Compare commits
78 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 90b76ddad3 | |||
| d5b3d1e061 | |||
| 41083de46a | |||
| 07ac7e54c5 | |||
| 879cb021b2 | |||
| 574f88bd6a | |||
| d5d7a222a4 | |||
| aabd89aea7 | |||
| 9189cbfc22 | |||
| 872c166ed7 | |||
| 6742def4d3 | |||
| 1b39f82117 | |||
| f7d821067f | |||
| a16f66cbc7 | |||
| 39485d87f6 | |||
| 61e5b0fe58 | |||
| f3c21c7d9f | |||
| bc5f5d3731 | |||
| 93a76f1d36 | |||
| 5c882f26a9 | |||
| 0d45ac6e3c | |||
| f7ad756fc3 | |||
| 4fe11b1b2b | |||
| 296f9b1817 | |||
| 7a8516b0c6 | |||
| 2df2f530b3 | |||
| 8b25ca72e5 | |||
| fa02791fe9 | |||
| 32767b4eb4 | |||
| e1e5351c5d | |||
| d60ef66f53 | |||
| 4669b73d38 | |||
| 6f91603168 | |||
| 3ad247039b | |||
| 32e2433628 | |||
| 712c6b92b8 | |||
| 89cafcefec | |||
| 1d7fac0a47 | |||
| 03d4f27d2b | |||
| 4ac2038331 | |||
| 14a7e7e04c | |||
| 5e522bad8b | |||
| 23872dd742 | |||
| 7ffbf5f6ca | |||
| d0dc30fcc9 | |||
| b38f7b4dc3 | |||
| 0512995931 | |||
| 049a5feadb | |||
| 1798e9c575 | |||
| 5d4bc8c2b9 | |||
| 63fb8fc132 | |||
| 4d4bb7a121 | |||
| 5dcb3cd4fc | |||
| efe346893c | |||
| c95d6fcfec | |||
| 0badb4364d | |||
| 1f63f8afbb | |||
| bc099a31ad | |||
| 4ff55221a3 | |||
| 8d8024099f | |||
| 18792fd8d1 | |||
| 8860aa033c | |||
| 3ca14d8b6e | |||
| 099e9c4a3e | |||
| cfe6b9408a | |||
| 6ececc749f | |||
| 79901fbb86 | |||
| 75fa0a030a | |||
| ef64966897 | |||
| c6c5e3cb69 | |||
| 3f4fd230b9 | |||
| 2091b58066 | |||
| c3fe88c9fa | |||
| 5830fda516 | |||
| 359e910572 | |||
| 4950a6bb14 | |||
| b69ba96811 | |||
| 941e2656e8 |
1
.gitignore
vendored
1
.gitignore
vendored
@@ -1,6 +1,5 @@
|
|||||||
# ---> Codex
|
# ---> Codex
|
||||||
.codex
|
.codex
|
||||||
AGENTS.md
|
|
||||||
|
|
||||||
# ---> Go
|
# ---> Go
|
||||||
# If you prefer the allow list template instead of the deny list, see community template:
|
# If you prefer the allow list template instead of the deny list, see community template:
|
||||||
|
|||||||
@@ -28,10 +28,6 @@ steps:
|
|||||||
|
|
||||||
build_binary linux amd64 ""
|
build_binary linux amd64 ""
|
||||||
build_binary linux arm64 ""
|
build_binary linux arm64 ""
|
||||||
build_binary darwin amd64 ""
|
|
||||||
build_binary darwin arm64 ""
|
|
||||||
build_binary windows amd64 ".exe"
|
|
||||||
build_binary windows arm64 ".exe"
|
|
||||||
|
|
||||||
- name: publish-release
|
- name: publish-release
|
||||||
image: woodpeckerci/plugin-release
|
image: woodpeckerci/plugin-release
|
||||||
|
|||||||
4
AGENTS.md
Normal file
4
AGENTS.md
Normal file
@@ -0,0 +1,4 @@
|
|||||||
|
Please carefully review the relevant documents in `docs/policy` before making any changes to this repository.
|
||||||
|
- `development.md` defines the contributor workflow for this application.
|
||||||
|
- `architecture.md` provides the canonical high-level architecture policy for this repository, and should be reviewed before writing or changing any code.
|
||||||
|
- `documentation.md` provides the canonical documentation policy for this repository, and should be reviewed before writing or changing any documentation.
|
||||||
2
LICENSE
2
LICENSE
@@ -1,4 +1,4 @@
|
|||||||
Copyright (c) 2026 eric.
|
Copyright (c) 2026 Eric Rakestraw.
|
||||||
|
|
||||||
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
|
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
|
||||||
|
|
||||||
|
|||||||
458
README.md
458
README.md
@@ -1,449 +1,45 @@
|
|||||||
# scriptorium
|
# scriptorium
|
||||||
|
|
||||||
Scriptorium is a generic prompt execution engine.
|
Scriptorium is a narrow prompt-execution application for rendering prompt
|
||||||
|
requests, running them against OpenAI-compatible chat-completions endpoints, and
|
||||||
|
serving the same run workflow over HTTP.
|
||||||
|
|
||||||
It takes:
|
It keeps prompt definitions, execution profiles, schemas, and input artifacts as
|
||||||
- a prompt definition
|
separate files so prompts can be reviewed and reused without baking model
|
||||||
- a selected or default execution profile
|
runtime settings into application code.
|
||||||
- named input artifacts
|
|
||||||
- template variables
|
|
||||||
- optional runtime overrides
|
|
||||||
|
|
||||||
It returns:
|
## Quickstart
|
||||||
- for `run`: generated artifact, validation result, metadata
|
|
||||||
- for `render`: prepared/rendered prompt data (no model output)
|
|
||||||
|
|
||||||
## Prompt vs Profile
|
From the repository root:
|
||||||
|
|
||||||
Scriptorium separates **what** to do (Prompt) from **how** to do it (Profile).
|
|
||||||
|
|
||||||
### Prompt Definition
|
|
||||||
Defines the task logic and output contract.
|
|
||||||
- Task description and version.
|
|
||||||
- Message templates (system, user, etc.).
|
|
||||||
- Required and optional input artifacts.
|
|
||||||
- Output format and validation rules.
|
|
||||||
- Repair settings for structured output.
|
|
||||||
- Optional `default_profile` for convenience.
|
|
||||||
|
|
||||||
### Execution Profile
|
|
||||||
Defines the runtime environment and model settings.
|
|
||||||
- LLM endpoint (URL).
|
|
||||||
- Model name.
|
|
||||||
- Generation parameters: `temperature`, `max_tokens`, `top_p`.
|
|
||||||
- Runtime settings: `timeout`, `reasoning_effort`.
|
|
||||||
- API key source via `api_key_env`.
|
|
||||||
|
|
||||||
Callers can explicitly provide a `profile_id` to override the prompt's `default_profile`.
|
|
||||||
|
|
||||||
## Precedence
|
|
||||||
|
|
||||||
Scriptorium uses two precedence layers:
|
|
||||||
|
|
||||||
### Application Configuration Precedence
|
|
||||||
|
|
||||||
For application-level adapter settings (for example prompt/profile/schema directories, server address, and render output default), precedence is:
|
|
||||||
|
|
||||||
1. **CLI Flags**
|
|
||||||
2. **`config.yml`**
|
|
||||||
3. **Built-in application defaults**
|
|
||||||
|
|
||||||
Application config loading behavior:
|
|
||||||
- Default config path: `/etc/scriptorium/config.yml`
|
|
||||||
- Override path: `--config <PATH>` (supported by `run`, `render`, and `serve`)
|
|
||||||
- If `--config` is provided, the file must exist and be valid.
|
|
||||||
- If `--config` is omitted, missing `/etc/scriptorium/config.yml` is allowed.
|
|
||||||
|
|
||||||
### Runtime Model Precedence
|
|
||||||
|
|
||||||
When resolving runtime model settings, Scriptorium follows this precedence model (highest to lowest):
|
|
||||||
|
|
||||||
1. **Runtime Overrides**: Provided via CLI flags or HTTP request `model` object.
|
|
||||||
2. **Execution Profile**: Settings defined in the selected profile.
|
|
||||||
3. **Application Defaults**: Built-in fallback values.
|
|
||||||
|
|
||||||
### Profile Selection Logic
|
|
||||||
The engine determines which profile to use in this order:
|
|
||||||
1. Explicit `profile_id` (via `--profile` or HTTP request).
|
|
||||||
2. The `default_profile` named in the Prompt Definition.
|
|
||||||
3. Error: If neither is provided and no default exists.
|
|
||||||
|
|
||||||
## API Key Policy
|
|
||||||
|
|
||||||
To ensure security, Scriptorium does not support raw API keys in configuration files, CLI arguments, or HTTP requests.
|
|
||||||
|
|
||||||
- **`api_key_env`**: Profiles and overrides specify the name of an environment variable (e.g., `SCRIPTORIUM_API_KEY`).
|
|
||||||
- **Runtime Resolution**: The value of the environment variable is read directly from the process environment at runtime.
|
|
||||||
- **Zero Leakage**: API key values are never included in metadata, logs, or response bodies.
|
|
||||||
|
|
||||||
## CLI Usage
|
|
||||||
|
|
||||||
Available CLI commands:
|
|
||||||
- `scriptorium run`
|
|
||||||
- `scriptorium render`
|
|
||||||
- `scriptorium serve`
|
|
||||||
|
|
||||||
All commands accept `--config <PATH>`.
|
|
||||||
|
|
||||||
`prompt_dir` and `profile_dir` may be supplied by CLI flags or `config.yml`:
|
|
||||||
- `--prompt-dir` or `config.yml` `prompt_dir`
|
|
||||||
- `--profile-dir` or `config.yml` `profile_dir`
|
|
||||||
|
|
||||||
`schema_dir` and `serve` `addr` may also be supplied by `config.yml` where applicable:
|
|
||||||
- `--schema-dir` or `config.yml` `schema_dir`
|
|
||||||
- `--addr` or `config.yml` `server.addr`
|
|
||||||
|
|
||||||
### `scriptorium run`
|
|
||||||
|
|
||||||
Runs a single prompt execution.
|
|
||||||
|
|
||||||
**Required Flags:**
|
|
||||||
- `--prompt`: The prompt ID to execute.
|
|
||||||
- `--input`: Input mapping `name=path` (repeatable).
|
|
||||||
|
|
||||||
**Required Effective Settings:**
|
|
||||||
- Prompt directory: `--prompt-dir` or `config.yml` `prompt_dir`
|
|
||||||
- Profile directory: `--profile-dir` or `config.yml` `profile_dir`
|
|
||||||
|
|
||||||
**Optional Flags:**
|
|
||||||
- `--config`: Application config file path. Default discovery path is `/etc/scriptorium/config.yml`.
|
|
||||||
- `--prompt-dir`: Override prompt directory from config.
|
|
||||||
- `--profile-dir`: Override profile directory from config.
|
|
||||||
- `--profile`: Override the prompt's default profile.
|
|
||||||
- `--var`: Template variable `name=value` (repeatable).
|
|
||||||
- `--out`: Write output to a file instead of stdout.
|
|
||||||
- `--llm-base-url`: Override endpoint.
|
|
||||||
- `--model`: Override model name.
|
|
||||||
- `--api-key-env`: Override API key environment variable name.
|
|
||||||
- `--temperature`: Override temperature.
|
|
||||||
- `--max-tokens`: Override max tokens.
|
|
||||||
- `--top-p`: Override top_p.
|
|
||||||
- `--timeout`: Override request timeout (e.g., `30s`, `1m`).
|
|
||||||
- `--schema-dir`: Base directory for validation schemas.
|
|
||||||
|
|
||||||
**Examples:**
|
|
||||||
|
|
||||||
Using `config.yml` for prompt/profile directories:
|
|
||||||
```bash
|
```bash
|
||||||
scriptorium run \
|
go run ./cmd/scriptorium render \
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Overriding config directories explicitly:
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Overriding the profile:
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--profile local-quality \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Overriding model and runtime values:
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--model gpt-4o \
|
|
||||||
--temperature 0.7 \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Using a local OpenAI-compatible vLLM endpoint:
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--llm-base-url http://localhost:8000/v1 \
|
|
||||||
--model meta-llama-3-8b \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
### `scriptorium render`
|
|
||||||
|
|
||||||
Prepares and renders a prompt without calling the LLM.
|
|
||||||
|
|
||||||
`render` uses the same prompt/profile/input/variable/runtime override resolution as `run`:
|
|
||||||
- Profile selection precedence: `--profile` -> prompt `default_profile` -> error.
|
|
||||||
- Runtime precedence: CLI runtime overrides -> selected profile -> built-in defaults.
|
|
||||||
|
|
||||||
`render` is useful for debugging:
|
|
||||||
- prompt template rendering
|
|
||||||
- input mappings
|
|
||||||
- selected profile behavior
|
|
||||||
- runtime override behavior
|
|
||||||
|
|
||||||
`render` does not:
|
|
||||||
- call the LLM
|
|
||||||
- validate model output
|
|
||||||
- perform repair
|
|
||||||
- expose resolved API key values
|
|
||||||
|
|
||||||
It may include `api_key_env` names where relevant.
|
|
||||||
|
|
||||||
**Required Flags:**
|
|
||||||
- `--prompt`: The prompt ID to render.
|
|
||||||
- `--input`: Input mapping `name=path` (repeatable).
|
|
||||||
|
|
||||||
**Required Effective Settings:**
|
|
||||||
- Prompt directory: `--prompt-dir` or `config.yml` `prompt_dir`
|
|
||||||
- Profile directory: `--profile-dir` or `config.yml` `profile_dir`
|
|
||||||
|
|
||||||
**Optional Flags:**
|
|
||||||
- `--config`: Application config file path. Default discovery path is `/etc/scriptorium/config.yml`.
|
|
||||||
- `--prompt-dir`: Override prompt directory from config.
|
|
||||||
- `--profile-dir`: Override profile directory from config.
|
|
||||||
- `--profile`: Override the prompt's default profile.
|
|
||||||
- `--var`: Template variable `name=value` (repeatable).
|
|
||||||
- `--out`: Write output to a file instead of stdout.
|
|
||||||
- `--format`: Render output format (`text` or `json`). Default: `text`.
|
|
||||||
- `--llm-base-url`: Runtime override for endpoint.
|
|
||||||
- `--model`: Runtime override for model name.
|
|
||||||
- `--api-key-env`: Runtime override for API key environment variable name.
|
|
||||||
- `--temperature`: Runtime override for temperature.
|
|
||||||
- `--max-tokens`: Runtime override for max tokens.
|
|
||||||
- `--top-p`: Runtime override for top_p.
|
|
||||||
- `--timeout`: Runtime override for timeout (e.g., `30s`, `1m`).
|
|
||||||
|
|
||||||
**Render Output Formats:**
|
|
||||||
- `text`: Human-readable output (default).
|
|
||||||
- `json`: Machine-readable structured output.
|
|
||||||
|
|
||||||
Render formatting is modular; additional output formats can be added later without changing prepare/run core logic.
|
|
||||||
|
|
||||||
**Examples:**
|
|
||||||
|
|
||||||
Default text output using `config.yml` directories:
|
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Explicit config path:
|
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--config ./examples/config.yml \
|
--config ./examples/config.yml \
|
||||||
--prompt generic.markdown_summary \
|
--prompt generic.markdown_summary \
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Explicit directory overrides:
|
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Explicit JSON output:
|
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md \
|
--input transcript=./examples/fixtures/transcript.md \
|
||||||
|
--input glossary=./examples/fixtures/glossary.yml \
|
||||||
--format json
|
--format json
|
||||||
```
|
```
|
||||||
|
|
||||||
Using prompt `default_profile` (omit `--profile`):
|
This command renders the prepared prompt and effective runtime settings without calling an LLM.
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Overriding profile selection:
|
## Documentation
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--profile local-quality \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Overriding runtime settings:
|
- [CLI reference](docs/cli.md)
|
||||||
```bash
|
- [Configuration reference](docs/config.md)
|
||||||
scriptorium render \
|
- [HTTP API reference](docs/api.md)
|
||||||
--prompt-dir ./prompts \
|
- [Operations guide](docs/operations.md)
|
||||||
--profile-dir ./profiles \
|
- [Troubleshooting](docs/troubleshooting.md)
|
||||||
--prompt generic.markdown_summary \
|
- [Consumer integration overview](docs/consumers/api.md)
|
||||||
--input transcript=./examples/fixtures/transcript.md \
|
- [Go library package](docs/consumers/pkg-scriptorium.md)
|
||||||
--llm-base-url http://localhost:8000/v1 \
|
- [Subprocess integration](docs/integrations/subprocess.md)
|
||||||
--model gpt-4o-mini \
|
- [OpenAI-compatible chat integration](docs/integrations/openai-compatible-chat.md)
|
||||||
--temperature 0.2 \
|
- [Architecture policy](docs/policy/architecture.md)
|
||||||
--max-tokens 800 \
|
|
||||||
--top-p 1.0 \
|
|
||||||
--timeout 45s
|
|
||||||
```
|
|
||||||
|
|
||||||
Writing rendered output to a file:
|
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md \
|
|
||||||
--format text \
|
|
||||||
--out ./rendered_prompt.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
### `scriptorium serve`
|
|
||||||
|
|
||||||
Starts the HTTP API.
|
|
||||||
|
|
||||||
**Required Effective Settings:**
|
|
||||||
- Prompt directory: `--prompt-dir` or `config.yml` `prompt_dir`
|
|
||||||
- Profile directory: `--profile-dir` or `config.yml` `profile_dir`
|
|
||||||
|
|
||||||
**Optional Flags:**
|
|
||||||
- `--config`: Application config file path. Default discovery path is `/etc/scriptorium/config.yml`.
|
|
||||||
- `--addr`: Listen address (default `:8080`).
|
|
||||||
- `--schema-dir`: Base directory for validation schemas.
|
|
||||||
|
|
||||||
**Examples:**
|
|
||||||
|
|
||||||
Using `config.yml`:
|
|
||||||
```bash
|
|
||||||
scriptorium serve
|
|
||||||
```
|
|
||||||
|
|
||||||
Overriding config for local use:
|
|
||||||
```bash
|
|
||||||
scriptorium serve \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--addr :9090
|
|
||||||
```
|
|
||||||
|
|
||||||
## HTTP API
|
|
||||||
|
|
||||||
### `POST /v1/runs`
|
|
||||||
|
|
||||||
Executes a prompt. No built-in authentication is provided; deploy behind a trusted gateway.
|
|
||||||
|
|
||||||
**Request Body:**
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"prompt_id": "generic.structured_events",
|
|
||||||
"profile_id": "local-quality",
|
|
||||||
"include_raw_output": false,
|
|
||||||
"inputs": {
|
|
||||||
"transcript": {"type": "file", "uri": "./examples/fixtures/transcript.md"}
|
|
||||||
},
|
|
||||||
"vars": {
|
|
||||||
"session_date": "2026-05-04"
|
|
||||||
},
|
|
||||||
"model": {
|
|
||||||
"endpoint": "http://localhost:8000/v1",
|
|
||||||
"model": "gpt-4o-mini",
|
|
||||||
"temperature": 0.0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
`profile_id` is optional. If omitted, Scriptorium uses the prompt's `default_profile`. If neither is available, the run fails.
|
|
||||||
|
|
||||||
**Response:**
|
|
||||||
Returns a `200 OK` with the generated artifact, validation results, and metadata including the `prompt_id` and the `selected_profile_id`.
|
|
||||||
|
|
||||||
**Validation Failures:**
|
|
||||||
If the model output fails validation (e.g., invalid JSON), the API returns `200 OK` with `validation.status = "failed"`.
|
|
||||||
|
|
||||||
**Raw Output Exposure:**
|
|
||||||
- `raw_model_output` is omitted by default.
|
|
||||||
- Set `include_raw_output: true` in the request to include it in the response.
|
|
||||||
- Raw output is preserved internally in run results regardless of HTTP exposure.
|
|
||||||
|
|
||||||
## Prompt Definition Authoring
|
|
||||||
|
|
||||||
Prompts are defined in YAML.
|
|
||||||
|
|
||||||
### Canonical Shape
|
|
||||||
```yaml
|
|
||||||
id: generic.structured_events
|
|
||||||
version: "1.0.0"
|
|
||||||
description: "Extracts structured events from a transcript"
|
|
||||||
default_profile: local-quality
|
|
||||||
|
|
||||||
inputs:
|
|
||||||
- name: transcript
|
|
||||||
required: true
|
|
||||||
content_type: text/markdown
|
|
||||||
description: "The raw session transcript"
|
|
||||||
- name: glossary
|
|
||||||
required: false
|
|
||||||
content_type: application/yaml
|
|
||||||
description: "Optional glossary terms"
|
|
||||||
|
|
||||||
messages:
|
|
||||||
- role: system
|
|
||||||
content: "You are a helpful assistant."
|
|
||||||
- role: user
|
|
||||||
content_file: messages/extract_events.tmpl
|
|
||||||
|
|
||||||
output:
|
|
||||||
format: json
|
|
||||||
validation_mode: json_schema
|
|
||||||
schema_path: structured_events.schema.json
|
|
||||||
repair_attempts: 2
|
|
||||||
```
|
|
||||||
|
|
||||||
**Key Features:**
|
|
||||||
- **Inline vs File**: Use `content` for short prompts or `content_file` for larger templates. Exactly one must be set per message.
|
|
||||||
- **Path Resolution**: `content_file` paths are resolved relative to the prompt YAML file.
|
|
||||||
- **Inputs**: Mark inputs as `required` to ensure the runner fails early if they are missing.
|
|
||||||
- **Input Metadata**: `content_type` is currently descriptive metadata and not enforced yet.
|
|
||||||
- **Validation**: Support `none`, `basic`, `json`, and `json_schema`.
|
|
||||||
- **Repair**: `repair_attempts` enables bounded retries to fix structured output.
|
|
||||||
|
|
||||||
## Execution Profile Authoring
|
|
||||||
|
|
||||||
Profiles are defined in YAML.
|
|
||||||
|
|
||||||
### Canonical Shape
|
|
||||||
```yaml
|
|
||||||
id: local-quality
|
|
||||||
endpoint: http://localhost:8000/v1
|
|
||||||
model: gpt-4o
|
|
||||||
temperature: 0.0
|
|
||||||
max_tokens: 4096
|
|
||||||
top_p: 1.0
|
|
||||||
timeout_seconds: 300
|
|
||||||
reasoning_effort: high
|
|
||||||
api_key_env: SCRIPTORIUM_API_KEY
|
|
||||||
```
|
|
||||||
|
|
||||||
**Constraints:**
|
|
||||||
- **No Raw Keys**: Do not include actual API keys. Only specify the environment variable name in `api_key_env`.
|
|
||||||
- **Local Profiles**: For local endpoints that don't require auth, `api_key_env` can be omitted.
|
|
||||||
|
|
||||||
## Examples
|
## Examples
|
||||||
|
|
||||||
- **Prompt Definitions**: `prompts/`
|
- `examples/config.yml`
|
||||||
- **Execution Profiles**: `profiles/`
|
- `examples/config.full.yml`
|
||||||
- **Schemas**: `schemas/`
|
- `examples/render-markdown-summary.sh`
|
||||||
- **Fixtures**: `examples/fixtures/`
|
- `examples/http-run.json`
|
||||||
|
- `examples/go-library/prepare`
|
||||||
## Build and Test
|
|
||||||
|
|
||||||
```bash
|
|
||||||
go build -o scriptorium ./cmd/scriptorium
|
|
||||||
go test ./...
|
|
||||||
```
|
|
||||||
|
|||||||
483
architecture.md
483
architecture.md
@@ -1,483 +0,0 @@
|
|||||||
# Scriptorium Architecture
|
|
||||||
|
|
||||||
## 1. Purpose and Non-Goals
|
|
||||||
|
|
||||||
Scriptorium is a prompt-definition execution engine.
|
|
||||||
|
|
||||||
It accepts named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata.
|
|
||||||
|
|
||||||
Scriptorium also supports rendering/preparing a prompt without calling an LLM. This allows users to inspect the fully rendered prompt messages and effective runtime settings before executing a run.
|
|
||||||
|
|
||||||
Scriptorium is not an orchestrator. It must not own transcription, transcript merge/polish steps, notifications, or cross-step workflow control.
|
|
||||||
|
|
||||||
For the motivating D&D workflow:
|
|
||||||
- Narratio orchestrates.
|
|
||||||
- WhisperX transcribes.
|
|
||||||
- Seriatim merges transcripts.
|
|
||||||
- Audita polishes transcripts.
|
|
||||||
- Scriptorium generates final artifacts from prepared inputs.
|
|
||||||
|
|
||||||
Core Go code remains generic.
|
|
||||||
|
|
||||||
## 2. Current Architecture
|
|
||||||
|
|
||||||
Scriptorium uses a ports-and-adapters architecture to decouple the core execution logic from external dependencies.
|
|
||||||
|
|
||||||
### Package Responsibilities
|
|
||||||
|
|
||||||
- `cmd/scriptorium`: Binary entrypoint for CLI and HTTP server.
|
|
||||||
- `internal/domain`: Core domain contracts, including `PromptDefinition`, `ExecutionProfile`, `PreparedRun`, `RunResult`, and related metadata.
|
|
||||||
- `internal/usecase`: `Runner` use case logic, including prompt preparation, profile selection, runtime override resolution, full run execution, validation, and bounded repair.
|
|
||||||
- `internal/config`: Application-level config model/loader for adapter settings (for example prompt/profile/schema directories, server address, and render format default).
|
|
||||||
- `internal/promptdef`: Repository for loading and validating Prompt Definitions from the filesystem.
|
|
||||||
- `internal/profile`: Repository for loading Execution Profiles from the filesystem.
|
|
||||||
- `internal/artifact`: Input artifact resolution (`inline`, `file`).
|
|
||||||
- `internal/prompt`: Template rendering via Go templates.
|
|
||||||
- `internal/llm`: Provider-neutral client interface and OpenAI-compatible HTTP adapter.
|
|
||||||
- `internal/validate`: Output validation implementation (`none/basic/json/json_schema`).
|
|
||||||
- `internal/adapter/cli`: CLI flag parsing, command dispatch, and output handling.
|
|
||||||
- `internal/adapter/http`: HTTP request/response mapping.
|
|
||||||
- `internal/format` or equivalent: Formatting of prepared/rendered prompt output for CLI or other adapters, if formatting grows beyond simple CLI-local helpers.
|
|
||||||
|
|
||||||
Exact package names may evolve, but the architectural boundaries should remain stable.
|
|
||||||
|
|
||||||
### Application Configuration
|
|
||||||
|
|
||||||
`config.yml` is adapter/application setup, not domain logic.
|
|
||||||
|
|
||||||
Application config is intended for application-level settings such as:
|
|
||||||
- `prompt_dir`
|
|
||||||
- `profile_dir`
|
|
||||||
- `schema_dir`
|
|
||||||
- `server.addr`
|
|
||||||
- `defaults.render_format`
|
|
||||||
|
|
||||||
Application config precedence is:
|
|
||||||
1. CLI flags
|
|
||||||
2. `config.yml`
|
|
||||||
3. Built-in application defaults
|
|
||||||
|
|
||||||
Runtime model settings are intentionally separate:
|
|
||||||
- Execution profiles and runtime overrides continue to own endpoint/model/runtime behavior.
|
|
||||||
- `config.yml` does not replace execution profiles.
|
|
||||||
|
|
||||||
The core use case (`Runner.Prepare`/`Runner.Run`) does not need to know whether adapter-level settings came from CLI flags or `config.yml`; it receives resolved dependencies and requests from adapters.
|
|
||||||
|
|
||||||
## 3. Core Execution Model
|
|
||||||
|
|
||||||
Scriptorium has two closely related execution paths:
|
|
||||||
|
|
||||||
1. Prepare/render path.
|
|
||||||
2. Full run path.
|
|
||||||
|
|
||||||
The full run path should reuse the prepare path rather than duplicating its logic.
|
|
||||||
|
|
||||||
### 3.1 Prepare / Render Data Flow
|
|
||||||
|
|
||||||
The prepare path should be represented in the use case layer, preferably as `Runner.Prepare(ctx, RunRequest)` or an equivalent method.
|
|
||||||
|
|
||||||
It executes all pre-LLM work:
|
|
||||||
|
|
||||||
1. **Validate Request**: Ensure the request includes a prompt ID and all minimum required fields.
|
|
||||||
2. **Load Prompt Definition**: Retrieve the `PromptDefinition` by ID from the prompt repository.
|
|
||||||
3. **Select Profile**: Determine the `profile_id` using this precedence:
|
|
||||||
- Explicit `profile_id` in `RunRequest`.
|
|
||||||
- `default_profile` specified in the `PromptDefinition`.
|
|
||||||
- Error if neither is available.
|
|
||||||
4. **Load Execution Profile**: Retrieve the `ExecutionProfile` from the profile repository.
|
|
||||||
5. **Resolve Runtime Overrides**: Merge settings based on precedence, highest to lowest:
|
|
||||||
- Runtime overrides from CLI flags or HTTP request `model` object.
|
|
||||||
- Execution Profile settings.
|
|
||||||
- Built-in application defaults.
|
|
||||||
6. **Resolve Artifacts**: Load all named input artifacts defined in the request.
|
|
||||||
7. **Render Prompt**: Apply template variables and input artifacts to the prompt templates.
|
|
||||||
8. **Compute Metadata**: Compute hashes, selected profile ID, prompt ID/version, effective runtime settings, input hashes, rendered prompt hash, and timing information as appropriate.
|
|
||||||
9. **Return PreparedRun**: Return a `PreparedRun` containing the rendered messages, effective runtime settings, resolved metadata, and input/prompt hashes.
|
|
||||||
|
|
||||||
The prepare path must not call the LLM.
|
|
||||||
|
|
||||||
The prepare path must not validate model output, because there is no model output.
|
|
||||||
|
|
||||||
The prepare path must not perform structured-output repair, because repair only applies after model output exists.
|
|
||||||
|
|
||||||
The prepare path should not resolve or expose raw API key values. It may include the selected `api_key_env` name in effective runtime settings or metadata, but never the environment variable value.
|
|
||||||
|
|
||||||
### 3.2 Full Run Data Flow
|
|
||||||
|
|
||||||
The `Runner.Run(ctx, RunRequest)` flow should reuse the prepare path:
|
|
||||||
|
|
||||||
1. **Prepare**: Call the shared prepare flow to load the prompt, select the profile, resolve artifacts, render prompt messages, and compute pre-run metadata.
|
|
||||||
2. **Call LLM**: Execute the generation request using the effective runtime settings from the prepared run.
|
|
||||||
3. **Build Output Artifact**: Convert the model response into the configured output artifact.
|
|
||||||
4. **Validate Output**:
|
|
||||||
- Validate the model output against the prompt definition's output contract.
|
|
||||||
- Validation content failures remain successful run results with `validation.status=failed`.
|
|
||||||
- Validator runtime/config errors are run errors.
|
|
||||||
5. **Repair If Configured**:
|
|
||||||
- If structured validation fails and `repair_attempts > 0`, perform bounded repair attempts.
|
|
||||||
- Re-validate after each repair attempt.
|
|
||||||
- Repair loops must remain strictly bounded.
|
|
||||||
6. **Return RunResult**: Produce a `RunResult` containing the final artifact, validation status, raw model output, usage information, and metadata.
|
|
||||||
|
|
||||||
`Runner.Run` should not duplicate profile selection, artifact resolution, or prompt rendering logic that already exists in `Runner.Prepare`.
|
|
||||||
|
|
||||||
## 4. Domain Model
|
|
||||||
|
|
||||||
Key domain types:
|
|
||||||
|
|
||||||
- `PromptDefinition`: Defines the "what" of the task: templates, inputs, output contract, validation settings, repair settings, and optional `default_profile`.
|
|
||||||
- `ExecutionProfile`: Defines the "how" of execution: endpoint, model, generation parameters, timeout, reasoning effort, and `api_key_env`.
|
|
||||||
- `RunRequest`: The intent to execute or prepare a prompt, including `prompt_id`, optional `profile_id`, inputs, variables, and optional runtime overrides.
|
|
||||||
- `PreparedRun`: The result of the prepare/render phase. Contains rendered messages, effective runtime settings, selected profile ID, prompt metadata, input hashes, prompt hash, and other pre-LLM metadata.
|
|
||||||
- `RunResult`: The result of a full run. Contains the generated `Artifact`, `ValidationResult`, raw model output, token usage, and `RunMetadata`.
|
|
||||||
- `RunMetadata`: Detailed tracing information, including prompt ID/version, selected profile ID, effective model parameters, usage tokens, hashes, timestamps, validation status, and repair attempts where applicable.
|
|
||||||
- `RenderedPrompt`: Provider-neutral rendered prompt structure.
|
|
||||||
- `RenderedMessage`: Provider-neutral rendered message with role and content.
|
|
||||||
- `ArtifactRef`: A reference to an input artifact, such as `file` or `inline`.
|
|
||||||
- `Artifact`: Loaded artifact content with name, content type, body, source URI, size, and hash.
|
|
||||||
- `ValidationResult`: Validation status and details for full runs.
|
|
||||||
|
|
||||||
`PreparedRun` should be serializable for JSON output and should also be representable in a human-readable text format.
|
|
||||||
|
|
||||||
## 5. Interfaces and Adapters
|
|
||||||
|
|
||||||
### Primary Ports
|
|
||||||
|
|
||||||
- `promptdef.Repository`: Lookup for prompt definitions.
|
|
||||||
- `profile.Repository`: Lookup for execution profiles.
|
|
||||||
- `artifact.Reader`: Loading of artifact content.
|
|
||||||
- `prompt.Renderer`: Template rendering.
|
|
||||||
- `llm.Client`: Model generation.
|
|
||||||
- `validate.Validator`: Output validation.
|
|
||||||
- `format.PreparedRunFormatter` or equivalent: Optional formatting abstraction for rendered/prepared output.
|
|
||||||
|
|
||||||
### Current Adapters
|
|
||||||
|
|
||||||
- **Repositories**: Filesystem YAML loaders for both prompts and profiles.
|
|
||||||
- **Artifact Reader**: Composite reader supporting `file` and `inline`.
|
|
||||||
- **Prompt Renderer**: Go templates with a custom `input` helper.
|
|
||||||
- **LLM Client**: OpenAI-compatible `/chat/completions` over HTTP.
|
|
||||||
- **Validator**: Standard validator supporting `none`, `basic`, `json`, and `json_schema`.
|
|
||||||
- **CLI Adapter**: Supports `run`, `render`, and `serve`.
|
|
||||||
- **HTTP Adapter**: Supports full run execution through `POST /v1/runs`.
|
|
||||||
|
|
||||||
## 6. Public Contracts
|
|
||||||
|
|
||||||
### CLI
|
|
||||||
|
|
||||||
Scriptorium should expose at least these commands:
|
|
||||||
|
|
||||||
- `scriptorium run`: Executes a prompt by preparing it, calling the LLM, validating output, optionally repairing structured output, and returning an artifact.
|
|
||||||
- `scriptorium render`: Prepares and renders a prompt without calling the LLM.
|
|
||||||
- `scriptorium serve`: Starts the HTTP API using infrastructure-only flags.
|
|
||||||
|
|
||||||
### `scriptorium run`
|
|
||||||
|
|
||||||
`run` uses flags such as:
|
|
||||||
|
|
||||||
- `--prompt-dir`
|
|
||||||
- `--profile-dir`
|
|
||||||
- `--prompt`
|
|
||||||
- `--profile`
|
|
||||||
- `--input`
|
|
||||||
- `--var`
|
|
||||||
- `--out`
|
|
||||||
- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.
|
|
||||||
|
|
||||||
`run` should produce the generated artifact as its primary output.
|
|
||||||
|
|
||||||
### `scriptorium render`
|
|
||||||
|
|
||||||
`render` prepares and renders a prompt without calling an LLM.
|
|
||||||
|
|
||||||
It should use the same prompt/profile/input/variable/runtime override flags as `run` where applicable:
|
|
||||||
|
|
||||||
- `--prompt-dir`
|
|
||||||
- `--profile-dir`
|
|
||||||
- `--prompt`
|
|
||||||
- `--profile`
|
|
||||||
- `--input`
|
|
||||||
- `--var`
|
|
||||||
- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.
|
|
||||||
- `--format`, with initial support for `text` and `json`.
|
|
||||||
|
|
||||||
Default render output format should be `text`.
|
|
||||||
|
|
||||||
`render` must not call the LLM.
|
|
||||||
|
|
||||||
`render` should show the same rendered messages and effective runtime settings that `run` would use.
|
|
||||||
|
|
||||||
`render` should include enough information to debug:
|
|
||||||
|
|
||||||
- prompt ID
|
|
||||||
- prompt version
|
|
||||||
- selected profile ID
|
|
||||||
- effective runtime settings
|
|
||||||
- input hashes
|
|
||||||
- prompt hash
|
|
||||||
- rendered messages
|
|
||||||
|
|
||||||
`render` must not include resolved API key values.
|
|
||||||
|
|
||||||
`render` may include the `api_key_env` name.
|
|
||||||
|
|
||||||
### Render Output Formats
|
|
||||||
|
|
||||||
Initial render output formats:
|
|
||||||
|
|
||||||
- `text`: Human-readable default format.
|
|
||||||
- `json`: Machine-readable structured representation of the prepared run.
|
|
||||||
|
|
||||||
Additional formats, such as `markdown`, may be added later.
|
|
||||||
|
|
||||||
Render output formatting should be modular. Adding a new output format should not require changing the prepare/run core logic.
|
|
||||||
|
|
||||||
The output formatting layer should consume a `PreparedRun` and produce bytes or text for the adapter. It should not reload prompts, re-resolve artifacts, re-render templates, call the LLM, or perform validation.
|
|
||||||
|
|
||||||
### `scriptorium serve`
|
|
||||||
|
|
||||||
`serve` starts the HTTP API.
|
|
||||||
|
|
||||||
It should use infrastructure-only flags such as:
|
|
||||||
|
|
||||||
- `--addr`
|
|
||||||
- `--prompt-dir`
|
|
||||||
- `--profile-dir`
|
|
||||||
- `--schema-dir`
|
|
||||||
|
|
||||||
`serve` should not introduce a server-level model/runtime precedence layer unless explicitly documented and intentionally implemented.
|
|
||||||
|
|
||||||
### HTTP API
|
|
||||||
|
|
||||||
Current HTTP API:
|
|
||||||
|
|
||||||
- `POST /v1/runs`: Accepts `RunRequest` JSON and returns `RunResponse` JSON. No built-in auth.
|
|
||||||
|
|
||||||
Request may include runtime overrides under `model` and an `include_raw_output` boolean.
|
|
||||||
|
|
||||||
`raw_model_output` is exposed only when explicitly requested with `include_raw_output=true`.
|
|
||||||
|
|
||||||
A future HTTP prepare/render endpoint may be added, such as `POST /v1/renders` or `POST /v1/runs/prepare`, but the initial render feature may be CLI-only. If added later, it should call the same usecase-level prepare path as `scriptorium render`.
|
|
||||||
|
|
||||||
### YAML Shapes
|
|
||||||
|
|
||||||
Prompt YAML includes:
|
|
||||||
|
|
||||||
- `id`
|
|
||||||
- `version`
|
|
||||||
- optional `default_profile`
|
|
||||||
- `inputs`
|
|
||||||
- `messages`
|
|
||||||
- `output`
|
|
||||||
|
|
||||||
Inputs support:
|
|
||||||
|
|
||||||
- `name`
|
|
||||||
- `required`
|
|
||||||
- optional `content_type`
|
|
||||||
- `description`
|
|
||||||
|
|
||||||
Messages require:
|
|
||||||
|
|
||||||
- `role`
|
|
||||||
- exactly one of `content` or `content_file`
|
|
||||||
|
|
||||||
`content_file` resolves relative to the prompt YAML location.
|
|
||||||
|
|
||||||
Profile YAML includes:
|
|
||||||
|
|
||||||
- `id`
|
|
||||||
- `endpoint`
|
|
||||||
- `model`
|
|
||||||
- generation parameters
|
|
||||||
- timeout settings
|
|
||||||
- reasoning settings
|
|
||||||
- `api_key_env`
|
|
||||||
|
|
||||||
Prompt content must not appear in profile YAML.
|
|
||||||
|
|
||||||
Model/runtime/API-key settings must not appear in prompt YAML, except that prompt YAML may specify `default_profile`.
|
|
||||||
|
|
||||||
## 7. Render Feature Design
|
|
||||||
|
|
||||||
The render feature is a first-class use case, not a CLI-only shortcut.
|
|
||||||
|
|
||||||
### Goals
|
|
||||||
|
|
||||||
The render feature should help users:
|
|
||||||
|
|
||||||
- inspect fully rendered prompt messages
|
|
||||||
- debug missing inputs
|
|
||||||
- verify template variable substitution
|
|
||||||
- verify selected profile resolution
|
|
||||||
- verify runtime override precedence
|
|
||||||
- verify file-backed prompt loading
|
|
||||||
- inspect input hashes and prompt hashes
|
|
||||||
- prepare for future token budgeting and prompt-size inspection
|
|
||||||
|
|
||||||
### Non-Goals
|
|
||||||
|
|
||||||
The render feature should not:
|
|
||||||
|
|
||||||
- call an LLM
|
|
||||||
- validate model output
|
|
||||||
- repair structured output
|
|
||||||
- resolve or print API key values
|
|
||||||
- mutate artifacts
|
|
||||||
- save outputs to artifact storage unless a future explicit output option is added
|
|
||||||
- become an orchestration step manager
|
|
||||||
|
|
||||||
### Usecase Shape
|
|
||||||
|
|
||||||
The preferred usecase shape is:
|
|
||||||
|
|
||||||
- `Runner.Prepare(ctx, RunRequest) (*PreparedRun, error)`
|
|
||||||
- `Runner.Run(ctx, RunRequest) (*RunResult, error)`
|
|
||||||
|
|
||||||
`Runner.Run` should call `Runner.Prepare`.
|
|
||||||
|
|
||||||
The prepare flow should be the only implementation of:
|
|
||||||
|
|
||||||
- prompt loading
|
|
||||||
- profile selection
|
|
||||||
- runtime override resolution
|
|
||||||
- artifact resolution
|
|
||||||
- prompt rendering
|
|
||||||
- pre-run metadata/hash calculation
|
|
||||||
|
|
||||||
### CLI Shape
|
|
||||||
|
|
||||||
The preferred command name is `render`.
|
|
||||||
|
|
||||||
The command should support:
|
|
||||||
|
|
||||||
- `--format text`
|
|
||||||
- `--format json`
|
|
||||||
|
|
||||||
Default format:
|
|
||||||
|
|
||||||
- `text`
|
|
||||||
|
|
||||||
Unknown formats should produce a clear error.
|
|
||||||
|
|
||||||
Formatting should be centralized through a small formatter registry, strategy, switch, or interface so new formats can be added without modifying usecase logic.
|
|
||||||
|
|
||||||
### Text Output Expectations
|
|
||||||
|
|
||||||
Text output should be optimized for human inspection.
|
|
||||||
|
|
||||||
It should include, at minimum:
|
|
||||||
|
|
||||||
- prompt ID and version
|
|
||||||
- selected profile ID
|
|
||||||
- model name
|
|
||||||
- endpoint
|
|
||||||
- effective generation settings
|
|
||||||
- input hashes
|
|
||||||
- rendered prompt hash
|
|
||||||
- rendered messages grouped by role
|
|
||||||
|
|
||||||
Text output should be readable and deterministic enough for tests.
|
|
||||||
|
|
||||||
It should not include raw API key values.
|
|
||||||
|
|
||||||
### JSON Output Expectations
|
|
||||||
|
|
||||||
JSON output should be a structured representation of `PreparedRun` or a DTO derived from it.
|
|
||||||
|
|
||||||
It should include, at minimum:
|
|
||||||
|
|
||||||
- prompt ID and version
|
|
||||||
- selected profile ID
|
|
||||||
- effective runtime settings
|
|
||||||
- input hashes
|
|
||||||
- rendered prompt hash
|
|
||||||
- rendered messages
|
|
||||||
|
|
||||||
JSON output should not include raw API key values.
|
|
||||||
|
|
||||||
JSON output should remain stable enough to be useful for automation and integration tests.
|
|
||||||
|
|
||||||
## 8. Guardrails
|
|
||||||
|
|
||||||
- **Separation of Concerns**: Prompt content must not belong in execution profiles; model/API settings must not belong in prompt definitions.
|
|
||||||
- **Security**: Raw API keys are unsupported in all configuration and transport layers. Only `api_key_env` is used.
|
|
||||||
- **Secret Handling**: Resolved API key values must never appear in rendered output, metadata, logs, HTTP responses, or CLI output.
|
|
||||||
- **Path Resolution**: `content_file` paths in prompt definitions resolve relative to the prompt YAML file.
|
|
||||||
- **Integrity**: No silent prompt truncation or omission of content.
|
|
||||||
- **Reliability**: Repair loops are strictly bounded by `repair_attempts`.
|
|
||||||
- **No Orchestration Creep**: Scriptorium prepares and executes a single prompt request. It does not coordinate multi-stage workflows.
|
|
||||||
- **Render Reuse**: The full run path must reuse the prepare/render path to avoid divergent behavior.
|
|
||||||
- **Formatter Isolation**: Render output formatters must not perform usecase work. They only format a completed `PreparedRun`.
|
|
||||||
|
|
||||||
## 9. Testing Strategy
|
|
||||||
|
|
||||||
Tests should protect both the run path and the prepare/render path.
|
|
||||||
|
|
||||||
### Prepare / Render Tests
|
|
||||||
|
|
||||||
Add tests for:
|
|
||||||
|
|
||||||
- preparing a prompt with explicit profile selection
|
|
||||||
- preparing a prompt using `default_profile`
|
|
||||||
- failing when no explicit profile and no `default_profile` exist
|
|
||||||
- runtime overrides beating profile values
|
|
||||||
- profile values beating application defaults
|
|
||||||
- file-backed prompt bodies rendering correctly
|
|
||||||
- required inputs failing when missing
|
|
||||||
- optional inputs being absent when not referenced
|
|
||||||
- unknown input references failing
|
|
||||||
- input hashes being included
|
|
||||||
- rendered prompt hash being included
|
|
||||||
- effective runtime settings being included
|
|
||||||
- `api_key_env` name being included where appropriate
|
|
||||||
- resolved API key values never appearing in `PreparedRun`
|
|
||||||
- prepare path not calling the LLM
|
|
||||||
|
|
||||||
### CLI Render Tests
|
|
||||||
|
|
||||||
Add tests for:
|
|
||||||
|
|
||||||
- `scriptorium render` mapping flags into `RunRequest`
|
|
||||||
- default text output
|
|
||||||
- explicit `--format text`
|
|
||||||
- explicit `--format json`
|
|
||||||
- unknown format failure
|
|
||||||
- text output includes prompt/profile/messages
|
|
||||||
- JSON output includes prompt/profile/messages
|
|
||||||
- rendered output never includes resolved API key values
|
|
||||||
|
|
||||||
### Run Reuse Tests
|
|
||||||
|
|
||||||
Add tests proving:
|
|
||||||
|
|
||||||
- `Runner.Run` reuses prepare behavior
|
|
||||||
- run and render resolve the same prompt/profile/runtime settings for equivalent inputs
|
|
||||||
- run still validates output
|
|
||||||
- run still performs bounded repair where configured
|
|
||||||
|
|
||||||
### Existing Tests
|
|
||||||
|
|
||||||
Continue testing:
|
|
||||||
|
|
||||||
- prompt definition loading
|
|
||||||
- execution profile loading
|
|
||||||
- artifact loading
|
|
||||||
- prompt rendering
|
|
||||||
- LLM adapter behavior
|
|
||||||
- validation behavior
|
|
||||||
- HTTP request/response mapping
|
|
||||||
|
|
||||||
## 10. Extension Points
|
|
||||||
|
|
||||||
Future work should remain grounded in the current architecture:
|
|
||||||
|
|
||||||
- **Artifacts**: Add S3 artifact references via a new `artifact.Reader`.
|
|
||||||
- **LLM**: Implement additional provider adapters, such as Anthropic or Google.
|
|
||||||
- **Execution**: Add token budgeting, streaming generation, and batch execution capabilities.
|
|
||||||
- **Prepare/Render**: Add token estimates, prompt-size summaries, or additional render output formats.
|
|
||||||
- **Repositories**: Implement database-backed repositories for prompts and profiles.
|
|
||||||
- **Profiles**: Support more granular profile versioning and environment-specific profiles.
|
|
||||||
- **HTTP**: Add an HTTP prepare/render endpoint if Narratio or another caller needs it.
|
|
||||||
|
|
||||||
Future render formats should plug into the formatter layer and should not require changes to the usecase layer.
|
|
||||||
406
convert.go
Normal file
406
convert.go
Normal file
@@ -0,0 +1,406 @@
|
|||||||
|
package scriptorium
|
||||||
|
|
||||||
|
import (
|
||||||
|
"reflect"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
|
)
|
||||||
|
|
||||||
|
func toDomainRunRequest(req RunRequest) (domain.RunRequest, error) {
|
||||||
|
execution, err := toDomainExecutionTargetOverride(req.Execution)
|
||||||
|
if err != nil {
|
||||||
|
return domain.RunRequest{}, err
|
||||||
|
}
|
||||||
|
return domain.RunRequest{
|
||||||
|
PromptID: req.PromptID,
|
||||||
|
PromptVersion: req.PromptVersion,
|
||||||
|
ProfileID: req.ProfileID,
|
||||||
|
APIKey: req.APIKey,
|
||||||
|
Inputs: toDomainArtifactRefMap(req.Inputs),
|
||||||
|
Vars: copyStringMap(req.Vars),
|
||||||
|
Execution: execution,
|
||||||
|
Validation: toDomainOutputContractPtr(req.Validation),
|
||||||
|
Metadata: copyStringMap(req.Metadata),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func fromDomainPreparedRun(prepared *domain.PreparedRun) *PreparedRun {
|
||||||
|
if prepared == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return &PreparedRun{
|
||||||
|
PromptID: prepared.PromptID,
|
||||||
|
PromptVersion: prepared.PromptVersion,
|
||||||
|
PromptHash: prepared.PromptHash,
|
||||||
|
SelectedProfileID: prepared.SelectedProfileID,
|
||||||
|
EffectiveModelParams: fromDomainExecutionTarget(prepared.EffectiveModelParams),
|
||||||
|
OutputContract: fromDomainOutputContract(prepared.OutputContract),
|
||||||
|
StructuredOutput: fromDomainStructuredOutputSpec(prepared.StructuredOutput),
|
||||||
|
InputHashes: copyStringMap(prepared.InputHashes),
|
||||||
|
SessionID: prepared.SessionID,
|
||||||
|
RenderedPromptHash: prepared.RenderedPromptHash,
|
||||||
|
Messages: fromDomainRenderedMessages(prepared.Messages),
|
||||||
|
StartTime: prepared.StartTime,
|
||||||
|
EndTime: prepared.EndTime,
|
||||||
|
DurationMS: prepared.DurationMS,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func fromDomainRunResult(result *domain.RunResult) *RunResult {
|
||||||
|
if result == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return &RunResult{
|
||||||
|
RunID: result.RunID,
|
||||||
|
Artifact: fromDomainArtifact(result.Artifact),
|
||||||
|
RawOutput: result.RawOutput,
|
||||||
|
Validation: fromDomainValidationResult(result.Validation),
|
||||||
|
PromptID: result.PromptID,
|
||||||
|
PromptVersion: result.PromptVersion,
|
||||||
|
PromptHash: result.PromptHash,
|
||||||
|
RenderedPromptHash: result.RenderedPromptHash,
|
||||||
|
SelectedProfileID: result.SelectedProfileID,
|
||||||
|
ModelName: result.ModelName,
|
||||||
|
Endpoint: result.Endpoint,
|
||||||
|
EffectiveModelParams: fromDomainExecutionTarget(result.EffectiveModelParams),
|
||||||
|
InputHashes: copyStringMap(result.InputHashes),
|
||||||
|
Usage: fromDomainTokenUsage(result.Usage),
|
||||||
|
StartTime: result.StartTime,
|
||||||
|
EndTime: result.EndTime,
|
||||||
|
Duration: result.Duration,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func fromDomainGenerateRequest(req domain.GenerateRequest) GenerateRequest {
|
||||||
|
return GenerateRequest{
|
||||||
|
Prompt: fromDomainRenderedPrompt(req.Prompt),
|
||||||
|
Target: fromDomainExecutionTarget(req.Target),
|
||||||
|
TargetPresence: fromDomainExecutionTargetPresence(req.TargetPresence),
|
||||||
|
StructuredOutput: fromDomainStructuredOutputSpec(req.StructuredOutput),
|
||||||
|
APIKey: req.Target.APIKey,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func toDomainGenerateResponse(resp *GenerateResponse) *domain.GenerateResponse {
|
||||||
|
if resp == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return &domain.GenerateResponse{
|
||||||
|
Content: resp.Content,
|
||||||
|
Usage: toDomainTokenUsage(resp.Usage),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func fromDomainRenderedPrompt(prompt domain.RenderedPrompt) RenderedPrompt {
|
||||||
|
return RenderedPrompt{
|
||||||
|
SessionID: prompt.SessionID,
|
||||||
|
Messages: fromDomainRenderedMessages(prompt.Messages),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func toDomainArtifactRefMap(src map[string]ArtifactRef) map[string]domain.ArtifactRef {
|
||||||
|
if src == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make(map[string]domain.ArtifactRef, len(src))
|
||||||
|
for k, v := range src {
|
||||||
|
out[k] = toDomainArtifactRef(v)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func toDomainArtifactRef(ref ArtifactRef) domain.ArtifactRef {
|
||||||
|
return domain.ArtifactRef{
|
||||||
|
Type: domain.ArtifactRefType(ref.Type),
|
||||||
|
URI: ref.URI,
|
||||||
|
Body: ref.Body,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func fromDomainArtifact(artifact domain.Artifact) Artifact {
|
||||||
|
return Artifact{
|
||||||
|
Name: artifact.Name,
|
||||||
|
ContentType: artifact.ContentType,
|
||||||
|
Body: copyBytes(artifact.Body),
|
||||||
|
URI: artifact.URI,
|
||||||
|
Size: artifact.Size,
|
||||||
|
Hash: artifact.Hash,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func toDomainExecutionTargetOverride(override *ExecutionTargetOverride) (*domain.ExecutionTargetOverride, error) {
|
||||||
|
if override == nil {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
extraParams, err := copyPublicJSONMap(override.ExtraParams)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return &domain.ExecutionTargetOverride{
|
||||||
|
Endpoint: override.Endpoint,
|
||||||
|
Model: override.Model,
|
||||||
|
Temperature: copyFloat64Ptr(override.Temperature),
|
||||||
|
MaxTokens: copyIntPtr(override.MaxTokens),
|
||||||
|
TopP: copyFloat64Ptr(override.TopP),
|
||||||
|
TimeoutSeconds: copyIntPtr(override.TimeoutSeconds),
|
||||||
|
ServiceTier: override.ServiceTier,
|
||||||
|
ReasoningEffort: override.ReasoningEffort,
|
||||||
|
APIKeyEnv: override.APIKeyEnv,
|
||||||
|
ExtraParams: extraParams,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func fromDomainExecutionTarget(target domain.ExecutionTarget) ExecutionTarget {
|
||||||
|
return ExecutionTarget{
|
||||||
|
Endpoint: target.Endpoint,
|
||||||
|
Model: target.Model,
|
||||||
|
Temperature: target.Temperature,
|
||||||
|
MaxTokens: target.MaxTokens,
|
||||||
|
TopP: target.TopP,
|
||||||
|
TimeoutSeconds: target.TimeoutSeconds,
|
||||||
|
ServiceTier: target.ServiceTier,
|
||||||
|
ReasoningEffort: target.ReasoningEffort,
|
||||||
|
APIKeyEnv: target.APIKeyEnv,
|
||||||
|
ExtraParams: copyAnyMap(target.ExtraParams),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func fromDomainExecutionTargetPresence(presence domain.ExecutionTargetPresence) ExecutionTargetPresence {
|
||||||
|
return ExecutionTargetPresence{
|
||||||
|
Temperature: presence.Temperature,
|
||||||
|
MaxTokens: presence.MaxTokens,
|
||||||
|
TopP: presence.TopP,
|
||||||
|
TimeoutSeconds: presence.TimeoutSeconds,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func toDomainOutputContractPtr(contract *OutputContract) *domain.OutputContract {
|
||||||
|
if contract == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := toDomainOutputContract(*contract)
|
||||||
|
return &out
|
||||||
|
}
|
||||||
|
|
||||||
|
func toDomainOutputContract(contract OutputContract) domain.OutputContract {
|
||||||
|
return domain.OutputContract{
|
||||||
|
Format: domain.OutputFormat(contract.Format),
|
||||||
|
ValidationMode: domain.ValidationMode(contract.ValidationMode),
|
||||||
|
SchemaPath: contract.SchemaPath,
|
||||||
|
RepairAttempts: contract.RepairAttempts,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func fromDomainOutputContract(contract domain.OutputContract) OutputContract {
|
||||||
|
return OutputContract{
|
||||||
|
Format: OutputFormat(contract.Format),
|
||||||
|
ValidationMode: ValidationMode(contract.ValidationMode),
|
||||||
|
SchemaPath: contract.SchemaPath,
|
||||||
|
RepairAttempts: contract.RepairAttempts,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func fromDomainValidationResult(result domain.ValidationResult) ValidationResult {
|
||||||
|
return ValidationResult{
|
||||||
|
Status: ValidationStatus(result.Status),
|
||||||
|
Mode: ValidationMode(result.Mode),
|
||||||
|
Errors: copyStringSlice(result.Errors),
|
||||||
|
SchemaPath: result.SchemaPath,
|
||||||
|
RepairAttempts: result.RepairAttempts,
|
||||||
|
IsValid: result.IsValid,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func fromDomainTokenUsage(usage domain.TokenUsage) TokenUsage {
|
||||||
|
return TokenUsage{
|
||||||
|
PromptTokens: usage.PromptTokens,
|
||||||
|
CompletionTokens: usage.CompletionTokens,
|
||||||
|
TotalTokens: usage.TotalTokens,
|
||||||
|
CachedTokens: usage.CachedTokens,
|
||||||
|
CacheWriteTokens: usage.CacheWriteTokens,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func toDomainTokenUsage(usage TokenUsage) domain.TokenUsage {
|
||||||
|
return domain.TokenUsage{
|
||||||
|
PromptTokens: usage.PromptTokens,
|
||||||
|
CompletionTokens: usage.CompletionTokens,
|
||||||
|
TotalTokens: usage.TotalTokens,
|
||||||
|
CachedTokens: usage.CachedTokens,
|
||||||
|
CacheWriteTokens: usage.CacheWriteTokens,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func fromDomainRenderedMessages(messages []domain.RenderedMessage) []RenderedMessage {
|
||||||
|
if messages == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make([]RenderedMessage, len(messages))
|
||||||
|
for i, msg := range messages {
|
||||||
|
out[i] = RenderedMessage{
|
||||||
|
Role: msg.Role,
|
||||||
|
Content: msg.Content,
|
||||||
|
CacheControl: fromDomainCacheControl(msg.CacheControl),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func fromDomainCacheControl(cacheControl *domain.CacheControl) *CacheControl {
|
||||||
|
if cacheControl == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return &CacheControl{
|
||||||
|
Type: CacheControlType(cacheControl.Type),
|
||||||
|
TTL: cacheControl.TTL,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func fromDomainStructuredOutputSpec(spec *domain.StructuredOutputSpec) *StructuredOutputSpec {
|
||||||
|
if spec == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := &StructuredOutputSpec{
|
||||||
|
Type: StructuredOutputType(spec.Type),
|
||||||
|
}
|
||||||
|
if spec.JSONSchema != nil {
|
||||||
|
out.JSONSchema = &StructuredOutputJSONSpec{
|
||||||
|
Name: spec.JSONSchema.Name,
|
||||||
|
Strict: spec.JSONSchema.Strict,
|
||||||
|
Schema: copyAny(spec.JSONSchema.Schema),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func copyStringMap(src map[string]string) map[string]string {
|
||||||
|
if src == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make(map[string]string, len(src))
|
||||||
|
for k, v := range src {
|
||||||
|
out[k] = v
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func copyAnyMap(src map[string]any) map[string]any {
|
||||||
|
if src == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make(map[string]any, len(src))
|
||||||
|
for k, v := range src {
|
||||||
|
out[k] = copyAny(v)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func copyAny(value any) any {
|
||||||
|
if value == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
switch v := value.(type) {
|
||||||
|
case map[string]any:
|
||||||
|
return copyAnyMap(v)
|
||||||
|
case []any:
|
||||||
|
out := make([]any, len(v))
|
||||||
|
for i, item := range v {
|
||||||
|
out[i] = copyAny(item)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
case []string:
|
||||||
|
return copyStringSlice(v)
|
||||||
|
case []byte:
|
||||||
|
return copyBytes(v)
|
||||||
|
default:
|
||||||
|
return copyReflectValue(reflect.ValueOf(value)).Interface()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func copyReflectValue(value reflect.Value) reflect.Value {
|
||||||
|
if !value.IsValid() {
|
||||||
|
return value
|
||||||
|
}
|
||||||
|
|
||||||
|
switch value.Kind() {
|
||||||
|
case reflect.Interface:
|
||||||
|
if value.IsNil() {
|
||||||
|
return reflect.Zero(value.Type())
|
||||||
|
}
|
||||||
|
copied := copyReflectValue(value.Elem())
|
||||||
|
if copied.IsValid() && copied.Type().AssignableTo(value.Type()) {
|
||||||
|
return copied
|
||||||
|
}
|
||||||
|
out := reflect.New(value.Type()).Elem()
|
||||||
|
out.Set(copied)
|
||||||
|
return out
|
||||||
|
case reflect.Pointer:
|
||||||
|
if value.IsNil() {
|
||||||
|
return reflect.Zero(value.Type())
|
||||||
|
}
|
||||||
|
out := reflect.New(value.Type().Elem())
|
||||||
|
out.Elem().Set(copyReflectValue(value.Elem()))
|
||||||
|
return out
|
||||||
|
case reflect.Map:
|
||||||
|
if value.IsNil() {
|
||||||
|
return reflect.Zero(value.Type())
|
||||||
|
}
|
||||||
|
out := reflect.MakeMapWithSize(value.Type(), value.Len())
|
||||||
|
iter := value.MapRange()
|
||||||
|
for iter.Next() {
|
||||||
|
out.SetMapIndex(copyReflectValue(iter.Key()), copyReflectValue(iter.Value()))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
case reflect.Slice:
|
||||||
|
if value.IsNil() {
|
||||||
|
return reflect.Zero(value.Type())
|
||||||
|
}
|
||||||
|
out := reflect.MakeSlice(value.Type(), value.Len(), value.Cap())
|
||||||
|
for i := 0; i < value.Len(); i++ {
|
||||||
|
out.Index(i).Set(copyReflectValue(value.Index(i)))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
case reflect.Array:
|
||||||
|
out := reflect.New(value.Type()).Elem()
|
||||||
|
for i := 0; i < value.Len(); i++ {
|
||||||
|
out.Index(i).Set(copyReflectValue(value.Index(i)))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
default:
|
||||||
|
return value
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func copyStringSlice(src []string) []string {
|
||||||
|
if src == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make([]string, len(src))
|
||||||
|
copy(out, src)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func copyBytes(src []byte) []byte {
|
||||||
|
if src == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make([]byte, len(src))
|
||||||
|
copy(out, src)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func copyFloat64Ptr(src *float64) *float64 {
|
||||||
|
if src == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
v := *src
|
||||||
|
return &v
|
||||||
|
}
|
||||||
|
|
||||||
|
func copyIntPtr(src *int) *int {
|
||||||
|
if src == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
v := *src
|
||||||
|
return &v
|
||||||
|
}
|
||||||
297
docs/api.md
Normal file
297
docs/api.md
Normal file
@@ -0,0 +1,297 @@
|
|||||||
|
# HTTP API Reference
|
||||||
|
|
||||||
|
This is the canonical public HTTP contract for Scriptorium.
|
||||||
|
|
||||||
|
Implemented route:
|
||||||
|
|
||||||
|
- `POST /v1/runs`
|
||||||
|
|
||||||
|
For CLI behavior, see [CLI reference](cli.md). For config and prompt/profile
|
||||||
|
file formats, see [Configuration reference](config.md).
|
||||||
|
|
||||||
|
The maintained request-shape example is `examples/http-run.json`. It requires a
|
||||||
|
running `serve` process with an artifact root that can read the referenced
|
||||||
|
files, plus a reachable model endpoint for full execution.
|
||||||
|
|
||||||
|
## Base URL And Deployment
|
||||||
|
|
||||||
|
`scriptorium serve` listens on `server.addr` or `serve --addr`. The default is
|
||||||
|
`:8080`.
|
||||||
|
|
||||||
|
The route path is always:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/v1/runs
|
||||||
|
```
|
||||||
|
|
||||||
|
The HTTP adapter has no built-in authentication or authorization. Deploy it
|
||||||
|
behind trusted network and authentication controls.
|
||||||
|
|
||||||
|
## Media Types
|
||||||
|
|
||||||
|
- Request body: JSON object.
|
||||||
|
- Response body: JSON object.
|
||||||
|
- Response `Content-Type`: `application/json`.
|
||||||
|
|
||||||
|
Requests are decoded as JSON regardless of the request `Content-Type` header.
|
||||||
|
There are no shared query parameters.
|
||||||
|
|
||||||
|
## Request Limits
|
||||||
|
|
||||||
|
HTTP limits are configured through `server.*` config fields or `serve` flags:
|
||||||
|
|
||||||
|
- `server.max_request_bytes`: encoded JSON request body limit, including inline input bodies.
|
||||||
|
- `server.max_artifact_bytes`: file artifact limit for HTTP `file` input references.
|
||||||
|
- `server.max_response_bytes`: encoded JSON response limit, including artifact body and optional raw output.
|
||||||
|
|
||||||
|
Each limit defaults to `16777216` bytes. `0` disables that limit.
|
||||||
|
|
||||||
|
## `POST /v1/runs`
|
||||||
|
|
||||||
|
Runs one prompt request and returns the generated artifact, validation result,
|
||||||
|
and metadata.
|
||||||
|
|
||||||
|
### Request Body
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"prompt_id": "generic.markdown_summary",
|
||||||
|
"profile_id": "local-fast",
|
||||||
|
"prompt_version": "1.0.0",
|
||||||
|
"inputs": {
|
||||||
|
"transcript": {
|
||||||
|
"type": "file",
|
||||||
|
"uri": "./examples/fixtures/transcript.md"
|
||||||
|
},
|
||||||
|
"glossary": {
|
||||||
|
"type": "inline",
|
||||||
|
"body": "party:\n - Rin"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"vars": {
|
||||||
|
"session_date": "2026-05-04"
|
||||||
|
},
|
||||||
|
"model": {
|
||||||
|
"endpoint": "http://localhost:8000/v1",
|
||||||
|
"model": "gpt-4o-mini",
|
||||||
|
"temperature": 0,
|
||||||
|
"max_tokens": 800,
|
||||||
|
"top_p": 1,
|
||||||
|
"timeout_seconds": 120,
|
||||||
|
"service_tier": "priority",
|
||||||
|
"reasoning_effort": "medium",
|
||||||
|
"api_key_env": "SCRIPTORIUM_API_KEY",
|
||||||
|
"extra_params": {
|
||||||
|
"provider_option": "enabled"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"include_raw_output": false
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Request fields:
|
||||||
|
|
||||||
|
| Field | Required | Description |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `prompt_id` | yes | Prompt ID. Must not be blank. |
|
||||||
|
| `prompt_version` | no | Prompt version filter. |
|
||||||
|
| `profile_id` | no | Execution profile ID. If omitted, the prompt must define `default_profile`. |
|
||||||
|
| `inputs` | yes | Object mapping prompt input names to input references. Must contain at least one entry. |
|
||||||
|
| `vars` | no | Object mapping template variable names to string values. |
|
||||||
|
| `model` | no | Runtime model override object. |
|
||||||
|
| `include_raw_output` | no | When `true`, include `raw_model_output` in the response. |
|
||||||
|
|
||||||
|
Input reference fields:
|
||||||
|
|
||||||
|
| Field | Required | Description |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `type` | yes | `file` or `inline`. |
|
||||||
|
| `uri` | for `file` | File URI/path. |
|
||||||
|
| `body` | for `inline` | Inline artifact body. |
|
||||||
|
|
||||||
|
HTTP `file` references require `server.artifact_root` or `serve
|
||||||
|
--artifact-root`. Relative file URIs resolve against that root. Absolute file
|
||||||
|
URIs are accepted only when lexically inside the root. Relative traversal and
|
||||||
|
absolute paths outside the root return `400 artifact_not_allowed`.
|
||||||
|
|
||||||
|
The containment check is lexical and does not resolve symlinks. Symlinks inside
|
||||||
|
the artifact root are followed by the operating system, including symlinks that
|
||||||
|
point outside the root. Keep the artifact root narrow and not writable by
|
||||||
|
untrusted users.
|
||||||
|
|
||||||
|
Model override fields:
|
||||||
|
|
||||||
|
| Field | Description |
|
||||||
|
| --- | --- |
|
||||||
|
| `endpoint` | Runtime endpoint override. |
|
||||||
|
| `model` | Runtime model override. |
|
||||||
|
| `temperature` | Number in range `0..2`. Explicit `0` is an override. |
|
||||||
|
| `max_tokens` | Integer greater than or equal to `0`. Explicit `0` is an override. |
|
||||||
|
| `top_p` | Number in range `0..1`. Explicit `0` is an override. |
|
||||||
|
| `timeout_seconds` | Integer greater than or equal to `0`. Explicit `0` disables the outbound client timeout. |
|
||||||
|
| `service_tier` | Provider-specific request tier. |
|
||||||
|
| `reasoning_effort` | Provider-specific reasoning setting. |
|
||||||
|
| `api_key_env` | Name of an environment variable containing the API key. |
|
||||||
|
| `extra_params` | JSON-compatible provider-specific top-level request fields. |
|
||||||
|
|
||||||
|
Raw API-key values are not accepted in HTTP payloads. A field such as
|
||||||
|
`api_key` is rejected as unknown JSON.
|
||||||
|
|
||||||
|
`extra_params` keys must not be empty and must not collide with reserved
|
||||||
|
outbound fields: `model`, `session_id`, `messages`, `temperature`,
|
||||||
|
`max_tokens`, `top_p`, `service_tier`, `reasoning_effort`, or
|
||||||
|
`response_format`.
|
||||||
|
|
||||||
|
### Strict JSON Rules
|
||||||
|
|
||||||
|
Request decoding is strict:
|
||||||
|
|
||||||
|
- malformed JSON returns `400 invalid_json`
|
||||||
|
- unknown request fields return `400 invalid_json`
|
||||||
|
- unknown `inputs` item fields return `400 invalid_json`
|
||||||
|
- unknown `model` fields return `400 invalid_json`
|
||||||
|
- trailing JSON tokens after the request object return `400 invalid_json`
|
||||||
|
- request bodies above the configured limit return `413 request_too_large`
|
||||||
|
|
||||||
|
### Success Response
|
||||||
|
|
||||||
|
Status: `200 OK`
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"artifact": {
|
||||||
|
"name": "output",
|
||||||
|
"content_type": "text/markdown",
|
||||||
|
"body": "Generated content",
|
||||||
|
"size": 17,
|
||||||
|
"hash": "..."
|
||||||
|
},
|
||||||
|
"validation": {
|
||||||
|
"status": "passed",
|
||||||
|
"mode": "basic",
|
||||||
|
"repair_attempts": 0,
|
||||||
|
"is_valid": true
|
||||||
|
},
|
||||||
|
"metadata": {
|
||||||
|
"run_id": "...",
|
||||||
|
"prompt_id": "generic.markdown_summary",
|
||||||
|
"prompt_version": "1.0.0",
|
||||||
|
"prompt_hash": "...",
|
||||||
|
"rendered_prompt_hash": "...",
|
||||||
|
"selected_profile_id": "local-fast",
|
||||||
|
"model_name": "gpt-4o-mini",
|
||||||
|
"endpoint": "http://localhost:8000/v1",
|
||||||
|
"model_params": {
|
||||||
|
"endpoint": "http://localhost:8000/v1",
|
||||||
|
"model": "gpt-4o-mini",
|
||||||
|
"temperature": 0.2,
|
||||||
|
"max_tokens": 500,
|
||||||
|
"top_p": 1,
|
||||||
|
"timeout_seconds": 90
|
||||||
|
},
|
||||||
|
"input_hashes": {
|
||||||
|
"transcript": "..."
|
||||||
|
},
|
||||||
|
"usage": {
|
||||||
|
"prompt_tokens": 11,
|
||||||
|
"completion_tokens": 22,
|
||||||
|
"total_tokens": 33,
|
||||||
|
"cached_tokens": 0,
|
||||||
|
"cache_write_tokens": 0
|
||||||
|
},
|
||||||
|
"start_time": "2026-05-04T12:00:00Z",
|
||||||
|
"end_time": "2026-05-04T12:00:01Z",
|
||||||
|
"duration_ms": 1000,
|
||||||
|
"validation_mode": "basic",
|
||||||
|
"validation_status": "passed",
|
||||||
|
"repair_attempts_used": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Response fields:
|
||||||
|
|
||||||
|
- `artifact`: generated output artifact.
|
||||||
|
- `validation`: validation result for the generated artifact.
|
||||||
|
- `metadata`: run and effective runtime metadata.
|
||||||
|
- `raw_model_output`: omitted unless `include_raw_output` is `true`.
|
||||||
|
|
||||||
|
`artifact.uri` is omitted when empty. `validation.errors` and
|
||||||
|
`validation.schema_path` are omitted when empty. `model_params.service_tier`,
|
||||||
|
`model_params.reasoning_effort`, `model_params.api_key_env`, and
|
||||||
|
`model_params.extra_params` are omitted when empty.
|
||||||
|
|
||||||
|
`metadata.usage.cached_tokens` and `metadata.usage.cache_write_tokens` are
|
||||||
|
always present as numbers. They are `0` when the provider omits compatible cache
|
||||||
|
usage fields or reports no cache activity.
|
||||||
|
|
||||||
|
### Validation Failure Response
|
||||||
|
|
||||||
|
Generated-content validation failures still return `200 OK`.
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"validation": {
|
||||||
|
"status": "failed",
|
||||||
|
"mode": "json",
|
||||||
|
"errors": ["invalid JSON: ..."],
|
||||||
|
"repair_attempts": 0,
|
||||||
|
"is_valid": false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The response still includes `artifact` and `metadata`.
|
||||||
|
|
||||||
|
## Error Responses
|
||||||
|
|
||||||
|
Error body shape:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"error": {
|
||||||
|
"code": "invalid_request",
|
||||||
|
"message": "prompt_id is required"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Current status/code mapping:
|
||||||
|
|
||||||
|
| Status | Code | Meaning |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `400` | `invalid_json` | Malformed JSON, unknown JSON field, or trailing JSON token. |
|
||||||
|
| `400` | `invalid_request` | Missing/invalid request fields or invalid runtime overrides. |
|
||||||
|
| `400` | `profile_required` | No `profile_id` and prompt has no `default_profile`. |
|
||||||
|
| `400` | `prompt_load_failed` | Prompt definition YAML/contract failed to load. |
|
||||||
|
| `400` | `profile_load_failed` | Profile YAML/contract failed to load, including raw `api_key`. |
|
||||||
|
| `400` | `artifact_not_allowed` | HTTP file refs are disabled or requested path is outside artifact root. |
|
||||||
|
| `400` | `artifact_read_failed` | Input artifact could not be read or input ref was unsupported/invalid. |
|
||||||
|
| `400` | `prompt_render_failed` | Prompt template rendering failed. |
|
||||||
|
| `400` | `api_key_env_missing` | Selected `api_key_env` variable is unset or empty. |
|
||||||
|
| `404` | `not_found` | Route path is unknown. |
|
||||||
|
| `404` | `prompt_not_found` | Prompt ID/version was not found. |
|
||||||
|
| `404` | `profile_not_found` | Profile ID was not found. |
|
||||||
|
| `405` | `method_not_allowed` | Method is not `POST` on `/v1/runs`. |
|
||||||
|
| `413` | `request_too_large` | Encoded JSON request body exceeds configured request limit. |
|
||||||
|
| `413` | `artifact_too_large` | HTTP file input artifact exceeds configured artifact limit. |
|
||||||
|
| `413` | `response_too_large` | Encoded JSON response exceeds configured response limit. |
|
||||||
|
| `500` | `validation_runtime_failed` | Validator runtime/schema loading failed. |
|
||||||
|
| `500` | `internal_error` | Unclassified server error. |
|
||||||
|
| `502` | `llm_failed` | Outbound model request failed. |
|
||||||
|
|
||||||
|
HTTP error messages are intentionally concise and do not include sensitive
|
||||||
|
internal causes.
|
||||||
|
|
||||||
|
## Retry And Idempotency
|
||||||
|
|
||||||
|
Scriptorium does not provide idempotency keys, pagination, caching headers, or
|
||||||
|
rate limiting.
|
||||||
|
|
||||||
|
Clients may retry transport failures or `5xx` responses when their surrounding
|
||||||
|
workflow can tolerate another model call. A retry can generate different output
|
||||||
|
and incur another provider request.
|
||||||
|
|
||||||
|
## Example File
|
||||||
|
|
||||||
|
- `examples/http-run.json`
|
||||||
233
docs/cli.md
Normal file
233
docs/cli.md
Normal file
@@ -0,0 +1,233 @@
|
|||||||
|
# CLI Reference
|
||||||
|
|
||||||
|
## Shortest Useful Command
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium render \
|
||||||
|
--config ./examples/config.yml \
|
||||||
|
--prompt generic.markdown_summary \
|
||||||
|
--input transcript=./examples/fixtures/transcript.md \
|
||||||
|
--input glossary=./examples/fixtures/glossary.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
`render` prepares the prompt, loads input artifacts, resolves the execution
|
||||||
|
profile, and prints the prepared request without calling an LLM.
|
||||||
|
|
||||||
|
## Command Overview
|
||||||
|
|
||||||
|
- `scriptorium run`: prepare a prompt, call the configured LLM, write generated output, and print a run summary.
|
||||||
|
- `scriptorium render`: prepare a prompt only; write prepared-run output as `text` or `json`.
|
||||||
|
- `scriptorium serve`: start the HTTP server for `POST /v1/runs`.
|
||||||
|
|
||||||
|
Canonical related references:
|
||||||
|
|
||||||
|
- [Configuration reference](config.md)
|
||||||
|
- [HTTP API reference](api.md)
|
||||||
|
- [Subprocess integration](integrations/subprocess.md)
|
||||||
|
|
||||||
|
## Common Rules
|
||||||
|
|
||||||
|
- `--config` is supported by `run`, `render`, and `serve`.
|
||||||
|
- Positional arguments are rejected.
|
||||||
|
- `run` and `render` require `--prompt`, at least one `--input`, and an effective `prompt_dir`.
|
||||||
|
- `serve` requires an effective `prompt_dir`.
|
||||||
|
- `profile_dir` is optional. Without it, only built-in profiles are available.
|
||||||
|
- If `profile_dir` is set, custom profiles override built-in profiles with the same ID.
|
||||||
|
- Prompt cache control, `session_id`, structured output, and provider-specific profile fields are configured in YAML, not with CLI flags.
|
||||||
|
|
||||||
|
Config precedence is:
|
||||||
|
|
||||||
|
1. built-in defaults
|
||||||
|
2. config file values
|
||||||
|
3. CLI flags
|
||||||
|
|
||||||
|
## Flag Reference
|
||||||
|
|
||||||
|
### `scriptorium run`
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scriptorium run [flags]
|
||||||
|
```
|
||||||
|
|
||||||
|
Required through flags or config:
|
||||||
|
|
||||||
|
- `--prompt-dir <dir>`: prompt definition directory.
|
||||||
|
|
||||||
|
Required as flags:
|
||||||
|
|
||||||
|
- `--prompt <id>`: prompt ID to execute.
|
||||||
|
- `--input name=path`: input file mapping. Repeat or use comma-separated mappings.
|
||||||
|
|
||||||
|
Optional flags:
|
||||||
|
|
||||||
|
- `--config <path>`: application config file.
|
||||||
|
- `--profile-dir <dir>`: custom profile definition directory.
|
||||||
|
- `--schema-dir <dir>`: schema base directory for `json_schema` validation.
|
||||||
|
- `--profile <id>`: execution profile override. If omitted, the prompt `default_profile` is used.
|
||||||
|
- `--var name=value`: template variable mapping. Repeat or use comma-separated mappings.
|
||||||
|
- `--out <path>`: write generated artifact body to a file instead of stdout.
|
||||||
|
- `--llm-base-url <url>`: runtime endpoint override.
|
||||||
|
- `--model <name>`: runtime model override.
|
||||||
|
- `--api-key-env <name>`: runtime API-key environment variable name override.
|
||||||
|
- `--temperature <float>`: runtime temperature override.
|
||||||
|
- `--max-tokens <int>`: runtime max tokens override.
|
||||||
|
- `--top-p <float>`: runtime top-p override.
|
||||||
|
- `--timeout <duration>`: runtime timeout override using Go duration syntax, such as `30s` or `2m`.
|
||||||
|
|
||||||
|
Deprecated aliases:
|
||||||
|
|
||||||
|
- `--prompt-id <id>`: alias for `--prompt`.
|
||||||
|
- `--profile-id <id>`: alias for `--profile`.
|
||||||
|
|
||||||
|
Runtime override notes:
|
||||||
|
|
||||||
|
- Omitted numeric override flags preserve the selected profile/default value.
|
||||||
|
- Explicit zero values override the selected profile/default value.
|
||||||
|
- `--timeout 0s` disables the outbound HTTP client timeout for that request.
|
||||||
|
- There is no raw API-key flag; use `--api-key-env`.
|
||||||
|
|
||||||
|
### `scriptorium render`
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scriptorium render [flags]
|
||||||
|
```
|
||||||
|
|
||||||
|
Required through flags or config:
|
||||||
|
|
||||||
|
- `--prompt-dir <dir>`: prompt definition directory.
|
||||||
|
|
||||||
|
Required as flags:
|
||||||
|
|
||||||
|
- `--prompt <id>`: prompt ID to render.
|
||||||
|
- `--input name=path`: input file mapping. Repeat or use comma-separated mappings.
|
||||||
|
|
||||||
|
Optional flags:
|
||||||
|
|
||||||
|
- `--config <path>`: application config file.
|
||||||
|
- `--prompt-dir <dir>`: prompt definition directory.
|
||||||
|
- `--profile-dir <dir>`: custom profile definition directory.
|
||||||
|
- `--profile <id>`: execution profile override.
|
||||||
|
- `--var name=value`: template variable mapping. Repeat or use comma-separated mappings.
|
||||||
|
- `--out <path>`: write prepared-run output to a file instead of stdout.
|
||||||
|
- `--llm-base-url <url>`: runtime endpoint override for the prepared request.
|
||||||
|
- `--model <name>`: runtime model override for the prepared request.
|
||||||
|
- `--api-key-env <name>`: runtime API-key environment variable name override.
|
||||||
|
- `--temperature <float>`: runtime temperature override.
|
||||||
|
- `--max-tokens <int>`: runtime max tokens override.
|
||||||
|
- `--top-p <float>`: runtime top-p override.
|
||||||
|
- `--timeout <duration>`: runtime timeout override using Go duration syntax.
|
||||||
|
- `--format text|json`: prepared-run output format. Defaults to config `defaults.render_format`, then `text`.
|
||||||
|
|
||||||
|
Deprecated aliases:
|
||||||
|
|
||||||
|
- `--prompt-id <id>`: alias for `--prompt`.
|
||||||
|
- `--profile-id <id>`: alias for `--profile`.
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
- `render` resolves profiles, loads schemas for `json_schema` prompts, and validates `api_key_env`.
|
||||||
|
- `render` does not accept `--schema-dir`; use config `schema_dir` for render-time schema lookup.
|
||||||
|
- `render` does not call the LLM.
|
||||||
|
|
||||||
|
### `scriptorium serve`
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scriptorium serve [flags]
|
||||||
|
```
|
||||||
|
|
||||||
|
Required through flags or config:
|
||||||
|
|
||||||
|
- `--prompt-dir <dir>`: prompt definition directory.
|
||||||
|
|
||||||
|
Optional flags:
|
||||||
|
|
||||||
|
- `--config <path>`: application config file.
|
||||||
|
- `--addr <listen-address>`: HTTP listen address.
|
||||||
|
- `--prompt-dir <dir>`: prompt definition directory.
|
||||||
|
- `--profile-dir <dir>`: custom profile definition directory.
|
||||||
|
- `--schema-dir <dir>`: schema base directory for `json_schema` validation.
|
||||||
|
- `--artifact-root <dir>`: base directory for HTTP `file` input references.
|
||||||
|
- `--max-request-bytes <n>`: maximum HTTP request body bytes; `0` disables the limit.
|
||||||
|
- `--max-artifact-bytes <n>`: maximum HTTP file artifact bytes; `0` disables the limit.
|
||||||
|
- `--max-response-bytes <n>`: maximum encoded HTTP response body bytes; `0` disables the limit.
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
- `serve` does not accept runtime model override flags such as `--model` or `--llm-base-url`.
|
||||||
|
- HTTP request fields and error codes are documented in the [HTTP API reference](api.md).
|
||||||
|
- HTTP `file` input references are rejected unless an artifact root is configured.
|
||||||
|
- HTTP size-limit flags affect only `serve`.
|
||||||
|
|
||||||
|
## Input And Variable Syntax
|
||||||
|
|
||||||
|
- `--input name=path` maps prompt input names to local file paths.
|
||||||
|
- `--var name=value` maps prompt template variables to string values.
|
||||||
|
- Both flags can be repeated.
|
||||||
|
- Both flags also accept comma-separated mappings, such as `--input transcript=./t.md,glossary=./g.yml`.
|
||||||
|
- Values may contain `=` after the first separator, such as `--var note=a=b=c`.
|
||||||
|
- Empty names and empty values are rejected.
|
||||||
|
|
||||||
|
CLI `run` and `render` convert every `--input` mapping to a `file` artifact
|
||||||
|
reference. HTTP also supports `inline` input references; see [HTTP API
|
||||||
|
reference](api.md).
|
||||||
|
|
||||||
|
## Output Behavior
|
||||||
|
|
||||||
|
`run`:
|
||||||
|
|
||||||
|
- Writes generated artifact content to stdout by default.
|
||||||
|
- Writes generated artifact content to `--out` when provided.
|
||||||
|
- Prints a success summary to stderr.
|
||||||
|
- Prints errors to stderr on failure.
|
||||||
|
|
||||||
|
`render`:
|
||||||
|
|
||||||
|
- Writes prepared-run output to stdout by default.
|
||||||
|
- Writes prepared-run output to `--out` when provided.
|
||||||
|
- Does not print a success summary.
|
||||||
|
|
||||||
|
`serve`:
|
||||||
|
|
||||||
|
- Logs startup and server errors to stderr.
|
||||||
|
|
||||||
|
## Exit Codes
|
||||||
|
|
||||||
|
- `0`: success.
|
||||||
|
- `1`: parse, config, load, render, generation, output-write, or runtime error.
|
||||||
|
- `2`: `run` completed and wrote output, but validation status is `failed`.
|
||||||
|
|
||||||
|
## Common Workflows
|
||||||
|
|
||||||
|
Render prompt inputs and variables as JSON:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium render \
|
||||||
|
--config ./examples/config.yml \
|
||||||
|
--prompt generic.markdown_summary \
|
||||||
|
--input transcript=./examples/fixtures/transcript.md \
|
||||||
|
--input glossary=./examples/fixtures/glossary.yml \
|
||||||
|
--var session_date=2026-05-04 \
|
||||||
|
--format json
|
||||||
|
```
|
||||||
|
|
||||||
|
Run a prompt with an explicit profile and file output:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium run \
|
||||||
|
--config ./examples/config.yml \
|
||||||
|
--prompt generic.markdown_summary \
|
||||||
|
--profile local-fast \
|
||||||
|
--input transcript=./examples/fixtures/transcript.md \
|
||||||
|
--input glossary=./examples/fixtures/glossary.yml \
|
||||||
|
--out ./summary.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Start the HTTP server with example config:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium serve --config ./examples/config.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Copyable maintained script:
|
||||||
|
|
||||||
|
- `examples/render-markdown-summary.sh`
|
||||||
324
docs/config.md
Normal file
324
docs/config.md
Normal file
@@ -0,0 +1,324 @@
|
|||||||
|
# Configuration Reference
|
||||||
|
|
||||||
|
## Config Discovery And Precedence
|
||||||
|
|
||||||
|
Application settings are resolved in this order:
|
||||||
|
|
||||||
|
1. built-in defaults
|
||||||
|
2. `config.yml` values
|
||||||
|
3. CLI overrides
|
||||||
|
|
||||||
|
When `--config` is omitted, Scriptorium searches:
|
||||||
|
|
||||||
|
1. `/usr/local/etc/scriptorium/config.yml`
|
||||||
|
2. `/etc/scriptorium/config.yml`
|
||||||
|
|
||||||
|
If neither file exists, Scriptorium uses built-in defaults. When
|
||||||
|
`--config <path>` is provided, that file must exist and decode successfully.
|
||||||
|
|
||||||
|
## Minimal Working Config
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
prompt_dir: ./examples/prompts
|
||||||
|
```
|
||||||
|
|
||||||
|
This is enough for `run` and `render` when selected prompts use built-in
|
||||||
|
profiles. Set `profile_dir` when prompts or requests use custom profiles.
|
||||||
|
|
||||||
|
The maintained repository example is `examples/config.yml`.
|
||||||
|
|
||||||
|
## Production-Oriented Config
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
prompt_dir: /opt/scriptorium/prompts
|
||||||
|
profile_dir: /opt/scriptorium/profiles
|
||||||
|
schema_dir: /opt/scriptorium/schemas
|
||||||
|
|
||||||
|
server:
|
||||||
|
addr: 127.0.0.1:8080
|
||||||
|
artifact_root: /var/lib/scriptorium/artifacts
|
||||||
|
max_request_bytes: 16777216
|
||||||
|
max_artifact_bytes: 16777216
|
||||||
|
max_response_bytes: 16777216
|
||||||
|
|
||||||
|
defaults:
|
||||||
|
render_format: text
|
||||||
|
```
|
||||||
|
|
||||||
|
The maintained full example is `examples/config.full.yml`.
|
||||||
|
|
||||||
|
## App Config Reference
|
||||||
|
|
||||||
|
Top-level fields:
|
||||||
|
|
||||||
|
| Field | Default | Description |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `prompt_dir` | unset | Directory containing prompt definition YAML files. Required effectively by `run`, `render`, and `serve`. |
|
||||||
|
| `profile_dir` | unset | Directory containing custom profile YAML files. Built-in profiles remain available when unset. |
|
||||||
|
| `schema_dir` | `.` | Base directory for relative JSON Schema paths. |
|
||||||
|
| `server` | `{}` | HTTP service settings used by `serve`. |
|
||||||
|
| `defaults` | `{}` | Adapter defaults. |
|
||||||
|
|
||||||
|
`server` fields:
|
||||||
|
|
||||||
|
| Field | Default | Description |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `server.addr` | `:8080` | Listen address for `serve`. |
|
||||||
|
| `server.artifact_root` | unset | Base directory for HTTP `file` input references. Without it, HTTP file refs are rejected. |
|
||||||
|
| `server.max_request_bytes` | `16777216` | Maximum encoded HTTP request body bytes. `0` disables the limit. |
|
||||||
|
| `server.max_artifact_bytes` | `16777216` | Maximum HTTP file artifact bytes. `0` disables the limit. |
|
||||||
|
| `server.max_response_bytes` | `16777216` | Maximum encoded HTTP response bytes. `0` disables the limit. |
|
||||||
|
|
||||||
|
`defaults` fields:
|
||||||
|
|
||||||
|
| Field | Default | Description |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `defaults.render_format` | `text` | Default `render` output format: `text` or `json`. |
|
||||||
|
|
||||||
|
Config rules:
|
||||||
|
|
||||||
|
- YAML decoding is strict; unknown fields are rejected.
|
||||||
|
- HTTP size limits must be greater than or equal to `0`.
|
||||||
|
- Empty string config values are ignored.
|
||||||
|
- Raw API key fields are not supported in app config.
|
||||||
|
|
||||||
|
## Prompt Definition Files
|
||||||
|
|
||||||
|
Prompt definitions are YAML files anywhere under `prompt_dir`. Nested
|
||||||
|
directories are organizational; callers select prompts by YAML `id`, not file
|
||||||
|
path.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
id: generic.structured_events
|
||||||
|
version: "1.0.0"
|
||||||
|
default_profile: local-quality
|
||||||
|
description: Produce structured event JSON from a transcript.
|
||||||
|
|
||||||
|
inputs:
|
||||||
|
- name: transcript
|
||||||
|
required: true
|
||||||
|
content_type: text/markdown
|
||||||
|
description: Source transcript content
|
||||||
|
- name: glossary
|
||||||
|
required: false
|
||||||
|
content_type: text/yaml
|
||||||
|
description: Optional glossary context
|
||||||
|
|
||||||
|
messages:
|
||||||
|
- role: system
|
||||||
|
content_file: ./generic.structured_events.system.md
|
||||||
|
- role: user
|
||||||
|
content_file: ./generic.structured_events.user.md
|
||||||
|
|
||||||
|
output:
|
||||||
|
format: json
|
||||||
|
validation_mode: json_schema
|
||||||
|
schema_path: structured_events.schema.json
|
||||||
|
repair_attempts: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
Prompt fields:
|
||||||
|
|
||||||
|
| Field | Required | Description |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `id` | yes | Prompt identifier used by `--prompt` and HTTP `prompt_id`. |
|
||||||
|
| `version` | yes | Prompt version. |
|
||||||
|
| `default_profile` | no | Profile ID used when a request does not provide a profile. |
|
||||||
|
| `description` | no | Human-readable description. |
|
||||||
|
| `session_id` | no | Go-template string rendered from request vars and forwarded as provider `session_id` when non-empty. |
|
||||||
|
| `inputs` | no | Named input declarations. |
|
||||||
|
| `messages` | yes | Chat message templates. |
|
||||||
|
| `output` | yes | Output format and validation contract. |
|
||||||
|
|
||||||
|
`inputs[]` fields:
|
||||||
|
|
||||||
|
- `name` (required)
|
||||||
|
- `required` (optional boolean)
|
||||||
|
- `content_type` (optional metadata)
|
||||||
|
- `description` (optional)
|
||||||
|
|
||||||
|
`messages[]` fields:
|
||||||
|
|
||||||
|
- `role` (required)
|
||||||
|
- exactly one of `content` or `content_file`
|
||||||
|
- `cache_control` (optional)
|
||||||
|
|
||||||
|
Message rules:
|
||||||
|
|
||||||
|
- `content_file` resolves relative to the prompt YAML file location.
|
||||||
|
- Repeated roles are allowed.
|
||||||
|
- Prompt YAML decoding is strict.
|
||||||
|
- Duplicate input names are invalid.
|
||||||
|
- Duplicate prompt IDs are invalid for a requested ID/version.
|
||||||
|
|
||||||
|
`messages[].cache_control` fields:
|
||||||
|
|
||||||
|
| Field | Required | Supported values |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `type` | yes | `ephemeral` |
|
||||||
|
| `ttl` | no | `1h` |
|
||||||
|
|
||||||
|
`session_id` behavior:
|
||||||
|
|
||||||
|
- Rendered with the same variable context as message templates.
|
||||||
|
- Trimmed and omitted when empty.
|
||||||
|
- Rejected when longer than 256 Unicode code points.
|
||||||
|
- CLI callers pass variables with `--var`; HTTP callers use `vars`.
|
||||||
|
|
||||||
|
`output` fields:
|
||||||
|
|
||||||
|
| Field | Required | Supported values |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `format` | yes | `text`, `markdown`, `json` |
|
||||||
|
| `validation_mode` | yes | `none`, `basic`, `json`, `json_schema` |
|
||||||
|
| `schema_path` | only for `json_schema` | Relative to `schema_dir` unless absolute. |
|
||||||
|
| `repair_attempts` | yes | Integer greater than or equal to `0`. |
|
||||||
|
|
||||||
|
Repair boundary:
|
||||||
|
|
||||||
|
- `repair_attempts` is part of the prompt contract.
|
||||||
|
- The current CLI and HTTP wiring constructs the runner without a repairer, so normal `run` and `serve` execution does not perform repair attempts.
|
||||||
|
|
||||||
|
## Profile Definition Files
|
||||||
|
|
||||||
|
Execution profiles are YAML files anywhere under `profile_dir`. Nested
|
||||||
|
directories are organizational; callers select profiles by YAML `id`, not file
|
||||||
|
path.
|
||||||
|
|
||||||
|
Scriptorium also ships built-in profiles. Custom profiles override built-ins
|
||||||
|
with the same ID.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
id: local-fast
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: gpt-4o-mini
|
||||||
|
temperature: 0.2
|
||||||
|
max_tokens: 500
|
||||||
|
top_p: 1.0
|
||||||
|
timeout_seconds: 90
|
||||||
|
api_key_env: SCRIPTORIUM_API_KEY
|
||||||
|
service_tier: priority
|
||||||
|
reasoning_effort: medium
|
||||||
|
extra_params:
|
||||||
|
provider_route: primary
|
||||||
|
```
|
||||||
|
|
||||||
|
Profile fields:
|
||||||
|
|
||||||
|
| Field | Required | Description |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `id` | yes | Profile identifier. |
|
||||||
|
| `endpoint` | yes | OpenAI-compatible base URL including `/v1`. |
|
||||||
|
| `model` | yes | Provider model name. |
|
||||||
|
| `temperature` | no | Range `0..2`. |
|
||||||
|
| `max_tokens` | no | Integer greater than or equal to `0`. |
|
||||||
|
| `top_p` | no | Range `0..1`. |
|
||||||
|
| `timeout_seconds` | no | Integer greater than or equal to `0`. |
|
||||||
|
| `service_tier` | no | Provider-specific request tier. |
|
||||||
|
| `reasoning_effort` | no | Provider-specific reasoning setting. |
|
||||||
|
| `api_key_env` | no | Environment variable name containing the API key. |
|
||||||
|
| `extra_params` | no | JSON-compatible provider-specific top-level request fields. |
|
||||||
|
|
||||||
|
Execution defaults before profile/request overrides:
|
||||||
|
|
||||||
|
| Field | Default |
|
||||||
|
| --- | --- |
|
||||||
|
| `temperature` | `0.0` |
|
||||||
|
| `max_tokens` | `0` |
|
||||||
|
| `top_p` | `1.0` |
|
||||||
|
| `timeout_seconds` | `600` |
|
||||||
|
|
||||||
|
Profile rules:
|
||||||
|
|
||||||
|
- Profile YAML decoding is strict.
|
||||||
|
- Duplicate custom profile IDs are invalid.
|
||||||
|
- Matching custom and built-in IDs are valid override behavior.
|
||||||
|
- Raw `api_key` is rejected; use `api_key_env`.
|
||||||
|
- If `api_key_env` is set, the named environment variable must be set before `run`, `render`, or HTTP execution can prepare the request.
|
||||||
|
- Profile numeric fields merge by non-zero value. Request overrides are presence-aware, so explicit zero values are supported through CLI flags or HTTP model overrides.
|
||||||
|
- `extra_params` keys must not be empty and must not collide with reserved outbound fields: `model`, `session_id`, `messages`, `temperature`, `max_tokens`, `top_p`, `service_tier`, `reasoning_effort`, or `response_format`.
|
||||||
|
|
||||||
|
Built-in profile catalog:
|
||||||
|
|
||||||
|
| Provider | ID | Model | API key env |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| aion-labs | `aion-2` | `aion-labs/aion-2.0` | `OPENROUTER_API_KEY` |
|
||||||
|
| anthropic | `claude-fable-latest` | `~anthropic/claude-fable-latest` | `OPENROUTER_API_KEY` |
|
||||||
|
| anthropic | `claude-haiku-latest` | `~anthropic/claude-haiku-latest` | `OPENROUTER_API_KEY` |
|
||||||
|
| anthropic | `claude-opus-latest` | `~anthropic/claude-opus-latest` | `OPENROUTER_API_KEY` |
|
||||||
|
| anthropic | `claude-sonnet-latest` | `~anthropic/claude-sonnet-latest` | `OPENROUTER_API_KEY` |
|
||||||
|
| deepseek | `deepseek-3-2` | `deepseek/deepseek-v3.2` | `OPENROUTER_API_KEY` |
|
||||||
|
| deepseek | `deepseek-4-pro` | `deepseek/deepseek-v4-pro` | `OPENROUTER_API_KEY` |
|
||||||
|
| google | `gemini-2-flash` | `google/gemini-2.5-flash` | `OPENROUTER_API_KEY` |
|
||||||
|
| google | `gemini-2-flash-lite` | `google/gemini-2.5-flash-lite` | `OPENROUTER_API_KEY` |
|
||||||
|
| google | `gemini-2-pro` | `google/gemini-2.5-pro` | `OPENROUTER_API_KEY` |
|
||||||
|
| google | `gemini-3-flash-lite` | `google/gemini-3.1-flash-lite` | `OPENROUTER_API_KEY` |
|
||||||
|
| google | `gemini-flash-latest` | `~google/gemini-flash-latest` | `OPENROUTER_API_KEY` |
|
||||||
|
| google | `gemini-pro-latest` | `~google/gemini-pro-latest` | `OPENROUTER_API_KEY` |
|
||||||
|
| google | `gemma-4-31b` | `google/gemma-4-31b-it:exacto` | `OPENROUTER_API_KEY` |
|
||||||
|
| minimax | `minimax-m2` | `minimax/minimax-m2.5` | `OPENROUTER_API_KEY` |
|
||||||
|
| minimax | `minimax-m3` | `minimax/minimax-m3` | `OPENROUTER_API_KEY` |
|
||||||
|
| mistral | `mistral-large-2512` | `mistralai/mistral-large-2512` | `OPENROUTER_API_KEY` |
|
||||||
|
| mistral | `mistral-medium-3-5` | `mistralai/mistral-medium-3-5` | `OPENROUTER_API_KEY` |
|
||||||
|
| mistral | `mistral-small-3` | `mistralai/mistral-small-3.2-24b-instruct` | `OPENROUTER_API_KEY` |
|
||||||
|
| mistral | `mistral-small-4` | `mistralai/mistral-small-2603` | `OPENROUTER_API_KEY` |
|
||||||
|
| nvidia | `nemotron-3-ultra` | `nvidia/nemotron-3-ultra-550b-a55b` | `OPENROUTER_API_KEY` |
|
||||||
|
| openai | `gpt-5-mini` | `openai/gpt-5.4-mini` | `OPENROUTER_API_KEY` |
|
||||||
|
| openai | `gpt-5-nano` | `openai/gpt-5.4-nano` | `OPENROUTER_API_KEY` |
|
||||||
|
|
||||||
|
## Schema Behavior
|
||||||
|
|
||||||
|
Schemas are JSON files, typically under `schema_dir`.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- `output.validation_mode: json_schema` requires `output.schema_path`.
|
||||||
|
- Relative `schema_path` values resolve from `schema_dir`.
|
||||||
|
- Absolute `schema_path` values are used directly.
|
||||||
|
- Nested schemas must be referenced by relative path; schemas are not searched recursively by basename.
|
||||||
|
- Missing or invalid schema documents are runtime validation errors.
|
||||||
|
- Invalid generated JSON produces validation status `failed`, not a runtime error.
|
||||||
|
|
||||||
|
## Artifact References
|
||||||
|
|
||||||
|
Supported request input artifact reference types are:
|
||||||
|
|
||||||
|
- `file`
|
||||||
|
- `inline`
|
||||||
|
|
||||||
|
CLI `run` and `render` create `file` references from `--input name=path`.
|
||||||
|
|
||||||
|
HTTP `file` references require `server.artifact_root` or `serve
|
||||||
|
--artifact-root`. Relative file URIs resolve under that root. Absolute paths
|
||||||
|
and relative traversal outside the root are rejected by lexical checks. Symlinks
|
||||||
|
inside the root are followed by the operating system, including symlinks that
|
||||||
|
point outside the root.
|
||||||
|
|
||||||
|
HTTP `inline` references do not require an artifact root.
|
||||||
|
|
||||||
|
## Secrets Handling
|
||||||
|
|
||||||
|
- Keep secret values in environment variables.
|
||||||
|
- Store only environment-variable names in `api_key_env`.
|
||||||
|
- Do not put raw API keys in config, prompts, profiles, CLI arguments, examples, or HTTP request bodies.
|
||||||
|
|
||||||
|
## Maintained Examples
|
||||||
|
|
||||||
|
- Minimal app config: `examples/config.yml`
|
||||||
|
- Full app config: `examples/config.full.yml`
|
||||||
|
- Prompt examples: `examples/prompts/`
|
||||||
|
- Custom profile examples: `examples/profiles/`
|
||||||
|
- Schema examples: `examples/schemas/`
|
||||||
|
- Input fixtures: `examples/fixtures/`
|
||||||
|
- Render script: `examples/render-markdown-summary.sh`
|
||||||
|
- HTTP request-shape example: `examples/http-run.json`
|
||||||
|
|
||||||
|
## Integration References
|
||||||
|
|
||||||
|
- [CLI reference](cli.md)
|
||||||
|
- [HTTP API reference](api.md)
|
||||||
|
- [Outbound OpenAI-compatible contract](integrations/openai-compatible-chat.md)
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
# Main `config.yml`
|
|
||||||
|
|
||||||
`config.yml` defines application-level defaults used by CLI commands.
|
|
||||||
|
|
||||||
By default, Scriptorium looks for `/usr/local/etc/scriptorium/config.yml` and, if not present, then for `/etc/scriptorium/config.yml`. You can also pass `--config PATH`.
|
|
||||||
|
|
||||||
## Complete Example
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
prompt_dir: ./prompts
|
|
||||||
profile_dir: ./profiles
|
|
||||||
schema_dir: ./schemas
|
|
||||||
|
|
||||||
server:
|
|
||||||
addr: :8080
|
|
||||||
|
|
||||||
defaults:
|
|
||||||
render_format: text
|
|
||||||
```
|
|
||||||
|
|
||||||
## Available Options
|
|
||||||
|
|
||||||
- `prompt_dir` (optional): Default directory for prompt definition YAML files.
|
|
||||||
- `profile_dir` (optional): Default directory for execution profile YAML files.
|
|
||||||
- `schema_dir` (optional): Base directory for JSON schema files used by `json_schema` validation.
|
|
||||||
|
|
||||||
### `server`
|
|
||||||
|
|
||||||
- `addr` (optional): HTTP server listen address for `scriptorium serve`.
|
|
||||||
|
|
||||||
### `defaults`
|
|
||||||
|
|
||||||
- `render_format` (optional): Default output format for `scriptorium render`.
|
|
||||||
- Allowed values: `text`, `json`.
|
|
||||||
|
|
||||||
## Precedence
|
|
||||||
|
|
||||||
For run/render/serve settings, precedence is:
|
|
||||||
|
|
||||||
1. Explicit CLI flags
|
|
||||||
2. `config.yml`
|
|
||||||
3. Built-in defaults
|
|
||||||
|
|
||||||
## Notes and Rules
|
|
||||||
|
|
||||||
- Unknown YAML fields fail to load (strict decoding).
|
|
||||||
- This file does not accept API keys.
|
|
||||||
- `config.yml` sets directory/server defaults only; prompt/profile content remains in their own files.
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
# Execution Profile Definitions
|
|
||||||
|
|
||||||
Execution Profiles define **how** Scriptorium calls an LLM endpoint.
|
|
||||||
|
|
||||||
A profile file is YAML, typically stored under `profiles/`, for example `profiles/local-quality.yaml`.
|
|
||||||
|
|
||||||
## Complete Example
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
id: local-quality
|
|
||||||
endpoint: http://localhost:8000/v1
|
|
||||||
model: gpt-4.1
|
|
||||||
temperature: 0.0
|
|
||||||
max_tokens: 1200
|
|
||||||
top_p: 1.0
|
|
||||||
timeout_seconds: 180
|
|
||||||
reasoning_effort: medium
|
|
||||||
api_key_env: SCRIPTORIUM_API_KEY
|
|
||||||
extra_params:
|
|
||||||
provider: openrouter
|
|
||||||
route: fallback
|
|
||||||
```
|
|
||||||
|
|
||||||
## Available Options
|
|
||||||
|
|
||||||
- `id` (required): Unique profile identifier used by `--profile` or prompt `default_profile`.
|
|
||||||
- `endpoint` (required): OpenAI-compatible base URL, usually ending in `/v1`.
|
|
||||||
- `model` (required): Model name to request at that endpoint.
|
|
||||||
- `temperature` (optional): Sampling temperature. Valid range is `0` to `2`.
|
|
||||||
- `max_tokens` (optional): Max completion tokens. Must be `>= 0`.
|
|
||||||
- `top_p` (optional): Nucleus sampling parameter. Valid range is `0` to `1`.
|
|
||||||
- `timeout_seconds` (optional): Request timeout in seconds. Must be `>= 0`.
|
|
||||||
- `reasoning_effort` (optional): Provider/model-specific reasoning level string.
|
|
||||||
- `api_key_env` (optional): Environment variable name that holds the API key.
|
|
||||||
- `extra_params` (optional): String key/value map for provider-specific parameters.
|
|
||||||
|
|
||||||
## Notes and Rules
|
|
||||||
|
|
||||||
- Raw API keys are not supported. Do **not** add `api_key` fields.
|
|
||||||
- Unknown YAML fields fail to load (strict decoding).
|
|
||||||
- If `api_key_env` is set, the environment variable must be present when the run executes.
|
|
||||||
@@ -1,73 +0,0 @@
|
|||||||
# Prompt Definition Files
|
|
||||||
|
|
||||||
Prompt Definitions define **what** Scriptorium should do.
|
|
||||||
|
|
||||||
A prompt file is YAML, typically stored under `prompts/`, for example `prompts/generic.structured_events.yaml`.
|
|
||||||
|
|
||||||
## Complete Example
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
id: generic.structured_events
|
|
||||||
version: "1.0.0"
|
|
||||||
default_profile: local-quality
|
|
||||||
description: Extract events from a transcript into structured JSON.
|
|
||||||
|
|
||||||
inputs:
|
|
||||||
- name: transcript
|
|
||||||
required: true
|
|
||||||
content_type: text/markdown
|
|
||||||
description: Source transcript
|
|
||||||
- name: glossary
|
|
||||||
required: false
|
|
||||||
content_type: text/yaml
|
|
||||||
description: Optional glossary context
|
|
||||||
|
|
||||||
messages:
|
|
||||||
- role: system
|
|
||||||
content: |
|
|
||||||
You are a structured extraction assistant.
|
|
||||||
Return only JSON.
|
|
||||||
- role: user
|
|
||||||
content_file: ./generic.structured_events.user.md
|
|
||||||
|
|
||||||
output:
|
|
||||||
format: json
|
|
||||||
validation_mode: json_schema
|
|
||||||
schema_path: structured_events.schema.json
|
|
||||||
repair_attempts: 1
|
|
||||||
```
|
|
||||||
|
|
||||||
## Available Options
|
|
||||||
|
|
||||||
- `id` (required): Prompt identifier used by `--prompt` / `prompt_id`.
|
|
||||||
- `version` (required): Prompt version string.
|
|
||||||
- `default_profile` (optional): Execution profile ID used when no explicit profile is provided.
|
|
||||||
- `description` (optional): Human-readable description.
|
|
||||||
|
|
||||||
### `inputs[]`
|
|
||||||
|
|
||||||
- `name` (required): Logical input name referenced in templates via `{{input "name"}}`.
|
|
||||||
- `required` (optional): If `true`, run fails when input is missing.
|
|
||||||
- `content_type` (optional): Metadata only (not enforced yet).
|
|
||||||
- `description` (optional): Human-readable input description.
|
|
||||||
|
|
||||||
### `messages[]`
|
|
||||||
|
|
||||||
- `role` (required): Message role such as `system` or `user`.
|
|
||||||
- `content` (optional): Inline Go-template message body.
|
|
||||||
- `content_file` (optional): Path to a template file.
|
|
||||||
|
|
||||||
Each message must set **exactly one** of `content` or `content_file`.
|
|
||||||
|
|
||||||
### `output`
|
|
||||||
|
|
||||||
- `format` (required): One of `text`, `markdown`, `json`.
|
|
||||||
- `validation_mode` (required): One of `none`, `basic`, `json`, `json_schema`.
|
|
||||||
- `schema_path` (required when `validation_mode: json_schema`): Path to JSON Schema file.
|
|
||||||
- `repair_attempts` (required): Number of bounded repair retries (`>= 0`).
|
|
||||||
|
|
||||||
## Notes and Rules
|
|
||||||
|
|
||||||
- Unknown YAML fields fail to load (strict decoding).
|
|
||||||
- `content_file` paths are resolved relative to the prompt YAML file.
|
|
||||||
- For `json_schema` validation mode, Scriptorium also sends provider-level structured output requests automatically.
|
|
||||||
@@ -1,82 +0,0 @@
|
|||||||
# JSON Schema Definition Files
|
|
||||||
|
|
||||||
Schema definition files describe the expected JSON output contract for prompts that use:
|
|
||||||
|
|
||||||
- `output.format: json`
|
|
||||||
- `output.validation_mode: json_schema`
|
|
||||||
|
|
||||||
Schema files are JSON, typically stored under `schemas/`, for example `schemas/structured_events.schema.json`.
|
|
||||||
|
|
||||||
## Complete Example
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
||||||
"$id": "https://example.com/schemas/structured-events.schema.json",
|
|
||||||
"title": "Structured Events",
|
|
||||||
"description": "Expected shape for extracted event output",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"summary": {
|
|
||||||
"type": "string",
|
|
||||||
"minLength": 1,
|
|
||||||
"description": "High-level session summary"
|
|
||||||
},
|
|
||||||
"events": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"title": { "type": "string" },
|
|
||||||
"type": {
|
|
||||||
"type": "string",
|
|
||||||
"enum": ["discovery", "combat", "social", "travel", "downtime", "other"]
|
|
||||||
},
|
|
||||||
"notes": { "type": "string" }
|
|
||||||
},
|
|
||||||
"required": ["title", "type"],
|
|
||||||
"additionalProperties": false
|
|
||||||
},
|
|
||||||
"minItems": 0
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": ["summary", "events"],
|
|
||||||
"additionalProperties": false,
|
|
||||||
"$defs": {
|
|
||||||
"nonEmptyString": {
|
|
||||||
"type": "string",
|
|
||||||
"minLength": 1
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Available Options
|
|
||||||
|
|
||||||
Scriptorium does not define custom schema keywords. It expects a valid JSON Schema document and passes it to the validator/provider.
|
|
||||||
|
|
||||||
Commonly used JSON Schema options include:
|
|
||||||
|
|
||||||
- `$schema`: Draft identifier URI.
|
|
||||||
- `$id`: Schema identifier URI.
|
|
||||||
- `title`: Human-readable schema title.
|
|
||||||
- `description`: Human-readable schema description.
|
|
||||||
- `type`: Expected JSON type (`object`, `array`, `string`, etc.).
|
|
||||||
- `properties`: Object field definitions.
|
|
||||||
- `required`: Required object fields.
|
|
||||||
- `additionalProperties`: Whether undeclared fields are allowed.
|
|
||||||
- `items`: Array item schema.
|
|
||||||
- `enum`: Allowed literal values.
|
|
||||||
- `const`: Single allowed literal value.
|
|
||||||
- `oneOf`, `anyOf`, `allOf`: Composition rules.
|
|
||||||
- `minimum`, `maximum`: Numeric bounds.
|
|
||||||
- `minLength`, `maxLength`, `pattern`: String constraints.
|
|
||||||
- `minItems`, `maxItems`: Array constraints.
|
|
||||||
- `$defs`: Reusable local definitions.
|
|
||||||
- `$ref`: Reference to another schema/definition.
|
|
||||||
|
|
||||||
## Notes and Rules
|
|
||||||
|
|
||||||
- Schema path comes from prompt `output.schema_path` and is resolved relative to `schema_dir`.
|
|
||||||
- If schema loading fails for `json_schema` mode, the run fails before the LLM request.
|
|
||||||
- Keep schemas strict (`additionalProperties: false`) when you want predictable output shape.
|
|
||||||
122
docs/consumers/api.md
Normal file
122
docs/consumers/api.md
Normal file
@@ -0,0 +1,122 @@
|
|||||||
|
# Consumer Integration Overview
|
||||||
|
|
||||||
|
This guide is for applications that call Scriptorium from another codebase.
|
||||||
|
|
||||||
|
Scriptorium exposes three integration surfaces:
|
||||||
|
|
||||||
|
| Surface | Use when |
|
||||||
|
| --- | --- |
|
||||||
|
| Go package | The consumer is Go, needs typed requests/results, or wants injected LLM clients for tests. |
|
||||||
|
| CLI subprocess | The consumer wants process isolation or is not written in Go. |
|
||||||
|
| HTTP API | The consumer needs a service boundary or remote access to `POST /v1/runs`. |
|
||||||
|
|
||||||
|
Canonical references:
|
||||||
|
|
||||||
|
- Go package: [Package scriptorium](pkg-scriptorium.md)
|
||||||
|
- CLI subprocess: [Subprocess integration](../integrations/subprocess.md)
|
||||||
|
- HTTP: [HTTP API reference](../api.md)
|
||||||
|
- File formats: [Configuration reference](../config.md)
|
||||||
|
|
||||||
|
## Required Deployment Inputs
|
||||||
|
|
||||||
|
Every integration needs operators to provide:
|
||||||
|
|
||||||
|
- prompt definitions;
|
||||||
|
- profile definitions or built-in profile IDs;
|
||||||
|
- schema files when prompts use `json_schema`;
|
||||||
|
- input artifacts or inline input bodies;
|
||||||
|
- API-key environment variables or direct per-request keys where supported.
|
||||||
|
|
||||||
|
Raw API keys do not belong in config, prompt files, profile YAML, CLI
|
||||||
|
arguments, or HTTP request bodies.
|
||||||
|
|
||||||
|
## Recommended Workflow
|
||||||
|
|
||||||
|
Use the Go package when:
|
||||||
|
|
||||||
|
- the consumer is a Go application;
|
||||||
|
- the application needs `context.Context` cancellation;
|
||||||
|
- repeated calls should avoid subprocess startup;
|
||||||
|
- tests need a fake LLM client;
|
||||||
|
- direct per-request `RunRequest.APIKey` is required.
|
||||||
|
|
||||||
|
Use the CLI subprocess when:
|
||||||
|
|
||||||
|
- the consumer is not Go;
|
||||||
|
- process isolation is useful;
|
||||||
|
- stdout/stderr separation and exit codes are enough;
|
||||||
|
- the consumer already manages local files and environment variables.
|
||||||
|
|
||||||
|
Use HTTP when:
|
||||||
|
|
||||||
|
- Scriptorium should run as a service;
|
||||||
|
- multiple clients need a shared prompt/profile deployment;
|
||||||
|
- clients can reach a trusted, protected HTTP boundary.
|
||||||
|
|
||||||
|
## Minimal Go Example
|
||||||
|
|
||||||
|
```go
|
||||||
|
engine, err := scriptorium.NewEngine(scriptorium.Config{
|
||||||
|
PromptDir: "./examples/prompts",
|
||||||
|
ProfileDir: "./examples/profiles",
|
||||||
|
SchemaDir: "./examples/schemas",
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
prepared, err := engine.Prepare(ctx, scriptorium.RunRequest{
|
||||||
|
PromptID: "generic.markdown_summary",
|
||||||
|
Inputs: map[string]scriptorium.ArtifactRef{
|
||||||
|
"transcript": scriptorium.File("./examples/fixtures/transcript.md"),
|
||||||
|
"glossary": scriptorium.File("./examples/fixtures/glossary.yml"),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
_ = prepared.Messages
|
||||||
|
```
|
||||||
|
|
||||||
|
Run the maintained package example:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./examples/go-library/prepare
|
||||||
|
```
|
||||||
|
|
||||||
|
## Subprocess Workflow
|
||||||
|
|
||||||
|
Invoke `scriptorium render` for preflight and `scriptorium run` for generation.
|
||||||
|
Capture stdout and stderr separately. Treat exit code `2` from `run` as a
|
||||||
|
completed generation with failed validation.
|
||||||
|
|
||||||
|
See [Subprocess integration](../integrations/subprocess.md) for the stable
|
||||||
|
invocation contract.
|
||||||
|
|
||||||
|
## HTTP Workflow
|
||||||
|
|
||||||
|
Run `scriptorium serve` behind trusted controls and send JSON requests to
|
||||||
|
`POST /v1/runs`.
|
||||||
|
|
||||||
|
Do not duplicate endpoint schemas in consumers. Use the [HTTP API
|
||||||
|
reference](../api.md) as the authoritative contract.
|
||||||
|
|
||||||
|
## Consumer Responsibilities
|
||||||
|
|
||||||
|
Consumers are responsible for:
|
||||||
|
|
||||||
|
- selecting prompt/profile IDs as deployment configuration;
|
||||||
|
- supplying all required inputs and vars;
|
||||||
|
- protecting generated artifacts and rendered prompts as sensitive data;
|
||||||
|
- deciding whether to keep output when validation fails;
|
||||||
|
- implementing retries only when another model call is acceptable.
|
||||||
|
|
||||||
|
Scriptorium does not persist run state. Retrying a failed or timed-out request
|
||||||
|
can produce different output and can incur another provider request.
|
||||||
|
|
||||||
|
## Status Behavior
|
||||||
|
|
||||||
|
- Go package methods return typed results or errors that support `errors.Is`.
|
||||||
|
- CLI `run` exits `2` when generation succeeds but validation fails.
|
||||||
|
- HTTP returns `200 OK` for generated-content validation failures and exposes the failed status in the response body.
|
||||||
|
- Runtime validation failures are errors.
|
||||||
284
docs/consumers/pkg-scriptorium.md
Normal file
284
docs/consumers/pkg-scriptorium.md
Normal file
@@ -0,0 +1,284 @@
|
|||||||
|
# Package scriptorium
|
||||||
|
|
||||||
|
Import path:
|
||||||
|
|
||||||
|
```go
|
||||||
|
import "gitea.maximumdirect.net/eric/scriptorium"
|
||||||
|
```
|
||||||
|
|
||||||
|
The root package is the public Go facade for Scriptorium's prompt prepare/run
|
||||||
|
workflow. It exposes typed requests, results, source options, injected LLM
|
||||||
|
clients, and stable public errors while keeping `internal/*` packages private.
|
||||||
|
|
||||||
|
## Intended Use Cases
|
||||||
|
|
||||||
|
Use the package when a Go application needs:
|
||||||
|
|
||||||
|
- in-process prompt preparation or execution;
|
||||||
|
- typed request/result structs;
|
||||||
|
- direct `context.Context` cancellation;
|
||||||
|
- injected/fake LLM clients for tests;
|
||||||
|
- direct per-request `RunRequest.APIKey`.
|
||||||
|
|
||||||
|
Use [Subprocess integration](../integrations/subprocess.md) or the [HTTP API](../api.md)
|
||||||
|
when a process or service boundary is preferred.
|
||||||
|
|
||||||
|
## Construct An Engine
|
||||||
|
|
||||||
|
```go
|
||||||
|
engine, err := scriptorium.NewEngine(scriptorium.Config{
|
||||||
|
PromptDir: "./examples/prompts",
|
||||||
|
ProfileDir: "./examples/profiles",
|
||||||
|
SchemaDir: "./examples/schemas",
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`Config` fields:
|
||||||
|
|
||||||
|
| Field | Description |
|
||||||
|
| --- | --- |
|
||||||
|
| `PromptDir` | Prompt definition directory. Required unless `WithPromptFS` or `WithPromptFile` is used. |
|
||||||
|
| `ProfileDir` | Optional custom profile directory overlaid above built-in profiles. |
|
||||||
|
| `SchemaDir` | Schema directory. Defaults to `.` when empty. |
|
||||||
|
| `Timeout` | Default timeout for the built-in OpenAI-compatible client. |
|
||||||
|
| `HTTPClient` | Optional HTTP client for the built-in OpenAI-compatible client. |
|
||||||
|
|
||||||
|
`NewEngine` accepts `nil` options and ignores them. Invalid construction wraps
|
||||||
|
`ErrInvalidConfig`.
|
||||||
|
|
||||||
|
## Source Options
|
||||||
|
|
||||||
|
Directory fields are the compatibility path. Explicit source options override
|
||||||
|
the matching directory field.
|
||||||
|
|
||||||
|
Prompt sources:
|
||||||
|
|
||||||
|
- `WithPromptFS(fsys, root)`
|
||||||
|
- `WithPromptFile(path)`
|
||||||
|
|
||||||
|
Profile sources:
|
||||||
|
|
||||||
|
- `WithProfileFS(fsys, root)`
|
||||||
|
- `WithProfileFile(path)`
|
||||||
|
- `WithProfiles(profiles...)`
|
||||||
|
|
||||||
|
Schema sources:
|
||||||
|
|
||||||
|
- `WithSchemaFS(fsys, root)`
|
||||||
|
- `WithSchemaFile(path)`
|
||||||
|
|
||||||
|
LLM source:
|
||||||
|
|
||||||
|
- `WithLLMClient(client)`
|
||||||
|
|
||||||
|
Source behavior:
|
||||||
|
|
||||||
|
- Prompt and profile YAML use the same strict rules as directory loading.
|
||||||
|
- Prompt `content_file` values resolve relative to the prompt file.
|
||||||
|
- `fs.FS` roots are containment boundaries for prompt content files and schema paths.
|
||||||
|
- File options expose the selected file by its base name.
|
||||||
|
- Profile source precedence is in-memory profiles, then explicit profile file/FS/directory source, then built-ins.
|
||||||
|
- `WithLLMClient(nil)` returns `ErrInvalidConfig`.
|
||||||
|
|
||||||
|
## In-Memory Profiles
|
||||||
|
|
||||||
|
Use `WithProfiles` when the application already has typed model settings:
|
||||||
|
|
||||||
|
```go
|
||||||
|
profile := scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
||||||
|
ID: "app.default",
|
||||||
|
Endpoint: "https://openrouter.ai/api/v1",
|
||||||
|
Model: "mistralai/mistral-small-3.2-24b-instruct",
|
||||||
|
APIKeyRequired: true,
|
||||||
|
})
|
||||||
|
|
||||||
|
engine, err := scriptorium.NewEngine(cfg, scriptorium.WithProfiles(profile))
|
||||||
|
```
|
||||||
|
|
||||||
|
`Profile` and `OpenAICompatibleProfileConfig` include:
|
||||||
|
|
||||||
|
- `ID`
|
||||||
|
- `Endpoint`
|
||||||
|
- `Model`
|
||||||
|
- `Temperature`
|
||||||
|
- `MaxTokens`
|
||||||
|
- `TopP`
|
||||||
|
- `TimeoutSeconds`
|
||||||
|
- `ServiceTier`
|
||||||
|
- `ReasoningEffort`
|
||||||
|
- `APIKeyRequired`
|
||||||
|
- `ExtraParams`
|
||||||
|
|
||||||
|
`WithProfiles` rejects duplicate IDs in one call. In-memory profiles do not
|
||||||
|
store raw keys. When `APIKeyRequired` is true, pass the secret on each request
|
||||||
|
with `RunRequest.APIKey`.
|
||||||
|
|
||||||
|
`ExtraParams` must be JSON-compatible: strings, booleans, finite numbers,
|
||||||
|
objects with string keys, arrays/slices, and nil. Unsupported values, non-string
|
||||||
|
map keys, non-finite floats, and cycles return `ErrInvalidConfig` for profiles
|
||||||
|
or `ErrInvalidRequest` for request overrides.
|
||||||
|
|
||||||
|
## Prepare Workflow
|
||||||
|
|
||||||
|
`Prepare` resolves prompt/profile/input/schema state and renders messages
|
||||||
|
without calling an LLM.
|
||||||
|
|
||||||
|
```go
|
||||||
|
prepared, err := engine.Prepare(ctx, scriptorium.RunRequest{
|
||||||
|
PromptID: "generic.markdown_summary",
|
||||||
|
Inputs: map[string]scriptorium.ArtifactRef{
|
||||||
|
"transcript": scriptorium.File("./examples/fixtures/transcript.md"),
|
||||||
|
"glossary": scriptorium.File("./examples/fixtures/glossary.yml"),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
_ = prepared.EffectiveModelParams
|
||||||
|
```
|
||||||
|
|
||||||
|
`PreparedRun` includes prompt ID/version/hash, selected profile, effective
|
||||||
|
model params, output contract, structured-output metadata, input hashes,
|
||||||
|
rendered prompt hash, rendered messages, and timing fields. It does not include
|
||||||
|
raw API-key values, model output, validation results, or internal target
|
||||||
|
presence metadata.
|
||||||
|
|
||||||
|
## Run Workflow
|
||||||
|
|
||||||
|
`Run` calls `Prepare`, invokes the configured LLM client, builds the output
|
||||||
|
artifact, and validates the output.
|
||||||
|
|
||||||
|
```go
|
||||||
|
result, err := engine.Run(ctx, scriptorium.RunRequest{
|
||||||
|
PromptID: "generic.markdown_summary",
|
||||||
|
APIKey: apiKey,
|
||||||
|
Inputs: map[string]scriptorium.ArtifactRef{
|
||||||
|
"transcript": scriptorium.File("./examples/fixtures/transcript.md"),
|
||||||
|
"glossary": scriptorium.File("./examples/fixtures/glossary.yml"),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
_ = result.Artifact
|
||||||
|
```
|
||||||
|
|
||||||
|
`RunResult` includes run ID, output artifact, raw output, validation result,
|
||||||
|
prompt/profile/model metadata, effective model params, input hashes, usage, and
|
||||||
|
timing fields.
|
||||||
|
|
||||||
|
Generated-content validation failures return a successful `RunResult` with
|
||||||
|
`Validation.Status == ValidationFailed`. Runtime/schema validation errors
|
||||||
|
return an error that matches `ErrValidation`.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
Input helpers:
|
||||||
|
|
||||||
|
- `File(path)`: file-backed artifact reference.
|
||||||
|
- `Inline(body)`: inline artifact body.
|
||||||
|
- `InlineWithURI(uri, body)`: inline artifact body with URI metadata.
|
||||||
|
|
||||||
|
Input map keys must match the prompt's expected input names.
|
||||||
|
|
||||||
|
## Injected LLM Clients
|
||||||
|
|
||||||
|
Use `WithLLMClient` for tests or custom model integrations:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type fakeLLM struct{}
|
||||||
|
|
||||||
|
func (fakeLLM) Generate(ctx context.Context, req scriptorium.GenerateRequest) (*scriptorium.GenerateResponse, error) {
|
||||||
|
return &scriptorium.GenerateResponse{
|
||||||
|
Content: "generated text",
|
||||||
|
Usage: scriptorium.TokenUsage{TotalTokens: 12},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
engine, err := scriptorium.NewEngine(cfg, scriptorium.WithLLMClient(fakeLLM{}))
|
||||||
|
```
|
||||||
|
|
||||||
|
Injected clients receive:
|
||||||
|
|
||||||
|
- rendered prompt;
|
||||||
|
- effective execution target;
|
||||||
|
- numeric target presence metadata;
|
||||||
|
- structured-output spec when applicable;
|
||||||
|
- direct request API key when provided.
|
||||||
|
|
||||||
|
Custom clients should not log raw prompts or API keys by default.
|
||||||
|
|
||||||
|
## Overrides And API Keys
|
||||||
|
|
||||||
|
`RunRequest` fields:
|
||||||
|
|
||||||
|
| Field | Description |
|
||||||
|
| --- | --- |
|
||||||
|
| `PromptID` | Prompt ID. |
|
||||||
|
| `PromptVersion` | Optional prompt version filter. |
|
||||||
|
| `ProfileID` | Optional profile override. |
|
||||||
|
| `APIKey` | Direct per-request API key. |
|
||||||
|
| `Inputs` | Input artifact references. |
|
||||||
|
| `Vars` | Template variables. |
|
||||||
|
| `Execution` | Per-request model overrides. |
|
||||||
|
| `Validation` | Per-request output contract override. |
|
||||||
|
| `Metadata` | Request metadata reserved for callers. |
|
||||||
|
|
||||||
|
`RunRequest.Execution` uses pointer fields for numeric values so explicit zero
|
||||||
|
overrides are preserved:
|
||||||
|
|
||||||
|
```go
|
||||||
|
zero := 0
|
||||||
|
req.Execution = &scriptorium.ExecutionTargetOverride{
|
||||||
|
MaxTokens: &zero,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Direct `RunRequest.APIKey` takes precedence over profile `api_key_env` for the
|
||||||
|
default OpenAI-compatible client. It is request-scoped, uses `json:"-"`, and is
|
||||||
|
not included in `PreparedRun` or `RunResult` JSON. Normal Go string formatting
|
||||||
|
of `RunRequest` and `GenerateRequest` reports only whether a direct key is set.
|
||||||
|
|
||||||
|
Raw API keys do not belong in profile YAML, in-memory profiles, or app config.
|
||||||
|
Avoid reflection-based debug dumps of request structs because exported fields
|
||||||
|
remain visible to tools that bypass `String` and `GoString`.
|
||||||
|
|
||||||
|
## Errors
|
||||||
|
|
||||||
|
Public methods wrap context while preserving stable sentinel checks with
|
||||||
|
`errors.Is`:
|
||||||
|
|
||||||
|
- `ErrInvalidConfig`
|
||||||
|
- `ErrInvalidRequest`
|
||||||
|
- `ErrPromptNotFound`
|
||||||
|
- `ErrProfileNotFound`
|
||||||
|
- `ErrPromptLoad`
|
||||||
|
- `ErrProfileLoad`
|
||||||
|
- `ErrArtifactLoad`
|
||||||
|
- `ErrPromptRender`
|
||||||
|
- `ErrLLMGenerate`
|
||||||
|
- `ErrValidation`
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```go
|
||||||
|
if errors.Is(err, scriptorium.ErrPromptNotFound) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
Run the maintained prepare-only example from the repository root:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./examples/go-library/prepare
|
||||||
|
```
|
||||||
|
|
||||||
|
See also:
|
||||||
|
|
||||||
|
- [Configuration reference](../config.md)
|
||||||
|
- [Consumer integration overview](api.md)
|
||||||
@@ -1,322 +0,0 @@
|
|||||||
# Narratio -> Scriptorium CLI Integration
|
|
||||||
|
|
||||||
## 1. Purpose
|
|
||||||
|
|
||||||
This document defines how Narratio should invoke Scriptorium through the **public CLI**.
|
|
||||||
|
|
||||||
This is a **subprocess integration contract**, not an internal Go API contract.
|
|
||||||
|
|
||||||
## 2. Assumptions
|
|
||||||
|
|
||||||
- `scriptorium` is installed and available on `PATH`.
|
|
||||||
- Scriptorium is configured with `config.yml`.
|
|
||||||
- `config.yml` provides `prompt_dir`, `profile_dir`, and `schema_dir` as needed.
|
|
||||||
- Prompt and profile libraries are already deployed for the environment.
|
|
||||||
- Narratio provides prepared artifact files (for example polished transcript, glossary, previous recap, campaign notes).
|
|
||||||
- Initial integration is synchronous subprocess execution.
|
|
||||||
- Narratio remains the orchestrator.
|
|
||||||
|
|
||||||
In normal operation, Narratio does not need to pass `--prompt-dir` and `--profile-dir` if they are supplied by Scriptorium config.
|
|
||||||
|
|
||||||
Narratio may pass `--config <PATH>` when it must use a non-default Scriptorium config file.
|
|
||||||
|
|
||||||
## 3. Core Commands Narratio May Call
|
|
||||||
|
|
||||||
Primary commands for subprocess integration:
|
|
||||||
|
|
||||||
- `scriptorium run`
|
|
||||||
- `scriptorium render`
|
|
||||||
|
|
||||||
For production generation, use `scriptorium run`.
|
|
||||||
|
|
||||||
`scriptorium render` is for debugging, dry-runs, test assertions, and validating command construction without LLM execution.
|
|
||||||
|
|
||||||
Note: `scriptorium serve` and HTTP API exist, but they are not the initial integration path.
|
|
||||||
|
|
||||||
## 4. Command Selection Guidance
|
|
||||||
|
|
||||||
- Use `run` to generate an output artifact.
|
|
||||||
- Use `render` to inspect the prepared prompt and effective settings without calling the LLM.
|
|
||||||
- Use `render --format json` when Narratio/tests need structured prepare output.
|
|
||||||
|
|
||||||
## 5. Recommended `run` Invocation Shape
|
|
||||||
|
|
||||||
Production shape:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt <prompt_id> \
|
|
||||||
--input transcript=<processed-transcript-path> \
|
|
||||||
--out <output-artifact-path>
|
|
||||||
```
|
|
||||||
|
|
||||||
Common optional additions:
|
|
||||||
|
|
||||||
- `--config <path>`: use a specific Scriptorium config file.
|
|
||||||
- `--profile <profile_id>`: override prompt default profile.
|
|
||||||
- `--var name=value` (repeatable): small metadata values.
|
|
||||||
- `--input name=path` (repeatable): additional named artifacts.
|
|
||||||
- `--timeout <duration>`: per-run timeout override.
|
|
||||||
- Runtime model override flags (`--llm-base-url`, `--model`, etc.) only for exceptional/operator-directed cases.
|
|
||||||
|
|
||||||
## 6. Recommended `render` Invocation Shape
|
|
||||||
|
|
||||||
Human-readable debug shape:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--prompt <prompt_id> \
|
|
||||||
--input transcript=<processed-transcript-path> \
|
|
||||||
--format text
|
|
||||||
```
|
|
||||||
|
|
||||||
Structured debug/test shape:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--prompt <prompt_id> \
|
|
||||||
--input transcript=<processed-transcript-path> \
|
|
||||||
--format json \
|
|
||||||
--out <render-debug-path>
|
|
||||||
```
|
|
||||||
|
|
||||||
`render` does **not** call the LLM, does **not** validate model output, and does **not** perform repair.
|
|
||||||
|
|
||||||
## 7. Inputs
|
|
||||||
|
|
||||||
- Pass inputs as repeated `--input name=path` flags.
|
|
||||||
- `name` must match the Prompt Definition input name.
|
|
||||||
- Prefer absolute paths, or paths relative to a working directory controlled by Narratio.
|
|
||||||
- Pass Audita output as the primary transcript input.
|
|
||||||
- Additional inputs may include glossary, previous recap, campaign notes, event logs, final state maps, or other prompt-specific artifacts.
|
|
||||||
- Scriptorium reads input files directly; Narratio does not need to inline file content for CLI use.
|
|
||||||
|
|
||||||
## 8. Variables
|
|
||||||
|
|
||||||
Use repeated `--var name=value` for small metadata values.
|
|
||||||
|
|
||||||
Typical examples:
|
|
||||||
|
|
||||||
- `session_date`
|
|
||||||
- `session_id`
|
|
||||||
- `campaign_name`
|
|
||||||
- `previous_session_id`
|
|
||||||
- `output_kind`
|
|
||||||
|
|
||||||
Large content belongs in input files, not `--var` values.
|
|
||||||
|
|
||||||
## 9. Prompt IDs and Output Artifact Types
|
|
||||||
|
|
||||||
Narratio should treat prompt IDs as configuration, not hardcoded business logic.
|
|
||||||
|
|
||||||
Narratio config may map stage/output names to prompt IDs, for example:
|
|
||||||
|
|
||||||
- session recap prompt
|
|
||||||
- structured event extraction prompt
|
|
||||||
- glossary suggestion prompt
|
|
||||||
- player-facing summary prompt
|
|
||||||
|
|
||||||
Prompt IDs used by Narratio should come from the deployed Scriptorium prompt library.
|
|
||||||
|
|
||||||
## 10. Profiles
|
|
||||||
|
|
||||||
- Prompts may declare `default_profile`.
|
|
||||||
- Narratio may omit `--profile` to use prompt default profile.
|
|
||||||
- Narratio may pass `--profile` to force profile selection.
|
|
||||||
- This enables environment/profile selection like `local-fast`, `local-quality`, `frontier`, `batch`, or test profiles.
|
|
||||||
- Profile names should generally be Narratio configuration values.
|
|
||||||
|
|
||||||
## 11. Runtime Overrides
|
|
||||||
|
|
||||||
Supported runtime override flags:
|
|
||||||
|
|
||||||
- `--llm-base-url`
|
|
||||||
- `--model`
|
|
||||||
- `--api-key-env`
|
|
||||||
- `--temperature`
|
|
||||||
- `--max-tokens`
|
|
||||||
- `--top-p`
|
|
||||||
- `--timeout`
|
|
||||||
|
|
||||||
Guidance:
|
|
||||||
|
|
||||||
- Keep normal model/runtime settings in Execution Profiles.
|
|
||||||
- Use runtime overrides only for explicit per-run exceptions, tests, or operator overrides.
|
|
||||||
- Never pass raw API keys on the command line.
|
|
||||||
- `--api-key-env` names an environment variable; Narratio must ensure that variable is set in subprocess environment.
|
|
||||||
|
|
||||||
## 12. Config Behavior
|
|
||||||
|
|
||||||
- Default config path: `/etc/scriptorium/config.yml`.
|
|
||||||
- `--config <PATH>` overrides default path.
|
|
||||||
- Missing default config is allowed by Scriptorium.
|
|
||||||
- If `--config` is provided explicitly, the file must exist and be valid.
|
|
||||||
- CLI flags override `config.yml`.
|
|
||||||
- `config.yml` overrides built-in application defaults.
|
|
||||||
|
|
||||||
Narratio can either:
|
|
||||||
|
|
||||||
- rely on system default config path, or
|
|
||||||
- carry an explicit config path and pass `--config`.
|
|
||||||
|
|
||||||
## 13. Environment Handling
|
|
||||||
|
|
||||||
Subprocess environment recommendations:
|
|
||||||
|
|
||||||
- Pass through required API-key environment variables referenced by `api_key_env`.
|
|
||||||
- Do not pass raw API keys as CLI arguments.
|
|
||||||
- Avoid logging full environment dumps.
|
|
||||||
- Capture stdout and stderr separately.
|
|
||||||
- Use a controlled working directory.
|
|
||||||
- Prefer absolute artifact paths.
|
|
||||||
|
|
||||||
## 14. Output Handling
|
|
||||||
|
|
||||||
For `scriptorium run`:
|
|
||||||
|
|
||||||
- Use `--out` when Narratio needs durable artifact files.
|
|
||||||
- Without `--out`, artifact content is written to stdout.
|
|
||||||
- Preferred orchestration pattern: always use `--out`, then treat the file as stage output artifact.
|
|
||||||
- Capture stderr for diagnostics.
|
|
||||||
|
|
||||||
For `scriptorium render`:
|
|
||||||
|
|
||||||
- Use `--out` to store render diagnostics.
|
|
||||||
- Use `--format json` when tests need to inspect selected profile, effective runtime settings, input hashes, prompt hash, and rendered messages.
|
|
||||||
|
|
||||||
## 15. Exit Status and Errors
|
|
||||||
|
|
||||||
Current CLI behavior (verified from implementation/tests):
|
|
||||||
|
|
||||||
- `0`: success.
|
|
||||||
- `1`: runtime/parse/config/load/render/generation/IO error.
|
|
||||||
- `2`: run completed but output validation failed (`ValidationFailed`).
|
|
||||||
|
|
||||||
Additional details:
|
|
||||||
|
|
||||||
- On `run`, output artifact write happens before exit code selection. If validation fails, artifact may still be written and exit code is `2`.
|
|
||||||
- `stderr` carries both errors and normal run summary output; non-empty stderr alone does not imply failure.
|
|
||||||
- `render` returns `0` on success and `1` on failures.
|
|
||||||
|
|
||||||
Narratio should treat non-zero exit codes as failed stage execution, but may record generated artifact paths if a run exited `2` and output file exists.
|
|
||||||
|
|
||||||
## 16. Recommended Narratio Integration Pattern
|
|
||||||
|
|
||||||
1. Build CLI args from Narratio stage configuration.
|
|
||||||
2. Use subprocess context cancellation/timeout.
|
|
||||||
3. Pass absolute input paths.
|
|
||||||
4. Pass `--out` to a session-scoped artifact path.
|
|
||||||
5. Add `--var` metadata values.
|
|
||||||
6. Optionally add `--config`.
|
|
||||||
7. Optionally add `--profile`.
|
|
||||||
8. Ensure required API-key env vars are present.
|
|
||||||
9. Run subprocess synchronously.
|
|
||||||
10. Capture stdout/stderr separately.
|
|
||||||
11. On success, store output artifact path and invocation metadata in stage artifacts.
|
|
||||||
12. On failure, store exit code and stderr diagnostics in stage status.
|
|
||||||
|
|
||||||
## 17. Suggested Narratio Configuration Shape
|
|
||||||
|
|
||||||
Illustrative (not required schema):
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
scriptorium:
|
|
||||||
config_path: /etc/scriptorium/config.yml
|
|
||||||
stages:
|
|
||||||
session_recap:
|
|
||||||
prompt_id: dnd.session_recap
|
|
||||||
profile_id: local-quality # optional
|
|
||||||
inputs: [transcript, glossary, previous_recap]
|
|
||||||
vars: [session_id, session_date, campaign_name]
|
|
||||||
output_path_template: artifacts/{session_id}/session_recap.md
|
|
||||||
timeout: 2m
|
|
||||||
render_debug: false
|
|
||||||
```
|
|
||||||
|
|
||||||
The key idea: map Narratio stage/artifact names to prompt ID, optional profile, expected inputs, and output destination.
|
|
||||||
|
|
||||||
## 18. Testing Strategy for Narratio Integration
|
|
||||||
|
|
||||||
- Use `scriptorium render --format json` to verify command construction without LLM calls.
|
|
||||||
- Use dedicated test prompt/profile libraries for integration tests.
|
|
||||||
- Use small fixture transcripts.
|
|
||||||
- Verify missing-input failure behavior.
|
|
||||||
- Verify prompt `default_profile` behavior.
|
|
||||||
- Verify explicit `--profile` override behavior.
|
|
||||||
- Verify `--config` behavior (default and explicit).
|
|
||||||
- Verify output file creation when `--out` is used.
|
|
||||||
- Verify stderr capture on failures.
|
|
||||||
- Avoid real API keys in tests.
|
|
||||||
|
|
||||||
## 19. Security and Privacy Notes
|
|
||||||
|
|
||||||
- Never pass raw API keys on command line.
|
|
||||||
- Do not log full rendered prompts by default; transcripts may contain sensitive content.
|
|
||||||
- Avoid logging prompt content unless explicit debug mode is enabled.
|
|
||||||
- Treat generated artifacts as potentially sensitive.
|
|
||||||
- Use session-scoped, access-controlled output paths.
|
|
||||||
- `api_key_env` names should come from environment management, not embedded secrets.
|
|
||||||
|
|
||||||
## 20. Initial D&D Artifact Generation Examples
|
|
||||||
|
|
||||||
These are examples only. Use prompt IDs from the deployed prompt library.
|
|
||||||
|
|
||||||
Session recap:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt dnd.session_recap \
|
|
||||||
--input transcript=/work/session-42/transcript.polished.md \
|
|
||||||
--input glossary=/work/session-42/glossary.yml \
|
|
||||||
--out /work/session-42/artifacts/session_recap.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Structured events:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt dnd.structured_events \
|
|
||||||
--input transcript=/work/session-42/transcript.polished.md \
|
|
||||||
--out /work/session-42/artifacts/structured_events.json
|
|
||||||
```
|
|
||||||
|
|
||||||
Glossary suggestions:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt dnd.glossary_suggestions \
|
|
||||||
--input transcript=/work/session-42/transcript.polished.md \
|
|
||||||
--input previous_recap=/work/session-41/artifacts/session_recap.md \
|
|
||||||
--out /work/session-42/artifacts/glossary_suggestions.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Player-facing summary:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt dnd.player_summary \
|
|
||||||
--input transcript=/work/session-42/transcript.polished.md \
|
|
||||||
--input structured_events=/work/session-42/artifacts/structured_events.json \
|
|
||||||
--out /work/session-42/artifacts/player_summary.md
|
|
||||||
```
|
|
||||||
|
|
||||||
## 21. Non-Goals
|
|
||||||
|
|
||||||
Initial Narratio integration should not:
|
|
||||||
|
|
||||||
- call Scriptorium internal Go packages
|
|
||||||
- use HTTP API as the primary path
|
|
||||||
- expect Scriptorium to read S3 refs directly
|
|
||||||
- make Scriptorium responsible for Narratio stage state
|
|
||||||
- make Scriptorium responsible for notification
|
|
||||||
- require Scriptorium to understand D&D workflow semantics beyond prompt definitions
|
|
||||||
|
|
||||||
## 22. Future Extension Notes
|
|
||||||
|
|
||||||
Possible later extensions:
|
|
||||||
|
|
||||||
- HTTP API integration
|
|
||||||
- S3 artifact references if Scriptorium adds S3 reader support
|
|
||||||
- storing render diagnostics alongside generated artifacts
|
|
||||||
- token budgeting/prompt-size checks
|
|
||||||
- batch execution if Scriptorium later adds batch support
|
|
||||||
193
docs/integrations/openai-compatible-chat.md
Normal file
193
docs/integrations/openai-compatible-chat.md
Normal file
@@ -0,0 +1,193 @@
|
|||||||
|
# OpenAI-Compatible Chat Integration
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document defines the outbound LLM contract implemented by `internal/llm/openai_compatible_client.go`.
|
||||||
|
|
||||||
|
It documents only fields and behaviors currently serialized by code.
|
||||||
|
|
||||||
|
## Endpoint Construction
|
||||||
|
|
||||||
|
Request endpoint is built as:
|
||||||
|
|
||||||
|
1. choose base URL:
|
||||||
|
- `GenerateRequest.Target.Endpoint` if set
|
||||||
|
- otherwise client config `BaseURL`
|
||||||
|
2. trim trailing slash
|
||||||
|
3. append `/chat/completions`
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
- base URL: `http://localhost:8000/v1`
|
||||||
|
- final URL: `http://localhost:8000/v1/chat/completions`
|
||||||
|
|
||||||
|
## Request Fields Sent
|
||||||
|
|
||||||
|
Serialized JSON fields:
|
||||||
|
|
||||||
|
- `model` (required after fallback resolution)
|
||||||
|
- `session_id` (only when the rendered prompt includes a non-empty session ID)
|
||||||
|
- `messages` (rendered prompt messages)
|
||||||
|
- `temperature` (when non-zero, or when explicitly overridden to zero)
|
||||||
|
- `max_tokens` (when non-zero, or when explicitly overridden to zero)
|
||||||
|
- `top_p` (when non-zero, or when explicitly overridden to zero)
|
||||||
|
- `service_tier` (only when non-empty)
|
||||||
|
- `reasoning_effort` (only when non-empty)
|
||||||
|
- `response_format` (only when structured output is provided)
|
||||||
|
- profile/request `extra_params` as additional provider-specific top-level fields
|
||||||
|
|
||||||
|
`service_tier` is provider-specific. OpenRouter currently documents request values such as `flex` and `priority`; Scriptorium forwards any non-empty configured value and lets the backend validate support.
|
||||||
|
|
||||||
|
`reasoning_effort` is provider-specific. Scriptorium forwards any non-empty configured value as top-level `reasoning_effort` and lets the backend validate support.
|
||||||
|
|
||||||
|
`extra_params` are flattened into the outbound JSON object. They are not wrapped in an `extra_params` object:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"model": "gpt-4o-mini",
|
||||||
|
"messages": [
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": "rendered text"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"provider_route": "primary",
|
||||||
|
"provider_options": {
|
||||||
|
"retry_budget": 2
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`extra_params` values must be JSON-compatible. Supported value shapes include strings, numbers, booleans, objects, and arrays.
|
||||||
|
|
||||||
|
Reserved `extra_params` keys are rejected before the HTTP request is made:
|
||||||
|
|
||||||
|
- `model`
|
||||||
|
- `session_id`
|
||||||
|
- `messages`
|
||||||
|
- `temperature`
|
||||||
|
- `max_tokens`
|
||||||
|
- `top_p`
|
||||||
|
- `service_tier`
|
||||||
|
- `reasoning_effort`
|
||||||
|
- `response_format`
|
||||||
|
|
||||||
|
Empty `extra_params` keys and values that cannot be encoded as JSON are also rejected before the HTTP request is made.
|
||||||
|
|
||||||
|
`session_id` is rendered from prompt YAML using request variables and serialized as a top-level JSON request field. Scriptorium does not send an `x-session-id` header. Empty rendered session IDs are omitted, and values longer than 256 characters are rejected before the HTTP request.
|
||||||
|
|
||||||
|
Messages without prompt cache control serialize with string `content`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"role": "system",
|
||||||
|
"content": "rendered text"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Messages with prompt cache control serialize as a single text content-block array:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"role": "system",
|
||||||
|
"content": [
|
||||||
|
{
|
||||||
|
"type": "text",
|
||||||
|
"text": "rendered text",
|
||||||
|
"cache_control": {
|
||||||
|
"type": "ephemeral",
|
||||||
|
"ttl": "1h"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
When cache-control `ttl` is unset in the prompt definition, `ttl` is omitted from the outbound payload.
|
||||||
|
|
||||||
|
Structured output is currently `json_schema` only, serialized as:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"response_format": {
|
||||||
|
"type": "json_schema",
|
||||||
|
"json_schema": {
|
||||||
|
"name": "...",
|
||||||
|
"strict": true,
|
||||||
|
"schema": {"type": "object"}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Authentication Header
|
||||||
|
|
||||||
|
If `Target.APIKey` is set:
|
||||||
|
|
||||||
|
- set `Authorization: Bearer <value>`
|
||||||
|
- do not read `Target.APIKeyEnv`
|
||||||
|
|
||||||
|
If `Target.APIKey` is empty and `Target.APIKeyEnv` is set:
|
||||||
|
|
||||||
|
- resolve environment variable value at request time
|
||||||
|
- set `Authorization: Bearer <value>`
|
||||||
|
|
||||||
|
If the environment variable is unset/empty:
|
||||||
|
|
||||||
|
- request fails before HTTP call (`ErrInvalidRequest`)
|
||||||
|
|
||||||
|
If both `Target.APIKey` and `Target.APIKeyEnv` are empty:
|
||||||
|
|
||||||
|
- no `Authorization` header is sent
|
||||||
|
|
||||||
|
## Timeout Behavior
|
||||||
|
|
||||||
|
Base timeout comes from client configuration.
|
||||||
|
|
||||||
|
Per-request override:
|
||||||
|
|
||||||
|
- if `Target.TimeoutSeconds > 0`, use that value for request timeout
|
||||||
|
- if `Target.TimeoutSeconds == 0` and the value came from an explicit request override, disable the HTTP client timeout
|
||||||
|
- if `Target.TimeoutSeconds < 0`, request is rejected (`ErrInvalidRequest`)
|
||||||
|
|
||||||
|
## Response Expectations
|
||||||
|
|
||||||
|
Expected successful response shape (subset used):
|
||||||
|
|
||||||
|
- `choices[0].message.content`
|
||||||
|
- `usage.prompt_tokens`
|
||||||
|
- `usage.completion_tokens`
|
||||||
|
- `usage.total_tokens`
|
||||||
|
- `usage.prompt_tokens_details.cached_tokens` (optional)
|
||||||
|
- `usage.cache_write_tokens` (optional)
|
||||||
|
|
||||||
|
Absent cache usage fields are treated as zero. Parsed cache usage is exposed through run results and adapter response surfaces as:
|
||||||
|
|
||||||
|
- `cached_tokens`
|
||||||
|
- `cache_write_tokens`
|
||||||
|
|
||||||
|
Malformed response conditions include:
|
||||||
|
|
||||||
|
- invalid JSON
|
||||||
|
- empty `choices`
|
||||||
|
- empty `choices[0].message.content`
|
||||||
|
|
||||||
|
Malformed responses return `ErrMalformedResponse`.
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
- network/request-construction failures: `ErrRequestFailed`
|
||||||
|
- non-2xx HTTP status: `ErrUnexpectedStatus` (includes status code; provider response bodies are not included)
|
||||||
|
- malformed response shape/content: `ErrMalformedResponse`
|
||||||
|
|
||||||
|
## Unsupported Or Non-Serialized Fields
|
||||||
|
|
||||||
|
The client does not serialize top-level `cache_control`.
|
||||||
|
|
||||||
|
No built-in retries, tool-calls, or multi-request payload modes are implemented in this client.
|
||||||
|
|
||||||
|
## Relationship To Runner
|
||||||
|
|
||||||
|
When prompt validation mode is `json_schema`, runner prepares a structured-output schema spec and passes it to the client as `StructuredOutput`.
|
||||||
|
|
||||||
|
The client only serializes the provider request payload; it does not load schema files itself.
|
||||||
130
docs/integrations/subprocess.md
Normal file
130
docs/integrations/subprocess.md
Normal file
@@ -0,0 +1,130 @@
|
|||||||
|
# Subprocess Integration
|
||||||
|
|
||||||
|
This document defines the supported subprocess contract for downstream
|
||||||
|
applications invoking Scriptorium through the public CLI.
|
||||||
|
|
||||||
|
This is a CLI contract. Go callers that want an in-process typed API should use
|
||||||
|
the [package guide](../consumers/pkg-scriptorium.md).
|
||||||
|
|
||||||
|
## Supported Commands
|
||||||
|
|
||||||
|
Downstream applications should invoke:
|
||||||
|
|
||||||
|
- `scriptorium render` for preflight/debug output without LLM execution.
|
||||||
|
- `scriptorium run` for generation.
|
||||||
|
|
||||||
|
`scriptorium serve` is an HTTP service command, not the recommended subprocess
|
||||||
|
contract for per-request execution.
|
||||||
|
|
||||||
|
## Recommended Invocation Shapes
|
||||||
|
|
||||||
|
Render:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scriptorium render \
|
||||||
|
--config <config_path> \
|
||||||
|
--prompt <prompt_id> \
|
||||||
|
--input transcript=<path> \
|
||||||
|
--format json
|
||||||
|
```
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scriptorium run \
|
||||||
|
--config <config_path> \
|
||||||
|
--prompt <prompt_id> \
|
||||||
|
--input transcript=<path> \
|
||||||
|
--out <artifact_path>
|
||||||
|
```
|
||||||
|
|
||||||
|
Callers may add:
|
||||||
|
|
||||||
|
- `--profile <profile_id>`
|
||||||
|
- repeatable `--input name=path`
|
||||||
|
- repeatable `--var name=value`
|
||||||
|
- runtime overrides when explicitly needed, such as `--model`, `--llm-base-url`, `--api-key-env`, and `--timeout`
|
||||||
|
|
||||||
|
Do not pass raw API keys as command arguments.
|
||||||
|
|
||||||
|
## Config And Directory Behavior
|
||||||
|
|
||||||
|
Callers can rely on resolved app config or pass explicit paths.
|
||||||
|
|
||||||
|
Default config search order:
|
||||||
|
|
||||||
|
1. `/usr/local/etc/scriptorium/config.yml`
|
||||||
|
2. `/etc/scriptorium/config.yml`
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- Explicit `--config` requires file existence and valid syntax.
|
||||||
|
- CLI flags override config values.
|
||||||
|
- `run` and `render` require an effective `prompt_dir`.
|
||||||
|
- `profile_dir` is optional because built-in profiles are available.
|
||||||
|
|
||||||
|
## Profile Selection
|
||||||
|
|
||||||
|
Profile selection follows runner behavior:
|
||||||
|
|
||||||
|
1. explicit `--profile`
|
||||||
|
2. prompt `default_profile`
|
||||||
|
3. error if neither is available
|
||||||
|
|
||||||
|
Treat prompt and profile IDs as deployment configuration, not hardcoded business
|
||||||
|
logic.
|
||||||
|
|
||||||
|
## Input And Variable Contract
|
||||||
|
|
||||||
|
- Inputs use repeated `--input name=path`.
|
||||||
|
- Input names must match prompt definition input names.
|
||||||
|
- Variables use repeated `--var name=value`.
|
||||||
|
- Both flags also accept comma-separated mappings.
|
||||||
|
- Prefer file inputs for large content.
|
||||||
|
|
||||||
|
CLI inputs are file references. HTTP-only `inline` references are documented in
|
||||||
|
the [HTTP API reference](../api.md).
|
||||||
|
|
||||||
|
## Environment Contract
|
||||||
|
|
||||||
|
- Pass through required API-key environment variables referenced by `api_key_env`.
|
||||||
|
- Keep subprocess environments scoped to required variables.
|
||||||
|
- Use `--api-key-env` only to name an environment variable.
|
||||||
|
- Never pass raw API keys via argv.
|
||||||
|
|
||||||
|
## Stdout And Stderr
|
||||||
|
|
||||||
|
`run`:
|
||||||
|
|
||||||
|
- stdout: generated artifact body unless `--out` is used.
|
||||||
|
- stderr: success summary and errors.
|
||||||
|
|
||||||
|
`render`:
|
||||||
|
|
||||||
|
- stdout: prepared-run output unless `--out` is used.
|
||||||
|
- stderr: errors.
|
||||||
|
|
||||||
|
Capture stdout and stderr separately. Do not parse stderr as a stable data
|
||||||
|
format beyond exit status handling.
|
||||||
|
|
||||||
|
## Exit Status Contract
|
||||||
|
|
||||||
|
- `0`: success.
|
||||||
|
- `1`: parse, config, load, render, generation, IO, or runtime error.
|
||||||
|
- `2`: `run` completed and output was written, but validation failed.
|
||||||
|
|
||||||
|
A `run` exit code `2` can still produce output on stdout or at `--out`.
|
||||||
|
Consumers must decide whether to keep or discard that output.
|
||||||
|
|
||||||
|
## Security Notes
|
||||||
|
|
||||||
|
- Treat generated artifacts, rendered prompts, stdout, and stderr as potentially sensitive.
|
||||||
|
- Use controlled output paths and access controls for persisted artifacts.
|
||||||
|
- Avoid logging full rendered prompts or generated artifacts by default.
|
||||||
|
|
||||||
|
## Canonical References
|
||||||
|
|
||||||
|
- CLI behavior: [CLI reference](../cli.md)
|
||||||
|
- Config and file formats: [Configuration reference](../config.md)
|
||||||
|
- Operations: [Operations guide](../operations.md)
|
||||||
|
- Troubleshooting: [Troubleshooting](../troubleshooting.md)
|
||||||
155
docs/internal/adapters.md
Normal file
155
docs/internal/adapters.md
Normal file
@@ -0,0 +1,155 @@
|
|||||||
|
# Adapter Internals
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Adapters translate external interfaces into domain requests and translate domain results back out. They wire dependencies, apply app config, and own IO concerns, but they do not make runner decisions.
|
||||||
|
|
||||||
|
Source-loading behavior belongs in `docs/internal/sources.md`. User-facing CLI, HTTP, and package contracts belong in `docs/cli.md`, `docs/api.md`, and `docs/consumers/pkg-scriptorium.md`.
|
||||||
|
|
||||||
|
## Adapter Map
|
||||||
|
|
||||||
|
- `cmd/scriptorium`: process entrypoint.
|
||||||
|
- `internal/adapter/cli`: command parsing, config handoff, runner construction, stdout/stderr, exit codes.
|
||||||
|
- `internal/adapter/http`: `POST /v1/runs` request/response mapping and HTTP error/status mapping.
|
||||||
|
- root package `scriptorium`: public Go facade over internal runner types and dependencies.
|
||||||
|
|
||||||
|
Supporting implementation packages used during adapter wiring:
|
||||||
|
|
||||||
|
- `internal/config`
|
||||||
|
- `internal/defaults`
|
||||||
|
- `internal/format`
|
||||||
|
- `internal/llm`
|
||||||
|
- `internal/prompt`
|
||||||
|
|
||||||
|
## Inputs And Outputs
|
||||||
|
|
||||||
|
CLI adapter:
|
||||||
|
|
||||||
|
- Input: process args, optional config file, filesystem sources, environment variables.
|
||||||
|
- Output: process exit code, stdout artifact/prepared output, stderr summaries and errors.
|
||||||
|
|
||||||
|
HTTP adapter:
|
||||||
|
|
||||||
|
- Input: HTTP request method/path/headers/body for `POST /v1/runs`.
|
||||||
|
- Output: JSON success or error body with mapped status code.
|
||||||
|
|
||||||
|
Public Go facade:
|
||||||
|
|
||||||
|
- Input: typed `scriptorium.Config`, `Option`, and `RunRequest` values.
|
||||||
|
- Output: typed `PreparedRun` and `RunResult` values plus public sentinel errors.
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
- Adapters convert external shapes to `domain.RunRequest` and back.
|
||||||
|
- Runner orchestration remains in `internal/usecase`.
|
||||||
|
- Prompt/profile/schema/artifact source rules remain in repository, validator, and artifact packages.
|
||||||
|
- LLM provider request serialization remains in `internal/llm`.
|
||||||
|
- Public package types are facade types; internal domain types do not leak across the package boundary.
|
||||||
|
|
||||||
|
## Config Fields Used
|
||||||
|
|
||||||
|
Adapter app settings:
|
||||||
|
|
||||||
|
- `prompt_dir`
|
||||||
|
- `profile_dir`
|
||||||
|
- `schema_dir`
|
||||||
|
- `server.addr`
|
||||||
|
- `server.artifact_root`
|
||||||
|
- `server.max_request_bytes`
|
||||||
|
- `server.max_artifact_bytes`
|
||||||
|
- `server.max_response_bytes`
|
||||||
|
- `defaults.render_format`
|
||||||
|
|
||||||
|
Execution request/profile settings passed through the runner:
|
||||||
|
|
||||||
|
- `endpoint`
|
||||||
|
- `model`
|
||||||
|
- `temperature`
|
||||||
|
- `max_tokens`
|
||||||
|
- `top_p`
|
||||||
|
- `timeout_seconds`
|
||||||
|
- `service_tier`
|
||||||
|
- `api_key_env`
|
||||||
|
- `reasoning_effort`
|
||||||
|
- `extra_params`
|
||||||
|
|
||||||
|
CLI and HTTP preserve numeric override presence so omitted values and explicit zero values remain distinct.
|
||||||
|
|
||||||
|
## CLI Adapter
|
||||||
|
|
||||||
|
Implemented commands:
|
||||||
|
|
||||||
|
- `run`
|
||||||
|
- `render`
|
||||||
|
- `serve`
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
|
||||||
|
- `run` constructs a runner with direct filesystem artifact reading and calls `Runner.Run`.
|
||||||
|
- `render` constructs a runner and calls `Runner.Prepare`; it does not call the LLM.
|
||||||
|
- `serve` constructs a restricted artifact reader and HTTP handler, then starts an unauthenticated HTTP server.
|
||||||
|
- `run` exits `2` when generation succeeds but validation fails.
|
||||||
|
- parse, runtime, and output-write errors exit `1`.
|
||||||
|
- deprecated `--prompt-id` and `--profile-id` aliases are accepted.
|
||||||
|
|
||||||
|
## HTTP Adapter
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
|
||||||
|
- Accepts only `POST /v1/runs`.
|
||||||
|
- Decodes JSON strictly and rejects unknown fields and trailing JSON tokens.
|
||||||
|
- Rejects empty `prompt_id` and empty `inputs` before calling the runner.
|
||||||
|
- Does not accept raw API key values in the request body.
|
||||||
|
- Returns validation failures as `200` responses with failed validation details.
|
||||||
|
- Maps request-body, artifact, and encoded-response size failures to `413`.
|
||||||
|
- Maps domain and repository errors to stable error codes without returning wrapped internal cause text.
|
||||||
|
|
||||||
|
The HTTP adapter has no built-in authentication or authorization. Deployment controls must be provided outside the process.
|
||||||
|
|
||||||
|
## Public Go Facade
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
|
||||||
|
- `NewEngine` wires the same default runner components as CLI/HTTP unless options override them.
|
||||||
|
- Prompt, profile, and schema sources may come from directories, single files, or `fs.FS` roots.
|
||||||
|
- `WithProfiles` adds in-memory profiles ahead of file-backed and built-in profiles.
|
||||||
|
- `WithLLMClient` injects custom model behavior.
|
||||||
|
- `RunRequest.APIKey` is request-scoped and direct; it is used only for generation and is stripped from public results.
|
||||||
|
- internal errors are mapped to public sentinels in `errors.go`.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Adapters should:
|
||||||
|
|
||||||
|
- keep external error payloads concise and stable.
|
||||||
|
- avoid leaking raw secret values.
|
||||||
|
- use sentinels and typed errors for mapping.
|
||||||
|
- preserve strict external input decoding.
|
||||||
|
- keep validation content failures distinct from runtime errors.
|
||||||
|
|
||||||
|
CLI writes human-readable summaries to stderr. HTTP writes JSON error envelopes. The public Go facade returns typed errors.
|
||||||
|
|
||||||
|
## State And Manifests
|
||||||
|
|
||||||
|
Adapters do not add durable run state.
|
||||||
|
|
||||||
|
- No adapter writes run manifests.
|
||||||
|
- No adapter implements checkpoint, skip, or resume behavior.
|
||||||
|
- CLI output files are caller-selected artifacts, not internal state.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/adapter/cli/run_test.go`
|
||||||
|
- `internal/adapter/http/handler_test.go`
|
||||||
|
- `engine_test.go`
|
||||||
|
- `internal/format/prepared_run_test.go`
|
||||||
|
- `internal/llm/openai_compatible_client_test.go`
|
||||||
|
|
||||||
|
## Architectural Invariants
|
||||||
|
|
||||||
|
- Adapter packages stay thin and translation-focused.
|
||||||
|
- App config is resolved before dependency construction.
|
||||||
|
- External input strictness is part of contract stability.
|
||||||
|
- CLI and HTTP construct runners without a repairer.
|
||||||
|
- HTTP endpoint details remain canonical in `docs/api.md`.
|
||||||
|
- Public Go package details remain canonical in `docs/consumers/pkg-scriptorium.md`.
|
||||||
146
docs/internal/runner.md
Normal file
146
docs/internal/runner.md
Normal file
@@ -0,0 +1,146 @@
|
|||||||
|
# Runner Internals
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
`internal/usecase.Runner` is the core prompt-execution orchestrator. It prepares prompt requests, calls the configured LLM client for `Run`, validates generated output, and returns domain results.
|
||||||
|
|
||||||
|
Transport parsing, DTOs, CLI output, HTTP status mapping, and public package type conversion belong outside the runner.
|
||||||
|
|
||||||
|
## Inputs And Outputs
|
||||||
|
|
||||||
|
Primary inputs:
|
||||||
|
|
||||||
|
- `domain.RunRequest`
|
||||||
|
- repositories/readers/renderers/validators injected at construction
|
||||||
|
- `context.Context` for cancellation
|
||||||
|
|
||||||
|
Primary outputs:
|
||||||
|
|
||||||
|
- `domain.PreparedRun` from `Prepare`
|
||||||
|
- `domain.RunResult` from `Run`
|
||||||
|
- wrapped sentinel errors for adapter mapping
|
||||||
|
|
||||||
|
LLM boundary types:
|
||||||
|
|
||||||
|
- `domain.GenerateRequest`
|
||||||
|
- `domain.GenerateResponse`
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
`Runner` depends on package interfaces instead of concrete adapter types:
|
||||||
|
|
||||||
|
- `promptdef.Repository`
|
||||||
|
- `profile.Repository`
|
||||||
|
- `artifact.Reader`
|
||||||
|
- `prompt.Renderer`
|
||||||
|
- `llm.Client`
|
||||||
|
- `validate.Validator`
|
||||||
|
- optional `usecase.OutputRepairer`
|
||||||
|
|
||||||
|
The CLI, HTTP adapter, and public Go package construct these dependencies and pass them in.
|
||||||
|
|
||||||
|
## Config Fields
|
||||||
|
|
||||||
|
`Runner` does not read app config files. Effective behavior is determined by injected dependencies and the `domain.RunRequest`.
|
||||||
|
|
||||||
|
Adapter wiring commonly reflects these app config fields:
|
||||||
|
|
||||||
|
- `prompt_dir`
|
||||||
|
- `profile_dir`
|
||||||
|
- `schema_dir`
|
||||||
|
- `server.artifact_root`
|
||||||
|
- HTTP request/artifact/response size limits
|
||||||
|
|
||||||
|
Runtime model settings are resolved from the selected profile plus request overrides.
|
||||||
|
|
||||||
|
## Prepare Flow
|
||||||
|
|
||||||
|
`Prepare`:
|
||||||
|
|
||||||
|
1. requires a non-empty prompt ID.
|
||||||
|
2. loads the prompt definition and computes its hash.
|
||||||
|
3. selects the profile from request `profile_id`, then prompt `default_profile`.
|
||||||
|
4. loads the selected execution profile.
|
||||||
|
5. merges built-in execution defaults, profile values, and request overrides.
|
||||||
|
6. applies request-scoped direct API key values for public Go callers.
|
||||||
|
7. validates endpoint, model, and credential requirements.
|
||||||
|
8. resolves the output contract and JSON Schema document when required.
|
||||||
|
9. reads input artifacts.
|
||||||
|
10. renders prompt messages and hashes the rendered prompt.
|
||||||
|
11. returns a prepared run without calling the LLM.
|
||||||
|
|
||||||
|
Numeric request overrides are presence-aware: omitted values preserve the current effective value, while explicit zero values are real overrides.
|
||||||
|
|
||||||
|
## Run Flow
|
||||||
|
|
||||||
|
`Run`:
|
||||||
|
|
||||||
|
1. creates a run ID and start timestamp.
|
||||||
|
2. calls `Prepare`.
|
||||||
|
3. calls the injected LLM client with rendered messages, effective target, target presence, and structured-output settings.
|
||||||
|
4. builds the output artifact.
|
||||||
|
5. validates the output.
|
||||||
|
6. optionally attempts bounded repair when a repairer is injected and the contract permits repair.
|
||||||
|
7. returns the run result with artifact, raw output, validation, hashes, selected profile/model metadata, usage, and timing.
|
||||||
|
|
||||||
|
`Run` must reuse `Prepare`; prepare logic should not be duplicated elsewhere.
|
||||||
|
|
||||||
|
## Validation And Repair
|
||||||
|
|
||||||
|
Validation content failures are returned as successful run results with `Validation.Status == failed`. They are not runtime errors.
|
||||||
|
|
||||||
|
Validation runtime failures, such as schema load or compile errors, return `ErrValidation`.
|
||||||
|
|
||||||
|
Repair attempts occur only when all conditions are true:
|
||||||
|
|
||||||
|
- a repairer is injected
|
||||||
|
- `repair_attempts` is greater than zero
|
||||||
|
- validation status is `failed`
|
||||||
|
- validation mode is `json` or `json_schema`
|
||||||
|
|
||||||
|
CLI and HTTP wiring call `usecase.NewRunner(...)`, which does not inject a repairer. Normal CLI and HTTP execution therefore does not repair invalid output.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Stable runner sentinels include:
|
||||||
|
|
||||||
|
- `ErrInvalidRequest`
|
||||||
|
- `ErrProfileRequired`
|
||||||
|
- `ErrAPIKeyEnvMissing`
|
||||||
|
- `ErrAPIKeyRequired`
|
||||||
|
- `ErrPromptLoad`
|
||||||
|
- `ErrProfileLoad`
|
||||||
|
- `ErrArtifactLoad`
|
||||||
|
- `ErrPromptRender`
|
||||||
|
- `ErrLLMGenerate`
|
||||||
|
- `ErrValidation`
|
||||||
|
|
||||||
|
Adapters should use `errors.Is` against sentinels and lower-level repository errors instead of matching message text.
|
||||||
|
|
||||||
|
Secret values must not appear in prepared output, run results, logs, HTTP responses, or serialized public package results. The effective API-key environment-variable name may appear.
|
||||||
|
|
||||||
|
## State And Manifests
|
||||||
|
|
||||||
|
The runner is stateless across requests.
|
||||||
|
|
||||||
|
- No durable run store.
|
||||||
|
- No manifest files.
|
||||||
|
- No checkpoint, skip, or resume behavior.
|
||||||
|
- Recovery is a new request after correcting inputs, config, or environment.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/usecase/runner_test.go`
|
||||||
|
- `internal/usecase/integration_test.go`
|
||||||
|
- `engine_test.go`
|
||||||
|
- `internal/adapter/cli/run_test.go`
|
||||||
|
- `internal/adapter/http/handler_test.go`
|
||||||
|
|
||||||
|
## Architectural Invariants
|
||||||
|
|
||||||
|
- Use-case decisions stay in `internal/usecase`.
|
||||||
|
- `Run` reuses `Prepare`.
|
||||||
|
- Prompt/profile/artifact/schema loading remains behind injected boundaries.
|
||||||
|
- Validation content failures are result state; validation runtime failures are errors.
|
||||||
|
- Repair loops are bounded by `repair_attempts` and repairer presence.
|
||||||
|
- Resolved secret values are never serialized or emitted.
|
||||||
157
docs/internal/sources.md
Normal file
157
docs/internal/sources.md
Normal file
@@ -0,0 +1,157 @@
|
|||||||
|
# Source Internals
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document covers implemented prompt, profile, schema, artifact, and catalog source behavior. It is for developers changing loaders or source wiring.
|
||||||
|
|
||||||
|
Full user-facing YAML and config reference material belongs in `docs/config.md`.
|
||||||
|
|
||||||
|
## Prompt Definition Sources
|
||||||
|
|
||||||
|
`internal/promptdef` provides directory-backed and `fs.FS` repositories.
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
|
||||||
|
- recursively scans `.yaml` and `.yml` files.
|
||||||
|
- decodes YAML with known-fields checking.
|
||||||
|
- looks up prompts by YAML `id`, not by path.
|
||||||
|
- optionally filters by prompt `version`.
|
||||||
|
- rejects duplicate matching prompt IDs.
|
||||||
|
- requires `id`, `version`, and at least one message.
|
||||||
|
- requires each message to set exactly one of `content` or `content_file`.
|
||||||
|
- resolves filesystem `content_file` values relative to the prompt YAML file.
|
||||||
|
- resolves `fs.FS` `content_file` values inside the configured source root.
|
||||||
|
- permits prompt subdirectories only as organization; they are not part of prompt identity.
|
||||||
|
|
||||||
|
For `fs.FS` roots, absolute paths and relative traversal outside the source root are rejected by catalog path helpers.
|
||||||
|
|
||||||
|
## Profile Sources
|
||||||
|
|
||||||
|
`internal/profile` provides directory-backed, `fs.FS`, and overlay repositories. `internal/profile/builtin` embeds built-in profile YAML assets and exposes them through the same repository interface.
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
|
||||||
|
- recursively scans `.yaml` and `.yml` files.
|
||||||
|
- decodes YAML with known-fields checking.
|
||||||
|
- looks up profiles by YAML `id`, not by path.
|
||||||
|
- rejects duplicate IDs inside the same source.
|
||||||
|
- rejects raw `api_key` fields in YAML; file-backed profiles must use `api_key_env`.
|
||||||
|
- validates required `endpoint` and `model` values.
|
||||||
|
- validates numeric profile ranges.
|
||||||
|
|
||||||
|
Overlay behavior:
|
||||||
|
|
||||||
|
- custom profiles are primary.
|
||||||
|
- built-in profiles are fallback.
|
||||||
|
- fallback occurs only after a primary `ErrProfileNotFound`.
|
||||||
|
- primary validation, YAML, duplicate, and raw-key errors are returned directly.
|
||||||
|
- duplicate IDs across custom and built-in sources are allowed because the custom profile overrides the built-in one.
|
||||||
|
|
||||||
|
The public Go facade can add in-memory profiles ahead of file-backed and built-in profiles.
|
||||||
|
|
||||||
|
## Schema Sources
|
||||||
|
|
||||||
|
`internal/validate` provides:
|
||||||
|
|
||||||
|
- `StandardValidator` for filesystem paths.
|
||||||
|
- `FSValidator` for `fs.FS` roots and single-file public schema sources.
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
|
||||||
|
- `json_schema` validation requires a non-empty `schema_path`.
|
||||||
|
- filesystem schema paths resolve relative to `schema_dir` unless absolute.
|
||||||
|
- directory-backed schema lookup uses the explicit `schema_path`; it does not search recursively by basename.
|
||||||
|
- `fs.FS` schema paths must remain inside the configured source root.
|
||||||
|
- single-file schema sources match by the configured file base name.
|
||||||
|
- schema documents are loaded before the LLM call for structured output.
|
||||||
|
- JSON parse failures are validation content failures.
|
||||||
|
- schema access, decode, registration, and compile failures are runtime validation errors.
|
||||||
|
|
||||||
|
## Artifact Sources
|
||||||
|
|
||||||
|
`internal/artifact` supports two input artifact reference types:
|
||||||
|
|
||||||
|
- `inline`
|
||||||
|
- `file`
|
||||||
|
|
||||||
|
Inline behavior:
|
||||||
|
|
||||||
|
- requires a non-empty body.
|
||||||
|
- produces text/plain artifacts.
|
||||||
|
- hashes the body bytes.
|
||||||
|
|
||||||
|
Direct file behavior:
|
||||||
|
|
||||||
|
- used by CLI `run`, CLI `render`, and the public Go facade.
|
||||||
|
- requires a non-empty URI.
|
||||||
|
- reads from the process filesystem without HTTP artifact-root restrictions.
|
||||||
|
- infers content type from file extension, defaulting to text/plain.
|
||||||
|
|
||||||
|
Restricted file behavior:
|
||||||
|
|
||||||
|
- used by HTTP `serve`.
|
||||||
|
- allows inline artifacts even when no artifact root is configured.
|
||||||
|
- denies file artifacts when no artifact root is configured.
|
||||||
|
- resolves relative file URIs against `server.artifact_root`.
|
||||||
|
- accepts absolute file URIs only when they pass containment checks.
|
||||||
|
- applies `server.max_artifact_bytes` when configured.
|
||||||
|
|
||||||
|
Restricted containment is lexical. It cleans paths and checks the relative path against the configured root; it does not resolve symlinks. Symlinks inside the root are followed by the operating system, including symlinks that target files outside the root.
|
||||||
|
|
||||||
|
## Catalog Helpers
|
||||||
|
|
||||||
|
`internal/filecatalog` centralizes shared source helpers:
|
||||||
|
|
||||||
|
- recursive YAML discovery for filesystem and `fs.FS` roots.
|
||||||
|
- deterministic sorting.
|
||||||
|
- `.yaml` and `.yml` filtering.
|
||||||
|
- display paths for diagnostics.
|
||||||
|
- YAML file stems.
|
||||||
|
- `fs.FS` root cleaning and containment checks.
|
||||||
|
|
||||||
|
Repository code should use these helpers instead of reimplementing path traversal and containment rules.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Common source failures:
|
||||||
|
|
||||||
|
- missing prompt/profile/schema/artifact files.
|
||||||
|
- invalid YAML or JSON.
|
||||||
|
- unknown YAML fields.
|
||||||
|
- duplicate prompt or profile IDs.
|
||||||
|
- prompt/profile validation errors.
|
||||||
|
- raw API key fields in profile YAML.
|
||||||
|
- unsupported artifact reference type.
|
||||||
|
- missing inline body or file URI.
|
||||||
|
- artifact outside HTTP root.
|
||||||
|
- artifact exceeding HTTP size limit.
|
||||||
|
- schema load or compile failure.
|
||||||
|
|
||||||
|
Prompt/profile repository lookup errors are mapped by adapters separately from runtime runner errors. Validation content failures remain result state; source and schema runtime failures return errors.
|
||||||
|
|
||||||
|
## State And Manifests
|
||||||
|
|
||||||
|
Source packages do not persist run state.
|
||||||
|
|
||||||
|
- No manifests are read or written.
|
||||||
|
- No source package implements skip or resume behavior.
|
||||||
|
- Source reads reflect the current filesystem or `fs.FS` state for each request.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/promptdef/repository_test.go`
|
||||||
|
- `internal/profile/repository_test.go`
|
||||||
|
- `internal/profile/builtin/repository_test.go`
|
||||||
|
- `internal/artifact/reader_test.go`
|
||||||
|
- `internal/validate/standard_validator_test.go`
|
||||||
|
- `internal/usecase/integration_test.go`
|
||||||
|
- `engine_test.go`
|
||||||
|
|
||||||
|
## Architectural Invariants
|
||||||
|
|
||||||
|
- Prompt/profile identity comes from YAML `id`.
|
||||||
|
- External YAML decoding remains strict.
|
||||||
|
- File-backed profile YAML never accepts raw API key values.
|
||||||
|
- Built-in profiles are fallback, not a replacement for custom source validation.
|
||||||
|
- HTTP file artifacts remain rooted by lexical containment.
|
||||||
|
- Schema runtime failures remain errors, while JSON/schema content mismatches remain validation results.
|
||||||
164
docs/operations.md
Normal file
164
docs/operations.md
Normal file
@@ -0,0 +1,164 @@
|
|||||||
|
# Operations Guide
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This guide covers operating the implemented CLI commands and HTTP service. It
|
||||||
|
does not replace the [CLI reference](cli.md), [Configuration reference](config.md),
|
||||||
|
or [HTTP API reference](api.md).
|
||||||
|
|
||||||
|
## Operational Model
|
||||||
|
|
||||||
|
Scriptorium executes one prompt request per CLI invocation or HTTP request.
|
||||||
|
|
||||||
|
Important boundaries:
|
||||||
|
|
||||||
|
- No durable run state is stored.
|
||||||
|
- No manifest, archive, checkpoint, or built-in backup workflow is written.
|
||||||
|
- No built-in resume behavior exists.
|
||||||
|
- Recovery is rerun-based: correct inputs, config, or environment, then run again.
|
||||||
|
|
||||||
|
## Filesystem Layout
|
||||||
|
|
||||||
|
Operational deployments usually provide:
|
||||||
|
|
||||||
|
- `prompt_dir`: prompt definition YAML files and adjacent `content_file` templates.
|
||||||
|
- `profile_dir`: optional custom profile YAML files.
|
||||||
|
- `schema_dir`: optional JSON Schema files.
|
||||||
|
- `server.artifact_root`: optional HTTP file-input root for `serve`.
|
||||||
|
|
||||||
|
Keep these directories readable by the Scriptorium process. Keep
|
||||||
|
`server.artifact_root` narrow and not writable by untrusted users.
|
||||||
|
|
||||||
|
## Normal CLI Workflow
|
||||||
|
|
||||||
|
Use `render` before `run` when changing prompt/profile/input wiring:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium render \
|
||||||
|
--config ./examples/config.yml \
|
||||||
|
--prompt generic.markdown_summary \
|
||||||
|
--input transcript=./examples/fixtures/transcript.md \
|
||||||
|
--input glossary=./examples/fixtures/glossary.yml \
|
||||||
|
--format json
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `run` for generation after preflight:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium run \
|
||||||
|
--config ./examples/config.yml \
|
||||||
|
--prompt generic.markdown_summary \
|
||||||
|
--input transcript=./examples/fixtures/transcript.md \
|
||||||
|
--input glossary=./examples/fixtures/glossary.yml \
|
||||||
|
--out ./summary.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Before production runs, confirm:
|
||||||
|
|
||||||
|
- the effective config path is the intended one;
|
||||||
|
- prompt/profile/schema directories are readable;
|
||||||
|
- input file paths exist and match prompt input names;
|
||||||
|
- required API-key environment variables are set;
|
||||||
|
- the selected model endpoint is reachable from the process environment.
|
||||||
|
|
||||||
|
## HTTP Service Operation
|
||||||
|
|
||||||
|
Start the service with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium serve --config ./examples/config.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
The implemented HTTP route is `POST /v1/runs`; request and response fields are
|
||||||
|
defined in the [HTTP API reference](api.md).
|
||||||
|
|
||||||
|
The maintained HTTP request-shape example is `examples/http-run.json`.
|
||||||
|
|
||||||
|
HTTP service notes:
|
||||||
|
|
||||||
|
- Unknown JSON fields are rejected.
|
||||||
|
- `inline` input references work without an artifact root.
|
||||||
|
- `file` input references require `server.artifact_root` or `serve --artifact-root`.
|
||||||
|
- Request bodies, HTTP file input artifacts, and encoded JSON responses are size-limited.
|
||||||
|
- Validation content failures return `200 OK` with `validation.status: "failed"`.
|
||||||
|
|
||||||
|
Security boundary:
|
||||||
|
|
||||||
|
- `serve` has no built-in authentication or authorization.
|
||||||
|
- Put it behind trusted controls such as a private network, authenticated reverse proxy, or API gateway.
|
||||||
|
- Do not expose an artifact root containing unrelated sensitive files.
|
||||||
|
- Symlinks inside the artifact root are followed by the operating system.
|
||||||
|
|
||||||
|
## Secrets Handling
|
||||||
|
|
||||||
|
Raw API keys are not accepted in app config, profiles, CLI flags, or HTTP
|
||||||
|
request bodies.
|
||||||
|
|
||||||
|
Use this pattern:
|
||||||
|
|
||||||
|
1. Set an environment variable containing the secret value.
|
||||||
|
2. Store only the variable name in profile `api_key_env` or request override `api_key_env`.
|
||||||
|
3. Scope the process environment to the minimum required variables.
|
||||||
|
|
||||||
|
## Output, Logs, And Exit Codes
|
||||||
|
|
||||||
|
`run`:
|
||||||
|
|
||||||
|
- stdout: generated artifact body unless `--out` is used.
|
||||||
|
- stderr: summary on success, errors on failure.
|
||||||
|
- exit `2`: generation completed and output was written, but validation failed.
|
||||||
|
|
||||||
|
`render`:
|
||||||
|
|
||||||
|
- stdout: prepared-run output unless `--out` is used.
|
||||||
|
- stderr: errors.
|
||||||
|
- exit `0` on success, `1` on failure.
|
||||||
|
|
||||||
|
`serve`:
|
||||||
|
|
||||||
|
- stderr: startup and server errors.
|
||||||
|
- HTTP response body: JSON success or error envelope.
|
||||||
|
|
||||||
|
## Validation Behavior
|
||||||
|
|
||||||
|
Prompt `output.validation_mode` controls validation:
|
||||||
|
|
||||||
|
- `none`: skipped.
|
||||||
|
- `basic`: output body must not be empty.
|
||||||
|
- `json`: output body must parse as JSON.
|
||||||
|
- `json_schema`: output body must parse as JSON and satisfy the configured schema.
|
||||||
|
|
||||||
|
Runtime/schema failures are hard failures (`run` exit `1`, HTTP error).
|
||||||
|
Generated-content validation failures are soft failures (`run` exit `2`, HTTP
|
||||||
|
`200 OK` with failed validation status).
|
||||||
|
|
||||||
|
## Size Limits
|
||||||
|
|
||||||
|
Defaults are documented in [Configuration reference](config.md). Operationally:
|
||||||
|
|
||||||
|
- Keep default HTTP limits unless larger payloads are measured and expected.
|
||||||
|
- Prefer `inline` HTTP inputs for small payloads.
|
||||||
|
- Prefer `file` HTTP inputs for larger local artifacts under a controlled artifact root.
|
||||||
|
- Increase `server.max_response_bytes` when generated artifacts or requested raw output are expected to be large.
|
||||||
|
- Use `0` only when another trusted layer enforces size limits.
|
||||||
|
|
||||||
|
## Maintained Examples
|
||||||
|
|
||||||
|
- `examples/config.yml`
|
||||||
|
- `examples/config.full.yml`
|
||||||
|
- `examples/render-markdown-summary.sh`
|
||||||
|
- `examples/http-run.json`
|
||||||
|
|
||||||
|
## Safe Recovery
|
||||||
|
|
||||||
|
For failed CLI commands or HTTP requests:
|
||||||
|
|
||||||
|
1. Capture stderr or the HTTP error `code` and `message`.
|
||||||
|
2. Confirm config path and effective directory settings.
|
||||||
|
3. Verify prompt ID, profile ID, schema path, and input mappings.
|
||||||
|
4. Verify required API-key environment variables.
|
||||||
|
5. Reproduce with `render --format json` when pre-LLM resolution is uncertain.
|
||||||
|
6. Rerun after correction.
|
||||||
|
|
||||||
|
Because Scriptorium does not persist run state, rerun is the supported recovery
|
||||||
|
path.
|
||||||
118
docs/policy/architecture.md
Normal file
118
docs/policy/architecture.md
Normal file
@@ -0,0 +1,118 @@
|
|||||||
|
# Architecture
|
||||||
|
|
||||||
|
This document is the development architecture policy for Scriptorium.
|
||||||
|
|
||||||
|
It is for developers and LLM coding agents. User-facing behavior belongs in `README.md` and the docs under `docs/` that target operators/users.
|
||||||
|
|
||||||
|
## Project Shape
|
||||||
|
|
||||||
|
Scriptorium is a narrow prompt-execution application with three entry paths:
|
||||||
|
|
||||||
|
- CLI `run`
|
||||||
|
- CLI `render`
|
||||||
|
- HTTP `POST /v1/runs` through `serve`
|
||||||
|
- public Go package `gitea.maximumdirect.net/eric/scriptorium`
|
||||||
|
|
||||||
|
Domain behavior is centralized in `internal/usecase` and `internal/domain`.
|
||||||
|
|
||||||
|
## Core Principles
|
||||||
|
|
||||||
|
- Keep orchestration narrow: Scriptorium executes one prompt request; it is not a multi-step workflow engine.
|
||||||
|
- Keep adapter logic thin: adapters map external shapes to domain requests/results and should not hold domain decisions.
|
||||||
|
- Keep boundaries explicit: repositories/loaders/renderers/validators/LLM client stay behind package interfaces.
|
||||||
|
- Keep config strict: YAML/JSON decoding for external inputs should reject unknown fields.
|
||||||
|
- Keep secrets out of payloads: raw API key values must not be accepted or emitted.
|
||||||
|
|
||||||
|
## Package Boundaries
|
||||||
|
|
||||||
|
Current package map:
|
||||||
|
|
||||||
|
- root package `scriptorium`: public Go facade over engine construction, source options, request/result types, and error mapping.
|
||||||
|
- `cmd/scriptorium`: process entrypoint.
|
||||||
|
- `internal/adapter/cli`: command parsing, app wiring for CLI commands, output behavior.
|
||||||
|
- `internal/adapter/http`: HTTP DTO mapping and error/status mapping.
|
||||||
|
- `internal/config`: application settings loading and CLI override precedence.
|
||||||
|
- `internal/defaults`: compile-time default constants.
|
||||||
|
- `internal/domain`: core request/result and contract types.
|
||||||
|
- `internal/usecase`: `Runner` prepare/run orchestration and repair-hook boundary.
|
||||||
|
- `internal/promptdef`: filesystem prompt-definition repository.
|
||||||
|
- `internal/profile`: filesystem, `fs.FS`, and overlay execution-profile repositories.
|
||||||
|
- `internal/profile/builtin`: embedded built-in execution profiles.
|
||||||
|
- `internal/filecatalog`: shared YAML discovery and `fs.FS` source helpers.
|
||||||
|
- `internal/artifact`: artifact reference readers.
|
||||||
|
- `internal/prompt`: template renderer.
|
||||||
|
- `internal/llm`: provider-neutral LLM client interface and OpenAI-compatible implementation.
|
||||||
|
- `internal/validate`: validator interfaces and standard implementation.
|
||||||
|
- `internal/format`: prepared-run output formatting.
|
||||||
|
|
||||||
|
Detailed component behavior is documented in:
|
||||||
|
|
||||||
|
- `docs/internal/runner.md`
|
||||||
|
- `docs/internal/adapters.md`
|
||||||
|
- `docs/internal/sources.md`
|
||||||
|
|
||||||
|
## Configuration And Precedence
|
||||||
|
|
||||||
|
Application settings are resolved as:
|
||||||
|
|
||||||
|
1. built-in defaults
|
||||||
|
2. config file values
|
||||||
|
3. CLI overrides
|
||||||
|
|
||||||
|
`config.yml` is for application wiring (directories, server address, render default format), not prompt/profile runtime execution settings.
|
||||||
|
|
||||||
|
Profile selection and runtime model resolution remain use-case concerns.
|
||||||
|
|
||||||
|
## State And Persistence Policy
|
||||||
|
|
||||||
|
Scriptorium has no durable run-state store.
|
||||||
|
|
||||||
|
- No built-in resume/checkpoint/archive behavior.
|
||||||
|
- Recovery model is rerun after correcting inputs/config/environment.
|
||||||
|
|
||||||
|
## External Integration Policy
|
||||||
|
|
||||||
|
Current external contracts:
|
||||||
|
|
||||||
|
- inbound HTTP contract: `POST /v1/runs`, documented canonically in `docs/api.md`
|
||||||
|
- outbound model contract: OpenAI-compatible chat completions subset
|
||||||
|
- subprocess contract for integrators: CLI `run`/`render`
|
||||||
|
- public Go package contract: `docs/consumers/pkg-scriptorium.md`
|
||||||
|
|
||||||
|
Integration docs belong under `docs/integrations/`.
|
||||||
|
|
||||||
|
## Error Handling And Logging
|
||||||
|
|
||||||
|
- Wrap errors with domain/operation context.
|
||||||
|
- Map domain errors to adapter-appropriate statuses/codes without leaking sensitive internals.
|
||||||
|
- Keep stderr summaries concise for CLI success/error paths.
|
||||||
|
- Never emit raw secret values.
|
||||||
|
|
||||||
|
## Testing Expectations
|
||||||
|
|
||||||
|
- Core runner behavior should be covered with isolated unit tests and fixture-based integration tests.
|
||||||
|
- Adapter behavior should be tested for parse/mapping/error semantics.
|
||||||
|
- Config parsing, prompt/profile loading, validator behavior, and LLM client error handling should remain covered by package tests.
|
||||||
|
- Repository-level docs/examples that claim runnable behavior should be validated by tests or smoke commands.
|
||||||
|
|
||||||
|
## Documentation Expectations
|
||||||
|
|
||||||
|
- Document implemented behavior only outside `docs/roadmap/`.
|
||||||
|
- Keep canonical reference locations stable (`docs/cli.md`, `docs/config.md`, `docs/operations.md`, `docs/troubleshooting.md`, `docs/internal/`).
|
||||||
|
- Update docs in the same change when architecture-relevant behavior changes.
|
||||||
|
|
||||||
|
## Architectural Invariants
|
||||||
|
|
||||||
|
- `Runner.Run` reuses `Runner.Prepare` flow.
|
||||||
|
- CLI and HTTP currently instantiate `Runner` without a repairer.
|
||||||
|
- Artifact reading supports `inline` and `file` references.
|
||||||
|
- Unknown input fields in config/prompt/profile/http JSON should be rejected by strict decoding.
|
||||||
|
- Raw API key values must not be accepted through config/HTTP payloads.
|
||||||
|
|
||||||
|
## Non-Goals
|
||||||
|
|
||||||
|
- Do not move orchestration responsibilities from external callers into Scriptorium.
|
||||||
|
- Do not add adapter-specific business logic in `internal/adapter/*` packages.
|
||||||
|
- Do not bypass repository/renderer/validator/LLM boundaries by introducing cross-package coupling.
|
||||||
|
|
||||||
|
Work that is not implemented belongs in `docs/roadmap/`.
|
||||||
110
docs/policy/development.md
Normal file
110
docs/policy/development.md
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
# Development Guide
|
||||||
|
|
||||||
|
This document defines contributor workflow for Scriptorium.
|
||||||
|
|
||||||
|
## Repository Layout
|
||||||
|
|
||||||
|
- root package `scriptorium`: public Go facade, options, types, and error mapping.
|
||||||
|
- `cmd/scriptorium`: application entrypoint.
|
||||||
|
- `internal/domain`: core contracts.
|
||||||
|
- `internal/usecase`: runner orchestration.
|
||||||
|
- `internal/adapter/cli`: CLI adapter.
|
||||||
|
- `internal/adapter/http`: HTTP adapter.
|
||||||
|
- `internal/config`: application settings loading and precedence.
|
||||||
|
- `internal/defaults`: default constants.
|
||||||
|
- `internal/promptdef`: prompt-definition repository.
|
||||||
|
- `internal/profile`: execution-profile repository.
|
||||||
|
- `internal/profile/builtin`: embedded built-in execution profiles.
|
||||||
|
- `internal/filecatalog`: shared source discovery and path helpers.
|
||||||
|
- `internal/artifact`: artifact readers.
|
||||||
|
- `internal/prompt`: prompt rendering.
|
||||||
|
- `internal/llm`: LLM client interface and OpenAI-compatible implementation.
|
||||||
|
- `internal/validate`: validation interfaces and implementation.
|
||||||
|
- `internal/format`: prepared-run formatting.
|
||||||
|
- `docs/`: canonical documentation.
|
||||||
|
- `examples/`: copyable maintained examples and fixtures.
|
||||||
|
|
||||||
|
## Common Commands
|
||||||
|
|
||||||
|
Build:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go build ./cmd/scriptorium
|
||||||
|
```
|
||||||
|
|
||||||
|
Test:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
Targeted test runs commonly used during changes:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test .
|
||||||
|
go test ./internal/adapter/cli ./internal/adapter/http ./internal/usecase
|
||||||
|
go test ./internal/...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Coding Conventions
|
||||||
|
|
||||||
|
- Prefer small interfaces at package boundaries.
|
||||||
|
- Keep adapter packages focused on translation and IO concerns.
|
||||||
|
- Keep domain/use-case logic outside adapters.
|
||||||
|
- Wrap errors with operation context.
|
||||||
|
- Use strict decoding for user-provided YAML/JSON where applicable.
|
||||||
|
- Avoid introducing dependencies unless they materially reduce risk/complexity.
|
||||||
|
|
||||||
|
## Dependency Policy
|
||||||
|
|
||||||
|
- Prefer standard library unless an external library is clearly justified.
|
||||||
|
- Current non-stdlib dependencies are intentionally small:
|
||||||
|
- `gopkg.in/yaml.v3` for YAML decoding.
|
||||||
|
- `github.com/santhosh-tekuri/jsonschema/v6` for JSON Schema validation.
|
||||||
|
- Do not leak dependency-specific types across unrelated package boundaries.
|
||||||
|
|
||||||
|
## How To Add App Config Fields
|
||||||
|
|
||||||
|
1. Add fields in `internal/config/config.go` (`Config`, `AppSettings`, and/or `CLIOverrides` as needed).
|
||||||
|
2. Apply defaults in `BuiltInDefaults()` when required.
|
||||||
|
3. Parse and validate in `applyConfig` / `ApplyCLIOverrides`.
|
||||||
|
4. Wire the field through the consuming adapter(s).
|
||||||
|
5. Add/update config tests in `internal/config/config_test.go`.
|
||||||
|
6. Update canonical docs (`docs/config.md`, and other affected docs).
|
||||||
|
|
||||||
|
## How To Add CLI Flags
|
||||||
|
|
||||||
|
1. Add flags in `internal/adapter/cli/run.go` for the relevant command.
|
||||||
|
2. Ensure precedence behavior remains consistent with app config rules.
|
||||||
|
3. Keep `run`, `render`, and `serve` flag surfaces intentionally scoped.
|
||||||
|
4. Add/update parser and command tests in `internal/adapter/cli/run_test.go`.
|
||||||
|
5. Update `docs/cli.md` and any related docs/examples.
|
||||||
|
|
||||||
|
## How To Add Adapters Or Adapter Capabilities
|
||||||
|
|
||||||
|
1. Define or reuse the appropriate interface boundary in domain/use-case packages.
|
||||||
|
2. Implement adapter code under `internal/adapter/<name>` (or relevant boundary package).
|
||||||
|
3. Keep business decisions in `internal/usecase`.
|
||||||
|
4. Add focused adapter tests for mapping, parse, and error behavior.
|
||||||
|
5. Document the new/changed boundary in `docs/internal/adapters.md`.
|
||||||
|
6. If source-loading behavior changes, update `docs/internal/sources.md`.
|
||||||
|
7. If an external contract changes, update the canonical public or integration doc in the same change.
|
||||||
|
|
||||||
|
## How To Update Prompt/Profile/Schema Assets
|
||||||
|
|
||||||
|
1. Keep prompt/profile/schema files valid under strict loaders.
|
||||||
|
2. Keep examples secret-free.
|
||||||
|
3. Re-run tests that cover prompt/profile/validation behavior.
|
||||||
|
4. Update `docs/config.md` and any docs that reference changed contracts.
|
||||||
|
|
||||||
|
## Documentation Update Expectations
|
||||||
|
|
||||||
|
When behavior changes:
|
||||||
|
|
||||||
|
1. Update canonical doc locations, not duplicate files.
|
||||||
|
2. Keep non-roadmap docs limited to implemented behavior.
|
||||||
|
3. Update links after file moves/renames.
|
||||||
|
4. Re-run relevant tests and smoke commands.
|
||||||
|
5. For internal boundary docs, check references with `rg "docs/internal|internal/sources" docs/policy docs/internal`.
|
||||||
|
|
||||||
|
Docs work is complete only when code/tests/examples/docs agree.
|
||||||
446
docs/policy/documentation.md
Normal file
446
docs/policy/documentation.md
Normal file
@@ -0,0 +1,446 @@
|
|||||||
|
# Go Project Documentation Policy
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Project documentation must help five audiences:
|
||||||
|
|
||||||
|
1. users who need to run the application;
|
||||||
|
2. administrators/operators who need to configure and operate it;
|
||||||
|
3. developers who need to understand and change it safely;
|
||||||
|
4. LLM coding agents that need clear scope, boundaries, and invariants;
|
||||||
|
5. developers and LLM coding agents integrating this project from another codebase.
|
||||||
|
|
||||||
|
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
|
||||||
|
|
||||||
|
## Core Rules
|
||||||
|
|
||||||
|
### 1. Keep docs concise
|
||||||
|
|
||||||
|
Each document should cover a defined scope and only the essentials for that scope.
|
||||||
|
|
||||||
|
Avoid:
|
||||||
|
- long background explanations;
|
||||||
|
- repeated reference material;
|
||||||
|
- implementation detail in user-facing docs;
|
||||||
|
- aspirational language outside roadmap docs;
|
||||||
|
- verbose examples where one minimal example is clearer.
|
||||||
|
|
||||||
|
### 2. Document only implemented behavior outside roadmap files
|
||||||
|
|
||||||
|
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
|
||||||
|
|
||||||
|
- `docs/roadmap/`
|
||||||
|
|
||||||
|
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
|
||||||
|
|
||||||
|
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
|
||||||
|
|
||||||
|
### 3. Use canonical homes
|
||||||
|
|
||||||
|
Each type of information should have one canonical location.
|
||||||
|
|
||||||
|
Canonical homes:
|
||||||
|
|
||||||
|
- project purpose and quickstart: `README.md`
|
||||||
|
- development principles: `docs/policy/architecture.md`
|
||||||
|
- public HTTP API reference: `docs/api.md`
|
||||||
|
- configuration reference: `docs/config.md`
|
||||||
|
- CLI reference: `docs/cli.md`
|
||||||
|
- operations and recovery: `docs/operations.md`
|
||||||
|
- troubleshooting: `docs/troubleshooting.md`
|
||||||
|
- public API/package consumer guidance: `docs/consumers/`
|
||||||
|
- implemented internals: `docs/internal/`
|
||||||
|
- external protocol, service, and file-format contracts: `docs/integrations/`
|
||||||
|
- future work: `docs/roadmap/`
|
||||||
|
- contributor workflow: `docs/policy/development.md`
|
||||||
|
- copyable examples: `examples/`
|
||||||
|
|
||||||
|
Other files should summarize briefly and link to the canonical source.
|
||||||
|
|
||||||
|
### 4. Keep examples real
|
||||||
|
|
||||||
|
Examples should be valid, maintained, and free of secrets.
|
||||||
|
|
||||||
|
Where practical:
|
||||||
|
- example configs should load successfully;
|
||||||
|
- example commands should match real CLI syntax;
|
||||||
|
- important examples should be covered by tests.
|
||||||
|
|
||||||
|
## Documentation Profiles
|
||||||
|
|
||||||
|
All projects require:
|
||||||
|
|
||||||
|
- `README.md`
|
||||||
|
- `docs/policy/architecture.md`
|
||||||
|
|
||||||
|
Additional docs depend on the project.
|
||||||
|
|
||||||
|
### Small library
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `docs/policy/development.md`, if contributor conventions are non-obvious
|
||||||
|
|
||||||
|
### Simple CLI
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/cli.md`
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `docs/policy/development.md`
|
||||||
|
|
||||||
|
### Config-driven CLI
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/cli.md`
|
||||||
|
- `docs/config.md`
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `examples/`
|
||||||
|
- `docs/policy/development.md`
|
||||||
|
|
||||||
|
### Stateful or operator-facing application
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/cli.md`, if CLI-based
|
||||||
|
- `docs/config.md`, if config-driven
|
||||||
|
- `docs/operations.md`
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `docs/troubleshooting.md`
|
||||||
|
- `examples/`
|
||||||
|
- `docs/policy/development.md`
|
||||||
|
|
||||||
|
### Modular, service-oriented, or orchestration application
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/cli.md`, if CLI-based
|
||||||
|
- `docs/config.md`, if config-driven
|
||||||
|
- `docs/operations.md`
|
||||||
|
- `docs/internal/`
|
||||||
|
- `docs/policy/development.md`
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `docs/troubleshooting.md`
|
||||||
|
- validated examples under `examples/`
|
||||||
|
|
||||||
|
### Public HTTP API service
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/api.md`
|
||||||
|
- `docs/cli.md`, if CLI-based
|
||||||
|
- `docs/config.md`, if config-driven
|
||||||
|
- `docs/operations.md`
|
||||||
|
- `docs/internal/`
|
||||||
|
- `docs/policy/development.md`
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `docs/troubleshooting.md`
|
||||||
|
- `docs/consumers/`, for task-oriented client integration guides
|
||||||
|
- `docs/integrations/`, for upstream/downstream service contracts
|
||||||
|
- validated examples under `examples/`
|
||||||
|
|
||||||
|
### Project with public packages or consumer APIs
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/consumers/api.md`
|
||||||
|
- one `docs/consumers/pkg-<name>.md` file per public package, if public packages exist
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- copyable consumer examples under `examples/`, if practical
|
||||||
|
|
||||||
|
## Required Documents
|
||||||
|
|
||||||
|
### README.md
|
||||||
|
|
||||||
|
**Audience:** users, administrators, operators
|
||||||
|
|
||||||
|
The README is the outward-facing project orientation page.
|
||||||
|
|
||||||
|
It should include, in order:
|
||||||
|
|
||||||
|
1. concise description;
|
||||||
|
2. elevator pitch;
|
||||||
|
3. shortest useful command or usage example;
|
||||||
|
4. links to targeted docs.
|
||||||
|
|
||||||
|
The README should be short. It is not a manual.
|
||||||
|
|
||||||
|
The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.)
|
||||||
|
|
||||||
|
### docs/policy/architecture.md
|
||||||
|
|
||||||
|
**Audience:** developers, LLM coding agents
|
||||||
|
|
||||||
|
`docs/policy/architecture.md` is required for every project.
|
||||||
|
|
||||||
|
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
|
||||||
|
|
||||||
|
It should include:
|
||||||
|
|
||||||
|
- project shape;
|
||||||
|
- core design principles;
|
||||||
|
- package and boundary philosophy;
|
||||||
|
- state/persistence philosophy, if applicable;
|
||||||
|
- external integration philosophy, if applicable;
|
||||||
|
- error-handling and logging principles;
|
||||||
|
- testing expectations;
|
||||||
|
- documentation expectations;
|
||||||
|
- architectural invariants;
|
||||||
|
- explicit non-goals, if useful.
|
||||||
|
|
||||||
|
Notably, this file should prescribe a core development *policy* that should remain unchanged as the application evolves. It is not a place for details (e.g., CLI flags) that could change over time.
|
||||||
|
|
||||||
|
The contents of `architecture.md` should be trim and concise. LLMs may be directed to review it routinely via AGENTS.md, CLAUDE.md, or similar.
|
||||||
|
|
||||||
|
### docs/api.md
|
||||||
|
|
||||||
|
**Audience:** external HTTP API consumers, developers, LLM coding agents integrating by HTTP
|
||||||
|
|
||||||
|
Required for projects whose primary public interface is HTTP.
|
||||||
|
|
||||||
|
`docs/api.md` is the canonical public HTTP API contract. It should be normative for external consumers and should not be duplicated by README, operations docs, consumer guides, or integration docs.
|
||||||
|
|
||||||
|
It should include:
|
||||||
|
|
||||||
|
1. base URL conventions;
|
||||||
|
2. authentication and authorization behavior, if implemented;
|
||||||
|
3. response envelope;
|
||||||
|
4. supported media types and content negotiation behavior;
|
||||||
|
5. shared query parameters;
|
||||||
|
6. endpoint reference grouped by route family;
|
||||||
|
7. request parameters and validation rules;
|
||||||
|
8. response fields, units, nullability, and optionality;
|
||||||
|
9. error response shape and status codes;
|
||||||
|
10. pagination, caching, rate-limit, idempotency, and retry behavior, if implemented;
|
||||||
|
11. compact request and response examples.
|
||||||
|
|
||||||
|
It must document only implemented endpoints and behavior. Planned endpoints, proposed fields, future filters, and experimental response shapes belong only under `docs/roadmap/`.
|
||||||
|
|
||||||
|
For HTTP API projects, `docs/consumers/` may provide task-oriented client integration guides, but those guides should link to `docs/api.md` for the authoritative endpoint contract.
|
||||||
|
|
||||||
|
### docs/policy/development.md
|
||||||
|
|
||||||
|
**Audience:** developers, LLM coding agents
|
||||||
|
|
||||||
|
Required for projects maintained by humans and LLM coding agents.
|
||||||
|
|
||||||
|
It should include:
|
||||||
|
|
||||||
|
- repository layout;
|
||||||
|
- build/test commands;
|
||||||
|
- coding conventions;
|
||||||
|
- dependency policy;
|
||||||
|
- how to add config fields;
|
||||||
|
- how to add CLI flags;
|
||||||
|
- how to add modules or adapters, if applicable;
|
||||||
|
- how to update examples;
|
||||||
|
- documentation update expectations.
|
||||||
|
|
||||||
|
### docs/config.md
|
||||||
|
|
||||||
|
**Audience:** administrators, operators, advanced users
|
||||||
|
|
||||||
|
Required for applications with configuration files.
|
||||||
|
|
||||||
|
It should include, in order:
|
||||||
|
|
||||||
|
1. config file locations and discovery precedence;
|
||||||
|
2. minimal working config;
|
||||||
|
3. production-oriented config;
|
||||||
|
4. full configuration reference;
|
||||||
|
5. secrets handling, if applicable;
|
||||||
|
6. links to maintained examples.
|
||||||
|
|
||||||
|
The full configuration reference should be canonical.
|
||||||
|
|
||||||
|
### docs/cli.md
|
||||||
|
|
||||||
|
**Audience:** users, administrators, operators
|
||||||
|
|
||||||
|
Required for CLI applications.
|
||||||
|
|
||||||
|
It should include, in order:
|
||||||
|
|
||||||
|
1. shortest useful command;
|
||||||
|
2. command overview;
|
||||||
|
3. complete flag reference;
|
||||||
|
4. common workflows;
|
||||||
|
5. diagnostic or recovery commands, if applicable.
|
||||||
|
|
||||||
|
Explain when commands are useful, not just their syntax.
|
||||||
|
|
||||||
|
### docs/operations.md
|
||||||
|
|
||||||
|
**Audience:** administrators, operators
|
||||||
|
|
||||||
|
Required for applications that maintain state, support resume behavior, run multi-step workflows, write durable artifacts, use remote storage, or require recovery procedures.
|
||||||
|
|
||||||
|
It should cover:
|
||||||
|
|
||||||
|
- normal workflow;
|
||||||
|
- filesystem layout;
|
||||||
|
- remote storage layout, if applicable;
|
||||||
|
- logs and manifests;
|
||||||
|
- resume/retry behavior;
|
||||||
|
- cleanup behavior;
|
||||||
|
- archive/backup behavior;
|
||||||
|
- safe recovery procedures;
|
||||||
|
- operational caveats.
|
||||||
|
|
||||||
|
### docs/troubleshooting.md
|
||||||
|
|
||||||
|
**Audience:** administrators, operators
|
||||||
|
|
||||||
|
Recommended once recurring failure modes exist.
|
||||||
|
|
||||||
|
Each entry should include:
|
||||||
|
|
||||||
|
- symptom;
|
||||||
|
- likely cause;
|
||||||
|
- diagnostic command or inspection step;
|
||||||
|
- safe fix;
|
||||||
|
- relevant links.
|
||||||
|
|
||||||
|
### docs/consumers/
|
||||||
|
|
||||||
|
**Audience:** developers and LLM coding agents integrating this project from another codebase
|
||||||
|
|
||||||
|
Required for projects with public packages, SDKs, client APIs, plugin APIs, or other application-facing integration surfaces.
|
||||||
|
|
||||||
|
This directory describes how an external codebase should consume the project's public API. It should be task-oriented and copyable where useful. It is not the place for internal implementation details or operator procedures.
|
||||||
|
|
||||||
|
For projects whose public API is HTTP, `docs/consumers/` is not required, and it should not duplicate the endpoint reference in `docs/api.md`. If present, it may provide practical integration workflows, client-specific examples, or migration notes that link back to `docs/api.md`.
|
||||||
|
|
||||||
|
`docs/consumers/api.md` should provide the consumer-facing overview and primary implementation workflow. It should include:
|
||||||
|
|
||||||
|
1. intended consumer audience and use cases;
|
||||||
|
2. required inputs supplied by operators or deployment configuration;
|
||||||
|
3. recommended public package or API workflow;
|
||||||
|
4. minimal copyable example;
|
||||||
|
5. consumer responsibilities and boundaries;
|
||||||
|
6. retry, idempotency, or status behavior, if applicable;
|
||||||
|
7. links to package-specific docs and canonical integration contracts.
|
||||||
|
|
||||||
|
Package-specific docs should be named `pkg-<name>.md` and should include:
|
||||||
|
|
||||||
|
1. import path;
|
||||||
|
2. intended use cases;
|
||||||
|
3. primary types and functions needed by consumers;
|
||||||
|
4. minimal examples;
|
||||||
|
5. validation, error, retry, and boundary behavior;
|
||||||
|
6. links to canonical file-format or wire-protocol contracts.
|
||||||
|
|
||||||
|
### docs/internal/
|
||||||
|
|
||||||
|
**Audience:** developers, LLM coding agents
|
||||||
|
|
||||||
|
Required for modular, service-oriented, or orchestration projects.
|
||||||
|
|
||||||
|
This directory describes implemented internal components. It is not the roadmap.
|
||||||
|
|
||||||
|
Use one file per major component where useful.
|
||||||
|
|
||||||
|
Each component doc should include:
|
||||||
|
|
||||||
|
1. purpose;
|
||||||
|
2. inputs and outputs;
|
||||||
|
3. boundaries;
|
||||||
|
4. config fields used;
|
||||||
|
5. external adapters used;
|
||||||
|
6. state or manifest behavior, if applicable;
|
||||||
|
7. skip/resume behavior, if applicable;
|
||||||
|
8. failure behavior;
|
||||||
|
9. tests to inspect before changing;
|
||||||
|
10. architectural invariants.
|
||||||
|
|
||||||
|
### docs/roadmap/
|
||||||
|
|
||||||
|
**Audience:** maintainers, developers, LLM coding agents
|
||||||
|
|
||||||
|
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
|
||||||
|
|
||||||
|
Roadmap docs should clearly distinguish:
|
||||||
|
|
||||||
|
- proposed work;
|
||||||
|
- accepted plans;
|
||||||
|
- deferred ideas;
|
||||||
|
- rejected ideas;
|
||||||
|
- implementation prompts or task breakdowns, if useful.
|
||||||
|
|
||||||
|
Roadmap docs should not be confused with current behavior.
|
||||||
|
|
||||||
|
### docs/integrations/
|
||||||
|
|
||||||
|
**Audience:** developers, LLM coding agents
|
||||||
|
|
||||||
|
Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
|
||||||
|
|
||||||
|
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses or exposes.
|
||||||
|
|
||||||
|
For public HTTP API services, `docs/integrations/` should document upstream, downstream, storage, protocol, or runtime contracts that the service depends on or bridges. It should not become a second copy of the public HTTP endpoint reference; that belongs in `docs/api.md`.
|
||||||
|
|
||||||
|
Use one file per integration where useful.
|
||||||
|
|
||||||
|
## Examples Directory
|
||||||
|
|
||||||
|
Projects with non-trivial configuration or workflows should include `examples/`.
|
||||||
|
|
||||||
|
Useful examples include:
|
||||||
|
|
||||||
|
- minimal working config;
|
||||||
|
- production-oriented config;
|
||||||
|
- full annotated config;
|
||||||
|
- local development config;
|
||||||
|
- remote/object-storage config;
|
||||||
|
- minimal session/input file.
|
||||||
|
|
||||||
|
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
|
||||||
|
|
||||||
|
## Security and Privacy
|
||||||
|
|
||||||
|
Docs and examples must not include:
|
||||||
|
|
||||||
|
- real API keys;
|
||||||
|
- tokens;
|
||||||
|
- passwords;
|
||||||
|
- private keys;
|
||||||
|
- private environment dumps;
|
||||||
|
- sensitive user data;
|
||||||
|
- raw private transcripts;
|
||||||
|
- private infrastructure details unless intentionally public.
|
||||||
|
|
||||||
|
Document secret-handling mechanisms, not actual secret values.
|
||||||
|
|
||||||
|
## Maintenance Rules
|
||||||
|
|
||||||
|
When docs change, verify the affected behavior.
|
||||||
|
|
||||||
|
Where practical:
|
||||||
|
|
||||||
|
- load example config files in tests;
|
||||||
|
- test CLI examples or command parser behavior;
|
||||||
|
- validate documented flags against real flags;
|
||||||
|
- remove stale references;
|
||||||
|
- update links after renames;
|
||||||
|
- keep roadmap content out of non-roadmap docs.
|
||||||
|
|
||||||
|
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
|
||||||
|
|
||||||
|
Documentation is complete only when it matches the current code.
|
||||||
|
|
||||||
|
## Documentation Change Checklist
|
||||||
|
|
||||||
|
Before merging documentation changes, verify:
|
||||||
|
|
||||||
|
- README is concise and orientation-focused.
|
||||||
|
- `docs/policy/architecture.md` describes development principles.
|
||||||
|
- `docs/api.md` is the canonical HTTP contract for HTTP API services.
|
||||||
|
- Future work appears only under `docs/roadmap/`.
|
||||||
|
- User-facing docs avoid unnecessary internals.
|
||||||
|
- Consumer-facing docs explain public APIs without duplicating HTTP endpoint or integration contracts.
|
||||||
|
- Developer-facing docs preserve boundaries and invariants.
|
||||||
|
- Config examples match the schema.
|
||||||
|
- CLI examples match real commands and flags.
|
||||||
|
- Defaults appear in the canonical config reference.
|
||||||
|
- No secrets or private data are included.
|
||||||
|
- Links are accurate.
|
||||||
361
docs/troubleshooting.md
Normal file
361
docs/troubleshooting.md
Normal file
@@ -0,0 +1,361 @@
|
|||||||
|
# Troubleshooting
|
||||||
|
|
||||||
|
This guide lists common implemented failure modes and safe fixes.
|
||||||
|
|
||||||
|
Canonical references:
|
||||||
|
|
||||||
|
- [CLI reference](cli.md)
|
||||||
|
- [Configuration reference](config.md)
|
||||||
|
- [HTTP API reference](api.md)
|
||||||
|
- [Operations guide](operations.md)
|
||||||
|
|
||||||
|
## Missing Or Invalid Config
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI error includes `application config error`, `config file not found`, `invalid config YAML`, or `invalid config`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- `--config` points to a missing file.
|
||||||
|
- YAML syntax is invalid.
|
||||||
|
- Config contains unknown fields or negative HTTP size limits.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium render --config /path/to/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Correct the config path.
|
||||||
|
- Fix YAML syntax.
|
||||||
|
- Remove unknown fields.
|
||||||
|
- Keep raw secrets out of config.
|
||||||
|
|
||||||
|
Relevant links: [Configuration reference](config.md), [CLI reference](cli.md)
|
||||||
|
|
||||||
|
## Missing Prompt Directory
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI parse error says the prompt directory is required.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Neither config nor CLI flags provide an effective `prompt_dir`.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Re-run once with explicit `--prompt-dir`.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Set `prompt_dir` in config or pass `--prompt-dir`.
|
||||||
|
|
||||||
|
Relevant links: [Configuration reference](config.md), [CLI reference](cli.md)
|
||||||
|
|
||||||
|
## Unknown Flags
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI parse error for an unknown flag.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Typo.
|
||||||
|
- Flag is valid for another command.
|
||||||
|
- `serve` was given runtime model override flags.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Compare the command with the command-specific flag list.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Remove unsupported flags.
|
||||||
|
- Use `run` or `render` for runtime model overrides.
|
||||||
|
|
||||||
|
Relevant links: [CLI reference](cli.md)
|
||||||
|
|
||||||
|
## Prompt Load Failures
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI run/render fails during prompt loading.
|
||||||
|
- HTTP returns `404 prompt_not_found` or `400 prompt_load_failed`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Prompt ID/version does not exist.
|
||||||
|
- Prompt YAML is invalid or has unknown fields.
|
||||||
|
- Prompt contract is invalid, such as missing messages, invalid output mode, bad `content_file`, or missing `schema_path` for `json_schema`.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium render --config ./examples/config.yml --prompt <prompt-id> --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml --format json
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Correct prompt ID/version.
|
||||||
|
- Fix prompt YAML and referenced `content_file` paths.
|
||||||
|
- Fix output contract fields.
|
||||||
|
|
||||||
|
Relevant links: [Configuration reference](config.md), [CLI reference](cli.md)
|
||||||
|
|
||||||
|
## Profile Load Failures
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI run/render fails during profile loading.
|
||||||
|
- HTTP returns `404 profile_not_found`, `400 profile_load_failed`, or `400 profile_required`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Profile ID does not exist.
|
||||||
|
- Request omitted profile and prompt has no `default_profile`.
|
||||||
|
- Profile YAML is invalid or has unknown fields.
|
||||||
|
- Profile contains raw `api_key`.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --profile <profile-id> --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Correct profile ID or prompt `default_profile`.
|
||||||
|
- Fix profile YAML and value ranges.
|
||||||
|
- Replace raw `api_key` with `api_key_env`.
|
||||||
|
|
||||||
|
Relevant links: [Configuration reference](config.md), [CLI reference](cli.md)
|
||||||
|
|
||||||
|
## Input Artifact Failures
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI run/render fails while reading inputs.
|
||||||
|
- HTTP returns `400 artifact_read_failed`, `400 artifact_not_allowed`, or `413 artifact_too_large`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Input file path is missing or unreadable.
|
||||||
|
- HTTP input type is unsupported or missing required fields.
|
||||||
|
- HTTP file refs are disabled because no artifact root is configured.
|
||||||
|
- HTTP file path is lexically outside the artifact root.
|
||||||
|
- HTTP file input exceeds `server.max_artifact_bytes`.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Verify each input path exists and is readable by the process.
|
||||||
|
- For HTTP, verify input refs use `file` or `inline`.
|
||||||
|
- For HTTP file refs, verify the artifact root and compare file size to `server.max_artifact_bytes`.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Correct paths and permissions.
|
||||||
|
- Configure a narrow artifact root for HTTP file refs.
|
||||||
|
- Use relative paths under the artifact root or switch to `inline`.
|
||||||
|
- Increase `server.max_artifact_bytes` only for expected larger inputs.
|
||||||
|
|
||||||
|
Relevant links: [HTTP API reference](api.md), [Configuration reference](config.md)
|
||||||
|
|
||||||
|
## Missing API-Key Environment Variable
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI render/run fails with an API-key environment error.
|
||||||
|
- HTTP returns `400 api_key_env_missing`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Selected profile or runtime override sets `api_key_env`, but the environment variable is unset or empty.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
printenv SCRIPTORIUM_API_KEY
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Set the required environment variable before starting the CLI command or HTTP service.
|
||||||
|
- Or use a profile that does not require provider API-key auth.
|
||||||
|
|
||||||
|
Relevant links: [Configuration reference](config.md), [Operations guide](operations.md)
|
||||||
|
|
||||||
|
## Prompt Template Render Failures
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI render/run fails during prompt rendering.
|
||||||
|
- HTTP returns `400 prompt_render_failed`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Template references an input that was not supplied.
|
||||||
|
- Template syntax or variable reference is invalid.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Run `render --format json` with the same prompt, inputs, vars, and profile.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Align `{{input "name"}}` references with request input names.
|
||||||
|
- Fix template syntax and variable names.
|
||||||
|
|
||||||
|
Relevant links: [Configuration reference](config.md), [CLI reference](cli.md)
|
||||||
|
|
||||||
|
## LLM Request Failures
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI `run` fails during generation.
|
||||||
|
- HTTP returns `502 llm_failed`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Endpoint is unreachable.
|
||||||
|
- Provider returns non-2xx.
|
||||||
|
- Request times out.
|
||||||
|
- Provider response is malformed.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Run `render` first to confirm pre-LLM preparation works.
|
||||||
|
- Check selected endpoint/model in prepared output.
|
||||||
|
- Check network/provider logs for timeout or non-2xx details.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Correct endpoint/model/profile settings.
|
||||||
|
- Adjust timeout when appropriate.
|
||||||
|
- Resolve provider or network issue.
|
||||||
|
|
||||||
|
Relevant links: [Operations guide](operations.md), [Configuration reference](config.md)
|
||||||
|
|
||||||
|
## Validation Failed
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI `run` exits `2`.
|
||||||
|
- HTTP returns `200 OK` with `validation.status` set to `failed`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Generated output failed `basic`, `json`, or `json_schema` content validation.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Inspect validation errors in CLI stderr or the HTTP response.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Refine prompt instructions.
|
||||||
|
- Adjust schema or model/profile settings.
|
||||||
|
- Rerun after correction.
|
||||||
|
|
||||||
|
Relevant links: [Operations guide](operations.md), [HTTP API reference](api.md)
|
||||||
|
|
||||||
|
## Validation Runtime Failure
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI `run` fails with validation runtime error.
|
||||||
|
- HTTP returns `500 validation_runtime_failed`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- `json_schema` schema file is missing or unreadable.
|
||||||
|
- Schema JSON is invalid.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Verify `schema_dir` and prompt `output.schema_path`.
|
||||||
|
- Check schema file readability and JSON syntax.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Correct schema path or permissions.
|
||||||
|
- Fix schema JSON.
|
||||||
|
- Rerun.
|
||||||
|
|
||||||
|
Relevant links: [Configuration reference](config.md), [Operations guide](operations.md)
|
||||||
|
|
||||||
|
## HTTP JSON Or Request Contract Errors
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- HTTP returns `400 invalid_json` or `400 invalid_request`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- JSON body is malformed.
|
||||||
|
- Request has unknown fields or trailing JSON tokens.
|
||||||
|
- Required `prompt_id` or `inputs` is missing.
|
||||||
|
- Runtime override values are out of range.
|
||||||
|
- `extra_params` collides with reserved outbound fields.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Revalidate request JSON and compare fields with the API reference.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Send one JSON object with only supported fields.
|
||||||
|
- Include `prompt_id` and at least one input.
|
||||||
|
- Use valid model override ranges.
|
||||||
|
- Remove reserved `extra_params` keys.
|
||||||
|
|
||||||
|
Relevant links: [HTTP API reference](api.md)
|
||||||
|
|
||||||
|
## HTTP Size Limit Errors
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- HTTP returns `413 request_too_large`, `413 artifact_too_large`, or `413 response_too_large`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- JSON request body exceeds `server.max_request_bytes`.
|
||||||
|
- HTTP file input exceeds `server.max_artifact_bytes`.
|
||||||
|
- Encoded JSON response exceeds `server.max_response_bytes`.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Compare request, file input, and expected response sizes with configured limits.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Use smaller inline inputs or switch to file inputs under the artifact root.
|
||||||
|
- Reduce generated output size.
|
||||||
|
- Omit `include_raw_output`.
|
||||||
|
- Increase limits only when the deployment expects larger payloads.
|
||||||
|
|
||||||
|
Relevant links: [HTTP API reference](api.md), [Operations guide](operations.md)
|
||||||
|
|
||||||
|
## HTTP Route Or Method Errors
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- HTTP returns `404 not_found` or `405 method_not_allowed`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Path is not `/v1/runs`.
|
||||||
|
- Method on `/v1/runs` is not `POST`.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Check the request URL and method.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Send `POST /v1/runs`.
|
||||||
|
|
||||||
|
Relevant links: [HTTP API reference](api.md)
|
||||||
318
engine.go
Normal file
318
engine.go
Normal file
@@ -0,0 +1,318 @@
|
|||||||
|
package scriptorium
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io/fs"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
artifactadapter "gitea.maximumdirect.net/eric/scriptorium/internal/artifact"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/llm"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/profile/builtin"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/prompt"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/usecase"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/validate"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ErrInvalidConfig indicates invalid public engine configuration.
|
||||||
|
var ErrInvalidConfig = errors.New("invalid engine configuration")
|
||||||
|
|
||||||
|
var (
|
||||||
|
ErrInvalidRequest = errors.New("invalid run request")
|
||||||
|
ErrPromptNotFound = errors.New("prompt not found")
|
||||||
|
ErrProfileNotFound = errors.New("profile not found")
|
||||||
|
ErrPromptLoad = errors.New("failed to load prompt definition")
|
||||||
|
ErrProfileLoad = errors.New("failed to load execution profile")
|
||||||
|
ErrArtifactLoad = errors.New("failed to load artifact")
|
||||||
|
ErrPromptRender = errors.New("failed to render prompt")
|
||||||
|
ErrLLMGenerate = errors.New("failed to generate output")
|
||||||
|
ErrValidation = errors.New("failed to validate output")
|
||||||
|
)
|
||||||
|
|
||||||
|
// Engine prepares and runs Scriptorium prompt requests.
|
||||||
|
type Engine struct {
|
||||||
|
runner *usecase.Runner
|
||||||
|
}
|
||||||
|
|
||||||
|
// Config configures a public Scriptorium engine.
|
||||||
|
type Config struct {
|
||||||
|
PromptDir string
|
||||||
|
ProfileDir string
|
||||||
|
SchemaDir string
|
||||||
|
Timeout time.Duration
|
||||||
|
HTTPClient *http.Client
|
||||||
|
}
|
||||||
|
|
||||||
|
// Option customizes engine construction.
|
||||||
|
type Option interface {
|
||||||
|
apply(*engineOptions) error
|
||||||
|
}
|
||||||
|
|
||||||
|
type optionFunc func(*engineOptions) error
|
||||||
|
|
||||||
|
func (f optionFunc) apply(options *engineOptions) error {
|
||||||
|
return f(options)
|
||||||
|
}
|
||||||
|
|
||||||
|
type engineOptions struct {
|
||||||
|
llmClient llm.Client
|
||||||
|
promptDefs promptdef.Repository
|
||||||
|
profiles profile.Repository
|
||||||
|
memoryProfiles profile.Repository
|
||||||
|
validator validate.Validator
|
||||||
|
promptSource bool
|
||||||
|
profileSource bool
|
||||||
|
memorySource bool
|
||||||
|
validatorSource bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// WithLLMClient injects a custom LLM client for execution.
|
||||||
|
func WithLLMClient(client LLMClient) Option {
|
||||||
|
return optionFunc(func(options *engineOptions) error {
|
||||||
|
if client == nil {
|
||||||
|
return ErrInvalidConfig
|
||||||
|
}
|
||||||
|
options.llmClient = publicLLMClientAdapter{client: client}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// WithPromptFS loads prompt definitions from fsys under root.
|
||||||
|
//
|
||||||
|
// The source uses the same strict prompt YAML rules as configured prompt
|
||||||
|
// directories, and prompt content_file paths resolve within this source.
|
||||||
|
func WithPromptFS(fsys fs.FS, root string) Option {
|
||||||
|
return optionFunc(func(options *engineOptions) error {
|
||||||
|
if fsys == nil {
|
||||||
|
return ErrInvalidConfig
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(root) == "" {
|
||||||
|
return ErrInvalidConfig
|
||||||
|
}
|
||||||
|
options.promptDefs = promptdef.NewFSRepository(fsys, root)
|
||||||
|
options.promptSource = true
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// WithPromptFile loads prompt definitions from the single prompt file at path.
|
||||||
|
//
|
||||||
|
// Relative prompt content_file paths resolve from the file's directory.
|
||||||
|
func WithPromptFile(path string) Option {
|
||||||
|
return optionFunc(func(options *engineOptions) error {
|
||||||
|
fsys, root, err := fileSource(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
options.promptDefs = promptdef.NewFSRepository(fsys, root)
|
||||||
|
options.promptSource = true
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// WithProfileFS loads execution profiles from fsys under root.
|
||||||
|
//
|
||||||
|
// Profiles from this source overlay built-in profiles. Profile YAML must use
|
||||||
|
// api_key_env for environment-based credentials; raw API keys are rejected.
|
||||||
|
func WithProfileFS(fsys fs.FS, root string) Option {
|
||||||
|
return optionFunc(func(options *engineOptions) error {
|
||||||
|
if fsys == nil {
|
||||||
|
return ErrInvalidConfig
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(root) == "" {
|
||||||
|
return ErrInvalidConfig
|
||||||
|
}
|
||||||
|
options.profiles = profile.NewFSRepository(fsys, root)
|
||||||
|
options.profileSource = true
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// WithProfileFile loads execution profiles from the single profile file at path.
|
||||||
|
//
|
||||||
|
// The profile overlays built-in profiles. Profile YAML must use api_key_env for
|
||||||
|
// environment-based credentials; raw API keys are rejected.
|
||||||
|
func WithProfileFile(path string) Option {
|
||||||
|
return optionFunc(func(options *engineOptions) error {
|
||||||
|
fsys, root, err := fileSource(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
options.profiles = profile.NewFSRepository(fsys, root)
|
||||||
|
options.profileSource = true
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// WithProfiles configures in-memory profiles that take precedence over
|
||||||
|
// configured profile files and built-in profiles.
|
||||||
|
func WithProfiles(profiles ...Profile) Option {
|
||||||
|
return optionFunc(func(options *engineOptions) error {
|
||||||
|
repo, err := newMemoryProfileRepository(profiles)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
options.memoryProfiles = repo
|
||||||
|
options.memorySource = true
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// WithSchemaFS loads JSON Schema documents from fsys under root.
|
||||||
|
//
|
||||||
|
// Prompt schema_path values resolve within this source when schema validation
|
||||||
|
// or structured output is requested.
|
||||||
|
func WithSchemaFS(fsys fs.FS, root string) Option {
|
||||||
|
return optionFunc(func(options *engineOptions) error {
|
||||||
|
if fsys == nil {
|
||||||
|
return ErrInvalidConfig
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(root) == "" {
|
||||||
|
return ErrInvalidConfig
|
||||||
|
}
|
||||||
|
options.validator = validate.NewFSValidator(fsys, root)
|
||||||
|
options.validatorSource = true
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// WithSchemaFile loads JSON Schema documents from the single schema file at path.
|
||||||
|
//
|
||||||
|
// Prompt schema_path values refer to the file's base name.
|
||||||
|
func WithSchemaFile(path string) Option {
|
||||||
|
return optionFunc(func(options *engineOptions) error {
|
||||||
|
fsys, root, err := fileSource(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
options.validator = validate.NewFSValidator(fsys, root)
|
||||||
|
options.validatorSource = true
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewEngine constructs an Engine using the same default internal components as
|
||||||
|
// the CLI and HTTP adapters.
|
||||||
|
func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
|
||||||
|
var options engineOptions
|
||||||
|
for _, opt := range opts {
|
||||||
|
if opt == nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err := opt.apply(&options); err != nil {
|
||||||
|
return nil, fmt.Errorf("%w: %v", ErrInvalidConfig, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
promptDefs := options.promptDefs
|
||||||
|
if !options.promptSource {
|
||||||
|
if strings.TrimSpace(cfg.PromptDir) == "" {
|
||||||
|
return nil, fmt.Errorf("%w: prompt directory is required", ErrInvalidConfig)
|
||||||
|
}
|
||||||
|
promptDefs = promptdef.NewFilesystemRepository(cfg.PromptDir)
|
||||||
|
}
|
||||||
|
|
||||||
|
profiles := builtin.NewRepositoryWithDirectory(cfg.ProfileDir)
|
||||||
|
if options.profileSource {
|
||||||
|
profiles = builtin.NewRepositoryWithPrimary(options.profiles)
|
||||||
|
}
|
||||||
|
if options.memorySource {
|
||||||
|
profiles = profile.NewOverlayRepository(options.memoryProfiles, profiles)
|
||||||
|
}
|
||||||
|
|
||||||
|
validator := options.validator
|
||||||
|
if !options.validatorSource {
|
||||||
|
schemaDir := cfg.SchemaDir
|
||||||
|
if strings.TrimSpace(schemaDir) == "" {
|
||||||
|
schemaDir = defaults.SchemaDirDefault
|
||||||
|
}
|
||||||
|
validator = validate.NewStandardValidator(schemaDir)
|
||||||
|
}
|
||||||
|
|
||||||
|
llmClient := options.llmClient
|
||||||
|
if llmClient == nil {
|
||||||
|
var err error
|
||||||
|
llmClient, err = llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
|
||||||
|
Timeout: cfg.Timeout,
|
||||||
|
HTTPClient: cfg.HTTPClient,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("%w: %v", ErrInvalidConfig, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return &Engine{
|
||||||
|
runner: usecase.NewRunner(
|
||||||
|
promptDefs,
|
||||||
|
profiles,
|
||||||
|
artifactadapter.NewCompositeReader(),
|
||||||
|
prompt.NewGoRenderer(),
|
||||||
|
llmClient,
|
||||||
|
validator,
|
||||||
|
),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func fileSource(name string) (fs.FS, string, error) {
|
||||||
|
cleanName := strings.TrimSpace(name)
|
||||||
|
if cleanName == "" {
|
||||||
|
return nil, "", ErrInvalidConfig
|
||||||
|
}
|
||||||
|
dir := filepath.Dir(cleanName)
|
||||||
|
base := filepath.Base(cleanName)
|
||||||
|
if base == "." || base == string(filepath.Separator) || strings.TrimSpace(base) == "" {
|
||||||
|
return nil, "", ErrInvalidConfig
|
||||||
|
}
|
||||||
|
info, err := os.Stat(cleanName)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", fmt.Errorf("%w: failed to access source file %q: %v", ErrInvalidConfig, cleanName, err)
|
||||||
|
}
|
||||||
|
if info.IsDir() {
|
||||||
|
return nil, "", fmt.Errorf("%w: source path %q must be a file", ErrInvalidConfig, cleanName)
|
||||||
|
}
|
||||||
|
return os.DirFS(dir), filepath.ToSlash(base), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Prepare resolves a prompt request without calling an LLM.
|
||||||
|
func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, error) {
|
||||||
|
if e == nil || e.runner == nil {
|
||||||
|
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
|
||||||
|
}
|
||||||
|
|
||||||
|
domainReq, err := toDomainRunRequest(req)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
prepared, err := e.runner.Prepare(ctx, domainReq)
|
||||||
|
if err != nil {
|
||||||
|
return nil, mapPublicError(err)
|
||||||
|
}
|
||||||
|
return fromDomainPreparedRun(prepared), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run executes a prompt request and returns the generated artifact and metadata.
|
||||||
|
func (e *Engine) Run(ctx context.Context, req RunRequest) (*RunResult, error) {
|
||||||
|
if e == nil || e.runner == nil {
|
||||||
|
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
|
||||||
|
}
|
||||||
|
|
||||||
|
domainReq, err := toDomainRunRequest(req)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := e.runner.Run(ctx, domainReq)
|
||||||
|
if err != nil {
|
||||||
|
return nil, mapPublicError(err)
|
||||||
|
}
|
||||||
|
return fromDomainRunResult(result), nil
|
||||||
|
}
|
||||||
1799
engine_test.go
Normal file
1799
engine_test.go
Normal file
File diff suppressed because it is too large
Load Diff
79
errors.go
Normal file
79
errors.go
Normal file
@@ -0,0 +1,79 @@
|
|||||||
|
package scriptorium
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/usecase"
|
||||||
|
)
|
||||||
|
|
||||||
|
func mapPublicError(err error) error {
|
||||||
|
if err == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if hasPublicError(err) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
publicErr := publicErrorFor(err)
|
||||||
|
if publicErr == nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return fmt.Errorf("%w: %w", publicErr, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
func hasPublicError(err error) bool {
|
||||||
|
for _, publicErr := range []error{
|
||||||
|
ErrInvalidConfig,
|
||||||
|
ErrInvalidRequest,
|
||||||
|
ErrPromptNotFound,
|
||||||
|
ErrProfileNotFound,
|
||||||
|
ErrPromptLoad,
|
||||||
|
ErrProfileLoad,
|
||||||
|
ErrArtifactLoad,
|
||||||
|
ErrPromptRender,
|
||||||
|
ErrLLMGenerate,
|
||||||
|
ErrValidation,
|
||||||
|
} {
|
||||||
|
if errors.Is(err, publicErr) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func publicErrorFor(err error) error {
|
||||||
|
switch {
|
||||||
|
case errors.Is(err, promptdef.ErrPromptDefinitionNotFound):
|
||||||
|
return ErrPromptNotFound
|
||||||
|
case errors.Is(err, profile.ErrProfileNotFound):
|
||||||
|
return ErrProfileNotFound
|
||||||
|
case errors.Is(err, usecase.ErrPromptLoad):
|
||||||
|
return ErrPromptLoad
|
||||||
|
case errors.Is(err, usecase.ErrProfileLoad):
|
||||||
|
return ErrProfileLoad
|
||||||
|
case errors.Is(err, promptdef.ErrInvalidYAML), errors.Is(err, promptdef.ErrInvalidPromptDefinition):
|
||||||
|
return ErrPromptLoad
|
||||||
|
case isProfileLoadCause(err):
|
||||||
|
return ErrProfileLoad
|
||||||
|
case errors.Is(err, usecase.ErrArtifactLoad):
|
||||||
|
return ErrArtifactLoad
|
||||||
|
case errors.Is(err, usecase.ErrPromptRender):
|
||||||
|
return ErrPromptRender
|
||||||
|
case errors.Is(err, usecase.ErrLLMGenerate):
|
||||||
|
return ErrLLMGenerate
|
||||||
|
case errors.Is(err, usecase.ErrValidation):
|
||||||
|
return ErrValidation
|
||||||
|
case errors.Is(err, usecase.ErrInvalidRequest):
|
||||||
|
return ErrInvalidRequest
|
||||||
|
default:
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func isProfileLoadCause(err error) bool {
|
||||||
|
return errors.Is(err, profile.ErrInvalidYAML) ||
|
||||||
|
errors.Is(err, profile.ErrInvalidProfile) ||
|
||||||
|
errors.Is(err, profile.ErrRawAPIKeyNotAllowed)
|
||||||
|
}
|
||||||
13
examples/config.full.yml
Normal file
13
examples/config.full.yml
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
prompt_dir: ./examples/prompts
|
||||||
|
profile_dir: ./examples/profiles
|
||||||
|
schema_dir: ./examples/schemas
|
||||||
|
|
||||||
|
server:
|
||||||
|
addr: 127.0.0.1:8080
|
||||||
|
artifact_root: .
|
||||||
|
max_request_bytes: 16777216
|
||||||
|
max_artifact_bytes: 16777216
|
||||||
|
max_response_bytes: 16777216
|
||||||
|
|
||||||
|
defaults:
|
||||||
|
render_format: text
|
||||||
@@ -1,9 +1,10 @@
|
|||||||
prompt_dir: ./prompts
|
prompt_dir: ./examples/prompts
|
||||||
profile_dir: ./profiles
|
profile_dir: ./examples/profiles
|
||||||
schema_dir: ./schemas
|
schema_dir: ./examples/schemas
|
||||||
|
|
||||||
server:
|
server:
|
||||||
addr: :8080
|
addr: :8080
|
||||||
|
artifact_root: .
|
||||||
|
|
||||||
defaults:
|
defaults:
|
||||||
render_format: text
|
render_format: text
|
||||||
|
|||||||
50
examples/go-library/prepare/main.go
Normal file
50
examples/go-library/prepare/main.go
Normal file
@@ -0,0 +1,50 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"log"
|
||||||
|
"os"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
engine, err := scriptorium.NewEngine(scriptorium.Config{
|
||||||
|
PromptDir: "./examples/prompts",
|
||||||
|
ProfileDir: "./examples/profiles",
|
||||||
|
SchemaDir: "./examples/schemas",
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
||||||
|
PromptID: "generic.markdown_summary",
|
||||||
|
Inputs: map[string]scriptorium.ArtifactRef{
|
||||||
|
"transcript": scriptorium.File("./examples/fixtures/transcript.md"),
|
||||||
|
"glossary": scriptorium.File("./examples/fixtures/glossary.yml"),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
summary := struct {
|
||||||
|
PromptID string `json:"prompt_id"`
|
||||||
|
SelectedProfileID string `json:"selected_profile_id"`
|
||||||
|
Model string `json:"model"`
|
||||||
|
MessageCount int `json:"message_count"`
|
||||||
|
InputHashes map[string]string `json:"input_hashes"`
|
||||||
|
}{
|
||||||
|
PromptID: prepared.PromptID,
|
||||||
|
SelectedProfileID: prepared.SelectedProfileID,
|
||||||
|
Model: prepared.EffectiveModelParams.Model,
|
||||||
|
MessageCount: len(prepared.Messages),
|
||||||
|
InputHashes: prepared.InputHashes,
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := json.NewEncoder(os.Stdout).Encode(summary); err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
18
examples/http-run.json
Normal file
18
examples/http-run.json
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
{
|
||||||
|
"prompt_id": "generic.markdown_summary",
|
||||||
|
"profile_id": "local-fast",
|
||||||
|
"inputs": {
|
||||||
|
"transcript": {
|
||||||
|
"type": "file",
|
||||||
|
"uri": "./examples/fixtures/transcript.md"
|
||||||
|
},
|
||||||
|
"glossary": {
|
||||||
|
"type": "file",
|
||||||
|
"uri": "./examples/fixtures/glossary.yml"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"vars": {
|
||||||
|
"session_date": "2026-05-04"
|
||||||
|
},
|
||||||
|
"include_raw_output": false
|
||||||
|
}
|
||||||
13
examples/render-markdown-summary.sh
Executable file
13
examples/render-markdown-summary.sh
Executable file
@@ -0,0 +1,13 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
|
||||||
|
cd "$repo_root"
|
||||||
|
|
||||||
|
go run ./cmd/scriptorium render \
|
||||||
|
--config ./examples/config.yml \
|
||||||
|
--prompt generic.markdown_summary \
|
||||||
|
--input transcript=./examples/fixtures/transcript.md \
|
||||||
|
--input glossary=./examples/fixtures/glossary.yml \
|
||||||
|
--format text
|
||||||
51
formatting.go
Normal file
51
formatting.go
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
package scriptorium
|
||||||
|
|
||||||
|
import "fmt"
|
||||||
|
|
||||||
|
// String returns a concise request summary without exposing direct API keys.
|
||||||
|
func (r RunRequest) String() string {
|
||||||
|
return r.redactedString()
|
||||||
|
}
|
||||||
|
|
||||||
|
// GoString returns a concise request summary without exposing direct API keys.
|
||||||
|
func (r RunRequest) GoString() string {
|
||||||
|
return r.redactedString()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r RunRequest) redactedString() string {
|
||||||
|
return fmt.Sprintf(
|
||||||
|
"scriptorium.RunRequest{PromptID:%q PromptVersion:%q ProfileID:%q APIKeySet:%t Inputs:%d Vars:%d ExecutionSet:%t ValidationSet:%t Metadata:%d}",
|
||||||
|
r.PromptID,
|
||||||
|
r.PromptVersion,
|
||||||
|
r.ProfileID,
|
||||||
|
r.APIKey != "",
|
||||||
|
len(r.Inputs),
|
||||||
|
len(r.Vars),
|
||||||
|
r.Execution != nil,
|
||||||
|
r.Validation != nil,
|
||||||
|
len(r.Metadata),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// String returns a concise request summary without exposing direct API keys or
|
||||||
|
// rendered prompt content.
|
||||||
|
func (r GenerateRequest) String() string {
|
||||||
|
return r.redactedString()
|
||||||
|
}
|
||||||
|
|
||||||
|
// GoString returns a concise request summary without exposing direct API keys or
|
||||||
|
// rendered prompt content.
|
||||||
|
func (r GenerateRequest) GoString() string {
|
||||||
|
return r.redactedString()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r GenerateRequest) redactedString() string {
|
||||||
|
return fmt.Sprintf(
|
||||||
|
"scriptorium.GenerateRequest{Messages:%d Model:%q APIKeySet:%t StructuredOutputSet:%t ExtraParams:%d}",
|
||||||
|
len(r.Prompt.Messages),
|
||||||
|
r.Target.Model,
|
||||||
|
r.APIKey != "",
|
||||||
|
r.StructuredOutput != nil,
|
||||||
|
len(r.Target.ExtraParams),
|
||||||
|
)
|
||||||
|
}
|
||||||
@@ -19,7 +19,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/llm"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/llm"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/profile/builtin"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/prompt"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/prompt"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/usecase"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/usecase"
|
||||||
@@ -33,8 +33,7 @@ const (
|
|||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
errPromptDirRequired = "prompt directory is required; provide --prompt-dir or config.yml prompt_dir"
|
errPromptDirRequired = "prompt directory is required; provide --prompt-dir or config.yml prompt_dir"
|
||||||
errProfileDirRequired = "profile directory is required; provide --profile-dir or config.yml profile_dir"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
type runConfig struct {
|
type runConfig struct {
|
||||||
@@ -75,10 +74,26 @@ type renderConfig struct {
|
|||||||
type serveConfig struct {
|
type serveConfig struct {
|
||||||
configPath string
|
configPath string
|
||||||
|
|
||||||
addr string
|
addr string
|
||||||
promptDir string
|
promptDir string
|
||||||
profileDir string
|
profileDir string
|
||||||
schemaDir string
|
schemaDir string
|
||||||
|
artifactRoot string
|
||||||
|
maxRequestBytes int64
|
||||||
|
maxArtifactBytes int64
|
||||||
|
maxResponseBytes int64
|
||||||
|
}
|
||||||
|
|
||||||
|
type commonCommandSettings struct {
|
||||||
|
promptDir string
|
||||||
|
profileDir string
|
||||||
|
schemaDir string
|
||||||
|
serverAddr string
|
||||||
|
artifactRoot string
|
||||||
|
maxRequestBytes int64
|
||||||
|
maxArtifactBytes int64
|
||||||
|
maxResponseBytes int64
|
||||||
|
defaultRenderFormat renderformat.PreparedRunOutputFormat
|
||||||
}
|
}
|
||||||
|
|
||||||
type listFlag []string
|
type listFlag []string
|
||||||
@@ -125,22 +140,13 @@ func runCommand(args []string, stdout, stderr io.Writer) int {
|
|||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
}
|
}
|
||||||
|
|
||||||
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
|
llmClient, err := newOpenAIClient()
|
||||||
Timeout: defaults.LLMRequestTimeoutDefault,
|
|
||||||
})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintf(stderr, "llm client error: %v\n", err)
|
fmt.Fprintf(stderr, "llm client error: %v\n", err)
|
||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
}
|
}
|
||||||
|
|
||||||
runner := usecase.NewRunner(
|
runner := newRunner(cfg.promptDir, cfg.profileDir, cfg.schemaDir, llmClient)
|
||||||
promptdef.NewFilesystemRepository(cfg.promptDir),
|
|
||||||
profile.NewFilesystemRepository(cfg.profileDir),
|
|
||||||
artifactadapter.NewCompositeReader(),
|
|
||||||
prompt.NewGoRenderer(),
|
|
||||||
llmClient,
|
|
||||||
validate.NewStandardValidator(cfg.schemaDir),
|
|
||||||
)
|
|
||||||
|
|
||||||
res, runErr := runner.Run(context.Background(), req)
|
res, runErr := runner.Run(context.Background(), req)
|
||||||
if runErr != nil {
|
if runErr != nil {
|
||||||
@@ -170,14 +176,7 @@ func renderCommand(args []string, stdout, stderr io.Writer) int {
|
|||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
}
|
}
|
||||||
|
|
||||||
runner := usecase.NewRunner(
|
runner := newRunner(cfg.promptDir, cfg.profileDir, cfg.schemaDir, nil)
|
||||||
promptdef.NewFilesystemRepository(cfg.promptDir),
|
|
||||||
profile.NewFilesystemRepository(cfg.profileDir),
|
|
||||||
artifactadapter.NewCompositeReader(),
|
|
||||||
prompt.NewGoRenderer(),
|
|
||||||
nil,
|
|
||||||
validate.NewStandardValidator(cfg.schemaDir),
|
|
||||||
)
|
|
||||||
|
|
||||||
prepared, prepErr := runner.Prepare(context.Background(), req)
|
prepared, prepErr := runner.Prepare(context.Background(), req)
|
||||||
if prepErr != nil {
|
if prepErr != nil {
|
||||||
@@ -205,24 +204,24 @@ func serveCommand(args []string, stderr io.Writer) int {
|
|||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
}
|
}
|
||||||
|
|
||||||
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
|
llmClient, err := newOpenAIClient()
|
||||||
Timeout: defaults.LLMRequestTimeoutDefault,
|
|
||||||
})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintf(stderr, "llm client error: %v\n", err)
|
fmt.Fprintf(stderr, "llm client error: %v\n", err)
|
||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
}
|
}
|
||||||
|
|
||||||
runner := usecase.NewRunner(
|
artifactReader, err := artifactadapter.NewRestrictedCompositeReaderWithLimit(cfg.artifactRoot, cfg.maxArtifactBytes)
|
||||||
promptdef.NewFilesystemRepository(cfg.promptDir),
|
if err != nil {
|
||||||
profile.NewFilesystemRepository(cfg.profileDir),
|
fmt.Fprintf(stderr, "artifact root error: %v\n", err)
|
||||||
artifactadapter.NewCompositeReader(),
|
return ExitRuntimeError
|
||||||
prompt.NewGoRenderer(),
|
}
|
||||||
llmClient,
|
|
||||||
validate.NewStandardValidator(cfg.schemaDir),
|
|
||||||
)
|
|
||||||
|
|
||||||
h := httpadapter.NewHandler(runner)
|
runner := newRunnerWithArtifactReader(cfg.promptDir, cfg.profileDir, cfg.schemaDir, llmClient, artifactReader)
|
||||||
|
|
||||||
|
h := httpadapter.NewHandlerWithOptions(runner, httpadapter.HandlerOptions{
|
||||||
|
MaxRequestBytes: cfg.maxRequestBytes,
|
||||||
|
MaxResponseBytes: cfg.maxResponseBytes,
|
||||||
|
})
|
||||||
srv := &http.Server{
|
srv := &http.Server{
|
||||||
Addr: cfg.addr,
|
Addr: cfg.addr,
|
||||||
Handler: h,
|
Handler: h,
|
||||||
@@ -299,6 +298,10 @@ func parseServeArgs(args []string) (*serveConfig, error) {
|
|||||||
fs.StringVar(&cfg.promptDir, "prompt-dir", "", "directory containing prompt definition YAML files")
|
fs.StringVar(&cfg.promptDir, "prompt-dir", "", "directory containing prompt definition YAML files")
|
||||||
fs.StringVar(&cfg.profileDir, "profile-dir", "", "directory containing execution profile YAML files")
|
fs.StringVar(&cfg.profileDir, "profile-dir", "", "directory containing execution profile YAML files")
|
||||||
fs.StringVar(&cfg.schemaDir, "schema-dir", "", "base directory for validation schemas")
|
fs.StringVar(&cfg.schemaDir, "schema-dir", "", "base directory for validation schemas")
|
||||||
|
fs.StringVar(&cfg.artifactRoot, "artifact-root", "", "base directory for HTTP file input artifacts")
|
||||||
|
fs.Int64Var(&cfg.maxRequestBytes, "max-request-bytes", 0, "maximum HTTP request body bytes; 0 disables the limit")
|
||||||
|
fs.Int64Var(&cfg.maxArtifactBytes, "max-artifact-bytes", 0, "maximum HTTP file artifact bytes; 0 disables the limit")
|
||||||
|
fs.Int64Var(&cfg.maxResponseBytes, "max-response-bytes", 0, "maximum HTTP response body bytes; 0 disables the limit")
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
if err := fs.Parse(args); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@@ -307,31 +310,41 @@ func parseServeArgs(args []string) (*serveConfig, error) {
|
|||||||
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
|
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
|
||||||
}
|
}
|
||||||
|
|
||||||
settings, err := resolveAppSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
settings, err := resolveCommonSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
||||||
PromptDir: cfg.promptDirIfSet(fs),
|
PromptDir: cfg.promptDirIfSet(fs),
|
||||||
ProfileDir: cfg.profileDirIfSet(fs),
|
ProfileDir: cfg.profileDirIfSet(fs),
|
||||||
SchemaDir: cfg.schemaDirIfSet(fs),
|
SchemaDir: cfg.schemaDirIfSet(fs),
|
||||||
ServerAddr: cfg.addrIfSet(fs),
|
ServerAddr: cfg.addrIfSet(fs),
|
||||||
|
ArtifactRoot: cfg.artifactRootIfSet(fs),
|
||||||
|
MaxRequestBytes: cfg.maxRequestBytesIfSet(fs),
|
||||||
|
MaxArtifactBytes: cfg.maxArtifactBytesIfSet(fs),
|
||||||
|
MaxResponseBytes: cfg.maxResponseBytesIfSet(fs),
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
cfg.promptDir = settings.PromptDir
|
cfg.promptDir = settings.promptDir
|
||||||
cfg.profileDir = settings.ProfileDir
|
cfg.profileDir = settings.profileDir
|
||||||
cfg.schemaDir = settings.SchemaDir
|
cfg.schemaDir = settings.schemaDir
|
||||||
cfg.addr = settings.ServerAddr
|
cfg.addr = settings.serverAddr
|
||||||
|
cfg.artifactRoot = settings.artifactRoot
|
||||||
|
cfg.maxRequestBytes = settings.maxRequestBytes
|
||||||
|
cfg.maxArtifactBytes = settings.maxArtifactBytes
|
||||||
|
cfg.maxResponseBytes = settings.maxResponseBytes
|
||||||
|
|
||||||
if strings.TrimSpace(cfg.promptDir) == "" {
|
if err := validateRequiredLibraryDirs(cfg.promptDir); err != nil {
|
||||||
return nil, errors.New(errPromptDirRequired)
|
return nil, err
|
||||||
}
|
|
||||||
if strings.TrimSpace(cfg.profileDir) == "" {
|
|
||||||
return nil, errors.New(errProfileDirRequired)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
cfg.promptDir = filepath.Clean(cfg.promptDir)
|
cfg.promptDir = filepath.Clean(cfg.promptDir)
|
||||||
cfg.profileDir = filepath.Clean(cfg.profileDir)
|
if strings.TrimSpace(cfg.profileDir) != "" {
|
||||||
|
cfg.profileDir = filepath.Clean(cfg.profileDir)
|
||||||
|
}
|
||||||
cfg.schemaDir = filepath.Clean(cfg.schemaDir)
|
cfg.schemaDir = filepath.Clean(cfg.schemaDir)
|
||||||
|
if strings.TrimSpace(cfg.artifactRoot) != "" {
|
||||||
|
cfg.artifactRoot = filepath.Clean(cfg.artifactRoot)
|
||||||
|
}
|
||||||
return cfg, nil
|
return cfg, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -359,7 +372,7 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
|
|||||||
return fmt.Errorf("unexpected positional args: %v", fs.Args())
|
return fmt.Errorf("unexpected positional args: %v", fs.Args())
|
||||||
}
|
}
|
||||||
|
|
||||||
settings, err := resolveAppSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
settings, err := resolveCommonSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
||||||
PromptDir: cfg.promptDirIfSet(fs),
|
PromptDir: cfg.promptDirIfSet(fs),
|
||||||
ProfileDir: cfg.profileDirIfSet(fs),
|
ProfileDir: cfg.profileDirIfSet(fs),
|
||||||
SchemaDir: cfg.schemaDirIfSet(fs),
|
SchemaDir: cfg.schemaDirIfSet(fs),
|
||||||
@@ -368,16 +381,13 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
cfg.promptDir = settings.PromptDir
|
cfg.promptDir = settings.promptDir
|
||||||
cfg.profileDir = settings.ProfileDir
|
cfg.profileDir = settings.profileDir
|
||||||
cfg.schemaDir = settings.SchemaDir
|
cfg.schemaDir = settings.schemaDir
|
||||||
cfg.defaultRenderFormat = settings.DefaultRenderFormat
|
cfg.defaultRenderFormat = settings.defaultRenderFormat
|
||||||
|
|
||||||
if strings.TrimSpace(cfg.promptDir) == "" {
|
if err := validateRequiredLibraryDirs(cfg.promptDir); err != nil {
|
||||||
return errors.New(errPromptDirRequired)
|
return err
|
||||||
}
|
|
||||||
if strings.TrimSpace(cfg.profileDir) == "" {
|
|
||||||
return errors.New(errProfileDirRequired)
|
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(cfg.promptID) == "" {
|
if strings.TrimSpace(cfg.promptID) == "" {
|
||||||
return errors.New("--prompt is required")
|
return errors.New("--prompt is required")
|
||||||
@@ -386,7 +396,9 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
|
|||||||
return errors.New("at least one --input is required")
|
return errors.New("at least one --input is required")
|
||||||
}
|
}
|
||||||
cfg.promptDir = filepath.Clean(cfg.promptDir)
|
cfg.promptDir = filepath.Clean(cfg.promptDir)
|
||||||
cfg.profileDir = filepath.Clean(cfg.profileDir)
|
if strings.TrimSpace(cfg.profileDir) != "" {
|
||||||
|
cfg.profileDir = filepath.Clean(cfg.profileDir)
|
||||||
|
}
|
||||||
if cfg.outputPath != "" {
|
if cfg.outputPath != "" {
|
||||||
cfg.outputPath = filepath.Clean(cfg.outputPath)
|
cfg.outputPath = filepath.Clean(cfg.outputPath)
|
||||||
}
|
}
|
||||||
@@ -449,6 +461,34 @@ func (c *serveConfig) addrIfSet(fs *flag.FlagSet) string {
|
|||||||
return ""
|
return ""
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (c *serveConfig) artifactRootIfSet(fs *flag.FlagSet) string {
|
||||||
|
if flagWasSet(fs, "artifact-root") {
|
||||||
|
return c.artifactRoot
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *serveConfig) maxRequestBytesIfSet(fs *flag.FlagSet) *int64 {
|
||||||
|
if flagWasSet(fs, "max-request-bytes") {
|
||||||
|
return &c.maxRequestBytes
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *serveConfig) maxArtifactBytesIfSet(fs *flag.FlagSet) *int64 {
|
||||||
|
if flagWasSet(fs, "max-artifact-bytes") {
|
||||||
|
return &c.maxArtifactBytes
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *serveConfig) maxResponseBytesIfSet(fs *flag.FlagSet) *int64 {
|
||||||
|
if flagWasSet(fs, "max-response-bytes") {
|
||||||
|
return &c.maxResponseBytes
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func registerConfigPathFlag(fs *flag.FlagSet, target *string) {
|
func registerConfigPathFlag(fs *flag.FlagSet, target *string) {
|
||||||
fs.StringVar(
|
fs.StringVar(
|
||||||
target,
|
target,
|
||||||
@@ -476,6 +516,55 @@ func resolveAppSettings(fs *flag.FlagSet, configPath string, overrides appconfig
|
|||||||
return merged, nil
|
return merged, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func resolveCommonSettings(fs *flag.FlagSet, configPath string, overrides appconfig.CLIOverrides) (commonCommandSettings, error) {
|
||||||
|
settings, err := resolveAppSettings(fs, configPath, overrides)
|
||||||
|
if err != nil {
|
||||||
|
return commonCommandSettings{}, err
|
||||||
|
}
|
||||||
|
return commonCommandSettings{
|
||||||
|
promptDir: settings.PromptDir,
|
||||||
|
profileDir: settings.ProfileDir,
|
||||||
|
schemaDir: settings.SchemaDir,
|
||||||
|
serverAddr: settings.ServerAddr,
|
||||||
|
artifactRoot: settings.ArtifactRoot,
|
||||||
|
maxRequestBytes: settings.MaxRequestBytes,
|
||||||
|
maxArtifactBytes: settings.MaxArtifactBytes,
|
||||||
|
maxResponseBytes: settings.MaxResponseBytes,
|
||||||
|
defaultRenderFormat: settings.DefaultRenderFormat,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateRequiredLibraryDirs(promptDir string) error {
|
||||||
|
if strings.TrimSpace(promptDir) == "" {
|
||||||
|
return errors.New(errPromptDirRequired)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func newRunner(promptDir, profileDir, schemaDir string, llmClient llm.Client) *usecase.Runner {
|
||||||
|
return newRunnerWithArtifactReader(promptDir, profileDir, schemaDir, llmClient, artifactadapter.NewCompositeReader())
|
||||||
|
}
|
||||||
|
|
||||||
|
func newRunnerWithArtifactReader(promptDir, profileDir, schemaDir string, llmClient llm.Client, artifactReader artifactadapter.Reader) *usecase.Runner {
|
||||||
|
if artifactReader == nil {
|
||||||
|
artifactReader = artifactadapter.NewCompositeReader()
|
||||||
|
}
|
||||||
|
return usecase.NewRunner(
|
||||||
|
promptdef.NewFilesystemRepository(promptDir),
|
||||||
|
builtin.NewRepositoryWithDirectory(profileDir),
|
||||||
|
artifactReader,
|
||||||
|
prompt.NewGoRenderer(),
|
||||||
|
llmClient,
|
||||||
|
validate.NewStandardValidator(schemaDir),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
func newOpenAIClient() (*llm.OpenAICompatibleClient, error) {
|
||||||
|
return llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
|
||||||
|
Timeout: defaults.LLMRequestTimeoutDefault,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
func buildRunRequestFromConfig(cfg *runConfig) (domain.RunRequest, error) {
|
func buildRunRequestFromConfig(cfg *runConfig) (domain.RunRequest, error) {
|
||||||
inputMappings, err := parseMappings(cfg.inputRaw, false)
|
inputMappings, err := parseMappings(cfg.inputRaw, false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -495,18 +584,25 @@ func buildRunRequestFromConfig(cfg *runConfig) (domain.RunRequest, error) {
|
|||||||
inputs[name] = domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: path}
|
inputs[name] = domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: path}
|
||||||
}
|
}
|
||||||
|
|
||||||
var modelOverride *domain.ExecutionTarget
|
var modelOverride *domain.ExecutionTargetOverride
|
||||||
if cfg.llmBaseURLSet || cfg.modelSet || cfg.temperatureSet || cfg.maxTokensSet || cfg.topPSet || cfg.apiKeyEnvSet || cfg.timeoutSet {
|
if cfg.llmBaseURLSet || cfg.modelSet || cfg.temperatureSet || cfg.maxTokensSet || cfg.topPSet || cfg.apiKeyEnvSet || cfg.timeoutSet {
|
||||||
modelOverride = &domain.ExecutionTarget{
|
modelOverride = &domain.ExecutionTargetOverride{
|
||||||
Endpoint: cfg.llmBaseURL,
|
Endpoint: cfg.llmBaseURL,
|
||||||
Model: cfg.model,
|
Model: cfg.model,
|
||||||
Temperature: cfg.temperature,
|
APIKeyEnv: cfg.apiKeyEnv,
|
||||||
MaxTokens: cfg.maxTokens,
|
}
|
||||||
TopP: cfg.topP,
|
if cfg.temperatureSet {
|
||||||
APIKeyEnv: cfg.apiKeyEnv,
|
modelOverride.Temperature = &cfg.temperature
|
||||||
|
}
|
||||||
|
if cfg.maxTokensSet {
|
||||||
|
modelOverride.MaxTokens = &cfg.maxTokens
|
||||||
|
}
|
||||||
|
if cfg.topPSet {
|
||||||
|
modelOverride.TopP = &cfg.topP
|
||||||
}
|
}
|
||||||
if cfg.timeoutSet {
|
if cfg.timeoutSet {
|
||||||
modelOverride.TimeoutSeconds = int(cfg.timeout.Seconds())
|
timeoutSeconds := int(cfg.timeout.Seconds())
|
||||||
|
modelOverride.TimeoutSeconds = &timeoutSeconds
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -588,7 +684,7 @@ func printSummary(stderr io.Writer, res *domain.RunResult) {
|
|||||||
if res == nil {
|
if res == nil {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
fmt.Fprintf(stderr, "prompt=%s@%s selected_profile=%s model=%s validation=%s mode=%s validation_errors=%d prompt_hash=%s inputs=%d usage=%d/%d/%d\n",
|
fmt.Fprintf(stderr, "prompt=%s@%s selected_profile=%s model=%s validation=%s mode=%s validation_errors=%d prompt_hash=%s inputs=%d usage=%d/%d/%d",
|
||||||
res.PromptID,
|
res.PromptID,
|
||||||
res.PromptVersion,
|
res.PromptVersion,
|
||||||
res.SelectedProfileID,
|
res.SelectedProfileID,
|
||||||
@@ -602,11 +698,15 @@ func printSummary(stderr io.Writer, res *domain.RunResult) {
|
|||||||
res.Usage.CompletionTokens,
|
res.Usage.CompletionTokens,
|
||||||
res.Usage.TotalTokens,
|
res.Usage.TotalTokens,
|
||||||
)
|
)
|
||||||
|
if res.Usage.CachedTokens != 0 || res.Usage.CacheWriteTokens != 0 {
|
||||||
|
fmt.Fprintf(stderr, " cached_tokens=%d cache_write_tokens=%d", res.Usage.CachedTokens, res.Usage.CacheWriteTokens)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(stderr)
|
||||||
}
|
}
|
||||||
|
|
||||||
func printUsage(w io.Writer) {
|
func printUsage(w io.Writer) {
|
||||||
fmt.Fprintln(w, "usage: scriptorium <run|render|serve> ...")
|
fmt.Fprintln(w, "usage: scriptorium <run|render|serve> ...")
|
||||||
fmt.Fprintln(w, " run: scriptorium run [--config PATH] [--prompt-dir DIR] [--profile-dir DIR] --prompt ID --input name=path [--input ...] [--profile ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--var k=v] [--out path] [--timeout 10m]")
|
fmt.Fprintln(w, " run: scriptorium run [--config PATH] [--prompt-dir DIR] [--profile-dir DIR] --prompt ID --input name=path [--input ...] [--profile ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--var k=v] [--out path] [--timeout 10m]")
|
||||||
fmt.Fprintln(w, " render: scriptorium render [--config PATH] [--prompt-dir DIR] [--profile-dir DIR] --prompt ID --input name=path [--input ...] [--profile ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--var k=v] [--format text|json] [--out path] [--timeout 10m]")
|
fmt.Fprintln(w, " render: scriptorium render [--config PATH] [--prompt-dir DIR] [--profile-dir DIR] --prompt ID --input name=path [--input ...] [--profile ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--var k=v] [--format text|json] [--out path] [--timeout 10m]")
|
||||||
fmt.Fprintf(w, " serve: scriptorium serve [--config PATH] [--addr %s] [--prompt-dir DIR] [--profile-dir DIR] [--schema-dir DIR]\n", defaults.HTTPAddrDefault)
|
fmt.Fprintf(w, " serve: scriptorium serve [--config PATH] [--addr %s] [--prompt-dir DIR] [--profile-dir DIR] [--schema-dir DIR] [--artifact-root DIR] [--max-request-bytes N] [--max-artifact-bytes N] [--max-response-bytes N]\n", defaults.HTTPAddrDefault)
|
||||||
}
|
}
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -21,15 +21,16 @@ type inputRefDTO struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type modelOverrideRequestDTO struct {
|
type modelOverrideRequestDTO struct {
|
||||||
Endpoint string `json:"endpoint,omitempty"`
|
Endpoint string `json:"endpoint,omitempty"`
|
||||||
Model string `json:"model,omitempty"`
|
Model string `json:"model,omitempty"`
|
||||||
Temperature float64 `json:"temperature,omitempty"`
|
Temperature *float64 `json:"temperature,omitempty"`
|
||||||
MaxTokens int `json:"max_tokens,omitempty"`
|
MaxTokens *int `json:"max_tokens,omitempty"`
|
||||||
TopP float64 `json:"top_p,omitempty"`
|
TopP *float64 `json:"top_p,omitempty"`
|
||||||
TimeoutSeconds int `json:"timeout_seconds,omitempty"`
|
TimeoutSeconds *int `json:"timeout_seconds,omitempty"`
|
||||||
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
ServiceTier string `json:"service_tier,omitempty"`
|
||||||
APIKeyEnv string `json:"api_key_env,omitempty"`
|
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||||
ExtraParams map[string]string `json:"extra_params,omitempty"`
|
APIKeyEnv string `json:"api_key_env,omitempty"`
|
||||||
|
ExtraParams map[string]any `json:"extra_params,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type runResponseDTO struct {
|
type runResponseDTO struct {
|
||||||
@@ -69,21 +70,24 @@ type metadataDTO struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type modelParamsDTO struct {
|
type modelParamsDTO struct {
|
||||||
Endpoint string `json:"endpoint"`
|
Endpoint string `json:"endpoint"`
|
||||||
Model string `json:"model"`
|
Model string `json:"model"`
|
||||||
Temperature float64 `json:"temperature"`
|
Temperature float64 `json:"temperature"`
|
||||||
MaxTokens int `json:"max_tokens"`
|
MaxTokens int `json:"max_tokens"`
|
||||||
TopP float64 `json:"top_p"`
|
TopP float64 `json:"top_p"`
|
||||||
TimeoutSeconds int `json:"timeout_seconds"`
|
TimeoutSeconds int `json:"timeout_seconds"`
|
||||||
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
ServiceTier string `json:"service_tier,omitempty"`
|
||||||
APIKeyEnv string `json:"api_key_env,omitempty"`
|
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||||
ExtraParams map[string]string `json:"extra_params,omitempty"`
|
APIKeyEnv string `json:"api_key_env,omitempty"`
|
||||||
|
ExtraParams map[string]any `json:"extra_params,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type tokenUsageDTO struct {
|
type tokenUsageDTO struct {
|
||||||
PromptTokens int `json:"prompt_tokens"`
|
PromptTokens int `json:"prompt_tokens"`
|
||||||
CompletionTokens int `json:"completion_tokens"`
|
CompletionTokens int `json:"completion_tokens"`
|
||||||
TotalTokens int `json:"total_tokens"`
|
TotalTokens int `json:"total_tokens"`
|
||||||
|
CachedTokens int `json:"cached_tokens"`
|
||||||
|
CacheWriteTokens int `json:"cache_write_tokens"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type validationDTO struct {
|
type validationDTO struct {
|
||||||
|
|||||||
@@ -4,9 +4,12 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/artifact"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
|
||||||
@@ -18,11 +21,24 @@ type Runner interface {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type Handler struct {
|
type Handler struct {
|
||||||
runner Runner
|
runner Runner
|
||||||
|
options HandlerOptions
|
||||||
|
}
|
||||||
|
|
||||||
|
type HandlerOptions struct {
|
||||||
|
MaxRequestBytes int64
|
||||||
|
MaxResponseBytes int64
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewHandler(runner Runner) *Handler {
|
func NewHandler(runner Runner) *Handler {
|
||||||
return &Handler{runner: runner}
|
return NewHandlerWithOptions(runner, HandlerOptions{
|
||||||
|
MaxRequestBytes: defaults.HTTPMaxRequestBytesDefault,
|
||||||
|
MaxResponseBytes: defaults.HTTPMaxResponseBytesDefault,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewHandlerWithOptions(runner Runner, options HandlerOptions) *Handler {
|
||||||
|
return &Handler{runner: runner, options: options}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||||
@@ -36,9 +52,26 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
var req runRequestDTO
|
var req runRequestDTO
|
||||||
dec := json.NewDecoder(r.Body)
|
body := r.Body
|
||||||
|
if h.options.MaxRequestBytes > 0 {
|
||||||
|
body = http.MaxBytesReader(w, r.Body, h.options.MaxRequestBytes)
|
||||||
|
}
|
||||||
|
dec := json.NewDecoder(body)
|
||||||
dec.DisallowUnknownFields()
|
dec.DisallowUnknownFields()
|
||||||
if err := dec.Decode(&req); err != nil {
|
if err := dec.Decode(&req); err != nil {
|
||||||
|
if isRequestTooLarge(err) {
|
||||||
|
writeError(w, http.StatusRequestEntityTooLarge, "request_too_large", "request body is too large")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeError(w, http.StatusBadRequest, "invalid_json", "invalid JSON request body")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var trailing any
|
||||||
|
if err := dec.Decode(&trailing); err != io.EOF {
|
||||||
|
if isRequestTooLarge(err) {
|
||||||
|
writeError(w, http.StatusRequestEntityTooLarge, "request_too_large", "request body is too large")
|
||||||
|
return
|
||||||
|
}
|
||||||
writeError(w, http.StatusBadRequest, "invalid_json", "invalid JSON request body")
|
writeError(w, http.StatusBadRequest, "invalid_json", "invalid JSON request body")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -61,19 +94,9 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var model *domain.ExecutionTarget
|
var model *domain.ExecutionTargetOverride
|
||||||
if req.Model != nil {
|
if req.Model != nil {
|
||||||
model = &domain.ExecutionTarget{
|
model = executionTargetOverrideFromModelOverrideDTO(req.Model)
|
||||||
Endpoint: req.Model.Endpoint,
|
|
||||||
Model: req.Model.Model,
|
|
||||||
Temperature: req.Model.Temperature,
|
|
||||||
MaxTokens: req.Model.MaxTokens,
|
|
||||||
TopP: req.Model.TopP,
|
|
||||||
TimeoutSeconds: req.Model.TimeoutSeconds,
|
|
||||||
ReasoningEffort: req.Model.ReasoningEffort,
|
|
||||||
APIKeyEnv: req.Model.APIKeyEnv,
|
|
||||||
ExtraParams: req.Model.ExtraParams,
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
res, err := h.runner.Run(r.Context(), domain.RunRequest{
|
res, err := h.runner.Run(r.Context(), domain.RunRequest{
|
||||||
@@ -109,22 +132,14 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
SelectedProfileID: res.SelectedProfileID,
|
SelectedProfileID: res.SelectedProfileID,
|
||||||
ModelName: res.ModelName,
|
ModelName: res.ModelName,
|
||||||
Endpoint: res.Endpoint,
|
Endpoint: res.Endpoint,
|
||||||
ModelParams: modelParamsDTO{
|
ModelParams: modelParamsDTOFromExecutionTarget(res.EffectiveModelParams),
|
||||||
Endpoint: res.EffectiveModelParams.Endpoint,
|
InputHashes: res.InputHashes,
|
||||||
Model: res.EffectiveModelParams.Model,
|
|
||||||
Temperature: res.EffectiveModelParams.Temperature,
|
|
||||||
MaxTokens: res.EffectiveModelParams.MaxTokens,
|
|
||||||
TopP: res.EffectiveModelParams.TopP,
|
|
||||||
TimeoutSeconds: res.EffectiveModelParams.TimeoutSeconds,
|
|
||||||
ReasoningEffort: res.EffectiveModelParams.ReasoningEffort,
|
|
||||||
APIKeyEnv: res.EffectiveModelParams.APIKeyEnv,
|
|
||||||
ExtraParams: res.EffectiveModelParams.ExtraParams,
|
|
||||||
},
|
|
||||||
InputHashes: res.InputHashes,
|
|
||||||
Usage: tokenUsageDTO{
|
Usage: tokenUsageDTO{
|
||||||
PromptTokens: res.Usage.PromptTokens,
|
PromptTokens: res.Usage.PromptTokens,
|
||||||
CompletionTokens: res.Usage.CompletionTokens,
|
CompletionTokens: res.Usage.CompletionTokens,
|
||||||
TotalTokens: res.Usage.TotalTokens,
|
TotalTokens: res.Usage.TotalTokens,
|
||||||
|
CachedTokens: res.Usage.CachedTokens,
|
||||||
|
CacheWriteTokens: res.Usage.CacheWriteTokens,
|
||||||
},
|
},
|
||||||
StartTime: res.StartTime,
|
StartTime: res.StartTime,
|
||||||
EndTime: res.EndTime,
|
EndTime: res.EndTime,
|
||||||
@@ -138,7 +153,40 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
raw := res.RawOutput
|
raw := res.RawOutput
|
||||||
resp.RawModelOutput = &raw
|
resp.RawModelOutput = &raw
|
||||||
}
|
}
|
||||||
writeJSON(w, http.StatusOK, resp)
|
writeLimitedJSON(w, http.StatusOK, resp, h.options.MaxResponseBytes)
|
||||||
|
}
|
||||||
|
|
||||||
|
func executionTargetOverrideFromModelOverrideDTO(dto *modelOverrideRequestDTO) *domain.ExecutionTargetOverride {
|
||||||
|
if dto == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return &domain.ExecutionTargetOverride{
|
||||||
|
Endpoint: dto.Endpoint,
|
||||||
|
Model: dto.Model,
|
||||||
|
Temperature: dto.Temperature,
|
||||||
|
MaxTokens: dto.MaxTokens,
|
||||||
|
TopP: dto.TopP,
|
||||||
|
TimeoutSeconds: dto.TimeoutSeconds,
|
||||||
|
ServiceTier: dto.ServiceTier,
|
||||||
|
ReasoningEffort: dto.ReasoningEffort,
|
||||||
|
APIKeyEnv: dto.APIKeyEnv,
|
||||||
|
ExtraParams: dto.ExtraParams,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func modelParamsDTOFromExecutionTarget(target domain.ExecutionTarget) modelParamsDTO {
|
||||||
|
return modelParamsDTO{
|
||||||
|
Endpoint: target.Endpoint,
|
||||||
|
Model: target.Model,
|
||||||
|
Temperature: target.Temperature,
|
||||||
|
MaxTokens: target.MaxTokens,
|
||||||
|
TopP: target.TopP,
|
||||||
|
TimeoutSeconds: target.TimeoutSeconds,
|
||||||
|
ServiceTier: target.ServiceTier,
|
||||||
|
ReasoningEffort: target.ReasoningEffort,
|
||||||
|
APIKeyEnv: target.APIKeyEnv,
|
||||||
|
ExtraParams: target.ExtraParams,
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func mapValidation(v domain.ValidationResult) validationDTO {
|
func mapValidation(v domain.ValidationResult) validationDTO {
|
||||||
@@ -160,16 +208,22 @@ func mapRunError(err error) (int, string, string) {
|
|||||||
return http.StatusNotFound, "profile_not_found", "execution profile not found"
|
return http.StatusNotFound, "profile_not_found", "execution profile not found"
|
||||||
case errors.Is(err, promptdef.ErrInvalidYAML), errors.Is(err, promptdef.ErrInvalidPromptDefinition):
|
case errors.Is(err, promptdef.ErrInvalidYAML), errors.Is(err, promptdef.ErrInvalidPromptDefinition):
|
||||||
return http.StatusBadRequest, "prompt_load_failed", "failed to load prompt definition"
|
return http.StatusBadRequest, "prompt_load_failed", "failed to load prompt definition"
|
||||||
case errors.Is(err, profile.ErrInvalidYAML), errors.Is(err, profile.ErrInvalidProfile):
|
case errors.Is(err, profile.ErrInvalidYAML), errors.Is(err, profile.ErrInvalidProfile), errors.Is(err, profile.ErrRawAPIKeyNotAllowed):
|
||||||
return http.StatusBadRequest, "profile_load_failed", "failed to load execution profile"
|
return http.StatusBadRequest, "profile_load_failed", "failed to load execution profile"
|
||||||
case errors.Is(err, usecase.ErrInvalidRequest) && strings.Contains(err.Error(), "profile id is required either in request or prompt default_profile"):
|
case errors.Is(err, usecase.ErrProfileRequired):
|
||||||
return http.StatusBadRequest, "profile_required", "profile_id is required when prompt default_profile is not set"
|
return http.StatusBadRequest, "profile_required", "profile_id is required when prompt default_profile is not set"
|
||||||
case errors.Is(err, usecase.ErrInvalidRequest) && strings.Contains(err.Error(), "api key environment variable"):
|
case errors.Is(err, usecase.ErrAPIKeyEnvMissing):
|
||||||
return http.StatusBadRequest, "api_key_env_missing", "api_key_env is set but the environment variable is missing"
|
return http.StatusBadRequest, "api_key_env_missing", "api_key_env is set but the environment variable is missing"
|
||||||
case errors.Is(err, usecase.ErrInvalidRequest):
|
case errors.Is(err, usecase.ErrInvalidRequest):
|
||||||
return http.StatusBadRequest, "invalid_request", "invalid run request"
|
return http.StatusBadRequest, "invalid_request", "invalid run request"
|
||||||
case errors.Is(err, usecase.ErrProfileLoad):
|
case errors.Is(err, usecase.ErrPromptLoad):
|
||||||
return http.StatusBadRequest, "prompt_load_failed", "failed to load prompt definition"
|
return http.StatusBadRequest, "prompt_load_failed", "failed to load prompt definition"
|
||||||
|
case errors.Is(err, usecase.ErrProfileLoad):
|
||||||
|
return http.StatusBadRequest, "profile_load_failed", "failed to load execution profile"
|
||||||
|
case errors.Is(err, artifact.ErrFileNotAllowed), errors.Is(err, artifact.ErrFileOutsideRoot):
|
||||||
|
return http.StatusBadRequest, "artifact_not_allowed", "file input artifact is not allowed"
|
||||||
|
case errors.Is(err, artifact.ErrFileTooLarge):
|
||||||
|
return http.StatusRequestEntityTooLarge, "artifact_too_large", "file input artifact is too large"
|
||||||
case errors.Is(err, usecase.ErrArtifactLoad):
|
case errors.Is(err, usecase.ErrArtifactLoad):
|
||||||
return http.StatusBadRequest, "artifact_read_failed", "failed to read input artifact"
|
return http.StatusBadRequest, "artifact_read_failed", "failed to read input artifact"
|
||||||
case errors.Is(err, usecase.ErrPromptRender):
|
case errors.Is(err, usecase.ErrPromptRender):
|
||||||
@@ -184,9 +238,23 @@ func mapRunError(err error) (int, string, string) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func writeJSON(w http.ResponseWriter, status int, v any) {
|
func writeJSON(w http.ResponseWriter, status int, v any) {
|
||||||
|
writeLimitedJSON(w, status, v, 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeLimitedJSON(w http.ResponseWriter, status int, v any, maxBytes int64) {
|
||||||
|
data, err := json.Marshal(v)
|
||||||
|
if err != nil {
|
||||||
|
writeError(w, http.StatusInternalServerError, "internal_error", "internal server error")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
data = append(data, '\n')
|
||||||
|
if maxBytes > 0 && int64(len(data)) > maxBytes {
|
||||||
|
writeError(w, http.StatusRequestEntityTooLarge, "response_too_large", "response body is too large")
|
||||||
|
return
|
||||||
|
}
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
w.WriteHeader(status)
|
w.WriteHeader(status)
|
||||||
_ = json.NewEncoder(w).Encode(v)
|
_, _ = w.Write(data)
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeError(w http.ResponseWriter, status int, code, message string) {
|
func writeError(w http.ResponseWriter, status int, code, message string) {
|
||||||
@@ -197,3 +265,8 @@ func writeError(w http.ResponseWriter, status int, code, message string) {
|
|||||||
},
|
},
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func isRequestTooLarge(err error) bool {
|
||||||
|
var maxBytesErr *http.MaxBytesError
|
||||||
|
return errors.As(err, &maxBytesErr)
|
||||||
|
}
|
||||||
|
|||||||
@@ -4,15 +4,19 @@ import (
|
|||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
|
||||||
"fmt"
|
"fmt"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/artifact"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/llm"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/usecase"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/usecase"
|
||||||
@@ -32,6 +36,40 @@ func (f *fakeRunner) Run(ctx context.Context, req domain.RunRequest) (*domain.Ru
|
|||||||
return f.result, nil
|
return f.result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type handlerPromptRepo struct {
|
||||||
|
def *domain.PromptDefinition
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r handlerPromptRepo) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error) {
|
||||||
|
return r.def, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type handlerProfileRepo struct {
|
||||||
|
profile *domain.ExecutionProfile
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r handlerProfileRepo) GetProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) {
|
||||||
|
return r.profile, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type handlerArtifactReader struct{}
|
||||||
|
|
||||||
|
func (handlerArtifactReader) Read(ctx context.Context, ref domain.ArtifactRef) (*domain.Artifact, error) {
|
||||||
|
return &domain.Artifact{Name: "input", Body: []byte("input"), Hash: "hash"}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type handlerRenderer struct{}
|
||||||
|
|
||||||
|
func (handlerRenderer) Render(ctx context.Context, definition *domain.PromptDefinition, inputs map[string]*domain.Artifact, vars map[string]string) (*domain.RenderedPrompt, error) {
|
||||||
|
return &domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type handlerLLMClient struct{}
|
||||||
|
|
||||||
|
func (handlerLLMClient) Generate(ctx context.Context, req domain.GenerateRequest) (*domain.GenerateResponse, error) {
|
||||||
|
return &domain.GenerateResponse{Content: "ok"}, nil
|
||||||
|
}
|
||||||
|
|
||||||
func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||||
start := time.Now().UTC()
|
start := time.Now().UTC()
|
||||||
end := start.Add(2 * time.Second)
|
end := start.Add(2 * time.Second)
|
||||||
@@ -62,14 +100,21 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
|||||||
MaxTokens: 42,
|
MaxTokens: 42,
|
||||||
TopP: 0.9,
|
TopP: 0.9,
|
||||||
TimeoutSeconds: 120,
|
TimeoutSeconds: 120,
|
||||||
|
ServiceTier: "priority",
|
||||||
APIKeyEnv: envName,
|
APIKeyEnv: envName,
|
||||||
},
|
},
|
||||||
InputHashes: map[string]string{"transcript": "h1"},
|
InputHashes: map[string]string{"transcript": "h1"},
|
||||||
Usage: domain.TokenUsage{PromptTokens: 1, CompletionTokens: 2, TotalTokens: 3},
|
Usage: domain.TokenUsage{
|
||||||
StartTime: start,
|
PromptTokens: 1,
|
||||||
EndTime: end,
|
CompletionTokens: 2,
|
||||||
Duration: 2 * time.Second,
|
TotalTokens: 3,
|
||||||
RawOutput: "hello",
|
CachedTokens: 4,
|
||||||
|
CacheWriteTokens: 5,
|
||||||
|
},
|
||||||
|
StartTime: start,
|
||||||
|
EndTime: end,
|
||||||
|
Duration: 2 * time.Second,
|
||||||
|
RawOutput: "hello",
|
||||||
}}
|
}}
|
||||||
|
|
||||||
h := NewHandler(r)
|
h := NewHandler(r)
|
||||||
@@ -81,7 +126,7 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
|||||||
"transcript": {"type": "file", "uri": "./t.md"}
|
"transcript": {"type": "file", "uri": "./t.md"}
|
||||||
},
|
},
|
||||||
"vars": {"k": "v"},
|
"vars": {"k": "v"},
|
||||||
"model": {"model": "gpt-x", "timeout_seconds": 120, "api_key_env": "SCRIPTORIUM_API_KEY"}
|
"model": {"model": "gpt-x", "timeout_seconds": 120, "service_tier": "flex", "api_key_env": "SCRIPTORIUM_API_KEY"}
|
||||||
}`)
|
}`)
|
||||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewReader(body))
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewReader(body))
|
||||||
w := httptest.NewRecorder()
|
w := httptest.NewRecorder()
|
||||||
@@ -110,10 +155,20 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
|||||||
if metadata["model_name"] != "m1" || metadata["endpoint"] != "http://llm/v1" {
|
if metadata["model_name"] != "m1" || metadata["endpoint"] != "http://llm/v1" {
|
||||||
t.Fatalf("unexpected model metadata: name=%#v endpoint=%#v", metadata["model_name"], metadata["endpoint"])
|
t.Fatalf("unexpected model metadata: name=%#v endpoint=%#v", metadata["model_name"], metadata["endpoint"])
|
||||||
}
|
}
|
||||||
|
usage := metadata["usage"].(map[string]any)
|
||||||
|
if usage["prompt_tokens"] != float64(1) || usage["completion_tokens"] != float64(2) || usage["total_tokens"] != float64(3) {
|
||||||
|
t.Fatalf("unexpected base usage metadata: %#v", usage)
|
||||||
|
}
|
||||||
|
if usage["cached_tokens"] != float64(4) || usage["cache_write_tokens"] != float64(5) {
|
||||||
|
t.Fatalf("unexpected cache usage metadata: %#v", usage)
|
||||||
|
}
|
||||||
modelParams := metadata["model_params"].(map[string]any)
|
modelParams := metadata["model_params"].(map[string]any)
|
||||||
if modelParams["api_key_env"] != envName {
|
if modelParams["api_key_env"] != envName {
|
||||||
t.Fatalf("expected model_params.api_key_env=%q, got %#v", envName, modelParams["api_key_env"])
|
t.Fatalf("expected model_params.api_key_env=%q, got %#v", envName, modelParams["api_key_env"])
|
||||||
}
|
}
|
||||||
|
if modelParams["service_tier"] != "priority" {
|
||||||
|
t.Fatalf("expected model_params.service_tier=priority, got %#v", modelParams["service_tier"])
|
||||||
|
}
|
||||||
if strings.Contains(w.Body.String(), secret) {
|
if strings.Contains(w.Body.String(), secret) {
|
||||||
t.Fatalf("response leaked raw API key value: %s", w.Body.String())
|
t.Fatalf("response leaked raw API key value: %s", w.Body.String())
|
||||||
}
|
}
|
||||||
@@ -130,9 +185,111 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
|||||||
if r.last.Execution == nil || r.last.Execution.Model != "gpt-x" {
|
if r.last.Execution == nil || r.last.Execution.Model != "gpt-x" {
|
||||||
t.Fatalf("expected model override, got %#v", r.last.Execution)
|
t.Fatalf("expected model override, got %#v", r.last.Execution)
|
||||||
}
|
}
|
||||||
if r.last.Execution.TimeoutSeconds != 120 {
|
if r.last.Execution.TimeoutSeconds == nil || *r.last.Execution.TimeoutSeconds != 120 {
|
||||||
t.Fatalf("expected timeout_seconds override 120, got %#v", r.last.Execution)
|
t.Fatalf("expected timeout_seconds override 120, got %#v", r.last.Execution)
|
||||||
}
|
}
|
||||||
|
if r.last.Execution.ServiceTier != "flex" {
|
||||||
|
t.Fatalf("expected service_tier override flex, got %#v", r.last.Execution)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerInlineRefsWorkWithoutArtifactRoot(t *testing.T) {
|
||||||
|
h := newArtifactRootHandler(t, "")
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||||
|
"prompt_id":"p",
|
||||||
|
"inputs":{"x":{"type":"inline","body":"inline body"}}
|
||||||
|
}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerFileRefsWithoutArtifactRootAreRejected(t *testing.T) {
|
||||||
|
h := newArtifactRootHandler(t, "")
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||||
|
"prompt_id":"p",
|
||||||
|
"inputs":{"x":{"type":"file","uri":"input.txt"}}
|
||||||
|
}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
assertHTTPErrorCode(t, w, http.StatusBadRequest, "artifact_not_allowed")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerFileRefsUnderArtifactRootWork(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
if err := os.WriteFile(filepath.Join(root, "input.txt"), []byte("allowed"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
h := newArtifactRootHandler(t, root)
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||||
|
"prompt_id":"p",
|
||||||
|
"inputs":{"x":{"type":"file","uri":"input.txt"}}
|
||||||
|
}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerFileRefsAboveArtifactLimitAreRejected(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
if err := os.WriteFile(filepath.Join(root, "large.txt"), []byte("123456"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
h := newArtifactRootHandlerWithLimit(t, root, 5)
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||||
|
"prompt_id":"p",
|
||||||
|
"inputs":{"x":{"type":"file","uri":"large.txt"}}
|
||||||
|
}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
assertHTTPErrorCode(t, w, http.StatusRequestEntityTooLarge, "artifact_too_large")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerFileRefsOutsideArtifactRootAreRejected(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
outside := t.TempDir()
|
||||||
|
if err := os.WriteFile(filepath.Join(outside, "secret.txt"), []byte("denied"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
h := newArtifactRootHandler(t, root)
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
uri string
|
||||||
|
}{
|
||||||
|
{name: "relative traversal", uri: filepath.Join("..", filepath.Base(outside), "secret.txt")},
|
||||||
|
{name: "absolute outside root", uri: filepath.Join(outside, "secret.txt")},
|
||||||
|
}
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
body := fmt.Sprintf(`{
|
||||||
|
"prompt_id":"p",
|
||||||
|
"inputs":{"x":{"type":"file","uri":%q}}
|
||||||
|
}`, tc.uri)
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(body))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
assertHTTPErrorCode(t, w, http.StatusBadRequest, "artifact_not_allowed")
|
||||||
|
})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
|
func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
|
||||||
@@ -164,6 +321,265 @@ func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
|
|||||||
if metadata["selected_profile_id"] != "prompt-default" {
|
if metadata["selected_profile_id"] != "prompt-default" {
|
||||||
t.Fatalf("expected selected_profile_id from result, got %#v", metadata["selected_profile_id"])
|
t.Fatalf("expected selected_profile_id from result, got %#v", metadata["selected_profile_id"])
|
||||||
}
|
}
|
||||||
|
usage := metadata["usage"].(map[string]any)
|
||||||
|
if usage["cached_tokens"] != float64(0) || usage["cache_write_tokens"] != float64(0) {
|
||||||
|
t.Fatalf("expected zero cache usage fields to be included, got %#v", usage)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerModelOverrideMapsAllSupportedExecutionFields(t *testing.T) {
|
||||||
|
r := &fakeRunner{result: &domain.RunResult{
|
||||||
|
Artifact: domain.Artifact{Body: []byte("ok")},
|
||||||
|
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
|
||||||
|
EffectiveModelParams: domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||||
|
}}
|
||||||
|
h := NewHandler(r)
|
||||||
|
|
||||||
|
reqBody := `{
|
||||||
|
"prompt_id": "prompt-1",
|
||||||
|
"inputs": {"transcript": {"type": "file", "uri": "./t.md"}},
|
||||||
|
"model": {
|
||||||
|
"endpoint": "http://override/v1",
|
||||||
|
"model": "override-model",
|
||||||
|
"temperature": 0.6,
|
||||||
|
"max_tokens": 250,
|
||||||
|
"top_p": 0.85,
|
||||||
|
"timeout_seconds": 33,
|
||||||
|
"service_tier": "flex",
|
||||||
|
"reasoning_effort": "medium",
|
||||||
|
"api_key_env": "SCRIPTORIUM_API_KEY",
|
||||||
|
"extra_params": {"provider_option":"on"}
|
||||||
|
}
|
||||||
|
}`
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(reqBody))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
if r.last.Execution == nil {
|
||||||
|
t.Fatalf("expected execution override in run request")
|
||||||
|
}
|
||||||
|
got := r.last.Execution
|
||||||
|
if got.Endpoint != "http://override/v1" ||
|
||||||
|
got.Model != "override-model" ||
|
||||||
|
got.ServiceTier != "flex" ||
|
||||||
|
got.ReasoningEffort != "medium" ||
|
||||||
|
got.APIKeyEnv != "SCRIPTORIUM_API_KEY" {
|
||||||
|
t.Fatalf("unexpected mapped execution target: %+v", got)
|
||||||
|
}
|
||||||
|
if got.Temperature == nil || *got.Temperature != 0.6 {
|
||||||
|
t.Fatalf("unexpected mapped temperature: %#v", got.Temperature)
|
||||||
|
}
|
||||||
|
if got.MaxTokens == nil || *got.MaxTokens != 250 {
|
||||||
|
t.Fatalf("unexpected mapped max_tokens: %#v", got.MaxTokens)
|
||||||
|
}
|
||||||
|
if got.TopP == nil || *got.TopP != 0.85 {
|
||||||
|
t.Fatalf("unexpected mapped top_p: %#v", got.TopP)
|
||||||
|
}
|
||||||
|
if got.TimeoutSeconds == nil || *got.TimeoutSeconds != 33 {
|
||||||
|
t.Fatalf("unexpected mapped timeout_seconds: %#v", got.TimeoutSeconds)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got.ExtraParams, map[string]any{"provider_option": "on"}) {
|
||||||
|
t.Fatalf("unexpected mapped extra_params: %#v", got.ExtraParams)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerModelOverrideAcceptsJSONCompatibleExtraParams(t *testing.T) {
|
||||||
|
r := &fakeRunner{result: &domain.RunResult{
|
||||||
|
Artifact: domain.Artifact{Body: []byte("ok")},
|
||||||
|
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
|
||||||
|
EffectiveModelParams: domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||||
|
}}
|
||||||
|
h := NewHandler(r)
|
||||||
|
|
||||||
|
reqBody := `{
|
||||||
|
"prompt_id": "prompt-1",
|
||||||
|
"inputs": {"transcript": {"type": "file", "uri": "./t.md"}},
|
||||||
|
"model": {
|
||||||
|
"extra_params": {
|
||||||
|
"string_value": "enabled",
|
||||||
|
"number_value": 42,
|
||||||
|
"boolean_value": true,
|
||||||
|
"object_value": {"nested": "value", "count": 2},
|
||||||
|
"array_value": ["first", 3, false]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}`
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(reqBody))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
if r.last.Execution == nil {
|
||||||
|
t.Fatal("expected execution override in run request")
|
||||||
|
}
|
||||||
|
want := map[string]any{
|
||||||
|
"string_value": "enabled",
|
||||||
|
"number_value": float64(42),
|
||||||
|
"boolean_value": true,
|
||||||
|
"object_value": map[string]any{"nested": "value", "count": float64(2)},
|
||||||
|
"array_value": []any{"first", float64(3), false},
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(r.last.Execution.ExtraParams, want) {
|
||||||
|
t.Fatalf("unexpected mapped extra_params:\ngot=%#v\nwant=%#v", r.last.Execution.ExtraParams, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerModelOverrideExplicitZeroTemperatureMapsAsPresent(t *testing.T) {
|
||||||
|
r := &fakeRunner{result: &domain.RunResult{
|
||||||
|
Artifact: domain.Artifact{Body: []byte("ok")},
|
||||||
|
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
|
||||||
|
EffectiveModelParams: domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1", Temperature: 0},
|
||||||
|
}}
|
||||||
|
h := NewHandler(r)
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||||
|
"prompt_id": "prompt-1",
|
||||||
|
"inputs": {"transcript": {"type": "file", "uri": "./t.md"}},
|
||||||
|
"model": {"temperature": 0}
|
||||||
|
}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
if r.last.Execution == nil || r.last.Execution.Temperature == nil {
|
||||||
|
t.Fatalf("expected temperature override to be present, got %#v", r.last.Execution)
|
||||||
|
}
|
||||||
|
if *r.last.Execution.Temperature != 0 {
|
||||||
|
t.Fatalf("expected zero temperature override, got %v", *r.last.Execution.Temperature)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerModelOverrideOmittedTemperatureMapsAsAbsent(t *testing.T) {
|
||||||
|
r := &fakeRunner{result: &domain.RunResult{
|
||||||
|
Artifact: domain.Artifact{Body: []byte("ok")},
|
||||||
|
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
|
||||||
|
EffectiveModelParams: domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1", Temperature: 0.7},
|
||||||
|
}}
|
||||||
|
h := NewHandler(r)
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||||
|
"prompt_id": "prompt-1",
|
||||||
|
"inputs": {"transcript": {"type": "file", "uri": "./t.md"}},
|
||||||
|
"model": {"model": "override-model"}
|
||||||
|
}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
if r.last.Execution == nil {
|
||||||
|
t.Fatal("expected model override")
|
||||||
|
}
|
||||||
|
if r.last.Execution.Temperature != nil {
|
||||||
|
t.Fatalf("expected omitted temperature to remain absent, got %#v", r.last.Execution.Temperature)
|
||||||
|
}
|
||||||
|
|
||||||
|
var resp map[string]any
|
||||||
|
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||||
|
t.Fatalf("invalid JSON response: %v", err)
|
||||||
|
}
|
||||||
|
metadata := resp["metadata"].(map[string]any)
|
||||||
|
params := metadata["model_params"].(map[string]any)
|
||||||
|
if params["temperature"] != 0.7 {
|
||||||
|
t.Fatalf("expected effective profile/default temperature in response, got %#v", params["temperature"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerResponseMetadataModelParamsIncludesAllSupportedFields(t *testing.T) {
|
||||||
|
r := &fakeRunner{result: &domain.RunResult{
|
||||||
|
Artifact: domain.Artifact{
|
||||||
|
Name: "output",
|
||||||
|
ContentType: "text/plain",
|
||||||
|
Body: []byte("ok"),
|
||||||
|
Size: 2,
|
||||||
|
Hash: "abc",
|
||||||
|
},
|
||||||
|
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
|
||||||
|
EffectiveModelParams: domain.ExecutionTarget{
|
||||||
|
Endpoint: "http://llm/v1",
|
||||||
|
Model: "gpt-test",
|
||||||
|
Temperature: 0.4,
|
||||||
|
MaxTokens: 321,
|
||||||
|
TopP: 0.7,
|
||||||
|
TimeoutSeconds: 45,
|
||||||
|
ServiceTier: "priority",
|
||||||
|
ReasoningEffort: "high",
|
||||||
|
APIKeyEnv: "SCRIPTORIUM_API_KEY",
|
||||||
|
ExtraParams: map[string]any{
|
||||||
|
"provider_option": "on",
|
||||||
|
"number_value": 42,
|
||||||
|
"object_value": map[string]any{"nested": "value"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}}
|
||||||
|
h := NewHandler(r)
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p","inputs":{"x":{"type":"file","uri":"a"}}}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
var resp map[string]any
|
||||||
|
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||||
|
t.Fatalf("invalid JSON response: %v", err)
|
||||||
|
}
|
||||||
|
metadata := resp["metadata"].(map[string]any)
|
||||||
|
params := metadata["model_params"].(map[string]any)
|
||||||
|
|
||||||
|
if params["endpoint"] != "http://llm/v1" {
|
||||||
|
t.Fatalf("unexpected endpoint: %#v", params["endpoint"])
|
||||||
|
}
|
||||||
|
if params["model"] != "gpt-test" {
|
||||||
|
t.Fatalf("unexpected model: %#v", params["model"])
|
||||||
|
}
|
||||||
|
if params["temperature"] != 0.4 {
|
||||||
|
t.Fatalf("unexpected temperature: %#v", params["temperature"])
|
||||||
|
}
|
||||||
|
if params["max_tokens"] != float64(321) {
|
||||||
|
t.Fatalf("unexpected max_tokens: %#v", params["max_tokens"])
|
||||||
|
}
|
||||||
|
if params["top_p"] != 0.7 {
|
||||||
|
t.Fatalf("unexpected top_p: %#v", params["top_p"])
|
||||||
|
}
|
||||||
|
if params["timeout_seconds"] != float64(45) {
|
||||||
|
t.Fatalf("unexpected timeout_seconds: %#v", params["timeout_seconds"])
|
||||||
|
}
|
||||||
|
if params["service_tier"] != "priority" {
|
||||||
|
t.Fatalf("unexpected service_tier: %#v", params["service_tier"])
|
||||||
|
}
|
||||||
|
if params["reasoning_effort"] != "high" {
|
||||||
|
t.Fatalf("unexpected reasoning_effort: %#v", params["reasoning_effort"])
|
||||||
|
}
|
||||||
|
if params["api_key_env"] != "SCRIPTORIUM_API_KEY" {
|
||||||
|
t.Fatalf("unexpected api_key_env: %#v", params["api_key_env"])
|
||||||
|
}
|
||||||
|
extraParams, ok := params["extra_params"].(map[string]any)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("expected extra_params object, got %#v", params["extra_params"])
|
||||||
|
}
|
||||||
|
if extraParams["provider_option"] != "on" {
|
||||||
|
t.Fatalf("unexpected extra_params.provider_option: %#v", extraParams["provider_option"])
|
||||||
|
}
|
||||||
|
if extraParams["number_value"] != float64(42) {
|
||||||
|
t.Fatalf("unexpected extra_params.number_value: %#v", extraParams["number_value"])
|
||||||
|
}
|
||||||
|
objectValue, ok := extraParams["object_value"].(map[string]any)
|
||||||
|
if !ok || objectValue["nested"] != "value" {
|
||||||
|
t.Fatalf("unexpected extra_params.object_value: %#v", extraParams["object_value"])
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestHandlerInvalidJSON(t *testing.T) {
|
func TestHandlerInvalidJSON(t *testing.T) {
|
||||||
@@ -178,6 +594,69 @@ func TestHandlerInvalidJSON(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestHandlerRejectsTrailingJSON(t *testing.T) {
|
||||||
|
h := NewHandler(&fakeRunner{})
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p","inputs":{"x":{"type":"file","uri":"a"}}} {}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
assertHTTPErrorCode(t, w, http.StatusBadRequest, "invalid_json")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerRequestTooLarge(t *testing.T) {
|
||||||
|
h := NewHandlerWithOptions(&fakeRunner{}, HandlerOptions{MaxRequestBytes: 12})
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p","inputs":{"x":{"type":"file","uri":"a"}}}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
assertHTTPErrorCode(t, w, http.StatusRequestEntityTooLarge, "request_too_large")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerMalformedJSONBelowLimitStillBadRequest(t *testing.T) {
|
||||||
|
h := NewHandlerWithOptions(&fakeRunner{}, HandlerOptions{MaxRequestBytes: 1024})
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString("{"))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
assertHTTPErrorCode(t, w, http.StatusBadRequest, "invalid_json")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerResponseTooLarge(t *testing.T) {
|
||||||
|
h := NewHandlerWithOptions(&fakeRunner{result: &domain.RunResult{
|
||||||
|
Artifact: domain.Artifact{Body: []byte(strings.Repeat("x", 128))},
|
||||||
|
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
|
||||||
|
EffectiveModelParams: domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||||
|
}}, HandlerOptions{MaxRequestBytes: 1024, MaxResponseBytes: 64})
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p","inputs":{"x":{"type":"file","uri":"a"}}}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
assertHTTPErrorCode(t, w, http.StatusRequestEntityTooLarge, "response_too_large")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerRawOutputDoesNotBypassResponseLimit(t *testing.T) {
|
||||||
|
h := NewHandlerWithOptions(&fakeRunner{result: &domain.RunResult{
|
||||||
|
Artifact: domain.Artifact{Body: []byte("ok")},
|
||||||
|
RawOutput: strings.Repeat("raw", 80),
|
||||||
|
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
|
||||||
|
EffectiveModelParams: domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||||
|
}}, HandlerOptions{MaxRequestBytes: 1024, MaxResponseBytes: 128})
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||||
|
"prompt_id":"p",
|
||||||
|
"inputs":{"x":{"type":"file","uri":"a"}},
|
||||||
|
"include_raw_output":true
|
||||||
|
}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
assertHTTPErrorCode(t, w, http.StatusRequestEntityTooLarge, "response_too_large")
|
||||||
|
}
|
||||||
|
|
||||||
func TestHandlerMissingPromptID(t *testing.T) {
|
func TestHandlerMissingPromptID(t *testing.T) {
|
||||||
h := NewHandler(&fakeRunner{})
|
h := NewHandler(&fakeRunner{})
|
||||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"inputs":{"x":{"type":"file","uri":"a"}}}`))
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"inputs":{"x":{"type":"file","uri":"a"}}}`))
|
||||||
@@ -198,6 +677,54 @@ func TestHandlerMissingPromptID(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestHandlerReservedExtraParamsThroughRunnerMapsToInvalidRequest(t *testing.T) {
|
||||||
|
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
runner := usecase.NewRunner(
|
||||||
|
handlerPromptRepo{def: &domain.PromptDefinition{
|
||||||
|
ID: "p",
|
||||||
|
Version: "1",
|
||||||
|
DefaultProfile: "exec",
|
||||||
|
Templates: []domain.PromptMessageTemplate{{Role: "user", Content: "hi"}},
|
||||||
|
OutputFormat: domain.FormatText,
|
||||||
|
Validation: domain.OutputContract{Format: domain.FormatText, ValidationMode: domain.ValidationNone},
|
||||||
|
}},
|
||||||
|
handlerProfileRepo{profile: &domain.ExecutionProfile{
|
||||||
|
ID: "exec",
|
||||||
|
Endpoint: "http://example.invalid/v1",
|
||||||
|
Model: "model",
|
||||||
|
}},
|
||||||
|
handlerArtifactReader{},
|
||||||
|
handlerRenderer{},
|
||||||
|
llmClient,
|
||||||
|
nil,
|
||||||
|
)
|
||||||
|
h := NewHandler(runner)
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||||
|
"prompt_id":"p",
|
||||||
|
"inputs":{"x":{"type":"file","uri":"a"}},
|
||||||
|
"model":{"extra_params":{"model":"collision"}}
|
||||||
|
}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusBadRequest {
|
||||||
|
t.Fatalf("expected 400, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
var resp map[string]any
|
||||||
|
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||||
|
t.Fatalf("invalid JSON response: %v", err)
|
||||||
|
}
|
||||||
|
errBody := resp["error"].(map[string]any)
|
||||||
|
if errBody["code"] != "invalid_request" {
|
||||||
|
t.Fatalf("expected invalid_request code, got %#v", errBody["code"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestHandlerUsecaseErrorMapping(t *testing.T) {
|
func TestHandlerUsecaseErrorMapping(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -207,12 +734,14 @@ func TestHandlerUsecaseErrorMapping(t *testing.T) {
|
|||||||
message string
|
message string
|
||||||
avoidCause string
|
avoidCause string
|
||||||
}{
|
}{
|
||||||
{name: "prompt not found", err: wrap(usecase.ErrProfileLoad, promptdef.ErrPromptDefinitionNotFound), status: http.StatusNotFound, code: "prompt_not_found", message: "prompt definition not found"},
|
{name: "prompt not found", err: wrap(usecase.ErrPromptLoad, promptdef.ErrPromptDefinitionNotFound), status: http.StatusNotFound, code: "prompt_not_found", message: "prompt definition not found"},
|
||||||
{name: "prompt load invalid", err: wrap(usecase.ErrProfileLoad, promptdef.ErrInvalidPromptDefinition), status: http.StatusBadRequest, code: "prompt_load_failed", message: "failed to load prompt definition"},
|
{name: "prompt load invalid", err: wrap(usecase.ErrPromptLoad, promptdef.ErrInvalidPromptDefinition), status: http.StatusBadRequest, code: "prompt_load_failed", message: "failed to load prompt definition"},
|
||||||
{name: "missing profile/default", err: wrap(usecase.ErrInvalidRequest, errors.New("profile id is required either in request or prompt default_profile")), status: http.StatusBadRequest, code: "profile_required", message: "profile_id is required when prompt default_profile is not set"},
|
{name: "prompt load generic", err: wrap(usecase.ErrPromptLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "prompt_load_failed", message: "failed to load prompt definition", avoidCause: "read failed"},
|
||||||
|
{name: "missing profile/default", err: wrap(usecase.ErrInvalidRequest, usecase.ErrProfileRequired), status: http.StatusBadRequest, code: "profile_required", message: "profile_id is required when prompt default_profile is not set"},
|
||||||
{name: "profile not found", err: wrap(usecase.ErrProfileLoad, profile.ErrProfileNotFound), status: http.StatusNotFound, code: "profile_not_found", message: "execution profile not found"},
|
{name: "profile not found", err: wrap(usecase.ErrProfileLoad, profile.ErrProfileNotFound), status: http.StatusNotFound, code: "profile_not_found", message: "execution profile not found"},
|
||||||
{name: "profile invalid", err: wrap(usecase.ErrProfileLoad, profile.ErrInvalidProfile), status: http.StatusBadRequest, code: "profile_load_failed", message: "failed to load execution profile"},
|
{name: "profile invalid", err: wrap(usecase.ErrProfileLoad, profile.ErrInvalidProfile), status: http.StatusBadRequest, code: "profile_load_failed", message: "failed to load execution profile"},
|
||||||
{name: "api key env missing", err: wrap(usecase.ErrInvalidRequest, errors.New(`api key environment variable "SCRIPTORIUM_API_KEY" is not set`)), status: http.StatusBadRequest, code: "api_key_env_missing", message: "api_key_env is set but the environment variable is missing"},
|
{name: "profile load generic", err: wrap(usecase.ErrProfileLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "profile_load_failed", message: "failed to load execution profile", avoidCause: "read failed"},
|
||||||
|
{name: "api key env missing", err: wrap(usecase.ErrInvalidRequest, usecase.ErrAPIKeyEnvMissing), status: http.StatusBadRequest, code: "api_key_env_missing", message: "api_key_env is set but the environment variable is missing"},
|
||||||
{name: "artifact", err: wrap(usecase.ErrArtifactLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "artifact_read_failed", message: "failed to read input artifact", avoidCause: "read failed"},
|
{name: "artifact", err: wrap(usecase.ErrArtifactLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "artifact_read_failed", message: "failed to read input artifact", avoidCause: "read failed"},
|
||||||
{name: "prompt render", err: wrap(usecase.ErrPromptRender, fmt.Errorf("render failed")), status: http.StatusBadRequest, code: "prompt_render_failed", message: "failed to render prompt", avoidCause: "render failed"},
|
{name: "prompt render", err: wrap(usecase.ErrPromptRender, fmt.Errorf("render failed")), status: http.StatusBadRequest, code: "prompt_render_failed", message: "failed to render prompt", avoidCause: "render failed"},
|
||||||
{name: "llm", err: wrap(usecase.ErrLLMGenerate, fmt.Errorf("llm failed")), status: http.StatusBadGateway, code: "llm_failed", message: "model generation request failed", avoidCause: "llm failed"},
|
{name: "llm", err: wrap(usecase.ErrLLMGenerate, fmt.Errorf("llm failed")), status: http.StatusBadGateway, code: "llm_failed", message: "model generation request failed", avoidCause: "llm failed"},
|
||||||
@@ -318,3 +847,54 @@ func TestHandlerValidationFailureStillSuccessAndRawOutputOptIn(t *testing.T) {
|
|||||||
func wrap(stage error, cause error) error {
|
func wrap(stage error, cause error) error {
|
||||||
return fmt.Errorf("%w: %w", stage, cause)
|
return fmt.Errorf("%w: %w", stage, cause)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func newArtifactRootHandler(t *testing.T, root string) *Handler {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
return newArtifactRootHandlerWithLimit(t, root, 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
func newArtifactRootHandlerWithLimit(t *testing.T, root string, maxArtifactBytes int64) *Handler {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
reader, err := artifact.NewRestrictedCompositeReaderWithLimit(root, maxArtifactBytes)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected restricted artifact reader: %v", err)
|
||||||
|
}
|
||||||
|
runner := usecase.NewRunner(
|
||||||
|
handlerPromptRepo{def: &domain.PromptDefinition{
|
||||||
|
ID: "p",
|
||||||
|
Version: "1",
|
||||||
|
DefaultProfile: "exec",
|
||||||
|
Templates: []domain.PromptMessageTemplate{{Role: "user", Content: "hi"}},
|
||||||
|
OutputFormat: domain.FormatText,
|
||||||
|
Validation: domain.OutputContract{Format: domain.FormatText, ValidationMode: domain.ValidationNone},
|
||||||
|
}},
|
||||||
|
handlerProfileRepo{profile: &domain.ExecutionProfile{
|
||||||
|
ID: "exec",
|
||||||
|
Endpoint: "http://example.invalid/v1",
|
||||||
|
Model: "model",
|
||||||
|
}},
|
||||||
|
reader,
|
||||||
|
handlerRenderer{},
|
||||||
|
handlerLLMClient{},
|
||||||
|
nil,
|
||||||
|
)
|
||||||
|
return NewHandler(runner)
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertHTTPErrorCode(t *testing.T, w *httptest.ResponseRecorder, status int, code string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if w.Code != status {
|
||||||
|
t.Fatalf("expected %d, got %d body=%s", status, w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
var resp map[string]any
|
||||||
|
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||||
|
t.Fatalf("invalid JSON response: %v", err)
|
||||||
|
}
|
||||||
|
errBody := resp["error"].(map[string]any)
|
||||||
|
if errBody["code"] != code {
|
||||||
|
t.Fatalf("expected code %q, got %#v", code, errBody["code"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -7,15 +7,20 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
|
"io"
|
||||||
"mime"
|
"mime"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
ErrUnsupportedRefType = errors.New("unsupported artifact reference type")
|
ErrUnsupportedRefType = errors.New("unsupported artifact reference type")
|
||||||
ErrMissingInlineBody = errors.New("missing body for inline artifact")
|
ErrMissingInlineBody = errors.New("missing body for inline artifact")
|
||||||
ErrMissingFilePath = errors.New("missing file path for file artifact")
|
ErrMissingFilePath = errors.New("missing file path for file artifact")
|
||||||
|
ErrFileNotAllowed = errors.New("file artifact references are not allowed")
|
||||||
|
ErrFileOutsideRoot = errors.New("file artifact path is outside artifact root")
|
||||||
|
ErrFileTooLarge = errors.New("file artifact exceeds size limit")
|
||||||
)
|
)
|
||||||
|
|
||||||
// Reader resolves artifact references into actual artifacts.
|
// Reader resolves artifact references into actual artifacts.
|
||||||
@@ -26,7 +31,7 @@ type Reader interface {
|
|||||||
// CompositeReader routes artifact resolution based on the reference type.
|
// CompositeReader routes artifact resolution based on the reference type.
|
||||||
type CompositeReader struct {
|
type CompositeReader struct {
|
||||||
inlineReader *inlineReader
|
inlineReader *inlineReader
|
||||||
fileReader *fileReader
|
fileReader Reader
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewCompositeReader() Reader {
|
func NewCompositeReader() Reader {
|
||||||
@@ -36,6 +41,21 @@ func NewCompositeReader() Reader {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func NewRestrictedCompositeReader(root string) (Reader, error) {
|
||||||
|
return NewRestrictedCompositeReaderWithLimit(root, 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewRestrictedCompositeReaderWithLimit(root string, maxBytes int64) (Reader, error) {
|
||||||
|
fileReader, err := newRestrictedFileReader(root, maxBytes)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return &CompositeReader{
|
||||||
|
inlineReader: &inlineReader{},
|
||||||
|
fileReader: fileReader,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (c *CompositeReader) Read(ctx context.Context, ref domain.ArtifactRef) (*domain.Artifact, error) {
|
func (c *CompositeReader) Read(ctx context.Context, ref domain.ArtifactRef) (*domain.Artifact, error) {
|
||||||
select {
|
select {
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
@@ -89,21 +109,133 @@ func (r *fileReader) Read(ctx context.Context, ref domain.ArtifactRef) (*domain.
|
|||||||
return nil, ErrMissingFilePath
|
return nil, ErrMissingFilePath
|
||||||
}
|
}
|
||||||
|
|
||||||
data, err := os.ReadFile(ref.URI)
|
return readFileArtifact(ref.URI)
|
||||||
if err != nil {
|
}
|
||||||
return nil, fmt.Errorf("failed to read file %s: %w", ref.URI, err)
|
|
||||||
|
type deniedFileReader struct{}
|
||||||
|
|
||||||
|
func (r deniedFileReader) Read(ctx context.Context, ref domain.ArtifactRef) (*domain.Artifact, error) {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return nil, ctx.Err()
|
||||||
|
default:
|
||||||
}
|
}
|
||||||
|
|
||||||
contentType := mime.TypeByExtension(filepath.Ext(ref.URI))
|
if ref.URI == "" {
|
||||||
|
return nil, ErrMissingFilePath
|
||||||
|
}
|
||||||
|
return nil, ErrFileNotAllowed
|
||||||
|
}
|
||||||
|
|
||||||
|
type restrictedFileReader struct {
|
||||||
|
root string
|
||||||
|
maxBytes int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func newRestrictedFileReader(root string, maxBytes int64) (Reader, error) {
|
||||||
|
if maxBytes < 0 {
|
||||||
|
return nil, fmt.Errorf("artifact size limit must be greater than or equal to 0")
|
||||||
|
}
|
||||||
|
cleanRoot := strings.TrimSpace(root)
|
||||||
|
if cleanRoot == "" {
|
||||||
|
return deniedFileReader{}, nil
|
||||||
|
}
|
||||||
|
absRoot, err := filepath.Abs(filepath.Clean(cleanRoot))
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("resolve artifact root: %w", err)
|
||||||
|
}
|
||||||
|
return &restrictedFileReader{root: absRoot, maxBytes: maxBytes}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *restrictedFileReader) Read(ctx context.Context, ref domain.ArtifactRef) (*domain.Artifact, error) {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return nil, ctx.Err()
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
|
||||||
|
if ref.URI == "" {
|
||||||
|
return nil, ErrMissingFilePath
|
||||||
|
}
|
||||||
|
|
||||||
|
path, err := r.resolveLexicalPath(ref.URI)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return readFileArtifactWithLimit(path, r.maxBytes)
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolveLexicalPath checks cleaned path containment without resolving symlinks.
|
||||||
|
func (r *restrictedFileReader) resolveLexicalPath(rawPath string) (string, error) {
|
||||||
|
cleanPath := filepath.Clean(strings.TrimSpace(rawPath))
|
||||||
|
var candidate string
|
||||||
|
if filepath.IsAbs(cleanPath) {
|
||||||
|
candidate = cleanPath
|
||||||
|
} else {
|
||||||
|
candidate = filepath.Join(r.root, cleanPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
absCandidate, err := filepath.Abs(candidate)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("resolve artifact path: %w", err)
|
||||||
|
}
|
||||||
|
absCandidate = filepath.Clean(absCandidate)
|
||||||
|
|
||||||
|
rel, err := filepath.Rel(r.root, absCandidate)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("compare artifact path to root: %w", err)
|
||||||
|
}
|
||||||
|
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || filepath.IsAbs(rel) {
|
||||||
|
return "", ErrFileOutsideRoot
|
||||||
|
}
|
||||||
|
return absCandidate, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func readFileArtifact(path string) (*domain.Artifact, error) {
|
||||||
|
return readFileArtifactWithLimit(path, 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
func readFileArtifactWithLimit(path string, maxBytes int64) (*domain.Artifact, error) {
|
||||||
|
if maxBytes < 0 {
|
||||||
|
return nil, fmt.Errorf("file size limit must be greater than or equal to 0")
|
||||||
|
}
|
||||||
|
|
||||||
|
file, err := os.Open(path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to read file %s: %w", path, err)
|
||||||
|
}
|
||||||
|
defer file.Close()
|
||||||
|
|
||||||
|
info, err := file.Stat()
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to stat file %s: %w", path, err)
|
||||||
|
}
|
||||||
|
if maxBytes > 0 && info.Size() > maxBytes {
|
||||||
|
return nil, ErrFileTooLarge
|
||||||
|
}
|
||||||
|
|
||||||
|
var reader io.Reader = file
|
||||||
|
if maxBytes > 0 {
|
||||||
|
reader = io.LimitReader(file, maxBytes+1)
|
||||||
|
}
|
||||||
|
data, err := io.ReadAll(reader)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to read file %s: %w", path, err)
|
||||||
|
}
|
||||||
|
if maxBytes > 0 && int64(len(data)) > maxBytes {
|
||||||
|
return nil, ErrFileTooLarge
|
||||||
|
}
|
||||||
|
|
||||||
|
contentType := mime.TypeByExtension(filepath.Ext(path))
|
||||||
if contentType == "" {
|
if contentType == "" {
|
||||||
contentType = defaults.ContentTypeTextPlain
|
contentType = defaults.ContentTypeTextPlain
|
||||||
}
|
}
|
||||||
|
|
||||||
return &domain.Artifact{
|
return &domain.Artifact{
|
||||||
Name: filepath.Base(ref.URI),
|
Name: filepath.Base(path),
|
||||||
ContentType: contentType,
|
ContentType: contentType,
|
||||||
Body: data,
|
Body: data,
|
||||||
URI: ref.URI,
|
URI: path,
|
||||||
Size: int64(len(data)),
|
Size: int64(len(data)),
|
||||||
Hash: fmt.Sprintf("%x", sha256.Sum256(data)),
|
Hash: fmt.Sprintf("%x", sha256.Sum256(data)),
|
||||||
}, nil
|
}, nil
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"os"
|
"os"
|
||||||
|
"path/filepath"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
@@ -46,8 +47,8 @@ func TestCompositeReader_Read(t *testing.T) {
|
|||||||
|
|
||||||
t.Run("unsupported ref type", func(t *testing.T) {
|
t.Run("unsupported ref type", func(t *testing.T) {
|
||||||
ref := domain.ArtifactRef{
|
ref := domain.ArtifactRef{
|
||||||
Type: domain.ArtifactRefS3,
|
Type: domain.ArtifactRefType("unsupported"),
|
||||||
URI: "s3://bucket/key",
|
URI: "unsupported://bucket/key",
|
||||||
}
|
}
|
||||||
_, err := reader.Read(ctx, ref)
|
_, err := reader.Read(ctx, ref)
|
||||||
if !errors.Is(err, ErrUnsupportedRefType) {
|
if !errors.Is(err, ErrUnsupportedRefType) {
|
||||||
@@ -56,6 +57,157 @@ func TestCompositeReader_Read(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRestrictedCompositeReader(t *testing.T) {
|
||||||
|
ctx := context.Background()
|
||||||
|
root := t.TempDir()
|
||||||
|
outside := t.TempDir()
|
||||||
|
|
||||||
|
if err := os.WriteFile(filepath.Join(root, "input.txt"), []byte("allowed"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.Mkdir(filepath.Join(root, "nested"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(outside, "secret.txt"), []byte("denied"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
reader, err := NewRestrictedCompositeReader(root)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected restricted reader construction, got %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Run("accepts relative contained path", func(t *testing.T) {
|
||||||
|
art, err := reader.Read(ctx, domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: "nested/../input.txt"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected contained relative path to succeed, got %v", err)
|
||||||
|
}
|
||||||
|
if string(art.Body) != "allowed" {
|
||||||
|
t.Fatalf("unexpected artifact body: %q", string(art.Body))
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("accepts absolute contained path", func(t *testing.T) {
|
||||||
|
art, err := reader.Read(ctx, domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: filepath.Join(root, "input.txt")})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected contained absolute path to succeed, got %v", err)
|
||||||
|
}
|
||||||
|
if art.Name != "input.txt" {
|
||||||
|
t.Fatalf("unexpected artifact name: %q", art.Name)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("rejects relative traversal outside root", func(t *testing.T) {
|
||||||
|
_, err := reader.Read(ctx, domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: filepath.Join("..", filepath.Base(outside), "secret.txt")})
|
||||||
|
if !errors.Is(err, ErrFileOutsideRoot) {
|
||||||
|
t.Fatalf("expected ErrFileOutsideRoot, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("rejects absolute path outside root", func(t *testing.T) {
|
||||||
|
_, err := reader.Read(ctx, domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: filepath.Join(outside, "secret.txt")})
|
||||||
|
if !errors.Is(err, ErrFileOutsideRoot) {
|
||||||
|
t.Fatalf("expected ErrFileOutsideRoot, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRestrictedCompositeReaderFollowsSymlinkInsideRoot(t *testing.T) {
|
||||||
|
ctx := context.Background()
|
||||||
|
root := t.TempDir()
|
||||||
|
outside := t.TempDir()
|
||||||
|
|
||||||
|
target := filepath.Join(outside, "linked.txt")
|
||||||
|
if err := os.WriteFile(target, []byte("linked outside root"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
link := filepath.Join(root, "linked.txt")
|
||||||
|
if err := os.Symlink(target, link); err != nil {
|
||||||
|
t.Skipf("symlink creation unavailable: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
reader, err := NewRestrictedCompositeReader(root)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected restricted reader construction, got %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
art, err := reader.Read(ctx, domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: "linked.txt"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected symlink inside root to be followed, got %v", err)
|
||||||
|
}
|
||||||
|
if string(art.Body) != "linked outside root" {
|
||||||
|
t.Fatalf("unexpected artifact body: %q", string(art.Body))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRestrictedCompositeReaderWithoutRootDeniesFileRefs(t *testing.T) {
|
||||||
|
reader, err := NewRestrictedCompositeReader("")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected restricted reader construction, got %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
art, err := reader.Read(context.Background(), domain.ArtifactRef{Type: domain.ArtifactRefInline, Body: "inline"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected inline ref to work without artifact root, got %v", err)
|
||||||
|
}
|
||||||
|
if string(art.Body) != "inline" {
|
||||||
|
t.Fatalf("unexpected inline body: %q", string(art.Body))
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = reader.Read(context.Background(), domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: "input.txt"})
|
||||||
|
if !errors.Is(err, ErrFileNotAllowed) {
|
||||||
|
t.Fatalf("expected ErrFileNotAllowed, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRestrictedCompositeReaderFileSizeLimit(t *testing.T) {
|
||||||
|
ctx := context.Background()
|
||||||
|
root := t.TempDir()
|
||||||
|
if err := os.WriteFile(filepath.Join(root, "exact.txt"), []byte("12345"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(root, "large.txt"), []byte("123456"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
reader, err := NewRestrictedCompositeReaderWithLimit(root, 5)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected restricted reader construction, got %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
art, err := reader.Read(ctx, domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: "exact.txt"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected file at limit to succeed, got %v", err)
|
||||||
|
}
|
||||||
|
if string(art.Body) != "12345" {
|
||||||
|
t.Fatalf("unexpected artifact body: %q", string(art.Body))
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = reader.Read(ctx, domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: "large.txt"})
|
||||||
|
if !errors.Is(err, ErrFileTooLarge) {
|
||||||
|
t.Fatalf("expected ErrFileTooLarge, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRestrictedCompositeReaderFileSizeLimitZeroDisablesLimit(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
if err := os.WriteFile(filepath.Join(root, "large.txt"), []byte("123456"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
reader, err := NewRestrictedCompositeReaderWithLimit(root, 0)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected restricted reader construction, got %v", err)
|
||||||
|
}
|
||||||
|
art, err := reader.Read(context.Background(), domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: "large.txt"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected unlimited reader to succeed, got %v", err)
|
||||||
|
}
|
||||||
|
if string(art.Body) != "123456" {
|
||||||
|
t.Fatalf("unexpected artifact body: %q", string(art.Body))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestFileReader_Read(t *testing.T) {
|
func TestFileReader_Read(t *testing.T) {
|
||||||
content := []byte("test file content")
|
content := []byte("test file content")
|
||||||
tmpFile, err := os.CreateTemp("", "artifact_test_*.txt")
|
tmpFile, err := os.CreateTemp("", "artifact_test_*.txt")
|
||||||
|
|||||||
@@ -40,7 +40,11 @@ type Config struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type ServerConfig struct {
|
type ServerConfig struct {
|
||||||
Addr string `yaml:"addr"`
|
Addr string `yaml:"addr"`
|
||||||
|
ArtifactRoot string `yaml:"artifact_root"`
|
||||||
|
MaxRequestBytes *int64 `yaml:"max_request_bytes"`
|
||||||
|
MaxArtifactBytes *int64 `yaml:"max_artifact_bytes"`
|
||||||
|
MaxResponseBytes *int64 `yaml:"max_response_bytes"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type DefaultsConfig struct {
|
type DefaultsConfig struct {
|
||||||
@@ -53,16 +57,24 @@ type AppSettings struct {
|
|||||||
ProfileDir string
|
ProfileDir string
|
||||||
SchemaDir string
|
SchemaDir string
|
||||||
ServerAddr string
|
ServerAddr string
|
||||||
|
ArtifactRoot string
|
||||||
|
MaxRequestBytes int64
|
||||||
|
MaxArtifactBytes int64
|
||||||
|
MaxResponseBytes int64
|
||||||
DefaultRenderFormat renderformat.PreparedRunOutputFormat
|
DefaultRenderFormat renderformat.PreparedRunOutputFormat
|
||||||
}
|
}
|
||||||
|
|
||||||
// CLIOverrides can be applied after config load to enforce precedence.
|
// CLIOverrides can be applied after config load to enforce precedence.
|
||||||
type CLIOverrides struct {
|
type CLIOverrides struct {
|
||||||
PromptDir string
|
PromptDir string
|
||||||
ProfileDir string
|
ProfileDir string
|
||||||
SchemaDir string
|
SchemaDir string
|
||||||
ServerAddr string
|
ServerAddr string
|
||||||
RenderFormat string
|
ArtifactRoot string
|
||||||
|
MaxRequestBytes *int64
|
||||||
|
MaxArtifactBytes *int64
|
||||||
|
MaxResponseBytes *int64
|
||||||
|
RenderFormat string
|
||||||
}
|
}
|
||||||
|
|
||||||
// BuiltInDefaults returns compile-time application defaults.
|
// BuiltInDefaults returns compile-time application defaults.
|
||||||
@@ -70,6 +82,9 @@ func BuiltInDefaults() AppSettings {
|
|||||||
return AppSettings{
|
return AppSettings{
|
||||||
SchemaDir: defaults.SchemaDirDefault,
|
SchemaDir: defaults.SchemaDirDefault,
|
||||||
ServerAddr: defaults.HTTPAddrDefault,
|
ServerAddr: defaults.HTTPAddrDefault,
|
||||||
|
MaxRequestBytes: defaults.HTTPMaxRequestBytesDefault,
|
||||||
|
MaxArtifactBytes: defaults.HTTPMaxArtifactBytesDefault,
|
||||||
|
MaxResponseBytes: defaults.HTTPMaxResponseBytesDefault,
|
||||||
DefaultRenderFormat: renderformat.DefaultPreparedRunOutputFormat,
|
DefaultRenderFormat: renderformat.DefaultPreparedRunOutputFormat,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -142,6 +157,27 @@ func ApplyCLIOverrides(base AppSettings, overrides CLIOverrides) (AppSettings, e
|
|||||||
if v := strings.TrimSpace(overrides.ServerAddr); v != "" {
|
if v := strings.TrimSpace(overrides.ServerAddr); v != "" {
|
||||||
out.ServerAddr = v
|
out.ServerAddr = v
|
||||||
}
|
}
|
||||||
|
if v := strings.TrimSpace(overrides.ArtifactRoot); v != "" {
|
||||||
|
out.ArtifactRoot = filepath.Clean(v)
|
||||||
|
}
|
||||||
|
if overrides.MaxRequestBytes != nil {
|
||||||
|
if *overrides.MaxRequestBytes < 0 {
|
||||||
|
return AppSettings{}, fmt.Errorf("%w: server.max_request_bytes must be greater than or equal to 0", ErrInvalidConfig)
|
||||||
|
}
|
||||||
|
out.MaxRequestBytes = *overrides.MaxRequestBytes
|
||||||
|
}
|
||||||
|
if overrides.MaxArtifactBytes != nil {
|
||||||
|
if *overrides.MaxArtifactBytes < 0 {
|
||||||
|
return AppSettings{}, fmt.Errorf("%w: server.max_artifact_bytes must be greater than or equal to 0", ErrInvalidConfig)
|
||||||
|
}
|
||||||
|
out.MaxArtifactBytes = *overrides.MaxArtifactBytes
|
||||||
|
}
|
||||||
|
if overrides.MaxResponseBytes != nil {
|
||||||
|
if *overrides.MaxResponseBytes < 0 {
|
||||||
|
return AppSettings{}, fmt.Errorf("%w: server.max_response_bytes must be greater than or equal to 0", ErrInvalidConfig)
|
||||||
|
}
|
||||||
|
out.MaxResponseBytes = *overrides.MaxResponseBytes
|
||||||
|
}
|
||||||
if rawFormat := strings.TrimSpace(overrides.RenderFormat); rawFormat != "" {
|
if rawFormat := strings.TrimSpace(overrides.RenderFormat); rawFormat != "" {
|
||||||
parsed, err := renderformat.ParsePreparedRunOutputFormat(rawFormat)
|
parsed, err := renderformat.ParsePreparedRunOutputFormat(rawFormat)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -181,6 +217,27 @@ func applyConfig(base AppSettings, cfg Config) (AppSettings, error) {
|
|||||||
if v := strings.TrimSpace(cfg.Server.Addr); v != "" {
|
if v := strings.TrimSpace(cfg.Server.Addr); v != "" {
|
||||||
out.ServerAddr = v
|
out.ServerAddr = v
|
||||||
}
|
}
|
||||||
|
if v := strings.TrimSpace(cfg.Server.ArtifactRoot); v != "" {
|
||||||
|
out.ArtifactRoot = filepath.Clean(v)
|
||||||
|
}
|
||||||
|
if cfg.Server.MaxRequestBytes != nil {
|
||||||
|
if *cfg.Server.MaxRequestBytes < 0 {
|
||||||
|
return AppSettings{}, fmt.Errorf("%w: server.max_request_bytes must be greater than or equal to 0", ErrInvalidConfig)
|
||||||
|
}
|
||||||
|
out.MaxRequestBytes = *cfg.Server.MaxRequestBytes
|
||||||
|
}
|
||||||
|
if cfg.Server.MaxArtifactBytes != nil {
|
||||||
|
if *cfg.Server.MaxArtifactBytes < 0 {
|
||||||
|
return AppSettings{}, fmt.Errorf("%w: server.max_artifact_bytes must be greater than or equal to 0", ErrInvalidConfig)
|
||||||
|
}
|
||||||
|
out.MaxArtifactBytes = *cfg.Server.MaxArtifactBytes
|
||||||
|
}
|
||||||
|
if cfg.Server.MaxResponseBytes != nil {
|
||||||
|
if *cfg.Server.MaxResponseBytes < 0 {
|
||||||
|
return AppSettings{}, fmt.Errorf("%w: server.max_response_bytes must be greater than or equal to 0", ErrInvalidConfig)
|
||||||
|
}
|
||||||
|
out.MaxResponseBytes = *cfg.Server.MaxResponseBytes
|
||||||
|
}
|
||||||
if rawFormat := strings.TrimSpace(cfg.Defaults.RenderFormat); rawFormat != "" {
|
if rawFormat := strings.TrimSpace(cfg.Defaults.RenderFormat); rawFormat != "" {
|
||||||
parsed, err := renderformat.ParsePreparedRunOutputFormat(rawFormat)
|
parsed, err := renderformat.ParsePreparedRunOutputFormat(rawFormat)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||||
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -24,6 +25,20 @@ func TestLoadConfigMissingImplicitPathUsesBuiltInDefaults(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestBuiltInDefaultsIncludeHTTPSizeLimits(t *testing.T) {
|
||||||
|
got := BuiltInDefaults()
|
||||||
|
|
||||||
|
if got.MaxRequestBytes != defaults.HTTPMaxRequestBytesDefault {
|
||||||
|
t.Fatalf("unexpected max request bytes: %d", got.MaxRequestBytes)
|
||||||
|
}
|
||||||
|
if got.MaxArtifactBytes != defaults.HTTPMaxArtifactBytesDefault {
|
||||||
|
t.Fatalf("unexpected max artifact bytes: %d", got.MaxArtifactBytes)
|
||||||
|
}
|
||||||
|
if got.MaxResponseBytes != defaults.HTTPMaxResponseBytesDefault {
|
||||||
|
t.Fatalf("unexpected max response bytes: %d", got.MaxResponseBytes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestLoadConfigMissingExplicitPathReturnsError(t *testing.T) {
|
func TestLoadConfigMissingExplicitPathReturnsError(t *testing.T) {
|
||||||
tmp := t.TempDir()
|
tmp := t.TempDir()
|
||||||
missing := filepath.Join(tmp, "missing.yml")
|
missing := filepath.Join(tmp, "missing.yml")
|
||||||
@@ -92,6 +107,10 @@ profile_dir: ./profiles
|
|||||||
schema_dir: ./schemas
|
schema_dir: ./schemas
|
||||||
server:
|
server:
|
||||||
addr: 127.0.0.1:9090
|
addr: 127.0.0.1:9090
|
||||||
|
artifact_root: ./artifacts
|
||||||
|
max_request_bytes: 1024
|
||||||
|
max_artifact_bytes: 2048
|
||||||
|
max_response_bytes: 4096
|
||||||
defaults:
|
defaults:
|
||||||
render_format: json
|
render_format: json
|
||||||
`)
|
`)
|
||||||
@@ -113,11 +132,61 @@ defaults:
|
|||||||
if got.ServerAddr != "127.0.0.1:9090" {
|
if got.ServerAddr != "127.0.0.1:9090" {
|
||||||
t.Fatalf("unexpected server.addr: %q", got.ServerAddr)
|
t.Fatalf("unexpected server.addr: %q", got.ServerAddr)
|
||||||
}
|
}
|
||||||
|
if got.ArtifactRoot != filepath.Clean("./artifacts") {
|
||||||
|
t.Fatalf("unexpected server.artifact_root: %q", got.ArtifactRoot)
|
||||||
|
}
|
||||||
|
if got.MaxRequestBytes != 1024 {
|
||||||
|
t.Fatalf("unexpected server.max_request_bytes: %d", got.MaxRequestBytes)
|
||||||
|
}
|
||||||
|
if got.MaxArtifactBytes != 2048 {
|
||||||
|
t.Fatalf("unexpected server.max_artifact_bytes: %d", got.MaxArtifactBytes)
|
||||||
|
}
|
||||||
|
if got.MaxResponseBytes != 4096 {
|
||||||
|
t.Fatalf("unexpected server.max_response_bytes: %d", got.MaxResponseBytes)
|
||||||
|
}
|
||||||
if got.DefaultRenderFormat != renderformat.PreparedRunFormatJSON {
|
if got.DefaultRenderFormat != renderformat.PreparedRunFormatJSON {
|
||||||
t.Fatalf("unexpected defaults.render_format: %q", got.DefaultRenderFormat)
|
t.Fatalf("unexpected defaults.render_format: %q", got.DefaultRenderFormat)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestLoadConfigAcceptsZeroHTTPSizeLimits(t *testing.T) {
|
||||||
|
path := writeConfigFile(t, "config.yml", `
|
||||||
|
server:
|
||||||
|
max_request_bytes: 0
|
||||||
|
max_artifact_bytes: 0
|
||||||
|
max_response_bytes: 0
|
||||||
|
`)
|
||||||
|
|
||||||
|
got, err := LoadConfig(path, true)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if got.MaxRequestBytes != 0 || got.MaxArtifactBytes != 0 || got.MaxResponseBytes != 0 {
|
||||||
|
t.Fatalf("expected zero limits to be preserved, got request=%d artifact=%d response=%d", got.MaxRequestBytes, got.MaxArtifactBytes, got.MaxResponseBytes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadConfigRejectsNegativeHTTPSizeLimits(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
body string
|
||||||
|
}{
|
||||||
|
{name: "request", body: "server:\n max_request_bytes: -1\n"},
|
||||||
|
{name: "artifact", body: "server:\n max_artifact_bytes: -1\n"},
|
||||||
|
{name: "response", body: "server:\n max_response_bytes: -1\n"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
path := writeConfigFile(t, "config.yml", tc.body)
|
||||||
|
_, err := LoadConfig(path, true)
|
||||||
|
if !errors.Is(err, ErrInvalidConfig) {
|
||||||
|
t.Fatalf("expected ErrInvalidConfig, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestLoadConfigEmptyFileResolvesToBuiltInDefaults(t *testing.T) {
|
func TestLoadConfigEmptyFileResolvesToBuiltInDefaults(t *testing.T) {
|
||||||
path := writeConfigFile(t, "config.yml", "")
|
path := writeConfigFile(t, "config.yml", "")
|
||||||
|
|
||||||
@@ -185,15 +254,26 @@ func TestApplyCLIOverridesAppliesPrecedence(t *testing.T) {
|
|||||||
ProfileDir: "/from/config/profiles",
|
ProfileDir: "/from/config/profiles",
|
||||||
SchemaDir: "/from/config/schemas",
|
SchemaDir: "/from/config/schemas",
|
||||||
ServerAddr: ":1234",
|
ServerAddr: ":1234",
|
||||||
|
ArtifactRoot: "/from/config/artifacts",
|
||||||
|
MaxRequestBytes: 111,
|
||||||
|
MaxArtifactBytes: 222,
|
||||||
|
MaxResponseBytes: 333,
|
||||||
DefaultRenderFormat: renderformat.PreparedRunFormatJSON,
|
DefaultRenderFormat: renderformat.PreparedRunFormatJSON,
|
||||||
}
|
}
|
||||||
|
maxRequestBytes := int64(0)
|
||||||
|
maxArtifactBytes := int64(444)
|
||||||
|
maxResponseBytes := int64(555)
|
||||||
|
|
||||||
got, err := ApplyCLIOverrides(base, CLIOverrides{
|
got, err := ApplyCLIOverrides(base, CLIOverrides{
|
||||||
PromptDir: "./prompts-cli",
|
PromptDir: "./prompts-cli",
|
||||||
ProfileDir: "./profiles-cli",
|
ProfileDir: "./profiles-cli",
|
||||||
SchemaDir: "./schemas-cli",
|
SchemaDir: "./schemas-cli",
|
||||||
ServerAddr: ":8081",
|
ServerAddr: ":8081",
|
||||||
RenderFormat: "text",
|
ArtifactRoot: "./artifacts-cli",
|
||||||
|
MaxRequestBytes: &maxRequestBytes,
|
||||||
|
MaxArtifactBytes: &maxArtifactBytes,
|
||||||
|
MaxResponseBytes: &maxResponseBytes,
|
||||||
|
RenderFormat: "text",
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("expected no error, got %v", err)
|
t.Fatalf("expected no error, got %v", err)
|
||||||
@@ -211,11 +291,45 @@ func TestApplyCLIOverridesAppliesPrecedence(t *testing.T) {
|
|||||||
if got.ServerAddr != ":8081" {
|
if got.ServerAddr != ":8081" {
|
||||||
t.Fatalf("unexpected server addr: %q", got.ServerAddr)
|
t.Fatalf("unexpected server addr: %q", got.ServerAddr)
|
||||||
}
|
}
|
||||||
|
if got.ArtifactRoot != filepath.Clean("./artifacts-cli") {
|
||||||
|
t.Fatalf("unexpected artifact root: %q", got.ArtifactRoot)
|
||||||
|
}
|
||||||
|
if got.MaxRequestBytes != 0 {
|
||||||
|
t.Fatalf("unexpected max request bytes: %d", got.MaxRequestBytes)
|
||||||
|
}
|
||||||
|
if got.MaxArtifactBytes != 444 {
|
||||||
|
t.Fatalf("unexpected max artifact bytes: %d", got.MaxArtifactBytes)
|
||||||
|
}
|
||||||
|
if got.MaxResponseBytes != 555 {
|
||||||
|
t.Fatalf("unexpected max response bytes: %d", got.MaxResponseBytes)
|
||||||
|
}
|
||||||
if got.DefaultRenderFormat != renderformat.PreparedRunFormatText {
|
if got.DefaultRenderFormat != renderformat.PreparedRunFormatText {
|
||||||
t.Fatalf("unexpected render format: %q", got.DefaultRenderFormat)
|
t.Fatalf("unexpected render format: %q", got.DefaultRenderFormat)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestApplyCLIOverridesRejectsNegativeHTTPSizeLimits(t *testing.T) {
|
||||||
|
negative := int64(-1)
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
overrides CLIOverrides
|
||||||
|
}{
|
||||||
|
{name: "request", overrides: CLIOverrides{MaxRequestBytes: &negative}},
|
||||||
|
{name: "artifact", overrides: CLIOverrides{MaxArtifactBytes: &negative}},
|
||||||
|
{name: "response", overrides: CLIOverrides{MaxResponseBytes: &negative}},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
_, err := ApplyCLIOverrides(BuiltInDefaults(), tc.overrides)
|
||||||
|
if !errors.Is(err, ErrInvalidConfig) {
|
||||||
|
t.Fatalf("expected ErrInvalidConfig, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestApplyCLIOverridesInvalidRenderFormatReturnsError(t *testing.T) {
|
func TestApplyCLIOverridesInvalidRenderFormatReturnsError(t *testing.T) {
|
||||||
_, err := ApplyCLIOverrides(BuiltInDefaults(), CLIOverrides{RenderFormat: "yaml"})
|
_, err := ApplyCLIOverrides(BuiltInDefaults(), CLIOverrides{RenderFormat: "yaml"})
|
||||||
if err == nil {
|
if err == nil {
|
||||||
|
|||||||
@@ -7,13 +7,16 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
HTTPAddrDefault = ":8080"
|
HTTPAddrDefault = ":8080"
|
||||||
SchemaDirDefault = "."
|
SchemaDirDefault = "."
|
||||||
OutputArtifactName = "output"
|
OutputArtifactName = "output"
|
||||||
ContentTypeTextPlain = "text/plain"
|
ContentTypeTextPlain = "text/plain"
|
||||||
ContentTypeTextMarkdown = "text/markdown"
|
ContentTypeTextMarkdown = "text/markdown"
|
||||||
ContentTypeApplicationJSON = "application/json"
|
ContentTypeApplicationJSON = "application/json"
|
||||||
OpenAIChatCompletionsPath = "/chat/completions"
|
OpenAIChatCompletionsPath = "/chat/completions"
|
||||||
|
HTTPMaxRequestBytesDefault = 16 * 1024 * 1024
|
||||||
|
HTTPMaxArtifactBytesDefault = 16 * 1024 * 1024
|
||||||
|
HTTPMaxResponseBytesDefault = 16 * 1024 * 1024
|
||||||
|
|
||||||
ExecutionDefaultTemperature = 0.0
|
ExecutionDefaultTemperature = 0.0
|
||||||
ExecutionDefaultMaxTokens = 0
|
ExecutionDefaultMaxTokens = 0
|
||||||
|
|||||||
@@ -10,7 +10,6 @@ type ArtifactRefType string
|
|||||||
const (
|
const (
|
||||||
ArtifactRefInline ArtifactRefType = "inline"
|
ArtifactRefInline ArtifactRefType = "inline"
|
||||||
ArtifactRefFile ArtifactRefType = "file"
|
ArtifactRefFile ArtifactRefType = "file"
|
||||||
ArtifactRefS3 ArtifactRefType = "s3"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// OutputFormat defines the desired format of the generated artifact.
|
// OutputFormat defines the desired format of the generated artifact.
|
||||||
@@ -41,14 +40,33 @@ const (
|
|||||||
ValidationSkipped ValidationStatus = "skipped"
|
ValidationSkipped ValidationStatus = "skipped"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// CacheControlType defines provider cache behavior for prompt content.
|
||||||
|
type CacheControlType string
|
||||||
|
|
||||||
|
const (
|
||||||
|
CacheControlEphemeral CacheControlType = "ephemeral"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
// SessionIDMaxLength is OpenRouter's documented maximum session_id length.
|
||||||
|
SessionIDMaxLength = 256
|
||||||
|
)
|
||||||
|
|
||||||
|
// CacheControl describes provider cache metadata attached to prompt content.
|
||||||
|
type CacheControl struct {
|
||||||
|
Type CacheControlType `yaml:"type" json:"type"`
|
||||||
|
TTL string `yaml:"ttl,omitempty" json:"ttl,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
// RunRequest represents a request to generate a single artifact.
|
// RunRequest represents a request to generate a single artifact.
|
||||||
type RunRequest struct {
|
type RunRequest struct {
|
||||||
PromptID string
|
PromptID string
|
||||||
PromptVersion string
|
PromptVersion string
|
||||||
ProfileID string
|
ProfileID string
|
||||||
|
APIKey string `json:"-" yaml:"-"`
|
||||||
Inputs map[string]ArtifactRef
|
Inputs map[string]ArtifactRef
|
||||||
Vars map[string]string
|
Vars map[string]string
|
||||||
Execution *ExecutionTarget
|
Execution *ExecutionTargetOverride
|
||||||
Validation *OutputContract
|
Validation *OutputContract
|
||||||
Metadata map[string]string
|
Metadata map[string]string
|
||||||
}
|
}
|
||||||
@@ -72,25 +90,26 @@ type RunResult struct {
|
|||||||
StartTime time.Time
|
StartTime time.Time
|
||||||
EndTime time.Time
|
EndTime time.Time
|
||||||
Duration time.Duration
|
Duration time.Duration
|
||||||
Error error
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// PreparedRun contains pre-LLM execution state from the prepare/render phase.
|
// PreparedRun contains pre-LLM execution state from the prepare/render phase.
|
||||||
// It must never include resolved API key values, model output, or validation data.
|
// It must never include resolved API key values, model output, or validation data.
|
||||||
type PreparedRun struct {
|
type PreparedRun struct {
|
||||||
PromptID string `json:"prompt_id"`
|
PromptID string `json:"prompt_id"`
|
||||||
PromptVersion string `json:"prompt_version,omitempty"`
|
PromptVersion string `json:"prompt_version,omitempty"`
|
||||||
PromptHash string `json:"prompt_hash,omitempty"`
|
PromptHash string `json:"prompt_hash,omitempty"`
|
||||||
SelectedProfileID string `json:"selected_profile_id"`
|
SelectedProfileID string `json:"selected_profile_id"`
|
||||||
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
||||||
OutputContract OutputContract `json:"output_contract"`
|
TargetPresence ExecutionTargetPresence `json:"-"`
|
||||||
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
|
OutputContract OutputContract `json:"output_contract"`
|
||||||
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
|
||||||
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
||||||
Messages []RenderedMessage `json:"messages"`
|
SessionID string `json:"session_id,omitempty"`
|
||||||
StartTime time.Time `json:"start_time,omitempty"`
|
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
||||||
EndTime time.Time `json:"end_time,omitempty"`
|
Messages []RenderedMessage `json:"messages"`
|
||||||
DurationMS int64 `json:"duration_ms,omitempty"`
|
StartTime time.Time `json:"start_time,omitempty"`
|
||||||
|
EndTime time.Time `json:"end_time,omitempty"`
|
||||||
|
DurationMS int64 `json:"duration_ms,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// ArtifactRef represents a reference to an input artifact.
|
// ArtifactRef represents a reference to an input artifact.
|
||||||
@@ -116,6 +135,7 @@ type PromptDefinition struct {
|
|||||||
Version string `yaml:"version"`
|
Version string `yaml:"version"`
|
||||||
DefaultProfile string `yaml:"default_profile"`
|
DefaultProfile string `yaml:"default_profile"`
|
||||||
Description string `yaml:"description"`
|
Description string `yaml:"description"`
|
||||||
|
SessionID string `yaml:"session_id" json:"session_id,omitempty"`
|
||||||
Inputs []PromptInput `yaml:"inputs"`
|
Inputs []PromptInput `yaml:"inputs"`
|
||||||
Templates []PromptMessageTemplate `yaml:"templates"`
|
Templates []PromptMessageTemplate `yaml:"templates"`
|
||||||
OutputFormat OutputFormat `yaml:"output_format"`
|
OutputFormat OutputFormat `yaml:"output_format"`
|
||||||
@@ -132,36 +152,65 @@ type PromptInput struct {
|
|||||||
|
|
||||||
// PromptMessageTemplate defines a template for a chat message.
|
// PromptMessageTemplate defines a template for a chat message.
|
||||||
type PromptMessageTemplate struct {
|
type PromptMessageTemplate struct {
|
||||||
Role string `yaml:"role"`
|
Role string `yaml:"role"`
|
||||||
Content string `yaml:"content"`
|
Content string `yaml:"content"`
|
||||||
ContentFile string `yaml:"content_file"`
|
ContentFile string `yaml:"content_file"`
|
||||||
|
CacheControl *CacheControl `yaml:"cache_control,omitempty" json:"cache_control,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// ExecutionProfile describes how and where to execute a model.
|
// ExecutionProfile describes how and where to execute a model.
|
||||||
type ExecutionProfile struct {
|
type ExecutionProfile struct {
|
||||||
ID string `yaml:"id"`
|
ID string `yaml:"id"`
|
||||||
Endpoint string `yaml:"endpoint"`
|
Endpoint string `yaml:"endpoint"`
|
||||||
Model string `yaml:"model"`
|
Model string `yaml:"model"`
|
||||||
Temperature float64 `yaml:"temperature"`
|
Temperature float64 `yaml:"temperature"`
|
||||||
MaxTokens int `yaml:"max_tokens"`
|
MaxTokens int `yaml:"max_tokens"`
|
||||||
TopP float64 `yaml:"top_p"`
|
TopP float64 `yaml:"top_p"`
|
||||||
TimeoutSeconds int `yaml:"timeout_seconds"`
|
TimeoutSeconds int `yaml:"timeout_seconds"`
|
||||||
ReasoningEffort string `yaml:"reasoning_effort"`
|
ServiceTier string `yaml:"service_tier"`
|
||||||
APIKeyEnv string `yaml:"api_key_env"`
|
ReasoningEffort string `yaml:"reasoning_effort"`
|
||||||
ExtraParams map[string]string `yaml:"extra_params"`
|
APIKeyEnv string `yaml:"api_key_env"`
|
||||||
|
APIKeyRequired bool `yaml:"-" json:"-"`
|
||||||
|
ExtraParams map[string]any `yaml:"extra_params"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// ExecutionTargetOverride represents per-request runtime setting overrides.
|
||||||
|
type ExecutionTargetOverride struct {
|
||||||
|
Endpoint string `json:"endpoint,omitempty"`
|
||||||
|
Model string `json:"model,omitempty"`
|
||||||
|
Temperature *float64 `json:"temperature,omitempty"`
|
||||||
|
MaxTokens *int `json:"max_tokens,omitempty"`
|
||||||
|
TopP *float64 `json:"top_p,omitempty"`
|
||||||
|
TimeoutSeconds *int `json:"timeout_seconds,omitempty"`
|
||||||
|
ServiceTier string `json:"service_tier,omitempty"`
|
||||||
|
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||||
|
APIKeyEnv string `json:"api_key_env,omitempty"`
|
||||||
|
ExtraParams map[string]any `json:"extra_params,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// ExecutionTargetPresence tracks which effective runtime fields came from an
|
||||||
|
// explicit request override even when the resolved value is a zero value.
|
||||||
|
type ExecutionTargetPresence struct {
|
||||||
|
Temperature bool
|
||||||
|
MaxTokens bool
|
||||||
|
TopP bool
|
||||||
|
TimeoutSeconds bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// ExecutionTarget represents effective model runtime settings for a run.
|
// ExecutionTarget represents effective model runtime settings for a run.
|
||||||
type ExecutionTarget struct {
|
type ExecutionTarget struct {
|
||||||
Endpoint string `yaml:"endpoint" json:"endpoint"`
|
Endpoint string `yaml:"endpoint" json:"endpoint"`
|
||||||
Model string `yaml:"model" json:"model"`
|
Model string `yaml:"model" json:"model"`
|
||||||
Temperature float64 `yaml:"temperature" json:"temperature"`
|
Temperature float64 `yaml:"temperature" json:"temperature"`
|
||||||
MaxTokens int `yaml:"max_tokens" json:"max_tokens"`
|
MaxTokens int `yaml:"max_tokens" json:"max_tokens"`
|
||||||
TopP float64 `yaml:"top_p" json:"top_p"`
|
TopP float64 `yaml:"top_p" json:"top_p"`
|
||||||
TimeoutSeconds int `yaml:"timeout_seconds" json:"timeout_seconds"`
|
TimeoutSeconds int `yaml:"timeout_seconds" json:"timeout_seconds"`
|
||||||
ReasoningEffort string `yaml:"reasoning_effort" json:"reasoning_effort"`
|
ServiceTier string `yaml:"service_tier" json:"service_tier"`
|
||||||
APIKeyEnv string `yaml:"api_key_env" json:"api_key_env"`
|
ReasoningEffort string `yaml:"reasoning_effort" json:"reasoning_effort"`
|
||||||
ExtraParams map[string]string `yaml:"extra_params" json:"extra_params"`
|
APIKeyEnv string `yaml:"api_key_env" json:"api_key_env"`
|
||||||
|
APIKey string `yaml:"-" json:"-"`
|
||||||
|
APIKeyRequired bool `yaml:"-" json:"-"`
|
||||||
|
ExtraParams map[string]any `yaml:"extra_params" json:"extra_params"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// OutputContract defines the requirements for the output artifact.
|
// OutputContract defines the requirements for the output artifact.
|
||||||
@@ -174,19 +223,22 @@ type OutputContract struct {
|
|||||||
|
|
||||||
// RenderedPrompt represents the prompt after template application.
|
// RenderedPrompt represents the prompt after template application.
|
||||||
type RenderedPrompt struct {
|
type RenderedPrompt struct {
|
||||||
Messages []RenderedMessage `json:"messages"`
|
SessionID string `json:"session_id,omitempty"`
|
||||||
|
Messages []RenderedMessage `json:"messages"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// RenderedMessage is a single message in a rendered prompt.
|
// RenderedMessage is a single message in a rendered prompt.
|
||||||
type RenderedMessage struct {
|
type RenderedMessage struct {
|
||||||
Role string `json:"role"`
|
Role string `json:"role"`
|
||||||
Content string `json:"content"`
|
Content string `json:"content"`
|
||||||
|
CacheControl *CacheControl `json:"cache_control,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// GenerateRequest is the internal request passed to the LLM client.
|
// GenerateRequest is the internal request passed to the LLM client.
|
||||||
type GenerateRequest struct {
|
type GenerateRequest struct {
|
||||||
Prompt RenderedPrompt
|
Prompt RenderedPrompt
|
||||||
Target ExecutionTarget
|
Target ExecutionTarget
|
||||||
|
TargetPresence ExecutionTargetPresence
|
||||||
StructuredOutput *StructuredOutputSpec
|
StructuredOutput *StructuredOutputSpec
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -221,6 +273,8 @@ type TokenUsage struct {
|
|||||||
PromptTokens int
|
PromptTokens int
|
||||||
CompletionTokens int
|
CompletionTokens int
|
||||||
TotalTokens int
|
TotalTokens int
|
||||||
|
CachedTokens int
|
||||||
|
CacheWriteTokens int
|
||||||
}
|
}
|
||||||
|
|
||||||
// ValidationResult represents the outcome of an output validation.
|
// ValidationResult represents the outcome of an output validation.
|
||||||
@@ -232,23 +286,3 @@ type ValidationResult struct {
|
|||||||
RepairAttempts int
|
RepairAttempts int
|
||||||
IsValid bool
|
IsValid bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// RunMetadata contains auditing information for a run.
|
|
||||||
type RunMetadata struct {
|
|
||||||
RunID string
|
|
||||||
PromptID string
|
|
||||||
PromptVersion string
|
|
||||||
PromptHash string
|
|
||||||
RenderedPromptHash string
|
|
||||||
SelectedProfileID string
|
|
||||||
InputHashes map[string]string
|
|
||||||
ModelEndpoint string
|
|
||||||
ModelName string
|
|
||||||
Params ExecutionTarget
|
|
||||||
Timestamp time.Time
|
|
||||||
Duration time.Duration
|
|
||||||
Usage TokenUsage
|
|
||||||
ValidationMode ValidationMode
|
|
||||||
ValidationStatus ValidationStatus
|
|
||||||
RepairAttempts int
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ func TestPreparedRunJSONDoesNotIncludeSecretValues(t *testing.T) {
|
|||||||
Endpoint: "http://llm/v1",
|
Endpoint: "http://llm/v1",
|
||||||
Model: "gpt-test",
|
Model: "gpt-test",
|
||||||
APIKeyEnv: envName,
|
APIKeyEnv: envName,
|
||||||
|
APIKey: secret,
|
||||||
},
|
},
|
||||||
InputHashes: map[string]string{"transcript": "hash-1"},
|
InputHashes: map[string]string{"transcript": "hash-1"},
|
||||||
RenderedPromptHash: "rendered-hash",
|
RenderedPromptHash: "rendered-hash",
|
||||||
@@ -53,3 +54,88 @@ func TestPreparedRunJSONDoesNotIncludeSecretValues(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestPreparedRunJSONIncludesMessageCacheControlOnlyWhenPresent(t *testing.T) {
|
||||||
|
prepared := PreparedRun{
|
||||||
|
PromptID: "prompt.id",
|
||||||
|
SelectedProfileID: "local-fast",
|
||||||
|
EffectiveModelParams: ExecutionTarget{
|
||||||
|
Endpoint: "http://llm/v1",
|
||||||
|
Model: "gpt-test",
|
||||||
|
},
|
||||||
|
RenderedPromptHash: "rendered-hash",
|
||||||
|
Messages: []RenderedMessage{
|
||||||
|
{
|
||||||
|
Role: "system",
|
||||||
|
Content: "You are helpful.",
|
||||||
|
CacheControl: &CacheControl{
|
||||||
|
Type: CacheControlEphemeral,
|
||||||
|
TTL: "1h",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{Role: "user", Content: "Summarize this."},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
b, err := json.Marshal(prepared)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("marshal failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var decoded struct {
|
||||||
|
Messages []map[string]any `json:"messages"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(b, &decoded); err != nil {
|
||||||
|
t.Fatalf("unmarshal failed: %v", err)
|
||||||
|
}
|
||||||
|
if len(decoded.Messages) != 2 {
|
||||||
|
t.Fatalf("expected 2 messages, got %d", len(decoded.Messages))
|
||||||
|
}
|
||||||
|
|
||||||
|
cacheControl, ok := decoded.Messages[0]["cache_control"].(map[string]any)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("expected cache_control on first message, got %#v", decoded.Messages[0])
|
||||||
|
}
|
||||||
|
if cacheControl["type"] != string(CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
|
||||||
|
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
|
||||||
|
}
|
||||||
|
if _, ok := decoded.Messages[1]["cache_control"]; ok {
|
||||||
|
t.Fatalf("expected second message to omit cache_control, got %#v", decoded.Messages[1])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPreparedRunJSONIncludesSessionIDOnlyWhenPresent(t *testing.T) {
|
||||||
|
prepared := PreparedRun{
|
||||||
|
PromptID: "prompt.id",
|
||||||
|
SelectedProfileID: "local-fast",
|
||||||
|
EffectiveModelParams: ExecutionTarget{
|
||||||
|
Endpoint: "http://llm/v1",
|
||||||
|
Model: "gpt-test",
|
||||||
|
},
|
||||||
|
SessionID: "session-123",
|
||||||
|
RenderedPromptHash: "rendered-hash",
|
||||||
|
Messages: []RenderedMessage{{Role: "user", Content: "Summarize this."}},
|
||||||
|
}
|
||||||
|
|
||||||
|
b, err := json.Marshal(prepared)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("marshal failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var decoded map[string]any
|
||||||
|
if err := json.Unmarshal(b, &decoded); err != nil {
|
||||||
|
t.Fatalf("unmarshal failed: %v", err)
|
||||||
|
}
|
||||||
|
if decoded["session_id"] != "session-123" {
|
||||||
|
t.Fatalf("expected session_id in prepared run JSON, got %#v", decoded["session_id"])
|
||||||
|
}
|
||||||
|
|
||||||
|
prepared.SessionID = ""
|
||||||
|
b, err = json.Marshal(prepared)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("marshal failed: %v", err)
|
||||||
|
}
|
||||||
|
if strings.Contains(string(b), "session_id") {
|
||||||
|
t.Fatalf("expected empty session_id to be omitted, got %s", b)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
142
internal/filecatalog/catalog.go
Normal file
142
internal/filecatalog/catalog.go
Normal file
@@ -0,0 +1,142 @@
|
|||||||
|
package filecatalog
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io/fs"
|
||||||
|
"os"
|
||||||
|
"path"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// FindYAMLFiles returns sorted full paths for .yaml and .yml files under root.
|
||||||
|
func FindYAMLFiles(ctx context.Context, root string) ([]string, error) {
|
||||||
|
var files []string
|
||||||
|
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err()
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
if d.IsDir() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if !IsYAMLFile(d.Name()) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
files = append(files, path)
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
sort.Strings(files)
|
||||||
|
return files, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// FindFSYAMLFiles returns sorted paths for .yaml and .yml files under root in fsys.
|
||||||
|
func FindFSYAMLFiles(ctx context.Context, fsys fs.FS, root string) ([]string, error) {
|
||||||
|
cleanRoot := CleanFSRoot(root)
|
||||||
|
var files []string
|
||||||
|
err := fs.WalkDir(fsys, cleanRoot, func(name string, d fs.DirEntry, err error) error {
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err()
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
if d.IsDir() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if !IsYAMLFile(d.Name()) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
files = append(files, name)
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
sort.Strings(files)
|
||||||
|
return files, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// RelativePath computes a clean relative path from root to path.
|
||||||
|
func RelativePath(root string, filePath string) string {
|
||||||
|
rel, err := filepath.Rel(root, filePath)
|
||||||
|
if err != nil {
|
||||||
|
return filepath.Clean(filePath)
|
||||||
|
}
|
||||||
|
return filepath.Clean(rel)
|
||||||
|
}
|
||||||
|
|
||||||
|
// CleanFSRoot normalizes a root path for use with fs.FS.
|
||||||
|
func CleanFSRoot(root string) string {
|
||||||
|
root = strings.TrimSpace(root)
|
||||||
|
if root == "" || root == "." {
|
||||||
|
return "."
|
||||||
|
}
|
||||||
|
return path.Clean(root)
|
||||||
|
}
|
||||||
|
|
||||||
|
// DisplayPath returns name relative to root for messages about fs.FS paths.
|
||||||
|
func DisplayPath(root string, name string) string {
|
||||||
|
cleanRoot := CleanFSRoot(root)
|
||||||
|
cleanName := path.Clean(name)
|
||||||
|
if cleanRoot == "." {
|
||||||
|
return cleanName
|
||||||
|
}
|
||||||
|
prefix := strings.TrimSuffix(cleanRoot, "/") + "/"
|
||||||
|
if strings.HasPrefix(cleanName, prefix) {
|
||||||
|
return strings.TrimPrefix(cleanName, prefix)
|
||||||
|
}
|
||||||
|
return cleanName
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResolveFSPath resolves userPath from baseDir and keeps it inside root.
|
||||||
|
func ResolveFSPath(root string, baseDir string, userPath string) (string, string, error) {
|
||||||
|
cleanRoot := CleanFSRoot(root)
|
||||||
|
cleanBase := path.Clean(strings.TrimSpace(baseDir))
|
||||||
|
if cleanBase == "" {
|
||||||
|
cleanBase = cleanRoot
|
||||||
|
}
|
||||||
|
if !containsFSPath(cleanRoot, cleanBase) {
|
||||||
|
return "", "", fmt.Errorf("base path %q is outside source root %q", cleanBase, cleanRoot)
|
||||||
|
}
|
||||||
|
|
||||||
|
cleanUserPath := strings.TrimSpace(userPath)
|
||||||
|
if cleanUserPath == "" {
|
||||||
|
return "", "", fmt.Errorf("path is required")
|
||||||
|
}
|
||||||
|
cleanUserPath = path.Clean(cleanUserPath)
|
||||||
|
if path.IsAbs(cleanUserPath) {
|
||||||
|
return "", "", fmt.Errorf("path %q must be relative", userPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
resolved := path.Clean(path.Join(cleanBase, cleanUserPath))
|
||||||
|
if !containsFSPath(cleanRoot, resolved) {
|
||||||
|
return "", "", fmt.Errorf("path %q escapes source root %q", userPath, cleanRoot)
|
||||||
|
}
|
||||||
|
return resolved, DisplayPath(cleanRoot, resolved), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func containsFSPath(root string, name string) bool {
|
||||||
|
root = CleanFSRoot(root)
|
||||||
|
name = path.Clean(name)
|
||||||
|
if root == "." {
|
||||||
|
return name == "." || (name != ".." && !strings.HasPrefix(name, "../"))
|
||||||
|
}
|
||||||
|
return name == root || strings.HasPrefix(name, strings.TrimSuffix(root, "/")+"/")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stem strips .yaml or .yml from a file name.
|
||||||
|
func Stem(name string) string {
|
||||||
|
name = strings.TrimSuffix(name, ".yaml")
|
||||||
|
name = strings.TrimSuffix(name, ".yml")
|
||||||
|
return name
|
||||||
|
}
|
||||||
|
|
||||||
|
func IsYAMLFile(name string) bool {
|
||||||
|
return strings.HasSuffix(name, ".yaml") || strings.HasSuffix(name, ".yml")
|
||||||
|
}
|
||||||
270
internal/filecatalog/catalog_test.go
Normal file
270
internal/filecatalog/catalog_test.go
Normal file
@@ -0,0 +1,270 @@
|
|||||||
|
package filecatalog
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"testing/fstest"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestFindYAMLFilesNestedSortedAndFiltered(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
mustWriteFile(t, filepath.Join(root, "z", "prompt.yml"), "id: z")
|
||||||
|
mustWriteFile(t, filepath.Join(root, "a", "profile.yaml"), "id: a")
|
||||||
|
mustWriteFile(t, filepath.Join(root, "a", "ignore.txt"), "not yaml")
|
||||||
|
mustWriteFile(t, filepath.Join(root, "b", "ignore.yaml.bak"), "not yaml")
|
||||||
|
|
||||||
|
got, err := FindYAMLFiles(context.Background(), root)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := []string{
|
||||||
|
filepath.Join(root, "a", "profile.yaml"),
|
||||||
|
filepath.Join(root, "z", "prompt.yml"),
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("expected sorted YAML files %v, got %v", want, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFindYAMLFilesHonorsContextCancellation(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
mustWriteFile(t, filepath.Join(root, "one.yaml"), "id: one")
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
|
||||||
|
_, err := FindYAMLFiles(ctx, root)
|
||||||
|
if !errors.Is(err, context.Canceled) {
|
||||||
|
t.Fatalf("expected context.Canceled, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFindFSYAMLFilesNestedSortedAndFiltered(t *testing.T) {
|
||||||
|
fsys := fstest.MapFS{
|
||||||
|
"prompts/z/prompt.yml": &fstest.MapFile{Data: []byte("id: z")},
|
||||||
|
"prompts/a/profile.yaml": &fstest.MapFile{Data: []byte("id: a")},
|
||||||
|
"prompts/a/ignore.txt": &fstest.MapFile{Data: []byte("not yaml")},
|
||||||
|
"prompts/b/ignore.yaml.bak": &fstest.MapFile{Data: []byte("not yaml")},
|
||||||
|
"other/ignored.yaml": &fstest.MapFile{Data: []byte("id: ignored")},
|
||||||
|
}
|
||||||
|
|
||||||
|
got, err := FindFSYAMLFiles(context.Background(), fsys, " prompts ")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := []string{
|
||||||
|
"prompts/a/profile.yaml",
|
||||||
|
"prompts/z/prompt.yml",
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("expected sorted YAML files %v, got %v", want, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFindFSYAMLFilesHonorsContextCancellation(t *testing.T) {
|
||||||
|
fsys := fstest.MapFS{
|
||||||
|
"one.yaml": &fstest.MapFile{Data: []byte("id: one")},
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
|
||||||
|
_, err := FindFSYAMLFiles(ctx, fsys, ".")
|
||||||
|
if !errors.Is(err, context.Canceled) {
|
||||||
|
t.Fatalf("expected context.Canceled, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRelativePathNested(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
path := filepath.Join(root, "nested", "profiles", "local.yaml")
|
||||||
|
got := RelativePath(root, path)
|
||||||
|
want := filepath.Join("nested", "profiles", "local.yaml")
|
||||||
|
if got != want {
|
||||||
|
t.Fatalf("expected relative path %q, got %q", want, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCleanFSRoot(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
root string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "empty", root: "", want: "."},
|
||||||
|
{name: "dot", root: ".", want: "."},
|
||||||
|
{name: "trimmed", root: " prompts/../profiles ", want: "profiles"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
if got := CleanFSRoot(tc.root); got != tc.want {
|
||||||
|
t.Fatalf("expected %q, got %q", tc.want, got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDisplayPath(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
root string
|
||||||
|
path string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "root dot", root: ".", path: "profiles/local.yaml", want: "profiles/local.yaml"},
|
||||||
|
{name: "nested root", root: "profiles", path: "profiles/local.yaml", want: "local.yaml"},
|
||||||
|
{name: "outside root", root: "profiles", path: "other/local.yaml", want: "other/local.yaml"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
if got := DisplayPath(tc.root, tc.path); got != tc.want {
|
||||||
|
t.Fatalf("expected %q, got %q", tc.want, got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveFSPath(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
root string
|
||||||
|
baseDir string
|
||||||
|
userPath string
|
||||||
|
wantPath string
|
||||||
|
wantDisplay string
|
||||||
|
wantErr string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "sibling inside root",
|
||||||
|
root: "prompts",
|
||||||
|
baseDir: "prompts/nested",
|
||||||
|
userPath: "./messages/user.tmpl",
|
||||||
|
wantPath: "prompts/nested/messages/user.tmpl",
|
||||||
|
wantDisplay: "nested/messages/user.tmpl",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "parent inside root",
|
||||||
|
root: "prompts",
|
||||||
|
baseDir: "prompts/nested",
|
||||||
|
userPath: "../shared/user.tmpl",
|
||||||
|
wantPath: "prompts/shared/user.tmpl",
|
||||||
|
wantDisplay: "shared/user.tmpl",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "escape rejected",
|
||||||
|
root: "prompts",
|
||||||
|
baseDir: "prompts/nested",
|
||||||
|
userPath: "../../outside.tmpl",
|
||||||
|
wantErr: "escapes source root",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "absolute path rejected",
|
||||||
|
root: "prompts",
|
||||||
|
baseDir: "prompts/nested",
|
||||||
|
userPath: "/outside.tmpl",
|
||||||
|
wantErr: "must be relative",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "empty path rejected",
|
||||||
|
root: "prompts",
|
||||||
|
baseDir: "prompts/nested",
|
||||||
|
userPath: " ",
|
||||||
|
wantErr: "path is required",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "dot root allows normal relative path",
|
||||||
|
root: ".",
|
||||||
|
baseDir: ".",
|
||||||
|
userPath: "schemas/events.schema.json",
|
||||||
|
wantPath: "schemas/events.schema.json",
|
||||||
|
wantDisplay: "schemas/events.schema.json",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "dot root rejects parent escape",
|
||||||
|
root: ".",
|
||||||
|
baseDir: ".",
|
||||||
|
userPath: "../outside.tmpl",
|
||||||
|
wantErr: "escapes source root",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
gotPath, gotDisplay, err := ResolveFSPath(tc.root, tc.baseDir, tc.userPath)
|
||||||
|
if tc.wantErr != "" {
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("expected error containing %q", tc.wantErr)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), tc.wantErr) {
|
||||||
|
t.Fatalf("expected error to contain %q, got %v", tc.wantErr, err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if gotPath != tc.wantPath || gotDisplay != tc.wantDisplay {
|
||||||
|
t.Fatalf("expected path/display %q/%q, got %q/%q", tc.wantPath, tc.wantDisplay, gotPath, gotDisplay)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStemStripsYAMLExtensions(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
in string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "yaml", in: "prompt.yaml", want: "prompt"},
|
||||||
|
{name: "yml", in: "profile.yml", want: "profile"},
|
||||||
|
{name: "other", in: "file.txt", want: "file.txt"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
if got := Stem(tc.in); got != tc.want {
|
||||||
|
t.Fatalf("expected %q, got %q", tc.want, got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIsYAMLFile(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
in string
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{name: "yaml", in: "prompt.yaml", want: true},
|
||||||
|
{name: "yml", in: "profile.yml", want: true},
|
||||||
|
{name: "backup", in: "profile.yaml.bak", want: false},
|
||||||
|
{name: "uppercase", in: "profile.YAML", want: false},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
if got := IsYAMLFile(tc.in); got != tc.want {
|
||||||
|
t.Fatalf("expected %v, got %v", tc.want, got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustWriteFile(t *testing.T, path string, content string) {
|
||||||
|
t.Helper()
|
||||||
|
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||||
|
t.Fatalf("failed to create directory: %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||||
|
t.Fatalf("failed to write file %q: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -96,6 +96,9 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
|
|||||||
if prepared.PromptHash != "" {
|
if prepared.PromptHash != "" {
|
||||||
fmt.Fprintf(&b, "prompt_hash: %s\n", prepared.PromptHash)
|
fmt.Fprintf(&b, "prompt_hash: %s\n", prepared.PromptHash)
|
||||||
}
|
}
|
||||||
|
if prepared.SessionID != "" {
|
||||||
|
fmt.Fprintf(&b, "session_id: %s\n", prepared.SessionID)
|
||||||
|
}
|
||||||
fmt.Fprintf(&b, "rendered_prompt_hash: %s\n", prepared.RenderedPromptHash)
|
fmt.Fprintf(&b, "rendered_prompt_hash: %s\n", prepared.RenderedPromptHash)
|
||||||
|
|
||||||
target := prepared.EffectiveModelParams
|
target := prepared.EffectiveModelParams
|
||||||
@@ -106,6 +109,9 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
|
|||||||
fmt.Fprintf(&b, " max_tokens: %d\n", target.MaxTokens)
|
fmt.Fprintf(&b, " max_tokens: %d\n", target.MaxTokens)
|
||||||
fmt.Fprintf(&b, " top_p: %g\n", target.TopP)
|
fmt.Fprintf(&b, " top_p: %g\n", target.TopP)
|
||||||
fmt.Fprintf(&b, " timeout_seconds: %d\n", target.TimeoutSeconds)
|
fmt.Fprintf(&b, " timeout_seconds: %d\n", target.TimeoutSeconds)
|
||||||
|
if target.ServiceTier != "" {
|
||||||
|
fmt.Fprintf(&b, " service_tier: %s\n", target.ServiceTier)
|
||||||
|
}
|
||||||
if target.ReasoningEffort != "" {
|
if target.ReasoningEffort != "" {
|
||||||
fmt.Fprintf(&b, " reasoning_effort: %s\n", target.ReasoningEffort)
|
fmt.Fprintf(&b, " reasoning_effort: %s\n", target.ReasoningEffort)
|
||||||
}
|
}
|
||||||
@@ -120,7 +126,11 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
|
|||||||
}
|
}
|
||||||
sort.Strings(keys)
|
sort.Strings(keys)
|
||||||
for _, k := range keys {
|
for _, k := range keys {
|
||||||
fmt.Fprintf(&b, " %s: %s\n", k, target.ExtraParams[k])
|
renderedValue, err := formatExtraParamTextValue(target.ExtraParams[k])
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to format extra_params.%s: %w", k, err)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(&b, " %s: %s\n", k, renderedValue)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -148,6 +158,13 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
|
|||||||
messages := byRole[role]
|
messages := byRole[role]
|
||||||
for i, msg := range messages {
|
for i, msg := range messages {
|
||||||
fmt.Fprintf(&b, " - message: %d\n", i+1)
|
fmt.Fprintf(&b, " - message: %d\n", i+1)
|
||||||
|
if msg.CacheControl != nil {
|
||||||
|
fmt.Fprintf(&b, " cache_control: %s", msg.CacheControl.Type)
|
||||||
|
if msg.CacheControl.TTL != "" {
|
||||||
|
fmt.Fprintf(&b, " ttl=%s", msg.CacheControl.TTL)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(&b)
|
||||||
|
}
|
||||||
fmt.Fprintln(&b, " content: |")
|
fmt.Fprintln(&b, " content: |")
|
||||||
content := msg.Content
|
content := msg.Content
|
||||||
if content == "" {
|
if content == "" {
|
||||||
@@ -162,3 +179,15 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
|
|||||||
|
|
||||||
return b.Bytes(), nil
|
return b.Bytes(), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func formatExtraParamTextValue(value any) (string, error) {
|
||||||
|
if s, ok := value.(string); ok {
|
||||||
|
return s, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
b, err := json.Marshal(value)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return string(b), nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -28,6 +28,7 @@ func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
|
|||||||
"max_tokens: 256",
|
"max_tokens: 256",
|
||||||
"top_p: 0.8",
|
"top_p: 0.8",
|
||||||
"timeout_seconds: 45",
|
"timeout_seconds: 45",
|
||||||
|
"service_tier: priority",
|
||||||
"reasoning_effort: medium",
|
"reasoning_effort: medium",
|
||||||
"api_key_env: SCRIPTORIUM_API_KEY",
|
"api_key_env: SCRIPTORIUM_API_KEY",
|
||||||
"prompt_hash: prompt-hash",
|
"prompt_hash: prompt-hash",
|
||||||
@@ -48,6 +49,36 @@ func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestTextFormatterRendersExtraParamsDeterministically(t *testing.T) {
|
||||||
|
prepared := samplePreparedRun()
|
||||||
|
prepared.EffectiveModelParams.ExtraParams = map[string]any{
|
||||||
|
"z_string": "enabled",
|
||||||
|
"b_number": 42,
|
||||||
|
"a_object": map[string]any{
|
||||||
|
"nested": "value",
|
||||||
|
"count": 2,
|
||||||
|
},
|
||||||
|
"c_array": []any{"first", 3, false},
|
||||||
|
}
|
||||||
|
|
||||||
|
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
s := string(out)
|
||||||
|
|
||||||
|
want := strings.Join([]string{
|
||||||
|
" extra_params:",
|
||||||
|
" a_object: {\"count\":2,\"nested\":\"value\"}",
|
||||||
|
" b_number: 42",
|
||||||
|
" c_array: [\"first\",3,false]",
|
||||||
|
" z_string: enabled",
|
||||||
|
}, "\n")
|
||||||
|
if !strings.Contains(s, want) {
|
||||||
|
t.Fatalf("expected deterministic extra_params block %q, got:\n%s", want, s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestTextFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
|
func TestTextFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
|
||||||
const secret = "super-secret-api-key"
|
const secret = "super-secret-api-key"
|
||||||
t.Setenv("SCRIPTORIUM_API_KEY", secret)
|
t.Setenv("SCRIPTORIUM_API_KEY", secret)
|
||||||
@@ -61,8 +92,94 @@ func TestTextFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestTextFormatterDoesNotIncludeDirectAPIKeyValue(t *testing.T) {
|
||||||
|
const directKey = "direct-format-key"
|
||||||
|
prepared := samplePreparedRun()
|
||||||
|
prepared.EffectiveModelParams.APIKey = directKey
|
||||||
|
|
||||||
|
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if strings.Contains(string(out), directKey) {
|
||||||
|
t.Fatalf("text output should not include direct api key value: %s", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTextFormatterIncludesMessageCacheControlBeforeContent(t *testing.T) {
|
||||||
|
prepared := samplePreparedRun()
|
||||||
|
prepared.Messages = []domain.RenderedMessage{
|
||||||
|
{
|
||||||
|
Role: "system",
|
||||||
|
Content: "System guidance.",
|
||||||
|
CacheControl: &domain.CacheControl{
|
||||||
|
Type: domain.CacheControlEphemeral,
|
||||||
|
TTL: "1h",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{Role: "user", Content: "Summarize the transcript."},
|
||||||
|
}
|
||||||
|
|
||||||
|
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
s := string(out)
|
||||||
|
if !strings.Contains(s, " system:\n - message: 1\n cache_control: ephemeral ttl=1h\n content: |") {
|
||||||
|
t.Fatalf("expected system message cache control before content, got:\n%s", s)
|
||||||
|
}
|
||||||
|
if strings.Count(s, "cache_control:") != 1 {
|
||||||
|
t.Fatalf("expected exactly one cache_control line, got:\n%s", s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTextFormatterIncludesSessionIDWhenPresent(t *testing.T) {
|
||||||
|
prepared := samplePreparedRun()
|
||||||
|
prepared.SessionID = "session-123"
|
||||||
|
|
||||||
|
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(out), "session_id: session-123\n") {
|
||||||
|
t.Fatalf("expected session_id in text output, got:\n%s", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTextFormatterOmitsEmptyCacheControlTTL(t *testing.T) {
|
||||||
|
prepared := samplePreparedRun()
|
||||||
|
prepared.Messages = []domain.RenderedMessage{
|
||||||
|
{
|
||||||
|
Role: "system",
|
||||||
|
Content: "System guidance.",
|
||||||
|
CacheControl: &domain.CacheControl{
|
||||||
|
Type: domain.CacheControlEphemeral,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
s := string(out)
|
||||||
|
if !strings.Contains(s, " cache_control: ephemeral\n") {
|
||||||
|
t.Fatalf("expected cache_control line without ttl, got:\n%s", s)
|
||||||
|
}
|
||||||
|
if strings.Contains(s, "ttl=") {
|
||||||
|
t.Fatalf("expected empty ttl to be omitted, got:\n%s", s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
|
func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
|
||||||
prepared := samplePreparedRun()
|
prepared := samplePreparedRun()
|
||||||
|
prepared.SessionID = "session-123"
|
||||||
|
prepared.EffectiveModelParams.ExtraParams = map[string]any{
|
||||||
|
"number": 42,
|
||||||
|
"nested": map[string]any{
|
||||||
|
"enabled": true,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
|
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -86,9 +203,24 @@ func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
|
|||||||
if decoded["rendered_prompt_hash"] != "rendered-hash" {
|
if decoded["rendered_prompt_hash"] != "rendered-hash" {
|
||||||
t.Fatalf("expected rendered_prompt_hash in json output, got %#v", decoded["rendered_prompt_hash"])
|
t.Fatalf("expected rendered_prompt_hash in json output, got %#v", decoded["rendered_prompt_hash"])
|
||||||
}
|
}
|
||||||
if _, ok := decoded["effective_model_params"]; !ok {
|
if decoded["session_id"] != "session-123" {
|
||||||
|
t.Fatalf("expected session_id in json output, got %#v", decoded["session_id"])
|
||||||
|
}
|
||||||
|
modelParams, ok := decoded["effective_model_params"].(map[string]any)
|
||||||
|
if !ok {
|
||||||
t.Fatalf("expected effective_model_params in json output, got %#v", decoded)
|
t.Fatalf("expected effective_model_params in json output, got %#v", decoded)
|
||||||
}
|
}
|
||||||
|
extraParams, ok := modelParams["extra_params"].(map[string]any)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("expected extra_params in json output, got %#v", modelParams["extra_params"])
|
||||||
|
}
|
||||||
|
if extraParams["number"] != float64(42) {
|
||||||
|
t.Fatalf("unexpected numeric extra param in json output: %#v", extraParams["number"])
|
||||||
|
}
|
||||||
|
nested, ok := extraParams["nested"].(map[string]any)
|
||||||
|
if !ok || nested["enabled"] != true {
|
||||||
|
t.Fatalf("unexpected nested extra param in json output: %#v", extraParams["nested"])
|
||||||
|
}
|
||||||
if _, ok := decoded["input_hashes"]; !ok {
|
if _, ok := decoded["input_hashes"]; !ok {
|
||||||
t.Fatalf("expected input_hashes in json output, got %#v", decoded)
|
t.Fatalf("expected input_hashes in json output, got %#v", decoded)
|
||||||
}
|
}
|
||||||
@@ -97,6 +229,47 @@ func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestJSONFormatterIncludesMessageCacheControlOnlyWhenPresent(t *testing.T) {
|
||||||
|
prepared := samplePreparedRun()
|
||||||
|
prepared.Messages = []domain.RenderedMessage{
|
||||||
|
{
|
||||||
|
Role: "system",
|
||||||
|
Content: "System guidance.",
|
||||||
|
CacheControl: &domain.CacheControl{
|
||||||
|
Type: domain.CacheControlEphemeral,
|
||||||
|
TTL: "1h",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{Role: "user", Content: "Summarize the transcript."},
|
||||||
|
}
|
||||||
|
|
||||||
|
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var decoded struct {
|
||||||
|
Messages []map[string]any `json:"messages"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(out, &decoded); err != nil {
|
||||||
|
t.Fatalf("expected valid json output, got %v", err)
|
||||||
|
}
|
||||||
|
if len(decoded.Messages) != 2 {
|
||||||
|
t.Fatalf("expected 2 messages, got %d", len(decoded.Messages))
|
||||||
|
}
|
||||||
|
|
||||||
|
cacheControl, ok := decoded.Messages[0]["cache_control"].(map[string]any)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("expected first message cache_control, got %#v", decoded.Messages[0])
|
||||||
|
}
|
||||||
|
if cacheControl["type"] != string(domain.CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
|
||||||
|
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
|
||||||
|
}
|
||||||
|
if _, ok := decoded.Messages[1]["cache_control"]; ok {
|
||||||
|
t.Fatalf("expected second message to omit cache_control, got %#v", decoded.Messages[1])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestJSONFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
|
func TestJSONFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
|
||||||
const secret = "super-secret-api-key"
|
const secret = "super-secret-api-key"
|
||||||
t.Setenv("SCRIPTORIUM_API_KEY", secret)
|
t.Setenv("SCRIPTORIUM_API_KEY", secret)
|
||||||
@@ -110,6 +283,20 @@ func TestJSONFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestJSONFormatterDoesNotIncludeDirectAPIKeyValue(t *testing.T) {
|
||||||
|
const directKey = "direct-format-key"
|
||||||
|
prepared := samplePreparedRun()
|
||||||
|
prepared.EffectiveModelParams.APIKey = directKey
|
||||||
|
|
||||||
|
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if strings.Contains(string(out), directKey) {
|
||||||
|
t.Fatalf("json output should not include direct api key value: %s", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestParsePreparedRunOutputFormatRecognizesSupportedNames(t *testing.T) {
|
func TestParsePreparedRunOutputFormatRecognizesSupportedNames(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -168,6 +355,7 @@ func samplePreparedRun() *domain.PreparedRun {
|
|||||||
MaxTokens: 256,
|
MaxTokens: 256,
|
||||||
TopP: 0.8,
|
TopP: 0.8,
|
||||||
TimeoutSeconds: 45,
|
TimeoutSeconds: 45,
|
||||||
|
ServiceTier: "priority",
|
||||||
ReasoningEffort: "medium",
|
ReasoningEffort: "medium",
|
||||||
APIKeyEnv: "SCRIPTORIUM_API_KEY",
|
APIKeyEnv: "SCRIPTORIUM_API_KEY",
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ import (
|
|||||||
"os"
|
"os"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
@@ -54,10 +55,11 @@ func NewOpenAICompatibleClient(cfg OpenAICompatibleConfig) (*OpenAICompatibleCli
|
|||||||
|
|
||||||
var client *http.Client
|
var client *http.Client
|
||||||
if cfg.HTTPClient != nil {
|
if cfg.HTTPClient != nil {
|
||||||
client = cfg.HTTPClient
|
cloned := *cfg.HTTPClient
|
||||||
if client.Timeout == 0 {
|
if cloned.Timeout == 0 {
|
||||||
client.Timeout = timeout
|
cloned.Timeout = timeout
|
||||||
}
|
}
|
||||||
|
client = &cloned
|
||||||
} else {
|
} else {
|
||||||
client = &http.Client{Timeout: timeout}
|
client = &http.Client{Timeout: timeout}
|
||||||
}
|
}
|
||||||
@@ -75,14 +77,6 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
|||||||
return nil, fmt.Errorf("%w: timeout_seconds must be greater than or equal to 0", ErrInvalidRequest)
|
return nil, fmt.Errorf("%w: timeout_seconds must be greater than or equal to 0", ErrInvalidRequest)
|
||||||
}
|
}
|
||||||
|
|
||||||
model := strings.TrimSpace(req.Target.Model)
|
|
||||||
if model == "" {
|
|
||||||
model = strings.TrimSpace(c.defaultModel)
|
|
||||||
}
|
|
||||||
if model == "" {
|
|
||||||
return nil, fmt.Errorf("%w: model is required", ErrInvalidRequest)
|
|
||||||
}
|
|
||||||
|
|
||||||
endpoint := strings.TrimSpace(req.Target.Endpoint)
|
endpoint := strings.TrimSpace(req.Target.Endpoint)
|
||||||
if endpoint == "" {
|
if endpoint == "" {
|
||||||
endpoint = c.baseURL
|
endpoint = c.baseURL
|
||||||
@@ -92,36 +86,17 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
|||||||
}
|
}
|
||||||
endpoint = strings.TrimRight(endpoint, "/") + defaults.OpenAIChatCompletionsPath
|
endpoint = strings.TrimRight(endpoint, "/") + defaults.OpenAIChatCompletionsPath
|
||||||
|
|
||||||
wireReq := openAIChatRequest{
|
wireReq, err := openAIChatRequestFromGenerateRequest(req, c.defaultModel)
|
||||||
Model: model,
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
wireReq.Messages = make([]openAIChatMessage, 0, len(req.Prompt.Messages))
|
wirePayload, err := openAIChatRequestPayload(wireReq)
|
||||||
for _, msg := range req.Prompt.Messages {
|
if err != nil {
|
||||||
wireReq.Messages = append(wireReq.Messages, openAIChatMessage{
|
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
|
||||||
Role: msg.Role,
|
|
||||||
Content: msg.Content,
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if req.Target.Temperature != 0 {
|
payload, err := json.Marshal(wirePayload)
|
||||||
wireReq.Temperature = &req.Target.Temperature
|
|
||||||
}
|
|
||||||
if req.Target.MaxTokens != 0 {
|
|
||||||
wireReq.MaxTokens = &req.Target.MaxTokens
|
|
||||||
}
|
|
||||||
if req.Target.TopP != 0 {
|
|
||||||
wireReq.TopP = &req.Target.TopP
|
|
||||||
}
|
|
||||||
if req.StructuredOutput != nil {
|
|
||||||
responseFormat, err := toOpenAIResponseFormat(req.StructuredOutput)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
|
|
||||||
}
|
|
||||||
wireReq.ResponseFormat = responseFormat
|
|
||||||
}
|
|
||||||
|
|
||||||
payload, err := json.Marshal(wireReq)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("%w: failed to encode request: %v", ErrRequestFailed, err)
|
return nil, fmt.Errorf("%w: failed to encode request: %v", ErrRequestFailed, err)
|
||||||
}
|
}
|
||||||
@@ -131,7 +106,9 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
|||||||
return nil, fmt.Errorf("%w: failed to create request: %v", ErrRequestFailed, err)
|
return nil, fmt.Errorf("%w: failed to create request: %v", ErrRequestFailed, err)
|
||||||
}
|
}
|
||||||
httpReq.Header.Set("Content-Type", "application/json")
|
httpReq.Header.Set("Content-Type", "application/json")
|
||||||
if envName := strings.TrimSpace(req.Target.APIKeyEnv); envName != "" {
|
if apiKey := strings.TrimSpace(req.Target.APIKey); apiKey != "" {
|
||||||
|
httpReq.Header.Set("Authorization", "Bearer "+apiKey)
|
||||||
|
} else if envName := strings.TrimSpace(req.Target.APIKeyEnv); envName != "" {
|
||||||
apiKey := strings.TrimSpace(os.Getenv(envName))
|
apiKey := strings.TrimSpace(os.Getenv(envName))
|
||||||
if apiKey == "" {
|
if apiKey == "" {
|
||||||
return nil, fmt.Errorf("%w: api key environment variable %q is not set", ErrInvalidRequest, envName)
|
return nil, fmt.Errorf("%w: api key environment variable %q is not set", ErrInvalidRequest, envName)
|
||||||
@@ -142,6 +119,8 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
|||||||
effectiveTimeout := c.timeout
|
effectiveTimeout := c.timeout
|
||||||
if req.Target.TimeoutSeconds > 0 {
|
if req.Target.TimeoutSeconds > 0 {
|
||||||
effectiveTimeout = time.Duration(req.Target.TimeoutSeconds) * time.Second
|
effectiveTimeout = time.Duration(req.Target.TimeoutSeconds) * time.Second
|
||||||
|
} else if req.TargetPresence.TimeoutSeconds {
|
||||||
|
effectiveTimeout = 0
|
||||||
}
|
}
|
||||||
|
|
||||||
httpClient := c.httpClient
|
httpClient := c.httpClient
|
||||||
@@ -160,8 +139,8 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
|||||||
defer httpResp.Body.Close()
|
defer httpResp.Body.Close()
|
||||||
|
|
||||||
if httpResp.StatusCode < 200 || httpResp.StatusCode >= 300 {
|
if httpResp.StatusCode < 200 || httpResp.StatusCode >= 300 {
|
||||||
body, _ := io.ReadAll(io.LimitReader(httpResp.Body, 4096))
|
_, _ = io.Copy(io.Discard, io.LimitReader(httpResp.Body, 4096))
|
||||||
return nil, fmt.Errorf("%w: status=%d body=%q", ErrUnexpectedStatus, httpResp.StatusCode, strings.TrimSpace(string(body)))
|
return nil, fmt.Errorf("%w: status=%d", ErrUnexpectedStatus, httpResp.StatusCode)
|
||||||
}
|
}
|
||||||
|
|
||||||
var wireResp openAIChatResponse
|
var wireResp openAIChatResponse
|
||||||
@@ -183,32 +162,166 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
|||||||
PromptTokens: wireResp.Usage.PromptTokens,
|
PromptTokens: wireResp.Usage.PromptTokens,
|
||||||
CompletionTokens: wireResp.Usage.CompletionTokens,
|
CompletionTokens: wireResp.Usage.CompletionTokens,
|
||||||
TotalTokens: wireResp.Usage.TotalTokens,
|
TotalTokens: wireResp.Usage.TotalTokens,
|
||||||
|
CachedTokens: wireResp.Usage.PromptTokensDetails.CachedTokens,
|
||||||
|
CacheWriteTokens: wireResp.Usage.CacheWriteTokens,
|
||||||
},
|
},
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type openAIChatRequest struct {
|
func openAIChatRequestFromGenerateRequest(req domain.GenerateRequest, defaultModel string) (openAIChatRequest, error) {
|
||||||
Model string `json:"model"`
|
model := strings.TrimSpace(req.Target.Model)
|
||||||
Messages []openAIChatMessage `json:"messages"`
|
if model == "" {
|
||||||
Temperature *float64 `json:"temperature,omitempty"`
|
model = strings.TrimSpace(defaultModel)
|
||||||
MaxTokens *int `json:"max_tokens,omitempty"`
|
}
|
||||||
TopP *float64 `json:"top_p,omitempty"`
|
if model == "" {
|
||||||
ResponseFormat *openAIResponseFormat `json:"response_format,omitempty"`
|
return openAIChatRequest{}, errors.New("model is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
wireReq := openAIChatRequest{
|
||||||
|
Model: model,
|
||||||
|
}
|
||||||
|
if sessionID := strings.TrimSpace(req.Prompt.SessionID); sessionID != "" {
|
||||||
|
if n := utf8.RuneCountInString(sessionID); n > domain.SessionIDMaxLength {
|
||||||
|
return openAIChatRequest{}, fmt.Errorf("session_id length %d exceeds maximum %d", n, domain.SessionIDMaxLength)
|
||||||
|
}
|
||||||
|
wireReq.SessionID = sessionID
|
||||||
|
}
|
||||||
|
|
||||||
|
wireReq.Messages = make([]openAIChatRequestMessage, 0, len(req.Prompt.Messages))
|
||||||
|
for _, msg := range req.Prompt.Messages {
|
||||||
|
wireReq.Messages = append(wireReq.Messages, openAIChatRequestMessageFromRenderedMessage(msg))
|
||||||
|
}
|
||||||
|
|
||||||
|
if req.Target.Temperature != 0 || req.TargetPresence.Temperature {
|
||||||
|
wireReq.Temperature = &req.Target.Temperature
|
||||||
|
}
|
||||||
|
if req.Target.MaxTokens != 0 || req.TargetPresence.MaxTokens {
|
||||||
|
wireReq.MaxTokens = &req.Target.MaxTokens
|
||||||
|
}
|
||||||
|
if req.Target.TopP != 0 || req.TargetPresence.TopP {
|
||||||
|
wireReq.TopP = &req.Target.TopP
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(req.Target.ServiceTier) != "" {
|
||||||
|
wireReq.ServiceTier = req.Target.ServiceTier
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(req.Target.ReasoningEffort) != "" {
|
||||||
|
wireReq.ReasoningEffort = req.Target.ReasoningEffort
|
||||||
|
}
|
||||||
|
if len(req.Target.ExtraParams) > 0 {
|
||||||
|
wireReq.ExtraParams = req.Target.ExtraParams
|
||||||
|
}
|
||||||
|
if req.StructuredOutput != nil {
|
||||||
|
responseFormat, err := toOpenAIResponseFormat(req.StructuredOutput)
|
||||||
|
if err != nil {
|
||||||
|
return openAIChatRequest{}, err
|
||||||
|
}
|
||||||
|
wireReq.ResponseFormat = responseFormat
|
||||||
|
}
|
||||||
|
|
||||||
|
return wireReq, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type openAIChatMessage struct {
|
type openAIChatRequest struct {
|
||||||
|
Model string `json:"model"`
|
||||||
|
SessionID string `json:"session_id,omitempty"`
|
||||||
|
Messages []openAIChatRequestMessage `json:"messages"`
|
||||||
|
Temperature *float64 `json:"temperature,omitempty"`
|
||||||
|
MaxTokens *int `json:"max_tokens,omitempty"`
|
||||||
|
TopP *float64 `json:"top_p,omitempty"`
|
||||||
|
ServiceTier string `json:"service_tier,omitempty"`
|
||||||
|
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||||
|
ResponseFormat *openAIResponseFormat `json:"response_format,omitempty"`
|
||||||
|
ExtraParams map[string]any `json:"-"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func openAIChatRequestPayload(req openAIChatRequest) (map[string]any, error) {
|
||||||
|
out := map[string]any{
|
||||||
|
"model": req.Model,
|
||||||
|
"messages": req.Messages,
|
||||||
|
}
|
||||||
|
if req.SessionID != "" {
|
||||||
|
out["session_id"] = req.SessionID
|
||||||
|
}
|
||||||
|
if req.Temperature != nil {
|
||||||
|
out["temperature"] = *req.Temperature
|
||||||
|
}
|
||||||
|
if req.MaxTokens != nil {
|
||||||
|
out["max_tokens"] = *req.MaxTokens
|
||||||
|
}
|
||||||
|
if req.TopP != nil {
|
||||||
|
out["top_p"] = *req.TopP
|
||||||
|
}
|
||||||
|
if req.ServiceTier != "" {
|
||||||
|
out["service_tier"] = req.ServiceTier
|
||||||
|
}
|
||||||
|
if req.ReasoningEffort != "" {
|
||||||
|
out["reasoning_effort"] = req.ReasoningEffort
|
||||||
|
}
|
||||||
|
if req.ResponseFormat != nil {
|
||||||
|
out["response_format"] = req.ResponseFormat
|
||||||
|
}
|
||||||
|
|
||||||
|
for key, value := range req.ExtraParams {
|
||||||
|
if key == "" {
|
||||||
|
return nil, errors.New("extra_params key must not be empty")
|
||||||
|
}
|
||||||
|
if _, reserved := reservedOpenAIChatRequestFields[key]; reserved {
|
||||||
|
return nil, fmt.Errorf("extra_params key %q collides with reserved request field", key)
|
||||||
|
}
|
||||||
|
if _, err := json.Marshal(value); err != nil {
|
||||||
|
return nil, fmt.Errorf("extra_params.%s must be JSON-serializable: %w", key, err)
|
||||||
|
}
|
||||||
|
out[key] = value
|
||||||
|
}
|
||||||
|
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var reservedOpenAIChatRequestFields = map[string]struct{}{
|
||||||
|
"model": {},
|
||||||
|
"session_id": {},
|
||||||
|
"messages": {},
|
||||||
|
"temperature": {},
|
||||||
|
"max_tokens": {},
|
||||||
|
"top_p": {},
|
||||||
|
"service_tier": {},
|
||||||
|
"reasoning_effort": {},
|
||||||
|
"response_format": {},
|
||||||
|
}
|
||||||
|
|
||||||
|
type openAIChatRequestMessage struct {
|
||||||
|
Role string `json:"role"`
|
||||||
|
Content any `json:"content"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type openAIChatTextContentBlock struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Text string `json:"text"`
|
||||||
|
CacheControl *openAICacheControl `json:"cache_control,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type openAICacheControl struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
TTL string `json:"ttl,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type openAIChatResponseMessage struct {
|
||||||
Role string `json:"role"`
|
Role string `json:"role"`
|
||||||
Content string `json:"content"`
|
Content string `json:"content"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type openAIChatResponse struct {
|
type openAIChatResponse struct {
|
||||||
Choices []struct {
|
Choices []struct {
|
||||||
Message openAIChatMessage `json:"message"`
|
Message openAIChatResponseMessage `json:"message"`
|
||||||
} `json:"choices"`
|
} `json:"choices"`
|
||||||
Usage struct {
|
Usage struct {
|
||||||
PromptTokens int `json:"prompt_tokens"`
|
PromptTokens int `json:"prompt_tokens"`
|
||||||
CompletionTokens int `json:"completion_tokens"`
|
CompletionTokens int `json:"completion_tokens"`
|
||||||
TotalTokens int `json:"total_tokens"`
|
TotalTokens int `json:"total_tokens"`
|
||||||
|
PromptTokensDetails struct {
|
||||||
|
CachedTokens int `json:"cached_tokens"`
|
||||||
|
} `json:"prompt_tokens_details"`
|
||||||
|
CacheWriteTokens int `json:"cache_write_tokens"`
|
||||||
} `json:"usage"`
|
} `json:"usage"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -223,6 +336,28 @@ type openAIJSONSchemaEnvelope struct {
|
|||||||
Schema any `json:"schema"`
|
Schema any `json:"schema"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func openAIChatRequestMessageFromRenderedMessage(msg domain.RenderedMessage) openAIChatRequestMessage {
|
||||||
|
wireMsg := openAIChatRequestMessage{
|
||||||
|
Role: msg.Role,
|
||||||
|
Content: msg.Content,
|
||||||
|
}
|
||||||
|
if msg.CacheControl == nil {
|
||||||
|
return wireMsg
|
||||||
|
}
|
||||||
|
|
||||||
|
wireMsg.Content = []openAIChatTextContentBlock{
|
||||||
|
{
|
||||||
|
Type: "text",
|
||||||
|
Text: msg.Content,
|
||||||
|
CacheControl: &openAICacheControl{
|
||||||
|
Type: string(msg.CacheControl.Type),
|
||||||
|
TTL: msg.CacheControl.TTL,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
return wireMsg
|
||||||
|
}
|
||||||
|
|
||||||
func toOpenAIResponseFormat(spec *domain.StructuredOutputSpec) (*openAIResponseFormat, error) {
|
func toOpenAIResponseFormat(spec *domain.StructuredOutputSpec) (*openAIResponseFormat, error) {
|
||||||
if spec == nil {
|
if spec == nil {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
|
"math"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -13,6 +14,64 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
func TestNewOpenAICompatibleClientDoesNotMutateSuppliedZeroTimeoutClient(t *testing.T) {
|
||||||
|
transport := http.DefaultTransport
|
||||||
|
supplied := &http.Client{Transport: transport}
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
|
||||||
|
HTTPClient: supplied,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected constructor error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if supplied.Timeout != 0 {
|
||||||
|
t.Fatalf("expected supplied client timeout to remain zero, got %v", supplied.Timeout)
|
||||||
|
}
|
||||||
|
if client.httpClient == supplied {
|
||||||
|
t.Fatal("expected constructed client to use a cloned HTTP client")
|
||||||
|
}
|
||||||
|
if client.httpClient.Timeout != client.timeout {
|
||||||
|
t.Fatalf("expected cloned client timeout %v, got %v", client.timeout, client.httpClient.Timeout)
|
||||||
|
}
|
||||||
|
if client.httpClient.Timeout <= 0 {
|
||||||
|
t.Fatalf("expected constructed client to use a positive default timeout, got %v", client.httpClient.Timeout)
|
||||||
|
}
|
||||||
|
if client.httpClient.Transport != transport {
|
||||||
|
t.Fatal("expected cloned client to preserve the supplied transport")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNewOpenAICompatibleClientDoesNotMutateSuppliedNonzeroTimeoutClient(t *testing.T) {
|
||||||
|
transport := http.DefaultTransport
|
||||||
|
suppliedTimeout := 37 * time.Second
|
||||||
|
supplied := &http.Client{
|
||||||
|
Timeout: suppliedTimeout,
|
||||||
|
Transport: transport,
|
||||||
|
}
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
|
||||||
|
Timeout: 2 * time.Second,
|
||||||
|
HTTPClient: supplied,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected constructor error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if supplied.Timeout != suppliedTimeout {
|
||||||
|
t.Fatalf("expected supplied client timeout to remain %v, got %v", suppliedTimeout, supplied.Timeout)
|
||||||
|
}
|
||||||
|
if client.httpClient == supplied {
|
||||||
|
t.Fatal("expected constructed client to use a cloned HTTP client")
|
||||||
|
}
|
||||||
|
if client.httpClient.Timeout != suppliedTimeout {
|
||||||
|
t.Fatalf("expected cloned client timeout %v, got %v", suppliedTimeout, client.httpClient.Timeout)
|
||||||
|
}
|
||||||
|
if client.httpClient.Transport != transport {
|
||||||
|
t.Fatal("expected cloned client to preserve the supplied transport")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
|
func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
|
||||||
type observedRequest struct {
|
type observedRequest struct {
|
||||||
Authorization string
|
Authorization string
|
||||||
@@ -61,6 +120,7 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
|
|||||||
Temperature: 0.4,
|
Temperature: 0.4,
|
||||||
MaxTokens: 123,
|
MaxTokens: 123,
|
||||||
TopP: 0.7,
|
TopP: 0.7,
|
||||||
|
ServiceTier: "priority",
|
||||||
APIKeyEnv: "SCRIPTORIUM_TEST_API_KEY",
|
APIKeyEnv: "SCRIPTORIUM_TEST_API_KEY",
|
||||||
},
|
},
|
||||||
StructuredOutput: &domain.StructuredOutputSpec{
|
StructuredOutput: &domain.StructuredOutputSpec{
|
||||||
@@ -88,6 +148,9 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
|
|||||||
if resp.Usage.PromptTokens != 11 || resp.Usage.CompletionTokens != 22 || resp.Usage.TotalTokens != 33 {
|
if resp.Usage.PromptTokens != 11 || resp.Usage.CompletionTokens != 22 || resp.Usage.TotalTokens != 33 {
|
||||||
t.Fatalf("unexpected usage: %+v", resp.Usage)
|
t.Fatalf("unexpected usage: %+v", resp.Usage)
|
||||||
}
|
}
|
||||||
|
if resp.Usage.CachedTokens != 0 || resp.Usage.CacheWriteTokens != 0 {
|
||||||
|
t.Fatalf("expected absent cache usage fields to remain zero, got %+v", resp.Usage)
|
||||||
|
}
|
||||||
|
|
||||||
if obs.Authorization != "Bearer secret-key" {
|
if obs.Authorization != "Bearer secret-key" {
|
||||||
t.Fatalf("unexpected Authorization header: %q", obs.Authorization)
|
t.Fatalf("unexpected Authorization header: %q", obs.Authorization)
|
||||||
@@ -95,6 +158,18 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
|
|||||||
if got, ok := obs.Body["model"].(string); !ok || got != "gpt-test" {
|
if got, ok := obs.Body["model"].(string); !ok || got != "gpt-test" {
|
||||||
t.Fatalf("unexpected model payload: %#v", obs.Body["model"])
|
t.Fatalf("unexpected model payload: %#v", obs.Body["model"])
|
||||||
}
|
}
|
||||||
|
if got, ok := obs.Body["temperature"].(float64); !ok || got != 0.4 {
|
||||||
|
t.Fatalf("unexpected temperature payload: %#v", obs.Body["temperature"])
|
||||||
|
}
|
||||||
|
if got, ok := obs.Body["max_tokens"].(float64); !ok || got != 123 {
|
||||||
|
t.Fatalf("unexpected max_tokens payload: %#v", obs.Body["max_tokens"])
|
||||||
|
}
|
||||||
|
if got, ok := obs.Body["top_p"].(float64); !ok || got != 0.7 {
|
||||||
|
t.Fatalf("unexpected top_p payload: %#v", obs.Body["top_p"])
|
||||||
|
}
|
||||||
|
if got, ok := obs.Body["service_tier"].(string); !ok || got != "priority" {
|
||||||
|
t.Fatalf("unexpected service_tier payload: %#v", obs.Body["service_tier"])
|
||||||
|
}
|
||||||
|
|
||||||
msgs, ok := obs.Body["messages"].([]any)
|
msgs, ok := obs.Body["messages"].([]any)
|
||||||
if !ok || len(msgs) != 2 {
|
if !ok || len(msgs) != 2 {
|
||||||
@@ -131,6 +206,277 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestOpenAICompatibleClientDirectAPIKeyPreferredOverEnv(t *testing.T) {
|
||||||
|
const directKey = "direct-llm-key"
|
||||||
|
t.Setenv("SCRIPTORIUM_TEST_API_KEY", "env-key")
|
||||||
|
|
||||||
|
var gotAuth string
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
gotAuth = r.Header.Get("Authorization")
|
||||||
|
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||||
|
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||||
|
Target: domain.ExecutionTarget{
|
||||||
|
Model: "model",
|
||||||
|
APIKeyEnv: "SCRIPTORIUM_TEST_API_KEY",
|
||||||
|
APIKey: directKey,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if gotAuth != "Bearer "+directKey {
|
||||||
|
t.Fatalf("unexpected Authorization header: %q", gotAuth)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOpenAICompatibleClientSerializesCacheControlledMessageAsContentBlock(t *testing.T) {
|
||||||
|
var observedBody map[string]any
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
defer r.Body.Close()
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||||
|
t.Fatalf("failed to decode request body: %v", err)
|
||||||
|
}
|
||||||
|
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||||
|
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{
|
||||||
|
{
|
||||||
|
Role: "system",
|
||||||
|
Content: "Stable instructions.",
|
||||||
|
CacheControl: &domain.CacheControl{
|
||||||
|
Type: domain.CacheControlEphemeral,
|
||||||
|
TTL: "1h",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{Role: "user", Content: "Dynamic request."},
|
||||||
|
}},
|
||||||
|
Target: domain.ExecutionTarget{Model: "model"},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, forbidden := range []string{"cache_control", "extra_params"} {
|
||||||
|
if _, exists := observedBody[forbidden]; exists {
|
||||||
|
t.Fatalf("expected top-level %s to be omitted, got %#v", forbidden, observedBody[forbidden])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
msgs, ok := observedBody["messages"].([]any)
|
||||||
|
if !ok || len(msgs) != 2 {
|
||||||
|
t.Fatalf("unexpected messages payload: %#v", observedBody["messages"])
|
||||||
|
}
|
||||||
|
msg0 := msgs[0].(map[string]any)
|
||||||
|
if msg0["role"] != "system" {
|
||||||
|
t.Fatalf("unexpected first message role: %#v", msg0["role"])
|
||||||
|
}
|
||||||
|
contentBlocks, ok := msg0["content"].([]any)
|
||||||
|
if !ok || len(contentBlocks) != 1 {
|
||||||
|
t.Fatalf("expected first message content block array, got %#v", msg0["content"])
|
||||||
|
}
|
||||||
|
block := contentBlocks[0].(map[string]any)
|
||||||
|
if block["type"] != "text" || block["text"] != "Stable instructions." {
|
||||||
|
t.Fatalf("unexpected text content block: %#v", block)
|
||||||
|
}
|
||||||
|
cacheControl, ok := block["cache_control"].(map[string]any)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("expected cache_control on content block, got %#v", block)
|
||||||
|
}
|
||||||
|
if cacheControl["type"] != string(domain.CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
|
||||||
|
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
|
||||||
|
}
|
||||||
|
|
||||||
|
msg1 := msgs[1].(map[string]any)
|
||||||
|
if msg1["role"] != "user" || msg1["content"] != "Dynamic request." {
|
||||||
|
t.Fatalf("expected uncached message to keep string content, got %#v", msg1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOpenAICompatibleClientOmitsEmptyCacheControlTTL(t *testing.T) {
|
||||||
|
var observedBody map[string]any
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
defer r.Body.Close()
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||||
|
t.Fatalf("failed to decode request body: %v", err)
|
||||||
|
}
|
||||||
|
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||||
|
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{
|
||||||
|
{
|
||||||
|
Role: "system",
|
||||||
|
Content: "Stable instructions.",
|
||||||
|
CacheControl: &domain.CacheControl{
|
||||||
|
Type: domain.CacheControlEphemeral,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}},
|
||||||
|
Target: domain.ExecutionTarget{Model: "model"},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
msgs := observedBody["messages"].([]any)
|
||||||
|
msg0 := msgs[0].(map[string]any)
|
||||||
|
contentBlocks := msg0["content"].([]any)
|
||||||
|
block := contentBlocks[0].(map[string]any)
|
||||||
|
cacheControl := block["cache_control"].(map[string]any)
|
||||||
|
if cacheControl["type"] != string(domain.CacheControlEphemeral) {
|
||||||
|
t.Fatalf("unexpected cache_control type: %#v", cacheControl)
|
||||||
|
}
|
||||||
|
if _, exists := cacheControl["ttl"]; exists {
|
||||||
|
t.Fatalf("expected empty ttl to be omitted, got %#v", cacheControl)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOpenAICompatibleClientSerializesSessionID(t *testing.T) {
|
||||||
|
var observedBody map[string]any
|
||||||
|
var observedSessionHeader string
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
observedSessionHeader = r.Header.Get("x-session-id")
|
||||||
|
defer r.Body.Close()
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||||
|
t.Fatalf("failed to decode request body: %v", err)
|
||||||
|
}
|
||||||
|
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||||
|
Prompt: domain.RenderedPrompt{
|
||||||
|
SessionID: " session-123 ",
|
||||||
|
Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}},
|
||||||
|
},
|
||||||
|
Target: domain.ExecutionTarget{Model: "model"},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if observedBody["session_id"] != "session-123" {
|
||||||
|
t.Fatalf("expected top-level session_id, got %#v", observedBody["session_id"])
|
||||||
|
}
|
||||||
|
if observedSessionHeader != "" {
|
||||||
|
t.Fatalf("did not expect x-session-id header, got %q", observedSessionHeader)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOpenAICompatibleClientOmitsEmptySessionID(t *testing.T) {
|
||||||
|
var observedBody map[string]any
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
defer r.Body.Close()
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||||
|
t.Fatalf("failed to decode request body: %v", err)
|
||||||
|
}
|
||||||
|
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||||
|
Prompt: domain.RenderedPrompt{
|
||||||
|
SessionID: " ",
|
||||||
|
Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}},
|
||||||
|
},
|
||||||
|
Target: domain.ExecutionTarget{Model: "model"},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if _, exists := observedBody["session_id"]; exists {
|
||||||
|
t.Fatalf("expected empty session_id to be omitted, got %#v", observedBody["session_id"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOpenAICompatibleClientRejectsTooLongSessionID(t *testing.T) {
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
|
||||||
|
BaseURL: "http://example.com/v1",
|
||||||
|
Model: "model",
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||||
|
Prompt: domain.RenderedPrompt{
|
||||||
|
SessionID: strings.Repeat("x", domain.SessionIDMaxLength+1),
|
||||||
|
Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected invalid request error")
|
||||||
|
}
|
||||||
|
if !errors.Is(err, ErrInvalidRequest) {
|
||||||
|
t.Fatalf("expected ErrInvalidRequest, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOpenAICompatibleClientParsesCacheUsage(t *testing.T) {
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
_, _ = w.Write([]byte(`{
|
||||||
|
"choices": [{"message": {"role": "assistant", "content": "ok"}}],
|
||||||
|
"usage": {
|
||||||
|
"prompt_tokens": 100,
|
||||||
|
"completion_tokens": 20,
|
||||||
|
"total_tokens": 120,
|
||||||
|
"prompt_tokens_details": {"cached_tokens": 80},
|
||||||
|
"cache_write_tokens": 60
|
||||||
|
}
|
||||||
|
}`))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1", Model: "model"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
resp, err := client.Generate(context.Background(), domain.GenerateRequest{
|
||||||
|
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if resp.Usage.PromptTokens != 100 || resp.Usage.CompletionTokens != 20 || resp.Usage.TotalTokens != 120 {
|
||||||
|
t.Fatalf("unexpected base usage fields: %+v", resp.Usage)
|
||||||
|
}
|
||||||
|
if resp.Usage.CachedTokens != 80 || resp.Usage.CacheWriteTokens != 60 {
|
||||||
|
t.Fatalf("unexpected cache usage fields: %+v", resp.Usage)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestOpenAICompatibleClientOmitsResponseFormatWhenNoStructuredOutput(t *testing.T) {
|
func TestOpenAICompatibleClientOmitsResponseFormatWhenNoStructuredOutput(t *testing.T) {
|
||||||
var observedBody map[string]any
|
var observedBody map[string]any
|
||||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
@@ -157,6 +503,284 @@ func TestOpenAICompatibleClientOmitsResponseFormatWhenNoStructuredOutput(t *test
|
|||||||
if _, exists := observedBody["response_format"]; exists {
|
if _, exists := observedBody["response_format"]; exists {
|
||||||
t.Fatalf("expected response_format omitted, got %#v", observedBody["response_format"])
|
t.Fatalf("expected response_format omitted, got %#v", observedBody["response_format"])
|
||||||
}
|
}
|
||||||
|
if _, exists := observedBody["service_tier"]; exists {
|
||||||
|
t.Fatalf("expected service_tier omitted, got %#v", observedBody["service_tier"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOpenAICompatibleClientSerializesReasoningEffortAndExtraParams(t *testing.T) {
|
||||||
|
var observedBody map[string]any
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
defer r.Body.Close()
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||||
|
t.Fatalf("failed to decode request body: %v", err)
|
||||||
|
}
|
||||||
|
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||||
|
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||||
|
Target: domain.ExecutionTarget{
|
||||||
|
Model: "model",
|
||||||
|
ReasoningEffort: "high",
|
||||||
|
ExtraParams: map[string]any{
|
||||||
|
"string_value": "on",
|
||||||
|
"number_value": 42,
|
||||||
|
"boolean_value": true,
|
||||||
|
"object_value": map[string]any{"nested": "value", "count": 2},
|
||||||
|
"array_value": []any{"first", 3, false},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if observedBody["reasoning_effort"] != "high" {
|
||||||
|
t.Fatalf("expected reasoning_effort high, got %#v", observedBody["reasoning_effort"])
|
||||||
|
}
|
||||||
|
if observedBody["string_value"] != "on" {
|
||||||
|
t.Fatalf("unexpected string extra param: %#v", observedBody["string_value"])
|
||||||
|
}
|
||||||
|
if observedBody["number_value"] != float64(42) {
|
||||||
|
t.Fatalf("unexpected number extra param: %#v", observedBody["number_value"])
|
||||||
|
}
|
||||||
|
if observedBody["boolean_value"] != true {
|
||||||
|
t.Fatalf("unexpected boolean extra param: %#v", observedBody["boolean_value"])
|
||||||
|
}
|
||||||
|
objectValue, ok := observedBody["object_value"].(map[string]any)
|
||||||
|
if !ok || objectValue["nested"] != "value" || objectValue["count"] != float64(2) {
|
||||||
|
t.Fatalf("unexpected object extra param: %#v", observedBody["object_value"])
|
||||||
|
}
|
||||||
|
if _, exists := observedBody["extra_params"]; exists {
|
||||||
|
t.Fatalf("expected extra_params wrapper omitted, got %#v", observedBody["extra_params"])
|
||||||
|
}
|
||||||
|
arrayValue, ok := observedBody["array_value"].([]any)
|
||||||
|
if !ok || len(arrayValue) != 3 || arrayValue[0] != "first" || arrayValue[1] != float64(3) || arrayValue[2] != false {
|
||||||
|
t.Fatalf("unexpected array extra param: %#v", observedBody["array_value"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOpenAICompatibleClientOmitsReasoningEffortWhenUnset(t *testing.T) {
|
||||||
|
var observedBody map[string]any
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
defer r.Body.Close()
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||||
|
t.Fatalf("failed to decode request body: %v", err)
|
||||||
|
}
|
||||||
|
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||||
|
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||||
|
Target: domain.ExecutionTarget{Model: "model"},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if _, exists := observedBody["reasoning_effort"]; exists {
|
||||||
|
t.Fatalf("expected reasoning_effort omitted, got %#v", observedBody["reasoning_effort"])
|
||||||
|
}
|
||||||
|
if _, exists := observedBody["extra_params"]; exists {
|
||||||
|
t.Fatalf("expected extra_params wrapper omitted, got %#v", observedBody["extra_params"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOpenAICompatibleClientSerializesExplicitZeroNumericOverrides(t *testing.T) {
|
||||||
|
var observedBody map[string]any
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
defer r.Body.Close()
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||||
|
t.Fatalf("failed to decode request body: %v", err)
|
||||||
|
}
|
||||||
|
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||||
|
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||||
|
Target: domain.ExecutionTarget{Model: "model"},
|
||||||
|
TargetPresence: domain.ExecutionTargetPresence{
|
||||||
|
Temperature: true,
|
||||||
|
MaxTokens: true,
|
||||||
|
TopP: true,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if observedBody["temperature"] != float64(0) {
|
||||||
|
t.Fatalf("expected explicit zero temperature, got %#v", observedBody["temperature"])
|
||||||
|
}
|
||||||
|
if observedBody["max_tokens"] != float64(0) {
|
||||||
|
t.Fatalf("expected explicit zero max_tokens, got %#v", observedBody["max_tokens"])
|
||||||
|
}
|
||||||
|
if observedBody["top_p"] != float64(0) {
|
||||||
|
t.Fatalf("expected explicit zero top_p, got %#v", observedBody["top_p"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOpenAICompatibleClientOmitsImplicitZeroNumericFields(t *testing.T) {
|
||||||
|
var observedBody map[string]any
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
defer r.Body.Close()
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||||
|
t.Fatalf("failed to decode request body: %v", err)
|
||||||
|
}
|
||||||
|
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||||
|
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||||
|
Target: domain.ExecutionTarget{Model: "model"},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
for _, field := range []string{"temperature", "max_tokens", "top_p"} {
|
||||||
|
if _, exists := observedBody[field]; exists {
|
||||||
|
t.Fatalf("expected implicit zero field %q to be omitted, got body %#v", field, observedBody)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOpenAICompatibleClientExplicitZeroTimeoutDisablesClientTimeout(t *testing.T) {
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
time.Sleep(20 * time.Millisecond)
|
||||||
|
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
|
||||||
|
BaseURL: ts.URL + "/v1",
|
||||||
|
Timeout: time.Nanosecond,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||||
|
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||||
|
Target: domain.ExecutionTarget{Model: "model", TimeoutSeconds: 0},
|
||||||
|
TargetPresence: domain.ExecutionTargetPresence{TimeoutSeconds: true},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected explicit zero timeout to disable client timeout, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOpenAICompatibleClientOmittedTimeoutUsesClientTimeout(t *testing.T) {
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
time.Sleep(20 * time.Millisecond)
|
||||||
|
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
|
||||||
|
BaseURL: ts.URL + "/v1",
|
||||||
|
Timeout: time.Nanosecond,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||||
|
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||||
|
Target: domain.ExecutionTarget{Model: "model", TimeoutSeconds: 0},
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected omitted timeout to use client timeout")
|
||||||
|
}
|
||||||
|
if !errors.Is(err, ErrRequestFailed) {
|
||||||
|
t.Fatalf("expected ErrRequestFailed, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOpenAICompatibleClientRejectsInvalidExtraParamsBeforeProviderCall(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
extraParams map[string]any
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "empty key", extraParams: map[string]any{"": "empty"}, want: "key must not be empty"},
|
||||||
|
{name: "unserializable value", extraParams: map[string]any{"bad": math.Inf(1)}, want: "JSON-serializable"},
|
||||||
|
}
|
||||||
|
for _, key := range []string{
|
||||||
|
"model",
|
||||||
|
"session_id",
|
||||||
|
"messages",
|
||||||
|
"temperature",
|
||||||
|
"max_tokens",
|
||||||
|
"top_p",
|
||||||
|
"service_tier",
|
||||||
|
"reasoning_effort",
|
||||||
|
"response_format",
|
||||||
|
} {
|
||||||
|
tests = append(tests, struct {
|
||||||
|
name string
|
||||||
|
extraParams map[string]any
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
name: "reserved key " + key,
|
||||||
|
extraParams: map[string]any{key: "collision"},
|
||||||
|
want: "reserved request field",
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
called := false
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
called = true
|
||||||
|
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||||
|
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||||
|
Target: domain.ExecutionTarget{Model: "model", ExtraParams: tc.extraParams},
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected invalid request error")
|
||||||
|
}
|
||||||
|
if !errors.Is(err, ErrInvalidRequest) {
|
||||||
|
t.Fatalf("expected ErrInvalidRequest, got %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), tc.want) {
|
||||||
|
t.Fatalf("expected error to contain %q, got %v", tc.want, err)
|
||||||
|
}
|
||||||
|
if called {
|
||||||
|
t.Fatal("provider should not be called for invalid extra_params")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestOpenAICompatibleClientNoAuthorizationHeaderWhenNoAPIKey(t *testing.T) {
|
func TestOpenAICompatibleClientNoAuthorizationHeaderWhenNoAPIKey(t *testing.T) {
|
||||||
@@ -279,9 +903,10 @@ func TestOpenAICompatibleClientEndpointOverride(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestOpenAICompatibleClientNon2xxError(t *testing.T) {
|
func TestOpenAICompatibleClientNon2xxError(t *testing.T) {
|
||||||
|
const sensitiveBody = `provider-secret-fragment request_payload_details`
|
||||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
w.WriteHeader(http.StatusBadRequest)
|
w.WriteHeader(http.StatusBadRequest)
|
||||||
_, _ = w.Write([]byte(`{"error":"bad request payload"}`))
|
_, _ = w.Write([]byte(`{"error":"` + sensitiveBody + `"}`))
|
||||||
}))
|
}))
|
||||||
defer ts.Close()
|
defer ts.Close()
|
||||||
|
|
||||||
@@ -299,8 +924,11 @@ func TestOpenAICompatibleClientNon2xxError(t *testing.T) {
|
|||||||
if !errors.Is(err, ErrUnexpectedStatus) {
|
if !errors.Is(err, ErrUnexpectedStatus) {
|
||||||
t.Fatalf("expected ErrUnexpectedStatus, got %v", err)
|
t.Fatalf("expected ErrUnexpectedStatus, got %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(err.Error(), "400") || !strings.Contains(err.Error(), "bad request payload") {
|
if !strings.Contains(err.Error(), "status=400") {
|
||||||
t.Fatalf("expected status/body details, got %v", err)
|
t.Fatalf("expected status detail, got %v", err)
|
||||||
|
}
|
||||||
|
if strings.Contains(err.Error(), sensitiveBody) {
|
||||||
|
t.Fatalf("expected provider response body to be redacted, got %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
9
internal/profile/builtin/assets/aion-labs/aion-2.yml
Normal file
9
internal/profile/builtin/assets/aion-labs/aion-2.yml
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
id: aion-2
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: aion-labs/aion-2.0
|
||||||
|
temperature: 0.72
|
||||||
|
reasoning_effort: high
|
||||||
|
top_p: 0.95
|
||||||
|
timeout_seconds: 180
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
id: claude-fable-latest
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: "~anthropic/claude-fable-latest"
|
||||||
|
reasoning_effort: high
|
||||||
|
timeout_seconds: 600
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
id: claude-haiku-latest
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: "~anthropic/claude-haiku-latest"
|
||||||
|
reasoning_effort: medium
|
||||||
|
timeout_seconds: 240
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
id: claude-opus-latest
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: "~anthropic/claude-opus-latest"
|
||||||
|
reasoning_effort: high
|
||||||
|
timeout_seconds: 240
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
id: claude-sonnet-latest
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: "~anthropic/claude-sonnet-latest"
|
||||||
|
reasoning_effort: high
|
||||||
|
timeout_seconds: 240
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
id: deepseek-3-2
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: deepseek/deepseek-v3.2
|
||||||
|
reasoning_effort: high
|
||||||
|
timeout_seconds: 180
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
id: deepseek-4-pro
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: deepseek/deepseek-v4-pro
|
||||||
|
reasoning_effort: high
|
||||||
|
timeout_seconds: 180
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
id: gemini-2-flash-lite
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: "google/gemini-2.5-flash-lite"
|
||||||
|
#temperature: 0.15
|
||||||
|
reasoning_effort: high
|
||||||
|
#top_p: 0.98
|
||||||
|
timeout_seconds: 240
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
id: gemini-2-flash
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: "google/gemini-2.5-flash"
|
||||||
|
#temperature: 0.15
|
||||||
|
reasoning_effort: high
|
||||||
|
#top_p: 0.98
|
||||||
|
timeout_seconds: 240
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
9
internal/profile/builtin/assets/google/gemini-2-pro.yml
Normal file
9
internal/profile/builtin/assets/google/gemini-2-pro.yml
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
id: gemini-2-pro
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: "google/gemini-2.5-pro"
|
||||||
|
#temperature: 0.15
|
||||||
|
reasoning_effort: high
|
||||||
|
#top_p: 0.98
|
||||||
|
timeout_seconds: 240
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
id: gemini-3-flash-lite
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: "google/gemini-3.1-flash-lite"
|
||||||
|
#temperature: 0.15
|
||||||
|
reasoning_effort: high
|
||||||
|
#top_p: 0.98
|
||||||
|
timeout_seconds: 240
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
id: gemini-flash-latest
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: "~google/gemini-flash-latest"
|
||||||
|
#temperature: 0.15
|
||||||
|
reasoning_effort: high
|
||||||
|
#top_p: 0.98
|
||||||
|
timeout_seconds: 240
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
id: gemini-pro-latest
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: "~google/gemini-pro-latest"
|
||||||
|
#temperature: 0.15
|
||||||
|
reasoning_effort: high
|
||||||
|
#top_p: 0.98
|
||||||
|
timeout_seconds: 240
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
9
internal/profile/builtin/assets/google/gemma-4-31b.yml
Normal file
9
internal/profile/builtin/assets/google/gemma-4-31b.yml
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
id: gemma-4-31b
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: google/gemma-4-31b-it:exacto
|
||||||
|
temperature: 0.15
|
||||||
|
reasoning_effort: high
|
||||||
|
top_p: 0.98
|
||||||
|
timeout_seconds: 240
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
9
internal/profile/builtin/assets/minimax/minimax-m2.yml
Normal file
9
internal/profile/builtin/assets/minimax/minimax-m2.yml
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
id: minimax-m2
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: minimax/minimax-m2.5
|
||||||
|
temperature: 0.5
|
||||||
|
reasoning_effort: high
|
||||||
|
top_p: 0.95
|
||||||
|
timeout_seconds: 180
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
9
internal/profile/builtin/assets/minimax/minimax-m3.yml
Normal file
9
internal/profile/builtin/assets/minimax/minimax-m3.yml
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
id: minimax-m3
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: minimax/minimax-m3
|
||||||
|
#temperature: 0.5
|
||||||
|
reasoning_effort: high
|
||||||
|
#top_p: 0.95
|
||||||
|
timeout_seconds: 180
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
id: mistral-large-2512
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: mistralai/mistral-large-2512
|
||||||
|
temperature: 0.15
|
||||||
|
top_p: 0.98
|
||||||
|
timeout_seconds: 180
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
id: mistral-medium-3-5
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: mistralai/mistral-medium-3-5
|
||||||
|
temperature: 0.15
|
||||||
|
reasoning_effort: high
|
||||||
|
top_p: 0.98
|
||||||
|
timeout_seconds: 180
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
id: mistral-small-3
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: mistralai/mistral-small-3.2-24b-instruct
|
||||||
|
temperature: 0.05
|
||||||
|
top_p: 1.0
|
||||||
|
timeout_seconds: 180
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
id: mistral-small-4
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: mistralai/mistral-small-2603
|
||||||
|
temperature: 0.1
|
||||||
|
reasoning_effort: high
|
||||||
|
top_p: 0.98
|
||||||
|
timeout_seconds: 180
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
id: nemotron-3-ultra
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: nvidia/nemotron-3-ultra-550b-a55b
|
||||||
|
reasoning_effort: high
|
||||||
|
timeout_seconds: 180
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
7
internal/profile/builtin/assets/openai/gpt-5-mini.yml
Normal file
7
internal/profile/builtin/assets/openai/gpt-5-mini.yml
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
id: gpt-5-mini
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: "openai/gpt-5.4-mini"
|
||||||
|
reasoning_effort: high
|
||||||
|
timeout_seconds: 240
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
7
internal/profile/builtin/assets/openai/gpt-5-nano.yml
Normal file
7
internal/profile/builtin/assets/openai/gpt-5-nano.yml
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
id: gpt-5-nano
|
||||||
|
endpoint: https://openrouter.ai/api/v1
|
||||||
|
model: "openai/gpt-5.4-nano"
|
||||||
|
reasoning_effort: high
|
||||||
|
timeout_seconds: 240
|
||||||
|
api_key_env: OPENROUTER_API_KEY
|
||||||
|
service_tier: flex
|
||||||
31
internal/profile/builtin/repository.go
Normal file
31
internal/profile/builtin/repository.go
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
package builtin
|
||||||
|
|
||||||
|
import (
|
||||||
|
"embed"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
|
||||||
|
)
|
||||||
|
|
||||||
|
const assetRoot = "assets"
|
||||||
|
|
||||||
|
//go:embed assets/**/*.yml
|
||||||
|
var assets embed.FS
|
||||||
|
|
||||||
|
func NewRepository() profile.Repository {
|
||||||
|
return profile.NewFSRepository(assets, assetRoot)
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewRepositoryWithPrimary(primary profile.Repository) profile.Repository {
|
||||||
|
if primary == nil {
|
||||||
|
return NewRepository()
|
||||||
|
}
|
||||||
|
return profile.NewOverlayRepository(primary, NewRepository())
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewRepositoryWithDirectory(dir string) profile.Repository {
|
||||||
|
if strings.TrimSpace(dir) == "" {
|
||||||
|
return NewRepository()
|
||||||
|
}
|
||||||
|
return NewRepositoryWithPrimary(profile.NewFilesystemRepository(dir))
|
||||||
|
}
|
||||||
127
internal/profile/builtin/repository_test.go
Normal file
127
internal/profile/builtin/repository_test.go
Normal file
@@ -0,0 +1,127 @@
|
|||||||
|
package builtin
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io/fs"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
|
||||||
|
"gopkg.in/yaml.v3"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestBuiltInProfilesValidateThroughRepository(t *testing.T) {
|
||||||
|
repo := NewRepository()
|
||||||
|
ids := loadBuiltInProfileIDs(t)
|
||||||
|
if len(ids) == 0 {
|
||||||
|
t.Fatal("expected built-in profiles")
|
||||||
|
}
|
||||||
|
|
||||||
|
for id := range ids {
|
||||||
|
t.Run(id, func(t *testing.T) {
|
||||||
|
p, err := repo.GetProfile(context.Background(), id)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected built-in profile %q to load, got %v", id, err)
|
||||||
|
}
|
||||||
|
if p.ID != id {
|
||||||
|
t.Fatalf("expected profile id %q, got %q", id, p.ID)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuiltInProfilesDoNotContainDuplicateIDsOrRawAPIKeys(t *testing.T) {
|
||||||
|
loadBuiltInProfileIDs(t)
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadBuiltInProfileIDs(t *testing.T) map[string]string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
ids := map[string]string{}
|
||||||
|
err := fs.WalkDir(assets, assetRoot, func(name string, d fs.DirEntry, err error) error {
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if d.IsDir() || !strings.HasSuffix(name, ".yml") {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := assets.ReadFile(name)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to read built-in profile %s: %v", name, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var raw map[string]any
|
||||||
|
if err := yaml.Unmarshal(data, &raw); err != nil {
|
||||||
|
t.Fatalf("failed to decode built-in profile %s: %v", name, err)
|
||||||
|
}
|
||||||
|
if _, ok := raw["api_key"]; ok {
|
||||||
|
t.Fatalf("built-in profile %s contains raw api_key", name)
|
||||||
|
}
|
||||||
|
id, ok := raw["id"].(string)
|
||||||
|
if !ok || strings.TrimSpace(id) == "" {
|
||||||
|
t.Fatalf("built-in profile %s has missing id", name)
|
||||||
|
}
|
||||||
|
if previous, ok := ids[id]; ok {
|
||||||
|
t.Fatalf("duplicate built-in profile id %q in %s and %s", id, previous, name)
|
||||||
|
}
|
||||||
|
ids[id] = name
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to walk built-in profiles: %v", err)
|
||||||
|
}
|
||||||
|
return ids
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRepositoryWithPrimaryUsesPrimaryBeforeBuiltIns(t *testing.T) {
|
||||||
|
repo := NewRepositoryWithPrimary(staticProfileRepo{
|
||||||
|
profiles: map[string]string{"mistral-small-3": "custom-model"},
|
||||||
|
})
|
||||||
|
|
||||||
|
p, err := repo.GetProfile(context.Background(), "mistral-small-3")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected profile to load, got %v", err)
|
||||||
|
}
|
||||||
|
if p.Model != "custom-model" {
|
||||||
|
t.Fatalf("expected primary profile to override built-in, got %+v", p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRepositoryWithPrimaryFallsBackToBuiltIns(t *testing.T) {
|
||||||
|
repo := NewRepositoryWithPrimary(staticProfileRepo{})
|
||||||
|
|
||||||
|
p, err := repo.GetProfile(context.Background(), "mistral-small-3")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected built-in profile to load, got %v", err)
|
||||||
|
}
|
||||||
|
if p.ID != "mistral-small-3" {
|
||||||
|
t.Fatalf("unexpected profile: %+v", p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRepositoryWithPrimaryDoesNotFallBackAfterPrimaryError(t *testing.T) {
|
||||||
|
repo := NewRepositoryWithPrimary(staticProfileRepo{err: profile.ErrInvalidProfile})
|
||||||
|
|
||||||
|
_, err := repo.GetProfile(context.Background(), "mistral-small-3")
|
||||||
|
if !errors.Is(err, profile.ErrInvalidProfile) {
|
||||||
|
t.Fatalf("expected primary error, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type staticProfileRepo struct {
|
||||||
|
profiles map[string]string
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r staticProfileRepo) GetProfile(_ context.Context, id string) (*domain.ExecutionProfile, error) {
|
||||||
|
if r.err != nil {
|
||||||
|
return nil, r.err
|
||||||
|
}
|
||||||
|
if model, ok := r.profiles[id]; ok {
|
||||||
|
return &domain.ExecutionProfile{ID: id, Endpoint: "http://primary/v1", Model: model}, nil
|
||||||
|
}
|
||||||
|
return nil, profile.ErrProfileNotFound
|
||||||
|
}
|
||||||
@@ -5,11 +5,13 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"io/fs"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/filecatalog"
|
||||||
"gopkg.in/yaml.v3"
|
"gopkg.in/yaml.v3"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -29,44 +31,89 @@ func NewFilesystemRepository(dir string) Repository {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (r *filesystemRepository) GetProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) {
|
func (r *filesystemRepository) GetProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) {
|
||||||
|
return loadProfile(ctx, os.DirFS(r.dir), ".", id)
|
||||||
|
}
|
||||||
|
|
||||||
|
type fsRepository struct {
|
||||||
|
fsys fs.FS
|
||||||
|
root string
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewFSRepository(fsys fs.FS, root string) Repository {
|
||||||
|
return &fsRepository{fsys: fsys, root: root}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *fsRepository) GetProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) {
|
||||||
|
return loadProfile(ctx, r.fsys, r.root, id)
|
||||||
|
}
|
||||||
|
|
||||||
|
type overlayRepository struct {
|
||||||
|
primary Repository
|
||||||
|
fallback Repository
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewOverlayRepository(primary, fallback Repository) Repository {
|
||||||
|
return &overlayRepository{primary: primary, fallback: fallback}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *overlayRepository) GetProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) {
|
||||||
|
if r.primary != nil {
|
||||||
|
prof, err := r.primary.GetProfile(ctx, id)
|
||||||
|
if err == nil {
|
||||||
|
return prof, nil
|
||||||
|
}
|
||||||
|
if !errors.Is(err, ErrProfileNotFound) {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if r.fallback == nil {
|
||||||
|
return nil, ErrProfileNotFound
|
||||||
|
}
|
||||||
|
return r.fallback.GetProfile(ctx, id)
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadProfile(ctx context.Context, fsys fs.FS, root string, id string) (*domain.ExecutionProfile, error) {
|
||||||
if strings.TrimSpace(id) == "" {
|
if strings.TrimSpace(id) == "" {
|
||||||
return nil, fmt.Errorf("%w: profile id is required", ErrInvalidProfile)
|
return nil, fmt.Errorf("%w: profile id is required", ErrInvalidProfile)
|
||||||
}
|
}
|
||||||
|
if fsys == nil {
|
||||||
|
return nil, fmt.Errorf("failed to read profile directory: filesystem is nil")
|
||||||
|
}
|
||||||
|
|
||||||
files, err := os.ReadDir(r.dir)
|
files, err := filecatalog.FindFSYAMLFiles(ctx, fsys, root)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to read profile directory: %w", err)
|
return nil, fmt.Errorf("failed to read profile directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, file := range files {
|
var matches []profileMatch
|
||||||
|
for _, fullPath := range files {
|
||||||
select {
|
select {
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
return nil, ctx.Err()
|
return nil, ctx.Err()
|
||||||
default:
|
default:
|
||||||
}
|
}
|
||||||
|
|
||||||
if file.IsDir() || (!strings.HasSuffix(file.Name(), ".yaml") && !strings.HasSuffix(file.Name(), ".yml")) {
|
relPath := filecatalog.DisplayPath(root, fullPath)
|
||||||
continue
|
fileMatch := filecatalog.Stem(path.Base(fullPath)) == id
|
||||||
}
|
data, err := fs.ReadFile(fsys, fullPath)
|
||||||
|
|
||||||
fullPath := filepath.Join(r.dir, file.Name())
|
|
||||||
data, err := os.ReadFile(fullPath)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to read profile file %s: %w", file.Name(), err)
|
return nil, fmt.Errorf("failed to read profile file %s: %w", relPath, err)
|
||||||
|
}
|
||||||
|
metadata := readProfileFileMetadata(data)
|
||||||
|
idMatch := fileMatch || metadata.id == id
|
||||||
|
if metadata.hasRawAPIKey {
|
||||||
|
if idMatch {
|
||||||
|
return nil, fmt.Errorf("%w: %s", ErrRawAPIKeyNotAllowed, relPath)
|
||||||
|
}
|
||||||
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
var prof domain.ExecutionProfile
|
var prof domain.ExecutionProfile
|
||||||
decoder := yaml.NewDecoder(bytes.NewReader(data))
|
decoder := yaml.NewDecoder(bytes.NewReader(data))
|
||||||
decoder.KnownFields(true)
|
decoder.KnownFields(true)
|
||||||
if err := decoder.Decode(&prof); err != nil {
|
if err := decoder.Decode(&prof); err != nil {
|
||||||
if strings.Contains(err.Error(), "field api_key not found") {
|
if idMatch {
|
||||||
if strings.TrimSuffix(strings.TrimSuffix(file.Name(), ".yaml"), ".yml") == id {
|
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
|
||||||
return nil, fmt.Errorf("%w: %s", ErrRawAPIKeyNotAllowed, file.Name())
|
|
||||||
}
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if strings.TrimSuffix(strings.TrimSuffix(file.Name(), ".yaml"), ".yml") == id {
|
|
||||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, file.Name(), err)
|
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -76,16 +123,68 @@ func (r *filesystemRepository) GetProfile(ctx context.Context, id string) (*doma
|
|||||||
}
|
}
|
||||||
if err := validateProfile(&prof); err != nil {
|
if err := validateProfile(&prof); err != nil {
|
||||||
if errors.Is(err, ErrRawAPIKeyNotAllowed) {
|
if errors.Is(err, ErrRawAPIKeyNotAllowed) {
|
||||||
return nil, fmt.Errorf("%w: %s", err, file.Name())
|
return nil, fmt.Errorf("%w: %s", err, relPath)
|
||||||
}
|
}
|
||||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, file.Name(), err)
|
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, relPath, err)
|
||||||
}
|
}
|
||||||
return &prof, nil
|
matches = append(matches, profileMatch{
|
||||||
|
profile: &prof,
|
||||||
|
path: relPath,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(matches) > 1 {
|
||||||
|
paths := make([]string, 0, len(matches))
|
||||||
|
for _, match := range matches {
|
||||||
|
paths = append(paths, match.path)
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("%w: duplicate execution profile id %q found in: %s", ErrInvalidProfile, id, strings.Join(paths, ", "))
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(matches) == 1 {
|
||||||
|
return matches[0].profile, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil, ErrProfileNotFound
|
return nil, ErrProfileNotFound
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type profileMatch struct {
|
||||||
|
profile *domain.ExecutionProfile
|
||||||
|
path string
|
||||||
|
}
|
||||||
|
|
||||||
|
type profileFileMetadata struct {
|
||||||
|
id string
|
||||||
|
hasRawAPIKey bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func readProfileFileMetadata(data []byte) profileFileMetadata {
|
||||||
|
var node yaml.Node
|
||||||
|
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&node); err != nil {
|
||||||
|
return profileFileMetadata{}
|
||||||
|
}
|
||||||
|
if node.Kind != yaml.DocumentNode || len(node.Content) == 0 {
|
||||||
|
return profileFileMetadata{}
|
||||||
|
}
|
||||||
|
mapping := node.Content[0]
|
||||||
|
if mapping.Kind != yaml.MappingNode {
|
||||||
|
return profileFileMetadata{}
|
||||||
|
}
|
||||||
|
|
||||||
|
var metadata profileFileMetadata
|
||||||
|
for i := 0; i+1 < len(mapping.Content); i += 2 {
|
||||||
|
key := mapping.Content[i]
|
||||||
|
value := mapping.Content[i+1]
|
||||||
|
switch key.Value {
|
||||||
|
case "id":
|
||||||
|
metadata.id = strings.TrimSpace(value.Value)
|
||||||
|
case "api_key":
|
||||||
|
metadata.hasRawAPIKey = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return metadata
|
||||||
|
}
|
||||||
|
|
||||||
func validateProfile(p *domain.ExecutionProfile) error {
|
func validateProfile(p *domain.ExecutionProfile) error {
|
||||||
if strings.TrimSpace(p.ID) == "" {
|
if strings.TrimSpace(p.ID) == "" {
|
||||||
return errors.New("id is required")
|
return errors.New("id is required")
|
||||||
|
|||||||
@@ -2,10 +2,15 @@ package profile
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
"testing/fstest"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestFilesystemRepository_GetProfile(t *testing.T) {
|
func TestFilesystemRepository_GetProfile(t *testing.T) {
|
||||||
@@ -58,6 +63,149 @@ func TestFilesystemRepository_GetProfile(t *testing.T) {
|
|||||||
if p.ReasoningEffort != "medium" {
|
if p.ReasoningEffort != "medium" {
|
||||||
t.Fatalf("unexpected reasoning_effort: %q", p.ReasoningEffort)
|
t.Fatalf("unexpected reasoning_effort: %q", p.ReasoningEffort)
|
||||||
}
|
}
|
||||||
|
if p.ServiceTier != "priority" {
|
||||||
|
t.Fatalf("unexpected service_tier: %q", p.ServiceTier)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("valid nested profile", func(t *testing.T) {
|
||||||
|
nestedDir := filepath.Join(tmpDir, "local")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writeProfileTestFile(t, filepath.Join(nestedDir, "nested-local.yaml"), `
|
||||||
|
id: nested-local
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: nested-model
|
||||||
|
temperature: 0.1
|
||||||
|
`)
|
||||||
|
|
||||||
|
p, err := repo.GetProfile(ctx, "nested-local")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if p.Model != "nested-model" {
|
||||||
|
t.Fatalf("unexpected model: %q", p.Model)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("valid profile with JSON-compatible extra params", func(t *testing.T) {
|
||||||
|
writeProfileTestFile(t, filepath.Join(tmpDir, "json-extra-params.yaml"), `
|
||||||
|
id: json-extra-params
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: nested-model
|
||||||
|
extra_params:
|
||||||
|
string_value: enabled
|
||||||
|
number_value: 42
|
||||||
|
boolean_value: true
|
||||||
|
object_value:
|
||||||
|
nested: value
|
||||||
|
count: 2
|
||||||
|
array_value:
|
||||||
|
- first
|
||||||
|
- 3
|
||||||
|
- false
|
||||||
|
`)
|
||||||
|
|
||||||
|
p, err := repo.GetProfile(ctx, "json-extra-params")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var got map[string]any
|
||||||
|
encoded, err := json.Marshal(p.ExtraParams)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected extra_params to marshal as JSON, got %v", err)
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(encoded, &got); err != nil {
|
||||||
|
t.Fatalf("expected extra_params JSON to decode, got %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got["string_value"] != "enabled" {
|
||||||
|
t.Fatalf("unexpected string extra param: %#v", got["string_value"])
|
||||||
|
}
|
||||||
|
if got["number_value"] != float64(42) {
|
||||||
|
t.Fatalf("unexpected number extra param: %#v", got["number_value"])
|
||||||
|
}
|
||||||
|
if got["boolean_value"] != true {
|
||||||
|
t.Fatalf("unexpected boolean extra param: %#v", got["boolean_value"])
|
||||||
|
}
|
||||||
|
objectValue, ok := got["object_value"].(map[string]any)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("expected object extra param, got %#v", got["object_value"])
|
||||||
|
}
|
||||||
|
if objectValue["nested"] != "value" || objectValue["count"] != float64(2) {
|
||||||
|
t.Fatalf("unexpected object extra param: %#v", objectValue)
|
||||||
|
}
|
||||||
|
arrayValue, ok := got["array_value"].([]any)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("expected array extra param, got %#v", got["array_value"])
|
||||||
|
}
|
||||||
|
if len(arrayValue) != 3 || arrayValue[0] != "first" || arrayValue[1] != float64(3) || arrayValue[2] != false {
|
||||||
|
t.Fatalf("unexpected array extra param: %#v", arrayValue)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("duplicate profile IDs fail as ambiguous", func(t *testing.T) {
|
||||||
|
writeProfileTestFile(t, filepath.Join(tmpDir, "duplicate-profile-a.yaml"), `
|
||||||
|
id: duplicate-profile
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: first-model
|
||||||
|
`)
|
||||||
|
nestedDir := filepath.Join(tmpDir, "duplicates")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writeProfileTestFile(t, filepath.Join(nestedDir, "duplicate-profile-b.yaml"), `
|
||||||
|
id: duplicate-profile
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: second-model
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetProfile(ctx, "duplicate-profile")
|
||||||
|
if !errors.Is(err, ErrInvalidProfile) {
|
||||||
|
t.Fatalf("expected duplicate profile to return ErrInvalidProfile, got %v", err)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"duplicate execution profile id", "duplicate-profile-a.yaml", filepath.Join("duplicates", "duplicate-profile-b.yaml")} {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("expected error to contain %q, got %v", want, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("nested raw api_key rejected for likely target file", func(t *testing.T) {
|
||||||
|
nestedDir := filepath.Join(tmpDir, "secure")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writeProfileTestFile(t, filepath.Join(nestedDir, "not_named_like_id.yaml"), `
|
||||||
|
id: nested_raw_api_key
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: m
|
||||||
|
api_key: secret
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetProfile(ctx, "nested_raw_api_key")
|
||||||
|
if !errors.Is(err, ErrRawAPIKeyNotAllowed) {
|
||||||
|
t.Fatalf("expected ErrRawAPIKeyNotAllowed, got %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), filepath.Join("secure", "not_named_like_id.yaml")) {
|
||||||
|
t.Fatalf("expected nested path in error, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("raw api_key in non-target profile is ignored", func(t *testing.T) {
|
||||||
|
writeProfileTestFile(t, filepath.Join(tmpDir, "raw-api-key-non-target.yaml"), `
|
||||||
|
id: raw-api-key-non-target
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: m
|
||||||
|
api_key: secret
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetProfile(ctx, "does-not-exist-with-raw-key-nearby")
|
||||||
|
if !errors.Is(err, ErrProfileNotFound) {
|
||||||
|
t.Fatalf("expected ErrProfileNotFound for non-target raw api_key file, got %v", err)
|
||||||
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
t.Run("invalid yaml", func(t *testing.T) {
|
t.Run("invalid yaml", func(t *testing.T) {
|
||||||
@@ -109,3 +257,223 @@ func TestFilesystemRepository_GetProfile(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writeProfileTestFile(t *testing.T, path string, content string) {
|
||||||
|
t.Helper()
|
||||||
|
if err := os.WriteFile(path, []byte(strings.TrimLeft(content, "\n")), 0o644); err != nil {
|
||||||
|
t.Fatalf("failed to write profile test file %q: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFSRepository(t *testing.T) {
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
t.Run("loads valid profiles from nested directories", func(t *testing.T) {
|
||||||
|
repo := NewFSRepository(fstest.MapFS{
|
||||||
|
"profiles/provider/nested.yaml": profileMapFile(`
|
||||||
|
id: nested-profile
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: nested-model
|
||||||
|
temperature: 0.1
|
||||||
|
`),
|
||||||
|
}, "profiles")
|
||||||
|
|
||||||
|
p, err := repo.GetProfile(ctx, "nested-profile")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if p.ID != "nested-profile" || p.Model != "nested-model" {
|
||||||
|
t.Fatalf("unexpected profile: %+v", p)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("rejects unknown YAML fields", func(t *testing.T) {
|
||||||
|
repo := NewFSRepository(fstest.MapFS{
|
||||||
|
"profiles/unknown.yaml": profileMapFile(`
|
||||||
|
id: unknown-profile
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: model
|
||||||
|
unknown: value
|
||||||
|
`),
|
||||||
|
}, "profiles")
|
||||||
|
|
||||||
|
_, err := repo.GetProfile(ctx, "unknown-profile")
|
||||||
|
if !errors.Is(err, ErrInvalidYAML) {
|
||||||
|
t.Fatalf("expected ErrInvalidYAML, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("rejects raw api_key in selected profile", func(t *testing.T) {
|
||||||
|
repo := NewFSRepository(fstest.MapFS{
|
||||||
|
"profiles/raw.yaml": profileMapFile(`
|
||||||
|
id: raw-profile
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: model
|
||||||
|
api_key: secret
|
||||||
|
`),
|
||||||
|
}, "profiles")
|
||||||
|
|
||||||
|
_, err := repo.GetProfile(ctx, "raw-profile")
|
||||||
|
if !errors.Is(err, ErrRawAPIKeyNotAllowed) {
|
||||||
|
t.Fatalf("expected ErrRawAPIKeyNotAllowed, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("ignores raw api_key in non-selected profiles", func(t *testing.T) {
|
||||||
|
repo := NewFSRepository(fstest.MapFS{
|
||||||
|
"profiles/raw.yaml": profileMapFile(`
|
||||||
|
id: raw-profile
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: model
|
||||||
|
api_key: secret
|
||||||
|
`),
|
||||||
|
"profiles/valid.yaml": profileMapFile(`
|
||||||
|
id: valid-profile
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: model
|
||||||
|
`),
|
||||||
|
}, "profiles")
|
||||||
|
|
||||||
|
p, err := repo.GetProfile(ctx, "valid-profile")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if p.ID != "valid-profile" {
|
||||||
|
t.Fatalf("unexpected profile: %+v", p)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("rejects duplicate IDs within one source", func(t *testing.T) {
|
||||||
|
repo := NewFSRepository(fstest.MapFS{
|
||||||
|
"profiles/a.yaml": profileMapFile(`
|
||||||
|
id: duplicate-profile
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: first
|
||||||
|
`),
|
||||||
|
"profiles/nested/b.yaml": profileMapFile(`
|
||||||
|
id: duplicate-profile
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: second
|
||||||
|
`),
|
||||||
|
}, "profiles")
|
||||||
|
|
||||||
|
_, err := repo.GetProfile(ctx, "duplicate-profile")
|
||||||
|
if !errors.Is(err, ErrInvalidProfile) {
|
||||||
|
t.Fatalf("expected ErrInvalidProfile, got %v", err)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"duplicate execution profile id", "a.yaml", "nested/b.yaml"} {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("expected error to contain %q, got %v", want, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOverlayRepository(t *testing.T) {
|
||||||
|
ctx := context.Background()
|
||||||
|
primaryProfile := &domain.ExecutionProfile{ID: "shared", Endpoint: "http://primary", Model: "primary"}
|
||||||
|
fallbackProfile := &domain.ExecutionProfile{ID: "shared", Endpoint: "http://fallback", Model: "fallback"}
|
||||||
|
|
||||||
|
t.Run("returns primary matches before fallback matches", func(t *testing.T) {
|
||||||
|
repo := NewOverlayRepository(
|
||||||
|
staticProfileRepo{profiles: map[string]*domain.ExecutionProfile{"shared": primaryProfile}},
|
||||||
|
staticProfileRepo{profiles: map[string]*domain.ExecutionProfile{"shared": fallbackProfile}},
|
||||||
|
)
|
||||||
|
|
||||||
|
p, err := repo.GetProfile(ctx, "shared")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if p.Model != "primary" {
|
||||||
|
t.Fatalf("expected primary profile, got %+v", p)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("falls back on primary not found", func(t *testing.T) {
|
||||||
|
repo := NewOverlayRepository(
|
||||||
|
staticProfileRepo{},
|
||||||
|
staticProfileRepo{profiles: map[string]*domain.ExecutionProfile{"shared": fallbackProfile}},
|
||||||
|
)
|
||||||
|
|
||||||
|
p, err := repo.GetProfile(ctx, "shared")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if p.Model != "fallback" {
|
||||||
|
t.Fatalf("expected fallback profile, got %+v", p)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("does not fall back after primary load errors", func(t *testing.T) {
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
err error
|
||||||
|
}{
|
||||||
|
{name: "invalid yaml", err: ErrInvalidYAML},
|
||||||
|
{name: "invalid profile", err: ErrInvalidProfile},
|
||||||
|
{name: "raw api key", err: ErrRawAPIKeyNotAllowed},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
repo := NewOverlayRepository(
|
||||||
|
staticProfileRepo{err: tc.err},
|
||||||
|
staticProfileRepo{profiles: map[string]*domain.ExecutionProfile{"shared": fallbackProfile}},
|
||||||
|
)
|
||||||
|
|
||||||
|
_, err := repo.GetProfile(ctx, "shared")
|
||||||
|
if !errors.Is(err, tc.err) {
|
||||||
|
t.Fatalf("expected %v, got %v", tc.err, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("returns not found when both sources miss", func(t *testing.T) {
|
||||||
|
repo := NewOverlayRepository(staticProfileRepo{}, staticProfileRepo{})
|
||||||
|
|
||||||
|
_, err := repo.GetProfile(ctx, "missing")
|
||||||
|
if !errors.Is(err, ErrProfileNotFound) {
|
||||||
|
t.Fatalf("expected ErrProfileNotFound, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("nil primary uses fallback", func(t *testing.T) {
|
||||||
|
repo := NewOverlayRepository(nil, staticProfileRepo{profiles: map[string]*domain.ExecutionProfile{"shared": fallbackProfile}})
|
||||||
|
|
||||||
|
p, err := repo.GetProfile(ctx, "shared")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if p.Model != "fallback" {
|
||||||
|
t.Fatalf("expected fallback profile, got %+v", p)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("nil fallback returns not found after primary miss", func(t *testing.T) {
|
||||||
|
repo := NewOverlayRepository(staticProfileRepo{}, nil)
|
||||||
|
|
||||||
|
_, err := repo.GetProfile(ctx, "missing")
|
||||||
|
if !errors.Is(err, ErrProfileNotFound) {
|
||||||
|
t.Fatalf("expected ErrProfileNotFound, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func profileMapFile(content string) *fstest.MapFile {
|
||||||
|
return &fstest.MapFile{Data: []byte(strings.TrimLeft(content, "\n"))}
|
||||||
|
}
|
||||||
|
|
||||||
|
type staticProfileRepo struct {
|
||||||
|
profiles map[string]*domain.ExecutionProfile
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r staticProfileRepo) GetProfile(_ context.Context, id string) (*domain.ExecutionProfile, error) {
|
||||||
|
if r.err != nil {
|
||||||
|
return nil, r.err
|
||||||
|
}
|
||||||
|
if p, ok := r.profiles[id]; ok {
|
||||||
|
cp := *p
|
||||||
|
return &cp, nil
|
||||||
|
}
|
||||||
|
return nil, ErrProfileNotFound
|
||||||
|
}
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ id: local-secure
|
|||||||
endpoint: http://localhost:8000/v1
|
endpoint: http://localhost:8000/v1
|
||||||
model: gpt-4o-mini
|
model: gpt-4o-mini
|
||||||
api_key_env: SCRIPTORIUM_API_KEY
|
api_key_env: SCRIPTORIUM_API_KEY
|
||||||
|
service_tier: priority
|
||||||
reasoning_effort: medium
|
reasoning_effort: medium
|
||||||
extra_params:
|
extra_params:
|
||||||
provider: local
|
provider: local
|
||||||
|
|||||||
@@ -6,7 +6,9 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
|
"strings"
|
||||||
"text/template"
|
"text/template"
|
||||||
|
"unicode/utf8"
|
||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
@@ -50,6 +52,11 @@ func (r *goRenderer) Render(ctx context.Context, definition *domain.PromptDefini
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
sessionID, err := renderSessionID(definition.SessionID, funcs, vars)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
var renderedMessages []domain.RenderedMessage
|
var renderedMessages []domain.RenderedMessage
|
||||||
|
|
||||||
for i, tmplMsg := range definition.Templates {
|
for i, tmplMsg := range definition.Templates {
|
||||||
@@ -75,12 +82,44 @@ func (r *goRenderer) Render(ctx context.Context, definition *domain.PromptDefini
|
|||||||
}
|
}
|
||||||
|
|
||||||
renderedMessages = append(renderedMessages, domain.RenderedMessage{
|
renderedMessages = append(renderedMessages, domain.RenderedMessage{
|
||||||
Role: tmplMsg.Role,
|
Role: tmplMsg.Role,
|
||||||
Content: buf.String(),
|
Content: buf.String(),
|
||||||
|
CacheControl: cloneCacheControl(tmplMsg.CacheControl),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
return &domain.RenderedPrompt{
|
return &domain.RenderedPrompt{
|
||||||
Messages: renderedMessages,
|
SessionID: sessionID,
|
||||||
|
Messages: renderedMessages,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func renderSessionID(raw string, funcs template.FuncMap, vars map[string]string) (string, error) {
|
||||||
|
if strings.TrimSpace(raw) == "" {
|
||||||
|
return "", nil
|
||||||
|
}
|
||||||
|
|
||||||
|
tmpl, err := template.New("session_id").Funcs(funcs).Option("missingkey=error").Parse(raw)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("%w: session_id: %v", ErrInvalidTemplate, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
if err := tmpl.Execute(&buf, vars); err != nil {
|
||||||
|
return "", fmt.Errorf("%w: session_id: %w", ErrRenderFailure, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
sessionID := strings.TrimSpace(buf.String())
|
||||||
|
if n := utf8.RuneCountInString(sessionID); n > domain.SessionIDMaxLength {
|
||||||
|
return "", fmt.Errorf("%w: session_id length %d exceeds maximum %d", ErrRenderFailure, n, domain.SessionIDMaxLength)
|
||||||
|
}
|
||||||
|
return sessionID, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneCacheControl(in *domain.CacheControl) *domain.CacheControl {
|
||||||
|
if in == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := *in
|
||||||
|
return &out
|
||||||
|
}
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package prompt
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
@@ -78,6 +79,66 @@ func TestGoRenderer_Render(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run("copying cache control to rendered messages", func(t *testing.T) {
|
||||||
|
def := &domain.PromptDefinition{
|
||||||
|
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
|
||||||
|
Templates: []domain.PromptMessageTemplate{
|
||||||
|
{
|
||||||
|
Role: "system",
|
||||||
|
Content: "You are concise.",
|
||||||
|
CacheControl: &domain.CacheControl{
|
||||||
|
Type: domain.CacheControlEphemeral,
|
||||||
|
TTL: "1h",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{Role: "user", Content: "Analyze this: {{input \"transcript\"}}"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
res, err := renderer.Render(ctx, def, inputs, vars)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
if len(res.Messages) != 2 {
|
||||||
|
t.Fatalf("expected 2 messages, got %d", len(res.Messages))
|
||||||
|
}
|
||||||
|
if res.Messages[0].CacheControl == nil {
|
||||||
|
t.Fatal("expected rendered cache control")
|
||||||
|
}
|
||||||
|
if res.Messages[0].CacheControl.Type != domain.CacheControlEphemeral {
|
||||||
|
t.Fatalf("unexpected cache control type: %q", res.Messages[0].CacheControl.Type)
|
||||||
|
}
|
||||||
|
if res.Messages[0].CacheControl.TTL != "1h" {
|
||||||
|
t.Fatalf("unexpected cache control ttl: %q", res.Messages[0].CacheControl.TTL)
|
||||||
|
}
|
||||||
|
if res.Messages[1].CacheControl != nil {
|
||||||
|
t.Fatalf("expected no cache control on second message, got %#v", res.Messages[1].CacheControl)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("rendered cache control does not alias source template", func(t *testing.T) {
|
||||||
|
source := &domain.CacheControl{Type: domain.CacheControlEphemeral, TTL: "1h"}
|
||||||
|
def := &domain.PromptDefinition{
|
||||||
|
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
|
||||||
|
Templates: []domain.PromptMessageTemplate{
|
||||||
|
{Role: "system", Content: "You are concise.", CacheControl: source},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
res, err := renderer.Render(ctx, def, inputs, vars)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
if res.Messages[0].CacheControl == source {
|
||||||
|
t.Fatal("expected rendered cache control to be cloned")
|
||||||
|
}
|
||||||
|
|
||||||
|
res.Messages[0].CacheControl.TTL = ""
|
||||||
|
if source.TTL != "1h" {
|
||||||
|
t.Fatalf("source cache control was mutated, ttl=%q", source.TTL)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
t.Run("accessing vars", func(t *testing.T) {
|
t.Run("accessing vars", func(t *testing.T) {
|
||||||
def := &domain.PromptDefinition{
|
def := &domain.PromptDefinition{
|
||||||
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
|
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
|
||||||
@@ -95,6 +156,78 @@ func TestGoRenderer_Render(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run("rendering session id from vars", func(t *testing.T) {
|
||||||
|
def := &domain.PromptDefinition{
|
||||||
|
SessionID: " {{ .session_id }} ",
|
||||||
|
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
|
||||||
|
Templates: []domain.PromptMessageTemplate{
|
||||||
|
{Role: "system", Content: "Speak in a {{.tone}} tone."},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
res, err := renderer.Render(ctx, def, inputs, map[string]string{
|
||||||
|
"tone": "concise",
|
||||||
|
"session_id": "agent-session-123",
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
if res.SessionID != "agent-session-123" {
|
||||||
|
t.Fatalf("unexpected session id: %q", res.SessionID)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("empty rendered session id is omitted", func(t *testing.T) {
|
||||||
|
def := &domain.PromptDefinition{
|
||||||
|
SessionID: " ",
|
||||||
|
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
|
||||||
|
Templates: []domain.PromptMessageTemplate{
|
||||||
|
{Role: "system", Content: "Speak in a {{.tone}} tone."},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
res, err := renderer.Render(ctx, def, inputs, vars)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
if res.SessionID != "" {
|
||||||
|
t.Fatalf("expected empty session id, got %q", res.SessionID)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("missing session id var fails rendering", func(t *testing.T) {
|
||||||
|
def := &domain.PromptDefinition{
|
||||||
|
SessionID: "{{ .session_id }}",
|
||||||
|
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
|
||||||
|
Templates: []domain.PromptMessageTemplate{
|
||||||
|
{Role: "system", Content: "Speak in a {{.tone}} tone."},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := renderer.Render(ctx, def, inputs, vars)
|
||||||
|
if !errors.Is(err, ErrRenderFailure) {
|
||||||
|
t.Fatalf("expected ErrRenderFailure, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("too long rendered session id fails rendering", func(t *testing.T) {
|
||||||
|
def := &domain.PromptDefinition{
|
||||||
|
SessionID: "{{ .session_id }}",
|
||||||
|
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
|
||||||
|
Templates: []domain.PromptMessageTemplate{
|
||||||
|
{Role: "system", Content: "Speak in a {{.tone}} tone."},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := renderer.Render(ctx, def, inputs, map[string]string{
|
||||||
|
"tone": "concise",
|
||||||
|
"session_id": strings.Repeat("x", domain.SessionIDMaxLength+1),
|
||||||
|
})
|
||||||
|
if !errors.Is(err, ErrRenderFailure) {
|
||||||
|
t.Fatalf("expected ErrRenderFailure, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
t.Run("inserting required input artifact", func(t *testing.T) {
|
t.Run("inserting required input artifact", func(t *testing.T) {
|
||||||
def := &domain.PromptDefinition{
|
def := &domain.PromptDefinition{
|
||||||
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
|
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
|
||||||
|
|||||||
@@ -5,11 +5,14 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"io/fs"
|
||||||
"os"
|
"os"
|
||||||
|
"path"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/filecatalog"
|
||||||
"gopkg.in/yaml.v3"
|
"gopkg.in/yaml.v3"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -23,11 +26,17 @@ type filesystemRepository struct {
|
|||||||
dir string
|
dir string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type fsRepository struct {
|
||||||
|
fsys fs.FS
|
||||||
|
root string
|
||||||
|
}
|
||||||
|
|
||||||
type promptDefinitionFile struct {
|
type promptDefinitionFile struct {
|
||||||
ID string `yaml:"id"`
|
ID string `yaml:"id"`
|
||||||
Version string `yaml:"version"`
|
Version string `yaml:"version"`
|
||||||
DefaultProfile *string `yaml:"default_profile"`
|
DefaultProfile *string `yaml:"default_profile"`
|
||||||
Description string `yaml:"description"`
|
Description string `yaml:"description"`
|
||||||
|
SessionID string `yaml:"session_id"`
|
||||||
Inputs []promptInputFile `yaml:"inputs"`
|
Inputs []promptInputFile `yaml:"inputs"`
|
||||||
Messages []promptMessageFile `yaml:"messages"`
|
Messages []promptMessageFile `yaml:"messages"`
|
||||||
Output promptOutputContractFile `yaml:"output"`
|
Output promptOutputContractFile `yaml:"output"`
|
||||||
@@ -41,9 +50,15 @@ type promptInputFile struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type promptMessageFile struct {
|
type promptMessageFile struct {
|
||||||
Role string `yaml:"role"`
|
Role string `yaml:"role"`
|
||||||
Content string `yaml:"content"`
|
Content string `yaml:"content"`
|
||||||
ContentFile string `yaml:"content_file"`
|
ContentFile string `yaml:"content_file"`
|
||||||
|
CacheControl *cacheControlFile `yaml:"cache_control"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type cacheControlFile struct {
|
||||||
|
Type string `yaml:"type"`
|
||||||
|
TTL string `yaml:"ttl"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type promptOutputContractFile struct {
|
type promptOutputContractFile struct {
|
||||||
@@ -57,34 +72,35 @@ func NewFilesystemRepository(dir string) Repository {
|
|||||||
return &filesystemRepository{dir: dir}
|
return &filesystemRepository{dir: dir}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func NewFSRepository(fsys fs.FS, root string) Repository {
|
||||||
|
return &fsRepository{fsys: fsys, root: root}
|
||||||
|
}
|
||||||
|
|
||||||
func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error) {
|
func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error) {
|
||||||
if strings.TrimSpace(id) == "" {
|
if strings.TrimSpace(id) == "" {
|
||||||
return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition)
|
return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition)
|
||||||
}
|
}
|
||||||
|
|
||||||
files, err := os.ReadDir(r.dir)
|
files, err := filecatalog.FindYAMLFiles(ctx, r.dir)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to read prompt definition directory: %w", err)
|
return nil, fmt.Errorf("failed to read prompt definition directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, file := range files {
|
var matches []promptDefinitionMatch
|
||||||
|
for _, fullPath := range files {
|
||||||
select {
|
select {
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
return nil, ctx.Err()
|
return nil, ctx.Err()
|
||||||
default:
|
default:
|
||||||
}
|
}
|
||||||
|
|
||||||
if file.IsDir() || !isYAMLFile(file.Name()) {
|
relPath := filecatalog.RelativePath(r.dir, fullPath)
|
||||||
continue
|
fileMatch := filecatalog.Stem(filepath.Base(fullPath)) == id
|
||||||
}
|
|
||||||
|
|
||||||
fullPath := filepath.Join(r.dir, file.Name())
|
|
||||||
fileMatch := promptIDFromFileName(file.Name()) == id
|
|
||||||
|
|
||||||
raw, err := loadPromptDefinitionFile(fullPath)
|
raw, err := loadPromptDefinitionFile(fullPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if fileMatch {
|
if fileMatch || promptDefinitionFileHasID(fullPath, id) {
|
||||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, file.Name(), err)
|
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -92,7 +108,7 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
|
|||||||
def, err := normalizePromptDefinition(raw, fullPath)
|
def, err := normalizePromptDefinition(raw, fullPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if fileMatch || strings.TrimSpace(raw.ID) == id {
|
if fileMatch || strings.TrimSpace(raw.ID) == id {
|
||||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, file.Name(), err)
|
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, relPath, err)
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -103,12 +119,39 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
|
|||||||
if version != "" && def.Version != version {
|
if version != "" && def.Version != version {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
return def, nil
|
matches = append(matches, promptDefinitionMatch{
|
||||||
|
def: def,
|
||||||
|
path: relPath,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(matches) > 1 {
|
||||||
|
paths := make([]string, 0, len(matches))
|
||||||
|
for _, match := range matches {
|
||||||
|
paths = append(paths, match.path)
|
||||||
|
}
|
||||||
|
if version != "" {
|
||||||
|
return nil, fmt.Errorf("%w: duplicate prompt definition id %q version %q found in: %s", ErrInvalidPromptDefinition, id, version, strings.Join(paths, ", "))
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("%w: duplicate prompt definition id %q found in: %s", ErrInvalidPromptDefinition, id, strings.Join(paths, ", "))
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(matches) == 1 {
|
||||||
|
return matches[0].def, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil, ErrPromptDefinitionNotFound
|
return nil, ErrPromptDefinitionNotFound
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (r *fsRepository) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error) {
|
||||||
|
return loadPromptDefinition(ctx, r.fsys, r.root, id, version)
|
||||||
|
}
|
||||||
|
|
||||||
|
type promptDefinitionMatch struct {
|
||||||
|
def *domain.PromptDefinition
|
||||||
|
path string
|
||||||
|
}
|
||||||
|
|
||||||
func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
|
func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
|
||||||
data, err := os.ReadFile(path)
|
data, err := os.ReadFile(path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -124,7 +167,166 @@ func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
|
|||||||
return &raw, nil
|
return &raw, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func promptDefinitionFileHasID(path string, id string) bool {
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
var raw struct {
|
||||||
|
ID string `yaml:"id"`
|
||||||
|
}
|
||||||
|
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&raw); err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(raw.ID) == id
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadPromptDefinition(ctx context.Context, fsys fs.FS, root string, id string, version string) (*domain.PromptDefinition, error) {
|
||||||
|
if strings.TrimSpace(id) == "" {
|
||||||
|
return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition)
|
||||||
|
}
|
||||||
|
if fsys == nil {
|
||||||
|
return nil, fmt.Errorf("failed to read prompt definition directory: filesystem is nil")
|
||||||
|
}
|
||||||
|
|
||||||
|
files, err := filecatalog.FindFSYAMLFiles(ctx, fsys, root)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to read prompt definition directory: %w", err)
|
||||||
|
}
|
||||||
|
cleanRoot := filecatalog.CleanFSRoot(root)
|
||||||
|
rootInfo, err := fs.Stat(fsys, cleanRoot)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to read prompt definition directory: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var matches []promptDefinitionMatch
|
||||||
|
for _, fullPath := range files {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return nil, ctx.Err()
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
|
||||||
|
relPath := filecatalog.DisplayPath(root, fullPath)
|
||||||
|
fileMatch := filecatalog.Stem(path.Base(fullPath)) == id
|
||||||
|
data, err := fs.ReadFile(fsys, fullPath)
|
||||||
|
if err != nil {
|
||||||
|
if fileMatch {
|
||||||
|
return nil, fmt.Errorf("%w: %s: failed to read prompt definition file: %v", ErrInvalidYAML, relPath, err)
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
raw, err := decodePromptDefinition(data)
|
||||||
|
if err != nil {
|
||||||
|
if fileMatch || promptDefinitionDataHasID(data, id) {
|
||||||
|
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
def, err := normalizePromptDefinitionFromFS(raw, fsys, root, fullPath, rootInfo.IsDir())
|
||||||
|
if err != nil {
|
||||||
|
if fileMatch || strings.TrimSpace(raw.ID) == id {
|
||||||
|
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, relPath, err)
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
if def.ID != id {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if version != "" && def.Version != version {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
matches = append(matches, promptDefinitionMatch{
|
||||||
|
def: def,
|
||||||
|
path: relPath,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(matches) > 1 {
|
||||||
|
paths := make([]string, 0, len(matches))
|
||||||
|
for _, match := range matches {
|
||||||
|
paths = append(paths, match.path)
|
||||||
|
}
|
||||||
|
if version != "" {
|
||||||
|
return nil, fmt.Errorf("%w: duplicate prompt definition id %q version %q found in: %s", ErrInvalidPromptDefinition, id, version, strings.Join(paths, ", "))
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("%w: duplicate prompt definition id %q found in: %s", ErrInvalidPromptDefinition, id, strings.Join(paths, ", "))
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(matches) == 1 {
|
||||||
|
return matches[0].def, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil, ErrPromptDefinitionNotFound
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodePromptDefinition(data []byte) (*promptDefinitionFile, error) {
|
||||||
|
var raw promptDefinitionFile
|
||||||
|
decoder := yaml.NewDecoder(bytes.NewReader(data))
|
||||||
|
decoder.KnownFields(true)
|
||||||
|
if err := decoder.Decode(&raw); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return &raw, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptDefinitionDataHasID(data []byte, id string) bool {
|
||||||
|
var raw struct {
|
||||||
|
ID string `yaml:"id"`
|
||||||
|
}
|
||||||
|
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&raw); err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(raw.ID) == id
|
||||||
|
}
|
||||||
|
|
||||||
func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*domain.PromptDefinition, error) {
|
func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*domain.PromptDefinition, error) {
|
||||||
|
promptDir := filepath.Dir(sourcePath)
|
||||||
|
return normalizePromptDefinitionWithContent(raw, func(contentFile string) (string, string, error) {
|
||||||
|
resolvedPath := strings.TrimSpace(contentFile)
|
||||||
|
if !filepath.IsAbs(resolvedPath) {
|
||||||
|
resolvedPath = filepath.Join(promptDir, resolvedPath)
|
||||||
|
}
|
||||||
|
resolvedPath = filepath.Clean(resolvedPath)
|
||||||
|
|
||||||
|
body, err := os.ReadFile(resolvedPath)
|
||||||
|
if err != nil {
|
||||||
|
return "", "", err
|
||||||
|
}
|
||||||
|
return string(body), resolvedPath, nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizePromptDefinitionFromFS(raw *promptDefinitionFile, fsys fs.FS, root string, sourcePath string, rootIsDir bool) (*domain.PromptDefinition, error) {
|
||||||
|
promptDir := path.Dir(sourcePath)
|
||||||
|
return normalizePromptDefinitionWithContent(raw, func(contentFile string) (string, string, error) {
|
||||||
|
var resolvedPath string
|
||||||
|
if rootIsDir {
|
||||||
|
var err error
|
||||||
|
resolvedPath, _, err = filecatalog.ResolveFSPath(root, promptDir, contentFile)
|
||||||
|
if err != nil {
|
||||||
|
return "", "", err
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
resolvedPath = strings.TrimSpace(contentFile)
|
||||||
|
if !path.IsAbs(resolvedPath) {
|
||||||
|
resolvedPath = path.Join(promptDir, resolvedPath)
|
||||||
|
}
|
||||||
|
resolvedPath = strings.TrimPrefix(path.Clean(resolvedPath), "/")
|
||||||
|
}
|
||||||
|
|
||||||
|
body, err := fs.ReadFile(fsys, resolvedPath)
|
||||||
|
if err != nil {
|
||||||
|
return "", "", err
|
||||||
|
}
|
||||||
|
return string(body), resolvedPath, nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizePromptDefinitionWithContent(raw *promptDefinitionFile, readContentFile func(string) (string, string, error)) (*domain.PromptDefinition, error) {
|
||||||
if raw == nil {
|
if raw == nil {
|
||||||
return nil, errors.New("prompt definition is nil")
|
return nil, errors.New("prompt definition is nil")
|
||||||
}
|
}
|
||||||
@@ -164,7 +366,6 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
|
|||||||
}
|
}
|
||||||
|
|
||||||
templates := make([]domain.PromptMessageTemplate, 0, len(raw.Messages))
|
templates := make([]domain.PromptMessageTemplate, 0, len(raw.Messages))
|
||||||
promptDir := filepath.Dir(sourcePath)
|
|
||||||
for i, msg := range raw.Messages {
|
for i, msg := range raw.Messages {
|
||||||
role := strings.TrimSpace(msg.Role)
|
role := strings.TrimSpace(msg.Role)
|
||||||
if role == "" {
|
if role == "" {
|
||||||
@@ -177,27 +378,27 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
|
|||||||
return nil, fmt.Errorf("message %d (%s) must set exactly one of content or content_file", i, role)
|
return nil, fmt.Errorf("message %d (%s) must set exactly one of content or content_file", i, role)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
cacheControl, err := normalizeCacheControl(msg.CacheControl)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("message %d (%s) cache_control: %w", i, role, err)
|
||||||
|
}
|
||||||
|
|
||||||
templateContent := msg.Content
|
templateContent := msg.Content
|
||||||
resolvedContentFile := ""
|
resolvedContentFile := ""
|
||||||
if hasContentFile {
|
if hasContentFile {
|
||||||
resolvedPath := strings.TrimSpace(msg.ContentFile)
|
body, resolvedPath, err := readContentFile(msg.ContentFile)
|
||||||
if !filepath.IsAbs(resolvedPath) {
|
|
||||||
resolvedPath = filepath.Join(promptDir, resolvedPath)
|
|
||||||
}
|
|
||||||
resolvedPath = filepath.Clean(resolvedPath)
|
|
||||||
|
|
||||||
body, err := os.ReadFile(resolvedPath)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("prompt %q message %d (%s): failed to read content_file %q: %w", id, i, role, msg.ContentFile, err)
|
return nil, fmt.Errorf("prompt %q message %d (%s): failed to read content_file %q: %w", id, i, role, msg.ContentFile, err)
|
||||||
}
|
}
|
||||||
templateContent = string(body)
|
templateContent = body
|
||||||
resolvedContentFile = resolvedPath
|
resolvedContentFile = resolvedPath
|
||||||
}
|
}
|
||||||
|
|
||||||
templates = append(templates, domain.PromptMessageTemplate{
|
templates = append(templates, domain.PromptMessageTemplate{
|
||||||
Role: role,
|
Role: role,
|
||||||
Content: templateContent,
|
Content: templateContent,
|
||||||
ContentFile: resolvedContentFile,
|
ContentFile: resolvedContentFile,
|
||||||
|
CacheControl: cacheControl,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -227,6 +428,7 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
|
|||||||
Version: version,
|
Version: version,
|
||||||
DefaultProfile: defaultProfile,
|
DefaultProfile: defaultProfile,
|
||||||
Description: strings.TrimSpace(raw.Description),
|
Description: strings.TrimSpace(raw.Description),
|
||||||
|
SessionID: strings.TrimSpace(raw.SessionID),
|
||||||
Inputs: inputs,
|
Inputs: inputs,
|
||||||
Templates: templates,
|
Templates: templates,
|
||||||
OutputFormat: raw.Output.Format,
|
OutputFormat: raw.Output.Format,
|
||||||
@@ -239,14 +441,28 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func isYAMLFile(name string) bool {
|
func normalizeCacheControl(raw *cacheControlFile) (*domain.CacheControl, error) {
|
||||||
return strings.HasSuffix(name, ".yaml") || strings.HasSuffix(name, ".yml")
|
if raw == nil {
|
||||||
}
|
return nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
func promptIDFromFileName(name string) string {
|
cacheType := strings.TrimSpace(raw.Type)
|
||||||
name = strings.TrimSuffix(name, ".yaml")
|
if cacheType == "" {
|
||||||
name = strings.TrimSuffix(name, ".yml")
|
return nil, errors.New("type is required")
|
||||||
return name
|
}
|
||||||
|
if domain.CacheControlType(cacheType) != domain.CacheControlEphemeral {
|
||||||
|
return nil, fmt.Errorf("unsupported type %q", cacheType)
|
||||||
|
}
|
||||||
|
|
||||||
|
ttl := strings.TrimSpace(raw.TTL)
|
||||||
|
if ttl != "" && ttl != "1h" {
|
||||||
|
return nil, fmt.Errorf("unsupported ttl %q", ttl)
|
||||||
|
}
|
||||||
|
|
||||||
|
return &domain.CacheControl{
|
||||||
|
Type: domain.CacheControlType(cacheType),
|
||||||
|
TTL: ttl,
|
||||||
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func isValidOutputFormat(f domain.OutputFormat) bool {
|
func isValidOutputFormat(f domain.OutputFormat) bool {
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
"testing/fstest"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
)
|
)
|
||||||
@@ -68,6 +69,77 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run("valid cache control with ttl", func(t *testing.T) {
|
||||||
|
p, err := repo.GetPromptDefinition(ctx, "valid-cache-control-ttl", "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if len(p.Templates) != 2 {
|
||||||
|
t.Fatalf("expected 2 messages, got %d", len(p.Templates))
|
||||||
|
}
|
||||||
|
assertCacheControl(t, p.Templates[0].CacheControl, domain.CacheControlEphemeral, "1h")
|
||||||
|
if p.Templates[1].CacheControl != nil {
|
||||||
|
t.Fatalf("expected second message cache control to be nil, got %#v", p.Templates[1].CacheControl)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("valid cache control without ttl", func(t *testing.T) {
|
||||||
|
p, err := repo.GetPromptDefinition(ctx, "valid-cache-control-without-ttl", "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if len(p.Templates) != 2 {
|
||||||
|
t.Fatalf("expected 2 messages, got %d", len(p.Templates))
|
||||||
|
}
|
||||||
|
assertCacheControl(t, p.Templates[0].CacheControl, domain.CacheControlEphemeral, "")
|
||||||
|
if p.Templates[1].CacheControl != nil {
|
||||||
|
t.Fatalf("expected second message cache control to be nil, got %#v", p.Templates[1].CacheControl)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("valid session id template", func(t *testing.T) {
|
||||||
|
p, err := repo.GetPromptDefinition(ctx, "valid-session-id", "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if p.SessionID != "{{ .session_id }}" {
|
||||||
|
t.Fatalf("expected trimmed session_id template, got %q", p.SessionID)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("valid nested file-backed prompt resolves content file relative to nested YAML", func(t *testing.T) {
|
||||||
|
nestedDir := filepath.Join(tmpDir, "dnd", "recap")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "nested_recap.yaml"), `
|
||||||
|
id: nested-recap
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content_file: ./nested_recap.user.tmpl
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "nested_recap.user.tmpl"), `Nested recap: {{input "transcript"}}`)
|
||||||
|
|
||||||
|
p, err := repo.GetPromptDefinition(ctx, "nested-recap", "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if len(p.Templates) != 1 {
|
||||||
|
t.Fatalf("expected one template, got %d", len(p.Templates))
|
||||||
|
}
|
||||||
|
if !strings.Contains(p.Templates[0].Content, "Nested recap") {
|
||||||
|
t.Fatalf("expected nested content file body, got %q", p.Templates[0].Content)
|
||||||
|
}
|
||||||
|
if !strings.Contains(p.Templates[0].ContentFile, filepath.Join("dnd", "recap", "nested_recap.user.tmpl")) {
|
||||||
|
t.Fatalf("expected nested content file path, got %q", p.Templates[0].ContentFile)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
t.Run("prompt with default_profile", func(t *testing.T) {
|
t.Run("prompt with default_profile", func(t *testing.T) {
|
||||||
p, err := repo.GetPromptDefinition(ctx, "with-default-profile", "")
|
p, err := repo.GetPromptDefinition(ctx, "with-default-profile", "")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -84,6 +156,124 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run("duplicate prompt IDs fail as ambiguous", func(t *testing.T) {
|
||||||
|
writePromptTestFile(t, filepath.Join(tmpDir, "duplicate_a.yaml"), `
|
||||||
|
id: duplicate-prompt
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: First duplicate.
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
nestedDir := filepath.Join(tmpDir, "nested")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "duplicate_b.yaml"), `
|
||||||
|
id: duplicate-prompt
|
||||||
|
version: "2.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: Second duplicate.
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetPromptDefinition(ctx, "duplicate-prompt", "")
|
||||||
|
if !errors.Is(err, ErrInvalidPromptDefinition) {
|
||||||
|
t.Fatalf("expected duplicate prompt to return ErrInvalidPromptDefinition, got %v", err)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"duplicate prompt definition id", "duplicate_a.yaml", filepath.Join("nested", "duplicate_b.yaml")} {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("expected error to contain %q, got %v", want, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("duplicate prompt ID and requested version fails as ambiguous", func(t *testing.T) {
|
||||||
|
writePromptTestFile(t, filepath.Join(tmpDir, "version_duplicate_a.yaml"), `
|
||||||
|
id: duplicate-version-prompt
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: First duplicate version.
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
nestedDir := filepath.Join(tmpDir, "versioned")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "version_duplicate_b.yaml"), `
|
||||||
|
id: duplicate-version-prompt
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: Second duplicate version.
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetPromptDefinition(ctx, "duplicate-version-prompt", "1.0.0")
|
||||||
|
if !errors.Is(err, ErrInvalidPromptDefinition) {
|
||||||
|
t.Fatalf("expected duplicate prompt version to return ErrInvalidPromptDefinition, got %v", err)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"duplicate prompt definition id", "version \"1.0.0\"", "version_duplicate_a.yaml", filepath.Join("versioned", "version_duplicate_b.yaml")} {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("expected error to contain %q, got %v", want, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("non-matching malformed nested prompt is ignored for not found lookup", func(t *testing.T) {
|
||||||
|
nestedDir := filepath.Join(tmpDir, "broken")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "unrelated.yaml"), "id: [")
|
||||||
|
|
||||||
|
_, err := repo.GetPromptDefinition(ctx, "does-not-exist-even-with-broken-nested-file", "")
|
||||||
|
if !errors.Is(err, ErrPromptDefinitionNotFound) {
|
||||||
|
t.Fatalf("expected ErrPromptDefinitionNotFound, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("strict decode failure in nested prompt matches by YAML ID", func(t *testing.T) {
|
||||||
|
nestedDir := filepath.Join(tmpDir, "strict")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "not_named_like_id.yaml"), `
|
||||||
|
id: nested-strict-error
|
||||||
|
version: "1.0.0"
|
||||||
|
unknown_field: true
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: Invalid because of unknown field.
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetPromptDefinition(ctx, "nested-strict-error", "")
|
||||||
|
if !errors.Is(err, ErrInvalidYAML) {
|
||||||
|
t.Fatalf("expected ErrInvalidYAML, got %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), filepath.Join("strict", "not_named_like_id.yaml")) {
|
||||||
|
t.Fatalf("expected nested path in error, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
t.Run("version lookup", func(t *testing.T) {
|
t.Run("version lookup", func(t *testing.T) {
|
||||||
_, err := repo.GetPromptDefinition(ctx, "valid-inline", "9.9.9")
|
_, err := repo.GetPromptDefinition(ctx, "valid-inline", "9.9.9")
|
||||||
if !errors.Is(err, ErrPromptDefinitionNotFound) {
|
if !errors.Is(err, ErrPromptDefinitionNotFound) {
|
||||||
@@ -107,6 +297,10 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
|
|||||||
{name: "invalid validation mode", id: "invalid_validation_mode", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"invalid validation mode"}},
|
{name: "invalid validation mode", id: "invalid_validation_mode", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"invalid validation mode"}},
|
||||||
{name: "json_schema without schema_path", id: "json_schema_without_schema_path", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"schema_path"}},
|
{name: "json_schema without schema_path", id: "json_schema_without_schema_path", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"schema_path"}},
|
||||||
{name: "unknown input field", id: "unknown_input_field", targetErr: ErrInvalidYAML, errSubstrs: []string{"field unknown_input_setting not found"}},
|
{name: "unknown input field", id: "unknown_input_field", targetErr: ErrInvalidYAML, errSubstrs: []string{"field unknown_input_setting not found"}},
|
||||||
|
{name: "empty cache control type", id: "empty_cache_control_type", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"cache_control", "type is required"}},
|
||||||
|
{name: "unsupported cache control type", id: "unsupported_cache_control_type", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"cache_control", "unsupported type"}},
|
||||||
|
{name: "unsupported cache control ttl", id: "unsupported_cache_control_ttl", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"cache_control", "unsupported ttl"}},
|
||||||
|
{name: "unknown cache control field", id: "unknown_cache_control_field", targetErr: ErrInvalidYAML, errSubstrs: []string{"field unexpected not found"}},
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, tc := range cases {
|
for _, tc := range cases {
|
||||||
@@ -131,6 +325,180 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFSRepositoryGetPromptDefinition(t *testing.T) {
|
||||||
|
repo := NewFSRepository(fstest.MapFS{
|
||||||
|
"prompts/nested/prompt.yaml": &fstest.MapFile{Data: []byte(`
|
||||||
|
id: fs-prompt
|
||||||
|
version: "1.0.0"
|
||||||
|
inputs:
|
||||||
|
- name: transcript
|
||||||
|
required: true
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content_file: ./messages/user.tmpl
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)},
|
||||||
|
"prompts/nested/messages/user.tmpl": &fstest.MapFile{Data: []byte(`Summarize {{input "transcript"}}.`)},
|
||||||
|
}, "prompts")
|
||||||
|
|
||||||
|
got, err := repo.GetPromptDefinition(context.Background(), "fs-prompt", "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if got.ID != "fs-prompt" {
|
||||||
|
t.Fatalf("unexpected prompt id: %q", got.ID)
|
||||||
|
}
|
||||||
|
if len(got.Templates) != 1 || !strings.Contains(got.Templates[0].Content, `{{input "transcript"}}`) {
|
||||||
|
t.Fatalf("expected content_file body to be loaded, got %+v", got.Templates)
|
||||||
|
}
|
||||||
|
if got.Templates[0].ContentFile != "prompts/nested/messages/user.tmpl" {
|
||||||
|
t.Fatalf("unexpected content file path: %q", got.Templates[0].ContentFile)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFSRepositoryContentFileContainment(t *testing.T) {
|
||||||
|
t.Run("nested prompt can reference file inside root", func(t *testing.T) {
|
||||||
|
repo := NewFSRepository(fstest.MapFS{
|
||||||
|
"prompts/nested/prompt.yaml": &fstest.MapFile{Data: []byte(`
|
||||||
|
id: fs-contained-prompt
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content_file: ../shared/user.tmpl
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)},
|
||||||
|
"prompts/shared/user.tmpl": &fstest.MapFile{Data: []byte(`Inside root.`)},
|
||||||
|
}, "prompts")
|
||||||
|
|
||||||
|
got, err := repo.GetPromptDefinition(context.Background(), "fs-contained-prompt", "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if len(got.Templates) != 1 || got.Templates[0].Content != "Inside root." {
|
||||||
|
t.Fatalf("expected contained content file, got %+v", got.Templates)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
contentFile string
|
||||||
|
wantErr string
|
||||||
|
}{
|
||||||
|
{name: "parent escape rejected", contentFile: "../outside.tmpl", wantErr: "escapes source root"},
|
||||||
|
{name: "absolute path rejected", contentFile: "/outside.tmpl", wantErr: "must be relative"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
repo := NewFSRepository(fstest.MapFS{
|
||||||
|
"prompts/prompt.yaml": &fstest.MapFile{Data: []byte(`
|
||||||
|
id: fs-escaped-prompt
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content_file: ` + tc.contentFile + `
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)},
|
||||||
|
"outside.tmpl": &fstest.MapFile{Data: []byte(`Outside root.`)},
|
||||||
|
}, "prompts")
|
||||||
|
|
||||||
|
_, err := repo.GetPromptDefinition(context.Background(), "fs-escaped-prompt", "")
|
||||||
|
if !errors.Is(err, ErrInvalidPromptDefinition) {
|
||||||
|
t.Fatalf("expected ErrInvalidPromptDefinition, got %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), tc.wantErr) {
|
||||||
|
t.Fatalf("expected error to contain %q, got %v", tc.wantErr, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFSRepositoryRejectsDuplicatePromptIDs(t *testing.T) {
|
||||||
|
repo := NewFSRepository(fstest.MapFS{
|
||||||
|
"one.yaml": &fstest.MapFile{Data: []byte(`
|
||||||
|
id: duplicate-fs-prompt
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: First.
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
repair_attempts: 0
|
||||||
|
`)},
|
||||||
|
"nested/two.yaml": &fstest.MapFile{Data: []byte(`
|
||||||
|
id: duplicate-fs-prompt
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: Second.
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
repair_attempts: 0
|
||||||
|
`)},
|
||||||
|
}, ".")
|
||||||
|
|
||||||
|
_, err := repo.GetPromptDefinition(context.Background(), "duplicate-fs-prompt", "")
|
||||||
|
if !errors.Is(err, ErrInvalidPromptDefinition) {
|
||||||
|
t.Fatalf("expected ErrInvalidPromptDefinition, got %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "one.yaml") || !strings.Contains(err.Error(), "nested/two.yaml") {
|
||||||
|
t.Fatalf("expected duplicate paths in error, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFSRepositoryRejectsUnknownYAMLFields(t *testing.T) {
|
||||||
|
repo := NewFSRepository(fstest.MapFS{
|
||||||
|
"not_named_like_id.yaml": &fstest.MapFile{Data: []byte(`
|
||||||
|
id: strict-fs-prompt
|
||||||
|
version: "1.0.0"
|
||||||
|
unknown: true
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: Invalid.
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
repair_attempts: 0
|
||||||
|
`)},
|
||||||
|
}, ".")
|
||||||
|
|
||||||
|
_, err := repo.GetPromptDefinition(context.Background(), "strict-fs-prompt", "")
|
||||||
|
if !errors.Is(err, ErrInvalidYAML) {
|
||||||
|
t.Fatalf("expected ErrInvalidYAML, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertCacheControl(t *testing.T, got *domain.CacheControl, wantType domain.CacheControlType, wantTTL string) {
|
||||||
|
t.Helper()
|
||||||
|
if got == nil {
|
||||||
|
t.Fatal("expected cache control, got nil")
|
||||||
|
}
|
||||||
|
if got.Type != wantType {
|
||||||
|
t.Fatalf("unexpected cache control type: got %q want %q", got.Type, wantType)
|
||||||
|
}
|
||||||
|
if got.TTL != wantTTL {
|
||||||
|
t.Fatalf("unexpected cache control ttl: got %q want %q", got.TTL, wantTTL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writePromptTestFile(t *testing.T, path string, content string) {
|
||||||
|
t.Helper()
|
||||||
|
if err := os.WriteFile(path, []byte(strings.TrimLeft(content, "\n")), 0o644); err != nil {
|
||||||
|
t.Fatalf("failed to write prompt test file %q: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func copyTree(src, dst string) error {
|
func copyTree(src, dst string) error {
|
||||||
return filepath.WalkDir(src, func(path string, d fs.DirEntry, err error) error {
|
return filepath.WalkDir(src, func(path string, d fs.DirEntry, err error) error {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
10
internal/promptdef/testdata/empty_cache_control_type.yaml
vendored
Normal file
10
internal/promptdef/testdata/empty_cache_control_type.yaml
vendored
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
id: empty-cache-control-type
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: system
|
||||||
|
content: "Use cached instructions."
|
||||||
|
cache_control: {}
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
12
internal/promptdef/testdata/unknown_cache_control_field.yaml
vendored
Normal file
12
internal/promptdef/testdata/unknown_cache_control_field.yaml
vendored
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
id: unknown-cache-control-field
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: system
|
||||||
|
content: "Use cached instructions."
|
||||||
|
cache_control:
|
||||||
|
type: ephemeral
|
||||||
|
unexpected: true
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user