Compare commits
119 Commits
v0.8.0
...
309fe9b7ea
| Author | SHA1 | Date | |
|---|---|---|---|
| 309fe9b7ea | |||
| adfd08ffe2 | |||
| c7263ab2a8 | |||
| 532c31c09a | |||
| 68cd90c657 | |||
| 416438d80d | |||
| 9298d8ae73 | |||
| 4c7278febc | |||
| 096208532e | |||
| 50bd19b9d1 | |||
| 2fc7204bd5 | |||
| c0d4ea0d4e | |||
| 4d7e1327ad | |||
| 3c33b52b15 | |||
| 280916bf4a | |||
| 033bc93d3c | |||
| 45c2644b9d | |||
| a74c03bd9b | |||
| ad115a2259 | |||
| 3074b3165a | |||
| 8703793b0c | |||
| 4cb4943a57 | |||
| c6c747e94d | |||
| 2bbf13e739 | |||
| 5edb24a9c1 | |||
| 99d5e96316 | |||
| 144d840fbe | |||
| ed0c9f6370 | |||
| a0e905ce46 | |||
| 719243e90c | |||
| a9e1b7435c | |||
| eb6dfb19b0 | |||
| d86b65adad | |||
| 31faaf4259 | |||
| 9932153b97 | |||
| f0ca233c25 | |||
| ff31f8daf8 | |||
| c927b7819d | |||
| 6d1fb66dd7 | |||
| e0b1d6a0dc | |||
| 33698903be | |||
| 90b76ddad3 | |||
| d5b3d1e061 | |||
| 41083de46a | |||
| 07ac7e54c5 | |||
| 879cb021b2 | |||
| 574f88bd6a | |||
| d5d7a222a4 | |||
| aabd89aea7 | |||
| 9189cbfc22 | |||
| 872c166ed7 | |||
| 6742def4d3 | |||
| 1b39f82117 | |||
| f7d821067f | |||
| a16f66cbc7 | |||
| 39485d87f6 | |||
| 61e5b0fe58 | |||
| f3c21c7d9f | |||
| bc5f5d3731 | |||
| 93a76f1d36 | |||
| 5c882f26a9 | |||
| 0d45ac6e3c | |||
| f7ad756fc3 | |||
| 4fe11b1b2b | |||
| 296f9b1817 | |||
| 7a8516b0c6 | |||
| 2df2f530b3 | |||
| 8b25ca72e5 | |||
| fa02791fe9 | |||
| 32767b4eb4 | |||
| e1e5351c5d | |||
| d60ef66f53 | |||
| 4669b73d38 | |||
| 6f91603168 | |||
| 3ad247039b | |||
| 32e2433628 | |||
| 712c6b92b8 | |||
| 89cafcefec | |||
| 1d7fac0a47 | |||
| 03d4f27d2b | |||
| 4ac2038331 | |||
| 14a7e7e04c | |||
| 5e522bad8b | |||
| 23872dd742 | |||
| 7ffbf5f6ca | |||
| d0dc30fcc9 | |||
| b38f7b4dc3 | |||
| 0512995931 | |||
| 049a5feadb | |||
| 1798e9c575 | |||
| 5d4bc8c2b9 | |||
| 63fb8fc132 | |||
| 4d4bb7a121 | |||
| 5dcb3cd4fc | |||
| efe346893c | |||
| c95d6fcfec | |||
| 0badb4364d | |||
| 1f63f8afbb | |||
| bc099a31ad | |||
| 4ff55221a3 | |||
| 8d8024099f | |||
| 18792fd8d1 | |||
| 8860aa033c | |||
| 3ca14d8b6e | |||
| 099e9c4a3e | |||
| cfe6b9408a | |||
| 6ececc749f | |||
| 79901fbb86 | |||
| 75fa0a030a | |||
| ef64966897 | |||
| c6c5e3cb69 | |||
| 3f4fd230b9 | |||
| 2091b58066 | |||
| c3fe88c9fa | |||
| 5830fda516 | |||
| 359e910572 | |||
| 4950a6bb14 | |||
| b69ba96811 | |||
| 941e2656e8 |
1
.gitignore
vendored
1
.gitignore
vendored
@@ -1,6 +1,5 @@
|
||||
# ---> Codex
|
||||
.codex
|
||||
AGENTS.md
|
||||
|
||||
# ---> Go
|
||||
# If you prefer the allow list template instead of the deny list, see community template:
|
||||
|
||||
@@ -28,10 +28,6 @@ steps:
|
||||
|
||||
build_binary linux amd64 ""
|
||||
build_binary linux arm64 ""
|
||||
build_binary darwin amd64 ""
|
||||
build_binary darwin arm64 ""
|
||||
build_binary windows amd64 ".exe"
|
||||
build_binary windows arm64 ".exe"
|
||||
|
||||
- name: publish-release
|
||||
image: woodpeckerci/plugin-release
|
||||
|
||||
1
AGENTS.md
Normal file
1
AGENTS.md
Normal file
@@ -0,0 +1 @@
|
||||
Please review `docs/development.md` for initial orientation in this repository and follow its task-specific reading guide.
|
||||
2
LICENSE
2
LICENSE
@@ -1,4 +1,4 @@
|
||||
Copyright (c) 2026 eric.
|
||||
Copyright (c) 2026 Eric Rakestraw.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
|
||||
|
||||
|
||||
458
README.md
458
README.md
@@ -1,449 +1,45 @@
|
||||
# scriptorium
|
||||
|
||||
Scriptorium is a generic prompt execution engine.
|
||||
Scriptorium is a narrow prompt-execution application for rendering prompt
|
||||
requests, running them against OpenAI-compatible chat-completions endpoints, and
|
||||
serving the same run workflow over HTTP.
|
||||
|
||||
It takes:
|
||||
- a prompt definition
|
||||
- a selected or default execution profile
|
||||
- named input artifacts
|
||||
- template variables
|
||||
- optional runtime overrides
|
||||
It keeps prompt definitions, execution profiles, schemas, and input artifacts as
|
||||
separate files so prompts can be reviewed and reused without baking model
|
||||
runtime settings into application code.
|
||||
|
||||
It returns:
|
||||
- for `run`: generated artifact, validation result, metadata
|
||||
- for `render`: prepared/rendered prompt data (no model output)
|
||||
## Quickstart
|
||||
|
||||
## Prompt vs Profile
|
||||
From the repository root:
|
||||
|
||||
Scriptorium separates **what** to do (Prompt) from **how** to do it (Profile).
|
||||
|
||||
### Prompt Definition
|
||||
Defines the task logic and output contract.
|
||||
- Task description and version.
|
||||
- Message templates (system, user, etc.).
|
||||
- Required and optional input artifacts.
|
||||
- Output format and validation rules.
|
||||
- Repair settings for structured output.
|
||||
- Optional `default_profile` for convenience.
|
||||
|
||||
### Execution Profile
|
||||
Defines the runtime environment and model settings.
|
||||
- LLM endpoint (URL).
|
||||
- Model name.
|
||||
- Generation parameters: `temperature`, `max_tokens`, `top_p`.
|
||||
- Runtime settings: `timeout`, `reasoning_effort`.
|
||||
- API key source via `api_key_env`.
|
||||
|
||||
Callers can explicitly provide a `profile_id` to override the prompt's `default_profile`.
|
||||
|
||||
## Precedence
|
||||
|
||||
Scriptorium uses two precedence layers:
|
||||
|
||||
### Application Configuration Precedence
|
||||
|
||||
For application-level adapter settings (for example prompt/profile/schema directories, server address, and render output default), precedence is:
|
||||
|
||||
1. **CLI Flags**
|
||||
2. **`config.yml`**
|
||||
3. **Built-in application defaults**
|
||||
|
||||
Application config loading behavior:
|
||||
- Default config path: `/etc/scriptorium/config.yml`
|
||||
- Override path: `--config <PATH>` (supported by `run`, `render`, and `serve`)
|
||||
- If `--config` is provided, the file must exist and be valid.
|
||||
- If `--config` is omitted, missing `/etc/scriptorium/config.yml` is allowed.
|
||||
|
||||
### Runtime Model Precedence
|
||||
|
||||
When resolving runtime model settings, Scriptorium follows this precedence model (highest to lowest):
|
||||
|
||||
1. **Runtime Overrides**: Provided via CLI flags or HTTP request `model` object.
|
||||
2. **Execution Profile**: Settings defined in the selected profile.
|
||||
3. **Application Defaults**: Built-in fallback values.
|
||||
|
||||
### Profile Selection Logic
|
||||
The engine determines which profile to use in this order:
|
||||
1. Explicit `profile_id` (via `--profile` or HTTP request).
|
||||
2. The `default_profile` named in the Prompt Definition.
|
||||
3. Error: If neither is provided and no default exists.
|
||||
|
||||
## API Key Policy
|
||||
|
||||
To ensure security, Scriptorium does not support raw API keys in configuration files, CLI arguments, or HTTP requests.
|
||||
|
||||
- **`api_key_env`**: Profiles and overrides specify the name of an environment variable (e.g., `SCRIPTORIUM_API_KEY`).
|
||||
- **Runtime Resolution**: The value of the environment variable is read directly from the process environment at runtime.
|
||||
- **Zero Leakage**: API key values are never included in metadata, logs, or response bodies.
|
||||
|
||||
## CLI Usage
|
||||
|
||||
Available CLI commands:
|
||||
- `scriptorium run`
|
||||
- `scriptorium render`
|
||||
- `scriptorium serve`
|
||||
|
||||
All commands accept `--config <PATH>`.
|
||||
|
||||
`prompt_dir` and `profile_dir` may be supplied by CLI flags or `config.yml`:
|
||||
- `--prompt-dir` or `config.yml` `prompt_dir`
|
||||
- `--profile-dir` or `config.yml` `profile_dir`
|
||||
|
||||
`schema_dir` and `serve` `addr` may also be supplied by `config.yml` where applicable:
|
||||
- `--schema-dir` or `config.yml` `schema_dir`
|
||||
- `--addr` or `config.yml` `server.addr`
|
||||
|
||||
### `scriptorium run`
|
||||
|
||||
Runs a single prompt execution.
|
||||
|
||||
**Required Flags:**
|
||||
- `--prompt`: The prompt ID to execute.
|
||||
- `--input`: Input mapping `name=path` (repeatable).
|
||||
|
||||
**Required Effective Settings:**
|
||||
- Prompt directory: `--prompt-dir` or `config.yml` `prompt_dir`
|
||||
- Profile directory: `--profile-dir` or `config.yml` `profile_dir`
|
||||
|
||||
**Optional Flags:**
|
||||
- `--config`: Application config file path. Default discovery path is `/etc/scriptorium/config.yml`.
|
||||
- `--prompt-dir`: Override prompt directory from config.
|
||||
- `--profile-dir`: Override profile directory from config.
|
||||
- `--profile`: Override the prompt's default profile.
|
||||
- `--var`: Template variable `name=value` (repeatable).
|
||||
- `--out`: Write output to a file instead of stdout.
|
||||
- `--llm-base-url`: Override endpoint.
|
||||
- `--model`: Override model name.
|
||||
- `--api-key-env`: Override API key environment variable name.
|
||||
- `--temperature`: Override temperature.
|
||||
- `--max-tokens`: Override max tokens.
|
||||
- `--top-p`: Override top_p.
|
||||
- `--timeout`: Override request timeout (e.g., `30s`, `1m`).
|
||||
- `--schema-dir`: Base directory for validation schemas.
|
||||
|
||||
**Examples:**
|
||||
|
||||
Using `config.yml` for prompt/profile directories:
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt generic.markdown_summary \
|
||||
--input transcript=./examples/fixtures/transcript.md
|
||||
```
|
||||
|
||||
Overriding config directories explicitly:
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt-dir ./prompts \
|
||||
--profile-dir ./profiles \
|
||||
--prompt generic.markdown_summary \
|
||||
--input transcript=./examples/fixtures/transcript.md
|
||||
```
|
||||
|
||||
Overriding the profile:
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt-dir ./prompts \
|
||||
--profile-dir ./profiles \
|
||||
--prompt generic.markdown_summary \
|
||||
--profile local-quality \
|
||||
--input transcript=./examples/fixtures/transcript.md
|
||||
```
|
||||
|
||||
Overriding model and runtime values:
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt-dir ./prompts \
|
||||
--profile-dir ./profiles \
|
||||
--prompt generic.markdown_summary \
|
||||
--model gpt-4o \
|
||||
--temperature 0.7 \
|
||||
--input transcript=./examples/fixtures/transcript.md
|
||||
```
|
||||
|
||||
Using a local OpenAI-compatible vLLM endpoint:
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt-dir ./prompts \
|
||||
--profile-dir ./profiles \
|
||||
--prompt generic.markdown_summary \
|
||||
--llm-base-url http://localhost:8000/v1 \
|
||||
--model meta-llama-3-8b \
|
||||
--input transcript=./examples/fixtures/transcript.md
|
||||
```
|
||||
|
||||
### `scriptorium render`
|
||||
|
||||
Prepares and renders a prompt without calling the LLM.
|
||||
|
||||
`render` uses the same prompt/profile/input/variable/runtime override resolution as `run`:
|
||||
- Profile selection precedence: `--profile` -> prompt `default_profile` -> error.
|
||||
- Runtime precedence: CLI runtime overrides -> selected profile -> built-in defaults.
|
||||
|
||||
`render` is useful for debugging:
|
||||
- prompt template rendering
|
||||
- input mappings
|
||||
- selected profile behavior
|
||||
- runtime override behavior
|
||||
|
||||
`render` does not:
|
||||
- call the LLM
|
||||
- validate model output
|
||||
- perform repair
|
||||
- expose resolved API key values
|
||||
|
||||
It may include `api_key_env` names where relevant.
|
||||
|
||||
**Required Flags:**
|
||||
- `--prompt`: The prompt ID to render.
|
||||
- `--input`: Input mapping `name=path` (repeatable).
|
||||
|
||||
**Required Effective Settings:**
|
||||
- Prompt directory: `--prompt-dir` or `config.yml` `prompt_dir`
|
||||
- Profile directory: `--profile-dir` or `config.yml` `profile_dir`
|
||||
|
||||
**Optional Flags:**
|
||||
- `--config`: Application config file path. Default discovery path is `/etc/scriptorium/config.yml`.
|
||||
- `--prompt-dir`: Override prompt directory from config.
|
||||
- `--profile-dir`: Override profile directory from config.
|
||||
- `--profile`: Override the prompt's default profile.
|
||||
- `--var`: Template variable `name=value` (repeatable).
|
||||
- `--out`: Write output to a file instead of stdout.
|
||||
- `--format`: Render output format (`text` or `json`). Default: `text`.
|
||||
- `--llm-base-url`: Runtime override for endpoint.
|
||||
- `--model`: Runtime override for model name.
|
||||
- `--api-key-env`: Runtime override for API key environment variable name.
|
||||
- `--temperature`: Runtime override for temperature.
|
||||
- `--max-tokens`: Runtime override for max tokens.
|
||||
- `--top-p`: Runtime override for top_p.
|
||||
- `--timeout`: Runtime override for timeout (e.g., `30s`, `1m`).
|
||||
|
||||
**Render Output Formats:**
|
||||
- `text`: Human-readable output (default).
|
||||
- `json`: Machine-readable structured output.
|
||||
|
||||
Render formatting is modular; additional output formats can be added later without changing prepare/run core logic.
|
||||
|
||||
**Examples:**
|
||||
|
||||
Default text output using `config.yml` directories:
|
||||
```bash
|
||||
scriptorium render \
|
||||
--prompt generic.markdown_summary \
|
||||
--input transcript=./examples/fixtures/transcript.md
|
||||
```
|
||||
|
||||
Explicit config path:
|
||||
```bash
|
||||
scriptorium render \
|
||||
go run ./cmd/scriptorium render \
|
||||
--config ./examples/config.yml \
|
||||
--prompt generic.markdown_summary \
|
||||
--input transcript=./examples/fixtures/transcript.md
|
||||
```
|
||||
|
||||
Explicit directory overrides:
|
||||
```bash
|
||||
scriptorium render \
|
||||
--prompt-dir ./prompts \
|
||||
--profile-dir ./profiles \
|
||||
--prompt generic.markdown_summary \
|
||||
--input transcript=./examples/fixtures/transcript.md
|
||||
```
|
||||
|
||||
Explicit JSON output:
|
||||
```bash
|
||||
scriptorium render \
|
||||
--prompt-dir ./prompts \
|
||||
--profile-dir ./profiles \
|
||||
--prompt generic.markdown_summary \
|
||||
--input transcript=./examples/fixtures/transcript.md \
|
||||
--input glossary=./examples/fixtures/glossary.yml \
|
||||
--format json
|
||||
```
|
||||
|
||||
Using prompt `default_profile` (omit `--profile`):
|
||||
```bash
|
||||
scriptorium render \
|
||||
--prompt-dir ./prompts \
|
||||
--profile-dir ./profiles \
|
||||
--prompt generic.markdown_summary \
|
||||
--input transcript=./examples/fixtures/transcript.md
|
||||
```
|
||||
This command renders the prepared prompt and effective runtime settings without calling an LLM.
|
||||
For complete invocation and output behavior, see the [CLI reference](docs/cli.md).
|
||||
|
||||
Overriding profile selection:
|
||||
```bash
|
||||
scriptorium render \
|
||||
--prompt-dir ./prompts \
|
||||
--profile-dir ./profiles \
|
||||
--prompt generic.markdown_summary \
|
||||
--profile local-quality \
|
||||
--input transcript=./examples/fixtures/transcript.md
|
||||
```
|
||||
## Documentation
|
||||
|
||||
Overriding runtime settings:
|
||||
```bash
|
||||
scriptorium render \
|
||||
--prompt-dir ./prompts \
|
||||
--profile-dir ./profiles \
|
||||
--prompt generic.markdown_summary \
|
||||
--input transcript=./examples/fixtures/transcript.md \
|
||||
--llm-base-url http://localhost:8000/v1 \
|
||||
--model gpt-4o-mini \
|
||||
--temperature 0.2 \
|
||||
--max-tokens 800 \
|
||||
--top-p 1.0 \
|
||||
--timeout 45s
|
||||
```
|
||||
|
||||
Writing rendered output to a file:
|
||||
```bash
|
||||
scriptorium render \
|
||||
--prompt-dir ./prompts \
|
||||
--profile-dir ./profiles \
|
||||
--prompt generic.markdown_summary \
|
||||
--input transcript=./examples/fixtures/transcript.md \
|
||||
--format text \
|
||||
--out ./rendered_prompt.txt
|
||||
```
|
||||
|
||||
### `scriptorium serve`
|
||||
|
||||
Starts the HTTP API.
|
||||
|
||||
**Required Effective Settings:**
|
||||
- Prompt directory: `--prompt-dir` or `config.yml` `prompt_dir`
|
||||
- Profile directory: `--profile-dir` or `config.yml` `profile_dir`
|
||||
|
||||
**Optional Flags:**
|
||||
- `--config`: Application config file path. Default discovery path is `/etc/scriptorium/config.yml`.
|
||||
- `--addr`: Listen address (default `:8080`).
|
||||
- `--schema-dir`: Base directory for validation schemas.
|
||||
|
||||
**Examples:**
|
||||
|
||||
Using `config.yml`:
|
||||
```bash
|
||||
scriptorium serve
|
||||
```
|
||||
|
||||
Overriding config for local use:
|
||||
```bash
|
||||
scriptorium serve \
|
||||
--prompt-dir ./prompts \
|
||||
--profile-dir ./profiles \
|
||||
--addr :9090
|
||||
```
|
||||
|
||||
## HTTP API
|
||||
|
||||
### `POST /v1/runs`
|
||||
|
||||
Executes a prompt. No built-in authentication is provided; deploy behind a trusted gateway.
|
||||
|
||||
**Request Body:**
|
||||
```json
|
||||
{
|
||||
"prompt_id": "generic.structured_events",
|
||||
"profile_id": "local-quality",
|
||||
"include_raw_output": false,
|
||||
"inputs": {
|
||||
"transcript": {"type": "file", "uri": "./examples/fixtures/transcript.md"}
|
||||
},
|
||||
"vars": {
|
||||
"session_date": "2026-05-04"
|
||||
},
|
||||
"model": {
|
||||
"endpoint": "http://localhost:8000/v1",
|
||||
"model": "gpt-4o-mini",
|
||||
"temperature": 0.0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`profile_id` is optional. If omitted, Scriptorium uses the prompt's `default_profile`. If neither is available, the run fails.
|
||||
|
||||
**Response:**
|
||||
Returns a `200 OK` with the generated artifact, validation results, and metadata including the `prompt_id` and the `selected_profile_id`.
|
||||
|
||||
**Validation Failures:**
|
||||
If the model output fails validation (e.g., invalid JSON), the API returns `200 OK` with `validation.status = "failed"`.
|
||||
|
||||
**Raw Output Exposure:**
|
||||
- `raw_model_output` is omitted by default.
|
||||
- Set `include_raw_output: true` in the request to include it in the response.
|
||||
- Raw output is preserved internally in run results regardless of HTTP exposure.
|
||||
|
||||
## Prompt Definition Authoring
|
||||
|
||||
Prompts are defined in YAML.
|
||||
|
||||
### Canonical Shape
|
||||
```yaml
|
||||
id: generic.structured_events
|
||||
version: "1.0.0"
|
||||
description: "Extracts structured events from a transcript"
|
||||
default_profile: local-quality
|
||||
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: text/markdown
|
||||
description: "The raw session transcript"
|
||||
- name: glossary
|
||||
required: false
|
||||
content_type: application/yaml
|
||||
description: "Optional glossary terms"
|
||||
|
||||
messages:
|
||||
- role: system
|
||||
content: "You are a helpful assistant."
|
||||
- role: user
|
||||
content_file: messages/extract_events.tmpl
|
||||
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: structured_events.schema.json
|
||||
repair_attempts: 2
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- **Inline vs File**: Use `content` for short prompts or `content_file` for larger templates. Exactly one must be set per message.
|
||||
- **Path Resolution**: `content_file` paths are resolved relative to the prompt YAML file.
|
||||
- **Inputs**: Mark inputs as `required` to ensure the runner fails early if they are missing.
|
||||
- **Input Metadata**: `content_type` is currently descriptive metadata and not enforced yet.
|
||||
- **Validation**: Support `none`, `basic`, `json`, and `json_schema`.
|
||||
- **Repair**: `repair_attempts` enables bounded retries to fix structured output.
|
||||
|
||||
## Execution Profile Authoring
|
||||
|
||||
Profiles are defined in YAML.
|
||||
|
||||
### Canonical Shape
|
||||
```yaml
|
||||
id: local-quality
|
||||
endpoint: http://localhost:8000/v1
|
||||
model: gpt-4o
|
||||
temperature: 0.0
|
||||
max_tokens: 4096
|
||||
top_p: 1.0
|
||||
timeout_seconds: 300
|
||||
reasoning_effort: high
|
||||
api_key_env: SCRIPTORIUM_API_KEY
|
||||
```
|
||||
|
||||
**Constraints:**
|
||||
- **No Raw Keys**: Do not include actual API keys. Only specify the environment variable name in `api_key_env`.
|
||||
- **Local Profiles**: For local endpoints that don't require auth, `api_key_env` can be omitted.
|
||||
- [CLI reference](docs/cli.md)
|
||||
- [Configuration reference](docs/config.md)
|
||||
- [HTTP API reference](docs/api.md)
|
||||
- [Operations guide](docs/operations.md)
|
||||
- [Consumer integration overview](docs/consumers/api.md)
|
||||
- [Go library package](docs/consumers/pkg-scriptorium.md)
|
||||
- [Subprocess integration](docs/integrations/subprocess.md)
|
||||
- [OpenAI-compatible chat integration](docs/integrations/openai-compatible-chat.md)
|
||||
- [Architecture policy](docs/policy/architecture.md)
|
||||
|
||||
## Examples
|
||||
|
||||
- **Prompt Definitions**: `prompts/`
|
||||
- **Execution Profiles**: `profiles/`
|
||||
- **Schemas**: `schemas/`
|
||||
- **Fixtures**: `examples/fixtures/`
|
||||
|
||||
## Build and Test
|
||||
|
||||
```bash
|
||||
go build -o scriptorium ./cmd/scriptorium
|
||||
go test ./...
|
||||
```
|
||||
- [Minimal configuration](examples/config.yml) and [complete configuration](examples/config.full.yml)
|
||||
- [Prompt definitions](examples/prompts/), [execution profiles](examples/profiles/), [schemas](examples/schemas/), and [synthetic input fixtures](examples/fixtures/)
|
||||
- [Render script](examples/render-markdown-summary.sh)
|
||||
- [HTTP request](examples/http-run.json)
|
||||
- [Go library example](examples/go-library/prepare/main.go)
|
||||
|
||||
483
architecture.md
483
architecture.md
@@ -1,483 +0,0 @@
|
||||
# Scriptorium Architecture
|
||||
|
||||
## 1. Purpose and Non-Goals
|
||||
|
||||
Scriptorium is a prompt-definition execution engine.
|
||||
|
||||
It accepts named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata.
|
||||
|
||||
Scriptorium also supports rendering/preparing a prompt without calling an LLM. This allows users to inspect the fully rendered prompt messages and effective runtime settings before executing a run.
|
||||
|
||||
Scriptorium is not an orchestrator. It must not own transcription, transcript merge/polish steps, notifications, or cross-step workflow control.
|
||||
|
||||
For the motivating D&D workflow:
|
||||
- Narratio orchestrates.
|
||||
- WhisperX transcribes.
|
||||
- Seriatim merges transcripts.
|
||||
- Audita polishes transcripts.
|
||||
- Scriptorium generates final artifacts from prepared inputs.
|
||||
|
||||
Core Go code remains generic.
|
||||
|
||||
## 2. Current Architecture
|
||||
|
||||
Scriptorium uses a ports-and-adapters architecture to decouple the core execution logic from external dependencies.
|
||||
|
||||
### Package Responsibilities
|
||||
|
||||
- `cmd/scriptorium`: Binary entrypoint for CLI and HTTP server.
|
||||
- `internal/domain`: Core domain contracts, including `PromptDefinition`, `ExecutionProfile`, `PreparedRun`, `RunResult`, and related metadata.
|
||||
- `internal/usecase`: `Runner` use case logic, including prompt preparation, profile selection, runtime override resolution, full run execution, validation, and bounded repair.
|
||||
- `internal/config`: Application-level config model/loader for adapter settings (for example prompt/profile/schema directories, server address, and render format default).
|
||||
- `internal/promptdef`: Repository for loading and validating Prompt Definitions from the filesystem.
|
||||
- `internal/profile`: Repository for loading Execution Profiles from the filesystem.
|
||||
- `internal/artifact`: Input artifact resolution (`inline`, `file`).
|
||||
- `internal/prompt`: Template rendering via Go templates.
|
||||
- `internal/llm`: Provider-neutral client interface and OpenAI-compatible HTTP adapter.
|
||||
- `internal/validate`: Output validation implementation (`none/basic/json/json_schema`).
|
||||
- `internal/adapter/cli`: CLI flag parsing, command dispatch, and output handling.
|
||||
- `internal/adapter/http`: HTTP request/response mapping.
|
||||
- `internal/format` or equivalent: Formatting of prepared/rendered prompt output for CLI or other adapters, if formatting grows beyond simple CLI-local helpers.
|
||||
|
||||
Exact package names may evolve, but the architectural boundaries should remain stable.
|
||||
|
||||
### Application Configuration
|
||||
|
||||
`config.yml` is adapter/application setup, not domain logic.
|
||||
|
||||
Application config is intended for application-level settings such as:
|
||||
- `prompt_dir`
|
||||
- `profile_dir`
|
||||
- `schema_dir`
|
||||
- `server.addr`
|
||||
- `defaults.render_format`
|
||||
|
||||
Application config precedence is:
|
||||
1. CLI flags
|
||||
2. `config.yml`
|
||||
3. Built-in application defaults
|
||||
|
||||
Runtime model settings are intentionally separate:
|
||||
- Execution profiles and runtime overrides continue to own endpoint/model/runtime behavior.
|
||||
- `config.yml` does not replace execution profiles.
|
||||
|
||||
The core use case (`Runner.Prepare`/`Runner.Run`) does not need to know whether adapter-level settings came from CLI flags or `config.yml`; it receives resolved dependencies and requests from adapters.
|
||||
|
||||
## 3. Core Execution Model
|
||||
|
||||
Scriptorium has two closely related execution paths:
|
||||
|
||||
1. Prepare/render path.
|
||||
2. Full run path.
|
||||
|
||||
The full run path should reuse the prepare path rather than duplicating its logic.
|
||||
|
||||
### 3.1 Prepare / Render Data Flow
|
||||
|
||||
The prepare path should be represented in the use case layer, preferably as `Runner.Prepare(ctx, RunRequest)` or an equivalent method.
|
||||
|
||||
It executes all pre-LLM work:
|
||||
|
||||
1. **Validate Request**: Ensure the request includes a prompt ID and all minimum required fields.
|
||||
2. **Load Prompt Definition**: Retrieve the `PromptDefinition` by ID from the prompt repository.
|
||||
3. **Select Profile**: Determine the `profile_id` using this precedence:
|
||||
- Explicit `profile_id` in `RunRequest`.
|
||||
- `default_profile` specified in the `PromptDefinition`.
|
||||
- Error if neither is available.
|
||||
4. **Load Execution Profile**: Retrieve the `ExecutionProfile` from the profile repository.
|
||||
5. **Resolve Runtime Overrides**: Merge settings based on precedence, highest to lowest:
|
||||
- Runtime overrides from CLI flags or HTTP request `model` object.
|
||||
- Execution Profile settings.
|
||||
- Built-in application defaults.
|
||||
6. **Resolve Artifacts**: Load all named input artifacts defined in the request.
|
||||
7. **Render Prompt**: Apply template variables and input artifacts to the prompt templates.
|
||||
8. **Compute Metadata**: Compute hashes, selected profile ID, prompt ID/version, effective runtime settings, input hashes, rendered prompt hash, and timing information as appropriate.
|
||||
9. **Return PreparedRun**: Return a `PreparedRun` containing the rendered messages, effective runtime settings, resolved metadata, and input/prompt hashes.
|
||||
|
||||
The prepare path must not call the LLM.
|
||||
|
||||
The prepare path must not validate model output, because there is no model output.
|
||||
|
||||
The prepare path must not perform structured-output repair, because repair only applies after model output exists.
|
||||
|
||||
The prepare path should not resolve or expose raw API key values. It may include the selected `api_key_env` name in effective runtime settings or metadata, but never the environment variable value.
|
||||
|
||||
### 3.2 Full Run Data Flow
|
||||
|
||||
The `Runner.Run(ctx, RunRequest)` flow should reuse the prepare path:
|
||||
|
||||
1. **Prepare**: Call the shared prepare flow to load the prompt, select the profile, resolve artifacts, render prompt messages, and compute pre-run metadata.
|
||||
2. **Call LLM**: Execute the generation request using the effective runtime settings from the prepared run.
|
||||
3. **Build Output Artifact**: Convert the model response into the configured output artifact.
|
||||
4. **Validate Output**:
|
||||
- Validate the model output against the prompt definition's output contract.
|
||||
- Validation content failures remain successful run results with `validation.status=failed`.
|
||||
- Validator runtime/config errors are run errors.
|
||||
5. **Repair If Configured**:
|
||||
- If structured validation fails and `repair_attempts > 0`, perform bounded repair attempts.
|
||||
- Re-validate after each repair attempt.
|
||||
- Repair loops must remain strictly bounded.
|
||||
6. **Return RunResult**: Produce a `RunResult` containing the final artifact, validation status, raw model output, usage information, and metadata.
|
||||
|
||||
`Runner.Run` should not duplicate profile selection, artifact resolution, or prompt rendering logic that already exists in `Runner.Prepare`.
|
||||
|
||||
## 4. Domain Model
|
||||
|
||||
Key domain types:
|
||||
|
||||
- `PromptDefinition`: Defines the "what" of the task: templates, inputs, output contract, validation settings, repair settings, and optional `default_profile`.
|
||||
- `ExecutionProfile`: Defines the "how" of execution: endpoint, model, generation parameters, timeout, reasoning effort, and `api_key_env`.
|
||||
- `RunRequest`: The intent to execute or prepare a prompt, including `prompt_id`, optional `profile_id`, inputs, variables, and optional runtime overrides.
|
||||
- `PreparedRun`: The result of the prepare/render phase. Contains rendered messages, effective runtime settings, selected profile ID, prompt metadata, input hashes, prompt hash, and other pre-LLM metadata.
|
||||
- `RunResult`: The result of a full run. Contains the generated `Artifact`, `ValidationResult`, raw model output, token usage, and `RunMetadata`.
|
||||
- `RunMetadata`: Detailed tracing information, including prompt ID/version, selected profile ID, effective model parameters, usage tokens, hashes, timestamps, validation status, and repair attempts where applicable.
|
||||
- `RenderedPrompt`: Provider-neutral rendered prompt structure.
|
||||
- `RenderedMessage`: Provider-neutral rendered message with role and content.
|
||||
- `ArtifactRef`: A reference to an input artifact, such as `file` or `inline`.
|
||||
- `Artifact`: Loaded artifact content with name, content type, body, source URI, size, and hash.
|
||||
- `ValidationResult`: Validation status and details for full runs.
|
||||
|
||||
`PreparedRun` should be serializable for JSON output and should also be representable in a human-readable text format.
|
||||
|
||||
## 5. Interfaces and Adapters
|
||||
|
||||
### Primary Ports
|
||||
|
||||
- `promptdef.Repository`: Lookup for prompt definitions.
|
||||
- `profile.Repository`: Lookup for execution profiles.
|
||||
- `artifact.Reader`: Loading of artifact content.
|
||||
- `prompt.Renderer`: Template rendering.
|
||||
- `llm.Client`: Model generation.
|
||||
- `validate.Validator`: Output validation.
|
||||
- `format.PreparedRunFormatter` or equivalent: Optional formatting abstraction for rendered/prepared output.
|
||||
|
||||
### Current Adapters
|
||||
|
||||
- **Repositories**: Filesystem YAML loaders for both prompts and profiles.
|
||||
- **Artifact Reader**: Composite reader supporting `file` and `inline`.
|
||||
- **Prompt Renderer**: Go templates with a custom `input` helper.
|
||||
- **LLM Client**: OpenAI-compatible `/chat/completions` over HTTP.
|
||||
- **Validator**: Standard validator supporting `none`, `basic`, `json`, and `json_schema`.
|
||||
- **CLI Adapter**: Supports `run`, `render`, and `serve`.
|
||||
- **HTTP Adapter**: Supports full run execution through `POST /v1/runs`.
|
||||
|
||||
## 6. Public Contracts
|
||||
|
||||
### CLI
|
||||
|
||||
Scriptorium should expose at least these commands:
|
||||
|
||||
- `scriptorium run`: Executes a prompt by preparing it, calling the LLM, validating output, optionally repairing structured output, and returning an artifact.
|
||||
- `scriptorium render`: Prepares and renders a prompt without calling the LLM.
|
||||
- `scriptorium serve`: Starts the HTTP API using infrastructure-only flags.
|
||||
|
||||
### `scriptorium run`
|
||||
|
||||
`run` uses flags such as:
|
||||
|
||||
- `--prompt-dir`
|
||||
- `--profile-dir`
|
||||
- `--prompt`
|
||||
- `--profile`
|
||||
- `--input`
|
||||
- `--var`
|
||||
- `--out`
|
||||
- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.
|
||||
|
||||
`run` should produce the generated artifact as its primary output.
|
||||
|
||||
### `scriptorium render`
|
||||
|
||||
`render` prepares and renders a prompt without calling an LLM.
|
||||
|
||||
It should use the same prompt/profile/input/variable/runtime override flags as `run` where applicable:
|
||||
|
||||
- `--prompt-dir`
|
||||
- `--profile-dir`
|
||||
- `--prompt`
|
||||
- `--profile`
|
||||
- `--input`
|
||||
- `--var`
|
||||
- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.
|
||||
- `--format`, with initial support for `text` and `json`.
|
||||
|
||||
Default render output format should be `text`.
|
||||
|
||||
`render` must not call the LLM.
|
||||
|
||||
`render` should show the same rendered messages and effective runtime settings that `run` would use.
|
||||
|
||||
`render` should include enough information to debug:
|
||||
|
||||
- prompt ID
|
||||
- prompt version
|
||||
- selected profile ID
|
||||
- effective runtime settings
|
||||
- input hashes
|
||||
- prompt hash
|
||||
- rendered messages
|
||||
|
||||
`render` must not include resolved API key values.
|
||||
|
||||
`render` may include the `api_key_env` name.
|
||||
|
||||
### Render Output Formats
|
||||
|
||||
Initial render output formats:
|
||||
|
||||
- `text`: Human-readable default format.
|
||||
- `json`: Machine-readable structured representation of the prepared run.
|
||||
|
||||
Additional formats, such as `markdown`, may be added later.
|
||||
|
||||
Render output formatting should be modular. Adding a new output format should not require changing the prepare/run core logic.
|
||||
|
||||
The output formatting layer should consume a `PreparedRun` and produce bytes or text for the adapter. It should not reload prompts, re-resolve artifacts, re-render templates, call the LLM, or perform validation.
|
||||
|
||||
### `scriptorium serve`
|
||||
|
||||
`serve` starts the HTTP API.
|
||||
|
||||
It should use infrastructure-only flags such as:
|
||||
|
||||
- `--addr`
|
||||
- `--prompt-dir`
|
||||
- `--profile-dir`
|
||||
- `--schema-dir`
|
||||
|
||||
`serve` should not introduce a server-level model/runtime precedence layer unless explicitly documented and intentionally implemented.
|
||||
|
||||
### HTTP API
|
||||
|
||||
Current HTTP API:
|
||||
|
||||
- `POST /v1/runs`: Accepts `RunRequest` JSON and returns `RunResponse` JSON. No built-in auth.
|
||||
|
||||
Request may include runtime overrides under `model` and an `include_raw_output` boolean.
|
||||
|
||||
`raw_model_output` is exposed only when explicitly requested with `include_raw_output=true`.
|
||||
|
||||
A future HTTP prepare/render endpoint may be added, such as `POST /v1/renders` or `POST /v1/runs/prepare`, but the initial render feature may be CLI-only. If added later, it should call the same usecase-level prepare path as `scriptorium render`.
|
||||
|
||||
### YAML Shapes
|
||||
|
||||
Prompt YAML includes:
|
||||
|
||||
- `id`
|
||||
- `version`
|
||||
- optional `default_profile`
|
||||
- `inputs`
|
||||
- `messages`
|
||||
- `output`
|
||||
|
||||
Inputs support:
|
||||
|
||||
- `name`
|
||||
- `required`
|
||||
- optional `content_type`
|
||||
- `description`
|
||||
|
||||
Messages require:
|
||||
|
||||
- `role`
|
||||
- exactly one of `content` or `content_file`
|
||||
|
||||
`content_file` resolves relative to the prompt YAML location.
|
||||
|
||||
Profile YAML includes:
|
||||
|
||||
- `id`
|
||||
- `endpoint`
|
||||
- `model`
|
||||
- generation parameters
|
||||
- timeout settings
|
||||
- reasoning settings
|
||||
- `api_key_env`
|
||||
|
||||
Prompt content must not appear in profile YAML.
|
||||
|
||||
Model/runtime/API-key settings must not appear in prompt YAML, except that prompt YAML may specify `default_profile`.
|
||||
|
||||
## 7. Render Feature Design
|
||||
|
||||
The render feature is a first-class use case, not a CLI-only shortcut.
|
||||
|
||||
### Goals
|
||||
|
||||
The render feature should help users:
|
||||
|
||||
- inspect fully rendered prompt messages
|
||||
- debug missing inputs
|
||||
- verify template variable substitution
|
||||
- verify selected profile resolution
|
||||
- verify runtime override precedence
|
||||
- verify file-backed prompt loading
|
||||
- inspect input hashes and prompt hashes
|
||||
- prepare for future token budgeting and prompt-size inspection
|
||||
|
||||
### Non-Goals
|
||||
|
||||
The render feature should not:
|
||||
|
||||
- call an LLM
|
||||
- validate model output
|
||||
- repair structured output
|
||||
- resolve or print API key values
|
||||
- mutate artifacts
|
||||
- save outputs to artifact storage unless a future explicit output option is added
|
||||
- become an orchestration step manager
|
||||
|
||||
### Usecase Shape
|
||||
|
||||
The preferred usecase shape is:
|
||||
|
||||
- `Runner.Prepare(ctx, RunRequest) (*PreparedRun, error)`
|
||||
- `Runner.Run(ctx, RunRequest) (*RunResult, error)`
|
||||
|
||||
`Runner.Run` should call `Runner.Prepare`.
|
||||
|
||||
The prepare flow should be the only implementation of:
|
||||
|
||||
- prompt loading
|
||||
- profile selection
|
||||
- runtime override resolution
|
||||
- artifact resolution
|
||||
- prompt rendering
|
||||
- pre-run metadata/hash calculation
|
||||
|
||||
### CLI Shape
|
||||
|
||||
The preferred command name is `render`.
|
||||
|
||||
The command should support:
|
||||
|
||||
- `--format text`
|
||||
- `--format json`
|
||||
|
||||
Default format:
|
||||
|
||||
- `text`
|
||||
|
||||
Unknown formats should produce a clear error.
|
||||
|
||||
Formatting should be centralized through a small formatter registry, strategy, switch, or interface so new formats can be added without modifying usecase logic.
|
||||
|
||||
### Text Output Expectations
|
||||
|
||||
Text output should be optimized for human inspection.
|
||||
|
||||
It should include, at minimum:
|
||||
|
||||
- prompt ID and version
|
||||
- selected profile ID
|
||||
- model name
|
||||
- endpoint
|
||||
- effective generation settings
|
||||
- input hashes
|
||||
- rendered prompt hash
|
||||
- rendered messages grouped by role
|
||||
|
||||
Text output should be readable and deterministic enough for tests.
|
||||
|
||||
It should not include raw API key values.
|
||||
|
||||
### JSON Output Expectations
|
||||
|
||||
JSON output should be a structured representation of `PreparedRun` or a DTO derived from it.
|
||||
|
||||
It should include, at minimum:
|
||||
|
||||
- prompt ID and version
|
||||
- selected profile ID
|
||||
- effective runtime settings
|
||||
- input hashes
|
||||
- rendered prompt hash
|
||||
- rendered messages
|
||||
|
||||
JSON output should not include raw API key values.
|
||||
|
||||
JSON output should remain stable enough to be useful for automation and integration tests.
|
||||
|
||||
## 8. Guardrails
|
||||
|
||||
- **Separation of Concerns**: Prompt content must not belong in execution profiles; model/API settings must not belong in prompt definitions.
|
||||
- **Security**: Raw API keys are unsupported in all configuration and transport layers. Only `api_key_env` is used.
|
||||
- **Secret Handling**: Resolved API key values must never appear in rendered output, metadata, logs, HTTP responses, or CLI output.
|
||||
- **Path Resolution**: `content_file` paths in prompt definitions resolve relative to the prompt YAML file.
|
||||
- **Integrity**: No silent prompt truncation or omission of content.
|
||||
- **Reliability**: Repair loops are strictly bounded by `repair_attempts`.
|
||||
- **No Orchestration Creep**: Scriptorium prepares and executes a single prompt request. It does not coordinate multi-stage workflows.
|
||||
- **Render Reuse**: The full run path must reuse the prepare/render path to avoid divergent behavior.
|
||||
- **Formatter Isolation**: Render output formatters must not perform usecase work. They only format a completed `PreparedRun`.
|
||||
|
||||
## 9. Testing Strategy
|
||||
|
||||
Tests should protect both the run path and the prepare/render path.
|
||||
|
||||
### Prepare / Render Tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- preparing a prompt with explicit profile selection
|
||||
- preparing a prompt using `default_profile`
|
||||
- failing when no explicit profile and no `default_profile` exist
|
||||
- runtime overrides beating profile values
|
||||
- profile values beating application defaults
|
||||
- file-backed prompt bodies rendering correctly
|
||||
- required inputs failing when missing
|
||||
- optional inputs being absent when not referenced
|
||||
- unknown input references failing
|
||||
- input hashes being included
|
||||
- rendered prompt hash being included
|
||||
- effective runtime settings being included
|
||||
- `api_key_env` name being included where appropriate
|
||||
- resolved API key values never appearing in `PreparedRun`
|
||||
- prepare path not calling the LLM
|
||||
|
||||
### CLI Render Tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- `scriptorium render` mapping flags into `RunRequest`
|
||||
- default text output
|
||||
- explicit `--format text`
|
||||
- explicit `--format json`
|
||||
- unknown format failure
|
||||
- text output includes prompt/profile/messages
|
||||
- JSON output includes prompt/profile/messages
|
||||
- rendered output never includes resolved API key values
|
||||
|
||||
### Run Reuse Tests
|
||||
|
||||
Add tests proving:
|
||||
|
||||
- `Runner.Run` reuses prepare behavior
|
||||
- run and render resolve the same prompt/profile/runtime settings for equivalent inputs
|
||||
- run still validates output
|
||||
- run still performs bounded repair where configured
|
||||
|
||||
### Existing Tests
|
||||
|
||||
Continue testing:
|
||||
|
||||
- prompt definition loading
|
||||
- execution profile loading
|
||||
- artifact loading
|
||||
- prompt rendering
|
||||
- LLM adapter behavior
|
||||
- validation behavior
|
||||
- HTTP request/response mapping
|
||||
|
||||
## 10. Extension Points
|
||||
|
||||
Future work should remain grounded in the current architecture:
|
||||
|
||||
- **Artifacts**: Add S3 artifact references via a new `artifact.Reader`.
|
||||
- **LLM**: Implement additional provider adapters, such as Anthropic or Google.
|
||||
- **Execution**: Add token budgeting, streaming generation, and batch execution capabilities.
|
||||
- **Prepare/Render**: Add token estimates, prompt-size summaries, or additional render output formats.
|
||||
- **Repositories**: Implement database-backed repositories for prompts and profiles.
|
||||
- **Profiles**: Support more granular profile versioning and environment-specific profiles.
|
||||
- **HTTP**: Add an HTTP prepare/render endpoint if Narratio or another caller needs it.
|
||||
|
||||
Future render formats should plug into the formatter layer and should not require changes to the usecase layer.
|
||||
39
artifact_reader.go
Normal file
39
artifact_reader.go
Normal file
@@ -0,0 +1,39 @@
|
||||
package scriptorium
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
|
||||
artifactadapter "gitea.maximumdirect.net/eric/scriptorium/internal/artifact"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||
)
|
||||
|
||||
var errNilArtifactReaderResponse = errors.New("artifact reader returned nil artifact without error")
|
||||
|
||||
type publicArtifactReaderAdapter struct {
|
||||
reader ArtifactReader
|
||||
}
|
||||
|
||||
var _ artifactadapter.Reader = publicArtifactReaderAdapter{}
|
||||
|
||||
func (a publicArtifactReaderAdapter) Read(ctx context.Context, ref domain.ArtifactRef) (*domain.Artifact, error) {
|
||||
artifact, err := a.reader.Read(ctx, ArtifactRef{
|
||||
Type: ArtifactRefType(ref.Type),
|
||||
URI: ref.URI,
|
||||
Body: ref.Body,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if artifact == nil {
|
||||
return nil, errNilArtifactReaderResponse
|
||||
}
|
||||
return &domain.Artifact{
|
||||
Name: artifact.Name,
|
||||
ContentType: artifact.ContentType,
|
||||
Body: copyBytes(artifact.Body),
|
||||
URI: artifact.URI,
|
||||
Size: artifact.Size,
|
||||
Hash: artifact.Hash,
|
||||
}, nil
|
||||
}
|
||||
36
artifact_reader_internal_test.go
Normal file
36
artifact_reader_internal_test.go
Normal file
@@ -0,0 +1,36 @@
|
||||
package scriptorium
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||
)
|
||||
|
||||
func TestPublicArtifactReaderAdapterCopiesBody(t *testing.T) {
|
||||
reader := internalArtifactReaderFake{
|
||||
artifact: &Artifact{Body: []byte("original")},
|
||||
}
|
||||
adapter := publicArtifactReaderAdapter{reader: &reader}
|
||||
|
||||
artifact, err := adapter.Read(context.Background(), domain.ArtifactRef{
|
||||
Type: domain.ArtifactRefInline,
|
||||
URI: "memory://input",
|
||||
Body: "input",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("read artifact: %v", err)
|
||||
}
|
||||
artifact.Body[0] = 'X'
|
||||
if got := string(reader.artifact.Body); got != "original" {
|
||||
t.Fatalf("reader artifact body was mutated: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
type internalArtifactReaderFake struct {
|
||||
artifact *Artifact
|
||||
}
|
||||
|
||||
func (r *internalArtifactReaderFake) Read(context.Context, ArtifactRef) (*Artifact, error) {
|
||||
return r.artifact, nil
|
||||
}
|
||||
406
convert.go
Normal file
406
convert.go
Normal file
@@ -0,0 +1,406 @@
|
||||
package scriptorium
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||
)
|
||||
|
||||
func toDomainRunRequest(req RunRequest) (domain.RunRequest, error) {
|
||||
execution, err := toDomainExecutionTargetOverride(req.Execution)
|
||||
if err != nil {
|
||||
return domain.RunRequest{}, err
|
||||
}
|
||||
return domain.RunRequest{
|
||||
PromptID: req.PromptID,
|
||||
PromptVersion: req.PromptVersion,
|
||||
ProfileID: req.ProfileID,
|
||||
APIKey: req.APIKey,
|
||||
Inputs: toDomainArtifactRefMap(req.Inputs),
|
||||
Vars: copyStringMap(req.Vars),
|
||||
Execution: execution,
|
||||
Validation: toDomainOutputContractPtr(req.Validation),
|
||||
Metadata: copyStringMap(req.Metadata),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func fromDomainPreparedRun(prepared *domain.PreparedRun) *PreparedRun {
|
||||
if prepared == nil {
|
||||
return nil
|
||||
}
|
||||
return &PreparedRun{
|
||||
PromptID: prepared.PromptID,
|
||||
PromptVersion: prepared.PromptVersion,
|
||||
PromptHash: prepared.PromptHash,
|
||||
SelectedProfileID: prepared.SelectedProfileID,
|
||||
EffectiveModelParams: fromDomainExecutionTarget(prepared.EffectiveModelParams),
|
||||
OutputContract: fromDomainOutputContract(prepared.OutputContract),
|
||||
StructuredOutput: fromDomainStructuredOutputSpec(prepared.StructuredOutput),
|
||||
InputHashes: copyStringMap(prepared.InputHashes),
|
||||
SessionID: prepared.SessionID,
|
||||
RenderedPromptHash: prepared.RenderedPromptHash,
|
||||
Messages: fromDomainRenderedMessages(prepared.Messages),
|
||||
StartTime: prepared.StartTime,
|
||||
EndTime: prepared.EndTime,
|
||||
DurationMS: prepared.DurationMS,
|
||||
}
|
||||
}
|
||||
|
||||
func fromDomainRunResult(result *domain.RunResult) *RunResult {
|
||||
if result == nil {
|
||||
return nil
|
||||
}
|
||||
return &RunResult{
|
||||
RunID: result.RunID,
|
||||
Artifact: fromDomainArtifact(result.Artifact),
|
||||
RawOutput: result.RawOutput,
|
||||
Validation: fromDomainValidationResult(result.Validation),
|
||||
PromptID: result.PromptID,
|
||||
PromptVersion: result.PromptVersion,
|
||||
PromptHash: result.PromptHash,
|
||||
RenderedPromptHash: result.RenderedPromptHash,
|
||||
SelectedProfileID: result.SelectedProfileID,
|
||||
ModelName: result.ModelName,
|
||||
Endpoint: result.Endpoint,
|
||||
EffectiveModelParams: fromDomainExecutionTarget(result.EffectiveModelParams),
|
||||
InputHashes: copyStringMap(result.InputHashes),
|
||||
Usage: fromDomainTokenUsage(result.Usage),
|
||||
StartTime: result.StartTime,
|
||||
EndTime: result.EndTime,
|
||||
Duration: result.Duration,
|
||||
}
|
||||
}
|
||||
|
||||
func fromDomainGenerateRequest(req domain.GenerateRequest) GenerateRequest {
|
||||
return GenerateRequest{
|
||||
Prompt: fromDomainRenderedPrompt(req.Prompt),
|
||||
Target: fromDomainExecutionTarget(req.Target),
|
||||
TargetPresence: fromDomainExecutionTargetPresence(req.TargetPresence),
|
||||
StructuredOutput: fromDomainStructuredOutputSpec(req.StructuredOutput),
|
||||
APIKey: req.Target.APIKey,
|
||||
}
|
||||
}
|
||||
|
||||
func toDomainGenerateResponse(resp *GenerateResponse) *domain.GenerateResponse {
|
||||
if resp == nil {
|
||||
return nil
|
||||
}
|
||||
return &domain.GenerateResponse{
|
||||
Content: resp.Content,
|
||||
Usage: toDomainTokenUsage(resp.Usage),
|
||||
}
|
||||
}
|
||||
|
||||
func fromDomainRenderedPrompt(prompt domain.RenderedPrompt) RenderedPrompt {
|
||||
return RenderedPrompt{
|
||||
SessionID: prompt.SessionID,
|
||||
Messages: fromDomainRenderedMessages(prompt.Messages),
|
||||
}
|
||||
}
|
||||
|
||||
func toDomainArtifactRefMap(src map[string]ArtifactRef) map[string]domain.ArtifactRef {
|
||||
if src == nil {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]domain.ArtifactRef, len(src))
|
||||
for k, v := range src {
|
||||
out[k] = toDomainArtifactRef(v)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func toDomainArtifactRef(ref ArtifactRef) domain.ArtifactRef {
|
||||
return domain.ArtifactRef{
|
||||
Type: domain.ArtifactRefType(ref.Type),
|
||||
URI: ref.URI,
|
||||
Body: ref.Body,
|
||||
}
|
||||
}
|
||||
|
||||
func fromDomainArtifact(artifact domain.Artifact) Artifact {
|
||||
return Artifact{
|
||||
Name: artifact.Name,
|
||||
ContentType: artifact.ContentType,
|
||||
Body: copyBytes(artifact.Body),
|
||||
URI: artifact.URI,
|
||||
Size: artifact.Size,
|
||||
Hash: artifact.Hash,
|
||||
}
|
||||
}
|
||||
|
||||
func toDomainExecutionTargetOverride(override *ExecutionTargetOverride) (*domain.ExecutionTargetOverride, error) {
|
||||
if override == nil {
|
||||
return nil, nil
|
||||
}
|
||||
extraParams, err := copyPublicJSONMap(override.ExtraParams)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &domain.ExecutionTargetOverride{
|
||||
Endpoint: override.Endpoint,
|
||||
Model: override.Model,
|
||||
Temperature: copyFloat64Ptr(override.Temperature),
|
||||
MaxTokens: copyIntPtr(override.MaxTokens),
|
||||
TopP: copyFloat64Ptr(override.TopP),
|
||||
TimeoutSeconds: copyIntPtr(override.TimeoutSeconds),
|
||||
ServiceTier: override.ServiceTier,
|
||||
ReasoningEffort: override.ReasoningEffort,
|
||||
APIKeyEnv: override.APIKeyEnv,
|
||||
ExtraParams: extraParams,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func fromDomainExecutionTarget(target domain.ExecutionTarget) ExecutionTarget {
|
||||
return ExecutionTarget{
|
||||
Endpoint: target.Endpoint,
|
||||
Model: target.Model,
|
||||
Temperature: target.Temperature,
|
||||
MaxTokens: target.MaxTokens,
|
||||
TopP: target.TopP,
|
||||
TimeoutSeconds: target.TimeoutSeconds,
|
||||
ServiceTier: target.ServiceTier,
|
||||
ReasoningEffort: target.ReasoningEffort,
|
||||
APIKeyEnv: target.APIKeyEnv,
|
||||
ExtraParams: copyAnyMap(target.ExtraParams),
|
||||
}
|
||||
}
|
||||
|
||||
func fromDomainExecutionTargetPresence(presence domain.ExecutionTargetPresence) ExecutionTargetPresence {
|
||||
return ExecutionTargetPresence{
|
||||
Temperature: presence.Temperature,
|
||||
MaxTokens: presence.MaxTokens,
|
||||
TopP: presence.TopP,
|
||||
TimeoutSeconds: presence.TimeoutSeconds,
|
||||
}
|
||||
}
|
||||
|
||||
func toDomainOutputContractPtr(contract *OutputContract) *domain.OutputContract {
|
||||
if contract == nil {
|
||||
return nil
|
||||
}
|
||||
out := toDomainOutputContract(*contract)
|
||||
return &out
|
||||
}
|
||||
|
||||
func toDomainOutputContract(contract OutputContract) domain.OutputContract {
|
||||
return domain.OutputContract{
|
||||
Format: domain.OutputFormat(contract.Format),
|
||||
ValidationMode: domain.ValidationMode(contract.ValidationMode),
|
||||
SchemaPath: contract.SchemaPath,
|
||||
RepairAttempts: contract.RepairAttempts,
|
||||
}
|
||||
}
|
||||
|
||||
func fromDomainOutputContract(contract domain.OutputContract) OutputContract {
|
||||
return OutputContract{
|
||||
Format: OutputFormat(contract.Format),
|
||||
ValidationMode: ValidationMode(contract.ValidationMode),
|
||||
SchemaPath: contract.SchemaPath,
|
||||
RepairAttempts: contract.RepairAttempts,
|
||||
}
|
||||
}
|
||||
|
||||
func fromDomainValidationResult(result domain.ValidationResult) ValidationResult {
|
||||
return ValidationResult{
|
||||
Status: ValidationStatus(result.Status),
|
||||
Mode: ValidationMode(result.Mode),
|
||||
Errors: copyStringSlice(result.Errors),
|
||||
SchemaPath: result.SchemaPath,
|
||||
RepairAttempts: result.RepairAttempts,
|
||||
IsValid: result.IsValid,
|
||||
}
|
||||
}
|
||||
|
||||
func fromDomainTokenUsage(usage domain.TokenUsage) TokenUsage {
|
||||
return TokenUsage{
|
||||
PromptTokens: usage.PromptTokens,
|
||||
CompletionTokens: usage.CompletionTokens,
|
||||
TotalTokens: usage.TotalTokens,
|
||||
CachedTokens: usage.CachedTokens,
|
||||
CacheWriteTokens: usage.CacheWriteTokens,
|
||||
}
|
||||
}
|
||||
|
||||
func toDomainTokenUsage(usage TokenUsage) domain.TokenUsage {
|
||||
return domain.TokenUsage{
|
||||
PromptTokens: usage.PromptTokens,
|
||||
CompletionTokens: usage.CompletionTokens,
|
||||
TotalTokens: usage.TotalTokens,
|
||||
CachedTokens: usage.CachedTokens,
|
||||
CacheWriteTokens: usage.CacheWriteTokens,
|
||||
}
|
||||
}
|
||||
|
||||
func fromDomainRenderedMessages(messages []domain.RenderedMessage) []RenderedMessage {
|
||||
if messages == nil {
|
||||
return nil
|
||||
}
|
||||
out := make([]RenderedMessage, len(messages))
|
||||
for i, msg := range messages {
|
||||
out[i] = RenderedMessage{
|
||||
Role: msg.Role,
|
||||
Content: msg.Content,
|
||||
CacheControl: fromDomainCacheControl(msg.CacheControl),
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func fromDomainCacheControl(cacheControl *domain.CacheControl) *CacheControl {
|
||||
if cacheControl == nil {
|
||||
return nil
|
||||
}
|
||||
return &CacheControl{
|
||||
Type: CacheControlType(cacheControl.Type),
|
||||
TTL: cacheControl.TTL,
|
||||
}
|
||||
}
|
||||
|
||||
func fromDomainStructuredOutputSpec(spec *domain.StructuredOutputSpec) *StructuredOutputSpec {
|
||||
if spec == nil {
|
||||
return nil
|
||||
}
|
||||
out := &StructuredOutputSpec{
|
||||
Type: StructuredOutputType(spec.Type),
|
||||
}
|
||||
if spec.JSONSchema != nil {
|
||||
out.JSONSchema = &StructuredOutputJSONSpec{
|
||||
Name: spec.JSONSchema.Name,
|
||||
Strict: spec.JSONSchema.Strict,
|
||||
Schema: copyAny(spec.JSONSchema.Schema),
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func copyStringMap(src map[string]string) map[string]string {
|
||||
if src == nil {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]string, len(src))
|
||||
for k, v := range src {
|
||||
out[k] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func copyAnyMap(src map[string]any) map[string]any {
|
||||
if src == nil {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]any, len(src))
|
||||
for k, v := range src {
|
||||
out[k] = copyAny(v)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func copyAny(value any) any {
|
||||
if value == nil {
|
||||
return nil
|
||||
}
|
||||
switch v := value.(type) {
|
||||
case map[string]any:
|
||||
return copyAnyMap(v)
|
||||
case []any:
|
||||
out := make([]any, len(v))
|
||||
for i, item := range v {
|
||||
out[i] = copyAny(item)
|
||||
}
|
||||
return out
|
||||
case []string:
|
||||
return copyStringSlice(v)
|
||||
case []byte:
|
||||
return copyBytes(v)
|
||||
default:
|
||||
return copyReflectValue(reflect.ValueOf(value)).Interface()
|
||||
}
|
||||
}
|
||||
|
||||
func copyReflectValue(value reflect.Value) reflect.Value {
|
||||
if !value.IsValid() {
|
||||
return value
|
||||
}
|
||||
|
||||
switch value.Kind() {
|
||||
case reflect.Interface:
|
||||
if value.IsNil() {
|
||||
return reflect.Zero(value.Type())
|
||||
}
|
||||
copied := copyReflectValue(value.Elem())
|
||||
if copied.IsValid() && copied.Type().AssignableTo(value.Type()) {
|
||||
return copied
|
||||
}
|
||||
out := reflect.New(value.Type()).Elem()
|
||||
out.Set(copied)
|
||||
return out
|
||||
case reflect.Pointer:
|
||||
if value.IsNil() {
|
||||
return reflect.Zero(value.Type())
|
||||
}
|
||||
out := reflect.New(value.Type().Elem())
|
||||
out.Elem().Set(copyReflectValue(value.Elem()))
|
||||
return out
|
||||
case reflect.Map:
|
||||
if value.IsNil() {
|
||||
return reflect.Zero(value.Type())
|
||||
}
|
||||
out := reflect.MakeMapWithSize(value.Type(), value.Len())
|
||||
iter := value.MapRange()
|
||||
for iter.Next() {
|
||||
out.SetMapIndex(copyReflectValue(iter.Key()), copyReflectValue(iter.Value()))
|
||||
}
|
||||
return out
|
||||
case reflect.Slice:
|
||||
if value.IsNil() {
|
||||
return reflect.Zero(value.Type())
|
||||
}
|
||||
out := reflect.MakeSlice(value.Type(), value.Len(), value.Cap())
|
||||
for i := 0; i < value.Len(); i++ {
|
||||
out.Index(i).Set(copyReflectValue(value.Index(i)))
|
||||
}
|
||||
return out
|
||||
case reflect.Array:
|
||||
out := reflect.New(value.Type()).Elem()
|
||||
for i := 0; i < value.Len(); i++ {
|
||||
out.Index(i).Set(copyReflectValue(value.Index(i)))
|
||||
}
|
||||
return out
|
||||
default:
|
||||
return value
|
||||
}
|
||||
}
|
||||
|
||||
func copyStringSlice(src []string) []string {
|
||||
if src == nil {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, len(src))
|
||||
copy(out, src)
|
||||
return out
|
||||
}
|
||||
|
||||
func copyBytes(src []byte) []byte {
|
||||
if src == nil {
|
||||
return nil
|
||||
}
|
||||
out := make([]byte, len(src))
|
||||
copy(out, src)
|
||||
return out
|
||||
}
|
||||
|
||||
func copyFloat64Ptr(src *float64) *float64 {
|
||||
if src == nil {
|
||||
return nil
|
||||
}
|
||||
v := *src
|
||||
return &v
|
||||
}
|
||||
|
||||
func copyIntPtr(src *int) *int {
|
||||
if src == nil {
|
||||
return nil
|
||||
}
|
||||
v := *src
|
||||
return &v
|
||||
}
|
||||
48
docs/adr/0001-adopt-canonical-documentation-ownership.md
Normal file
48
docs/adr/0001-adopt-canonical-documentation-ownership.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# ADR 0001: Adopt Canonical Documentation Ownership
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
## Date
|
||||
|
||||
2026-07-26
|
||||
|
||||
## Context
|
||||
|
||||
Scriptorium's documentation grew alongside its CLI, HTTP, public Go, and
|
||||
integration interfaces. As a result, several documents repeated mutable
|
||||
contracts such as flags, configuration fields, and status behavior. Those
|
||||
parallel definitions made it unclear which document to update when behavior
|
||||
changed and increased the risk of documentation drift.
|
||||
|
||||
## Decision
|
||||
|
||||
Assign each documentation topic one canonical owner, as defined in
|
||||
[`docs/policy/documentation.md`](../policy/documentation.md). Non-owning
|
||||
documents may provide short orientation and links, but do not redefine volatile
|
||||
contracts. Current behavior is documented outside `docs/roadmap/`; roadmaps own
|
||||
future work, sequencing, and implementation status.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
- Keep broad reference material in several audience-specific documents. This
|
||||
would preserve local convenience but leave conflicting contract definitions
|
||||
likely.
|
||||
- Consolidate all documentation into one reference. This would reduce duplicate
|
||||
text but would not serve the distinct needs of users, operators, consumers,
|
||||
and contributors.
|
||||
|
||||
## Rationale
|
||||
|
||||
Canonical ownership retains audience-specific guidance while making the source
|
||||
of truth for each contract discoverable. It also makes documentation changes
|
||||
reviewable alongside the implementation change that requires them.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Changes to behavior must update the canonical owner in the same change.
|
||||
- Cross-cutting documentation links to the owner instead of copying its
|
||||
details.
|
||||
- Documentation restructuring followed a dedicated implementation roadmap;
|
||||
repository history, not this ADR, records its completion.
|
||||
288
docs/adr/0002-split-promptkit-from-scriptorium.md
Normal file
288
docs/adr/0002-split-promptkit-from-scriptorium.md
Normal file
@@ -0,0 +1,288 @@
|
||||
# ADR 0002: Split Promptkit From Scriptorium
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
## Date
|
||||
|
||||
2026-07-26
|
||||
|
||||
## Context
|
||||
|
||||
Scriptorium currently combines two products in one Go module:
|
||||
|
||||
- a reusable prompt-execution framework with a public Go facade; and
|
||||
- a runnable application with CLI and HTTP interfaces.
|
||||
|
||||
Downstream Go projects increasingly import the framework directly and do not
|
||||
use the executable interfaces. Keeping both products in one module couples
|
||||
framework releases, dependencies, documentation, and public API evolution to
|
||||
application-specific transport concerns.
|
||||
|
||||
Promptkit will become the framework project, and Scriptorium will become a slim
|
||||
application that consumes it. This ADR records that end-state boundary. It does
|
||||
not assert that the split has been implemented; until then, the current
|
||||
repository structure and contracts remain authoritative.
|
||||
|
||||
## Decision
|
||||
|
||||
### Projects And Module Paths
|
||||
|
||||
Create a repository named `promptkit` alongside Scriptorium:
|
||||
|
||||
| Project | Repository and Go module path | Root Go package |
|
||||
| --- | --- | --- |
|
||||
| Promptkit | `gitea.maximumdirect.net/eric/promptkit` | `promptkit` |
|
||||
| Scriptorium | `gitea.maximumdirect.net/eric/scriptorium` | No reusable root facade after migration |
|
||||
|
||||
Promptkit will expose its supported public API from the module root. Its
|
||||
implementation packages will remain under `internal/` unless a real consumer
|
||||
extension point requires a public type or interface.
|
||||
|
||||
Scriptorium will import only Promptkit's supported public packages. It will not
|
||||
import Promptkit implementation packages or reproduce Promptkit orchestration.
|
||||
|
||||
### Product Responsibilities
|
||||
|
||||
Promptkit owns application-neutral framework behavior:
|
||||
|
||||
- the engine and its `Prepare` and `Run` workflow;
|
||||
- public request, result, profile, option, extension, and error APIs;
|
||||
- prompt-definition loading and rendering;
|
||||
- profile loading, overlays, and the embedded built-in profile registry;
|
||||
- schema loading and output validation;
|
||||
- provider-neutral model-client boundaries and the OpenAI-compatible client;
|
||||
- artifact types, artifact-reader injection, and general-purpose inline and
|
||||
caller-selected file readers;
|
||||
- execution-setting resolution and framework defaults; and
|
||||
- framework-level secret redaction and error classification.
|
||||
|
||||
Scriptorium owns executable and transport behavior:
|
||||
|
||||
- the `scriptorium` process and its `run`, `render`, and `serve` commands;
|
||||
- CLI parsing, streams, output files, formatting, exit codes, and process
|
||||
cancellation behavior;
|
||||
- application-configuration discovery and CLI-over-configuration precedence;
|
||||
- HTTP routing, strict request decoding, DTO mapping, response encoding,
|
||||
status codes, and transport limits;
|
||||
- HTTP artifact-root containment and deployment policy;
|
||||
- server construction, server defaults, and process logging; and
|
||||
- executable release artifacts.
|
||||
|
||||
The dependency direction is:
|
||||
|
||||
```text
|
||||
Scriptorium CLI and HTTP adapters
|
||||
|
|
||||
v
|
||||
Promptkit public API
|
||||
|
|
||||
v
|
||||
injected sources, readers, and model clients
|
||||
```
|
||||
|
||||
### Current Package Disposition
|
||||
|
||||
Implementation may reorganize files during extraction, but each current package
|
||||
has this target owner:
|
||||
|
||||
| Current package or file group | Target owner | Disposition |
|
||||
| --- | --- | --- |
|
||||
| Root `scriptorium` facade files and tests | Promptkit | Move and rename the public package to `promptkit`; Scriptorium retains no compatibility facade. |
|
||||
| `internal/domain`, `internal/usecase` | Promptkit | Move as internal engine implementation. |
|
||||
| `internal/promptdef`, `internal/prompt` | Promptkit | Move as internal prompt loading and rendering. |
|
||||
| `internal/profile`, `internal/profile/builtin` | Promptkit | Move with embedded built-in assets and registry tests. |
|
||||
| `internal/filecatalog` | Promptkit | Move as source-loading support. |
|
||||
| `internal/validate` | Promptkit | Move as schema and output-validation implementation. |
|
||||
| `internal/llm` | Promptkit | Move with the OpenAI-compatible integration. |
|
||||
| `internal/artifact` | Split | Move general inline/file reading to Promptkit; keep rooted, denied, and byte-limited HTTP file reading in Scriptorium behind a Promptkit reader interface. |
|
||||
| `internal/defaults` | Split | Move framework, execution, output-artifact, content-type, and model-client defaults to Promptkit; keep CLI, HTTP, and server defaults in Scriptorium. |
|
||||
| `internal/adapter/cli`, `internal/adapter/http` | Scriptorium | Keep and refactor to use Promptkit's public API. |
|
||||
| `internal/config` | Scriptorium | Keep application settings, discovery, validation, and CLI precedence. |
|
||||
| `internal/format` | Scriptorium | Keep prepared-run presentation, rewritten against Promptkit public values. |
|
||||
| `cmd/scriptorium` | Scriptorium | Keep as the process entrypoint. |
|
||||
|
||||
Tests move with the behavior they protect. Cross-boundary tests will live with
|
||||
the consuming side: Promptkit protects framework contracts, while Scriptorium
|
||||
protects adapter mapping, HTTP containment, and executable behavior.
|
||||
|
||||
### Public Boundary
|
||||
|
||||
Promptkit's initial facade will preserve the useful shape of the current
|
||||
Scriptorium Go API where that reduces extraction risk. It will expose only the
|
||||
capabilities required by Promptkit consumers and by Scriptorium:
|
||||
|
||||
- engine construction, preparation, and execution;
|
||||
- public request, result, profile, and error values;
|
||||
- prompt, profile, schema, artifact-reader, validator, and model-client source
|
||||
or injection options that have demonstrated consumers; and
|
||||
- enough stable error identity for Scriptorium to map CLI and HTTP outcomes.
|
||||
|
||||
Promptkit will not export its domain package, runner implementation,
|
||||
repositories, adapter DTOs, or general internal constructors merely to
|
||||
simplify the move.
|
||||
|
||||
Scriptorium's CLI and HTTP adapters will depend on a small consumer-facing
|
||||
`Prepare`/`Run` interface where test substitution is needed. That interface
|
||||
belongs at the consuming boundary rather than forcing adapter concepts into
|
||||
Promptkit.
|
||||
|
||||
### Artifact Reading And HTTP Containment
|
||||
|
||||
Promptkit will define the artifact-reader extension point used during
|
||||
preparation. Its ordinary file reader may read a path deliberately supplied by
|
||||
an in-process or CLI caller and does not claim to be a deployment sandbox.
|
||||
|
||||
Scriptorium will implement the HTTP-specific reader that:
|
||||
|
||||
- denies file references when no artifact root is configured;
|
||||
- applies the configured artifact byte limit;
|
||||
- enforces Scriptorium's documented lexical root-containment rule; and
|
||||
- maps reader failures to Scriptorium HTTP error responses.
|
||||
|
||||
Scriptorium will inject that reader through Promptkit's public construction
|
||||
boundary. Promptkit will not know about HTTP roots, status codes, request DTOs,
|
||||
or deployment policy.
|
||||
|
||||
### Configuration And Default Ownership
|
||||
|
||||
Configuration ownership follows the behavior configured, not the current file
|
||||
location:
|
||||
|
||||
| Configuration category | Owner |
|
||||
| --- | --- |
|
||||
| Application configuration discovery, configuration-file precedence, `prompt_dir`, `profile_dir`, and `schema_dir` | Scriptorium |
|
||||
| CLI flags and their mapping to application settings or request overrides | Scriptorium |
|
||||
| `server.*`, render-output settings, HTTP byte limits, and server defaults | Scriptorium |
|
||||
| Prompt-definition, profile, and output-contract file formats | Promptkit |
|
||||
| Prompt/profile source selection, overlays, schema behavior, and built-in profiles | Promptkit |
|
||||
| Execution settings, presence-aware request overrides, and execution defaults | Promptkit |
|
||||
| Built-in OpenAI-compatible client settings, timeout behavior, and provider wire mapping | Promptkit |
|
||||
| HTTP request and response fields, including their mapping to framework values | Scriptorium |
|
||||
|
||||
Scriptorium will translate its application settings and external request
|
||||
values into Promptkit construction options and requests. When an omitted
|
||||
Scriptorium setting means “use the framework default,” Scriptorium will omit
|
||||
the override rather than copy Promptkit's numeric default.
|
||||
|
||||
### Compatibility And Versioning
|
||||
|
||||
This migration is intentionally breaking:
|
||||
|
||||
- new Go consumers will import `gitea.maximumdirect.net/eric/promptkit`;
|
||||
- Scriptorium will not provide aliases, forwarding wrappers, or deprecated
|
||||
compatibility packages for its former Go facade;
|
||||
- existing consumers may remain pinned to the final framework-bearing
|
||||
Scriptorium tag until migrated; and
|
||||
- intermediate migration phases need not preserve source compatibility, but
|
||||
each merged phase must be internally buildable and tested.
|
||||
|
||||
Promptkit's first release will be `v0.1.0`. During the migration, incompatible
|
||||
Promptkit changes may advance its minor version until a stable `v1` contract is
|
||||
declared. The first slim Scriptorium release will advance the Scriptorium minor
|
||||
version beyond the final framework-bearing release. Normal semantic-versioning
|
||||
rules apply independently to both projects after the migration.
|
||||
|
||||
Promptkit must be tagged before Scriptorium or another consumer publishes a
|
||||
release that depends on it. Release branches must use tagged module
|
||||
dependencies, not local replacements or unpublished revisions.
|
||||
|
||||
### Local Development And Cross-Repository Coordination
|
||||
|
||||
For coordinated local work, place both repositories in a temporary Go
|
||||
workspace or use an uncommitted module replacement. `go.work`,
|
||||
`go.work.sum`, and local filesystem `replace` directives must not be committed
|
||||
to release branches.
|
||||
|
||||
Cross-repository changes follow this order:
|
||||
|
||||
1. land and tag the required Promptkit capability;
|
||||
2. update Scriptorium and other consumers to that tag;
|
||||
3. run each repository's own CI and smoke checks; and
|
||||
4. release consumers only after the Promptkit tag is available.
|
||||
|
||||
Migration coordination must confirm out-of-band repository creation, Promptkit
|
||||
tags, and downstream migrations before dependent work proceeds.
|
||||
Cross-repository changes are coordinated, not treated as atomic commits.
|
||||
|
||||
### Documentation And Maintained Assets
|
||||
|
||||
Each repository will maintain its own README, contributor guide, architecture,
|
||||
documentation, testing, release, and operations material appropriate to that
|
||||
project. Cross-project documents will link to the canonical owner rather than
|
||||
copy its contract.
|
||||
|
||||
Existing documentation and maintained assets have these target owners:
|
||||
|
||||
| Current material | Target owner |
|
||||
| --- | --- |
|
||||
| Current README and executable quickstart | Scriptorium; Promptkit creates its own framework orientation |
|
||||
| Public Go package and Go-consumer guidance | Promptkit |
|
||||
| Prompt, profile, schema, execution-setting, and framework credential reference | Promptkit |
|
||||
| OpenAI-compatible integration contract and framework internal documents | Promptkit |
|
||||
| CLI, HTTP API, subprocess, and Scriptorium operations contracts | Scriptorium |
|
||||
| Consumer interface overview | Scriptorium, revised to route Go consumers to Promptkit |
|
||||
| Application-configuration discovery, server settings, and adapter internals | Scriptorium |
|
||||
| Current internal overview and source documentation | Split into repository-local overviews; Promptkit owns framework sources and Scriptorium owns HTTP containment |
|
||||
| This ADR and cross-project migration records | Scriptorium |
|
||||
| `examples/go-library` | Promptkit |
|
||||
| `examples/config*.yml`, `examples/render-markdown-summary.sh`, and `examples/http-run.json` | Scriptorium |
|
||||
| Example prompts, profiles, schemas, and synthetic fixtures used by the executable examples | Scriptorium |
|
||||
| Embedded built-in profile assets | Promptkit |
|
||||
| Scriptorium release workflow and executable packaging | Scriptorium |
|
||||
| Repository-level license, ignore rules, agent guidance, and development policies | Each repository maintains its own applicable copy |
|
||||
|
||||
Promptkit will create or retain its own minimal framework examples and test
|
||||
fixtures rather than making either repository's tests depend on the other's
|
||||
working tree. Scriptorium's framework-format documentation will become a short
|
||||
version-appropriate link to Promptkit, while its maintained executable examples
|
||||
remain self-contained.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
- Keep the current combined repository and improve package naming. This avoids
|
||||
migration work but retains release and ownership coupling between the
|
||||
framework and executable.
|
||||
- Add Promptkit as a wrapper around the Scriptorium public package. This gives
|
||||
consumers a new import path but leaves framework ownership and dependency
|
||||
direction inverted.
|
||||
- Extract Promptkit while retaining a Scriptorium compatibility facade. This
|
||||
reduces immediate consumer changes but creates a second public API surface
|
||||
and prolongs duplicate maintenance.
|
||||
- Move all artifact reading into Promptkit. This would place HTTP containment,
|
||||
byte limits, and deployment policy in the application-neutral framework.
|
||||
- Keep Promptkit and Scriptorium as separate modules in one repository. This
|
||||
separates imports but not repository permissions, release workflows,
|
||||
issue ownership, or independent project evolution.
|
||||
|
||||
## Rationale
|
||||
|
||||
A separate Promptkit project makes the reusable framework the direct owner of
|
||||
the API that downstream Go projects already consume. Keeping Scriptorium as a
|
||||
public-API consumer exercises the same boundary as other consumers and prevents
|
||||
its adapters from relying on framework internals.
|
||||
|
||||
The selected split keeps transport and deployment policy close to the
|
||||
Scriptorium interfaces that expose it, while allowing Promptkit to remain
|
||||
useful to in-process consumers with different IO and security requirements.
|
||||
Explicit package, configuration, documentation, and asset ownership reduces
|
||||
ambiguity during extraction and after release.
|
||||
|
||||
## Consequences
|
||||
|
||||
- All Go consumers of the framework must change their import path.
|
||||
- Promptkit and Scriptorium gain independent issue, release, CI, policy, and
|
||||
documentation lifecycles.
|
||||
- Scriptorium becomes a real downstream integration test of Promptkit's public
|
||||
facade.
|
||||
- Framework changes that affect Scriptorium require tagged, ordered
|
||||
cross-repository coordination.
|
||||
- Some current packages, especially artifact reading and defaults, must be
|
||||
separated by responsibility rather than moved intact.
|
||||
- Scriptorium's current configuration and documentation references must be
|
||||
split between application and framework owners.
|
||||
- Maintainers must inventory and migrate downstream consumers explicitly; no
|
||||
compatibility facade will hide incomplete migration.
|
||||
- Until the split is implemented, the current repository structure and
|
||||
contracts remain authoritative.
|
||||
@@ -0,0 +1,67 @@
|
||||
# ADR 0003: Use Maintainer-Run Validation and Tag-Only Releases for Promptkit
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
## Date
|
||||
|
||||
2026-07-28
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0002](0002-split-promptkit-from-scriptorium.md) established Promptkit as
|
||||
an independent Go library with its own repository, version history, validation,
|
||||
and release coordination. It anticipated independent hosted CI for Promptkit
|
||||
alongside Scriptorium's existing executable build and CI policy.
|
||||
|
||||
Promptkit is presently a single-maintainer library. It does not produce a
|
||||
runnable command, so executable packaging and binary-release automation do not
|
||||
apply. Its validation and release model should be explicit before repository
|
||||
guidance relies on it.
|
||||
|
||||
## Decision
|
||||
|
||||
Promptkit will use maintainer-run validation rather than hosted CI at this
|
||||
stage. From a clean checkout, the maintainer will run the repository-documented
|
||||
test, vet, build, formatting, documentation-link, and repository-hygiene checks
|
||||
before changes are accepted and before a release tag is published.
|
||||
|
||||
Promptkit releases consist of source commits and semantic Go module tags. The
|
||||
project does not release runnable binaries or maintain binary-packaging
|
||||
automation.
|
||||
|
||||
Scriptorium's executable build, hosted CI, and binary-release policies are
|
||||
unaffected. The repository boundary, independent version history, release
|
||||
ordering, and other migration decisions accepted by ADR 0002 remain in force.
|
||||
Where ADR 0002 anticipated independent hosted CI for Promptkit, this later ADR
|
||||
controls Promptkit validation.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
- Add hosted Promptkit CI now. This would provide automated remote enforcement,
|
||||
but its setup and maintenance are not proportionate to the present
|
||||
single-maintainer library and do not replace the maintainer's release
|
||||
responsibility.
|
||||
- Require local Git hooks. Hooks can provide fast feedback, but they are
|
||||
machine-local, can be bypassed, and are not a durable substitute for the
|
||||
documented clean-checkout validation procedure.
|
||||
|
||||
## Rationale
|
||||
|
||||
A documented maintainer-run procedure provides a clear acceptance and release
|
||||
gate with little operational overhead for the project's current contribution
|
||||
pattern. If maintenance load or contributor patterns change, a later ADR may
|
||||
introduce hosted CI without changing Promptkit's library or tag-based release
|
||||
model.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Promptkit repository guidance must define the complete local validation
|
||||
procedure and the checks required before accepting or tagging a change.
|
||||
- Release evidence is the maintainer's successful clean-checkout validation,
|
||||
not a hosted CI result.
|
||||
- Promptkit releases contain source and semantic Go module tags only.
|
||||
- A future move to hosted CI requires a later architectural decision.
|
||||
- Scriptorium continues to validate, build, package, and release its executable
|
||||
under its own policies.
|
||||
145
docs/api.md
Normal file
145
docs/api.md
Normal file
@@ -0,0 +1,145 @@
|
||||
# HTTP API Reference
|
||||
|
||||
This is the canonical public HTTP contract for Scriptorium.
|
||||
|
||||
## Service And Route
|
||||
|
||||
`POST /v1/runs` runs one prompt request and returns generated output,
|
||||
validation, and metadata. The service has no built-in authentication or
|
||||
authorization; deploy it behind appropriate network and authentication controls.
|
||||
|
||||
The service address and HTTP limits are configured as described in the
|
||||
[configuration reference](config.md). `serve` invocation is defined in the
|
||||
[CLI reference](cli.md).
|
||||
|
||||
Requests and responses are JSON objects. Requests are decoded as JSON regardless
|
||||
of their `Content-Type`; successful JSON responses use
|
||||
`Content-Type: application/json`. There are no query parameters.
|
||||
|
||||
## Request Limits
|
||||
|
||||
The configured request-body limit includes inline artifact bodies. The artifact
|
||||
limit applies to HTTP `file` inputs. The response limit applies to the encoded
|
||||
response, including the artifact body and optional raw output. A limit of zero
|
||||
disables that limit.
|
||||
|
||||
A request body over its limit returns `413 request_too_large`; an oversized
|
||||
file input returns `413 artifact_too_large`; an oversized encoded response
|
||||
returns `413 response_too_large`.
|
||||
|
||||
## `POST /v1/runs`
|
||||
|
||||
### Request Body
|
||||
|
||||
The maintained [request example](../examples/http-run.json) is a complete
|
||||
copyable shape. The smallest valid shape is:
|
||||
|
||||
```json
|
||||
{
|
||||
"prompt_id": "generic.markdown_summary",
|
||||
"inputs": {
|
||||
"transcript": {"type": "inline", "body": "Source text"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Required | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `prompt_id` | yes | Non-blank prompt ID. |
|
||||
| `prompt_version` | no | Prompt version filter. |
|
||||
| `profile_id` | no | Execution-profile ID; otherwise the prompt must set `default_profile`. |
|
||||
| `inputs` | yes | Non-empty object mapping input names to references. |
|
||||
| `vars` | no | Object mapping template-variable names to strings. |
|
||||
| `model` | no | Runtime model-override object. |
|
||||
| `include_raw_output` | no | Include `raw_model_output` when true. |
|
||||
|
||||
An input reference has a required `type` of `file` or `inline`. A `file`
|
||||
reference requires `uri`; an `inline` reference requires `body`.
|
||||
|
||||
HTTP file references require a configured artifact root. Relative paths resolve
|
||||
within that root. Absolute paths must be lexically within it; traversal outside
|
||||
it is rejected with `400 artifact_not_allowed`. This lexical check does not
|
||||
resolve symlinks: the operating system follows symlinks inside the root,
|
||||
including ones that target outside it. Keep the root narrow and inaccessible to
|
||||
untrusted writers.
|
||||
|
||||
The optional `model` object accepts `endpoint`, `model`, `temperature`,
|
||||
`max_tokens`, `top_p`, `timeout_seconds`, `service_tier`,
|
||||
`reasoning_effort`, `api_key_env`, and `extra_params`. Numeric ranges and
|
||||
credential supply are defined by the [configuration reference](config.md).
|
||||
Explicit zero values for the numeric fields are overrides; zero
|
||||
`timeout_seconds` disables the per-generation deadline only, retaining the
|
||||
request context and configured transport cap. The timeout layers are defined in
|
||||
the [outbound integration contract](integrations/openai-compatible-chat.md#authentication-and-timeout).
|
||||
|
||||
Raw API-key values are not accepted. `api_key` and any other unknown model
|
||||
field cause `400 invalid_json`.
|
||||
|
||||
### Strict JSON
|
||||
|
||||
Request decoding rejects malformed JSON, unknown fields at every request level,
|
||||
and trailing JSON tokens with `400 invalid_json`. A blank `prompt_id` or
|
||||
empty `inputs` object returns `400 invalid_request`.
|
||||
|
||||
### Success Response
|
||||
|
||||
A completed run returns `200 OK`, including when generated content fails its
|
||||
validation contract. The response contains:
|
||||
|
||||
- `artifact`: `name`, `content_type`, `body`, `size`, `hash`, and
|
||||
optional `uri`;
|
||||
- `validation`: `status`, `mode`, `repair_attempts`, `is_valid`, plus
|
||||
optional `errors` and `schema_path`;
|
||||
- `metadata`: run, prompt, rendered-prompt, profile, model, input-hash, usage,
|
||||
timing, validation, and repair-attempt metadata; and
|
||||
- optional `raw_model_output` when requested.
|
||||
|
||||
`metadata.model_params` has `endpoint`, `model`, `temperature`,
|
||||
`max_tokens`, `top_p`, and `timeout_seconds`, plus optional
|
||||
`service_tier`, `reasoning_effort`, `api_key_env`, and `extra_params`.
|
||||
`metadata.usage` always includes `prompt_tokens`, `completion_tokens`,
|
||||
`total_tokens`, `cached_tokens`, and `cache_write_tokens`; unavailable
|
||||
cache usage is reported as zero.
|
||||
|
||||
A validation failure has `validation.status: "failed"`, `is_valid: false`,
|
||||
and any available diagnostic errors, while still returning the artifact and
|
||||
metadata.
|
||||
|
||||
## Error Responses
|
||||
|
||||
Errors have this shape:
|
||||
|
||||
```json
|
||||
{"error":{"code":"invalid_request","message":"prompt_id is required"}}
|
||||
```
|
||||
|
||||
Messages are concise and do not expose wrapped internal causes.
|
||||
|
||||
| Status | Code | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `400` | `invalid_json` | Malformed JSON, unknown field, or trailing JSON. |
|
||||
| `400` | `invalid_request` | Missing or invalid request data or runtime override. |
|
||||
| `400` | `profile_required` | No profile ID and no prompt default profile. |
|
||||
| `400` | `prompt_load_failed` | Prompt definition failed to load. |
|
||||
| `400` | `profile_load_failed` | Profile failed to load. |
|
||||
| `400` | `artifact_not_allowed` | HTTP file input is disabled or outside the artifact root. |
|
||||
| `400` | `artifact_read_failed` | Input artifact is invalid or cannot be read. |
|
||||
| `400` | `prompt_render_failed` | Prompt template rendering failed. |
|
||||
| `400` | `api_key_env_missing` | The selected credential environment variable is unset or empty. |
|
||||
| `404` | `not_found` | Route does not exist. |
|
||||
| `404` | `prompt_not_found` | Prompt ID or version does not exist. |
|
||||
| `404` | `profile_not_found` | Profile ID does not exist. |
|
||||
| `405` | `method_not_allowed` | The route does not accept the method. |
|
||||
| `413` | `request_too_large` | Encoded request exceeds its limit. |
|
||||
| `413` | `artifact_too_large` | File input exceeds its limit. |
|
||||
| `413` | `response_too_large` | Encoded response exceeds its limit. |
|
||||
| `500` | `validation_runtime_failed` | Schema or validator runtime failure. |
|
||||
| `500` | `internal_error` | Unclassified server failure. |
|
||||
| `502` | `llm_failed` | Outbound model request failed. |
|
||||
|
||||
## Retry And Idempotency
|
||||
|
||||
Scriptorium provides no idempotency keys, pagination, caching headers, or rate
|
||||
limits. Clients may retry transport failures or `5xx` responses only when
|
||||
their workflow tolerates another model call: a retry can produce different
|
||||
output and incur another provider request.
|
||||
148
docs/cli.md
Normal file
148
docs/cli.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# CLI Reference
|
||||
|
||||
This is the canonical contract for invoking Scriptorium. Configuration discovery,
|
||||
precedence, directories, profiles, and schemas are defined in the
|
||||
[configuration reference](config.md). The [HTTP API reference](api.md) owns
|
||||
service request and response behavior.
|
||||
|
||||
## Shortest Useful Command
|
||||
|
||||
```bash
|
||||
go run ./cmd/scriptorium render \
|
||||
--config ./examples/config.yml \
|
||||
--prompt generic.markdown_summary \
|
||||
--input transcript=./examples/fixtures/transcript.md \
|
||||
--input glossary=./examples/fixtures/glossary.yml
|
||||
```
|
||||
|
||||
`render` prepares a request without calling an LLM.
|
||||
|
||||
## Commands
|
||||
|
||||
- `scriptorium run`: prepare a prompt, call the configured LLM, and write the
|
||||
generated artifact.
|
||||
- `scriptorium render`: prepare a prompt and write prepared-run output.
|
||||
- `scriptorium serve`: start the HTTP server.
|
||||
|
||||
All commands accept `--config <path>` and reject positional arguments. An
|
||||
effective `prompt_dir` is required for every command. Supply it through the
|
||||
configuration contract or the command's `--prompt-dir` flag.
|
||||
|
||||
## `scriptorium run`
|
||||
|
||||
```text
|
||||
scriptorium run [flags]
|
||||
```
|
||||
|
||||
Required flags:
|
||||
|
||||
| Flag | Meaning |
|
||||
| --- | --- |
|
||||
| `--prompt <id>` | Prompt ID to execute. |
|
||||
| `--input name=path` | Input file mapping; repeat or use comma-separated mappings. |
|
||||
|
||||
Optional flags:
|
||||
|
||||
| Flag | Meaning |
|
||||
| --- | --- |
|
||||
| `--config <path>` | Application configuration file. |
|
||||
| `--prompt-dir <dir>` | Prompt-definition directory override. |
|
||||
| `--profile-dir <dir>` | Custom profile-directory override. |
|
||||
| `--schema-dir <dir>` | Schema base-directory override. |
|
||||
| `--profile <id>` | Execution-profile override. |
|
||||
| `--var name=value` | Template-variable mapping; repeat or use comma-separated mappings. |
|
||||
| `--out <path>` | Write generated content to this file instead of stdout. |
|
||||
| `--llm-base-url <url>` | Runtime endpoint override. |
|
||||
| `--model <name>` | Runtime model override. |
|
||||
| `--api-key-env <name>` | Runtime API-key environment-variable name override. |
|
||||
| `--temperature <float>` | Runtime temperature override. |
|
||||
| `--max-tokens <int>` | Runtime maximum-token override. |
|
||||
| `--top-p <float>` | Runtime top-p override. |
|
||||
| `--timeout <duration>` | Runtime timeout override using Go duration syntax. |
|
||||
|
||||
Deprecated aliases: `--prompt-id` for `--prompt`, and `--profile-id` for
|
||||
`--profile`.
|
||||
|
||||
Omitted numeric runtime flags preserve the selected effective value; explicit
|
||||
zero values override it. `--timeout 0s` disables the per-generation deadline
|
||||
only; the caller context and configured transport cap remain active. CLI
|
||||
durations are converted to whole seconds by truncation toward zero, so any
|
||||
duration whose absolute value is below one second becomes an explicit
|
||||
zero-second override. The timeout layers are defined in the
|
||||
[outbound integration contract](integrations/openai-compatible-chat.md#authentication-and-timeout).
|
||||
|
||||
There is no raw API-key flag. Use `--api-key-env`.
|
||||
|
||||
## `scriptorium render`
|
||||
|
||||
```text
|
||||
scriptorium render [flags]
|
||||
```
|
||||
|
||||
`--prompt <id>` and at least one `--input name=path` are required. The
|
||||
following optional flags are supported: `--config`, `--prompt-dir`,
|
||||
`--profile-dir`, `--profile`, `--var`, `--out`, `--llm-base-url`,
|
||||
`--model`, `--api-key-env`, `--temperature`, `--max-tokens`, `--top-p`,
|
||||
`--timeout`, and `--format text|json`. Their meanings match the corresponding
|
||||
`run` flags; `--format` selects prepared-run output and otherwise uses
|
||||
`defaults.render_format`.
|
||||
|
||||
The same deprecated aliases and numeric/timeout behavior as `run` apply.
|
||||
`render` does not accept `--schema-dir`; configure `schema_dir` through the
|
||||
configuration file. It resolves profiles and schemas as part of preparation but
|
||||
does not call an LLM.
|
||||
|
||||
## `scriptorium serve`
|
||||
|
||||
```text
|
||||
scriptorium serve [flags]
|
||||
```
|
||||
|
||||
Optional flags:
|
||||
|
||||
| Flag | Meaning |
|
||||
| --- | --- |
|
||||
| `--config <path>` | Application configuration file. |
|
||||
| `--addr <listen-address>` | HTTP listen-address override. |
|
||||
| `--prompt-dir <dir>` | Prompt-definition directory override. |
|
||||
| `--profile-dir <dir>` | Custom profile-directory override. |
|
||||
| `--schema-dir <dir>` | Schema base-directory override. |
|
||||
| `--artifact-root <dir>` | Root for HTTP `file` input references. |
|
||||
| `--max-request-bytes <n>` | Maximum encoded HTTP request-body bytes; `0` disables the limit. |
|
||||
| `--max-artifact-bytes <n>` | Maximum HTTP file-input artifact bytes; `0` disables the limit. |
|
||||
| `--max-response-bytes <n>` | Maximum encoded HTTP response bytes; `0` disables the limit. |
|
||||
|
||||
`serve` accepts no runtime model override flags. HTTP request fields, response
|
||||
schemas, and error codes are defined in the [HTTP API reference](api.md).
|
||||
|
||||
## Input And Variable Syntax
|
||||
|
||||
`--input name=path` maps an input name to a local file; `--var name=value`
|
||||
maps a template variable to a string. Both flags can be repeated or contain
|
||||
comma-separated mappings. Values may contain `=` after the first separator.
|
||||
Empty names and values are rejected.
|
||||
|
||||
CLI inputs are file references. HTTP inline inputs are defined by the
|
||||
[HTTP API reference](api.md).
|
||||
|
||||
## Output And Exit Behavior
|
||||
|
||||
- `run` writes generated content to stdout, or to `--out` when supplied, and
|
||||
writes a concise summary to stderr.
|
||||
- `render` writes prepared-run output to stdout, or to `--out` when supplied,
|
||||
without a success summary.
|
||||
- `serve` writes startup and server errors to stderr.
|
||||
|
||||
Exit statuses:
|
||||
|
||||
| Status | Meaning |
|
||||
| --- | --- |
|
||||
| `0` | Success. |
|
||||
| `1` | Parse, configuration, loading, rendering, generation, output-write, or other runtime error. |
|
||||
| `2` | `run` generated and wrote output, but validation failed. |
|
||||
|
||||
## Workflows And Examples
|
||||
|
||||
The [maintained render script](../examples/render-markdown-summary.sh) is a
|
||||
copyable render workflow. The [HTTP request example](../examples/http-run.json)
|
||||
is for a running `serve` process.
|
||||
173
docs/config.md
Normal file
173
docs/config.md
Normal file
@@ -0,0 +1,173 @@
|
||||
# Configuration Reference
|
||||
|
||||
This is the canonical reference for Scriptorium application settings and the
|
||||
prompt, profile, and schema files those settings select. For command syntax,
|
||||
see the [CLI reference](cli.md); for HTTP request shapes, limits, and outcomes,
|
||||
see the [HTTP API reference](api.md).
|
||||
|
||||
## Discovery And Precedence
|
||||
|
||||
Application settings are resolved in this order:
|
||||
|
||||
1. built-in defaults;
|
||||
2. a configuration file; then
|
||||
3. CLI overrides.
|
||||
|
||||
When `--config` is omitted, Scriptorium searches
|
||||
`/usr/local/etc/scriptorium/config.yml` and then `/etc/scriptorium/config.yml`.
|
||||
If neither exists, it uses built-in defaults. An explicit `--config` path must
|
||||
exist and decode successfully.
|
||||
|
||||
The maintained [minimal configuration](../examples/config.yml) and
|
||||
[full configuration](../examples/config.full.yml) are copyable examples.
|
||||
|
||||
## Application Configuration File
|
||||
|
||||
Configuration is strict YAML: unknown fields are rejected. Empty string values
|
||||
do not override a prior value. Raw API-key fields are not accepted.
|
||||
|
||||
| Field | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `prompt_dir` | unset | Directory containing prompt-definition YAML. `run`, `render`, and `serve` require an effective value. |
|
||||
| `profile_dir` | unset | Directory containing custom profile YAML. Built-in profiles remain available. |
|
||||
| `schema_dir` | `.` | Base directory for relative JSON Schema paths. |
|
||||
| `server.addr` | `:8080` | Address used by `serve`. |
|
||||
| `server.artifact_root` | unset | Root that enables HTTP `file` input references. |
|
||||
| `server.max_request_bytes` | `16777216` | Maximum encoded HTTP request body bytes; `0` disables the limit. |
|
||||
| `server.max_artifact_bytes` | `16777216` | Maximum HTTP file-input artifact bytes; `0` disables the limit. |
|
||||
| `server.max_response_bytes` | `16777216` | Maximum encoded HTTP response bytes; `0` disables the limit. |
|
||||
| `defaults.render_format` | `text` | Default `render` output format: `text` or `json`. |
|
||||
|
||||
The three size fields must be zero or greater. The HTTP contract defines how
|
||||
each limit is enforced and reported. `server.artifact_root` configures the
|
||||
deployment boundary; see the [HTTP API reference](api.md) for request-path and
|
||||
containment behavior, and [operations](operations.md) for deployment handling.
|
||||
|
||||
## Prompt Definition Files
|
||||
|
||||
Prompt definitions are strict YAML files anywhere below `prompt_dir`. A prompt
|
||||
is selected by its YAML `id`, not by file path; nested directories are only for
|
||||
organization. See [maintained prompt examples](../examples/prompts/).
|
||||
|
||||
| Field | Required | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `id` | yes | Prompt identifier. |
|
||||
| `version` | yes | Prompt version. |
|
||||
| `default_profile` | no | Profile used when a request omits a profile ID. |
|
||||
| `description` | no | Human-readable description. |
|
||||
| `session_id` | no | Go-template string rendered from request variables and sent to a compatible provider when non-empty. |
|
||||
| `inputs` | no | Declared input metadata. |
|
||||
| `messages` | yes | Chat-message templates. |
|
||||
| `output` | yes | Output format and validation contract. |
|
||||
|
||||
### Inputs And Messages
|
||||
|
||||
Each `inputs` item has a required `name` and optional `required`,
|
||||
`content_type`, and `description` fields. Input names must be unique.
|
||||
|
||||
Each message has a required `role`, exactly one of `content` or `content_file`,
|
||||
and optional `cache_control`. A `content_file` path is relative to the prompt
|
||||
file. `cache_control.type` must be `ephemeral`; its optional `ttl` is `1h`.
|
||||
|
||||
`session_id` uses the same template variables as messages. Empty rendered
|
||||
values are omitted. A rendered value may contain at most 256 Unicode code
|
||||
points.
|
||||
|
||||
### Output Contract
|
||||
|
||||
| Field | Required | Values or behavior |
|
||||
| --- | --- | --- |
|
||||
| `format` | yes | `text`, `markdown`, or `json`. |
|
||||
| `validation_mode` | yes | `none`, `basic`, `json`, or `json_schema`. |
|
||||
| `schema_path` | for `json_schema` | Schema path, relative to `schema_dir` unless absolute. |
|
||||
| `repair_attempts` | no | Integer greater than or equal to `0`; omitted means `0`. |
|
||||
|
||||
## Profile Definition Files
|
||||
|
||||
Profiles are strict YAML files anywhere below `profile_dir`. A profile is
|
||||
selected by YAML `id`; nested directories are organizational. See the
|
||||
[maintained profile examples](../examples/profiles/).
|
||||
|
||||
| Field | Required | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `id` | yes | Profile identifier. |
|
||||
| `endpoint` | yes | OpenAI-compatible base URL, including its API version path when needed. |
|
||||
| `model` | yes | Provider model name. |
|
||||
| `temperature` | no | Number from `0` through `2`. |
|
||||
| `max_tokens` | no | Integer zero or greater. |
|
||||
| `top_p` | no | Number from `0` through `1`. |
|
||||
| `timeout_seconds` | no | Per-generation-call deadline in whole seconds; integer zero or greater. |
|
||||
| `service_tier` | no | Non-empty provider-specific request tier. |
|
||||
| `reasoning_effort` | no | Non-empty provider-specific reasoning setting. |
|
||||
| `api_key_env` | no | Environment-variable name containing the API key. |
|
||||
| `extra_params` | no | JSON-compatible provider-specific outbound request fields. |
|
||||
|
||||
Execution defaults before profile and request overrides are `temperature: 0`,
|
||||
`max_tokens: 0`, `top_p: 1`, and `timeout_seconds: 600`. Profile numeric values
|
||||
merge by non-zero value. Request overrides preserve presence, so an explicit
|
||||
zero can override a profile value. For `timeout_seconds`, explicit request zero
|
||||
disables the generation deadline while retaining the caller context and the
|
||||
built-in client's transport cap. See the
|
||||
[OpenAI-compatible integration contract](integrations/openai-compatible-chat.md#authentication-and-timeout)
|
||||
for the complete timeout interaction.
|
||||
|
||||
Custom profiles take precedence over built-ins with the same ID. Invalid custom
|
||||
profiles are errors; they do not fall back to a built-in profile. Raw `api_key`
|
||||
is rejected. Use `api_key_env`, or the public Go package's request-scoped key
|
||||
mechanism described in the [package contract](consumers/pkg-scriptorium.md).
|
||||
|
||||
`extra_params` keys must be non-empty and cannot be `model`, `session_id`,
|
||||
`messages`, `temperature`, `max_tokens`, `top_p`, `service_tier`,
|
||||
`reasoning_effort`, or `response_format`.
|
||||
|
||||
### Built-In Profile Catalog
|
||||
|
||||
Each embedded profile uses `OPENROUTER_API_KEY`.
|
||||
|
||||
| Provider | ID | Model |
|
||||
| --- | --- | --- |
|
||||
| aion-labs | `aion-2` | `aion-labs/aion-2.0` |
|
||||
| anthropic | `claude-fable-latest` | `~anthropic/claude-fable-latest` |
|
||||
| anthropic | `claude-haiku-latest` | `~anthropic/claude-haiku-latest` |
|
||||
| anthropic | `claude-opus-latest` | `~anthropic/claude-opus-latest` |
|
||||
| anthropic | `claude-sonnet-latest` | `~anthropic/claude-sonnet-latest` |
|
||||
| deepseek | `deepseek-3-2` | `deepseek/deepseek-v3.2` |
|
||||
| deepseek | `deepseek-4-flash` | `deepseek/deepseek-v4-flash` |
|
||||
| deepseek | `deepseek-4-pro` | `deepseek/deepseek-v4-pro` |
|
||||
| google | `gemini-2-flash` | `google/gemini-2.5-flash` |
|
||||
| google | `gemini-2-flash-lite` | `google/gemini-2.5-flash-lite` |
|
||||
| google | `gemini-2-pro` | `google/gemini-2.5-pro` |
|
||||
| google | `gemini-3-flash-lite` | `google/gemini-3.1-flash-lite` |
|
||||
| google | `gemini-flash-latest` | `~google/gemini-flash-latest` |
|
||||
| google | `gemini-pro-latest` | `~google/gemini-pro-latest` |
|
||||
| google | `gemma-4-31b` | `google/gemma-4-31b-it:exacto` |
|
||||
| minimax | `minimax-m2` | `minimax/minimax-m2.5` |
|
||||
| minimax | `minimax-m3` | `minimax/minimax-m3` |
|
||||
| mistral | `mistral-large-2512` | `mistralai/mistral-large-2512` |
|
||||
| mistral | `mistral-medium-3-5` | `mistralai/mistral-medium-3-5` |
|
||||
| mistral | `mistral-small-3` | `mistralai/mistral-small-3.2-24b-instruct` |
|
||||
| mistral | `mistral-small-4` | `mistralai/mistral-small-2603` |
|
||||
| nvidia | `nemotron-3-ultra` | `nvidia/nemotron-3-ultra-550b-a55b` |
|
||||
| openai | `gpt-5-mini` | `openai/gpt-5.4-mini` |
|
||||
| openai | `gpt-5-nano` | `openai/gpt-5.4-nano` |
|
||||
|
||||
## Schemas
|
||||
|
||||
Schemas are JSON files, normally below `schema_dir`. `json_schema` output
|
||||
requires a `schema_path`. Relative paths resolve from `schema_dir`; absolute
|
||||
paths are used directly. Referenced nested schemas use relative paths and are
|
||||
not discovered by basename. An unreadable or invalid schema is a runtime
|
||||
validation error; generated content that fails JSON or schema validation is a
|
||||
validation result.
|
||||
|
||||
## Credentials
|
||||
|
||||
Keep secrets in environment variables. Store only an environment-variable name
|
||||
in `api_key_env`; do not place raw keys in configuration, prompt or profile
|
||||
files, CLI arguments, examples, or HTTP payloads.
|
||||
|
||||
## Related References
|
||||
|
||||
- [CLI reference](cli.md)
|
||||
- [HTTP API reference](api.md)
|
||||
- [OpenAI-compatible outbound contract](integrations/openai-compatible-chat.md)
|
||||
@@ -1,48 +0,0 @@
|
||||
# Main `config.yml`
|
||||
|
||||
`config.yml` defines application-level defaults used by CLI commands.
|
||||
|
||||
By default, Scriptorium looks for `/usr/local/etc/scriptorium/config.yml` and, if not present, then for `/etc/scriptorium/config.yml`. You can also pass `--config PATH`.
|
||||
|
||||
## Complete Example
|
||||
|
||||
```yaml
|
||||
prompt_dir: ./prompts
|
||||
profile_dir: ./profiles
|
||||
schema_dir: ./schemas
|
||||
|
||||
server:
|
||||
addr: :8080
|
||||
|
||||
defaults:
|
||||
render_format: text
|
||||
```
|
||||
|
||||
## Available Options
|
||||
|
||||
- `prompt_dir` (optional): Default directory for prompt definition YAML files.
|
||||
- `profile_dir` (optional): Default directory for execution profile YAML files.
|
||||
- `schema_dir` (optional): Base directory for JSON schema files used by `json_schema` validation.
|
||||
|
||||
### `server`
|
||||
|
||||
- `addr` (optional): HTTP server listen address for `scriptorium serve`.
|
||||
|
||||
### `defaults`
|
||||
|
||||
- `render_format` (optional): Default output format for `scriptorium render`.
|
||||
- Allowed values: `text`, `json`.
|
||||
|
||||
## Precedence
|
||||
|
||||
For run/render/serve settings, precedence is:
|
||||
|
||||
1. Explicit CLI flags
|
||||
2. `config.yml`
|
||||
3. Built-in defaults
|
||||
|
||||
## Notes and Rules
|
||||
|
||||
- Unknown YAML fields fail to load (strict decoding).
|
||||
- This file does not accept API keys.
|
||||
- `config.yml` sets directory/server defaults only; prompt/profile content remains in their own files.
|
||||
@@ -1,41 +0,0 @@
|
||||
# Execution Profile Definitions
|
||||
|
||||
Execution Profiles define **how** Scriptorium calls an LLM endpoint.
|
||||
|
||||
A profile file is YAML, typically stored under `profiles/`, for example `profiles/local-quality.yaml`.
|
||||
|
||||
## Complete Example
|
||||
|
||||
```yaml
|
||||
id: local-quality
|
||||
endpoint: http://localhost:8000/v1
|
||||
model: gpt-4.1
|
||||
temperature: 0.0
|
||||
max_tokens: 1200
|
||||
top_p: 1.0
|
||||
timeout_seconds: 180
|
||||
reasoning_effort: medium
|
||||
api_key_env: SCRIPTORIUM_API_KEY
|
||||
extra_params:
|
||||
provider: openrouter
|
||||
route: fallback
|
||||
```
|
||||
|
||||
## Available Options
|
||||
|
||||
- `id` (required): Unique profile identifier used by `--profile` or prompt `default_profile`.
|
||||
- `endpoint` (required): OpenAI-compatible base URL, usually ending in `/v1`.
|
||||
- `model` (required): Model name to request at that endpoint.
|
||||
- `temperature` (optional): Sampling temperature. Valid range is `0` to `2`.
|
||||
- `max_tokens` (optional): Max completion tokens. Must be `>= 0`.
|
||||
- `top_p` (optional): Nucleus sampling parameter. Valid range is `0` to `1`.
|
||||
- `timeout_seconds` (optional): Request timeout in seconds. Must be `>= 0`.
|
||||
- `reasoning_effort` (optional): Provider/model-specific reasoning level string.
|
||||
- `api_key_env` (optional): Environment variable name that holds the API key.
|
||||
- `extra_params` (optional): String key/value map for provider-specific parameters.
|
||||
|
||||
## Notes and Rules
|
||||
|
||||
- Raw API keys are not supported. Do **not** add `api_key` fields.
|
||||
- Unknown YAML fields fail to load (strict decoding).
|
||||
- If `api_key_env` is set, the environment variable must be present when the run executes.
|
||||
@@ -1,73 +0,0 @@
|
||||
# Prompt Definition Files
|
||||
|
||||
Prompt Definitions define **what** Scriptorium should do.
|
||||
|
||||
A prompt file is YAML, typically stored under `prompts/`, for example `prompts/generic.structured_events.yaml`.
|
||||
|
||||
## Complete Example
|
||||
|
||||
```yaml
|
||||
id: generic.structured_events
|
||||
version: "1.0.0"
|
||||
default_profile: local-quality
|
||||
description: Extract events from a transcript into structured JSON.
|
||||
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: text/markdown
|
||||
description: Source transcript
|
||||
- name: glossary
|
||||
required: false
|
||||
content_type: text/yaml
|
||||
description: Optional glossary context
|
||||
|
||||
messages:
|
||||
- role: system
|
||||
content: |
|
||||
You are a structured extraction assistant.
|
||||
Return only JSON.
|
||||
- role: user
|
||||
content_file: ./generic.structured_events.user.md
|
||||
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: structured_events.schema.json
|
||||
repair_attempts: 1
|
||||
```
|
||||
|
||||
## Available Options
|
||||
|
||||
- `id` (required): Prompt identifier used by `--prompt` / `prompt_id`.
|
||||
- `version` (required): Prompt version string.
|
||||
- `default_profile` (optional): Execution profile ID used when no explicit profile is provided.
|
||||
- `description` (optional): Human-readable description.
|
||||
|
||||
### `inputs[]`
|
||||
|
||||
- `name` (required): Logical input name referenced in templates via `{{input "name"}}`.
|
||||
- `required` (optional): If `true`, run fails when input is missing.
|
||||
- `content_type` (optional): Metadata only (not enforced yet).
|
||||
- `description` (optional): Human-readable input description.
|
||||
|
||||
### `messages[]`
|
||||
|
||||
- `role` (required): Message role such as `system` or `user`.
|
||||
- `content` (optional): Inline Go-template message body.
|
||||
- `content_file` (optional): Path to a template file.
|
||||
|
||||
Each message must set **exactly one** of `content` or `content_file`.
|
||||
|
||||
### `output`
|
||||
|
||||
- `format` (required): One of `text`, `markdown`, `json`.
|
||||
- `validation_mode` (required): One of `none`, `basic`, `json`, `json_schema`.
|
||||
- `schema_path` (required when `validation_mode: json_schema`): Path to JSON Schema file.
|
||||
- `repair_attempts` (required): Number of bounded repair retries (`>= 0`).
|
||||
|
||||
## Notes and Rules
|
||||
|
||||
- Unknown YAML fields fail to load (strict decoding).
|
||||
- `content_file` paths are resolved relative to the prompt YAML file.
|
||||
- For `json_schema` validation mode, Scriptorium also sends provider-level structured output requests automatically.
|
||||
@@ -1,82 +0,0 @@
|
||||
# JSON Schema Definition Files
|
||||
|
||||
Schema definition files describe the expected JSON output contract for prompts that use:
|
||||
|
||||
- `output.format: json`
|
||||
- `output.validation_mode: json_schema`
|
||||
|
||||
Schema files are JSON, typically stored under `schemas/`, for example `schemas/structured_events.schema.json`.
|
||||
|
||||
## Complete Example
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://example.com/schemas/structured-events.schema.json",
|
||||
"title": "Structured Events",
|
||||
"description": "Expected shape for extracted event output",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"summary": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "High-level session summary"
|
||||
},
|
||||
"events": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"title": { "type": "string" },
|
||||
"type": {
|
||||
"type": "string",
|
||||
"enum": ["discovery", "combat", "social", "travel", "downtime", "other"]
|
||||
},
|
||||
"notes": { "type": "string" }
|
||||
},
|
||||
"required": ["title", "type"],
|
||||
"additionalProperties": false
|
||||
},
|
||||
"minItems": 0
|
||||
}
|
||||
},
|
||||
"required": ["summary", "events"],
|
||||
"additionalProperties": false,
|
||||
"$defs": {
|
||||
"nonEmptyString": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Available Options
|
||||
|
||||
Scriptorium does not define custom schema keywords. It expects a valid JSON Schema document and passes it to the validator/provider.
|
||||
|
||||
Commonly used JSON Schema options include:
|
||||
|
||||
- `$schema`: Draft identifier URI.
|
||||
- `$id`: Schema identifier URI.
|
||||
- `title`: Human-readable schema title.
|
||||
- `description`: Human-readable schema description.
|
||||
- `type`: Expected JSON type (`object`, `array`, `string`, etc.).
|
||||
- `properties`: Object field definitions.
|
||||
- `required`: Required object fields.
|
||||
- `additionalProperties`: Whether undeclared fields are allowed.
|
||||
- `items`: Array item schema.
|
||||
- `enum`: Allowed literal values.
|
||||
- `const`: Single allowed literal value.
|
||||
- `oneOf`, `anyOf`, `allOf`: Composition rules.
|
||||
- `minimum`, `maximum`: Numeric bounds.
|
||||
- `minLength`, `maxLength`, `pattern`: String constraints.
|
||||
- `minItems`, `maxItems`: Array constraints.
|
||||
- `$defs`: Reusable local definitions.
|
||||
- `$ref`: Reference to another schema/definition.
|
||||
|
||||
## Notes and Rules
|
||||
|
||||
- Schema path comes from prompt `output.schema_path` and is resolved relative to `schema_dir`.
|
||||
- If schema loading fails for `json_schema` mode, the run fails before the LLM request.
|
||||
- Keep schemas strict (`additionalProperties: false`) when you want predictable output shape.
|
||||
59
docs/consumers/api.md
Normal file
59
docs/consumers/api.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# Consumer Integration Overview
|
||||
|
||||
This guide helps applications choose a Scriptorium interface and understand
|
||||
their responsibilities. The linked contracts own interface syntax and wire
|
||||
semantics.
|
||||
|
||||
| Interface | Use when |
|
||||
| --- | --- |
|
||||
| Go package | The consumer is Go and needs typed requests, results, or an injected LLM client. |
|
||||
| CLI subprocess | The consumer needs process isolation or is not written in Go. |
|
||||
| HTTP API | The consumer needs a service boundary or remote access. |
|
||||
|
||||
- Go package: [package contract](pkg-scriptorium.md)
|
||||
- CLI subprocess: [subprocess integration](../integrations/subprocess.md)
|
||||
- HTTP service: [HTTP API reference](../api.md)
|
||||
- Prompt, profile, schema, and credential configuration: [configuration reference](../config.md)
|
||||
|
||||
## Minimal Go Use
|
||||
|
||||
```go
|
||||
engine, err := scriptorium.NewEngine(scriptorium.Config{
|
||||
PromptDir: "./examples/prompts",
|
||||
ProfileDir: "./examples/profiles",
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
prepared, err := engine.Prepare(ctx, scriptorium.RunRequest{
|
||||
PromptID: "generic.markdown_summary",
|
||||
Inputs: map[string]scriptorium.ArtifactRef{
|
||||
"transcript": scriptorium.File("./examples/fixtures/transcript.md"),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_ = prepared
|
||||
```
|
||||
|
||||
For a maintained program, see
|
||||
[`examples/go-library/prepare`](../../examples/go-library/prepare).
|
||||
|
||||
## Consumer Responsibilities
|
||||
|
||||
Consumers are responsible for:
|
||||
|
||||
- selecting and deploying prompt, profile, and schema assets;
|
||||
- supplying required inputs and template variables;
|
||||
- supplying credentials through the applicable interface;
|
||||
- protecting rendered prompts and generated artifacts as potentially sensitive;
|
||||
- deciding whether validation-failed output is usable; and
|
||||
- retrying only when another model call is acceptable.
|
||||
|
||||
Scriptorium does not persist run state. A retry can produce different output and
|
||||
can incur another provider request. CLI exit behavior belongs to the
|
||||
[CLI reference](../cli.md); HTTP status behavior belongs to the
|
||||
[HTTP API reference](../api.md); package errors and results belong to the
|
||||
[package contract](pkg-scriptorium.md).
|
||||
203
docs/consumers/pkg-scriptorium.md
Normal file
203
docs/consumers/pkg-scriptorium.md
Normal file
@@ -0,0 +1,203 @@
|
||||
# Package `scriptorium`
|
||||
|
||||
Import path:
|
||||
|
||||
```go
|
||||
import "gitea.maximumdirect.net/eric/scriptorium"
|
||||
```
|
||||
|
||||
This is the canonical public Go contract for in-process prompt preparation and
|
||||
execution. Prompt, profile, and schema file formats are defined in the
|
||||
[configuration reference](../config.md).
|
||||
|
||||
## Engine Construction
|
||||
|
||||
`NewEngine(Config, ...Option)` constructs an engine. `Config` has these
|
||||
fields:
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `PromptDir` | Prompt-definition directory, required unless a prompt source option is supplied. |
|
||||
| `ProfileDir` | Optional custom profile directory over built-ins. |
|
||||
| `SchemaDir` | Schema directory; empty uses `.`. |
|
||||
| `Timeout` | Transport-wide safety cap for the built-in OpenAI-compatible client when `HTTPClient` is absent or has a non-positive timeout. A non-positive value uses the internal ten-minute default. |
|
||||
| `HTTPClient` | Optional HTTP client for that built-in client. It is cloned; a positive `Timeout` on it is the transport cap and takes precedence over `Config.Timeout`. A non-positive client timeout is treated as unset. |
|
||||
|
||||
Nil options are ignored. Invalid construction, including
|
||||
`WithLLMClient(nil)` and `WithArtifactReader(nil)`, returns an error matching
|
||||
`ErrInvalidConfig`.
|
||||
|
||||
Profile and request `timeout_seconds` values select a per-generation-call
|
||||
deadline independently of the transport cap. An explicit request override of
|
||||
zero disables that generation deadline only. The complete interaction with the
|
||||
caller context is defined in the
|
||||
[OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md#authentication-and-timeout).
|
||||
|
||||
Source options replace their matching directory source:
|
||||
|
||||
- prompts: `WithPromptFS(fsys, root)`, `WithPromptFile(path)`;
|
||||
- profiles: `WithProfileFS(fsys, root)`, `WithProfileFile(path)`, and
|
||||
`WithProfiles(profiles...)`;
|
||||
- schemas: `WithSchemaFS(fsys, root)`, `WithSchemaFile(path)`; and
|
||||
- LLM client: `WithLLMClient(client)`; and
|
||||
- artifact reader: `WithArtifactReader(reader)`.
|
||||
|
||||
`fs.FS` prompt-content and schema paths stay inside their configured roots.
|
||||
Single-file prompt and profile sources are selected by their YAML `id`, not
|
||||
their file names. `WithPromptFile` resolves relative `content_file` paths from
|
||||
the prompt file's directory. `WithSchemaFile` exposes its schema by the schema
|
||||
file's base name. In-memory profiles take precedence over an explicit or
|
||||
directory-backed profile source, which in turn takes precedence over built-ins.
|
||||
File and filesystem sources use the format and credential rules in the
|
||||
[configuration reference](../config.md).
|
||||
|
||||
## Prepare And Run
|
||||
|
||||
`Prepare(ctx, request)` resolves the prompt, profile, input artifacts,
|
||||
validation contract, and rendered messages without calling an LLM.
|
||||
`Run(ctx, request)` performs that preparation, calls the configured client,
|
||||
and validates generated content.
|
||||
|
||||
```go
|
||||
engine, err := scriptorium.NewEngine(scriptorium.Config{
|
||||
PromptDir: "./examples/prompts",
|
||||
ProfileDir: "./examples/profiles",
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
prepared, err := engine.Prepare(ctx, scriptorium.RunRequest{
|
||||
PromptID: "generic.markdown_summary",
|
||||
Inputs: map[string]scriptorium.ArtifactRef{
|
||||
"transcript": scriptorium.File("./examples/fixtures/transcript.md"),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_ = prepared.Messages
|
||||
```
|
||||
|
||||
The maintained package example is
|
||||
[`examples/go-library/prepare`](../../examples/go-library/prepare).
|
||||
|
||||
`PreparedRun` exposes prompt, selected-profile, effective-model, output
|
||||
contract, structured-output, input-hash, rendered-message, and timing
|
||||
information. It does not include a resolved API key, model output, validation
|
||||
result, or target-presence metadata.
|
||||
|
||||
`RunResult` adds run ID, artifact, raw output, validation, model metadata,
|
||||
usage, and duration. Generated-content validation failures return a result with
|
||||
`Validation.Status == ValidationFailed`; schema or validator runtime failures
|
||||
return an error matching `ErrValidation`.
|
||||
|
||||
## Public Values
|
||||
|
||||
`ArtifactRef` has `Type`, `URI`, and `Body`; `Artifact` has `Name`,
|
||||
`ContentType`, `Body`, `URI`, `Size`, and `Hash`. `ExecutionTarget` exposes the
|
||||
effective endpoint, model, numeric settings, credential-environment name,
|
||||
service tier, reasoning effort, and extra parameters. `ValidationResult`
|
||||
contains status, mode, errors, schema path, repair attempts, and validity.
|
||||
|
||||
The exported constants define these serialized values:
|
||||
|
||||
- artifact types: `inline` and `file`;
|
||||
- output formats: `text`, `markdown`, and `json`;
|
||||
- validation modes: `none`, `basic`, `json`, and `json_schema`; and
|
||||
- validation statuses: `passed`, `failed`, and `skipped`.
|
||||
|
||||
`TokenUsage` reports prompt, completion, total, cached, and cache-write token
|
||||
counts. `RenderedPrompt`, `RenderedMessage`, `CacheControl`, and
|
||||
`StructuredOutputSpec` are the public shapes used by injected LLM clients.
|
||||
|
||||
`ArtifactReader` implements
|
||||
`Read(context.Context, ArtifactRef) (*Artifact, error)`. Supplying it through
|
||||
`WithArtifactReader` replaces, rather than extends, the engine's default inline
|
||||
and file reader for every input. Omitting the option retains that default;
|
||||
`WithArtifactReader(nil)` makes engine construction fail with
|
||||
`ErrInvalidConfig`.
|
||||
|
||||
Reader failures are surfaced as errors matching `ErrArtifactLoad` while
|
||||
preserving the reader's original error identity for `errors.Is`. A `(nil, nil)`
|
||||
reader response is also an artifact-load failure. Readers are responsible for
|
||||
artifact metadata, although the engine assigns the input-map name when the
|
||||
returned name is empty; readers should not retain or mutate caller values.
|
||||
|
||||
## Requests, Inputs, And Overrides
|
||||
|
||||
`RunRequest` fields are `PromptID`, `PromptVersion`, `ProfileID`,
|
||||
`APIKey`, `Inputs`, `Vars`, `Execution`, `Validation`, and
|
||||
`Metadata`.
|
||||
|
||||
Input helpers are:
|
||||
|
||||
- `File(path)` for a file-backed artifact;
|
||||
- `Inline(body)` for inline content; and
|
||||
- `InlineWithURI(uri, body)` for inline content with URI metadata.
|
||||
|
||||
Required declared inputs must be supplied. Template rendering must also resolve
|
||||
every input name the prompt actually references. Extra entries in `Inputs`
|
||||
are not rejected solely because they are undeclared.
|
||||
|
||||
`ExecutionTargetOverride` supplies endpoint, model, credential-environment,
|
||||
service-tier, reasoning-effort, and extra-parameter overrides. Its numeric
|
||||
fields (`Temperature`, `MaxTokens`, `TopP`, and `TimeoutSeconds`) are
|
||||
pointers so explicit zero values are preserved. `OutputContract` supplies
|
||||
`Format`, `ValidationMode`, `SchemaPath`, and `RepairAttempts`.
|
||||
|
||||
`ExtraParams` accepts JSON-compatible values: strings, booleans, finite
|
||||
numbers, objects with string keys, arrays or slices, and nil. Unsupported
|
||||
values, non-string map keys, non-finite floats, and cycles return
|
||||
`ErrInvalidConfig` for profiles or `ErrInvalidRequest` for request
|
||||
overrides.
|
||||
|
||||
## Profiles And Credentials
|
||||
|
||||
`OpenAICompatibleProfile(OpenAICompatibleProfileConfig)` creates an
|
||||
in-memory `Profile`. Its public fields are `ID`, `Endpoint`, `Model`,
|
||||
`Temperature`, `MaxTokens`, `TopP`, `TimeoutSeconds`, `ServiceTier`,
|
||||
`ReasoningEffort`, `APIKeyRequired`, and `ExtraParams`.
|
||||
`WithProfiles` rejects duplicate IDs in one call.
|
||||
|
||||
A direct `RunRequest.APIKey` is request-scoped and takes precedence over
|
||||
`api_key_env` for the built-in client. It is excluded from JSON output and
|
||||
from `PreparedRun` and `RunResult`. The package's `String` and
|
||||
`GoString` methods report only whether a direct key is set. Do not use
|
||||
reflection-based dumps of request structs, which can bypass that redaction.
|
||||
|
||||
## Injected LLM Clients
|
||||
|
||||
`LLMClient` implements:
|
||||
|
||||
```go
|
||||
Generate(context.Context, GenerateRequest) (*GenerateResponse, error)
|
||||
```
|
||||
|
||||
Injected clients receive the rendered prompt, effective execution target, numeric
|
||||
target-presence metadata, optional structured-output specification, and direct
|
||||
request API key. `GenerateResponse` returns content and `TokenUsage`.
|
||||
Custom clients should avoid logging raw prompts or credentials.
|
||||
|
||||
## Errors
|
||||
|
||||
Public methods preserve these sentinel checks through `errors.Is`:
|
||||
|
||||
- `ErrInvalidConfig`
|
||||
- `ErrInvalidRequest`
|
||||
- `ErrPromptNotFound`
|
||||
- `ErrProfileNotFound`
|
||||
- `ErrProfileRequired`
|
||||
- `ErrPromptLoad`
|
||||
- `ErrProfileLoad`
|
||||
- `ErrAPIKeyEnvMissing`
|
||||
- `ErrArtifactLoad`
|
||||
- `ErrPromptRender`
|
||||
- `ErrLLMGenerate`
|
||||
- `ErrValidation`
|
||||
|
||||
`ErrProfileRequired` and `ErrAPIKeyEnvMissing` each also match
|
||||
`ErrInvalidRequest`, so callers can select either the broad request category or
|
||||
the specific condition.
|
||||
|
||||
For the HTTP interface, see the [HTTP API reference](../api.md).
|
||||
63
docs/development.md
Normal file
63
docs/development.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# Development
|
||||
|
||||
This is the contributor entry point for Scriptorium. Use the task-specific
|
||||
reading guide below before making changes. Canonical architecture, contracts,
|
||||
component behavior, and policies remain in their owning documents.
|
||||
|
||||
## Initial Orientation
|
||||
|
||||
Before starting work:
|
||||
|
||||
1. inspect the working tree and preserve unrelated changes;
|
||||
2. read the architecture policy for code or design work;
|
||||
3. read the policy, contract, and internal documents listed for the task;
|
||||
4. inspect the relevant implementation and tests before deciding how to change
|
||||
them.
|
||||
|
||||
Start with:
|
||||
|
||||
- [Architecture policy](policy/architecture.md) for system boundaries,
|
||||
invariants, and non-goals;
|
||||
- [Internal component overview](internal/overview.md) for the current package
|
||||
and component map;
|
||||
- [Documentation policy](policy/documentation.md) before changing
|
||||
documentation;
|
||||
- [Testing policy](policy/testing.md) before adding, rewriting, or deleting
|
||||
tests.
|
||||
|
||||
## Task-Specific Reading Guide
|
||||
|
||||
| Task | Read before changing |
|
||||
| --- | --- |
|
||||
| Repository orientation or component responsibility | [Internal component overview](internal/overview.md) and [architecture policy](policy/architecture.md) |
|
||||
| Public Go package or engine behavior | [Go package consumer contract](consumers/pkg-scriptorium.md), [internal component overview](internal/overview.md), [runner internals](internal/runner.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) |
|
||||
| CLI commands, flags, output, or exit behavior | [CLI contract](cli.md), [internal component overview](internal/overview.md), and [adapter internals](internal/adapters.md) |
|
||||
| HTTP routes, DTOs, limits, or status mapping | [HTTP API contract](api.md), [internal component overview](internal/overview.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) |
|
||||
| Application configuration | [Configuration contract](config.md), [internal component overview](internal/overview.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) |
|
||||
| Prompt, profile, schema, or artifact loading | [Configuration contract](config.md), [internal component overview](internal/overview.md), and [source internals](internal/sources.md) |
|
||||
| Runner orchestration, rendering, validation, or repair | [Runner internals](internal/runner.md) and [source internals](internal/sources.md) |
|
||||
| OpenAI-compatible request or response behavior | [OpenAI-compatible integration](integrations/openai-compatible-chat.md), [LLM internals](internal/llm.md), [runner internals](internal/runner.md), and [adapter internals](internal/adapters.md) |
|
||||
| Subprocess behavior | [Subprocess integration](integrations/subprocess.md) and [CLI contract](cli.md) |
|
||||
| Runtime operation or recovery | [Operations](operations.md) |
|
||||
| Examples or copyable assets | The owning contract for the demonstrated behavior and the related files under `examples/` |
|
||||
| Architecture decisions or future work | The [documentation policy](policy/documentation.md), relevant accepted ADRs such as [ADR 0001](adr/0001-adopt-canonical-documentation-ownership.md), and relevant roadmap documents under `roadmap/` |
|
||||
|
||||
For cross-cutting changes, follow every applicable row. Internal component
|
||||
documents own detailed subsystem change recipes.
|
||||
|
||||
## Baseline Validation
|
||||
|
||||
Use focused checks while iterating, then run validation proportionate to the
|
||||
change and the risks described by the testing policy.
|
||||
|
||||
The repository-level baseline for code changes is:
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/scriptorium
|
||||
```
|
||||
|
||||
Documentation-only work does not require the full Go suite unless it changes
|
||||
commands, examples, generated output, or another behavior that the suite
|
||||
validates. Always check changed links, paths, examples, and canonical ownership.
|
||||
@@ -1,322 +0,0 @@
|
||||
# Narratio -> Scriptorium CLI Integration
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
This document defines how Narratio should invoke Scriptorium through the **public CLI**.
|
||||
|
||||
This is a **subprocess integration contract**, not an internal Go API contract.
|
||||
|
||||
## 2. Assumptions
|
||||
|
||||
- `scriptorium` is installed and available on `PATH`.
|
||||
- Scriptorium is configured with `config.yml`.
|
||||
- `config.yml` provides `prompt_dir`, `profile_dir`, and `schema_dir` as needed.
|
||||
- Prompt and profile libraries are already deployed for the environment.
|
||||
- Narratio provides prepared artifact files (for example polished transcript, glossary, previous recap, campaign notes).
|
||||
- Initial integration is synchronous subprocess execution.
|
||||
- Narratio remains the orchestrator.
|
||||
|
||||
In normal operation, Narratio does not need to pass `--prompt-dir` and `--profile-dir` if they are supplied by Scriptorium config.
|
||||
|
||||
Narratio may pass `--config <PATH>` when it must use a non-default Scriptorium config file.
|
||||
|
||||
## 3. Core Commands Narratio May Call
|
||||
|
||||
Primary commands for subprocess integration:
|
||||
|
||||
- `scriptorium run`
|
||||
- `scriptorium render`
|
||||
|
||||
For production generation, use `scriptorium run`.
|
||||
|
||||
`scriptorium render` is for debugging, dry-runs, test assertions, and validating command construction without LLM execution.
|
||||
|
||||
Note: `scriptorium serve` and HTTP API exist, but they are not the initial integration path.
|
||||
|
||||
## 4. Command Selection Guidance
|
||||
|
||||
- Use `run` to generate an output artifact.
|
||||
- Use `render` to inspect the prepared prompt and effective settings without calling the LLM.
|
||||
- Use `render --format json` when Narratio/tests need structured prepare output.
|
||||
|
||||
## 5. Recommended `run` Invocation Shape
|
||||
|
||||
Production shape:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt <prompt_id> \
|
||||
--input transcript=<processed-transcript-path> \
|
||||
--out <output-artifact-path>
|
||||
```
|
||||
|
||||
Common optional additions:
|
||||
|
||||
- `--config <path>`: use a specific Scriptorium config file.
|
||||
- `--profile <profile_id>`: override prompt default profile.
|
||||
- `--var name=value` (repeatable): small metadata values.
|
||||
- `--input name=path` (repeatable): additional named artifacts.
|
||||
- `--timeout <duration>`: per-run timeout override.
|
||||
- Runtime model override flags (`--llm-base-url`, `--model`, etc.) only for exceptional/operator-directed cases.
|
||||
|
||||
## 6. Recommended `render` Invocation Shape
|
||||
|
||||
Human-readable debug shape:
|
||||
|
||||
```bash
|
||||
scriptorium render \
|
||||
--prompt <prompt_id> \
|
||||
--input transcript=<processed-transcript-path> \
|
||||
--format text
|
||||
```
|
||||
|
||||
Structured debug/test shape:
|
||||
|
||||
```bash
|
||||
scriptorium render \
|
||||
--prompt <prompt_id> \
|
||||
--input transcript=<processed-transcript-path> \
|
||||
--format json \
|
||||
--out <render-debug-path>
|
||||
```
|
||||
|
||||
`render` does **not** call the LLM, does **not** validate model output, and does **not** perform repair.
|
||||
|
||||
## 7. Inputs
|
||||
|
||||
- Pass inputs as repeated `--input name=path` flags.
|
||||
- `name` must match the Prompt Definition input name.
|
||||
- Prefer absolute paths, or paths relative to a working directory controlled by Narratio.
|
||||
- Pass Audita output as the primary transcript input.
|
||||
- Additional inputs may include glossary, previous recap, campaign notes, event logs, final state maps, or other prompt-specific artifacts.
|
||||
- Scriptorium reads input files directly; Narratio does not need to inline file content for CLI use.
|
||||
|
||||
## 8. Variables
|
||||
|
||||
Use repeated `--var name=value` for small metadata values.
|
||||
|
||||
Typical examples:
|
||||
|
||||
- `session_date`
|
||||
- `session_id`
|
||||
- `campaign_name`
|
||||
- `previous_session_id`
|
||||
- `output_kind`
|
||||
|
||||
Large content belongs in input files, not `--var` values.
|
||||
|
||||
## 9. Prompt IDs and Output Artifact Types
|
||||
|
||||
Narratio should treat prompt IDs as configuration, not hardcoded business logic.
|
||||
|
||||
Narratio config may map stage/output names to prompt IDs, for example:
|
||||
|
||||
- session recap prompt
|
||||
- structured event extraction prompt
|
||||
- glossary suggestion prompt
|
||||
- player-facing summary prompt
|
||||
|
||||
Prompt IDs used by Narratio should come from the deployed Scriptorium prompt library.
|
||||
|
||||
## 10. Profiles
|
||||
|
||||
- Prompts may declare `default_profile`.
|
||||
- Narratio may omit `--profile` to use prompt default profile.
|
||||
- Narratio may pass `--profile` to force profile selection.
|
||||
- This enables environment/profile selection like `local-fast`, `local-quality`, `frontier`, `batch`, or test profiles.
|
||||
- Profile names should generally be Narratio configuration values.
|
||||
|
||||
## 11. Runtime Overrides
|
||||
|
||||
Supported runtime override flags:
|
||||
|
||||
- `--llm-base-url`
|
||||
- `--model`
|
||||
- `--api-key-env`
|
||||
- `--temperature`
|
||||
- `--max-tokens`
|
||||
- `--top-p`
|
||||
- `--timeout`
|
||||
|
||||
Guidance:
|
||||
|
||||
- Keep normal model/runtime settings in Execution Profiles.
|
||||
- Use runtime overrides only for explicit per-run exceptions, tests, or operator overrides.
|
||||
- Never pass raw API keys on the command line.
|
||||
- `--api-key-env` names an environment variable; Narratio must ensure that variable is set in subprocess environment.
|
||||
|
||||
## 12. Config Behavior
|
||||
|
||||
- Default config path: `/etc/scriptorium/config.yml`.
|
||||
- `--config <PATH>` overrides default path.
|
||||
- Missing default config is allowed by Scriptorium.
|
||||
- If `--config` is provided explicitly, the file must exist and be valid.
|
||||
- CLI flags override `config.yml`.
|
||||
- `config.yml` overrides built-in application defaults.
|
||||
|
||||
Narratio can either:
|
||||
|
||||
- rely on system default config path, or
|
||||
- carry an explicit config path and pass `--config`.
|
||||
|
||||
## 13. Environment Handling
|
||||
|
||||
Subprocess environment recommendations:
|
||||
|
||||
- Pass through required API-key environment variables referenced by `api_key_env`.
|
||||
- Do not pass raw API keys as CLI arguments.
|
||||
- Avoid logging full environment dumps.
|
||||
- Capture stdout and stderr separately.
|
||||
- Use a controlled working directory.
|
||||
- Prefer absolute artifact paths.
|
||||
|
||||
## 14. Output Handling
|
||||
|
||||
For `scriptorium run`:
|
||||
|
||||
- Use `--out` when Narratio needs durable artifact files.
|
||||
- Without `--out`, artifact content is written to stdout.
|
||||
- Preferred orchestration pattern: always use `--out`, then treat the file as stage output artifact.
|
||||
- Capture stderr for diagnostics.
|
||||
|
||||
For `scriptorium render`:
|
||||
|
||||
- Use `--out` to store render diagnostics.
|
||||
- Use `--format json` when tests need to inspect selected profile, effective runtime settings, input hashes, prompt hash, and rendered messages.
|
||||
|
||||
## 15. Exit Status and Errors
|
||||
|
||||
Current CLI behavior (verified from implementation/tests):
|
||||
|
||||
- `0`: success.
|
||||
- `1`: runtime/parse/config/load/render/generation/IO error.
|
||||
- `2`: run completed but output validation failed (`ValidationFailed`).
|
||||
|
||||
Additional details:
|
||||
|
||||
- On `run`, output artifact write happens before exit code selection. If validation fails, artifact may still be written and exit code is `2`.
|
||||
- `stderr` carries both errors and normal run summary output; non-empty stderr alone does not imply failure.
|
||||
- `render` returns `0` on success and `1` on failures.
|
||||
|
||||
Narratio should treat non-zero exit codes as failed stage execution, but may record generated artifact paths if a run exited `2` and output file exists.
|
||||
|
||||
## 16. Recommended Narratio Integration Pattern
|
||||
|
||||
1. Build CLI args from Narratio stage configuration.
|
||||
2. Use subprocess context cancellation/timeout.
|
||||
3. Pass absolute input paths.
|
||||
4. Pass `--out` to a session-scoped artifact path.
|
||||
5. Add `--var` metadata values.
|
||||
6. Optionally add `--config`.
|
||||
7. Optionally add `--profile`.
|
||||
8. Ensure required API-key env vars are present.
|
||||
9. Run subprocess synchronously.
|
||||
10. Capture stdout/stderr separately.
|
||||
11. On success, store output artifact path and invocation metadata in stage artifacts.
|
||||
12. On failure, store exit code and stderr diagnostics in stage status.
|
||||
|
||||
## 17. Suggested Narratio Configuration Shape
|
||||
|
||||
Illustrative (not required schema):
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
config_path: /etc/scriptorium/config.yml
|
||||
stages:
|
||||
session_recap:
|
||||
prompt_id: dnd.session_recap
|
||||
profile_id: local-quality # optional
|
||||
inputs: [transcript, glossary, previous_recap]
|
||||
vars: [session_id, session_date, campaign_name]
|
||||
output_path_template: artifacts/{session_id}/session_recap.md
|
||||
timeout: 2m
|
||||
render_debug: false
|
||||
```
|
||||
|
||||
The key idea: map Narratio stage/artifact names to prompt ID, optional profile, expected inputs, and output destination.
|
||||
|
||||
## 18. Testing Strategy for Narratio Integration
|
||||
|
||||
- Use `scriptorium render --format json` to verify command construction without LLM calls.
|
||||
- Use dedicated test prompt/profile libraries for integration tests.
|
||||
- Use small fixture transcripts.
|
||||
- Verify missing-input failure behavior.
|
||||
- Verify prompt `default_profile` behavior.
|
||||
- Verify explicit `--profile` override behavior.
|
||||
- Verify `--config` behavior (default and explicit).
|
||||
- Verify output file creation when `--out` is used.
|
||||
- Verify stderr capture on failures.
|
||||
- Avoid real API keys in tests.
|
||||
|
||||
## 19. Security and Privacy Notes
|
||||
|
||||
- Never pass raw API keys on command line.
|
||||
- Do not log full rendered prompts by default; transcripts may contain sensitive content.
|
||||
- Avoid logging prompt content unless explicit debug mode is enabled.
|
||||
- Treat generated artifacts as potentially sensitive.
|
||||
- Use session-scoped, access-controlled output paths.
|
||||
- `api_key_env` names should come from environment management, not embedded secrets.
|
||||
|
||||
## 20. Initial D&D Artifact Generation Examples
|
||||
|
||||
These are examples only. Use prompt IDs from the deployed prompt library.
|
||||
|
||||
Session recap:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.session_recap \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--input glossary=/work/session-42/glossary.yml \
|
||||
--out /work/session-42/artifacts/session_recap.md
|
||||
```
|
||||
|
||||
Structured events:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.structured_events \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--out /work/session-42/artifacts/structured_events.json
|
||||
```
|
||||
|
||||
Glossary suggestions:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.glossary_suggestions \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--input previous_recap=/work/session-41/artifacts/session_recap.md \
|
||||
--out /work/session-42/artifacts/glossary_suggestions.md
|
||||
```
|
||||
|
||||
Player-facing summary:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.player_summary \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--input structured_events=/work/session-42/artifacts/structured_events.json \
|
||||
--out /work/session-42/artifacts/player_summary.md
|
||||
```
|
||||
|
||||
## 21. Non-Goals
|
||||
|
||||
Initial Narratio integration should not:
|
||||
|
||||
- call Scriptorium internal Go packages
|
||||
- use HTTP API as the primary path
|
||||
- expect Scriptorium to read S3 refs directly
|
||||
- make Scriptorium responsible for Narratio stage state
|
||||
- make Scriptorium responsible for notification
|
||||
- require Scriptorium to understand D&D workflow semantics beyond prompt definitions
|
||||
|
||||
## 22. Future Extension Notes
|
||||
|
||||
Possible later extensions:
|
||||
|
||||
- HTTP API integration
|
||||
- S3 artifact references if Scriptorium adds S3 reader support
|
||||
- storing render diagnostics alongside generated artifacts
|
||||
- token budgeting/prompt-size checks
|
||||
- batch execution if Scriptorium later adds batch support
|
||||
121
docs/integrations/openai-compatible-chat.md
Normal file
121
docs/integrations/openai-compatible-chat.md
Normal file
@@ -0,0 +1,121 @@
|
||||
# OpenAI-Compatible Chat Integration
|
||||
|
||||
This is the outbound wire contract for Scriptorium's OpenAI-compatible
|
||||
chat-completions client.
|
||||
|
||||
## Endpoint And Method
|
||||
|
||||
Scriptorium uses the request endpoint override when present; otherwise it uses
|
||||
the configured client base URL. It removes a trailing slash and sends
|
||||
`POST /chat/completions`.
|
||||
|
||||
For example, `http://localhost:8000/v1` becomes
|
||||
`http://localhost:8000/v1/chat/completions`.
|
||||
|
||||
## Request Payload
|
||||
|
||||
The payload always contains `model` and rendered `messages`. It additionally
|
||||
contains these fields when applicable:
|
||||
|
||||
| Field | Inclusion |
|
||||
| --- | --- |
|
||||
| `session_id` | Non-empty rendered prompt session ID. |
|
||||
| `temperature` | Non-zero effective value or an explicit zero override. |
|
||||
| `max_tokens` | Non-zero effective value or an explicit zero override. |
|
||||
| `top_p` | Non-zero effective value or an explicit zero override. |
|
||||
| `service_tier` | Any non-empty configured value. |
|
||||
| `reasoning_effort` | Any non-empty configured value. |
|
||||
| `response_format` | Structured output is requested. |
|
||||
| provider-specific fields | Flattened from `extra_params`. |
|
||||
|
||||
`service_tier` and `reasoning_effort` are forwarded without a provider value
|
||||
catalog; the selected backend decides which values it supports.
|
||||
|
||||
`extra_params` are top-level JSON fields, not a nested object. Keys cannot be
|
||||
empty or collide with `model`, `session_id`, `messages`, `temperature`,
|
||||
`max_tokens`, `top_p`, `service_tier`, `reasoning_effort`, or
|
||||
`response_format`. Values must be JSON-serializable.
|
||||
|
||||
A rendered `session_id` is sent as a top-level JSON field, not as a header.
|
||||
Empty values are omitted. The maximum length is 256 Unicode code points.
|
||||
|
||||
Messages without cache control use string `content`. A message with cache
|
||||
control uses one text block:
|
||||
|
||||
```json
|
||||
{
|
||||
"role": "system",
|
||||
"content": [{
|
||||
"type": "text",
|
||||
"text": "rendered text",
|
||||
"cache_control": {"type": "ephemeral", "ttl": "1h"}
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
When the prompt omits cache-control `ttl`, the payload omits `ttl`.
|
||||
Structured JSON Schema output is sent as:
|
||||
|
||||
```json
|
||||
{
|
||||
"response_format": {
|
||||
"type": "json_schema",
|
||||
"json_schema": {
|
||||
"name": "schema name",
|
||||
"strict": true,
|
||||
"schema": {"type": "object"}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Authentication And Timeout
|
||||
|
||||
When a direct request API key is present, Scriptorium sends
|
||||
`Authorization: Bearer <key>` and does not read `api_key_env`. Otherwise, it
|
||||
resolves the configured non-empty `api_key_env` at request time and sends the
|
||||
same header. If neither mechanism supplies a key, it sends no
|
||||
`Authorization` header.
|
||||
|
||||
The transport-wide safety cap is chosen at client construction. A positive
|
||||
timeout on a supplied `http.Client` takes precedence over a positive
|
||||
`Config.Timeout`; if neither is positive, the internal ten-minute default is
|
||||
used. The supplied client is cloned, and zero or negative timeout values are
|
||||
treated as unset.
|
||||
|
||||
Separately, a positive effective `timeout_seconds` creates a deadline for each
|
||||
outbound generation call. Its value follows the execution-setting hierarchy:
|
||||
an explicit request override, then a non-zero profile value, then the
|
||||
600-second framework default. An explicit request override of zero disables
|
||||
only this generation deadline. Negative values are rejected before a request
|
||||
is sent.
|
||||
|
||||
The complete observable rule is that the earliest caller-context deadline,
|
||||
transport cap, or positive generation deadline terminates the call. Transport
|
||||
and cancellation failures retain the generation-error classification.
|
||||
|
||||
## Response Subset And Failures
|
||||
|
||||
A successful provider response must supply non-empty
|
||||
`choices[0].message.content`. Scriptorium reads these optional or required
|
||||
usage fields when present:
|
||||
|
||||
- `usage.prompt_tokens`
|
||||
- `usage.completion_tokens`
|
||||
- `usage.total_tokens`
|
||||
- `usage.prompt_tokens_details.cached_tokens`
|
||||
- `usage.cache_write_tokens`
|
||||
|
||||
Missing cache usage is reported as zero. Invalid JSON, an empty choices array,
|
||||
or empty first-choice content is a malformed provider response. Network and
|
||||
request-construction failures, non-2xx responses, and malformed responses fail
|
||||
the outbound call. Provider response bodies are not exposed by this client.
|
||||
|
||||
The client does not implement built-in retries, tool calls, top-level
|
||||
`cache_control`, or multi-request payload modes.
|
||||
|
||||
## Related References
|
||||
|
||||
Prompt schema preparation and runner orchestration are described in
|
||||
[runner internals](../internal/runner.md). Prompt and profile configuration is
|
||||
defined by the [configuration reference](../config.md).
|
||||
41
docs/integrations/subprocess.md
Normal file
41
docs/integrations/subprocess.md
Normal file
@@ -0,0 +1,41 @@
|
||||
# Subprocess Integration
|
||||
|
||||
This document covers process-boundary behavior for callers that invoke
|
||||
Scriptorium as a child process. Command syntax, flags, output, and exit codes
|
||||
are defined by the [CLI reference](../cli.md). Interface selection belongs in
|
||||
the [consumer integration overview](../consumers/api.md).
|
||||
|
||||
## Process Contract
|
||||
|
||||
Use `scriptorium render` when the caller needs prepared output without a model
|
||||
call, and `scriptorium run` for generation. Pass an explicit `--config` or
|
||||
make the configuration search paths available to the child process; configuration
|
||||
discovery, fields, profile selection, and credential mechanisms are defined in
|
||||
the [configuration reference](../config.md).
|
||||
|
||||
Pass required API-key environment variables through the child environment. Do
|
||||
not place raw API keys in arguments. Keep the environment limited to the values
|
||||
needed for the selected profile.
|
||||
|
||||
## Streams And Output Ownership
|
||||
|
||||
Capture stdout and stderr separately. Stdout contains the requested artifact or
|
||||
prepared output unless the caller selects an output file; stderr contains
|
||||
summaries, diagnostics, and server messages. The exact destinations and status
|
||||
meanings are part of the [CLI reference](../cli.md), not a stable stderr data
|
||||
protocol.
|
||||
|
||||
When using `--out`, the caller owns the output path, its permissions, and
|
||||
cleanup. Treat rendered prompts, generated artifacts, stdout, and stderr as
|
||||
potentially sensitive.
|
||||
|
||||
## Cancellation And Recovery
|
||||
|
||||
A CLI invocation performs one synchronous request and creates no durable run
|
||||
state. A supervising process that needs cancellation must terminate the child
|
||||
process according to its own process-management policy. A later invocation is a
|
||||
new request and can make another model call; there is no resume or checkpoint
|
||||
protocol.
|
||||
|
||||
For deployment, filesystem permissions, and sensitive-artifact handling, see
|
||||
the [operations guide](../operations.md).
|
||||
116
docs/internal/adapters.md
Normal file
116
docs/internal/adapters.md
Normal file
@@ -0,0 +1,116 @@
|
||||
# Adapter Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
Adapters translate external inputs into public engine requests and translate
|
||||
public results or errors back to their interface. They own IO and presentation
|
||||
mechanics; use-case decisions remain behind the root `scriptorium` facade.
|
||||
|
||||
External contracts are canonical in the [CLI reference](../cli.md), [HTTP API
|
||||
reference](../api.md), and [Go package contract](../consumers/pkg-scriptorium.md).
|
||||
|
||||
## Components And Collaborators
|
||||
|
||||
- `cmd/scriptorium` passes process arguments and streams to
|
||||
`internal/adapter/cli`.
|
||||
- `internal/adapter/cli` parses commands, resolves application settings through
|
||||
`internal/config`, constructs the public engine, and owns process output
|
||||
handling.
|
||||
- `internal/adapter/http` decodes DTOs, maps them to public run requests,
|
||||
calls its local public `Runner` interface, and maps public errors and results
|
||||
to HTTP DTOs.
|
||||
- The root `scriptorium` package maps its public types and options to internal
|
||||
collaborators and maps selected internal errors to public sentinels.
|
||||
- `internal/format` formats public prepared runs for the CLI.
|
||||
|
||||
## Wiring Flows
|
||||
|
||||
### CLI
|
||||
|
||||
The CLI resolves configuration before constructing the public engine. `run`
|
||||
calls `Engine.Run` with a public request and `render` calls `Engine.Prepare`
|
||||
with the same request mapping. `serve` constructs the HTTP-owned restricted
|
||||
artifact reader, injects it with `WithArtifactReader`, passes the resulting
|
||||
engine directly to the HTTP handler, and starts the server.
|
||||
|
||||
Parser state records whether numeric runtime values were explicitly supplied.
|
||||
That presence is carried into `scriptorium.ExecutionTargetOverride`, allowing
|
||||
the engine to distinguish omitted values from explicit zero overrides.
|
||||
|
||||
### HTTP
|
||||
|
||||
The handler first enforces transport limits, strict JSON decoding, and the
|
||||
minimal request shape. It maps DTO values to public types without deciding
|
||||
prompt selection, source behavior, or validation semantics. On success it maps
|
||||
the public result to the response DTO; on failure it uses `errors.Is` over
|
||||
public framework errors and HTTP-local artifact-policy errors to choose the
|
||||
public error mapping.
|
||||
|
||||
The [HTTP API reference](../api.md) owns the route, DTO schema, status codes,
|
||||
and externally observable limit behavior.
|
||||
|
||||
### Public Go Facade
|
||||
|
||||
`NewEngine` applies public options, selects filesystem, `fs.FS`, single-file,
|
||||
or in-memory dependencies, and constructs a runner. The conversion functions
|
||||
copy maps and slices across the boundary so callers do not receive internal
|
||||
domain values. The facade maps selected internal errors to the public sentinel
|
||||
set and keeps direct request API keys out of public results.
|
||||
|
||||
## Package-Local Guarantees
|
||||
|
||||
- Adapters do not embed framework orchestration or source-loading decisions.
|
||||
- Configuration is resolved before adapter dependency composition.
|
||||
- CLI and HTTP consume the public engine without a repairer; a repairer remains
|
||||
available only through explicit internal runner construction.
|
||||
- DTO conversion preserves explicit numeric-override presence.
|
||||
- Error mapping matches error identities, not error text.
|
||||
- No adapter creates durable run state; caller-selected output files are not
|
||||
application state.
|
||||
|
||||
## Failure And Verification Boundaries
|
||||
|
||||
Keep external error payloads concise, preserve strict external decoding, and do
|
||||
not serialize resolved secret values. Validation content failures remain result
|
||||
state; runtime failures remain errors for the relevant adapter to map.
|
||||
|
||||
Inspect focused tests when changing this area:
|
||||
|
||||
- `internal/adapter/cli/run_test.go`
|
||||
- `internal/adapter/http/handler_test.go`
|
||||
- `engine_test.go`
|
||||
- `internal/format/prepared_run_test.go`
|
||||
|
||||
Run the affected adapter package tests and recheck the relevant canonical
|
||||
contract. The [testing policy](../policy/testing.md) owns global test
|
||||
sufficiency guidance.
|
||||
|
||||
## Change Recipes
|
||||
|
||||
### Application Configuration Fields
|
||||
|
||||
1. Add the field to the relevant `internal/config` shape and default handling.
|
||||
2. Parse and validate it, then preserve configuration and CLI-override
|
||||
precedence while wiring it through its consuming adapter.
|
||||
3. Add focused configuration and adapter tests for parsing, mapping, and
|
||||
effective behavior.
|
||||
4. Update the [configuration contract](../config.md) and any affected external
|
||||
contract.
|
||||
|
||||
### CLI Flags
|
||||
|
||||
1. Add the flag to the relevant parser in `internal/adapter/cli/run.go`.
|
||||
2. Keep command scope and application-configuration precedence intentional.
|
||||
3. Add or update parser and command tests in
|
||||
`internal/adapter/cli/run_test.go`.
|
||||
4. Update the [CLI contract](../cli.md) and affected maintained examples.
|
||||
|
||||
### Adapter Capabilities
|
||||
|
||||
1. Define or reuse an adapter-local consumer interface with public facade
|
||||
types when a test seam is needed.
|
||||
2. Implement translation and IO behavior without moving framework decisions out
|
||||
of the public engine.
|
||||
3. Add focused mapping, parsing, and error-behavior tests.
|
||||
4. Update this document and the affected public or integration contract. Update
|
||||
[source internals](sources.md) when source-loading behavior changes.
|
||||
86
docs/internal/llm.md
Normal file
86
docs/internal/llm.md
Normal file
@@ -0,0 +1,86 @@
|
||||
# LLM Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
`internal/llm` defines the provider-neutral `Client` interface and the
|
||||
OpenAI-compatible client implementation. The [OpenAI-compatible integration
|
||||
contract](../integrations/openai-compatible-chat.md) owns the outbound HTTP wire
|
||||
format and protocol behavior.
|
||||
|
||||
## Construction
|
||||
|
||||
`NewOpenAICompatibleClient` validates a non-empty configured base URL, records
|
||||
an optional default model, and resolves one transport cap. A supplied client
|
||||
with a positive timeout supplies that cap; otherwise a positive configured
|
||||
timeout is used, then the internal default.
|
||||
|
||||
When callers supply an `http.Client`, construction clones it rather than
|
||||
mutating the caller's instance. A supplied client with a zero or negative
|
||||
timeout receives the resolved transport cap in the clone. The client stores the
|
||||
trimmed base URL, default model, and cloned client.
|
||||
|
||||
## Generate Flow
|
||||
|
||||
`Generate` receives a `domain.GenerateRequest` from the runner:
|
||||
|
||||
1. validate the effective timeout and choose the request endpoint;
|
||||
2. map the domain request to the internal wire-request representation;
|
||||
3. validate and flatten extra parameters and encode JSON;
|
||||
4. derive a child context when the effective generation timeout is positive,
|
||||
then create the HTTP request with that context;
|
||||
5. prefer a direct API key, otherwise resolve the configured key environment
|
||||
variable;
|
||||
6. execute with the construction-time HTTP client, reject non-success status
|
||||
responses without returning
|
||||
provider response bodies; and
|
||||
7. decode the response subset into `domain.GenerateResponse`.
|
||||
|
||||
`openAIChatRequestFromGenerateRequest` is the conversion boundary for effective
|
||||
model defaults, explicit numeric-presence state, rendered messages, structured
|
||||
output, and session-ID validation. `openAIChatRequestPayload` protects reserved
|
||||
fields and JSON encoding before an HTTP call. The external payload shape is
|
||||
defined only in the [integration contract](../integrations/openai-compatible-chat.md).
|
||||
|
||||
## Error Categories
|
||||
|
||||
The package uses these internal sentinels:
|
||||
|
||||
- `ErrInvalidConfig` for invalid client construction;
|
||||
- `ErrInvalidRequest` for invalid effective generation input;
|
||||
- `ErrRequestFailed` for request construction or transport failures;
|
||||
- `ErrUnexpectedStatus` for non-success HTTP responses; and
|
||||
- `ErrMalformedResponse` for invalid or incomplete successful-response data.
|
||||
|
||||
The runner maps an invalid LLM request to its invalid-request category and
|
||||
other LLM failures to its generation category. Adapters then apply their public
|
||||
error contracts.
|
||||
|
||||
## Package-Local Guarantees
|
||||
|
||||
- The default-model fallback happens before wire encoding.
|
||||
- Per-generation timeout handling derives a request context; it never replaces
|
||||
or mutates the configured HTTP client's transport cap.
|
||||
- Direct API keys take precedence over environment lookup within this client.
|
||||
- Provider response bodies are discarded for non-success status responses.
|
||||
- The client does not implement retries, tool calls, or a stateful session
|
||||
store.
|
||||
|
||||
## Verification And Change Recipe
|
||||
|
||||
Inspect:
|
||||
|
||||
- `internal/llm/openai_compatible_client_test.go`
|
||||
- `internal/usecase/runner_test.go`
|
||||
- `internal/adapter/http/handler_test.go`
|
||||
|
||||
When changing the client:
|
||||
|
||||
1. keep domain-to-wire mapping inside `internal/llm` and preserve the `Client`
|
||||
interface;
|
||||
2. test construction, timeout selection, mapping, and error categorization;
|
||||
3. update the [OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md)
|
||||
for any observable wire or protocol change; and
|
||||
4. update [runner internals](runner.md) if the client boundary or structured
|
||||
output handoff changes.
|
||||
|
||||
The [testing policy](../policy/testing.md) owns global test sufficiency.
|
||||
48
docs/internal/overview.md
Normal file
48
docs/internal/overview.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# Internal Component Overview
|
||||
|
||||
## Purpose
|
||||
|
||||
This is the inventory of Scriptorium's implemented components for contributors.
|
||||
The [architecture policy](../policy/architecture.md) owns normative boundaries
|
||||
and invariants; public behavior belongs in the linked contracts.
|
||||
|
||||
## Public And Command Entrypoints
|
||||
|
||||
| Component | Implemented responsibility | References |
|
||||
| --- | --- | --- |
|
||||
| Root package `scriptorium` | Public Go facade that constructs the engine, exposes request/result types and options, and maps internal errors. | [Go package contract](../consumers/pkg-scriptorium.md), [adapter internals](adapters.md) |
|
||||
| `cmd/scriptorium` | Process entrypoint that delegates command execution to the CLI adapter. | [CLI contract](../cli.md), [adapter internals](adapters.md) |
|
||||
|
||||
## Adapters, Domain, And Use Case
|
||||
|
||||
| Component | Implemented responsibility | References |
|
||||
| --- | --- | --- |
|
||||
| `internal/adapter/cli` | Parses CLI commands, constructs the public engine from application settings, and handles process input and output. | [CLI contract](../cli.md), [adapter internals](adapters.md) |
|
||||
| `internal/adapter/http` | Maps HTTP requests and responses through public engine values, maps public errors, and owns restricted HTTP artifact policy. | [HTTP API contract](../api.md), [adapter internals](adapters.md) |
|
||||
| `internal/domain` | Defines core request, result, output-contract, and LLM-boundary types. | [runner internals](runner.md) |
|
||||
| `internal/usecase` | Implements `Runner` preparation, execution, validation coordination, and the repairer boundary. | [runner internals](runner.md) |
|
||||
|
||||
## Configuration And Sources
|
||||
|
||||
| Component | Implemented responsibility | References |
|
||||
| --- | --- | --- |
|
||||
| `internal/config` | Loads application settings, applies defaults, and applies CLI overrides. | [configuration contract](../config.md), [adapter internals](adapters.md) |
|
||||
| `internal/defaults` | Holds compile-time default values used when application settings are resolved. | [configuration contract](../config.md) |
|
||||
| `internal/promptdef` | Loads prompt definitions from filesystem and `fs.FS` sources. | [configuration contract](../config.md), [source internals](sources.md) |
|
||||
| `internal/profile` | Loads filesystem and `fs.FS` execution profiles and combines profile repositories. | [configuration contract](../config.md), [source internals](sources.md) |
|
||||
| `internal/profile/builtin` | Provides embedded built-in execution profiles as a repository. | [configuration contract](../config.md), [source internals](sources.md) |
|
||||
| `internal/filecatalog` | Provides shared YAML discovery and source-root helpers. | [source internals](sources.md) |
|
||||
| `internal/artifact` | Provides the framework's ordinary inline and unrestricted file artifact reader. | [configuration contract](../config.md), [source internals](sources.md) |
|
||||
| `internal/prompt` | Renders prompt templates into messages. | [runner internals](runner.md) |
|
||||
|
||||
## Formatting, Validation, And Model Access
|
||||
|
||||
| Component | Implemented responsibility | References |
|
||||
| --- | --- | --- |
|
||||
| `internal/format` | Formats public prepared-run information for CLI output. | [CLI contract](../cli.md), [adapter internals](adapters.md) |
|
||||
| `internal/validate` | Defines validation interfaces and provides standard filesystem and `fs.FS` schema validation. | [configuration contract](../config.md), [source internals](sources.md), [runner internals](runner.md) |
|
||||
| `internal/llm` | Defines the provider-neutral LLM client boundary and its OpenAI-compatible implementation. | [OpenAI-compatible integration](../integrations/openai-compatible-chat.md), [LLM internals](llm.md), [runner internals](runner.md) |
|
||||
|
||||
Focused internal documents describe the components that have detailed
|
||||
orchestration, adapter, or source behavior. Package tests live alongside the
|
||||
implementation and are identified in those focused documents where relevant.
|
||||
120
docs/internal/runner.md
Normal file
120
docs/internal/runner.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# Runner Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
`internal/usecase.Runner` is the prompt-execution orchestrator. It prepares
|
||||
domain requests, invokes an injected LLM client, validates output, and returns
|
||||
domain results. Transport parsing, response mapping, and public type conversion
|
||||
remain outside this package.
|
||||
|
||||
The [configuration reference](../config.md) owns prompt, profile, schema, and
|
||||
runtime-setting definitions. Public error behavior is defined by the
|
||||
[HTTP API](../api.md) and [Go package](../consumers/pkg-scriptorium.md)
|
||||
contracts.
|
||||
|
||||
## Dependencies And Construction
|
||||
|
||||
`Runner` receives these collaborators:
|
||||
|
||||
- `promptdef.Repository`;
|
||||
- `profile.Repository`;
|
||||
- `artifact.Reader`;
|
||||
- `prompt.Renderer`;
|
||||
- `llm.Client`;
|
||||
- `validate.Validator`; and
|
||||
- an optional `OutputRepairer`.
|
||||
|
||||
`NewRunner` constructs a runner without a repairer. `NewRunnerWithRepairer`
|
||||
accepts one explicitly. The public engine chooses concrete repositories and
|
||||
readers; executable adapters reach the runner only through that engine. The
|
||||
runner does not load application configuration.
|
||||
|
||||
## Prepare Flow
|
||||
|
||||
`Prepare` performs one deterministic preparation pass for a request:
|
||||
|
||||
1. validate the prompt ID and load the prompt definition;
|
||||
2. hash the definition and select the explicit or default profile;
|
||||
3. load the profile and resolve effective execution settings;
|
||||
4. validate endpoint, model, and credential availability;
|
||||
5. resolve the output contract and, for JSON Schema output, load a structured
|
||||
schema document before model execution;
|
||||
6. read and hash input artifacts;
|
||||
7. render messages and the session ID; and
|
||||
8. return a `PreparedRun` containing the effective state and rendered-prompt
|
||||
hash.
|
||||
|
||||
Execution settings merge defaults, profile values, and a request override.
|
||||
Numeric override presence is retained so explicit zero values are not confused
|
||||
with omissions.
|
||||
|
||||
## Run And Validation Flow
|
||||
|
||||
`Run` creates a run ID and timestamps, then calls `Prepare` rather than
|
||||
duplicating preparation. It sends the prepared prompt, effective target,
|
||||
target-presence state, and optional structured-output specification to the LLM
|
||||
client. It converts the returned content to an output artifact, validates it,
|
||||
and returns the artifact, validation, hashes, usage, and timing metadata.
|
||||
|
||||
A validator can return a content result or an operational error. Content
|
||||
failures stay in the result; schema loading, compilation, and validator
|
||||
operational failures are returned as `ErrValidation`. The canonical distinction
|
||||
for callers is documented by the public contracts.
|
||||
|
||||
## Repair Boundary
|
||||
|
||||
Repair is an internal optional loop. It starts only when a repairer is present,
|
||||
the output contract permits one or more attempts, validation failed, and the
|
||||
validation mode is JSON or JSON Schema. Each repair receives the previous
|
||||
output, validation errors, effective target, structured-output specification,
|
||||
and attempt metadata; every repaired result is validated again.
|
||||
|
||||
`NewDefaultOutputRepairer` delegates to the injected LLM client. The public
|
||||
engine, and therefore CLI and HTTP, uses `NewRunner` and does not inject this
|
||||
repairer.
|
||||
|
||||
## Error Translation
|
||||
|
||||
Runner sentinels identify failure categories for adapters:
|
||||
|
||||
- `ErrInvalidRequest`
|
||||
- `ErrProfileRequired`
|
||||
- `ErrAPIKeyEnvMissing` and `ErrAPIKeyRequired`
|
||||
- `ErrPromptLoad`, `ErrProfileLoad`, and `ErrArtifactLoad`
|
||||
- `ErrPromptRender`
|
||||
- `ErrLLMGenerate`
|
||||
- `ErrValidation`
|
||||
|
||||
Wrap errors with those sentinels and preserve their identities through
|
||||
`errors.Is`; adapters must not classify errors by message text. The runner
|
||||
passes direct keys only to the LLM boundary and never includes resolved key
|
||||
values in prepared or run results.
|
||||
|
||||
## Package-Local Guarantees
|
||||
|
||||
- `Run` always reuses `Prepare`.
|
||||
- Schema documents are loaded before the initial LLM call when structured output
|
||||
is required.
|
||||
- Output validation records attempts used, including repair attempts.
|
||||
- Runner state is per request; the package does not create a durable run store
|
||||
or manifest.
|
||||
- Source, renderer, validator, and LLM implementations remain injected
|
||||
boundaries.
|
||||
|
||||
## Verification And Change Recipe
|
||||
|
||||
Inspect:
|
||||
|
||||
- `internal/usecase/runner_test.go`
|
||||
- `engine_test.go`
|
||||
|
||||
When changing orchestration:
|
||||
|
||||
1. identify the collaborator boundary and the affected `Prepare` or `Run` state;
|
||||
2. preserve the `Run`-through-`Prepare` path and error identity;
|
||||
3. add focused runner or integration tests for changed state transitions,
|
||||
validation, or repair behavior; and
|
||||
4. update the owning external contract and any affected source or LLM internal
|
||||
document.
|
||||
|
||||
The [testing policy](../policy/testing.md) owns global test sufficiency.
|
||||
99
docs/internal/sources.md
Normal file
99
docs/internal/sources.md
Normal file
@@ -0,0 +1,99 @@
|
||||
# Source Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
This document describes how source packages load prompt definitions, profiles,
|
||||
schemas, and artifacts. The [configuration reference](../config.md) owns their
|
||||
user-facing formats and settings. The [HTTP API reference](../api.md) owns
|
||||
HTTP-visible artifact outcomes; [operations](../operations.md) owns deployment
|
||||
handling.
|
||||
|
||||
## Prompt Definitions
|
||||
|
||||
`internal/promptdef` provides filesystem and `fs.FS` repositories. Both use
|
||||
`internal/filecatalog` for recursive YAML discovery, deterministic ordering,
|
||||
display paths, and root cleaning.
|
||||
|
||||
Repositories select a prompt by YAML ID and optional version rather than by
|
||||
path. They decode through strict YAML handling, reject duplicate matching
|
||||
definitions, and resolve `content_file` relative to the definition. The `fs.FS`
|
||||
implementation resolves content paths inside its source root; absolute paths and
|
||||
traversal outside that root are rejected before file access.
|
||||
|
||||
## Profiles And Built-Ins
|
||||
|
||||
`internal/profile` provides filesystem, `fs.FS`, and overlay repositories.
|
||||
`internal/profile/builtin` exposes embedded assets through the same repository
|
||||
interface.
|
||||
|
||||
An overlay asks its primary source first. It falls back only when the primary
|
||||
reports `ErrProfileNotFound`; invalid YAML, duplicate IDs, validation failures,
|
||||
and raw-key failures are returned rather than hidden by fallback. This makes a
|
||||
custom ID override a built-in ID while retaining errors in the custom source.
|
||||
|
||||
The public engine can overlay in-memory profiles ahead of both file-backed and
|
||||
built-in repositories. Profile field definitions, validation ranges, and the
|
||||
built-in catalog remain in the [configuration reference](../config.md).
|
||||
|
||||
## Schemas
|
||||
|
||||
`internal/validate` supplies `StandardValidator` for filesystem sources and
|
||||
`FSValidator` for `fs.FS` sources. Directory-backed validation loads the named
|
||||
schema path; it does not search directories by basename. `fs.FS` schema paths
|
||||
are cleaned and checked against their configured root, while a single-file
|
||||
source matches its file base name.
|
||||
|
||||
The runner requests a schema document before generation when it needs
|
||||
structured output. JSON and schema mismatches in generated content are
|
||||
validation results; source access, decoding, registration, and compilation
|
||||
failures are operational errors.
|
||||
|
||||
## Artifacts
|
||||
|
||||
`internal/artifact` owns the framework's ordinary inline and unrestricted file
|
||||
reader. The public engine uses it by default and permits consumers to replace it
|
||||
for every input through the public `ArtifactReader` extension. The
|
||||
HTTP adapter owns its restricted reader for HTTP containment: `serve` injects
|
||||
that reader into the public engine with `WithArtifactReader`.
|
||||
|
||||
The rooted reader cleans paths and applies lexical containment without resolving
|
||||
symlinks. It checks relative references against the configured root and accepts
|
||||
absolute references only when they remain inside that lexical root. The OS still
|
||||
follows symlinks after that check. The public containment outcome is documented
|
||||
by the [HTTP API reference](../api.md); deployment permissions belong in
|
||||
[operations](../operations.md).
|
||||
|
||||
## Failure Boundaries
|
||||
|
||||
Source packages report repository, decoding, duplicate, validation, and read
|
||||
failures to their callers. They do not select public status codes or response
|
||||
schemas. The runner categorizes source failures and the public engine preserves
|
||||
the corresponding public error identities; adapters map those identities to
|
||||
their own external contract.
|
||||
|
||||
Source reads use current filesystem or `fs.FS` content for each request. These
|
||||
packages create no manifests, checkpoints, or durable run state.
|
||||
|
||||
## Verification And Change Recipe
|
||||
|
||||
Inspect:
|
||||
|
||||
- `internal/promptdef/repository_test.go`
|
||||
- `internal/profile/repository_test.go`
|
||||
- `internal/profile/builtin/repository_test.go`
|
||||
- `internal/artifact/reader_test.go`
|
||||
- `internal/adapter/http/artifact_reader_test.go`
|
||||
- `internal/validate/standard_validator_test.go`
|
||||
- `engine_test.go`
|
||||
|
||||
When updating prompt, profile, schema, or built-in assets:
|
||||
|
||||
1. keep assets valid for the strict loader and the relevant source boundary;
|
||||
2. update the [configuration reference](../config.md) when a file-format,
|
||||
catalog, or default changes;
|
||||
3. run focused source and integration tests, including the built-in repository
|
||||
test when embedded assets change; and
|
||||
4. update this document when discovery, precedence, containment, or failure
|
||||
mechanics change.
|
||||
|
||||
The [testing policy](../policy/testing.md) owns global test sufficiency.
|
||||
157
docs/operations.md
Normal file
157
docs/operations.md
Normal file
@@ -0,0 +1,157 @@
|
||||
# Operations Guide
|
||||
|
||||
## Scope And References
|
||||
|
||||
This runbook covers deployment, normal operation, capacity planning, and safe
|
||||
recovery for Scriptorium. It does not redefine invocation syntax, configuration
|
||||
fields, or HTTP wire behavior.
|
||||
|
||||
- [CLI reference](cli.md): commands, output destinations, and exit codes.
|
||||
- [Configuration reference](config.md): configuration, prompt/profile/schema
|
||||
formats, defaults, and credentials.
|
||||
- [HTTP API reference](api.md): route, request/response schema, status codes,
|
||||
limits, and HTTP artifact access.
|
||||
- [Consumer integration overview](consumers/api.md): caller responsibilities.
|
||||
|
||||
## Operational Model And State
|
||||
|
||||
Scriptorium handles one prompt request for each CLI invocation or HTTP request.
|
||||
It has no durable run store, archive, checkpoint, cache, or resume mechanism.
|
||||
A failed or interrupted request is recovered by correcting its inputs,
|
||||
configuration, or environment and submitting a new request.
|
||||
|
||||
Generated artifacts, rendered prompts, model output, and run metadata are
|
||||
caller-owned data. Retention, encryption, backup, and deletion are deployment
|
||||
responsibilities.
|
||||
|
||||
## Deploy The Filesystem And Process
|
||||
|
||||
Provide the process with readable prompt, profile, and schema sources. Keep
|
||||
prompt templates adjacent to the prompt definitions that reference them. For an
|
||||
HTTP deployment that accepts file artifacts, use a dedicated, narrow artifact
|
||||
directory rather than a general-purpose or sensitive filesystem tree.
|
||||
|
||||
Run Scriptorium under an identity that can:
|
||||
|
||||
- read only the prompt, profile, schema, and allowed input-artifact paths it
|
||||
needs;
|
||||
- read the required credential environment variables without writing them to
|
||||
files or logs; and
|
||||
- write only caller-selected output locations when CLI output files are used.
|
||||
|
||||
Do not make the HTTP artifact directory writable by untrusted users. The HTTP
|
||||
artifact containment behavior is lexical and the operating system follows
|
||||
symlinks; account for that when choosing ownership and mount boundaries. See
|
||||
the [HTTP API reference](api.md) for the externally observable behavior.
|
||||
|
||||
## Supply Credentials And Protect Runtime Data
|
||||
|
||||
Set secret values in the process environment and configure only their
|
||||
environment-variable names. Do not put raw keys in configuration, prompt or
|
||||
profile files, process arguments, HTTP payloads, captured command lines, or
|
||||
debug dumps.
|
||||
|
||||
Treat stdout, stderr, prepared-run output, generated artifacts, and HTTP
|
||||
responses as potentially sensitive. Send service logs to a controlled collector
|
||||
and apply the same retention and access rules as for model input and output.
|
||||
|
||||
## Run A Normal Workflow
|
||||
|
||||
Before changing production inputs, profiles, or schemas:
|
||||
|
||||
1. confirm the deployed configuration selects the intended sources and model
|
||||
credentials;
|
||||
2. use [`render`](cli.md) with the same request inputs and variables to confirm
|
||||
preparation without a model call;
|
||||
3. use [`run`](cli.md) for generation; and
|
||||
4. retain or discard validation-failed output according to the caller's
|
||||
policy.
|
||||
|
||||
The [maintained render script](../examples/render-markdown-summary.sh) is a
|
||||
copyable preflight example. The CLI reference owns its complete invocation and
|
||||
exit semantics.
|
||||
|
||||
## Expose The HTTP Service
|
||||
|
||||
The HTTP service has no built-in authentication or authorization. Place it on a
|
||||
trusted network or behind an authenticated reverse proxy, API gateway, or
|
||||
equivalent access control. Restrict who can reach it and who can read the
|
||||
artifact root.
|
||||
|
||||
Use a service manager or supervisor appropriate to the deployment to manage
|
||||
process lifetime, restart policy, log capture, and environment injection. The
|
||||
[HTTP API reference](api.md) owns client request shapes, status behavior, and
|
||||
artifact-access outcomes.
|
||||
|
||||
## Plan Capacity And Limits
|
||||
|
||||
Capacity is primarily determined by concurrent model calls, input and output
|
||||
sizes, schema complexity, provider latency, and network behavior. Size limits
|
||||
protect request bodies, HTTP file artifacts, and encoded responses; configure
|
||||
them through the [configuration reference](config.md) and rely on the
|
||||
[HTTP API reference](api.md) for their response effects.
|
||||
|
||||
Before increasing a limit:
|
||||
|
||||
1. measure representative input, generated-output, and optional raw-output
|
||||
sizes;
|
||||
2. confirm memory, network, and upstream-provider capacity;
|
||||
3. retain an upstream request-size and authentication boundary; and
|
||||
4. test the intended workload in a non-production environment.
|
||||
|
||||
For large local inputs, prefer a controlled file-artifact directory over
|
||||
placing arbitrary paths on the service host. Avoid disabling a limit unless an
|
||||
equivalent trusted control exists elsewhere.
|
||||
|
||||
## Diagnose And Recover
|
||||
|
||||
### Preparation Or Configuration Failure
|
||||
|
||||
Capture the CLI diagnostic or HTTP error response, then verify the selected
|
||||
configuration, prompt ID, profile selection, source readability, and input
|
||||
mapping. Use `render` with the same request when it is unclear whether failure
|
||||
occurs before model execution. Consult the [CLI reference](cli.md), the
|
||||
[configuration reference](config.md), and the [HTTP API reference](api.md) for
|
||||
the exact interface contract.
|
||||
|
||||
### Credential Or Provider Failure
|
||||
|
||||
Confirm that the process environment contains the configured credential name
|
||||
without printing the secret. Check endpoint reachability and provider health
|
||||
from the process network. If preparation succeeds but generation fails, inspect
|
||||
the selected model settings in prepared output and the service's controlled
|
||||
logs. Correct the deployment or provider issue, then submit a new request.
|
||||
|
||||
### Artifact Or Permission Failure
|
||||
|
||||
Verify that the process can read the intended local input. For HTTP file
|
||||
artifacts, verify the deployment's artifact root, ownership, path layout, and
|
||||
file size. Do not widen filesystem permissions or the allowed root merely to
|
||||
make an arbitrary path work; move or copy the required artifact into the
|
||||
controlled location instead.
|
||||
|
||||
### Validation Failure
|
||||
|
||||
A generated-content validation failure is distinct from a runtime failure.
|
||||
CLI `run` reports the validation result and error count in its success summary;
|
||||
it does not print the individual validation messages. For HTTP, inspect the
|
||||
validation object in the response according to the [HTTP API reference](api.md).
|
||||
|
||||
Use rendered input and generated output to determine whether prompt instructions,
|
||||
the selected model, or the schema needs correction. If schema loading or
|
||||
compilation itself fails, correct the source deployment or schema document
|
||||
before rerunning.
|
||||
|
||||
### HTTP Limit Or Request Failure
|
||||
|
||||
Compare the request, artifact, or expected response size with the deployed
|
||||
configuration, and validate the request against the [HTTP API reference](api.md).
|
||||
Reduce the payload, use an appropriate controlled artifact source, omit
|
||||
unneeded raw output, or adjust the deployment limit after capacity review.
|
||||
|
||||
## Cleanup And Reruns
|
||||
|
||||
Because no run state is retained, cleanup concerns caller-owned output files,
|
||||
logs, and artifacts only. Remove or rotate them using the deployment's normal
|
||||
retention policy. After a correction, rerun the request from the beginning;
|
||||
there is no safe resume point.
|
||||
83
docs/policy/architecture.md
Normal file
83
docs/policy/architecture.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# Architecture
|
||||
|
||||
This document is the development architecture policy for Scriptorium.
|
||||
|
||||
It is for developers and LLM coding agents. User-facing behavior belongs in `README.md` and the docs under `docs/` that target operators/users.
|
||||
|
||||
## System Shape
|
||||
|
||||
Scriptorium is a narrow prompt-execution application with three executable
|
||||
entry paths: CLI `run`, CLI `render`, and the HTTP service started by `serve`.
|
||||
It also provides a public Go package for in-process use. Executable adapters
|
||||
consume framework behavior through that public facade; the facade continues to
|
||||
compose the framework implementation inside this single repository. Its current
|
||||
component inventory is maintained in the [internal overview](../internal/overview.md).
|
||||
|
||||
Domain behavior is centralized in `internal/usecase` and `internal/domain`.
|
||||
|
||||
## Core Principles
|
||||
|
||||
- Keep orchestration narrow: Scriptorium executes one prompt request; it is not a multi-step workflow engine.
|
||||
- Keep adapter logic thin: adapters map external shapes to public engine
|
||||
requests/results and should not hold framework decisions.
|
||||
- Keep boundaries explicit: repositories/loaders/renderers/validators/LLM client stay behind package interfaces.
|
||||
- Keep external decoding strict: configuration, prompt, and profile YAML and
|
||||
HTTP JSON should reject unknown fields.
|
||||
- Keep secrets out of payloads: raw API key values must not be accepted or emitted.
|
||||
|
||||
## Dependency Direction
|
||||
|
||||
- Adapters translate external shapes and IO concerns; they do not make
|
||||
use-case decisions.
|
||||
- Executable adapters and prepared-run formatting use the public facade for
|
||||
framework behavior rather than importing framework implementation packages
|
||||
directly.
|
||||
- Use-case and domain code depend on explicit repository, renderer, validator,
|
||||
and LLM interfaces rather than adapter implementations.
|
||||
- Source, rendering, validation, and LLM implementations remain behind their
|
||||
package boundaries.
|
||||
- Dependency-specific types must not leak across unrelated package boundaries.
|
||||
- Prefer the standard library; add an external dependency only when it
|
||||
materially reduces risk or complexity.
|
||||
|
||||
## State And Persistence Policy
|
||||
|
||||
Scriptorium has no durable run-state store.
|
||||
|
||||
- No built-in resume/checkpoint/archive behavior.
|
||||
- Recovery model is rerun after correcting inputs/config/environment.
|
||||
|
||||
## Contract Ownership
|
||||
|
||||
The [CLI](../cli.md), [configuration](../config.md), [HTTP API](../api.md),
|
||||
[public Go package](../consumers/pkg-scriptorium.md), and
|
||||
[integration](../integrations/) documents own their respective external
|
||||
contracts. This policy keeps only the architectural boundaries that govern
|
||||
their implementation.
|
||||
|
||||
## Error Handling And Logging
|
||||
|
||||
- Wrap errors with domain/operation context.
|
||||
- Map public error identities to adapter-appropriate statuses/codes without
|
||||
leaking sensitive internals.
|
||||
- Never emit raw secret values.
|
||||
|
||||
## Testing And Documentation
|
||||
|
||||
Testing philosophy and change-validation expectations are defined by the
|
||||
[testing policy](testing.md). Documentation ownership and maintenance rules are
|
||||
defined by the [documentation policy](documentation.md).
|
||||
|
||||
## Architectural Invariants
|
||||
|
||||
- `Runner.Run` reuses `Runner.Prepare` flow.
|
||||
- Raw API key values must not be accepted through external configuration or
|
||||
request payloads, and resolved secret values must not be emitted.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not move orchestration responsibilities from external callers into Scriptorium.
|
||||
- Do not add adapter-specific business logic in `internal/adapter/*` packages.
|
||||
- Do not bypass repository/renderer/validator/LLM boundaries by introducing cross-package coupling.
|
||||
|
||||
Work that is not implemented belongs in `docs/roadmap/`.
|
||||
163
docs/policy/documentation.md
Normal file
163
docs/policy/documentation.md
Normal file
@@ -0,0 +1,163 @@
|
||||
# Documentation Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
This policy assigns each documentation topic to one canonical owner. Its goal is
|
||||
to keep this repository's documentation accurate, concise, discoverable, and
|
||||
resistant to drift for users, operators, developers, integrators, and LLM
|
||||
coding agents.
|
||||
|
||||
## Core Rules
|
||||
|
||||
### One Canonical Owner
|
||||
|
||||
Each authoritative fact belongs in one document. A non-owning document may give
|
||||
a short, stable summary for orientation, but it must link to the canonical owner
|
||||
instead of repeating volatile details.
|
||||
|
||||
Volatile details include commands, flags, configuration fields and defaults,
|
||||
module keys, schemas, file names, paths, status codes, retry behavior, and
|
||||
runtime guarantees. If readers could reasonably treat a statement as a
|
||||
contract, maintain it only in the owning document.
|
||||
|
||||
Minimal tested usage examples are allowed outside the owning contract when this
|
||||
policy assigns them an orientation or instructional purpose. They must link to
|
||||
the canonical contract and must not redefine complete syntax, defaults, or
|
||||
semantics.
|
||||
|
||||
### Current And Future Behavior
|
||||
|
||||
Outside `docs/roadmap/`, documentation describes implemented behavior only.
|
||||
Partial features may be described only to their implemented boundary.
|
||||
|
||||
ADRs are the narrow exception: an ADR may record an accepted architectural
|
||||
decision before implementation, but acceptance must not be presented as proof
|
||||
that the behavior exists. The roadmap owns implementation status and sequencing
|
||||
until the decision is implemented. Current architecture, user, operator,
|
||||
integration, and internal documentation are updated when the behavior lands.
|
||||
|
||||
### Audience And Detail
|
||||
|
||||
Write for the document's stated audience and include only the detail needed for
|
||||
its owned topic. User and operator docs should not expose implementation detail.
|
||||
Developer docs should link to user-facing and external contracts rather than
|
||||
restate them.
|
||||
|
||||
### Examples
|
||||
|
||||
Complete copyable files belong in `examples/`. Documentation may use the
|
||||
smallest illustrative snippet needed to explain its owned topic, but should link
|
||||
to maintained examples instead of embedding a second complete copy.
|
||||
|
||||
Examples must be valid, secret-free, and tested where practical. Commands and
|
||||
configuration used in documentation should match the application.
|
||||
|
||||
### Security And Privacy
|
||||
|
||||
Documentation and examples must not contain real credentials, private keys,
|
||||
private environment dumps, sensitive source material, or private infrastructure
|
||||
details unless intentionally public. Document secret-handling mechanisms, not
|
||||
secret values.
|
||||
|
||||
## Canonical Ownership
|
||||
|
||||
| Topic | Canonical owner | Owned content | Content owned elsewhere |
|
||||
| --- | --- | --- | --- |
|
||||
| Product orientation and minimal end-to-end quickstart | `README.md` | What this project is, why it is useful, one shortest successful invocation, and links onward. | Complete command reference, configuration reference, operational procedures, implementation detail. |
|
||||
| Contributor entry point | `docs/development.md` | Task-oriented reading guide, minimal contributor orientation, baseline validation commands, and links to canonical docs. | Package inventory, architecture rules, subsystem behavior, and detailed change recipes, which belong in the relevant internal component document. |
|
||||
| Current application architecture | `docs/policy/architecture.md` | System shape, normative ownership, dependency direction, architectural boundaries, invariants, safety properties, and non-goals. | Concrete package inventory, implementation mechanics, contributor procedures, decision history, future work. |
|
||||
| Documentation organization | `docs/policy/documentation.md` | Documentation ownership, audience boundaries, maintenance rules, and ADR/document lifecycle. | Application architecture or product behavior. |
|
||||
| Testing policy | `docs/policy/testing.md` | Test philosophy, risk-based sufficiency, test boundaries, doubles, coverage guidance, regression-test policy, and criteria for adding, rewriting, or deleting tests. | Subsystem behavior, application contracts, subsystem-specific test inventories, and implementation plans. |
|
||||
| CLI contract | `docs/cli.md` | Commands, arguments, flags, invocation semantics, and exit codes. | End-to-end operating procedures, configuration field definitions, runtime filesystem layout, module implementation details. |
|
||||
| Configuration contract | `docs/config.md` | Discovery and precedence, file schema, fields, defaults, environment overrides, validation rules, and user-selectable module or validator keys. | Complete example files, CLI syntax, runtime state lifecycle, module implementation details. |
|
||||
| Operations | `docs/operations.md` | Runtime workflows, physical filesystem and state layout, output, cache, and debug handling, resume, cleanup, permissions, recovery, and operational limits. | CLI flag syntax, configuration field definitions, logical output schemas, implementation mechanics. |
|
||||
| Public HTTP contract | `docs/api.md` | Routes, authentication, media types, request and response schemas, status codes, pagination, caching, idempotency, rate limits, and HTTP retry semantics. | Client walkthroughs, upstream or downstream integration internals, implementation detail. |
|
||||
| Consumer guidance | `docs/consumers/` | Task-oriented use of the public interface, minimal client examples, and consumer responsibilities. | HTTP wire semantics, external protocol contracts, internal implementation detail. |
|
||||
| External and durable integration contracts | `docs/integrations/` | External file formats and protocols, upstream and downstream contracts, logical output bundle paths and schemas, media types, and compatibility behavior. | Physical runtime placement and lifecycle, internal transformations, CLI syntax, configuration defaults. |
|
||||
| Implemented component inventory | `docs/internal/overview.md` | Current packages and components, their implemented responsibilities, and links to focused internal docs. | Normative architecture, contributor reading policy, external contracts. |
|
||||
| Internal component behavior | Other files under `docs/internal/` | Implementation flow, internal collaborators and state transitions, package-local guarantees and failures, and relevant tests. | Global architecture invariants, configuration definitions and defaults, external schemas, operator procedures. |
|
||||
| Architectural decision history | `docs/adr/` | Significant decisions, context, alternatives, rationale, consequences, and supersession history. | Current behavior reference, implementation status, task sequencing. |
|
||||
| Future work and implementation status | `docs/roadmap/` | Proposed, accepted, deferred, or rejected work; implementation status; sequencing; and task breakdowns. | Implemented behavior reference and architectural decision rationale. |
|
||||
| Complete copyable artifacts | `examples/` | Maintained configuration, inputs, and other files intended to be copied or run. | Field-by-field reference, command reference, prose explanation. |
|
||||
|
||||
Documents that do not exist are required only when the corresponding interface
|
||||
or responsibility exists. Do not create placeholder API, consumer, integration,
|
||||
or operations documents for behavior the application does not have.
|
||||
|
||||
## Boundary Rules
|
||||
|
||||
### Orientation
|
||||
|
||||
The README owns product orientation. The developer guide routes contributors.
|
||||
Architecture owns normative structure. Internal overview owns the current
|
||||
concrete component map. These documents may link to one another but should not
|
||||
maintain parallel package or behavior descriptions.
|
||||
|
||||
### Commands, Configuration, And Operations
|
||||
|
||||
CLI documentation answers how to invoke the application. Configuration
|
||||
documentation answers what settings mean. Operations answers what happens to
|
||||
runtime state and how to operate or recover the application. When a workflow
|
||||
crosses these topics, choose the document that owns the task and link to the
|
||||
other contracts.
|
||||
|
||||
### Contracts And Implementation
|
||||
|
||||
Integration and API documents define externally observable shapes and
|
||||
semantics. Internal documents explain how this project implements or consumes
|
||||
those contracts. Internal docs may name a field, file, or protocol to identify
|
||||
a dependency, but must link to its canonical contract for the definition.
|
||||
|
||||
### Security Topics
|
||||
|
||||
This policy owns what documentation and examples may contain. Architecture owns
|
||||
application security invariants. Configuration owns credential-supply
|
||||
mechanisms. Operations owns permissions and handling of sensitive runtime
|
||||
artifacts. Internal docs own implementation mechanisms only.
|
||||
|
||||
## Architecture Decision Records
|
||||
|
||||
Use sequentially numbered ADR filenames such as
|
||||
`0001-record-architecture-decisions.md`. Follow the lightweight Nygard format:
|
||||
|
||||
1. title;
|
||||
2. status;
|
||||
3. date;
|
||||
4. context;
|
||||
5. decision;
|
||||
6. alternatives considered;
|
||||
7. consequences.
|
||||
|
||||
Use one of these statuses:
|
||||
|
||||
- **Proposed:** the decision is under consideration and may change;
|
||||
- **Accepted:** the decision is approved, whether or not implementation is
|
||||
complete;
|
||||
- **Rejected:** the proposed decision was considered and not adopted;
|
||||
- **Superseded:** a later ADR replaces the accepted decision.
|
||||
|
||||
A proposed ADR transitions to accepted or rejected. An accepted ADR transitions
|
||||
to superseded only when a later accepted ADR replaces it. An ADR may be created
|
||||
as accepted when the decision has already been made.
|
||||
|
||||
Treat the decision content of an accepted ADR as immutable. Its status and
|
||||
supersession metadata may be updated, but a changed decision requires a new ADR.
|
||||
A superseded ADR must link to its replacement, and the replacement must link
|
||||
back to the superseded ADR. Rejected architectural alternatives belong in the
|
||||
ADR; rejected product ideas belong in the roadmap.
|
||||
|
||||
## Maintenance
|
||||
|
||||
When behavior changes, update its canonical owner in the same change. If
|
||||
ownership moves, remove the old definition and replace it with a link where
|
||||
navigation remains useful.
|
||||
|
||||
Before completing documentation work:
|
||||
|
||||
- verify affected behavior and examples;
|
||||
- check commands, flags, fields, defaults, schemas, and paths against their
|
||||
implementation;
|
||||
- keep unimplemented behavior in the roadmap, subject to the ADR exception;
|
||||
- remove stale references and validate links;
|
||||
- confirm that non-owning documents summarize and link rather than redefine;
|
||||
- confirm that no secrets or sensitive private data were added.
|
||||
296
docs/policy/testing.md
Normal file
296
docs/policy/testing.md
Normal file
@@ -0,0 +1,296 @@
|
||||
# Testing Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
Our tests exist to make **incorrect changes expensive and correct changes cheap**.
|
||||
|
||||
We do not optimize for test count, line coverage, exhaustive isolation, or the fewest possible tests. We optimize for sufficient confidence in important behavior while imposing as little unnecessary friction as possible on future development.
|
||||
|
||||
## Every test has a cost
|
||||
|
||||
Testing is not an unqualified good. Every test imposes both an immediate cost and a continuing lifetime cost.
|
||||
|
||||
A test must be:
|
||||
|
||||
- written and reviewed;
|
||||
- understood by future maintainers and coding agents;
|
||||
- executed in local and CI workflows;
|
||||
- diagnosed when it fails;
|
||||
- updated when legitimate behavior changes;
|
||||
- maintained as fixtures, APIs, and dependencies evolve; and
|
||||
- removed or rewritten when it becomes redundant, brittle, misleading, or obsolete.
|
||||
|
||||
Tests also create cognitive and architectural friction. They can constrain refactoring, duplicate policy, slow feedback loops, add noise to failures, and cause harmless implementation changes to require unrelated edits across the suite.
|
||||
|
||||
A test is warranted only when the confidence it provides justifies these costs.
|
||||
|
||||
Apply this cost-benefit analysis at two levels:
|
||||
|
||||
1. **Per test:** What realistic defect does this test detect, how consequential would that defect be, and is that protection worth the test's lifetime cost?
|
||||
2. **Across the suite:** Does this collection provide materially more confidence than a smaller, simpler suite would?
|
||||
|
||||
The preferred test suite is a **lean suite that provides sufficient confidence in the risks that matter, without redundant or low-value tests**. We seek sufficient confidence with the least unnecessary testing friction, not the fewest possible tests.
|
||||
|
||||
Some friction is intentional. Tests should make dangerous changes—such as breaking compatibility, corrupting data, violating security boundaries, or reintroducing subtle bugs—require deliberate review. They should not make ordinary internal changes needlessly expensive.
|
||||
|
||||
The cost of a test is not a reason to omit testing by default. Do not cite maintenance cost abstractly. When omitting a plausible test, be able to state why the protected failure is low-risk, already covered, obvious, reversible, or cheaper to detect elsewhere. For consequential, subtle, or difficult-to-observe behavior, the presumption should favor testing.
|
||||
|
||||
## Default testing style
|
||||
|
||||
Use a **classical/Detroit-style** approach:
|
||||
|
||||
- Test observable behavior, resulting state, contracts, and invariants.
|
||||
- Use real internal collaborators when they are fast and deterministic.
|
||||
- Use fakes, stubs, or mocks primarily at expensive, nondeterministic, destructive, or external boundaries.
|
||||
- Prefer package-level behavioral tests over tests coupled to private helpers or internal call sequences.
|
||||
- Treat exact collaborator interactions as testable behavior only when the interaction itself is a requirement.
|
||||
|
||||
Examples of appropriate seams include clocks, randomness, subprocesses, remote APIs, object storage, email, and paid LLM calls.
|
||||
|
||||
## Test execution requirements
|
||||
|
||||
Tests in the default suite must be deterministic, offline, and independent of real credentials. They must not invoke paid APIs or depend on mutable external services. Tests that require live infrastructure must be explicitly opt-in and clearly separated from the default suite.
|
||||
|
||||
Control clocks, randomness, environment variables, and other process-global or machine-specific state when they affect behavior. Tests should be safe to run repeatedly and alongside other tests without depending on execution order or state left by an earlier test.
|
||||
|
||||
## What deserves tests
|
||||
|
||||
Prioritize tests for:
|
||||
|
||||
1. Public and package-level contracts.
|
||||
2. Domain rules and important invariants.
|
||||
3. Boundary conditions and malformed input.
|
||||
4. Failure handling, cancellation, retries, recovery, and partial success.
|
||||
5. Serialization, schemas, compatibility, and round trips.
|
||||
6. Previously observed or plausible regressions.
|
||||
7. Representative integration and end-to-end workflows.
|
||||
|
||||
A package-level contract is behavior relied upon by another package or major collaborator, not every observable detail of a package implementation.
|
||||
|
||||
For behavior involving **data integrity, destructive operations, compatibility, security, concurrency, idempotency, or recovery**, presume that durable tests are required unless the behavior is already credibly protected at another layer.
|
||||
|
||||
Do not add tests merely because a function, branch, or line exists. Do not add a test when the same meaningful risk is already adequately protected elsewhere.
|
||||
|
||||
## Choose the right test boundary
|
||||
|
||||
Test through the narrowest stable boundary that expresses the behavior clearly.
|
||||
|
||||
This is often the package API, but it may instead be:
|
||||
|
||||
- a smaller pure function when dense domain logic is most clearly isolated there;
|
||||
- a package-level operation when several internal collaborators jointly produce the behavior; or
|
||||
- a larger integration boundary when correctness emerges from interaction with a real dependency.
|
||||
|
||||
Do not force all behavior through oversized end-to-end tests. Do not test every private helper merely because it exists. Choose the boundary that gives durable confidence with the least incidental coupling.
|
||||
|
||||
## Test behavior, not implementation
|
||||
|
||||
A test should protect a decision, contract, or invariant—not memorialize the current implementation.
|
||||
|
||||
Before adding or retaining a test, ask:
|
||||
|
||||
> What realistic defect would this test catch?
|
||||
|
||||
A test is suspect when its main purpose is to detect that someone:
|
||||
|
||||
- changed an internal constant;
|
||||
- renamed or split a private helper;
|
||||
- reordered equivalent internal operations;
|
||||
- changed incidental formatting;
|
||||
- replaced one correct algorithm with another; or
|
||||
- refactored internal object structure without changing behavior.
|
||||
|
||||
Refactoring should normally require no test edits unless the refactored structure is itself part of the contract.
|
||||
|
||||
A test can be factually correct and still have negative value. Accurately describing current behavior is not enough; the protected behavior must be important enough to justify the future friction.
|
||||
|
||||
## Expected effects of different changes
|
||||
|
||||
Use the following expectations when evaluating test failures and test maintenance:
|
||||
|
||||
| Change | Expected effect on tests |
|
||||
|---|---|
|
||||
| Internal refactor that preserves behavior | Existing tests should normally remain unchanged and continue to pass. |
|
||||
| Change to an internal default with no contractual significance | Behavioral tests should normally remain unchanged; tests should derive expectations from configuration or relationships rather than duplicate the old value. |
|
||||
| Intentional change to public behavior, policy, schema, or compatibility guarantees | The relevant tests should be reviewed and changed deliberately. |
|
||||
| Accidental violation of a contract or invariant | Tests should fail; fix the production code rather than rewriting the tests to accept the defect. |
|
||||
|
||||
A test failing is not the same as a test needing to be edited. Many tests may correctly fail because of one production defect. The maintenance smell is a correct internal change that requires unrelated expectation updates throughout the suite.
|
||||
|
||||
## Separate mechanism from policy
|
||||
|
||||
Configurable thresholds and defaults must not be duplicated throughout the test suite.
|
||||
|
||||
For example, do not encode an internal concurrency limit indirectly:
|
||||
|
||||
```go
|
||||
// Production policy:
|
||||
const maxConcurrency = 4
|
||||
|
||||
// Brittle test:
|
||||
err := startProcesses(5)
|
||||
require.Error(t, err)
|
||||
```
|
||||
|
||||
Instead, test the mechanism relationally:
|
||||
|
||||
```go
|
||||
const limit = 2
|
||||
runner := NewRunner(limit)
|
||||
|
||||
require.NoError(t, runner.Start(limit))
|
||||
require.ErrorIs(t, runner.Start(limit+1), ErrTooMuchConcurrency)
|
||||
```
|
||||
|
||||
The test should prove:
|
||||
|
||||
- the configured limit is accepted; and
|
||||
- one beyond the configured limit is rejected.
|
||||
|
||||
The production default should be tested exactly only when its literal value is itself a public, operational, safety, protocol, or compatibility requirement.
|
||||
|
||||
Apply the same rule to limits, timeouts, capacities, retry counts, and ranges: test relationships and behavior, not duplicated literals.
|
||||
|
||||
For concurrency limits, test both kinds of behavior when relevant:
|
||||
|
||||
1. **Configuration enforcement:** invalid or excessive requested values are handled correctly.
|
||||
2. **Runtime enforcement:** observed peak concurrency never exceeds the configured limit.
|
||||
|
||||
Use a test-controlled limit and measure the behavior relative to that limit. Do not merely assert today's default value.
|
||||
|
||||
## Avoid semantic duplication across layers
|
||||
|
||||
Each behavior should have a clear test owner.
|
||||
|
||||
- Parser tests own parsing cases.
|
||||
- Validator tests own validation rules.
|
||||
- Domain tests own transformations and invariants.
|
||||
- Adapter tests own external integration behavior.
|
||||
- Orchestrator tests own coordination and failure propagation.
|
||||
- CLI tests own argument and configuration mapping.
|
||||
- End-to-end tests prove that representative assembled workflows work.
|
||||
|
||||
Higher-level tests should not repeat every lower-level case. A single intentional policy change should not require unrelated edits across many test files.
|
||||
|
||||
Tests that are individually reasonable may still be collectively redundant. Evaluate the marginal value of each additional test in light of the protection already provided by the rest of the suite.
|
||||
|
||||
## Use test doubles deliberately
|
||||
|
||||
Choose the least elaborate test double that provides the required control or observation.
|
||||
|
||||
As a default:
|
||||
|
||||
1. Prefer real collaborators when they are fast and deterministic.
|
||||
2. Use small in-memory fakes when realistic stateful behavior is helpful.
|
||||
3. Use stubs when a dependency only needs to provide controlled responses.
|
||||
4. Use mocks when the interaction itself is contractual.
|
||||
|
||||
Mocks are appropriate when the contract includes facts such as:
|
||||
|
||||
- a notification is sent exactly once;
|
||||
- a transaction is committed only after successful writes;
|
||||
- cancellation reaches a subprocess;
|
||||
- an expensive API is called no more than once; or
|
||||
- a security audit event is emitted.
|
||||
|
||||
Do not use mocks merely to isolate every object or reproduce the implementation's call graph.
|
||||
|
||||
## Go-specific guidance
|
||||
|
||||
Use:
|
||||
|
||||
- table-driven tests for meaningful behavioral categories and boundaries;
|
||||
- `t.TempDir()` for real filesystem behavior;
|
||||
- `httptest.Server` for realistic HTTP interactions;
|
||||
- fuzz tests for parsers, normalization, path handling, and broad input spaces;
|
||||
- golden files only when the complete output is intentionally stable;
|
||||
- integration tests where correctness depends on component interaction; and
|
||||
- a small number of representative end-to-end tests.
|
||||
|
||||
Avoid exact error-string assertions unless the wording is itself contractual. Prefer `errors.Is`, `errors.As`, typed errors, or structured error fields.
|
||||
|
||||
At CLI boundaries, prefer exit classifications, structured output, and the smallest stable semantic fragment needed to identify the error. Do not snapshot complete diagnostic wording unless it is contractual.
|
||||
|
||||
Golden-file updates must require an explicit local flag. CI must not update golden files automatically, and reviewers must inspect the semantic diff before accepting an update.
|
||||
|
||||
Keep tests readable and direct. Test helpers and fixture frameworks must earn their own maintenance cost; do not build elaborate test infrastructure for small or isolated needs.
|
||||
|
||||
## Coverage
|
||||
|
||||
Coverage is a diagnostic, not a target.
|
||||
|
||||
Use it to find untested critical branches and unexpectedly weak packages. Do not write low-value tests solely to increase a percentage, and do not infer test quality from coverage alone.
|
||||
|
||||
Pure domain logic will often warrant higher coverage than CLI wiring or external adapters. Uneven coverage is acceptable when it reflects risk.
|
||||
|
||||
Increasing coverage is valuable only when the newly covered behavior protects a meaningful risk at an acceptable cost.
|
||||
|
||||
## Regression tests
|
||||
|
||||
A bug fix should normally include a regression test that fails before the fix and passes afterward.
|
||||
|
||||
Retain the test when the defect could realistically recur and its consequences justify the ongoing cost. Prefer the narrowest durable test of the violated contract or invariant; do not preserve accidental implementation details from the original bug.
|
||||
|
||||
Not every historical bug requires a permanent test. If the underlying design has made recurrence impossible, the test has become redundant, or a stronger invariant test now subsumes it, remove or consolidate it.
|
||||
|
||||
## Deleting or rewriting tests
|
||||
|
||||
Tests are maintained code, not permanent historical artifacts.
|
||||
|
||||
Delete or rewrite a test when its maintenance cost exceeds the confidence it provides.
|
||||
|
||||
Strong candidates include tests that:
|
||||
|
||||
- require updates after harmless internal changes;
|
||||
- directly assert private constants without protecting a real contract;
|
||||
- duplicate the same policy across several layers;
|
||||
- verify mock choreography rather than outcomes;
|
||||
- snapshot large amounts of incidental output;
|
||||
- test trivial private helpers already exercised through stable package behavior;
|
||||
- protect risks already covered more effectively elsewhere;
|
||||
- are flaky, misleading, obsolete, or disproportionately expensive to diagnose; or
|
||||
- no longer correspond to a plausible failure mode.
|
||||
|
||||
Several brittle tests may encode one genuine requirement. Replace them with one durable behavior-level or invariant test rather than preserving all of them.
|
||||
|
||||
Deleting a low-value test can improve the quality of the suite by reducing noise, maintenance burden, and friction around legitimate change.
|
||||
|
||||
## Reviewing a proposed test
|
||||
|
||||
Use the following questions when the value, boundary, or durability of a proposed test is not self-evident. Significant test additions should be reviewable against them, but written answers are not required for every routine test.
|
||||
|
||||
1. What realistic defect would it catch?
|
||||
2. How likely is that defect?
|
||||
3. How consequential would it be?
|
||||
4. Is the behavior already protected elsewhere?
|
||||
5. At which layer should this behavior be owned?
|
||||
6. Does the test assert a durable contract or an incidental implementation detail?
|
||||
7. Could the implementation be refactored without changing the behavior and without editing this test?
|
||||
8. What should cause this test to fail?
|
||||
9. What legitimate changes should not cause this test to fail?
|
||||
10. What ongoing maintenance, execution, and diagnostic cost will the test impose?
|
||||
11. Is there a smaller or more direct test that protects the same risk?
|
||||
|
||||
Do not add the test when its expected lifetime cost exceeds its expected protective value.
|
||||
|
||||
When deciding not to test plausible behavior, record or be able to explain why the risk is low, already protected, obvious, reversible, or cheaper to detect elsewhere.
|
||||
|
||||
## Definition of sufficient
|
||||
|
||||
A test suite is sufficient when:
|
||||
|
||||
- important contracts and invariants are protected;
|
||||
- meaningful boundaries and failure modes are exercised;
|
||||
- realistic and consequential regressions are credibly protected against silent recurrence;
|
||||
- behavior involving data integrity, destructive operations, compatibility, security, concurrency, idempotency, and recovery is credibly protected;
|
||||
- important external boundaries have realistic integration coverage;
|
||||
- representative complete workflows are tested;
|
||||
- failures provide useful signal rather than redundant noise;
|
||||
- legitimate internal changes usually do not require test edits; and
|
||||
- additional tests would mostly repeat existing protection or preserve inconsequential implementation details.
|
||||
|
||||
Sufficiency is a risk judgment, not a coverage percentage or test count. Reassess it as the application, its users, and the consequences of failure evolve.
|
||||
|
||||
The governing rule is:
|
||||
|
||||
> Test heavily where failure is consequential, subtle, or difficult to detect after the fact. Test lightly where failure is obvious, reversible, and inexpensive—and retain no test whose lifetime cost exceeds the confidence it provides.
|
||||
637
docs/roadmap/implementation.md
Normal file
637
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,637 @@
|
||||
# Step 7 Implementation Plan
|
||||
|
||||
## Status
|
||||
|
||||
Accepted for implementation.
|
||||
|
||||
Implement each stage in order. Do not begin a later stage until the current
|
||||
stage's exit criteria pass. The
|
||||
[Step 7 feature roadmap](step7.md) is the north star for scope, policy, desired
|
||||
behavior, and completion. [ADR 0002](../adr/0002-split-promptkit-from-scriptorium.md)
|
||||
controls the durable project boundary.
|
||||
|
||||
This plan changes Scriptorium only. Promptkit `v0.1.0` is a published,
|
||||
read-only input to the cutover. Do not edit Promptkit, create a new Promptkit
|
||||
tag, or use the sibling Promptkit checkout to make an intermediate
|
||||
Scriptorium state pass.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
Apply these rules throughout every stage:
|
||||
|
||||
- Read both repositories' development, architecture, documentation, and
|
||||
testing policies before implementation.
|
||||
- Preserve unrelated working-tree changes. The removals of completed roadmap
|
||||
files that predate this plan are user-owned changes and must not be restored.
|
||||
- Run Scriptorium commands outside a Go workspace. Do not commit `go.work`,
|
||||
`go.work.sum`, a local `replace`, vendored Promptkit source, or generated
|
||||
dependency output.
|
||||
- Depend only on
|
||||
`gitea.maximumdirect.net/eric/promptkit v0.1.0` and its root `promptkit`
|
||||
package. Do not import `promptkit/internal/...`.
|
||||
- Treat the former Scriptorium Go package as intentionally removed. Do not add
|
||||
aliases, forwarding wrappers, deprecated shims, or an application-local copy
|
||||
of Promptkit values.
|
||||
- Preserve CLI, HTTP, configuration, formatting, artifact-containment,
|
||||
security, error-mapping, and process behavior unless the feature roadmap
|
||||
explicitly says otherwise.
|
||||
- Prefer adapting existing application tests to Promptkit values over adding
|
||||
parallel test suites. Delete tests whose only owner is the removed framework.
|
||||
- Keep permanent documentation aligned with implemented state. Do not describe
|
||||
a later stage as complete before its code has landed.
|
||||
- Use `gofmt` for Go formatting and `go mod tidy` for module metadata. Do not
|
||||
hand-edit `go.sum`.
|
||||
|
||||
If a required capability is absent from Promptkit `v0.1.0`, stop the Step 7
|
||||
implementation and report the exact public-boundary gap. Do not work around it
|
||||
with a Scriptorium framework copy or an unpublished Promptkit revision.
|
||||
|
||||
## Stage 1: Cut The Application Seam Over To Promptkit
|
||||
|
||||
### Objective
|
||||
|
||||
Make all retained Scriptorium application code and tests consume the published
|
||||
Promptkit facade while the old local framework remains temporarily available
|
||||
for comparison. This stage proves that `v0.1.0` is sufficient before deleting
|
||||
the old implementation.
|
||||
|
||||
### Module Dependency
|
||||
|
||||
Add an exact direct requirement:
|
||||
|
||||
```text
|
||||
gitea.maximumdirect.net/eric/promptkit v0.1.0
|
||||
```
|
||||
|
||||
Resolve it from the configured module source with `GOWORK=off`. Do not add a
|
||||
replacement. Do not prune the old framework-only dependencies yet; the local
|
||||
framework still compiles during this stage.
|
||||
|
||||
Confirm before changing imports that the selected module resolves to
|
||||
`v0.1.0` and that its origin hash is the accepted Promptkit release commit
|
||||
`9e68a2bbf779545995270c47842048a3bc6c85dc`.
|
||||
|
||||
### Production Import And Type Migration
|
||||
|
||||
Replace the former root-package import with
|
||||
`gitea.maximumdirect.net/eric/promptkit` in every retained application package.
|
||||
Use the package name `promptkit` in code rather than aliasing it back to
|
||||
`scriptorium`.
|
||||
|
||||
Update `internal/adapter/cli/run.go` so:
|
||||
|
||||
- engine construction calls `promptkit.NewEngine` with `promptkit.Config`;
|
||||
- `run` and `render` requests use `promptkit.RunRequest`;
|
||||
- input mappings use `promptkit.File` and Promptkit artifact references;
|
||||
- execution overrides use `promptkit.ExecutionTargetOverride`;
|
||||
- the `serve` path injects the HTTP reader with
|
||||
`promptkit.WithArtifactReader`;
|
||||
- result summaries, validation classification, and exit-code selection use
|
||||
Promptkit result and status values; and
|
||||
- no CLI flag, precedence, output, or error wording changes solely because of
|
||||
the import migration.
|
||||
|
||||
Update `internal/adapter/http` so:
|
||||
|
||||
- the adapter-owned `Runner` interface accepts `promptkit.RunRequest` and
|
||||
returns `*promptkit.RunResult`;
|
||||
- DTO mapping constructs Promptkit artifact references and execution
|
||||
overrides;
|
||||
- response mapping reads Promptkit artifact, validation, model, usage, and
|
||||
metadata values directly;
|
||||
- HTTP error mapping uses Promptkit's public sentinel errors with `errors.Is`;
|
||||
and
|
||||
- the restricted reader implements `promptkit.ArtifactReader` and uses
|
||||
Promptkit artifact reference, artifact, and reference-type values.
|
||||
|
||||
Keep `ErrFileNotAllowed`, `ErrFileOutsideRoot`, and `ErrFileTooLarge` in
|
||||
Scriptorium. Confirm that Promptkit's artifact-load wrapping preserves those
|
||||
underlying identities so the existing HTTP mappings continue to work.
|
||||
|
||||
Update `internal/format` so every formatter interface, entry point, message
|
||||
collection, and test fixture uses Promptkit prepared-run and message values
|
||||
directly.
|
||||
|
||||
### Test Migration
|
||||
|
||||
Update retained CLI, HTTP, restricted-reader, and formatter tests to import and
|
||||
construct Promptkit public values. In particular:
|
||||
|
||||
- fake runners must use the Promptkit request/result signature;
|
||||
- handler integration helpers must construct a real `promptkit.Engine`;
|
||||
- test LLM clients must implement `promptkit.LLMClient`;
|
||||
- engine options must use Promptkit options;
|
||||
- error-mapping tables must use Promptkit sentinels; and
|
||||
- formatting fixtures must use Promptkit public prepared-run types.
|
||||
|
||||
Do not change expected CLI output, HTTP payloads, status codes, error codes,
|
||||
artifact policy, or formatted render output merely to accommodate the new
|
||||
package name.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
go mod download gitea.maximumdirect.net/eric/promptkit@v0.1.0
|
||||
go list -m -f '{{.Path}} {{.Version}}' gitea.maximumdirect.net/eric/promptkit
|
||||
go test ./internal/adapter/... ./internal/format/...
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/scriptorium
|
||||
```
|
||||
|
||||
Also run the maintained render script and the existing CLI command tests that
|
||||
exercise built-in profiles, custom profiles, explicit zero-valued overrides,
|
||||
restricted HTTP artifacts, public error mapping, and prepared-run formatting.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- All retained application production code uses Promptkit public types.
|
||||
- Application tests use Promptkit at the same boundaries as production.
|
||||
- The exact direct `v0.1.0` dependency is selected without a replacement.
|
||||
- The full repository still builds and tests while the local framework copy is
|
||||
present but unused by retained application code.
|
||||
- No Promptkit API gap has been discovered.
|
||||
|
||||
## Stage 2: Remove The Duplicated Framework
|
||||
|
||||
### Objective
|
||||
|
||||
Delete every Promptkit-owned implementation, facade, test, fixture, and example
|
||||
from Scriptorium, then reduce application defaults and module dependencies to
|
||||
the slim repository boundary.
|
||||
|
||||
### Remove The Former Root Package
|
||||
|
||||
Delete the complete former root facade:
|
||||
|
||||
- `artifact_reader.go`
|
||||
- `artifact_reader_internal_test.go`
|
||||
- `convert.go`
|
||||
- `engine.go`
|
||||
- `engine_test.go`
|
||||
- `errors.go`
|
||||
- `formatting.go`
|
||||
- `json_copy.go`
|
||||
- `llm_adapter.go`
|
||||
- `profiles.go`
|
||||
- `types.go`
|
||||
|
||||
After deletion, the Scriptorium module root must contain no `.go` files and
|
||||
must not be importable as package `gitea.maximumdirect.net/eric/scriptorium`.
|
||||
Do not replace these files with a stub package or migration shim.
|
||||
|
||||
### Remove Promptkit-Owned Internal Packages
|
||||
|
||||
Delete these directories in full, including their package tests, testdata, and
|
||||
embedded assets:
|
||||
|
||||
- `internal/artifact`
|
||||
- `internal/domain`
|
||||
- `internal/filecatalog`
|
||||
- `internal/llm`
|
||||
- `internal/profile`
|
||||
- `internal/prompt`
|
||||
- `internal/promptdef`
|
||||
- `internal/usecase`
|
||||
- `internal/validate`
|
||||
|
||||
Delete `testdata/framework`. Promptkit owns the framework contract corpus and
|
||||
already contains its maintained copy.
|
||||
|
||||
Delete `examples/go-library`. The Scriptorium Go-library example is obsolete;
|
||||
Promptkit owns the maintained in-process preparation example.
|
||||
|
||||
Do not delete the executable assets under `examples/config*.yml`,
|
||||
`examples/prompts`, `examples/profiles`, `examples/schemas`,
|
||||
`examples/fixtures`, `examples/http-run.json`, or
|
||||
`examples/render-markdown-summary.sh`. Those remain Scriptorium application
|
||||
examples even though Promptkit owns their file-format semantics.
|
||||
|
||||
### Split Defaults By Ownership
|
||||
|
||||
Reduce `internal/defaults/defaults.go` to values still owned and used by
|
||||
Scriptorium:
|
||||
|
||||
- `HTTPAddrDefault`
|
||||
- `SchemaDirDefault`, which remains the application configuration default for
|
||||
Scriptorium's schema source location;
|
||||
- `HTTPMaxRequestBytesDefault`
|
||||
- `HTTPMaxArtifactBytesDefault`
|
||||
- `HTTPMaxResponseBytesDefault`; and
|
||||
- `HTTPReadHeaderTimeoutDefault`.
|
||||
|
||||
Remove framework-owned output-artifact names, content types,
|
||||
OpenAI-compatible paths, execution defaults, model-client timeouts, domain
|
||||
imports, and `ExecutionTargetDefault`.
|
||||
|
||||
Do not copy constants from Promptkit merely to make tests convenient. When the
|
||||
CLI does not supply a framework setting, continue passing the zero value so
|
||||
Promptkit applies its own default.
|
||||
|
||||
### Prune Module Metadata
|
||||
|
||||
Run `go mod tidy` only after all Promptkit-owned source and tests are gone.
|
||||
Keep `gopkg.in/yaml.v3` as a direct dependency because Scriptorium still
|
||||
decodes application configuration. Allow `go mod tidy` to remove or reclassify
|
||||
JSON Schema and other modules according to the actual remaining import graph.
|
||||
|
||||
The final `go.mod` must have:
|
||||
|
||||
- Promptkit `v0.1.0` as a direct dependency;
|
||||
- no `replace` directive;
|
||||
- no dependency retained solely for deleted framework code; and
|
||||
- the existing compatible Go language version.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
go mod tidy
|
||||
go list ./...
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/scriptorium
|
||||
```
|
||||
|
||||
Inspect `go list ./...` and the filesystem. The package inventory must consist
|
||||
only of the command and retained application packages: adapters,
|
||||
configuration, defaults, and formatting. The module root and removed framework
|
||||
package families must be absent.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- Scriptorium exposes no root Go package.
|
||||
- No duplicated framework package, built-in registry, framework testdata, or
|
||||
Go-library example remains.
|
||||
- Application defaults contain no Promptkit-owned behavior.
|
||||
- Module metadata reflects the actual slim application graph.
|
||||
- All remaining packages test, vet, and build against Promptkit `v0.1.0`.
|
||||
|
||||
## Stage 3: Harden The Boundary And Rationalize Tests
|
||||
|
||||
### Objective
|
||||
|
||||
Make the slim dependency direction durable and leave Scriptorium with a lean
|
||||
test suite that protects application behavior rather than retesting Promptkit.
|
||||
|
||||
### Architecture Guard
|
||||
|
||||
Rewrite `internal/adapter/dependency_test.go` as a repository-level
|
||||
application-boundary guard. It may remain in the `internal/adapter` test
|
||||
package, but it must locate the module root reliably and recursively inspect
|
||||
all non-test production `.go` files under the repository.
|
||||
|
||||
The scanner must:
|
||||
|
||||
- skip `.git`, `vendor`, and other non-source output directories;
|
||||
- parse imports with the Go parser instead of searching raw source text;
|
||||
- reject the exact former root import
|
||||
`gitea.maximumdirect.net/eric/scriptorium`;
|
||||
- reject the former Scriptorium framework package families removed in Stage 2;
|
||||
- reject `gitea.maximumdirect.net/eric/promptkit/internal` and every path below
|
||||
it; and
|
||||
- continue allowing retained Scriptorium application-internal imports such as
|
||||
`internal/config`, `internal/defaults`, `internal/format`, and adapter
|
||||
packages.
|
||||
|
||||
Add focused scanner tests using temporary nested source trees to prove exact
|
||||
root-package detection, nested former-family detection, Promptkit-internal
|
||||
detection, and allowance of retained application packages.
|
||||
|
||||
Add an architecture assertion that:
|
||||
|
||||
- the module root has no production `.go` file; and
|
||||
- each removed framework directory is absent.
|
||||
|
||||
This is an intentional architecture invariant, not a generic assertion that
|
||||
the whole repository tree can never change.
|
||||
|
||||
### Test Ownership Audit
|
||||
|
||||
Review every remaining test after the mechanical import migration:
|
||||
|
||||
- `internal/adapter/cli` owns CLI parsing, configuration/flag mapping, command
|
||||
output, summaries, exit codes, and representative real-engine command
|
||||
workflows;
|
||||
- `internal/adapter/http` owns DTO mapping, strict JSON, status/error mapping,
|
||||
limits, restricted artifact policy, cancellation, and representative
|
||||
real-engine handler workflows;
|
||||
- `internal/config` owns strict application YAML, discovery, precedence,
|
||||
validation, and application defaults;
|
||||
- `internal/format` owns deterministic text/JSON presentation; and
|
||||
- the architecture guard owns dependency direction and removal invariants.
|
||||
|
||||
Retain the existing real Promptkit-engine adapter tests because they protect
|
||||
the consumer integration seam. Keep their prompt and profile fixtures
|
||||
package-local or generated with `t.TempDir`; do not recreate
|
||||
`testdata/framework`.
|
||||
|
||||
Delete or simplify any assertion that only duplicates Promptkit parser,
|
||||
orchestration, profile, validation, model-client, or public-contract tests.
|
||||
Do not add a Scriptorium table for every Promptkit sentinel or field. Preserve
|
||||
only mappings Scriptorium exposes through CLI or HTTP behavior.
|
||||
|
||||
Add a new test only when the cutover exposes an application-owned risk that is
|
||||
not already covered. Prefer updating an existing adapter integration test over
|
||||
creating a new end-to-end harness.
|
||||
|
||||
### Release Workflow Check
|
||||
|
||||
Inspect `.woodpecker/release.yml` and any packaging metadata for assumptions
|
||||
about the removed root package or framework directories. Keep the release
|
||||
target at `./cmd/scriptorium` (or its canonical module package path) and ensure
|
||||
the workflow obtains Promptkit through ordinary module resolution.
|
||||
|
||||
Do not add a Promptkit checkout, workspace, replacement, vendor step, or
|
||||
cross-repository artifact to Scriptorium CI or release packaging.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
go build ./cmd/scriptorium
|
||||
```
|
||||
|
||||
Run the architecture guard directly by name as a focused diagnostic. Confirm
|
||||
that its negative fixtures fail the scanner for the intended import and that
|
||||
the real repository produces no violation.
|
||||
|
||||
Exercise the release workflow's Go build commands locally with temporary
|
||||
outputs for its supported target platforms. Do not create `dist/` or another
|
||||
tracked output directory during validation.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- The repository guard protects the complete new dependency direction.
|
||||
- Remaining tests have an explicit application-owned purpose.
|
||||
- No deleted Promptkit behavior is redundantly reimplemented in tests or
|
||||
fixtures.
|
||||
- Race-enabled tests, vet, application builds, and release-equivalent builds
|
||||
pass.
|
||||
|
||||
## Stage 4: Reconcile Permanent Documentation And Examples
|
||||
|
||||
### Objective
|
||||
|
||||
Make all permanent Scriptorium documentation describe the implemented slim
|
||||
application and route framework readers to the version of Promptkit the
|
||||
application actually consumes.
|
||||
|
||||
### Version-Appropriate Promptkit Links
|
||||
|
||||
Use the tagged Promptkit `v0.1.0` documents as the canonical framework
|
||||
references during this step. Link to the corresponding files under:
|
||||
|
||||
```text
|
||||
https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/
|
||||
```
|
||||
|
||||
Use these owners where applicable:
|
||||
|
||||
- `docs/consumers/pkg-promptkit.md` for Go-library consumer guidance;
|
||||
- `docs/formats.md` for prompt, profile, schema, execution-setting, built-in
|
||||
profile, and framework credential contracts; and
|
||||
- `docs/integrations/openai-compatible-chat.md` for outbound provider wire and
|
||||
timeout behavior.
|
||||
|
||||
Do not copy their field tables, defaults, error inventories, or timeout rules
|
||||
into Scriptorium. Scriptorium documents only its own mapping or invocation
|
||||
surface and links to the Promptkit owner for the framework definition.
|
||||
|
||||
### Orientation And Policy
|
||||
|
||||
Update:
|
||||
|
||||
- `README.md` to present only the runnable CLI and HTTP application, remove the
|
||||
Scriptorium Go-package link and Go-library example, and identify Promptkit as
|
||||
the imported framework;
|
||||
- `docs/policy/architecture.md` to define the current application-only shape,
|
||||
retained package boundaries, Promptkit dependency direction, HTTP artifact
|
||||
security boundary, application invariants, and non-goals;
|
||||
- `docs/development.md` to route contributors to the remaining adapter,
|
||||
configuration, formatting, packaging, and Promptkit-consumer boundaries;
|
||||
- `docs/policy/documentation.md` where its ownership table or boundary rules
|
||||
still imply that Scriptorium owns a public Go package or framework
|
||||
integration; and
|
||||
- `docs/policy/testing.md` where framework-domain, validator, model-client, or
|
||||
orchestration examples imply those are still Scriptorium test owners.
|
||||
|
||||
Do not rewrite accepted ADR decision content. ADR 0002 remains the permanent
|
||||
decision record even where its context describes the pre-split state.
|
||||
|
||||
### External And User Contracts
|
||||
|
||||
Audit and update:
|
||||
|
||||
- `docs/cli.md`
|
||||
- `docs/api.md`
|
||||
- `docs/config.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/consumers/api.md`
|
||||
- `docs/integrations/subprocess.md`
|
||||
|
||||
Preserve Scriptorium-owned command syntax, configuration discovery and server
|
||||
fields, HTTP schemas and mappings, operations, and subprocess behavior. Remove
|
||||
claims that Scriptorium provides an in-process Go API. Where these documents
|
||||
name prompt/profile fields, model semantics, built-in profiles, framework
|
||||
credentials, or timeout behavior, keep only the Scriptorium mapping context
|
||||
and link to the tagged Promptkit owner.
|
||||
|
||||
Delete `docs/consumers/pkg-scriptorium.md`. Update every incoming link so Go
|
||||
framework consumers are directed to Promptkit rather than to a compatibility
|
||||
document.
|
||||
|
||||
Delete `docs/integrations/openai-compatible-chat.md` after replacing all
|
||||
incoming Scriptorium links with the tagged Promptkit integration contract.
|
||||
Scriptorium no longer owns the outbound provider protocol.
|
||||
|
||||
### Internal Documentation
|
||||
|
||||
Update `docs/internal/overview.md` to inventory exactly:
|
||||
|
||||
- `cmd/scriptorium`;
|
||||
- `internal/adapter/cli`;
|
||||
- `internal/adapter/http`;
|
||||
- `internal/config`;
|
||||
- `internal/defaults`; and
|
||||
- `internal/format`.
|
||||
|
||||
Update `docs/internal/adapters.md` for Promptkit request/result mapping, engine
|
||||
construction, consumer-owned interfaces, error mapping, output behavior, and
|
||||
restricted-reader injection.
|
||||
|
||||
Rewrite `docs/internal/sources.md` around Scriptorium-owned application source
|
||||
locations and the restricted HTTP artifact reader. Link Promptkit for ordinary
|
||||
artifact loading and framework source semantics.
|
||||
|
||||
Delete:
|
||||
|
||||
- `docs/internal/runner.md`
|
||||
- `docs/internal/llm.md`
|
||||
|
||||
Promptkit owns those subsystem documents. Remove all incoming links rather than
|
||||
leaving placeholder internal documents.
|
||||
|
||||
### Examples And Navigation
|
||||
|
||||
Keep the executable configuration, prompt, profile, schema, fixture, HTTP, and
|
||||
render-script examples. Confirm they still run against the tagged dependency
|
||||
and that prose describes them as Scriptorium application examples.
|
||||
|
||||
Remove every link to `examples/go-library`, root Go declarations, removed
|
||||
framework directories, deleted Scriptorium consumer documentation, and deleted
|
||||
internal or integration documents.
|
||||
|
||||
Search all permanent Markdown outside `docs/roadmap/` for:
|
||||
|
||||
- the former root import path used as a Go package;
|
||||
- claims that Scriptorium owns or implements the framework;
|
||||
- references to removed packages, tests, fixtures, or examples; and
|
||||
- duplicate Promptkit framework contracts.
|
||||
|
||||
The migration roadmap and accepted ADRs may retain historical and future
|
||||
language appropriate to their lifecycle.
|
||||
|
||||
### Validation
|
||||
|
||||
Validate every local Markdown link and follow every new external Promptkit
|
||||
link. Run:
|
||||
|
||||
```sh
|
||||
git diff --check
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/scriptorium
|
||||
```
|
||||
|
||||
Run the maintained render script and both `examples/config.yml` and
|
||||
`examples/config.full.yml` through a credential-free `render` workflow. No
|
||||
documentation validation may invoke a paid or live model endpoint.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- Permanent docs describe only the implemented application boundary.
|
||||
- Framework contracts have one Promptkit owner and version-appropriate links.
|
||||
- Removed code and examples have no live incoming links.
|
||||
- All retained examples are valid, secret-free, and runnable offline where
|
||||
intended.
|
||||
- Documentation links, whitespace, tests, vet, and the executable build pass.
|
||||
|
||||
## Stage 5: Perform Independent Acceptance And Record Completion
|
||||
|
||||
### Objective
|
||||
|
||||
Validate the exact final state without sibling-repository coupling, then record
|
||||
the Step 7 gate only after every feature-roadmap completion criterion is
|
||||
satisfied.
|
||||
|
||||
### Scriptorium Acceptance
|
||||
|
||||
From the Scriptorium repository root, confirm no Go workspace is active and no
|
||||
replacement or vendored dependency exists. Validate module identity and
|
||||
selection:
|
||||
|
||||
```sh
|
||||
go list -m -f '{{.Path}} {{.GoVersion}}'
|
||||
go list -m -f '{{.Path}} {{.Version}}' gitea.maximumdirect.net/eric/promptkit
|
||||
go mod graph
|
||||
go mod tidy -diff
|
||||
```
|
||||
|
||||
Require the Scriptorium module path and existing Go version, and require
|
||||
Promptkit to resolve exactly to `v0.1.0`. Inspect the graph for a normal tagged
|
||||
module edge rather than a local path.
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
go build ./cmd/scriptorium
|
||||
gofmt -l $(git ls-files '*.go')
|
||||
git diff --check
|
||||
```
|
||||
|
||||
The formatting command must produce no paths. Build the executable to a
|
||||
temporary directory rather than the repository. Reproduce the release
|
||||
workflow's supported cross-compilation targets and linker flags with temporary
|
||||
outputs.
|
||||
|
||||
Run the maintained render script and credential-free render commands using
|
||||
both configuration examples. Verify representative HTTP behavior through the
|
||||
retained handler tests, including strict decoding, request/response size
|
||||
limits, restricted file access, public error mapping, and optional raw output.
|
||||
|
||||
Validate all maintained Markdown links, including the tagged Promptkit links.
|
||||
Search the final tracked tree for former root-facade imports, removed framework
|
||||
packages, built-in asset copies, framework fixtures, Go-library examples,
|
||||
workspaces, replacements, generated binaries, credentials, and stale
|
||||
documentation references.
|
||||
|
||||
### Clean Remote-Dependency Verification
|
||||
|
||||
Use a fresh temporary module cache with `GOWORK=off` and ordinary remote module
|
||||
resolution to download dependencies and run at least the Scriptorium build and
|
||||
test suite. The verification must not read the sibling Promptkit working tree.
|
||||
Confirm the downloaded Promptkit module reports version `v0.1.0` and origin
|
||||
commit `9e68a2bbf779545995270c47842048a3bc6c85dc`.
|
||||
|
||||
Do not clear or destructively rewrite the maintainer's normal module cache.
|
||||
Use a temporary directory and remove it after validation.
|
||||
|
||||
### Independent Promptkit Reconfirmation
|
||||
|
||||
Without changing Promptkit, validate the exact `v0.1.0` source from a temporary
|
||||
detached worktree or the downloaded module directory. If the clean Promptkit
|
||||
checkout still has `HEAD` exactly at the peeled `v0.1.0` commit, it may be used
|
||||
directly. Run Promptkit's documented sequence from that exact source:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
go build ./...
|
||||
go run ./examples/go-library/prepare
|
||||
```
|
||||
|
||||
Also check Promptkit formatting, links, whitespace, module tidiness, absence of
|
||||
a workspace/replacement, clean status, and the local and remote `v0.1.0` tag
|
||||
target. Remove any temporary detached worktree after validation. This proves
|
||||
that Scriptorium's adoption did not introduce cross-repository state or require
|
||||
an unpublished Promptkit change.
|
||||
|
||||
### Completion Records
|
||||
|
||||
Only after all acceptance checks pass:
|
||||
|
||||
- update `docs/roadmap/step7.md` status from Proposed to Complete and replace
|
||||
future-oriented starting-state material as needed with a concise result,
|
||||
retained application boundary, validation evidence, and Step 8 handoff;
|
||||
- update `docs/roadmap/migration.md` so its status says Steps 1 through 7 are
|
||||
complete and its Step 7 gate records the tagged dependency, removal of the
|
||||
framework copy and public facade, independent validation, and readiness for
|
||||
downstream migration; and
|
||||
- replace this staged checklist with a concise Step 7 implementation
|
||||
completion record containing the dependency version, resulting package
|
||||
boundary, removed ownership, validation evidence, and next gate.
|
||||
|
||||
Do not mark Step 8 or Step 9 complete. Do not publish a Scriptorium release or
|
||||
claim that external consumers have migrated.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- Every completion criterion in `docs/roadmap/step7.md` is demonstrably true.
|
||||
- Scriptorium passes release-grade validation against remotely resolved
|
||||
Promptkit `v0.1.0`.
|
||||
- Promptkit remains independently clean and valid at its published tag.
|
||||
- Permanent documentation, examples, module metadata, and architecture checks
|
||||
match the final slim application.
|
||||
- Completion records accurately authorize Step 8 without claiming later work.
|
||||
327
docs/roadmap/migration.md
Normal file
327
docs/roadmap/migration.md
Normal file
@@ -0,0 +1,327 @@
|
||||
# Promptkit Migration Roadmap
|
||||
|
||||
## Status
|
||||
|
||||
Accepted plan. Steps 1 through 6 are complete. Steps 7 through 9 remain
|
||||
proposed and are not yet implemented.
|
||||
|
||||
## Objective
|
||||
|
||||
Split the current repository into two projects:
|
||||
|
||||
- **Promptkit**: the reusable Go framework, public Go facade, execution engine,
|
||||
source and validation support, OpenAI-compatible client, extension
|
||||
interfaces, and built-in execution-profile registry.
|
||||
- **Scriptorium**: a slim runnable application that imports Promptkit and
|
||||
provides the CLI and HTTP interfaces.
|
||||
|
||||
Scriptorium will become another downstream Promptkit consumer rather than the
|
||||
owner of the framework.
|
||||
|
||||
## Compatibility And Migration Policy
|
||||
|
||||
This is an intentionally breaking change.
|
||||
|
||||
- New and migrated Go consumers must import Promptkit instead of Scriptorium.
|
||||
- Scriptorium will not retain type aliases, forwarding packages, deprecated
|
||||
facade APIs, or other source-compatibility shims.
|
||||
- Existing consumers may continue using a previously tagged Scriptorium module
|
||||
version until they are migrated.
|
||||
- The migration does not need to preserve compatibility between intermediate
|
||||
development states. Each completed phase must instead leave the affected
|
||||
repository internally consistent and tested.
|
||||
- Promptkit should initially preserve the useful shape and behavior of the
|
||||
current public Go facade where doing so reduces extraction risk. Broader API
|
||||
redesign should follow the split unless required to establish the new
|
||||
boundary.
|
||||
|
||||
## Target Ownership
|
||||
|
||||
Promptkit should own application-neutral framework behavior:
|
||||
|
||||
- public engine, request, result, option, extension, and error APIs;
|
||||
- prompt-definition loading and rendering;
|
||||
- execution profiles, overlays, and the built-in profile registry;
|
||||
- artifact-loading interfaces and general-purpose `file` and `inline` support;
|
||||
- schema loading and output validation;
|
||||
- LLM client boundaries and the OpenAI-compatible implementation;
|
||||
- preparation and execution orchestration;
|
||||
- framework and execution defaults.
|
||||
|
||||
Scriptorium should own executable and transport concerns:
|
||||
|
||||
- the `scriptorium` command and its `run`, `render`, and `serve` interfaces;
|
||||
- CLI parsing, output formatting, exit codes, and process behavior;
|
||||
- application-config discovery and CLI precedence;
|
||||
- HTTP routing, request and response DTOs, limits, and error/status mapping;
|
||||
- HTTP artifact-root and deployment security policy;
|
||||
- server and adapter defaults;
|
||||
- executable examples, operations guidance, and transport documentation.
|
||||
|
||||
The intended dependency direction is:
|
||||
|
||||
```text
|
||||
Scriptorium CLI and HTTP adapters
|
||||
|
|
||||
v
|
||||
Promptkit
|
||||
|
|
||||
v
|
||||
consumer-supplied sources and clients
|
||||
```
|
||||
|
||||
Scriptorium must use Promptkit's public API. It must not depend on Promptkit
|
||||
implementation packages or reproduce framework orchestration.
|
||||
|
||||
## Migration Steps
|
||||
|
||||
### Step 1: Refresh And Synchronize Documentation
|
||||
|
||||
Perform a repository-wide documentation refresh before migration development.
|
||||
|
||||
At minimum:
|
||||
|
||||
- reconcile all current-behavior documentation with the code, tests, examples,
|
||||
defaults, and current public contracts;
|
||||
- introduce the planned documentation-policy updates;
|
||||
- establish an architecture decision record policy and canonical ADR location;
|
||||
- resolve stale, duplicated, or misplaced material;
|
||||
- validate documentation links and maintained examples;
|
||||
- leave future migration behavior in `docs/roadmap/` until implemented.
|
||||
|
||||
**Gate:** Do not begin architectural migration work until the documentation
|
||||
refresh and policy updates are merged and the repository has an agreed,
|
||||
accurate baseline.
|
||||
|
||||
**Gate status:** Complete as of 2026-07-26. The completed documentation
|
||||
refresh, follow-up verification, and layered-timeout correction remain recorded
|
||||
in repository history. Step 1 remains complete after that validation.
|
||||
|
||||
### Step 2: Record The Architectural Decision And Detailed Boundary
|
||||
|
||||
Create an ADR, under the policy established in Step 1, that records:
|
||||
|
||||
- the decision to split Promptkit from Scriptorium;
|
||||
- the target ownership and dependency direction;
|
||||
- the selected Promptkit repository and Go module paths;
|
||||
- the breaking-change and versioning policy;
|
||||
- ownership of configuration fields and defaults;
|
||||
- artifact-reader and HTTP containment responsibilities;
|
||||
- local multi-repository development and release coordination;
|
||||
- documentation ownership after the split.
|
||||
|
||||
Use the ADR to resolve any remaining public-boundary decisions before code is
|
||||
moved.
|
||||
|
||||
**Gate:** The ADR is accepted, and every existing package, public contract,
|
||||
configuration category, and maintained asset has a target owner.
|
||||
|
||||
**Gate status:** Complete as of 2026-07-26.
|
||||
[ADR 0002: Split Promptkit From Scriptorium](../adr/0002-split-promptkit-from-scriptorium.md)
|
||||
is accepted and records the required ownership and coordination decisions.
|
||||
|
||||
### Step 3: Characterize Existing Framework Behavior
|
||||
|
||||
Strengthen or add contract-focused tests where needed so extraction can be
|
||||
verified without relying on package placement.
|
||||
|
||||
The completed Step 3 gate records the accepted implementation scope and
|
||||
intended completion state.
|
||||
|
||||
Preserve coverage of:
|
||||
|
||||
- `Prepare` and `Run` behavior;
|
||||
- prompt, profile, execution-default, and request-override precedence;
|
||||
- presence-aware numeric overrides;
|
||||
- built-in profile fallback and custom-profile overlays;
|
||||
- strict YAML and JSON decoding;
|
||||
- prompt, profile, schema, and artifact source behavior;
|
||||
- structured-output requests and output validation;
|
||||
- validation failures versus validation runtime errors;
|
||||
- secret handling and redaction;
|
||||
- public error classification;
|
||||
- HTTP artifact restrictions and transport mappings.
|
||||
|
||||
**Gate:** Current framework and adapter contracts are represented by passing
|
||||
tests sufficient to detect behavioral regressions during the split.
|
||||
|
||||
**Gate status:** Complete as of 2026-07-27. The framework contract corpus,
|
||||
public `Engine` characterization, ownership audit, full test and vet suites,
|
||||
temporary executable build, and maintained offline examples passed.
|
||||
|
||||
### Step 4: Make Scriptorium Adapters Consume The Public Facade
|
||||
|
||||
Within the current repository, refactor the CLI and HTTP adapters to use the
|
||||
public framework facade rather than constructing or importing internal runner
|
||||
components directly.
|
||||
|
||||
Add only the minimum public capabilities needed to support this boundary. These
|
||||
may include:
|
||||
|
||||
- a small `Run`/`Prepare` consumer interface;
|
||||
- injectable artifact-reading behavior for Scriptorium's restricted HTTP
|
||||
policy;
|
||||
- source options currently available only through internal constructors;
|
||||
- prepared-run formatting based on public types;
|
||||
- stable public error classification required by CLI and HTTP mappings.
|
||||
|
||||
Do not broadly export internal repositories, domain types, or use-case
|
||||
implementations.
|
||||
|
||||
**Gate:** The CLI and HTTP adapters use only the public framework API for
|
||||
framework behavior, and all tests and documented smoke commands pass.
|
||||
|
||||
**Gate status:** Complete as of 2026-07-28. CLI `run`, `render`, and `serve`,
|
||||
the HTTP handler, and prepared-run formatting use the public facade; the
|
||||
restricted HTTP reader is injected through the public extension point. The
|
||||
post-implementation public-error, deterministic MIME, and recursive
|
||||
dependency-guard corrections passed full tests, vet, build, race checks,
|
||||
maintained examples, and configuration smoke checks.
|
||||
|
||||
### Step 5: Create The Promptkit Repository
|
||||
|
||||
Promptkit was established as an independent repository and Go module through
|
||||
the completed out-of-band workflow recorded in repository history. Its
|
||||
foundation includes:
|
||||
|
||||
- confirmed repository access, governance, origin, and default-branch tracking;
|
||||
- module `gitea.maximumdirect.net/eric/promptkit` at Go `1.25.5`;
|
||||
- a minimal root `promptkit` public package boundary with no placeholder
|
||||
framework packages;
|
||||
- library-specific development, architecture, documentation, testing, and
|
||||
release policies;
|
||||
- documented maintainer-run test, vet, build, formatting, documentation-link,
|
||||
and repository-hygiene validation;
|
||||
- source-commit and semantic Go module tag releases without runnable binaries
|
||||
or binary packaging; and
|
||||
- temporary workspace and uncommitted replacement workflows for coordinated
|
||||
development without committed repository coupling.
|
||||
|
||||
[ADR 0003](../adr/0003-use-maintainer-run-validation-and-tag-only-releases-for-promptkit.md)
|
||||
records the controlling Promptkit validation and release decision.
|
||||
|
||||
**Gate status:** Complete as of 2026-07-28. Promptkit passed its documented
|
||||
validation independently, all maintained links and repository-hygiene checks
|
||||
passed, and no workspace, replacement, CI configuration, binary, tag, command,
|
||||
or placeholder package was added. The completed repository foundation
|
||||
supported the Step 6 extraction.
|
||||
|
||||
### Step 6: Extract And Stabilize Promptkit
|
||||
|
||||
Move the application-neutral framework and built-in profile assets into
|
||||
Promptkit. Preserve implementation packages as internal where practical.
|
||||
|
||||
The initial public API should remain focused on the established engine workflow
|
||||
and the source and client extension points required by real consumers. Avoid
|
||||
combining the extraction with unrelated API redesign.
|
||||
|
||||
Move or recreate the relevant:
|
||||
|
||||
- framework implementation;
|
||||
- public package tests and framework contract tests;
|
||||
- built-in profile assets and registry tests;
|
||||
- Go consumer examples;
|
||||
- framework, consumer, configuration-format, and integration documentation.
|
||||
|
||||
Verify that Promptkit can be built, tested, and consumed independently of the
|
||||
Scriptorium repository.
|
||||
|
||||
**Gate:** Promptkit independently provides the agreed framework contract,
|
||||
passes its documented validation, and has published its first versioned tag
|
||||
before Scriptorium or another consumer adopts it, as required by
|
||||
[ADR 0003](../adr/0003-use-maintainer-run-validation-and-tag-only-releases-for-promptkit.md).
|
||||
|
||||
**Gate status:** Complete as of 2026-07-28. The
|
||||
[Step 6 completion record](step6.md) records source Scriptorium commit
|
||||
`c7263ab2a8e58f7fb97280082d327a820c7cece7`, accepted Promptkit commit
|
||||
`9e68a2bbf779545995270c47842048a3bc6c85dc`, independently passing acceptance,
|
||||
published annotated tag `v0.1.0`, and successful remote-consumer validation.
|
||||
Scriptorium remains unchanged at its pre-cutover boundary. Step 7 adoption of
|
||||
the tagged module and removal of the duplicated framework is the next gate.
|
||||
|
||||
### Step 7: Slim Scriptorium And Adopt Promptkit
|
||||
|
||||
Update Scriptorium to import the tagged Promptkit module and remove the
|
||||
framework implementation and public Go facade that Promptkit replaces.
|
||||
|
||||
Retain only Scriptorium-owned executable and transport behavior. In particular:
|
||||
|
||||
- wire CLI and HTTP requests through Promptkit's public API;
|
||||
- keep application config and transport defaults in Scriptorium;
|
||||
- keep restricted HTTP artifact policy in Scriptorium while injecting it
|
||||
through Promptkit's supported boundary;
|
||||
- remove obsolete framework packages, tests, and documentation;
|
||||
- update Scriptorium examples and docs to describe the CLI and HTTP application;
|
||||
- direct Go framework consumers to Promptkit without providing compatibility
|
||||
aliases or forwarding APIs.
|
||||
|
||||
**Gate:** Scriptorium builds and passes all tests using a tagged Promptkit
|
||||
dependency, contains no duplicate framework implementation, and its current
|
||||
documentation describes only the slimmed application.
|
||||
|
||||
### Step 8: Migrate Downstream Consumers To Promptkit
|
||||
|
||||
Inventory downstream Go consumers and migrate each from the Scriptorium package
|
||||
to Promptkit. This work may occur in external repositories and must be tracked
|
||||
explicitly.
|
||||
|
||||
For each consumer:
|
||||
|
||||
- update module imports and dependencies;
|
||||
- adapt to any intentionally changed public API;
|
||||
- run its tests and relevant integration or smoke checks;
|
||||
- confirm configuration, source, validation, and error behavior;
|
||||
- release or deploy the migrated consumer through its normal process.
|
||||
|
||||
Consumers that cannot migrate immediately may remain pinned to the last
|
||||
framework-bearing Scriptorium tag. No compatibility work is required in the new
|
||||
Scriptorium project for those consumers.
|
||||
|
||||
**Gate:** All in-scope downstream consumers are either migrated and verified or
|
||||
explicitly recorded as remaining on the previous Scriptorium version with an
|
||||
owner and follow-up plan. Do not declare the ecosystem migration complete until
|
||||
the required out-of-band consumer changes are confirmed.
|
||||
|
||||
### Step 9: Complete Release And Documentation Cutover
|
||||
|
||||
Complete the coordinated project transition:
|
||||
|
||||
- publish Promptkit before dependent Scriptorium releases;
|
||||
- release the breaking Scriptorium version against the tagged Promptkit
|
||||
dependency;
|
||||
- publish migration guidance that maps the former Scriptorium Go API to
|
||||
Promptkit;
|
||||
- update cross-project links, examples, package documentation, and release
|
||||
notes;
|
||||
- verify that no release artifact depends on local workspaces or replacements;
|
||||
- archive completed roadmap material according to the documentation policy in
|
||||
effect at that time.
|
||||
|
||||
**Gate:** Promptkit and Scriptorium are independently releasable, their
|
||||
documentation has distinct and accurate ownership, and the migration status of
|
||||
all identified downstream consumers is recorded.
|
||||
|
||||
## Cross-Cutting Constraints
|
||||
|
||||
- Preserve the invariant that execution orchestration remains narrow and
|
||||
application-neutral.
|
||||
- Keep adapter-specific decisions out of Promptkit.
|
||||
- Keep Scriptorium dependent only on Promptkit's supported public API.
|
||||
- Preserve strict external decoding, error classification, validation
|
||||
semantics, and secret redaction throughout the migration.
|
||||
- Keep each repository buildable and testable at merged phase boundaries.
|
||||
- Coordinate cross-repository changes through tagged dependencies and explicit
|
||||
gates rather than assuming atomic commits across repositories.
|
||||
- Document only implemented behavior outside roadmap files.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
The migration is complete when:
|
||||
|
||||
- Promptkit is the independent owner of the reusable framework and built-in
|
||||
profiles;
|
||||
- Scriptorium is a slim CLI and HTTP consumer of Promptkit;
|
||||
- Scriptorium no longer exposes or maintains the former public Go framework;
|
||||
- all required downstream migrations and external repository work have been
|
||||
completed or explicitly dispositioned;
|
||||
- both repositories build, test, document, version, and release independently.
|
||||
339
docs/roadmap/step7.md
Normal file
339
docs/roadmap/step7.md
Normal file
@@ -0,0 +1,339 @@
|
||||
# Migration Step 7: Slim Scriptorium And Adopt Promptkit
|
||||
|
||||
## Status
|
||||
|
||||
Proposed.
|
||||
|
||||
## Purpose
|
||||
|
||||
Complete Scriptorium's application-side cutover to the independently published
|
||||
Promptkit library. After this work, Scriptorium is a runnable CLI and HTTP
|
||||
application built on Promptkit's supported public API rather than a second
|
||||
owner of the prompt-execution framework.
|
||||
|
||||
This roadmap defines the required end state for Step 7. The
|
||||
[main migration roadmap](migration.md) owns the overall migration sequence,
|
||||
while
|
||||
[ADR 0002](../adr/0002-split-promptkit-from-scriptorium.md) owns the durable
|
||||
project, package, configuration, compatibility, and documentation boundaries.
|
||||
|
||||
## Starting Point
|
||||
|
||||
Promptkit is independently published as module
|
||||
`gitea.maximumdirect.net/eric/promptkit` at annotated tag `v0.1.0`. That release
|
||||
contains the characterized framework, root public facade, built-in profile
|
||||
registry, maintained framework tests, and consumer documentation extracted in
|
||||
Step 6.
|
||||
|
||||
Scriptorium intentionally still contains the pre-cutover copy of that
|
||||
framework. Its executable adapters already consume framework behavior through
|
||||
the local root facade, which provides a narrow migration seam. Step 7 replaces
|
||||
that local facade with Promptkit and removes the duplicated implementation; it
|
||||
does not redesign the framework or the executable interfaces.
|
||||
|
||||
## Desired End State
|
||||
|
||||
Scriptorium is an application-only Go module with this dependency direction:
|
||||
|
||||
```text
|
||||
scriptorium command
|
||||
|
|
||||
v
|
||||
CLI and HTTP adapters, application config, and output formatting
|
||||
|
|
||||
v
|
||||
gitea.maximumdirect.net/eric/promptkit public package
|
||||
|
|
||||
v
|
||||
Promptkit-owned framework implementation
|
||||
```
|
||||
|
||||
The Scriptorium module root no longer provides an importable Go package.
|
||||
Scriptorium has no compatibility facade, type aliases, forwarding functions,
|
||||
or deprecated wrappers for the former `scriptorium` public API. Go consumers
|
||||
must import Promptkit; consumers that have not migrated may remain pinned to a
|
||||
previous framework-bearing Scriptorium version as established by ADR 0002.
|
||||
|
||||
## Promptkit Dependency
|
||||
|
||||
Scriptorium must declare
|
||||
`gitea.maximumdirect.net/eric/promptkit v0.1.0` as a direct module dependency.
|
||||
Production code, tests, examples, builds, and release configuration must
|
||||
resolve that published tag without a committed workspace, local filesystem
|
||||
replacement, vendored Promptkit copy, or unpublished revision.
|
||||
|
||||
The cutover must use only Promptkit's supported root package. Scriptorium must
|
||||
not import, copy, or depend conceptually on Promptkit implementation packages.
|
||||
No Promptkit API expansion is expected for this step: Step 4 established the
|
||||
required application seam, and Step 6 published it. If implementation discovers
|
||||
a genuine missing Promptkit capability, that is a cross-repository blocker:
|
||||
the capability must be accepted, implemented, validated, and tagged in
|
||||
Promptkit before Scriptorium can depend on it.
|
||||
|
||||
Dependencies used only by the removed framework must leave Scriptorium's
|
||||
`go.mod` and `go.sum`. Scriptorium retains only dependencies required by its
|
||||
application-owned code. The module's Go version remains compatible with the
|
||||
selected Promptkit release.
|
||||
|
||||
## Application Assembly And Adapter Boundary
|
||||
|
||||
The CLI must construct `promptkit.Engine` values from Scriptorium's resolved
|
||||
application settings. It must translate Scriptorium CLI inputs and execution
|
||||
overrides into `promptkit.RunRequest`, `promptkit.ArtifactRef`, and
|
||||
`promptkit.ExecutionTargetOverride` values without introducing a parallel
|
||||
application model.
|
||||
|
||||
The `run` command must invoke `promptkit.Engine.Run`; the `render` command must
|
||||
invoke `promptkit.Engine.Prepare`. The `serve` command must inject
|
||||
Scriptorium's restricted artifact reader through
|
||||
`promptkit.WithArtifactReader` and pass the engine to the HTTP adapter through
|
||||
the adapter-owned consumer interface.
|
||||
|
||||
The HTTP adapter's `Runner` interface remains owned by Scriptorium because it
|
||||
is a consumer-side test and substitution boundary. Its method uses Promptkit
|
||||
request and result types. Request DTOs, response DTOs, strict JSON decoding,
|
||||
HTTP limits, error-to-status mapping, response encoding, and route behavior
|
||||
remain Scriptorium concerns and must not move into Promptkit.
|
||||
|
||||
Prepared-run formatting remains in Scriptorium because text and JSON output
|
||||
selection is an executable presentation concern. The formatter operates
|
||||
directly on Promptkit public prepared-run values; it must not introduce copied
|
||||
framework types.
|
||||
|
||||
## Artifact And Security Boundary
|
||||
|
||||
Scriptorium retains its HTTP-specific restricted artifact reader. The reader
|
||||
must implement `promptkit.ArtifactReader` and consume and return Promptkit
|
||||
artifact values directly.
|
||||
|
||||
The cutover must preserve the current HTTP artifact policy:
|
||||
|
||||
- an empty artifact root permits inline artifacts and denies file references;
|
||||
- configured byte limits apply to file artifacts, with zero meaning unlimited;
|
||||
- file paths are checked using the documented lexical root-containment rule;
|
||||
- symlinks retain their currently documented behavior;
|
||||
- content type, size, hash, name, URI, and cancellation behavior remain
|
||||
observable through the same HTTP contract; and
|
||||
- Scriptorium reader errors remain identifiable so the HTTP adapter can
|
||||
preserve its existing status and error-code mappings.
|
||||
|
||||
Ordinary in-process and CLI file reading belongs to Promptkit. Scriptorium must
|
||||
not retain its former general-purpose artifact reader or framework artifact
|
||||
package after the cutover.
|
||||
|
||||
## Configuration And Defaults
|
||||
|
||||
Scriptorium continues to own:
|
||||
|
||||
- configuration discovery and strict YAML decoding;
|
||||
- configuration-file and CLI precedence;
|
||||
- `prompt_dir`, `profile_dir`, and `schema_dir` as application source
|
||||
locations;
|
||||
- `server.*` settings and transport byte limits;
|
||||
- the default prepared-run output format; and
|
||||
- CLI, HTTP server, and process defaults.
|
||||
|
||||
These settings are translated into Promptkit construction options and request
|
||||
values at the application boundary. When omission means “use Promptkit's
|
||||
framework default,” Scriptorium must leave the value unset rather than
|
||||
redeclare a Promptkit constant.
|
||||
|
||||
Promptkit owns prompt, profile, and output-contract file semantics; built-in
|
||||
profiles; execution-setting resolution; validation behavior; output-artifact
|
||||
and framework content-type defaults; OpenAI-compatible request behavior; and
|
||||
generation and transport timeout semantics. Scriptorium's defaults package
|
||||
must be reduced to application and transport defaults still used by the CLI,
|
||||
HTTP server, configuration loader, or output formatter.
|
||||
|
||||
The cutover must not change documented configuration discovery, CLI
|
||||
precedence, source-path interpretation, server limits, or render-format
|
||||
behavior.
|
||||
|
||||
## Package And Asset Disposition
|
||||
|
||||
The following Scriptorium components remain:
|
||||
|
||||
| Component | Retained responsibility |
|
||||
| --- | --- |
|
||||
| `cmd/scriptorium` | Runnable process entrypoint. |
|
||||
| `internal/adapter/cli` | CLI parsing, application assembly, streams, files, summaries, and exit codes. |
|
||||
| `internal/adapter/http` | Routes, DTOs, strict decoding, HTTP limits and mappings, and restricted artifact reading. |
|
||||
| `internal/config` | Application configuration discovery, decoding, validation, defaults, and CLI precedence. |
|
||||
| `internal/format` | Prepared-run text and JSON presentation using Promptkit public values. |
|
||||
| `internal/defaults` | Scriptorium-only CLI, HTTP, server, and application defaults. |
|
||||
| `.woodpecker`, release metadata, and executable packaging | Scriptorium build and binary-release behavior. |
|
||||
|
||||
The following duplicated framework components must be removed from
|
||||
Scriptorium:
|
||||
|
||||
- all root-package facade source and tests;
|
||||
- `internal/artifact`, `internal/domain`, `internal/filecatalog`,
|
||||
`internal/llm`, `internal/profile`, `internal/prompt`,
|
||||
`internal/promptdef`, `internal/usecase`, and `internal/validate`;
|
||||
- embedded built-in profile assets and framework-package test fixtures;
|
||||
- the root framework contract tests and `testdata/framework`; and
|
||||
- `examples/go-library`, which is owned and maintained by Promptkit.
|
||||
|
||||
Tests remain with the behavior they protect. Scriptorium retains and adapts
|
||||
application configuration, adapter, formatting, HTTP containment, command, and
|
||||
representative executable-workflow coverage. It must not retain duplicate
|
||||
Promptkit unit or contract tests merely to exercise dependency internals.
|
||||
|
||||
The executable examples under `examples/` remain when they support
|
||||
Scriptorium's CLI or HTTP workflows. This includes application configuration,
|
||||
render scripts, HTTP requests, prompt/profile/schema inputs, and synthetic
|
||||
fixtures needed by those workflows. Their format semantics are owned by
|
||||
Promptkit documentation even though the files remain runnable Scriptorium
|
||||
assets.
|
||||
|
||||
## Observable Behavior To Preserve
|
||||
|
||||
Step 7 is an ownership and dependency cutover, not an intentional CLI or HTTP
|
||||
contract change. Subject to the deliberate removal of the public Scriptorium
|
||||
Go package, preserve:
|
||||
|
||||
- the `run`, `render`, and `serve` command names, flags, precedence, output
|
||||
destinations, summaries, exit classifications, and process behavior;
|
||||
- application configuration schema, discovery, validation, and defaults;
|
||||
- HTTP route, method, request and response shapes, strict decoding, media
|
||||
types, size enforcement, status codes, and stable error codes;
|
||||
- prompt, profile, execution-setting, request-override, and validation
|
||||
behavior as supplied by Promptkit `v0.1.0`;
|
||||
- built-in profile availability and custom-profile overlay behavior;
|
||||
- presence-aware zero-valued execution overrides;
|
||||
- the rule that explicit timeout zero disables only the generation deadline,
|
||||
while caller cancellation and the transport cap remain active;
|
||||
- public error identity as consumed by Scriptorium's CLI and HTTP mappings;
|
||||
- structured-output requests, output validation, and validation-failure
|
||||
classification;
|
||||
- strict external YAML and JSON decoding;
|
||||
- secret resolution and redaction; and
|
||||
- deterministic prepared-run formatting and output metadata.
|
||||
|
||||
Any observable application change discovered during implementation must be
|
||||
treated as a regression unless this roadmap, an accepted ADR, or a separately
|
||||
approved feature decision authorizes it.
|
||||
|
||||
## Architecture Enforcement
|
||||
|
||||
Repository checks must make the new dependency direction durable. They must
|
||||
detect production imports of:
|
||||
|
||||
- the removed Scriptorium root facade;
|
||||
- former Scriptorium framework package families; and
|
||||
- Promptkit `internal` packages.
|
||||
|
||||
The checks must cover the command, adapters, configuration, formatting, and
|
||||
other remaining production packages recursively without treating test-only
|
||||
fixtures as the application architecture. Go's own `internal` enforcement is
|
||||
useful but does not replace a repository-level check that explains the intended
|
||||
boundary.
|
||||
|
||||
The final tree must contain no duplicate framework directories, embedded
|
||||
built-in registry, copied Promptkit source, or dormant compatibility package.
|
||||
|
||||
## Documentation End State
|
||||
|
||||
Permanent Scriptorium documentation must describe the implemented slim
|
||||
application only:
|
||||
|
||||
- `README.md` presents the runnable CLI and HTTP application and contains no
|
||||
in-process Go-library positioning;
|
||||
- `docs/policy/architecture.md` defines Scriptorium as a Promptkit consumer and
|
||||
removes the single-repository framework architecture;
|
||||
- `docs/development.md` routes contributors through the retained application
|
||||
packages and the Promptkit contracts relevant to cross-project work;
|
||||
- `docs/internal/overview.md` inventories only the remaining Scriptorium
|
||||
packages;
|
||||
- CLI, HTTP, configuration, operations, adapter, source, and subprocess
|
||||
documents retain their application-owned contracts and mechanics;
|
||||
- the former Scriptorium Go-package consumer guide is retired, and incoming
|
||||
navigation directs Go framework consumers to Promptkit;
|
||||
- framework format and OpenAI-compatible behavior are linked to their
|
||||
canonical Promptkit owners rather than redefined in Scriptorium;
|
||||
- former runner, model-client, source, and other internal framework
|
||||
documentation is removed or reduced to the application-owned boundary where
|
||||
such a boundary still exists;
|
||||
- maintained examples and all documentation links refer only to files and
|
||||
commands that remain; and
|
||||
- future release and downstream-consumer work remains in the migration roadmap
|
||||
until Steps 8 and 9 are completed.
|
||||
|
||||
Promptkit's existing references to Scriptorium as a downstream application
|
||||
become accurate when this cutover lands. Step 7 does not otherwise change
|
||||
Promptkit's public API, implementation, version, or release documentation.
|
||||
|
||||
## Validation Expectations
|
||||
|
||||
Validation must demonstrate the resulting boundary and behavior, not the
|
||||
deleted implementation structure.
|
||||
|
||||
Scriptorium must pass:
|
||||
|
||||
- all ordinary and race-enabled Go tests;
|
||||
- `go vet` for all remaining packages;
|
||||
- an executable build to a temporary output path;
|
||||
- formatting, module-tidiness, whitespace, documentation-link, and repository
|
||||
hygiene checks;
|
||||
- maintained executable examples other than the retired Go-library example;
|
||||
- both maintained application configuration examples;
|
||||
- representative HTTP handler and restricted-artifact workflows; and
|
||||
- architecture checks proving the absence of the former framework dependency
|
||||
direction.
|
||||
|
||||
Validation must run with no active Go workspace and no module replacement.
|
||||
Module inspection must show the tagged Promptkit dependency selected directly.
|
||||
Promptkit must continue to pass its own documented validation independently;
|
||||
Scriptorium validation must not rely on the sibling Promptkit checkout.
|
||||
|
||||
Scriptorium's hosted validation and executable release workflow must remain
|
||||
capable of resolving the public Promptkit tag and building the command without
|
||||
cross-repository filesystem state.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
Step 7 does not:
|
||||
|
||||
- redesign or broaden Promptkit's public API;
|
||||
- preserve source compatibility for the former Scriptorium Go package;
|
||||
- migrate downstream repositories other than Scriptorium;
|
||||
- publish the breaking Scriptorium release or general downstream migration
|
||||
guide assigned to Steps 8 and 9;
|
||||
- change CLI, HTTP, configuration, prompt, profile, schema, validation, model,
|
||||
or timeout contracts;
|
||||
- move Scriptorium transport, deployment, or presentation policy into
|
||||
Promptkit;
|
||||
- add hosted CI or binary releases to Promptkit; or
|
||||
- retain duplicated code or documentation as a fallback.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
Step 7 is complete only when all of the following are true:
|
||||
|
||||
- Scriptorium declares and resolves
|
||||
`gitea.maximumdirect.net/eric/promptkit v0.1.0` as a direct dependency with no
|
||||
workspace, replacement, vendored copy, or unpublished revision.
|
||||
- The command, CLI adapter, HTTP adapter, restricted artifact reader, and
|
||||
prepared-run formatter use Promptkit public values and interfaces.
|
||||
- The Scriptorium module root exposes no Go package or compatibility facade.
|
||||
- Every Promptkit-owned framework package, built-in asset copy, framework test,
|
||||
fixture corpus, and Go-library example has been removed from Scriptorium.
|
||||
- Only application-owned configuration, adapter, formatting, transport,
|
||||
process, packaging, and executable-example responsibilities remain.
|
||||
- CLI, HTTP, configuration, containment, formatting, error-mapping, security,
|
||||
and representative end-to-end behavior remain protected by passing
|
||||
Scriptorium-owned tests.
|
||||
- Architecture checks prevent imports of the former local framework and
|
||||
Promptkit internals.
|
||||
- Scriptorium's current documentation describes the slim application and links
|
||||
to Promptkit for framework contracts without duplicating them.
|
||||
- Both repositories validate independently, and Scriptorium's full test, race,
|
||||
vet, build, example, link, module, and hygiene checks pass against the
|
||||
published Promptkit tag.
|
||||
- The working tree contains no generated binaries, temporary workspaces,
|
||||
replacement directives, credentials, or migration residue.
|
||||
|
||||
**Gate:** Scriptorium is a clean, independently buildable CLI and HTTP consumer
|
||||
of the published Promptkit module, contains no reusable framework
|
||||
implementation or public Go facade, preserves its application contracts, and
|
||||
is ready for the out-of-band downstream-consumer migrations in Step 8.
|
||||
343
engine.go
Normal file
343
engine.go
Normal file
@@ -0,0 +1,343 @@
|
||||
package scriptorium
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
artifactadapter "gitea.maximumdirect.net/eric/scriptorium/internal/artifact"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/llm"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/profile/builtin"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/prompt"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/usecase"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/validate"
|
||||
)
|
||||
|
||||
// ErrInvalidConfig indicates invalid public engine configuration.
|
||||
var ErrInvalidConfig = errors.New("invalid engine configuration")
|
||||
|
||||
var (
|
||||
ErrInvalidRequest = errors.New("invalid run request")
|
||||
ErrPromptNotFound = errors.New("prompt not found")
|
||||
ErrProfileNotFound = errors.New("profile not found")
|
||||
ErrProfileRequired = errors.New("profile selection is required")
|
||||
ErrPromptLoad = errors.New("failed to load prompt definition")
|
||||
ErrProfileLoad = errors.New("failed to load execution profile")
|
||||
ErrAPIKeyEnvMissing = errors.New("api_key_env points to an unset environment variable")
|
||||
ErrArtifactLoad = errors.New("failed to load artifact")
|
||||
ErrPromptRender = errors.New("failed to render prompt")
|
||||
ErrLLMGenerate = errors.New("failed to generate output")
|
||||
ErrValidation = errors.New("failed to validate output")
|
||||
)
|
||||
|
||||
// Engine prepares and runs Scriptorium prompt requests.
|
||||
type Engine struct {
|
||||
runner *usecase.Runner
|
||||
}
|
||||
|
||||
// Config configures a public Scriptorium engine.
|
||||
type Config struct {
|
||||
PromptDir string
|
||||
ProfileDir string
|
||||
SchemaDir string
|
||||
// Timeout is the transport-wide safety cap for the built-in LLM client
|
||||
// when HTTPClient is absent or has a non-positive timeout.
|
||||
Timeout time.Duration
|
||||
// HTTPClient is cloned for the built-in LLM client. Its positive Timeout
|
||||
// takes precedence over Config.Timeout as the transport-wide safety cap.
|
||||
HTTPClient *http.Client
|
||||
}
|
||||
|
||||
// Option customizes engine construction.
|
||||
type Option interface {
|
||||
apply(*engineOptions) error
|
||||
}
|
||||
|
||||
type optionFunc func(*engineOptions) error
|
||||
|
||||
func (f optionFunc) apply(options *engineOptions) error {
|
||||
return f(options)
|
||||
}
|
||||
|
||||
type engineOptions struct {
|
||||
llmClient llm.Client
|
||||
artifactReader artifactadapter.Reader
|
||||
promptDefs promptdef.Repository
|
||||
profiles profile.Repository
|
||||
memoryProfiles profile.Repository
|
||||
validator validate.Validator
|
||||
promptSource bool
|
||||
profileSource bool
|
||||
memorySource bool
|
||||
validatorSource bool
|
||||
artifactSource bool
|
||||
}
|
||||
|
||||
// WithLLMClient injects a custom LLM client for execution.
|
||||
func WithLLMClient(client LLMClient) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
if client == nil {
|
||||
return ErrInvalidConfig
|
||||
}
|
||||
options.llmClient = publicLLMClientAdapter{client: client}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// WithArtifactReader injects a reader for every input artifact reference.
|
||||
func WithArtifactReader(reader ArtifactReader) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
if reader == nil {
|
||||
return ErrInvalidConfig
|
||||
}
|
||||
options.artifactReader = publicArtifactReaderAdapter{reader: reader}
|
||||
options.artifactSource = true
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// WithPromptFS loads prompt definitions from fsys under root.
|
||||
//
|
||||
// The source uses the same strict prompt YAML rules as configured prompt
|
||||
// directories, and prompt content_file paths resolve within this source.
|
||||
func WithPromptFS(fsys fs.FS, root string) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
if fsys == nil {
|
||||
return ErrInvalidConfig
|
||||
}
|
||||
if strings.TrimSpace(root) == "" {
|
||||
return ErrInvalidConfig
|
||||
}
|
||||
options.promptDefs = promptdef.NewFSRepository(fsys, root)
|
||||
options.promptSource = true
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// WithPromptFile loads prompt definitions from the single prompt file at path.
|
||||
//
|
||||
// Relative prompt content_file paths resolve from the file's directory.
|
||||
func WithPromptFile(path string) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
fsys, root, err := fileSource(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
options.promptDefs = promptdef.NewFSRepository(fsys, root)
|
||||
options.promptSource = true
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// WithProfileFS loads execution profiles from fsys under root.
|
||||
//
|
||||
// Profiles from this source overlay built-in profiles. Profile YAML must use
|
||||
// api_key_env for environment-based credentials; raw API keys are rejected.
|
||||
func WithProfileFS(fsys fs.FS, root string) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
if fsys == nil {
|
||||
return ErrInvalidConfig
|
||||
}
|
||||
if strings.TrimSpace(root) == "" {
|
||||
return ErrInvalidConfig
|
||||
}
|
||||
options.profiles = profile.NewFSRepository(fsys, root)
|
||||
options.profileSource = true
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// WithProfileFile loads execution profiles from the single profile file at path.
|
||||
//
|
||||
// The profile overlays built-in profiles. Profile YAML must use api_key_env for
|
||||
// environment-based credentials; raw API keys are rejected.
|
||||
func WithProfileFile(path string) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
fsys, root, err := fileSource(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
options.profiles = profile.NewFSRepository(fsys, root)
|
||||
options.profileSource = true
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// WithProfiles configures in-memory profiles that take precedence over
|
||||
// configured profile files and built-in profiles.
|
||||
func WithProfiles(profiles ...Profile) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
repo, err := newMemoryProfileRepository(profiles)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
options.memoryProfiles = repo
|
||||
options.memorySource = true
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// WithSchemaFS loads JSON Schema documents from fsys under root.
|
||||
//
|
||||
// Prompt schema_path values resolve within this source when schema validation
|
||||
// or structured output is requested.
|
||||
func WithSchemaFS(fsys fs.FS, root string) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
if fsys == nil {
|
||||
return ErrInvalidConfig
|
||||
}
|
||||
if strings.TrimSpace(root) == "" {
|
||||
return ErrInvalidConfig
|
||||
}
|
||||
options.validator = validate.NewFSValidator(fsys, root)
|
||||
options.validatorSource = true
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// WithSchemaFile loads JSON Schema documents from the single schema file at path.
|
||||
//
|
||||
// Prompt schema_path values refer to the file's base name.
|
||||
func WithSchemaFile(path string) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
fsys, root, err := fileSource(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
options.validator = validate.NewFSValidator(fsys, root)
|
||||
options.validatorSource = true
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// NewEngine constructs an Engine using the same default internal components as
|
||||
// the CLI and HTTP adapters.
|
||||
func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
|
||||
var options engineOptions
|
||||
for _, opt := range opts {
|
||||
if opt == nil {
|
||||
continue
|
||||
}
|
||||
if err := opt.apply(&options); err != nil {
|
||||
return nil, fmt.Errorf("%w: %v", ErrInvalidConfig, err)
|
||||
}
|
||||
}
|
||||
|
||||
promptDefs := options.promptDefs
|
||||
if !options.promptSource {
|
||||
if strings.TrimSpace(cfg.PromptDir) == "" {
|
||||
return nil, fmt.Errorf("%w: prompt directory is required", ErrInvalidConfig)
|
||||
}
|
||||
promptDefs = promptdef.NewFilesystemRepository(cfg.PromptDir)
|
||||
}
|
||||
|
||||
profiles := builtin.NewRepositoryWithDirectory(cfg.ProfileDir)
|
||||
if options.profileSource {
|
||||
profiles = builtin.NewRepositoryWithPrimary(options.profiles)
|
||||
}
|
||||
if options.memorySource {
|
||||
profiles = profile.NewOverlayRepository(options.memoryProfiles, profiles)
|
||||
}
|
||||
|
||||
validator := options.validator
|
||||
if !options.validatorSource {
|
||||
schemaDir := cfg.SchemaDir
|
||||
if strings.TrimSpace(schemaDir) == "" {
|
||||
schemaDir = defaults.SchemaDirDefault
|
||||
}
|
||||
validator = validate.NewStandardValidator(schemaDir)
|
||||
}
|
||||
|
||||
llmClient := options.llmClient
|
||||
if llmClient == nil {
|
||||
var err error
|
||||
llmClient, err = llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
|
||||
Timeout: cfg.Timeout,
|
||||
HTTPClient: cfg.HTTPClient,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %v", ErrInvalidConfig, err)
|
||||
}
|
||||
}
|
||||
|
||||
artifacts := options.artifactReader
|
||||
if !options.artifactSource {
|
||||
artifacts = artifactadapter.NewCompositeReader()
|
||||
}
|
||||
|
||||
return &Engine{
|
||||
runner: usecase.NewRunner(
|
||||
promptDefs,
|
||||
profiles,
|
||||
artifacts,
|
||||
prompt.NewGoRenderer(),
|
||||
llmClient,
|
||||
validator,
|
||||
),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func fileSource(name string) (fs.FS, string, error) {
|
||||
cleanName := strings.TrimSpace(name)
|
||||
if cleanName == "" {
|
||||
return nil, "", ErrInvalidConfig
|
||||
}
|
||||
dir := filepath.Dir(cleanName)
|
||||
base := filepath.Base(cleanName)
|
||||
if base == "." || base == string(filepath.Separator) || strings.TrimSpace(base) == "" {
|
||||
return nil, "", ErrInvalidConfig
|
||||
}
|
||||
info, err := os.Stat(cleanName)
|
||||
if err != nil {
|
||||
return nil, "", fmt.Errorf("%w: failed to access source file %q: %v", ErrInvalidConfig, cleanName, err)
|
||||
}
|
||||
if info.IsDir() {
|
||||
return nil, "", fmt.Errorf("%w: source path %q must be a file", ErrInvalidConfig, cleanName)
|
||||
}
|
||||
return os.DirFS(dir), filepath.ToSlash(base), nil
|
||||
}
|
||||
|
||||
// Prepare resolves a prompt request without calling an LLM.
|
||||
func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, error) {
|
||||
if e == nil || e.runner == nil {
|
||||
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
|
||||
}
|
||||
|
||||
domainReq, err := toDomainRunRequest(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
|
||||
}
|
||||
|
||||
prepared, err := e.runner.Prepare(ctx, domainReq)
|
||||
if err != nil {
|
||||
return nil, mapPublicError(err)
|
||||
}
|
||||
return fromDomainPreparedRun(prepared), nil
|
||||
}
|
||||
|
||||
// Run executes a prompt request and returns the generated artifact and metadata.
|
||||
func (e *Engine) Run(ctx context.Context, req RunRequest) (*RunResult, error) {
|
||||
if e == nil || e.runner == nil {
|
||||
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
|
||||
}
|
||||
|
||||
domainReq, err := toDomainRunRequest(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
|
||||
}
|
||||
|
||||
result, err := e.runner.Run(ctx, domainReq)
|
||||
if err != nil {
|
||||
return nil, mapPublicError(err)
|
||||
}
|
||||
return fromDomainRunResult(result), nil
|
||||
}
|
||||
2559
engine_test.go
Normal file
2559
engine_test.go
Normal file
File diff suppressed because it is too large
Load Diff
60
errors.go
Normal file
60
errors.go
Normal file
@@ -0,0 +1,60 @@
|
||||
package scriptorium
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/usecase"
|
||||
)
|
||||
|
||||
func mapPublicError(err error) error {
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
publicErr := publicErrorFor(err)
|
||||
if publicErr == nil {
|
||||
return err
|
||||
}
|
||||
return fmt.Errorf("%w: %w", publicErr, err)
|
||||
}
|
||||
|
||||
func publicErrorFor(err error) error {
|
||||
switch {
|
||||
case errors.Is(err, promptdef.ErrPromptDefinitionNotFound):
|
||||
return ErrPromptNotFound
|
||||
case errors.Is(err, profile.ErrProfileNotFound):
|
||||
return ErrProfileNotFound
|
||||
case errors.Is(err, usecase.ErrProfileRequired):
|
||||
return errors.Join(ErrInvalidRequest, ErrProfileRequired)
|
||||
case errors.Is(err, usecase.ErrPromptLoad):
|
||||
return ErrPromptLoad
|
||||
case errors.Is(err, usecase.ErrProfileLoad):
|
||||
return ErrProfileLoad
|
||||
case errors.Is(err, promptdef.ErrInvalidYAML), errors.Is(err, promptdef.ErrInvalidPromptDefinition):
|
||||
return ErrPromptLoad
|
||||
case isProfileLoadCause(err):
|
||||
return ErrProfileLoad
|
||||
case errors.Is(err, usecase.ErrAPIKeyEnvMissing):
|
||||
return errors.Join(ErrInvalidRequest, ErrAPIKeyEnvMissing)
|
||||
case errors.Is(err, usecase.ErrArtifactLoad):
|
||||
return ErrArtifactLoad
|
||||
case errors.Is(err, usecase.ErrPromptRender):
|
||||
return ErrPromptRender
|
||||
case errors.Is(err, usecase.ErrLLMGenerate):
|
||||
return ErrLLMGenerate
|
||||
case errors.Is(err, usecase.ErrValidation):
|
||||
return ErrValidation
|
||||
case errors.Is(err, usecase.ErrInvalidRequest):
|
||||
return ErrInvalidRequest
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
func isProfileLoadCause(err error) bool {
|
||||
return errors.Is(err, profile.ErrInvalidYAML) ||
|
||||
errors.Is(err, profile.ErrInvalidProfile) ||
|
||||
errors.Is(err, profile.ErrRawAPIKeyNotAllowed)
|
||||
}
|
||||
13
examples/config.full.yml
Normal file
13
examples/config.full.yml
Normal file
@@ -0,0 +1,13 @@
|
||||
prompt_dir: ./examples/prompts
|
||||
profile_dir: ./examples/profiles
|
||||
schema_dir: ./examples/schemas
|
||||
|
||||
server:
|
||||
addr: 127.0.0.1:8080
|
||||
artifact_root: .
|
||||
max_request_bytes: 16777216
|
||||
max_artifact_bytes: 16777216
|
||||
max_response_bytes: 16777216
|
||||
|
||||
defaults:
|
||||
render_format: text
|
||||
@@ -1,9 +1,10 @@
|
||||
prompt_dir: ./prompts
|
||||
profile_dir: ./profiles
|
||||
schema_dir: ./schemas
|
||||
prompt_dir: ./examples/prompts
|
||||
profile_dir: ./examples/profiles
|
||||
schema_dir: ./examples/schemas
|
||||
|
||||
server:
|
||||
addr: :8080
|
||||
artifact_root: .
|
||||
|
||||
defaults:
|
||||
render_format: text
|
||||
|
||||
50
examples/go-library/prepare/main.go
Normal file
50
examples/go-library/prepare/main.go
Normal file
@@ -0,0 +1,50 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log"
|
||||
"os"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium"
|
||||
)
|
||||
|
||||
func main() {
|
||||
engine, err := scriptorium.NewEngine(scriptorium.Config{
|
||||
PromptDir: "./examples/prompts",
|
||||
ProfileDir: "./examples/profiles",
|
||||
SchemaDir: "./examples/schemas",
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
||||
PromptID: "generic.markdown_summary",
|
||||
Inputs: map[string]scriptorium.ArtifactRef{
|
||||
"transcript": scriptorium.File("./examples/fixtures/transcript.md"),
|
||||
"glossary": scriptorium.File("./examples/fixtures/glossary.yml"),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
summary := struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
SelectedProfileID string `json:"selected_profile_id"`
|
||||
Model string `json:"model"`
|
||||
MessageCount int `json:"message_count"`
|
||||
InputHashes map[string]string `json:"input_hashes"`
|
||||
}{
|
||||
PromptID: prepared.PromptID,
|
||||
SelectedProfileID: prepared.SelectedProfileID,
|
||||
Model: prepared.EffectiveModelParams.Model,
|
||||
MessageCount: len(prepared.Messages),
|
||||
InputHashes: prepared.InputHashes,
|
||||
}
|
||||
|
||||
if err := json.NewEncoder(os.Stdout).Encode(summary); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
}
|
||||
18
examples/http-run.json
Normal file
18
examples/http-run.json
Normal file
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"prompt_id": "generic.markdown_summary",
|
||||
"profile_id": "local-fast",
|
||||
"inputs": {
|
||||
"transcript": {
|
||||
"type": "file",
|
||||
"uri": "./examples/fixtures/transcript.md"
|
||||
},
|
||||
"glossary": {
|
||||
"type": "file",
|
||||
"uri": "./examples/fixtures/glossary.yml"
|
||||
}
|
||||
},
|
||||
"vars": {
|
||||
"session_date": "2026-05-04"
|
||||
},
|
||||
"include_raw_output": false
|
||||
}
|
||||
13
examples/render-markdown-summary.sh
Executable file
13
examples/render-markdown-summary.sh
Executable file
@@ -0,0 +1,13 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
|
||||
cd "$repo_root"
|
||||
|
||||
go run ./cmd/scriptorium render \
|
||||
--config ./examples/config.yml \
|
||||
--prompt generic.markdown_summary \
|
||||
--input transcript=./examples/fixtures/transcript.md \
|
||||
--input glossary=./examples/fixtures/glossary.yml \
|
||||
--format text
|
||||
51
formatting.go
Normal file
51
formatting.go
Normal file
@@ -0,0 +1,51 @@
|
||||
package scriptorium
|
||||
|
||||
import "fmt"
|
||||
|
||||
// String returns a concise request summary without exposing direct API keys.
|
||||
func (r RunRequest) String() string {
|
||||
return r.redactedString()
|
||||
}
|
||||
|
||||
// GoString returns a concise request summary without exposing direct API keys.
|
||||
func (r RunRequest) GoString() string {
|
||||
return r.redactedString()
|
||||
}
|
||||
|
||||
func (r RunRequest) redactedString() string {
|
||||
return fmt.Sprintf(
|
||||
"scriptorium.RunRequest{PromptID:%q PromptVersion:%q ProfileID:%q APIKeySet:%t Inputs:%d Vars:%d ExecutionSet:%t ValidationSet:%t Metadata:%d}",
|
||||
r.PromptID,
|
||||
r.PromptVersion,
|
||||
r.ProfileID,
|
||||
r.APIKey != "",
|
||||
len(r.Inputs),
|
||||
len(r.Vars),
|
||||
r.Execution != nil,
|
||||
r.Validation != nil,
|
||||
len(r.Metadata),
|
||||
)
|
||||
}
|
||||
|
||||
// String returns a concise request summary without exposing direct API keys or
|
||||
// rendered prompt content.
|
||||
func (r GenerateRequest) String() string {
|
||||
return r.redactedString()
|
||||
}
|
||||
|
||||
// GoString returns a concise request summary without exposing direct API keys or
|
||||
// rendered prompt content.
|
||||
func (r GenerateRequest) GoString() string {
|
||||
return r.redactedString()
|
||||
}
|
||||
|
||||
func (r GenerateRequest) redactedString() string {
|
||||
return fmt.Sprintf(
|
||||
"scriptorium.GenerateRequest{Messages:%d Model:%q APIKeySet:%t StructuredOutputSet:%t ExtraParams:%d}",
|
||||
len(r.Prompt.Messages),
|
||||
r.Target.Model,
|
||||
r.APIKey != "",
|
||||
r.StructuredOutput != nil,
|
||||
len(r.Target.ExtraParams),
|
||||
)
|
||||
}
|
||||
@@ -12,18 +12,11 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium"
|
||||
httpadapter "gitea.maximumdirect.net/eric/scriptorium/internal/adapter/http"
|
||||
artifactadapter "gitea.maximumdirect.net/eric/scriptorium/internal/artifact"
|
||||
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/llm"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/prompt"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/usecase"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/validate"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -33,8 +26,7 @@ const (
|
||||
)
|
||||
|
||||
const (
|
||||
errPromptDirRequired = "prompt directory is required; provide --prompt-dir or config.yml prompt_dir"
|
||||
errProfileDirRequired = "profile directory is required; provide --profile-dir or config.yml profile_dir"
|
||||
errPromptDirRequired = "prompt directory is required; provide --prompt-dir or config.yml prompt_dir"
|
||||
)
|
||||
|
||||
type runConfig struct {
|
||||
@@ -75,10 +67,26 @@ type renderConfig struct {
|
||||
type serveConfig struct {
|
||||
configPath string
|
||||
|
||||
addr string
|
||||
promptDir string
|
||||
profileDir string
|
||||
schemaDir string
|
||||
addr string
|
||||
promptDir string
|
||||
profileDir string
|
||||
schemaDir string
|
||||
artifactRoot string
|
||||
maxRequestBytes int64
|
||||
maxArtifactBytes int64
|
||||
maxResponseBytes int64
|
||||
}
|
||||
|
||||
type commonCommandSettings struct {
|
||||
promptDir string
|
||||
profileDir string
|
||||
schemaDir string
|
||||
serverAddr string
|
||||
artifactRoot string
|
||||
maxRequestBytes int64
|
||||
maxArtifactBytes int64
|
||||
maxResponseBytes int64
|
||||
defaultRenderFormat renderformat.PreparedRunOutputFormat
|
||||
}
|
||||
|
||||
type listFlag []string
|
||||
@@ -125,24 +133,13 @@ func runCommand(args []string, stdout, stderr io.Writer) int {
|
||||
return ExitRuntimeError
|
||||
}
|
||||
|
||||
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
|
||||
Timeout: defaults.LLMRequestTimeoutDefault,
|
||||
})
|
||||
engine, err := newEngine(cfg)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "llm client error: %v\n", err)
|
||||
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
|
||||
runner := usecase.NewRunner(
|
||||
promptdef.NewFilesystemRepository(cfg.promptDir),
|
||||
profile.NewFilesystemRepository(cfg.profileDir),
|
||||
artifactadapter.NewCompositeReader(),
|
||||
prompt.NewGoRenderer(),
|
||||
llmClient,
|
||||
validate.NewStandardValidator(cfg.schemaDir),
|
||||
)
|
||||
|
||||
res, runErr := runner.Run(context.Background(), req)
|
||||
res, runErr := engine.Run(context.Background(), req)
|
||||
if runErr != nil {
|
||||
fmt.Fprintf(stderr, "run error: %v\n", runErr)
|
||||
return ExitRuntimeError
|
||||
@@ -170,16 +167,13 @@ func renderCommand(args []string, stdout, stderr io.Writer) int {
|
||||
return ExitRuntimeError
|
||||
}
|
||||
|
||||
runner := usecase.NewRunner(
|
||||
promptdef.NewFilesystemRepository(cfg.promptDir),
|
||||
profile.NewFilesystemRepository(cfg.profileDir),
|
||||
artifactadapter.NewCompositeReader(),
|
||||
prompt.NewGoRenderer(),
|
||||
nil,
|
||||
validate.NewStandardValidator(cfg.schemaDir),
|
||||
)
|
||||
engine, err := newEngine(&cfg.runConfig)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
|
||||
prepared, prepErr := runner.Prepare(context.Background(), req)
|
||||
prepared, prepErr := engine.Prepare(context.Background(), req)
|
||||
if prepErr != nil {
|
||||
fmt.Fprintf(stderr, "render error: %v\n", prepErr)
|
||||
return ExitRuntimeError
|
||||
@@ -205,24 +199,26 @@ func serveCommand(args []string, stderr io.Writer) int {
|
||||
return ExitRuntimeError
|
||||
}
|
||||
|
||||
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
|
||||
Timeout: defaults.LLMRequestTimeoutDefault,
|
||||
})
|
||||
artifactReader, err := httpadapter.NewRestrictedArtifactReader(cfg.artifactRoot, cfg.maxArtifactBytes)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "llm client error: %v\n", err)
|
||||
fmt.Fprintf(stderr, "artifact root error: %v\n", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
|
||||
runner := usecase.NewRunner(
|
||||
promptdef.NewFilesystemRepository(cfg.promptDir),
|
||||
profile.NewFilesystemRepository(cfg.profileDir),
|
||||
artifactadapter.NewCompositeReader(),
|
||||
prompt.NewGoRenderer(),
|
||||
llmClient,
|
||||
validate.NewStandardValidator(cfg.schemaDir),
|
||||
)
|
||||
engine, err := newEngine(&runConfig{
|
||||
promptDir: cfg.promptDir,
|
||||
profileDir: cfg.profileDir,
|
||||
schemaDir: cfg.schemaDir,
|
||||
}, scriptorium.WithArtifactReader(artifactReader))
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
|
||||
h := httpadapter.NewHandler(runner)
|
||||
h := httpadapter.NewHandlerWithOptions(engine, httpadapter.HandlerOptions{
|
||||
MaxRequestBytes: cfg.maxRequestBytes,
|
||||
MaxResponseBytes: cfg.maxResponseBytes,
|
||||
})
|
||||
srv := &http.Server{
|
||||
Addr: cfg.addr,
|
||||
Handler: h,
|
||||
@@ -299,6 +295,10 @@ func parseServeArgs(args []string) (*serveConfig, error) {
|
||||
fs.StringVar(&cfg.promptDir, "prompt-dir", "", "directory containing prompt definition YAML files")
|
||||
fs.StringVar(&cfg.profileDir, "profile-dir", "", "directory containing execution profile YAML files")
|
||||
fs.StringVar(&cfg.schemaDir, "schema-dir", "", "base directory for validation schemas")
|
||||
fs.StringVar(&cfg.artifactRoot, "artifact-root", "", "base directory for HTTP file input artifacts")
|
||||
fs.Int64Var(&cfg.maxRequestBytes, "max-request-bytes", 0, "maximum HTTP request body bytes; 0 disables the limit")
|
||||
fs.Int64Var(&cfg.maxArtifactBytes, "max-artifact-bytes", 0, "maximum HTTP file artifact bytes; 0 disables the limit")
|
||||
fs.Int64Var(&cfg.maxResponseBytes, "max-response-bytes", 0, "maximum HTTP response body bytes; 0 disables the limit")
|
||||
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return nil, err
|
||||
@@ -307,31 +307,41 @@ func parseServeArgs(args []string) (*serveConfig, error) {
|
||||
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
|
||||
}
|
||||
|
||||
settings, err := resolveAppSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
||||
PromptDir: cfg.promptDirIfSet(fs),
|
||||
ProfileDir: cfg.profileDirIfSet(fs),
|
||||
SchemaDir: cfg.schemaDirIfSet(fs),
|
||||
ServerAddr: cfg.addrIfSet(fs),
|
||||
settings, err := resolveCommonSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
||||
PromptDir: cfg.promptDirIfSet(fs),
|
||||
ProfileDir: cfg.profileDirIfSet(fs),
|
||||
SchemaDir: cfg.schemaDirIfSet(fs),
|
||||
ServerAddr: cfg.addrIfSet(fs),
|
||||
ArtifactRoot: cfg.artifactRootIfSet(fs),
|
||||
MaxRequestBytes: cfg.maxRequestBytesIfSet(fs),
|
||||
MaxArtifactBytes: cfg.maxArtifactBytesIfSet(fs),
|
||||
MaxResponseBytes: cfg.maxResponseBytesIfSet(fs),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
cfg.promptDir = settings.PromptDir
|
||||
cfg.profileDir = settings.ProfileDir
|
||||
cfg.schemaDir = settings.SchemaDir
|
||||
cfg.addr = settings.ServerAddr
|
||||
cfg.promptDir = settings.promptDir
|
||||
cfg.profileDir = settings.profileDir
|
||||
cfg.schemaDir = settings.schemaDir
|
||||
cfg.addr = settings.serverAddr
|
||||
cfg.artifactRoot = settings.artifactRoot
|
||||
cfg.maxRequestBytes = settings.maxRequestBytes
|
||||
cfg.maxArtifactBytes = settings.maxArtifactBytes
|
||||
cfg.maxResponseBytes = settings.maxResponseBytes
|
||||
|
||||
if strings.TrimSpace(cfg.promptDir) == "" {
|
||||
return nil, errors.New(errPromptDirRequired)
|
||||
}
|
||||
if strings.TrimSpace(cfg.profileDir) == "" {
|
||||
return nil, errors.New(errProfileDirRequired)
|
||||
if err := validateRequiredLibraryDirs(cfg.promptDir); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
cfg.promptDir = filepath.Clean(cfg.promptDir)
|
||||
cfg.profileDir = filepath.Clean(cfg.profileDir)
|
||||
if strings.TrimSpace(cfg.profileDir) != "" {
|
||||
cfg.profileDir = filepath.Clean(cfg.profileDir)
|
||||
}
|
||||
cfg.schemaDir = filepath.Clean(cfg.schemaDir)
|
||||
if strings.TrimSpace(cfg.artifactRoot) != "" {
|
||||
cfg.artifactRoot = filepath.Clean(cfg.artifactRoot)
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
@@ -359,7 +369,7 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
|
||||
return fmt.Errorf("unexpected positional args: %v", fs.Args())
|
||||
}
|
||||
|
||||
settings, err := resolveAppSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
||||
settings, err := resolveCommonSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
||||
PromptDir: cfg.promptDirIfSet(fs),
|
||||
ProfileDir: cfg.profileDirIfSet(fs),
|
||||
SchemaDir: cfg.schemaDirIfSet(fs),
|
||||
@@ -368,16 +378,13 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
|
||||
return err
|
||||
}
|
||||
|
||||
cfg.promptDir = settings.PromptDir
|
||||
cfg.profileDir = settings.ProfileDir
|
||||
cfg.schemaDir = settings.SchemaDir
|
||||
cfg.defaultRenderFormat = settings.DefaultRenderFormat
|
||||
cfg.promptDir = settings.promptDir
|
||||
cfg.profileDir = settings.profileDir
|
||||
cfg.schemaDir = settings.schemaDir
|
||||
cfg.defaultRenderFormat = settings.defaultRenderFormat
|
||||
|
||||
if strings.TrimSpace(cfg.promptDir) == "" {
|
||||
return errors.New(errPromptDirRequired)
|
||||
}
|
||||
if strings.TrimSpace(cfg.profileDir) == "" {
|
||||
return errors.New(errProfileDirRequired)
|
||||
if err := validateRequiredLibraryDirs(cfg.promptDir); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(cfg.promptID) == "" {
|
||||
return errors.New("--prompt is required")
|
||||
@@ -386,7 +393,9 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
|
||||
return errors.New("at least one --input is required")
|
||||
}
|
||||
cfg.promptDir = filepath.Clean(cfg.promptDir)
|
||||
cfg.profileDir = filepath.Clean(cfg.profileDir)
|
||||
if strings.TrimSpace(cfg.profileDir) != "" {
|
||||
cfg.profileDir = filepath.Clean(cfg.profileDir)
|
||||
}
|
||||
if cfg.outputPath != "" {
|
||||
cfg.outputPath = filepath.Clean(cfg.outputPath)
|
||||
}
|
||||
@@ -449,6 +458,34 @@ func (c *serveConfig) addrIfSet(fs *flag.FlagSet) string {
|
||||
return ""
|
||||
}
|
||||
|
||||
func (c *serveConfig) artifactRootIfSet(fs *flag.FlagSet) string {
|
||||
if flagWasSet(fs, "artifact-root") {
|
||||
return c.artifactRoot
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (c *serveConfig) maxRequestBytesIfSet(fs *flag.FlagSet) *int64 {
|
||||
if flagWasSet(fs, "max-request-bytes") {
|
||||
return &c.maxRequestBytes
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *serveConfig) maxArtifactBytesIfSet(fs *flag.FlagSet) *int64 {
|
||||
if flagWasSet(fs, "max-artifact-bytes") {
|
||||
return &c.maxArtifactBytes
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *serveConfig) maxResponseBytesIfSet(fs *flag.FlagSet) *int64 {
|
||||
if flagWasSet(fs, "max-response-bytes") {
|
||||
return &c.maxResponseBytes
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func registerConfigPathFlag(fs *flag.FlagSet, target *string) {
|
||||
fs.StringVar(
|
||||
target,
|
||||
@@ -476,41 +513,81 @@ func resolveAppSettings(fs *flag.FlagSet, configPath string, overrides appconfig
|
||||
return merged, nil
|
||||
}
|
||||
|
||||
func buildRunRequestFromConfig(cfg *runConfig) (domain.RunRequest, error) {
|
||||
func resolveCommonSettings(fs *flag.FlagSet, configPath string, overrides appconfig.CLIOverrides) (commonCommandSettings, error) {
|
||||
settings, err := resolveAppSettings(fs, configPath, overrides)
|
||||
if err != nil {
|
||||
return commonCommandSettings{}, err
|
||||
}
|
||||
return commonCommandSettings{
|
||||
promptDir: settings.PromptDir,
|
||||
profileDir: settings.ProfileDir,
|
||||
schemaDir: settings.SchemaDir,
|
||||
serverAddr: settings.ServerAddr,
|
||||
artifactRoot: settings.ArtifactRoot,
|
||||
maxRequestBytes: settings.MaxRequestBytes,
|
||||
maxArtifactBytes: settings.MaxArtifactBytes,
|
||||
maxResponseBytes: settings.MaxResponseBytes,
|
||||
defaultRenderFormat: settings.DefaultRenderFormat,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func validateRequiredLibraryDirs(promptDir string) error {
|
||||
if strings.TrimSpace(promptDir) == "" {
|
||||
return errors.New(errPromptDirRequired)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func newEngine(cfg *runConfig, options ...scriptorium.Option) (*scriptorium.Engine, error) {
|
||||
return scriptorium.NewEngine(scriptorium.Config{
|
||||
PromptDir: cfg.promptDir,
|
||||
ProfileDir: cfg.profileDir,
|
||||
SchemaDir: cfg.schemaDir,
|
||||
}, options...)
|
||||
}
|
||||
|
||||
func buildRunRequestFromConfig(cfg *runConfig) (scriptorium.RunRequest, error) {
|
||||
inputMappings, err := parseMappings(cfg.inputRaw, false)
|
||||
if err != nil {
|
||||
return domain.RunRequest{}, fmt.Errorf("input parse error: %w", err)
|
||||
return scriptorium.RunRequest{}, fmt.Errorf("input parse error: %w", err)
|
||||
}
|
||||
|
||||
varMappings := map[string]string{}
|
||||
if len(cfg.varRaw) > 0 {
|
||||
varMappings, err = parseMappings(cfg.varRaw, false)
|
||||
if err != nil {
|
||||
return domain.RunRequest{}, fmt.Errorf("var parse error: %w", err)
|
||||
return scriptorium.RunRequest{}, fmt.Errorf("var parse error: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
inputs := make(map[string]domain.ArtifactRef, len(inputMappings))
|
||||
inputs := make(map[string]scriptorium.ArtifactRef, len(inputMappings))
|
||||
for name, path := range inputMappings {
|
||||
inputs[name] = domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: path}
|
||||
inputs[name] = scriptorium.File(path)
|
||||
}
|
||||
|
||||
var modelOverride *domain.ExecutionTarget
|
||||
var modelOverride *scriptorium.ExecutionTargetOverride
|
||||
if cfg.llmBaseURLSet || cfg.modelSet || cfg.temperatureSet || cfg.maxTokensSet || cfg.topPSet || cfg.apiKeyEnvSet || cfg.timeoutSet {
|
||||
modelOverride = &domain.ExecutionTarget{
|
||||
Endpoint: cfg.llmBaseURL,
|
||||
Model: cfg.model,
|
||||
Temperature: cfg.temperature,
|
||||
MaxTokens: cfg.maxTokens,
|
||||
TopP: cfg.topP,
|
||||
APIKeyEnv: cfg.apiKeyEnv,
|
||||
modelOverride = &scriptorium.ExecutionTargetOverride{
|
||||
Endpoint: cfg.llmBaseURL,
|
||||
Model: cfg.model,
|
||||
APIKeyEnv: cfg.apiKeyEnv,
|
||||
}
|
||||
if cfg.temperatureSet {
|
||||
modelOverride.Temperature = &cfg.temperature
|
||||
}
|
||||
if cfg.maxTokensSet {
|
||||
modelOverride.MaxTokens = &cfg.maxTokens
|
||||
}
|
||||
if cfg.topPSet {
|
||||
modelOverride.TopP = &cfg.topP
|
||||
}
|
||||
if cfg.timeoutSet {
|
||||
modelOverride.TimeoutSeconds = int(cfg.timeout.Seconds())
|
||||
timeoutSeconds := int(cfg.timeout.Seconds())
|
||||
modelOverride.TimeoutSeconds = &timeoutSeconds
|
||||
}
|
||||
}
|
||||
|
||||
return domain.RunRequest{
|
||||
return scriptorium.RunRequest{
|
||||
PromptID: cfg.promptID,
|
||||
ProfileID: cfg.profileID,
|
||||
Inputs: inputs,
|
||||
@@ -574,21 +651,21 @@ func writeOutput(stdout io.Writer, outputPath string, body []byte) error {
|
||||
return os.WriteFile(outputPath, body, 0644)
|
||||
}
|
||||
|
||||
func determineExitCode(runErr error, result *domain.RunResult) int {
|
||||
func determineExitCode(runErr error, result *scriptorium.RunResult) int {
|
||||
if runErr != nil {
|
||||
return ExitRuntimeError
|
||||
}
|
||||
if result != nil && result.Validation.Status == domain.ValidationFailed {
|
||||
if result != nil && result.Validation.Status == scriptorium.ValidationFailed {
|
||||
return ExitValidationFailed
|
||||
}
|
||||
return ExitOK
|
||||
}
|
||||
|
||||
func printSummary(stderr io.Writer, res *domain.RunResult) {
|
||||
func printSummary(stderr io.Writer, res *scriptorium.RunResult) {
|
||||
if res == nil {
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(stderr, "prompt=%s@%s selected_profile=%s model=%s validation=%s mode=%s validation_errors=%d prompt_hash=%s inputs=%d usage=%d/%d/%d\n",
|
||||
fmt.Fprintf(stderr, "prompt=%s@%s selected_profile=%s model=%s validation=%s mode=%s validation_errors=%d prompt_hash=%s inputs=%d usage=%d/%d/%d",
|
||||
res.PromptID,
|
||||
res.PromptVersion,
|
||||
res.SelectedProfileID,
|
||||
@@ -602,11 +679,15 @@ func printSummary(stderr io.Writer, res *domain.RunResult) {
|
||||
res.Usage.CompletionTokens,
|
||||
res.Usage.TotalTokens,
|
||||
)
|
||||
if res.Usage.CachedTokens != 0 || res.Usage.CacheWriteTokens != 0 {
|
||||
fmt.Fprintf(stderr, " cached_tokens=%d cache_write_tokens=%d", res.Usage.CachedTokens, res.Usage.CacheWriteTokens)
|
||||
}
|
||||
fmt.Fprintln(stderr)
|
||||
}
|
||||
|
||||
func printUsage(w io.Writer) {
|
||||
fmt.Fprintln(w, "usage: scriptorium <run|render|serve> ...")
|
||||
fmt.Fprintln(w, " run: scriptorium run [--config PATH] [--prompt-dir DIR] [--profile-dir DIR] --prompt ID --input name=path [--input ...] [--profile ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--var k=v] [--out path] [--timeout 10m]")
|
||||
fmt.Fprintln(w, " render: scriptorium render [--config PATH] [--prompt-dir DIR] [--profile-dir DIR] --prompt ID --input name=path [--input ...] [--profile ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--var k=v] [--format text|json] [--out path] [--timeout 10m]")
|
||||
fmt.Fprintf(w, " serve: scriptorium serve [--config PATH] [--addr %s] [--prompt-dir DIR] [--profile-dir DIR] [--schema-dir DIR]\n", defaults.HTTPAddrDefault)
|
||||
fmt.Fprintf(w, " serve: scriptorium serve [--config PATH] [--addr %s] [--prompt-dir DIR] [--profile-dir DIR] [--schema-dir DIR] [--artifact-root DIR] [--max-request-bytes N] [--max-artifact-bytes N] [--max-response-bytes N]\n", defaults.HTTPAddrDefault)
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
132
internal/adapter/dependency_test.go
Normal file
132
internal/adapter/dependency_test.go
Normal file
@@ -0,0 +1,132 @@
|
||||
package adapter_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const scriptoriumModulePath = "gitea.maximumdirect.net/eric/scriptorium"
|
||||
|
||||
var forbiddenFrameworkPackageRoots = []string{
|
||||
scriptoriumModulePath + "/internal/domain",
|
||||
scriptoriumModulePath + "/internal/usecase",
|
||||
scriptoriumModulePath + "/internal/promptdef",
|
||||
scriptoriumModulePath + "/internal/prompt",
|
||||
scriptoriumModulePath + "/internal/profile",
|
||||
scriptoriumModulePath + "/internal/validate",
|
||||
scriptoriumModulePath + "/internal/llm",
|
||||
scriptoriumModulePath + "/internal/artifact",
|
||||
}
|
||||
|
||||
type forbiddenFrameworkImport struct {
|
||||
filePath string
|
||||
importPath string
|
||||
}
|
||||
|
||||
func TestScriptoriumAdaptersUseOnlyPublicFrameworkBoundary(t *testing.T) {
|
||||
_, testFile, _, ok := runtime.Caller(0)
|
||||
if !ok {
|
||||
t.Fatal("locate dependency guard source")
|
||||
}
|
||||
|
||||
adapterDir := filepath.Dir(testFile)
|
||||
violations, err := findForbiddenFrameworkImports([]string{
|
||||
filepath.Join(adapterDir, "cli"),
|
||||
filepath.Join(adapterDir, "http"),
|
||||
filepath.Join(adapterDir, "..", "format"),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("scan framework imports: %v", err)
|
||||
}
|
||||
for _, violation := range violations {
|
||||
t.Errorf("%s imports forbidden framework package %s", violation.filePath, violation.importPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestForbiddenFrameworkImportScannerDetectsNestedPackageFamilies(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
nestedDir := filepath.Join(root, "nested", "adapter")
|
||||
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||
t.Fatalf("create nested source directory: %v", err)
|
||||
}
|
||||
|
||||
sourcePath := filepath.Join(nestedDir, "imports.go")
|
||||
source := `package nested
|
||||
|
||||
import (
|
||||
_ "gitea.maximumdirect.net/eric/scriptorium"
|
||||
_ "gitea.maximumdirect.net/eric/scriptorium/internal/profile/builtin"
|
||||
)
|
||||
`
|
||||
if err := os.WriteFile(sourcePath, []byte(source), 0o644); err != nil {
|
||||
t.Fatalf("write nested source fixture: %v", err)
|
||||
}
|
||||
|
||||
violations, err := findForbiddenFrameworkImports([]string{root})
|
||||
if err != nil {
|
||||
t.Fatalf("scan nested source fixture: %v", err)
|
||||
}
|
||||
if len(violations) != 1 {
|
||||
t.Fatalf("expected one forbidden import, got %#v", violations)
|
||||
}
|
||||
if violations[0].filePath != sourcePath {
|
||||
t.Fatalf("unexpected importing file: %q", violations[0].filePath)
|
||||
}
|
||||
wantImport := scriptoriumModulePath + "/internal/profile/builtin"
|
||||
if violations[0].importPath != wantImport {
|
||||
t.Fatalf("unexpected forbidden import: %q", violations[0].importPath)
|
||||
}
|
||||
}
|
||||
|
||||
func findForbiddenFrameworkImports(roots []string) ([]forbiddenFrameworkImport, error) {
|
||||
var violations []forbiddenFrameworkImport
|
||||
for _, root := range roots {
|
||||
err := filepath.WalkDir(root, func(path string, entry fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".go") || strings.HasSuffix(entry.Name(), "_test.go") {
|
||||
return nil
|
||||
}
|
||||
|
||||
file, err := parser.ParseFile(token.NewFileSet(), path, nil, parser.ImportsOnly)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse imports in %s: %w", path, err)
|
||||
}
|
||||
for _, imported := range file.Imports {
|
||||
importPath, err := strconv.Unquote(imported.Path.Value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse import path in %s: %w", path, err)
|
||||
}
|
||||
if isForbiddenFrameworkImport(importPath) {
|
||||
violations = append(violations, forbiddenFrameworkImport{
|
||||
filePath: path,
|
||||
importPath: importPath,
|
||||
})
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("walk source root %s: %w", root, err)
|
||||
}
|
||||
}
|
||||
return violations, nil
|
||||
}
|
||||
|
||||
func isForbiddenFrameworkImport(importPath string) bool {
|
||||
for _, root := range forbiddenFrameworkPackageRoots {
|
||||
if importPath == root || strings.HasPrefix(importPath, root+"/") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
165
internal/adapter/http/artifact_reader.go
Normal file
165
internal/adapter/http/artifact_reader.go
Normal file
@@ -0,0 +1,165 @@
|
||||
package httpadapter
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"mime"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium"
|
||||
)
|
||||
|
||||
var (
|
||||
ErrFileNotAllowed = errors.New("file artifact references are not allowed")
|
||||
ErrFileOutsideRoot = errors.New("file artifact path is outside artifact root")
|
||||
ErrFileTooLarge = errors.New("file artifact exceeds size limit")
|
||||
)
|
||||
|
||||
const fallbackArtifactContentType = "text/plain"
|
||||
|
||||
// NewRestrictedArtifactReader creates the HTTP artifact reader for a rooted
|
||||
// filesystem and optional byte limit. An empty root permits inline artifacts
|
||||
// but denies file references; a zero limit permits artifacts of any size.
|
||||
func NewRestrictedArtifactReader(root string, maxBytes int64) (scriptorium.ArtifactReader, error) {
|
||||
if maxBytes < 0 {
|
||||
return nil, fmt.Errorf("artifact size limit must be greater than or equal to 0")
|
||||
}
|
||||
|
||||
cleanRoot := strings.TrimSpace(root)
|
||||
if cleanRoot == "" {
|
||||
return &restrictedArtifactReader{maxBytes: maxBytes}, nil
|
||||
}
|
||||
absRoot, err := filepath.Abs(filepath.Clean(cleanRoot))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve artifact root: %w", err)
|
||||
}
|
||||
return &restrictedArtifactReader{root: absRoot, maxBytes: maxBytes}, nil
|
||||
}
|
||||
|
||||
type restrictedArtifactReader struct {
|
||||
root string
|
||||
maxBytes int64
|
||||
}
|
||||
|
||||
var _ scriptorium.ArtifactReader = (*restrictedArtifactReader)(nil)
|
||||
|
||||
func (r *restrictedArtifactReader) Read(ctx context.Context, ref scriptorium.ArtifactRef) (*scriptorium.Artifact, error) {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
default:
|
||||
}
|
||||
|
||||
switch ref.Type {
|
||||
case scriptorium.ArtifactRefInline:
|
||||
return readInlineArtifact(ref)
|
||||
case scriptorium.ArtifactRefFile:
|
||||
return r.readFileArtifact(ref)
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupported artifact reference type %q", ref.Type)
|
||||
}
|
||||
}
|
||||
|
||||
func readInlineArtifact(ref scriptorium.ArtifactRef) (*scriptorium.Artifact, error) {
|
||||
if ref.Body == "" {
|
||||
return nil, errors.New("inline artifact body is required")
|
||||
}
|
||||
|
||||
body := []byte(ref.Body)
|
||||
return &scriptorium.Artifact{
|
||||
ContentType: fallbackArtifactContentType,
|
||||
Body: body,
|
||||
Size: int64(len(body)),
|
||||
Hash: artifactHash(body),
|
||||
URI: ref.URI,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (r *restrictedArtifactReader) readFileArtifact(ref scriptorium.ArtifactRef) (*scriptorium.Artifact, error) {
|
||||
if ref.URI == "" {
|
||||
return nil, errors.New("file artifact path is required")
|
||||
}
|
||||
if r.root == "" {
|
||||
return nil, ErrFileNotAllowed
|
||||
}
|
||||
|
||||
path, err := r.resolveLexicalPath(ref.URI)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return readArtifactFile(path, r.maxBytes)
|
||||
}
|
||||
|
||||
// resolveLexicalPath checks cleaned path containment without resolving symlinks.
|
||||
func (r *restrictedArtifactReader) resolveLexicalPath(rawPath string) (string, error) {
|
||||
cleanPath := filepath.Clean(strings.TrimSpace(rawPath))
|
||||
candidate := cleanPath
|
||||
if !filepath.IsAbs(cleanPath) {
|
||||
candidate = filepath.Join(r.root, cleanPath)
|
||||
}
|
||||
|
||||
absCandidate, err := filepath.Abs(candidate)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("resolve artifact path: %w", err)
|
||||
}
|
||||
absCandidate = filepath.Clean(absCandidate)
|
||||
|
||||
rel, err := filepath.Rel(r.root, absCandidate)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("compare artifact path to root: %w", err)
|
||||
}
|
||||
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || filepath.IsAbs(rel) {
|
||||
return "", ErrFileOutsideRoot
|
||||
}
|
||||
return absCandidate, nil
|
||||
}
|
||||
|
||||
func readArtifactFile(path string, maxBytes int64) (*scriptorium.Artifact, error) {
|
||||
file, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read file %s: %w", path, err)
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
info, err := file.Stat()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to stat file %s: %w", path, err)
|
||||
}
|
||||
if maxBytes > 0 && info.Size() > maxBytes {
|
||||
return nil, ErrFileTooLarge
|
||||
}
|
||||
|
||||
var reader io.Reader = file
|
||||
if maxBytes > 0 {
|
||||
reader = io.LimitReader(file, maxBytes+1)
|
||||
}
|
||||
body, err := io.ReadAll(reader)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read file %s: %w", path, err)
|
||||
}
|
||||
if maxBytes > 0 && int64(len(body)) > maxBytes {
|
||||
return nil, ErrFileTooLarge
|
||||
}
|
||||
|
||||
contentType := mime.TypeByExtension(filepath.Ext(path))
|
||||
if contentType == "" {
|
||||
contentType = fallbackArtifactContentType
|
||||
}
|
||||
return &scriptorium.Artifact{
|
||||
Name: filepath.Base(path),
|
||||
ContentType: contentType,
|
||||
Body: body,
|
||||
URI: path,
|
||||
Size: int64(len(body)),
|
||||
Hash: artifactHash(body),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func artifactHash(body []byte) string {
|
||||
return fmt.Sprintf("%x", sha256.Sum256(body))
|
||||
}
|
||||
185
internal/adapter/http/artifact_reader_test.go
Normal file
185
internal/adapter/http/artifact_reader_test.go
Normal file
@@ -0,0 +1,185 @@
|
||||
package httpadapter
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"mime"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium"
|
||||
)
|
||||
|
||||
func TestRestrictedArtifactReaderReadsContainedFiles(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
outside := t.TempDir()
|
||||
inputPath := filepath.Join(root, "input.html")
|
||||
if err := os.WriteFile(inputPath, []byte("allowed"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(root, "input.unknown"), []byte("unknown type"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Mkdir(filepath.Join(root, "nested"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(outside, "secret.txt"), []byte("denied"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
expectedContentType := mime.TypeByExtension(filepath.Ext(inputPath))
|
||||
if expectedContentType == "" {
|
||||
t.Fatal("expected built-in HTML content type")
|
||||
}
|
||||
|
||||
reader, err := NewRestrictedArtifactReader(root, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("construct restricted reader: %v", err)
|
||||
}
|
||||
|
||||
for _, ref := range []scriptorium.ArtifactRef{
|
||||
{Type: scriptorium.ArtifactRefFile, URI: "nested/../input.html"},
|
||||
{Type: scriptorium.ArtifactRefFile, URI: inputPath},
|
||||
} {
|
||||
artifact, err := reader.Read(context.Background(), ref)
|
||||
if err != nil {
|
||||
t.Fatalf("read contained path %q: %v", ref.URI, err)
|
||||
}
|
||||
if artifact.Name != "input.html" || artifact.URI != inputPath || artifact.Size != int64(len("allowed")) || string(artifact.Body) != "allowed" {
|
||||
t.Fatalf("unexpected artifact metadata: %#v", artifact)
|
||||
}
|
||||
if artifact.ContentType != expectedContentType {
|
||||
t.Fatalf("unexpected artifact content type: got %q, want %q", artifact.ContentType, expectedContentType)
|
||||
}
|
||||
if artifact.Hash != artifactHash([]byte("allowed")) {
|
||||
t.Fatalf("unexpected artifact hash: %q", artifact.Hash)
|
||||
}
|
||||
}
|
||||
|
||||
artifact, err := reader.Read(context.Background(), scriptorium.File("input.unknown"))
|
||||
if err != nil {
|
||||
t.Fatalf("read unknown-extension path: %v", err)
|
||||
}
|
||||
if artifact.ContentType != fallbackArtifactContentType {
|
||||
t.Fatalf("unexpected fallback content type: %q", artifact.ContentType)
|
||||
}
|
||||
|
||||
for _, ref := range []scriptorium.ArtifactRef{
|
||||
{Type: scriptorium.ArtifactRefFile, URI: filepath.Join("..", filepath.Base(outside), "secret.txt")},
|
||||
{Type: scriptorium.ArtifactRefFile, URI: filepath.Join(outside, "secret.txt")},
|
||||
} {
|
||||
_, err := reader.Read(context.Background(), ref)
|
||||
if !errors.Is(err, ErrFileOutsideRoot) {
|
||||
t.Fatalf("expected ErrFileOutsideRoot for %q, got %v", ref.URI, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestrictedArtifactReaderFollowsSymlinkAfterLexicalCheck(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
outside := t.TempDir()
|
||||
target := filepath.Join(outside, "linked.txt")
|
||||
if err := os.WriteFile(target, []byte("linked outside root"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Symlink(target, filepath.Join(root, "linked.txt")); err != nil {
|
||||
t.Skipf("symlink creation unavailable: %v", err)
|
||||
}
|
||||
|
||||
reader, err := NewRestrictedArtifactReader(root, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("construct restricted reader: %v", err)
|
||||
}
|
||||
artifact, err := reader.Read(context.Background(), scriptorium.File("linked.txt"))
|
||||
if err != nil {
|
||||
t.Fatalf("read symlink inside root: %v", err)
|
||||
}
|
||||
if string(artifact.Body) != "linked outside root" {
|
||||
t.Fatalf("unexpected symlink artifact body: %q", artifact.Body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestrictedArtifactReaderWithoutRootDeniesFiles(t *testing.T) {
|
||||
reader, err := NewRestrictedArtifactReader("", 0)
|
||||
if err != nil {
|
||||
t.Fatalf("construct rootless reader: %v", err)
|
||||
}
|
||||
|
||||
artifact, err := reader.Read(context.Background(), scriptorium.Inline("inline"))
|
||||
if err != nil {
|
||||
t.Fatalf("read inline artifact: %v", err)
|
||||
}
|
||||
if artifact.ContentType != fallbackArtifactContentType || string(artifact.Body) != "inline" || artifact.Hash != artifactHash([]byte("inline")) {
|
||||
t.Fatalf("unexpected inline artifact: %#v", artifact)
|
||||
}
|
||||
|
||||
_, err = reader.Read(context.Background(), scriptorium.File("input.txt"))
|
||||
if !errors.Is(err, ErrFileNotAllowed) {
|
||||
t.Fatalf("expected ErrFileNotAllowed, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestrictedArtifactReaderEnforcesLimits(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(root, "exact.txt"), []byte("12345"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(root, "large.txt"), []byte("123456"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reader, err := NewRestrictedArtifactReader(root, 5)
|
||||
if err != nil {
|
||||
t.Fatalf("construct limited reader: %v", err)
|
||||
}
|
||||
artifact, err := reader.Read(context.Background(), scriptorium.File("exact.txt"))
|
||||
if err != nil || string(artifact.Body) != "12345" {
|
||||
t.Fatalf("expected exact-limit artifact, got %#v and %v", artifact, err)
|
||||
}
|
||||
_, err = reader.Read(context.Background(), scriptorium.File("large.txt"))
|
||||
if !errors.Is(err, ErrFileTooLarge) {
|
||||
t.Fatalf("expected ErrFileTooLarge, got %v", err)
|
||||
}
|
||||
|
||||
unlimited, err := NewRestrictedArtifactReader(root, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("construct unlimited reader: %v", err)
|
||||
}
|
||||
artifact, err = unlimited.Read(context.Background(), scriptorium.File("large.txt"))
|
||||
if err != nil || string(artifact.Body) != "123456" {
|
||||
t.Fatalf("expected unlimited artifact, got %#v and %v", artifact, err)
|
||||
}
|
||||
|
||||
if _, err := NewRestrictedArtifactReader(root, -1); err == nil {
|
||||
t.Fatal("expected negative limit to fail")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestrictedArtifactReaderRejectsCanceledAndMalformedReferences(t *testing.T) {
|
||||
reader, err := NewRestrictedArtifactReader(t.TempDir(), 0)
|
||||
if err != nil {
|
||||
t.Fatalf("construct reader: %v", err)
|
||||
}
|
||||
|
||||
canceledCtx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
for _, ref := range []scriptorium.ArtifactRef{
|
||||
scriptorium.Inline("input"),
|
||||
scriptorium.File("input.txt"),
|
||||
} {
|
||||
_, err := reader.Read(canceledCtx, ref)
|
||||
if !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("expected cancellation for %#v, got %v", ref, err)
|
||||
}
|
||||
}
|
||||
|
||||
for _, ref := range []scriptorium.ArtifactRef{
|
||||
{Type: scriptorium.ArtifactRefType("unsupported")},
|
||||
{Type: scriptorium.ArtifactRefInline},
|
||||
{Type: scriptorium.ArtifactRefFile},
|
||||
} {
|
||||
if _, err := reader.Read(context.Background(), ref); err == nil {
|
||||
t.Fatalf("expected malformed reference %#v to fail", ref)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -21,15 +21,16 @@ type inputRefDTO struct {
|
||||
}
|
||||
|
||||
type modelOverrideRequestDTO struct {
|
||||
Endpoint string `json:"endpoint,omitempty"`
|
||||
Model string `json:"model,omitempty"`
|
||||
Temperature float64 `json:"temperature,omitempty"`
|
||||
MaxTokens int `json:"max_tokens,omitempty"`
|
||||
TopP float64 `json:"top_p,omitempty"`
|
||||
TimeoutSeconds int `json:"timeout_seconds,omitempty"`
|
||||
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||
APIKeyEnv string `json:"api_key_env,omitempty"`
|
||||
ExtraParams map[string]string `json:"extra_params,omitempty"`
|
||||
Endpoint string `json:"endpoint,omitempty"`
|
||||
Model string `json:"model,omitempty"`
|
||||
Temperature *float64 `json:"temperature,omitempty"`
|
||||
MaxTokens *int `json:"max_tokens,omitempty"`
|
||||
TopP *float64 `json:"top_p,omitempty"`
|
||||
TimeoutSeconds *int `json:"timeout_seconds,omitempty"`
|
||||
ServiceTier string `json:"service_tier,omitempty"`
|
||||
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||
APIKeyEnv string `json:"api_key_env,omitempty"`
|
||||
ExtraParams map[string]any `json:"extra_params,omitempty"`
|
||||
}
|
||||
|
||||
type runResponseDTO struct {
|
||||
@@ -69,21 +70,24 @@ type metadataDTO struct {
|
||||
}
|
||||
|
||||
type modelParamsDTO struct {
|
||||
Endpoint string `json:"endpoint"`
|
||||
Model string `json:"model"`
|
||||
Temperature float64 `json:"temperature"`
|
||||
MaxTokens int `json:"max_tokens"`
|
||||
TopP float64 `json:"top_p"`
|
||||
TimeoutSeconds int `json:"timeout_seconds"`
|
||||
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||
APIKeyEnv string `json:"api_key_env,omitempty"`
|
||||
ExtraParams map[string]string `json:"extra_params,omitempty"`
|
||||
Endpoint string `json:"endpoint"`
|
||||
Model string `json:"model"`
|
||||
Temperature float64 `json:"temperature"`
|
||||
MaxTokens int `json:"max_tokens"`
|
||||
TopP float64 `json:"top_p"`
|
||||
TimeoutSeconds int `json:"timeout_seconds"`
|
||||
ServiceTier string `json:"service_tier,omitempty"`
|
||||
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||
APIKeyEnv string `json:"api_key_env,omitempty"`
|
||||
ExtraParams map[string]any `json:"extra_params,omitempty"`
|
||||
}
|
||||
|
||||
type tokenUsageDTO struct {
|
||||
PromptTokens int `json:"prompt_tokens"`
|
||||
CompletionTokens int `json:"completion_tokens"`
|
||||
TotalTokens int `json:"total_tokens"`
|
||||
CachedTokens int `json:"cached_tokens"`
|
||||
CacheWriteTokens int `json:"cache_write_tokens"`
|
||||
}
|
||||
|
||||
type validationDTO struct {
|
||||
|
||||
@@ -4,25 +4,37 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/usecase"
|
||||
"gitea.maximumdirect.net/eric/scriptorium"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||
)
|
||||
|
||||
type Runner interface {
|
||||
Run(ctx context.Context, req domain.RunRequest) (*domain.RunResult, error)
|
||||
Run(ctx context.Context, req scriptorium.RunRequest) (*scriptorium.RunResult, error)
|
||||
}
|
||||
|
||||
type Handler struct {
|
||||
runner Runner
|
||||
runner Runner
|
||||
options HandlerOptions
|
||||
}
|
||||
|
||||
type HandlerOptions struct {
|
||||
MaxRequestBytes int64
|
||||
MaxResponseBytes int64
|
||||
}
|
||||
|
||||
func NewHandler(runner Runner) *Handler {
|
||||
return &Handler{runner: runner}
|
||||
return NewHandlerWithOptions(runner, HandlerOptions{
|
||||
MaxRequestBytes: defaults.HTTPMaxRequestBytesDefault,
|
||||
MaxResponseBytes: defaults.HTTPMaxResponseBytesDefault,
|
||||
})
|
||||
}
|
||||
|
||||
func NewHandlerWithOptions(runner Runner, options HandlerOptions) *Handler {
|
||||
return &Handler{runner: runner, options: options}
|
||||
}
|
||||
|
||||
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -36,9 +48,26 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
|
||||
var req runRequestDTO
|
||||
dec := json.NewDecoder(r.Body)
|
||||
body := r.Body
|
||||
if h.options.MaxRequestBytes > 0 {
|
||||
body = http.MaxBytesReader(w, r.Body, h.options.MaxRequestBytes)
|
||||
}
|
||||
dec := json.NewDecoder(body)
|
||||
dec.DisallowUnknownFields()
|
||||
if err := dec.Decode(&req); err != nil {
|
||||
if isRequestTooLarge(err) {
|
||||
writeError(w, http.StatusRequestEntityTooLarge, "request_too_large", "request body is too large")
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusBadRequest, "invalid_json", "invalid JSON request body")
|
||||
return
|
||||
}
|
||||
var trailing any
|
||||
if err := dec.Decode(&trailing); err != io.EOF {
|
||||
if isRequestTooLarge(err) {
|
||||
writeError(w, http.StatusRequestEntityTooLarge, "request_too_large", "request body is too large")
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusBadRequest, "invalid_json", "invalid JSON request body")
|
||||
return
|
||||
}
|
||||
@@ -52,31 +81,21 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
|
||||
mappedInputs := make(map[string]domain.ArtifactRef, len(req.Inputs))
|
||||
mappedInputs := make(map[string]scriptorium.ArtifactRef, len(req.Inputs))
|
||||
for name, in := range req.Inputs {
|
||||
mappedInputs[name] = domain.ArtifactRef{
|
||||
Type: domain.ArtifactRefType(in.Type),
|
||||
mappedInputs[name] = scriptorium.ArtifactRef{
|
||||
Type: scriptorium.ArtifactRefType(in.Type),
|
||||
URI: in.URI,
|
||||
Body: in.Body,
|
||||
}
|
||||
}
|
||||
|
||||
var model *domain.ExecutionTarget
|
||||
var model *scriptorium.ExecutionTargetOverride
|
||||
if req.Model != nil {
|
||||
model = &domain.ExecutionTarget{
|
||||
Endpoint: req.Model.Endpoint,
|
||||
Model: req.Model.Model,
|
||||
Temperature: req.Model.Temperature,
|
||||
MaxTokens: req.Model.MaxTokens,
|
||||
TopP: req.Model.TopP,
|
||||
TimeoutSeconds: req.Model.TimeoutSeconds,
|
||||
ReasoningEffort: req.Model.ReasoningEffort,
|
||||
APIKeyEnv: req.Model.APIKeyEnv,
|
||||
ExtraParams: req.Model.ExtraParams,
|
||||
}
|
||||
model = executionTargetOverrideFromModelOverrideDTO(req.Model)
|
||||
}
|
||||
|
||||
res, err := h.runner.Run(r.Context(), domain.RunRequest{
|
||||
res, err := h.runner.Run(r.Context(), scriptorium.RunRequest{
|
||||
PromptID: req.PromptID,
|
||||
PromptVersion: req.PromptVersion,
|
||||
ProfileID: req.ProfileID,
|
||||
@@ -109,22 +128,14 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
SelectedProfileID: res.SelectedProfileID,
|
||||
ModelName: res.ModelName,
|
||||
Endpoint: res.Endpoint,
|
||||
ModelParams: modelParamsDTO{
|
||||
Endpoint: res.EffectiveModelParams.Endpoint,
|
||||
Model: res.EffectiveModelParams.Model,
|
||||
Temperature: res.EffectiveModelParams.Temperature,
|
||||
MaxTokens: res.EffectiveModelParams.MaxTokens,
|
||||
TopP: res.EffectiveModelParams.TopP,
|
||||
TimeoutSeconds: res.EffectiveModelParams.TimeoutSeconds,
|
||||
ReasoningEffort: res.EffectiveModelParams.ReasoningEffort,
|
||||
APIKeyEnv: res.EffectiveModelParams.APIKeyEnv,
|
||||
ExtraParams: res.EffectiveModelParams.ExtraParams,
|
||||
},
|
||||
InputHashes: res.InputHashes,
|
||||
ModelParams: modelParamsDTOFromExecutionTarget(res.EffectiveModelParams),
|
||||
InputHashes: res.InputHashes,
|
||||
Usage: tokenUsageDTO{
|
||||
PromptTokens: res.Usage.PromptTokens,
|
||||
CompletionTokens: res.Usage.CompletionTokens,
|
||||
TotalTokens: res.Usage.TotalTokens,
|
||||
CachedTokens: res.Usage.CachedTokens,
|
||||
CacheWriteTokens: res.Usage.CacheWriteTokens,
|
||||
},
|
||||
StartTime: res.StartTime,
|
||||
EndTime: res.EndTime,
|
||||
@@ -138,10 +149,43 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
raw := res.RawOutput
|
||||
resp.RawModelOutput = &raw
|
||||
}
|
||||
writeJSON(w, http.StatusOK, resp)
|
||||
writeLimitedJSON(w, http.StatusOK, resp, h.options.MaxResponseBytes)
|
||||
}
|
||||
|
||||
func mapValidation(v domain.ValidationResult) validationDTO {
|
||||
func executionTargetOverrideFromModelOverrideDTO(dto *modelOverrideRequestDTO) *scriptorium.ExecutionTargetOverride {
|
||||
if dto == nil {
|
||||
return nil
|
||||
}
|
||||
return &scriptorium.ExecutionTargetOverride{
|
||||
Endpoint: dto.Endpoint,
|
||||
Model: dto.Model,
|
||||
Temperature: dto.Temperature,
|
||||
MaxTokens: dto.MaxTokens,
|
||||
TopP: dto.TopP,
|
||||
TimeoutSeconds: dto.TimeoutSeconds,
|
||||
ServiceTier: dto.ServiceTier,
|
||||
ReasoningEffort: dto.ReasoningEffort,
|
||||
APIKeyEnv: dto.APIKeyEnv,
|
||||
ExtraParams: dto.ExtraParams,
|
||||
}
|
||||
}
|
||||
|
||||
func modelParamsDTOFromExecutionTarget(target scriptorium.ExecutionTarget) modelParamsDTO {
|
||||
return modelParamsDTO{
|
||||
Endpoint: target.Endpoint,
|
||||
Model: target.Model,
|
||||
Temperature: target.Temperature,
|
||||
MaxTokens: target.MaxTokens,
|
||||
TopP: target.TopP,
|
||||
TimeoutSeconds: target.TimeoutSeconds,
|
||||
ServiceTier: target.ServiceTier,
|
||||
ReasoningEffort: target.ReasoningEffort,
|
||||
APIKeyEnv: target.APIKeyEnv,
|
||||
ExtraParams: target.ExtraParams,
|
||||
}
|
||||
}
|
||||
|
||||
func mapValidation(v scriptorium.ValidationResult) validationDTO {
|
||||
return validationDTO{
|
||||
Status: string(v.Status),
|
||||
Mode: string(v.Mode),
|
||||
@@ -154,29 +198,31 @@ func mapValidation(v domain.ValidationResult) validationDTO {
|
||||
|
||||
func mapRunError(err error) (int, string, string) {
|
||||
switch {
|
||||
case errors.Is(err, promptdef.ErrPromptDefinitionNotFound):
|
||||
case errors.Is(err, scriptorium.ErrPromptNotFound):
|
||||
return http.StatusNotFound, "prompt_not_found", "prompt definition not found"
|
||||
case errors.Is(err, profile.ErrProfileNotFound):
|
||||
case errors.Is(err, scriptorium.ErrProfileNotFound):
|
||||
return http.StatusNotFound, "profile_not_found", "execution profile not found"
|
||||
case errors.Is(err, promptdef.ErrInvalidYAML), errors.Is(err, promptdef.ErrInvalidPromptDefinition):
|
||||
return http.StatusBadRequest, "prompt_load_failed", "failed to load prompt definition"
|
||||
case errors.Is(err, profile.ErrInvalidYAML), errors.Is(err, profile.ErrInvalidProfile):
|
||||
return http.StatusBadRequest, "profile_load_failed", "failed to load execution profile"
|
||||
case errors.Is(err, usecase.ErrInvalidRequest) && strings.Contains(err.Error(), "profile id is required either in request or prompt default_profile"):
|
||||
case errors.Is(err, scriptorium.ErrProfileRequired):
|
||||
return http.StatusBadRequest, "profile_required", "profile_id is required when prompt default_profile is not set"
|
||||
case errors.Is(err, usecase.ErrInvalidRequest) && strings.Contains(err.Error(), "api key environment variable"):
|
||||
case errors.Is(err, scriptorium.ErrAPIKeyEnvMissing):
|
||||
return http.StatusBadRequest, "api_key_env_missing", "api_key_env is set but the environment variable is missing"
|
||||
case errors.Is(err, usecase.ErrInvalidRequest):
|
||||
return http.StatusBadRequest, "invalid_request", "invalid run request"
|
||||
case errors.Is(err, usecase.ErrProfileLoad):
|
||||
case errors.Is(err, scriptorium.ErrPromptLoad):
|
||||
return http.StatusBadRequest, "prompt_load_failed", "failed to load prompt definition"
|
||||
case errors.Is(err, usecase.ErrArtifactLoad):
|
||||
case errors.Is(err, scriptorium.ErrProfileLoad):
|
||||
return http.StatusBadRequest, "profile_load_failed", "failed to load execution profile"
|
||||
case errors.Is(err, scriptorium.ErrInvalidRequest):
|
||||
return http.StatusBadRequest, "invalid_request", "invalid run request"
|
||||
case errors.Is(err, ErrFileNotAllowed), errors.Is(err, ErrFileOutsideRoot):
|
||||
return http.StatusBadRequest, "artifact_not_allowed", "file input artifact is not allowed"
|
||||
case errors.Is(err, ErrFileTooLarge):
|
||||
return http.StatusRequestEntityTooLarge, "artifact_too_large", "file input artifact is too large"
|
||||
case errors.Is(err, scriptorium.ErrArtifactLoad):
|
||||
return http.StatusBadRequest, "artifact_read_failed", "failed to read input artifact"
|
||||
case errors.Is(err, usecase.ErrPromptRender):
|
||||
case errors.Is(err, scriptorium.ErrPromptRender):
|
||||
return http.StatusBadRequest, "prompt_render_failed", "failed to render prompt"
|
||||
case errors.Is(err, usecase.ErrLLMGenerate):
|
||||
case errors.Is(err, scriptorium.ErrLLMGenerate):
|
||||
return http.StatusBadGateway, "llm_failed", "model generation request failed"
|
||||
case errors.Is(err, usecase.ErrValidation):
|
||||
case errors.Is(err, scriptorium.ErrValidation):
|
||||
return http.StatusInternalServerError, "validation_runtime_failed", "validation runtime failed"
|
||||
default:
|
||||
return http.StatusInternalServerError, "internal_error", "internal server error"
|
||||
@@ -184,9 +230,23 @@ func mapRunError(err error) (int, string, string) {
|
||||
}
|
||||
|
||||
func writeJSON(w http.ResponseWriter, status int, v any) {
|
||||
writeLimitedJSON(w, status, v, 0)
|
||||
}
|
||||
|
||||
func writeLimitedJSON(w http.ResponseWriter, status int, v any, maxBytes int64) {
|
||||
data, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "internal server error")
|
||||
return
|
||||
}
|
||||
data = append(data, '\n')
|
||||
if maxBytes > 0 && int64(len(data)) > maxBytes {
|
||||
writeError(w, http.StatusRequestEntityTooLarge, "response_too_large", "response body is too large")
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
_ = json.NewEncoder(w).Encode(v)
|
||||
_, _ = w.Write(data)
|
||||
}
|
||||
|
||||
func writeError(w http.ResponseWriter, status int, code, message string) {
|
||||
@@ -197,3 +257,8 @@ func writeError(w http.ResponseWriter, status int, code, message string) {
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
func isRequestTooLarge(err error) bool {
|
||||
var maxBytesErr *http.MaxBytesError
|
||||
return errors.As(err, &maxBytesErr)
|
||||
}
|
||||
|
||||
@@ -4,27 +4,26 @@ import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/usecase"
|
||||
"gitea.maximumdirect.net/eric/scriptorium"
|
||||
)
|
||||
|
||||
type fakeRunner struct {
|
||||
result *domain.RunResult
|
||||
result *scriptorium.RunResult
|
||||
err error
|
||||
last domain.RunRequest
|
||||
last scriptorium.RunRequest
|
||||
}
|
||||
|
||||
func (f *fakeRunner) Run(ctx context.Context, req domain.RunRequest) (*domain.RunResult, error) {
|
||||
func (f *fakeRunner) Run(ctx context.Context, req scriptorium.RunRequest) (*scriptorium.RunResult, error) {
|
||||
f.last = req
|
||||
if f.err != nil {
|
||||
return nil, f.err
|
||||
@@ -32,22 +31,60 @@ func (f *fakeRunner) Run(ctx context.Context, req domain.RunRequest) (*domain.Ru
|
||||
return f.result, nil
|
||||
}
|
||||
|
||||
func TestMaintainedHTTPRunExampleMatchesRequestContract(t *testing.T) {
|
||||
body, err := os.ReadFile(filepath.Join("..", "..", "..", "examples", "http-run.json"))
|
||||
if err != nil {
|
||||
t.Fatalf("read maintained HTTP request example: %v", err)
|
||||
}
|
||||
|
||||
runner := &fakeRunner{result: &scriptorium.RunResult{}}
|
||||
h := NewHandler(runner)
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected maintained HTTP request example to be accepted, got %d: %s", w.Code, w.Body.String())
|
||||
}
|
||||
|
||||
var invalidExample map[string]json.RawMessage
|
||||
if err := json.Unmarshal(body, &invalidExample); err != nil {
|
||||
t.Fatalf("decode maintained HTTP request example: %v", err)
|
||||
}
|
||||
invalidExample["unexpected"] = json.RawMessage(`true`)
|
||||
invalidBody, err := json.Marshal(invalidExample)
|
||||
if err != nil {
|
||||
t.Fatalf("encode structurally invalid request example: %v", err)
|
||||
}
|
||||
invalidReq := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewReader(invalidBody))
|
||||
invalidW := httptest.NewRecorder()
|
||||
h.ServeHTTP(invalidW, invalidReq)
|
||||
assertHTTPErrorCode(t, invalidW, http.StatusBadRequest, "invalid_json")
|
||||
}
|
||||
|
||||
type handlerLLMClient struct{}
|
||||
|
||||
func (handlerLLMClient) Generate(ctx context.Context, req scriptorium.GenerateRequest) (*scriptorium.GenerateResponse, error) {
|
||||
return &scriptorium.GenerateResponse{Content: "ok"}, nil
|
||||
}
|
||||
|
||||
func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||
start := time.Now().UTC()
|
||||
end := start.Add(2 * time.Second)
|
||||
const envName = "SCRIPTORIUM_API_KEY"
|
||||
const secret = "never-include-me"
|
||||
|
||||
r := &fakeRunner{result: &domain.RunResult{
|
||||
r := &fakeRunner{result: &scriptorium.RunResult{
|
||||
RunID: "11111111-1111-4111-8111-111111111111",
|
||||
Artifact: domain.Artifact{
|
||||
Artifact: scriptorium.Artifact{
|
||||
Name: "output",
|
||||
ContentType: "text/plain",
|
||||
Body: []byte("hello"),
|
||||
Size: 5,
|
||||
Hash: "abc",
|
||||
},
|
||||
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
|
||||
Validation: scriptorium.ValidationResult{Status: scriptorium.ValidationPassed, Mode: scriptorium.ValidationBasic, IsValid: true},
|
||||
PromptID: "prompt-1",
|
||||
PromptVersion: "1.0.0",
|
||||
PromptHash: "phash",
|
||||
@@ -55,21 +92,28 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||
SelectedProfileID: "exec-default",
|
||||
ModelName: "m1",
|
||||
Endpoint: "http://llm/v1",
|
||||
EffectiveModelParams: domain.ExecutionTarget{
|
||||
EffectiveModelParams: scriptorium.ExecutionTarget{
|
||||
Endpoint: "http://llm/v1",
|
||||
Model: "m1",
|
||||
Temperature: 0.2,
|
||||
MaxTokens: 42,
|
||||
TopP: 0.9,
|
||||
TimeoutSeconds: 120,
|
||||
ServiceTier: "priority",
|
||||
APIKeyEnv: envName,
|
||||
},
|
||||
InputHashes: map[string]string{"transcript": "h1"},
|
||||
Usage: domain.TokenUsage{PromptTokens: 1, CompletionTokens: 2, TotalTokens: 3},
|
||||
StartTime: start,
|
||||
EndTime: end,
|
||||
Duration: 2 * time.Second,
|
||||
RawOutput: "hello",
|
||||
Usage: scriptorium.TokenUsage{
|
||||
PromptTokens: 1,
|
||||
CompletionTokens: 2,
|
||||
TotalTokens: 3,
|
||||
CachedTokens: 4,
|
||||
CacheWriteTokens: 5,
|
||||
},
|
||||
StartTime: start,
|
||||
EndTime: end,
|
||||
Duration: 2 * time.Second,
|
||||
RawOutput: "hello",
|
||||
}}
|
||||
|
||||
h := NewHandler(r)
|
||||
@@ -81,7 +125,7 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||
"transcript": {"type": "file", "uri": "./t.md"}
|
||||
},
|
||||
"vars": {"k": "v"},
|
||||
"model": {"model": "gpt-x", "timeout_seconds": 120, "api_key_env": "SCRIPTORIUM_API_KEY"}
|
||||
"model": {"model": "gpt-x", "timeout_seconds": 120, "service_tier": "flex", "api_key_env": "SCRIPTORIUM_API_KEY"}
|
||||
}`)
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
@@ -110,10 +154,20 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||
if metadata["model_name"] != "m1" || metadata["endpoint"] != "http://llm/v1" {
|
||||
t.Fatalf("unexpected model metadata: name=%#v endpoint=%#v", metadata["model_name"], metadata["endpoint"])
|
||||
}
|
||||
usage := metadata["usage"].(map[string]any)
|
||||
if usage["prompt_tokens"] != float64(1) || usage["completion_tokens"] != float64(2) || usage["total_tokens"] != float64(3) {
|
||||
t.Fatalf("unexpected base usage metadata: %#v", usage)
|
||||
}
|
||||
if usage["cached_tokens"] != float64(4) || usage["cache_write_tokens"] != float64(5) {
|
||||
t.Fatalf("unexpected cache usage metadata: %#v", usage)
|
||||
}
|
||||
modelParams := metadata["model_params"].(map[string]any)
|
||||
if modelParams["api_key_env"] != envName {
|
||||
t.Fatalf("expected model_params.api_key_env=%q, got %#v", envName, modelParams["api_key_env"])
|
||||
}
|
||||
if modelParams["service_tier"] != "priority" {
|
||||
t.Fatalf("expected model_params.service_tier=priority, got %#v", modelParams["service_tier"])
|
||||
}
|
||||
if strings.Contains(w.Body.String(), secret) {
|
||||
t.Fatalf("response leaked raw API key value: %s", w.Body.String())
|
||||
}
|
||||
@@ -130,19 +184,121 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||
if r.last.Execution == nil || r.last.Execution.Model != "gpt-x" {
|
||||
t.Fatalf("expected model override, got %#v", r.last.Execution)
|
||||
}
|
||||
if r.last.Execution.TimeoutSeconds != 120 {
|
||||
if r.last.Execution.TimeoutSeconds == nil || *r.last.Execution.TimeoutSeconds != 120 {
|
||||
t.Fatalf("expected timeout_seconds override 120, got %#v", r.last.Execution)
|
||||
}
|
||||
if r.last.Execution.ServiceTier != "flex" {
|
||||
t.Fatalf("expected service_tier override flex, got %#v", r.last.Execution)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerInlineRefsWorkWithoutArtifactRoot(t *testing.T) {
|
||||
h := newArtifactRootHandler(t, "")
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||
"prompt_id":"p",
|
||||
"inputs":{"x":{"type":"inline","body":"inline body"}}
|
||||
}`))
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerFileRefsWithoutArtifactRootAreRejected(t *testing.T) {
|
||||
h := newArtifactRootHandler(t, "")
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||
"prompt_id":"p",
|
||||
"inputs":{"x":{"type":"file","uri":"input.txt"}}
|
||||
}`))
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
assertHTTPErrorCode(t, w, http.StatusBadRequest, "artifact_not_allowed")
|
||||
}
|
||||
|
||||
func TestHandlerFileRefsUnderArtifactRootWork(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(root, "input.txt"), []byte("allowed"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h := newArtifactRootHandler(t, root)
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||
"prompt_id":"p",
|
||||
"inputs":{"x":{"type":"file","uri":"input.txt"}}
|
||||
}`))
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerFileRefsAboveArtifactLimitAreRejected(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(root, "large.txt"), []byte("123456"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h := newArtifactRootHandlerWithLimit(t, root, 5)
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||
"prompt_id":"p",
|
||||
"inputs":{"x":{"type":"file","uri":"large.txt"}}
|
||||
}`))
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
assertHTTPErrorCode(t, w, http.StatusRequestEntityTooLarge, "artifact_too_large")
|
||||
}
|
||||
|
||||
func TestHandlerFileRefsOutsideArtifactRootAreRejected(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
outside := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(outside, "secret.txt"), []byte("denied"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h := newArtifactRootHandler(t, root)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
uri string
|
||||
}{
|
||||
{name: "relative traversal", uri: filepath.Join("..", filepath.Base(outside), "secret.txt")},
|
||||
{name: "absolute outside root", uri: filepath.Join(outside, "secret.txt")},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
body := fmt.Sprintf(`{
|
||||
"prompt_id":"p",
|
||||
"inputs":{"x":{"type":"file","uri":%q}}
|
||||
}`, tc.uri)
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(body))
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
assertHTTPErrorCode(t, w, http.StatusBadRequest, "artifact_not_allowed")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
|
||||
r := &fakeRunner{result: &domain.RunResult{
|
||||
Artifact: domain.Artifact{Body: []byte("ok")},
|
||||
r := &fakeRunner{result: &scriptorium.RunResult{
|
||||
Artifact: scriptorium.Artifact{Body: []byte("ok")},
|
||||
PromptID: "prompt-1",
|
||||
PromptVersion: "1.0.0",
|
||||
SelectedProfileID: "prompt-default",
|
||||
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
|
||||
EffectiveModelParams: domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||
Validation: scriptorium.ValidationResult{Status: scriptorium.ValidationPassed, Mode: scriptorium.ValidationBasic, IsValid: true},
|
||||
EffectiveModelParams: scriptorium.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||
}}
|
||||
h := NewHandler(r)
|
||||
|
||||
@@ -164,6 +320,265 @@ func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
|
||||
if metadata["selected_profile_id"] != "prompt-default" {
|
||||
t.Fatalf("expected selected_profile_id from result, got %#v", metadata["selected_profile_id"])
|
||||
}
|
||||
usage := metadata["usage"].(map[string]any)
|
||||
if usage["cached_tokens"] != float64(0) || usage["cache_write_tokens"] != float64(0) {
|
||||
t.Fatalf("expected zero cache usage fields to be included, got %#v", usage)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerModelOverrideMapsAllSupportedExecutionFields(t *testing.T) {
|
||||
r := &fakeRunner{result: &scriptorium.RunResult{
|
||||
Artifact: scriptorium.Artifact{Body: []byte("ok")},
|
||||
Validation: scriptorium.ValidationResult{Status: scriptorium.ValidationPassed, Mode: scriptorium.ValidationBasic, IsValid: true},
|
||||
EffectiveModelParams: scriptorium.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||
}}
|
||||
h := NewHandler(r)
|
||||
|
||||
reqBody := `{
|
||||
"prompt_id": "prompt-1",
|
||||
"inputs": {"transcript": {"type": "file", "uri": "./t.md"}},
|
||||
"model": {
|
||||
"endpoint": "http://override/v1",
|
||||
"model": "override-model",
|
||||
"temperature": 0.6,
|
||||
"max_tokens": 250,
|
||||
"top_p": 0.85,
|
||||
"timeout_seconds": 33,
|
||||
"service_tier": "flex",
|
||||
"reasoning_effort": "medium",
|
||||
"api_key_env": "SCRIPTORIUM_API_KEY",
|
||||
"extra_params": {"provider_option":"on"}
|
||||
}
|
||||
}`
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(reqBody))
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
if r.last.Execution == nil {
|
||||
t.Fatalf("expected execution override in run request")
|
||||
}
|
||||
got := r.last.Execution
|
||||
if got.Endpoint != "http://override/v1" ||
|
||||
got.Model != "override-model" ||
|
||||
got.ServiceTier != "flex" ||
|
||||
got.ReasoningEffort != "medium" ||
|
||||
got.APIKeyEnv != "SCRIPTORIUM_API_KEY" {
|
||||
t.Fatalf("unexpected mapped execution target: %+v", got)
|
||||
}
|
||||
if got.Temperature == nil || *got.Temperature != 0.6 {
|
||||
t.Fatalf("unexpected mapped temperature: %#v", got.Temperature)
|
||||
}
|
||||
if got.MaxTokens == nil || *got.MaxTokens != 250 {
|
||||
t.Fatalf("unexpected mapped max_tokens: %#v", got.MaxTokens)
|
||||
}
|
||||
if got.TopP == nil || *got.TopP != 0.85 {
|
||||
t.Fatalf("unexpected mapped top_p: %#v", got.TopP)
|
||||
}
|
||||
if got.TimeoutSeconds == nil || *got.TimeoutSeconds != 33 {
|
||||
t.Fatalf("unexpected mapped timeout_seconds: %#v", got.TimeoutSeconds)
|
||||
}
|
||||
if !reflect.DeepEqual(got.ExtraParams, map[string]any{"provider_option": "on"}) {
|
||||
t.Fatalf("unexpected mapped extra_params: %#v", got.ExtraParams)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerModelOverrideAcceptsJSONCompatibleExtraParams(t *testing.T) {
|
||||
r := &fakeRunner{result: &scriptorium.RunResult{
|
||||
Artifact: scriptorium.Artifact{Body: []byte("ok")},
|
||||
Validation: scriptorium.ValidationResult{Status: scriptorium.ValidationPassed, Mode: scriptorium.ValidationBasic, IsValid: true},
|
||||
EffectiveModelParams: scriptorium.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||
}}
|
||||
h := NewHandler(r)
|
||||
|
||||
reqBody := `{
|
||||
"prompt_id": "prompt-1",
|
||||
"inputs": {"transcript": {"type": "file", "uri": "./t.md"}},
|
||||
"model": {
|
||||
"extra_params": {
|
||||
"string_value": "enabled",
|
||||
"number_value": 42,
|
||||
"boolean_value": true,
|
||||
"object_value": {"nested": "value", "count": 2},
|
||||
"array_value": ["first", 3, false]
|
||||
}
|
||||
}
|
||||
}`
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(reqBody))
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
if r.last.Execution == nil {
|
||||
t.Fatal("expected execution override in run request")
|
||||
}
|
||||
want := map[string]any{
|
||||
"string_value": "enabled",
|
||||
"number_value": float64(42),
|
||||
"boolean_value": true,
|
||||
"object_value": map[string]any{"nested": "value", "count": float64(2)},
|
||||
"array_value": []any{"first", float64(3), false},
|
||||
}
|
||||
if !reflect.DeepEqual(r.last.Execution.ExtraParams, want) {
|
||||
t.Fatalf("unexpected mapped extra_params:\ngot=%#v\nwant=%#v", r.last.Execution.ExtraParams, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerModelOverrideExplicitZeroTemperatureMapsAsPresent(t *testing.T) {
|
||||
r := &fakeRunner{result: &scriptorium.RunResult{
|
||||
Artifact: scriptorium.Artifact{Body: []byte("ok")},
|
||||
Validation: scriptorium.ValidationResult{Status: scriptorium.ValidationPassed, Mode: scriptorium.ValidationBasic, IsValid: true},
|
||||
EffectiveModelParams: scriptorium.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1", Temperature: 0},
|
||||
}}
|
||||
h := NewHandler(r)
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||
"prompt_id": "prompt-1",
|
||||
"inputs": {"transcript": {"type": "file", "uri": "./t.md"}},
|
||||
"model": {"temperature": 0}
|
||||
}`))
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
if r.last.Execution == nil || r.last.Execution.Temperature == nil {
|
||||
t.Fatalf("expected temperature override to be present, got %#v", r.last.Execution)
|
||||
}
|
||||
if *r.last.Execution.Temperature != 0 {
|
||||
t.Fatalf("expected zero temperature override, got %v", *r.last.Execution.Temperature)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerModelOverrideOmittedTemperatureMapsAsAbsent(t *testing.T) {
|
||||
r := &fakeRunner{result: &scriptorium.RunResult{
|
||||
Artifact: scriptorium.Artifact{Body: []byte("ok")},
|
||||
Validation: scriptorium.ValidationResult{Status: scriptorium.ValidationPassed, Mode: scriptorium.ValidationBasic, IsValid: true},
|
||||
EffectiveModelParams: scriptorium.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1", Temperature: 0.7},
|
||||
}}
|
||||
h := NewHandler(r)
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||
"prompt_id": "prompt-1",
|
||||
"inputs": {"transcript": {"type": "file", "uri": "./t.md"}},
|
||||
"model": {"model": "override-model"}
|
||||
}`))
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
if r.last.Execution == nil {
|
||||
t.Fatal("expected model override")
|
||||
}
|
||||
if r.last.Execution.Temperature != nil {
|
||||
t.Fatalf("expected omitted temperature to remain absent, got %#v", r.last.Execution.Temperature)
|
||||
}
|
||||
|
||||
var resp map[string]any
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("invalid JSON response: %v", err)
|
||||
}
|
||||
metadata := resp["metadata"].(map[string]any)
|
||||
params := metadata["model_params"].(map[string]any)
|
||||
if params["temperature"] != 0.7 {
|
||||
t.Fatalf("expected effective profile/default temperature in response, got %#v", params["temperature"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerResponseMetadataModelParamsIncludesAllSupportedFields(t *testing.T) {
|
||||
r := &fakeRunner{result: &scriptorium.RunResult{
|
||||
Artifact: scriptorium.Artifact{
|
||||
Name: "output",
|
||||
ContentType: "text/plain",
|
||||
Body: []byte("ok"),
|
||||
Size: 2,
|
||||
Hash: "abc",
|
||||
},
|
||||
Validation: scriptorium.ValidationResult{Status: scriptorium.ValidationPassed, Mode: scriptorium.ValidationBasic, IsValid: true},
|
||||
EffectiveModelParams: scriptorium.ExecutionTarget{
|
||||
Endpoint: "http://llm/v1",
|
||||
Model: "gpt-test",
|
||||
Temperature: 0.4,
|
||||
MaxTokens: 321,
|
||||
TopP: 0.7,
|
||||
TimeoutSeconds: 45,
|
||||
ServiceTier: "priority",
|
||||
ReasoningEffort: "high",
|
||||
APIKeyEnv: "SCRIPTORIUM_API_KEY",
|
||||
ExtraParams: map[string]any{
|
||||
"provider_option": "on",
|
||||
"number_value": 42,
|
||||
"object_value": map[string]any{"nested": "value"},
|
||||
},
|
||||
},
|
||||
}}
|
||||
h := NewHandler(r)
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p","inputs":{"x":{"type":"file","uri":"a"}}}`))
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
|
||||
var resp map[string]any
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("invalid JSON response: %v", err)
|
||||
}
|
||||
metadata := resp["metadata"].(map[string]any)
|
||||
params := metadata["model_params"].(map[string]any)
|
||||
|
||||
if params["endpoint"] != "http://llm/v1" {
|
||||
t.Fatalf("unexpected endpoint: %#v", params["endpoint"])
|
||||
}
|
||||
if params["model"] != "gpt-test" {
|
||||
t.Fatalf("unexpected model: %#v", params["model"])
|
||||
}
|
||||
if params["temperature"] != 0.4 {
|
||||
t.Fatalf("unexpected temperature: %#v", params["temperature"])
|
||||
}
|
||||
if params["max_tokens"] != float64(321) {
|
||||
t.Fatalf("unexpected max_tokens: %#v", params["max_tokens"])
|
||||
}
|
||||
if params["top_p"] != 0.7 {
|
||||
t.Fatalf("unexpected top_p: %#v", params["top_p"])
|
||||
}
|
||||
if params["timeout_seconds"] != float64(45) {
|
||||
t.Fatalf("unexpected timeout_seconds: %#v", params["timeout_seconds"])
|
||||
}
|
||||
if params["service_tier"] != "priority" {
|
||||
t.Fatalf("unexpected service_tier: %#v", params["service_tier"])
|
||||
}
|
||||
if params["reasoning_effort"] != "high" {
|
||||
t.Fatalf("unexpected reasoning_effort: %#v", params["reasoning_effort"])
|
||||
}
|
||||
if params["api_key_env"] != "SCRIPTORIUM_API_KEY" {
|
||||
t.Fatalf("unexpected api_key_env: %#v", params["api_key_env"])
|
||||
}
|
||||
extraParams, ok := params["extra_params"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("expected extra_params object, got %#v", params["extra_params"])
|
||||
}
|
||||
if extraParams["provider_option"] != "on" {
|
||||
t.Fatalf("unexpected extra_params.provider_option: %#v", extraParams["provider_option"])
|
||||
}
|
||||
if extraParams["number_value"] != float64(42) {
|
||||
t.Fatalf("unexpected extra_params.number_value: %#v", extraParams["number_value"])
|
||||
}
|
||||
objectValue, ok := extraParams["object_value"].(map[string]any)
|
||||
if !ok || objectValue["nested"] != "value" {
|
||||
t.Fatalf("unexpected extra_params.object_value: %#v", extraParams["object_value"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerInvalidJSON(t *testing.T) {
|
||||
@@ -178,6 +593,69 @@ func TestHandlerInvalidJSON(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerRejectsTrailingJSON(t *testing.T) {
|
||||
h := NewHandler(&fakeRunner{})
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p","inputs":{"x":{"type":"file","uri":"a"}}} {}`))
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
assertHTTPErrorCode(t, w, http.StatusBadRequest, "invalid_json")
|
||||
}
|
||||
|
||||
func TestHandlerRequestTooLarge(t *testing.T) {
|
||||
h := NewHandlerWithOptions(&fakeRunner{}, HandlerOptions{MaxRequestBytes: 12})
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p","inputs":{"x":{"type":"file","uri":"a"}}}`))
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
assertHTTPErrorCode(t, w, http.StatusRequestEntityTooLarge, "request_too_large")
|
||||
}
|
||||
|
||||
func TestHandlerMalformedJSONBelowLimitStillBadRequest(t *testing.T) {
|
||||
h := NewHandlerWithOptions(&fakeRunner{}, HandlerOptions{MaxRequestBytes: 1024})
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString("{"))
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
assertHTTPErrorCode(t, w, http.StatusBadRequest, "invalid_json")
|
||||
}
|
||||
|
||||
func TestHandlerResponseTooLarge(t *testing.T) {
|
||||
h := NewHandlerWithOptions(&fakeRunner{result: &scriptorium.RunResult{
|
||||
Artifact: scriptorium.Artifact{Body: []byte(strings.Repeat("x", 128))},
|
||||
Validation: scriptorium.ValidationResult{Status: scriptorium.ValidationPassed, Mode: scriptorium.ValidationBasic, IsValid: true},
|
||||
EffectiveModelParams: scriptorium.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||
}}, HandlerOptions{MaxRequestBytes: 1024, MaxResponseBytes: 64})
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p","inputs":{"x":{"type":"file","uri":"a"}}}`))
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
assertHTTPErrorCode(t, w, http.StatusRequestEntityTooLarge, "response_too_large")
|
||||
}
|
||||
|
||||
func TestHandlerRawOutputDoesNotBypassResponseLimit(t *testing.T) {
|
||||
h := NewHandlerWithOptions(&fakeRunner{result: &scriptorium.RunResult{
|
||||
Artifact: scriptorium.Artifact{Body: []byte("ok")},
|
||||
RawOutput: strings.Repeat("raw", 80),
|
||||
Validation: scriptorium.ValidationResult{Status: scriptorium.ValidationPassed, Mode: scriptorium.ValidationBasic, IsValid: true},
|
||||
EffectiveModelParams: scriptorium.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||
}}, HandlerOptions{MaxRequestBytes: 1024, MaxResponseBytes: 128})
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||
"prompt_id":"p",
|
||||
"inputs":{"x":{"type":"file","uri":"a"}},
|
||||
"include_raw_output":true
|
||||
}`))
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
assertHTTPErrorCode(t, w, http.StatusRequestEntityTooLarge, "response_too_large")
|
||||
}
|
||||
|
||||
func TestHandlerMissingPromptID(t *testing.T) {
|
||||
h := NewHandler(&fakeRunner{})
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"inputs":{"x":{"type":"file","uri":"a"}}}`))
|
||||
@@ -198,7 +676,32 @@ func TestHandlerMissingPromptID(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerUsecaseErrorMapping(t *testing.T) {
|
||||
func TestHandlerReservedExtraParamsThroughEngineMapsToInvalidRequest(t *testing.T) {
|
||||
h := NewHandler(newHandlerEngineWithDefaultClient(t))
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
|
||||
"prompt_id":"p",
|
||||
"inputs":{"x":{"type":"inline","body":"input"}},
|
||||
"model":{"extra_params":{"model":"collision"}}
|
||||
}`))
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusBadRequest {
|
||||
t.Fatalf("expected 400, got %d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
var resp map[string]any
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("invalid JSON response: %v", err)
|
||||
}
|
||||
errBody := resp["error"].(map[string]any)
|
||||
if errBody["code"] != "invalid_request" {
|
||||
t.Fatalf("expected invalid_request code, got %#v", errBody["code"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerPublicErrorMapping(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
err error
|
||||
@@ -207,16 +710,20 @@ func TestHandlerUsecaseErrorMapping(t *testing.T) {
|
||||
message string
|
||||
avoidCause string
|
||||
}{
|
||||
{name: "prompt not found", err: wrap(usecase.ErrProfileLoad, promptdef.ErrPromptDefinitionNotFound), status: http.StatusNotFound, code: "prompt_not_found", message: "prompt definition not found"},
|
||||
{name: "prompt load invalid", err: wrap(usecase.ErrProfileLoad, promptdef.ErrInvalidPromptDefinition), status: http.StatusBadRequest, code: "prompt_load_failed", message: "failed to load prompt definition"},
|
||||
{name: "missing profile/default", err: wrap(usecase.ErrInvalidRequest, errors.New("profile id is required either in request or prompt default_profile")), status: http.StatusBadRequest, code: "profile_required", message: "profile_id is required when prompt default_profile is not set"},
|
||||
{name: "profile not found", err: wrap(usecase.ErrProfileLoad, profile.ErrProfileNotFound), status: http.StatusNotFound, code: "profile_not_found", message: "execution profile not found"},
|
||||
{name: "profile invalid", err: wrap(usecase.ErrProfileLoad, profile.ErrInvalidProfile), status: http.StatusBadRequest, code: "profile_load_failed", message: "failed to load execution profile"},
|
||||
{name: "api key env missing", err: wrap(usecase.ErrInvalidRequest, errors.New(`api key environment variable "SCRIPTORIUM_API_KEY" is not set`)), status: http.StatusBadRequest, code: "api_key_env_missing", message: "api_key_env is set but the environment variable is missing"},
|
||||
{name: "artifact", err: wrap(usecase.ErrArtifactLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "artifact_read_failed", message: "failed to read input artifact", avoidCause: "read failed"},
|
||||
{name: "prompt render", err: wrap(usecase.ErrPromptRender, fmt.Errorf("render failed")), status: http.StatusBadRequest, code: "prompt_render_failed", message: "failed to render prompt", avoidCause: "render failed"},
|
||||
{name: "llm", err: wrap(usecase.ErrLLMGenerate, fmt.Errorf("llm failed")), status: http.StatusBadGateway, code: "llm_failed", message: "model generation request failed", avoidCause: "llm failed"},
|
||||
{name: "validation runtime", err: wrap(usecase.ErrValidation, fmt.Errorf("validator broke")), status: http.StatusInternalServerError, code: "validation_runtime_failed", message: "validation runtime failed", avoidCause: "validator broke"},
|
||||
{name: "prompt not found", err: scriptorium.ErrPromptNotFound, status: http.StatusNotFound, code: "prompt_not_found", message: "prompt definition not found"},
|
||||
{name: "prompt load", err: wrap(scriptorium.ErrPromptLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "prompt_load_failed", message: "failed to load prompt definition", avoidCause: "read failed"},
|
||||
{name: "missing profile/default", err: wrap(scriptorium.ErrProfileRequired, scriptorium.ErrInvalidRequest), status: http.StatusBadRequest, code: "profile_required", message: "profile_id is required when prompt default_profile is not set"},
|
||||
{name: "profile not found", err: scriptorium.ErrProfileNotFound, status: http.StatusNotFound, code: "profile_not_found", message: "execution profile not found"},
|
||||
{name: "profile load", err: wrap(scriptorium.ErrProfileLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "profile_load_failed", message: "failed to load execution profile", avoidCause: "read failed"},
|
||||
{name: "api key env missing", err: wrap(scriptorium.ErrAPIKeyEnvMissing, scriptorium.ErrInvalidRequest), status: http.StatusBadRequest, code: "api_key_env_missing", message: "api_key_env is set but the environment variable is missing"},
|
||||
{name: "invalid request", err: scriptorium.ErrInvalidRequest, status: http.StatusBadRequest, code: "invalid_request", message: "invalid run request"},
|
||||
{name: "file denied", err: ErrFileNotAllowed, status: http.StatusBadRequest, code: "artifact_not_allowed", message: "file input artifact is not allowed"},
|
||||
{name: "file outside root", err: ErrFileOutsideRoot, status: http.StatusBadRequest, code: "artifact_not_allowed", message: "file input artifact is not allowed"},
|
||||
{name: "file too large", err: ErrFileTooLarge, status: http.StatusRequestEntityTooLarge, code: "artifact_too_large", message: "file input artifact is too large"},
|
||||
{name: "artifact", err: wrap(scriptorium.ErrArtifactLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "artifact_read_failed", message: "failed to read input artifact", avoidCause: "read failed"},
|
||||
{name: "prompt render", err: wrap(scriptorium.ErrPromptRender, fmt.Errorf("render failed")), status: http.StatusBadRequest, code: "prompt_render_failed", message: "failed to render prompt", avoidCause: "render failed"},
|
||||
{name: "llm", err: wrap(scriptorium.ErrLLMGenerate, fmt.Errorf("llm failed")), status: http.StatusBadGateway, code: "llm_failed", message: "model generation request failed", avoidCause: "llm failed"},
|
||||
{name: "validation runtime", err: wrap(scriptorium.ErrValidation, fmt.Errorf("validator broke")), status: http.StatusInternalServerError, code: "validation_runtime_failed", message: "validation runtime failed", avoidCause: "validator broke"},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
@@ -269,12 +776,12 @@ func TestHandlerRawAPIKeyRejectedByStrictJSON(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestHandlerValidationFailureStillSuccessAndRawOutputOptIn(t *testing.T) {
|
||||
h := NewHandler(&fakeRunner{result: &domain.RunResult{
|
||||
Artifact: domain.Artifact{Body: []byte("bad json")},
|
||||
h := NewHandler(&fakeRunner{result: &scriptorium.RunResult{
|
||||
Artifact: scriptorium.Artifact{Body: []byte("bad json")},
|
||||
RawOutput: "bad json",
|
||||
Validation: domain.ValidationResult{
|
||||
Status: domain.ValidationFailed,
|
||||
Mode: domain.ValidationJSON,
|
||||
Validation: scriptorium.ValidationResult{
|
||||
Status: scriptorium.ValidationFailed,
|
||||
Mode: scriptorium.ValidationJSON,
|
||||
Errors: []string{"invalid JSON"},
|
||||
},
|
||||
}})
|
||||
@@ -318,3 +825,82 @@ func TestHandlerValidationFailureStillSuccessAndRawOutputOptIn(t *testing.T) {
|
||||
func wrap(stage error, cause error) error {
|
||||
return fmt.Errorf("%w: %w", stage, cause)
|
||||
}
|
||||
|
||||
func newArtifactRootHandler(t *testing.T, root string) *Handler {
|
||||
t.Helper()
|
||||
|
||||
return newArtifactRootHandlerWithLimit(t, root, 0)
|
||||
}
|
||||
|
||||
func newArtifactRootHandlerWithLimit(t *testing.T, root string, maxArtifactBytes int64) *Handler {
|
||||
t.Helper()
|
||||
|
||||
reader, err := NewRestrictedArtifactReader(root, maxArtifactBytes)
|
||||
if err != nil {
|
||||
t.Fatalf("expected restricted artifact reader: %v", err)
|
||||
}
|
||||
return NewHandler(newHandlerEngine(t, scriptorium.WithArtifactReader(reader)))
|
||||
}
|
||||
|
||||
func newHandlerEngine(t *testing.T, options ...scriptorium.Option) *scriptorium.Engine {
|
||||
t.Helper()
|
||||
|
||||
return newHandlerEngineWithOptions(t, append(options, scriptorium.WithLLMClient(handlerLLMClient{}))...)
|
||||
}
|
||||
|
||||
func newHandlerEngineWithDefaultClient(t *testing.T, options ...scriptorium.Option) *scriptorium.Engine {
|
||||
t.Helper()
|
||||
|
||||
return newHandlerEngineWithOptions(t, options...)
|
||||
}
|
||||
|
||||
func newHandlerEngineWithOptions(t *testing.T, options ...scriptorium.Option) *scriptorium.Engine {
|
||||
t.Helper()
|
||||
|
||||
promptDir := t.TempDir()
|
||||
profileDir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(promptDir, "prompt.yaml"), []byte(`id: p
|
||||
version: "1"
|
||||
default_profile: exec
|
||||
messages:
|
||||
- role: user
|
||||
content: "hi"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
repair_attempts: 0
|
||||
`), 0o644); err != nil {
|
||||
t.Fatalf("write prompt fixture: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte(`id: exec
|
||||
endpoint: http://example.invalid/v1
|
||||
model: model
|
||||
`), 0o644); err != nil {
|
||||
t.Fatalf("write profile fixture: %v", err)
|
||||
}
|
||||
|
||||
engine, err := scriptorium.NewEngine(scriptorium.Config{
|
||||
PromptDir: promptDir,
|
||||
ProfileDir: profileDir,
|
||||
}, options...)
|
||||
if err != nil {
|
||||
t.Fatalf("construct public engine: %v", err)
|
||||
}
|
||||
return engine
|
||||
}
|
||||
|
||||
func assertHTTPErrorCode(t *testing.T, w *httptest.ResponseRecorder, status int, code string) {
|
||||
t.Helper()
|
||||
|
||||
if w.Code != status {
|
||||
t.Fatalf("expected %d, got %d body=%s", status, w.Code, w.Body.String())
|
||||
}
|
||||
var resp map[string]any
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("invalid JSON response: %v", err)
|
||||
}
|
||||
errBody := resp["error"].(map[string]any)
|
||||
if errBody["code"] != code {
|
||||
t.Fatalf("expected code %q, got %#v", code, errBody["code"])
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,11 +5,13 @@ import (
|
||||
"crypto/sha256"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||
"io"
|
||||
"mime"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||
)
|
||||
|
||||
var (
|
||||
@@ -26,7 +28,7 @@ type Reader interface {
|
||||
// CompositeReader routes artifact resolution based on the reference type.
|
||||
type CompositeReader struct {
|
||||
inlineReader *inlineReader
|
||||
fileReader *fileReader
|
||||
fileReader Reader
|
||||
}
|
||||
|
||||
func NewCompositeReader() Reader {
|
||||
@@ -89,21 +91,31 @@ func (r *fileReader) Read(ctx context.Context, ref domain.ArtifactRef) (*domain.
|
||||
return nil, ErrMissingFilePath
|
||||
}
|
||||
|
||||
data, err := os.ReadFile(ref.URI)
|
||||
return readFileArtifact(ref.URI)
|
||||
}
|
||||
|
||||
func readFileArtifact(path string) (*domain.Artifact, error) {
|
||||
file, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read file %s: %w", ref.URI, err)
|
||||
return nil, fmt.Errorf("failed to read file %s: %w", path, err)
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
data, err := io.ReadAll(file)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read file %s: %w", path, err)
|
||||
}
|
||||
|
||||
contentType := mime.TypeByExtension(filepath.Ext(ref.URI))
|
||||
contentType := mime.TypeByExtension(filepath.Ext(path))
|
||||
if contentType == "" {
|
||||
contentType = defaults.ContentTypeTextPlain
|
||||
}
|
||||
|
||||
return &domain.Artifact{
|
||||
Name: filepath.Base(ref.URI),
|
||||
Name: filepath.Base(path),
|
||||
ContentType: contentType,
|
||||
Body: data,
|
||||
URI: ref.URI,
|
||||
URI: path,
|
||||
Size: int64(len(data)),
|
||||
Hash: fmt.Sprintf("%x", sha256.Sum256(data)),
|
||||
}, nil
|
||||
|
||||
@@ -46,8 +46,8 @@ func TestCompositeReader_Read(t *testing.T) {
|
||||
|
||||
t.Run("unsupported ref type", func(t *testing.T) {
|
||||
ref := domain.ArtifactRef{
|
||||
Type: domain.ArtifactRefS3,
|
||||
URI: "s3://bucket/key",
|
||||
Type: domain.ArtifactRefType("unsupported"),
|
||||
URI: "unsupported://bucket/key",
|
||||
}
|
||||
_, err := reader.Read(ctx, ref)
|
||||
if !errors.Is(err, ErrUnsupportedRefType) {
|
||||
|
||||
@@ -40,7 +40,11 @@ type Config struct {
|
||||
}
|
||||
|
||||
type ServerConfig struct {
|
||||
Addr string `yaml:"addr"`
|
||||
Addr string `yaml:"addr"`
|
||||
ArtifactRoot string `yaml:"artifact_root"`
|
||||
MaxRequestBytes *int64 `yaml:"max_request_bytes"`
|
||||
MaxArtifactBytes *int64 `yaml:"max_artifact_bytes"`
|
||||
MaxResponseBytes *int64 `yaml:"max_response_bytes"`
|
||||
}
|
||||
|
||||
type DefaultsConfig struct {
|
||||
@@ -53,16 +57,24 @@ type AppSettings struct {
|
||||
ProfileDir string
|
||||
SchemaDir string
|
||||
ServerAddr string
|
||||
ArtifactRoot string
|
||||
MaxRequestBytes int64
|
||||
MaxArtifactBytes int64
|
||||
MaxResponseBytes int64
|
||||
DefaultRenderFormat renderformat.PreparedRunOutputFormat
|
||||
}
|
||||
|
||||
// CLIOverrides can be applied after config load to enforce precedence.
|
||||
type CLIOverrides struct {
|
||||
PromptDir string
|
||||
ProfileDir string
|
||||
SchemaDir string
|
||||
ServerAddr string
|
||||
RenderFormat string
|
||||
PromptDir string
|
||||
ProfileDir string
|
||||
SchemaDir string
|
||||
ServerAddr string
|
||||
ArtifactRoot string
|
||||
MaxRequestBytes *int64
|
||||
MaxArtifactBytes *int64
|
||||
MaxResponseBytes *int64
|
||||
RenderFormat string
|
||||
}
|
||||
|
||||
// BuiltInDefaults returns compile-time application defaults.
|
||||
@@ -70,6 +82,9 @@ func BuiltInDefaults() AppSettings {
|
||||
return AppSettings{
|
||||
SchemaDir: defaults.SchemaDirDefault,
|
||||
ServerAddr: defaults.HTTPAddrDefault,
|
||||
MaxRequestBytes: defaults.HTTPMaxRequestBytesDefault,
|
||||
MaxArtifactBytes: defaults.HTTPMaxArtifactBytesDefault,
|
||||
MaxResponseBytes: defaults.HTTPMaxResponseBytesDefault,
|
||||
DefaultRenderFormat: renderformat.DefaultPreparedRunOutputFormat,
|
||||
}
|
||||
}
|
||||
@@ -142,6 +157,27 @@ func ApplyCLIOverrides(base AppSettings, overrides CLIOverrides) (AppSettings, e
|
||||
if v := strings.TrimSpace(overrides.ServerAddr); v != "" {
|
||||
out.ServerAddr = v
|
||||
}
|
||||
if v := strings.TrimSpace(overrides.ArtifactRoot); v != "" {
|
||||
out.ArtifactRoot = filepath.Clean(v)
|
||||
}
|
||||
if overrides.MaxRequestBytes != nil {
|
||||
if *overrides.MaxRequestBytes < 0 {
|
||||
return AppSettings{}, fmt.Errorf("%w: server.max_request_bytes must be greater than or equal to 0", ErrInvalidConfig)
|
||||
}
|
||||
out.MaxRequestBytes = *overrides.MaxRequestBytes
|
||||
}
|
||||
if overrides.MaxArtifactBytes != nil {
|
||||
if *overrides.MaxArtifactBytes < 0 {
|
||||
return AppSettings{}, fmt.Errorf("%w: server.max_artifact_bytes must be greater than or equal to 0", ErrInvalidConfig)
|
||||
}
|
||||
out.MaxArtifactBytes = *overrides.MaxArtifactBytes
|
||||
}
|
||||
if overrides.MaxResponseBytes != nil {
|
||||
if *overrides.MaxResponseBytes < 0 {
|
||||
return AppSettings{}, fmt.Errorf("%w: server.max_response_bytes must be greater than or equal to 0", ErrInvalidConfig)
|
||||
}
|
||||
out.MaxResponseBytes = *overrides.MaxResponseBytes
|
||||
}
|
||||
if rawFormat := strings.TrimSpace(overrides.RenderFormat); rawFormat != "" {
|
||||
parsed, err := renderformat.ParsePreparedRunOutputFormat(rawFormat)
|
||||
if err != nil {
|
||||
@@ -181,6 +217,27 @@ func applyConfig(base AppSettings, cfg Config) (AppSettings, error) {
|
||||
if v := strings.TrimSpace(cfg.Server.Addr); v != "" {
|
||||
out.ServerAddr = v
|
||||
}
|
||||
if v := strings.TrimSpace(cfg.Server.ArtifactRoot); v != "" {
|
||||
out.ArtifactRoot = filepath.Clean(v)
|
||||
}
|
||||
if cfg.Server.MaxRequestBytes != nil {
|
||||
if *cfg.Server.MaxRequestBytes < 0 {
|
||||
return AppSettings{}, fmt.Errorf("%w: server.max_request_bytes must be greater than or equal to 0", ErrInvalidConfig)
|
||||
}
|
||||
out.MaxRequestBytes = *cfg.Server.MaxRequestBytes
|
||||
}
|
||||
if cfg.Server.MaxArtifactBytes != nil {
|
||||
if *cfg.Server.MaxArtifactBytes < 0 {
|
||||
return AppSettings{}, fmt.Errorf("%w: server.max_artifact_bytes must be greater than or equal to 0", ErrInvalidConfig)
|
||||
}
|
||||
out.MaxArtifactBytes = *cfg.Server.MaxArtifactBytes
|
||||
}
|
||||
if cfg.Server.MaxResponseBytes != nil {
|
||||
if *cfg.Server.MaxResponseBytes < 0 {
|
||||
return AppSettings{}, fmt.Errorf("%w: server.max_response_bytes must be greater than or equal to 0", ErrInvalidConfig)
|
||||
}
|
||||
out.MaxResponseBytes = *cfg.Server.MaxResponseBytes
|
||||
}
|
||||
if rawFormat := strings.TrimSpace(cfg.Defaults.RenderFormat); rawFormat != "" {
|
||||
parsed, err := renderformat.ParsePreparedRunOutputFormat(rawFormat)
|
||||
if err != nil {
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
||||
)
|
||||
|
||||
@@ -24,6 +25,20 @@ func TestLoadConfigMissingImplicitPathUsesBuiltInDefaults(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuiltInDefaultsIncludeHTTPSizeLimits(t *testing.T) {
|
||||
got := BuiltInDefaults()
|
||||
|
||||
if got.MaxRequestBytes != defaults.HTTPMaxRequestBytesDefault {
|
||||
t.Fatalf("unexpected max request bytes: %d", got.MaxRequestBytes)
|
||||
}
|
||||
if got.MaxArtifactBytes != defaults.HTTPMaxArtifactBytesDefault {
|
||||
t.Fatalf("unexpected max artifact bytes: %d", got.MaxArtifactBytes)
|
||||
}
|
||||
if got.MaxResponseBytes != defaults.HTTPMaxResponseBytesDefault {
|
||||
t.Fatalf("unexpected max response bytes: %d", got.MaxResponseBytes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadConfigMissingExplicitPathReturnsError(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
missing := filepath.Join(tmp, "missing.yml")
|
||||
@@ -92,6 +107,10 @@ profile_dir: ./profiles
|
||||
schema_dir: ./schemas
|
||||
server:
|
||||
addr: 127.0.0.1:9090
|
||||
artifact_root: ./artifacts
|
||||
max_request_bytes: 1024
|
||||
max_artifact_bytes: 2048
|
||||
max_response_bytes: 4096
|
||||
defaults:
|
||||
render_format: json
|
||||
`)
|
||||
@@ -113,11 +132,61 @@ defaults:
|
||||
if got.ServerAddr != "127.0.0.1:9090" {
|
||||
t.Fatalf("unexpected server.addr: %q", got.ServerAddr)
|
||||
}
|
||||
if got.ArtifactRoot != filepath.Clean("./artifacts") {
|
||||
t.Fatalf("unexpected server.artifact_root: %q", got.ArtifactRoot)
|
||||
}
|
||||
if got.MaxRequestBytes != 1024 {
|
||||
t.Fatalf("unexpected server.max_request_bytes: %d", got.MaxRequestBytes)
|
||||
}
|
||||
if got.MaxArtifactBytes != 2048 {
|
||||
t.Fatalf("unexpected server.max_artifact_bytes: %d", got.MaxArtifactBytes)
|
||||
}
|
||||
if got.MaxResponseBytes != 4096 {
|
||||
t.Fatalf("unexpected server.max_response_bytes: %d", got.MaxResponseBytes)
|
||||
}
|
||||
if got.DefaultRenderFormat != renderformat.PreparedRunFormatJSON {
|
||||
t.Fatalf("unexpected defaults.render_format: %q", got.DefaultRenderFormat)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadConfigAcceptsZeroHTTPSizeLimits(t *testing.T) {
|
||||
path := writeConfigFile(t, "config.yml", `
|
||||
server:
|
||||
max_request_bytes: 0
|
||||
max_artifact_bytes: 0
|
||||
max_response_bytes: 0
|
||||
`)
|
||||
|
||||
got, err := LoadConfig(path, true)
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if got.MaxRequestBytes != 0 || got.MaxArtifactBytes != 0 || got.MaxResponseBytes != 0 {
|
||||
t.Fatalf("expected zero limits to be preserved, got request=%d artifact=%d response=%d", got.MaxRequestBytes, got.MaxArtifactBytes, got.MaxResponseBytes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadConfigRejectsNegativeHTTPSizeLimits(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
body string
|
||||
}{
|
||||
{name: "request", body: "server:\n max_request_bytes: -1\n"},
|
||||
{name: "artifact", body: "server:\n max_artifact_bytes: -1\n"},
|
||||
{name: "response", body: "server:\n max_response_bytes: -1\n"},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
path := writeConfigFile(t, "config.yml", tc.body)
|
||||
_, err := LoadConfig(path, true)
|
||||
if !errors.Is(err, ErrInvalidConfig) {
|
||||
t.Fatalf("expected ErrInvalidConfig, got %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadConfigEmptyFileResolvesToBuiltInDefaults(t *testing.T) {
|
||||
path := writeConfigFile(t, "config.yml", "")
|
||||
|
||||
@@ -185,15 +254,26 @@ func TestApplyCLIOverridesAppliesPrecedence(t *testing.T) {
|
||||
ProfileDir: "/from/config/profiles",
|
||||
SchemaDir: "/from/config/schemas",
|
||||
ServerAddr: ":1234",
|
||||
ArtifactRoot: "/from/config/artifacts",
|
||||
MaxRequestBytes: 111,
|
||||
MaxArtifactBytes: 222,
|
||||
MaxResponseBytes: 333,
|
||||
DefaultRenderFormat: renderformat.PreparedRunFormatJSON,
|
||||
}
|
||||
maxRequestBytes := int64(0)
|
||||
maxArtifactBytes := int64(444)
|
||||
maxResponseBytes := int64(555)
|
||||
|
||||
got, err := ApplyCLIOverrides(base, CLIOverrides{
|
||||
PromptDir: "./prompts-cli",
|
||||
ProfileDir: "./profiles-cli",
|
||||
SchemaDir: "./schemas-cli",
|
||||
ServerAddr: ":8081",
|
||||
RenderFormat: "text",
|
||||
PromptDir: "./prompts-cli",
|
||||
ProfileDir: "./profiles-cli",
|
||||
SchemaDir: "./schemas-cli",
|
||||
ServerAddr: ":8081",
|
||||
ArtifactRoot: "./artifacts-cli",
|
||||
MaxRequestBytes: &maxRequestBytes,
|
||||
MaxArtifactBytes: &maxArtifactBytes,
|
||||
MaxResponseBytes: &maxResponseBytes,
|
||||
RenderFormat: "text",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
@@ -211,11 +291,45 @@ func TestApplyCLIOverridesAppliesPrecedence(t *testing.T) {
|
||||
if got.ServerAddr != ":8081" {
|
||||
t.Fatalf("unexpected server addr: %q", got.ServerAddr)
|
||||
}
|
||||
if got.ArtifactRoot != filepath.Clean("./artifacts-cli") {
|
||||
t.Fatalf("unexpected artifact root: %q", got.ArtifactRoot)
|
||||
}
|
||||
if got.MaxRequestBytes != 0 {
|
||||
t.Fatalf("unexpected max request bytes: %d", got.MaxRequestBytes)
|
||||
}
|
||||
if got.MaxArtifactBytes != 444 {
|
||||
t.Fatalf("unexpected max artifact bytes: %d", got.MaxArtifactBytes)
|
||||
}
|
||||
if got.MaxResponseBytes != 555 {
|
||||
t.Fatalf("unexpected max response bytes: %d", got.MaxResponseBytes)
|
||||
}
|
||||
if got.DefaultRenderFormat != renderformat.PreparedRunFormatText {
|
||||
t.Fatalf("unexpected render format: %q", got.DefaultRenderFormat)
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyCLIOverridesRejectsNegativeHTTPSizeLimits(t *testing.T) {
|
||||
negative := int64(-1)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
overrides CLIOverrides
|
||||
}{
|
||||
{name: "request", overrides: CLIOverrides{MaxRequestBytes: &negative}},
|
||||
{name: "artifact", overrides: CLIOverrides{MaxArtifactBytes: &negative}},
|
||||
{name: "response", overrides: CLIOverrides{MaxResponseBytes: &negative}},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
_, err := ApplyCLIOverrides(BuiltInDefaults(), tc.overrides)
|
||||
if !errors.Is(err, ErrInvalidConfig) {
|
||||
t.Fatalf("expected ErrInvalidConfig, got %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyCLIOverridesInvalidRenderFormatReturnsError(t *testing.T) {
|
||||
_, err := ApplyCLIOverrides(BuiltInDefaults(), CLIOverrides{RenderFormat: "yaml"})
|
||||
if err == nil {
|
||||
|
||||
@@ -7,13 +7,16 @@ import (
|
||||
)
|
||||
|
||||
const (
|
||||
HTTPAddrDefault = ":8080"
|
||||
SchemaDirDefault = "."
|
||||
OutputArtifactName = "output"
|
||||
ContentTypeTextPlain = "text/plain"
|
||||
ContentTypeTextMarkdown = "text/markdown"
|
||||
ContentTypeApplicationJSON = "application/json"
|
||||
OpenAIChatCompletionsPath = "/chat/completions"
|
||||
HTTPAddrDefault = ":8080"
|
||||
SchemaDirDefault = "."
|
||||
OutputArtifactName = "output"
|
||||
ContentTypeTextPlain = "text/plain"
|
||||
ContentTypeTextMarkdown = "text/markdown"
|
||||
ContentTypeApplicationJSON = "application/json"
|
||||
OpenAIChatCompletionsPath = "/chat/completions"
|
||||
HTTPMaxRequestBytesDefault = 16 * 1024 * 1024
|
||||
HTTPMaxArtifactBytesDefault = 16 * 1024 * 1024
|
||||
HTTPMaxResponseBytesDefault = 16 * 1024 * 1024
|
||||
|
||||
ExecutionDefaultTemperature = 0.0
|
||||
ExecutionDefaultMaxTokens = 0
|
||||
|
||||
@@ -10,7 +10,6 @@ type ArtifactRefType string
|
||||
const (
|
||||
ArtifactRefInline ArtifactRefType = "inline"
|
||||
ArtifactRefFile ArtifactRefType = "file"
|
||||
ArtifactRefS3 ArtifactRefType = "s3"
|
||||
)
|
||||
|
||||
// OutputFormat defines the desired format of the generated artifact.
|
||||
@@ -41,14 +40,33 @@ const (
|
||||
ValidationSkipped ValidationStatus = "skipped"
|
||||
)
|
||||
|
||||
// CacheControlType defines provider cache behavior for prompt content.
|
||||
type CacheControlType string
|
||||
|
||||
const (
|
||||
CacheControlEphemeral CacheControlType = "ephemeral"
|
||||
)
|
||||
|
||||
const (
|
||||
// SessionIDMaxLength is OpenRouter's documented maximum session_id length.
|
||||
SessionIDMaxLength = 256
|
||||
)
|
||||
|
||||
// CacheControl describes provider cache metadata attached to prompt content.
|
||||
type CacheControl struct {
|
||||
Type CacheControlType `yaml:"type" json:"type"`
|
||||
TTL string `yaml:"ttl,omitempty" json:"ttl,omitempty"`
|
||||
}
|
||||
|
||||
// RunRequest represents a request to generate a single artifact.
|
||||
type RunRequest struct {
|
||||
PromptID string
|
||||
PromptVersion string
|
||||
ProfileID string
|
||||
APIKey string `json:"-" yaml:"-"`
|
||||
Inputs map[string]ArtifactRef
|
||||
Vars map[string]string
|
||||
Execution *ExecutionTarget
|
||||
Execution *ExecutionTargetOverride
|
||||
Validation *OutputContract
|
||||
Metadata map[string]string
|
||||
}
|
||||
@@ -72,25 +90,26 @@ type RunResult struct {
|
||||
StartTime time.Time
|
||||
EndTime time.Time
|
||||
Duration time.Duration
|
||||
Error error
|
||||
}
|
||||
|
||||
// PreparedRun contains pre-LLM execution state from the prepare/render phase.
|
||||
// It must never include resolved API key values, model output, or validation data.
|
||||
type PreparedRun struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
PromptHash string `json:"prompt_hash,omitempty"`
|
||||
SelectedProfileID string `json:"selected_profile_id"`
|
||||
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
||||
OutputContract OutputContract `json:"output_contract"`
|
||||
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
|
||||
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
||||
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
||||
Messages []RenderedMessage `json:"messages"`
|
||||
StartTime time.Time `json:"start_time,omitempty"`
|
||||
EndTime time.Time `json:"end_time,omitempty"`
|
||||
DurationMS int64 `json:"duration_ms,omitempty"`
|
||||
PromptID string `json:"prompt_id"`
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
PromptHash string `json:"prompt_hash,omitempty"`
|
||||
SelectedProfileID string `json:"selected_profile_id"`
|
||||
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
||||
TargetPresence ExecutionTargetPresence `json:"-"`
|
||||
OutputContract OutputContract `json:"output_contract"`
|
||||
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
|
||||
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
||||
Messages []RenderedMessage `json:"messages"`
|
||||
StartTime time.Time `json:"start_time,omitempty"`
|
||||
EndTime time.Time `json:"end_time,omitempty"`
|
||||
DurationMS int64 `json:"duration_ms,omitempty"`
|
||||
}
|
||||
|
||||
// ArtifactRef represents a reference to an input artifact.
|
||||
@@ -116,6 +135,7 @@ type PromptDefinition struct {
|
||||
Version string `yaml:"version"`
|
||||
DefaultProfile string `yaml:"default_profile"`
|
||||
Description string `yaml:"description"`
|
||||
SessionID string `yaml:"session_id" json:"session_id,omitempty"`
|
||||
Inputs []PromptInput `yaml:"inputs"`
|
||||
Templates []PromptMessageTemplate `yaml:"templates"`
|
||||
OutputFormat OutputFormat `yaml:"output_format"`
|
||||
@@ -132,36 +152,65 @@ type PromptInput struct {
|
||||
|
||||
// PromptMessageTemplate defines a template for a chat message.
|
||||
type PromptMessageTemplate struct {
|
||||
Role string `yaml:"role"`
|
||||
Content string `yaml:"content"`
|
||||
ContentFile string `yaml:"content_file"`
|
||||
Role string `yaml:"role"`
|
||||
Content string `yaml:"content"`
|
||||
ContentFile string `yaml:"content_file"`
|
||||
CacheControl *CacheControl `yaml:"cache_control,omitempty" json:"cache_control,omitempty"`
|
||||
}
|
||||
|
||||
// ExecutionProfile describes how and where to execute a model.
|
||||
type ExecutionProfile struct {
|
||||
ID string `yaml:"id"`
|
||||
Endpoint string `yaml:"endpoint"`
|
||||
Model string `yaml:"model"`
|
||||
Temperature float64 `yaml:"temperature"`
|
||||
MaxTokens int `yaml:"max_tokens"`
|
||||
TopP float64 `yaml:"top_p"`
|
||||
TimeoutSeconds int `yaml:"timeout_seconds"`
|
||||
ReasoningEffort string `yaml:"reasoning_effort"`
|
||||
APIKeyEnv string `yaml:"api_key_env"`
|
||||
ExtraParams map[string]string `yaml:"extra_params"`
|
||||
ID string `yaml:"id"`
|
||||
Endpoint string `yaml:"endpoint"`
|
||||
Model string `yaml:"model"`
|
||||
Temperature float64 `yaml:"temperature"`
|
||||
MaxTokens int `yaml:"max_tokens"`
|
||||
TopP float64 `yaml:"top_p"`
|
||||
TimeoutSeconds int `yaml:"timeout_seconds"`
|
||||
ServiceTier string `yaml:"service_tier"`
|
||||
ReasoningEffort string `yaml:"reasoning_effort"`
|
||||
APIKeyEnv string `yaml:"api_key_env"`
|
||||
APIKeyRequired bool `yaml:"-" json:"-"`
|
||||
ExtraParams map[string]any `yaml:"extra_params"`
|
||||
}
|
||||
|
||||
// ExecutionTargetOverride represents per-request runtime setting overrides.
|
||||
type ExecutionTargetOverride struct {
|
||||
Endpoint string `json:"endpoint,omitempty"`
|
||||
Model string `json:"model,omitempty"`
|
||||
Temperature *float64 `json:"temperature,omitempty"`
|
||||
MaxTokens *int `json:"max_tokens,omitempty"`
|
||||
TopP *float64 `json:"top_p,omitempty"`
|
||||
TimeoutSeconds *int `json:"timeout_seconds,omitempty"`
|
||||
ServiceTier string `json:"service_tier,omitempty"`
|
||||
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||
APIKeyEnv string `json:"api_key_env,omitempty"`
|
||||
ExtraParams map[string]any `json:"extra_params,omitempty"`
|
||||
}
|
||||
|
||||
// ExecutionTargetPresence tracks which effective runtime fields came from an
|
||||
// explicit request override even when the resolved value is a zero value.
|
||||
type ExecutionTargetPresence struct {
|
||||
Temperature bool
|
||||
MaxTokens bool
|
||||
TopP bool
|
||||
TimeoutSeconds bool
|
||||
}
|
||||
|
||||
// ExecutionTarget represents effective model runtime settings for a run.
|
||||
type ExecutionTarget struct {
|
||||
Endpoint string `yaml:"endpoint" json:"endpoint"`
|
||||
Model string `yaml:"model" json:"model"`
|
||||
Temperature float64 `yaml:"temperature" json:"temperature"`
|
||||
MaxTokens int `yaml:"max_tokens" json:"max_tokens"`
|
||||
TopP float64 `yaml:"top_p" json:"top_p"`
|
||||
TimeoutSeconds int `yaml:"timeout_seconds" json:"timeout_seconds"`
|
||||
ReasoningEffort string `yaml:"reasoning_effort" json:"reasoning_effort"`
|
||||
APIKeyEnv string `yaml:"api_key_env" json:"api_key_env"`
|
||||
ExtraParams map[string]string `yaml:"extra_params" json:"extra_params"`
|
||||
Endpoint string `yaml:"endpoint" json:"endpoint"`
|
||||
Model string `yaml:"model" json:"model"`
|
||||
Temperature float64 `yaml:"temperature" json:"temperature"`
|
||||
MaxTokens int `yaml:"max_tokens" json:"max_tokens"`
|
||||
TopP float64 `yaml:"top_p" json:"top_p"`
|
||||
TimeoutSeconds int `yaml:"timeout_seconds" json:"timeout_seconds"`
|
||||
ServiceTier string `yaml:"service_tier" json:"service_tier"`
|
||||
ReasoningEffort string `yaml:"reasoning_effort" json:"reasoning_effort"`
|
||||
APIKeyEnv string `yaml:"api_key_env" json:"api_key_env"`
|
||||
APIKey string `yaml:"-" json:"-"`
|
||||
APIKeyRequired bool `yaml:"-" json:"-"`
|
||||
ExtraParams map[string]any `yaml:"extra_params" json:"extra_params"`
|
||||
}
|
||||
|
||||
// OutputContract defines the requirements for the output artifact.
|
||||
@@ -174,19 +223,22 @@ type OutputContract struct {
|
||||
|
||||
// RenderedPrompt represents the prompt after template application.
|
||||
type RenderedPrompt struct {
|
||||
Messages []RenderedMessage `json:"messages"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
Messages []RenderedMessage `json:"messages"`
|
||||
}
|
||||
|
||||
// RenderedMessage is a single message in a rendered prompt.
|
||||
type RenderedMessage struct {
|
||||
Role string `json:"role"`
|
||||
Content string `json:"content"`
|
||||
Role string `json:"role"`
|
||||
Content string `json:"content"`
|
||||
CacheControl *CacheControl `json:"cache_control,omitempty"`
|
||||
}
|
||||
|
||||
// GenerateRequest is the internal request passed to the LLM client.
|
||||
type GenerateRequest struct {
|
||||
Prompt RenderedPrompt
|
||||
Target ExecutionTarget
|
||||
TargetPresence ExecutionTargetPresence
|
||||
StructuredOutput *StructuredOutputSpec
|
||||
}
|
||||
|
||||
@@ -221,6 +273,8 @@ type TokenUsage struct {
|
||||
PromptTokens int
|
||||
CompletionTokens int
|
||||
TotalTokens int
|
||||
CachedTokens int
|
||||
CacheWriteTokens int
|
||||
}
|
||||
|
||||
// ValidationResult represents the outcome of an output validation.
|
||||
@@ -232,23 +286,3 @@ type ValidationResult struct {
|
||||
RepairAttempts int
|
||||
IsValid bool
|
||||
}
|
||||
|
||||
// RunMetadata contains auditing information for a run.
|
||||
type RunMetadata struct {
|
||||
RunID string
|
||||
PromptID string
|
||||
PromptVersion string
|
||||
PromptHash string
|
||||
RenderedPromptHash string
|
||||
SelectedProfileID string
|
||||
InputHashes map[string]string
|
||||
ModelEndpoint string
|
||||
ModelName string
|
||||
Params ExecutionTarget
|
||||
Timestamp time.Time
|
||||
Duration time.Duration
|
||||
Usage TokenUsage
|
||||
ValidationMode ValidationMode
|
||||
ValidationStatus ValidationStatus
|
||||
RepairAttempts int
|
||||
}
|
||||
|
||||
@@ -20,6 +20,7 @@ func TestPreparedRunJSONDoesNotIncludeSecretValues(t *testing.T) {
|
||||
Endpoint: "http://llm/v1",
|
||||
Model: "gpt-test",
|
||||
APIKeyEnv: envName,
|
||||
APIKey: secret,
|
||||
},
|
||||
InputHashes: map[string]string{"transcript": "hash-1"},
|
||||
RenderedPromptHash: "rendered-hash",
|
||||
@@ -53,3 +54,88 @@ func TestPreparedRunJSONDoesNotIncludeSecretValues(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPreparedRunJSONIncludesMessageCacheControlOnlyWhenPresent(t *testing.T) {
|
||||
prepared := PreparedRun{
|
||||
PromptID: "prompt.id",
|
||||
SelectedProfileID: "local-fast",
|
||||
EffectiveModelParams: ExecutionTarget{
|
||||
Endpoint: "http://llm/v1",
|
||||
Model: "gpt-test",
|
||||
},
|
||||
RenderedPromptHash: "rendered-hash",
|
||||
Messages: []RenderedMessage{
|
||||
{
|
||||
Role: "system",
|
||||
Content: "You are helpful.",
|
||||
CacheControl: &CacheControl{
|
||||
Type: CacheControlEphemeral,
|
||||
TTL: "1h",
|
||||
},
|
||||
},
|
||||
{Role: "user", Content: "Summarize this."},
|
||||
},
|
||||
}
|
||||
|
||||
b, err := json.Marshal(prepared)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal failed: %v", err)
|
||||
}
|
||||
|
||||
var decoded struct {
|
||||
Messages []map[string]any `json:"messages"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &decoded); err != nil {
|
||||
t.Fatalf("unmarshal failed: %v", err)
|
||||
}
|
||||
if len(decoded.Messages) != 2 {
|
||||
t.Fatalf("expected 2 messages, got %d", len(decoded.Messages))
|
||||
}
|
||||
|
||||
cacheControl, ok := decoded.Messages[0]["cache_control"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("expected cache_control on first message, got %#v", decoded.Messages[0])
|
||||
}
|
||||
if cacheControl["type"] != string(CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
|
||||
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
|
||||
}
|
||||
if _, ok := decoded.Messages[1]["cache_control"]; ok {
|
||||
t.Fatalf("expected second message to omit cache_control, got %#v", decoded.Messages[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestPreparedRunJSONIncludesSessionIDOnlyWhenPresent(t *testing.T) {
|
||||
prepared := PreparedRun{
|
||||
PromptID: "prompt.id",
|
||||
SelectedProfileID: "local-fast",
|
||||
EffectiveModelParams: ExecutionTarget{
|
||||
Endpoint: "http://llm/v1",
|
||||
Model: "gpt-test",
|
||||
},
|
||||
SessionID: "session-123",
|
||||
RenderedPromptHash: "rendered-hash",
|
||||
Messages: []RenderedMessage{{Role: "user", Content: "Summarize this."}},
|
||||
}
|
||||
|
||||
b, err := json.Marshal(prepared)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal failed: %v", err)
|
||||
}
|
||||
|
||||
var decoded map[string]any
|
||||
if err := json.Unmarshal(b, &decoded); err != nil {
|
||||
t.Fatalf("unmarshal failed: %v", err)
|
||||
}
|
||||
if decoded["session_id"] != "session-123" {
|
||||
t.Fatalf("expected session_id in prepared run JSON, got %#v", decoded["session_id"])
|
||||
}
|
||||
|
||||
prepared.SessionID = ""
|
||||
b, err = json.Marshal(prepared)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal failed: %v", err)
|
||||
}
|
||||
if strings.Contains(string(b), "session_id") {
|
||||
t.Fatalf("expected empty session_id to be omitted, got %s", b)
|
||||
}
|
||||
}
|
||||
|
||||
142
internal/filecatalog/catalog.go
Normal file
142
internal/filecatalog/catalog.go
Normal file
@@ -0,0 +1,142 @@
|
||||
package filecatalog
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// FindYAMLFiles returns sorted full paths for .yaml and .yml files under root.
|
||||
func FindYAMLFiles(ctx context.Context, root string) ([]string, error) {
|
||||
var files []string
|
||||
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
default:
|
||||
}
|
||||
if d.IsDir() {
|
||||
return nil
|
||||
}
|
||||
if !IsYAMLFile(d.Name()) {
|
||||
return nil
|
||||
}
|
||||
files = append(files, path)
|
||||
return nil
|
||||
})
|
||||
sort.Strings(files)
|
||||
return files, err
|
||||
}
|
||||
|
||||
// FindFSYAMLFiles returns sorted paths for .yaml and .yml files under root in fsys.
|
||||
func FindFSYAMLFiles(ctx context.Context, fsys fs.FS, root string) ([]string, error) {
|
||||
cleanRoot := CleanFSRoot(root)
|
||||
var files []string
|
||||
err := fs.WalkDir(fsys, cleanRoot, func(name string, d fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
default:
|
||||
}
|
||||
if d.IsDir() {
|
||||
return nil
|
||||
}
|
||||
if !IsYAMLFile(d.Name()) {
|
||||
return nil
|
||||
}
|
||||
files = append(files, name)
|
||||
return nil
|
||||
})
|
||||
sort.Strings(files)
|
||||
return files, err
|
||||
}
|
||||
|
||||
// RelativePath computes a clean relative path from root to path.
|
||||
func RelativePath(root string, filePath string) string {
|
||||
rel, err := filepath.Rel(root, filePath)
|
||||
if err != nil {
|
||||
return filepath.Clean(filePath)
|
||||
}
|
||||
return filepath.Clean(rel)
|
||||
}
|
||||
|
||||
// CleanFSRoot normalizes a root path for use with fs.FS.
|
||||
func CleanFSRoot(root string) string {
|
||||
root = strings.TrimSpace(root)
|
||||
if root == "" || root == "." {
|
||||
return "."
|
||||
}
|
||||
return path.Clean(root)
|
||||
}
|
||||
|
||||
// DisplayPath returns name relative to root for messages about fs.FS paths.
|
||||
func DisplayPath(root string, name string) string {
|
||||
cleanRoot := CleanFSRoot(root)
|
||||
cleanName := path.Clean(name)
|
||||
if cleanRoot == "." {
|
||||
return cleanName
|
||||
}
|
||||
prefix := strings.TrimSuffix(cleanRoot, "/") + "/"
|
||||
if strings.HasPrefix(cleanName, prefix) {
|
||||
return strings.TrimPrefix(cleanName, prefix)
|
||||
}
|
||||
return cleanName
|
||||
}
|
||||
|
||||
// ResolveFSPath resolves userPath from baseDir and keeps it inside root.
|
||||
func ResolveFSPath(root string, baseDir string, userPath string) (string, string, error) {
|
||||
cleanRoot := CleanFSRoot(root)
|
||||
cleanBase := path.Clean(strings.TrimSpace(baseDir))
|
||||
if cleanBase == "" {
|
||||
cleanBase = cleanRoot
|
||||
}
|
||||
if !containsFSPath(cleanRoot, cleanBase) {
|
||||
return "", "", fmt.Errorf("base path %q is outside source root %q", cleanBase, cleanRoot)
|
||||
}
|
||||
|
||||
cleanUserPath := strings.TrimSpace(userPath)
|
||||
if cleanUserPath == "" {
|
||||
return "", "", fmt.Errorf("path is required")
|
||||
}
|
||||
cleanUserPath = path.Clean(cleanUserPath)
|
||||
if path.IsAbs(cleanUserPath) {
|
||||
return "", "", fmt.Errorf("path %q must be relative", userPath)
|
||||
}
|
||||
|
||||
resolved := path.Clean(path.Join(cleanBase, cleanUserPath))
|
||||
if !containsFSPath(cleanRoot, resolved) {
|
||||
return "", "", fmt.Errorf("path %q escapes source root %q", userPath, cleanRoot)
|
||||
}
|
||||
return resolved, DisplayPath(cleanRoot, resolved), nil
|
||||
}
|
||||
|
||||
func containsFSPath(root string, name string) bool {
|
||||
root = CleanFSRoot(root)
|
||||
name = path.Clean(name)
|
||||
if root == "." {
|
||||
return name == "." || (name != ".." && !strings.HasPrefix(name, "../"))
|
||||
}
|
||||
return name == root || strings.HasPrefix(name, strings.TrimSuffix(root, "/")+"/")
|
||||
}
|
||||
|
||||
// Stem strips .yaml or .yml from a file name.
|
||||
func Stem(name string) string {
|
||||
name = strings.TrimSuffix(name, ".yaml")
|
||||
name = strings.TrimSuffix(name, ".yml")
|
||||
return name
|
||||
}
|
||||
|
||||
func IsYAMLFile(name string) bool {
|
||||
return strings.HasSuffix(name, ".yaml") || strings.HasSuffix(name, ".yml")
|
||||
}
|
||||
270
internal/filecatalog/catalog_test.go
Normal file
270
internal/filecatalog/catalog_test.go
Normal file
@@ -0,0 +1,270 @@
|
||||
package filecatalog
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
)
|
||||
|
||||
func TestFindYAMLFilesNestedSortedAndFiltered(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
mustWriteFile(t, filepath.Join(root, "z", "prompt.yml"), "id: z")
|
||||
mustWriteFile(t, filepath.Join(root, "a", "profile.yaml"), "id: a")
|
||||
mustWriteFile(t, filepath.Join(root, "a", "ignore.txt"), "not yaml")
|
||||
mustWriteFile(t, filepath.Join(root, "b", "ignore.yaml.bak"), "not yaml")
|
||||
|
||||
got, err := FindYAMLFiles(context.Background(), root)
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
|
||||
want := []string{
|
||||
filepath.Join(root, "a", "profile.yaml"),
|
||||
filepath.Join(root, "z", "prompt.yml"),
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("expected sorted YAML files %v, got %v", want, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFindYAMLFilesHonorsContextCancellation(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
mustWriteFile(t, filepath.Join(root, "one.yaml"), "id: one")
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
|
||||
_, err := FindYAMLFiles(ctx, root)
|
||||
if !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("expected context.Canceled, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFindFSYAMLFilesNestedSortedAndFiltered(t *testing.T) {
|
||||
fsys := fstest.MapFS{
|
||||
"prompts/z/prompt.yml": &fstest.MapFile{Data: []byte("id: z")},
|
||||
"prompts/a/profile.yaml": &fstest.MapFile{Data: []byte("id: a")},
|
||||
"prompts/a/ignore.txt": &fstest.MapFile{Data: []byte("not yaml")},
|
||||
"prompts/b/ignore.yaml.bak": &fstest.MapFile{Data: []byte("not yaml")},
|
||||
"other/ignored.yaml": &fstest.MapFile{Data: []byte("id: ignored")},
|
||||
}
|
||||
|
||||
got, err := FindFSYAMLFiles(context.Background(), fsys, " prompts ")
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
|
||||
want := []string{
|
||||
"prompts/a/profile.yaml",
|
||||
"prompts/z/prompt.yml",
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("expected sorted YAML files %v, got %v", want, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFindFSYAMLFilesHonorsContextCancellation(t *testing.T) {
|
||||
fsys := fstest.MapFS{
|
||||
"one.yaml": &fstest.MapFile{Data: []byte("id: one")},
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
|
||||
_, err := FindFSYAMLFiles(ctx, fsys, ".")
|
||||
if !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("expected context.Canceled, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRelativePathNested(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
path := filepath.Join(root, "nested", "profiles", "local.yaml")
|
||||
got := RelativePath(root, path)
|
||||
want := filepath.Join("nested", "profiles", "local.yaml")
|
||||
if got != want {
|
||||
t.Fatalf("expected relative path %q, got %q", want, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanFSRoot(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
root string
|
||||
want string
|
||||
}{
|
||||
{name: "empty", root: "", want: "."},
|
||||
{name: "dot", root: ".", want: "."},
|
||||
{name: "trimmed", root: " prompts/../profiles ", want: "profiles"},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := CleanFSRoot(tc.root); got != tc.want {
|
||||
t.Fatalf("expected %q, got %q", tc.want, got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDisplayPath(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
root string
|
||||
path string
|
||||
want string
|
||||
}{
|
||||
{name: "root dot", root: ".", path: "profiles/local.yaml", want: "profiles/local.yaml"},
|
||||
{name: "nested root", root: "profiles", path: "profiles/local.yaml", want: "local.yaml"},
|
||||
{name: "outside root", root: "profiles", path: "other/local.yaml", want: "other/local.yaml"},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := DisplayPath(tc.root, tc.path); got != tc.want {
|
||||
t.Fatalf("expected %q, got %q", tc.want, got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveFSPath(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
root string
|
||||
baseDir string
|
||||
userPath string
|
||||
wantPath string
|
||||
wantDisplay string
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "sibling inside root",
|
||||
root: "prompts",
|
||||
baseDir: "prompts/nested",
|
||||
userPath: "./messages/user.tmpl",
|
||||
wantPath: "prompts/nested/messages/user.tmpl",
|
||||
wantDisplay: "nested/messages/user.tmpl",
|
||||
},
|
||||
{
|
||||
name: "parent inside root",
|
||||
root: "prompts",
|
||||
baseDir: "prompts/nested",
|
||||
userPath: "../shared/user.tmpl",
|
||||
wantPath: "prompts/shared/user.tmpl",
|
||||
wantDisplay: "shared/user.tmpl",
|
||||
},
|
||||
{
|
||||
name: "escape rejected",
|
||||
root: "prompts",
|
||||
baseDir: "prompts/nested",
|
||||
userPath: "../../outside.tmpl",
|
||||
wantErr: "escapes source root",
|
||||
},
|
||||
{
|
||||
name: "absolute path rejected",
|
||||
root: "prompts",
|
||||
baseDir: "prompts/nested",
|
||||
userPath: "/outside.tmpl",
|
||||
wantErr: "must be relative",
|
||||
},
|
||||
{
|
||||
name: "empty path rejected",
|
||||
root: "prompts",
|
||||
baseDir: "prompts/nested",
|
||||
userPath: " ",
|
||||
wantErr: "path is required",
|
||||
},
|
||||
{
|
||||
name: "dot root allows normal relative path",
|
||||
root: ".",
|
||||
baseDir: ".",
|
||||
userPath: "schemas/events.schema.json",
|
||||
wantPath: "schemas/events.schema.json",
|
||||
wantDisplay: "schemas/events.schema.json",
|
||||
},
|
||||
{
|
||||
name: "dot root rejects parent escape",
|
||||
root: ".",
|
||||
baseDir: ".",
|
||||
userPath: "../outside.tmpl",
|
||||
wantErr: "escapes source root",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
gotPath, gotDisplay, err := ResolveFSPath(tc.root, tc.baseDir, tc.userPath)
|
||||
if tc.wantErr != "" {
|
||||
if err == nil {
|
||||
t.Fatalf("expected error containing %q", tc.wantErr)
|
||||
}
|
||||
if !strings.Contains(err.Error(), tc.wantErr) {
|
||||
t.Fatalf("expected error to contain %q, got %v", tc.wantErr, err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if gotPath != tc.wantPath || gotDisplay != tc.wantDisplay {
|
||||
t.Fatalf("expected path/display %q/%q, got %q/%q", tc.wantPath, tc.wantDisplay, gotPath, gotDisplay)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestStemStripsYAMLExtensions(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
in string
|
||||
want string
|
||||
}{
|
||||
{name: "yaml", in: "prompt.yaml", want: "prompt"},
|
||||
{name: "yml", in: "profile.yml", want: "profile"},
|
||||
{name: "other", in: "file.txt", want: "file.txt"},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := Stem(tc.in); got != tc.want {
|
||||
t.Fatalf("expected %q, got %q", tc.want, got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsYAMLFile(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
in string
|
||||
want bool
|
||||
}{
|
||||
{name: "yaml", in: "prompt.yaml", want: true},
|
||||
{name: "yml", in: "profile.yml", want: true},
|
||||
{name: "backup", in: "profile.yaml.bak", want: false},
|
||||
{name: "uppercase", in: "profile.YAML", want: false},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := IsYAMLFile(tc.in); got != tc.want {
|
||||
t.Fatalf("expected %v, got %v", tc.want, got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func mustWriteFile(t *testing.T, path string, content string) {
|
||||
t.Helper()
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
t.Fatalf("failed to create directory: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||
t.Fatalf("failed to write file %q: %v", path, err)
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
// Package format formats already-prepared domain data for adapters.
|
||||
// Package format formats already-prepared public data for adapters.
|
||||
package format
|
||||
|
||||
import (
|
||||
@@ -9,7 +9,7 @@ import (
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||
"gitea.maximumdirect.net/eric/scriptorium"
|
||||
)
|
||||
|
||||
var ErrUnknownPreparedRunFormat = errors.New("unknown prepared run format")
|
||||
@@ -26,7 +26,7 @@ const (
|
||||
|
||||
// PreparedRunFormatter serializes a prepared run without performing use case work.
|
||||
type PreparedRunFormatter interface {
|
||||
Format(prepared *domain.PreparedRun) ([]byte, error)
|
||||
Format(prepared *scriptorium.PreparedRun) ([]byte, error)
|
||||
}
|
||||
|
||||
// ParsePreparedRunOutputFormat parses a format name.
|
||||
@@ -56,7 +56,7 @@ func FormatterForPreparedRun(outputFormat PreparedRunOutputFormat) (PreparedRunF
|
||||
}
|
||||
|
||||
// FormatPreparedRun formats a prepared run using the selected format.
|
||||
func FormatPreparedRun(prepared *domain.PreparedRun, outputFormat PreparedRunOutputFormat) ([]byte, error) {
|
||||
func FormatPreparedRun(prepared *scriptorium.PreparedRun, outputFormat PreparedRunOutputFormat) ([]byte, error) {
|
||||
formatter, err := FormatterForPreparedRun(outputFormat)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -65,7 +65,7 @@ func FormatPreparedRun(prepared *domain.PreparedRun, outputFormat PreparedRunOut
|
||||
}
|
||||
|
||||
// FormatPreparedRunByName parses a format name and formats a prepared run.
|
||||
func FormatPreparedRunByName(prepared *domain.PreparedRun, rawFormat string) ([]byte, error) {
|
||||
func FormatPreparedRunByName(prepared *scriptorium.PreparedRun, rawFormat string) ([]byte, error) {
|
||||
outputFormat, err := ParsePreparedRunOutputFormat(rawFormat)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -75,7 +75,7 @@ func FormatPreparedRunByName(prepared *domain.PreparedRun, rawFormat string) ([]
|
||||
|
||||
type jsonPreparedRunFormatter struct{}
|
||||
|
||||
func (jsonPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, error) {
|
||||
func (jsonPreparedRunFormatter) Format(prepared *scriptorium.PreparedRun) ([]byte, error) {
|
||||
if prepared == nil {
|
||||
return nil, errors.New("prepared run is nil")
|
||||
}
|
||||
@@ -84,7 +84,7 @@ func (jsonPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
|
||||
|
||||
type textPreparedRunFormatter struct{}
|
||||
|
||||
func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, error) {
|
||||
func (textPreparedRunFormatter) Format(prepared *scriptorium.PreparedRun) ([]byte, error) {
|
||||
if prepared == nil {
|
||||
return nil, errors.New("prepared run is nil")
|
||||
}
|
||||
@@ -96,6 +96,9 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
|
||||
if prepared.PromptHash != "" {
|
||||
fmt.Fprintf(&b, "prompt_hash: %s\n", prepared.PromptHash)
|
||||
}
|
||||
if prepared.SessionID != "" {
|
||||
fmt.Fprintf(&b, "session_id: %s\n", prepared.SessionID)
|
||||
}
|
||||
fmt.Fprintf(&b, "rendered_prompt_hash: %s\n", prepared.RenderedPromptHash)
|
||||
|
||||
target := prepared.EffectiveModelParams
|
||||
@@ -106,6 +109,9 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
|
||||
fmt.Fprintf(&b, " max_tokens: %d\n", target.MaxTokens)
|
||||
fmt.Fprintf(&b, " top_p: %g\n", target.TopP)
|
||||
fmt.Fprintf(&b, " timeout_seconds: %d\n", target.TimeoutSeconds)
|
||||
if target.ServiceTier != "" {
|
||||
fmt.Fprintf(&b, " service_tier: %s\n", target.ServiceTier)
|
||||
}
|
||||
if target.ReasoningEffort != "" {
|
||||
fmt.Fprintf(&b, " reasoning_effort: %s\n", target.ReasoningEffort)
|
||||
}
|
||||
@@ -120,7 +126,11 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, k := range keys {
|
||||
fmt.Fprintf(&b, " %s: %s\n", k, target.ExtraParams[k])
|
||||
renderedValue, err := formatExtraParamTextValue(target.ExtraParams[k])
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to format extra_params.%s: %w", k, err)
|
||||
}
|
||||
fmt.Fprintf(&b, " %s: %s\n", k, renderedValue)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -136,7 +146,7 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
|
||||
|
||||
fmt.Fprintln(&b, "messages:")
|
||||
roleOrder := make([]string, 0)
|
||||
byRole := make(map[string][]domain.RenderedMessage)
|
||||
byRole := make(map[string][]scriptorium.RenderedMessage)
|
||||
for _, msg := range prepared.Messages {
|
||||
if _, exists := byRole[msg.Role]; !exists {
|
||||
roleOrder = append(roleOrder, msg.Role)
|
||||
@@ -148,6 +158,13 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
|
||||
messages := byRole[role]
|
||||
for i, msg := range messages {
|
||||
fmt.Fprintf(&b, " - message: %d\n", i+1)
|
||||
if msg.CacheControl != nil {
|
||||
fmt.Fprintf(&b, " cache_control: %s", msg.CacheControl.Type)
|
||||
if msg.CacheControl.TTL != "" {
|
||||
fmt.Fprintf(&b, " ttl=%s", msg.CacheControl.TTL)
|
||||
}
|
||||
fmt.Fprintln(&b)
|
||||
}
|
||||
fmt.Fprintln(&b, " content: |")
|
||||
content := msg.Content
|
||||
if content == "" {
|
||||
@@ -162,3 +179,15 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
|
||||
|
||||
return b.Bytes(), nil
|
||||
}
|
||||
|
||||
func formatExtraParamTextValue(value any) (string, error) {
|
||||
if s, ok := value.(string); ok {
|
||||
return s, nil
|
||||
}
|
||||
|
||||
b, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return string(b), nil
|
||||
}
|
||||
|
||||
@@ -6,7 +6,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||
"gitea.maximumdirect.net/eric/scriptorium"
|
||||
)
|
||||
|
||||
func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
|
||||
@@ -28,6 +28,7 @@ func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
|
||||
"max_tokens: 256",
|
||||
"top_p: 0.8",
|
||||
"timeout_seconds: 45",
|
||||
"service_tier: priority",
|
||||
"reasoning_effort: medium",
|
||||
"api_key_env: SCRIPTORIUM_API_KEY",
|
||||
"prompt_hash: prompt-hash",
|
||||
@@ -48,6 +49,36 @@ func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTextFormatterRendersExtraParamsDeterministically(t *testing.T) {
|
||||
prepared := samplePreparedRun()
|
||||
prepared.EffectiveModelParams.ExtraParams = map[string]any{
|
||||
"z_string": "enabled",
|
||||
"b_number": 42,
|
||||
"a_object": map[string]any{
|
||||
"nested": "value",
|
||||
"count": 2,
|
||||
},
|
||||
"c_array": []any{"first", 3, false},
|
||||
}
|
||||
|
||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
s := string(out)
|
||||
|
||||
want := strings.Join([]string{
|
||||
" extra_params:",
|
||||
" a_object: {\"count\":2,\"nested\":\"value\"}",
|
||||
" b_number: 42",
|
||||
" c_array: [\"first\",3,false]",
|
||||
" z_string: enabled",
|
||||
}, "\n")
|
||||
if !strings.Contains(s, want) {
|
||||
t.Fatalf("expected deterministic extra_params block %q, got:\n%s", want, s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTextFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
|
||||
const secret = "super-secret-api-key"
|
||||
t.Setenv("SCRIPTORIUM_API_KEY", secret)
|
||||
@@ -61,8 +92,93 @@ func TestTextFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTextFormatterDoesNotIncludeDirectAPIKeyValue(t *testing.T) {
|
||||
const directKey = "direct-format-key"
|
||||
// PreparedRun intentionally has no field for direct API keys.
|
||||
prepared := samplePreparedRun()
|
||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if strings.Contains(string(out), directKey) {
|
||||
t.Fatalf("text output should not include direct api key value: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTextFormatterIncludesMessageCacheControlBeforeContent(t *testing.T) {
|
||||
prepared := samplePreparedRun()
|
||||
prepared.Messages = []scriptorium.RenderedMessage{
|
||||
{
|
||||
Role: "system",
|
||||
Content: "System guidance.",
|
||||
CacheControl: &scriptorium.CacheControl{
|
||||
Type: scriptorium.CacheControlEphemeral,
|
||||
TTL: "1h",
|
||||
},
|
||||
},
|
||||
{Role: "user", Content: "Summarize the transcript."},
|
||||
}
|
||||
|
||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
s := string(out)
|
||||
if !strings.Contains(s, " system:\n - message: 1\n cache_control: ephemeral ttl=1h\n content: |") {
|
||||
t.Fatalf("expected system message cache control before content, got:\n%s", s)
|
||||
}
|
||||
if strings.Count(s, "cache_control:") != 1 {
|
||||
t.Fatalf("expected exactly one cache_control line, got:\n%s", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTextFormatterIncludesSessionIDWhenPresent(t *testing.T) {
|
||||
prepared := samplePreparedRun()
|
||||
prepared.SessionID = "session-123"
|
||||
|
||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if !strings.Contains(string(out), "session_id: session-123\n") {
|
||||
t.Fatalf("expected session_id in text output, got:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTextFormatterOmitsEmptyCacheControlTTL(t *testing.T) {
|
||||
prepared := samplePreparedRun()
|
||||
prepared.Messages = []scriptorium.RenderedMessage{
|
||||
{
|
||||
Role: "system",
|
||||
Content: "System guidance.",
|
||||
CacheControl: &scriptorium.CacheControl{
|
||||
Type: scriptorium.CacheControlEphemeral,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
s := string(out)
|
||||
if !strings.Contains(s, " cache_control: ephemeral\n") {
|
||||
t.Fatalf("expected cache_control line without ttl, got:\n%s", s)
|
||||
}
|
||||
if strings.Contains(s, "ttl=") {
|
||||
t.Fatalf("expected empty ttl to be omitted, got:\n%s", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
|
||||
prepared := samplePreparedRun()
|
||||
prepared.SessionID = "session-123"
|
||||
prepared.EffectiveModelParams.ExtraParams = map[string]any{
|
||||
"number": 42,
|
||||
"nested": map[string]any{
|
||||
"enabled": true,
|
||||
},
|
||||
}
|
||||
|
||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
|
||||
if err != nil {
|
||||
@@ -86,9 +202,24 @@ func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
|
||||
if decoded["rendered_prompt_hash"] != "rendered-hash" {
|
||||
t.Fatalf("expected rendered_prompt_hash in json output, got %#v", decoded["rendered_prompt_hash"])
|
||||
}
|
||||
if _, ok := decoded["effective_model_params"]; !ok {
|
||||
if decoded["session_id"] != "session-123" {
|
||||
t.Fatalf("expected session_id in json output, got %#v", decoded["session_id"])
|
||||
}
|
||||
modelParams, ok := decoded["effective_model_params"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("expected effective_model_params in json output, got %#v", decoded)
|
||||
}
|
||||
extraParams, ok := modelParams["extra_params"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("expected extra_params in json output, got %#v", modelParams["extra_params"])
|
||||
}
|
||||
if extraParams["number"] != float64(42) {
|
||||
t.Fatalf("unexpected numeric extra param in json output: %#v", extraParams["number"])
|
||||
}
|
||||
nested, ok := extraParams["nested"].(map[string]any)
|
||||
if !ok || nested["enabled"] != true {
|
||||
t.Fatalf("unexpected nested extra param in json output: %#v", extraParams["nested"])
|
||||
}
|
||||
if _, ok := decoded["input_hashes"]; !ok {
|
||||
t.Fatalf("expected input_hashes in json output, got %#v", decoded)
|
||||
}
|
||||
@@ -97,6 +228,47 @@ func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestJSONFormatterIncludesMessageCacheControlOnlyWhenPresent(t *testing.T) {
|
||||
prepared := samplePreparedRun()
|
||||
prepared.Messages = []scriptorium.RenderedMessage{
|
||||
{
|
||||
Role: "system",
|
||||
Content: "System guidance.",
|
||||
CacheControl: &scriptorium.CacheControl{
|
||||
Type: scriptorium.CacheControlEphemeral,
|
||||
TTL: "1h",
|
||||
},
|
||||
},
|
||||
{Role: "user", Content: "Summarize the transcript."},
|
||||
}
|
||||
|
||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
|
||||
var decoded struct {
|
||||
Messages []map[string]any `json:"messages"`
|
||||
}
|
||||
if err := json.Unmarshal(out, &decoded); err != nil {
|
||||
t.Fatalf("expected valid json output, got %v", err)
|
||||
}
|
||||
if len(decoded.Messages) != 2 {
|
||||
t.Fatalf("expected 2 messages, got %d", len(decoded.Messages))
|
||||
}
|
||||
|
||||
cacheControl, ok := decoded.Messages[0]["cache_control"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("expected first message cache_control, got %#v", decoded.Messages[0])
|
||||
}
|
||||
if cacheControl["type"] != string(scriptorium.CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
|
||||
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
|
||||
}
|
||||
if _, ok := decoded.Messages[1]["cache_control"]; ok {
|
||||
t.Fatalf("expected second message to omit cache_control, got %#v", decoded.Messages[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestJSONFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
|
||||
const secret = "super-secret-api-key"
|
||||
t.Setenv("SCRIPTORIUM_API_KEY", secret)
|
||||
@@ -110,6 +282,19 @@ func TestJSONFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestJSONFormatterDoesNotIncludeDirectAPIKeyValue(t *testing.T) {
|
||||
const directKey = "direct-format-key"
|
||||
// PreparedRun intentionally has no field for direct API keys.
|
||||
prepared := samplePreparedRun()
|
||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if strings.Contains(string(out), directKey) {
|
||||
t.Fatalf("json output should not include direct api key value: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParsePreparedRunOutputFormatRecognizesSupportedNames(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
@@ -155,19 +340,20 @@ func TestFormatPreparedRunByNameUnknownFailsClearly(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func samplePreparedRun() *domain.PreparedRun {
|
||||
return &domain.PreparedRun{
|
||||
func samplePreparedRun() *scriptorium.PreparedRun {
|
||||
return &scriptorium.PreparedRun{
|
||||
PromptID: "prompt.id",
|
||||
PromptVersion: "v1",
|
||||
PromptHash: "prompt-hash",
|
||||
SelectedProfileID: "local-fast",
|
||||
EffectiveModelParams: domain.ExecutionTarget{
|
||||
EffectiveModelParams: scriptorium.ExecutionTarget{
|
||||
Endpoint: "http://llm/v1",
|
||||
Model: "gpt-test",
|
||||
Temperature: 0.4,
|
||||
MaxTokens: 256,
|
||||
TopP: 0.8,
|
||||
TimeoutSeconds: 45,
|
||||
ServiceTier: "priority",
|
||||
ReasoningEffort: "medium",
|
||||
APIKeyEnv: "SCRIPTORIUM_API_KEY",
|
||||
},
|
||||
@@ -176,7 +362,7 @@ func samplePreparedRun() *domain.PreparedRun {
|
||||
"glossary": "hash-glossary",
|
||||
},
|
||||
RenderedPromptHash: "rendered-hash",
|
||||
Messages: []domain.RenderedMessage{
|
||||
Messages: []scriptorium.RenderedMessage{
|
||||
{Role: "system", Content: "System guidance."},
|
||||
{Role: "user", Content: "Summarize the transcript.\nInclude key entities."},
|
||||
{Role: "user", Content: "Second user message."},
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||
@@ -35,7 +36,6 @@ type OpenAICompatibleConfig struct {
|
||||
type OpenAICompatibleClient struct {
|
||||
baseURL string
|
||||
defaultModel string
|
||||
timeout time.Duration
|
||||
httpClient *http.Client
|
||||
}
|
||||
|
||||
@@ -54,10 +54,11 @@ func NewOpenAICompatibleClient(cfg OpenAICompatibleConfig) (*OpenAICompatibleCli
|
||||
|
||||
var client *http.Client
|
||||
if cfg.HTTPClient != nil {
|
||||
client = cfg.HTTPClient
|
||||
if client.Timeout == 0 {
|
||||
client.Timeout = timeout
|
||||
cloned := *cfg.HTTPClient
|
||||
if cloned.Timeout <= 0 {
|
||||
cloned.Timeout = timeout
|
||||
}
|
||||
client = &cloned
|
||||
} else {
|
||||
client = &http.Client{Timeout: timeout}
|
||||
}
|
||||
@@ -65,7 +66,6 @@ func NewOpenAICompatibleClient(cfg OpenAICompatibleConfig) (*OpenAICompatibleCli
|
||||
return &OpenAICompatibleClient{
|
||||
baseURL: strings.TrimRight(baseURL, "/"),
|
||||
defaultModel: cfg.Model,
|
||||
timeout: timeout,
|
||||
httpClient: client,
|
||||
}, nil
|
||||
}
|
||||
@@ -75,14 +75,6 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
||||
return nil, fmt.Errorf("%w: timeout_seconds must be greater than or equal to 0", ErrInvalidRequest)
|
||||
}
|
||||
|
||||
model := strings.TrimSpace(req.Target.Model)
|
||||
if model == "" {
|
||||
model = strings.TrimSpace(c.defaultModel)
|
||||
}
|
||||
if model == "" {
|
||||
return nil, fmt.Errorf("%w: model is required", ErrInvalidRequest)
|
||||
}
|
||||
|
||||
endpoint := strings.TrimSpace(req.Target.Endpoint)
|
||||
if endpoint == "" {
|
||||
endpoint = c.baseURL
|
||||
@@ -92,46 +84,39 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
||||
}
|
||||
endpoint = strings.TrimRight(endpoint, "/") + defaults.OpenAIChatCompletionsPath
|
||||
|
||||
wireReq := openAIChatRequest{
|
||||
Model: model,
|
||||
wireReq, err := openAIChatRequestFromGenerateRequest(req, c.defaultModel)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
|
||||
}
|
||||
|
||||
wireReq.Messages = make([]openAIChatMessage, 0, len(req.Prompt.Messages))
|
||||
for _, msg := range req.Prompt.Messages {
|
||||
wireReq.Messages = append(wireReq.Messages, openAIChatMessage{
|
||||
Role: msg.Role,
|
||||
Content: msg.Content,
|
||||
})
|
||||
wirePayload, err := openAIChatRequestPayload(wireReq)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
|
||||
}
|
||||
|
||||
if req.Target.Temperature != 0 {
|
||||
wireReq.Temperature = &req.Target.Temperature
|
||||
}
|
||||
if req.Target.MaxTokens != 0 {
|
||||
wireReq.MaxTokens = &req.Target.MaxTokens
|
||||
}
|
||||
if req.Target.TopP != 0 {
|
||||
wireReq.TopP = &req.Target.TopP
|
||||
}
|
||||
if req.StructuredOutput != nil {
|
||||
responseFormat, err := toOpenAIResponseFormat(req.StructuredOutput)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
|
||||
}
|
||||
wireReq.ResponseFormat = responseFormat
|
||||
}
|
||||
|
||||
payload, err := json.Marshal(wireReq)
|
||||
payload, err := json.Marshal(wirePayload)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: failed to encode request: %v", ErrRequestFailed, err)
|
||||
}
|
||||
|
||||
httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(payload))
|
||||
requestContext := ctx
|
||||
if req.Target.TimeoutSeconds > 0 {
|
||||
var cancel context.CancelFunc
|
||||
requestContext, cancel = context.WithTimeout(
|
||||
ctx,
|
||||
time.Duration(req.Target.TimeoutSeconds)*time.Second,
|
||||
)
|
||||
defer cancel()
|
||||
}
|
||||
|
||||
httpReq, err := http.NewRequestWithContext(requestContext, http.MethodPost, endpoint, bytes.NewReader(payload))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: failed to create request: %v", ErrRequestFailed, err)
|
||||
}
|
||||
httpReq.Header.Set("Content-Type", "application/json")
|
||||
if envName := strings.TrimSpace(req.Target.APIKeyEnv); envName != "" {
|
||||
if apiKey := strings.TrimSpace(req.Target.APIKey); apiKey != "" {
|
||||
httpReq.Header.Set("Authorization", "Bearer "+apiKey)
|
||||
} else if envName := strings.TrimSpace(req.Target.APIKeyEnv); envName != "" {
|
||||
apiKey := strings.TrimSpace(os.Getenv(envName))
|
||||
if apiKey == "" {
|
||||
return nil, fmt.Errorf("%w: api key environment variable %q is not set", ErrInvalidRequest, envName)
|
||||
@@ -139,18 +124,9 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
||||
httpReq.Header.Set("Authorization", "Bearer "+apiKey)
|
||||
}
|
||||
|
||||
effectiveTimeout := c.timeout
|
||||
if req.Target.TimeoutSeconds > 0 {
|
||||
effectiveTimeout = time.Duration(req.Target.TimeoutSeconds) * time.Second
|
||||
}
|
||||
|
||||
httpClient := c.httpClient
|
||||
if httpClient == nil {
|
||||
httpClient = &http.Client{Timeout: effectiveTimeout}
|
||||
} else if httpClient.Timeout != effectiveTimeout {
|
||||
cloned := *httpClient
|
||||
cloned.Timeout = effectiveTimeout
|
||||
httpClient = &cloned
|
||||
httpClient = &http.Client{Timeout: defaults.LLMRequestTimeoutDefault}
|
||||
}
|
||||
|
||||
httpResp, err := httpClient.Do(httpReq)
|
||||
@@ -160,8 +136,8 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
||||
defer httpResp.Body.Close()
|
||||
|
||||
if httpResp.StatusCode < 200 || httpResp.StatusCode >= 300 {
|
||||
body, _ := io.ReadAll(io.LimitReader(httpResp.Body, 4096))
|
||||
return nil, fmt.Errorf("%w: status=%d body=%q", ErrUnexpectedStatus, httpResp.StatusCode, strings.TrimSpace(string(body)))
|
||||
_, _ = io.Copy(io.Discard, io.LimitReader(httpResp.Body, 4096))
|
||||
return nil, fmt.Errorf("%w: status=%d", ErrUnexpectedStatus, httpResp.StatusCode)
|
||||
}
|
||||
|
||||
var wireResp openAIChatResponse
|
||||
@@ -183,32 +159,166 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
||||
PromptTokens: wireResp.Usage.PromptTokens,
|
||||
CompletionTokens: wireResp.Usage.CompletionTokens,
|
||||
TotalTokens: wireResp.Usage.TotalTokens,
|
||||
CachedTokens: wireResp.Usage.PromptTokensDetails.CachedTokens,
|
||||
CacheWriteTokens: wireResp.Usage.CacheWriteTokens,
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type openAIChatRequest struct {
|
||||
Model string `json:"model"`
|
||||
Messages []openAIChatMessage `json:"messages"`
|
||||
Temperature *float64 `json:"temperature,omitempty"`
|
||||
MaxTokens *int `json:"max_tokens,omitempty"`
|
||||
TopP *float64 `json:"top_p,omitempty"`
|
||||
ResponseFormat *openAIResponseFormat `json:"response_format,omitempty"`
|
||||
func openAIChatRequestFromGenerateRequest(req domain.GenerateRequest, defaultModel string) (openAIChatRequest, error) {
|
||||
model := strings.TrimSpace(req.Target.Model)
|
||||
if model == "" {
|
||||
model = strings.TrimSpace(defaultModel)
|
||||
}
|
||||
if model == "" {
|
||||
return openAIChatRequest{}, errors.New("model is required")
|
||||
}
|
||||
|
||||
wireReq := openAIChatRequest{
|
||||
Model: model,
|
||||
}
|
||||
if sessionID := strings.TrimSpace(req.Prompt.SessionID); sessionID != "" {
|
||||
if n := utf8.RuneCountInString(sessionID); n > domain.SessionIDMaxLength {
|
||||
return openAIChatRequest{}, fmt.Errorf("session_id length %d exceeds maximum %d", n, domain.SessionIDMaxLength)
|
||||
}
|
||||
wireReq.SessionID = sessionID
|
||||
}
|
||||
|
||||
wireReq.Messages = make([]openAIChatRequestMessage, 0, len(req.Prompt.Messages))
|
||||
for _, msg := range req.Prompt.Messages {
|
||||
wireReq.Messages = append(wireReq.Messages, openAIChatRequestMessageFromRenderedMessage(msg))
|
||||
}
|
||||
|
||||
if req.Target.Temperature != 0 || req.TargetPresence.Temperature {
|
||||
wireReq.Temperature = &req.Target.Temperature
|
||||
}
|
||||
if req.Target.MaxTokens != 0 || req.TargetPresence.MaxTokens {
|
||||
wireReq.MaxTokens = &req.Target.MaxTokens
|
||||
}
|
||||
if req.Target.TopP != 0 || req.TargetPresence.TopP {
|
||||
wireReq.TopP = &req.Target.TopP
|
||||
}
|
||||
if strings.TrimSpace(req.Target.ServiceTier) != "" {
|
||||
wireReq.ServiceTier = req.Target.ServiceTier
|
||||
}
|
||||
if strings.TrimSpace(req.Target.ReasoningEffort) != "" {
|
||||
wireReq.ReasoningEffort = req.Target.ReasoningEffort
|
||||
}
|
||||
if len(req.Target.ExtraParams) > 0 {
|
||||
wireReq.ExtraParams = req.Target.ExtraParams
|
||||
}
|
||||
if req.StructuredOutput != nil {
|
||||
responseFormat, err := toOpenAIResponseFormat(req.StructuredOutput)
|
||||
if err != nil {
|
||||
return openAIChatRequest{}, err
|
||||
}
|
||||
wireReq.ResponseFormat = responseFormat
|
||||
}
|
||||
|
||||
return wireReq, nil
|
||||
}
|
||||
|
||||
type openAIChatMessage struct {
|
||||
type openAIChatRequest struct {
|
||||
Model string `json:"model"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
Messages []openAIChatRequestMessage `json:"messages"`
|
||||
Temperature *float64 `json:"temperature,omitempty"`
|
||||
MaxTokens *int `json:"max_tokens,omitempty"`
|
||||
TopP *float64 `json:"top_p,omitempty"`
|
||||
ServiceTier string `json:"service_tier,omitempty"`
|
||||
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||
ResponseFormat *openAIResponseFormat `json:"response_format,omitempty"`
|
||||
ExtraParams map[string]any `json:"-"`
|
||||
}
|
||||
|
||||
func openAIChatRequestPayload(req openAIChatRequest) (map[string]any, error) {
|
||||
out := map[string]any{
|
||||
"model": req.Model,
|
||||
"messages": req.Messages,
|
||||
}
|
||||
if req.SessionID != "" {
|
||||
out["session_id"] = req.SessionID
|
||||
}
|
||||
if req.Temperature != nil {
|
||||
out["temperature"] = *req.Temperature
|
||||
}
|
||||
if req.MaxTokens != nil {
|
||||
out["max_tokens"] = *req.MaxTokens
|
||||
}
|
||||
if req.TopP != nil {
|
||||
out["top_p"] = *req.TopP
|
||||
}
|
||||
if req.ServiceTier != "" {
|
||||
out["service_tier"] = req.ServiceTier
|
||||
}
|
||||
if req.ReasoningEffort != "" {
|
||||
out["reasoning_effort"] = req.ReasoningEffort
|
||||
}
|
||||
if req.ResponseFormat != nil {
|
||||
out["response_format"] = req.ResponseFormat
|
||||
}
|
||||
|
||||
for key, value := range req.ExtraParams {
|
||||
if key == "" {
|
||||
return nil, errors.New("extra_params key must not be empty")
|
||||
}
|
||||
if _, reserved := reservedOpenAIChatRequestFields[key]; reserved {
|
||||
return nil, fmt.Errorf("extra_params key %q collides with reserved request field", key)
|
||||
}
|
||||
if _, err := json.Marshal(value); err != nil {
|
||||
return nil, fmt.Errorf("extra_params.%s must be JSON-serializable: %w", key, err)
|
||||
}
|
||||
out[key] = value
|
||||
}
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
var reservedOpenAIChatRequestFields = map[string]struct{}{
|
||||
"model": {},
|
||||
"session_id": {},
|
||||
"messages": {},
|
||||
"temperature": {},
|
||||
"max_tokens": {},
|
||||
"top_p": {},
|
||||
"service_tier": {},
|
||||
"reasoning_effort": {},
|
||||
"response_format": {},
|
||||
}
|
||||
|
||||
type openAIChatRequestMessage struct {
|
||||
Role string `json:"role"`
|
||||
Content any `json:"content"`
|
||||
}
|
||||
|
||||
type openAIChatTextContentBlock struct {
|
||||
Type string `json:"type"`
|
||||
Text string `json:"text"`
|
||||
CacheControl *openAICacheControl `json:"cache_control,omitempty"`
|
||||
}
|
||||
|
||||
type openAICacheControl struct {
|
||||
Type string `json:"type"`
|
||||
TTL string `json:"ttl,omitempty"`
|
||||
}
|
||||
|
||||
type openAIChatResponseMessage struct {
|
||||
Role string `json:"role"`
|
||||
Content string `json:"content"`
|
||||
}
|
||||
|
||||
type openAIChatResponse struct {
|
||||
Choices []struct {
|
||||
Message openAIChatMessage `json:"message"`
|
||||
Message openAIChatResponseMessage `json:"message"`
|
||||
} `json:"choices"`
|
||||
Usage struct {
|
||||
PromptTokens int `json:"prompt_tokens"`
|
||||
CompletionTokens int `json:"completion_tokens"`
|
||||
TotalTokens int `json:"total_tokens"`
|
||||
PromptTokens int `json:"prompt_tokens"`
|
||||
CompletionTokens int `json:"completion_tokens"`
|
||||
TotalTokens int `json:"total_tokens"`
|
||||
PromptTokensDetails struct {
|
||||
CachedTokens int `json:"cached_tokens"`
|
||||
} `json:"prompt_tokens_details"`
|
||||
CacheWriteTokens int `json:"cache_write_tokens"`
|
||||
} `json:"usage"`
|
||||
}
|
||||
|
||||
@@ -223,6 +333,28 @@ type openAIJSONSchemaEnvelope struct {
|
||||
Schema any `json:"schema"`
|
||||
}
|
||||
|
||||
func openAIChatRequestMessageFromRenderedMessage(msg domain.RenderedMessage) openAIChatRequestMessage {
|
||||
wireMsg := openAIChatRequestMessage{
|
||||
Role: msg.Role,
|
||||
Content: msg.Content,
|
||||
}
|
||||
if msg.CacheControl == nil {
|
||||
return wireMsg
|
||||
}
|
||||
|
||||
wireMsg.Content = []openAIChatTextContentBlock{
|
||||
{
|
||||
Type: "text",
|
||||
Text: msg.Content,
|
||||
CacheControl: &openAICacheControl{
|
||||
Type: string(msg.CacheControl.Type),
|
||||
TTL: msg.CacheControl.TTL,
|
||||
},
|
||||
},
|
||||
}
|
||||
return wireMsg
|
||||
}
|
||||
|
||||
func toOpenAIResponseFormat(spec *domain.StructuredOutputSpec) (*openAIResponseFormat, error) {
|
||||
if spec == nil {
|
||||
return nil, nil
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"math"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
@@ -13,6 +14,91 @@ import (
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||
)
|
||||
|
||||
func TestNewOpenAICompatibleClientDoesNotMutateSuppliedZeroTimeoutClient(t *testing.T) {
|
||||
transport := http.DefaultTransport
|
||||
supplied := &http.Client{Transport: transport}
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
|
||||
HTTPClient: supplied,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected constructor error: %v", err)
|
||||
}
|
||||
|
||||
if supplied.Timeout != 0 {
|
||||
t.Fatalf("expected supplied client timeout to remain zero, got %v", supplied.Timeout)
|
||||
}
|
||||
if client.httpClient == supplied {
|
||||
t.Fatal("expected constructed client to use a cloned HTTP client")
|
||||
}
|
||||
if client.httpClient.Timeout <= 0 {
|
||||
t.Fatalf("expected constructed client to use a positive default timeout, got %v", client.httpClient.Timeout)
|
||||
}
|
||||
if client.httpClient.Transport != transport {
|
||||
t.Fatal("expected cloned client to preserve the supplied transport")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewOpenAICompatibleClientDoesNotMutateSuppliedNonzeroTimeoutClient(t *testing.T) {
|
||||
transport := http.DefaultTransport
|
||||
suppliedTimeout := 37 * time.Second
|
||||
supplied := &http.Client{
|
||||
Timeout: suppliedTimeout,
|
||||
Transport: transport,
|
||||
}
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
|
||||
Timeout: 2 * time.Second,
|
||||
HTTPClient: supplied,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected constructor error: %v", err)
|
||||
}
|
||||
|
||||
if supplied.Timeout != suppliedTimeout {
|
||||
t.Fatalf("expected supplied client timeout to remain %v, got %v", suppliedTimeout, supplied.Timeout)
|
||||
}
|
||||
if client.httpClient == supplied {
|
||||
t.Fatal("expected constructed client to use a cloned HTTP client")
|
||||
}
|
||||
if client.httpClient.Timeout != suppliedTimeout {
|
||||
t.Fatalf("expected cloned client timeout %v, got %v", suppliedTimeout, client.httpClient.Timeout)
|
||||
}
|
||||
if client.httpClient.Transport != transport {
|
||||
t.Fatal("expected cloned client to preserve the supplied transport")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewOpenAICompatibleClientTreatsSuppliedNegativeTimeoutAsUnset(t *testing.T) {
|
||||
transport := http.DefaultTransport
|
||||
supplied := &http.Client{
|
||||
Timeout: -time.Second,
|
||||
Transport: transport,
|
||||
}
|
||||
configuredTimeout := 23 * time.Second
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
|
||||
Timeout: configuredTimeout,
|
||||
HTTPClient: supplied,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected constructor error: %v", err)
|
||||
}
|
||||
|
||||
if supplied.Timeout != -time.Second {
|
||||
t.Fatalf("expected supplied client timeout to remain negative, got %v", supplied.Timeout)
|
||||
}
|
||||
if client.httpClient == supplied {
|
||||
t.Fatal("expected constructed client to use a cloned HTTP client")
|
||||
}
|
||||
if client.httpClient.Timeout != configuredTimeout {
|
||||
t.Fatalf("expected cloned client timeout %v, got %v", configuredTimeout, client.httpClient.Timeout)
|
||||
}
|
||||
if client.httpClient.Transport != transport {
|
||||
t.Fatal("expected cloned client to preserve the supplied transport")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
|
||||
type observedRequest struct {
|
||||
Authorization string
|
||||
@@ -61,6 +147,7 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
|
||||
Temperature: 0.4,
|
||||
MaxTokens: 123,
|
||||
TopP: 0.7,
|
||||
ServiceTier: "priority",
|
||||
APIKeyEnv: "SCRIPTORIUM_TEST_API_KEY",
|
||||
},
|
||||
StructuredOutput: &domain.StructuredOutputSpec{
|
||||
@@ -88,6 +175,9 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
|
||||
if resp.Usage.PromptTokens != 11 || resp.Usage.CompletionTokens != 22 || resp.Usage.TotalTokens != 33 {
|
||||
t.Fatalf("unexpected usage: %+v", resp.Usage)
|
||||
}
|
||||
if resp.Usage.CachedTokens != 0 || resp.Usage.CacheWriteTokens != 0 {
|
||||
t.Fatalf("expected absent cache usage fields to remain zero, got %+v", resp.Usage)
|
||||
}
|
||||
|
||||
if obs.Authorization != "Bearer secret-key" {
|
||||
t.Fatalf("unexpected Authorization header: %q", obs.Authorization)
|
||||
@@ -95,6 +185,18 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
|
||||
if got, ok := obs.Body["model"].(string); !ok || got != "gpt-test" {
|
||||
t.Fatalf("unexpected model payload: %#v", obs.Body["model"])
|
||||
}
|
||||
if got, ok := obs.Body["temperature"].(float64); !ok || got != 0.4 {
|
||||
t.Fatalf("unexpected temperature payload: %#v", obs.Body["temperature"])
|
||||
}
|
||||
if got, ok := obs.Body["max_tokens"].(float64); !ok || got != 123 {
|
||||
t.Fatalf("unexpected max_tokens payload: %#v", obs.Body["max_tokens"])
|
||||
}
|
||||
if got, ok := obs.Body["top_p"].(float64); !ok || got != 0.7 {
|
||||
t.Fatalf("unexpected top_p payload: %#v", obs.Body["top_p"])
|
||||
}
|
||||
if got, ok := obs.Body["service_tier"].(string); !ok || got != "priority" {
|
||||
t.Fatalf("unexpected service_tier payload: %#v", obs.Body["service_tier"])
|
||||
}
|
||||
|
||||
msgs, ok := obs.Body["messages"].([]any)
|
||||
if !ok || len(msgs) != 2 {
|
||||
@@ -131,6 +233,277 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientDirectAPIKeyPreferredOverEnv(t *testing.T) {
|
||||
const directKey = "direct-llm-key"
|
||||
t.Setenv("SCRIPTORIUM_TEST_API_KEY", "env-key")
|
||||
|
||||
var gotAuth string
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
gotAuth = r.Header.Get("Authorization")
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||
Target: domain.ExecutionTarget{
|
||||
Model: "model",
|
||||
APIKeyEnv: "SCRIPTORIUM_TEST_API_KEY",
|
||||
APIKey: directKey,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if gotAuth != "Bearer "+directKey {
|
||||
t.Fatalf("unexpected Authorization header: %q", gotAuth)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientSerializesCacheControlledMessageAsContentBlock(t *testing.T) {
|
||||
var observedBody map[string]any
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
defer r.Body.Close()
|
||||
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||
t.Fatalf("failed to decode request body: %v", err)
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{
|
||||
{
|
||||
Role: "system",
|
||||
Content: "Stable instructions.",
|
||||
CacheControl: &domain.CacheControl{
|
||||
Type: domain.CacheControlEphemeral,
|
||||
TTL: "1h",
|
||||
},
|
||||
},
|
||||
{Role: "user", Content: "Dynamic request."},
|
||||
}},
|
||||
Target: domain.ExecutionTarget{Model: "model"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
|
||||
for _, forbidden := range []string{"cache_control", "extra_params"} {
|
||||
if _, exists := observedBody[forbidden]; exists {
|
||||
t.Fatalf("expected top-level %s to be omitted, got %#v", forbidden, observedBody[forbidden])
|
||||
}
|
||||
}
|
||||
|
||||
msgs, ok := observedBody["messages"].([]any)
|
||||
if !ok || len(msgs) != 2 {
|
||||
t.Fatalf("unexpected messages payload: %#v", observedBody["messages"])
|
||||
}
|
||||
msg0 := msgs[0].(map[string]any)
|
||||
if msg0["role"] != "system" {
|
||||
t.Fatalf("unexpected first message role: %#v", msg0["role"])
|
||||
}
|
||||
contentBlocks, ok := msg0["content"].([]any)
|
||||
if !ok || len(contentBlocks) != 1 {
|
||||
t.Fatalf("expected first message content block array, got %#v", msg0["content"])
|
||||
}
|
||||
block := contentBlocks[0].(map[string]any)
|
||||
if block["type"] != "text" || block["text"] != "Stable instructions." {
|
||||
t.Fatalf("unexpected text content block: %#v", block)
|
||||
}
|
||||
cacheControl, ok := block["cache_control"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("expected cache_control on content block, got %#v", block)
|
||||
}
|
||||
if cacheControl["type"] != string(domain.CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
|
||||
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
|
||||
}
|
||||
|
||||
msg1 := msgs[1].(map[string]any)
|
||||
if msg1["role"] != "user" || msg1["content"] != "Dynamic request." {
|
||||
t.Fatalf("expected uncached message to keep string content, got %#v", msg1)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientOmitsEmptyCacheControlTTL(t *testing.T) {
|
||||
var observedBody map[string]any
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
defer r.Body.Close()
|
||||
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||
t.Fatalf("failed to decode request body: %v", err)
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{
|
||||
{
|
||||
Role: "system",
|
||||
Content: "Stable instructions.",
|
||||
CacheControl: &domain.CacheControl{
|
||||
Type: domain.CacheControlEphemeral,
|
||||
},
|
||||
},
|
||||
}},
|
||||
Target: domain.ExecutionTarget{Model: "model"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
|
||||
msgs := observedBody["messages"].([]any)
|
||||
msg0 := msgs[0].(map[string]any)
|
||||
contentBlocks := msg0["content"].([]any)
|
||||
block := contentBlocks[0].(map[string]any)
|
||||
cacheControl := block["cache_control"].(map[string]any)
|
||||
if cacheControl["type"] != string(domain.CacheControlEphemeral) {
|
||||
t.Fatalf("unexpected cache_control type: %#v", cacheControl)
|
||||
}
|
||||
if _, exists := cacheControl["ttl"]; exists {
|
||||
t.Fatalf("expected empty ttl to be omitted, got %#v", cacheControl)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientSerializesSessionID(t *testing.T) {
|
||||
var observedBody map[string]any
|
||||
var observedSessionHeader string
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
observedSessionHeader = r.Header.Get("x-session-id")
|
||||
defer r.Body.Close()
|
||||
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||
t.Fatalf("failed to decode request body: %v", err)
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{
|
||||
SessionID: " session-123 ",
|
||||
Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}},
|
||||
},
|
||||
Target: domain.ExecutionTarget{Model: "model"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if observedBody["session_id"] != "session-123" {
|
||||
t.Fatalf("expected top-level session_id, got %#v", observedBody["session_id"])
|
||||
}
|
||||
if observedSessionHeader != "" {
|
||||
t.Fatalf("did not expect x-session-id header, got %q", observedSessionHeader)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientOmitsEmptySessionID(t *testing.T) {
|
||||
var observedBody map[string]any
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
defer r.Body.Close()
|
||||
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||
t.Fatalf("failed to decode request body: %v", err)
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{
|
||||
SessionID: " ",
|
||||
Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}},
|
||||
},
|
||||
Target: domain.ExecutionTarget{Model: "model"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if _, exists := observedBody["session_id"]; exists {
|
||||
t.Fatalf("expected empty session_id to be omitted, got %#v", observedBody["session_id"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientRejectsTooLongSessionID(t *testing.T) {
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
|
||||
BaseURL: "http://example.com/v1",
|
||||
Model: "model",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{
|
||||
SessionID: strings.Repeat("x", domain.SessionIDMaxLength+1),
|
||||
Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}},
|
||||
},
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("expected invalid request error")
|
||||
}
|
||||
if !errors.Is(err, ErrInvalidRequest) {
|
||||
t.Fatalf("expected ErrInvalidRequest, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientParsesCacheUsage(t *testing.T) {
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
_, _ = w.Write([]byte(`{
|
||||
"choices": [{"message": {"role": "assistant", "content": "ok"}}],
|
||||
"usage": {
|
||||
"prompt_tokens": 100,
|
||||
"completion_tokens": 20,
|
||||
"total_tokens": 120,
|
||||
"prompt_tokens_details": {"cached_tokens": 80},
|
||||
"cache_write_tokens": 60
|
||||
}
|
||||
}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1", Model: "model"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
resp, err := client.Generate(context.Background(), domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if resp.Usage.PromptTokens != 100 || resp.Usage.CompletionTokens != 20 || resp.Usage.TotalTokens != 120 {
|
||||
t.Fatalf("unexpected base usage fields: %+v", resp.Usage)
|
||||
}
|
||||
if resp.Usage.CachedTokens != 80 || resp.Usage.CacheWriteTokens != 60 {
|
||||
t.Fatalf("unexpected cache usage fields: %+v", resp.Usage)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientOmitsResponseFormatWhenNoStructuredOutput(t *testing.T) {
|
||||
var observedBody map[string]any
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -157,6 +530,259 @@ func TestOpenAICompatibleClientOmitsResponseFormatWhenNoStructuredOutput(t *test
|
||||
if _, exists := observedBody["response_format"]; exists {
|
||||
t.Fatalf("expected response_format omitted, got %#v", observedBody["response_format"])
|
||||
}
|
||||
if _, exists := observedBody["service_tier"]; exists {
|
||||
t.Fatalf("expected service_tier omitted, got %#v", observedBody["service_tier"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientSerializesReasoningEffortAndExtraParams(t *testing.T) {
|
||||
var observedBody map[string]any
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
defer r.Body.Close()
|
||||
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||
t.Fatalf("failed to decode request body: %v", err)
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||
Target: domain.ExecutionTarget{
|
||||
Model: "model",
|
||||
ReasoningEffort: "high",
|
||||
ExtraParams: map[string]any{
|
||||
"string_value": "on",
|
||||
"number_value": 42,
|
||||
"boolean_value": true,
|
||||
"object_value": map[string]any{"nested": "value", "count": 2},
|
||||
"array_value": []any{"first", 3, false},
|
||||
},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if observedBody["reasoning_effort"] != "high" {
|
||||
t.Fatalf("expected reasoning_effort high, got %#v", observedBody["reasoning_effort"])
|
||||
}
|
||||
if observedBody["string_value"] != "on" {
|
||||
t.Fatalf("unexpected string extra param: %#v", observedBody["string_value"])
|
||||
}
|
||||
if observedBody["number_value"] != float64(42) {
|
||||
t.Fatalf("unexpected number extra param: %#v", observedBody["number_value"])
|
||||
}
|
||||
if observedBody["boolean_value"] != true {
|
||||
t.Fatalf("unexpected boolean extra param: %#v", observedBody["boolean_value"])
|
||||
}
|
||||
objectValue, ok := observedBody["object_value"].(map[string]any)
|
||||
if !ok || objectValue["nested"] != "value" || objectValue["count"] != float64(2) {
|
||||
t.Fatalf("unexpected object extra param: %#v", observedBody["object_value"])
|
||||
}
|
||||
if _, exists := observedBody["extra_params"]; exists {
|
||||
t.Fatalf("expected extra_params wrapper omitted, got %#v", observedBody["extra_params"])
|
||||
}
|
||||
arrayValue, ok := observedBody["array_value"].([]any)
|
||||
if !ok || len(arrayValue) != 3 || arrayValue[0] != "first" || arrayValue[1] != float64(3) || arrayValue[2] != false {
|
||||
t.Fatalf("unexpected array extra param: %#v", observedBody["array_value"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientOmitsReasoningEffortWhenUnset(t *testing.T) {
|
||||
var observedBody map[string]any
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
defer r.Body.Close()
|
||||
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||
t.Fatalf("failed to decode request body: %v", err)
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||
Target: domain.ExecutionTarget{Model: "model"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if _, exists := observedBody["reasoning_effort"]; exists {
|
||||
t.Fatalf("expected reasoning_effort omitted, got %#v", observedBody["reasoning_effort"])
|
||||
}
|
||||
if _, exists := observedBody["extra_params"]; exists {
|
||||
t.Fatalf("expected extra_params wrapper omitted, got %#v", observedBody["extra_params"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientSerializesExplicitZeroNumericOverrides(t *testing.T) {
|
||||
var observedBody map[string]any
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
defer r.Body.Close()
|
||||
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||
t.Fatalf("failed to decode request body: %v", err)
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||
Target: domain.ExecutionTarget{Model: "model"},
|
||||
TargetPresence: domain.ExecutionTargetPresence{
|
||||
Temperature: true,
|
||||
MaxTokens: true,
|
||||
TopP: true,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if observedBody["temperature"] != float64(0) {
|
||||
t.Fatalf("expected explicit zero temperature, got %#v", observedBody["temperature"])
|
||||
}
|
||||
if observedBody["max_tokens"] != float64(0) {
|
||||
t.Fatalf("expected explicit zero max_tokens, got %#v", observedBody["max_tokens"])
|
||||
}
|
||||
if observedBody["top_p"] != float64(0) {
|
||||
t.Fatalf("expected explicit zero top_p, got %#v", observedBody["top_p"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientOmitsImplicitZeroNumericFields(t *testing.T) {
|
||||
var observedBody map[string]any
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
defer r.Body.Close()
|
||||
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||
t.Fatalf("failed to decode request body: %v", err)
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||
Target: domain.ExecutionTarget{Model: "model"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
for _, field := range []string{"temperature", "max_tokens", "top_p"} {
|
||||
if _, exists := observedBody[field]; exists {
|
||||
t.Fatalf("expected implicit zero field %q to be omitted, got body %#v", field, observedBody)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientOmittedTimeoutUsesClientTimeout(t *testing.T) {
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
|
||||
BaseURL: ts.URL + "/v1",
|
||||
Timeout: time.Nanosecond,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||
Target: domain.ExecutionTarget{Model: "model", TimeoutSeconds: 0},
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("expected omitted timeout to use client timeout")
|
||||
}
|
||||
if !errors.Is(err, ErrRequestFailed) {
|
||||
t.Fatalf("expected ErrRequestFailed, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientRejectsInvalidExtraParamsBeforeProviderCall(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
extraParams map[string]any
|
||||
want string
|
||||
}{
|
||||
{name: "empty key", extraParams: map[string]any{"": "empty"}, want: "key must not be empty"},
|
||||
{name: "unserializable value", extraParams: map[string]any{"bad": math.Inf(1)}, want: "JSON-serializable"},
|
||||
}
|
||||
for _, key := range []string{
|
||||
"model",
|
||||
"session_id",
|
||||
"messages",
|
||||
"temperature",
|
||||
"max_tokens",
|
||||
"top_p",
|
||||
"service_tier",
|
||||
"reasoning_effort",
|
||||
"response_format",
|
||||
} {
|
||||
tests = append(tests, struct {
|
||||
name string
|
||||
extraParams map[string]any
|
||||
want string
|
||||
}{
|
||||
name: "reserved key " + key,
|
||||
extraParams: map[string]any{key: "collision"},
|
||||
want: "reserved request field",
|
||||
})
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
called := false
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
called = true
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||
Target: domain.ExecutionTarget{Model: "model", ExtraParams: tc.extraParams},
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("expected invalid request error")
|
||||
}
|
||||
if !errors.Is(err, ErrInvalidRequest) {
|
||||
t.Fatalf("expected ErrInvalidRequest, got %v", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), tc.want) {
|
||||
t.Fatalf("expected error to contain %q, got %v", tc.want, err)
|
||||
}
|
||||
if called {
|
||||
t.Fatal("provider should not be called for invalid extra_params")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientNoAuthorizationHeaderWhenNoAPIKey(t *testing.T) {
|
||||
@@ -279,9 +905,10 @@ func TestOpenAICompatibleClientEndpointOverride(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientNon2xxError(t *testing.T) {
|
||||
const sensitiveBody = `provider-secret-fragment request_payload_details`
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
_, _ = w.Write([]byte(`{"error":"bad request payload"}`))
|
||||
_, _ = w.Write([]byte(`{"error":"` + sensitiveBody + `"}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
@@ -299,8 +926,11 @@ func TestOpenAICompatibleClientNon2xxError(t *testing.T) {
|
||||
if !errors.Is(err, ErrUnexpectedStatus) {
|
||||
t.Fatalf("expected ErrUnexpectedStatus, got %v", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "400") || !strings.Contains(err.Error(), "bad request payload") {
|
||||
t.Fatalf("expected status/body details, got %v", err)
|
||||
if !strings.Contains(err.Error(), "status=400") {
|
||||
t.Fatalf("expected status detail, got %v", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), sensitiveBody) {
|
||||
t.Fatalf("expected provider response body to be redacted, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -375,34 +1005,6 @@ func TestOpenAICompatibleClientTimeout(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientRequestTimeoutOverride(t *testing.T) {
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
|
||||
BaseURL: ts.URL + "/v1",
|
||||
Model: "m",
|
||||
Timeout: 50 * time.Millisecond,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
resp, err := client.Generate(context.Background(), domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||
Target: domain.ExecutionTarget{TimeoutSeconds: 1},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected request-level timeout override to succeed, got %v", err)
|
||||
}
|
||||
if resp.Content != "ok" {
|
||||
t.Fatalf("expected response content ok, got %q", resp.Content)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientNegativeTimeoutRejected(t *testing.T) {
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
|
||||
BaseURL: "http://example.com/v1",
|
||||
|
||||
9
internal/profile/builtin/assets/aion-labs/aion-2.yml
Normal file
9
internal/profile/builtin/assets/aion-labs/aion-2.yml
Normal file
@@ -0,0 +1,9 @@
|
||||
id: aion-2
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: aion-labs/aion-2.0
|
||||
temperature: 0.72
|
||||
reasoning_effort: high
|
||||
top_p: 0.95
|
||||
timeout_seconds: 180
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
@@ -0,0 +1,7 @@
|
||||
id: claude-fable-latest
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: "~anthropic/claude-fable-latest"
|
||||
reasoning_effort: high
|
||||
timeout_seconds: 600
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
@@ -0,0 +1,7 @@
|
||||
id: claude-haiku-latest
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: "~anthropic/claude-haiku-latest"
|
||||
reasoning_effort: medium
|
||||
timeout_seconds: 240
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
@@ -0,0 +1,7 @@
|
||||
id: claude-opus-latest
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: "~anthropic/claude-opus-latest"
|
||||
reasoning_effort: high
|
||||
timeout_seconds: 240
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
@@ -0,0 +1,7 @@
|
||||
id: claude-sonnet-latest
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: "~anthropic/claude-sonnet-latest"
|
||||
reasoning_effort: high
|
||||
timeout_seconds: 240
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
@@ -0,0 +1,7 @@
|
||||
id: deepseek-3-2
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: deepseek/deepseek-v3.2
|
||||
reasoning_effort: high
|
||||
timeout_seconds: 180
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
@@ -0,0 +1,7 @@
|
||||
id: deepseek-4-flash
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: deepseek/deepseek-v4-flash
|
||||
#reasoning_effort: medium
|
||||
timeout_seconds: 180
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
@@ -0,0 +1,7 @@
|
||||
id: deepseek-4-pro
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: deepseek/deepseek-v4-pro
|
||||
reasoning_effort: high
|
||||
timeout_seconds: 180
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
@@ -0,0 +1,9 @@
|
||||
id: gemini-2-flash-lite
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: "google/gemini-2.5-flash-lite"
|
||||
#temperature: 0.15
|
||||
reasoning_effort: high
|
||||
#top_p: 0.98
|
||||
timeout_seconds: 240
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
@@ -0,0 +1,9 @@
|
||||
id: gemini-2-flash
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: "google/gemini-2.5-flash"
|
||||
#temperature: 0.15
|
||||
reasoning_effort: high
|
||||
#top_p: 0.98
|
||||
timeout_seconds: 240
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
9
internal/profile/builtin/assets/google/gemini-2-pro.yml
Normal file
9
internal/profile/builtin/assets/google/gemini-2-pro.yml
Normal file
@@ -0,0 +1,9 @@
|
||||
id: gemini-2-pro
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: "google/gemini-2.5-pro"
|
||||
#temperature: 0.15
|
||||
reasoning_effort: high
|
||||
#top_p: 0.98
|
||||
timeout_seconds: 240
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
@@ -0,0 +1,9 @@
|
||||
id: gemini-3-flash-lite
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: "google/gemini-3.1-flash-lite"
|
||||
#temperature: 0.15
|
||||
reasoning_effort: high
|
||||
#top_p: 0.98
|
||||
timeout_seconds: 240
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
@@ -0,0 +1,9 @@
|
||||
id: gemini-flash-latest
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: "~google/gemini-flash-latest"
|
||||
#temperature: 0.15
|
||||
reasoning_effort: high
|
||||
#top_p: 0.98
|
||||
timeout_seconds: 240
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
@@ -0,0 +1,9 @@
|
||||
id: gemini-pro-latest
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: "~google/gemini-pro-latest"
|
||||
#temperature: 0.15
|
||||
reasoning_effort: high
|
||||
#top_p: 0.98
|
||||
timeout_seconds: 240
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
9
internal/profile/builtin/assets/google/gemma-4-31b.yml
Normal file
9
internal/profile/builtin/assets/google/gemma-4-31b.yml
Normal file
@@ -0,0 +1,9 @@
|
||||
id: gemma-4-31b
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: google/gemma-4-31b-it:exacto
|
||||
temperature: 0.15
|
||||
reasoning_effort: high
|
||||
top_p: 0.98
|
||||
timeout_seconds: 240
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
9
internal/profile/builtin/assets/minimax/minimax-m2.yml
Normal file
9
internal/profile/builtin/assets/minimax/minimax-m2.yml
Normal file
@@ -0,0 +1,9 @@
|
||||
id: minimax-m2
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: minimax/minimax-m2.5
|
||||
temperature: 0.5
|
||||
reasoning_effort: high
|
||||
top_p: 0.95
|
||||
timeout_seconds: 180
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
9
internal/profile/builtin/assets/minimax/minimax-m3.yml
Normal file
9
internal/profile/builtin/assets/minimax/minimax-m3.yml
Normal file
@@ -0,0 +1,9 @@
|
||||
id: minimax-m3
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: minimax/minimax-m3
|
||||
#temperature: 0.5
|
||||
reasoning_effort: high
|
||||
#top_p: 0.95
|
||||
timeout_seconds: 180
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
service_tier: flex
|
||||
@@ -0,0 +1,7 @@
|
||||
id: mistral-large-2512
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: mistralai/mistral-large-2512
|
||||
temperature: 0.15
|
||||
top_p: 0.98
|
||||
timeout_seconds: 180
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
@@ -0,0 +1,8 @@
|
||||
id: mistral-medium-3-5
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: mistralai/mistral-medium-3-5
|
||||
temperature: 0.15
|
||||
reasoning_effort: high
|
||||
top_p: 0.98
|
||||
timeout_seconds: 180
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
@@ -0,0 +1,7 @@
|
||||
id: mistral-small-3
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: mistralai/mistral-small-3.2-24b-instruct
|
||||
temperature: 0.05
|
||||
top_p: 1.0
|
||||
timeout_seconds: 180
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
@@ -0,0 +1,8 @@
|
||||
id: mistral-small-4
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
model: mistralai/mistral-small-2603
|
||||
temperature: 0.1
|
||||
reasoning_effort: high
|
||||
top_p: 0.98
|
||||
timeout_seconds: 180
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user