40 Commits

Author SHA1 Message Date
03d4f27d2b Document public library usage 2026-07-04 14:24:06 +00:00
4ac2038331 Add public run API with injectable LLM 2026-07-04 14:21:37 +00:00
14a7e7e04c Add public prepare engine API 2026-07-04 14:16:19 +00:00
5e522bad8b Add a roadmap and implementation plan for an initial public library package 2026-07-04 09:09:29 -05:00
23872dd742 Implement runtime parameter completion fixes
All checks were successful
ci/woodpecker/tag/release Pipeline was successful
2026-07-04 09:00:19 -05:00
7ffbf5f6ca Update woodpecker config to prepare only linux binaries 2026-07-04 08:53:30 -05:00
d0dc30fcc9 Document runtime provider parameters 2026-07-04 13:29:38 +00:00
b38f7b4dc3 Serialize runtime extra parameters outbound 2026-07-04 13:26:53 +00:00
0512995931 Allow JSON-compatible extra params 2026-07-04 13:24:27 +00:00
049a5feadb Make request execution overrides presence-aware 2026-07-04 13:20:39 +00:00
1798e9c575 Add a roadmap and implementation plan to support reasoning_effort and extra_params in outbound requests 2026-07-04 08:13:10 -05:00
5d4bc8c2b9 Remove completed feature roadmap docs 2026-07-02 20:12:10 -05:00
63fb8fc132 Implement support for OpenRouter sticky routing via a session_id variable 2026-07-02 20:08:44 -05:00
4d4bb7a121 Document prompt cache control behavior 2026-07-02 23:10:24 +00:00
5dcb3cd4fc Expose cache usage in adapters 2026-07-02 23:07:57 +00:00
efe346893c Serialize cache-controlled chat messages 2026-07-02 23:05:56 +00:00
c95d6fcfec Preserve cache control in rendered prompts 2026-07-02 23:03:50 +00:00
0badb4364d Add prompt cache control loading 2026-07-02 23:00:43 +00:00
1f63f8afbb Add a feature roadmap and implementation plan for cache_control values 2026-07-02 17:55:56 -05:00
bc099a31ad Finish cleanup roadmap follow-through
All checks were successful
ci/woodpecker/tag/release Pipeline was successful
2026-05-26 11:05:08 -05:00
4ff55221a3 Update internal docs for stable runner error reasons and serialized model fields 2026-05-26 15:00:06 +00:00
8d8024099f Complete final cleanup verification and align unsupported artifact test naming 2026-05-26 13:23:45 +00:00
18792fd8d1 Remove unsupported S3 artifact reference placeholder 2026-05-26 13:21:52 +00:00
8860aa033c Reduce CLI test fixture duplication with local setup helpers 2026-05-26 13:20:07 +00:00
3ca14d8b6e Add regression test for JSON schema load failure during prepare 2026-05-26 13:16:26 +00:00
099e9c4a3e Use stable usecase sentinels for HTTP invalid-request mapping 2026-05-26 13:14:57 +00:00
cfe6b9408a Refactor CLI command wiring with shared settings and runner helpers 2026-05-26 13:12:34 +00:00
6ececc749f Centralize YAML catalog scanning for prompt and profile repositories 2026-05-26 13:10:09 +00:00
79901fbb86 Refine execution target mapping helpers and coverage across usecase, HTTP, and LLM 2026-05-26 13:07:35 +00:00
75fa0a030a Added a roadmap to address the issues identifed by the audit 2026-05-26 08:01:33 -05:00
ef64966897 Audit code quality and deduplication opportunities 2026-05-26 07:52:48 -05:00
c6c5e3cb69 Added support for the service_tier key in profiles 2026-05-26 07:44:49 -05:00
3f4fd230b9 Implemented support for loading configuration from nested subdirectories 2026-05-26 07:36:10 -05:00
2091b58066 Completed the documentation update and removed the completed roadmap 2026-05-26 07:15:36 -05:00
c3fe88c9fa Add maintained examples and clean up legacy documentation paths 2026-05-26 03:33:25 +00:00
5830fda516 Add HTTP and OpenAI integration documentation and rewrite Narratio contract 2026-05-26 03:31:01 +00:00
359e910572 Add development policy and internal architecture docs 2026-05-26 03:28:28 +00:00
4950a6bb14 Add operations and troubleshooting documentation 2026-05-26 03:25:34 +00:00
b69ba96811 Rewrite README and add canonical CLI/config documentation 2026-05-26 03:21:11 +00:00
941e2656e8 Add policy documentation and prepare a roadmap to update the remaining documentation accordingly 2026-05-25 22:16:33 -05:00
79 changed files with 7868 additions and 2000 deletions

1
.gitignore vendored
View File

@@ -1,6 +1,5 @@
# ---> Codex
.codex
AGENTS.md
# ---> Go
# If you prefer the allow list template instead of the deny list, see community template:

View File

@@ -28,10 +28,6 @@ steps:
build_binary linux amd64 ""
build_binary linux arm64 ""
build_binary darwin amd64 ""
build_binary darwin arm64 ""
build_binary windows amd64 ".exe"
build_binary windows arm64 ".exe"
- name: publish-release
image: woodpeckerci/plugin-release

4
AGENTS.md Normal file
View File

@@ -0,0 +1,4 @@
Please carefully review the relevant documents in `docs/policy` before making any changes to this repository.
- `development.md` defines the contributor workflow for this application.
- `architecture.md` provides the canonical high-level architecture policy for this repository, and should be reviewed before writing or changing any code.
- `documentation.md` provides the canonical documentation policy for this repository, and should be reviewed before writing or changing any documentation.

451
README.md
View File

@@ -1,449 +1,38 @@
# scriptorium
Scriptorium is a generic prompt execution engine.
Scriptorium is a config-driven prompt execution engine.
It takes:
- a prompt definition
- a selected or default execution profile
- named input artifacts
- template variables
- optional runtime overrides
It separates prompt definitions (what to generate) from execution profiles (how to call an OpenAI-compatible model endpoint), then runs or renders a prepared request from named input artifacts.
It returns:
- for `run`: generated artifact, validation result, metadata
- for `render`: prepared/rendered prompt data (no model output)
## Quickstart
## Prompt vs Profile
From the repository root:
Scriptorium separates **what** to do (Prompt) from **how** to do it (Profile).
### Prompt Definition
Defines the task logic and output contract.
- Task description and version.
- Message templates (system, user, etc.).
- Required and optional input artifacts.
- Output format and validation rules.
- Repair settings for structured output.
- Optional `default_profile` for convenience.
### Execution Profile
Defines the runtime environment and model settings.
- LLM endpoint (URL).
- Model name.
- Generation parameters: `temperature`, `max_tokens`, `top_p`.
- Runtime settings: `timeout`, `reasoning_effort`.
- API key source via `api_key_env`.
Callers can explicitly provide a `profile_id` to override the prompt's `default_profile`.
## Precedence
Scriptorium uses two precedence layers:
### Application Configuration Precedence
For application-level adapter settings (for example prompt/profile/schema directories, server address, and render output default), precedence is:
1. **CLI Flags**
2. **`config.yml`**
3. **Built-in application defaults**
Application config loading behavior:
- Default config path: `/etc/scriptorium/config.yml`
- Override path: `--config <PATH>` (supported by `run`, `render`, and `serve`)
- If `--config` is provided, the file must exist and be valid.
- If `--config` is omitted, missing `/etc/scriptorium/config.yml` is allowed.
### Runtime Model Precedence
When resolving runtime model settings, Scriptorium follows this precedence model (highest to lowest):
1. **Runtime Overrides**: Provided via CLI flags or HTTP request `model` object.
2. **Execution Profile**: Settings defined in the selected profile.
3. **Application Defaults**: Built-in fallback values.
### Profile Selection Logic
The engine determines which profile to use in this order:
1. Explicit `profile_id` (via `--profile` or HTTP request).
2. The `default_profile` named in the Prompt Definition.
3. Error: If neither is provided and no default exists.
## API Key Policy
To ensure security, Scriptorium does not support raw API keys in configuration files, CLI arguments, or HTTP requests.
- **`api_key_env`**: Profiles and overrides specify the name of an environment variable (e.g., `SCRIPTORIUM_API_KEY`).
- **Runtime Resolution**: The value of the environment variable is read directly from the process environment at runtime.
- **Zero Leakage**: API key values are never included in metadata, logs, or response bodies.
## CLI Usage
Available CLI commands:
- `scriptorium run`
- `scriptorium render`
- `scriptorium serve`
All commands accept `--config <PATH>`.
`prompt_dir` and `profile_dir` may be supplied by CLI flags or `config.yml`:
- `--prompt-dir` or `config.yml` `prompt_dir`
- `--profile-dir` or `config.yml` `profile_dir`
`schema_dir` and `serve` `addr` may also be supplied by `config.yml` where applicable:
- `--schema-dir` or `config.yml` `schema_dir`
- `--addr` or `config.yml` `server.addr`
### `scriptorium run`
Runs a single prompt execution.
**Required Flags:**
- `--prompt`: The prompt ID to execute.
- `--input`: Input mapping `name=path` (repeatable).
**Required Effective Settings:**
- Prompt directory: `--prompt-dir` or `config.yml` `prompt_dir`
- Profile directory: `--profile-dir` or `config.yml` `profile_dir`
**Optional Flags:**
- `--config`: Application config file path. Default discovery path is `/etc/scriptorium/config.yml`.
- `--prompt-dir`: Override prompt directory from config.
- `--profile-dir`: Override profile directory from config.
- `--profile`: Override the prompt's default profile.
- `--var`: Template variable `name=value` (repeatable).
- `--out`: Write output to a file instead of stdout.
- `--llm-base-url`: Override endpoint.
- `--model`: Override model name.
- `--api-key-env`: Override API key environment variable name.
- `--temperature`: Override temperature.
- `--max-tokens`: Override max tokens.
- `--top-p`: Override top_p.
- `--timeout`: Override request timeout (e.g., `30s`, `1m`).
- `--schema-dir`: Base directory for validation schemas.
**Examples:**
Using `config.yml` for prompt/profile directories:
```bash
scriptorium run \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md
```
Overriding config directories explicitly:
```bash
scriptorium run \
--prompt-dir ./prompts \
--profile-dir ./profiles \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md
```
Overriding the profile:
```bash
scriptorium run \
--prompt-dir ./prompts \
--profile-dir ./profiles \
--prompt generic.markdown_summary \
--profile local-quality \
--input transcript=./examples/fixtures/transcript.md
```
Overriding model and runtime values:
```bash
scriptorium run \
--prompt-dir ./prompts \
--profile-dir ./profiles \
--prompt generic.markdown_summary \
--model gpt-4o \
--temperature 0.7 \
--input transcript=./examples/fixtures/transcript.md
```
Using a local OpenAI-compatible vLLM endpoint:
```bash
scriptorium run \
--prompt-dir ./prompts \
--profile-dir ./profiles \
--prompt generic.markdown_summary \
--llm-base-url http://localhost:8000/v1 \
--model meta-llama-3-8b \
--input transcript=./examples/fixtures/transcript.md
```
### `scriptorium render`
Prepares and renders a prompt without calling the LLM.
`render` uses the same prompt/profile/input/variable/runtime override resolution as `run`:
- Profile selection precedence: `--profile` -> prompt `default_profile` -> error.
- Runtime precedence: CLI runtime overrides -> selected profile -> built-in defaults.
`render` is useful for debugging:
- prompt template rendering
- input mappings
- selected profile behavior
- runtime override behavior
`render` does not:
- call the LLM
- validate model output
- perform repair
- expose resolved API key values
It may include `api_key_env` names where relevant.
**Required Flags:**
- `--prompt`: The prompt ID to render.
- `--input`: Input mapping `name=path` (repeatable).
**Required Effective Settings:**
- Prompt directory: `--prompt-dir` or `config.yml` `prompt_dir`
- Profile directory: `--profile-dir` or `config.yml` `profile_dir`
**Optional Flags:**
- `--config`: Application config file path. Default discovery path is `/etc/scriptorium/config.yml`.
- `--prompt-dir`: Override prompt directory from config.
- `--profile-dir`: Override profile directory from config.
- `--profile`: Override the prompt's default profile.
- `--var`: Template variable `name=value` (repeatable).
- `--out`: Write output to a file instead of stdout.
- `--format`: Render output format (`text` or `json`). Default: `text`.
- `--llm-base-url`: Runtime override for endpoint.
- `--model`: Runtime override for model name.
- `--api-key-env`: Runtime override for API key environment variable name.
- `--temperature`: Runtime override for temperature.
- `--max-tokens`: Runtime override for max tokens.
- `--top-p`: Runtime override for top_p.
- `--timeout`: Runtime override for timeout (e.g., `30s`, `1m`).
**Render Output Formats:**
- `text`: Human-readable output (default).
- `json`: Machine-readable structured output.
Render formatting is modular; additional output formats can be added later without changing prepare/run core logic.
**Examples:**
Default text output using `config.yml` directories:
```bash
scriptorium render \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md
```
Explicit config path:
```bash
scriptorium render \
go run ./cmd/scriptorium render \
--config ./examples/config.yml \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md
```
Explicit directory overrides:
```bash
scriptorium render \
--prompt-dir ./prompts \
--profile-dir ./profiles \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md
```
Explicit JSON output:
```bash
scriptorium render \
--prompt-dir ./prompts \
--profile-dir ./profiles \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \
--format json
```
Using prompt `default_profile` (omit `--profile`):
```bash
scriptorium render \
--prompt-dir ./prompts \
--profile-dir ./profiles \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md
```
This command renders the prepared prompt and effective runtime settings without calling an LLM.
Overriding profile selection:
```bash
scriptorium render \
--prompt-dir ./prompts \
--profile-dir ./profiles \
--prompt generic.markdown_summary \
--profile local-quality \
--input transcript=./examples/fixtures/transcript.md
```
## Documentation
Overriding runtime settings:
```bash
scriptorium render \
--prompt-dir ./prompts \
--profile-dir ./profiles \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--llm-base-url http://localhost:8000/v1 \
--model gpt-4o-mini \
--temperature 0.2 \
--max-tokens 800 \
--top-p 1.0 \
--timeout 45s
```
Writing rendered output to a file:
```bash
scriptorium render \
--prompt-dir ./prompts \
--profile-dir ./profiles \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--format text \
--out ./rendered_prompt.txt
```
### `scriptorium serve`
Starts the HTTP API.
**Required Effective Settings:**
- Prompt directory: `--prompt-dir` or `config.yml` `prompt_dir`
- Profile directory: `--profile-dir` or `config.yml` `profile_dir`
**Optional Flags:**
- `--config`: Application config file path. Default discovery path is `/etc/scriptorium/config.yml`.
- `--addr`: Listen address (default `:8080`).
- `--schema-dir`: Base directory for validation schemas.
**Examples:**
Using `config.yml`:
```bash
scriptorium serve
```
Overriding config for local use:
```bash
scriptorium serve \
--prompt-dir ./prompts \
--profile-dir ./profiles \
--addr :9090
```
## HTTP API
### `POST /v1/runs`
Executes a prompt. No built-in authentication is provided; deploy behind a trusted gateway.
**Request Body:**
```json
{
"prompt_id": "generic.structured_events",
"profile_id": "local-quality",
"include_raw_output": false,
"inputs": {
"transcript": {"type": "file", "uri": "./examples/fixtures/transcript.md"}
},
"vars": {
"session_date": "2026-05-04"
},
"model": {
"endpoint": "http://localhost:8000/v1",
"model": "gpt-4o-mini",
"temperature": 0.0
}
}
```
`profile_id` is optional. If omitted, Scriptorium uses the prompt's `default_profile`. If neither is available, the run fails.
**Response:**
Returns a `200 OK` with the generated artifact, validation results, and metadata including the `prompt_id` and the `selected_profile_id`.
**Validation Failures:**
If the model output fails validation (e.g., invalid JSON), the API returns `200 OK` with `validation.status = "failed"`.
**Raw Output Exposure:**
- `raw_model_output` is omitted by default.
- Set `include_raw_output: true` in the request to include it in the response.
- Raw output is preserved internally in run results regardless of HTTP exposure.
## Prompt Definition Authoring
Prompts are defined in YAML.
### Canonical Shape
```yaml
id: generic.structured_events
version: "1.0.0"
description: "Extracts structured events from a transcript"
default_profile: local-quality
inputs:
- name: transcript
required: true
content_type: text/markdown
description: "The raw session transcript"
- name: glossary
required: false
content_type: application/yaml
description: "Optional glossary terms"
messages:
- role: system
content: "You are a helpful assistant."
- role: user
content_file: messages/extract_events.tmpl
output:
format: json
validation_mode: json_schema
schema_path: structured_events.schema.json
repair_attempts: 2
```
**Key Features:**
- **Inline vs File**: Use `content` for short prompts or `content_file` for larger templates. Exactly one must be set per message.
- **Path Resolution**: `content_file` paths are resolved relative to the prompt YAML file.
- **Inputs**: Mark inputs as `required` to ensure the runner fails early if they are missing.
- **Input Metadata**: `content_type` is currently descriptive metadata and not enforced yet.
- **Validation**: Support `none`, `basic`, `json`, and `json_schema`.
- **Repair**: `repair_attempts` enables bounded retries to fix structured output.
## Execution Profile Authoring
Profiles are defined in YAML.
### Canonical Shape
```yaml
id: local-quality
endpoint: http://localhost:8000/v1
model: gpt-4o
temperature: 0.0
max_tokens: 4096
top_p: 1.0
timeout_seconds: 300
reasoning_effort: high
api_key_env: SCRIPTORIUM_API_KEY
```
**Constraints:**
- **No Raw Keys**: Do not include actual API keys. Only specify the environment variable name in `api_key_env`.
- **Local Profiles**: For local endpoints that don't require auth, `api_key_env` can be omitted.
- [CLI reference](docs/cli.md)
- [Configuration reference](docs/config.md)
- [Operations guide](docs/operations.md)
- [Troubleshooting](docs/troubleshooting.md)
- [Go library package](docs/consumers/pkg-scriptorium.md)
- [HTTP API integration](docs/integrations/http-api.md)
- [OpenAI-compatible chat integration](docs/integrations/openai-compatible-chat.md)
- [Narratio subprocess integration](docs/integrations/narratio.md)
- [Architecture policy](docs/policy/architecture.md)
## Examples
- **Prompt Definitions**: `prompts/`
- **Execution Profiles**: `profiles/`
- **Schemas**: `schemas/`
- **Fixtures**: `examples/fixtures/`
## Build and Test
```bash
go build -o scriptorium ./cmd/scriptorium
go test ./...
```
- `examples/render-markdown-summary.sh`
- `examples/http-run.json`
- `examples/go-library/prepare`

View File

@@ -1,483 +0,0 @@
# Scriptorium Architecture
## 1. Purpose and Non-Goals
Scriptorium is a prompt-definition execution engine.
It accepts named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata.
Scriptorium also supports rendering/preparing a prompt without calling an LLM. This allows users to inspect the fully rendered prompt messages and effective runtime settings before executing a run.
Scriptorium is not an orchestrator. It must not own transcription, transcript merge/polish steps, notifications, or cross-step workflow control.
For the motivating D&D workflow:
- Narratio orchestrates.
- WhisperX transcribes.
- Seriatim merges transcripts.
- Audita polishes transcripts.
- Scriptorium generates final artifacts from prepared inputs.
Core Go code remains generic.
## 2. Current Architecture
Scriptorium uses a ports-and-adapters architecture to decouple the core execution logic from external dependencies.
### Package Responsibilities
- `cmd/scriptorium`: Binary entrypoint for CLI and HTTP server.
- `internal/domain`: Core domain contracts, including `PromptDefinition`, `ExecutionProfile`, `PreparedRun`, `RunResult`, and related metadata.
- `internal/usecase`: `Runner` use case logic, including prompt preparation, profile selection, runtime override resolution, full run execution, validation, and bounded repair.
- `internal/config`: Application-level config model/loader for adapter settings (for example prompt/profile/schema directories, server address, and render format default).
- `internal/promptdef`: Repository for loading and validating Prompt Definitions from the filesystem.
- `internal/profile`: Repository for loading Execution Profiles from the filesystem.
- `internal/artifact`: Input artifact resolution (`inline`, `file`).
- `internal/prompt`: Template rendering via Go templates.
- `internal/llm`: Provider-neutral client interface and OpenAI-compatible HTTP adapter.
- `internal/validate`: Output validation implementation (`none/basic/json/json_schema`).
- `internal/adapter/cli`: CLI flag parsing, command dispatch, and output handling.
- `internal/adapter/http`: HTTP request/response mapping.
- `internal/format` or equivalent: Formatting of prepared/rendered prompt output for CLI or other adapters, if formatting grows beyond simple CLI-local helpers.
Exact package names may evolve, but the architectural boundaries should remain stable.
### Application Configuration
`config.yml` is adapter/application setup, not domain logic.
Application config is intended for application-level settings such as:
- `prompt_dir`
- `profile_dir`
- `schema_dir`
- `server.addr`
- `defaults.render_format`
Application config precedence is:
1. CLI flags
2. `config.yml`
3. Built-in application defaults
Runtime model settings are intentionally separate:
- Execution profiles and runtime overrides continue to own endpoint/model/runtime behavior.
- `config.yml` does not replace execution profiles.
The core use case (`Runner.Prepare`/`Runner.Run`) does not need to know whether adapter-level settings came from CLI flags or `config.yml`; it receives resolved dependencies and requests from adapters.
## 3. Core Execution Model
Scriptorium has two closely related execution paths:
1. Prepare/render path.
2. Full run path.
The full run path should reuse the prepare path rather than duplicating its logic.
### 3.1 Prepare / Render Data Flow
The prepare path should be represented in the use case layer, preferably as `Runner.Prepare(ctx, RunRequest)` or an equivalent method.
It executes all pre-LLM work:
1. **Validate Request**: Ensure the request includes a prompt ID and all minimum required fields.
2. **Load Prompt Definition**: Retrieve the `PromptDefinition` by ID from the prompt repository.
3. **Select Profile**: Determine the `profile_id` using this precedence:
- Explicit `profile_id` in `RunRequest`.
- `default_profile` specified in the `PromptDefinition`.
- Error if neither is available.
4. **Load Execution Profile**: Retrieve the `ExecutionProfile` from the profile repository.
5. **Resolve Runtime Overrides**: Merge settings based on precedence, highest to lowest:
- Runtime overrides from CLI flags or HTTP request `model` object.
- Execution Profile settings.
- Built-in application defaults.
6. **Resolve Artifacts**: Load all named input artifacts defined in the request.
7. **Render Prompt**: Apply template variables and input artifacts to the prompt templates.
8. **Compute Metadata**: Compute hashes, selected profile ID, prompt ID/version, effective runtime settings, input hashes, rendered prompt hash, and timing information as appropriate.
9. **Return PreparedRun**: Return a `PreparedRun` containing the rendered messages, effective runtime settings, resolved metadata, and input/prompt hashes.
The prepare path must not call the LLM.
The prepare path must not validate model output, because there is no model output.
The prepare path must not perform structured-output repair, because repair only applies after model output exists.
The prepare path should not resolve or expose raw API key values. It may include the selected `api_key_env` name in effective runtime settings or metadata, but never the environment variable value.
### 3.2 Full Run Data Flow
The `Runner.Run(ctx, RunRequest)` flow should reuse the prepare path:
1. **Prepare**: Call the shared prepare flow to load the prompt, select the profile, resolve artifacts, render prompt messages, and compute pre-run metadata.
2. **Call LLM**: Execute the generation request using the effective runtime settings from the prepared run.
3. **Build Output Artifact**: Convert the model response into the configured output artifact.
4. **Validate Output**:
- Validate the model output against the prompt definition's output contract.
- Validation content failures remain successful run results with `validation.status=failed`.
- Validator runtime/config errors are run errors.
5. **Repair If Configured**:
- If structured validation fails and `repair_attempts > 0`, perform bounded repair attempts.
- Re-validate after each repair attempt.
- Repair loops must remain strictly bounded.
6. **Return RunResult**: Produce a `RunResult` containing the final artifact, validation status, raw model output, usage information, and metadata.
`Runner.Run` should not duplicate profile selection, artifact resolution, or prompt rendering logic that already exists in `Runner.Prepare`.
## 4. Domain Model
Key domain types:
- `PromptDefinition`: Defines the "what" of the task: templates, inputs, output contract, validation settings, repair settings, and optional `default_profile`.
- `ExecutionProfile`: Defines the "how" of execution: endpoint, model, generation parameters, timeout, reasoning effort, and `api_key_env`.
- `RunRequest`: The intent to execute or prepare a prompt, including `prompt_id`, optional `profile_id`, inputs, variables, and optional runtime overrides.
- `PreparedRun`: The result of the prepare/render phase. Contains rendered messages, effective runtime settings, selected profile ID, prompt metadata, input hashes, prompt hash, and other pre-LLM metadata.
- `RunResult`: The result of a full run. Contains the generated `Artifact`, `ValidationResult`, raw model output, token usage, and `RunMetadata`.
- `RunMetadata`: Detailed tracing information, including prompt ID/version, selected profile ID, effective model parameters, usage tokens, hashes, timestamps, validation status, and repair attempts where applicable.
- `RenderedPrompt`: Provider-neutral rendered prompt structure.
- `RenderedMessage`: Provider-neutral rendered message with role and content.
- `ArtifactRef`: A reference to an input artifact, such as `file` or `inline`.
- `Artifact`: Loaded artifact content with name, content type, body, source URI, size, and hash.
- `ValidationResult`: Validation status and details for full runs.
`PreparedRun` should be serializable for JSON output and should also be representable in a human-readable text format.
## 5. Interfaces and Adapters
### Primary Ports
- `promptdef.Repository`: Lookup for prompt definitions.
- `profile.Repository`: Lookup for execution profiles.
- `artifact.Reader`: Loading of artifact content.
- `prompt.Renderer`: Template rendering.
- `llm.Client`: Model generation.
- `validate.Validator`: Output validation.
- `format.PreparedRunFormatter` or equivalent: Optional formatting abstraction for rendered/prepared output.
### Current Adapters
- **Repositories**: Filesystem YAML loaders for both prompts and profiles.
- **Artifact Reader**: Composite reader supporting `file` and `inline`.
- **Prompt Renderer**: Go templates with a custom `input` helper.
- **LLM Client**: OpenAI-compatible `/chat/completions` over HTTP.
- **Validator**: Standard validator supporting `none`, `basic`, `json`, and `json_schema`.
- **CLI Adapter**: Supports `run`, `render`, and `serve`.
- **HTTP Adapter**: Supports full run execution through `POST /v1/runs`.
## 6. Public Contracts
### CLI
Scriptorium should expose at least these commands:
- `scriptorium run`: Executes a prompt by preparing it, calling the LLM, validating output, optionally repairing structured output, and returning an artifact.
- `scriptorium render`: Prepares and renders a prompt without calling the LLM.
- `scriptorium serve`: Starts the HTTP API using infrastructure-only flags.
### `scriptorium run`
`run` uses flags such as:
- `--prompt-dir`
- `--profile-dir`
- `--prompt`
- `--profile`
- `--input`
- `--var`
- `--out`
- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.
`run` should produce the generated artifact as its primary output.
### `scriptorium render`
`render` prepares and renders a prompt without calling an LLM.
It should use the same prompt/profile/input/variable/runtime override flags as `run` where applicable:
- `--prompt-dir`
- `--profile-dir`
- `--prompt`
- `--profile`
- `--input`
- `--var`
- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.
- `--format`, with initial support for `text` and `json`.
Default render output format should be `text`.
`render` must not call the LLM.
`render` should show the same rendered messages and effective runtime settings that `run` would use.
`render` should include enough information to debug:
- prompt ID
- prompt version
- selected profile ID
- effective runtime settings
- input hashes
- prompt hash
- rendered messages
`render` must not include resolved API key values.
`render` may include the `api_key_env` name.
### Render Output Formats
Initial render output formats:
- `text`: Human-readable default format.
- `json`: Machine-readable structured representation of the prepared run.
Additional formats, such as `markdown`, may be added later.
Render output formatting should be modular. Adding a new output format should not require changing the prepare/run core logic.
The output formatting layer should consume a `PreparedRun` and produce bytes or text for the adapter. It should not reload prompts, re-resolve artifacts, re-render templates, call the LLM, or perform validation.
### `scriptorium serve`
`serve` starts the HTTP API.
It should use infrastructure-only flags such as:
- `--addr`
- `--prompt-dir`
- `--profile-dir`
- `--schema-dir`
`serve` should not introduce a server-level model/runtime precedence layer unless explicitly documented and intentionally implemented.
### HTTP API
Current HTTP API:
- `POST /v1/runs`: Accepts `RunRequest` JSON and returns `RunResponse` JSON. No built-in auth.
Request may include runtime overrides under `model` and an `include_raw_output` boolean.
`raw_model_output` is exposed only when explicitly requested with `include_raw_output=true`.
A future HTTP prepare/render endpoint may be added, such as `POST /v1/renders` or `POST /v1/runs/prepare`, but the initial render feature may be CLI-only. If added later, it should call the same usecase-level prepare path as `scriptorium render`.
### YAML Shapes
Prompt YAML includes:
- `id`
- `version`
- optional `default_profile`
- `inputs`
- `messages`
- `output`
Inputs support:
- `name`
- `required`
- optional `content_type`
- `description`
Messages require:
- `role`
- exactly one of `content` or `content_file`
`content_file` resolves relative to the prompt YAML location.
Profile YAML includes:
- `id`
- `endpoint`
- `model`
- generation parameters
- timeout settings
- reasoning settings
- `api_key_env`
Prompt content must not appear in profile YAML.
Model/runtime/API-key settings must not appear in prompt YAML, except that prompt YAML may specify `default_profile`.
## 7. Render Feature Design
The render feature is a first-class use case, not a CLI-only shortcut.
### Goals
The render feature should help users:
- inspect fully rendered prompt messages
- debug missing inputs
- verify template variable substitution
- verify selected profile resolution
- verify runtime override precedence
- verify file-backed prompt loading
- inspect input hashes and prompt hashes
- prepare for future token budgeting and prompt-size inspection
### Non-Goals
The render feature should not:
- call an LLM
- validate model output
- repair structured output
- resolve or print API key values
- mutate artifacts
- save outputs to artifact storage unless a future explicit output option is added
- become an orchestration step manager
### Usecase Shape
The preferred usecase shape is:
- `Runner.Prepare(ctx, RunRequest) (*PreparedRun, error)`
- `Runner.Run(ctx, RunRequest) (*RunResult, error)`
`Runner.Run` should call `Runner.Prepare`.
The prepare flow should be the only implementation of:
- prompt loading
- profile selection
- runtime override resolution
- artifact resolution
- prompt rendering
- pre-run metadata/hash calculation
### CLI Shape
The preferred command name is `render`.
The command should support:
- `--format text`
- `--format json`
Default format:
- `text`
Unknown formats should produce a clear error.
Formatting should be centralized through a small formatter registry, strategy, switch, or interface so new formats can be added without modifying usecase logic.
### Text Output Expectations
Text output should be optimized for human inspection.
It should include, at minimum:
- prompt ID and version
- selected profile ID
- model name
- endpoint
- effective generation settings
- input hashes
- rendered prompt hash
- rendered messages grouped by role
Text output should be readable and deterministic enough for tests.
It should not include raw API key values.
### JSON Output Expectations
JSON output should be a structured representation of `PreparedRun` or a DTO derived from it.
It should include, at minimum:
- prompt ID and version
- selected profile ID
- effective runtime settings
- input hashes
- rendered prompt hash
- rendered messages
JSON output should not include raw API key values.
JSON output should remain stable enough to be useful for automation and integration tests.
## 8. Guardrails
- **Separation of Concerns**: Prompt content must not belong in execution profiles; model/API settings must not belong in prompt definitions.
- **Security**: Raw API keys are unsupported in all configuration and transport layers. Only `api_key_env` is used.
- **Secret Handling**: Resolved API key values must never appear in rendered output, metadata, logs, HTTP responses, or CLI output.
- **Path Resolution**: `content_file` paths in prompt definitions resolve relative to the prompt YAML file.
- **Integrity**: No silent prompt truncation or omission of content.
- **Reliability**: Repair loops are strictly bounded by `repair_attempts`.
- **No Orchestration Creep**: Scriptorium prepares and executes a single prompt request. It does not coordinate multi-stage workflows.
- **Render Reuse**: The full run path must reuse the prepare/render path to avoid divergent behavior.
- **Formatter Isolation**: Render output formatters must not perform usecase work. They only format a completed `PreparedRun`.
## 9. Testing Strategy
Tests should protect both the run path and the prepare/render path.
### Prepare / Render Tests
Add tests for:
- preparing a prompt with explicit profile selection
- preparing a prompt using `default_profile`
- failing when no explicit profile and no `default_profile` exist
- runtime overrides beating profile values
- profile values beating application defaults
- file-backed prompt bodies rendering correctly
- required inputs failing when missing
- optional inputs being absent when not referenced
- unknown input references failing
- input hashes being included
- rendered prompt hash being included
- effective runtime settings being included
- `api_key_env` name being included where appropriate
- resolved API key values never appearing in `PreparedRun`
- prepare path not calling the LLM
### CLI Render Tests
Add tests for:
- `scriptorium render` mapping flags into `RunRequest`
- default text output
- explicit `--format text`
- explicit `--format json`
- unknown format failure
- text output includes prompt/profile/messages
- JSON output includes prompt/profile/messages
- rendered output never includes resolved API key values
### Run Reuse Tests
Add tests proving:
- `Runner.Run` reuses prepare behavior
- run and render resolve the same prompt/profile/runtime settings for equivalent inputs
- run still validates output
- run still performs bounded repair where configured
### Existing Tests
Continue testing:
- prompt definition loading
- execution profile loading
- artifact loading
- prompt rendering
- LLM adapter behavior
- validation behavior
- HTTP request/response mapping
## 10. Extension Points
Future work should remain grounded in the current architecture:
- **Artifacts**: Add S3 artifact references via a new `artifact.Reader`.
- **LLM**: Implement additional provider adapters, such as Anthropic or Google.
- **Execution**: Add token budgeting, streaming generation, and batch execution capabilities.
- **Prepare/Render**: Add token estimates, prompt-size summaries, or additional render output formats.
- **Repositories**: Implement database-backed repositories for prompts and profiles.
- **Profiles**: Support more granular profile versioning and environment-specific profiles.
- **HTTP**: Add an HTTP prepare/render endpoint if Narratio or another caller needs it.
Future render formats should plug into the formatter layer and should not require changes to the usecase layer.

335
convert.go Normal file
View File

@@ -0,0 +1,335 @@
package scriptorium
import "gitea.maximumdirect.net/eric/scriptorium/internal/domain"
func toDomainRunRequest(req RunRequest) domain.RunRequest {
return domain.RunRequest{
PromptID: req.PromptID,
PromptVersion: req.PromptVersion,
ProfileID: req.ProfileID,
Inputs: toDomainArtifactRefMap(req.Inputs),
Vars: copyStringMap(req.Vars),
Execution: toDomainExecutionTargetOverride(req.Execution),
Validation: toDomainOutputContractPtr(req.Validation),
Metadata: copyStringMap(req.Metadata),
}
}
func fromDomainPreparedRun(prepared *domain.PreparedRun) *PreparedRun {
if prepared == nil {
return nil
}
return &PreparedRun{
PromptID: prepared.PromptID,
PromptVersion: prepared.PromptVersion,
PromptHash: prepared.PromptHash,
SelectedProfileID: prepared.SelectedProfileID,
EffectiveModelParams: fromDomainExecutionTarget(prepared.EffectiveModelParams),
OutputContract: fromDomainOutputContract(prepared.OutputContract),
StructuredOutput: fromDomainStructuredOutputSpec(prepared.StructuredOutput),
InputHashes: copyStringMap(prepared.InputHashes),
SessionID: prepared.SessionID,
RenderedPromptHash: prepared.RenderedPromptHash,
Messages: fromDomainRenderedMessages(prepared.Messages),
StartTime: prepared.StartTime,
EndTime: prepared.EndTime,
DurationMS: prepared.DurationMS,
}
}
func fromDomainRunResult(result *domain.RunResult) *RunResult {
if result == nil {
return nil
}
return &RunResult{
RunID: result.RunID,
Artifact: fromDomainArtifact(result.Artifact),
RawOutput: result.RawOutput,
Validation: fromDomainValidationResult(result.Validation),
PromptID: result.PromptID,
PromptVersion: result.PromptVersion,
PromptHash: result.PromptHash,
RenderedPromptHash: result.RenderedPromptHash,
SelectedProfileID: result.SelectedProfileID,
ModelName: result.ModelName,
Endpoint: result.Endpoint,
EffectiveModelParams: fromDomainExecutionTarget(result.EffectiveModelParams),
InputHashes: copyStringMap(result.InputHashes),
Usage: fromDomainTokenUsage(result.Usage),
StartTime: result.StartTime,
EndTime: result.EndTime,
Duration: result.Duration,
}
}
func fromDomainGenerateRequest(req domain.GenerateRequest) GenerateRequest {
return GenerateRequest{
Prompt: fromDomainRenderedPrompt(req.Prompt),
Target: fromDomainExecutionTarget(req.Target),
TargetPresence: fromDomainExecutionTargetPresence(req.TargetPresence),
StructuredOutput: fromDomainStructuredOutputSpec(req.StructuredOutput),
}
}
func toDomainGenerateResponse(resp *GenerateResponse) *domain.GenerateResponse {
if resp == nil {
return nil
}
return &domain.GenerateResponse{
Content: resp.Content,
Usage: toDomainTokenUsage(resp.Usage),
}
}
func fromDomainRenderedPrompt(prompt domain.RenderedPrompt) RenderedPrompt {
return RenderedPrompt{
SessionID: prompt.SessionID,
Messages: fromDomainRenderedMessages(prompt.Messages),
}
}
func toDomainArtifactRefMap(src map[string]ArtifactRef) map[string]domain.ArtifactRef {
if src == nil {
return nil
}
out := make(map[string]domain.ArtifactRef, len(src))
for k, v := range src {
out[k] = toDomainArtifactRef(v)
}
return out
}
func toDomainArtifactRef(ref ArtifactRef) domain.ArtifactRef {
return domain.ArtifactRef{
Type: domain.ArtifactRefType(ref.Type),
URI: ref.URI,
Body: ref.Body,
}
}
func fromDomainArtifact(artifact domain.Artifact) Artifact {
return Artifact{
Name: artifact.Name,
ContentType: artifact.ContentType,
Body: copyBytes(artifact.Body),
URI: artifact.URI,
Size: artifact.Size,
Hash: artifact.Hash,
}
}
func toDomainExecutionTargetOverride(override *ExecutionTargetOverride) *domain.ExecutionTargetOverride {
if override == nil {
return nil
}
return &domain.ExecutionTargetOverride{
Endpoint: override.Endpoint,
Model: override.Model,
Temperature: copyFloat64Ptr(override.Temperature),
MaxTokens: copyIntPtr(override.MaxTokens),
TopP: copyFloat64Ptr(override.TopP),
TimeoutSeconds: copyIntPtr(override.TimeoutSeconds),
ServiceTier: override.ServiceTier,
ReasoningEffort: override.ReasoningEffort,
APIKeyEnv: override.APIKeyEnv,
ExtraParams: copyAnyMap(override.ExtraParams),
}
}
func fromDomainExecutionTarget(target domain.ExecutionTarget) ExecutionTarget {
return ExecutionTarget{
Endpoint: target.Endpoint,
Model: target.Model,
Temperature: target.Temperature,
MaxTokens: target.MaxTokens,
TopP: target.TopP,
TimeoutSeconds: target.TimeoutSeconds,
ServiceTier: target.ServiceTier,
ReasoningEffort: target.ReasoningEffort,
APIKeyEnv: target.APIKeyEnv,
ExtraParams: copyAnyMap(target.ExtraParams),
}
}
func fromDomainExecutionTargetPresence(presence domain.ExecutionTargetPresence) ExecutionTargetPresence {
return ExecutionTargetPresence{
Temperature: presence.Temperature,
MaxTokens: presence.MaxTokens,
TopP: presence.TopP,
TimeoutSeconds: presence.TimeoutSeconds,
}
}
func toDomainOutputContractPtr(contract *OutputContract) *domain.OutputContract {
if contract == nil {
return nil
}
out := toDomainOutputContract(*contract)
return &out
}
func toDomainOutputContract(contract OutputContract) domain.OutputContract {
return domain.OutputContract{
Format: domain.OutputFormat(contract.Format),
ValidationMode: domain.ValidationMode(contract.ValidationMode),
SchemaPath: contract.SchemaPath,
RepairAttempts: contract.RepairAttempts,
}
}
func fromDomainOutputContract(contract domain.OutputContract) OutputContract {
return OutputContract{
Format: OutputFormat(contract.Format),
ValidationMode: ValidationMode(contract.ValidationMode),
SchemaPath: contract.SchemaPath,
RepairAttempts: contract.RepairAttempts,
}
}
func fromDomainValidationResult(result domain.ValidationResult) ValidationResult {
return ValidationResult{
Status: ValidationStatus(result.Status),
Mode: ValidationMode(result.Mode),
Errors: copyStringSlice(result.Errors),
SchemaPath: result.SchemaPath,
RepairAttempts: result.RepairAttempts,
IsValid: result.IsValid,
}
}
func fromDomainTokenUsage(usage domain.TokenUsage) TokenUsage {
return TokenUsage{
PromptTokens: usage.PromptTokens,
CompletionTokens: usage.CompletionTokens,
TotalTokens: usage.TotalTokens,
CachedTokens: usage.CachedTokens,
CacheWriteTokens: usage.CacheWriteTokens,
}
}
func toDomainTokenUsage(usage TokenUsage) domain.TokenUsage {
return domain.TokenUsage{
PromptTokens: usage.PromptTokens,
CompletionTokens: usage.CompletionTokens,
TotalTokens: usage.TotalTokens,
CachedTokens: usage.CachedTokens,
CacheWriteTokens: usage.CacheWriteTokens,
}
}
func fromDomainRenderedMessages(messages []domain.RenderedMessage) []RenderedMessage {
if messages == nil {
return nil
}
out := make([]RenderedMessage, len(messages))
for i, msg := range messages {
out[i] = RenderedMessage{
Role: msg.Role,
Content: msg.Content,
CacheControl: fromDomainCacheControl(msg.CacheControl),
}
}
return out
}
func fromDomainCacheControl(cacheControl *domain.CacheControl) *CacheControl {
if cacheControl == nil {
return nil
}
return &CacheControl{
Type: CacheControlType(cacheControl.Type),
TTL: cacheControl.TTL,
}
}
func fromDomainStructuredOutputSpec(spec *domain.StructuredOutputSpec) *StructuredOutputSpec {
if spec == nil {
return nil
}
out := &StructuredOutputSpec{
Type: StructuredOutputType(spec.Type),
}
if spec.JSONSchema != nil {
out.JSONSchema = &StructuredOutputJSONSpec{
Name: spec.JSONSchema.Name,
Strict: spec.JSONSchema.Strict,
Schema: copyAny(spec.JSONSchema.Schema),
}
}
return out
}
func copyStringMap(src map[string]string) map[string]string {
if src == nil {
return nil
}
out := make(map[string]string, len(src))
for k, v := range src {
out[k] = v
}
return out
}
func copyAnyMap(src map[string]any) map[string]any {
if src == nil {
return nil
}
out := make(map[string]any, len(src))
for k, v := range src {
out[k] = copyAny(v)
}
return out
}
func copyAny(value any) any {
switch v := value.(type) {
case map[string]any:
return copyAnyMap(v)
case []any:
out := make([]any, len(v))
for i, item := range v {
out[i] = copyAny(item)
}
return out
case []string:
return copyStringSlice(v)
case []byte:
return copyBytes(v)
default:
return value
}
}
func copyStringSlice(src []string) []string {
if src == nil {
return nil
}
out := make([]string, len(src))
copy(out, src)
return out
}
func copyBytes(src []byte) []byte {
if src == nil {
return nil
}
out := make([]byte, len(src))
copy(out, src)
return out
}
func copyFloat64Ptr(src *float64) *float64 {
if src == nil {
return nil
}
v := *src
return &v
}
func copyIntPtr(src *int) *int {
if src == nil {
return nil
}
v := *src
return &v
}

157
docs/cli.md Normal file
View File

@@ -0,0 +1,157 @@
# CLI Reference
## Shortest Useful Command
```bash
go run ./cmd/scriptorium render \
--config ./examples/config.yml \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml
```
`render` prepares and formats the prompt without calling an LLM.
## Command Overview
- `scriptorium run`: prepare prompt, call the configured LLM, write generated output, print a run summary.
- `scriptorium render`: prepare prompt only; write prepared-run output as `text` or `json`.
- `scriptorium serve`: start the HTTP server.
Integration references:
- [HTTP contract](integrations/http-api.md)
- [Narratio subprocess contract](integrations/narratio.md)
## Common Argument Rules
- `--config` is supported by `run`, `render`, and `serve`.
- `run` and `render` require:
- `--prompt`
- at least one `--input`
- an effective `prompt_dir` and `profile_dir` (from flags or config)
- `serve` requires an effective `prompt_dir` and `profile_dir` (from flags or config).
- Positional arguments are rejected.
- Prompt cache control is configured in prompt YAML (`messages[].cache_control`), not with CLI flags.
- Provider-specific `reasoning_effort` and `extra_params` are configured in profile YAML or HTTP model overrides, not with CLI flags.
## Flag Reference
### `scriptorium run`
- `--config <path>`: app config file path.
- `--prompt-dir <dir>`: prompt definition directory.
- `--profile-dir <dir>`: profile definition directory.
- `--schema-dir <dir>`: schema base directory for `json_schema` validation.
- `--prompt <id>`: prompt ID to execute. Required.
- `--prompt-id <id>`: deprecated alias for `--prompt`.
- `--profile <id>`: explicit profile override.
- `--profile-id <id>`: deprecated alias for `--profile`.
- `--input name=path`: input mapping (repeatable, comma-separated accepted).
- `--var name=value`: template variable mapping (repeatable, comma-separated accepted).
- `--out <path>`: write artifact body to file instead of stdout.
- `--llm-base-url <url>`: runtime endpoint override.
- `--model <name>`: runtime model override.
- `--api-key-env <name>`: runtime API key environment-variable name override.
- `--temperature <float>`: runtime temperature override.
- `--max-tokens <int>`: runtime max tokens override.
- `--top-p <float>`: runtime top-p override.
- `--timeout <duration>`: runtime timeout override (Go duration syntax, for example `30s`, `2m`).
Numeric runtime override flags are presence-aware:
- omitted numeric flags preserve the selected profile/default value
- explicit zero values override the selected profile/default value (`--temperature 0`, `--max-tokens 0`, `--top-p 0`, `--timeout 0s`)
### `scriptorium render`
- Supports the same flags as `run`, except:
- no `--schema-dir` flag.
- Adds:
- `--format text|json`: prepared-run output format.
Notes:
- `render` still resolves profile and runtime settings.
- `render` still validates that `api_key_env` exists if the selected profile or overrides require it.
### `scriptorium serve`
- `--config <path>`: app config file path.
- `--addr <listen-address>`: HTTP listen address.
- `--prompt-dir <dir>`: prompt definition directory.
- `--profile-dir <dir>`: profile definition directory.
- `--schema-dir <dir>`: schema base directory for `json_schema` validation.
Notes:
- `serve` does not accept runtime model override flags such as `--model` or `--llm-base-url`.
## Input And Variable Syntax
- `--input name=path` maps prompt input names to local file paths.
- `--var name=value` maps template variable names to values.
- If a prompt defines `session_id: "{{ .session_id }}"`, pass the OpenRouter sticky-routing value with `--var session_id=<value>`.
- Both flags can be repeated.
- Both flags also support comma-separated batches, for example:
- `--input transcript=./t.md,glossary=./g.yml`
- `--var session_id=42,session_date=2026-05-04`
## Output Behavior
`run`:
- Writes generated artifact content to stdout by default.
- Writes generated artifact content to `--out` when provided.
- Prints run summary metadata to stderr on success.
- Appends `cached_tokens=<n> cache_write_tokens=<n>` to the summary only when the provider reports non-zero cache usage.
- Prints errors to stderr on failure.
`render`:
- Writes prepared-run output to stdout by default.
- Writes prepared-run output to `--out` when provided.
- Does not print a success summary line.
`serve`:
- Logs startup and server errors to stderr.
## Exit Codes
- `0`: success.
- `1`: runtime/parse/config/load/render/generation/output-write error.
- `2`: `run` completed, output was generated, but validation status is `failed`.
When `run` exits `2`, output may already be written to stdout or `--out`.
## Common Workflows
Render prompt inputs and template variables as JSON:
```bash
go run ./cmd/scriptorium render \
--config ./examples/config.yml \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \
--var session_date=2026-05-04 \
--format json
```
Run a prompt with profile override and file output:
```bash
go run ./cmd/scriptorium run \
--config ./examples/config.yml \
--prompt generic.markdown_summary \
--profile local-fast \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \
--out ./summary.md
```
Start the HTTP server with explicit config:
```bash
go run ./cmd/scriptorium serve --config ./examples/config.yml
```
Copyable example script:
- `examples/render-markdown-summary.sh`

270
docs/config.md Normal file
View File

@@ -0,0 +1,270 @@
# Configuration Reference
## Config Discovery And Precedence
Application settings are loaded in this order:
1. Built-in defaults
2. `config.yml` values
3. CLI overrides
When `--config` is not provided, Scriptorium searches for config files in this order:
1. `/usr/local/etc/scriptorium/config.yml`
2. `/etc/scriptorium/config.yml`
If neither file exists, Scriptorium continues with built-in defaults.
When `--config <path>` is provided, that file is required.
## Minimal App Config
```yaml
prompt_dir: ./examples/prompts
profile_dir: ./examples/profiles
```
This is enough to use `run` and `render` when prompt/profile files are valid.
## Production-Oriented App Config
```yaml
prompt_dir: /opt/scriptorium/prompts
profile_dir: /opt/scriptorium/profiles
schema_dir: /opt/scriptorium/schemas
server:
addr: 127.0.0.1:8080
defaults:
render_format: text
```
## App Config File (`config.yml`)
Top-level fields:
- `prompt_dir` (optional): default prompt definition directory.
- `profile_dir` (optional): default profile definition directory.
- `schema_dir` (optional): base directory for schema files used by `json_schema` validation.
- `server.addr` (optional): default listen address for `serve`.
- `defaults.render_format` (optional): default `render` output format (`text` or `json`).
Built-in defaults:
- `schema_dir`: `.`
- `server.addr`: `:8080`
- `defaults.render_format`: `text`
Validation behavior:
- Config decoding is strict; unknown YAML fields are rejected.
- Raw API key fields are not supported in `config.yml`.
## Prompt Definition Files
Prompt definitions are YAML files anywhere under `prompt_dir`, including nested subdirectories.
Subdirectories are organizational only. Callers still select prompts by the YAML `id`, not by file path. For example, `prompts/dnd/recap.yaml` may still declare `id: dnd.recap`, and callers use `--prompt dnd.recap`.
Example:
```yaml
id: generic.structured_events
version: "1.0.0"
default_profile: local-quality
description: Produce structured event JSON from a transcript.
inputs:
- name: transcript
required: true
content_type: text/markdown
description: Source transcript content
- name: glossary
required: false
content_type: text/yaml
description: Optional glossary context
messages:
- role: system
content_file: ./generic.structured_events.system.md
- role: user
content_file: ./generic.structured_events.user.md
output:
format: json
validation_mode: json_schema
schema_path: structured_events.schema.json
repair_attempts: 0
```
Field reference:
- `id` (required): prompt identifier.
- `version` (required): prompt version.
- `default_profile` (optional): profile ID used when request does not provide `profile_id`.
- `description` (optional): prompt description.
- `session_id` (optional): Go-template string for OpenRouter sticky-routing `session_id`; rendered from request vars.
- `inputs` (optional list): expected named inputs.
- `messages` (required list): prompt message templates.
- `output` (required object): output contract.
`inputs[]` fields:
- `name` (required)
- `required` (optional, boolean)
- `content_type` (optional metadata)
- `description` (optional)
`messages[]` fields:
- `role` (required)
- `content` or `content_file` (exactly one is required)
- `cache_control` (optional object): provider prompt-cache metadata for this message
Message rules:
- Repeated roles are allowed.
- `content_file` is resolved relative to the prompt YAML file location.
- Nested prompt files keep the same relative `content_file` behavior; `./recap.user.md` next to `dnd/recap.yaml` resolves from `dnd/`.
- Prompt decoding is strict; unknown YAML fields are rejected.
- Duplicate prompt IDs are invalid. If multiple files declare the requested prompt ID, Scriptorium fails instead of choosing one.
`messages[].cache_control` fields:
- `type` (required when `cache_control` is present): currently only `ephemeral`.
- `ttl` (optional): currently only `1h`; omitted from outbound requests when unset.
Example cache-controlled message:
```yaml
messages:
- role: system
content_file: ./stable_context.md
cache_control:
type: ephemeral
ttl: 1h
- role: user
content: |
{{input "transcript"}}
```
Use cache control on stable reusable prompt content. Dynamic per-run inputs before the cache-controlled message change the provider cache key.
Example prompt-level session ID:
```yaml
session_id: "{{ .session_id }}"
```
When configured, `session_id` is rendered with the same variable context as messages. The rendered value is trimmed, omitted when empty, and rejected if longer than 256 characters. CLI callers pass the value through `--var session_id=<value>`; HTTP callers pass it through `"vars": {"session_id": "<value>"}`.
`output` fields:
- `format` (required): `text`, `markdown`, or `json`.
- `validation_mode` (required): `none`, `basic`, `json`, or `json_schema`.
- `schema_path` (required when `validation_mode: json_schema`).
- `repair_attempts` (required): integer `>= 0`.
Repair behavior boundary:
- `repair_attempts` is part of the prompt contract.
- CLI and HTTP currently construct the runner without a repairer, so normal `run`/`serve` execution does not perform output repair attempts.
## Profile Definition Files
Execution profiles are YAML files anywhere under `profile_dir`, including nested subdirectories.
Subdirectories are organizational only. Callers still select profiles by the YAML `id`, not by file path. For example, `profiles/local/local-quality.yaml` may still declare `id: local-quality`, and callers use `--profile local-quality`.
Example:
```yaml
id: local-fast
endpoint: http://localhost:8000/v1
model: gpt-4o-mini
temperature: 0.2
max_tokens: 500
top_p: 1.0
timeout_seconds: 90
api_key_env: SCRIPTORIUM_API_KEY
service_tier: priority
reasoning_effort: medium
extra_params:
provider_route: primary
provider_options:
retry_budget: 2
```
Field reference:
- `id` (required)
- `endpoint` (required)
- `model` (required)
- `temperature` (optional): range `0..2`
- `max_tokens` (optional): `>= 0`
- `top_p` (optional): range `0..1`
- `timeout_seconds` (optional): `>= 0`
- `service_tier` (optional): provider-specific request tier such as OpenRouter `flex` or `priority`
- `reasoning_effort` (optional): serialized as top-level `reasoning_effort` in outbound chat-completions requests
- `api_key_env` (optional)
- `extra_params` (optional map): JSON-compatible provider-specific parameters. Values may be strings, numbers, booleans, objects, or arrays.
Profile rules:
- Profile decoding is strict; unknown YAML fields are rejected.
- Raw `api_key` is rejected; use `api_key_env`.
- If `api_key_env` is set, that environment variable must be set when preparing/running.
- Duplicate profile IDs are invalid. If multiple files declare the requested profile ID, Scriptorium fails instead of choosing one.
- `extra_params` keys must not be empty and must not collide with reserved outbound request fields: `model`, `session_id`, `messages`, `temperature`, `max_tokens`, `top_p`, `service_tier`, `reasoning_effort`, or `response_format`.
Current outbound request behavior:
- The OpenAI-compatible client currently serializes: `model`, optional `session_id`, `messages`, `temperature`, `max_tokens`, `top_p`, `service_tier`, `reasoning_effort`, optional `response_format` for `json_schema` prompts, and `extra_params`.
- `extra_params` are flattened into provider-specific top-level JSON request fields. They are not wrapped in an `extra_params` object on the outbound provider request.
- Messages without `cache_control` serialize with string `content`.
- Messages with `cache_control` serialize as a single text content-block array containing `cache_control`.
## Schema Behavior
Schemas are JSON files, typically in `schema_dir`.
Rules:
- `output.validation_mode: json_schema` requires `output.schema_path`.
- Relative `schema_path` values resolve from `schema_dir`, including explicit nested paths such as `dnd/structured_events.schema.json`.
- Absolute `schema_path` values are used directly.
- Scriptorium does not recursively search schemas by basename; nested schemas must be referenced by their relative path.
- Missing or invalid schema documents cause runtime validation errors.
- Invalid generated JSON causes validation status `failed` (not a runtime error).
Supported artifact reference types for request inputs are `file` and `inline`.
## Secrets Handling
- Keep secret values in environment variables.
- Store only environment-variable names in profile `api_key_env`.
- Do not put raw API keys in config, prompts, profiles, CLI flags, or HTTP request bodies.
## Maintained Examples
- App config: `examples/config.yml`
- Prompt examples: `examples/prompts/`
- Profile examples: `examples/profiles/`
- Schema examples: `examples/schemas/`
- Input fixtures: `examples/fixtures/`
- Render example script: `examples/render-markdown-summary.sh`
- HTTP request example: `examples/http-run.json`
Example organizational layout:
```text
examples/prompts/dnd/recap.yaml
examples/profiles/local/local-quality.yaml
examples/schemas/dnd/structured_events.schema.json
```
## Integration References
- [Inbound HTTP contract](integrations/http-api.md)
- [Outbound OpenAI-compatible contract](integrations/openai-compatible-chat.md)

View File

@@ -1,48 +0,0 @@
# Main `config.yml`
`config.yml` defines application-level defaults used by CLI commands.
By default, Scriptorium looks for `/usr/local/etc/scriptorium/config.yml` and, if not present, then for `/etc/scriptorium/config.yml`. You can also pass `--config PATH`.
## Complete Example
```yaml
prompt_dir: ./prompts
profile_dir: ./profiles
schema_dir: ./schemas
server:
addr: :8080
defaults:
render_format: text
```
## Available Options
- `prompt_dir` (optional): Default directory for prompt definition YAML files.
- `profile_dir` (optional): Default directory for execution profile YAML files.
- `schema_dir` (optional): Base directory for JSON schema files used by `json_schema` validation.
### `server`
- `addr` (optional): HTTP server listen address for `scriptorium serve`.
### `defaults`
- `render_format` (optional): Default output format for `scriptorium render`.
- Allowed values: `text`, `json`.
## Precedence
For run/render/serve settings, precedence is:
1. Explicit CLI flags
2. `config.yml`
3. Built-in defaults
## Notes and Rules
- Unknown YAML fields fail to load (strict decoding).
- This file does not accept API keys.
- `config.yml` sets directory/server defaults only; prompt/profile content remains in their own files.

View File

@@ -1,41 +0,0 @@
# Execution Profile Definitions
Execution Profiles define **how** Scriptorium calls an LLM endpoint.
A profile file is YAML, typically stored under `profiles/`, for example `profiles/local-quality.yaml`.
## Complete Example
```yaml
id: local-quality
endpoint: http://localhost:8000/v1
model: gpt-4.1
temperature: 0.0
max_tokens: 1200
top_p: 1.0
timeout_seconds: 180
reasoning_effort: medium
api_key_env: SCRIPTORIUM_API_KEY
extra_params:
provider: openrouter
route: fallback
```
## Available Options
- `id` (required): Unique profile identifier used by `--profile` or prompt `default_profile`.
- `endpoint` (required): OpenAI-compatible base URL, usually ending in `/v1`.
- `model` (required): Model name to request at that endpoint.
- `temperature` (optional): Sampling temperature. Valid range is `0` to `2`.
- `max_tokens` (optional): Max completion tokens. Must be `>= 0`.
- `top_p` (optional): Nucleus sampling parameter. Valid range is `0` to `1`.
- `timeout_seconds` (optional): Request timeout in seconds. Must be `>= 0`.
- `reasoning_effort` (optional): Provider/model-specific reasoning level string.
- `api_key_env` (optional): Environment variable name that holds the API key.
- `extra_params` (optional): String key/value map for provider-specific parameters.
## Notes and Rules
- Raw API keys are not supported. Do **not** add `api_key` fields.
- Unknown YAML fields fail to load (strict decoding).
- If `api_key_env` is set, the environment variable must be present when the run executes.

View File

@@ -1,73 +0,0 @@
# Prompt Definition Files
Prompt Definitions define **what** Scriptorium should do.
A prompt file is YAML, typically stored under `prompts/`, for example `prompts/generic.structured_events.yaml`.
## Complete Example
```yaml
id: generic.structured_events
version: "1.0.0"
default_profile: local-quality
description: Extract events from a transcript into structured JSON.
inputs:
- name: transcript
required: true
content_type: text/markdown
description: Source transcript
- name: glossary
required: false
content_type: text/yaml
description: Optional glossary context
messages:
- role: system
content: |
You are a structured extraction assistant.
Return only JSON.
- role: user
content_file: ./generic.structured_events.user.md
output:
format: json
validation_mode: json_schema
schema_path: structured_events.schema.json
repair_attempts: 1
```
## Available Options
- `id` (required): Prompt identifier used by `--prompt` / `prompt_id`.
- `version` (required): Prompt version string.
- `default_profile` (optional): Execution profile ID used when no explicit profile is provided.
- `description` (optional): Human-readable description.
### `inputs[]`
- `name` (required): Logical input name referenced in templates via `{{input "name"}}`.
- `required` (optional): If `true`, run fails when input is missing.
- `content_type` (optional): Metadata only (not enforced yet).
- `description` (optional): Human-readable input description.
### `messages[]`
- `role` (required): Message role such as `system` or `user`.
- `content` (optional): Inline Go-template message body.
- `content_file` (optional): Path to a template file.
Each message must set **exactly one** of `content` or `content_file`.
### `output`
- `format` (required): One of `text`, `markdown`, `json`.
- `validation_mode` (required): One of `none`, `basic`, `json`, `json_schema`.
- `schema_path` (required when `validation_mode: json_schema`): Path to JSON Schema file.
- `repair_attempts` (required): Number of bounded repair retries (`>= 0`).
## Notes and Rules
- Unknown YAML fields fail to load (strict decoding).
- `content_file` paths are resolved relative to the prompt YAML file.
- For `json_schema` validation mode, Scriptorium also sends provider-level structured output requests automatically.

View File

@@ -1,82 +0,0 @@
# JSON Schema Definition Files
Schema definition files describe the expected JSON output contract for prompts that use:
- `output.format: json`
- `output.validation_mode: json_schema`
Schema files are JSON, typically stored under `schemas/`, for example `schemas/structured_events.schema.json`.
## Complete Example
```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://example.com/schemas/structured-events.schema.json",
"title": "Structured Events",
"description": "Expected shape for extracted event output",
"type": "object",
"properties": {
"summary": {
"type": "string",
"minLength": 1,
"description": "High-level session summary"
},
"events": {
"type": "array",
"items": {
"type": "object",
"properties": {
"title": { "type": "string" },
"type": {
"type": "string",
"enum": ["discovery", "combat", "social", "travel", "downtime", "other"]
},
"notes": { "type": "string" }
},
"required": ["title", "type"],
"additionalProperties": false
},
"minItems": 0
}
},
"required": ["summary", "events"],
"additionalProperties": false,
"$defs": {
"nonEmptyString": {
"type": "string",
"minLength": 1
}
}
}
```
## Available Options
Scriptorium does not define custom schema keywords. It expects a valid JSON Schema document and passes it to the validator/provider.
Commonly used JSON Schema options include:
- `$schema`: Draft identifier URI.
- `$id`: Schema identifier URI.
- `title`: Human-readable schema title.
- `description`: Human-readable schema description.
- `type`: Expected JSON type (`object`, `array`, `string`, etc.).
- `properties`: Object field definitions.
- `required`: Required object fields.
- `additionalProperties`: Whether undeclared fields are allowed.
- `items`: Array item schema.
- `enum`: Allowed literal values.
- `const`: Single allowed literal value.
- `oneOf`, `anyOf`, `allOf`: Composition rules.
- `minimum`, `maximum`: Numeric bounds.
- `minLength`, `maxLength`, `pattern`: String constraints.
- `minItems`, `maxItems`: Array constraints.
- `$defs`: Reusable local definitions.
- `$ref`: Reference to another schema/definition.
## Notes and Rules
- Schema path comes from prompt `output.schema_path` and is resolved relative to `schema_dir`.
- If schema loading fails for `json_schema` mode, the run fails before the LLM request.
- Keep schemas strict (`additionalProperties: false`) when you want predictable output shape.

13
docs/consumers/api.md Normal file
View File

@@ -0,0 +1,13 @@
# Consumer API Overview
Scriptorium can be used by consumers through three implemented surfaces:
- CLI commands, documented in [CLI reference](../cli.md).
- HTTP `POST /v1/runs`, documented in [HTTP API integration](../integrations/http-api.md).
- Go package `gitea.maximumdirect.net/eric/scriptorium`, documented in [pkg-scriptorium](pkg-scriptorium.md).
The Go package is the typed in-process API. It prepares prompts, runs prompts, accepts file or inline artifacts, supports per-request execution overrides, and exposes stable public errors for `errors.Is`.
Use the Go package when the caller is a Go program that wants typed requests/results, context cancellation, repeated calls without subprocess overhead, or fake LLM injection for tests. Use the CLI or HTTP surfaces when process isolation, language neutrality, or an HTTP boundary is preferred.
Raw API key values are not accepted in public payloads and are not returned in prepared or run results. Execution profiles may reference an environment variable name through `api_key_env`.

View File

@@ -0,0 +1,129 @@
# Package scriptorium
Import path:
```go
import "gitea.maximumdirect.net/eric/scriptorium"
```
The root package is a public facade over Scriptorium's prompt execution use case. It keeps `internal/*` packages private while exposing typed construction, preparation, execution, inputs, results, and errors.
## Construct An Engine
```go
engine, err := scriptorium.NewEngine(scriptorium.Config{
PromptDir: "./examples/prompts",
ProfileDir: "./examples/profiles",
SchemaDir: "./examples/schemas",
})
if err != nil {
return err
}
```
`PromptDir` and `ProfileDir` are required. `SchemaDir` defaults to the built-in schema directory. `Timeout` and `HTTPClient` configure the default OpenAI-compatible client used by `Run` when no custom LLM client is supplied.
## Prepare A Prompt
`Prepare` resolves the prompt definition, profile, inputs, variables, output contract, structured-output metadata, and rendered messages without calling an LLM.
```go
prepared, err := engine.Prepare(ctx, scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.File("./examples/fixtures/transcript.md"),
"glossary": scriptorium.File("./examples/fixtures/glossary.yml"),
},
})
if err != nil {
return err
}
_ = prepared.Messages
```
Input helpers:
- `scriptorium.File(path)` loads an input artifact from a file.
- `scriptorium.Inline(body)` passes inline input content.
- `scriptorium.InlineWithURI(uri, body)` passes inline content with URI metadata.
## Run A Prompt
`Run` prepares the prompt, calls the configured LLM client, builds the output artifact, and validates the output.
```go
result, err := engine.Run(ctx, scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.File("./examples/fixtures/transcript.md"),
"glossary": scriptorium.File("./examples/fixtures/glossary.yml"),
},
})
if err != nil {
return err
}
_ = result.Artifact
```
`RunResult` includes the run ID, output artifact, raw output, validation result, prompt/profile/model metadata, effective model parameters, input hashes, token/cache usage, and timing fields. Validation content failures return a successful `RunResult` with failed validation status. Runtime validation errors return `ErrValidation`.
## Inject An LLM Client
Use `WithLLMClient` for tests or custom model integrations:
```go
type fakeLLM struct{}
func (fakeLLM) Generate(ctx context.Context, req scriptorium.GenerateRequest) (*scriptorium.GenerateResponse, error) {
return &scriptorium.GenerateResponse{
Content: "generated text",
Usage: scriptorium.TokenUsage{TotalTokens: 12},
}, nil
}
engine, err := scriptorium.NewEngine(cfg, scriptorium.WithLLMClient(fakeLLM{}))
```
The injected client receives the rendered prompt, effective execution target, target presence metadata for explicit numeric overrides, and structured-output spec. `WithLLMClient(nil)` returns `ErrInvalidConfig`.
## Request Overrides
`RunRequest.Execution` accepts per-request overrides. Numeric override fields are pointers so explicit zero values are preserved:
```go
zero := 0
req.Execution = &scriptorium.ExecutionTargetOverride{
MaxTokens: &zero,
}
```
## Errors
Public methods wrap context while preserving stable sentinel checks with `errors.Is`:
- `ErrInvalidConfig`
- `ErrInvalidRequest`
- `ErrPromptNotFound`
- `ErrProfileNotFound`
- `ErrPromptLoad`
- `ErrProfileLoad`
- `ErrArtifactLoad`
- `ErrPromptRender`
- `ErrLLMGenerate`
- `ErrValidation`
Example:
```go
if errors.Is(err, scriptorium.ErrPromptNotFound) {
return err
}
```
## Examples
Run the prepare-only example from the repository root:
```bash
go run ./examples/go-library/prepare
```

View File

@@ -0,0 +1,210 @@
# HTTP API Integration
## Scope
This document defines the implemented inbound HTTP contract for Scriptorium.
Current scope is only:
- `POST /v1/runs`
For CLI behavior, see the [CLI reference](../cli.md).
## Endpoint
- Method: `POST`
- Path: `/v1/runs`
- Content type: JSON request/response
Route behavior:
- unknown path: `404 not_found`
- unsupported method on `/v1/runs`: `405 method_not_allowed`
Copyable request example file:
- `examples/http-run.json`
## Request Body
```json
{
"prompt_id": "generic.structured_events",
"profile_id": "local-quality",
"prompt_version": "1.0.0",
"inputs": {
"transcript": {"type": "file", "uri": "./examples/fixtures/transcript.md"},
"glossary": {"type": "inline", "body": "party:\n - Rin"}
},
"vars": {
"session_date": "2026-05-04"
},
"model": {
"endpoint": "http://localhost:8000/v1",
"model": "gpt-4o-mini",
"temperature": 0.0,
"max_tokens": 800,
"top_p": 1.0,
"timeout_seconds": 120,
"service_tier": "priority",
"reasoning_effort": "medium",
"api_key_env": "SCRIPTORIUM_API_KEY",
"extra_params": {
"route": "primary",
"provider_options": {
"retry_budget": 2
}
}
},
"include_raw_output": false
}
```
Required fields:
- `prompt_id`
- `inputs` (must contain at least one named input)
Input reference types currently supported by runtime artifact loading:
- `file`
- `inline`
Model override notes:
- Numeric model override fields distinguish omitted values from explicit zero values. For example, omitting `temperature` preserves the selected profile/default value, while `"temperature": 0` explicitly sets the effective temperature to zero.
- `extra_params` accepts JSON-compatible values: strings, numbers, booleans, objects, and arrays.
- `extra_params` are passed through effective model metadata and flattened into top-level provider request fields by the OpenAI-compatible client.
- `extra_params` keys must not be empty and must not collide with reserved outbound fields: `model`, `session_id`, `messages`, `temperature`, `max_tokens`, `top_p`, `service_tier`, `reasoning_effort`, or `response_format`.
- Raw API-key values are not accepted. Use `api_key_env` to name an environment variable.
## Strict JSON Rules
Request decoding uses strict JSON field checks:
- unknown request fields are rejected with `400 invalid_json`
- unknown `model` fields are rejected with `400 invalid_json`
- raw API-key payload fields such as `api_key` are rejected as unknown fields
## Success Response
Status: `200 OK`
Response shape:
```json
{
"artifact": {
"name": "output",
"content_type": "application/json",
"body": "{\"summary\":\"...\"}",
"uri": "",
"size": 123,
"hash": "..."
},
"validation": {
"status": "passed",
"mode": "json_schema",
"errors": [],
"schema_path": "structured_events.schema.json",
"repair_attempts": 0,
"is_valid": true
},
"metadata": {
"run_id": "...",
"prompt_id": "generic.structured_events",
"prompt_version": "1.0.0",
"prompt_hash": "...",
"rendered_prompt_hash": "...",
"selected_profile_id": "local-quality",
"model_name": "gpt-4o-mini",
"endpoint": "http://localhost:8000/v1",
"model_params": {
"endpoint": "http://localhost:8000/v1",
"model": "gpt-4o-mini",
"temperature": 0,
"max_tokens": 800,
"top_p": 1,
"timeout_seconds": 120,
"service_tier": "priority",
"reasoning_effort": "medium",
"api_key_env": "SCRIPTORIUM_API_KEY",
"extra_params": {
"route": "primary",
"provider_options": {
"retry_budget": 2
}
}
},
"input_hashes": {
"transcript": "..."
},
"usage": {
"prompt_tokens": 11,
"completion_tokens": 22,
"total_tokens": 33,
"cached_tokens": 0,
"cache_write_tokens": 0
},
"start_time": "2026-05-04T12:00:00Z",
"end_time": "2026-05-04T12:00:01Z",
"duration_ms": 1000,
"validation_mode": "json_schema",
"validation_status": "passed",
"repair_attempts_used": 0
}
}
```
`raw_model_output` is omitted by default.
`metadata.usage.cached_tokens` and `metadata.usage.cache_write_tokens` are always present as numbers. They are `0` when the provider omits compatible cache usage fields or reports no cache activity.
To include it, send:
- `"include_raw_output": true`
## Validation Failure Behavior
Validation content failures do not map to HTTP error status.
Behavior:
- status remains `200 OK`
- `validation.status` is `failed`
- validation errors are returned in `validation.errors`
## Error Responses
Error body shape:
```json
{
"error": {
"code": "invalid_request",
"message": "prompt_id is required"
}
}
```
Current error mapping (non-exhaustive):
- `400 invalid_json`: malformed JSON or unknown JSON fields
- `400 invalid_request`: missing/invalid request fields
- `400 profile_required`: no explicit `profile_id` and prompt has no `default_profile`
- `400 prompt_load_failed`: prompt definition invalid/unloadable
- `400 profile_load_failed`: profile invalid/unloadable
- `400 artifact_read_failed`: input artifact loading failed
- `400 prompt_render_failed`: template render failed
- `400 api_key_env_missing`: named API-key environment variable is missing
- `404 prompt_not_found`
- `404 profile_not_found`
- `502 llm_failed`: outbound model request failed
- `500 validation_runtime_failed`: validator runtime/schema-load failure
- `500 internal_error`
## Security And Deployment Note
The HTTP adapter has no built-in authentication or authorization.
Deploy behind trusted controls (for example authenticated gateway/reverse proxy and network boundaries).

View File

@@ -1,322 +1,114 @@
# Narratio -> Scriptorium CLI Integration
# Narratio Subprocess Integration
## 1. Purpose
## Purpose
This document defines how Narratio should invoke Scriptorium through the **public CLI**.
This document defines the supported subprocess contract for Narratio invoking Scriptorium through the public CLI.
This is a **subprocess integration contract**, not an internal Go API contract.
This is a CLI contract, not an internal Go package integration.
## 2. Assumptions
## Supported Commands
- `scriptorium` is installed and available on `PATH`.
- Scriptorium is configured with `config.yml`.
- `config.yml` provides `prompt_dir`, `profile_dir`, and `schema_dir` as needed.
- Prompt and profile libraries are already deployed for the environment.
- Narratio provides prepared artifact files (for example polished transcript, glossary, previous recap, campaign notes).
- Initial integration is synchronous subprocess execution.
- Narratio remains the orchestrator.
In normal operation, Narratio does not need to pass `--prompt-dir` and `--profile-dir` if they are supplied by Scriptorium config.
Narratio may pass `--config <PATH>` when it must use a non-default Scriptorium config file.
## 3. Core Commands Narratio May Call
Primary commands for subprocess integration:
Narratio should invoke:
- `scriptorium run`
- `scriptorium render`
For production generation, use `scriptorium run`.
Use `run` for generation.
`scriptorium render` is for debugging, dry-runs, test assertions, and validating command construction without LLM execution.
Use `render` for preflight/debug output without LLM execution.
Note: `scriptorium serve` and HTTP API exist, but they are not the initial integration path.
## Recommended Invocation Shapes
## 4. Command Selection Guidance
- Use `run` to generate an output artifact.
- Use `render` to inspect the prepared prompt and effective settings without calling the LLM.
- Use `render --format json` when Narratio/tests need structured prepare output.
## 5. Recommended `run` Invocation Shape
Production shape:
Run:
```bash
scriptorium run \
--prompt <prompt_id> \
--input transcript=<processed-transcript-path> \
--out <output-artifact-path>
--input transcript=<path> \
--out <artifact_path>
```
Common optional additions:
- `--config <path>`: use a specific Scriptorium config file.
- `--profile <profile_id>`: override prompt default profile.
- `--var name=value` (repeatable): small metadata values.
- `--input name=path` (repeatable): additional named artifacts.
- `--timeout <duration>`: per-run timeout override.
- Runtime model override flags (`--llm-base-url`, `--model`, etc.) only for exceptional/operator-directed cases.
## 6. Recommended `render` Invocation Shape
Human-readable debug shape:
Render:
```bash
scriptorium render \
--prompt <prompt_id> \
--input transcript=<processed-transcript-path> \
--format text
--input transcript=<path> \
--format json
```
Structured debug/test shape:
Narratio may add:
```bash
scriptorium render \
--prompt <prompt_id> \
--input transcript=<processed-transcript-path> \
--format json \
--out <render-debug-path>
```
- `--config <path>`
- `--profile <profile_id>`
- repeatable `--input name=path`
- repeatable `--var name=value`
- runtime overrides when explicitly needed (`--model`, `--llm-base-url`, `--timeout`, etc.)
`render` does **not** call the LLM, does **not** validate model output, and does **not** perform repair.
## Config And Directory Behavior
## 7. Inputs
Narratio can rely on resolved app config or pass explicit paths.
- Pass inputs as repeated `--input name=path` flags.
- `name` must match the Prompt Definition input name.
- Prefer absolute paths, or paths relative to a working directory controlled by Narratio.
- Pass Audita output as the primary transcript input.
- Additional inputs may include glossary, previous recap, campaign notes, event logs, final state maps, or other prompt-specific artifacts.
- Scriptorium reads input files directly; Narratio does not need to inline file content for CLI use.
- default config search order:
1. `/usr/local/etc/scriptorium/config.yml`
2. `/etc/scriptorium/config.yml`
- explicit `--config` requires file existence and valid syntax
- CLI flags override config values
## 8. Variables
## Profile Selection
Use repeated `--var name=value` for small metadata values.
Profile selection follows runner behavior:
Typical examples:
1. explicit `--profile`
2. prompt `default_profile`
3. error if neither is available
- `session_date`
- `session_id`
- `campaign_name`
- `previous_session_id`
- `output_kind`
Narratio should treat prompt/profile IDs as deployment configuration, not hardcoded logic.
Large content belongs in input files, not `--var` values.
## Input And Variable Contract
## 9. Prompt IDs and Output Artifact Types
- Inputs use repeated `--input name=path`.
- Input names must match prompt definition input names.
- Variables use repeated `--var name=value` for small metadata values.
- Prefer file inputs for large content.
Narratio should treat prompt IDs as configuration, not hardcoded business logic.
Narratio config may map stage/output names to prompt IDs, for example:
- session recap prompt
- structured event extraction prompt
- glossary suggestion prompt
- player-facing summary prompt
Prompt IDs used by Narratio should come from the deployed Scriptorium prompt library.
## 10. Profiles
- Prompts may declare `default_profile`.
- Narratio may omit `--profile` to use prompt default profile.
- Narratio may pass `--profile` to force profile selection.
- This enables environment/profile selection like `local-fast`, `local-quality`, `frontier`, `batch`, or test profiles.
- Profile names should generally be Narratio configuration values.
## 11. Runtime Overrides
Supported runtime override flags:
- `--llm-base-url`
- `--model`
- `--api-key-env`
- `--temperature`
- `--max-tokens`
- `--top-p`
- `--timeout`
Guidance:
- Keep normal model/runtime settings in Execution Profiles.
- Use runtime overrides only for explicit per-run exceptions, tests, or operator overrides.
- Never pass raw API keys on the command line.
- `--api-key-env` names an environment variable; Narratio must ensure that variable is set in subprocess environment.
## 12. Config Behavior
- Default config path: `/etc/scriptorium/config.yml`.
- `--config <PATH>` overrides default path.
- Missing default config is allowed by Scriptorium.
- If `--config` is provided explicitly, the file must exist and be valid.
- CLI flags override `config.yml`.
- `config.yml` overrides built-in application defaults.
Narratio can either:
- rely on system default config path, or
- carry an explicit config path and pass `--config`.
## 13. Environment Handling
Subprocess environment recommendations:
## Environment Contract
- Pass through required API-key environment variables referenced by `api_key_env`.
- Do not pass raw API keys as CLI arguments.
- Avoid logging full environment dumps.
- Capture stdout and stderr separately.
- Use a controlled working directory.
- Prefer absolute artifact paths.
- Never pass raw API keys via CLI arguments.
- Keep subprocess environment scoped to required variables.
## 14. Output Handling
## Output And Error Handling
For `scriptorium run`:
`run`:
- Use `--out` when Narratio needs durable artifact files.
- Without `--out`, artifact content is written to stdout.
- Preferred orchestration pattern: always use `--out`, then treat the file as stage output artifact.
- Capture stderr for diagnostics.
- stdout: artifact body unless `--out` is used
- `--out`: writes artifact to file
- stderr: success summary and errors
For `scriptorium render`:
`render`:
- Use `--out` to store render diagnostics.
- Use `--format json` when tests need to inspect selected profile, effective runtime settings, input hashes, prompt hash, and rendered messages.
- stdout: prepared-run output unless `--out` is used
- stderr: errors
## 15. Exit Status and Errors
Narratio should capture stdout and stderr separately.
Current CLI behavior (verified from implementation/tests):
## Exit Status Contract
- `0`: success.
- `1`: runtime/parse/config/load/render/generation/IO error.
- `2`: run completed but output validation failed (`ValidationFailed`).
- `0`: success
- `1`: parse/config/load/render/generation/IO/runtime error
- `2`: run completed but validation failed
Additional details:
A `run` exit code `2` can still produce output (stdout or `--out`).
- On `run`, output artifact write happens before exit code selection. If validation fails, artifact may still be written and exit code is `2`.
- `stderr` carries both errors and normal run summary output; non-empty stderr alone does not imply failure.
- `render` returns `0` on success and `1` on failures.
## Security Notes
Narratio should treat non-zero exit codes as failed stage execution, but may record generated artifact paths if a run exited `2` and output file exists.
- Treat generated artifacts and stderr logs as potentially sensitive.
- Avoid logging full rendered prompts by default in production contexts.
- Use controlled output paths and access controls for persisted artifacts.
## 16. Recommended Narratio Integration Pattern
## Canonical References
1. Build CLI args from Narratio stage configuration.
2. Use subprocess context cancellation/timeout.
3. Pass absolute input paths.
4. Pass `--out` to a session-scoped artifact path.
5. Add `--var` metadata values.
6. Optionally add `--config`.
7. Optionally add `--profile`.
8. Ensure required API-key env vars are present.
9. Run subprocess synchronously.
10. Capture stdout/stderr separately.
11. On success, store output artifact path and invocation metadata in stage artifacts.
12. On failure, store exit code and stderr diagnostics in stage status.
## 17. Suggested Narratio Configuration Shape
Illustrative (not required schema):
```yaml
scriptorium:
config_path: /etc/scriptorium/config.yml
stages:
session_recap:
prompt_id: dnd.session_recap
profile_id: local-quality # optional
inputs: [transcript, glossary, previous_recap]
vars: [session_id, session_date, campaign_name]
output_path_template: artifacts/{session_id}/session_recap.md
timeout: 2m
render_debug: false
```
The key idea: map Narratio stage/artifact names to prompt ID, optional profile, expected inputs, and output destination.
## 18. Testing Strategy for Narratio Integration
- Use `scriptorium render --format json` to verify command construction without LLM calls.
- Use dedicated test prompt/profile libraries for integration tests.
- Use small fixture transcripts.
- Verify missing-input failure behavior.
- Verify prompt `default_profile` behavior.
- Verify explicit `--profile` override behavior.
- Verify `--config` behavior (default and explicit).
- Verify output file creation when `--out` is used.
- Verify stderr capture on failures.
- Avoid real API keys in tests.
## 19. Security and Privacy Notes
- Never pass raw API keys on command line.
- Do not log full rendered prompts by default; transcripts may contain sensitive content.
- Avoid logging prompt content unless explicit debug mode is enabled.
- Treat generated artifacts as potentially sensitive.
- Use session-scoped, access-controlled output paths.
- `api_key_env` names should come from environment management, not embedded secrets.
## 20. Initial D&D Artifact Generation Examples
These are examples only. Use prompt IDs from the deployed prompt library.
Session recap:
```bash
scriptorium run \
--prompt dnd.session_recap \
--input transcript=/work/session-42/transcript.polished.md \
--input glossary=/work/session-42/glossary.yml \
--out /work/session-42/artifacts/session_recap.md
```
Structured events:
```bash
scriptorium run \
--prompt dnd.structured_events \
--input transcript=/work/session-42/transcript.polished.md \
--out /work/session-42/artifacts/structured_events.json
```
Glossary suggestions:
```bash
scriptorium run \
--prompt dnd.glossary_suggestions \
--input transcript=/work/session-42/transcript.polished.md \
--input previous_recap=/work/session-41/artifacts/session_recap.md \
--out /work/session-42/artifacts/glossary_suggestions.md
```
Player-facing summary:
```bash
scriptorium run \
--prompt dnd.player_summary \
--input transcript=/work/session-42/transcript.polished.md \
--input structured_events=/work/session-42/artifacts/structured_events.json \
--out /work/session-42/artifacts/player_summary.md
```
## 21. Non-Goals
Initial Narratio integration should not:
- call Scriptorium internal Go packages
- use HTTP API as the primary path
- expect Scriptorium to read S3 refs directly
- make Scriptorium responsible for Narratio stage state
- make Scriptorium responsible for notification
- require Scriptorium to understand D&D workflow semantics beyond prompt definitions
## 22. Future Extension Notes
Possible later extensions:
- HTTP API integration
- S3 artifact references if Scriptorium adds S3 reader support
- storing render diagnostics alongside generated artifacts
- token budgeting/prompt-size checks
- batch execution if Scriptorium later adds batch support
- CLI behavior: [CLI reference](../cli.md)
- Config behavior: [Configuration reference](../config.md)
- Operations and failure handling: [Operations guide](../operations.md), [Troubleshooting](../troubleshooting.md)

View File

@@ -0,0 +1,188 @@
# OpenAI-Compatible Chat Integration
## Scope
This document defines the outbound LLM contract implemented by `internal/llm/openai_compatible_client.go`.
It documents only fields and behaviors currently serialized by code.
## Endpoint Construction
Request endpoint is built as:
1. choose base URL:
- `GenerateRequest.Target.Endpoint` if set
- otherwise client config `BaseURL`
2. trim trailing slash
3. append `/chat/completions`
Example:
- base URL: `http://localhost:8000/v1`
- final URL: `http://localhost:8000/v1/chat/completions`
## Request Fields Sent
Serialized JSON fields:
- `model` (required after fallback resolution)
- `session_id` (only when the rendered prompt includes a non-empty session ID)
- `messages` (rendered prompt messages)
- `temperature` (when non-zero, or when explicitly overridden to zero)
- `max_tokens` (when non-zero, or when explicitly overridden to zero)
- `top_p` (when non-zero, or when explicitly overridden to zero)
- `service_tier` (only when non-empty)
- `reasoning_effort` (only when non-empty)
- `response_format` (only when structured output is provided)
- profile/request `extra_params` as additional provider-specific top-level fields
`service_tier` is provider-specific. OpenRouter currently documents request values such as `flex` and `priority`; Scriptorium forwards any non-empty configured value and lets the backend validate support.
`reasoning_effort` is provider-specific. Scriptorium forwards any non-empty configured value as top-level `reasoning_effort` and lets the backend validate support.
`extra_params` are flattened into the outbound JSON object. They are not wrapped in an `extra_params` object:
```json
{
"model": "gpt-4o-mini",
"messages": [
{
"role": "user",
"content": "rendered text"
}
],
"provider_route": "primary",
"provider_options": {
"retry_budget": 2
}
}
```
`extra_params` values must be JSON-compatible. Supported value shapes include strings, numbers, booleans, objects, and arrays.
Reserved `extra_params` keys are rejected before the HTTP request is made:
- `model`
- `session_id`
- `messages`
- `temperature`
- `max_tokens`
- `top_p`
- `service_tier`
- `reasoning_effort`
- `response_format`
Empty `extra_params` keys and values that cannot be encoded as JSON are also rejected before the HTTP request is made.
`session_id` is rendered from prompt YAML using request variables and serialized as a top-level JSON request field. Scriptorium does not send an `x-session-id` header. Empty rendered session IDs are omitted, and values longer than 256 characters are rejected before the HTTP request.
Messages without prompt cache control serialize with string `content`:
```json
{
"role": "system",
"content": "rendered text"
}
```
Messages with prompt cache control serialize as a single text content-block array:
```json
{
"role": "system",
"content": [
{
"type": "text",
"text": "rendered text",
"cache_control": {
"type": "ephemeral",
"ttl": "1h"
}
}
]
}
```
When cache-control `ttl` is unset in the prompt definition, `ttl` is omitted from the outbound payload.
Structured output is currently `json_schema` only, serialized as:
```json
{
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "...",
"strict": true,
"schema": {"type": "object"}
}
}
}
```
## Authentication Header
If `Target.APIKeyEnv` is set:
- resolve environment variable value at request time
- set `Authorization: Bearer <value>`
If the environment variable is unset/empty:
- request fails before HTTP call (`ErrInvalidRequest`)
If `Target.APIKeyEnv` is empty:
- no `Authorization` header is sent
## Timeout Behavior
Base timeout comes from client configuration.
Per-request override:
- if `Target.TimeoutSeconds > 0`, use that value for request timeout
- if `Target.TimeoutSeconds == 0` and the value came from an explicit request override, disable the HTTP client timeout
- if `Target.TimeoutSeconds < 0`, request is rejected (`ErrInvalidRequest`)
## Response Expectations
Expected successful response shape (subset used):
- `choices[0].message.content`
- `usage.prompt_tokens`
- `usage.completion_tokens`
- `usage.total_tokens`
- `usage.prompt_tokens_details.cached_tokens` (optional)
- `usage.cache_write_tokens` (optional)
Absent cache usage fields are treated as zero. Parsed cache usage is exposed through run results and adapter response surfaces as:
- `cached_tokens`
- `cache_write_tokens`
Malformed response conditions include:
- invalid JSON
- empty `choices`
- empty `choices[0].message.content`
Malformed responses return `ErrMalformedResponse`.
## Error Handling
- network/request-construction failures: `ErrRequestFailed`
- non-2xx HTTP status: `ErrUnexpectedStatus` (includes status code and trimmed response body snippet)
- malformed response shape/content: `ErrMalformedResponse`
## Unsupported Or Non-Serialized Fields
The client does not serialize top-level `cache_control`.
No built-in retries, tool-calls, or multi-request payload modes are implemented in this client.
## Relationship To Runner
When prompt validation mode is `json_schema`, runner prepares a structured-output schema spec and passes it to the client as `StructuredOutput`.
The client only serializes the provider request payload; it does not load schema files itself.

164
docs/internal/adapters.md Normal file
View File

@@ -0,0 +1,164 @@
# Adapter And Repository Internals
## Purpose
This document describes implemented adapter/repository boundaries and their current behavior.
## Adapter Map
- `internal/adapter/cli`: CLI command parsing, app wiring, stdout/stderr handling, exit codes.
- `internal/adapter/http`: HTTP request/response mapping for `POST /v1/runs`.
- root package `scriptorium`: public Go library facade for preparing and running prompt requests.
- `internal/promptdef`: filesystem prompt-definition repository.
- `internal/profile`: filesystem execution-profile repository.
- `internal/artifact`: input artifact reader.
- `internal/prompt`: Go-template renderer.
- `internal/llm`: OpenAI-compatible LLM client implementation.
- `internal/validate`: output validator.
- `internal/format`: prepared-run formatters for `render` output.
## Inputs And Outputs
CLI adapter:
- Input: process args, filesystem config/assets, environment.
- Output: exit code, stdout artifact/prepared output, stderr summaries/errors.
- `run` summaries include cache usage counters only when either parsed cache counter is non-zero.
HTTP adapter:
- Input: JSON request body (`runRequestDTO`).
- Output: JSON success/error body with mapped status codes.
- Success metadata includes token usage plus cache usage counters.
Public library facade:
- Input: typed `scriptorium.RunRequest` values.
- Output: typed `PreparedRun` and `RunResult` values plus public sentinel errors.
- Custom LLM behavior is injected with `WithLLMClient`; otherwise the default OpenAI-compatible client is used.
- Public types are facade types converted at the package boundary; internal domain types remain internal.
Filesystem repositories:
- Input: prompt/profile YAML files under configured directories.
- Output: normalized domain definitions/profiles or typed errors.
Artifact reader:
- Input: `domain.ArtifactRef`.
- Output: loaded `domain.Artifact`.
LLM adapter:
- Input: `domain.GenerateRequest`.
- Output: `domain.GenerateResponse`.
Validator:
- Input: artifact body + output contract.
- Output: validation result or runtime validation error.
## Boundaries
- Adapters convert external representations to domain requests and back.
- Use-case decisions remain in `internal/usecase`.
- External dependency details stay scoped to adapter packages.
## Config Fields Used
Primary app settings consumed by adapters:
- `prompt_dir`
- `profile_dir`
- `schema_dir`
- `server.addr`
- `defaults.render_format`
Execution profile/request settings used through runner:
- `endpoint`, `model`, `temperature`, `max_tokens`, `top_p`, `timeout_seconds`, `service_tier`, `api_key_env`, `reasoning_effort`, `extra_params`
- CLI and HTTP request adapters preserve caller intent for numeric runtime overrides. Omitted values remain absent; explicit zero values are mapped as explicit overrides.
- HTTP `extra_params` accepts JSON-compatible values and maps them to domain request overrides without provider-specific adapter logic.
## External Dependencies
- YAML decoding: `gopkg.in/yaml.v3` (strict known-fields mode in config/prompt/profile loaders).
- JSON Schema validation: `github.com/santhosh-tekuri/jsonschema/v6`.
- HTTP client/server: Go standard library.
## Failure Behavior
Strict decoding and input checks:
- config/prompt/profile loaders reject unknown YAML fields.
- prompt/profile repositories scan nested subdirectories recursively.
- prompt/profile lookup uses YAML `id` values; subdirectory paths are organizational only.
- duplicate prompt/profile IDs are invalid and fail instead of using first-match behavior.
- HTTP DTO decoder rejects unknown JSON fields.
- raw API key payload fields are rejected by strict decoding in profile/http paths.
Artifact refs:
- Supported reference types: `inline`, `file`.
- Unsupported types return `ErrUnsupportedRefType`.
LLM adapter:
- endpoint appends `/chat/completions`.
- rendered messages without cache control serialize with string `content`.
- rendered messages with cache control serialize as one text content block with `cache_control`.
- non-empty `reasoning_effort` serializes as a top-level provider request field.
- `extra_params` flatten into provider-specific top-level JSON request fields.
- reserved `extra_params` keys are rejected before the provider call: `model`, `session_id`, `messages`, `temperature`, `max_tokens`, `top_p`, `service_tier`, `reasoning_effort`, and `response_format`.
- empty `extra_params` keys and values that cannot be JSON-encoded are rejected before the provider call.
- compatible cache usage response fields are parsed into domain token usage.
- non-2xx responses map to request failure errors.
- malformed responses (including missing/empty first choice content) are errors.
Validator:
- `basic`, `json`, `json_schema` content failures return `ValidationFailed` results.
- schema load/compile/path failures are runtime errors.
- schema lookup uses explicit `schema_path` values relative to `schema_dir`; it does not recursively search by basename.
HTTP error mapping:
- maps domain/use-case errors to stable HTTP code + error code/message.
- distinguishes missing profile selection and missing `api_key_env` variable using stable use-case sentinel errors.
- avoids returning internal wrapped-cause details in response payload.
## CLI Adapter Semantics
Implemented commands:
- `run`
- `render`
- `serve`
Behavior highlights:
- `run` exit `2` indicates validation failed after generation.
- `render` does not call the LLM.
- `serve` exposes HTTP handler only; no built-in auth.
- `render` supports `--format text|json`; `render` does not expose `--schema-dir`.
- deprecated aliases `--prompt-id` and `--profile-id` are still accepted.
## Tests To Inspect Before Changing
- `internal/adapter/cli/run_test.go`
- `internal/adapter/http/handler_test.go`
- `internal/promptdef/repository_test.go`
- `internal/profile/repository_test.go`
- `internal/artifact/reader_test.go`
- `internal/prompt/renderer_test.go`
- `internal/llm/openai_compatible_client_test.go`
- `internal/validate/standard_validator_test.go`
- `internal/format/prepared_run_test.go`
## Architectural Invariants
- Adapter packages do not own runner decision logic.
- External request/response strictness is part of contract stability.
- Prepared-render output never includes resolved API key values.
- Outbound OpenAI-compatible request includes currently serialized first-class fields (`model`, optional `session_id`, `messages`, optional `temperature`, `max_tokens`, `top_p`, optional `service_tier`, optional `reasoning_effort`, optional `response_format`) plus validated `extra_params` flattened as provider-specific top-level fields.
- Outbound cache control is message-level only; no top-level cache-control field is serialized.

166
docs/internal/runner.md Normal file
View File

@@ -0,0 +1,166 @@
# Runner Internals
## Purpose
`internal/usecase.Runner` is the core use case orchestrator for prompt preparation and execution.
It owns request validation, prompt/profile resolution, runtime-parameter merge, artifact loading, prompt rendering, structured-output setup, LLM invocation, output validation, and result metadata.
## Inputs And Outputs
Primary input type:
- `domain.RunRequest`
Primary output types:
- `domain.PreparedRun` from `Prepare`
- `domain.RunResult` from `Run`
LLM boundary types:
- `domain.GenerateRequest`
- `domain.GenerateResponse`
## Boundaries
`Runner` coordinates the following interfaces:
- `promptdef.Repository`
- `profile.Repository`
- `artifact.Reader`
- `prompt.Renderer`
- `llm.Client`
- `validate.Validator`
- optional `usecase.OutputRepairer`
Transport concerns (CLI flags, HTTP DTO parsing, status-code mapping) stay outside runner.
## Config Fields Used
`Runner` does not read app config files directly.
It receives fully constructed repositories/readers/validators from adapters. Effective behavior depends on adapter wiring, including:
- prompt/profile directories
- schema base directory
- selected profile/runtime overrides in request
## External Adapters Used
`Runner` works with adapter implementations via interfaces. Current wiring from CLI/HTTP uses:
- filesystem prompt/profile repositories
- composite artifact reader
- Go-template prompt renderer
- OpenAI-compatible LLM client
- standard validator
## State And Resume Behavior
`Runner` is stateless across requests.
- No durable run-state storage.
- No built-in resume/skip checkpoints.
- Each `Run`/`Prepare` executes from request inputs and current repositories.
## Failure Behavior
Primary runner error classes:
- `ErrInvalidRequest`: invalid run request envelope.
- `ErrProfileRequired`: specific invalid-request reason when neither request `profile_id` nor prompt `default_profile` is available.
- `ErrAPIKeyEnvMissing`: specific invalid-request reason when `api_key_env` is set but the named environment variable is unset/empty.
- `ErrProfileLoad`: prompt/profile repository load failures.
- `ErrArtifactLoad`: artifact read failures.
- `ErrPromptRender`: template render failures.
- `ErrLLMGenerate`: outbound model request failures.
- `ErrValidation`: validation runtime failures (including structured-output schema load/compile failures).
Reason sentinel behavior:
- `ErrProfileRequired` and `ErrAPIKeyEnvMissing` are wrapped with `ErrInvalidRequest`.
- Adapters can use `errors.Is` for stable reason mapping without matching runner prose.
Validation content failures are not run errors:
- `Run` can succeed with `Validation.Status == failed`.
- CLI maps this to exit code `2`.
- HTTP returns `200` with failed validation details.
## Prepare Flow
`Prepare` performs:
1. validate request basics (prompt ID present).
2. load prompt definition by ID/version.
3. select profile ID:
- explicit request profile ID
- prompt `default_profile`
- otherwise return an invalid request with `ErrProfileRequired`
4. load execution profile.
5. merge effective runtime target:
- built-in execution defaults
- selected profile values
- request overrides
- request numeric overrides are presence-aware, so omitted values preserve the current effective value and explicit zero values override it
6. verify required `api_key_env` environment variable:
- missing/empty env value returns an invalid request with `ErrAPIKeyEnvMissing`
- only the environment-variable name is retained; secret value is never returned
7. resolve output contract and structured-output schema payload when `json_schema` mode is active.
8. read input artifacts.
9. render prompt messages, including any normalized message cache-control metadata.
10. compute prompt/input/render hashes and return `PreparedRun`.
`rendered_prompt_hash` includes cache-control metadata when present because it affects the outbound provider request. Prompts without cache control keep the role/content hash behavior.
`Prepare` does not call the LLM.
Runtime target notes:
- Profile `extra_params` and request `extra_params` carry JSON-compatible values through prepared output, run metadata, and `domain.GenerateRequest.Target`.
- The OpenAI-compatible client serializes non-empty `reasoning_effort` as a top-level provider request field.
- The OpenAI-compatible client flattens `extra_params` into provider-specific top-level JSON request fields.
- Empty `extra_params` keys, reserved outbound field names, and values that cannot be JSON-encoded fail before the provider request.
- Resolved API-key values are never stored in `PreparedRun`, `RunResult`, logs, or HTTP responses.
## Run Flow
`Run` performs:
1. generate run ID.
2. call `Prepare`.
3. call LLM with prepared messages/effective target/structured-output spec.
4. build output artifact content type from output format.
5. validate output.
6. optionally attempt bounded repair when repairer is injected and contract allows it.
7. return `RunResult` with artifact, raw output, validation, hashes, profile/model metadata, token/cache usage, and timestamps.
## Repair Hook Boundary
Repair attempts occur only when all are true:
- repairer is injected
- `repair_attempts > 0`
- validation status is `failed`
- validation mode is `json` or `json_schema`
Current production wiring boundary:
- CLI and HTTP adapters call `usecase.NewRunner(...)` (no repairer argument).
- Therefore normal CLI/HTTP execution does not perform repair attempts today.
## Tests To Inspect Before Changing
- `internal/usecase/runner_test.go`
- `internal/usecase/integration_test.go`
- `internal/adapter/cli/run_test.go`
- `internal/adapter/http/handler_test.go`
## Architectural Invariants
- `Run` reuses `Prepare`; prepare logic is not duplicated.
- Effective API-key environment-variable name may appear; resolved secret value must not.
- Structured-output schema document must load before LLM call for `json_schema` mode.
- Repair loops are bounded by `repair_attempts` and repairer presence.
- Runner stays transport-agnostic.

125
docs/operations.md Normal file
View File

@@ -0,0 +1,125 @@
# Operations Guide
## Scope
This document covers day-to-day operation of the CLI and HTTP service for currently implemented behavior.
For command syntax, see [CLI reference](cli.md). For file formats and defaults, see [Configuration reference](config.md).
## Operational Model
Scriptorium executes one request at a time per CLI invocation or HTTP request.
Important boundaries:
- No durable run state is stored.
- No built-in resume, checkpoint, archive, or backup workflow exists.
- Recovery is rerun-based: fix inputs/config, then rerun.
## Filesystem Layout And Config
Scriptorium depends on:
- prompt definition files (`prompt_dir`)
- execution profile files (`profile_dir`)
- optional JSON schemas (`schema_dir`)
Config discovery order when `--config` is omitted:
1. `/usr/local/etc/scriptorium/config.yml`
2. `/etc/scriptorium/config.yml`
If neither exists, built-in defaults are used. If `--config <path>` is provided, that file must exist and parse successfully.
Built-in defaults relevant to operations:
- `schema_dir: .`
- `server.addr: :8080`
- `defaults.render_format: text`
## Normal CLI Workflow
Use `render` first when you need to verify prompt resolution and runtime settings without calling a model.
Use `run` for generation.
Typical sequence:
1. Confirm prompt/profile directories resolve through config or flags.
2. Confirm required input files exist and map to prompt input names.
3. Confirm required API-key environment variables are set.
4. Confirm the selected profile's model endpoint is reachable from the process environment.
5. Run `render` for preflight when changing prompt/profile/input wiring.
6. Run `run` for actual generation.
## Secrets Handling
Raw API keys are not accepted in config files, profile files as `api_key`, CLI flags, or HTTP request bodies.
Operational pattern:
- Set environment variables that hold secret values.
- Set profile `api_key_env` (or runtime override `api_key_env`) to the environment variable name.
- Keep process environments scoped to only required variables.
## HTTP Service Operation
Start service with:
```bash
go run ./cmd/scriptorium serve --config ./examples/config.yml
```
Current inbound API behavior:
- Route: `POST /v1/runs`
- JSON request parsing rejects unknown fields.
- Validation content failures still return `200 OK` with `validation.status: "failed"`.
Security caveat:
- `serve` has no built-in authentication or authorization.
- Deploy only behind trusted controls (private network boundary, authenticated reverse proxy, API gateway, or equivalent).
## Output, Logs, And Exit Codes
`run` command:
- Generated artifact body goes to stdout by default.
- `--out` writes generated artifact to a file.
- Summary metadata line is written to stderr on success.
- Exit code `2` means generation completed but validation failed.
`render` command:
- Prepared-run output goes to stdout by default.
- `--out` writes prepared-run output to a file.
- Exit code is `0` on success and `1` on failure.
`serve` command:
- Startup and server errors are written to stderr.
## Validation Behavior In Operations
Validation modes (`none`, `basic`, `json`, `json_schema`) are defined by prompt output contract.
Operational interpretation:
- Validation runtime errors are hard failures (`run` exit `1`; HTTP error response).
- Validation content failures are soft failures (`run` exit `2`; HTTP `200` with failed status).
A failed validation run can still produce output. Decide whether to keep or discard that output in your surrounding workflow.
## Safe Recovery Steps
For failed runs or requests:
1. Capture stderr output or HTTP error code/message.
2. Confirm config path and directory settings.
3. Verify prompt/profile IDs and input mappings.
4. Verify API-key environment-variable presence when required.
5. Reproduce with `render --format json` when prompt/profile/input resolution is uncertain.
6. Rerun after correction.
Because Scriptorium does not persist run state, rerun is the canonical recovery path.

112
docs/policy/architecture.md Normal file
View File

@@ -0,0 +1,112 @@
# Architecture
This document is the development architecture policy for Scriptorium.
It is for developers and LLM coding agents. User-facing behavior belongs in `README.md` and the docs under `docs/` that target operators/users.
## Project Shape
Scriptorium is a narrow prompt-execution application with three entry paths:
- CLI `run`
- CLI `render`
- HTTP `POST /v1/runs` through `serve`
Domain behavior is centralized in `internal/usecase` and `internal/domain`.
## Core Principles
- Keep orchestration narrow: Scriptorium executes one prompt request; it is not a multi-step workflow engine.
- Keep adapter logic thin: adapters map external shapes to domain requests/results and should not hold domain decisions.
- Keep boundaries explicit: repositories/loaders/renderers/validators/LLM client stay behind package interfaces.
- Keep config strict: YAML/JSON decoding for external inputs should reject unknown fields.
- Keep secrets out of payloads: raw API key values must not be accepted or emitted.
## Package Boundaries
Current package map:
- `cmd/scriptorium`: process entrypoint.
- `internal/adapter/cli`: command parsing, app wiring for CLI commands, output behavior.
- `internal/adapter/http`: HTTP DTO mapping and error/status mapping.
- `internal/config`: application settings loading and CLI override precedence.
- `internal/defaults`: compile-time default constants.
- `internal/domain`: core request/result and contract types.
- `internal/usecase`: `Runner` prepare/run orchestration and repair-hook boundary.
- `internal/promptdef`: filesystem prompt-definition repository.
- `internal/profile`: filesystem execution-profile repository.
- `internal/artifact`: artifact reference readers.
- `internal/prompt`: template renderer.
- `internal/llm`: provider-neutral LLM client interface and OpenAI-compatible implementation.
- `internal/validate`: validator interfaces and standard implementation.
- `internal/format`: prepared-run output formatting.
Detailed component behavior is documented in:
- `docs/internal/runner.md`
- `docs/internal/adapters.md`
## Configuration And Precedence
Application settings are resolved as:
1. built-in defaults
2. config file values
3. CLI overrides
`config.yml` is for application wiring (directories, server address, render default format), not prompt/profile runtime execution settings.
Profile selection and runtime model resolution remain use-case concerns.
## State And Persistence Policy
Scriptorium has no durable run-state store.
- No built-in resume/checkpoint/archive behavior.
- Recovery model is rerun after correcting inputs/config/environment.
## External Integration Policy
Current external contracts:
- inbound HTTP contract: `POST /v1/runs`
- outbound model contract: OpenAI-compatible chat completions subset
- subprocess contract for integrators: CLI `run`/`render`
Integration docs belong under `docs/integrations/`.
## Error Handling And Logging
- Wrap errors with domain/operation context.
- Map domain errors to adapter-appropriate statuses/codes without leaking sensitive internals.
- Keep stderr summaries concise for CLI success/error paths.
- Never emit raw secret values.
## Testing Expectations
- Core runner behavior should be covered with isolated unit tests and fixture-based integration tests.
- Adapter behavior should be tested for parse/mapping/error semantics.
- Config parsing, prompt/profile loading, validator behavior, and LLM client error handling should remain covered by package tests.
- Repository-level docs/examples that claim runnable behavior should be validated by tests or smoke commands.
## Documentation Expectations
- Document implemented behavior only outside `docs/roadmap/`.
- Keep canonical reference locations stable (`docs/cli.md`, `docs/config.md`, `docs/operations.md`, `docs/troubleshooting.md`, `docs/internal/`).
- Update docs in the same change when architecture-relevant behavior changes.
## Architectural Invariants
- `Runner.Run` reuses `Runner.Prepare` flow.
- CLI and HTTP currently instantiate `Runner` without a repairer.
- Artifact reading supports `inline` and `file` references.
- Unknown input fields in config/prompt/profile/http JSON should be rejected by strict decoding.
- Raw API key values must not be accepted through config/HTTP payloads.
## Non-Goals
- Do not move orchestration responsibilities from external callers into Scriptorium.
- Do not add adapter-specific business logic in `internal/adapter/*` packages.
- Do not bypass repository/renderer/validator/LLM boundaries by introducing cross-package coupling.
Work that is not implemented belongs in `docs/roadmap/`.

103
docs/policy/development.md Normal file
View File

@@ -0,0 +1,103 @@
# Development Guide
This document defines contributor workflow for Scriptorium.
## Repository Layout
- `cmd/scriptorium`: application entrypoint.
- `internal/domain`: core contracts.
- `internal/usecase`: runner orchestration.
- `internal/adapter/cli`: CLI adapter.
- `internal/adapter/http`: HTTP adapter.
- `internal/config`: application settings loading and precedence.
- `internal/defaults`: default constants.
- `internal/promptdef`: prompt-definition repository.
- `internal/profile`: execution-profile repository.
- `internal/artifact`: artifact readers.
- `internal/prompt`: prompt rendering.
- `internal/llm`: LLM client interface and OpenAI-compatible implementation.
- `internal/validate`: validation interfaces and implementation.
- `internal/format`: prepared-run formatting.
- `docs/`: canonical documentation.
- `examples/`: copyable maintained examples and fixtures.
## Common Commands
Build:
```bash
go build ./cmd/scriptorium
```
Test:
```bash
go test ./...
```
Targeted test runs commonly used during changes:
```bash
go test ./internal/adapter/cli ./internal/adapter/http ./internal/usecase
```
## Coding Conventions
- Prefer small interfaces at package boundaries.
- Keep adapter packages focused on translation and IO concerns.
- Keep domain/use-case logic outside adapters.
- Wrap errors with operation context.
- Use strict decoding for user-provided YAML/JSON where applicable.
- Avoid introducing dependencies unless they materially reduce risk/complexity.
## Dependency Policy
- Prefer standard library unless an external library is clearly justified.
- Current non-stdlib dependencies are intentionally small:
- `gopkg.in/yaml.v3` for YAML decoding.
- `github.com/santhosh-tekuri/jsonschema/v6` for JSON Schema validation.
- Do not leak dependency-specific types across unrelated package boundaries.
## How To Add App Config Fields
1. Add fields in `internal/config/config.go` (`Config`, `AppSettings`, and/or `CLIOverrides` as needed).
2. Apply defaults in `BuiltInDefaults()` when required.
3. Parse and validate in `applyConfig` / `ApplyCLIOverrides`.
4. Wire the field through the consuming adapter(s).
5. Add/update config tests in `internal/config/config_test.go`.
6. Update canonical docs (`docs/config.md`, and other affected docs).
## How To Add CLI Flags
1. Add flags in `internal/adapter/cli/run.go` for the relevant command.
2. Ensure precedence behavior remains consistent with app config rules.
3. Keep `run`, `render`, and `serve` flag surfaces intentionally scoped.
4. Add/update parser and command tests in `internal/adapter/cli/run_test.go`.
5. Update `docs/cli.md` and any related docs/examples.
## How To Add Adapters Or Adapter Capabilities
1. Define or reuse the appropriate interface boundary in domain/use-case packages.
2. Implement adapter code under `internal/adapter/<name>` (or relevant boundary package).
3. Keep business decisions in `internal/usecase`.
4. Add focused adapter tests for mapping, parse, and error behavior.
5. Document the new/changed boundary in `docs/internal/adapters.md`.
6. If external contract changes, update `docs/integrations/` in the same change.
## How To Update Prompt/Profile/Schema Assets
1. Keep prompt/profile/schema files valid under strict loaders.
2. Keep examples secret-free.
3. Re-run tests that cover prompt/profile/validation behavior.
4. Update `docs/config.md` and any docs that reference changed contracts.
## Documentation Update Expectations
When behavior changes:
1. Update canonical doc locations, not duplicate files.
2. Keep non-roadmap docs limited to implemented behavior.
3. Update links after file moves/renames.
4. Re-run relevant tests and smoke commands.
Docs work is complete only when code/tests/examples/docs agree.

View File

@@ -0,0 +1,446 @@
# Go Project Documentation Policy
## Purpose
Project documentation must help five audiences:
1. users who need to run the application;
2. administrators/operators who need to configure and operate it;
3. developers who need to understand and change it safely;
4. LLM coding agents that need clear scope, boundaries, and invariants;
5. developers and LLM coding agents integrating this project from another codebase.
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
## Core Rules
### 1. Keep docs concise
Each document should cover a defined scope and only the essentials for that scope.
Avoid:
- long background explanations;
- repeated reference material;
- implementation detail in user-facing docs;
- aspirational language outside roadmap docs;
- verbose examples where one minimal example is clearer.
### 2. Document only implemented behavior outside roadmap files
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
- `docs/roadmap/`
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
### 3. Use canonical homes
Each type of information should have one canonical location.
Canonical homes:
- project purpose and quickstart: `README.md`
- development principles: `docs/policy/architecture.md`
- public HTTP API reference: `docs/api.md`
- configuration reference: `docs/config.md`
- CLI reference: `docs/cli.md`
- operations and recovery: `docs/operations.md`
- troubleshooting: `docs/troubleshooting.md`
- public API/package consumer guidance: `docs/consumers/`
- implemented internals: `docs/internal/`
- external protocol, service, and file-format contracts: `docs/integrations/`
- future work: `docs/roadmap/`
- contributor workflow: `docs/policy/development.md`
- copyable examples: `examples/`
Other files should summarize briefly and link to the canonical source.
### 4. Keep examples real
Examples should be valid, maintained, and free of secrets.
Where practical:
- example configs should load successfully;
- example commands should match real CLI syntax;
- important examples should be covered by tests.
## Documentation Profiles
All projects require:
- `README.md`
- `docs/policy/architecture.md`
Additional docs depend on the project.
### Small library
Recommended:
- `docs/policy/development.md`, if contributor conventions are non-obvious
### Simple CLI
Required:
- `docs/cli.md`
Recommended:
- `docs/policy/development.md`
### Config-driven CLI
Required:
- `docs/cli.md`
- `docs/config.md`
Recommended:
- `examples/`
- `docs/policy/development.md`
### Stateful or operator-facing application
Required:
- `docs/cli.md`, if CLI-based
- `docs/config.md`, if config-driven
- `docs/operations.md`
Recommended:
- `docs/troubleshooting.md`
- `examples/`
- `docs/policy/development.md`
### Modular, service-oriented, or orchestration application
Required:
- `docs/cli.md`, if CLI-based
- `docs/config.md`, if config-driven
- `docs/operations.md`
- `docs/internal/`
- `docs/policy/development.md`
Recommended:
- `docs/troubleshooting.md`
- validated examples under `examples/`
### Public HTTP API service
Required:
- `docs/api.md`
- `docs/cli.md`, if CLI-based
- `docs/config.md`, if config-driven
- `docs/operations.md`
- `docs/internal/`
- `docs/policy/development.md`
Recommended:
- `docs/troubleshooting.md`
- `docs/consumers/`, for task-oriented client integration guides
- `docs/integrations/`, for upstream/downstream service contracts
- validated examples under `examples/`
### Project with public packages or consumer APIs
Required:
- `docs/consumers/api.md`
- one `docs/consumers/pkg-<name>.md` file per public package, if public packages exist
Recommended:
- copyable consumer examples under `examples/`, if practical
## Required Documents
### README.md
**Audience:** users, administrators, operators
The README is the outward-facing project orientation page.
It should include, in order:
1. concise description;
2. elevator pitch;
3. shortest useful command or usage example;
4. links to targeted docs.
The README should be short. It is not a manual.
The “shortest useful command” means the simplest command that performs the projects core use case. (It does not mean `app --help`.)
### docs/policy/architecture.md
**Audience:** developers, LLM coding agents
`docs/policy/architecture.md` is required for every project.
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
It should include:
- project shape;
- core design principles;
- package and boundary philosophy;
- state/persistence philosophy, if applicable;
- external integration philosophy, if applicable;
- error-handling and logging principles;
- testing expectations;
- documentation expectations;
- architectural invariants;
- explicit non-goals, if useful.
Notably, this file should prescribe a core development *policy* that should remain unchanged as the application evolves. It is not a place for details (e.g., CLI flags) that could change over time.
The contents of `architecture.md` should be trim and concise. LLMs may be directed to review it routinely via AGENTS.md, CLAUDE.md, or similar.
### docs/api.md
**Audience:** external HTTP API consumers, developers, LLM coding agents integrating by HTTP
Required for projects whose primary public interface is HTTP.
`docs/api.md` is the canonical public HTTP API contract. It should be normative for external consumers and should not be duplicated by README, operations docs, consumer guides, or integration docs.
It should include:
1. base URL conventions;
2. authentication and authorization behavior, if implemented;
3. response envelope;
4. supported media types and content negotiation behavior;
5. shared query parameters;
6. endpoint reference grouped by route family;
7. request parameters and validation rules;
8. response fields, units, nullability, and optionality;
9. error response shape and status codes;
10. pagination, caching, rate-limit, idempotency, and retry behavior, if implemented;
11. compact request and response examples.
It must document only implemented endpoints and behavior. Planned endpoints, proposed fields, future filters, and experimental response shapes belong only under `docs/roadmap/`.
For HTTP API projects, `docs/consumers/` may provide task-oriented client integration guides, but those guides should link to `docs/api.md` for the authoritative endpoint contract.
### docs/policy/development.md
**Audience:** developers, LLM coding agents
Required for projects maintained by humans and LLM coding agents.
It should include:
- repository layout;
- build/test commands;
- coding conventions;
- dependency policy;
- how to add config fields;
- how to add CLI flags;
- how to add modules or adapters, if applicable;
- how to update examples;
- documentation update expectations.
### docs/config.md
**Audience:** administrators, operators, advanced users
Required for applications with configuration files.
It should include, in order:
1. config file locations and discovery precedence;
2. minimal working config;
3. production-oriented config;
4. full configuration reference;
5. secrets handling, if applicable;
6. links to maintained examples.
The full configuration reference should be canonical.
### docs/cli.md
**Audience:** users, administrators, operators
Required for CLI applications.
It should include, in order:
1. shortest useful command;
2. command overview;
3. complete flag reference;
4. common workflows;
5. diagnostic or recovery commands, if applicable.
Explain when commands are useful, not just their syntax.
### docs/operations.md
**Audience:** administrators, operators
Required for applications that maintain state, support resume behavior, run multi-step workflows, write durable artifacts, use remote storage, or require recovery procedures.
It should cover:
- normal workflow;
- filesystem layout;
- remote storage layout, if applicable;
- logs and manifests;
- resume/retry behavior;
- cleanup behavior;
- archive/backup behavior;
- safe recovery procedures;
- operational caveats.
### docs/troubleshooting.md
**Audience:** administrators, operators
Recommended once recurring failure modes exist.
Each entry should include:
- symptom;
- likely cause;
- diagnostic command or inspection step;
- safe fix;
- relevant links.
### docs/consumers/
**Audience:** developers and LLM coding agents integrating this project from another codebase
Required for projects with public packages, SDKs, client APIs, plugin APIs, or other application-facing integration surfaces.
This directory describes how an external codebase should consume the project's public API. It should be task-oriented and copyable where useful. It is not the place for internal implementation details or operator procedures.
For projects whose public API is HTTP, `docs/consumers/` is not required, and it should not duplicate the endpoint reference in `docs/api.md`. If present, it may provide practical integration workflows, client-specific examples, or migration notes that link back to `docs/api.md`.
`docs/consumers/api.md` should provide the consumer-facing overview and primary implementation workflow. It should include:
1. intended consumer audience and use cases;
2. required inputs supplied by operators or deployment configuration;
3. recommended public package or API workflow;
4. minimal copyable example;
5. consumer responsibilities and boundaries;
6. retry, idempotency, or status behavior, if applicable;
7. links to package-specific docs and canonical integration contracts.
Package-specific docs should be named `pkg-<name>.md` and should include:
1. import path;
2. intended use cases;
3. primary types and functions needed by consumers;
4. minimal examples;
5. validation, error, retry, and boundary behavior;
6. links to canonical file-format or wire-protocol contracts.
### docs/internal/
**Audience:** developers, LLM coding agents
Required for modular, service-oriented, or orchestration projects.
This directory describes implemented internal components. It is not the roadmap.
Use one file per major component where useful.
Each component doc should include:
1. purpose;
2. inputs and outputs;
3. boundaries;
4. config fields used;
5. external adapters used;
6. state or manifest behavior, if applicable;
7. skip/resume behavior, if applicable;
8. failure behavior;
9. tests to inspect before changing;
10. architectural invariants.
### docs/roadmap/
**Audience:** maintainers, developers, LLM coding agents
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
Roadmap docs should clearly distinguish:
- proposed work;
- accepted plans;
- deferred ideas;
- rejected ideas;
- implementation prompts or task breakdowns, if useful.
Roadmap docs should not be confused with current behavior.
### docs/integrations/
**Audience:** developers, LLM coding agents
Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses or exposes.
For public HTTP API services, `docs/integrations/` should document upstream, downstream, storage, protocol, or runtime contracts that the service depends on or bridges. It should not become a second copy of the public HTTP endpoint reference; that belongs in `docs/api.md`.
Use one file per integration where useful.
## Examples Directory
Projects with non-trivial configuration or workflows should include `examples/`.
Useful examples include:
- minimal working config;
- production-oriented config;
- full annotated config;
- local development config;
- remote/object-storage config;
- minimal session/input file.
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
## Security and Privacy
Docs and examples must not include:
- real API keys;
- tokens;
- passwords;
- private keys;
- private environment dumps;
- sensitive user data;
- raw private transcripts;
- private infrastructure details unless intentionally public.
Document secret-handling mechanisms, not actual secret values.
## Maintenance Rules
When docs change, verify the affected behavior.
Where practical:
- load example config files in tests;
- test CLI examples or command parser behavior;
- validate documented flags against real flags;
- remove stale references;
- update links after renames;
- keep roadmap content out of non-roadmap docs.
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
Documentation is complete only when it matches the current code.
## Documentation Change Checklist
Before merging documentation changes, verify:
- README is concise and orientation-focused.
- `docs/policy/architecture.md` describes development principles.
- `docs/api.md` is the canonical HTTP contract for HTTP API services.
- Future work appears only under `docs/roadmap/`.
- User-facing docs avoid unnecessary internals.
- Consumer-facing docs explain public APIs without duplicating HTTP endpoint or integration contracts.
- Developer-facing docs preserve boundaries and invariants.
- Config examples match the schema.
- CLI examples match real commands and flags.
- Defaults appear in the canonical config reference.
- No secrets or private data are included.
- Links are accurate.

View File

@@ -0,0 +1,321 @@
# Library API Implementation Plan
This plan implements the target state in `docs/roadmap/library.md`.
Audience: LLM coding agents implementing the feature in order. Follow `docs/policy/architecture.md`, `docs/policy/development.md`, and `docs/policy/documentation.md` before changing code.
## Constraints
- Add a public root package named `scriptorium`; keep existing `internal/*` packages internal.
- Define public facade types and convert to/from internal domain types. Do not alias internal domain types as the public API.
- Do not rewire CLI or HTTP through the public facade in this implementation.
- Preserve current CLI, HTTP, prompt/profile loading, validation, secret-handling, and outbound LLM behavior.
- Do not add dependencies.
- Keep each stage passing `go test ./...` before moving to the next stage.
## Stage 1: Public Types, Engine Construction, And Prepare
Goal: make prompt preparation usable from an imported root package without calling an LLM.
### Public Package
Create Go files at the module root using:
```go
package scriptorium
```
Expose:
```go
type Engine struct { /* unexported fields */ }
type Config struct {
PromptDir string
ProfileDir string
SchemaDir string
Timeout time.Duration
HTTPClient *http.Client
}
type Option func(*engineOptions) error
func NewEngine(cfg Config, opts ...Option) (*Engine, error)
func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, error)
```
Construction rules:
- `PromptDir` and `ProfileDir` are required.
- `SchemaDir` defaults to the same built-in default used by app config.
- `Timeout`, when non-zero, configures the default OpenAI-compatible client timeout.
- `HTTPClient`, when non-nil, is used by the default OpenAI-compatible client.
- `NewEngine` wires the same internal components used by CLI/HTTP: filesystem prompt/profile repositories, composite artifact reader, Go template renderer, standard validator, and OpenAI-compatible LLM client.
- Return public `ErrInvalidConfig` for invalid engine configuration.
### Public Types
Define public facade types with exported fields:
- `RunRequest`
- `PreparedRun`
- `ArtifactRef`
- `Artifact`
- `ExecutionTarget`
- `ExecutionTargetOverride`
- `ExecutionTargetPresence`
- `OutputContract`
- `ValidationResult`
- `TokenUsage`
- `RenderedMessage`
- `CacheControl`
- `StructuredOutputSpec`
Use the same enum string values as internal domain types for formats, validation modes, validation statuses, artifact ref types, cache-control type, and structured-output type.
Required request shape:
```go
type RunRequest struct {
PromptID string
PromptVersion string
ProfileID string
Inputs map[string]ArtifactRef
Vars map[string]string
Execution *ExecutionTargetOverride
Validation *OutputContract
Metadata map[string]string
}
```
`ExecutionTargetOverride` must preserve numeric override presence using pointer fields:
```go
Temperature *float64
MaxTokens *int
TopP *float64
TimeoutSeconds *int
```
`PreparedRun` should include the same user-observable fields as internal `domain.PreparedRun`, but should not expose internal-only target presence metadata.
### Input Helpers
Expose:
```go
func File(path string) ArtifactRef
func Inline(body string) ArtifactRef
func InlineWithURI(uri string, body string) ArtifactRef
```
Mapping:
- `File(path)` maps to artifact type `file` with `URI: path`.
- `Inline(body)` maps to artifact type `inline` with `Body: body`.
- `InlineWithURI(uri, body)` maps to artifact type `inline` with both fields set.
### Conversion Layer
Implement unexported conversion helpers in the public package:
- public run request to internal `domain.RunRequest`
- internal `domain.PreparedRun` to public `PreparedRun`
- internal artifacts/messages/contracts/validation/usage/structured-output to public equivalents
- public execution override to internal `domain.ExecutionTargetOverride`
Conversions must deep-copy maps and slices that cross the public/internal boundary.
### Tests
Add root package tests.
Required tests:
- `NewEngine` rejects missing `PromptDir`.
- `NewEngine` rejects missing `ProfileDir`.
- `Prepare` works with `examples/config.yml` directories when passed directly through `Config`.
- `Prepare` works with `File` input refs.
- `Prepare` works with `Inline` input refs.
- `Prepare` output does not expose raw API-key values or internal target presence metadata in JSON.
- Explicit zero execution overrides survive into prepared effective settings.
### Verification
Run:
```bash
go test ./...
```
## Stage 2: Run, LLM Injection, And Public Errors
Goal: make full execution usable and testable without real provider credentials.
### Public Run Method
Expose:
```go
func (e *Engine) Run(ctx context.Context, req RunRequest) (*RunResult, error)
```
`RunResult` should expose:
- run ID
- artifact
- raw output
- validation result
- prompt/profile/model metadata
- effective model params
- input hashes
- token/cache usage
- start/end/duration timing
Do not expose raw API-key values.
### Public LLM Injection
Expose:
```go
type LLMClient interface {
Generate(context.Context, GenerateRequest) (*GenerateResponse, error)
}
func WithLLMClient(client LLMClient) Option
```
Public `GenerateRequest` must include:
- rendered prompt
- effective execution target
- execution target presence
- structured-output spec
Public `GenerateResponse` must include:
- content
- token usage
Implementation rule:
- `WithLLMClient` wraps the public client in an unexported adapter that satisfies `internal/llm.Client`.
- The adapter converts internal generate requests to public generate requests and converts public generate responses back to internal responses.
- A nil client passed to `WithLLMClient` returns `ErrInvalidConfig`.
Default behavior:
- If no custom LLM client is supplied, `NewEngine` uses `internal/llm.NewOpenAICompatibleClient`.
- `Config.Timeout` and `Config.HTTPClient` apply only to the default OpenAI-compatible client.
### Public Errors
Define public sentinel errors:
- `ErrInvalidConfig`
- `ErrInvalidRequest`
- `ErrPromptNotFound`
- `ErrProfileNotFound`
- `ErrPromptLoad`
- `ErrProfileLoad`
- `ErrArtifactLoad`
- `ErrPromptRender`
- `ErrLLMGenerate`
- `ErrValidation`
Public methods must map internal errors to public sentinels while preserving wrapped context. Callers must be able to use `errors.Is`.
Mapping rules:
- missing/invalid public engine config -> `ErrInvalidConfig`
- internal `usecase.ErrInvalidRequest` -> `ErrInvalidRequest`
- internal prompt not found -> `ErrPromptNotFound`
- internal profile not found -> `ErrProfileNotFound`
- internal prompt load errors -> `ErrPromptLoad`
- internal profile load errors -> `ErrProfileLoad`
- internal artifact load errors -> `ErrArtifactLoad`
- internal prompt render errors -> `ErrPromptRender`
- internal LLM generate errors -> `ErrLLMGenerate`
- internal validation runtime errors -> `ErrValidation`
Do not expose internal sentinel values as public API.
### Tests
Required tests:
- `Run` succeeds with `WithLLMClient` fake and returns typed artifact, raw output, validation, metadata, and usage.
- `Run` passes rendered prompt, effective execution target, and target presence to the injected LLM client.
- `Run` validation failure returns a successful result with failed validation, not an error.
- public errors support `errors.Is` for invalid request, prompt not found, profile not found, artifact load, render failure, LLM failure, and validation runtime failure.
- nil `WithLLMClient(nil)` returns `ErrInvalidConfig`.
- default OpenAI-compatible client can still be constructed without real provider credentials.
### Verification
Run:
```bash
go test ./...
```
## Stage 3: Public Documentation And Consumer Examples
Goal: document implemented library behavior in canonical public-consumer docs.
### Docs
After Stages 1 and 2 are implemented, update:
- `README.md`: add a short link to library usage without turning the README into a manual.
- `docs/internal/adapters.md`: list the public library facade as an implemented adapter surface.
- `docs/consumers/api.md`: describe the public consumer API at a high level.
- `docs/consumers/pkg-scriptorium.md`: document the root package usage, types, errors, and examples.
Create `docs/consumers/` if it does not exist.
Do not document unimplemented future library features outside `docs/roadmap/`.
### Examples
Add copyable library examples only if they can be tested without real credentials.
Recommended example:
- `examples/go-library/prepare/main.go` or equivalent prepare-only example using `examples/` prompt/profile/fixture assets.
If adding a run example, it must use an injected fake LLM client and must not require provider credentials.
### Tests
Required tests:
- doc/example smoke coverage for any added Go example using `go test` or `go test ./...`.
- existing CLI/HTTP tests continue to pass unchanged.
### Verification
Run:
```bash
go test ./...
go run ./cmd/scriptorium render \
--config ./examples/config.yml \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \
--format json
```
## Final Checks
Before considering the feature complete:
1. Confirm the root package can be imported as `gitea.maximumdirect.net/eric/scriptorium`.
2. Confirm public package tests do not require real provider credentials.
3. Confirm `go test ./...` passes.
4. Confirm the render smoke command passes.
5. Confirm non-roadmap docs describe only implemented behavior.
6. Confirm no public result or rendered/prepared output exposes raw API-key values.
7. Confirm `git diff` does not include unrelated CLI/HTTP behavior changes.

134
docs/roadmap/library.md Normal file
View File

@@ -0,0 +1,134 @@
# Library API Roadmap
This roadmap defines the target behavior for making Scriptorium usable as an imported Go library while retaining the current standalone CLI and HTTP application behavior.
The implementation plan for this feature lives in `docs/roadmap/implementation.md`.
## Motivation
Scriptorium is currently optimized for subprocess use by other applications. That contract remains useful because it is language-neutral, operationally simple, and process-isolated.
For Go callers, an imported library should provide:
- typed requests and results instead of stdout/stderr parsing;
- direct `context.Context` cancellation;
- lower overhead for repeated calls;
- easier test integration through injected clients or fixtures;
- direct access to prepared-run data without process management;
- fewer integration points where secrets or output metadata can be mishandled.
The library is an additional adapter surface, not a replacement for the CLI or HTTP API.
## Target State
Scriptorium should expose a small public Go API suitable for common embedding use cases:
- construct an engine from app-level settings such as prompt, profile, and schema directories;
- prepare a prompt request without calling an LLM;
- run a prompt request and receive a typed result;
- pass file and inline artifacts;
- apply profile selection, runtime overrides, vars, validation behavior, cache-control behavior, and structured-output behavior consistently with CLI/HTTP;
- inject a custom LLM client or HTTP client where needed;
- preserve existing CLI and HTTP behavior by continuing to route all entry paths through the same use-case layer.
The public library API should be stable, narrow, and intentionally higher-level than the current `internal/*` package layout.
## Public Package Policy
The public package should be the module root:
```go
import "gitea.maximumdirect.net/eric/scriptorium"
```
Recommended usage shape:
```go
engine, err := scriptorium.NewEngine(scriptorium.Config{
PromptDir: "./prompts",
ProfileDir: "./profiles",
SchemaDir: "./schemas",
})
if err != nil {
return err
}
prepared, err := engine.Prepare(ctx, scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.File("./transcript.md"),
},
})
result, err := engine.Run(ctx, scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.File("./transcript.md"),
},
})
```
## Policy Decisions
### Public Package Scope
Expose a narrow root facade package and keep existing `internal/*` packages internal.
Reasoning:
This gives callers the workflow they need without freezing the internal architecture as public API. It also preserves the current package-boundary policy and keeps future refactoring possible.
### Public Type Strategy
Define public facade types and map them to internal domain types.
Reasoning:
Public types can be designed around caller needs and long-term stability. Internal types can continue to evolve with implementation details such as adapter metadata, validation internals, and provider-specific behavior.
### CLI And HTTP Reuse
Keep CLI and HTTP on current internal wiring for the initial library release. Consider migrating them to the public facade only after the facade proves stable.
Reasoning:
This minimizes risk to the existing subprocess and HTTP contracts while adding the new API. It also avoids forcing the first public facade to satisfy every adapter edge case immediately.
### Error Surface
Expose public sentinel errors or typed error categories and map internal errors to them while preserving wrapped context.
Reasoning:
Library callers need stable, idiomatic error checks. Mapping internal errors avoids exposing internal package paths as public compatibility promises.
## Scope
In scope:
- Public facade package for library consumers.
- Public request, result, prepared-run, artifact reference, execution override, validation, and config types.
- Public constructors for common file and inline input references.
- Public engine methods for `Prepare` and `Run`.
- Optional dependency injection for LLM behavior and HTTP behavior.
- Stable error behavior suitable for `errors.Is` and `errors.As`.
- Tests proving public API behavior matches CLI/use-case behavior.
- Documentation and examples for library usage after implementation.
Out of scope for the first library release:
- Making every `internal/*` package public.
- Replacing or rewiring the CLI or HTTP adapters.
- Adding a durable run store or workflow engine.
- Adding broad provider-specific SDK surfaces.
- Adding non-Go language bindings.
- Adding global mutable configuration.
## Acceptance Criteria
- A Go caller can import the root module and run a prompt without invoking a subprocess.
- A Go caller can prepare a prompt without invoking an LLM.
- Public library behavior matches current CLI/HTTP use-case semantics for prompt/profile loading, artifact reading, rendering, validation, and model invocation.
- Existing CLI and HTTP behavior remains unchanged.
- Library tests use injected/fake LLM behavior and do not require real provider credentials.
- Public documentation is concise and limited to implemented behavior once code exists.

367
docs/troubleshooting.md Normal file
View File

@@ -0,0 +1,367 @@
# Troubleshooting
This guide lists recurring implemented failure modes and safe fixes.
For command syntax, see [CLI reference](cli.md). For configuration and file formats, see [Configuration reference](config.md). For operational behavior, see [Operations guide](operations.md).
## Missing Or Invalid Config File
Symptom:
- CLI errors such as `application config error: config file not found` or `invalid config YAML`.
Likely cause:
- `--config` points to a missing file.
- Config YAML has syntax errors or unknown fields.
Diagnostic step:
```bash
go run ./cmd/scriptorium render --config /path/to/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml
```
Safe fix:
- Correct file path.
- Remove unknown fields.
- Fix YAML syntax.
- Keep secrets out of config.
Relevant links:
- [Configuration reference](config.md)
- [CLI reference](cli.md)
## Missing Prompt/Profile Directory Settings
Symptom:
- CLI parse errors saying prompt directory or profile directory is required.
Likely cause:
- Neither CLI flags nor config provide effective `prompt_dir` / `profile_dir`.
Diagnostic step:
- Run the failing command with explicit `--prompt-dir` and `--profile-dir` once to verify.
Safe fix:
- Set `prompt_dir` and `profile_dir` in config, or always pass both flags.
Relevant links:
- [Configuration reference](config.md)
- [CLI reference](cli.md)
## Unknown Or Unsupported Flags
Symptom:
- CLI parse error for an unknown flag.
Likely cause:
- Typo or command mismatch (for example, `serve` with runtime model override flags).
Diagnostic step:
- Compare command against the command-specific flag list.
Safe fix:
- Remove unsupported flags.
- Use `run`/`render` for runtime model overrides.
Relevant links:
- [CLI reference](cli.md)
## Prompt Definition Load Failures
Symptom:
- CLI run/render error from prompt loading.
- HTTP `404 prompt_not_found` or `400 prompt_load_failed`.
Likely cause:
- Prompt ID not found.
- Invalid prompt YAML.
- Invalid prompt contract (for example bad validation mode, message content/content_file rule violation, missing schema path for `json_schema`).
Diagnostic step:
```bash
go run ./cmd/scriptorium render --config ./examples/config.yml --prompt <prompt-id> --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml --format json
```
Safe fix:
- Correct prompt ID.
- Fix prompt YAML and contract fields.
- Ensure referenced `content_file` paths exist.
Relevant links:
- [Configuration reference](config.md)
- [CLI reference](cli.md)
## Profile Definition Load Failures
Symptom:
- CLI run/render error from profile loading.
- HTTP `404 profile_not_found` or `400 profile_load_failed`.
Likely cause:
- Profile ID missing/not found.
- Invalid profile YAML.
- Invalid profile values.
- Raw `api_key` field present (rejected).
Diagnostic step:
```bash
go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --profile <profile-id> --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml
```
Safe fix:
- Correct profile ID.
- Fix profile YAML and value ranges.
- Replace `api_key` with `api_key_env`.
Relevant links:
- [Configuration reference](config.md)
- [CLI reference](cli.md)
## Input Artifact Read Failures
Symptom:
- CLI run/render error reading input artifacts.
- HTTP `400 artifact_read_failed`.
Likely cause:
- File path in input mapping does not exist or is unreadable.
- Unsupported artifact reference type in HTTP request.
Diagnostic step:
- Verify every mapped file path exists and is readable by the process.
- For HTTP, verify each input uses supported `type` values.
Safe fix:
- Correct file paths and permissions.
- Use supported input types (`file`, `inline`).
Relevant links:
- [CLI reference](cli.md)
- [Configuration reference](config.md)
## Prompt Template Render Failures
Symptom:
- CLI run/render error from prompt rendering.
- HTTP `400 prompt_render_failed`.
Likely cause:
- Template references missing input names.
- Template syntax or data reference issues.
Diagnostic step:
- Run `render --format json` with the same prompt, inputs, vars, and profile selection.
Safe fix:
- Align template `{{input "name"}}` references with actual input mappings.
- Fix template syntax and variable names.
Relevant links:
- [CLI reference](cli.md)
- [Configuration reference](config.md)
## Missing API-Key Environment Variable
Symptom:
- CLI run/render invalid request error about missing API-key environment variable.
- HTTP `400 api_key_env_missing`.
Likely cause:
- Selected profile or override sets `api_key_env`, but that environment variable is unset/empty.
Diagnostic step:
```bash
printenv SCRIPTORIUM_API_KEY
```
Safe fix:
- Set the required environment variable before invoking CLI/service.
- Or use a profile that does not require API key auth for the target endpoint.
Relevant links:
- [Configuration reference](config.md)
- [Operations guide](operations.md)
## LLM Request Failures
Symptom:
- CLI `run` fails with LLM generation errors.
- HTTP returns `502 llm_failed`.
Likely cause:
- Endpoint unreachable.
- Non-2xx response from provider.
- Timeout.
- Malformed provider response.
Diagnostic step:
- Confirm endpoint URL and model in selected profile/overrides.
- Retry with `render` first to confirm pre-LLM preparation works.
- Check provider/network logs for non-2xx responses and timeouts.
Safe fix:
- Correct endpoint/model settings.
- Adjust timeout if needed.
- Resolve provider-side or network issues.
Relevant links:
- [CLI reference](cli.md)
- [Configuration reference](config.md)
- [Operations guide](operations.md)
## Prompt Cache Misses Or No Cache Usage
Symptom:
- CLI run summary omits `cached_tokens` / `cache_write_tokens`.
- HTTP `metadata.usage.cached_tokens` and `metadata.usage.cache_write_tokens` are both `0`.
- Provider cost or latency does not improve after repeated similar runs.
Likely cause:
- The selected prompt has no `messages[].cache_control`.
- Dynamic per-run input appears before the cache-controlled message and changes the provider cache key.
- The provider does not support the serialized cache-control shape for the selected model.
- The provider imposes minimum token thresholds or cache-breakpoint limits.
Diagnostic step:
- Run `render --format json` and verify the intended rendered message includes `cache_control`.
- Confirm stable reusable context appears before the cache-controlled message, with dynamic input after it.
- Check provider docs/logs for model support, minimum token thresholds, and breakpoint limits.
Safe fix:
- Move stable reusable context before the cache-controlled message.
- Move highly dynamic input after the cache breakpoint.
- Keep `cache_control.type: ephemeral` and, when using `ttl`, set `ttl: 1h`.
- Use CLI cache counters or HTTP cache usage fields to verify cache reads/writes after rerunning.
Relevant links:
- [Configuration reference](config.md)
- [OpenAI-compatible chat integration](integrations/openai-compatible-chat.md)
## Validation Status Failed (`run` Exit 2 Or HTTP 200 With Failed Status)
Symptom:
- CLI exits with code `2`.
- HTTP returns `200`, but `validation.status` is `failed`.
Likely cause:
- Generated output failed `basic`, `json`, or `json_schema` content validation.
Diagnostic step:
- Inspect validation mode and validation errors in CLI summary/HTTP response.
Safe fix:
- Refine prompt constraints.
- Tighten schema or adjust model/profile settings.
- Rerun after correction.
Relevant links:
- [Configuration reference](config.md)
- [Operations guide](operations.md)
## Validation Runtime Failure
Symptom:
- CLI `run` fails with validation runtime error.
- HTTP returns `500 validation_runtime_failed`.
Likely cause:
- `json_schema` schema file missing/inaccessible.
- Invalid schema JSON document.
Diagnostic step:
- Verify `schema_dir` and `output.schema_path` resolution.
- Check schema file readability and valid JSON syntax.
Safe fix:
- Correct schema path.
- Fix schema JSON content.
- Rerun.
Relevant links:
- [Configuration reference](config.md)
- [Operations guide](operations.md)
## HTTP Request Parsing/Contract Errors
Symptom:
- HTTP `400 invalid_json` or `400 invalid_request`.
Likely cause:
- Malformed JSON body.
- Unknown JSON fields.
- Missing required `prompt_id` or `inputs`.
Diagnostic step:
- Revalidate request JSON.
- Confirm required request fields are present.
Safe fix:
- Send valid JSON with only supported fields.
- Ensure `prompt_id` and at least one input mapping are included.
Relevant links:
- [Operations guide](operations.md)
- [CLI reference](cli.md)

141
engine.go Normal file
View File

@@ -0,0 +1,141 @@
package scriptorium
import (
"context"
"errors"
"fmt"
"net/http"
"strings"
"time"
artifactadapter "gitea.maximumdirect.net/eric/scriptorium/internal/artifact"
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
"gitea.maximumdirect.net/eric/scriptorium/internal/llm"
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
"gitea.maximumdirect.net/eric/scriptorium/internal/prompt"
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
"gitea.maximumdirect.net/eric/scriptorium/internal/usecase"
"gitea.maximumdirect.net/eric/scriptorium/internal/validate"
)
// ErrInvalidConfig indicates invalid public engine configuration.
var ErrInvalidConfig = errors.New("invalid engine configuration")
var (
ErrInvalidRequest = errors.New("invalid run request")
ErrPromptNotFound = errors.New("prompt not found")
ErrProfileNotFound = errors.New("profile not found")
ErrPromptLoad = errors.New("failed to load prompt definition")
ErrProfileLoad = errors.New("failed to load execution profile")
ErrArtifactLoad = errors.New("failed to load artifact")
ErrPromptRender = errors.New("failed to render prompt")
ErrLLMGenerate = errors.New("failed to generate output")
ErrValidation = errors.New("failed to validate output")
)
// Engine prepares and runs Scriptorium prompt requests.
type Engine struct {
runner *usecase.Runner
}
// Config configures a public Scriptorium engine.
type Config struct {
PromptDir string
ProfileDir string
SchemaDir string
Timeout time.Duration
HTTPClient *http.Client
}
// Option customizes engine construction.
type Option func(*engineOptions) error
type engineOptions struct {
llmClient llm.Client
}
// WithLLMClient injects a custom LLM client for execution.
func WithLLMClient(client LLMClient) Option {
return func(options *engineOptions) error {
if client == nil {
return ErrInvalidConfig
}
options.llmClient = publicLLMClientAdapter{client: client}
return nil
}
}
// NewEngine constructs an Engine using the same default internal components as
// the CLI and HTTP adapters.
func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
if strings.TrimSpace(cfg.PromptDir) == "" {
return nil, fmt.Errorf("%w: prompt directory is required", ErrInvalidConfig)
}
if strings.TrimSpace(cfg.ProfileDir) == "" {
return nil, fmt.Errorf("%w: profile directory is required", ErrInvalidConfig)
}
var options engineOptions
for _, opt := range opts {
if opt == nil {
continue
}
if err := opt(&options); err != nil {
return nil, fmt.Errorf("%w: %v", ErrInvalidConfig, err)
}
}
schemaDir := cfg.SchemaDir
if strings.TrimSpace(schemaDir) == "" {
schemaDir = defaults.SchemaDirDefault
}
llmClient := options.llmClient
if llmClient == nil {
var err error
llmClient, err = llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
Timeout: cfg.Timeout,
HTTPClient: cfg.HTTPClient,
})
if err != nil {
return nil, fmt.Errorf("%w: %v", ErrInvalidConfig, err)
}
}
return &Engine{
runner: usecase.NewRunner(
promptdef.NewFilesystemRepository(cfg.PromptDir),
profile.NewFilesystemRepository(cfg.ProfileDir),
artifactadapter.NewCompositeReader(),
prompt.NewGoRenderer(),
llmClient,
validate.NewStandardValidator(schemaDir),
),
}, nil
}
// Prepare resolves a prompt request without calling an LLM.
func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, error) {
if e == nil || e.runner == nil {
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
}
prepared, err := e.runner.Prepare(ctx, toDomainRunRequest(req))
if err != nil {
return nil, mapPublicError(err)
}
return fromDomainPreparedRun(prepared), nil
}
// Run executes a prompt request and returns the generated artifact and metadata.
func (e *Engine) Run(ctx context.Context, req RunRequest) (*RunResult, error) {
if e == nil || e.runner == nil {
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
}
result, err := e.runner.Run(ctx, toDomainRunRequest(req))
if err != nil {
return nil, mapPublicError(err)
}
return fromDomainRunResult(result), nil
}

429
engine_test.go Normal file
View File

@@ -0,0 +1,429 @@
package scriptorium_test
import (
"context"
"encoding/json"
"errors"
"os"
"strings"
"testing"
"gitea.maximumdirect.net/eric/scriptorium"
)
func TestNewEngineRejectsMissingPromptDir(t *testing.T) {
_, err := scriptorium.NewEngine(scriptorium.Config{ProfileDir: "./examples/profiles"})
if !errors.Is(err, scriptorium.ErrInvalidConfig) {
t.Fatalf("expected ErrInvalidConfig, got %v", err)
}
}
func TestNewEngineRejectsMissingProfileDir(t *testing.T) {
_, err := scriptorium.NewEngine(scriptorium.Config{PromptDir: "./examples/prompts"})
if !errors.Is(err, scriptorium.ErrInvalidConfig) {
t.Fatalf("expected ErrInvalidConfig, got %v", err)
}
}
func TestPrepareWorksWithExampleDirectoriesAndFileInputs(t *testing.T) {
engine := newExampleEngine(t)
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.File("./examples/fixtures/transcript.md"),
"glossary": scriptorium.File("./examples/fixtures/glossary.yml"),
},
})
if err != nil {
t.Fatalf("expected prepare to succeed, got %v", err)
}
if prepared.PromptID != "generic.markdown_summary" {
t.Fatalf("unexpected prompt id: %q", prepared.PromptID)
}
if prepared.SelectedProfileID != "local-fast" {
t.Fatalf("unexpected selected profile: %q", prepared.SelectedProfileID)
}
if prepared.EffectiveModelParams.Model != "gpt-4o-mini" {
t.Fatalf("unexpected effective model: %q", prepared.EffectiveModelParams.Model)
}
if len(prepared.Messages) != 2 {
t.Fatalf("expected rendered messages, got %d", len(prepared.Messages))
}
if prepared.InputHashes["transcript"] == "" || prepared.InputHashes["glossary"] == "" {
t.Fatalf("expected input hashes, got %#v", prepared.InputHashes)
}
}
func TestPrepareWorksWithInlineInputs(t *testing.T) {
engine := newExampleEngine(t)
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.Inline("Rin scouts the tower.\nKara lights a lantern."),
"glossary": scriptorium.InlineWithURI("memory://glossary.yml", "party:\n - Rin\n - Kara\n"),
},
})
if err != nil {
t.Fatalf("expected prepare to succeed, got %v", err)
}
if len(prepared.Messages) != 2 {
t.Fatalf("expected rendered messages, got %d", len(prepared.Messages))
}
rendered := prepared.Messages[1].Content
if !strings.Contains(rendered, "Rin scouts the tower.") || !strings.Contains(rendered, "party:") {
t.Fatalf("expected inline inputs in rendered prompt, got %q", rendered)
}
}
func TestPreparedRunJSONDoesNotExposeSecretOrTargetPresence(t *testing.T) {
const envName = "SCRIPTORIUM_API_KEY"
const secret = "public-api-test-secret"
t.Setenv(envName, secret)
engine := newExampleEngine(t)
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
PromptID: "generic.structured_events",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.File("./examples/fixtures/transcript.md"),
"glossary": scriptorium.File("./examples/fixtures/glossary.yml"),
},
})
if err != nil {
t.Fatalf("expected prepare to succeed, got %v", err)
}
payload, err := json.Marshal(prepared)
if err != nil {
t.Fatalf("expected prepared run to marshal, got %v", err)
}
out := string(payload)
if strings.Contains(out, secret) {
t.Fatalf("prepared run JSON leaked raw API key value: %s", out)
}
if !strings.Contains(out, envName) {
t.Fatalf("prepared run JSON should retain api_key_env name, got %s", out)
}
for _, forbidden := range []string{"TargetPresence", "target_presence"} {
if strings.Contains(out, forbidden) {
t.Fatalf("prepared run JSON exposed internal target presence metadata %q: %s", forbidden, out)
}
}
}
func TestPreparePreservesExplicitZeroExecutionOverrides(t *testing.T) {
engine := newExampleEngine(t)
zeroFloat := 0.0
zeroInt := 0
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.File("./examples/fixtures/transcript.md"),
"glossary": scriptorium.File("./examples/fixtures/glossary.yml"),
},
Execution: &scriptorium.ExecutionTargetOverride{
Temperature: &zeroFloat,
MaxTokens: &zeroInt,
TopP: &zeroFloat,
TimeoutSeconds: &zeroInt,
},
})
if err != nil {
t.Fatalf("expected prepare to succeed, got %v", err)
}
target := prepared.EffectiveModelParams
if target.Temperature != 0 || target.MaxTokens != 0 || target.TopP != 0 || target.TimeoutSeconds != 0 {
t.Fatalf("expected explicit zero overrides in effective target, got %+v", target)
}
}
func TestRunSucceedsWithInjectedLLMClient(t *testing.T) {
const envName = "SCRIPTORIUM_API_KEY"
const secret = "run-secret-value"
t.Setenv(envName, secret)
fake := &fakeLLMClient{
response: &scriptorium.GenerateResponse{
Content: "# Summary\n\nDone.",
Usage: scriptorium.TokenUsage{
PromptTokens: 10,
CompletionTokens: 5,
TotalTokens: 15,
CachedTokens: 3,
CacheWriteTokens: 2,
},
},
}
engine := newExampleEngineWithOptions(t, "./examples/schemas", scriptorium.WithLLMClient(fake))
result, err := engine.Run(context.Background(), scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.Inline("Rin opens the gate."),
"glossary": scriptorium.Inline("gate: A guarded passage."),
},
Execution: &scriptorium.ExecutionTargetOverride{
APIKeyEnv: envName,
},
})
if err != nil {
t.Fatalf("expected run to succeed, got %v", err)
}
if result.RunID == "" {
t.Fatalf("expected run id")
}
if result.RawOutput != fake.response.Content {
t.Fatalf("unexpected raw output: %q", result.RawOutput)
}
if string(result.Artifact.Body) != fake.response.Content {
t.Fatalf("unexpected artifact body: %q", string(result.Artifact.Body))
}
if result.Artifact.ContentType != "text/markdown" {
t.Fatalf("unexpected artifact content type: %q", result.Artifact.ContentType)
}
if result.Validation.Status != scriptorium.ValidationPassed || !result.Validation.IsValid {
t.Fatalf("expected passed validation, got %+v", result.Validation)
}
if result.PromptID != "generic.markdown_summary" || result.SelectedProfileID != "local-fast" || result.ModelName != "gpt-4o-mini" {
t.Fatalf("unexpected run metadata: %+v", result)
}
if result.Usage.TotalTokens != 15 || result.Usage.CachedTokens != 3 || result.Usage.CacheWriteTokens != 2 {
t.Fatalf("unexpected usage: %+v", result.Usage)
}
payload, err := json.Marshal(result)
if err != nil {
t.Fatalf("expected run result to marshal, got %v", err)
}
if strings.Contains(string(payload), secret) {
t.Fatalf("run result JSON leaked raw API key value: %s", payload)
}
}
func TestRunPassesPreparedRequestToInjectedLLMClient(t *testing.T) {
fake := &fakeLLMClient{
response: &scriptorium.GenerateResponse{Content: "ok"},
}
engine := newExampleEngineWithOptions(t, "./examples/schemas", scriptorium.WithLLMClient(fake))
zeroFloat := 0.0
zeroInt := 0
_, err := engine.Run(context.Background(), scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.Inline("Rin opens the gate."),
"glossary": scriptorium.Inline("gate: A guarded passage."),
},
Execution: &scriptorium.ExecutionTargetOverride{
Temperature: &zeroFloat,
MaxTokens: &zeroInt,
TopP: &zeroFloat,
TimeoutSeconds: &zeroInt,
},
})
if err != nil {
t.Fatalf("expected run to succeed, got %v", err)
}
if len(fake.requests) != 1 {
t.Fatalf("expected one generate request, got %d", len(fake.requests))
}
req := fake.requests[0]
if len(req.Prompt.Messages) != 2 || !strings.Contains(req.Prompt.Messages[1].Content, "Rin opens the gate.") {
t.Fatalf("expected rendered prompt in generate request, got %+v", req.Prompt)
}
if req.Target.Model != "gpt-4o-mini" || req.Target.Temperature != 0 || req.Target.MaxTokens != 0 || req.Target.TopP != 0 || req.Target.TimeoutSeconds != 0 {
t.Fatalf("unexpected effective target: %+v", req.Target)
}
if !req.TargetPresence.Temperature || !req.TargetPresence.MaxTokens || !req.TargetPresence.TopP || !req.TargetPresence.TimeoutSeconds {
t.Fatalf("expected explicit zero target presence, got %+v", req.TargetPresence)
}
if req.StructuredOutput != nil {
t.Fatalf("did not expect structured output for markdown prompt: %+v", req.StructuredOutput)
}
}
func TestRunValidationFailureReturnsResult(t *testing.T) {
fake := &fakeLLMClient{
response: &scriptorium.GenerateResponse{Content: ""},
}
engine := newExampleEngineWithOptions(t, "./examples/schemas", scriptorium.WithLLMClient(fake))
result, err := engine.Run(context.Background(), scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.Inline("Rin opens the gate."),
"glossary": scriptorium.Inline("gate: A guarded passage."),
},
})
if err != nil {
t.Fatalf("expected validation failure as successful result, got %v", err)
}
if result.Validation.Status != scriptorium.ValidationFailed || result.Validation.IsValid {
t.Fatalf("expected failed validation result, got %+v", result.Validation)
}
if len(result.Validation.Errors) == 0 {
t.Fatalf("expected validation errors")
}
}
func TestPublicErrorsSupportErrorsIs(t *testing.T) {
llmErr := errors.New("llm failed")
tests := []struct {
name string
req scriptorium.RunRequest
client scriptorium.LLMClient
schemaDir string
want error
}{
{
name: "invalid request",
req: scriptorium.RunRequest{},
client: &fakeLLMClient{response: &scriptorium.GenerateResponse{Content: "ok"}},
want: scriptorium.ErrInvalidRequest,
},
{
name: "prompt not found",
req: scriptorium.RunRequest{PromptID: "missing.prompt"},
client: &fakeLLMClient{response: &scriptorium.GenerateResponse{Content: "ok"}},
want: scriptorium.ErrPromptNotFound,
},
{
name: "profile not found",
req: scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
ProfileID: "missing-profile",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.Inline("Rin opens the gate."),
},
},
client: &fakeLLMClient{response: &scriptorium.GenerateResponse{Content: "ok"}},
want: scriptorium.ErrProfileNotFound,
},
{
name: "artifact load",
req: scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.File("./examples/fixtures/does-not-exist.md"),
},
},
client: &fakeLLMClient{response: &scriptorium.GenerateResponse{Content: "ok"}},
want: scriptorium.ErrArtifactLoad,
},
{
name: "prompt render",
req: scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
},
client: &fakeLLMClient{response: &scriptorium.GenerateResponse{Content: "ok"}},
want: scriptorium.ErrPromptRender,
},
{
name: "llm failure",
req: scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.Inline("Rin opens the gate."),
"glossary": scriptorium.Inline("gate: A guarded passage."),
},
},
client: &fakeLLMClient{err: llmErr},
want: scriptorium.ErrLLMGenerate,
},
{
name: "validation runtime failure",
req: scriptorium.RunRequest{
PromptID: "generic.structured_events",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.Inline("Rin opens the gate."),
},
},
client: &fakeLLMClient{response: &scriptorium.GenerateResponse{Content: `{"events":[]}`}},
schemaDir: t.TempDir(),
want: scriptorium.ErrValidation,
},
}
t.Setenv("SCRIPTORIUM_API_KEY", "test-secret")
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
schemaDir := tc.schemaDir
if schemaDir == "" {
schemaDir = "./examples/schemas"
}
engine := newExampleEngineWithOptions(t, schemaDir, scriptorium.WithLLMClient(tc.client))
_, err := engine.Run(context.Background(), tc.req)
if !errors.Is(err, tc.want) {
t.Fatalf("expected errors.Is(%v), got %v", tc.want, err)
}
})
}
}
func TestWithLLMClientRejectsNilClient(t *testing.T) {
_, err := scriptorium.NewEngine(exampleConfig("./examples/schemas"), scriptorium.WithLLMClient(nil))
if !errors.Is(err, scriptorium.ErrInvalidConfig) {
t.Fatalf("expected ErrInvalidConfig, got %v", err)
}
}
func TestNewEngineConstructsDefaultLLMClientWithoutCredentials(t *testing.T) {
if _, err := scriptorium.NewEngine(exampleConfig("./examples/schemas")); err != nil {
t.Fatalf("expected default engine construction without credentials to succeed, got %v", err)
}
}
func newExampleEngine(t *testing.T) *scriptorium.Engine {
t.Helper()
for _, path := range []string{
"./examples/prompts",
"./examples/profiles",
"./examples/schemas",
} {
if _, err := os.Stat(path); err != nil {
t.Fatalf("expected example path %s to exist: %v", path, err)
}
}
engine, err := scriptorium.NewEngine(exampleConfig("./examples/schemas"))
if err != nil {
t.Fatalf("expected engine construction to succeed, got %v", err)
}
return engine
}
func newExampleEngineWithOptions(t *testing.T, schemaDir string, opts ...scriptorium.Option) *scriptorium.Engine {
t.Helper()
engine, err := scriptorium.NewEngine(exampleConfig(schemaDir), opts...)
if err != nil {
t.Fatalf("expected engine construction to succeed, got %v", err)
}
return engine
}
func exampleConfig(schemaDir string) scriptorium.Config {
return scriptorium.Config{
PromptDir: "./examples/prompts",
ProfileDir: "./examples/profiles",
SchemaDir: schemaDir,
}
}
type fakeLLMClient struct {
response *scriptorium.GenerateResponse
err error
requests []scriptorium.GenerateRequest
}
func (f *fakeLLMClient) Generate(_ context.Context, req scriptorium.GenerateRequest) (*scriptorium.GenerateResponse, error) {
f.requests = append(f.requests, req)
if f.err != nil {
return nil, f.err
}
return f.response, nil
}

71
errors.go Normal file
View File

@@ -0,0 +1,71 @@
package scriptorium
import (
"errors"
"fmt"
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
"gitea.maximumdirect.net/eric/scriptorium/internal/usecase"
)
func mapPublicError(err error) error {
if err == nil {
return nil
}
if hasPublicError(err) {
return err
}
publicErr := publicErrorFor(err)
if publicErr == nil {
return err
}
return fmt.Errorf("%w: %w", publicErr, err)
}
func hasPublicError(err error) bool {
for _, publicErr := range []error{
ErrInvalidConfig,
ErrInvalidRequest,
ErrPromptNotFound,
ErrProfileNotFound,
ErrPromptLoad,
ErrProfileLoad,
ErrArtifactLoad,
ErrPromptRender,
ErrLLMGenerate,
ErrValidation,
} {
if errors.Is(err, publicErr) {
return true
}
}
return false
}
func publicErrorFor(err error) error {
switch {
case errors.Is(err, promptdef.ErrPromptDefinitionNotFound):
return ErrPromptNotFound
case errors.Is(err, profile.ErrProfileNotFound):
return ErrProfileNotFound
case errors.Is(err, promptdef.ErrInvalidYAML), errors.Is(err, promptdef.ErrInvalidPromptDefinition):
return ErrPromptLoad
case errors.Is(err, profile.ErrInvalidYAML), errors.Is(err, profile.ErrInvalidProfile):
return ErrProfileLoad
case errors.Is(err, usecase.ErrArtifactLoad):
return ErrArtifactLoad
case errors.Is(err, usecase.ErrPromptRender):
return ErrPromptRender
case errors.Is(err, usecase.ErrLLMGenerate):
return ErrLLMGenerate
case errors.Is(err, usecase.ErrValidation):
return ErrValidation
case errors.Is(err, usecase.ErrInvalidRequest):
return ErrInvalidRequest
case errors.Is(err, usecase.ErrProfileLoad):
return ErrPromptLoad
default:
return nil
}
}

View File

@@ -1,6 +1,6 @@
prompt_dir: ./prompts
profile_dir: ./profiles
schema_dir: ./schemas
prompt_dir: ./examples/prompts
profile_dir: ./examples/profiles
schema_dir: ./examples/schemas
server:
addr: :8080

View File

@@ -0,0 +1,50 @@
package main
import (
"context"
"encoding/json"
"log"
"os"
"gitea.maximumdirect.net/eric/scriptorium"
)
func main() {
engine, err := scriptorium.NewEngine(scriptorium.Config{
PromptDir: "./examples/prompts",
ProfileDir: "./examples/profiles",
SchemaDir: "./examples/schemas",
})
if err != nil {
log.Fatal(err)
}
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
PromptID: "generic.markdown_summary",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.File("./examples/fixtures/transcript.md"),
"glossary": scriptorium.File("./examples/fixtures/glossary.yml"),
},
})
if err != nil {
log.Fatal(err)
}
summary := struct {
PromptID string `json:"prompt_id"`
SelectedProfileID string `json:"selected_profile_id"`
Model string `json:"model"`
MessageCount int `json:"message_count"`
InputHashes map[string]string `json:"input_hashes"`
}{
PromptID: prepared.PromptID,
SelectedProfileID: prepared.SelectedProfileID,
Model: prepared.EffectiveModelParams.Model,
MessageCount: len(prepared.Messages),
InputHashes: prepared.InputHashes,
}
if err := json.NewEncoder(os.Stdout).Encode(summary); err != nil {
log.Fatal(err)
}
}

18
examples/http-run.json Normal file
View File

@@ -0,0 +1,18 @@
{
"prompt_id": "generic.markdown_summary",
"profile_id": "local-fast",
"inputs": {
"transcript": {
"type": "file",
"uri": "./examples/fixtures/transcript.md"
},
"glossary": {
"type": "file",
"uri": "./examples/fixtures/glossary.yml"
}
},
"vars": {
"session_date": "2026-05-04"
},
"include_raw_output": false
}

View File

@@ -0,0 +1,13 @@
#!/usr/bin/env bash
set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$repo_root"
go run ./cmd/scriptorium render \
--config ./examples/config.yml \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \
--format text

View File

@@ -81,6 +81,14 @@ type serveConfig struct {
schemaDir string
}
type commonCommandSettings struct {
promptDir string
profileDir string
schemaDir string
serverAddr string
defaultRenderFormat renderformat.PreparedRunOutputFormat
}
type listFlag []string
func (l *listFlag) String() string {
@@ -125,22 +133,13 @@ func runCommand(args []string, stdout, stderr io.Writer) int {
return ExitRuntimeError
}
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
Timeout: defaults.LLMRequestTimeoutDefault,
})
llmClient, err := newOpenAIClient()
if err != nil {
fmt.Fprintf(stderr, "llm client error: %v\n", err)
return ExitRuntimeError
}
runner := usecase.NewRunner(
promptdef.NewFilesystemRepository(cfg.promptDir),
profile.NewFilesystemRepository(cfg.profileDir),
artifactadapter.NewCompositeReader(),
prompt.NewGoRenderer(),
llmClient,
validate.NewStandardValidator(cfg.schemaDir),
)
runner := newRunner(cfg.promptDir, cfg.profileDir, cfg.schemaDir, llmClient)
res, runErr := runner.Run(context.Background(), req)
if runErr != nil {
@@ -170,14 +169,7 @@ func renderCommand(args []string, stdout, stderr io.Writer) int {
return ExitRuntimeError
}
runner := usecase.NewRunner(
promptdef.NewFilesystemRepository(cfg.promptDir),
profile.NewFilesystemRepository(cfg.profileDir),
artifactadapter.NewCompositeReader(),
prompt.NewGoRenderer(),
nil,
validate.NewStandardValidator(cfg.schemaDir),
)
runner := newRunner(cfg.promptDir, cfg.profileDir, cfg.schemaDir, nil)
prepared, prepErr := runner.Prepare(context.Background(), req)
if prepErr != nil {
@@ -205,22 +197,13 @@ func serveCommand(args []string, stderr io.Writer) int {
return ExitRuntimeError
}
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
Timeout: defaults.LLMRequestTimeoutDefault,
})
llmClient, err := newOpenAIClient()
if err != nil {
fmt.Fprintf(stderr, "llm client error: %v\n", err)
return ExitRuntimeError
}
runner := usecase.NewRunner(
promptdef.NewFilesystemRepository(cfg.promptDir),
profile.NewFilesystemRepository(cfg.profileDir),
artifactadapter.NewCompositeReader(),
prompt.NewGoRenderer(),
llmClient,
validate.NewStandardValidator(cfg.schemaDir),
)
runner := newRunner(cfg.promptDir, cfg.profileDir, cfg.schemaDir, llmClient)
h := httpadapter.NewHandler(runner)
srv := &http.Server{
@@ -307,7 +290,7 @@ func parseServeArgs(args []string) (*serveConfig, error) {
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
}
settings, err := resolveAppSettings(fs, cfg.configPath, appconfig.CLIOverrides{
settings, err := resolveCommonSettings(fs, cfg.configPath, appconfig.CLIOverrides{
PromptDir: cfg.promptDirIfSet(fs),
ProfileDir: cfg.profileDirIfSet(fs),
SchemaDir: cfg.schemaDirIfSet(fs),
@@ -317,16 +300,13 @@ func parseServeArgs(args []string) (*serveConfig, error) {
return nil, err
}
cfg.promptDir = settings.PromptDir
cfg.profileDir = settings.ProfileDir
cfg.schemaDir = settings.SchemaDir
cfg.addr = settings.ServerAddr
cfg.promptDir = settings.promptDir
cfg.profileDir = settings.profileDir
cfg.schemaDir = settings.schemaDir
cfg.addr = settings.serverAddr
if strings.TrimSpace(cfg.promptDir) == "" {
return nil, errors.New(errPromptDirRequired)
}
if strings.TrimSpace(cfg.profileDir) == "" {
return nil, errors.New(errProfileDirRequired)
if err := validateRequiredLibraryDirs(cfg.promptDir, cfg.profileDir); err != nil {
return nil, err
}
cfg.promptDir = filepath.Clean(cfg.promptDir)
@@ -359,7 +339,7 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
return fmt.Errorf("unexpected positional args: %v", fs.Args())
}
settings, err := resolveAppSettings(fs, cfg.configPath, appconfig.CLIOverrides{
settings, err := resolveCommonSettings(fs, cfg.configPath, appconfig.CLIOverrides{
PromptDir: cfg.promptDirIfSet(fs),
ProfileDir: cfg.profileDirIfSet(fs),
SchemaDir: cfg.schemaDirIfSet(fs),
@@ -368,16 +348,13 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
return err
}
cfg.promptDir = settings.PromptDir
cfg.profileDir = settings.ProfileDir
cfg.schemaDir = settings.SchemaDir
cfg.defaultRenderFormat = settings.DefaultRenderFormat
cfg.promptDir = settings.promptDir
cfg.profileDir = settings.profileDir
cfg.schemaDir = settings.schemaDir
cfg.defaultRenderFormat = settings.defaultRenderFormat
if strings.TrimSpace(cfg.promptDir) == "" {
return errors.New(errPromptDirRequired)
}
if strings.TrimSpace(cfg.profileDir) == "" {
return errors.New(errProfileDirRequired)
if err := validateRequiredLibraryDirs(cfg.promptDir, cfg.profileDir); err != nil {
return err
}
if strings.TrimSpace(cfg.promptID) == "" {
return errors.New("--prompt is required")
@@ -476,6 +453,47 @@ func resolveAppSettings(fs *flag.FlagSet, configPath string, overrides appconfig
return merged, nil
}
func resolveCommonSettings(fs *flag.FlagSet, configPath string, overrides appconfig.CLIOverrides) (commonCommandSettings, error) {
settings, err := resolveAppSettings(fs, configPath, overrides)
if err != nil {
return commonCommandSettings{}, err
}
return commonCommandSettings{
promptDir: settings.PromptDir,
profileDir: settings.ProfileDir,
schemaDir: settings.SchemaDir,
serverAddr: settings.ServerAddr,
defaultRenderFormat: settings.DefaultRenderFormat,
}, nil
}
func validateRequiredLibraryDirs(promptDir, profileDir string) error {
if strings.TrimSpace(promptDir) == "" {
return errors.New(errPromptDirRequired)
}
if strings.TrimSpace(profileDir) == "" {
return errors.New(errProfileDirRequired)
}
return nil
}
func newRunner(promptDir, profileDir, schemaDir string, llmClient llm.Client) *usecase.Runner {
return usecase.NewRunner(
promptdef.NewFilesystemRepository(promptDir),
profile.NewFilesystemRepository(profileDir),
artifactadapter.NewCompositeReader(),
prompt.NewGoRenderer(),
llmClient,
validate.NewStandardValidator(schemaDir),
)
}
func newOpenAIClient() (*llm.OpenAICompatibleClient, error) {
return llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
Timeout: defaults.LLMRequestTimeoutDefault,
})
}
func buildRunRequestFromConfig(cfg *runConfig) (domain.RunRequest, error) {
inputMappings, err := parseMappings(cfg.inputRaw, false)
if err != nil {
@@ -495,18 +513,25 @@ func buildRunRequestFromConfig(cfg *runConfig) (domain.RunRequest, error) {
inputs[name] = domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: path}
}
var modelOverride *domain.ExecutionTarget
var modelOverride *domain.ExecutionTargetOverride
if cfg.llmBaseURLSet || cfg.modelSet || cfg.temperatureSet || cfg.maxTokensSet || cfg.topPSet || cfg.apiKeyEnvSet || cfg.timeoutSet {
modelOverride = &domain.ExecutionTarget{
Endpoint: cfg.llmBaseURL,
Model: cfg.model,
Temperature: cfg.temperature,
MaxTokens: cfg.maxTokens,
TopP: cfg.topP,
APIKeyEnv: cfg.apiKeyEnv,
modelOverride = &domain.ExecutionTargetOverride{
Endpoint: cfg.llmBaseURL,
Model: cfg.model,
APIKeyEnv: cfg.apiKeyEnv,
}
if cfg.temperatureSet {
modelOverride.Temperature = &cfg.temperature
}
if cfg.maxTokensSet {
modelOverride.MaxTokens = &cfg.maxTokens
}
if cfg.topPSet {
modelOverride.TopP = &cfg.topP
}
if cfg.timeoutSet {
modelOverride.TimeoutSeconds = int(cfg.timeout.Seconds())
timeoutSeconds := int(cfg.timeout.Seconds())
modelOverride.TimeoutSeconds = &timeoutSeconds
}
}
@@ -588,7 +613,7 @@ func printSummary(stderr io.Writer, res *domain.RunResult) {
if res == nil {
return
}
fmt.Fprintf(stderr, "prompt=%s@%s selected_profile=%s model=%s validation=%s mode=%s validation_errors=%d prompt_hash=%s inputs=%d usage=%d/%d/%d\n",
fmt.Fprintf(stderr, "prompt=%s@%s selected_profile=%s model=%s validation=%s mode=%s validation_errors=%d prompt_hash=%s inputs=%d usage=%d/%d/%d",
res.PromptID,
res.PromptVersion,
res.SelectedProfileID,
@@ -602,6 +627,10 @@ func printSummary(stderr io.Writer, res *domain.RunResult) {
res.Usage.CompletionTokens,
res.Usage.TotalTokens,
)
if res.Usage.CachedTokens != 0 || res.Usage.CacheWriteTokens != 0 {
fmt.Fprintf(stderr, " cached_tokens=%d cache_write_tokens=%d", res.Usage.CachedTokens, res.Usage.CacheWriteTokens)
}
fmt.Fprintln(stderr)
}
func printUsage(w io.Writer) {

View File

@@ -11,6 +11,7 @@ import (
"net/http/httptest"
"os"
"path/filepath"
"reflect"
"strings"
"sync/atomic"
"testing"
@@ -410,6 +411,28 @@ defaults:
}
}
func TestParseRenderArgsExplicitFormatOverridesConfigDefaultFormat(t *testing.T) {
configPath := writeAppConfigFile(t, `
prompt_dir: ./from-config/prompts
profile_dir: ./from-config/profiles
defaults:
render_format: json
`)
cfg, err := parseRenderArgs([]string{
"--config", configPath,
"--prompt", "p",
"--input", "a=b",
"--format", "text",
})
if err != nil {
t.Fatalf("expected valid args, got %v", err)
}
if cfg.outputFormat != renderformat.PreparedRunFormatText {
t.Fatalf("expected explicit --format text to override config default, got %q", cfg.outputFormat)
}
}
func TestParseServeArgsWithExplicitConfigLoadsSettingsAndCLIAddrOverrides(t *testing.T) {
configPath := writeAppConfigFile(t, `
prompt_dir: ./from-config/prompts
@@ -471,6 +494,59 @@ server:
}
}
func TestRunAndRenderBuildEquivalentRuntimeOverrideRequestsForSharedFlags(t *testing.T) {
runCfg, err := parseRunArgs([]string{
"--prompt-dir", "./prompts",
"--profile-dir", "./profiles",
"--prompt", "prompt-1",
"--profile", "profile-1",
"--input", "transcript=./transcript.md",
"--var", "session_date=2026-05-01",
"--llm-base-url", "http://localhost:8000/v1",
"--model", "model-x",
"--temperature", "0.8",
"--max-tokens", "123",
"--top-p", "0.6",
"--timeout", "90s",
"--api-key-env", "SCRIPTORIUM_API_KEY",
})
if err != nil {
t.Fatalf("expected valid run args, got %v", err)
}
renderCfg, err := parseRenderArgs([]string{
"--prompt-dir", "./prompts",
"--profile-dir", "./profiles",
"--prompt", "prompt-1",
"--profile", "profile-1",
"--input", "transcript=./transcript.md",
"--var", "session_date=2026-05-01",
"--llm-base-url", "http://localhost:8000/v1",
"--model", "model-x",
"--temperature", "0.8",
"--max-tokens", "123",
"--top-p", "0.6",
"--timeout", "90s",
"--api-key-env", "SCRIPTORIUM_API_KEY",
})
if err != nil {
t.Fatalf("expected valid render args, got %v", err)
}
runReq, err := buildRunRequestFromConfig(runCfg)
if err != nil {
t.Fatalf("expected run request build success, got %v", err)
}
renderReq, err := buildRunRequestFromConfig(&renderCfg.runConfig)
if err != nil {
t.Fatalf("expected render request build success, got %v", err)
}
if !reflect.DeepEqual(runReq, renderReq) {
t.Fatalf("expected run/render shared flag requests to match.\nrun=%#v\nrender=%#v", runReq, renderReq)
}
}
func TestParseRunArgsFailsClearlyWhenNoEffectivePromptDir(t *testing.T) {
configPath := writeAppConfigFile(t, `
profile_dir: ./profiles
@@ -586,42 +662,29 @@ func TestRunCommandVarsOptional(t *testing.T) {
}
func TestRunCommandSucceedsWithPromptAndProfileDirsFromConfig(t *testing.T) {
tmp := t.TempDir()
promptDir := filepath.Join(tmp, "prompts")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
t.Fatal(err)
}
lib := newCLITestLibrary(t)
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
ts := newTestLLMServer("from-config-dirs", nil)
defer ts.Close()
writePromptFile(t, promptDir, "prompt.default", "local-default")
writeProfileFile(t, profileDir, "local-default", ts.URL+"/v1", "profile-model")
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
writeProfileFile(t, lib.profileDir, "local-default", ts.URL+"/v1", "profile-model")
configPath := writeAppConfigFile(t, fmt.Sprintf(`
prompt_dir: %s
profile_dir: %s
`, promptDir, profileDir))
`, lib.promptDir, lib.profileDir))
var stdout bytes.Buffer
var stderr bytes.Buffer
code := runCommand([]string{
code, stdout, stderr := runCLICommand(t, runCommand, []string{
"--config", configPath,
"--prompt", "prompt.default",
"--input", "transcript=" + inputPath,
}, &stdout, &stderr)
})
if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
}
if stdout.String() != "from-config-dirs" {
t.Fatalf("unexpected stdout output: %q", stdout.String())
if stdout != "from-config-dirs" {
t.Fatalf("unexpected stdout output: %q", stdout)
}
}
@@ -630,28 +693,15 @@ func TestRenderCommandDefaultFormatTextIncludesPreparedDetailsAndNoSecrets(t *te
const secret = "super-secret-render-key"
t.Setenv(envName, secret)
tmp := t.TempDir()
promptDir := filepath.Join(tmp, "prompts")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
t.Fatal(err)
}
lib := newCLITestLibrary(t)
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
writePromptFileWithTemplate(t, promptDir, "prompt.render", "local-default", "Date {{.session_date}} - Summarize: {{input \"transcript\"}}")
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
writePromptFileWithTemplate(t, lib.promptDir, "prompt.render", "local-default", "Date {{.session_date}} - Summarize: {{input \"transcript\"}}")
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
var stdout bytes.Buffer
var stderr bytes.Buffer
code := renderCommand([]string{
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
"--prompt-dir", lib.promptDir,
"--profile-dir", lib.profileDir,
"--prompt", "prompt.render",
"--profile", "local-default",
"--input", "transcript=" + inputPath,
@@ -663,15 +713,15 @@ func TestRenderCommandDefaultFormatTextIncludesPreparedDetailsAndNoSecrets(t *te
"--top-p", "0.2",
"--timeout", "20s",
"--api-key-env", envName,
}, &stdout, &stderr)
})
if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
}
if stderr.Len() != 0 {
t.Fatalf("expected empty stderr on success, got %q", stderr.String())
if stderr != "" {
t.Fatalf("expected empty stderr on success, got %q", stderr)
}
out := stdout.String()
out := stdout
for _, want := range []string{
"prompt: prompt.render",
"selected_profile_id: local-default",
@@ -697,112 +747,102 @@ func TestRenderCommandDefaultFormatTextIncludesPreparedDetailsAndNoSecrets(t *te
}
}
func TestRenderCommandSucceedsWithPromptAndProfileDirsFromConfig(t *testing.T) {
tmp := t.TempDir()
promptDir := filepath.Join(tmp, "prompts")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
t.Fatal(err)
func TestRenderCommandExplicitZeroTemperatureReachesEffectiveSettings(t *testing.T) {
lib := newCLITestLibrary(t)
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
profile := `id: local-default
endpoint: http://127.0.0.1:1/v1
model: profile-model
temperature: 0.7
`
if err := os.WriteFile(filepath.Join(lib.profileDir, "local-default.yaml"), []byte(profile), 0o644); err != nil {
t.Fatalf("failed to write profile fixture: %v", err)
}
writePromptFile(t, promptDir, "prompt.render", "local-default")
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
"--prompt-dir", lib.promptDir,
"--profile-dir", lib.profileDir,
"--prompt", "prompt.render",
"--input", "transcript=" + inputPath,
"--temperature", "0",
})
if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
}
if !strings.Contains(stdout, "\n temperature: 0\n") {
t.Fatalf("expected explicit zero temperature in effective settings, got:\n%s", stdout)
}
}
func TestRenderCommandSucceedsWithPromptAndProfileDirsFromConfig(t *testing.T) {
lib := newCLITestLibrary(t)
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
configPath := writeAppConfigFile(t, fmt.Sprintf(`
prompt_dir: %s
profile_dir: %s
`, promptDir, profileDir))
`, lib.promptDir, lib.profileDir))
var stdout bytes.Buffer
var stderr bytes.Buffer
code := renderCommand([]string{
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
"--config", configPath,
"--prompt", "prompt.render",
"--input", "transcript=" + inputPath,
}, &stdout, &stderr)
})
if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
}
if !strings.Contains(stdout.String(), "prompt: prompt.render") {
t.Fatalf("expected rendered output, got %q", stdout.String())
if !strings.Contains(stdout, "prompt: prompt.render") {
t.Fatalf("expected rendered output, got %q", stdout)
}
}
func TestRenderCommandExplicitTextFormatWorks(t *testing.T) {
tmp := t.TempDir()
promptDir := filepath.Join(tmp, "prompts")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
t.Fatal(err)
}
lib := newCLITestLibrary(t)
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
writePromptFile(t, promptDir, "prompt.render", "local-default")
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
var stdout bytes.Buffer
var stderr bytes.Buffer
code := renderCommand([]string{
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
"--prompt-dir", lib.promptDir,
"--profile-dir", lib.profileDir,
"--prompt", "prompt.render",
"--input", "transcript=" + inputPath,
"--format", "text",
}, &stdout, &stderr)
})
if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
}
if !strings.Contains(stdout.String(), "prompt: prompt.render") {
t.Fatalf("expected text output for explicit --format text, got %q", stdout.String())
if !strings.Contains(stdout, "prompt: prompt.render") {
t.Fatalf("expected text output for explicit --format text, got %q", stdout)
}
}
func TestRenderCommandExplicitJSONFormatOutputsValidJSON(t *testing.T) {
tmp := t.TempDir()
promptDir := filepath.Join(tmp, "prompts")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
t.Fatal(err)
}
lib := newCLITestLibrary(t)
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
writePromptFile(t, promptDir, "prompt.render", "local-default")
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
var stdout bytes.Buffer
var stderr bytes.Buffer
code := renderCommand([]string{
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
"--prompt-dir", lib.promptDir,
"--profile-dir", lib.profileDir,
"--prompt", "prompt.render",
"--input", "transcript=" + inputPath,
"--format", "json",
}, &stdout, &stderr)
})
if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
}
var payload map[string]any
if err := json.Unmarshal(stdout.Bytes(), &payload); err != nil {
t.Fatalf("expected valid json output, got %v\nbody=%s", err, stdout.String())
if err := json.Unmarshal([]byte(stdout), &payload); err != nil {
t.Fatalf("expected valid json output, got %v\nbody=%s", err, stdout)
}
if payload["prompt_id"] != "prompt.render" {
t.Fatalf("expected prompt_id, got %#v", payload["prompt_id"])
@@ -834,38 +874,25 @@ func TestRenderCommandUnknownFormatFailsClearly(t *testing.T) {
}
func TestRenderCommandOutWritesToFile(t *testing.T) {
tmp := t.TempDir()
promptDir := filepath.Join(tmp, "prompts")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
t.Fatal(err)
}
outPath := filepath.Join(tmp, "render.txt")
lib := newCLITestLibrary(t)
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
outPath := filepath.Join(lib.rootDir, "render.txt")
writePromptFile(t, promptDir, "prompt.render", "local-default")
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
var stdout bytes.Buffer
var stderr bytes.Buffer
code := renderCommand([]string{
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
"--prompt-dir", lib.promptDir,
"--profile-dir", lib.profileDir,
"--prompt", "prompt.render",
"--input", "transcript=" + inputPath,
"--out", outPath,
}, &stdout, &stderr)
})
if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
}
if stdout.Len() != 0 {
t.Fatalf("expected empty stdout when --out is set, got %q", stdout.String())
if stdout != "" {
t.Fatalf("expected empty stdout when --out is set, got %q", stdout)
}
out, err := os.ReadFile(outPath)
if err != nil {
@@ -877,36 +904,23 @@ func TestRenderCommandOutWritesToFile(t *testing.T) {
}
func TestRenderCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
tmp := t.TempDir()
promptDir := filepath.Join(tmp, "prompts")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
t.Fatal(err)
}
lib := newCLITestLibrary(t)
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
writePromptFile(t, promptDir, "prompt.default", "local-default")
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model")
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model")
var stdout bytes.Buffer
var stderr bytes.Buffer
code := renderCommand([]string{
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
"--prompt-dir", lib.promptDir,
"--profile-dir", lib.profileDir,
"--prompt", "prompt.default",
"--input", "transcript=" + inputPath,
}, &stdout, &stderr)
})
if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
}
out := stdout.String()
out := stdout
if !strings.Contains(out, "selected_profile_id: local-default") {
t.Fatalf("expected prompt default profile in output, got %q", out)
}
@@ -916,38 +930,25 @@ func TestRenderCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
}
func TestRenderCommandExplicitProfileOverridesPromptDefault(t *testing.T) {
tmp := t.TempDir()
promptDir := filepath.Join(tmp, "prompts")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
t.Fatal(err)
}
lib := newCLITestLibrary(t)
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
writePromptFile(t, promptDir, "prompt.default", "local-default")
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model")
writeProfileFile(t, profileDir, "quality", "http://127.0.0.1:1/v1", "quality-model")
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model")
writeProfileFile(t, lib.profileDir, "quality", "http://127.0.0.1:1/v1", "quality-model")
var stdout bytes.Buffer
var stderr bytes.Buffer
code := renderCommand([]string{
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
"--prompt-dir", lib.promptDir,
"--profile-dir", lib.profileDir,
"--prompt", "prompt.default",
"--profile", "quality",
"--input", "transcript=" + inputPath,
}, &stdout, &stderr)
})
if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
}
out := stdout.String()
out := stdout
if !strings.Contains(out, "selected_profile_id: quality") {
t.Fatalf("expected explicit profile in output, got %q", out)
}
@@ -957,104 +958,67 @@ func TestRenderCommandExplicitProfileOverridesPromptDefault(t *testing.T) {
}
func TestRunCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
tmp := t.TempDir()
promptDir := filepath.Join(tmp, "prompts")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
t.Fatal(err)
}
lib := newCLITestLibrary(t)
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
ts := newTestLLMServer("default-output", nil)
defer ts.Close()
writePromptFile(t, promptDir, "prompt.default", "local-default")
writeProfileFile(t, profileDir, "local-default", ts.URL+"/v1", "profile-model")
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
writeProfileFile(t, lib.profileDir, "local-default", ts.URL+"/v1", "profile-model")
var stdout bytes.Buffer
var stderr bytes.Buffer
code := runCommand([]string{
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
code, stdout, stderr := runCLICommand(t, runCommand, []string{
"--prompt-dir", lib.promptDir,
"--profile-dir", lib.profileDir,
"--prompt", "prompt.default",
"--input", "transcript=" + inputPath,
}, &stdout, &stderr)
})
if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
}
if stdout.String() != "default-output" {
t.Fatalf("unexpected stdout output: %q", stdout.String())
if stdout != "default-output" {
t.Fatalf("unexpected stdout output: %q", stdout)
}
if !strings.Contains(stderr.String(), "selected_profile=local-default") {
t.Fatalf("expected selected profile in summary, got %q", stderr.String())
if !strings.Contains(stderr, "selected_profile=local-default") {
t.Fatalf("expected selected profile in summary, got %q", stderr)
}
}
func TestRunCommandExplicitProfileOverridesPromptDefault(t *testing.T) {
tmp := t.TempDir()
promptDir := filepath.Join(tmp, "prompts")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
t.Fatal(err)
}
lib := newCLITestLibrary(t)
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
defaultServer := newTestLLMServer("from-default", nil)
defer defaultServer.Close()
overrideServer := newTestLLMServer("from-override", nil)
defer overrideServer.Close()
writePromptFile(t, promptDir, "prompt.default", "local-default")
writeProfileFile(t, profileDir, "local-default", defaultServer.URL+"/v1", "default-model")
writeProfileFile(t, profileDir, "quality", overrideServer.URL+"/v1", "quality-model")
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
writeProfileFile(t, lib.profileDir, "local-default", defaultServer.URL+"/v1", "default-model")
writeProfileFile(t, lib.profileDir, "quality", overrideServer.URL+"/v1", "quality-model")
var stdout bytes.Buffer
var stderr bytes.Buffer
code := runCommand([]string{
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
code, stdout, stderr := runCLICommand(t, runCommand, []string{
"--prompt-dir", lib.promptDir,
"--profile-dir", lib.profileDir,
"--prompt", "prompt.default",
"--profile", "quality",
"--input", "transcript=" + inputPath,
}, &stdout, &stderr)
})
if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
}
if stdout.String() != "from-override" {
t.Fatalf("expected explicit profile output, got %q", stdout.String())
if stdout != "from-override" {
t.Fatalf("expected explicit profile output, got %q", stdout)
}
if !strings.Contains(stderr.String(), "selected_profile=quality") {
t.Fatalf("expected selected profile quality, got %q", stderr.String())
if !strings.Contains(stderr, "selected_profile=quality") {
t.Fatalf("expected selected profile quality, got %q", stderr)
}
}
func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
tmp := t.TempDir()
promptDir := filepath.Join(tmp, "prompts")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
t.Fatal(err)
}
lib := newCLITestLibrary(t)
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
var baseHits int32
baseServer := newTestLLMServer("base", &baseHits)
@@ -1071,14 +1035,12 @@ func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
}))
defer overrideServer.Close()
writePromptFile(t, promptDir, "prompt.default", "local-default")
writeProfileFile(t, profileDir, "local-default", baseServer.URL+"/v1", "profile-model")
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
writeProfileFile(t, lib.profileDir, "local-default", baseServer.URL+"/v1", "profile-model")
var stdout bytes.Buffer
var stderr bytes.Buffer
code := runCommand([]string{
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
code, stdout, stderr := runCLICommand(t, runCommand, []string{
"--prompt-dir", lib.promptDir,
"--profile-dir", lib.profileDir,
"--prompt", "prompt.default",
"--input", "transcript=" + inputPath,
"--llm-base-url", overrideServer.URL + "/v1",
@@ -1087,9 +1049,9 @@ func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
"--max-tokens", "55",
"--top-p", "0.2",
"--timeout", "20s",
}, &stdout, &stderr)
})
if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
}
if atomic.LoadInt32(&baseHits) != 0 {
t.Fatalf("expected base profile endpoint not to be hit, got %d", baseHits)
@@ -1097,8 +1059,8 @@ func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
if atomic.LoadInt32(&overrideHits) != 1 {
t.Fatalf("expected override endpoint to be hit once, got %d", overrideHits)
}
if stdout.String() != "override" {
t.Fatalf("unexpected stdout output: %q", stdout.String())
if stdout != "override" {
t.Fatalf("unexpected stdout output: %q", stdout)
}
if !strings.Contains(observedBody, `"model":"override-model"`) {
t.Fatalf("expected override model in request body, got %s", observedBody)
@@ -1131,6 +1093,81 @@ func TestWriteOutputAndSummaryUseSeparateWriters(t *testing.T) {
if !strings.Contains(stderr.String(), "prompt=p@1") {
t.Fatalf("expected summary on stderr, got %q", stderr.String())
}
if strings.Contains(stderr.String(), "cached_tokens=") || strings.Contains(stderr.String(), "cache_write_tokens=") {
t.Fatalf("expected zero cache usage to be omitted from summary, got %q", stderr.String())
}
}
func TestPrintSummaryIncludesCacheUsageWhenPresent(t *testing.T) {
var stderr bytes.Buffer
printSummary(&stderr, &domain.RunResult{
PromptID: "p",
PromptVersion: "1",
SelectedProfileID: "exec",
ModelName: "m",
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic},
RenderedPromptHash: "h",
InputHashes: map[string]string{"in": "x"},
Usage: domain.TokenUsage{
PromptTokens: 10,
CompletionTokens: 5,
TotalTokens: 15,
CachedTokens: 0,
CacheWriteTokens: 3,
},
})
summary := stderr.String()
if !strings.Contains(summary, "usage=10/5/15") {
t.Fatalf("expected base usage summary, got %q", summary)
}
if !strings.Contains(summary, "cached_tokens=0 cache_write_tokens=3") {
t.Fatalf("expected cache usage in summary, got %q", summary)
}
}
type cliTestLibrary struct {
rootDir string
promptDir string
profileDir string
}
func newCLITestLibrary(t *testing.T) *cliTestLibrary {
t.Helper()
root := t.TempDir()
lib := &cliTestLibrary{
rootDir: root,
promptDir: filepath.Join(root, "prompts"),
profileDir: filepath.Join(root, "profiles"),
}
if err := os.MkdirAll(lib.promptDir, 0o755); err != nil {
t.Fatalf("failed to create prompt fixture directory: %v", err)
}
if err := os.MkdirAll(lib.profileDir, 0o755); err != nil {
t.Fatalf("failed to create profile fixture directory: %v", err)
}
return lib
}
func (l *cliTestLibrary) writeInputFile(t *testing.T, name, body string) string {
t.Helper()
path := filepath.Join(l.rootDir, name)
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatalf("failed to create input fixture directory: %v", err)
}
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
t.Fatalf("failed to write input fixture: %v", err)
}
return path
}
func runCLICommand(t *testing.T, command func([]string, io.Writer, io.Writer) int, args []string) (int, string, string) {
t.Helper()
var stdout bytes.Buffer
var stderr bytes.Buffer
code := command(args, &stdout, &stderr)
return code, stdout.String(), stderr.String()
}
func writePromptFile(t *testing.T, dir, id, defaultProfile string) {

View File

@@ -21,15 +21,16 @@ type inputRefDTO struct {
}
type modelOverrideRequestDTO struct {
Endpoint string `json:"endpoint,omitempty"`
Model string `json:"model,omitempty"`
Temperature float64 `json:"temperature,omitempty"`
MaxTokens int `json:"max_tokens,omitempty"`
TopP float64 `json:"top_p,omitempty"`
TimeoutSeconds int `json:"timeout_seconds,omitempty"`
ReasoningEffort string `json:"reasoning_effort,omitempty"`
APIKeyEnv string `json:"api_key_env,omitempty"`
ExtraParams map[string]string `json:"extra_params,omitempty"`
Endpoint string `json:"endpoint,omitempty"`
Model string `json:"model,omitempty"`
Temperature *float64 `json:"temperature,omitempty"`
MaxTokens *int `json:"max_tokens,omitempty"`
TopP *float64 `json:"top_p,omitempty"`
TimeoutSeconds *int `json:"timeout_seconds,omitempty"`
ServiceTier string `json:"service_tier,omitempty"`
ReasoningEffort string `json:"reasoning_effort,omitempty"`
APIKeyEnv string `json:"api_key_env,omitempty"`
ExtraParams map[string]any `json:"extra_params,omitempty"`
}
type runResponseDTO struct {
@@ -69,21 +70,24 @@ type metadataDTO struct {
}
type modelParamsDTO struct {
Endpoint string `json:"endpoint"`
Model string `json:"model"`
Temperature float64 `json:"temperature"`
MaxTokens int `json:"max_tokens"`
TopP float64 `json:"top_p"`
TimeoutSeconds int `json:"timeout_seconds"`
ReasoningEffort string `json:"reasoning_effort,omitempty"`
APIKeyEnv string `json:"api_key_env,omitempty"`
ExtraParams map[string]string `json:"extra_params,omitempty"`
Endpoint string `json:"endpoint"`
Model string `json:"model"`
Temperature float64 `json:"temperature"`
MaxTokens int `json:"max_tokens"`
TopP float64 `json:"top_p"`
TimeoutSeconds int `json:"timeout_seconds"`
ServiceTier string `json:"service_tier,omitempty"`
ReasoningEffort string `json:"reasoning_effort,omitempty"`
APIKeyEnv string `json:"api_key_env,omitempty"`
ExtraParams map[string]any `json:"extra_params,omitempty"`
}
type tokenUsageDTO struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"`
CachedTokens int `json:"cached_tokens"`
CacheWriteTokens int `json:"cache_write_tokens"`
}
type validationDTO struct {

View File

@@ -61,19 +61,9 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
}
}
var model *domain.ExecutionTarget
var model *domain.ExecutionTargetOverride
if req.Model != nil {
model = &domain.ExecutionTarget{
Endpoint: req.Model.Endpoint,
Model: req.Model.Model,
Temperature: req.Model.Temperature,
MaxTokens: req.Model.MaxTokens,
TopP: req.Model.TopP,
TimeoutSeconds: req.Model.TimeoutSeconds,
ReasoningEffort: req.Model.ReasoningEffort,
APIKeyEnv: req.Model.APIKeyEnv,
ExtraParams: req.Model.ExtraParams,
}
model = executionTargetOverrideFromModelOverrideDTO(req.Model)
}
res, err := h.runner.Run(r.Context(), domain.RunRequest{
@@ -109,22 +99,14 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
SelectedProfileID: res.SelectedProfileID,
ModelName: res.ModelName,
Endpoint: res.Endpoint,
ModelParams: modelParamsDTO{
Endpoint: res.EffectiveModelParams.Endpoint,
Model: res.EffectiveModelParams.Model,
Temperature: res.EffectiveModelParams.Temperature,
MaxTokens: res.EffectiveModelParams.MaxTokens,
TopP: res.EffectiveModelParams.TopP,
TimeoutSeconds: res.EffectiveModelParams.TimeoutSeconds,
ReasoningEffort: res.EffectiveModelParams.ReasoningEffort,
APIKeyEnv: res.EffectiveModelParams.APIKeyEnv,
ExtraParams: res.EffectiveModelParams.ExtraParams,
},
InputHashes: res.InputHashes,
ModelParams: modelParamsDTOFromExecutionTarget(res.EffectiveModelParams),
InputHashes: res.InputHashes,
Usage: tokenUsageDTO{
PromptTokens: res.Usage.PromptTokens,
CompletionTokens: res.Usage.CompletionTokens,
TotalTokens: res.Usage.TotalTokens,
CachedTokens: res.Usage.CachedTokens,
CacheWriteTokens: res.Usage.CacheWriteTokens,
},
StartTime: res.StartTime,
EndTime: res.EndTime,
@@ -141,6 +123,39 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, resp)
}
func executionTargetOverrideFromModelOverrideDTO(dto *modelOverrideRequestDTO) *domain.ExecutionTargetOverride {
if dto == nil {
return nil
}
return &domain.ExecutionTargetOverride{
Endpoint: dto.Endpoint,
Model: dto.Model,
Temperature: dto.Temperature,
MaxTokens: dto.MaxTokens,
TopP: dto.TopP,
TimeoutSeconds: dto.TimeoutSeconds,
ServiceTier: dto.ServiceTier,
ReasoningEffort: dto.ReasoningEffort,
APIKeyEnv: dto.APIKeyEnv,
ExtraParams: dto.ExtraParams,
}
}
func modelParamsDTOFromExecutionTarget(target domain.ExecutionTarget) modelParamsDTO {
return modelParamsDTO{
Endpoint: target.Endpoint,
Model: target.Model,
Temperature: target.Temperature,
MaxTokens: target.MaxTokens,
TopP: target.TopP,
TimeoutSeconds: target.TimeoutSeconds,
ServiceTier: target.ServiceTier,
ReasoningEffort: target.ReasoningEffort,
APIKeyEnv: target.APIKeyEnv,
ExtraParams: target.ExtraParams,
}
}
func mapValidation(v domain.ValidationResult) validationDTO {
return validationDTO{
Status: string(v.Status),
@@ -162,9 +177,9 @@ func mapRunError(err error) (int, string, string) {
return http.StatusBadRequest, "prompt_load_failed", "failed to load prompt definition"
case errors.Is(err, profile.ErrInvalidYAML), errors.Is(err, profile.ErrInvalidProfile):
return http.StatusBadRequest, "profile_load_failed", "failed to load execution profile"
case errors.Is(err, usecase.ErrInvalidRequest) && strings.Contains(err.Error(), "profile id is required either in request or prompt default_profile"):
case errors.Is(err, usecase.ErrProfileRequired):
return http.StatusBadRequest, "profile_required", "profile_id is required when prompt default_profile is not set"
case errors.Is(err, usecase.ErrInvalidRequest) && strings.Contains(err.Error(), "api key environment variable"):
case errors.Is(err, usecase.ErrAPIKeyEnvMissing):
return http.StatusBadRequest, "api_key_env_missing", "api_key_env is set but the environment variable is missing"
case errors.Is(err, usecase.ErrInvalidRequest):
return http.StatusBadRequest, "invalid_request", "invalid run request"

View File

@@ -4,15 +4,16 @@ import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"reflect"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
"gitea.maximumdirect.net/eric/scriptorium/internal/llm"
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
"gitea.maximumdirect.net/eric/scriptorium/internal/usecase"
@@ -32,6 +33,34 @@ func (f *fakeRunner) Run(ctx context.Context, req domain.RunRequest) (*domain.Ru
return f.result, nil
}
type handlerPromptRepo struct {
def *domain.PromptDefinition
}
func (r handlerPromptRepo) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error) {
return r.def, nil
}
type handlerProfileRepo struct {
profile *domain.ExecutionProfile
}
func (r handlerProfileRepo) GetProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) {
return r.profile, nil
}
type handlerArtifactReader struct{}
func (handlerArtifactReader) Read(ctx context.Context, ref domain.ArtifactRef) (*domain.Artifact, error) {
return &domain.Artifact{Name: "input", Body: []byte("input"), Hash: "hash"}, nil
}
type handlerRenderer struct{}
func (handlerRenderer) Render(ctx context.Context, definition *domain.PromptDefinition, inputs map[string]*domain.Artifact, vars map[string]string) (*domain.RenderedPrompt, error) {
return &domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}}, nil
}
func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
start := time.Now().UTC()
end := start.Add(2 * time.Second)
@@ -62,14 +91,21 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
MaxTokens: 42,
TopP: 0.9,
TimeoutSeconds: 120,
ServiceTier: "priority",
APIKeyEnv: envName,
},
InputHashes: map[string]string{"transcript": "h1"},
Usage: domain.TokenUsage{PromptTokens: 1, CompletionTokens: 2, TotalTokens: 3},
StartTime: start,
EndTime: end,
Duration: 2 * time.Second,
RawOutput: "hello",
Usage: domain.TokenUsage{
PromptTokens: 1,
CompletionTokens: 2,
TotalTokens: 3,
CachedTokens: 4,
CacheWriteTokens: 5,
},
StartTime: start,
EndTime: end,
Duration: 2 * time.Second,
RawOutput: "hello",
}}
h := NewHandler(r)
@@ -81,7 +117,7 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
"transcript": {"type": "file", "uri": "./t.md"}
},
"vars": {"k": "v"},
"model": {"model": "gpt-x", "timeout_seconds": 120, "api_key_env": "SCRIPTORIUM_API_KEY"}
"model": {"model": "gpt-x", "timeout_seconds": 120, "service_tier": "flex", "api_key_env": "SCRIPTORIUM_API_KEY"}
}`)
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewReader(body))
w := httptest.NewRecorder()
@@ -110,10 +146,20 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
if metadata["model_name"] != "m1" || metadata["endpoint"] != "http://llm/v1" {
t.Fatalf("unexpected model metadata: name=%#v endpoint=%#v", metadata["model_name"], metadata["endpoint"])
}
usage := metadata["usage"].(map[string]any)
if usage["prompt_tokens"] != float64(1) || usage["completion_tokens"] != float64(2) || usage["total_tokens"] != float64(3) {
t.Fatalf("unexpected base usage metadata: %#v", usage)
}
if usage["cached_tokens"] != float64(4) || usage["cache_write_tokens"] != float64(5) {
t.Fatalf("unexpected cache usage metadata: %#v", usage)
}
modelParams := metadata["model_params"].(map[string]any)
if modelParams["api_key_env"] != envName {
t.Fatalf("expected model_params.api_key_env=%q, got %#v", envName, modelParams["api_key_env"])
}
if modelParams["service_tier"] != "priority" {
t.Fatalf("expected model_params.service_tier=priority, got %#v", modelParams["service_tier"])
}
if strings.Contains(w.Body.String(), secret) {
t.Fatalf("response leaked raw API key value: %s", w.Body.String())
}
@@ -130,9 +176,12 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
if r.last.Execution == nil || r.last.Execution.Model != "gpt-x" {
t.Fatalf("expected model override, got %#v", r.last.Execution)
}
if r.last.Execution.TimeoutSeconds != 120 {
if r.last.Execution.TimeoutSeconds == nil || *r.last.Execution.TimeoutSeconds != 120 {
t.Fatalf("expected timeout_seconds override 120, got %#v", r.last.Execution)
}
if r.last.Execution.ServiceTier != "flex" {
t.Fatalf("expected service_tier override flex, got %#v", r.last.Execution)
}
}
func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
@@ -164,6 +213,265 @@ func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
if metadata["selected_profile_id"] != "prompt-default" {
t.Fatalf("expected selected_profile_id from result, got %#v", metadata["selected_profile_id"])
}
usage := metadata["usage"].(map[string]any)
if usage["cached_tokens"] != float64(0) || usage["cache_write_tokens"] != float64(0) {
t.Fatalf("expected zero cache usage fields to be included, got %#v", usage)
}
}
func TestHandlerModelOverrideMapsAllSupportedExecutionFields(t *testing.T) {
r := &fakeRunner{result: &domain.RunResult{
Artifact: domain.Artifact{Body: []byte("ok")},
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
EffectiveModelParams: domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
}}
h := NewHandler(r)
reqBody := `{
"prompt_id": "prompt-1",
"inputs": {"transcript": {"type": "file", "uri": "./t.md"}},
"model": {
"endpoint": "http://override/v1",
"model": "override-model",
"temperature": 0.6,
"max_tokens": 250,
"top_p": 0.85,
"timeout_seconds": 33,
"service_tier": "flex",
"reasoning_effort": "medium",
"api_key_env": "SCRIPTORIUM_API_KEY",
"extra_params": {"provider_option":"on"}
}
}`
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(reqBody))
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
}
if r.last.Execution == nil {
t.Fatalf("expected execution override in run request")
}
got := r.last.Execution
if got.Endpoint != "http://override/v1" ||
got.Model != "override-model" ||
got.ServiceTier != "flex" ||
got.ReasoningEffort != "medium" ||
got.APIKeyEnv != "SCRIPTORIUM_API_KEY" {
t.Fatalf("unexpected mapped execution target: %+v", got)
}
if got.Temperature == nil || *got.Temperature != 0.6 {
t.Fatalf("unexpected mapped temperature: %#v", got.Temperature)
}
if got.MaxTokens == nil || *got.MaxTokens != 250 {
t.Fatalf("unexpected mapped max_tokens: %#v", got.MaxTokens)
}
if got.TopP == nil || *got.TopP != 0.85 {
t.Fatalf("unexpected mapped top_p: %#v", got.TopP)
}
if got.TimeoutSeconds == nil || *got.TimeoutSeconds != 33 {
t.Fatalf("unexpected mapped timeout_seconds: %#v", got.TimeoutSeconds)
}
if !reflect.DeepEqual(got.ExtraParams, map[string]any{"provider_option": "on"}) {
t.Fatalf("unexpected mapped extra_params: %#v", got.ExtraParams)
}
}
func TestHandlerModelOverrideAcceptsJSONCompatibleExtraParams(t *testing.T) {
r := &fakeRunner{result: &domain.RunResult{
Artifact: domain.Artifact{Body: []byte("ok")},
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
EffectiveModelParams: domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
}}
h := NewHandler(r)
reqBody := `{
"prompt_id": "prompt-1",
"inputs": {"transcript": {"type": "file", "uri": "./t.md"}},
"model": {
"extra_params": {
"string_value": "enabled",
"number_value": 42,
"boolean_value": true,
"object_value": {"nested": "value", "count": 2},
"array_value": ["first", 3, false]
}
}
}`
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(reqBody))
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
}
if r.last.Execution == nil {
t.Fatal("expected execution override in run request")
}
want := map[string]any{
"string_value": "enabled",
"number_value": float64(42),
"boolean_value": true,
"object_value": map[string]any{"nested": "value", "count": float64(2)},
"array_value": []any{"first", float64(3), false},
}
if !reflect.DeepEqual(r.last.Execution.ExtraParams, want) {
t.Fatalf("unexpected mapped extra_params:\ngot=%#v\nwant=%#v", r.last.Execution.ExtraParams, want)
}
}
func TestHandlerModelOverrideExplicitZeroTemperatureMapsAsPresent(t *testing.T) {
r := &fakeRunner{result: &domain.RunResult{
Artifact: domain.Artifact{Body: []byte("ok")},
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
EffectiveModelParams: domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1", Temperature: 0},
}}
h := NewHandler(r)
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
"prompt_id": "prompt-1",
"inputs": {"transcript": {"type": "file", "uri": "./t.md"}},
"model": {"temperature": 0}
}`))
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
}
if r.last.Execution == nil || r.last.Execution.Temperature == nil {
t.Fatalf("expected temperature override to be present, got %#v", r.last.Execution)
}
if *r.last.Execution.Temperature != 0 {
t.Fatalf("expected zero temperature override, got %v", *r.last.Execution.Temperature)
}
}
func TestHandlerModelOverrideOmittedTemperatureMapsAsAbsent(t *testing.T) {
r := &fakeRunner{result: &domain.RunResult{
Artifact: domain.Artifact{Body: []byte("ok")},
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
EffectiveModelParams: domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1", Temperature: 0.7},
}}
h := NewHandler(r)
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
"prompt_id": "prompt-1",
"inputs": {"transcript": {"type": "file", "uri": "./t.md"}},
"model": {"model": "override-model"}
}`))
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
}
if r.last.Execution == nil {
t.Fatal("expected model override")
}
if r.last.Execution.Temperature != nil {
t.Fatalf("expected omitted temperature to remain absent, got %#v", r.last.Execution.Temperature)
}
var resp map[string]any
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("invalid JSON response: %v", err)
}
metadata := resp["metadata"].(map[string]any)
params := metadata["model_params"].(map[string]any)
if params["temperature"] != 0.7 {
t.Fatalf("expected effective profile/default temperature in response, got %#v", params["temperature"])
}
}
func TestHandlerResponseMetadataModelParamsIncludesAllSupportedFields(t *testing.T) {
r := &fakeRunner{result: &domain.RunResult{
Artifact: domain.Artifact{
Name: "output",
ContentType: "text/plain",
Body: []byte("ok"),
Size: 2,
Hash: "abc",
},
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
EffectiveModelParams: domain.ExecutionTarget{
Endpoint: "http://llm/v1",
Model: "gpt-test",
Temperature: 0.4,
MaxTokens: 321,
TopP: 0.7,
TimeoutSeconds: 45,
ServiceTier: "priority",
ReasoningEffort: "high",
APIKeyEnv: "SCRIPTORIUM_API_KEY",
ExtraParams: map[string]any{
"provider_option": "on",
"number_value": 42,
"object_value": map[string]any{"nested": "value"},
},
},
}}
h := NewHandler(r)
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p","inputs":{"x":{"type":"file","uri":"a"}}}`))
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
}
var resp map[string]any
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("invalid JSON response: %v", err)
}
metadata := resp["metadata"].(map[string]any)
params := metadata["model_params"].(map[string]any)
if params["endpoint"] != "http://llm/v1" {
t.Fatalf("unexpected endpoint: %#v", params["endpoint"])
}
if params["model"] != "gpt-test" {
t.Fatalf("unexpected model: %#v", params["model"])
}
if params["temperature"] != 0.4 {
t.Fatalf("unexpected temperature: %#v", params["temperature"])
}
if params["max_tokens"] != float64(321) {
t.Fatalf("unexpected max_tokens: %#v", params["max_tokens"])
}
if params["top_p"] != 0.7 {
t.Fatalf("unexpected top_p: %#v", params["top_p"])
}
if params["timeout_seconds"] != float64(45) {
t.Fatalf("unexpected timeout_seconds: %#v", params["timeout_seconds"])
}
if params["service_tier"] != "priority" {
t.Fatalf("unexpected service_tier: %#v", params["service_tier"])
}
if params["reasoning_effort"] != "high" {
t.Fatalf("unexpected reasoning_effort: %#v", params["reasoning_effort"])
}
if params["api_key_env"] != "SCRIPTORIUM_API_KEY" {
t.Fatalf("unexpected api_key_env: %#v", params["api_key_env"])
}
extraParams, ok := params["extra_params"].(map[string]any)
if !ok {
t.Fatalf("expected extra_params object, got %#v", params["extra_params"])
}
if extraParams["provider_option"] != "on" {
t.Fatalf("unexpected extra_params.provider_option: %#v", extraParams["provider_option"])
}
if extraParams["number_value"] != float64(42) {
t.Fatalf("unexpected extra_params.number_value: %#v", extraParams["number_value"])
}
objectValue, ok := extraParams["object_value"].(map[string]any)
if !ok || objectValue["nested"] != "value" {
t.Fatalf("unexpected extra_params.object_value: %#v", extraParams["object_value"])
}
}
func TestHandlerInvalidJSON(t *testing.T) {
@@ -198,6 +506,54 @@ func TestHandlerMissingPromptID(t *testing.T) {
}
}
func TestHandlerReservedExtraParamsThroughRunnerMapsToInvalidRequest(t *testing.T) {
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{})
if err != nil {
t.Fatal(err)
}
runner := usecase.NewRunner(
handlerPromptRepo{def: &domain.PromptDefinition{
ID: "p",
Version: "1",
DefaultProfile: "exec",
Templates: []domain.PromptMessageTemplate{{Role: "user", Content: "hi"}},
OutputFormat: domain.FormatText,
Validation: domain.OutputContract{Format: domain.FormatText, ValidationMode: domain.ValidationNone},
}},
handlerProfileRepo{profile: &domain.ExecutionProfile{
ID: "exec",
Endpoint: "http://example.invalid/v1",
Model: "model",
}},
handlerArtifactReader{},
handlerRenderer{},
llmClient,
nil,
)
h := NewHandler(runner)
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{
"prompt_id":"p",
"inputs":{"x":{"type":"file","uri":"a"}},
"model":{"extra_params":{"model":"collision"}}
}`))
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
if w.Code != http.StatusBadRequest {
t.Fatalf("expected 400, got %d body=%s", w.Code, w.Body.String())
}
var resp map[string]any
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("invalid JSON response: %v", err)
}
errBody := resp["error"].(map[string]any)
if errBody["code"] != "invalid_request" {
t.Fatalf("expected invalid_request code, got %#v", errBody["code"])
}
}
func TestHandlerUsecaseErrorMapping(t *testing.T) {
tests := []struct {
name string
@@ -209,10 +565,10 @@ func TestHandlerUsecaseErrorMapping(t *testing.T) {
}{
{name: "prompt not found", err: wrap(usecase.ErrProfileLoad, promptdef.ErrPromptDefinitionNotFound), status: http.StatusNotFound, code: "prompt_not_found", message: "prompt definition not found"},
{name: "prompt load invalid", err: wrap(usecase.ErrProfileLoad, promptdef.ErrInvalidPromptDefinition), status: http.StatusBadRequest, code: "prompt_load_failed", message: "failed to load prompt definition"},
{name: "missing profile/default", err: wrap(usecase.ErrInvalidRequest, errors.New("profile id is required either in request or prompt default_profile")), status: http.StatusBadRequest, code: "profile_required", message: "profile_id is required when prompt default_profile is not set"},
{name: "missing profile/default", err: wrap(usecase.ErrInvalidRequest, usecase.ErrProfileRequired), status: http.StatusBadRequest, code: "profile_required", message: "profile_id is required when prompt default_profile is not set"},
{name: "profile not found", err: wrap(usecase.ErrProfileLoad, profile.ErrProfileNotFound), status: http.StatusNotFound, code: "profile_not_found", message: "execution profile not found"},
{name: "profile invalid", err: wrap(usecase.ErrProfileLoad, profile.ErrInvalidProfile), status: http.StatusBadRequest, code: "profile_load_failed", message: "failed to load execution profile"},
{name: "api key env missing", err: wrap(usecase.ErrInvalidRequest, errors.New(`api key environment variable "SCRIPTORIUM_API_KEY" is not set`)), status: http.StatusBadRequest, code: "api_key_env_missing", message: "api_key_env is set but the environment variable is missing"},
{name: "api key env missing", err: wrap(usecase.ErrInvalidRequest, usecase.ErrAPIKeyEnvMissing), status: http.StatusBadRequest, code: "api_key_env_missing", message: "api_key_env is set but the environment variable is missing"},
{name: "artifact", err: wrap(usecase.ErrArtifactLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "artifact_read_failed", message: "failed to read input artifact", avoidCause: "read failed"},
{name: "prompt render", err: wrap(usecase.ErrPromptRender, fmt.Errorf("render failed")), status: http.StatusBadRequest, code: "prompt_render_failed", message: "failed to render prompt", avoidCause: "render failed"},
{name: "llm", err: wrap(usecase.ErrLLMGenerate, fmt.Errorf("llm failed")), status: http.StatusBadGateway, code: "llm_failed", message: "model generation request failed", avoidCause: "llm failed"},

View File

@@ -46,8 +46,8 @@ func TestCompositeReader_Read(t *testing.T) {
t.Run("unsupported ref type", func(t *testing.T) {
ref := domain.ArtifactRef{
Type: domain.ArtifactRefS3,
URI: "s3://bucket/key",
Type: domain.ArtifactRefType("unsupported"),
URI: "unsupported://bucket/key",
}
_, err := reader.Read(ctx, ref)
if !errors.Is(err, ErrUnsupportedRefType) {

View File

@@ -10,7 +10,6 @@ type ArtifactRefType string
const (
ArtifactRefInline ArtifactRefType = "inline"
ArtifactRefFile ArtifactRefType = "file"
ArtifactRefS3 ArtifactRefType = "s3"
)
// OutputFormat defines the desired format of the generated artifact.
@@ -41,6 +40,24 @@ const (
ValidationSkipped ValidationStatus = "skipped"
)
// CacheControlType defines provider cache behavior for prompt content.
type CacheControlType string
const (
CacheControlEphemeral CacheControlType = "ephemeral"
)
const (
// SessionIDMaxLength is OpenRouter's documented maximum session_id length.
SessionIDMaxLength = 256
)
// CacheControl describes provider cache metadata attached to prompt content.
type CacheControl struct {
Type CacheControlType `yaml:"type" json:"type"`
TTL string `yaml:"ttl,omitempty" json:"ttl,omitempty"`
}
// RunRequest represents a request to generate a single artifact.
type RunRequest struct {
PromptID string
@@ -48,7 +65,7 @@ type RunRequest struct {
ProfileID string
Inputs map[string]ArtifactRef
Vars map[string]string
Execution *ExecutionTarget
Execution *ExecutionTargetOverride
Validation *OutputContract
Metadata map[string]string
}
@@ -78,19 +95,21 @@ type RunResult struct {
// PreparedRun contains pre-LLM execution state from the prepare/render phase.
// It must never include resolved API key values, model output, or validation data.
type PreparedRun struct {
PromptID string `json:"prompt_id"`
PromptVersion string `json:"prompt_version,omitempty"`
PromptHash string `json:"prompt_hash,omitempty"`
SelectedProfileID string `json:"selected_profile_id"`
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
OutputContract OutputContract `json:"output_contract"`
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
InputHashes map[string]string `json:"input_hashes,omitempty"`
RenderedPromptHash string `json:"rendered_prompt_hash"`
Messages []RenderedMessage `json:"messages"`
StartTime time.Time `json:"start_time,omitempty"`
EndTime time.Time `json:"end_time,omitempty"`
DurationMS int64 `json:"duration_ms,omitempty"`
PromptID string `json:"prompt_id"`
PromptVersion string `json:"prompt_version,omitempty"`
PromptHash string `json:"prompt_hash,omitempty"`
SelectedProfileID string `json:"selected_profile_id"`
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
TargetPresence ExecutionTargetPresence `json:"-"`
OutputContract OutputContract `json:"output_contract"`
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
InputHashes map[string]string `json:"input_hashes,omitempty"`
SessionID string `json:"session_id,omitempty"`
RenderedPromptHash string `json:"rendered_prompt_hash"`
Messages []RenderedMessage `json:"messages"`
StartTime time.Time `json:"start_time,omitempty"`
EndTime time.Time `json:"end_time,omitempty"`
DurationMS int64 `json:"duration_ms,omitempty"`
}
// ArtifactRef represents a reference to an input artifact.
@@ -116,6 +135,7 @@ type PromptDefinition struct {
Version string `yaml:"version"`
DefaultProfile string `yaml:"default_profile"`
Description string `yaml:"description"`
SessionID string `yaml:"session_id" json:"session_id,omitempty"`
Inputs []PromptInput `yaml:"inputs"`
Templates []PromptMessageTemplate `yaml:"templates"`
OutputFormat OutputFormat `yaml:"output_format"`
@@ -132,36 +152,62 @@ type PromptInput struct {
// PromptMessageTemplate defines a template for a chat message.
type PromptMessageTemplate struct {
Role string `yaml:"role"`
Content string `yaml:"content"`
ContentFile string `yaml:"content_file"`
Role string `yaml:"role"`
Content string `yaml:"content"`
ContentFile string `yaml:"content_file"`
CacheControl *CacheControl `yaml:"cache_control,omitempty" json:"cache_control,omitempty"`
}
// ExecutionProfile describes how and where to execute a model.
type ExecutionProfile struct {
ID string `yaml:"id"`
Endpoint string `yaml:"endpoint"`
Model string `yaml:"model"`
Temperature float64 `yaml:"temperature"`
MaxTokens int `yaml:"max_tokens"`
TopP float64 `yaml:"top_p"`
TimeoutSeconds int `yaml:"timeout_seconds"`
ReasoningEffort string `yaml:"reasoning_effort"`
APIKeyEnv string `yaml:"api_key_env"`
ExtraParams map[string]string `yaml:"extra_params"`
ID string `yaml:"id"`
Endpoint string `yaml:"endpoint"`
Model string `yaml:"model"`
Temperature float64 `yaml:"temperature"`
MaxTokens int `yaml:"max_tokens"`
TopP float64 `yaml:"top_p"`
TimeoutSeconds int `yaml:"timeout_seconds"`
ServiceTier string `yaml:"service_tier"`
ReasoningEffort string `yaml:"reasoning_effort"`
APIKeyEnv string `yaml:"api_key_env"`
ExtraParams map[string]any `yaml:"extra_params"`
}
// ExecutionTargetOverride represents per-request runtime setting overrides.
type ExecutionTargetOverride struct {
Endpoint string `json:"endpoint,omitempty"`
Model string `json:"model,omitempty"`
Temperature *float64 `json:"temperature,omitempty"`
MaxTokens *int `json:"max_tokens,omitempty"`
TopP *float64 `json:"top_p,omitempty"`
TimeoutSeconds *int `json:"timeout_seconds,omitempty"`
ServiceTier string `json:"service_tier,omitempty"`
ReasoningEffort string `json:"reasoning_effort,omitempty"`
APIKeyEnv string `json:"api_key_env,omitempty"`
ExtraParams map[string]any `json:"extra_params,omitempty"`
}
// ExecutionTargetPresence tracks which effective runtime fields came from an
// explicit request override even when the resolved value is a zero value.
type ExecutionTargetPresence struct {
Temperature bool
MaxTokens bool
TopP bool
TimeoutSeconds bool
}
// ExecutionTarget represents effective model runtime settings for a run.
type ExecutionTarget struct {
Endpoint string `yaml:"endpoint" json:"endpoint"`
Model string `yaml:"model" json:"model"`
Temperature float64 `yaml:"temperature" json:"temperature"`
MaxTokens int `yaml:"max_tokens" json:"max_tokens"`
TopP float64 `yaml:"top_p" json:"top_p"`
TimeoutSeconds int `yaml:"timeout_seconds" json:"timeout_seconds"`
ReasoningEffort string `yaml:"reasoning_effort" json:"reasoning_effort"`
APIKeyEnv string `yaml:"api_key_env" json:"api_key_env"`
ExtraParams map[string]string `yaml:"extra_params" json:"extra_params"`
Endpoint string `yaml:"endpoint" json:"endpoint"`
Model string `yaml:"model" json:"model"`
Temperature float64 `yaml:"temperature" json:"temperature"`
MaxTokens int `yaml:"max_tokens" json:"max_tokens"`
TopP float64 `yaml:"top_p" json:"top_p"`
TimeoutSeconds int `yaml:"timeout_seconds" json:"timeout_seconds"`
ServiceTier string `yaml:"service_tier" json:"service_tier"`
ReasoningEffort string `yaml:"reasoning_effort" json:"reasoning_effort"`
APIKeyEnv string `yaml:"api_key_env" json:"api_key_env"`
ExtraParams map[string]any `yaml:"extra_params" json:"extra_params"`
}
// OutputContract defines the requirements for the output artifact.
@@ -174,19 +220,22 @@ type OutputContract struct {
// RenderedPrompt represents the prompt after template application.
type RenderedPrompt struct {
Messages []RenderedMessage `json:"messages"`
SessionID string `json:"session_id,omitempty"`
Messages []RenderedMessage `json:"messages"`
}
// RenderedMessage is a single message in a rendered prompt.
type RenderedMessage struct {
Role string `json:"role"`
Content string `json:"content"`
Role string `json:"role"`
Content string `json:"content"`
CacheControl *CacheControl `json:"cache_control,omitempty"`
}
// GenerateRequest is the internal request passed to the LLM client.
type GenerateRequest struct {
Prompt RenderedPrompt
Target ExecutionTarget
TargetPresence ExecutionTargetPresence
StructuredOutput *StructuredOutputSpec
}
@@ -221,6 +270,8 @@ type TokenUsage struct {
PromptTokens int
CompletionTokens int
TotalTokens int
CachedTokens int
CacheWriteTokens int
}
// ValidationResult represents the outcome of an output validation.

View File

@@ -53,3 +53,88 @@ func TestPreparedRunJSONDoesNotIncludeSecretValues(t *testing.T) {
}
}
}
func TestPreparedRunJSONIncludesMessageCacheControlOnlyWhenPresent(t *testing.T) {
prepared := PreparedRun{
PromptID: "prompt.id",
SelectedProfileID: "local-fast",
EffectiveModelParams: ExecutionTarget{
Endpoint: "http://llm/v1",
Model: "gpt-test",
},
RenderedPromptHash: "rendered-hash",
Messages: []RenderedMessage{
{
Role: "system",
Content: "You are helpful.",
CacheControl: &CacheControl{
Type: CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "Summarize this."},
},
}
b, err := json.Marshal(prepared)
if err != nil {
t.Fatalf("marshal failed: %v", err)
}
var decoded struct {
Messages []map[string]any `json:"messages"`
}
if err := json.Unmarshal(b, &decoded); err != nil {
t.Fatalf("unmarshal failed: %v", err)
}
if len(decoded.Messages) != 2 {
t.Fatalf("expected 2 messages, got %d", len(decoded.Messages))
}
cacheControl, ok := decoded.Messages[0]["cache_control"].(map[string]any)
if !ok {
t.Fatalf("expected cache_control on first message, got %#v", decoded.Messages[0])
}
if cacheControl["type"] != string(CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
}
if _, ok := decoded.Messages[1]["cache_control"]; ok {
t.Fatalf("expected second message to omit cache_control, got %#v", decoded.Messages[1])
}
}
func TestPreparedRunJSONIncludesSessionIDOnlyWhenPresent(t *testing.T) {
prepared := PreparedRun{
PromptID: "prompt.id",
SelectedProfileID: "local-fast",
EffectiveModelParams: ExecutionTarget{
Endpoint: "http://llm/v1",
Model: "gpt-test",
},
SessionID: "session-123",
RenderedPromptHash: "rendered-hash",
Messages: []RenderedMessage{{Role: "user", Content: "Summarize this."}},
}
b, err := json.Marshal(prepared)
if err != nil {
t.Fatalf("marshal failed: %v", err)
}
var decoded map[string]any
if err := json.Unmarshal(b, &decoded); err != nil {
t.Fatalf("unmarshal failed: %v", err)
}
if decoded["session_id"] != "session-123" {
t.Fatalf("expected session_id in prepared run JSON, got %#v", decoded["session_id"])
}
prepared.SessionID = ""
b, err = json.Marshal(prepared)
if err != nil {
t.Fatalf("marshal failed: %v", err)
}
if strings.Contains(string(b), "session_id") {
t.Fatalf("expected empty session_id to be omitted, got %s", b)
}
}

View File

@@ -0,0 +1,54 @@
package filecatalog
import (
"context"
"os"
"path/filepath"
"sort"
"strings"
)
// FindYAMLFiles returns sorted full paths for .yaml and .yml files under root.
func FindYAMLFiles(ctx context.Context, root string) ([]string, error) {
var files []string
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
select {
case <-ctx.Done():
return ctx.Err()
default:
}
if d.IsDir() {
return nil
}
if !isYAMLFile(d.Name()) {
return nil
}
files = append(files, path)
return nil
})
sort.Strings(files)
return files, err
}
// RelativePath computes a clean relative path from root to path.
func RelativePath(root string, path string) string {
rel, err := filepath.Rel(root, path)
if err != nil {
return filepath.Clean(path)
}
return filepath.Clean(rel)
}
// Stem strips .yaml or .yml from a file name.
func Stem(name string) string {
name = strings.TrimSuffix(name, ".yaml")
name = strings.TrimSuffix(name, ".yml")
return name
}
func isYAMLFile(name string) bool {
return strings.HasSuffix(name, ".yaml") || strings.HasSuffix(name, ".yml")
}

View File

@@ -0,0 +1,84 @@
package filecatalog
import (
"context"
"errors"
"os"
"path/filepath"
"reflect"
"testing"
)
func TestFindYAMLFilesNestedSortedAndFiltered(t *testing.T) {
root := t.TempDir()
mustWriteFile(t, filepath.Join(root, "z", "prompt.yml"), "id: z")
mustWriteFile(t, filepath.Join(root, "a", "profile.yaml"), "id: a")
mustWriteFile(t, filepath.Join(root, "a", "ignore.txt"), "not yaml")
mustWriteFile(t, filepath.Join(root, "b", "ignore.yaml.bak"), "not yaml")
got, err := FindYAMLFiles(context.Background(), root)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
want := []string{
filepath.Join(root, "a", "profile.yaml"),
filepath.Join(root, "z", "prompt.yml"),
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("expected sorted YAML files %v, got %v", want, got)
}
}
func TestFindYAMLFilesHonorsContextCancellation(t *testing.T) {
root := t.TempDir()
mustWriteFile(t, filepath.Join(root, "one.yaml"), "id: one")
ctx, cancel := context.WithCancel(context.Background())
cancel()
_, err := FindYAMLFiles(ctx, root)
if !errors.Is(err, context.Canceled) {
t.Fatalf("expected context.Canceled, got %v", err)
}
}
func TestRelativePathNested(t *testing.T) {
root := t.TempDir()
path := filepath.Join(root, "nested", "profiles", "local.yaml")
got := RelativePath(root, path)
want := filepath.Join("nested", "profiles", "local.yaml")
if got != want {
t.Fatalf("expected relative path %q, got %q", want, got)
}
}
func TestStemStripsYAMLExtensions(t *testing.T) {
tests := []struct {
name string
in string
want string
}{
{name: "yaml", in: "prompt.yaml", want: "prompt"},
{name: "yml", in: "profile.yml", want: "profile"},
{name: "other", in: "file.txt", want: "file.txt"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if got := Stem(tc.in); got != tc.want {
t.Fatalf("expected %q, got %q", tc.want, got)
}
})
}
}
func mustWriteFile(t *testing.T, path string, content string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatalf("failed to create directory: %v", err)
}
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatalf("failed to write file %q: %v", path, err)
}
}

View File

@@ -96,6 +96,9 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
if prepared.PromptHash != "" {
fmt.Fprintf(&b, "prompt_hash: %s\n", prepared.PromptHash)
}
if prepared.SessionID != "" {
fmt.Fprintf(&b, "session_id: %s\n", prepared.SessionID)
}
fmt.Fprintf(&b, "rendered_prompt_hash: %s\n", prepared.RenderedPromptHash)
target := prepared.EffectiveModelParams
@@ -106,6 +109,9 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
fmt.Fprintf(&b, " max_tokens: %d\n", target.MaxTokens)
fmt.Fprintf(&b, " top_p: %g\n", target.TopP)
fmt.Fprintf(&b, " timeout_seconds: %d\n", target.TimeoutSeconds)
if target.ServiceTier != "" {
fmt.Fprintf(&b, " service_tier: %s\n", target.ServiceTier)
}
if target.ReasoningEffort != "" {
fmt.Fprintf(&b, " reasoning_effort: %s\n", target.ReasoningEffort)
}
@@ -120,7 +126,11 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
}
sort.Strings(keys)
for _, k := range keys {
fmt.Fprintf(&b, " %s: %s\n", k, target.ExtraParams[k])
renderedValue, err := formatExtraParamTextValue(target.ExtraParams[k])
if err != nil {
return nil, fmt.Errorf("failed to format extra_params.%s: %w", k, err)
}
fmt.Fprintf(&b, " %s: %s\n", k, renderedValue)
}
}
@@ -148,6 +158,13 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
messages := byRole[role]
for i, msg := range messages {
fmt.Fprintf(&b, " - message: %d\n", i+1)
if msg.CacheControl != nil {
fmt.Fprintf(&b, " cache_control: %s", msg.CacheControl.Type)
if msg.CacheControl.TTL != "" {
fmt.Fprintf(&b, " ttl=%s", msg.CacheControl.TTL)
}
fmt.Fprintln(&b)
}
fmt.Fprintln(&b, " content: |")
content := msg.Content
if content == "" {
@@ -162,3 +179,15 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
return b.Bytes(), nil
}
func formatExtraParamTextValue(value any) (string, error) {
if s, ok := value.(string); ok {
return s, nil
}
b, err := json.Marshal(value)
if err != nil {
return "", err
}
return string(b), nil
}

View File

@@ -28,6 +28,7 @@ func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
"max_tokens: 256",
"top_p: 0.8",
"timeout_seconds: 45",
"service_tier: priority",
"reasoning_effort: medium",
"api_key_env: SCRIPTORIUM_API_KEY",
"prompt_hash: prompt-hash",
@@ -48,6 +49,36 @@ func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
}
}
func TestTextFormatterRendersExtraParamsDeterministically(t *testing.T) {
prepared := samplePreparedRun()
prepared.EffectiveModelParams.ExtraParams = map[string]any{
"z_string": "enabled",
"b_number": 42,
"a_object": map[string]any{
"nested": "value",
"count": 2,
},
"c_array": []any{"first", 3, false},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
s := string(out)
want := strings.Join([]string{
" extra_params:",
" a_object: {\"count\":2,\"nested\":\"value\"}",
" b_number: 42",
" c_array: [\"first\",3,false]",
" z_string: enabled",
}, "\n")
if !strings.Contains(s, want) {
t.Fatalf("expected deterministic extra_params block %q, got:\n%s", want, s)
}
}
func TestTextFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
const secret = "super-secret-api-key"
t.Setenv("SCRIPTORIUM_API_KEY", secret)
@@ -61,8 +92,80 @@ func TestTextFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
}
}
func TestTextFormatterIncludesMessageCacheControlBeforeContent(t *testing.T) {
prepared := samplePreparedRun()
prepared.Messages = []domain.RenderedMessage{
{
Role: "system",
Content: "System guidance.",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "Summarize the transcript."},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
s := string(out)
if !strings.Contains(s, " system:\n - message: 1\n cache_control: ephemeral ttl=1h\n content: |") {
t.Fatalf("expected system message cache control before content, got:\n%s", s)
}
if strings.Count(s, "cache_control:") != 1 {
t.Fatalf("expected exactly one cache_control line, got:\n%s", s)
}
}
func TestTextFormatterIncludesSessionIDWhenPresent(t *testing.T) {
prepared := samplePreparedRun()
prepared.SessionID = "session-123"
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if !strings.Contains(string(out), "session_id: session-123\n") {
t.Fatalf("expected session_id in text output, got:\n%s", out)
}
}
func TestTextFormatterOmitsEmptyCacheControlTTL(t *testing.T) {
prepared := samplePreparedRun()
prepared.Messages = []domain.RenderedMessage{
{
Role: "system",
Content: "System guidance.",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
},
},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
s := string(out)
if !strings.Contains(s, " cache_control: ephemeral\n") {
t.Fatalf("expected cache_control line without ttl, got:\n%s", s)
}
if strings.Contains(s, "ttl=") {
t.Fatalf("expected empty ttl to be omitted, got:\n%s", s)
}
}
func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
prepared := samplePreparedRun()
prepared.SessionID = "session-123"
prepared.EffectiveModelParams.ExtraParams = map[string]any{
"number": 42,
"nested": map[string]any{
"enabled": true,
},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
if err != nil {
@@ -86,9 +189,24 @@ func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
if decoded["rendered_prompt_hash"] != "rendered-hash" {
t.Fatalf("expected rendered_prompt_hash in json output, got %#v", decoded["rendered_prompt_hash"])
}
if _, ok := decoded["effective_model_params"]; !ok {
if decoded["session_id"] != "session-123" {
t.Fatalf("expected session_id in json output, got %#v", decoded["session_id"])
}
modelParams, ok := decoded["effective_model_params"].(map[string]any)
if !ok {
t.Fatalf("expected effective_model_params in json output, got %#v", decoded)
}
extraParams, ok := modelParams["extra_params"].(map[string]any)
if !ok {
t.Fatalf("expected extra_params in json output, got %#v", modelParams["extra_params"])
}
if extraParams["number"] != float64(42) {
t.Fatalf("unexpected numeric extra param in json output: %#v", extraParams["number"])
}
nested, ok := extraParams["nested"].(map[string]any)
if !ok || nested["enabled"] != true {
t.Fatalf("unexpected nested extra param in json output: %#v", extraParams["nested"])
}
if _, ok := decoded["input_hashes"]; !ok {
t.Fatalf("expected input_hashes in json output, got %#v", decoded)
}
@@ -97,6 +215,47 @@ func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
}
}
func TestJSONFormatterIncludesMessageCacheControlOnlyWhenPresent(t *testing.T) {
prepared := samplePreparedRun()
prepared.Messages = []domain.RenderedMessage{
{
Role: "system",
Content: "System guidance.",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "Summarize the transcript."},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
var decoded struct {
Messages []map[string]any `json:"messages"`
}
if err := json.Unmarshal(out, &decoded); err != nil {
t.Fatalf("expected valid json output, got %v", err)
}
if len(decoded.Messages) != 2 {
t.Fatalf("expected 2 messages, got %d", len(decoded.Messages))
}
cacheControl, ok := decoded.Messages[0]["cache_control"].(map[string]any)
if !ok {
t.Fatalf("expected first message cache_control, got %#v", decoded.Messages[0])
}
if cacheControl["type"] != string(domain.CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
}
if _, ok := decoded.Messages[1]["cache_control"]; ok {
t.Fatalf("expected second message to omit cache_control, got %#v", decoded.Messages[1])
}
}
func TestJSONFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
const secret = "super-secret-api-key"
t.Setenv("SCRIPTORIUM_API_KEY", secret)
@@ -168,6 +327,7 @@ func samplePreparedRun() *domain.PreparedRun {
MaxTokens: 256,
TopP: 0.8,
TimeoutSeconds: 45,
ServiceTier: "priority",
ReasoningEffort: "medium",
APIKeyEnv: "SCRIPTORIUM_API_KEY",
},

View File

@@ -12,6 +12,7 @@ import (
"os"
"strings"
"time"
"unicode/utf8"
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
@@ -75,14 +76,6 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
return nil, fmt.Errorf("%w: timeout_seconds must be greater than or equal to 0", ErrInvalidRequest)
}
model := strings.TrimSpace(req.Target.Model)
if model == "" {
model = strings.TrimSpace(c.defaultModel)
}
if model == "" {
return nil, fmt.Errorf("%w: model is required", ErrInvalidRequest)
}
endpoint := strings.TrimSpace(req.Target.Endpoint)
if endpoint == "" {
endpoint = c.baseURL
@@ -92,36 +85,17 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
}
endpoint = strings.TrimRight(endpoint, "/") + defaults.OpenAIChatCompletionsPath
wireReq := openAIChatRequest{
Model: model,
wireReq, err := openAIChatRequestFromGenerateRequest(req, c.defaultModel)
if err != nil {
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
}
wireReq.Messages = make([]openAIChatMessage, 0, len(req.Prompt.Messages))
for _, msg := range req.Prompt.Messages {
wireReq.Messages = append(wireReq.Messages, openAIChatMessage{
Role: msg.Role,
Content: msg.Content,
})
wirePayload, err := openAIChatRequestPayload(wireReq)
if err != nil {
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
}
if req.Target.Temperature != 0 {
wireReq.Temperature = &req.Target.Temperature
}
if req.Target.MaxTokens != 0 {
wireReq.MaxTokens = &req.Target.MaxTokens
}
if req.Target.TopP != 0 {
wireReq.TopP = &req.Target.TopP
}
if req.StructuredOutput != nil {
responseFormat, err := toOpenAIResponseFormat(req.StructuredOutput)
if err != nil {
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
}
wireReq.ResponseFormat = responseFormat
}
payload, err := json.Marshal(wireReq)
payload, err := json.Marshal(wirePayload)
if err != nil {
return nil, fmt.Errorf("%w: failed to encode request: %v", ErrRequestFailed, err)
}
@@ -142,6 +116,8 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
effectiveTimeout := c.timeout
if req.Target.TimeoutSeconds > 0 {
effectiveTimeout = time.Duration(req.Target.TimeoutSeconds) * time.Second
} else if req.TargetPresence.TimeoutSeconds {
effectiveTimeout = 0
}
httpClient := c.httpClient
@@ -183,32 +159,166 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
PromptTokens: wireResp.Usage.PromptTokens,
CompletionTokens: wireResp.Usage.CompletionTokens,
TotalTokens: wireResp.Usage.TotalTokens,
CachedTokens: wireResp.Usage.PromptTokensDetails.CachedTokens,
CacheWriteTokens: wireResp.Usage.CacheWriteTokens,
},
}, nil
}
type openAIChatRequest struct {
Model string `json:"model"`
Messages []openAIChatMessage `json:"messages"`
Temperature *float64 `json:"temperature,omitempty"`
MaxTokens *int `json:"max_tokens,omitempty"`
TopP *float64 `json:"top_p,omitempty"`
ResponseFormat *openAIResponseFormat `json:"response_format,omitempty"`
func openAIChatRequestFromGenerateRequest(req domain.GenerateRequest, defaultModel string) (openAIChatRequest, error) {
model := strings.TrimSpace(req.Target.Model)
if model == "" {
model = strings.TrimSpace(defaultModel)
}
if model == "" {
return openAIChatRequest{}, errors.New("model is required")
}
wireReq := openAIChatRequest{
Model: model,
}
if sessionID := strings.TrimSpace(req.Prompt.SessionID); sessionID != "" {
if n := utf8.RuneCountInString(sessionID); n > domain.SessionIDMaxLength {
return openAIChatRequest{}, fmt.Errorf("session_id length %d exceeds maximum %d", n, domain.SessionIDMaxLength)
}
wireReq.SessionID = sessionID
}
wireReq.Messages = make([]openAIChatRequestMessage, 0, len(req.Prompt.Messages))
for _, msg := range req.Prompt.Messages {
wireReq.Messages = append(wireReq.Messages, openAIChatRequestMessageFromRenderedMessage(msg))
}
if req.Target.Temperature != 0 || req.TargetPresence.Temperature {
wireReq.Temperature = &req.Target.Temperature
}
if req.Target.MaxTokens != 0 || req.TargetPresence.MaxTokens {
wireReq.MaxTokens = &req.Target.MaxTokens
}
if req.Target.TopP != 0 || req.TargetPresence.TopP {
wireReq.TopP = &req.Target.TopP
}
if strings.TrimSpace(req.Target.ServiceTier) != "" {
wireReq.ServiceTier = req.Target.ServiceTier
}
if strings.TrimSpace(req.Target.ReasoningEffort) != "" {
wireReq.ReasoningEffort = req.Target.ReasoningEffort
}
if len(req.Target.ExtraParams) > 0 {
wireReq.ExtraParams = req.Target.ExtraParams
}
if req.StructuredOutput != nil {
responseFormat, err := toOpenAIResponseFormat(req.StructuredOutput)
if err != nil {
return openAIChatRequest{}, err
}
wireReq.ResponseFormat = responseFormat
}
return wireReq, nil
}
type openAIChatMessage struct {
type openAIChatRequest struct {
Model string `json:"model"`
SessionID string `json:"session_id,omitempty"`
Messages []openAIChatRequestMessage `json:"messages"`
Temperature *float64 `json:"temperature,omitempty"`
MaxTokens *int `json:"max_tokens,omitempty"`
TopP *float64 `json:"top_p,omitempty"`
ServiceTier string `json:"service_tier,omitempty"`
ReasoningEffort string `json:"reasoning_effort,omitempty"`
ResponseFormat *openAIResponseFormat `json:"response_format,omitempty"`
ExtraParams map[string]any `json:"-"`
}
func openAIChatRequestPayload(req openAIChatRequest) (map[string]any, error) {
out := map[string]any{
"model": req.Model,
"messages": req.Messages,
}
if req.SessionID != "" {
out["session_id"] = req.SessionID
}
if req.Temperature != nil {
out["temperature"] = *req.Temperature
}
if req.MaxTokens != nil {
out["max_tokens"] = *req.MaxTokens
}
if req.TopP != nil {
out["top_p"] = *req.TopP
}
if req.ServiceTier != "" {
out["service_tier"] = req.ServiceTier
}
if req.ReasoningEffort != "" {
out["reasoning_effort"] = req.ReasoningEffort
}
if req.ResponseFormat != nil {
out["response_format"] = req.ResponseFormat
}
for key, value := range req.ExtraParams {
if key == "" {
return nil, errors.New("extra_params key must not be empty")
}
if _, reserved := reservedOpenAIChatRequestFields[key]; reserved {
return nil, fmt.Errorf("extra_params key %q collides with reserved request field", key)
}
if _, err := json.Marshal(value); err != nil {
return nil, fmt.Errorf("extra_params.%s must be JSON-serializable: %w", key, err)
}
out[key] = value
}
return out, nil
}
var reservedOpenAIChatRequestFields = map[string]struct{}{
"model": {},
"session_id": {},
"messages": {},
"temperature": {},
"max_tokens": {},
"top_p": {},
"service_tier": {},
"reasoning_effort": {},
"response_format": {},
}
type openAIChatRequestMessage struct {
Role string `json:"role"`
Content any `json:"content"`
}
type openAIChatTextContentBlock struct {
Type string `json:"type"`
Text string `json:"text"`
CacheControl *openAICacheControl `json:"cache_control,omitempty"`
}
type openAICacheControl struct {
Type string `json:"type"`
TTL string `json:"ttl,omitempty"`
}
type openAIChatResponseMessage struct {
Role string `json:"role"`
Content string `json:"content"`
}
type openAIChatResponse struct {
Choices []struct {
Message openAIChatMessage `json:"message"`
Message openAIChatResponseMessage `json:"message"`
} `json:"choices"`
Usage struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"`
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"`
PromptTokensDetails struct {
CachedTokens int `json:"cached_tokens"`
} `json:"prompt_tokens_details"`
CacheWriteTokens int `json:"cache_write_tokens"`
} `json:"usage"`
}
@@ -223,6 +333,28 @@ type openAIJSONSchemaEnvelope struct {
Schema any `json:"schema"`
}
func openAIChatRequestMessageFromRenderedMessage(msg domain.RenderedMessage) openAIChatRequestMessage {
wireMsg := openAIChatRequestMessage{
Role: msg.Role,
Content: msg.Content,
}
if msg.CacheControl == nil {
return wireMsg
}
wireMsg.Content = []openAIChatTextContentBlock{
{
Type: "text",
Text: msg.Content,
CacheControl: &openAICacheControl{
Type: string(msg.CacheControl.Type),
TTL: msg.CacheControl.TTL,
},
},
}
return wireMsg
}
func toOpenAIResponseFormat(spec *domain.StructuredOutputSpec) (*openAIResponseFormat, error) {
if spec == nil {
return nil, nil

View File

@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"errors"
"math"
"net/http"
"net/http/httptest"
"strings"
@@ -61,6 +62,7 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
Temperature: 0.4,
MaxTokens: 123,
TopP: 0.7,
ServiceTier: "priority",
APIKeyEnv: "SCRIPTORIUM_TEST_API_KEY",
},
StructuredOutput: &domain.StructuredOutputSpec{
@@ -88,6 +90,9 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
if resp.Usage.PromptTokens != 11 || resp.Usage.CompletionTokens != 22 || resp.Usage.TotalTokens != 33 {
t.Fatalf("unexpected usage: %+v", resp.Usage)
}
if resp.Usage.CachedTokens != 0 || resp.Usage.CacheWriteTokens != 0 {
t.Fatalf("expected absent cache usage fields to remain zero, got %+v", resp.Usage)
}
if obs.Authorization != "Bearer secret-key" {
t.Fatalf("unexpected Authorization header: %q", obs.Authorization)
@@ -95,6 +100,18 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
if got, ok := obs.Body["model"].(string); !ok || got != "gpt-test" {
t.Fatalf("unexpected model payload: %#v", obs.Body["model"])
}
if got, ok := obs.Body["temperature"].(float64); !ok || got != 0.4 {
t.Fatalf("unexpected temperature payload: %#v", obs.Body["temperature"])
}
if got, ok := obs.Body["max_tokens"].(float64); !ok || got != 123 {
t.Fatalf("unexpected max_tokens payload: %#v", obs.Body["max_tokens"])
}
if got, ok := obs.Body["top_p"].(float64); !ok || got != 0.7 {
t.Fatalf("unexpected top_p payload: %#v", obs.Body["top_p"])
}
if got, ok := obs.Body["service_tier"].(string); !ok || got != "priority" {
t.Fatalf("unexpected service_tier payload: %#v", obs.Body["service_tier"])
}
msgs, ok := obs.Body["messages"].([]any)
if !ok || len(msgs) != 2 {
@@ -131,6 +148,245 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
}
}
func TestOpenAICompatibleClientSerializesCacheControlledMessageAsContentBlock(t *testing.T) {
var observedBody map[string]any
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
t.Fatalf("failed to decode request body: %v", err)
}
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{
{
Role: "system",
Content: "Stable instructions.",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "Dynamic request."},
}},
Target: domain.ExecutionTarget{Model: "model"},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
for _, forbidden := range []string{"cache_control", "extra_params"} {
if _, exists := observedBody[forbidden]; exists {
t.Fatalf("expected top-level %s to be omitted, got %#v", forbidden, observedBody[forbidden])
}
}
msgs, ok := observedBody["messages"].([]any)
if !ok || len(msgs) != 2 {
t.Fatalf("unexpected messages payload: %#v", observedBody["messages"])
}
msg0 := msgs[0].(map[string]any)
if msg0["role"] != "system" {
t.Fatalf("unexpected first message role: %#v", msg0["role"])
}
contentBlocks, ok := msg0["content"].([]any)
if !ok || len(contentBlocks) != 1 {
t.Fatalf("expected first message content block array, got %#v", msg0["content"])
}
block := contentBlocks[0].(map[string]any)
if block["type"] != "text" || block["text"] != "Stable instructions." {
t.Fatalf("unexpected text content block: %#v", block)
}
cacheControl, ok := block["cache_control"].(map[string]any)
if !ok {
t.Fatalf("expected cache_control on content block, got %#v", block)
}
if cacheControl["type"] != string(domain.CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
}
msg1 := msgs[1].(map[string]any)
if msg1["role"] != "user" || msg1["content"] != "Dynamic request." {
t.Fatalf("expected uncached message to keep string content, got %#v", msg1)
}
}
func TestOpenAICompatibleClientOmitsEmptyCacheControlTTL(t *testing.T) {
var observedBody map[string]any
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
t.Fatalf("failed to decode request body: %v", err)
}
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{
{
Role: "system",
Content: "Stable instructions.",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
},
},
}},
Target: domain.ExecutionTarget{Model: "model"},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
msgs := observedBody["messages"].([]any)
msg0 := msgs[0].(map[string]any)
contentBlocks := msg0["content"].([]any)
block := contentBlocks[0].(map[string]any)
cacheControl := block["cache_control"].(map[string]any)
if cacheControl["type"] != string(domain.CacheControlEphemeral) {
t.Fatalf("unexpected cache_control type: %#v", cacheControl)
}
if _, exists := cacheControl["ttl"]; exists {
t.Fatalf("expected empty ttl to be omitted, got %#v", cacheControl)
}
}
func TestOpenAICompatibleClientSerializesSessionID(t *testing.T) {
var observedBody map[string]any
var observedSessionHeader string
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
observedSessionHeader = r.Header.Get("x-session-id")
defer r.Body.Close()
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
t.Fatalf("failed to decode request body: %v", err)
}
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{
SessionID: " session-123 ",
Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}},
},
Target: domain.ExecutionTarget{Model: "model"},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if observedBody["session_id"] != "session-123" {
t.Fatalf("expected top-level session_id, got %#v", observedBody["session_id"])
}
if observedSessionHeader != "" {
t.Fatalf("did not expect x-session-id header, got %q", observedSessionHeader)
}
}
func TestOpenAICompatibleClientOmitsEmptySessionID(t *testing.T) {
var observedBody map[string]any
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
t.Fatalf("failed to decode request body: %v", err)
}
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{
SessionID: " ",
Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}},
},
Target: domain.ExecutionTarget{Model: "model"},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if _, exists := observedBody["session_id"]; exists {
t.Fatalf("expected empty session_id to be omitted, got %#v", observedBody["session_id"])
}
}
func TestOpenAICompatibleClientRejectsTooLongSessionID(t *testing.T) {
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
BaseURL: "http://example.com/v1",
Model: "model",
})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{
SessionID: strings.Repeat("x", domain.SessionIDMaxLength+1),
Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}},
},
})
if err == nil {
t.Fatal("expected invalid request error")
}
if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected ErrInvalidRequest, got %v", err)
}
}
func TestOpenAICompatibleClientParsesCacheUsage(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{
"choices": [{"message": {"role": "assistant", "content": "ok"}}],
"usage": {
"prompt_tokens": 100,
"completion_tokens": 20,
"total_tokens": 120,
"prompt_tokens_details": {"cached_tokens": 80},
"cache_write_tokens": 60
}
}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1", Model: "model"})
if err != nil {
t.Fatal(err)
}
resp, err := client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if resp.Usage.PromptTokens != 100 || resp.Usage.CompletionTokens != 20 || resp.Usage.TotalTokens != 120 {
t.Fatalf("unexpected base usage fields: %+v", resp.Usage)
}
if resp.Usage.CachedTokens != 80 || resp.Usage.CacheWriteTokens != 60 {
t.Fatalf("unexpected cache usage fields: %+v", resp.Usage)
}
}
func TestOpenAICompatibleClientOmitsResponseFormatWhenNoStructuredOutput(t *testing.T) {
var observedBody map[string]any
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -157,6 +413,284 @@ func TestOpenAICompatibleClientOmitsResponseFormatWhenNoStructuredOutput(t *test
if _, exists := observedBody["response_format"]; exists {
t.Fatalf("expected response_format omitted, got %#v", observedBody["response_format"])
}
if _, exists := observedBody["service_tier"]; exists {
t.Fatalf("expected service_tier omitted, got %#v", observedBody["service_tier"])
}
}
func TestOpenAICompatibleClientSerializesReasoningEffortAndExtraParams(t *testing.T) {
var observedBody map[string]any
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
t.Fatalf("failed to decode request body: %v", err)
}
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ExecutionTarget{
Model: "model",
ReasoningEffort: "high",
ExtraParams: map[string]any{
"string_value": "on",
"number_value": 42,
"boolean_value": true,
"object_value": map[string]any{"nested": "value", "count": 2},
"array_value": []any{"first", 3, false},
},
},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if observedBody["reasoning_effort"] != "high" {
t.Fatalf("expected reasoning_effort high, got %#v", observedBody["reasoning_effort"])
}
if observedBody["string_value"] != "on" {
t.Fatalf("unexpected string extra param: %#v", observedBody["string_value"])
}
if observedBody["number_value"] != float64(42) {
t.Fatalf("unexpected number extra param: %#v", observedBody["number_value"])
}
if observedBody["boolean_value"] != true {
t.Fatalf("unexpected boolean extra param: %#v", observedBody["boolean_value"])
}
objectValue, ok := observedBody["object_value"].(map[string]any)
if !ok || objectValue["nested"] != "value" || objectValue["count"] != float64(2) {
t.Fatalf("unexpected object extra param: %#v", observedBody["object_value"])
}
if _, exists := observedBody["extra_params"]; exists {
t.Fatalf("expected extra_params wrapper omitted, got %#v", observedBody["extra_params"])
}
arrayValue, ok := observedBody["array_value"].([]any)
if !ok || len(arrayValue) != 3 || arrayValue[0] != "first" || arrayValue[1] != float64(3) || arrayValue[2] != false {
t.Fatalf("unexpected array extra param: %#v", observedBody["array_value"])
}
}
func TestOpenAICompatibleClientOmitsReasoningEffortWhenUnset(t *testing.T) {
var observedBody map[string]any
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
t.Fatalf("failed to decode request body: %v", err)
}
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ExecutionTarget{Model: "model"},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if _, exists := observedBody["reasoning_effort"]; exists {
t.Fatalf("expected reasoning_effort omitted, got %#v", observedBody["reasoning_effort"])
}
if _, exists := observedBody["extra_params"]; exists {
t.Fatalf("expected extra_params wrapper omitted, got %#v", observedBody["extra_params"])
}
}
func TestOpenAICompatibleClientSerializesExplicitZeroNumericOverrides(t *testing.T) {
var observedBody map[string]any
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
t.Fatalf("failed to decode request body: %v", err)
}
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ExecutionTarget{Model: "model"},
TargetPresence: domain.ExecutionTargetPresence{
Temperature: true,
MaxTokens: true,
TopP: true,
},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if observedBody["temperature"] != float64(0) {
t.Fatalf("expected explicit zero temperature, got %#v", observedBody["temperature"])
}
if observedBody["max_tokens"] != float64(0) {
t.Fatalf("expected explicit zero max_tokens, got %#v", observedBody["max_tokens"])
}
if observedBody["top_p"] != float64(0) {
t.Fatalf("expected explicit zero top_p, got %#v", observedBody["top_p"])
}
}
func TestOpenAICompatibleClientOmitsImplicitZeroNumericFields(t *testing.T) {
var observedBody map[string]any
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
t.Fatalf("failed to decode request body: %v", err)
}
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ExecutionTarget{Model: "model"},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
for _, field := range []string{"temperature", "max_tokens", "top_p"} {
if _, exists := observedBody[field]; exists {
t.Fatalf("expected implicit zero field %q to be omitted, got body %#v", field, observedBody)
}
}
}
func TestOpenAICompatibleClientExplicitZeroTimeoutDisablesClientTimeout(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
time.Sleep(20 * time.Millisecond)
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
BaseURL: ts.URL + "/v1",
Timeout: time.Nanosecond,
})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ExecutionTarget{Model: "model", TimeoutSeconds: 0},
TargetPresence: domain.ExecutionTargetPresence{TimeoutSeconds: true},
})
if err != nil {
t.Fatalf("expected explicit zero timeout to disable client timeout, got %v", err)
}
}
func TestOpenAICompatibleClientOmittedTimeoutUsesClientTimeout(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
time.Sleep(20 * time.Millisecond)
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
BaseURL: ts.URL + "/v1",
Timeout: time.Nanosecond,
})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ExecutionTarget{Model: "model", TimeoutSeconds: 0},
})
if err == nil {
t.Fatal("expected omitted timeout to use client timeout")
}
if !errors.Is(err, ErrRequestFailed) {
t.Fatalf("expected ErrRequestFailed, got %v", err)
}
}
func TestOpenAICompatibleClientRejectsInvalidExtraParamsBeforeProviderCall(t *testing.T) {
tests := []struct {
name string
extraParams map[string]any
want string
}{
{name: "empty key", extraParams: map[string]any{"": "empty"}, want: "key must not be empty"},
{name: "unserializable value", extraParams: map[string]any{"bad": math.Inf(1)}, want: "JSON-serializable"},
}
for _, key := range []string{
"model",
"session_id",
"messages",
"temperature",
"max_tokens",
"top_p",
"service_tier",
"reasoning_effort",
"response_format",
} {
tests = append(tests, struct {
name string
extraParams map[string]any
want string
}{
name: "reserved key " + key,
extraParams: map[string]any{key: "collision"},
want: "reserved request field",
})
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
called := false
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
called = true
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ExecutionTarget{Model: "model", ExtraParams: tc.extraParams},
})
if err == nil {
t.Fatal("expected invalid request error")
}
if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected ErrInvalidRequest, got %v", err)
}
if !strings.Contains(err.Error(), tc.want) {
t.Fatalf("expected error to contain %q, got %v", tc.want, err)
}
if called {
t.Fatal("provider should not be called for invalid extra_params")
}
})
}
}
func TestOpenAICompatibleClientNoAuthorizationHeaderWhenNoAPIKey(t *testing.T) {

View File

@@ -10,6 +10,7 @@ import (
"strings"
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
"gitea.maximumdirect.net/eric/scriptorium/internal/filecatalog"
"gopkg.in/yaml.v3"
)
@@ -33,40 +34,40 @@ func (r *filesystemRepository) GetProfile(ctx context.Context, id string) (*doma
return nil, fmt.Errorf("%w: profile id is required", ErrInvalidProfile)
}
files, err := os.ReadDir(r.dir)
files, err := filecatalog.FindYAMLFiles(ctx, r.dir)
if err != nil {
return nil, fmt.Errorf("failed to read profile directory: %w", err)
}
for _, file := range files {
var matches []profileMatch
for _, fullPath := range files {
select {
case <-ctx.Done():
return nil, ctx.Err()
default:
}
if file.IsDir() || (!strings.HasSuffix(file.Name(), ".yaml") && !strings.HasSuffix(file.Name(), ".yml")) {
continue
}
fullPath := filepath.Join(r.dir, file.Name())
relPath := filecatalog.RelativePath(r.dir, fullPath)
fileMatch := filecatalog.Stem(filepath.Base(fullPath)) == id
data, err := os.ReadFile(fullPath)
if err != nil {
return nil, fmt.Errorf("failed to read profile file %s: %w", file.Name(), err)
return nil, fmt.Errorf("failed to read profile file %s: %w", relPath, err)
}
metadata := readProfileFileMetadata(data)
idMatch := fileMatch || metadata.id == id
if metadata.hasRawAPIKey {
if idMatch {
return nil, fmt.Errorf("%w: %s", ErrRawAPIKeyNotAllowed, relPath)
}
continue
}
var prof domain.ExecutionProfile
decoder := yaml.NewDecoder(bytes.NewReader(data))
decoder.KnownFields(true)
if err := decoder.Decode(&prof); err != nil {
if strings.Contains(err.Error(), "field api_key not found") {
if strings.TrimSuffix(strings.TrimSuffix(file.Name(), ".yaml"), ".yml") == id {
return nil, fmt.Errorf("%w: %s", ErrRawAPIKeyNotAllowed, file.Name())
}
continue
}
if strings.TrimSuffix(strings.TrimSuffix(file.Name(), ".yaml"), ".yml") == id {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, file.Name(), err)
if idMatch {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
}
continue
}
@@ -76,16 +77,68 @@ func (r *filesystemRepository) GetProfile(ctx context.Context, id string) (*doma
}
if err := validateProfile(&prof); err != nil {
if errors.Is(err, ErrRawAPIKeyNotAllowed) {
return nil, fmt.Errorf("%w: %s", err, file.Name())
return nil, fmt.Errorf("%w: %s", err, relPath)
}
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, file.Name(), err)
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, relPath, err)
}
return &prof, nil
matches = append(matches, profileMatch{
profile: &prof,
path: relPath,
})
}
if len(matches) > 1 {
paths := make([]string, 0, len(matches))
for _, match := range matches {
paths = append(paths, match.path)
}
return nil, fmt.Errorf("%w: duplicate execution profile id %q found in: %s", ErrInvalidProfile, id, strings.Join(paths, ", "))
}
if len(matches) == 1 {
return matches[0].profile, nil
}
return nil, ErrProfileNotFound
}
type profileMatch struct {
profile *domain.ExecutionProfile
path string
}
type profileFileMetadata struct {
id string
hasRawAPIKey bool
}
func readProfileFileMetadata(data []byte) profileFileMetadata {
var node yaml.Node
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&node); err != nil {
return profileFileMetadata{}
}
if node.Kind != yaml.DocumentNode || len(node.Content) == 0 {
return profileFileMetadata{}
}
mapping := node.Content[0]
if mapping.Kind != yaml.MappingNode {
return profileFileMetadata{}
}
var metadata profileFileMetadata
for i := 0; i+1 < len(mapping.Content); i += 2 {
key := mapping.Content[i]
value := mapping.Content[i+1]
switch key.Value {
case "id":
metadata.id = strings.TrimSpace(value.Value)
case "api_key":
metadata.hasRawAPIKey = true
}
}
return metadata
}
func validateProfile(p *domain.ExecutionProfile) error {
if strings.TrimSpace(p.ID) == "" {
return errors.New("id is required")

View File

@@ -2,9 +2,11 @@ package profile
import (
"context"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
"testing"
)
@@ -58,6 +60,149 @@ func TestFilesystemRepository_GetProfile(t *testing.T) {
if p.ReasoningEffort != "medium" {
t.Fatalf("unexpected reasoning_effort: %q", p.ReasoningEffort)
}
if p.ServiceTier != "priority" {
t.Fatalf("unexpected service_tier: %q", p.ServiceTier)
}
})
t.Run("valid nested profile", func(t *testing.T) {
nestedDir := filepath.Join(tmpDir, "local")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writeProfileTestFile(t, filepath.Join(nestedDir, "nested-local.yaml"), `
id: nested-local
endpoint: http://localhost:8000/v1
model: nested-model
temperature: 0.1
`)
p, err := repo.GetProfile(ctx, "nested-local")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if p.Model != "nested-model" {
t.Fatalf("unexpected model: %q", p.Model)
}
})
t.Run("valid profile with JSON-compatible extra params", func(t *testing.T) {
writeProfileTestFile(t, filepath.Join(tmpDir, "json-extra-params.yaml"), `
id: json-extra-params
endpoint: http://localhost:8000/v1
model: nested-model
extra_params:
string_value: enabled
number_value: 42
boolean_value: true
object_value:
nested: value
count: 2
array_value:
- first
- 3
- false
`)
p, err := repo.GetProfile(ctx, "json-extra-params")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
var got map[string]any
encoded, err := json.Marshal(p.ExtraParams)
if err != nil {
t.Fatalf("expected extra_params to marshal as JSON, got %v", err)
}
if err := json.Unmarshal(encoded, &got); err != nil {
t.Fatalf("expected extra_params JSON to decode, got %v", err)
}
if got["string_value"] != "enabled" {
t.Fatalf("unexpected string extra param: %#v", got["string_value"])
}
if got["number_value"] != float64(42) {
t.Fatalf("unexpected number extra param: %#v", got["number_value"])
}
if got["boolean_value"] != true {
t.Fatalf("unexpected boolean extra param: %#v", got["boolean_value"])
}
objectValue, ok := got["object_value"].(map[string]any)
if !ok {
t.Fatalf("expected object extra param, got %#v", got["object_value"])
}
if objectValue["nested"] != "value" || objectValue["count"] != float64(2) {
t.Fatalf("unexpected object extra param: %#v", objectValue)
}
arrayValue, ok := got["array_value"].([]any)
if !ok {
t.Fatalf("expected array extra param, got %#v", got["array_value"])
}
if len(arrayValue) != 3 || arrayValue[0] != "first" || arrayValue[1] != float64(3) || arrayValue[2] != false {
t.Fatalf("unexpected array extra param: %#v", arrayValue)
}
})
t.Run("duplicate profile IDs fail as ambiguous", func(t *testing.T) {
writeProfileTestFile(t, filepath.Join(tmpDir, "duplicate-profile-a.yaml"), `
id: duplicate-profile
endpoint: http://localhost:8000/v1
model: first-model
`)
nestedDir := filepath.Join(tmpDir, "duplicates")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writeProfileTestFile(t, filepath.Join(nestedDir, "duplicate-profile-b.yaml"), `
id: duplicate-profile
endpoint: http://localhost:8000/v1
model: second-model
`)
_, err := repo.GetProfile(ctx, "duplicate-profile")
if !errors.Is(err, ErrInvalidProfile) {
t.Fatalf("expected duplicate profile to return ErrInvalidProfile, got %v", err)
}
for _, want := range []string{"duplicate execution profile id", "duplicate-profile-a.yaml", filepath.Join("duplicates", "duplicate-profile-b.yaml")} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("expected error to contain %q, got %v", want, err)
}
}
})
t.Run("nested raw api_key rejected for likely target file", func(t *testing.T) {
nestedDir := filepath.Join(tmpDir, "secure")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writeProfileTestFile(t, filepath.Join(nestedDir, "not_named_like_id.yaml"), `
id: nested_raw_api_key
endpoint: http://localhost:8000/v1
model: m
api_key: secret
`)
_, err := repo.GetProfile(ctx, "nested_raw_api_key")
if !errors.Is(err, ErrRawAPIKeyNotAllowed) {
t.Fatalf("expected ErrRawAPIKeyNotAllowed, got %v", err)
}
if !strings.Contains(err.Error(), filepath.Join("secure", "not_named_like_id.yaml")) {
t.Fatalf("expected nested path in error, got %v", err)
}
})
t.Run("raw api_key in non-target profile is ignored", func(t *testing.T) {
writeProfileTestFile(t, filepath.Join(tmpDir, "raw-api-key-non-target.yaml"), `
id: raw-api-key-non-target
endpoint: http://localhost:8000/v1
model: m
api_key: secret
`)
_, err := repo.GetProfile(ctx, "does-not-exist-with-raw-key-nearby")
if !errors.Is(err, ErrProfileNotFound) {
t.Fatalf("expected ErrProfileNotFound for non-target raw api_key file, got %v", err)
}
})
t.Run("invalid yaml", func(t *testing.T) {
@@ -109,3 +254,10 @@ func TestFilesystemRepository_GetProfile(t *testing.T) {
}
})
}
func writeProfileTestFile(t *testing.T, path string, content string) {
t.Helper()
if err := os.WriteFile(path, []byte(strings.TrimLeft(content, "\n")), 0o644); err != nil {
t.Fatalf("failed to write profile test file %q: %v", path, err)
}
}

View File

@@ -2,6 +2,7 @@ id: local-secure
endpoint: http://localhost:8000/v1
model: gpt-4o-mini
api_key_env: SCRIPTORIUM_API_KEY
service_tier: priority
reasoning_effort: medium
extra_params:
provider: local

View File

@@ -6,7 +6,9 @@ import (
"errors"
"fmt"
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
"strings"
"text/template"
"unicode/utf8"
)
var (
@@ -50,6 +52,11 @@ func (r *goRenderer) Render(ctx context.Context, definition *domain.PromptDefini
},
}
sessionID, err := renderSessionID(definition.SessionID, funcs, vars)
if err != nil {
return nil, err
}
var renderedMessages []domain.RenderedMessage
for i, tmplMsg := range definition.Templates {
@@ -75,12 +82,44 @@ func (r *goRenderer) Render(ctx context.Context, definition *domain.PromptDefini
}
renderedMessages = append(renderedMessages, domain.RenderedMessage{
Role: tmplMsg.Role,
Content: buf.String(),
Role: tmplMsg.Role,
Content: buf.String(),
CacheControl: cloneCacheControl(tmplMsg.CacheControl),
})
}
return &domain.RenderedPrompt{
Messages: renderedMessages,
SessionID: sessionID,
Messages: renderedMessages,
}, nil
}
func renderSessionID(raw string, funcs template.FuncMap, vars map[string]string) (string, error) {
if strings.TrimSpace(raw) == "" {
return "", nil
}
tmpl, err := template.New("session_id").Funcs(funcs).Option("missingkey=error").Parse(raw)
if err != nil {
return "", fmt.Errorf("%w: session_id: %v", ErrInvalidTemplate, err)
}
var buf bytes.Buffer
if err := tmpl.Execute(&buf, vars); err != nil {
return "", fmt.Errorf("%w: session_id: %w", ErrRenderFailure, err)
}
sessionID := strings.TrimSpace(buf.String())
if n := utf8.RuneCountInString(sessionID); n > domain.SessionIDMaxLength {
return "", fmt.Errorf("%w: session_id length %d exceeds maximum %d", ErrRenderFailure, n, domain.SessionIDMaxLength)
}
return sessionID, nil
}
func cloneCacheControl(in *domain.CacheControl) *domain.CacheControl {
if in == nil {
return nil
}
out := *in
return &out
}

View File

@@ -3,6 +3,7 @@ package prompt
import (
"context"
"errors"
"strings"
"testing"
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
@@ -78,6 +79,66 @@ func TestGoRenderer_Render(t *testing.T) {
}
})
t.Run("copying cache control to rendered messages", func(t *testing.T) {
def := &domain.PromptDefinition{
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
Templates: []domain.PromptMessageTemplate{
{
Role: "system",
Content: "You are concise.",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "Analyze this: {{input \"transcript\"}}"},
},
}
res, err := renderer.Render(ctx, def, inputs, vars)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(res.Messages) != 2 {
t.Fatalf("expected 2 messages, got %d", len(res.Messages))
}
if res.Messages[0].CacheControl == nil {
t.Fatal("expected rendered cache control")
}
if res.Messages[0].CacheControl.Type != domain.CacheControlEphemeral {
t.Fatalf("unexpected cache control type: %q", res.Messages[0].CacheControl.Type)
}
if res.Messages[0].CacheControl.TTL != "1h" {
t.Fatalf("unexpected cache control ttl: %q", res.Messages[0].CacheControl.TTL)
}
if res.Messages[1].CacheControl != nil {
t.Fatalf("expected no cache control on second message, got %#v", res.Messages[1].CacheControl)
}
})
t.Run("rendered cache control does not alias source template", func(t *testing.T) {
source := &domain.CacheControl{Type: domain.CacheControlEphemeral, TTL: "1h"}
def := &domain.PromptDefinition{
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
Templates: []domain.PromptMessageTemplate{
{Role: "system", Content: "You are concise.", CacheControl: source},
},
}
res, err := renderer.Render(ctx, def, inputs, vars)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if res.Messages[0].CacheControl == source {
t.Fatal("expected rendered cache control to be cloned")
}
res.Messages[0].CacheControl.TTL = ""
if source.TTL != "1h" {
t.Fatalf("source cache control was mutated, ttl=%q", source.TTL)
}
})
t.Run("accessing vars", func(t *testing.T) {
def := &domain.PromptDefinition{
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
@@ -95,6 +156,78 @@ func TestGoRenderer_Render(t *testing.T) {
}
})
t.Run("rendering session id from vars", func(t *testing.T) {
def := &domain.PromptDefinition{
SessionID: " {{ .session_id }} ",
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
Templates: []domain.PromptMessageTemplate{
{Role: "system", Content: "Speak in a {{.tone}} tone."},
},
}
res, err := renderer.Render(ctx, def, inputs, map[string]string{
"tone": "concise",
"session_id": "agent-session-123",
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if res.SessionID != "agent-session-123" {
t.Fatalf("unexpected session id: %q", res.SessionID)
}
})
t.Run("empty rendered session id is omitted", func(t *testing.T) {
def := &domain.PromptDefinition{
SessionID: " ",
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
Templates: []domain.PromptMessageTemplate{
{Role: "system", Content: "Speak in a {{.tone}} tone."},
},
}
res, err := renderer.Render(ctx, def, inputs, vars)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if res.SessionID != "" {
t.Fatalf("expected empty session id, got %q", res.SessionID)
}
})
t.Run("missing session id var fails rendering", func(t *testing.T) {
def := &domain.PromptDefinition{
SessionID: "{{ .session_id }}",
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
Templates: []domain.PromptMessageTemplate{
{Role: "system", Content: "Speak in a {{.tone}} tone."},
},
}
_, err := renderer.Render(ctx, def, inputs, vars)
if !errors.Is(err, ErrRenderFailure) {
t.Fatalf("expected ErrRenderFailure, got %v", err)
}
})
t.Run("too long rendered session id fails rendering", func(t *testing.T) {
def := &domain.PromptDefinition{
SessionID: "{{ .session_id }}",
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
Templates: []domain.PromptMessageTemplate{
{Role: "system", Content: "Speak in a {{.tone}} tone."},
},
}
_, err := renderer.Render(ctx, def, inputs, map[string]string{
"tone": "concise",
"session_id": strings.Repeat("x", domain.SessionIDMaxLength+1),
})
if !errors.Is(err, ErrRenderFailure) {
t.Fatalf("expected ErrRenderFailure, got %v", err)
}
})
t.Run("inserting required input artifact", func(t *testing.T) {
def := &domain.PromptDefinition{
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},

View File

@@ -10,6 +10,7 @@ import (
"strings"
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
"gitea.maximumdirect.net/eric/scriptorium/internal/filecatalog"
"gopkg.in/yaml.v3"
)
@@ -28,6 +29,7 @@ type promptDefinitionFile struct {
Version string `yaml:"version"`
DefaultProfile *string `yaml:"default_profile"`
Description string `yaml:"description"`
SessionID string `yaml:"session_id"`
Inputs []promptInputFile `yaml:"inputs"`
Messages []promptMessageFile `yaml:"messages"`
Output promptOutputContractFile `yaml:"output"`
@@ -41,9 +43,15 @@ type promptInputFile struct {
}
type promptMessageFile struct {
Role string `yaml:"role"`
Content string `yaml:"content"`
ContentFile string `yaml:"content_file"`
Role string `yaml:"role"`
Content string `yaml:"content"`
ContentFile string `yaml:"content_file"`
CacheControl *cacheControlFile `yaml:"cache_control"`
}
type cacheControlFile struct {
Type string `yaml:"type"`
TTL string `yaml:"ttl"`
}
type promptOutputContractFile struct {
@@ -62,29 +70,26 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition)
}
files, err := os.ReadDir(r.dir)
files, err := filecatalog.FindYAMLFiles(ctx, r.dir)
if err != nil {
return nil, fmt.Errorf("failed to read prompt definition directory: %w", err)
}
for _, file := range files {
var matches []promptDefinitionMatch
for _, fullPath := range files {
select {
case <-ctx.Done():
return nil, ctx.Err()
default:
}
if file.IsDir() || !isYAMLFile(file.Name()) {
continue
}
fullPath := filepath.Join(r.dir, file.Name())
fileMatch := promptIDFromFileName(file.Name()) == id
relPath := filecatalog.RelativePath(r.dir, fullPath)
fileMatch := filecatalog.Stem(filepath.Base(fullPath)) == id
raw, err := loadPromptDefinitionFile(fullPath)
if err != nil {
if fileMatch {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, file.Name(), err)
if fileMatch || promptDefinitionFileHasID(fullPath, id) {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
}
continue
}
@@ -92,7 +97,7 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
def, err := normalizePromptDefinition(raw, fullPath)
if err != nil {
if fileMatch || strings.TrimSpace(raw.ID) == id {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, file.Name(), err)
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, relPath, err)
}
continue
}
@@ -103,12 +108,35 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
if version != "" && def.Version != version {
continue
}
return def, nil
matches = append(matches, promptDefinitionMatch{
def: def,
path: relPath,
})
}
if len(matches) > 1 {
paths := make([]string, 0, len(matches))
for _, match := range matches {
paths = append(paths, match.path)
}
if version != "" {
return nil, fmt.Errorf("%w: duplicate prompt definition id %q version %q found in: %s", ErrInvalidPromptDefinition, id, version, strings.Join(paths, ", "))
}
return nil, fmt.Errorf("%w: duplicate prompt definition id %q found in: %s", ErrInvalidPromptDefinition, id, strings.Join(paths, ", "))
}
if len(matches) == 1 {
return matches[0].def, nil
}
return nil, ErrPromptDefinitionNotFound
}
type promptDefinitionMatch struct {
def *domain.PromptDefinition
path string
}
func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
data, err := os.ReadFile(path)
if err != nil {
@@ -124,6 +152,20 @@ func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
return &raw, nil
}
func promptDefinitionFileHasID(path string, id string) bool {
data, err := os.ReadFile(path)
if err != nil {
return false
}
var raw struct {
ID string `yaml:"id"`
}
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&raw); err != nil {
return false
}
return strings.TrimSpace(raw.ID) == id
}
func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*domain.PromptDefinition, error) {
if raw == nil {
return nil, errors.New("prompt definition is nil")
@@ -177,6 +219,11 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
return nil, fmt.Errorf("message %d (%s) must set exactly one of content or content_file", i, role)
}
cacheControl, err := normalizeCacheControl(msg.CacheControl)
if err != nil {
return nil, fmt.Errorf("message %d (%s) cache_control: %w", i, role, err)
}
templateContent := msg.Content
resolvedContentFile := ""
if hasContentFile {
@@ -195,9 +242,10 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
}
templates = append(templates, domain.PromptMessageTemplate{
Role: role,
Content: templateContent,
ContentFile: resolvedContentFile,
Role: role,
Content: templateContent,
ContentFile: resolvedContentFile,
CacheControl: cacheControl,
})
}
@@ -227,6 +275,7 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
Version: version,
DefaultProfile: defaultProfile,
Description: strings.TrimSpace(raw.Description),
SessionID: strings.TrimSpace(raw.SessionID),
Inputs: inputs,
Templates: templates,
OutputFormat: raw.Output.Format,
@@ -239,14 +288,28 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
}, nil
}
func isYAMLFile(name string) bool {
return strings.HasSuffix(name, ".yaml") || strings.HasSuffix(name, ".yml")
}
func normalizeCacheControl(raw *cacheControlFile) (*domain.CacheControl, error) {
if raw == nil {
return nil, nil
}
func promptIDFromFileName(name string) string {
name = strings.TrimSuffix(name, ".yaml")
name = strings.TrimSuffix(name, ".yml")
return name
cacheType := strings.TrimSpace(raw.Type)
if cacheType == "" {
return nil, errors.New("type is required")
}
if domain.CacheControlType(cacheType) != domain.CacheControlEphemeral {
return nil, fmt.Errorf("unsupported type %q", cacheType)
}
ttl := strings.TrimSpace(raw.TTL)
if ttl != "" && ttl != "1h" {
return nil, fmt.Errorf("unsupported ttl %q", ttl)
}
return &domain.CacheControl{
Type: domain.CacheControlType(cacheType),
TTL: ttl,
}, nil
}
func isValidOutputFormat(f domain.OutputFormat) bool {

View File

@@ -68,6 +68,77 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
}
})
t.Run("valid cache control with ttl", func(t *testing.T) {
p, err := repo.GetPromptDefinition(ctx, "valid-cache-control-ttl", "")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if len(p.Templates) != 2 {
t.Fatalf("expected 2 messages, got %d", len(p.Templates))
}
assertCacheControl(t, p.Templates[0].CacheControl, domain.CacheControlEphemeral, "1h")
if p.Templates[1].CacheControl != nil {
t.Fatalf("expected second message cache control to be nil, got %#v", p.Templates[1].CacheControl)
}
})
t.Run("valid cache control without ttl", func(t *testing.T) {
p, err := repo.GetPromptDefinition(ctx, "valid-cache-control-without-ttl", "")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if len(p.Templates) != 2 {
t.Fatalf("expected 2 messages, got %d", len(p.Templates))
}
assertCacheControl(t, p.Templates[0].CacheControl, domain.CacheControlEphemeral, "")
if p.Templates[1].CacheControl != nil {
t.Fatalf("expected second message cache control to be nil, got %#v", p.Templates[1].CacheControl)
}
})
t.Run("valid session id template", func(t *testing.T) {
p, err := repo.GetPromptDefinition(ctx, "valid-session-id", "")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if p.SessionID != "{{ .session_id }}" {
t.Fatalf("expected trimmed session_id template, got %q", p.SessionID)
}
})
t.Run("valid nested file-backed prompt resolves content file relative to nested YAML", func(t *testing.T) {
nestedDir := filepath.Join(tmpDir, "dnd", "recap")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writePromptTestFile(t, filepath.Join(nestedDir, "nested_recap.yaml"), `
id: nested-recap
version: "1.0.0"
messages:
- role: user
content_file: ./nested_recap.user.tmpl
output:
format: markdown
validation_mode: basic
repair_attempts: 0
`)
writePromptTestFile(t, filepath.Join(nestedDir, "nested_recap.user.tmpl"), `Nested recap: {{input "transcript"}}`)
p, err := repo.GetPromptDefinition(ctx, "nested-recap", "")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if len(p.Templates) != 1 {
t.Fatalf("expected one template, got %d", len(p.Templates))
}
if !strings.Contains(p.Templates[0].Content, "Nested recap") {
t.Fatalf("expected nested content file body, got %q", p.Templates[0].Content)
}
if !strings.Contains(p.Templates[0].ContentFile, filepath.Join("dnd", "recap", "nested_recap.user.tmpl")) {
t.Fatalf("expected nested content file path, got %q", p.Templates[0].ContentFile)
}
})
t.Run("prompt with default_profile", func(t *testing.T) {
p, err := repo.GetPromptDefinition(ctx, "with-default-profile", "")
if err != nil {
@@ -84,6 +155,124 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
}
})
t.Run("duplicate prompt IDs fail as ambiguous", func(t *testing.T) {
writePromptTestFile(t, filepath.Join(tmpDir, "duplicate_a.yaml"), `
id: duplicate-prompt
version: "1.0.0"
messages:
- role: user
content: First duplicate.
output:
format: markdown
validation_mode: basic
repair_attempts: 0
`)
nestedDir := filepath.Join(tmpDir, "nested")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writePromptTestFile(t, filepath.Join(nestedDir, "duplicate_b.yaml"), `
id: duplicate-prompt
version: "2.0.0"
messages:
- role: user
content: Second duplicate.
output:
format: markdown
validation_mode: basic
repair_attempts: 0
`)
_, err := repo.GetPromptDefinition(ctx, "duplicate-prompt", "")
if !errors.Is(err, ErrInvalidPromptDefinition) {
t.Fatalf("expected duplicate prompt to return ErrInvalidPromptDefinition, got %v", err)
}
for _, want := range []string{"duplicate prompt definition id", "duplicate_a.yaml", filepath.Join("nested", "duplicate_b.yaml")} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("expected error to contain %q, got %v", want, err)
}
}
})
t.Run("duplicate prompt ID and requested version fails as ambiguous", func(t *testing.T) {
writePromptTestFile(t, filepath.Join(tmpDir, "version_duplicate_a.yaml"), `
id: duplicate-version-prompt
version: "1.0.0"
messages:
- role: user
content: First duplicate version.
output:
format: markdown
validation_mode: basic
repair_attempts: 0
`)
nestedDir := filepath.Join(tmpDir, "versioned")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writePromptTestFile(t, filepath.Join(nestedDir, "version_duplicate_b.yaml"), `
id: duplicate-version-prompt
version: "1.0.0"
messages:
- role: user
content: Second duplicate version.
output:
format: markdown
validation_mode: basic
repair_attempts: 0
`)
_, err := repo.GetPromptDefinition(ctx, "duplicate-version-prompt", "1.0.0")
if !errors.Is(err, ErrInvalidPromptDefinition) {
t.Fatalf("expected duplicate prompt version to return ErrInvalidPromptDefinition, got %v", err)
}
for _, want := range []string{"duplicate prompt definition id", "version \"1.0.0\"", "version_duplicate_a.yaml", filepath.Join("versioned", "version_duplicate_b.yaml")} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("expected error to contain %q, got %v", want, err)
}
}
})
t.Run("non-matching malformed nested prompt is ignored for not found lookup", func(t *testing.T) {
nestedDir := filepath.Join(tmpDir, "broken")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writePromptTestFile(t, filepath.Join(nestedDir, "unrelated.yaml"), "id: [")
_, err := repo.GetPromptDefinition(ctx, "does-not-exist-even-with-broken-nested-file", "")
if !errors.Is(err, ErrPromptDefinitionNotFound) {
t.Fatalf("expected ErrPromptDefinitionNotFound, got %v", err)
}
})
t.Run("strict decode failure in nested prompt matches by YAML ID", func(t *testing.T) {
nestedDir := filepath.Join(tmpDir, "strict")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writePromptTestFile(t, filepath.Join(nestedDir, "not_named_like_id.yaml"), `
id: nested-strict-error
version: "1.0.0"
unknown_field: true
messages:
- role: user
content: Invalid because of unknown field.
output:
format: markdown
validation_mode: basic
repair_attempts: 0
`)
_, err := repo.GetPromptDefinition(ctx, "nested-strict-error", "")
if !errors.Is(err, ErrInvalidYAML) {
t.Fatalf("expected ErrInvalidYAML, got %v", err)
}
if !strings.Contains(err.Error(), filepath.Join("strict", "not_named_like_id.yaml")) {
t.Fatalf("expected nested path in error, got %v", err)
}
})
t.Run("version lookup", func(t *testing.T) {
_, err := repo.GetPromptDefinition(ctx, "valid-inline", "9.9.9")
if !errors.Is(err, ErrPromptDefinitionNotFound) {
@@ -107,6 +296,10 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
{name: "invalid validation mode", id: "invalid_validation_mode", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"invalid validation mode"}},
{name: "json_schema without schema_path", id: "json_schema_without_schema_path", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"schema_path"}},
{name: "unknown input field", id: "unknown_input_field", targetErr: ErrInvalidYAML, errSubstrs: []string{"field unknown_input_setting not found"}},
{name: "empty cache control type", id: "empty_cache_control_type", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"cache_control", "type is required"}},
{name: "unsupported cache control type", id: "unsupported_cache_control_type", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"cache_control", "unsupported type"}},
{name: "unsupported cache control ttl", id: "unsupported_cache_control_ttl", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"cache_control", "unsupported ttl"}},
{name: "unknown cache control field", id: "unknown_cache_control_field", targetErr: ErrInvalidYAML, errSubstrs: []string{"field unexpected not found"}},
}
for _, tc := range cases {
@@ -131,6 +324,26 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
})
}
func assertCacheControl(t *testing.T, got *domain.CacheControl, wantType domain.CacheControlType, wantTTL string) {
t.Helper()
if got == nil {
t.Fatal("expected cache control, got nil")
}
if got.Type != wantType {
t.Fatalf("unexpected cache control type: got %q want %q", got.Type, wantType)
}
if got.TTL != wantTTL {
t.Fatalf("unexpected cache control ttl: got %q want %q", got.TTL, wantTTL)
}
}
func writePromptTestFile(t *testing.T, path string, content string) {
t.Helper()
if err := os.WriteFile(path, []byte(strings.TrimLeft(content, "\n")), 0o644); err != nil {
t.Fatalf("failed to write prompt test file %q: %v", path, err)
}
}
func copyTree(src, dst string) error {
return filepath.WalkDir(src, func(path string, d fs.DirEntry, err error) error {
if err != nil {

View File

@@ -0,0 +1,10 @@
id: empty-cache-control-type
version: "1.0.0"
messages:
- role: system
content: "Use cached instructions."
cache_control: {}
output:
format: markdown
validation_mode: basic
repair_attempts: 0

View File

@@ -0,0 +1,12 @@
id: unknown-cache-control-field
version: "1.0.0"
messages:
- role: system
content: "Use cached instructions."
cache_control:
type: ephemeral
unexpected: true
output:
format: markdown
validation_mode: basic
repair_attempts: 0

View File

@@ -0,0 +1,12 @@
id: unsupported-cache-control-ttl
version: "1.0.0"
messages:
- role: system
content: "Use cached instructions."
cache_control:
type: ephemeral
ttl: 5m
output:
format: markdown
validation_mode: basic
repair_attempts: 0

View File

@@ -0,0 +1,11 @@
id: unsupported-cache-control-type
version: "1.0.0"
messages:
- role: system
content: "Use cached instructions."
cache_control:
type: persistent
output:
format: markdown
validation_mode: basic
repair_attempts: 0

View File

@@ -0,0 +1,14 @@
id: valid-cache-control-ttl
version: "1.0.0"
messages:
- role: system
content: "Use cached instructions."
cache_control:
type: ephemeral
ttl: 1h
- role: user
content: "Summarize the input."
output:
format: markdown
validation_mode: basic
repair_attempts: 0

View File

@@ -0,0 +1,13 @@
id: valid-cache-control-without-ttl
version: "1.0.0"
messages:
- role: system
content: "Use cached instructions."
cache_control:
type: ephemeral
- role: user
content: "Summarize the input."
output:
format: markdown
validation_mode: basic
repair_attempts: 0

View File

@@ -0,0 +1,10 @@
id: valid-session-id
version: "1.0.0"
session_id: " {{ .session_id }} "
messages:
- role: user
content: Hello.
output:
format: markdown
validation_mode: basic
repair_attempts: 0

View File

@@ -35,9 +35,9 @@ func TestRunnerIntegrationWithPromptAndProfileFixturesAndValidation(t *testing.T
t.Fatalf("failed to resolve repo root: %v", err)
}
promptsDir := filepath.Join(root, "prompts")
profilesDir := filepath.Join(root, "profiles")
schemasDir := filepath.Join(root, "schemas")
promptsDir := filepath.Join(root, "examples", "prompts")
profilesDir := filepath.Join(root, "examples", "profiles")
schemasDir := filepath.Join(root, "examples", "schemas")
fixturesDir := filepath.Join(root, "examples", "fixtures")
t.Setenv("SCRIPTORIUM_API_KEY", "test-key")

View File

@@ -24,12 +24,14 @@ import (
)
var (
ErrInvalidRequest = errors.New("invalid run request")
ErrProfileLoad = errors.New("failed to load prompt definition")
ErrArtifactLoad = errors.New("failed to load artifact")
ErrPromptRender = errors.New("failed to render prompt")
ErrLLMGenerate = errors.New("failed to generate output")
ErrValidation = errors.New("failed to validate output")
ErrInvalidRequest = errors.New("invalid run request")
ErrProfileRequired = errors.New("profile selection is required")
ErrAPIKeyEnvMissing = errors.New("api_key_env points to an unset environment variable")
ErrProfileLoad = errors.New("failed to load prompt definition")
ErrArtifactLoad = errors.New("failed to load artifact")
ErrPromptRender = errors.New("failed to render prompt")
ErrLLMGenerate = errors.New("failed to generate output")
ErrValidation = errors.New("failed to validate output")
)
// Runner executes the Scriptorium core use case.
@@ -88,11 +90,15 @@ func (r *Runner) Run(ctx context.Context, req domain.RunRequest) (*domain.RunRes
}
genResp, err := r.llm.Generate(ctx, domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: prepared.Messages},
Prompt: domain.RenderedPrompt{SessionID: prepared.SessionID, Messages: prepared.Messages},
Target: prepared.EffectiveModelParams,
TargetPresence: prepared.TargetPresence,
StructuredOutput: prepared.StructuredOutput,
})
if err != nil {
if errors.Is(err, llm.ErrInvalidRequest) {
return nil, fmt.Errorf("%w: %w", ErrInvalidRequest, err)
}
return nil, fmt.Errorf("%w: %w", ErrLLMGenerate, err)
}
@@ -177,7 +183,7 @@ func (r *Runner) Prepare(ctx context.Context, req domain.RunRequest) (*domain.Pr
selectedProfileID = strings.TrimSpace(def.DefaultProfile)
}
if selectedProfileID == "" {
return nil, fmt.Errorf("%w: profile id is required either in request or prompt default_profile", ErrInvalidRequest)
return nil, fmt.Errorf("%w: %w: profile id is required either in request or prompt default_profile", ErrInvalidRequest, ErrProfileRequired)
}
execProfile, err := r.profiles.GetProfile(ctx, selectedProfileID)
@@ -185,7 +191,10 @@ func (r *Runner) Prepare(ctx context.Context, req domain.RunRequest) (*domain.Pr
return nil, fmt.Errorf("%w: %w", ErrProfileLoad, err)
}
effectiveModel := resolveExecutionTarget(execProfile, req.Execution)
effectiveModel, targetPresence, err := resolveExecutionTarget(execProfile, req.Execution)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrInvalidRequest, err)
}
if strings.TrimSpace(effectiveModel.Endpoint) == "" {
return nil, fmt.Errorf("%w: execution endpoint is required", ErrInvalidRequest)
}
@@ -228,9 +237,11 @@ func (r *Runner) Prepare(ctx context.Context, req domain.RunRequest) (*domain.Pr
PromptHash: promptDefinitionHash,
SelectedProfileID: selectedProfileID,
EffectiveModelParams: effectiveModel,
TargetPresence: targetPresence,
OutputContract: effectiveContract,
StructuredOutput: structuredOutput,
InputHashes: inputHashes,
SessionID: renderedPrompt.SessionID,
RenderedPromptHash: hashRenderedPrompt(*renderedPrompt),
Messages: renderedPrompt.Messages,
StartTime: start,
@@ -342,6 +353,9 @@ func mergeExecutionTarget(base domain.ExecutionTarget, override domain.Execution
if override.TimeoutSeconds != 0 {
out.TimeoutSeconds = override.TimeoutSeconds
}
if strings.TrimSpace(override.ServiceTier) != "" {
out.ServiceTier = override.ServiceTier
}
if strings.TrimSpace(override.ReasoningEffort) != "" {
out.ReasoningEffort = override.ReasoningEffort
}
@@ -349,22 +363,75 @@ func mergeExecutionTarget(base domain.ExecutionTarget, override domain.Execution
out.APIKeyEnv = override.APIKeyEnv
}
if len(override.ExtraParams) > 0 {
cp := make(map[string]string, len(override.ExtraParams))
for k, v := range override.ExtraParams {
cp[k] = v
}
out.ExtraParams = cp
out.ExtraParams = copyExtraParams(override.ExtraParams)
}
return out
}
func resolveExecutionTarget(profileValue *domain.ExecutionProfile, override *domain.ExecutionTarget) domain.ExecutionTarget {
func mergeExecutionTargetOverride(base domain.ExecutionTarget, override domain.ExecutionTargetOverride) (domain.ExecutionTarget, domain.ExecutionTargetPresence, error) {
out := base
var presence domain.ExecutionTargetPresence
if override.Endpoint != "" {
out.Endpoint = override.Endpoint
}
if override.Model != "" {
out.Model = override.Model
}
if override.Temperature != nil {
if *override.Temperature < 0 || *override.Temperature > 2 {
return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("temperature must be between 0 and 2")
}
out.Temperature = *override.Temperature
presence.Temperature = true
}
if override.MaxTokens != nil {
if *override.MaxTokens < 0 {
return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("max_tokens must be greater than or equal to 0")
}
out.MaxTokens = *override.MaxTokens
presence.MaxTokens = true
}
if override.TopP != nil {
if *override.TopP < 0 || *override.TopP > 1 {
return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("top_p must be between 0 and 1")
}
out.TopP = *override.TopP
presence.TopP = true
}
if override.TimeoutSeconds != nil {
if *override.TimeoutSeconds < 0 {
return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("timeout_seconds must be greater than or equal to 0")
}
out.TimeoutSeconds = *override.TimeoutSeconds
presence.TimeoutSeconds = true
}
if strings.TrimSpace(override.ServiceTier) != "" {
out.ServiceTier = override.ServiceTier
}
if strings.TrimSpace(override.ReasoningEffort) != "" {
out.ReasoningEffort = override.ReasoningEffort
}
if strings.TrimSpace(override.APIKeyEnv) != "" {
out.APIKeyEnv = override.APIKeyEnv
}
if len(override.ExtraParams) > 0 {
out.ExtraParams = copyExtraParams(override.ExtraParams)
}
return out, presence, nil
}
func resolveExecutionTarget(profileValue *domain.ExecutionProfile, override *domain.ExecutionTargetOverride) (domain.ExecutionTarget, domain.ExecutionTargetPresence, error) {
out := defaults.ExecutionTargetDefault()
out = mergeExecutionTarget(out, executionProfileToTarget(profileValue))
var presence domain.ExecutionTargetPresence
if override != nil {
out = mergeExecutionTarget(out, *override)
var err error
out, presence, err = mergeExecutionTargetOverride(out, *override)
if err != nil {
return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, err
}
}
return out
return out, presence, nil
}
func validateAPIKeyEnv(apiKeyEnv string) error {
@@ -373,7 +440,7 @@ func validateAPIKeyEnv(apiKeyEnv string) error {
return nil
}
if strings.TrimSpace(os.Getenv(envName)) == "" {
return fmt.Errorf("api key environment variable %q is not set", envName)
return fmt.Errorf("%w: api key environment variable %q is not set", ErrAPIKeyEnvMissing, envName)
}
return nil
}
@@ -382,13 +449,6 @@ func executionProfileToTarget(p *domain.ExecutionProfile) domain.ExecutionTarget
if p == nil {
return domain.ExecutionTarget{}
}
cp := map[string]string(nil)
if len(p.ExtraParams) > 0 {
cp = make(map[string]string, len(p.ExtraParams))
for k, v := range p.ExtraParams {
cp[k] = v
}
}
return domain.ExecutionTarget{
Endpoint: p.Endpoint,
Model: p.Model,
@@ -396,12 +456,24 @@ func executionProfileToTarget(p *domain.ExecutionProfile) domain.ExecutionTarget
MaxTokens: p.MaxTokens,
TopP: p.TopP,
TimeoutSeconds: p.TimeoutSeconds,
ServiceTier: p.ServiceTier,
ReasoningEffort: p.ReasoningEffort,
APIKeyEnv: p.APIKeyEnv,
ExtraParams: cp,
ExtraParams: copyExtraParams(p.ExtraParams),
}
}
func copyExtraParams(src map[string]any) map[string]any {
if len(src) == 0 {
return nil
}
cp := make(map[string]any, len(src))
for k, v := range src {
cp[k] = v
}
return cp
}
func resolveOutputContract(def *domain.PromptDefinition, override *domain.OutputContract) domain.OutputContract {
contract := def.Validation
if contract.Format == "" {
@@ -418,10 +490,23 @@ func resolveOutputContract(def *domain.PromptDefinition, override *domain.Output
func hashRenderedPrompt(p domain.RenderedPrompt) string {
var b strings.Builder
if p.SessionID != "" {
b.WriteString("session_id=")
b.WriteString(p.SessionID)
b.WriteString("\n---\n")
}
for _, msg := range p.Messages {
b.WriteString(msg.Role)
b.WriteByte('\n')
b.WriteString(msg.Content)
if msg.CacheControl != nil {
b.WriteString("\ncache_control.type=")
b.WriteString(string(msg.CacheControl.Type))
if msg.CacheControl.TTL != "" {
b.WriteString("\ncache_control.ttl=")
b.WriteString(msg.CacheControl.TTL)
}
}
b.WriteString("\n---\n")
}
h := sha256.Sum256([]byte(b.String()))

View File

@@ -14,6 +14,7 @@ import (
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
"gitea.maximumdirect.net/eric/scriptorium/internal/llm"
"gitea.maximumdirect.net/eric/scriptorium/internal/profile"
"gitea.maximumdirect.net/eric/scriptorium/internal/prompt"
"gitea.maximumdirect.net/eric/scriptorium/internal/promptdef"
@@ -160,7 +161,7 @@ func TestRunnerPrepareWithExplicitProfileSelection(t *testing.T) {
"a://t": {Body: []byte("transcript"), Hash: hashString("transcript")},
"a://g": {Body: []byte("glossary"), Hash: hashString("glossary")},
}}
renderer := &fakeRenderer{rendered: &domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "system", Content: "sys"}, {Role: "user", Content: "usr"}}}}
renderer := &fakeRenderer{rendered: &domain.RenderedPrompt{SessionID: "session-123", Messages: []domain.RenderedMessage{{Role: "system", Content: "sys"}, {Role: "user", Content: "usr"}}}}
llmClient := &fakeLLM{forbid: true}
runner := NewRunner(promptRepo, execRepo, reader, renderer, llmClient, nil)
@@ -172,7 +173,7 @@ func TestRunnerPrepareWithExplicitProfileSelection(t *testing.T) {
"transcript": {Type: domain.ArtifactRefFile, URI: "a://t"},
"glossary": {Type: domain.ArtifactRefFile, URI: "a://g"},
},
Execution: &domain.ExecutionTarget{Endpoint: "http://override/v1", Model: "m", Temperature: 0.3, TimeoutSeconds: 90},
Execution: &domain.ExecutionTargetOverride{Endpoint: "http://override/v1", Model: "m", Temperature: float64Ptr(0.3), TimeoutSeconds: intPtr(90)},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
@@ -198,6 +199,9 @@ func TestRunnerPrepareWithExplicitProfileSelection(t *testing.T) {
if len(prepared.Messages) != 2 {
t.Fatalf("expected two messages, got %d", len(prepared.Messages))
}
if prepared.SessionID != "session-123" {
t.Fatalf("expected prepared session id, got %q", prepared.SessionID)
}
if llmClient.calls != 0 {
t.Fatalf("prepare should not call llm, calls=%d", llmClient.calls)
}
@@ -234,6 +238,9 @@ func TestRunnerPrepareMissingExplicitProfileAndMissingDefaultProfileFails(t *tes
if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected ErrInvalidRequest, got %v", err)
}
if !errors.Is(err, ErrProfileRequired) {
t.Fatalf("expected ErrProfileRequired, got %v", err)
}
}
func TestRunnerPrepareSelectedProfileDoesNotExistFails(t *testing.T) {
@@ -257,6 +264,7 @@ func TestRunnerPrepareRuntimeOverrideBeatsSelectedProfileValue(t *testing.T) {
MaxTokens: 500,
TopP: 0.9,
TimeoutSeconds: 120,
ServiceTier: "priority",
},
}}
runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{forbid: true}, nil)
@@ -265,11 +273,12 @@ func TestRunnerPrepareRuntimeOverrideBeatsSelectedProfileValue(t *testing.T) {
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
Execution: &domain.ExecutionTarget{
Execution: &domain.ExecutionTargetOverride{
Endpoint: "http://override/v1",
Model: "override-model",
Temperature: 0.7,
TimeoutSeconds: 30,
Temperature: float64Ptr(0.7),
TimeoutSeconds: intPtr(30),
ServiceTier: "flex",
},
})
if err != nil {
@@ -281,6 +290,146 @@ func TestRunnerPrepareRuntimeOverrideBeatsSelectedProfileValue(t *testing.T) {
if prepared.EffectiveModelParams.TopP != 0.9 {
t.Fatalf("expected profile top_p to remain, got %v", prepared.EffectiveModelParams.TopP)
}
if prepared.EffectiveModelParams.ServiceTier != "flex" {
t.Fatalf("expected service_tier override to win, got %q", prepared.EffectiveModelParams.ServiceTier)
}
}
func TestRunnerPrepareRequestNumericOverridePresence(t *testing.T) {
tests := []struct {
name string
override *domain.ExecutionTargetOverride
wantTemperature float64
wantMaxTokens int
wantTopP float64
wantTimeoutSecs int
wantPresence domain.ExecutionTargetPresence
}{
{
name: "omitted preserves profile values",
override: &domain.ExecutionTargetOverride{},
wantTemperature: 0.7,
wantMaxTokens: 321,
wantTopP: 0.8,
wantTimeoutSecs: 45,
},
{
name: "explicit zero temperature",
override: &domain.ExecutionTargetOverride{Temperature: float64Ptr(0)},
wantTemperature: 0,
wantMaxTokens: 321,
wantTopP: 0.8,
wantTimeoutSecs: 45,
wantPresence: domain.ExecutionTargetPresence{Temperature: true},
},
{
name: "explicit zero max tokens",
override: &domain.ExecutionTargetOverride{MaxTokens: intPtr(0)},
wantTemperature: 0.7,
wantMaxTokens: 0,
wantTopP: 0.8,
wantTimeoutSecs: 45,
wantPresence: domain.ExecutionTargetPresence{MaxTokens: true},
},
{
name: "explicit zero top p",
override: &domain.ExecutionTargetOverride{TopP: float64Ptr(0)},
wantTemperature: 0.7,
wantMaxTokens: 321,
wantTopP: 0,
wantTimeoutSecs: 45,
wantPresence: domain.ExecutionTargetPresence{TopP: true},
},
{
name: "explicit zero timeout",
override: &domain.ExecutionTargetOverride{TimeoutSeconds: intPtr(0)},
wantTemperature: 0.7,
wantMaxTokens: 321,
wantTopP: 0.8,
wantTimeoutSecs: 0,
wantPresence: domain.ExecutionTargetPresence{TimeoutSeconds: true},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
runner := NewRunner(
&fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{
"exec": {
ID: "exec",
Endpoint: "http://profile/v1",
Model: "profile-model",
Temperature: 0.7,
MaxTokens: 321,
TopP: 0.8,
TimeoutSeconds: 45,
},
}},
defaultArtifactReader(),
defaultRenderer(),
&fakeLLM{forbid: true},
nil,
)
prepared, err := runner.Prepare(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
Execution: tc.override,
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
got := prepared.EffectiveModelParams
if got.Temperature != tc.wantTemperature ||
got.MaxTokens != tc.wantMaxTokens ||
got.TopP != tc.wantTopP ||
got.TimeoutSeconds != tc.wantTimeoutSecs {
t.Fatalf("unexpected effective numeric settings: %+v", got)
}
if prepared.TargetPresence != tc.wantPresence {
t.Fatalf("unexpected target presence: got %+v want %+v", prepared.TargetPresence, tc.wantPresence)
}
})
}
}
func TestRunnerPrepareInvalidRequestNumericOverridesFail(t *testing.T) {
tests := []struct {
name string
override *domain.ExecutionTargetOverride
}{
{name: "temperature below range", override: &domain.ExecutionTargetOverride{Temperature: float64Ptr(-0.1)}},
{name: "temperature above range", override: &domain.ExecutionTargetOverride{Temperature: float64Ptr(2.1)}},
{name: "max tokens below range", override: &domain.ExecutionTargetOverride{MaxTokens: intPtr(-1)}},
{name: "top p below range", override: &domain.ExecutionTargetOverride{TopP: float64Ptr(-0.1)}},
{name: "top p above range", override: &domain.ExecutionTargetOverride{TopP: float64Ptr(1.1)}},
{name: "timeout below range", override: &domain.ExecutionTargetOverride{TimeoutSeconds: intPtr(-1)}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
runner := NewRunner(
&fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}},
defaultArtifactReader(),
defaultRenderer(),
&fakeLLM{forbid: true},
nil,
)
_, err := runner.Prepare(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
Execution: tc.override,
})
if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected ErrInvalidRequest, got %v", err)
}
})
}
}
func TestRunnerPrepareSelectedProfileBeatsBuiltInDefault(t *testing.T) {
@@ -292,6 +441,7 @@ func TestRunnerPrepareSelectedProfileBeatsBuiltInDefault(t *testing.T) {
Model: "profile-model",
TopP: 0.8,
TimeoutSeconds: 90,
ServiceTier: "priority",
},
}}
runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{forbid: true}, nil)
@@ -310,6 +460,9 @@ func TestRunnerPrepareSelectedProfileBeatsBuiltInDefault(t *testing.T) {
if prepared.EffectiveModelParams.TimeoutSeconds != 90 {
t.Fatalf("expected profile timeout to beat default, got %d", prepared.EffectiveModelParams.TimeoutSeconds)
}
if prepared.EffectiveModelParams.ServiceTier != "priority" {
t.Fatalf("expected profile service_tier to beat default, got %q", prepared.EffectiveModelParams.ServiceTier)
}
}
func TestRunnerPrepareFileBackedPromptBodiesRenderCorrectly(t *testing.T) {
@@ -478,6 +631,35 @@ func TestRunnerPrepareJSONSchemaBuildsStructuredOutputSpec(t *testing.T) {
}
}
func TestRunnerPrepareJSONSchemaSchemaLoadFailureReturnsValidationError(t *testing.T) {
def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 0)
def.Validation.SchemaPath = "missing.schema.json"
validator := &fakeValidator{schemaErr: errors.New("schema unavailable")}
runner := NewRunner(
&fakePromptRepo{def: def},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}},
defaultArtifactReader(),
defaultRenderer(),
&fakeLLM{forbid: true},
validator,
)
_, err := runner.Prepare(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
})
if !errors.Is(err, ErrValidation) {
t.Fatalf("expected ErrValidation, got %v", err)
}
if validator.schemaLoads != 1 {
t.Fatalf("expected one schema load attempt, got %d", validator.schemaLoads)
}
if validator.schemaLoadPath != "missing.schema.json" {
t.Fatalf("expected schema path missing.schema.json, got %q", validator.schemaLoadPath)
}
}
func TestRunnerRunJSONSchemaSchemaLoadFailureFailsBeforeLLM(t *testing.T) {
def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 0)
def.Validation.SchemaPath = "missing.schema.json"
@@ -536,6 +718,100 @@ func TestDeriveStructuredSchemaName(t *testing.T) {
}
}
func TestHashRenderedPromptIncludesCacheControlWhenPresent(t *testing.T) {
uncached := domain.RenderedPrompt{Messages: []domain.RenderedMessage{
{Role: "system", Content: "sys"},
{Role: "user", Content: "usr"},
}}
wantLegacyHash := hashString("system\nsys\n---\nuser\nusr\n---\n")
if got := hashRenderedPrompt(uncached); got != wantLegacyHash {
t.Fatalf("expected no-cache hash to preserve legacy input, got %q want %q", got, wantLegacyHash)
}
withCache := domain.RenderedPrompt{Messages: []domain.RenderedMessage{
{
Role: "system",
Content: "sys",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "usr"},
}}
alsoWithCache := domain.RenderedPrompt{Messages: []domain.RenderedMessage{
{
Role: "system",
Content: "sys",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "usr"},
}}
withoutTTL := domain.RenderedPrompt{Messages: []domain.RenderedMessage{
{
Role: "system",
Content: "sys",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
},
},
{Role: "user", Content: "usr"},
}}
cachedHash := hashRenderedPrompt(withCache)
if cachedHash == hashRenderedPrompt(uncached) {
t.Fatal("expected cache control to change rendered prompt hash")
}
if cachedHash != hashRenderedPrompt(alsoWithCache) {
t.Fatal("expected identical cache control metadata to produce stable hash")
}
if cachedHash == hashRenderedPrompt(withoutTTL) {
t.Fatal("expected ttl changes to affect rendered prompt hash")
}
}
func TestHashRenderedPromptIncludesSessionIDWhenPresent(t *testing.T) {
withoutSession := domain.RenderedPrompt{Messages: []domain.RenderedMessage{
{Role: "system", Content: "sys"},
{Role: "user", Content: "usr"},
}}
withSession := domain.RenderedPrompt{
SessionID: "session-123",
Messages: []domain.RenderedMessage{
{Role: "system", Content: "sys"},
{Role: "user", Content: "usr"},
},
}
alsoWithSession := domain.RenderedPrompt{
SessionID: "session-123",
Messages: []domain.RenderedMessage{
{Role: "system", Content: "sys"},
{Role: "user", Content: "usr"},
},
}
otherSession := domain.RenderedPrompt{
SessionID: "session-456",
Messages: []domain.RenderedMessage{
{Role: "system", Content: "sys"},
{Role: "user", Content: "usr"},
},
}
sessionHash := hashRenderedPrompt(withSession)
if sessionHash == hashRenderedPrompt(withoutSession) {
t.Fatal("expected session_id to change rendered prompt hash")
}
if sessionHash != hashRenderedPrompt(alsoWithSession) {
t.Fatal("expected identical session_id to produce stable hash")
}
if sessionHash == hashRenderedPrompt(otherSession) {
t.Fatal("expected session_id value changes to affect rendered prompt hash")
}
}
func TestRunnerRunSuccessful(t *testing.T) {
promptRepo := &fakePromptRepo{def: promptDef(domain.FormatMarkdown, domain.ValidationBasic, 0)}
execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}
@@ -543,7 +819,7 @@ func TestRunnerRunSuccessful(t *testing.T) {
"a://t": {Body: []byte("transcript"), Hash: hashString("transcript")},
"a://g": {Body: []byte("glossary"), Hash: hashString("glossary")},
}}
renderer := &fakeRenderer{rendered: &domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "system", Content: "sys"}, {Role: "user", Content: "usr"}}}}
renderer := &fakeRenderer{rendered: &domain.RenderedPrompt{SessionID: "session-123", Messages: []domain.RenderedMessage{{Role: "system", Content: "sys"}, {Role: "user", Content: "usr"}}}}
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "# recap", Usage: domain.TokenUsage{TotalTokens: 7}}}
runner := NewRunner(promptRepo, execRepo, reader, renderer, llmClient, nil)
@@ -555,7 +831,7 @@ func TestRunnerRunSuccessful(t *testing.T) {
"transcript": {Type: domain.ArtifactRefFile, URI: "a://t"},
"glossary": {Type: domain.ArtifactRefFile, URI: "a://g"},
},
Execution: &domain.ExecutionTarget{Endpoint: "http://override/v1", Model: "m", Temperature: 0.3, TimeoutSeconds: 90},
Execution: &domain.ExecutionTargetOverride{Endpoint: "http://override/v1", Model: "m", Temperature: float64Ptr(0.3), TimeoutSeconds: intPtr(90)},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
@@ -590,6 +866,48 @@ func TestRunnerRunSuccessful(t *testing.T) {
if llmClient.lastReq.Target.TimeoutSeconds != 90 {
t.Fatalf("expected timeout propagation, got %d", llmClient.lastReq.Target.TimeoutSeconds)
}
if !llmClient.lastReq.TargetPresence.Temperature || !llmClient.lastReq.TargetPresence.TimeoutSeconds {
t.Fatalf("expected numeric override presence to be sent to llm, got %+v", llmClient.lastReq.TargetPresence)
}
if llmClient.lastReq.Prompt.SessionID != "session-123" {
t.Fatalf("expected session id to be sent to llm, got %q", llmClient.lastReq.Prompt.SessionID)
}
}
func TestRunnerRunPassesExtraParamsToGenerateRequestTarget(t *testing.T) {
extraParams := map[string]any{
"string_value": "enabled",
"number_value": 42,
"boolean_value": true,
"object_value": map[string]any{"nested": "value"},
"array_value": []any{"first", 3, false},
}
promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)}
execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{
"exec": {
ID: "exec",
Endpoint: "http://profile/v1",
Model: "profile-model",
ExtraParams: extraParams,
},
}}
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}
runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), llmClient, nil)
res, err := runner.Run(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if !reflect.DeepEqual(res.EffectiveModelParams.ExtraParams, extraParams) {
t.Fatalf("expected run result extra_params to match profile values, got %#v", res.EffectiveModelParams.ExtraParams)
}
if !reflect.DeepEqual(llmClient.lastReq.Target.ExtraParams, extraParams) {
t.Fatalf("expected generate request extra_params to match profile values, got %#v", llmClient.lastReq.Target.ExtraParams)
}
}
func TestRunnerRunAndPrepareResolveSameProfileAndEffectiveSettings(t *testing.T) {
@@ -608,7 +926,7 @@ func TestRunnerRunAndPrepareResolveSameProfileAndEffectiveSettings(t *testing.T)
Inputs: map[string]domain.ArtifactRef{
"transcript": {Type: domain.ArtifactRefFile, URI: "a://t"},
},
Execution: &domain.ExecutionTarget{Endpoint: "http://override/v1", Model: "m", Temperature: 0.3, TimeoutSeconds: 90},
Execution: &domain.ExecutionTargetOverride{Endpoint: "http://override/v1", Model: "m", Temperature: float64Ptr(0.3), TimeoutSeconds: intPtr(90)},
}
prepared, err := runner.Prepare(context.Background(), req)
@@ -726,6 +1044,7 @@ func TestRunnerRunExplicitRuntimeOverrideBeatsSelectedProfileValue(t *testing.T)
MaxTokens: 500,
TopP: 0.9,
TimeoutSeconds: 120,
ServiceTier: "priority",
},
}}
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}
@@ -735,11 +1054,12 @@ func TestRunnerRunExplicitRuntimeOverrideBeatsSelectedProfileValue(t *testing.T)
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
Execution: &domain.ExecutionTarget{
Execution: &domain.ExecutionTargetOverride{
Endpoint: "http://override/v1",
Model: "override-model",
Temperature: 0.7,
TimeoutSeconds: 30,
Temperature: float64Ptr(0.7),
TimeoutSeconds: intPtr(30),
ServiceTier: "flex",
},
})
if err != nil {
@@ -754,6 +1074,9 @@ func TestRunnerRunExplicitRuntimeOverrideBeatsSelectedProfileValue(t *testing.T)
if res.EffectiveModelParams.TopP != 0.9 {
t.Fatalf("expected non-overridden profile top_p to remain, got %v", res.EffectiveModelParams.TopP)
}
if res.EffectiveModelParams.ServiceTier != "flex" {
t.Fatalf("expected service_tier override to win, got %q", res.EffectiveModelParams.ServiceTier)
}
}
func TestRunnerRunSelectedProfileBeatsBuiltInDefault(t *testing.T) {
@@ -765,6 +1088,7 @@ func TestRunnerRunSelectedProfileBeatsBuiltInDefault(t *testing.T) {
Model: "profile-model",
TopP: 0.8,
TimeoutSeconds: 90,
ServiceTier: "priority",
},
}}
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}
@@ -784,6 +1108,9 @@ func TestRunnerRunSelectedProfileBeatsBuiltInDefault(t *testing.T) {
if res.EffectiveModelParams.TimeoutSeconds != 90 {
t.Fatalf("expected profile timeout to beat default, got %d", res.EffectiveModelParams.TimeoutSeconds)
}
if res.EffectiveModelParams.ServiceTier != "priority" {
t.Fatalf("expected profile service_tier to beat default, got %q", res.EffectiveModelParams.ServiceTier)
}
}
func TestRunnerRunBuiltInDefaultsUsedWhenProfileOmitsOptionalFields(t *testing.T) {
@@ -844,6 +1171,9 @@ func TestRunnerRunAPIKeyEnvMissingEnvironmentValueFailsClearly(t *testing.T) {
if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected ErrInvalidRequest, got %v", err)
}
if !errors.Is(err, ErrAPIKeyEnvMissing) {
t.Fatalf("expected ErrAPIKeyEnvMissing, got %v", err)
}
if !strings.Contains(err.Error(), "SCRIPTORIUM_MISSING_KEY") {
t.Fatalf("expected missing env name in error, got %v", err)
}
@@ -863,7 +1193,7 @@ func TestRunnerRunRuntimeAPIKeyEnvOverrideWorks(t *testing.T) {
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
Execution: &domain.ExecutionTarget{APIKeyEnv: envName},
Execution: &domain.ExecutionTargetOverride{APIKeyEnv: envName},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
@@ -888,7 +1218,7 @@ func TestRunnerRunRuntimeAPIKeyEnvOverrideBeatsProfile(t *testing.T) {
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
Execution: &domain.ExecutionTarget{APIKeyEnv: runtimeEnv},
Execution: &domain.ExecutionTargetOverride{APIKeyEnv: runtimeEnv},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
@@ -987,6 +1317,28 @@ func TestRunnerRunLLMFailure(t *testing.T) {
}
}
func TestRunnerRunLLMInvalidRequestMapsToUsecaseInvalidRequest(t *testing.T) {
runner := NewRunner(
&fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}},
defaultArtifactReader(),
defaultRenderer(),
&fakeLLM{err: llm.ErrInvalidRequest},
nil,
)
_, err := runner.Run(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
})
if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected ErrInvalidRequest, got %v", err)
}
if errors.Is(err, ErrLLMGenerate) {
t.Fatalf("did not expect ErrLLMGenerate, got %v", err)
}
}
func TestRunnerRunValidationStillWorks(t *testing.T) {
validator := &fakeValidator{result: domain.ValidationResult{Status: domain.ValidationFailed, Mode: domain.ValidationBasic, Errors: []string{"bad"}, IsValid: false}}
runner := NewRunner(
@@ -1029,7 +1381,7 @@ func TestRunnerRunStructuredRepairRemainsBoundedAndUsesEffectiveModelSettings(t
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
Execution: &domain.ExecutionTarget{Endpoint: "http://override/v1", Model: "override-model", TimeoutSeconds: 22},
Execution: &domain.ExecutionTargetOverride{Endpoint: "http://override/v1", Model: "override-model", TimeoutSeconds: intPtr(22)},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
@@ -1104,6 +1456,186 @@ func TestRunnerRunJSONSchemaRepairCarriesStructuredOutputSpec(t *testing.T) {
}
}
func TestExecutionProfileToTargetPopulatesAllFieldsAndCopiesExtraParams(t *testing.T) {
src := &domain.ExecutionProfile{
ID: "exec",
Endpoint: "http://profile/v1",
Model: "profile-model",
Temperature: 0.2,
MaxTokens: 123,
TopP: 0.75,
TimeoutSeconds: 90,
ServiceTier: "priority",
ReasoningEffort: "medium",
APIKeyEnv: "SCRIPTORIUM_API_KEY",
ExtraParams: map[string]any{
"provider_option": "on",
},
}
target := executionProfileToTarget(src)
if target.Endpoint != src.Endpoint ||
target.Model != src.Model ||
target.Temperature != src.Temperature ||
target.MaxTokens != src.MaxTokens ||
target.TopP != src.TopP ||
target.TimeoutSeconds != src.TimeoutSeconds ||
target.ServiceTier != src.ServiceTier ||
target.ReasoningEffort != src.ReasoningEffort ||
target.APIKeyEnv != src.APIKeyEnv {
t.Fatalf("expected all profile fields to populate target, got %+v", target)
}
if !reflect.DeepEqual(target.ExtraParams, src.ExtraParams) {
t.Fatalf("expected extra_params to match, got %#v", target.ExtraParams)
}
src.ExtraParams["provider_option"] = "changed"
if target.ExtraParams["provider_option"] != "on" {
t.Fatalf("expected extra_params copy to be independent, got %#v", target.ExtraParams)
}
}
func TestResolveExecutionTargetProfileValuesPopulateAllSupportedFields(t *testing.T) {
profileValue := &domain.ExecutionProfile{
ID: "exec",
Endpoint: "http://profile/v1",
Model: "profile-model",
Temperature: 0.3,
MaxTokens: 222,
TopP: 0.6,
TimeoutSeconds: 77,
ServiceTier: "priority",
ReasoningEffort: "low",
APIKeyEnv: "PROFILE_KEY",
ExtraParams: map[string]any{
"profile_option": "enabled",
},
}
target, presence, err := resolveExecutionTarget(profileValue, nil)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if presence != (domain.ExecutionTargetPresence{}) {
t.Fatalf("expected no request override presence, got %+v", presence)
}
if target.Endpoint != profileValue.Endpoint ||
target.Model != profileValue.Model ||
target.Temperature != profileValue.Temperature ||
target.MaxTokens != profileValue.MaxTokens ||
target.TopP != profileValue.TopP ||
target.TimeoutSeconds != profileValue.TimeoutSeconds ||
target.ServiceTier != profileValue.ServiceTier ||
target.ReasoningEffort != profileValue.ReasoningEffort ||
target.APIKeyEnv != profileValue.APIKeyEnv {
t.Fatalf("expected profile values to populate target, got %+v", target)
}
if !reflect.DeepEqual(target.ExtraParams, profileValue.ExtraParams) {
t.Fatalf("expected profile extra_params in target, got %#v", target.ExtraParams)
}
}
func TestResolveExecutionTargetRuntimeOverridesBeatProfileForAllOverrideableFields(t *testing.T) {
profileValue := &domain.ExecutionProfile{
ID: "exec",
Endpoint: "http://profile/v1",
Model: "profile-model",
Temperature: 0.2,
MaxTokens: 200,
TopP: 0.8,
TimeoutSeconds: 90,
ServiceTier: "priority",
ReasoningEffort: "medium",
APIKeyEnv: "PROFILE_KEY",
ExtraParams: map[string]any{
"profile_only": "yes",
},
}
override := &domain.ExecutionTargetOverride{
Endpoint: "http://override/v1",
Model: "override-model",
Temperature: float64Ptr(0.9),
MaxTokens: intPtr(111),
TopP: float64Ptr(0.5),
TimeoutSeconds: intPtr(30),
ServiceTier: "flex",
ReasoningEffort: "high",
APIKeyEnv: "RUNTIME_KEY",
ExtraParams: map[string]any{
"runtime_only": "yes",
},
}
target, presence, err := resolveExecutionTarget(profileValue, override)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if presence != (domain.ExecutionTargetPresence{Temperature: true, MaxTokens: true, TopP: true, TimeoutSeconds: true}) {
t.Fatalf("unexpected override presence: %+v", presence)
}
if target.Endpoint != override.Endpoint ||
target.Model != override.Model ||
target.Temperature != *override.Temperature ||
target.MaxTokens != *override.MaxTokens ||
target.TopP != *override.TopP ||
target.TimeoutSeconds != *override.TimeoutSeconds ||
target.ServiceTier != override.ServiceTier ||
target.ReasoningEffort != override.ReasoningEffort ||
target.APIKeyEnv != override.APIKeyEnv {
t.Fatalf("expected runtime overrides to win for all fields, got %+v", target)
}
if !reflect.DeepEqual(target.ExtraParams, override.ExtraParams) {
t.Fatalf("expected runtime extra_params to replace profile extra_params, got %#v", target.ExtraParams)
}
}
func TestMergeExecutionTargetEmptyStringOverridesDoNotErase(t *testing.T) {
base := domain.ExecutionTarget{
Endpoint: "http://base/v1",
Model: "base-model",
ServiceTier: "priority",
ReasoningEffort: "medium",
APIKeyEnv: "BASE_KEY",
}
override := domain.ExecutionTarget{
Endpoint: "http://override/v1",
Model: "override-model",
ServiceTier: " ",
ReasoningEffort: " ",
APIKeyEnv: "",
}
merged := mergeExecutionTarget(base, override)
if merged.Endpoint != "http://override/v1" || merged.Model != "override-model" {
t.Fatalf("expected endpoint/model to override, got %+v", merged)
}
if merged.ServiceTier != "priority" {
t.Fatalf("expected empty service_tier override to be ignored, got %q", merged.ServiceTier)
}
if merged.ReasoningEffort != "medium" {
t.Fatalf("expected empty reasoning_effort override to be ignored, got %q", merged.ReasoningEffort)
}
if merged.APIKeyEnv != "BASE_KEY" {
t.Fatalf("expected empty api_key_env override to be ignored, got %q", merged.APIKeyEnv)
}
}
func TestMergeExecutionTargetEmptyExtraParamsDoesNotErase(t *testing.T) {
base := domain.ExecutionTarget{
ExtraParams: map[string]any{
"keep": "value",
},
}
override := domain.ExecutionTarget{
ExtraParams: map[string]any{},
}
merged := mergeExecutionTarget(base, override)
if !reflect.DeepEqual(merged.ExtraParams, base.ExtraParams) {
t.Fatalf("expected empty extra_params override not to erase base values, got %#v", merged.ExtraParams)
}
}
func TestBuildOutputArtifactDefaults(t *testing.T) {
tests := []struct {
name string
@@ -1171,6 +1703,14 @@ func singleInputRef() map[string]domain.ArtifactRef {
return map[string]domain.ArtifactRef{"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"}}
}
func float64Ptr(v float64) *float64 {
return &v
}
func intPtr(v int) *int {
return &v
}
func newMinimalRunner(promptRepo *fakePromptRepo, execRepo *fakeExecutionProfileRepo) *Runner {
return NewRunner(
promptRepo,

View File

@@ -116,6 +116,49 @@ func TestStandardValidatorJSONSchemaSuccess(t *testing.T) {
}
}
func TestStandardValidatorJSONSchemaNestedSchemaPathSuccess(t *testing.T) {
tmp := t.TempDir()
nestedDir := filepath.Join(tmp, "dnd")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(nestedDir, "schema.json"), []byte(`{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["name"],
"properties": {
"name": {"type": "string"}
}
}`), 0644); err != nil {
t.Fatal(err)
}
v := NewStandardValidator(tmp)
res, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"name":"eris"}`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: filepath.Join("dnd", "schema.json"),
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if res.Status != domain.ValidationPassed || !res.IsValid {
t.Fatalf("expected passed/valid, got status=%q valid=%v", res.Status, res.IsValid)
}
}
func TestStandardValidatorJSONSchemaNestedSchemaPathMissing(t *testing.T) {
v := NewStandardValidator(t.TempDir())
_, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"name":"eris"}`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: filepath.Join("dnd", "missing.json"),
})
if err == nil {
t.Fatal("expected nested schema load error")
}
}
func TestStandardValidatorJSONSchemaFailure(t *testing.T) {
tmp := t.TempDir()
schemaPath := filepath.Join(tmp, "schema.json")

23
llm_adapter.go Normal file
View File

@@ -0,0 +1,23 @@
package scriptorium
import (
"context"
"fmt"
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
)
type publicLLMClientAdapter struct {
client LLMClient
}
func (a publicLLMClientAdapter) Generate(ctx context.Context, req domain.GenerateRequest) (*domain.GenerateResponse, error) {
resp, err := a.client.Generate(ctx, fromDomainGenerateRequest(req))
if err != nil {
return nil, err
}
if resp == nil {
return nil, fmt.Errorf("%w: llm client returned nil response", ErrLLMGenerate)
}
return toDomainGenerateResponse(resp), nil
}

256
types.go Normal file
View File

@@ -0,0 +1,256 @@
package scriptorium
import (
"context"
"time"
)
// ArtifactRefType defines how an artifact is referenced.
type ArtifactRefType string
const (
ArtifactRefInline ArtifactRefType = "inline"
ArtifactRefFile ArtifactRefType = "file"
)
// OutputFormat defines the desired output format.
type OutputFormat string
const (
FormatText OutputFormat = "text"
FormatMarkdown OutputFormat = "markdown"
FormatJSON OutputFormat = "json"
)
// ValidationMode defines the output validation strategy.
type ValidationMode string
const (
ValidationNone ValidationMode = "none"
ValidationBasic ValidationMode = "basic"
ValidationJSON ValidationMode = "json"
ValidationJSONSchema ValidationMode = "json_schema"
)
// ValidationStatus defines the result of a validation check.
type ValidationStatus string
const (
ValidationPassed ValidationStatus = "passed"
ValidationFailed ValidationStatus = "failed"
ValidationSkipped ValidationStatus = "skipped"
)
// CacheControlType defines provider cache behavior for prompt content.
type CacheControlType string
const (
CacheControlEphemeral CacheControlType = "ephemeral"
)
// StructuredOutputType identifies provider-level structured output modes.
type StructuredOutputType string
const (
StructuredOutputJSONSchema StructuredOutputType = "json_schema"
)
// RunRequest represents a request to prepare or run a single prompt.
type RunRequest struct {
PromptID string
PromptVersion string
ProfileID string
Inputs map[string]ArtifactRef
Vars map[string]string
Execution *ExecutionTargetOverride
Validation *OutputContract
Metadata map[string]string
}
// PreparedRun contains prepared prompt execution state. It does not include
// resolved API key values, model output, validation results, or internal target
// presence metadata.
type PreparedRun struct {
PromptID string `json:"prompt_id"`
PromptVersion string `json:"prompt_version,omitempty"`
PromptHash string `json:"prompt_hash,omitempty"`
SelectedProfileID string `json:"selected_profile_id"`
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
OutputContract OutputContract `json:"output_contract"`
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
InputHashes map[string]string `json:"input_hashes,omitempty"`
SessionID string `json:"session_id,omitempty"`
RenderedPromptHash string `json:"rendered_prompt_hash"`
Messages []RenderedMessage `json:"messages"`
StartTime time.Time `json:"start_time,omitempty"`
EndTime time.Time `json:"end_time,omitempty"`
DurationMS int64 `json:"duration_ms,omitempty"`
}
// RunResult contains generated output, validation state, and run metadata.
type RunResult struct {
RunID string `json:"run_id"`
Artifact Artifact `json:"artifact"`
RawOutput string `json:"raw_output"`
Validation ValidationResult `json:"validation"`
PromptID string `json:"prompt_id"`
PromptVersion string `json:"prompt_version,omitempty"`
PromptHash string `json:"prompt_hash,omitempty"`
RenderedPromptHash string `json:"rendered_prompt_hash"`
SelectedProfileID string `json:"selected_profile_id"`
ModelName string `json:"model_name"`
Endpoint string `json:"endpoint"`
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
InputHashes map[string]string `json:"input_hashes,omitempty"`
Usage TokenUsage `json:"usage"`
StartTime time.Time `json:"start_time,omitempty"`
EndTime time.Time `json:"end_time,omitempty"`
Duration time.Duration `json:"duration,omitempty"`
}
// ArtifactRef represents a reference to prompt input content.
type ArtifactRef struct {
Type ArtifactRefType
URI string
Body string
}
// Artifact represents loaded artifact content.
type Artifact struct {
Name string
ContentType string
Body []byte
URI string
Size int64
Hash string
}
// ExecutionTarget represents effective model runtime settings.
type ExecutionTarget struct {
Endpoint string `json:"endpoint"`
Model string `json:"model"`
Temperature float64 `json:"temperature"`
MaxTokens int `json:"max_tokens"`
TopP float64 `json:"top_p"`
TimeoutSeconds int `json:"timeout_seconds"`
ServiceTier string `json:"service_tier"`
ReasoningEffort string `json:"reasoning_effort"`
APIKeyEnv string `json:"api_key_env"`
ExtraParams map[string]any `json:"extra_params"`
}
// ExecutionTargetOverride represents per-request runtime setting overrides.
type ExecutionTargetOverride struct {
Endpoint string
Model string
Temperature *float64
MaxTokens *int
TopP *float64
TimeoutSeconds *int
ServiceTier string
ReasoningEffort string
APIKeyEnv string
ExtraParams map[string]any
}
// ExecutionTargetPresence tracks which numeric runtime settings were explicit
// request overrides.
type ExecutionTargetPresence struct {
Temperature bool
MaxTokens bool
TopP bool
TimeoutSeconds bool
}
// OutputContract defines output and validation requirements.
type OutputContract struct {
Format OutputFormat `json:"format"`
ValidationMode ValidationMode `json:"validation_mode"`
SchemaPath string `json:"schema_path"`
RepairAttempts int `json:"repair_attempts"`
}
// ValidationResult represents output validation state.
type ValidationResult struct {
Status ValidationStatus `json:"status"`
Mode ValidationMode `json:"mode"`
Errors []string `json:"errors,omitempty"`
SchemaPath string `json:"schema_path,omitempty"`
RepairAttempts int `json:"repair_attempts"`
IsValid bool `json:"is_valid"`
}
// TokenUsage tracks token consumption.
type TokenUsage struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"`
CachedTokens int `json:"cached_tokens"`
CacheWriteTokens int `json:"cache_write_tokens"`
}
// RenderedPrompt is the fully rendered prompt passed to an LLM client.
type RenderedPrompt struct {
SessionID string `json:"session_id,omitempty"`
Messages []RenderedMessage `json:"messages"`
}
// RenderedMessage is a rendered chat message.
type RenderedMessage struct {
Role string `json:"role"`
Content string `json:"content"`
CacheControl *CacheControl `json:"cache_control,omitempty"`
}
// CacheControl describes provider cache metadata attached to prompt content.
type CacheControl struct {
Type CacheControlType `json:"type"`
TTL string `json:"ttl,omitempty"`
}
// StructuredOutputSpec describes provider-level structured output.
type StructuredOutputSpec struct {
Type StructuredOutputType `json:"type"`
JSONSchema *StructuredOutputJSONSpec `json:"json_schema,omitempty"`
}
// StructuredOutputJSONSpec contains JSON Schema output constraints.
type StructuredOutputJSONSpec struct {
Name string `json:"name"`
Strict bool `json:"strict"`
Schema any `json:"schema"`
}
// LLMClient executes rendered prompts for Engine.Run.
type LLMClient interface {
Generate(context.Context, GenerateRequest) (*GenerateResponse, error)
}
// GenerateRequest is passed to an injected LLM client.
type GenerateRequest struct {
Prompt RenderedPrompt `json:"prompt"`
Target ExecutionTarget `json:"target"`
TargetPresence ExecutionTargetPresence `json:"target_presence"`
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
}
// GenerateResponse is returned by an injected LLM client.
type GenerateResponse struct {
Content string `json:"content"`
Usage TokenUsage `json:"usage"`
}
// File returns a file-backed artifact reference.
func File(path string) ArtifactRef {
return ArtifactRef{Type: ArtifactRefFile, URI: path}
}
// Inline returns an inline artifact reference.
func Inline(body string) ArtifactRef {
return ArtifactRef{Type: ArtifactRefInline, Body: body}
}
// InlineWithURI returns an inline artifact reference with URI metadata.
func InlineWithURI(uri string, body string) ArtifactRef {
return ArtifactRef{Type: ArtifactRefInline, URI: uri, Body: body}
}