Compare commits
19 Commits
v0.8.0
...
8d8024099f
| Author | SHA1 | Date | |
|---|---|---|---|
| 8d8024099f | |||
| 18792fd8d1 | |||
| 8860aa033c | |||
| 3ca14d8b6e | |||
| 099e9c4a3e | |||
| cfe6b9408a | |||
| 6ececc749f | |||
| 79901fbb86 | |||
| 75fa0a030a | |||
| ef64966897 | |||
| c6c5e3cb69 | |||
| 3f4fd230b9 | |||
| 2091b58066 | |||
| c3fe88c9fa | |||
| 5830fda516 | |||
| 359e910572 | |||
| 4950a6bb14 | |||
| b69ba96811 | |||
| 941e2656e8 |
449
README.md
449
README.md
@@ -1,449 +1,36 @@
|
|||||||
# scriptorium
|
# scriptorium
|
||||||
|
|
||||||
Scriptorium is a generic prompt execution engine.
|
Scriptorium is a config-driven prompt execution engine.
|
||||||
|
|
||||||
It takes:
|
It separates prompt definitions (what to generate) from execution profiles (how to call an OpenAI-compatible model endpoint), then runs or renders a prepared request from named input artifacts.
|
||||||
- a prompt definition
|
|
||||||
- a selected or default execution profile
|
|
||||||
- named input artifacts
|
|
||||||
- template variables
|
|
||||||
- optional runtime overrides
|
|
||||||
|
|
||||||
It returns:
|
## Quickstart
|
||||||
- for `run`: generated artifact, validation result, metadata
|
|
||||||
- for `render`: prepared/rendered prompt data (no model output)
|
|
||||||
|
|
||||||
## Prompt vs Profile
|
From the repository root:
|
||||||
|
|
||||||
Scriptorium separates **what** to do (Prompt) from **how** to do it (Profile).
|
|
||||||
|
|
||||||
### Prompt Definition
|
|
||||||
Defines the task logic and output contract.
|
|
||||||
- Task description and version.
|
|
||||||
- Message templates (system, user, etc.).
|
|
||||||
- Required and optional input artifacts.
|
|
||||||
- Output format and validation rules.
|
|
||||||
- Repair settings for structured output.
|
|
||||||
- Optional `default_profile` for convenience.
|
|
||||||
|
|
||||||
### Execution Profile
|
|
||||||
Defines the runtime environment and model settings.
|
|
||||||
- LLM endpoint (URL).
|
|
||||||
- Model name.
|
|
||||||
- Generation parameters: `temperature`, `max_tokens`, `top_p`.
|
|
||||||
- Runtime settings: `timeout`, `reasoning_effort`.
|
|
||||||
- API key source via `api_key_env`.
|
|
||||||
|
|
||||||
Callers can explicitly provide a `profile_id` to override the prompt's `default_profile`.
|
|
||||||
|
|
||||||
## Precedence
|
|
||||||
|
|
||||||
Scriptorium uses two precedence layers:
|
|
||||||
|
|
||||||
### Application Configuration Precedence
|
|
||||||
|
|
||||||
For application-level adapter settings (for example prompt/profile/schema directories, server address, and render output default), precedence is:
|
|
||||||
|
|
||||||
1. **CLI Flags**
|
|
||||||
2. **`config.yml`**
|
|
||||||
3. **Built-in application defaults**
|
|
||||||
|
|
||||||
Application config loading behavior:
|
|
||||||
- Default config path: `/etc/scriptorium/config.yml`
|
|
||||||
- Override path: `--config <PATH>` (supported by `run`, `render`, and `serve`)
|
|
||||||
- If `--config` is provided, the file must exist and be valid.
|
|
||||||
- If `--config` is omitted, missing `/etc/scriptorium/config.yml` is allowed.
|
|
||||||
|
|
||||||
### Runtime Model Precedence
|
|
||||||
|
|
||||||
When resolving runtime model settings, Scriptorium follows this precedence model (highest to lowest):
|
|
||||||
|
|
||||||
1. **Runtime Overrides**: Provided via CLI flags or HTTP request `model` object.
|
|
||||||
2. **Execution Profile**: Settings defined in the selected profile.
|
|
||||||
3. **Application Defaults**: Built-in fallback values.
|
|
||||||
|
|
||||||
### Profile Selection Logic
|
|
||||||
The engine determines which profile to use in this order:
|
|
||||||
1. Explicit `profile_id` (via `--profile` or HTTP request).
|
|
||||||
2. The `default_profile` named in the Prompt Definition.
|
|
||||||
3. Error: If neither is provided and no default exists.
|
|
||||||
|
|
||||||
## API Key Policy
|
|
||||||
|
|
||||||
To ensure security, Scriptorium does not support raw API keys in configuration files, CLI arguments, or HTTP requests.
|
|
||||||
|
|
||||||
- **`api_key_env`**: Profiles and overrides specify the name of an environment variable (e.g., `SCRIPTORIUM_API_KEY`).
|
|
||||||
- **Runtime Resolution**: The value of the environment variable is read directly from the process environment at runtime.
|
|
||||||
- **Zero Leakage**: API key values are never included in metadata, logs, or response bodies.
|
|
||||||
|
|
||||||
## CLI Usage
|
|
||||||
|
|
||||||
Available CLI commands:
|
|
||||||
- `scriptorium run`
|
|
||||||
- `scriptorium render`
|
|
||||||
- `scriptorium serve`
|
|
||||||
|
|
||||||
All commands accept `--config <PATH>`.
|
|
||||||
|
|
||||||
`prompt_dir` and `profile_dir` may be supplied by CLI flags or `config.yml`:
|
|
||||||
- `--prompt-dir` or `config.yml` `prompt_dir`
|
|
||||||
- `--profile-dir` or `config.yml` `profile_dir`
|
|
||||||
|
|
||||||
`schema_dir` and `serve` `addr` may also be supplied by `config.yml` where applicable:
|
|
||||||
- `--schema-dir` or `config.yml` `schema_dir`
|
|
||||||
- `--addr` or `config.yml` `server.addr`
|
|
||||||
|
|
||||||
### `scriptorium run`
|
|
||||||
|
|
||||||
Runs a single prompt execution.
|
|
||||||
|
|
||||||
**Required Flags:**
|
|
||||||
- `--prompt`: The prompt ID to execute.
|
|
||||||
- `--input`: Input mapping `name=path` (repeatable).
|
|
||||||
|
|
||||||
**Required Effective Settings:**
|
|
||||||
- Prompt directory: `--prompt-dir` or `config.yml` `prompt_dir`
|
|
||||||
- Profile directory: `--profile-dir` or `config.yml` `profile_dir`
|
|
||||||
|
|
||||||
**Optional Flags:**
|
|
||||||
- `--config`: Application config file path. Default discovery path is `/etc/scriptorium/config.yml`.
|
|
||||||
- `--prompt-dir`: Override prompt directory from config.
|
|
||||||
- `--profile-dir`: Override profile directory from config.
|
|
||||||
- `--profile`: Override the prompt's default profile.
|
|
||||||
- `--var`: Template variable `name=value` (repeatable).
|
|
||||||
- `--out`: Write output to a file instead of stdout.
|
|
||||||
- `--llm-base-url`: Override endpoint.
|
|
||||||
- `--model`: Override model name.
|
|
||||||
- `--api-key-env`: Override API key environment variable name.
|
|
||||||
- `--temperature`: Override temperature.
|
|
||||||
- `--max-tokens`: Override max tokens.
|
|
||||||
- `--top-p`: Override top_p.
|
|
||||||
- `--timeout`: Override request timeout (e.g., `30s`, `1m`).
|
|
||||||
- `--schema-dir`: Base directory for validation schemas.
|
|
||||||
|
|
||||||
**Examples:**
|
|
||||||
|
|
||||||
Using `config.yml` for prompt/profile directories:
|
|
||||||
```bash
|
```bash
|
||||||
scriptorium run \
|
go run ./cmd/scriptorium render \
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Overriding config directories explicitly:
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Overriding the profile:
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--profile local-quality \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Overriding model and runtime values:
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--model gpt-4o \
|
|
||||||
--temperature 0.7 \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Using a local OpenAI-compatible vLLM endpoint:
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--llm-base-url http://localhost:8000/v1 \
|
|
||||||
--model meta-llama-3-8b \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
### `scriptorium render`
|
|
||||||
|
|
||||||
Prepares and renders a prompt without calling the LLM.
|
|
||||||
|
|
||||||
`render` uses the same prompt/profile/input/variable/runtime override resolution as `run`:
|
|
||||||
- Profile selection precedence: `--profile` -> prompt `default_profile` -> error.
|
|
||||||
- Runtime precedence: CLI runtime overrides -> selected profile -> built-in defaults.
|
|
||||||
|
|
||||||
`render` is useful for debugging:
|
|
||||||
- prompt template rendering
|
|
||||||
- input mappings
|
|
||||||
- selected profile behavior
|
|
||||||
- runtime override behavior
|
|
||||||
|
|
||||||
`render` does not:
|
|
||||||
- call the LLM
|
|
||||||
- validate model output
|
|
||||||
- perform repair
|
|
||||||
- expose resolved API key values
|
|
||||||
|
|
||||||
It may include `api_key_env` names where relevant.
|
|
||||||
|
|
||||||
**Required Flags:**
|
|
||||||
- `--prompt`: The prompt ID to render.
|
|
||||||
- `--input`: Input mapping `name=path` (repeatable).
|
|
||||||
|
|
||||||
**Required Effective Settings:**
|
|
||||||
- Prompt directory: `--prompt-dir` or `config.yml` `prompt_dir`
|
|
||||||
- Profile directory: `--profile-dir` or `config.yml` `profile_dir`
|
|
||||||
|
|
||||||
**Optional Flags:**
|
|
||||||
- `--config`: Application config file path. Default discovery path is `/etc/scriptorium/config.yml`.
|
|
||||||
- `--prompt-dir`: Override prompt directory from config.
|
|
||||||
- `--profile-dir`: Override profile directory from config.
|
|
||||||
- `--profile`: Override the prompt's default profile.
|
|
||||||
- `--var`: Template variable `name=value` (repeatable).
|
|
||||||
- `--out`: Write output to a file instead of stdout.
|
|
||||||
- `--format`: Render output format (`text` or `json`). Default: `text`.
|
|
||||||
- `--llm-base-url`: Runtime override for endpoint.
|
|
||||||
- `--model`: Runtime override for model name.
|
|
||||||
- `--api-key-env`: Runtime override for API key environment variable name.
|
|
||||||
- `--temperature`: Runtime override for temperature.
|
|
||||||
- `--max-tokens`: Runtime override for max tokens.
|
|
||||||
- `--top-p`: Runtime override for top_p.
|
|
||||||
- `--timeout`: Runtime override for timeout (e.g., `30s`, `1m`).
|
|
||||||
|
|
||||||
**Render Output Formats:**
|
|
||||||
- `text`: Human-readable output (default).
|
|
||||||
- `json`: Machine-readable structured output.
|
|
||||||
|
|
||||||
Render formatting is modular; additional output formats can be added later without changing prepare/run core logic.
|
|
||||||
|
|
||||||
**Examples:**
|
|
||||||
|
|
||||||
Default text output using `config.yml` directories:
|
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Explicit config path:
|
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--config ./examples/config.yml \
|
--config ./examples/config.yml \
|
||||||
--prompt generic.markdown_summary \
|
--prompt generic.markdown_summary \
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Explicit directory overrides:
|
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Explicit JSON output:
|
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md \
|
--input transcript=./examples/fixtures/transcript.md \
|
||||||
|
--input glossary=./examples/fixtures/glossary.yml \
|
||||||
--format json
|
--format json
|
||||||
```
|
```
|
||||||
|
|
||||||
Using prompt `default_profile` (omit `--profile`):
|
This command renders the prepared prompt and effective runtime settings without calling an LLM.
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Overriding profile selection:
|
## Documentation
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--profile local-quality \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Overriding runtime settings:
|
- [CLI reference](docs/cli.md)
|
||||||
```bash
|
- [Configuration reference](docs/config.md)
|
||||||
scriptorium render \
|
- [Operations guide](docs/operations.md)
|
||||||
--prompt-dir ./prompts \
|
- [Troubleshooting](docs/troubleshooting.md)
|
||||||
--profile-dir ./profiles \
|
- [HTTP API integration](docs/integrations/http-api.md)
|
||||||
--prompt generic.markdown_summary \
|
- [OpenAI-compatible chat integration](docs/integrations/openai-compatible-chat.md)
|
||||||
--input transcript=./examples/fixtures/transcript.md \
|
- [Narratio subprocess integration](docs/integrations/narratio.md)
|
||||||
--llm-base-url http://localhost:8000/v1 \
|
- [Architecture policy](docs/policy/architecture.md)
|
||||||
--model gpt-4o-mini \
|
|
||||||
--temperature 0.2 \
|
|
||||||
--max-tokens 800 \
|
|
||||||
--top-p 1.0 \
|
|
||||||
--timeout 45s
|
|
||||||
```
|
|
||||||
|
|
||||||
Writing rendered output to a file:
|
|
||||||
```bash
|
|
||||||
scriptorium render \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md \
|
|
||||||
--format text \
|
|
||||||
--out ./rendered_prompt.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
### `scriptorium serve`
|
|
||||||
|
|
||||||
Starts the HTTP API.
|
|
||||||
|
|
||||||
**Required Effective Settings:**
|
|
||||||
- Prompt directory: `--prompt-dir` or `config.yml` `prompt_dir`
|
|
||||||
- Profile directory: `--profile-dir` or `config.yml` `profile_dir`
|
|
||||||
|
|
||||||
**Optional Flags:**
|
|
||||||
- `--config`: Application config file path. Default discovery path is `/etc/scriptorium/config.yml`.
|
|
||||||
- `--addr`: Listen address (default `:8080`).
|
|
||||||
- `--schema-dir`: Base directory for validation schemas.
|
|
||||||
|
|
||||||
**Examples:**
|
|
||||||
|
|
||||||
Using `config.yml`:
|
|
||||||
```bash
|
|
||||||
scriptorium serve
|
|
||||||
```
|
|
||||||
|
|
||||||
Overriding config for local use:
|
|
||||||
```bash
|
|
||||||
scriptorium serve \
|
|
||||||
--prompt-dir ./prompts \
|
|
||||||
--profile-dir ./profiles \
|
|
||||||
--addr :9090
|
|
||||||
```
|
|
||||||
|
|
||||||
## HTTP API
|
|
||||||
|
|
||||||
### `POST /v1/runs`
|
|
||||||
|
|
||||||
Executes a prompt. No built-in authentication is provided; deploy behind a trusted gateway.
|
|
||||||
|
|
||||||
**Request Body:**
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"prompt_id": "generic.structured_events",
|
|
||||||
"profile_id": "local-quality",
|
|
||||||
"include_raw_output": false,
|
|
||||||
"inputs": {
|
|
||||||
"transcript": {"type": "file", "uri": "./examples/fixtures/transcript.md"}
|
|
||||||
},
|
|
||||||
"vars": {
|
|
||||||
"session_date": "2026-05-04"
|
|
||||||
},
|
|
||||||
"model": {
|
|
||||||
"endpoint": "http://localhost:8000/v1",
|
|
||||||
"model": "gpt-4o-mini",
|
|
||||||
"temperature": 0.0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
`profile_id` is optional. If omitted, Scriptorium uses the prompt's `default_profile`. If neither is available, the run fails.
|
|
||||||
|
|
||||||
**Response:**
|
|
||||||
Returns a `200 OK` with the generated artifact, validation results, and metadata including the `prompt_id` and the `selected_profile_id`.
|
|
||||||
|
|
||||||
**Validation Failures:**
|
|
||||||
If the model output fails validation (e.g., invalid JSON), the API returns `200 OK` with `validation.status = "failed"`.
|
|
||||||
|
|
||||||
**Raw Output Exposure:**
|
|
||||||
- `raw_model_output` is omitted by default.
|
|
||||||
- Set `include_raw_output: true` in the request to include it in the response.
|
|
||||||
- Raw output is preserved internally in run results regardless of HTTP exposure.
|
|
||||||
|
|
||||||
## Prompt Definition Authoring
|
|
||||||
|
|
||||||
Prompts are defined in YAML.
|
|
||||||
|
|
||||||
### Canonical Shape
|
|
||||||
```yaml
|
|
||||||
id: generic.structured_events
|
|
||||||
version: "1.0.0"
|
|
||||||
description: "Extracts structured events from a transcript"
|
|
||||||
default_profile: local-quality
|
|
||||||
|
|
||||||
inputs:
|
|
||||||
- name: transcript
|
|
||||||
required: true
|
|
||||||
content_type: text/markdown
|
|
||||||
description: "The raw session transcript"
|
|
||||||
- name: glossary
|
|
||||||
required: false
|
|
||||||
content_type: application/yaml
|
|
||||||
description: "Optional glossary terms"
|
|
||||||
|
|
||||||
messages:
|
|
||||||
- role: system
|
|
||||||
content: "You are a helpful assistant."
|
|
||||||
- role: user
|
|
||||||
content_file: messages/extract_events.tmpl
|
|
||||||
|
|
||||||
output:
|
|
||||||
format: json
|
|
||||||
validation_mode: json_schema
|
|
||||||
schema_path: structured_events.schema.json
|
|
||||||
repair_attempts: 2
|
|
||||||
```
|
|
||||||
|
|
||||||
**Key Features:**
|
|
||||||
- **Inline vs File**: Use `content` for short prompts or `content_file` for larger templates. Exactly one must be set per message.
|
|
||||||
- **Path Resolution**: `content_file` paths are resolved relative to the prompt YAML file.
|
|
||||||
- **Inputs**: Mark inputs as `required` to ensure the runner fails early if they are missing.
|
|
||||||
- **Input Metadata**: `content_type` is currently descriptive metadata and not enforced yet.
|
|
||||||
- **Validation**: Support `none`, `basic`, `json`, and `json_schema`.
|
|
||||||
- **Repair**: `repair_attempts` enables bounded retries to fix structured output.
|
|
||||||
|
|
||||||
## Execution Profile Authoring
|
|
||||||
|
|
||||||
Profiles are defined in YAML.
|
|
||||||
|
|
||||||
### Canonical Shape
|
|
||||||
```yaml
|
|
||||||
id: local-quality
|
|
||||||
endpoint: http://localhost:8000/v1
|
|
||||||
model: gpt-4o
|
|
||||||
temperature: 0.0
|
|
||||||
max_tokens: 4096
|
|
||||||
top_p: 1.0
|
|
||||||
timeout_seconds: 300
|
|
||||||
reasoning_effort: high
|
|
||||||
api_key_env: SCRIPTORIUM_API_KEY
|
|
||||||
```
|
|
||||||
|
|
||||||
**Constraints:**
|
|
||||||
- **No Raw Keys**: Do not include actual API keys. Only specify the environment variable name in `api_key_env`.
|
|
||||||
- **Local Profiles**: For local endpoints that don't require auth, `api_key_env` can be omitted.
|
|
||||||
|
|
||||||
## Examples
|
## Examples
|
||||||
|
|
||||||
- **Prompt Definitions**: `prompts/`
|
- `examples/render-markdown-summary.sh`
|
||||||
- **Execution Profiles**: `profiles/`
|
- `examples/http-run.json`
|
||||||
- **Schemas**: `schemas/`
|
|
||||||
- **Fixtures**: `examples/fixtures/`
|
|
||||||
|
|
||||||
## Build and Test
|
|
||||||
|
|
||||||
```bash
|
|
||||||
go build -o scriptorium ./cmd/scriptorium
|
|
||||||
go test ./...
|
|
||||||
```
|
|
||||||
|
|||||||
483
architecture.md
483
architecture.md
@@ -1,483 +0,0 @@
|
|||||||
# Scriptorium Architecture
|
|
||||||
|
|
||||||
## 1. Purpose and Non-Goals
|
|
||||||
|
|
||||||
Scriptorium is a prompt-definition execution engine.
|
|
||||||
|
|
||||||
It accepts named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata.
|
|
||||||
|
|
||||||
Scriptorium also supports rendering/preparing a prompt without calling an LLM. This allows users to inspect the fully rendered prompt messages and effective runtime settings before executing a run.
|
|
||||||
|
|
||||||
Scriptorium is not an orchestrator. It must not own transcription, transcript merge/polish steps, notifications, or cross-step workflow control.
|
|
||||||
|
|
||||||
For the motivating D&D workflow:
|
|
||||||
- Narratio orchestrates.
|
|
||||||
- WhisperX transcribes.
|
|
||||||
- Seriatim merges transcripts.
|
|
||||||
- Audita polishes transcripts.
|
|
||||||
- Scriptorium generates final artifacts from prepared inputs.
|
|
||||||
|
|
||||||
Core Go code remains generic.
|
|
||||||
|
|
||||||
## 2. Current Architecture
|
|
||||||
|
|
||||||
Scriptorium uses a ports-and-adapters architecture to decouple the core execution logic from external dependencies.
|
|
||||||
|
|
||||||
### Package Responsibilities
|
|
||||||
|
|
||||||
- `cmd/scriptorium`: Binary entrypoint for CLI and HTTP server.
|
|
||||||
- `internal/domain`: Core domain contracts, including `PromptDefinition`, `ExecutionProfile`, `PreparedRun`, `RunResult`, and related metadata.
|
|
||||||
- `internal/usecase`: `Runner` use case logic, including prompt preparation, profile selection, runtime override resolution, full run execution, validation, and bounded repair.
|
|
||||||
- `internal/config`: Application-level config model/loader for adapter settings (for example prompt/profile/schema directories, server address, and render format default).
|
|
||||||
- `internal/promptdef`: Repository for loading and validating Prompt Definitions from the filesystem.
|
|
||||||
- `internal/profile`: Repository for loading Execution Profiles from the filesystem.
|
|
||||||
- `internal/artifact`: Input artifact resolution (`inline`, `file`).
|
|
||||||
- `internal/prompt`: Template rendering via Go templates.
|
|
||||||
- `internal/llm`: Provider-neutral client interface and OpenAI-compatible HTTP adapter.
|
|
||||||
- `internal/validate`: Output validation implementation (`none/basic/json/json_schema`).
|
|
||||||
- `internal/adapter/cli`: CLI flag parsing, command dispatch, and output handling.
|
|
||||||
- `internal/adapter/http`: HTTP request/response mapping.
|
|
||||||
- `internal/format` or equivalent: Formatting of prepared/rendered prompt output for CLI or other adapters, if formatting grows beyond simple CLI-local helpers.
|
|
||||||
|
|
||||||
Exact package names may evolve, but the architectural boundaries should remain stable.
|
|
||||||
|
|
||||||
### Application Configuration
|
|
||||||
|
|
||||||
`config.yml` is adapter/application setup, not domain logic.
|
|
||||||
|
|
||||||
Application config is intended for application-level settings such as:
|
|
||||||
- `prompt_dir`
|
|
||||||
- `profile_dir`
|
|
||||||
- `schema_dir`
|
|
||||||
- `server.addr`
|
|
||||||
- `defaults.render_format`
|
|
||||||
|
|
||||||
Application config precedence is:
|
|
||||||
1. CLI flags
|
|
||||||
2. `config.yml`
|
|
||||||
3. Built-in application defaults
|
|
||||||
|
|
||||||
Runtime model settings are intentionally separate:
|
|
||||||
- Execution profiles and runtime overrides continue to own endpoint/model/runtime behavior.
|
|
||||||
- `config.yml` does not replace execution profiles.
|
|
||||||
|
|
||||||
The core use case (`Runner.Prepare`/`Runner.Run`) does not need to know whether adapter-level settings came from CLI flags or `config.yml`; it receives resolved dependencies and requests from adapters.
|
|
||||||
|
|
||||||
## 3. Core Execution Model
|
|
||||||
|
|
||||||
Scriptorium has two closely related execution paths:
|
|
||||||
|
|
||||||
1. Prepare/render path.
|
|
||||||
2. Full run path.
|
|
||||||
|
|
||||||
The full run path should reuse the prepare path rather than duplicating its logic.
|
|
||||||
|
|
||||||
### 3.1 Prepare / Render Data Flow
|
|
||||||
|
|
||||||
The prepare path should be represented in the use case layer, preferably as `Runner.Prepare(ctx, RunRequest)` or an equivalent method.
|
|
||||||
|
|
||||||
It executes all pre-LLM work:
|
|
||||||
|
|
||||||
1. **Validate Request**: Ensure the request includes a prompt ID and all minimum required fields.
|
|
||||||
2. **Load Prompt Definition**: Retrieve the `PromptDefinition` by ID from the prompt repository.
|
|
||||||
3. **Select Profile**: Determine the `profile_id` using this precedence:
|
|
||||||
- Explicit `profile_id` in `RunRequest`.
|
|
||||||
- `default_profile` specified in the `PromptDefinition`.
|
|
||||||
- Error if neither is available.
|
|
||||||
4. **Load Execution Profile**: Retrieve the `ExecutionProfile` from the profile repository.
|
|
||||||
5. **Resolve Runtime Overrides**: Merge settings based on precedence, highest to lowest:
|
|
||||||
- Runtime overrides from CLI flags or HTTP request `model` object.
|
|
||||||
- Execution Profile settings.
|
|
||||||
- Built-in application defaults.
|
|
||||||
6. **Resolve Artifacts**: Load all named input artifacts defined in the request.
|
|
||||||
7. **Render Prompt**: Apply template variables and input artifacts to the prompt templates.
|
|
||||||
8. **Compute Metadata**: Compute hashes, selected profile ID, prompt ID/version, effective runtime settings, input hashes, rendered prompt hash, and timing information as appropriate.
|
|
||||||
9. **Return PreparedRun**: Return a `PreparedRun` containing the rendered messages, effective runtime settings, resolved metadata, and input/prompt hashes.
|
|
||||||
|
|
||||||
The prepare path must not call the LLM.
|
|
||||||
|
|
||||||
The prepare path must not validate model output, because there is no model output.
|
|
||||||
|
|
||||||
The prepare path must not perform structured-output repair, because repair only applies after model output exists.
|
|
||||||
|
|
||||||
The prepare path should not resolve or expose raw API key values. It may include the selected `api_key_env` name in effective runtime settings or metadata, but never the environment variable value.
|
|
||||||
|
|
||||||
### 3.2 Full Run Data Flow
|
|
||||||
|
|
||||||
The `Runner.Run(ctx, RunRequest)` flow should reuse the prepare path:
|
|
||||||
|
|
||||||
1. **Prepare**: Call the shared prepare flow to load the prompt, select the profile, resolve artifacts, render prompt messages, and compute pre-run metadata.
|
|
||||||
2. **Call LLM**: Execute the generation request using the effective runtime settings from the prepared run.
|
|
||||||
3. **Build Output Artifact**: Convert the model response into the configured output artifact.
|
|
||||||
4. **Validate Output**:
|
|
||||||
- Validate the model output against the prompt definition's output contract.
|
|
||||||
- Validation content failures remain successful run results with `validation.status=failed`.
|
|
||||||
- Validator runtime/config errors are run errors.
|
|
||||||
5. **Repair If Configured**:
|
|
||||||
- If structured validation fails and `repair_attempts > 0`, perform bounded repair attempts.
|
|
||||||
- Re-validate after each repair attempt.
|
|
||||||
- Repair loops must remain strictly bounded.
|
|
||||||
6. **Return RunResult**: Produce a `RunResult` containing the final artifact, validation status, raw model output, usage information, and metadata.
|
|
||||||
|
|
||||||
`Runner.Run` should not duplicate profile selection, artifact resolution, or prompt rendering logic that already exists in `Runner.Prepare`.
|
|
||||||
|
|
||||||
## 4. Domain Model
|
|
||||||
|
|
||||||
Key domain types:
|
|
||||||
|
|
||||||
- `PromptDefinition`: Defines the "what" of the task: templates, inputs, output contract, validation settings, repair settings, and optional `default_profile`.
|
|
||||||
- `ExecutionProfile`: Defines the "how" of execution: endpoint, model, generation parameters, timeout, reasoning effort, and `api_key_env`.
|
|
||||||
- `RunRequest`: The intent to execute or prepare a prompt, including `prompt_id`, optional `profile_id`, inputs, variables, and optional runtime overrides.
|
|
||||||
- `PreparedRun`: The result of the prepare/render phase. Contains rendered messages, effective runtime settings, selected profile ID, prompt metadata, input hashes, prompt hash, and other pre-LLM metadata.
|
|
||||||
- `RunResult`: The result of a full run. Contains the generated `Artifact`, `ValidationResult`, raw model output, token usage, and `RunMetadata`.
|
|
||||||
- `RunMetadata`: Detailed tracing information, including prompt ID/version, selected profile ID, effective model parameters, usage tokens, hashes, timestamps, validation status, and repair attempts where applicable.
|
|
||||||
- `RenderedPrompt`: Provider-neutral rendered prompt structure.
|
|
||||||
- `RenderedMessage`: Provider-neutral rendered message with role and content.
|
|
||||||
- `ArtifactRef`: A reference to an input artifact, such as `file` or `inline`.
|
|
||||||
- `Artifact`: Loaded artifact content with name, content type, body, source URI, size, and hash.
|
|
||||||
- `ValidationResult`: Validation status and details for full runs.
|
|
||||||
|
|
||||||
`PreparedRun` should be serializable for JSON output and should also be representable in a human-readable text format.
|
|
||||||
|
|
||||||
## 5. Interfaces and Adapters
|
|
||||||
|
|
||||||
### Primary Ports
|
|
||||||
|
|
||||||
- `promptdef.Repository`: Lookup for prompt definitions.
|
|
||||||
- `profile.Repository`: Lookup for execution profiles.
|
|
||||||
- `artifact.Reader`: Loading of artifact content.
|
|
||||||
- `prompt.Renderer`: Template rendering.
|
|
||||||
- `llm.Client`: Model generation.
|
|
||||||
- `validate.Validator`: Output validation.
|
|
||||||
- `format.PreparedRunFormatter` or equivalent: Optional formatting abstraction for rendered/prepared output.
|
|
||||||
|
|
||||||
### Current Adapters
|
|
||||||
|
|
||||||
- **Repositories**: Filesystem YAML loaders for both prompts and profiles.
|
|
||||||
- **Artifact Reader**: Composite reader supporting `file` and `inline`.
|
|
||||||
- **Prompt Renderer**: Go templates with a custom `input` helper.
|
|
||||||
- **LLM Client**: OpenAI-compatible `/chat/completions` over HTTP.
|
|
||||||
- **Validator**: Standard validator supporting `none`, `basic`, `json`, and `json_schema`.
|
|
||||||
- **CLI Adapter**: Supports `run`, `render`, and `serve`.
|
|
||||||
- **HTTP Adapter**: Supports full run execution through `POST /v1/runs`.
|
|
||||||
|
|
||||||
## 6. Public Contracts
|
|
||||||
|
|
||||||
### CLI
|
|
||||||
|
|
||||||
Scriptorium should expose at least these commands:
|
|
||||||
|
|
||||||
- `scriptorium run`: Executes a prompt by preparing it, calling the LLM, validating output, optionally repairing structured output, and returning an artifact.
|
|
||||||
- `scriptorium render`: Prepares and renders a prompt without calling the LLM.
|
|
||||||
- `scriptorium serve`: Starts the HTTP API using infrastructure-only flags.
|
|
||||||
|
|
||||||
### `scriptorium run`
|
|
||||||
|
|
||||||
`run` uses flags such as:
|
|
||||||
|
|
||||||
- `--prompt-dir`
|
|
||||||
- `--profile-dir`
|
|
||||||
- `--prompt`
|
|
||||||
- `--profile`
|
|
||||||
- `--input`
|
|
||||||
- `--var`
|
|
||||||
- `--out`
|
|
||||||
- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.
|
|
||||||
|
|
||||||
`run` should produce the generated artifact as its primary output.
|
|
||||||
|
|
||||||
### `scriptorium render`
|
|
||||||
|
|
||||||
`render` prepares and renders a prompt without calling an LLM.
|
|
||||||
|
|
||||||
It should use the same prompt/profile/input/variable/runtime override flags as `run` where applicable:
|
|
||||||
|
|
||||||
- `--prompt-dir`
|
|
||||||
- `--profile-dir`
|
|
||||||
- `--prompt`
|
|
||||||
- `--profile`
|
|
||||||
- `--input`
|
|
||||||
- `--var`
|
|
||||||
- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.
|
|
||||||
- `--format`, with initial support for `text` and `json`.
|
|
||||||
|
|
||||||
Default render output format should be `text`.
|
|
||||||
|
|
||||||
`render` must not call the LLM.
|
|
||||||
|
|
||||||
`render` should show the same rendered messages and effective runtime settings that `run` would use.
|
|
||||||
|
|
||||||
`render` should include enough information to debug:
|
|
||||||
|
|
||||||
- prompt ID
|
|
||||||
- prompt version
|
|
||||||
- selected profile ID
|
|
||||||
- effective runtime settings
|
|
||||||
- input hashes
|
|
||||||
- prompt hash
|
|
||||||
- rendered messages
|
|
||||||
|
|
||||||
`render` must not include resolved API key values.
|
|
||||||
|
|
||||||
`render` may include the `api_key_env` name.
|
|
||||||
|
|
||||||
### Render Output Formats
|
|
||||||
|
|
||||||
Initial render output formats:
|
|
||||||
|
|
||||||
- `text`: Human-readable default format.
|
|
||||||
- `json`: Machine-readable structured representation of the prepared run.
|
|
||||||
|
|
||||||
Additional formats, such as `markdown`, may be added later.
|
|
||||||
|
|
||||||
Render output formatting should be modular. Adding a new output format should not require changing the prepare/run core logic.
|
|
||||||
|
|
||||||
The output formatting layer should consume a `PreparedRun` and produce bytes or text for the adapter. It should not reload prompts, re-resolve artifacts, re-render templates, call the LLM, or perform validation.
|
|
||||||
|
|
||||||
### `scriptorium serve`
|
|
||||||
|
|
||||||
`serve` starts the HTTP API.
|
|
||||||
|
|
||||||
It should use infrastructure-only flags such as:
|
|
||||||
|
|
||||||
- `--addr`
|
|
||||||
- `--prompt-dir`
|
|
||||||
- `--profile-dir`
|
|
||||||
- `--schema-dir`
|
|
||||||
|
|
||||||
`serve` should not introduce a server-level model/runtime precedence layer unless explicitly documented and intentionally implemented.
|
|
||||||
|
|
||||||
### HTTP API
|
|
||||||
|
|
||||||
Current HTTP API:
|
|
||||||
|
|
||||||
- `POST /v1/runs`: Accepts `RunRequest` JSON and returns `RunResponse` JSON. No built-in auth.
|
|
||||||
|
|
||||||
Request may include runtime overrides under `model` and an `include_raw_output` boolean.
|
|
||||||
|
|
||||||
`raw_model_output` is exposed only when explicitly requested with `include_raw_output=true`.
|
|
||||||
|
|
||||||
A future HTTP prepare/render endpoint may be added, such as `POST /v1/renders` or `POST /v1/runs/prepare`, but the initial render feature may be CLI-only. If added later, it should call the same usecase-level prepare path as `scriptorium render`.
|
|
||||||
|
|
||||||
### YAML Shapes
|
|
||||||
|
|
||||||
Prompt YAML includes:
|
|
||||||
|
|
||||||
- `id`
|
|
||||||
- `version`
|
|
||||||
- optional `default_profile`
|
|
||||||
- `inputs`
|
|
||||||
- `messages`
|
|
||||||
- `output`
|
|
||||||
|
|
||||||
Inputs support:
|
|
||||||
|
|
||||||
- `name`
|
|
||||||
- `required`
|
|
||||||
- optional `content_type`
|
|
||||||
- `description`
|
|
||||||
|
|
||||||
Messages require:
|
|
||||||
|
|
||||||
- `role`
|
|
||||||
- exactly one of `content` or `content_file`
|
|
||||||
|
|
||||||
`content_file` resolves relative to the prompt YAML location.
|
|
||||||
|
|
||||||
Profile YAML includes:
|
|
||||||
|
|
||||||
- `id`
|
|
||||||
- `endpoint`
|
|
||||||
- `model`
|
|
||||||
- generation parameters
|
|
||||||
- timeout settings
|
|
||||||
- reasoning settings
|
|
||||||
- `api_key_env`
|
|
||||||
|
|
||||||
Prompt content must not appear in profile YAML.
|
|
||||||
|
|
||||||
Model/runtime/API-key settings must not appear in prompt YAML, except that prompt YAML may specify `default_profile`.
|
|
||||||
|
|
||||||
## 7. Render Feature Design
|
|
||||||
|
|
||||||
The render feature is a first-class use case, not a CLI-only shortcut.
|
|
||||||
|
|
||||||
### Goals
|
|
||||||
|
|
||||||
The render feature should help users:
|
|
||||||
|
|
||||||
- inspect fully rendered prompt messages
|
|
||||||
- debug missing inputs
|
|
||||||
- verify template variable substitution
|
|
||||||
- verify selected profile resolution
|
|
||||||
- verify runtime override precedence
|
|
||||||
- verify file-backed prompt loading
|
|
||||||
- inspect input hashes and prompt hashes
|
|
||||||
- prepare for future token budgeting and prompt-size inspection
|
|
||||||
|
|
||||||
### Non-Goals
|
|
||||||
|
|
||||||
The render feature should not:
|
|
||||||
|
|
||||||
- call an LLM
|
|
||||||
- validate model output
|
|
||||||
- repair structured output
|
|
||||||
- resolve or print API key values
|
|
||||||
- mutate artifacts
|
|
||||||
- save outputs to artifact storage unless a future explicit output option is added
|
|
||||||
- become an orchestration step manager
|
|
||||||
|
|
||||||
### Usecase Shape
|
|
||||||
|
|
||||||
The preferred usecase shape is:
|
|
||||||
|
|
||||||
- `Runner.Prepare(ctx, RunRequest) (*PreparedRun, error)`
|
|
||||||
- `Runner.Run(ctx, RunRequest) (*RunResult, error)`
|
|
||||||
|
|
||||||
`Runner.Run` should call `Runner.Prepare`.
|
|
||||||
|
|
||||||
The prepare flow should be the only implementation of:
|
|
||||||
|
|
||||||
- prompt loading
|
|
||||||
- profile selection
|
|
||||||
- runtime override resolution
|
|
||||||
- artifact resolution
|
|
||||||
- prompt rendering
|
|
||||||
- pre-run metadata/hash calculation
|
|
||||||
|
|
||||||
### CLI Shape
|
|
||||||
|
|
||||||
The preferred command name is `render`.
|
|
||||||
|
|
||||||
The command should support:
|
|
||||||
|
|
||||||
- `--format text`
|
|
||||||
- `--format json`
|
|
||||||
|
|
||||||
Default format:
|
|
||||||
|
|
||||||
- `text`
|
|
||||||
|
|
||||||
Unknown formats should produce a clear error.
|
|
||||||
|
|
||||||
Formatting should be centralized through a small formatter registry, strategy, switch, or interface so new formats can be added without modifying usecase logic.
|
|
||||||
|
|
||||||
### Text Output Expectations
|
|
||||||
|
|
||||||
Text output should be optimized for human inspection.
|
|
||||||
|
|
||||||
It should include, at minimum:
|
|
||||||
|
|
||||||
- prompt ID and version
|
|
||||||
- selected profile ID
|
|
||||||
- model name
|
|
||||||
- endpoint
|
|
||||||
- effective generation settings
|
|
||||||
- input hashes
|
|
||||||
- rendered prompt hash
|
|
||||||
- rendered messages grouped by role
|
|
||||||
|
|
||||||
Text output should be readable and deterministic enough for tests.
|
|
||||||
|
|
||||||
It should not include raw API key values.
|
|
||||||
|
|
||||||
### JSON Output Expectations
|
|
||||||
|
|
||||||
JSON output should be a structured representation of `PreparedRun` or a DTO derived from it.
|
|
||||||
|
|
||||||
It should include, at minimum:
|
|
||||||
|
|
||||||
- prompt ID and version
|
|
||||||
- selected profile ID
|
|
||||||
- effective runtime settings
|
|
||||||
- input hashes
|
|
||||||
- rendered prompt hash
|
|
||||||
- rendered messages
|
|
||||||
|
|
||||||
JSON output should not include raw API key values.
|
|
||||||
|
|
||||||
JSON output should remain stable enough to be useful for automation and integration tests.
|
|
||||||
|
|
||||||
## 8. Guardrails
|
|
||||||
|
|
||||||
- **Separation of Concerns**: Prompt content must not belong in execution profiles; model/API settings must not belong in prompt definitions.
|
|
||||||
- **Security**: Raw API keys are unsupported in all configuration and transport layers. Only `api_key_env` is used.
|
|
||||||
- **Secret Handling**: Resolved API key values must never appear in rendered output, metadata, logs, HTTP responses, or CLI output.
|
|
||||||
- **Path Resolution**: `content_file` paths in prompt definitions resolve relative to the prompt YAML file.
|
|
||||||
- **Integrity**: No silent prompt truncation or omission of content.
|
|
||||||
- **Reliability**: Repair loops are strictly bounded by `repair_attempts`.
|
|
||||||
- **No Orchestration Creep**: Scriptorium prepares and executes a single prompt request. It does not coordinate multi-stage workflows.
|
|
||||||
- **Render Reuse**: The full run path must reuse the prepare/render path to avoid divergent behavior.
|
|
||||||
- **Formatter Isolation**: Render output formatters must not perform usecase work. They only format a completed `PreparedRun`.
|
|
||||||
|
|
||||||
## 9. Testing Strategy
|
|
||||||
|
|
||||||
Tests should protect both the run path and the prepare/render path.
|
|
||||||
|
|
||||||
### Prepare / Render Tests
|
|
||||||
|
|
||||||
Add tests for:
|
|
||||||
|
|
||||||
- preparing a prompt with explicit profile selection
|
|
||||||
- preparing a prompt using `default_profile`
|
|
||||||
- failing when no explicit profile and no `default_profile` exist
|
|
||||||
- runtime overrides beating profile values
|
|
||||||
- profile values beating application defaults
|
|
||||||
- file-backed prompt bodies rendering correctly
|
|
||||||
- required inputs failing when missing
|
|
||||||
- optional inputs being absent when not referenced
|
|
||||||
- unknown input references failing
|
|
||||||
- input hashes being included
|
|
||||||
- rendered prompt hash being included
|
|
||||||
- effective runtime settings being included
|
|
||||||
- `api_key_env` name being included where appropriate
|
|
||||||
- resolved API key values never appearing in `PreparedRun`
|
|
||||||
- prepare path not calling the LLM
|
|
||||||
|
|
||||||
### CLI Render Tests
|
|
||||||
|
|
||||||
Add tests for:
|
|
||||||
|
|
||||||
- `scriptorium render` mapping flags into `RunRequest`
|
|
||||||
- default text output
|
|
||||||
- explicit `--format text`
|
|
||||||
- explicit `--format json`
|
|
||||||
- unknown format failure
|
|
||||||
- text output includes prompt/profile/messages
|
|
||||||
- JSON output includes prompt/profile/messages
|
|
||||||
- rendered output never includes resolved API key values
|
|
||||||
|
|
||||||
### Run Reuse Tests
|
|
||||||
|
|
||||||
Add tests proving:
|
|
||||||
|
|
||||||
- `Runner.Run` reuses prepare behavior
|
|
||||||
- run and render resolve the same prompt/profile/runtime settings for equivalent inputs
|
|
||||||
- run still validates output
|
|
||||||
- run still performs bounded repair where configured
|
|
||||||
|
|
||||||
### Existing Tests
|
|
||||||
|
|
||||||
Continue testing:
|
|
||||||
|
|
||||||
- prompt definition loading
|
|
||||||
- execution profile loading
|
|
||||||
- artifact loading
|
|
||||||
- prompt rendering
|
|
||||||
- LLM adapter behavior
|
|
||||||
- validation behavior
|
|
||||||
- HTTP request/response mapping
|
|
||||||
|
|
||||||
## 10. Extension Points
|
|
||||||
|
|
||||||
Future work should remain grounded in the current architecture:
|
|
||||||
|
|
||||||
- **Artifacts**: Add S3 artifact references via a new `artifact.Reader`.
|
|
||||||
- **LLM**: Implement additional provider adapters, such as Anthropic or Google.
|
|
||||||
- **Execution**: Add token budgeting, streaming generation, and batch execution capabilities.
|
|
||||||
- **Prepare/Render**: Add token estimates, prompt-size summaries, or additional render output formats.
|
|
||||||
- **Repositories**: Implement database-backed repositories for prompts and profiles.
|
|
||||||
- **Profiles**: Support more granular profile versioning and environment-specific profiles.
|
|
||||||
- **HTTP**: Add an HTTP prepare/render endpoint if Narratio or another caller needs it.
|
|
||||||
|
|
||||||
Future render formats should plug into the formatter layer and should not require changes to the usecase layer.
|
|
||||||
148
docs/cli.md
Normal file
148
docs/cli.md
Normal file
@@ -0,0 +1,148 @@
|
|||||||
|
# CLI Reference
|
||||||
|
|
||||||
|
## Shortest Useful Command
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium render \
|
||||||
|
--config ./examples/config.yml \
|
||||||
|
--prompt generic.markdown_summary \
|
||||||
|
--input transcript=./examples/fixtures/transcript.md \
|
||||||
|
--input glossary=./examples/fixtures/glossary.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
`render` prepares and formats the prompt without calling an LLM.
|
||||||
|
|
||||||
|
## Command Overview
|
||||||
|
|
||||||
|
- `scriptorium run`: prepare prompt, call the configured LLM, write generated output, print a run summary.
|
||||||
|
- `scriptorium render`: prepare prompt only; write prepared-run output as `text` or `json`.
|
||||||
|
- `scriptorium serve`: start the HTTP server.
|
||||||
|
|
||||||
|
Integration references:
|
||||||
|
|
||||||
|
- [HTTP contract](integrations/http-api.md)
|
||||||
|
- [Narratio subprocess contract](integrations/narratio.md)
|
||||||
|
|
||||||
|
## Common Argument Rules
|
||||||
|
|
||||||
|
- `--config` is supported by `run`, `render`, and `serve`.
|
||||||
|
- `run` and `render` require:
|
||||||
|
- `--prompt`
|
||||||
|
- at least one `--input`
|
||||||
|
- an effective `prompt_dir` and `profile_dir` (from flags or config)
|
||||||
|
- `serve` requires an effective `prompt_dir` and `profile_dir` (from flags or config).
|
||||||
|
- Positional arguments are rejected.
|
||||||
|
|
||||||
|
## Flag Reference
|
||||||
|
|
||||||
|
### `scriptorium run`
|
||||||
|
|
||||||
|
- `--config <path>`: app config file path.
|
||||||
|
- `--prompt-dir <dir>`: prompt definition directory.
|
||||||
|
- `--profile-dir <dir>`: profile definition directory.
|
||||||
|
- `--schema-dir <dir>`: schema base directory for `json_schema` validation.
|
||||||
|
- `--prompt <id>`: prompt ID to execute. Required.
|
||||||
|
- `--prompt-id <id>`: deprecated alias for `--prompt`.
|
||||||
|
- `--profile <id>`: explicit profile override.
|
||||||
|
- `--profile-id <id>`: deprecated alias for `--profile`.
|
||||||
|
- `--input name=path`: input mapping (repeatable, comma-separated accepted).
|
||||||
|
- `--var name=value`: template variable mapping (repeatable, comma-separated accepted).
|
||||||
|
- `--out <path>`: write artifact body to file instead of stdout.
|
||||||
|
- `--llm-base-url <url>`: runtime endpoint override.
|
||||||
|
- `--model <name>`: runtime model override.
|
||||||
|
- `--api-key-env <name>`: runtime API key environment-variable name override.
|
||||||
|
- `--temperature <float>`: runtime temperature override.
|
||||||
|
- `--max-tokens <int>`: runtime max tokens override.
|
||||||
|
- `--top-p <float>`: runtime top-p override.
|
||||||
|
- `--timeout <duration>`: runtime timeout override (Go duration syntax, for example `30s`, `2m`).
|
||||||
|
|
||||||
|
### `scriptorium render`
|
||||||
|
|
||||||
|
- Supports the same flags as `run`, except:
|
||||||
|
- no `--schema-dir` flag.
|
||||||
|
- Adds:
|
||||||
|
- `--format text|json`: prepared-run output format.
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
- `render` still resolves profile and runtime settings.
|
||||||
|
- `render` still validates that `api_key_env` exists if the selected profile or overrides require it.
|
||||||
|
|
||||||
|
### `scriptorium serve`
|
||||||
|
|
||||||
|
- `--config <path>`: app config file path.
|
||||||
|
- `--addr <listen-address>`: HTTP listen address.
|
||||||
|
- `--prompt-dir <dir>`: prompt definition directory.
|
||||||
|
- `--profile-dir <dir>`: profile definition directory.
|
||||||
|
- `--schema-dir <dir>`: schema base directory for `json_schema` validation.
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
- `serve` does not accept runtime model override flags such as `--model` or `--llm-base-url`.
|
||||||
|
|
||||||
|
## Input And Variable Syntax
|
||||||
|
|
||||||
|
- `--input name=path` maps prompt input names to local file paths.
|
||||||
|
- `--var name=value` maps template variable names to values.
|
||||||
|
- Both flags can be repeated.
|
||||||
|
- Both flags also support comma-separated batches, for example:
|
||||||
|
- `--input transcript=./t.md,glossary=./g.yml`
|
||||||
|
- `--var session_id=42,session_date=2026-05-04`
|
||||||
|
|
||||||
|
## Output Behavior
|
||||||
|
|
||||||
|
`run`:
|
||||||
|
- Writes generated artifact content to stdout by default.
|
||||||
|
- Writes generated artifact content to `--out` when provided.
|
||||||
|
- Prints run summary metadata to stderr on success.
|
||||||
|
- Prints errors to stderr on failure.
|
||||||
|
|
||||||
|
`render`:
|
||||||
|
- Writes prepared-run output to stdout by default.
|
||||||
|
- Writes prepared-run output to `--out` when provided.
|
||||||
|
- Does not print a success summary line.
|
||||||
|
|
||||||
|
`serve`:
|
||||||
|
- Logs startup and server errors to stderr.
|
||||||
|
|
||||||
|
## Exit Codes
|
||||||
|
|
||||||
|
- `0`: success.
|
||||||
|
- `1`: runtime/parse/config/load/render/generation/output-write error.
|
||||||
|
- `2`: `run` completed, output was generated, but validation status is `failed`.
|
||||||
|
|
||||||
|
When `run` exits `2`, output may already be written to stdout or `--out`.
|
||||||
|
|
||||||
|
## Common Workflows
|
||||||
|
|
||||||
|
Render prompt inputs and template variables as JSON:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium render \
|
||||||
|
--config ./examples/config.yml \
|
||||||
|
--prompt generic.markdown_summary \
|
||||||
|
--input transcript=./examples/fixtures/transcript.md \
|
||||||
|
--input glossary=./examples/fixtures/glossary.yml \
|
||||||
|
--var session_date=2026-05-04 \
|
||||||
|
--format json
|
||||||
|
```
|
||||||
|
|
||||||
|
Run a prompt with profile override and file output:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium run \
|
||||||
|
--config ./examples/config.yml \
|
||||||
|
--prompt generic.markdown_summary \
|
||||||
|
--profile local-fast \
|
||||||
|
--input transcript=./examples/fixtures/transcript.md \
|
||||||
|
--input glossary=./examples/fixtures/glossary.yml \
|
||||||
|
--out ./summary.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Start the HTTP server with explicit config:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium serve --config ./examples/config.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Copyable example script:
|
||||||
|
|
||||||
|
- `examples/render-markdown-summary.sh`
|
||||||
231
docs/config.md
Normal file
231
docs/config.md
Normal file
@@ -0,0 +1,231 @@
|
|||||||
|
# Configuration Reference
|
||||||
|
|
||||||
|
## Config Discovery And Precedence
|
||||||
|
|
||||||
|
Application settings are loaded in this order:
|
||||||
|
|
||||||
|
1. Built-in defaults
|
||||||
|
2. `config.yml` values
|
||||||
|
3. CLI overrides
|
||||||
|
|
||||||
|
When `--config` is not provided, Scriptorium searches for config files in this order:
|
||||||
|
|
||||||
|
1. `/usr/local/etc/scriptorium/config.yml`
|
||||||
|
2. `/etc/scriptorium/config.yml`
|
||||||
|
|
||||||
|
If neither file exists, Scriptorium continues with built-in defaults.
|
||||||
|
|
||||||
|
When `--config <path>` is provided, that file is required.
|
||||||
|
|
||||||
|
## Minimal App Config
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
prompt_dir: ./examples/prompts
|
||||||
|
profile_dir: ./examples/profiles
|
||||||
|
```
|
||||||
|
|
||||||
|
This is enough to use `run` and `render` when prompt/profile files are valid.
|
||||||
|
|
||||||
|
## Production-Oriented App Config
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
prompt_dir: /opt/scriptorium/prompts
|
||||||
|
profile_dir: /opt/scriptorium/profiles
|
||||||
|
schema_dir: /opt/scriptorium/schemas
|
||||||
|
|
||||||
|
server:
|
||||||
|
addr: 127.0.0.1:8080
|
||||||
|
|
||||||
|
defaults:
|
||||||
|
render_format: text
|
||||||
|
```
|
||||||
|
|
||||||
|
## App Config File (`config.yml`)
|
||||||
|
|
||||||
|
Top-level fields:
|
||||||
|
|
||||||
|
- `prompt_dir` (optional): default prompt definition directory.
|
||||||
|
- `profile_dir` (optional): default profile definition directory.
|
||||||
|
- `schema_dir` (optional): base directory for schema files used by `json_schema` validation.
|
||||||
|
- `server.addr` (optional): default listen address for `serve`.
|
||||||
|
- `defaults.render_format` (optional): default `render` output format (`text` or `json`).
|
||||||
|
|
||||||
|
Built-in defaults:
|
||||||
|
|
||||||
|
- `schema_dir`: `.`
|
||||||
|
- `server.addr`: `:8080`
|
||||||
|
- `defaults.render_format`: `text`
|
||||||
|
|
||||||
|
Validation behavior:
|
||||||
|
|
||||||
|
- Config decoding is strict; unknown YAML fields are rejected.
|
||||||
|
- Raw API key fields are not supported in `config.yml`.
|
||||||
|
|
||||||
|
## Prompt Definition Files
|
||||||
|
|
||||||
|
Prompt definitions are YAML files anywhere under `prompt_dir`, including nested subdirectories.
|
||||||
|
|
||||||
|
Subdirectories are organizational only. Callers still select prompts by the YAML `id`, not by file path. For example, `prompts/dnd/recap.yaml` may still declare `id: dnd.recap`, and callers use `--prompt dnd.recap`.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
id: generic.structured_events
|
||||||
|
version: "1.0.0"
|
||||||
|
default_profile: local-quality
|
||||||
|
description: Produce structured event JSON from a transcript.
|
||||||
|
|
||||||
|
inputs:
|
||||||
|
- name: transcript
|
||||||
|
required: true
|
||||||
|
content_type: text/markdown
|
||||||
|
description: Source transcript content
|
||||||
|
- name: glossary
|
||||||
|
required: false
|
||||||
|
content_type: text/yaml
|
||||||
|
description: Optional glossary context
|
||||||
|
|
||||||
|
messages:
|
||||||
|
- role: system
|
||||||
|
content_file: ./generic.structured_events.system.md
|
||||||
|
- role: user
|
||||||
|
content_file: ./generic.structured_events.user.md
|
||||||
|
|
||||||
|
output:
|
||||||
|
format: json
|
||||||
|
validation_mode: json_schema
|
||||||
|
schema_path: structured_events.schema.json
|
||||||
|
repair_attempts: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
Field reference:
|
||||||
|
|
||||||
|
- `id` (required): prompt identifier.
|
||||||
|
- `version` (required): prompt version.
|
||||||
|
- `default_profile` (optional): profile ID used when request does not provide `profile_id`.
|
||||||
|
- `description` (optional): prompt description.
|
||||||
|
- `inputs` (optional list): expected named inputs.
|
||||||
|
- `messages` (required list): prompt message templates.
|
||||||
|
- `output` (required object): output contract.
|
||||||
|
|
||||||
|
`inputs[]` fields:
|
||||||
|
|
||||||
|
- `name` (required)
|
||||||
|
- `required` (optional, boolean)
|
||||||
|
- `content_type` (optional metadata)
|
||||||
|
- `description` (optional)
|
||||||
|
|
||||||
|
`messages[]` fields:
|
||||||
|
|
||||||
|
- `role` (required)
|
||||||
|
- `content` or `content_file` (exactly one is required)
|
||||||
|
|
||||||
|
Message rules:
|
||||||
|
|
||||||
|
- Repeated roles are allowed.
|
||||||
|
- `content_file` is resolved relative to the prompt YAML file location.
|
||||||
|
- Nested prompt files keep the same relative `content_file` behavior; `./recap.user.md` next to `dnd/recap.yaml` resolves from `dnd/`.
|
||||||
|
- Prompt decoding is strict; unknown YAML fields are rejected.
|
||||||
|
- Duplicate prompt IDs are invalid. If multiple files declare the requested prompt ID, Scriptorium fails instead of choosing one.
|
||||||
|
|
||||||
|
`output` fields:
|
||||||
|
|
||||||
|
- `format` (required): `text`, `markdown`, or `json`.
|
||||||
|
- `validation_mode` (required): `none`, `basic`, `json`, or `json_schema`.
|
||||||
|
- `schema_path` (required when `validation_mode: json_schema`).
|
||||||
|
- `repair_attempts` (required): integer `>= 0`.
|
||||||
|
|
||||||
|
Repair behavior boundary:
|
||||||
|
|
||||||
|
- `repair_attempts` is part of the prompt contract.
|
||||||
|
- CLI and HTTP currently construct the runner without a repairer, so normal `run`/`serve` execution does not perform output repair attempts.
|
||||||
|
|
||||||
|
## Profile Definition Files
|
||||||
|
|
||||||
|
Execution profiles are YAML files anywhere under `profile_dir`, including nested subdirectories.
|
||||||
|
|
||||||
|
Subdirectories are organizational only. Callers still select profiles by the YAML `id`, not by file path. For example, `profiles/local/local-quality.yaml` may still declare `id: local-quality`, and callers use `--profile local-quality`.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
id: local-fast
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: gpt-4o-mini
|
||||||
|
temperature: 0.2
|
||||||
|
max_tokens: 500
|
||||||
|
top_p: 1.0
|
||||||
|
timeout_seconds: 90
|
||||||
|
api_key_env: SCRIPTORIUM_API_KEY
|
||||||
|
service_tier: priority
|
||||||
|
```
|
||||||
|
|
||||||
|
Field reference:
|
||||||
|
|
||||||
|
- `id` (required)
|
||||||
|
- `endpoint` (required)
|
||||||
|
- `model` (required)
|
||||||
|
- `temperature` (optional): range `0..2`
|
||||||
|
- `max_tokens` (optional): `>= 0`
|
||||||
|
- `top_p` (optional): range `0..1`
|
||||||
|
- `timeout_seconds` (optional): `>= 0`
|
||||||
|
- `service_tier` (optional): provider-specific request tier such as OpenRouter `flex` or `priority`
|
||||||
|
- `reasoning_effort` (optional)
|
||||||
|
- `api_key_env` (optional)
|
||||||
|
- `extra_params` (optional map of strings)
|
||||||
|
|
||||||
|
Profile rules:
|
||||||
|
|
||||||
|
- Profile decoding is strict; unknown YAML fields are rejected.
|
||||||
|
- Raw `api_key` is rejected; use `api_key_env`.
|
||||||
|
- If `api_key_env` is set, that environment variable must be set when preparing/running.
|
||||||
|
- Duplicate profile IDs are invalid. If multiple files declare the requested profile ID, Scriptorium fails instead of choosing one.
|
||||||
|
|
||||||
|
Current outbound request behavior:
|
||||||
|
|
||||||
|
- The OpenAI-compatible client currently serializes: `model`, `messages`, `temperature`, `max_tokens`, `top_p`, `service_tier`, and optional `response_format` for `json_schema` prompts.
|
||||||
|
- `reasoning_effort` and `extra_params` are parsed and carried in effective settings, but are not currently serialized into outbound chat-completions requests.
|
||||||
|
|
||||||
|
## Schema Behavior
|
||||||
|
|
||||||
|
Schemas are JSON files, typically in `schema_dir`.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- `output.validation_mode: json_schema` requires `output.schema_path`.
|
||||||
|
- Relative `schema_path` values resolve from `schema_dir`, including explicit nested paths such as `dnd/structured_events.schema.json`.
|
||||||
|
- Absolute `schema_path` values are used directly.
|
||||||
|
- Scriptorium does not recursively search schemas by basename; nested schemas must be referenced by their relative path.
|
||||||
|
- Missing or invalid schema documents cause runtime validation errors.
|
||||||
|
- Invalid generated JSON causes validation status `failed` (not a runtime error).
|
||||||
|
|
||||||
|
Supported artifact reference types for request inputs are `file` and `inline`.
|
||||||
|
|
||||||
|
## Secrets Handling
|
||||||
|
|
||||||
|
- Keep secret values in environment variables.
|
||||||
|
- Store only environment-variable names in profile `api_key_env`.
|
||||||
|
- Do not put raw API keys in config, prompts, profiles, CLI flags, or HTTP request bodies.
|
||||||
|
|
||||||
|
## Maintained Examples
|
||||||
|
|
||||||
|
- App config: `examples/config.yml`
|
||||||
|
- Prompt examples: `examples/prompts/`
|
||||||
|
- Profile examples: `examples/profiles/`
|
||||||
|
- Schema examples: `examples/schemas/`
|
||||||
|
- Input fixtures: `examples/fixtures/`
|
||||||
|
- Render example script: `examples/render-markdown-summary.sh`
|
||||||
|
- HTTP request example: `examples/http-run.json`
|
||||||
|
|
||||||
|
Example organizational layout:
|
||||||
|
|
||||||
|
```text
|
||||||
|
examples/prompts/dnd/recap.yaml
|
||||||
|
examples/profiles/local/local-quality.yaml
|
||||||
|
examples/schemas/dnd/structured_events.schema.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Integration References
|
||||||
|
|
||||||
|
- [Inbound HTTP contract](integrations/http-api.md)
|
||||||
|
- [Outbound OpenAI-compatible contract](integrations/openai-compatible-chat.md)
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
# Main `config.yml`
|
|
||||||
|
|
||||||
`config.yml` defines application-level defaults used by CLI commands.
|
|
||||||
|
|
||||||
By default, Scriptorium looks for `/usr/local/etc/scriptorium/config.yml` and, if not present, then for `/etc/scriptorium/config.yml`. You can also pass `--config PATH`.
|
|
||||||
|
|
||||||
## Complete Example
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
prompt_dir: ./prompts
|
|
||||||
profile_dir: ./profiles
|
|
||||||
schema_dir: ./schemas
|
|
||||||
|
|
||||||
server:
|
|
||||||
addr: :8080
|
|
||||||
|
|
||||||
defaults:
|
|
||||||
render_format: text
|
|
||||||
```
|
|
||||||
|
|
||||||
## Available Options
|
|
||||||
|
|
||||||
- `prompt_dir` (optional): Default directory for prompt definition YAML files.
|
|
||||||
- `profile_dir` (optional): Default directory for execution profile YAML files.
|
|
||||||
- `schema_dir` (optional): Base directory for JSON schema files used by `json_schema` validation.
|
|
||||||
|
|
||||||
### `server`
|
|
||||||
|
|
||||||
- `addr` (optional): HTTP server listen address for `scriptorium serve`.
|
|
||||||
|
|
||||||
### `defaults`
|
|
||||||
|
|
||||||
- `render_format` (optional): Default output format for `scriptorium render`.
|
|
||||||
- Allowed values: `text`, `json`.
|
|
||||||
|
|
||||||
## Precedence
|
|
||||||
|
|
||||||
For run/render/serve settings, precedence is:
|
|
||||||
|
|
||||||
1. Explicit CLI flags
|
|
||||||
2. `config.yml`
|
|
||||||
3. Built-in defaults
|
|
||||||
|
|
||||||
## Notes and Rules
|
|
||||||
|
|
||||||
- Unknown YAML fields fail to load (strict decoding).
|
|
||||||
- This file does not accept API keys.
|
|
||||||
- `config.yml` sets directory/server defaults only; prompt/profile content remains in their own files.
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
# Execution Profile Definitions
|
|
||||||
|
|
||||||
Execution Profiles define **how** Scriptorium calls an LLM endpoint.
|
|
||||||
|
|
||||||
A profile file is YAML, typically stored under `profiles/`, for example `profiles/local-quality.yaml`.
|
|
||||||
|
|
||||||
## Complete Example
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
id: local-quality
|
|
||||||
endpoint: http://localhost:8000/v1
|
|
||||||
model: gpt-4.1
|
|
||||||
temperature: 0.0
|
|
||||||
max_tokens: 1200
|
|
||||||
top_p: 1.0
|
|
||||||
timeout_seconds: 180
|
|
||||||
reasoning_effort: medium
|
|
||||||
api_key_env: SCRIPTORIUM_API_KEY
|
|
||||||
extra_params:
|
|
||||||
provider: openrouter
|
|
||||||
route: fallback
|
|
||||||
```
|
|
||||||
|
|
||||||
## Available Options
|
|
||||||
|
|
||||||
- `id` (required): Unique profile identifier used by `--profile` or prompt `default_profile`.
|
|
||||||
- `endpoint` (required): OpenAI-compatible base URL, usually ending in `/v1`.
|
|
||||||
- `model` (required): Model name to request at that endpoint.
|
|
||||||
- `temperature` (optional): Sampling temperature. Valid range is `0` to `2`.
|
|
||||||
- `max_tokens` (optional): Max completion tokens. Must be `>= 0`.
|
|
||||||
- `top_p` (optional): Nucleus sampling parameter. Valid range is `0` to `1`.
|
|
||||||
- `timeout_seconds` (optional): Request timeout in seconds. Must be `>= 0`.
|
|
||||||
- `reasoning_effort` (optional): Provider/model-specific reasoning level string.
|
|
||||||
- `api_key_env` (optional): Environment variable name that holds the API key.
|
|
||||||
- `extra_params` (optional): String key/value map for provider-specific parameters.
|
|
||||||
|
|
||||||
## Notes and Rules
|
|
||||||
|
|
||||||
- Raw API keys are not supported. Do **not** add `api_key` fields.
|
|
||||||
- Unknown YAML fields fail to load (strict decoding).
|
|
||||||
- If `api_key_env` is set, the environment variable must be present when the run executes.
|
|
||||||
@@ -1,73 +0,0 @@
|
|||||||
# Prompt Definition Files
|
|
||||||
|
|
||||||
Prompt Definitions define **what** Scriptorium should do.
|
|
||||||
|
|
||||||
A prompt file is YAML, typically stored under `prompts/`, for example `prompts/generic.structured_events.yaml`.
|
|
||||||
|
|
||||||
## Complete Example
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
id: generic.structured_events
|
|
||||||
version: "1.0.0"
|
|
||||||
default_profile: local-quality
|
|
||||||
description: Extract events from a transcript into structured JSON.
|
|
||||||
|
|
||||||
inputs:
|
|
||||||
- name: transcript
|
|
||||||
required: true
|
|
||||||
content_type: text/markdown
|
|
||||||
description: Source transcript
|
|
||||||
- name: glossary
|
|
||||||
required: false
|
|
||||||
content_type: text/yaml
|
|
||||||
description: Optional glossary context
|
|
||||||
|
|
||||||
messages:
|
|
||||||
- role: system
|
|
||||||
content: |
|
|
||||||
You are a structured extraction assistant.
|
|
||||||
Return only JSON.
|
|
||||||
- role: user
|
|
||||||
content_file: ./generic.structured_events.user.md
|
|
||||||
|
|
||||||
output:
|
|
||||||
format: json
|
|
||||||
validation_mode: json_schema
|
|
||||||
schema_path: structured_events.schema.json
|
|
||||||
repair_attempts: 1
|
|
||||||
```
|
|
||||||
|
|
||||||
## Available Options
|
|
||||||
|
|
||||||
- `id` (required): Prompt identifier used by `--prompt` / `prompt_id`.
|
|
||||||
- `version` (required): Prompt version string.
|
|
||||||
- `default_profile` (optional): Execution profile ID used when no explicit profile is provided.
|
|
||||||
- `description` (optional): Human-readable description.
|
|
||||||
|
|
||||||
### `inputs[]`
|
|
||||||
|
|
||||||
- `name` (required): Logical input name referenced in templates via `{{input "name"}}`.
|
|
||||||
- `required` (optional): If `true`, run fails when input is missing.
|
|
||||||
- `content_type` (optional): Metadata only (not enforced yet).
|
|
||||||
- `description` (optional): Human-readable input description.
|
|
||||||
|
|
||||||
### `messages[]`
|
|
||||||
|
|
||||||
- `role` (required): Message role such as `system` or `user`.
|
|
||||||
- `content` (optional): Inline Go-template message body.
|
|
||||||
- `content_file` (optional): Path to a template file.
|
|
||||||
|
|
||||||
Each message must set **exactly one** of `content` or `content_file`.
|
|
||||||
|
|
||||||
### `output`
|
|
||||||
|
|
||||||
- `format` (required): One of `text`, `markdown`, `json`.
|
|
||||||
- `validation_mode` (required): One of `none`, `basic`, `json`, `json_schema`.
|
|
||||||
- `schema_path` (required when `validation_mode: json_schema`): Path to JSON Schema file.
|
|
||||||
- `repair_attempts` (required): Number of bounded repair retries (`>= 0`).
|
|
||||||
|
|
||||||
## Notes and Rules
|
|
||||||
|
|
||||||
- Unknown YAML fields fail to load (strict decoding).
|
|
||||||
- `content_file` paths are resolved relative to the prompt YAML file.
|
|
||||||
- For `json_schema` validation mode, Scriptorium also sends provider-level structured output requests automatically.
|
|
||||||
@@ -1,82 +0,0 @@
|
|||||||
# JSON Schema Definition Files
|
|
||||||
|
|
||||||
Schema definition files describe the expected JSON output contract for prompts that use:
|
|
||||||
|
|
||||||
- `output.format: json`
|
|
||||||
- `output.validation_mode: json_schema`
|
|
||||||
|
|
||||||
Schema files are JSON, typically stored under `schemas/`, for example `schemas/structured_events.schema.json`.
|
|
||||||
|
|
||||||
## Complete Example
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
||||||
"$id": "https://example.com/schemas/structured-events.schema.json",
|
|
||||||
"title": "Structured Events",
|
|
||||||
"description": "Expected shape for extracted event output",
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"summary": {
|
|
||||||
"type": "string",
|
|
||||||
"minLength": 1,
|
|
||||||
"description": "High-level session summary"
|
|
||||||
},
|
|
||||||
"events": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"title": { "type": "string" },
|
|
||||||
"type": {
|
|
||||||
"type": "string",
|
|
||||||
"enum": ["discovery", "combat", "social", "travel", "downtime", "other"]
|
|
||||||
},
|
|
||||||
"notes": { "type": "string" }
|
|
||||||
},
|
|
||||||
"required": ["title", "type"],
|
|
||||||
"additionalProperties": false
|
|
||||||
},
|
|
||||||
"minItems": 0
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": ["summary", "events"],
|
|
||||||
"additionalProperties": false,
|
|
||||||
"$defs": {
|
|
||||||
"nonEmptyString": {
|
|
||||||
"type": "string",
|
|
||||||
"minLength": 1
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Available Options
|
|
||||||
|
|
||||||
Scriptorium does not define custom schema keywords. It expects a valid JSON Schema document and passes it to the validator/provider.
|
|
||||||
|
|
||||||
Commonly used JSON Schema options include:
|
|
||||||
|
|
||||||
- `$schema`: Draft identifier URI.
|
|
||||||
- `$id`: Schema identifier URI.
|
|
||||||
- `title`: Human-readable schema title.
|
|
||||||
- `description`: Human-readable schema description.
|
|
||||||
- `type`: Expected JSON type (`object`, `array`, `string`, etc.).
|
|
||||||
- `properties`: Object field definitions.
|
|
||||||
- `required`: Required object fields.
|
|
||||||
- `additionalProperties`: Whether undeclared fields are allowed.
|
|
||||||
- `items`: Array item schema.
|
|
||||||
- `enum`: Allowed literal values.
|
|
||||||
- `const`: Single allowed literal value.
|
|
||||||
- `oneOf`, `anyOf`, `allOf`: Composition rules.
|
|
||||||
- `minimum`, `maximum`: Numeric bounds.
|
|
||||||
- `minLength`, `maxLength`, `pattern`: String constraints.
|
|
||||||
- `minItems`, `maxItems`: Array constraints.
|
|
||||||
- `$defs`: Reusable local definitions.
|
|
||||||
- `$ref`: Reference to another schema/definition.
|
|
||||||
|
|
||||||
## Notes and Rules
|
|
||||||
|
|
||||||
- Schema path comes from prompt `output.schema_path` and is resolved relative to `schema_dir`.
|
|
||||||
- If schema loading fails for `json_schema` mode, the run fails before the LLM request.
|
|
||||||
- Keep schemas strict (`additionalProperties: false`) when you want predictable output shape.
|
|
||||||
192
docs/integrations/http-api.md
Normal file
192
docs/integrations/http-api.md
Normal file
@@ -0,0 +1,192 @@
|
|||||||
|
# HTTP API Integration
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document defines the implemented inbound HTTP contract for Scriptorium.
|
||||||
|
|
||||||
|
Current scope is only:
|
||||||
|
|
||||||
|
- `POST /v1/runs`
|
||||||
|
|
||||||
|
For CLI behavior, see the [CLI reference](../cli.md).
|
||||||
|
|
||||||
|
## Endpoint
|
||||||
|
|
||||||
|
- Method: `POST`
|
||||||
|
- Path: `/v1/runs`
|
||||||
|
- Content type: JSON request/response
|
||||||
|
|
||||||
|
Route behavior:
|
||||||
|
|
||||||
|
- unknown path: `404 not_found`
|
||||||
|
- unsupported method on `/v1/runs`: `405 method_not_allowed`
|
||||||
|
|
||||||
|
Copyable request example file:
|
||||||
|
|
||||||
|
- `examples/http-run.json`
|
||||||
|
|
||||||
|
## Request Body
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"prompt_id": "generic.structured_events",
|
||||||
|
"profile_id": "local-quality",
|
||||||
|
"prompt_version": "1.0.0",
|
||||||
|
"inputs": {
|
||||||
|
"transcript": {"type": "file", "uri": "./examples/fixtures/transcript.md"},
|
||||||
|
"glossary": {"type": "inline", "body": "party:\n - Rin"}
|
||||||
|
},
|
||||||
|
"vars": {
|
||||||
|
"session_date": "2026-05-04"
|
||||||
|
},
|
||||||
|
"model": {
|
||||||
|
"endpoint": "http://localhost:8000/v1",
|
||||||
|
"model": "gpt-4o-mini",
|
||||||
|
"temperature": 0.0,
|
||||||
|
"max_tokens": 800,
|
||||||
|
"top_p": 1.0,
|
||||||
|
"timeout_seconds": 120,
|
||||||
|
"service_tier": "priority",
|
||||||
|
"reasoning_effort": "medium",
|
||||||
|
"api_key_env": "SCRIPTORIUM_API_KEY",
|
||||||
|
"extra_params": {
|
||||||
|
"route": "primary"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"include_raw_output": false
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Required fields:
|
||||||
|
|
||||||
|
- `prompt_id`
|
||||||
|
- `inputs` (must contain at least one named input)
|
||||||
|
|
||||||
|
Input reference types currently supported by runtime artifact loading:
|
||||||
|
|
||||||
|
- `file`
|
||||||
|
- `inline`
|
||||||
|
|
||||||
|
## Strict JSON Rules
|
||||||
|
|
||||||
|
Request decoding uses strict JSON field checks:
|
||||||
|
|
||||||
|
- unknown request fields are rejected with `400 invalid_json`
|
||||||
|
- unknown `model` fields are rejected with `400 invalid_json`
|
||||||
|
- raw API-key payload fields such as `api_key` are rejected as unknown fields
|
||||||
|
|
||||||
|
## Success Response
|
||||||
|
|
||||||
|
Status: `200 OK`
|
||||||
|
|
||||||
|
Response shape:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"artifact": {
|
||||||
|
"name": "output",
|
||||||
|
"content_type": "application/json",
|
||||||
|
"body": "{\"summary\":\"...\"}",
|
||||||
|
"uri": "",
|
||||||
|
"size": 123,
|
||||||
|
"hash": "..."
|
||||||
|
},
|
||||||
|
"validation": {
|
||||||
|
"status": "passed",
|
||||||
|
"mode": "json_schema",
|
||||||
|
"errors": [],
|
||||||
|
"schema_path": "structured_events.schema.json",
|
||||||
|
"repair_attempts": 0,
|
||||||
|
"is_valid": true
|
||||||
|
},
|
||||||
|
"metadata": {
|
||||||
|
"run_id": "...",
|
||||||
|
"prompt_id": "generic.structured_events",
|
||||||
|
"prompt_version": "1.0.0",
|
||||||
|
"prompt_hash": "...",
|
||||||
|
"rendered_prompt_hash": "...",
|
||||||
|
"selected_profile_id": "local-quality",
|
||||||
|
"model_name": "gpt-4o-mini",
|
||||||
|
"endpoint": "http://localhost:8000/v1",
|
||||||
|
"model_params": {
|
||||||
|
"endpoint": "http://localhost:8000/v1",
|
||||||
|
"model": "gpt-4o-mini",
|
||||||
|
"temperature": 0,
|
||||||
|
"max_tokens": 800,
|
||||||
|
"top_p": 1,
|
||||||
|
"timeout_seconds": 120,
|
||||||
|
"service_tier": "priority",
|
||||||
|
"reasoning_effort": "medium",
|
||||||
|
"api_key_env": "SCRIPTORIUM_API_KEY",
|
||||||
|
"extra_params": {
|
||||||
|
"route": "primary"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"input_hashes": {
|
||||||
|
"transcript": "..."
|
||||||
|
},
|
||||||
|
"usage": {
|
||||||
|
"prompt_tokens": 11,
|
||||||
|
"completion_tokens": 22,
|
||||||
|
"total_tokens": 33
|
||||||
|
},
|
||||||
|
"start_time": "2026-05-04T12:00:00Z",
|
||||||
|
"end_time": "2026-05-04T12:00:01Z",
|
||||||
|
"duration_ms": 1000,
|
||||||
|
"validation_mode": "json_schema",
|
||||||
|
"validation_status": "passed",
|
||||||
|
"repair_attempts_used": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`raw_model_output` is omitted by default.
|
||||||
|
|
||||||
|
To include it, send:
|
||||||
|
|
||||||
|
- `"include_raw_output": true`
|
||||||
|
|
||||||
|
## Validation Failure Behavior
|
||||||
|
|
||||||
|
Validation content failures do not map to HTTP error status.
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
|
||||||
|
- status remains `200 OK`
|
||||||
|
- `validation.status` is `failed`
|
||||||
|
- validation errors are returned in `validation.errors`
|
||||||
|
|
||||||
|
## Error Responses
|
||||||
|
|
||||||
|
Error body shape:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"error": {
|
||||||
|
"code": "invalid_request",
|
||||||
|
"message": "prompt_id is required"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Current error mapping (non-exhaustive):
|
||||||
|
|
||||||
|
- `400 invalid_json`: malformed JSON or unknown JSON fields
|
||||||
|
- `400 invalid_request`: missing/invalid request fields
|
||||||
|
- `400 profile_required`: no explicit `profile_id` and prompt has no `default_profile`
|
||||||
|
- `400 prompt_load_failed`: prompt definition invalid/unloadable
|
||||||
|
- `400 profile_load_failed`: profile invalid/unloadable
|
||||||
|
- `400 artifact_read_failed`: input artifact loading failed
|
||||||
|
- `400 prompt_render_failed`: template render failed
|
||||||
|
- `400 api_key_env_missing`: named API-key environment variable is missing
|
||||||
|
- `404 prompt_not_found`
|
||||||
|
- `404 profile_not_found`
|
||||||
|
- `502 llm_failed`: outbound model request failed
|
||||||
|
- `500 validation_runtime_failed`: validator runtime/schema-load failure
|
||||||
|
- `500 internal_error`
|
||||||
|
|
||||||
|
## Security And Deployment Note
|
||||||
|
|
||||||
|
The HTTP adapter has no built-in authentication or authorization.
|
||||||
|
|
||||||
|
Deploy behind trusted controls (for example authenticated gateway/reverse proxy and network boundaries).
|
||||||
@@ -1,322 +1,114 @@
|
|||||||
# Narratio -> Scriptorium CLI Integration
|
# Narratio Subprocess Integration
|
||||||
|
|
||||||
## 1. Purpose
|
## Purpose
|
||||||
|
|
||||||
This document defines how Narratio should invoke Scriptorium through the **public CLI**.
|
This document defines the supported subprocess contract for Narratio invoking Scriptorium through the public CLI.
|
||||||
|
|
||||||
This is a **subprocess integration contract**, not an internal Go API contract.
|
This is a CLI contract, not an internal Go package integration.
|
||||||
|
|
||||||
## 2. Assumptions
|
## Supported Commands
|
||||||
|
|
||||||
- `scriptorium` is installed and available on `PATH`.
|
Narratio should invoke:
|
||||||
- Scriptorium is configured with `config.yml`.
|
|
||||||
- `config.yml` provides `prompt_dir`, `profile_dir`, and `schema_dir` as needed.
|
|
||||||
- Prompt and profile libraries are already deployed for the environment.
|
|
||||||
- Narratio provides prepared artifact files (for example polished transcript, glossary, previous recap, campaign notes).
|
|
||||||
- Initial integration is synchronous subprocess execution.
|
|
||||||
- Narratio remains the orchestrator.
|
|
||||||
|
|
||||||
In normal operation, Narratio does not need to pass `--prompt-dir` and `--profile-dir` if they are supplied by Scriptorium config.
|
|
||||||
|
|
||||||
Narratio may pass `--config <PATH>` when it must use a non-default Scriptorium config file.
|
|
||||||
|
|
||||||
## 3. Core Commands Narratio May Call
|
|
||||||
|
|
||||||
Primary commands for subprocess integration:
|
|
||||||
|
|
||||||
- `scriptorium run`
|
- `scriptorium run`
|
||||||
- `scriptorium render`
|
- `scriptorium render`
|
||||||
|
|
||||||
For production generation, use `scriptorium run`.
|
Use `run` for generation.
|
||||||
|
|
||||||
`scriptorium render` is for debugging, dry-runs, test assertions, and validating command construction without LLM execution.
|
Use `render` for preflight/debug output without LLM execution.
|
||||||
|
|
||||||
Note: `scriptorium serve` and HTTP API exist, but they are not the initial integration path.
|
## Recommended Invocation Shapes
|
||||||
|
|
||||||
## 4. Command Selection Guidance
|
Run:
|
||||||
|
|
||||||
- Use `run` to generate an output artifact.
|
|
||||||
- Use `render` to inspect the prepared prompt and effective settings without calling the LLM.
|
|
||||||
- Use `render --format json` when Narratio/tests need structured prepare output.
|
|
||||||
|
|
||||||
## 5. Recommended `run` Invocation Shape
|
|
||||||
|
|
||||||
Production shape:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
scriptorium run \
|
scriptorium run \
|
||||||
--prompt <prompt_id> \
|
--prompt <prompt_id> \
|
||||||
--input transcript=<processed-transcript-path> \
|
--input transcript=<path> \
|
||||||
--out <output-artifact-path>
|
--out <artifact_path>
|
||||||
```
|
```
|
||||||
|
|
||||||
Common optional additions:
|
Render:
|
||||||
|
|
||||||
- `--config <path>`: use a specific Scriptorium config file.
|
|
||||||
- `--profile <profile_id>`: override prompt default profile.
|
|
||||||
- `--var name=value` (repeatable): small metadata values.
|
|
||||||
- `--input name=path` (repeatable): additional named artifacts.
|
|
||||||
- `--timeout <duration>`: per-run timeout override.
|
|
||||||
- Runtime model override flags (`--llm-base-url`, `--model`, etc.) only for exceptional/operator-directed cases.
|
|
||||||
|
|
||||||
## 6. Recommended `render` Invocation Shape
|
|
||||||
|
|
||||||
Human-readable debug shape:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
scriptorium render \
|
scriptorium render \
|
||||||
--prompt <prompt_id> \
|
--prompt <prompt_id> \
|
||||||
--input transcript=<processed-transcript-path> \
|
--input transcript=<path> \
|
||||||
--format text
|
--format json
|
||||||
```
|
```
|
||||||
|
|
||||||
Structured debug/test shape:
|
Narratio may add:
|
||||||
|
|
||||||
```bash
|
- `--config <path>`
|
||||||
scriptorium render \
|
- `--profile <profile_id>`
|
||||||
--prompt <prompt_id> \
|
- repeatable `--input name=path`
|
||||||
--input transcript=<processed-transcript-path> \
|
- repeatable `--var name=value`
|
||||||
--format json \
|
- runtime overrides when explicitly needed (`--model`, `--llm-base-url`, `--timeout`, etc.)
|
||||||
--out <render-debug-path>
|
|
||||||
```
|
|
||||||
|
|
||||||
`render` does **not** call the LLM, does **not** validate model output, and does **not** perform repair.
|
## Config And Directory Behavior
|
||||||
|
|
||||||
## 7. Inputs
|
Narratio can rely on resolved app config or pass explicit paths.
|
||||||
|
|
||||||
- Pass inputs as repeated `--input name=path` flags.
|
- default config search order:
|
||||||
- `name` must match the Prompt Definition input name.
|
1. `/usr/local/etc/scriptorium/config.yml`
|
||||||
- Prefer absolute paths, or paths relative to a working directory controlled by Narratio.
|
2. `/etc/scriptorium/config.yml`
|
||||||
- Pass Audita output as the primary transcript input.
|
- explicit `--config` requires file existence and valid syntax
|
||||||
- Additional inputs may include glossary, previous recap, campaign notes, event logs, final state maps, or other prompt-specific artifacts.
|
- CLI flags override config values
|
||||||
- Scriptorium reads input files directly; Narratio does not need to inline file content for CLI use.
|
|
||||||
|
|
||||||
## 8. Variables
|
## Profile Selection
|
||||||
|
|
||||||
Use repeated `--var name=value` for small metadata values.
|
Profile selection follows runner behavior:
|
||||||
|
|
||||||
Typical examples:
|
1. explicit `--profile`
|
||||||
|
2. prompt `default_profile`
|
||||||
|
3. error if neither is available
|
||||||
|
|
||||||
- `session_date`
|
Narratio should treat prompt/profile IDs as deployment configuration, not hardcoded logic.
|
||||||
- `session_id`
|
|
||||||
- `campaign_name`
|
|
||||||
- `previous_session_id`
|
|
||||||
- `output_kind`
|
|
||||||
|
|
||||||
Large content belongs in input files, not `--var` values.
|
## Input And Variable Contract
|
||||||
|
|
||||||
## 9. Prompt IDs and Output Artifact Types
|
- Inputs use repeated `--input name=path`.
|
||||||
|
- Input names must match prompt definition input names.
|
||||||
|
- Variables use repeated `--var name=value` for small metadata values.
|
||||||
|
- Prefer file inputs for large content.
|
||||||
|
|
||||||
Narratio should treat prompt IDs as configuration, not hardcoded business logic.
|
## Environment Contract
|
||||||
|
|
||||||
Narratio config may map stage/output names to prompt IDs, for example:
|
|
||||||
|
|
||||||
- session recap prompt
|
|
||||||
- structured event extraction prompt
|
|
||||||
- glossary suggestion prompt
|
|
||||||
- player-facing summary prompt
|
|
||||||
|
|
||||||
Prompt IDs used by Narratio should come from the deployed Scriptorium prompt library.
|
|
||||||
|
|
||||||
## 10. Profiles
|
|
||||||
|
|
||||||
- Prompts may declare `default_profile`.
|
|
||||||
- Narratio may omit `--profile` to use prompt default profile.
|
|
||||||
- Narratio may pass `--profile` to force profile selection.
|
|
||||||
- This enables environment/profile selection like `local-fast`, `local-quality`, `frontier`, `batch`, or test profiles.
|
|
||||||
- Profile names should generally be Narratio configuration values.
|
|
||||||
|
|
||||||
## 11. Runtime Overrides
|
|
||||||
|
|
||||||
Supported runtime override flags:
|
|
||||||
|
|
||||||
- `--llm-base-url`
|
|
||||||
- `--model`
|
|
||||||
- `--api-key-env`
|
|
||||||
- `--temperature`
|
|
||||||
- `--max-tokens`
|
|
||||||
- `--top-p`
|
|
||||||
- `--timeout`
|
|
||||||
|
|
||||||
Guidance:
|
|
||||||
|
|
||||||
- Keep normal model/runtime settings in Execution Profiles.
|
|
||||||
- Use runtime overrides only for explicit per-run exceptions, tests, or operator overrides.
|
|
||||||
- Never pass raw API keys on the command line.
|
|
||||||
- `--api-key-env` names an environment variable; Narratio must ensure that variable is set in subprocess environment.
|
|
||||||
|
|
||||||
## 12. Config Behavior
|
|
||||||
|
|
||||||
- Default config path: `/etc/scriptorium/config.yml`.
|
|
||||||
- `--config <PATH>` overrides default path.
|
|
||||||
- Missing default config is allowed by Scriptorium.
|
|
||||||
- If `--config` is provided explicitly, the file must exist and be valid.
|
|
||||||
- CLI flags override `config.yml`.
|
|
||||||
- `config.yml` overrides built-in application defaults.
|
|
||||||
|
|
||||||
Narratio can either:
|
|
||||||
|
|
||||||
- rely on system default config path, or
|
|
||||||
- carry an explicit config path and pass `--config`.
|
|
||||||
|
|
||||||
## 13. Environment Handling
|
|
||||||
|
|
||||||
Subprocess environment recommendations:
|
|
||||||
|
|
||||||
- Pass through required API-key environment variables referenced by `api_key_env`.
|
- Pass through required API-key environment variables referenced by `api_key_env`.
|
||||||
- Do not pass raw API keys as CLI arguments.
|
- Never pass raw API keys via CLI arguments.
|
||||||
- Avoid logging full environment dumps.
|
- Keep subprocess environment scoped to required variables.
|
||||||
- Capture stdout and stderr separately.
|
|
||||||
- Use a controlled working directory.
|
|
||||||
- Prefer absolute artifact paths.
|
|
||||||
|
|
||||||
## 14. Output Handling
|
## Output And Error Handling
|
||||||
|
|
||||||
For `scriptorium run`:
|
`run`:
|
||||||
|
|
||||||
- Use `--out` when Narratio needs durable artifact files.
|
- stdout: artifact body unless `--out` is used
|
||||||
- Without `--out`, artifact content is written to stdout.
|
- `--out`: writes artifact to file
|
||||||
- Preferred orchestration pattern: always use `--out`, then treat the file as stage output artifact.
|
- stderr: success summary and errors
|
||||||
- Capture stderr for diagnostics.
|
|
||||||
|
|
||||||
For `scriptorium render`:
|
`render`:
|
||||||
|
|
||||||
- Use `--out` to store render diagnostics.
|
- stdout: prepared-run output unless `--out` is used
|
||||||
- Use `--format json` when tests need to inspect selected profile, effective runtime settings, input hashes, prompt hash, and rendered messages.
|
- stderr: errors
|
||||||
|
|
||||||
## 15. Exit Status and Errors
|
Narratio should capture stdout and stderr separately.
|
||||||
|
|
||||||
Current CLI behavior (verified from implementation/tests):
|
## Exit Status Contract
|
||||||
|
|
||||||
- `0`: success.
|
- `0`: success
|
||||||
- `1`: runtime/parse/config/load/render/generation/IO error.
|
- `1`: parse/config/load/render/generation/IO/runtime error
|
||||||
- `2`: run completed but output validation failed (`ValidationFailed`).
|
- `2`: run completed but validation failed
|
||||||
|
|
||||||
Additional details:
|
A `run` exit code `2` can still produce output (stdout or `--out`).
|
||||||
|
|
||||||
- On `run`, output artifact write happens before exit code selection. If validation fails, artifact may still be written and exit code is `2`.
|
## Security Notes
|
||||||
- `stderr` carries both errors and normal run summary output; non-empty stderr alone does not imply failure.
|
|
||||||
- `render` returns `0` on success and `1` on failures.
|
|
||||||
|
|
||||||
Narratio should treat non-zero exit codes as failed stage execution, but may record generated artifact paths if a run exited `2` and output file exists.
|
- Treat generated artifacts and stderr logs as potentially sensitive.
|
||||||
|
- Avoid logging full rendered prompts by default in production contexts.
|
||||||
|
- Use controlled output paths and access controls for persisted artifacts.
|
||||||
|
|
||||||
## 16. Recommended Narratio Integration Pattern
|
## Canonical References
|
||||||
|
|
||||||
1. Build CLI args from Narratio stage configuration.
|
- CLI behavior: [CLI reference](../cli.md)
|
||||||
2. Use subprocess context cancellation/timeout.
|
- Config behavior: [Configuration reference](../config.md)
|
||||||
3. Pass absolute input paths.
|
- Operations and failure handling: [Operations guide](../operations.md), [Troubleshooting](../troubleshooting.md)
|
||||||
4. Pass `--out` to a session-scoped artifact path.
|
|
||||||
5. Add `--var` metadata values.
|
|
||||||
6. Optionally add `--config`.
|
|
||||||
7. Optionally add `--profile`.
|
|
||||||
8. Ensure required API-key env vars are present.
|
|
||||||
9. Run subprocess synchronously.
|
|
||||||
10. Capture stdout/stderr separately.
|
|
||||||
11. On success, store output artifact path and invocation metadata in stage artifacts.
|
|
||||||
12. On failure, store exit code and stderr diagnostics in stage status.
|
|
||||||
|
|
||||||
## 17. Suggested Narratio Configuration Shape
|
|
||||||
|
|
||||||
Illustrative (not required schema):
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
scriptorium:
|
|
||||||
config_path: /etc/scriptorium/config.yml
|
|
||||||
stages:
|
|
||||||
session_recap:
|
|
||||||
prompt_id: dnd.session_recap
|
|
||||||
profile_id: local-quality # optional
|
|
||||||
inputs: [transcript, glossary, previous_recap]
|
|
||||||
vars: [session_id, session_date, campaign_name]
|
|
||||||
output_path_template: artifacts/{session_id}/session_recap.md
|
|
||||||
timeout: 2m
|
|
||||||
render_debug: false
|
|
||||||
```
|
|
||||||
|
|
||||||
The key idea: map Narratio stage/artifact names to prompt ID, optional profile, expected inputs, and output destination.
|
|
||||||
|
|
||||||
## 18. Testing Strategy for Narratio Integration
|
|
||||||
|
|
||||||
- Use `scriptorium render --format json` to verify command construction without LLM calls.
|
|
||||||
- Use dedicated test prompt/profile libraries for integration tests.
|
|
||||||
- Use small fixture transcripts.
|
|
||||||
- Verify missing-input failure behavior.
|
|
||||||
- Verify prompt `default_profile` behavior.
|
|
||||||
- Verify explicit `--profile` override behavior.
|
|
||||||
- Verify `--config` behavior (default and explicit).
|
|
||||||
- Verify output file creation when `--out` is used.
|
|
||||||
- Verify stderr capture on failures.
|
|
||||||
- Avoid real API keys in tests.
|
|
||||||
|
|
||||||
## 19. Security and Privacy Notes
|
|
||||||
|
|
||||||
- Never pass raw API keys on command line.
|
|
||||||
- Do not log full rendered prompts by default; transcripts may contain sensitive content.
|
|
||||||
- Avoid logging prompt content unless explicit debug mode is enabled.
|
|
||||||
- Treat generated artifacts as potentially sensitive.
|
|
||||||
- Use session-scoped, access-controlled output paths.
|
|
||||||
- `api_key_env` names should come from environment management, not embedded secrets.
|
|
||||||
|
|
||||||
## 20. Initial D&D Artifact Generation Examples
|
|
||||||
|
|
||||||
These are examples only. Use prompt IDs from the deployed prompt library.
|
|
||||||
|
|
||||||
Session recap:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt dnd.session_recap \
|
|
||||||
--input transcript=/work/session-42/transcript.polished.md \
|
|
||||||
--input glossary=/work/session-42/glossary.yml \
|
|
||||||
--out /work/session-42/artifacts/session_recap.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Structured events:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt dnd.structured_events \
|
|
||||||
--input transcript=/work/session-42/transcript.polished.md \
|
|
||||||
--out /work/session-42/artifacts/structured_events.json
|
|
||||||
```
|
|
||||||
|
|
||||||
Glossary suggestions:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt dnd.glossary_suggestions \
|
|
||||||
--input transcript=/work/session-42/transcript.polished.md \
|
|
||||||
--input previous_recap=/work/session-41/artifacts/session_recap.md \
|
|
||||||
--out /work/session-42/artifacts/glossary_suggestions.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Player-facing summary:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scriptorium run \
|
|
||||||
--prompt dnd.player_summary \
|
|
||||||
--input transcript=/work/session-42/transcript.polished.md \
|
|
||||||
--input structured_events=/work/session-42/artifacts/structured_events.json \
|
|
||||||
--out /work/session-42/artifacts/player_summary.md
|
|
||||||
```
|
|
||||||
|
|
||||||
## 21. Non-Goals
|
|
||||||
|
|
||||||
Initial Narratio integration should not:
|
|
||||||
|
|
||||||
- call Scriptorium internal Go packages
|
|
||||||
- use HTTP API as the primary path
|
|
||||||
- expect Scriptorium to read S3 refs directly
|
|
||||||
- make Scriptorium responsible for Narratio stage state
|
|
||||||
- make Scriptorium responsible for notification
|
|
||||||
- require Scriptorium to understand D&D workflow semantics beyond prompt definitions
|
|
||||||
|
|
||||||
## 22. Future Extension Notes
|
|
||||||
|
|
||||||
Possible later extensions:
|
|
||||||
|
|
||||||
- HTTP API integration
|
|
||||||
- S3 artifact references if Scriptorium adds S3 reader support
|
|
||||||
- storing render diagnostics alongside generated artifacts
|
|
||||||
- token budgeting/prompt-size checks
|
|
||||||
- batch execution if Scriptorium later adds batch support
|
|
||||||
|
|||||||
113
docs/integrations/openai-compatible-chat.md
Normal file
113
docs/integrations/openai-compatible-chat.md
Normal file
@@ -0,0 +1,113 @@
|
|||||||
|
# OpenAI-Compatible Chat Integration
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document defines the outbound LLM contract implemented by `internal/llm/openai_compatible_client.go`.
|
||||||
|
|
||||||
|
It documents only fields and behaviors currently serialized by code.
|
||||||
|
|
||||||
|
## Endpoint Construction
|
||||||
|
|
||||||
|
Request endpoint is built as:
|
||||||
|
|
||||||
|
1. choose base URL:
|
||||||
|
- `GenerateRequest.Target.Endpoint` if set
|
||||||
|
- otherwise client config `BaseURL`
|
||||||
|
2. trim trailing slash
|
||||||
|
3. append `/chat/completions`
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
- base URL: `http://localhost:8000/v1`
|
||||||
|
- final URL: `http://localhost:8000/v1/chat/completions`
|
||||||
|
|
||||||
|
## Request Fields Sent
|
||||||
|
|
||||||
|
Serialized JSON fields:
|
||||||
|
|
||||||
|
- `model` (required after fallback resolution)
|
||||||
|
- `messages` (role/content pairs from rendered prompt)
|
||||||
|
- `temperature` (only when non-zero)
|
||||||
|
- `max_tokens` (only when non-zero)
|
||||||
|
- `top_p` (only when non-zero)
|
||||||
|
- `service_tier` (only when non-empty)
|
||||||
|
- `response_format` (only when structured output is provided)
|
||||||
|
|
||||||
|
`service_tier` is provider-specific. OpenRouter currently documents request values such as `flex` and `priority`; Scriptorium forwards any non-empty configured value and lets the backend validate support.
|
||||||
|
|
||||||
|
Structured output is currently `json_schema` only, serialized as:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"response_format": {
|
||||||
|
"type": "json_schema",
|
||||||
|
"json_schema": {
|
||||||
|
"name": "...",
|
||||||
|
"strict": true,
|
||||||
|
"schema": {"type": "object"}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Authentication Header
|
||||||
|
|
||||||
|
If `Target.APIKeyEnv` is set:
|
||||||
|
|
||||||
|
- resolve environment variable value at request time
|
||||||
|
- set `Authorization: Bearer <value>`
|
||||||
|
|
||||||
|
If the environment variable is unset/empty:
|
||||||
|
|
||||||
|
- request fails before HTTP call (`ErrInvalidRequest`)
|
||||||
|
|
||||||
|
If `Target.APIKeyEnv` is empty:
|
||||||
|
|
||||||
|
- no `Authorization` header is sent
|
||||||
|
|
||||||
|
## Timeout Behavior
|
||||||
|
|
||||||
|
Base timeout comes from client configuration.
|
||||||
|
|
||||||
|
Per-request override:
|
||||||
|
|
||||||
|
- if `Target.TimeoutSeconds > 0`, use that value for request timeout
|
||||||
|
- if `Target.TimeoutSeconds < 0`, request is rejected (`ErrInvalidRequest`)
|
||||||
|
|
||||||
|
## Response Expectations
|
||||||
|
|
||||||
|
Expected successful response shape (subset used):
|
||||||
|
|
||||||
|
- `choices[0].message.content`
|
||||||
|
- `usage.prompt_tokens`
|
||||||
|
- `usage.completion_tokens`
|
||||||
|
- `usage.total_tokens`
|
||||||
|
|
||||||
|
Malformed response conditions include:
|
||||||
|
|
||||||
|
- invalid JSON
|
||||||
|
- empty `choices`
|
||||||
|
- empty `choices[0].message.content`
|
||||||
|
|
||||||
|
Malformed responses return `ErrMalformedResponse`.
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
- network/request-construction failures: `ErrRequestFailed`
|
||||||
|
- non-2xx HTTP status: `ErrUnexpectedStatus` (includes status code and trimmed response body snippet)
|
||||||
|
- malformed response shape/content: `ErrMalformedResponse`
|
||||||
|
|
||||||
|
## Unsupported Or Non-Serialized Fields
|
||||||
|
|
||||||
|
The following fields may exist in profile/effective settings but are not currently serialized into outbound chat-completions payloads:
|
||||||
|
|
||||||
|
- `reasoning_effort`
|
||||||
|
- `extra_params`
|
||||||
|
|
||||||
|
No built-in retries, tool-calls, or multi-request payload modes are implemented in this client.
|
||||||
|
|
||||||
|
## Relationship To Runner
|
||||||
|
|
||||||
|
When prompt validation mode is `json_schema`, runner prepares a structured-output schema spec and passes it to the client as `StructuredOutput`.
|
||||||
|
|
||||||
|
The client only serializes the provider request payload; it does not load schema files itself.
|
||||||
143
docs/internal/adapters.md
Normal file
143
docs/internal/adapters.md
Normal file
@@ -0,0 +1,143 @@
|
|||||||
|
# Adapter And Repository Internals
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document describes implemented adapter/repository boundaries and their current behavior.
|
||||||
|
|
||||||
|
## Adapter Map
|
||||||
|
|
||||||
|
- `internal/adapter/cli`: CLI command parsing, app wiring, stdout/stderr handling, exit codes.
|
||||||
|
- `internal/adapter/http`: HTTP request/response mapping for `POST /v1/runs`.
|
||||||
|
- `internal/promptdef`: filesystem prompt-definition repository.
|
||||||
|
- `internal/profile`: filesystem execution-profile repository.
|
||||||
|
- `internal/artifact`: input artifact reader.
|
||||||
|
- `internal/prompt`: Go-template renderer.
|
||||||
|
- `internal/llm`: OpenAI-compatible LLM client implementation.
|
||||||
|
- `internal/validate`: output validator.
|
||||||
|
- `internal/format`: prepared-run formatters for `render` output.
|
||||||
|
|
||||||
|
## Inputs And Outputs
|
||||||
|
|
||||||
|
CLI adapter:
|
||||||
|
|
||||||
|
- Input: process args, filesystem config/assets, environment.
|
||||||
|
- Output: exit code, stdout artifact/prepared output, stderr summaries/errors.
|
||||||
|
|
||||||
|
HTTP adapter:
|
||||||
|
|
||||||
|
- Input: JSON request body (`runRequestDTO`).
|
||||||
|
- Output: JSON success/error body with mapped status codes.
|
||||||
|
|
||||||
|
Filesystem repositories:
|
||||||
|
|
||||||
|
- Input: prompt/profile YAML files under configured directories.
|
||||||
|
- Output: normalized domain definitions/profiles or typed errors.
|
||||||
|
|
||||||
|
Artifact reader:
|
||||||
|
|
||||||
|
- Input: `domain.ArtifactRef`.
|
||||||
|
- Output: loaded `domain.Artifact`.
|
||||||
|
|
||||||
|
LLM adapter:
|
||||||
|
|
||||||
|
- Input: `domain.GenerateRequest`.
|
||||||
|
- Output: `domain.GenerateResponse`.
|
||||||
|
|
||||||
|
Validator:
|
||||||
|
|
||||||
|
- Input: artifact body + output contract.
|
||||||
|
- Output: validation result or runtime validation error.
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
- Adapters convert external representations to domain requests and back.
|
||||||
|
- Use-case decisions remain in `internal/usecase`.
|
||||||
|
- External dependency details stay scoped to adapter packages.
|
||||||
|
|
||||||
|
## Config Fields Used
|
||||||
|
|
||||||
|
Primary app settings consumed by adapters:
|
||||||
|
|
||||||
|
- `prompt_dir`
|
||||||
|
- `profile_dir`
|
||||||
|
- `schema_dir`
|
||||||
|
- `server.addr`
|
||||||
|
- `defaults.render_format`
|
||||||
|
|
||||||
|
Execution profile/request settings used through runner:
|
||||||
|
|
||||||
|
- `endpoint`, `model`, `temperature`, `max_tokens`, `top_p`, `timeout_seconds`, `service_tier`, `api_key_env`, `reasoning_effort`, `extra_params`
|
||||||
|
|
||||||
|
## External Dependencies
|
||||||
|
|
||||||
|
- YAML decoding: `gopkg.in/yaml.v3` (strict known-fields mode in config/prompt/profile loaders).
|
||||||
|
- JSON Schema validation: `github.com/santhosh-tekuri/jsonschema/v6`.
|
||||||
|
- HTTP client/server: Go standard library.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Strict decoding and input checks:
|
||||||
|
|
||||||
|
- config/prompt/profile loaders reject unknown YAML fields.
|
||||||
|
- prompt/profile repositories scan nested subdirectories recursively.
|
||||||
|
- prompt/profile lookup uses YAML `id` values; subdirectory paths are organizational only.
|
||||||
|
- duplicate prompt/profile IDs are invalid and fail instead of using first-match behavior.
|
||||||
|
- HTTP DTO decoder rejects unknown JSON fields.
|
||||||
|
- raw API key payload fields are rejected by strict decoding in profile/http paths.
|
||||||
|
|
||||||
|
Artifact refs:
|
||||||
|
|
||||||
|
- Supported reference types: `inline`, `file`.
|
||||||
|
- Unsupported types return `ErrUnsupportedRefType`.
|
||||||
|
|
||||||
|
LLM adapter:
|
||||||
|
|
||||||
|
- endpoint appends `/chat/completions`.
|
||||||
|
- non-2xx responses map to request failure errors.
|
||||||
|
- malformed responses (including missing/empty first choice content) are errors.
|
||||||
|
|
||||||
|
Validator:
|
||||||
|
|
||||||
|
- `basic`, `json`, `json_schema` content failures return `ValidationFailed` results.
|
||||||
|
- schema load/compile/path failures are runtime errors.
|
||||||
|
- schema lookup uses explicit `schema_path` values relative to `schema_dir`; it does not recursively search by basename.
|
||||||
|
|
||||||
|
HTTP error mapping:
|
||||||
|
|
||||||
|
- maps domain/use-case errors to stable HTTP code + error code/message.
|
||||||
|
- avoids returning internal wrapped-cause details in response payload.
|
||||||
|
|
||||||
|
## CLI Adapter Semantics
|
||||||
|
|
||||||
|
Implemented commands:
|
||||||
|
|
||||||
|
- `run`
|
||||||
|
- `render`
|
||||||
|
- `serve`
|
||||||
|
|
||||||
|
Behavior highlights:
|
||||||
|
|
||||||
|
- `run` exit `2` indicates validation failed after generation.
|
||||||
|
- `render` does not call the LLM.
|
||||||
|
- `serve` exposes HTTP handler only; no built-in auth.
|
||||||
|
- `render` supports `--format text|json`; `render` does not expose `--schema-dir`.
|
||||||
|
- deprecated aliases `--prompt-id` and `--profile-id` are still accepted.
|
||||||
|
|
||||||
|
## Tests To Inspect Before Changing
|
||||||
|
|
||||||
|
- `internal/adapter/cli/run_test.go`
|
||||||
|
- `internal/adapter/http/handler_test.go`
|
||||||
|
- `internal/promptdef/repository_test.go`
|
||||||
|
- `internal/profile/repository_test.go`
|
||||||
|
- `internal/artifact/reader_test.go`
|
||||||
|
- `internal/prompt/renderer_test.go`
|
||||||
|
- `internal/llm/openai_compatible_client_test.go`
|
||||||
|
- `internal/validate/standard_validator_test.go`
|
||||||
|
- `internal/format/prepared_run_test.go`
|
||||||
|
|
||||||
|
## Architectural Invariants
|
||||||
|
|
||||||
|
- Adapter packages do not own runner decision logic.
|
||||||
|
- External request/response strictness is part of contract stability.
|
||||||
|
- Prepared-render output never includes resolved API key values.
|
||||||
|
- Outbound OpenAI-compatible request includes only currently serialized fields (`model`, `messages`, optional `temperature`, `max_tokens`, `top_p`, optional `response_format`).
|
||||||
146
docs/internal/runner.md
Normal file
146
docs/internal/runner.md
Normal file
@@ -0,0 +1,146 @@
|
|||||||
|
# Runner Internals
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
`internal/usecase.Runner` is the core use case orchestrator for prompt preparation and execution.
|
||||||
|
|
||||||
|
It owns request validation, prompt/profile resolution, runtime-parameter merge, artifact loading, prompt rendering, structured-output setup, LLM invocation, output validation, and result metadata.
|
||||||
|
|
||||||
|
## Inputs And Outputs
|
||||||
|
|
||||||
|
Primary input type:
|
||||||
|
|
||||||
|
- `domain.RunRequest`
|
||||||
|
|
||||||
|
Primary output types:
|
||||||
|
|
||||||
|
- `domain.PreparedRun` from `Prepare`
|
||||||
|
- `domain.RunResult` from `Run`
|
||||||
|
|
||||||
|
LLM boundary types:
|
||||||
|
|
||||||
|
- `domain.GenerateRequest`
|
||||||
|
- `domain.GenerateResponse`
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
`Runner` coordinates the following interfaces:
|
||||||
|
|
||||||
|
- `promptdef.Repository`
|
||||||
|
- `profile.Repository`
|
||||||
|
- `artifact.Reader`
|
||||||
|
- `prompt.Renderer`
|
||||||
|
- `llm.Client`
|
||||||
|
- `validate.Validator`
|
||||||
|
- optional `usecase.OutputRepairer`
|
||||||
|
|
||||||
|
Transport concerns (CLI flags, HTTP DTO parsing, status-code mapping) stay outside runner.
|
||||||
|
|
||||||
|
## Config Fields Used
|
||||||
|
|
||||||
|
`Runner` does not read app config files directly.
|
||||||
|
|
||||||
|
It receives fully constructed repositories/readers/validators from adapters. Effective behavior depends on adapter wiring, including:
|
||||||
|
|
||||||
|
- prompt/profile directories
|
||||||
|
- schema base directory
|
||||||
|
- selected profile/runtime overrides in request
|
||||||
|
|
||||||
|
## External Adapters Used
|
||||||
|
|
||||||
|
`Runner` works with adapter implementations via interfaces. Current wiring from CLI/HTTP uses:
|
||||||
|
|
||||||
|
- filesystem prompt/profile repositories
|
||||||
|
- composite artifact reader
|
||||||
|
- Go-template prompt renderer
|
||||||
|
- OpenAI-compatible LLM client
|
||||||
|
- standard validator
|
||||||
|
|
||||||
|
## State And Resume Behavior
|
||||||
|
|
||||||
|
`Runner` is stateless across requests.
|
||||||
|
|
||||||
|
- No durable run-state storage.
|
||||||
|
- No built-in resume/skip checkpoints.
|
||||||
|
- Each `Run`/`Prepare` executes from request inputs and current repositories.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Key error classes surfaced from `Runner`:
|
||||||
|
|
||||||
|
- `ErrInvalidRequest`: invalid prompt/profile/request/runtime/API-key-env prerequisites.
|
||||||
|
- `ErrProfileLoad`: prompt or profile load failures.
|
||||||
|
- `ErrArtifactLoad`: artifact read failures.
|
||||||
|
- `ErrPromptRender`: template render failures.
|
||||||
|
- `ErrLLMGenerate`: model request failures.
|
||||||
|
- `ErrValidation`: validation runtime failures (including schema load/compile failures).
|
||||||
|
|
||||||
|
Validation content failures are not run errors:
|
||||||
|
|
||||||
|
- `Run` can succeed with `Validation.Status == failed`.
|
||||||
|
- CLI maps this to exit code `2`.
|
||||||
|
- HTTP returns `200` with failed validation details.
|
||||||
|
|
||||||
|
## Prepare Flow
|
||||||
|
|
||||||
|
`Prepare` performs:
|
||||||
|
|
||||||
|
1. validate request basics (prompt ID present).
|
||||||
|
2. load prompt definition by ID/version.
|
||||||
|
3. select profile ID:
|
||||||
|
- explicit request profile ID
|
||||||
|
- prompt `default_profile`
|
||||||
|
- otherwise request error
|
||||||
|
4. load execution profile.
|
||||||
|
5. merge effective runtime target:
|
||||||
|
- built-in execution defaults
|
||||||
|
- selected profile values
|
||||||
|
- request overrides
|
||||||
|
6. verify required `api_key_env` environment variable (name only; value is not returned).
|
||||||
|
7. resolve output contract and structured-output schema payload when `json_schema` mode is active.
|
||||||
|
8. read input artifacts.
|
||||||
|
9. render prompt messages.
|
||||||
|
10. compute prompt/input/render hashes and return `PreparedRun`.
|
||||||
|
|
||||||
|
`Prepare` does not call the LLM.
|
||||||
|
|
||||||
|
## Run Flow
|
||||||
|
|
||||||
|
`Run` performs:
|
||||||
|
|
||||||
|
1. generate run ID.
|
||||||
|
2. call `Prepare`.
|
||||||
|
3. call LLM with prepared messages/effective target/structured-output spec.
|
||||||
|
4. build output artifact content type from output format.
|
||||||
|
5. validate output.
|
||||||
|
6. optionally attempt bounded repair when repairer is injected and contract allows it.
|
||||||
|
7. return `RunResult` with artifact, raw output, validation, hashes, profile/model metadata, usage, and timestamps.
|
||||||
|
|
||||||
|
## Repair Hook Boundary
|
||||||
|
|
||||||
|
Repair attempts occur only when all are true:
|
||||||
|
|
||||||
|
- repairer is injected
|
||||||
|
- `repair_attempts > 0`
|
||||||
|
- validation status is `failed`
|
||||||
|
- validation mode is `json` or `json_schema`
|
||||||
|
|
||||||
|
Current production wiring boundary:
|
||||||
|
|
||||||
|
- CLI and HTTP adapters call `usecase.NewRunner(...)` (no repairer argument).
|
||||||
|
- Therefore normal CLI/HTTP execution does not perform repair attempts today.
|
||||||
|
|
||||||
|
## Tests To Inspect Before Changing
|
||||||
|
|
||||||
|
- `internal/usecase/runner_test.go`
|
||||||
|
- `internal/usecase/integration_test.go`
|
||||||
|
- `internal/adapter/cli/run_test.go`
|
||||||
|
- `internal/adapter/http/handler_test.go`
|
||||||
|
|
||||||
|
## Architectural Invariants
|
||||||
|
|
||||||
|
- `Run` reuses `Prepare`; prepare logic is not duplicated.
|
||||||
|
- Effective API-key environment-variable name may appear; resolved secret value must not.
|
||||||
|
- Structured-output schema document must load before LLM call for `json_schema` mode.
|
||||||
|
- Repair loops are bounded by `repair_attempts` and repairer presence.
|
||||||
|
- Runner stays transport-agnostic.
|
||||||
125
docs/operations.md
Normal file
125
docs/operations.md
Normal file
@@ -0,0 +1,125 @@
|
|||||||
|
# Operations Guide
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document covers day-to-day operation of the CLI and HTTP service for currently implemented behavior.
|
||||||
|
|
||||||
|
For command syntax, see [CLI reference](cli.md). For file formats and defaults, see [Configuration reference](config.md).
|
||||||
|
|
||||||
|
## Operational Model
|
||||||
|
|
||||||
|
Scriptorium executes one request at a time per CLI invocation or HTTP request.
|
||||||
|
|
||||||
|
Important boundaries:
|
||||||
|
|
||||||
|
- No durable run state is stored.
|
||||||
|
- No built-in resume, checkpoint, archive, or backup workflow exists.
|
||||||
|
- Recovery is rerun-based: fix inputs/config, then rerun.
|
||||||
|
|
||||||
|
## Filesystem Layout And Config
|
||||||
|
|
||||||
|
Scriptorium depends on:
|
||||||
|
|
||||||
|
- prompt definition files (`prompt_dir`)
|
||||||
|
- execution profile files (`profile_dir`)
|
||||||
|
- optional JSON schemas (`schema_dir`)
|
||||||
|
|
||||||
|
Config discovery order when `--config` is omitted:
|
||||||
|
|
||||||
|
1. `/usr/local/etc/scriptorium/config.yml`
|
||||||
|
2. `/etc/scriptorium/config.yml`
|
||||||
|
|
||||||
|
If neither exists, built-in defaults are used. If `--config <path>` is provided, that file must exist and parse successfully.
|
||||||
|
|
||||||
|
Built-in defaults relevant to operations:
|
||||||
|
|
||||||
|
- `schema_dir: .`
|
||||||
|
- `server.addr: :8080`
|
||||||
|
- `defaults.render_format: text`
|
||||||
|
|
||||||
|
## Normal CLI Workflow
|
||||||
|
|
||||||
|
Use `render` first when you need to verify prompt resolution and runtime settings without calling a model.
|
||||||
|
|
||||||
|
Use `run` for generation.
|
||||||
|
|
||||||
|
Typical sequence:
|
||||||
|
|
||||||
|
1. Confirm prompt/profile directories resolve through config or flags.
|
||||||
|
2. Confirm required input files exist and map to prompt input names.
|
||||||
|
3. Confirm required API-key environment variables are set.
|
||||||
|
4. Confirm the selected profile's model endpoint is reachable from the process environment.
|
||||||
|
5. Run `render` for preflight when changing prompt/profile/input wiring.
|
||||||
|
6. Run `run` for actual generation.
|
||||||
|
|
||||||
|
## Secrets Handling
|
||||||
|
|
||||||
|
Raw API keys are not accepted in config files, profile files as `api_key`, CLI flags, or HTTP request bodies.
|
||||||
|
|
||||||
|
Operational pattern:
|
||||||
|
|
||||||
|
- Set environment variables that hold secret values.
|
||||||
|
- Set profile `api_key_env` (or runtime override `api_key_env`) to the environment variable name.
|
||||||
|
- Keep process environments scoped to only required variables.
|
||||||
|
|
||||||
|
## HTTP Service Operation
|
||||||
|
|
||||||
|
Start service with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium serve --config ./examples/config.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Current inbound API behavior:
|
||||||
|
|
||||||
|
- Route: `POST /v1/runs`
|
||||||
|
- JSON request parsing rejects unknown fields.
|
||||||
|
- Validation content failures still return `200 OK` with `validation.status: "failed"`.
|
||||||
|
|
||||||
|
Security caveat:
|
||||||
|
|
||||||
|
- `serve` has no built-in authentication or authorization.
|
||||||
|
- Deploy only behind trusted controls (private network boundary, authenticated reverse proxy, API gateway, or equivalent).
|
||||||
|
|
||||||
|
## Output, Logs, And Exit Codes
|
||||||
|
|
||||||
|
`run` command:
|
||||||
|
|
||||||
|
- Generated artifact body goes to stdout by default.
|
||||||
|
- `--out` writes generated artifact to a file.
|
||||||
|
- Summary metadata line is written to stderr on success.
|
||||||
|
- Exit code `2` means generation completed but validation failed.
|
||||||
|
|
||||||
|
`render` command:
|
||||||
|
|
||||||
|
- Prepared-run output goes to stdout by default.
|
||||||
|
- `--out` writes prepared-run output to a file.
|
||||||
|
- Exit code is `0` on success and `1` on failure.
|
||||||
|
|
||||||
|
`serve` command:
|
||||||
|
|
||||||
|
- Startup and server errors are written to stderr.
|
||||||
|
|
||||||
|
## Validation Behavior In Operations
|
||||||
|
|
||||||
|
Validation modes (`none`, `basic`, `json`, `json_schema`) are defined by prompt output contract.
|
||||||
|
|
||||||
|
Operational interpretation:
|
||||||
|
|
||||||
|
- Validation runtime errors are hard failures (`run` exit `1`; HTTP error response).
|
||||||
|
- Validation content failures are soft failures (`run` exit `2`; HTTP `200` with failed status).
|
||||||
|
|
||||||
|
A failed validation run can still produce output. Decide whether to keep or discard that output in your surrounding workflow.
|
||||||
|
|
||||||
|
## Safe Recovery Steps
|
||||||
|
|
||||||
|
For failed runs or requests:
|
||||||
|
|
||||||
|
1. Capture stderr output or HTTP error code/message.
|
||||||
|
2. Confirm config path and directory settings.
|
||||||
|
3. Verify prompt/profile IDs and input mappings.
|
||||||
|
4. Verify API-key environment-variable presence when required.
|
||||||
|
5. Reproduce with `render --format json` when prompt/profile/input resolution is uncertain.
|
||||||
|
6. Rerun after correction.
|
||||||
|
|
||||||
|
Because Scriptorium does not persist run state, rerun is the canonical recovery path.
|
||||||
112
docs/policy/architecture.md
Normal file
112
docs/policy/architecture.md
Normal file
@@ -0,0 +1,112 @@
|
|||||||
|
# Architecture
|
||||||
|
|
||||||
|
This document is the development architecture policy for Scriptorium.
|
||||||
|
|
||||||
|
It is for developers and LLM coding agents. User-facing behavior belongs in `README.md` and the docs under `docs/` that target operators/users.
|
||||||
|
|
||||||
|
## Project Shape
|
||||||
|
|
||||||
|
Scriptorium is a narrow prompt-execution application with three entry paths:
|
||||||
|
|
||||||
|
- CLI `run`
|
||||||
|
- CLI `render`
|
||||||
|
- HTTP `POST /v1/runs` through `serve`
|
||||||
|
|
||||||
|
Domain behavior is centralized in `internal/usecase` and `internal/domain`.
|
||||||
|
|
||||||
|
## Core Principles
|
||||||
|
|
||||||
|
- Keep orchestration narrow: Scriptorium executes one prompt request; it is not a multi-step workflow engine.
|
||||||
|
- Keep adapter logic thin: adapters map external shapes to domain requests/results and should not hold domain decisions.
|
||||||
|
- Keep boundaries explicit: repositories/loaders/renderers/validators/LLM client stay behind package interfaces.
|
||||||
|
- Keep config strict: YAML/JSON decoding for external inputs should reject unknown fields.
|
||||||
|
- Keep secrets out of payloads: raw API key values must not be accepted or emitted.
|
||||||
|
|
||||||
|
## Package Boundaries
|
||||||
|
|
||||||
|
Current package map:
|
||||||
|
|
||||||
|
- `cmd/scriptorium`: process entrypoint.
|
||||||
|
- `internal/adapter/cli`: command parsing, app wiring for CLI commands, output behavior.
|
||||||
|
- `internal/adapter/http`: HTTP DTO mapping and error/status mapping.
|
||||||
|
- `internal/config`: application settings loading and CLI override precedence.
|
||||||
|
- `internal/defaults`: compile-time default constants.
|
||||||
|
- `internal/domain`: core request/result and contract types.
|
||||||
|
- `internal/usecase`: `Runner` prepare/run orchestration and repair-hook boundary.
|
||||||
|
- `internal/promptdef`: filesystem prompt-definition repository.
|
||||||
|
- `internal/profile`: filesystem execution-profile repository.
|
||||||
|
- `internal/artifact`: artifact reference readers.
|
||||||
|
- `internal/prompt`: template renderer.
|
||||||
|
- `internal/llm`: provider-neutral LLM client interface and OpenAI-compatible implementation.
|
||||||
|
- `internal/validate`: validator interfaces and standard implementation.
|
||||||
|
- `internal/format`: prepared-run output formatting.
|
||||||
|
|
||||||
|
Detailed component behavior is documented in:
|
||||||
|
|
||||||
|
- `docs/internal/runner.md`
|
||||||
|
- `docs/internal/adapters.md`
|
||||||
|
|
||||||
|
## Configuration And Precedence
|
||||||
|
|
||||||
|
Application settings are resolved as:
|
||||||
|
|
||||||
|
1. built-in defaults
|
||||||
|
2. config file values
|
||||||
|
3. CLI overrides
|
||||||
|
|
||||||
|
`config.yml` is for application wiring (directories, server address, render default format), not prompt/profile runtime execution settings.
|
||||||
|
|
||||||
|
Profile selection and runtime model resolution remain use-case concerns.
|
||||||
|
|
||||||
|
## State And Persistence Policy
|
||||||
|
|
||||||
|
Scriptorium has no durable run-state store.
|
||||||
|
|
||||||
|
- No built-in resume/checkpoint/archive behavior.
|
||||||
|
- Recovery model is rerun after correcting inputs/config/environment.
|
||||||
|
|
||||||
|
## External Integration Policy
|
||||||
|
|
||||||
|
Current external contracts:
|
||||||
|
|
||||||
|
- inbound HTTP contract: `POST /v1/runs`
|
||||||
|
- outbound model contract: OpenAI-compatible chat completions subset
|
||||||
|
- subprocess contract for integrators: CLI `run`/`render`
|
||||||
|
|
||||||
|
Integration docs belong under `docs/integrations/`.
|
||||||
|
|
||||||
|
## Error Handling And Logging
|
||||||
|
|
||||||
|
- Wrap errors with domain/operation context.
|
||||||
|
- Map domain errors to adapter-appropriate statuses/codes without leaking sensitive internals.
|
||||||
|
- Keep stderr summaries concise for CLI success/error paths.
|
||||||
|
- Never emit raw secret values.
|
||||||
|
|
||||||
|
## Testing Expectations
|
||||||
|
|
||||||
|
- Core runner behavior should be covered with isolated unit tests and fixture-based integration tests.
|
||||||
|
- Adapter behavior should be tested for parse/mapping/error semantics.
|
||||||
|
- Config parsing, prompt/profile loading, validator behavior, and LLM client error handling should remain covered by package tests.
|
||||||
|
- Repository-level docs/examples that claim runnable behavior should be validated by tests or smoke commands.
|
||||||
|
|
||||||
|
## Documentation Expectations
|
||||||
|
|
||||||
|
- Document implemented behavior only outside `docs/roadmap/`.
|
||||||
|
- Keep canonical reference locations stable (`docs/cli.md`, `docs/config.md`, `docs/operations.md`, `docs/troubleshooting.md`, `docs/internal/`).
|
||||||
|
- Update docs in the same change when architecture-relevant behavior changes.
|
||||||
|
|
||||||
|
## Architectural Invariants
|
||||||
|
|
||||||
|
- `Runner.Run` reuses `Runner.Prepare` flow.
|
||||||
|
- CLI and HTTP currently instantiate `Runner` without a repairer.
|
||||||
|
- Artifact reading supports `inline` and `file` references.
|
||||||
|
- Unknown input fields in config/prompt/profile/http JSON should be rejected by strict decoding.
|
||||||
|
- Raw API key values must not be accepted through config/HTTP payloads.
|
||||||
|
|
||||||
|
## Non-Goals
|
||||||
|
|
||||||
|
- Do not move orchestration responsibilities from external callers into Scriptorium.
|
||||||
|
- Do not add adapter-specific business logic in `internal/adapter/*` packages.
|
||||||
|
- Do not bypass repository/renderer/validator/LLM boundaries by introducing cross-package coupling.
|
||||||
|
|
||||||
|
Work that is not implemented belongs in `docs/roadmap/`.
|
||||||
103
docs/policy/development.md
Normal file
103
docs/policy/development.md
Normal file
@@ -0,0 +1,103 @@
|
|||||||
|
# Development Guide
|
||||||
|
|
||||||
|
This document defines contributor workflow for Scriptorium.
|
||||||
|
|
||||||
|
## Repository Layout
|
||||||
|
|
||||||
|
- `cmd/scriptorium`: application entrypoint.
|
||||||
|
- `internal/domain`: core contracts.
|
||||||
|
- `internal/usecase`: runner orchestration.
|
||||||
|
- `internal/adapter/cli`: CLI adapter.
|
||||||
|
- `internal/adapter/http`: HTTP adapter.
|
||||||
|
- `internal/config`: application settings loading and precedence.
|
||||||
|
- `internal/defaults`: default constants.
|
||||||
|
- `internal/promptdef`: prompt-definition repository.
|
||||||
|
- `internal/profile`: execution-profile repository.
|
||||||
|
- `internal/artifact`: artifact readers.
|
||||||
|
- `internal/prompt`: prompt rendering.
|
||||||
|
- `internal/llm`: LLM client interface and OpenAI-compatible implementation.
|
||||||
|
- `internal/validate`: validation interfaces and implementation.
|
||||||
|
- `internal/format`: prepared-run formatting.
|
||||||
|
- `docs/`: canonical documentation.
|
||||||
|
- `examples/`: copyable maintained examples and fixtures.
|
||||||
|
|
||||||
|
## Common Commands
|
||||||
|
|
||||||
|
Build:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go build ./cmd/scriptorium
|
||||||
|
```
|
||||||
|
|
||||||
|
Test:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
Targeted test runs commonly used during changes:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/adapter/cli ./internal/adapter/http ./internal/usecase
|
||||||
|
```
|
||||||
|
|
||||||
|
## Coding Conventions
|
||||||
|
|
||||||
|
- Prefer small interfaces at package boundaries.
|
||||||
|
- Keep adapter packages focused on translation and IO concerns.
|
||||||
|
- Keep domain/use-case logic outside adapters.
|
||||||
|
- Wrap errors with operation context.
|
||||||
|
- Use strict decoding for user-provided YAML/JSON where applicable.
|
||||||
|
- Avoid introducing dependencies unless they materially reduce risk/complexity.
|
||||||
|
|
||||||
|
## Dependency Policy
|
||||||
|
|
||||||
|
- Prefer standard library unless an external library is clearly justified.
|
||||||
|
- Current non-stdlib dependencies are intentionally small:
|
||||||
|
- `gopkg.in/yaml.v3` for YAML decoding.
|
||||||
|
- `github.com/santhosh-tekuri/jsonschema/v6` for JSON Schema validation.
|
||||||
|
- Do not leak dependency-specific types across unrelated package boundaries.
|
||||||
|
|
||||||
|
## How To Add App Config Fields
|
||||||
|
|
||||||
|
1. Add fields in `internal/config/config.go` (`Config`, `AppSettings`, and/or `CLIOverrides` as needed).
|
||||||
|
2. Apply defaults in `BuiltInDefaults()` when required.
|
||||||
|
3. Parse and validate in `applyConfig` / `ApplyCLIOverrides`.
|
||||||
|
4. Wire the field through the consuming adapter(s).
|
||||||
|
5. Add/update config tests in `internal/config/config_test.go`.
|
||||||
|
6. Update canonical docs (`docs/config.md`, and other affected docs).
|
||||||
|
|
||||||
|
## How To Add CLI Flags
|
||||||
|
|
||||||
|
1. Add flags in `internal/adapter/cli/run.go` for the relevant command.
|
||||||
|
2. Ensure precedence behavior remains consistent with app config rules.
|
||||||
|
3. Keep `run`, `render`, and `serve` flag surfaces intentionally scoped.
|
||||||
|
4. Add/update parser and command tests in `internal/adapter/cli/run_test.go`.
|
||||||
|
5. Update `docs/cli.md` and any related docs/examples.
|
||||||
|
|
||||||
|
## How To Add Adapters Or Adapter Capabilities
|
||||||
|
|
||||||
|
1. Define or reuse the appropriate interface boundary in domain/use-case packages.
|
||||||
|
2. Implement adapter code under `internal/adapter/<name>` (or relevant boundary package).
|
||||||
|
3. Keep business decisions in `internal/usecase`.
|
||||||
|
4. Add focused adapter tests for mapping, parse, and error behavior.
|
||||||
|
5. Document the new/changed boundary in `docs/internal/adapters.md`.
|
||||||
|
6. If external contract changes, update `docs/integrations/` in the same change.
|
||||||
|
|
||||||
|
## How To Update Prompt/Profile/Schema Assets
|
||||||
|
|
||||||
|
1. Keep prompt/profile/schema files valid under strict loaders.
|
||||||
|
2. Keep examples secret-free.
|
||||||
|
3. Re-run tests that cover prompt/profile/validation behavior.
|
||||||
|
4. Update `docs/config.md` and any docs that reference changed contracts.
|
||||||
|
|
||||||
|
## Documentation Update Expectations
|
||||||
|
|
||||||
|
When behavior changes:
|
||||||
|
|
||||||
|
1. Update canonical doc locations, not duplicate files.
|
||||||
|
2. Keep non-roadmap docs limited to implemented behavior.
|
||||||
|
3. Update links after file moves/renames.
|
||||||
|
4. Re-run relevant tests and smoke commands.
|
||||||
|
|
||||||
|
Docs work is complete only when code/tests/examples/docs agree.
|
||||||
356
docs/policy/documentation.md
Normal file
356
docs/policy/documentation.md
Normal file
@@ -0,0 +1,356 @@
|
|||||||
|
# Go Project Documentation Policy
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Project documentation must help four audiences:
|
||||||
|
|
||||||
|
1. users who need to run the application;
|
||||||
|
2. administrators/operators who need to configure and operate it;
|
||||||
|
3. developers who need to understand and change it safely;
|
||||||
|
4. LLM coding agents that need clear scope, boundaries, and invariants.
|
||||||
|
|
||||||
|
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
|
||||||
|
|
||||||
|
## Core Rules
|
||||||
|
|
||||||
|
### 1. Keep docs concise
|
||||||
|
|
||||||
|
Each document should cover a defined scope and only the essentials for that scope.
|
||||||
|
|
||||||
|
Avoid:
|
||||||
|
- long background explanations;
|
||||||
|
- repeated reference material;
|
||||||
|
- implementation detail in user-facing docs;
|
||||||
|
- aspirational language outside roadmap docs;
|
||||||
|
- verbose examples where one minimal example is clearer.
|
||||||
|
|
||||||
|
### 2. Document only implemented behavior outside roadmap files
|
||||||
|
|
||||||
|
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
|
||||||
|
|
||||||
|
- `docs/roadmap/`
|
||||||
|
|
||||||
|
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
|
||||||
|
|
||||||
|
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
|
||||||
|
|
||||||
|
### 3. Use canonical homes
|
||||||
|
|
||||||
|
Each type of information should have one canonical location.
|
||||||
|
|
||||||
|
Canonical homes:
|
||||||
|
|
||||||
|
- project purpose and quickstart: `README.md`
|
||||||
|
- development principles: `docs/policy/architecture.md`
|
||||||
|
- configuration reference: `docs/config.md`
|
||||||
|
- CLI reference: `docs/cli.md`
|
||||||
|
- operations and recovery: `docs/operations.md`
|
||||||
|
- troubleshooting: `docs/troubleshooting.md`
|
||||||
|
- implemented internals: `docs/internal/`
|
||||||
|
- future work: `docs/roadmap/`
|
||||||
|
- contributor workflow: `docs/policy/development.md`
|
||||||
|
- copyable examples: `examples/`
|
||||||
|
|
||||||
|
Other files should summarize briefly and link to the canonical source.
|
||||||
|
|
||||||
|
### 4. Keep examples real
|
||||||
|
|
||||||
|
Examples should be valid, maintained, and free of secrets.
|
||||||
|
|
||||||
|
Where practical:
|
||||||
|
- example configs should load successfully;
|
||||||
|
- example commands should match real CLI syntax;
|
||||||
|
- important examples should be covered by tests.
|
||||||
|
|
||||||
|
## Documentation Profiles
|
||||||
|
|
||||||
|
All projects require:
|
||||||
|
|
||||||
|
- `README.md`
|
||||||
|
- `docs/policy/architecture.md`
|
||||||
|
|
||||||
|
Additional docs depend on the project.
|
||||||
|
|
||||||
|
### Small library
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `docs/policy/development.md`, if contributor conventions are non-obvious
|
||||||
|
|
||||||
|
### Simple CLI
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/cli.md`
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `docs/policy/development.md`
|
||||||
|
|
||||||
|
### Config-driven CLI
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/cli.md`
|
||||||
|
- `docs/config.md`
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `examples/`
|
||||||
|
- `docs/policy/development.md`
|
||||||
|
|
||||||
|
### Stateful or operator-facing application
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/cli.md`, if CLI-based
|
||||||
|
- `docs/config.md`, if config-driven
|
||||||
|
- `docs/operations.md`
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `docs/troubleshooting.md`
|
||||||
|
- `examples/`
|
||||||
|
- `docs/policy/development.md`
|
||||||
|
|
||||||
|
### Modular, staged, service-oriented, or orchestration application
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/cli.md`, if CLI-based
|
||||||
|
- `docs/config.md`, if config-driven
|
||||||
|
- `docs/operations.md`
|
||||||
|
- `docs/internal/`
|
||||||
|
- `docs/policy/development.md`
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `docs/troubleshooting.md`
|
||||||
|
- validated examples under `examples/`
|
||||||
|
|
||||||
|
## Required Documents
|
||||||
|
|
||||||
|
### README.md
|
||||||
|
|
||||||
|
**Audience:** users, administrators, operators
|
||||||
|
|
||||||
|
The README is the outward-facing project orientation page.
|
||||||
|
|
||||||
|
It should include, in order:
|
||||||
|
|
||||||
|
1. concise description;
|
||||||
|
2. elevator pitch;
|
||||||
|
3. shortest useful command or usage example;
|
||||||
|
4. links to targeted docs.
|
||||||
|
|
||||||
|
The README should be short. It is not a manual.
|
||||||
|
|
||||||
|
The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.)
|
||||||
|
|
||||||
|
### docs/policy/architecture.md
|
||||||
|
|
||||||
|
**Audience:** developers, LLM coding agents
|
||||||
|
|
||||||
|
`docs/policy/architecture.md` is required for every project.
|
||||||
|
|
||||||
|
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
|
||||||
|
|
||||||
|
It should include:
|
||||||
|
|
||||||
|
- project shape;
|
||||||
|
- core design principles;
|
||||||
|
- package and boundary philosophy;
|
||||||
|
- state/persistence philosophy, if applicable;
|
||||||
|
- external integration philosophy, if applicable;
|
||||||
|
- error-handling and logging principles;
|
||||||
|
- testing expectations;
|
||||||
|
- documentation expectations;
|
||||||
|
- architectural invariants;
|
||||||
|
- explicit non-goals, if useful.
|
||||||
|
|
||||||
|
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
|
||||||
|
|
||||||
|
### docs/policy/development.md
|
||||||
|
|
||||||
|
**Audience:** developers, LLM coding agents
|
||||||
|
|
||||||
|
Required for projects maintained by humans and LLM coding agents.
|
||||||
|
|
||||||
|
It should include:
|
||||||
|
|
||||||
|
- repository layout;
|
||||||
|
- build/test commands;
|
||||||
|
- coding conventions;
|
||||||
|
- dependency policy;
|
||||||
|
- how to add config fields;
|
||||||
|
- how to add CLI flags;
|
||||||
|
- how to add stages/modules/adapters, if applicable;
|
||||||
|
- how to update examples;
|
||||||
|
- documentation update expectations.
|
||||||
|
|
||||||
|
### docs/config.md
|
||||||
|
|
||||||
|
**Audience:** administrators, operators, advanced users
|
||||||
|
|
||||||
|
Required for applications with configuration files.
|
||||||
|
|
||||||
|
It should include, in order:
|
||||||
|
|
||||||
|
1. config file locations and discovery precedence;
|
||||||
|
2. minimal working config;
|
||||||
|
3. production-oriented config;
|
||||||
|
4. full configuration reference;
|
||||||
|
5. secrets handling, if applicable;
|
||||||
|
6. links to maintained examples.
|
||||||
|
|
||||||
|
The full configuration reference should be canonical.
|
||||||
|
|
||||||
|
### docs/cli.md
|
||||||
|
|
||||||
|
**Audience:** users, administrators, operators
|
||||||
|
|
||||||
|
Required for CLI applications.
|
||||||
|
|
||||||
|
It should include, in order:
|
||||||
|
|
||||||
|
1. shortest useful command;
|
||||||
|
2. command overview;
|
||||||
|
3. complete flag reference;
|
||||||
|
4. common workflows;
|
||||||
|
5. diagnostic or recovery commands, if applicable.
|
||||||
|
|
||||||
|
Explain when commands are useful, not just their syntax.
|
||||||
|
|
||||||
|
### docs/operations.md
|
||||||
|
|
||||||
|
**Audience:** administrators, operators
|
||||||
|
|
||||||
|
Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures.
|
||||||
|
|
||||||
|
It should cover:
|
||||||
|
|
||||||
|
- normal workflow;
|
||||||
|
- filesystem layout;
|
||||||
|
- remote storage layout, if applicable;
|
||||||
|
- logs and manifests;
|
||||||
|
- resume/retry behavior;
|
||||||
|
- cleanup behavior;
|
||||||
|
- archive/backup behavior;
|
||||||
|
- safe recovery procedures;
|
||||||
|
- operational caveats.
|
||||||
|
|
||||||
|
### docs/troubleshooting.md
|
||||||
|
|
||||||
|
**Audience:** administrators, operators
|
||||||
|
|
||||||
|
Recommended once recurring failure modes exist.
|
||||||
|
|
||||||
|
Each entry should include:
|
||||||
|
|
||||||
|
- symptom;
|
||||||
|
- likely cause;
|
||||||
|
- diagnostic command or inspection step;
|
||||||
|
- safe fix;
|
||||||
|
- relevant links.
|
||||||
|
|
||||||
|
### docs/internal/
|
||||||
|
|
||||||
|
**Audience:** developers, LLM coding agents
|
||||||
|
|
||||||
|
Required for modular, staged, service-oriented, or orchestration projects.
|
||||||
|
|
||||||
|
This directory describes implemented internal components. It is not the roadmap.
|
||||||
|
|
||||||
|
Use one file per major component where useful.
|
||||||
|
|
||||||
|
Each component doc should include:
|
||||||
|
|
||||||
|
1. purpose;
|
||||||
|
2. inputs and outputs;
|
||||||
|
3. boundaries;
|
||||||
|
4. config fields used;
|
||||||
|
5. external adapters used;
|
||||||
|
6. state or manifest behavior, if applicable;
|
||||||
|
7. skip/resume behavior, if applicable;
|
||||||
|
8. failure behavior;
|
||||||
|
9. tests to inspect before changing;
|
||||||
|
10. architectural invariants.
|
||||||
|
|
||||||
|
### docs/roadmap/
|
||||||
|
|
||||||
|
**Audience:** maintainers, developers, LLM coding agents
|
||||||
|
|
||||||
|
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
|
||||||
|
|
||||||
|
Roadmap docs should clearly distinguish:
|
||||||
|
|
||||||
|
- proposed work;
|
||||||
|
- accepted plans;
|
||||||
|
- deferred ideas;
|
||||||
|
- rejected ideas;
|
||||||
|
- implementation prompts or task breakdowns, if useful.
|
||||||
|
|
||||||
|
Roadmap docs should not be confused with current behavior.
|
||||||
|
|
||||||
|
### docs/integrations/
|
||||||
|
|
||||||
|
**Audience:** developers, LLM coding agents
|
||||||
|
|
||||||
|
Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
|
||||||
|
|
||||||
|
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses.
|
||||||
|
|
||||||
|
Use one file per integration where useful.
|
||||||
|
|
||||||
|
## Examples Directory
|
||||||
|
|
||||||
|
Projects with non-trivial configuration or workflows should include `examples/`.
|
||||||
|
|
||||||
|
Useful examples include:
|
||||||
|
|
||||||
|
- minimal working config;
|
||||||
|
- production-oriented config;
|
||||||
|
- full annotated config;
|
||||||
|
- local development config;
|
||||||
|
- remote/object-storage config;
|
||||||
|
- minimal session/input file.
|
||||||
|
|
||||||
|
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
|
||||||
|
|
||||||
|
## Security and Privacy
|
||||||
|
|
||||||
|
Docs and examples must not include:
|
||||||
|
|
||||||
|
- real API keys;
|
||||||
|
- tokens;
|
||||||
|
- passwords;
|
||||||
|
- private keys;
|
||||||
|
- private environment dumps;
|
||||||
|
- sensitive user data;
|
||||||
|
- raw private transcripts;
|
||||||
|
- private infrastructure details unless intentionally public.
|
||||||
|
|
||||||
|
Document secret-handling mechanisms, not actual secret values.
|
||||||
|
|
||||||
|
## Maintenance Rules
|
||||||
|
|
||||||
|
When docs change, verify the affected behavior.
|
||||||
|
|
||||||
|
Where practical:
|
||||||
|
|
||||||
|
- load example config files in tests;
|
||||||
|
- test CLI examples or command parser behavior;
|
||||||
|
- validate documented flags against real flags;
|
||||||
|
- remove stale references;
|
||||||
|
- update links after renames;
|
||||||
|
- keep roadmap content out of non-roadmap docs.
|
||||||
|
|
||||||
|
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
|
||||||
|
|
||||||
|
Documentation is complete only when it matches the current code.
|
||||||
|
|
||||||
|
## Documentation Change Checklist
|
||||||
|
|
||||||
|
Before merging documentation changes, verify:
|
||||||
|
|
||||||
|
- README is concise and orientation-focused.
|
||||||
|
- `docs/policy/architecture.md` describes development principles.
|
||||||
|
- Future work appears only under `docs/roadmap/`.
|
||||||
|
- User-facing docs avoid unnecessary internals.
|
||||||
|
- Developer-facing docs preserve boundaries and invariants.
|
||||||
|
- Config examples match the schema.
|
||||||
|
- CLI examples match real commands and flags.
|
||||||
|
- Defaults appear in the canonical config reference.
|
||||||
|
- No secrets or private data are included.
|
||||||
|
- Links are accurate.
|
||||||
519
docs/roadmap/audit.md
Normal file
519
docs/roadmap/audit.md
Normal file
@@ -0,0 +1,519 @@
|
|||||||
|
# Code Quality And Deduplication Audit
|
||||||
|
|
||||||
|
## Executive Summary
|
||||||
|
|
||||||
|
Overall code quality is solid for the current project size. The repository follows the documented shape: thin CLI/HTTP adapters, a central `Runner`, strict config/prompt/profile loading, narrow filesystem and LLM adapters, and package tests around the important public behaviors.
|
||||||
|
|
||||||
|
The top three refactoring targets are:
|
||||||
|
|
||||||
|
1. centralize execution-target field mapping across profiles, CLI overrides, HTTP DTOs, prepared output, and LLM request serialization;
|
||||||
|
2. share filesystem repository scanning helpers between prompt and profile repositories;
|
||||||
|
3. reduce CLI command wiring duplication for app settings, runner construction, and common preflight behavior.
|
||||||
|
|
||||||
|
The codebase appears ready for a limited cleanup pass. No major architectural rewrite is warranted. The main risk before a release is maintenance drift when adding fields or policy to execution settings, repositories, or command setup.
|
||||||
|
|
||||||
|
## Repository Map Reviewed
|
||||||
|
|
||||||
|
Main areas inspected:
|
||||||
|
|
||||||
|
- `cmd/scriptorium`: process entrypoint path, via repository layout and command docs.
|
||||||
|
- `internal/adapter/cli`: `run`, `render`, `serve` parsing, config loading, app wiring, stdout/stderr behavior, summary output, exit-code policy.
|
||||||
|
- `internal/adapter/http`: request DTOs, strict JSON decoding, domain mapping, response metadata mapping, error mapping.
|
||||||
|
- `internal/config`: app config loading, defaults, strict YAML decoding, CLI override precedence.
|
||||||
|
- `internal/defaults`: built-in runtime/config/content-type defaults.
|
||||||
|
- `internal/domain`: domain request/result/profile/target/artifact/validation types.
|
||||||
|
- `internal/usecase`: `Runner.Prepare`, `Runner.Run`, execution target merge, schema structured-output setup, validation and repair policy.
|
||||||
|
- `internal/promptdef`: recursive prompt YAML loading, prompt normalization, content-file resolution, prompt output-contract validation.
|
||||||
|
- `internal/profile`: recursive profile YAML loading, strict decoding, duplicate ID handling, profile validation, raw API-key rejection.
|
||||||
|
- `internal/artifact`: `inline` and `file` artifact resolution.
|
||||||
|
- `internal/prompt`: Go template rendering and required input checks.
|
||||||
|
- `internal/llm`: OpenAI-compatible request construction, timeout/API-key handling, response parsing.
|
||||||
|
- `internal/validate`: basic/JSON/JSON Schema validation and schema document loading.
|
||||||
|
- `internal/format`: prepared-run text/JSON output.
|
||||||
|
- `examples/`: maintained example config, prompts, profiles, schemas, fixtures, HTTP/render examples.
|
||||||
|
- Package tests under `internal/**`.
|
||||||
|
|
||||||
|
Major execution paths reviewed:
|
||||||
|
|
||||||
|
- CLI `run`: parse flags/config, build `RunRequest`, construct runner, call LLM, write artifact, print summary, choose exit code.
|
||||||
|
- CLI `render`: parse flags/config, build `RunRequest`, construct runner without LLM, prepare only, format prepared run.
|
||||||
|
- CLI `serve`: parse flags/config, construct runner, expose HTTP `POST /v1/runs`.
|
||||||
|
- HTTP `POST /v1/runs`: strict decode DTO, map to `RunRequest`, map `RunResult` to JSON response.
|
||||||
|
- Runner `Prepare` and `Run`: prompt/profile/artifact/schema resolution, render, LLM generate, validation, repair hook.
|
||||||
|
|
||||||
|
Areas not deeply inspected:
|
||||||
|
|
||||||
|
- No `internal/app`, `internal/stage`, `internal/modules`, `internal/storage`, `internal/manifest`, `pkg`, or external test-suite directories exist in the current repository.
|
||||||
|
- Full test execution was not run because this is a report-only pass; inspection was static.
|
||||||
|
|
||||||
|
## High-Confidence Deduplication Opportunities
|
||||||
|
|
||||||
|
### 1. Execution Target Field Mapping Is Repeated Across Boundaries
|
||||||
|
|
||||||
|
Affected files/packages:
|
||||||
|
|
||||||
|
- `internal/domain/domain.go`
|
||||||
|
- `internal/usecase/runner.go`
|
||||||
|
- `internal/adapter/cli/run.go`
|
||||||
|
- `internal/adapter/http/dto.go`
|
||||||
|
- `internal/adapter/http/handler.go`
|
||||||
|
- `internal/llm/openai_compatible_client.go`
|
||||||
|
- `internal/format/prepared_run.go`
|
||||||
|
- tests in `internal/usecase`, `internal/adapter/http`, `internal/adapter/cli`, `internal/llm`, and `internal/format`
|
||||||
|
|
||||||
|
Duplicated or near-duplicated behavior:
|
||||||
|
|
||||||
|
- Execution settings fields are listed and copied in several places: profile-to-target conversion, target merge, CLI runtime override construction, HTTP request mapping, HTTP metadata mapping, prepared-run text output, and OpenAI-compatible request construction.
|
||||||
|
- Adding `service_tier` required coordinated edits across many of these sites, which is a strong signal that field-level mapping is too scattered.
|
||||||
|
|
||||||
|
Why it matters:
|
||||||
|
|
||||||
|
- A future profile/runtime field can easily be parsed but not sent, displayed but not merged, accepted over HTTP but not included in metadata, or tested in one command but not another.
|
||||||
|
- This is public-interface drift risk because CLI, HTTP, render output, and provider requests all expose different slices of the same effective execution target.
|
||||||
|
|
||||||
|
Recommended refactor:
|
||||||
|
|
||||||
|
- Keep transport DTOs local to adapters, but add small mapping helpers at each boundary:
|
||||||
|
- a use-case/domain helper for profile-to-target copy and execution-target merge;
|
||||||
|
- an HTTP helper such as `executionTargetFromModelOverrideDTO` and `modelParamsDTOFromExecutionTarget`;
|
||||||
|
- an LLM helper such as `openAIChatRequestFromGenerateRequest` for outbound serialization.
|
||||||
|
- Add a table-driven test that constructs an `ExecutionTarget` with every supported field and verifies the HTTP metadata DTO and LLM request payload retain the intended fields.
|
||||||
|
- Avoid reflection-based generic copying; explicit mapping is still clearer here.
|
||||||
|
|
||||||
|
Suggested tests:
|
||||||
|
|
||||||
|
- Expand runner merge tests to use an all-fields target.
|
||||||
|
- Add HTTP DTO mapping tests that fail when a domain execution field is omitted from request or metadata mapping.
|
||||||
|
- Add LLM request serialization tests for all serialized fields and explicit omission of non-serialized fields.
|
||||||
|
|
||||||
|
Risk level: high. The refactor itself is low-to-medium implementation risk, but the duplicated behavior has high drift risk.
|
||||||
|
|
||||||
|
### 2. Prompt And Profile Filesystem Repository Scanning Is Duplicated
|
||||||
|
|
||||||
|
Affected files/packages:
|
||||||
|
|
||||||
|
- `internal/promptdef/filesystem_repository.go`
|
||||||
|
- `internal/profile/filesystem_repository.go`
|
||||||
|
- repository tests in `internal/promptdef` and `internal/profile`
|
||||||
|
|
||||||
|
Duplicated or near-duplicated behavior:
|
||||||
|
|
||||||
|
- Both repositories recursively walk configured directories, filter `.yaml`/`.yml`, sort paths, compute relative paths, derive likely IDs from filenames, use strict YAML decoding, partially decode IDs to decide whether to surface malformed likely-target files, detect duplicate IDs, and include relative paths in errors.
|
||||||
|
|
||||||
|
Why it matters:
|
||||||
|
|
||||||
|
- Prompt/profile repository policy should remain aligned: nested scanning, stable ordering, duplicate ID errors, likely-target malformed file behavior, and relative-path diagnostics.
|
||||||
|
- Future changes to repository scanning or extension rules would need to be made in two packages.
|
||||||
|
|
||||||
|
Recommended refactor:
|
||||||
|
|
||||||
|
- Introduce a narrow internal helper for filesystem catalog behavior, for example `internal/filecatalog` or another small package whose scope is only:
|
||||||
|
- recursive YAML file discovery with context cancellation;
|
||||||
|
- stable sorting;
|
||||||
|
- relative path formatting;
|
||||||
|
- YAML extension checks;
|
||||||
|
- filename stem extraction.
|
||||||
|
- Keep prompt/profile-specific normalization and validation in their current packages.
|
||||||
|
- Do not make a generic repository framework.
|
||||||
|
|
||||||
|
Suggested tests:
|
||||||
|
|
||||||
|
- Add helper-level tests for recursive YAML discovery, ordering, extension filtering, and relative path output.
|
||||||
|
- Keep existing prompt/profile behavior tests unchanged to confirm public error behavior survives.
|
||||||
|
|
||||||
|
Risk level: high for drift prevention; low implementation risk if the helper stays small.
|
||||||
|
|
||||||
|
### 3. CLI Command Setup And Runner Wiring Are Repeated
|
||||||
|
|
||||||
|
Affected files/packages:
|
||||||
|
|
||||||
|
- `internal/adapter/cli/run.go`
|
||||||
|
- `internal/adapter/cli/run_test.go`
|
||||||
|
|
||||||
|
Duplicated or near-duplicated behavior:
|
||||||
|
|
||||||
|
- `run`, `render`, and `serve` each construct similar runner dependencies.
|
||||||
|
- `run` and `serve` both create an OpenAI-compatible client with default timeout.
|
||||||
|
- `run`, `render`, and `serve` all resolve app settings, clean dirs, validate required prompt/profile dirs, and create filesystem repositories/validator/renderer/readers.
|
||||||
|
- `run` and `render` share most execution request flags and request construction, with command-specific differences around `--schema-dir`, `--format`, and LLM use.
|
||||||
|
|
||||||
|
Why it matters:
|
||||||
|
|
||||||
|
- Adding or changing a dependency, default, or preflight check can drift between commands.
|
||||||
|
- The current duplication is still readable, but it is large enough that future command additions or option changes will be error-prone.
|
||||||
|
|
||||||
|
Recommended refactor:
|
||||||
|
|
||||||
|
- Add a small CLI-local wiring helper, such as `newRunnerFromDirs(promptDir, profileDir, schemaDir string, llmClient llm.Client) *usecase.Runner`.
|
||||||
|
- Add a small CLI-local resolved settings struct for common `prompt_dir`, `profile_dir`, `schema_dir`, and render-format handling.
|
||||||
|
- Preserve the existing command-specific parse functions and flag surfaces; do not hide command behavior behind a generic command framework.
|
||||||
|
|
||||||
|
Suggested tests:
|
||||||
|
|
||||||
|
- Keep parser tests for each command.
|
||||||
|
- Add one test that verifies `run`, `render`, and `serve` use the same configured prompt/profile/schema dirs by exercising config-derived dirs.
|
||||||
|
- Keep command-level success tests for `run` and `render`.
|
||||||
|
|
||||||
|
Risk level: medium. Behavior is public, but a small CLI-local helper can be behavior-preserving.
|
||||||
|
|
||||||
|
### 4. HTTP Error Mapping Depends On Error Message Substrings
|
||||||
|
|
||||||
|
Affected files/packages:
|
||||||
|
|
||||||
|
- `internal/usecase/runner.go`
|
||||||
|
- `internal/adapter/http/handler.go`
|
||||||
|
- `internal/adapter/http/handler_test.go`
|
||||||
|
|
||||||
|
Duplicated or near-duplicated behavior:
|
||||||
|
|
||||||
|
- Runner creates `ErrInvalidRequest` with human-readable details for cases such as missing profile selection and missing API-key env.
|
||||||
|
- HTTP maps some specific invalid-request cases by checking `strings.Contains(err.Error(), ...)`.
|
||||||
|
|
||||||
|
Why it matters:
|
||||||
|
|
||||||
|
- User-facing HTTP error codes can drift if a runner error message is clarified.
|
||||||
|
- This crosses package boundaries in a brittle way: HTTP should depend on stable error identity, not exact prose from `internal/usecase`.
|
||||||
|
|
||||||
|
Recommended refactor:
|
||||||
|
|
||||||
|
- Add narrow sentinel errors or typed invalid-request reasons in `internal/usecase`, for example profile-required and API-key-env-missing.
|
||||||
|
- Keep HTTP response messages stable and adapter-owned.
|
||||||
|
- Do not expose HTTP-specific error codes from the runner.
|
||||||
|
|
||||||
|
Suggested tests:
|
||||||
|
|
||||||
|
- HTTP error mapping tests should assert `profile_required` and `api_key_env_missing` via sentinel wrapping, not via message matching.
|
||||||
|
- Runner tests should assert `errors.Is` for the new reason errors.
|
||||||
|
|
||||||
|
Risk level: high for public API stability; low-to-medium implementation risk.
|
||||||
|
|
||||||
|
## Medium-Confidence Opportunities
|
||||||
|
|
||||||
|
### 1. Strict YAML Decode Setup Is Repeated
|
||||||
|
|
||||||
|
Affected files/packages:
|
||||||
|
|
||||||
|
- `internal/config/config.go`
|
||||||
|
- `internal/promptdef/filesystem_repository.go`
|
||||||
|
- `internal/profile/filesystem_repository.go`
|
||||||
|
|
||||||
|
Duplicated or near-duplicated behavior:
|
||||||
|
|
||||||
|
- Each package constructs a YAML decoder and enables `KnownFields(true)`.
|
||||||
|
|
||||||
|
Semantic differences that may be intentional:
|
||||||
|
|
||||||
|
- Prompt/profile loaders need likely-target behavior and raw `api_key` handling.
|
||||||
|
- Config loading has its own explicit/implicit file search policy.
|
||||||
|
|
||||||
|
Why it matters:
|
||||||
|
|
||||||
|
- Strict decoding is an architectural invariant. A future YAML loader could forget to enable it.
|
||||||
|
|
||||||
|
Recommended refactor:
|
||||||
|
|
||||||
|
- Consider a tiny helper for strict YAML decode from bytes.
|
||||||
|
- Keep package-specific error wrapping and partial ID decode logic local.
|
||||||
|
|
||||||
|
Suggested tests:
|
||||||
|
|
||||||
|
- Existing unknown-field tests in config, promptdef, and profile should remain.
|
||||||
|
- Add any new YAML-consuming package with an unknown-field test.
|
||||||
|
|
||||||
|
Risk level: medium.
|
||||||
|
|
||||||
|
### 2. Schema Path Resolution Is Centralized, But Schema Loading Happens Twice
|
||||||
|
|
||||||
|
Affected files/packages:
|
||||||
|
|
||||||
|
- `internal/usecase/runner.go`
|
||||||
|
- `internal/validate/standard_validator.go`
|
||||||
|
|
||||||
|
Duplicated or near-duplicated behavior:
|
||||||
|
|
||||||
|
- For `json_schema`, `Runner.Prepare` asks the validator to load the schema document for provider-level structured output.
|
||||||
|
- Later validation resolves and compiles the same schema path again.
|
||||||
|
|
||||||
|
Semantic differences that may be intentional:
|
||||||
|
|
||||||
|
- Provider structured-output payload needs the raw JSON schema document.
|
||||||
|
- Runtime validation needs a compiled schema.
|
||||||
|
|
||||||
|
Why it matters:
|
||||||
|
|
||||||
|
- The current behavior is correct, but schema file errors can surface at different phases and the same file is read more than once during `Run`.
|
||||||
|
- Future caching or schema behavior changes should have one clear owner.
|
||||||
|
|
||||||
|
Recommended refactor:
|
||||||
|
|
||||||
|
- Keep path resolution inside `internal/validate`.
|
||||||
|
- Consider a schema service/loader interface that can return both raw document and compiled schema from one path, only if schema-related work grows.
|
||||||
|
- Do not add caching unless repeated schema loads become a measured cost.
|
||||||
|
|
||||||
|
Suggested tests:
|
||||||
|
|
||||||
|
- Keep existing nested schema path tests.
|
||||||
|
- Add a test that `Prepare` fails before LLM call when structured-output schema cannot load.
|
||||||
|
|
||||||
|
Risk level: medium.
|
||||||
|
|
||||||
|
### 3. Artifact Hashing And Output Artifact Construction Are Split
|
||||||
|
|
||||||
|
Affected files/packages:
|
||||||
|
|
||||||
|
- `internal/artifact/reader.go`
|
||||||
|
- `internal/usecase/runner.go`
|
||||||
|
|
||||||
|
Duplicated or near-duplicated behavior:
|
||||||
|
|
||||||
|
- Input artifacts and generated output artifacts both compute SHA-256 hashes and fill size/body/content-type fields.
|
||||||
|
|
||||||
|
Semantic differences that may be intentional:
|
||||||
|
|
||||||
|
- Input artifacts derive content type from source extension or inline default.
|
||||||
|
- Output artifacts derive content type from prompt output format and always use the default output artifact name.
|
||||||
|
|
||||||
|
Why it matters:
|
||||||
|
|
||||||
|
- Hash algorithm and artifact metadata policy should remain consistent.
|
||||||
|
|
||||||
|
Recommended refactor:
|
||||||
|
|
||||||
|
- Consider an `artifact.Build` or `artifact.HashBody` helper only for shared hash/size construction.
|
||||||
|
- Keep source-specific content type and naming decisions local.
|
||||||
|
|
||||||
|
Suggested tests:
|
||||||
|
|
||||||
|
- Existing artifact reader tests and runner output content-type tests should cover behavior.
|
||||||
|
- Add a small helper test if a shared hash function is introduced.
|
||||||
|
|
||||||
|
Risk level: medium-low.
|
||||||
|
|
||||||
|
### 4. Domain Contains An Unsupported `s3` Artifact Reference Constant
|
||||||
|
|
||||||
|
Affected files/packages:
|
||||||
|
|
||||||
|
- `internal/domain/domain.go`
|
||||||
|
- `internal/artifact/reader.go`
|
||||||
|
- docs that state only `inline` and `file` are implemented
|
||||||
|
|
||||||
|
Duplicated or near-duplicated behavior:
|
||||||
|
|
||||||
|
- Not a duplication issue. This is a cleanup issue: `domain.ArtifactRefS3` exists, but no reader supports it and docs do not document it as implemented.
|
||||||
|
|
||||||
|
Semantic differences that may be intentional:
|
||||||
|
|
||||||
|
- The constant may be a placeholder for a future adapter.
|
||||||
|
|
||||||
|
Why it matters:
|
||||||
|
|
||||||
|
- Future-facing code outside `docs/roadmap/` can confuse maintainers and tests. It also weakens the otherwise clean "implemented behavior only" policy.
|
||||||
|
|
||||||
|
Recommended refactor:
|
||||||
|
|
||||||
|
- Remove the constant if no near-term S3 implementation is planned.
|
||||||
|
- If kept, add a code comment that it is intentionally unsupported today and ensure docs continue to state only `inline` and `file` are supported.
|
||||||
|
|
||||||
|
Suggested tests:
|
||||||
|
|
||||||
|
- Existing unsupported artifact type tests should continue to pass.
|
||||||
|
|
||||||
|
Risk level: medium-low.
|
||||||
|
|
||||||
|
### 5. Test Fixture Setup Is Repeated In CLI And Use-Case Tests
|
||||||
|
|
||||||
|
Affected files/packages:
|
||||||
|
|
||||||
|
- `internal/adapter/cli/run_test.go`
|
||||||
|
- `internal/usecase/runner_test.go`
|
||||||
|
- `internal/usecase/integration_test.go`
|
||||||
|
|
||||||
|
Duplicated or near-duplicated behavior:
|
||||||
|
|
||||||
|
- Tests repeatedly construct temp prompt/profile dirs, write prompt/profile YAML, set up fake LLMs/readers/renderers/validators, and build runners.
|
||||||
|
|
||||||
|
Semantic differences that may be intentional:
|
||||||
|
|
||||||
|
- Package-local tests avoid exporting test helpers and keep each package independent.
|
||||||
|
|
||||||
|
Why it matters:
|
||||||
|
|
||||||
|
- Repeated setup makes behavior-preserving refactors noisier and can obscure the exact behavior under test.
|
||||||
|
|
||||||
|
Recommended refactor:
|
||||||
|
|
||||||
|
- Add package-local helper builders where duplication is highest, especially in CLI command tests.
|
||||||
|
- Avoid a cross-package test utility package unless multiple packages need the same public fixture contract.
|
||||||
|
|
||||||
|
Suggested tests:
|
||||||
|
|
||||||
|
- This is test cleanup only; existing test assertions should remain equivalent.
|
||||||
|
|
||||||
|
Risk level: low.
|
||||||
|
|
||||||
|
## Boundary And Responsibility Concerns
|
||||||
|
|
||||||
|
- HTTP error mapping currently depends on runner error message text. Stable reason identity should live in `internal/usecase`; HTTP status/code/message mapping should remain in `internal/adapter/http`.
|
||||||
|
- Execution-target merge policy lives in `internal/usecase`, which fits the architecture. The concern is not placement but incomplete centralization of field mapping around that policy.
|
||||||
|
- CLI app wiring currently constructs concrete repositories/readers/renderers/validators inline in each command. That is acceptable adapter responsibility, but repeated wiring should be centralized within the CLI adapter package.
|
||||||
|
- Prompt/profile filesystem traversal is duplicated in two repository packages. A narrow filesystem catalog helper would fit the architecture because it would not own prompt/profile policy.
|
||||||
|
- LLM provider request shape is properly isolated in `internal/llm`. It should remain there; do not move OpenAI/OpenRouter request-field policy into runner or prompt definitions.
|
||||||
|
|
||||||
|
## Path, Key, And Naming Construction Review
|
||||||
|
|
||||||
|
Path and naming construction is mostly explicit and low-risk:
|
||||||
|
|
||||||
|
- Config paths are cleaned in `internal/config` and again in CLI finalization.
|
||||||
|
- Prompt `content_file` paths are resolved relative to the prompt YAML file in `internal/promptdef`.
|
||||||
|
- Schema paths are resolved through `internal/validate.StandardValidator`.
|
||||||
|
- OpenAI-compatible endpoint paths use `defaults.OpenAIChatCompletionsPath`.
|
||||||
|
- Output artifact name comes from `defaults.OutputArtifactName`.
|
||||||
|
|
||||||
|
Areas needing cleanup:
|
||||||
|
|
||||||
|
- Prompt/profile recursive YAML discovery and relative-path formatting should share one helper.
|
||||||
|
- CLI path cleaning after config resolution is repeated and should be consolidated with common command settings finalization.
|
||||||
|
- Structured schema names are derived in `internal/usecase`; this is currently one place and should stay there unless structured-output support expands.
|
||||||
|
|
||||||
|
No remote keys, cache paths, manifest paths, lock files, or generated report paths exist in the current implementation.
|
||||||
|
|
||||||
|
## Resolution And Catalog Review
|
||||||
|
|
||||||
|
Named concept resolution is mostly consistent:
|
||||||
|
|
||||||
|
- Prompt resolution uses YAML `id` and optional `version`, not file path.
|
||||||
|
- Profile resolution uses YAML `id`, not file path.
|
||||||
|
- Prompt/profile subdirectories are organizational only.
|
||||||
|
- Duplicate prompt/profile IDs fail instead of choosing first match.
|
||||||
|
- Schema resolution is explicit path-based relative to `schema_dir`; no basename search.
|
||||||
|
- Input artifacts are resolved by `artifact.Reader`, with `inline` and `file` as implemented types.
|
||||||
|
- Prompt required-input checks happen in the renderer.
|
||||||
|
|
||||||
|
Recommended centralization:
|
||||||
|
|
||||||
|
- Share only filesystem catalog mechanics between prompt/profile repositories.
|
||||||
|
- Keep prompt ID/version, profile ID, schema path, and input resolution policies in their current owning packages.
|
||||||
|
|
||||||
|
## Config And Command-Loading Review
|
||||||
|
|
||||||
|
Config precedence is consistent with policy:
|
||||||
|
|
||||||
|
1. built-in defaults;
|
||||||
|
2. config file values;
|
||||||
|
3. CLI overrides.
|
||||||
|
|
||||||
|
Intentional differences:
|
||||||
|
|
||||||
|
- `render` supports `--format`; `run` does not.
|
||||||
|
- `serve` supports `--addr`; `run`/`render` do not.
|
||||||
|
- `serve` rejects runtime model override flags.
|
||||||
|
- `render` does not expose `--schema-dir`, even though it can prepare `json_schema` prompts through config/default schema settings.
|
||||||
|
- CLI runtime model override flags are narrower than HTTP model override fields; this is currently documented by omission in CLI docs.
|
||||||
|
|
||||||
|
Likely cleanup areas:
|
||||||
|
|
||||||
|
- Common command settings finalization can be made smaller and less repetitive.
|
||||||
|
- Runner dependency construction should be a CLI-local helper.
|
||||||
|
- The `runConfig` flag-set booleans work, but adding more runtime override fields will continue to require touching several fields and the override condition.
|
||||||
|
|
||||||
|
## State, Manifest, Or Progress Handling Review
|
||||||
|
|
||||||
|
Scriptorium has no durable state, manifests, checkpoints, progress records, cache state, or resume behavior. This matches `docs/policy/architecture.md` and `docs/operations.md`.
|
||||||
|
|
||||||
|
There is no drift affecting resume, retry, force, dry-run, or audit behavior because those features do not exist. Validation repair hooks exist in the runner but are not wired by CLI/HTTP today; that boundary is documented.
|
||||||
|
|
||||||
|
## Refactors To Avoid
|
||||||
|
|
||||||
|
- Do not introduce a generic workflow/stage engine. Scriptorium intentionally executes one prompt request.
|
||||||
|
- Do not build a plugin architecture for prompt/profile/schema/artifact backends before another backend exists.
|
||||||
|
- Do not replace explicit CLI parse functions with a broad command framework.
|
||||||
|
- Do not merge CLI and HTTP adapters. Their public interfaces and failure semantics differ.
|
||||||
|
- Do not create a generic reflection-based mapper for domain/DTO/request structs.
|
||||||
|
- Do not centralize prompt/profile normalization into one generic YAML repository; their validation and error semantics are different.
|
||||||
|
- Do not add schema caching or a manifest/state store as part of deduplication.
|
||||||
|
- Do not document or implement unsupported artifact backends as cleanup.
|
||||||
|
|
||||||
|
## Recommended Implementation Sequence
|
||||||
|
|
||||||
|
1. Execution-target mapping cleanup
|
||||||
|
- Goal: reduce drift when adding profile/runtime/provider fields.
|
||||||
|
- Files to update: `internal/usecase`, `internal/adapter/http`, `internal/llm`, targeted tests.
|
||||||
|
- Acceptance criteria: all existing field mapping tests pass; a new all-fields test fails if a mapped execution field is omitted.
|
||||||
|
- Suggested validation: `go test ./internal/usecase ./internal/adapter/http ./internal/llm ./internal/format`.
|
||||||
|
- Small enough for one implementation prompt: yes.
|
||||||
|
|
||||||
|
2. Prompt/profile filesystem catalog helper
|
||||||
|
- Goal: centralize recursive YAML discovery, path sorting, extension filtering, and relative path formatting.
|
||||||
|
- Files to update: prompt/profile repositories plus new narrow helper package.
|
||||||
|
- Acceptance criteria: existing repository tests pass unchanged; new helper tests cover nested discovery and stable ordering.
|
||||||
|
- Suggested validation: `go test ./internal/promptdef ./internal/profile`.
|
||||||
|
- Small enough for one implementation prompt: yes.
|
||||||
|
|
||||||
|
3. CLI wiring and config finalization helper
|
||||||
|
- Goal: centralize common command settings resolution and runner construction.
|
||||||
|
- Files to update: `internal/adapter/cli/run.go` and CLI tests.
|
||||||
|
- Acceptance criteria: no flag behavior changes; run/render/serve config precedence tests pass.
|
||||||
|
- Suggested validation: `go test ./internal/adapter/cli`.
|
||||||
|
- Small enough for one implementation prompt: yes.
|
||||||
|
|
||||||
|
4. Stable use-case invalid-request reasons
|
||||||
|
- Goal: remove HTTP dependency on runner error message substrings.
|
||||||
|
- Files to update: `internal/usecase/runner.go`, `internal/adapter/http/handler.go`, tests.
|
||||||
|
- Acceptance criteria: HTTP error codes/messages remain unchanged; runner exposes stable `errors.Is` reason errors.
|
||||||
|
- Suggested validation: `go test ./internal/usecase ./internal/adapter/http`.
|
||||||
|
- Small enough for one implementation prompt: yes.
|
||||||
|
|
||||||
|
5. Test helper cleanup
|
||||||
|
- Goal: reduce repeated fixture construction after behavior-preserving refactors are complete.
|
||||||
|
- Files to update: primarily `internal/adapter/cli/run_test.go`, optionally `internal/usecase/runner_test.go`.
|
||||||
|
- Acceptance criteria: test intent remains clear; no cross-package helper package unless strongly justified.
|
||||||
|
- Suggested validation: `go test ./internal/adapter/cli ./internal/usecase`.
|
||||||
|
- Small enough for one implementation prompt: yes.
|
||||||
|
|
||||||
|
6. Dead-code/legacy sweep
|
||||||
|
- Goal: remove or clearly mark unsupported placeholders such as `ArtifactRefS3`.
|
||||||
|
- Files to update: domain/artifact tests/docs only if needed.
|
||||||
|
- Acceptance criteria: docs continue to describe only implemented behavior outside roadmap.
|
||||||
|
- Suggested validation: `go test ./internal/domain ./internal/artifact`.
|
||||||
|
- Small enough for one implementation prompt: yes.
|
||||||
|
|
||||||
|
## Test Strategy
|
||||||
|
|
||||||
|
Tests to add before or during cleanup:
|
||||||
|
|
||||||
|
- Execution target all-fields mapping tests:
|
||||||
|
- runner profile/default/request merge;
|
||||||
|
- HTTP request DTO to domain target;
|
||||||
|
- HTTP domain result to metadata DTO;
|
||||||
|
- LLM domain target to outbound JSON payload.
|
||||||
|
- Repository catalog helper tests:
|
||||||
|
- recursive scan;
|
||||||
|
- `.yaml` and `.yml` filtering;
|
||||||
|
- stable sorted paths;
|
||||||
|
- relative clean path formatting;
|
||||||
|
- context cancellation if the helper preserves current behavior.
|
||||||
|
- HTTP error mapping tests:
|
||||||
|
- profile-required and API-key-env-missing should rely on sentinel errors, not message text.
|
||||||
|
- CLI wiring regression tests:
|
||||||
|
- `run`, `render`, and `serve` preserve config precedence and required directory checks.
|
||||||
|
- Schema behavior tests:
|
||||||
|
- `Prepare` fails before LLM generation when `json_schema` structured-output schema cannot be loaded.
|
||||||
|
|
||||||
|
Validation commands for cleanup work:
|
||||||
|
|
||||||
|
- `go test ./internal/usecase ./internal/adapter/http ./internal/llm ./internal/format`
|
||||||
|
- `go test ./internal/promptdef ./internal/profile`
|
||||||
|
- `go test ./internal/adapter/cli`
|
||||||
|
- `go test ./...` before merging broader cleanup.
|
||||||
|
|
||||||
|
No automated docs or link checker is currently present.
|
||||||
|
|
||||||
|
## Appendix: Findings Not Worth Acting On
|
||||||
|
|
||||||
|
- Repeated `select { case <-ctx.Done(): ... }` checks are acceptable. They are local, simple, and appear at IO/loop boundaries where behavior is easy to read.
|
||||||
|
- CLI and HTTP request validation should remain separate. Their external contracts differ, and centralizing all request validation would blur adapter responsibilities.
|
||||||
|
- Prompt and profile validation should not be merged. They both use YAML and IDs, but their schemas, normalization rules, and error policies differ.
|
||||||
|
- Prepared-run text formatting is verbose but intentionally presentation-specific. Avoid abstracting it until another formatter needs the same layout policy.
|
||||||
|
- Hashing appears in artifact loading and runner metadata, but not all hashes represent the same thing. A tiny body-hash helper may be useful later; a generic hashing subsystem is not justified.
|
||||||
|
- HTTP DTOs duplicate domain field names by design. They should remain transport-owned so JSON compatibility can evolve deliberately.
|
||||||
|
- Config `applyConfig` and `ApplyCLIOverrides` look similar, but they apply different source labels and validation contexts. A broad merge abstraction would likely reduce clarity.
|
||||||
504
docs/roadmap/cleanup.md
Normal file
504
docs/roadmap/cleanup.md
Normal file
@@ -0,0 +1,504 @@
|
|||||||
|
# Cleanup Implementation Roadmap
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This roadmap turns the findings in [audit.md](audit.md) into a staged, decision-complete cleanup plan for Scriptorium.
|
||||||
|
|
||||||
|
Audience: LLM coding agents implementing the cleanup in order.
|
||||||
|
|
||||||
|
Controlling policies:
|
||||||
|
|
||||||
|
- [Documentation policy](../policy/documentation.md)
|
||||||
|
- [Architecture policy](../policy/architecture.md)
|
||||||
|
- [Development guide](../policy/development.md)
|
||||||
|
|
||||||
|
## Global Implementation Rules
|
||||||
|
|
||||||
|
- Implement stages in order.
|
||||||
|
- Keep public CLI flags, HTTP request/response shapes, config precedence, prompt/profile ID semantics, and validation behavior stable unless a stage explicitly says otherwise.
|
||||||
|
- Keep adapters thin and use-case policy in `internal/usecase`.
|
||||||
|
- Prefer explicit helpers over reflection, generic workflow abstractions, or broad framework-style rewrites.
|
||||||
|
- Update non-roadmap docs only when implemented behavior changes or when a stale implemented-behavior statement is found during a stage.
|
||||||
|
- Do not document future behavior outside `docs/roadmap/`.
|
||||||
|
- Run the stage-specific tests before moving to the next stage.
|
||||||
|
- Run `go test ./...` after the final stage.
|
||||||
|
|
||||||
|
## Stage 1: Execution Target Mapping Cleanup
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Reduce drift when adding or changing execution/runtime fields such as `service_tier`, `api_key_env`, `reasoning_effort`, or future provider request keys.
|
||||||
|
|
||||||
|
### Scope
|
||||||
|
|
||||||
|
Update only explicit execution-target mapping and serialization paths. Do not add new CLI flags or new provider features.
|
||||||
|
|
||||||
|
### Implementation
|
||||||
|
|
||||||
|
In `internal/usecase`:
|
||||||
|
|
||||||
|
- Keep execution-target merge policy in `internal/usecase`.
|
||||||
|
- Add focused helper coverage around `resolveExecutionTarget`, `mergeExecutionTarget`, and profile-to-target conversion.
|
||||||
|
- Keep the existing semantics:
|
||||||
|
- built-in defaults first;
|
||||||
|
- profile values override defaults;
|
||||||
|
- request overrides override profile values;
|
||||||
|
- zero numeric values do not override;
|
||||||
|
- empty/whitespace string values do not override;
|
||||||
|
- non-empty `ExtraParams` replaces the previous map with a copy.
|
||||||
|
|
||||||
|
In `internal/adapter/http`:
|
||||||
|
|
||||||
|
- Add local helper functions:
|
||||||
|
- `executionTargetFromModelOverrideDTO(*modelOverrideRequestDTO) *domain.ExecutionTarget`
|
||||||
|
- `modelParamsDTOFromExecutionTarget(domain.ExecutionTarget) modelParamsDTO`
|
||||||
|
- Use those helpers in `handler.go`.
|
||||||
|
- Keep DTO types unexported and transport-owned.
|
||||||
|
- Keep HTTP response field names and omission behavior unchanged.
|
||||||
|
|
||||||
|
In `internal/llm`:
|
||||||
|
|
||||||
|
- Add a local helper such as `openAIChatRequestFromGenerateRequest(req domain.GenerateRequest, defaultModel string) (openAIChatRequest, error)` or an equivalent small function.
|
||||||
|
- Keep endpoint construction, HTTP client timeout handling, API-key environment lookup, and response parsing in `Generate`.
|
||||||
|
- Keep outbound serialization behavior unchanged:
|
||||||
|
- send `model` and `messages`;
|
||||||
|
- send `temperature`, `max_tokens`, `top_p`, and `service_tier` only when currently sent;
|
||||||
|
- send `response_format` only when structured output is present;
|
||||||
|
- do not serialize `reasoning_effort` or `extra_params`.
|
||||||
|
|
||||||
|
Do not:
|
||||||
|
|
||||||
|
- use reflection to copy fields;
|
||||||
|
- move HTTP DTOs into `internal/domain`;
|
||||||
|
- add generic mapper packages;
|
||||||
|
- change prepared-run JSON tags.
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
|
||||||
|
Add or update tests so an all-fields `domain.ExecutionTarget` catches omissions.
|
||||||
|
|
||||||
|
Required tests:
|
||||||
|
|
||||||
|
- Runner merge/profile conversion:
|
||||||
|
- profile values populate all supported execution fields;
|
||||||
|
- runtime overrides beat profile values for all overrideable fields;
|
||||||
|
- empty string overrides do not erase profile values;
|
||||||
|
- empty `ExtraParams` does not erase profile values.
|
||||||
|
- HTTP adapter:
|
||||||
|
- request `model` object maps every supported field into `RunRequest.Execution`;
|
||||||
|
- response `metadata.model_params` includes every supported field according to current DTO tags.
|
||||||
|
- LLM client:
|
||||||
|
- outbound JSON includes every serialized execution field;
|
||||||
|
- outbound JSON omits `service_tier` when empty;
|
||||||
|
- outbound JSON still omits `reasoning_effort` and `extra_params`.
|
||||||
|
|
||||||
|
### Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/usecase ./internal/adapter/http ./internal/llm ./internal/format
|
||||||
|
```
|
||||||
|
|
||||||
|
### Acceptance Criteria
|
||||||
|
|
||||||
|
- No public behavior changes.
|
||||||
|
- Adding a new execution target field later has obvious mapping/test locations.
|
||||||
|
- Existing HTTP and LLM behavior remains stable.
|
||||||
|
- Stage is small enough for one implementation prompt.
|
||||||
|
|
||||||
|
## Stage 2: Prompt/Profile Filesystem Catalog Helper
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Centralize shared recursive YAML discovery mechanics while preserving prompt/profile-specific validation and error behavior.
|
||||||
|
|
||||||
|
### Scope
|
||||||
|
|
||||||
|
Create a narrow helper package for filesystem catalog mechanics only.
|
||||||
|
|
||||||
|
Recommended package:
|
||||||
|
|
||||||
|
- `internal/filecatalog`
|
||||||
|
|
||||||
|
### Implementation
|
||||||
|
|
||||||
|
Add helper functions with explicit, small responsibilities:
|
||||||
|
|
||||||
|
- recursively find YAML files under a root directory;
|
||||||
|
- honor context cancellation during walking;
|
||||||
|
- accept `.yaml` and `.yml`;
|
||||||
|
- return stable sorted full paths;
|
||||||
|
- compute clean relative paths from a root;
|
||||||
|
- return filename stems with `.yaml`/`.yml` stripped.
|
||||||
|
|
||||||
|
Use the helper in:
|
||||||
|
|
||||||
|
- `internal/promptdef/filesystem_repository.go`
|
||||||
|
- `internal/profile/filesystem_repository.go`
|
||||||
|
|
||||||
|
Preserve existing behavior:
|
||||||
|
|
||||||
|
- prompt/profile lookup uses YAML `id`, not file path;
|
||||||
|
- subdirectories are organizational only;
|
||||||
|
- duplicate prompt/profile IDs are invalid;
|
||||||
|
- malformed likely-target files still surface errors;
|
||||||
|
- relative nested paths still appear in errors;
|
||||||
|
- prompt `content_file` resolution remains relative to the prompt YAML file;
|
||||||
|
- prompt/profile strict YAML and validation stay in their existing packages.
|
||||||
|
|
||||||
|
Do not:
|
||||||
|
|
||||||
|
- create a generic repository framework;
|
||||||
|
- merge prompt and profile normalization;
|
||||||
|
- move prompt/profile domain policy into the helper;
|
||||||
|
- change error messages except for unavoidable wording caused by helper extraction.
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
|
||||||
|
Add tests for `internal/filecatalog`:
|
||||||
|
|
||||||
|
- nested YAML discovery;
|
||||||
|
- `.yaml` and `.yml` accepted;
|
||||||
|
- non-YAML files ignored;
|
||||||
|
- returned paths sorted deterministically;
|
||||||
|
- relative path formatting works for nested files;
|
||||||
|
- filename stem stripping handles both extensions.
|
||||||
|
|
||||||
|
Keep existing prompt/profile repository tests passing.
|
||||||
|
|
||||||
|
### Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/filecatalog ./internal/promptdef ./internal/profile
|
||||||
|
```
|
||||||
|
|
||||||
|
### Acceptance Criteria
|
||||||
|
|
||||||
|
- Prompt/profile repository tests pass without behavior expectation changes.
|
||||||
|
- Shared filesystem scanning logic exists in one place.
|
||||||
|
- Prompt/profile packages still own their own validation and normalization.
|
||||||
|
- Stage is small enough for one implementation prompt.
|
||||||
|
|
||||||
|
## Stage 3: CLI Wiring And Settings Finalization Cleanup
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Reduce duplicated command setup while preserving each command's public flag surface and behavior.
|
||||||
|
|
||||||
|
### Scope
|
||||||
|
|
||||||
|
Clean up `internal/adapter/cli` only, except for tests.
|
||||||
|
|
||||||
|
### Implementation
|
||||||
|
|
||||||
|
Add CLI-local helpers. Recommended helpers:
|
||||||
|
|
||||||
|
- `commonCommandSettings` or similar struct containing resolved `promptDir`, `profileDir`, `schemaDir`, `serverAddr`, and `defaultRenderFormat` where applicable.
|
||||||
|
- `resolveCommonSettings(fs *flag.FlagSet, configPath string, overrides appconfig.CLIOverrides) (commonCommandSettings, error)`.
|
||||||
|
- `validateRequiredLibraryDirs(promptDir, profileDir string) error`.
|
||||||
|
- `newRunner(promptDir, profileDir, schemaDir string, llmClient llm.Client) *usecase.Runner`.
|
||||||
|
- Optionally `newOpenAIClient() (*llm.OpenAICompatibleClient, error)` if it removes exact duplication without obscuring command behavior.
|
||||||
|
|
||||||
|
Preserve command differences:
|
||||||
|
|
||||||
|
- `run` exposes runtime model override flags and `--schema-dir`;
|
||||||
|
- `render` exposes runtime model override flags and `--format`, but not `--schema-dir`;
|
||||||
|
- `serve` exposes `--addr` and `--schema-dir`, but no runtime model override flags;
|
||||||
|
- `render` default format comes from `defaults.render_format` unless `--format` is set;
|
||||||
|
- deprecated `--prompt-id` and `--profile-id` aliases remain accepted.
|
||||||
|
|
||||||
|
Keep existing parse functions:
|
||||||
|
|
||||||
|
- `parseRunArgs`
|
||||||
|
- `parseRenderArgs`
|
||||||
|
- `parseServeArgs`
|
||||||
|
|
||||||
|
Do not:
|
||||||
|
|
||||||
|
- replace the standard library `flag` package;
|
||||||
|
- introduce a command framework;
|
||||||
|
- make `serve` accept runtime model override flags;
|
||||||
|
- change error prefixes such as `run parse error`, `render parse error`, or `serve parse error`;
|
||||||
|
- change CLI output behavior.
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
|
||||||
|
Required regression tests:
|
||||||
|
|
||||||
|
- `run`, `render`, and `serve` still apply config precedence correctly.
|
||||||
|
- Missing effective `prompt_dir` and `profile_dir` still return the same guidance.
|
||||||
|
- `render` still uses config default render format and explicit `--format` override.
|
||||||
|
- `serve` still rejects runtime model override flags.
|
||||||
|
- `run` and `render` still build equivalent runtime override requests for shared flags.
|
||||||
|
|
||||||
|
### Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/adapter/cli
|
||||||
|
```
|
||||||
|
|
||||||
|
### Acceptance Criteria
|
||||||
|
|
||||||
|
- No CLI flag, output, exit-code, or precedence changes.
|
||||||
|
- Runner dependency construction is centralized inside the CLI adapter.
|
||||||
|
- Command-specific behavior remains easy to read.
|
||||||
|
- Stage is small enough for one implementation prompt.
|
||||||
|
|
||||||
|
## Stage 4: Stable Use-Case Error Reasons For HTTP Mapping
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Remove HTTP error mapping's dependency on runner error message substrings.
|
||||||
|
|
||||||
|
### Scope
|
||||||
|
|
||||||
|
Change error identity, not public HTTP error responses.
|
||||||
|
|
||||||
|
### Implementation
|
||||||
|
|
||||||
|
In `internal/usecase`:
|
||||||
|
|
||||||
|
- Add stable sentinel errors for invalid-request reasons that HTTP currently distinguishes by message text.
|
||||||
|
- Required sentinels:
|
||||||
|
- missing profile selection, for the case where neither request profile nor prompt `default_profile` is available;
|
||||||
|
- missing API-key environment value, for the case where `api_key_env` is set but the named environment variable is unset or empty.
|
||||||
|
- Wrap these sentinels with `ErrInvalidRequest` so existing broad invalid-request checks keep working.
|
||||||
|
- Preserve clear human-readable runner errors.
|
||||||
|
|
||||||
|
In `internal/adapter/http`:
|
||||||
|
|
||||||
|
- Replace `strings.Contains(err.Error(), ...)` checks for these cases with `errors.Is`.
|
||||||
|
- Keep current HTTP status codes, error codes, and response messages:
|
||||||
|
- `400 profile_required`;
|
||||||
|
- `400 api_key_env_missing`.
|
||||||
|
|
||||||
|
Do not:
|
||||||
|
|
||||||
|
- expose HTTP-specific error codes from `internal/usecase`;
|
||||||
|
- change the HTTP JSON error body shape;
|
||||||
|
- remove broad fallback handling for `usecase.ErrInvalidRequest`.
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
|
||||||
|
Required tests:
|
||||||
|
|
||||||
|
- Runner tests assert `errors.Is(err, usecase.ErrProfileRequired)` or the chosen sentinel name for missing profile selection.
|
||||||
|
- Runner tests assert `errors.Is(err, usecase.ErrAPIKeyEnvMissing)` or the chosen sentinel name for missing API-key environment value.
|
||||||
|
- HTTP handler tests still assert unchanged status/code/message for both cases.
|
||||||
|
- HTTP handler tests should not construct errors by relying on exact runner prose for these two cases.
|
||||||
|
|
||||||
|
### Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/usecase ./internal/adapter/http
|
||||||
|
```
|
||||||
|
|
||||||
|
### Acceptance Criteria
|
||||||
|
|
||||||
|
- HTTP mapping no longer depends on runner message substrings for the two distinguished invalid-request cases.
|
||||||
|
- Public HTTP behavior is unchanged.
|
||||||
|
- Runner errors remain clear in CLI output.
|
||||||
|
- Stage is small enough for one implementation prompt.
|
||||||
|
|
||||||
|
## Stage 5: Schema Failure Regression Coverage
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Protect the current structured-output invariant before future schema cleanup: `Prepare` must load a `json_schema` document before any LLM call.
|
||||||
|
|
||||||
|
### Scope
|
||||||
|
|
||||||
|
Add regression coverage only. Do not add schema caching or change schema loading architecture in this stage.
|
||||||
|
|
||||||
|
### Implementation
|
||||||
|
|
||||||
|
In `internal/usecase/runner_test.go` or an appropriate package test:
|
||||||
|
|
||||||
|
- Add a test where a prompt uses `validation_mode: json_schema` with a missing or failing schema document.
|
||||||
|
- Assert `Runner.Prepare` fails with `ErrValidation`.
|
||||||
|
- Assert no LLM call is made for `Runner.Run` when structured-output schema loading fails.
|
||||||
|
|
||||||
|
If existing tests already cover part of this behavior, consolidate assertions without making the test suite harder to read.
|
||||||
|
|
||||||
|
Do not:
|
||||||
|
|
||||||
|
- cache compiled schemas;
|
||||||
|
- change `validate.StandardValidator` behavior;
|
||||||
|
- introduce a schema service abstraction.
|
||||||
|
|
||||||
|
### Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/usecase ./internal/validate
|
||||||
|
```
|
||||||
|
|
||||||
|
### Acceptance Criteria
|
||||||
|
|
||||||
|
- Missing structured-output schema fails before generation.
|
||||||
|
- Existing JSON Schema validation behavior is unchanged.
|
||||||
|
- Stage is small enough for one implementation prompt.
|
||||||
|
|
||||||
|
## Stage 6: Test Fixture Cleanup
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Reduce repeated test setup after behavior-preserving production refactors are complete.
|
||||||
|
|
||||||
|
### Scope
|
||||||
|
|
||||||
|
Prefer package-local test helpers. Avoid cross-package test utility packages unless a helper is needed by more than two packages and represents a stable public fixture contract.
|
||||||
|
|
||||||
|
### Implementation
|
||||||
|
|
||||||
|
In `internal/adapter/cli/run_test.go`:
|
||||||
|
|
||||||
|
- Consolidate repeated temp prompt/profile/input setup into local helper functions.
|
||||||
|
- Keep helper names behavior-focused, for example:
|
||||||
|
- `newCLITestLibrary`
|
||||||
|
- `writePromptFileWithDefaultProfile`
|
||||||
|
- `writeProfileFile`
|
||||||
|
- `runCLICommand`
|
||||||
|
- Do not hide assertions inside helpers unless the assertion is truly setup validation.
|
||||||
|
|
||||||
|
In `internal/usecase/runner_test.go`:
|
||||||
|
|
||||||
|
- Keep existing fake interfaces package-local.
|
||||||
|
- Remove only high-volume duplication that obscures test intent.
|
||||||
|
|
||||||
|
Do not:
|
||||||
|
|
||||||
|
- move package-private fake types into production code;
|
||||||
|
- create a broad `internal/testutil` package unless a later cleanup stage proves it necessary;
|
||||||
|
- rewrite tests into table-driven form when cases have meaningfully different setup.
|
||||||
|
|
||||||
|
### Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/adapter/cli ./internal/usecase
|
||||||
|
```
|
||||||
|
|
||||||
|
### Acceptance Criteria
|
||||||
|
|
||||||
|
- Test intent is at least as clear as before.
|
||||||
|
- No production behavior changes.
|
||||||
|
- Test fixture setup has less repeated boilerplate in CLI tests.
|
||||||
|
- Stage is small enough for one implementation prompt.
|
||||||
|
|
||||||
|
## Stage 7: Unsupported Placeholder Sweep
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Remove code that suggests unimplemented artifact behavior outside roadmap documentation.
|
||||||
|
|
||||||
|
### Scope
|
||||||
|
|
||||||
|
Remove unsupported placeholders only when they are not needed by current tests or public docs.
|
||||||
|
|
||||||
|
### Implementation
|
||||||
|
|
||||||
|
Remove `domain.ArtifactRefS3` from `internal/domain/domain.go` unless new evidence shows it is intentionally needed by implemented code.
|
||||||
|
|
||||||
|
Preserve current behavior:
|
||||||
|
|
||||||
|
- supported artifact reference types remain `inline` and `file`;
|
||||||
|
- unsupported artifact reference types still return `artifact.ErrUnsupportedRefType`;
|
||||||
|
- docs continue to describe only `inline` and `file` outside roadmap files.
|
||||||
|
|
||||||
|
Update tests only if they reference the removed constant. Prefer testing unsupported artifact behavior with a literal custom type such as `domain.ArtifactRefType("s3")` or `domain.ArtifactRefType("unsupported")`.
|
||||||
|
|
||||||
|
Do not:
|
||||||
|
|
||||||
|
- add S3 support;
|
||||||
|
- document S3 as implemented;
|
||||||
|
- add future-backend placeholders elsewhere.
|
||||||
|
|
||||||
|
### Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/domain ./internal/artifact
|
||||||
|
```
|
||||||
|
|
||||||
|
### Acceptance Criteria
|
||||||
|
|
||||||
|
- Unsupported placeholder constant is removed.
|
||||||
|
- Unsupported artifact-type behavior remains covered.
|
||||||
|
- No non-roadmap doc claims unimplemented artifact support.
|
||||||
|
- Stage is small enough for one implementation prompt.
|
||||||
|
|
||||||
|
## Stage 8: Final Verification And Documentation Alignment
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Confirm the cleanup sequence preserved behavior and documentation accuracy.
|
||||||
|
|
||||||
|
### Implementation
|
||||||
|
|
||||||
|
Run the full test suite:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
Run the maintained render smoke command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml --format json
|
||||||
|
```
|
||||||
|
|
||||||
|
Search for stale or unsupported terms:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -n "ArtifactRefS3|s3|strings\\.Contains\\(err\\.Error\\(\\)|TODO|future|planned" internal docs README.md examples
|
||||||
|
```
|
||||||
|
|
||||||
|
Review results manually:
|
||||||
|
|
||||||
|
- `s3` should not appear as an implemented artifact type.
|
||||||
|
- `strings.Contains(err.Error())` should not be used for stable use-case reason mapping.
|
||||||
|
- Any `future` or `planned` wording outside `docs/roadmap/` must describe current boundaries, not aspirational behavior.
|
||||||
|
|
||||||
|
Update docs only if cleanup changed implemented behavior or if the search reveals stale implemented-behavior docs.
|
||||||
|
|
||||||
|
### Acceptance Criteria
|
||||||
|
|
||||||
|
- Full test suite passes.
|
||||||
|
- Maintained render smoke command succeeds.
|
||||||
|
- No stale unsupported artifact placeholder remains.
|
||||||
|
- Non-roadmap docs describe implemented behavior only.
|
||||||
|
- Working tree contains only intentional cleanup changes.
|
||||||
|
|
||||||
|
## Deferred Work
|
||||||
|
|
||||||
|
Do not implement these during the staged cleanup unless a later audit makes them high-confidence:
|
||||||
|
|
||||||
|
- schema caching or a combined raw/compiled schema service;
|
||||||
|
- artifact hash/build helper beyond a small helper introduced opportunistically during touched code;
|
||||||
|
- generic YAML repository framework;
|
||||||
|
- generic CLI command framework;
|
||||||
|
- plugin architecture for future prompt/profile/schema/artifact backends;
|
||||||
|
- durable state, manifests, checkpoints, or resume behavior.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
The cleanup roadmap is complete when all stages have been implemented in order, the final verification passes, and the resulting code still satisfies:
|
||||||
|
|
||||||
|
- `Runner.Run` reuses `Runner.Prepare`;
|
||||||
|
- CLI and HTTP adapters instantiate `Runner` without a repairer;
|
||||||
|
- unknown config/prompt/profile YAML and HTTP JSON fields are rejected;
|
||||||
|
- raw API key values are not accepted or emitted;
|
||||||
|
- prompt/profile subdirectories remain organizational only;
|
||||||
|
- schema paths remain explicit and relative to `schema_dir` when not absolute;
|
||||||
|
- public CLI and HTTP behavior remains stable.
|
||||||
334
docs/troubleshooting.md
Normal file
334
docs/troubleshooting.md
Normal file
@@ -0,0 +1,334 @@
|
|||||||
|
# Troubleshooting
|
||||||
|
|
||||||
|
This guide lists recurring implemented failure modes and safe fixes.
|
||||||
|
|
||||||
|
For command syntax, see [CLI reference](cli.md). For configuration and file formats, see [Configuration reference](config.md). For operational behavior, see [Operations guide](operations.md).
|
||||||
|
|
||||||
|
## Missing Or Invalid Config File
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI errors such as `application config error: config file not found` or `invalid config YAML`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- `--config` points to a missing file.
|
||||||
|
- Config YAML has syntax errors or unknown fields.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium render --config /path/to/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Correct file path.
|
||||||
|
- Remove unknown fields.
|
||||||
|
- Fix YAML syntax.
|
||||||
|
- Keep secrets out of config.
|
||||||
|
|
||||||
|
Relevant links:
|
||||||
|
|
||||||
|
- [Configuration reference](config.md)
|
||||||
|
- [CLI reference](cli.md)
|
||||||
|
|
||||||
|
## Missing Prompt/Profile Directory Settings
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI parse errors saying prompt directory or profile directory is required.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Neither CLI flags nor config provide effective `prompt_dir` / `profile_dir`.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Run the failing command with explicit `--prompt-dir` and `--profile-dir` once to verify.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Set `prompt_dir` and `profile_dir` in config, or always pass both flags.
|
||||||
|
|
||||||
|
Relevant links:
|
||||||
|
|
||||||
|
- [Configuration reference](config.md)
|
||||||
|
- [CLI reference](cli.md)
|
||||||
|
|
||||||
|
## Unknown Or Unsupported Flags
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI parse error for an unknown flag.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Typo or command mismatch (for example, `serve` with runtime model override flags).
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Compare command against the command-specific flag list.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Remove unsupported flags.
|
||||||
|
- Use `run`/`render` for runtime model overrides.
|
||||||
|
|
||||||
|
Relevant links:
|
||||||
|
|
||||||
|
- [CLI reference](cli.md)
|
||||||
|
|
||||||
|
## Prompt Definition Load Failures
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI run/render error from prompt loading.
|
||||||
|
- HTTP `404 prompt_not_found` or `400 prompt_load_failed`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Prompt ID not found.
|
||||||
|
- Invalid prompt YAML.
|
||||||
|
- Invalid prompt contract (for example bad validation mode, message content/content_file rule violation, missing schema path for `json_schema`).
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium render --config ./examples/config.yml --prompt <prompt-id> --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml --format json
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Correct prompt ID.
|
||||||
|
- Fix prompt YAML and contract fields.
|
||||||
|
- Ensure referenced `content_file` paths exist.
|
||||||
|
|
||||||
|
Relevant links:
|
||||||
|
|
||||||
|
- [Configuration reference](config.md)
|
||||||
|
- [CLI reference](cli.md)
|
||||||
|
|
||||||
|
## Profile Definition Load Failures
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI run/render error from profile loading.
|
||||||
|
- HTTP `404 profile_not_found` or `400 profile_load_failed`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Profile ID missing/not found.
|
||||||
|
- Invalid profile YAML.
|
||||||
|
- Invalid profile values.
|
||||||
|
- Raw `api_key` field present (rejected).
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --profile <profile-id> --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Correct profile ID.
|
||||||
|
- Fix profile YAML and value ranges.
|
||||||
|
- Replace `api_key` with `api_key_env`.
|
||||||
|
|
||||||
|
Relevant links:
|
||||||
|
|
||||||
|
- [Configuration reference](config.md)
|
||||||
|
- [CLI reference](cli.md)
|
||||||
|
|
||||||
|
## Input Artifact Read Failures
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI run/render error reading input artifacts.
|
||||||
|
- HTTP `400 artifact_read_failed`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- File path in input mapping does not exist or is unreadable.
|
||||||
|
- Unsupported artifact reference type in HTTP request.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Verify every mapped file path exists and is readable by the process.
|
||||||
|
- For HTTP, verify each input uses supported `type` values.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Correct file paths and permissions.
|
||||||
|
- Use supported input types (`file`, `inline`).
|
||||||
|
|
||||||
|
Relevant links:
|
||||||
|
|
||||||
|
- [CLI reference](cli.md)
|
||||||
|
- [Configuration reference](config.md)
|
||||||
|
|
||||||
|
## Prompt Template Render Failures
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI run/render error from prompt rendering.
|
||||||
|
- HTTP `400 prompt_render_failed`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Template references missing input names.
|
||||||
|
- Template syntax or data reference issues.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Run `render --format json` with the same prompt, inputs, vars, and profile selection.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Align template `{{input "name"}}` references with actual input mappings.
|
||||||
|
- Fix template syntax and variable names.
|
||||||
|
|
||||||
|
Relevant links:
|
||||||
|
|
||||||
|
- [CLI reference](cli.md)
|
||||||
|
- [Configuration reference](config.md)
|
||||||
|
|
||||||
|
## Missing API-Key Environment Variable
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI run/render invalid request error about missing API-key environment variable.
|
||||||
|
- HTTP `400 api_key_env_missing`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Selected profile or override sets `api_key_env`, but that environment variable is unset/empty.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
printenv SCRIPTORIUM_API_KEY
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Set the required environment variable before invoking CLI/service.
|
||||||
|
- Or use a profile that does not require API key auth for the target endpoint.
|
||||||
|
|
||||||
|
Relevant links:
|
||||||
|
|
||||||
|
- [Configuration reference](config.md)
|
||||||
|
- [Operations guide](operations.md)
|
||||||
|
|
||||||
|
## LLM Request Failures
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI `run` fails with LLM generation errors.
|
||||||
|
- HTTP returns `502 llm_failed`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Endpoint unreachable.
|
||||||
|
- Non-2xx response from provider.
|
||||||
|
- Timeout.
|
||||||
|
- Malformed provider response.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Confirm endpoint URL and model in selected profile/overrides.
|
||||||
|
- Retry with `render` first to confirm pre-LLM preparation works.
|
||||||
|
- Check provider/network logs for non-2xx responses and timeouts.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Correct endpoint/model settings.
|
||||||
|
- Adjust timeout if needed.
|
||||||
|
- Resolve provider-side or network issues.
|
||||||
|
|
||||||
|
Relevant links:
|
||||||
|
|
||||||
|
- [CLI reference](cli.md)
|
||||||
|
- [Configuration reference](config.md)
|
||||||
|
- [Operations guide](operations.md)
|
||||||
|
|
||||||
|
## Validation Status Failed (`run` Exit 2 Or HTTP 200 With Failed Status)
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI exits with code `2`.
|
||||||
|
- HTTP returns `200`, but `validation.status` is `failed`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Generated output failed `basic`, `json`, or `json_schema` content validation.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Inspect validation mode and validation errors in CLI summary/HTTP response.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Refine prompt constraints.
|
||||||
|
- Tighten schema or adjust model/profile settings.
|
||||||
|
- Rerun after correction.
|
||||||
|
|
||||||
|
Relevant links:
|
||||||
|
|
||||||
|
- [Configuration reference](config.md)
|
||||||
|
- [Operations guide](operations.md)
|
||||||
|
|
||||||
|
## Validation Runtime Failure
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- CLI `run` fails with validation runtime error.
|
||||||
|
- HTTP returns `500 validation_runtime_failed`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- `json_schema` schema file missing/inaccessible.
|
||||||
|
- Invalid schema JSON document.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Verify `schema_dir` and `output.schema_path` resolution.
|
||||||
|
- Check schema file readability and valid JSON syntax.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Correct schema path.
|
||||||
|
- Fix schema JSON content.
|
||||||
|
- Rerun.
|
||||||
|
|
||||||
|
Relevant links:
|
||||||
|
|
||||||
|
- [Configuration reference](config.md)
|
||||||
|
- [Operations guide](operations.md)
|
||||||
|
|
||||||
|
## HTTP Request Parsing/Contract Errors
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- HTTP `400 invalid_json` or `400 invalid_request`.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Malformed JSON body.
|
||||||
|
- Unknown JSON fields.
|
||||||
|
- Missing required `prompt_id` or `inputs`.
|
||||||
|
|
||||||
|
Diagnostic step:
|
||||||
|
|
||||||
|
- Revalidate request JSON.
|
||||||
|
- Confirm required request fields are present.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- Send valid JSON with only supported fields.
|
||||||
|
- Ensure `prompt_id` and at least one input mapping are included.
|
||||||
|
|
||||||
|
Relevant links:
|
||||||
|
|
||||||
|
- [Operations guide](operations.md)
|
||||||
|
- [CLI reference](cli.md)
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
prompt_dir: ./prompts
|
prompt_dir: ./examples/prompts
|
||||||
profile_dir: ./profiles
|
profile_dir: ./examples/profiles
|
||||||
schema_dir: ./schemas
|
schema_dir: ./examples/schemas
|
||||||
|
|
||||||
server:
|
server:
|
||||||
addr: :8080
|
addr: :8080
|
||||||
|
|||||||
18
examples/http-run.json
Normal file
18
examples/http-run.json
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
{
|
||||||
|
"prompt_id": "generic.markdown_summary",
|
||||||
|
"profile_id": "local-fast",
|
||||||
|
"inputs": {
|
||||||
|
"transcript": {
|
||||||
|
"type": "file",
|
||||||
|
"uri": "./examples/fixtures/transcript.md"
|
||||||
|
},
|
||||||
|
"glossary": {
|
||||||
|
"type": "file",
|
||||||
|
"uri": "./examples/fixtures/glossary.yml"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"vars": {
|
||||||
|
"session_date": "2026-05-04"
|
||||||
|
},
|
||||||
|
"include_raw_output": false
|
||||||
|
}
|
||||||
13
examples/render-markdown-summary.sh
Executable file
13
examples/render-markdown-summary.sh
Executable file
@@ -0,0 +1,13 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
|
||||||
|
cd "$repo_root"
|
||||||
|
|
||||||
|
go run ./cmd/scriptorium render \
|
||||||
|
--config ./examples/config.yml \
|
||||||
|
--prompt generic.markdown_summary \
|
||||||
|
--input transcript=./examples/fixtures/transcript.md \
|
||||||
|
--input glossary=./examples/fixtures/glossary.yml \
|
||||||
|
--format text
|
||||||
@@ -81,6 +81,14 @@ type serveConfig struct {
|
|||||||
schemaDir string
|
schemaDir string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type commonCommandSettings struct {
|
||||||
|
promptDir string
|
||||||
|
profileDir string
|
||||||
|
schemaDir string
|
||||||
|
serverAddr string
|
||||||
|
defaultRenderFormat renderformat.PreparedRunOutputFormat
|
||||||
|
}
|
||||||
|
|
||||||
type listFlag []string
|
type listFlag []string
|
||||||
|
|
||||||
func (l *listFlag) String() string {
|
func (l *listFlag) String() string {
|
||||||
@@ -125,22 +133,13 @@ func runCommand(args []string, stdout, stderr io.Writer) int {
|
|||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
}
|
}
|
||||||
|
|
||||||
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
|
llmClient, err := newOpenAIClient()
|
||||||
Timeout: defaults.LLMRequestTimeoutDefault,
|
|
||||||
})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintf(stderr, "llm client error: %v\n", err)
|
fmt.Fprintf(stderr, "llm client error: %v\n", err)
|
||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
}
|
}
|
||||||
|
|
||||||
runner := usecase.NewRunner(
|
runner := newRunner(cfg.promptDir, cfg.profileDir, cfg.schemaDir, llmClient)
|
||||||
promptdef.NewFilesystemRepository(cfg.promptDir),
|
|
||||||
profile.NewFilesystemRepository(cfg.profileDir),
|
|
||||||
artifactadapter.NewCompositeReader(),
|
|
||||||
prompt.NewGoRenderer(),
|
|
||||||
llmClient,
|
|
||||||
validate.NewStandardValidator(cfg.schemaDir),
|
|
||||||
)
|
|
||||||
|
|
||||||
res, runErr := runner.Run(context.Background(), req)
|
res, runErr := runner.Run(context.Background(), req)
|
||||||
if runErr != nil {
|
if runErr != nil {
|
||||||
@@ -170,14 +169,7 @@ func renderCommand(args []string, stdout, stderr io.Writer) int {
|
|||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
}
|
}
|
||||||
|
|
||||||
runner := usecase.NewRunner(
|
runner := newRunner(cfg.promptDir, cfg.profileDir, cfg.schemaDir, nil)
|
||||||
promptdef.NewFilesystemRepository(cfg.promptDir),
|
|
||||||
profile.NewFilesystemRepository(cfg.profileDir),
|
|
||||||
artifactadapter.NewCompositeReader(),
|
|
||||||
prompt.NewGoRenderer(),
|
|
||||||
nil,
|
|
||||||
validate.NewStandardValidator(cfg.schemaDir),
|
|
||||||
)
|
|
||||||
|
|
||||||
prepared, prepErr := runner.Prepare(context.Background(), req)
|
prepared, prepErr := runner.Prepare(context.Background(), req)
|
||||||
if prepErr != nil {
|
if prepErr != nil {
|
||||||
@@ -205,22 +197,13 @@ func serveCommand(args []string, stderr io.Writer) int {
|
|||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
}
|
}
|
||||||
|
|
||||||
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
|
llmClient, err := newOpenAIClient()
|
||||||
Timeout: defaults.LLMRequestTimeoutDefault,
|
|
||||||
})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintf(stderr, "llm client error: %v\n", err)
|
fmt.Fprintf(stderr, "llm client error: %v\n", err)
|
||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
}
|
}
|
||||||
|
|
||||||
runner := usecase.NewRunner(
|
runner := newRunner(cfg.promptDir, cfg.profileDir, cfg.schemaDir, llmClient)
|
||||||
promptdef.NewFilesystemRepository(cfg.promptDir),
|
|
||||||
profile.NewFilesystemRepository(cfg.profileDir),
|
|
||||||
artifactadapter.NewCompositeReader(),
|
|
||||||
prompt.NewGoRenderer(),
|
|
||||||
llmClient,
|
|
||||||
validate.NewStandardValidator(cfg.schemaDir),
|
|
||||||
)
|
|
||||||
|
|
||||||
h := httpadapter.NewHandler(runner)
|
h := httpadapter.NewHandler(runner)
|
||||||
srv := &http.Server{
|
srv := &http.Server{
|
||||||
@@ -307,7 +290,7 @@ func parseServeArgs(args []string) (*serveConfig, error) {
|
|||||||
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
|
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
|
||||||
}
|
}
|
||||||
|
|
||||||
settings, err := resolveAppSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
settings, err := resolveCommonSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
||||||
PromptDir: cfg.promptDirIfSet(fs),
|
PromptDir: cfg.promptDirIfSet(fs),
|
||||||
ProfileDir: cfg.profileDirIfSet(fs),
|
ProfileDir: cfg.profileDirIfSet(fs),
|
||||||
SchemaDir: cfg.schemaDirIfSet(fs),
|
SchemaDir: cfg.schemaDirIfSet(fs),
|
||||||
@@ -317,16 +300,13 @@ func parseServeArgs(args []string) (*serveConfig, error) {
|
|||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
cfg.promptDir = settings.PromptDir
|
cfg.promptDir = settings.promptDir
|
||||||
cfg.profileDir = settings.ProfileDir
|
cfg.profileDir = settings.profileDir
|
||||||
cfg.schemaDir = settings.SchemaDir
|
cfg.schemaDir = settings.schemaDir
|
||||||
cfg.addr = settings.ServerAddr
|
cfg.addr = settings.serverAddr
|
||||||
|
|
||||||
if strings.TrimSpace(cfg.promptDir) == "" {
|
if err := validateRequiredLibraryDirs(cfg.promptDir, cfg.profileDir); err != nil {
|
||||||
return nil, errors.New(errPromptDirRequired)
|
return nil, err
|
||||||
}
|
|
||||||
if strings.TrimSpace(cfg.profileDir) == "" {
|
|
||||||
return nil, errors.New(errProfileDirRequired)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
cfg.promptDir = filepath.Clean(cfg.promptDir)
|
cfg.promptDir = filepath.Clean(cfg.promptDir)
|
||||||
@@ -359,7 +339,7 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
|
|||||||
return fmt.Errorf("unexpected positional args: %v", fs.Args())
|
return fmt.Errorf("unexpected positional args: %v", fs.Args())
|
||||||
}
|
}
|
||||||
|
|
||||||
settings, err := resolveAppSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
settings, err := resolveCommonSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
||||||
PromptDir: cfg.promptDirIfSet(fs),
|
PromptDir: cfg.promptDirIfSet(fs),
|
||||||
ProfileDir: cfg.profileDirIfSet(fs),
|
ProfileDir: cfg.profileDirIfSet(fs),
|
||||||
SchemaDir: cfg.schemaDirIfSet(fs),
|
SchemaDir: cfg.schemaDirIfSet(fs),
|
||||||
@@ -368,16 +348,13 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
cfg.promptDir = settings.PromptDir
|
cfg.promptDir = settings.promptDir
|
||||||
cfg.profileDir = settings.ProfileDir
|
cfg.profileDir = settings.profileDir
|
||||||
cfg.schemaDir = settings.SchemaDir
|
cfg.schemaDir = settings.schemaDir
|
||||||
cfg.defaultRenderFormat = settings.DefaultRenderFormat
|
cfg.defaultRenderFormat = settings.defaultRenderFormat
|
||||||
|
|
||||||
if strings.TrimSpace(cfg.promptDir) == "" {
|
if err := validateRequiredLibraryDirs(cfg.promptDir, cfg.profileDir); err != nil {
|
||||||
return errors.New(errPromptDirRequired)
|
return err
|
||||||
}
|
|
||||||
if strings.TrimSpace(cfg.profileDir) == "" {
|
|
||||||
return errors.New(errProfileDirRequired)
|
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(cfg.promptID) == "" {
|
if strings.TrimSpace(cfg.promptID) == "" {
|
||||||
return errors.New("--prompt is required")
|
return errors.New("--prompt is required")
|
||||||
@@ -476,6 +453,47 @@ func resolveAppSettings(fs *flag.FlagSet, configPath string, overrides appconfig
|
|||||||
return merged, nil
|
return merged, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func resolveCommonSettings(fs *flag.FlagSet, configPath string, overrides appconfig.CLIOverrides) (commonCommandSettings, error) {
|
||||||
|
settings, err := resolveAppSettings(fs, configPath, overrides)
|
||||||
|
if err != nil {
|
||||||
|
return commonCommandSettings{}, err
|
||||||
|
}
|
||||||
|
return commonCommandSettings{
|
||||||
|
promptDir: settings.PromptDir,
|
||||||
|
profileDir: settings.ProfileDir,
|
||||||
|
schemaDir: settings.SchemaDir,
|
||||||
|
serverAddr: settings.ServerAddr,
|
||||||
|
defaultRenderFormat: settings.DefaultRenderFormat,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateRequiredLibraryDirs(promptDir, profileDir string) error {
|
||||||
|
if strings.TrimSpace(promptDir) == "" {
|
||||||
|
return errors.New(errPromptDirRequired)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(profileDir) == "" {
|
||||||
|
return errors.New(errProfileDirRequired)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func newRunner(promptDir, profileDir, schemaDir string, llmClient llm.Client) *usecase.Runner {
|
||||||
|
return usecase.NewRunner(
|
||||||
|
promptdef.NewFilesystemRepository(promptDir),
|
||||||
|
profile.NewFilesystemRepository(profileDir),
|
||||||
|
artifactadapter.NewCompositeReader(),
|
||||||
|
prompt.NewGoRenderer(),
|
||||||
|
llmClient,
|
||||||
|
validate.NewStandardValidator(schemaDir),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
func newOpenAIClient() (*llm.OpenAICompatibleClient, error) {
|
||||||
|
return llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
|
||||||
|
Timeout: defaults.LLMRequestTimeoutDefault,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
func buildRunRequestFromConfig(cfg *runConfig) (domain.RunRequest, error) {
|
func buildRunRequestFromConfig(cfg *runConfig) (domain.RunRequest, error) {
|
||||||
inputMappings, err := parseMappings(cfg.inputRaw, false)
|
inputMappings, err := parseMappings(cfg.inputRaw, false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -410,6 +411,28 @@ defaults:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestParseRenderArgsExplicitFormatOverridesConfigDefaultFormat(t *testing.T) {
|
||||||
|
configPath := writeAppConfigFile(t, `
|
||||||
|
prompt_dir: ./from-config/prompts
|
||||||
|
profile_dir: ./from-config/profiles
|
||||||
|
defaults:
|
||||||
|
render_format: json
|
||||||
|
`)
|
||||||
|
|
||||||
|
cfg, err := parseRenderArgs([]string{
|
||||||
|
"--config", configPath,
|
||||||
|
"--prompt", "p",
|
||||||
|
"--input", "a=b",
|
||||||
|
"--format", "text",
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected valid args, got %v", err)
|
||||||
|
}
|
||||||
|
if cfg.outputFormat != renderformat.PreparedRunFormatText {
|
||||||
|
t.Fatalf("expected explicit --format text to override config default, got %q", cfg.outputFormat)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestParseServeArgsWithExplicitConfigLoadsSettingsAndCLIAddrOverrides(t *testing.T) {
|
func TestParseServeArgsWithExplicitConfigLoadsSettingsAndCLIAddrOverrides(t *testing.T) {
|
||||||
configPath := writeAppConfigFile(t, `
|
configPath := writeAppConfigFile(t, `
|
||||||
prompt_dir: ./from-config/prompts
|
prompt_dir: ./from-config/prompts
|
||||||
@@ -471,6 +494,59 @@ server:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunAndRenderBuildEquivalentRuntimeOverrideRequestsForSharedFlags(t *testing.T) {
|
||||||
|
runCfg, err := parseRunArgs([]string{
|
||||||
|
"--prompt-dir", "./prompts",
|
||||||
|
"--profile-dir", "./profiles",
|
||||||
|
"--prompt", "prompt-1",
|
||||||
|
"--profile", "profile-1",
|
||||||
|
"--input", "transcript=./transcript.md",
|
||||||
|
"--var", "session_date=2026-05-01",
|
||||||
|
"--llm-base-url", "http://localhost:8000/v1",
|
||||||
|
"--model", "model-x",
|
||||||
|
"--temperature", "0.8",
|
||||||
|
"--max-tokens", "123",
|
||||||
|
"--top-p", "0.6",
|
||||||
|
"--timeout", "90s",
|
||||||
|
"--api-key-env", "SCRIPTORIUM_API_KEY",
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected valid run args, got %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
renderCfg, err := parseRenderArgs([]string{
|
||||||
|
"--prompt-dir", "./prompts",
|
||||||
|
"--profile-dir", "./profiles",
|
||||||
|
"--prompt", "prompt-1",
|
||||||
|
"--profile", "profile-1",
|
||||||
|
"--input", "transcript=./transcript.md",
|
||||||
|
"--var", "session_date=2026-05-01",
|
||||||
|
"--llm-base-url", "http://localhost:8000/v1",
|
||||||
|
"--model", "model-x",
|
||||||
|
"--temperature", "0.8",
|
||||||
|
"--max-tokens", "123",
|
||||||
|
"--top-p", "0.6",
|
||||||
|
"--timeout", "90s",
|
||||||
|
"--api-key-env", "SCRIPTORIUM_API_KEY",
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected valid render args, got %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
runReq, err := buildRunRequestFromConfig(runCfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected run request build success, got %v", err)
|
||||||
|
}
|
||||||
|
renderReq, err := buildRunRequestFromConfig(&renderCfg.runConfig)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected render request build success, got %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !reflect.DeepEqual(runReq, renderReq) {
|
||||||
|
t.Fatalf("expected run/render shared flag requests to match.\nrun=%#v\nrender=%#v", runReq, renderReq)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestParseRunArgsFailsClearlyWhenNoEffectivePromptDir(t *testing.T) {
|
func TestParseRunArgsFailsClearlyWhenNoEffectivePromptDir(t *testing.T) {
|
||||||
configPath := writeAppConfigFile(t, `
|
configPath := writeAppConfigFile(t, `
|
||||||
profile_dir: ./profiles
|
profile_dir: ./profiles
|
||||||
@@ -586,42 +662,29 @@ func TestRunCommandVarsOptional(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRunCommandSucceedsWithPromptAndProfileDirsFromConfig(t *testing.T) {
|
func TestRunCommandSucceedsWithPromptAndProfileDirsFromConfig(t *testing.T) {
|
||||||
tmp := t.TempDir()
|
lib := newCLITestLibrary(t)
|
||||||
promptDir := filepath.Join(tmp, "prompts")
|
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
|
||||||
profileDir := filepath.Join(tmp, "profiles")
|
|
||||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
inputPath := filepath.Join(tmp, "transcript.md")
|
|
||||||
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
ts := newTestLLMServer("from-config-dirs", nil)
|
ts := newTestLLMServer("from-config-dirs", nil)
|
||||||
defer ts.Close()
|
defer ts.Close()
|
||||||
|
|
||||||
writePromptFile(t, promptDir, "prompt.default", "local-default")
|
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
|
||||||
writeProfileFile(t, profileDir, "local-default", ts.URL+"/v1", "profile-model")
|
writeProfileFile(t, lib.profileDir, "local-default", ts.URL+"/v1", "profile-model")
|
||||||
configPath := writeAppConfigFile(t, fmt.Sprintf(`
|
configPath := writeAppConfigFile(t, fmt.Sprintf(`
|
||||||
prompt_dir: %s
|
prompt_dir: %s
|
||||||
profile_dir: %s
|
profile_dir: %s
|
||||||
`, promptDir, profileDir))
|
`, lib.promptDir, lib.profileDir))
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
code, stdout, stderr := runCLICommand(t, runCommand, []string{
|
||||||
var stderr bytes.Buffer
|
|
||||||
code := runCommand([]string{
|
|
||||||
"--config", configPath,
|
"--config", configPath,
|
||||||
"--prompt", "prompt.default",
|
"--prompt", "prompt.default",
|
||||||
"--input", "transcript=" + inputPath,
|
"--input", "transcript=" + inputPath,
|
||||||
}, &stdout, &stderr)
|
})
|
||||||
if code != ExitOK {
|
if code != ExitOK {
|
||||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
}
|
}
|
||||||
if stdout.String() != "from-config-dirs" {
|
if stdout != "from-config-dirs" {
|
||||||
t.Fatalf("unexpected stdout output: %q", stdout.String())
|
t.Fatalf("unexpected stdout output: %q", stdout)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -630,28 +693,15 @@ func TestRenderCommandDefaultFormatTextIncludesPreparedDetailsAndNoSecrets(t *te
|
|||||||
const secret = "super-secret-render-key"
|
const secret = "super-secret-render-key"
|
||||||
t.Setenv(envName, secret)
|
t.Setenv(envName, secret)
|
||||||
|
|
||||||
tmp := t.TempDir()
|
lib := newCLITestLibrary(t)
|
||||||
promptDir := filepath.Join(tmp, "prompts")
|
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
|
||||||
profileDir := filepath.Join(tmp, "profiles")
|
|
||||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
inputPath := filepath.Join(tmp, "transcript.md")
|
|
||||||
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
writePromptFileWithTemplate(t, promptDir, "prompt.render", "local-default", "Date {{.session_date}} - Summarize: {{input \"transcript\"}}")
|
writePromptFileWithTemplate(t, lib.promptDir, "prompt.render", "local-default", "Date {{.session_date}} - Summarize: {{input \"transcript\"}}")
|
||||||
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||||
var stderr bytes.Buffer
|
"--prompt-dir", lib.promptDir,
|
||||||
code := renderCommand([]string{
|
"--profile-dir", lib.profileDir,
|
||||||
"--prompt-dir", promptDir,
|
|
||||||
"--profile-dir", profileDir,
|
|
||||||
"--prompt", "prompt.render",
|
"--prompt", "prompt.render",
|
||||||
"--profile", "local-default",
|
"--profile", "local-default",
|
||||||
"--input", "transcript=" + inputPath,
|
"--input", "transcript=" + inputPath,
|
||||||
@@ -663,15 +713,15 @@ func TestRenderCommandDefaultFormatTextIncludesPreparedDetailsAndNoSecrets(t *te
|
|||||||
"--top-p", "0.2",
|
"--top-p", "0.2",
|
||||||
"--timeout", "20s",
|
"--timeout", "20s",
|
||||||
"--api-key-env", envName,
|
"--api-key-env", envName,
|
||||||
}, &stdout, &stderr)
|
})
|
||||||
if code != ExitOK {
|
if code != ExitOK {
|
||||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
}
|
}
|
||||||
if stderr.Len() != 0 {
|
if stderr != "" {
|
||||||
t.Fatalf("expected empty stderr on success, got %q", stderr.String())
|
t.Fatalf("expected empty stderr on success, got %q", stderr)
|
||||||
}
|
}
|
||||||
|
|
||||||
out := stdout.String()
|
out := stdout
|
||||||
for _, want := range []string{
|
for _, want := range []string{
|
||||||
"prompt: prompt.render",
|
"prompt: prompt.render",
|
||||||
"selected_profile_id: local-default",
|
"selected_profile_id: local-default",
|
||||||
@@ -698,111 +748,72 @@ func TestRenderCommandDefaultFormatTextIncludesPreparedDetailsAndNoSecrets(t *te
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRenderCommandSucceedsWithPromptAndProfileDirsFromConfig(t *testing.T) {
|
func TestRenderCommandSucceedsWithPromptAndProfileDirsFromConfig(t *testing.T) {
|
||||||
tmp := t.TempDir()
|
lib := newCLITestLibrary(t)
|
||||||
promptDir := filepath.Join(tmp, "prompts")
|
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
|
||||||
profileDir := filepath.Join(tmp, "profiles")
|
|
||||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
inputPath := filepath.Join(tmp, "transcript.md")
|
|
||||||
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
writePromptFile(t, promptDir, "prompt.render", "local-default")
|
writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
|
||||||
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
||||||
configPath := writeAppConfigFile(t, fmt.Sprintf(`
|
configPath := writeAppConfigFile(t, fmt.Sprintf(`
|
||||||
prompt_dir: %s
|
prompt_dir: %s
|
||||||
profile_dir: %s
|
profile_dir: %s
|
||||||
`, promptDir, profileDir))
|
`, lib.promptDir, lib.profileDir))
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||||
var stderr bytes.Buffer
|
|
||||||
code := renderCommand([]string{
|
|
||||||
"--config", configPath,
|
"--config", configPath,
|
||||||
"--prompt", "prompt.render",
|
"--prompt", "prompt.render",
|
||||||
"--input", "transcript=" + inputPath,
|
"--input", "transcript=" + inputPath,
|
||||||
}, &stdout, &stderr)
|
})
|
||||||
if code != ExitOK {
|
if code != ExitOK {
|
||||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "prompt: prompt.render") {
|
if !strings.Contains(stdout, "prompt: prompt.render") {
|
||||||
t.Fatalf("expected rendered output, got %q", stdout.String())
|
t.Fatalf("expected rendered output, got %q", stdout)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRenderCommandExplicitTextFormatWorks(t *testing.T) {
|
func TestRenderCommandExplicitTextFormatWorks(t *testing.T) {
|
||||||
tmp := t.TempDir()
|
lib := newCLITestLibrary(t)
|
||||||
promptDir := filepath.Join(tmp, "prompts")
|
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
|
||||||
profileDir := filepath.Join(tmp, "profiles")
|
|
||||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
inputPath := filepath.Join(tmp, "transcript.md")
|
|
||||||
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
writePromptFile(t, promptDir, "prompt.render", "local-default")
|
writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
|
||||||
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||||
var stderr bytes.Buffer
|
"--prompt-dir", lib.promptDir,
|
||||||
code := renderCommand([]string{
|
"--profile-dir", lib.profileDir,
|
||||||
"--prompt-dir", promptDir,
|
|
||||||
"--profile-dir", profileDir,
|
|
||||||
"--prompt", "prompt.render",
|
"--prompt", "prompt.render",
|
||||||
"--input", "transcript=" + inputPath,
|
"--input", "transcript=" + inputPath,
|
||||||
"--format", "text",
|
"--format", "text",
|
||||||
}, &stdout, &stderr)
|
})
|
||||||
if code != ExitOK {
|
if code != ExitOK {
|
||||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "prompt: prompt.render") {
|
if !strings.Contains(stdout, "prompt: prompt.render") {
|
||||||
t.Fatalf("expected text output for explicit --format text, got %q", stdout.String())
|
t.Fatalf("expected text output for explicit --format text, got %q", stdout)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRenderCommandExplicitJSONFormatOutputsValidJSON(t *testing.T) {
|
func TestRenderCommandExplicitJSONFormatOutputsValidJSON(t *testing.T) {
|
||||||
tmp := t.TempDir()
|
lib := newCLITestLibrary(t)
|
||||||
promptDir := filepath.Join(tmp, "prompts")
|
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
|
||||||
profileDir := filepath.Join(tmp, "profiles")
|
|
||||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
inputPath := filepath.Join(tmp, "transcript.md")
|
|
||||||
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
writePromptFile(t, promptDir, "prompt.render", "local-default")
|
writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
|
||||||
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||||
var stderr bytes.Buffer
|
"--prompt-dir", lib.promptDir,
|
||||||
code := renderCommand([]string{
|
"--profile-dir", lib.profileDir,
|
||||||
"--prompt-dir", promptDir,
|
|
||||||
"--profile-dir", profileDir,
|
|
||||||
"--prompt", "prompt.render",
|
"--prompt", "prompt.render",
|
||||||
"--input", "transcript=" + inputPath,
|
"--input", "transcript=" + inputPath,
|
||||||
"--format", "json",
|
"--format", "json",
|
||||||
}, &stdout, &stderr)
|
})
|
||||||
if code != ExitOK {
|
if code != ExitOK {
|
||||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
}
|
}
|
||||||
|
|
||||||
var payload map[string]any
|
var payload map[string]any
|
||||||
if err := json.Unmarshal(stdout.Bytes(), &payload); err != nil {
|
if err := json.Unmarshal([]byte(stdout), &payload); err != nil {
|
||||||
t.Fatalf("expected valid json output, got %v\nbody=%s", err, stdout.String())
|
t.Fatalf("expected valid json output, got %v\nbody=%s", err, stdout)
|
||||||
}
|
}
|
||||||
if payload["prompt_id"] != "prompt.render" {
|
if payload["prompt_id"] != "prompt.render" {
|
||||||
t.Fatalf("expected prompt_id, got %#v", payload["prompt_id"])
|
t.Fatalf("expected prompt_id, got %#v", payload["prompt_id"])
|
||||||
@@ -834,38 +845,25 @@ func TestRenderCommandUnknownFormatFailsClearly(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRenderCommandOutWritesToFile(t *testing.T) {
|
func TestRenderCommandOutWritesToFile(t *testing.T) {
|
||||||
tmp := t.TempDir()
|
lib := newCLITestLibrary(t)
|
||||||
promptDir := filepath.Join(tmp, "prompts")
|
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
|
||||||
profileDir := filepath.Join(tmp, "profiles")
|
outPath := filepath.Join(lib.rootDir, "render.txt")
|
||||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
inputPath := filepath.Join(tmp, "transcript.md")
|
|
||||||
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
outPath := filepath.Join(tmp, "render.txt")
|
|
||||||
|
|
||||||
writePromptFile(t, promptDir, "prompt.render", "local-default")
|
writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
|
||||||
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||||
var stderr bytes.Buffer
|
"--prompt-dir", lib.promptDir,
|
||||||
code := renderCommand([]string{
|
"--profile-dir", lib.profileDir,
|
||||||
"--prompt-dir", promptDir,
|
|
||||||
"--profile-dir", profileDir,
|
|
||||||
"--prompt", "prompt.render",
|
"--prompt", "prompt.render",
|
||||||
"--input", "transcript=" + inputPath,
|
"--input", "transcript=" + inputPath,
|
||||||
"--out", outPath,
|
"--out", outPath,
|
||||||
}, &stdout, &stderr)
|
})
|
||||||
if code != ExitOK {
|
if code != ExitOK {
|
||||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
}
|
}
|
||||||
if stdout.Len() != 0 {
|
if stdout != "" {
|
||||||
t.Fatalf("expected empty stdout when --out is set, got %q", stdout.String())
|
t.Fatalf("expected empty stdout when --out is set, got %q", stdout)
|
||||||
}
|
}
|
||||||
out, err := os.ReadFile(outPath)
|
out, err := os.ReadFile(outPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -877,36 +875,23 @@ func TestRenderCommandOutWritesToFile(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRenderCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
|
func TestRenderCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
|
||||||
tmp := t.TempDir()
|
lib := newCLITestLibrary(t)
|
||||||
promptDir := filepath.Join(tmp, "prompts")
|
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
|
||||||
profileDir := filepath.Join(tmp, "profiles")
|
|
||||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
inputPath := filepath.Join(tmp, "transcript.md")
|
|
||||||
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
writePromptFile(t, promptDir, "prompt.default", "local-default")
|
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
|
||||||
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model")
|
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model")
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||||
var stderr bytes.Buffer
|
"--prompt-dir", lib.promptDir,
|
||||||
code := renderCommand([]string{
|
"--profile-dir", lib.profileDir,
|
||||||
"--prompt-dir", promptDir,
|
|
||||||
"--profile-dir", profileDir,
|
|
||||||
"--prompt", "prompt.default",
|
"--prompt", "prompt.default",
|
||||||
"--input", "transcript=" + inputPath,
|
"--input", "transcript=" + inputPath,
|
||||||
}, &stdout, &stderr)
|
})
|
||||||
|
|
||||||
if code != ExitOK {
|
if code != ExitOK {
|
||||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
}
|
}
|
||||||
out := stdout.String()
|
out := stdout
|
||||||
if !strings.Contains(out, "selected_profile_id: local-default") {
|
if !strings.Contains(out, "selected_profile_id: local-default") {
|
||||||
t.Fatalf("expected prompt default profile in output, got %q", out)
|
t.Fatalf("expected prompt default profile in output, got %q", out)
|
||||||
}
|
}
|
||||||
@@ -916,38 +901,25 @@ func TestRenderCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRenderCommandExplicitProfileOverridesPromptDefault(t *testing.T) {
|
func TestRenderCommandExplicitProfileOverridesPromptDefault(t *testing.T) {
|
||||||
tmp := t.TempDir()
|
lib := newCLITestLibrary(t)
|
||||||
promptDir := filepath.Join(tmp, "prompts")
|
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
|
||||||
profileDir := filepath.Join(tmp, "profiles")
|
|
||||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
inputPath := filepath.Join(tmp, "transcript.md")
|
|
||||||
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
writePromptFile(t, promptDir, "prompt.default", "local-default")
|
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
|
||||||
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model")
|
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model")
|
||||||
writeProfileFile(t, profileDir, "quality", "http://127.0.0.1:1/v1", "quality-model")
|
writeProfileFile(t, lib.profileDir, "quality", "http://127.0.0.1:1/v1", "quality-model")
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||||
var stderr bytes.Buffer
|
"--prompt-dir", lib.promptDir,
|
||||||
code := renderCommand([]string{
|
"--profile-dir", lib.profileDir,
|
||||||
"--prompt-dir", promptDir,
|
|
||||||
"--profile-dir", profileDir,
|
|
||||||
"--prompt", "prompt.default",
|
"--prompt", "prompt.default",
|
||||||
"--profile", "quality",
|
"--profile", "quality",
|
||||||
"--input", "transcript=" + inputPath,
|
"--input", "transcript=" + inputPath,
|
||||||
}, &stdout, &stderr)
|
})
|
||||||
|
|
||||||
if code != ExitOK {
|
if code != ExitOK {
|
||||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
}
|
}
|
||||||
out := stdout.String()
|
out := stdout
|
||||||
if !strings.Contains(out, "selected_profile_id: quality") {
|
if !strings.Contains(out, "selected_profile_id: quality") {
|
||||||
t.Fatalf("expected explicit profile in output, got %q", out)
|
t.Fatalf("expected explicit profile in output, got %q", out)
|
||||||
}
|
}
|
||||||
@@ -957,104 +929,67 @@ func TestRenderCommandExplicitProfileOverridesPromptDefault(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRunCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
|
func TestRunCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
|
||||||
tmp := t.TempDir()
|
lib := newCLITestLibrary(t)
|
||||||
promptDir := filepath.Join(tmp, "prompts")
|
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
|
||||||
profileDir := filepath.Join(tmp, "profiles")
|
|
||||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
inputPath := filepath.Join(tmp, "transcript.md")
|
|
||||||
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
ts := newTestLLMServer("default-output", nil)
|
ts := newTestLLMServer("default-output", nil)
|
||||||
defer ts.Close()
|
defer ts.Close()
|
||||||
|
|
||||||
writePromptFile(t, promptDir, "prompt.default", "local-default")
|
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
|
||||||
writeProfileFile(t, profileDir, "local-default", ts.URL+"/v1", "profile-model")
|
writeProfileFile(t, lib.profileDir, "local-default", ts.URL+"/v1", "profile-model")
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
code, stdout, stderr := runCLICommand(t, runCommand, []string{
|
||||||
var stderr bytes.Buffer
|
"--prompt-dir", lib.promptDir,
|
||||||
code := runCommand([]string{
|
"--profile-dir", lib.profileDir,
|
||||||
"--prompt-dir", promptDir,
|
|
||||||
"--profile-dir", profileDir,
|
|
||||||
"--prompt", "prompt.default",
|
"--prompt", "prompt.default",
|
||||||
"--input", "transcript=" + inputPath,
|
"--input", "transcript=" + inputPath,
|
||||||
}, &stdout, &stderr)
|
})
|
||||||
|
|
||||||
if code != ExitOK {
|
if code != ExitOK {
|
||||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
}
|
}
|
||||||
if stdout.String() != "default-output" {
|
if stdout != "default-output" {
|
||||||
t.Fatalf("unexpected stdout output: %q", stdout.String())
|
t.Fatalf("unexpected stdout output: %q", stdout)
|
||||||
}
|
}
|
||||||
if !strings.Contains(stderr.String(), "selected_profile=local-default") {
|
if !strings.Contains(stderr, "selected_profile=local-default") {
|
||||||
t.Fatalf("expected selected profile in summary, got %q", stderr.String())
|
t.Fatalf("expected selected profile in summary, got %q", stderr)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunCommandExplicitProfileOverridesPromptDefault(t *testing.T) {
|
func TestRunCommandExplicitProfileOverridesPromptDefault(t *testing.T) {
|
||||||
tmp := t.TempDir()
|
lib := newCLITestLibrary(t)
|
||||||
promptDir := filepath.Join(tmp, "prompts")
|
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
|
||||||
profileDir := filepath.Join(tmp, "profiles")
|
|
||||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
inputPath := filepath.Join(tmp, "transcript.md")
|
|
||||||
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defaultServer := newTestLLMServer("from-default", nil)
|
defaultServer := newTestLLMServer("from-default", nil)
|
||||||
defer defaultServer.Close()
|
defer defaultServer.Close()
|
||||||
overrideServer := newTestLLMServer("from-override", nil)
|
overrideServer := newTestLLMServer("from-override", nil)
|
||||||
defer overrideServer.Close()
|
defer overrideServer.Close()
|
||||||
|
|
||||||
writePromptFile(t, promptDir, "prompt.default", "local-default")
|
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
|
||||||
writeProfileFile(t, profileDir, "local-default", defaultServer.URL+"/v1", "default-model")
|
writeProfileFile(t, lib.profileDir, "local-default", defaultServer.URL+"/v1", "default-model")
|
||||||
writeProfileFile(t, profileDir, "quality", overrideServer.URL+"/v1", "quality-model")
|
writeProfileFile(t, lib.profileDir, "quality", overrideServer.URL+"/v1", "quality-model")
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
code, stdout, stderr := runCLICommand(t, runCommand, []string{
|
||||||
var stderr bytes.Buffer
|
"--prompt-dir", lib.promptDir,
|
||||||
code := runCommand([]string{
|
"--profile-dir", lib.profileDir,
|
||||||
"--prompt-dir", promptDir,
|
|
||||||
"--profile-dir", profileDir,
|
|
||||||
"--prompt", "prompt.default",
|
"--prompt", "prompt.default",
|
||||||
"--profile", "quality",
|
"--profile", "quality",
|
||||||
"--input", "transcript=" + inputPath,
|
"--input", "transcript=" + inputPath,
|
||||||
}, &stdout, &stderr)
|
})
|
||||||
if code != ExitOK {
|
if code != ExitOK {
|
||||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
}
|
}
|
||||||
if stdout.String() != "from-override" {
|
if stdout != "from-override" {
|
||||||
t.Fatalf("expected explicit profile output, got %q", stdout.String())
|
t.Fatalf("expected explicit profile output, got %q", stdout)
|
||||||
}
|
}
|
||||||
if !strings.Contains(stderr.String(), "selected_profile=quality") {
|
if !strings.Contains(stderr, "selected_profile=quality") {
|
||||||
t.Fatalf("expected selected profile quality, got %q", stderr.String())
|
t.Fatalf("expected selected profile quality, got %q", stderr)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
|
func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
|
||||||
tmp := t.TempDir()
|
lib := newCLITestLibrary(t)
|
||||||
promptDir := filepath.Join(tmp, "prompts")
|
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
|
||||||
profileDir := filepath.Join(tmp, "profiles")
|
|
||||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
inputPath := filepath.Join(tmp, "transcript.md")
|
|
||||||
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var baseHits int32
|
var baseHits int32
|
||||||
baseServer := newTestLLMServer("base", &baseHits)
|
baseServer := newTestLLMServer("base", &baseHits)
|
||||||
@@ -1071,14 +1006,12 @@ func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
|
|||||||
}))
|
}))
|
||||||
defer overrideServer.Close()
|
defer overrideServer.Close()
|
||||||
|
|
||||||
writePromptFile(t, promptDir, "prompt.default", "local-default")
|
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
|
||||||
writeProfileFile(t, profileDir, "local-default", baseServer.URL+"/v1", "profile-model")
|
writeProfileFile(t, lib.profileDir, "local-default", baseServer.URL+"/v1", "profile-model")
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
code, stdout, stderr := runCLICommand(t, runCommand, []string{
|
||||||
var stderr bytes.Buffer
|
"--prompt-dir", lib.promptDir,
|
||||||
code := runCommand([]string{
|
"--profile-dir", lib.profileDir,
|
||||||
"--prompt-dir", promptDir,
|
|
||||||
"--profile-dir", profileDir,
|
|
||||||
"--prompt", "prompt.default",
|
"--prompt", "prompt.default",
|
||||||
"--input", "transcript=" + inputPath,
|
"--input", "transcript=" + inputPath,
|
||||||
"--llm-base-url", overrideServer.URL + "/v1",
|
"--llm-base-url", overrideServer.URL + "/v1",
|
||||||
@@ -1087,9 +1020,9 @@ func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
|
|||||||
"--max-tokens", "55",
|
"--max-tokens", "55",
|
||||||
"--top-p", "0.2",
|
"--top-p", "0.2",
|
||||||
"--timeout", "20s",
|
"--timeout", "20s",
|
||||||
}, &stdout, &stderr)
|
})
|
||||||
if code != ExitOK {
|
if code != ExitOK {
|
||||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
}
|
}
|
||||||
if atomic.LoadInt32(&baseHits) != 0 {
|
if atomic.LoadInt32(&baseHits) != 0 {
|
||||||
t.Fatalf("expected base profile endpoint not to be hit, got %d", baseHits)
|
t.Fatalf("expected base profile endpoint not to be hit, got %d", baseHits)
|
||||||
@@ -1097,8 +1030,8 @@ func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
|
|||||||
if atomic.LoadInt32(&overrideHits) != 1 {
|
if atomic.LoadInt32(&overrideHits) != 1 {
|
||||||
t.Fatalf("expected override endpoint to be hit once, got %d", overrideHits)
|
t.Fatalf("expected override endpoint to be hit once, got %d", overrideHits)
|
||||||
}
|
}
|
||||||
if stdout.String() != "override" {
|
if stdout != "override" {
|
||||||
t.Fatalf("unexpected stdout output: %q", stdout.String())
|
t.Fatalf("unexpected stdout output: %q", stdout)
|
||||||
}
|
}
|
||||||
if !strings.Contains(observedBody, `"model":"override-model"`) {
|
if !strings.Contains(observedBody, `"model":"override-model"`) {
|
||||||
t.Fatalf("expected override model in request body, got %s", observedBody)
|
t.Fatalf("expected override model in request body, got %s", observedBody)
|
||||||
@@ -1133,6 +1066,49 @@ func TestWriteOutputAndSummaryUseSeparateWriters(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type cliTestLibrary struct {
|
||||||
|
rootDir string
|
||||||
|
promptDir string
|
||||||
|
profileDir string
|
||||||
|
}
|
||||||
|
|
||||||
|
func newCLITestLibrary(t *testing.T) *cliTestLibrary {
|
||||||
|
t.Helper()
|
||||||
|
root := t.TempDir()
|
||||||
|
lib := &cliTestLibrary{
|
||||||
|
rootDir: root,
|
||||||
|
promptDir: filepath.Join(root, "prompts"),
|
||||||
|
profileDir: filepath.Join(root, "profiles"),
|
||||||
|
}
|
||||||
|
if err := os.MkdirAll(lib.promptDir, 0o755); err != nil {
|
||||||
|
t.Fatalf("failed to create prompt fixture directory: %v", err)
|
||||||
|
}
|
||||||
|
if err := os.MkdirAll(lib.profileDir, 0o755); err != nil {
|
||||||
|
t.Fatalf("failed to create profile fixture directory: %v", err)
|
||||||
|
}
|
||||||
|
return lib
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *cliTestLibrary) writeInputFile(t *testing.T, name, body string) string {
|
||||||
|
t.Helper()
|
||||||
|
path := filepath.Join(l.rootDir, name)
|
||||||
|
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||||
|
t.Fatalf("failed to create input fixture directory: %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
|
||||||
|
t.Fatalf("failed to write input fixture: %v", err)
|
||||||
|
}
|
||||||
|
return path
|
||||||
|
}
|
||||||
|
|
||||||
|
func runCLICommand(t *testing.T, command func([]string, io.Writer, io.Writer) int, args []string) (int, string, string) {
|
||||||
|
t.Helper()
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := command(args, &stdout, &stderr)
|
||||||
|
return code, stdout.String(), stderr.String()
|
||||||
|
}
|
||||||
|
|
||||||
func writePromptFile(t *testing.T, dir, id, defaultProfile string) {
|
func writePromptFile(t *testing.T, dir, id, defaultProfile string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
writePromptFileWithTemplate(t, dir, id, defaultProfile, "Summarize: {{input \"transcript\"}}")
|
writePromptFileWithTemplate(t, dir, id, defaultProfile, "Summarize: {{input \"transcript\"}}")
|
||||||
|
|||||||
@@ -27,6 +27,7 @@ type modelOverrideRequestDTO struct {
|
|||||||
MaxTokens int `json:"max_tokens,omitempty"`
|
MaxTokens int `json:"max_tokens,omitempty"`
|
||||||
TopP float64 `json:"top_p,omitempty"`
|
TopP float64 `json:"top_p,omitempty"`
|
||||||
TimeoutSeconds int `json:"timeout_seconds,omitempty"`
|
TimeoutSeconds int `json:"timeout_seconds,omitempty"`
|
||||||
|
ServiceTier string `json:"service_tier,omitempty"`
|
||||||
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||||
APIKeyEnv string `json:"api_key_env,omitempty"`
|
APIKeyEnv string `json:"api_key_env,omitempty"`
|
||||||
ExtraParams map[string]string `json:"extra_params,omitempty"`
|
ExtraParams map[string]string `json:"extra_params,omitempty"`
|
||||||
@@ -75,6 +76,7 @@ type modelParamsDTO struct {
|
|||||||
MaxTokens int `json:"max_tokens"`
|
MaxTokens int `json:"max_tokens"`
|
||||||
TopP float64 `json:"top_p"`
|
TopP float64 `json:"top_p"`
|
||||||
TimeoutSeconds int `json:"timeout_seconds"`
|
TimeoutSeconds int `json:"timeout_seconds"`
|
||||||
|
ServiceTier string `json:"service_tier,omitempty"`
|
||||||
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||||
APIKeyEnv string `json:"api_key_env,omitempty"`
|
APIKeyEnv string `json:"api_key_env,omitempty"`
|
||||||
ExtraParams map[string]string `json:"extra_params,omitempty"`
|
ExtraParams map[string]string `json:"extra_params,omitempty"`
|
||||||
|
|||||||
@@ -63,17 +63,7 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
|
|
||||||
var model *domain.ExecutionTarget
|
var model *domain.ExecutionTarget
|
||||||
if req.Model != nil {
|
if req.Model != nil {
|
||||||
model = &domain.ExecutionTarget{
|
model = executionTargetFromModelOverrideDTO(req.Model)
|
||||||
Endpoint: req.Model.Endpoint,
|
|
||||||
Model: req.Model.Model,
|
|
||||||
Temperature: req.Model.Temperature,
|
|
||||||
MaxTokens: req.Model.MaxTokens,
|
|
||||||
TopP: req.Model.TopP,
|
|
||||||
TimeoutSeconds: req.Model.TimeoutSeconds,
|
|
||||||
ReasoningEffort: req.Model.ReasoningEffort,
|
|
||||||
APIKeyEnv: req.Model.APIKeyEnv,
|
|
||||||
ExtraParams: req.Model.ExtraParams,
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
res, err := h.runner.Run(r.Context(), domain.RunRequest{
|
res, err := h.runner.Run(r.Context(), domain.RunRequest{
|
||||||
@@ -109,18 +99,8 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
SelectedProfileID: res.SelectedProfileID,
|
SelectedProfileID: res.SelectedProfileID,
|
||||||
ModelName: res.ModelName,
|
ModelName: res.ModelName,
|
||||||
Endpoint: res.Endpoint,
|
Endpoint: res.Endpoint,
|
||||||
ModelParams: modelParamsDTO{
|
ModelParams: modelParamsDTOFromExecutionTarget(res.EffectiveModelParams),
|
||||||
Endpoint: res.EffectiveModelParams.Endpoint,
|
InputHashes: res.InputHashes,
|
||||||
Model: res.EffectiveModelParams.Model,
|
|
||||||
Temperature: res.EffectiveModelParams.Temperature,
|
|
||||||
MaxTokens: res.EffectiveModelParams.MaxTokens,
|
|
||||||
TopP: res.EffectiveModelParams.TopP,
|
|
||||||
TimeoutSeconds: res.EffectiveModelParams.TimeoutSeconds,
|
|
||||||
ReasoningEffort: res.EffectiveModelParams.ReasoningEffort,
|
|
||||||
APIKeyEnv: res.EffectiveModelParams.APIKeyEnv,
|
|
||||||
ExtraParams: res.EffectiveModelParams.ExtraParams,
|
|
||||||
},
|
|
||||||
InputHashes: res.InputHashes,
|
|
||||||
Usage: tokenUsageDTO{
|
Usage: tokenUsageDTO{
|
||||||
PromptTokens: res.Usage.PromptTokens,
|
PromptTokens: res.Usage.PromptTokens,
|
||||||
CompletionTokens: res.Usage.CompletionTokens,
|
CompletionTokens: res.Usage.CompletionTokens,
|
||||||
@@ -141,6 +121,39 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
writeJSON(w, http.StatusOK, resp)
|
writeJSON(w, http.StatusOK, resp)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func executionTargetFromModelOverrideDTO(dto *modelOverrideRequestDTO) *domain.ExecutionTarget {
|
||||||
|
if dto == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return &domain.ExecutionTarget{
|
||||||
|
Endpoint: dto.Endpoint,
|
||||||
|
Model: dto.Model,
|
||||||
|
Temperature: dto.Temperature,
|
||||||
|
MaxTokens: dto.MaxTokens,
|
||||||
|
TopP: dto.TopP,
|
||||||
|
TimeoutSeconds: dto.TimeoutSeconds,
|
||||||
|
ServiceTier: dto.ServiceTier,
|
||||||
|
ReasoningEffort: dto.ReasoningEffort,
|
||||||
|
APIKeyEnv: dto.APIKeyEnv,
|
||||||
|
ExtraParams: dto.ExtraParams,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func modelParamsDTOFromExecutionTarget(target domain.ExecutionTarget) modelParamsDTO {
|
||||||
|
return modelParamsDTO{
|
||||||
|
Endpoint: target.Endpoint,
|
||||||
|
Model: target.Model,
|
||||||
|
Temperature: target.Temperature,
|
||||||
|
MaxTokens: target.MaxTokens,
|
||||||
|
TopP: target.TopP,
|
||||||
|
TimeoutSeconds: target.TimeoutSeconds,
|
||||||
|
ServiceTier: target.ServiceTier,
|
||||||
|
ReasoningEffort: target.ReasoningEffort,
|
||||||
|
APIKeyEnv: target.APIKeyEnv,
|
||||||
|
ExtraParams: target.ExtraParams,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func mapValidation(v domain.ValidationResult) validationDTO {
|
func mapValidation(v domain.ValidationResult) validationDTO {
|
||||||
return validationDTO{
|
return validationDTO{
|
||||||
Status: string(v.Status),
|
Status: string(v.Status),
|
||||||
@@ -162,9 +175,9 @@ func mapRunError(err error) (int, string, string) {
|
|||||||
return http.StatusBadRequest, "prompt_load_failed", "failed to load prompt definition"
|
return http.StatusBadRequest, "prompt_load_failed", "failed to load prompt definition"
|
||||||
case errors.Is(err, profile.ErrInvalidYAML), errors.Is(err, profile.ErrInvalidProfile):
|
case errors.Is(err, profile.ErrInvalidYAML), errors.Is(err, profile.ErrInvalidProfile):
|
||||||
return http.StatusBadRequest, "profile_load_failed", "failed to load execution profile"
|
return http.StatusBadRequest, "profile_load_failed", "failed to load execution profile"
|
||||||
case errors.Is(err, usecase.ErrInvalidRequest) && strings.Contains(err.Error(), "profile id is required either in request or prompt default_profile"):
|
case errors.Is(err, usecase.ErrProfileRequired):
|
||||||
return http.StatusBadRequest, "profile_required", "profile_id is required when prompt default_profile is not set"
|
return http.StatusBadRequest, "profile_required", "profile_id is required when prompt default_profile is not set"
|
||||||
case errors.Is(err, usecase.ErrInvalidRequest) && strings.Contains(err.Error(), "api key environment variable"):
|
case errors.Is(err, usecase.ErrAPIKeyEnvMissing):
|
||||||
return http.StatusBadRequest, "api_key_env_missing", "api_key_env is set but the environment variable is missing"
|
return http.StatusBadRequest, "api_key_env_missing", "api_key_env is set but the environment variable is missing"
|
||||||
case errors.Is(err, usecase.ErrInvalidRequest):
|
case errors.Is(err, usecase.ErrInvalidRequest):
|
||||||
return http.StatusBadRequest, "invalid_request", "invalid run request"
|
return http.StatusBadRequest, "invalid_request", "invalid run request"
|
||||||
|
|||||||
@@ -4,10 +4,10 @@ import (
|
|||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
|
||||||
"fmt"
|
"fmt"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
@@ -62,6 +62,7 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
|||||||
MaxTokens: 42,
|
MaxTokens: 42,
|
||||||
TopP: 0.9,
|
TopP: 0.9,
|
||||||
TimeoutSeconds: 120,
|
TimeoutSeconds: 120,
|
||||||
|
ServiceTier: "priority",
|
||||||
APIKeyEnv: envName,
|
APIKeyEnv: envName,
|
||||||
},
|
},
|
||||||
InputHashes: map[string]string{"transcript": "h1"},
|
InputHashes: map[string]string{"transcript": "h1"},
|
||||||
@@ -81,7 +82,7 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
|||||||
"transcript": {"type": "file", "uri": "./t.md"}
|
"transcript": {"type": "file", "uri": "./t.md"}
|
||||||
},
|
},
|
||||||
"vars": {"k": "v"},
|
"vars": {"k": "v"},
|
||||||
"model": {"model": "gpt-x", "timeout_seconds": 120, "api_key_env": "SCRIPTORIUM_API_KEY"}
|
"model": {"model": "gpt-x", "timeout_seconds": 120, "service_tier": "flex", "api_key_env": "SCRIPTORIUM_API_KEY"}
|
||||||
}`)
|
}`)
|
||||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewReader(body))
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewReader(body))
|
||||||
w := httptest.NewRecorder()
|
w := httptest.NewRecorder()
|
||||||
@@ -114,6 +115,9 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
|||||||
if modelParams["api_key_env"] != envName {
|
if modelParams["api_key_env"] != envName {
|
||||||
t.Fatalf("expected model_params.api_key_env=%q, got %#v", envName, modelParams["api_key_env"])
|
t.Fatalf("expected model_params.api_key_env=%q, got %#v", envName, modelParams["api_key_env"])
|
||||||
}
|
}
|
||||||
|
if modelParams["service_tier"] != "priority" {
|
||||||
|
t.Fatalf("expected model_params.service_tier=priority, got %#v", modelParams["service_tier"])
|
||||||
|
}
|
||||||
if strings.Contains(w.Body.String(), secret) {
|
if strings.Contains(w.Body.String(), secret) {
|
||||||
t.Fatalf("response leaked raw API key value: %s", w.Body.String())
|
t.Fatalf("response leaked raw API key value: %s", w.Body.String())
|
||||||
}
|
}
|
||||||
@@ -133,6 +137,9 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
|||||||
if r.last.Execution.TimeoutSeconds != 120 {
|
if r.last.Execution.TimeoutSeconds != 120 {
|
||||||
t.Fatalf("expected timeout_seconds override 120, got %#v", r.last.Execution)
|
t.Fatalf("expected timeout_seconds override 120, got %#v", r.last.Execution)
|
||||||
}
|
}
|
||||||
|
if r.last.Execution.ServiceTier != "flex" {
|
||||||
|
t.Fatalf("expected service_tier override flex, got %#v", r.last.Execution)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
|
func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
|
||||||
@@ -166,6 +173,136 @@ func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestHandlerModelOverrideMapsAllSupportedExecutionFields(t *testing.T) {
|
||||||
|
r := &fakeRunner{result: &domain.RunResult{
|
||||||
|
Artifact: domain.Artifact{Body: []byte("ok")},
|
||||||
|
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
|
||||||
|
EffectiveModelParams: domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||||
|
}}
|
||||||
|
h := NewHandler(r)
|
||||||
|
|
||||||
|
reqBody := `{
|
||||||
|
"prompt_id": "prompt-1",
|
||||||
|
"inputs": {"transcript": {"type": "file", "uri": "./t.md"}},
|
||||||
|
"model": {
|
||||||
|
"endpoint": "http://override/v1",
|
||||||
|
"model": "override-model",
|
||||||
|
"temperature": 0.6,
|
||||||
|
"max_tokens": 250,
|
||||||
|
"top_p": 0.85,
|
||||||
|
"timeout_seconds": 33,
|
||||||
|
"service_tier": "flex",
|
||||||
|
"reasoning_effort": "medium",
|
||||||
|
"api_key_env": "SCRIPTORIUM_API_KEY",
|
||||||
|
"extra_params": {"provider_option":"on"}
|
||||||
|
}
|
||||||
|
}`
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(reqBody))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
if r.last.Execution == nil {
|
||||||
|
t.Fatalf("expected execution override in run request")
|
||||||
|
}
|
||||||
|
got := r.last.Execution
|
||||||
|
if got.Endpoint != "http://override/v1" ||
|
||||||
|
got.Model != "override-model" ||
|
||||||
|
got.Temperature != 0.6 ||
|
||||||
|
got.MaxTokens != 250 ||
|
||||||
|
got.TopP != 0.85 ||
|
||||||
|
got.TimeoutSeconds != 33 ||
|
||||||
|
got.ServiceTier != "flex" ||
|
||||||
|
got.ReasoningEffort != "medium" ||
|
||||||
|
got.APIKeyEnv != "SCRIPTORIUM_API_KEY" {
|
||||||
|
t.Fatalf("unexpected mapped execution target: %+v", got)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got.ExtraParams, map[string]string{"provider_option": "on"}) {
|
||||||
|
t.Fatalf("unexpected mapped extra_params: %#v", got.ExtraParams)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerResponseMetadataModelParamsIncludesAllSupportedFields(t *testing.T) {
|
||||||
|
r := &fakeRunner{result: &domain.RunResult{
|
||||||
|
Artifact: domain.Artifact{
|
||||||
|
Name: "output",
|
||||||
|
ContentType: "text/plain",
|
||||||
|
Body: []byte("ok"),
|
||||||
|
Size: 2,
|
||||||
|
Hash: "abc",
|
||||||
|
},
|
||||||
|
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
|
||||||
|
EffectiveModelParams: domain.ExecutionTarget{
|
||||||
|
Endpoint: "http://llm/v1",
|
||||||
|
Model: "gpt-test",
|
||||||
|
Temperature: 0.4,
|
||||||
|
MaxTokens: 321,
|
||||||
|
TopP: 0.7,
|
||||||
|
TimeoutSeconds: 45,
|
||||||
|
ServiceTier: "priority",
|
||||||
|
ReasoningEffort: "high",
|
||||||
|
APIKeyEnv: "SCRIPTORIUM_API_KEY",
|
||||||
|
ExtraParams: map[string]string{
|
||||||
|
"provider_option": "on",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}}
|
||||||
|
h := NewHandler(r)
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p","inputs":{"x":{"type":"file","uri":"a"}}}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
var resp map[string]any
|
||||||
|
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||||
|
t.Fatalf("invalid JSON response: %v", err)
|
||||||
|
}
|
||||||
|
metadata := resp["metadata"].(map[string]any)
|
||||||
|
params := metadata["model_params"].(map[string]any)
|
||||||
|
|
||||||
|
if params["endpoint"] != "http://llm/v1" {
|
||||||
|
t.Fatalf("unexpected endpoint: %#v", params["endpoint"])
|
||||||
|
}
|
||||||
|
if params["model"] != "gpt-test" {
|
||||||
|
t.Fatalf("unexpected model: %#v", params["model"])
|
||||||
|
}
|
||||||
|
if params["temperature"] != 0.4 {
|
||||||
|
t.Fatalf("unexpected temperature: %#v", params["temperature"])
|
||||||
|
}
|
||||||
|
if params["max_tokens"] != float64(321) {
|
||||||
|
t.Fatalf("unexpected max_tokens: %#v", params["max_tokens"])
|
||||||
|
}
|
||||||
|
if params["top_p"] != 0.7 {
|
||||||
|
t.Fatalf("unexpected top_p: %#v", params["top_p"])
|
||||||
|
}
|
||||||
|
if params["timeout_seconds"] != float64(45) {
|
||||||
|
t.Fatalf("unexpected timeout_seconds: %#v", params["timeout_seconds"])
|
||||||
|
}
|
||||||
|
if params["service_tier"] != "priority" {
|
||||||
|
t.Fatalf("unexpected service_tier: %#v", params["service_tier"])
|
||||||
|
}
|
||||||
|
if params["reasoning_effort"] != "high" {
|
||||||
|
t.Fatalf("unexpected reasoning_effort: %#v", params["reasoning_effort"])
|
||||||
|
}
|
||||||
|
if params["api_key_env"] != "SCRIPTORIUM_API_KEY" {
|
||||||
|
t.Fatalf("unexpected api_key_env: %#v", params["api_key_env"])
|
||||||
|
}
|
||||||
|
extraParams, ok := params["extra_params"].(map[string]any)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("expected extra_params object, got %#v", params["extra_params"])
|
||||||
|
}
|
||||||
|
if extraParams["provider_option"] != "on" {
|
||||||
|
t.Fatalf("unexpected extra_params.provider_option: %#v", extraParams["provider_option"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestHandlerInvalidJSON(t *testing.T) {
|
func TestHandlerInvalidJSON(t *testing.T) {
|
||||||
h := NewHandler(&fakeRunner{})
|
h := NewHandler(&fakeRunner{})
|
||||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString("{"))
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString("{"))
|
||||||
@@ -209,10 +346,10 @@ func TestHandlerUsecaseErrorMapping(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{name: "prompt not found", err: wrap(usecase.ErrProfileLoad, promptdef.ErrPromptDefinitionNotFound), status: http.StatusNotFound, code: "prompt_not_found", message: "prompt definition not found"},
|
{name: "prompt not found", err: wrap(usecase.ErrProfileLoad, promptdef.ErrPromptDefinitionNotFound), status: http.StatusNotFound, code: "prompt_not_found", message: "prompt definition not found"},
|
||||||
{name: "prompt load invalid", err: wrap(usecase.ErrProfileLoad, promptdef.ErrInvalidPromptDefinition), status: http.StatusBadRequest, code: "prompt_load_failed", message: "failed to load prompt definition"},
|
{name: "prompt load invalid", err: wrap(usecase.ErrProfileLoad, promptdef.ErrInvalidPromptDefinition), status: http.StatusBadRequest, code: "prompt_load_failed", message: "failed to load prompt definition"},
|
||||||
{name: "missing profile/default", err: wrap(usecase.ErrInvalidRequest, errors.New("profile id is required either in request or prompt default_profile")), status: http.StatusBadRequest, code: "profile_required", message: "profile_id is required when prompt default_profile is not set"},
|
{name: "missing profile/default", err: wrap(usecase.ErrInvalidRequest, usecase.ErrProfileRequired), status: http.StatusBadRequest, code: "profile_required", message: "profile_id is required when prompt default_profile is not set"},
|
||||||
{name: "profile not found", err: wrap(usecase.ErrProfileLoad, profile.ErrProfileNotFound), status: http.StatusNotFound, code: "profile_not_found", message: "execution profile not found"},
|
{name: "profile not found", err: wrap(usecase.ErrProfileLoad, profile.ErrProfileNotFound), status: http.StatusNotFound, code: "profile_not_found", message: "execution profile not found"},
|
||||||
{name: "profile invalid", err: wrap(usecase.ErrProfileLoad, profile.ErrInvalidProfile), status: http.StatusBadRequest, code: "profile_load_failed", message: "failed to load execution profile"},
|
{name: "profile invalid", err: wrap(usecase.ErrProfileLoad, profile.ErrInvalidProfile), status: http.StatusBadRequest, code: "profile_load_failed", message: "failed to load execution profile"},
|
||||||
{name: "api key env missing", err: wrap(usecase.ErrInvalidRequest, errors.New(`api key environment variable "SCRIPTORIUM_API_KEY" is not set`)), status: http.StatusBadRequest, code: "api_key_env_missing", message: "api_key_env is set but the environment variable is missing"},
|
{name: "api key env missing", err: wrap(usecase.ErrInvalidRequest, usecase.ErrAPIKeyEnvMissing), status: http.StatusBadRequest, code: "api_key_env_missing", message: "api_key_env is set but the environment variable is missing"},
|
||||||
{name: "artifact", err: wrap(usecase.ErrArtifactLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "artifact_read_failed", message: "failed to read input artifact", avoidCause: "read failed"},
|
{name: "artifact", err: wrap(usecase.ErrArtifactLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "artifact_read_failed", message: "failed to read input artifact", avoidCause: "read failed"},
|
||||||
{name: "prompt render", err: wrap(usecase.ErrPromptRender, fmt.Errorf("render failed")), status: http.StatusBadRequest, code: "prompt_render_failed", message: "failed to render prompt", avoidCause: "render failed"},
|
{name: "prompt render", err: wrap(usecase.ErrPromptRender, fmt.Errorf("render failed")), status: http.StatusBadRequest, code: "prompt_render_failed", message: "failed to render prompt", avoidCause: "render failed"},
|
||||||
{name: "llm", err: wrap(usecase.ErrLLMGenerate, fmt.Errorf("llm failed")), status: http.StatusBadGateway, code: "llm_failed", message: "model generation request failed", avoidCause: "llm failed"},
|
{name: "llm", err: wrap(usecase.ErrLLMGenerate, fmt.Errorf("llm failed")), status: http.StatusBadGateway, code: "llm_failed", message: "model generation request failed", avoidCause: "llm failed"},
|
||||||
|
|||||||
@@ -46,8 +46,8 @@ func TestCompositeReader_Read(t *testing.T) {
|
|||||||
|
|
||||||
t.Run("unsupported ref type", func(t *testing.T) {
|
t.Run("unsupported ref type", func(t *testing.T) {
|
||||||
ref := domain.ArtifactRef{
|
ref := domain.ArtifactRef{
|
||||||
Type: domain.ArtifactRefS3,
|
Type: domain.ArtifactRefType("unsupported"),
|
||||||
URI: "s3://bucket/key",
|
URI: "unsupported://bucket/key",
|
||||||
}
|
}
|
||||||
_, err := reader.Read(ctx, ref)
|
_, err := reader.Read(ctx, ref)
|
||||||
if !errors.Is(err, ErrUnsupportedRefType) {
|
if !errors.Is(err, ErrUnsupportedRefType) {
|
||||||
|
|||||||
@@ -10,7 +10,6 @@ type ArtifactRefType string
|
|||||||
const (
|
const (
|
||||||
ArtifactRefInline ArtifactRefType = "inline"
|
ArtifactRefInline ArtifactRefType = "inline"
|
||||||
ArtifactRefFile ArtifactRefType = "file"
|
ArtifactRefFile ArtifactRefType = "file"
|
||||||
ArtifactRefS3 ArtifactRefType = "s3"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// OutputFormat defines the desired format of the generated artifact.
|
// OutputFormat defines the desired format of the generated artifact.
|
||||||
@@ -146,6 +145,7 @@ type ExecutionProfile struct {
|
|||||||
MaxTokens int `yaml:"max_tokens"`
|
MaxTokens int `yaml:"max_tokens"`
|
||||||
TopP float64 `yaml:"top_p"`
|
TopP float64 `yaml:"top_p"`
|
||||||
TimeoutSeconds int `yaml:"timeout_seconds"`
|
TimeoutSeconds int `yaml:"timeout_seconds"`
|
||||||
|
ServiceTier string `yaml:"service_tier"`
|
||||||
ReasoningEffort string `yaml:"reasoning_effort"`
|
ReasoningEffort string `yaml:"reasoning_effort"`
|
||||||
APIKeyEnv string `yaml:"api_key_env"`
|
APIKeyEnv string `yaml:"api_key_env"`
|
||||||
ExtraParams map[string]string `yaml:"extra_params"`
|
ExtraParams map[string]string `yaml:"extra_params"`
|
||||||
@@ -159,6 +159,7 @@ type ExecutionTarget struct {
|
|||||||
MaxTokens int `yaml:"max_tokens" json:"max_tokens"`
|
MaxTokens int `yaml:"max_tokens" json:"max_tokens"`
|
||||||
TopP float64 `yaml:"top_p" json:"top_p"`
|
TopP float64 `yaml:"top_p" json:"top_p"`
|
||||||
TimeoutSeconds int `yaml:"timeout_seconds" json:"timeout_seconds"`
|
TimeoutSeconds int `yaml:"timeout_seconds" json:"timeout_seconds"`
|
||||||
|
ServiceTier string `yaml:"service_tier" json:"service_tier"`
|
||||||
ReasoningEffort string `yaml:"reasoning_effort" json:"reasoning_effort"`
|
ReasoningEffort string `yaml:"reasoning_effort" json:"reasoning_effort"`
|
||||||
APIKeyEnv string `yaml:"api_key_env" json:"api_key_env"`
|
APIKeyEnv string `yaml:"api_key_env" json:"api_key_env"`
|
||||||
ExtraParams map[string]string `yaml:"extra_params" json:"extra_params"`
|
ExtraParams map[string]string `yaml:"extra_params" json:"extra_params"`
|
||||||
|
|||||||
54
internal/filecatalog/catalog.go
Normal file
54
internal/filecatalog/catalog.go
Normal file
@@ -0,0 +1,54 @@
|
|||||||
|
package filecatalog
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// FindYAMLFiles returns sorted full paths for .yaml and .yml files under root.
|
||||||
|
func FindYAMLFiles(ctx context.Context, root string) ([]string, error) {
|
||||||
|
var files []string
|
||||||
|
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err()
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
if d.IsDir() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if !isYAMLFile(d.Name()) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
files = append(files, path)
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
sort.Strings(files)
|
||||||
|
return files, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// RelativePath computes a clean relative path from root to path.
|
||||||
|
func RelativePath(root string, path string) string {
|
||||||
|
rel, err := filepath.Rel(root, path)
|
||||||
|
if err != nil {
|
||||||
|
return filepath.Clean(path)
|
||||||
|
}
|
||||||
|
return filepath.Clean(rel)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stem strips .yaml or .yml from a file name.
|
||||||
|
func Stem(name string) string {
|
||||||
|
name = strings.TrimSuffix(name, ".yaml")
|
||||||
|
name = strings.TrimSuffix(name, ".yml")
|
||||||
|
return name
|
||||||
|
}
|
||||||
|
|
||||||
|
func isYAMLFile(name string) bool {
|
||||||
|
return strings.HasSuffix(name, ".yaml") || strings.HasSuffix(name, ".yml")
|
||||||
|
}
|
||||||
84
internal/filecatalog/catalog_test.go
Normal file
84
internal/filecatalog/catalog_test.go
Normal file
@@ -0,0 +1,84 @@
|
|||||||
|
package filecatalog
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestFindYAMLFilesNestedSortedAndFiltered(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
mustWriteFile(t, filepath.Join(root, "z", "prompt.yml"), "id: z")
|
||||||
|
mustWriteFile(t, filepath.Join(root, "a", "profile.yaml"), "id: a")
|
||||||
|
mustWriteFile(t, filepath.Join(root, "a", "ignore.txt"), "not yaml")
|
||||||
|
mustWriteFile(t, filepath.Join(root, "b", "ignore.yaml.bak"), "not yaml")
|
||||||
|
|
||||||
|
got, err := FindYAMLFiles(context.Background(), root)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := []string{
|
||||||
|
filepath.Join(root, "a", "profile.yaml"),
|
||||||
|
filepath.Join(root, "z", "prompt.yml"),
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("expected sorted YAML files %v, got %v", want, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFindYAMLFilesHonorsContextCancellation(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
mustWriteFile(t, filepath.Join(root, "one.yaml"), "id: one")
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
|
||||||
|
_, err := FindYAMLFiles(ctx, root)
|
||||||
|
if !errors.Is(err, context.Canceled) {
|
||||||
|
t.Fatalf("expected context.Canceled, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRelativePathNested(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
path := filepath.Join(root, "nested", "profiles", "local.yaml")
|
||||||
|
got := RelativePath(root, path)
|
||||||
|
want := filepath.Join("nested", "profiles", "local.yaml")
|
||||||
|
if got != want {
|
||||||
|
t.Fatalf("expected relative path %q, got %q", want, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStemStripsYAMLExtensions(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
in string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "yaml", in: "prompt.yaml", want: "prompt"},
|
||||||
|
{name: "yml", in: "profile.yml", want: "profile"},
|
||||||
|
{name: "other", in: "file.txt", want: "file.txt"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
if got := Stem(tc.in); got != tc.want {
|
||||||
|
t.Fatalf("expected %q, got %q", tc.want, got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustWriteFile(t *testing.T, path string, content string) {
|
||||||
|
t.Helper()
|
||||||
|
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||||
|
t.Fatalf("failed to create directory: %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||||
|
t.Fatalf("failed to write file %q: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -106,6 +106,9 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
|
|||||||
fmt.Fprintf(&b, " max_tokens: %d\n", target.MaxTokens)
|
fmt.Fprintf(&b, " max_tokens: %d\n", target.MaxTokens)
|
||||||
fmt.Fprintf(&b, " top_p: %g\n", target.TopP)
|
fmt.Fprintf(&b, " top_p: %g\n", target.TopP)
|
||||||
fmt.Fprintf(&b, " timeout_seconds: %d\n", target.TimeoutSeconds)
|
fmt.Fprintf(&b, " timeout_seconds: %d\n", target.TimeoutSeconds)
|
||||||
|
if target.ServiceTier != "" {
|
||||||
|
fmt.Fprintf(&b, " service_tier: %s\n", target.ServiceTier)
|
||||||
|
}
|
||||||
if target.ReasoningEffort != "" {
|
if target.ReasoningEffort != "" {
|
||||||
fmt.Fprintf(&b, " reasoning_effort: %s\n", target.ReasoningEffort)
|
fmt.Fprintf(&b, " reasoning_effort: %s\n", target.ReasoningEffort)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -28,6 +28,7 @@ func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
|
|||||||
"max_tokens: 256",
|
"max_tokens: 256",
|
||||||
"top_p: 0.8",
|
"top_p: 0.8",
|
||||||
"timeout_seconds: 45",
|
"timeout_seconds: 45",
|
||||||
|
"service_tier: priority",
|
||||||
"reasoning_effort: medium",
|
"reasoning_effort: medium",
|
||||||
"api_key_env: SCRIPTORIUM_API_KEY",
|
"api_key_env: SCRIPTORIUM_API_KEY",
|
||||||
"prompt_hash: prompt-hash",
|
"prompt_hash: prompt-hash",
|
||||||
@@ -168,6 +169,7 @@ func samplePreparedRun() *domain.PreparedRun {
|
|||||||
MaxTokens: 256,
|
MaxTokens: 256,
|
||||||
TopP: 0.8,
|
TopP: 0.8,
|
||||||
TimeoutSeconds: 45,
|
TimeoutSeconds: 45,
|
||||||
|
ServiceTier: "priority",
|
||||||
ReasoningEffort: "medium",
|
ReasoningEffort: "medium",
|
||||||
APIKeyEnv: "SCRIPTORIUM_API_KEY",
|
APIKeyEnv: "SCRIPTORIUM_API_KEY",
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -75,14 +75,6 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
|||||||
return nil, fmt.Errorf("%w: timeout_seconds must be greater than or equal to 0", ErrInvalidRequest)
|
return nil, fmt.Errorf("%w: timeout_seconds must be greater than or equal to 0", ErrInvalidRequest)
|
||||||
}
|
}
|
||||||
|
|
||||||
model := strings.TrimSpace(req.Target.Model)
|
|
||||||
if model == "" {
|
|
||||||
model = strings.TrimSpace(c.defaultModel)
|
|
||||||
}
|
|
||||||
if model == "" {
|
|
||||||
return nil, fmt.Errorf("%w: model is required", ErrInvalidRequest)
|
|
||||||
}
|
|
||||||
|
|
||||||
endpoint := strings.TrimSpace(req.Target.Endpoint)
|
endpoint := strings.TrimSpace(req.Target.Endpoint)
|
||||||
if endpoint == "" {
|
if endpoint == "" {
|
||||||
endpoint = c.baseURL
|
endpoint = c.baseURL
|
||||||
@@ -92,33 +84,9 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
|||||||
}
|
}
|
||||||
endpoint = strings.TrimRight(endpoint, "/") + defaults.OpenAIChatCompletionsPath
|
endpoint = strings.TrimRight(endpoint, "/") + defaults.OpenAIChatCompletionsPath
|
||||||
|
|
||||||
wireReq := openAIChatRequest{
|
wireReq, err := openAIChatRequestFromGenerateRequest(req, c.defaultModel)
|
||||||
Model: model,
|
if err != nil {
|
||||||
}
|
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
|
||||||
|
|
||||||
wireReq.Messages = make([]openAIChatMessage, 0, len(req.Prompt.Messages))
|
|
||||||
for _, msg := range req.Prompt.Messages {
|
|
||||||
wireReq.Messages = append(wireReq.Messages, openAIChatMessage{
|
|
||||||
Role: msg.Role,
|
|
||||||
Content: msg.Content,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
if req.Target.Temperature != 0 {
|
|
||||||
wireReq.Temperature = &req.Target.Temperature
|
|
||||||
}
|
|
||||||
if req.Target.MaxTokens != 0 {
|
|
||||||
wireReq.MaxTokens = &req.Target.MaxTokens
|
|
||||||
}
|
|
||||||
if req.Target.TopP != 0 {
|
|
||||||
wireReq.TopP = &req.Target.TopP
|
|
||||||
}
|
|
||||||
if req.StructuredOutput != nil {
|
|
||||||
responseFormat, err := toOpenAIResponseFormat(req.StructuredOutput)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
|
|
||||||
}
|
|
||||||
wireReq.ResponseFormat = responseFormat
|
|
||||||
}
|
}
|
||||||
|
|
||||||
payload, err := json.Marshal(wireReq)
|
payload, err := json.Marshal(wireReq)
|
||||||
@@ -187,12 +155,57 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func openAIChatRequestFromGenerateRequest(req domain.GenerateRequest, defaultModel string) (openAIChatRequest, error) {
|
||||||
|
model := strings.TrimSpace(req.Target.Model)
|
||||||
|
if model == "" {
|
||||||
|
model = strings.TrimSpace(defaultModel)
|
||||||
|
}
|
||||||
|
if model == "" {
|
||||||
|
return openAIChatRequest{}, errors.New("model is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
wireReq := openAIChatRequest{
|
||||||
|
Model: model,
|
||||||
|
}
|
||||||
|
|
||||||
|
wireReq.Messages = make([]openAIChatMessage, 0, len(req.Prompt.Messages))
|
||||||
|
for _, msg := range req.Prompt.Messages {
|
||||||
|
wireReq.Messages = append(wireReq.Messages, openAIChatMessage{
|
||||||
|
Role: msg.Role,
|
||||||
|
Content: msg.Content,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
if req.Target.Temperature != 0 {
|
||||||
|
wireReq.Temperature = &req.Target.Temperature
|
||||||
|
}
|
||||||
|
if req.Target.MaxTokens != 0 {
|
||||||
|
wireReq.MaxTokens = &req.Target.MaxTokens
|
||||||
|
}
|
||||||
|
if req.Target.TopP != 0 {
|
||||||
|
wireReq.TopP = &req.Target.TopP
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(req.Target.ServiceTier) != "" {
|
||||||
|
wireReq.ServiceTier = req.Target.ServiceTier
|
||||||
|
}
|
||||||
|
if req.StructuredOutput != nil {
|
||||||
|
responseFormat, err := toOpenAIResponseFormat(req.StructuredOutput)
|
||||||
|
if err != nil {
|
||||||
|
return openAIChatRequest{}, err
|
||||||
|
}
|
||||||
|
wireReq.ResponseFormat = responseFormat
|
||||||
|
}
|
||||||
|
|
||||||
|
return wireReq, nil
|
||||||
|
}
|
||||||
|
|
||||||
type openAIChatRequest struct {
|
type openAIChatRequest struct {
|
||||||
Model string `json:"model"`
|
Model string `json:"model"`
|
||||||
Messages []openAIChatMessage `json:"messages"`
|
Messages []openAIChatMessage `json:"messages"`
|
||||||
Temperature *float64 `json:"temperature,omitempty"`
|
Temperature *float64 `json:"temperature,omitempty"`
|
||||||
MaxTokens *int `json:"max_tokens,omitempty"`
|
MaxTokens *int `json:"max_tokens,omitempty"`
|
||||||
TopP *float64 `json:"top_p,omitempty"`
|
TopP *float64 `json:"top_p,omitempty"`
|
||||||
|
ServiceTier string `json:"service_tier,omitempty"`
|
||||||
ResponseFormat *openAIResponseFormat `json:"response_format,omitempty"`
|
ResponseFormat *openAIResponseFormat `json:"response_format,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -61,6 +61,7 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
|
|||||||
Temperature: 0.4,
|
Temperature: 0.4,
|
||||||
MaxTokens: 123,
|
MaxTokens: 123,
|
||||||
TopP: 0.7,
|
TopP: 0.7,
|
||||||
|
ServiceTier: "priority",
|
||||||
APIKeyEnv: "SCRIPTORIUM_TEST_API_KEY",
|
APIKeyEnv: "SCRIPTORIUM_TEST_API_KEY",
|
||||||
},
|
},
|
||||||
StructuredOutput: &domain.StructuredOutputSpec{
|
StructuredOutput: &domain.StructuredOutputSpec{
|
||||||
@@ -95,6 +96,18 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
|
|||||||
if got, ok := obs.Body["model"].(string); !ok || got != "gpt-test" {
|
if got, ok := obs.Body["model"].(string); !ok || got != "gpt-test" {
|
||||||
t.Fatalf("unexpected model payload: %#v", obs.Body["model"])
|
t.Fatalf("unexpected model payload: %#v", obs.Body["model"])
|
||||||
}
|
}
|
||||||
|
if got, ok := obs.Body["temperature"].(float64); !ok || got != 0.4 {
|
||||||
|
t.Fatalf("unexpected temperature payload: %#v", obs.Body["temperature"])
|
||||||
|
}
|
||||||
|
if got, ok := obs.Body["max_tokens"].(float64); !ok || got != 123 {
|
||||||
|
t.Fatalf("unexpected max_tokens payload: %#v", obs.Body["max_tokens"])
|
||||||
|
}
|
||||||
|
if got, ok := obs.Body["top_p"].(float64); !ok || got != 0.7 {
|
||||||
|
t.Fatalf("unexpected top_p payload: %#v", obs.Body["top_p"])
|
||||||
|
}
|
||||||
|
if got, ok := obs.Body["service_tier"].(string); !ok || got != "priority" {
|
||||||
|
t.Fatalf("unexpected service_tier payload: %#v", obs.Body["service_tier"])
|
||||||
|
}
|
||||||
|
|
||||||
msgs, ok := obs.Body["messages"].([]any)
|
msgs, ok := obs.Body["messages"].([]any)
|
||||||
if !ok || len(msgs) != 2 {
|
if !ok || len(msgs) != 2 {
|
||||||
@@ -157,6 +170,46 @@ func TestOpenAICompatibleClientOmitsResponseFormatWhenNoStructuredOutput(t *test
|
|||||||
if _, exists := observedBody["response_format"]; exists {
|
if _, exists := observedBody["response_format"]; exists {
|
||||||
t.Fatalf("expected response_format omitted, got %#v", observedBody["response_format"])
|
t.Fatalf("expected response_format omitted, got %#v", observedBody["response_format"])
|
||||||
}
|
}
|
||||||
|
if _, exists := observedBody["service_tier"]; exists {
|
||||||
|
t.Fatalf("expected service_tier omitted, got %#v", observedBody["service_tier"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOpenAICompatibleClientOmitsReasoningEffortAndExtraParams(t *testing.T) {
|
||||||
|
var observedBody map[string]any
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
defer r.Body.Close()
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||||
|
t.Fatalf("failed to decode request body: %v", err)
|
||||||
|
}
|
||||||
|
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||||
|
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||||
|
Target: domain.ExecutionTarget{
|
||||||
|
Model: "model",
|
||||||
|
ReasoningEffort: "high",
|
||||||
|
ExtraParams: map[string]string{
|
||||||
|
"provider_option": "on",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if _, exists := observedBody["reasoning_effort"]; exists {
|
||||||
|
t.Fatalf("expected reasoning_effort omitted, got %#v", observedBody["reasoning_effort"])
|
||||||
|
}
|
||||||
|
if _, exists := observedBody["extra_params"]; exists {
|
||||||
|
t.Fatalf("expected extra_params omitted, got %#v", observedBody["extra_params"])
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestOpenAICompatibleClientNoAuthorizationHeaderWhenNoAPIKey(t *testing.T) {
|
func TestOpenAICompatibleClientNoAuthorizationHeaderWhenNoAPIKey(t *testing.T) {
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/filecatalog"
|
||||||
"gopkg.in/yaml.v3"
|
"gopkg.in/yaml.v3"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -33,40 +34,39 @@ func (r *filesystemRepository) GetProfile(ctx context.Context, id string) (*doma
|
|||||||
return nil, fmt.Errorf("%w: profile id is required", ErrInvalidProfile)
|
return nil, fmt.Errorf("%w: profile id is required", ErrInvalidProfile)
|
||||||
}
|
}
|
||||||
|
|
||||||
files, err := os.ReadDir(r.dir)
|
files, err := filecatalog.FindYAMLFiles(ctx, r.dir)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to read profile directory: %w", err)
|
return nil, fmt.Errorf("failed to read profile directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, file := range files {
|
var matches []profileMatch
|
||||||
|
for _, fullPath := range files {
|
||||||
select {
|
select {
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
return nil, ctx.Err()
|
return nil, ctx.Err()
|
||||||
default:
|
default:
|
||||||
}
|
}
|
||||||
|
|
||||||
if file.IsDir() || (!strings.HasSuffix(file.Name(), ".yaml") && !strings.HasSuffix(file.Name(), ".yml")) {
|
relPath := filecatalog.RelativePath(r.dir, fullPath)
|
||||||
continue
|
fileMatch := filecatalog.Stem(filepath.Base(fullPath)) == id
|
||||||
}
|
|
||||||
|
|
||||||
fullPath := filepath.Join(r.dir, file.Name())
|
|
||||||
data, err := os.ReadFile(fullPath)
|
data, err := os.ReadFile(fullPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to read profile file %s: %w", file.Name(), err)
|
return nil, fmt.Errorf("failed to read profile file %s: %w", relPath, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
var prof domain.ExecutionProfile
|
var prof domain.ExecutionProfile
|
||||||
decoder := yaml.NewDecoder(bytes.NewReader(data))
|
decoder := yaml.NewDecoder(bytes.NewReader(data))
|
||||||
decoder.KnownFields(true)
|
decoder.KnownFields(true)
|
||||||
if err := decoder.Decode(&prof); err != nil {
|
if err := decoder.Decode(&prof); err != nil {
|
||||||
|
idMatch := fileMatch || profileFileHasID(data, id)
|
||||||
if strings.Contains(err.Error(), "field api_key not found") {
|
if strings.Contains(err.Error(), "field api_key not found") {
|
||||||
if strings.TrimSuffix(strings.TrimSuffix(file.Name(), ".yaml"), ".yml") == id {
|
if idMatch {
|
||||||
return nil, fmt.Errorf("%w: %s", ErrRawAPIKeyNotAllowed, file.Name())
|
return nil, fmt.Errorf("%w: %s", ErrRawAPIKeyNotAllowed, relPath)
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if strings.TrimSuffix(strings.TrimSuffix(file.Name(), ".yaml"), ".yml") == id {
|
if idMatch {
|
||||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, file.Name(), err)
|
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -76,16 +76,46 @@ func (r *filesystemRepository) GetProfile(ctx context.Context, id string) (*doma
|
|||||||
}
|
}
|
||||||
if err := validateProfile(&prof); err != nil {
|
if err := validateProfile(&prof); err != nil {
|
||||||
if errors.Is(err, ErrRawAPIKeyNotAllowed) {
|
if errors.Is(err, ErrRawAPIKeyNotAllowed) {
|
||||||
return nil, fmt.Errorf("%w: %s", err, file.Name())
|
return nil, fmt.Errorf("%w: %s", err, relPath)
|
||||||
}
|
}
|
||||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, file.Name(), err)
|
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, relPath, err)
|
||||||
}
|
}
|
||||||
return &prof, nil
|
matches = append(matches, profileMatch{
|
||||||
|
profile: &prof,
|
||||||
|
path: relPath,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(matches) > 1 {
|
||||||
|
paths := make([]string, 0, len(matches))
|
||||||
|
for _, match := range matches {
|
||||||
|
paths = append(paths, match.path)
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("%w: duplicate execution profile id %q found in: %s", ErrInvalidProfile, id, strings.Join(paths, ", "))
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(matches) == 1 {
|
||||||
|
return matches[0].profile, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil, ErrProfileNotFound
|
return nil, ErrProfileNotFound
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type profileMatch struct {
|
||||||
|
profile *domain.ExecutionProfile
|
||||||
|
path string
|
||||||
|
}
|
||||||
|
|
||||||
|
func profileFileHasID(data []byte, id string) bool {
|
||||||
|
var raw struct {
|
||||||
|
ID string `yaml:"id"`
|
||||||
|
}
|
||||||
|
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&raw); err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(raw.ID) == id
|
||||||
|
}
|
||||||
|
|
||||||
func validateProfile(p *domain.ExecutionProfile) error {
|
func validateProfile(p *domain.ExecutionProfile) error {
|
||||||
if strings.TrimSpace(p.ID) == "" {
|
if strings.TrimSpace(p.ID) == "" {
|
||||||
return errors.New("id is required")
|
return errors.New("id is required")
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -58,6 +59,78 @@ func TestFilesystemRepository_GetProfile(t *testing.T) {
|
|||||||
if p.ReasoningEffort != "medium" {
|
if p.ReasoningEffort != "medium" {
|
||||||
t.Fatalf("unexpected reasoning_effort: %q", p.ReasoningEffort)
|
t.Fatalf("unexpected reasoning_effort: %q", p.ReasoningEffort)
|
||||||
}
|
}
|
||||||
|
if p.ServiceTier != "priority" {
|
||||||
|
t.Fatalf("unexpected service_tier: %q", p.ServiceTier)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("valid nested profile", func(t *testing.T) {
|
||||||
|
nestedDir := filepath.Join(tmpDir, "local")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writeProfileTestFile(t, filepath.Join(nestedDir, "nested-local.yaml"), `
|
||||||
|
id: nested-local
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: nested-model
|
||||||
|
temperature: 0.1
|
||||||
|
`)
|
||||||
|
|
||||||
|
p, err := repo.GetProfile(ctx, "nested-local")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if p.Model != "nested-model" {
|
||||||
|
t.Fatalf("unexpected model: %q", p.Model)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("duplicate profile IDs fail as ambiguous", func(t *testing.T) {
|
||||||
|
writeProfileTestFile(t, filepath.Join(tmpDir, "duplicate-profile-a.yaml"), `
|
||||||
|
id: duplicate-profile
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: first-model
|
||||||
|
`)
|
||||||
|
nestedDir := filepath.Join(tmpDir, "duplicates")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writeProfileTestFile(t, filepath.Join(nestedDir, "duplicate-profile-b.yaml"), `
|
||||||
|
id: duplicate-profile
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: second-model
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetProfile(ctx, "duplicate-profile")
|
||||||
|
if !errors.Is(err, ErrInvalidProfile) {
|
||||||
|
t.Fatalf("expected duplicate profile to return ErrInvalidProfile, got %v", err)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"duplicate execution profile id", "duplicate-profile-a.yaml", filepath.Join("duplicates", "duplicate-profile-b.yaml")} {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("expected error to contain %q, got %v", want, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("nested raw api_key rejected for likely target file", func(t *testing.T) {
|
||||||
|
nestedDir := filepath.Join(tmpDir, "secure")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writeProfileTestFile(t, filepath.Join(nestedDir, "not_named_like_id.yaml"), `
|
||||||
|
id: nested_raw_api_key
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
model: m
|
||||||
|
api_key: secret
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetProfile(ctx, "nested_raw_api_key")
|
||||||
|
if !errors.Is(err, ErrRawAPIKeyNotAllowed) {
|
||||||
|
t.Fatalf("expected ErrRawAPIKeyNotAllowed, got %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), filepath.Join("secure", "not_named_like_id.yaml")) {
|
||||||
|
t.Fatalf("expected nested path in error, got %v", err)
|
||||||
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
t.Run("invalid yaml", func(t *testing.T) {
|
t.Run("invalid yaml", func(t *testing.T) {
|
||||||
@@ -109,3 +182,10 @@ func TestFilesystemRepository_GetProfile(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writeProfileTestFile(t *testing.T, path string, content string) {
|
||||||
|
t.Helper()
|
||||||
|
if err := os.WriteFile(path, []byte(strings.TrimLeft(content, "\n")), 0o644); err != nil {
|
||||||
|
t.Fatalf("failed to write profile test file %q: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ id: local-secure
|
|||||||
endpoint: http://localhost:8000/v1
|
endpoint: http://localhost:8000/v1
|
||||||
model: gpt-4o-mini
|
model: gpt-4o-mini
|
||||||
api_key_env: SCRIPTORIUM_API_KEY
|
api_key_env: SCRIPTORIUM_API_KEY
|
||||||
|
service_tier: priority
|
||||||
reasoning_effort: medium
|
reasoning_effort: medium
|
||||||
extra_params:
|
extra_params:
|
||||||
provider: local
|
provider: local
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium/internal/filecatalog"
|
||||||
"gopkg.in/yaml.v3"
|
"gopkg.in/yaml.v3"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -62,29 +63,26 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
|
|||||||
return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition)
|
return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition)
|
||||||
}
|
}
|
||||||
|
|
||||||
files, err := os.ReadDir(r.dir)
|
files, err := filecatalog.FindYAMLFiles(ctx, r.dir)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to read prompt definition directory: %w", err)
|
return nil, fmt.Errorf("failed to read prompt definition directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, file := range files {
|
var matches []promptDefinitionMatch
|
||||||
|
for _, fullPath := range files {
|
||||||
select {
|
select {
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
return nil, ctx.Err()
|
return nil, ctx.Err()
|
||||||
default:
|
default:
|
||||||
}
|
}
|
||||||
|
|
||||||
if file.IsDir() || !isYAMLFile(file.Name()) {
|
relPath := filecatalog.RelativePath(r.dir, fullPath)
|
||||||
continue
|
fileMatch := filecatalog.Stem(filepath.Base(fullPath)) == id
|
||||||
}
|
|
||||||
|
|
||||||
fullPath := filepath.Join(r.dir, file.Name())
|
|
||||||
fileMatch := promptIDFromFileName(file.Name()) == id
|
|
||||||
|
|
||||||
raw, err := loadPromptDefinitionFile(fullPath)
|
raw, err := loadPromptDefinitionFile(fullPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if fileMatch {
|
if fileMatch || promptDefinitionFileHasID(fullPath, id) {
|
||||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, file.Name(), err)
|
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -92,7 +90,7 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
|
|||||||
def, err := normalizePromptDefinition(raw, fullPath)
|
def, err := normalizePromptDefinition(raw, fullPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if fileMatch || strings.TrimSpace(raw.ID) == id {
|
if fileMatch || strings.TrimSpace(raw.ID) == id {
|
||||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, file.Name(), err)
|
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, relPath, err)
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -103,12 +101,35 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
|
|||||||
if version != "" && def.Version != version {
|
if version != "" && def.Version != version {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
return def, nil
|
matches = append(matches, promptDefinitionMatch{
|
||||||
|
def: def,
|
||||||
|
path: relPath,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(matches) > 1 {
|
||||||
|
paths := make([]string, 0, len(matches))
|
||||||
|
for _, match := range matches {
|
||||||
|
paths = append(paths, match.path)
|
||||||
|
}
|
||||||
|
if version != "" {
|
||||||
|
return nil, fmt.Errorf("%w: duplicate prompt definition id %q version %q found in: %s", ErrInvalidPromptDefinition, id, version, strings.Join(paths, ", "))
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("%w: duplicate prompt definition id %q found in: %s", ErrInvalidPromptDefinition, id, strings.Join(paths, ", "))
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(matches) == 1 {
|
||||||
|
return matches[0].def, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil, ErrPromptDefinitionNotFound
|
return nil, ErrPromptDefinitionNotFound
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type promptDefinitionMatch struct {
|
||||||
|
def *domain.PromptDefinition
|
||||||
|
path string
|
||||||
|
}
|
||||||
|
|
||||||
func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
|
func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
|
||||||
data, err := os.ReadFile(path)
|
data, err := os.ReadFile(path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -124,6 +145,20 @@ func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
|
|||||||
return &raw, nil
|
return &raw, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func promptDefinitionFileHasID(path string, id string) bool {
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
var raw struct {
|
||||||
|
ID string `yaml:"id"`
|
||||||
|
}
|
||||||
|
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&raw); err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(raw.ID) == id
|
||||||
|
}
|
||||||
|
|
||||||
func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*domain.PromptDefinition, error) {
|
func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*domain.PromptDefinition, error) {
|
||||||
if raw == nil {
|
if raw == nil {
|
||||||
return nil, errors.New("prompt definition is nil")
|
return nil, errors.New("prompt definition is nil")
|
||||||
@@ -239,16 +274,6 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func isYAMLFile(name string) bool {
|
|
||||||
return strings.HasSuffix(name, ".yaml") || strings.HasSuffix(name, ".yml")
|
|
||||||
}
|
|
||||||
|
|
||||||
func promptIDFromFileName(name string) string {
|
|
||||||
name = strings.TrimSuffix(name, ".yaml")
|
|
||||||
name = strings.TrimSuffix(name, ".yml")
|
|
||||||
return name
|
|
||||||
}
|
|
||||||
|
|
||||||
func isValidOutputFormat(f domain.OutputFormat) bool {
|
func isValidOutputFormat(f domain.OutputFormat) bool {
|
||||||
switch f {
|
switch f {
|
||||||
case domain.FormatText, domain.FormatMarkdown, domain.FormatJSON:
|
case domain.FormatText, domain.FormatMarkdown, domain.FormatJSON:
|
||||||
|
|||||||
@@ -68,6 +68,39 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run("valid nested file-backed prompt resolves content file relative to nested YAML", func(t *testing.T) {
|
||||||
|
nestedDir := filepath.Join(tmpDir, "dnd", "recap")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "nested_recap.yaml"), `
|
||||||
|
id: nested-recap
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content_file: ./nested_recap.user.tmpl
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "nested_recap.user.tmpl"), `Nested recap: {{input "transcript"}}`)
|
||||||
|
|
||||||
|
p, err := repo.GetPromptDefinition(ctx, "nested-recap", "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if len(p.Templates) != 1 {
|
||||||
|
t.Fatalf("expected one template, got %d", len(p.Templates))
|
||||||
|
}
|
||||||
|
if !strings.Contains(p.Templates[0].Content, "Nested recap") {
|
||||||
|
t.Fatalf("expected nested content file body, got %q", p.Templates[0].Content)
|
||||||
|
}
|
||||||
|
if !strings.Contains(p.Templates[0].ContentFile, filepath.Join("dnd", "recap", "nested_recap.user.tmpl")) {
|
||||||
|
t.Fatalf("expected nested content file path, got %q", p.Templates[0].ContentFile)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
t.Run("prompt with default_profile", func(t *testing.T) {
|
t.Run("prompt with default_profile", func(t *testing.T) {
|
||||||
p, err := repo.GetPromptDefinition(ctx, "with-default-profile", "")
|
p, err := repo.GetPromptDefinition(ctx, "with-default-profile", "")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -84,6 +117,124 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run("duplicate prompt IDs fail as ambiguous", func(t *testing.T) {
|
||||||
|
writePromptTestFile(t, filepath.Join(tmpDir, "duplicate_a.yaml"), `
|
||||||
|
id: duplicate-prompt
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: First duplicate.
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
nestedDir := filepath.Join(tmpDir, "nested")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "duplicate_b.yaml"), `
|
||||||
|
id: duplicate-prompt
|
||||||
|
version: "2.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: Second duplicate.
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetPromptDefinition(ctx, "duplicate-prompt", "")
|
||||||
|
if !errors.Is(err, ErrInvalidPromptDefinition) {
|
||||||
|
t.Fatalf("expected duplicate prompt to return ErrInvalidPromptDefinition, got %v", err)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"duplicate prompt definition id", "duplicate_a.yaml", filepath.Join("nested", "duplicate_b.yaml")} {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("expected error to contain %q, got %v", want, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("duplicate prompt ID and requested version fails as ambiguous", func(t *testing.T) {
|
||||||
|
writePromptTestFile(t, filepath.Join(tmpDir, "version_duplicate_a.yaml"), `
|
||||||
|
id: duplicate-version-prompt
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: First duplicate version.
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
nestedDir := filepath.Join(tmpDir, "versioned")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "version_duplicate_b.yaml"), `
|
||||||
|
id: duplicate-version-prompt
|
||||||
|
version: "1.0.0"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: Second duplicate version.
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetPromptDefinition(ctx, "duplicate-version-prompt", "1.0.0")
|
||||||
|
if !errors.Is(err, ErrInvalidPromptDefinition) {
|
||||||
|
t.Fatalf("expected duplicate prompt version to return ErrInvalidPromptDefinition, got %v", err)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"duplicate prompt definition id", "version \"1.0.0\"", "version_duplicate_a.yaml", filepath.Join("versioned", "version_duplicate_b.yaml")} {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("expected error to contain %q, got %v", want, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("non-matching malformed nested prompt is ignored for not found lookup", func(t *testing.T) {
|
||||||
|
nestedDir := filepath.Join(tmpDir, "broken")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "unrelated.yaml"), "id: [")
|
||||||
|
|
||||||
|
_, err := repo.GetPromptDefinition(ctx, "does-not-exist-even-with-broken-nested-file", "")
|
||||||
|
if !errors.Is(err, ErrPromptDefinitionNotFound) {
|
||||||
|
t.Fatalf("expected ErrPromptDefinitionNotFound, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("strict decode failure in nested prompt matches by YAML ID", func(t *testing.T) {
|
||||||
|
nestedDir := filepath.Join(tmpDir, "strict")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePromptTestFile(t, filepath.Join(nestedDir, "not_named_like_id.yaml"), `
|
||||||
|
id: nested-strict-error
|
||||||
|
version: "1.0.0"
|
||||||
|
unknown_field: true
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: Invalid because of unknown field.
|
||||||
|
output:
|
||||||
|
format: markdown
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 0
|
||||||
|
`)
|
||||||
|
|
||||||
|
_, err := repo.GetPromptDefinition(ctx, "nested-strict-error", "")
|
||||||
|
if !errors.Is(err, ErrInvalidYAML) {
|
||||||
|
t.Fatalf("expected ErrInvalidYAML, got %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), filepath.Join("strict", "not_named_like_id.yaml")) {
|
||||||
|
t.Fatalf("expected nested path in error, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
t.Run("version lookup", func(t *testing.T) {
|
t.Run("version lookup", func(t *testing.T) {
|
||||||
_, err := repo.GetPromptDefinition(ctx, "valid-inline", "9.9.9")
|
_, err := repo.GetPromptDefinition(ctx, "valid-inline", "9.9.9")
|
||||||
if !errors.Is(err, ErrPromptDefinitionNotFound) {
|
if !errors.Is(err, ErrPromptDefinitionNotFound) {
|
||||||
@@ -131,6 +282,13 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writePromptTestFile(t *testing.T, path string, content string) {
|
||||||
|
t.Helper()
|
||||||
|
if err := os.WriteFile(path, []byte(strings.TrimLeft(content, "\n")), 0o644); err != nil {
|
||||||
|
t.Fatalf("failed to write prompt test file %q: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func copyTree(src, dst string) error {
|
func copyTree(src, dst string) error {
|
||||||
return filepath.WalkDir(src, func(path string, d fs.DirEntry, err error) error {
|
return filepath.WalkDir(src, func(path string, d fs.DirEntry, err error) error {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -35,9 +35,9 @@ func TestRunnerIntegrationWithPromptAndProfileFixturesAndValidation(t *testing.T
|
|||||||
t.Fatalf("failed to resolve repo root: %v", err)
|
t.Fatalf("failed to resolve repo root: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
promptsDir := filepath.Join(root, "prompts")
|
promptsDir := filepath.Join(root, "examples", "prompts")
|
||||||
profilesDir := filepath.Join(root, "profiles")
|
profilesDir := filepath.Join(root, "examples", "profiles")
|
||||||
schemasDir := filepath.Join(root, "schemas")
|
schemasDir := filepath.Join(root, "examples", "schemas")
|
||||||
fixturesDir := filepath.Join(root, "examples", "fixtures")
|
fixturesDir := filepath.Join(root, "examples", "fixtures")
|
||||||
t.Setenv("SCRIPTORIUM_API_KEY", "test-key")
|
t.Setenv("SCRIPTORIUM_API_KEY", "test-key")
|
||||||
|
|
||||||
|
|||||||
@@ -24,12 +24,14 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
ErrInvalidRequest = errors.New("invalid run request")
|
ErrInvalidRequest = errors.New("invalid run request")
|
||||||
ErrProfileLoad = errors.New("failed to load prompt definition")
|
ErrProfileRequired = errors.New("profile selection is required")
|
||||||
ErrArtifactLoad = errors.New("failed to load artifact")
|
ErrAPIKeyEnvMissing = errors.New("api_key_env points to an unset environment variable")
|
||||||
ErrPromptRender = errors.New("failed to render prompt")
|
ErrProfileLoad = errors.New("failed to load prompt definition")
|
||||||
ErrLLMGenerate = errors.New("failed to generate output")
|
ErrArtifactLoad = errors.New("failed to load artifact")
|
||||||
ErrValidation = errors.New("failed to validate output")
|
ErrPromptRender = errors.New("failed to render prompt")
|
||||||
|
ErrLLMGenerate = errors.New("failed to generate output")
|
||||||
|
ErrValidation = errors.New("failed to validate output")
|
||||||
)
|
)
|
||||||
|
|
||||||
// Runner executes the Scriptorium core use case.
|
// Runner executes the Scriptorium core use case.
|
||||||
@@ -177,7 +179,7 @@ func (r *Runner) Prepare(ctx context.Context, req domain.RunRequest) (*domain.Pr
|
|||||||
selectedProfileID = strings.TrimSpace(def.DefaultProfile)
|
selectedProfileID = strings.TrimSpace(def.DefaultProfile)
|
||||||
}
|
}
|
||||||
if selectedProfileID == "" {
|
if selectedProfileID == "" {
|
||||||
return nil, fmt.Errorf("%w: profile id is required either in request or prompt default_profile", ErrInvalidRequest)
|
return nil, fmt.Errorf("%w: %w: profile id is required either in request or prompt default_profile", ErrInvalidRequest, ErrProfileRequired)
|
||||||
}
|
}
|
||||||
|
|
||||||
execProfile, err := r.profiles.GetProfile(ctx, selectedProfileID)
|
execProfile, err := r.profiles.GetProfile(ctx, selectedProfileID)
|
||||||
@@ -342,6 +344,9 @@ func mergeExecutionTarget(base domain.ExecutionTarget, override domain.Execution
|
|||||||
if override.TimeoutSeconds != 0 {
|
if override.TimeoutSeconds != 0 {
|
||||||
out.TimeoutSeconds = override.TimeoutSeconds
|
out.TimeoutSeconds = override.TimeoutSeconds
|
||||||
}
|
}
|
||||||
|
if strings.TrimSpace(override.ServiceTier) != "" {
|
||||||
|
out.ServiceTier = override.ServiceTier
|
||||||
|
}
|
||||||
if strings.TrimSpace(override.ReasoningEffort) != "" {
|
if strings.TrimSpace(override.ReasoningEffort) != "" {
|
||||||
out.ReasoningEffort = override.ReasoningEffort
|
out.ReasoningEffort = override.ReasoningEffort
|
||||||
}
|
}
|
||||||
@@ -373,7 +378,7 @@ func validateAPIKeyEnv(apiKeyEnv string) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(os.Getenv(envName)) == "" {
|
if strings.TrimSpace(os.Getenv(envName)) == "" {
|
||||||
return fmt.Errorf("api key environment variable %q is not set", envName)
|
return fmt.Errorf("%w: api key environment variable %q is not set", ErrAPIKeyEnvMissing, envName)
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -396,6 +401,7 @@ func executionProfileToTarget(p *domain.ExecutionProfile) domain.ExecutionTarget
|
|||||||
MaxTokens: p.MaxTokens,
|
MaxTokens: p.MaxTokens,
|
||||||
TopP: p.TopP,
|
TopP: p.TopP,
|
||||||
TimeoutSeconds: p.TimeoutSeconds,
|
TimeoutSeconds: p.TimeoutSeconds,
|
||||||
|
ServiceTier: p.ServiceTier,
|
||||||
ReasoningEffort: p.ReasoningEffort,
|
ReasoningEffort: p.ReasoningEffort,
|
||||||
APIKeyEnv: p.APIKeyEnv,
|
APIKeyEnv: p.APIKeyEnv,
|
||||||
ExtraParams: cp,
|
ExtraParams: cp,
|
||||||
|
|||||||
@@ -234,6 +234,9 @@ func TestRunnerPrepareMissingExplicitProfileAndMissingDefaultProfileFails(t *tes
|
|||||||
if !errors.Is(err, ErrInvalidRequest) {
|
if !errors.Is(err, ErrInvalidRequest) {
|
||||||
t.Fatalf("expected ErrInvalidRequest, got %v", err)
|
t.Fatalf("expected ErrInvalidRequest, got %v", err)
|
||||||
}
|
}
|
||||||
|
if !errors.Is(err, ErrProfileRequired) {
|
||||||
|
t.Fatalf("expected ErrProfileRequired, got %v", err)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunnerPrepareSelectedProfileDoesNotExistFails(t *testing.T) {
|
func TestRunnerPrepareSelectedProfileDoesNotExistFails(t *testing.T) {
|
||||||
@@ -257,6 +260,7 @@ func TestRunnerPrepareRuntimeOverrideBeatsSelectedProfileValue(t *testing.T) {
|
|||||||
MaxTokens: 500,
|
MaxTokens: 500,
|
||||||
TopP: 0.9,
|
TopP: 0.9,
|
||||||
TimeoutSeconds: 120,
|
TimeoutSeconds: 120,
|
||||||
|
ServiceTier: "priority",
|
||||||
},
|
},
|
||||||
}}
|
}}
|
||||||
runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{forbid: true}, nil)
|
runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{forbid: true}, nil)
|
||||||
@@ -270,6 +274,7 @@ func TestRunnerPrepareRuntimeOverrideBeatsSelectedProfileValue(t *testing.T) {
|
|||||||
Model: "override-model",
|
Model: "override-model",
|
||||||
Temperature: 0.7,
|
Temperature: 0.7,
|
||||||
TimeoutSeconds: 30,
|
TimeoutSeconds: 30,
|
||||||
|
ServiceTier: "flex",
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -281,6 +286,9 @@ func TestRunnerPrepareRuntimeOverrideBeatsSelectedProfileValue(t *testing.T) {
|
|||||||
if prepared.EffectiveModelParams.TopP != 0.9 {
|
if prepared.EffectiveModelParams.TopP != 0.9 {
|
||||||
t.Fatalf("expected profile top_p to remain, got %v", prepared.EffectiveModelParams.TopP)
|
t.Fatalf("expected profile top_p to remain, got %v", prepared.EffectiveModelParams.TopP)
|
||||||
}
|
}
|
||||||
|
if prepared.EffectiveModelParams.ServiceTier != "flex" {
|
||||||
|
t.Fatalf("expected service_tier override to win, got %q", prepared.EffectiveModelParams.ServiceTier)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunnerPrepareSelectedProfileBeatsBuiltInDefault(t *testing.T) {
|
func TestRunnerPrepareSelectedProfileBeatsBuiltInDefault(t *testing.T) {
|
||||||
@@ -292,6 +300,7 @@ func TestRunnerPrepareSelectedProfileBeatsBuiltInDefault(t *testing.T) {
|
|||||||
Model: "profile-model",
|
Model: "profile-model",
|
||||||
TopP: 0.8,
|
TopP: 0.8,
|
||||||
TimeoutSeconds: 90,
|
TimeoutSeconds: 90,
|
||||||
|
ServiceTier: "priority",
|
||||||
},
|
},
|
||||||
}}
|
}}
|
||||||
runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{forbid: true}, nil)
|
runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{forbid: true}, nil)
|
||||||
@@ -310,6 +319,9 @@ func TestRunnerPrepareSelectedProfileBeatsBuiltInDefault(t *testing.T) {
|
|||||||
if prepared.EffectiveModelParams.TimeoutSeconds != 90 {
|
if prepared.EffectiveModelParams.TimeoutSeconds != 90 {
|
||||||
t.Fatalf("expected profile timeout to beat default, got %d", prepared.EffectiveModelParams.TimeoutSeconds)
|
t.Fatalf("expected profile timeout to beat default, got %d", prepared.EffectiveModelParams.TimeoutSeconds)
|
||||||
}
|
}
|
||||||
|
if prepared.EffectiveModelParams.ServiceTier != "priority" {
|
||||||
|
t.Fatalf("expected profile service_tier to beat default, got %q", prepared.EffectiveModelParams.ServiceTier)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunnerPrepareFileBackedPromptBodiesRenderCorrectly(t *testing.T) {
|
func TestRunnerPrepareFileBackedPromptBodiesRenderCorrectly(t *testing.T) {
|
||||||
@@ -478,6 +490,35 @@ func TestRunnerPrepareJSONSchemaBuildsStructuredOutputSpec(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunnerPrepareJSONSchemaSchemaLoadFailureReturnsValidationError(t *testing.T) {
|
||||||
|
def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 0)
|
||||||
|
def.Validation.SchemaPath = "missing.schema.json"
|
||||||
|
validator := &fakeValidator{schemaErr: errors.New("schema unavailable")}
|
||||||
|
runner := NewRunner(
|
||||||
|
&fakePromptRepo{def: def},
|
||||||
|
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}},
|
||||||
|
defaultArtifactReader(),
|
||||||
|
defaultRenderer(),
|
||||||
|
&fakeLLM{forbid: true},
|
||||||
|
validator,
|
||||||
|
)
|
||||||
|
|
||||||
|
_, err := runner.Prepare(context.Background(), domain.RunRequest{
|
||||||
|
PromptID: "p",
|
||||||
|
ProfileID: "exec",
|
||||||
|
Inputs: singleInputRef(),
|
||||||
|
})
|
||||||
|
if !errors.Is(err, ErrValidation) {
|
||||||
|
t.Fatalf("expected ErrValidation, got %v", err)
|
||||||
|
}
|
||||||
|
if validator.schemaLoads != 1 {
|
||||||
|
t.Fatalf("expected one schema load attempt, got %d", validator.schemaLoads)
|
||||||
|
}
|
||||||
|
if validator.schemaLoadPath != "missing.schema.json" {
|
||||||
|
t.Fatalf("expected schema path missing.schema.json, got %q", validator.schemaLoadPath)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunnerRunJSONSchemaSchemaLoadFailureFailsBeforeLLM(t *testing.T) {
|
func TestRunnerRunJSONSchemaSchemaLoadFailureFailsBeforeLLM(t *testing.T) {
|
||||||
def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 0)
|
def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 0)
|
||||||
def.Validation.SchemaPath = "missing.schema.json"
|
def.Validation.SchemaPath = "missing.schema.json"
|
||||||
@@ -726,6 +767,7 @@ func TestRunnerRunExplicitRuntimeOverrideBeatsSelectedProfileValue(t *testing.T)
|
|||||||
MaxTokens: 500,
|
MaxTokens: 500,
|
||||||
TopP: 0.9,
|
TopP: 0.9,
|
||||||
TimeoutSeconds: 120,
|
TimeoutSeconds: 120,
|
||||||
|
ServiceTier: "priority",
|
||||||
},
|
},
|
||||||
}}
|
}}
|
||||||
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}
|
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}
|
||||||
@@ -740,6 +782,7 @@ func TestRunnerRunExplicitRuntimeOverrideBeatsSelectedProfileValue(t *testing.T)
|
|||||||
Model: "override-model",
|
Model: "override-model",
|
||||||
Temperature: 0.7,
|
Temperature: 0.7,
|
||||||
TimeoutSeconds: 30,
|
TimeoutSeconds: 30,
|
||||||
|
ServiceTier: "flex",
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -754,6 +797,9 @@ func TestRunnerRunExplicitRuntimeOverrideBeatsSelectedProfileValue(t *testing.T)
|
|||||||
if res.EffectiveModelParams.TopP != 0.9 {
|
if res.EffectiveModelParams.TopP != 0.9 {
|
||||||
t.Fatalf("expected non-overridden profile top_p to remain, got %v", res.EffectiveModelParams.TopP)
|
t.Fatalf("expected non-overridden profile top_p to remain, got %v", res.EffectiveModelParams.TopP)
|
||||||
}
|
}
|
||||||
|
if res.EffectiveModelParams.ServiceTier != "flex" {
|
||||||
|
t.Fatalf("expected service_tier override to win, got %q", res.EffectiveModelParams.ServiceTier)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunnerRunSelectedProfileBeatsBuiltInDefault(t *testing.T) {
|
func TestRunnerRunSelectedProfileBeatsBuiltInDefault(t *testing.T) {
|
||||||
@@ -765,6 +811,7 @@ func TestRunnerRunSelectedProfileBeatsBuiltInDefault(t *testing.T) {
|
|||||||
Model: "profile-model",
|
Model: "profile-model",
|
||||||
TopP: 0.8,
|
TopP: 0.8,
|
||||||
TimeoutSeconds: 90,
|
TimeoutSeconds: 90,
|
||||||
|
ServiceTier: "priority",
|
||||||
},
|
},
|
||||||
}}
|
}}
|
||||||
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}
|
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}
|
||||||
@@ -784,6 +831,9 @@ func TestRunnerRunSelectedProfileBeatsBuiltInDefault(t *testing.T) {
|
|||||||
if res.EffectiveModelParams.TimeoutSeconds != 90 {
|
if res.EffectiveModelParams.TimeoutSeconds != 90 {
|
||||||
t.Fatalf("expected profile timeout to beat default, got %d", res.EffectiveModelParams.TimeoutSeconds)
|
t.Fatalf("expected profile timeout to beat default, got %d", res.EffectiveModelParams.TimeoutSeconds)
|
||||||
}
|
}
|
||||||
|
if res.EffectiveModelParams.ServiceTier != "priority" {
|
||||||
|
t.Fatalf("expected profile service_tier to beat default, got %q", res.EffectiveModelParams.ServiceTier)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunnerRunBuiltInDefaultsUsedWhenProfileOmitsOptionalFields(t *testing.T) {
|
func TestRunnerRunBuiltInDefaultsUsedWhenProfileOmitsOptionalFields(t *testing.T) {
|
||||||
@@ -844,6 +894,9 @@ func TestRunnerRunAPIKeyEnvMissingEnvironmentValueFailsClearly(t *testing.T) {
|
|||||||
if !errors.Is(err, ErrInvalidRequest) {
|
if !errors.Is(err, ErrInvalidRequest) {
|
||||||
t.Fatalf("expected ErrInvalidRequest, got %v", err)
|
t.Fatalf("expected ErrInvalidRequest, got %v", err)
|
||||||
}
|
}
|
||||||
|
if !errors.Is(err, ErrAPIKeyEnvMissing) {
|
||||||
|
t.Fatalf("expected ErrAPIKeyEnvMissing, got %v", err)
|
||||||
|
}
|
||||||
if !strings.Contains(err.Error(), "SCRIPTORIUM_MISSING_KEY") {
|
if !strings.Contains(err.Error(), "SCRIPTORIUM_MISSING_KEY") {
|
||||||
t.Fatalf("expected missing env name in error, got %v", err)
|
t.Fatalf("expected missing env name in error, got %v", err)
|
||||||
}
|
}
|
||||||
@@ -1104,6 +1157,174 @@ func TestRunnerRunJSONSchemaRepairCarriesStructuredOutputSpec(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecutionProfileToTargetPopulatesAllFieldsAndCopiesExtraParams(t *testing.T) {
|
||||||
|
src := &domain.ExecutionProfile{
|
||||||
|
ID: "exec",
|
||||||
|
Endpoint: "http://profile/v1",
|
||||||
|
Model: "profile-model",
|
||||||
|
Temperature: 0.2,
|
||||||
|
MaxTokens: 123,
|
||||||
|
TopP: 0.75,
|
||||||
|
TimeoutSeconds: 90,
|
||||||
|
ServiceTier: "priority",
|
||||||
|
ReasoningEffort: "medium",
|
||||||
|
APIKeyEnv: "SCRIPTORIUM_API_KEY",
|
||||||
|
ExtraParams: map[string]string{
|
||||||
|
"provider_option": "on",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
target := executionProfileToTarget(src)
|
||||||
|
if target.Endpoint != src.Endpoint ||
|
||||||
|
target.Model != src.Model ||
|
||||||
|
target.Temperature != src.Temperature ||
|
||||||
|
target.MaxTokens != src.MaxTokens ||
|
||||||
|
target.TopP != src.TopP ||
|
||||||
|
target.TimeoutSeconds != src.TimeoutSeconds ||
|
||||||
|
target.ServiceTier != src.ServiceTier ||
|
||||||
|
target.ReasoningEffort != src.ReasoningEffort ||
|
||||||
|
target.APIKeyEnv != src.APIKeyEnv {
|
||||||
|
t.Fatalf("expected all profile fields to populate target, got %+v", target)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(target.ExtraParams, src.ExtraParams) {
|
||||||
|
t.Fatalf("expected extra_params to match, got %#v", target.ExtraParams)
|
||||||
|
}
|
||||||
|
|
||||||
|
src.ExtraParams["provider_option"] = "changed"
|
||||||
|
if target.ExtraParams["provider_option"] != "on" {
|
||||||
|
t.Fatalf("expected extra_params copy to be independent, got %#v", target.ExtraParams)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveExecutionTargetProfileValuesPopulateAllSupportedFields(t *testing.T) {
|
||||||
|
profileValue := &domain.ExecutionProfile{
|
||||||
|
ID: "exec",
|
||||||
|
Endpoint: "http://profile/v1",
|
||||||
|
Model: "profile-model",
|
||||||
|
Temperature: 0.3,
|
||||||
|
MaxTokens: 222,
|
||||||
|
TopP: 0.6,
|
||||||
|
TimeoutSeconds: 77,
|
||||||
|
ServiceTier: "priority",
|
||||||
|
ReasoningEffort: "low",
|
||||||
|
APIKeyEnv: "PROFILE_KEY",
|
||||||
|
ExtraParams: map[string]string{
|
||||||
|
"profile_option": "enabled",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
target := resolveExecutionTarget(profileValue, nil)
|
||||||
|
if target.Endpoint != profileValue.Endpoint ||
|
||||||
|
target.Model != profileValue.Model ||
|
||||||
|
target.Temperature != profileValue.Temperature ||
|
||||||
|
target.MaxTokens != profileValue.MaxTokens ||
|
||||||
|
target.TopP != profileValue.TopP ||
|
||||||
|
target.TimeoutSeconds != profileValue.TimeoutSeconds ||
|
||||||
|
target.ServiceTier != profileValue.ServiceTier ||
|
||||||
|
target.ReasoningEffort != profileValue.ReasoningEffort ||
|
||||||
|
target.APIKeyEnv != profileValue.APIKeyEnv {
|
||||||
|
t.Fatalf("expected profile values to populate target, got %+v", target)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(target.ExtraParams, profileValue.ExtraParams) {
|
||||||
|
t.Fatalf("expected profile extra_params in target, got %#v", target.ExtraParams)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveExecutionTargetRuntimeOverridesBeatProfileForAllOverrideableFields(t *testing.T) {
|
||||||
|
profileValue := &domain.ExecutionProfile{
|
||||||
|
ID: "exec",
|
||||||
|
Endpoint: "http://profile/v1",
|
||||||
|
Model: "profile-model",
|
||||||
|
Temperature: 0.2,
|
||||||
|
MaxTokens: 200,
|
||||||
|
TopP: 0.8,
|
||||||
|
TimeoutSeconds: 90,
|
||||||
|
ServiceTier: "priority",
|
||||||
|
ReasoningEffort: "medium",
|
||||||
|
APIKeyEnv: "PROFILE_KEY",
|
||||||
|
ExtraParams: map[string]string{
|
||||||
|
"profile_only": "yes",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
override := &domain.ExecutionTarget{
|
||||||
|
Endpoint: "http://override/v1",
|
||||||
|
Model: "override-model",
|
||||||
|
Temperature: 0.9,
|
||||||
|
MaxTokens: 111,
|
||||||
|
TopP: 0.5,
|
||||||
|
TimeoutSeconds: 30,
|
||||||
|
ServiceTier: "flex",
|
||||||
|
ReasoningEffort: "high",
|
||||||
|
APIKeyEnv: "RUNTIME_KEY",
|
||||||
|
ExtraParams: map[string]string{
|
||||||
|
"runtime_only": "yes",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
target := resolveExecutionTarget(profileValue, override)
|
||||||
|
if target.Endpoint != override.Endpoint ||
|
||||||
|
target.Model != override.Model ||
|
||||||
|
target.Temperature != override.Temperature ||
|
||||||
|
target.MaxTokens != override.MaxTokens ||
|
||||||
|
target.TopP != override.TopP ||
|
||||||
|
target.TimeoutSeconds != override.TimeoutSeconds ||
|
||||||
|
target.ServiceTier != override.ServiceTier ||
|
||||||
|
target.ReasoningEffort != override.ReasoningEffort ||
|
||||||
|
target.APIKeyEnv != override.APIKeyEnv {
|
||||||
|
t.Fatalf("expected runtime overrides to win for all fields, got %+v", target)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(target.ExtraParams, override.ExtraParams) {
|
||||||
|
t.Fatalf("expected runtime extra_params to replace profile extra_params, got %#v", target.ExtraParams)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMergeExecutionTargetEmptyStringOverridesDoNotErase(t *testing.T) {
|
||||||
|
base := domain.ExecutionTarget{
|
||||||
|
Endpoint: "http://base/v1",
|
||||||
|
Model: "base-model",
|
||||||
|
ServiceTier: "priority",
|
||||||
|
ReasoningEffort: "medium",
|
||||||
|
APIKeyEnv: "BASE_KEY",
|
||||||
|
}
|
||||||
|
override := domain.ExecutionTarget{
|
||||||
|
Endpoint: "http://override/v1",
|
||||||
|
Model: "override-model",
|
||||||
|
ServiceTier: " ",
|
||||||
|
ReasoningEffort: " ",
|
||||||
|
APIKeyEnv: "",
|
||||||
|
}
|
||||||
|
|
||||||
|
merged := mergeExecutionTarget(base, override)
|
||||||
|
if merged.Endpoint != "http://override/v1" || merged.Model != "override-model" {
|
||||||
|
t.Fatalf("expected endpoint/model to override, got %+v", merged)
|
||||||
|
}
|
||||||
|
if merged.ServiceTier != "priority" {
|
||||||
|
t.Fatalf("expected empty service_tier override to be ignored, got %q", merged.ServiceTier)
|
||||||
|
}
|
||||||
|
if merged.ReasoningEffort != "medium" {
|
||||||
|
t.Fatalf("expected empty reasoning_effort override to be ignored, got %q", merged.ReasoningEffort)
|
||||||
|
}
|
||||||
|
if merged.APIKeyEnv != "BASE_KEY" {
|
||||||
|
t.Fatalf("expected empty api_key_env override to be ignored, got %q", merged.APIKeyEnv)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMergeExecutionTargetEmptyExtraParamsDoesNotErase(t *testing.T) {
|
||||||
|
base := domain.ExecutionTarget{
|
||||||
|
ExtraParams: map[string]string{
|
||||||
|
"keep": "value",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
override := domain.ExecutionTarget{
|
||||||
|
ExtraParams: map[string]string{},
|
||||||
|
}
|
||||||
|
|
||||||
|
merged := mergeExecutionTarget(base, override)
|
||||||
|
if !reflect.DeepEqual(merged.ExtraParams, base.ExtraParams) {
|
||||||
|
t.Fatalf("expected empty extra_params override not to erase base values, got %#v", merged.ExtraParams)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestBuildOutputArtifactDefaults(t *testing.T) {
|
func TestBuildOutputArtifactDefaults(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
|
|||||||
@@ -116,6 +116,49 @@ func TestStandardValidatorJSONSchemaSuccess(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestStandardValidatorJSONSchemaNestedSchemaPathSuccess(t *testing.T) {
|
||||||
|
tmp := t.TempDir()
|
||||||
|
nestedDir := filepath.Join(tmp, "dnd")
|
||||||
|
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(nestedDir, "schema.json"), []byte(`{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"type": "object",
|
||||||
|
"required": ["name"],
|
||||||
|
"properties": {
|
||||||
|
"name": {"type": "string"}
|
||||||
|
}
|
||||||
|
}`), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
v := NewStandardValidator(tmp)
|
||||||
|
|
||||||
|
res, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"name":"eris"}`)}, domain.OutputContract{
|
||||||
|
ValidationMode: domain.ValidationJSONSchema,
|
||||||
|
SchemaPath: filepath.Join("dnd", "schema.json"),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if res.Status != domain.ValidationPassed || !res.IsValid {
|
||||||
|
t.Fatalf("expected passed/valid, got status=%q valid=%v", res.Status, res.IsValid)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStandardValidatorJSONSchemaNestedSchemaPathMissing(t *testing.T) {
|
||||||
|
v := NewStandardValidator(t.TempDir())
|
||||||
|
|
||||||
|
_, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"name":"eris"}`)}, domain.OutputContract{
|
||||||
|
ValidationMode: domain.ValidationJSONSchema,
|
||||||
|
SchemaPath: filepath.Join("dnd", "missing.json"),
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected nested schema load error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestStandardValidatorJSONSchemaFailure(t *testing.T) {
|
func TestStandardValidatorJSONSchemaFailure(t *testing.T) {
|
||||||
tmp := t.TempDir()
|
tmp := t.TempDir()
|
||||||
schemaPath := filepath.Join(tmp, "schema.json")
|
schemaPath := filepath.Join(tmp, "schema.json")
|
||||||
|
|||||||
Reference in New Issue
Block a user