eric/scriptorium
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Refactor: split prompt definition from execution settings and migrate run contracts to prompt_* + execution_target

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-05 10:09:31 -05:00
parent fdfd8641f5
commit a633c67538
28 changed files with 712 additions and 1021 deletions
Download Patch File Download Diff File Expand all files Collapse all files

150
README.md
View File

@@ -1,10 +1,10 @@
# scriptorium
Scriptorium is a generic prompt-profile execution engine written in Go.
Scriptorium is a generic prompt-definition execution engine written in Go.
Given named input artifacts and a prompt profile, Scriptorium:
Given named input artifacts and a prompt definition, Scriptorium:
1. Loads the profile.
1. Loads the prompt definition.
2. Resolves input artifact references.
3. Renders prompt messages from templates.
4. Calls an OpenAI-compatible LLM endpoint.
@@ -28,36 +28,23 @@ D&D-specific behavior belongs in profiles, schemas, fixtures, and caller inputs,
## Core Concepts
- Prompt profile: YAML config that defines templates, model defaults, output format, and validation behavior.
- Named inputs: logical input names (for example `transcript`, `glossary`) mapped to artifact references.
- Artifact refs: currently `file` and `inline` are supported by readers used in v1 flows.
- Template variables: key/value vars provided at run time and accessed in templates as `{{.var_name}}`.
- Model target: endpoint/model and generation parameters (`temperature`, `max_tokens`, `top_p`, `timeout_seconds`).
- Prompt definition: YAML config for templates, inputs, output format, and validation behavior.
- Execution profile: conceptual runtime config (endpoint/model/timeouts/auth source). In this transition, execution settings are supplied as run-time overrides.
- Named inputs: logical names (for example `transcript`, `glossary`) mapped to artifact references.
- Artifact refs: currently `file` and `inline` are supported.
- Template variables: key/value vars passed at run time and referenced as `{{.var_name}}`.
- Execution target: endpoint/model plus generation/runtime parameters (`temperature`, `max_tokens`, `top_p`, `timeout_seconds`, `reasoning_effort`, `api_key_env`).
- Output format: `text`, `markdown`, or `json`.
- Validation mode: `none`, `basic`, `json`, `json_schema`.
- Repair attempts: bounded retries for structured modes (`json`, `json_schema`) when output validation fails.
- Run metadata: IDs/hashes/model/timing/usage/validation details for auditability.
## Build and Test
Build:
```bash
go build -o scriptorium ./cmd/scriptorium
```
Run tests:
```bash
go test ./...
```
Run CLI locally:
```bash
go run ./cmd/scriptorium run --help
```
## CLI Usage
### `scriptorium run`
@@ -65,57 +52,38 @@ go run ./cmd/scriptorium run --help
Required flags:
- `--profile-dir`
- `--profile-id`
- `--prompt-id`
- `--input` (repeatable `name=path`)
Optional flags:
Common optional flags:
- `--profile-id` (execution profile selector; falls back to prompt `default_profile`)
- `--var` (repeatable `name=value`)
- `--out`
- `--llm-base-url`
- `--llm-api-key`
- `--model`
- `--api-key-env`
- `--temperature`
- `--max-tokens`
- `--schema-dir`
- `--timeout`
If `--llm-base-url` and/or `--model` are omitted, profile `model_defaults` must provide them.
Current transitional behavior: execution-profile loading is not implemented yet, so run-time execution settings must be supplied via overrides. In practice, provide at least endpoint and model (`--llm-base-url` and `--model`).
Markdown summary example:
Example:
```bash
export SCRIPTORIUM_API_KEY="your-key"
go run ./cmd/scriptorium run \
--profile-dir ./profiles \
--profile-id generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \
--out ./out.md
```
Same run with explicit local OpenAI-compatible endpoint (for example vLLM):
```bash
go run ./cmd/scriptorium run \
--profile-dir ./profiles \
--profile-id generic.markdown_summary \
--prompt-id generic.markdown_summary \
--profile-id local-default \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \
--llm-base-url http://localhost:8000/v1 \
--model gpt-4o-mini \
--out ./out.md
```
Passing template variables:
```bash
go run ./cmd/scriptorium run \
--profile-dir ./profiles \
--profile-id generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \
--var session_date=2026-05-04 \
--var facilitator="Eris" \
--api-key-env SCRIPTORIUM_API_KEY \
--out ./out.md
```
@@ -123,7 +91,7 @@ Output behavior:
- Artifact content goes to stdout unless `--out` is set.
- Summaries and errors are written to stderr.
- Exit code `2` indicates run succeeded but validation status is `failed`.
- Exit code `2` means the run succeeded but validation status is `failed`.
### `scriptorium serve`
@@ -138,23 +106,24 @@ Common optional flags:
- `--addr` (default `:8080`)
- `--schema-dir` (default `.`)
- `--llm-api-key`
- `--model`
- `--timeout` (default `10m`)
## HTTP API
Run endpoint:
Endpoint:
- `POST /v1/runs`
- No built-in authentication is provided in the current implementation; deploy behind a trusted boundary or gateway.
No built-in authentication is provided by the server itself. Deploy behind a trusted boundary or gateway.
Request example:
```json
{
"profile_id": "generic.structured_events",
"profile_version": "1.0.0",
"prompt_id": "generic.structured_events",
"prompt_version": "1.0.0",
"profile_id": "local-default",
"inputs": {
"transcript": {"type": "file", "uri": "./examples/fixtures/transcript.md"},
"glossary": {"type": "file", "uri": "./examples/fixtures/glossary.yml"}
@@ -168,7 +137,8 @@ Request example:
"temperature": 0.0,
"max_tokens": 600,
"top_p": 1.0,
"timeout_seconds": 120
"timeout_seconds": 120,
"api_key_env": "SCRIPTORIUM_API_KEY"
}
}
```
@@ -195,9 +165,11 @@ Response shape:
},
"metadata": {
"run_id": "xxxxxxxx-xxxx-4xxx-8xxx-xxxxxxxxxxxx",
"profile_id": "generic.structured_events",
"profile_version": "1.0.0",
"profile_hash": "...",
"prompt_id": "generic.structured_events",
"prompt_version": "1.0.0",
"prompt_hash": "...",
"rendered_prompt_hash": "...",
"selected_profile_id": "local-default",
"model_name": "gpt-4o-mini",
"endpoint": "http://localhost:8000/v1",
"model_params": {
@@ -206,10 +178,10 @@ Response shape:
"temperature": 0,
"max_tokens": 600,
"top_p": 1,
"timeout_seconds": 120
"timeout_seconds": 120,
"api_key_env": "SCRIPTORIUM_API_KEY"
},
"input_hashes": {"transcript": "...", "glossary": "..."},
"prompt_hash": "...",
"usage": {"prompt_tokens": 10, "completion_tokens": 20, "total_tokens": 30},
"start_time": "...",
"end_time": "...",
@@ -222,7 +194,7 @@ Response shape:
}
```
Validation content failures are returned as successful run responses (`200`) with `validation.status = "failed"`; raw model output is preserved in `raw_model_output`.
Validation content failures return `200` with `validation.status = "failed"` and preserve `raw_model_output`.
Error response shape:
@@ -235,15 +207,17 @@ Error response shape:
}
```
## Prompt Profile Authoring
## Prompt Definition Authoring
### Minimal Markdown profile
### Minimal Markdown prompt definition
```yaml
id: generic.markdown_summary
version: "1.0.0"
expected_inputs:
- transcript
default_profile: local-default
inputs:
- name: transcript
required: true
templates:
- role: system
content: "You are a concise assistant."
@@ -251,23 +225,20 @@ templates:
content: |
Summarize:
{{input "transcript"}}
model_defaults:
endpoint: http://localhost:8000/v1
model: gpt-4o-mini
temperature: 0.2
max_tokens: 700
output_format: markdown
validation:
validation_mode: basic
```
### Structured JSON profile with schema validation
### Structured JSON prompt definition with schema validation
```yaml
id: generic.structured_events
version: "1.0.0"
expected_inputs:
- transcript
default_profile: local-default
inputs:
- name: transcript
required: true
templates:
- role: system
content: "Return only JSON."
@@ -275,9 +246,6 @@ templates:
content: |
Extract events from:
{{input "transcript"}}
model_defaults:
endpoint: http://localhost:8000/v1
model: gpt-4o-mini
output_format: json
validation:
format: json
@@ -286,30 +254,28 @@ validation:
repair_attempts: 1
```
`repair_attempts` is bounded. Repair is attempted only for structured validation modes.
`repair_attempts` is strictly bounded and only applies to structured validation modes.
## Validation Modes
Supported modes:
- `none`: skipped validation result.
- `basic`: fails if output is empty/whitespace.
- `basic`: fails for empty/whitespace output.
- `json`: output must parse as JSON.
- `json_schema`: output must parse as JSON and satisfy the configured schema.
- `json_schema`: output must parse as JSON and satisfy configured schema.
Validation failures caused by output content are represented in `validation` and do not discard raw model output.
Validation content failures are returned in the structured result; raw model output is preserved.
## Repository Examples
## Examples
- Profiles: `profiles/`
- Prompt definitions: `profiles/`
- Schemas: `schemas/`
- Fixtures: `examples/fixtures/`
- Local experimentation: `local-test/`
## Development Notes
- Core is generic and follows a ports-and-adapters style.
- Domain/usecase packages do not depend on HTTP/CLI/wire types.
- Core follows ports-and-adapters and remains domain-generic.
- Domain/usecase packages do not depend on HTTP/CLI wire DTOs.
- To add a new LLM adapter: implement `internal/llm.Client`.
- To add a new artifact reader: implement/extend `internal/artifact.Reader` routing.
- To add a new validation mode: extend `internal/validate` and keep run semantics stable.
- To add a new artifact reader: extend `internal/artifact.Reader` routing.
- To add a new validation mode: extend `internal/validate` and preserve run semantics.