eric/scriptorium
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Refactor: split prompt definition from execution settings and migrate run contracts to prompt_* + execution_target

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-05 10:09:31 -05:00
parent fdfd8641f5
commit a633c67538
28 changed files with 712 additions and 1021 deletions
Download Patch File Download Diff File Expand all files Collapse all files

150
README.md
View File

@@ -1,10 +1,10 @@
# scriptorium # scriptorium
Scriptorium is a generic prompt-profile execution engine written in Go. Scriptorium is a generic prompt-definition execution engine written in Go.
Given named input artifacts and a prompt profile, Scriptorium: Given named input artifacts and a prompt definition, Scriptorium:
1. Loads the profile. 1. Loads the prompt definition.
2. Resolves input artifact references. 2. Resolves input artifact references.
3. Renders prompt messages from templates. 3. Renders prompt messages from templates.
4. Calls an OpenAI-compatible LLM endpoint. 4. Calls an OpenAI-compatible LLM endpoint.
@@ -28,36 +28,23 @@ D&D-specific behavior belongs in profiles, schemas, fixtures, and caller inputs,
## Core Concepts ## Core Concepts
- Prompt profile: YAML config that defines templates, model defaults, output format, and validation behavior. - Prompt definition: YAML config for templates, inputs, output format, and validation behavior.
- Named inputs: logical input names (for example `transcript`, `glossary`) mapped to artifact references. - Execution profile: conceptual runtime config (endpoint/model/timeouts/auth source). In this transition, execution settings are supplied as run-time overrides.
- Artifact refs: currently `file` and `inline` are supported by readers used in v1 flows. - Named inputs: logical names (for example `transcript`, `glossary`) mapped to artifact references.
- Template variables: key/value vars provided at run time and accessed in templates as `{{.var_name}}`. - Artifact refs: currently `file` and `inline` are supported.
- Model target: endpoint/model and generation parameters (`temperature`, `max_tokens`, `top_p`, `timeout_seconds`). - Template variables: key/value vars passed at run time and referenced as `{{.var_name}}`.
- Execution target: endpoint/model plus generation/runtime parameters (`temperature`, `max_tokens`, `top_p`, `timeout_seconds`, `reasoning_effort`, `api_key_env`).
- Output format: `text`, `markdown`, or `json`. - Output format: `text`, `markdown`, or `json`.
- Validation mode: `none`, `basic`, `json`, `json_schema`. - Validation mode: `none`, `basic`, `json`, `json_schema`.
- Repair attempts: bounded retries for structured modes (`json`, `json_schema`) when output validation fails. - Repair attempts: bounded retries for structured modes (`json`, `json_schema`) when output validation fails.
- Run metadata: IDs/hashes/model/timing/usage/validation details for auditability.
## Build and Test ## Build and Test
Build:
```bash ```bash
go build -o scriptorium ./cmd/scriptorium go build -o scriptorium ./cmd/scriptorium
```
Run tests:
```bash
go test ./... go test ./...
``` ```
Run CLI locally:
```bash
go run ./cmd/scriptorium run --help
```
## CLI Usage ## CLI Usage
### `scriptorium run` ### `scriptorium run`
@@ -65,57 +52,38 @@ go run ./cmd/scriptorium run --help
Required flags: Required flags:
- `--profile-dir` - `--profile-dir`
- `--profile-id` - `--prompt-id`
- `--input` (repeatable `name=path`) - `--input` (repeatable `name=path`)
Optional flags: Common optional flags:
- `--profile-id` (execution profile selector; falls back to prompt `default_profile`)
- `--var` (repeatable `name=value`) - `--var` (repeatable `name=value`)
- `--out` - `--out`
- `--llm-base-url` - `--llm-base-url`
- `--llm-api-key`
- `--model` - `--model`
- `--api-key-env`
- `--temperature` - `--temperature`
- `--max-tokens` - `--max-tokens`
- `--schema-dir` - `--schema-dir`
- `--timeout` - `--timeout`
If `--llm-base-url` and/or `--model` are omitted, profile `model_defaults` must provide them. Current transitional behavior: execution-profile loading is not implemented yet, so run-time execution settings must be supplied via overrides. In practice, provide at least endpoint and model (`--llm-base-url` and `--model`).
Markdown summary example: Example:
```bash ```bash
export SCRIPTORIUM_API_KEY="your-key"
go run ./cmd/scriptorium run \ go run ./cmd/scriptorium run \
--profile-dir ./profiles \ --profile-dir ./profiles \
--profile-id generic.markdown_summary \ --prompt-id generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \ --profile-id local-default \
--input glossary=./examples/fixtures/glossary.yml \
--out ./out.md
```
Same run with explicit local OpenAI-compatible endpoint (for example vLLM):
```bash
go run ./cmd/scriptorium run \
--profile-dir ./profiles \
--profile-id generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \ --input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \ --input glossary=./examples/fixtures/glossary.yml \
--llm-base-url http://localhost:8000/v1 \ --llm-base-url http://localhost:8000/v1 \
--model gpt-4o-mini \ --model gpt-4o-mini \
--out ./out.md --api-key-env SCRIPTORIUM_API_KEY \
```
Passing template variables:
```bash
go run ./cmd/scriptorium run \
--profile-dir ./profiles \
--profile-id generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \
--var session_date=2026-05-04 \
--var facilitator="Eris" \
--out ./out.md --out ./out.md
``` ```
@@ -123,7 +91,7 @@ Output behavior:
- Artifact content goes to stdout unless `--out` is set. - Artifact content goes to stdout unless `--out` is set.
- Summaries and errors are written to stderr. - Summaries and errors are written to stderr.
- Exit code `2` indicates run succeeded but validation status is `failed`. - Exit code `2` means the run succeeded but validation status is `failed`.
### `scriptorium serve` ### `scriptorium serve`
@@ -138,23 +106,24 @@ Common optional flags:
- `--addr` (default `:8080`) - `--addr` (default `:8080`)
- `--schema-dir` (default `.`) - `--schema-dir` (default `.`)
- `--llm-api-key`
- `--model` - `--model`
- `--timeout` (default `10m`) - `--timeout` (default `10m`)
## HTTP API ## HTTP API
Run endpoint: Endpoint:
- `POST /v1/runs` - `POST /v1/runs`
- No built-in authentication is provided in the current implementation; deploy behind a trusted boundary or gateway.
No built-in authentication is provided by the server itself. Deploy behind a trusted boundary or gateway.
Request example: Request example:
```json ```json
{ {
"profile_id": "generic.structured_events", "prompt_id": "generic.structured_events",
"profile_version": "1.0.0", "prompt_version": "1.0.0",
"profile_id": "local-default",
"inputs": { "inputs": {
"transcript": {"type": "file", "uri": "./examples/fixtures/transcript.md"}, "transcript": {"type": "file", "uri": "./examples/fixtures/transcript.md"},
"glossary": {"type": "file", "uri": "./examples/fixtures/glossary.yml"} "glossary": {"type": "file", "uri": "./examples/fixtures/glossary.yml"}
@@ -168,7 +137,8 @@ Request example:
"temperature": 0.0, "temperature": 0.0,
"max_tokens": 600, "max_tokens": 600,
"top_p": 1.0, "top_p": 1.0,
"timeout_seconds": 120 "timeout_seconds": 120,
"api_key_env": "SCRIPTORIUM_API_KEY"
} }
} }
``` ```
@@ -195,9 +165,11 @@ Response shape:
}, },
"metadata": { "metadata": {
"run_id": "xxxxxxxx-xxxx-4xxx-8xxx-xxxxxxxxxxxx", "run_id": "xxxxxxxx-xxxx-4xxx-8xxx-xxxxxxxxxxxx",
"profile_id": "generic.structured_events", "prompt_id": "generic.structured_events",
"profile_version": "1.0.0", "prompt_version": "1.0.0",
"profile_hash": "...", "prompt_hash": "...",
"rendered_prompt_hash": "...",
"selected_profile_id": "local-default",
"model_name": "gpt-4o-mini", "model_name": "gpt-4o-mini",
"endpoint": "http://localhost:8000/v1", "endpoint": "http://localhost:8000/v1",
"model_params": { "model_params": {
@@ -206,10 +178,10 @@ Response shape:
"temperature": 0, "temperature": 0,
"max_tokens": 600, "max_tokens": 600,
"top_p": 1, "top_p": 1,
"timeout_seconds": 120 "timeout_seconds": 120,
"api_key_env": "SCRIPTORIUM_API_KEY"
}, },
"input_hashes": {"transcript": "...", "glossary": "..."}, "input_hashes": {"transcript": "...", "glossary": "..."},
"prompt_hash": "...",
"usage": {"prompt_tokens": 10, "completion_tokens": 20, "total_tokens": 30}, "usage": {"prompt_tokens": 10, "completion_tokens": 20, "total_tokens": 30},
"start_time": "...", "start_time": "...",
"end_time": "...", "end_time": "...",
@@ -222,7 +194,7 @@ Response shape:
} }
``` ```
Validation content failures are returned as successful run responses (`200`) with `validation.status = "failed"`; raw model output is preserved in `raw_model_output`. Validation content failures return `200` with `validation.status = "failed"` and preserve `raw_model_output`.
Error response shape: Error response shape:
@@ -235,15 +207,17 @@ Error response shape:
} }
``` ```
## Prompt Profile Authoring ## Prompt Definition Authoring
### Minimal Markdown profile ### Minimal Markdown prompt definition
```yaml ```yaml
id: generic.markdown_summary id: generic.markdown_summary
version: "1.0.0" version: "1.0.0"
expected_inputs: default_profile: local-default
- transcript inputs:
- name: transcript
required: true
templates: templates:
- role: system - role: system
content: "You are a concise assistant." content: "You are a concise assistant."
@@ -251,23 +225,20 @@ templates:
content: | content: |
Summarize: Summarize:
{{input "transcript"}} {{input "transcript"}}
model_defaults:
endpoint: http://localhost:8000/v1
model: gpt-4o-mini
temperature: 0.2
max_tokens: 700
output_format: markdown output_format: markdown
validation: validation:
validation_mode: basic validation_mode: basic
``` ```
### Structured JSON profile with schema validation ### Structured JSON prompt definition with schema validation
```yaml ```yaml
id: generic.structured_events id: generic.structured_events
version: "1.0.0" version: "1.0.0"
expected_inputs: default_profile: local-default
- transcript inputs:
- name: transcript
required: true
templates: templates:
- role: system - role: system
content: "Return only JSON." content: "Return only JSON."
@@ -275,9 +246,6 @@ templates:
content: | content: |
Extract events from: Extract events from:
{{input "transcript"}} {{input "transcript"}}
model_defaults:
endpoint: http://localhost:8000/v1
model: gpt-4o-mini
output_format: json output_format: json
validation: validation:
format: json format: json
@@ -286,30 +254,28 @@ validation:
repair_attempts: 1 repair_attempts: 1
``` ```
`repair_attempts` is bounded. Repair is attempted only for structured validation modes. `repair_attempts` is strictly bounded and only applies to structured validation modes.
## Validation Modes ## Validation Modes
Supported modes:
- `none`: skipped validation result. - `none`: skipped validation result.
- `basic`: fails if output is empty/whitespace. - `basic`: fails for empty/whitespace output.
- `json`: output must parse as JSON. - `json`: output must parse as JSON.
- `json_schema`: output must parse as JSON and satisfy the configured schema. - `json_schema`: output must parse as JSON and satisfy configured schema.
Validation failures caused by output content are represented in `validation` and do not discard raw model output. Validation content failures are returned in the structured result; raw model output is preserved.
## Repository Examples ## Examples
- Profiles: `profiles/` - Prompt definitions: `profiles/`
- Schemas: `schemas/` - Schemas: `schemas/`
- Fixtures: `examples/fixtures/` - Fixtures: `examples/fixtures/`
- Local experimentation: `local-test/` - Local experimentation: `local-test/`
## Development Notes ## Development Notes
- Core is generic and follows a ports-and-adapters style. - Core follows ports-and-adapters and remains domain-generic.
- Domain/usecase packages do not depend on HTTP/CLI/wire types. - Domain/usecase packages do not depend on HTTP/CLI wire DTOs.
- To add a new LLM adapter: implement `internal/llm.Client`. - To add a new LLM adapter: implement `internal/llm.Client`.
- To add a new artifact reader: implement/extend `internal/artifact.Reader` routing. - To add a new artifact reader: extend `internal/artifact.Reader` routing.
- To add a new validation mode: extend `internal/validate` and keep run semantics stable. - To add a new validation mode: extend `internal/validate` and preserve run semantics.

234
architecture.md
View File

@@ -2,11 +2,11 @@
## 1. Purpose and Non-Goals ## 1. Purpose and Non-Goals
Scriptorium is a prompt-profile execution engine. Scriptorium is a prompt-definition execution engine.
It takes named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata. It accepts named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata.
Scriptorium is not an orchestrator. It should not own transcription, transcript merging, transcript polishing, notification, or cross-step workflow control. Scriptorium is not an orchestrator. It must not own transcription, transcript merge/polish steps, notifications, or cross-step workflow control.
For the motivating D&D workflow: For the motivating D&D workflow:
@@ -14,111 +14,116 @@ For the motivating D&D workflow:
- WhisperX transcribes. - WhisperX transcribes.
- Seriatim merges transcripts. - Seriatim merges transcripts.
- Audita polishes transcripts. - Audita polishes transcripts.
- Scriptorium generates output artifacts from prepared inputs. - Scriptorium generates final artifacts from prepared inputs.
Core Go code must remain domain-generic. Core Go code remains generic.
## 2. Current Architecture ## 2. Current Architecture
Current high-level structure: Current implementation structure:
- `cmd/scriptorium`: binary entrypoint. - `cmd/scriptorium`: binary entrypoint.
- `internal/domain`: core domain types. - `internal/domain`: core domain contracts.
- `internal/usecase`: `Runner` use case and repair loop orchestration. - `internal/usecase`: `Runner` run flow, validation integration, bounded repair coordination.
- `internal/profile`: filesystem prompt profile repository and profile validation. - `internal/profile`: transitional filesystem prompt-definition repository (package rename deferred).
- `internal/artifact`: artifact reference readers (`inline`, `file`) and routing. - `internal/artifact`: input artifact resolution (`inline`, `file`).
- `internal/prompt`: Go template-based prompt renderer. - `internal/prompt`: template rendering.
- `internal/llm`: LLM client interface + OpenAI-compatible HTTP adapter. - `internal/llm`: provider-neutral client interface + OpenAI-compatible HTTP adapter.
- `internal/validate`: output validator implementation (`none/basic/json/json_schema`). - `internal/validate`: validation implementation (`none/basic/json/json_schema`).
- `internal/adapter/cli`: CLI adapter. - `internal/adapter/cli`: CLI adapter.
- `internal/adapter/http`: HTTP adapter (`POST /v1/runs`). - `internal/adapter/http`: HTTP adapter (`POST /v1/runs`).
This is a practical ports-and-adapters implementation.
## 3. Run Data Flow ## 3. Run Data Flow
`Runner.Run(ctx, RunRequest)` currently executes: `Runner.Run(ctx, RunRequest)` currently executes:
1. Validate minimum request requirements (`profile_id`). 1. Validate request (`prompt_id` required).
2. Load prompt profile by ID/version. 2. Load `PromptDefinition` by ID/version.
3. Merge effective model target (profile defaults + request override). 3. Determine selected profile ID (`request.profile_id` or prompt `default_profile`).
4. Resolve effective output contract (profile + optional request override). 4. Resolve effective execution target from request override (execution-profile loading is deferred in this phase).
5. Resolve named artifact refs to loaded artifacts. 5. Resolve named input artifact refs.
6. Render prompt messages from templates. 6. Render prompt messages.
7. Hash rendered prompt for auditability. 7. Hash prompt definition and rendered prompt.
8. Call LLM client with provider-neutral `GenerateRequest`. 8. Call LLM client with `GenerateRequest`.
9. Build output artifact from model content. 9. Build output artifact.
10. Validate output. 10. Validate output.
11. If structured validation failed and repair is enabled/bounded, run repair attempts and re-validate. 11. If structured validation failed and repair is enabled, run bounded repair attempts and re-validate.
12. Return `RunResult` with artifact, validation, raw output, and metadata. 12. Return `RunResult` with artifact, raw output, validation result, metadata.
Validation content failure remains a successful run result with `validation.status=failed`. Validation content failures are returned as successful runs with `validation.status=failed`.
## 4. Package Responsibilities ## 4. Package Responsibilities
- `domain` - `domain`
- Owns core nouns and contracts. - Owns core nouns/contracts.
- Must not import adapters/provider SDK types. - Must not depend on adapters/provider SDK types.
- `usecase` - `usecase`
- Owns execution sequence and cross-port orchestration for a single run. - Owns single-run orchestration across ports.
- May coordinate validation and bounded repair. - Owns bounded repair control flow.
- Must not contain HTTP/CLI/wire concerns. - Must not own transport/wire concerns.
- `profile` - `profile` (transitional)
- Owns prompt profile loading/parsing/validation. - Currently loads prompt definitions from YAML.
- Handles YAML strict decoding and profile-level constraints. - Package naming split (`prompt definition repo` vs `execution profile repo`) is deferred follow-up.
- `artifact` - `artifact`
- Owns artifact ref resolution and content loading. - Loads artifacts from refs and normalizes payload metadata.
- Produces normalized `Artifact` values with size/hash/content type.
- `prompt` - `prompt`
- Owns template rendering and required-input enforcement. - Renders templates and enforces required inputs.
- `llm` - `llm`
- Owns generation port and provider adapters. - Defines generation client contract and protocol adapters.
- Current adapter: OpenAI-compatible chat completions over `net/http`.
- `validate` - `validate`
- Owns output validation semantics and JSON Schema integration. - Owns output validation semantics and schema validation.
- `adapter/http`, `adapter/cli` - `adapter/http`, `adapter/cli`
- Owns transport/wire/flag concerns only. - Own request/response/flag mapping only.
- Should stay thin and delegate business flow to `usecase.Runner`. - Delegate business flow to `usecase.Runner`.
## 5. Domain Model (Current) ## 5. Domain Model (Current)
Key types in `internal/domain`: Key types:
- `RunRequest`: profile selector, named input refs, vars, optional model override, optional validation override. - `PromptDefinition`
- `RunResult`: output artifact, validation result, raw output, profile/model metadata, hashes, usage, timestamps, duration. - `id`, `version`, `default_profile`, `inputs`, `templates`, `output_format`, `validation`.
- `ArtifactRef`: `{type, uri, body}` reference contract. - `ExecutionProfile`
- `Artifact`: loaded payload (`name`, `content_type`, `body`, `uri`, `size`, `hash`). - Execution/runtime settings shape (`endpoint`, `model`, timeouts, `api_key_env`, etc.).
- `PromptProfile`: YAML-backed profile definition. - Loading/persistence is deferred in this pass.
- `RenderedPrompt` / `RenderedMessage`: provider-neutral prompt structure. - `ExecutionTarget`
- `GenerateRequest` / `GenerateResponse`: provider-neutral model I/O. - Effective execution settings for a run.
- `ValidationResult`: passed/failed/skipped + mode/errors/schema/repair attempts. - `RunRequest`
- `prompt_id`, `prompt_version`, optional `profile_id`, `inputs`, `vars`, optional `execution` override, optional validation override.
- `RunResult`
- Output artifact, validation, raw output, prompt/profile/model metadata, hashes, timing, usage.
- `ArtifactRef` / `Artifact`
- Input reference and loaded content contracts.
- `RenderedPrompt` / `RenderedMessage`
- Provider-neutral rendered prompt.
- `GenerateRequest` / `GenerateResponse`
- Provider-neutral model I/O.
## 6. Interfaces and Adapters ## 6. Interfaces and Adapters
Primary ports: Primary ports:
- `profile.Repository` - `profile.Repository` (transitional prompt-definition lookup)
- `artifact.Reader` - `artifact.Reader`
- `prompt.Renderer` - `prompt.Renderer`
- `llm.Client` - `llm.Client`
- `validate.Validator` - `validate.Validator`
- `usecase.OutputRepairer` (usecase-local abstraction) - `usecase.OutputRepairer` (usecase-local)
Current adapters: Current adapters:
- Profile repository: filesystem YAML loader. - Prompt definition repository: filesystem YAML loader.
- Artifact reader: composite reader for `inline` and `file`. - Artifact readers: `file`, `inline` via composite reader.
- Prompt renderer: Go templates with input helper + vars. - Prompt renderer: Go templates with `input` helper.
- LLM adapter: OpenAI-compatible `/chat/completions`. - LLM adapter: OpenAI-compatible `/chat/completions` over `net/http`.
- Validator: standard validator with `none/basic/json/json_schema`. - Validator: standard validator (`none/basic/json/json_schema`).
- CLI/HTTP adapters: thin request mapping and response mapping. - CLI/HTTP adapters.
## 7. Validation and Repair Model ## 7. Validation and Repair Model
@@ -131,12 +136,11 @@ Validation modes:
Repair behavior: Repair behavior:
- Only applies to structured modes (`json`, `json_schema`). - Applies only to structured modes (`json`, `json_schema`).
- Attempted only when validation fails, repairer exists, and `repair_attempts > 0`. - Triggered only on failed validation and only when `repair_attempts > 0`.
- Bounded strictly by `repair_attempts`. - Strictly bounded by `repair_attempts`.
- Uses a narrow JSON-repair prompt and re-validates each attempt. - Uses a narrow repair prompt asking for corrected JSON only.
- If still invalid, run succeeds with failed validation and preserved final raw output. - Runtime validator/repair errors are run errors.
- Validator runtime/config errors are run errors.
## 8. Public Contracts ## 8. Public Contracts
@@ -147,86 +151,76 @@ Commands:
- `scriptorium run` - `scriptorium run`
- `scriptorium serve` - `scriptorium serve`
`run`: `run` flags:
- Required: `--profile-dir`, `--profile-id`, `--input`. - Required: `--profile-dir`, `--prompt-id`, `--input`.
- Optional: model/endpoint overrides (`--model`, `--llm-base-url`), vars, output path, schema dir, timeout. - Optional: `--profile-id`, `--var`, `--out`, `--llm-base-url`, `--model`, `--api-key-env`, `--temperature`, `--max-tokens`, `--schema-dir`, `--timeout`.
- Artifact bytes go to stdout (or `--out` file); summaries/errors go to stderr.
`serve`: Current transitional runtime behavior:
- Required: `--profile-dir`, `--llm-base-url`. - Prompt definitions may provide `default_profile` selection.
- Exposes HTTP run endpoint. - Execution-profile loading is deferred; execution settings must currently be supplied via run-time overrides.
### HTTP ### HTTP
- Endpoint: `POST /v1/runs`. - Endpoint: `POST /v1/runs`.
- Request maps to `RunRequest` (`profile_id`, `inputs`, `vars`, optional `model` override). - Request maps to `RunRequest` with `prompt_id` (required), `inputs`, optional `profile_id`, `vars`, optional execution override (`model` object).
- Response includes `artifact`, `validation`, `metadata`, `raw_model_output`. - Response includes `artifact`, `validation`, `metadata`, `raw_model_output`.
- Validation content failures are represented as `200` with `validation.status=failed`. - Validation content failures return `200` with failed validation status.
- Error responses are `{error:{code,message}}` with stable code mapping. - Error response shape: `{ "error": { "code": "...", "message": "..." } }`.
### Prompt Profile YAML ### Prompt Definition YAML
- `id`, `version`, `expected_inputs`, `templates`, `model_defaults`, `output_format`, `validation`. Current prompt-definition fields:
- Strict YAML decoding (`KnownFields`) rejects unknown fields.
- `validation.schema_path` required when `validation_mode=json_schema`.
- `validation.repair_attempts` must be non-negative.
### Metadata - `id`, `version`, optional `default_profile`, optional `description`
- `inputs[]` with `name`, `required`, optional `content_type`, optional `description`
- `templates[]` with `role` and either `content` or `content_file`
- `output_format`
- `validation` (`format`, `validation_mode`, `schema_path`, `repair_attempts`)
Current run metadata includes: Strict YAML decoding (`KnownFields`) is enabled.
- `run_id` (UUID v4) ### API Key Policy
- `profile_id`, `profile_version`, `profile_hash`
- `model_name`, `endpoint`, effective `model_params` - Raw API keys are not accepted in YAML, CLI flags, HTTP body, or domain metadata.
- `input_hashes`, `prompt_hash` - Auth is configured only by env var reference (`api_key_env`), resolved at request time by the LLM adapter.
- token usage
- start/end timestamps
- duration
- validation mode/status
- repair attempts used
## 9. Extension Points (Future Work) ## 9. Extension Points (Future Work)
Future features should plug into existing boundaries, not bypass them. Planned next extensions should reuse current boundaries:
Candidate extensions: - Execution-profile repository/loader implementation.
- Split transitional `internal/profile` into clearer prompt-definition/profile repositories.
- S3 artifact refs via `artifact.Reader` extension. - S3 artifact refs.
- Token budgeting in usecase/model-target policy layer. - Token budgeting/policy layer.
- Streaming LLM output via additional `llm.Client` methods/adapters. - Streaming generation.
- Batch execution as a separate use case (not hidden in single-run path). - Batch run use case.
- Additional LLM providers implementing `llm.Client`. - Additional provider adapters.
- Additional validators/modes in `validate`. - Additional validation modes.
- Additional profile repositories (embedded, remote, object storage).
These are future work, not part of current default behavior.
## 10. Architectural Guardrails ## 10. Architectural Guardrails
Contributors should preserve these constraints: - No D&D-specific logic in core Go packages.
- No D&D-specific behavior in core Go packages.
- No orchestration creep into Scriptorium. - No orchestration creep into Scriptorium.
- No unbounded repair loops. - No unbounded repair loops.
- No silent truncation/omission of rendered inputs or outputs. - No silent content truncation/omission.
- Do not log full artifacts/prompts by default. - Do not log full prompts/artifacts by default.
- Keep provider-specific wire/SDK details out of domain types. - Keep provider-specific wire/SDK details out of domain types.
- Keep adapter boundaries explicit and thin. - Keep adapters thin.
## 11. Testing Strategy ## 11. Testing Strategy
Protect behavior at boundaries and in usecase flow: Protect these behaviors with focused tests:
- Profile loading/parsing/validation errors. - Prompt-definition loading/validation errors.
- Artifact reading for inline/file + hash/content type behavior. - Artifact loading/hash/content-type behavior.
- Prompt rendering required inputs/template error behavior. - Prompt rendering required-input and template error paths.
- LLM adapter request/response/error/timeout behavior. - LLM adapter request/response/auth/error/timeout behavior.
- Runner success path and metadata population. - Runner success/failure/metadata behavior.
- Validation failure raw-output preservation. - Validation failure raw-output preservation.
- Successful/failed/bounded repair flows. - Bounded repair behavior.
- HTTP request mapping, response shape, and error mapping. - HTTP mapping and error mapping.
- CLI parsing helpers, required flags, and output stream separation. - CLI parsing and output stream separation.
Prefer focused unit tests and small integration-style tests with fake LLMs. Prefer small unit tests and minimal integration-style tests with fake LLMs.

54
internal/adapter/cli/run.go
View File

@@ -30,12 +30,13 @@ const (
type runConfig struct { type runConfig struct {
profileDir string profileDir string
promptID string
profileID string profileID string
inputRaw listFlag inputRaw listFlag
varRaw listFlag varRaw listFlag
outputPath string outputPath string
llmBaseURL string llmBaseURL string
llmAPIKey string apiKeyEnv string
model string model string
temperature float64 temperature float64
maxTokens int maxTokens int
@@ -43,6 +44,7 @@ type runConfig struct {
timeout time.Duration timeout time.Duration
llmBaseURLSet bool llmBaseURLSet bool
apiKeyEnvSet bool
modelSet bool modelSet bool
temperatureSet bool temperatureSet bool
maxTokensSet bool maxTokensSet bool
@@ -53,7 +55,6 @@ type serveConfig struct {
profileDir string profileDir string
schemaDir string schemaDir string
llmBaseURL string llmBaseURL string
llmAPIKey string
model string model string
timeout time.Duration timeout time.Duration
} }
@@ -115,7 +116,6 @@ func runCommand(args []string, stdout, stderr io.Writer) int {
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{ llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
BaseURL: cfg.llmBaseURL, BaseURL: cfg.llmBaseURL,
APIKey: cfg.llmAPIKey,
Model: cfg.model, Model: cfg.model,
Timeout: cfg.timeout, Timeout: cfg.timeout,
}) })
@@ -132,21 +132,24 @@ func runCommand(args []string, stdout, stderr io.Writer) int {
validate.NewStandardValidator(cfg.schemaDir), validate.NewStandardValidator(cfg.schemaDir),
) )
var modelOverride *domain.ModelTarget var modelOverride *domain.ExecutionTarget
if cfg.llmBaseURLSet || cfg.modelSet || cfg.temperatureSet || cfg.maxTokensSet { if cfg.llmBaseURLSet || cfg.modelSet || cfg.temperatureSet || cfg.maxTokensSet || cfg.apiKeyEnvSet {
modelOverride = &domain.ModelTarget{ modelOverride = &domain.ExecutionTarget{
Endpoint: cfg.llmBaseURL, Endpoint: cfg.llmBaseURL,
Model: cfg.model, Model: cfg.model,
Temperature: cfg.temperature, Temperature: cfg.temperature,
MaxTokens: cfg.maxTokens, MaxTokens: cfg.maxTokens,
TimeoutSeconds: int(cfg.timeout.Seconds()),
APIKeyEnv: cfg.apiKeyEnv,
} }
} }
res, runErr := runner.Run(context.Background(), domain.RunRequest{ res, runErr := runner.Run(context.Background(), domain.RunRequest{
PromptID: cfg.promptID,
ProfileID: cfg.profileID, ProfileID: cfg.profileID,
Inputs: inputs, Inputs: inputs,
Vars: varMappings, Vars: varMappings,
Model: modelOverride, Execution: modelOverride,
}) })
if runErr != nil { if runErr != nil {
fmt.Fprintf(stderr, "run error: %v\n", runErr) fmt.Fprintf(stderr, "run error: %v\n", runErr)
@@ -171,7 +174,6 @@ func serveCommand(args []string, stderr io.Writer) int {
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{ llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
BaseURL: cfg.llmBaseURL, BaseURL: cfg.llmBaseURL,
APIKey: cfg.llmAPIKey,
Model: cfg.model, Model: cfg.model,
Timeout: cfg.timeout, Timeout: cfg.timeout,
}) })
@@ -208,13 +210,14 @@ func parseRunArgs(args []string) (*runConfig, error) {
fs := flag.NewFlagSet("run", flag.ContinueOnError) fs := flag.NewFlagSet("run", flag.ContinueOnError)
fs.SetOutput(io.Discard) fs.SetOutput(io.Discard)
fs.StringVar(&cfg.profileDir, "profile-dir", "", "directory containing prompt profile YAML files") fs.StringVar(&cfg.profileDir, "profile-dir", "", "directory containing prompt definition YAML files")
fs.StringVar(&cfg.profileID, "profile-id", "", "profile ID to run") fs.StringVar(&cfg.promptID, "prompt-id", "", "prompt ID to run")
fs.StringVar(&cfg.profileID, "profile-id", "", "optional execution profile ID; if omitted, prompt default_profile is used")
fs.Var(&cfg.inputRaw, "input", "input mapping(s): name=path (repeatable, comma-separated)") fs.Var(&cfg.inputRaw, "input", "input mapping(s): name=path (repeatable, comma-separated)")
fs.Var(&cfg.varRaw, "var", "variable mapping(s): name=value (repeatable, comma-separated)") fs.Var(&cfg.varRaw, "var", "variable mapping(s): name=value (repeatable, comma-separated)")
fs.StringVar(&cfg.outputPath, "out", "", "optional output file path") fs.StringVar(&cfg.outputPath, "out", "", "optional output file path")
fs.StringVar(&cfg.llmBaseURL, "llm-base-url", "", "OpenAI-compatible base URL including /v1") fs.StringVar(&cfg.llmBaseURL, "llm-base-url", "", "OpenAI-compatible base URL including /v1")
fs.StringVar(&cfg.llmAPIKey, "llm-api-key", "", "optional API key") fs.StringVar(&cfg.apiKeyEnv, "api-key-env", "", "environment variable name containing API key")
fs.StringVar(&cfg.model, "model", "", "model name") fs.StringVar(&cfg.model, "model", "", "model name")
fs.Float64Var(&cfg.temperature, "temperature", 0, "optional temperature override") fs.Float64Var(&cfg.temperature, "temperature", 0, "optional temperature override")
fs.IntVar(&cfg.maxTokens, "max-tokens", 0, "optional max tokens override") fs.IntVar(&cfg.maxTokens, "max-tokens", 0, "optional max tokens override")
@@ -231,8 +234,8 @@ func parseRunArgs(args []string) (*runConfig, error) {
if strings.TrimSpace(cfg.profileDir) == "" { if strings.TrimSpace(cfg.profileDir) == "" {
return nil, errors.New("--profile-dir is required") return nil, errors.New("--profile-dir is required")
} }
if strings.TrimSpace(cfg.profileID) == "" { if strings.TrimSpace(cfg.promptID) == "" {
return nil, errors.New("--profile-id is required") return nil, errors.New("--prompt-id is required")
} }
if len(cfg.inputRaw) == 0 { if len(cfg.inputRaw) == 0 {
return nil, errors.New("at least one --input is required") return nil, errors.New("at least one --input is required")
@@ -243,6 +246,7 @@ func parseRunArgs(args []string) (*runConfig, error) {
cfg.outputPath = filepath.Clean(cfg.outputPath) cfg.outputPath = filepath.Clean(cfg.outputPath)
} }
cfg.llmBaseURLSet = flagWasSet(fs, "llm-base-url") cfg.llmBaseURLSet = flagWasSet(fs, "llm-base-url")
cfg.apiKeyEnvSet = flagWasSet(fs, "api-key-env")
cfg.modelSet = flagWasSet(fs, "model") cfg.modelSet = flagWasSet(fs, "model")
cfg.temperatureSet = flagWasSet(fs, "temperature") cfg.temperatureSet = flagWasSet(fs, "temperature")
cfg.maxTokensSet = flagWasSet(fs, "max-tokens") cfg.maxTokensSet = flagWasSet(fs, "max-tokens")
@@ -256,10 +260,9 @@ func parseServeArgs(args []string) (*serveConfig, error) {
fs.SetOutput(io.Discard) fs.SetOutput(io.Discard)
fs.StringVar(&cfg.addr, "addr", ":8080", "HTTP listen address") fs.StringVar(&cfg.addr, "addr", ":8080", "HTTP listen address")
fs.StringVar(&cfg.profileDir, "profile-dir", "", "directory containing prompt profile YAML files") fs.StringVar(&cfg.profileDir, "profile-dir", "", "directory containing prompt definition YAML files")
fs.StringVar(&cfg.schemaDir, "schema-dir", ".", "base directory for validation schemas") fs.StringVar(&cfg.schemaDir, "schema-dir", ".", "base directory for validation schemas")
fs.StringVar(&cfg.llmBaseURL, "llm-base-url", "", "OpenAI-compatible base URL including /v1") fs.StringVar(&cfg.llmBaseURL, "llm-base-url", "", "OpenAI-compatible base URL including /v1")
fs.StringVar(&cfg.llmAPIKey, "llm-api-key", "", "optional API key")
fs.StringVar(&cfg.model, "model", "", "optional default model") fs.StringVar(&cfg.model, "model", "", "optional default model")
fs.DurationVar(&cfg.timeout, "timeout", 10*time.Minute, "LLM request timeout") fs.DurationVar(&cfg.timeout, "timeout", 10*time.Minute, "LLM request timeout")
@@ -351,14 +354,15 @@ func printSummary(stderr io.Writer, res *domain.RunResult) {
if res == nil { if res == nil {
return return
} }
fmt.Fprintf(stderr, "profile=%s@%s model=%s validation=%s mode=%s validation_errors=%d prompt_hash=%s inputs=%d usage=%d/%d/%d\n", fmt.Fprintf(stderr, "prompt=%s@%s selected_profile=%s model=%s validation=%s mode=%s validation_errors=%d prompt_hash=%s inputs=%d usage=%d/%d/%d\n",
res.ProfileID, res.PromptID,
res.ProfileVersion, res.PromptVersion,
res.SelectedProfileID,
res.ModelName, res.ModelName,
res.Validation.Status, res.Validation.Status,
res.Validation.Mode, res.Validation.Mode,
len(res.Validation.Errors), len(res.Validation.Errors),
res.PromptHash, res.RenderedPromptHash,
len(res.InputHashes), len(res.InputHashes),
res.Usage.PromptTokens, res.Usage.PromptTokens,
res.Usage.CompletionTokens, res.Usage.CompletionTokens,
@@ -368,6 +372,6 @@ func printSummary(stderr io.Writer, res *domain.RunResult) {
func printUsage(w io.Writer) { func printUsage(w io.Writer) {
fmt.Fprintln(w, "usage: scriptorium <run|serve> ...") fmt.Fprintln(w, "usage: scriptorium <run|serve> ...")
fmt.Fprintln(w, " run: scriptorium run --profile-dir DIR --profile-id ID --input name=path [--input ...] [--llm-base-url URL] [--model NAME] [--var k=v] [--out path] [--timeout 10m]") fmt.Fprintln(w, " run: scriptorium run --profile-dir DIR --prompt-id ID --input name=path [--input ...] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--var k=v] [--out path] [--timeout 10m]")
fmt.Fprintln(w, " serve: scriptorium serve --addr :8080 --profile-dir DIR --llm-base-url URL [--schema-dir DIR] [--llm-api-key KEY] [--model NAME] [--timeout 10m]") fmt.Fprintln(w, " serve: scriptorium serve --addr :8080 --profile-dir DIR --llm-base-url URL [--schema-dir DIR] [--model NAME] [--timeout 10m]")
} }

29
internal/adapter/cli/run_test.go
View File

@@ -51,17 +51,17 @@ func TestParseMappingsMalformed(t *testing.T) {
} }
func TestParseRunArgsRequiredFlags(t *testing.T) { func TestParseRunArgsRequiredFlags(t *testing.T) {
_, err := parseRunArgs([]string{"--profile-id", "p", "--input", "a=b", "--llm-base-url", "http://x/v1", "--model", "m"}) _, err := parseRunArgs([]string{"--prompt-id", "p", "--input", "a=b", "--llm-base-url", "http://x/v1", "--model", "m"})
if err == nil { if err == nil {
t.Fatal("expected missing --profile-dir error") t.Fatal("expected missing --profile-dir error")
} }
_, err = parseRunArgs([]string{"--profile-dir", "./profiles", "--input", "a=b", "--llm-base-url", "http://x/v1", "--model", "m"}) _, err = parseRunArgs([]string{"--profile-dir", "./profiles", "--input", "a=b", "--llm-base-url", "http://x/v1", "--model", "m"})
if err == nil { if err == nil {
t.Fatal("expected missing --profile-id error") t.Fatal("expected missing --prompt-id error")
} }
_, err = parseRunArgs([]string{"--profile-dir", "./profiles", "--profile-id", "p", "--llm-base-url", "http://x/v1", "--model", "m"}) _, err = parseRunArgs([]string{"--profile-dir", "./profiles", "--prompt-id", "p", "--llm-base-url", "http://x/v1", "--model", "m"})
if err == nil { if err == nil {
t.Fatal("expected missing --input error") t.Fatal("expected missing --input error")
} }
@@ -70,7 +70,7 @@ func TestParseRunArgsRequiredFlags(t *testing.T) {
func TestParseRunArgsAllowsOmittedModelAndBaseURL(t *testing.T) { func TestParseRunArgsAllowsOmittedModelAndBaseURL(t *testing.T) {
cfg, err := parseRunArgs([]string{ cfg, err := parseRunArgs([]string{
"--profile-dir", "./profiles", "--profile-dir", "./profiles",
"--profile-id", "p", "--prompt-id", "p",
"--input", "a=b", "--input", "a=b",
}) })
if err != nil { if err != nil {
@@ -107,7 +107,7 @@ func TestParseServeArgsRequiredFlags(t *testing.T) {
func TestParseRunArgsTimeout(t *testing.T) { func TestParseRunArgsTimeout(t *testing.T) {
cfg, err := parseRunArgs([]string{ cfg, err := parseRunArgs([]string{
"--profile-dir", "./profiles", "--profile-dir", "./profiles",
"--profile-id", "p", "--prompt-id", "p",
"--input", "a=b", "--input", "a=b",
"--llm-base-url", "http://x/v1", "--llm-base-url", "http://x/v1",
"--model", "m", "--model", "m",
@@ -121,7 +121,7 @@ func TestParseRunArgsTimeout(t *testing.T) {
cfg, err = parseRunArgs([]string{ cfg, err = parseRunArgs([]string{
"--profile-dir", "./profiles", "--profile-dir", "./profiles",
"--profile-id", "p", "--prompt-id", "p",
"--input", "a=b", "--input", "a=b",
"--llm-base-url", "http://x/v1", "--llm-base-url", "http://x/v1",
"--model", "m", "--model", "m",
@@ -156,7 +156,7 @@ func TestRunCommandVarsOptional(t *testing.T) {
code := runCommand([]string{ code := runCommand([]string{
"--profile-dir", "./profiles", "--profile-dir", "./profiles",
"--profile-id", "p", "--prompt-id", "p",
"--input", "transcript=./t.md", "--input", "transcript=./t.md",
"--llm-base-url", "://bad-url", "--llm-base-url", "://bad-url",
"--model", "m", "--model", "m",
@@ -184,18 +184,19 @@ func TestWriteOutputAndSummaryUseSeparateWriters(t *testing.T) {
t.Fatalf("unexpected writeOutput error: %v", err) t.Fatalf("unexpected writeOutput error: %v", err)
} }
printSummary(&stderr, &domain.RunResult{ printSummary(&stderr, &domain.RunResult{
ProfileID: "p", PromptID: "p",
ProfileVersion: "1", PromptVersion: "1",
ModelName: "m", SelectedProfileID: "exec",
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic}, ModelName: "m",
PromptHash: "h", Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic},
InputHashes: map[string]string{"in": "x"}, RenderedPromptHash: "h",
InputHashes: map[string]string{"in": "x"},
}) })
if stdout.String() != "artifact-body" { if stdout.String() != "artifact-body" {
t.Fatalf("expected artifact output on stdout, got %q", stdout.String()) t.Fatalf("expected artifact output on stdout, got %q", stdout.String())
} }
if !strings.Contains(stderr.String(), "profile=p@1") { if !strings.Contains(stderr.String(), "prompt=p@1") {
t.Fatalf("expected summary on stderr, got %q", stderr.String()) t.Fatalf("expected summary on stderr, got %q", stderr.String())
} }
} }

50
internal/adapter/http/dto.go
View File

@@ -5,11 +5,12 @@ import (
) )
type runRequestDTO struct { type runRequestDTO struct {
ProfileID string `json:"profile_id"` PromptID string `json:"prompt_id"`
ProfileVersion string `json:"profile_version,omitempty"` PromptVersion string `json:"prompt_version,omitempty"`
Inputs map[string]inputRefDTO `json:"inputs"` ProfileID string `json:"profile_id,omitempty"`
Vars map[string]string `json:"vars,omitempty"` Inputs map[string]inputRefDTO `json:"inputs"`
Model *modelOverrideRequestDTO `json:"model,omitempty"` Vars map[string]string `json:"vars,omitempty"`
Model *modelOverrideRequestDTO `json:"model,omitempty"`
} }
type inputRefDTO struct { type inputRefDTO struct {
@@ -19,12 +20,15 @@ type inputRefDTO struct {
} }
type modelOverrideRequestDTO struct { type modelOverrideRequestDTO struct {
Endpoint string `json:"endpoint,omitempty"` Endpoint string `json:"endpoint,omitempty"`
Model string `json:"model,omitempty"` Model string `json:"model,omitempty"`
Temperature float64 `json:"temperature,omitempty"` Temperature float64 `json:"temperature,omitempty"`
MaxTokens int `json:"max_tokens,omitempty"` MaxTokens int `json:"max_tokens,omitempty"`
TopP float64 `json:"top_p,omitempty"` TopP float64 `json:"top_p,omitempty"`
TimeoutSeconds int `json:"timeout_seconds,omitempty"` TimeoutSeconds int `json:"timeout_seconds,omitempty"`
ReasoningEffort string `json:"reasoning_effort,omitempty"`
APIKeyEnv string `json:"api_key_env,omitempty"`
ExtraParams map[string]string `json:"extra_params,omitempty"`
} }
type runResponseDTO struct { type runResponseDTO struct {
@@ -45,14 +49,15 @@ type artifactDTO struct {
type metadataDTO struct { type metadataDTO struct {
RunID string `json:"run_id"` RunID string `json:"run_id"`
ProfileID string `json:"profile_id"` PromptID string `json:"prompt_id"`
ProfileVersion string `json:"profile_version"` PromptVersion string `json:"prompt_version"`
ProfileHash string `json:"profile_hash"` PromptHash string `json:"prompt_hash"`
RenderedPromptHash string `json:"rendered_prompt_hash"`
SelectedProfileID string `json:"selected_profile_id"`
ModelName string `json:"model_name"` ModelName string `json:"model_name"`
Endpoint string `json:"endpoint"` Endpoint string `json:"endpoint"`
ModelParams modelParamsDTO `json:"model_params"` ModelParams modelParamsDTO `json:"model_params"`
InputHashes map[string]string `json:"input_hashes"` InputHashes map[string]string `json:"input_hashes"`
PromptHash string `json:"prompt_hash"`
Usage tokenUsageDTO `json:"usage"` Usage tokenUsageDTO `json:"usage"`
StartTime time.Time `json:"start_time"` StartTime time.Time `json:"start_time"`
EndTime time.Time `json:"end_time"` EndTime time.Time `json:"end_time"`
@@ -63,12 +68,15 @@ type metadataDTO struct {
} }
type modelParamsDTO struct { type modelParamsDTO struct {
Endpoint string `json:"endpoint"` Endpoint string `json:"endpoint"`
Model string `json:"model"` Model string `json:"model"`
Temperature float64 `json:"temperature"` Temperature float64 `json:"temperature"`
MaxTokens int `json:"max_tokens"` MaxTokens int `json:"max_tokens"`
TopP float64 `json:"top_p"` TopP float64 `json:"top_p"`
TimeoutSeconds int `json:"timeout_seconds"` TimeoutSeconds int `json:"timeout_seconds"`
ReasoningEffort string `json:"reasoning_effort,omitempty"`
APIKeyEnv string `json:"api_key_env,omitempty"`
ExtraParams map[string]string `json:"extra_params,omitempty"`
} }
type tokenUsageDTO struct { type tokenUsageDTO struct {

68
internal/adapter/http/handler.go
View File

@@ -40,8 +40,8 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
return return
} }
if strings.TrimSpace(req.ProfileID) == "" { if strings.TrimSpace(req.PromptID) == "" {
writeError(w, http.StatusBadRequest, "invalid_request", "profile_id is required") writeError(w, http.StatusBadRequest, "invalid_request", "prompt_id is required")
return return
} }
if len(req.Inputs) == 0 { if len(req.Inputs) == 0 {
@@ -58,24 +58,28 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
} }
} }
var model *domain.ModelTarget var model *domain.ExecutionTarget
if req.Model != nil { if req.Model != nil {
model = &domain.ModelTarget{ model = &domain.ExecutionTarget{
Endpoint: req.Model.Endpoint, Endpoint: req.Model.Endpoint,
Model: req.Model.Model, Model: req.Model.Model,
Temperature: req.Model.Temperature, Temperature: req.Model.Temperature,
MaxTokens: req.Model.MaxTokens, MaxTokens: req.Model.MaxTokens,
TopP: req.Model.TopP, TopP: req.Model.TopP,
TimeoutSeconds: req.Model.TimeoutSeconds, TimeoutSeconds: req.Model.TimeoutSeconds,
ReasoningEffort: req.Model.ReasoningEffort,
APIKeyEnv: req.Model.APIKeyEnv,
ExtraParams: req.Model.ExtraParams,
} }
} }
res, err := h.runner.Run(r.Context(), domain.RunRequest{ res, err := h.runner.Run(r.Context(), domain.RunRequest{
ProfileID: req.ProfileID, PromptID: req.PromptID,
ProfileVersion: req.ProfileVersion, PromptVersion: req.PromptVersion,
Inputs: mappedInputs, ProfileID: req.ProfileID,
Vars: req.Vars, Inputs: mappedInputs,
Model: model, Vars: req.Vars,
Execution: model,
}) })
if err != nil { if err != nil {
status, code, message := mapRunError(err) status, code, message := mapRunError(err)
@@ -94,22 +98,26 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
}, },
Validation: mapValidation(res.Validation), Validation: mapValidation(res.Validation),
Metadata: metadataDTO{ Metadata: metadataDTO{
RunID: res.RunID, RunID: res.RunID,
ProfileID: res.ProfileID, PromptID: res.PromptID,
ProfileVersion: res.ProfileVersion, PromptVersion: res.PromptVersion,
ProfileHash: res.ProfileHash, PromptHash: res.PromptHash,
ModelName: res.ModelName, RenderedPromptHash: res.RenderedPromptHash,
Endpoint: res.Endpoint, SelectedProfileID: res.SelectedProfileID,
ModelName: res.ModelName,
Endpoint: res.Endpoint,
ModelParams: modelParamsDTO{ ModelParams: modelParamsDTO{
Endpoint: res.ModelParams.Endpoint, Endpoint: res.EffectiveModelParams.Endpoint,
Model: res.ModelParams.Model, Model: res.EffectiveModelParams.Model,
Temperature: res.ModelParams.Temperature, Temperature: res.EffectiveModelParams.Temperature,
MaxTokens: res.ModelParams.MaxTokens, MaxTokens: res.EffectiveModelParams.MaxTokens,
TopP: res.ModelParams.TopP, TopP: res.EffectiveModelParams.TopP,
TimeoutSeconds: res.ModelParams.TimeoutSeconds, TimeoutSeconds: res.EffectiveModelParams.TimeoutSeconds,
ReasoningEffort: res.EffectiveModelParams.ReasoningEffort,
APIKeyEnv: res.EffectiveModelParams.APIKeyEnv,
ExtraParams: res.EffectiveModelParams.ExtraParams,
}, },
InputHashes: res.InputHashes, InputHashes: res.InputHashes,
PromptHash: res.PromptHash,
Usage: tokenUsageDTO{ Usage: tokenUsageDTO{
PromptTokens: res.Usage.PromptTokens, PromptTokens: res.Usage.PromptTokens,
CompletionTokens: res.Usage.CompletionTokens, CompletionTokens: res.Usage.CompletionTokens,
@@ -140,11 +148,11 @@ func mapValidation(v domain.ValidationResult) validationDTO {
func mapRunError(err error) (int, string, string) { func mapRunError(err error) (int, string, string) {
switch { switch {
case errors.Is(err, profile.ErrProfileNotFound): case errors.Is(err, profile.ErrProfileNotFound):
return http.StatusNotFound, "profile_not_found", "profile not found" return http.StatusNotFound, "prompt_not_found", "prompt definition not found"
case errors.Is(err, usecase.ErrInvalidRequest): case errors.Is(err, usecase.ErrInvalidRequest):
return http.StatusBadRequest, "invalid_request", "invalid run request" return http.StatusBadRequest, "invalid_request", "invalid run request"
case errors.Is(err, usecase.ErrProfileLoad): case errors.Is(err, usecase.ErrProfileLoad):
return http.StatusBadRequest, "profile_load_failed", "failed to load profile" return http.StatusBadRequest, "prompt_load_failed", "failed to load prompt definition"
case errors.Is(err, usecase.ErrArtifactLoad): case errors.Is(err, usecase.ErrArtifactLoad):
return http.StatusBadRequest, "artifact_read_failed", "failed to read input artifact" return http.StatusBadRequest, "artifact_read_failed", "failed to read input artifact"
case errors.Is(err, usecase.ErrPromptRender): case errors.Is(err, usecase.ErrPromptRender):

45
internal/adapter/http/handler_test.go
View File

@@ -42,13 +42,15 @@ func TestHandlerPostRunsSuccess(t *testing.T) {
Size: 5, Size: 5,
Hash: "abc", Hash: "abc",
}, },
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true}, Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
ProfileID: "p1", PromptID: "prompt-1",
ProfileVersion: "1.0.0", PromptVersion: "1.0.0",
ProfileHash: "phash", PromptHash: "phash",
ModelName: "m1", RenderedPromptHash: "rhash",
Endpoint: "http://llm/v1", SelectedProfileID: "exec-default",
ModelParams: domain.ModelTarget{ ModelName: "m1",
Endpoint: "http://llm/v1",
EffectiveModelParams: domain.ExecutionTarget{
Endpoint: "http://llm/v1", Endpoint: "http://llm/v1",
Model: "m1", Model: "m1",
Temperature: 0.2, Temperature: 0.2,
@@ -57,7 +59,6 @@ func TestHandlerPostRunsSuccess(t *testing.T) {
TimeoutSeconds: 120, TimeoutSeconds: 120,
}, },
InputHashes: map[string]string{"transcript": "h1"}, InputHashes: map[string]string{"transcript": "h1"},
PromptHash: "ph",
Usage: domain.TokenUsage{PromptTokens: 1, CompletionTokens: 2, TotalTokens: 3}, Usage: domain.TokenUsage{PromptTokens: 1, CompletionTokens: 2, TotalTokens: 3},
StartTime: start, StartTime: start,
EndTime: end, EndTime: end,
@@ -68,7 +69,8 @@ func TestHandlerPostRunsSuccess(t *testing.T) {
h := NewHandler(r) h := NewHandler(r)
body := []byte(`{ body := []byte(`{
"profile_id": "p1", "prompt_id": "prompt-1",
"profile_id": "exec-default",
"inputs": { "inputs": {
"transcript": {"type": "file", "uri": "./t.md"} "transcript": {"type": "file", "uri": "./t.md"}
}, },
@@ -101,8 +103,8 @@ func TestHandlerPostRunsSuccess(t *testing.T) {
if metadata["run_id"] != "11111111-1111-4111-8111-111111111111" { if metadata["run_id"] != "11111111-1111-4111-8111-111111111111" {
t.Fatalf("unexpected metadata.run_id: %#v", metadata["run_id"]) t.Fatalf("unexpected metadata.run_id: %#v", metadata["run_id"])
} }
if metadata["profile_hash"] != "phash" { if metadata["prompt_hash"] != "phash" {
t.Fatalf("unexpected metadata.profile_hash: %#v", metadata["profile_hash"]) t.Fatalf("unexpected metadata.prompt_hash: %#v", metadata["prompt_hash"])
} }
usage := metadata["usage"].(map[string]any) usage := metadata["usage"].(map[string]any)
if usage["total_tokens"] != float64(3) { if usage["total_tokens"] != float64(3) {
@@ -122,14 +124,17 @@ func TestHandlerPostRunsSuccess(t *testing.T) {
t.Fatalf("expected raw model output hello, got %#v", resp["raw_model_output"]) t.Fatalf("expected raw model output hello, got %#v", resp["raw_model_output"])
} }
if r.last.ProfileID != "p1" { if r.last.PromptID != "prompt-1" {
t.Fatalf("expected request profile_id p1, got %q", r.last.ProfileID) t.Fatalf("expected request prompt_id prompt-1, got %q", r.last.PromptID)
} }
if r.last.Model == nil || r.last.Model.Model != "gpt-x" { if r.last.ProfileID != "exec-default" {
t.Fatalf("expected model override, got %#v", r.last.Model) t.Fatalf("expected request profile_id exec-default, got %q", r.last.ProfileID)
} }
if r.last.Model.TimeoutSeconds != 120 { if r.last.Execution == nil || r.last.Execution.Model != "gpt-x" {
t.Fatalf("expected timeout_seconds override 120, got %#v", r.last.Model) t.Fatalf("expected model override, got %#v", r.last.Execution)
}
if r.last.Execution.TimeoutSeconds != 120 {
t.Fatalf("expected timeout_seconds override 120, got %#v", r.last.Execution)
} }
} }
@@ -145,7 +150,7 @@ func TestHandlerInvalidJSON(t *testing.T) {
} }
} }
func TestHandlerMissingProfileID(t *testing.T) { func TestHandlerMissingPromptID(t *testing.T) {
h := NewHandler(&fakeRunner{}) h := NewHandler(&fakeRunner{})
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"inputs":{"x":{"type":"file","uri":"a"}}}`)) req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"inputs":{"x":{"type":"file","uri":"a"}}}`))
w := httptest.NewRecorder() w := httptest.NewRecorder()
@@ -173,7 +178,7 @@ func TestHandlerUsecaseErrorMapping(t *testing.T) {
for _, tc := range tests { for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
h := NewHandler(&fakeRunner{err: tc.err}) h := NewHandler(&fakeRunner{err: tc.err})
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"profile_id":"p","inputs":{"x":{"type":"file","uri":"a"}}}`)) req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p","inputs":{"x":{"type":"file","uri":"a"}}}`))
w := httptest.NewRecorder() w := httptest.NewRecorder()
h.ServeHTTP(w, req) h.ServeHTTP(w, req)
@@ -210,7 +215,7 @@ func TestHandlerValidationFailureStillSuccess(t *testing.T) {
}, },
}}) }})
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"profile_id":"p","inputs":{"x":{"type":"file","uri":"a"}}}`)) req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p","inputs":{"x":{"type":"file","uri":"a"}}}`))
w := httptest.NewRecorder() w := httptest.NewRecorder()
h.ServeHTTP(w, req) h.ServeHTTP(w, req)

141
internal/domain/domain.go
View File

@@ -43,34 +43,36 @@ const (
// RunRequest represents a request to generate a single artifact. // RunRequest represents a request to generate a single artifact.
type RunRequest struct { type RunRequest struct {
ProfileID string PromptID string
ProfileVersion string PromptVersion string
Inputs map[string]ArtifactRef ProfileID string
Vars map[string]string Inputs map[string]ArtifactRef
Model *ModelTarget Vars map[string]string
Validation *OutputContract Execution *ExecutionTarget
Metadata map[string]string Validation *OutputContract
Metadata map[string]string
} }
// RunResult represents the complete result of a prompt execution run. // RunResult represents the complete result of a prompt execution run.
type RunResult struct { type RunResult struct {
RunID string RunID string
Artifact Artifact Artifact Artifact
RawOutput string RawOutput string
Validation ValidationResult Validation ValidationResult
ProfileID string PromptID string
ProfileVersion string PromptVersion string
ProfileHash string PromptHash string
ModelName string RenderedPromptHash string
Endpoint string SelectedProfileID string
ModelParams ModelTarget ModelName string
InputHashes map[string]string Endpoint string
PromptHash string EffectiveModelParams ExecutionTarget
Usage TokenUsage InputHashes map[string]string
StartTime time.Time Usage TokenUsage
EndTime time.Time StartTime time.Time
Duration time.Duration EndTime time.Time
Error error Duration time.Duration
Error error
} }
// ArtifactRef represents a reference to an input artifact. // ArtifactRef represents a reference to an input artifact.
@@ -90,32 +92,58 @@ type Artifact struct {
Hash string Hash string
} }
// PromptProfile represents a configured prompt execution profile. // PromptDefinition represents a configured prompt execution definition.
type PromptProfile struct { type PromptDefinition struct {
ID string `yaml:"id"` ID string `yaml:"id"`
Version string `yaml:"version"` Version string `yaml:"version"`
DefaultProfile string `yaml:"default_profile"`
Description string `yaml:"description"` Description string `yaml:"description"`
ExpectedInputs []string `yaml:"expected_inputs"` Inputs []PromptInput `yaml:"inputs"`
Templates []PromptMessageTemplate `yaml:"templates"` Templates []PromptMessageTemplate `yaml:"templates"`
ModelDefaults ModelTarget `yaml:"model_defaults"`
OutputFormat OutputFormat `yaml:"output_format"` OutputFormat OutputFormat `yaml:"output_format"`
Validation OutputContract `yaml:"validation"` Validation OutputContract `yaml:"validation"`
} }
// PromptMessageTemplate defines a template for a chat message. // PromptInput describes one named input expected by a prompt definition.
type PromptMessageTemplate struct { type PromptInput struct {
Role string `yaml:"role"` Name string `yaml:"name"`
Content string `yaml:"content"` Required bool `yaml:"required"`
ContentType string `yaml:"content_type"`
Description string `yaml:"description"`
} }
// ModelTarget represents the LLM endpoint and configuration. // PromptMessageTemplate defines a template for a chat message.
type ModelTarget struct { type PromptMessageTemplate struct {
Endpoint string `yaml:"endpoint"` Role string `yaml:"role"`
Model string `yaml:"model"` Content string `yaml:"content"`
Temperature float64 `yaml:"temperature"` ContentFile string `yaml:"content_file"`
MaxTokens int `yaml:"max_tokens"` }
TopP float64 `yaml:"top_p"`
TimeoutSeconds int `yaml:"timeout_seconds"` // ExecutionProfile describes how and where to execute a model.
type ExecutionProfile struct {
ID string `yaml:"id"`
Endpoint string `yaml:"endpoint"`
Model string `yaml:"model"`
Temperature float64 `yaml:"temperature"`
MaxTokens int `yaml:"max_tokens"`
TopP float64 `yaml:"top_p"`
TimeoutSeconds int `yaml:"timeout_seconds"`
ReasoningEffort string `yaml:"reasoning_effort"`
APIKeyEnv string `yaml:"api_key_env"`
ExtraParams map[string]string `yaml:"extra_params"`
}
// ExecutionTarget represents effective model runtime settings for a run.
type ExecutionTarget struct {
Endpoint string `yaml:"endpoint" json:"endpoint"`
Model string `yaml:"model" json:"model"`
Temperature float64 `yaml:"temperature" json:"temperature"`
MaxTokens int `yaml:"max_tokens" json:"max_tokens"`
TopP float64 `yaml:"top_p" json:"top_p"`
TimeoutSeconds int `yaml:"timeout_seconds" json:"timeout_seconds"`
ReasoningEffort string `yaml:"reasoning_effort" json:"reasoning_effort"`
APIKeyEnv string `yaml:"api_key_env" json:"api_key_env"`
ExtraParams map[string]string `yaml:"extra_params" json:"extra_params"`
} }
// OutputContract defines the requirements for the output artifact. // OutputContract defines the requirements for the output artifact.
@@ -140,7 +168,7 @@ type RenderedMessage struct {
// GenerateRequest is the internal request passed to the LLM client. // GenerateRequest is the internal request passed to the LLM client.
type GenerateRequest struct { type GenerateRequest struct {
Prompt RenderedPrompt Prompt RenderedPrompt
Target ModelTarget Target ExecutionTarget
} }
// GenerateResponse is the response received from the LLM client. // GenerateResponse is the response received from the LLM client.
@@ -168,19 +196,20 @@ type ValidationResult struct {
// RunMetadata contains auditing information for a run. // RunMetadata contains auditing information for a run.
type RunMetadata struct { type RunMetadata struct {
RunID string RunID string
ProfileID string PromptID string
ProfileVersion string PromptVersion string
ProfileHash string PromptHash string
PromptHash string RenderedPromptHash string
InputHashes map[string]string SelectedProfileID string
ModelEndpoint string InputHashes map[string]string
ModelName string ModelEndpoint string
Params ModelTarget ModelName string
Timestamp time.Time Params ExecutionTarget
Duration time.Duration Timestamp time.Time
Usage TokenUsage Duration time.Duration
ValidationMode ValidationMode Usage TokenUsage
ValidationStatus ValidationStatus ValidationMode ValidationMode
RepairAttempts int ValidationStatus ValidationStatus
RepairAttempts int
} }

12
internal/llm/openai_compatible_client.go
View File

@@ -7,6 +7,7 @@ import (
"errors" "errors"
"fmt" "fmt"
"io" "io"
"os"
"net/http" "net/http"
"net/url" "net/url"
"strings" "strings"
@@ -25,7 +26,6 @@ var (
type OpenAICompatibleConfig struct { type OpenAICompatibleConfig struct {
BaseURL string BaseURL string
APIKey string
Model string Model string
Timeout time.Duration Timeout time.Duration
HTTPClient *http.Client HTTPClient *http.Client
@@ -33,7 +33,6 @@ type OpenAICompatibleConfig struct {
type OpenAICompatibleClient struct { type OpenAICompatibleClient struct {
baseURL string baseURL string
apiKey string
defaultModel string defaultModel string
timeout time.Duration timeout time.Duration
httpClient *http.Client httpClient *http.Client
@@ -64,7 +63,6 @@ func NewOpenAICompatibleClient(cfg OpenAICompatibleConfig) (*OpenAICompatibleCli
return &OpenAICompatibleClient{ return &OpenAICompatibleClient{
baseURL: strings.TrimRight(baseURL, "/"), baseURL: strings.TrimRight(baseURL, "/"),
apiKey: cfg.APIKey,
defaultModel: cfg.Model, defaultModel: cfg.Model,
timeout: timeout, timeout: timeout,
httpClient: client, httpClient: client,
@@ -125,8 +123,12 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
return nil, fmt.Errorf("%w: failed to create request: %v", ErrRequestFailed, err) return nil, fmt.Errorf("%w: failed to create request: %v", ErrRequestFailed, err)
} }
httpReq.Header.Set("Content-Type", "application/json") httpReq.Header.Set("Content-Type", "application/json")
if strings.TrimSpace(c.apiKey) != "" { if envName := strings.TrimSpace(req.Target.APIKeyEnv); envName != "" {
httpReq.Header.Set("Authorization", "Bearer "+c.apiKey) apiKey := strings.TrimSpace(os.Getenv(envName))
if apiKey == "" {
return nil, fmt.Errorf("%w: api key environment variable %q is not set", ErrInvalidRequest, envName)
}
httpReq.Header.Set("Authorization", "Bearer "+apiKey)
} }
effectiveTimeout := c.timeout effectiveTimeout := c.timeout

42
internal/llm/openai_compatible_client_test.go
View File

@@ -44,23 +44,24 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{ client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{
BaseURL: ts.URL + "/v1", BaseURL: ts.URL + "/v1",
APIKey: "secret-key",
Timeout: 2 * time.Second, Timeout: 2 * time.Second,
}) })
if err != nil { if err != nil {
t.Fatalf("unexpected constructor error: %v", err) t.Fatalf("unexpected constructor error: %v", err)
} }
t.Setenv("SCRIPTORIUM_TEST_API_KEY", "secret-key")
resp, err := client.Generate(context.Background(), domain.GenerateRequest{ resp, err := client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{ Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{
{Role: "system", Content: "You are helpful."}, {Role: "system", Content: "You are helpful."},
{Role: "user", Content: "Say hello"}, {Role: "user", Content: "Say hello"},
}}, }},
Target: domain.ModelTarget{ Target: domain.ExecutionTarget{
Model: "gpt-test", Model: "gpt-test",
Temperature: 0.4, Temperature: 0.4,
MaxTokens: 123, MaxTokens: 123,
TopP: 0.7, TopP: 0.7,
APIKeyEnv: "SCRIPTORIUM_TEST_API_KEY",
}, },
}) })
if err != nil { if err != nil {
@@ -110,7 +111,7 @@ func TestOpenAICompatibleClientNoAuthorizationHeaderWhenNoAPIKey(t *testing.T) {
_, err = client.Generate(context.Background(), domain.GenerateRequest{ _, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}}, Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ModelTarget{Model: "model"}, Target: domain.ExecutionTarget{Model: "model"},
}) })
if err != nil { if err != nil {
t.Fatalf("expected no error, got %v", err) t.Fatalf("expected no error, got %v", err)
@@ -120,6 +121,29 @@ func TestOpenAICompatibleClientNoAuthorizationHeaderWhenNoAPIKey(t *testing.T) {
} }
} }
func TestOpenAICompatibleClientAPIKeyEnvMissing(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1", Model: "model"})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ExecutionTarget{APIKeyEnv: "SCRIPTORIUM_MISSING_KEY"},
})
if err == nil {
t.Fatal("expected missing API key env error")
}
if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected ErrInvalidRequest, got %v", err)
}
}
func TestOpenAICompatibleClientModelFallbackFromConfig(t *testing.T) { func TestOpenAICompatibleClientModelFallbackFromConfig(t *testing.T) {
gotModel := "" gotModel := ""
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -139,7 +163,7 @@ func TestOpenAICompatibleClientModelFallbackFromConfig(t *testing.T) {
_, err = client.Generate(context.Background(), domain.GenerateRequest{ _, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}}, Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ModelTarget{}, Target: domain.ExecutionTarget{},
}) })
if err != nil { if err != nil {
t.Fatalf("expected no error, got %v", err) t.Fatalf("expected no error, got %v", err)
@@ -175,7 +199,7 @@ func TestOpenAICompatibleClientEndpointOverride(t *testing.T) {
resp, err := client.Generate(context.Background(), domain.GenerateRequest{ resp, err := client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}}, Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ModelTarget{Endpoint: overrideServer.URL + "/v1"}, Target: domain.ExecutionTarget{Endpoint: overrideServer.URL + "/v1"},
}) })
if err != nil { if err != nil {
t.Fatalf("expected no error, got %v", err) t.Fatalf("expected no error, got %v", err)
@@ -306,7 +330,7 @@ func TestOpenAICompatibleClientRequestTimeoutOverride(t *testing.T) {
resp, err := client.Generate(context.Background(), domain.GenerateRequest{ resp, err := client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}}, Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ModelTarget{TimeoutSeconds: 1}, Target: domain.ExecutionTarget{TimeoutSeconds: 1},
}) })
if err != nil { if err != nil {
t.Fatalf("expected request-level timeout override to succeed, got %v", err) t.Fatalf("expected request-level timeout override to succeed, got %v", err)
@@ -327,7 +351,7 @@ func TestOpenAICompatibleClientNegativeTimeoutRejected(t *testing.T) {
_, err = client.Generate(context.Background(), domain.GenerateRequest{ _, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}}, Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ModelTarget{TimeoutSeconds: -1}, Target: domain.ExecutionTarget{TimeoutSeconds: -1},
}) })
if err == nil { if err == nil {
t.Fatal("expected invalid request error") t.Fatal("expected invalid request error")
@@ -348,7 +372,7 @@ func TestOpenAICompatibleClientAllowsEmptyConfiguredBaseURL(t *testing.T) {
_, err = client.Generate(context.Background(), domain.GenerateRequest{ _, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}}, Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ModelTarget{Endpoint: "http://localhost:9999/v1"}, Target: domain.ExecutionTarget{Endpoint: "http://localhost:9999/v1"},
}) })
if err == nil { if err == nil {
t.Fatal("expected request failure due to unreachable endpoint") t.Fatal("expected request failure due to unreachable endpoint")
@@ -369,7 +393,7 @@ func TestOpenAICompatibleClientRequiresEndpointWhenUnsetEverywhere(t *testing.T)
_, err = client.Generate(context.Background(), domain.GenerateRequest{ _, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}}, Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ModelTarget{}, Target: domain.ExecutionTarget{},
}) })
if err == nil { if err == nil {
t.Fatal("expected endpoint-required error") t.Fatal("expected endpoint-required error")

39
internal/profile/filesystem_repository.go
View File

@@ -13,9 +13,9 @@ import (
) )
var ( var (
ErrProfileNotFound = errors.New("prompt profile not found") ErrProfileNotFound = errors.New("prompt definition not found")
ErrInvalidYAML = errors.New("invalid YAML format") ErrInvalidYAML = errors.New("invalid YAML format")
ErrInvalidProfile = errors.New("invalid profile configuration") ErrInvalidProfile = errors.New("invalid prompt definition configuration")
) )
type filesystemRepository struct { type filesystemRepository struct {
@@ -26,9 +26,9 @@ func NewFilesystemRepository(dir string) Repository {
return &filesystemRepository{dir: dir} return &filesystemRepository{dir: dir}
} }
func (r *filesystemRepository) GetProfile(ctx context.Context, id string, version string) (*domain.PromptProfile, error) { func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error) {
if strings.TrimSpace(id) == "" { if strings.TrimSpace(id) == "" {
return nil, fmt.Errorf("%w: profile id is required", ErrInvalidProfile) return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidProfile)
} }
files, err := os.ReadDir(r.dir) files, err := os.ReadDir(r.dir)
@@ -53,7 +53,7 @@ func (r *filesystemRepository) GetProfile(ctx context.Context, id string, versio
return nil, fmt.Errorf("failed to read profile file %s: %w", file.Name(), err) return nil, fmt.Errorf("failed to read profile file %s: %w", file.Name(), err)
} }
var prof domain.PromptProfile var prof domain.PromptDefinition
decoder := yaml.NewDecoder(bytes.NewReader(data)) decoder := yaml.NewDecoder(bytes.NewReader(data))
decoder.KnownFields(true) decoder.KnownFields(true)
if err := decoder.Decode(&prof); err != nil { if err := decoder.Decode(&prof); err != nil {
@@ -77,22 +77,33 @@ func (r *filesystemRepository) GetProfile(ctx context.Context, id string, versio
return nil, ErrProfileNotFound return nil, ErrProfileNotFound
} }
func validateProfile(p *domain.PromptProfile) error { func validateProfile(p *domain.PromptDefinition) error {
if p.ID == "" { if p.ID == "" {
return errors.New("profile id is required") return errors.New("prompt id is required")
} }
if p.Version == "" { if p.Version == "" {
return errors.New("profile version is required") return errors.New("prompt version is required")
} }
if len(p.Templates) == 0 { if len(p.Templates) == 0 {
return errors.New("at least one prompt template message is required") return errors.New("at least one prompt template message is required")
} }
if len(p.Inputs) == 0 {
return errors.New("at least one prompt input is required")
}
for i, input := range p.Inputs {
if strings.TrimSpace(input.Name) == "" {
return fmt.Errorf("input %d has empty name", i)
}
}
for i, t := range p.Templates { for i, t := range p.Templates {
if !isValidMessageRole(t.Role) { if !isValidMessageRole(t.Role) {
return fmt.Errorf("template message %d has invalid role %q", i, t.Role) return fmt.Errorf("template message %d has invalid role %q", i, t.Role)
} }
if t.Content == "" { if strings.TrimSpace(t.Content) == "" && strings.TrimSpace(t.ContentFile) == "" {
return fmt.Errorf("template message %d is missing content", i) return fmt.Errorf("template message %d must provide content or content_file", i)
}
if strings.TrimSpace(t.Content) != "" && strings.TrimSpace(t.ContentFile) != "" {
return fmt.Errorf("template message %d cannot set both content and content_file", i)
} }
} }
if !isValidOutputFormat(p.OutputFormat) { if !isValidOutputFormat(p.OutputFormat) {
@@ -107,17 +118,9 @@ func validateProfile(p *domain.PromptProfile) error {
if p.Validation.ValidationMode == domain.ValidationJSONSchema && strings.TrimSpace(p.Validation.SchemaPath) == "" { if p.Validation.ValidationMode == domain.ValidationJSONSchema && strings.TrimSpace(p.Validation.SchemaPath) == "" {
return errors.New("validation.schema_path is required when validation_mode is json_schema") return errors.New("validation.schema_path is required when validation_mode is json_schema")
} }
if p.ModelDefaults.TimeoutSeconds < 0 {
return errors.New("model_defaults.timeout_seconds must be greater than or equal to 0")
}
if p.Validation.Format != "" && p.Validation.Format != p.OutputFormat { if p.Validation.Format != "" && p.Validation.Format != p.OutputFormat {
return fmt.Errorf("validation format %q does not match output format %q", p.Validation.Format, p.OutputFormat) return fmt.Errorf("validation format %q does not match output format %q", p.Validation.Format, p.OutputFormat)
} }
for i, input := range p.ExpectedInputs {
if strings.TrimSpace(input) == "" {
return fmt.Errorf("expected input %d has empty name", i)
}
}
return nil return nil
} }

6
internal/profile/repository.go
View File

@@ -5,7 +5,9 @@ import (
"gitea.maximumdirect.net/eric/scriptorium/internal/domain" "gitea.maximumdirect.net/eric/scriptorium/internal/domain"
) )
// Repository handles loading and storing prompt profiles. // Repository is a transitional prompt-definition repository.
// It currently lives in internal/profile until package responsibilities
// are split in a follow-up refactor.
type Repository interface { type Repository interface {
GetProfile(ctx context.Context, id string, version string) (*domain.PromptProfile, error) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error)
} }

35
internal/profile/repository_test.go
View File

@@ -10,7 +10,7 @@ import (
"gitea.maximumdirect.net/eric/scriptorium/internal/domain" "gitea.maximumdirect.net/eric/scriptorium/internal/domain"
) )
func TestFilesystemRepository_GetProfile(t *testing.T) { func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
tmpDir, err := os.MkdirTemp("", "profile_test") tmpDir, err := os.MkdirTemp("", "profile_test")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -38,19 +38,19 @@ func TestFilesystemRepository_GetProfile(t *testing.T) {
repo := NewFilesystemRepository(tmpDir) repo := NewFilesystemRepository(tmpDir)
ctx := context.Background() ctx := context.Background()
t.Run("valid profile", func(t *testing.T) { t.Run("valid prompt definition", func(t *testing.T) {
p, err := repo.GetProfile(ctx, "test-profile", "") p, err := repo.GetPromptDefinition(ctx, "test-profile", "")
if err != nil { if err != nil {
t.Fatalf("expected no error, got %v", err) t.Fatalf("expected no error, got %v", err)
} }
if p == nil || p.ID != "test-profile" { if p == nil || p.ID != "test-profile" {
t.Errorf("expected profile test-profile, got %v", p) t.Errorf("expected prompt definition test-profile, got %v", p)
} }
if p.Version != "1.0.0" { if p.Version != "1.0.0" {
t.Fatalf("expected version 1.0.0, got %q", p.Version) t.Fatalf("expected version 1.0.0, got %q", p.Version)
} }
if len(p.ExpectedInputs) != 2 || p.ExpectedInputs[0] != "transcript" || p.ExpectedInputs[1] != "glossary" { if len(p.Inputs) != 2 || p.Inputs[0].Name != "transcript" || p.Inputs[1].Name != "glossary" {
t.Fatalf("unexpected expected_inputs: %#v", p.ExpectedInputs) t.Fatalf("unexpected inputs: %#v", p.Inputs)
} }
if len(p.Templates) != 2 { if len(p.Templates) != 2 {
t.Fatalf("expected 2 templates, got %d", len(p.Templates)) t.Fatalf("expected 2 templates, got %d", len(p.Templates))
@@ -64,48 +64,41 @@ func TestFilesystemRepository_GetProfile(t *testing.T) {
if p.Validation.ValidationMode != domain.ValidationBasic { if p.Validation.ValidationMode != domain.ValidationBasic {
t.Fatalf("expected validation mode basic, got %q", p.Validation.ValidationMode) t.Fatalf("expected validation mode basic, got %q", p.Validation.ValidationMode)
} }
if p.ModelDefaults.TimeoutSeconds != 120 { if p.DefaultProfile != "test-exec" {
t.Fatalf("expected timeout_seconds 120, got %d", p.ModelDefaults.TimeoutSeconds) t.Fatalf("expected default profile test-exec, got %q", p.DefaultProfile)
} }
}) })
t.Run("invalid YAML", func(t *testing.T) { t.Run("invalid YAML", func(t *testing.T) {
_, err := repo.GetProfile(ctx, "invalid_yaml", "") _, err := repo.GetPromptDefinition(ctx, "invalid_yaml", "")
if !errors.Is(err, ErrInvalidYAML) { if !errors.Is(err, ErrInvalidYAML) {
t.Errorf("expected ErrInvalidYAML, got %v", err) t.Errorf("expected ErrInvalidYAML, got %v", err)
} }
}) })
t.Run("missing ID", func(t *testing.T) { t.Run("missing ID", func(t *testing.T) {
_, err := repo.GetProfile(ctx, "missing-id", "") _, err := repo.GetPromptDefinition(ctx, "missing-id", "")
if !errors.Is(err, ErrProfileNotFound) { if !errors.Is(err, ErrProfileNotFound) {
t.Errorf("expected ErrProfileNotFound for profile with missing ID, got %v", err) t.Errorf("expected ErrProfileNotFound for profile with missing ID, got %v", err)
} }
}) })
t.Run("no templates", func(t *testing.T) { t.Run("no templates", func(t *testing.T) {
_, err := repo.GetProfile(ctx, "no-templates", "") _, err := repo.GetPromptDefinition(ctx, "no-templates", "")
if !errors.Is(err, ErrInvalidProfile) { if !errors.Is(err, ErrInvalidProfile) {
t.Errorf("expected ErrInvalidProfile for profile with no templates, got %v", err) t.Errorf("expected ErrInvalidProfile for profile with no templates, got %v", err)
} }
}) })
t.Run("json schema mode missing schema path", func(t *testing.T) { t.Run("json schema mode missing schema path", func(t *testing.T) {
_, err := repo.GetProfile(ctx, "json-schema-missing-path", "") _, err := repo.GetPromptDefinition(ctx, "json-schema-missing-path", "")
if !errors.Is(err, ErrInvalidProfile) { if !errors.Is(err, ErrInvalidProfile) {
t.Errorf("expected ErrInvalidProfile for json_schema profile without schema_path, got %v", err) t.Errorf("expected ErrInvalidProfile for json_schema profile without schema_path, got %v", err)
} }
}) })
t.Run("negative timeout seconds", func(t *testing.T) { t.Run("prompt definition not found", func(t *testing.T) {
_, err := repo.GetProfile(ctx, "negative-timeout", "") _, err := repo.GetPromptDefinition(ctx, "unknown", "")
if !errors.Is(err, ErrInvalidProfile) {
t.Errorf("expected ErrInvalidProfile for negative timeout_seconds, got %v", err)
}
})
t.Run("profile not found", func(t *testing.T) {
_, err := repo.GetProfile(ctx, "unknown", "")
if !errors.Is(err, ErrProfileNotFound) { if !errors.Is(err, ErrProfileNotFound) {
t.Errorf("expected ErrProfileNotFound, got %v", err) t.Errorf("expected ErrProfileNotFound, got %v", err)
} }

3
internal/profile/testdata/json_schema_missing_path.yaml vendored
View File

@@ -1,5 +1,8 @@
id: json-schema-missing-path id: json-schema-missing-path
version: "1.0.0" version: "1.0.0"
inputs:
- name: transcript
required: true
templates: templates:
- role: user - role: user
content: "Return JSON" content: "Return JSON"

3
internal/profile/testdata/missing_id.yaml vendored
View File

@@ -1,5 +1,8 @@
version: 1.0.0 version: 1.0.0
description: Missing ID description: Missing ID
inputs:
- name: transcript
required: true
templates: templates:
- role: system - role: system
content: Hello content: Hello

5
internal/profile/testdata/negative_timeout.yaml vendored
View File

@@ -1,10 +1,11 @@
id: negative-timeout id: negative-timeout
version: "1.0.0" version: "1.0.0"
inputs:
- name: transcript
required: true
templates: templates:
- role: user - role: user
content: "Say hi" content: "Say hi"
model_defaults:
timeout_seconds: -1
output_format: text output_format: text
validation: validation:
validation_mode: none validation_mode: none

3
internal/profile/testdata/no_templates.yaml vendored
View File

@@ -1,5 +1,8 @@
id: no-templates id: no-templates
version: 1.0.0 version: 1.0.0
inputs:
- name: transcript
required: true
templates: [] templates: []
output_format: text output_format: text
validation: validation:

17
internal/profile/testdata/valid.yaml vendored
View File

@@ -1,18 +1,19 @@
id: test-profile id: test-profile
version: "1.0.0" version: "1.0.0"
description: A valid test profile default_profile: test-exec
expected_inputs: description: A valid test prompt definition
- transcript inputs:
- glossary - name: transcript
required: true
content_type: text/markdown
- name: glossary
required: false
content_type: text/yaml
templates: templates:
- role: system - role: system
content: "You are a helpful assistant." content: "You are a helpful assistant."
- role: user - role: user
content: 'Analyze this: {{input "transcript"}}' content: 'Analyze this: {{input "transcript"}}'
model_defaults:
model: gpt-4o
temperature: 0.7
timeout_seconds: 120
output_format: markdown output_format: markdown
validation: validation:
validation_mode: basic validation_mode: basic

20
internal/prompt/go_renderer.go
View File

@@ -23,16 +23,19 @@ func NewGoRenderer() Renderer {
return &goRenderer{} return &goRenderer{}
} }
func (r *goRenderer) Render(ctx context.Context, profile *domain.PromptProfile, inputs map[string]*domain.Artifact, vars map[string]string) (*domain.RenderedPrompt, error) { func (r *goRenderer) Render(ctx context.Context, definition *domain.PromptDefinition, inputs map[string]*domain.Artifact, vars map[string]string) (*domain.RenderedPrompt, error) {
if profile == nil { if definition == nil {
return nil, fmt.Errorf("%w: nil profile", ErrRenderFailure) return nil, fmt.Errorf("%w: nil prompt definition", ErrRenderFailure)
} }
// 1. Verify required inputs // 1. Verify required inputs
for _, req := range profile.ExpectedInputs { for _, in := range definition.Inputs {
art, ok := inputs[req] if !in.Required {
continue
}
art, ok := inputs[in.Name]
if !ok || art == nil { if !ok || art == nil {
return nil, fmt.Errorf("%w: %s", ErrMissingRequiredInput, req) return nil, fmt.Errorf("%w: %s", ErrMissingRequiredInput, in.Name)
} }
} }
@@ -49,7 +52,7 @@ func (r *goRenderer) Render(ctx context.Context, profile *domain.PromptProfile,
var renderedMessages []domain.RenderedMessage var renderedMessages []domain.RenderedMessage
for i, tmplMsg := range profile.Templates { for i, tmplMsg := range definition.Templates {
select { select {
case <-ctx.Done(): case <-ctx.Done():
return nil, ctx.Err() return nil, ctx.Err()
@@ -61,6 +64,9 @@ func (r *goRenderer) Render(ctx context.Context, profile *domain.PromptProfile,
} }
// Parse and execute template // Parse and execute template
if tmplMsg.ContentFile != "" {
return nil, fmt.Errorf("%w: message %d: content_file is not implemented yet", ErrRenderFailure, i)
}
tmpl, err := template.New(fmt.Sprintf("msg_%d", i)).Funcs(funcs).Option("missingkey=error").Parse(tmplMsg.Content) tmpl, err := template.New(fmt.Sprintf("msg_%d", i)).Funcs(funcs).Option("missingkey=error").Parse(tmplMsg.Content)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: message %d: %v", ErrInvalidTemplate, i, err) return nil, fmt.Errorf("%w: message %d: %v", ErrInvalidTemplate, i, err)

2
internal/prompt/renderer.go
View File

@@ -7,5 +7,5 @@ import (
// Renderer renders prompt templates using named artifacts and variables. // Renderer renders prompt templates using named artifacts and variables.
type Renderer interface { type Renderer interface {
Render(ctx context.Context, profile *domain.PromptProfile, inputs map[string]*domain.Artifact, vars map[string]string) (*domain.RenderedPrompt, error) Render(ctx context.Context, definition *domain.PromptDefinition, inputs map[string]*domain.Artifact, vars map[string]string) (*domain.RenderedPrompt, error)
} }

16
internal/prompt/renderer_test.go
View File

@@ -12,9 +12,9 @@ func TestGoRenderer_Render(t *testing.T) {
renderer := NewGoRenderer() renderer := NewGoRenderer()
ctx := context.Background() ctx := context.Background()
profile := &domain.PromptProfile{ profile := &domain.PromptDefinition{
ID: "test-profile", ID: "test-profile",
ExpectedInputs: []string{"transcript"}, Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
Templates: []domain.PromptMessageTemplate{ Templates: []domain.PromptMessageTemplate{
{Role: "system", Content: "You are a {{.role}}."}, {Role: "system", Content: "You are a {{.role}}."},
{Role: "user", Content: "Analyze this: {{input \"transcript\"}}"}, {Role: "user", Content: "Analyze this: {{input \"transcript\"}}"},
@@ -54,7 +54,8 @@ func TestGoRenderer_Render(t *testing.T) {
}) })
t.Run("unknown input in template", func(t *testing.T) { t.Run("unknown input in template", func(t *testing.T) {
profileUnknown := &domain.PromptProfile{ profileUnknown := &domain.PromptDefinition{
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
Templates: []domain.PromptMessageTemplate{ Templates: []domain.PromptMessageTemplate{
{Role: "user", Content: "Hello {{input \"ghost\"}}"}, {Role: "user", Content: "Hello {{input \"ghost\"}}"},
}, },
@@ -69,7 +70,8 @@ func TestGoRenderer_Render(t *testing.T) {
}) })
t.Run("invalid template syntax", func(t *testing.T) { t.Run("invalid template syntax", func(t *testing.T) {
profileInvalid := &domain.PromptProfile{ profileInvalid := &domain.PromptDefinition{
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
Templates: []domain.PromptMessageTemplate{ Templates: []domain.PromptMessageTemplate{
{Role: "user", Content: "Hello {{.unclosed"}, {Role: "user", Content: "Hello {{.unclosed"},
}, },
@@ -81,7 +83,8 @@ func TestGoRenderer_Render(t *testing.T) {
}) })
t.Run("empty message role", func(t *testing.T) { t.Run("empty message role", func(t *testing.T) {
profileNoRole := &domain.PromptProfile{ profileNoRole := &domain.PromptDefinition{
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
Templates: []domain.PromptMessageTemplate{ Templates: []domain.PromptMessageTemplate{
{Role: "", Content: "Hello"}, {Role: "", Content: "Hello"},
}, },
@@ -93,7 +96,8 @@ func TestGoRenderer_Render(t *testing.T) {
}) })
t.Run("missing variable in template", func(t *testing.T) { t.Run("missing variable in template", func(t *testing.T) {
profileMissingVar := &domain.PromptProfile{ profileMissingVar := &domain.PromptDefinition{
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
Templates: []domain.PromptMessageTemplate{ Templates: []domain.PromptMessageTemplate{
{Role: "system", Content: "You are {{.missing}}"}, {Role: "system", Content: "You are {{.missing}}"},
}, },

19
internal/usecase/integration_test.go
View File

@@ -44,7 +44,12 @@ func TestRunnerIntegrationWithProfilesFixturesAndValidation(t *testing.T) {
) )
res, err := runner.Run(context.Background(), domain.RunRequest{ res, err := runner.Run(context.Background(), domain.RunRequest{
ProfileID: "generic.structured_events", PromptID: "generic.structured_events",
ProfileID: "exec",
Execution: &domain.ExecutionTarget{
Endpoint: "http://llm/v1",
Model: "test-model",
},
Inputs: map[string]domain.ArtifactRef{ Inputs: map[string]domain.ArtifactRef{
"transcript": { "transcript": {
Type: domain.ArtifactRefFile, Type: domain.ArtifactRefFile,
@@ -60,17 +65,17 @@ func TestRunnerIntegrationWithProfilesFixturesAndValidation(t *testing.T) {
t.Fatalf("expected no error, got %v", err) t.Fatalf("expected no error, got %v", err)
} }
if res.ProfileID != "generic.structured_events" { if res.PromptID != "generic.structured_events" {
t.Fatalf("unexpected profile id: %q", res.ProfileID) t.Fatalf("unexpected prompt id: %q", res.PromptID)
} }
if res.RunID == "" { if res.RunID == "" {
t.Fatal("expected run id") t.Fatal("expected run id")
} }
if res.ProfileHash == "" { if res.PromptHash == "" {
t.Fatal("expected profile hash") t.Fatal("expected prompt hash")
} }
if res.ProfileVersion != "1.0.0" { if res.PromptVersion != "1.0.0" {
t.Fatalf("unexpected profile version: %q", res.ProfileVersion) t.Fatalf("unexpected prompt version: %q", res.PromptVersion)
} }
if res.Validation.Status != domain.ValidationPassed { if res.Validation.Status != domain.ValidationPassed {
t.Fatalf("expected passed validation, got %q", res.Validation.Status) t.Fatalf("expected passed validation, got %q", res.Validation.Status)

2
internal/usecase/repairer.go
View File

@@ -17,7 +17,7 @@ type OutputRepairer interface {
type RepairRequest struct { type RepairRequest struct {
PreviousOutput string PreviousOutput string
ValidationErrors []string ValidationErrors []string
Target domain.ModelTarget Target domain.ExecutionTarget
Attempt int Attempt int
MaxAttempts int MaxAttempts int
Mode domain.ValidationMode Mode domain.ValidationMode

95
internal/usecase/runner.go
View File

@@ -21,7 +21,7 @@ import (
var ( var (
ErrInvalidRequest = errors.New("invalid run request") ErrInvalidRequest = errors.New("invalid run request")
ErrProfileLoad = errors.New("failed to load profile") ErrProfileLoad = errors.New("failed to load prompt definition")
ErrArtifactLoad = errors.New("failed to load artifact") ErrArtifactLoad = errors.New("failed to load artifact")
ErrPromptRender = errors.New("failed to render prompt") ErrPromptRender = errors.New("failed to render prompt")
ErrLLMGenerate = errors.New("failed to generate output") ErrLLMGenerate = errors.New("failed to generate output")
@@ -67,8 +67,8 @@ func NewRunnerWithRepairer(
} }
func (r *Runner) Run(ctx context.Context, req domain.RunRequest) (*domain.RunResult, error) { func (r *Runner) Run(ctx context.Context, req domain.RunRequest) (*domain.RunResult, error) {
if strings.TrimSpace(req.ProfileID) == "" { if strings.TrimSpace(req.PromptID) == "" {
return nil, fmt.Errorf("%w: profile id is required", ErrInvalidRequest) return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidRequest)
} }
runID, err := newRunID() runID, err := newRunID()
@@ -78,17 +78,32 @@ func (r *Runner) Run(ctx context.Context, req domain.RunRequest) (*domain.RunRes
start := time.Now().UTC() start := time.Now().UTC()
prof, err := r.profiles.GetProfile(ctx, req.ProfileID, req.ProfileVersion) def, err := r.profiles.GetPromptDefinition(ctx, req.PromptID, req.PromptVersion)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: %w", ErrProfileLoad, err) return nil, fmt.Errorf("%w: %w", ErrProfileLoad, err)
} }
profileHash, err := hashProfile(prof) promptDefinitionHash, err := hashPromptDefinition(def)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: failed to hash profile: %v", ErrProfileLoad, err) return nil, fmt.Errorf("%w: failed to hash prompt definition: %v", ErrProfileLoad, err)
} }
selectedProfileID := strings.TrimSpace(req.ProfileID)
effectiveModel := mergeModelTarget(prof.ModelDefaults, req.Model) if selectedProfileID == "" {
effectiveContract := resolveOutputContract(prof, req.Validation) selectedProfileID = strings.TrimSpace(def.DefaultProfile)
}
if selectedProfileID == "" {
return nil, fmt.Errorf("%w: profile id is required either in request or prompt default_profile", ErrInvalidRequest)
}
if req.Execution == nil {
return nil, fmt.Errorf("%w: execution override is required until execution profile loading is implemented", ErrInvalidRequest)
}
effectiveModel := mergeExecutionTarget(domain.ExecutionTarget{}, req.Execution)
if strings.TrimSpace(effectiveModel.Endpoint) == "" {
return nil, fmt.Errorf("%w: execution endpoint is required", ErrInvalidRequest)
}
if strings.TrimSpace(effectiveModel.Model) == "" {
return nil, fmt.Errorf("%w: execution model is required", ErrInvalidRequest)
}
effectiveContract := resolveOutputContract(def, req.Validation)
resolvedInputs := make(map[string]*domain.Artifact, len(req.Inputs)) resolvedInputs := make(map[string]*domain.Artifact, len(req.Inputs))
inputHashes := make(map[string]string, len(req.Inputs)) inputHashes := make(map[string]string, len(req.Inputs))
@@ -104,12 +119,12 @@ func (r *Runner) Run(ctx context.Context, req domain.RunRequest) (*domain.RunRes
inputHashes[name] = art.Hash inputHashes[name] = art.Hash
} }
renderedPrompt, err := r.renderer.Render(ctx, prof, resolvedInputs, req.Vars) renderedPrompt, err := r.renderer.Render(ctx, def, resolvedInputs, req.Vars)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: %w", ErrPromptRender, err) return nil, fmt.Errorf("%w: %w", ErrPromptRender, err)
} }
promptHash := hashRenderedPrompt(*renderedPrompt) renderedPromptHash := hashRenderedPrompt(*renderedPrompt)
genResp, err := r.llm.Generate(ctx, domain.GenerateRequest{ genResp, err := r.llm.Generate(ctx, domain.GenerateRequest{
Prompt: *renderedPrompt, Prompt: *renderedPrompt,
@@ -158,22 +173,23 @@ func (r *Runner) Run(ctx context.Context, req domain.RunRequest) (*domain.RunRes
end := time.Now().UTC() end := time.Now().UTC()
return &domain.RunResult{ return &domain.RunResult{
RunID: runID, RunID: runID,
Artifact: outputArtifact, Artifact: outputArtifact,
RawOutput: genResp.Content, RawOutput: genResp.Content,
Validation: validationResult, Validation: validationResult,
ProfileID: prof.ID, PromptID: def.ID,
ProfileVersion: prof.Version, PromptVersion: def.Version,
ProfileHash: profileHash, PromptHash: promptDefinitionHash,
ModelName: effectiveModel.Model, RenderedPromptHash: renderedPromptHash,
Endpoint: effectiveModel.Endpoint, SelectedProfileID: selectedProfileID,
ModelParams: effectiveModel, ModelName: effectiveModel.Model,
InputHashes: inputHashes, Endpoint: effectiveModel.Endpoint,
PromptHash: promptHash, EffectiveModelParams: effectiveModel,
Usage: genResp.Usage, InputHashes: inputHashes,
StartTime: start, Usage: genResp.Usage,
EndTime: end, StartTime: start,
Duration: end.Sub(start), EndTime: end,
Duration: end.Sub(start),
}, nil }, nil
} }
@@ -209,7 +225,7 @@ func (r *Runner) shouldAttemptRepair(contract domain.OutputContract, validationR
return contract.ValidationMode == domain.ValidationJSON || contract.ValidationMode == domain.ValidationJSONSchema return contract.ValidationMode == domain.ValidationJSON || contract.ValidationMode == domain.ValidationJSONSchema
} }
func mergeModelTarget(base domain.ModelTarget, override *domain.ModelTarget) domain.ModelTarget { func mergeExecutionTarget(base domain.ExecutionTarget, override *domain.ExecutionTarget) domain.ExecutionTarget {
if override == nil { if override == nil {
return base return base
} }
@@ -233,13 +249,26 @@ func mergeModelTarget(base domain.ModelTarget, override *domain.ModelTarget) dom
if override.TimeoutSeconds != 0 { if override.TimeoutSeconds != 0 {
out.TimeoutSeconds = override.TimeoutSeconds out.TimeoutSeconds = override.TimeoutSeconds
} }
if strings.TrimSpace(override.ReasoningEffort) != "" {
out.ReasoningEffort = override.ReasoningEffort
}
if strings.TrimSpace(override.APIKeyEnv) != "" {
out.APIKeyEnv = override.APIKeyEnv
}
if len(override.ExtraParams) > 0 {
cp := make(map[string]string, len(override.ExtraParams))
for k, v := range override.ExtraParams {
cp[k] = v
}
out.ExtraParams = cp
}
return out return out
} }
func resolveOutputContract(prof *domain.PromptProfile, override *domain.OutputContract) domain.OutputContract { func resolveOutputContract(def *domain.PromptDefinition, override *domain.OutputContract) domain.OutputContract {
contract := prof.Validation contract := def.Validation
if contract.Format == "" { if contract.Format == "" {
contract.Format = prof.OutputFormat contract.Format = def.OutputFormat
} }
if override != nil { if override != nil {
contract = *override contract = *override
@@ -283,8 +312,8 @@ func buildOutputArtifact(content string, format domain.OutputFormat) domain.Arti
} }
} }
func hashProfile(prof *domain.PromptProfile) (string, error) { func hashPromptDefinition(def *domain.PromptDefinition) (string, error) {
b, err := json.Marshal(prof) b, err := json.Marshal(def)
if err != nil { if err != nil {
return "", err return "", err
} }

591
internal/usecase/runner_test.go
View File

@@ -5,8 +5,6 @@ import (
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
"errors" "errors"
"os"
"path/filepath"
"regexp" "regexp"
"testing" "testing"
@@ -14,20 +12,20 @@ import (
"gitea.maximumdirect.net/eric/scriptorium/internal/validate" "gitea.maximumdirect.net/eric/scriptorium/internal/validate"
) )
type fakeProfileRepo struct { type fakePromptRepo struct {
profile *domain.PromptProfile def *domain.PromptDefinition
err error err error
lastID string lastID string
lastVersion string lastVersion string
} }
func (f *fakeProfileRepo) GetProfile(ctx context.Context, id string, version string) (*domain.PromptProfile, error) { func (f *fakePromptRepo) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error) {
f.lastID = id f.lastID = id
f.lastVersion = version f.lastVersion = version
if f.err != nil { if f.err != nil {
return nil, f.err return nil, f.err
} }
return f.profile, nil return f.def, nil
} }
type fakeArtifactReader struct { type fakeArtifactReader struct {
@@ -40,8 +38,8 @@ func (f *fakeArtifactReader) Read(ctx context.Context, ref domain.ArtifactRef) (
return nil, err return nil, err
} }
if art, ok := f.artifactsByURI[ref.URI]; ok { if art, ok := f.artifactsByURI[ref.URI]; ok {
copy := *art cp := *art
return &copy, nil return &cp, nil
} }
return nil, errors.New("artifact not found") return nil, errors.New("artifact not found")
} }
@@ -51,7 +49,7 @@ type fakeRenderer struct {
err error err error
} }
func (f *fakeRenderer) Render(ctx context.Context, profile *domain.PromptProfile, inputs map[string]*domain.Artifact, vars map[string]string) (*domain.RenderedPrompt, error) { func (f *fakeRenderer) Render(ctx context.Context, def *domain.PromptDefinition, inputs map[string]*domain.Artifact, vars map[string]string) (*domain.RenderedPrompt, error) {
if f.err != nil { if f.err != nil {
return nil, f.err return nil, f.err
} }
@@ -73,15 +71,11 @@ func (f *fakeLLM) Generate(ctx context.Context, req domain.GenerateRequest) (*do
} }
type fakeValidator struct { type fakeValidator struct {
result domain.ValidationResult result domain.ValidationResult
err error err error
called bool
lastContract domain.OutputContract
} }
func (f *fakeValidator) Validate(ctx context.Context, artifact *domain.Artifact, contract domain.OutputContract) (domain.ValidationResult, error) { func (f *fakeValidator) Validate(ctx context.Context, artifact *domain.Artifact, contract domain.OutputContract) (domain.ValidationResult, error) {
f.called = true
f.lastContract = contract
if f.err != nil { if f.err != nil {
return domain.ValidationResult{}, f.err return domain.ValidationResult{}, f.err
} }
@@ -92,12 +86,10 @@ type fakeRepairer struct {
responses []*domain.GenerateResponse responses []*domain.GenerateResponse
err error err error
calls int calls int
lastReq RepairRequest
} }
func (f *fakeRepairer) Repair(ctx context.Context, req RepairRequest) (*domain.GenerateResponse, error) { func (f *fakeRepairer) Repair(ctx context.Context, req RepairRequest) (*domain.GenerateResponse, error) {
f.calls++ f.calls++
f.lastReq = req
if f.err != nil { if f.err != nil {
return nil, f.err return nil, f.err
} }
@@ -112,156 +104,83 @@ func (f *fakeRepairer) Repair(ctx context.Context, req RepairRequest) (*domain.G
} }
func TestRunnerRunSuccessful(t *testing.T) { func TestRunnerRunSuccessful(t *testing.T) {
repo := &fakeProfileRepo{ repo := &fakePromptRepo{def: promptDef(domain.FormatMarkdown, domain.ValidationBasic, 0)}
profile: &domain.PromptProfile{
ID: "p1",
Version: "1.0.0",
OutputFormat: domain.FormatMarkdown,
ModelDefaults: domain.ModelTarget{
Endpoint: "ep1",
Model: "model-default",
Temperature: 0.4,
MaxTokens: 200,
TopP: 0.9,
TimeoutSeconds: 90,
},
Validation: domain.OutputContract{
ValidationMode: domain.ValidationBasic,
},
},
}
reader := &fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{ reader := &fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{
"a://t": {Body: []byte("transcript body"), Hash: hashString("transcript body")}, "a://t": {Body: []byte("transcript"), Hash: hashString("transcript")},
"a://g": {Body: []byte("glossary body"), Hash: hashString("glossary body")}, "a://g": {Body: []byte("glossary"), Hash: hashString("glossary")},
}}
renderer := &fakeRenderer{rendered: &domain.RenderedPrompt{Messages: []domain.RenderedMessage{
{Role: "system", Content: "System context"},
{Role: "user", Content: "Please summarize"},
}}}
llmClient := &fakeLLM{resp: &domain.GenerateResponse{
Content: "# recap\n- item",
Usage: domain.TokenUsage{
PromptTokens: 10,
CompletionTokens: 20,
TotalTokens: 30,
},
}} }}
renderer := &fakeRenderer{rendered: &domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "system", Content: "sys"}, {Role: "user", Content: "usr"}}}}
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "# recap", Usage: domain.TokenUsage{TotalTokens: 7}}}
runner := NewRunner(repo, reader, renderer, llmClient, nil) runner := NewRunner(repo, reader, renderer, llmClient, nil)
res, err := runner.Run(context.Background(), domain.RunRequest{ res, err := runner.Run(context.Background(), domain.RunRequest{
ProfileID: "p1", PromptID: "p",
ProfileVersion: "1.0.0", PromptVersion: "1",
ProfileID: "exec",
Inputs: map[string]domain.ArtifactRef{ Inputs: map[string]domain.ArtifactRef{
"transcript": {Type: domain.ArtifactRefFile, URI: "a://t"}, "transcript": {Type: domain.ArtifactRefFile, URI: "a://t"},
"glossary": {Type: domain.ArtifactRefFile, URI: "a://g"}, "glossary": {Type: domain.ArtifactRefFile, URI: "a://g"},
}, },
Model: &domain.ModelTarget{ Execution: &domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m", Temperature: 0.3, TimeoutSeconds: 90},
Model: "model-override",
Temperature: 0,
MaxTokens: 0,
TimeoutSeconds: 0,
},
}) })
if err != nil { if err != nil {
t.Fatalf("expected no error, got %v", err) t.Fatalf("expected no error, got %v", err)
} }
if res.PromptID != "p" || res.PromptVersion != "1" {
if res.ProfileID != "p1" || res.ProfileVersion != "1.0.0" { t.Fatalf("unexpected prompt metadata: %+v", res)
t.Fatalf("unexpected profile metadata: id=%q version=%q", res.ProfileID, res.ProfileVersion) }
if res.SelectedProfileID != "exec" {
t.Fatalf("expected selected profile exec, got %q", res.SelectedProfileID)
} }
if ok, _ := regexp.MatchString(`^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`, res.RunID); !ok { if ok, _ := regexp.MatchString(`^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`, res.RunID); !ok {
t.Fatalf("expected UUIDv4 run id, got %q", res.RunID) t.Fatalf("invalid run id: %q", res.RunID)
} }
if res.ProfileHash == "" { if res.PromptHash == "" || res.RenderedPromptHash == "" {
t.Fatal("expected non-empty profile hash") t.Fatal("expected prompt hashes")
} }
if res.ModelName != "model-override" { if res.EffectiveModelParams.Model != "m" || res.Endpoint != "http://llm/v1" {
t.Fatalf("expected model override to apply, got %q", res.ModelName) t.Fatalf("unexpected model params: %+v", res.EffectiveModelParams)
} }
if res.Endpoint != "ep1" { if res.RawOutput != "# recap" {
t.Fatalf("expected endpoint from profile default, got %q", res.Endpoint) t.Fatalf("expected raw output, got %q", res.RawOutput)
}
if res.Artifact.ContentType != "text/markdown" {
t.Fatalf("expected markdown content type, got %q", res.Artifact.ContentType)
}
if string(res.Artifact.Body) != "# recap\n- item" {
t.Fatalf("unexpected artifact body: %q", string(res.Artifact.Body))
}
if res.RawOutput != "# recap\n- item" {
t.Fatalf("expected raw output preserved, got %q", res.RawOutput)
} }
if res.Validation.Status != domain.ValidationSkipped { if res.Validation.Status != domain.ValidationSkipped {
t.Fatalf("expected skipped validation, got %q", res.Validation.Status) t.Fatalf("expected skipped validation, got %q", res.Validation.Status)
} }
if res.Validation.Mode != domain.ValidationBasic {
t.Fatalf("expected validation mode basic in skipped result, got %q", res.Validation.Mode)
}
if res.PromptHash == "" {
t.Fatal("expected non-empty prompt hash")
}
if res.Usage.TotalTokens != 30 {
t.Fatalf("expected usage to propagate, got %+v", res.Usage)
}
if res.StartTime.IsZero() || res.EndTime.IsZero() {
t.Fatal("expected start and end times")
}
if res.EndTime.Before(res.StartTime) {
t.Fatalf("expected end >= start, got start=%v end=%v", res.StartTime, res.EndTime)
}
if res.Duration < 0 {
t.Fatalf("expected non-negative duration, got %s", res.Duration)
}
if got := res.InputHashes["transcript"]; got != hashString("transcript body") {
t.Fatalf("unexpected transcript hash: %q", got)
}
if got := res.InputHashes["glossary"]; got != hashString("glossary body") {
t.Fatalf("unexpected glossary hash: %q", got)
}
if llmClient.lastReq.Target.Temperature != 0.4 {
t.Fatalf("expected zero-valued request field not to override default temperature, got %v", llmClient.lastReq.Target.Temperature)
}
if llmClient.lastReq.Target.TimeoutSeconds != 90 { if llmClient.lastReq.Target.TimeoutSeconds != 90 {
t.Fatalf("expected zero-valued request timeout not to override default timeout, got %d", llmClient.lastReq.Target.TimeoutSeconds) t.Fatalf("expected timeout propagation, got %d", llmClient.lastReq.Target.TimeoutSeconds)
}
if res.ModelParams.Model != "model-override" || res.ModelParams.Endpoint != "ep1" {
t.Fatalf("expected effective model params in result, got %+v", res.ModelParams)
} }
} }
func TestRunnerRunProfileLoadFailure(t *testing.T) { func TestRunnerRunPromptLoadFailure(t *testing.T) {
runner := NewRunner( runner := NewRunner(&fakePromptRepo{err: errors.New("boom")}, &fakeArtifactReader{}, &fakeRenderer{}, &fakeLLM{}, nil)
&fakeProfileRepo{err: errors.New("boom")}, _, err := runner.Run(context.Background(), domain.RunRequest{PromptID: "p"})
&fakeArtifactReader{},
&fakeRenderer{},
&fakeLLM{},
nil,
)
_, err := runner.Run(context.Background(), domain.RunRequest{ProfileID: "p"})
if !errors.Is(err, ErrProfileLoad) { if !errors.Is(err, ErrProfileLoad) {
t.Fatalf("expected ErrProfileLoad, got %v", err) t.Fatalf("expected ErrProfileLoad, got %v", err)
} }
} }
func TestMergeModelTargetTimeoutOverride(t *testing.T) { func TestRunnerRunMissingProfileSelection(t *testing.T) {
base := domain.ModelTarget{TimeoutSeconds: 30} repo := &fakePromptRepo{def: &domain.PromptDefinition{ID: "p", Version: "1", Inputs: []domain.PromptInput{{Name: "transcript", Required: true}}, Templates: []domain.PromptMessageTemplate{{Role: "user", Content: "x"}}, OutputFormat: domain.FormatText, Validation: domain.OutputContract{ValidationMode: domain.ValidationNone}}}
override := &domain.ModelTarget{TimeoutSeconds: 75} runner := NewRunner(repo, &fakeArtifactReader{}, &fakeRenderer{}, &fakeLLM{}, nil)
_, err := runner.Run(context.Background(), domain.RunRequest{PromptID: "p"})
if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected invalid request, got %v", err)
}
}
got := mergeModelTarget(base, override) func TestRunnerRunMissingExecutionOverride(t *testing.T) {
if got.TimeoutSeconds != 75 { repo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)}
t.Fatalf("expected timeout override to apply, got %d", got.TimeoutSeconds) runner := NewRunner(repo, &fakeArtifactReader{}, &fakeRenderer{}, &fakeLLM{}, nil)
_, err := runner.Run(context.Background(), domain.RunRequest{PromptID: "p", ProfileID: "exec"})
if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected invalid request, got %v", err)
} }
} }
func TestRunnerRunArtifactLoadFailure(t *testing.T) { func TestRunnerRunArtifactLoadFailure(t *testing.T) {
runner := NewRunner( runner := NewRunner(
&fakeProfileRepo{profile: minimalProfile()}, &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)},
&fakeArtifactReader{errByURI: map[string]error{"a://bad": errors.New("read failed")}}, &fakeArtifactReader{errByURI: map[string]error{"a://bad": errors.New("read failed")}},
&fakeRenderer{rendered: &domain.RenderedPrompt{}}, &fakeRenderer{rendered: &domain.RenderedPrompt{}},
&fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}, &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}},
@@ -269,10 +188,10 @@ func TestRunnerRunArtifactLoadFailure(t *testing.T) {
) )
_, err := runner.Run(context.Background(), domain.RunRequest{ _, err := runner.Run(context.Background(), domain.RunRequest{
ProfileID: "p", PromptID: "p",
Inputs: map[string]domain.ArtifactRef{ ProfileID: "exec",
"transcript": {Type: domain.ArtifactRefFile, URI: "a://bad"}, Execution: &domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m"},
}, Inputs: map[string]domain.ArtifactRef{"transcript": {Type: domain.ArtifactRefFile, URI: "a://bad"}},
}) })
if !errors.Is(err, ErrArtifactLoad) { if !errors.Is(err, ErrArtifactLoad) {
t.Fatalf("expected ErrArtifactLoad, got %v", err) t.Fatalf("expected ErrArtifactLoad, got %v", err)
@@ -281,18 +200,17 @@ func TestRunnerRunArtifactLoadFailure(t *testing.T) {
func TestRunnerRunPromptRenderFailure(t *testing.T) { func TestRunnerRunPromptRenderFailure(t *testing.T) {
runner := NewRunner( runner := NewRunner(
&fakeProfileRepo{profile: minimalProfile()}, &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)},
&fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{"a://ok": {Body: []byte("x"), Hash: hashString("x")}}}, &fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{"a://ok": {Body: []byte("x"), Hash: hashString("x")}}},
&fakeRenderer{err: errors.New("render failed")}, &fakeRenderer{err: errors.New("render failed")},
&fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}, &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}},
nil, nil,
) )
_, err := runner.Run(context.Background(), domain.RunRequest{ _, err := runner.Run(context.Background(), domain.RunRequest{
ProfileID: "p", PromptID: "p",
Inputs: map[string]domain.ArtifactRef{ ProfileID: "exec",
"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"}, Execution: &domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m"},
}, Inputs: map[string]domain.ArtifactRef{"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"}},
}) })
if !errors.Is(err, ErrPromptRender) { if !errors.Is(err, ErrPromptRender) {
t.Fatalf("expected ErrPromptRender, got %v", err) t.Fatalf("expected ErrPromptRender, got %v", err)
@@ -301,405 +219,82 @@ func TestRunnerRunPromptRenderFailure(t *testing.T) {
func TestRunnerRunLLMFailure(t *testing.T) { func TestRunnerRunLLMFailure(t *testing.T) {
runner := NewRunner( runner := NewRunner(
&fakeProfileRepo{profile: minimalProfile()}, &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)},
&fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{"a://ok": {Body: []byte("x"), Hash: hashString("x")}}}, &fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{"a://ok": {Body: []byte("x"), Hash: hashString("x")}}},
&fakeRenderer{rendered: &domain.RenderedPrompt{}}, &fakeRenderer{rendered: &domain.RenderedPrompt{}},
&fakeLLM{err: errors.New("llm failed")}, &fakeLLM{err: errors.New("llm failed")},
nil, nil,
) )
_, err := runner.Run(context.Background(), domain.RunRequest{ _, err := runner.Run(context.Background(), domain.RunRequest{
ProfileID: "p", PromptID: "p",
Inputs: map[string]domain.ArtifactRef{ ProfileID: "exec",
"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"}, Execution: &domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m"},
}, Inputs: map[string]domain.ArtifactRef{"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"}},
}) })
if !errors.Is(err, ErrLLMGenerate) { if !errors.Is(err, ErrLLMGenerate) {
t.Fatalf("expected ErrLLMGenerate, got %v", err) t.Fatalf("expected ErrLLMGenerate, got %v", err)
} }
} }
func TestRunnerRunValidationFailureNonError(t *testing.T) { func TestRunnerRunValidationFailurePreservesRawOutput(t *testing.T) {
validator := &fakeValidator{result: domain.ValidationResult{ validator := &fakeValidator{result: domain.ValidationResult{Status: domain.ValidationFailed, Mode: domain.ValidationBasic, Errors: []string{"bad"}, IsValid: false}}
Status: domain.ValidationFailed,
Mode: domain.ValidationBasic,
Errors: []string{"bad output"},
IsValid: false,
}}
runner := NewRunner( runner := NewRunner(
&fakeProfileRepo{profile: minimalProfile()}, &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationBasic, 0)},
&fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{"a://ok": {Body: []byte("x"), Hash: hashString("x")}}}, &fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{"a://ok": {Body: []byte("x"), Hash: hashString("x")}}},
&fakeRenderer{rendered: &domain.RenderedPrompt{}}, &fakeRenderer{rendered: &domain.RenderedPrompt{}},
&fakeLLM{resp: &domain.GenerateResponse{Content: "raw output"}}, &fakeLLM{resp: &domain.GenerateResponse{Content: "raw output"}},
validator, validator,
) )
res, err := runner.Run(context.Background(), domain.RunRequest{ res, err := runner.Run(context.Background(), domain.RunRequest{
ProfileID: "p", PromptID: "p",
Inputs: map[string]domain.ArtifactRef{ ProfileID: "exec",
"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"}, Execution: &domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m"},
}, Inputs: map[string]domain.ArtifactRef{"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"}},
}) })
if err != nil { if err != nil {
t.Fatalf("expected no error, got %v", err) t.Fatalf("expected no error, got %v", err)
} }
if res.Validation.Status != domain.ValidationFailed { if res.Validation.Status != domain.ValidationFailed || res.RawOutput != "raw output" {
t.Fatalf("expected validation failed result, got %q", res.Validation.Status) t.Fatalf("unexpected validation/raw output: %+v", res)
}
if res.RawOutput != "raw output" {
t.Fatalf("expected raw output preserved, got %q", res.RawOutput)
}
if !validator.called {
t.Fatal("expected validator to be called")
} }
} }
func TestRunnerRunValidationRuntimeError(t *testing.T) { func TestRunnerRunRepairBounded(t *testing.T) {
validator := &fakeValidator{err: errors.New("validator unavailable")} repairer := &fakeRepairer{responses: []*domain.GenerateResponse{{Content: `{"broken":`}, {Content: `{"still":`}}}
runner := NewRunner(
&fakeProfileRepo{profile: minimalProfile()},
&fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{"a://ok": {Body: []byte("x"), Hash: hashString("x")}}},
&fakeRenderer{rendered: &domain.RenderedPrompt{}},
&fakeLLM{resp: &domain.GenerateResponse{Content: "raw output"}},
validator,
)
_, err := runner.Run(context.Background(), domain.RunRequest{
ProfileID: "p",
Inputs: map[string]domain.ArtifactRef{
"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"},
},
})
if !errors.Is(err, ErrValidation) {
t.Fatalf("expected ErrValidation, got %v", err)
}
}
func TestRunnerRunValidationFailureWithRealValidatorPreservesRawOutput(t *testing.T) {
tmp := t.TempDir()
if err := os.WriteFile(filepath.Join(tmp, "schema.json"), []byte(`{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["name"],
"properties": {
"name": {"type": "string"}
}
}`), 0644); err != nil {
t.Fatal(err)
}
runner := NewRunner(
&fakeProfileRepo{profile: &domain.PromptProfile{
ID: "p-json",
Version: "1",
OutputFormat: domain.FormatJSON,
ModelDefaults: domain.ModelTarget{
Endpoint: "ep",
Model: "m",
},
Validation: domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "schema.json",
Format: domain.FormatJSON,
},
}},
&fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{"a://ok": {Body: []byte("x"), Hash: hashString("x")}}},
&fakeRenderer{rendered: &domain.RenderedPrompt{}},
&fakeLLM{resp: &domain.GenerateResponse{Content: `{"count":1}`}},
validate.NewStandardValidator(tmp),
)
res, err := runner.Run(context.Background(), domain.RunRequest{
ProfileID: "p-json",
Inputs: map[string]domain.ArtifactRef{
"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"},
},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if res.Validation.Status != domain.ValidationFailed {
t.Fatalf("expected validation failed, got %q", res.Validation.Status)
}
if res.RawOutput != `{"count":1}` {
t.Fatalf("expected raw output preserved, got %q", res.RawOutput)
}
}
func TestRunnerRunNoRepairWhenDisabled(t *testing.T) {
repairer := &fakeRepairer{
responses: []*domain.GenerateResponse{{Content: `{"ok":true}`}},
}
runner := NewRunnerWithRepairer( runner := NewRunnerWithRepairer(
&fakeProfileRepo{profile: &domain.PromptProfile{ &fakePromptRepo{def: promptDef(domain.FormatJSON, domain.ValidationJSON, 1)},
ID: "p-json",
Version: "1",
OutputFormat: domain.FormatJSON,
ModelDefaults: domain.ModelTarget{
Endpoint: "ep",
Model: "m",
},
Validation: domain.OutputContract{
ValidationMode: domain.ValidationJSON,
Format: domain.FormatJSON,
RepairAttempts: 0,
},
}},
&fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{"a://ok": {Body: []byte("x"), Hash: hashString("x")}}},
&fakeRenderer{rendered: &domain.RenderedPrompt{}},
&fakeLLM{resp: &domain.GenerateResponse{Content: `{"broken":`}},
validate.NewStandardValidator(t.TempDir()),
repairer,
)
res, err := runner.Run(context.Background(), domain.RunRequest{
ProfileID: "p-json",
Inputs: map[string]domain.ArtifactRef{
"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"},
},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if repairer.calls != 0 {
t.Fatalf("expected no repair calls, got %d", repairer.calls)
}
if res.Validation.Status != domain.ValidationFailed {
t.Fatalf("expected failed validation, got %q", res.Validation.Status)
}
if res.RawOutput != `{"broken":` {
t.Fatalf("expected original output preserved, got %q", res.RawOutput)
}
if res.Validation.RepairAttempts != 0 {
t.Fatalf("expected repair attempts 0, got %d", res.Validation.RepairAttempts)
}
}
func TestRunnerRunSuccessfulRepairAfterInvalidJSON(t *testing.T) {
repairer := &fakeRepairer{
responses: []*domain.GenerateResponse{{Content: `{"ok":true}`, Usage: domain.TokenUsage{TotalTokens: 5}}},
}
runner := NewRunnerWithRepairer(
&fakeProfileRepo{profile: &domain.PromptProfile{
ID: "p-json",
Version: "1",
OutputFormat: domain.FormatJSON,
ModelDefaults: domain.ModelTarget{
Endpoint: "ep",
Model: "m",
},
Validation: domain.OutputContract{
ValidationMode: domain.ValidationJSON,
Format: domain.FormatJSON,
RepairAttempts: 1,
},
}},
&fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{"a://ok": {Body: []byte("x"), Hash: hashString("x")}}},
&fakeRenderer{rendered: &domain.RenderedPrompt{}},
&fakeLLM{resp: &domain.GenerateResponse{Content: `{"broken":`, Usage: domain.TokenUsage{TotalTokens: 3}}},
validate.NewStandardValidator(t.TempDir()),
repairer,
)
res, err := runner.Run(context.Background(), domain.RunRequest{
ProfileID: "p-json",
Inputs: map[string]domain.ArtifactRef{
"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"},
},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if repairer.calls != 1 {
t.Fatalf("expected one repair call, got %d", repairer.calls)
}
if res.Validation.Status != domain.ValidationPassed {
t.Fatalf("expected passed validation, got %q", res.Validation.Status)
}
if res.Validation.RepairAttempts != 1 {
t.Fatalf("expected repair attempts 1, got %d", res.Validation.RepairAttempts)
}
if res.RawOutput != `{"ok":true}` {
t.Fatalf("expected repaired output, got %q", res.RawOutput)
}
}
func TestRunnerRunSuccessfulRepairAfterSchemaFailure(t *testing.T) {
tmp := t.TempDir()
if err := os.WriteFile(filepath.Join(tmp, "schema.json"), []byte(`{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["name"],
"properties": {
"name": {"type": "string"}
}
}`), 0644); err != nil {
t.Fatal(err)
}
repairer := &fakeRepairer{
responses: []*domain.GenerateResponse{{Content: `{"name":"eris"}`}},
}
runner := NewRunnerWithRepairer(
&fakeProfileRepo{profile: &domain.PromptProfile{
ID: "p-json",
Version: "1",
OutputFormat: domain.FormatJSON,
ModelDefaults: domain.ModelTarget{
Endpoint: "ep",
Model: "m",
},
Validation: domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "schema.json",
Format: domain.FormatJSON,
RepairAttempts: 1,
},
}},
&fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{"a://ok": {Body: []byte("x"), Hash: hashString("x")}}},
&fakeRenderer{rendered: &domain.RenderedPrompt{}},
&fakeLLM{resp: &domain.GenerateResponse{Content: `{"count":1}`}},
validate.NewStandardValidator(tmp),
repairer,
)
res, err := runner.Run(context.Background(), domain.RunRequest{
ProfileID: "p-json",
Inputs: map[string]domain.ArtifactRef{
"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"},
},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if res.Validation.Status != domain.ValidationPassed {
t.Fatalf("expected passed validation, got %q", res.Validation.Status)
}
if res.Validation.RepairAttempts != 1 {
t.Fatalf("expected repair attempts 1, got %d", res.Validation.RepairAttempts)
}
if res.RawOutput != `{"name":"eris"}` {
t.Fatalf("expected repaired output, got %q", res.RawOutput)
}
}
func TestRunnerRunFailedRepairPreservesRawOutputAndErrors(t *testing.T) {
repairer := &fakeRepairer{
responses: []*domain.GenerateResponse{
{Content: `{"repair1":`},
{Content: `{"repair2":`},
},
}
runner := NewRunnerWithRepairer(
&fakeProfileRepo{profile: &domain.PromptProfile{
ID: "p-json",
Version: "1",
OutputFormat: domain.FormatJSON,
ModelDefaults: domain.ModelTarget{
Endpoint: "ep",
Model: "m",
},
Validation: domain.OutputContract{
ValidationMode: domain.ValidationJSON,
Format: domain.FormatJSON,
RepairAttempts: 2,
},
}},
&fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{"a://ok": {Body: []byte("x"), Hash: hashString("x")}}}, &fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{"a://ok": {Body: []byte("x"), Hash: hashString("x")}}},
&fakeRenderer{rendered: &domain.RenderedPrompt{}}, &fakeRenderer{rendered: &domain.RenderedPrompt{}},
&fakeLLM{resp: &domain.GenerateResponse{Content: `{"initial":`}}, &fakeLLM{resp: &domain.GenerateResponse{Content: `{"initial":`}},
validate.NewStandardValidator(t.TempDir()), validate.NewStandardValidator("."),
repairer, repairer,
) )
res, err := runner.Run(context.Background(), domain.RunRequest{ res, err := runner.Run(context.Background(), domain.RunRequest{
ProfileID: "p-json", PromptID: "p",
Inputs: map[string]domain.ArtifactRef{ ProfileID: "exec",
"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"}, Execution: &domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m"},
}, Inputs: map[string]domain.ArtifactRef{"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"}},
}) })
if err != nil { if err != nil {
t.Fatalf("expected no error, got %v", err) t.Fatalf("expected no error, got %v", err)
} }
if res.Validation.Status != domain.ValidationFailed { if repairer.calls != 1 || res.Validation.RepairAttempts != 1 {
t.Fatalf("expected failed validation, got %q", res.Validation.Status) t.Fatalf("expected one bounded repair, calls=%d attempts=%d", repairer.calls, res.Validation.RepairAttempts)
}
if len(res.Validation.Errors) == 0 {
t.Fatal("expected validation errors after failed repair")
}
if res.Validation.RepairAttempts != 2 {
t.Fatalf("expected repair attempts 2, got %d", res.Validation.RepairAttempts)
}
if res.RawOutput != `{"repair2":` {
t.Fatalf("expected final repaired output preserved, got %q", res.RawOutput)
} }
} }
func TestRunnerRunRepairAttemptsBounded(t *testing.T) { func promptDef(format domain.OutputFormat, mode domain.ValidationMode, attempts int) *domain.PromptDefinition {
repairer := &fakeRepairer{ return &domain.PromptDefinition{
responses: []*domain.GenerateResponse{ ID: "p",
{Content: `{"repair1":`}, Version: "1",
{Content: `{"repair2":`}, DefaultProfile: "exec",
{Content: `{"repair3":`}, Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
}, Templates: []domain.PromptMessageTemplate{{Role: "user", Content: "x"}},
} OutputFormat: format,
runner := NewRunnerWithRepairer(
&fakeProfileRepo{profile: &domain.PromptProfile{
ID: "p-json",
Version: "1",
OutputFormat: domain.FormatJSON,
ModelDefaults: domain.ModelTarget{
Endpoint: "ep",
Model: "m",
},
Validation: domain.OutputContract{
ValidationMode: domain.ValidationJSON,
Format: domain.FormatJSON,
RepairAttempts: 1,
},
}},
&fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{"a://ok": {Body: []byte("x"), Hash: hashString("x")}}},
&fakeRenderer{rendered: &domain.RenderedPrompt{}},
&fakeLLM{resp: &domain.GenerateResponse{Content: `{"initial":`}},
validate.NewStandardValidator(t.TempDir()),
repairer,
)
res, err := runner.Run(context.Background(), domain.RunRequest{
ProfileID: "p-json",
Inputs: map[string]domain.ArtifactRef{
"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"},
},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if repairer.calls != 1 {
t.Fatalf("expected repair calls bounded to 1, got %d", repairer.calls)
}
if res.Validation.RepairAttempts != 1 {
t.Fatalf("expected repair attempts 1, got %d", res.Validation.RepairAttempts)
}
}
func minimalProfile() *domain.PromptProfile {
return &domain.PromptProfile{
ID: "p",
Version: "1",
OutputFormat: domain.FormatText,
ModelDefaults: domain.ModelTarget{
Endpoint: "ep",
Model: "m",
},
Validation: domain.OutputContract{ Validation: domain.OutputContract{
ValidationMode: domain.ValidationBasic, ValidationMode: mode,
RepairAttempts: attempts,
Format: format,
}, },
} }
} }

18
profiles/dnd.session_recap.yaml
View File

@@ -1,9 +1,14 @@
id: dnd.session_recap id: dnd.session_recap
version: "1.0.0" version: "1.0.0"
description: Example D&D session recap profile for demonstration only. default_profile: local-default
expected_inputs: description: Example D&D session recap prompt definition for demonstration only.
- transcript inputs:
- glossary - name: transcript
required: true
content_type: text/markdown
- name: glossary
required: false
content_type: text/yaml
templates: templates:
- role: system - role: system
content: | content: |
@@ -21,11 +26,6 @@ templates:
Glossary: Glossary:
{{input "glossary"}} {{input "glossary"}}
model_defaults:
endpoint: http://localhost:8000/v1
model: gpt-4o-mini
temperature: 0.3
max_tokens: 900
output_format: markdown output_format: markdown
validation: validation:
validation_mode: basic validation_mode: basic

18
profiles/generic.markdown_summary.yaml
View File

@@ -1,9 +1,16 @@
id: generic.markdown_summary id: generic.markdown_summary
version: "1.0.0" version: "1.0.0"
default_profile: local-default
description: Generic markdown summary from transcript and glossary. description: Generic markdown summary from transcript and glossary.
expected_inputs: inputs:
- transcript - name: transcript
- glossary required: true
content_type: text/markdown
description: Source transcript content
- name: glossary
required: false
content_type: text/yaml
description: Optional glossary context
templates: templates:
- role: system - role: system
content: | content: |
@@ -18,11 +25,6 @@ templates:
Reference glossary: Reference glossary:
{{input "glossary"}} {{input "glossary"}}
model_defaults:
endpoint: http://localhost:8000/v1
model: gpt-4o-mini
temperature: 0.2
max_tokens: 700
output_format: markdown output_format: markdown
validation: validation:
validation_mode: basic validation_mode: basic

16
profiles/generic.structured_events.yaml
View File

@@ -1,9 +1,14 @@
id: generic.structured_events id: generic.structured_events
version: "1.0.0" version: "1.0.0"
default_profile: local-default
description: Produce structured event JSON from a transcript. description: Produce structured event JSON from a transcript.
expected_inputs: inputs:
- transcript - name: transcript
- glossary required: true
content_type: text/markdown
- name: glossary
required: false
content_type: text/yaml
templates: templates:
- role: system - role: system
content: | content: |
@@ -18,11 +23,6 @@ templates:
Glossary: Glossary:
{{input "glossary"}} {{input "glossary"}}
model_defaults:
endpoint: http://localhost:8000/v1
model: gpt-4o-mini
temperature: 0.0
max_tokens: 500
output_format: json output_format: json
validation: validation:
format: json format: json