Files

Eric Rakestraw 7fbc6cd9fe Updated architecture documentation to describe the roadmap for the upcoming render feature

2026-05-06 09:13:32 -05:00

17 KiB

Raw Blame History

Scriptorium Architecture

1. Purpose and Non-Goals

Scriptorium is a prompt-definition execution engine.

It accepts named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata.

Scriptorium also supports rendering/preparing a prompt without calling an LLM. This allows users to inspect the fully rendered prompt messages and effective runtime settings before executing a run.

Scriptorium is not an orchestrator. It must not own transcription, transcript merge/polish steps, notifications, or cross-step workflow control.

For the motivating D&D workflow:

Narratio orchestrates.
WhisperX transcribes.
Seriatim merges transcripts.
Audita polishes transcripts.
Scriptorium generates final artifacts from prepared inputs.

Core Go code remains generic.

2. Current Architecture

Scriptorium uses a ports-and-adapters architecture to decouple the core execution logic from external dependencies.

Package Responsibilities

cmd/scriptorium: Binary entrypoint for CLI and HTTP server.
internal/domain: Core domain contracts, including PromptDefinition, ExecutionProfile, PreparedRun, RunResult, and related metadata.
internal/usecase: Runner use case logic, including prompt preparation, profile selection, runtime override resolution, full run execution, validation, and bounded repair.
internal/promptdef: Repository for loading and validating Prompt Definitions from the filesystem.
internal/profile: Repository for loading Execution Profiles from the filesystem.
internal/artifact: Input artifact resolution (inline, file).
internal/prompt: Template rendering via Go templates.
internal/llm: Provider-neutral client interface and OpenAI-compatible HTTP adapter.
internal/validate: Output validation implementation (none/basic/json/json_schema).
internal/adapter/cli: CLI flag parsing, command dispatch, and output handling.
internal/adapter/http: HTTP request/response mapping.
internal/format or equivalent: Formatting of prepared/rendered prompt output for CLI or other adapters, if formatting grows beyond simple CLI-local helpers.

Exact package names may evolve, but the architectural boundaries should remain stable.

3. Core Execution Model

Scriptorium has two closely related execution paths:

Prepare/render path.
Full run path.

The full run path should reuse the prepare path rather than duplicating its logic.

3.1 Prepare / Render Data Flow

The prepare path should be represented in the use case layer, preferably as Runner.Prepare(ctx, RunRequest) or an equivalent method.

It executes all pre-LLM work:

Validate Request: Ensure the request includes a prompt ID and all minimum required fields.
Load Prompt Definition: Retrieve the PromptDefinition by ID from the prompt repository.
Select Profile: Determine the profile_id using this precedence:
- Explicit profile_id in RunRequest.
- default_profile specified in the PromptDefinition.
- Error if neither is available.
Load Execution Profile: Retrieve the ExecutionProfile from the profile repository.
Resolve Runtime Overrides: Merge settings based on precedence, highest to lowest:
- Runtime overrides from CLI flags or HTTP request model object.
- Execution Profile settings.
- Built-in application defaults.
Resolve Artifacts: Load all named input artifacts defined in the request.
Render Prompt: Apply template variables and input artifacts to the prompt templates.
Compute Metadata: Compute hashes, selected profile ID, prompt ID/version, effective runtime settings, input hashes, rendered prompt hash, and timing information as appropriate.
Return PreparedRun: Return a PreparedRun containing the rendered messages, effective runtime settings, resolved metadata, and input/prompt hashes.

The prepare path must not call the LLM.

The prepare path must not validate model output, because there is no model output.

The prepare path must not perform structured-output repair, because repair only applies after model output exists.

The prepare path should not resolve or expose raw API key values. It may include the selected api_key_env name in effective runtime settings or metadata, but never the environment variable value.

3.2 Full Run Data Flow

The Runner.Run(ctx, RunRequest) flow should reuse the prepare path:

Prepare: Call the shared prepare flow to load the prompt, select the profile, resolve artifacts, render prompt messages, and compute pre-run metadata.
Call LLM: Execute the generation request using the effective runtime settings from the prepared run.
Build Output Artifact: Convert the model response into the configured output artifact.
Validate Output:
- Validate the model output against the prompt definition's output contract.
- Validation content failures remain successful run results with validation.status=failed.
- Validator runtime/config errors are run errors.
Repair If Configured:
- If structured validation fails and repair_attempts > 0, perform bounded repair attempts.
- Re-validate after each repair attempt.
- Repair loops must remain strictly bounded.
Return RunResult: Produce a RunResult containing the final artifact, validation status, raw model output, usage information, and metadata.

Runner.Run should not duplicate profile selection, artifact resolution, or prompt rendering logic that already exists in Runner.Prepare.

4. Domain Model

Key domain types:

PromptDefinition: Defines the "what" of the task: templates, inputs, output contract, validation settings, repair settings, and optional default_profile.
ExecutionProfile: Defines the "how" of execution: endpoint, model, generation parameters, timeout, reasoning effort, and api_key_env.
RunRequest: The intent to execute or prepare a prompt, including prompt_id, optional profile_id, inputs, variables, and optional runtime overrides.
PreparedRun: The result of the prepare/render phase. Contains rendered messages, effective runtime settings, selected profile ID, prompt metadata, input hashes, prompt hash, and other pre-LLM metadata.
RunResult: The result of a full run. Contains the generated Artifact, ValidationResult, raw model output, token usage, and RunMetadata.
RunMetadata: Detailed tracing information, including prompt ID/version, selected profile ID, effective model parameters, usage tokens, hashes, timestamps, validation status, and repair attempts where applicable.
RenderedPrompt: Provider-neutral rendered prompt structure.
RenderedMessage: Provider-neutral rendered message with role and content.
ArtifactRef: A reference to an input artifact, such as file or inline.
Artifact: Loaded artifact content with name, content type, body, source URI, size, and hash.
ValidationResult: Validation status and details for full runs.

PreparedRun should be serializable for JSON output and should also be representable in a human-readable text format.

5. Interfaces and Adapters

Primary Ports

promptdef.Repository: Lookup for prompt definitions.
profile.Repository: Lookup for execution profiles.
artifact.Reader: Loading of artifact content.
prompt.Renderer: Template rendering.
llm.Client: Model generation.
validate.Validator: Output validation.
format.PreparedRunFormatter or equivalent: Optional formatting abstraction for rendered/prepared output.

Current Adapters

Repositories: Filesystem YAML loaders for both prompts and profiles.
Artifact Reader: Composite reader supporting file and inline.
Prompt Renderer: Go templates with a custom input helper.
LLM Client: OpenAI-compatible /chat/completions over HTTP.
Validator: Standard validator supporting none, basic, json, and json_schema.
CLI Adapter: Supports run, render, and serve.
HTTP Adapter: Supports full run execution through POST /v1/runs.

6. Public Contracts

CLI

Scriptorium should expose at least these commands:

scriptorium run: Executes a prompt by preparing it, calling the LLM, validating output, optionally repairing structured output, and returning an artifact.
scriptorium render: Prepares and renders a prompt without calling the LLM.
scriptorium serve: Starts the HTTP API using infrastructure-only flags.

`scriptorium run`

run uses flags such as:

--prompt-dir
--profile-dir
--prompt
--profile
--input
--var
--out
runtime overrides such as --model, --llm-base-url, --temperature, --max-tokens, --top-p, --timeout, and --api-key-env if supported.

run should produce the generated artifact as its primary output.

`scriptorium render`

render prepares and renders a prompt without calling an LLM.

It should use the same prompt/profile/input/variable/runtime override flags as run where applicable:

--prompt-dir
--profile-dir
--prompt
--profile
--input
--var
runtime overrides such as --model, --llm-base-url, --temperature, --max-tokens, --top-p, --timeout, and --api-key-env if supported.
--format, with initial support for text and json.

Default render output format should be text.

render must not call the LLM.

render should show the same rendered messages and effective runtime settings that run would use.

render should include enough information to debug:

prompt ID
prompt version
selected profile ID
effective runtime settings
input hashes
prompt hash
rendered messages

render must not include resolved API key values.

render may include the api_key_env name.

Render Output Formats

Initial render output formats:

text: Human-readable default format.
json: Machine-readable structured representation of the prepared run.

Additional formats, such as markdown, may be added later.

Render output formatting should be modular. Adding a new output format should not require changing the prepare/run core logic.

The output formatting layer should consume a PreparedRun and produce bytes or text for the adapter. It should not reload prompts, re-resolve artifacts, re-render templates, call the LLM, or perform validation.

`scriptorium serve`

serve starts the HTTP API.

It should use infrastructure-only flags such as:

--addr
--prompt-dir
--profile-dir
--schema-dir

serve should not introduce a server-level model/runtime precedence layer unless explicitly documented and intentionally implemented.

HTTP API

Current HTTP API:

POST /v1/runs: Accepts RunRequest JSON and returns RunResponse JSON. No built-in auth.

Request may include runtime overrides under model and an include_raw_output boolean.

raw_model_output is exposed only when explicitly requested with include_raw_output=true.

A future HTTP prepare/render endpoint may be added, such as POST /v1/renders or POST /v1/runs/prepare, but the initial render feature may be CLI-only. If added later, it should call the same usecase-level prepare path as scriptorium render.

YAML Shapes

Prompt YAML includes:

id
version
optional default_profile
inputs
messages
output

Inputs support:

name
required
optional content_type
description

Messages require:

role
exactly one of content or content_file

content_file resolves relative to the prompt YAML location.

Profile YAML includes:

id
endpoint
model
generation parameters
timeout settings
reasoning settings
api_key_env

Prompt content must not appear in profile YAML.

Model/runtime/API-key settings must not appear in prompt YAML, except that prompt YAML may specify default_profile.

7. Render Feature Design

The render feature is a first-class use case, not a CLI-only shortcut.

Goals

The render feature should help users:

inspect fully rendered prompt messages
debug missing inputs
verify template variable substitution
verify selected profile resolution
verify runtime override precedence
verify file-backed prompt loading
inspect input hashes and prompt hashes
prepare for future token budgeting and prompt-size inspection

Non-Goals

The render feature should not:

call an LLM
validate model output
repair structured output
resolve or print API key values
mutate artifacts
save outputs to artifact storage unless a future explicit output option is added
become an orchestration step manager

Usecase Shape

The preferred usecase shape is:

Runner.Prepare(ctx, RunRequest) (*PreparedRun, error)
Runner.Run(ctx, RunRequest) (*RunResult, error)

Runner.Run should call Runner.Prepare.

The prepare flow should be the only implementation of:

prompt loading
profile selection
runtime override resolution
artifact resolution
prompt rendering
pre-run metadata/hash calculation

CLI Shape

The preferred command name is render.

The command should support:

--format text
--format json

Default format:

text

Unknown formats should produce a clear error.

Formatting should be centralized through a small formatter registry, strategy, switch, or interface so new formats can be added without modifying usecase logic.

Text Output Expectations

Text output should be optimized for human inspection.

It should include, at minimum:

prompt ID and version
selected profile ID
model name
endpoint
effective generation settings
input hashes
rendered prompt hash
rendered messages grouped by role

Text output should be readable and deterministic enough for tests.

It should not include raw API key values.

JSON Output Expectations

JSON output should be a structured representation of PreparedRun or a DTO derived from it.

It should include, at minimum:

prompt ID and version
selected profile ID
effective runtime settings
input hashes
rendered prompt hash
rendered messages

JSON output should not include raw API key values.

JSON output should remain stable enough to be useful for automation and integration tests.

8. Guardrails

Separation of Concerns: Prompt content must not belong in execution profiles; model/API settings must not belong in prompt definitions.
Security: Raw API keys are unsupported in all configuration and transport layers. Only api_key_env is used.
Secret Handling: Resolved API key values must never appear in rendered output, metadata, logs, HTTP responses, or CLI output.
Path Resolution: content_file paths in prompt definitions resolve relative to the prompt YAML file.
Integrity: No silent prompt truncation or omission of content.
Reliability: Repair loops are strictly bounded by repair_attempts.
No Orchestration Creep: Scriptorium prepares and executes a single prompt request. It does not coordinate multi-stage workflows.
Render Reuse: The full run path must reuse the prepare/render path to avoid divergent behavior.
Formatter Isolation: Render output formatters must not perform usecase work. They only format a completed PreparedRun.

9. Testing Strategy

Tests should protect both the run path and the prepare/render path.

Prepare / Render Tests

Add tests for:

preparing a prompt with explicit profile selection
preparing a prompt using default_profile
failing when no explicit profile and no default_profile exist
runtime overrides beating profile values
profile values beating application defaults
file-backed prompt bodies rendering correctly
required inputs failing when missing
optional inputs being absent when not referenced
unknown input references failing
input hashes being included
rendered prompt hash being included
effective runtime settings being included
api_key_env name being included where appropriate
resolved API key values never appearing in PreparedRun
prepare path not calling the LLM

CLI Render Tests