scriptorium/architecture.md

# Scriptorium Architecture

## 1. Purpose and Non-Goals

Scriptorium is a prompt-definition execution engine.

It accepts named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata.

Scriptorium also supports rendering/preparing a prompt without calling an LLM. This allows users to inspect the fully rendered prompt messages and effective runtime settings before executing a run.

Scriptorium is not an orchestrator. It must not own transcription, transcript merge/polish steps, notifications, or cross-step workflow control.

For the motivating D&D workflow:
- Narratio orchestrates.
- WhisperX transcribes.
- Seriatim merges transcripts.
- Audita polishes transcripts.
- Scriptorium generates final artifacts from prepared inputs.

Core Go code remains generic.

## 2. Current Architecture

Scriptorium uses a ports-and-adapters architecture to decouple the core execution logic from external dependencies.

### Package Responsibilities

- `cmd/scriptorium`: Binary entrypoint for CLI and HTTP server.
- `internal/domain`: Core domain contracts, including `PromptDefinition`, `ExecutionProfile`, `PreparedRun`, `RunResult`, and related metadata.
- `internal/usecase`: `Runner` use case logic, including prompt preparation, profile selection, runtime override resolution, full run execution, validation, and bounded repair.
- `internal/config`: Application-level config model/loader for adapter settings (for example prompt/profile/schema directories, server address, and render format default).
- `internal/promptdef`: Repository for loading and validating Prompt Definitions from the filesystem.
- `internal/profile`: Repository for loading Execution Profiles from the filesystem.
- `internal/artifact`: Input artifact resolution (`inline`, `file`).
- `internal/prompt`: Template rendering via Go templates.
- `internal/llm`: Provider-neutral client interface and OpenAI-compatible HTTP adapter.
- `internal/validate`: Output validation implementation (`none/basic/json/json_schema`).
- `internal/adapter/cli`: CLI flag parsing, command dispatch, and output handling.
- `internal/adapter/http`: HTTP request/response mapping.
- `internal/format` or equivalent: Formatting of prepared/rendered prompt output for CLI or other adapters, if formatting grows beyond simple CLI-local helpers.

Exact package names may evolve, but the architectural boundaries should remain stable.

### Application Configuration

`config.yml` is adapter/application setup, not domain logic.

Application config is intended for application-level settings such as:
- `prompt_dir`
- `profile_dir`
- `schema_dir`
- `server.addr`
- `defaults.render_format`

Application config precedence is:
1. CLI flags
2. `config.yml`
3. Built-in application defaults

Runtime model settings are intentionally separate:
- Execution profiles and runtime overrides continue to own endpoint/model/runtime behavior.
- `config.yml` does not replace execution profiles.

The core use case (`Runner.Prepare`/`Runner.Run`) does not need to know whether adapter-level settings came from CLI flags or `config.yml`; it receives resolved dependencies and requests from adapters.

## 3. Core Execution Model

Scriptorium has two closely related execution paths:

1. Prepare/render path.
2. Full run path.

The full run path should reuse the prepare path rather than duplicating its logic.

### 3.1 Prepare / Render Data Flow

The prepare path should be represented in the use case layer, preferably as `Runner.Prepare(ctx, RunRequest)` or an equivalent method.

It executes all pre-LLM work:

1. **Validate Request**: Ensure the request includes a prompt ID and all minimum required fields.
2. **Load Prompt Definition**: Retrieve the `PromptDefinition` by ID from the prompt repository.
3. **Select Profile**: Determine the `profile_id` using this precedence:
   - Explicit `profile_id` in `RunRequest`.
   - `default_profile` specified in the `PromptDefinition`.
   - Error if neither is available.
4. **Load Execution Profile**: Retrieve the `ExecutionProfile` from the profile repository.
5. **Resolve Runtime Overrides**: Merge settings based on precedence, highest to lowest:
   - Runtime overrides from CLI flags or HTTP request `model` object.
   - Execution Profile settings.
   - Built-in application defaults.
6. **Resolve Artifacts**: Load all named input artifacts defined in the request.
7. **Render Prompt**: Apply template variables and input artifacts to the prompt templates.
8. **Compute Metadata**: Compute hashes, selected profile ID, prompt ID/version, effective runtime settings, input hashes, rendered prompt hash, and timing information as appropriate.
9. **Return PreparedRun**: Return a `PreparedRun` containing the rendered messages, effective runtime settings, resolved metadata, and input/prompt hashes.

The prepare path must not call the LLM.

The prepare path must not validate model output, because there is no model output.

The prepare path must not perform structured-output repair, because repair only applies after model output exists.

The prepare path should not resolve or expose raw API key values. It may include the selected `api_key_env` name in effective runtime settings or metadata, but never the environment variable value.

### 3.2 Full Run Data Flow

The `Runner.Run(ctx, RunRequest)` flow should reuse the prepare path:

1. **Prepare**: Call the shared prepare flow to load the prompt, select the profile, resolve artifacts, render prompt messages, and compute pre-run metadata.
2. **Call LLM**: Execute the generation request using the effective runtime settings from the prepared run.
3. **Build Output Artifact**: Convert the model response into the configured output artifact.
4. **Validate Output**:
   - Validate the model output against the prompt definition's output contract.
   - Validation content failures remain successful run results with `validation.status=failed`.
   - Validator runtime/config errors are run errors.
5. **Repair If Configured**:
   - If structured validation fails and `repair_attempts > 0`, perform bounded repair attempts.
   - Re-validate after each repair attempt.
   - Repair loops must remain strictly bounded.
6. **Return RunResult**: Produce a `RunResult` containing the final artifact, validation status, raw model output, usage information, and metadata.

`Runner.Run` should not duplicate profile selection, artifact resolution, or prompt rendering logic that already exists in `Runner.Prepare`.

## 4. Domain Model

Key domain types:

- `PromptDefinition`: Defines the "what" of the task: templates, inputs, output contract, validation settings, repair settings, and optional `default_profile`.
- `ExecutionProfile`: Defines the "how" of execution: endpoint, model, generation parameters, timeout, reasoning effort, and `api_key_env`.
- `RunRequest`: The intent to execute or prepare a prompt, including `prompt_id`, optional `profile_id`, inputs, variables, and optional runtime overrides.
- `PreparedRun`: The result of the prepare/render phase. Contains rendered messages, effective runtime settings, selected profile ID, prompt metadata, input hashes, prompt hash, and other pre-LLM metadata.
- `RunResult`: The result of a full run. Contains the generated `Artifact`, `ValidationResult`, raw model output, token usage, and `RunMetadata`.
- `RunMetadata`: Detailed tracing information, including prompt ID/version, selected profile ID, effective model parameters, usage tokens, hashes, timestamps, validation status, and repair attempts where applicable.
- `RenderedPrompt`: Provider-neutral rendered prompt structure.
- `RenderedMessage`: Provider-neutral rendered message with role and content.
- `ArtifactRef`: A reference to an input artifact, such as `file` or `inline`.
- `Artifact`: Loaded artifact content with name, content type, body, source URI, size, and hash.
- `ValidationResult`: Validation status and details for full runs.

`PreparedRun` should be serializable for JSON output and should also be representable in a human-readable text format.

## 5. Interfaces and Adapters

### Primary Ports

- `promptdef.Repository`: Lookup for prompt definitions.
- `profile.Repository`: Lookup for execution profiles.
- `artifact.Reader`: Loading of artifact content.
- `prompt.Renderer`: Template rendering.
- `llm.Client`: Model generation.
- `validate.Validator`: Output validation.
- `format.PreparedRunFormatter` or equivalent: Optional formatting abstraction for rendered/prepared output.

### Current Adapters

- **Repositories**: Filesystem YAML loaders for both prompts and profiles.
- **Artifact Reader**: Composite reader supporting `file` and `inline`.
- **Prompt Renderer**: Go templates with a custom `input` helper.
- **LLM Client**: OpenAI-compatible `/chat/completions` over HTTP.
- **Validator**: Standard validator supporting `none`, `basic`, `json`, and `json_schema`.
- **CLI Adapter**: Supports `run`, `render`, and `serve`.
- **HTTP Adapter**: Supports full run execution through `POST /v1/runs`.

## 6. Public Contracts

### CLI

Scriptorium should expose at least these commands:

- `scriptorium run`: Executes a prompt by preparing it, calling the LLM, validating output, optionally repairing structured output, and returning an artifact.
- `scriptorium render`: Prepares and renders a prompt without calling the LLM.
- `scriptorium serve`: Starts the HTTP API using infrastructure-only flags.

### `scriptorium run`

`run` uses flags such as:

- `--prompt-dir`
- `--profile-dir`
- `--prompt`
- `--profile`
- `--input`
- `--var`
- `--out`
- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.

`run` should produce the generated artifact as its primary output.

### `scriptorium render`

`render` prepares and renders a prompt without calling an LLM.

It should use the same prompt/profile/input/variable/runtime override flags as `run` where applicable:

- `--prompt-dir`
- `--profile-dir`
- `--prompt`
- `--profile`
- `--input`
- `--var`
- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.
- `--format`, with initial support for `text` and `json`.

Default render output format should be `text`.

`render` must not call the LLM.

`render` should show the same rendered messages and effective runtime settings that `run` would use.

`render` should include enough information to debug:

- prompt ID
- prompt version
- selected profile ID
- effective runtime settings
- input hashes
- prompt hash
- rendered messages

`render` must not include resolved API key values.

`render` may include the `api_key_env` name.

### Render Output Formats

Initial render output formats:

- `text`: Human-readable default format.
- `json`: Machine-readable structured representation of the prepared run.

Additional formats, such as `markdown`, may be added later.

Render output formatting should be modular. Adding a new output format should not require changing the prepare/run core logic.

The output formatting layer should consume a `PreparedRun` and produce bytes or text for the adapter. It should not reload prompts, re-resolve artifacts, re-render templates, call the LLM, or perform validation.

### `scriptorium serve`

`serve` starts the HTTP API.

It should use infrastructure-only flags such as:

- `--addr`
- `--prompt-dir`
- `--profile-dir`
- `--schema-dir`

`serve` should not introduce a server-level model/runtime precedence layer unless explicitly documented and intentionally implemented.

### HTTP API

Current HTTP API:

- `POST /v1/runs`: Accepts `RunRequest` JSON and returns `RunResponse` JSON. No built-in auth.

Request may include runtime overrides under `model` and an `include_raw_output` boolean.

`raw_model_output` is exposed only when explicitly requested with `include_raw_output=true`.

A future HTTP prepare/render endpoint may be added, such as `POST /v1/renders` or `POST /v1/runs/prepare`, but the initial render feature may be CLI-only. If added later, it should call the same usecase-level prepare path as `scriptorium render`.

### YAML Shapes

Prompt YAML includes:

- `id`
- `version`
- optional `default_profile`
- `inputs`
- `messages`
- `output`

Inputs support:

- `name`
- `required`
- optional `content_type`
- `description`

Messages require:

- `role`
- exactly one of `content` or `content_file`

`content_file` resolves relative to the prompt YAML location.

Profile YAML includes:

- `id`
- `endpoint`
- `model`
- generation parameters
- timeout settings
- reasoning settings
- `api_key_env`

Prompt content must not appear in profile YAML.

Model/runtime/API-key settings must not appear in prompt YAML, except that prompt YAML may specify `default_profile`.

## 7. Render Feature Design

The render feature is a first-class use case, not a CLI-only shortcut.

### Goals

The render feature should help users:

- inspect fully rendered prompt messages
- debug missing inputs
- verify template variable substitution
- verify selected profile resolution
- verify runtime override precedence
- verify file-backed prompt loading
- inspect input hashes and prompt hashes
- prepare for future token budgeting and prompt-size inspection

### Non-Goals

The render feature should not:

- call an LLM
- validate model output
- repair structured output
- resolve or print API key values
- mutate artifacts
- save outputs to artifact storage unless a future explicit output option is added
- become an orchestration step manager

### Usecase Shape

The preferred usecase shape is:

- `Runner.Prepare(ctx, RunRequest) (*PreparedRun, error)`
- `Runner.Run(ctx, RunRequest) (*RunResult, error)`

`Runner.Run` should call `Runner.Prepare`.

The prepare flow should be the only implementation of:

- prompt loading
- profile selection
- runtime override resolution
- artifact resolution
- prompt rendering
- pre-run metadata/hash calculation

### CLI Shape

The preferred command name is `render`.

The command should support:

- `--format text`
- `--format json`

Default format:

- `text`

Unknown formats should produce a clear error.

Formatting should be centralized through a small formatter registry, strategy, switch, or interface so new formats can be added without modifying usecase logic.

### Text Output Expectations

Text output should be optimized for human inspection.

It should include, at minimum:

- prompt ID and version
- selected profile ID
- model name
- endpoint
- effective generation settings
- input hashes
- rendered prompt hash
- rendered messages grouped by role

Text output should be readable and deterministic enough for tests.

It should not include raw API key values.

### JSON Output Expectations

JSON output should be a structured representation of `PreparedRun` or a DTO derived from it.

It should include, at minimum:

- prompt ID and version
- selected profile ID
- effective runtime settings
- input hashes
- rendered prompt hash
- rendered messages

JSON output should not include raw API key values.

JSON output should remain stable enough to be useful for automation and integration tests.

## 8. Guardrails

- **Separation of Concerns**: Prompt content must not belong in execution profiles; model/API settings must not belong in prompt definitions.
- **Security**: Raw API keys are unsupported in all configuration and transport layers. Only `api_key_env` is used.
- **Secret Handling**: Resolved API key values must never appear in rendered output, metadata, logs, HTTP responses, or CLI output.
- **Path Resolution**: `content_file` paths in prompt definitions resolve relative to the prompt YAML file.
- **Integrity**: No silent prompt truncation or omission of content.
- **Reliability**: Repair loops are strictly bounded by `repair_attempts`.
- **No Orchestration Creep**: Scriptorium prepares and executes a single prompt request. It does not coordinate multi-stage workflows.
- **Render Reuse**: The full run path must reuse the prepare/render path to avoid divergent behavior.
- **Formatter Isolation**: Render output formatters must not perform usecase work. They only format a completed `PreparedRun`.

## 9. Testing Strategy

Tests should protect both the run path and the prepare/render path.

### Prepare / Render Tests

Add tests for:

- preparing a prompt with explicit profile selection
- preparing a prompt using `default_profile`
- failing when no explicit profile and no `default_profile` exist
- runtime overrides beating profile values
- profile values beating application defaults
- file-backed prompt bodies rendering correctly
- required inputs failing when missing
- optional inputs being absent when not referenced
- unknown input references failing
- input hashes being included
- rendered prompt hash being included
- effective runtime settings being included
- `api_key_env` name being included where appropriate
- resolved API key values never appearing in `PreparedRun`
- prepare path not calling the LLM

### CLI Render Tests

Add tests for:

- `scriptorium render` mapping flags into `RunRequest`
- default text output
- explicit `--format text`
- explicit `--format json`
- unknown format failure
- text output includes prompt/profile/messages
- JSON output includes prompt/profile/messages
- rendered output never includes resolved API key values

### Run Reuse Tests

Add tests proving:

- `Runner.Run` reuses prepare behavior
- run and render resolve the same prompt/profile/runtime settings for equivalent inputs
- run still validates output
- run still performs bounded repair where configured

### Existing Tests

Continue testing:

- prompt definition loading
- execution profile loading
- artifact loading
- prompt rendering
- LLM adapter behavior
- validation behavior
- HTTP request/response mapping

## 10. Extension Points

Future work should remain grounded in the current architecture:

- **Artifacts**: Add S3 artifact references via a new `artifact.Reader`.
- **LLM**: Implement additional provider adapters, such as Anthropic or Google.
- **Execution**: Add token budgeting, streaming generation, and batch execution capabilities.
- **Prepare/Render**: Add token estimates, prompt-size summaries, or additional render output formats.
- **Repositories**: Implement database-backed repositories for prompts and profiles.
- **Profiles**: Support more granular profile versioning and environment-specific profiles.
- **HTTP**: Add an HTTP prepare/render endpoint if Narratio or another caller needs it.

Future render formats should plug into the formatter layer and should not require changes to the usecase layer.