diff --git a/architecture.md b/architecture.md index 3f2dc51..46cb976 100644 --- a/architecture.md +++ b/architecture.md @@ -6,6 +6,8 @@ Scriptorium is a prompt-definition execution engine. It accepts named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata. +Scriptorium also supports rendering/preparing a prompt without calling an LLM. This allows users to inspect the fully rendered prompt messages and effective runtime settings before executing a run. + Scriptorium is not an orchestrator. It must not own transcription, transcript merge/polish steps, notifications, or cross-step workflow control. For the motivating D&D workflow: @@ -22,97 +24,437 @@ Core Go code remains generic. Scriptorium uses a ports-and-adapters architecture to decouple the core execution logic from external dependencies. ### Package Responsibilities + - `cmd/scriptorium`: Binary entrypoint for CLI and HTTP server. -- `internal/domain`: Core domain contracts, including `PromptDefinition`, `ExecutionProfile`, and `RunResult`. -- `internal/usecase`: `Runner` orchestration, including the logic for profile selection, runtime override resolution, and bounded repair. +- `internal/domain`: Core domain contracts, including `PromptDefinition`, `ExecutionProfile`, `PreparedRun`, `RunResult`, and related metadata. +- `internal/usecase`: `Runner` use case logic, including prompt preparation, profile selection, runtime override resolution, full run execution, validation, and bounded repair. - `internal/promptdef`: Repository for loading and validating Prompt Definitions from the filesystem. - `internal/profile`: Repository for loading Execution Profiles from the filesystem. - `internal/artifact`: Input artifact resolution (`inline`, `file`). - `internal/prompt`: Template rendering via Go templates. - `internal/llm`: Provider-neutral client interface and OpenAI-compatible HTTP adapter. - `internal/validate`: Output validation implementation (`none/basic/json/json_schema`). -- `internal/adapter/cli`: CLI flag parsing and output handling. +- `internal/adapter/cli`: CLI flag parsing, command dispatch, and output handling. - `internal/adapter/http`: HTTP request/response mapping. +- `internal/format` or equivalent: Formatting of prepared/rendered prompt output for CLI or other adapters, if formatting grows beyond simple CLI-local helpers. -## 3. Run Data Flow +Exact package names may evolve, but the architectural boundaries should remain stable. -The `Runner.Run` flow executes the following steps: +## 3. Core Execution Model -1. **Load Prompt Definition**: Retrieve the `PromptDefinition` by ID from the prompt repository. -2. **Select Profile**: Determine the `profile_id` using the precedence: +Scriptorium has two closely related execution paths: + +1. Prepare/render path. +2. Full run path. + +The full run path should reuse the prepare path rather than duplicating its logic. + +### 3.1 Prepare / Render Data Flow + +The prepare path should be represented in the use case layer, preferably as `Runner.Prepare(ctx, RunRequest)` or an equivalent method. + +It executes all pre-LLM work: + +1. **Validate Request**: Ensure the request includes a prompt ID and all minimum required fields. +2. **Load Prompt Definition**: Retrieve the `PromptDefinition` by ID from the prompt repository. +3. **Select Profile**: Determine the `profile_id` using this precedence: - Explicit `profile_id` in `RunRequest`. - `default_profile` specified in the `PromptDefinition`. - Error if neither is available. -3. **Load Execution Profile**: Retrieve the `ExecutionProfile` from the profile repository. -4. **Resolve Runtime Overrides**: Merge settings based on precedence (Highest to Lowest): - - Runtime overrides (CLI flags or HTTP `model` object). +4. **Load Execution Profile**: Retrieve the `ExecutionProfile` from the profile repository. +5. **Resolve Runtime Overrides**: Merge settings based on precedence, highest to lowest: + - Runtime overrides from CLI flags or HTTP request `model` object. - Execution Profile settings. - Built-in application defaults. -5. **Resolve Artifacts**: Load all named input artifacts defined in the request. -6. **Render Prompt**: Apply template variables and input artifacts to the prompt templates. -7. **Call LLM**: Execute the generation request using the resolved execution target (effective runtime settings). -8. **Validate/Repair**: - - Validate the model output against the output contract. - - If structured validation fails and `repair_attempts > 0`, perform bounded repair and re-validate. -9. **Return Result**: Produce a `RunResult` containing the final artifact, metadata, and validation status. +6. **Resolve Artifacts**: Load all named input artifacts defined in the request. +7. **Render Prompt**: Apply template variables and input artifacts to the prompt templates. +8. **Compute Metadata**: Compute hashes, selected profile ID, prompt ID/version, effective runtime settings, input hashes, rendered prompt hash, and timing information as appropriate. +9. **Return PreparedRun**: Return a `PreparedRun` containing the rendered messages, effective runtime settings, resolved metadata, and input/prompt hashes. + +The prepare path must not call the LLM. + +The prepare path must not validate model output, because there is no model output. + +The prepare path must not perform structured-output repair, because repair only applies after model output exists. + +The prepare path should not resolve or expose raw API key values. It may include the selected `api_key_env` name in effective runtime settings or metadata, but never the environment variable value. + +### 3.2 Full Run Data Flow + +The `Runner.Run(ctx, RunRequest)` flow should reuse the prepare path: + +1. **Prepare**: Call the shared prepare flow to load the prompt, select the profile, resolve artifacts, render prompt messages, and compute pre-run metadata. +2. **Call LLM**: Execute the generation request using the effective runtime settings from the prepared run. +3. **Build Output Artifact**: Convert the model response into the configured output artifact. +4. **Validate Output**: + - Validate the model output against the prompt definition's output contract. + - Validation content failures remain successful run results with `validation.status=failed`. + - Validator runtime/config errors are run errors. +5. **Repair If Configured**: + - If structured validation fails and `repair_attempts > 0`, perform bounded repair attempts. + - Re-validate after each repair attempt. + - Repair loops must remain strictly bounded. +6. **Return RunResult**: Produce a `RunResult` containing the final artifact, validation status, raw model output, usage information, and metadata. + +`Runner.Run` should not duplicate profile selection, artifact resolution, or prompt rendering logic that already exists in `Runner.Prepare`. ## 4. Domain Model Key domain types: -- `PromptDefinition`: Defines the "what" (templates, inputs, validation contract, and an optional `default_profile`). -- `ExecutionProfile`: Defines the "how" (endpoint, model, generation parameters, and `api_key_env`). -- `RunRequest`: The intent to execute a prompt, including `prompt_id`, optional `profile_id`, inputs, variables, and optional runtime overrides. -- `RunResult`: The outcome of a run, including the generated `Artifact`, `ValidationResult`, and auditing `RunMetadata`. -- `RunMetadata`: Detailed tracing info: `prompt_id`, `selected_profile_id`, model params, usage tokens, and hashes. + +- `PromptDefinition`: Defines the "what" of the task: templates, inputs, output contract, validation settings, repair settings, and optional `default_profile`. +- `ExecutionProfile`: Defines the "how" of execution: endpoint, model, generation parameters, timeout, reasoning effort, and `api_key_env`. +- `RunRequest`: The intent to execute or prepare a prompt, including `prompt_id`, optional `profile_id`, inputs, variables, and optional runtime overrides. +- `PreparedRun`: The result of the prepare/render phase. Contains rendered messages, effective runtime settings, selected profile ID, prompt metadata, input hashes, prompt hash, and other pre-LLM metadata. +- `RunResult`: The result of a full run. Contains the generated `Artifact`, `ValidationResult`, raw model output, token usage, and `RunMetadata`. +- `RunMetadata`: Detailed tracing information, including prompt ID/version, selected profile ID, effective model parameters, usage tokens, hashes, timestamps, validation status, and repair attempts where applicable. +- `RenderedPrompt`: Provider-neutral rendered prompt structure. +- `RenderedMessage`: Provider-neutral rendered message with role and content. +- `ArtifactRef`: A reference to an input artifact, such as `file` or `inline`. +- `Artifact`: Loaded artifact content with name, content type, body, source URI, size, and hash. +- `ValidationResult`: Validation status and details for full runs. + +`PreparedRun` should be serializable for JSON output and should also be representable in a human-readable text format. ## 5. Interfaces and Adapters ### Primary Ports + - `promptdef.Repository`: Lookup for prompt definitions. - `profile.Repository`: Lookup for execution profiles. - `artifact.Reader`: Loading of artifact content. - `prompt.Renderer`: Template rendering. - `llm.Client`: Model generation. - `validate.Validator`: Output validation. +- `format.PreparedRunFormatter` or equivalent: Optional formatting abstraction for rendered/prepared output. ### Current Adapters + - **Repositories**: Filesystem YAML loaders for both prompts and profiles. - **Artifact Reader**: Composite reader supporting `file` and `inline`. - **Prompt Renderer**: Go templates with a custom `input` helper. - **LLM Client**: OpenAI-compatible `/chat/completions` over HTTP. - **Validator**: Standard validator supporting `none`, `basic`, `json`, and `json_schema`. +- **CLI Adapter**: Supports `run`, `render`, and `serve`. +- **HTTP Adapter**: Supports full run execution through `POST /v1/runs`. ## 6. Public Contracts ### CLI -- `run`: Executes a prompt. Uses flags like `--prompt`, `--profile`, `--input`, and various runtime overrides (e.g., `--model`, `--temperature`). -- `serve`: Starts the HTTP API using infrastructure-only flags (`--addr`, `--prompt-dir`, `--profile-dir`, `--schema-dir`). It does not introduce a server-level model/runtime precedence layer. + +Scriptorium should expose at least these commands: + +- `scriptorium run`: Executes a prompt by preparing it, calling the LLM, validating output, optionally repairing structured output, and returning an artifact. +- `scriptorium render`: Prepares and renders a prompt without calling the LLM. +- `scriptorium serve`: Starts the HTTP API using infrastructure-only flags. + +### `scriptorium run` + +`run` uses flags such as: + +- `--prompt-dir` +- `--profile-dir` +- `--prompt` +- `--profile` +- `--input` +- `--var` +- `--out` +- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported. + +`run` should produce the generated artifact as its primary output. + +### `scriptorium render` + +`render` prepares and renders a prompt without calling an LLM. + +It should use the same prompt/profile/input/variable/runtime override flags as `run` where applicable: + +- `--prompt-dir` +- `--profile-dir` +- `--prompt` +- `--profile` +- `--input` +- `--var` +- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported. +- `--format`, with initial support for `text` and `json`. + +Default render output format should be `text`. + +`render` must not call the LLM. + +`render` should show the same rendered messages and effective runtime settings that `run` would use. + +`render` should include enough information to debug: + +- prompt ID +- prompt version +- selected profile ID +- effective runtime settings +- input hashes +- prompt hash +- rendered messages + +`render` must not include resolved API key values. + +`render` may include the `api_key_env` name. + +### Render Output Formats + +Initial render output formats: + +- `text`: Human-readable default format. +- `json`: Machine-readable structured representation of the prepared run. + +Additional formats, such as `markdown`, may be added later. + +Render output formatting should be modular. Adding a new output format should not require changing the prepare/run core logic. + +The output formatting layer should consume a `PreparedRun` and produce bytes or text for the adapter. It should not reload prompts, re-resolve artifacts, re-render templates, call the LLM, or perform validation. + +### `scriptorium serve` + +`serve` starts the HTTP API. + +It should use infrastructure-only flags such as: + +- `--addr` +- `--prompt-dir` +- `--profile-dir` +- `--schema-dir` + +`serve` should not introduce a server-level model/runtime precedence layer unless explicitly documented and intentionally implemented. ### HTTP API + +Current HTTP API: + - `POST /v1/runs`: Accepts `RunRequest` JSON and returns `RunResponse` JSON. No built-in auth. -- Request may include runtime overrides under `model` and an `include_raw_output` boolean. -- `raw_model_output` is exposed only when explicitly requested with `include_raw_output=true`. + +Request may include runtime overrides under `model` and an `include_raw_output` boolean. + +`raw_model_output` is exposed only when explicitly requested with `include_raw_output=true`. + +A future HTTP prepare/render endpoint may be added, such as `POST /v1/renders` or `POST /v1/runs/prepare`, but the initial render feature may be CLI-only. If added later, it should call the same usecase-level prepare path as `scriptorium render`. ### YAML Shapes -- **Prompt YAML**: Includes `id`, `version`, optional `default_profile`, `inputs`, `messages`, and `output`. - - Inputs support `name`, `required`, optional `content_type`, and `description`. - - Messages require `role` and exactly one of `content` or `content_file`. - - `content_file` resolves relative to the prompt YAML location. -- **Profile YAML**: Includes `id`, `endpoint`, `model`, generation params, and `api_key_env`. -## 7. Guardrails +Prompt YAML includes: + +- `id` +- `version` +- optional `default_profile` +- `inputs` +- `messages` +- `output` + +Inputs support: + +- `name` +- `required` +- optional `content_type` +- `description` + +Messages require: + +- `role` +- exactly one of `content` or `content_file` + +`content_file` resolves relative to the prompt YAML location. + +Profile YAML includes: + +- `id` +- `endpoint` +- `model` +- generation parameters +- timeout settings +- reasoning settings +- `api_key_env` + +Prompt content must not appear in profile YAML. + +Model/runtime/API-key settings must not appear in prompt YAML, except that prompt YAML may specify `default_profile`. + +## 7. Render Feature Design + +The render feature is a first-class use case, not a CLI-only shortcut. + +### Goals + +The render feature should help users: + +- inspect fully rendered prompt messages +- debug missing inputs +- verify template variable substitution +- verify selected profile resolution +- verify runtime override precedence +- verify file-backed prompt loading +- inspect input hashes and prompt hashes +- prepare for future token budgeting and prompt-size inspection + +### Non-Goals + +The render feature should not: + +- call an LLM +- validate model output +- repair structured output +- resolve or print API key values +- mutate artifacts +- save outputs to artifact storage unless a future explicit output option is added +- become an orchestration step manager + +### Usecase Shape + +The preferred usecase shape is: + +- `Runner.Prepare(ctx, RunRequest) (*PreparedRun, error)` +- `Runner.Run(ctx, RunRequest) (*RunResult, error)` + +`Runner.Run` should call `Runner.Prepare`. + +The prepare flow should be the only implementation of: + +- prompt loading +- profile selection +- runtime override resolution +- artifact resolution +- prompt rendering +- pre-run metadata/hash calculation + +### CLI Shape + +The preferred command name is `render`. + +The command should support: + +- `--format text` +- `--format json` + +Default format: + +- `text` + +Unknown formats should produce a clear error. + +Formatting should be centralized through a small formatter registry, strategy, switch, or interface so new formats can be added without modifying usecase logic. + +### Text Output Expectations + +Text output should be optimized for human inspection. + +It should include, at minimum: + +- prompt ID and version +- selected profile ID +- model name +- endpoint +- effective generation settings +- input hashes +- rendered prompt hash +- rendered messages grouped by role + +Text output should be readable and deterministic enough for tests. + +It should not include raw API key values. + +### JSON Output Expectations + +JSON output should be a structured representation of `PreparedRun` or a DTO derived from it. + +It should include, at minimum: + +- prompt ID and version +- selected profile ID +- effective runtime settings +- input hashes +- rendered prompt hash +- rendered messages + +JSON output should not include raw API key values. + +JSON output should remain stable enough to be useful for automation and integration tests. + +## 8. Guardrails - **Separation of Concerns**: Prompt content must not belong in execution profiles; model/API settings must not belong in prompt definitions. - **Security**: Raw API keys are unsupported in all configuration and transport layers. Only `api_key_env` is used. +- **Secret Handling**: Resolved API key values must never appear in rendered output, metadata, logs, HTTP responses, or CLI output. - **Path Resolution**: `content_file` paths in prompt definitions resolve relative to the prompt YAML file. - **Integrity**: No silent prompt truncation or omission of content. - **Reliability**: Repair loops are strictly bounded by `repair_attempts`. +- **No Orchestration Creep**: Scriptorium prepares and executes a single prompt request. It does not coordinate multi-stage workflows. +- **Render Reuse**: The full run path must reuse the prepare/render path to avoid divergent behavior. +- **Formatter Isolation**: Render output formatters must not perform usecase work. They only format a completed `PreparedRun`. -## 8. Extension Points +## 9. Testing Strategy + +Tests should protect both the run path and the prepare/render path. + +### Prepare / Render Tests + +Add tests for: + +- preparing a prompt with explicit profile selection +- preparing a prompt using `default_profile` +- failing when no explicit profile and no `default_profile` exist +- runtime overrides beating profile values +- profile values beating application defaults +- file-backed prompt bodies rendering correctly +- required inputs failing when missing +- optional inputs being absent when not referenced +- unknown input references failing +- input hashes being included +- rendered prompt hash being included +- effective runtime settings being included +- `api_key_env` name being included where appropriate +- resolved API key values never appearing in `PreparedRun` +- prepare path not calling the LLM + +### CLI Render Tests + +Add tests for: + +- `scriptorium render` mapping flags into `RunRequest` +- default text output +- explicit `--format text` +- explicit `--format json` +- unknown format failure +- text output includes prompt/profile/messages +- JSON output includes prompt/profile/messages +- rendered output never includes resolved API key values + +### Run Reuse Tests + +Add tests proving: + +- `Runner.Run` reuses prepare behavior +- run and render resolve the same prompt/profile/runtime settings for equivalent inputs +- run still validates output +- run still performs bounded repair where configured + +### Existing Tests + +Continue testing: + +- prompt definition loading +- execution profile loading +- artifact loading +- prompt rendering +- LLM adapter behavior +- validation behavior +- HTTP request/response mapping + +## 10. Extension Points Future work should remain grounded in the current architecture: + - **Artifacts**: Add S3 artifact references via a new `artifact.Reader`. -- **LLM**: Implement additional provider adapters (e.g., Anthropic, Google). +- **LLM**: Implement additional provider adapters, such as Anthropic or Google. - **Execution**: Add token budgeting, streaming generation, and batch execution capabilities. +- **Prepare/Render**: Add token estimates, prompt-size summaries, or additional render output formats. - **Repositories**: Implement database-backed repositories for prompts and profiles. - **Profiles**: Support more granular profile versioning and environment-specific profiles. +- **HTTP**: Add an HTTP prepare/render endpoint if Narratio or another caller needs it. + +Future render formats should plug into the formatter layer and should not require changes to the usecase layer. \ No newline at end of file