From c3fe88c9fabe1708ff4e838f16da20447921e87b Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 26 May 2026 03:33:25 +0000 Subject: [PATCH] Add maintained examples and clean up legacy documentation paths --- README.md | 5 + architecture.md | 485 +--------------------------- docs/cli.md | 4 + docs/config.md | 2 + docs/config/config-yml.md | 5 - docs/config/profile-definitions.md | 5 - docs/config/prompt-definitions.md | 5 - docs/config/schema-definitions.md | 5 - docs/integrations/http-api.md | 4 + examples/http-run.json | 18 ++ examples/render-markdown-summary.sh | 13 + 11 files changed, 54 insertions(+), 497 deletions(-) delete mode 100644 docs/config/config-yml.md delete mode 100644 docs/config/profile-definitions.md delete mode 100644 docs/config/prompt-definitions.md delete mode 100644 docs/config/schema-definitions.md create mode 100644 examples/http-run.json create mode 100755 examples/render-markdown-summary.sh diff --git a/README.md b/README.md index 13fd396..f82f404 100644 --- a/README.md +++ b/README.md @@ -30,3 +30,8 @@ This command renders the prepared prompt and effective runtime settings without - [Narratio subprocess integration](docs/integrations/narratio.md) - [Architecture policy](docs/policy/architecture.md) - [Documentation roadmap](docs/roadmap/documentation.md) + +## Examples + +- `examples/render-markdown-summary.sh` +- `examples/http-run.json` diff --git a/architecture.md b/architecture.md index 28d9c9b..233aa20 100644 --- a/architecture.md +++ b/architecture.md @@ -1,483 +1,14 @@ -# Scriptorium Architecture +# Architecture (Moved) -## 1. Purpose and Non-Goals +The canonical architecture policy is: -Scriptorium is a prompt-definition execution engine. +- `docs/policy/architecture.md` -It accepts named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata. +Implemented internal component documentation is: -Scriptorium also supports rendering/preparing a prompt without calling an LLM. This allows users to inspect the fully rendered prompt messages and effective runtime settings before executing a run. +- `docs/internal/runner.md` +- `docs/internal/adapters.md` -Scriptorium is not an orchestrator. It must not own transcription, transcript merge/polish steps, notifications, or cross-step workflow control. +Roadmap-only planning content is: -For the motivating D&D workflow: -- Narratio orchestrates. -- WhisperX transcribes. -- Seriatim merges transcripts. -- Audita polishes transcripts. -- Scriptorium generates final artifacts from prepared inputs. - -Core Go code remains generic. - -## 2. Current Architecture - -Scriptorium uses a ports-and-adapters architecture to decouple the core execution logic from external dependencies. - -### Package Responsibilities - -- `cmd/scriptorium`: Binary entrypoint for CLI and HTTP server. -- `internal/domain`: Core domain contracts, including `PromptDefinition`, `ExecutionProfile`, `PreparedRun`, `RunResult`, and related metadata. -- `internal/usecase`: `Runner` use case logic, including prompt preparation, profile selection, runtime override resolution, full run execution, validation, and bounded repair. -- `internal/config`: Application-level config model/loader for adapter settings (for example prompt/profile/schema directories, server address, and render format default). -- `internal/promptdef`: Repository for loading and validating Prompt Definitions from the filesystem. -- `internal/profile`: Repository for loading Execution Profiles from the filesystem. -- `internal/artifact`: Input artifact resolution (`inline`, `file`). -- `internal/prompt`: Template rendering via Go templates. -- `internal/llm`: Provider-neutral client interface and OpenAI-compatible HTTP adapter. -- `internal/validate`: Output validation implementation (`none/basic/json/json_schema`). -- `internal/adapter/cli`: CLI flag parsing, command dispatch, and output handling. -- `internal/adapter/http`: HTTP request/response mapping. -- `internal/format` or equivalent: Formatting of prepared/rendered prompt output for CLI or other adapters, if formatting grows beyond simple CLI-local helpers. - -Exact package names may evolve, but the architectural boundaries should remain stable. - -### Application Configuration - -`config.yml` is adapter/application setup, not domain logic. - -Application config is intended for application-level settings such as: -- `prompt_dir` -- `profile_dir` -- `schema_dir` -- `server.addr` -- `defaults.render_format` - -Application config precedence is: -1. CLI flags -2. `config.yml` -3. Built-in application defaults - -Runtime model settings are intentionally separate: -- Execution profiles and runtime overrides continue to own endpoint/model/runtime behavior. -- `config.yml` does not replace execution profiles. - -The core use case (`Runner.Prepare`/`Runner.Run`) does not need to know whether adapter-level settings came from CLI flags or `config.yml`; it receives resolved dependencies and requests from adapters. - -## 3. Core Execution Model - -Scriptorium has two closely related execution paths: - -1. Prepare/render path. -2. Full run path. - -The full run path should reuse the prepare path rather than duplicating its logic. - -### 3.1 Prepare / Render Data Flow - -The prepare path should be represented in the use case layer, preferably as `Runner.Prepare(ctx, RunRequest)` or an equivalent method. - -It executes all pre-LLM work: - -1. **Validate Request**: Ensure the request includes a prompt ID and all minimum required fields. -2. **Load Prompt Definition**: Retrieve the `PromptDefinition` by ID from the prompt repository. -3. **Select Profile**: Determine the `profile_id` using this precedence: - - Explicit `profile_id` in `RunRequest`. - - `default_profile` specified in the `PromptDefinition`. - - Error if neither is available. -4. **Load Execution Profile**: Retrieve the `ExecutionProfile` from the profile repository. -5. **Resolve Runtime Overrides**: Merge settings based on precedence, highest to lowest: - - Runtime overrides from CLI flags or HTTP request `model` object. - - Execution Profile settings. - - Built-in application defaults. -6. **Resolve Artifacts**: Load all named input artifacts defined in the request. -7. **Render Prompt**: Apply template variables and input artifacts to the prompt templates. -8. **Compute Metadata**: Compute hashes, selected profile ID, prompt ID/version, effective runtime settings, input hashes, rendered prompt hash, and timing information as appropriate. -9. **Return PreparedRun**: Return a `PreparedRun` containing the rendered messages, effective runtime settings, resolved metadata, and input/prompt hashes. - -The prepare path must not call the LLM. - -The prepare path must not validate model output, because there is no model output. - -The prepare path must not perform structured-output repair, because repair only applies after model output exists. - -The prepare path should not resolve or expose raw API key values. It may include the selected `api_key_env` name in effective runtime settings or metadata, but never the environment variable value. - -### 3.2 Full Run Data Flow - -The `Runner.Run(ctx, RunRequest)` flow should reuse the prepare path: - -1. **Prepare**: Call the shared prepare flow to load the prompt, select the profile, resolve artifacts, render prompt messages, and compute pre-run metadata. -2. **Call LLM**: Execute the generation request using the effective runtime settings from the prepared run. -3. **Build Output Artifact**: Convert the model response into the configured output artifact. -4. **Validate Output**: - - Validate the model output against the prompt definition's output contract. - - Validation content failures remain successful run results with `validation.status=failed`. - - Validator runtime/config errors are run errors. -5. **Repair If Configured**: - - If structured validation fails and `repair_attempts > 0`, perform bounded repair attempts. - - Re-validate after each repair attempt. - - Repair loops must remain strictly bounded. -6. **Return RunResult**: Produce a `RunResult` containing the final artifact, validation status, raw model output, usage information, and metadata. - -`Runner.Run` should not duplicate profile selection, artifact resolution, or prompt rendering logic that already exists in `Runner.Prepare`. - -## 4. Domain Model - -Key domain types: - -- `PromptDefinition`: Defines the "what" of the task: templates, inputs, output contract, validation settings, repair settings, and optional `default_profile`. -- `ExecutionProfile`: Defines the "how" of execution: endpoint, model, generation parameters, timeout, reasoning effort, and `api_key_env`. -- `RunRequest`: The intent to execute or prepare a prompt, including `prompt_id`, optional `profile_id`, inputs, variables, and optional runtime overrides. -- `PreparedRun`: The result of the prepare/render phase. Contains rendered messages, effective runtime settings, selected profile ID, prompt metadata, input hashes, prompt hash, and other pre-LLM metadata. -- `RunResult`: The result of a full run. Contains the generated `Artifact`, `ValidationResult`, raw model output, token usage, and `RunMetadata`. -- `RunMetadata`: Detailed tracing information, including prompt ID/version, selected profile ID, effective model parameters, usage tokens, hashes, timestamps, validation status, and repair attempts where applicable. -- `RenderedPrompt`: Provider-neutral rendered prompt structure. -- `RenderedMessage`: Provider-neutral rendered message with role and content. -- `ArtifactRef`: A reference to an input artifact, such as `file` or `inline`. -- `Artifact`: Loaded artifact content with name, content type, body, source URI, size, and hash. -- `ValidationResult`: Validation status and details for full runs. - -`PreparedRun` should be serializable for JSON output and should also be representable in a human-readable text format. - -## 5. Interfaces and Adapters - -### Primary Ports - -- `promptdef.Repository`: Lookup for prompt definitions. -- `profile.Repository`: Lookup for execution profiles. -- `artifact.Reader`: Loading of artifact content. -- `prompt.Renderer`: Template rendering. -- `llm.Client`: Model generation. -- `validate.Validator`: Output validation. -- `format.PreparedRunFormatter` or equivalent: Optional formatting abstraction for rendered/prepared output. - -### Current Adapters - -- **Repositories**: Filesystem YAML loaders for both prompts and profiles. -- **Artifact Reader**: Composite reader supporting `file` and `inline`. -- **Prompt Renderer**: Go templates with a custom `input` helper. -- **LLM Client**: OpenAI-compatible `/chat/completions` over HTTP. -- **Validator**: Standard validator supporting `none`, `basic`, `json`, and `json_schema`. -- **CLI Adapter**: Supports `run`, `render`, and `serve`. -- **HTTP Adapter**: Supports full run execution through `POST /v1/runs`. - -## 6. Public Contracts - -### CLI - -Scriptorium should expose at least these commands: - -- `scriptorium run`: Executes a prompt by preparing it, calling the LLM, validating output, optionally repairing structured output, and returning an artifact. -- `scriptorium render`: Prepares and renders a prompt without calling the LLM. -- `scriptorium serve`: Starts the HTTP API using infrastructure-only flags. - -### `scriptorium run` - -`run` uses flags such as: - -- `--prompt-dir` -- `--profile-dir` -- `--prompt` -- `--profile` -- `--input` -- `--var` -- `--out` -- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported. - -`run` should produce the generated artifact as its primary output. - -### `scriptorium render` - -`render` prepares and renders a prompt without calling an LLM. - -It should use the same prompt/profile/input/variable/runtime override flags as `run` where applicable: - -- `--prompt-dir` -- `--profile-dir` -- `--prompt` -- `--profile` -- `--input` -- `--var` -- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported. -- `--format`, with initial support for `text` and `json`. - -Default render output format should be `text`. - -`render` must not call the LLM. - -`render` should show the same rendered messages and effective runtime settings that `run` would use. - -`render` should include enough information to debug: - -- prompt ID -- prompt version -- selected profile ID -- effective runtime settings -- input hashes -- prompt hash -- rendered messages - -`render` must not include resolved API key values. - -`render` may include the `api_key_env` name. - -### Render Output Formats - -Initial render output formats: - -- `text`: Human-readable default format. -- `json`: Machine-readable structured representation of the prepared run. - -Additional formats, such as `markdown`, may be added later. - -Render output formatting should be modular. Adding a new output format should not require changing the prepare/run core logic. - -The output formatting layer should consume a `PreparedRun` and produce bytes or text for the adapter. It should not reload prompts, re-resolve artifacts, re-render templates, call the LLM, or perform validation. - -### `scriptorium serve` - -`serve` starts the HTTP API. - -It should use infrastructure-only flags such as: - -- `--addr` -- `--prompt-dir` -- `--profile-dir` -- `--schema-dir` - -`serve` should not introduce a server-level model/runtime precedence layer unless explicitly documented and intentionally implemented. - -### HTTP API - -Current HTTP API: - -- `POST /v1/runs`: Accepts `RunRequest` JSON and returns `RunResponse` JSON. No built-in auth. - -Request may include runtime overrides under `model` and an `include_raw_output` boolean. - -`raw_model_output` is exposed only when explicitly requested with `include_raw_output=true`. - -A future HTTP prepare/render endpoint may be added, such as `POST /v1/renders` or `POST /v1/runs/prepare`, but the initial render feature may be CLI-only. If added later, it should call the same usecase-level prepare path as `scriptorium render`. - -### YAML Shapes - -Prompt YAML includes: - -- `id` -- `version` -- optional `default_profile` -- `inputs` -- `messages` -- `output` - -Inputs support: - -- `name` -- `required` -- optional `content_type` -- `description` - -Messages require: - -- `role` -- exactly one of `content` or `content_file` - -`content_file` resolves relative to the prompt YAML location. - -Profile YAML includes: - -- `id` -- `endpoint` -- `model` -- generation parameters -- timeout settings -- reasoning settings -- `api_key_env` - -Prompt content must not appear in profile YAML. - -Model/runtime/API-key settings must not appear in prompt YAML, except that prompt YAML may specify `default_profile`. - -## 7. Render Feature Design - -The render feature is a first-class use case, not a CLI-only shortcut. - -### Goals - -The render feature should help users: - -- inspect fully rendered prompt messages -- debug missing inputs -- verify template variable substitution -- verify selected profile resolution -- verify runtime override precedence -- verify file-backed prompt loading -- inspect input hashes and prompt hashes -- prepare for future token budgeting and prompt-size inspection - -### Non-Goals - -The render feature should not: - -- call an LLM -- validate model output -- repair structured output -- resolve or print API key values -- mutate artifacts -- save outputs to artifact storage unless a future explicit output option is added -- become an orchestration step manager - -### Usecase Shape - -The preferred usecase shape is: - -- `Runner.Prepare(ctx, RunRequest) (*PreparedRun, error)` -- `Runner.Run(ctx, RunRequest) (*RunResult, error)` - -`Runner.Run` should call `Runner.Prepare`. - -The prepare flow should be the only implementation of: - -- prompt loading -- profile selection -- runtime override resolution -- artifact resolution -- prompt rendering -- pre-run metadata/hash calculation - -### CLI Shape - -The preferred command name is `render`. - -The command should support: - -- `--format text` -- `--format json` - -Default format: - -- `text` - -Unknown formats should produce a clear error. - -Formatting should be centralized through a small formatter registry, strategy, switch, or interface so new formats can be added without modifying usecase logic. - -### Text Output Expectations - -Text output should be optimized for human inspection. - -It should include, at minimum: - -- prompt ID and version -- selected profile ID -- model name -- endpoint -- effective generation settings -- input hashes -- rendered prompt hash -- rendered messages grouped by role - -Text output should be readable and deterministic enough for tests. - -It should not include raw API key values. - -### JSON Output Expectations - -JSON output should be a structured representation of `PreparedRun` or a DTO derived from it. - -It should include, at minimum: - -- prompt ID and version -- selected profile ID -- effective runtime settings -- input hashes -- rendered prompt hash -- rendered messages - -JSON output should not include raw API key values. - -JSON output should remain stable enough to be useful for automation and integration tests. - -## 8. Guardrails - -- **Separation of Concerns**: Prompt content must not belong in execution profiles; model/API settings must not belong in prompt definitions. -- **Security**: Raw API keys are unsupported in all configuration and transport layers. Only `api_key_env` is used. -- **Secret Handling**: Resolved API key values must never appear in rendered output, metadata, logs, HTTP responses, or CLI output. -- **Path Resolution**: `content_file` paths in prompt definitions resolve relative to the prompt YAML file. -- **Integrity**: No silent prompt truncation or omission of content. -- **Reliability**: Repair loops are strictly bounded by `repair_attempts`. -- **No Orchestration Creep**: Scriptorium prepares and executes a single prompt request. It does not coordinate multi-stage workflows. -- **Render Reuse**: The full run path must reuse the prepare/render path to avoid divergent behavior. -- **Formatter Isolation**: Render output formatters must not perform usecase work. They only format a completed `PreparedRun`. - -## 9. Testing Strategy - -Tests should protect both the run path and the prepare/render path. - -### Prepare / Render Tests - -Add tests for: - -- preparing a prompt with explicit profile selection -- preparing a prompt using `default_profile` -- failing when no explicit profile and no `default_profile` exist -- runtime overrides beating profile values -- profile values beating application defaults -- file-backed prompt bodies rendering correctly -- required inputs failing when missing -- optional inputs being absent when not referenced -- unknown input references failing -- input hashes being included -- rendered prompt hash being included -- effective runtime settings being included -- `api_key_env` name being included where appropriate -- resolved API key values never appearing in `PreparedRun` -- prepare path not calling the LLM - -### CLI Render Tests - -Add tests for: - -- `scriptorium render` mapping flags into `RunRequest` -- default text output -- explicit `--format text` -- explicit `--format json` -- unknown format failure -- text output includes prompt/profile/messages -- JSON output includes prompt/profile/messages -- rendered output never includes resolved API key values - -### Run Reuse Tests - -Add tests proving: - -- `Runner.Run` reuses prepare behavior -- run and render resolve the same prompt/profile/runtime settings for equivalent inputs -- run still validates output -- run still performs bounded repair where configured - -### Existing Tests - -Continue testing: - -- prompt definition loading -- execution profile loading -- artifact loading -- prompt rendering -- LLM adapter behavior -- validation behavior -- HTTP request/response mapping - -## 10. Extension Points - -Future work should remain grounded in the current architecture: - -- **Artifacts**: Add S3 artifact references via a new `artifact.Reader`. -- **LLM**: Implement additional provider adapters, such as Anthropic or Google. -- **Execution**: Add token budgeting, streaming generation, and batch execution capabilities. -- **Prepare/Render**: Add token estimates, prompt-size summaries, or additional render output formats. -- **Repositories**: Implement database-backed repositories for prompts and profiles. -- **Profiles**: Support more granular profile versioning and environment-specific profiles. -- **HTTP**: Add an HTTP prepare/render endpoint if Narratio or another caller needs it. - -Future render formats should plug into the formatter layer and should not require changes to the usecase layer. +- `docs/roadmap/documentation.md` diff --git a/docs/cli.md b/docs/cli.md index 21c9a74..5f50406 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -142,3 +142,7 @@ Start the HTTP server with explicit config: ```bash go run ./cmd/scriptorium serve --config ./examples/config.yml ``` + +Copyable example script: + +- `examples/render-markdown-summary.sh` diff --git a/docs/config.md b/docs/config.md index 3109dd9..a576912 100644 --- a/docs/config.md +++ b/docs/config.md @@ -204,6 +204,8 @@ Supported artifact reference types for request inputs are `file` and `inline`. - Profile examples: `profiles/` - Schema examples: `schemas/` - Input fixtures: `examples/fixtures/` +- Render example script: `examples/render-markdown-summary.sh` +- HTTP request example: `examples/http-run.json` ## Integration References diff --git a/docs/config/config-yml.md b/docs/config/config-yml.md deleted file mode 100644 index b20c7cd..0000000 --- a/docs/config/config-yml.md +++ /dev/null @@ -1,5 +0,0 @@ -# Moved - -This page has moved. - -Use the canonical configuration reference: [../config.md](../config.md). diff --git a/docs/config/profile-definitions.md b/docs/config/profile-definitions.md deleted file mode 100644 index b20c7cd..0000000 --- a/docs/config/profile-definitions.md +++ /dev/null @@ -1,5 +0,0 @@ -# Moved - -This page has moved. - -Use the canonical configuration reference: [../config.md](../config.md). diff --git a/docs/config/prompt-definitions.md b/docs/config/prompt-definitions.md deleted file mode 100644 index b20c7cd..0000000 --- a/docs/config/prompt-definitions.md +++ /dev/null @@ -1,5 +0,0 @@ -# Moved - -This page has moved. - -Use the canonical configuration reference: [../config.md](../config.md). diff --git a/docs/config/schema-definitions.md b/docs/config/schema-definitions.md deleted file mode 100644 index b20c7cd..0000000 --- a/docs/config/schema-definitions.md +++ /dev/null @@ -1,5 +0,0 @@ -# Moved - -This page has moved. - -Use the canonical configuration reference: [../config.md](../config.md). diff --git a/docs/integrations/http-api.md b/docs/integrations/http-api.md index b281dae..af59d5c 100644 --- a/docs/integrations/http-api.md +++ b/docs/integrations/http-api.md @@ -21,6 +21,10 @@ Route behavior: - unknown path: `404 not_found` - unsupported method on `/v1/runs`: `405 method_not_allowed` +Copyable request example file: + +- `examples/http-run.json` + ## Request Body ```json diff --git a/examples/http-run.json b/examples/http-run.json new file mode 100644 index 0000000..f56bf66 --- /dev/null +++ b/examples/http-run.json @@ -0,0 +1,18 @@ +{ + "prompt_id": "generic.markdown_summary", + "profile_id": "local-fast", + "inputs": { + "transcript": { + "type": "file", + "uri": "./examples/fixtures/transcript.md" + }, + "glossary": { + "type": "file", + "uri": "./examples/fixtures/glossary.yml" + } + }, + "vars": { + "session_date": "2026-05-04" + }, + "include_raw_output": false +} diff --git a/examples/render-markdown-summary.sh b/examples/render-markdown-summary.sh new file mode 100755 index 0000000..12b6ef0 --- /dev/null +++ b/examples/render-markdown-summary.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +set -euo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +cd "$repo_root" + +go run ./cmd/scriptorium render \ + --config ./examples/config.yml \ + --prompt generic.markdown_summary \ + --input transcript=./examples/fixtures/transcript.md \ + --input glossary=./examples/fixtures/glossary.yml \ + --format text