diff --git a/README.md b/README.md index fe7e44c..da67439 100644 --- a/README.md +++ b/README.md @@ -2,24 +2,87 @@ Scriptorium is a generic prompt-profile execution engine written in Go. -It loads a prompt profile, resolves named input artifacts, renders a prompt, calls an OpenAI-compatible LLM endpoint, validates output, and returns an artifact plus metadata. +Given named input artifacts and a prompt profile, Scriptorium: -## Relationship to Narratio +1. Loads the profile. +2. Resolves input artifact references. +3. Renders prompt messages from templates. +4. Calls an OpenAI-compatible LLM endpoint. +5. Validates output if configured. +6. Optionally performs bounded structured-output repair. +7. Returns a generated artifact plus run metadata. -In the broader workflow, Narratio handles pipeline orchestration (transcription, cleanup, storage, notifications). Scriptorium handles only prompt-profile execution for a single run. +## Where Scriptorium Fits -## Repository Example Assets +Scriptorium is not an orchestrator. -- Profiles: `profiles/` -- Schemas: `schemas/` -- Tiny fixtures: `examples/fixtures/` +In the D&D workflow: -Included profiles: -- `generic.markdown_summary` -- `dnd.session_recap` (example content only; no D&D-specific Go logic) -- `generic.structured_events` (JSON + JSON Schema validation) +- Narratio orchestrates the full pipeline. +- WhisperX transcribes audio. +- Seriatim merges transcripts. +- Audita polishes transcripts. +- Scriptorium generates final artifacts from prepared inputs. -## Run a Local Profile (CLI) +D&D-specific behavior belongs in profiles, schemas, fixtures, and caller inputs, not in core Go logic. + +## Core Concepts + +- Prompt profile: YAML config that defines templates, model defaults, output format, and validation behavior. +- Named inputs: logical input names (for example `transcript`, `glossary`) mapped to artifact references. +- Artifact refs: currently `file` and `inline` are supported by readers used in v1 flows. +- Template variables: key/value vars provided at run time and accessed in templates as `{{.var_name}}`. +- Model target: endpoint/model and generation parameters (`temperature`, `max_tokens`, `top_p`, `timeout_seconds`). +- Output format: `text`, `markdown`, or `json`. +- Validation mode: `none`, `basic`, `json`, `json_schema`. +- Repair attempts: bounded retries for structured modes (`json`, `json_schema`) when output validation fails. +- Run metadata: IDs/hashes/model/timing/usage/validation details for auditability. + +## Build and Test + +Build: + +```bash +go build -o scriptorium ./cmd/scriptorium +``` + +Run tests: + +```bash +go test ./... +``` + +Run CLI locally: + +```bash +go run ./cmd/scriptorium run --help +``` + +## CLI Usage + +### `scriptorium run` + +Required flags: + +- `--profile-dir` +- `--profile-id` +- `--input` (repeatable `name=path`) + +Optional flags: + +- `--var` (repeatable `name=value`) +- `--out` +- `--llm-base-url` +- `--llm-api-key` +- `--model` +- `--temperature` +- `--max-tokens` +- `--schema-dir` +- `--timeout` + +If `--llm-base-url` and/or `--model` are omitted, profile `model_defaults` must provide them. + +Markdown summary example: ```bash go run ./cmd/scriptorium run \ @@ -30,81 +93,222 @@ go run ./cmd/scriptorium run \ --out ./out.md ``` -This relies on `model_defaults.endpoint` and `model_defaults.model` in the selected profile. -You can override either at runtime with `--llm-base-url` and/or `--model`. - -For schema-validated JSON output: +Same run with explicit local OpenAI-compatible endpoint (for example vLLM): ```bash go run ./cmd/scriptorium run \ --profile-dir ./profiles \ - --profile-id generic.structured_events \ + --profile-id generic.markdown_summary \ --input transcript=./examples/fixtures/transcript.md \ --input glossary=./examples/fixtures/glossary.yml \ --llm-base-url http://localhost:8000/v1 \ --model gpt-4o-mini \ - --schema-dir ./schemas \ - --out ./events.json + --out ./out.md ``` -## Start Local HTTP API +Passing template variables: ```bash -go run ./cmd/scriptorium serve \ - --addr :8080 \ +go run ./cmd/scriptorium run \ --profile-dir ./profiles \ - --schema-dir ./schemas \ - --llm-base-url http://localhost:8000/v1 \ - --model gpt-4o-mini + --profile-id generic.markdown_summary \ + --input transcript=./examples/fixtures/transcript.md \ + --input glossary=./examples/fixtures/glossary.yml \ + --var session_date=2026-05-04 \ + --var facilitator="Eris" \ + --out ./out.md ``` -## Call `POST /v1/runs` +Output behavior: -```bash -curl -sS http://localhost:8080/v1/runs \ - -H 'Content-Type: application/json' \ - -d '{ - "profile_id": "generic.structured_events", - "inputs": { - "transcript": {"type": "file", "uri": "./examples/fixtures/transcript.md"}, - "glossary": {"type": "file", "uri": "./examples/fixtures/glossary.yml"} - }, - "model": {"model": "gpt-4o-mini"} - }' +- Artifact content goes to stdout unless `--out` is set. +- Summaries and errors are written to stderr. +- Exit code `2` indicates run succeeded but validation status is `failed`. + +### `scriptorium serve` + +Starts HTTP API. + +Required flags: + +- `--profile-dir` +- `--llm-base-url` + +Common optional flags: + +- `--addr` (default `:8080`) +- `--schema-dir` (default `.`) +- `--llm-api-key` +- `--model` +- `--timeout` (default `10m`) + +## HTTP API + +Run endpoint: + +- `POST /v1/runs` + +Request example: + +```json +{ + "profile_id": "generic.structured_events", + "profile_version": "1.0.0", + "inputs": { + "transcript": {"type": "file", "uri": "./examples/fixtures/transcript.md"}, + "glossary": {"type": "file", "uri": "./examples/fixtures/glossary.yml"} + }, + "vars": { + "session_date": "2026-05-04" + }, + "model": { + "endpoint": "http://localhost:8000/v1", + "model": "gpt-4o-mini", + "temperature": 0.0, + "max_tokens": 600, + "top_p": 1.0, + "timeout_seconds": 120 + } +} ``` Response shape: -- `artifact` -- `validation` -- `metadata` -- `raw_model_output` -`metadata` includes stable audit fields such as run/profile IDs, profile hash, effective model params, prompt/input hashes, timing, usage, and validation summary. +```json +{ + "artifact": { + "name": "output", + "content_type": "application/json", + "body": "{...}", + "uri": "", + "size": 123, + "hash": "..." + }, + "validation": { + "status": "passed", + "mode": "json_schema", + "errors": [], + "schema_path": "structured_events.schema.json", + "repair_attempts": 0, + "is_valid": true + }, + "metadata": { + "run_id": "xxxxxxxx-xxxx-4xxx-8xxx-xxxxxxxxxxxx", + "profile_id": "generic.structured_events", + "profile_version": "1.0.0", + "profile_hash": "...", + "model_name": "gpt-4o-mini", + "endpoint": "http://localhost:8000/v1", + "model_params": { + "endpoint": "http://localhost:8000/v1", + "model": "gpt-4o-mini", + "temperature": 0, + "max_tokens": 600, + "top_p": 1, + "timeout_seconds": 120 + }, + "input_hashes": {"transcript": "...", "glossary": "..."}, + "prompt_hash": "...", + "usage": {"prompt_tokens": 10, "completion_tokens": 20, "total_tokens": 30}, + "start_time": "...", + "end_time": "...", + "duration_ms": 1523, + "validation_mode": "json_schema", + "validation_status": "passed", + "repair_attempts_used": 0 + }, + "raw_model_output": "{...}" +} +``` -## Add a New Prompt Profile +Validation content failures are returned as successful run responses (`200`) with `validation.status = "failed"`; raw model output is preserved in `raw_model_output`. -1. Add a YAML file under `profiles/` with: - - `id`, `version`, `expected_inputs`, `templates`, `model_defaults`, `output_format`, `validation` - - optional model timeout via `model_defaults.timeout_seconds` (per-run LLM timeout override) -2. Ensure endpoint/model are available from either: - - profile defaults (`model_defaults.endpoint`, `model_defaults.model`), or - - request overrides (`--llm-base-url`, `--model`, or HTTP `model.endpoint`/`model.model`). -3. Use template helpers such as `{{input "transcript"}}` and template vars like `{{.session_date}}`. -4. For structured JSON output, set: - - `output_format: json` - - `validation.validation_mode: json_schema` - - `validation.schema_path: ` -5. Place schema files in `schemas/` and pass `--schema-dir ./schemas` for CLI/serve. -6. `validation.repair_attempts` is bounded and applies only to structured modes (`json`, `json_schema`). +Error response shape: -## Validation Behavior +```json +{ + "error": { + "code": "artifact_read_failed", + "message": "failed to read input artifact" + } +} +``` -Validation modes currently implemented: -- `none` -- `basic` (non-empty output) -- `json` (must parse as JSON) -- `json_schema` (must parse JSON and satisfy schema) +## Prompt Profile Authoring -Important behavior: -- Validation content failures are returned as structured run results (`validation.status = failed`) and preserve `raw_model_output`. -- Validation runtime/configuration failures are treated as run errors. +### Minimal Markdown profile + +```yaml +id: generic.markdown_summary +version: "1.0.0" +expected_inputs: + - transcript +templates: + - role: system + content: "You are a concise assistant." + - role: user + content: | + Summarize: + {{input "transcript"}} +model_defaults: + endpoint: http://localhost:8000/v1 + model: gpt-4o-mini + temperature: 0.2 + max_tokens: 700 +output_format: markdown +validation: + validation_mode: basic +``` + +### Structured JSON profile with schema validation + +```yaml +id: generic.structured_events +version: "1.0.0" +expected_inputs: + - transcript +templates: + - role: system + content: "Return only JSON." + - role: user + content: | + Extract events from: + {{input "transcript"}} +model_defaults: + endpoint: http://localhost:8000/v1 + model: gpt-4o-mini +output_format: json +validation: + format: json + validation_mode: json_schema + schema_path: structured_events.schema.json + repair_attempts: 1 +``` + +`repair_attempts` is bounded. Repair is attempted only for structured validation modes. + +## Validation Modes + +Supported modes: + +- `none`: skipped validation result. +- `basic`: fails if output is empty/whitespace. +- `json`: output must parse as JSON. +- `json_schema`: output must parse as JSON and satisfy the configured schema. + +Validation failures caused by output content are represented in `validation` and do not discard raw model output. + +## Repository Examples + +- Profiles: `profiles/` +- Schemas: `schemas/` +- Fixtures: `examples/fixtures/` +- Local experimentation: `local-test/` + +## Development Notes + +- Core is generic and follows a ports-and-adapters style. +- Domain/usecase packages do not depend on HTTP/CLI/wire types. +- To add a new LLM adapter: implement `internal/llm.Client`. +- To add a new artifact reader: implement/extend `internal/artifact.Reader` routing. +- To add a new validation mode: extend `internal/validate` and keep run semantics stable. diff --git a/architecture.md b/architecture.md index 5c81868..b7e2e0c 100644 --- a/architecture.md +++ b/architecture.md @@ -1,752 +1,232 @@ # Scriptorium Architecture -## Purpose +## 1. Purpose and Non-Goals -Scriptorium is a general-purpose prompt-profile execution service. +Scriptorium is a prompt-profile execution engine. -Its job is to take one or more named input artifacts, render a configured prompt profile, execute that prompt against an LLM endpoint, optionally validate the output, and return a generated artifact with useful metadata. +It takes named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata. -The initial concrete use case is generating artifacts from cleaned Dungeons & Dragons session transcripts, such as session recaps, player analysis, structured event extraction, and glossary update suggestions. +Scriptorium is not an orchestrator. It should not own transcription, transcript merging, transcript polishing, notification, or cross-step workflow control. -However, Scriptorium must not be D&D-specific. D&D behavior belongs in prompt profiles, schemas, and caller-provided inputs. The Go application should remain a generic engine for prompt execution and output validation. +For the motivating D&D workflow: -## Intended Audience +- Narratio orchestrates. +- WhisperX transcribes. +- Seriatim merges transcripts. +- Audita polishes transcripts. +- Scriptorium generates output artifacts from prepared inputs. -This document is written for 5.3-Codex and future maintainers. +Core Go code must remain domain-generic. -When implementing this repository, prefer simple, idiomatic Go over elaborate framework code. The architecture should be modular, testable, and composable, but not over-engineered. +## 2. Current Architecture -The desired implementation style is: +Current high-level structure: -- Clear domain types. -- Small interfaces at architectural boundaries. -- Explicit dependencies. -- No hidden global state. -- No domain-specific D&D logic in core packages. -- Practical hexagonal / ports-and-adapters structure. -- Boring, inspectable behavior. +- `cmd/scriptorium`: binary entrypoint. +- `internal/domain`: core domain types. +- `internal/usecase`: `Runner` use case and repair loop orchestration. +- `internal/profile`: filesystem prompt profile repository and profile validation. +- `internal/artifact`: artifact reference readers (`inline`, `file`) and routing. +- `internal/prompt`: Go template-based prompt renderer. +- `internal/llm`: LLM client interface + OpenAI-compatible HTTP adapter. +- `internal/validate`: output validator implementation (`none/basic/json/json_schema`). +- `internal/adapter/cli`: CLI adapter. +- `internal/adapter/http`: HTTP adapter (`POST /v1/runs`). -## Core Concept +This is a practical ports-and-adapters implementation. -Scriptorium transforms: +## 3. Run Data Flow -- Prompt profile -- Named input artifacts -- Template variables -- Model target -- Optional output contract +`Runner.Run(ctx, RunRequest)` currently executes: -Into: +1. Validate minimum request requirements (`profile_id`). +2. Load prompt profile by ID/version. +3. Merge effective model target (profile defaults + request override). +4. Resolve effective output contract (profile + optional request override). +5. Resolve named artifact refs to loaded artifacts. +6. Render prompt messages from templates. +7. Hash rendered prompt for auditability. +8. Call LLM client with provider-neutral `GenerateRequest`. +9. Build output artifact from model content. +10. Validate output. +11. If structured validation failed and repair is enabled/bounded, run repair attempts and re-validate. +12. Return `RunResult` with artifact, validation, raw output, and metadata. -- Generated artifact -- Validation result -- Prompt/model/input metadata -- Raw model output -- Structured error details, if applicable +Validation content failure remains a successful run result with `validation.status=failed`. -Scriptorium should be thought of as a deterministic wrapper around a nondeterministic model call. +## 4. Package Responsibilities -The system should make the model call as auditable and reproducible as possible, even though LLM output itself may not be exactly reproducible. +- `domain` + - Owns core nouns and contracts. + - Must not import adapters/provider SDK types. -## Application Boundary +- `usecase` + - Owns execution sequence and cross-port orchestration for a single run. + - May coordinate validation and bounded repair. + - Must not contain HTTP/CLI/wire concerns. -Scriptorium is not an orchestrator. +- `profile` + - Owns prompt profile loading/parsing/validation. + - Handles YAML strict decoding and profile-level constraints. -The broader workflow may include audio transcription, transcript merging, transcript polishing, artifact persistence, and notifications. Those responsibilities belong to the external orchestrator, currently expected to be Narratio. +- `artifact` + - Owns artifact ref resolution and content loading. + - Produces normalized `Artifact` values with size/hash/content type. -Scriptorium should not know about WhisperX, Seriatim, Audita, or any other pipeline stage. +- `prompt` + - Owns template rendering and required-input enforcement. -Scriptorium only knows how to: +- `llm` + - Owns generation port and provider adapters. + - Current adapter: OpenAI-compatible chat completions over `net/http`. -1. Load a prompt profile. -2. Load or receive named input artifacts. -3. Render a prompt. -4. Call an LLM. -5. Validate the output, if configured. -6. Return an output artifact and metadata. +- `validate` + - Owns output validation semantics and JSON Schema integration. -## Initial Workflow Context +- `adapter/http`, `adapter/cli` + - Owns transport/wire/flag concerns only. + - Should stay thin and delegate business flow to `usecase.Runner`. -The initial D&D workflow is expected to look like this: +## 5. Domain Model (Current) -1. Narratio transcribes audio tracks using WhisperX. -2. Narratio normalizes speaker names and saves per-speaker transcripts. -3. Narratio calls Seriatim to merge transcripts. -4. Narratio saves the merged transcript. -5. Narratio calls Audita to polish the transcript. -6. Narratio saves the processed transcript. -7. Narratio calls Scriptorium one or more times to generate output artifacts. -8. Narratio saves each generated artifact. -9. Narratio optionally sends a completion notification. +Key types in `internal/domain`: -Scriptorium only owns step 7. +- `RunRequest`: profile selector, named input refs, vars, optional model override, optional validation override. +- `RunResult`: output artifact, validation result, raw output, profile/model metadata, hashes, usage, timestamps, duration. +- `ArtifactRef`: `{type, uri, body}` reference contract. +- `Artifact`: loaded payload (`name`, `content_type`, `body`, `uri`, `size`, `hash`). +- `PromptProfile`: YAML-backed profile definition. +- `RenderedPrompt` / `RenderedMessage`: provider-neutral prompt structure. +- `GenerateRequest` / `GenerateResponse`: provider-neutral model I/O. +- `ValidationResult`: passed/failed/skipped + mode/errors/schema/repair attempts. -Each Scriptorium request should initially produce one artifact. If multiple artifacts are needed, the orchestrator should call Scriptorium multiple times. +## 6. Interfaces and Adapters -Batch execution can be added later, but should not be part of the core v1 design unless there is an immediate need. +Primary ports: -## Primary Use Cases +- `profile.Repository` +- `artifact.Reader` +- `prompt.Renderer` +- `llm.Client` +- `validate.Validator` +- `usecase.OutputRepairer` (usecase-local abstraction) -Scriptorium should support the following v1 use cases: +Current adapters: -1. Generate a freeform Markdown artifact from a transcript and prompt profile. -2. Generate a structured JSON artifact from a transcript and prompt profile. -3. Validate JSON output against a JSON Schema. -4. Return raw model output when validation fails. -5. Optionally attempt one bounded repair pass for invalid structured output. -6. Record metadata about the profile, model, inputs, prompt hash, and validation result. -7. Support local development through a CLI. -8. Support service usage through an HTTP API. +- Profile repository: filesystem YAML loader. +- Artifact reader: composite reader for `inline` and `file`. +- Prompt renderer: Go templates with input helper + vars. +- LLM adapter: OpenAI-compatible `/chat/completions`. +- Validator: standard validator with `none/basic/json/json_schema`. +- CLI/HTTP adapters: thin request mapping and response mapping. -## Non-Goals for v1 - -Do not implement these in the initial version unless explicitly requested: - -- Multi-agent workflows. -- Arbitrary DAG execution. -- Long-running job queues. -- Automatic RAG. -- Automatic prompt chaining. -- Automatic chunking and summarization. -- Model selection logic. -- Complex retry policies beyond basic HTTP/model retry and optional validation repair. -- D&D-specific Go packages. -- UI. -- Database persistence. -- Full artifact lifecycle management. - -These may be valid future features, but v1 should remain a focused prompt-profile execution engine. - -## Architectural Style - -Use a practical hexagonal architecture. - -The core domain and use case packages should not depend on infrastructure details such as HTTP, S3, local filesystems, or specific LLM providers. - -External concerns should be implemented as adapters. - -The central use case should be easy to test with fake prompt repositories, fake artifact readers, fake LLM clients, and fake validators. - -Recommended high-level structure: - -- cmd/scriptorium: application entrypoint -- internal/domain: core domain types -- internal/usecase: application use cases -- internal/profile: prompt profile loading and parsing -- internal/prompt: prompt rendering -- internal/llm: LLM client interfaces and adapters -- internal/validate: output validation implementations -- internal/artifact: artifact loading and storage adapters -- internal/adapter/http: HTTP API -- internal/adapter/cli: CLI interface -- internal/config: application configuration -- profiles: example prompt profiles -- schemas: example output schemas -- testdata: fixtures for tests - -Exact package names may evolve, but the boundary principles should remain stable. - -## Domain Model - -The core domain should include these concepts. - -### RunRequest - -Represents one request to generate one artifact. - -Fields should include: - -- ProfileID -- Inputs -- Vars -- Optional model override -- Optional validation override, if needed -- Optional caller metadata - -Inputs should be keyed by logical input name, not by filename. - -Example logical input names: - -- transcript -- glossary -- previous_recap -- campaign_notes -- source_document - -### ArtifactRef - -Represents a reference to an input artifact. - -Artifact references should support at least inline content and local file paths in v1. - -S3 references may be supported in v1 if needed, but should be implemented behind an interface. - -Likely artifact reference types: - -- inline -- file -- s3 - -The core use case should not care which reference type is used. - -### Artifact - -Represents loaded content. - -Fields should include: - -- Name -- ContentType -- Body -- Optional URI or source reference -- Optional size -- Optional SHA-256 hash - -Artifacts are the actual input and output payloads after references have been resolved. - -### PromptProfile - -Represents a configured prompt execution profile. - -A profile should include: - -- ID -- Version -- Description -- Expected inputs -- Prompt templates -- Model defaults -- Output format -- Optional validation configuration -- Optional repair configuration - -Prompt profiles should be serializable from YAML. - -Prompt profiles are where domain-specific behavior belongs. - -### RenderedPrompt - -Represents the prompt after input artifacts and variables have been applied. - -For OpenAI-compatible chat models, this should contain a list of chat messages. - -At minimum, support system and user messages. - -Future support for developer messages, assistant prefill, or multimodal parts can be added later. - -### ModelTarget - -Represents the LLM endpoint and model configuration. - -Fields should include: - -- Endpoint name or URL -- Model name -- Temperature -- Max tokens -- Top-p, if supported -- Additional provider-specific options, if needed - -For v1, the main adapter should support OpenAI-compatible chat completion APIs. - -### RunResult - -Represents the complete result of a run. - -Fields should include: - -- Generated artifact -- Raw model output -- Validation result -- Profile ID and version -- Model name -- Endpoint name -- Input hashes -- Prompt hash -- Token usage, if available -- Start and end timestamps -- Error details, if applicable - -### ValidationResult - -Represents validation status. - -Fields should include: - -- Status: passed, failed, skipped -- Validation mode -- Error messages -- Schema path, if applicable -- Repair attempts used -- Final output validity - -## Core Interfaces - -Keep interfaces small and focused. - -### PromptRepository - -Responsible for loading prompt profiles. - -Expected implementations: - -- FilesystemPromptRepository -- EmbeddedPromptRepository, optional later -- S3PromptRepository, optional later - -The use case should ask for a profile by ID or ID plus version. - -### ArtifactReader - -Responsible for resolving ArtifactRef values into Artifact values. - -Expected implementations: - -- InlineArtifactReader -- FileArtifactReader -- S3ArtifactReader -- CompositeArtifactReader - -The CompositeArtifactReader can route by reference type. - -### PromptRenderer - -Responsible for rendering prompt templates using named artifacts and variables. - -Use Go templates unless there is a strong reason to choose something else. - -Renderer responsibilities: - -- Verify required inputs exist. -- Expose safe template functions. -- Insert artifact content by logical name. -- Render system and user prompt sections. -- Return a RenderedPrompt. - -Do not silently omit missing required inputs. - -Do not silently truncate large inputs in v1. - -### LLMClient - -Responsible for executing a rendered prompt against a model endpoint. - -The initial implementation should support OpenAI-compatible chat completions. - -This should work with: - -- vLLM -- LiteLLM -- OpenAI-compatible local endpoints -- OpenAI-compatible hosted endpoints, if configured - -The domain should not depend on provider-specific SDK types. - -### OutputValidator - -Responsible for validating the generated artifact. - -Expected validation modes: - -- none -- basic -- json_schema - -Basic validation may check things like non-empty output, required headings, or forbidden boilerplate. - -JSON Schema validation should parse the output as JSON and validate it against the configured schema. - -### OutputRepairer - -Responsible for making a bounded attempt to repair invalid structured output. - -This should be optional. - -The repairer may use the same LLMClient with a repair prompt. - -Repair attempts must be bounded by configuration. Default should be zero or one. - -Do not implement unbounded repair loops. - -## Prompt Profiles - -Prompt profiles are the main extension mechanism. - -The Go application should stay generic. Prompt profiles should define domain behavior. - -A profile should be able to specify: - -- ID -- Version -- Description -- Required and optional inputs -- System prompt template -- User prompt template -- Default model configuration -- Output format -- Validation mode -- Schema path, if applicable -- Repair attempts, if applicable - -Profiles should live outside compiled Go code. - -Example profile categories for the initial D&D use case: - -- dnd.session_recap -- dnd.meta_analysis -- dnd.table_read -- dnd.structured_events -- dnd.glossary_update_suggestions - -The code should not special-case these names. - -## Template Rendering - -Prompt rendering must be predictable and explicit. - -Templates should be able to reference: - -- Named input artifacts -- Template variables -- Profile metadata - -The renderer should provide a helper equivalent to input(name), which inserts the content of a named artifact. - -The renderer should fail when: - -- A required input is missing. -- A template references an unknown input. -- A template references a missing required variable. -- The rendered prompt exceeds a configured token or size limit, if such a limit is configured. - -In v1, do not silently truncate inputs. - -If token counting is not implemented initially, use byte-size limits or leave token budgeting as a clearly marked future improvement. - -## Output Formats - -Scriptorium should support at least these output formats: - -- markdown -- text -- json - -For markdown and text, validation may be skipped or basic. - -For JSON, validation should at minimum require valid JSON. If a schema is configured, validate against the schema. - -The output artifact should preserve content type. - -Suggested content types: - -- text/markdown -- text/plain -- application/json - -## Validation - -Validation should be explicit and profile-driven. +## 7. Validation and Repair Model Validation modes: -- none: no validation beyond successful generation -- basic: simple textual validation -- json: parse as JSON -- json_schema: parse as JSON and validate against schema +- `none` +- `basic` +- `json` +- `json_schema` -For invalid structured output, Scriptorium should return: +Repair behavior: -- Validation status -- Validation errors -- Raw model output -- Repair attempts used -- Final output, if repair succeeded +- Only applies to structured modes (`json`, `json_schema`). +- Attempted only when validation fails, repairer exists, and `repair_attempts > 0`. +- Bounded strictly by `repair_attempts`. +- Uses a narrow JSON-repair prompt and re-validates each attempt. +- If still invalid, run succeeds with failed validation and preserved final raw output. +- Validator runtime/config errors are run errors. -Validation failure should not discard the raw output. +## 8. Public Contracts -## Repair +### CLI -Repair is only for structured output. +Commands: -The initial repair use case is invalid JSON or JSON that fails schema validation. +- `scriptorium run` +- `scriptorium serve` -The repair prompt should be deterministic and narrow: +`run`: -- Explain that the previous output failed validation. -- Provide validation errors. -- Provide the previous output. -- Ask the model to return only corrected JSON. -- Do not ask the model to improve the answer substantively. +- Required: `--profile-dir`, `--profile-id`, `--input`. +- Optional: model/endpoint overrides (`--model`, `--llm-base-url`), vars, output path, schema dir, timeout. +- Artifact bytes go to stdout (or `--out` file); summaries/errors go to stderr. -Repair must be bounded. +`serve`: -Recommended default: +- Required: `--profile-dir`, `--llm-base-url`. +- Exposes HTTP run endpoint. -- repair_attempts: 0 for freeform output -- repair_attempts: 1 for JSON schema output, if configured +### HTTP -## LLM Adapter +- Endpoint: `POST /v1/runs`. +- Request maps to `RunRequest` (`profile_id`, `inputs`, `vars`, optional `model` override). +- Response includes `artifact`, `validation`, `metadata`, `raw_model_output`. +- Validation content failures are represented as `200` with `validation.status=failed`. +- Error responses are `{error:{code,message}}` with stable code mapping. -The initial LLM adapter should target OpenAI-compatible chat completions. +### Prompt Profile YAML -The adapter should support: +- `id`, `version`, `expected_inputs`, `templates`, `model_defaults`, `output_format`, `validation`. +- Strict YAML decoding (`KnownFields`) rejects unknown fields. +- `validation.schema_path` required when `validation_mode=json_schema`. +- `validation.repair_attempts` must be non-negative. -- Base URL -- API key, optional for local endpoints -- Model name -- Temperature -- Max tokens -- Basic generation parameters -- Request timeout -- Token usage extraction, if returned by the endpoint +### Metadata -Do not couple the core domain to OpenAI SDK request or response structs. +Current run metadata includes: -The adapter should translate between internal GenerateRequest / GenerateResponse types and the provider wire format. +- `run_id` (UUID v4) +- `profile_id`, `profile_version`, `profile_hash` +- `model_name`, `endpoint`, effective `model_params` +- `input_hashes`, `prompt_hash` +- token usage +- start/end timestamps +- duration +- validation mode/status +- repair attempts used -## HTTP API +## 9. Extension Points (Future Work) -The HTTP API should be thin. +Future features should plug into existing boundaries, not bypass them. -It should translate HTTP requests into RunRequest values, call the use case, and translate RunResult values into HTTP responses. +Candidate extensions: -Suggested initial endpoint: +- S3 artifact refs via `artifact.Reader` extension. +- Token budgeting in usecase/model-target policy layer. +- Streaming LLM output via additional `llm.Client` methods/adapters. +- Batch execution as a separate use case (not hidden in single-run path). +- Additional LLM providers implementing `llm.Client`. +- Additional validators/modes in `validate`. +- Additional profile repositories (embedded, remote, object storage). -- POST /v1/runs +These are future work, not part of current default behavior. -The request should include: +## 10. Architectural Guardrails -- profile_id -- inputs -- vars -- optional model override -- optional caller metadata +Contributors should preserve these constraints: -The response should include: +- No D&D-specific behavior in core Go packages. +- No orchestration creep into Scriptorium. +- No unbounded repair loops. +- No silent truncation/omission of rendered inputs or outputs. +- Do not log full artifacts/prompts by default. +- Keep provider-specific wire/SDK details out of domain types. +- Keep adapter boundaries explicit and thin. -- artifact -- validation -- metadata -- raw_model_output -- error details, if applicable +## 11. Testing Strategy -The HTTP layer should not contain business logic. +Protect behavior at boundaries and in usecase flow: -## CLI +- Profile loading/parsing/validation errors. +- Artifact reading for inline/file + hash/content type behavior. +- Prompt rendering required inputs/template error behavior. +- LLM adapter request/response/error/timeout behavior. +- Runner success path and metadata population. +- Validation failure raw-output preservation. +- Successful/failed/bounded repair flows. +- HTTP request mapping, response shape, and error mapping. +- CLI parsing helpers, required flags, and output stream separation. -The CLI should be thin and call the same core use case as HTTP. - -Current command surface: - -- scriptorium run -- scriptorium serve - -The `run` command should accept: - -- profile ID -- input mappings -- variable mappings -- output path, optional -- profile directory -- optional model/endpoint overrides - -If model/endpoint overrides are omitted, profile model defaults should be used. - -## Configuration - -Application configuration should include: - -- Prompt profile directory -- Schema directory -- LLM endpoints -- Default endpoint -- Timeout settings -- Optional artifact store settings -- Logging settings - -Configuration should be file-based with environment variable overrides where appropriate. - -Avoid hardcoding local paths. - -Avoid hardcoding D&D-specific defaults. - -## Artifact Storage - -Scriptorium does not need to own artifact persistence in v1. - -The default behavior should be: - -- Read input artifacts. -- Return generated artifact to caller. - -Narratio or another orchestrator can save the result to S3. - -However, Scriptorium should be designed so that artifact readers and writers can be added later. - -If an ArtifactWriter is added, it should be optional and should not change the core use case. - -## Error Handling - -Errors should be explicit and typed where useful. - -Important error categories: - -- Profile not found -- Invalid profile -- Required input missing -- Artifact read failure -- Template render failure -- LLM request failure -- LLM response parse failure -- Output validation failure -- Repair failure - -Validation failure is not necessarily the same as application failure. - -If the model returns output but the output fails validation, Scriptorium should return a structured RunResult with failed validation status when possible. - -Transport-level errors, missing inputs, invalid profiles, and failed model calls should be returned as application errors. - -## Observability - -Use structured logging. - -Log important lifecycle events: - -- Run started -- Profile loaded -- Inputs loaded -- Prompt rendered -- LLM request started -- LLM response received -- Validation completed -- Repair attempted -- Run completed - -Do not log full prompt content or full artifact content by default. - -Do log hashes, sizes, profile IDs, model names, durations, and validation status. - -## Metadata and Reproducibility - -Every successful or partially successful run should include metadata. - -Recommended metadata: - -- Run ID -- Profile ID -- Profile version -- Profile hash -- Prompt hash -- Input artifact hashes -- Model endpoint -- Model name -- Generation parameters -- Created timestamp -- Duration -- Token usage, if available -- Validation mode -- Validation status -- Repair attempts used - -This metadata is important for auditing and regeneration. - -## Security and Safety Considerations - -Scriptorium will often handle private transcripts or documents. - -Default behavior should avoid accidental disclosure. - -Guidelines: - -- Do not log full inputs by default. -- Do not log full model outputs by default unless debug logging is explicitly enabled. -- Keep API keys in configuration or environment variables, not in prompt profiles. -- Avoid exposing local filesystem paths in public error messages when running as a service. -- Treat prompt profiles as trusted configuration. -- Treat input artifacts as untrusted content. -- Avoid shell execution entirely. - -## Testing Strategy - -Tests should focus on the use case and boundaries. - -Recommended tests: - -- Load valid prompt profile. -- Reject invalid prompt profile. -- Reject missing required input. -- Render prompt with named inputs. -- Render prompt with variables. -- Execute run with fake LLM client. -- Validate successful Markdown output. -- Validate successful JSON output. -- Detect invalid JSON output. -- Detect JSON Schema validation errors. -- Perform successful repair with fake LLM client. -- Preserve raw output on validation failure. -- Return useful metadata. -- HTTP handler maps request to use case correctly. -- CLI command maps flags to use case correctly. - -Use fixtures in testdata. - -The core use case should be testable without network access. - -## Development Priorities - -Implementation should proceed in this order: - -1. Define domain types. -2. Define core interfaces. -3. Implement prompt profile loading from YAML. -4. Implement artifact loading for inline and local file inputs. -5. Implement prompt rendering. -6. Implement fake LLM client tests. -7. Implement OpenAI-compatible LLM client. -8. Implement basic validation. -9. Implement JSON validation. -10. Implement JSON Schema validation. -11. Implement optional repair. -12. Implement CLI. -13. Implement HTTP API. -14. Add example D&D profiles and schemas. -15. Add integration-style tests using fake adapters. - -Do not start with HTTP or CLI. Start with the core use case. - -## Design Principles - -Prefer boring code. - -Prefer explicit configuration. - -Prefer small packages with clear responsibilities. - -Prefer interfaces only at real boundaries. - -Do not create abstractions before they are needed. - -Do not let prompt profile complexity leak into Go code. - -Do not let D&D assumptions leak into the core engine. - -Do not silently truncate inputs. - -Do not discard invalid model output. - -Do not hide validation errors. - -Do not implement an orchestrator inside Scriptorium. - -## Summary - -Scriptorium is a reusable prompt-profile execution engine. - -It should provide this core transformation: - -Named artifacts plus prompt profile plus model target produces generated artifact plus validation plus metadata. - -The D&D transcript analysis workflow is the first use case, not the architecture itself. - -The correct implementation is a small, modular Go service with a clean core use case and replaceable adapters for profiles, artifacts, prompt rendering, LLM calls, validation, CLI, and HTTP. +Prefer focused unit tests and small integration-style tests with fake LLMs.