|
|
|
|
@@ -1,483 +1,14 @@
|
|
|
|
|
# Scriptorium Architecture
|
|
|
|
|
# Architecture (Moved)
|
|
|
|
|
|
|
|
|
|
## 1. Purpose and Non-Goals
|
|
|
|
|
The canonical architecture policy is:
|
|
|
|
|
|
|
|
|
|
Scriptorium is a prompt-definition execution engine.
|
|
|
|
|
- `docs/policy/architecture.md`
|
|
|
|
|
|
|
|
|
|
It accepts named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata.
|
|
|
|
|
Implemented internal component documentation is:
|
|
|
|
|
|
|
|
|
|
Scriptorium also supports rendering/preparing a prompt without calling an LLM. This allows users to inspect the fully rendered prompt messages and effective runtime settings before executing a run.
|
|
|
|
|
- `docs/internal/runner.md`
|
|
|
|
|
- `docs/internal/adapters.md`
|
|
|
|
|
|
|
|
|
|
Scriptorium is not an orchestrator. It must not own transcription, transcript merge/polish steps, notifications, or cross-step workflow control.
|
|
|
|
|
Roadmap-only planning content is:
|
|
|
|
|
|
|
|
|
|
For the motivating D&D workflow:
|
|
|
|
|
- Narratio orchestrates.
|
|
|
|
|
- WhisperX transcribes.
|
|
|
|
|
- Seriatim merges transcripts.
|
|
|
|
|
- Audita polishes transcripts.
|
|
|
|
|
- Scriptorium generates final artifacts from prepared inputs.
|
|
|
|
|
|
|
|
|
|
Core Go code remains generic.
|
|
|
|
|
|
|
|
|
|
## 2. Current Architecture
|
|
|
|
|
|
|
|
|
|
Scriptorium uses a ports-and-adapters architecture to decouple the core execution logic from external dependencies.
|
|
|
|
|
|
|
|
|
|
### Package Responsibilities
|
|
|
|
|
|
|
|
|
|
- `cmd/scriptorium`: Binary entrypoint for CLI and HTTP server.
|
|
|
|
|
- `internal/domain`: Core domain contracts, including `PromptDefinition`, `ExecutionProfile`, `PreparedRun`, `RunResult`, and related metadata.
|
|
|
|
|
- `internal/usecase`: `Runner` use case logic, including prompt preparation, profile selection, runtime override resolution, full run execution, validation, and bounded repair.
|
|
|
|
|
- `internal/config`: Application-level config model/loader for adapter settings (for example prompt/profile/schema directories, server address, and render format default).
|
|
|
|
|
- `internal/promptdef`: Repository for loading and validating Prompt Definitions from the filesystem.
|
|
|
|
|
- `internal/profile`: Repository for loading Execution Profiles from the filesystem.
|
|
|
|
|
- `internal/artifact`: Input artifact resolution (`inline`, `file`).
|
|
|
|
|
- `internal/prompt`: Template rendering via Go templates.
|
|
|
|
|
- `internal/llm`: Provider-neutral client interface and OpenAI-compatible HTTP adapter.
|
|
|
|
|
- `internal/validate`: Output validation implementation (`none/basic/json/json_schema`).
|
|
|
|
|
- `internal/adapter/cli`: CLI flag parsing, command dispatch, and output handling.
|
|
|
|
|
- `internal/adapter/http`: HTTP request/response mapping.
|
|
|
|
|
- `internal/format` or equivalent: Formatting of prepared/rendered prompt output for CLI or other adapters, if formatting grows beyond simple CLI-local helpers.
|
|
|
|
|
|
|
|
|
|
Exact package names may evolve, but the architectural boundaries should remain stable.
|
|
|
|
|
|
|
|
|
|
### Application Configuration
|
|
|
|
|
|
|
|
|
|
`config.yml` is adapter/application setup, not domain logic.
|
|
|
|
|
|
|
|
|
|
Application config is intended for application-level settings such as:
|
|
|
|
|
- `prompt_dir`
|
|
|
|
|
- `profile_dir`
|
|
|
|
|
- `schema_dir`
|
|
|
|
|
- `server.addr`
|
|
|
|
|
- `defaults.render_format`
|
|
|
|
|
|
|
|
|
|
Application config precedence is:
|
|
|
|
|
1. CLI flags
|
|
|
|
|
2. `config.yml`
|
|
|
|
|
3. Built-in application defaults
|
|
|
|
|
|
|
|
|
|
Runtime model settings are intentionally separate:
|
|
|
|
|
- Execution profiles and runtime overrides continue to own endpoint/model/runtime behavior.
|
|
|
|
|
- `config.yml` does not replace execution profiles.
|
|
|
|
|
|
|
|
|
|
The core use case (`Runner.Prepare`/`Runner.Run`) does not need to know whether adapter-level settings came from CLI flags or `config.yml`; it receives resolved dependencies and requests from adapters.
|
|
|
|
|
|
|
|
|
|
## 3. Core Execution Model
|
|
|
|
|
|
|
|
|
|
Scriptorium has two closely related execution paths:
|
|
|
|
|
|
|
|
|
|
1. Prepare/render path.
|
|
|
|
|
2. Full run path.
|
|
|
|
|
|
|
|
|
|
The full run path should reuse the prepare path rather than duplicating its logic.
|
|
|
|
|
|
|
|
|
|
### 3.1 Prepare / Render Data Flow
|
|
|
|
|
|
|
|
|
|
The prepare path should be represented in the use case layer, preferably as `Runner.Prepare(ctx, RunRequest)` or an equivalent method.
|
|
|
|
|
|
|
|
|
|
It executes all pre-LLM work:
|
|
|
|
|
|
|
|
|
|
1. **Validate Request**: Ensure the request includes a prompt ID and all minimum required fields.
|
|
|
|
|
2. **Load Prompt Definition**: Retrieve the `PromptDefinition` by ID from the prompt repository.
|
|
|
|
|
3. **Select Profile**: Determine the `profile_id` using this precedence:
|
|
|
|
|
- Explicit `profile_id` in `RunRequest`.
|
|
|
|
|
- `default_profile` specified in the `PromptDefinition`.
|
|
|
|
|
- Error if neither is available.
|
|
|
|
|
4. **Load Execution Profile**: Retrieve the `ExecutionProfile` from the profile repository.
|
|
|
|
|
5. **Resolve Runtime Overrides**: Merge settings based on precedence, highest to lowest:
|
|
|
|
|
- Runtime overrides from CLI flags or HTTP request `model` object.
|
|
|
|
|
- Execution Profile settings.
|
|
|
|
|
- Built-in application defaults.
|
|
|
|
|
6. **Resolve Artifacts**: Load all named input artifacts defined in the request.
|
|
|
|
|
7. **Render Prompt**: Apply template variables and input artifacts to the prompt templates.
|
|
|
|
|
8. **Compute Metadata**: Compute hashes, selected profile ID, prompt ID/version, effective runtime settings, input hashes, rendered prompt hash, and timing information as appropriate.
|
|
|
|
|
9. **Return PreparedRun**: Return a `PreparedRun` containing the rendered messages, effective runtime settings, resolved metadata, and input/prompt hashes.
|
|
|
|
|
|
|
|
|
|
The prepare path must not call the LLM.
|
|
|
|
|
|
|
|
|
|
The prepare path must not validate model output, because there is no model output.
|
|
|
|
|
|
|
|
|
|
The prepare path must not perform structured-output repair, because repair only applies after model output exists.
|
|
|
|
|
|
|
|
|
|
The prepare path should not resolve or expose raw API key values. It may include the selected `api_key_env` name in effective runtime settings or metadata, but never the environment variable value.
|
|
|
|
|
|
|
|
|
|
### 3.2 Full Run Data Flow
|
|
|
|
|
|
|
|
|
|
The `Runner.Run(ctx, RunRequest)` flow should reuse the prepare path:
|
|
|
|
|
|
|
|
|
|
1. **Prepare**: Call the shared prepare flow to load the prompt, select the profile, resolve artifacts, render prompt messages, and compute pre-run metadata.
|
|
|
|
|
2. **Call LLM**: Execute the generation request using the effective runtime settings from the prepared run.
|
|
|
|
|
3. **Build Output Artifact**: Convert the model response into the configured output artifact.
|
|
|
|
|
4. **Validate Output**:
|
|
|
|
|
- Validate the model output against the prompt definition's output contract.
|
|
|
|
|
- Validation content failures remain successful run results with `validation.status=failed`.
|
|
|
|
|
- Validator runtime/config errors are run errors.
|
|
|
|
|
5. **Repair If Configured**:
|
|
|
|
|
- If structured validation fails and `repair_attempts > 0`, perform bounded repair attempts.
|
|
|
|
|
- Re-validate after each repair attempt.
|
|
|
|
|
- Repair loops must remain strictly bounded.
|
|
|
|
|
6. **Return RunResult**: Produce a `RunResult` containing the final artifact, validation status, raw model output, usage information, and metadata.
|
|
|
|
|
|
|
|
|
|
`Runner.Run` should not duplicate profile selection, artifact resolution, or prompt rendering logic that already exists in `Runner.Prepare`.
|
|
|
|
|
|
|
|
|
|
## 4. Domain Model
|
|
|
|
|
|
|
|
|
|
Key domain types:
|
|
|
|
|
|
|
|
|
|
- `PromptDefinition`: Defines the "what" of the task: templates, inputs, output contract, validation settings, repair settings, and optional `default_profile`.
|
|
|
|
|
- `ExecutionProfile`: Defines the "how" of execution: endpoint, model, generation parameters, timeout, reasoning effort, and `api_key_env`.
|
|
|
|
|
- `RunRequest`: The intent to execute or prepare a prompt, including `prompt_id`, optional `profile_id`, inputs, variables, and optional runtime overrides.
|
|
|
|
|
- `PreparedRun`: The result of the prepare/render phase. Contains rendered messages, effective runtime settings, selected profile ID, prompt metadata, input hashes, prompt hash, and other pre-LLM metadata.
|
|
|
|
|
- `RunResult`: The result of a full run. Contains the generated `Artifact`, `ValidationResult`, raw model output, token usage, and `RunMetadata`.
|
|
|
|
|
- `RunMetadata`: Detailed tracing information, including prompt ID/version, selected profile ID, effective model parameters, usage tokens, hashes, timestamps, validation status, and repair attempts where applicable.
|
|
|
|
|
- `RenderedPrompt`: Provider-neutral rendered prompt structure.
|
|
|
|
|
- `RenderedMessage`: Provider-neutral rendered message with role and content.
|
|
|
|
|
- `ArtifactRef`: A reference to an input artifact, such as `file` or `inline`.
|
|
|
|
|
- `Artifact`: Loaded artifact content with name, content type, body, source URI, size, and hash.
|
|
|
|
|
- `ValidationResult`: Validation status and details for full runs.
|
|
|
|
|
|
|
|
|
|
`PreparedRun` should be serializable for JSON output and should also be representable in a human-readable text format.
|
|
|
|
|
|
|
|
|
|
## 5. Interfaces and Adapters
|
|
|
|
|
|
|
|
|
|
### Primary Ports
|
|
|
|
|
|
|
|
|
|
- `promptdef.Repository`: Lookup for prompt definitions.
|
|
|
|
|
- `profile.Repository`: Lookup for execution profiles.
|
|
|
|
|
- `artifact.Reader`: Loading of artifact content.
|
|
|
|
|
- `prompt.Renderer`: Template rendering.
|
|
|
|
|
- `llm.Client`: Model generation.
|
|
|
|
|
- `validate.Validator`: Output validation.
|
|
|
|
|
- `format.PreparedRunFormatter` or equivalent: Optional formatting abstraction for rendered/prepared output.
|
|
|
|
|
|
|
|
|
|
### Current Adapters
|
|
|
|
|
|
|
|
|
|
- **Repositories**: Filesystem YAML loaders for both prompts and profiles.
|
|
|
|
|
- **Artifact Reader**: Composite reader supporting `file` and `inline`.
|
|
|
|
|
- **Prompt Renderer**: Go templates with a custom `input` helper.
|
|
|
|
|
- **LLM Client**: OpenAI-compatible `/chat/completions` over HTTP.
|
|
|
|
|
- **Validator**: Standard validator supporting `none`, `basic`, `json`, and `json_schema`.
|
|
|
|
|
- **CLI Adapter**: Supports `run`, `render`, and `serve`.
|
|
|
|
|
- **HTTP Adapter**: Supports full run execution through `POST /v1/runs`.
|
|
|
|
|
|
|
|
|
|
## 6. Public Contracts
|
|
|
|
|
|
|
|
|
|
### CLI
|
|
|
|
|
|
|
|
|
|
Scriptorium should expose at least these commands:
|
|
|
|
|
|
|
|
|
|
- `scriptorium run`: Executes a prompt by preparing it, calling the LLM, validating output, optionally repairing structured output, and returning an artifact.
|
|
|
|
|
- `scriptorium render`: Prepares and renders a prompt without calling the LLM.
|
|
|
|
|
- `scriptorium serve`: Starts the HTTP API using infrastructure-only flags.
|
|
|
|
|
|
|
|
|
|
### `scriptorium run`
|
|
|
|
|
|
|
|
|
|
`run` uses flags such as:
|
|
|
|
|
|
|
|
|
|
- `--prompt-dir`
|
|
|
|
|
- `--profile-dir`
|
|
|
|
|
- `--prompt`
|
|
|
|
|
- `--profile`
|
|
|
|
|
- `--input`
|
|
|
|
|
- `--var`
|
|
|
|
|
- `--out`
|
|
|
|
|
- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.
|
|
|
|
|
|
|
|
|
|
`run` should produce the generated artifact as its primary output.
|
|
|
|
|
|
|
|
|
|
### `scriptorium render`
|
|
|
|
|
|
|
|
|
|
`render` prepares and renders a prompt without calling an LLM.
|
|
|
|
|
|
|
|
|
|
It should use the same prompt/profile/input/variable/runtime override flags as `run` where applicable:
|
|
|
|
|
|
|
|
|
|
- `--prompt-dir`
|
|
|
|
|
- `--profile-dir`
|
|
|
|
|
- `--prompt`
|
|
|
|
|
- `--profile`
|
|
|
|
|
- `--input`
|
|
|
|
|
- `--var`
|
|
|
|
|
- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.
|
|
|
|
|
- `--format`, with initial support for `text` and `json`.
|
|
|
|
|
|
|
|
|
|
Default render output format should be `text`.
|
|
|
|
|
|
|
|
|
|
`render` must not call the LLM.
|
|
|
|
|
|
|
|
|
|
`render` should show the same rendered messages and effective runtime settings that `run` would use.
|
|
|
|
|
|
|
|
|
|
`render` should include enough information to debug:
|
|
|
|
|
|
|
|
|
|
- prompt ID
|
|
|
|
|
- prompt version
|
|
|
|
|
- selected profile ID
|
|
|
|
|
- effective runtime settings
|
|
|
|
|
- input hashes
|
|
|
|
|
- prompt hash
|
|
|
|
|
- rendered messages
|
|
|
|
|
|
|
|
|
|
`render` must not include resolved API key values.
|
|
|
|
|
|
|
|
|
|
`render` may include the `api_key_env` name.
|
|
|
|
|
|
|
|
|
|
### Render Output Formats
|
|
|
|
|
|
|
|
|
|
Initial render output formats:
|
|
|
|
|
|
|
|
|
|
- `text`: Human-readable default format.
|
|
|
|
|
- `json`: Machine-readable structured representation of the prepared run.
|
|
|
|
|
|
|
|
|
|
Additional formats, such as `markdown`, may be added later.
|
|
|
|
|
|
|
|
|
|
Render output formatting should be modular. Adding a new output format should not require changing the prepare/run core logic.
|
|
|
|
|
|
|
|
|
|
The output formatting layer should consume a `PreparedRun` and produce bytes or text for the adapter. It should not reload prompts, re-resolve artifacts, re-render templates, call the LLM, or perform validation.
|
|
|
|
|
|
|
|
|
|
### `scriptorium serve`
|
|
|
|
|
|
|
|
|
|
`serve` starts the HTTP API.
|
|
|
|
|
|
|
|
|
|
It should use infrastructure-only flags such as:
|
|
|
|
|
|
|
|
|
|
- `--addr`
|
|
|
|
|
- `--prompt-dir`
|
|
|
|
|
- `--profile-dir`
|
|
|
|
|
- `--schema-dir`
|
|
|
|
|
|
|
|
|
|
`serve` should not introduce a server-level model/runtime precedence layer unless explicitly documented and intentionally implemented.
|
|
|
|
|
|
|
|
|
|
### HTTP API
|
|
|
|
|
|
|
|
|
|
Current HTTP API:
|
|
|
|
|
|
|
|
|
|
- `POST /v1/runs`: Accepts `RunRequest` JSON and returns `RunResponse` JSON. No built-in auth.
|
|
|
|
|
|
|
|
|
|
Request may include runtime overrides under `model` and an `include_raw_output` boolean.
|
|
|
|
|
|
|
|
|
|
`raw_model_output` is exposed only when explicitly requested with `include_raw_output=true`.
|
|
|
|
|
|
|
|
|
|
A future HTTP prepare/render endpoint may be added, such as `POST /v1/renders` or `POST /v1/runs/prepare`, but the initial render feature may be CLI-only. If added later, it should call the same usecase-level prepare path as `scriptorium render`.
|
|
|
|
|
|
|
|
|
|
### YAML Shapes
|
|
|
|
|
|
|
|
|
|
Prompt YAML includes:
|
|
|
|
|
|
|
|
|
|
- `id`
|
|
|
|
|
- `version`
|
|
|
|
|
- optional `default_profile`
|
|
|
|
|
- `inputs`
|
|
|
|
|
- `messages`
|
|
|
|
|
- `output`
|
|
|
|
|
|
|
|
|
|
Inputs support:
|
|
|
|
|
|
|
|
|
|
- `name`
|
|
|
|
|
- `required`
|
|
|
|
|
- optional `content_type`
|
|
|
|
|
- `description`
|
|
|
|
|
|
|
|
|
|
Messages require:
|
|
|
|
|
|
|
|
|
|
- `role`
|
|
|
|
|
- exactly one of `content` or `content_file`
|
|
|
|
|
|
|
|
|
|
`content_file` resolves relative to the prompt YAML location.
|
|
|
|
|
|
|
|
|
|
Profile YAML includes:
|
|
|
|
|
|
|
|
|
|
- `id`
|
|
|
|
|
- `endpoint`
|
|
|
|
|
- `model`
|
|
|
|
|
- generation parameters
|
|
|
|
|
- timeout settings
|
|
|
|
|
- reasoning settings
|
|
|
|
|
- `api_key_env`
|
|
|
|
|
|
|
|
|
|
Prompt content must not appear in profile YAML.
|
|
|
|
|
|
|
|
|
|
Model/runtime/API-key settings must not appear in prompt YAML, except that prompt YAML may specify `default_profile`.
|
|
|
|
|
|
|
|
|
|
## 7. Render Feature Design
|
|
|
|
|
|
|
|
|
|
The render feature is a first-class use case, not a CLI-only shortcut.
|
|
|
|
|
|
|
|
|
|
### Goals
|
|
|
|
|
|
|
|
|
|
The render feature should help users:
|
|
|
|
|
|
|
|
|
|
- inspect fully rendered prompt messages
|
|
|
|
|
- debug missing inputs
|
|
|
|
|
- verify template variable substitution
|
|
|
|
|
- verify selected profile resolution
|
|
|
|
|
- verify runtime override precedence
|
|
|
|
|
- verify file-backed prompt loading
|
|
|
|
|
- inspect input hashes and prompt hashes
|
|
|
|
|
- prepare for future token budgeting and prompt-size inspection
|
|
|
|
|
|
|
|
|
|
### Non-Goals
|
|
|
|
|
|
|
|
|
|
The render feature should not:
|
|
|
|
|
|
|
|
|
|
- call an LLM
|
|
|
|
|
- validate model output
|
|
|
|
|
- repair structured output
|
|
|
|
|
- resolve or print API key values
|
|
|
|
|
- mutate artifacts
|
|
|
|
|
- save outputs to artifact storage unless a future explicit output option is added
|
|
|
|
|
- become an orchestration step manager
|
|
|
|
|
|
|
|
|
|
### Usecase Shape
|
|
|
|
|
|
|
|
|
|
The preferred usecase shape is:
|
|
|
|
|
|
|
|
|
|
- `Runner.Prepare(ctx, RunRequest) (*PreparedRun, error)`
|
|
|
|
|
- `Runner.Run(ctx, RunRequest) (*RunResult, error)`
|
|
|
|
|
|
|
|
|
|
`Runner.Run` should call `Runner.Prepare`.
|
|
|
|
|
|
|
|
|
|
The prepare flow should be the only implementation of:
|
|
|
|
|
|
|
|
|
|
- prompt loading
|
|
|
|
|
- profile selection
|
|
|
|
|
- runtime override resolution
|
|
|
|
|
- artifact resolution
|
|
|
|
|
- prompt rendering
|
|
|
|
|
- pre-run metadata/hash calculation
|
|
|
|
|
|
|
|
|
|
### CLI Shape
|
|
|
|
|
|
|
|
|
|
The preferred command name is `render`.
|
|
|
|
|
|
|
|
|
|
The command should support:
|
|
|
|
|
|
|
|
|
|
- `--format text`
|
|
|
|
|
- `--format json`
|
|
|
|
|
|
|
|
|
|
Default format:
|
|
|
|
|
|
|
|
|
|
- `text`
|
|
|
|
|
|
|
|
|
|
Unknown formats should produce a clear error.
|
|
|
|
|
|
|
|
|
|
Formatting should be centralized through a small formatter registry, strategy, switch, or interface so new formats can be added without modifying usecase logic.
|
|
|
|
|
|
|
|
|
|
### Text Output Expectations
|
|
|
|
|
|
|
|
|
|
Text output should be optimized for human inspection.
|
|
|
|
|
|
|
|
|
|
It should include, at minimum:
|
|
|
|
|
|
|
|
|
|
- prompt ID and version
|
|
|
|
|
- selected profile ID
|
|
|
|
|
- model name
|
|
|
|
|
- endpoint
|
|
|
|
|
- effective generation settings
|
|
|
|
|
- input hashes
|
|
|
|
|
- rendered prompt hash
|
|
|
|
|
- rendered messages grouped by role
|
|
|
|
|
|
|
|
|
|
Text output should be readable and deterministic enough for tests.
|
|
|
|
|
|
|
|
|
|
It should not include raw API key values.
|
|
|
|
|
|
|
|
|
|
### JSON Output Expectations
|
|
|
|
|
|
|
|
|
|
JSON output should be a structured representation of `PreparedRun` or a DTO derived from it.
|
|
|
|
|
|
|
|
|
|
It should include, at minimum:
|
|
|
|
|
|
|
|
|
|
- prompt ID and version
|
|
|
|
|
- selected profile ID
|
|
|
|
|
- effective runtime settings
|
|
|
|
|
- input hashes
|
|
|
|
|
- rendered prompt hash
|
|
|
|
|
- rendered messages
|
|
|
|
|
|
|
|
|
|
JSON output should not include raw API key values.
|
|
|
|
|
|
|
|
|
|
JSON output should remain stable enough to be useful for automation and integration tests.
|
|
|
|
|
|
|
|
|
|
## 8. Guardrails
|
|
|
|
|
|
|
|
|
|
- **Separation of Concerns**: Prompt content must not belong in execution profiles; model/API settings must not belong in prompt definitions.
|
|
|
|
|
- **Security**: Raw API keys are unsupported in all configuration and transport layers. Only `api_key_env` is used.
|
|
|
|
|
- **Secret Handling**: Resolved API key values must never appear in rendered output, metadata, logs, HTTP responses, or CLI output.
|
|
|
|
|
- **Path Resolution**: `content_file` paths in prompt definitions resolve relative to the prompt YAML file.
|
|
|
|
|
- **Integrity**: No silent prompt truncation or omission of content.
|
|
|
|
|
- **Reliability**: Repair loops are strictly bounded by `repair_attempts`.
|
|
|
|
|
- **No Orchestration Creep**: Scriptorium prepares and executes a single prompt request. It does not coordinate multi-stage workflows.
|
|
|
|
|
- **Render Reuse**: The full run path must reuse the prepare/render path to avoid divergent behavior.
|
|
|
|
|
- **Formatter Isolation**: Render output formatters must not perform usecase work. They only format a completed `PreparedRun`.
|
|
|
|
|
|
|
|
|
|
## 9. Testing Strategy
|
|
|
|
|
|
|
|
|
|
Tests should protect both the run path and the prepare/render path.
|
|
|
|
|
|
|
|
|
|
### Prepare / Render Tests
|
|
|
|
|
|
|
|
|
|
Add tests for:
|
|
|
|
|
|
|
|
|
|
- preparing a prompt with explicit profile selection
|
|
|
|
|
- preparing a prompt using `default_profile`
|
|
|
|
|
- failing when no explicit profile and no `default_profile` exist
|
|
|
|
|
- runtime overrides beating profile values
|
|
|
|
|
- profile values beating application defaults
|
|
|
|
|
- file-backed prompt bodies rendering correctly
|
|
|
|
|
- required inputs failing when missing
|
|
|
|
|
- optional inputs being absent when not referenced
|
|
|
|
|
- unknown input references failing
|
|
|
|
|
- input hashes being included
|
|
|
|
|
- rendered prompt hash being included
|
|
|
|
|
- effective runtime settings being included
|
|
|
|
|
- `api_key_env` name being included where appropriate
|
|
|
|
|
- resolved API key values never appearing in `PreparedRun`
|
|
|
|
|
- prepare path not calling the LLM
|
|
|
|
|
|
|
|
|
|
### CLI Render Tests
|
|
|
|
|
|
|
|
|
|
Add tests for:
|
|
|
|
|
|
|
|
|
|
- `scriptorium render` mapping flags into `RunRequest`
|
|
|
|
|
- default text output
|
|
|
|
|
- explicit `--format text`
|
|
|
|
|
- explicit `--format json`
|
|
|
|
|
- unknown format failure
|
|
|
|
|
- text output includes prompt/profile/messages
|
|
|
|
|
- JSON output includes prompt/profile/messages
|
|
|
|
|
- rendered output never includes resolved API key values
|
|
|
|
|
|
|
|
|
|
### Run Reuse Tests
|
|
|
|
|
|
|
|
|
|
Add tests proving:
|
|
|
|
|
|
|
|
|
|
- `Runner.Run` reuses prepare behavior
|
|
|
|
|
- run and render resolve the same prompt/profile/runtime settings for equivalent inputs
|
|
|
|
|
- run still validates output
|
|
|
|
|
- run still performs bounded repair where configured
|
|
|
|
|
|
|
|
|
|
### Existing Tests
|
|
|
|
|
|
|
|
|
|
Continue testing:
|
|
|
|
|
|
|
|
|
|
- prompt definition loading
|
|
|
|
|
- execution profile loading
|
|
|
|
|
- artifact loading
|
|
|
|
|
- prompt rendering
|
|
|
|
|
- LLM adapter behavior
|
|
|
|
|
- validation behavior
|
|
|
|
|
- HTTP request/response mapping
|
|
|
|
|
|
|
|
|
|
## 10. Extension Points
|
|
|
|
|
|
|
|
|
|
Future work should remain grounded in the current architecture:
|
|
|
|
|
|
|
|
|
|
- **Artifacts**: Add S3 artifact references via a new `artifact.Reader`.
|
|
|
|
|
- **LLM**: Implement additional provider adapters, such as Anthropic or Google.
|
|
|
|
|
- **Execution**: Add token budgeting, streaming generation, and batch execution capabilities.
|
|
|
|
|
- **Prepare/Render**: Add token estimates, prompt-size summaries, or additional render output formats.
|
|
|
|
|
- **Repositories**: Implement database-backed repositories for prompts and profiles.
|
|
|
|
|
- **Profiles**: Support more granular profile versioning and environment-specific profiles.
|
|
|
|
|
- **HTTP**: Add an HTTP prepare/render endpoint if Narratio or another caller needs it.
|
|
|
|
|
|
|
|
|
|
Future render formats should plug into the formatter layer and should not require changes to the usecase layer.
|
|
|
|
|
- `docs/roadmap/documentation.md`
|
|
|
|
|
|
