Updated architecture documentation to describe the roadmap for the upcoming render feature

architecture.md

@@ -6,6 +6,8 @@ Scriptorium is a prompt-definition execution engine.
 
 It accepts named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata.
 
+Scriptorium also supports rendering/preparing a prompt without calling an LLM. This allows users to inspect the fully rendered prompt messages and effective runtime settings before executing a run.
+
 Scriptorium is not an orchestrator. It must not own transcription, transcript merge/polish steps, notifications, or cross-step workflow control.
 
 For the motivating D&D workflow:
@@ -22,97 +24,437 @@ Core Go code remains generic.
 Scriptorium uses a ports-and-adapters architecture to decouple the core execution logic from external dependencies.
 
 ### Package Responsibilities
+
 - `cmd/scriptorium`: Binary entrypoint for CLI and HTTP server.
-- `internal/domain`: Core domain contracts, including `PromptDefinition`, `ExecutionProfile`, and `RunResult`.
+- `internal/domain`: Core domain contracts, including `PromptDefinition`, `ExecutionProfile`, `PreparedRun`, `RunResult`, and related metadata.
-- `internal/usecase`: `Runner` orchestration, including the logic for profile selection, runtime override resolution, and bounded repair.
+- `internal/usecase`: `Runner` use case logic, including prompt preparation, profile selection, runtime override resolution, full run execution, validation, and bounded repair.
 - `internal/promptdef`: Repository for loading and validating Prompt Definitions from the filesystem.
 - `internal/profile`: Repository for loading Execution Profiles from the filesystem.
 - `internal/artifact`: Input artifact resolution (`inline`, `file`).
 - `internal/prompt`: Template rendering via Go templates.
 - `internal/llm`: Provider-neutral client interface and OpenAI-compatible HTTP adapter.
 - `internal/validate`: Output validation implementation (`none/basic/json/json_schema`).
-- `internal/adapter/cli`: CLI flag parsing and output handling.
+- `internal/adapter/cli`: CLI flag parsing, command dispatch, and output handling.
 - `internal/adapter/http`: HTTP request/response mapping.
+- `internal/format` or equivalent: Formatting of prepared/rendered prompt output for CLI or other adapters, if formatting grows beyond simple CLI-local helpers.
 
-## 3. Run Data Flow
+Exact package names may evolve, but the architectural boundaries should remain stable.
 
-The `Runner.Run` flow executes the following steps:
+## 3. Core Execution Model
 
-1. **Load Prompt Definition**: Retrieve the `PromptDefinition` by ID from the prompt repository.
+Scriptorium has two closely related execution paths:
-2. **Select Profile**: Determine the `profile_id` using the precedence:
+
+1. Prepare/render path.
+2. Full run path.
+
+The full run path should reuse the prepare path rather than duplicating its logic.
+
+### 3.1 Prepare / Render Data Flow
+
+The prepare path should be represented in the use case layer, preferably as `Runner.Prepare(ctx, RunRequest)` or an equivalent method.
+
+It executes all pre-LLM work:
+
+1. **Validate Request**: Ensure the request includes a prompt ID and all minimum required fields.
+2. **Load Prompt Definition**: Retrieve the `PromptDefinition` by ID from the prompt repository.
+3. **Select Profile**: Determine the `profile_id` using this precedence:
    - Explicit `profile_id` in `RunRequest`.
    - `default_profile` specified in the `PromptDefinition`.
    - Error if neither is available.
-3. **Load Execution Profile**: Retrieve the `ExecutionProfile` from the profile repository.
+4. **Load Execution Profile**: Retrieve the `ExecutionProfile` from the profile repository.
-4. **Resolve Runtime Overrides**: Merge settings based on precedence (Highest to Lowest):
+5. **Resolve Runtime Overrides**: Merge settings based on precedence, highest to lowest:
-   - Runtime overrides (CLI flags or HTTP `model` object).
+   - Runtime overrides from CLI flags or HTTP request `model` object.
    - Execution Profile settings.
    - Built-in application defaults.
-5. **Resolve Artifacts**: Load all named input artifacts defined in the request.
+6. **Resolve Artifacts**: Load all named input artifacts defined in the request.
-6. **Render Prompt**: Apply template variables and input artifacts to the prompt templates.
+7. **Render Prompt**: Apply template variables and input artifacts to the prompt templates.
-7. **Call LLM**: Execute the generation request using the resolved execution target (effective runtime settings).
+8. **Compute Metadata**: Compute hashes, selected profile ID, prompt ID/version, effective runtime settings, input hashes, rendered prompt hash, and timing information as appropriate.
-8. **Validate/Repair**: 
+9. **Return PreparedRun**: Return a `PreparedRun` containing the rendered messages, effective runtime settings, resolved metadata, and input/prompt hashes.
-   - Validate the model output against the output contract.
+
-   - If structured validation fails and `repair_attempts > 0`, perform bounded repair and re-validate.
+The prepare path must not call the LLM.
-9. **Return Result**: Produce a `RunResult` containing the final artifact, metadata, and validation status.
+
+The prepare path must not validate model output, because there is no model output.
+
+The prepare path must not perform structured-output repair, because repair only applies after model output exists.
+
+The prepare path should not resolve or expose raw API key values. It may include the selected `api_key_env` name in effective runtime settings or metadata, but never the environment variable value.
+
+### 3.2 Full Run Data Flow
+
+The `Runner.Run(ctx, RunRequest)` flow should reuse the prepare path:
+
+1. **Prepare**: Call the shared prepare flow to load the prompt, select the profile, resolve artifacts, render prompt messages, and compute pre-run metadata.
+2. **Call LLM**: Execute the generation request using the effective runtime settings from the prepared run.
+3. **Build Output Artifact**: Convert the model response into the configured output artifact.
+4. **Validate Output**:
+   - Validate the model output against the prompt definition's output contract.
+   - Validation content failures remain successful run results with `validation.status=failed`.
+   - Validator runtime/config errors are run errors.
+5. **Repair If Configured**:
+   - If structured validation fails and `repair_attempts > 0`, perform bounded repair attempts.
+   - Re-validate after each repair attempt.
+   - Repair loops must remain strictly bounded.
+6. **Return RunResult**: Produce a `RunResult` containing the final artifact, validation status, raw model output, usage information, and metadata.
+
+`Runner.Run` should not duplicate profile selection, artifact resolution, or prompt rendering logic that already exists in `Runner.Prepare`.
 
 ## 4. Domain Model
 
 Key domain types:
-- `PromptDefinition`: Defines the "what" (templates, inputs, validation contract, and an optional `default_profile`).
+
-- `ExecutionProfile`: Defines the "how" (endpoint, model, generation parameters, and `api_key_env`).
+- `PromptDefinition`: Defines the "what" of the task: templates, inputs, output contract, validation settings, repair settings, and optional `default_profile`.
-- `RunRequest`: The intent to execute a prompt, including `prompt_id`, optional `profile_id`, inputs, variables, and optional runtime overrides.
+- `ExecutionProfile`: Defines the "how" of execution: endpoint, model, generation parameters, timeout, reasoning effort, and `api_key_env`.
-- `RunResult`: The outcome of a run, including the generated `Artifact`, `ValidationResult`, and auditing `RunMetadata`.
+- `RunRequest`: The intent to execute or prepare a prompt, including `prompt_id`, optional `profile_id`, inputs, variables, and optional runtime overrides.
-- `RunMetadata`: Detailed tracing info: `prompt_id`, `selected_profile_id`, model params, usage tokens, and hashes.
+- `PreparedRun`: The result of the prepare/render phase. Contains rendered messages, effective runtime settings, selected profile ID, prompt metadata, input hashes, prompt hash, and other pre-LLM metadata.
+- `RunResult`: The result of a full run. Contains the generated `Artifact`, `ValidationResult`, raw model output, token usage, and `RunMetadata`.
+- `RunMetadata`: Detailed tracing information, including prompt ID/version, selected profile ID, effective model parameters, usage tokens, hashes, timestamps, validation status, and repair attempts where applicable.
+- `RenderedPrompt`: Provider-neutral rendered prompt structure.
+- `RenderedMessage`: Provider-neutral rendered message with role and content.
+- `ArtifactRef`: A reference to an input artifact, such as `file` or `inline`.
+- `Artifact`: Loaded artifact content with name, content type, body, source URI, size, and hash.
+- `ValidationResult`: Validation status and details for full runs.
+
+`PreparedRun` should be serializable for JSON output and should also be representable in a human-readable text format.
 
 ## 5. Interfaces and Adapters
 
 ### Primary Ports
+
 - `promptdef.Repository`: Lookup for prompt definitions.
 - `profile.Repository`: Lookup for execution profiles.
 - `artifact.Reader`: Loading of artifact content.
 - `prompt.Renderer`: Template rendering.
 - `llm.Client`: Model generation.
 - `validate.Validator`: Output validation.
+- `format.PreparedRunFormatter` or equivalent: Optional formatting abstraction for rendered/prepared output.
 
 ### Current Adapters
+
 - **Repositories**: Filesystem YAML loaders for both prompts and profiles.
 - **Artifact Reader**: Composite reader supporting `file` and `inline`.
 - **Prompt Renderer**: Go templates with a custom `input` helper.
 - **LLM Client**: OpenAI-compatible `/chat/completions` over HTTP.
 - **Validator**: Standard validator supporting `none`, `basic`, `json`, and `json_schema`.
+- **CLI Adapter**: Supports `run`, `render`, and `serve`.
+- **HTTP Adapter**: Supports full run execution through `POST /v1/runs`.
 
 ## 6. Public Contracts
 
 ### CLI
-- `run`: Executes a prompt. Uses flags like `--prompt`, `--profile`, `--input`, and various runtime overrides (e.g., `--model`, `--temperature`).
+
-- `serve`: Starts the HTTP API using infrastructure-only flags (`--addr`, `--prompt-dir`, `--profile-dir`, `--schema-dir`). It does not introduce a server-level model/runtime precedence layer.
+Scriptorium should expose at least these commands:
+
+- `scriptorium run`: Executes a prompt by preparing it, calling the LLM, validating output, optionally repairing structured output, and returning an artifact.
+- `scriptorium render`: Prepares and renders a prompt without calling the LLM.
+- `scriptorium serve`: Starts the HTTP API using infrastructure-only flags.
+
+### `scriptorium run`
+
+`run` uses flags such as:
+
+- `--prompt-dir`
+- `--profile-dir`
+- `--prompt`
+- `--profile`
+- `--input`
+- `--var`
+- `--out`
+- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.
+
+`run` should produce the generated artifact as its primary output.
+
+### `scriptorium render`
+
+`render` prepares and renders a prompt without calling an LLM.
+
+It should use the same prompt/profile/input/variable/runtime override flags as `run` where applicable:
+
+- `--prompt-dir`
+- `--profile-dir`
+- `--prompt`
+- `--profile`
+- `--input`
+- `--var`
+- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.
+- `--format`, with initial support for `text` and `json`.
+
+Default render output format should be `text`.
+
+`render` must not call the LLM.
+
+`render` should show the same rendered messages and effective runtime settings that `run` would use.
+
+`render` should include enough information to debug:
+
+- prompt ID
+- prompt version
+- selected profile ID
+- effective runtime settings
+- input hashes
+- prompt hash
+- rendered messages
+
+`render` must not include resolved API key values.
+
+`render` may include the `api_key_env` name.
+
+### Render Output Formats
+
+Initial render output formats:
+
+- `text`: Human-readable default format.
+- `json`: Machine-readable structured representation of the prepared run.
+
+Additional formats, such as `markdown`, may be added later.
+
+Render output formatting should be modular. Adding a new output format should not require changing the prepare/run core logic.
+
+The output formatting layer should consume a `PreparedRun` and produce bytes or text for the adapter. It should not reload prompts, re-resolve artifacts, re-render templates, call the LLM, or perform validation.
+
+### `scriptorium serve`
+
+`serve` starts the HTTP API.
+
+It should use infrastructure-only flags such as:
+
+- `--addr`
+- `--prompt-dir`
+- `--profile-dir`
+- `--schema-dir`
+
+`serve` should not introduce a server-level model/runtime precedence layer unless explicitly documented and intentionally implemented.
 
 ### HTTP API
+
+Current HTTP API:
+
 - `POST /v1/runs`: Accepts `RunRequest` JSON and returns `RunResponse` JSON. No built-in auth.
-- Request may include runtime overrides under `model` and an `include_raw_output` boolean.
+
-- `raw_model_output` is exposed only when explicitly requested with `include_raw_output=true`.
+Request may include runtime overrides under `model` and an `include_raw_output` boolean.
+
+`raw_model_output` is exposed only when explicitly requested with `include_raw_output=true`.
+
+A future HTTP prepare/render endpoint may be added, such as `POST /v1/renders` or `POST /v1/runs/prepare`, but the initial render feature may be CLI-only. If added later, it should call the same usecase-level prepare path as `scriptorium render`.
 
 ### YAML Shapes
-- **Prompt YAML**: Includes `id`, `version`, optional `default_profile`, `inputs`, `messages`, and `output`.
-  - Inputs support `name`, `required`, optional `content_type`, and `description`.
-  - Messages require `role` and exactly one of `content` or `content_file`.
-  - `content_file` resolves relative to the prompt YAML location.
-- **Profile YAML**: Includes `id`, `endpoint`, `model`, generation params, and `api_key_env`.
 
-## 7. Guardrails
+Prompt YAML includes:
+
+- `id`
+- `version`
+- optional `default_profile`
+- `inputs`
+- `messages`
+- `output`
+
+Inputs support:
+
+- `name`
+- `required`
+- optional `content_type`
+- `description`
+
+Messages require:
+
+- `role`
+- exactly one of `content` or `content_file`
+
+`content_file` resolves relative to the prompt YAML location.
+
+Profile YAML includes:
+
+- `id`
+- `endpoint`
+- `model`
+- generation parameters
+- timeout settings
+- reasoning settings
+- `api_key_env`
+
+Prompt content must not appear in profile YAML.
+
+Model/runtime/API-key settings must not appear in prompt YAML, except that prompt YAML may specify `default_profile`.
+
+## 7. Render Feature Design
+
+The render feature is a first-class use case, not a CLI-only shortcut.
+
+### Goals
+
+The render feature should help users:
+
+- inspect fully rendered prompt messages
+- debug missing inputs
+- verify template variable substitution
+- verify selected profile resolution
+- verify runtime override precedence
+- verify file-backed prompt loading
+- inspect input hashes and prompt hashes
+- prepare for future token budgeting and prompt-size inspection
+
+### Non-Goals
+
+The render feature should not:
+
+- call an LLM
+- validate model output
+- repair structured output
+- resolve or print API key values
+- mutate artifacts
+- save outputs to artifact storage unless a future explicit output option is added
+- become an orchestration step manager
+
+### Usecase Shape
+
+The preferred usecase shape is:
+
+- `Runner.Prepare(ctx, RunRequest) (*PreparedRun, error)`
+- `Runner.Run(ctx, RunRequest) (*RunResult, error)`
+
+`Runner.Run` should call `Runner.Prepare`.
+
+The prepare flow should be the only implementation of:
+
+- prompt loading
+- profile selection
+- runtime override resolution
+- artifact resolution
+- prompt rendering
+- pre-run metadata/hash calculation
+
+### CLI Shape
+
+The preferred command name is `render`.
+
+The command should support:
+
+- `--format text`
+- `--format json`
+
+Default format:
+
+- `text`
+
+Unknown formats should produce a clear error.
+
+Formatting should be centralized through a small formatter registry, strategy, switch, or interface so new formats can be added without modifying usecase logic.
+
+### Text Output Expectations
+
+Text output should be optimized for human inspection.
+
+It should include, at minimum:
+
+- prompt ID and version
+- selected profile ID
+- model name
+- endpoint
+- effective generation settings
+- input hashes
+- rendered prompt hash
+- rendered messages grouped by role
+
+Text output should be readable and deterministic enough for tests.
+
+It should not include raw API key values.
+
+### JSON Output Expectations
+
+JSON output should be a structured representation of `PreparedRun` or a DTO derived from it.
+
+It should include, at minimum:
+
+- prompt ID and version
+- selected profile ID
+- effective runtime settings
+- input hashes
+- rendered prompt hash
+- rendered messages
+
+JSON output should not include raw API key values.
+
+JSON output should remain stable enough to be useful for automation and integration tests.
+
+## 8. Guardrails
 
 - **Separation of Concerns**: Prompt content must not belong in execution profiles; model/API settings must not belong in prompt definitions.
 - **Security**: Raw API keys are unsupported in all configuration and transport layers. Only `api_key_env` is used.
+- **Secret Handling**: Resolved API key values must never appear in rendered output, metadata, logs, HTTP responses, or CLI output.
 - **Path Resolution**: `content_file` paths in prompt definitions resolve relative to the prompt YAML file.
 - **Integrity**: No silent prompt truncation or omission of content.
 - **Reliability**: Repair loops are strictly bounded by `repair_attempts`.
+- **No Orchestration Creep**: Scriptorium prepares and executes a single prompt request. It does not coordinate multi-stage workflows.
+- **Render Reuse**: The full run path must reuse the prepare/render path to avoid divergent behavior.
+- **Formatter Isolation**: Render output formatters must not perform usecase work. They only format a completed `PreparedRun`.
 
-## 8. Extension Points
+## 9. Testing Strategy
+
+Tests should protect both the run path and the prepare/render path.
+
+### Prepare / Render Tests
+
+Add tests for:
+
+- preparing a prompt with explicit profile selection
+- preparing a prompt using `default_profile`
+- failing when no explicit profile and no `default_profile` exist
+- runtime overrides beating profile values
+- profile values beating application defaults
+- file-backed prompt bodies rendering correctly
+- required inputs failing when missing
+- optional inputs being absent when not referenced
+- unknown input references failing
+- input hashes being included
+- rendered prompt hash being included
+- effective runtime settings being included
+- `api_key_env` name being included where appropriate
+- resolved API key values never appearing in `PreparedRun`
+- prepare path not calling the LLM
+
+### CLI Render Tests
+
+Add tests for:
+
+- `scriptorium render` mapping flags into `RunRequest`
+- default text output
+- explicit `--format text`
+- explicit `--format json`
+- unknown format failure
+- text output includes prompt/profile/messages
+- JSON output includes prompt/profile/messages
+- rendered output never includes resolved API key values
+
+### Run Reuse Tests
+
+Add tests proving:
+
+- `Runner.Run` reuses prepare behavior
+- run and render resolve the same prompt/profile/runtime settings for equivalent inputs
+- run still validates output
+- run still performs bounded repair where configured
+
+### Existing Tests
+
+Continue testing:
+
+- prompt definition loading
+- execution profile loading
+- artifact loading
+- prompt rendering
+- LLM adapter behavior
+- validation behavior
+- HTTP request/response mapping
+
+## 10. Extension Points
 
 Future work should remain grounded in the current architecture:
+
 - **Artifacts**: Add S3 artifact references via a new `artifact.Reader`.
-- **LLM**: Implement additional provider adapters (e.g., Anthropic, Google).
+- **LLM**: Implement additional provider adapters, such as Anthropic or Google.
 - **Execution**: Add token budgeting, streaming generation, and batch execution capabilities.
+- **Prepare/Render**: Add token estimates, prompt-size summaries, or additional render output formats.
 - **Repositories**: Implement database-backed repositories for prompts and profiles.
 - **Profiles**: Support more granular profile versioning and environment-specific profiles.
+- **HTTP**: Add an HTTP prepare/render endpoint if Narratio or another caller needs it.
+
+Future render formats should plug into the formatter layer and should not require changes to the usecase layer.