15 Commits

Author SHA1 Message Date
bc099a31ad Finish cleanup roadmap follow-through
All checks were successful
ci/woodpecker/tag/release Pipeline was successful
2026-05-26 11:05:08 -05:00
4ff55221a3 Update internal docs for stable runner error reasons and serialized model fields 2026-05-26 15:00:06 +00:00
8d8024099f Complete final cleanup verification and align unsupported artifact test naming 2026-05-26 13:23:45 +00:00
18792fd8d1 Remove unsupported S3 artifact reference placeholder 2026-05-26 13:21:52 +00:00
8860aa033c Reduce CLI test fixture duplication with local setup helpers 2026-05-26 13:20:07 +00:00
3ca14d8b6e Add regression test for JSON schema load failure during prepare 2026-05-26 13:16:26 +00:00
099e9c4a3e Use stable usecase sentinels for HTTP invalid-request mapping 2026-05-26 13:14:57 +00:00
cfe6b9408a Refactor CLI command wiring with shared settings and runner helpers 2026-05-26 13:12:34 +00:00
6ececc749f Centralize YAML catalog scanning for prompt and profile repositories 2026-05-26 13:10:09 +00:00
79901fbb86 Refine execution target mapping helpers and coverage across usecase, HTTP, and LLM 2026-05-26 13:07:35 +00:00
75fa0a030a Added a roadmap to address the issues identifed by the audit 2026-05-26 08:01:33 -05:00
ef64966897 Audit code quality and deduplication opportunities 2026-05-26 07:52:48 -05:00
c6c5e3cb69 Added support for the service_tier key in profiles 2026-05-26 07:44:49 -05:00
3f4fd230b9 Implemented support for loading configuration from nested subdirectories 2026-05-26 07:36:10 -05:00
2091b58066 Completed the documentation update and removed the completed roadmap 2026-05-26 07:15:36 -05:00
46 changed files with 1465 additions and 840 deletions

View File

@@ -29,7 +29,6 @@ This command renders the prepared prompt and effective runtime settings without
- [OpenAI-compatible chat integration](docs/integrations/openai-compatible-chat.md) - [OpenAI-compatible chat integration](docs/integrations/openai-compatible-chat.md)
- [Narratio subprocess integration](docs/integrations/narratio.md) - [Narratio subprocess integration](docs/integrations/narratio.md)
- [Architecture policy](docs/policy/architecture.md) - [Architecture policy](docs/policy/architecture.md)
- [Documentation roadmap](docs/roadmap/documentation.md)
## Examples ## Examples

View File

@@ -1,14 +0,0 @@
# Architecture (Moved)
The canonical architecture policy is:
- `docs/policy/architecture.md`
Implemented internal component documentation is:
- `docs/internal/runner.md`
- `docs/internal/adapters.md`
Roadmap-only planning content is:
- `docs/roadmap/documentation.md`

View File

@@ -20,8 +20,8 @@ go run ./cmd/scriptorium render \
Integration references: Integration references:
- HTTP contract: `docs/integrations/http-api.md` - [HTTP contract](integrations/http-api.md)
- Narratio subprocess contract: `docs/integrations/narratio.md` - [Narratio subprocess contract](integrations/narratio.md)
## Common Argument Rules ## Common Argument Rules
@@ -134,7 +134,7 @@ go run ./cmd/scriptorium run \
--profile local-fast \ --profile local-fast \
--input transcript=./examples/fixtures/transcript.md \ --input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \ --input glossary=./examples/fixtures/glossary.yml \
--out ./out/summary.md --out ./summary.md
``` ```
Start the HTTP server with explicit config: Start the HTTP server with explicit config:

View File

@@ -20,8 +20,8 @@ When `--config <path>` is provided, that file is required.
## Minimal App Config ## Minimal App Config
```yaml ```yaml
prompt_dir: ./prompts prompt_dir: ./examples/prompts
profile_dir: ./profiles profile_dir: ./examples/profiles
``` ```
This is enough to use `run` and `render` when prompt/profile files are valid. This is enough to use `run` and `render` when prompt/profile files are valid.
@@ -63,7 +63,9 @@ Validation behavior:
## Prompt Definition Files ## Prompt Definition Files
Prompt definitions are YAML files in `prompt_dir`. Prompt definitions are YAML files anywhere under `prompt_dir`, including nested subdirectories.
Subdirectories are organizational only. Callers still select prompts by the YAML `id`, not by file path. For example, `prompts/dnd/recap.yaml` may still declare `id: dnd.recap`, and callers use `--prompt dnd.recap`.
Example: Example:
@@ -122,7 +124,9 @@ Message rules:
- Repeated roles are allowed. - Repeated roles are allowed.
- `content_file` is resolved relative to the prompt YAML file location. - `content_file` is resolved relative to the prompt YAML file location.
- Nested prompt files keep the same relative `content_file` behavior; `./recap.user.md` next to `dnd/recap.yaml` resolves from `dnd/`.
- Prompt decoding is strict; unknown YAML fields are rejected. - Prompt decoding is strict; unknown YAML fields are rejected.
- Duplicate prompt IDs are invalid. If multiple files declare the requested prompt ID, Scriptorium fails instead of choosing one.
`output` fields: `output` fields:
@@ -138,7 +142,9 @@ Repair behavior boundary:
## Profile Definition Files ## Profile Definition Files
Execution profiles are YAML files in `profile_dir`. Execution profiles are YAML files anywhere under `profile_dir`, including nested subdirectories.
Subdirectories are organizational only. Callers still select profiles by the YAML `id`, not by file path. For example, `profiles/local/local-quality.yaml` may still declare `id: local-quality`, and callers use `--profile local-quality`.
Example: Example:
@@ -151,6 +157,7 @@ max_tokens: 500
top_p: 1.0 top_p: 1.0
timeout_seconds: 90 timeout_seconds: 90
api_key_env: SCRIPTORIUM_API_KEY api_key_env: SCRIPTORIUM_API_KEY
service_tier: priority
``` ```
Field reference: Field reference:
@@ -162,6 +169,7 @@ Field reference:
- `max_tokens` (optional): `>= 0` - `max_tokens` (optional): `>= 0`
- `top_p` (optional): range `0..1` - `top_p` (optional): range `0..1`
- `timeout_seconds` (optional): `>= 0` - `timeout_seconds` (optional): `>= 0`
- `service_tier` (optional): provider-specific request tier such as OpenRouter `flex` or `priority`
- `reasoning_effort` (optional) - `reasoning_effort` (optional)
- `api_key_env` (optional) - `api_key_env` (optional)
- `extra_params` (optional map of strings) - `extra_params` (optional map of strings)
@@ -171,10 +179,11 @@ Profile rules:
- Profile decoding is strict; unknown YAML fields are rejected. - Profile decoding is strict; unknown YAML fields are rejected.
- Raw `api_key` is rejected; use `api_key_env`. - Raw `api_key` is rejected; use `api_key_env`.
- If `api_key_env` is set, that environment variable must be set when preparing/running. - If `api_key_env` is set, that environment variable must be set when preparing/running.
- Duplicate profile IDs are invalid. If multiple files declare the requested profile ID, Scriptorium fails instead of choosing one.
Current outbound request behavior: Current outbound request behavior:
- The OpenAI-compatible client currently serializes: `model`, `messages`, `temperature`, `max_tokens`, `top_p`, and optional `response_format` for `json_schema` prompts. - The OpenAI-compatible client currently serializes: `model`, `messages`, `temperature`, `max_tokens`, `top_p`, `service_tier`, and optional `response_format` for `json_schema` prompts.
- `reasoning_effort` and `extra_params` are parsed and carried in effective settings, but are not currently serialized into outbound chat-completions requests. - `reasoning_effort` and `extra_params` are parsed and carried in effective settings, but are not currently serialized into outbound chat-completions requests.
## Schema Behavior ## Schema Behavior
@@ -184,8 +193,9 @@ Schemas are JSON files, typically in `schema_dir`.
Rules: Rules:
- `output.validation_mode: json_schema` requires `output.schema_path`. - `output.validation_mode: json_schema` requires `output.schema_path`.
- Relative `schema_path` values resolve from `schema_dir`. - Relative `schema_path` values resolve from `schema_dir`, including explicit nested paths such as `dnd/structured_events.schema.json`.
- Absolute `schema_path` values are used directly. - Absolute `schema_path` values are used directly.
- Scriptorium does not recursively search schemas by basename; nested schemas must be referenced by their relative path.
- Missing or invalid schema documents cause runtime validation errors. - Missing or invalid schema documents cause runtime validation errors.
- Invalid generated JSON causes validation status `failed` (not a runtime error). - Invalid generated JSON causes validation status `failed` (not a runtime error).
@@ -200,14 +210,22 @@ Supported artifact reference types for request inputs are `file` and `inline`.
## Maintained Examples ## Maintained Examples
- App config: `examples/config.yml` - App config: `examples/config.yml`
- Prompt examples: `prompts/` - Prompt examples: `examples/prompts/`
- Profile examples: `profiles/` - Profile examples: `examples/profiles/`
- Schema examples: `schemas/` - Schema examples: `examples/schemas/`
- Input fixtures: `examples/fixtures/` - Input fixtures: `examples/fixtures/`
- Render example script: `examples/render-markdown-summary.sh` - Render example script: `examples/render-markdown-summary.sh`
- HTTP request example: `examples/http-run.json` - HTTP request example: `examples/http-run.json`
Example organizational layout:
```text
examples/prompts/dnd/recap.yaml
examples/profiles/local/local-quality.yaml
examples/schemas/dnd/structured_events.schema.json
```
## Integration References ## Integration References
- Inbound HTTP contract: `docs/integrations/http-api.md` - [Inbound HTTP contract](integrations/http-api.md)
- Outbound OpenAI-compatible contract: `docs/integrations/openai-compatible-chat.md` - [Outbound OpenAI-compatible contract](integrations/openai-compatible-chat.md)

View File

@@ -8,7 +8,7 @@ Current scope is only:
- `POST /v1/runs` - `POST /v1/runs`
For CLI behavior, see `docs/cli.md`. For CLI behavior, see the [CLI reference](../cli.md).
## Endpoint ## Endpoint
@@ -46,6 +46,7 @@ Copyable request example file:
"max_tokens": 800, "max_tokens": 800,
"top_p": 1.0, "top_p": 1.0,
"timeout_seconds": 120, "timeout_seconds": 120,
"service_tier": "priority",
"reasoning_effort": "medium", "reasoning_effort": "medium",
"api_key_env": "SCRIPTORIUM_API_KEY", "api_key_env": "SCRIPTORIUM_API_KEY",
"extra_params": { "extra_params": {
@@ -114,6 +115,7 @@ Response shape:
"max_tokens": 800, "max_tokens": 800,
"top_p": 1, "top_p": 1,
"timeout_seconds": 120, "timeout_seconds": 120,
"service_tier": "priority",
"reasoning_effort": "medium", "reasoning_effort": "medium",
"api_key_env": "SCRIPTORIUM_API_KEY", "api_key_env": "SCRIPTORIUM_API_KEY",
"extra_params": { "extra_params": {

View File

@@ -109,6 +109,6 @@ A `run` exit code `2` can still produce output (stdout or `--out`).
## Canonical References ## Canonical References
- CLI behavior: `docs/cli.md` - CLI behavior: [CLI reference](../cli.md)
- Config behavior: `docs/config.md` - Config behavior: [Configuration reference](../config.md)
- Operations and failure handling: `docs/operations.md`, `docs/troubleshooting.md` - Operations and failure handling: [Operations guide](../operations.md), [Troubleshooting](../troubleshooting.md)

View File

@@ -30,8 +30,11 @@ Serialized JSON fields:
- `temperature` (only when non-zero) - `temperature` (only when non-zero)
- `max_tokens` (only when non-zero) - `max_tokens` (only when non-zero)
- `top_p` (only when non-zero) - `top_p` (only when non-zero)
- `service_tier` (only when non-empty)
- `response_format` (only when structured output is provided) - `response_format` (only when structured output is provided)
`service_tier` is provider-specific. OpenRouter currently documents request values such as `flex` and `priority`; Scriptorium forwards any non-empty configured value and lets the backend validate support.
Structured output is currently `json_schema` only, serialized as: Structured output is currently `json_schema` only, serialized as:
```json ```json

View File

@@ -30,7 +30,7 @@ HTTP adapter:
Filesystem repositories: Filesystem repositories:
- Input: prompt/profile YAML files. - Input: prompt/profile YAML files under configured directories.
- Output: normalized domain definitions/profiles or typed errors. - Output: normalized domain definitions/profiles or typed errors.
Artifact reader: Artifact reader:
@@ -66,7 +66,7 @@ Primary app settings consumed by adapters:
Execution profile/request settings used through runner: Execution profile/request settings used through runner:
- `endpoint`, `model`, `temperature`, `max_tokens`, `top_p`, `timeout_seconds`, `api_key_env`, `reasoning_effort`, `extra_params` - `endpoint`, `model`, `temperature`, `max_tokens`, `top_p`, `timeout_seconds`, `service_tier`, `api_key_env`, `reasoning_effort`, `extra_params`
## External Dependencies ## External Dependencies
@@ -79,6 +79,9 @@ Execution profile/request settings used through runner:
Strict decoding and input checks: Strict decoding and input checks:
- config/prompt/profile loaders reject unknown YAML fields. - config/prompt/profile loaders reject unknown YAML fields.
- prompt/profile repositories scan nested subdirectories recursively.
- prompt/profile lookup uses YAML `id` values; subdirectory paths are organizational only.
- duplicate prompt/profile IDs are invalid and fail instead of using first-match behavior.
- HTTP DTO decoder rejects unknown JSON fields. - HTTP DTO decoder rejects unknown JSON fields.
- raw API key payload fields are rejected by strict decoding in profile/http paths. - raw API key payload fields are rejected by strict decoding in profile/http paths.
@@ -97,10 +100,12 @@ Validator:
- `basic`, `json`, `json_schema` content failures return `ValidationFailed` results. - `basic`, `json`, `json_schema` content failures return `ValidationFailed` results.
- schema load/compile/path failures are runtime errors. - schema load/compile/path failures are runtime errors.
- schema lookup uses explicit `schema_path` values relative to `schema_dir`; it does not recursively search by basename.
HTTP error mapping: HTTP error mapping:
- maps domain/use-case errors to stable HTTP code + error code/message. - maps domain/use-case errors to stable HTTP code + error code/message.
- distinguishes missing profile selection and missing `api_key_env` variable using stable use-case sentinel errors.
- avoids returning internal wrapped-cause details in response payload. - avoids returning internal wrapped-cause details in response payload.
## CLI Adapter Semantics ## CLI Adapter Semantics
@@ -136,4 +141,4 @@ Behavior highlights:
- Adapter packages do not own runner decision logic. - Adapter packages do not own runner decision logic.
- External request/response strictness is part of contract stability. - External request/response strictness is part of contract stability.
- Prepared-render output never includes resolved API key values. - Prepared-render output never includes resolved API key values.
- Outbound OpenAI-compatible request includes only currently serialized fields (`model`, `messages`, optional `temperature`, `max_tokens`, `top_p`, optional `response_format`). - Outbound OpenAI-compatible request includes only currently serialized fields (`model`, `messages`, optional `temperature`, `max_tokens`, `top_p`, optional `service_tier`, optional `response_format`).

View File

@@ -66,14 +66,21 @@ It receives fully constructed repositories/readers/validators from adapters. Eff
## Failure Behavior ## Failure Behavior
Key error classes surfaced from `Runner`: Primary runner error classes:
- `ErrInvalidRequest`: invalid prompt/profile/request/runtime/API-key-env prerequisites. - `ErrInvalidRequest`: invalid run request envelope.
- `ErrProfileLoad`: prompt or profile load failures. - `ErrProfileRequired`: specific invalid-request reason when neither request `profile_id` nor prompt `default_profile` is available.
- `ErrAPIKeyEnvMissing`: specific invalid-request reason when `api_key_env` is set but the named environment variable is unset/empty.
- `ErrProfileLoad`: prompt/profile repository load failures.
- `ErrArtifactLoad`: artifact read failures. - `ErrArtifactLoad`: artifact read failures.
- `ErrPromptRender`: template render failures. - `ErrPromptRender`: template render failures.
- `ErrLLMGenerate`: model request failures. - `ErrLLMGenerate`: outbound model request failures.
- `ErrValidation`: validation runtime failures (including schema load/compile failures). - `ErrValidation`: validation runtime failures (including structured-output schema load/compile failures).
Reason sentinel behavior:
- `ErrProfileRequired` and `ErrAPIKeyEnvMissing` are wrapped with `ErrInvalidRequest`.
- Adapters can use `errors.Is` for stable reason mapping without matching runner prose.
Validation content failures are not run errors: Validation content failures are not run errors:
@@ -90,13 +97,15 @@ Validation content failures are not run errors:
3. select profile ID: 3. select profile ID:
- explicit request profile ID - explicit request profile ID
- prompt `default_profile` - prompt `default_profile`
- otherwise request error - otherwise return an invalid request with `ErrProfileRequired`
4. load execution profile. 4. load execution profile.
5. merge effective runtime target: 5. merge effective runtime target:
- built-in execution defaults - built-in execution defaults
- selected profile values - selected profile values
- request overrides - request overrides
6. verify required `api_key_env` environment variable (name only; value is not returned). 6. verify required `api_key_env` environment variable:
- missing/empty env value returns an invalid request with `ErrAPIKeyEnvMissing`
- only the environment-variable name is retained; secret value is never returned
7. resolve output contract and structured-output schema payload when `json_schema` mode is active. 7. resolve output contract and structured-output schema payload when `json_schema` mode is active.
8. read input artifacts. 8. read input artifacts.
9. render prompt messages. 9. render prompt messages.

View File

@@ -48,8 +48,9 @@ Typical sequence:
1. Confirm prompt/profile directories resolve through config or flags. 1. Confirm prompt/profile directories resolve through config or flags.
2. Confirm required input files exist and map to prompt input names. 2. Confirm required input files exist and map to prompt input names.
3. Confirm required API-key environment variables are set. 3. Confirm required API-key environment variables are set.
4. Run `render` for preflight when changing prompt/profile/input wiring. 4. Confirm the selected profile's model endpoint is reachable from the process environment.
5. Run `run` for actual generation. 5. Run `render` for preflight when changing prompt/profile/input wiring.
6. Run `run` for actual generation.
## Secrets Handling ## Secrets Handling

View File

@@ -1,355 +0,0 @@
# Documentation Roadmap
## Purpose
This roadmap defines the work required to bring scriptorium's documentation into compliance with `docs/policy/documentation.md` and the current implementation. It is a planning document only; future agents should use it to update the canonical documentation without describing unimplemented behavior outside `docs/roadmap/`.
## Repository Documentation Inventory
- `README.md` - keep and rewrite. It is currently a full manual covering config, CLI, HTTP, prompt/profile authoring, examples, and build commands; policy says README should be a short orientation page with a quickstart and links.
- `architecture.md` - move/merge/delete after rewrite. It overlaps with `docs/policy/architecture.md`, contains "should" guidance and future extension notes outside `docs/roadmap/`, and references architecture that is partly stale or aspirational.
- `docs/policy/documentation.md` - keep and lightly update only if policy itself changes. It is the controlling documentation policy.
- `docs/policy/architecture.md` - keep and lightly update. It is the canonical development architecture policy, but some package-layout defaults do not exactly match this repository (`internal/adapter/...` versus policy examples such as `internal/adapters/...`).
- `docs/policy/development.md` - create new. Required by policy for projects maintained by humans and LLM coding agents.
- `docs/config/config-yml.md` - merge into `docs/config.md`. The content mostly matches code but lives in the wrong canonical home.
- `docs/config/prompt-definitions.md` - merge into `docs/config.md`. The field reference is mostly accurate, but it needs caveats about `repair_attempts`, repeated message roles, schema path resolution, and examples.
- `docs/config/profile-definitions.md` - merge into `docs/config.md`. It must stop implying that every profile field is sent to the provider; the current OpenAI-compatible client does not send `reasoning_effort` or `extra_params`.
- `docs/config/schema-definitions.md` - merge into `docs/config.md` or link from it. Schema behavior is implemented, but the canonical config reference should own this material.
- `docs/cli.md` - create new. Required by policy for the implemented CLI.
- `docs/operations.md` - create new. Required by policy for this CLI/service application; scope should cover normal operation, config paths, secret handling, stdout/stderr, exit codes, HTTP serving, and the fact that there is no durable run state/resume behavior.
- `docs/troubleshooting.md` - create new. Recommended by policy and justified by implemented failure modes in parser, config, prompt/profile loading, artifact reading, validation, LLM calls, and HTTP error mapping.
- `docs/internal/` - create new. Required by policy for this modular application.
- `docs/integrations/narratio.md` - keep and rewrite. It documents an actual CLI integration contract, but it includes future extension notes outside roadmap and illustrative prompt IDs that are not all present in the repository.
- `docs/integrations/http-api.md` - create new. The implemented `POST /v1/runs` API is an external integration contract and should not live in README.
- `docs/integrations/openai-compatible-chat.md` - create new. The outbound LLM contract is important and implemented in `internal/llm/openai_compatible_client.go`.
- `examples/config.yml` - keep and lightly update if paths move. It is a valid app config example for the current root `prompts/`, `profiles/`, and `schemas/` directories.
- `examples/fixtures/transcript.md` - keep. It is used by integration tests.
- `examples/fixtures/glossary.yml` - keep. It is used by integration tests.
- `prompts/` - keep as maintained sample prompt library for now; recommended to move or mirror under `examples/` only if tests and docs are updated together.
- `profiles/` - keep as maintained sample profile library for now; recommended to move or mirror under `examples/` only if tests and docs are updated together.
- `schemas/` - keep as maintained sample schema library for now; recommended to move or mirror under `examples/` only if tests and docs are updated together.
- `local-test/` - delete, move out of the repository, or explicitly exclude from maintained docs. It contains ad hoc local artifacts and provider profiles; it should not be linked from canonical docs unless promoted to maintained examples with tests and secret review.
## Policy Compliance Assessment
Required documents that are missing:
- `docs/cli.md`
- `docs/config.md`
- `docs/operations.md`
- `docs/internal/`
- `docs/policy/development.md`
Recommended documents that should be added:
- `docs/troubleshooting.md`
- Validated examples under `examples/` beyond the current config and fixtures, especially command examples that can be checked with `render`.
- Integration docs for the implemented HTTP API and outbound OpenAI-compatible chat API.
Documents that exist but are stale or in the wrong canonical home:
- `README.md` duplicates material that belongs in `docs/cli.md`, `docs/config.md`, `docs/integrations/`, and `docs/internal/`.
- `docs/config/*.md` should be merged into `docs/config.md`.
- `architecture.md` should be merged into `docs/policy/architecture.md`, `docs/internal/`, or `docs/roadmap/`, then removed.
- `docs/integrations/narratio.md` should remain under integrations but must be narrowed to implemented CLI behavior and actual integration guidance.
Content that appears to describe deprecated, historical, planned, or unimplemented behavior outside `docs/roadmap/`:
- `architecture.md` has future extension notes for S3 artifact references, additional LLM providers, streaming, batch execution, database-backed repositories, profile versioning, and HTTP render endpoints.
- `architecture.md` and `README.md` describe bounded repair as if it is generally active. The code has an injected repairer hook, but the CLI and HTTP server construct `Runner` without a repairer, so production commands do not currently perform repair attempts.
- `README.md` says "additional output formats can be added later"; this belongs in roadmap only.
- `README.md` references `go build -o scriptorium ./cmd/scriptorium`, which is valid, but the README should not be the build/test manual after `docs/policy/development.md` exists.
- `docs/integrations/narratio.md` has "Future Extension Notes" and examples for prompt IDs not present in the repository, such as `dnd.structured_events`, `dnd.glossary_suggestions`, and `dnd.player_summary`.
- Any docs implying S3 artifact support should be removed from current-behavior docs. `domain.ArtifactRefS3` exists, but `artifact.CompositeReader` supports only `inline` and `file`.
Examples that are missing, stale, invalid, or untested:
- `examples/config.yml` points at root `prompts/`, `profiles/`, and `schemas/`; it is valid for repository-root execution but should be tested or explicitly checked.
- There are no copyable CLI example scripts or expected-output files under `examples/`.
- The maintained prompt/profile/schema examples live outside `examples/`; this is usable, but policy prefers copyable examples under `examples/`.
- `local-test/` appears unmaintained and should not be treated as documentation.
Links that are likely stale or need verification:
- Existing links from README to docs should be rewritten after canonical files are created.
- Any references to `docs/config/config-yml.md`, `docs/config/prompt-definitions.md`, `docs/config/profile-definitions.md`, or `docs/config/schema-definitions.md` should be updated after those files are merged.
- References to default config paths must use the implemented search order: `/usr/local/etc/scriptorium/config.yml`, then `/etc/scriptorium/config.yml`.
## Target Documentation Set
### `README.md`
- Audience: users, administrators, operators.
- Purpose: concise orientation and shortest useful command.
- Canonical scope: project purpose, elevator pitch, quickstart, and links.
- Recommended outline: description; why scriptorium exists; shortest useful `scriptorium render` or `scriptorium run` example; documentation links.
- Source of truth: `cmd/scriptorium/main.go`, `internal/adapter/cli/run.go`, `examples/config.yml`, `prompts/generic.markdown_summary.yaml`.
- Acceptance criteria: no full flag reference; no HTTP schema; no prompt/profile field tables; no future work; all links point to existing target docs.
### `docs/cli.md`
- Audience: users, administrators, operators.
- Purpose: complete CLI reference and common workflows.
- Canonical scope: commands, flags, outputs, exit codes, and command examples.
- Recommended outline: shortest useful command; command overview; `run`; `render`; `serve`; flag reference; input and variable mapping syntax; output behavior; exit codes; common workflows.
- Source of truth: `internal/adapter/cli/run.go`, `internal/adapter/cli/run_test.go`, `internal/format/prepared_run.go`, `cmd/scriptorium/main.go`.
- Acceptance criteria: documents real flags only; notes deprecated aliases `--prompt-id` and `--profile-id`; documents that `render` does not currently accept `--schema-dir`; documents stdout/stderr split and exit code `2` for validation failure.
### `docs/config.md`
- Audience: administrators, operators, advanced users.
- Purpose: canonical reference for app config, prompt definitions, profiles, and schemas.
- Canonical scope: all implemented YAML/JSON file formats and precedence rules.
- Recommended outline: config file discovery and precedence; minimal app config; production-oriented app config; app config reference; prompt definition reference; profile definition reference; schema behavior; secrets handling; maintained examples.
- Source of truth: `internal/config/config.go`, `internal/defaults/defaults.go`, `internal/promptdef/filesystem_repository.go`, `internal/profile/filesystem_repository.go`, `internal/validate/standard_validator.go`, repository `prompts/`, `profiles/`, `schemas/`, and config/profile/prompt tests.
- Acceptance criteria: replaces split `docs/config/*.md`; documents strict YAML decoding; documents raw API key rejection; documents `schema_dir` default `.`; documents prompt `content_file` relative to prompt YAML; states `content_type` is metadata only; does not claim operational repair unless a repairer is configured.
### `docs/operations.md`
- Audience: administrators, operators.
- Purpose: operational use of CLI and HTTP service.
- Canonical scope: normal workflow, filesystem expectations, config deployment, secrets, logs/output, validation behavior, and recovery from failed runs.
- Recommended outline: normal run/render workflow; config and library directories; environment variables for API keys; serving HTTP; output and stderr summaries; validation failure handling; no durable state/resume/archive behavior; safe recovery steps.
- Source of truth: `internal/adapter/cli/run.go`, `internal/adapter/http/handler.go`, `internal/config/config.go`, `internal/llm/openai_compatible_client.go`, `internal/usecase/runner.go`.
- Acceptance criteria: makes clear scriptorium does not persist run state; does not invent cleanup/archive/resume; documents that HTTP has no built-in authentication and should be deployed behind trusted controls.
### `docs/troubleshooting.md`
- Audience: administrators, operators.
- Purpose: safe diagnosis and fixes for recurring implemented failure modes.
- Canonical scope: symptoms, likely causes, diagnostics, safe fixes, and links.
- Recommended outline: missing config; missing prompt/profile dirs; unknown flags; prompt/profile load failures; missing input files; template render failures; missing API-key environment values; LLM non-2xx/malformed responses; JSON/schema validation failures; HTTP error codes.
- Source of truth: `internal/adapter/cli/run_test.go`, `internal/adapter/http/handler_test.go`, `internal/config/config_test.go`, `internal/promptdef/repository_test.go`, `internal/profile/repository_test.go`, `internal/validate/standard_validator_test.go`, `internal/llm/openai_compatible_client_test.go`.
- Acceptance criteria: every entry includes symptom, likely cause, diagnostic step, safe fix, and links to canonical CLI/config/operations docs.
### `docs/policy/architecture.md`
- Audience: developers, LLM coding agents.
- Purpose: controlling development architecture and invariants.
- Canonical scope: development principles, boundaries, invariants, non-goals.
- Recommended outline: keep current policy shape; add scriptorium-specific package map or link to `docs/internal/`; clarify no orchestration creep; clarify current adapters.
- Source of truth: existing policy, `internal/` package layout, `architecture.md`.
- Acceptance criteria: remains policy-oriented; does not become user docs; future work stays in roadmap; no stale package names.
### `docs/policy/development.md`
- Audience: developers, LLM coding agents.
- Purpose: contributor workflow and change checklist.
- Canonical scope: repository layout, build/test commands, coding conventions, dependency policy, adding config/CLI/adapters, updating examples/docs.
- Recommended outline: repository layout; common commands; coding conventions; dependency policy; how to add config fields; how to add CLI flags; how to add adapters; how to update examples; documentation expectations.
- Source of truth: `go.mod`, `cmd/scriptorium/main.go`, `internal/adapter/cli/run.go`, `internal/config/config.go`, `docs/policy/architecture.md`, existing tests.
- Acceptance criteria: includes `go test ./...`; references `go build ./cmd/scriptorium`; tells contributors to update docs and tests with behavior changes.
### `docs/internal/runner.md`
- Audience: developers, LLM coding agents.
- Purpose: implemented core prepare/run behavior.
- Canonical scope: `Runner.Prepare`, `Runner.Run`, profile selection, runtime merge, artifact loading, rendering, structured output setup, validation, repair hook boundary.
- Recommended outline: purpose; inputs/outputs; prepare flow; run flow; boundary contracts; failure behavior; tests; invariants.
- Source of truth: `internal/usecase/runner.go`, `internal/usecase/repairer.go`, `internal/usecase/runner_test.go`, `internal/usecase/integration_test.go`.
- Acceptance criteria: states CLI/HTTP currently construct `Runner` without a repairer; documents validation content failures versus runtime validation errors; no provider-specific details except through ports.
### `docs/internal/adapters.md`
- Audience: developers, LLM coding agents.
- Purpose: implemented adapter boundaries.
- Canonical scope: CLI adapter, HTTP adapter, filesystem repositories, artifact reader, prompt renderer, OpenAI-compatible LLM client, validator, prepared-run formatter.
- Recommended outline: adapter map; inputs/outputs; config fields used; external dependencies; failure behavior; tests to inspect.
- Source of truth: `internal/adapter/cli`, `internal/adapter/http`, `internal/promptdef`, `internal/profile`, `internal/artifact`, `internal/prompt`, `internal/llm`, `internal/validate`, `internal/format`.
- Acceptance criteria: documents only implemented adapters; states `inline` and `file` artifact refs are supported and S3 is not; states OpenAI request fields actually sent.
### `docs/integrations/http-api.md`
- Audience: developers, LLM coding agents, API clients.
- Purpose: implemented inbound HTTP contract.
- Canonical scope: `POST /v1/runs`, request/response shape, raw output opt-in, error mapping, validation-failed status behavior.
- Recommended outline: scope; endpoint; request fields; response fields; error responses; validation behavior; security/auth note.
- Source of truth: `internal/adapter/http/dto.go`, `internal/adapter/http/handler.go`, `internal/adapter/http/handler_test.go`.
- Acceptance criteria: no unimplemented render endpoint; no built-in auth claim; unknown JSON fields rejected; raw API key fields rejected by strict JSON.
### `docs/integrations/openai-compatible-chat.md`
- Audience: developers, LLM operators, LLM adapter maintainers.
- Purpose: implemented outbound LLM API contract.
- Canonical scope: OpenAI-compatible chat completions request/response subset and provider-level structured output behavior.
- Recommended outline: endpoint construction; request fields sent; auth header from `api_key_env`; timeout behavior; response expectations; error handling; unsupported profile fields.
- Source of truth: `internal/llm/openai_compatible_client.go`, `internal/llm/openai_compatible_client_test.go`, `internal/usecase/runner.go`.
- Acceptance criteria: says endpoint appends `/chat/completions`; says empty first choice content is malformed; says `reasoning_effort` and `extra_params` are not currently serialized into the outbound request.
### `docs/integrations/narratio.md`
- Audience: developers, LLM coding agents maintaining Narratio integration.
- Purpose: CLI subprocess contract for Narratio.
- Canonical scope: how Narratio should call implemented `scriptorium run` and `scriptorium render`.
- Recommended outline: purpose; assumptions; command shapes; inputs/vars; profile selection; runtime overrides; config behavior; environment handling; output handling; exit statuses; security notes; non-goals.
- Source of truth: `internal/adapter/cli/run.go`, `internal/adapter/cli/run_test.go`, `docs/cli.md`, `docs/config.md`.
- Acceptance criteria: removes future extensions; labels any Narratio-specific prompt IDs as external examples only or removes them; links to canonical CLI/config docs.
## File-by-File Rewrite Guidance
- `README.md`: cover what scriptorium does and show one minimal command. Avoid field tables, complete flag lists, HTTP schema, internal package details, future extensions, and long examples. Link to `docs/cli.md`, `docs/config.md`, `docs/operations.md`, `docs/troubleshooting.md`, and `docs/integrations/`.
- `docs/cli.md`: cover `run`, `render`, `serve`, flags, output behavior, and exit codes. Avoid duplicating prompt/profile YAML field references; link to `docs/config.md`. Inspect CLI parser tests before writing examples. Do not carry forward README's claim that render supports `--schema-dir`.
- `docs/config.md`: cover app config, prompt YAML, profile YAML, and schema behavior. Avoid provider API details except where needed for profile fields; link to OpenAI integration doc. Do not carry forward claims that `repair_attempts` enables repair for normal CLI/HTTP runs unless code later wires a repairer.
- `docs/operations.md`: cover deployed operation and recovery boundaries. Avoid inventing durable state, resume behavior, cleanup, backups, or archives. State that rerunning a command is the recovery model.
- `docs/troubleshooting.md`: use tested errors and behavior. Avoid exposing internal wrapped error details that HTTP intentionally suppresses. Link to CLI/config/operations instead of repeating full references.
- `docs/policy/architecture.md`: preserve policy authority and update only scriptorium-specific facts. Avoid copying the long historical `architecture.md` wholesale. Move future extension ideas to roadmap docs only.
- `docs/policy/development.md`: cover contributor mechanics and how to update behavior safely. Avoid user-facing manuals. Include tests and docs update expectations.
- `docs/internal/runner.md`: explain implemented core flow and invariants. Avoid CLI flag tables and HTTP DTO detail; link to adapter docs.
- `docs/internal/adapters.md`: explain implemented adapter boundaries and tests. Avoid proposing new adapters. Do not imply `ArtifactRefS3` works.
- `docs/integrations/http-api.md`: document only `POST /v1/runs`. Avoid documenting a render/prepare HTTP endpoint.
- `docs/integrations/openai-compatible-chat.md`: document the outbound request subset. Avoid documenting unsupported OpenAI fields or provider-specific options unless code sends them.
- `docs/integrations/narratio.md`: keep it as a subprocess contract. Avoid future work, S3, HTTP-as-primary-path, and undeployed prompt IDs as current examples.
- `architecture.md`: after target docs exist, delete it or replace it with a short pointer to `docs/policy/architecture.md` and `docs/internal/`. Do not leave future notes in this root file.
- `docs/config/*.md`: after `docs/config.md` exists and links are updated, delete these split files or replace them with pointers only if backwards-compatible links are necessary.
## Examples Plan
Existing maintained examples:
- `examples/config.yml`: minimal app config pointing at root prompt/profile/schema libraries. Validity check: `go test ./internal/config ./internal/adapter/cli` and `go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --format json`. Link from README, `docs/config.md`, and `docs/cli.md`.
- `examples/fixtures/transcript.md`: sample transcript input. Validity check: used by `internal/usecase/integration_test.go` and render smoke command. Link from README and `docs/cli.md`.
- `examples/fixtures/glossary.yml`: sample optional glossary input. Validity check: used by `internal/usecase/integration_test.go`. Link from `docs/config.md` and examples section in `docs/cli.md`.
- `prompts/generic.markdown_summary.yaml`: sample markdown prompt. Validity check: render smoke command. Link from config docs until or unless it is moved under `examples/`.
- `prompts/generic.structured_events.yaml` plus `schemas/structured_events.schema.json`: sample JSON-schema prompt. Validity check: `go test ./internal/usecase`. Link from `docs/config.md`.
- `profiles/local-fast.yaml` and `profiles/local-quality.yaml`: sample profiles. Validity check: profile repository tests plus integration test. Link from `docs/config.md`, with a note that `local-quality` requires `SCRIPTORIUM_API_KEY` because it sets `api_key_env`.
Recommended example additions, all based on implemented behavior:
- `examples/render-markdown-summary.sh`: copyable render smoke command using `generic.markdown_summary`. Expected check: run script or equivalent `go run` command exits `0`. Link from README and `docs/cli.md`.
- `examples/http-run.json`: copyable `POST /v1/runs` request body using `inline` or `file` artifact refs. Expected check: parse as JSON and keep aligned with `internal/adapter/http/dto.go`. Link from `docs/integrations/http-api.md`.
- `examples/prompts/`, `examples/profiles/`, `examples/schemas/`: optional future move or mirror of maintained sample libraries. Expected check: update integration tests and `examples/config.yml` together. This is recommended for policy alignment but should be done as its own implementation stage to avoid breaking tests.
Do not document `local-test/` as maintained examples.
## Internal Documentation Plan
### Core runner
- Path: `docs/internal/runner.md`
- Purpose: explain implemented prepare/run lifecycle.
- Inputs and outputs: `domain.RunRequest`, `domain.PreparedRun`, `domain.RunResult`, `domain.GenerateRequest`.
- Boundaries: usecase owns profile selection, runtime merge, artifact resolution orchestration, rendering orchestration, structured output setup, validation, and run metadata; adapters own transport/config parsing.
- Config fields used: none directly; adapters pass resolved repositories, validators, and request values.
- Adapters used: promptdef repository, profile repository, artifact reader, prompt renderer, LLM client, validator, optional injected repairer.
- Failure behavior: invalid request, prompt/profile load, artifact load, render failure, LLM failure, validation runtime failure; validation content failures return a result.
- Tests to inspect before changing: `internal/usecase/runner_test.go`, `internal/usecase/integration_test.go`.
- Architectural invariants: `Run` reuses `Prepare`; no resolved API key values in prepared data; repair attempts bounded and only possible when a repairer is injected; no orchestration creep.
### Adapters and repositories
- Path: `docs/internal/adapters.md`
- Purpose: explain implemented external boundaries.
- Inputs and outputs: CLI args/stdout/stderr/exit codes; HTTP JSON DTOs; YAML prompt/profile/config files; file/inline artifacts; OpenAI-compatible HTTP requests; prepared-run text/JSON output.
- Boundaries: adapters translate external forms into domain requests/results and must not own domain decisions.
- Config fields used: `prompt_dir`, `profile_dir`, `schema_dir`, `server.addr`, `defaults.render_format`; profile `endpoint`, `model`, generation fields, timeout, `api_key_env`.
- Adapters used: CLI, HTTP, filesystem repositories, artifact reader, Go template renderer, OpenAI-compatible client, standard validator, prepared-run formatter.
- Failure behavior: strict YAML/JSON decoding, unknown fields rejected, unsupported artifact refs rejected, LLM non-2xx/malformed responses become errors.
- Tests to inspect before changing: adapter, repository, artifact, renderer, LLM, validator, and formatter tests under `internal/**`.
- Architectural invariants: no raw API keys; no S3 docs until reader exists; OpenAI client sends only implemented request fields.
### Validation and structured output
- Path: include in `docs/internal/runner.md` or create `docs/internal/validation.md` if the section grows.
- Purpose: explain `none`, `basic`, `json`, `json_schema`, schema loading, and provider-level JSON schema request setup.
- Inputs and outputs: `domain.Artifact`, `domain.OutputContract`, `domain.ValidationResult`, `domain.StructuredOutputSpec`.
- Boundaries: validator checks output; runner creates provider-level structured output spec for `json_schema`; OpenAI adapter serializes `response_format`.
- Config fields used: `schema_dir`; prompt `output.schema_path`, `output.validation_mode`, `output.format`.
- Adapters used: standard validator and OpenAI-compatible client.
- Failure behavior: invalid generated JSON is validation failure; missing/invalid schema file is runtime validation error before or during run preparation.
- Tests to inspect before changing: `internal/validate/standard_validator_test.go`, `internal/usecase/runner_test.go`, `internal/llm/openai_compatible_client_test.go`.
- Architectural invariants: schema docs must load before `json_schema` LLM request; schema paths resolve relative to `schema_dir`.
## Integration Documentation Plan
### HTTP API
- Path: `docs/integrations/http-api.md`
- External system or contract: inbound HTTP clients of scriptorium.
- Current usage in scriptorium: `scriptorium serve` exposes `POST /v1/runs`.
- Version or compatibility notes: route is `/v1/runs`; request decoding rejects unknown JSON fields.
- What should be documented: request fields, `file` and `inline` artifact refs, model overrides, raw output opt-in, response shape, error codes, validation-failed `200 OK`, no built-in auth.
- What should not be documented: unimplemented render endpoint, streaming, batch, authentication middleware, remote artifact storage.
### OpenAI-compatible chat completions
- Path: `docs/integrations/openai-compatible-chat.md`
- External system or contract: outbound OpenAI-compatible `/chat/completions` API.
- Current usage in scriptorium: `OpenAICompatibleClient.Generate` posts chat messages and optional JSON schema response format.
- Version or compatibility notes: compatibility is defined by the subset used in code, not by a pinned OpenAI API version.
- What should be documented: endpoint construction, request fields, auth header behavior, timeout behavior, expected response shape, error handling, structured output payload.
- What should not be documented: provider features not serialized by code, retries, streaming, tool calls, reasoning controls, or extra provider params.
### Narratio CLI subprocess
- Path: `docs/integrations/narratio.md`
- External system or contract: Narratio calling scriptorium as a subprocess.
- Current usage in scriptorium: public CLI commands `run` and `render`.
- Version or compatibility notes: contract should be tied to implemented CLI flags and exit codes.
- What should be documented: command construction, config use, input files, vars, profile overrides, timeout override, output paths, stdout/stderr handling, exit status semantics.
- What should not be documented: future S3 support, future HTTP primary integration, unimplemented prompt IDs as current examples, or Narratio stage state internals.
JSON Schema is important but does not need a separate integration doc in the first migration; keep schema behavior in `docs/config.md` and internal validation docs unless compatibility issues require a dedicated page later.
## Recommended Implementation Sequence
### Stage 1: Canonical README, CLI, and Config
- Goal: make user-facing docs accurate and move reference material to canonical homes.
- Files to create/update/delete/move: rewrite `README.md`; create `docs/cli.md`; create `docs/config.md`; leave old `docs/config/*.md` temporarily with pointers or delete them only after links are updated.
- Repository areas to inspect: `cmd/scriptorium/main.go`, `internal/adapter/cli/run.go`, `internal/adapter/cli/run_test.go`, `internal/config`, `internal/promptdef`, `internal/profile`, `internal/validate`, `examples/config.yml`, `prompts/`, `profiles/`, `schemas/`.
- Acceptance criteria: README is short; CLI flags match parser; config paths and precedence match code; no future work outside roadmap; no repair claims beyond implemented behavior.
- Suggested validation commands: `go test ./...`; `go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --format json`; `rg -n "future|planned|may be added|can be added later|S3|streaming|batch" README.md docs/cli.md docs/config.md`.
- One-prompt size: yes, if old split config files are deleted or replaced with pointers in the same change.
### Stage 2: Operations and Troubleshooting
- Goal: document operational behavior and known failure modes.
- Files to create/update/delete/move: create `docs/operations.md`; create `docs/troubleshooting.md`; update README links.
- Repository areas to inspect: CLI and HTTP adapters, config tests, LLM client tests, validator tests, prompt/profile repository tests.
- Acceptance criteria: no invented state/resume/backup behavior; troubleshooting entries are actionable and link to canonical docs; HTTP no-auth caveat is clear.
- Suggested validation commands: `go test ./internal/adapter/cli ./internal/adapter/http ./internal/config ./internal/llm ./internal/validate`; `rg -n "resume|archive|backup|cleanup|state" docs/operations.md docs/troubleshooting.md`.
- One-prompt size: yes.
### Stage 3: Development Policy and Internal Docs
- Goal: give developers and LLM agents accurate package boundaries and invariants.
- Files to create/update/delete/move: create `docs/policy/development.md`; update `docs/policy/architecture.md`; create `docs/internal/runner.md`; create `docs/internal/adapters.md`; optionally create `docs/internal/validation.md`.
- Repository areas to inspect: all `internal/` packages, `go.mod`, root `architecture.md`, tests.
- Acceptance criteria: package names match repository; current adapters only; future extension ideas absent except links to roadmap; repairer hook boundary is accurate.
- Suggested validation commands: `go test ./...`; `rg -n "should expose|may be added|future|S3|batch|streaming|database-backed|additional providers" docs/policy docs/internal`.
- One-prompt size: maybe split into two prompts if `docs/internal/` becomes too large.
### Stage 4: Integration Docs
- Goal: move external contracts out of README and make integration docs precise.
- Files to create/update/delete/move: create `docs/integrations/http-api.md`; create `docs/integrations/openai-compatible-chat.md`; rewrite `docs/integrations/narratio.md`; update README and CLI/config links.
- Repository areas to inspect: `internal/adapter/http`, `internal/llm`, `internal/usecase`, CLI tests, HTTP tests, LLM tests.
- Acceptance criteria: HTTP docs cover only `POST /v1/runs`; OpenAI docs cover only serialized fields; Narratio docs include only implemented CLI integration and no future notes.
- Suggested validation commands: `go test ./internal/adapter/http ./internal/llm ./internal/adapter/cli`; `rg -n "POST /v1/renders|S3|future|later|batch|streaming" docs/integrations`.
- One-prompt size: yes.
### Stage 5: Examples and Link Cleanup
- Goal: make examples policy-compliant and verify links after moves.
- Files to create/update/delete/move: optionally add `examples/render-markdown-summary.sh`; optionally add `examples/http-run.json`; decide whether to move or mirror `prompts/`, `profiles/`, `schemas/` under `examples/`; delete or exclude `local-test/`; remove or replace root `architecture.md`; delete obsolete `docs/config/*.md` if not already removed.
- Repository areas to inspect: `examples/`, `prompts/`, `profiles/`, `schemas/`, `internal/usecase/integration_test.go`, docs links.
- Acceptance criteria: examples are copyable, secret-free, and tested where practical; no links to deleted docs; no maintained docs link to `local-test/`.
- Suggested validation commands: `go test ./...`; `go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --format text`; `rg -n "docs/config/|architecture.md|local-test|dnd.structured_events|dnd.glossary_suggestions|dnd.player_summary" README.md docs examples`.
- One-prompt size: split if moving prompt/profile/schema assets because tests and paths must be updated carefully.
## Validation Plan
- Run `go test ./...` after documentation changes that touch examples, paths, command examples, or config references.
- Smoke-test the documented render quickstart with `go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --format json`.
- If documenting JSON-schema render/run examples, set `SCRIPTORIUM_API_KEY` or use a profile without `api_key_env`; `Runner.Prepare` validates the named environment variable.
- Validate CLI flags against `internal/adapter/cli/run.go` and parser tests, especially `render` lacking `--schema-dir` and `serve` rejecting runtime override flags.
- Validate app config examples against `internal/config/config.go` strict YAML decoding.
- Validate prompt/profile examples against `internal/promptdef/filesystem_repository.go` and `internal/profile/filesystem_repository.go`.
- Validate HTTP request examples against `internal/adapter/http/dto.go` strict JSON decoding.
- Run grep checks for stale or roadmap-only terms outside `docs/roadmap/`: `future`, `planned`, `may be added`, `can be added later`, `S3`, `streaming`, `batch`, `database-backed`, `render endpoint`, and prompt IDs not present in `prompts/`.
- Run grep checks for stale paths after file moves: `docs/config/`, `architecture.md`, and `local-test`.
- No automated documentation link checker is currently configured; perform manual link review or add a link checker in a separate roadmap item if desired.
## Open Questions
No open questions block the documentation migration. The recommended path is to document the current implementation conservatively, move future ideas into `docs/roadmap/`, and avoid claiming production behavior for hooks that are present in code but not wired into CLI or HTTP adapters.

View File

@@ -1,6 +1,6 @@
prompt_dir: ./prompts prompt_dir: ./examples/prompts
profile_dir: ./profiles profile_dir: ./examples/profiles
schema_dir: ./schemas schema_dir: ./examples/schemas
server: server:
addr: :8080 addr: :8080

View File

@@ -81,6 +81,14 @@ type serveConfig struct {
schemaDir string schemaDir string
} }
type commonCommandSettings struct {
promptDir string
profileDir string
schemaDir string
serverAddr string
defaultRenderFormat renderformat.PreparedRunOutputFormat
}
type listFlag []string type listFlag []string
func (l *listFlag) String() string { func (l *listFlag) String() string {
@@ -125,22 +133,13 @@ func runCommand(args []string, stdout, stderr io.Writer) int {
return ExitRuntimeError return ExitRuntimeError
} }
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{ llmClient, err := newOpenAIClient()
Timeout: defaults.LLMRequestTimeoutDefault,
})
if err != nil { if err != nil {
fmt.Fprintf(stderr, "llm client error: %v\n", err) fmt.Fprintf(stderr, "llm client error: %v\n", err)
return ExitRuntimeError return ExitRuntimeError
} }
runner := usecase.NewRunner( runner := newRunner(cfg.promptDir, cfg.profileDir, cfg.schemaDir, llmClient)
promptdef.NewFilesystemRepository(cfg.promptDir),
profile.NewFilesystemRepository(cfg.profileDir),
artifactadapter.NewCompositeReader(),
prompt.NewGoRenderer(),
llmClient,
validate.NewStandardValidator(cfg.schemaDir),
)
res, runErr := runner.Run(context.Background(), req) res, runErr := runner.Run(context.Background(), req)
if runErr != nil { if runErr != nil {
@@ -170,14 +169,7 @@ func renderCommand(args []string, stdout, stderr io.Writer) int {
return ExitRuntimeError return ExitRuntimeError
} }
runner := usecase.NewRunner( runner := newRunner(cfg.promptDir, cfg.profileDir, cfg.schemaDir, nil)
promptdef.NewFilesystemRepository(cfg.promptDir),
profile.NewFilesystemRepository(cfg.profileDir),
artifactadapter.NewCompositeReader(),
prompt.NewGoRenderer(),
nil,
validate.NewStandardValidator(cfg.schemaDir),
)
prepared, prepErr := runner.Prepare(context.Background(), req) prepared, prepErr := runner.Prepare(context.Background(), req)
if prepErr != nil { if prepErr != nil {
@@ -205,22 +197,13 @@ func serveCommand(args []string, stderr io.Writer) int {
return ExitRuntimeError return ExitRuntimeError
} }
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{ llmClient, err := newOpenAIClient()
Timeout: defaults.LLMRequestTimeoutDefault,
})
if err != nil { if err != nil {
fmt.Fprintf(stderr, "llm client error: %v\n", err) fmt.Fprintf(stderr, "llm client error: %v\n", err)
return ExitRuntimeError return ExitRuntimeError
} }
runner := usecase.NewRunner( runner := newRunner(cfg.promptDir, cfg.profileDir, cfg.schemaDir, llmClient)
promptdef.NewFilesystemRepository(cfg.promptDir),
profile.NewFilesystemRepository(cfg.profileDir),
artifactadapter.NewCompositeReader(),
prompt.NewGoRenderer(),
llmClient,
validate.NewStandardValidator(cfg.schemaDir),
)
h := httpadapter.NewHandler(runner) h := httpadapter.NewHandler(runner)
srv := &http.Server{ srv := &http.Server{
@@ -307,7 +290,7 @@ func parseServeArgs(args []string) (*serveConfig, error) {
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args()) return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
} }
settings, err := resolveAppSettings(fs, cfg.configPath, appconfig.CLIOverrides{ settings, err := resolveCommonSettings(fs, cfg.configPath, appconfig.CLIOverrides{
PromptDir: cfg.promptDirIfSet(fs), PromptDir: cfg.promptDirIfSet(fs),
ProfileDir: cfg.profileDirIfSet(fs), ProfileDir: cfg.profileDirIfSet(fs),
SchemaDir: cfg.schemaDirIfSet(fs), SchemaDir: cfg.schemaDirIfSet(fs),
@@ -317,16 +300,13 @@ func parseServeArgs(args []string) (*serveConfig, error) {
return nil, err return nil, err
} }
cfg.promptDir = settings.PromptDir cfg.promptDir = settings.promptDir
cfg.profileDir = settings.ProfileDir cfg.profileDir = settings.profileDir
cfg.schemaDir = settings.SchemaDir cfg.schemaDir = settings.schemaDir
cfg.addr = settings.ServerAddr cfg.addr = settings.serverAddr
if strings.TrimSpace(cfg.promptDir) == "" { if err := validateRequiredLibraryDirs(cfg.promptDir, cfg.profileDir); err != nil {
return nil, errors.New(errPromptDirRequired) return nil, err
}
if strings.TrimSpace(cfg.profileDir) == "" {
return nil, errors.New(errProfileDirRequired)
} }
cfg.promptDir = filepath.Clean(cfg.promptDir) cfg.promptDir = filepath.Clean(cfg.promptDir)
@@ -359,7 +339,7 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
return fmt.Errorf("unexpected positional args: %v", fs.Args()) return fmt.Errorf("unexpected positional args: %v", fs.Args())
} }
settings, err := resolveAppSettings(fs, cfg.configPath, appconfig.CLIOverrides{ settings, err := resolveCommonSettings(fs, cfg.configPath, appconfig.CLIOverrides{
PromptDir: cfg.promptDirIfSet(fs), PromptDir: cfg.promptDirIfSet(fs),
ProfileDir: cfg.profileDirIfSet(fs), ProfileDir: cfg.profileDirIfSet(fs),
SchemaDir: cfg.schemaDirIfSet(fs), SchemaDir: cfg.schemaDirIfSet(fs),
@@ -368,16 +348,13 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
return err return err
} }
cfg.promptDir = settings.PromptDir cfg.promptDir = settings.promptDir
cfg.profileDir = settings.ProfileDir cfg.profileDir = settings.profileDir
cfg.schemaDir = settings.SchemaDir cfg.schemaDir = settings.schemaDir
cfg.defaultRenderFormat = settings.DefaultRenderFormat cfg.defaultRenderFormat = settings.defaultRenderFormat
if strings.TrimSpace(cfg.promptDir) == "" { if err := validateRequiredLibraryDirs(cfg.promptDir, cfg.profileDir); err != nil {
return errors.New(errPromptDirRequired) return err
}
if strings.TrimSpace(cfg.profileDir) == "" {
return errors.New(errProfileDirRequired)
} }
if strings.TrimSpace(cfg.promptID) == "" { if strings.TrimSpace(cfg.promptID) == "" {
return errors.New("--prompt is required") return errors.New("--prompt is required")
@@ -476,6 +453,47 @@ func resolveAppSettings(fs *flag.FlagSet, configPath string, overrides appconfig
return merged, nil return merged, nil
} }
func resolveCommonSettings(fs *flag.FlagSet, configPath string, overrides appconfig.CLIOverrides) (commonCommandSettings, error) {
settings, err := resolveAppSettings(fs, configPath, overrides)
if err != nil {
return commonCommandSettings{}, err
}
return commonCommandSettings{
promptDir: settings.PromptDir,
profileDir: settings.ProfileDir,
schemaDir: settings.SchemaDir,
serverAddr: settings.ServerAddr,
defaultRenderFormat: settings.DefaultRenderFormat,
}, nil
}
func validateRequiredLibraryDirs(promptDir, profileDir string) error {
if strings.TrimSpace(promptDir) == "" {
return errors.New(errPromptDirRequired)
}
if strings.TrimSpace(profileDir) == "" {
return errors.New(errProfileDirRequired)
}
return nil
}
func newRunner(promptDir, profileDir, schemaDir string, llmClient llm.Client) *usecase.Runner {
return usecase.NewRunner(
promptdef.NewFilesystemRepository(promptDir),
profile.NewFilesystemRepository(profileDir),
artifactadapter.NewCompositeReader(),
prompt.NewGoRenderer(),
llmClient,
validate.NewStandardValidator(schemaDir),
)
}
func newOpenAIClient() (*llm.OpenAICompatibleClient, error) {
return llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
Timeout: defaults.LLMRequestTimeoutDefault,
})
}
func buildRunRequestFromConfig(cfg *runConfig) (domain.RunRequest, error) { func buildRunRequestFromConfig(cfg *runConfig) (domain.RunRequest, error) {
inputMappings, err := parseMappings(cfg.inputRaw, false) inputMappings, err := parseMappings(cfg.inputRaw, false)
if err != nil { if err != nil {

View File

@@ -11,6 +11,7 @@ import (
"net/http/httptest" "net/http/httptest"
"os" "os"
"path/filepath" "path/filepath"
"reflect"
"strings" "strings"
"sync/atomic" "sync/atomic"
"testing" "testing"
@@ -410,6 +411,28 @@ defaults:
} }
} }
func TestParseRenderArgsExplicitFormatOverridesConfigDefaultFormat(t *testing.T) {
configPath := writeAppConfigFile(t, `
prompt_dir: ./from-config/prompts
profile_dir: ./from-config/profiles
defaults:
render_format: json
`)
cfg, err := parseRenderArgs([]string{
"--config", configPath,
"--prompt", "p",
"--input", "a=b",
"--format", "text",
})
if err != nil {
t.Fatalf("expected valid args, got %v", err)
}
if cfg.outputFormat != renderformat.PreparedRunFormatText {
t.Fatalf("expected explicit --format text to override config default, got %q", cfg.outputFormat)
}
}
func TestParseServeArgsWithExplicitConfigLoadsSettingsAndCLIAddrOverrides(t *testing.T) { func TestParseServeArgsWithExplicitConfigLoadsSettingsAndCLIAddrOverrides(t *testing.T) {
configPath := writeAppConfigFile(t, ` configPath := writeAppConfigFile(t, `
prompt_dir: ./from-config/prompts prompt_dir: ./from-config/prompts
@@ -471,6 +494,59 @@ server:
} }
} }
func TestRunAndRenderBuildEquivalentRuntimeOverrideRequestsForSharedFlags(t *testing.T) {
runCfg, err := parseRunArgs([]string{
"--prompt-dir", "./prompts",
"--profile-dir", "./profiles",
"--prompt", "prompt-1",
"--profile", "profile-1",
"--input", "transcript=./transcript.md",
"--var", "session_date=2026-05-01",
"--llm-base-url", "http://localhost:8000/v1",
"--model", "model-x",
"--temperature", "0.8",
"--max-tokens", "123",
"--top-p", "0.6",
"--timeout", "90s",
"--api-key-env", "SCRIPTORIUM_API_KEY",
})
if err != nil {
t.Fatalf("expected valid run args, got %v", err)
}
renderCfg, err := parseRenderArgs([]string{
"--prompt-dir", "./prompts",
"--profile-dir", "./profiles",
"--prompt", "prompt-1",
"--profile", "profile-1",
"--input", "transcript=./transcript.md",
"--var", "session_date=2026-05-01",
"--llm-base-url", "http://localhost:8000/v1",
"--model", "model-x",
"--temperature", "0.8",
"--max-tokens", "123",
"--top-p", "0.6",
"--timeout", "90s",
"--api-key-env", "SCRIPTORIUM_API_KEY",
})
if err != nil {
t.Fatalf("expected valid render args, got %v", err)
}
runReq, err := buildRunRequestFromConfig(runCfg)
if err != nil {
t.Fatalf("expected run request build success, got %v", err)
}
renderReq, err := buildRunRequestFromConfig(&renderCfg.runConfig)
if err != nil {
t.Fatalf("expected render request build success, got %v", err)
}
if !reflect.DeepEqual(runReq, renderReq) {
t.Fatalf("expected run/render shared flag requests to match.\nrun=%#v\nrender=%#v", runReq, renderReq)
}
}
func TestParseRunArgsFailsClearlyWhenNoEffectivePromptDir(t *testing.T) { func TestParseRunArgsFailsClearlyWhenNoEffectivePromptDir(t *testing.T) {
configPath := writeAppConfigFile(t, ` configPath := writeAppConfigFile(t, `
profile_dir: ./profiles profile_dir: ./profiles
@@ -586,42 +662,29 @@ func TestRunCommandVarsOptional(t *testing.T) {
} }
func TestRunCommandSucceedsWithPromptAndProfileDirsFromConfig(t *testing.T) { func TestRunCommandSucceedsWithPromptAndProfileDirsFromConfig(t *testing.T) {
tmp := t.TempDir() lib := newCLITestLibrary(t)
promptDir := filepath.Join(tmp, "prompts") inputPath := lib.writeInputFile(t, "transcript.md", "hello")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
t.Fatal(err)
}
ts := newTestLLMServer("from-config-dirs", nil) ts := newTestLLMServer("from-config-dirs", nil)
defer ts.Close() defer ts.Close()
writePromptFile(t, promptDir, "prompt.default", "local-default") writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
writeProfileFile(t, profileDir, "local-default", ts.URL+"/v1", "profile-model") writeProfileFile(t, lib.profileDir, "local-default", ts.URL+"/v1", "profile-model")
configPath := writeAppConfigFile(t, fmt.Sprintf(` configPath := writeAppConfigFile(t, fmt.Sprintf(`
prompt_dir: %s prompt_dir: %s
profile_dir: %s profile_dir: %s
`, promptDir, profileDir)) `, lib.promptDir, lib.profileDir))
var stdout bytes.Buffer code, stdout, stderr := runCLICommand(t, runCommand, []string{
var stderr bytes.Buffer
code := runCommand([]string{
"--config", configPath, "--config", configPath,
"--prompt", "prompt.default", "--prompt", "prompt.default",
"--input", "transcript=" + inputPath, "--input", "transcript=" + inputPath,
}, &stdout, &stderr) })
if code != ExitOK { if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String()) t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
} }
if stdout.String() != "from-config-dirs" { if stdout != "from-config-dirs" {
t.Fatalf("unexpected stdout output: %q", stdout.String()) t.Fatalf("unexpected stdout output: %q", stdout)
} }
} }
@@ -630,28 +693,15 @@ func TestRenderCommandDefaultFormatTextIncludesPreparedDetailsAndNoSecrets(t *te
const secret = "super-secret-render-key" const secret = "super-secret-render-key"
t.Setenv(envName, secret) t.Setenv(envName, secret)
tmp := t.TempDir() lib := newCLITestLibrary(t)
promptDir := filepath.Join(tmp, "prompts") inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
t.Fatal(err)
}
writePromptFileWithTemplate(t, promptDir, "prompt.render", "local-default", "Date {{.session_date}} - Summarize: {{input \"transcript\"}}") writePromptFileWithTemplate(t, lib.promptDir, "prompt.render", "local-default", "Date {{.session_date}} - Summarize: {{input \"transcript\"}}")
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model") writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
var stdout bytes.Buffer code, stdout, stderr := runCLICommand(t, renderCommand, []string{
var stderr bytes.Buffer "--prompt-dir", lib.promptDir,
code := renderCommand([]string{ "--profile-dir", lib.profileDir,
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
"--prompt", "prompt.render", "--prompt", "prompt.render",
"--profile", "local-default", "--profile", "local-default",
"--input", "transcript=" + inputPath, "--input", "transcript=" + inputPath,
@@ -663,15 +713,15 @@ func TestRenderCommandDefaultFormatTextIncludesPreparedDetailsAndNoSecrets(t *te
"--top-p", "0.2", "--top-p", "0.2",
"--timeout", "20s", "--timeout", "20s",
"--api-key-env", envName, "--api-key-env", envName,
}, &stdout, &stderr) })
if code != ExitOK { if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String()) t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
} }
if stderr.Len() != 0 { if stderr != "" {
t.Fatalf("expected empty stderr on success, got %q", stderr.String()) t.Fatalf("expected empty stderr on success, got %q", stderr)
} }
out := stdout.String() out := stdout
for _, want := range []string{ for _, want := range []string{
"prompt: prompt.render", "prompt: prompt.render",
"selected_profile_id: local-default", "selected_profile_id: local-default",
@@ -698,111 +748,72 @@ func TestRenderCommandDefaultFormatTextIncludesPreparedDetailsAndNoSecrets(t *te
} }
func TestRenderCommandSucceedsWithPromptAndProfileDirsFromConfig(t *testing.T) { func TestRenderCommandSucceedsWithPromptAndProfileDirsFromConfig(t *testing.T) {
tmp := t.TempDir() lib := newCLITestLibrary(t)
promptDir := filepath.Join(tmp, "prompts") inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
t.Fatal(err)
}
writePromptFile(t, promptDir, "prompt.render", "local-default") writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model") writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
configPath := writeAppConfigFile(t, fmt.Sprintf(` configPath := writeAppConfigFile(t, fmt.Sprintf(`
prompt_dir: %s prompt_dir: %s
profile_dir: %s profile_dir: %s
`, promptDir, profileDir)) `, lib.promptDir, lib.profileDir))
var stdout bytes.Buffer code, stdout, stderr := runCLICommand(t, renderCommand, []string{
var stderr bytes.Buffer
code := renderCommand([]string{
"--config", configPath, "--config", configPath,
"--prompt", "prompt.render", "--prompt", "prompt.render",
"--input", "transcript=" + inputPath, "--input", "transcript=" + inputPath,
}, &stdout, &stderr) })
if code != ExitOK { if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String()) t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
} }
if !strings.Contains(stdout.String(), "prompt: prompt.render") { if !strings.Contains(stdout, "prompt: prompt.render") {
t.Fatalf("expected rendered output, got %q", stdout.String()) t.Fatalf("expected rendered output, got %q", stdout)
} }
} }
func TestRenderCommandExplicitTextFormatWorks(t *testing.T) { func TestRenderCommandExplicitTextFormatWorks(t *testing.T) {
tmp := t.TempDir() lib := newCLITestLibrary(t)
promptDir := filepath.Join(tmp, "prompts") inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
t.Fatal(err)
}
writePromptFile(t, promptDir, "prompt.render", "local-default") writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model") writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
var stdout bytes.Buffer code, stdout, stderr := runCLICommand(t, renderCommand, []string{
var stderr bytes.Buffer "--prompt-dir", lib.promptDir,
code := renderCommand([]string{ "--profile-dir", lib.profileDir,
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
"--prompt", "prompt.render", "--prompt", "prompt.render",
"--input", "transcript=" + inputPath, "--input", "transcript=" + inputPath,
"--format", "text", "--format", "text",
}, &stdout, &stderr) })
if code != ExitOK { if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String()) t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
} }
if !strings.Contains(stdout.String(), "prompt: prompt.render") { if !strings.Contains(stdout, "prompt: prompt.render") {
t.Fatalf("expected text output for explicit --format text, got %q", stdout.String()) t.Fatalf("expected text output for explicit --format text, got %q", stdout)
} }
} }
func TestRenderCommandExplicitJSONFormatOutputsValidJSON(t *testing.T) { func TestRenderCommandExplicitJSONFormatOutputsValidJSON(t *testing.T) {
tmp := t.TempDir() lib := newCLITestLibrary(t)
promptDir := filepath.Join(tmp, "prompts") inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
t.Fatal(err)
}
writePromptFile(t, promptDir, "prompt.render", "local-default") writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model") writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
var stdout bytes.Buffer code, stdout, stderr := runCLICommand(t, renderCommand, []string{
var stderr bytes.Buffer "--prompt-dir", lib.promptDir,
code := renderCommand([]string{ "--profile-dir", lib.profileDir,
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
"--prompt", "prompt.render", "--prompt", "prompt.render",
"--input", "transcript=" + inputPath, "--input", "transcript=" + inputPath,
"--format", "json", "--format", "json",
}, &stdout, &stderr) })
if code != ExitOK { if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String()) t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
} }
var payload map[string]any var payload map[string]any
if err := json.Unmarshal(stdout.Bytes(), &payload); err != nil { if err := json.Unmarshal([]byte(stdout), &payload); err != nil {
t.Fatalf("expected valid json output, got %v\nbody=%s", err, stdout.String()) t.Fatalf("expected valid json output, got %v\nbody=%s", err, stdout)
} }
if payload["prompt_id"] != "prompt.render" { if payload["prompt_id"] != "prompt.render" {
t.Fatalf("expected prompt_id, got %#v", payload["prompt_id"]) t.Fatalf("expected prompt_id, got %#v", payload["prompt_id"])
@@ -834,38 +845,25 @@ func TestRenderCommandUnknownFormatFailsClearly(t *testing.T) {
} }
func TestRenderCommandOutWritesToFile(t *testing.T) { func TestRenderCommandOutWritesToFile(t *testing.T) {
tmp := t.TempDir() lib := newCLITestLibrary(t)
promptDir := filepath.Join(tmp, "prompts") inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
profileDir := filepath.Join(tmp, "profiles") outPath := filepath.Join(lib.rootDir, "render.txt")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
t.Fatal(err)
}
outPath := filepath.Join(tmp, "render.txt")
writePromptFile(t, promptDir, "prompt.render", "local-default") writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model") writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
var stdout bytes.Buffer code, stdout, stderr := runCLICommand(t, renderCommand, []string{
var stderr bytes.Buffer "--prompt-dir", lib.promptDir,
code := renderCommand([]string{ "--profile-dir", lib.profileDir,
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
"--prompt", "prompt.render", "--prompt", "prompt.render",
"--input", "transcript=" + inputPath, "--input", "transcript=" + inputPath,
"--out", outPath, "--out", outPath,
}, &stdout, &stderr) })
if code != ExitOK { if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String()) t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
} }
if stdout.Len() != 0 { if stdout != "" {
t.Fatalf("expected empty stdout when --out is set, got %q", stdout.String()) t.Fatalf("expected empty stdout when --out is set, got %q", stdout)
} }
out, err := os.ReadFile(outPath) out, err := os.ReadFile(outPath)
if err != nil { if err != nil {
@@ -877,36 +875,23 @@ func TestRenderCommandOutWritesToFile(t *testing.T) {
} }
func TestRenderCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) { func TestRenderCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
tmp := t.TempDir() lib := newCLITestLibrary(t)
promptDir := filepath.Join(tmp, "prompts") inputPath := lib.writeInputFile(t, "transcript.md", "hello")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
t.Fatal(err)
}
writePromptFile(t, promptDir, "prompt.default", "local-default") writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model") writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model")
var stdout bytes.Buffer code, stdout, stderr := runCLICommand(t, renderCommand, []string{
var stderr bytes.Buffer "--prompt-dir", lib.promptDir,
code := renderCommand([]string{ "--profile-dir", lib.profileDir,
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
"--prompt", "prompt.default", "--prompt", "prompt.default",
"--input", "transcript=" + inputPath, "--input", "transcript=" + inputPath,
}, &stdout, &stderr) })
if code != ExitOK { if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String()) t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
} }
out := stdout.String() out := stdout
if !strings.Contains(out, "selected_profile_id: local-default") { if !strings.Contains(out, "selected_profile_id: local-default") {
t.Fatalf("expected prompt default profile in output, got %q", out) t.Fatalf("expected prompt default profile in output, got %q", out)
} }
@@ -916,38 +901,25 @@ func TestRenderCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
} }
func TestRenderCommandExplicitProfileOverridesPromptDefault(t *testing.T) { func TestRenderCommandExplicitProfileOverridesPromptDefault(t *testing.T) {
tmp := t.TempDir() lib := newCLITestLibrary(t)
promptDir := filepath.Join(tmp, "prompts") inputPath := lib.writeInputFile(t, "transcript.md", "hello")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
t.Fatal(err)
}
writePromptFile(t, promptDir, "prompt.default", "local-default") writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model") writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model")
writeProfileFile(t, profileDir, "quality", "http://127.0.0.1:1/v1", "quality-model") writeProfileFile(t, lib.profileDir, "quality", "http://127.0.0.1:1/v1", "quality-model")
var stdout bytes.Buffer code, stdout, stderr := runCLICommand(t, renderCommand, []string{
var stderr bytes.Buffer "--prompt-dir", lib.promptDir,
code := renderCommand([]string{ "--profile-dir", lib.profileDir,
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
"--prompt", "prompt.default", "--prompt", "prompt.default",
"--profile", "quality", "--profile", "quality",
"--input", "transcript=" + inputPath, "--input", "transcript=" + inputPath,
}, &stdout, &stderr) })
if code != ExitOK { if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String()) t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
} }
out := stdout.String() out := stdout
if !strings.Contains(out, "selected_profile_id: quality") { if !strings.Contains(out, "selected_profile_id: quality") {
t.Fatalf("expected explicit profile in output, got %q", out) t.Fatalf("expected explicit profile in output, got %q", out)
} }
@@ -957,104 +929,67 @@ func TestRenderCommandExplicitProfileOverridesPromptDefault(t *testing.T) {
} }
func TestRunCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) { func TestRunCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
tmp := t.TempDir() lib := newCLITestLibrary(t)
promptDir := filepath.Join(tmp, "prompts") inputPath := lib.writeInputFile(t, "transcript.md", "hello")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
t.Fatal(err)
}
ts := newTestLLMServer("default-output", nil) ts := newTestLLMServer("default-output", nil)
defer ts.Close() defer ts.Close()
writePromptFile(t, promptDir, "prompt.default", "local-default") writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
writeProfileFile(t, profileDir, "local-default", ts.URL+"/v1", "profile-model") writeProfileFile(t, lib.profileDir, "local-default", ts.URL+"/v1", "profile-model")
var stdout bytes.Buffer code, stdout, stderr := runCLICommand(t, runCommand, []string{
var stderr bytes.Buffer "--prompt-dir", lib.promptDir,
code := runCommand([]string{ "--profile-dir", lib.profileDir,
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
"--prompt", "prompt.default", "--prompt", "prompt.default",
"--input", "transcript=" + inputPath, "--input", "transcript=" + inputPath,
}, &stdout, &stderr) })
if code != ExitOK { if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String()) t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
} }
if stdout.String() != "default-output" { if stdout != "default-output" {
t.Fatalf("unexpected stdout output: %q", stdout.String()) t.Fatalf("unexpected stdout output: %q", stdout)
} }
if !strings.Contains(stderr.String(), "selected_profile=local-default") { if !strings.Contains(stderr, "selected_profile=local-default") {
t.Fatalf("expected selected profile in summary, got %q", stderr.String()) t.Fatalf("expected selected profile in summary, got %q", stderr)
} }
} }
func TestRunCommandExplicitProfileOverridesPromptDefault(t *testing.T) { func TestRunCommandExplicitProfileOverridesPromptDefault(t *testing.T) {
tmp := t.TempDir() lib := newCLITestLibrary(t)
promptDir := filepath.Join(tmp, "prompts") inputPath := lib.writeInputFile(t, "transcript.md", "hello")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
t.Fatal(err)
}
defaultServer := newTestLLMServer("from-default", nil) defaultServer := newTestLLMServer("from-default", nil)
defer defaultServer.Close() defer defaultServer.Close()
overrideServer := newTestLLMServer("from-override", nil) overrideServer := newTestLLMServer("from-override", nil)
defer overrideServer.Close() defer overrideServer.Close()
writePromptFile(t, promptDir, "prompt.default", "local-default") writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
writeProfileFile(t, profileDir, "local-default", defaultServer.URL+"/v1", "default-model") writeProfileFile(t, lib.profileDir, "local-default", defaultServer.URL+"/v1", "default-model")
writeProfileFile(t, profileDir, "quality", overrideServer.URL+"/v1", "quality-model") writeProfileFile(t, lib.profileDir, "quality", overrideServer.URL+"/v1", "quality-model")
var stdout bytes.Buffer code, stdout, stderr := runCLICommand(t, runCommand, []string{
var stderr bytes.Buffer "--prompt-dir", lib.promptDir,
code := runCommand([]string{ "--profile-dir", lib.profileDir,
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
"--prompt", "prompt.default", "--prompt", "prompt.default",
"--profile", "quality", "--profile", "quality",
"--input", "transcript=" + inputPath, "--input", "transcript=" + inputPath,
}, &stdout, &stderr) })
if code != ExitOK { if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String()) t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
} }
if stdout.String() != "from-override" { if stdout != "from-override" {
t.Fatalf("expected explicit profile output, got %q", stdout.String()) t.Fatalf("expected explicit profile output, got %q", stdout)
} }
if !strings.Contains(stderr.String(), "selected_profile=quality") { if !strings.Contains(stderr, "selected_profile=quality") {
t.Fatalf("expected selected profile quality, got %q", stderr.String()) t.Fatalf("expected selected profile quality, got %q", stderr)
} }
} }
func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) { func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
tmp := t.TempDir() lib := newCLITestLibrary(t)
promptDir := filepath.Join(tmp, "prompts") inputPath := lib.writeInputFile(t, "transcript.md", "hello")
profileDir := filepath.Join(tmp, "profiles")
if err := os.MkdirAll(promptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(profileDir, 0o755); err != nil {
t.Fatal(err)
}
inputPath := filepath.Join(tmp, "transcript.md")
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
t.Fatal(err)
}
var baseHits int32 var baseHits int32
baseServer := newTestLLMServer("base", &baseHits) baseServer := newTestLLMServer("base", &baseHits)
@@ -1071,14 +1006,12 @@ func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
})) }))
defer overrideServer.Close() defer overrideServer.Close()
writePromptFile(t, promptDir, "prompt.default", "local-default") writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
writeProfileFile(t, profileDir, "local-default", baseServer.URL+"/v1", "profile-model") writeProfileFile(t, lib.profileDir, "local-default", baseServer.URL+"/v1", "profile-model")
var stdout bytes.Buffer code, stdout, stderr := runCLICommand(t, runCommand, []string{
var stderr bytes.Buffer "--prompt-dir", lib.promptDir,
code := runCommand([]string{ "--profile-dir", lib.profileDir,
"--prompt-dir", promptDir,
"--profile-dir", profileDir,
"--prompt", "prompt.default", "--prompt", "prompt.default",
"--input", "transcript=" + inputPath, "--input", "transcript=" + inputPath,
"--llm-base-url", overrideServer.URL + "/v1", "--llm-base-url", overrideServer.URL + "/v1",
@@ -1087,9 +1020,9 @@ func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
"--max-tokens", "55", "--max-tokens", "55",
"--top-p", "0.2", "--top-p", "0.2",
"--timeout", "20s", "--timeout", "20s",
}, &stdout, &stderr) })
if code != ExitOK { if code != ExitOK {
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String()) t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
} }
if atomic.LoadInt32(&baseHits) != 0 { if atomic.LoadInt32(&baseHits) != 0 {
t.Fatalf("expected base profile endpoint not to be hit, got %d", baseHits) t.Fatalf("expected base profile endpoint not to be hit, got %d", baseHits)
@@ -1097,8 +1030,8 @@ func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
if atomic.LoadInt32(&overrideHits) != 1 { if atomic.LoadInt32(&overrideHits) != 1 {
t.Fatalf("expected override endpoint to be hit once, got %d", overrideHits) t.Fatalf("expected override endpoint to be hit once, got %d", overrideHits)
} }
if stdout.String() != "override" { if stdout != "override" {
t.Fatalf("unexpected stdout output: %q", stdout.String()) t.Fatalf("unexpected stdout output: %q", stdout)
} }
if !strings.Contains(observedBody, `"model":"override-model"`) { if !strings.Contains(observedBody, `"model":"override-model"`) {
t.Fatalf("expected override model in request body, got %s", observedBody) t.Fatalf("expected override model in request body, got %s", observedBody)
@@ -1133,6 +1066,49 @@ func TestWriteOutputAndSummaryUseSeparateWriters(t *testing.T) {
} }
} }
type cliTestLibrary struct {
rootDir string
promptDir string
profileDir string
}
func newCLITestLibrary(t *testing.T) *cliTestLibrary {
t.Helper()
root := t.TempDir()
lib := &cliTestLibrary{
rootDir: root,
promptDir: filepath.Join(root, "prompts"),
profileDir: filepath.Join(root, "profiles"),
}
if err := os.MkdirAll(lib.promptDir, 0o755); err != nil {
t.Fatalf("failed to create prompt fixture directory: %v", err)
}
if err := os.MkdirAll(lib.profileDir, 0o755); err != nil {
t.Fatalf("failed to create profile fixture directory: %v", err)
}
return lib
}
func (l *cliTestLibrary) writeInputFile(t *testing.T, name, body string) string {
t.Helper()
path := filepath.Join(l.rootDir, name)
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatalf("failed to create input fixture directory: %v", err)
}
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
t.Fatalf("failed to write input fixture: %v", err)
}
return path
}
func runCLICommand(t *testing.T, command func([]string, io.Writer, io.Writer) int, args []string) (int, string, string) {
t.Helper()
var stdout bytes.Buffer
var stderr bytes.Buffer
code := command(args, &stdout, &stderr)
return code, stdout.String(), stderr.String()
}
func writePromptFile(t *testing.T, dir, id, defaultProfile string) { func writePromptFile(t *testing.T, dir, id, defaultProfile string) {
t.Helper() t.Helper()
writePromptFileWithTemplate(t, dir, id, defaultProfile, "Summarize: {{input \"transcript\"}}") writePromptFileWithTemplate(t, dir, id, defaultProfile, "Summarize: {{input \"transcript\"}}")

View File

@@ -27,6 +27,7 @@ type modelOverrideRequestDTO struct {
MaxTokens int `json:"max_tokens,omitempty"` MaxTokens int `json:"max_tokens,omitempty"`
TopP float64 `json:"top_p,omitempty"` TopP float64 `json:"top_p,omitempty"`
TimeoutSeconds int `json:"timeout_seconds,omitempty"` TimeoutSeconds int `json:"timeout_seconds,omitempty"`
ServiceTier string `json:"service_tier,omitempty"`
ReasoningEffort string `json:"reasoning_effort,omitempty"` ReasoningEffort string `json:"reasoning_effort,omitempty"`
APIKeyEnv string `json:"api_key_env,omitempty"` APIKeyEnv string `json:"api_key_env,omitempty"`
ExtraParams map[string]string `json:"extra_params,omitempty"` ExtraParams map[string]string `json:"extra_params,omitempty"`
@@ -75,6 +76,7 @@ type modelParamsDTO struct {
MaxTokens int `json:"max_tokens"` MaxTokens int `json:"max_tokens"`
TopP float64 `json:"top_p"` TopP float64 `json:"top_p"`
TimeoutSeconds int `json:"timeout_seconds"` TimeoutSeconds int `json:"timeout_seconds"`
ServiceTier string `json:"service_tier,omitempty"`
ReasoningEffort string `json:"reasoning_effort,omitempty"` ReasoningEffort string `json:"reasoning_effort,omitempty"`
APIKeyEnv string `json:"api_key_env,omitempty"` APIKeyEnv string `json:"api_key_env,omitempty"`
ExtraParams map[string]string `json:"extra_params,omitempty"` ExtraParams map[string]string `json:"extra_params,omitempty"`

View File

@@ -63,17 +63,7 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
var model *domain.ExecutionTarget var model *domain.ExecutionTarget
if req.Model != nil { if req.Model != nil {
model = &domain.ExecutionTarget{ model = executionTargetFromModelOverrideDTO(req.Model)
Endpoint: req.Model.Endpoint,
Model: req.Model.Model,
Temperature: req.Model.Temperature,
MaxTokens: req.Model.MaxTokens,
TopP: req.Model.TopP,
TimeoutSeconds: req.Model.TimeoutSeconds,
ReasoningEffort: req.Model.ReasoningEffort,
APIKeyEnv: req.Model.APIKeyEnv,
ExtraParams: req.Model.ExtraParams,
}
} }
res, err := h.runner.Run(r.Context(), domain.RunRequest{ res, err := h.runner.Run(r.Context(), domain.RunRequest{
@@ -109,17 +99,7 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
SelectedProfileID: res.SelectedProfileID, SelectedProfileID: res.SelectedProfileID,
ModelName: res.ModelName, ModelName: res.ModelName,
Endpoint: res.Endpoint, Endpoint: res.Endpoint,
ModelParams: modelParamsDTO{ ModelParams: modelParamsDTOFromExecutionTarget(res.EffectiveModelParams),
Endpoint: res.EffectiveModelParams.Endpoint,
Model: res.EffectiveModelParams.Model,
Temperature: res.EffectiveModelParams.Temperature,
MaxTokens: res.EffectiveModelParams.MaxTokens,
TopP: res.EffectiveModelParams.TopP,
TimeoutSeconds: res.EffectiveModelParams.TimeoutSeconds,
ReasoningEffort: res.EffectiveModelParams.ReasoningEffort,
APIKeyEnv: res.EffectiveModelParams.APIKeyEnv,
ExtraParams: res.EffectiveModelParams.ExtraParams,
},
InputHashes: res.InputHashes, InputHashes: res.InputHashes,
Usage: tokenUsageDTO{ Usage: tokenUsageDTO{
PromptTokens: res.Usage.PromptTokens, PromptTokens: res.Usage.PromptTokens,
@@ -141,6 +121,39 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, resp) writeJSON(w, http.StatusOK, resp)
} }
func executionTargetFromModelOverrideDTO(dto *modelOverrideRequestDTO) *domain.ExecutionTarget {
if dto == nil {
return nil
}
return &domain.ExecutionTarget{
Endpoint: dto.Endpoint,
Model: dto.Model,
Temperature: dto.Temperature,
MaxTokens: dto.MaxTokens,
TopP: dto.TopP,
TimeoutSeconds: dto.TimeoutSeconds,
ServiceTier: dto.ServiceTier,
ReasoningEffort: dto.ReasoningEffort,
APIKeyEnv: dto.APIKeyEnv,
ExtraParams: dto.ExtraParams,
}
}
func modelParamsDTOFromExecutionTarget(target domain.ExecutionTarget) modelParamsDTO {
return modelParamsDTO{
Endpoint: target.Endpoint,
Model: target.Model,
Temperature: target.Temperature,
MaxTokens: target.MaxTokens,
TopP: target.TopP,
TimeoutSeconds: target.TimeoutSeconds,
ServiceTier: target.ServiceTier,
ReasoningEffort: target.ReasoningEffort,
APIKeyEnv: target.APIKeyEnv,
ExtraParams: target.ExtraParams,
}
}
func mapValidation(v domain.ValidationResult) validationDTO { func mapValidation(v domain.ValidationResult) validationDTO {
return validationDTO{ return validationDTO{
Status: string(v.Status), Status: string(v.Status),
@@ -162,9 +175,9 @@ func mapRunError(err error) (int, string, string) {
return http.StatusBadRequest, "prompt_load_failed", "failed to load prompt definition" return http.StatusBadRequest, "prompt_load_failed", "failed to load prompt definition"
case errors.Is(err, profile.ErrInvalidYAML), errors.Is(err, profile.ErrInvalidProfile): case errors.Is(err, profile.ErrInvalidYAML), errors.Is(err, profile.ErrInvalidProfile):
return http.StatusBadRequest, "profile_load_failed", "failed to load execution profile" return http.StatusBadRequest, "profile_load_failed", "failed to load execution profile"
case errors.Is(err, usecase.ErrInvalidRequest) && strings.Contains(err.Error(), "profile id is required either in request or prompt default_profile"): case errors.Is(err, usecase.ErrProfileRequired):
return http.StatusBadRequest, "profile_required", "profile_id is required when prompt default_profile is not set" return http.StatusBadRequest, "profile_required", "profile_id is required when prompt default_profile is not set"
case errors.Is(err, usecase.ErrInvalidRequest) && strings.Contains(err.Error(), "api key environment variable"): case errors.Is(err, usecase.ErrAPIKeyEnvMissing):
return http.StatusBadRequest, "api_key_env_missing", "api_key_env is set but the environment variable is missing" return http.StatusBadRequest, "api_key_env_missing", "api_key_env is set but the environment variable is missing"
case errors.Is(err, usecase.ErrInvalidRequest): case errors.Is(err, usecase.ErrInvalidRequest):
return http.StatusBadRequest, "invalid_request", "invalid run request" return http.StatusBadRequest, "invalid_request", "invalid run request"

View File

@@ -4,10 +4,10 @@ import (
"bytes" "bytes"
"context" "context"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"reflect"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -62,6 +62,7 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
MaxTokens: 42, MaxTokens: 42,
TopP: 0.9, TopP: 0.9,
TimeoutSeconds: 120, TimeoutSeconds: 120,
ServiceTier: "priority",
APIKeyEnv: envName, APIKeyEnv: envName,
}, },
InputHashes: map[string]string{"transcript": "h1"}, InputHashes: map[string]string{"transcript": "h1"},
@@ -81,7 +82,7 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
"transcript": {"type": "file", "uri": "./t.md"} "transcript": {"type": "file", "uri": "./t.md"}
}, },
"vars": {"k": "v"}, "vars": {"k": "v"},
"model": {"model": "gpt-x", "timeout_seconds": 120, "api_key_env": "SCRIPTORIUM_API_KEY"} "model": {"model": "gpt-x", "timeout_seconds": 120, "service_tier": "flex", "api_key_env": "SCRIPTORIUM_API_KEY"}
}`) }`)
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewReader(body)) req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewReader(body))
w := httptest.NewRecorder() w := httptest.NewRecorder()
@@ -114,6 +115,9 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
if modelParams["api_key_env"] != envName { if modelParams["api_key_env"] != envName {
t.Fatalf("expected model_params.api_key_env=%q, got %#v", envName, modelParams["api_key_env"]) t.Fatalf("expected model_params.api_key_env=%q, got %#v", envName, modelParams["api_key_env"])
} }
if modelParams["service_tier"] != "priority" {
t.Fatalf("expected model_params.service_tier=priority, got %#v", modelParams["service_tier"])
}
if strings.Contains(w.Body.String(), secret) { if strings.Contains(w.Body.String(), secret) {
t.Fatalf("response leaked raw API key value: %s", w.Body.String()) t.Fatalf("response leaked raw API key value: %s", w.Body.String())
} }
@@ -133,6 +137,9 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
if r.last.Execution.TimeoutSeconds != 120 { if r.last.Execution.TimeoutSeconds != 120 {
t.Fatalf("expected timeout_seconds override 120, got %#v", r.last.Execution) t.Fatalf("expected timeout_seconds override 120, got %#v", r.last.Execution)
} }
if r.last.Execution.ServiceTier != "flex" {
t.Fatalf("expected service_tier override flex, got %#v", r.last.Execution)
}
} }
func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) { func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
@@ -166,6 +173,136 @@ func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
} }
} }
func TestHandlerModelOverrideMapsAllSupportedExecutionFields(t *testing.T) {
r := &fakeRunner{result: &domain.RunResult{
Artifact: domain.Artifact{Body: []byte("ok")},
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
EffectiveModelParams: domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
}}
h := NewHandler(r)
reqBody := `{
"prompt_id": "prompt-1",
"inputs": {"transcript": {"type": "file", "uri": "./t.md"}},
"model": {
"endpoint": "http://override/v1",
"model": "override-model",
"temperature": 0.6,
"max_tokens": 250,
"top_p": 0.85,
"timeout_seconds": 33,
"service_tier": "flex",
"reasoning_effort": "medium",
"api_key_env": "SCRIPTORIUM_API_KEY",
"extra_params": {"provider_option":"on"}
}
}`
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(reqBody))
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
}
if r.last.Execution == nil {
t.Fatalf("expected execution override in run request")
}
got := r.last.Execution
if got.Endpoint != "http://override/v1" ||
got.Model != "override-model" ||
got.Temperature != 0.6 ||
got.MaxTokens != 250 ||
got.TopP != 0.85 ||
got.TimeoutSeconds != 33 ||
got.ServiceTier != "flex" ||
got.ReasoningEffort != "medium" ||
got.APIKeyEnv != "SCRIPTORIUM_API_KEY" {
t.Fatalf("unexpected mapped execution target: %+v", got)
}
if !reflect.DeepEqual(got.ExtraParams, map[string]string{"provider_option": "on"}) {
t.Fatalf("unexpected mapped extra_params: %#v", got.ExtraParams)
}
}
func TestHandlerResponseMetadataModelParamsIncludesAllSupportedFields(t *testing.T) {
r := &fakeRunner{result: &domain.RunResult{
Artifact: domain.Artifact{
Name: "output",
ContentType: "text/plain",
Body: []byte("ok"),
Size: 2,
Hash: "abc",
},
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
EffectiveModelParams: domain.ExecutionTarget{
Endpoint: "http://llm/v1",
Model: "gpt-test",
Temperature: 0.4,
MaxTokens: 321,
TopP: 0.7,
TimeoutSeconds: 45,
ServiceTier: "priority",
ReasoningEffort: "high",
APIKeyEnv: "SCRIPTORIUM_API_KEY",
ExtraParams: map[string]string{
"provider_option": "on",
},
},
}}
h := NewHandler(r)
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p","inputs":{"x":{"type":"file","uri":"a"}}}`))
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
}
var resp map[string]any
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("invalid JSON response: %v", err)
}
metadata := resp["metadata"].(map[string]any)
params := metadata["model_params"].(map[string]any)
if params["endpoint"] != "http://llm/v1" {
t.Fatalf("unexpected endpoint: %#v", params["endpoint"])
}
if params["model"] != "gpt-test" {
t.Fatalf("unexpected model: %#v", params["model"])
}
if params["temperature"] != 0.4 {
t.Fatalf("unexpected temperature: %#v", params["temperature"])
}
if params["max_tokens"] != float64(321) {
t.Fatalf("unexpected max_tokens: %#v", params["max_tokens"])
}
if params["top_p"] != 0.7 {
t.Fatalf("unexpected top_p: %#v", params["top_p"])
}
if params["timeout_seconds"] != float64(45) {
t.Fatalf("unexpected timeout_seconds: %#v", params["timeout_seconds"])
}
if params["service_tier"] != "priority" {
t.Fatalf("unexpected service_tier: %#v", params["service_tier"])
}
if params["reasoning_effort"] != "high" {
t.Fatalf("unexpected reasoning_effort: %#v", params["reasoning_effort"])
}
if params["api_key_env"] != "SCRIPTORIUM_API_KEY" {
t.Fatalf("unexpected api_key_env: %#v", params["api_key_env"])
}
extraParams, ok := params["extra_params"].(map[string]any)
if !ok {
t.Fatalf("expected extra_params object, got %#v", params["extra_params"])
}
if extraParams["provider_option"] != "on" {
t.Fatalf("unexpected extra_params.provider_option: %#v", extraParams["provider_option"])
}
}
func TestHandlerInvalidJSON(t *testing.T) { func TestHandlerInvalidJSON(t *testing.T) {
h := NewHandler(&fakeRunner{}) h := NewHandler(&fakeRunner{})
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString("{")) req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString("{"))
@@ -209,10 +346,10 @@ func TestHandlerUsecaseErrorMapping(t *testing.T) {
}{ }{
{name: "prompt not found", err: wrap(usecase.ErrProfileLoad, promptdef.ErrPromptDefinitionNotFound), status: http.StatusNotFound, code: "prompt_not_found", message: "prompt definition not found"}, {name: "prompt not found", err: wrap(usecase.ErrProfileLoad, promptdef.ErrPromptDefinitionNotFound), status: http.StatusNotFound, code: "prompt_not_found", message: "prompt definition not found"},
{name: "prompt load invalid", err: wrap(usecase.ErrProfileLoad, promptdef.ErrInvalidPromptDefinition), status: http.StatusBadRequest, code: "prompt_load_failed", message: "failed to load prompt definition"}, {name: "prompt load invalid", err: wrap(usecase.ErrProfileLoad, promptdef.ErrInvalidPromptDefinition), status: http.StatusBadRequest, code: "prompt_load_failed", message: "failed to load prompt definition"},
{name: "missing profile/default", err: wrap(usecase.ErrInvalidRequest, errors.New("profile id is required either in request or prompt default_profile")), status: http.StatusBadRequest, code: "profile_required", message: "profile_id is required when prompt default_profile is not set"}, {name: "missing profile/default", err: wrap(usecase.ErrInvalidRequest, usecase.ErrProfileRequired), status: http.StatusBadRequest, code: "profile_required", message: "profile_id is required when prompt default_profile is not set"},
{name: "profile not found", err: wrap(usecase.ErrProfileLoad, profile.ErrProfileNotFound), status: http.StatusNotFound, code: "profile_not_found", message: "execution profile not found"}, {name: "profile not found", err: wrap(usecase.ErrProfileLoad, profile.ErrProfileNotFound), status: http.StatusNotFound, code: "profile_not_found", message: "execution profile not found"},
{name: "profile invalid", err: wrap(usecase.ErrProfileLoad, profile.ErrInvalidProfile), status: http.StatusBadRequest, code: "profile_load_failed", message: "failed to load execution profile"}, {name: "profile invalid", err: wrap(usecase.ErrProfileLoad, profile.ErrInvalidProfile), status: http.StatusBadRequest, code: "profile_load_failed", message: "failed to load execution profile"},
{name: "api key env missing", err: wrap(usecase.ErrInvalidRequest, errors.New(`api key environment variable "SCRIPTORIUM_API_KEY" is not set`)), status: http.StatusBadRequest, code: "api_key_env_missing", message: "api_key_env is set but the environment variable is missing"}, {name: "api key env missing", err: wrap(usecase.ErrInvalidRequest, usecase.ErrAPIKeyEnvMissing), status: http.StatusBadRequest, code: "api_key_env_missing", message: "api_key_env is set but the environment variable is missing"},
{name: "artifact", err: wrap(usecase.ErrArtifactLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "artifact_read_failed", message: "failed to read input artifact", avoidCause: "read failed"}, {name: "artifact", err: wrap(usecase.ErrArtifactLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "artifact_read_failed", message: "failed to read input artifact", avoidCause: "read failed"},
{name: "prompt render", err: wrap(usecase.ErrPromptRender, fmt.Errorf("render failed")), status: http.StatusBadRequest, code: "prompt_render_failed", message: "failed to render prompt", avoidCause: "render failed"}, {name: "prompt render", err: wrap(usecase.ErrPromptRender, fmt.Errorf("render failed")), status: http.StatusBadRequest, code: "prompt_render_failed", message: "failed to render prompt", avoidCause: "render failed"},
{name: "llm", err: wrap(usecase.ErrLLMGenerate, fmt.Errorf("llm failed")), status: http.StatusBadGateway, code: "llm_failed", message: "model generation request failed", avoidCause: "llm failed"}, {name: "llm", err: wrap(usecase.ErrLLMGenerate, fmt.Errorf("llm failed")), status: http.StatusBadGateway, code: "llm_failed", message: "model generation request failed", avoidCause: "llm failed"},

View File

@@ -46,8 +46,8 @@ func TestCompositeReader_Read(t *testing.T) {
t.Run("unsupported ref type", func(t *testing.T) { t.Run("unsupported ref type", func(t *testing.T) {
ref := domain.ArtifactRef{ ref := domain.ArtifactRef{
Type: domain.ArtifactRefS3, Type: domain.ArtifactRefType("unsupported"),
URI: "s3://bucket/key", URI: "unsupported://bucket/key",
} }
_, err := reader.Read(ctx, ref) _, err := reader.Read(ctx, ref)
if !errors.Is(err, ErrUnsupportedRefType) { if !errors.Is(err, ErrUnsupportedRefType) {

View File

@@ -10,7 +10,6 @@ type ArtifactRefType string
const ( const (
ArtifactRefInline ArtifactRefType = "inline" ArtifactRefInline ArtifactRefType = "inline"
ArtifactRefFile ArtifactRefType = "file" ArtifactRefFile ArtifactRefType = "file"
ArtifactRefS3 ArtifactRefType = "s3"
) )
// OutputFormat defines the desired format of the generated artifact. // OutputFormat defines the desired format of the generated artifact.
@@ -146,6 +145,7 @@ type ExecutionProfile struct {
MaxTokens int `yaml:"max_tokens"` MaxTokens int `yaml:"max_tokens"`
TopP float64 `yaml:"top_p"` TopP float64 `yaml:"top_p"`
TimeoutSeconds int `yaml:"timeout_seconds"` TimeoutSeconds int `yaml:"timeout_seconds"`
ServiceTier string `yaml:"service_tier"`
ReasoningEffort string `yaml:"reasoning_effort"` ReasoningEffort string `yaml:"reasoning_effort"`
APIKeyEnv string `yaml:"api_key_env"` APIKeyEnv string `yaml:"api_key_env"`
ExtraParams map[string]string `yaml:"extra_params"` ExtraParams map[string]string `yaml:"extra_params"`
@@ -159,6 +159,7 @@ type ExecutionTarget struct {
MaxTokens int `yaml:"max_tokens" json:"max_tokens"` MaxTokens int `yaml:"max_tokens" json:"max_tokens"`
TopP float64 `yaml:"top_p" json:"top_p"` TopP float64 `yaml:"top_p" json:"top_p"`
TimeoutSeconds int `yaml:"timeout_seconds" json:"timeout_seconds"` TimeoutSeconds int `yaml:"timeout_seconds" json:"timeout_seconds"`
ServiceTier string `yaml:"service_tier" json:"service_tier"`
ReasoningEffort string `yaml:"reasoning_effort" json:"reasoning_effort"` ReasoningEffort string `yaml:"reasoning_effort" json:"reasoning_effort"`
APIKeyEnv string `yaml:"api_key_env" json:"api_key_env"` APIKeyEnv string `yaml:"api_key_env" json:"api_key_env"`
ExtraParams map[string]string `yaml:"extra_params" json:"extra_params"` ExtraParams map[string]string `yaml:"extra_params" json:"extra_params"`

View File

@@ -0,0 +1,54 @@
package filecatalog
import (
"context"
"os"
"path/filepath"
"sort"
"strings"
)
// FindYAMLFiles returns sorted full paths for .yaml and .yml files under root.
func FindYAMLFiles(ctx context.Context, root string) ([]string, error) {
var files []string
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
select {
case <-ctx.Done():
return ctx.Err()
default:
}
if d.IsDir() {
return nil
}
if !isYAMLFile(d.Name()) {
return nil
}
files = append(files, path)
return nil
})
sort.Strings(files)
return files, err
}
// RelativePath computes a clean relative path from root to path.
func RelativePath(root string, path string) string {
rel, err := filepath.Rel(root, path)
if err != nil {
return filepath.Clean(path)
}
return filepath.Clean(rel)
}
// Stem strips .yaml or .yml from a file name.
func Stem(name string) string {
name = strings.TrimSuffix(name, ".yaml")
name = strings.TrimSuffix(name, ".yml")
return name
}
func isYAMLFile(name string) bool {
return strings.HasSuffix(name, ".yaml") || strings.HasSuffix(name, ".yml")
}

View File

@@ -0,0 +1,84 @@
package filecatalog
import (
"context"
"errors"
"os"
"path/filepath"
"reflect"
"testing"
)
func TestFindYAMLFilesNestedSortedAndFiltered(t *testing.T) {
root := t.TempDir()
mustWriteFile(t, filepath.Join(root, "z", "prompt.yml"), "id: z")
mustWriteFile(t, filepath.Join(root, "a", "profile.yaml"), "id: a")
mustWriteFile(t, filepath.Join(root, "a", "ignore.txt"), "not yaml")
mustWriteFile(t, filepath.Join(root, "b", "ignore.yaml.bak"), "not yaml")
got, err := FindYAMLFiles(context.Background(), root)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
want := []string{
filepath.Join(root, "a", "profile.yaml"),
filepath.Join(root, "z", "prompt.yml"),
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("expected sorted YAML files %v, got %v", want, got)
}
}
func TestFindYAMLFilesHonorsContextCancellation(t *testing.T) {
root := t.TempDir()
mustWriteFile(t, filepath.Join(root, "one.yaml"), "id: one")
ctx, cancel := context.WithCancel(context.Background())
cancel()
_, err := FindYAMLFiles(ctx, root)
if !errors.Is(err, context.Canceled) {
t.Fatalf("expected context.Canceled, got %v", err)
}
}
func TestRelativePathNested(t *testing.T) {
root := t.TempDir()
path := filepath.Join(root, "nested", "profiles", "local.yaml")
got := RelativePath(root, path)
want := filepath.Join("nested", "profiles", "local.yaml")
if got != want {
t.Fatalf("expected relative path %q, got %q", want, got)
}
}
func TestStemStripsYAMLExtensions(t *testing.T) {
tests := []struct {
name string
in string
want string
}{
{name: "yaml", in: "prompt.yaml", want: "prompt"},
{name: "yml", in: "profile.yml", want: "profile"},
{name: "other", in: "file.txt", want: "file.txt"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if got := Stem(tc.in); got != tc.want {
t.Fatalf("expected %q, got %q", tc.want, got)
}
})
}
}
func mustWriteFile(t *testing.T, path string, content string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatalf("failed to create directory: %v", err)
}
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatalf("failed to write file %q: %v", path, err)
}
}

View File

@@ -106,6 +106,9 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
fmt.Fprintf(&b, " max_tokens: %d\n", target.MaxTokens) fmt.Fprintf(&b, " max_tokens: %d\n", target.MaxTokens)
fmt.Fprintf(&b, " top_p: %g\n", target.TopP) fmt.Fprintf(&b, " top_p: %g\n", target.TopP)
fmt.Fprintf(&b, " timeout_seconds: %d\n", target.TimeoutSeconds) fmt.Fprintf(&b, " timeout_seconds: %d\n", target.TimeoutSeconds)
if target.ServiceTier != "" {
fmt.Fprintf(&b, " service_tier: %s\n", target.ServiceTier)
}
if target.ReasoningEffort != "" { if target.ReasoningEffort != "" {
fmt.Fprintf(&b, " reasoning_effort: %s\n", target.ReasoningEffort) fmt.Fprintf(&b, " reasoning_effort: %s\n", target.ReasoningEffort)
} }

View File

@@ -28,6 +28,7 @@ func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
"max_tokens: 256", "max_tokens: 256",
"top_p: 0.8", "top_p: 0.8",
"timeout_seconds: 45", "timeout_seconds: 45",
"service_tier: priority",
"reasoning_effort: medium", "reasoning_effort: medium",
"api_key_env: SCRIPTORIUM_API_KEY", "api_key_env: SCRIPTORIUM_API_KEY",
"prompt_hash: prompt-hash", "prompt_hash: prompt-hash",
@@ -168,6 +169,7 @@ func samplePreparedRun() *domain.PreparedRun {
MaxTokens: 256, MaxTokens: 256,
TopP: 0.8, TopP: 0.8,
TimeoutSeconds: 45, TimeoutSeconds: 45,
ServiceTier: "priority",
ReasoningEffort: "medium", ReasoningEffort: "medium",
APIKeyEnv: "SCRIPTORIUM_API_KEY", APIKeyEnv: "SCRIPTORIUM_API_KEY",
}, },

View File

@@ -75,14 +75,6 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
return nil, fmt.Errorf("%w: timeout_seconds must be greater than or equal to 0", ErrInvalidRequest) return nil, fmt.Errorf("%w: timeout_seconds must be greater than or equal to 0", ErrInvalidRequest)
} }
model := strings.TrimSpace(req.Target.Model)
if model == "" {
model = strings.TrimSpace(c.defaultModel)
}
if model == "" {
return nil, fmt.Errorf("%w: model is required", ErrInvalidRequest)
}
endpoint := strings.TrimSpace(req.Target.Endpoint) endpoint := strings.TrimSpace(req.Target.Endpoint)
if endpoint == "" { if endpoint == "" {
endpoint = c.baseURL endpoint = c.baseURL
@@ -92,34 +84,10 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
} }
endpoint = strings.TrimRight(endpoint, "/") + defaults.OpenAIChatCompletionsPath endpoint = strings.TrimRight(endpoint, "/") + defaults.OpenAIChatCompletionsPath
wireReq := openAIChatRequest{ wireReq, err := openAIChatRequestFromGenerateRequest(req, c.defaultModel)
Model: model,
}
wireReq.Messages = make([]openAIChatMessage, 0, len(req.Prompt.Messages))
for _, msg := range req.Prompt.Messages {
wireReq.Messages = append(wireReq.Messages, openAIChatMessage{
Role: msg.Role,
Content: msg.Content,
})
}
if req.Target.Temperature != 0 {
wireReq.Temperature = &req.Target.Temperature
}
if req.Target.MaxTokens != 0 {
wireReq.MaxTokens = &req.Target.MaxTokens
}
if req.Target.TopP != 0 {
wireReq.TopP = &req.Target.TopP
}
if req.StructuredOutput != nil {
responseFormat, err := toOpenAIResponseFormat(req.StructuredOutput)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err) return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
} }
wireReq.ResponseFormat = responseFormat
}
payload, err := json.Marshal(wireReq) payload, err := json.Marshal(wireReq)
if err != nil { if err != nil {
@@ -187,12 +155,57 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
}, nil }, nil
} }
func openAIChatRequestFromGenerateRequest(req domain.GenerateRequest, defaultModel string) (openAIChatRequest, error) {
model := strings.TrimSpace(req.Target.Model)
if model == "" {
model = strings.TrimSpace(defaultModel)
}
if model == "" {
return openAIChatRequest{}, errors.New("model is required")
}
wireReq := openAIChatRequest{
Model: model,
}
wireReq.Messages = make([]openAIChatMessage, 0, len(req.Prompt.Messages))
for _, msg := range req.Prompt.Messages {
wireReq.Messages = append(wireReq.Messages, openAIChatMessage{
Role: msg.Role,
Content: msg.Content,
})
}
if req.Target.Temperature != 0 {
wireReq.Temperature = &req.Target.Temperature
}
if req.Target.MaxTokens != 0 {
wireReq.MaxTokens = &req.Target.MaxTokens
}
if req.Target.TopP != 0 {
wireReq.TopP = &req.Target.TopP
}
if strings.TrimSpace(req.Target.ServiceTier) != "" {
wireReq.ServiceTier = req.Target.ServiceTier
}
if req.StructuredOutput != nil {
responseFormat, err := toOpenAIResponseFormat(req.StructuredOutput)
if err != nil {
return openAIChatRequest{}, err
}
wireReq.ResponseFormat = responseFormat
}
return wireReq, nil
}
type openAIChatRequest struct { type openAIChatRequest struct {
Model string `json:"model"` Model string `json:"model"`
Messages []openAIChatMessage `json:"messages"` Messages []openAIChatMessage `json:"messages"`
Temperature *float64 `json:"temperature,omitempty"` Temperature *float64 `json:"temperature,omitempty"`
MaxTokens *int `json:"max_tokens,omitempty"` MaxTokens *int `json:"max_tokens,omitempty"`
TopP *float64 `json:"top_p,omitempty"` TopP *float64 `json:"top_p,omitempty"`
ServiceTier string `json:"service_tier,omitempty"`
ResponseFormat *openAIResponseFormat `json:"response_format,omitempty"` ResponseFormat *openAIResponseFormat `json:"response_format,omitempty"`
} }

View File

@@ -61,6 +61,7 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
Temperature: 0.4, Temperature: 0.4,
MaxTokens: 123, MaxTokens: 123,
TopP: 0.7, TopP: 0.7,
ServiceTier: "priority",
APIKeyEnv: "SCRIPTORIUM_TEST_API_KEY", APIKeyEnv: "SCRIPTORIUM_TEST_API_KEY",
}, },
StructuredOutput: &domain.StructuredOutputSpec{ StructuredOutput: &domain.StructuredOutputSpec{
@@ -95,6 +96,18 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
if got, ok := obs.Body["model"].(string); !ok || got != "gpt-test" { if got, ok := obs.Body["model"].(string); !ok || got != "gpt-test" {
t.Fatalf("unexpected model payload: %#v", obs.Body["model"]) t.Fatalf("unexpected model payload: %#v", obs.Body["model"])
} }
if got, ok := obs.Body["temperature"].(float64); !ok || got != 0.4 {
t.Fatalf("unexpected temperature payload: %#v", obs.Body["temperature"])
}
if got, ok := obs.Body["max_tokens"].(float64); !ok || got != 123 {
t.Fatalf("unexpected max_tokens payload: %#v", obs.Body["max_tokens"])
}
if got, ok := obs.Body["top_p"].(float64); !ok || got != 0.7 {
t.Fatalf("unexpected top_p payload: %#v", obs.Body["top_p"])
}
if got, ok := obs.Body["service_tier"].(string); !ok || got != "priority" {
t.Fatalf("unexpected service_tier payload: %#v", obs.Body["service_tier"])
}
msgs, ok := obs.Body["messages"].([]any) msgs, ok := obs.Body["messages"].([]any)
if !ok || len(msgs) != 2 { if !ok || len(msgs) != 2 {
@@ -157,6 +170,46 @@ func TestOpenAICompatibleClientOmitsResponseFormatWhenNoStructuredOutput(t *test
if _, exists := observedBody["response_format"]; exists { if _, exists := observedBody["response_format"]; exists {
t.Fatalf("expected response_format omitted, got %#v", observedBody["response_format"]) t.Fatalf("expected response_format omitted, got %#v", observedBody["response_format"])
} }
if _, exists := observedBody["service_tier"]; exists {
t.Fatalf("expected service_tier omitted, got %#v", observedBody["service_tier"])
}
}
func TestOpenAICompatibleClientOmitsReasoningEffortAndExtraParams(t *testing.T) {
var observedBody map[string]any
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
t.Fatalf("failed to decode request body: %v", err)
}
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
Target: domain.ExecutionTarget{
Model: "model",
ReasoningEffort: "high",
ExtraParams: map[string]string{
"provider_option": "on",
},
},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if _, exists := observedBody["reasoning_effort"]; exists {
t.Fatalf("expected reasoning_effort omitted, got %#v", observedBody["reasoning_effort"])
}
if _, exists := observedBody["extra_params"]; exists {
t.Fatalf("expected extra_params omitted, got %#v", observedBody["extra_params"])
}
} }
func TestOpenAICompatibleClientNoAuthorizationHeaderWhenNoAPIKey(t *testing.T) { func TestOpenAICompatibleClientNoAuthorizationHeaderWhenNoAPIKey(t *testing.T) {

View File

@@ -10,6 +10,7 @@ import (
"strings" "strings"
"gitea.maximumdirect.net/eric/scriptorium/internal/domain" "gitea.maximumdirect.net/eric/scriptorium/internal/domain"
"gitea.maximumdirect.net/eric/scriptorium/internal/filecatalog"
"gopkg.in/yaml.v3" "gopkg.in/yaml.v3"
) )
@@ -33,40 +34,40 @@ func (r *filesystemRepository) GetProfile(ctx context.Context, id string) (*doma
return nil, fmt.Errorf("%w: profile id is required", ErrInvalidProfile) return nil, fmt.Errorf("%w: profile id is required", ErrInvalidProfile)
} }
files, err := os.ReadDir(r.dir) files, err := filecatalog.FindYAMLFiles(ctx, r.dir)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read profile directory: %w", err) return nil, fmt.Errorf("failed to read profile directory: %w", err)
} }
for _, file := range files { var matches []profileMatch
for _, fullPath := range files {
select { select {
case <-ctx.Done(): case <-ctx.Done():
return nil, ctx.Err() return nil, ctx.Err()
default: default:
} }
if file.IsDir() || (!strings.HasSuffix(file.Name(), ".yaml") && !strings.HasSuffix(file.Name(), ".yml")) { relPath := filecatalog.RelativePath(r.dir, fullPath)
continue fileMatch := filecatalog.Stem(filepath.Base(fullPath)) == id
}
fullPath := filepath.Join(r.dir, file.Name())
data, err := os.ReadFile(fullPath) data, err := os.ReadFile(fullPath)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read profile file %s: %w", file.Name(), err) return nil, fmt.Errorf("failed to read profile file %s: %w", relPath, err)
}
metadata := readProfileFileMetadata(data)
idMatch := fileMatch || metadata.id == id
if metadata.hasRawAPIKey {
if idMatch {
return nil, fmt.Errorf("%w: %s", ErrRawAPIKeyNotAllowed, relPath)
}
continue
} }
var prof domain.ExecutionProfile var prof domain.ExecutionProfile
decoder := yaml.NewDecoder(bytes.NewReader(data)) decoder := yaml.NewDecoder(bytes.NewReader(data))
decoder.KnownFields(true) decoder.KnownFields(true)
if err := decoder.Decode(&prof); err != nil { if err := decoder.Decode(&prof); err != nil {
if strings.Contains(err.Error(), "field api_key not found") { if idMatch {
if strings.TrimSuffix(strings.TrimSuffix(file.Name(), ".yaml"), ".yml") == id { return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
return nil, fmt.Errorf("%w: %s", ErrRawAPIKeyNotAllowed, file.Name())
}
continue
}
if strings.TrimSuffix(strings.TrimSuffix(file.Name(), ".yaml"), ".yml") == id {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, file.Name(), err)
} }
continue continue
} }
@@ -76,16 +77,68 @@ func (r *filesystemRepository) GetProfile(ctx context.Context, id string) (*doma
} }
if err := validateProfile(&prof); err != nil { if err := validateProfile(&prof); err != nil {
if errors.Is(err, ErrRawAPIKeyNotAllowed) { if errors.Is(err, ErrRawAPIKeyNotAllowed) {
return nil, fmt.Errorf("%w: %s", err, file.Name()) return nil, fmt.Errorf("%w: %s", err, relPath)
} }
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, file.Name(), err) return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, relPath, err)
} }
return &prof, nil matches = append(matches, profileMatch{
profile: &prof,
path: relPath,
})
}
if len(matches) > 1 {
paths := make([]string, 0, len(matches))
for _, match := range matches {
paths = append(paths, match.path)
}
return nil, fmt.Errorf("%w: duplicate execution profile id %q found in: %s", ErrInvalidProfile, id, strings.Join(paths, ", "))
}
if len(matches) == 1 {
return matches[0].profile, nil
} }
return nil, ErrProfileNotFound return nil, ErrProfileNotFound
} }
type profileMatch struct {
profile *domain.ExecutionProfile
path string
}
type profileFileMetadata struct {
id string
hasRawAPIKey bool
}
func readProfileFileMetadata(data []byte) profileFileMetadata {
var node yaml.Node
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&node); err != nil {
return profileFileMetadata{}
}
if node.Kind != yaml.DocumentNode || len(node.Content) == 0 {
return profileFileMetadata{}
}
mapping := node.Content[0]
if mapping.Kind != yaml.MappingNode {
return profileFileMetadata{}
}
var metadata profileFileMetadata
for i := 0; i+1 < len(mapping.Content); i += 2 {
key := mapping.Content[i]
value := mapping.Content[i+1]
switch key.Value {
case "id":
metadata.id = strings.TrimSpace(value.Value)
case "api_key":
metadata.hasRawAPIKey = true
}
}
return metadata
}
func validateProfile(p *domain.ExecutionProfile) error { func validateProfile(p *domain.ExecutionProfile) error {
if strings.TrimSpace(p.ID) == "" { if strings.TrimSpace(p.ID) == "" {
return errors.New("id is required") return errors.New("id is required")

View File

@@ -5,6 +5,7 @@ import (
"errors" "errors"
"os" "os"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
) )
@@ -58,6 +59,92 @@ func TestFilesystemRepository_GetProfile(t *testing.T) {
if p.ReasoningEffort != "medium" { if p.ReasoningEffort != "medium" {
t.Fatalf("unexpected reasoning_effort: %q", p.ReasoningEffort) t.Fatalf("unexpected reasoning_effort: %q", p.ReasoningEffort)
} }
if p.ServiceTier != "priority" {
t.Fatalf("unexpected service_tier: %q", p.ServiceTier)
}
})
t.Run("valid nested profile", func(t *testing.T) {
nestedDir := filepath.Join(tmpDir, "local")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writeProfileTestFile(t, filepath.Join(nestedDir, "nested-local.yaml"), `
id: nested-local
endpoint: http://localhost:8000/v1
model: nested-model
temperature: 0.1
`)
p, err := repo.GetProfile(ctx, "nested-local")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if p.Model != "nested-model" {
t.Fatalf("unexpected model: %q", p.Model)
}
})
t.Run("duplicate profile IDs fail as ambiguous", func(t *testing.T) {
writeProfileTestFile(t, filepath.Join(tmpDir, "duplicate-profile-a.yaml"), `
id: duplicate-profile
endpoint: http://localhost:8000/v1
model: first-model
`)
nestedDir := filepath.Join(tmpDir, "duplicates")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writeProfileTestFile(t, filepath.Join(nestedDir, "duplicate-profile-b.yaml"), `
id: duplicate-profile
endpoint: http://localhost:8000/v1
model: second-model
`)
_, err := repo.GetProfile(ctx, "duplicate-profile")
if !errors.Is(err, ErrInvalidProfile) {
t.Fatalf("expected duplicate profile to return ErrInvalidProfile, got %v", err)
}
for _, want := range []string{"duplicate execution profile id", "duplicate-profile-a.yaml", filepath.Join("duplicates", "duplicate-profile-b.yaml")} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("expected error to contain %q, got %v", want, err)
}
}
})
t.Run("nested raw api_key rejected for likely target file", func(t *testing.T) {
nestedDir := filepath.Join(tmpDir, "secure")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writeProfileTestFile(t, filepath.Join(nestedDir, "not_named_like_id.yaml"), `
id: nested_raw_api_key
endpoint: http://localhost:8000/v1
model: m
api_key: secret
`)
_, err := repo.GetProfile(ctx, "nested_raw_api_key")
if !errors.Is(err, ErrRawAPIKeyNotAllowed) {
t.Fatalf("expected ErrRawAPIKeyNotAllowed, got %v", err)
}
if !strings.Contains(err.Error(), filepath.Join("secure", "not_named_like_id.yaml")) {
t.Fatalf("expected nested path in error, got %v", err)
}
})
t.Run("raw api_key in non-target profile is ignored", func(t *testing.T) {
writeProfileTestFile(t, filepath.Join(tmpDir, "raw-api-key-non-target.yaml"), `
id: raw-api-key-non-target
endpoint: http://localhost:8000/v1
model: m
api_key: secret
`)
_, err := repo.GetProfile(ctx, "does-not-exist-with-raw-key-nearby")
if !errors.Is(err, ErrProfileNotFound) {
t.Fatalf("expected ErrProfileNotFound for non-target raw api_key file, got %v", err)
}
}) })
t.Run("invalid yaml", func(t *testing.T) { t.Run("invalid yaml", func(t *testing.T) {
@@ -109,3 +196,10 @@ func TestFilesystemRepository_GetProfile(t *testing.T) {
} }
}) })
} }
func writeProfileTestFile(t *testing.T, path string, content string) {
t.Helper()
if err := os.WriteFile(path, []byte(strings.TrimLeft(content, "\n")), 0o644); err != nil {
t.Fatalf("failed to write profile test file %q: %v", path, err)
}
}

View File

@@ -2,6 +2,7 @@ id: local-secure
endpoint: http://localhost:8000/v1 endpoint: http://localhost:8000/v1
model: gpt-4o-mini model: gpt-4o-mini
api_key_env: SCRIPTORIUM_API_KEY api_key_env: SCRIPTORIUM_API_KEY
service_tier: priority
reasoning_effort: medium reasoning_effort: medium
extra_params: extra_params:
provider: local provider: local

View File

@@ -10,6 +10,7 @@ import (
"strings" "strings"
"gitea.maximumdirect.net/eric/scriptorium/internal/domain" "gitea.maximumdirect.net/eric/scriptorium/internal/domain"
"gitea.maximumdirect.net/eric/scriptorium/internal/filecatalog"
"gopkg.in/yaml.v3" "gopkg.in/yaml.v3"
) )
@@ -62,29 +63,26 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition) return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition)
} }
files, err := os.ReadDir(r.dir) files, err := filecatalog.FindYAMLFiles(ctx, r.dir)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read prompt definition directory: %w", err) return nil, fmt.Errorf("failed to read prompt definition directory: %w", err)
} }
for _, file := range files { var matches []promptDefinitionMatch
for _, fullPath := range files {
select { select {
case <-ctx.Done(): case <-ctx.Done():
return nil, ctx.Err() return nil, ctx.Err()
default: default:
} }
if file.IsDir() || !isYAMLFile(file.Name()) { relPath := filecatalog.RelativePath(r.dir, fullPath)
continue fileMatch := filecatalog.Stem(filepath.Base(fullPath)) == id
}
fullPath := filepath.Join(r.dir, file.Name())
fileMatch := promptIDFromFileName(file.Name()) == id
raw, err := loadPromptDefinitionFile(fullPath) raw, err := loadPromptDefinitionFile(fullPath)
if err != nil { if err != nil {
if fileMatch { if fileMatch || promptDefinitionFileHasID(fullPath, id) {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, file.Name(), err) return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
} }
continue continue
} }
@@ -92,7 +90,7 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
def, err := normalizePromptDefinition(raw, fullPath) def, err := normalizePromptDefinition(raw, fullPath)
if err != nil { if err != nil {
if fileMatch || strings.TrimSpace(raw.ID) == id { if fileMatch || strings.TrimSpace(raw.ID) == id {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, file.Name(), err) return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, relPath, err)
} }
continue continue
} }
@@ -103,12 +101,35 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
if version != "" && def.Version != version { if version != "" && def.Version != version {
continue continue
} }
return def, nil matches = append(matches, promptDefinitionMatch{
def: def,
path: relPath,
})
}
if len(matches) > 1 {
paths := make([]string, 0, len(matches))
for _, match := range matches {
paths = append(paths, match.path)
}
if version != "" {
return nil, fmt.Errorf("%w: duplicate prompt definition id %q version %q found in: %s", ErrInvalidPromptDefinition, id, version, strings.Join(paths, ", "))
}
return nil, fmt.Errorf("%w: duplicate prompt definition id %q found in: %s", ErrInvalidPromptDefinition, id, strings.Join(paths, ", "))
}
if len(matches) == 1 {
return matches[0].def, nil
} }
return nil, ErrPromptDefinitionNotFound return nil, ErrPromptDefinitionNotFound
} }
type promptDefinitionMatch struct {
def *domain.PromptDefinition
path string
}
func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) { func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
data, err := os.ReadFile(path) data, err := os.ReadFile(path)
if err != nil { if err != nil {
@@ -124,6 +145,20 @@ func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
return &raw, nil return &raw, nil
} }
func promptDefinitionFileHasID(path string, id string) bool {
data, err := os.ReadFile(path)
if err != nil {
return false
}
var raw struct {
ID string `yaml:"id"`
}
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&raw); err != nil {
return false
}
return strings.TrimSpace(raw.ID) == id
}
func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*domain.PromptDefinition, error) { func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*domain.PromptDefinition, error) {
if raw == nil { if raw == nil {
return nil, errors.New("prompt definition is nil") return nil, errors.New("prompt definition is nil")
@@ -239,16 +274,6 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
}, nil }, nil
} }
func isYAMLFile(name string) bool {
return strings.HasSuffix(name, ".yaml") || strings.HasSuffix(name, ".yml")
}
func promptIDFromFileName(name string) string {
name = strings.TrimSuffix(name, ".yaml")
name = strings.TrimSuffix(name, ".yml")
return name
}
func isValidOutputFormat(f domain.OutputFormat) bool { func isValidOutputFormat(f domain.OutputFormat) bool {
switch f { switch f {
case domain.FormatText, domain.FormatMarkdown, domain.FormatJSON: case domain.FormatText, domain.FormatMarkdown, domain.FormatJSON:

View File

@@ -68,6 +68,39 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
} }
}) })
t.Run("valid nested file-backed prompt resolves content file relative to nested YAML", func(t *testing.T) {
nestedDir := filepath.Join(tmpDir, "dnd", "recap")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writePromptTestFile(t, filepath.Join(nestedDir, "nested_recap.yaml"), `
id: nested-recap
version: "1.0.0"
messages:
- role: user
content_file: ./nested_recap.user.tmpl
output:
format: markdown
validation_mode: basic
repair_attempts: 0
`)
writePromptTestFile(t, filepath.Join(nestedDir, "nested_recap.user.tmpl"), `Nested recap: {{input "transcript"}}`)
p, err := repo.GetPromptDefinition(ctx, "nested-recap", "")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if len(p.Templates) != 1 {
t.Fatalf("expected one template, got %d", len(p.Templates))
}
if !strings.Contains(p.Templates[0].Content, "Nested recap") {
t.Fatalf("expected nested content file body, got %q", p.Templates[0].Content)
}
if !strings.Contains(p.Templates[0].ContentFile, filepath.Join("dnd", "recap", "nested_recap.user.tmpl")) {
t.Fatalf("expected nested content file path, got %q", p.Templates[0].ContentFile)
}
})
t.Run("prompt with default_profile", func(t *testing.T) { t.Run("prompt with default_profile", func(t *testing.T) {
p, err := repo.GetPromptDefinition(ctx, "with-default-profile", "") p, err := repo.GetPromptDefinition(ctx, "with-default-profile", "")
if err != nil { if err != nil {
@@ -84,6 +117,124 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
} }
}) })
t.Run("duplicate prompt IDs fail as ambiguous", func(t *testing.T) {
writePromptTestFile(t, filepath.Join(tmpDir, "duplicate_a.yaml"), `
id: duplicate-prompt
version: "1.0.0"
messages:
- role: user
content: First duplicate.
output:
format: markdown
validation_mode: basic
repair_attempts: 0
`)
nestedDir := filepath.Join(tmpDir, "nested")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writePromptTestFile(t, filepath.Join(nestedDir, "duplicate_b.yaml"), `
id: duplicate-prompt
version: "2.0.0"
messages:
- role: user
content: Second duplicate.
output:
format: markdown
validation_mode: basic
repair_attempts: 0
`)
_, err := repo.GetPromptDefinition(ctx, "duplicate-prompt", "")
if !errors.Is(err, ErrInvalidPromptDefinition) {
t.Fatalf("expected duplicate prompt to return ErrInvalidPromptDefinition, got %v", err)
}
for _, want := range []string{"duplicate prompt definition id", "duplicate_a.yaml", filepath.Join("nested", "duplicate_b.yaml")} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("expected error to contain %q, got %v", want, err)
}
}
})
t.Run("duplicate prompt ID and requested version fails as ambiguous", func(t *testing.T) {
writePromptTestFile(t, filepath.Join(tmpDir, "version_duplicate_a.yaml"), `
id: duplicate-version-prompt
version: "1.0.0"
messages:
- role: user
content: First duplicate version.
output:
format: markdown
validation_mode: basic
repair_attempts: 0
`)
nestedDir := filepath.Join(tmpDir, "versioned")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writePromptTestFile(t, filepath.Join(nestedDir, "version_duplicate_b.yaml"), `
id: duplicate-version-prompt
version: "1.0.0"
messages:
- role: user
content: Second duplicate version.
output:
format: markdown
validation_mode: basic
repair_attempts: 0
`)
_, err := repo.GetPromptDefinition(ctx, "duplicate-version-prompt", "1.0.0")
if !errors.Is(err, ErrInvalidPromptDefinition) {
t.Fatalf("expected duplicate prompt version to return ErrInvalidPromptDefinition, got %v", err)
}
for _, want := range []string{"duplicate prompt definition id", "version \"1.0.0\"", "version_duplicate_a.yaml", filepath.Join("versioned", "version_duplicate_b.yaml")} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("expected error to contain %q, got %v", want, err)
}
}
})
t.Run("non-matching malformed nested prompt is ignored for not found lookup", func(t *testing.T) {
nestedDir := filepath.Join(tmpDir, "broken")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writePromptTestFile(t, filepath.Join(nestedDir, "unrelated.yaml"), "id: [")
_, err := repo.GetPromptDefinition(ctx, "does-not-exist-even-with-broken-nested-file", "")
if !errors.Is(err, ErrPromptDefinitionNotFound) {
t.Fatalf("expected ErrPromptDefinitionNotFound, got %v", err)
}
})
t.Run("strict decode failure in nested prompt matches by YAML ID", func(t *testing.T) {
nestedDir := filepath.Join(tmpDir, "strict")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
writePromptTestFile(t, filepath.Join(nestedDir, "not_named_like_id.yaml"), `
id: nested-strict-error
version: "1.0.0"
unknown_field: true
messages:
- role: user
content: Invalid because of unknown field.
output:
format: markdown
validation_mode: basic
repair_attempts: 0
`)
_, err := repo.GetPromptDefinition(ctx, "nested-strict-error", "")
if !errors.Is(err, ErrInvalidYAML) {
t.Fatalf("expected ErrInvalidYAML, got %v", err)
}
if !strings.Contains(err.Error(), filepath.Join("strict", "not_named_like_id.yaml")) {
t.Fatalf("expected nested path in error, got %v", err)
}
})
t.Run("version lookup", func(t *testing.T) { t.Run("version lookup", func(t *testing.T) {
_, err := repo.GetPromptDefinition(ctx, "valid-inline", "9.9.9") _, err := repo.GetPromptDefinition(ctx, "valid-inline", "9.9.9")
if !errors.Is(err, ErrPromptDefinitionNotFound) { if !errors.Is(err, ErrPromptDefinitionNotFound) {
@@ -131,6 +282,13 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
}) })
} }
func writePromptTestFile(t *testing.T, path string, content string) {
t.Helper()
if err := os.WriteFile(path, []byte(strings.TrimLeft(content, "\n")), 0o644); err != nil {
t.Fatalf("failed to write prompt test file %q: %v", path, err)
}
}
func copyTree(src, dst string) error { func copyTree(src, dst string) error {
return filepath.WalkDir(src, func(path string, d fs.DirEntry, err error) error { return filepath.WalkDir(src, func(path string, d fs.DirEntry, err error) error {
if err != nil { if err != nil {

View File

@@ -35,9 +35,9 @@ func TestRunnerIntegrationWithPromptAndProfileFixturesAndValidation(t *testing.T
t.Fatalf("failed to resolve repo root: %v", err) t.Fatalf("failed to resolve repo root: %v", err)
} }
promptsDir := filepath.Join(root, "prompts") promptsDir := filepath.Join(root, "examples", "prompts")
profilesDir := filepath.Join(root, "profiles") profilesDir := filepath.Join(root, "examples", "profiles")
schemasDir := filepath.Join(root, "schemas") schemasDir := filepath.Join(root, "examples", "schemas")
fixturesDir := filepath.Join(root, "examples", "fixtures") fixturesDir := filepath.Join(root, "examples", "fixtures")
t.Setenv("SCRIPTORIUM_API_KEY", "test-key") t.Setenv("SCRIPTORIUM_API_KEY", "test-key")

View File

@@ -25,6 +25,8 @@ import (
var ( var (
ErrInvalidRequest = errors.New("invalid run request") ErrInvalidRequest = errors.New("invalid run request")
ErrProfileRequired = errors.New("profile selection is required")
ErrAPIKeyEnvMissing = errors.New("api_key_env points to an unset environment variable")
ErrProfileLoad = errors.New("failed to load prompt definition") ErrProfileLoad = errors.New("failed to load prompt definition")
ErrArtifactLoad = errors.New("failed to load artifact") ErrArtifactLoad = errors.New("failed to load artifact")
ErrPromptRender = errors.New("failed to render prompt") ErrPromptRender = errors.New("failed to render prompt")
@@ -177,7 +179,7 @@ func (r *Runner) Prepare(ctx context.Context, req domain.RunRequest) (*domain.Pr
selectedProfileID = strings.TrimSpace(def.DefaultProfile) selectedProfileID = strings.TrimSpace(def.DefaultProfile)
} }
if selectedProfileID == "" { if selectedProfileID == "" {
return nil, fmt.Errorf("%w: profile id is required either in request or prompt default_profile", ErrInvalidRequest) return nil, fmt.Errorf("%w: %w: profile id is required either in request or prompt default_profile", ErrInvalidRequest, ErrProfileRequired)
} }
execProfile, err := r.profiles.GetProfile(ctx, selectedProfileID) execProfile, err := r.profiles.GetProfile(ctx, selectedProfileID)
@@ -342,6 +344,9 @@ func mergeExecutionTarget(base domain.ExecutionTarget, override domain.Execution
if override.TimeoutSeconds != 0 { if override.TimeoutSeconds != 0 {
out.TimeoutSeconds = override.TimeoutSeconds out.TimeoutSeconds = override.TimeoutSeconds
} }
if strings.TrimSpace(override.ServiceTier) != "" {
out.ServiceTier = override.ServiceTier
}
if strings.TrimSpace(override.ReasoningEffort) != "" { if strings.TrimSpace(override.ReasoningEffort) != "" {
out.ReasoningEffort = override.ReasoningEffort out.ReasoningEffort = override.ReasoningEffort
} }
@@ -373,7 +378,7 @@ func validateAPIKeyEnv(apiKeyEnv string) error {
return nil return nil
} }
if strings.TrimSpace(os.Getenv(envName)) == "" { if strings.TrimSpace(os.Getenv(envName)) == "" {
return fmt.Errorf("api key environment variable %q is not set", envName) return fmt.Errorf("%w: api key environment variable %q is not set", ErrAPIKeyEnvMissing, envName)
} }
return nil return nil
} }
@@ -396,6 +401,7 @@ func executionProfileToTarget(p *domain.ExecutionProfile) domain.ExecutionTarget
MaxTokens: p.MaxTokens, MaxTokens: p.MaxTokens,
TopP: p.TopP, TopP: p.TopP,
TimeoutSeconds: p.TimeoutSeconds, TimeoutSeconds: p.TimeoutSeconds,
ServiceTier: p.ServiceTier,
ReasoningEffort: p.ReasoningEffort, ReasoningEffort: p.ReasoningEffort,
APIKeyEnv: p.APIKeyEnv, APIKeyEnv: p.APIKeyEnv,
ExtraParams: cp, ExtraParams: cp,

View File

@@ -234,6 +234,9 @@ func TestRunnerPrepareMissingExplicitProfileAndMissingDefaultProfileFails(t *tes
if !errors.Is(err, ErrInvalidRequest) { if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected ErrInvalidRequest, got %v", err) t.Fatalf("expected ErrInvalidRequest, got %v", err)
} }
if !errors.Is(err, ErrProfileRequired) {
t.Fatalf("expected ErrProfileRequired, got %v", err)
}
} }
func TestRunnerPrepareSelectedProfileDoesNotExistFails(t *testing.T) { func TestRunnerPrepareSelectedProfileDoesNotExistFails(t *testing.T) {
@@ -257,6 +260,7 @@ func TestRunnerPrepareRuntimeOverrideBeatsSelectedProfileValue(t *testing.T) {
MaxTokens: 500, MaxTokens: 500,
TopP: 0.9, TopP: 0.9,
TimeoutSeconds: 120, TimeoutSeconds: 120,
ServiceTier: "priority",
}, },
}} }}
runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{forbid: true}, nil) runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{forbid: true}, nil)
@@ -270,6 +274,7 @@ func TestRunnerPrepareRuntimeOverrideBeatsSelectedProfileValue(t *testing.T) {
Model: "override-model", Model: "override-model",
Temperature: 0.7, Temperature: 0.7,
TimeoutSeconds: 30, TimeoutSeconds: 30,
ServiceTier: "flex",
}, },
}) })
if err != nil { if err != nil {
@@ -281,6 +286,9 @@ func TestRunnerPrepareRuntimeOverrideBeatsSelectedProfileValue(t *testing.T) {
if prepared.EffectiveModelParams.TopP != 0.9 { if prepared.EffectiveModelParams.TopP != 0.9 {
t.Fatalf("expected profile top_p to remain, got %v", prepared.EffectiveModelParams.TopP) t.Fatalf("expected profile top_p to remain, got %v", prepared.EffectiveModelParams.TopP)
} }
if prepared.EffectiveModelParams.ServiceTier != "flex" {
t.Fatalf("expected service_tier override to win, got %q", prepared.EffectiveModelParams.ServiceTier)
}
} }
func TestRunnerPrepareSelectedProfileBeatsBuiltInDefault(t *testing.T) { func TestRunnerPrepareSelectedProfileBeatsBuiltInDefault(t *testing.T) {
@@ -292,6 +300,7 @@ func TestRunnerPrepareSelectedProfileBeatsBuiltInDefault(t *testing.T) {
Model: "profile-model", Model: "profile-model",
TopP: 0.8, TopP: 0.8,
TimeoutSeconds: 90, TimeoutSeconds: 90,
ServiceTier: "priority",
}, },
}} }}
runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{forbid: true}, nil) runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{forbid: true}, nil)
@@ -310,6 +319,9 @@ func TestRunnerPrepareSelectedProfileBeatsBuiltInDefault(t *testing.T) {
if prepared.EffectiveModelParams.TimeoutSeconds != 90 { if prepared.EffectiveModelParams.TimeoutSeconds != 90 {
t.Fatalf("expected profile timeout to beat default, got %d", prepared.EffectiveModelParams.TimeoutSeconds) t.Fatalf("expected profile timeout to beat default, got %d", prepared.EffectiveModelParams.TimeoutSeconds)
} }
if prepared.EffectiveModelParams.ServiceTier != "priority" {
t.Fatalf("expected profile service_tier to beat default, got %q", prepared.EffectiveModelParams.ServiceTier)
}
} }
func TestRunnerPrepareFileBackedPromptBodiesRenderCorrectly(t *testing.T) { func TestRunnerPrepareFileBackedPromptBodiesRenderCorrectly(t *testing.T) {
@@ -478,6 +490,35 @@ func TestRunnerPrepareJSONSchemaBuildsStructuredOutputSpec(t *testing.T) {
} }
} }
func TestRunnerPrepareJSONSchemaSchemaLoadFailureReturnsValidationError(t *testing.T) {
def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 0)
def.Validation.SchemaPath = "missing.schema.json"
validator := &fakeValidator{schemaErr: errors.New("schema unavailable")}
runner := NewRunner(
&fakePromptRepo{def: def},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}},
defaultArtifactReader(),
defaultRenderer(),
&fakeLLM{forbid: true},
validator,
)
_, err := runner.Prepare(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
})
if !errors.Is(err, ErrValidation) {
t.Fatalf("expected ErrValidation, got %v", err)
}
if validator.schemaLoads != 1 {
t.Fatalf("expected one schema load attempt, got %d", validator.schemaLoads)
}
if validator.schemaLoadPath != "missing.schema.json" {
t.Fatalf("expected schema path missing.schema.json, got %q", validator.schemaLoadPath)
}
}
func TestRunnerRunJSONSchemaSchemaLoadFailureFailsBeforeLLM(t *testing.T) { func TestRunnerRunJSONSchemaSchemaLoadFailureFailsBeforeLLM(t *testing.T) {
def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 0) def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 0)
def.Validation.SchemaPath = "missing.schema.json" def.Validation.SchemaPath = "missing.schema.json"
@@ -726,6 +767,7 @@ func TestRunnerRunExplicitRuntimeOverrideBeatsSelectedProfileValue(t *testing.T)
MaxTokens: 500, MaxTokens: 500,
TopP: 0.9, TopP: 0.9,
TimeoutSeconds: 120, TimeoutSeconds: 120,
ServiceTier: "priority",
}, },
}} }}
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}} llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}
@@ -740,6 +782,7 @@ func TestRunnerRunExplicitRuntimeOverrideBeatsSelectedProfileValue(t *testing.T)
Model: "override-model", Model: "override-model",
Temperature: 0.7, Temperature: 0.7,
TimeoutSeconds: 30, TimeoutSeconds: 30,
ServiceTier: "flex",
}, },
}) })
if err != nil { if err != nil {
@@ -754,6 +797,9 @@ func TestRunnerRunExplicitRuntimeOverrideBeatsSelectedProfileValue(t *testing.T)
if res.EffectiveModelParams.TopP != 0.9 { if res.EffectiveModelParams.TopP != 0.9 {
t.Fatalf("expected non-overridden profile top_p to remain, got %v", res.EffectiveModelParams.TopP) t.Fatalf("expected non-overridden profile top_p to remain, got %v", res.EffectiveModelParams.TopP)
} }
if res.EffectiveModelParams.ServiceTier != "flex" {
t.Fatalf("expected service_tier override to win, got %q", res.EffectiveModelParams.ServiceTier)
}
} }
func TestRunnerRunSelectedProfileBeatsBuiltInDefault(t *testing.T) { func TestRunnerRunSelectedProfileBeatsBuiltInDefault(t *testing.T) {
@@ -765,6 +811,7 @@ func TestRunnerRunSelectedProfileBeatsBuiltInDefault(t *testing.T) {
Model: "profile-model", Model: "profile-model",
TopP: 0.8, TopP: 0.8,
TimeoutSeconds: 90, TimeoutSeconds: 90,
ServiceTier: "priority",
}, },
}} }}
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}} llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}
@@ -784,6 +831,9 @@ func TestRunnerRunSelectedProfileBeatsBuiltInDefault(t *testing.T) {
if res.EffectiveModelParams.TimeoutSeconds != 90 { if res.EffectiveModelParams.TimeoutSeconds != 90 {
t.Fatalf("expected profile timeout to beat default, got %d", res.EffectiveModelParams.TimeoutSeconds) t.Fatalf("expected profile timeout to beat default, got %d", res.EffectiveModelParams.TimeoutSeconds)
} }
if res.EffectiveModelParams.ServiceTier != "priority" {
t.Fatalf("expected profile service_tier to beat default, got %q", res.EffectiveModelParams.ServiceTier)
}
} }
func TestRunnerRunBuiltInDefaultsUsedWhenProfileOmitsOptionalFields(t *testing.T) { func TestRunnerRunBuiltInDefaultsUsedWhenProfileOmitsOptionalFields(t *testing.T) {
@@ -844,6 +894,9 @@ func TestRunnerRunAPIKeyEnvMissingEnvironmentValueFailsClearly(t *testing.T) {
if !errors.Is(err, ErrInvalidRequest) { if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected ErrInvalidRequest, got %v", err) t.Fatalf("expected ErrInvalidRequest, got %v", err)
} }
if !errors.Is(err, ErrAPIKeyEnvMissing) {
t.Fatalf("expected ErrAPIKeyEnvMissing, got %v", err)
}
if !strings.Contains(err.Error(), "SCRIPTORIUM_MISSING_KEY") { if !strings.Contains(err.Error(), "SCRIPTORIUM_MISSING_KEY") {
t.Fatalf("expected missing env name in error, got %v", err) t.Fatalf("expected missing env name in error, got %v", err)
} }
@@ -1104,6 +1157,174 @@ func TestRunnerRunJSONSchemaRepairCarriesStructuredOutputSpec(t *testing.T) {
} }
} }
func TestExecutionProfileToTargetPopulatesAllFieldsAndCopiesExtraParams(t *testing.T) {
src := &domain.ExecutionProfile{
ID: "exec",
Endpoint: "http://profile/v1",
Model: "profile-model",
Temperature: 0.2,
MaxTokens: 123,
TopP: 0.75,
TimeoutSeconds: 90,
ServiceTier: "priority",
ReasoningEffort: "medium",
APIKeyEnv: "SCRIPTORIUM_API_KEY",
ExtraParams: map[string]string{
"provider_option": "on",
},
}
target := executionProfileToTarget(src)
if target.Endpoint != src.Endpoint ||
target.Model != src.Model ||
target.Temperature != src.Temperature ||
target.MaxTokens != src.MaxTokens ||
target.TopP != src.TopP ||
target.TimeoutSeconds != src.TimeoutSeconds ||
target.ServiceTier != src.ServiceTier ||
target.ReasoningEffort != src.ReasoningEffort ||
target.APIKeyEnv != src.APIKeyEnv {
t.Fatalf("expected all profile fields to populate target, got %+v", target)
}
if !reflect.DeepEqual(target.ExtraParams, src.ExtraParams) {
t.Fatalf("expected extra_params to match, got %#v", target.ExtraParams)
}
src.ExtraParams["provider_option"] = "changed"
if target.ExtraParams["provider_option"] != "on" {
t.Fatalf("expected extra_params copy to be independent, got %#v", target.ExtraParams)
}
}
func TestResolveExecutionTargetProfileValuesPopulateAllSupportedFields(t *testing.T) {
profileValue := &domain.ExecutionProfile{
ID: "exec",
Endpoint: "http://profile/v1",
Model: "profile-model",
Temperature: 0.3,
MaxTokens: 222,
TopP: 0.6,
TimeoutSeconds: 77,
ServiceTier: "priority",
ReasoningEffort: "low",
APIKeyEnv: "PROFILE_KEY",
ExtraParams: map[string]string{
"profile_option": "enabled",
},
}
target := resolveExecutionTarget(profileValue, nil)
if target.Endpoint != profileValue.Endpoint ||
target.Model != profileValue.Model ||
target.Temperature != profileValue.Temperature ||
target.MaxTokens != profileValue.MaxTokens ||
target.TopP != profileValue.TopP ||
target.TimeoutSeconds != profileValue.TimeoutSeconds ||
target.ServiceTier != profileValue.ServiceTier ||
target.ReasoningEffort != profileValue.ReasoningEffort ||
target.APIKeyEnv != profileValue.APIKeyEnv {
t.Fatalf("expected profile values to populate target, got %+v", target)
}
if !reflect.DeepEqual(target.ExtraParams, profileValue.ExtraParams) {
t.Fatalf("expected profile extra_params in target, got %#v", target.ExtraParams)
}
}
func TestResolveExecutionTargetRuntimeOverridesBeatProfileForAllOverrideableFields(t *testing.T) {
profileValue := &domain.ExecutionProfile{
ID: "exec",
Endpoint: "http://profile/v1",
Model: "profile-model",
Temperature: 0.2,
MaxTokens: 200,
TopP: 0.8,
TimeoutSeconds: 90,
ServiceTier: "priority",
ReasoningEffort: "medium",
APIKeyEnv: "PROFILE_KEY",
ExtraParams: map[string]string{
"profile_only": "yes",
},
}
override := &domain.ExecutionTarget{
Endpoint: "http://override/v1",
Model: "override-model",
Temperature: 0.9,
MaxTokens: 111,
TopP: 0.5,
TimeoutSeconds: 30,
ServiceTier: "flex",
ReasoningEffort: "high",
APIKeyEnv: "RUNTIME_KEY",
ExtraParams: map[string]string{
"runtime_only": "yes",
},
}
target := resolveExecutionTarget(profileValue, override)
if target.Endpoint != override.Endpoint ||
target.Model != override.Model ||
target.Temperature != override.Temperature ||
target.MaxTokens != override.MaxTokens ||
target.TopP != override.TopP ||
target.TimeoutSeconds != override.TimeoutSeconds ||
target.ServiceTier != override.ServiceTier ||
target.ReasoningEffort != override.ReasoningEffort ||
target.APIKeyEnv != override.APIKeyEnv {
t.Fatalf("expected runtime overrides to win for all fields, got %+v", target)
}
if !reflect.DeepEqual(target.ExtraParams, override.ExtraParams) {
t.Fatalf("expected runtime extra_params to replace profile extra_params, got %#v", target.ExtraParams)
}
}
func TestMergeExecutionTargetEmptyStringOverridesDoNotErase(t *testing.T) {
base := domain.ExecutionTarget{
Endpoint: "http://base/v1",
Model: "base-model",
ServiceTier: "priority",
ReasoningEffort: "medium",
APIKeyEnv: "BASE_KEY",
}
override := domain.ExecutionTarget{
Endpoint: "http://override/v1",
Model: "override-model",
ServiceTier: " ",
ReasoningEffort: " ",
APIKeyEnv: "",
}
merged := mergeExecutionTarget(base, override)
if merged.Endpoint != "http://override/v1" || merged.Model != "override-model" {
t.Fatalf("expected endpoint/model to override, got %+v", merged)
}
if merged.ServiceTier != "priority" {
t.Fatalf("expected empty service_tier override to be ignored, got %q", merged.ServiceTier)
}
if merged.ReasoningEffort != "medium" {
t.Fatalf("expected empty reasoning_effort override to be ignored, got %q", merged.ReasoningEffort)
}
if merged.APIKeyEnv != "BASE_KEY" {
t.Fatalf("expected empty api_key_env override to be ignored, got %q", merged.APIKeyEnv)
}
}
func TestMergeExecutionTargetEmptyExtraParamsDoesNotErase(t *testing.T) {
base := domain.ExecutionTarget{
ExtraParams: map[string]string{
"keep": "value",
},
}
override := domain.ExecutionTarget{
ExtraParams: map[string]string{},
}
merged := mergeExecutionTarget(base, override)
if !reflect.DeepEqual(merged.ExtraParams, base.ExtraParams) {
t.Fatalf("expected empty extra_params override not to erase base values, got %#v", merged.ExtraParams)
}
}
func TestBuildOutputArtifactDefaults(t *testing.T) { func TestBuildOutputArtifactDefaults(t *testing.T) {
tests := []struct { tests := []struct {
name string name string

View File

@@ -116,6 +116,49 @@ func TestStandardValidatorJSONSchemaSuccess(t *testing.T) {
} }
} }
func TestStandardValidatorJSONSchemaNestedSchemaPathSuccess(t *testing.T) {
tmp := t.TempDir()
nestedDir := filepath.Join(tmp, "dnd")
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(nestedDir, "schema.json"), []byte(`{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["name"],
"properties": {
"name": {"type": "string"}
}
}`), 0644); err != nil {
t.Fatal(err)
}
v := NewStandardValidator(tmp)
res, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"name":"eris"}`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: filepath.Join("dnd", "schema.json"),
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if res.Status != domain.ValidationPassed || !res.IsValid {
t.Fatalf("expected passed/valid, got status=%q valid=%v", res.Status, res.IsValid)
}
}
func TestStandardValidatorJSONSchemaNestedSchemaPathMissing(t *testing.T) {
v := NewStandardValidator(t.TempDir())
_, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"name":"eris"}`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: filepath.Join("dnd", "missing.json"),
})
if err == nil {
t.Fatal("expected nested schema load error")
}
}
func TestStandardValidatorJSONSchemaFailure(t *testing.T) { func TestStandardValidatorJSONSchemaFailure(t *testing.T) {
tmp := t.TempDir() tmp := t.TempDir()
schemaPath := filepath.Join(tmp, "schema.json") schemaPath := filepath.Join(tmp, "schema.json")