Compare commits
15 Commits
c3fe88c9fa
...
v0.9.0
| Author | SHA1 | Date | |
|---|---|---|---|
| bc099a31ad | |||
| 4ff55221a3 | |||
| 8d8024099f | |||
| 18792fd8d1 | |||
| 8860aa033c | |||
| 3ca14d8b6e | |||
| 099e9c4a3e | |||
| cfe6b9408a | |||
| 6ececc749f | |||
| 79901fbb86 | |||
| 75fa0a030a | |||
| ef64966897 | |||
| c6c5e3cb69 | |||
| 3f4fd230b9 | |||
| 2091b58066 |
@@ -29,7 +29,6 @@ This command renders the prepared prompt and effective runtime settings without
|
||||
- [OpenAI-compatible chat integration](docs/integrations/openai-compatible-chat.md)
|
||||
- [Narratio subprocess integration](docs/integrations/narratio.md)
|
||||
- [Architecture policy](docs/policy/architecture.md)
|
||||
- [Documentation roadmap](docs/roadmap/documentation.md)
|
||||
|
||||
## Examples
|
||||
|
||||
|
||||
@@ -1,14 +0,0 @@
|
||||
# Architecture (Moved)
|
||||
|
||||
The canonical architecture policy is:
|
||||
|
||||
- `docs/policy/architecture.md`
|
||||
|
||||
Implemented internal component documentation is:
|
||||
|
||||
- `docs/internal/runner.md`
|
||||
- `docs/internal/adapters.md`
|
||||
|
||||
Roadmap-only planning content is:
|
||||
|
||||
- `docs/roadmap/documentation.md`
|
||||
@@ -20,8 +20,8 @@ go run ./cmd/scriptorium render \
|
||||
|
||||
Integration references:
|
||||
|
||||
- HTTP contract: `docs/integrations/http-api.md`
|
||||
- Narratio subprocess contract: `docs/integrations/narratio.md`
|
||||
- [HTTP contract](integrations/http-api.md)
|
||||
- [Narratio subprocess contract](integrations/narratio.md)
|
||||
|
||||
## Common Argument Rules
|
||||
|
||||
@@ -134,7 +134,7 @@ go run ./cmd/scriptorium run \
|
||||
--profile local-fast \
|
||||
--input transcript=./examples/fixtures/transcript.md \
|
||||
--input glossary=./examples/fixtures/glossary.yml \
|
||||
--out ./out/summary.md
|
||||
--out ./summary.md
|
||||
```
|
||||
|
||||
Start the HTTP server with explicit config:
|
||||
|
||||
@@ -20,8 +20,8 @@ When `--config <path>` is provided, that file is required.
|
||||
## Minimal App Config
|
||||
|
||||
```yaml
|
||||
prompt_dir: ./prompts
|
||||
profile_dir: ./profiles
|
||||
prompt_dir: ./examples/prompts
|
||||
profile_dir: ./examples/profiles
|
||||
```
|
||||
|
||||
This is enough to use `run` and `render` when prompt/profile files are valid.
|
||||
@@ -63,7 +63,9 @@ Validation behavior:
|
||||
|
||||
## Prompt Definition Files
|
||||
|
||||
Prompt definitions are YAML files in `prompt_dir`.
|
||||
Prompt definitions are YAML files anywhere under `prompt_dir`, including nested subdirectories.
|
||||
|
||||
Subdirectories are organizational only. Callers still select prompts by the YAML `id`, not by file path. For example, `prompts/dnd/recap.yaml` may still declare `id: dnd.recap`, and callers use `--prompt dnd.recap`.
|
||||
|
||||
Example:
|
||||
|
||||
@@ -122,7 +124,9 @@ Message rules:
|
||||
|
||||
- Repeated roles are allowed.
|
||||
- `content_file` is resolved relative to the prompt YAML file location.
|
||||
- Nested prompt files keep the same relative `content_file` behavior; `./recap.user.md` next to `dnd/recap.yaml` resolves from `dnd/`.
|
||||
- Prompt decoding is strict; unknown YAML fields are rejected.
|
||||
- Duplicate prompt IDs are invalid. If multiple files declare the requested prompt ID, Scriptorium fails instead of choosing one.
|
||||
|
||||
`output` fields:
|
||||
|
||||
@@ -138,7 +142,9 @@ Repair behavior boundary:
|
||||
|
||||
## Profile Definition Files
|
||||
|
||||
Execution profiles are YAML files in `profile_dir`.
|
||||
Execution profiles are YAML files anywhere under `profile_dir`, including nested subdirectories.
|
||||
|
||||
Subdirectories are organizational only. Callers still select profiles by the YAML `id`, not by file path. For example, `profiles/local/local-quality.yaml` may still declare `id: local-quality`, and callers use `--profile local-quality`.
|
||||
|
||||
Example:
|
||||
|
||||
@@ -151,6 +157,7 @@ max_tokens: 500
|
||||
top_p: 1.0
|
||||
timeout_seconds: 90
|
||||
api_key_env: SCRIPTORIUM_API_KEY
|
||||
service_tier: priority
|
||||
```
|
||||
|
||||
Field reference:
|
||||
@@ -162,6 +169,7 @@ Field reference:
|
||||
- `max_tokens` (optional): `>= 0`
|
||||
- `top_p` (optional): range `0..1`
|
||||
- `timeout_seconds` (optional): `>= 0`
|
||||
- `service_tier` (optional): provider-specific request tier such as OpenRouter `flex` or `priority`
|
||||
- `reasoning_effort` (optional)
|
||||
- `api_key_env` (optional)
|
||||
- `extra_params` (optional map of strings)
|
||||
@@ -171,10 +179,11 @@ Profile rules:
|
||||
- Profile decoding is strict; unknown YAML fields are rejected.
|
||||
- Raw `api_key` is rejected; use `api_key_env`.
|
||||
- If `api_key_env` is set, that environment variable must be set when preparing/running.
|
||||
- Duplicate profile IDs are invalid. If multiple files declare the requested profile ID, Scriptorium fails instead of choosing one.
|
||||
|
||||
Current outbound request behavior:
|
||||
|
||||
- The OpenAI-compatible client currently serializes: `model`, `messages`, `temperature`, `max_tokens`, `top_p`, and optional `response_format` for `json_schema` prompts.
|
||||
- The OpenAI-compatible client currently serializes: `model`, `messages`, `temperature`, `max_tokens`, `top_p`, `service_tier`, and optional `response_format` for `json_schema` prompts.
|
||||
- `reasoning_effort` and `extra_params` are parsed and carried in effective settings, but are not currently serialized into outbound chat-completions requests.
|
||||
|
||||
## Schema Behavior
|
||||
@@ -184,8 +193,9 @@ Schemas are JSON files, typically in `schema_dir`.
|
||||
Rules:
|
||||
|
||||
- `output.validation_mode: json_schema` requires `output.schema_path`.
|
||||
- Relative `schema_path` values resolve from `schema_dir`.
|
||||
- Relative `schema_path` values resolve from `schema_dir`, including explicit nested paths such as `dnd/structured_events.schema.json`.
|
||||
- Absolute `schema_path` values are used directly.
|
||||
- Scriptorium does not recursively search schemas by basename; nested schemas must be referenced by their relative path.
|
||||
- Missing or invalid schema documents cause runtime validation errors.
|
||||
- Invalid generated JSON causes validation status `failed` (not a runtime error).
|
||||
|
||||
@@ -200,14 +210,22 @@ Supported artifact reference types for request inputs are `file` and `inline`.
|
||||
## Maintained Examples
|
||||
|
||||
- App config: `examples/config.yml`
|
||||
- Prompt examples: `prompts/`
|
||||
- Profile examples: `profiles/`
|
||||
- Schema examples: `schemas/`
|
||||
- Prompt examples: `examples/prompts/`
|
||||
- Profile examples: `examples/profiles/`
|
||||
- Schema examples: `examples/schemas/`
|
||||
- Input fixtures: `examples/fixtures/`
|
||||
- Render example script: `examples/render-markdown-summary.sh`
|
||||
- HTTP request example: `examples/http-run.json`
|
||||
|
||||
Example organizational layout:
|
||||
|
||||
```text
|
||||
examples/prompts/dnd/recap.yaml
|
||||
examples/profiles/local/local-quality.yaml
|
||||
examples/schemas/dnd/structured_events.schema.json
|
||||
```
|
||||
|
||||
## Integration References
|
||||
|
||||
- Inbound HTTP contract: `docs/integrations/http-api.md`
|
||||
- Outbound OpenAI-compatible contract: `docs/integrations/openai-compatible-chat.md`
|
||||
- [Inbound HTTP contract](integrations/http-api.md)
|
||||
- [Outbound OpenAI-compatible contract](integrations/openai-compatible-chat.md)
|
||||
|
||||
@@ -8,7 +8,7 @@ Current scope is only:
|
||||
|
||||
- `POST /v1/runs`
|
||||
|
||||
For CLI behavior, see `docs/cli.md`.
|
||||
For CLI behavior, see the [CLI reference](../cli.md).
|
||||
|
||||
## Endpoint
|
||||
|
||||
@@ -46,6 +46,7 @@ Copyable request example file:
|
||||
"max_tokens": 800,
|
||||
"top_p": 1.0,
|
||||
"timeout_seconds": 120,
|
||||
"service_tier": "priority",
|
||||
"reasoning_effort": "medium",
|
||||
"api_key_env": "SCRIPTORIUM_API_KEY",
|
||||
"extra_params": {
|
||||
@@ -114,6 +115,7 @@ Response shape:
|
||||
"max_tokens": 800,
|
||||
"top_p": 1,
|
||||
"timeout_seconds": 120,
|
||||
"service_tier": "priority",
|
||||
"reasoning_effort": "medium",
|
||||
"api_key_env": "SCRIPTORIUM_API_KEY",
|
||||
"extra_params": {
|
||||
|
||||
@@ -109,6 +109,6 @@ A `run` exit code `2` can still produce output (stdout or `--out`).
|
||||
|
||||
## Canonical References
|
||||
|
||||
- CLI behavior: `docs/cli.md`
|
||||
- Config behavior: `docs/config.md`
|
||||
- Operations and failure handling: `docs/operations.md`, `docs/troubleshooting.md`
|
||||
- CLI behavior: [CLI reference](../cli.md)
|
||||
- Config behavior: [Configuration reference](../config.md)
|
||||
- Operations and failure handling: [Operations guide](../operations.md), [Troubleshooting](../troubleshooting.md)
|
||||
|
||||
@@ -30,8 +30,11 @@ Serialized JSON fields:
|
||||
- `temperature` (only when non-zero)
|
||||
- `max_tokens` (only when non-zero)
|
||||
- `top_p` (only when non-zero)
|
||||
- `service_tier` (only when non-empty)
|
||||
- `response_format` (only when structured output is provided)
|
||||
|
||||
`service_tier` is provider-specific. OpenRouter currently documents request values such as `flex` and `priority`; Scriptorium forwards any non-empty configured value and lets the backend validate support.
|
||||
|
||||
Structured output is currently `json_schema` only, serialized as:
|
||||
|
||||
```json
|
||||
|
||||
@@ -30,7 +30,7 @@ HTTP adapter:
|
||||
|
||||
Filesystem repositories:
|
||||
|
||||
- Input: prompt/profile YAML files.
|
||||
- Input: prompt/profile YAML files under configured directories.
|
||||
- Output: normalized domain definitions/profiles or typed errors.
|
||||
|
||||
Artifact reader:
|
||||
@@ -66,7 +66,7 @@ Primary app settings consumed by adapters:
|
||||
|
||||
Execution profile/request settings used through runner:
|
||||
|
||||
- `endpoint`, `model`, `temperature`, `max_tokens`, `top_p`, `timeout_seconds`, `api_key_env`, `reasoning_effort`, `extra_params`
|
||||
- `endpoint`, `model`, `temperature`, `max_tokens`, `top_p`, `timeout_seconds`, `service_tier`, `api_key_env`, `reasoning_effort`, `extra_params`
|
||||
|
||||
## External Dependencies
|
||||
|
||||
@@ -79,6 +79,9 @@ Execution profile/request settings used through runner:
|
||||
Strict decoding and input checks:
|
||||
|
||||
- config/prompt/profile loaders reject unknown YAML fields.
|
||||
- prompt/profile repositories scan nested subdirectories recursively.
|
||||
- prompt/profile lookup uses YAML `id` values; subdirectory paths are organizational only.
|
||||
- duplicate prompt/profile IDs are invalid and fail instead of using first-match behavior.
|
||||
- HTTP DTO decoder rejects unknown JSON fields.
|
||||
- raw API key payload fields are rejected by strict decoding in profile/http paths.
|
||||
|
||||
@@ -97,10 +100,12 @@ Validator:
|
||||
|
||||
- `basic`, `json`, `json_schema` content failures return `ValidationFailed` results.
|
||||
- schema load/compile/path failures are runtime errors.
|
||||
- schema lookup uses explicit `schema_path` values relative to `schema_dir`; it does not recursively search by basename.
|
||||
|
||||
HTTP error mapping:
|
||||
|
||||
- maps domain/use-case errors to stable HTTP code + error code/message.
|
||||
- distinguishes missing profile selection and missing `api_key_env` variable using stable use-case sentinel errors.
|
||||
- avoids returning internal wrapped-cause details in response payload.
|
||||
|
||||
## CLI Adapter Semantics
|
||||
@@ -136,4 +141,4 @@ Behavior highlights:
|
||||
- Adapter packages do not own runner decision logic.
|
||||
- External request/response strictness is part of contract stability.
|
||||
- Prepared-render output never includes resolved API key values.
|
||||
- Outbound OpenAI-compatible request includes only currently serialized fields (`model`, `messages`, optional `temperature`, `max_tokens`, `top_p`, optional `response_format`).
|
||||
- Outbound OpenAI-compatible request includes only currently serialized fields (`model`, `messages`, optional `temperature`, `max_tokens`, `top_p`, optional `service_tier`, optional `response_format`).
|
||||
|
||||
@@ -66,14 +66,21 @@ It receives fully constructed repositories/readers/validators from adapters. Eff
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Key error classes surfaced from `Runner`:
|
||||
Primary runner error classes:
|
||||
|
||||
- `ErrInvalidRequest`: invalid prompt/profile/request/runtime/API-key-env prerequisites.
|
||||
- `ErrProfileLoad`: prompt or profile load failures.
|
||||
- `ErrInvalidRequest`: invalid run request envelope.
|
||||
- `ErrProfileRequired`: specific invalid-request reason when neither request `profile_id` nor prompt `default_profile` is available.
|
||||
- `ErrAPIKeyEnvMissing`: specific invalid-request reason when `api_key_env` is set but the named environment variable is unset/empty.
|
||||
- `ErrProfileLoad`: prompt/profile repository load failures.
|
||||
- `ErrArtifactLoad`: artifact read failures.
|
||||
- `ErrPromptRender`: template render failures.
|
||||
- `ErrLLMGenerate`: model request failures.
|
||||
- `ErrValidation`: validation runtime failures (including schema load/compile failures).
|
||||
- `ErrLLMGenerate`: outbound model request failures.
|
||||
- `ErrValidation`: validation runtime failures (including structured-output schema load/compile failures).
|
||||
|
||||
Reason sentinel behavior:
|
||||
|
||||
- `ErrProfileRequired` and `ErrAPIKeyEnvMissing` are wrapped with `ErrInvalidRequest`.
|
||||
- Adapters can use `errors.Is` for stable reason mapping without matching runner prose.
|
||||
|
||||
Validation content failures are not run errors:
|
||||
|
||||
@@ -90,13 +97,15 @@ Validation content failures are not run errors:
|
||||
3. select profile ID:
|
||||
- explicit request profile ID
|
||||
- prompt `default_profile`
|
||||
- otherwise request error
|
||||
- otherwise return an invalid request with `ErrProfileRequired`
|
||||
4. load execution profile.
|
||||
5. merge effective runtime target:
|
||||
- built-in execution defaults
|
||||
- selected profile values
|
||||
- request overrides
|
||||
6. verify required `api_key_env` environment variable (name only; value is not returned).
|
||||
6. verify required `api_key_env` environment variable:
|
||||
- missing/empty env value returns an invalid request with `ErrAPIKeyEnvMissing`
|
||||
- only the environment-variable name is retained; secret value is never returned
|
||||
7. resolve output contract and structured-output schema payload when `json_schema` mode is active.
|
||||
8. read input artifacts.
|
||||
9. render prompt messages.
|
||||
|
||||
@@ -48,8 +48,9 @@ Typical sequence:
|
||||
1. Confirm prompt/profile directories resolve through config or flags.
|
||||
2. Confirm required input files exist and map to prompt input names.
|
||||
3. Confirm required API-key environment variables are set.
|
||||
4. Run `render` for preflight when changing prompt/profile/input wiring.
|
||||
5. Run `run` for actual generation.
|
||||
4. Confirm the selected profile's model endpoint is reachable from the process environment.
|
||||
5. Run `render` for preflight when changing prompt/profile/input wiring.
|
||||
6. Run `run` for actual generation.
|
||||
|
||||
## Secrets Handling
|
||||
|
||||
|
||||
@@ -1,355 +0,0 @@
|
||||
# Documentation Roadmap
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap defines the work required to bring scriptorium's documentation into compliance with `docs/policy/documentation.md` and the current implementation. It is a planning document only; future agents should use it to update the canonical documentation without describing unimplemented behavior outside `docs/roadmap/`.
|
||||
|
||||
## Repository Documentation Inventory
|
||||
|
||||
- `README.md` - keep and rewrite. It is currently a full manual covering config, CLI, HTTP, prompt/profile authoring, examples, and build commands; policy says README should be a short orientation page with a quickstart and links.
|
||||
- `architecture.md` - move/merge/delete after rewrite. It overlaps with `docs/policy/architecture.md`, contains "should" guidance and future extension notes outside `docs/roadmap/`, and references architecture that is partly stale or aspirational.
|
||||
- `docs/policy/documentation.md` - keep and lightly update only if policy itself changes. It is the controlling documentation policy.
|
||||
- `docs/policy/architecture.md` - keep and lightly update. It is the canonical development architecture policy, but some package-layout defaults do not exactly match this repository (`internal/adapter/...` versus policy examples such as `internal/adapters/...`).
|
||||
- `docs/policy/development.md` - create new. Required by policy for projects maintained by humans and LLM coding agents.
|
||||
- `docs/config/config-yml.md` - merge into `docs/config.md`. The content mostly matches code but lives in the wrong canonical home.
|
||||
- `docs/config/prompt-definitions.md` - merge into `docs/config.md`. The field reference is mostly accurate, but it needs caveats about `repair_attempts`, repeated message roles, schema path resolution, and examples.
|
||||
- `docs/config/profile-definitions.md` - merge into `docs/config.md`. It must stop implying that every profile field is sent to the provider; the current OpenAI-compatible client does not send `reasoning_effort` or `extra_params`.
|
||||
- `docs/config/schema-definitions.md` - merge into `docs/config.md` or link from it. Schema behavior is implemented, but the canonical config reference should own this material.
|
||||
- `docs/cli.md` - create new. Required by policy for the implemented CLI.
|
||||
- `docs/operations.md` - create new. Required by policy for this CLI/service application; scope should cover normal operation, config paths, secret handling, stdout/stderr, exit codes, HTTP serving, and the fact that there is no durable run state/resume behavior.
|
||||
- `docs/troubleshooting.md` - create new. Recommended by policy and justified by implemented failure modes in parser, config, prompt/profile loading, artifact reading, validation, LLM calls, and HTTP error mapping.
|
||||
- `docs/internal/` - create new. Required by policy for this modular application.
|
||||
- `docs/integrations/narratio.md` - keep and rewrite. It documents an actual CLI integration contract, but it includes future extension notes outside roadmap and illustrative prompt IDs that are not all present in the repository.
|
||||
- `docs/integrations/http-api.md` - create new. The implemented `POST /v1/runs` API is an external integration contract and should not live in README.
|
||||
- `docs/integrations/openai-compatible-chat.md` - create new. The outbound LLM contract is important and implemented in `internal/llm/openai_compatible_client.go`.
|
||||
- `examples/config.yml` - keep and lightly update if paths move. It is a valid app config example for the current root `prompts/`, `profiles/`, and `schemas/` directories.
|
||||
- `examples/fixtures/transcript.md` - keep. It is used by integration tests.
|
||||
- `examples/fixtures/glossary.yml` - keep. It is used by integration tests.
|
||||
- `prompts/` - keep as maintained sample prompt library for now; recommended to move or mirror under `examples/` only if tests and docs are updated together.
|
||||
- `profiles/` - keep as maintained sample profile library for now; recommended to move or mirror under `examples/` only if tests and docs are updated together.
|
||||
- `schemas/` - keep as maintained sample schema library for now; recommended to move or mirror under `examples/` only if tests and docs are updated together.
|
||||
- `local-test/` - delete, move out of the repository, or explicitly exclude from maintained docs. It contains ad hoc local artifacts and provider profiles; it should not be linked from canonical docs unless promoted to maintained examples with tests and secret review.
|
||||
|
||||
## Policy Compliance Assessment
|
||||
|
||||
Required documents that are missing:
|
||||
|
||||
- `docs/cli.md`
|
||||
- `docs/config.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/internal/`
|
||||
- `docs/policy/development.md`
|
||||
|
||||
Recommended documents that should be added:
|
||||
|
||||
- `docs/troubleshooting.md`
|
||||
- Validated examples under `examples/` beyond the current config and fixtures, especially command examples that can be checked with `render`.
|
||||
- Integration docs for the implemented HTTP API and outbound OpenAI-compatible chat API.
|
||||
|
||||
Documents that exist but are stale or in the wrong canonical home:
|
||||
|
||||
- `README.md` duplicates material that belongs in `docs/cli.md`, `docs/config.md`, `docs/integrations/`, and `docs/internal/`.
|
||||
- `docs/config/*.md` should be merged into `docs/config.md`.
|
||||
- `architecture.md` should be merged into `docs/policy/architecture.md`, `docs/internal/`, or `docs/roadmap/`, then removed.
|
||||
- `docs/integrations/narratio.md` should remain under integrations but must be narrowed to implemented CLI behavior and actual integration guidance.
|
||||
|
||||
Content that appears to describe deprecated, historical, planned, or unimplemented behavior outside `docs/roadmap/`:
|
||||
|
||||
- `architecture.md` has future extension notes for S3 artifact references, additional LLM providers, streaming, batch execution, database-backed repositories, profile versioning, and HTTP render endpoints.
|
||||
- `architecture.md` and `README.md` describe bounded repair as if it is generally active. The code has an injected repairer hook, but the CLI and HTTP server construct `Runner` without a repairer, so production commands do not currently perform repair attempts.
|
||||
- `README.md` says "additional output formats can be added later"; this belongs in roadmap only.
|
||||
- `README.md` references `go build -o scriptorium ./cmd/scriptorium`, which is valid, but the README should not be the build/test manual after `docs/policy/development.md` exists.
|
||||
- `docs/integrations/narratio.md` has "Future Extension Notes" and examples for prompt IDs not present in the repository, such as `dnd.structured_events`, `dnd.glossary_suggestions`, and `dnd.player_summary`.
|
||||
- Any docs implying S3 artifact support should be removed from current-behavior docs. `domain.ArtifactRefS3` exists, but `artifact.CompositeReader` supports only `inline` and `file`.
|
||||
|
||||
Examples that are missing, stale, invalid, or untested:
|
||||
|
||||
- `examples/config.yml` points at root `prompts/`, `profiles/`, and `schemas/`; it is valid for repository-root execution but should be tested or explicitly checked.
|
||||
- There are no copyable CLI example scripts or expected-output files under `examples/`.
|
||||
- The maintained prompt/profile/schema examples live outside `examples/`; this is usable, but policy prefers copyable examples under `examples/`.
|
||||
- `local-test/` appears unmaintained and should not be treated as documentation.
|
||||
|
||||
Links that are likely stale or need verification:
|
||||
|
||||
- Existing links from README to docs should be rewritten after canonical files are created.
|
||||
- Any references to `docs/config/config-yml.md`, `docs/config/prompt-definitions.md`, `docs/config/profile-definitions.md`, or `docs/config/schema-definitions.md` should be updated after those files are merged.
|
||||
- References to default config paths must use the implemented search order: `/usr/local/etc/scriptorium/config.yml`, then `/etc/scriptorium/config.yml`.
|
||||
|
||||
## Target Documentation Set
|
||||
|
||||
### `README.md`
|
||||
|
||||
- Audience: users, administrators, operators.
|
||||
- Purpose: concise orientation and shortest useful command.
|
||||
- Canonical scope: project purpose, elevator pitch, quickstart, and links.
|
||||
- Recommended outline: description; why scriptorium exists; shortest useful `scriptorium render` or `scriptorium run` example; documentation links.
|
||||
- Source of truth: `cmd/scriptorium/main.go`, `internal/adapter/cli/run.go`, `examples/config.yml`, `prompts/generic.markdown_summary.yaml`.
|
||||
- Acceptance criteria: no full flag reference; no HTTP schema; no prompt/profile field tables; no future work; all links point to existing target docs.
|
||||
|
||||
### `docs/cli.md`
|
||||
|
||||
- Audience: users, administrators, operators.
|
||||
- Purpose: complete CLI reference and common workflows.
|
||||
- Canonical scope: commands, flags, outputs, exit codes, and command examples.
|
||||
- Recommended outline: shortest useful command; command overview; `run`; `render`; `serve`; flag reference; input and variable mapping syntax; output behavior; exit codes; common workflows.
|
||||
- Source of truth: `internal/adapter/cli/run.go`, `internal/adapter/cli/run_test.go`, `internal/format/prepared_run.go`, `cmd/scriptorium/main.go`.
|
||||
- Acceptance criteria: documents real flags only; notes deprecated aliases `--prompt-id` and `--profile-id`; documents that `render` does not currently accept `--schema-dir`; documents stdout/stderr split and exit code `2` for validation failure.
|
||||
|
||||
### `docs/config.md`
|
||||
|
||||
- Audience: administrators, operators, advanced users.
|
||||
- Purpose: canonical reference for app config, prompt definitions, profiles, and schemas.
|
||||
- Canonical scope: all implemented YAML/JSON file formats and precedence rules.
|
||||
- Recommended outline: config file discovery and precedence; minimal app config; production-oriented app config; app config reference; prompt definition reference; profile definition reference; schema behavior; secrets handling; maintained examples.
|
||||
- Source of truth: `internal/config/config.go`, `internal/defaults/defaults.go`, `internal/promptdef/filesystem_repository.go`, `internal/profile/filesystem_repository.go`, `internal/validate/standard_validator.go`, repository `prompts/`, `profiles/`, `schemas/`, and config/profile/prompt tests.
|
||||
- Acceptance criteria: replaces split `docs/config/*.md`; documents strict YAML decoding; documents raw API key rejection; documents `schema_dir` default `.`; documents prompt `content_file` relative to prompt YAML; states `content_type` is metadata only; does not claim operational repair unless a repairer is configured.
|
||||
|
||||
### `docs/operations.md`
|
||||
|
||||
- Audience: administrators, operators.
|
||||
- Purpose: operational use of CLI and HTTP service.
|
||||
- Canonical scope: normal workflow, filesystem expectations, config deployment, secrets, logs/output, validation behavior, and recovery from failed runs.
|
||||
- Recommended outline: normal run/render workflow; config and library directories; environment variables for API keys; serving HTTP; output and stderr summaries; validation failure handling; no durable state/resume/archive behavior; safe recovery steps.
|
||||
- Source of truth: `internal/adapter/cli/run.go`, `internal/adapter/http/handler.go`, `internal/config/config.go`, `internal/llm/openai_compatible_client.go`, `internal/usecase/runner.go`.
|
||||
- Acceptance criteria: makes clear scriptorium does not persist run state; does not invent cleanup/archive/resume; documents that HTTP has no built-in authentication and should be deployed behind trusted controls.
|
||||
|
||||
### `docs/troubleshooting.md`
|
||||
|
||||
- Audience: administrators, operators.
|
||||
- Purpose: safe diagnosis and fixes for recurring implemented failure modes.
|
||||
- Canonical scope: symptoms, likely causes, diagnostics, safe fixes, and links.
|
||||
- Recommended outline: missing config; missing prompt/profile dirs; unknown flags; prompt/profile load failures; missing input files; template render failures; missing API-key environment values; LLM non-2xx/malformed responses; JSON/schema validation failures; HTTP error codes.
|
||||
- Source of truth: `internal/adapter/cli/run_test.go`, `internal/adapter/http/handler_test.go`, `internal/config/config_test.go`, `internal/promptdef/repository_test.go`, `internal/profile/repository_test.go`, `internal/validate/standard_validator_test.go`, `internal/llm/openai_compatible_client_test.go`.
|
||||
- Acceptance criteria: every entry includes symptom, likely cause, diagnostic step, safe fix, and links to canonical CLI/config/operations docs.
|
||||
|
||||
### `docs/policy/architecture.md`
|
||||
|
||||
- Audience: developers, LLM coding agents.
|
||||
- Purpose: controlling development architecture and invariants.
|
||||
- Canonical scope: development principles, boundaries, invariants, non-goals.
|
||||
- Recommended outline: keep current policy shape; add scriptorium-specific package map or link to `docs/internal/`; clarify no orchestration creep; clarify current adapters.
|
||||
- Source of truth: existing policy, `internal/` package layout, `architecture.md`.
|
||||
- Acceptance criteria: remains policy-oriented; does not become user docs; future work stays in roadmap; no stale package names.
|
||||
|
||||
### `docs/policy/development.md`
|
||||
|
||||
- Audience: developers, LLM coding agents.
|
||||
- Purpose: contributor workflow and change checklist.
|
||||
- Canonical scope: repository layout, build/test commands, coding conventions, dependency policy, adding config/CLI/adapters, updating examples/docs.
|
||||
- Recommended outline: repository layout; common commands; coding conventions; dependency policy; how to add config fields; how to add CLI flags; how to add adapters; how to update examples; documentation expectations.
|
||||
- Source of truth: `go.mod`, `cmd/scriptorium/main.go`, `internal/adapter/cli/run.go`, `internal/config/config.go`, `docs/policy/architecture.md`, existing tests.
|
||||
- Acceptance criteria: includes `go test ./...`; references `go build ./cmd/scriptorium`; tells contributors to update docs and tests with behavior changes.
|
||||
|
||||
### `docs/internal/runner.md`
|
||||
|
||||
- Audience: developers, LLM coding agents.
|
||||
- Purpose: implemented core prepare/run behavior.
|
||||
- Canonical scope: `Runner.Prepare`, `Runner.Run`, profile selection, runtime merge, artifact loading, rendering, structured output setup, validation, repair hook boundary.
|
||||
- Recommended outline: purpose; inputs/outputs; prepare flow; run flow; boundary contracts; failure behavior; tests; invariants.
|
||||
- Source of truth: `internal/usecase/runner.go`, `internal/usecase/repairer.go`, `internal/usecase/runner_test.go`, `internal/usecase/integration_test.go`.
|
||||
- Acceptance criteria: states CLI/HTTP currently construct `Runner` without a repairer; documents validation content failures versus runtime validation errors; no provider-specific details except through ports.
|
||||
|
||||
### `docs/internal/adapters.md`
|
||||
|
||||
- Audience: developers, LLM coding agents.
|
||||
- Purpose: implemented adapter boundaries.
|
||||
- Canonical scope: CLI adapter, HTTP adapter, filesystem repositories, artifact reader, prompt renderer, OpenAI-compatible LLM client, validator, prepared-run formatter.
|
||||
- Recommended outline: adapter map; inputs/outputs; config fields used; external dependencies; failure behavior; tests to inspect.
|
||||
- Source of truth: `internal/adapter/cli`, `internal/adapter/http`, `internal/promptdef`, `internal/profile`, `internal/artifact`, `internal/prompt`, `internal/llm`, `internal/validate`, `internal/format`.
|
||||
- Acceptance criteria: documents only implemented adapters; states `inline` and `file` artifact refs are supported and S3 is not; states OpenAI request fields actually sent.
|
||||
|
||||
### `docs/integrations/http-api.md`
|
||||
|
||||
- Audience: developers, LLM coding agents, API clients.
|
||||
- Purpose: implemented inbound HTTP contract.
|
||||
- Canonical scope: `POST /v1/runs`, request/response shape, raw output opt-in, error mapping, validation-failed status behavior.
|
||||
- Recommended outline: scope; endpoint; request fields; response fields; error responses; validation behavior; security/auth note.
|
||||
- Source of truth: `internal/adapter/http/dto.go`, `internal/adapter/http/handler.go`, `internal/adapter/http/handler_test.go`.
|
||||
- Acceptance criteria: no unimplemented render endpoint; no built-in auth claim; unknown JSON fields rejected; raw API key fields rejected by strict JSON.
|
||||
|
||||
### `docs/integrations/openai-compatible-chat.md`
|
||||
|
||||
- Audience: developers, LLM operators, LLM adapter maintainers.
|
||||
- Purpose: implemented outbound LLM API contract.
|
||||
- Canonical scope: OpenAI-compatible chat completions request/response subset and provider-level structured output behavior.
|
||||
- Recommended outline: endpoint construction; request fields sent; auth header from `api_key_env`; timeout behavior; response expectations; error handling; unsupported profile fields.
|
||||
- Source of truth: `internal/llm/openai_compatible_client.go`, `internal/llm/openai_compatible_client_test.go`, `internal/usecase/runner.go`.
|
||||
- Acceptance criteria: says endpoint appends `/chat/completions`; says empty first choice content is malformed; says `reasoning_effort` and `extra_params` are not currently serialized into the outbound request.
|
||||
|
||||
### `docs/integrations/narratio.md`
|
||||
|
||||
- Audience: developers, LLM coding agents maintaining Narratio integration.
|
||||
- Purpose: CLI subprocess contract for Narratio.
|
||||
- Canonical scope: how Narratio should call implemented `scriptorium run` and `scriptorium render`.
|
||||
- Recommended outline: purpose; assumptions; command shapes; inputs/vars; profile selection; runtime overrides; config behavior; environment handling; output handling; exit statuses; security notes; non-goals.
|
||||
- Source of truth: `internal/adapter/cli/run.go`, `internal/adapter/cli/run_test.go`, `docs/cli.md`, `docs/config.md`.
|
||||
- Acceptance criteria: removes future extensions; labels any Narratio-specific prompt IDs as external examples only or removes them; links to canonical CLI/config docs.
|
||||
|
||||
## File-by-File Rewrite Guidance
|
||||
|
||||
- `README.md`: cover what scriptorium does and show one minimal command. Avoid field tables, complete flag lists, HTTP schema, internal package details, future extensions, and long examples. Link to `docs/cli.md`, `docs/config.md`, `docs/operations.md`, `docs/troubleshooting.md`, and `docs/integrations/`.
|
||||
- `docs/cli.md`: cover `run`, `render`, `serve`, flags, output behavior, and exit codes. Avoid duplicating prompt/profile YAML field references; link to `docs/config.md`. Inspect CLI parser tests before writing examples. Do not carry forward README's claim that render supports `--schema-dir`.
|
||||
- `docs/config.md`: cover app config, prompt YAML, profile YAML, and schema behavior. Avoid provider API details except where needed for profile fields; link to OpenAI integration doc. Do not carry forward claims that `repair_attempts` enables repair for normal CLI/HTTP runs unless code later wires a repairer.
|
||||
- `docs/operations.md`: cover deployed operation and recovery boundaries. Avoid inventing durable state, resume behavior, cleanup, backups, or archives. State that rerunning a command is the recovery model.
|
||||
- `docs/troubleshooting.md`: use tested errors and behavior. Avoid exposing internal wrapped error details that HTTP intentionally suppresses. Link to CLI/config/operations instead of repeating full references.
|
||||
- `docs/policy/architecture.md`: preserve policy authority and update only scriptorium-specific facts. Avoid copying the long historical `architecture.md` wholesale. Move future extension ideas to roadmap docs only.
|
||||
- `docs/policy/development.md`: cover contributor mechanics and how to update behavior safely. Avoid user-facing manuals. Include tests and docs update expectations.
|
||||
- `docs/internal/runner.md`: explain implemented core flow and invariants. Avoid CLI flag tables and HTTP DTO detail; link to adapter docs.
|
||||
- `docs/internal/adapters.md`: explain implemented adapter boundaries and tests. Avoid proposing new adapters. Do not imply `ArtifactRefS3` works.
|
||||
- `docs/integrations/http-api.md`: document only `POST /v1/runs`. Avoid documenting a render/prepare HTTP endpoint.
|
||||
- `docs/integrations/openai-compatible-chat.md`: document the outbound request subset. Avoid documenting unsupported OpenAI fields or provider-specific options unless code sends them.
|
||||
- `docs/integrations/narratio.md`: keep it as a subprocess contract. Avoid future work, S3, HTTP-as-primary-path, and undeployed prompt IDs as current examples.
|
||||
- `architecture.md`: after target docs exist, delete it or replace it with a short pointer to `docs/policy/architecture.md` and `docs/internal/`. Do not leave future notes in this root file.
|
||||
- `docs/config/*.md`: after `docs/config.md` exists and links are updated, delete these split files or replace them with pointers only if backwards-compatible links are necessary.
|
||||
|
||||
## Examples Plan
|
||||
|
||||
Existing maintained examples:
|
||||
|
||||
- `examples/config.yml`: minimal app config pointing at root prompt/profile/schema libraries. Validity check: `go test ./internal/config ./internal/adapter/cli` and `go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --format json`. Link from README, `docs/config.md`, and `docs/cli.md`.
|
||||
- `examples/fixtures/transcript.md`: sample transcript input. Validity check: used by `internal/usecase/integration_test.go` and render smoke command. Link from README and `docs/cli.md`.
|
||||
- `examples/fixtures/glossary.yml`: sample optional glossary input. Validity check: used by `internal/usecase/integration_test.go`. Link from `docs/config.md` and examples section in `docs/cli.md`.
|
||||
- `prompts/generic.markdown_summary.yaml`: sample markdown prompt. Validity check: render smoke command. Link from config docs until or unless it is moved under `examples/`.
|
||||
- `prompts/generic.structured_events.yaml` plus `schemas/structured_events.schema.json`: sample JSON-schema prompt. Validity check: `go test ./internal/usecase`. Link from `docs/config.md`.
|
||||
- `profiles/local-fast.yaml` and `profiles/local-quality.yaml`: sample profiles. Validity check: profile repository tests plus integration test. Link from `docs/config.md`, with a note that `local-quality` requires `SCRIPTORIUM_API_KEY` because it sets `api_key_env`.
|
||||
|
||||
Recommended example additions, all based on implemented behavior:
|
||||
|
||||
- `examples/render-markdown-summary.sh`: copyable render smoke command using `generic.markdown_summary`. Expected check: run script or equivalent `go run` command exits `0`. Link from README and `docs/cli.md`.
|
||||
- `examples/http-run.json`: copyable `POST /v1/runs` request body using `inline` or `file` artifact refs. Expected check: parse as JSON and keep aligned with `internal/adapter/http/dto.go`. Link from `docs/integrations/http-api.md`.
|
||||
- `examples/prompts/`, `examples/profiles/`, `examples/schemas/`: optional future move or mirror of maintained sample libraries. Expected check: update integration tests and `examples/config.yml` together. This is recommended for policy alignment but should be done as its own implementation stage to avoid breaking tests.
|
||||
|
||||
Do not document `local-test/` as maintained examples.
|
||||
|
||||
## Internal Documentation Plan
|
||||
|
||||
### Core runner
|
||||
|
||||
- Path: `docs/internal/runner.md`
|
||||
- Purpose: explain implemented prepare/run lifecycle.
|
||||
- Inputs and outputs: `domain.RunRequest`, `domain.PreparedRun`, `domain.RunResult`, `domain.GenerateRequest`.
|
||||
- Boundaries: usecase owns profile selection, runtime merge, artifact resolution orchestration, rendering orchestration, structured output setup, validation, and run metadata; adapters own transport/config parsing.
|
||||
- Config fields used: none directly; adapters pass resolved repositories, validators, and request values.
|
||||
- Adapters used: promptdef repository, profile repository, artifact reader, prompt renderer, LLM client, validator, optional injected repairer.
|
||||
- Failure behavior: invalid request, prompt/profile load, artifact load, render failure, LLM failure, validation runtime failure; validation content failures return a result.
|
||||
- Tests to inspect before changing: `internal/usecase/runner_test.go`, `internal/usecase/integration_test.go`.
|
||||
- Architectural invariants: `Run` reuses `Prepare`; no resolved API key values in prepared data; repair attempts bounded and only possible when a repairer is injected; no orchestration creep.
|
||||
|
||||
### Adapters and repositories
|
||||
|
||||
- Path: `docs/internal/adapters.md`
|
||||
- Purpose: explain implemented external boundaries.
|
||||
- Inputs and outputs: CLI args/stdout/stderr/exit codes; HTTP JSON DTOs; YAML prompt/profile/config files; file/inline artifacts; OpenAI-compatible HTTP requests; prepared-run text/JSON output.
|
||||
- Boundaries: adapters translate external forms into domain requests/results and must not own domain decisions.
|
||||
- Config fields used: `prompt_dir`, `profile_dir`, `schema_dir`, `server.addr`, `defaults.render_format`; profile `endpoint`, `model`, generation fields, timeout, `api_key_env`.
|
||||
- Adapters used: CLI, HTTP, filesystem repositories, artifact reader, Go template renderer, OpenAI-compatible client, standard validator, prepared-run formatter.
|
||||
- Failure behavior: strict YAML/JSON decoding, unknown fields rejected, unsupported artifact refs rejected, LLM non-2xx/malformed responses become errors.
|
||||
- Tests to inspect before changing: adapter, repository, artifact, renderer, LLM, validator, and formatter tests under `internal/**`.
|
||||
- Architectural invariants: no raw API keys; no S3 docs until reader exists; OpenAI client sends only implemented request fields.
|
||||
|
||||
### Validation and structured output
|
||||
|
||||
- Path: include in `docs/internal/runner.md` or create `docs/internal/validation.md` if the section grows.
|
||||
- Purpose: explain `none`, `basic`, `json`, `json_schema`, schema loading, and provider-level JSON schema request setup.
|
||||
- Inputs and outputs: `domain.Artifact`, `domain.OutputContract`, `domain.ValidationResult`, `domain.StructuredOutputSpec`.
|
||||
- Boundaries: validator checks output; runner creates provider-level structured output spec for `json_schema`; OpenAI adapter serializes `response_format`.
|
||||
- Config fields used: `schema_dir`; prompt `output.schema_path`, `output.validation_mode`, `output.format`.
|
||||
- Adapters used: standard validator and OpenAI-compatible client.
|
||||
- Failure behavior: invalid generated JSON is validation failure; missing/invalid schema file is runtime validation error before or during run preparation.
|
||||
- Tests to inspect before changing: `internal/validate/standard_validator_test.go`, `internal/usecase/runner_test.go`, `internal/llm/openai_compatible_client_test.go`.
|
||||
- Architectural invariants: schema docs must load before `json_schema` LLM request; schema paths resolve relative to `schema_dir`.
|
||||
|
||||
## Integration Documentation Plan
|
||||
|
||||
### HTTP API
|
||||
|
||||
- Path: `docs/integrations/http-api.md`
|
||||
- External system or contract: inbound HTTP clients of scriptorium.
|
||||
- Current usage in scriptorium: `scriptorium serve` exposes `POST /v1/runs`.
|
||||
- Version or compatibility notes: route is `/v1/runs`; request decoding rejects unknown JSON fields.
|
||||
- What should be documented: request fields, `file` and `inline` artifact refs, model overrides, raw output opt-in, response shape, error codes, validation-failed `200 OK`, no built-in auth.
|
||||
- What should not be documented: unimplemented render endpoint, streaming, batch, authentication middleware, remote artifact storage.
|
||||
|
||||
### OpenAI-compatible chat completions
|
||||
|
||||
- Path: `docs/integrations/openai-compatible-chat.md`
|
||||
- External system or contract: outbound OpenAI-compatible `/chat/completions` API.
|
||||
- Current usage in scriptorium: `OpenAICompatibleClient.Generate` posts chat messages and optional JSON schema response format.
|
||||
- Version or compatibility notes: compatibility is defined by the subset used in code, not by a pinned OpenAI API version.
|
||||
- What should be documented: endpoint construction, request fields, auth header behavior, timeout behavior, expected response shape, error handling, structured output payload.
|
||||
- What should not be documented: provider features not serialized by code, retries, streaming, tool calls, reasoning controls, or extra provider params.
|
||||
|
||||
### Narratio CLI subprocess
|
||||
|
||||
- Path: `docs/integrations/narratio.md`
|
||||
- External system or contract: Narratio calling scriptorium as a subprocess.
|
||||
- Current usage in scriptorium: public CLI commands `run` and `render`.
|
||||
- Version or compatibility notes: contract should be tied to implemented CLI flags and exit codes.
|
||||
- What should be documented: command construction, config use, input files, vars, profile overrides, timeout override, output paths, stdout/stderr handling, exit status semantics.
|
||||
- What should not be documented: future S3 support, future HTTP primary integration, unimplemented prompt IDs as current examples, or Narratio stage state internals.
|
||||
|
||||
JSON Schema is important but does not need a separate integration doc in the first migration; keep schema behavior in `docs/config.md` and internal validation docs unless compatibility issues require a dedicated page later.
|
||||
|
||||
## Recommended Implementation Sequence
|
||||
|
||||
### Stage 1: Canonical README, CLI, and Config
|
||||
|
||||
- Goal: make user-facing docs accurate and move reference material to canonical homes.
|
||||
- Files to create/update/delete/move: rewrite `README.md`; create `docs/cli.md`; create `docs/config.md`; leave old `docs/config/*.md` temporarily with pointers or delete them only after links are updated.
|
||||
- Repository areas to inspect: `cmd/scriptorium/main.go`, `internal/adapter/cli/run.go`, `internal/adapter/cli/run_test.go`, `internal/config`, `internal/promptdef`, `internal/profile`, `internal/validate`, `examples/config.yml`, `prompts/`, `profiles/`, `schemas/`.
|
||||
- Acceptance criteria: README is short; CLI flags match parser; config paths and precedence match code; no future work outside roadmap; no repair claims beyond implemented behavior.
|
||||
- Suggested validation commands: `go test ./...`; `go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --format json`; `rg -n "future|planned|may be added|can be added later|S3|streaming|batch" README.md docs/cli.md docs/config.md`.
|
||||
- One-prompt size: yes, if old split config files are deleted or replaced with pointers in the same change.
|
||||
|
||||
### Stage 2: Operations and Troubleshooting
|
||||
|
||||
- Goal: document operational behavior and known failure modes.
|
||||
- Files to create/update/delete/move: create `docs/operations.md`; create `docs/troubleshooting.md`; update README links.
|
||||
- Repository areas to inspect: CLI and HTTP adapters, config tests, LLM client tests, validator tests, prompt/profile repository tests.
|
||||
- Acceptance criteria: no invented state/resume/backup behavior; troubleshooting entries are actionable and link to canonical docs; HTTP no-auth caveat is clear.
|
||||
- Suggested validation commands: `go test ./internal/adapter/cli ./internal/adapter/http ./internal/config ./internal/llm ./internal/validate`; `rg -n "resume|archive|backup|cleanup|state" docs/operations.md docs/troubleshooting.md`.
|
||||
- One-prompt size: yes.
|
||||
|
||||
### Stage 3: Development Policy and Internal Docs
|
||||
|
||||
- Goal: give developers and LLM agents accurate package boundaries and invariants.
|
||||
- Files to create/update/delete/move: create `docs/policy/development.md`; update `docs/policy/architecture.md`; create `docs/internal/runner.md`; create `docs/internal/adapters.md`; optionally create `docs/internal/validation.md`.
|
||||
- Repository areas to inspect: all `internal/` packages, `go.mod`, root `architecture.md`, tests.
|
||||
- Acceptance criteria: package names match repository; current adapters only; future extension ideas absent except links to roadmap; repairer hook boundary is accurate.
|
||||
- Suggested validation commands: `go test ./...`; `rg -n "should expose|may be added|future|S3|batch|streaming|database-backed|additional providers" docs/policy docs/internal`.
|
||||
- One-prompt size: maybe split into two prompts if `docs/internal/` becomes too large.
|
||||
|
||||
### Stage 4: Integration Docs
|
||||
|
||||
- Goal: move external contracts out of README and make integration docs precise.
|
||||
- Files to create/update/delete/move: create `docs/integrations/http-api.md`; create `docs/integrations/openai-compatible-chat.md`; rewrite `docs/integrations/narratio.md`; update README and CLI/config links.
|
||||
- Repository areas to inspect: `internal/adapter/http`, `internal/llm`, `internal/usecase`, CLI tests, HTTP tests, LLM tests.
|
||||
- Acceptance criteria: HTTP docs cover only `POST /v1/runs`; OpenAI docs cover only serialized fields; Narratio docs include only implemented CLI integration and no future notes.
|
||||
- Suggested validation commands: `go test ./internal/adapter/http ./internal/llm ./internal/adapter/cli`; `rg -n "POST /v1/renders|S3|future|later|batch|streaming" docs/integrations`.
|
||||
- One-prompt size: yes.
|
||||
|
||||
### Stage 5: Examples and Link Cleanup
|
||||
|
||||
- Goal: make examples policy-compliant and verify links after moves.
|
||||
- Files to create/update/delete/move: optionally add `examples/render-markdown-summary.sh`; optionally add `examples/http-run.json`; decide whether to move or mirror `prompts/`, `profiles/`, `schemas/` under `examples/`; delete or exclude `local-test/`; remove or replace root `architecture.md`; delete obsolete `docs/config/*.md` if not already removed.
|
||||
- Repository areas to inspect: `examples/`, `prompts/`, `profiles/`, `schemas/`, `internal/usecase/integration_test.go`, docs links.
|
||||
- Acceptance criteria: examples are copyable, secret-free, and tested where practical; no links to deleted docs; no maintained docs link to `local-test/`.
|
||||
- Suggested validation commands: `go test ./...`; `go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --format text`; `rg -n "docs/config/|architecture.md|local-test|dnd.structured_events|dnd.glossary_suggestions|dnd.player_summary" README.md docs examples`.
|
||||
- One-prompt size: split if moving prompt/profile/schema assets because tests and paths must be updated carefully.
|
||||
|
||||
## Validation Plan
|
||||
|
||||
- Run `go test ./...` after documentation changes that touch examples, paths, command examples, or config references.
|
||||
- Smoke-test the documented render quickstart with `go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --format json`.
|
||||
- If documenting JSON-schema render/run examples, set `SCRIPTORIUM_API_KEY` or use a profile without `api_key_env`; `Runner.Prepare` validates the named environment variable.
|
||||
- Validate CLI flags against `internal/adapter/cli/run.go` and parser tests, especially `render` lacking `--schema-dir` and `serve` rejecting runtime override flags.
|
||||
- Validate app config examples against `internal/config/config.go` strict YAML decoding.
|
||||
- Validate prompt/profile examples against `internal/promptdef/filesystem_repository.go` and `internal/profile/filesystem_repository.go`.
|
||||
- Validate HTTP request examples against `internal/adapter/http/dto.go` strict JSON decoding.
|
||||
- Run grep checks for stale or roadmap-only terms outside `docs/roadmap/`: `future`, `planned`, `may be added`, `can be added later`, `S3`, `streaming`, `batch`, `database-backed`, `render endpoint`, and prompt IDs not present in `prompts/`.
|
||||
- Run grep checks for stale paths after file moves: `docs/config/`, `architecture.md`, and `local-test`.
|
||||
- No automated documentation link checker is currently configured; perform manual link review or add a link checker in a separate roadmap item if desired.
|
||||
|
||||
## Open Questions
|
||||
|
||||
No open questions block the documentation migration. The recommended path is to document the current implementation conservatively, move future ideas into `docs/roadmap/`, and avoid claiming production behavior for hooks that are present in code but not wired into CLI or HTTP adapters.
|
||||
@@ -1,6 +1,6 @@
|
||||
prompt_dir: ./prompts
|
||||
profile_dir: ./profiles
|
||||
schema_dir: ./schemas
|
||||
prompt_dir: ./examples/prompts
|
||||
profile_dir: ./examples/profiles
|
||||
schema_dir: ./examples/schemas
|
||||
|
||||
server:
|
||||
addr: :8080
|
||||
|
||||
@@ -81,6 +81,14 @@ type serveConfig struct {
|
||||
schemaDir string
|
||||
}
|
||||
|
||||
type commonCommandSettings struct {
|
||||
promptDir string
|
||||
profileDir string
|
||||
schemaDir string
|
||||
serverAddr string
|
||||
defaultRenderFormat renderformat.PreparedRunOutputFormat
|
||||
}
|
||||
|
||||
type listFlag []string
|
||||
|
||||
func (l *listFlag) String() string {
|
||||
@@ -125,22 +133,13 @@ func runCommand(args []string, stdout, stderr io.Writer) int {
|
||||
return ExitRuntimeError
|
||||
}
|
||||
|
||||
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
|
||||
Timeout: defaults.LLMRequestTimeoutDefault,
|
||||
})
|
||||
llmClient, err := newOpenAIClient()
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "llm client error: %v\n", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
|
||||
runner := usecase.NewRunner(
|
||||
promptdef.NewFilesystemRepository(cfg.promptDir),
|
||||
profile.NewFilesystemRepository(cfg.profileDir),
|
||||
artifactadapter.NewCompositeReader(),
|
||||
prompt.NewGoRenderer(),
|
||||
llmClient,
|
||||
validate.NewStandardValidator(cfg.schemaDir),
|
||||
)
|
||||
runner := newRunner(cfg.promptDir, cfg.profileDir, cfg.schemaDir, llmClient)
|
||||
|
||||
res, runErr := runner.Run(context.Background(), req)
|
||||
if runErr != nil {
|
||||
@@ -170,14 +169,7 @@ func renderCommand(args []string, stdout, stderr io.Writer) int {
|
||||
return ExitRuntimeError
|
||||
}
|
||||
|
||||
runner := usecase.NewRunner(
|
||||
promptdef.NewFilesystemRepository(cfg.promptDir),
|
||||
profile.NewFilesystemRepository(cfg.profileDir),
|
||||
artifactadapter.NewCompositeReader(),
|
||||
prompt.NewGoRenderer(),
|
||||
nil,
|
||||
validate.NewStandardValidator(cfg.schemaDir),
|
||||
)
|
||||
runner := newRunner(cfg.promptDir, cfg.profileDir, cfg.schemaDir, nil)
|
||||
|
||||
prepared, prepErr := runner.Prepare(context.Background(), req)
|
||||
if prepErr != nil {
|
||||
@@ -205,22 +197,13 @@ func serveCommand(args []string, stderr io.Writer) int {
|
||||
return ExitRuntimeError
|
||||
}
|
||||
|
||||
llmClient, err := llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
|
||||
Timeout: defaults.LLMRequestTimeoutDefault,
|
||||
})
|
||||
llmClient, err := newOpenAIClient()
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "llm client error: %v\n", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
|
||||
runner := usecase.NewRunner(
|
||||
promptdef.NewFilesystemRepository(cfg.promptDir),
|
||||
profile.NewFilesystemRepository(cfg.profileDir),
|
||||
artifactadapter.NewCompositeReader(),
|
||||
prompt.NewGoRenderer(),
|
||||
llmClient,
|
||||
validate.NewStandardValidator(cfg.schemaDir),
|
||||
)
|
||||
runner := newRunner(cfg.promptDir, cfg.profileDir, cfg.schemaDir, llmClient)
|
||||
|
||||
h := httpadapter.NewHandler(runner)
|
||||
srv := &http.Server{
|
||||
@@ -307,7 +290,7 @@ func parseServeArgs(args []string) (*serveConfig, error) {
|
||||
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
|
||||
}
|
||||
|
||||
settings, err := resolveAppSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
||||
settings, err := resolveCommonSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
||||
PromptDir: cfg.promptDirIfSet(fs),
|
||||
ProfileDir: cfg.profileDirIfSet(fs),
|
||||
SchemaDir: cfg.schemaDirIfSet(fs),
|
||||
@@ -317,16 +300,13 @@ func parseServeArgs(args []string) (*serveConfig, error) {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
cfg.promptDir = settings.PromptDir
|
||||
cfg.profileDir = settings.ProfileDir
|
||||
cfg.schemaDir = settings.SchemaDir
|
||||
cfg.addr = settings.ServerAddr
|
||||
cfg.promptDir = settings.promptDir
|
||||
cfg.profileDir = settings.profileDir
|
||||
cfg.schemaDir = settings.schemaDir
|
||||
cfg.addr = settings.serverAddr
|
||||
|
||||
if strings.TrimSpace(cfg.promptDir) == "" {
|
||||
return nil, errors.New(errPromptDirRequired)
|
||||
}
|
||||
if strings.TrimSpace(cfg.profileDir) == "" {
|
||||
return nil, errors.New(errProfileDirRequired)
|
||||
if err := validateRequiredLibraryDirs(cfg.promptDir, cfg.profileDir); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
cfg.promptDir = filepath.Clean(cfg.promptDir)
|
||||
@@ -359,7 +339,7 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
|
||||
return fmt.Errorf("unexpected positional args: %v", fs.Args())
|
||||
}
|
||||
|
||||
settings, err := resolveAppSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
||||
settings, err := resolveCommonSettings(fs, cfg.configPath, appconfig.CLIOverrides{
|
||||
PromptDir: cfg.promptDirIfSet(fs),
|
||||
ProfileDir: cfg.profileDirIfSet(fs),
|
||||
SchemaDir: cfg.schemaDirIfSet(fs),
|
||||
@@ -368,16 +348,13 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
|
||||
return err
|
||||
}
|
||||
|
||||
cfg.promptDir = settings.PromptDir
|
||||
cfg.profileDir = settings.ProfileDir
|
||||
cfg.schemaDir = settings.SchemaDir
|
||||
cfg.defaultRenderFormat = settings.DefaultRenderFormat
|
||||
cfg.promptDir = settings.promptDir
|
||||
cfg.profileDir = settings.profileDir
|
||||
cfg.schemaDir = settings.schemaDir
|
||||
cfg.defaultRenderFormat = settings.defaultRenderFormat
|
||||
|
||||
if strings.TrimSpace(cfg.promptDir) == "" {
|
||||
return errors.New(errPromptDirRequired)
|
||||
}
|
||||
if strings.TrimSpace(cfg.profileDir) == "" {
|
||||
return errors.New(errProfileDirRequired)
|
||||
if err := validateRequiredLibraryDirs(cfg.promptDir, cfg.profileDir); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(cfg.promptID) == "" {
|
||||
return errors.New("--prompt is required")
|
||||
@@ -476,6 +453,47 @@ func resolveAppSettings(fs *flag.FlagSet, configPath string, overrides appconfig
|
||||
return merged, nil
|
||||
}
|
||||
|
||||
func resolveCommonSettings(fs *flag.FlagSet, configPath string, overrides appconfig.CLIOverrides) (commonCommandSettings, error) {
|
||||
settings, err := resolveAppSettings(fs, configPath, overrides)
|
||||
if err != nil {
|
||||
return commonCommandSettings{}, err
|
||||
}
|
||||
return commonCommandSettings{
|
||||
promptDir: settings.PromptDir,
|
||||
profileDir: settings.ProfileDir,
|
||||
schemaDir: settings.SchemaDir,
|
||||
serverAddr: settings.ServerAddr,
|
||||
defaultRenderFormat: settings.DefaultRenderFormat,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func validateRequiredLibraryDirs(promptDir, profileDir string) error {
|
||||
if strings.TrimSpace(promptDir) == "" {
|
||||
return errors.New(errPromptDirRequired)
|
||||
}
|
||||
if strings.TrimSpace(profileDir) == "" {
|
||||
return errors.New(errProfileDirRequired)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func newRunner(promptDir, profileDir, schemaDir string, llmClient llm.Client) *usecase.Runner {
|
||||
return usecase.NewRunner(
|
||||
promptdef.NewFilesystemRepository(promptDir),
|
||||
profile.NewFilesystemRepository(profileDir),
|
||||
artifactadapter.NewCompositeReader(),
|
||||
prompt.NewGoRenderer(),
|
||||
llmClient,
|
||||
validate.NewStandardValidator(schemaDir),
|
||||
)
|
||||
}
|
||||
|
||||
func newOpenAIClient() (*llm.OpenAICompatibleClient, error) {
|
||||
return llm.NewOpenAICompatibleClient(llm.OpenAICompatibleConfig{
|
||||
Timeout: defaults.LLMRequestTimeoutDefault,
|
||||
})
|
||||
}
|
||||
|
||||
func buildRunRequestFromConfig(cfg *runConfig) (domain.RunRequest, error) {
|
||||
inputMappings, err := parseMappings(cfg.inputRaw, false)
|
||||
if err != nil {
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
@@ -410,6 +411,28 @@ defaults:
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseRenderArgsExplicitFormatOverridesConfigDefaultFormat(t *testing.T) {
|
||||
configPath := writeAppConfigFile(t, `
|
||||
prompt_dir: ./from-config/prompts
|
||||
profile_dir: ./from-config/profiles
|
||||
defaults:
|
||||
render_format: json
|
||||
`)
|
||||
|
||||
cfg, err := parseRenderArgs([]string{
|
||||
"--config", configPath,
|
||||
"--prompt", "p",
|
||||
"--input", "a=b",
|
||||
"--format", "text",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected valid args, got %v", err)
|
||||
}
|
||||
if cfg.outputFormat != renderformat.PreparedRunFormatText {
|
||||
t.Fatalf("expected explicit --format text to override config default, got %q", cfg.outputFormat)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseServeArgsWithExplicitConfigLoadsSettingsAndCLIAddrOverrides(t *testing.T) {
|
||||
configPath := writeAppConfigFile(t, `
|
||||
prompt_dir: ./from-config/prompts
|
||||
@@ -471,6 +494,59 @@ server:
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunAndRenderBuildEquivalentRuntimeOverrideRequestsForSharedFlags(t *testing.T) {
|
||||
runCfg, err := parseRunArgs([]string{
|
||||
"--prompt-dir", "./prompts",
|
||||
"--profile-dir", "./profiles",
|
||||
"--prompt", "prompt-1",
|
||||
"--profile", "profile-1",
|
||||
"--input", "transcript=./transcript.md",
|
||||
"--var", "session_date=2026-05-01",
|
||||
"--llm-base-url", "http://localhost:8000/v1",
|
||||
"--model", "model-x",
|
||||
"--temperature", "0.8",
|
||||
"--max-tokens", "123",
|
||||
"--top-p", "0.6",
|
||||
"--timeout", "90s",
|
||||
"--api-key-env", "SCRIPTORIUM_API_KEY",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected valid run args, got %v", err)
|
||||
}
|
||||
|
||||
renderCfg, err := parseRenderArgs([]string{
|
||||
"--prompt-dir", "./prompts",
|
||||
"--profile-dir", "./profiles",
|
||||
"--prompt", "prompt-1",
|
||||
"--profile", "profile-1",
|
||||
"--input", "transcript=./transcript.md",
|
||||
"--var", "session_date=2026-05-01",
|
||||
"--llm-base-url", "http://localhost:8000/v1",
|
||||
"--model", "model-x",
|
||||
"--temperature", "0.8",
|
||||
"--max-tokens", "123",
|
||||
"--top-p", "0.6",
|
||||
"--timeout", "90s",
|
||||
"--api-key-env", "SCRIPTORIUM_API_KEY",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected valid render args, got %v", err)
|
||||
}
|
||||
|
||||
runReq, err := buildRunRequestFromConfig(runCfg)
|
||||
if err != nil {
|
||||
t.Fatalf("expected run request build success, got %v", err)
|
||||
}
|
||||
renderReq, err := buildRunRequestFromConfig(&renderCfg.runConfig)
|
||||
if err != nil {
|
||||
t.Fatalf("expected render request build success, got %v", err)
|
||||
}
|
||||
|
||||
if !reflect.DeepEqual(runReq, renderReq) {
|
||||
t.Fatalf("expected run/render shared flag requests to match.\nrun=%#v\nrender=%#v", runReq, renderReq)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseRunArgsFailsClearlyWhenNoEffectivePromptDir(t *testing.T) {
|
||||
configPath := writeAppConfigFile(t, `
|
||||
profile_dir: ./profiles
|
||||
@@ -586,42 +662,29 @@ func TestRunCommandVarsOptional(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRunCommandSucceedsWithPromptAndProfileDirsFromConfig(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
promptDir := filepath.Join(tmp, "prompts")
|
||||
profileDir := filepath.Join(tmp, "profiles")
|
||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inputPath := filepath.Join(tmp, "transcript.md")
|
||||
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lib := newCLITestLibrary(t)
|
||||
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
|
||||
|
||||
ts := newTestLLMServer("from-config-dirs", nil)
|
||||
defer ts.Close()
|
||||
|
||||
writePromptFile(t, promptDir, "prompt.default", "local-default")
|
||||
writeProfileFile(t, profileDir, "local-default", ts.URL+"/v1", "profile-model")
|
||||
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
|
||||
writeProfileFile(t, lib.profileDir, "local-default", ts.URL+"/v1", "profile-model")
|
||||
configPath := writeAppConfigFile(t, fmt.Sprintf(`
|
||||
prompt_dir: %s
|
||||
profile_dir: %s
|
||||
`, promptDir, profileDir))
|
||||
`, lib.promptDir, lib.profileDir))
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := runCommand([]string{
|
||||
code, stdout, stderr := runCLICommand(t, runCommand, []string{
|
||||
"--config", configPath,
|
||||
"--prompt", "prompt.default",
|
||||
"--input", "transcript=" + inputPath,
|
||||
}, &stdout, &stderr)
|
||||
})
|
||||
if code != ExitOK {
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||
}
|
||||
if stdout.String() != "from-config-dirs" {
|
||||
t.Fatalf("unexpected stdout output: %q", stdout.String())
|
||||
if stdout != "from-config-dirs" {
|
||||
t.Fatalf("unexpected stdout output: %q", stdout)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -630,28 +693,15 @@ func TestRenderCommandDefaultFormatTextIncludesPreparedDetailsAndNoSecrets(t *te
|
||||
const secret = "super-secret-render-key"
|
||||
t.Setenv(envName, secret)
|
||||
|
||||
tmp := t.TempDir()
|
||||
promptDir := filepath.Join(tmp, "prompts")
|
||||
profileDir := filepath.Join(tmp, "profiles")
|
||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inputPath := filepath.Join(tmp, "transcript.md")
|
||||
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lib := newCLITestLibrary(t)
|
||||
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
|
||||
|
||||
writePromptFileWithTemplate(t, promptDir, "prompt.render", "local-default", "Date {{.session_date}} - Summarize: {{input \"transcript\"}}")
|
||||
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
||||
writePromptFileWithTemplate(t, lib.promptDir, "prompt.render", "local-default", "Date {{.session_date}} - Summarize: {{input \"transcript\"}}")
|
||||
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := renderCommand([]string{
|
||||
"--prompt-dir", promptDir,
|
||||
"--profile-dir", profileDir,
|
||||
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||
"--prompt-dir", lib.promptDir,
|
||||
"--profile-dir", lib.profileDir,
|
||||
"--prompt", "prompt.render",
|
||||
"--profile", "local-default",
|
||||
"--input", "transcript=" + inputPath,
|
||||
@@ -663,15 +713,15 @@ func TestRenderCommandDefaultFormatTextIncludesPreparedDetailsAndNoSecrets(t *te
|
||||
"--top-p", "0.2",
|
||||
"--timeout", "20s",
|
||||
"--api-key-env", envName,
|
||||
}, &stdout, &stderr)
|
||||
})
|
||||
if code != ExitOK {
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||
}
|
||||
if stderr.Len() != 0 {
|
||||
t.Fatalf("expected empty stderr on success, got %q", stderr.String())
|
||||
if stderr != "" {
|
||||
t.Fatalf("expected empty stderr on success, got %q", stderr)
|
||||
}
|
||||
|
||||
out := stdout.String()
|
||||
out := stdout
|
||||
for _, want := range []string{
|
||||
"prompt: prompt.render",
|
||||
"selected_profile_id: local-default",
|
||||
@@ -698,111 +748,72 @@ func TestRenderCommandDefaultFormatTextIncludesPreparedDetailsAndNoSecrets(t *te
|
||||
}
|
||||
|
||||
func TestRenderCommandSucceedsWithPromptAndProfileDirsFromConfig(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
promptDir := filepath.Join(tmp, "prompts")
|
||||
profileDir := filepath.Join(tmp, "profiles")
|
||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inputPath := filepath.Join(tmp, "transcript.md")
|
||||
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lib := newCLITestLibrary(t)
|
||||
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
|
||||
|
||||
writePromptFile(t, promptDir, "prompt.render", "local-default")
|
||||
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
||||
writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
|
||||
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
||||
configPath := writeAppConfigFile(t, fmt.Sprintf(`
|
||||
prompt_dir: %s
|
||||
profile_dir: %s
|
||||
`, promptDir, profileDir))
|
||||
`, lib.promptDir, lib.profileDir))
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := renderCommand([]string{
|
||||
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||
"--config", configPath,
|
||||
"--prompt", "prompt.render",
|
||||
"--input", "transcript=" + inputPath,
|
||||
}, &stdout, &stderr)
|
||||
})
|
||||
if code != ExitOK {
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "prompt: prompt.render") {
|
||||
t.Fatalf("expected rendered output, got %q", stdout.String())
|
||||
if !strings.Contains(stdout, "prompt: prompt.render") {
|
||||
t.Fatalf("expected rendered output, got %q", stdout)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderCommandExplicitTextFormatWorks(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
promptDir := filepath.Join(tmp, "prompts")
|
||||
profileDir := filepath.Join(tmp, "profiles")
|
||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inputPath := filepath.Join(tmp, "transcript.md")
|
||||
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lib := newCLITestLibrary(t)
|
||||
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
|
||||
|
||||
writePromptFile(t, promptDir, "prompt.render", "local-default")
|
||||
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
||||
writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
|
||||
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := renderCommand([]string{
|
||||
"--prompt-dir", promptDir,
|
||||
"--profile-dir", profileDir,
|
||||
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||
"--prompt-dir", lib.promptDir,
|
||||
"--profile-dir", lib.profileDir,
|
||||
"--prompt", "prompt.render",
|
||||
"--input", "transcript=" + inputPath,
|
||||
"--format", "text",
|
||||
}, &stdout, &stderr)
|
||||
})
|
||||
if code != ExitOK {
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "prompt: prompt.render") {
|
||||
t.Fatalf("expected text output for explicit --format text, got %q", stdout.String())
|
||||
if !strings.Contains(stdout, "prompt: prompt.render") {
|
||||
t.Fatalf("expected text output for explicit --format text, got %q", stdout)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderCommandExplicitJSONFormatOutputsValidJSON(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
promptDir := filepath.Join(tmp, "prompts")
|
||||
profileDir := filepath.Join(tmp, "profiles")
|
||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inputPath := filepath.Join(tmp, "transcript.md")
|
||||
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lib := newCLITestLibrary(t)
|
||||
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
|
||||
|
||||
writePromptFile(t, promptDir, "prompt.render", "local-default")
|
||||
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
||||
writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
|
||||
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := renderCommand([]string{
|
||||
"--prompt-dir", promptDir,
|
||||
"--profile-dir", profileDir,
|
||||
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||
"--prompt-dir", lib.promptDir,
|
||||
"--profile-dir", lib.profileDir,
|
||||
"--prompt", "prompt.render",
|
||||
"--input", "transcript=" + inputPath,
|
||||
"--format", "json",
|
||||
}, &stdout, &stderr)
|
||||
})
|
||||
if code != ExitOK {
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||
}
|
||||
|
||||
var payload map[string]any
|
||||
if err := json.Unmarshal(stdout.Bytes(), &payload); err != nil {
|
||||
t.Fatalf("expected valid json output, got %v\nbody=%s", err, stdout.String())
|
||||
if err := json.Unmarshal([]byte(stdout), &payload); err != nil {
|
||||
t.Fatalf("expected valid json output, got %v\nbody=%s", err, stdout)
|
||||
}
|
||||
if payload["prompt_id"] != "prompt.render" {
|
||||
t.Fatalf("expected prompt_id, got %#v", payload["prompt_id"])
|
||||
@@ -834,38 +845,25 @@ func TestRenderCommandUnknownFormatFailsClearly(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRenderCommandOutWritesToFile(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
promptDir := filepath.Join(tmp, "prompts")
|
||||
profileDir := filepath.Join(tmp, "profiles")
|
||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inputPath := filepath.Join(tmp, "transcript.md")
|
||||
if err := os.WriteFile(inputPath, []byte("hello transcript"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
outPath := filepath.Join(tmp, "render.txt")
|
||||
lib := newCLITestLibrary(t)
|
||||
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
|
||||
outPath := filepath.Join(lib.rootDir, "render.txt")
|
||||
|
||||
writePromptFile(t, promptDir, "prompt.render", "local-default")
|
||||
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
||||
writePromptFile(t, lib.promptDir, "prompt.render", "local-default")
|
||||
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "profile-model")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := renderCommand([]string{
|
||||
"--prompt-dir", promptDir,
|
||||
"--profile-dir", profileDir,
|
||||
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||
"--prompt-dir", lib.promptDir,
|
||||
"--profile-dir", lib.profileDir,
|
||||
"--prompt", "prompt.render",
|
||||
"--input", "transcript=" + inputPath,
|
||||
"--out", outPath,
|
||||
}, &stdout, &stderr)
|
||||
})
|
||||
if code != ExitOK {
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||
}
|
||||
if stdout.Len() != 0 {
|
||||
t.Fatalf("expected empty stdout when --out is set, got %q", stdout.String())
|
||||
if stdout != "" {
|
||||
t.Fatalf("expected empty stdout when --out is set, got %q", stdout)
|
||||
}
|
||||
out, err := os.ReadFile(outPath)
|
||||
if err != nil {
|
||||
@@ -877,36 +875,23 @@ func TestRenderCommandOutWritesToFile(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRenderCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
promptDir := filepath.Join(tmp, "prompts")
|
||||
profileDir := filepath.Join(tmp, "profiles")
|
||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inputPath := filepath.Join(tmp, "transcript.md")
|
||||
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lib := newCLITestLibrary(t)
|
||||
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
|
||||
|
||||
writePromptFile(t, promptDir, "prompt.default", "local-default")
|
||||
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model")
|
||||
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
|
||||
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := renderCommand([]string{
|
||||
"--prompt-dir", promptDir,
|
||||
"--profile-dir", profileDir,
|
||||
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||
"--prompt-dir", lib.promptDir,
|
||||
"--profile-dir", lib.profileDir,
|
||||
"--prompt", "prompt.default",
|
||||
"--input", "transcript=" + inputPath,
|
||||
}, &stdout, &stderr)
|
||||
})
|
||||
|
||||
if code != ExitOK {
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||
}
|
||||
out := stdout.String()
|
||||
out := stdout
|
||||
if !strings.Contains(out, "selected_profile_id: local-default") {
|
||||
t.Fatalf("expected prompt default profile in output, got %q", out)
|
||||
}
|
||||
@@ -916,38 +901,25 @@ func TestRenderCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRenderCommandExplicitProfileOverridesPromptDefault(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
promptDir := filepath.Join(tmp, "prompts")
|
||||
profileDir := filepath.Join(tmp, "profiles")
|
||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inputPath := filepath.Join(tmp, "transcript.md")
|
||||
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lib := newCLITestLibrary(t)
|
||||
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
|
||||
|
||||
writePromptFile(t, promptDir, "prompt.default", "local-default")
|
||||
writeProfileFile(t, profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model")
|
||||
writeProfileFile(t, profileDir, "quality", "http://127.0.0.1:1/v1", "quality-model")
|
||||
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
|
||||
writeProfileFile(t, lib.profileDir, "local-default", "http://127.0.0.1:1/v1", "default-model")
|
||||
writeProfileFile(t, lib.profileDir, "quality", "http://127.0.0.1:1/v1", "quality-model")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := renderCommand([]string{
|
||||
"--prompt-dir", promptDir,
|
||||
"--profile-dir", profileDir,
|
||||
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||
"--prompt-dir", lib.promptDir,
|
||||
"--profile-dir", lib.profileDir,
|
||||
"--prompt", "prompt.default",
|
||||
"--profile", "quality",
|
||||
"--input", "transcript=" + inputPath,
|
||||
}, &stdout, &stderr)
|
||||
})
|
||||
|
||||
if code != ExitOK {
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||
}
|
||||
out := stdout.String()
|
||||
out := stdout
|
||||
if !strings.Contains(out, "selected_profile_id: quality") {
|
||||
t.Fatalf("expected explicit profile in output, got %q", out)
|
||||
}
|
||||
@@ -957,104 +929,67 @@ func TestRenderCommandExplicitProfileOverridesPromptDefault(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRunCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
promptDir := filepath.Join(tmp, "prompts")
|
||||
profileDir := filepath.Join(tmp, "profiles")
|
||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inputPath := filepath.Join(tmp, "transcript.md")
|
||||
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lib := newCLITestLibrary(t)
|
||||
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
|
||||
|
||||
ts := newTestLLMServer("default-output", nil)
|
||||
defer ts.Close()
|
||||
|
||||
writePromptFile(t, promptDir, "prompt.default", "local-default")
|
||||
writeProfileFile(t, profileDir, "local-default", ts.URL+"/v1", "profile-model")
|
||||
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
|
||||
writeProfileFile(t, lib.profileDir, "local-default", ts.URL+"/v1", "profile-model")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := runCommand([]string{
|
||||
"--prompt-dir", promptDir,
|
||||
"--profile-dir", profileDir,
|
||||
code, stdout, stderr := runCLICommand(t, runCommand, []string{
|
||||
"--prompt-dir", lib.promptDir,
|
||||
"--profile-dir", lib.profileDir,
|
||||
"--prompt", "prompt.default",
|
||||
"--input", "transcript=" + inputPath,
|
||||
}, &stdout, &stderr)
|
||||
})
|
||||
|
||||
if code != ExitOK {
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||
}
|
||||
if stdout.String() != "default-output" {
|
||||
t.Fatalf("unexpected stdout output: %q", stdout.String())
|
||||
if stdout != "default-output" {
|
||||
t.Fatalf("unexpected stdout output: %q", stdout)
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "selected_profile=local-default") {
|
||||
t.Fatalf("expected selected profile in summary, got %q", stderr.String())
|
||||
if !strings.Contains(stderr, "selected_profile=local-default") {
|
||||
t.Fatalf("expected selected profile in summary, got %q", stderr)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunCommandExplicitProfileOverridesPromptDefault(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
promptDir := filepath.Join(tmp, "prompts")
|
||||
profileDir := filepath.Join(tmp, "profiles")
|
||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inputPath := filepath.Join(tmp, "transcript.md")
|
||||
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lib := newCLITestLibrary(t)
|
||||
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
|
||||
|
||||
defaultServer := newTestLLMServer("from-default", nil)
|
||||
defer defaultServer.Close()
|
||||
overrideServer := newTestLLMServer("from-override", nil)
|
||||
defer overrideServer.Close()
|
||||
|
||||
writePromptFile(t, promptDir, "prompt.default", "local-default")
|
||||
writeProfileFile(t, profileDir, "local-default", defaultServer.URL+"/v1", "default-model")
|
||||
writeProfileFile(t, profileDir, "quality", overrideServer.URL+"/v1", "quality-model")
|
||||
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
|
||||
writeProfileFile(t, lib.profileDir, "local-default", defaultServer.URL+"/v1", "default-model")
|
||||
writeProfileFile(t, lib.profileDir, "quality", overrideServer.URL+"/v1", "quality-model")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := runCommand([]string{
|
||||
"--prompt-dir", promptDir,
|
||||
"--profile-dir", profileDir,
|
||||
code, stdout, stderr := runCLICommand(t, runCommand, []string{
|
||||
"--prompt-dir", lib.promptDir,
|
||||
"--profile-dir", lib.profileDir,
|
||||
"--prompt", "prompt.default",
|
||||
"--profile", "quality",
|
||||
"--input", "transcript=" + inputPath,
|
||||
}, &stdout, &stderr)
|
||||
})
|
||||
if code != ExitOK {
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||
}
|
||||
if stdout.String() != "from-override" {
|
||||
t.Fatalf("expected explicit profile output, got %q", stdout.String())
|
||||
if stdout != "from-override" {
|
||||
t.Fatalf("expected explicit profile output, got %q", stdout)
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "selected_profile=quality") {
|
||||
t.Fatalf("expected selected profile quality, got %q", stderr.String())
|
||||
if !strings.Contains(stderr, "selected_profile=quality") {
|
||||
t.Fatalf("expected selected profile quality, got %q", stderr)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
promptDir := filepath.Join(tmp, "prompts")
|
||||
profileDir := filepath.Join(tmp, "profiles")
|
||||
if err := os.MkdirAll(promptDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(profileDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inputPath := filepath.Join(tmp, "transcript.md")
|
||||
if err := os.WriteFile(inputPath, []byte("hello"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lib := newCLITestLibrary(t)
|
||||
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
|
||||
|
||||
var baseHits int32
|
||||
baseServer := newTestLLMServer("base", &baseHits)
|
||||
@@ -1071,14 +1006,12 @@ func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
|
||||
}))
|
||||
defer overrideServer.Close()
|
||||
|
||||
writePromptFile(t, promptDir, "prompt.default", "local-default")
|
||||
writeProfileFile(t, profileDir, "local-default", baseServer.URL+"/v1", "profile-model")
|
||||
writePromptFile(t, lib.promptDir, "prompt.default", "local-default")
|
||||
writeProfileFile(t, lib.profileDir, "local-default", baseServer.URL+"/v1", "profile-model")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := runCommand([]string{
|
||||
"--prompt-dir", promptDir,
|
||||
"--profile-dir", profileDir,
|
||||
code, stdout, stderr := runCLICommand(t, runCommand, []string{
|
||||
"--prompt-dir", lib.promptDir,
|
||||
"--profile-dir", lib.profileDir,
|
||||
"--prompt", "prompt.default",
|
||||
"--input", "transcript=" + inputPath,
|
||||
"--llm-base-url", overrideServer.URL + "/v1",
|
||||
@@ -1087,9 +1020,9 @@ func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
|
||||
"--max-tokens", "55",
|
||||
"--top-p", "0.2",
|
||||
"--timeout", "20s",
|
||||
}, &stdout, &stderr)
|
||||
})
|
||||
if code != ExitOK {
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr.String())
|
||||
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||
}
|
||||
if atomic.LoadInt32(&baseHits) != 0 {
|
||||
t.Fatalf("expected base profile endpoint not to be hit, got %d", baseHits)
|
||||
@@ -1097,8 +1030,8 @@ func TestRunCommandRuntimeFlagsOverrideSelectedProfileValues(t *testing.T) {
|
||||
if atomic.LoadInt32(&overrideHits) != 1 {
|
||||
t.Fatalf("expected override endpoint to be hit once, got %d", overrideHits)
|
||||
}
|
||||
if stdout.String() != "override" {
|
||||
t.Fatalf("unexpected stdout output: %q", stdout.String())
|
||||
if stdout != "override" {
|
||||
t.Fatalf("unexpected stdout output: %q", stdout)
|
||||
}
|
||||
if !strings.Contains(observedBody, `"model":"override-model"`) {
|
||||
t.Fatalf("expected override model in request body, got %s", observedBody)
|
||||
@@ -1133,6 +1066,49 @@ func TestWriteOutputAndSummaryUseSeparateWriters(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
type cliTestLibrary struct {
|
||||
rootDir string
|
||||
promptDir string
|
||||
profileDir string
|
||||
}
|
||||
|
||||
func newCLITestLibrary(t *testing.T) *cliTestLibrary {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
lib := &cliTestLibrary{
|
||||
rootDir: root,
|
||||
promptDir: filepath.Join(root, "prompts"),
|
||||
profileDir: filepath.Join(root, "profiles"),
|
||||
}
|
||||
if err := os.MkdirAll(lib.promptDir, 0o755); err != nil {
|
||||
t.Fatalf("failed to create prompt fixture directory: %v", err)
|
||||
}
|
||||
if err := os.MkdirAll(lib.profileDir, 0o755); err != nil {
|
||||
t.Fatalf("failed to create profile fixture directory: %v", err)
|
||||
}
|
||||
return lib
|
||||
}
|
||||
|
||||
func (l *cliTestLibrary) writeInputFile(t *testing.T, name, body string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(l.rootDir, name)
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
t.Fatalf("failed to create input fixture directory: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
|
||||
t.Fatalf("failed to write input fixture: %v", err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func runCLICommand(t *testing.T, command func([]string, io.Writer, io.Writer) int, args []string) (int, string, string) {
|
||||
t.Helper()
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := command(args, &stdout, &stderr)
|
||||
return code, stdout.String(), stderr.String()
|
||||
}
|
||||
|
||||
func writePromptFile(t *testing.T, dir, id, defaultProfile string) {
|
||||
t.Helper()
|
||||
writePromptFileWithTemplate(t, dir, id, defaultProfile, "Summarize: {{input \"transcript\"}}")
|
||||
|
||||
@@ -27,6 +27,7 @@ type modelOverrideRequestDTO struct {
|
||||
MaxTokens int `json:"max_tokens,omitempty"`
|
||||
TopP float64 `json:"top_p,omitempty"`
|
||||
TimeoutSeconds int `json:"timeout_seconds,omitempty"`
|
||||
ServiceTier string `json:"service_tier,omitempty"`
|
||||
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||
APIKeyEnv string `json:"api_key_env,omitempty"`
|
||||
ExtraParams map[string]string `json:"extra_params,omitempty"`
|
||||
@@ -75,6 +76,7 @@ type modelParamsDTO struct {
|
||||
MaxTokens int `json:"max_tokens"`
|
||||
TopP float64 `json:"top_p"`
|
||||
TimeoutSeconds int `json:"timeout_seconds"`
|
||||
ServiceTier string `json:"service_tier,omitempty"`
|
||||
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||
APIKeyEnv string `json:"api_key_env,omitempty"`
|
||||
ExtraParams map[string]string `json:"extra_params,omitempty"`
|
||||
|
||||
@@ -63,17 +63,7 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
var model *domain.ExecutionTarget
|
||||
if req.Model != nil {
|
||||
model = &domain.ExecutionTarget{
|
||||
Endpoint: req.Model.Endpoint,
|
||||
Model: req.Model.Model,
|
||||
Temperature: req.Model.Temperature,
|
||||
MaxTokens: req.Model.MaxTokens,
|
||||
TopP: req.Model.TopP,
|
||||
TimeoutSeconds: req.Model.TimeoutSeconds,
|
||||
ReasoningEffort: req.Model.ReasoningEffort,
|
||||
APIKeyEnv: req.Model.APIKeyEnv,
|
||||
ExtraParams: req.Model.ExtraParams,
|
||||
}
|
||||
model = executionTargetFromModelOverrideDTO(req.Model)
|
||||
}
|
||||
|
||||
res, err := h.runner.Run(r.Context(), domain.RunRequest{
|
||||
@@ -109,18 +99,8 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
SelectedProfileID: res.SelectedProfileID,
|
||||
ModelName: res.ModelName,
|
||||
Endpoint: res.Endpoint,
|
||||
ModelParams: modelParamsDTO{
|
||||
Endpoint: res.EffectiveModelParams.Endpoint,
|
||||
Model: res.EffectiveModelParams.Model,
|
||||
Temperature: res.EffectiveModelParams.Temperature,
|
||||
MaxTokens: res.EffectiveModelParams.MaxTokens,
|
||||
TopP: res.EffectiveModelParams.TopP,
|
||||
TimeoutSeconds: res.EffectiveModelParams.TimeoutSeconds,
|
||||
ReasoningEffort: res.EffectiveModelParams.ReasoningEffort,
|
||||
APIKeyEnv: res.EffectiveModelParams.APIKeyEnv,
|
||||
ExtraParams: res.EffectiveModelParams.ExtraParams,
|
||||
},
|
||||
InputHashes: res.InputHashes,
|
||||
ModelParams: modelParamsDTOFromExecutionTarget(res.EffectiveModelParams),
|
||||
InputHashes: res.InputHashes,
|
||||
Usage: tokenUsageDTO{
|
||||
PromptTokens: res.Usage.PromptTokens,
|
||||
CompletionTokens: res.Usage.CompletionTokens,
|
||||
@@ -141,6 +121,39 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, http.StatusOK, resp)
|
||||
}
|
||||
|
||||
func executionTargetFromModelOverrideDTO(dto *modelOverrideRequestDTO) *domain.ExecutionTarget {
|
||||
if dto == nil {
|
||||
return nil
|
||||
}
|
||||
return &domain.ExecutionTarget{
|
||||
Endpoint: dto.Endpoint,
|
||||
Model: dto.Model,
|
||||
Temperature: dto.Temperature,
|
||||
MaxTokens: dto.MaxTokens,
|
||||
TopP: dto.TopP,
|
||||
TimeoutSeconds: dto.TimeoutSeconds,
|
||||
ServiceTier: dto.ServiceTier,
|
||||
ReasoningEffort: dto.ReasoningEffort,
|
||||
APIKeyEnv: dto.APIKeyEnv,
|
||||
ExtraParams: dto.ExtraParams,
|
||||
}
|
||||
}
|
||||
|
||||
func modelParamsDTOFromExecutionTarget(target domain.ExecutionTarget) modelParamsDTO {
|
||||
return modelParamsDTO{
|
||||
Endpoint: target.Endpoint,
|
||||
Model: target.Model,
|
||||
Temperature: target.Temperature,
|
||||
MaxTokens: target.MaxTokens,
|
||||
TopP: target.TopP,
|
||||
TimeoutSeconds: target.TimeoutSeconds,
|
||||
ServiceTier: target.ServiceTier,
|
||||
ReasoningEffort: target.ReasoningEffort,
|
||||
APIKeyEnv: target.APIKeyEnv,
|
||||
ExtraParams: target.ExtraParams,
|
||||
}
|
||||
}
|
||||
|
||||
func mapValidation(v domain.ValidationResult) validationDTO {
|
||||
return validationDTO{
|
||||
Status: string(v.Status),
|
||||
@@ -162,9 +175,9 @@ func mapRunError(err error) (int, string, string) {
|
||||
return http.StatusBadRequest, "prompt_load_failed", "failed to load prompt definition"
|
||||
case errors.Is(err, profile.ErrInvalidYAML), errors.Is(err, profile.ErrInvalidProfile):
|
||||
return http.StatusBadRequest, "profile_load_failed", "failed to load execution profile"
|
||||
case errors.Is(err, usecase.ErrInvalidRequest) && strings.Contains(err.Error(), "profile id is required either in request or prompt default_profile"):
|
||||
case errors.Is(err, usecase.ErrProfileRequired):
|
||||
return http.StatusBadRequest, "profile_required", "profile_id is required when prompt default_profile is not set"
|
||||
case errors.Is(err, usecase.ErrInvalidRequest) && strings.Contains(err.Error(), "api key environment variable"):
|
||||
case errors.Is(err, usecase.ErrAPIKeyEnvMissing):
|
||||
return http.StatusBadRequest, "api_key_env_missing", "api_key_env is set but the environment variable is missing"
|
||||
case errors.Is(err, usecase.ErrInvalidRequest):
|
||||
return http.StatusBadRequest, "invalid_request", "invalid run request"
|
||||
|
||||
@@ -4,10 +4,10 @@ import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -62,6 +62,7 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||
MaxTokens: 42,
|
||||
TopP: 0.9,
|
||||
TimeoutSeconds: 120,
|
||||
ServiceTier: "priority",
|
||||
APIKeyEnv: envName,
|
||||
},
|
||||
InputHashes: map[string]string{"transcript": "h1"},
|
||||
@@ -81,7 +82,7 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||
"transcript": {"type": "file", "uri": "./t.md"}
|
||||
},
|
||||
"vars": {"k": "v"},
|
||||
"model": {"model": "gpt-x", "timeout_seconds": 120, "api_key_env": "SCRIPTORIUM_API_KEY"}
|
||||
"model": {"model": "gpt-x", "timeout_seconds": 120, "service_tier": "flex", "api_key_env": "SCRIPTORIUM_API_KEY"}
|
||||
}`)
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
@@ -114,6 +115,9 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||
if modelParams["api_key_env"] != envName {
|
||||
t.Fatalf("expected model_params.api_key_env=%q, got %#v", envName, modelParams["api_key_env"])
|
||||
}
|
||||
if modelParams["service_tier"] != "priority" {
|
||||
t.Fatalf("expected model_params.service_tier=priority, got %#v", modelParams["service_tier"])
|
||||
}
|
||||
if strings.Contains(w.Body.String(), secret) {
|
||||
t.Fatalf("response leaked raw API key value: %s", w.Body.String())
|
||||
}
|
||||
@@ -133,6 +137,9 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||
if r.last.Execution.TimeoutSeconds != 120 {
|
||||
t.Fatalf("expected timeout_seconds override 120, got %#v", r.last.Execution)
|
||||
}
|
||||
if r.last.Execution.ServiceTier != "flex" {
|
||||
t.Fatalf("expected service_tier override flex, got %#v", r.last.Execution)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
|
||||
@@ -166,6 +173,136 @@ func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerModelOverrideMapsAllSupportedExecutionFields(t *testing.T) {
|
||||
r := &fakeRunner{result: &domain.RunResult{
|
||||
Artifact: domain.Artifact{Body: []byte("ok")},
|
||||
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
|
||||
EffectiveModelParams: domain.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||
}}
|
||||
h := NewHandler(r)
|
||||
|
||||
reqBody := `{
|
||||
"prompt_id": "prompt-1",
|
||||
"inputs": {"transcript": {"type": "file", "uri": "./t.md"}},
|
||||
"model": {
|
||||
"endpoint": "http://override/v1",
|
||||
"model": "override-model",
|
||||
"temperature": 0.6,
|
||||
"max_tokens": 250,
|
||||
"top_p": 0.85,
|
||||
"timeout_seconds": 33,
|
||||
"service_tier": "flex",
|
||||
"reasoning_effort": "medium",
|
||||
"api_key_env": "SCRIPTORIUM_API_KEY",
|
||||
"extra_params": {"provider_option":"on"}
|
||||
}
|
||||
}`
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(reqBody))
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
if r.last.Execution == nil {
|
||||
t.Fatalf("expected execution override in run request")
|
||||
}
|
||||
got := r.last.Execution
|
||||
if got.Endpoint != "http://override/v1" ||
|
||||
got.Model != "override-model" ||
|
||||
got.Temperature != 0.6 ||
|
||||
got.MaxTokens != 250 ||
|
||||
got.TopP != 0.85 ||
|
||||
got.TimeoutSeconds != 33 ||
|
||||
got.ServiceTier != "flex" ||
|
||||
got.ReasoningEffort != "medium" ||
|
||||
got.APIKeyEnv != "SCRIPTORIUM_API_KEY" {
|
||||
t.Fatalf("unexpected mapped execution target: %+v", got)
|
||||
}
|
||||
if !reflect.DeepEqual(got.ExtraParams, map[string]string{"provider_option": "on"}) {
|
||||
t.Fatalf("unexpected mapped extra_params: %#v", got.ExtraParams)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerResponseMetadataModelParamsIncludesAllSupportedFields(t *testing.T) {
|
||||
r := &fakeRunner{result: &domain.RunResult{
|
||||
Artifact: domain.Artifact{
|
||||
Name: "output",
|
||||
ContentType: "text/plain",
|
||||
Body: []byte("ok"),
|
||||
Size: 2,
|
||||
Hash: "abc",
|
||||
},
|
||||
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic, IsValid: true},
|
||||
EffectiveModelParams: domain.ExecutionTarget{
|
||||
Endpoint: "http://llm/v1",
|
||||
Model: "gpt-test",
|
||||
Temperature: 0.4,
|
||||
MaxTokens: 321,
|
||||
TopP: 0.7,
|
||||
TimeoutSeconds: 45,
|
||||
ServiceTier: "priority",
|
||||
ReasoningEffort: "high",
|
||||
APIKeyEnv: "SCRIPTORIUM_API_KEY",
|
||||
ExtraParams: map[string]string{
|
||||
"provider_option": "on",
|
||||
},
|
||||
},
|
||||
}}
|
||||
h := NewHandler(r)
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p","inputs":{"x":{"type":"file","uri":"a"}}}`))
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
|
||||
var resp map[string]any
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("invalid JSON response: %v", err)
|
||||
}
|
||||
metadata := resp["metadata"].(map[string]any)
|
||||
params := metadata["model_params"].(map[string]any)
|
||||
|
||||
if params["endpoint"] != "http://llm/v1" {
|
||||
t.Fatalf("unexpected endpoint: %#v", params["endpoint"])
|
||||
}
|
||||
if params["model"] != "gpt-test" {
|
||||
t.Fatalf("unexpected model: %#v", params["model"])
|
||||
}
|
||||
if params["temperature"] != 0.4 {
|
||||
t.Fatalf("unexpected temperature: %#v", params["temperature"])
|
||||
}
|
||||
if params["max_tokens"] != float64(321) {
|
||||
t.Fatalf("unexpected max_tokens: %#v", params["max_tokens"])
|
||||
}
|
||||
if params["top_p"] != 0.7 {
|
||||
t.Fatalf("unexpected top_p: %#v", params["top_p"])
|
||||
}
|
||||
if params["timeout_seconds"] != float64(45) {
|
||||
t.Fatalf("unexpected timeout_seconds: %#v", params["timeout_seconds"])
|
||||
}
|
||||
if params["service_tier"] != "priority" {
|
||||
t.Fatalf("unexpected service_tier: %#v", params["service_tier"])
|
||||
}
|
||||
if params["reasoning_effort"] != "high" {
|
||||
t.Fatalf("unexpected reasoning_effort: %#v", params["reasoning_effort"])
|
||||
}
|
||||
if params["api_key_env"] != "SCRIPTORIUM_API_KEY" {
|
||||
t.Fatalf("unexpected api_key_env: %#v", params["api_key_env"])
|
||||
}
|
||||
extraParams, ok := params["extra_params"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("expected extra_params object, got %#v", params["extra_params"])
|
||||
}
|
||||
if extraParams["provider_option"] != "on" {
|
||||
t.Fatalf("unexpected extra_params.provider_option: %#v", extraParams["provider_option"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerInvalidJSON(t *testing.T) {
|
||||
h := NewHandler(&fakeRunner{})
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString("{"))
|
||||
@@ -209,10 +346,10 @@ func TestHandlerUsecaseErrorMapping(t *testing.T) {
|
||||
}{
|
||||
{name: "prompt not found", err: wrap(usecase.ErrProfileLoad, promptdef.ErrPromptDefinitionNotFound), status: http.StatusNotFound, code: "prompt_not_found", message: "prompt definition not found"},
|
||||
{name: "prompt load invalid", err: wrap(usecase.ErrProfileLoad, promptdef.ErrInvalidPromptDefinition), status: http.StatusBadRequest, code: "prompt_load_failed", message: "failed to load prompt definition"},
|
||||
{name: "missing profile/default", err: wrap(usecase.ErrInvalidRequest, errors.New("profile id is required either in request or prompt default_profile")), status: http.StatusBadRequest, code: "profile_required", message: "profile_id is required when prompt default_profile is not set"},
|
||||
{name: "missing profile/default", err: wrap(usecase.ErrInvalidRequest, usecase.ErrProfileRequired), status: http.StatusBadRequest, code: "profile_required", message: "profile_id is required when prompt default_profile is not set"},
|
||||
{name: "profile not found", err: wrap(usecase.ErrProfileLoad, profile.ErrProfileNotFound), status: http.StatusNotFound, code: "profile_not_found", message: "execution profile not found"},
|
||||
{name: "profile invalid", err: wrap(usecase.ErrProfileLoad, profile.ErrInvalidProfile), status: http.StatusBadRequest, code: "profile_load_failed", message: "failed to load execution profile"},
|
||||
{name: "api key env missing", err: wrap(usecase.ErrInvalidRequest, errors.New(`api key environment variable "SCRIPTORIUM_API_KEY" is not set`)), status: http.StatusBadRequest, code: "api_key_env_missing", message: "api_key_env is set but the environment variable is missing"},
|
||||
{name: "api key env missing", err: wrap(usecase.ErrInvalidRequest, usecase.ErrAPIKeyEnvMissing), status: http.StatusBadRequest, code: "api_key_env_missing", message: "api_key_env is set but the environment variable is missing"},
|
||||
{name: "artifact", err: wrap(usecase.ErrArtifactLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "artifact_read_failed", message: "failed to read input artifact", avoidCause: "read failed"},
|
||||
{name: "prompt render", err: wrap(usecase.ErrPromptRender, fmt.Errorf("render failed")), status: http.StatusBadRequest, code: "prompt_render_failed", message: "failed to render prompt", avoidCause: "render failed"},
|
||||
{name: "llm", err: wrap(usecase.ErrLLMGenerate, fmt.Errorf("llm failed")), status: http.StatusBadGateway, code: "llm_failed", message: "model generation request failed", avoidCause: "llm failed"},
|
||||
|
||||
@@ -46,8 +46,8 @@ func TestCompositeReader_Read(t *testing.T) {
|
||||
|
||||
t.Run("unsupported ref type", func(t *testing.T) {
|
||||
ref := domain.ArtifactRef{
|
||||
Type: domain.ArtifactRefS3,
|
||||
URI: "s3://bucket/key",
|
||||
Type: domain.ArtifactRefType("unsupported"),
|
||||
URI: "unsupported://bucket/key",
|
||||
}
|
||||
_, err := reader.Read(ctx, ref)
|
||||
if !errors.Is(err, ErrUnsupportedRefType) {
|
||||
|
||||
@@ -10,7 +10,6 @@ type ArtifactRefType string
|
||||
const (
|
||||
ArtifactRefInline ArtifactRefType = "inline"
|
||||
ArtifactRefFile ArtifactRefType = "file"
|
||||
ArtifactRefS3 ArtifactRefType = "s3"
|
||||
)
|
||||
|
||||
// OutputFormat defines the desired format of the generated artifact.
|
||||
@@ -146,6 +145,7 @@ type ExecutionProfile struct {
|
||||
MaxTokens int `yaml:"max_tokens"`
|
||||
TopP float64 `yaml:"top_p"`
|
||||
TimeoutSeconds int `yaml:"timeout_seconds"`
|
||||
ServiceTier string `yaml:"service_tier"`
|
||||
ReasoningEffort string `yaml:"reasoning_effort"`
|
||||
APIKeyEnv string `yaml:"api_key_env"`
|
||||
ExtraParams map[string]string `yaml:"extra_params"`
|
||||
@@ -159,6 +159,7 @@ type ExecutionTarget struct {
|
||||
MaxTokens int `yaml:"max_tokens" json:"max_tokens"`
|
||||
TopP float64 `yaml:"top_p" json:"top_p"`
|
||||
TimeoutSeconds int `yaml:"timeout_seconds" json:"timeout_seconds"`
|
||||
ServiceTier string `yaml:"service_tier" json:"service_tier"`
|
||||
ReasoningEffort string `yaml:"reasoning_effort" json:"reasoning_effort"`
|
||||
APIKeyEnv string `yaml:"api_key_env" json:"api_key_env"`
|
||||
ExtraParams map[string]string `yaml:"extra_params" json:"extra_params"`
|
||||
|
||||
54
internal/filecatalog/catalog.go
Normal file
54
internal/filecatalog/catalog.go
Normal file
@@ -0,0 +1,54 @@
|
||||
package filecatalog
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// FindYAMLFiles returns sorted full paths for .yaml and .yml files under root.
|
||||
func FindYAMLFiles(ctx context.Context, root string) ([]string, error) {
|
||||
var files []string
|
||||
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
default:
|
||||
}
|
||||
if d.IsDir() {
|
||||
return nil
|
||||
}
|
||||
if !isYAMLFile(d.Name()) {
|
||||
return nil
|
||||
}
|
||||
files = append(files, path)
|
||||
return nil
|
||||
})
|
||||
sort.Strings(files)
|
||||
return files, err
|
||||
}
|
||||
|
||||
// RelativePath computes a clean relative path from root to path.
|
||||
func RelativePath(root string, path string) string {
|
||||
rel, err := filepath.Rel(root, path)
|
||||
if err != nil {
|
||||
return filepath.Clean(path)
|
||||
}
|
||||
return filepath.Clean(rel)
|
||||
}
|
||||
|
||||
// Stem strips .yaml or .yml from a file name.
|
||||
func Stem(name string) string {
|
||||
name = strings.TrimSuffix(name, ".yaml")
|
||||
name = strings.TrimSuffix(name, ".yml")
|
||||
return name
|
||||
}
|
||||
|
||||
func isYAMLFile(name string) bool {
|
||||
return strings.HasSuffix(name, ".yaml") || strings.HasSuffix(name, ".yml")
|
||||
}
|
||||
84
internal/filecatalog/catalog_test.go
Normal file
84
internal/filecatalog/catalog_test.go
Normal file
@@ -0,0 +1,84 @@
|
||||
package filecatalog
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestFindYAMLFilesNestedSortedAndFiltered(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
mustWriteFile(t, filepath.Join(root, "z", "prompt.yml"), "id: z")
|
||||
mustWriteFile(t, filepath.Join(root, "a", "profile.yaml"), "id: a")
|
||||
mustWriteFile(t, filepath.Join(root, "a", "ignore.txt"), "not yaml")
|
||||
mustWriteFile(t, filepath.Join(root, "b", "ignore.yaml.bak"), "not yaml")
|
||||
|
||||
got, err := FindYAMLFiles(context.Background(), root)
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
|
||||
want := []string{
|
||||
filepath.Join(root, "a", "profile.yaml"),
|
||||
filepath.Join(root, "z", "prompt.yml"),
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("expected sorted YAML files %v, got %v", want, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFindYAMLFilesHonorsContextCancellation(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
mustWriteFile(t, filepath.Join(root, "one.yaml"), "id: one")
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
|
||||
_, err := FindYAMLFiles(ctx, root)
|
||||
if !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("expected context.Canceled, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRelativePathNested(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
path := filepath.Join(root, "nested", "profiles", "local.yaml")
|
||||
got := RelativePath(root, path)
|
||||
want := filepath.Join("nested", "profiles", "local.yaml")
|
||||
if got != want {
|
||||
t.Fatalf("expected relative path %q, got %q", want, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStemStripsYAMLExtensions(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
in string
|
||||
want string
|
||||
}{
|
||||
{name: "yaml", in: "prompt.yaml", want: "prompt"},
|
||||
{name: "yml", in: "profile.yml", want: "profile"},
|
||||
{name: "other", in: "file.txt", want: "file.txt"},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := Stem(tc.in); got != tc.want {
|
||||
t.Fatalf("expected %q, got %q", tc.want, got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func mustWriteFile(t *testing.T, path string, content string) {
|
||||
t.Helper()
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
t.Fatalf("failed to create directory: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||
t.Fatalf("failed to write file %q: %v", path, err)
|
||||
}
|
||||
}
|
||||
@@ -106,6 +106,9 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
|
||||
fmt.Fprintf(&b, " max_tokens: %d\n", target.MaxTokens)
|
||||
fmt.Fprintf(&b, " top_p: %g\n", target.TopP)
|
||||
fmt.Fprintf(&b, " timeout_seconds: %d\n", target.TimeoutSeconds)
|
||||
if target.ServiceTier != "" {
|
||||
fmt.Fprintf(&b, " service_tier: %s\n", target.ServiceTier)
|
||||
}
|
||||
if target.ReasoningEffort != "" {
|
||||
fmt.Fprintf(&b, " reasoning_effort: %s\n", target.ReasoningEffort)
|
||||
}
|
||||
|
||||
@@ -28,6 +28,7 @@ func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
|
||||
"max_tokens: 256",
|
||||
"top_p: 0.8",
|
||||
"timeout_seconds: 45",
|
||||
"service_tier: priority",
|
||||
"reasoning_effort: medium",
|
||||
"api_key_env: SCRIPTORIUM_API_KEY",
|
||||
"prompt_hash: prompt-hash",
|
||||
@@ -168,6 +169,7 @@ func samplePreparedRun() *domain.PreparedRun {
|
||||
MaxTokens: 256,
|
||||
TopP: 0.8,
|
||||
TimeoutSeconds: 45,
|
||||
ServiceTier: "priority",
|
||||
ReasoningEffort: "medium",
|
||||
APIKeyEnv: "SCRIPTORIUM_API_KEY",
|
||||
},
|
||||
|
||||
@@ -75,14 +75,6 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
||||
return nil, fmt.Errorf("%w: timeout_seconds must be greater than or equal to 0", ErrInvalidRequest)
|
||||
}
|
||||
|
||||
model := strings.TrimSpace(req.Target.Model)
|
||||
if model == "" {
|
||||
model = strings.TrimSpace(c.defaultModel)
|
||||
}
|
||||
if model == "" {
|
||||
return nil, fmt.Errorf("%w: model is required", ErrInvalidRequest)
|
||||
}
|
||||
|
||||
endpoint := strings.TrimSpace(req.Target.Endpoint)
|
||||
if endpoint == "" {
|
||||
endpoint = c.baseURL
|
||||
@@ -92,33 +84,9 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
||||
}
|
||||
endpoint = strings.TrimRight(endpoint, "/") + defaults.OpenAIChatCompletionsPath
|
||||
|
||||
wireReq := openAIChatRequest{
|
||||
Model: model,
|
||||
}
|
||||
|
||||
wireReq.Messages = make([]openAIChatMessage, 0, len(req.Prompt.Messages))
|
||||
for _, msg := range req.Prompt.Messages {
|
||||
wireReq.Messages = append(wireReq.Messages, openAIChatMessage{
|
||||
Role: msg.Role,
|
||||
Content: msg.Content,
|
||||
})
|
||||
}
|
||||
|
||||
if req.Target.Temperature != 0 {
|
||||
wireReq.Temperature = &req.Target.Temperature
|
||||
}
|
||||
if req.Target.MaxTokens != 0 {
|
||||
wireReq.MaxTokens = &req.Target.MaxTokens
|
||||
}
|
||||
if req.Target.TopP != 0 {
|
||||
wireReq.TopP = &req.Target.TopP
|
||||
}
|
||||
if req.StructuredOutput != nil {
|
||||
responseFormat, err := toOpenAIResponseFormat(req.StructuredOutput)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
|
||||
}
|
||||
wireReq.ResponseFormat = responseFormat
|
||||
wireReq, err := openAIChatRequestFromGenerateRequest(req, c.defaultModel)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
|
||||
}
|
||||
|
||||
payload, err := json.Marshal(wireReq)
|
||||
@@ -187,12 +155,57 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
||||
}, nil
|
||||
}
|
||||
|
||||
func openAIChatRequestFromGenerateRequest(req domain.GenerateRequest, defaultModel string) (openAIChatRequest, error) {
|
||||
model := strings.TrimSpace(req.Target.Model)
|
||||
if model == "" {
|
||||
model = strings.TrimSpace(defaultModel)
|
||||
}
|
||||
if model == "" {
|
||||
return openAIChatRequest{}, errors.New("model is required")
|
||||
}
|
||||
|
||||
wireReq := openAIChatRequest{
|
||||
Model: model,
|
||||
}
|
||||
|
||||
wireReq.Messages = make([]openAIChatMessage, 0, len(req.Prompt.Messages))
|
||||
for _, msg := range req.Prompt.Messages {
|
||||
wireReq.Messages = append(wireReq.Messages, openAIChatMessage{
|
||||
Role: msg.Role,
|
||||
Content: msg.Content,
|
||||
})
|
||||
}
|
||||
|
||||
if req.Target.Temperature != 0 {
|
||||
wireReq.Temperature = &req.Target.Temperature
|
||||
}
|
||||
if req.Target.MaxTokens != 0 {
|
||||
wireReq.MaxTokens = &req.Target.MaxTokens
|
||||
}
|
||||
if req.Target.TopP != 0 {
|
||||
wireReq.TopP = &req.Target.TopP
|
||||
}
|
||||
if strings.TrimSpace(req.Target.ServiceTier) != "" {
|
||||
wireReq.ServiceTier = req.Target.ServiceTier
|
||||
}
|
||||
if req.StructuredOutput != nil {
|
||||
responseFormat, err := toOpenAIResponseFormat(req.StructuredOutput)
|
||||
if err != nil {
|
||||
return openAIChatRequest{}, err
|
||||
}
|
||||
wireReq.ResponseFormat = responseFormat
|
||||
}
|
||||
|
||||
return wireReq, nil
|
||||
}
|
||||
|
||||
type openAIChatRequest struct {
|
||||
Model string `json:"model"`
|
||||
Messages []openAIChatMessage `json:"messages"`
|
||||
Temperature *float64 `json:"temperature,omitempty"`
|
||||
MaxTokens *int `json:"max_tokens,omitempty"`
|
||||
TopP *float64 `json:"top_p,omitempty"`
|
||||
ServiceTier string `json:"service_tier,omitempty"`
|
||||
ResponseFormat *openAIResponseFormat `json:"response_format,omitempty"`
|
||||
}
|
||||
|
||||
|
||||
@@ -61,6 +61,7 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
|
||||
Temperature: 0.4,
|
||||
MaxTokens: 123,
|
||||
TopP: 0.7,
|
||||
ServiceTier: "priority",
|
||||
APIKeyEnv: "SCRIPTORIUM_TEST_API_KEY",
|
||||
},
|
||||
StructuredOutput: &domain.StructuredOutputSpec{
|
||||
@@ -95,6 +96,18 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
|
||||
if got, ok := obs.Body["model"].(string); !ok || got != "gpt-test" {
|
||||
t.Fatalf("unexpected model payload: %#v", obs.Body["model"])
|
||||
}
|
||||
if got, ok := obs.Body["temperature"].(float64); !ok || got != 0.4 {
|
||||
t.Fatalf("unexpected temperature payload: %#v", obs.Body["temperature"])
|
||||
}
|
||||
if got, ok := obs.Body["max_tokens"].(float64); !ok || got != 123 {
|
||||
t.Fatalf("unexpected max_tokens payload: %#v", obs.Body["max_tokens"])
|
||||
}
|
||||
if got, ok := obs.Body["top_p"].(float64); !ok || got != 0.7 {
|
||||
t.Fatalf("unexpected top_p payload: %#v", obs.Body["top_p"])
|
||||
}
|
||||
if got, ok := obs.Body["service_tier"].(string); !ok || got != "priority" {
|
||||
t.Fatalf("unexpected service_tier payload: %#v", obs.Body["service_tier"])
|
||||
}
|
||||
|
||||
msgs, ok := obs.Body["messages"].([]any)
|
||||
if !ok || len(msgs) != 2 {
|
||||
@@ -157,6 +170,46 @@ func TestOpenAICompatibleClientOmitsResponseFormatWhenNoStructuredOutput(t *test
|
||||
if _, exists := observedBody["response_format"]; exists {
|
||||
t.Fatalf("expected response_format omitted, got %#v", observedBody["response_format"])
|
||||
}
|
||||
if _, exists := observedBody["service_tier"]; exists {
|
||||
t.Fatalf("expected service_tier omitted, got %#v", observedBody["service_tier"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientOmitsReasoningEffortAndExtraParams(t *testing.T) {
|
||||
var observedBody map[string]any
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
defer r.Body.Close()
|
||||
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
|
||||
t.Fatalf("failed to decode request body: %v", err)
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, err = client.Generate(context.Background(), domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
|
||||
Target: domain.ExecutionTarget{
|
||||
Model: "model",
|
||||
ReasoningEffort: "high",
|
||||
ExtraParams: map[string]string{
|
||||
"provider_option": "on",
|
||||
},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if _, exists := observedBody["reasoning_effort"]; exists {
|
||||
t.Fatalf("expected reasoning_effort omitted, got %#v", observedBody["reasoning_effort"])
|
||||
}
|
||||
if _, exists := observedBody["extra_params"]; exists {
|
||||
t.Fatalf("expected extra_params omitted, got %#v", observedBody["extra_params"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientNoAuthorizationHeaderWhenNoAPIKey(t *testing.T) {
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/filecatalog"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
@@ -33,40 +34,40 @@ func (r *filesystemRepository) GetProfile(ctx context.Context, id string) (*doma
|
||||
return nil, fmt.Errorf("%w: profile id is required", ErrInvalidProfile)
|
||||
}
|
||||
|
||||
files, err := os.ReadDir(r.dir)
|
||||
files, err := filecatalog.FindYAMLFiles(ctx, r.dir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read profile directory: %w", err)
|
||||
}
|
||||
|
||||
for _, file := range files {
|
||||
var matches []profileMatch
|
||||
for _, fullPath := range files {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
default:
|
||||
}
|
||||
|
||||
if file.IsDir() || (!strings.HasSuffix(file.Name(), ".yaml") && !strings.HasSuffix(file.Name(), ".yml")) {
|
||||
continue
|
||||
}
|
||||
|
||||
fullPath := filepath.Join(r.dir, file.Name())
|
||||
relPath := filecatalog.RelativePath(r.dir, fullPath)
|
||||
fileMatch := filecatalog.Stem(filepath.Base(fullPath)) == id
|
||||
data, err := os.ReadFile(fullPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read profile file %s: %w", file.Name(), err)
|
||||
return nil, fmt.Errorf("failed to read profile file %s: %w", relPath, err)
|
||||
}
|
||||
metadata := readProfileFileMetadata(data)
|
||||
idMatch := fileMatch || metadata.id == id
|
||||
if metadata.hasRawAPIKey {
|
||||
if idMatch {
|
||||
return nil, fmt.Errorf("%w: %s", ErrRawAPIKeyNotAllowed, relPath)
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
var prof domain.ExecutionProfile
|
||||
decoder := yaml.NewDecoder(bytes.NewReader(data))
|
||||
decoder.KnownFields(true)
|
||||
if err := decoder.Decode(&prof); err != nil {
|
||||
if strings.Contains(err.Error(), "field api_key not found") {
|
||||
if strings.TrimSuffix(strings.TrimSuffix(file.Name(), ".yaml"), ".yml") == id {
|
||||
return nil, fmt.Errorf("%w: %s", ErrRawAPIKeyNotAllowed, file.Name())
|
||||
}
|
||||
continue
|
||||
}
|
||||
if strings.TrimSuffix(strings.TrimSuffix(file.Name(), ".yaml"), ".yml") == id {
|
||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, file.Name(), err)
|
||||
if idMatch {
|
||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
|
||||
}
|
||||
continue
|
||||
}
|
||||
@@ -76,16 +77,68 @@ func (r *filesystemRepository) GetProfile(ctx context.Context, id string) (*doma
|
||||
}
|
||||
if err := validateProfile(&prof); err != nil {
|
||||
if errors.Is(err, ErrRawAPIKeyNotAllowed) {
|
||||
return nil, fmt.Errorf("%w: %s", err, file.Name())
|
||||
return nil, fmt.Errorf("%w: %s", err, relPath)
|
||||
}
|
||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, file.Name(), err)
|
||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, relPath, err)
|
||||
}
|
||||
return &prof, nil
|
||||
matches = append(matches, profileMatch{
|
||||
profile: &prof,
|
||||
path: relPath,
|
||||
})
|
||||
}
|
||||
|
||||
if len(matches) > 1 {
|
||||
paths := make([]string, 0, len(matches))
|
||||
for _, match := range matches {
|
||||
paths = append(paths, match.path)
|
||||
}
|
||||
return nil, fmt.Errorf("%w: duplicate execution profile id %q found in: %s", ErrInvalidProfile, id, strings.Join(paths, ", "))
|
||||
}
|
||||
|
||||
if len(matches) == 1 {
|
||||
return matches[0].profile, nil
|
||||
}
|
||||
|
||||
return nil, ErrProfileNotFound
|
||||
}
|
||||
|
||||
type profileMatch struct {
|
||||
profile *domain.ExecutionProfile
|
||||
path string
|
||||
}
|
||||
|
||||
type profileFileMetadata struct {
|
||||
id string
|
||||
hasRawAPIKey bool
|
||||
}
|
||||
|
||||
func readProfileFileMetadata(data []byte) profileFileMetadata {
|
||||
var node yaml.Node
|
||||
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&node); err != nil {
|
||||
return profileFileMetadata{}
|
||||
}
|
||||
if node.Kind != yaml.DocumentNode || len(node.Content) == 0 {
|
||||
return profileFileMetadata{}
|
||||
}
|
||||
mapping := node.Content[0]
|
||||
if mapping.Kind != yaml.MappingNode {
|
||||
return profileFileMetadata{}
|
||||
}
|
||||
|
||||
var metadata profileFileMetadata
|
||||
for i := 0; i+1 < len(mapping.Content); i += 2 {
|
||||
key := mapping.Content[i]
|
||||
value := mapping.Content[i+1]
|
||||
switch key.Value {
|
||||
case "id":
|
||||
metadata.id = strings.TrimSpace(value.Value)
|
||||
case "api_key":
|
||||
metadata.hasRawAPIKey = true
|
||||
}
|
||||
}
|
||||
return metadata
|
||||
}
|
||||
|
||||
func validateProfile(p *domain.ExecutionProfile) error {
|
||||
if strings.TrimSpace(p.ID) == "" {
|
||||
return errors.New("id is required")
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
@@ -58,6 +59,92 @@ func TestFilesystemRepository_GetProfile(t *testing.T) {
|
||||
if p.ReasoningEffort != "medium" {
|
||||
t.Fatalf("unexpected reasoning_effort: %q", p.ReasoningEffort)
|
||||
}
|
||||
if p.ServiceTier != "priority" {
|
||||
t.Fatalf("unexpected service_tier: %q", p.ServiceTier)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("valid nested profile", func(t *testing.T) {
|
||||
nestedDir := filepath.Join(tmpDir, "local")
|
||||
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeProfileTestFile(t, filepath.Join(nestedDir, "nested-local.yaml"), `
|
||||
id: nested-local
|
||||
endpoint: http://localhost:8000/v1
|
||||
model: nested-model
|
||||
temperature: 0.1
|
||||
`)
|
||||
|
||||
p, err := repo.GetProfile(ctx, "nested-local")
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if p.Model != "nested-model" {
|
||||
t.Fatalf("unexpected model: %q", p.Model)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("duplicate profile IDs fail as ambiguous", func(t *testing.T) {
|
||||
writeProfileTestFile(t, filepath.Join(tmpDir, "duplicate-profile-a.yaml"), `
|
||||
id: duplicate-profile
|
||||
endpoint: http://localhost:8000/v1
|
||||
model: first-model
|
||||
`)
|
||||
nestedDir := filepath.Join(tmpDir, "duplicates")
|
||||
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeProfileTestFile(t, filepath.Join(nestedDir, "duplicate-profile-b.yaml"), `
|
||||
id: duplicate-profile
|
||||
endpoint: http://localhost:8000/v1
|
||||
model: second-model
|
||||
`)
|
||||
|
||||
_, err := repo.GetProfile(ctx, "duplicate-profile")
|
||||
if !errors.Is(err, ErrInvalidProfile) {
|
||||
t.Fatalf("expected duplicate profile to return ErrInvalidProfile, got %v", err)
|
||||
}
|
||||
for _, want := range []string{"duplicate execution profile id", "duplicate-profile-a.yaml", filepath.Join("duplicates", "duplicate-profile-b.yaml")} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Fatalf("expected error to contain %q, got %v", want, err)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("nested raw api_key rejected for likely target file", func(t *testing.T) {
|
||||
nestedDir := filepath.Join(tmpDir, "secure")
|
||||
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeProfileTestFile(t, filepath.Join(nestedDir, "not_named_like_id.yaml"), `
|
||||
id: nested_raw_api_key
|
||||
endpoint: http://localhost:8000/v1
|
||||
model: m
|
||||
api_key: secret
|
||||
`)
|
||||
|
||||
_, err := repo.GetProfile(ctx, "nested_raw_api_key")
|
||||
if !errors.Is(err, ErrRawAPIKeyNotAllowed) {
|
||||
t.Fatalf("expected ErrRawAPIKeyNotAllowed, got %v", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), filepath.Join("secure", "not_named_like_id.yaml")) {
|
||||
t.Fatalf("expected nested path in error, got %v", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("raw api_key in non-target profile is ignored", func(t *testing.T) {
|
||||
writeProfileTestFile(t, filepath.Join(tmpDir, "raw-api-key-non-target.yaml"), `
|
||||
id: raw-api-key-non-target
|
||||
endpoint: http://localhost:8000/v1
|
||||
model: m
|
||||
api_key: secret
|
||||
`)
|
||||
|
||||
_, err := repo.GetProfile(ctx, "does-not-exist-with-raw-key-nearby")
|
||||
if !errors.Is(err, ErrProfileNotFound) {
|
||||
t.Fatalf("expected ErrProfileNotFound for non-target raw api_key file, got %v", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("invalid yaml", func(t *testing.T) {
|
||||
@@ -109,3 +196,10 @@ func TestFilesystemRepository_GetProfile(t *testing.T) {
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func writeProfileTestFile(t *testing.T, path string, content string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(path, []byte(strings.TrimLeft(content, "\n")), 0o644); err != nil {
|
||||
t.Fatalf("failed to write profile test file %q: %v", path, err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ id: local-secure
|
||||
endpoint: http://localhost:8000/v1
|
||||
model: gpt-4o-mini
|
||||
api_key_env: SCRIPTORIUM_API_KEY
|
||||
service_tier: priority
|
||||
reasoning_effort: medium
|
||||
extra_params:
|
||||
provider: local
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/domain"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/filecatalog"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
@@ -62,29 +63,26 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
|
||||
return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition)
|
||||
}
|
||||
|
||||
files, err := os.ReadDir(r.dir)
|
||||
files, err := filecatalog.FindYAMLFiles(ctx, r.dir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read prompt definition directory: %w", err)
|
||||
}
|
||||
|
||||
for _, file := range files {
|
||||
var matches []promptDefinitionMatch
|
||||
for _, fullPath := range files {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
default:
|
||||
}
|
||||
|
||||
if file.IsDir() || !isYAMLFile(file.Name()) {
|
||||
continue
|
||||
}
|
||||
|
||||
fullPath := filepath.Join(r.dir, file.Name())
|
||||
fileMatch := promptIDFromFileName(file.Name()) == id
|
||||
relPath := filecatalog.RelativePath(r.dir, fullPath)
|
||||
fileMatch := filecatalog.Stem(filepath.Base(fullPath)) == id
|
||||
|
||||
raw, err := loadPromptDefinitionFile(fullPath)
|
||||
if err != nil {
|
||||
if fileMatch {
|
||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, file.Name(), err)
|
||||
if fileMatch || promptDefinitionFileHasID(fullPath, id) {
|
||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
|
||||
}
|
||||
continue
|
||||
}
|
||||
@@ -92,7 +90,7 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
|
||||
def, err := normalizePromptDefinition(raw, fullPath)
|
||||
if err != nil {
|
||||
if fileMatch || strings.TrimSpace(raw.ID) == id {
|
||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, file.Name(), err)
|
||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, relPath, err)
|
||||
}
|
||||
continue
|
||||
}
|
||||
@@ -103,12 +101,35 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
|
||||
if version != "" && def.Version != version {
|
||||
continue
|
||||
}
|
||||
return def, nil
|
||||
matches = append(matches, promptDefinitionMatch{
|
||||
def: def,
|
||||
path: relPath,
|
||||
})
|
||||
}
|
||||
|
||||
if len(matches) > 1 {
|
||||
paths := make([]string, 0, len(matches))
|
||||
for _, match := range matches {
|
||||
paths = append(paths, match.path)
|
||||
}
|
||||
if version != "" {
|
||||
return nil, fmt.Errorf("%w: duplicate prompt definition id %q version %q found in: %s", ErrInvalidPromptDefinition, id, version, strings.Join(paths, ", "))
|
||||
}
|
||||
return nil, fmt.Errorf("%w: duplicate prompt definition id %q found in: %s", ErrInvalidPromptDefinition, id, strings.Join(paths, ", "))
|
||||
}
|
||||
|
||||
if len(matches) == 1 {
|
||||
return matches[0].def, nil
|
||||
}
|
||||
|
||||
return nil, ErrPromptDefinitionNotFound
|
||||
}
|
||||
|
||||
type promptDefinitionMatch struct {
|
||||
def *domain.PromptDefinition
|
||||
path string
|
||||
}
|
||||
|
||||
func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
@@ -124,6 +145,20 @@ func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
|
||||
return &raw, nil
|
||||
}
|
||||
|
||||
func promptDefinitionFileHasID(path string, id string) bool {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
var raw struct {
|
||||
ID string `yaml:"id"`
|
||||
}
|
||||
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&raw); err != nil {
|
||||
return false
|
||||
}
|
||||
return strings.TrimSpace(raw.ID) == id
|
||||
}
|
||||
|
||||
func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*domain.PromptDefinition, error) {
|
||||
if raw == nil {
|
||||
return nil, errors.New("prompt definition is nil")
|
||||
@@ -239,16 +274,6 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
|
||||
}, nil
|
||||
}
|
||||
|
||||
func isYAMLFile(name string) bool {
|
||||
return strings.HasSuffix(name, ".yaml") || strings.HasSuffix(name, ".yml")
|
||||
}
|
||||
|
||||
func promptIDFromFileName(name string) string {
|
||||
name = strings.TrimSuffix(name, ".yaml")
|
||||
name = strings.TrimSuffix(name, ".yml")
|
||||
return name
|
||||
}
|
||||
|
||||
func isValidOutputFormat(f domain.OutputFormat) bool {
|
||||
switch f {
|
||||
case domain.FormatText, domain.FormatMarkdown, domain.FormatJSON:
|
||||
|
||||
@@ -68,6 +68,39 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("valid nested file-backed prompt resolves content file relative to nested YAML", func(t *testing.T) {
|
||||
nestedDir := filepath.Join(tmpDir, "dnd", "recap")
|
||||
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writePromptTestFile(t, filepath.Join(nestedDir, "nested_recap.yaml"), `
|
||||
id: nested-recap
|
||||
version: "1.0.0"
|
||||
messages:
|
||||
- role: user
|
||||
content_file: ./nested_recap.user.tmpl
|
||||
output:
|
||||
format: markdown
|
||||
validation_mode: basic
|
||||
repair_attempts: 0
|
||||
`)
|
||||
writePromptTestFile(t, filepath.Join(nestedDir, "nested_recap.user.tmpl"), `Nested recap: {{input "transcript"}}`)
|
||||
|
||||
p, err := repo.GetPromptDefinition(ctx, "nested-recap", "")
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if len(p.Templates) != 1 {
|
||||
t.Fatalf("expected one template, got %d", len(p.Templates))
|
||||
}
|
||||
if !strings.Contains(p.Templates[0].Content, "Nested recap") {
|
||||
t.Fatalf("expected nested content file body, got %q", p.Templates[0].Content)
|
||||
}
|
||||
if !strings.Contains(p.Templates[0].ContentFile, filepath.Join("dnd", "recap", "nested_recap.user.tmpl")) {
|
||||
t.Fatalf("expected nested content file path, got %q", p.Templates[0].ContentFile)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("prompt with default_profile", func(t *testing.T) {
|
||||
p, err := repo.GetPromptDefinition(ctx, "with-default-profile", "")
|
||||
if err != nil {
|
||||
@@ -84,6 +117,124 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("duplicate prompt IDs fail as ambiguous", func(t *testing.T) {
|
||||
writePromptTestFile(t, filepath.Join(tmpDir, "duplicate_a.yaml"), `
|
||||
id: duplicate-prompt
|
||||
version: "1.0.0"
|
||||
messages:
|
||||
- role: user
|
||||
content: First duplicate.
|
||||
output:
|
||||
format: markdown
|
||||
validation_mode: basic
|
||||
repair_attempts: 0
|
||||
`)
|
||||
nestedDir := filepath.Join(tmpDir, "nested")
|
||||
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writePromptTestFile(t, filepath.Join(nestedDir, "duplicate_b.yaml"), `
|
||||
id: duplicate-prompt
|
||||
version: "2.0.0"
|
||||
messages:
|
||||
- role: user
|
||||
content: Second duplicate.
|
||||
output:
|
||||
format: markdown
|
||||
validation_mode: basic
|
||||
repair_attempts: 0
|
||||
`)
|
||||
|
||||
_, err := repo.GetPromptDefinition(ctx, "duplicate-prompt", "")
|
||||
if !errors.Is(err, ErrInvalidPromptDefinition) {
|
||||
t.Fatalf("expected duplicate prompt to return ErrInvalidPromptDefinition, got %v", err)
|
||||
}
|
||||
for _, want := range []string{"duplicate prompt definition id", "duplicate_a.yaml", filepath.Join("nested", "duplicate_b.yaml")} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Fatalf("expected error to contain %q, got %v", want, err)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("duplicate prompt ID and requested version fails as ambiguous", func(t *testing.T) {
|
||||
writePromptTestFile(t, filepath.Join(tmpDir, "version_duplicate_a.yaml"), `
|
||||
id: duplicate-version-prompt
|
||||
version: "1.0.0"
|
||||
messages:
|
||||
- role: user
|
||||
content: First duplicate version.
|
||||
output:
|
||||
format: markdown
|
||||
validation_mode: basic
|
||||
repair_attempts: 0
|
||||
`)
|
||||
nestedDir := filepath.Join(tmpDir, "versioned")
|
||||
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writePromptTestFile(t, filepath.Join(nestedDir, "version_duplicate_b.yaml"), `
|
||||
id: duplicate-version-prompt
|
||||
version: "1.0.0"
|
||||
messages:
|
||||
- role: user
|
||||
content: Second duplicate version.
|
||||
output:
|
||||
format: markdown
|
||||
validation_mode: basic
|
||||
repair_attempts: 0
|
||||
`)
|
||||
|
||||
_, err := repo.GetPromptDefinition(ctx, "duplicate-version-prompt", "1.0.0")
|
||||
if !errors.Is(err, ErrInvalidPromptDefinition) {
|
||||
t.Fatalf("expected duplicate prompt version to return ErrInvalidPromptDefinition, got %v", err)
|
||||
}
|
||||
for _, want := range []string{"duplicate prompt definition id", "version \"1.0.0\"", "version_duplicate_a.yaml", filepath.Join("versioned", "version_duplicate_b.yaml")} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Fatalf("expected error to contain %q, got %v", want, err)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("non-matching malformed nested prompt is ignored for not found lookup", func(t *testing.T) {
|
||||
nestedDir := filepath.Join(tmpDir, "broken")
|
||||
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writePromptTestFile(t, filepath.Join(nestedDir, "unrelated.yaml"), "id: [")
|
||||
|
||||
_, err := repo.GetPromptDefinition(ctx, "does-not-exist-even-with-broken-nested-file", "")
|
||||
if !errors.Is(err, ErrPromptDefinitionNotFound) {
|
||||
t.Fatalf("expected ErrPromptDefinitionNotFound, got %v", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("strict decode failure in nested prompt matches by YAML ID", func(t *testing.T) {
|
||||
nestedDir := filepath.Join(tmpDir, "strict")
|
||||
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writePromptTestFile(t, filepath.Join(nestedDir, "not_named_like_id.yaml"), `
|
||||
id: nested-strict-error
|
||||
version: "1.0.0"
|
||||
unknown_field: true
|
||||
messages:
|
||||
- role: user
|
||||
content: Invalid because of unknown field.
|
||||
output:
|
||||
format: markdown
|
||||
validation_mode: basic
|
||||
repair_attempts: 0
|
||||
`)
|
||||
|
||||
_, err := repo.GetPromptDefinition(ctx, "nested-strict-error", "")
|
||||
if !errors.Is(err, ErrInvalidYAML) {
|
||||
t.Fatalf("expected ErrInvalidYAML, got %v", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), filepath.Join("strict", "not_named_like_id.yaml")) {
|
||||
t.Fatalf("expected nested path in error, got %v", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("version lookup", func(t *testing.T) {
|
||||
_, err := repo.GetPromptDefinition(ctx, "valid-inline", "9.9.9")
|
||||
if !errors.Is(err, ErrPromptDefinitionNotFound) {
|
||||
@@ -131,6 +282,13 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
|
||||
})
|
||||
}
|
||||
|
||||
func writePromptTestFile(t *testing.T, path string, content string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(path, []byte(strings.TrimLeft(content, "\n")), 0o644); err != nil {
|
||||
t.Fatalf("failed to write prompt test file %q: %v", path, err)
|
||||
}
|
||||
}
|
||||
|
||||
func copyTree(src, dst string) error {
|
||||
return filepath.WalkDir(src, func(path string, d fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
|
||||
@@ -35,9 +35,9 @@ func TestRunnerIntegrationWithPromptAndProfileFixturesAndValidation(t *testing.T
|
||||
t.Fatalf("failed to resolve repo root: %v", err)
|
||||
}
|
||||
|
||||
promptsDir := filepath.Join(root, "prompts")
|
||||
profilesDir := filepath.Join(root, "profiles")
|
||||
schemasDir := filepath.Join(root, "schemas")
|
||||
promptsDir := filepath.Join(root, "examples", "prompts")
|
||||
profilesDir := filepath.Join(root, "examples", "profiles")
|
||||
schemasDir := filepath.Join(root, "examples", "schemas")
|
||||
fixturesDir := filepath.Join(root, "examples", "fixtures")
|
||||
t.Setenv("SCRIPTORIUM_API_KEY", "test-key")
|
||||
|
||||
|
||||
@@ -24,12 +24,14 @@ import (
|
||||
)
|
||||
|
||||
var (
|
||||
ErrInvalidRequest = errors.New("invalid run request")
|
||||
ErrProfileLoad = errors.New("failed to load prompt definition")
|
||||
ErrArtifactLoad = errors.New("failed to load artifact")
|
||||
ErrPromptRender = errors.New("failed to render prompt")
|
||||
ErrLLMGenerate = errors.New("failed to generate output")
|
||||
ErrValidation = errors.New("failed to validate output")
|
||||
ErrInvalidRequest = errors.New("invalid run request")
|
||||
ErrProfileRequired = errors.New("profile selection is required")
|
||||
ErrAPIKeyEnvMissing = errors.New("api_key_env points to an unset environment variable")
|
||||
ErrProfileLoad = errors.New("failed to load prompt definition")
|
||||
ErrArtifactLoad = errors.New("failed to load artifact")
|
||||
ErrPromptRender = errors.New("failed to render prompt")
|
||||
ErrLLMGenerate = errors.New("failed to generate output")
|
||||
ErrValidation = errors.New("failed to validate output")
|
||||
)
|
||||
|
||||
// Runner executes the Scriptorium core use case.
|
||||
@@ -177,7 +179,7 @@ func (r *Runner) Prepare(ctx context.Context, req domain.RunRequest) (*domain.Pr
|
||||
selectedProfileID = strings.TrimSpace(def.DefaultProfile)
|
||||
}
|
||||
if selectedProfileID == "" {
|
||||
return nil, fmt.Errorf("%w: profile id is required either in request or prompt default_profile", ErrInvalidRequest)
|
||||
return nil, fmt.Errorf("%w: %w: profile id is required either in request or prompt default_profile", ErrInvalidRequest, ErrProfileRequired)
|
||||
}
|
||||
|
||||
execProfile, err := r.profiles.GetProfile(ctx, selectedProfileID)
|
||||
@@ -342,6 +344,9 @@ func mergeExecutionTarget(base domain.ExecutionTarget, override domain.Execution
|
||||
if override.TimeoutSeconds != 0 {
|
||||
out.TimeoutSeconds = override.TimeoutSeconds
|
||||
}
|
||||
if strings.TrimSpace(override.ServiceTier) != "" {
|
||||
out.ServiceTier = override.ServiceTier
|
||||
}
|
||||
if strings.TrimSpace(override.ReasoningEffort) != "" {
|
||||
out.ReasoningEffort = override.ReasoningEffort
|
||||
}
|
||||
@@ -373,7 +378,7 @@ func validateAPIKeyEnv(apiKeyEnv string) error {
|
||||
return nil
|
||||
}
|
||||
if strings.TrimSpace(os.Getenv(envName)) == "" {
|
||||
return fmt.Errorf("api key environment variable %q is not set", envName)
|
||||
return fmt.Errorf("%w: api key environment variable %q is not set", ErrAPIKeyEnvMissing, envName)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -396,6 +401,7 @@ func executionProfileToTarget(p *domain.ExecutionProfile) domain.ExecutionTarget
|
||||
MaxTokens: p.MaxTokens,
|
||||
TopP: p.TopP,
|
||||
TimeoutSeconds: p.TimeoutSeconds,
|
||||
ServiceTier: p.ServiceTier,
|
||||
ReasoningEffort: p.ReasoningEffort,
|
||||
APIKeyEnv: p.APIKeyEnv,
|
||||
ExtraParams: cp,
|
||||
|
||||
@@ -234,6 +234,9 @@ func TestRunnerPrepareMissingExplicitProfileAndMissingDefaultProfileFails(t *tes
|
||||
if !errors.Is(err, ErrInvalidRequest) {
|
||||
t.Fatalf("expected ErrInvalidRequest, got %v", err)
|
||||
}
|
||||
if !errors.Is(err, ErrProfileRequired) {
|
||||
t.Fatalf("expected ErrProfileRequired, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerPrepareSelectedProfileDoesNotExistFails(t *testing.T) {
|
||||
@@ -257,6 +260,7 @@ func TestRunnerPrepareRuntimeOverrideBeatsSelectedProfileValue(t *testing.T) {
|
||||
MaxTokens: 500,
|
||||
TopP: 0.9,
|
||||
TimeoutSeconds: 120,
|
||||
ServiceTier: "priority",
|
||||
},
|
||||
}}
|
||||
runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{forbid: true}, nil)
|
||||
@@ -270,6 +274,7 @@ func TestRunnerPrepareRuntimeOverrideBeatsSelectedProfileValue(t *testing.T) {
|
||||
Model: "override-model",
|
||||
Temperature: 0.7,
|
||||
TimeoutSeconds: 30,
|
||||
ServiceTier: "flex",
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
@@ -281,6 +286,9 @@ func TestRunnerPrepareRuntimeOverrideBeatsSelectedProfileValue(t *testing.T) {
|
||||
if prepared.EffectiveModelParams.TopP != 0.9 {
|
||||
t.Fatalf("expected profile top_p to remain, got %v", prepared.EffectiveModelParams.TopP)
|
||||
}
|
||||
if prepared.EffectiveModelParams.ServiceTier != "flex" {
|
||||
t.Fatalf("expected service_tier override to win, got %q", prepared.EffectiveModelParams.ServiceTier)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerPrepareSelectedProfileBeatsBuiltInDefault(t *testing.T) {
|
||||
@@ -292,6 +300,7 @@ func TestRunnerPrepareSelectedProfileBeatsBuiltInDefault(t *testing.T) {
|
||||
Model: "profile-model",
|
||||
TopP: 0.8,
|
||||
TimeoutSeconds: 90,
|
||||
ServiceTier: "priority",
|
||||
},
|
||||
}}
|
||||
runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{forbid: true}, nil)
|
||||
@@ -310,6 +319,9 @@ func TestRunnerPrepareSelectedProfileBeatsBuiltInDefault(t *testing.T) {
|
||||
if prepared.EffectiveModelParams.TimeoutSeconds != 90 {
|
||||
t.Fatalf("expected profile timeout to beat default, got %d", prepared.EffectiveModelParams.TimeoutSeconds)
|
||||
}
|
||||
if prepared.EffectiveModelParams.ServiceTier != "priority" {
|
||||
t.Fatalf("expected profile service_tier to beat default, got %q", prepared.EffectiveModelParams.ServiceTier)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerPrepareFileBackedPromptBodiesRenderCorrectly(t *testing.T) {
|
||||
@@ -478,6 +490,35 @@ func TestRunnerPrepareJSONSchemaBuildsStructuredOutputSpec(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerPrepareJSONSchemaSchemaLoadFailureReturnsValidationError(t *testing.T) {
|
||||
def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 0)
|
||||
def.Validation.SchemaPath = "missing.schema.json"
|
||||
validator := &fakeValidator{schemaErr: errors.New("schema unavailable")}
|
||||
runner := NewRunner(
|
||||
&fakePromptRepo{def: def},
|
||||
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}},
|
||||
defaultArtifactReader(),
|
||||
defaultRenderer(),
|
||||
&fakeLLM{forbid: true},
|
||||
validator,
|
||||
)
|
||||
|
||||
_, err := runner.Prepare(context.Background(), domain.RunRequest{
|
||||
PromptID: "p",
|
||||
ProfileID: "exec",
|
||||
Inputs: singleInputRef(),
|
||||
})
|
||||
if !errors.Is(err, ErrValidation) {
|
||||
t.Fatalf("expected ErrValidation, got %v", err)
|
||||
}
|
||||
if validator.schemaLoads != 1 {
|
||||
t.Fatalf("expected one schema load attempt, got %d", validator.schemaLoads)
|
||||
}
|
||||
if validator.schemaLoadPath != "missing.schema.json" {
|
||||
t.Fatalf("expected schema path missing.schema.json, got %q", validator.schemaLoadPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerRunJSONSchemaSchemaLoadFailureFailsBeforeLLM(t *testing.T) {
|
||||
def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 0)
|
||||
def.Validation.SchemaPath = "missing.schema.json"
|
||||
@@ -726,6 +767,7 @@ func TestRunnerRunExplicitRuntimeOverrideBeatsSelectedProfileValue(t *testing.T)
|
||||
MaxTokens: 500,
|
||||
TopP: 0.9,
|
||||
TimeoutSeconds: 120,
|
||||
ServiceTier: "priority",
|
||||
},
|
||||
}}
|
||||
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}
|
||||
@@ -740,6 +782,7 @@ func TestRunnerRunExplicitRuntimeOverrideBeatsSelectedProfileValue(t *testing.T)
|
||||
Model: "override-model",
|
||||
Temperature: 0.7,
|
||||
TimeoutSeconds: 30,
|
||||
ServiceTier: "flex",
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
@@ -754,6 +797,9 @@ func TestRunnerRunExplicitRuntimeOverrideBeatsSelectedProfileValue(t *testing.T)
|
||||
if res.EffectiveModelParams.TopP != 0.9 {
|
||||
t.Fatalf("expected non-overridden profile top_p to remain, got %v", res.EffectiveModelParams.TopP)
|
||||
}
|
||||
if res.EffectiveModelParams.ServiceTier != "flex" {
|
||||
t.Fatalf("expected service_tier override to win, got %q", res.EffectiveModelParams.ServiceTier)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerRunSelectedProfileBeatsBuiltInDefault(t *testing.T) {
|
||||
@@ -765,6 +811,7 @@ func TestRunnerRunSelectedProfileBeatsBuiltInDefault(t *testing.T) {
|
||||
Model: "profile-model",
|
||||
TopP: 0.8,
|
||||
TimeoutSeconds: 90,
|
||||
ServiceTier: "priority",
|
||||
},
|
||||
}}
|
||||
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}
|
||||
@@ -784,6 +831,9 @@ func TestRunnerRunSelectedProfileBeatsBuiltInDefault(t *testing.T) {
|
||||
if res.EffectiveModelParams.TimeoutSeconds != 90 {
|
||||
t.Fatalf("expected profile timeout to beat default, got %d", res.EffectiveModelParams.TimeoutSeconds)
|
||||
}
|
||||
if res.EffectiveModelParams.ServiceTier != "priority" {
|
||||
t.Fatalf("expected profile service_tier to beat default, got %q", res.EffectiveModelParams.ServiceTier)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerRunBuiltInDefaultsUsedWhenProfileOmitsOptionalFields(t *testing.T) {
|
||||
@@ -844,6 +894,9 @@ func TestRunnerRunAPIKeyEnvMissingEnvironmentValueFailsClearly(t *testing.T) {
|
||||
if !errors.Is(err, ErrInvalidRequest) {
|
||||
t.Fatalf("expected ErrInvalidRequest, got %v", err)
|
||||
}
|
||||
if !errors.Is(err, ErrAPIKeyEnvMissing) {
|
||||
t.Fatalf("expected ErrAPIKeyEnvMissing, got %v", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "SCRIPTORIUM_MISSING_KEY") {
|
||||
t.Fatalf("expected missing env name in error, got %v", err)
|
||||
}
|
||||
@@ -1104,6 +1157,174 @@ func TestRunnerRunJSONSchemaRepairCarriesStructuredOutputSpec(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutionProfileToTargetPopulatesAllFieldsAndCopiesExtraParams(t *testing.T) {
|
||||
src := &domain.ExecutionProfile{
|
||||
ID: "exec",
|
||||
Endpoint: "http://profile/v1",
|
||||
Model: "profile-model",
|
||||
Temperature: 0.2,
|
||||
MaxTokens: 123,
|
||||
TopP: 0.75,
|
||||
TimeoutSeconds: 90,
|
||||
ServiceTier: "priority",
|
||||
ReasoningEffort: "medium",
|
||||
APIKeyEnv: "SCRIPTORIUM_API_KEY",
|
||||
ExtraParams: map[string]string{
|
||||
"provider_option": "on",
|
||||
},
|
||||
}
|
||||
|
||||
target := executionProfileToTarget(src)
|
||||
if target.Endpoint != src.Endpoint ||
|
||||
target.Model != src.Model ||
|
||||
target.Temperature != src.Temperature ||
|
||||
target.MaxTokens != src.MaxTokens ||
|
||||
target.TopP != src.TopP ||
|
||||
target.TimeoutSeconds != src.TimeoutSeconds ||
|
||||
target.ServiceTier != src.ServiceTier ||
|
||||
target.ReasoningEffort != src.ReasoningEffort ||
|
||||
target.APIKeyEnv != src.APIKeyEnv {
|
||||
t.Fatalf("expected all profile fields to populate target, got %+v", target)
|
||||
}
|
||||
if !reflect.DeepEqual(target.ExtraParams, src.ExtraParams) {
|
||||
t.Fatalf("expected extra_params to match, got %#v", target.ExtraParams)
|
||||
}
|
||||
|
||||
src.ExtraParams["provider_option"] = "changed"
|
||||
if target.ExtraParams["provider_option"] != "on" {
|
||||
t.Fatalf("expected extra_params copy to be independent, got %#v", target.ExtraParams)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveExecutionTargetProfileValuesPopulateAllSupportedFields(t *testing.T) {
|
||||
profileValue := &domain.ExecutionProfile{
|
||||
ID: "exec",
|
||||
Endpoint: "http://profile/v1",
|
||||
Model: "profile-model",
|
||||
Temperature: 0.3,
|
||||
MaxTokens: 222,
|
||||
TopP: 0.6,
|
||||
TimeoutSeconds: 77,
|
||||
ServiceTier: "priority",
|
||||
ReasoningEffort: "low",
|
||||
APIKeyEnv: "PROFILE_KEY",
|
||||
ExtraParams: map[string]string{
|
||||
"profile_option": "enabled",
|
||||
},
|
||||
}
|
||||
|
||||
target := resolveExecutionTarget(profileValue, nil)
|
||||
if target.Endpoint != profileValue.Endpoint ||
|
||||
target.Model != profileValue.Model ||
|
||||
target.Temperature != profileValue.Temperature ||
|
||||
target.MaxTokens != profileValue.MaxTokens ||
|
||||
target.TopP != profileValue.TopP ||
|
||||
target.TimeoutSeconds != profileValue.TimeoutSeconds ||
|
||||
target.ServiceTier != profileValue.ServiceTier ||
|
||||
target.ReasoningEffort != profileValue.ReasoningEffort ||
|
||||
target.APIKeyEnv != profileValue.APIKeyEnv {
|
||||
t.Fatalf("expected profile values to populate target, got %+v", target)
|
||||
}
|
||||
if !reflect.DeepEqual(target.ExtraParams, profileValue.ExtraParams) {
|
||||
t.Fatalf("expected profile extra_params in target, got %#v", target.ExtraParams)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveExecutionTargetRuntimeOverridesBeatProfileForAllOverrideableFields(t *testing.T) {
|
||||
profileValue := &domain.ExecutionProfile{
|
||||
ID: "exec",
|
||||
Endpoint: "http://profile/v1",
|
||||
Model: "profile-model",
|
||||
Temperature: 0.2,
|
||||
MaxTokens: 200,
|
||||
TopP: 0.8,
|
||||
TimeoutSeconds: 90,
|
||||
ServiceTier: "priority",
|
||||
ReasoningEffort: "medium",
|
||||
APIKeyEnv: "PROFILE_KEY",
|
||||
ExtraParams: map[string]string{
|
||||
"profile_only": "yes",
|
||||
},
|
||||
}
|
||||
override := &domain.ExecutionTarget{
|
||||
Endpoint: "http://override/v1",
|
||||
Model: "override-model",
|
||||
Temperature: 0.9,
|
||||
MaxTokens: 111,
|
||||
TopP: 0.5,
|
||||
TimeoutSeconds: 30,
|
||||
ServiceTier: "flex",
|
||||
ReasoningEffort: "high",
|
||||
APIKeyEnv: "RUNTIME_KEY",
|
||||
ExtraParams: map[string]string{
|
||||
"runtime_only": "yes",
|
||||
},
|
||||
}
|
||||
|
||||
target := resolveExecutionTarget(profileValue, override)
|
||||
if target.Endpoint != override.Endpoint ||
|
||||
target.Model != override.Model ||
|
||||
target.Temperature != override.Temperature ||
|
||||
target.MaxTokens != override.MaxTokens ||
|
||||
target.TopP != override.TopP ||
|
||||
target.TimeoutSeconds != override.TimeoutSeconds ||
|
||||
target.ServiceTier != override.ServiceTier ||
|
||||
target.ReasoningEffort != override.ReasoningEffort ||
|
||||
target.APIKeyEnv != override.APIKeyEnv {
|
||||
t.Fatalf("expected runtime overrides to win for all fields, got %+v", target)
|
||||
}
|
||||
if !reflect.DeepEqual(target.ExtraParams, override.ExtraParams) {
|
||||
t.Fatalf("expected runtime extra_params to replace profile extra_params, got %#v", target.ExtraParams)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeExecutionTargetEmptyStringOverridesDoNotErase(t *testing.T) {
|
||||
base := domain.ExecutionTarget{
|
||||
Endpoint: "http://base/v1",
|
||||
Model: "base-model",
|
||||
ServiceTier: "priority",
|
||||
ReasoningEffort: "medium",
|
||||
APIKeyEnv: "BASE_KEY",
|
||||
}
|
||||
override := domain.ExecutionTarget{
|
||||
Endpoint: "http://override/v1",
|
||||
Model: "override-model",
|
||||
ServiceTier: " ",
|
||||
ReasoningEffort: " ",
|
||||
APIKeyEnv: "",
|
||||
}
|
||||
|
||||
merged := mergeExecutionTarget(base, override)
|
||||
if merged.Endpoint != "http://override/v1" || merged.Model != "override-model" {
|
||||
t.Fatalf("expected endpoint/model to override, got %+v", merged)
|
||||
}
|
||||
if merged.ServiceTier != "priority" {
|
||||
t.Fatalf("expected empty service_tier override to be ignored, got %q", merged.ServiceTier)
|
||||
}
|
||||
if merged.ReasoningEffort != "medium" {
|
||||
t.Fatalf("expected empty reasoning_effort override to be ignored, got %q", merged.ReasoningEffort)
|
||||
}
|
||||
if merged.APIKeyEnv != "BASE_KEY" {
|
||||
t.Fatalf("expected empty api_key_env override to be ignored, got %q", merged.APIKeyEnv)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeExecutionTargetEmptyExtraParamsDoesNotErase(t *testing.T) {
|
||||
base := domain.ExecutionTarget{
|
||||
ExtraParams: map[string]string{
|
||||
"keep": "value",
|
||||
},
|
||||
}
|
||||
override := domain.ExecutionTarget{
|
||||
ExtraParams: map[string]string{},
|
||||
}
|
||||
|
||||
merged := mergeExecutionTarget(base, override)
|
||||
if !reflect.DeepEqual(merged.ExtraParams, base.ExtraParams) {
|
||||
t.Fatalf("expected empty extra_params override not to erase base values, got %#v", merged.ExtraParams)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildOutputArtifactDefaults(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
|
||||
@@ -116,6 +116,49 @@ func TestStandardValidatorJSONSchemaSuccess(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestStandardValidatorJSONSchemaNestedSchemaPathSuccess(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
nestedDir := filepath.Join(tmp, "dnd")
|
||||
if err := os.MkdirAll(nestedDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(nestedDir, "schema.json"), []byte(`{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"type": "object",
|
||||
"required": ["name"],
|
||||
"properties": {
|
||||
"name": {"type": "string"}
|
||||
}
|
||||
}`), 0644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
v := NewStandardValidator(tmp)
|
||||
|
||||
res, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"name":"eris"}`)}, domain.OutputContract{
|
||||
ValidationMode: domain.ValidationJSONSchema,
|
||||
SchemaPath: filepath.Join("dnd", "schema.json"),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
if res.Status != domain.ValidationPassed || !res.IsValid {
|
||||
t.Fatalf("expected passed/valid, got status=%q valid=%v", res.Status, res.IsValid)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStandardValidatorJSONSchemaNestedSchemaPathMissing(t *testing.T) {
|
||||
v := NewStandardValidator(t.TempDir())
|
||||
|
||||
_, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{"name":"eris"}`)}, domain.OutputContract{
|
||||
ValidationMode: domain.ValidationJSONSchema,
|
||||
SchemaPath: filepath.Join("dnd", "missing.json"),
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("expected nested schema load error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestStandardValidatorJSONSchemaFailure(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
schemaPath := filepath.Join(tmp, "schema.json")
|
||||
|
||||
Reference in New Issue
Block a user