Compare commits
18 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| f33af20524 | |||
| 85d14784ca | |||
| edca9fbbc1 | |||
| 6c2541c1ee | |||
| b9b98b2aec | |||
| 8936ca7c18 | |||
| 3e5f5c5198 | |||
| 68ebe9ee50 | |||
| 36c7c5a358 | |||
| 157209f097 | |||
| 06b37c2bae | |||
| 1a0f15e210 | |||
| 5f946a5a1f | |||
| 1806df9888 | |||
| c35ac5a0cb | |||
| 40f8a1c628 | |||
| 27d7ad5057 | |||
| 1f08a1a94c |
@@ -6,7 +6,7 @@ OpenAI-compatible model endpoints, and returns generated output with validation
|
|||||||
metadata.
|
metadata.
|
||||||
|
|
||||||
The application uses
|
The application uses
|
||||||
[Promptkit v0.1.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/)
|
[Promptkit v0.9.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/)
|
||||||
for prompt, profile, schema, preparation, generation, and validation behavior.
|
for prompt, profile, schema, preparation, generation, and validation behavior.
|
||||||
Scriptorium owns executable configuration, CLI and HTTP mapping, process
|
Scriptorium owns executable configuration, CLI and HTTP mapping, process
|
||||||
behavior, output presentation, and HTTP artifact-containment policy.
|
behavior, output presentation, and HTTP artifact-containment policy.
|
||||||
@@ -38,8 +38,8 @@ a model. For complete invocation and output behavior, see the
|
|||||||
- [Migration from the former Go package](docs/consumers/migrating-to-promptkit.md)
|
- [Migration from the former Go package](docs/consumers/migrating-to-promptkit.md)
|
||||||
- [Subprocess integration](docs/integrations/subprocess.md)
|
- [Subprocess integration](docs/integrations/subprocess.md)
|
||||||
- [Architecture policy](docs/policy/architecture.md)
|
- [Architecture policy](docs/policy/architecture.md)
|
||||||
- [Promptkit framework formats](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/formats.md)
|
- [Promptkit framework formats](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md)
|
||||||
- [Promptkit Go consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/consumers/pkg-promptkit.md)
|
- [Promptkit Go consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/consumers/pkg-promptkit.md)
|
||||||
|
|
||||||
## Examples
|
## Examples
|
||||||
|
|
||||||
|
|||||||
112
docs/adr/0004-definition-boundary.md
Normal file
112
docs/adr/0004-definition-boundary.md
Normal file
@@ -0,0 +1,112 @@
|
|||||||
|
# Preserve Definition Compatibility And A Simple Execution Model
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Accepted.
|
||||||
|
|
||||||
|
## Date
|
||||||
|
|
||||||
|
2026-08-29.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Scriptorium is a user-facing application built on Promptkit. Promptkit is a Go
|
||||||
|
library with both declarative framework formats and public APIs intended for
|
||||||
|
applications that need to assemble specialized integrations or workflows.
|
||||||
|
|
||||||
|
Scriptorium needs a durable rule for deciding which upstream capabilities it
|
||||||
|
must support. Treating every new Promptkit public API as an application feature
|
||||||
|
would steadily expand Scriptorium into a workflow framework. Selecting an
|
||||||
|
arbitrary subset of prompt or profile fields would instead make valid Promptkit
|
||||||
|
definitions unexpectedly unusable through Scriptorium.
|
||||||
|
|
||||||
|
The intended product is narrower: Scriptorium provides simple executable
|
||||||
|
interfaces that select and execute predefined prompts and return their results
|
||||||
|
in a consistent, repeatable form. It is not intended to own conversations,
|
||||||
|
modify prompts dynamically, coordinate follow-up messages, or manage complex
|
||||||
|
multi-step workflows.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Scriptorium will aim to support the complete set of features expressible in
|
||||||
|
valid Promptkit prompt and profile definitions for the Promptkit version it
|
||||||
|
selects. This includes the schema, backend, source, and request plumbing needed
|
||||||
|
to prepare and execute those definitions without Scriptorium imposing a
|
||||||
|
narrower field-level format contract.
|
||||||
|
|
||||||
|
Promptkit remains the canonical parser and validator for its definitions.
|
||||||
|
Scriptorium will pass its configured framework sources and mapped request
|
||||||
|
values through the supported Promptkit public facade rather than copying
|
||||||
|
Promptkit formats, defaults, or validation rules.
|
||||||
|
|
||||||
|
Promptkit library features that are not required to use valid prompt or profile
|
||||||
|
definitions are considered individually. They may be incorporated when they
|
||||||
|
serve Scriptorium's simple execution model and fit its application-owned CLI,
|
||||||
|
HTTP, configuration, presentation, security, and process boundaries. A public
|
||||||
|
Promptkit declaration does not by itself require a corresponding Scriptorium
|
||||||
|
interface.
|
||||||
|
|
||||||
|
Each execution request will remain self-contained from the caller's
|
||||||
|
perspective. A request must not depend on conversation or workflow state
|
||||||
|
retained from a completed request, and Scriptorium will not become a general
|
||||||
|
workflow or conversation system. Application-managed prompt modification,
|
||||||
|
appended or follow-up message flows, conversation state, prepared-handle
|
||||||
|
coordination, multi-step orchestration, and similar higher-level workflows are
|
||||||
|
outside its intended scope.
|
||||||
|
|
||||||
|
Scriptorium may, and when necessary must, maintain process-scoped operational
|
||||||
|
state shared by concurrent requests. This includes the configured Promptkit
|
||||||
|
engine and backend registry, capacity admission and queue accounting,
|
||||||
|
in-flight request coordination, cancellation, and ordinary server lifecycle
|
||||||
|
resources. The state must be scoped so configured backend limits apply across
|
||||||
|
all relevant in-flight requests. But this is operational implementation state,
|
||||||
|
not durable user workflow or conversation state.
|
||||||
|
|
||||||
|
## Alternatives Considered
|
||||||
|
|
||||||
|
### Expose Every Promptkit Public Feature
|
||||||
|
|
||||||
|
This would make the application surface track the library API closely, but it
|
||||||
|
would blur the boundary between a reusable Go library and a simple executable
|
||||||
|
consumer. Library-oriented lifecycle, source-construction, extension, and
|
||||||
|
workflow primitives would add commands, wire contracts, configuration, and
|
||||||
|
state without necessarily improving predefined prompt execution.
|
||||||
|
|
||||||
|
### Support A Curated Subset Of Definition Fields
|
||||||
|
|
||||||
|
This would keep the application small in the short term, but valid Promptkit
|
||||||
|
definitions could fail or behave differently solely because they were invoked
|
||||||
|
through Scriptorium. Maintaining a parallel field-level compatibility list
|
||||||
|
would also duplicate upstream format ownership and create recurring drift.
|
||||||
|
|
||||||
|
### Expand Scriptorium Into A Workflow Service
|
||||||
|
|
||||||
|
Scriptorium could own conversations, prompt changes, follow-up messages,
|
||||||
|
retries, checkpoints, and multi-step execution. That is a different product
|
||||||
|
with durable state, lifecycle, recovery, privacy, and operational requirements
|
||||||
|
that are not justified by Scriptorium's current purpose.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- Promptkit upgrades require an audit of the complete prompt and profile format
|
||||||
|
contracts, not only a source-compatibility build.
|
||||||
|
- Scriptorium adapters must not add unconditional request requirements that
|
||||||
|
reject a definition Promptkit can validly prepare or execute.
|
||||||
|
- Application configuration may need new plumbing, such as custom backend
|
||||||
|
registration, when it is necessary to use a valid profile feature.
|
||||||
|
- Definition parsing, validation, precedence, inheritance, defaults, and
|
||||||
|
built-in catalogs remain Promptkit responsibilities.
|
||||||
|
- A running HTTP server must share its appropriately scoped Promptkit engine
|
||||||
|
across requests so the engine's backend admission and concurrency limits
|
||||||
|
cannot be bypassed by per-request construction.
|
||||||
|
- Process-scoped counters, queues, in-flight request records, cancellation, and
|
||||||
|
lifecycle resources are permitted implementation state. They must be
|
||||||
|
concurrency-safe and do not create a durable resume or conversation
|
||||||
|
contract.
|
||||||
|
- Library-only source constructors, injected collaborators, opaque lifecycle
|
||||||
|
handles, and request-composition primitives remain optional Scriptorium
|
||||||
|
features rather than automatic requirements.
|
||||||
|
- New application features must be evaluated against the simple,
|
||||||
|
request-independent predefined-prompt execution model.
|
||||||
|
- Work that implements this decision remains tracked in roadmaps until it is
|
||||||
|
delivered and incorporated into current-state contracts.
|
||||||
43
docs/api.md
43
docs/api.md
@@ -32,14 +32,11 @@ returns `413 response_too_large`.
|
|||||||
### Request Body
|
### Request Body
|
||||||
|
|
||||||
The maintained [request example](../examples/http-run.json) is a complete
|
The maintained [request example](../examples/http-run.json) is a complete
|
||||||
copyable shape. The smallest valid shape is:
|
copyable shape. At the HTTP adapter boundary, the smallest valid shape is:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"prompt_id": "generic.markdown_summary",
|
"prompt_id": "generic.markdown_summary"
|
||||||
"inputs": {
|
|
||||||
"transcript": {"type": "inline", "body": "Source text"}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -48,7 +45,8 @@ copyable shape. The smallest valid shape is:
|
|||||||
| `prompt_id` | yes | Non-blank prompt ID. |
|
| `prompt_id` | yes | Non-blank prompt ID. |
|
||||||
| `prompt_version` | no | Prompt version filter. |
|
| `prompt_version` | no | Prompt version filter. |
|
||||||
| `profile_id` | no | Execution-profile ID; otherwise the prompt must set `default_profile`. |
|
| `profile_id` | no | Execution-profile ID; otherwise the prompt must set `default_profile`. |
|
||||||
| `inputs` | yes | Non-empty object mapping input names to references. |
|
| `session_id` | no | Optional direct, non-secret session identifier. |
|
||||||
|
| `inputs` | no | Optional object mapping input names to references. Promptkit decides whether the selected definition needs them. |
|
||||||
| `vars` | no | Object mapping template-variable names to strings. |
|
| `vars` | no | Object mapping template-variable names to strings. |
|
||||||
| `model` | no | Runtime model-override object. |
|
| `model` | no | Runtime model-override object. |
|
||||||
| `include_raw_output` | no | Include `raw_model_output` when true. |
|
| `include_raw_output` | no | Include `raw_model_output` when true. |
|
||||||
@@ -67,20 +65,31 @@ The optional `model` object accepts `endpoint`, `model`, `temperature`,
|
|||||||
`max_tokens`, `top_p`, `timeout_seconds`, `service_tier`,
|
`max_tokens`, `top_p`, `timeout_seconds`, `service_tier`,
|
||||||
`reasoning_effort`, `api_key_env`, and `extra_params`. Numeric ranges and
|
`reasoning_effort`, `api_key_env`, and `extra_params`. Numeric ranges and
|
||||||
framework credential semantics are defined by the
|
framework credential semantics are defined by the
|
||||||
[Promptkit format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/formats.md).
|
[Promptkit format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md).
|
||||||
Explicit zero values for the numeric fields are overrides; zero
|
Explicit zero values for the numeric fields are overrides; zero
|
||||||
`timeout_seconds` disables the per-generation deadline only, retaining the
|
`timeout_seconds` disables the per-generation deadline only, retaining the
|
||||||
request context and configured transport cap. The timeout layers are defined in
|
request context and configured transport cap. The timeout layers are defined in
|
||||||
the [Promptkit outbound integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/integrations/openai-compatible-chat.md#timeout-and-cancellation).
|
the [Promptkit outbound integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/integrations/openai-compatible-chat.md#timeout-and-cancellation).
|
||||||
|
|
||||||
Raw API-key values are not accepted. `api_key` and any other unknown model
|
Raw API-key values are not accepted. `api_key` and any other unknown model
|
||||||
field cause `400 invalid_json`.
|
field cause `400 invalid_json`.
|
||||||
|
|
||||||
|
`model.reasoning_effort` is an optional JSON string with three states: omission
|
||||||
|
inherits the selected profile, a non-empty string replaces its value, and an
|
||||||
|
empty string explicitly clears it. JSON `null` is treated as omission.
|
||||||
|
|
||||||
|
`session_id` is passed directly to Promptkit. A nonblank value replaces a
|
||||||
|
definition-rendered session ID; omission or a blank value lets the definition
|
||||||
|
provide one. Promptkit trims direct values and limits them to 256 Unicode code
|
||||||
|
points. Session IDs are not credentials and may be included in prepared data,
|
||||||
|
results, and provider-facing requests, so use stable non-sensitive identifiers.
|
||||||
|
|
||||||
### Strict JSON
|
### Strict JSON
|
||||||
|
|
||||||
Request decoding rejects malformed JSON, unknown fields at every request level,
|
Request decoding rejects malformed JSON, unknown fields at every request level,
|
||||||
and trailing JSON tokens with `400 invalid_json`. A blank `prompt_id` or
|
and trailing JSON tokens with `400 invalid_json`. A blank `prompt_id` returns
|
||||||
empty `inputs` object returns `400 invalid_request`.
|
`400 invalid_request`. Omitted or empty `inputs` are passed to Promptkit, which
|
||||||
|
reports any definition-required or template-referenced inputs.
|
||||||
|
|
||||||
### Success Response
|
### Success Response
|
||||||
|
|
||||||
@@ -91,17 +100,26 @@ validation contract. The response contains:
|
|||||||
optional `uri`;
|
optional `uri`;
|
||||||
- `validation`: `status`, `mode`, `repair_attempts`, `is_valid`, plus
|
- `validation`: `status`, `mode`, `repair_attempts`, `is_valid`, plus
|
||||||
optional `errors` and `schema_path`;
|
optional `errors` and `schema_path`;
|
||||||
- `metadata`: run, prompt, rendered-prompt, profile, model, input-hash, usage,
|
- `metadata`: run, prompt, rendered-prompt, profile, optional backend identity, model, input-hash, usage,
|
||||||
timing, validation, and repair-attempt metadata; and
|
timing, validation, and repair-attempt metadata; and
|
||||||
- optional `raw_model_output` when requested.
|
- optional `raw_model_output` when requested.
|
||||||
|
|
||||||
`metadata.model_params` has `endpoint`, `model`, `temperature`,
|
`metadata.model_params` has `endpoint`, `model`, `temperature`,
|
||||||
`max_tokens`, `top_p`, and `timeout_seconds`, plus optional
|
`max_tokens`, `top_p`, and `timeout_seconds`, plus optional
|
||||||
`service_tier`, `reasoning_effort`, `api_key_env`, and `extra_params`.
|
`backend_id`, `service_tier`, `reasoning_effort`, `api_key_env`, and `extra_params`.
|
||||||
`metadata.usage` always includes `prompt_tokens`, `completion_tokens`,
|
`metadata.usage` always includes `prompt_tokens`, `completion_tokens`,
|
||||||
`total_tokens`, `cached_tokens`, and `cache_write_tokens`; unavailable
|
`total_tokens`, `cached_tokens`, and `cache_write_tokens`; unavailable
|
||||||
cache usage is reported as zero.
|
cache usage is reported as zero.
|
||||||
|
|
||||||
|
When Promptkit resolves a direct or definition-rendered session ID,
|
||||||
|
`metadata.session_id` contains that effective result value. It is omitted when
|
||||||
|
no effective session ID exists.
|
||||||
|
|
||||||
|
`metadata.selected_backend_id` and `metadata.model_params.backend_id` report
|
||||||
|
the corresponding Promptkit result fields independently when present. Both are
|
||||||
|
omitted for an endpoint-only profile; Scriptorium does not infer backend
|
||||||
|
identity from an endpoint.
|
||||||
|
|
||||||
A validation failure has `validation.status: "failed"`, `is_valid: false`,
|
A validation failure has `validation.status: "failed"`, `is_valid: false`,
|
||||||
and any available diagnostic errors, while still returning the artifact and
|
and any available diagnostic errors, while still returning the artifact and
|
||||||
metadata.
|
metadata.
|
||||||
@@ -137,6 +155,7 @@ Messages are concise and do not expose wrapped internal causes.
|
|||||||
| `500` | `validation_runtime_failed` | Schema or validator runtime failure. |
|
| `500` | `validation_runtime_failed` | Schema or validator runtime failure. |
|
||||||
| `500` | `internal_error` | Unclassified server failure. |
|
| `500` | `internal_error` | Unclassified server failure. |
|
||||||
| `502` | `llm_failed` | Outbound model request failed. |
|
| `502` | `llm_failed` | Outbound model request failed. |
|
||||||
|
| `503` | `capacity_exceeded` | The selected model backend has no admission capacity. No retry timing is supplied. |
|
||||||
|
|
||||||
## Retry And Idempotency
|
## Retry And Idempotency
|
||||||
|
|
||||||
|
|||||||
72
docs/cli.md
72
docs/cli.md
@@ -23,6 +23,8 @@ go run ./cmd/scriptorium render \
|
|||||||
generated artifact.
|
generated artifact.
|
||||||
- `scriptorium render`: prepare a prompt and write prepared-run output.
|
- `scriptorium render`: prepare a prompt and write prepared-run output.
|
||||||
- `scriptorium serve`: start the HTTP server.
|
- `scriptorium serve`: start the HTTP server.
|
||||||
|
- `scriptorium inspect prompt`: inspect one prompt definition without model execution.
|
||||||
|
- `scriptorium inspect profile`: inspect one effective profile without model execution.
|
||||||
|
|
||||||
All commands accept `--config <path>` and reject positional arguments. An
|
All commands accept `--config <path>` and reject positional arguments. An
|
||||||
effective `prompt_dir` is required for every command. Supply it through the
|
effective `prompt_dir` is required for every command. Supply it through the
|
||||||
@@ -39,7 +41,6 @@ Required flags:
|
|||||||
| Flag | Meaning |
|
| Flag | Meaning |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `--prompt <id>` | Prompt ID to execute. |
|
| `--prompt <id>` | Prompt ID to execute. |
|
||||||
| `--input name=path` | Input file mapping; repeat or use comma-separated mappings. |
|
|
||||||
|
|
||||||
Optional flags:
|
Optional flags:
|
||||||
|
|
||||||
@@ -49,7 +50,10 @@ Optional flags:
|
|||||||
| `--prompt-dir <dir>` | Prompt-definition directory override. |
|
| `--prompt-dir <dir>` | Prompt-definition directory override. |
|
||||||
| `--profile-dir <dir>` | Custom profile-directory override. |
|
| `--profile-dir <dir>` | Custom profile-directory override. |
|
||||||
| `--schema-dir <dir>` | Schema base-directory override. |
|
| `--schema-dir <dir>` | Schema base-directory override. |
|
||||||
|
| `--prompt-version <version>` | Optional prompt-definition version selector. |
|
||||||
| `--profile <id>` | Execution-profile override. |
|
| `--profile <id>` | Execution-profile override. |
|
||||||
|
| `--session-id <id>` | Optional direct session identifier. |
|
||||||
|
| `--input name=path` | Optional input file mapping; repeat or use comma-separated mappings. |
|
||||||
| `--var name=value` | Template-variable mapping; repeat or use comma-separated mappings. |
|
| `--var name=value` | Template-variable mapping; repeat or use comma-separated mappings. |
|
||||||
| `--out <path>` | Write generated content to this file instead of stdout. |
|
| `--out <path>` | Write generated content to this file instead of stdout. |
|
||||||
| `--llm-base-url <url>` | Runtime endpoint override. |
|
| `--llm-base-url <url>` | Runtime endpoint override. |
|
||||||
@@ -58,6 +62,7 @@ Optional flags:
|
|||||||
| `--temperature <float>` | Runtime temperature override. |
|
| `--temperature <float>` | Runtime temperature override. |
|
||||||
| `--max-tokens <int>` | Runtime maximum-token override. |
|
| `--max-tokens <int>` | Runtime maximum-token override. |
|
||||||
| `--top-p <float>` | Runtime top-p override. |
|
| `--top-p <float>` | Runtime top-p override. |
|
||||||
|
| `--reasoning-effort <value>` | Runtime reasoning-effort override. |
|
||||||
| `--timeout <duration>` | Runtime timeout override using Go duration syntax. |
|
| `--timeout <duration>` | Runtime timeout override using Go duration syntax. |
|
||||||
|
|
||||||
Deprecated aliases: `--prompt-id` for `--prompt`, and `--profile-id` for
|
Deprecated aliases: `--prompt-id` for `--prompt`, and `--profile-id` for
|
||||||
@@ -69,25 +74,44 @@ only; the caller context and configured transport cap remain active. CLI
|
|||||||
durations are converted to whole seconds by truncation toward zero, so any
|
durations are converted to whole seconds by truncation toward zero, so any
|
||||||
duration whose absolute value is below one second becomes an explicit
|
duration whose absolute value is below one second becomes an explicit
|
||||||
zero-second override. The timeout layers are defined in the
|
zero-second override. The timeout layers are defined in the
|
||||||
[Promptkit outbound integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/integrations/openai-compatible-chat.md#timeout-and-cancellation).
|
[Promptkit outbound integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/integrations/openai-compatible-chat.md#timeout-and-cancellation).
|
||||||
|
|
||||||
There is no raw API-key flag. Use `--api-key-env`.
|
There is no raw API-key flag. Use `--api-key-env`.
|
||||||
|
|
||||||
|
`--prompt-version` is passed directly to Promptkit. When it is omitted, the
|
||||||
|
selected prompt ID must have exactly one available version. `--input` is
|
||||||
|
optional at the CLI boundary: Promptkit decides whether the selected definition
|
||||||
|
requires declared inputs or template-referenced values.
|
||||||
|
|
||||||
|
`--session-id` supplies a direct, non-secret session identifier. A nonblank
|
||||||
|
value replaces a definition-rendered session ID; an omitted or blank value lets
|
||||||
|
the definition supply one. Promptkit trims direct values and limits them to 256
|
||||||
|
Unicode code points. Use stable, non-sensitive identifiers because effective
|
||||||
|
session IDs may appear in prepared output, run metadata, and provider-facing
|
||||||
|
requests.
|
||||||
|
|
||||||
|
`--reasoning-effort` is presence-aware: omitting it inherits the selected
|
||||||
|
profile value, a nonblank value replaces that value, and
|
||||||
|
`--reasoning-effort=` explicitly clears inherited reasoning. Promptkit treats
|
||||||
|
nonblank values as provider-specific opaque strings.
|
||||||
|
|
||||||
## `scriptorium render`
|
## `scriptorium render`
|
||||||
|
|
||||||
```text
|
```text
|
||||||
scriptorium render [flags]
|
scriptorium render [flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
`--prompt <id>` and at least one `--input name=path` are required. The
|
`--prompt <id>` is required. The following optional flags are supported:
|
||||||
following optional flags are supported: `--config`, `--prompt-dir`,
|
`--config`, `--prompt-dir`, `--profile-dir`, `--prompt-version`, `--profile`,
|
||||||
`--profile-dir`, `--profile`, `--var`, `--out`, `--llm-base-url`,
|
`--input`, `--var`, `--out`, `--llm-base-url`,
|
||||||
`--model`, `--api-key-env`, `--temperature`, `--max-tokens`, `--top-p`,
|
`--model`, `--api-key-env`, `--temperature`, `--max-tokens`, `--top-p`,
|
||||||
`--timeout`, and `--format text|json`. Their meanings match the corresponding
|
`--reasoning-effort`, `--session-id`, `--timeout`, and `--format text|json`. Their meanings match the corresponding
|
||||||
`run` flags; `--format` selects prepared-run output and otherwise uses
|
`run` flags; `--format` selects prepared-run output and otherwise uses
|
||||||
`defaults.render_format`.
|
`defaults.render_format`.
|
||||||
|
|
||||||
The same deprecated aliases and numeric/timeout behavior as `run` apply.
|
The same deprecated aliases and numeric/timeout behavior as `run` apply.
|
||||||
|
The same session and reasoning inheritance, replacement, and clearing behavior
|
||||||
|
also applies.
|
||||||
`render` does not accept `--schema-dir`; configure `schema_dir` through the
|
`render` does not accept `--schema-dir`; configure `schema_dir` through the
|
||||||
configuration file. It resolves profiles and schemas as part of preparation but
|
configuration file. It resolves profiles and schemas as part of preparation but
|
||||||
does not call an LLM.
|
does not call an LLM.
|
||||||
@@ -115,6 +139,31 @@ Optional flags:
|
|||||||
`serve` accepts no runtime model override flags. HTTP request fields, response
|
`serve` accepts no runtime model override flags. HTTP request fields, response
|
||||||
schemas, and error codes are defined in the [HTTP API reference](api.md).
|
schemas, and error codes are defined in the [HTTP API reference](api.md).
|
||||||
|
|
||||||
|
## `scriptorium inspect prompt`
|
||||||
|
|
||||||
|
```text
|
||||||
|
scriptorium inspect prompt --prompt ID [--prompt-version VERSION]
|
||||||
|
[--config PATH] [--prompt-dir DIR] [--format text|json] [--out PATH]
|
||||||
|
```
|
||||||
|
|
||||||
|
`--prompt` is required. Inspection uses normal configuration discovery and a
|
||||||
|
`--prompt-dir` override, defaults to text regardless of `defaults.render_format`,
|
||||||
|
and writes to stdout unless `--out` is supplied. It loads and normalizes the
|
||||||
|
selected definition but does not resolve a profile, load a schema, render a
|
||||||
|
template, reserve backend capacity, or call a model.
|
||||||
|
|
||||||
|
## `scriptorium inspect profile`
|
||||||
|
|
||||||
|
```text
|
||||||
|
scriptorium inspect profile --profile ID
|
||||||
|
[--config PATH] [--profile-dir DIR] [--format text|json] [--out PATH]
|
||||||
|
```
|
||||||
|
|
||||||
|
`--profile` is required; built-in profiles need no prompt directory. Inspection
|
||||||
|
resolves profile inheritance and backend defaults, but never reads a credential
|
||||||
|
value, loads a prompt, reserves capacity, or calls a model. Unset provider
|
||||||
|
controls are shown as zero values when Promptkit leaves them unspecified.
|
||||||
|
|
||||||
## Input And Variable Syntax
|
## Input And Variable Syntax
|
||||||
|
|
||||||
`--input name=path` maps an input name to a local file; `--var name=value`
|
`--input name=path` maps an input name to a local file; `--var name=value`
|
||||||
@@ -128,9 +177,12 @@ CLI inputs are file references. HTTP inline inputs are defined by the
|
|||||||
## Output And Exit Behavior
|
## Output And Exit Behavior
|
||||||
|
|
||||||
- `run` writes generated content to stdout, or to `--out` when supplied, and
|
- `run` writes generated content to stdout, or to `--out` when supplied, and
|
||||||
writes a concise summary to stderr.
|
writes a concise summary to stderr. The summary includes `backend=<id>` when
|
||||||
|
Promptkit selected a backend; endpoint-only profiles omit it.
|
||||||
- `render` writes prepared-run output to stdout, or to `--out` when supplied,
|
- `render` writes prepared-run output to stdout, or to `--out` when supplied,
|
||||||
without a success summary.
|
without a success summary. Text output includes `selected_backend_id` after
|
||||||
|
`selected_profile_id` when Promptkit selected one; endpoint-only profiles
|
||||||
|
omit it.
|
||||||
- `serve` writes startup and server errors to stderr.
|
- `serve` writes startup and server errors to stderr.
|
||||||
|
|
||||||
Exit statuses:
|
Exit statuses:
|
||||||
@@ -141,6 +193,10 @@ Exit statuses:
|
|||||||
| `1` | Parse, configuration, loading, rendering, generation, output-write, or other runtime error. |
|
| `1` | Parse, configuration, loading, rendering, generation, output-write, or other runtime error. |
|
||||||
| `2` | `run` generated and wrote output, but validation failed. |
|
| `2` | `run` generated and wrote output, but validation failed. |
|
||||||
|
|
||||||
|
A backend admission rejection is a runtime error and prints `run error: model
|
||||||
|
backend capacity is exhausted`. The HTTP capacity response is defined in the
|
||||||
|
[HTTP API reference](api.md).
|
||||||
|
|
||||||
## Workflows And Examples
|
## Workflows And Examples
|
||||||
|
|
||||||
The [maintained render script](../examples/render-markdown-summary.sh) is a
|
The [maintained render script](../examples/render-markdown-summary.sh) is a
|
||||||
|
|||||||
@@ -3,7 +3,7 @@
|
|||||||
This is the canonical reference for Scriptorium application settings. Prompt,
|
This is the canonical reference for Scriptorium application settings. Prompt,
|
||||||
profile, schema, execution-setting, built-in profile, and framework credential
|
profile, schema, execution-setting, built-in profile, and framework credential
|
||||||
semantics are defined by the
|
semantics are defined by the
|
||||||
[Promptkit v0.1.0 format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/formats.md).
|
[Promptkit v0.9.0 format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md).
|
||||||
For command syntax, see the [CLI reference](cli.md); for HTTP request shapes and
|
For command syntax, see the [CLI reference](cli.md); for HTTP request shapes and
|
||||||
outcomes, see the [HTTP API reference](api.md).
|
outcomes, see the [HTTP API reference](api.md).
|
||||||
|
|
||||||
@@ -39,11 +39,39 @@ do not override a prior value. Raw API-key fields are not accepted.
|
|||||||
| `server.max_artifact_bytes` | `16777216` | Maximum HTTP file-input artifact bytes; `0` disables the limit. |
|
| `server.max_artifact_bytes` | `16777216` | Maximum HTTP file-input artifact bytes; `0` disables the limit. |
|
||||||
| `server.max_response_bytes` | `16777216` | Maximum encoded HTTP response bytes; `0` disables the limit. |
|
| `server.max_response_bytes` | `16777216` | Maximum encoded HTTP response bytes; `0` disables the limit. |
|
||||||
| `defaults.render_format` | `text` | Default prepared-run output format: `text` or `json`. |
|
| `defaults.render_format` | `text` | Default prepared-run output format: `text` or `json`. |
|
||||||
|
| `backends` | unset | Optional mapping of custom Promptkit backend IDs to engine-scoped connection and capacity settings. |
|
||||||
|
|
||||||
The size fields must be zero or greater. The [HTTP API](api.md) defines how
|
The size fields must be zero or greater. The [HTTP API](api.md) defines how
|
||||||
each limit is enforced and reported. `server.artifact_root` configures an HTTP
|
each limit is enforced and reported. `server.artifact_root` configures an HTTP
|
||||||
deployment boundary; see [operations](operations.md) for deployment handling.
|
deployment boundary; see [operations](operations.md) for deployment handling.
|
||||||
|
|
||||||
|
## Custom Backends
|
||||||
|
|
||||||
|
Use `backends` when a profile selects an application-defined backend ID:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
backends:
|
||||||
|
local-gpu:
|
||||||
|
endpoint: http://localhost:11434/v1
|
||||||
|
api_key_env: LOCAL_GPU_API_KEY
|
||||||
|
extra_params:
|
||||||
|
provider_option: enabled
|
||||||
|
concurrency_limit: 2
|
||||||
|
queue_capacity: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
Each mapping key is the case-sensitive backend ID. `endpoint` is required;
|
||||||
|
`api_key_env`, `extra_params`, `concurrency_limit`, and `queue_capacity` are
|
||||||
|
optional. `concurrency_limit: 0` leaves the backend unlimited. Omitting
|
||||||
|
`queue_capacity` lets Promptkit use its default for a limited backend, while
|
||||||
|
an explicit `queue_capacity: 0` disables queueing.
|
||||||
|
|
||||||
|
Configuration strictly owns the YAML shape and rejects unknown fields. Promptkit
|
||||||
|
validates backend IDs, endpoints, environment-variable names, extra parameters,
|
||||||
|
and capacity relationships when Scriptorium constructs its engine. There are no
|
||||||
|
backend command-line overrides. Store only an environment-variable name in
|
||||||
|
`api_key_env`; raw API-key fields are not accepted.
|
||||||
|
|
||||||
## Framework Source Mapping
|
## Framework Source Mapping
|
||||||
|
|
||||||
Scriptorium passes `prompt_dir`, `profile_dir`, and `schema_dir` to Promptkit
|
Scriptorium passes `prompt_dir`, `profile_dir`, and `schema_dir` to Promptkit
|
||||||
@@ -58,7 +86,7 @@ parse those framework file formats.
|
|||||||
contracts.
|
contracts.
|
||||||
|
|
||||||
See the
|
See the
|
||||||
[tagged Promptkit format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/formats.md)
|
[tagged Promptkit format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md)
|
||||||
for all of those definitions. The files under
|
for all of those definitions. The files under
|
||||||
[`examples/prompts`](../examples/prompts/),
|
[`examples/prompts`](../examples/prompts/),
|
||||||
[`examples/profiles`](../examples/profiles/), and
|
[`examples/profiles`](../examples/profiles/), and
|
||||||
@@ -73,8 +101,20 @@ their names in configuration or framework source files. Do not place raw keys
|
|||||||
in configuration, prompts, profiles, CLI arguments, examples, or HTTP
|
in configuration, prompts, profiles, CLI arguments, examples, or HTTP
|
||||||
payloads.
|
payloads.
|
||||||
|
|
||||||
|
An optional `api_key_env` whose environment value is absent or empty can result
|
||||||
|
in an unauthenticated provider request. A profile that declares credentials
|
||||||
|
required still fails as an invalid request when no credential source is
|
||||||
|
selected, and fails with `ErrAPIKeyEnvMissing` when its selected environment
|
||||||
|
source is absent or empty. Scriptorium never reads or emits the environment
|
||||||
|
value itself.
|
||||||
|
|
||||||
|
Unset optional provider controls are omitted from compatible provider requests.
|
||||||
|
A positive Promptkit `repair_attempts` budget can add provider calls, latency,
|
||||||
|
token use, and cost; see the tagged format reference for its permitted values
|
||||||
|
and validation-mode requirements.
|
||||||
|
|
||||||
Promptkit's
|
Promptkit's
|
||||||
[OpenAI-compatible integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/integrations/openai-compatible-chat.md)
|
[OpenAI-compatible integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/integrations/openai-compatible-chat.md)
|
||||||
defines outbound authentication, provider request mapping, transport limits,
|
defines outbound authentication, provider request mapping, transport limits,
|
||||||
and timeout layering.
|
and timeout layering.
|
||||||
|
|
||||||
@@ -83,4 +123,4 @@ and timeout layering.
|
|||||||
- [CLI reference](cli.md)
|
- [CLI reference](cli.md)
|
||||||
- [HTTP API reference](api.md)
|
- [HTTP API reference](api.md)
|
||||||
- [Operations guide](operations.md)
|
- [Operations guide](operations.md)
|
||||||
- [Promptkit framework formats](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/formats.md)
|
- [Promptkit framework formats](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md)
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ and the HTTP service according to the boundary your application needs.
|
|||||||
|
|
||||||
Go applications that need an in-process prompt framework should import
|
Go applications that need an in-process prompt framework should import
|
||||||
Promptkit directly. The tagged
|
Promptkit directly. The tagged
|
||||||
[Promptkit Go consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/consumers/pkg-promptkit.md)
|
[Promptkit Go consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/consumers/pkg-promptkit.md)
|
||||||
owns that interface; Scriptorium does not provide a Go library package.
|
owns that interface; Scriptorium does not provide a Go library package.
|
||||||
Consumers arriving from the former Scriptorium Go API should follow the
|
Consumers arriving from the former Scriptorium Go API should follow the
|
||||||
[migration guide](migrating-to-promptkit.md).
|
[migration guide](migrating-to-promptkit.md).
|
||||||
@@ -35,4 +35,4 @@ Scriptorium does not persist run state. A retry can produce different output
|
|||||||
and can incur another provider request. CLI exits belong to the
|
and can incur another provider request. CLI exits belong to the
|
||||||
[CLI reference](../cli.md), HTTP status behavior belongs to the
|
[CLI reference](../cli.md), HTTP status behavior belongs to the
|
||||||
[HTTP API reference](../api.md), and framework semantics belong to
|
[HTTP API reference](../api.md), and framework semantics belong to
|
||||||
[Promptkit v0.1.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/formats.md).
|
[Promptkit v0.9.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md).
|
||||||
|
|||||||
@@ -4,7 +4,7 @@
|
|||||||
|
|
||||||
Scriptorium `v0.11.1` at
|
Scriptorium `v0.11.1` at
|
||||||
`gitea.maximumdirect.net/eric/scriptorium` is the final release that provides
|
`gitea.maximumdirect.net/eric/scriptorium` is the final release that provides
|
||||||
the former in-process Go framework. Promptkit `v0.1.0` at
|
the former in-process Go framework. Promptkit `v0.9.0` at
|
||||||
`gitea.maximumdirect.net/eric/promptkit` is the destination for that framework
|
`gitea.maximumdirect.net/eric/promptkit` is the destination for that framework
|
||||||
API. Scriptorium `v0.12.0` and later provide the CLI and HTTP application only.
|
API. Scriptorium `v0.12.0` and later provide the CLI and HTTP application only.
|
||||||
|
|
||||||
@@ -19,7 +19,7 @@ Start from a clean consumer checkout and review the pending diff before
|
|||||||
committing it. Add the published Promptkit module:
|
committing it. Add the published Promptkit module:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
go get gitea.maximumdirect.net/eric/promptkit@v0.1.0
|
go get gitea.maximumdirect.net/eric/promptkit@v0.9.0
|
||||||
```
|
```
|
||||||
|
|
||||||
For an ordinary consumer that imports the former root package under its
|
For an ordinary consumer that imports the former root package under its
|
||||||
@@ -51,14 +51,14 @@ go mod tidy
|
|||||||
go test ./...
|
go test ./...
|
||||||
```
|
```
|
||||||
|
|
||||||
Confirm that `go.mod` selects Promptkit `v0.1.0` and that no Go file imports
|
Confirm that `go.mod` selects Promptkit `v0.9.0` and that no Go file imports
|
||||||
the former Scriptorium package:
|
the former Scriptorium package:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
test "$(
|
test "$(
|
||||||
go list -m -f '{{.Path}}@{{.Version}}' \
|
go list -m -f '{{.Path}}@{{.Version}}' \
|
||||||
gitea.maximumdirect.net/eric/promptkit
|
gitea.maximumdirect.net/eric/promptkit
|
||||||
)" = 'gitea.maximumdirect.net/eric/promptkit@v0.1.0'
|
)" = 'gitea.maximumdirect.net/eric/promptkit@v0.9.0'
|
||||||
if git grep -n \
|
if git grep -n \
|
||||||
'gitea.maximumdirect.net/eric/scriptorium' \
|
'gitea.maximumdirect.net/eric/scriptorium' \
|
||||||
-- '*.go'
|
-- '*.go'
|
||||||
@@ -73,20 +73,20 @@ fi
|
|||||||
Promptkit preserves the established engine, request, result, profile,
|
Promptkit preserves the established engine, request, result, profile,
|
||||||
source-option, model-client, artifact, validation-value, and public-error
|
source-option, model-client, artifact, validation-value, and public-error
|
||||||
shapes where practical. Exact declarations and current behavior belong to the
|
shapes where practical. Exact declarations and current behavior belong to the
|
||||||
tagged [Promptkit consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/consumers/pkg-promptkit.md)
|
tagged [Promptkit consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/consumers/pkg-promptkit.md)
|
||||||
and Go source.
|
and Go source.
|
||||||
|
|
||||||
Promptkit also includes migration-relevant public contracts that were not in
|
Promptkit also includes migration-relevant public contracts that were not in
|
||||||
Scriptorium `v0.11.1`:
|
Scriptorium `v0.11.1`:
|
||||||
|
|
||||||
- [`WithArtifactReader`](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/engine.go#L96-L105)
|
- [`WithArtifactReader`](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/engine.go#L182)
|
||||||
and the
|
and the
|
||||||
[`ArtifactReader` declaration](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/types.go#L129-L135)
|
[`ArtifactReader` declaration](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/types.go#L304)
|
||||||
provide the artifact-reading extension described by the tagged
|
provide the artifact-reading extension described by the tagged
|
||||||
[extension-interface guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/consumers/pkg-promptkit.md#extension-interfaces).
|
[extension-interface guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/consumers/pkg-promptkit.md#extension-interfaces).
|
||||||
- [`ErrProfileRequired` and `ErrAPIKeyEnvMissing`](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/engine.go#L28-L40)
|
- [`ErrProfileRequired` and `ErrAPIKeyEnvMissing`](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/engine.go#L46-L61)
|
||||||
provide the specific identities described by the tagged
|
provide the specific identities described by the tagged
|
||||||
[error guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/consumers/pkg-promptkit.md#errors).
|
[error guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/consumers/pkg-promptkit.md#errors).
|
||||||
|
|
||||||
Use those tagged owners for exact signatures, wrapping guarantees, and
|
Use those tagged owners for exact signatures, wrapping guarantees, and
|
||||||
extension behavior.
|
extension behavior.
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
This is the contributor entry point for Scriptorium. Scriptorium is an
|
This is the contributor entry point for Scriptorium. Scriptorium is an
|
||||||
application that consumes the public
|
application that consumes the public
|
||||||
[Promptkit v0.1.0 package](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/consumers/pkg-promptkit.md);
|
[Promptkit v0.9.0 package](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/consumers/pkg-promptkit.md);
|
||||||
framework implementation work belongs in Promptkit.
|
framework implementation work belongs in Promptkit.
|
||||||
|
|
||||||
## Initial Orientation
|
## Initial Orientation
|
||||||
@@ -27,12 +27,12 @@ tests.
|
|||||||
| HTTP routes, DTOs, limits, status mapping, or artifact policy | [HTTP API contract](api.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) |
|
| HTTP routes, DTOs, limits, status mapping, or artifact policy | [HTTP API contract](api.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) |
|
||||||
| Application configuration or precedence | [Configuration contract](config.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) |
|
| Application configuration or precedence | [Configuration contract](config.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) |
|
||||||
| Prepared-run presentation | [CLI contract](cli.md), [adapter internals](internal/adapters.md), and `internal/format` |
|
| Prepared-run presentation | [CLI contract](cli.md), [adapter internals](internal/adapters.md), and `internal/format` |
|
||||||
| Prompt, profile, schema, generation, or validation semantics | [Promptkit framework formats](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/formats.md) and the [Promptkit consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/consumers/pkg-promptkit.md) |
|
| Prompt, profile, schema, generation, or validation semantics | [Promptkit framework formats](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md) and the [Promptkit consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/consumers/pkg-promptkit.md) |
|
||||||
| OpenAI-compatible outbound behavior or timeout layering | [Promptkit integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/integrations/openai-compatible-chat.md) |
|
| OpenAI-compatible outbound behavior or timeout layering | [Promptkit integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/integrations/openai-compatible-chat.md) |
|
||||||
| Subprocess behavior | [Subprocess integration](integrations/subprocess.md) and [CLI contract](cli.md) |
|
| Subprocess behavior | [Subprocess integration](integrations/subprocess.md) and [CLI contract](cli.md) |
|
||||||
| Runtime operation or recovery | [Operations](operations.md) |
|
| Runtime operation or recovery | [Operations](operations.md) |
|
||||||
| Release packaging or publication | The [release procedure](release.md), [hosted release workflow](../.woodpecker/release.yml), and [architecture policy](policy/architecture.md) |
|
| Release packaging or publication | The [release procedure](release.md), [hosted release workflow](../.woodpecker/release.yml), and [architecture policy](policy/architecture.md) |
|
||||||
| Examples or copyable assets | The owning Scriptorium contract, the relevant [Promptkit format contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/formats.md), and the related files under `examples/` |
|
| Examples or copyable assets | The owning Scriptorium contract, the relevant [Promptkit format contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md), and the related files under `examples/` |
|
||||||
| Architecture decisions or future work | The [documentation policy](policy/documentation.md), relevant accepted ADRs, and relevant roadmap documents |
|
| Architecture decisions or future work | The [documentation policy](policy/documentation.md), relevant accepted ADRs, and relevant roadmap documents |
|
||||||
|
|
||||||
Cross-project changes land and release in Promptkit before Scriptorium adopts
|
Cross-project changes land and release in Promptkit before Scriptorium adopts
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ own IO and presentation mechanics, not framework decisions.
|
|||||||
External contracts are canonical in the [CLI reference](../cli.md) and
|
External contracts are canonical in the [CLI reference](../cli.md) and
|
||||||
[HTTP API reference](../api.md). Promptkit's public engine contract is
|
[HTTP API reference](../api.md). Promptkit's public engine contract is
|
||||||
described by its tagged
|
described by its tagged
|
||||||
[Go consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/consumers/pkg-promptkit.md).
|
[Go consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/consumers/pkg-promptkit.md).
|
||||||
|
|
||||||
## Components And Collaborators
|
## Components And Collaborators
|
||||||
|
|
||||||
@@ -29,15 +29,30 @@ described by its tagged
|
|||||||
### CLI
|
### CLI
|
||||||
|
|
||||||
`run` calls `promptkit.Engine.Run`; `render` calls
|
`run` calls `promptkit.Engine.Run`; `render` calls
|
||||||
`promptkit.Engine.Prepare`. Both share request mapping for prompt/profile
|
`promptkit.Engine.Prepare`. Both share request mapping for prompt ID/version
|
||||||
selection, file inputs, variables, and presence-aware execution overrides.
|
and profile selection, optional file inputs, variables, and presence-aware
|
||||||
Omitted framework settings remain zero values so Promptkit resolves its own
|
execution overrides. Omitted framework settings remain zero values so Promptkit
|
||||||
defaults.
|
resolves its own defaults and definition-required inputs.
|
||||||
|
|
||||||
|
`inspect prompt` maps the selected ID and optional version to
|
||||||
|
`promptkit.Engine.InspectPrompt`, then formats Scriptorium-owned inspection
|
||||||
|
data. It constructs the same configuration-aware engine but does not perform
|
||||||
|
preparation or generation.
|
||||||
|
|
||||||
|
`inspect profile` maps an explicit profile ID to `Engine.InspectProfile` and
|
||||||
|
formats a safe application-owned effective profile view. It intentionally does
|
||||||
|
not require prompt, schema, or artifact sources.
|
||||||
|
|
||||||
`serve` constructs Scriptorium's restricted HTTP artifact reader, injects it
|
`serve` constructs Scriptorium's restricted HTTP artifact reader, injects it
|
||||||
with `promptkit.WithArtifactReader`, passes the engine through the HTTP
|
with `promptkit.WithArtifactReader`, passes the engine through the HTTP
|
||||||
adapter's consumer-owned `Runner` interface, and starts the server.
|
adapter's consumer-owned `Runner` interface, and starts the server.
|
||||||
|
|
||||||
|
All three CLI paths assemble the engine from the same resolved prompt, profile,
|
||||||
|
and schema directories plus configured custom backends. Each backend is mapped
|
||||||
|
to Promptkit's public `Backend` value and registered during engine construction,
|
||||||
|
so one constructed server engine retains one immutable backend registry and its
|
||||||
|
associated capacity state.
|
||||||
|
|
||||||
### HTTP
|
### HTTP
|
||||||
|
|
||||||
The handler enforces transport limits and strict JSON decoding before mapping
|
The handler enforces transport limits and strict JSON decoding before mapping
|
||||||
|
|||||||
@@ -11,8 +11,8 @@ public behavior belongs in the linked contracts.
|
|||||||
| `internal/adapter/http` | Owns routes, DTOs, strict decoding, limits, Promptkit request/result mapping, public error mapping, and restricted HTTP artifact reading. | [HTTP API](../api.md), [adapter internals](adapters.md), [source internals](sources.md) |
|
| `internal/adapter/http` | Owns routes, DTOs, strict decoding, limits, Promptkit request/result mapping, public error mapping, and restricted HTTP artifact reading. | [HTTP API](../api.md), [adapter internals](adapters.md), [source internals](sources.md) |
|
||||||
| `internal/config` | Discovers and strictly decodes application configuration and applies built-in and CLI precedence. | [configuration contract](../config.md), [adapter internals](adapters.md) |
|
| `internal/config` | Discovers and strictly decodes application configuration and applies built-in and CLI precedence. | [configuration contract](../config.md), [adapter internals](adapters.md) |
|
||||||
| `internal/defaults` | Holds Scriptorium-owned application and HTTP defaults. | [configuration contract](../config.md) |
|
| `internal/defaults` | Holds Scriptorium-owned application and HTTP defaults. | [configuration contract](../config.md) |
|
||||||
| `internal/format` | Formats Promptkit prepared-run values for CLI text or JSON output. | [CLI contract](../cli.md), [adapter internals](adapters.md) |
|
| `internal/format` | Formats Promptkit prepared-run and prompt/profile inspection values for CLI text or JSON output. | [CLI contract](../cli.md), [adapter internals](adapters.md) |
|
||||||
|
|
||||||
Framework implementation packages are provided by
|
Framework implementation packages are provided by
|
||||||
[Promptkit v0.1.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/consumers/pkg-promptkit.md)
|
[Promptkit v0.9.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/consumers/pkg-promptkit.md)
|
||||||
and are not part of this repository.
|
and are not part of this repository.
|
||||||
|
|||||||
@@ -5,14 +5,16 @@
|
|||||||
This document covers Scriptorium-owned source locations and the restricted HTTP
|
This document covers Scriptorium-owned source locations and the restricted HTTP
|
||||||
artifact reader. Prompt, profile, schema, and ordinary artifact semantics are
|
artifact reader. Prompt, profile, schema, and ordinary artifact semantics are
|
||||||
owned by the tagged
|
owned by the tagged
|
||||||
[Promptkit format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/formats.md).
|
[Promptkit format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md).
|
||||||
|
|
||||||
## Application Source Locations
|
## Application Source Locations
|
||||||
|
|
||||||
`internal/config` resolves `prompt_dir`, `profile_dir`, and `schema_dir` from
|
`internal/config` resolves `prompt_dir`, `profile_dir`, and `schema_dir` from
|
||||||
Scriptorium defaults, configuration files, and CLI overrides.
|
Scriptorium defaults, configuration files, and CLI overrides. It also resolves
|
||||||
`internal/adapter/cli` passes those paths into `promptkit.Config` when
|
the application-owned `backends` mapping into sorted engine settings.
|
||||||
constructing the engine.
|
`internal/adapter/cli` passes the directories into `promptkit.Config` and maps
|
||||||
|
each configured backend to Promptkit's public engine registration when
|
||||||
|
constructing an engine shared by the command path.
|
||||||
|
|
||||||
Scriptorium does not search, parse, validate, or overlay framework source files
|
Scriptorium does not search, parse, validate, or overlay framework source files
|
||||||
itself. Promptkit owns prompt selection, profile built-ins and overlays, schema
|
itself. Promptkit owns prompt selection, profile built-ins and overlays, schema
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ fields, or HTTP wire behavior.
|
|||||||
- [CLI reference](cli.md): commands, output destinations, and exit codes.
|
- [CLI reference](cli.md): commands, output destinations, and exit codes.
|
||||||
- [Configuration reference](config.md): application settings, source
|
- [Configuration reference](config.md): application settings, source
|
||||||
locations, defaults, and credential mapping.
|
locations, defaults, and credential mapping.
|
||||||
- [Promptkit framework formats](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/formats.md):
|
- [Promptkit framework formats](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md):
|
||||||
prompt, profile, schema, execution-setting, and framework credential
|
prompt, profile, schema, execution-setting, and framework credential
|
||||||
contracts.
|
contracts.
|
||||||
- [HTTP API reference](api.md): route, request/response schema, status codes,
|
- [HTTP API reference](api.md): route, request/response schema, status codes,
|
||||||
@@ -94,6 +94,14 @@ protect request bodies, HTTP file artifacts, and encoded responses; configure
|
|||||||
them through the [configuration reference](config.md) and rely on the
|
them through the [configuration reference](config.md) and rely on the
|
||||||
[HTTP API reference](api.md) for their response effects.
|
[HTTP API reference](api.md) for their response effects.
|
||||||
|
|
||||||
|
Configured backend concurrency and queue capacity are enforced per constructed
|
||||||
|
Promptkit engine. A `serve` process constructs one engine for its handler, so
|
||||||
|
concurrent HTTP requests share that transient admission state. Scriptorium does
|
||||||
|
not retain workflow state: capacity is neither durable nor a queue of resumable
|
||||||
|
runs. When admission is exhausted, HTTP returns `503 capacity_exceeded` without
|
||||||
|
retry timing; callers choose any retry policy that is safe for another model
|
||||||
|
call.
|
||||||
|
|
||||||
Before increasing a limit:
|
Before increasing a limit:
|
||||||
|
|
||||||
1. measure representative input, generated-output, and optional raw-output
|
1. measure representative input, generated-output, and optional raw-output
|
||||||
|
|||||||
@@ -5,16 +5,18 @@ durable development boundaries.
|
|||||||
|
|
||||||
## System Shape
|
## System Shape
|
||||||
|
|
||||||
Scriptorium is an executable application with three entry paths: CLI `run`, CLI
|
Scriptorium is an executable application with four command entry paths: CLI
|
||||||
`render`, and the HTTP service started by `serve`. It does not expose a reusable
|
`run`, CLI `render`, CLI `inspect`, and the HTTP service started by `serve`.
|
||||||
root Go package.
|
The `inspect` command has prompt and profile modes. Scriptorium does not expose
|
||||||
|
a reusable root Go package.
|
||||||
|
|
||||||
The application consumes
|
The application consumes
|
||||||
[Promptkit v0.1.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/consumers/pkg-promptkit.md)
|
[Promptkit v0.9.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/consumers/pkg-promptkit.md)
|
||||||
through its supported root package. Promptkit owns prompt execution,
|
through its supported root package. Promptkit owns prompt execution,
|
||||||
preparation, source formats, built-in profiles, model-client behavior, and
|
preparation, source formats, built-in profiles, model-client behavior, and
|
||||||
validation. Scriptorium owns application configuration, executable adapters,
|
validation. Scriptorium owns application configuration, executable adapters,
|
||||||
prepared-run presentation, process behavior, and HTTP deployment policy.
|
prepared-run and definition-inspection presentation, process behavior, and HTTP
|
||||||
|
deployment policy.
|
||||||
|
|
||||||
The concrete package inventory is maintained in the
|
The concrete package inventory is maintained in the
|
||||||
[internal overview](../internal/overview.md).
|
[internal overview](../internal/overview.md).
|
||||||
@@ -33,6 +35,7 @@ gitea.maximumdirect.net/eric/promptkit
|
|||||||
|
|
||||||
- Retained application packages may import Promptkit's root package.
|
- Retained application packages may import Promptkit's root package.
|
||||||
- They must not import Promptkit `internal` packages.
|
- They must not import Promptkit `internal` packages.
|
||||||
|
- They must not import Promptkit catalog modules directly.
|
||||||
- They must not import the removed Scriptorium root facade or recreate former
|
- They must not import the removed Scriptorium root facade or recreate former
|
||||||
framework package families.
|
framework package families.
|
||||||
- Adapter-owned interfaces use Promptkit public values when a consumer-side
|
- Adapter-owned interfaces use Promptkit public values when a consumer-side
|
||||||
@@ -52,11 +55,12 @@ invariants.
|
|||||||
- `internal/config` owns discovery and strict decoding of Scriptorium
|
- `internal/config` owns discovery and strict decoding of Scriptorium
|
||||||
application configuration.
|
application configuration.
|
||||||
- `internal/defaults` owns Scriptorium application and HTTP defaults only.
|
- `internal/defaults` owns Scriptorium application and HTTP defaults only.
|
||||||
- `internal/format` owns deterministic prepared-run text and JSON presentation.
|
- `internal/format` owns deterministic prepared-run and definition-inspection
|
||||||
|
text and JSON presentation.
|
||||||
- Promptkit owns framework orchestration and contracts. Its
|
- Promptkit owns framework orchestration and contracts. Its
|
||||||
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/formats.md)
|
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md)
|
||||||
and
|
and
|
||||||
[outbound integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/integrations/openai-compatible-chat.md)
|
[outbound integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/integrations/openai-compatible-chat.md)
|
||||||
are canonical.
|
are canonical.
|
||||||
|
|
||||||
## HTTP Artifact Security Boundary
|
## HTTP Artifact Security Boundary
|
||||||
|
|||||||
@@ -8,11 +8,11 @@ releases. The hosted
|
|||||||
executables, publishes their SHA-256 checksums, and uses the matching file
|
executables, publishes their SHA-256 checksums, and uses the matching file
|
||||||
under `docs/releases/` as the hosted release body.
|
under `docs/releases/` as the hosted release body.
|
||||||
|
|
||||||
`v0.12.0` is the selected version for the pending first application-only
|
`v0.12.0` is the first published application-only release. For each later
|
||||||
release. It remains an unreleased candidate until its annotated tag is
|
release, select a new `vMAJOR.MINOR.PATCH` version according to the intended
|
||||||
published, the hosted workflow succeeds, and every published artifact is
|
compatibility change. A selected version remains an unreleased candidate until
|
||||||
verified. Later releases select a new `vMAJOR.MINOR.PATCH` version according to
|
its annotated tag is published, the hosted workflow succeeds, and every
|
||||||
the intended compatibility change.
|
published artifact is verified.
|
||||||
|
|
||||||
Run this procedure from the Scriptorium repository root. A release must not
|
Run this procedure from the Scriptorium repository root. A release must not
|
||||||
depend on a Go workspace, module replacement, vendor tree, sibling checkout,
|
depend on a Go workspace, module replacement, vendor tree, sibling checkout,
|
||||||
@@ -20,14 +20,17 @@ unpublished dependency, or unpushed source commit.
|
|||||||
|
|
||||||
## Establish The Candidate
|
## Establish The Candidate
|
||||||
|
|
||||||
For the pending application-only release, start a POSIX shell and select:
|
Start a POSIX shell, choose a semantic version that has not been published, and
|
||||||
|
export it as `RELEASE_VERSION`. For example, if `v0.12.1` is the intended next
|
||||||
|
version and remains unpublished, select:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
export RELEASE_VERSION=v0.12.0
|
export RELEASE_VERSION=v0.12.1
|
||||||
```
|
```
|
||||||
|
|
||||||
For a later release, export its not-yet-published semantic version instead.
|
Use the version appropriate to the actual compatibility change rather than
|
||||||
Then run the following guard in that same shell:
|
assuming that the example is the next release. Then run the following guard in
|
||||||
|
that same shell:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
set -eu
|
set -eu
|
||||||
@@ -100,7 +103,7 @@ test "$(
|
|||||||
)" = 'gitea.maximumdirect.net/eric/scriptorium 1.25.5'
|
)" = 'gitea.maximumdirect.net/eric/scriptorium 1.25.5'
|
||||||
```
|
```
|
||||||
|
|
||||||
Require Promptkit `v0.1.0` as both the direct module-graph edge and the selected
|
Require Promptkit `v0.9.0` as both the direct module-graph edge and the selected
|
||||||
module version:
|
module version:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
@@ -114,11 +117,11 @@ direct_promptkit=$(
|
|||||||
'
|
'
|
||||||
)
|
)
|
||||||
test "$direct_promptkit" = \
|
test "$direct_promptkit" = \
|
||||||
'gitea.maximumdirect.net/eric/promptkit@v0.1.0'
|
'gitea.maximumdirect.net/eric/promptkit@v0.9.0'
|
||||||
test "$(
|
test "$(
|
||||||
GOWORK=off go list -m -f '{{.Path}}@{{.Version}}' \
|
GOWORK=off go list -m -f '{{.Path}}@{{.Version}}' \
|
||||||
gitea.maximumdirect.net/eric/promptkit
|
gitea.maximumdirect.net/eric/promptkit
|
||||||
)" = 'gitea.maximumdirect.net/eric/promptkit@v0.1.0'
|
)" = 'gitea.maximumdirect.net/eric/promptkit@v0.9.0'
|
||||||
GOWORK=off go list -m all
|
GOWORK=off go list -m all
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -169,11 +172,12 @@ unformatted=$(
|
|||||||
test -z "$unformatted"
|
test -z "$unformatted"
|
||||||
```
|
```
|
||||||
|
|
||||||
Run the maintained render script and smoke-test both maintained configuration
|
Run the maintained render and Promptkit v0.9 feature scripts, then smoke-test
|
||||||
examples without a model call:
|
both maintained configuration examples without a model call:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
GOWORK=off ./examples/render-markdown-summary.sh
|
GOWORK=off ./examples/render-markdown-summary.sh
|
||||||
|
GOWORK=off ./examples/render-v0.9-features.sh
|
||||||
for config_file in examples/config.yml examples/config.full.yml
|
for config_file in examples/config.yml examples/config.full.yml
|
||||||
do
|
do
|
||||||
GOWORK=off go run ./cmd/scriptorium render \
|
GOWORK=off go run ./cmd/scriptorium render \
|
||||||
|
|||||||
59
docs/releases/v0.13.0.md
Normal file
59
docs/releases/v0.13.0.md
Normal file
@@ -0,0 +1,59 @@
|
|||||||
|
# Scriptorium v0.13.0
|
||||||
|
|
||||||
|
## Promptkit v0.9 Adoption
|
||||||
|
|
||||||
|
Scriptorium now uses
|
||||||
|
[Promptkit `v0.9.0`](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/)
|
||||||
|
and supports the complete Promptkit v0.9 prompt and profile definition format
|
||||||
|
through its configured directory sources. This includes prompt versions,
|
||||||
|
optional inputs, all supported message roles, cache control, session templates,
|
||||||
|
output schemas and repair budgets, profile inheritance, backend selection,
|
||||||
|
provider controls, optional credential environment names, and JSON-compatible
|
||||||
|
extra parameters.
|
||||||
|
|
||||||
|
Promptkit remains the canonical owner of definition parsing and framework
|
||||||
|
validation. Scriptorium continues to own its CLI, HTTP, configuration,
|
||||||
|
presentation, and deployment contracts.
|
||||||
|
|
||||||
|
## Application Features
|
||||||
|
|
||||||
|
- `run` and `render` can select an explicit prompt version and can execute
|
||||||
|
prompts that do not declare or reference inputs.
|
||||||
|
- CLI and HTTP requests can provide a direct session ID. Reasoning controls can
|
||||||
|
inherit, replace, or explicitly clear the selected profile value.
|
||||||
|
- Application configuration can register custom OpenAI-compatible backends
|
||||||
|
with optional credential environment names, extra parameters, concurrency
|
||||||
|
limits, and queue policy.
|
||||||
|
- `inspect prompt` and `inspect profile` provide deterministic text or JSON
|
||||||
|
views without invoking a model. Profile inspection does not read credential
|
||||||
|
values or require a prompt definition.
|
||||||
|
- Prepared output, CLI summaries, and HTTP metadata report effective backend
|
||||||
|
identity when available.
|
||||||
|
- HTTP requests rejected by backend capacity policy return `503` with the
|
||||||
|
stable `capacity_exceeded` code. One server-scoped Promptkit engine enforces
|
||||||
|
those limits across concurrent requests.
|
||||||
|
|
||||||
|
See the versioned application contracts for exact behavior:
|
||||||
|
|
||||||
|
- [CLI reference](https://gitea.maximumdirect.net/eric/scriptorium/src/tag/v0.13.0/docs/cli.md)
|
||||||
|
- [HTTP API reference](https://gitea.maximumdirect.net/eric/scriptorium/src/tag/v0.13.0/docs/api.md)
|
||||||
|
- [Configuration reference](https://gitea.maximumdirect.net/eric/scriptorium/src/tag/v0.13.0/docs/config.md)
|
||||||
|
- [Operations guide](https://gitea.maximumdirect.net/eric/scriptorium/src/tag/v0.13.0/docs/operations.md)
|
||||||
|
- [Promptkit v0.9 format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md)
|
||||||
|
|
||||||
|
## Compatibility And Operations
|
||||||
|
|
||||||
|
There are no intentional removals from the v0.12 application interfaces. The
|
||||||
|
Promptkit upgrade does enforce its current definition-safety rules, including
|
||||||
|
supported message roles, absolute HTTP or HTTPS endpoints, contained content
|
||||||
|
and schema paths, bounded repair budgets, and JSON-compatible extra parameters.
|
||||||
|
Definitions rejected by these rules must be corrected before use.
|
||||||
|
|
||||||
|
Credential values remain outside configuration, definition files, CLI flags,
|
||||||
|
and HTTP payloads. Named environment sources are optional unless a selected
|
||||||
|
Promptkit profile explicitly requires a key. A positive repair budget permits
|
||||||
|
additional provider calls and can increase latency, token use, and cost.
|
||||||
|
|
||||||
|
The HTTP service retains only the shared operational state needed for backend
|
||||||
|
admission and in-flight request handling. Scriptorium does not retain durable
|
||||||
|
conversation, workflow, checkpoint, or resume state.
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -1,341 +0,0 @@
|
|||||||
# Promptkit Migration Roadmap
|
|
||||||
|
|
||||||
## Status
|
|
||||||
|
|
||||||
Accepted plan. Steps 1 through 8 are complete. Step 9 remains proposed and is
|
|
||||||
not yet implemented.
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
|
|
||||||
Split the current repository into two projects:
|
|
||||||
|
|
||||||
- **Promptkit**: the reusable Go framework, public Go facade, execution engine,
|
|
||||||
source and validation support, OpenAI-compatible client, extension
|
|
||||||
interfaces, and built-in execution-profile registry.
|
|
||||||
- **Scriptorium**: a slim runnable application that imports Promptkit and
|
|
||||||
provides the CLI and HTTP interfaces.
|
|
||||||
|
|
||||||
Scriptorium will become another downstream Promptkit consumer rather than the
|
|
||||||
owner of the framework.
|
|
||||||
|
|
||||||
## Compatibility And Migration Policy
|
|
||||||
|
|
||||||
This is an intentionally breaking change.
|
|
||||||
|
|
||||||
- New and migrated Go consumers must import Promptkit instead of Scriptorium.
|
|
||||||
- Scriptorium will not retain type aliases, forwarding packages, deprecated
|
|
||||||
facade APIs, or other source-compatibility shims.
|
|
||||||
- Existing consumers may continue using a previously tagged Scriptorium module
|
|
||||||
version until they are migrated.
|
|
||||||
- The migration does not need to preserve compatibility between intermediate
|
|
||||||
development states. Each completed phase must instead leave the affected
|
|
||||||
repository internally consistent and tested.
|
|
||||||
- Promptkit should initially preserve the useful shape and behavior of the
|
|
||||||
current public Go facade where doing so reduces extraction risk. Broader API
|
|
||||||
redesign should follow the split unless required to establish the new
|
|
||||||
boundary.
|
|
||||||
|
|
||||||
## Target Ownership
|
|
||||||
|
|
||||||
Promptkit should own application-neutral framework behavior:
|
|
||||||
|
|
||||||
- public engine, request, result, option, extension, and error APIs;
|
|
||||||
- prompt-definition loading and rendering;
|
|
||||||
- execution profiles, overlays, and the built-in profile registry;
|
|
||||||
- artifact-loading interfaces and general-purpose `file` and `inline` support;
|
|
||||||
- schema loading and output validation;
|
|
||||||
- LLM client boundaries and the OpenAI-compatible implementation;
|
|
||||||
- preparation and execution orchestration;
|
|
||||||
- framework and execution defaults.
|
|
||||||
|
|
||||||
Scriptorium should own executable and transport concerns:
|
|
||||||
|
|
||||||
- the `scriptorium` command and its `run`, `render`, and `serve` interfaces;
|
|
||||||
- CLI parsing, output formatting, exit codes, and process behavior;
|
|
||||||
- application-config discovery and CLI precedence;
|
|
||||||
- HTTP routing, request and response DTOs, limits, and error/status mapping;
|
|
||||||
- HTTP artifact-root and deployment security policy;
|
|
||||||
- server and adapter defaults;
|
|
||||||
- executable examples, operations guidance, and transport documentation.
|
|
||||||
|
|
||||||
The intended dependency direction is:
|
|
||||||
|
|
||||||
```text
|
|
||||||
Scriptorium CLI and HTTP adapters
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Promptkit
|
|
||||||
|
|
|
||||||
v
|
|
||||||
consumer-supplied sources and clients
|
|
||||||
```
|
|
||||||
|
|
||||||
Scriptorium must use Promptkit's public API. It must not depend on Promptkit
|
|
||||||
implementation packages or reproduce framework orchestration.
|
|
||||||
|
|
||||||
## Migration Steps
|
|
||||||
|
|
||||||
### Step 1: Refresh And Synchronize Documentation
|
|
||||||
|
|
||||||
Perform a repository-wide documentation refresh before migration development.
|
|
||||||
|
|
||||||
At minimum:
|
|
||||||
|
|
||||||
- reconcile all current-behavior documentation with the code, tests, examples,
|
|
||||||
defaults, and current public contracts;
|
|
||||||
- introduce the planned documentation-policy updates;
|
|
||||||
- establish an architecture decision record policy and canonical ADR location;
|
|
||||||
- resolve stale, duplicated, or misplaced material;
|
|
||||||
- validate documentation links and maintained examples;
|
|
||||||
- leave future migration behavior in `docs/roadmap/` until implemented.
|
|
||||||
|
|
||||||
**Gate:** Do not begin architectural migration work until the documentation
|
|
||||||
refresh and policy updates are merged and the repository has an agreed,
|
|
||||||
accurate baseline.
|
|
||||||
|
|
||||||
**Gate status:** Complete as of 2026-07-26. The completed documentation
|
|
||||||
refresh, follow-up verification, and layered-timeout correction remain recorded
|
|
||||||
in repository history. Step 1 remains complete after that validation.
|
|
||||||
|
|
||||||
### Step 2: Record The Architectural Decision And Detailed Boundary
|
|
||||||
|
|
||||||
Create an ADR, under the policy established in Step 1, that records:
|
|
||||||
|
|
||||||
- the decision to split Promptkit from Scriptorium;
|
|
||||||
- the target ownership and dependency direction;
|
|
||||||
- the selected Promptkit repository and Go module paths;
|
|
||||||
- the breaking-change and versioning policy;
|
|
||||||
- ownership of configuration fields and defaults;
|
|
||||||
- artifact-reader and HTTP containment responsibilities;
|
|
||||||
- local multi-repository development and release coordination;
|
|
||||||
- documentation ownership after the split.
|
|
||||||
|
|
||||||
Use the ADR to resolve any remaining public-boundary decisions before code is
|
|
||||||
moved.
|
|
||||||
|
|
||||||
**Gate:** The ADR is accepted, and every existing package, public contract,
|
|
||||||
configuration category, and maintained asset has a target owner.
|
|
||||||
|
|
||||||
**Gate status:** Complete as of 2026-07-26.
|
|
||||||
[ADR 0002: Split Promptkit From Scriptorium](../adr/0002-split-promptkit-from-scriptorium.md)
|
|
||||||
is accepted and records the required ownership and coordination decisions.
|
|
||||||
|
|
||||||
### Step 3: Characterize Existing Framework Behavior
|
|
||||||
|
|
||||||
Strengthen or add contract-focused tests where needed so extraction can be
|
|
||||||
verified without relying on package placement.
|
|
||||||
|
|
||||||
The completed Step 3 gate records the accepted implementation scope and
|
|
||||||
intended completion state.
|
|
||||||
|
|
||||||
Preserve coverage of:
|
|
||||||
|
|
||||||
- `Prepare` and `Run` behavior;
|
|
||||||
- prompt, profile, execution-default, and request-override precedence;
|
|
||||||
- presence-aware numeric overrides;
|
|
||||||
- built-in profile fallback and custom-profile overlays;
|
|
||||||
- strict YAML and JSON decoding;
|
|
||||||
- prompt, profile, schema, and artifact source behavior;
|
|
||||||
- structured-output requests and output validation;
|
|
||||||
- validation failures versus validation runtime errors;
|
|
||||||
- secret handling and redaction;
|
|
||||||
- public error classification;
|
|
||||||
- HTTP artifact restrictions and transport mappings.
|
|
||||||
|
|
||||||
**Gate:** Current framework and adapter contracts are represented by passing
|
|
||||||
tests sufficient to detect behavioral regressions during the split.
|
|
||||||
|
|
||||||
**Gate status:** Complete as of 2026-07-27. The framework contract corpus,
|
|
||||||
public `Engine` characterization, ownership audit, full test and vet suites,
|
|
||||||
temporary executable build, and maintained offline examples passed.
|
|
||||||
|
|
||||||
### Step 4: Make Scriptorium Adapters Consume The Public Facade
|
|
||||||
|
|
||||||
Within the current repository, refactor the CLI and HTTP adapters to use the
|
|
||||||
public framework facade rather than constructing or importing internal runner
|
|
||||||
components directly.
|
|
||||||
|
|
||||||
Add only the minimum public capabilities needed to support this boundary. These
|
|
||||||
may include:
|
|
||||||
|
|
||||||
- a small `Run`/`Prepare` consumer interface;
|
|
||||||
- injectable artifact-reading behavior for Scriptorium's restricted HTTP
|
|
||||||
policy;
|
|
||||||
- source options currently available only through internal constructors;
|
|
||||||
- prepared-run formatting based on public types;
|
|
||||||
- stable public error classification required by CLI and HTTP mappings.
|
|
||||||
|
|
||||||
Do not broadly export internal repositories, domain types, or use-case
|
|
||||||
implementations.
|
|
||||||
|
|
||||||
**Gate:** The CLI and HTTP adapters use only the public framework API for
|
|
||||||
framework behavior, and all tests and documented smoke commands pass.
|
|
||||||
|
|
||||||
**Gate status:** Complete as of 2026-07-28. CLI `run`, `render`, and `serve`,
|
|
||||||
the HTTP handler, and prepared-run formatting use the public facade; the
|
|
||||||
restricted HTTP reader is injected through the public extension point. The
|
|
||||||
post-implementation public-error, deterministic MIME, and recursive
|
|
||||||
dependency-guard corrections passed full tests, vet, build, race checks,
|
|
||||||
maintained examples, and configuration smoke checks.
|
|
||||||
|
|
||||||
### Step 5: Create The Promptkit Repository
|
|
||||||
|
|
||||||
Promptkit was established as an independent repository and Go module through
|
|
||||||
the completed out-of-band workflow recorded in repository history. Its
|
|
||||||
foundation includes:
|
|
||||||
|
|
||||||
- confirmed repository access, governance, origin, and default-branch tracking;
|
|
||||||
- module `gitea.maximumdirect.net/eric/promptkit` at Go `1.25.5`;
|
|
||||||
- a minimal root `promptkit` public package boundary with no placeholder
|
|
||||||
framework packages;
|
|
||||||
- library-specific development, architecture, documentation, testing, and
|
|
||||||
release policies;
|
|
||||||
- documented maintainer-run test, vet, build, formatting, documentation-link,
|
|
||||||
and repository-hygiene validation;
|
|
||||||
- source-commit and semantic Go module tag releases without runnable binaries
|
|
||||||
or binary packaging; and
|
|
||||||
- temporary workspace and uncommitted replacement workflows for coordinated
|
|
||||||
development without committed repository coupling.
|
|
||||||
|
|
||||||
[ADR 0003](../adr/0003-use-maintainer-run-validation-and-tag-only-releases-for-promptkit.md)
|
|
||||||
records the controlling Promptkit validation and release decision.
|
|
||||||
|
|
||||||
**Gate status:** Complete as of 2026-07-28. Promptkit passed its documented
|
|
||||||
validation independently, all maintained links and repository-hygiene checks
|
|
||||||
passed, and no workspace, replacement, CI configuration, binary, tag, command,
|
|
||||||
or placeholder package was added. The completed repository foundation
|
|
||||||
supported the Step 6 extraction.
|
|
||||||
|
|
||||||
### Step 6: Extract And Stabilize Promptkit
|
|
||||||
|
|
||||||
Move the application-neutral framework and built-in profile assets into
|
|
||||||
Promptkit. Preserve implementation packages as internal where practical.
|
|
||||||
|
|
||||||
The initial public API should remain focused on the established engine workflow
|
|
||||||
and the source and client extension points required by real consumers. Avoid
|
|
||||||
combining the extraction with unrelated API redesign.
|
|
||||||
|
|
||||||
Move or recreate the relevant:
|
|
||||||
|
|
||||||
- framework implementation;
|
|
||||||
- public package tests and framework contract tests;
|
|
||||||
- built-in profile assets and registry tests;
|
|
||||||
- Go consumer examples;
|
|
||||||
- framework, consumer, configuration-format, and integration documentation.
|
|
||||||
|
|
||||||
Verify that Promptkit can be built, tested, and consumed independently of the
|
|
||||||
Scriptorium repository.
|
|
||||||
|
|
||||||
**Gate:** Promptkit independently provides the agreed framework contract,
|
|
||||||
passes its documented validation, and has published its first versioned tag
|
|
||||||
before Scriptorium or another consumer adopts it, as required by
|
|
||||||
[ADR 0003](../adr/0003-use-maintainer-run-validation-and-tag-only-releases-for-promptkit.md).
|
|
||||||
|
|
||||||
**Gate status:** Complete as of 2026-07-28. Repository history records source
|
|
||||||
Scriptorium commit `c7263ab2a8e58f7fb97280082d327a820c7cece7`,
|
|
||||||
accepted Promptkit commit
|
|
||||||
`9e68a2bbf779545995270c47842048a3bc6c85dc`, independently passing acceptance,
|
|
||||||
published annotated tag `v0.1.0`, and successful remote-consumer validation.
|
|
||||||
Scriptorium remains unchanged at its pre-cutover boundary. Step 7 adoption of
|
|
||||||
the tagged module and removal of the duplicated framework is the next gate.
|
|
||||||
|
|
||||||
### Step 7: Slim Scriptorium And Adopt Promptkit
|
|
||||||
|
|
||||||
Update Scriptorium to import the tagged Promptkit module and remove the
|
|
||||||
framework implementation and public Go facade that Promptkit replaces.
|
|
||||||
|
|
||||||
Retain only Scriptorium-owned executable and transport behavior. In particular:
|
|
||||||
|
|
||||||
- wire CLI and HTTP requests through Promptkit's public API;
|
|
||||||
- keep application config and transport defaults in Scriptorium;
|
|
||||||
- keep restricted HTTP artifact policy in Scriptorium while injecting it
|
|
||||||
through Promptkit's supported boundary;
|
|
||||||
- remove obsolete framework packages, tests, and documentation;
|
|
||||||
- update Scriptorium examples and docs to describe the CLI and HTTP application;
|
|
||||||
- direct Go framework consumers to Promptkit without providing compatibility
|
|
||||||
aliases or forwarding APIs.
|
|
||||||
|
|
||||||
**Gate:** Scriptorium builds and passes all tests using a tagged Promptkit
|
|
||||||
dependency, contains no duplicate framework implementation, and its current
|
|
||||||
documentation describes only the slimmed application.
|
|
||||||
|
|
||||||
**Gate status:** Complete as of 2026-07-28. Scriptorium directly resolves
|
|
||||||
Promptkit `v0.1.0`, no longer contains the framework copy or public Go facade,
|
|
||||||
and retains only its application, adapter, configuration, presentation,
|
|
||||||
transport, packaging, and executable-example responsibilities. Release-grade
|
|
||||||
Scriptorium validation passed with a fresh remote dependency cache, and the
|
|
||||||
published Promptkit tag passed its documented validation independently. The
|
|
||||||
application is ready for the downstream-consumer migrations in Step 8.
|
|
||||||
|
|
||||||
### Step 8: Migrate Downstream Consumers To Promptkit
|
|
||||||
|
|
||||||
Inventory downstream Go consumers and migrate each from the Scriptorium package
|
|
||||||
to Promptkit. This work may occur in external repositories and must be tracked
|
|
||||||
explicitly.
|
|
||||||
|
|
||||||
For each consumer:
|
|
||||||
|
|
||||||
- update module imports and dependencies;
|
|
||||||
- adapt to any intentionally changed public API;
|
|
||||||
- run its tests and relevant integration or smoke checks;
|
|
||||||
- confirm configuration, source, validation, and error behavior;
|
|
||||||
- release or deploy the migrated consumer through its normal process.
|
|
||||||
|
|
||||||
Consumers that cannot migrate immediately may remain pinned to the last
|
|
||||||
framework-bearing Scriptorium tag. No compatibility work is required in the new
|
|
||||||
Scriptorium project for those consumers.
|
|
||||||
|
|
||||||
**Gate:** All in-scope downstream consumers are either migrated and verified or
|
|
||||||
explicitly recorded as remaining on the previous Scriptorium version with an
|
|
||||||
owner and follow-up plan. Do not declare the ecosystem migration complete until
|
|
||||||
the required out-of-band consumer changes are confirmed.
|
|
||||||
|
|
||||||
**Gate status:** Complete as of 2026-07-28. The maintainer confirmed that
|
|
||||||
Notarius was the only downstream consumer of Scriptorium's former Go package.
|
|
||||||
Its clean, synchronized main branch now directly requires Promptkit `v0.1.0`,
|
|
||||||
all relevant Go imports use Promptkit rather than Scriptorium, and its full Go
|
|
||||||
test suite passes. No downstream consumer remains to migrate or disposition.
|
|
||||||
|
|
||||||
### Step 9: Complete Release And Documentation Cutover
|
|
||||||
|
|
||||||
Complete the coordinated project transition:
|
|
||||||
|
|
||||||
- publish Promptkit before dependent Scriptorium releases;
|
|
||||||
- release the breaking Scriptorium version against the tagged Promptkit
|
|
||||||
dependency;
|
|
||||||
- publish migration guidance that maps the former Scriptorium Go API to
|
|
||||||
Promptkit;
|
|
||||||
- update cross-project links, examples, package documentation, and release
|
|
||||||
notes;
|
|
||||||
- verify that no release artifact depends on local workspaces or replacements;
|
|
||||||
- archive completed roadmap material according to the documentation policy in
|
|
||||||
effect at that time.
|
|
||||||
|
|
||||||
**Gate:** Promptkit and Scriptorium are independently releasable, their
|
|
||||||
documentation has distinct and accurate ownership, and the migration status of
|
|
||||||
all identified downstream consumers is recorded.
|
|
||||||
|
|
||||||
## Cross-Cutting Constraints
|
|
||||||
|
|
||||||
- Preserve the invariant that execution orchestration remains narrow and
|
|
||||||
application-neutral.
|
|
||||||
- Keep adapter-specific decisions out of Promptkit.
|
|
||||||
- Keep Scriptorium dependent only on Promptkit's supported public API.
|
|
||||||
- Preserve strict external decoding, error classification, validation
|
|
||||||
semantics, and secret redaction throughout the migration.
|
|
||||||
- Keep each repository buildable and testable at merged phase boundaries.
|
|
||||||
- Coordinate cross-repository changes through tagged dependencies and explicit
|
|
||||||
gates rather than assuming atomic commits across repositories.
|
|
||||||
- Document only implemented behavior outside roadmap files.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
|
|
||||||
The migration is complete when:
|
|
||||||
|
|
||||||
- Promptkit is the independent owner of the reusable framework and built-in
|
|
||||||
profiles;
|
|
||||||
- Scriptorium is a slim CLI and HTTP consumer of Promptkit;
|
|
||||||
- Scriptorium no longer exposes or maintains the former public Go framework;
|
|
||||||
- all required downstream migrations and external repository work have been
|
|
||||||
completed or explicitly dispositioned;
|
|
||||||
- both repositories build, test, document, version, and release independently.
|
|
||||||
351
docs/roadmap/promptkit-v0.9.0-adoption.md
Normal file
351
docs/roadmap/promptkit-v0.9.0-adoption.md
Normal file
@@ -0,0 +1,351 @@
|
|||||||
|
# Promptkit v0.9.0 Adoption Roadmap
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Complete. This document records the delivered feature scope and targeted end
|
||||||
|
state. The completed delivery plan is maintained in
|
||||||
|
[implementation.md](implementation.md); current behavior remains defined by
|
||||||
|
the canonical application contracts linked below.
|
||||||
|
|
||||||
|
The durable product and integration boundary for this work is established by
|
||||||
|
[ADR 0004](../adr/0004-definition-boundary.md). This roadmap applies that
|
||||||
|
decision to Promptkit v0.9.0 and owns implementation scope and completion
|
||||||
|
status.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Bring Scriptorium from Promptkit v0.1.0 to v0.9.0 and expose the newer
|
||||||
|
framework capabilities that fit Scriptorium's existing CLI, HTTP,
|
||||||
|
configuration, presentation, and process boundaries.
|
||||||
|
|
||||||
|
Scriptorium must accept the complete Promptkit v0.9.0 feature set expressed by
|
||||||
|
directory-backed prompt, profile, and schema definitions. Application adapters
|
||||||
|
must not narrow that format contract by requiring request values that the
|
||||||
|
selected definition does not require or by omitting selection values needed to
|
||||||
|
address a valid definition. Promptkit's tagged
|
||||||
|
[framework format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md)
|
||||||
|
remains canonical; Scriptorium must delegate parsing and framework validation
|
||||||
|
rather than reproduce those rules.
|
||||||
|
|
||||||
|
The completed application will retain Promptkit as the owner of reusable
|
||||||
|
prompt execution, source formats, profiles, backend definitions, capacity,
|
||||||
|
inspection, generation, and validation. Scriptorium will continue to own
|
||||||
|
application configuration, command and HTTP contracts, error and exit mapping,
|
||||||
|
output presentation, and deployment policy.
|
||||||
|
|
||||||
|
## Targeted End State
|
||||||
|
|
||||||
|
When this roadmap is complete:
|
||||||
|
|
||||||
|
- Scriptorium pins the released Promptkit v0.9.0 module without a workspace,
|
||||||
|
replacement, vendored source, direct catalog import, or Promptkit internal
|
||||||
|
import.
|
||||||
|
- Existing valid CLI, HTTP, configuration, prompt, profile, schema, and render
|
||||||
|
workflows remain supported subject to Promptkit's documented v0.2.0 through
|
||||||
|
v0.9.0 compatibility corrections.
|
||||||
|
- Every Promptkit v0.9.0 prompt and profile feature expressible in definitions
|
||||||
|
under Scriptorium's configured directory sources is usable through the
|
||||||
|
executable application, including prompts with no inputs and explicitly
|
||||||
|
selected prompt versions.
|
||||||
|
- CLI and HTTP requests can supply a direct non-secret session ID.
|
||||||
|
- HTTP requests and a presence-aware CLI flag can inherit, replace, or
|
||||||
|
explicitly clear reasoning effort.
|
||||||
|
- Prepared output, CLI run summaries, and HTTP result metadata expose effective
|
||||||
|
backend identity when Promptkit supplies one.
|
||||||
|
- HTTP capacity rejection has a stable Scriptorium-owned overload response.
|
||||||
|
- Application configuration can register reusable engine-scoped custom
|
||||||
|
backends, including bounded concurrency and queue policy, for selection by
|
||||||
|
Promptkit profiles.
|
||||||
|
- CLI users can inspect prompt and profile definitions without supplying
|
||||||
|
placeholder inputs or invoking a model.
|
||||||
|
- Scriptorium's contracts and examples point to Promptkit v0.9.0 and accurately
|
||||||
|
describe its current credential, validation-repair, provider-control,
|
||||||
|
message-role, backend, and source-validation semantics.
|
||||||
|
|
||||||
|
## Feature Scope
|
||||||
|
|
||||||
|
### Complete Prompt And Profile Definition Compatibility
|
||||||
|
|
||||||
|
Scriptorium will treat the tagged Promptkit v0.9.0 format contract as one
|
||||||
|
indivisible downstream compatibility boundary. This includes prompt identity
|
||||||
|
and version selection, optional and required input declarations, message and
|
||||||
|
content-file templates, supported roles, cache control, session templates,
|
||||||
|
default profiles, output contracts, schemas, and repair budgets. It also
|
||||||
|
includes standalone and inherited profiles, built-in and custom backend
|
||||||
|
selection, endpoint overrides, execution controls, optional credential
|
||||||
|
environment sources, and JSON-compatible extra parameters.
|
||||||
|
|
||||||
|
Most of this support is provided by passing configured prompt, profile, and
|
||||||
|
schema directories directly to Promptkit. Scriptorium will not independently
|
||||||
|
decode these framework definitions or maintain a field-level subset. Two
|
||||||
|
current adapter restrictions must be removed so the executable interfaces do
|
||||||
|
not reject definitions that Promptkit accepts:
|
||||||
|
|
||||||
|
- `run` and `render` will accept an optional prompt-version flag and map it to
|
||||||
|
`promptkit.RunRequest.PromptVersion`; and
|
||||||
|
- `run`, `render`, and `POST /v1/runs` will permit omitted or empty input maps.
|
||||||
|
Promptkit will decide whether an input is required by the selected
|
||||||
|
definition or referenced template.
|
||||||
|
|
||||||
|
The HTTP API already carries `prompt_version`; its mapping must remain covered
|
||||||
|
by adapter tests. CLI and HTTP callers may continue to supply extra inputs,
|
||||||
|
subject to their existing file, containment, and size policies. Removing the
|
||||||
|
application-level nonempty-input requirement does not weaken Promptkit's
|
||||||
|
declared-required-input or template-reference validation.
|
||||||
|
|
||||||
|
Profiles that name built-in backend IDs require no application registration.
|
||||||
|
Profiles that name other backend IDs become fully usable through the custom
|
||||||
|
backend configuration in this roadmap. Profile inheritance and all other
|
||||||
|
profile-field resolution remain Promptkit responsibilities.
|
||||||
|
|
||||||
|
This compatibility requirement concerns fields and behavior expressed in
|
||||||
|
Promptkit YAML definitions. Go-consumer construction alternatives such as
|
||||||
|
single-file sources, injected `fs.FS` sources, in-memory profiles, fallback
|
||||||
|
profile filesystems, and prepared-execution handles are library-integration
|
||||||
|
features and are not required merely to support the complete definition
|
||||||
|
format.
|
||||||
|
|
||||||
|
### Promptkit v0.9.0 Compatibility Baseline
|
||||||
|
|
||||||
|
Required dependency and source changes:
|
||||||
|
|
||||||
|
- update `go.mod` and `go.sum` to select Promptkit v0.9.0 and its selected
|
||||||
|
OpenRouter and Rakestrawhome catalog dependencies;
|
||||||
|
- change Scriptorium's inbound HTTP reasoning-effort representation to preserve
|
||||||
|
Promptkit's pointer semantics;
|
||||||
|
- retain keyed Promptkit public struct literals and confirm no removed
|
||||||
|
`RunRequest.Metadata` use exists;
|
||||||
|
- update tagged Promptkit links and version references throughout Scriptorium;
|
||||||
|
and
|
||||||
|
- preserve Scriptorium's architecture guard against replacements, workspaces,
|
||||||
|
vendored Promptkit code, former facade packages, and Promptkit internal
|
||||||
|
imports.
|
||||||
|
|
||||||
|
Required compatibility review:
|
||||||
|
|
||||||
|
- verify application and maintained example endpoints satisfy Promptkit's
|
||||||
|
absolute HTTP/HTTPS endpoint rules;
|
||||||
|
- verify prompt content paths, file artifacts, identities, JSON documents, and
|
||||||
|
JSON-compatible extra parameters satisfy the v0.6.0 safety boundaries;
|
||||||
|
- verify maintained prompt roles are limited to `developer`, `system`, `user`,
|
||||||
|
and `assistant`;
|
||||||
|
- verify every positive repair budget is no greater than three and is paired
|
||||||
|
with `basic`, `json`, or `json_schema` validation;
|
||||||
|
- document that positive repair budgets authorize additional provider calls,
|
||||||
|
latency, token use, and cost;
|
||||||
|
- document that unset optional provider controls are omitted and that a zero
|
||||||
|
effective value can represent an unspecified provider control unless an
|
||||||
|
explicit request override supplied it; and
|
||||||
|
- update credential guidance for Promptkit's optional environment lookup
|
||||||
|
behavior, profile `api_key_env` semantics, and Scriptorium's prohibition on
|
||||||
|
raw API-key inputs.
|
||||||
|
|
||||||
|
### Session And Reasoning Request Controls
|
||||||
|
|
||||||
|
The CLI `run` and `render` commands will accept a direct session-ID flag and map
|
||||||
|
it to `promptkit.RunRequest.SessionID`. The value remains non-secret
|
||||||
|
application correlation metadata and is subject to Promptkit's normalization
|
||||||
|
and length rules.
|
||||||
|
|
||||||
|
The HTTP run request will accept `session_id` and map it through the existing
|
||||||
|
strict DTO boundary. Successful HTTP metadata will report the effective
|
||||||
|
session ID when present.
|
||||||
|
|
||||||
|
The CLI `run` and `render` commands will also accept a presence-aware reasoning
|
||||||
|
effort flag. Omission inherits the selected profile, a nonblank value replaces
|
||||||
|
it, and an explicitly supplied empty value clears it. The existing HTTP
|
||||||
|
`model.reasoning_effort` field will gain the same three-state behavior while
|
||||||
|
remaining a JSON string when present.
|
||||||
|
|
||||||
|
These additions must not introduce raw direct API-key flags or fields.
|
||||||
|
|
||||||
|
### Backend Identity And Capacity Outcomes
|
||||||
|
|
||||||
|
Scriptorium presentation will expose `SelectedBackendID` and the effective
|
||||||
|
target's `BackendID` without deriving identity from endpoint text. Backend
|
||||||
|
identity will be included where applicable in:
|
||||||
|
|
||||||
|
- prepared-run text and JSON output;
|
||||||
|
- the CLI run summary; and
|
||||||
|
- HTTP response metadata and effective model parameters.
|
||||||
|
|
||||||
|
Endpoint-only profiles continue to have no backend ID, and empty identities
|
||||||
|
must remain distinguishable from registered built-in or custom backends.
|
||||||
|
|
||||||
|
The HTTP adapter will classify `promptkit.ErrCapacityExceeded` separately from
|
||||||
|
provider generation failures. The target public outcome is HTTP `503` with a
|
||||||
|
stable `capacity_exceeded` code and a generic message. Scriptorium will not
|
||||||
|
invent retry timing or expose an untrusted diagnostic. CLI capacity rejection
|
||||||
|
continues to be a runtime failure, with a clear safe diagnostic and the
|
||||||
|
existing runtime-error exit status.
|
||||||
|
|
||||||
|
The HTTP server will use one appropriately scoped Promptkit engine across its
|
||||||
|
requests so backend admission, active-generation limits, and queue capacity
|
||||||
|
apply across concurrent in-flight work. The server may retain the operational
|
||||||
|
state needed for those limits and for request cancellation and lifecycle
|
||||||
|
management. This does not create durable per-request, conversation, or resume
|
||||||
|
state.
|
||||||
|
|
||||||
|
Scriptorium will continue to return generic public model-generation failures.
|
||||||
|
Promptkit `GenerationError` provider code, type, and message values will not be
|
||||||
|
added to the public HTTP response because they are untrusted and potentially
|
||||||
|
sensitive.
|
||||||
|
|
||||||
|
### Custom Backend Configuration
|
||||||
|
|
||||||
|
The strict Scriptorium application configuration will gain an optional custom
|
||||||
|
backend collection. Each entry will support the application-owned mapping
|
||||||
|
needed to construct a public `promptkit.Backend`:
|
||||||
|
|
||||||
|
- backend ID;
|
||||||
|
- OpenAI-compatible endpoint;
|
||||||
|
- optional API-key environment-variable name;
|
||||||
|
- optional JSON-compatible request-wide extra parameters;
|
||||||
|
- non-negative concurrency limit; and
|
||||||
|
- queue capacity with presence preserved so omission and explicit zero remain
|
||||||
|
different.
|
||||||
|
|
||||||
|
Raw API-key values will remain invalid. Unknown fields will remain errors.
|
||||||
|
Configured backends will be registered through `promptkit.WithBackend` during
|
||||||
|
engine construction and will be immutable and engine-scoped. Promptkit will
|
||||||
|
remain the canonical validator for backend IDs, endpoints, reserved request
|
||||||
|
parameters, capacity bounds, duplicate or reserved IDs, environment-variable
|
||||||
|
names, and JSON-compatible values. Scriptorium will add context appropriate to
|
||||||
|
its configuration error boundary without copying those framework rules.
|
||||||
|
|
||||||
|
The collection will be configuration-file-owned. This roadmap does not add
|
||||||
|
per-backend CLI flags. Existing endpoint-only profiles remain valid, while
|
||||||
|
profiles may select configured custom backend IDs through Promptkit's existing
|
||||||
|
`backend` field. Promptkit's built-in OpenRouter and Rakestrawhome backends and
|
||||||
|
profiles remain available without Scriptorium registration.
|
||||||
|
|
||||||
|
The application configuration shape is a `backends` mapping whose keys are the
|
||||||
|
backend IDs registered with Promptkit:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
backends:
|
||||||
|
local-gpu:
|
||||||
|
endpoint: http://localhost:11434/v1
|
||||||
|
api_key_env: LOCAL_GPU_API_KEY
|
||||||
|
extra_params:
|
||||||
|
provider_option: enabled
|
||||||
|
concurrency_limit: 2
|
||||||
|
queue_capacity: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
`endpoint` is required for each entry. The other fields are optional. Omitted
|
||||||
|
`queue_capacity` uses Promptkit's default queue policy when concurrency is
|
||||||
|
bounded, while an explicit zero disables queuing. The configuration contract
|
||||||
|
and complete maintained configuration example will document this shape when
|
||||||
|
the feature is implemented.
|
||||||
|
|
||||||
|
### Prompt And Profile Inspection CLI
|
||||||
|
|
||||||
|
Scriptorium will add a CLI inspection command with prompt and profile modes.
|
||||||
|
The intended command family is:
|
||||||
|
|
||||||
|
- `scriptorium inspect prompt`, backed by `Engine.InspectPrompt`; and
|
||||||
|
- `scriptorium inspect profile`, backed by `Engine.InspectProfile`.
|
||||||
|
|
||||||
|
Prompt inspection will select an ID and optional version and report the
|
||||||
|
normalized prompt identity, opaque prompt hash, declared default profile,
|
||||||
|
declared inputs, and normalized output contract. It will not require input
|
||||||
|
artifacts, resolve a profile, load a schema, render templates, reserve backend
|
||||||
|
capacity, or contact a model.
|
||||||
|
|
||||||
|
Profile inspection will select an explicit profile ID and report its resolved
|
||||||
|
effective target, backend identity, optional credential source name, and
|
||||||
|
whether a later request must provide credential configuration. It will not
|
||||||
|
read credential values, load a prompt, reserve capacity, or contact a model.
|
||||||
|
|
||||||
|
Inspection will support deterministic Scriptorium-owned text and JSON output
|
||||||
|
and normal output-file handling. Scriptorium will define explicit output DTOs
|
||||||
|
rather than treating Promptkit inspection structs as stable wire formats.
|
||||||
|
Inspection failures will use existing public Promptkit error identities and
|
||||||
|
Scriptorium-owned CLI diagnostics and exit behavior.
|
||||||
|
|
||||||
|
The prompt form accepts `--prompt`, optional `--prompt-version`, `--config`,
|
||||||
|
`--prompt-dir`, `--format text|json`, and `--out`. The profile form accepts
|
||||||
|
`--profile`, `--config`, `--profile-dir`, `--format text|json`, and `--out`.
|
||||||
|
Both forms reject positional arguments and default to text output independently
|
||||||
|
of `defaults.render_format`. Configuration discovery and CLI-over-file source
|
||||||
|
precedence match the existing command family. Prompt inspection requires an
|
||||||
|
effective prompt directory; profile inspection does not, because Promptkit's
|
||||||
|
built-in profiles remain inspectable without one.
|
||||||
|
|
||||||
|
## Documentation And Example Changes
|
||||||
|
|
||||||
|
Implementation of this roadmap requires coordinated updates to the canonical
|
||||||
|
owners of affected behavior:
|
||||||
|
|
||||||
|
- `docs/cli.md` for session and reasoning flags, inspection commands, output,
|
||||||
|
prompt-version selection, optional inputs, and exit behavior;
|
||||||
|
- `docs/api.md` for `session_id`, effective backend/session metadata, reasoning
|
||||||
|
presence semantics, optional inputs, and the capacity response;
|
||||||
|
- `docs/config.md` for custom backend configuration and current Promptkit
|
||||||
|
credential and provider-default semantics;
|
||||||
|
- `docs/operations.md` for capacity and custom-backend operational guidance
|
||||||
|
where deployment handling is affected;
|
||||||
|
- `docs/internal/adapters.md` and `docs/internal/sources.md` for implemented
|
||||||
|
mapping and engine-assembly behavior;
|
||||||
|
- `docs/policy/architecture.md` and `docs/internal/overview.md` only if concrete
|
||||||
|
component responsibilities or durable boundaries change; and
|
||||||
|
- maintained examples for valid v0.9.0 prompt/profile formats and custom
|
||||||
|
backend configuration.
|
||||||
|
|
||||||
|
Current-state documents must not describe these features as implemented until
|
||||||
|
the corresponding code lands.
|
||||||
|
|
||||||
|
## Validation And Completion Criteria
|
||||||
|
|
||||||
|
The roadmap is complete only when:
|
||||||
|
|
||||||
|
- ordinary and race-enabled tests pass against the tagged Promptkit v0.9.0
|
||||||
|
dependency;
|
||||||
|
- `go vet ./...` and `go build ./cmd/scriptorium` pass;
|
||||||
|
- Go formatting and `git diff --check` pass;
|
||||||
|
- the architecture guard passes with no workspace, replacement, vendor tree,
|
||||||
|
former facade, or Promptkit internal import;
|
||||||
|
- maintained render, custom-backend, session, reasoning, and inspection
|
||||||
|
examples execute successfully without real credentials or provider calls;
|
||||||
|
- CLI tests cover explicit and omitted prompt versions for prompt IDs with one
|
||||||
|
or multiple definitions;
|
||||||
|
- CLI and HTTP tests cover prompts with no declared inputs, omitted optional
|
||||||
|
inputs, missing required inputs, and template-referenced inputs;
|
||||||
|
- HTTP tests cover omitted, replacement, and clearing reasoning states;
|
||||||
|
- CLI and HTTP tests cover session propagation and invalid session handling;
|
||||||
|
- presentation tests cover registered and endpoint-only backend identity;
|
||||||
|
- HTTP tests cover capacity rejection independently from provider failure;
|
||||||
|
- concurrent HTTP tests demonstrate that one server's requests share backend
|
||||||
|
admission and cannot bypass capacity through per-request engine creation;
|
||||||
|
- configuration tests cover strict decoding, precedence, queue-capacity
|
||||||
|
presence, invalid backends, and secret-field rejection;
|
||||||
|
- inspection tests cover deterministic text and JSON, source selection,
|
||||||
|
output-file handling, safe credential presentation, and public error
|
||||||
|
mapping;
|
||||||
|
- all maintained prompt roles and repair budgets are valid under Promptkit
|
||||||
|
v0.9.0; and
|
||||||
|
- representative definition fixtures cover inline and file-backed messages,
|
||||||
|
cache control, session templates, every output validation mode, schemas,
|
||||||
|
profile inheritance, built-in and custom backends, execution controls,
|
||||||
|
credential environment names, and extra parameters without Scriptorium
|
||||||
|
independently parsing those fields; and
|
||||||
|
- affected documentation links and copyable examples validate successfully.
|
||||||
|
|
||||||
|
## Explicit Non-Goals
|
||||||
|
|
||||||
|
This roadmap does not include:
|
||||||
|
|
||||||
|
- appended request messages in either executable interface;
|
||||||
|
- retained or durable prepared-execution handles;
|
||||||
|
- HTTP prompt or profile inspection routes;
|
||||||
|
- embedded application fallback profiles;
|
||||||
|
- single-file, injected `fs.FS`, or in-memory prompt and profile source
|
||||||
|
configuration;
|
||||||
|
- request-level output-contract replacement through the executable interfaces;
|
||||||
|
- public disclosure of structured provider error details;
|
||||||
|
- direct imports or configuration of Promptkit's external catalog modules;
|
||||||
|
- raw API-key configuration, CLI flags, or HTTP fields;
|
||||||
|
- durable run state, conversations, retries, archives, checkpoints, or resume;
|
||||||
|
or
|
||||||
|
- changes to Promptkit itself.
|
||||||
@@ -1,282 +0,0 @@
|
|||||||
# Migration Step 9: Complete Release And Documentation Cutover
|
|
||||||
|
|
||||||
## Status
|
|
||||||
|
|
||||||
Proposed. Steps 1 through 8 of the
|
|
||||||
[migration roadmap](migration.md) are complete. This is the final migration
|
|
||||||
gate.
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
Finish the Promptkit split as a released, documented, and independently
|
|
||||||
maintainable project boundary. Step 9 turns the already-implemented repository
|
|
||||||
state into the supported public release state, gives former Scriptorium Go
|
|
||||||
consumers a durable migration path, reconciles release guidance in both
|
|
||||||
repositories, and retires the temporary migration records once their work is
|
|
||||||
complete.
|
|
||||||
|
|
||||||
This roadmap defines the intended end state.
|
|
||||||
|
|
||||||
## Release Baseline And Version Decisions
|
|
||||||
|
|
||||||
The coordinated release boundary is:
|
|
||||||
|
|
||||||
- Promptkit `v0.1.0`, already published from commit
|
|
||||||
`9e68a2bbf779545995270c47842048a3bc6c85dc`, is the framework version consumed
|
|
||||||
by Scriptorium;
|
|
||||||
- Scriptorium `v0.11.1` is the final published framework-bearing Scriptorium
|
|
||||||
release; and
|
|
||||||
- Scriptorium `v0.12.0` is the first slim application-only release.
|
|
||||||
|
|
||||||
The `v0.12.0` version satisfies
|
|
||||||
[ADR 0002](../adr/0002-split-promptkit-from-scriptorium.md), which requires
|
|
||||||
the first slim pre-`v1` Scriptorium release to advance the minor version beyond
|
|
||||||
the final framework-bearing release.
|
|
||||||
|
|
||||||
Promptkit does not need another tag merely to complete this migration.
|
|
||||||
Promptkit documentation corrections that do not change the library contract
|
|
||||||
may land on its main branch without changing Scriptorium's dependency. If Step
|
|
||||||
9 discovers that a Promptkit code or consumer-visible contract change is
|
|
||||||
required, Promptkit must instead publish an appropriate later semantic version
|
|
||||||
first, and Scriptorium must adopt and validate that tag before `v0.12.0` is
|
|
||||||
published.
|
|
||||||
|
|
||||||
No release may depend on a Go workspace, local module replacement, vendored
|
|
||||||
sibling source, unpublished commit, or an unpushed tag.
|
|
||||||
|
|
||||||
## Scriptorium Release Readiness
|
|
||||||
|
|
||||||
Scriptorium must have a durable, canonical release procedure appropriate to
|
|
||||||
its hosted tag-triggered binary workflow. The procedure and contributor
|
|
||||||
reading guide must collectively define:
|
|
||||||
|
|
||||||
- semantic-version selection and the clean-checkout preconditions;
|
|
||||||
- validation outside a Go workspace and without a module replacement;
|
|
||||||
- module identity, dependency resolution, module tidiness, tests, race tests,
|
|
||||||
vet, formatting, link, example, and repository-hygiene checks;
|
|
||||||
- release-equivalent Linux `amd64` and `arm64` builds;
|
|
||||||
- annotated tag creation and publication;
|
|
||||||
- observation of the hosted release workflow;
|
|
||||||
- verification of published binaries and checksums; and
|
|
||||||
- post-publication smoke checks using downloaded release artifacts where the
|
|
||||||
execution platform permits.
|
|
||||||
|
|
||||||
The documentation policy must assign the release procedure one canonical
|
|
||||||
owner, and `docs/development.md` must route release work to it. Operations
|
|
||||||
documentation should link to release material only when an operator task
|
|
||||||
requires it; it must not become a second release procedure.
|
|
||||||
|
|
||||||
The tag workflow must accurately describe the current application. In
|
|
||||||
particular, remove the obsolete linker assignment to the deleted
|
|
||||||
`internal/buildinfo.Version` symbol. The release continues to use stripped,
|
|
||||||
trimmed binaries unless a separate supported application-version interface is
|
|
||||||
introduced. Adding a new `--version` command or other product behavior is not
|
|
||||||
part of this migration.
|
|
||||||
|
|
||||||
Release validation must exercise the same build commands and artifact names as
|
|
||||||
the hosted workflow. The resulting `v0.12.0` release must contain the supported
|
|
||||||
Linux `amd64` and `arm64` Scriptorium binaries and published SHA-256 checksums,
|
|
||||||
with no framework source or Promptkit binary artifact.
|
|
||||||
|
|
||||||
## Promptkit Release Readiness
|
|
||||||
|
|
||||||
Promptkit remains governed by
|
|
||||||
[ADR 0003](../adr/0003-use-maintainer-run-validation-and-tag-only-releases-for-promptkit.md):
|
|
||||||
maintainer-run validation, semantic Go module tags, and no hosted CI or binary
|
|
||||||
release artifacts.
|
|
||||||
|
|
||||||
Its release procedure must be corrected from pre-release language to current,
|
|
||||||
reusable guidance. It must no longer describe `v0.1.0` as an uncreated planned
|
|
||||||
release or instruct a maintainer to create an existing tag. It should:
|
|
||||||
|
|
||||||
- acknowledge `v0.1.0` as the initial published release;
|
|
||||||
- use version-agnostic instructions for later releases;
|
|
||||||
- retain the clean-checkout, full-validation, tag-ordering, and remote-tag
|
|
||||||
verification requirements; and
|
|
||||||
- require pre-`v1` release notes to identify public API changes and consumer
|
|
||||||
migration requirements.
|
|
||||||
|
|
||||||
Before Scriptorium `v0.12.0` is tagged, independently reconfirm that the local
|
|
||||||
and remote Promptkit `v0.1.0` tags resolve to the expected source commit, that
|
|
||||||
the tagged module is available through ordinary Go module resolution, and that
|
|
||||||
Promptkit passes its documented release validation without Scriptorium or
|
|
||||||
sibling-repository state.
|
|
||||||
|
|
||||||
## Go Consumer Migration Guidance
|
|
||||||
|
|
||||||
Scriptorium must publish a permanent migration guide under `docs/consumers/`
|
|
||||||
for consumers of the former Go package. Scriptorium owns this guide because it
|
|
||||||
describes departure from Scriptorium's removed API; Promptkit's declarations,
|
|
||||||
GoDoc, format reference, and consumer guide remain canonical for the
|
|
||||||
destination contract.
|
|
||||||
|
|
||||||
The guide must identify the supported migration baseline:
|
|
||||||
|
|
||||||
- source: Scriptorium `v0.11.1` and import path
|
|
||||||
`gitea.maximumdirect.net/eric/scriptorium`;
|
|
||||||
- destination: Promptkit `v0.1.0` and import path
|
|
||||||
`gitea.maximumdirect.net/eric/promptkit`; and
|
|
||||||
- Scriptorium `v0.12.0` and later: executable application only, with no root Go
|
|
||||||
package or compatibility facade.
|
|
||||||
|
|
||||||
It must provide a minimal, copyable migration workflow:
|
|
||||||
|
|
||||||
1. replace the Scriptorium module requirement and Go imports with Promptkit
|
|
||||||
`v0.1.0`;
|
|
||||||
2. update package qualifiers from `scriptorium` to `promptkit`;
|
|
||||||
3. run `go mod tidy`;
|
|
||||||
4. compile and test the consuming project; and
|
|
||||||
5. verify prompt, profile, schema, credential, timeout, validation, injected
|
|
||||||
client, and error-handling behavior relevant to that consumer.
|
|
||||||
|
|
||||||
The guide should explain that the established engine, request, result, profile,
|
|
||||||
source-option, model-client, artifact, validation, and error shapes were
|
|
||||||
intentionally preserved where practical, while Promptkit also owns the
|
|
||||||
post-extraction public error identities and artifact-reader extension point.
|
|
||||||
It must direct exact API questions to Promptkit's tagged GoDoc and consumer
|
|
||||||
guide rather than duplicating the declaration reference.
|
|
||||||
|
|
||||||
The guide must also state the deliberate compatibility policy: there are no
|
|
||||||
Scriptorium aliases, forwarding packages, or deprecated wrappers. A consumer
|
|
||||||
that cannot migrate may remain pinned to `v0.11.1`, but it will not receive the
|
|
||||||
application-only Scriptorium line through that package API.
|
|
||||||
|
|
||||||
## Documentation And Project Identity Cutover
|
|
||||||
|
|
||||||
Review both repositories as separate products and reconcile every maintained
|
|
||||||
link, example, package comment, and current-state statement with the released
|
|
||||||
boundary.
|
|
||||||
|
|
||||||
Scriptorium documentation must:
|
|
||||||
|
|
||||||
- present Scriptorium as a CLI and HTTP application, not a Go framework;
|
|
||||||
- link in-process Go consumers and framework contract questions to tagged
|
|
||||||
Promptkit `v0.1.0` documentation;
|
|
||||||
- link former Scriptorium Go consumers to the migration guide;
|
|
||||||
- keep CLI, HTTP, application configuration, operations, subprocess, and
|
|
||||||
executable examples under Scriptorium ownership; and
|
|
||||||
- avoid reproducing Promptkit fields, defaults, public declarations, or
|
|
||||||
integration contracts.
|
|
||||||
|
|
||||||
Promptkit documentation must:
|
|
||||||
|
|
||||||
- present Promptkit as the reusable Go framework and owner of its root API,
|
|
||||||
file formats, built-in profiles, validation, and outbound integration;
|
|
||||||
- retain Scriptorium only as a downstream application example or related
|
|
||||||
project, not as a framework owner or dependency;
|
|
||||||
- link to Scriptorium only for executable CLI and HTTP workflows when that
|
|
||||||
navigation is useful; and
|
|
||||||
- contain no stale extraction, planned-first-release, or pre-cutover claims.
|
|
||||||
|
|
||||||
Cross-project links must point to the canonical owner. Scriptorium links that
|
|
||||||
define the framework version it consumes remain pinned to Promptkit `v0.1.0`;
|
|
||||||
general project-navigation links may point to the other repository's current
|
|
||||||
project entry point. Maintained examples must stay repository-local and must
|
|
||||||
not require a sibling checkout.
|
|
||||||
|
|
||||||
## Release Notes And Public Communication
|
|
||||||
|
|
||||||
The Scriptorium `v0.12.0` release notes must clearly identify the release as a
|
|
||||||
breaking project-boundary change. They must:
|
|
||||||
|
|
||||||
- state that Scriptorium is now an executable-only CLI and HTTP application;
|
|
||||||
- state that the former Go framework moved to Promptkit;
|
|
||||||
- link the Scriptorium migration guide and Promptkit `v0.1.0` consumer
|
|
||||||
documentation;
|
|
||||||
- identify `v0.11.1` as the final framework-bearing Scriptorium release;
|
|
||||||
- summarize the retained Scriptorium interfaces and the removed root package;
|
|
||||||
- record that all known downstream Go consumers were migrated before release;
|
|
||||||
and
|
|
||||||
- identify the Promptkit version used by the released binary.
|
|
||||||
|
|
||||||
Release notes must not serve as a duplicate CLI, HTTP, configuration, or
|
|
||||||
Promptkit API reference. They should route readers to the corresponding
|
|
||||||
canonical documents.
|
|
||||||
|
|
||||||
## Independent Release And Artifact Verification
|
|
||||||
|
|
||||||
The final acceptance run must treat the repositories as independent remote
|
|
||||||
projects:
|
|
||||||
|
|
||||||
- validate Promptkit from its exact published tag without Scriptorium;
|
|
||||||
- validate Scriptorium from its intended release commit outside any workspace
|
|
||||||
and with a fresh module and build cache that cannot read the sibling
|
|
||||||
Promptkit checkout;
|
|
||||||
- confirm the Scriptorium module graph selects the intended published Promptkit
|
|
||||||
tag;
|
|
||||||
- verify both working trees contain no tracked workspace, replacement, vendored
|
|
||||||
cross-project source, generated binary, credential, or temporary release
|
|
||||||
residue;
|
|
||||||
- publish and verify the annotated Scriptorium `v0.12.0` tag;
|
|
||||||
- verify the hosted release completes and publishes the expected binaries and
|
|
||||||
checksums;
|
|
||||||
- download the published artifacts into a temporary location, verify their
|
|
||||||
checksums, file types, target architectures, and basic executable behavior;
|
|
||||||
and
|
|
||||||
- recheck maintained local and cross-project documentation links after
|
|
||||||
publication.
|
|
||||||
|
|
||||||
Ordinary ignored developer files, including an ignored local Scriptorium
|
|
||||||
binary, do not fail repository hygiene. Acceptance concerns tracked content,
|
|
||||||
release inputs, generated files introduced by the release work, and published
|
|
||||||
artifacts.
|
|
||||||
|
|
||||||
## Roadmap Retirement
|
|
||||||
|
|
||||||
Roadmaps are temporary coordination documents. After every Step 9 completion
|
|
||||||
criterion is satisfied and durable release and migration records exist:
|
|
||||||
|
|
||||||
- mark Step 9 and the overall migration complete before cleanup;
|
|
||||||
- preserve any still-useful current contract in its canonical permanent owner;
|
|
||||||
- rely on ADRs, Git history, tags, release notes, and the migration guide for
|
|
||||||
durable decision and release history;
|
|
||||||
- remove completed migration, step, and implementation roadmaps rather than
|
|
||||||
retaining them as a second current-state reference; and
|
|
||||||
- repair every incoming link affected by that removal.
|
|
||||||
|
|
||||||
The roadmap files must remain until the out-of-band tag and hosted release have
|
|
||||||
been verified. Creating a release candidate or merging documentation is not
|
|
||||||
enough to declare the migration complete.
|
|
||||||
|
|
||||||
## Non-Goals
|
|
||||||
|
|
||||||
Step 9 does not:
|
|
||||||
|
|
||||||
- redesign Promptkit's public API or Scriptorium's CLI or HTTP contracts;
|
|
||||||
- restore a Scriptorium Go facade or add compatibility shims;
|
|
||||||
- add hosted CI or binary artifacts to Promptkit;
|
|
||||||
- add new Scriptorium target platforms beyond the existing Linux `amd64` and
|
|
||||||
`arm64` release policy;
|
|
||||||
- introduce an application version command solely to preserve a stale linker
|
|
||||||
flag;
|
|
||||||
- redo the completed downstream migration inventory; or
|
|
||||||
- require a new Promptkit release when no Promptkit contract change is needed.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
|
|
||||||
Step 9 is complete only when all of the following are true:
|
|
||||||
|
|
||||||
- Promptkit `v0.1.0` remains independently available, validated, and correctly
|
|
||||||
documented as the published framework dependency;
|
|
||||||
- Scriptorium has an accurate, canonical, and tested release procedure;
|
|
||||||
- the Scriptorium release workflow contains no reference to removed framework
|
|
||||||
or build-information packages and produces only the intended application
|
|
||||||
artifacts;
|
|
||||||
- the permanent Go-consumer migration guide is complete, copyable, and linked
|
|
||||||
from appropriate Scriptorium entry points;
|
|
||||||
- both repositories' permanent documentation, examples, package comments, and
|
|
||||||
cross-project links reflect distinct and canonical ownership;
|
|
||||||
- Scriptorium `v0.12.0` is published from a clean, independently validated
|
|
||||||
commit that directly requires a published Promptkit tag;
|
|
||||||
- the hosted Scriptorium release publishes verified Linux `amd64` and `arm64`
|
|
||||||
binaries and SHA-256 checksums;
|
|
||||||
- the `v0.12.0` release notes communicate the breaking package move and link to
|
|
||||||
the migration path;
|
|
||||||
- all known downstream Go consumers remain migrated or explicitly
|
|
||||||
dispositioned;
|
|
||||||
- neither release depends on local multi-repository state; and
|
|
||||||
- completed migration roadmaps are removed after their useful content and
|
|
||||||
completion evidence have durable owners.
|
|
||||||
|
|
||||||
When these criteria are satisfied, the Promptkit split is complete and both
|
|
||||||
projects can evolve, validate, version, document, and release independently.
|
|
||||||
@@ -2,6 +2,15 @@ prompt_dir: ./examples/prompts
|
|||||||
profile_dir: ./examples/profiles
|
profile_dir: ./examples/profiles
|
||||||
schema_dir: ./examples/schemas
|
schema_dir: ./examples/schemas
|
||||||
|
|
||||||
|
backends:
|
||||||
|
local-gpu:
|
||||||
|
endpoint: http://localhost:11434/v1
|
||||||
|
api_key_env: LOCAL_GPU_API_KEY
|
||||||
|
extra_params:
|
||||||
|
provider_option: enabled
|
||||||
|
concurrency_limit: 2
|
||||||
|
queue_capacity: 0
|
||||||
|
|
||||||
server:
|
server:
|
||||||
addr: 127.0.0.1:8080
|
addr: 127.0.0.1:8080
|
||||||
artifact_root: .
|
artifact_root: .
|
||||||
|
|||||||
4
examples/profiles/local-gpu.yml
Normal file
4
examples/profiles/local-gpu.yml
Normal file
@@ -0,0 +1,4 @@
|
|||||||
|
id: local-gpu
|
||||||
|
backend: local-gpu
|
||||||
|
model: local-model
|
||||||
|
reasoning_effort: low
|
||||||
25
examples/render-v0.9-features.sh
Executable file
25
examples/render-v0.9-features.sh
Executable file
@@ -0,0 +1,25 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
cd "$repo_root"
|
||||||
|
|
||||||
|
go run ./cmd/scriptorium render \
|
||||||
|
--config ./examples/config.full.yml \
|
||||||
|
--prompt generic.markdown_summary \
|
||||||
|
--prompt-version 1.0.0 \
|
||||||
|
--profile local-gpu \
|
||||||
|
--session-id example-session \
|
||||||
|
--reasoning-effort= \
|
||||||
|
--input transcript=./examples/fixtures/transcript.md \
|
||||||
|
--input glossary=./examples/fixtures/glossary.yml
|
||||||
|
|
||||||
|
go run ./cmd/scriptorium inspect prompt \
|
||||||
|
--config ./examples/config.full.yml \
|
||||||
|
--prompt generic.markdown_summary \
|
||||||
|
--format json
|
||||||
|
|
||||||
|
go run ./cmd/scriptorium inspect profile \
|
||||||
|
--config ./examples/config.full.yml \
|
||||||
|
--profile local-gpu \
|
||||||
|
--format json
|
||||||
4
go.mod
4
go.mod
@@ -3,11 +3,13 @@ module gitea.maximumdirect.net/eric/scriptorium
|
|||||||
go 1.25.5
|
go 1.25.5
|
||||||
|
|
||||||
require (
|
require (
|
||||||
gitea.maximumdirect.net/eric/promptkit v0.1.0
|
gitea.maximumdirect.net/eric/promptkit v0.9.0
|
||||||
gopkg.in/yaml.v3 v3.0.1
|
gopkg.in/yaml.v3 v3.0.1
|
||||||
)
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
|
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0 // indirect
|
||||||
|
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0 // indirect
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 // indirect
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 // indirect
|
||||||
golang.org/x/text v0.14.0 // indirect
|
golang.org/x/text v0.14.0 // indirect
|
||||||
)
|
)
|
||||||
|
|||||||
8
go.sum
8
go.sum
@@ -1,5 +1,9 @@
|
|||||||
gitea.maximumdirect.net/eric/promptkit v0.1.0 h1:vuKeBxkiY8E54LRFbLQFjlJJCiOfMvB1++DYBCrD/ug=
|
gitea.maximumdirect.net/eric/promptkit v0.9.0 h1:IpvDRC8L6xRxQ9hpuyKOmMc5b6MeLTKYyx+h1YAjy08=
|
||||||
gitea.maximumdirect.net/eric/promptkit v0.1.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
gitea.maximumdirect.net/eric/promptkit v0.9.0/go.mod h1:oMJ/WUJImUtwJ5e+6MAGECPYAErAkOaKel0G+3T/b4E=
|
||||||
|
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0 h1:lc062euk2qseO//D762i3JaFyulDNML3eQQX7DkYTho=
|
||||||
|
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0/go.mod h1:AIa7kAu2mfrRQgcspe4L+DW51WqgnALQT60lqkEywJI=
|
||||||
|
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0 h1:j9YY7wsTVjzke2kHH4YAzpU0oUpM+x+nXwl1IeS+2eg=
|
||||||
|
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0/go.mod h1:4RNS+LILDg4JbS4Ts9Lwy1C92wauXJIbeQaalps4Koo=
|
||||||
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
||||||
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
||||||
|
|||||||
200
internal/adapter/cli/inspect.go
Normal file
200
internal/adapter/cli/inspect.go
Normal file
@@ -0,0 +1,200 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"io/fs"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
|
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
|
||||||
|
appformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
||||||
|
)
|
||||||
|
|
||||||
|
type promptInspectionConfig struct {
|
||||||
|
configPath, promptDir, promptID, promptVersion, outputPath string
|
||||||
|
configExplicit bool
|
||||||
|
outputFormat appformat.OutputFormat
|
||||||
|
}
|
||||||
|
|
||||||
|
type profileInspectionConfig struct {
|
||||||
|
configPath, profileDir, profileID, outputPath string
|
||||||
|
configExplicit bool
|
||||||
|
outputFormat appformat.OutputFormat
|
||||||
|
}
|
||||||
|
|
||||||
|
// emptyPromptDefinitionFS satisfies Promptkit's engine-level prompt-source
|
||||||
|
// requirement without exposing the caller's working directory. Profile
|
||||||
|
// inspection never reads this source.
|
||||||
|
type emptyPromptDefinitionFS struct{}
|
||||||
|
|
||||||
|
func (emptyPromptDefinitionFS) Open(name string) (fs.File, error) {
|
||||||
|
return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist}
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectCommand(args []string, stdout, stderr io.Writer) int {
|
||||||
|
if len(args) == 0 {
|
||||||
|
fmt.Fprintln(stderr, "inspect parse error: inspection mode is required")
|
||||||
|
return ExitRuntimeError
|
||||||
|
}
|
||||||
|
if args[0] == "profile" {
|
||||||
|
return inspectProfileCommand(args[1:], stdout, stderr)
|
||||||
|
}
|
||||||
|
if args[0] != "prompt" {
|
||||||
|
fmt.Fprintln(stderr, "inspect parse error: unknown inspection mode")
|
||||||
|
return ExitRuntimeError
|
||||||
|
}
|
||||||
|
cfg, err := parsePromptInspectionArgs(args[1:])
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(stderr, "inspect parse error: %v\n", err)
|
||||||
|
return ExitRuntimeError
|
||||||
|
}
|
||||||
|
settings, err := resolveAppSettingsForPromptInspection(cfg)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(stderr, "inspect error: %v\n", err)
|
||||||
|
return ExitRuntimeError
|
||||||
|
}
|
||||||
|
engine, err := newEngine(settings)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
||||||
|
return ExitRuntimeError
|
||||||
|
}
|
||||||
|
inspection, err := engine.InspectPrompt(context.Background(), cfg.promptID, cfg.promptVersion)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(stderr, "inspect error: %v\n", err)
|
||||||
|
return ExitRuntimeError
|
||||||
|
}
|
||||||
|
data, err := appformat.FormatPromptInspection(inspection, cfg.outputFormat)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(stderr, "inspect error: %v\n", err)
|
||||||
|
return ExitRuntimeError
|
||||||
|
}
|
||||||
|
if err := writeOutput(stdout, cfg.outputPath, data); err != nil {
|
||||||
|
fmt.Fprintf(stderr, "output write error: %v\n", err)
|
||||||
|
return ExitRuntimeError
|
||||||
|
}
|
||||||
|
return ExitOK
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectProfileCommand(args []string, stdout, stderr io.Writer) int {
|
||||||
|
cfg, err := parseProfileInspectionArgs(args)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(stderr, "inspect parse error: %v\n", err)
|
||||||
|
return ExitRuntimeError
|
||||||
|
}
|
||||||
|
settings, err := resolveAppSettingsForProfileInspection(cfg)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(stderr, "inspect error: %v\n", err)
|
||||||
|
return ExitRuntimeError
|
||||||
|
}
|
||||||
|
engine, err := newEngine(settings, promptkit.WithPromptFS(emptyPromptDefinitionFS{}, "."))
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
||||||
|
return ExitRuntimeError
|
||||||
|
}
|
||||||
|
inspection, err := engine.InspectProfile(context.Background(), cfg.profileID)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(stderr, "inspect error: %v\n", err)
|
||||||
|
return ExitRuntimeError
|
||||||
|
}
|
||||||
|
data, err := appformat.FormatProfileInspection(inspection, cfg.outputFormat)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(stderr, "inspect error: %v\n", err)
|
||||||
|
return ExitRuntimeError
|
||||||
|
}
|
||||||
|
if err := writeOutput(stdout, cfg.outputPath, data); err != nil {
|
||||||
|
fmt.Fprintf(stderr, "output write error: %v\n", err)
|
||||||
|
return ExitRuntimeError
|
||||||
|
}
|
||||||
|
return ExitOK
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseProfileInspectionArgs(args []string) (*profileInspectionConfig, error) {
|
||||||
|
cfg := &profileInspectionConfig{outputFormat: appformat.DefaultOutputFormat}
|
||||||
|
fs := flag.NewFlagSet("inspect profile", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
registerConfigPathFlag(fs, &cfg.configPath)
|
||||||
|
fs.StringVar(&cfg.profileDir, "profile-dir", "", "directory containing execution profiles")
|
||||||
|
fs.StringVar(&cfg.profileID, "profile", "", "profile ID to inspect")
|
||||||
|
rawFormat := ""
|
||||||
|
fs.StringVar(&rawFormat, "format", "", "output format: text or json")
|
||||||
|
fs.StringVar(&cfg.outputPath, "out", "", "optional output file path")
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
cfg.configExplicit = flagWasSet(fs, "config")
|
||||||
|
if fs.NArg() > 0 {
|
||||||
|
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(cfg.profileID) == "" {
|
||||||
|
return nil, errors.New("--profile is required")
|
||||||
|
}
|
||||||
|
format, err := appformat.ParseOutputFormat(rawFormat)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
cfg.outputFormat = format
|
||||||
|
if cfg.outputPath != "" {
|
||||||
|
cfg.outputPath = filepath.Clean(cfg.outputPath)
|
||||||
|
}
|
||||||
|
return cfg, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveAppSettingsForProfileInspection(cfg *profileInspectionConfig) (engineSettings, error) {
|
||||||
|
settings, err := resolveAppSettingsWithConfigPresence(cfg.configPath, cfg.configExplicit, appconfig.CLIOverrides{
|
||||||
|
ProfileDir: cfg.profileDir,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return engineSettings{}, err
|
||||||
|
}
|
||||||
|
return engineSettings{profileDir: settings.ProfileDir, backends: settings.Backends}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func parsePromptInspectionArgs(args []string) (*promptInspectionConfig, error) {
|
||||||
|
cfg := &promptInspectionConfig{outputFormat: appformat.DefaultOutputFormat}
|
||||||
|
fs := flag.NewFlagSet("inspect prompt", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
registerConfigPathFlag(fs, &cfg.configPath)
|
||||||
|
fs.StringVar(&cfg.promptDir, "prompt-dir", "", "directory containing prompt definitions")
|
||||||
|
fs.StringVar(&cfg.promptID, "prompt", "", "prompt ID to inspect")
|
||||||
|
fs.StringVar(&cfg.promptVersion, "prompt-version", "", "optional prompt version")
|
||||||
|
rawFormat := ""
|
||||||
|
fs.StringVar(&rawFormat, "format", "", "output format: text or json")
|
||||||
|
fs.StringVar(&cfg.outputPath, "out", "", "optional output file path")
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
cfg.configExplicit = flagWasSet(fs, "config")
|
||||||
|
if fs.NArg() > 0 {
|
||||||
|
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(cfg.promptID) == "" {
|
||||||
|
return nil, errors.New("--prompt is required")
|
||||||
|
}
|
||||||
|
format, err := appformat.ParseOutputFormat(rawFormat)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
cfg.outputFormat = format
|
||||||
|
if cfg.outputPath != "" {
|
||||||
|
cfg.outputPath = filepath.Clean(cfg.outputPath)
|
||||||
|
}
|
||||||
|
return cfg, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveAppSettingsForPromptInspection(cfg *promptInspectionConfig) (engineSettings, error) {
|
||||||
|
settings, err := resolveAppSettingsWithConfigPresence(cfg.configPath, cfg.configExplicit, appconfig.CLIOverrides{
|
||||||
|
PromptDir: cfg.promptDir,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return engineSettings{}, err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(settings.PromptDir) == "" {
|
||||||
|
return engineSettings{}, errors.New(errPromptDirRequired)
|
||||||
|
}
|
||||||
|
return engineSettings{promptDir: settings.PromptDir, backends: settings.Backends}, nil
|
||||||
|
}
|
||||||
@@ -35,7 +35,9 @@ type runConfig struct {
|
|||||||
promptDir string
|
promptDir string
|
||||||
profileDir string
|
profileDir string
|
||||||
promptID string
|
promptID string
|
||||||
|
promptVersion string
|
||||||
profileID string
|
profileID string
|
||||||
|
sessionID string
|
||||||
inputRaw listFlag
|
inputRaw listFlag
|
||||||
varRaw listFlag
|
varRaw listFlag
|
||||||
outputPath string
|
outputPath string
|
||||||
@@ -45,7 +47,9 @@ type runConfig struct {
|
|||||||
temperature float64
|
temperature float64
|
||||||
maxTokens int
|
maxTokens int
|
||||||
topP float64
|
topP float64
|
||||||
|
reasoningEffort string
|
||||||
schemaDir string
|
schemaDir string
|
||||||
|
backends []appconfig.BackendSettings
|
||||||
timeout time.Duration
|
timeout time.Duration
|
||||||
|
|
||||||
defaultRenderFormat renderformat.PreparedRunOutputFormat
|
defaultRenderFormat renderformat.PreparedRunOutputFormat
|
||||||
@@ -56,6 +60,7 @@ type runConfig struct {
|
|||||||
temperatureSet bool
|
temperatureSet bool
|
||||||
maxTokensSet bool
|
maxTokensSet bool
|
||||||
topPSet bool
|
topPSet bool
|
||||||
|
reasoningEffortSet bool
|
||||||
timeoutSet bool
|
timeoutSet bool
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -75,6 +80,7 @@ type serveConfig struct {
|
|||||||
maxRequestBytes int64
|
maxRequestBytes int64
|
||||||
maxArtifactBytes int64
|
maxArtifactBytes int64
|
||||||
maxResponseBytes int64
|
maxResponseBytes int64
|
||||||
|
backends []appconfig.BackendSettings
|
||||||
}
|
}
|
||||||
|
|
||||||
type commonCommandSettings struct {
|
type commonCommandSettings struct {
|
||||||
@@ -87,6 +93,7 @@ type commonCommandSettings struct {
|
|||||||
maxArtifactBytes int64
|
maxArtifactBytes int64
|
||||||
maxResponseBytes int64
|
maxResponseBytes int64
|
||||||
defaultRenderFormat renderformat.PreparedRunOutputFormat
|
defaultRenderFormat renderformat.PreparedRunOutputFormat
|
||||||
|
backends []appconfig.BackendSettings
|
||||||
}
|
}
|
||||||
|
|
||||||
type listFlag []string
|
type listFlag []string
|
||||||
@@ -113,6 +120,8 @@ func Run(args []string, stdout, stderr io.Writer) int {
|
|||||||
return renderCommand(args[1:], stdout, stderr)
|
return renderCommand(args[1:], stdout, stderr)
|
||||||
case "serve":
|
case "serve":
|
||||||
return serveCommand(args[1:], stderr)
|
return serveCommand(args[1:], stderr)
|
||||||
|
case "inspect":
|
||||||
|
return inspectCommand(args[1:], stdout, stderr)
|
||||||
default:
|
default:
|
||||||
fmt.Fprintf(stderr, "unknown command %q\n", args[0])
|
fmt.Fprintf(stderr, "unknown command %q\n", args[0])
|
||||||
printUsage(stderr)
|
printUsage(stderr)
|
||||||
@@ -133,7 +142,7 @@ func runCommand(args []string, stdout, stderr io.Writer) int {
|
|||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
}
|
}
|
||||||
|
|
||||||
engine, err := newEngine(cfg)
|
engine, err := newEngine(cfg.engineSettings())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
@@ -141,7 +150,7 @@ func runCommand(args []string, stdout, stderr io.Writer) int {
|
|||||||
|
|
||||||
res, runErr := engine.Run(context.Background(), req)
|
res, runErr := engine.Run(context.Background(), req)
|
||||||
if runErr != nil {
|
if runErr != nil {
|
||||||
fmt.Fprintf(stderr, "run error: %v\n", runErr)
|
fmt.Fprintln(stderr, runErrorMessage(runErr))
|
||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -167,7 +176,7 @@ func renderCommand(args []string, stdout, stderr io.Writer) int {
|
|||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
}
|
}
|
||||||
|
|
||||||
engine, err := newEngine(&cfg.runConfig)
|
engine, err := newEngine(cfg.runConfig.engineSettings())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
@@ -205,11 +214,7 @@ func serveCommand(args []string, stderr io.Writer) int {
|
|||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
}
|
}
|
||||||
|
|
||||||
engine, err := newEngine(&runConfig{
|
engine, err := newEngine(cfg.engineSettings(), promptkit.WithArtifactReader(artifactReader))
|
||||||
promptDir: cfg.promptDir,
|
|
||||||
profileDir: cfg.profileDir,
|
|
||||||
schemaDir: cfg.schemaDir,
|
|
||||||
}, promptkit.WithArtifactReader(artifactReader))
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
||||||
return ExitRuntimeError
|
return ExitRuntimeError
|
||||||
@@ -329,6 +334,7 @@ func parseServeArgs(args []string) (*serveConfig, error) {
|
|||||||
cfg.maxRequestBytes = settings.maxRequestBytes
|
cfg.maxRequestBytes = settings.maxRequestBytes
|
||||||
cfg.maxArtifactBytes = settings.maxArtifactBytes
|
cfg.maxArtifactBytes = settings.maxArtifactBytes
|
||||||
cfg.maxResponseBytes = settings.maxResponseBytes
|
cfg.maxResponseBytes = settings.maxResponseBytes
|
||||||
|
cfg.backends = settings.backends
|
||||||
|
|
||||||
if err := validateRequiredLibraryDirs(cfg.promptDir); err != nil {
|
if err := validateRequiredLibraryDirs(cfg.promptDir); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@@ -349,7 +355,9 @@ func registerExecutionRequestFlags(fs *flag.FlagSet, cfg *runConfig) {
|
|||||||
fs.StringVar(&cfg.promptDir, "prompt-dir", "", "directory containing prompt definition YAML files")
|
fs.StringVar(&cfg.promptDir, "prompt-dir", "", "directory containing prompt definition YAML files")
|
||||||
fs.StringVar(&cfg.profileDir, "profile-dir", "", "directory containing execution profile YAML files")
|
fs.StringVar(&cfg.profileDir, "profile-dir", "", "directory containing execution profile YAML files")
|
||||||
fs.StringVar(&cfg.promptID, "prompt", "", "prompt ID to run")
|
fs.StringVar(&cfg.promptID, "prompt", "", "prompt ID to run")
|
||||||
|
fs.StringVar(&cfg.promptVersion, "prompt-version", "", "optional prompt definition version")
|
||||||
fs.StringVar(&cfg.profileID, "profile", "", "optional execution profile ID; if omitted, prompt default_profile is used")
|
fs.StringVar(&cfg.profileID, "profile", "", "optional execution profile ID; if omitted, prompt default_profile is used")
|
||||||
|
fs.StringVar(&cfg.sessionID, "session-id", "", "optional session ID")
|
||||||
fs.Var(&cfg.inputRaw, "input", "input mapping(s): name=path (repeatable, comma-separated)")
|
fs.Var(&cfg.inputRaw, "input", "input mapping(s): name=path (repeatable, comma-separated)")
|
||||||
fs.Var(&cfg.varRaw, "var", "variable mapping(s): name=value (repeatable, comma-separated)")
|
fs.Var(&cfg.varRaw, "var", "variable mapping(s): name=value (repeatable, comma-separated)")
|
||||||
fs.StringVar(&cfg.outputPath, "out", "", "optional output file path")
|
fs.StringVar(&cfg.outputPath, "out", "", "optional output file path")
|
||||||
@@ -359,6 +367,7 @@ func registerExecutionRequestFlags(fs *flag.FlagSet, cfg *runConfig) {
|
|||||||
fs.Float64Var(&cfg.temperature, "temperature", 0, "optional temperature override")
|
fs.Float64Var(&cfg.temperature, "temperature", 0, "optional temperature override")
|
||||||
fs.IntVar(&cfg.maxTokens, "max-tokens", 0, "optional max tokens override")
|
fs.IntVar(&cfg.maxTokens, "max-tokens", 0, "optional max tokens override")
|
||||||
fs.Float64Var(&cfg.topP, "top-p", 0, "optional top_p override")
|
fs.Float64Var(&cfg.topP, "top-p", 0, "optional top_p override")
|
||||||
|
fs.StringVar(&cfg.reasoningEffort, "reasoning-effort", "", "optional reasoning effort override")
|
||||||
fs.DurationVar(&cfg.timeout, "timeout", 0, "LLM request timeout")
|
fs.DurationVar(&cfg.timeout, "timeout", 0, "LLM request timeout")
|
||||||
fs.StringVar(&cfg.promptID, "prompt-id", "", "deprecated alias for --prompt")
|
fs.StringVar(&cfg.promptID, "prompt-id", "", "deprecated alias for --prompt")
|
||||||
fs.StringVar(&cfg.profileID, "profile-id", "", "deprecated alias for --profile")
|
fs.StringVar(&cfg.profileID, "profile-id", "", "deprecated alias for --profile")
|
||||||
@@ -382,6 +391,7 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
|
|||||||
cfg.profileDir = settings.profileDir
|
cfg.profileDir = settings.profileDir
|
||||||
cfg.schemaDir = settings.schemaDir
|
cfg.schemaDir = settings.schemaDir
|
||||||
cfg.defaultRenderFormat = settings.defaultRenderFormat
|
cfg.defaultRenderFormat = settings.defaultRenderFormat
|
||||||
|
cfg.backends = settings.backends
|
||||||
|
|
||||||
if err := validateRequiredLibraryDirs(cfg.promptDir); err != nil {
|
if err := validateRequiredLibraryDirs(cfg.promptDir); err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -389,9 +399,6 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
|
|||||||
if strings.TrimSpace(cfg.promptID) == "" {
|
if strings.TrimSpace(cfg.promptID) == "" {
|
||||||
return errors.New("--prompt is required")
|
return errors.New("--prompt is required")
|
||||||
}
|
}
|
||||||
if len(cfg.inputRaw) == 0 {
|
|
||||||
return errors.New("at least one --input is required")
|
|
||||||
}
|
|
||||||
cfg.promptDir = filepath.Clean(cfg.promptDir)
|
cfg.promptDir = filepath.Clean(cfg.promptDir)
|
||||||
if strings.TrimSpace(cfg.profileDir) != "" {
|
if strings.TrimSpace(cfg.profileDir) != "" {
|
||||||
cfg.profileDir = filepath.Clean(cfg.profileDir)
|
cfg.profileDir = filepath.Clean(cfg.profileDir)
|
||||||
@@ -405,6 +412,7 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
|
|||||||
cfg.temperatureSet = flagWasSet(fs, "temperature")
|
cfg.temperatureSet = flagWasSet(fs, "temperature")
|
||||||
cfg.maxTokensSet = flagWasSet(fs, "max-tokens")
|
cfg.maxTokensSet = flagWasSet(fs, "max-tokens")
|
||||||
cfg.topPSet = flagWasSet(fs, "top-p")
|
cfg.topPSet = flagWasSet(fs, "top-p")
|
||||||
|
cfg.reasoningEffortSet = flagWasSet(fs, "reasoning-effort")
|
||||||
cfg.timeoutSet = flagWasSet(fs, "timeout")
|
cfg.timeoutSet = flagWasSet(fs, "timeout")
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -500,7 +508,11 @@ func registerConfigPathFlag(fs *flag.FlagSet, target *string) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func resolveAppSettings(fs *flag.FlagSet, configPath string, overrides appconfig.CLIOverrides) (appconfig.AppSettings, error) {
|
func resolveAppSettings(fs *flag.FlagSet, configPath string, overrides appconfig.CLIOverrides) (appconfig.AppSettings, error) {
|
||||||
settings, err := appconfig.LoadConfig(configPath, flagWasSet(fs, "config"))
|
return resolveAppSettingsWithConfigPresence(configPath, flagWasSet(fs, "config"), overrides)
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveAppSettingsWithConfigPresence(configPath string, configExplicit bool, overrides appconfig.CLIOverrides) (appconfig.AppSettings, error) {
|
||||||
|
settings, err := appconfig.LoadConfig(configPath, configExplicit)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return appconfig.AppSettings{}, fmt.Errorf("application config error: %w", err)
|
return appconfig.AppSettings{}, fmt.Errorf("application config error: %w", err)
|
||||||
}
|
}
|
||||||
@@ -528,6 +540,7 @@ func resolveCommonSettings(fs *flag.FlagSet, configPath string, overrides appcon
|
|||||||
maxArtifactBytes: settings.MaxArtifactBytes,
|
maxArtifactBytes: settings.MaxArtifactBytes,
|
||||||
maxResponseBytes: settings.MaxResponseBytes,
|
maxResponseBytes: settings.MaxResponseBytes,
|
||||||
defaultRenderFormat: settings.DefaultRenderFormat,
|
defaultRenderFormat: settings.DefaultRenderFormat,
|
||||||
|
backends: settings.Backends,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -538,19 +551,54 @@ func validateRequiredLibraryDirs(promptDir string) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func newEngine(cfg *runConfig, options ...promptkit.Option) (*promptkit.Engine, error) {
|
type engineSettings struct {
|
||||||
return promptkit.NewEngine(promptkit.Config{
|
promptDir string
|
||||||
PromptDir: cfg.promptDir,
|
profileDir string
|
||||||
ProfileDir: cfg.profileDir,
|
schemaDir string
|
||||||
SchemaDir: cfg.schemaDir,
|
backends []appconfig.BackendSettings
|
||||||
}, options...)
|
}
|
||||||
|
|
||||||
|
func (c runConfig) engineSettings() engineSettings {
|
||||||
|
return engineSettings{promptDir: c.promptDir, profileDir: c.profileDir, schemaDir: c.schemaDir, backends: c.backends}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c serveConfig) engineSettings() engineSettings {
|
||||||
|
return engineSettings{promptDir: c.promptDir, profileDir: c.profileDir, schemaDir: c.schemaDir, backends: c.backends}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newEngine(settings engineSettings, options ...promptkit.Option) (*promptkit.Engine, error) {
|
||||||
|
engineOptions := make([]promptkit.Option, 0, len(settings.backends)+len(options))
|
||||||
|
for _, configured := range settings.backends {
|
||||||
|
engineOptions = append(engineOptions, promptkit.WithBackend(promptkit.Backend{
|
||||||
|
ID: configured.ID,
|
||||||
|
Endpoint: configured.Endpoint,
|
||||||
|
APIKeyEnv: configured.APIKeyEnv,
|
||||||
|
ExtraParams: configured.ExtraParams,
|
||||||
|
ConcurrencyLimit: configured.ConcurrencyLimit,
|
||||||
|
QueueCapacity: configured.QueueCapacity,
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
engineOptions = append(engineOptions, options...)
|
||||||
|
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||||
|
PromptDir: settings.promptDir,
|
||||||
|
ProfileDir: settings.profileDir,
|
||||||
|
SchemaDir: settings.schemaDir,
|
||||||
|
}, engineOptions...)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("engine initialization from application configuration: %w", err)
|
||||||
|
}
|
||||||
|
return engine, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func buildRunRequestFromConfig(cfg *runConfig) (promptkit.RunRequest, error) {
|
func buildRunRequestFromConfig(cfg *runConfig) (promptkit.RunRequest, error) {
|
||||||
inputMappings, err := parseMappings(cfg.inputRaw, false)
|
var inputMappings map[string]string
|
||||||
|
var err error
|
||||||
|
if len(cfg.inputRaw) > 0 {
|
||||||
|
inputMappings, err = parseMappings(cfg.inputRaw, false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return promptkit.RunRequest{}, fmt.Errorf("input parse error: %w", err)
|
return promptkit.RunRequest{}, fmt.Errorf("input parse error: %w", err)
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
varMappings := map[string]string{}
|
varMappings := map[string]string{}
|
||||||
if len(cfg.varRaw) > 0 {
|
if len(cfg.varRaw) > 0 {
|
||||||
@@ -560,13 +608,16 @@ func buildRunRequestFromConfig(cfg *runConfig) (promptkit.RunRequest, error) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
inputs := make(map[string]promptkit.ArtifactRef, len(inputMappings))
|
var inputs map[string]promptkit.ArtifactRef
|
||||||
|
if len(inputMappings) > 0 {
|
||||||
|
inputs = make(map[string]promptkit.ArtifactRef, len(inputMappings))
|
||||||
for name, path := range inputMappings {
|
for name, path := range inputMappings {
|
||||||
inputs[name] = promptkit.File(path)
|
inputs[name] = promptkit.File(path)
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
var modelOverride *promptkit.ExecutionTargetOverride
|
var modelOverride *promptkit.ExecutionTargetOverride
|
||||||
if cfg.llmBaseURLSet || cfg.modelSet || cfg.temperatureSet || cfg.maxTokensSet || cfg.topPSet || cfg.apiKeyEnvSet || cfg.timeoutSet {
|
if cfg.llmBaseURLSet || cfg.modelSet || cfg.temperatureSet || cfg.maxTokensSet || cfg.topPSet || cfg.reasoningEffortSet || cfg.apiKeyEnvSet || cfg.timeoutSet {
|
||||||
modelOverride = &promptkit.ExecutionTargetOverride{
|
modelOverride = &promptkit.ExecutionTargetOverride{
|
||||||
Endpoint: cfg.llmBaseURL,
|
Endpoint: cfg.llmBaseURL,
|
||||||
Model: cfg.model,
|
Model: cfg.model,
|
||||||
@@ -581,6 +632,9 @@ func buildRunRequestFromConfig(cfg *runConfig) (promptkit.RunRequest, error) {
|
|||||||
if cfg.topPSet {
|
if cfg.topPSet {
|
||||||
modelOverride.TopP = &cfg.topP
|
modelOverride.TopP = &cfg.topP
|
||||||
}
|
}
|
||||||
|
if cfg.reasoningEffortSet {
|
||||||
|
modelOverride.ReasoningEffort = &cfg.reasoningEffort
|
||||||
|
}
|
||||||
if cfg.timeoutSet {
|
if cfg.timeoutSet {
|
||||||
timeoutSeconds := int(cfg.timeout.Seconds())
|
timeoutSeconds := int(cfg.timeout.Seconds())
|
||||||
modelOverride.TimeoutSeconds = &timeoutSeconds
|
modelOverride.TimeoutSeconds = &timeoutSeconds
|
||||||
@@ -589,7 +643,9 @@ func buildRunRequestFromConfig(cfg *runConfig) (promptkit.RunRequest, error) {
|
|||||||
|
|
||||||
return promptkit.RunRequest{
|
return promptkit.RunRequest{
|
||||||
PromptID: cfg.promptID,
|
PromptID: cfg.promptID,
|
||||||
|
PromptVersion: cfg.promptVersion,
|
||||||
ProfileID: cfg.profileID,
|
ProfileID: cfg.profileID,
|
||||||
|
SessionID: cfg.sessionID,
|
||||||
Inputs: inputs,
|
Inputs: inputs,
|
||||||
Vars: varMappings,
|
Vars: varMappings,
|
||||||
Execution: modelOverride,
|
Execution: modelOverride,
|
||||||
@@ -682,12 +738,24 @@ func printSummary(stderr io.Writer, res *promptkit.RunResult) {
|
|||||||
if res.Usage.CachedTokens != 0 || res.Usage.CacheWriteTokens != 0 {
|
if res.Usage.CachedTokens != 0 || res.Usage.CacheWriteTokens != 0 {
|
||||||
fmt.Fprintf(stderr, " cached_tokens=%d cache_write_tokens=%d", res.Usage.CachedTokens, res.Usage.CacheWriteTokens)
|
fmt.Fprintf(stderr, " cached_tokens=%d cache_write_tokens=%d", res.Usage.CachedTokens, res.Usage.CacheWriteTokens)
|
||||||
}
|
}
|
||||||
|
if res.SelectedBackendID != "" {
|
||||||
|
fmt.Fprintf(stderr, " backend=%s", res.SelectedBackendID)
|
||||||
|
}
|
||||||
fmt.Fprintln(stderr)
|
fmt.Fprintln(stderr)
|
||||||
}
|
}
|
||||||
|
|
||||||
func printUsage(w io.Writer) {
|
func runErrorMessage(err error) string {
|
||||||
fmt.Fprintln(w, "usage: scriptorium <run|render|serve> ...")
|
if errors.Is(err, promptkit.ErrCapacityExceeded) {
|
||||||
fmt.Fprintln(w, " run: scriptorium run [--config PATH] [--prompt-dir DIR] [--profile-dir DIR] --prompt ID --input name=path [--input ...] [--profile ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--var k=v] [--out path] [--timeout 10m]")
|
return "run error: model backend capacity is exhausted"
|
||||||
fmt.Fprintln(w, " render: scriptorium render [--config PATH] [--prompt-dir DIR] [--profile-dir DIR] --prompt ID --input name=path [--input ...] [--profile ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--var k=v] [--format text|json] [--out path] [--timeout 10m]")
|
}
|
||||||
fmt.Fprintf(w, " serve: scriptorium serve [--config PATH] [--addr %s] [--prompt-dir DIR] [--profile-dir DIR] [--schema-dir DIR] [--artifact-root DIR] [--max-request-bytes N] [--max-artifact-bytes N] [--max-response-bytes N]\n", defaults.HTTPAddrDefault)
|
return fmt.Sprintf("run error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
func printUsage(w io.Writer) {
|
||||||
|
fmt.Fprintln(w, "usage: scriptorium <run|render|serve|inspect> ...")
|
||||||
|
fmt.Fprintln(w, " run: scriptorium run [--config PATH] [--prompt-dir DIR] [--profile-dir DIR] --prompt ID [--prompt-version VERSION] [--input name=path] [--profile ID] [--session-id ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--reasoning-effort VALUE] [--var k=v] [--out path] [--timeout 10m]")
|
||||||
|
fmt.Fprintln(w, " render: scriptorium render [--config PATH] [--prompt-dir DIR] [--profile-dir DIR] --prompt ID [--prompt-version VERSION] [--input name=path] [--profile ID] [--session-id ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--reasoning-effort VALUE] [--var k=v] [--format text|json] [--out path] [--timeout 10m]")
|
||||||
|
fmt.Fprintf(w, " serve: scriptorium serve [--config PATH] [--addr %s] [--prompt-dir DIR] [--profile-dir DIR] [--schema-dir DIR] [--artifact-root DIR] [--max-request-bytes N] [--max-artifact-bytes N] [--max-response-bytes N]\n", defaults.HTTPAddrDefault)
|
||||||
|
fmt.Fprintln(w, " inspect prompt: scriptorium inspect prompt --prompt ID [--prompt-version VERSION] [--config PATH] [--prompt-dir DIR] [--format text|json] [--out path]")
|
||||||
|
fmt.Fprintln(w, " inspect profile: scriptorium inspect profile --profile ID [--config PATH] [--profile-dir DIR] [--format text|json] [--out path]")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -87,9 +87,12 @@ func TestParseRunArgsRequiredFlags(t *testing.T) {
|
|||||||
t.Fatal("expected missing --prompt error")
|
t.Fatal("expected missing --prompt error")
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err = parseRunArgs([]string{"--config", configPath, "--prompt-dir", "./prompts", "--profile-dir", "./profiles", "--prompt", "p"})
|
cfg, err = parseRunArgs([]string{"--config", configPath, "--prompt-dir", "./prompts", "--profile-dir", "./profiles", "--prompt", "p"})
|
||||||
if err == nil {
|
if err != nil {
|
||||||
t.Fatal("expected missing --input error")
|
t.Fatalf("expected omitted --input to be accepted, got %v", err)
|
||||||
|
}
|
||||||
|
if len(cfg.inputRaw) != 0 {
|
||||||
|
t.Fatalf("expected no input mappings, got %#v", cfg.inputRaw)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -98,13 +101,16 @@ func TestParseRunArgsFlagMapping(t *testing.T) {
|
|||||||
"--prompt-dir", "./prompts",
|
"--prompt-dir", "./prompts",
|
||||||
"--profile-dir", "./profiles",
|
"--profile-dir", "./profiles",
|
||||||
"--prompt", "prompt.a",
|
"--prompt", "prompt.a",
|
||||||
|
"--prompt-version", "2",
|
||||||
"--profile", "profile.a",
|
"--profile", "profile.a",
|
||||||
|
"--session-id", "session-1",
|
||||||
"--input", "a=b",
|
"--input", "a=b",
|
||||||
"--llm-base-url", "http://x/v1",
|
"--llm-base-url", "http://x/v1",
|
||||||
"--model", "m",
|
"--model", "m",
|
||||||
"--temperature", "0.7",
|
"--temperature", "0.7",
|
||||||
"--max-tokens", "111",
|
"--max-tokens", "111",
|
||||||
"--top-p", "0.8",
|
"--top-p", "0.8",
|
||||||
|
"--reasoning-effort", "medium",
|
||||||
"--timeout", "30s",
|
"--timeout", "30s",
|
||||||
"--api-key-env", "SCRIPTORIUM_API_KEY",
|
"--api-key-env", "SCRIPTORIUM_API_KEY",
|
||||||
})
|
})
|
||||||
@@ -114,8 +120,11 @@ func TestParseRunArgsFlagMapping(t *testing.T) {
|
|||||||
if cfg.promptDir != filepath.Clean("./prompts") || cfg.profileDir != filepath.Clean("./profiles") {
|
if cfg.promptDir != filepath.Clean("./prompts") || cfg.profileDir != filepath.Clean("./profiles") {
|
||||||
t.Fatalf("unexpected dirs: prompt=%q profile=%q", cfg.promptDir, cfg.profileDir)
|
t.Fatalf("unexpected dirs: prompt=%q profile=%q", cfg.promptDir, cfg.profileDir)
|
||||||
}
|
}
|
||||||
if cfg.promptID != "prompt.a" || cfg.profileID != "profile.a" {
|
if cfg.promptID != "prompt.a" || cfg.promptVersion != "2" || cfg.profileID != "profile.a" {
|
||||||
t.Fatalf("unexpected prompt/profile ids: %q %q", cfg.promptID, cfg.profileID)
|
t.Fatalf("unexpected prompt/version/profile ids: %q %q %q", cfg.promptID, cfg.promptVersion, cfg.profileID)
|
||||||
|
}
|
||||||
|
if cfg.sessionID != "session-1" || cfg.reasoningEffort != "medium" || !cfg.reasoningEffortSet {
|
||||||
|
t.Fatalf("unexpected session or reasoning configuration: %+v", cfg)
|
||||||
}
|
}
|
||||||
if !cfg.llmBaseURLSet || !cfg.modelSet || !cfg.temperatureSet || !cfg.maxTokensSet || !cfg.topPSet || !cfg.timeoutSet || !cfg.apiKeyEnvSet {
|
if !cfg.llmBaseURLSet || !cfg.modelSet || !cfg.temperatureSet || !cfg.maxTokensSet || !cfg.topPSet || !cfg.timeoutSet || !cfg.apiKeyEnvSet {
|
||||||
t.Fatalf("expected override flags set, got %+v", cfg)
|
t.Fatalf("expected override flags set, got %+v", cfg)
|
||||||
@@ -192,7 +201,7 @@ func TestParseServeArgsRejectsRuntimeOverrideFlags(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestUsageIncludesServeFileAndSizeLimitFlags(t *testing.T) {
|
func TestUsageIncludesExecutionAndServeFlags(t *testing.T) {
|
||||||
var stderr bytes.Buffer
|
var stderr bytes.Buffer
|
||||||
code := Run(nil, io.Discard, &stderr)
|
code := Run(nil, io.Discard, &stderr)
|
||||||
if code != ExitRuntimeError {
|
if code != ExitRuntimeError {
|
||||||
@@ -201,10 +210,15 @@ func TestUsageIncludesServeFileAndSizeLimitFlags(t *testing.T) {
|
|||||||
|
|
||||||
usage := stderr.String()
|
usage := stderr.String()
|
||||||
for _, want := range []string{
|
for _, want := range []string{
|
||||||
|
"--prompt-version VERSION",
|
||||||
|
"--session-id ID",
|
||||||
|
"--reasoning-effort VALUE",
|
||||||
"--artifact-root",
|
"--artifact-root",
|
||||||
"--max-request-bytes",
|
"--max-request-bytes",
|
||||||
"--max-artifact-bytes",
|
"--max-artifact-bytes",
|
||||||
"--max-response-bytes",
|
"--max-response-bytes",
|
||||||
|
"inspect prompt",
|
||||||
|
"inspect profile",
|
||||||
} {
|
} {
|
||||||
if !strings.Contains(usage, want) {
|
if !strings.Contains(usage, want) {
|
||||||
t.Fatalf("expected usage to include %q, got %q", want, usage)
|
t.Fatalf("expected usage to include %q, got %q", want, usage)
|
||||||
@@ -689,6 +703,55 @@ func TestBuildRunRequestPreservesNumericOverridePresence(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestBuildRunRequestPreservesReasoningEffortPresenceAndSessionID(t *testing.T) {
|
||||||
|
omitted, err := buildRunRequestFromConfig(&runConfig{promptID: "prompt-1"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected omitted request to build, got %v", err)
|
||||||
|
}
|
||||||
|
if omitted.Execution != nil {
|
||||||
|
t.Fatalf("expected omitted reasoning flag to leave execution nil, got %#v", omitted.Execution)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
value string
|
||||||
|
}{
|
||||||
|
{name: "replacement", value: "high"},
|
||||||
|
{name: "clear", value: ""},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
req, err := buildRunRequestFromConfig(&runConfig{
|
||||||
|
promptID: "prompt-1",
|
||||||
|
sessionID: "session-1",
|
||||||
|
reasoningEffort: tc.value,
|
||||||
|
reasoningEffortSet: true,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected request to build, got %v", err)
|
||||||
|
}
|
||||||
|
if req.SessionID != "session-1" || req.Execution == nil || req.Execution.ReasoningEffort == nil || *req.Execution.ReasoningEffort != tc.value {
|
||||||
|
t.Fatalf("unexpected mapped request: %#v", req)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildRunRequestAllowsOmittedInputsAndMapsPromptVersion(t *testing.T) {
|
||||||
|
req, err := buildRunRequestFromConfig(&runConfig{
|
||||||
|
promptID: "prompt-1",
|
||||||
|
promptVersion: "2",
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected request without inputs to build, got %v", err)
|
||||||
|
}
|
||||||
|
if req.PromptVersion != "2" {
|
||||||
|
t.Fatalf("expected prompt version to be mapped, got %q", req.PromptVersion)
|
||||||
|
}
|
||||||
|
if req.Inputs != nil {
|
||||||
|
t.Fatalf("expected omitted inputs to remain nil, got %#v", req.Inputs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestParseRunArgsFailsClearlyWhenNoEffectivePromptDir(t *testing.T) {
|
func TestParseRunArgsFailsClearlyWhenNoEffectivePromptDir(t *testing.T) {
|
||||||
configPath := writeAppConfigFile(t, `
|
configPath := writeAppConfigFile(t, `
|
||||||
profile_dir: ./profiles
|
profile_dir: ./profiles
|
||||||
@@ -942,6 +1005,452 @@ profile_dir: %s
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRenderCommandUsesConfiguredCustomBackend(t *testing.T) {
|
||||||
|
lib := newCLITestLibrary(t)
|
||||||
|
writePromptDefinition(t, lib.promptDir, "custom.yaml", `id: custom
|
||||||
|
version: "1"
|
||||||
|
default_profile: local-gpu
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "hello"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`)
|
||||||
|
if err := os.WriteFile(filepath.Join(lib.profileDir, "local-gpu.yaml"), []byte(`id: local-gpu
|
||||||
|
backend: local-gpu
|
||||||
|
model: local-model
|
||||||
|
`), 0o644); err != nil {
|
||||||
|
t.Fatalf("write profile fixture: %v", err)
|
||||||
|
}
|
||||||
|
configPath := writeAppConfigFile(t, fmt.Sprintf(`
|
||||||
|
prompt_dir: %s
|
||||||
|
profile_dir: %s
|
||||||
|
backends:
|
||||||
|
local-gpu:
|
||||||
|
endpoint: http://localhost:11434/v1
|
||||||
|
extra_params:
|
||||||
|
provider_option: enabled
|
||||||
|
concurrency_limit: 2
|
||||||
|
queue_capacity: 0
|
||||||
|
`, lib.promptDir, lib.profileDir))
|
||||||
|
|
||||||
|
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||||
|
"--config", configPath,
|
||||||
|
"--prompt", "custom",
|
||||||
|
})
|
||||||
|
if code != ExitOK {
|
||||||
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout, "selected_backend_id: local-gpu") {
|
||||||
|
t.Fatalf("expected configured backend in prepared output, got:\n%s", stdout)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRenderCommandMapsReasoningEffortAndSessionID(t *testing.T) {
|
||||||
|
lib := newCLITestLibrary(t)
|
||||||
|
writePromptDefinition(t, lib.promptDir, "session.yaml", `id: session
|
||||||
|
version: "1"
|
||||||
|
default_profile: local
|
||||||
|
session_id: definition-session
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "hello"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`)
|
||||||
|
if err := os.WriteFile(filepath.Join(lib.profileDir, "local.yaml"), []byte(`id: local
|
||||||
|
endpoint: http://127.0.0.1:1/v1
|
||||||
|
model: local-model
|
||||||
|
reasoning_effort: low
|
||||||
|
`), 0o644); err != nil {
|
||||||
|
t.Fatalf("write profile fixture: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
args []string
|
||||||
|
wantReasoning string
|
||||||
|
wantSessionID string
|
||||||
|
absentReasoning bool
|
||||||
|
}{
|
||||||
|
{name: "omitted reasoning inherits profile", wantReasoning: "low", wantSessionID: "definition-session"},
|
||||||
|
{name: "nonblank reasoning replaces profile", args: []string{"--reasoning-effort", "high"}, wantReasoning: "high", wantSessionID: "definition-session"},
|
||||||
|
{name: "empty reasoning clears profile", args: []string{"--reasoning-effort="}, wantSessionID: "definition-session", absentReasoning: true},
|
||||||
|
{name: "direct session replaces definition", args: []string{"--session-id", "direct-session"}, wantReasoning: "low", wantSessionID: "direct-session"},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
args := []string{"--prompt-dir", lib.promptDir, "--profile-dir", lib.profileDir, "--prompt", "session"}
|
||||||
|
args = append(args, tc.args...)
|
||||||
|
code, stdout, stderr := runCLICommand(t, renderCommand, args)
|
||||||
|
if code != ExitOK {
|
||||||
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout, "session_id: "+tc.wantSessionID) {
|
||||||
|
t.Fatalf("expected session ID %q, got:\n%s", tc.wantSessionID, stdout)
|
||||||
|
}
|
||||||
|
hasReasoning := strings.Contains(stdout, "reasoning_effort:")
|
||||||
|
if tc.absentReasoning {
|
||||||
|
if hasReasoning {
|
||||||
|
t.Fatalf("expected cleared reasoning to be omitted, got:\n%s", stdout)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout, "reasoning_effort: "+tc.wantReasoning) {
|
||||||
|
t.Fatalf("expected reasoning effort %q, got:\n%s", tc.wantReasoning, stdout)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRenderCommandOmitsEmptyEffectiveSessionID(t *testing.T) {
|
||||||
|
lib := newCLITestLibrary(t)
|
||||||
|
writePromptDefinition(t, lib.promptDir, "plain.yaml", `id: plain
|
||||||
|
version: "1"
|
||||||
|
default_profile: local
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "hello"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`)
|
||||||
|
writeProfileFile(t, lib.profileDir, "local", "http://127.0.0.1:1/v1", "local-model")
|
||||||
|
|
||||||
|
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||||
|
"--prompt-dir", lib.promptDir,
|
||||||
|
"--profile-dir", lib.profileDir,
|
||||||
|
"--prompt", "plain",
|
||||||
|
})
|
||||||
|
if code != ExitOK {
|
||||||
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
|
}
|
||||||
|
if strings.Contains(stdout, "session_id:") {
|
||||||
|
t.Fatalf("expected no effective session ID, got:\n%s", stdout)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRenderCommandRejectsOverlongSessionID(t *testing.T) {
|
||||||
|
lib := newCLITestLibrary(t)
|
||||||
|
writePromptDefinition(t, lib.promptDir, "session.yaml", `id: session
|
||||||
|
version: "1"
|
||||||
|
default_profile: local
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "hello"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`)
|
||||||
|
writeProfileFile(t, lib.profileDir, "local", "http://127.0.0.1:1/v1", "local-model")
|
||||||
|
|
||||||
|
code, _, stderr := runCLICommand(t, renderCommand, []string{
|
||||||
|
"--prompt-dir", lib.promptDir,
|
||||||
|
"--profile-dir", lib.profileDir,
|
||||||
|
"--prompt", "session",
|
||||||
|
"--session-id", strings.Repeat("x", 257),
|
||||||
|
})
|
||||||
|
if code != ExitRuntimeError {
|
||||||
|
t.Fatalf("expected runtime error, got %d stderr=%q", code, stderr)
|
||||||
|
}
|
||||||
|
if !strings.Contains(stderr, "invalid run request") {
|
||||||
|
t.Fatalf("expected invalid-request context, got %q", stderr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestInspectPromptCommandFormatsDefinitionWithoutProfileOrGeneration(t *testing.T) {
|
||||||
|
lib := newCLITestLibrary(t)
|
||||||
|
writePromptDefinition(t, lib.promptDir, "inspect.yaml", `id: inspect
|
||||||
|
version: "1"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "hello"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`)
|
||||||
|
code, stdout, stderr := runCLICommand(t, inspectCommand, []string{"prompt", "--prompt-dir", lib.promptDir, "--prompt", "inspect", "--format", "json"})
|
||||||
|
if code != ExitOK {
|
||||||
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout, `"prompt_id": "inspect"`) || !strings.Contains(stdout, `"inputs": []`) {
|
||||||
|
t.Fatalf("unexpected inspection output: %s", stdout)
|
||||||
|
}
|
||||||
|
code, stdout, stderr = runCLICommand(t, inspectCommand, []string{"prompt", "--prompt-dir", lib.promptDir, "--prompt", "missing"})
|
||||||
|
if code != ExitRuntimeError || stdout != "" || !strings.Contains(stderr, "inspect error") {
|
||||||
|
t.Fatalf("expected failed inspection without output, got code=%d stdout=%q stderr=%q", code, stdout, stderr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestInspectionParsersPreserveExplicitEmptyConfigPath(t *testing.T) {
|
||||||
|
promptConfig, err := parsePromptInspectionArgs([]string{"--config=", "--prompt", "fixture"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse prompt inspection: %v", err)
|
||||||
|
}
|
||||||
|
if promptConfig.configPath != "" || !promptConfig.configExplicit {
|
||||||
|
t.Fatalf("expected explicit empty prompt config path, got %+v", promptConfig)
|
||||||
|
}
|
||||||
|
|
||||||
|
profileConfig, err := parseProfileInspectionArgs([]string{"--config=", "--profile", "fixture"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse profile inspection: %v", err)
|
||||||
|
}
|
||||||
|
if profileConfig.configPath != "" || !profileConfig.configExplicit {
|
||||||
|
t.Fatalf("expected explicit empty profile config path, got %+v", profileConfig)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptkitV09DefinitionsRenderThroughCLI(t *testing.T) {
|
||||||
|
fixtureRoot := promptkitV09FixtureRoot(t)
|
||||||
|
configPath := writePromptkitV09Config(t, fixtureRoot, true)
|
||||||
|
inputPath := filepath.Join(fixtureRoot, "inputs", "source.md")
|
||||||
|
|
||||||
|
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||||
|
"--config", configPath,
|
||||||
|
"--prompt", "compat.complete",
|
||||||
|
"--prompt-version", "1.0.0",
|
||||||
|
"--input", "source=" + inputPath,
|
||||||
|
"--var", "topic=testing",
|
||||||
|
"--format", "json",
|
||||||
|
})
|
||||||
|
if code != ExitOK {
|
||||||
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
|
}
|
||||||
|
|
||||||
|
var payload map[string]any
|
||||||
|
if err := json.Unmarshal([]byte(stdout), &payload); err != nil {
|
||||||
|
t.Fatalf("decode rendered fixture: %v\nbody=%s", err, stdout)
|
||||||
|
}
|
||||||
|
if payload["prompt_version"] != "1.0.0" || payload["selected_profile_id"] != "custom-derived" || payload["selected_backend_id"] != "fixture-custom" {
|
||||||
|
t.Fatalf("unexpected selected definition and target: %#v", payload)
|
||||||
|
}
|
||||||
|
if payload["session_id"] != "fixture-testing" {
|
||||||
|
t.Fatalf("expected rendered session template, got %#v", payload["session_id"])
|
||||||
|
}
|
||||||
|
|
||||||
|
outputContract := payload["output_contract"].(map[string]any)
|
||||||
|
if outputContract["format"] != "json" || outputContract["validation_mode"] != "json_schema" || outputContract["repair_attempts"] != float64(2) {
|
||||||
|
t.Fatalf("unexpected output contract: %#v", outputContract)
|
||||||
|
}
|
||||||
|
if _, ok := payload["structured_output"].(map[string]any)["json_schema"]; !ok {
|
||||||
|
t.Fatalf("expected loaded JSON schema metadata, got %#v", payload["structured_output"])
|
||||||
|
}
|
||||||
|
|
||||||
|
messages := payload["messages"].([]any)
|
||||||
|
wantRoles := []string{"developer", "system", "user", "assistant"}
|
||||||
|
for i, wantRole := range wantRoles {
|
||||||
|
message := messages[i].(map[string]any)
|
||||||
|
if message["role"] != wantRole {
|
||||||
|
t.Fatalf("message %d: expected role %q, got %#v", i, wantRole, message["role"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
cacheControl := messages[0].(map[string]any)["cache_control"].(map[string]any)
|
||||||
|
if cacheControl["type"] != "ephemeral" || cacheControl["ttl"] != "1h" {
|
||||||
|
t.Fatalf("unexpected cache control: %#v", cacheControl)
|
||||||
|
}
|
||||||
|
if !strings.Contains(messages[2].(map[string]any)["content"].(string), "stable, synthetic material") {
|
||||||
|
t.Fatalf("file-backed input template was not rendered: %#v", messages[2])
|
||||||
|
}
|
||||||
|
inputHashes := payload["input_hashes"].(map[string]any)
|
||||||
|
if len(inputHashes) != 1 || inputHashes["source"] == "" {
|
||||||
|
t.Fatalf("required and omitted optional inputs were not preserved: %#v", inputHashes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptkitV09PromptInspectionSelectsVersionsAndContracts(t *testing.T) {
|
||||||
|
fixtureRoot := promptkitV09FixtureRoot(t)
|
||||||
|
configPath := writePromptkitV09Config(t, fixtureRoot, true)
|
||||||
|
tests := []struct {
|
||||||
|
promptID string
|
||||||
|
version string
|
||||||
|
format string
|
||||||
|
mode string
|
||||||
|
}{
|
||||||
|
{promptID: "compat.complete", version: "1.0.0", format: "json", mode: "json_schema"},
|
||||||
|
{promptID: "compat.complete", version: "2.0.0", format: "text", mode: "basic"},
|
||||||
|
{promptID: "compat.json", version: "1.0.0", format: "json", mode: "json"},
|
||||||
|
{promptID: "compat.none", version: "1.0.0", format: "text", mode: "none"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.promptID+"@"+tc.version, func(t *testing.T) {
|
||||||
|
code, stdout, stderr := runCLICommand(t, inspectCommand, []string{
|
||||||
|
"prompt",
|
||||||
|
"--config", configPath,
|
||||||
|
"--prompt", tc.promptID,
|
||||||
|
"--prompt-version", tc.version,
|
||||||
|
"--format", "json",
|
||||||
|
})
|
||||||
|
if code != ExitOK {
|
||||||
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
|
}
|
||||||
|
var inspection renderformat.PromptInspection
|
||||||
|
if err := json.Unmarshal([]byte(stdout), &inspection); err != nil {
|
||||||
|
t.Fatalf("decode prompt inspection: %v\nbody=%s", err, stdout)
|
||||||
|
}
|
||||||
|
if inspection.PromptVersion != tc.version || inspection.OutputContract.Format != tc.format || inspection.OutputContract.ValidationMode != tc.mode {
|
||||||
|
t.Fatalf("unexpected prompt inspection: %+v", inspection)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptkitV09ProfileInspectionResolvesSupportedTargets(t *testing.T) {
|
||||||
|
const secret = "sentinel-profile-secret"
|
||||||
|
t.Setenv("FIXTURE_PROFILE_API_KEY", secret)
|
||||||
|
|
||||||
|
fixtureRoot := promptkitV09FixtureRoot(t)
|
||||||
|
configPath := writePromptkitV09Config(t, fixtureRoot, true)
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
profileID string
|
||||||
|
wantBackend string
|
||||||
|
wantModel string
|
||||||
|
wantAPIKeyEnv string
|
||||||
|
}{
|
||||||
|
{name: "inherited custom backend", profileID: "custom-derived", wantBackend: "fixture-custom", wantModel: "fixture-derived-model", wantAPIKeyEnv: "FIXTURE_PROFILE_API_KEY"},
|
||||||
|
{name: "endpoint only", profileID: "endpoint-only", wantModel: "fixture-endpoint-model"},
|
||||||
|
{name: "built in", profileID: "deepseek-4-flash", wantBackend: "openrouter", wantModel: "deepseek/deepseek-v4-flash", wantAPIKeyEnv: "OPENROUTER_API_KEY"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
args := []string{"profile", "--config", configPath, "--profile", tc.profileID, "--format", "json"}
|
||||||
|
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
|
||||||
|
if code != ExitOK {
|
||||||
|
t.Fatalf("expected ExitOK, got %d stderr=%q", code, stderr)
|
||||||
|
}
|
||||||
|
if strings.Contains(stdout, secret) || strings.Contains(stderr, secret) {
|
||||||
|
t.Fatalf("inspection exposed environment secret: stdout=%q stderr=%q", stdout, stderr)
|
||||||
|
}
|
||||||
|
var inspection renderformat.ProfileInspection
|
||||||
|
if err := json.Unmarshal([]byte(stdout), &inspection); err != nil {
|
||||||
|
t.Fatalf("decode profile inspection: %v\nbody=%s", err, stdout)
|
||||||
|
}
|
||||||
|
params := inspection.EffectiveModelParams
|
||||||
|
if inspection.ProfileID != tc.profileID || params.BackendID != tc.wantBackend || params.Model != tc.wantModel || params.APIKeyEnv != tc.wantAPIKeyEnv {
|
||||||
|
t.Fatalf("unexpected effective profile: %+v", inspection)
|
||||||
|
}
|
||||||
|
if tc.profileID == "custom-derived" {
|
||||||
|
if params.ServiceTier != "flex" || params.ReasoningEffort != "high" || params.TimeoutSeconds != 45 || len(params.ExtraParams) == 0 {
|
||||||
|
t.Fatalf("inherited profile controls were not resolved: %+v", params)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
configWithoutPromptDir := writePromptkitV09Config(t, fixtureRoot, false)
|
||||||
|
code, _, stderr := runCLICommand(t, inspectCommand, []string{
|
||||||
|
"profile",
|
||||||
|
"--config", configWithoutPromptDir,
|
||||||
|
"--profile-dir", filepath.Join(fixtureRoot, "profiles"),
|
||||||
|
"--profile", "custom-derived",
|
||||||
|
})
|
||||||
|
if code != ExitOK {
|
||||||
|
t.Fatalf("profile inspection unexpectedly required a prompt directory: %q", stderr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProfileInspectionHonorsDirectoryPrecedenceOutputAndFailures(t *testing.T) {
|
||||||
|
fixtureRoot := promptkitV09FixtureRoot(t)
|
||||||
|
configPath := writePromptkitV09Config(t, fixtureRoot, true)
|
||||||
|
overrideDir := t.TempDir()
|
||||||
|
writeProfileFile(t, overrideDir, "custom-derived", "http://127.0.0.1:9000/v1", "override-model")
|
||||||
|
outPath := filepath.Join(t.TempDir(), "inspection.json")
|
||||||
|
|
||||||
|
code, stdout, stderr := runCLICommand(t, inspectCommand, []string{
|
||||||
|
"profile",
|
||||||
|
"--config", configPath,
|
||||||
|
"--profile-dir", overrideDir,
|
||||||
|
"--profile", "custom-derived",
|
||||||
|
"--format", "json",
|
||||||
|
"--out", outPath,
|
||||||
|
})
|
||||||
|
if code != ExitOK || stdout != "" {
|
||||||
|
t.Fatalf("expected file output, got code=%d stdout=%q stderr=%q", code, stdout, stderr)
|
||||||
|
}
|
||||||
|
output, err := os.ReadFile(outPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read profile inspection output: %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(output), `"model": "override-model"`) || !strings.Contains(string(output), `"backend_id": ""`) {
|
||||||
|
t.Fatalf("profile directory override was not used: %s", output)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, profileID := range []string{"missing", "invalid"} {
|
||||||
|
t.Run(profileID, func(t *testing.T) {
|
||||||
|
profileDir := filepath.Join(fixtureRoot, "profiles")
|
||||||
|
if profileID == "invalid" {
|
||||||
|
profileDir = t.TempDir()
|
||||||
|
writePromptDefinition(t, profileDir, "invalid.yaml", "id: invalid\nendpoint: not-a-url\nmodel: fixture\n")
|
||||||
|
}
|
||||||
|
code, stdout, stderr := runCLICommand(t, inspectCommand, []string{
|
||||||
|
"profile",
|
||||||
|
"--config", configPath,
|
||||||
|
"--profile-dir", profileDir,
|
||||||
|
"--profile", profileID,
|
||||||
|
})
|
||||||
|
if code != ExitRuntimeError || stdout != "" || !strings.Contains(stderr, "inspect error") {
|
||||||
|
t.Fatalf("expected inspection failure without partial output, got code=%d stdout=%q stderr=%q", code, stdout, stderr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConfiguredBackendValidationComesFromPromptkit(t *testing.T) {
|
||||||
|
lib := newCLITestLibrary(t)
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
backend string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "reserved ID",
|
||||||
|
backend: `openrouter:
|
||||||
|
endpoint: http://localhost:11434/v1`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "invalid endpoint",
|
||||||
|
backend: `local-gpu:
|
||||||
|
endpoint: not-a-url`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "invalid capacity relationship",
|
||||||
|
backend: `local-gpu:
|
||||||
|
endpoint: http://localhost:11434/v1
|
||||||
|
queue_capacity: 0`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "reserved extra parameter",
|
||||||
|
backend: `local-gpu:
|
||||||
|
endpoint: http://localhost:11434/v1
|
||||||
|
extra_params:
|
||||||
|
model: forbidden`,
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
configPath := writeAppConfigFile(t, fmt.Sprintf(`
|
||||||
|
prompt_dir: %s
|
||||||
|
profile_dir: %s
|
||||||
|
backends:
|
||||||
|
%s
|
||||||
|
`, lib.promptDir, lib.profileDir, tc.backend))
|
||||||
|
cfg, err := parseRenderArgs([]string{"--config", configPath, "--prompt", "custom"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected config decoding to succeed, got %v", err)
|
||||||
|
}
|
||||||
|
_, err = newEngine(cfg.runConfig.engineSettings())
|
||||||
|
if !errors.Is(err, promptkit.ErrInvalidConfig) {
|
||||||
|
t.Fatalf("expected Promptkit ErrInvalidConfig, got %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "engine initialization from application configuration") {
|
||||||
|
t.Fatalf("expected application context, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRenderCommandExplicitTextFormatWorks(t *testing.T) {
|
func TestRenderCommandExplicitTextFormatWorks(t *testing.T) {
|
||||||
lib := newCLITestLibrary(t)
|
lib := newCLITestLibrary(t)
|
||||||
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
|
inputPath := lib.writeInputFile(t, "transcript.md", "hello transcript")
|
||||||
@@ -1122,6 +1631,104 @@ func TestRenderCommandExplicitProfileOverridesPromptDefault(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRenderCommandUsesDefinitionInputRulesAndPromptVersions(t *testing.T) {
|
||||||
|
lib := newCLITestLibrary(t)
|
||||||
|
writeProfileFile(t, lib.profileDir, "local", "http://127.0.0.1:1/v1", "model")
|
||||||
|
|
||||||
|
writePromptDefinition(t, lib.promptDir, "sole.yaml", `id: sole
|
||||||
|
version: "1"
|
||||||
|
default_profile: local
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "hello"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`)
|
||||||
|
writePromptDefinition(t, lib.promptDir, "versioned-one.yaml", `id: versioned
|
||||||
|
version: "1"
|
||||||
|
default_profile: local
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "one"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`)
|
||||||
|
writePromptDefinition(t, lib.promptDir, "versioned-two.yaml", `id: versioned
|
||||||
|
version: "2"
|
||||||
|
default_profile: local
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "two"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`)
|
||||||
|
writePromptDefinition(t, lib.promptDir, "optional.yaml", `id: optional
|
||||||
|
version: "1"
|
||||||
|
default_profile: local
|
||||||
|
inputs:
|
||||||
|
- name: note
|
||||||
|
required: false
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "hello"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`)
|
||||||
|
writePromptDefinition(t, lib.promptDir, "required.yaml", `id: required
|
||||||
|
version: "1"
|
||||||
|
default_profile: local
|
||||||
|
inputs:
|
||||||
|
- name: note
|
||||||
|
required: true
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "hello"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`)
|
||||||
|
writePromptDefinition(t, lib.promptDir, "template.yaml", `id: template
|
||||||
|
version: "1"
|
||||||
|
default_profile: local
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: '{{input "note"}}'
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`)
|
||||||
|
|
||||||
|
baseArgs := []string{"--prompt-dir", lib.promptDir, "--profile-dir", lib.profileDir}
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
args []string
|
||||||
|
wantCode int
|
||||||
|
wantText string
|
||||||
|
}{
|
||||||
|
{name: "sole version selected when omitted", args: []string{"--prompt", "sole"}, wantCode: ExitOK, wantText: "prompt_version: 1"},
|
||||||
|
{name: "explicit version selected", args: []string{"--prompt", "versioned", "--prompt-version", "2"}, wantCode: ExitOK, wantText: "prompt_version: 2"},
|
||||||
|
{name: "multiple versions require selection", args: []string{"--prompt", "versioned"}, wantCode: ExitRuntimeError, wantText: "duplicate prompt definition id"},
|
||||||
|
{name: "no declared inputs", args: []string{"--prompt", "sole"}, wantCode: ExitOK, wantText: "prompt: sole"},
|
||||||
|
{name: "optional input omitted", args: []string{"--prompt", "optional"}, wantCode: ExitOK, wantText: "prompt: optional"},
|
||||||
|
{name: "required input omitted", args: []string{"--prompt", "required"}, wantCode: ExitRuntimeError, wantText: "required"},
|
||||||
|
{name: "template input omitted", args: []string{"--prompt", "template"}, wantCode: ExitRuntimeError, wantText: "note"},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
code, stdout, stderr := runCLICommand(t, renderCommand, append(append([]string{}, baseArgs...), tc.args...))
|
||||||
|
if code != tc.wantCode {
|
||||||
|
t.Fatalf("expected exit %d, got %d stderr=%q", tc.wantCode, code, stderr)
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout+stderr, tc.wantText) {
|
||||||
|
t.Fatalf("expected output to contain %q, stdout=%q stderr=%q", tc.wantText, stdout, stderr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
|
func TestRunCommandPromptDefaultProfileWorksThroughCLIPath(t *testing.T) {
|
||||||
lib := newCLITestLibrary(t)
|
lib := newCLITestLibrary(t)
|
||||||
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
|
inputPath := lib.writeInputFile(t, "transcript.md", "hello")
|
||||||
@@ -1261,6 +1868,9 @@ func TestWriteOutputAndSummaryUseSeparateWriters(t *testing.T) {
|
|||||||
if strings.Contains(stderr.String(), "cached_tokens=") || strings.Contains(stderr.String(), "cache_write_tokens=") {
|
if strings.Contains(stderr.String(), "cached_tokens=") || strings.Contains(stderr.String(), "cache_write_tokens=") {
|
||||||
t.Fatalf("expected zero cache usage to be omitted from summary, got %q", stderr.String())
|
t.Fatalf("expected zero cache usage to be omitted from summary, got %q", stderr.String())
|
||||||
}
|
}
|
||||||
|
if strings.Contains(stderr.String(), "backend=") {
|
||||||
|
t.Fatalf("expected endpoint-only backend to be omitted from summary, got %q", stderr.String())
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPrintSummaryIncludesCacheUsageWhenPresent(t *testing.T) {
|
func TestPrintSummaryIncludesCacheUsageWhenPresent(t *testing.T) {
|
||||||
@@ -1290,6 +1900,32 @@ func TestPrintSummaryIncludesCacheUsageWhenPresent(t *testing.T) {
|
|||||||
if !strings.Contains(summary, "cached_tokens=0 cache_write_tokens=3") {
|
if !strings.Contains(summary, "cached_tokens=0 cache_write_tokens=3") {
|
||||||
t.Fatalf("expected cache usage in summary, got %q", summary)
|
t.Fatalf("expected cache usage in summary, got %q", summary)
|
||||||
}
|
}
|
||||||
|
if strings.Contains(summary, "backend=") {
|
||||||
|
t.Fatalf("expected endpoint-only backend to be omitted from summary, got %q", summary)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPrintSummaryIncludesBackendWhenPresent(t *testing.T) {
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
printSummary(&stderr, &promptkit.RunResult{
|
||||||
|
PromptID: "p",
|
||||||
|
PromptVersion: "1",
|
||||||
|
SelectedProfileID: "exec",
|
||||||
|
SelectedBackendID: "local",
|
||||||
|
ModelName: "m",
|
||||||
|
Validation: promptkit.ValidationResult{Status: promptkit.ValidationPassed, Mode: promptkit.ValidationBasic},
|
||||||
|
RenderedPromptHash: "h",
|
||||||
|
})
|
||||||
|
if !strings.Contains(stderr.String(), "backend=local") {
|
||||||
|
t.Fatalf("expected backend in summary, got %q", stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunErrorMessageDoesNotExposeCapacityDetails(t *testing.T) {
|
||||||
|
got := runErrorMessage(&promptkit.CapacityError{BackendID: "private-backend"})
|
||||||
|
if got != "run error: model backend capacity is exhausted" {
|
||||||
|
t.Fatalf("unexpected capacity diagnostic: %q", got)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
type cliTestLibrary struct {
|
type cliTestLibrary struct {
|
||||||
@@ -1335,11 +1971,48 @@ func runCLICommand(t *testing.T, command func([]string, io.Writer, io.Writer) in
|
|||||||
return code, stdout.String(), stderr.String()
|
return code, stdout.String(), stderr.String()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func promptkitV09FixtureRoot(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
root, err := filepath.Abs(filepath.Join("..", "..", "..", "testdata", "promptkit-v0.9"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("resolve Promptkit v0.9 fixture root: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(root); err != nil {
|
||||||
|
t.Fatalf("stat Promptkit v0.9 fixture root: %v", err)
|
||||||
|
}
|
||||||
|
return root
|
||||||
|
}
|
||||||
|
|
||||||
|
func writePromptkitV09Config(t *testing.T, fixtureRoot string, includeSources bool) string {
|
||||||
|
t.Helper()
|
||||||
|
sources := ""
|
||||||
|
if includeSources {
|
||||||
|
sources = fmt.Sprintf("prompt_dir: %q\nprofile_dir: %q\nschema_dir: %q\n", filepath.Join(fixtureRoot, "prompts"), filepath.Join(fixtureRoot, "profiles"), filepath.Join(fixtureRoot, "schemas"))
|
||||||
|
}
|
||||||
|
return writeAppConfigFile(t, sources+`backends:
|
||||||
|
fixture-custom:
|
||||||
|
endpoint: http://127.0.0.1:11434/v1
|
||||||
|
api_key_env: FIXTURE_BACKEND_API_KEY
|
||||||
|
extra_params:
|
||||||
|
backend_option:
|
||||||
|
enabled: true
|
||||||
|
concurrency_limit: 2
|
||||||
|
queue_capacity: 0
|
||||||
|
`)
|
||||||
|
}
|
||||||
|
|
||||||
func writePromptFile(t *testing.T, dir, id, defaultProfile string) {
|
func writePromptFile(t *testing.T, dir, id, defaultProfile string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
writePromptFileWithTemplate(t, dir, id, defaultProfile, "Summarize: {{input \"transcript\"}}")
|
writePromptFileWithTemplate(t, dir, id, defaultProfile, "Summarize: {{input \"transcript\"}}")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writePromptDefinition(t *testing.T, dir, name, definition string) {
|
||||||
|
t.Helper()
|
||||||
|
if err := os.WriteFile(filepath.Join(dir, name), []byte(definition), 0o644); err != nil {
|
||||||
|
t.Fatalf("write prompt definition: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func writePromptFileWithTemplate(t *testing.T, dir, id, defaultProfile, templateContent string) {
|
func writePromptFileWithTemplate(t *testing.T, dir, id, defaultProfile, templateContent string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
data := fmt.Sprintf(`id: %s
|
data := fmt.Sprintf(`id: %s
|
||||||
|
|||||||
@@ -16,6 +16,8 @@ import (
|
|||||||
const (
|
const (
|
||||||
scriptoriumModulePath = "gitea.maximumdirect.net/eric/scriptorium"
|
scriptoriumModulePath = "gitea.maximumdirect.net/eric/scriptorium"
|
||||||
promptkitInternalPath = "gitea.maximumdirect.net/eric/promptkit/internal"
|
promptkitInternalPath = "gitea.maximumdirect.net/eric/promptkit/internal"
|
||||||
|
promptkitOpenRouterCatalogPath = "gitea.maximumdirect.net/eric/promptkit-backend-openrouter"
|
||||||
|
promptkitRakestrawhomeCatalogPath = "gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome"
|
||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
@@ -120,6 +122,24 @@ func TestForbiddenImportScannerDetectsPromptkitInternalPackages(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestForbiddenImportScannerDetectsPromptkitCatalogPackages(t *testing.T) {
|
||||||
|
for _, importPath := range []string{
|
||||||
|
promptkitOpenRouterCatalogPath,
|
||||||
|
promptkitRakestrawhomeCatalogPath,
|
||||||
|
} {
|
||||||
|
t.Run(importPath, func(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
sourcePath := writeGoSource(t, root, "nested/consumer/catalog.go", importPath)
|
||||||
|
|
||||||
|
violations, err := findForbiddenProductionImports(root)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("scan source fixture: %v", err)
|
||||||
|
}
|
||||||
|
assertSingleViolation(t, violations, sourcePath, importPath)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestForbiddenImportScannerAllowsRetainedApplicationPackages(t *testing.T) {
|
func TestForbiddenImportScannerAllowsRetainedApplicationPackages(t *testing.T) {
|
||||||
root := t.TempDir()
|
root := t.TempDir()
|
||||||
sourcePath := filepath.Join(root, "nested/consumer/application.go")
|
sourcePath := filepath.Join(root, "nested/consumer/application.go")
|
||||||
@@ -201,6 +221,12 @@ func isForbiddenProductionImport(importPath string) bool {
|
|||||||
if importPath == promptkitInternalPath || strings.HasPrefix(importPath, promptkitInternalPath+"/") {
|
if importPath == promptkitInternalPath || strings.HasPrefix(importPath, promptkitInternalPath+"/") {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
if importPath == promptkitOpenRouterCatalogPath || strings.HasPrefix(importPath, promptkitOpenRouterCatalogPath+"/") {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if importPath == promptkitRakestrawhomeCatalogPath || strings.HasPrefix(importPath, promptkitRakestrawhomeCatalogPath+"/") {
|
||||||
|
return true
|
||||||
|
}
|
||||||
for _, root := range removedFrameworkPackageRoots {
|
for _, root := range removedFrameworkPackageRoots {
|
||||||
if importPath == root || strings.HasPrefix(importPath, root+"/") {
|
if importPath == root || strings.HasPrefix(importPath, root+"/") {
|
||||||
return true
|
return true
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ type runRequestDTO struct {
|
|||||||
PromptID string `json:"prompt_id"`
|
PromptID string `json:"prompt_id"`
|
||||||
PromptVersion string `json:"prompt_version,omitempty"`
|
PromptVersion string `json:"prompt_version,omitempty"`
|
||||||
ProfileID string `json:"profile_id,omitempty"`
|
ProfileID string `json:"profile_id,omitempty"`
|
||||||
|
SessionID string `json:"session_id,omitempty"`
|
||||||
Inputs map[string]inputRefDTO `json:"inputs"`
|
Inputs map[string]inputRefDTO `json:"inputs"`
|
||||||
Vars map[string]string `json:"vars,omitempty"`
|
Vars map[string]string `json:"vars,omitempty"`
|
||||||
Model *modelOverrideRequestDTO `json:"model,omitempty"`
|
Model *modelOverrideRequestDTO `json:"model,omitempty"`
|
||||||
@@ -28,7 +29,7 @@ type modelOverrideRequestDTO struct {
|
|||||||
TopP *float64 `json:"top_p,omitempty"`
|
TopP *float64 `json:"top_p,omitempty"`
|
||||||
TimeoutSeconds *int `json:"timeout_seconds,omitempty"`
|
TimeoutSeconds *int `json:"timeout_seconds,omitempty"`
|
||||||
ServiceTier string `json:"service_tier,omitempty"`
|
ServiceTier string `json:"service_tier,omitempty"`
|
||||||
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
ReasoningEffort *string `json:"reasoning_effort,omitempty"`
|
||||||
APIKeyEnv string `json:"api_key_env,omitempty"`
|
APIKeyEnv string `json:"api_key_env,omitempty"`
|
||||||
ExtraParams map[string]any `json:"extra_params,omitempty"`
|
ExtraParams map[string]any `json:"extra_params,omitempty"`
|
||||||
}
|
}
|
||||||
@@ -56,6 +57,8 @@ type metadataDTO struct {
|
|||||||
PromptHash string `json:"prompt_hash"`
|
PromptHash string `json:"prompt_hash"`
|
||||||
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
||||||
SelectedProfileID string `json:"selected_profile_id"`
|
SelectedProfileID string `json:"selected_profile_id"`
|
||||||
|
SelectedBackendID string `json:"selected_backend_id,omitempty"`
|
||||||
|
SessionID string `json:"session_id,omitempty"`
|
||||||
ModelName string `json:"model_name"`
|
ModelName string `json:"model_name"`
|
||||||
Endpoint string `json:"endpoint"`
|
Endpoint string `json:"endpoint"`
|
||||||
ModelParams modelParamsDTO `json:"model_params"`
|
ModelParams modelParamsDTO `json:"model_params"`
|
||||||
@@ -71,6 +74,7 @@ type metadataDTO struct {
|
|||||||
|
|
||||||
type modelParamsDTO struct {
|
type modelParamsDTO struct {
|
||||||
Endpoint string `json:"endpoint"`
|
Endpoint string `json:"endpoint"`
|
||||||
|
BackendID string `json:"backend_id,omitempty"`
|
||||||
Model string `json:"model"`
|
Model string `json:"model"`
|
||||||
Temperature float64 `json:"temperature"`
|
Temperature float64 `json:"temperature"`
|
||||||
MaxTokens int `json:"max_tokens"`
|
MaxTokens int `json:"max_tokens"`
|
||||||
|
|||||||
@@ -76,12 +76,9 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
writeError(w, http.StatusBadRequest, "invalid_request", "prompt_id is required")
|
writeError(w, http.StatusBadRequest, "invalid_request", "prompt_id is required")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if len(req.Inputs) == 0 {
|
var mappedInputs map[string]promptkit.ArtifactRef
|
||||||
writeError(w, http.StatusBadRequest, "invalid_request", "inputs is required")
|
if len(req.Inputs) > 0 {
|
||||||
return
|
mappedInputs = make(map[string]promptkit.ArtifactRef, len(req.Inputs))
|
||||||
}
|
|
||||||
|
|
||||||
mappedInputs := make(map[string]promptkit.ArtifactRef, len(req.Inputs))
|
|
||||||
for name, in := range req.Inputs {
|
for name, in := range req.Inputs {
|
||||||
mappedInputs[name] = promptkit.ArtifactRef{
|
mappedInputs[name] = promptkit.ArtifactRef{
|
||||||
Type: promptkit.ArtifactRefType(in.Type),
|
Type: promptkit.ArtifactRefType(in.Type),
|
||||||
@@ -89,6 +86,7 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
Body: in.Body,
|
Body: in.Body,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
var model *promptkit.ExecutionTargetOverride
|
var model *promptkit.ExecutionTargetOverride
|
||||||
if req.Model != nil {
|
if req.Model != nil {
|
||||||
@@ -99,6 +97,7 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
PromptID: req.PromptID,
|
PromptID: req.PromptID,
|
||||||
PromptVersion: req.PromptVersion,
|
PromptVersion: req.PromptVersion,
|
||||||
ProfileID: req.ProfileID,
|
ProfileID: req.ProfileID,
|
||||||
|
SessionID: req.SessionID,
|
||||||
Inputs: mappedInputs,
|
Inputs: mappedInputs,
|
||||||
Vars: req.Vars,
|
Vars: req.Vars,
|
||||||
Execution: model,
|
Execution: model,
|
||||||
@@ -126,6 +125,8 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
PromptHash: res.PromptHash,
|
PromptHash: res.PromptHash,
|
||||||
RenderedPromptHash: res.RenderedPromptHash,
|
RenderedPromptHash: res.RenderedPromptHash,
|
||||||
SelectedProfileID: res.SelectedProfileID,
|
SelectedProfileID: res.SelectedProfileID,
|
||||||
|
SelectedBackendID: res.SelectedBackendID,
|
||||||
|
SessionID: res.SessionID,
|
||||||
ModelName: res.ModelName,
|
ModelName: res.ModelName,
|
||||||
Endpoint: res.Endpoint,
|
Endpoint: res.Endpoint,
|
||||||
ModelParams: modelParamsDTOFromExecutionTarget(res.EffectiveModelParams),
|
ModelParams: modelParamsDTOFromExecutionTarget(res.EffectiveModelParams),
|
||||||
@@ -173,6 +174,7 @@ func executionTargetOverrideFromModelOverrideDTO(dto *modelOverrideRequestDTO) *
|
|||||||
func modelParamsDTOFromExecutionTarget(target promptkit.ExecutionTarget) modelParamsDTO {
|
func modelParamsDTOFromExecutionTarget(target promptkit.ExecutionTarget) modelParamsDTO {
|
||||||
return modelParamsDTO{
|
return modelParamsDTO{
|
||||||
Endpoint: target.Endpoint,
|
Endpoint: target.Endpoint,
|
||||||
|
BackendID: target.BackendID,
|
||||||
Model: target.Model,
|
Model: target.Model,
|
||||||
Temperature: target.Temperature,
|
Temperature: target.Temperature,
|
||||||
MaxTokens: target.MaxTokens,
|
MaxTokens: target.MaxTokens,
|
||||||
@@ -220,6 +222,8 @@ func mapRunError(err error) (int, string, string) {
|
|||||||
return http.StatusBadRequest, "artifact_read_failed", "failed to read input artifact"
|
return http.StatusBadRequest, "artifact_read_failed", "failed to read input artifact"
|
||||||
case errors.Is(err, promptkit.ErrPromptRender):
|
case errors.Is(err, promptkit.ErrPromptRender):
|
||||||
return http.StatusBadRequest, "prompt_render_failed", "failed to render prompt"
|
return http.StatusBadRequest, "prompt_render_failed", "failed to render prompt"
|
||||||
|
case errors.Is(err, promptkit.ErrCapacityExceeded):
|
||||||
|
return http.StatusServiceUnavailable, "capacity_exceeded", "model backend capacity is exhausted"
|
||||||
case errors.Is(err, promptkit.ErrLLMGenerate):
|
case errors.Is(err, promptkit.ErrLLMGenerate):
|
||||||
return http.StatusBadGateway, "llm_failed", "model generation request failed"
|
return http.StatusBadGateway, "llm_failed", "model generation request failed"
|
||||||
case errors.Is(err, promptkit.ErrValidation):
|
case errors.Is(err, promptkit.ErrValidation):
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"reflect"
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -69,6 +70,34 @@ func (handlerLLMClient) Generate(ctx context.Context, req promptkit.GenerateRequ
|
|||||||
return &promptkit.GenerateResponse{Content: "ok"}, nil
|
return &promptkit.GenerateResponse{Content: "ok"}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type blockingLLMClient struct {
|
||||||
|
started chan struct{}
|
||||||
|
release chan struct{}
|
||||||
|
current int32
|
||||||
|
peak int32
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *blockingLLMClient) Generate(ctx context.Context, req promptkit.GenerateRequest) (*promptkit.GenerateResponse, error) {
|
||||||
|
current := atomic.AddInt32(&c.current, 1)
|
||||||
|
defer atomic.AddInt32(&c.current, -1)
|
||||||
|
for {
|
||||||
|
peak := atomic.LoadInt32(&c.peak)
|
||||||
|
if current <= peak || atomic.CompareAndSwapInt32(&c.peak, peak, current) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case c.started <- struct{}{}:
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-c.release:
|
||||||
|
return &promptkit.GenerateResponse{Content: "ok"}, nil
|
||||||
|
case <-ctx.Done():
|
||||||
|
return nil, ctx.Err()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||||
start := time.Now().UTC()
|
start := time.Now().UTC()
|
||||||
end := start.Add(2 * time.Second)
|
end := start.Add(2 * time.Second)
|
||||||
@@ -90,9 +119,11 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
|||||||
PromptHash: "phash",
|
PromptHash: "phash",
|
||||||
RenderedPromptHash: "rhash",
|
RenderedPromptHash: "rhash",
|
||||||
SelectedProfileID: "exec-default",
|
SelectedProfileID: "exec-default",
|
||||||
|
SelectedBackendID: "local",
|
||||||
ModelName: "m1",
|
ModelName: "m1",
|
||||||
Endpoint: "http://llm/v1",
|
Endpoint: "http://llm/v1",
|
||||||
EffectiveModelParams: promptkit.ExecutionTarget{
|
EffectiveModelParams: promptkit.ExecutionTarget{
|
||||||
|
BackendID: "local",
|
||||||
Endpoint: "http://llm/v1",
|
Endpoint: "http://llm/v1",
|
||||||
Model: "m1",
|
Model: "m1",
|
||||||
Temperature: 0.2,
|
Temperature: 0.2,
|
||||||
@@ -151,6 +182,9 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
|||||||
if metadata["selected_profile_id"] != "exec-default" {
|
if metadata["selected_profile_id"] != "exec-default" {
|
||||||
t.Fatalf("unexpected metadata.selected_profile_id: %#v", metadata["selected_profile_id"])
|
t.Fatalf("unexpected metadata.selected_profile_id: %#v", metadata["selected_profile_id"])
|
||||||
}
|
}
|
||||||
|
if metadata["selected_backend_id"] != "local" {
|
||||||
|
t.Fatalf("unexpected metadata.selected_backend_id: %#v", metadata["selected_backend_id"])
|
||||||
|
}
|
||||||
if metadata["model_name"] != "m1" || metadata["endpoint"] != "http://llm/v1" {
|
if metadata["model_name"] != "m1" || metadata["endpoint"] != "http://llm/v1" {
|
||||||
t.Fatalf("unexpected model metadata: name=%#v endpoint=%#v", metadata["model_name"], metadata["endpoint"])
|
t.Fatalf("unexpected model metadata: name=%#v endpoint=%#v", metadata["model_name"], metadata["endpoint"])
|
||||||
}
|
}
|
||||||
@@ -162,6 +196,9 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
|||||||
t.Fatalf("unexpected cache usage metadata: %#v", usage)
|
t.Fatalf("unexpected cache usage metadata: %#v", usage)
|
||||||
}
|
}
|
||||||
modelParams := metadata["model_params"].(map[string]any)
|
modelParams := metadata["model_params"].(map[string]any)
|
||||||
|
if modelParams["backend_id"] != "local" {
|
||||||
|
t.Fatalf("expected model_params.backend_id=local, got %#v", modelParams["backend_id"])
|
||||||
|
}
|
||||||
if modelParams["api_key_env"] != envName {
|
if modelParams["api_key_env"] != envName {
|
||||||
t.Fatalf("expected model_params.api_key_env=%q, got %#v", envName, modelParams["api_key_env"])
|
t.Fatalf("expected model_params.api_key_env=%q, got %#v", envName, modelParams["api_key_env"])
|
||||||
}
|
}
|
||||||
@@ -192,6 +229,124 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestHandlerAllowsOmittedInputsAndMapsPromptVersion(t *testing.T) {
|
||||||
|
r := &fakeRunner{result: &promptkit.RunResult{
|
||||||
|
Artifact: promptkit.Artifact{Body: []byte("ok")},
|
||||||
|
Validation: promptkit.ValidationResult{Status: promptkit.ValidationPassed, Mode: promptkit.ValidationNone, IsValid: true},
|
||||||
|
EffectiveModelParams: promptkit.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||||
|
}}
|
||||||
|
h := NewHandler(r)
|
||||||
|
|
||||||
|
for _, body := range []string{
|
||||||
|
`{"prompt_id":"prompt-1","prompt_version":"2"}`,
|
||||||
|
`{"prompt_id":"prompt-1","prompt_version":"2","inputs":{}}`,
|
||||||
|
} {
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(body))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
if r.last.PromptVersion != "2" {
|
||||||
|
t.Fatalf("expected prompt version to be mapped, got %q", r.last.PromptVersion)
|
||||||
|
}
|
||||||
|
if r.last.Inputs != nil {
|
||||||
|
t.Fatalf("expected omitted or empty inputs to remain nil, got %#v", r.last.Inputs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerDelegatesDefinitionInputRequirements(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
definition string
|
||||||
|
wantStatus int
|
||||||
|
wantCode string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "no declared inputs",
|
||||||
|
definition: `id: p
|
||||||
|
version: "1"
|
||||||
|
default_profile: exec
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "hello"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`,
|
||||||
|
wantStatus: http.StatusOK,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "optional input omitted",
|
||||||
|
definition: `id: p
|
||||||
|
version: "1"
|
||||||
|
default_profile: exec
|
||||||
|
inputs:
|
||||||
|
- name: note
|
||||||
|
required: false
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "hello"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`,
|
||||||
|
wantStatus: http.StatusOK,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "required input omitted",
|
||||||
|
definition: `id: p
|
||||||
|
version: "1"
|
||||||
|
default_profile: exec
|
||||||
|
inputs:
|
||||||
|
- name: note
|
||||||
|
required: true
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "hello"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`,
|
||||||
|
wantStatus: http.StatusBadRequest,
|
||||||
|
wantCode: "prompt_render_failed",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "template input omitted",
|
||||||
|
definition: `id: p
|
||||||
|
version: "1"
|
||||||
|
default_profile: exec
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: '{{input "note"}}'
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`,
|
||||||
|
wantStatus: http.StatusBadRequest,
|
||||||
|
wantCode: "prompt_render_failed",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
h := newDefinitionHandler(t, tc.definition)
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p"}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != tc.wantStatus {
|
||||||
|
t.Fatalf("expected %d, got %d body=%s", tc.wantStatus, w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
if tc.wantCode != "" {
|
||||||
|
assertHTTPErrorCode(t, w, tc.wantStatus, tc.wantCode)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestHandlerInlineRefsWorkWithoutArtifactRoot(t *testing.T) {
|
func TestHandlerInlineRefsWorkWithoutArtifactRoot(t *testing.T) {
|
||||||
h := newArtifactRootHandler(t, "")
|
h := newArtifactRootHandler(t, "")
|
||||||
|
|
||||||
@@ -365,10 +520,12 @@ func TestHandlerModelOverrideMapsAllSupportedExecutionFields(t *testing.T) {
|
|||||||
if got.Endpoint != "http://override/v1" ||
|
if got.Endpoint != "http://override/v1" ||
|
||||||
got.Model != "override-model" ||
|
got.Model != "override-model" ||
|
||||||
got.ServiceTier != "flex" ||
|
got.ServiceTier != "flex" ||
|
||||||
got.ReasoningEffort != "medium" ||
|
|
||||||
got.APIKeyEnv != "SCRIPTORIUM_API_KEY" {
|
got.APIKeyEnv != "SCRIPTORIUM_API_KEY" {
|
||||||
t.Fatalf("unexpected mapped execution target: %+v", got)
|
t.Fatalf("unexpected mapped execution target: %+v", got)
|
||||||
}
|
}
|
||||||
|
if got.ReasoningEffort == nil || *got.ReasoningEffort != "medium" {
|
||||||
|
t.Fatalf("unexpected mapped reasoning_effort: %#v", got.ReasoningEffort)
|
||||||
|
}
|
||||||
if got.Temperature == nil || *got.Temperature != 0.6 {
|
if got.Temperature == nil || *got.Temperature != 0.6 {
|
||||||
t.Fatalf("unexpected mapped temperature: %#v", got.Temperature)
|
t.Fatalf("unexpected mapped temperature: %#v", got.Temperature)
|
||||||
}
|
}
|
||||||
@@ -386,6 +543,251 @@ func TestHandlerModelOverrideMapsAllSupportedExecutionFields(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestHandlerModelOverridePreservesReasoningEffortPresence(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
model string
|
||||||
|
wantPresent bool
|
||||||
|
wantValue string
|
||||||
|
}{
|
||||||
|
{name: "omitted", model: `{}`},
|
||||||
|
{name: "nonblank", model: `{"reasoning_effort":"high"}`, wantPresent: true, wantValue: "high"},
|
||||||
|
{name: "explicit empty", model: `{"reasoning_effort":""}`, wantPresent: true},
|
||||||
|
{name: "null", model: `{"reasoning_effort":null}`},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
r := &fakeRunner{result: &promptkit.RunResult{
|
||||||
|
Artifact: promptkit.Artifact{Body: []byte("ok")},
|
||||||
|
Validation: promptkit.ValidationResult{Status: promptkit.ValidationPassed, Mode: promptkit.ValidationBasic, IsValid: true},
|
||||||
|
EffectiveModelParams: promptkit.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||||
|
}}
|
||||||
|
h := NewHandler(r)
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"prompt-1","inputs":{"transcript":{"type":"file","uri":"./t.md"}},"model":`+tc.model+`}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
if r.last.Execution == nil {
|
||||||
|
t.Fatal("expected execution override")
|
||||||
|
}
|
||||||
|
if (r.last.Execution.ReasoningEffort != nil) != tc.wantPresent {
|
||||||
|
t.Fatalf("unexpected reasoning_effort presence: %#v", r.last.Execution.ReasoningEffort)
|
||||||
|
}
|
||||||
|
if tc.wantPresent && *r.last.Execution.ReasoningEffort != tc.wantValue {
|
||||||
|
t.Fatalf("unexpected reasoning_effort: got %q want %q", *r.last.Execution.ReasoningEffort, tc.wantValue)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerMapsSessionIDAndReportsEffectiveResultSessionID(t *testing.T) {
|
||||||
|
r := &fakeRunner{result: &promptkit.RunResult{
|
||||||
|
Artifact: promptkit.Artifact{Body: []byte("ok")},
|
||||||
|
Validation: promptkit.ValidationResult{Status: promptkit.ValidationPassed, Mode: promptkit.ValidationBasic, IsValid: true},
|
||||||
|
EffectiveModelParams: promptkit.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||||
|
SessionID: "effective-session",
|
||||||
|
}}
|
||||||
|
h := NewHandler(r)
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"prompt-1","session_id":"request-session"}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
if r.last.SessionID != "request-session" {
|
||||||
|
t.Fatalf("expected session ID in run request, got %q", r.last.SessionID)
|
||||||
|
}
|
||||||
|
var resp map[string]any
|
||||||
|
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||||
|
t.Fatalf("invalid JSON response: %v", err)
|
||||||
|
}
|
||||||
|
metadata := resp["metadata"].(map[string]any)
|
||||||
|
if metadata["session_id"] != "effective-session" {
|
||||||
|
t.Fatalf("expected effective session ID in response, got %#v", metadata["session_id"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerOmitsEmptyEffectiveSessionID(t *testing.T) {
|
||||||
|
r := &fakeRunner{result: &promptkit.RunResult{
|
||||||
|
Artifact: promptkit.Artifact{Body: []byte("ok")},
|
||||||
|
Validation: promptkit.ValidationResult{Status: promptkit.ValidationPassed, Mode: promptkit.ValidationBasic, IsValid: true},
|
||||||
|
EffectiveModelParams: promptkit.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||||
|
}}
|
||||||
|
h := NewHandler(r)
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"prompt-1"}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
h.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
var resp map[string]any
|
||||||
|
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||||
|
t.Fatalf("invalid JSON response: %v", err)
|
||||||
|
}
|
||||||
|
metadata := resp["metadata"].(map[string]any)
|
||||||
|
if _, ok := metadata["session_id"]; ok {
|
||||||
|
t.Fatalf("expected empty effective session ID to be omitted, got %#v", metadata)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerSessionIDUsesPromptkitResolution(t *testing.T) {
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
definitionSession string
|
||||||
|
requestSession string
|
||||||
|
wantSession string
|
||||||
|
}{
|
||||||
|
{name: "definition session", definitionSession: "definition-session", wantSession: "definition-session"},
|
||||||
|
{name: "direct session", definitionSession: "definition-session", requestSession: "direct-session", wantSession: "direct-session"},
|
||||||
|
{name: "no effective session"},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
promptDir := t.TempDir()
|
||||||
|
profileDir := t.TempDir()
|
||||||
|
sessionLine := ""
|
||||||
|
if tc.definitionSession != "" {
|
||||||
|
sessionLine = "session_id: " + tc.definitionSession + "\n"
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(promptDir, "prompt.yaml"), []byte("id: p\nversion: \"1\"\ndefault_profile: exec\n"+sessionLine+`messages:
|
||||||
|
- role: user
|
||||||
|
content: "hi"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`), 0o644); err != nil {
|
||||||
|
t.Fatalf("write prompt fixture: %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte("id: exec\nendpoint: http://127.0.0.1:1/v1\nmodel: test\n"), 0o644); err != nil {
|
||||||
|
t.Fatalf("write profile fixture: %v", err)
|
||||||
|
}
|
||||||
|
engine, err := promptkit.NewEngine(promptkit.Config{PromptDir: promptDir, ProfileDir: profileDir}, promptkit.WithLLMClient(handlerLLMClient{}))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("new engine: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
body := `{"prompt_id":"p"}`
|
||||||
|
if tc.requestSession != "" {
|
||||||
|
body = `{"prompt_id":"p","session_id":"` + tc.requestSession + `"}`
|
||||||
|
}
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
NewHandler(engine).ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(body)))
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
var resp map[string]any
|
||||||
|
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||||
|
t.Fatalf("invalid JSON response: %v", err)
|
||||||
|
}
|
||||||
|
metadata := resp["metadata"].(map[string]any)
|
||||||
|
if tc.wantSession == "" {
|
||||||
|
if _, ok := metadata["session_id"]; ok {
|
||||||
|
t.Fatalf("expected session_id to be omitted, got %#v", metadata)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if metadata["session_id"] != tc.wantSession {
|
||||||
|
t.Fatalf("expected effective session ID %q, got %#v", tc.wantSession, metadata["session_id"])
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerSharesBackendCapacityAcrossConcurrentRequests(t *testing.T) {
|
||||||
|
promptDir := t.TempDir()
|
||||||
|
profileDir := t.TempDir()
|
||||||
|
if err := os.WriteFile(filepath.Join(promptDir, "prompt.yaml"), []byte(`id: p
|
||||||
|
version: "1"
|
||||||
|
default_profile: limited
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "hi"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`), 0o644); err != nil {
|
||||||
|
t.Fatalf("write prompt fixture: %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte(`id: limited
|
||||||
|
backend: limited
|
||||||
|
model: test
|
||||||
|
`), 0o644); err != nil {
|
||||||
|
t.Fatalf("write profile fixture: %v", err)
|
||||||
|
}
|
||||||
|
queueCapacity := 0
|
||||||
|
client := &blockingLLMClient{started: make(chan struct{}, 1), release: make(chan struct{})}
|
||||||
|
engine, err := promptkit.NewEngine(
|
||||||
|
promptkit.Config{PromptDir: promptDir, ProfileDir: profileDir},
|
||||||
|
promptkit.WithBackend(promptkit.Backend{
|
||||||
|
ID: "limited",
|
||||||
|
Endpoint: "http://127.0.0.1:1/v1",
|
||||||
|
ConcurrencyLimit: 1,
|
||||||
|
QueueCapacity: &queueCapacity,
|
||||||
|
}),
|
||||||
|
promptkit.WithLLMClient(client),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("new engine: %v", err)
|
||||||
|
}
|
||||||
|
h := NewHandler(engine)
|
||||||
|
first := httptest.NewRecorder()
|
||||||
|
firstDone := make(chan struct{})
|
||||||
|
go func() {
|
||||||
|
defer close(firstDone)
|
||||||
|
h.ServeHTTP(first, httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p"}`)))
|
||||||
|
}()
|
||||||
|
select {
|
||||||
|
case <-client.started:
|
||||||
|
case <-time.After(time.Second):
|
||||||
|
t.Fatal("first request did not reach generation")
|
||||||
|
}
|
||||||
|
|
||||||
|
second := httptest.NewRecorder()
|
||||||
|
h.ServeHTTP(second, httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(`{"prompt_id":"p"}`)))
|
||||||
|
assertHTTPErrorCode(t, second, http.StatusServiceUnavailable, "capacity_exceeded")
|
||||||
|
if second.Header().Get("Retry-After") != "" {
|
||||||
|
t.Fatalf("expected no Retry-After header, got %q", second.Header().Get("Retry-After"))
|
||||||
|
}
|
||||||
|
if strings.Contains(second.Body.String(), "limited") {
|
||||||
|
t.Fatalf("capacity response leaked backend details: %s", second.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
close(client.release)
|
||||||
|
select {
|
||||||
|
case <-firstDone:
|
||||||
|
case <-time.After(time.Second):
|
||||||
|
t.Fatal("first request did not complete")
|
||||||
|
}
|
||||||
|
if first.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected first request to succeed, got %d body=%s", first.Code, first.Body.String())
|
||||||
|
}
|
||||||
|
if atomic.LoadInt32(&client.peak) != 1 {
|
||||||
|
t.Fatalf("expected peak generation concurrency of one, got %d", atomic.LoadInt32(&client.peak))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerRejectsOverlongSessionIDAndNonStringReasoningEffort(t *testing.T) {
|
||||||
|
engine := newHandlerEngine(t)
|
||||||
|
for _, body := range []string{
|
||||||
|
`{"prompt_id":"p","session_id":"` + strings.Repeat("x", 257) + `"}`,
|
||||||
|
`{"prompt_id":"p","model":{"reasoning_effort":1}}`,
|
||||||
|
} {
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
NewHandler(engine).ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(body)))
|
||||||
|
if w.Code != http.StatusBadRequest {
|
||||||
|
t.Fatalf("expected 400, got %d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestHandlerModelOverrideAcceptsJSONCompatibleExtraParams(t *testing.T) {
|
func TestHandlerModelOverrideAcceptsJSONCompatibleExtraParams(t *testing.T) {
|
||||||
r := &fakeRunner{result: &promptkit.RunResult{
|
r := &fakeRunner{result: &promptkit.RunResult{
|
||||||
Artifact: promptkit.Artifact{Body: []byte("ok")},
|
Artifact: promptkit.Artifact{Body: []byte("ok")},
|
||||||
@@ -722,6 +1124,7 @@ func TestHandlerPublicErrorMapping(t *testing.T) {
|
|||||||
{name: "file too large", err: ErrFileTooLarge, status: http.StatusRequestEntityTooLarge, code: "artifact_too_large", message: "file input artifact is too large"},
|
{name: "file too large", err: ErrFileTooLarge, status: http.StatusRequestEntityTooLarge, code: "artifact_too_large", message: "file input artifact is too large"},
|
||||||
{name: "artifact", err: wrap(promptkit.ErrArtifactLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "artifact_read_failed", message: "failed to read input artifact", avoidCause: "read failed"},
|
{name: "artifact", err: wrap(promptkit.ErrArtifactLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "artifact_read_failed", message: "failed to read input artifact", avoidCause: "read failed"},
|
||||||
{name: "prompt render", err: wrap(promptkit.ErrPromptRender, fmt.Errorf("render failed")), status: http.StatusBadRequest, code: "prompt_render_failed", message: "failed to render prompt", avoidCause: "render failed"},
|
{name: "prompt render", err: wrap(promptkit.ErrPromptRender, fmt.Errorf("render failed")), status: http.StatusBadRequest, code: "prompt_render_failed", message: "failed to render prompt", avoidCause: "render failed"},
|
||||||
|
{name: "capacity", err: &promptkit.CapacityError{BackendID: "private-backend"}, status: http.StatusServiceUnavailable, code: "capacity_exceeded", message: "model backend capacity is exhausted", avoidCause: "private-backend"},
|
||||||
{name: "llm", err: wrap(promptkit.ErrLLMGenerate, fmt.Errorf("llm failed")), status: http.StatusBadGateway, code: "llm_failed", message: "model generation request failed", avoidCause: "llm failed"},
|
{name: "llm", err: wrap(promptkit.ErrLLMGenerate, fmt.Errorf("llm failed")), status: http.StatusBadGateway, code: "llm_failed", message: "model generation request failed", avoidCause: "llm failed"},
|
||||||
{name: "validation runtime", err: wrap(promptkit.ErrValidation, fmt.Errorf("validator broke")), status: http.StatusInternalServerError, code: "validation_runtime_failed", message: "validation runtime failed", avoidCause: "validator broke"},
|
{name: "validation runtime", err: wrap(promptkit.ErrValidation, fmt.Errorf("validator broke")), status: http.StatusInternalServerError, code: "validation_runtime_failed", message: "validation runtime failed", avoidCause: "validator broke"},
|
||||||
}
|
}
|
||||||
@@ -889,6 +1292,31 @@ model: model
|
|||||||
return engine
|
return engine
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func newDefinitionHandler(t *testing.T, definition string) *Handler {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
promptDir := t.TempDir()
|
||||||
|
profileDir := t.TempDir()
|
||||||
|
if err := os.WriteFile(filepath.Join(promptDir, "prompt.yaml"), []byte(definition), 0o644); err != nil {
|
||||||
|
t.Fatalf("write prompt fixture: %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte(`id: exec
|
||||||
|
endpoint: http://example.invalid/v1
|
||||||
|
model: model
|
||||||
|
`), 0o644); err != nil {
|
||||||
|
t.Fatalf("write profile fixture: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||||
|
PromptDir: promptDir,
|
||||||
|
ProfileDir: profileDir,
|
||||||
|
}, promptkit.WithLLMClient(handlerLLMClient{}))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("construct public engine: %v", err)
|
||||||
|
}
|
||||||
|
return NewHandler(engine)
|
||||||
|
}
|
||||||
|
|
||||||
func assertHTTPErrorCode(t *testing.T, w *httptest.ResponseRecorder, status int, code string) {
|
func assertHTTPErrorCode(t *testing.T, w *httptest.ResponseRecorder, status int, code string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||||
@@ -37,6 +38,24 @@ type Config struct {
|
|||||||
SchemaDir string `yaml:"schema_dir"`
|
SchemaDir string `yaml:"schema_dir"`
|
||||||
Server ServerConfig `yaml:"server"`
|
Server ServerConfig `yaml:"server"`
|
||||||
Defaults DefaultsConfig `yaml:"defaults"`
|
Defaults DefaultsConfig `yaml:"defaults"`
|
||||||
|
Backends map[string]BackendConfig `yaml:"backends"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type BackendConfig struct {
|
||||||
|
Endpoint string `yaml:"endpoint"`
|
||||||
|
APIKeyEnv string `yaml:"api_key_env"`
|
||||||
|
ExtraParams map[string]any `yaml:"extra_params"`
|
||||||
|
ConcurrencyLimit int `yaml:"concurrency_limit"`
|
||||||
|
QueueCapacity *int `yaml:"queue_capacity"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type BackendSettings struct {
|
||||||
|
ID string
|
||||||
|
Endpoint string
|
||||||
|
APIKeyEnv string
|
||||||
|
ExtraParams map[string]any
|
||||||
|
ConcurrencyLimit int
|
||||||
|
QueueCapacity *int
|
||||||
}
|
}
|
||||||
|
|
||||||
type ServerConfig struct {
|
type ServerConfig struct {
|
||||||
@@ -62,6 +81,7 @@ type AppSettings struct {
|
|||||||
MaxArtifactBytes int64
|
MaxArtifactBytes int64
|
||||||
MaxResponseBytes int64
|
MaxResponseBytes int64
|
||||||
DefaultRenderFormat renderformat.PreparedRunOutputFormat
|
DefaultRenderFormat renderformat.PreparedRunOutputFormat
|
||||||
|
Backends []BackendSettings
|
||||||
}
|
}
|
||||||
|
|
||||||
// CLIOverrides can be applied after config load to enforce precedence.
|
// CLIOverrides can be applied after config load to enforce precedence.
|
||||||
@@ -245,6 +265,25 @@ func applyConfig(base AppSettings, cfg Config) (AppSettings, error) {
|
|||||||
}
|
}
|
||||||
out.DefaultRenderFormat = parsed
|
out.DefaultRenderFormat = parsed
|
||||||
}
|
}
|
||||||
|
if len(cfg.Backends) > 0 {
|
||||||
|
ids := make([]string, 0, len(cfg.Backends))
|
||||||
|
for id := range cfg.Backends {
|
||||||
|
ids = append(ids, id)
|
||||||
|
}
|
||||||
|
sort.Strings(ids)
|
||||||
|
out.Backends = make([]BackendSettings, 0, len(ids))
|
||||||
|
for _, id := range ids {
|
||||||
|
backend := cfg.Backends[id]
|
||||||
|
out.Backends = append(out.Backends, BackendSettings{
|
||||||
|
ID: id,
|
||||||
|
Endpoint: backend.Endpoint,
|
||||||
|
APIKeyEnv: backend.APIKeyEnv,
|
||||||
|
ExtraParams: backend.ExtraParams,
|
||||||
|
ConcurrencyLimit: backend.ConcurrencyLimit,
|
||||||
|
QueueCapacity: backend.QueueCapacity,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
return out, nil
|
return out, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||||
@@ -20,7 +21,7 @@ func TestLoadConfigMissingImplicitPathUsesBuiltInDefaults(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
want := BuiltInDefaults()
|
want := BuiltInDefaults()
|
||||||
if got != want {
|
if !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("unexpected settings: got=%+v want=%+v", got, want)
|
t.Fatalf("unexpected settings: got=%+v want=%+v", got, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -100,6 +101,78 @@ func TestLoadConfigAPIKeyFieldIsRejectedAsUnknown(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestLoadConfigBackendsRetainsResolvedSettingsInSortedOrder(t *testing.T) {
|
||||||
|
path := writeConfigFile(t, "config.yml", `
|
||||||
|
backends:
|
||||||
|
zebra:
|
||||||
|
endpoint: https://zebra.example/v1
|
||||||
|
api_key_env: ZEBRA_API_KEY
|
||||||
|
extra_params:
|
||||||
|
provider_option: enabled
|
||||||
|
nested:
|
||||||
|
enabled: true
|
||||||
|
attempts: 2
|
||||||
|
concurrency_limit: 2
|
||||||
|
alpha:
|
||||||
|
endpoint: http://alpha.example/v1
|
||||||
|
queue_capacity: 0
|
||||||
|
`)
|
||||||
|
|
||||||
|
got, err := LoadConfig(path, true)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected no error, got %v", err)
|
||||||
|
}
|
||||||
|
if len(got.Backends) != 2 {
|
||||||
|
t.Fatalf("expected two backends, got %#v", got.Backends)
|
||||||
|
}
|
||||||
|
if got.Backends[0].ID != "alpha" || got.Backends[1].ID != "zebra" {
|
||||||
|
t.Fatalf("expected sorted backend IDs, got %#v", got.Backends)
|
||||||
|
}
|
||||||
|
if got.Backends[0].QueueCapacity == nil || *got.Backends[0].QueueCapacity != 0 {
|
||||||
|
t.Fatalf("expected explicit zero queue capacity, got %#v", got.Backends[0].QueueCapacity)
|
||||||
|
}
|
||||||
|
if got.Backends[1].QueueCapacity != nil {
|
||||||
|
t.Fatalf("expected omitted queue capacity to remain nil, got %#v", got.Backends[1].QueueCapacity)
|
||||||
|
}
|
||||||
|
wantParams := map[string]any{
|
||||||
|
"provider_option": "enabled",
|
||||||
|
"nested": map[string]any{
|
||||||
|
"enabled": true,
|
||||||
|
"attempts": 2,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got.Backends[1].ExtraParams, wantParams) {
|
||||||
|
t.Fatalf("unexpected extra params: got=%#v want=%#v", got.Backends[1].ExtraParams, wantParams)
|
||||||
|
}
|
||||||
|
if got.Backends[1].Endpoint != "https://zebra.example/v1" || got.Backends[1].APIKeyEnv != "ZEBRA_API_KEY" || got.Backends[1].ConcurrencyLimit != 2 {
|
||||||
|
t.Fatalf("unexpected zebra backend: %#v", got.Backends[1])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadConfigRejectsUnknownOrSecretBackendFields(t *testing.T) {
|
||||||
|
for name, body := range map[string]string{
|
||||||
|
"unknown": "backends:\n local:\n endpoint: http://localhost:11434/v1\n unexpected: value\n",
|
||||||
|
"secret": "backends:\n local:\n endpoint: http://localhost:11434/v1\n api_key: secret\n",
|
||||||
|
} {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
path := writeConfigFile(t, "config.yml", body)
|
||||||
|
_, err := LoadConfig(path, true)
|
||||||
|
if !errors.Is(err, ErrInvalidConfigYAML) {
|
||||||
|
t.Fatalf("expected ErrInvalidConfigYAML, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadConfigRejectsInvalidBackendFieldTypes(t *testing.T) {
|
||||||
|
path := writeConfigFile(t, "config.yml", "backends:\n local:\n endpoint: http://localhost:11434/v1\n queue_capacity: not-a-number\n")
|
||||||
|
|
||||||
|
_, err := LoadConfig(path, true)
|
||||||
|
if !errors.Is(err, ErrInvalidConfigYAML) {
|
||||||
|
t.Fatalf("expected ErrInvalidConfigYAML, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestLoadConfigValidConfigSetsDirectoriesAndServerAddr(t *testing.T) {
|
func TestLoadConfigValidConfigSetsDirectoriesAndServerAddr(t *testing.T) {
|
||||||
path := writeConfigFile(t, "config.yml", `
|
path := writeConfigFile(t, "config.yml", `
|
||||||
prompt_dir: ./prompts
|
prompt_dir: ./prompts
|
||||||
@@ -196,7 +269,7 @@ func TestLoadConfigEmptyFileResolvesToBuiltInDefaults(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
want := BuiltInDefaults()
|
want := BuiltInDefaults()
|
||||||
if got != want {
|
if !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("unexpected settings: got=%+v want=%+v", got, want)
|
t.Fatalf("unexpected settings: got=%+v want=%+v", got, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
122
internal/format/inspection.go
Normal file
122
internal/format/inspection.go
Normal file
@@ -0,0 +1,122 @@
|
|||||||
|
package format
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strconv"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
|
)
|
||||||
|
|
||||||
|
type PromptInspection struct {
|
||||||
|
PromptID string `json:"prompt_id"`
|
||||||
|
PromptVersion string `json:"prompt_version"`
|
||||||
|
PromptHash string `json:"prompt_hash"`
|
||||||
|
DefaultProfileID string `json:"default_profile_id"`
|
||||||
|
Inputs []PromptInspectionInput `json:"inputs"`
|
||||||
|
OutputContract OutputContract `json:"output_contract"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptInspectionInput struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Required bool `json:"required"`
|
||||||
|
ContentType string `json:"content_type"`
|
||||||
|
Description string `json:"description"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type OutputContract struct {
|
||||||
|
Format string `json:"format"`
|
||||||
|
ValidationMode string `json:"validation_mode"`
|
||||||
|
SchemaPath string `json:"schema_path"`
|
||||||
|
RepairAttempts int `json:"repair_attempts"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type ProfileInspection struct {
|
||||||
|
ProfileID string `json:"profile_id"`
|
||||||
|
EffectiveModelParams ProfileModelParams `json:"effective_model_params"`
|
||||||
|
APIKeyRequired bool `json:"api_key_required"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type ProfileModelParams struct {
|
||||||
|
BackendID string `json:"backend_id"`
|
||||||
|
Endpoint string `json:"endpoint"`
|
||||||
|
Model string `json:"model"`
|
||||||
|
Temperature float64 `json:"temperature"`
|
||||||
|
MaxTokens int `json:"max_tokens"`
|
||||||
|
TopP float64 `json:"top_p"`
|
||||||
|
TimeoutSeconds int `json:"timeout_seconds"`
|
||||||
|
ServiceTier string `json:"service_tier"`
|
||||||
|
ReasoningEffort string `json:"reasoning_effort"`
|
||||||
|
APIKeyEnv string `json:"api_key_env"`
|
||||||
|
ExtraParams map[string]any `json:"extra_params"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func FormatPromptInspection(value *promptkit.PromptInspection, outputFormat OutputFormat) ([]byte, error) {
|
||||||
|
if value == nil {
|
||||||
|
return nil, errors.New("prompt inspection is nil")
|
||||||
|
}
|
||||||
|
dto := PromptInspection{
|
||||||
|
PromptID: value.PromptID, PromptVersion: value.PromptVersion, PromptHash: value.PromptHash, DefaultProfileID: value.DefaultProfileID,
|
||||||
|
Inputs: make([]PromptInspectionInput, len(value.Inputs)),
|
||||||
|
OutputContract: OutputContract{Format: string(value.OutputContract.Format), ValidationMode: string(value.OutputContract.ValidationMode), SchemaPath: value.OutputContract.SchemaPath, RepairAttempts: value.OutputContract.RepairAttempts},
|
||||||
|
}
|
||||||
|
for i, input := range value.Inputs {
|
||||||
|
dto.Inputs[i] = PromptInspectionInput{Name: input.Name, Required: input.Required, ContentType: input.ContentType, Description: input.Description}
|
||||||
|
}
|
||||||
|
switch outputFormat {
|
||||||
|
case OutputFormatJSON:
|
||||||
|
data, err := json.MarshalIndent(dto, "", " ")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return append(data, '\n'), nil
|
||||||
|
case OutputFormatText:
|
||||||
|
var b bytes.Buffer
|
||||||
|
fmt.Fprintf(&b, "prompt_id: %s\nprompt_version: %s\nprompt_hash: %s\ndefault_profile_id: %s\ninputs:", dto.PromptID, dto.PromptVersion, dto.PromptHash, dto.DefaultProfileID)
|
||||||
|
if len(dto.Inputs) == 0 {
|
||||||
|
fmt.Fprintln(&b, " []")
|
||||||
|
} else {
|
||||||
|
fmt.Fprintln(&b)
|
||||||
|
for _, input := range dto.Inputs {
|
||||||
|
fmt.Fprintf(&b, " - name: %s\n required: %t\n content_type: %s\n description: %s\n", input.Name, input.Required, input.ContentType, strconv.Quote(input.Description))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fmt.Fprintf(&b, "output_contract:\n format: %s\n validation_mode: %s\n schema_path: %s\n repair_attempts: %d\n", dto.OutputContract.Format, dto.OutputContract.ValidationMode, dto.OutputContract.SchemaPath, dto.OutputContract.RepairAttempts)
|
||||||
|
return b.Bytes(), nil
|
||||||
|
default:
|
||||||
|
return nil, fmt.Errorf("%w: %q", ErrUnknownPreparedRunFormat, outputFormat)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func FormatProfileInspection(value *promptkit.ProfileInspection, outputFormat OutputFormat) ([]byte, error) {
|
||||||
|
if value == nil {
|
||||||
|
return nil, errors.New("profile inspection is nil")
|
||||||
|
}
|
||||||
|
target := value.EffectiveModelParams
|
||||||
|
params := map[string]any{}
|
||||||
|
for key, item := range target.ExtraParams {
|
||||||
|
params[key] = item
|
||||||
|
}
|
||||||
|
dto := ProfileInspection{ProfileID: value.ProfileID, APIKeyRequired: value.APIKeyRequired, EffectiveModelParams: ProfileModelParams{BackendID: target.BackendID, Endpoint: target.Endpoint, Model: target.Model, Temperature: target.Temperature, MaxTokens: target.MaxTokens, TopP: target.TopP, TimeoutSeconds: target.TimeoutSeconds, ServiceTier: target.ServiceTier, ReasoningEffort: target.ReasoningEffort, APIKeyEnv: target.APIKeyEnv, ExtraParams: params}}
|
||||||
|
switch outputFormat {
|
||||||
|
case OutputFormatJSON:
|
||||||
|
data, err := json.MarshalIndent(dto, "", " ")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return append(data, '\n'), nil
|
||||||
|
case OutputFormatText:
|
||||||
|
var b bytes.Buffer
|
||||||
|
fmt.Fprintf(&b, "profile_id: %s\neffective_model_params:\n backend_id: %s\n endpoint: %s\n model: %s\n temperature: %g\n max_tokens: %d\n top_p: %g\n timeout_seconds: %d\n service_tier: %s\n reasoning_effort: %s\n api_key_env: %s\n extra_params: ", dto.ProfileID, dto.EffectiveModelParams.BackendID, dto.EffectiveModelParams.Endpoint, dto.EffectiveModelParams.Model, dto.EffectiveModelParams.Temperature, dto.EffectiveModelParams.MaxTokens, dto.EffectiveModelParams.TopP, dto.EffectiveModelParams.TimeoutSeconds, dto.EffectiveModelParams.ServiceTier, dto.EffectiveModelParams.ReasoningEffort, dto.EffectiveModelParams.APIKeyEnv)
|
||||||
|
paramsJSON, err := json.Marshal(dto.EffectiveModelParams.ExtraParams)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
fmt.Fprintf(&b, "%s\napi_key_required: %t\n", paramsJSON, dto.APIKeyRequired)
|
||||||
|
return b.Bytes(), nil
|
||||||
|
default:
|
||||||
|
return nil, fmt.Errorf("%w: %q", ErrUnknownPreparedRunFormat, outputFormat)
|
||||||
|
}
|
||||||
|
}
|
||||||
124
internal/format/inspection_test.go
Normal file
124
internal/format/inspection_test.go
Normal file
@@ -0,0 +1,124 @@
|
|||||||
|
package format
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestFormatPromptInspectionPreservesDeclaredOrderAndEmptyValues(t *testing.T) {
|
||||||
|
value := &promptkit.PromptInspection{PromptID: "p", PromptVersion: "1", PromptHash: "hash", Inputs: []promptkit.PromptInputDefinition{{Name: "first", Required: true, ContentType: "text/plain", Description: "first input"}, {Name: "second"}}, OutputContract: promptkit.OutputContract{Format: "text", ValidationMode: promptkit.ValidationNone}}
|
||||||
|
text, err := FormatPromptInspection(value, OutputFormatText)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(text), "default_profile_id: \ninputs:\n - name: first") || strings.Index(string(text), "name: first") > strings.Index(string(text), "name: second") {
|
||||||
|
t.Fatalf("unexpected text inspection: %s", text)
|
||||||
|
}
|
||||||
|
jsonOutput, err := FormatPromptInspection(value, OutputFormatJSON)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !strings.HasSuffix(string(jsonOutput), "\n") || !strings.Contains(string(jsonOutput), `"default_profile_id": ""`) {
|
||||||
|
t.Fatalf("unexpected json inspection: %s", jsonOutput)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFormatPromptInspectionRejectsNil(t *testing.T) {
|
||||||
|
if _, err := FormatPromptInspection(nil, OutputFormatText); err == nil {
|
||||||
|
t.Fatal("expected nil inspection error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFormatProfileInspectionPreservesSafeEffectiveValues(t *testing.T) {
|
||||||
|
const secret = "sentinel-secret-must-not-appear"
|
||||||
|
t.Setenv("FIXTURE_PROFILE_API_KEY", secret)
|
||||||
|
|
||||||
|
value := &promptkit.ProfileInspection{
|
||||||
|
ProfileID: "custom-derived",
|
||||||
|
EffectiveModelParams: promptkit.ExecutionTarget{
|
||||||
|
BackendID: "fixture-custom",
|
||||||
|
Endpoint: "http://127.0.0.1:11434/v1",
|
||||||
|
Model: "fixture-model",
|
||||||
|
Temperature: 0.25,
|
||||||
|
MaxTokens: 640,
|
||||||
|
TopP: 0.9,
|
||||||
|
TimeoutSeconds: 45,
|
||||||
|
ServiceTier: "flex",
|
||||||
|
ReasoningEffort: "high",
|
||||||
|
APIKeyEnv: "FIXTURE_PROFILE_API_KEY",
|
||||||
|
ExtraParams: map[string]any{
|
||||||
|
"zeta": true,
|
||||||
|
"alpha": map[string]any{"nested": []any{"first", 2.0}},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
APIKeyRequired: true,
|
||||||
|
}
|
||||||
|
|
||||||
|
jsonOutput, err := FormatProfileInspection(value, OutputFormatJSON)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
secondJSONOutput, err := FormatProfileInspection(value, OutputFormatJSON)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if string(jsonOutput) != string(secondJSONOutput) || !strings.HasSuffix(string(jsonOutput), "\n") {
|
||||||
|
t.Fatalf("expected deterministic newline-terminated JSON, got %q", jsonOutput)
|
||||||
|
}
|
||||||
|
if strings.Contains(string(jsonOutput), secret) {
|
||||||
|
t.Fatalf("inspection exposed an environment secret: %s", jsonOutput)
|
||||||
|
}
|
||||||
|
|
||||||
|
var decoded ProfileInspection
|
||||||
|
if err := json.Unmarshal(jsonOutput, &decoded); err != nil {
|
||||||
|
t.Fatalf("decode profile inspection: %v", err)
|
||||||
|
}
|
||||||
|
if decoded.ProfileID != "custom-derived" || decoded.EffectiveModelParams.BackendID != "fixture-custom" {
|
||||||
|
t.Fatalf("unexpected profile identity: %+v", decoded)
|
||||||
|
}
|
||||||
|
if decoded.EffectiveModelParams.APIKeyEnv != "FIXTURE_PROFILE_API_KEY" || !decoded.APIKeyRequired {
|
||||||
|
t.Fatalf("unexpected credential metadata: %+v", decoded)
|
||||||
|
}
|
||||||
|
alpha, ok := decoded.EffectiveModelParams.ExtraParams["alpha"].(map[string]any)
|
||||||
|
if !ok || len(alpha["nested"].([]any)) != 2 {
|
||||||
|
t.Fatalf("nested extra parameters were not preserved: %#v", decoded.EffectiveModelParams.ExtraParams)
|
||||||
|
}
|
||||||
|
|
||||||
|
textOutput, err := FormatProfileInspection(value, OutputFormatText)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
textValue := string(textOutput)
|
||||||
|
if !strings.Contains(textValue, "backend_id: fixture-custom") ||
|
||||||
|
!strings.Contains(textValue, "api_key_env: FIXTURE_PROFILE_API_KEY") ||
|
||||||
|
!strings.Contains(textValue, `extra_params: {"alpha":{"nested":["first",2]},"zeta":true}`) ||
|
||||||
|
!strings.Contains(textValue, "api_key_required: true") {
|
||||||
|
t.Fatalf("unexpected text inspection: %s", textValue)
|
||||||
|
}
|
||||||
|
if strings.Contains(textValue, secret) {
|
||||||
|
t.Fatalf("text inspection exposed an environment secret: %s", textValue)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFormatProfileInspectionPreservesEmptyBackendAndRejectsNil(t *testing.T) {
|
||||||
|
output, err := FormatProfileInspection(&promptkit.ProfileInspection{
|
||||||
|
ProfileID: "endpoint-only",
|
||||||
|
EffectiveModelParams: promptkit.ExecutionTarget{
|
||||||
|
Endpoint: "http://127.0.0.1:8000/v1",
|
||||||
|
Model: "fixture-model",
|
||||||
|
},
|
||||||
|
}, OutputFormatJSON)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(output), `"backend_id": ""`) ||
|
||||||
|
!strings.Contains(string(output), `"extra_params": {}`) {
|
||||||
|
t.Fatalf("expected explicit empty backend and object extra params, got %s", output)
|
||||||
|
}
|
||||||
|
if _, err := FormatProfileInspection(nil, OutputFormatText); err == nil {
|
||||||
|
t.Fatal("expected nil profile inspection error")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -14,14 +14,24 @@ import (
|
|||||||
|
|
||||||
var ErrUnknownPreparedRunFormat = errors.New("unknown prepared run format")
|
var ErrUnknownPreparedRunFormat = errors.New("unknown prepared run format")
|
||||||
|
|
||||||
// PreparedRunOutputFormat is the output format for prepared render data.
|
// OutputFormat selects an application-owned textual or JSON representation.
|
||||||
type PreparedRunOutputFormat string
|
type OutputFormat string
|
||||||
|
|
||||||
const (
|
const (
|
||||||
PreparedRunFormatText PreparedRunOutputFormat = "text"
|
OutputFormatText OutputFormat = "text"
|
||||||
PreparedRunFormatJSON PreparedRunOutputFormat = "json"
|
OutputFormatJSON OutputFormat = "json"
|
||||||
|
|
||||||
DefaultPreparedRunOutputFormat PreparedRunOutputFormat = PreparedRunFormatText
|
DefaultOutputFormat OutputFormat = OutputFormatText
|
||||||
|
)
|
||||||
|
|
||||||
|
// PreparedRunOutputFormat is the output format for prepared render data.
|
||||||
|
type PreparedRunOutputFormat = OutputFormat
|
||||||
|
|
||||||
|
const (
|
||||||
|
PreparedRunFormatText = OutputFormatText
|
||||||
|
PreparedRunFormatJSON = OutputFormatJSON
|
||||||
|
|
||||||
|
DefaultPreparedRunOutputFormat = DefaultOutputFormat
|
||||||
)
|
)
|
||||||
|
|
||||||
// PreparedRunFormatter serializes a prepared run without performing use case work.
|
// PreparedRunFormatter serializes a prepared run without performing use case work.
|
||||||
@@ -31,15 +41,20 @@ type PreparedRunFormatter interface {
|
|||||||
|
|
||||||
// ParsePreparedRunOutputFormat parses a format name.
|
// ParsePreparedRunOutputFormat parses a format name.
|
||||||
func ParsePreparedRunOutputFormat(raw string) (PreparedRunOutputFormat, error) {
|
func ParsePreparedRunOutputFormat(raw string) (PreparedRunOutputFormat, error) {
|
||||||
switch PreparedRunOutputFormat(strings.ToLower(strings.TrimSpace(raw))) {
|
return ParseOutputFormat(raw)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseOutputFormat parses the shared inspection and prepared-run format names.
|
||||||
|
func ParseOutputFormat(raw string) (OutputFormat, error) {
|
||||||
|
switch OutputFormat(strings.ToLower(strings.TrimSpace(raw))) {
|
||||||
case "":
|
case "":
|
||||||
return DefaultPreparedRunOutputFormat, nil
|
return DefaultOutputFormat, nil
|
||||||
case PreparedRunFormatText:
|
case OutputFormatText:
|
||||||
return PreparedRunFormatText, nil
|
return OutputFormatText, nil
|
||||||
case PreparedRunFormatJSON:
|
case OutputFormatJSON:
|
||||||
return PreparedRunFormatJSON, nil
|
return OutputFormatJSON, nil
|
||||||
default:
|
default:
|
||||||
return "", fmt.Errorf("%w: %q (supported: %s, %s)", ErrUnknownPreparedRunFormat, raw, PreparedRunFormatText, PreparedRunFormatJSON)
|
return "", fmt.Errorf("%w: %q (supported: %s, %s)", ErrUnknownPreparedRunFormat, raw, OutputFormatText, OutputFormatJSON)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -93,6 +108,9 @@ func (textPreparedRunFormatter) Format(prepared *promptkit.PreparedRun) ([]byte,
|
|||||||
fmt.Fprintf(&b, "prompt: %s\n", prepared.PromptID)
|
fmt.Fprintf(&b, "prompt: %s\n", prepared.PromptID)
|
||||||
fmt.Fprintf(&b, "prompt_version: %s\n", prepared.PromptVersion)
|
fmt.Fprintf(&b, "prompt_version: %s\n", prepared.PromptVersion)
|
||||||
fmt.Fprintf(&b, "selected_profile_id: %s\n", prepared.SelectedProfileID)
|
fmt.Fprintf(&b, "selected_profile_id: %s\n", prepared.SelectedProfileID)
|
||||||
|
if prepared.SelectedBackendID != "" {
|
||||||
|
fmt.Fprintf(&b, "selected_backend_id: %s\n", prepared.SelectedBackendID)
|
||||||
|
}
|
||||||
if prepared.PromptHash != "" {
|
if prepared.PromptHash != "" {
|
||||||
fmt.Fprintf(&b, "prompt_hash: %s\n", prepared.PromptHash)
|
fmt.Fprintf(&b, "prompt_hash: %s\n", prepared.PromptHash)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -22,6 +22,7 @@ func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
|
|||||||
"prompt: prompt.id",
|
"prompt: prompt.id",
|
||||||
"prompt_version: v1",
|
"prompt_version: v1",
|
||||||
"selected_profile_id: local-fast",
|
"selected_profile_id: local-fast",
|
||||||
|
"selected_backend_id: local",
|
||||||
"endpoint: http://llm/v1",
|
"endpoint: http://llm/v1",
|
||||||
"model: gpt-test",
|
"model: gpt-test",
|
||||||
"temperature: 0.4",
|
"temperature: 0.4",
|
||||||
@@ -346,7 +347,9 @@ func samplePreparedRun() *promptkit.PreparedRun {
|
|||||||
PromptVersion: "v1",
|
PromptVersion: "v1",
|
||||||
PromptHash: "prompt-hash",
|
PromptHash: "prompt-hash",
|
||||||
SelectedProfileID: "local-fast",
|
SelectedProfileID: "local-fast",
|
||||||
|
SelectedBackendID: "local",
|
||||||
EffectiveModelParams: promptkit.ExecutionTarget{
|
EffectiveModelParams: promptkit.ExecutionTarget{
|
||||||
|
BackendID: "local",
|
||||||
Endpoint: "http://llm/v1",
|
Endpoint: "http://llm/v1",
|
||||||
Model: "gpt-test",
|
Model: "gpt-test",
|
||||||
Temperature: 0.4,
|
Temperature: 0.4,
|
||||||
|
|||||||
12
testdata/promptkit-v0.9/config.yml
vendored
Normal file
12
testdata/promptkit-v0.9/config.yml
vendored
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
prompt_dir: ./testdata/promptkit-v0.9/prompts
|
||||||
|
profile_dir: ./testdata/promptkit-v0.9/profiles
|
||||||
|
schema_dir: ./testdata/promptkit-v0.9/schemas
|
||||||
|
backends:
|
||||||
|
fixture-custom:
|
||||||
|
endpoint: http://127.0.0.1:11434/v1
|
||||||
|
api_key_env: FIXTURE_BACKEND_API_KEY
|
||||||
|
extra_params:
|
||||||
|
backend_option:
|
||||||
|
enabled: true
|
||||||
|
concurrency_limit: 2
|
||||||
|
queue_capacity: 0
|
||||||
1
testdata/promptkit-v0.9/inputs/source.md
vendored
Normal file
1
testdata/promptkit-v0.9/inputs/source.md
vendored
Normal file
@@ -0,0 +1 @@
|
|||||||
|
The fixture source contains stable, synthetic material.
|
||||||
16
testdata/promptkit-v0.9/profiles/custom-base.yaml
vendored
Normal file
16
testdata/promptkit-v0.9/profiles/custom-base.yaml
vendored
Normal file
@@ -0,0 +1,16 @@
|
|||||||
|
id: custom-base
|
||||||
|
backend: fixture-custom
|
||||||
|
model: fixture-base-model
|
||||||
|
temperature: 0.25
|
||||||
|
max_tokens: 640
|
||||||
|
top_p: 0.9
|
||||||
|
timeout_seconds: 45
|
||||||
|
service_tier: flex
|
||||||
|
reasoning_effort: medium
|
||||||
|
api_key_env: FIXTURE_PROFILE_API_KEY
|
||||||
|
extra_params:
|
||||||
|
routing:
|
||||||
|
order:
|
||||||
|
- primary
|
||||||
|
- fallback
|
||||||
|
allow_fallback: true
|
||||||
4
testdata/promptkit-v0.9/profiles/custom-derived.yml
vendored
Normal file
4
testdata/promptkit-v0.9/profiles/custom-derived.yml
vendored
Normal file
@@ -0,0 +1,4 @@
|
|||||||
|
id: custom-derived
|
||||||
|
base_profile: custom-base
|
||||||
|
model: fixture-derived-model
|
||||||
|
reasoning_effort: high
|
||||||
10
testdata/promptkit-v0.9/profiles/endpoint-only.yaml
vendored
Normal file
10
testdata/promptkit-v0.9/profiles/endpoint-only.yaml
vendored
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
id: endpoint-only
|
||||||
|
endpoint: http://127.0.0.1:8000/v1
|
||||||
|
model: fixture-endpoint-model
|
||||||
|
temperature: 0.1
|
||||||
|
max_tokens: 128
|
||||||
|
top_p: 0.75
|
||||||
|
timeout_seconds: 30
|
||||||
|
extra_params:
|
||||||
|
endpoint_option:
|
||||||
|
nested: true
|
||||||
31
testdata/promptkit-v0.9/prompts/complete-v1.yaml
vendored
Normal file
31
testdata/promptkit-v0.9/prompts/complete-v1.yaml
vendored
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
id: compat.complete
|
||||||
|
version: "1.0.0"
|
||||||
|
description: Exercise the complete Promptkit v0.9 prompt-definition surface.
|
||||||
|
default_profile: custom-derived
|
||||||
|
session_id: 'fixture-{{.topic}}'
|
||||||
|
inputs:
|
||||||
|
- name: source
|
||||||
|
required: true
|
||||||
|
content_type: text/markdown
|
||||||
|
description: Source material rendered by a file-backed message.
|
||||||
|
- name: context
|
||||||
|
required: false
|
||||||
|
content_type: text/plain
|
||||||
|
description: Optional caller context.
|
||||||
|
messages:
|
||||||
|
- role: developer
|
||||||
|
content: 'Use the topic {{.topic}}.'
|
||||||
|
cache_control:
|
||||||
|
type: ephemeral
|
||||||
|
ttl: 1h
|
||||||
|
- role: system
|
||||||
|
content_file: ./messages/complete.system.md
|
||||||
|
- role: user
|
||||||
|
content_file: ./messages/complete.user.md
|
||||||
|
- role: assistant
|
||||||
|
content: Return one JSON object.
|
||||||
|
output:
|
||||||
|
format: json
|
||||||
|
validation_mode: json_schema
|
||||||
|
schema_path: complete.schema.json
|
||||||
|
repair_attempts: 2
|
||||||
10
testdata/promptkit-v0.9/prompts/complete-v2.yml
vendored
Normal file
10
testdata/promptkit-v0.9/prompts/complete-v2.yml
vendored
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
id: compat.complete
|
||||||
|
version: "2.0.0"
|
||||||
|
default_profile: endpoint-only
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: Version two has no declared inputs.
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 1
|
||||||
10
testdata/promptkit-v0.9/prompts/json.yaml
vendored
Normal file
10
testdata/promptkit-v0.9/prompts/json.yaml
vendored
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
id: compat.json
|
||||||
|
version: "1.0.0"
|
||||||
|
default_profile: endpoint-only
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: Return JSON.
|
||||||
|
output:
|
||||||
|
format: json
|
||||||
|
validation_mode: json
|
||||||
|
repair_attempts: 0
|
||||||
1
testdata/promptkit-v0.9/prompts/messages/complete.system.md
vendored
Normal file
1
testdata/promptkit-v0.9/prompts/messages/complete.system.md
vendored
Normal file
@@ -0,0 +1 @@
|
|||||||
|
Use the supplied source without inventing details.
|
||||||
4
testdata/promptkit-v0.9/prompts/messages/complete.user.md
vendored
Normal file
4
testdata/promptkit-v0.9/prompts/messages/complete.user.md
vendored
Normal file
@@ -0,0 +1,4 @@
|
|||||||
|
Topic: {{.topic}}
|
||||||
|
|
||||||
|
Source:
|
||||||
|
{{input "source"}}
|
||||||
10
testdata/promptkit-v0.9/prompts/none.yaml
vendored
Normal file
10
testdata/promptkit-v0.9/prompts/none.yaml
vendored
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
id: compat.none
|
||||||
|
version: "1.0.0"
|
||||||
|
default_profile: endpoint-only
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: Return text.
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
repair_attempts: 0
|
||||||
11
testdata/promptkit-v0.9/schemas/complete.schema.json
vendored
Normal file
11
testdata/promptkit-v0.9/schemas/complete.schema.json
vendored
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"type": "object",
|
||||||
|
"required": ["summary"],
|
||||||
|
"properties": {
|
||||||
|
"summary": {
|
||||||
|
"type": "string"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"additionalProperties": false
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user