Harden backend contracts and documentation

This commit is contained in:
2026-07-29 17:22:55 +00:00
parent ae210b3c26
commit 359b7313f4
16 changed files with 315 additions and 113 deletions

View File

@@ -5,7 +5,8 @@ import (
"gitea.maximumdirect.net/eric/promptkit/internal/domain" "gitea.maximumdirect.net/eric/promptkit/internal/domain"
) )
// BackendOpenRouter identifies Promptkit's built-in OpenRouter backend. // BackendOpenRouter is the reserved ID of Promptkit's built-in OpenRouter
// backend.
const BackendOpenRouter = backend.OpenRouterID const BackendOpenRouter = backend.OpenRouterID
// Backend configures one engine-scoped OpenAI-compatible backend. // Backend configures one engine-scoped OpenAI-compatible backend.

21
doc.go
View File

@@ -2,9 +2,10 @@
// prompt-defined LLM workflows. // prompt-defined LLM workflows.
// //
// Applications construct an [Engine] with [NewEngine], select filesystem or // Applications construct an [Engine] with [NewEngine], select filesystem or
// in-memory sources with options, and call [Engine.Prepare] or [Engine.Run]. // in-memory sources and optional engine-scoped [Backend] registrations, and
// Concrete repositories, validators, and the built-in OpenAI-compatible client // call [Engine.Prepare] or [Engine.Run]. Concrete registries, repositories,
// remain internal implementation details. // validators, and the built-in OpenAI-compatible client remain internal
// implementation details.
// //
// # Concurrency and ownership // # Concurrency and ownership
// //
@@ -12,11 +13,11 @@
// or [ArtifactReader] can therefore receive concurrent calls and must be safe // or [ArtifactReader] can therefore receive concurrent calls and must be safe
// for that use. // for that use.
// //
// NewEngine copies in-memory profiles. Prepare and Run copy request maps, // NewEngine copies in-memory profiles and backend definitions. Prepare and Run
// slices, pointer values, and JSON-compatible extra parameters before using // copy request maps, slices, pointer values, and JSON-compatible extra
// them. Returned values and values passed to extension interfaces are likewise // parameters before using them. Returned values and values passed to extension
// isolated from engine state. Callers own those copies and may mutate them // interfaces are likewise isolated from engine state. Callers own those copies
// after the call that supplied or returned them. // and may mutate them after the call that supplied or returned them.
// //
// # Security and sensitive data // # Security and sensitive data
// //
@@ -42,8 +43,8 @@
// [GenerateResponse], [ExecutionTargetPresence], and the string value types // [GenerateResponse], [ExecutionTargetPresence], and the string value types
// used by those values. // used by those values.
// //
// Construction values, including [Config], [RunRequest], [ArtifactRef], // Construction values, including [Config], [Backend], [RunRequest],
// [ExecutionTargetOverride], [Profile], and // [ArtifactRef], [ExecutionTargetOverride], [Profile], and
// [OpenAICompatibleProfileConfig], do not have stable JSON representations. // [OpenAICompatibleProfileConfig], do not have stable JSON representations.
// Direct API keys are nevertheless excluded from JSON for every public value. // Direct API keys are nevertheless excluded from JSON for every public value.
// //

View File

@@ -33,8 +33,8 @@ engine, err := promptkit.NewEngine(promptkit.Config{
}) })
``` ```
Options support single-file or `fs.FS` sources, in-memory profiles, and Options support single-file or `fs.FS` sources, in-memory profiles,
injected artifact or model clients. Consult the engine-scoped backends, and injected artifact or model clients. Consult the
[constructor and option GoDoc](../../engine.go) for composition, precedence, [constructor and option GoDoc](../../engine.go) for composition, precedence,
validation, and default transport behavior. Source discovery, format validation, and default transport behavior. Source discovery, format
validation, and profile precedence are defined by the validation, and profile precedence are defined by the
@@ -123,6 +123,13 @@ Registrations belong to one engine and custom IDs cannot replace built-ins.
The [`Backend` and `WithBackend` GoDoc](../../backends.go) defines validation, The [`Backend` and `WithBackend` GoDoc](../../backends.go) defines validation,
copying, uniqueness, and request-default behavior. copying, uniqueness, and request-default behavior.
Both file-backed and in-memory profiles select a registration through
`backend` or `Profile.BackendID`. Profile and request endpoint overrides retain
that routing identity. `PreparedRun.SelectedBackendID`,
`RunResult.SelectedBackendID`, and the effective `ExecutionTarget.BackendID`
expose it to consumers and injected model clients. Endpoint-only profiles
remain supported and expose an empty backend ID.
## Credentials ## Credentials
File-backed profiles name an environment variable; in-memory profiles can File-backed profiles name an environment variable; in-memory profiles can
@@ -163,7 +170,8 @@ distinguish invalid construction, invalid requests, absent sources,
source-loading failures, collaborator failures, and operational validation source-loading failures, collaborator failures, and operational validation
failures. Specific request conditions may also match the broader failures. Specific request conditions may also match the broader
`ErrInvalidRequest`, and injected collaborator identities are preserved where `ErrInvalidRequest`, and injected collaborator identities are preserved where
documented. documented. Invalid or duplicate backend registrations match
`ErrInvalidConfig`; selecting an unknown backend matches `ErrProfileLoad`.
## Application Boundary ## Application Boundary

View File

@@ -168,6 +168,10 @@ variable name in `api_key_env`.
Promptkit does not infer a backend from a model or endpoint. Endpoint-only Promptkit does not infer a backend from a model or endpoint. Endpoint-only
profiles remain supported and have no effective backend ID. profiles remain supported and have no effective backend ID.
The engine always provides the built-in `openrouter` ID. Consumers can add
engine-scoped IDs with
[`WithBackend`](../backends.go); exact registration validation belongs to its
GoDoc.
`extra_params` accepts null, booleans, finite numbers, strings, arrays, and `extra_params` accepts null, booleans, finite numbers, strings, arrays, and
objects with string keys. Keys must be non-empty. With the built-in client, objects with string keys. Keys must be non-empty. With the built-in client,
@@ -226,8 +230,9 @@ invalid matching profile is an error and does not fall back. In-memory
Every built-in selects the `openrouter` backend. The engine's built-in backend Every built-in selects the `openrouter` backend. The engine's built-in backend
registry supplies `https://openrouter.ai/api/v1` and the environment-variable registry supplies `https://openrouter.ai/api/v1` and the environment-variable
name `OPENROUTER_API_KEY`, so individual profiles contain only model and name `OPENROUTER_API_KEY`, so individual profiles contain only model and
generation settings. A custom or in-memory profile with the same profile ID generation settings. Built-in profile files do not repeat those connection
takes precedence. values. A custom or in-memory profile with the same profile ID takes
precedence.
| Provider | ID | Model | | Provider | ID | Model |
| --- | --- | --- | | --- | --- | --- |

View File

@@ -13,10 +13,15 @@ that produce these outbound settings.
## Endpoint And Method ## Endpoint And Method
Generation sends an HTTP `POST` with `Content-Type: application/json`. Generation sends an HTTP `POST` with `Content-Type: application/json`.
A non-empty endpoint from the execution target overrides the client's Before the client is called, the engine resolves framework, backend, profile,
configured base URL. After trailing slashes are removed, and request values into one execution target. A non-empty endpoint from that
`/chat/completions` is appended. Generation fails before sending when neither target overrides the client's configured base URL. After trailing slashes are
source supplies an endpoint. removed, `/chat/completions` is appended. Generation fails before sending when
neither source supplies an endpoint.
The target's backend ID is routing metadata for prepared values, results, and
injected clients. The built-in client does not derive the URL from that ID and
does not serialize it in the provider request.
## Authentication ## Authentication
@@ -26,6 +31,12 @@ the client reads that variable and requires a non-empty value. The selected
key is sent as `Authorization: Bearer <key>`. No authorization header is sent key is sent as `Authorization: Bearer <key>`. No authorization header is sent
when neither mechanism is configured. when neither mechanism is configured.
The target contains the already resolved environment-variable name: an
explicit request override takes precedence over profile metadata, which takes
precedence over the backend default. Only the name reaches prepared metadata;
the environment value is read just before the provider call and is never added
to the JSON body.
## Request Body ## Request Body
The request body always contains `model` and `messages`. The execution The request body always contains `model` and `messages`. The execution
@@ -47,9 +58,11 @@ The client conditionally includes:
- `response_format` for JSON Schema structured output, including its name, - `response_format` for JSON Schema structured output, including its name,
strict flag, and schema document. strict flag, and schema document.
Extra parameters are merged directly into the top-level body after JSON The engine resolves backend, profile, and request extra-parameter maps by
serialization is verified. Empty keys and collisions with these reserved whole-map replacement rather than key merging. The resulting effective map is
fields are rejected before any provider call: then merged directly into the top-level body after JSON serialization is
verified. Empty keys and collisions with these reserved fields are rejected
before any provider call:
- `model` - `model`
- `session_id` - `session_id`
@@ -61,6 +74,9 @@ fields are rejected before any provider call:
- `reasoning_effort` - `reasoning_effort`
- `response_format` - `response_format`
`backend_id`, `api_key_env`, and resolved credential values are not provider
request fields.
## Response Handling ## Response Handling
Any 2xx response is decoded as an OpenAI-compatible chat response. The client Any 2xx response is decoded as an OpenAI-compatible chat response. The client

View File

@@ -20,6 +20,11 @@ orchestration. `OpenAICompatibleClient` is the built-in implementation. It
uses internal domain values for rendered prompts, execution targets, uses internal domain values for rendered prompts, execution targets,
structured output, responses, and token usage. structured output, responses, and token usage.
The runner supplies a fully resolved target after applying backend, profile,
and request precedence. The client uses its endpoint, credential metadata,
generation fields, and extra parameters. `BackendID` remains routing metadata
for the generation boundary and is not mapped into the provider payload.
Construction validates the configured base URL and clones any supplied Construction validates the configured base URL and clones any supplied
`http.Client` so Promptkit can apply its timeout default without mutating the `http.Client` so Promptkit can apply its timeout default without mutating the
caller's client. Generation then: caller's client. Generation then:
@@ -51,5 +56,7 @@ The
[OpenAI-compatible client tests](../../internal/llm/openai_compatible_client_test.go) [OpenAI-compatible client tests](../../internal/llm/openai_compatible_client_test.go)
own configuration, client cloning, deterministic deadline precedence, own configuration, client cloning, deterministic deadline precedence,
authentication, request and response mapping, malformed data, error identity, authentication, request and response mapping, malformed data, error identity,
cancellation, and response-body suppression. They use local test servers and cancellation, and response-body suppression. The root transport contract test
test transports; the default suite makes no live or paid provider requests. also verifies that resolved backend settings reach this client without
serializing backend identity. All use local test servers or test transports;
the default suite makes no live or paid provider requests.

View File

@@ -11,21 +11,21 @@ contributor workflow and validation.
| Component | Implemented responsibility | References | | Component | Implemented responsibility | References |
| --- | --- | --- | | --- | --- | --- |
| Root `promptkit` package | Provides the supported engine facade, source and injection options, public request and result values, built-in profile construction, extension interfaces, value conversion, redacted formatting, and public error mapping. | [Package GoDoc](../../doc.go), [engine assembly](../../engine.go) | | Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request and result values, profile construction, extension interfaces, value conversion, redacted formatting, and public error mapping. | [Package GoDoc](../../doc.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
| `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) | | `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) | | `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
| `internal/backend` | Validates and defensively copies immutable OpenAI-compatible backend definitions, supplies the built-in OpenRouter definition, and owns the shared reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) | | `internal/backend` | Constructs each engine's immutable registry from the built-in OpenRouter definition and consumer additions, validates and defensively copies definitions, and owns the shared reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) |
| `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, profiles, execution targets, rendering, generation, and validation. | [Domain declarations](../../internal/domain/domain.go) | | `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, profiles, execution targets, rendering, generation, and validation. | [Domain declarations](../../internal/domain/domain.go) |
| `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) | | `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) |
| `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) | | `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |
| `internal/promptdef` | Loads strictly decoded, validated prompt definitions from filesystem and `fs.FS` sources, including version selection and contained file-backed message content. | [Framework formats](../formats.md), [prompt-definition repository](../../internal/promptdef/filesystem_repository.go) | | `internal/promptdef` | Loads strictly decoded, validated prompt definitions from filesystem and `fs.FS` sources, including version selection and contained file-backed message content. | [Framework formats](../formats.md), [prompt-definition repository](../../internal/promptdef/filesystem_repository.go) |
| `internal/profile` | Loads strictly decoded, validated execution profiles from filesystem and `fs.FS` sources and composes repositories with error-preserving fallback. | [Framework formats](../formats.md), [profile repositories](../../internal/profile/filesystem_repository.go) | | `internal/profile` | Loads strictly decoded, validated execution profiles, including backend selection, from filesystem and `fs.FS` sources and composes repositories with error-preserving fallback. | [Framework formats](../formats.md), [profile repositories](../../internal/profile/filesystem_repository.go) |
| `internal/profile/builtin` | Embeds the built-in execution profile catalog and combines it with an optional primary repository. | [Built-in catalog](../formats.md#built-in-profile-catalog), [repository](../../internal/profile/builtin/repository.go) | | `internal/profile/builtin` | Embeds the built-in profile catalog, whose entries select OpenRouter, and combines it with an optional primary repository. | [Built-in catalog](../formats.md#built-in-profile-catalog), [repository](../../internal/profile/builtin/repository.go) |
| `internal/prompt` | Renders prompt messages from Go templates with artifact, variable, session, and cache-control data. | [Go-template renderer](../../internal/prompt/go_renderer.go) | | `internal/prompt` | Renders prompt messages from Go templates with artifact, variable, session, and cache-control data. | [Go-template renderer](../../internal/prompt/go_renderer.go) |
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) | | `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
| `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) | | `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests, response decoding, authentication, and deadline handling. | [Internal model client](llm.md) | | `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including response decoding, authentication, and deadline handling. | [Internal model client](llm.md) |
| `internal/usecase` | Coordinates preparation and execution across internal sources, rendering, artifact loading, generation, validation, and optional repair. | [Internal runner](runner.md) | | `internal/usecase` | Resolves backend, profile, and request settings and coordinates preparation and execution across internal sources, rendering, artifact loading, generation, validation, and optional repair. | [Internal runner](runner.md) |
The root package assembles these internal components without exposing their The root package assembles these internal components without exposing their
representations. Consumers depend only on the root facade. representations. Consumers depend only on the root facade.

View File

@@ -18,10 +18,11 @@ and override semantics consumed by the runner.
`Runner` coordinates narrow internal interfaces for prompt definitions, `Runner` coordinates narrow internal interfaces for prompt definitions,
profiles, backend resolution, artifacts, rendering, model generation, and profiles, backend resolution, artifacts, rendering, model generation, and
validation. The root engine supplies an immutable built-in backend registry. validation. The root engine supplies one immutable registry containing the
Schema documents are loaded through the validator's optional schema-loader built-in backend and validated consumer additions. Schema documents are loaded
interface. An output repairer can be injected internally, but the ordinary through the validator's optional schema-loader interface. An output repairer
runner constructor does not enable one. can be injected internally, but the ordinary runner constructor does not
enable one.
Each invocation carries its state in request, prepared-run, and result values. Each invocation carries its state in request, prepared-run, and result values.
The runner has no durable run or session store. The runner has no durable run or session store.
@@ -53,6 +54,10 @@ apply in that order. A profile requiring a direct key clears an inherited
backend environment name unless the request supplies its own name. Secret backend environment name unless the request supplies its own name. Secret
values remain excluded from serialized metadata. values remain excluded from serialized metadata.
The registry is read-only after engine construction. Concurrent `Prepare` and
`Run` calls resolve independent defensive backend values and keep all
invocation state local.
## Run Flow ## Run Flow
`Run` calls `Prepare` rather than maintaining a second preparation path. It `Run` calls `Prepare` rather than maintaining a second preparation path. It

View File

@@ -36,7 +36,8 @@ profile selects `openrouter` and inherits its endpoint and credential
environment-variable name from the built-in backend registry rather than environment-variable name from the built-in backend registry rather than
repeating those values. Profile behavior is owned by the repeating those values. Profile behavior is owned by the
[profile repository tests](../../internal/profile/repository_test.go), while [profile repository tests](../../internal/profile/repository_test.go), while
catalog completeness, backend-selection invariant, duplicate IDs, and overlay behavior are owned by the catalog completeness, the backend-selection invariant, duplicate IDs, and
overlay behavior are owned by the
[built-in repository tests](../../internal/profile/builtin/repository_test.go). [built-in repository tests](../../internal/profile/builtin/repository_test.go).
## Ordinary Artifacts ## Ordinary Artifacts

View File

@@ -47,17 +47,19 @@ The `examples/go-library/prepare` and `examples/go-library/run` packages are
maintained downstream consumers of the root facade. They do not expose library maintained downstream consumers of the root facade. They do not expose library
packages or participate in internal assembly. packages or participate in internal assembly.
The root facade assembles the internal repositories, renderer, validator, The root facade assembles one immutable backend registry, the internal
outbound client, and use-case runner while translating public values and repositories, renderer, validator, outbound client, and use-case runner while
errors at the library boundary. The defaults and renderer depend on the domain translating public values and errors at the library boundary. The registry
model. Prompt-definition and profile repositories use the domain model, file contains built-ins plus validated engine-scoped consumer additions. The
catalog, and YAML decoder. The built-in profile repository supplies an defaults and renderer depend on the domain model. Prompt-definition and
embedded `fs.FS` to the profile package. Artifact reading uses the domain model profile repositories use the domain model, file catalog, and YAML decoder. The
and application-neutral defaults. Validation uses the domain model, file built-in profile repository supplies an embedded `fs.FS` to the profile
catalog, and JSON Schema implementation. The model client uses the domain package. Artifact reading uses the domain model and application-neutral
model, application-neutral defaults, and an injected or standard-library HTTP defaults. Validation uses the domain model, file catalog, and JSON Schema
client. The use-case runner depends on the narrow interfaces owned by each implementation. The model client uses the domain model, application-neutral
internal component. defaults, and an injected or standard-library HTTP client. The use-case runner
depends on the narrow interfaces owned by each internal component, including
backend lookup.
The current implementation follows this dependency direction: The current implementation follows this dependency direction:
@@ -74,11 +76,11 @@ downstream consumers, including Scriptorium
narrow injected abstractions narrow injected abstractions
``` ```
The backend registry depends on the domain model, and the model client reuses The backend registry depends on the domain model, has no mutation API after
its OpenAI-compatible reserved request-field rule. The facade coordinates construction, and shares its OpenAI-compatible reserved request-field rule
internal components and adapts the supported public extension interfaces to with the model client. The facade coordinates internal components and adapts
narrow internal abstractions. Internal components must not depend on consumers the supported public extension interfaces to narrow internal abstractions.
or on Scriptorium. Internal components must not depend on consumers or on Scriptorium.
## Repository And Consumer Boundary ## Repository And Consumer Boundary

View File

@@ -1,5 +1,7 @@
# Extensible LLM Backend Registry # Extensible LLM Backend Registry
**Status:** Complete.
## Purpose ## Purpose
This roadmap defines the scope and target end state for formal This roadmap defines the scope and target end state for formal

View File

@@ -428,6 +428,8 @@ external endpoint. Keep lower-level registry validation cases in
## Stage 4 — Contract Hardening And Documentation Completion ## Stage 4 — Contract Hardening And Documentation Completion
**Status:** Complete.
### Goal ### Goal
Audit the completed feature across public, profile, use-case, and transport Audit the completed feature across public, profile, use-case, and transport

View File

@@ -25,7 +25,8 @@ import (
) )
// ErrInvalidConfig identifies invalid engine construction, including missing // ErrInvalidConfig identifies invalid engine construction, including missing
// required configuration, invalid options, and a nil Engine receiver. // required configuration, invalid options or backend registrations, and a nil
// Engine receiver.
var ErrInvalidConfig = errors.New("invalid engine configuration") var ErrInvalidConfig = errors.New("invalid engine configuration")
var ( var (
@@ -48,8 +49,8 @@ var (
// ErrPromptNotFound. // ErrPromptNotFound.
ErrPromptLoad = errors.New("failed to load prompt definition") ErrPromptLoad = errors.New("failed to load prompt definition")
// ErrProfileLoad identifies a failure to read, decode, validate, or select // ErrProfileLoad identifies a failure to read, decode, validate, or select
// an execution profile, except for the not-found case represented by // an execution profile or resolve its backend, except for the profile
// ErrProfileNotFound. // not-found case represented by ErrProfileNotFound.
ErrProfileLoad = errors.New("failed to load execution profile") ErrProfileLoad = errors.New("failed to load execution profile")
// ErrAPIKeyEnvMissing identifies an APIKeyEnv whose environment variable is // ErrAPIKeyEnvMissing identifies an APIKeyEnv whose environment variable is
// unset or empty when no direct RunRequest.APIKey takes precedence. Such an // unset or empty when no direct RunRequest.APIKey takes precedence. Such an
@@ -407,11 +408,12 @@ func fileSource(name string) (fs.FS, string, error) {
// Prepare resolves and renders a prompt request without calling an LLM. // Prepare resolves and renders a prompt request without calling an LLM.
// //
// Prepare selects the prompt and profile, resolves effective execution // Prepare selects the prompt and profile, resolves any selected backend and
// settings and the output contract, loads and hashes inputs, loads structured // effective execution settings, resolves the output contract, loads and hashes
// output schema metadata when required, and renders the session ID and // inputs, loads structured-output schema metadata when required, and renders
// messages. The returned PreparedRun is owned by the caller and never contains // the session ID and messages. The returned PreparedRun is owned by the caller
// a resolved API-key value, model output, or validation result. // and never contains a resolved API-key value, model output, or validation
// result.
// //
// A nil Engine returns an error matching ErrInvalidConfig. Request and // A nil Engine returns an error matching ErrInvalidConfig. Request and
// preparation failures may match ErrInvalidRequest, ErrPromptNotFound, // preparation failures may match ErrInvalidRequest, ErrPromptNotFound,

View File

@@ -755,6 +755,92 @@ func TestRunUsesDirectAPIKeyWithDefaultLLMClient(t *testing.T) {
} }
} }
func TestRunUsesResolvedBackendWithBuiltInLLMClient(t *testing.T) {
const (
backendID = "local-test"
envName = "PROMPTKIT_BACKEND_TRANSPORT_KEY"
apiKey = "synthetic-backend-key"
)
t.Setenv(envName, apiKey)
var (
gotAuth string
gotBody map[string]any
)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotAuth = r.Header.Get("Authorization")
if r.URL.Path != "/v1/chat/completions" {
t.Errorf("unexpected path: %s", r.URL.Path)
}
if err := json.NewDecoder(r.Body).Decode(&gotBody); err != nil {
t.Errorf("decode request body: %v", err)
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{
"choices": [{"message": {"role": "assistant", "content": "# Summary\n\nDone."}}],
"usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7}
}`))
}))
defer server.Close()
engine, err := promptkit.NewEngine(promptkit.Config{
PromptDir: frameworkPromptDir,
SchemaDir: frameworkSchemaDir,
},
promptkit.WithBackend(promptkit.Backend{
ID: backendID,
Endpoint: server.URL + "/v1",
APIKeyEnv: envName,
ExtraParams: map[string]any{
"provider": "synthetic",
},
}),
promptkit.WithProfiles(promptkit.Profile{
ID: "backend-transport",
BackendID: backendID,
Model: "test-model",
}),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
result, err := engine.Run(context.Background(), promptkit.RunRequest{
PromptID: frameworkMarkdownSummaryPromptID,
ProfileID: "backend-transport",
Inputs: map[string]promptkit.ArtifactRef{
"transcript": promptkit.Inline("Rin opens the gate."),
"glossary": promptkit.Inline("gate: A guarded passage."),
},
})
if err != nil {
t.Fatalf("run with resolved backend: %v", err)
}
if gotAuth != "Bearer "+apiKey {
t.Fatalf("unexpected Authorization header: %q", gotAuth)
}
if gotBody["model"] != "test-model" || gotBody["provider"] != "synthetic" {
t.Fatalf("backend defaults did not reach provider payload: %#v", gotBody)
}
for _, field := range []string{"backend_id", "api_key_env"} {
if _, ok := gotBody[field]; ok {
t.Fatalf("internal metadata field %q was serialized to provider payload: %#v", field, gotBody)
}
}
bodyJSON, err := json.Marshal(gotBody)
if err != nil {
t.Fatalf("marshal captured provider payload: %v", err)
}
if strings.Contains(string(bodyJSON), apiKey) {
t.Fatalf("credential value was serialized to provider payload: %s", bodyJSON)
}
if result.SelectedBackendID != backendID ||
result.EffectiveModelParams.Endpoint != server.URL+"/v1" ||
result.EffectiveModelParams.APIKeyEnv != envName {
t.Fatalf("unexpected resolved backend metadata: %+v", result)
}
}
func TestPrepareDirectAPIKeyBypassesMissingEnvWithoutLeakingOrHashing(t *testing.T) { func TestPrepareDirectAPIKeyBypassesMissingEnvWithoutLeakingOrHashing(t *testing.T) {
const missingEnv = "PROMPTKIT_PUBLIC_PREPARE_MISSING" const missingEnv = "PROMPTKIT_PUBLIC_PREPARE_MISSING"
const firstKey = "first-direct-key" const firstKey = "first-direct-key"

View File

@@ -28,39 +28,94 @@ func TestPreparedRunJSONOmitsZeroTimingValues(t *testing.T) {
} }
func TestBackendIdentityJSONNamesAndOmission(t *testing.T) { func TestBackendIdentityJSONNamesAndOmission(t *testing.T) {
values := []struct { t.Run("execution target round trip", func(t *testing.T) {
name string value := promptkit.ExecutionTarget{BackendID: promptkit.BackendOpenRouter}
value any
field string
}{
{name: "execution target", value: promptkit.ExecutionTarget{BackendID: promptkit.BackendOpenRouter}, field: "backend_id"},
{name: "prepared run", value: promptkit.PreparedRun{SelectedBackendID: promptkit.BackendOpenRouter}, field: "selected_backend_id"},
{name: "run result", value: promptkit.RunResult{SelectedBackendID: promptkit.BackendOpenRouter}, field: "selected_backend_id"},
}
for _, tt := range values {
t.Run(tt.name, func(t *testing.T) {
payload, err := json.Marshal(tt.value)
if err != nil {
t.Fatalf("marshal populated value: %v", err)
}
var object map[string]any
if err := json.Unmarshal(payload, &object); err != nil {
t.Fatalf("decode populated value: %v", err)
}
if object[tt.field] != promptkit.BackendOpenRouter {
t.Fatalf("expected %s=%q, got %s", tt.field, promptkit.BackendOpenRouter, payload)
}
})
}
emptyValues := []any{promptkit.ExecutionTarget{}, promptkit.PreparedRun{}, promptkit.RunResult{}}
for _, value := range emptyValues {
payload, err := json.Marshal(value) payload, err := json.Marshal(value)
if err != nil { if err != nil {
t.Fatalf("marshal empty value: %v", err) t.Fatalf("marshal execution target: %v", err)
}
var decoded promptkit.ExecutionTarget
if err := json.Unmarshal(payload, &decoded); err != nil {
t.Fatalf("unmarshal execution target: %v", err)
}
if decoded.BackendID != value.BackendID {
t.Fatalf("backend identity did not round trip: got %q want %q", decoded.BackendID, value.BackendID)
}
})
t.Run("prepared run round trip", func(t *testing.T) {
value := promptkit.PreparedRun{SelectedBackendID: promptkit.BackendOpenRouter}
payload, err := json.Marshal(value)
if err != nil {
t.Fatalf("marshal prepared run: %v", err)
}
var decoded promptkit.PreparedRun
if err := json.Unmarshal(payload, &decoded); err != nil {
t.Fatalf("unmarshal prepared run: %v", err)
}
if decoded.SelectedBackendID != value.SelectedBackendID {
t.Fatalf("backend identity did not round trip: got %q want %q", decoded.SelectedBackendID, value.SelectedBackendID)
}
})
t.Run("run result round trip", func(t *testing.T) {
value := promptkit.RunResult{SelectedBackendID: promptkit.BackendOpenRouter}
payload, err := json.Marshal(value)
if err != nil {
t.Fatalf("marshal run result: %v", err)
}
var decoded promptkit.RunResult
if err := json.Unmarshal(payload, &decoded); err != nil {
t.Fatalf("unmarshal run result: %v", err)
}
if decoded.SelectedBackendID != value.SelectedBackendID {
t.Fatalf("backend identity did not round trip: got %q want %q", decoded.SelectedBackendID, value.SelectedBackendID)
}
})
payload, err := json.Marshal(promptkit.ExecutionTarget{})
if err != nil {
t.Fatalf("marshal empty execution target: %v", err)
}
if strings.Contains(string(payload), `"backend_id"`) {
t.Fatalf("empty backend identity was not omitted: %s", payload)
}
}
func TestEndpointOnlyProfileOmitsBackendIdentityFromStableJSON(t *testing.T) {
client := &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}}
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
promptkit.WithProfiles(promptkit.Profile{
ID: "profile", Endpoint: "http://example.test/v1", Model: "model",
}),
promptkit.WithLLMClient(client),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
if err != nil {
t.Fatalf("prepare endpoint-only profile: %v", err)
}
result, err := engine.Run(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
if err != nil {
t.Fatalf("run endpoint-only profile: %v", err)
}
if prepared.SelectedBackendID != "" ||
prepared.EffectiveModelParams.BackendID != "" ||
result.SelectedBackendID != "" ||
result.EffectiveModelParams.BackendID != "" {
t.Fatalf("endpoint-only profile acquired backend identity: prepared=%+v result=%+v", prepared, result)
}
for _, value := range []any{prepared, result} {
payload, err := json.Marshal(value)
if err != nil {
t.Fatalf("marshal endpoint-only value: %v", err)
} }
if strings.Contains(string(payload), `"backend_id"`) || strings.Contains(string(payload), `"selected_backend_id"`) { if strings.Contains(string(payload), `"backend_id"`) || strings.Contains(string(payload), `"selected_backend_id"`) {
t.Fatalf("empty backend identity was not omitted: %s", payload) t.Fatalf("endpoint-only backend identity was not omitted: %s", payload)
} }
} }
} }
@@ -537,10 +592,14 @@ func TestRepeatedOptionsUseLastValueInEachCategory(t *testing.T) {
func TestEngineSupportsConcurrentPrepareAndRun(t *testing.T) { func TestEngineSupportsConcurrentPrepareAndRun(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{}, engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."), promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
promptkit.WithProfiles(promptkit.Profile{ promptkit.WithBackend(promptkit.Backend{
ID: "profile", ID: "concurrent",
Endpoint: "http://example.test/v1", Endpoint: "http://example.test/v1",
Model: "model", }),
promptkit.WithProfiles(promptkit.Profile{
ID: "profile",
BackendID: "concurrent",
Model: "model",
}), }),
promptkit.WithLLMClient(countingLLMClient{}), promptkit.WithLLMClient(countingLLMClient{}),
) )

View File

@@ -101,8 +101,8 @@ type RunRequest struct {
// Vars supplies Go-template data for messages and the session ID. Nil and // Vars supplies Go-template data for messages and the session ID. Nil and
// empty maps are equivalent. // empty maps are equivalent.
Vars map[string]string Vars map[string]string
// Execution optionally overrides individual profile execution settings. // Execution optionally overrides individual execution settings. Nil uses
// Nil uses the selected profile over framework defaults. // the selected profile over its backend, when any, and framework defaults.
Execution *ExecutionTargetOverride Execution *ExecutionTargetOverride
// Validation optionally replaces the prompt's complete output contract. It // Validation optionally replaces the prompt's complete output contract. It
// does not merge individual fields. Nil uses the prompt contract. // does not merge individual fields. Nil uses the prompt contract.
@@ -126,8 +126,8 @@ type PreparedRun struct {
// SelectedProfileID is the explicit request profile or prompt default that // SelectedProfileID is the explicit request profile or prompt default that
// supplied execution settings. // supplied execution settings.
SelectedProfileID string `json:"selected_profile_id"` SelectedProfileID string `json:"selected_profile_id"`
// SelectedBackendID is the selected profile's normalized backend ID. It is // SelectedBackendID equals EffectiveModelParams.BackendID. It is empty for
// empty for an endpoint-only profile. // an endpoint-only profile.
SelectedBackendID string `json:"selected_backend_id,omitempty"` SelectedBackendID string `json:"selected_backend_id,omitempty"`
// EffectiveModelParams contains framework defaults overlaid by the selected // EffectiveModelParams contains framework defaults overlaid by the selected
// backend, profile, and then request overrides. It excludes resolved API-key // backend, profile, and then request overrides. It excludes resolved API-key
@@ -184,8 +184,8 @@ type RunResult struct {
RenderedPromptHash string `json:"rendered_prompt_hash"` RenderedPromptHash string `json:"rendered_prompt_hash"`
// SelectedProfileID identifies the profile used for execution. // SelectedProfileID identifies the profile used for execution.
SelectedProfileID string `json:"selected_profile_id"` SelectedProfileID string `json:"selected_profile_id"`
// SelectedBackendID is the selected profile's normalized backend ID. It is // SelectedBackendID equals EffectiveModelParams.BackendID. It is empty for
// empty for an endpoint-only profile. // an endpoint-only profile.
SelectedBackendID string `json:"selected_backend_id,omitempty"` SelectedBackendID string `json:"selected_backend_id,omitempty"`
// ModelName is the effective model name and equals // ModelName is the effective model name and equals
// EffectiveModelParams.Model. // EffectiveModelParams.Model.
@@ -268,7 +268,8 @@ type ArtifactReader interface {
type ExecutionTarget struct { type ExecutionTarget struct {
// BackendID is the effective routing identity selected by the profile. It // BackendID is the effective routing identity selected by the profile. It
// remains unchanged when a profile or request overrides Endpoint and is // remains unchanged when a profile or request overrides Endpoint and is
// empty for endpoint-only profiles. // empty for endpoint-only profiles. It is supplied to injected LLMClient
// implementations as part of the effective target.
BackendID string `json:"backend_id,omitempty"` BackendID string `json:"backend_id,omitempty"`
// Endpoint is the model-provider base URL. // Endpoint is the model-provider base URL.
Endpoint string `json:"endpoint"` Endpoint string `json:"endpoint"`
@@ -298,13 +299,15 @@ type ExecutionTarget struct {
// ExecutionTargetOverride represents per-request runtime setting overrides and // ExecutionTargetOverride represents per-request runtime setting overrides and
// has no stable JSON representation. // has no stable JSON representation.
// //
// Non-empty string fields replace profile values. Non-nil numeric pointers // Non-empty string fields replace profile and backend values. Non-nil numeric
// replace profile values and preserve explicit zero. A non-empty ExtraParams // pointers replace profile values and preserve explicit zero. A non-empty
// map replaces the complete profile map rather than merging keys. Empty string // ExtraParams map replaces the complete profile or backend map rather than
// fields, nil pointers, and a nil or empty ExtraParams map inherit the selected // merging keys. Empty string fields, nil pointers, and a nil or empty
// profile over framework defaults. // ExtraParams map inherit the selected profile over its backend, when any, and
// framework defaults.
type ExecutionTargetOverride struct { type ExecutionTargetOverride struct {
// Endpoint replaces the profile endpoint when non-empty. // Endpoint replaces the profile or backend endpoint when non-empty without
// changing the effective BackendID.
Endpoint string Endpoint string
// Model replaces the profile model when non-empty. // Model replaces the profile model when non-empty.
Model string Model string
@@ -322,12 +325,13 @@ type ExecutionTargetOverride struct {
// ReasoningEffort replaces the profile value when non-blank. An empty value // ReasoningEffort replaces the profile value when non-blank. An empty value
// cannot clear a profile setting. // cannot clear a profile setting.
ReasoningEffort string ReasoningEffort string
// APIKeyEnv replaces the profile environment-variable name when non-blank. // APIKeyEnv replaces the profile or backend environment-variable name when
// A direct RunRequest.APIKey still takes precedence over environment lookup. // non-blank. A direct RunRequest.APIKey still takes precedence over
// environment lookup.
APIKeyEnv string APIKeyEnv string
// ExtraParams, when non-empty, replaces the profile map. Values must be // ExtraParams, when non-empty, replaces the complete profile or backend map.
// JSON-compatible: nil, booleans, finite numbers, strings, arrays or slices, // Values must be JSON-compatible: nil, booleans, finite numbers, strings,
// and maps with non-empty string keys. Cycles are invalid. // arrays or slices, and maps with non-empty string keys. Cycles are invalid.
ExtraParams map[string]any ExtraParams map[string]any
} }
@@ -346,7 +350,8 @@ type Profile struct {
// ID is the required non-blank profile identifier. WithProfiles trims it. // ID is the required non-blank profile identifier. WithProfiles trims it.
ID string ID string
// BackendID optionally selects an engine backend. WithProfiles trims it. // BackendID optionally selects an engine backend. WithProfiles trims it.
// Backend membership is checked when a request selects the profile. // Backend membership is checked when a request selects the profile; an
// unknown ID makes preparation fail with ErrProfileLoad.
BackendID string BackendID string
// Endpoint is the model-provider base URL. It is required only when // Endpoint is the model-provider base URL. It is required only when
// BackendID is blank and otherwise overrides the backend endpoint when // BackendID is blank and otherwise overrides the backend endpoint when
@@ -373,8 +378,8 @@ type Profile struct {
// supplies ExecutionTargetOverride.APIKeyEnv. It does not store a credential. // supplies ExecutionTargetOverride.APIKeyEnv. It does not store a credential.
APIKeyRequired bool APIKeyRequired bool
// ExtraParams contains provider-specific JSON-compatible values. An empty // ExtraParams contains provider-specific JSON-compatible values. An empty
// map inherits framework defaults. WithProfiles validates and deeply copies // map inherits backend request defaults, when any. WithProfiles validates
// it during NewEngine. // and deeply copies it during NewEngine.
ExtraParams map[string]any ExtraParams map[string]any
} }