9 Commits

31 changed files with 1590 additions and 954 deletions

View File

@@ -33,10 +33,13 @@ boundary and constraints that framework work must preserve.
## Release Guidance
Consumers upgrading from `v0.5.0` to `v0.6.0` should read the
[v0.6.0 changelog and migration guide](docs/releases/v0.6.0.md).
Consumers upgrading from `v0.6.0` to `v0.7.0` should read the
[v0.7.0 changelog and migration guide](docs/releases/v0.7.0.md).
Earlier adopters can consult the
[v0.6.0 changelog and migration guide](docs/releases/v0.6.0.md).
Consumers upgrading from `v0.4.0` to `v0.5.0` can consult the
[v0.5.0 changelog and migration guide](docs/releases/v0.5.0.md).
Consumers upgrading from `v0.3.0` to `v0.4.0` should read the

View File

@@ -30,9 +30,12 @@ type Backend struct {
// requires an absolute HTTP or HTTPS URL with a host and without user
// information, a query string, or a fragment. Paths are allowed.
Endpoint string
// APIKeyEnv optionally names the environment variable containing the API
// key. NewEngine trims it and requires the portable form
// [A-Za-z_][A-Za-z0-9_]*. Store only the name, never a credential value.
// APIKeyEnv optionally names an environment lookup source for an API key.
// NewEngine trims it and requires the portable form [A-Za-z_][A-Za-z0-9_]*.
// A direct RunRequest.APIKey takes precedence. When no usable credential is
// available, the built-in client omits Authorization; injected clients own
// their own credential-resolution behavior. Store only the name, never a
// credential value.
APIKeyEnv string
// ExtraParams contains backend-wide request defaults. Values must be
// JSON-compatible, finite, acyclic, and keyed by non-empty strings. Keys

View File

@@ -189,6 +189,26 @@ For programmatic profiles,
[`OpenAICompatibleProfile`](../../profiles.go) converts ordinary
OpenAI-compatible settings into a value accepted by `WithProfiles`.
### Alias A Built-In Profile
Give an application-owned profile ID a built-in base when prompts should select
the application ID while inheriting the built-in target. The child can override
only the setting it owns:
```go
promptkit.WithProfiles(promptkit.Profile{
ID: "weather-light",
BaseProfileID: "deepseek-4-flash",
ReasoningEffort: "high",
})
```
Select `weather-light` in a prompt or `RunRequest.ProfileID`; it remains the
reported selected profile. See the [profile inheritance format
reference](../formats.md#profile-inheritance) and the
[`Profile` GoDoc](../../types.go) for exact lookup, merging, and validation
behavior.
### Use The Rakestrawhome Built-In Profile
Set `RAKESTRAWHOME_INFERENCE_API_KEY` in the application environment, then
@@ -231,7 +251,7 @@ if err != nil {
target := inspection.EffectiveModelParams
if target.APIKeyEnv != "" {
// Apply application policy for the named environment variable.
// This is a configured optional environment lookup source.
} else if inspection.APIKeyRequired {
// Arrange a direct credential before later execution.
}
@@ -240,9 +260,11 @@ if target.APIKeyEnv != "" {
Use this configuration-time boundary when only the profile and its target need
checking. Use `Prepare` when the application also needs prompt, input, schema,
or rendering work; use prepared execution when that work must remain tied to a
later execution. Inspection reports credential requirements but leaves the
timing of credential enforcement to the application. The method's
[GoDoc](../../engine.go) owns its exact result and error contract.
later execution. A reported `APIKeyEnv` is a configured optional source, while
`APIKeyRequired` is the explicit local requirement. The
[credential format reference](../formats.md#credentials) and the method's
[GoDoc](../../engine.go) own the exact precedence, timing, result, and error
contracts.
### Set A Per-Run Session And Reasoning

View File

@@ -173,9 +173,19 @@ extra_params:
provider_option: enabled
```
A derived profile can use a named base and override only the settings it owns:
```yaml
id: local-summary-fast
base_profile: local-summary
timeout_seconds: 30
reasoning_effort: low
```
| Field | Required | Meaning |
| --- | --- | --- |
| `id` | yes | Profile identifier, trimmed before selection and publication. It must be non-empty after trimming and unique within one source after normalization. |
| `base_profile` | no | One optional parent profile ID. A derived profile may inherit target fields from it. |
| `backend` | unless `endpoint` is present | Backend registry ID. It is trimmed and registry membership is checked when the profile is prepared or inspected. |
| `endpoint` | unless `backend` is present | OpenAI-compatible base URL, including an API version path when required. A nonempty value is trimmed and must be absolute HTTP or HTTPS with a host and without user information, a query, or a fragment. When both connection fields are present, this overrides the backend endpoint without changing backend identity. |
| `model` | yes | Non-empty provider model name. |
@@ -185,12 +195,17 @@ extra_params:
| `timeout_seconds` | no | Per-generation deadline in whole seconds; integer zero or greater. |
| `service_tier` | no | Provider-specific request tier. |
| `reasoning_effort` | no | Provider-specific reasoning setting. |
| `api_key_env` | no | Name of an environment variable containing the API key. |
| `api_key_env` | no | Optional environment-variable lookup source for an API key. |
| `extra_params` | no | JSON-compatible provider-specific outbound fields. |
Raw `api_key` is prohibited in profile YAML. Store only an environment
variable name in `api_key_env`.
A standalone profile must provide a model and at least one of `backend` or
`endpoint`. A derived profile may omit those target fields because its selected
base chain can provide them. Local parsing still validates a derived profile's
own ID, supplied endpoint, execution-setting bounds, and `extra_params`.
Promptkit does not infer a backend from a model or endpoint. Endpoint-only
profiles remain supported and have no effective backend ID.
The engine always provides the built-in `openrouter` and `rakestrawhome` IDs.
@@ -266,6 +281,32 @@ profiles. They use `APIKeyRequired` for request-scoped credentials instead of
`api_key_env`. Preparation and exact profile inspection use this same source
precedence.
When a selected definition names `base_profile`, every profile ID in that
chain is looked up through this same precedence order. A higher-precedence
definition therefore shadows a lower-precedence definition of the same base
ID, including a built-in. References are not source-qualified.
### Profile Inheritance
Promptkit resolves one linear base chain of at most 32 profiles, including the
selected profile. It merges settings from the root base to the selected leaf.
The leaf's `id` remains the selected profile identity. Nonblank string fields
(`backend`, `endpoint`, `model`, `service_tier`, `reasoning_effort`, and
`api_key_env`) and nonzero numeric fields replace inherited values. A nonempty
`extra_params` map replaces the complete inherited map rather than merging
keys, and `APIKeyRequired: true` remains true through the chain. Backend and
endpoint are independent: replacing one does not clear the other.
There is no profile-level clearing syntax. Blank strings, zero numbers, false,
and empty maps remain unspecified and inherit from a base. Use existing
presence-aware request overrides where an execution needs an explicit zero or
empty reasoning setting.
An absent directly selected profile reports the ordinary not-found error. Once
the selected profile exists, a missing base, cycle, overlong chain, or
incomplete resolved target is a profile-load failure. Ordinary operations
resolve chains afresh; prepared execution retains the fully resolved target.
## Built-In Profile Catalog
Every built-in profile selects one maintained built-in backend and inherits
@@ -318,16 +359,25 @@ schema produces a failed validation result.
Credential values belong at the request or environment boundary, never in
prompt, profile, schema, or example files:
- a file profile names an environment variable with `api_key_env`;
- an in-memory profile may set `APIKeyRequired`;
- a request can provide a direct `APIKey` or override `APIKeyEnv`; and
- a backend or file profile can name an optional environment lookup source
with `APIKeyEnv` or `api_key_env`;
- an in-memory profile may set `APIKeyRequired` as an explicit local
requirement;
- a request can provide a direct `APIKey` or override the optional `APIKeyEnv`
source; and
- a direct request key takes precedence over environment lookup.
After a direct request key, the credential-source precedence is request
`APIKeyEnv`, profile `api_key_env`, then the backend default. An in-memory
profile with `APIKeyRequired` clears an inherited backend environment name and
requires a direct key unless the request explicitly supplies `APIKeyEnv`.
Promptkit validates required credential availability during preparation.
Named environment sources are optional: when the selected source is absent,
empty, or whitespace-only, the built-in client omits the `Authorization`
header and handles the provider response normally. `APIKeyRequired` is the
only explicit local availability requirement. Promptkit validates required
credential availability during preparation and rechecks it when a prepared
execution runs. Injected clients receive resolved source metadata but define
their own credential-resolution behavior.
Direct keys are excluded from JSON results and redacted by public string
formatters. Environment-variable names may appear in prepared metadata, but
their values do not.

View File

@@ -31,11 +31,13 @@ does not serialize it in the provider request.
## Authentication
A non-empty API key supplied directly on the execution target takes
precedence. Otherwise, when an API-key environment-variable name is supplied,
the client reads that variable and requires a non-empty value. The selected
key is sent as `Authorization: Bearer <key>`. No authorization header is sent
when neither mechanism is configured.
A usable API key supplied directly on the execution target takes precedence.
Otherwise, when an API-key environment-variable name is supplied, the client
reads and trims that variable. A bearer header is sent only when the resolved
direct or environment credential is non-empty. When neither source is usable,
the client omits `Authorization` and handles the provider response normally.
An explicitly required target with no usable source is rejected before
transport.
The target contains the already resolved environment-variable name: an
explicit request override takes precedence over profile metadata, which takes

View File

@@ -55,12 +55,14 @@ the target, rendered messages, and structured-output constraint retained by
executable preparation. Execution does not reopen or rerender consumer
sources.
Before backend admission, the runner rechecks that the frozen credential
environment-variable name is available. The handle does not retain the
environment value; the model client resolves the value visible when generation
begins. A direct request key remains in private execution state only until the
claimed execution finishes or an unclaimed handle is discarded. Exact public
ownership and redaction semantics belong to the
Before backend admission, the runner rechecks a frozen credential
environment-variable name only when the target explicitly requires a
credential. The handle does not retain the environment value; the model client
resolves the value visible when generation begins. For optional sources with no
usable value, the built-in client omits `Authorization` and continues to the
provider. A direct request key remains in private execution state only until
the claimed execution finishes or an unclaimed handle is discarded. Exact
public ownership and redaction semantics belong to the
[`PreparedExecution` GoDoc](../../prepared_execution.go).
## Failure Categories
@@ -68,12 +70,18 @@ ownership and redaction semantics belong to the
The package preserves distinct error identities for invalid client
configuration, invalid generation requests, request execution failures,
non-success provider statuses, and malformed successful responses. Provider
response bodies are not included in non-success errors.
response bodies are never exposed in raw form through non-success errors.
Invalid nonempty configured endpoints are configuration failures. A missing or
invalid final selected endpoint is an invalid generation request and is
rejected before transport.
Authentication resolves a trimmed direct key before a trimmed configured
environment value. Optional missing, empty, or whitespace-only sources do not
block transport and produce no `Authorization` header. An explicitly required
target with no usable source is rejected before transport with the existing
invalid-request diagnostics.
Successful response bodies have a fixed 16 MiB limit enforced by declared
length and by reading at most one byte beyond the boundary. The decoder accepts
exactly one JSON object plus trailing whitespace and EOF. Size overflow,

View File

@@ -21,7 +21,7 @@ contributor workflow and validation.
| `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |
| `internal/jsonvalue` | Validates and deeply copies bounded JSON-compatible extra-parameter and prepared-schema trees while preserving supported concrete value types and rejecting cycles or excessive depth and work. | [JSON values](../../internal/jsonvalue/jsonvalue.go) |
| `internal/promptdef` | Loads strictly decoded, validated prompt definitions from filesystem and `fs.FS` sources, including version selection and contained file-backed message content. | [Framework formats](../formats.md), [prompt-definition repository](../../internal/promptdef/filesystem_repository.go) |
| `internal/profile` | Loads strictly decoded, validated execution profiles, including backend selection, from filesystem and `fs.FS` sources and composes repositories with error-preserving fallback. | [Framework formats](../formats.md), [profile repositories](../../internal/profile/filesystem_repository.go) |
| `internal/profile` | Loads strictly decoded, locally validated execution profiles from filesystem and `fs.FS` sources, overlays raw sources with error-preserving fallback, and resolves inherited profiles. | [Framework formats](../formats.md), [profile repositories](../../internal/profile/filesystem_repository.go), [internal sources](sources.md#profiles-and-built-ins) |
| `internal/profile/builtin` | Embeds the built-in profile catalog, whose entries select maintained built-in backends. | [Built-in catalog](../formats.md#built-in-profile-catalog), [repository](../../internal/profile/builtin/repository.go) |
| `internal/prompt` | Renders prompt messages from Go templates with artifact, variable, session, and cache-control data. | [Go-template renderer](../../internal/prompt/go_renderer.go) |
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |

View File

@@ -39,17 +39,17 @@ duplicate detection, and source containment:
## Profiles And Built-Ins
`internal/profile` loads and validates execution profiles from an
operating-system filesystem or an `fs.FS`. A file contains exactly one YAML
document and its trimmed YAML `id` is its only selection identity; filenames do
not confer authority. Each point lookup reads discovered files once for their
metadata and reuses the selected file's bytes for strict decoding; unrelated
profiles are not fully decoded. Strict selected decoding recognizes the
optional `backend` field, trims its value, and requires a model plus at least
one non-blank backend or endpoint. File-backed `extra_params` values are
validated and defensively copied through the shared bounded JSON-value owner
before a profile is published. OpenAI-compatible reserved-field policy remains
with the model-client and backend-registry owners.
`internal/profile` loads, locally validates, overlays, and resolves execution
profiles from an operating-system filesystem or an `fs.FS`. A file contains
exactly one YAML document and its trimmed YAML `id` is its only selection
identity; filenames do not confer authority. Each point lookup reads discovered
files once for their metadata and reuses the selected file's bytes for strict
decoding; unrelated profiles are not fully decoded. Strict selected decoding
recognizes `base_profile` and the optional `backend` field, trims their values,
and permits inherited target fields only when a base is named. File-backed
`extra_params` values are validated and defensively copied through the shared
bounded JSON-value owner before a profile is published. OpenAI-compatible
reserved-field policy remains with the model-client and backend-registry owners.
The overlay repository consults the next repository only when the
higher-precedence repository reports that a profile is absent. A reliably
@@ -59,14 +59,25 @@ backend registry membership because the available registry belongs to the
assembled engine; the runner checks membership during preparation and exact
profile inspection.
The root engine assembles profile repositories in precedence order: in-memory
profiles, one ordinary configured source, an application fallback source, then
the embedded built-in catalog. An explicit file or `fs.FS` profile source
replaces `Config.ProfileDir` within the ordinary configured-source category.
The root engine assembles one raw composite catalog in precedence order:
in-memory profiles, one ordinary configured source, an application fallback
source, then the embedded built-in catalog. An explicit file or `fs.FS` profile
source replaces `Config.ProfileDir` within the ordinary configured-source
category. One outer resolving repository wraps that complete raw catalog, so
each base lookup observes the same precedence and shadowing rules.
Exact profile inspection performs one point-in-time lookup through those
profile sources and checks the resolved target without reading prompt, input,
The resolving repository traverses every selected chain afresh, retains no
cache, detects cycles, limits a chain to 32 profiles, merges root-to-leaf into a
new caller-owned value, and validates the final target before publishing it. It
does not check backend registry membership. Exact `base_profile` syntax, merge
rules, and consumer-visible failure behavior belong to the [framework format
reference](../formats.md#profile-inheritance).
Exact profile inspection performs one point-in-time resolved lookup through
those profile sources and checks the final target without reading prompt, input,
or schema sources. It does not retain that lookup for a later execution.
Prepared execution instead freezes the fully resolved target; a later ordinary
operation performs a fresh traversal.
`internal/profile/builtin` embeds the maintained built-in profile catalog.
Every embedded profile selects a maintained built-in backend and inherits that

155
docs/releases/v0.7.0.md Normal file
View File

@@ -0,0 +1,155 @@
# Promptkit v0.7.0
This supplemental changelog and migration guide summarizes the consumer-facing
changes from `v0.6.0` to `v0.7.0`. The annotated `v0.7.0` tag is the
authoritative release record. Exact current contracts belong to the linked
GoDoc and durable documentation.
## Summary
`v0.7.0` expands provider integration and profile composition while making
credential and generation-failure handling more flexible:
- Promptkit now includes the `rakestrawhome` backend and its Gemma profile;
- built-in generation failures expose bounded structured provider details;
- an unavailable optional API-key environment source no longer prevents a
request from reaching an upstream that permits unauthenticated access; and
- profiles can inherit from and selectively refine another profile.
## Compatibility
This release adds public declarations and fields but removes none. Existing
keyed configuration literals and ordinary `errors.Is` handling continue to
work.
Adding `BaseProfileID` to `Profile` and `OpenAICompatibleProfileConfig` changes
their struct shape. Consumers using positional composite literals for either
type must convert them to keyed literals. Existing keyed literals require no
change.
The `rakestrawhome` backend ID is now built in and reserved. A consumer that
previously registered that exact ID with `WithBackend` must remove its manual
registration before upgrading. Other custom backend registrations are
unchanged.
When an optional backend, profile, or request `APIKeyEnv` is unset, empty, or
whitespace-only, the built-in client now omits `Authorization` and sends the
request. Previously this condition could fail before transport. Set
`Profile.APIKeyRequired` when missing credentials must remain a local
preflight error.
Provider non-success responses continue to match `ErrLLMGenerate`. Their
rendered wording is not a compatibility contract; consumers can now use
`errors.As` with `*GenerationError` when structured status information is
needed.
## Upgrade
Update the module dependency with:
```sh
go get gitea.maximumdirect.net/eric/promptkit@v0.7.0
go mod tidy
```
Remove any manual `rakestrawhome` backend registration, convert positional
profile literals to keyed literals, and run the consuming project's ordinary
and race-enabled tests.
## Rakestrawhome Built-In Backend And Profile
Every engine now includes the reserved `rakestrawhome` backend, identified by
`BackendRakestrawHome`. The built-in `rakestrawhome-gemma-4-31b` profile
selects that backend. Consumers can use the maintained endpoint, credential,
capacity, and model defaults without registering either definition themselves.
See the [built-in backend and profile catalogs](../formats.md#built-in-backends)
and the [consumer adoption example](../consumers/pkg-promptkit.md#use-the-rakestrawhome-built-in-profile)
for the current contracts.
## Structured Generation Errors
Non-2xx responses from the built-in OpenAI-compatible client now return an
immutable `*GenerationError`. Consumers can inspect the HTTP status and any
safely extracted provider code, type, or message while retaining the ordinary
generation-error category:
```go
var generationErr *promptkit.GenerationError
if errors.As(err, &generationErr) {
status := generationErr.StatusCode()
_ = status
}
```
Provider fields are bounded and normalized but remain untrusted and may
contain sensitive request or schema details. Default and Go-syntax formatting
omit those fields. Applications must apply their own disclosure policy before
logging or presenting accessor values.
See the [`GenerationError` GoDoc](../../generation_error.go), the
[consumer error-handling guide](../consumers/pkg-promptkit.md#handle-errors),
and the [OpenAI-compatible response contract](../integrations/openai-compatible-chat.md#response-handling).
## Optional Credential Sources
`APIKeyEnv` names an optional environment lookup source unless the selected
profile explicitly sets `APIKeyRequired`. When neither a direct request key nor
a usable environment value exists, the built-in client omits the bearer header
and handles the upstream response normally. This supports local and other
OpenAI-compatible providers that permit unauthenticated requests without
hiding an authentication error returned by a provider that requires one.
The [credential format reference](../formats.md#credentials), the
[`Backend` GoDoc](../../backends.go), the
[`ExecutionTargetOverride` GoDoc](../../types.go), and the
[authentication integration contract](../integrations/openai-compatible-chat.md#authentication)
define the current precedence and availability rules.
## Profile Inheritance
YAML profiles can name one parent with `base_profile`; in-memory profiles use
`Profile.BaseProfileID`, and `OpenAICompatibleProfileConfig` forwards the same
field. A profile can act as an application-owned alias of a built-in or refine
selected inherited settings:
```go
promptkit.WithProfiles(promptkit.Profile{
ID: "weather-light",
BaseProfileID: "deepseek-4-flash",
ReasoningEffort: "high",
})
```
Base lookup observes the existing source precedence. Chains are linear,
cycle-safe, and resolved afresh for ordinary operations. Prepared execution
freezes the fully resolved target. The selected leaf ID remains public while
effective execution settings reflect the resolved chain.
See the [profile inheritance format reference](../formats.md#profile-inheritance),
the [consumer alias example](../consumers/pkg-promptkit.md#alias-a-built-in-profile),
and the [`Profile` GoDoc](../../types.go) for exact merge and validation
behavior.
## Public API Changes
The release adds:
- `BackendRakestrawHome`;
- `GenerationError`, including `StatusCode`, `ProviderCode`, `ProviderType`,
`ProviderMessage`, `Error`, `GoString`, and `Unwrap`;
- `Profile.BaseProfileID`; and
- `OpenAICompatibleProfileConfig.BaseProfileID`.
No public declaration was removed.
## Consumer Action
- Remove a manual backend registration whose ID is exactly `rakestrawhome`.
- Convert positional `Profile` or `OpenAICompatibleProfileConfig` literals to
keyed literals.
- Set `Profile.APIKeyRequired` where a missing credential must fail locally
instead of reaching the provider unauthenticated.
- Treat `GenerationError` provider fields as untrusted and potentially
sensitive when adopting the new accessors.
- Run consumer ordinary and race-enabled tests after updating the module.

View File

@@ -40,12 +40,11 @@ consumers.
### Public bounded output repair
After the codebase-audit remediations are complete, Promptkit should make its
bounded output-repair capability available through the public engine. A
consumer should be able to request a limited number of corrective generation
attempts when JSON or JSON Schema output fails content validation, without
having to reproduce Promptkit's generation, validation, capacity, and result-
accounting orchestration.
Promptkit should make its bounded output-repair capability available through
the public engine. A consumer should be able to request a limited number of
corrective generation attempts when JSON or JSON Schema output fails content
validation, without having to reproduce Promptkit's generation, validation,
capacity, and result-accounting orchestration.
- Repair is validation recovery, not a general provider retry, failover, or
backoff policy. Transport failures, cancellation, and operational schema or
@@ -62,10 +61,6 @@ accounting orchestration.
- Ordinary and prepared execution should expose coherent behavior, including
cancellation, frozen prepared state, error identity, and capacity lifetime.
Select this work only after the accepted audit findings affecting shared
execution invariants, validation, orchestration, transport, and repair
internals have been remediated.
## Entry Format
Use a short heading followed by a concise summary. Add focused bullets when

View File

@@ -1,437 +0,0 @@
# Structured Generation Errors Implementation Plan
## Purpose
Implement the target state defined in the
[structured generation errors roadmap](structured-generation-errors.md): turn
every non-2xx response from the built-in OpenAI-compatible client into a
bounded, immutable public error that exposes deliberate provider diagnostics
through `errors.As` while retaining `ErrLLMGenerate` through `errors.Is`.
This document owns implementation sequencing. The feature roadmap owns the
consumer intent, public-policy decisions, safety limits, compatibility
boundaries, and non-goals. Follow the architecture, documentation, and testing
policies under [`docs/policy/`](../policy/) throughout the work.
## Fixed Decisions
- The public type is `GenerationError`. Engine-produced values are pointers;
fields are unexported; no public constructor or mutation API is added.
- Public methods are `StatusCode() int`, `ProviderCode() string`,
`ProviderType() string`, `ProviderMessage() string`, `Error() string`,
`GoString() string`, and `Unwrap() error`.
- Public `Unwrap` returns `ErrLLMGenerate`. Accessors, formatting, and unwrapping
are safe on a nil receiver and a zero value.
- Engine-produced `Error()` text is exactly
`failed to generate output: provider returned HTTP status N`, where `N` is
the received status. A nil receiver or zero status returns exactly
`failed to generate output`. `GoString()` returns the same redacted text as
`Error()` so `%#v` cannot reveal unexported provider fields.
- Default formatting contains no provider-controlled code, type, message, or
raw response content. The type has no stable JSON representation.
- The recognized body is one JSON document with a top-level object-valued
`error`. `message` and `type` accept strings; `code` accepts a string or an
exact `json.Number`; supported fields are independent and unknown fields are
ignored.
- The non-success-body limit is 65,536 bytes. Declared oversize bodies are not
read; other bodies are read through a 65,537-byte bound. Oversize, malformed,
unrecognized, or unreadable bodies produce status-only detail.
- Normalization converts invalid UTF-8, collapses Unicode whitespace, control,
and format-character runs to one ASCII space, trims the result, omits blank
values, and produces one-line strings.
- Codes and types longer than 256 Unicode code points are omitted. Messages
longer than 4,096 code points retain the first 4,095 code points followed by
`…`, for a total limit of 4,096.
- Only the concrete built-in transport error is converted into a new public
`GenerationError`. Arbitrary injected-client errors are never inspected or
enriched.
- The use-case and domain layers remain provider-neutral. No retryability,
retry, logging, presentation, provider-specific envelope, header, or success-
response behavior is added.
## Execution Rules
- Complete the stages in numerical order. Each stage is scoped for one
gpt-5.6-terra implementation prompt and must finish its focused tests before
the next begins.
- Treat all stages as one feature delivery. Intermediate stages intentionally
create internal machinery before exposing it; do not release, tag, or claim
the feature is available until Stage 5 is complete.
- At the start of each stage, reread the feature roadmap and the task-specific
references in [`docs/development.md`](../development.md). Preserve unrelated
working-tree changes.
- Use classical behavior tests at the narrowest owner. Table-drive parser and
boundary cases, use controlled transports or local servers, and do not
duplicate the internal envelope matrix at the root engine boundary.
- Do not contact a live provider, add dependencies, commit, tag, push, or edit
release documentation unless separately instructed.
- Current-state prose documentation changes in Stage 5. Public GoDoc changes
alongside the public declarations in Stage 4 because GoDoc owns that API.
## Stage 1: Add the Internal Structured Status Error and Envelope Parser
### Objective
Create the transport-owned structured value and pure parsing and normalization
logic without changing `OpenAICompatibleClient.Generate` yet.
### Implementation
1. Add `internal/llm/provider_http_error.go`. Keep all provider HTTP mechanics
in `internal/llm`; do not add an HTTP error DTO to `internal/domain` or
`internal/usecase`.
2. Define these private constants:
- `maxProviderErrorResponseBytes int64 = 64 << 10`;
- `maxProviderErrorIdentifierRunes = 256`; and
- `maxProviderErrorMessageRunes = 4096`.
3. Add an exported-within-`internal` `ProviderHTTPError` type with unexported
`statusCode`, `providerCode`, `providerType`, and `providerMessage` fields.
The root facade will need to name this concrete type in Stage 4, but no
representation is public outside the module's `internal` boundary.
4. Give `ProviderHTTPError` nil-safe read-only accessors with the same four
names as the planned public type. Implement:
- `Error()` as `llm returned non-success status: status=N` when status is
nonzero and `llm returned non-success status` otherwise;
- `GoString()` by returning `Error()`; and
- `Unwrap()` by returning `ErrUnexpectedStatus`.
Never include provider-derived strings in either formatter.
5. Add a private `providerErrorDetails` value and a private constructor that
builds `*ProviderHTTPError` from a status plus already normalized details.
6. Add `parseProviderErrorEnvelope([]byte) providerErrorDetails` with these
rules:
- use `json.Decoder` with `UseNumber` and require EOF after trailing JSON
whitespace;
- require a top-level object and object-valued `error` member;
- retain supported fields as `json.RawMessage` so each can be decoded and
validated independently;
- accept string `message` and `type` values;
- accept string or `json.Number` `code`, preserving validated number text
without float conversion;
- ignore unknown fields and treat invalid supported-field values as absent;
and
- return empty details for malformed framing, a missing or invalid `error`
object, or an object with no usable fields.
7. Add private normalization helpers that implement the roadmap's UTF-8,
single-line, whitespace/control/format handling and exact rune limits. Use
rune-aware operations; do not truncate bytes in the middle of UTF-8. Omit
overlong code and type identifiers, and truncate overlong messages to 4,095
code points plus `…`.
### Tests
Add `internal/llm/provider_http_error_test.go` in package `llm` with focused,
table-driven tests:
1. `TestProviderHTTPErrorEnvelopeParsing` covers:
- all supported string fields;
- string, integer, fractional, and exponent-form numeric codes without
float coercion;
- `null` and invalid field types handled independently;
- unknown top-level and nested fields;
- missing, null, scalar, and empty `error` values;
- malformed, truncated, trailing-garbage, and second-document input; and
- no raw or unsupported metadata retained.
2. `TestProviderErrorTextNormalizationAndLimits` covers valid multibyte text,
invalid UTF-8 replacement, leading/trailing and repeated whitespace,
newline/tab/control/format characters, blank normalization, exact identifier
and message boundaries, identifier omission one rune over, and rune-safe
message ellipsis one rune over.
3. `TestProviderHTTPErrorIdentityAndFormatting` covers exact accessors,
`errors.Is(err, ErrUnexpectedStatus)`, nil receivers, zero values, and safe
`%v`, `%+v`, and `%#v` formatting with distinctive provider markers absent.
Do not test body reading, HTTP response ownership, the root public type, or
assembled engine behavior in this stage.
### Verification
```sh
go test ./internal/llm -run 'TestProvider'
go test ./internal/llm
go test ./...
```
Stage 1 is complete when the parser and internal error are fully protected but
the live non-2xx branch remains unchanged.
**Status:** Complete.
## Stage 2: Add the Bounded Non-Success Body Reader
### Objective
Implement and test bounded response-body extraction independently from HTTP
client integration, keeping status preservation separate from envelope
validity.
### Implementation
1. In `internal/llm/provider_http_error.go`, add a private helper with the
equivalent contract of:
```go
func providerHTTPErrorFromBody(
statusCode int,
contentLength int64,
body io.Reader,
) *ProviderHTTPError
```
2. Always return a nonnil `ProviderHTTPError` carrying `statusCode`.
3. When `contentLength` is greater than 65,536, return status-only detail
without reading `body`.
4. Otherwise read through an `io.LimitedReader` capped at 65,537 bytes. Return
status-only detail on a read error or when the extra byte is consumed. Do
not parse a bounded prefix of an incomplete oversized body.
5. For a complete body at or below the limit, call
`parseProviderErrorEnvelope` and construct the error from its normalized
details.
6. The helper does not close or drain `body`; the HTTP caller retains response-
body ownership. It must never read beyond the one-byte overflow probe.
### Tests
Extend `internal/llm/provider_http_error_test.go` with
`TestProviderHTTPErrorBodyBounds`, using counting, failing, and guarded readers
instead of an HTTP server. Cover:
- an ordinary recognized envelope;
- an exact 65,536-byte body, using trailing JSON whitespace to reach the
boundary while remaining one valid document;
- a declared 65,537-byte body with zero reads;
- unknown-length and underreported 65,537-byte bodies with exactly 65,537 bytes
read and status-only detail;
- an early read failure with status-only detail; and
- an empty body with status-only detail.
Assert the provider markers are absent whenever extraction is discarded. Do
not add body-closure assertions here because this helper does not own closing.
### Verification
```sh
go test ./internal/llm -run 'TestProviderHTTPErrorBodyBounds|TestProvider'
go test ./internal/llm
go test ./...
```
Stage 2 is complete when every read path is deterministically bounded and the
HTTP client's current branch is still untouched.
**Status:** Complete.
## Stage 3: Integrate Structured Status Errors into the Built-In Client
### Objective
Replace the built-in client's status-only discard branch with the bounded
internal error while preserving all successful, cancellation, transport, and
body-ownership behavior.
### Implementation
1. In `internal/llm/openai_compatible_client.go`, replace the non-2xx branch's
4,096-byte discard and formatted sentinel with
`providerHTTPErrorFromBody(httpResp.StatusCode, httpResp.ContentLength,
httpResp.Body)`.
2. Keep the existing `defer httpResp.Body.Close()` as the single body-closure
owner. Do not close in the helper, drain after the bound, or reuse the 16 MiB
successful-response decoder or limit.
3. Return no partial `GenerateResponse` for every non-2xx response.
4. Preserve `errors.Is(err, ErrUnexpectedStatus)` through
`ProviderHTTPError.Unwrap`. Do not change `requestFailedError`, endpoint or
request validation, authentication, timeout handling, successful response
decoding, or response-size behavior.
### Tests
1. Update the existing non-success case in
`internal/llm/openai_compatible_client_test.go` to assert
`errors.As(err, &providerHTTPError)`, exact status, and continued
`ErrUnexpectedStatus` identity. Keep the existing raw-body redaction check.
2. Add `internal/llm/provider_http_error_transport_test.go` with:
- `TestOpenAICompatibleClientStructuredNonSuccessResponse`, proving a
recognized envelope supplies all normalized internal fields and returns
no generation result;
- `TestOpenAICompatibleClientNonSuccessBodyOwnership`, table-driving normal,
declared-oversize, streamed-oversize, underreported, malformed, and read-
failure cases through a controlled transport; and
- assertions that every body is closed, no case reads beyond its bound,
declared oversize performs no read, and discarded details remain empty.
3. Reuse existing controlled-transport and counting-reader helpers when they
are clear and package-local. Do not duplicate successful-response framing,
timeout, authentication, or endpoint matrices.
### Verification
```sh
go test ./internal/llm -run 'TestOpenAICompatibleClient.*NonSuccess|TestProvider'
go test ./internal/llm
go test ./...
```
Stage 3 is complete when the built-in client emits the structured internal
error for every non-2xx response and all prior internal identities remain
green.
**Status:** Complete.
## Stage 4: Add the Public Error and Root Mapping Contract
### Objective
Translate only the built-in transport's structured error at the root facade,
publish the immutable consumer API, and prove ordinary, prepared, and injected-
client behavior.
### Implementation
1. Add `generation_error.go` in package `promptkit` with an immutable public
`GenerationError` whose four fields are unexported strings or integers. Add
one private constructor that accepts the four normalized scalar values. The
public error file must not import `internal/llm` or retain the internal error
or raw body; `errors.go` performs that adaptation at the facade boundary.
2. Implement the four public accessors exactly as fixed above. Each returns
zero or empty on a nil receiver.
3. Implement `Error()` with the fixed strings from this plan, `GoString()` by
returning `Error()`, and `Unwrap()` by returning `ErrLLMGenerate` even for a
nil receiver. Do not implement mutable fields, an exported constructor,
retry helpers, HTTP mapping, `fmt.Formatter`, or JSON methods.
4. Write complete GoDoc covering:
- built-in-client and non-2xx scope;
- ordinary and prepared execution;
- `errors.Is` and pointer-target `errors.As` usage;
- immutable and caller-owned semantics;
- nil and zero behavior;
- lack of stable JSON;
- safe default formatting; and
- the fact that every provider accessor is untrusted and may contain
sensitive request or schema fragments.
5. In `errors.go`, after the special capacity conversion and before generic
sentinel wrapping, use `errors.As` for a nonnil concrete
`*llm.ProviderHTTPError`. Convert it to `*GenerationError` and return that
public value directly. Do not parse error text or recognize an interface
that an injected client could accidentally satisfy.
6. Preserve the existing generic `publicErrorFor` path for all other failures.
In particular, arbitrary injected-client errors remain wrapped with
`ErrLLMGenerate` and retain their original identity.
7. Update public GoDoc in the same stage:
- the `ErrLLMGenerate` declaration points to `GenerationError` for built-in
non-2xx responses;
- `Engine.Run` and `Engine.RunPrepared` mention the typed error without
restating its accessors;
- `doc.go` distinguishes mutable `CapacityError` from immutable
`GenerationError`, lists both as lacking stable JSON, and calls out the
provider-detail trust boundary; and
- injected `LLMClient` GoDoc remains clear that arbitrary client errors are
preserved rather than translated.
### Tests
1. Add package-internal tests for `GenerationError` nil and zero receivers,
exact fixed formatting, and unwrapping. Do not expose a test-only public
constructor or turn the lack of stable JSON into a serialized-output
contract.
2. Add a focused external-package contract test, preferably in
`generation_error_contract_test.go`, that obtains errors through a real
assembled engine with a controlled HTTP transport:
- an ordinary `Run` with a recognized envelope asserts a nil result,
`errors.Is(err, ErrLLMGenerate)`, pointer-target `errors.As`, all four
accessors, exact status-bearing formatting, and absence of distinctive
code/type/message markers from `%v`, `%+v`, and `%#v`;
- one `RunPrepared` case proves the same public type and status cross the
prepared boundary without repeating every parser field; and
- neither case contacts a live provider or uses a real credential.
3. Extend the existing injected-client preservation owner with one assertion
that an arbitrary injected error does not become a `*GenerationError`, while
still matching both `ErrLLMGenerate` and the injected error.
4. Keep detailed envelope, normalization, size, and body-ownership matrices in
`internal/llm`; root tests remain representative.
### Verification
```sh
go test . -run 'TestGenerationError|TestBuiltInGenerationError|TestRunAddsLLMGenerate'
go test ./internal/llm
go test ./...
```
Stage 4 is complete when consumers can inspect built-in non-2xx details through
the stable root contract and injected errors remain untouched.
**Status:** Complete.
## Stage 5: Update Canonical Documentation and Run Full Validation
### Objective
Make durable documentation match the implemented contract, remove the roadmap
links that describe parsing as future work, and complete repository-wide
validation.
### Documentation
1. Update `docs/integrations/openai-compatible-chat.md` as the canonical wire
owner. Replace the status-only paragraph with:
- the recognized top-level envelope and independent field types;
- strict single-document framing and unknown-field behavior;
- the exact 65,536-byte read policy and status-only fallbacks;
- exact normalization and field limits;
- response closure and no-overread behavior; and
- the prohibition on raw bodies, headers, endpoints, credentials, request
data, schemas, and generated content.
2. Update `docs/internal/llm.md` with the internal `ProviderHTTPError`, bounded
reader and parser flow, retained `ErrUnexpectedStatus` identity, root
conversion boundary, and narrow test ownership. Remove its link that defers
parsing to the feature roadmap.
3. Update `docs/consumers/pkg-promptkit.md` under `Handle Errors` with one short
`errors.As` example. Show status and deliberate message access, warn that all
provider fields are untrusted and potentially sensitive, leave retry and
presentation policy to the application, and link to `GenerationError`
GoDoc rather than duplicating its full contract.
4. Update `docs/internal/overview.md` only enough to inventory implemented
responsibilities: the root facade owns typed capacity and generation error
mapping, and `internal/llm` owns bounded structured non-success response
decoding. Do not duplicate limits or accessor details there.
5. Do not change `docs/policy/architecture.md`, `docs/formats.md`, backend
documentation, release notes, or the README unless implementation reveals a
concrete inaccurate statement. Their canonical topics do not own this
contract.
### Final Validation
Run the complete maintainer workflow from
[`docs/development.md#maintainer-validation`](../development.md#maintainer-validation):
```sh
go test ./...
go test -race ./...
go vet ./...
go build ./...
go run ./examples/go-library/prepare
go run ./examples/go-library/run
```
Also perform the documented Go-formatting, local Markdown-link, repository-
hygiene, ignored-file, and credential scans. Review both example outputs and
confirm that all commands remain deterministic, offline, and credential-free.
Finally review the complete diff against the feature roadmap and confirm:
- every built-in non-2xx response produces a status-bearing public type;
- malformed and oversized bodies cannot erase status or leak partial content;
- provider-derived strings appear only through deliberate accessors;
- default and Go-syntax formatting are redacted;
- successful, cancellation, capacity, validation, repair, and injected-client
behavior is unchanged;
- no provider policy entered the use-case or domain layers; and
- each exact contract has one canonical documentation and test owner.
**Status:** Complete.
## Open Questions
None. The roadmap and this plan fix the public API, internal representation,
wire envelope, normalization and bounds, status fallback, formatting,
propagation, compatibility, documentation, and verification decisions required
for implementation.

View File

@@ -1,260 +0,0 @@
# Structured Generation Errors
## Purpose
Promptkit should give downstream applications actionable, machine-readable
details when the built-in OpenAI-compatible client receives a non-success HTTP
response. Today the client reports only the status code and discards the
provider response body. This makes ordinary configuration failures, such as an
unsupported strict JSON Schema keyword, unnecessarily difficult to diagnose.
This feature supplies bounded facts about the provider response. It does not
make retry, presentation, or logging decisions for consumers.
## Target End State
Every non-2xx response received by Promptkit's built-in OpenAI-compatible
client becomes a public typed generation error. A consumer can use
`errors.As` to obtain the HTTP status and any safely extracted provider fields,
and `errors.Is` continues to match `ErrLLMGenerate`.
The typed contract is available from both `Run` and `RunPrepared`. It is not
produced during preparation, which performs no model request. Successful
responses, transport failures before a response is received, cancellation,
capacity failures, validation failures, and nil responses from injected model
clients retain their existing categories and behavior.
An unusable response body never hides the known HTTP status. Empty, malformed,
unrecognized, unreadable, or oversized bodies therefore produce the same typed
error with status-only detail rather than falling back to an unstructured
error or becoming a malformed-success response.
## Public Contract
The root package exposes an immutable `GenerationError` type with unexported
state and these read-only accessors:
- `StatusCode() int` returns the received HTTP status code;
- `ProviderCode() string` returns a normalized provider code, when present;
- `ProviderType() string` returns a normalized provider error type, when
present; and
- `ProviderMessage() string` returns the bounded normalized diagnostic message,
when present.
The engine returns a `*GenerationError`, so the idiomatic inspection form is:
```go
var generationErr *promptkit.GenerationError
if errors.As(err, &generationErr) {
status := generationErr.StatusCode()
message := generationErr.ProviderMessage()
_, _ = status, message
}
```
There is no public constructor or mutation API. The type implements `error`,
unwraps to `ErrLLMGenerate`, and provides safe ordinary and Go-syntax
formatting. `Error()` and `GoString()` include the HTTP status but no provider-
controlled code, type, or message. Consumers must use the accessors
deliberately when they want provider details and must not classify failures by
matching error text.
The zero value and a nil `*GenerationError` receiver are safe: accessors return
zero or empty values, formatting returns a generic redacted generation-failure
description, and unwrapping still identifies `ErrLLMGenerate`. Engine-produced
values always have the non-2xx status received from the provider. The type has
no stable JSON representation.
All provider-derived strings remain untrusted even after normalization. GoDoc
must warn consumers that provider fields can contain sensitive request or
schema fragments and must not be logged, displayed, or returned to another
caller without an application-appropriate disclosure policy.
## Recognized Provider Envelope
Promptkit recognizes only the conventional OpenAI-compatible top-level error
object:
```json
{
"error": {
"message": "diagnostic text",
"type": "invalid_request_error",
"code": "unsupported_parameter"
}
}
```
The envelope must be one JSON document followed only by JSON whitespace. The
top-level `error` value must be an object. Unknown top-level and error-object
fields are ignored. The optional supported fields are interpreted
independently:
- `message` and `type` must be JSON strings;
- `code` may be a JSON string or number and is exposed as normalized text;
numeric codes retain their validated JSON number text without floating-point
coercion; and
- `null`, booleans, arrays, objects, or otherwise invalid values are treated as
absent for that field.
An invalid optional field does not discard other valid supported fields. An
absent `error` object, malformed or multiply framed JSON, or an object with no
usable supported fields simply leaves all provider accessors empty while
preserving the typed status error.
Promptkit does not expose `param`, metadata objects, nested causes, headers, or
provider-specific extensions in this feature.
## Bounded Reading And Normalization
Non-success bodies have a separate fixed limit of 64 KiB (65,536 bytes). This
is intentionally much smaller than the successful completion-body limit while
remaining large enough for useful schema diagnostics.
- A declared `Content-Length` above the limit is rejected without reading the
body for detail extraction.
- Otherwise Promptkit reads at most one byte beyond the limit so streamed,
chunked, and underreported bodies are bounded.
- A body over the limit contributes no provider fields; Promptkit does not
parse or retain a prefix as though it were a complete envelope.
- Read failures likewise discard provider fields while preserving the status.
- The response body is closed on every outcome and is not drained beyond the
bounded read.
Extracted strings are converted to valid UTF-8, trimmed, and made single-line:
invalid UTF-8 is replaced, and runs of Unicode whitespace, control characters,
and formatting controls are replaced with one ASCII space. Empty normalized
values are treated as absent.
Normalized provider codes and types are retained only when they contain at
most 256 Unicode code points. Longer values are omitted rather than truncated
so consumers never classify on a fabricated partial identifier. A provider
message is limited to 4,096 Unicode code points; a longer normalized message is
truncated at a code-point boundary with a visible ellipsis inside that limit.
The raw response body and pre-normalized strings are never exposed or retained
in the public error.
## Error Propagation And Compatibility
- Every built-in-client non-2xx response matches `ErrLLMGenerate` and supports
`errors.As` to `*GenerationError`, including status-only cases.
- The internal model client retains its non-success-status identity for its
own package tests. The use-case layer remains provider-neutral and continues
to add only its generation category.
- The root error boundary converts only the built-in transport's structured
status error. It does not parse arbitrary error text, inspect consumer error
fields, or fabricate HTTP details for an injected `LLMClient`.
- Errors returned by injected clients remain in the chain exactly as today.
If an injected client deliberately returns an existing `*GenerationError`,
its identity may pass through ordinary wrapping, but Promptkit does not
construct or enrich one on that client's behalf.
- Existing cancellation and deadline identities, capacity errors, validation
behavior, repair behavior, and successful response decoding remain
unchanged.
- This is an additive public API. Existing consumers that use
`errors.Is(err, ErrLLMGenerate)` continue to work; consumers should not rely
on the previous rendered wording of non-success errors.
## Architecture And Ownership
The provider-envelope parser and bounded body reader belong in `internal/llm`,
which owns the OpenAI-compatible transport. The internal transport error owns
only normalized status facts and continues to match the package's existing
non-success-status sentinel.
The use-case package does not gain HTTP DTOs, status policy, or a provider-
specific branch. Its existing wrapping carries the internal error to the root
facade. The root error mapper recognizes the internal structured status error
and constructs the public `GenerationError` without exposing an internal type
or raw cause through public fields. No transport error is added to
`internal/domain`.
The public type and its exact Go semantics are owned by its declaration and
GoDoc. The
[OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md)
owns recognized wire shapes, limits, and observable response behavior. The
[internal model-client document](../internal/llm.md) owns implementation flow,
internal failure categories, and test ownership. Architecture policy does not
need a new package or dependency rule for this feature.
## Documentation End State
Canonical documentation at the target state has these responsibilities:
- the `GenerationError` declaration and GoDoc define the exact public methods,
formatting, unwrapping, zero-value behavior, and trust boundary;
- `Engine.Run` and `Engine.RunPrepared` GoDoc identify the typed error without
duplicating its accessor contract;
- the consumer guide includes one short `errors.As` example and links to the
public declaration;
- the integration document replaces its status-only description with the
bounded envelope contract; and
- the internal model-client document describes parsing, conversion ownership,
and narrow test owners.
The architecture policy, framework format reference, and built-in backend
catalog do not duplicate this API or wire contract.
## Verification Expectations
Verification protects each behavior at its narrowest stable owner:
- internal model-client tests cover recognized string and numeric codes,
independent optional-field handling, unknown fields, empty and malformed
envelopes, single-document framing, read failures, declared and streamed
size boundaries, body closure, normalization, field limits, and absence of
raw provider content from rendered errors;
- root error-boundary tests cover conversion to the immutable public type,
every accessor, `errors.Is`, `errors.As`, and safe `%v`, `%+v`, and `%#v`
formatting;
- one representative ordinary run and one prepared run prove that the built-in
transport contract crosses the assembled engine boundary, without repeating
the complete parser matrix;
- existing injected-client tests continue to prove preservation of consumer
error identity without fabricated provider details; and
- all tests use controlled transports or local servers and never contact a
live or paid provider.
Security limits and their exact boundaries are contractual enough to warrant
literal boundary tests. Higher-level tests should remain representative and
must not duplicate the internal transport matrix.
## Acceptance Criteria
- A consumer can distinguish an HTTP 400 from other generation failures and
deliberately obtain a bounded provider explanation when one is available.
- The same typed error remains available through ordinary and prepared
execution and still satisfies `errors.Is(err, ErrLLMGenerate)`.
- Default and Go-syntax error formatting cannot disclose any provider-derived
string or raw response content.
- Empty, malformed, unreadable, unrecognized, and oversized bodies preserve a
typed status-only error.
- No read, retained field, or formatted representation can exceed its stated
bound, and the body is closed on every outcome.
- Existing success, cancellation, capacity, validation, repair, and injected-
client contracts remain unchanged.
- Current-state documentation changes only when the implementation exists and
follows the repository's canonical ownership policy.
## Non-Goals
This feature does not add:
- retryability classification, retry loops, backoff, failover, or routing;
- parsing of success bodies as errors or changes to successful-response limits;
- provider-specific envelope variants beyond the conventional top-level
`error` object;
- response headers such as `Retry-After`, raw bodies, request data, endpoints,
credentials, schema documents, generated content, or provider metadata;
- logging, telemetry, redaction policy for downstream applications, HTTP status
mapping for consumer servers, or user-facing presentation;
- translation or enrichment of arbitrary injected-client errors; or
- a new public package, public constructor, mutable error value, or transport
type in the domain model.
## Open Questions
None. The public type direction, accessor surface, formatting and error-chain
behavior, envelope scope, normalization, safety limits, fallback behavior,
layer ownership, compatibility boundaries, documentation ownership, and test
boundaries are fixed by this roadmap.

View File

@@ -53,9 +53,9 @@ var (
// an execution profile or resolve its backend, except for the profile
// not-found case represented by ErrProfileNotFound.
ErrProfileLoad = errors.New("failed to load execution profile")
// ErrAPIKeyEnvMissing identifies an APIKeyEnv whose environment variable is
// unset or empty when no direct RunRequest.APIKey takes precedence. Such an
// error also matches ErrInvalidRequest.
// ErrAPIKeyEnvMissing identifies an explicitly required APIKeyEnv whose
// environment variable is unset or empty after direct RunRequest.APIKey
// precedence is applied. Such an error also matches ErrInvalidRequest.
ErrAPIKeyEnvMissing = errors.New("api_key_env points to an unset environment variable")
// ErrArtifactLoad identifies a failure to resolve an input artifact. Errors
// returned by an injected ArtifactReader remain available through errors.Is.
@@ -304,10 +304,12 @@ func WithFallbackProfileFS(fsys fs.FS, root string) Option {
// WithProfiles configures in-memory profiles that take precedence over
// ordinary configured, application fallback, and built-in profiles.
//
// NewEngine validates and copies every profile. IDs must be unique within one
// call. An invalid profile, duplicate ID, or unsupported ExtraParams value
// makes construction fail with ErrInvalidConfig. Repeating WithProfiles
// replaces the complete earlier in-memory set rather than merging it.
// NewEngine locally validates and copies every profile. IDs must be unique
// within one call. An invalid local definition, duplicate ID, or unsupported
// ExtraParams value makes construction fail with ErrInvalidConfig. A derived
// profile's base reference and resolved target completeness are checked when it
// is selected or inspected. Repeating WithProfiles replaces the complete
// earlier in-memory set rather than merging it.
func WithProfiles(profiles ...Profile) Option {
return optionFunc(func(options *engineOptions) error {
repo, err := newMemoryProfileRepository(profiles)
@@ -458,7 +460,7 @@ func newProfileRepository(profileDir string, options engineOptions) profile.Repo
repository = profile.NewOverlayRepository(options.memoryProfiles, repository)
}
return repository
return profile.NewResolvingRepository(repository)
}
func fileSource(name string) (fs.FS, string, error) {

View File

@@ -977,37 +977,72 @@ func TestPrepareDirectAPIKeyBypassesMissingEnvWithoutLeakingOrHashing(t *testing
}
}
func TestMissingCredentialsFailClearlyWhenProfileRequiresAuth(t *testing.T) {
func TestOptionalMissingCredentialsReachUpstream(t *testing.T) {
const missingEnv = "PROMPTKIT_PUBLIC_AUTH_MISSING"
const providerBody = `{"error":{"message":"authentication failed","type":"authentication_error","code":"invalid_api_key"}}`
t.Setenv(missingEnv, "")
profileDir := t.TempDir()
writePublicProfileFileWithAPIKeyEnv(t, profileDir, "requires-auth", "http://localhost:8000/v1", "test-model", missingEnv)
engine, err := promptkit.NewEngine(promptkit.Config{
PromptDir: frameworkPromptDir,
ProfileDir: profileDir,
SchemaDir: frameworkSchemaDir,
})
called := false
config := promptkit.Config{
PromptDir: frameworkPromptDir,
SchemaDir: frameworkSchemaDir,
HTTPClient: &http.Client{Transport: roundTripFunc(func(req *http.Request) (*http.Response, error) {
called = true
if values := req.Header.Values("Authorization"); len(values) != 0 {
t.Fatalf("Authorization values = %q, want absent", values)
}
return &http.Response{
StatusCode: http.StatusUnauthorized,
ContentLength: int64(len(providerBody)),
Body: io.NopCloser(strings.NewReader(providerBody)),
}, nil
})},
}
engine, err := promptkit.NewEngine(config,
promptkit.WithBackend(promptkit.Backend{
ID: "optional-auth",
Endpoint: "http://provider.test/v1",
APIKeyEnv: missingEnv,
}),
promptkit.WithProfiles(promptkit.Profile{
ID: "optional-auth-profile",
BackendID: "optional-auth",
Model: "test-model",
}),
)
if err != nil {
t.Fatalf("expected engine construction to succeed, got %v", err)
}
_, err = engine.Prepare(context.Background(), promptkit.RunRequest{
result, err := engine.Run(context.Background(), promptkit.RunRequest{
PromptID: frameworkMarkdownSummaryPromptID,
ProfileID: "requires-auth",
ProfileID: "optional-auth-profile",
Inputs: map[string]promptkit.ArtifactRef{
"transcript": promptkit.Inline("Rin opens the gate."),
"glossary": promptkit.Inline("gate: A guarded passage."),
},
})
if !errors.Is(err, promptkit.ErrInvalidRequest) {
t.Fatalf("expected invalid request for missing credentials, got %v", err)
if !called {
t.Fatal("optional missing credential did not reach upstream")
}
if !errors.Is(err, promptkit.ErrAPIKeyEnvMissing) {
t.Fatalf("expected missing credential environment error, got %v", err)
if result != nil {
t.Fatalf("result = %+v, want nil", result)
}
if err == nil || !strings.Contains(err.Error(), missingEnv) {
t.Fatalf("expected missing env name in error, got %v", err)
if errors.Is(err, promptkit.ErrInvalidRequest) || errors.Is(err, promptkit.ErrAPIKeyEnvMissing) {
t.Fatalf("error = %v, want upstream generation error without credential identities", err)
}
if !errors.Is(err, promptkit.ErrLLMGenerate) {
t.Fatalf("error = %v, want ErrLLMGenerate", err)
}
var generationErr *promptkit.GenerationError
if !errors.As(err, &generationErr) {
t.Fatalf("error = %v, want GenerationError", err)
}
if generationErr.StatusCode() != http.StatusUnauthorized ||
generationErr.ProviderType() != "authentication_error" ||
generationErr.ProviderCode() != "invalid_api_key" ||
generationErr.ProviderMessage() != "authentication failed" {
t.Fatalf("GenerationError = %+v, want structured upstream authentication failure", generationErr)
}
}
@@ -1983,6 +2018,25 @@ func TestWithProfilesRejectsDuplicateIDs(t *testing.T) {
}
}
func TestWithProfilesAcceptsDerivedDefinitionWithoutTargetFields(t *testing.T) {
_, err := promptkit.NewEngine(promptkit.Config{PromptDir: frameworkPromptDir},
promptkit.WithProfiles(promptkit.Profile{
ID: " derived-profile ",
BaseProfileID: " base-profile ",
}),
)
if err != nil {
t.Fatalf("derived profile should be accepted during construction: %v", err)
}
_, err = promptkit.NewEngine(promptkit.Config{PromptDir: frameworkPromptDir},
promptkit.WithProfiles(promptkit.Profile{ID: "standalone-profile"}),
)
if !errors.Is(err, promptkit.ErrInvalidConfig) {
t.Fatalf("standalone incomplete profile error = %v, want ErrInvalidConfig", err)
}
}
func TestWithProfilesRejectsInvalidExecutionSettings(t *testing.T) {
type testCase struct {
name string
@@ -2135,6 +2189,7 @@ func TestOpenAICompatibleProfileMapsEveryField(t *testing.T) {
extraParams := map[string]any{"provider_option": "distinct-extra-params"}
got := promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
ID: "distinct-id",
BaseProfileID: "distinct-base",
BackendID: "distinct-backend",
Endpoint: "https://distinct.example/v1",
Model: "distinct-model",
@@ -2149,6 +2204,7 @@ func TestOpenAICompatibleProfileMapsEveryField(t *testing.T) {
})
want := promptkit.Profile{
ID: "distinct-id",
BaseProfileID: "distinct-base",
BackendID: "distinct-backend",
Endpoint: "https://distinct.example/v1",
Model: "distinct-model",

View File

@@ -21,16 +21,20 @@ func mapPublicError(err error) error {
strings.TrimSpace(internalCapacityError.BackendID) != "" {
return &CapacityError{BackendID: internalCapacityError.BackendID}
}
publicErr := publicErrorFor(err)
var providerHTTPError *llm.ProviderHTTPError
if errors.As(err, &providerHTTPError) && providerHTTPError != nil {
return newGenerationError(
generationErr := newGenerationError(
providerHTTPError.StatusCode(),
providerHTTPError.ProviderCode(),
providerHTTPError.ProviderType(),
providerHTTPError.ProviderMessage(),
)
if publicErr != nil && !errors.Is(publicErr, ErrLLMGenerate) {
return fmt.Errorf("%w: %w", publicErr, generationErr)
}
return generationErr
}
publicErr := publicErrorFor(err)
if publicErr == nil {
return err
}

View File

@@ -6,6 +6,7 @@ import (
"fmt"
"testing"
"gitea.maximumdirect.net/eric/promptkit/internal/llm"
"gitea.maximumdirect.net/eric/promptkit/internal/usecase"
)
@@ -48,3 +49,27 @@ func TestMapPublicErrorTranslatesCapacityError(t *testing.T) {
t.Fatalf("mapped backend ID changed with source error: %q", publicErr.BackendID)
}
}
func TestMapPublicErrorPreservesValidationAroundGenerationError(t *testing.T) {
internalErr := fmt.Errorf(
"%w: %w",
usecase.ErrValidation,
&llm.ProviderHTTPError{},
)
err := mapPublicError(internalErr)
if !errors.Is(err, ErrValidation) {
t.Fatalf("mapped error=%v, want ErrValidation", err)
}
if !errors.Is(err, ErrLLMGenerate) {
t.Fatalf("mapped error=%v, want ErrLLMGenerate", err)
}
var generationErr *GenerationError
if !errors.As(err, &generationErr) || generationErr == nil {
t.Fatalf("mapped error=%v, want GenerationError", err)
}
var leakedInternalErr *llm.ProviderHTTPError
if errors.As(err, &leakedInternalErr) {
t.Fatalf("mapped error exposes internal ProviderHTTPError: %v", err)
}
}

View File

@@ -192,6 +192,7 @@ type BackendCapacityPolicy struct {
// ExecutionProfile describes how and where to execute a model.
type ExecutionProfile struct {
ID string `yaml:"id"`
BaseProfileID string `yaml:"base_profile"`
BackendID string `yaml:"backend"`
Endpoint string `yaml:"endpoint"`
Model string `yaml:"model"`

View File

@@ -133,13 +133,18 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
return nil, fmt.Errorf("%w: failed to create request: %v", ErrRequestFailed, err)
}
httpReq.Header.Set("Content-Type", "application/json")
if apiKey := strings.TrimSpace(req.Target.APIKey); apiKey != "" {
httpReq.Header.Set("Authorization", "Bearer "+apiKey)
} else if envName := strings.TrimSpace(req.Target.APIKeyEnv); envName != "" {
apiKey := strings.TrimSpace(os.Getenv(envName))
if apiKey == "" {
apiKey := strings.TrimSpace(req.Target.APIKey)
envName := strings.TrimSpace(req.Target.APIKeyEnv)
if apiKey == "" && envName != "" {
apiKey = strings.TrimSpace(os.Getenv(envName))
}
if apiKey == "" && req.Target.APIKeyRequired {
if envName != "" {
return nil, fmt.Errorf("%w: api key environment variable %q is not set", ErrInvalidRequest, envName)
}
return nil, fmt.Errorf("%w: api key is required", ErrInvalidRequest)
}
if apiKey != "" {
httpReq.Header.Set("Authorization", "Bearer "+apiKey)
}

View File

@@ -501,36 +501,61 @@ func checkCompleteRequestAndResponseMapping(t *testing.T) {
func TestOpenAICompatibleClientAuthentication(t *testing.T) {
tests := []struct {
name string
configureEnv func(*testing.T)
target domain.ExecutionTarget
wantAuth string
wantErr error
wantCallCount int
name string
configureEnv func(*testing.T)
target domain.ExecutionTarget
wantAuthorization string
wantError error
wantCallCount int
}{
{
name: "direct key takes precedence over environment",
configureEnv: func(t *testing.T) {
t.Setenv("PROMPTKIT_TEST_API_KEY", "env-key")
t.Setenv("PROMPTKIT_TEST_API_KEY", " env-key ")
},
target: domain.ExecutionTarget{
APIKeyEnv: "PROMPTKIT_TEST_API_KEY",
APIKey: "direct-llm-key",
APIKey: " direct-llm-key ",
},
wantAuth: "Bearer direct-llm-key",
wantCallCount: 1,
wantAuthorization: "Bearer direct-llm-key",
wantCallCount: 1,
},
{
name: "environment key supplies authorization",
configureEnv: func(t *testing.T) {
t.Setenv("PROMPTKIT_TEST_API_KEY", " env-key ")
},
target: domain.ExecutionTarget{APIKeyEnv: "PROMPTKIT_TEST_API_KEY"},
wantAuthorization: "Bearer env-key",
wantCallCount: 1,
},
{
name: "no key omits authorization",
wantCallCount: 1,
},
{
name: "missing environment key fails before transport",
name: "optional missing environment omits authorization",
configureEnv: func(t *testing.T) {
t.Setenv("PROMPTKIT_MISSING_KEY", "")
},
target: domain.ExecutionTarget{APIKeyEnv: "PROMPTKIT_MISSING_KEY"},
wantErr: ErrInvalidRequest,
target: domain.ExecutionTarget{APIKeyEnv: "PROMPTKIT_MISSING_KEY"},
wantCallCount: 1,
},
{
name: "required missing environment fails before transport",
configureEnv: func(t *testing.T) {
t.Setenv("PROMPTKIT_MISSING_KEY", "")
},
target: domain.ExecutionTarget{
APIKeyEnv: "PROMPTKIT_MISSING_KEY",
APIKeyRequired: true,
},
wantError: ErrInvalidRequest,
},
{
name: "required target without source fails before transport",
target: domain.ExecutionTarget{APIKeyRequired: true},
wantError: ErrInvalidRequest,
},
}
@@ -545,9 +570,9 @@ func TestOpenAICompatibleClientAuthentication(t *testing.T) {
request.Target = tc.target
_, err := client.Generate(context.Background(), request)
if tc.wantErr != nil {
if !errors.Is(err, tc.wantErr) {
t.Fatalf("error = %v, want %v", err, tc.wantErr)
if tc.wantError != nil {
if !errors.Is(err, tc.wantError) {
t.Fatalf("error = %v, want %v", err, tc.wantError)
}
} else if err != nil {
t.Fatalf("generate: %v", err)
@@ -556,8 +581,13 @@ func TestOpenAICompatibleClientAuthentication(t *testing.T) {
t.Fatalf("provider calls = %d, want %d", got, tc.wantCallCount)
}
if tc.wantCallCount == 1 {
if got := provider.lastRequest(t).header.Get("Authorization"); got != tc.wantAuth {
t.Fatalf("Authorization = %q, want %q", got, tc.wantAuth)
values := provider.lastRequest(t).header.Values("Authorization")
if tc.wantAuthorization == "" {
if len(values) != 0 {
t.Fatalf("Authorization values = %q, want absent", values)
}
} else if len(values) != 1 || values[0] != tc.wantAuthorization {
t.Fatalf("Authorization values = %q, want [%q]", values, tc.wantAuthorization)
}
}
})

View File

@@ -0,0 +1,47 @@
package profile
import (
"errors"
"strings"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
)
// NormalizeAndValidateDefinition normalizes and validates one source-local
// profile definition without resolving a base profile.
func NormalizeAndValidateDefinition(profile *domain.ExecutionProfile) error {
if profile == nil {
return errors.New("profile is required")
}
profile.ID = strings.TrimSpace(profile.ID)
profile.BaseProfileID = strings.TrimSpace(profile.BaseProfileID)
profile.BackendID = strings.TrimSpace(profile.BackendID)
profile.Endpoint = strings.TrimSpace(profile.Endpoint)
if profile.ID == "" {
return errors.New("id is required")
}
if profile.Endpoint != "" {
endpoint, err := domain.NormalizeOpenAICompatibleBaseEndpoint(profile.Endpoint)
if err != nil {
return err
}
profile.Endpoint = endpoint
}
if profile.BaseProfileID == "" {
if profile.BackendID == "" && profile.Endpoint == "" {
return errors.New("backend or endpoint is required")
}
if strings.TrimSpace(profile.Model) == "" {
return errors.New("model is required")
}
}
return domain.ValidateExecutionTargetSettings(domain.ExecutionTarget{
Temperature: profile.Temperature,
MaxTokens: profile.MaxTokens,
TopP: profile.TopP,
TimeoutSeconds: profile.TimeoutSeconds,
})
}

View File

@@ -127,12 +127,11 @@ func loadProfile(ctx context.Context, fsys fs.FS, root string, id string) (*doma
if prof.ID != id {
continue
}
prof.BackendID = strings.TrimSpace(prof.BackendID)
prof.ExtraParams, err = jsonvalue.CopyMap(prof.ExtraParams)
if err != nil {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, relPath, err)
}
if err := normalizeAndValidateProfile(prof); err != nil {
if err := NormalizeAndValidateDefinition(prof); err != nil {
if errors.Is(err, ErrRawAPIKeyNotAllowed) {
return nil, fmt.Errorf("%w: %s", err, relPath)
}
@@ -255,30 +254,3 @@ func requireYAMLStreamEnd(decoder *yaml.Decoder) error {
}
return errors.New("profile file must contain exactly one YAML document")
}
func normalizeAndValidateProfile(p *domain.ExecutionProfile) error {
if strings.TrimSpace(p.ID) == "" {
return errors.New("id is required")
}
p.Endpoint = strings.TrimSpace(p.Endpoint)
if strings.TrimSpace(p.BackendID) == "" && p.Endpoint == "" {
return errors.New("backend or endpoint is required")
}
if p.Endpoint != "" {
endpoint, err := domain.NormalizeOpenAICompatibleBaseEndpoint(p.Endpoint)
if err != nil {
return err
}
p.Endpoint = endpoint
}
if strings.TrimSpace(p.Model) == "" {
return errors.New("model is required")
}
return domain.ValidateExecutionTargetSettings(domain.ExecutionTarget{
Temperature: p.Temperature,
MaxTokens: p.MaxTokens,
TopP: p.TopP,
TimeoutSeconds: p.TimeoutSeconds,
})
}

View File

@@ -858,6 +858,107 @@ top_p: .inf
})
}
func TestProfileRepositoriesValidateDerivedDefinitions(t *testing.T) {
tests := []struct {
name string
files map[string]string
wantError error
wantBaseID string
wantProfile bool
}{
{
name: "alias is locally valid and normalizes base id",
files: map[string]string{"alias.yaml": `
id: selected-profile
base_profile: " base-profile "
`},
wantBaseID: "base-profile",
wantProfile: true,
},
{
name: "derived endpoint remains valid",
files: map[string]string{"invalid.yaml": `
id: selected-profile
base_profile: base-profile
endpoint: /v1
`},
wantError: ErrInvalidProfile,
},
{
name: "derived settings remain valid",
files: map[string]string{"invalid.yaml": `
id: selected-profile
base_profile: base-profile
top_p: 1.1
`},
wantError: ErrInvalidProfile,
},
{
name: "derived extra params remain valid",
files: map[string]string{"invalid.yaml": `
id: selected-profile
base_profile: base-profile
extra_params:
timestamp: 2026-08-11T12:34:56Z
`},
wantError: ErrInvalidProfile,
},
{
name: "derived raw key remains prohibited",
files: map[string]string{"invalid.yaml": `
id: selected-profile
base_profile: base-profile
api_key: secret
`},
wantError: ErrRawAPIKeyNotAllowed,
},
{
name: "derived duplicate id remains invalid",
files: map[string]string{
"first.yaml": "id: selected-profile\nbase_profile: first-base\n",
"second.yaml": "id: selected-profile\nbase_profile: second-base\n",
},
wantError: ErrInvalidProfile,
},
{
name: "derived extra document remains invalid",
files: map[string]string{"invalid.yaml": `
id: selected-profile
base_profile: base-profile
---
id: other
`},
wantError: ErrInvalidYAML,
},
{
name: "standalone profile remains complete",
files: map[string]string{"invalid.yaml": "id: selected-profile\n"},
wantError: ErrInvalidProfile,
},
}
for _, source := range profileRepositorySources() {
for _, tc := range tests {
t.Run(source.name+"/"+tc.name, func(t *testing.T) {
repo := source.newRepository(t, tc.files)
got, err := repo.GetProfile(context.Background(), "selected-profile")
if tc.wantError != nil {
if !errors.Is(err, tc.wantError) {
t.Fatalf("error = %v, want %v", err, tc.wantError)
}
return
}
if err != nil || !tc.wantProfile {
t.Fatalf("profile = %+v, error = %v, want valid derived definition", got, err)
}
if got.BaseProfileID != tc.wantBaseID {
t.Fatalf("BaseProfileID = %q, want %q", got.BaseProfileID, tc.wantBaseID)
}
})
}
}
}
func TestOverlayRepository(t *testing.T) {
ctx := context.Background()
primaryProfile := &domain.ExecutionProfile{ID: "shared", Endpoint: "http://primary", Model: "primary"}

View File

@@ -0,0 +1,160 @@
package profile
import (
"context"
"errors"
"fmt"
"strings"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
"gitea.maximumdirect.net/eric/promptkit/internal/jsonvalue"
)
const maximumProfileChainLength = 32
type resolvingRepository struct {
source Repository
}
// NewResolvingRepository resolves inherited profile definitions from source.
func NewResolvingRepository(source Repository) Repository {
return &resolvingRepository{source: source}
}
func (r *resolvingRepository) GetProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) {
if r == nil || r.source == nil {
return nil, fmt.Errorf("%w: profile repository is required", ErrInvalidProfile)
}
requestedID := strings.TrimSpace(id)
if requestedID == "" {
return nil, fmt.Errorf("%w: profile id is required", ErrInvalidProfile)
}
if err := ctx.Err(); err != nil {
return nil, err
}
profile, err := r.getRawProfile(ctx, requestedID)
if err != nil {
return nil, err
}
if profile == nil {
return nil, fmt.Errorf("%w: selected profile %q is nil", ErrInvalidProfile, requestedID)
}
chain := []*domain.ExecutionProfile{profile}
chainIDs := []string{requestedID}
visited := map[string]struct{}{requestedID: {}}
current := profile
for {
baseID := strings.TrimSpace(current.BaseProfileID)
if baseID == "" {
break
}
if err := ctx.Err(); err != nil {
return nil, err
}
if _, seen := visited[baseID]; seen {
return nil, fmt.Errorf("%w: profile inheritance cycle %s", ErrInvalidProfile, joinProfileChain(chainIDs, baseID))
}
if len(chain) >= maximumProfileChainLength {
return nil, fmt.Errorf("%w: profile inheritance chain exceeds %d profiles: %s", ErrInvalidProfile, maximumProfileChainLength, joinProfileChain(chainIDs, baseID))
}
base, err := r.getRawProfile(ctx, baseID)
if err != nil {
if errors.Is(err, ErrProfileNotFound) {
return nil, fmt.Errorf("%w: base profile %q is missing in chain %s", ErrInvalidProfile, baseID, joinProfileChain(chainIDs, baseID))
}
return nil, fmt.Errorf("%w: failed to load base profile %q in chain %s: %w", ErrInvalidProfile, baseID, joinProfileChain(chainIDs, baseID), err)
}
if base == nil {
return nil, fmt.Errorf("%w: base profile %q is nil in chain %s", ErrInvalidProfile, baseID, joinProfileChain(chainIDs, baseID))
}
chain = append(chain, base)
chainIDs = append(chainIDs, baseID)
visited[baseID] = struct{}{}
current = base
}
resolved, err := mergeProfileChain(chain)
if err != nil {
return nil, fmt.Errorf("%w: resolved profile chain %s: %w", ErrInvalidProfile, strings.Join(chainIDs, " -> "), err)
}
if err := validateResolvedProfile(resolved); err != nil {
return nil, fmt.Errorf("%w: resolved profile chain %s: %w", ErrInvalidProfile, strings.Join(chainIDs, " -> "), err)
}
return resolved, nil
}
func (r *resolvingRepository) getRawProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) {
profile, err := r.source.GetProfile(ctx, id)
if err != nil {
return nil, err
}
if err := ctx.Err(); err != nil {
return nil, err
}
return profile, nil
}
func joinProfileChain(chain []string, next string) string {
return strings.Join(append(append([]string(nil), chain...), next), " -> ")
}
func mergeProfileChain(chain []*domain.ExecutionProfile) (*domain.ExecutionProfile, error) {
resolved := &domain.ExecutionProfile{ID: chain[0].ID}
for index := len(chain) - 1; index >= 0; index-- {
definition := chain[index]
if strings.TrimSpace(definition.BackendID) != "" {
resolved.BackendID = definition.BackendID
}
if strings.TrimSpace(definition.Endpoint) != "" {
resolved.Endpoint = definition.Endpoint
}
if strings.TrimSpace(definition.Model) != "" {
resolved.Model = definition.Model
}
if definition.Temperature != 0 {
resolved.Temperature = definition.Temperature
}
if definition.MaxTokens != 0 {
resolved.MaxTokens = definition.MaxTokens
}
if definition.TopP != 0 {
resolved.TopP = definition.TopP
}
if definition.TimeoutSeconds != 0 {
resolved.TimeoutSeconds = definition.TimeoutSeconds
}
if strings.TrimSpace(definition.ServiceTier) != "" {
resolved.ServiceTier = definition.ServiceTier
}
if strings.TrimSpace(definition.ReasoningEffort) != "" {
resolved.ReasoningEffort = definition.ReasoningEffort
}
if strings.TrimSpace(definition.APIKeyEnv) != "" {
resolved.APIKeyEnv = definition.APIKeyEnv
}
resolved.APIKeyRequired = resolved.APIKeyRequired || definition.APIKeyRequired
if len(definition.ExtraParams) != 0 {
extraParams, err := jsonvalue.CopyMap(definition.ExtraParams)
if err != nil {
return nil, err
}
resolved.ExtraParams = extraParams
}
}
resolved.ID = chain[0].ID
resolved.BaseProfileID = ""
return resolved, nil
}
func validateResolvedProfile(profile *domain.ExecutionProfile) error {
if profile == nil {
return errors.New("resolved profile is required")
}
profile.BaseProfileID = ""
return NormalizeAndValidateDefinition(profile)
}

View File

@@ -0,0 +1,418 @@
package profile
import (
"context"
"errors"
"fmt"
"io/fs"
"reflect"
"strings"
"sync"
"testing"
"testing/fstest"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
)
func TestResolvingRepositoryMergesProfileChain(t *testing.T) {
repo := &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"leaf": {
ID: "leaf",
BaseProfileID: "middle",
BackendID: "leaf-backend",
TopP: 0.8,
TimeoutSeconds: 45,
ReasoningEffort: "high",
},
"middle": {
ID: "middle",
BaseProfileID: "root",
Endpoint: "https://middle.example/v1",
Model: "middle-model",
MaxTokens: 256,
APIKeyEnv: "MIDDLE_API_KEY",
APIKeyRequired: true,
ExtraParams: map[string]any{"middle": map[string]any{"value": "middle"}},
},
"root": {
ID: "root",
BackendID: "root-backend",
Endpoint: "https://root.example/v1",
Model: "root-model",
Temperature: 0.3,
ServiceTier: "priority",
ExtraParams: map[string]any{"root": "value"},
},
}}
got, err := NewResolvingRepository(repo).GetProfile(context.Background(), "leaf")
if err != nil {
t.Fatalf("resolve profile: %v", err)
}
want := &domain.ExecutionProfile{
ID: "leaf",
BackendID: "leaf-backend",
Endpoint: "https://middle.example/v1",
Model: "middle-model",
Temperature: 0.3,
MaxTokens: 256,
TopP: 0.8,
TimeoutSeconds: 45,
ServiceTier: "priority",
ReasoningEffort: "high",
APIKeyEnv: "MIDDLE_API_KEY",
APIKeyRequired: true,
ExtraParams: map[string]any{"middle": map[string]any{"value": "middle"}},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("resolved profile:\n got %#v\nwant %#v", got, want)
}
}
func TestResolvingRepositoryRejectsMissingSourceAndProfileID(t *testing.T) {
if _, err := NewResolvingRepository(nil).GetProfile(context.Background(), "profile"); !errors.Is(err, ErrInvalidProfile) {
t.Fatalf("nil source error = %v, want ErrInvalidProfile", err)
}
repo := &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{}}
if _, err := NewResolvingRepository(repo).GetProfile(context.Background(), " \t "); !errors.Is(err, ErrInvalidProfile) {
t.Fatalf("blank id error = %v, want ErrInvalidProfile", err)
}
if got := repo.callCount(" "); got != 0 {
t.Fatalf("blank id looked up source %d times", got)
}
}
func TestResolvingRepositoryCopiesExtraParams(t *testing.T) {
baseParams := map[string]any{"nested": map[string]any{"value": "base"}}
repo := &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"child": {ID: "child", BaseProfileID: "base"},
"base": {
ID: "base",
Endpoint: "https://base.example/v1",
Model: "model",
ExtraParams: baseParams,
},
}}
resolver := NewResolvingRepository(repo)
first, err := resolver.GetProfile(context.Background(), "child")
if err != nil {
t.Fatalf("resolve inherited map: %v", err)
}
first.ExtraParams["nested"].(map[string]any)["value"] = "mutated"
second, err := resolver.GetProfile(context.Background(), "child")
if err != nil {
t.Fatalf("resolve inherited map again: %v", err)
}
if got := second.ExtraParams["nested"].(map[string]any)["value"]; got != "base" {
t.Fatalf("later result retained mutation: %v", got)
}
if got := baseParams["nested"].(map[string]any)["value"]; got != "base" {
t.Fatalf("source map retained mutation: %v", got)
}
repo.set("child", &domain.ExecutionProfile{
ID: "child",
BaseProfileID: "base",
ExtraParams: map[string]any{"child": "replacement"},
})
replaced, err := resolver.GetProfile(context.Background(), "child")
if err != nil {
t.Fatalf("resolve replacement map: %v", err)
}
if !reflect.DeepEqual(replaced.ExtraParams, map[string]any{"child": "replacement"}) {
t.Fatalf("extra params = %#v, want complete child replacement", replaced.ExtraParams)
}
}
func TestResolvingRepositoryUsesRawOverlayForEachLookup(t *testing.T) {
leafSource := NewFSRepository(profileTestFS(map[string]string{
"leaf.yaml": "id: leaf\nbase_profile: base\n",
}), ".")
fallback := NewFSRepository(profileTestFS(map[string]string{
"base.yaml": "id: base\nendpoint: https://fallback.example/v1\nmodel: fallback-model\n",
}), ".")
overlay := NewOverlayRepository(leafSource, fallback)
resolver := NewResolvingRepository(overlay)
got, err := resolver.GetProfile(context.Background(), "leaf")
if err != nil {
t.Fatalf("resolve fallback base: %v", err)
}
if got.Model != "fallback-model" {
t.Fatalf("fallback base model = %q", got.Model)
}
shadowing := NewOverlayRepository(NewFSRepository(profileTestFS(map[string]string{
"leaf.yaml": "id: leaf\nbase_profile: base\n",
"base.yaml": "id: base\nendpoint: https://primary.example/v1\nmodel: primary-model\n",
}), "."), fallback)
got, err = NewResolvingRepository(shadowing).GetProfile(context.Background(), "leaf")
if err != nil {
t.Fatalf("resolve shadowed base: %v", err)
}
if got.Model != "primary-model" || got.Endpoint != "https://primary.example/v1" {
t.Fatalf("shadowed base = %+v", got)
}
}
func TestResolvingRepositoryReportsSafetyAndSourceErrors(t *testing.T) {
sourceErr := errors.New("source failure")
tests := []struct {
name string
repo *resolvingTestRepository
id string
want []error
wantNot error
contains []string
}{
{
name: "missing selected profile preserves not found",
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{}},
id: "missing",
want: []error{ErrProfileNotFound},
wantNot: ErrInvalidProfile,
},
{
name: "missing base is invalid but not not found",
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"leaf": {ID: "leaf", BaseProfileID: "missing"},
}},
id: "leaf",
want: []error{ErrInvalidProfile},
wantNot: ErrProfileNotFound,
contains: []string{"missing", "leaf -> missing"},
},
{
name: "direct cycle",
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"a": {ID: "a", BaseProfileID: "a"},
}},
id: "a",
want: []error{ErrInvalidProfile},
contains: []string{"a -> a"},
},
{
name: "indirect cycle",
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"a": {ID: "a", BaseProfileID: "b"},
"b": {ID: "b", BaseProfileID: "c"},
"c": {ID: "c", BaseProfileID: "a"},
}},
id: "a",
want: []error{ErrInvalidProfile},
contains: []string{"a -> b -> c -> a"},
},
{
name: "nil result",
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"leaf": nil,
}},
id: "leaf",
want: []error{ErrInvalidProfile},
},
{
name: "incomplete resolved profile",
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"leaf": {ID: "leaf", BaseProfileID: "base"},
"base": {ID: "base", Model: "model"},
}},
id: "leaf",
want: []error{ErrInvalidProfile},
},
{
name: "base source error is retained",
repo: &resolvingTestRepository{
profiles: map[string]*domain.ExecutionProfile{"leaf": {ID: "leaf", BaseProfileID: "base"}},
errors: map[string]error{"base": sourceErr},
},
id: "leaf",
want: []error{ErrInvalidProfile, sourceErr},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
_, err := NewResolvingRepository(tc.repo).GetProfile(context.Background(), tc.id)
for _, want := range tc.want {
if !errors.Is(err, want) {
t.Fatalf("error = %v, want %v", err, want)
}
}
if tc.wantNot != nil && errors.Is(err, tc.wantNot) {
t.Fatalf("error = %v, must not match %v", err, tc.wantNot)
}
for _, fragment := range tc.contains {
if !strings.Contains(err.Error(), fragment) {
t.Fatalf("error = %v, want %q", err, fragment)
}
}
})
}
}
func TestResolvingRepositoryEnforcesChainLength(t *testing.T) {
for _, count := range []int{maximumProfileChainLength, maximumProfileChainLength + 1} {
t.Run(fmt.Sprintf("%d profiles", count), func(t *testing.T) {
profiles := make(map[string]*domain.ExecutionProfile, count)
for index := 1; index <= count; index++ {
id := fmt.Sprintf("profile-%d", index)
definition := &domain.ExecutionProfile{ID: id}
if index == count {
definition.Endpoint = "https://root.example/v1"
definition.Model = "model"
} else {
definition.BaseProfileID = fmt.Sprintf("profile-%d", index+1)
}
profiles[id] = definition
}
got, err := NewResolvingRepository(&resolvingTestRepository{profiles: profiles}).GetProfile(context.Background(), "profile-1")
if count == maximumProfileChainLength {
if err != nil || got == nil {
t.Fatalf("profile = %+v, error = %v, want accepted chain", got, err)
}
return
}
if !errors.Is(err, ErrInvalidProfile) {
t.Fatalf("error = %v, want ErrInvalidProfile", err)
}
})
}
}
func TestResolvingRepositoryIsFreshAndCancellationAware(t *testing.T) {
repo := &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"leaf": {ID: "leaf", BaseProfileID: "base"},
"base": {ID: "base", Endpoint: "https://base.example/v1", Model: "first", ExtraParams: map[string]any{"nested": map[string]any{"value": "first"}}},
}}
resolver := NewResolvingRepository(repo)
first, err := resolver.GetProfile(context.Background(), "leaf")
if err != nil || first.Model != "first" {
t.Fatalf("first result=(%+v, %v)", first, err)
}
repo.set("base", &domain.ExecutionProfile{ID: "base", Endpoint: "https://base.example/v1", Model: "second", ExtraParams: map[string]any{"nested": map[string]any{"value": "second"}}})
second, err := resolver.GetProfile(context.Background(), "leaf")
if err != nil || second.Model != "second" {
t.Fatalf("second result=(%+v, %v)", second, err)
}
canceled, cancel := context.WithCancel(context.Background())
cancel()
if _, err := resolver.GetProfile(canceled, "leaf"); !errors.Is(err, context.Canceled) {
t.Fatalf("canceled lookup error = %v", err)
}
if got := repo.callCount("leaf"); got != 2 {
t.Fatalf("calls after canceled lookup = %d, want 2", got)
}
duringTraversal, cancelDuringTraversal := context.WithCancel(context.Background())
repo.afterGet = func(id string) {
if id == "leaf" {
cancelDuringTraversal()
}
}
if _, err := resolver.GetProfile(duringTraversal, "leaf"); !errors.Is(err, context.Canceled) {
t.Fatalf("during traversal error = %v", err)
}
if got := repo.callCount("base"); got != 2 {
t.Fatalf("base calls after cancellation = %d, want 2", got)
}
terminalLookup, cancelTerminalLookup := context.WithCancel(context.Background())
repo.afterGet = func(id string) {
if id == "base" {
cancelTerminalLookup()
}
}
if _, err := resolver.GetProfile(terminalLookup, "leaf"); !errors.Is(err, context.Canceled) {
t.Fatalf("terminal lookup cancellation error = %v", err)
}
if got := repo.callCount("base"); got != 3 {
t.Fatalf("base calls after terminal cancellation = %d, want 3", got)
}
repo.afterGet = nil
var wg sync.WaitGroup
errors := make(chan error, 8)
for index := 0; index < cap(errors); index++ {
wg.Add(1)
go func() {
defer wg.Done()
resolved, err := resolver.GetProfile(context.Background(), "leaf")
if err != nil {
errors <- err
return
}
resolved.ExtraParams["nested"].(map[string]any)["value"] = "mutated"
}()
}
wg.Wait()
close(errors)
for err := range errors {
t.Errorf("concurrent resolution: %v", err)
}
latest, err := resolver.GetProfile(context.Background(), "leaf")
if err != nil || latest.ExtraParams["nested"].(map[string]any)["value"] != "second" {
t.Fatalf("latest result=(%+v, %v)", latest, err)
}
}
type resolvingTestRepository struct {
mu sync.Mutex
profiles map[string]*domain.ExecutionProfile
errors map[string]error
calls map[string]int
afterGet func(string)
}
func (r *resolvingTestRepository) GetProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
r.mu.Lock()
if r.calls == nil {
r.calls = make(map[string]int)
}
r.calls[id]++
err := r.errors[id]
profile := r.profiles[id]
afterGet := r.afterGet
r.mu.Unlock()
if afterGet != nil {
afterGet(id)
}
if err != nil {
return nil, err
}
if profile == nil {
if _, exists := r.profiles[id]; exists {
return nil, nil
}
return nil, ErrProfileNotFound
}
copy := *profile
return &copy, nil
}
func (r *resolvingTestRepository) set(id string, profile *domain.ExecutionProfile) {
r.mu.Lock()
defer r.mu.Unlock()
r.profiles[id] = profile
}
func (r *resolvingTestRepository) callCount(id string) int {
r.mu.Lock()
defer r.mu.Unlock()
return r.calls[id]
}
func profileTestFS(files map[string]string) fs.FS {
fsys := make(fstest.MapFS, len(files))
for name, content := range files {
fsys[name] = profileMapFile(content)
}
return fsys
}

View File

@@ -3,7 +3,6 @@ package usecase
import (
"context"
"errors"
"os"
"reflect"
"testing"
@@ -241,49 +240,76 @@ func excessivelyDeepPreparedJSONValue() any {
return value
}
func TestRunnerRunPreparedRechecksEnvironmentCredentialBeforeAdmission(t *testing.T) {
func TestRunnerRunPreparedCredentialAvailabilityBeforeAdmission(t *testing.T) {
const environmentName = "PROMPTKIT_PREPARED_EXECUTION_TEST_KEY"
t.Setenv(environmentName, "available-during-preparation")
profile := defaultExecutionProfile()
profile.APIKeyEnv = environmentName
validator := &recordingValidationPreparer{plan: &recordingPreparedValidation{}}
admitter := &fakeRunAdmitter{}
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "unexpected"}}
runner := NewRunner(
&fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": profile}},
nil,
defaultArtifactReader(),
defaultRenderer(),
llmClient,
validator,
admitter,
)
prepared, err := runner.PrepareExecution(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
})
if err != nil {
t.Fatalf("prepare execution: %v", err)
}
if err := os.Unsetenv(environmentName); err != nil {
t.Fatalf("unset credential environment: %v", err)
tests := []struct {
name string
apiKeyRequired bool
profileEnv bool
overrideEnv bool
wantFailure bool
}{
{name: "optional environment becomes unavailable", profileEnv: true},
{
name: "required request environment becomes unavailable",
apiKeyRequired: true,
overrideEnv: true,
wantFailure: true,
},
}
result, err := runner.RunPrepared(context.Background(), prepared)
if result != nil {
t.Fatalf("credential failure returned partial result: %+v", result)
}
if !errors.Is(err, ErrInvalidRequest) || !errors.Is(err, ErrAPIKeyEnvMissing) {
t.Fatalf("credential error identities are missing: %v", err)
}
if len(admitter.backendIDs) != 0 || llmClient.calls != 0 {
t.Fatalf("credential failure reached admission or generation: admission=%v generation=%d", admitter.backendIDs, llmClient.calls)
}
if _, err := runner.RunPrepared(context.Background(), prepared); !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("credential failure did not consume execution: %v", err)
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Setenv(environmentName, "available-during-preparation")
profile := defaultExecutionProfile()
profile.APIKeyRequired = tc.apiKeyRequired
if tc.profileEnv {
profile.APIKeyEnv = environmentName
}
validator := &recordingValidationPreparer{plan: &recordingPreparedValidation{}}
admitter := &fakeRunAdmitter{}
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}
runner := NewRunner(
&fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": profile}},
nil,
defaultArtifactReader(),
defaultRenderer(),
llmClient,
validator,
admitter,
)
request := domain.RunRequest{PromptID: "p", ProfileID: "exec", Inputs: singleInputRef()}
if tc.overrideEnv {
request.Execution = &domain.ExecutionTargetOverride{APIKeyEnv: environmentName}
}
prepared, err := runner.PrepareExecution(context.Background(), request)
if err != nil {
t.Fatalf("prepare execution: %v", err)
}
t.Setenv(environmentName, "")
result, err := runner.RunPrepared(context.Background(), prepared)
if tc.wantFailure {
if result != nil || !errors.Is(err, ErrInvalidRequest) || !errors.Is(err, ErrAPIKeyEnvMissing) {
t.Fatalf("required credential result = (%+v, %v)", result, err)
}
if len(admitter.backendIDs) != 0 || llmClient.calls != 0 {
t.Fatalf("required credential reached admission or generation: admission=%v generation=%d", admitter.backendIDs, llmClient.calls)
}
} else {
if result == nil || err != nil {
t.Fatalf("optional credential result = (%+v, %v), want success", result, err)
}
if len(admitter.backendIDs) != 1 || llmClient.calls != 1 {
t.Fatalf("optional credential admission=%v generation=%d, want one each", admitter.backendIDs, llmClient.calls)
}
}
if _, err := runner.RunPrepared(context.Background(), prepared); !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("execution outcome did not consume handle: %v", err)
}
})
}
}

View File

@@ -624,12 +624,12 @@ func validateAPIKey(apiKeyEnv string, apiKey string, apiKeyRequired bool) error
if strings.TrimSpace(apiKey) != "" {
return nil
}
if !apiKeyRequired {
return nil
}
envName := strings.TrimSpace(apiKeyEnv)
if envName == "" {
if apiKeyRequired {
return ErrAPIKeyRequired
}
return nil
return ErrAPIKeyRequired
}
if strings.TrimSpace(os.Getenv(envName)) == "" {
return fmt.Errorf("%w: api key environment variable %q is not set", ErrAPIKeyEnvMissing, envName)

View File

@@ -1787,22 +1787,26 @@ func TestRunnerRunAPIKeyEnvResolvesFromEnvironment(t *testing.T) {
}
}
func TestRunnerRunAPIKeyEnvMissingEnvironmentValueFailsClearly(t *testing.T) {
func TestRunnerRunOptionalAPIKeyEnvMissingEnvironmentValueReachesLLM(t *testing.T) {
const environmentName = "PROMPTKIT_MISSING_KEY"
t.Setenv(environmentName, "")
promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)}
execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{
"exec": {ID: "exec", Endpoint: "http://profile/v1", Model: "profile-model", APIKeyEnv: "PROMPTKIT_MISSING_KEY"},
"exec": {ID: "exec", Endpoint: "http://profile/v1", Model: "profile-model", APIKeyEnv: environmentName},
}}
runner := NewRunner(promptRepo, execRepo, nil, defaultArtifactReader(), defaultRenderer(), &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}, nil, nil)
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}
runner := NewRunner(promptRepo, execRepo, nil, defaultArtifactReader(), defaultRenderer(), llmClient, nil, nil)
_, err := runner.Run(context.Background(), domain.RunRequest{PromptID: "p", ProfileID: "exec", Inputs: singleInputRef()})
if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected ErrInvalidRequest, got %v", err)
result, err := runner.Run(context.Background(), domain.RunRequest{PromptID: "p", ProfileID: "exec", Inputs: singleInputRef()})
if err != nil || result == nil {
t.Fatalf("optional credential run = (%+v, %v), want success", result, err)
}
if !errors.Is(err, ErrAPIKeyEnvMissing) {
t.Fatalf("expected ErrAPIKeyEnvMissing, got %v", err)
if llmClient.calls != 1 {
t.Fatalf("LLM calls = %d, want 1", llmClient.calls)
}
if !strings.Contains(err.Error(), "PROMPTKIT_MISSING_KEY") {
t.Fatalf("expected missing env name in error, got %v", err)
if llmClient.lastReq.Target.APIKeyEnv != environmentName {
t.Fatalf("LLM api_key_env = %q, want %q", llmClient.lastReq.Target.APIKeyEnv, environmentName)
}
}

View File

@@ -5,7 +5,6 @@ import (
"encoding/json"
"errors"
"fmt"
"os"
"reflect"
"strings"
"sync"
@@ -524,19 +523,25 @@ func TestPreparedExecutionCredentialCapacityAndTimingBoundaries(t *testing.T) {
engine, err := promptkit.NewEngine(
promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prepared", "profile", "content"), "."),
promptkit.WithProfileFS(preparedCredentialProfileSource(environmentName), "."),
promptkit.WithProfiles(promptkit.Profile{
ID: "profile",
Endpoint: "http://example.test/v1",
Model: "model",
APIKeyRequired: true,
}),
promptkit.WithLLMClient(client),
)
if err != nil {
t.Fatalf("construct credential engine: %v", err)
}
prepared, err := engine.PrepareExecution(context.Background(), promptkit.RunRequest{PromptID: "prepared"})
prepared, err := engine.PrepareExecution(context.Background(), promptkit.RunRequest{
PromptID: "prepared",
Execution: &promptkit.ExecutionTargetOverride{APIKeyEnv: environmentName},
})
if err != nil {
t.Fatalf("prepare credential execution: %v", err)
}
if err := os.Unsetenv(environmentName); err != nil {
t.Fatalf("unset credential environment: %v", err)
}
t.Setenv(environmentName, "")
result, err := engine.RunPrepared(context.Background(), prepared)
if result != nil ||
@@ -755,16 +760,6 @@ model: ` + model + `
}
}
func preparedCredentialProfileSource(environmentName string) fstest.MapFS {
return fstest.MapFS{
"profile.yaml": &fstest.MapFile{Data: []byte(`id: profile
endpoint: http://example.test/v1
model: model
api_key_env: ` + environmentName + `
`)},
}
}
func preparedSchemaSource() fstest.MapFS {
return fstest.MapFS{
"schema.json": &fstest.MapFile{Data: []byte(`{

View File

@@ -0,0 +1,243 @@
package promptkit_test
import (
"context"
"errors"
"io/fs"
"strings"
"sync"
"testing"
"testing/fstest"
"gitea.maximumdirect.net/eric/promptkit"
)
func TestProfileInheritanceBuiltInAliasWorkflow(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{
PromptDir: frameworkPromptDir,
SchemaDir: frameworkSchemaDir,
}, promptkit.WithProfiles(promptkit.Profile{
ID: "weather-light",
BaseProfileID: "deepseek-4-flash",
ReasoningEffort: "high",
TimeoutSeconds: 120,
}))
if err != nil {
t.Fatalf("construct alias engine: %v", err)
}
base, err := engine.InspectProfile(context.Background(), "deepseek-4-flash")
if err != nil {
t.Fatalf("inspect base: %v", err)
}
child, err := engine.InspectProfile(context.Background(), "weather-light")
if err != nil {
t.Fatalf("inspect alias: %v", err)
}
if child.ProfileID != "weather-light" ||
child.EffectiveModelParams.BackendID != base.EffectiveModelParams.BackendID ||
child.EffectiveModelParams.Model != base.EffectiveModelParams.Model ||
child.EffectiveModelParams.ReasoningEffort != "high" ||
child.EffectiveModelParams.TimeoutSeconds != 120 {
t.Fatalf("alias inspection = %+v, base = %+v", child, base)
}
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
PromptID: frameworkMarkdownSummaryPromptID,
ProfileID: "weather-light",
Inputs: map[string]promptkit.ArtifactRef{
"transcript": promptkit.Inline("Rin opens the gate."),
"glossary": promptkit.Inline("gate: A guarded passage."),
},
})
if err != nil {
t.Fatalf("prepare alias: %v", err)
}
if prepared.SelectedProfileID != "weather-light" {
t.Fatalf("SelectedProfileID = %q", prepared.SelectedProfileID)
}
timeout := 15
reasoning := "low"
overridden, err := engine.Prepare(context.Background(), promptkit.RunRequest{
PromptID: frameworkMarkdownSummaryPromptID,
ProfileID: "weather-light",
Execution: &promptkit.ExecutionTargetOverride{
TimeoutSeconds: &timeout,
ReasoningEffort: &reasoning,
},
Inputs: map[string]promptkit.ArtifactRef{
"transcript": promptkit.Inline("Rin opens the gate."),
"glossary": promptkit.Inline("gate: A guarded passage."),
},
})
if err != nil {
t.Fatalf("prepare override: %v", err)
}
if overridden.EffectiveModelParams.TimeoutSeconds != timeout ||
overridden.EffectiveModelParams.ReasoningEffort != reasoning {
t.Fatalf("runtime override target = %+v", overridden.EffectiveModelParams)
}
}
func TestProfileInheritanceYAMLAliasOfBuiltIn(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(fstest.MapFS{}, "."),
promptkit.WithProfileFS(fstest.MapFS{
"alias.yaml": &fstest.MapFile{Data: []byte("id: yaml-alias\nbase_profile: deepseek-4-flash\n")},
}, "."),
)
if err != nil {
t.Fatalf("construct YAML alias engine: %v", err)
}
inspection, err := engine.InspectProfile(context.Background(), "yaml-alias")
if err != nil {
t.Fatalf("inspect YAML alias: %v", err)
}
if inspection.ProfileID != "yaml-alias" || inspection.EffectiveModelParams.Model == "" {
t.Fatalf("YAML alias inspection = %+v", inspection)
}
}
func TestProfileInheritanceRetainsRequiredCredentialBehavior(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(fstest.MapFS{}, "."),
promptkit.WithBackend(promptkit.Backend{
ID: "credential-backend",
Endpoint: "https://credential.example/v1",
APIKeyEnv: "OPTIONAL_BACKEND_KEY",
}),
promptkit.WithProfiles(
promptkit.Profile{
ID: "credential-base",
BackendID: "credential-backend",
Model: "model",
APIKeyRequired: true,
},
promptkit.Profile{ID: "credential-child", BaseProfileID: "credential-base"},
),
)
if err != nil {
t.Fatalf("construct credential inheritance engine: %v", err)
}
inspection, err := engine.InspectProfile(context.Background(), "credential-child")
if err != nil {
t.Fatalf("inspect credential child: %v", err)
}
if !inspection.APIKeyRequired || inspection.EffectiveModelParams.APIKeyEnv != "" {
t.Fatalf("credential inspection = %+v", inspection)
}
}
func TestProfileInheritancePreservesPublicErrorIdentities(t *testing.T) {
newEngine := func(t *testing.T, profiles fs.FS) *promptkit.Engine {
t.Helper()
options := []promptkit.Option{promptkit.WithPromptFS(fstest.MapFS{}, ".")}
if profiles != nil {
options = append(options, promptkit.WithProfileFS(profiles, "."))
}
engine, err := promptkit.NewEngine(promptkit.Config{}, options...)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
return engine
}
tests := []struct {
name string
profiles fs.FS
profile string
contains []string
want error
wantNot error
}{
{
name: "missing selected profile",
profile: "missing",
want: promptkit.ErrProfileNotFound,
wantNot: promptkit.ErrProfileLoad,
},
{
name: "missing base",
profiles: fstest.MapFS{
"child.yaml": &fstest.MapFile{Data: []byte("id: child\nbase_profile: missing\n")},
},
profile: "child",
contains: []string{"child", "missing"},
want: promptkit.ErrProfileLoad,
wantNot: promptkit.ErrProfileNotFound,
},
{
name: "cycle",
profiles: fstest.MapFS{
"a.yaml": &fstest.MapFile{Data: []byte("id: a\nbase_profile: b\n")},
"b.yaml": &fstest.MapFile{Data: []byte("id: b\nbase_profile: a\n")},
},
profile: "a",
contains: []string{"a", "b"},
want: promptkit.ErrProfileLoad,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
result, err := newEngine(t, tc.profiles).InspectProfile(context.Background(), tc.profile)
if result != nil || !errors.Is(err, tc.want) || (tc.wantNot != nil && errors.Is(err, tc.wantNot)) {
t.Fatalf("inspection=(%+v, %v), want %v without %v", result, err, tc.want, tc.wantNot)
}
for _, fragment := range tc.contains {
if !strings.Contains(err.Error(), fragment) {
t.Fatalf("error = %v, want %q", err, fragment)
}
}
})
}
}
func TestProfileInheritanceFreezesPreparedExecution(t *testing.T) {
profiles := &mutableInheritanceProfileFS{files: fstest.MapFS{
"child.yaml": &fstest.MapFile{Data: []byte("id: child\nbase_profile: base\n")},
"base.yaml": &fstest.MapFile{Data: []byte("id: base\nendpoint: https://base.example/v1\nmodel: first-model\n")},
}}
client := &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}}
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prepared", "child", "content"), "."),
promptkit.WithProfileFS(profiles, "."),
promptkit.WithLLMClient(client),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
prepared, err := engine.PrepareExecution(context.Background(), promptkit.RunRequest{PromptID: "prepared"})
if err != nil {
t.Fatalf("prepare execution: %v", err)
}
profiles.set("base.yaml", "id: base\nendpoint: https://base.example/v1\nmodel: second-model\n")
result, err := engine.RunPrepared(context.Background(), prepared)
if err != nil || result == nil || len(client.requests) != 1 || client.requests[0].Target.Model != "first-model" {
t.Fatalf("prepared execution=(%+v, %v), requests=%+v", result, err, client.requests)
}
inspection, err := engine.InspectProfile(context.Background(), "child")
if err != nil || inspection.EffectiveModelParams.Model != "second-model" {
t.Fatalf("fresh inspection=(%+v, %v)", inspection, err)
}
}
type mutableInheritanceProfileFS struct {
mu sync.RWMutex
files fstest.MapFS
}
func (f *mutableInheritanceProfileFS) Open(name string) (fs.File, error) {
f.mu.RLock()
defer f.mu.RUnlock()
return f.files.Open(name)
}
func (f *mutableInheritanceProfileFS) set(name, content string) {
f.mu.Lock()
defer f.mu.Unlock()
f.files[name] = &fstest.MapFile{Data: []byte(content)}
}

View File

@@ -2,17 +2,16 @@ package promptkit
import (
"context"
"errors"
"fmt"
"strings"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
"gitea.maximumdirect.net/eric/promptkit/internal/jsonvalue"
"gitea.maximumdirect.net/eric/promptkit/internal/profile"
)
// OpenAICompatibleProfile returns an ordinary in-memory Profile for an
// OpenAI-compatible chat-completions endpoint.
// OpenAICompatibleProfile returns an in-memory Profile for an OpenAI-compatible
// chat-completions endpoint. A non-blank BaseProfileID permits its target
// fields to be inherited when the profile is selected or inspected.
//
// It does not register global state, maintain a model catalog, or resolve
// credentials. If APIKeyRequired is true, callers satisfy it with
@@ -25,6 +24,7 @@ import (
func OpenAICompatibleProfile(cfg OpenAICompatibleProfileConfig) Profile {
return Profile{
ID: cfg.ID,
BaseProfileID: cfg.BaseProfileID,
BackendID: cfg.BackendID,
Endpoint: cfg.Endpoint,
Model: cfg.Model,
@@ -87,8 +87,9 @@ func toDomainProfile(publicProfile Profile) (domain.ExecutionProfile, error) {
return domain.ExecutionProfile{}, err
}
prof := domain.ExecutionProfile{
ID: strings.TrimSpace(publicProfile.ID),
BackendID: strings.TrimSpace(publicProfile.BackendID),
ID: publicProfile.ID,
BaseProfileID: publicProfile.BaseProfileID,
BackendID: publicProfile.BackendID,
Endpoint: publicProfile.Endpoint,
Model: publicProfile.Model,
Temperature: publicProfile.Temperature,
@@ -100,34 +101,8 @@ func toDomainProfile(publicProfile Profile) (domain.ExecutionProfile, error) {
APIKeyRequired: publicProfile.APIKeyRequired,
ExtraParams: extraParams,
}
if err := normalizeAndValidatePublicProfile(&prof); err != nil {
if err := profile.NormalizeAndValidateDefinition(&prof); err != nil {
return domain.ExecutionProfile{}, err
}
return prof, nil
}
func normalizeAndValidatePublicProfile(prof *domain.ExecutionProfile) error {
if strings.TrimSpace(prof.ID) == "" {
return errors.New("id is required")
}
prof.Endpoint = strings.TrimSpace(prof.Endpoint)
if strings.TrimSpace(prof.BackendID) == "" && prof.Endpoint == "" {
return errors.New("backend or endpoint is required")
}
if prof.Endpoint != "" {
endpoint, err := domain.NormalizeOpenAICompatibleBaseEndpoint(prof.Endpoint)
if err != nil {
return err
}
prof.Endpoint = endpoint
}
if strings.TrimSpace(prof.Model) == "" {
return errors.New("model is required")
}
return domain.ValidateExecutionTargetSettings(domain.ExecutionTarget{
Temperature: prof.Temperature,
MaxTokens: prof.MaxTokens,
TopP: prof.TopP,
TimeoutSeconds: prof.TimeoutSeconds,
})
}

View File

@@ -322,7 +322,10 @@ type ExecutionTarget struct {
// ReasoningEffort is the effective opaque provider-specific reasoning
// setting. An empty value instructs model clients to omit reasoning.
ReasoningEffort string `json:"reasoning_effort"`
// APIKeyEnv is an environment-variable name, not its credential value.
// APIKeyEnv is the resolved name of an optional environment lookup source,
// not its credential value. The built-in client omits Authorization when no
// usable direct or environment credential is available; injected clients may
// resolve this metadata differently.
APIKeyEnv string `json:"api_key_env"`
// ExtraParams contains copied JSON-compatible provider parameters.
ExtraParams map[string]any `json:"extra_params"`
@@ -426,9 +429,10 @@ type ExecutionTargetOverride struct {
// inherited value and disables reasoning for this run. Non-blank values
// are opaque and are not validated against a fixed vocabulary.
ReasoningEffort *string
// APIKeyEnv replaces the profile or backend environment-variable name when
// non-blank. A direct RunRequest.APIKey still takes precedence over
// environment lookup.
// APIKeyEnv replaces the profile or backend optional environment lookup
// source when non-blank. A direct RunRequest.APIKey still takes precedence.
// The built-in client omits Authorization when neither source has a usable
// value; injected clients may resolve this metadata differently.
APIKeyEnv string
// ExtraParams, when non-empty, replaces the complete profile or backend map.
// Values must be JSON-compatible: nil, booleans, finite numbers, strings,
@@ -439,31 +443,43 @@ type ExecutionTargetOverride struct {
// Profile is an in-memory execution profile for library consumers.
//
// It is equivalent to a loaded profile file after validation. Raw API keys do
// not belong in profiles; use APIKeyRequired to require callers to provide a
// RunRequest.APIKey or explicit request ExecutionTargetOverride.APIKeyEnv, or
// use profile YAML api_key_env with file and FS profile sources. Profile has no
// stable JSON representation.
// A standalone Profile is equivalent to a loaded profile file after local
// validation. A derived profile names BaseProfileID and can inherit target
// fields when selected or inspected. Raw API keys do not belong in profiles;
// use APIKeyRequired to require callers to provide a RunRequest.APIKey or
// explicit request ExecutionTargetOverride.APIKeyEnv, or use profile YAML
// api_key_env with file and FS profile sources. Profile has no stable JSON
// representation.
//
// WithProfiles validates and copies Profile values during NewEngine. Zero
// Temperature, MaxTokens, and TopP values and blank ServiceTier and
// ReasoningEffort values leave those provider controls unspecified. A zero
// TimeoutSeconds retains the framework deadline, while an empty ExtraParams map
// inherits backend request defaults. Use ExecutionTargetOverride pointer fields
// to request an explicit numeric zero.
// WithProfiles locally validates and copies Profile values during NewEngine.
// It checks base-reference existence and resolved target completeness when a
// derived profile is selected or inspected. Zero Temperature, MaxTokens, and
// TopP values and blank ServiceTier and ReasoningEffort values leave those
// provider controls unspecified. A zero TimeoutSeconds retains the framework
// deadline, while an empty ExtraParams map inherits backend request defaults.
// Use ExecutionTargetOverride pointer fields to request an explicit numeric
// zero.
type Profile struct {
// ID is the required non-blank profile identifier. WithProfiles trims it.
ID string
// BaseProfileID optionally names one base profile. WithProfiles trims it. A
// non-blank value permits required target fields to be inherited when the
// profile is selected or inspected, which is also when reference existence
// and resolved completeness are checked. A blank value leaves this as a
// standalone profile.
BaseProfileID string
// BackendID optionally selects an engine backend. WithProfiles trims it.
// Backend membership is checked when a request selects the profile; an
// unknown ID makes preparation fail with ErrProfileLoad.
BackendID string
// Endpoint is the model-provider base URL. It is required only when
// BackendID is blank and otherwise overrides the backend endpoint when
// Endpoint is the model-provider base URL. A standalone Profile requires an
// endpoint when BackendID is blank; a derived Profile may inherit either
// field. A non-blank endpoint overrides the backend endpoint when
// non-blank. WithProfiles trims it and requires an absolute HTTP or HTTPS URL
// with a host and no user information, query, or fragment.
Endpoint string
// Model is the required non-blank provider model identifier.
// Model is the provider model identifier. It is required for a standalone
// Profile and may be inherited by a derived Profile.
Model string
// Temperature is from 0 through 2. Zero leaves the provider control
// unspecified.
@@ -481,7 +497,8 @@ type Profile struct {
ReasoningEffort string
// APIKeyRequired clears a backend's inherited API-key environment name and
// requires a non-blank RunRequest.APIKey unless the request explicitly
// supplies ExecutionTargetOverride.APIKeyEnv. It does not store a credential.
// supplies ExecutionTargetOverride.APIKeyEnv. When false, a named
// environment source remains optional. It does not store a credential.
APIKeyRequired bool
// ExtraParams contains provider-specific JSON-compatible values. An empty
// map inherits backend request defaults, when any. WithProfiles validates
@@ -494,13 +511,16 @@ type Profile struct {
// profile.
//
// It contains ordinary profile fields for OpenAI-compatible chat-completions
// endpoints. APIKeyRequired follows Profile.APIKeyRequired. Raw API keys do not
// belong in this config. OpenAICompatibleProfileConfig has no stable JSON
// endpoints. BaseProfileID and APIKeyRequired follow Profile. Raw API keys do
// not belong in this config. OpenAICompatibleProfileConfig has no stable JSON
// representation and is not validated until its resulting Profile is supplied
// through WithProfiles to NewEngine.
type OpenAICompatibleProfileConfig struct {
// ID becomes Profile.ID.
ID string
// BaseProfileID becomes Profile.BaseProfileID. A non-blank value permits the
// resulting Profile to inherit target fields when it is selected or inspected.
BaseProfileID string
// BackendID becomes Profile.BackendID.
BackendID string
// Endpoint becomes Profile.Endpoint.