6 Commits

20 changed files with 1293 additions and 705 deletions

View File

@@ -33,10 +33,13 @@ boundary and constraints that framework work must preserve.
## Release Guidance
Consumers upgrading from `v0.5.0` to `v0.6.0` should read the
[v0.6.0 changelog and migration guide](docs/releases/v0.6.0.md).
Consumers upgrading from `v0.6.0` to `v0.7.0` should read the
[v0.7.0 changelog and migration guide](docs/releases/v0.7.0.md).
Earlier adopters can consult the
[v0.6.0 changelog and migration guide](docs/releases/v0.6.0.md).
Consumers upgrading from `v0.4.0` to `v0.5.0` can consult the
[v0.5.0 changelog and migration guide](docs/releases/v0.5.0.md).
Consumers upgrading from `v0.3.0` to `v0.4.0` should read the

View File

@@ -189,6 +189,26 @@ For programmatic profiles,
[`OpenAICompatibleProfile`](../../profiles.go) converts ordinary
OpenAI-compatible settings into a value accepted by `WithProfiles`.
### Alias A Built-In Profile
Give an application-owned profile ID a built-in base when prompts should select
the application ID while inheriting the built-in target. The child can override
only the setting it owns:
```go
promptkit.WithProfiles(promptkit.Profile{
ID: "weather-light",
BaseProfileID: "deepseek-4-flash",
ReasoningEffort: "high",
})
```
Select `weather-light` in a prompt or `RunRequest.ProfileID`; it remains the
reported selected profile. See the [profile inheritance format
reference](../formats.md#profile-inheritance) and the
[`Profile` GoDoc](../../types.go) for exact lookup, merging, and validation
behavior.
### Use The Rakestrawhome Built-In Profile
Set `RAKESTRAWHOME_INFERENCE_API_KEY` in the application environment, then

View File

@@ -173,9 +173,19 @@ extra_params:
provider_option: enabled
```
A derived profile can use a named base and override only the settings it owns:
```yaml
id: local-summary-fast
base_profile: local-summary
timeout_seconds: 30
reasoning_effort: low
```
| Field | Required | Meaning |
| --- | --- | --- |
| `id` | yes | Profile identifier, trimmed before selection and publication. It must be non-empty after trimming and unique within one source after normalization. |
| `base_profile` | no | One optional parent profile ID. A derived profile may inherit target fields from it. |
| `backend` | unless `endpoint` is present | Backend registry ID. It is trimmed and registry membership is checked when the profile is prepared or inspected. |
| `endpoint` | unless `backend` is present | OpenAI-compatible base URL, including an API version path when required. A nonempty value is trimmed and must be absolute HTTP or HTTPS with a host and without user information, a query, or a fragment. When both connection fields are present, this overrides the backend endpoint without changing backend identity. |
| `model` | yes | Non-empty provider model name. |
@@ -191,6 +201,11 @@ extra_params:
Raw `api_key` is prohibited in profile YAML. Store only an environment
variable name in `api_key_env`.
A standalone profile must provide a model and at least one of `backend` or
`endpoint`. A derived profile may omit those target fields because its selected
base chain can provide them. Local parsing still validates a derived profile's
own ID, supplied endpoint, execution-setting bounds, and `extra_params`.
Promptkit does not infer a backend from a model or endpoint. Endpoint-only
profiles remain supported and have no effective backend ID.
The engine always provides the built-in `openrouter` and `rakestrawhome` IDs.
@@ -266,6 +281,32 @@ profiles. They use `APIKeyRequired` for request-scoped credentials instead of
`api_key_env`. Preparation and exact profile inspection use this same source
precedence.
When a selected definition names `base_profile`, every profile ID in that
chain is looked up through this same precedence order. A higher-precedence
definition therefore shadows a lower-precedence definition of the same base
ID, including a built-in. References are not source-qualified.
### Profile Inheritance
Promptkit resolves one linear base chain of at most 32 profiles, including the
selected profile. It merges settings from the root base to the selected leaf.
The leaf's `id` remains the selected profile identity. Nonblank string fields
(`backend`, `endpoint`, `model`, `service_tier`, `reasoning_effort`, and
`api_key_env`) and nonzero numeric fields replace inherited values. A nonempty
`extra_params` map replaces the complete inherited map rather than merging
keys, and `APIKeyRequired: true` remains true through the chain. Backend and
endpoint are independent: replacing one does not clear the other.
There is no profile-level clearing syntax. Blank strings, zero numbers, false,
and empty maps remain unspecified and inherit from a base. Use existing
presence-aware request overrides where an execution needs an explicit zero or
empty reasoning setting.
An absent directly selected profile reports the ordinary not-found error. Once
the selected profile exists, a missing base, cycle, overlong chain, or
incomplete resolved target is a profile-load failure. Ordinary operations
resolve chains afresh; prepared execution retains the fully resolved target.
## Built-In Profile Catalog
Every built-in profile selects one maintained built-in backend and inherits

View File

@@ -21,7 +21,7 @@ contributor workflow and validation.
| `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |
| `internal/jsonvalue` | Validates and deeply copies bounded JSON-compatible extra-parameter and prepared-schema trees while preserving supported concrete value types and rejecting cycles or excessive depth and work. | [JSON values](../../internal/jsonvalue/jsonvalue.go) |
| `internal/promptdef` | Loads strictly decoded, validated prompt definitions from filesystem and `fs.FS` sources, including version selection and contained file-backed message content. | [Framework formats](../formats.md), [prompt-definition repository](../../internal/promptdef/filesystem_repository.go) |
| `internal/profile` | Loads strictly decoded, validated execution profiles, including backend selection, from filesystem and `fs.FS` sources and composes repositories with error-preserving fallback. | [Framework formats](../formats.md), [profile repositories](../../internal/profile/filesystem_repository.go) |
| `internal/profile` | Loads strictly decoded, locally validated execution profiles from filesystem and `fs.FS` sources, overlays raw sources with error-preserving fallback, and resolves inherited profiles. | [Framework formats](../formats.md), [profile repositories](../../internal/profile/filesystem_repository.go), [internal sources](sources.md#profiles-and-built-ins) |
| `internal/profile/builtin` | Embeds the built-in profile catalog, whose entries select maintained built-in backends. | [Built-in catalog](../formats.md#built-in-profile-catalog), [repository](../../internal/profile/builtin/repository.go) |
| `internal/prompt` | Renders prompt messages from Go templates with artifact, variable, session, and cache-control data. | [Go-template renderer](../../internal/prompt/go_renderer.go) |
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |

View File

@@ -39,17 +39,17 @@ duplicate detection, and source containment:
## Profiles And Built-Ins
`internal/profile` loads and validates execution profiles from an
operating-system filesystem or an `fs.FS`. A file contains exactly one YAML
document and its trimmed YAML `id` is its only selection identity; filenames do
not confer authority. Each point lookup reads discovered files once for their
metadata and reuses the selected file's bytes for strict decoding; unrelated
profiles are not fully decoded. Strict selected decoding recognizes the
optional `backend` field, trims its value, and requires a model plus at least
one non-blank backend or endpoint. File-backed `extra_params` values are
validated and defensively copied through the shared bounded JSON-value owner
before a profile is published. OpenAI-compatible reserved-field policy remains
with the model-client and backend-registry owners.
`internal/profile` loads, locally validates, overlays, and resolves execution
profiles from an operating-system filesystem or an `fs.FS`. A file contains
exactly one YAML document and its trimmed YAML `id` is its only selection
identity; filenames do not confer authority. Each point lookup reads discovered
files once for their metadata and reuses the selected file's bytes for strict
decoding; unrelated profiles are not fully decoded. Strict selected decoding
recognizes `base_profile` and the optional `backend` field, trims their values,
and permits inherited target fields only when a base is named. File-backed
`extra_params` values are validated and defensively copied through the shared
bounded JSON-value owner before a profile is published. OpenAI-compatible
reserved-field policy remains with the model-client and backend-registry owners.
The overlay repository consults the next repository only when the
higher-precedence repository reports that a profile is absent. A reliably
@@ -59,14 +59,25 @@ backend registry membership because the available registry belongs to the
assembled engine; the runner checks membership during preparation and exact
profile inspection.
The root engine assembles profile repositories in precedence order: in-memory
profiles, one ordinary configured source, an application fallback source, then
the embedded built-in catalog. An explicit file or `fs.FS` profile source
replaces `Config.ProfileDir` within the ordinary configured-source category.
The root engine assembles one raw composite catalog in precedence order:
in-memory profiles, one ordinary configured source, an application fallback
source, then the embedded built-in catalog. An explicit file or `fs.FS` profile
source replaces `Config.ProfileDir` within the ordinary configured-source
category. One outer resolving repository wraps that complete raw catalog, so
each base lookup observes the same precedence and shadowing rules.
Exact profile inspection performs one point-in-time lookup through those
profile sources and checks the resolved target without reading prompt, input,
The resolving repository traverses every selected chain afresh, retains no
cache, detects cycles, limits a chain to 32 profiles, merges root-to-leaf into a
new caller-owned value, and validates the final target before publishing it. It
does not check backend registry membership. Exact `base_profile` syntax, merge
rules, and consumer-visible failure behavior belong to the [framework format
reference](../formats.md#profile-inheritance).
Exact profile inspection performs one point-in-time resolved lookup through
those profile sources and checks the final target without reading prompt, input,
or schema sources. It does not retain that lookup for a later execution.
Prepared execution instead freezes the fully resolved target; a later ordinary
operation performs a fresh traversal.
`internal/profile/builtin` embeds the maintained built-in profile catalog.
Every embedded profile selects a maintained built-in backend and inherits that

155
docs/releases/v0.7.0.md Normal file
View File

@@ -0,0 +1,155 @@
# Promptkit v0.7.0
This supplemental changelog and migration guide summarizes the consumer-facing
changes from `v0.6.0` to `v0.7.0`. The annotated `v0.7.0` tag is the
authoritative release record. Exact current contracts belong to the linked
GoDoc and durable documentation.
## Summary
`v0.7.0` expands provider integration and profile composition while making
credential and generation-failure handling more flexible:
- Promptkit now includes the `rakestrawhome` backend and its Gemma profile;
- built-in generation failures expose bounded structured provider details;
- an unavailable optional API-key environment source no longer prevents a
request from reaching an upstream that permits unauthenticated access; and
- profiles can inherit from and selectively refine another profile.
## Compatibility
This release adds public declarations and fields but removes none. Existing
keyed configuration literals and ordinary `errors.Is` handling continue to
work.
Adding `BaseProfileID` to `Profile` and `OpenAICompatibleProfileConfig` changes
their struct shape. Consumers using positional composite literals for either
type must convert them to keyed literals. Existing keyed literals require no
change.
The `rakestrawhome` backend ID is now built in and reserved. A consumer that
previously registered that exact ID with `WithBackend` must remove its manual
registration before upgrading. Other custom backend registrations are
unchanged.
When an optional backend, profile, or request `APIKeyEnv` is unset, empty, or
whitespace-only, the built-in client now omits `Authorization` and sends the
request. Previously this condition could fail before transport. Set
`Profile.APIKeyRequired` when missing credentials must remain a local
preflight error.
Provider non-success responses continue to match `ErrLLMGenerate`. Their
rendered wording is not a compatibility contract; consumers can now use
`errors.As` with `*GenerationError` when structured status information is
needed.
## Upgrade
Update the module dependency with:
```sh
go get gitea.maximumdirect.net/eric/promptkit@v0.7.0
go mod tidy
```
Remove any manual `rakestrawhome` backend registration, convert positional
profile literals to keyed literals, and run the consuming project's ordinary
and race-enabled tests.
## Rakestrawhome Built-In Backend And Profile
Every engine now includes the reserved `rakestrawhome` backend, identified by
`BackendRakestrawHome`. The built-in `rakestrawhome-gemma-4-31b` profile
selects that backend. Consumers can use the maintained endpoint, credential,
capacity, and model defaults without registering either definition themselves.
See the [built-in backend and profile catalogs](../formats.md#built-in-backends)
and the [consumer adoption example](../consumers/pkg-promptkit.md#use-the-rakestrawhome-built-in-profile)
for the current contracts.
## Structured Generation Errors
Non-2xx responses from the built-in OpenAI-compatible client now return an
immutable `*GenerationError`. Consumers can inspect the HTTP status and any
safely extracted provider code, type, or message while retaining the ordinary
generation-error category:
```go
var generationErr *promptkit.GenerationError
if errors.As(err, &generationErr) {
status := generationErr.StatusCode()
_ = status
}
```
Provider fields are bounded and normalized but remain untrusted and may
contain sensitive request or schema details. Default and Go-syntax formatting
omit those fields. Applications must apply their own disclosure policy before
logging or presenting accessor values.
See the [`GenerationError` GoDoc](../../generation_error.go), the
[consumer error-handling guide](../consumers/pkg-promptkit.md#handle-errors),
and the [OpenAI-compatible response contract](../integrations/openai-compatible-chat.md#response-handling).
## Optional Credential Sources
`APIKeyEnv` names an optional environment lookup source unless the selected
profile explicitly sets `APIKeyRequired`. When neither a direct request key nor
a usable environment value exists, the built-in client omits the bearer header
and handles the upstream response normally. This supports local and other
OpenAI-compatible providers that permit unauthenticated requests without
hiding an authentication error returned by a provider that requires one.
The [credential format reference](../formats.md#credentials), the
[`Backend` GoDoc](../../backends.go), the
[`ExecutionTargetOverride` GoDoc](../../types.go), and the
[authentication integration contract](../integrations/openai-compatible-chat.md#authentication)
define the current precedence and availability rules.
## Profile Inheritance
YAML profiles can name one parent with `base_profile`; in-memory profiles use
`Profile.BaseProfileID`, and `OpenAICompatibleProfileConfig` forwards the same
field. A profile can act as an application-owned alias of a built-in or refine
selected inherited settings:
```go
promptkit.WithProfiles(promptkit.Profile{
ID: "weather-light",
BaseProfileID: "deepseek-4-flash",
ReasoningEffort: "high",
})
```
Base lookup observes the existing source precedence. Chains are linear,
cycle-safe, and resolved afresh for ordinary operations. Prepared execution
freezes the fully resolved target. The selected leaf ID remains public while
effective execution settings reflect the resolved chain.
See the [profile inheritance format reference](../formats.md#profile-inheritance),
the [consumer alias example](../consumers/pkg-promptkit.md#alias-a-built-in-profile),
and the [`Profile` GoDoc](../../types.go) for exact merge and validation
behavior.
## Public API Changes
The release adds:
- `BackendRakestrawHome`;
- `GenerationError`, including `StatusCode`, `ProviderCode`, `ProviderType`,
`ProviderMessage`, `Error`, `GoString`, and `Unwrap`;
- `Profile.BaseProfileID`; and
- `OpenAICompatibleProfileConfig.BaseProfileID`.
No public declaration was removed.
## Consumer Action
- Remove a manual backend registration whose ID is exactly `rakestrawhome`.
- Convert positional `Profile` or `OpenAICompatibleProfileConfig` literals to
keyed literals.
- Set `Profile.APIKeyRequired` where a missing credential must fail locally
instead of reaching the provider unauthenticated.
- Treat `GenerationError` provider fields as untrusted and potentially
sensitive when adopting the new accessors.
- Run consumer ordinary and race-enabled tests after updating the module.

View File

@@ -40,12 +40,11 @@ consumers.
### Public bounded output repair
After the codebase-audit remediations are complete, Promptkit should make its
bounded output-repair capability available through the public engine. A
consumer should be able to request a limited number of corrective generation
attempts when JSON or JSON Schema output fails content validation, without
having to reproduce Promptkit's generation, validation, capacity, and result-
accounting orchestration.
Promptkit should make its bounded output-repair capability available through
the public engine. A consumer should be able to request a limited number of
corrective generation attempts when JSON or JSON Schema output fails content
validation, without having to reproduce Promptkit's generation, validation,
capacity, and result-accounting orchestration.
- Repair is validation recovery, not a general provider retry, failover, or
backoff policy. Transport failures, cancellation, and operational schema or
@@ -62,10 +61,6 @@ accounting orchestration.
- Ordinary and prepared execution should expose coherent behavior, including
cancellation, frozen prepared state, error identity, and capacity lifetime.
Select this work only after the accepted audit findings affecting shared
execution invariants, validation, orchestration, transport, and repair
internals have been remediated.
## Entry Format
Use a short heading followed by a concise summary. Add focused bullets when

View File

@@ -1,332 +0,0 @@
# Optional API-Key Environment Implementation Plan
## Purpose
Change Promptkit's credential handling so an effective `APIKeyEnv` names an
optional credential source rather than implicitly requiring that environment
variable to be populated. When neither a direct key nor a populated optional
environment variable is available, the built-in OpenAI-compatible client must
omit the `Authorization` header and let the upstream provider accept or reject
the unauthenticated request.
This document owns implementation sequencing for that change. Follow the
architecture, documentation, and testing policies under [`docs/policy/`](../policy/)
and the task-specific reading guide in [`docs/development.md`](../development.md)
throughout the work.
## Target Outcome
For ordinary and prepared execution:
1. a nonblank direct `RunRequest.APIKey` remains the highest-precedence
credential and produces `Authorization: Bearer <key>`;
2. otherwise, a nonblank effective `APIKeyEnv` is read when generation begins;
3. a nonblank trimmed environment value produces the bearer header;
4. an absent, empty, or whitespace-only optional environment value produces no
`Authorization` header and does not prevent preparation or execution; and
5. the provider response follows the normal success or error path, including
public `GenerationError` conversion for a non-2xx response.
An explicitly required credential retains local validation. `APIKeyRequired`
continues to require either a nonblank direct key or a populated environment
variable explicitly selected by the request. Missing required credentials
fail before provider transport.
## Fixed Decisions
- `APIKeyEnv` is a lookup location, not an assertion that authentication is
required. This applies whether the name comes from a built-in backend, a
consumer backend, a file profile, or a request override.
- An environment value is usable only after `strings.TrimSpace`; unset, empty,
and whitespace-only values are equivalent and cause header omission when
optional.
- Direct keys retain precedence. When a direct key is nonblank, Promptkit does
not need the environment value and must not let a missing environment value
block the request.
- `APIKeyRequired` remains the only existing explicit local requirement flag.
Do not add a backend field, YAML field, request field, or new public API.
- With `APIKeyRequired` true:
- a nonblank direct key satisfies the requirement;
- an explicitly selected nonblank `APIKeyEnv` satisfies it only when that
variable currently contains a nonblank value;
- no selected environment name retains the existing required-key failure;
and
- a selected but unset environment retains `ErrAPIKeyEnvMissing` and
`ErrInvalidRequest` at the public boundary.
- `ErrAPIKeyEnvMissing` remains exported for compatibility but is narrowed to
a missing environment credential in an explicitly required flow. Optional
missing environment values do not return it.
- Optional credential availability is not frozen during preparation. The
built-in client reads the selected environment variable for each generation,
including repair generation. Prepared execution therefore uses the value
visible when it runs.
- Required prepared execution continues to recheck credential availability
before admission. Optional prepared execution proceeds even if the variable
becomes unset after preparation.
- The built-in transport sets no `Authorization` header at all when no usable
credential exists. It must not send `Bearer ` with an empty value.
- Injected `LLMClient` behavior is not redefined. Promptkit stops rejecting an
optional missing environment before the injected client is called and
continues to pass the effective target through the public adapter without
resolving or exposing a secret on the client's behalf.
- Upstream authentication policy remains upstream. Promptkit adds no provider-
specific authentication rules, retry behavior, status mapping, or special
handling beyond the existing structured non-2xx error path.
## Execution Rules
- Complete the stages in numerical order. Each stage is scoped for one coding-
agent prompt and must finish its focused tests before the next stage begins.
- Treat all stages as one behavior change. Intermediate stages intentionally
leave the use-case and transport policies temporarily different; do not tag,
release, or claim the change is complete until Stage 3 passes.
- At the start of each stage, inspect the working tree and preserve all
unrelated changes. In particular, merge rather than overwrite any existing
edits in files touched by this plan.
- Use controlled transports, local servers, and `t.Setenv`; never contact a
live provider or depend on the developer machine's credentials.
- Update existing tests whose asserted policy is changing instead of retaining
contradictory tests or adding duplicate coverage under new names.
- Do not add dependencies, commit, tag, push, or edit release documentation
unless separately instructed.
- Current-state prose documentation changes only in Stage 3, after the runtime
behavior exists. GoDoc changes alongside the public contract in that stage.
## Stage 1: Make Use-Case Credential Validation Requirement-Aware
### Objective
Stop preparation and execution orchestration from treating every named
environment variable as required, while preserving the explicit
`APIKeyRequired` and prepared-execution contracts.
### Implementation
1. Update `validateAPIKey` in `internal/usecase/runner.go` with this policy:
- return success immediately for a nonblank direct key;
- return success when `apiKeyRequired` is false, regardless of whether
`apiKeyEnv` is blank, populated, or missing;
- when `apiKeyRequired` is true and the trimmed environment name is blank,
return the existing `ErrAPIKeyRequired`;
- when `apiKeyRequired` is true and the selected environment variable is
unset, empty, or whitespace-only, return the existing wrapped
`ErrAPIKeyEnvMissing` diagnostic naming the variable; and
- otherwise return success.
2. Keep the calls from preparation and `RunPrepared` in place. Do not move
environment lookup into backend/profile resolution, admission, the domain
model, or the root facade.
3. Do not clear `APIKeyEnv` from effective or prepared metadata merely because
its current value is absent. The name remains the configured lookup source;
the secret value remains excluded.
4. Preserve credential-source precedence and `APIKeyRequired` merge behavior.
A request environment override remains capable of satisfying an in-memory
profile's explicit requirement when its value is populated.
### Tests
Update the narrow `internal/usecase` behavior owners rather than adding tests
for the private helper itself:
1. Replace `TestRunnerRunAPIKeyEnvMissingEnvironmentValueFailsClearly` with a
test proving an unset optional profile `APIKeyEnv` reaches the injected
model client and completes successfully. Explicitly set the named variable
to an empty value so the test is independent of the process environment.
2. Preserve the existing populated-environment and direct-key-precedence tests.
3. Preserve `TestRunnerPrepareAPIKeyRequiredFailsWithoutDirectKey` and the
existing direct-key success coverage.
4. Revise the prepared credential test matrix to prove both distinct cases:
- an optional environment value that disappears after
`PrepareExecution` does not prevent admission and generation; and
- an `APIKeyRequired` profile using an explicit request `APIKeyEnv` override
still fails with `ErrInvalidRequest` and `ErrAPIKeyEnvMissing` if that
value disappears before `RunPrepared`, before admission or generation.
5. Keep prepared handles single-use in both outcomes and avoid duplicating
unrelated preparation, validation, or admission assertions.
### Verification
```sh
go test ./internal/usecase -run 'APIKey|Credential|Prepared'
go test ./internal/usecase
go test ./...
```
Stage 1 is complete when optional missing environment values no longer block
the use-case layer and explicit requirements retain their prior identities and
timing.
**Status:** Complete.
## Stage 2: Omit Authentication in the Built-In Client When Optional Keys Are Missing
### Objective
Make the OpenAI-compatible transport implement the optional lookup contract at
the boundary that owns the outbound `Authorization` header.
### Implementation
1. Refactor the authentication branch in
`internal/llm/openai_compatible_client.go` so it resolves one credential in
this order:
- trimmed `req.Target.APIKey`; then
- the trimmed value of the environment variable named by the trimmed
`req.Target.APIKeyEnv`.
2. Set `Authorization` to `Bearer <credential>` only when the resolved value is
nonblank. If the direct key is blank and the optional environment lookup is
absent, empty, or whitespace-only, leave the header unset and continue to
`http.Client.Do`.
3. Retain defensive enforcement for direct internal callers of the transport:
- if `req.Target.APIKeyRequired` is true and no usable source exists, return
`ErrInvalidRequest` before transport;
- use the existing missing-environment diagnostic when a nonblank
environment name was selected; and
- use a concise required-key diagnostic when no environment name exists.
Do not add an internal or public error type for this branch.
4. Preserve request construction, endpoint validation, timeout behavior,
response-body ownership, and direct-key redaction. Never serialize
`APIKeyEnv`, `APIKeyRequired`, or a credential into the JSON request body.
5. Allow an unauthenticated upstream non-2xx response to flow through the
existing `ProviderHTTPError` and public `GenerationError` machinery without
adding a credential-specific translation.
### Tests
Update `TestOpenAICompatibleClientAuthentication` in
`internal/llm/openai_compatible_client_test.go` as the transport contract owner.
Its table should cover only the meaningful credential states:
- a direct key takes precedence over a populated environment value;
- a populated environment value supplies the bearer header when no direct key
is present;
- no configured source sends the request without `Authorization`;
- a named but unset optional environment sends the request without
`Authorization`;
- a named but unset required environment fails with `ErrInvalidRequest` before
transport; and
- a required target with no source also fails before transport.
Use exact header assertions because presence or absence of `Authorization` is
the wire contract. Do not repeat endpoint, body, timeout, or response-error
matrices in this test.
### Verification
```sh
go test ./internal/llm -run TestOpenAICompatibleClientAuthentication
go test ./internal/llm
go test ./...
```
Stage 2 is complete when the built-in transport sends unauthenticated requests
for optional missing sources, still enforces explicit requirements, and all
existing provider-response behavior remains green.
**Status:** Complete.
## Stage 3: Align the Public Contract, Durable Documentation, and Full Validation
### Objective
Prove the assembled consumer workflow, update canonical contract owners, and
complete repository-wide validation.
### Public Contract And Tests
1. Update the GoDoc for `ErrAPIKeyEnvMissing` in `engine.go` so it applies only
when an explicitly required credential names an unset or empty environment
variable. Retain the exported value and its `ErrInvalidRequest` identity.
2. Update credential GoDoc where the exact public semantics are exposed:
- `Backend.APIKeyEnv` in `backends.go`;
- `ExecutionTarget.APIKeyEnv` and
`ExecutionTargetOverride.APIKeyEnv` in `types.go`; and
- any nearby `APIKeyRequired` wording that would otherwise imply every
named environment source is mandatory.
State that optional missing values cause the built-in client to omit
`Authorization`; do not promise that injected clients resolve environment
variables identically.
3. Replace `TestMissingCredentialsFailClearlyWhenProfileRequiresAuth` in
`engine_test.go`, whose old assertion is intentionally obsolete, with one
focused external-package contract test. Assemble a real engine with a
controlled HTTP client or local server, select a backend or file profile
whose `APIKeyEnv` is explicitly empty in the process environment, and have
the controlled upstream return a structured authentication failure. Assert:
- the request reaches upstream;
- `Authorization` is absent;
- the result is nil;
- the error does not match `ErrInvalidRequest` or
`ErrAPIKeyEnvMissing`; and
- the error matches `ErrLLMGenerate` and is discoverable as a
`*GenerationError` with the upstream status and representative provider
detail.
4. Keep this root test representative. The transport authentication table owns
the full credential matrix, and the existing generation-error tests own
envelope parsing, bounds, and formatting.
### Documentation
Update each canonical owner only for its topic:
1. In `docs/formats.md`, define profile, backend, and request `api_key_env`
values as optional lookup sources. In the credentials section, distinguish
them from `APIKeyRequired`, document header omission for a missing optional
value, retain direct-key precedence, and state that required availability is
validated during preparation and rechecked for prepared execution.
2. In `docs/integrations/openai-compatible-chat.md`, document the outbound
authentication wire behavior: a bearer header is sent only for a usable
direct or environment credential; otherwise the header is omitted and the
provider response is handled normally.
3. In `docs/internal/llm.md`, update the internal flow and failure categories to
distinguish optional omission from explicit required-key rejection. Preserve
any unrelated edits already present in this file.
4. In `docs/consumers/pkg-promptkit.md`, clarify near profile inspection or
credential guidance that a reported `APIKeyEnv` is a configured optional
source, while `APIKeyRequired` is the explicit local requirement. Link to
the exact public GoDoc or format reference rather than reproducing the full
precedence contract.
5. Do not change architecture policy, backend IDs/default names, release notes,
the README, or examples unless implementation uncovers a concrete
inaccurate current-state statement in one of those owners.
### Final Validation
Run the complete maintainer workflow from
[`docs/development.md#maintainer-validation`](../development.md#maintainer-validation):
```sh
go test ./...
go test -race ./...
go vet ./...
go build ./...
go run ./examples/go-library/prepare
go run ./examples/go-library/run
```
Also run the documented tracked-Go formatting check, local Markdown-link
validation, repository-hygiene checks, ignored-file check, credential scan, and
`git diff --check`. Review the complete diff and confirm that:
- optional absent, empty, and whitespace-only environment values omit
authentication and reach transport;
- direct and populated environment credentials still produce the correct
bearer header;
- explicit required flows still fail locally with the intended identities;
- ordinary, prepared, built-in, injected-client, and repair paths follow their
stated ownership boundaries;
- upstream non-2xx responses remain structured generation errors rather than
local credential errors;
- no credential value is serialized, retained in public metadata, or exposed
by formatting; and
- durable documentation describes only the now-implemented behavior with one
canonical owner per exact contract.
Stage 3 is complete when the public workflow, documentation, and full
maintainer validation all match the target outcome.
**Status:** Complete.
## Open Questions
None. Optional environment lookup, explicit requirement behavior, precedence,
prepared-execution timing, outbound header semantics, error compatibility,
documentation ownership, and test boundaries are fixed by this plan.

View File

@@ -1,260 +0,0 @@
# Structured Generation Errors
## Purpose
Promptkit should give downstream applications actionable, machine-readable
details when the built-in OpenAI-compatible client receives a non-success HTTP
response. Today the client reports only the status code and discards the
provider response body. This makes ordinary configuration failures, such as an
unsupported strict JSON Schema keyword, unnecessarily difficult to diagnose.
This feature supplies bounded facts about the provider response. It does not
make retry, presentation, or logging decisions for consumers.
## Target End State
Every non-2xx response received by Promptkit's built-in OpenAI-compatible
client becomes a public typed generation error. A consumer can use
`errors.As` to obtain the HTTP status and any safely extracted provider fields,
and `errors.Is` continues to match `ErrLLMGenerate`.
The typed contract is available from both `Run` and `RunPrepared`. It is not
produced during preparation, which performs no model request. Successful
responses, transport failures before a response is received, cancellation,
capacity failures, validation failures, and nil responses from injected model
clients retain their existing categories and behavior.
An unusable response body never hides the known HTTP status. Empty, malformed,
unrecognized, unreadable, or oversized bodies therefore produce the same typed
error with status-only detail rather than falling back to an unstructured
error or becoming a malformed-success response.
## Public Contract
The root package exposes an immutable `GenerationError` type with unexported
state and these read-only accessors:
- `StatusCode() int` returns the received HTTP status code;
- `ProviderCode() string` returns a normalized provider code, when present;
- `ProviderType() string` returns a normalized provider error type, when
present; and
- `ProviderMessage() string` returns the bounded normalized diagnostic message,
when present.
The engine returns a `*GenerationError`, so the idiomatic inspection form is:
```go
var generationErr *promptkit.GenerationError
if errors.As(err, &generationErr) {
status := generationErr.StatusCode()
message := generationErr.ProviderMessage()
_, _ = status, message
}
```
There is no public constructor or mutation API. The type implements `error`,
unwraps to `ErrLLMGenerate`, and provides safe ordinary and Go-syntax
formatting. `Error()` and `GoString()` include the HTTP status but no provider-
controlled code, type, or message. Consumers must use the accessors
deliberately when they want provider details and must not classify failures by
matching error text.
The zero value and a nil `*GenerationError` receiver are safe: accessors return
zero or empty values, formatting returns a generic redacted generation-failure
description, and unwrapping still identifies `ErrLLMGenerate`. Engine-produced
values always have the non-2xx status received from the provider. The type has
no stable JSON representation.
All provider-derived strings remain untrusted even after normalization. GoDoc
must warn consumers that provider fields can contain sensitive request or
schema fragments and must not be logged, displayed, or returned to another
caller without an application-appropriate disclosure policy.
## Recognized Provider Envelope
Promptkit recognizes only the conventional OpenAI-compatible top-level error
object:
```json
{
"error": {
"message": "diagnostic text",
"type": "invalid_request_error",
"code": "unsupported_parameter"
}
}
```
The envelope must be one JSON document followed only by JSON whitespace. The
top-level `error` value must be an object. Unknown top-level and error-object
fields are ignored. The optional supported fields are interpreted
independently:
- `message` and `type` must be JSON strings;
- `code` may be a JSON string or number and is exposed as normalized text;
numeric codes retain their validated JSON number text without floating-point
coercion; and
- `null`, booleans, arrays, objects, or otherwise invalid values are treated as
absent for that field.
An invalid optional field does not discard other valid supported fields. An
absent `error` object, malformed or multiply framed JSON, or an object with no
usable supported fields simply leaves all provider accessors empty while
preserving the typed status error.
Promptkit does not expose `param`, metadata objects, nested causes, headers, or
provider-specific extensions in this feature.
## Bounded Reading And Normalization
Non-success bodies have a separate fixed limit of 64 KiB (65,536 bytes). This
is intentionally much smaller than the successful completion-body limit while
remaining large enough for useful schema diagnostics.
- A declared `Content-Length` above the limit is rejected without reading the
body for detail extraction.
- Otherwise Promptkit reads at most one byte beyond the limit so streamed,
chunked, and underreported bodies are bounded.
- A body over the limit contributes no provider fields; Promptkit does not
parse or retain a prefix as though it were a complete envelope.
- Read failures likewise discard provider fields while preserving the status.
- The response body is closed on every outcome and is not drained beyond the
bounded read.
Extracted strings are converted to valid UTF-8, trimmed, and made single-line:
invalid UTF-8 is replaced, and runs of Unicode whitespace, control characters,
and formatting controls are replaced with one ASCII space. Empty normalized
values are treated as absent.
Normalized provider codes and types are retained only when they contain at
most 256 Unicode code points. Longer values are omitted rather than truncated
so consumers never classify on a fabricated partial identifier. A provider
message is limited to 4,096 Unicode code points; a longer normalized message is
truncated at a code-point boundary with a visible ellipsis inside that limit.
The raw response body and pre-normalized strings are never exposed or retained
in the public error.
## Error Propagation And Compatibility
- Every built-in-client non-2xx response matches `ErrLLMGenerate` and supports
`errors.As` to `*GenerationError`, including status-only cases.
- The internal model client retains its non-success-status identity for its
own package tests. The use-case layer remains provider-neutral and continues
to add only its generation category.
- The root error boundary converts only the built-in transport's structured
status error. It does not parse arbitrary error text, inspect consumer error
fields, or fabricate HTTP details for an injected `LLMClient`.
- Errors returned by injected clients remain in the chain exactly as today.
If an injected client deliberately returns an existing `*GenerationError`,
its identity may pass through ordinary wrapping, but Promptkit does not
construct or enrich one on that client's behalf.
- Existing cancellation and deadline identities, capacity errors, validation
behavior, repair behavior, and successful response decoding remain
unchanged.
- This is an additive public API. Existing consumers that use
`errors.Is(err, ErrLLMGenerate)` continue to work; consumers should not rely
on the previous rendered wording of non-success errors.
## Architecture And Ownership
The provider-envelope parser and bounded body reader belong in `internal/llm`,
which owns the OpenAI-compatible transport. The internal transport error owns
only normalized status facts and continues to match the package's existing
non-success-status sentinel.
The use-case package does not gain HTTP DTOs, status policy, or a provider-
specific branch. Its existing wrapping carries the internal error to the root
facade. The root error mapper recognizes the internal structured status error
and constructs the public `GenerationError` without exposing an internal type
or raw cause through public fields. No transport error is added to
`internal/domain`.
The public type and its exact Go semantics are owned by its declaration and
GoDoc. The
[OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md)
owns recognized wire shapes, limits, and observable response behavior. The
[internal model-client document](../internal/llm.md) owns implementation flow,
internal failure categories, and test ownership. Architecture policy does not
need a new package or dependency rule for this feature.
## Documentation End State
Canonical documentation at the target state has these responsibilities:
- the `GenerationError` declaration and GoDoc define the exact public methods,
formatting, unwrapping, zero-value behavior, and trust boundary;
- `Engine.Run` and `Engine.RunPrepared` GoDoc identify the typed error without
duplicating its accessor contract;
- the consumer guide includes one short `errors.As` example and links to the
public declaration;
- the integration document replaces its status-only description with the
bounded envelope contract; and
- the internal model-client document describes parsing, conversion ownership,
and narrow test owners.
The architecture policy, framework format reference, and built-in backend
catalog do not duplicate this API or wire contract.
## Verification Expectations
Verification protects each behavior at its narrowest stable owner:
- internal model-client tests cover recognized string and numeric codes,
independent optional-field handling, unknown fields, empty and malformed
envelopes, single-document framing, read failures, declared and streamed
size boundaries, body closure, normalization, field limits, and absence of
raw provider content from rendered errors;
- root error-boundary tests cover conversion to the immutable public type,
every accessor, `errors.Is`, `errors.As`, and safe `%v`, `%+v`, and `%#v`
formatting;
- one representative ordinary run and one prepared run prove that the built-in
transport contract crosses the assembled engine boundary, without repeating
the complete parser matrix;
- existing injected-client tests continue to prove preservation of consumer
error identity without fabricated provider details; and
- all tests use controlled transports or local servers and never contact a
live or paid provider.
Security limits and their exact boundaries are contractual enough to warrant
literal boundary tests. Higher-level tests should remain representative and
must not duplicate the internal transport matrix.
## Acceptance Criteria
- A consumer can distinguish an HTTP 400 from other generation failures and
deliberately obtain a bounded provider explanation when one is available.
- The same typed error remains available through ordinary and prepared
execution and still satisfies `errors.Is(err, ErrLLMGenerate)`.
- Default and Go-syntax error formatting cannot disclose any provider-derived
string or raw response content.
- Empty, malformed, unreadable, unrecognized, and oversized bodies preserve a
typed status-only error.
- No read, retained field, or formatted representation can exceed its stated
bound, and the body is closed on every outcome.
- Existing success, cancellation, capacity, validation, repair, and injected-
client contracts remain unchanged.
- Current-state documentation changes only when the implementation exists and
follows the repository's canonical ownership policy.
## Non-Goals
This feature does not add:
- retryability classification, retry loops, backoff, failover, or routing;
- parsing of success bodies as errors or changes to successful-response limits;
- provider-specific envelope variants beyond the conventional top-level
`error` object;
- response headers such as `Retry-After`, raw bodies, request data, endpoints,
credentials, schema documents, generated content, or provider metadata;
- logging, telemetry, redaction policy for downstream applications, HTTP status
mapping for consumer servers, or user-facing presentation;
- translation or enrichment of arbitrary injected-client errors; or
- a new public package, public constructor, mutable error value, or transport
type in the domain model.
## Open Questions
None. The public type direction, accessor surface, formatting and error-chain
behavior, envelope scope, normalization, safety limits, fallback behavior,
layer ownership, compatibility boundaries, documentation ownership, and test
boundaries are fixed by this roadmap.

View File

@@ -304,10 +304,12 @@ func WithFallbackProfileFS(fsys fs.FS, root string) Option {
// WithProfiles configures in-memory profiles that take precedence over
// ordinary configured, application fallback, and built-in profiles.
//
// NewEngine validates and copies every profile. IDs must be unique within one
// call. An invalid profile, duplicate ID, or unsupported ExtraParams value
// makes construction fail with ErrInvalidConfig. Repeating WithProfiles
// replaces the complete earlier in-memory set rather than merging it.
// NewEngine locally validates and copies every profile. IDs must be unique
// within one call. An invalid local definition, duplicate ID, or unsupported
// ExtraParams value makes construction fail with ErrInvalidConfig. A derived
// profile's base reference and resolved target completeness are checked when it
// is selected or inspected. Repeating WithProfiles replaces the complete
// earlier in-memory set rather than merging it.
func WithProfiles(profiles ...Profile) Option {
return optionFunc(func(options *engineOptions) error {
repo, err := newMemoryProfileRepository(profiles)
@@ -458,7 +460,7 @@ func newProfileRepository(profileDir string, options engineOptions) profile.Repo
repository = profile.NewOverlayRepository(options.memoryProfiles, repository)
}
return repository
return profile.NewResolvingRepository(repository)
}
func fileSource(name string) (fs.FS, string, error) {

View File

@@ -2018,6 +2018,25 @@ func TestWithProfilesRejectsDuplicateIDs(t *testing.T) {
}
}
func TestWithProfilesAcceptsDerivedDefinitionWithoutTargetFields(t *testing.T) {
_, err := promptkit.NewEngine(promptkit.Config{PromptDir: frameworkPromptDir},
promptkit.WithProfiles(promptkit.Profile{
ID: " derived-profile ",
BaseProfileID: " base-profile ",
}),
)
if err != nil {
t.Fatalf("derived profile should be accepted during construction: %v", err)
}
_, err = promptkit.NewEngine(promptkit.Config{PromptDir: frameworkPromptDir},
promptkit.WithProfiles(promptkit.Profile{ID: "standalone-profile"}),
)
if !errors.Is(err, promptkit.ErrInvalidConfig) {
t.Fatalf("standalone incomplete profile error = %v, want ErrInvalidConfig", err)
}
}
func TestWithProfilesRejectsInvalidExecutionSettings(t *testing.T) {
type testCase struct {
name string
@@ -2170,6 +2189,7 @@ func TestOpenAICompatibleProfileMapsEveryField(t *testing.T) {
extraParams := map[string]any{"provider_option": "distinct-extra-params"}
got := promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
ID: "distinct-id",
BaseProfileID: "distinct-base",
BackendID: "distinct-backend",
Endpoint: "https://distinct.example/v1",
Model: "distinct-model",
@@ -2184,6 +2204,7 @@ func TestOpenAICompatibleProfileMapsEveryField(t *testing.T) {
})
want := promptkit.Profile{
ID: "distinct-id",
BaseProfileID: "distinct-base",
BackendID: "distinct-backend",
Endpoint: "https://distinct.example/v1",
Model: "distinct-model",

View File

@@ -192,6 +192,7 @@ type BackendCapacityPolicy struct {
// ExecutionProfile describes how and where to execute a model.
type ExecutionProfile struct {
ID string `yaml:"id"`
BaseProfileID string `yaml:"base_profile"`
BackendID string `yaml:"backend"`
Endpoint string `yaml:"endpoint"`
Model string `yaml:"model"`

View File

@@ -0,0 +1,47 @@
package profile
import (
"errors"
"strings"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
)
// NormalizeAndValidateDefinition normalizes and validates one source-local
// profile definition without resolving a base profile.
func NormalizeAndValidateDefinition(profile *domain.ExecutionProfile) error {
if profile == nil {
return errors.New("profile is required")
}
profile.ID = strings.TrimSpace(profile.ID)
profile.BaseProfileID = strings.TrimSpace(profile.BaseProfileID)
profile.BackendID = strings.TrimSpace(profile.BackendID)
profile.Endpoint = strings.TrimSpace(profile.Endpoint)
if profile.ID == "" {
return errors.New("id is required")
}
if profile.Endpoint != "" {
endpoint, err := domain.NormalizeOpenAICompatibleBaseEndpoint(profile.Endpoint)
if err != nil {
return err
}
profile.Endpoint = endpoint
}
if profile.BaseProfileID == "" {
if profile.BackendID == "" && profile.Endpoint == "" {
return errors.New("backend or endpoint is required")
}
if strings.TrimSpace(profile.Model) == "" {
return errors.New("model is required")
}
}
return domain.ValidateExecutionTargetSettings(domain.ExecutionTarget{
Temperature: profile.Temperature,
MaxTokens: profile.MaxTokens,
TopP: profile.TopP,
TimeoutSeconds: profile.TimeoutSeconds,
})
}

View File

@@ -127,12 +127,11 @@ func loadProfile(ctx context.Context, fsys fs.FS, root string, id string) (*doma
if prof.ID != id {
continue
}
prof.BackendID = strings.TrimSpace(prof.BackendID)
prof.ExtraParams, err = jsonvalue.CopyMap(prof.ExtraParams)
if err != nil {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, relPath, err)
}
if err := normalizeAndValidateProfile(prof); err != nil {
if err := NormalizeAndValidateDefinition(prof); err != nil {
if errors.Is(err, ErrRawAPIKeyNotAllowed) {
return nil, fmt.Errorf("%w: %s", err, relPath)
}
@@ -255,30 +254,3 @@ func requireYAMLStreamEnd(decoder *yaml.Decoder) error {
}
return errors.New("profile file must contain exactly one YAML document")
}
func normalizeAndValidateProfile(p *domain.ExecutionProfile) error {
if strings.TrimSpace(p.ID) == "" {
return errors.New("id is required")
}
p.Endpoint = strings.TrimSpace(p.Endpoint)
if strings.TrimSpace(p.BackendID) == "" && p.Endpoint == "" {
return errors.New("backend or endpoint is required")
}
if p.Endpoint != "" {
endpoint, err := domain.NormalizeOpenAICompatibleBaseEndpoint(p.Endpoint)
if err != nil {
return err
}
p.Endpoint = endpoint
}
if strings.TrimSpace(p.Model) == "" {
return errors.New("model is required")
}
return domain.ValidateExecutionTargetSettings(domain.ExecutionTarget{
Temperature: p.Temperature,
MaxTokens: p.MaxTokens,
TopP: p.TopP,
TimeoutSeconds: p.TimeoutSeconds,
})
}

View File

@@ -858,6 +858,107 @@ top_p: .inf
})
}
func TestProfileRepositoriesValidateDerivedDefinitions(t *testing.T) {
tests := []struct {
name string
files map[string]string
wantError error
wantBaseID string
wantProfile bool
}{
{
name: "alias is locally valid and normalizes base id",
files: map[string]string{"alias.yaml": `
id: selected-profile
base_profile: " base-profile "
`},
wantBaseID: "base-profile",
wantProfile: true,
},
{
name: "derived endpoint remains valid",
files: map[string]string{"invalid.yaml": `
id: selected-profile
base_profile: base-profile
endpoint: /v1
`},
wantError: ErrInvalidProfile,
},
{
name: "derived settings remain valid",
files: map[string]string{"invalid.yaml": `
id: selected-profile
base_profile: base-profile
top_p: 1.1
`},
wantError: ErrInvalidProfile,
},
{
name: "derived extra params remain valid",
files: map[string]string{"invalid.yaml": `
id: selected-profile
base_profile: base-profile
extra_params:
timestamp: 2026-08-11T12:34:56Z
`},
wantError: ErrInvalidProfile,
},
{
name: "derived raw key remains prohibited",
files: map[string]string{"invalid.yaml": `
id: selected-profile
base_profile: base-profile
api_key: secret
`},
wantError: ErrRawAPIKeyNotAllowed,
},
{
name: "derived duplicate id remains invalid",
files: map[string]string{
"first.yaml": "id: selected-profile\nbase_profile: first-base\n",
"second.yaml": "id: selected-profile\nbase_profile: second-base\n",
},
wantError: ErrInvalidProfile,
},
{
name: "derived extra document remains invalid",
files: map[string]string{"invalid.yaml": `
id: selected-profile
base_profile: base-profile
---
id: other
`},
wantError: ErrInvalidYAML,
},
{
name: "standalone profile remains complete",
files: map[string]string{"invalid.yaml": "id: selected-profile\n"},
wantError: ErrInvalidProfile,
},
}
for _, source := range profileRepositorySources() {
for _, tc := range tests {
t.Run(source.name+"/"+tc.name, func(t *testing.T) {
repo := source.newRepository(t, tc.files)
got, err := repo.GetProfile(context.Background(), "selected-profile")
if tc.wantError != nil {
if !errors.Is(err, tc.wantError) {
t.Fatalf("error = %v, want %v", err, tc.wantError)
}
return
}
if err != nil || !tc.wantProfile {
t.Fatalf("profile = %+v, error = %v, want valid derived definition", got, err)
}
if got.BaseProfileID != tc.wantBaseID {
t.Fatalf("BaseProfileID = %q, want %q", got.BaseProfileID, tc.wantBaseID)
}
})
}
}
}
func TestOverlayRepository(t *testing.T) {
ctx := context.Background()
primaryProfile := &domain.ExecutionProfile{ID: "shared", Endpoint: "http://primary", Model: "primary"}

View File

@@ -0,0 +1,160 @@
package profile
import (
"context"
"errors"
"fmt"
"strings"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
"gitea.maximumdirect.net/eric/promptkit/internal/jsonvalue"
)
const maximumProfileChainLength = 32
type resolvingRepository struct {
source Repository
}
// NewResolvingRepository resolves inherited profile definitions from source.
func NewResolvingRepository(source Repository) Repository {
return &resolvingRepository{source: source}
}
func (r *resolvingRepository) GetProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) {
if r == nil || r.source == nil {
return nil, fmt.Errorf("%w: profile repository is required", ErrInvalidProfile)
}
requestedID := strings.TrimSpace(id)
if requestedID == "" {
return nil, fmt.Errorf("%w: profile id is required", ErrInvalidProfile)
}
if err := ctx.Err(); err != nil {
return nil, err
}
profile, err := r.getRawProfile(ctx, requestedID)
if err != nil {
return nil, err
}
if profile == nil {
return nil, fmt.Errorf("%w: selected profile %q is nil", ErrInvalidProfile, requestedID)
}
chain := []*domain.ExecutionProfile{profile}
chainIDs := []string{requestedID}
visited := map[string]struct{}{requestedID: {}}
current := profile
for {
baseID := strings.TrimSpace(current.BaseProfileID)
if baseID == "" {
break
}
if err := ctx.Err(); err != nil {
return nil, err
}
if _, seen := visited[baseID]; seen {
return nil, fmt.Errorf("%w: profile inheritance cycle %s", ErrInvalidProfile, joinProfileChain(chainIDs, baseID))
}
if len(chain) >= maximumProfileChainLength {
return nil, fmt.Errorf("%w: profile inheritance chain exceeds %d profiles: %s", ErrInvalidProfile, maximumProfileChainLength, joinProfileChain(chainIDs, baseID))
}
base, err := r.getRawProfile(ctx, baseID)
if err != nil {
if errors.Is(err, ErrProfileNotFound) {
return nil, fmt.Errorf("%w: base profile %q is missing in chain %s", ErrInvalidProfile, baseID, joinProfileChain(chainIDs, baseID))
}
return nil, fmt.Errorf("%w: failed to load base profile %q in chain %s: %w", ErrInvalidProfile, baseID, joinProfileChain(chainIDs, baseID), err)
}
if base == nil {
return nil, fmt.Errorf("%w: base profile %q is nil in chain %s", ErrInvalidProfile, baseID, joinProfileChain(chainIDs, baseID))
}
chain = append(chain, base)
chainIDs = append(chainIDs, baseID)
visited[baseID] = struct{}{}
current = base
}
resolved, err := mergeProfileChain(chain)
if err != nil {
return nil, fmt.Errorf("%w: resolved profile chain %s: %w", ErrInvalidProfile, strings.Join(chainIDs, " -> "), err)
}
if err := validateResolvedProfile(resolved); err != nil {
return nil, fmt.Errorf("%w: resolved profile chain %s: %w", ErrInvalidProfile, strings.Join(chainIDs, " -> "), err)
}
return resolved, nil
}
func (r *resolvingRepository) getRawProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) {
profile, err := r.source.GetProfile(ctx, id)
if err != nil {
return nil, err
}
if err := ctx.Err(); err != nil {
return nil, err
}
return profile, nil
}
func joinProfileChain(chain []string, next string) string {
return strings.Join(append(append([]string(nil), chain...), next), " -> ")
}
func mergeProfileChain(chain []*domain.ExecutionProfile) (*domain.ExecutionProfile, error) {
resolved := &domain.ExecutionProfile{ID: chain[0].ID}
for index := len(chain) - 1; index >= 0; index-- {
definition := chain[index]
if strings.TrimSpace(definition.BackendID) != "" {
resolved.BackendID = definition.BackendID
}
if strings.TrimSpace(definition.Endpoint) != "" {
resolved.Endpoint = definition.Endpoint
}
if strings.TrimSpace(definition.Model) != "" {
resolved.Model = definition.Model
}
if definition.Temperature != 0 {
resolved.Temperature = definition.Temperature
}
if definition.MaxTokens != 0 {
resolved.MaxTokens = definition.MaxTokens
}
if definition.TopP != 0 {
resolved.TopP = definition.TopP
}
if definition.TimeoutSeconds != 0 {
resolved.TimeoutSeconds = definition.TimeoutSeconds
}
if strings.TrimSpace(definition.ServiceTier) != "" {
resolved.ServiceTier = definition.ServiceTier
}
if strings.TrimSpace(definition.ReasoningEffort) != "" {
resolved.ReasoningEffort = definition.ReasoningEffort
}
if strings.TrimSpace(definition.APIKeyEnv) != "" {
resolved.APIKeyEnv = definition.APIKeyEnv
}
resolved.APIKeyRequired = resolved.APIKeyRequired || definition.APIKeyRequired
if len(definition.ExtraParams) != 0 {
extraParams, err := jsonvalue.CopyMap(definition.ExtraParams)
if err != nil {
return nil, err
}
resolved.ExtraParams = extraParams
}
}
resolved.ID = chain[0].ID
resolved.BaseProfileID = ""
return resolved, nil
}
func validateResolvedProfile(profile *domain.ExecutionProfile) error {
if profile == nil {
return errors.New("resolved profile is required")
}
profile.BaseProfileID = ""
return NormalizeAndValidateDefinition(profile)
}

View File

@@ -0,0 +1,418 @@
package profile
import (
"context"
"errors"
"fmt"
"io/fs"
"reflect"
"strings"
"sync"
"testing"
"testing/fstest"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
)
func TestResolvingRepositoryMergesProfileChain(t *testing.T) {
repo := &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"leaf": {
ID: "leaf",
BaseProfileID: "middle",
BackendID: "leaf-backend",
TopP: 0.8,
TimeoutSeconds: 45,
ReasoningEffort: "high",
},
"middle": {
ID: "middle",
BaseProfileID: "root",
Endpoint: "https://middle.example/v1",
Model: "middle-model",
MaxTokens: 256,
APIKeyEnv: "MIDDLE_API_KEY",
APIKeyRequired: true,
ExtraParams: map[string]any{"middle": map[string]any{"value": "middle"}},
},
"root": {
ID: "root",
BackendID: "root-backend",
Endpoint: "https://root.example/v1",
Model: "root-model",
Temperature: 0.3,
ServiceTier: "priority",
ExtraParams: map[string]any{"root": "value"},
},
}}
got, err := NewResolvingRepository(repo).GetProfile(context.Background(), "leaf")
if err != nil {
t.Fatalf("resolve profile: %v", err)
}
want := &domain.ExecutionProfile{
ID: "leaf",
BackendID: "leaf-backend",
Endpoint: "https://middle.example/v1",
Model: "middle-model",
Temperature: 0.3,
MaxTokens: 256,
TopP: 0.8,
TimeoutSeconds: 45,
ServiceTier: "priority",
ReasoningEffort: "high",
APIKeyEnv: "MIDDLE_API_KEY",
APIKeyRequired: true,
ExtraParams: map[string]any{"middle": map[string]any{"value": "middle"}},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("resolved profile:\n got %#v\nwant %#v", got, want)
}
}
func TestResolvingRepositoryRejectsMissingSourceAndProfileID(t *testing.T) {
if _, err := NewResolvingRepository(nil).GetProfile(context.Background(), "profile"); !errors.Is(err, ErrInvalidProfile) {
t.Fatalf("nil source error = %v, want ErrInvalidProfile", err)
}
repo := &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{}}
if _, err := NewResolvingRepository(repo).GetProfile(context.Background(), " \t "); !errors.Is(err, ErrInvalidProfile) {
t.Fatalf("blank id error = %v, want ErrInvalidProfile", err)
}
if got := repo.callCount(" "); got != 0 {
t.Fatalf("blank id looked up source %d times", got)
}
}
func TestResolvingRepositoryCopiesExtraParams(t *testing.T) {
baseParams := map[string]any{"nested": map[string]any{"value": "base"}}
repo := &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"child": {ID: "child", BaseProfileID: "base"},
"base": {
ID: "base",
Endpoint: "https://base.example/v1",
Model: "model",
ExtraParams: baseParams,
},
}}
resolver := NewResolvingRepository(repo)
first, err := resolver.GetProfile(context.Background(), "child")
if err != nil {
t.Fatalf("resolve inherited map: %v", err)
}
first.ExtraParams["nested"].(map[string]any)["value"] = "mutated"
second, err := resolver.GetProfile(context.Background(), "child")
if err != nil {
t.Fatalf("resolve inherited map again: %v", err)
}
if got := second.ExtraParams["nested"].(map[string]any)["value"]; got != "base" {
t.Fatalf("later result retained mutation: %v", got)
}
if got := baseParams["nested"].(map[string]any)["value"]; got != "base" {
t.Fatalf("source map retained mutation: %v", got)
}
repo.set("child", &domain.ExecutionProfile{
ID: "child",
BaseProfileID: "base",
ExtraParams: map[string]any{"child": "replacement"},
})
replaced, err := resolver.GetProfile(context.Background(), "child")
if err != nil {
t.Fatalf("resolve replacement map: %v", err)
}
if !reflect.DeepEqual(replaced.ExtraParams, map[string]any{"child": "replacement"}) {
t.Fatalf("extra params = %#v, want complete child replacement", replaced.ExtraParams)
}
}
func TestResolvingRepositoryUsesRawOverlayForEachLookup(t *testing.T) {
leafSource := NewFSRepository(profileTestFS(map[string]string{
"leaf.yaml": "id: leaf\nbase_profile: base\n",
}), ".")
fallback := NewFSRepository(profileTestFS(map[string]string{
"base.yaml": "id: base\nendpoint: https://fallback.example/v1\nmodel: fallback-model\n",
}), ".")
overlay := NewOverlayRepository(leafSource, fallback)
resolver := NewResolvingRepository(overlay)
got, err := resolver.GetProfile(context.Background(), "leaf")
if err != nil {
t.Fatalf("resolve fallback base: %v", err)
}
if got.Model != "fallback-model" {
t.Fatalf("fallback base model = %q", got.Model)
}
shadowing := NewOverlayRepository(NewFSRepository(profileTestFS(map[string]string{
"leaf.yaml": "id: leaf\nbase_profile: base\n",
"base.yaml": "id: base\nendpoint: https://primary.example/v1\nmodel: primary-model\n",
}), "."), fallback)
got, err = NewResolvingRepository(shadowing).GetProfile(context.Background(), "leaf")
if err != nil {
t.Fatalf("resolve shadowed base: %v", err)
}
if got.Model != "primary-model" || got.Endpoint != "https://primary.example/v1" {
t.Fatalf("shadowed base = %+v", got)
}
}
func TestResolvingRepositoryReportsSafetyAndSourceErrors(t *testing.T) {
sourceErr := errors.New("source failure")
tests := []struct {
name string
repo *resolvingTestRepository
id string
want []error
wantNot error
contains []string
}{
{
name: "missing selected profile preserves not found",
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{}},
id: "missing",
want: []error{ErrProfileNotFound},
wantNot: ErrInvalidProfile,
},
{
name: "missing base is invalid but not not found",
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"leaf": {ID: "leaf", BaseProfileID: "missing"},
}},
id: "leaf",
want: []error{ErrInvalidProfile},
wantNot: ErrProfileNotFound,
contains: []string{"missing", "leaf -> missing"},
},
{
name: "direct cycle",
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"a": {ID: "a", BaseProfileID: "a"},
}},
id: "a",
want: []error{ErrInvalidProfile},
contains: []string{"a -> a"},
},
{
name: "indirect cycle",
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"a": {ID: "a", BaseProfileID: "b"},
"b": {ID: "b", BaseProfileID: "c"},
"c": {ID: "c", BaseProfileID: "a"},
}},
id: "a",
want: []error{ErrInvalidProfile},
contains: []string{"a -> b -> c -> a"},
},
{
name: "nil result",
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"leaf": nil,
}},
id: "leaf",
want: []error{ErrInvalidProfile},
},
{
name: "incomplete resolved profile",
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"leaf": {ID: "leaf", BaseProfileID: "base"},
"base": {ID: "base", Model: "model"},
}},
id: "leaf",
want: []error{ErrInvalidProfile},
},
{
name: "base source error is retained",
repo: &resolvingTestRepository{
profiles: map[string]*domain.ExecutionProfile{"leaf": {ID: "leaf", BaseProfileID: "base"}},
errors: map[string]error{"base": sourceErr},
},
id: "leaf",
want: []error{ErrInvalidProfile, sourceErr},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
_, err := NewResolvingRepository(tc.repo).GetProfile(context.Background(), tc.id)
for _, want := range tc.want {
if !errors.Is(err, want) {
t.Fatalf("error = %v, want %v", err, want)
}
}
if tc.wantNot != nil && errors.Is(err, tc.wantNot) {
t.Fatalf("error = %v, must not match %v", err, tc.wantNot)
}
for _, fragment := range tc.contains {
if !strings.Contains(err.Error(), fragment) {
t.Fatalf("error = %v, want %q", err, fragment)
}
}
})
}
}
func TestResolvingRepositoryEnforcesChainLength(t *testing.T) {
for _, count := range []int{maximumProfileChainLength, maximumProfileChainLength + 1} {
t.Run(fmt.Sprintf("%d profiles", count), func(t *testing.T) {
profiles := make(map[string]*domain.ExecutionProfile, count)
for index := 1; index <= count; index++ {
id := fmt.Sprintf("profile-%d", index)
definition := &domain.ExecutionProfile{ID: id}
if index == count {
definition.Endpoint = "https://root.example/v1"
definition.Model = "model"
} else {
definition.BaseProfileID = fmt.Sprintf("profile-%d", index+1)
}
profiles[id] = definition
}
got, err := NewResolvingRepository(&resolvingTestRepository{profiles: profiles}).GetProfile(context.Background(), "profile-1")
if count == maximumProfileChainLength {
if err != nil || got == nil {
t.Fatalf("profile = %+v, error = %v, want accepted chain", got, err)
}
return
}
if !errors.Is(err, ErrInvalidProfile) {
t.Fatalf("error = %v, want ErrInvalidProfile", err)
}
})
}
}
func TestResolvingRepositoryIsFreshAndCancellationAware(t *testing.T) {
repo := &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
"leaf": {ID: "leaf", BaseProfileID: "base"},
"base": {ID: "base", Endpoint: "https://base.example/v1", Model: "first", ExtraParams: map[string]any{"nested": map[string]any{"value": "first"}}},
}}
resolver := NewResolvingRepository(repo)
first, err := resolver.GetProfile(context.Background(), "leaf")
if err != nil || first.Model != "first" {
t.Fatalf("first result=(%+v, %v)", first, err)
}
repo.set("base", &domain.ExecutionProfile{ID: "base", Endpoint: "https://base.example/v1", Model: "second", ExtraParams: map[string]any{"nested": map[string]any{"value": "second"}}})
second, err := resolver.GetProfile(context.Background(), "leaf")
if err != nil || second.Model != "second" {
t.Fatalf("second result=(%+v, %v)", second, err)
}
canceled, cancel := context.WithCancel(context.Background())
cancel()
if _, err := resolver.GetProfile(canceled, "leaf"); !errors.Is(err, context.Canceled) {
t.Fatalf("canceled lookup error = %v", err)
}
if got := repo.callCount("leaf"); got != 2 {
t.Fatalf("calls after canceled lookup = %d, want 2", got)
}
duringTraversal, cancelDuringTraversal := context.WithCancel(context.Background())
repo.afterGet = func(id string) {
if id == "leaf" {
cancelDuringTraversal()
}
}
if _, err := resolver.GetProfile(duringTraversal, "leaf"); !errors.Is(err, context.Canceled) {
t.Fatalf("during traversal error = %v", err)
}
if got := repo.callCount("base"); got != 2 {
t.Fatalf("base calls after cancellation = %d, want 2", got)
}
terminalLookup, cancelTerminalLookup := context.WithCancel(context.Background())
repo.afterGet = func(id string) {
if id == "base" {
cancelTerminalLookup()
}
}
if _, err := resolver.GetProfile(terminalLookup, "leaf"); !errors.Is(err, context.Canceled) {
t.Fatalf("terminal lookup cancellation error = %v", err)
}
if got := repo.callCount("base"); got != 3 {
t.Fatalf("base calls after terminal cancellation = %d, want 3", got)
}
repo.afterGet = nil
var wg sync.WaitGroup
errors := make(chan error, 8)
for index := 0; index < cap(errors); index++ {
wg.Add(1)
go func() {
defer wg.Done()
resolved, err := resolver.GetProfile(context.Background(), "leaf")
if err != nil {
errors <- err
return
}
resolved.ExtraParams["nested"].(map[string]any)["value"] = "mutated"
}()
}
wg.Wait()
close(errors)
for err := range errors {
t.Errorf("concurrent resolution: %v", err)
}
latest, err := resolver.GetProfile(context.Background(), "leaf")
if err != nil || latest.ExtraParams["nested"].(map[string]any)["value"] != "second" {
t.Fatalf("latest result=(%+v, %v)", latest, err)
}
}
type resolvingTestRepository struct {
mu sync.Mutex
profiles map[string]*domain.ExecutionProfile
errors map[string]error
calls map[string]int
afterGet func(string)
}
func (r *resolvingTestRepository) GetProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
r.mu.Lock()
if r.calls == nil {
r.calls = make(map[string]int)
}
r.calls[id]++
err := r.errors[id]
profile := r.profiles[id]
afterGet := r.afterGet
r.mu.Unlock()
if afterGet != nil {
afterGet(id)
}
if err != nil {
return nil, err
}
if profile == nil {
if _, exists := r.profiles[id]; exists {
return nil, nil
}
return nil, ErrProfileNotFound
}
copy := *profile
return &copy, nil
}
func (r *resolvingTestRepository) set(id string, profile *domain.ExecutionProfile) {
r.mu.Lock()
defer r.mu.Unlock()
r.profiles[id] = profile
}
func (r *resolvingTestRepository) callCount(id string) int {
r.mu.Lock()
defer r.mu.Unlock()
return r.calls[id]
}
func profileTestFS(files map[string]string) fs.FS {
fsys := make(fstest.MapFS, len(files))
for name, content := range files {
fsys[name] = profileMapFile(content)
}
return fsys
}

View File

@@ -0,0 +1,243 @@
package promptkit_test
import (
"context"
"errors"
"io/fs"
"strings"
"sync"
"testing"
"testing/fstest"
"gitea.maximumdirect.net/eric/promptkit"
)
func TestProfileInheritanceBuiltInAliasWorkflow(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{
PromptDir: frameworkPromptDir,
SchemaDir: frameworkSchemaDir,
}, promptkit.WithProfiles(promptkit.Profile{
ID: "weather-light",
BaseProfileID: "deepseek-4-flash",
ReasoningEffort: "high",
TimeoutSeconds: 120,
}))
if err != nil {
t.Fatalf("construct alias engine: %v", err)
}
base, err := engine.InspectProfile(context.Background(), "deepseek-4-flash")
if err != nil {
t.Fatalf("inspect base: %v", err)
}
child, err := engine.InspectProfile(context.Background(), "weather-light")
if err != nil {
t.Fatalf("inspect alias: %v", err)
}
if child.ProfileID != "weather-light" ||
child.EffectiveModelParams.BackendID != base.EffectiveModelParams.BackendID ||
child.EffectiveModelParams.Model != base.EffectiveModelParams.Model ||
child.EffectiveModelParams.ReasoningEffort != "high" ||
child.EffectiveModelParams.TimeoutSeconds != 120 {
t.Fatalf("alias inspection = %+v, base = %+v", child, base)
}
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
PromptID: frameworkMarkdownSummaryPromptID,
ProfileID: "weather-light",
Inputs: map[string]promptkit.ArtifactRef{
"transcript": promptkit.Inline("Rin opens the gate."),
"glossary": promptkit.Inline("gate: A guarded passage."),
},
})
if err != nil {
t.Fatalf("prepare alias: %v", err)
}
if prepared.SelectedProfileID != "weather-light" {
t.Fatalf("SelectedProfileID = %q", prepared.SelectedProfileID)
}
timeout := 15
reasoning := "low"
overridden, err := engine.Prepare(context.Background(), promptkit.RunRequest{
PromptID: frameworkMarkdownSummaryPromptID,
ProfileID: "weather-light",
Execution: &promptkit.ExecutionTargetOverride{
TimeoutSeconds: &timeout,
ReasoningEffort: &reasoning,
},
Inputs: map[string]promptkit.ArtifactRef{
"transcript": promptkit.Inline("Rin opens the gate."),
"glossary": promptkit.Inline("gate: A guarded passage."),
},
})
if err != nil {
t.Fatalf("prepare override: %v", err)
}
if overridden.EffectiveModelParams.TimeoutSeconds != timeout ||
overridden.EffectiveModelParams.ReasoningEffort != reasoning {
t.Fatalf("runtime override target = %+v", overridden.EffectiveModelParams)
}
}
func TestProfileInheritanceYAMLAliasOfBuiltIn(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(fstest.MapFS{}, "."),
promptkit.WithProfileFS(fstest.MapFS{
"alias.yaml": &fstest.MapFile{Data: []byte("id: yaml-alias\nbase_profile: deepseek-4-flash\n")},
}, "."),
)
if err != nil {
t.Fatalf("construct YAML alias engine: %v", err)
}
inspection, err := engine.InspectProfile(context.Background(), "yaml-alias")
if err != nil {
t.Fatalf("inspect YAML alias: %v", err)
}
if inspection.ProfileID != "yaml-alias" || inspection.EffectiveModelParams.Model == "" {
t.Fatalf("YAML alias inspection = %+v", inspection)
}
}
func TestProfileInheritanceRetainsRequiredCredentialBehavior(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(fstest.MapFS{}, "."),
promptkit.WithBackend(promptkit.Backend{
ID: "credential-backend",
Endpoint: "https://credential.example/v1",
APIKeyEnv: "OPTIONAL_BACKEND_KEY",
}),
promptkit.WithProfiles(
promptkit.Profile{
ID: "credential-base",
BackendID: "credential-backend",
Model: "model",
APIKeyRequired: true,
},
promptkit.Profile{ID: "credential-child", BaseProfileID: "credential-base"},
),
)
if err != nil {
t.Fatalf("construct credential inheritance engine: %v", err)
}
inspection, err := engine.InspectProfile(context.Background(), "credential-child")
if err != nil {
t.Fatalf("inspect credential child: %v", err)
}
if !inspection.APIKeyRequired || inspection.EffectiveModelParams.APIKeyEnv != "" {
t.Fatalf("credential inspection = %+v", inspection)
}
}
func TestProfileInheritancePreservesPublicErrorIdentities(t *testing.T) {
newEngine := func(t *testing.T, profiles fs.FS) *promptkit.Engine {
t.Helper()
options := []promptkit.Option{promptkit.WithPromptFS(fstest.MapFS{}, ".")}
if profiles != nil {
options = append(options, promptkit.WithProfileFS(profiles, "."))
}
engine, err := promptkit.NewEngine(promptkit.Config{}, options...)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
return engine
}
tests := []struct {
name string
profiles fs.FS
profile string
contains []string
want error
wantNot error
}{
{
name: "missing selected profile",
profile: "missing",
want: promptkit.ErrProfileNotFound,
wantNot: promptkit.ErrProfileLoad,
},
{
name: "missing base",
profiles: fstest.MapFS{
"child.yaml": &fstest.MapFile{Data: []byte("id: child\nbase_profile: missing\n")},
},
profile: "child",
contains: []string{"child", "missing"},
want: promptkit.ErrProfileLoad,
wantNot: promptkit.ErrProfileNotFound,
},
{
name: "cycle",
profiles: fstest.MapFS{
"a.yaml": &fstest.MapFile{Data: []byte("id: a\nbase_profile: b\n")},
"b.yaml": &fstest.MapFile{Data: []byte("id: b\nbase_profile: a\n")},
},
profile: "a",
contains: []string{"a", "b"},
want: promptkit.ErrProfileLoad,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
result, err := newEngine(t, tc.profiles).InspectProfile(context.Background(), tc.profile)
if result != nil || !errors.Is(err, tc.want) || (tc.wantNot != nil && errors.Is(err, tc.wantNot)) {
t.Fatalf("inspection=(%+v, %v), want %v without %v", result, err, tc.want, tc.wantNot)
}
for _, fragment := range tc.contains {
if !strings.Contains(err.Error(), fragment) {
t.Fatalf("error = %v, want %q", err, fragment)
}
}
})
}
}
func TestProfileInheritanceFreezesPreparedExecution(t *testing.T) {
profiles := &mutableInheritanceProfileFS{files: fstest.MapFS{
"child.yaml": &fstest.MapFile{Data: []byte("id: child\nbase_profile: base\n")},
"base.yaml": &fstest.MapFile{Data: []byte("id: base\nendpoint: https://base.example/v1\nmodel: first-model\n")},
}}
client := &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}}
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prepared", "child", "content"), "."),
promptkit.WithProfileFS(profiles, "."),
promptkit.WithLLMClient(client),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
prepared, err := engine.PrepareExecution(context.Background(), promptkit.RunRequest{PromptID: "prepared"})
if err != nil {
t.Fatalf("prepare execution: %v", err)
}
profiles.set("base.yaml", "id: base\nendpoint: https://base.example/v1\nmodel: second-model\n")
result, err := engine.RunPrepared(context.Background(), prepared)
if err != nil || result == nil || len(client.requests) != 1 || client.requests[0].Target.Model != "first-model" {
t.Fatalf("prepared execution=(%+v, %v), requests=%+v", result, err, client.requests)
}
inspection, err := engine.InspectProfile(context.Background(), "child")
if err != nil || inspection.EffectiveModelParams.Model != "second-model" {
t.Fatalf("fresh inspection=(%+v, %v)", inspection, err)
}
}
type mutableInheritanceProfileFS struct {
mu sync.RWMutex
files fstest.MapFS
}
func (f *mutableInheritanceProfileFS) Open(name string) (fs.File, error) {
f.mu.RLock()
defer f.mu.RUnlock()
return f.files.Open(name)
}
func (f *mutableInheritanceProfileFS) set(name, content string) {
f.mu.Lock()
defer f.mu.Unlock()
f.files[name] = &fstest.MapFile{Data: []byte(content)}
}

View File

@@ -2,17 +2,16 @@ package promptkit
import (
"context"
"errors"
"fmt"
"strings"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
"gitea.maximumdirect.net/eric/promptkit/internal/jsonvalue"
"gitea.maximumdirect.net/eric/promptkit/internal/profile"
)
// OpenAICompatibleProfile returns an ordinary in-memory Profile for an
// OpenAI-compatible chat-completions endpoint.
// OpenAICompatibleProfile returns an in-memory Profile for an OpenAI-compatible
// chat-completions endpoint. A non-blank BaseProfileID permits its target
// fields to be inherited when the profile is selected or inspected.
//
// It does not register global state, maintain a model catalog, or resolve
// credentials. If APIKeyRequired is true, callers satisfy it with
@@ -25,6 +24,7 @@ import (
func OpenAICompatibleProfile(cfg OpenAICompatibleProfileConfig) Profile {
return Profile{
ID: cfg.ID,
BaseProfileID: cfg.BaseProfileID,
BackendID: cfg.BackendID,
Endpoint: cfg.Endpoint,
Model: cfg.Model,
@@ -87,8 +87,9 @@ func toDomainProfile(publicProfile Profile) (domain.ExecutionProfile, error) {
return domain.ExecutionProfile{}, err
}
prof := domain.ExecutionProfile{
ID: strings.TrimSpace(publicProfile.ID),
BackendID: strings.TrimSpace(publicProfile.BackendID),
ID: publicProfile.ID,
BaseProfileID: publicProfile.BaseProfileID,
BackendID: publicProfile.BackendID,
Endpoint: publicProfile.Endpoint,
Model: publicProfile.Model,
Temperature: publicProfile.Temperature,
@@ -100,34 +101,8 @@ func toDomainProfile(publicProfile Profile) (domain.ExecutionProfile, error) {
APIKeyRequired: publicProfile.APIKeyRequired,
ExtraParams: extraParams,
}
if err := normalizeAndValidatePublicProfile(&prof); err != nil {
if err := profile.NormalizeAndValidateDefinition(&prof); err != nil {
return domain.ExecutionProfile{}, err
}
return prof, nil
}
func normalizeAndValidatePublicProfile(prof *domain.ExecutionProfile) error {
if strings.TrimSpace(prof.ID) == "" {
return errors.New("id is required")
}
prof.Endpoint = strings.TrimSpace(prof.Endpoint)
if strings.TrimSpace(prof.BackendID) == "" && prof.Endpoint == "" {
return errors.New("backend or endpoint is required")
}
if prof.Endpoint != "" {
endpoint, err := domain.NormalizeOpenAICompatibleBaseEndpoint(prof.Endpoint)
if err != nil {
return err
}
prof.Endpoint = endpoint
}
if strings.TrimSpace(prof.Model) == "" {
return errors.New("model is required")
}
return domain.ValidateExecutionTargetSettings(domain.ExecutionTarget{
Temperature: prof.Temperature,
MaxTokens: prof.MaxTokens,
TopP: prof.TopP,
TimeoutSeconds: prof.TimeoutSeconds,
})
}

View File

@@ -443,31 +443,43 @@ type ExecutionTargetOverride struct {
// Profile is an in-memory execution profile for library consumers.
//
// It is equivalent to a loaded profile file after validation. Raw API keys do
// not belong in profiles; use APIKeyRequired to require callers to provide a
// RunRequest.APIKey or explicit request ExecutionTargetOverride.APIKeyEnv, or
// use profile YAML api_key_env with file and FS profile sources. Profile has no
// stable JSON representation.
// A standalone Profile is equivalent to a loaded profile file after local
// validation. A derived profile names BaseProfileID and can inherit target
// fields when selected or inspected. Raw API keys do not belong in profiles;
// use APIKeyRequired to require callers to provide a RunRequest.APIKey or
// explicit request ExecutionTargetOverride.APIKeyEnv, or use profile YAML
// api_key_env with file and FS profile sources. Profile has no stable JSON
// representation.
//
// WithProfiles validates and copies Profile values during NewEngine. Zero
// Temperature, MaxTokens, and TopP values and blank ServiceTier and
// ReasoningEffort values leave those provider controls unspecified. A zero
// TimeoutSeconds retains the framework deadline, while an empty ExtraParams map
// inherits backend request defaults. Use ExecutionTargetOverride pointer fields
// to request an explicit numeric zero.
// WithProfiles locally validates and copies Profile values during NewEngine.
// It checks base-reference existence and resolved target completeness when a
// derived profile is selected or inspected. Zero Temperature, MaxTokens, and
// TopP values and blank ServiceTier and ReasoningEffort values leave those
// provider controls unspecified. A zero TimeoutSeconds retains the framework
// deadline, while an empty ExtraParams map inherits backend request defaults.
// Use ExecutionTargetOverride pointer fields to request an explicit numeric
// zero.
type Profile struct {
// ID is the required non-blank profile identifier. WithProfiles trims it.
ID string
// BaseProfileID optionally names one base profile. WithProfiles trims it. A
// non-blank value permits required target fields to be inherited when the
// profile is selected or inspected, which is also when reference existence
// and resolved completeness are checked. A blank value leaves this as a
// standalone profile.
BaseProfileID string
// BackendID optionally selects an engine backend. WithProfiles trims it.
// Backend membership is checked when a request selects the profile; an
// unknown ID makes preparation fail with ErrProfileLoad.
BackendID string
// Endpoint is the model-provider base URL. It is required only when
// BackendID is blank and otherwise overrides the backend endpoint when
// Endpoint is the model-provider base URL. A standalone Profile requires an
// endpoint when BackendID is blank; a derived Profile may inherit either
// field. A non-blank endpoint overrides the backend endpoint when
// non-blank. WithProfiles trims it and requires an absolute HTTP or HTTPS URL
// with a host and no user information, query, or fragment.
Endpoint string
// Model is the required non-blank provider model identifier.
// Model is the provider model identifier. It is required for a standalone
// Profile and may be inherited by a derived Profile.
Model string
// Temperature is from 0 through 2. Zero leaves the provider control
// unspecified.
@@ -499,13 +511,16 @@ type Profile struct {
// profile.
//
// It contains ordinary profile fields for OpenAI-compatible chat-completions
// endpoints. APIKeyRequired follows Profile.APIKeyRequired. Raw API keys do not
// belong in this config. OpenAICompatibleProfileConfig has no stable JSON
// endpoints. BaseProfileID and APIKeyRequired follow Profile. Raw API keys do
// not belong in this config. OpenAICompatibleProfileConfig has no stable JSON
// representation and is not validated until its resulting Profile is supplied
// through WithProfiles to NewEngine.
type OpenAICompatibleProfileConfig struct {
// ID becomes Profile.ID.
ID string
// BaseProfileID becomes Profile.BaseProfileID. A non-blank value permits the
// resulting Profile to inherit target fields when it is selected or inspected.
BaseProfileID string
// BackendID becomes Profile.BackendID.
BackendID string
// Endpoint becomes Profile.Endpoint.