Compare commits
6 Commits
2d44305a8a
...
c53250f023
| Author | SHA1 | Date | |
|---|---|---|---|
| c53250f023 | |||
| 3d99483219 | |||
| 67f788b1e2 | |||
| 764103a2e2 | |||
| a08dd83d1f | |||
| e8922d8ec5 |
@@ -33,10 +33,13 @@ boundary and constraints that framework work must preserve.
|
||||
|
||||
## Release Guidance
|
||||
|
||||
Consumers upgrading from `v0.5.0` to `v0.6.0` should read the
|
||||
[v0.6.0 changelog and migration guide](docs/releases/v0.6.0.md).
|
||||
Consumers upgrading from `v0.6.0` to `v0.7.0` should read the
|
||||
[v0.7.0 changelog and migration guide](docs/releases/v0.7.0.md).
|
||||
|
||||
Earlier adopters can consult the
|
||||
[v0.6.0 changelog and migration guide](docs/releases/v0.6.0.md).
|
||||
|
||||
Consumers upgrading from `v0.4.0` to `v0.5.0` can consult the
|
||||
[v0.5.0 changelog and migration guide](docs/releases/v0.5.0.md).
|
||||
|
||||
Consumers upgrading from `v0.3.0` to `v0.4.0` should read the
|
||||
|
||||
@@ -189,6 +189,26 @@ For programmatic profiles,
|
||||
[`OpenAICompatibleProfile`](../../profiles.go) converts ordinary
|
||||
OpenAI-compatible settings into a value accepted by `WithProfiles`.
|
||||
|
||||
### Alias A Built-In Profile
|
||||
|
||||
Give an application-owned profile ID a built-in base when prompts should select
|
||||
the application ID while inheriting the built-in target. The child can override
|
||||
only the setting it owns:
|
||||
|
||||
```go
|
||||
promptkit.WithProfiles(promptkit.Profile{
|
||||
ID: "weather-light",
|
||||
BaseProfileID: "deepseek-4-flash",
|
||||
ReasoningEffort: "high",
|
||||
})
|
||||
```
|
||||
|
||||
Select `weather-light` in a prompt or `RunRequest.ProfileID`; it remains the
|
||||
reported selected profile. See the [profile inheritance format
|
||||
reference](../formats.md#profile-inheritance) and the
|
||||
[`Profile` GoDoc](../../types.go) for exact lookup, merging, and validation
|
||||
behavior.
|
||||
|
||||
### Use The Rakestrawhome Built-In Profile
|
||||
|
||||
Set `RAKESTRAWHOME_INFERENCE_API_KEY` in the application environment, then
|
||||
|
||||
@@ -173,9 +173,19 @@ extra_params:
|
||||
provider_option: enabled
|
||||
```
|
||||
|
||||
A derived profile can use a named base and override only the settings it owns:
|
||||
|
||||
```yaml
|
||||
id: local-summary-fast
|
||||
base_profile: local-summary
|
||||
timeout_seconds: 30
|
||||
reasoning_effort: low
|
||||
```
|
||||
|
||||
| Field | Required | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `id` | yes | Profile identifier, trimmed before selection and publication. It must be non-empty after trimming and unique within one source after normalization. |
|
||||
| `base_profile` | no | One optional parent profile ID. A derived profile may inherit target fields from it. |
|
||||
| `backend` | unless `endpoint` is present | Backend registry ID. It is trimmed and registry membership is checked when the profile is prepared or inspected. |
|
||||
| `endpoint` | unless `backend` is present | OpenAI-compatible base URL, including an API version path when required. A nonempty value is trimmed and must be absolute HTTP or HTTPS with a host and without user information, a query, or a fragment. When both connection fields are present, this overrides the backend endpoint without changing backend identity. |
|
||||
| `model` | yes | Non-empty provider model name. |
|
||||
@@ -191,6 +201,11 @@ extra_params:
|
||||
Raw `api_key` is prohibited in profile YAML. Store only an environment
|
||||
variable name in `api_key_env`.
|
||||
|
||||
A standalone profile must provide a model and at least one of `backend` or
|
||||
`endpoint`. A derived profile may omit those target fields because its selected
|
||||
base chain can provide them. Local parsing still validates a derived profile's
|
||||
own ID, supplied endpoint, execution-setting bounds, and `extra_params`.
|
||||
|
||||
Promptkit does not infer a backend from a model or endpoint. Endpoint-only
|
||||
profiles remain supported and have no effective backend ID.
|
||||
The engine always provides the built-in `openrouter` and `rakestrawhome` IDs.
|
||||
@@ -266,6 +281,32 @@ profiles. They use `APIKeyRequired` for request-scoped credentials instead of
|
||||
`api_key_env`. Preparation and exact profile inspection use this same source
|
||||
precedence.
|
||||
|
||||
When a selected definition names `base_profile`, every profile ID in that
|
||||
chain is looked up through this same precedence order. A higher-precedence
|
||||
definition therefore shadows a lower-precedence definition of the same base
|
||||
ID, including a built-in. References are not source-qualified.
|
||||
|
||||
### Profile Inheritance
|
||||
|
||||
Promptkit resolves one linear base chain of at most 32 profiles, including the
|
||||
selected profile. It merges settings from the root base to the selected leaf.
|
||||
The leaf's `id` remains the selected profile identity. Nonblank string fields
|
||||
(`backend`, `endpoint`, `model`, `service_tier`, `reasoning_effort`, and
|
||||
`api_key_env`) and nonzero numeric fields replace inherited values. A nonempty
|
||||
`extra_params` map replaces the complete inherited map rather than merging
|
||||
keys, and `APIKeyRequired: true` remains true through the chain. Backend and
|
||||
endpoint are independent: replacing one does not clear the other.
|
||||
|
||||
There is no profile-level clearing syntax. Blank strings, zero numbers, false,
|
||||
and empty maps remain unspecified and inherit from a base. Use existing
|
||||
presence-aware request overrides where an execution needs an explicit zero or
|
||||
empty reasoning setting.
|
||||
|
||||
An absent directly selected profile reports the ordinary not-found error. Once
|
||||
the selected profile exists, a missing base, cycle, overlong chain, or
|
||||
incomplete resolved target is a profile-load failure. Ordinary operations
|
||||
resolve chains afresh; prepared execution retains the fully resolved target.
|
||||
|
||||
## Built-In Profile Catalog
|
||||
|
||||
Every built-in profile selects one maintained built-in backend and inherits
|
||||
|
||||
@@ -21,7 +21,7 @@ contributor workflow and validation.
|
||||
| `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |
|
||||
| `internal/jsonvalue` | Validates and deeply copies bounded JSON-compatible extra-parameter and prepared-schema trees while preserving supported concrete value types and rejecting cycles or excessive depth and work. | [JSON values](../../internal/jsonvalue/jsonvalue.go) |
|
||||
| `internal/promptdef` | Loads strictly decoded, validated prompt definitions from filesystem and `fs.FS` sources, including version selection and contained file-backed message content. | [Framework formats](../formats.md), [prompt-definition repository](../../internal/promptdef/filesystem_repository.go) |
|
||||
| `internal/profile` | Loads strictly decoded, validated execution profiles, including backend selection, from filesystem and `fs.FS` sources and composes repositories with error-preserving fallback. | [Framework formats](../formats.md), [profile repositories](../../internal/profile/filesystem_repository.go) |
|
||||
| `internal/profile` | Loads strictly decoded, locally validated execution profiles from filesystem and `fs.FS` sources, overlays raw sources with error-preserving fallback, and resolves inherited profiles. | [Framework formats](../formats.md), [profile repositories](../../internal/profile/filesystem_repository.go), [internal sources](sources.md#profiles-and-built-ins) |
|
||||
| `internal/profile/builtin` | Embeds the built-in profile catalog, whose entries select maintained built-in backends. | [Built-in catalog](../formats.md#built-in-profile-catalog), [repository](../../internal/profile/builtin/repository.go) |
|
||||
| `internal/prompt` | Renders prompt messages from Go templates with artifact, variable, session, and cache-control data. | [Go-template renderer](../../internal/prompt/go_renderer.go) |
|
||||
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
|
||||
|
||||
@@ -39,17 +39,17 @@ duplicate detection, and source containment:
|
||||
|
||||
## Profiles And Built-Ins
|
||||
|
||||
`internal/profile` loads and validates execution profiles from an
|
||||
operating-system filesystem or an `fs.FS`. A file contains exactly one YAML
|
||||
document and its trimmed YAML `id` is its only selection identity; filenames do
|
||||
not confer authority. Each point lookup reads discovered files once for their
|
||||
metadata and reuses the selected file's bytes for strict decoding; unrelated
|
||||
profiles are not fully decoded. Strict selected decoding recognizes the
|
||||
optional `backend` field, trims its value, and requires a model plus at least
|
||||
one non-blank backend or endpoint. File-backed `extra_params` values are
|
||||
validated and defensively copied through the shared bounded JSON-value owner
|
||||
before a profile is published. OpenAI-compatible reserved-field policy remains
|
||||
with the model-client and backend-registry owners.
|
||||
`internal/profile` loads, locally validates, overlays, and resolves execution
|
||||
profiles from an operating-system filesystem or an `fs.FS`. A file contains
|
||||
exactly one YAML document and its trimmed YAML `id` is its only selection
|
||||
identity; filenames do not confer authority. Each point lookup reads discovered
|
||||
files once for their metadata and reuses the selected file's bytes for strict
|
||||
decoding; unrelated profiles are not fully decoded. Strict selected decoding
|
||||
recognizes `base_profile` and the optional `backend` field, trims their values,
|
||||
and permits inherited target fields only when a base is named. File-backed
|
||||
`extra_params` values are validated and defensively copied through the shared
|
||||
bounded JSON-value owner before a profile is published. OpenAI-compatible
|
||||
reserved-field policy remains with the model-client and backend-registry owners.
|
||||
|
||||
The overlay repository consults the next repository only when the
|
||||
higher-precedence repository reports that a profile is absent. A reliably
|
||||
@@ -59,14 +59,25 @@ backend registry membership because the available registry belongs to the
|
||||
assembled engine; the runner checks membership during preparation and exact
|
||||
profile inspection.
|
||||
|
||||
The root engine assembles profile repositories in precedence order: in-memory
|
||||
profiles, one ordinary configured source, an application fallback source, then
|
||||
the embedded built-in catalog. An explicit file or `fs.FS` profile source
|
||||
replaces `Config.ProfileDir` within the ordinary configured-source category.
|
||||
The root engine assembles one raw composite catalog in precedence order:
|
||||
in-memory profiles, one ordinary configured source, an application fallback
|
||||
source, then the embedded built-in catalog. An explicit file or `fs.FS` profile
|
||||
source replaces `Config.ProfileDir` within the ordinary configured-source
|
||||
category. One outer resolving repository wraps that complete raw catalog, so
|
||||
each base lookup observes the same precedence and shadowing rules.
|
||||
|
||||
Exact profile inspection performs one point-in-time lookup through those
|
||||
profile sources and checks the resolved target without reading prompt, input,
|
||||
The resolving repository traverses every selected chain afresh, retains no
|
||||
cache, detects cycles, limits a chain to 32 profiles, merges root-to-leaf into a
|
||||
new caller-owned value, and validates the final target before publishing it. It
|
||||
does not check backend registry membership. Exact `base_profile` syntax, merge
|
||||
rules, and consumer-visible failure behavior belong to the [framework format
|
||||
reference](../formats.md#profile-inheritance).
|
||||
|
||||
Exact profile inspection performs one point-in-time resolved lookup through
|
||||
those profile sources and checks the final target without reading prompt, input,
|
||||
or schema sources. It does not retain that lookup for a later execution.
|
||||
Prepared execution instead freezes the fully resolved target; a later ordinary
|
||||
operation performs a fresh traversal.
|
||||
|
||||
`internal/profile/builtin` embeds the maintained built-in profile catalog.
|
||||
Every embedded profile selects a maintained built-in backend and inherits that
|
||||
|
||||
155
docs/releases/v0.7.0.md
Normal file
155
docs/releases/v0.7.0.md
Normal file
@@ -0,0 +1,155 @@
|
||||
# Promptkit v0.7.0
|
||||
|
||||
This supplemental changelog and migration guide summarizes the consumer-facing
|
||||
changes from `v0.6.0` to `v0.7.0`. The annotated `v0.7.0` tag is the
|
||||
authoritative release record. Exact current contracts belong to the linked
|
||||
GoDoc and durable documentation.
|
||||
|
||||
## Summary
|
||||
|
||||
`v0.7.0` expands provider integration and profile composition while making
|
||||
credential and generation-failure handling more flexible:
|
||||
|
||||
- Promptkit now includes the `rakestrawhome` backend and its Gemma profile;
|
||||
- built-in generation failures expose bounded structured provider details;
|
||||
- an unavailable optional API-key environment source no longer prevents a
|
||||
request from reaching an upstream that permits unauthenticated access; and
|
||||
- profiles can inherit from and selectively refine another profile.
|
||||
|
||||
## Compatibility
|
||||
|
||||
This release adds public declarations and fields but removes none. Existing
|
||||
keyed configuration literals and ordinary `errors.Is` handling continue to
|
||||
work.
|
||||
|
||||
Adding `BaseProfileID` to `Profile` and `OpenAICompatibleProfileConfig` changes
|
||||
their struct shape. Consumers using positional composite literals for either
|
||||
type must convert them to keyed literals. Existing keyed literals require no
|
||||
change.
|
||||
|
||||
The `rakestrawhome` backend ID is now built in and reserved. A consumer that
|
||||
previously registered that exact ID with `WithBackend` must remove its manual
|
||||
registration before upgrading. Other custom backend registrations are
|
||||
unchanged.
|
||||
|
||||
When an optional backend, profile, or request `APIKeyEnv` is unset, empty, or
|
||||
whitespace-only, the built-in client now omits `Authorization` and sends the
|
||||
request. Previously this condition could fail before transport. Set
|
||||
`Profile.APIKeyRequired` when missing credentials must remain a local
|
||||
preflight error.
|
||||
|
||||
Provider non-success responses continue to match `ErrLLMGenerate`. Their
|
||||
rendered wording is not a compatibility contract; consumers can now use
|
||||
`errors.As` with `*GenerationError` when structured status information is
|
||||
needed.
|
||||
|
||||
## Upgrade
|
||||
|
||||
Update the module dependency with:
|
||||
|
||||
```sh
|
||||
go get gitea.maximumdirect.net/eric/promptkit@v0.7.0
|
||||
go mod tidy
|
||||
```
|
||||
|
||||
Remove any manual `rakestrawhome` backend registration, convert positional
|
||||
profile literals to keyed literals, and run the consuming project's ordinary
|
||||
and race-enabled tests.
|
||||
|
||||
## Rakestrawhome Built-In Backend And Profile
|
||||
|
||||
Every engine now includes the reserved `rakestrawhome` backend, identified by
|
||||
`BackendRakestrawHome`. The built-in `rakestrawhome-gemma-4-31b` profile
|
||||
selects that backend. Consumers can use the maintained endpoint, credential,
|
||||
capacity, and model defaults without registering either definition themselves.
|
||||
|
||||
See the [built-in backend and profile catalogs](../formats.md#built-in-backends)
|
||||
and the [consumer adoption example](../consumers/pkg-promptkit.md#use-the-rakestrawhome-built-in-profile)
|
||||
for the current contracts.
|
||||
|
||||
## Structured Generation Errors
|
||||
|
||||
Non-2xx responses from the built-in OpenAI-compatible client now return an
|
||||
immutable `*GenerationError`. Consumers can inspect the HTTP status and any
|
||||
safely extracted provider code, type, or message while retaining the ordinary
|
||||
generation-error category:
|
||||
|
||||
```go
|
||||
var generationErr *promptkit.GenerationError
|
||||
if errors.As(err, &generationErr) {
|
||||
status := generationErr.StatusCode()
|
||||
_ = status
|
||||
}
|
||||
```
|
||||
|
||||
Provider fields are bounded and normalized but remain untrusted and may
|
||||
contain sensitive request or schema details. Default and Go-syntax formatting
|
||||
omit those fields. Applications must apply their own disclosure policy before
|
||||
logging or presenting accessor values.
|
||||
|
||||
See the [`GenerationError` GoDoc](../../generation_error.go), the
|
||||
[consumer error-handling guide](../consumers/pkg-promptkit.md#handle-errors),
|
||||
and the [OpenAI-compatible response contract](../integrations/openai-compatible-chat.md#response-handling).
|
||||
|
||||
## Optional Credential Sources
|
||||
|
||||
`APIKeyEnv` names an optional environment lookup source unless the selected
|
||||
profile explicitly sets `APIKeyRequired`. When neither a direct request key nor
|
||||
a usable environment value exists, the built-in client omits the bearer header
|
||||
and handles the upstream response normally. This supports local and other
|
||||
OpenAI-compatible providers that permit unauthenticated requests without
|
||||
hiding an authentication error returned by a provider that requires one.
|
||||
|
||||
The [credential format reference](../formats.md#credentials), the
|
||||
[`Backend` GoDoc](../../backends.go), the
|
||||
[`ExecutionTargetOverride` GoDoc](../../types.go), and the
|
||||
[authentication integration contract](../integrations/openai-compatible-chat.md#authentication)
|
||||
define the current precedence and availability rules.
|
||||
|
||||
## Profile Inheritance
|
||||
|
||||
YAML profiles can name one parent with `base_profile`; in-memory profiles use
|
||||
`Profile.BaseProfileID`, and `OpenAICompatibleProfileConfig` forwards the same
|
||||
field. A profile can act as an application-owned alias of a built-in or refine
|
||||
selected inherited settings:
|
||||
|
||||
```go
|
||||
promptkit.WithProfiles(promptkit.Profile{
|
||||
ID: "weather-light",
|
||||
BaseProfileID: "deepseek-4-flash",
|
||||
ReasoningEffort: "high",
|
||||
})
|
||||
```
|
||||
|
||||
Base lookup observes the existing source precedence. Chains are linear,
|
||||
cycle-safe, and resolved afresh for ordinary operations. Prepared execution
|
||||
freezes the fully resolved target. The selected leaf ID remains public while
|
||||
effective execution settings reflect the resolved chain.
|
||||
|
||||
See the [profile inheritance format reference](../formats.md#profile-inheritance),
|
||||
the [consumer alias example](../consumers/pkg-promptkit.md#alias-a-built-in-profile),
|
||||
and the [`Profile` GoDoc](../../types.go) for exact merge and validation
|
||||
behavior.
|
||||
|
||||
## Public API Changes
|
||||
|
||||
The release adds:
|
||||
|
||||
- `BackendRakestrawHome`;
|
||||
- `GenerationError`, including `StatusCode`, `ProviderCode`, `ProviderType`,
|
||||
`ProviderMessage`, `Error`, `GoString`, and `Unwrap`;
|
||||
- `Profile.BaseProfileID`; and
|
||||
- `OpenAICompatibleProfileConfig.BaseProfileID`.
|
||||
|
||||
No public declaration was removed.
|
||||
|
||||
## Consumer Action
|
||||
|
||||
- Remove a manual backend registration whose ID is exactly `rakestrawhome`.
|
||||
- Convert positional `Profile` or `OpenAICompatibleProfileConfig` literals to
|
||||
keyed literals.
|
||||
- Set `Profile.APIKeyRequired` where a missing credential must fail locally
|
||||
instead of reaching the provider unauthenticated.
|
||||
- Treat `GenerationError` provider fields as untrusted and potentially
|
||||
sensitive when adopting the new accessors.
|
||||
- Run consumer ordinary and race-enabled tests after updating the module.
|
||||
@@ -40,12 +40,11 @@ consumers.
|
||||
|
||||
### Public bounded output repair
|
||||
|
||||
After the codebase-audit remediations are complete, Promptkit should make its
|
||||
bounded output-repair capability available through the public engine. A
|
||||
consumer should be able to request a limited number of corrective generation
|
||||
attempts when JSON or JSON Schema output fails content validation, without
|
||||
having to reproduce Promptkit's generation, validation, capacity, and result-
|
||||
accounting orchestration.
|
||||
Promptkit should make its bounded output-repair capability available through
|
||||
the public engine. A consumer should be able to request a limited number of
|
||||
corrective generation attempts when JSON or JSON Schema output fails content
|
||||
validation, without having to reproduce Promptkit's generation, validation,
|
||||
capacity, and result-accounting orchestration.
|
||||
|
||||
- Repair is validation recovery, not a general provider retry, failover, or
|
||||
backoff policy. Transport failures, cancellation, and operational schema or
|
||||
@@ -62,10 +61,6 @@ accounting orchestration.
|
||||
- Ordinary and prepared execution should expose coherent behavior, including
|
||||
cancellation, frozen prepared state, error identity, and capacity lifetime.
|
||||
|
||||
Select this work only after the accepted audit findings affecting shared
|
||||
execution invariants, validation, orchestration, transport, and repair
|
||||
internals have been remediated.
|
||||
|
||||
## Entry Format
|
||||
|
||||
Use a short heading followed by a concise summary. Add focused bullets when
|
||||
|
||||
@@ -1,332 +0,0 @@
|
||||
# Optional API-Key Environment Implementation Plan
|
||||
|
||||
## Purpose
|
||||
|
||||
Change Promptkit's credential handling so an effective `APIKeyEnv` names an
|
||||
optional credential source rather than implicitly requiring that environment
|
||||
variable to be populated. When neither a direct key nor a populated optional
|
||||
environment variable is available, the built-in OpenAI-compatible client must
|
||||
omit the `Authorization` header and let the upstream provider accept or reject
|
||||
the unauthenticated request.
|
||||
|
||||
This document owns implementation sequencing for that change. Follow the
|
||||
architecture, documentation, and testing policies under [`docs/policy/`](../policy/)
|
||||
and the task-specific reading guide in [`docs/development.md`](../development.md)
|
||||
throughout the work.
|
||||
|
||||
## Target Outcome
|
||||
|
||||
For ordinary and prepared execution:
|
||||
|
||||
1. a nonblank direct `RunRequest.APIKey` remains the highest-precedence
|
||||
credential and produces `Authorization: Bearer <key>`;
|
||||
2. otherwise, a nonblank effective `APIKeyEnv` is read when generation begins;
|
||||
3. a nonblank trimmed environment value produces the bearer header;
|
||||
4. an absent, empty, or whitespace-only optional environment value produces no
|
||||
`Authorization` header and does not prevent preparation or execution; and
|
||||
5. the provider response follows the normal success or error path, including
|
||||
public `GenerationError` conversion for a non-2xx response.
|
||||
|
||||
An explicitly required credential retains local validation. `APIKeyRequired`
|
||||
continues to require either a nonblank direct key or a populated environment
|
||||
variable explicitly selected by the request. Missing required credentials
|
||||
fail before provider transport.
|
||||
|
||||
## Fixed Decisions
|
||||
|
||||
- `APIKeyEnv` is a lookup location, not an assertion that authentication is
|
||||
required. This applies whether the name comes from a built-in backend, a
|
||||
consumer backend, a file profile, or a request override.
|
||||
- An environment value is usable only after `strings.TrimSpace`; unset, empty,
|
||||
and whitespace-only values are equivalent and cause header omission when
|
||||
optional.
|
||||
- Direct keys retain precedence. When a direct key is nonblank, Promptkit does
|
||||
not need the environment value and must not let a missing environment value
|
||||
block the request.
|
||||
- `APIKeyRequired` remains the only existing explicit local requirement flag.
|
||||
Do not add a backend field, YAML field, request field, or new public API.
|
||||
- With `APIKeyRequired` true:
|
||||
- a nonblank direct key satisfies the requirement;
|
||||
- an explicitly selected nonblank `APIKeyEnv` satisfies it only when that
|
||||
variable currently contains a nonblank value;
|
||||
- no selected environment name retains the existing required-key failure;
|
||||
and
|
||||
- a selected but unset environment retains `ErrAPIKeyEnvMissing` and
|
||||
`ErrInvalidRequest` at the public boundary.
|
||||
- `ErrAPIKeyEnvMissing` remains exported for compatibility but is narrowed to
|
||||
a missing environment credential in an explicitly required flow. Optional
|
||||
missing environment values do not return it.
|
||||
- Optional credential availability is not frozen during preparation. The
|
||||
built-in client reads the selected environment variable for each generation,
|
||||
including repair generation. Prepared execution therefore uses the value
|
||||
visible when it runs.
|
||||
- Required prepared execution continues to recheck credential availability
|
||||
before admission. Optional prepared execution proceeds even if the variable
|
||||
becomes unset after preparation.
|
||||
- The built-in transport sets no `Authorization` header at all when no usable
|
||||
credential exists. It must not send `Bearer ` with an empty value.
|
||||
- Injected `LLMClient` behavior is not redefined. Promptkit stops rejecting an
|
||||
optional missing environment before the injected client is called and
|
||||
continues to pass the effective target through the public adapter without
|
||||
resolving or exposing a secret on the client's behalf.
|
||||
- Upstream authentication policy remains upstream. Promptkit adds no provider-
|
||||
specific authentication rules, retry behavior, status mapping, or special
|
||||
handling beyond the existing structured non-2xx error path.
|
||||
|
||||
## Execution Rules
|
||||
|
||||
- Complete the stages in numerical order. Each stage is scoped for one coding-
|
||||
agent prompt and must finish its focused tests before the next stage begins.
|
||||
- Treat all stages as one behavior change. Intermediate stages intentionally
|
||||
leave the use-case and transport policies temporarily different; do not tag,
|
||||
release, or claim the change is complete until Stage 3 passes.
|
||||
- At the start of each stage, inspect the working tree and preserve all
|
||||
unrelated changes. In particular, merge rather than overwrite any existing
|
||||
edits in files touched by this plan.
|
||||
- Use controlled transports, local servers, and `t.Setenv`; never contact a
|
||||
live provider or depend on the developer machine's credentials.
|
||||
- Update existing tests whose asserted policy is changing instead of retaining
|
||||
contradictory tests or adding duplicate coverage under new names.
|
||||
- Do not add dependencies, commit, tag, push, or edit release documentation
|
||||
unless separately instructed.
|
||||
- Current-state prose documentation changes only in Stage 3, after the runtime
|
||||
behavior exists. GoDoc changes alongside the public contract in that stage.
|
||||
|
||||
## Stage 1: Make Use-Case Credential Validation Requirement-Aware
|
||||
|
||||
### Objective
|
||||
|
||||
Stop preparation and execution orchestration from treating every named
|
||||
environment variable as required, while preserving the explicit
|
||||
`APIKeyRequired` and prepared-execution contracts.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Update `validateAPIKey` in `internal/usecase/runner.go` with this policy:
|
||||
- return success immediately for a nonblank direct key;
|
||||
- return success when `apiKeyRequired` is false, regardless of whether
|
||||
`apiKeyEnv` is blank, populated, or missing;
|
||||
- when `apiKeyRequired` is true and the trimmed environment name is blank,
|
||||
return the existing `ErrAPIKeyRequired`;
|
||||
- when `apiKeyRequired` is true and the selected environment variable is
|
||||
unset, empty, or whitespace-only, return the existing wrapped
|
||||
`ErrAPIKeyEnvMissing` diagnostic naming the variable; and
|
||||
- otherwise return success.
|
||||
2. Keep the calls from preparation and `RunPrepared` in place. Do not move
|
||||
environment lookup into backend/profile resolution, admission, the domain
|
||||
model, or the root facade.
|
||||
3. Do not clear `APIKeyEnv` from effective or prepared metadata merely because
|
||||
its current value is absent. The name remains the configured lookup source;
|
||||
the secret value remains excluded.
|
||||
4. Preserve credential-source precedence and `APIKeyRequired` merge behavior.
|
||||
A request environment override remains capable of satisfying an in-memory
|
||||
profile's explicit requirement when its value is populated.
|
||||
|
||||
### Tests
|
||||
|
||||
Update the narrow `internal/usecase` behavior owners rather than adding tests
|
||||
for the private helper itself:
|
||||
|
||||
1. Replace `TestRunnerRunAPIKeyEnvMissingEnvironmentValueFailsClearly` with a
|
||||
test proving an unset optional profile `APIKeyEnv` reaches the injected
|
||||
model client and completes successfully. Explicitly set the named variable
|
||||
to an empty value so the test is independent of the process environment.
|
||||
2. Preserve the existing populated-environment and direct-key-precedence tests.
|
||||
3. Preserve `TestRunnerPrepareAPIKeyRequiredFailsWithoutDirectKey` and the
|
||||
existing direct-key success coverage.
|
||||
4. Revise the prepared credential test matrix to prove both distinct cases:
|
||||
- an optional environment value that disappears after
|
||||
`PrepareExecution` does not prevent admission and generation; and
|
||||
- an `APIKeyRequired` profile using an explicit request `APIKeyEnv` override
|
||||
still fails with `ErrInvalidRequest` and `ErrAPIKeyEnvMissing` if that
|
||||
value disappears before `RunPrepared`, before admission or generation.
|
||||
5. Keep prepared handles single-use in both outcomes and avoid duplicating
|
||||
unrelated preparation, validation, or admission assertions.
|
||||
|
||||
### Verification
|
||||
|
||||
```sh
|
||||
go test ./internal/usecase -run 'APIKey|Credential|Prepared'
|
||||
go test ./internal/usecase
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Stage 1 is complete when optional missing environment values no longer block
|
||||
the use-case layer and explicit requirements retain their prior identities and
|
||||
timing.
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Stage 2: Omit Authentication in the Built-In Client When Optional Keys Are Missing
|
||||
|
||||
### Objective
|
||||
|
||||
Make the OpenAI-compatible transport implement the optional lookup contract at
|
||||
the boundary that owns the outbound `Authorization` header.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Refactor the authentication branch in
|
||||
`internal/llm/openai_compatible_client.go` so it resolves one credential in
|
||||
this order:
|
||||
- trimmed `req.Target.APIKey`; then
|
||||
- the trimmed value of the environment variable named by the trimmed
|
||||
`req.Target.APIKeyEnv`.
|
||||
2. Set `Authorization` to `Bearer <credential>` only when the resolved value is
|
||||
nonblank. If the direct key is blank and the optional environment lookup is
|
||||
absent, empty, or whitespace-only, leave the header unset and continue to
|
||||
`http.Client.Do`.
|
||||
3. Retain defensive enforcement for direct internal callers of the transport:
|
||||
- if `req.Target.APIKeyRequired` is true and no usable source exists, return
|
||||
`ErrInvalidRequest` before transport;
|
||||
- use the existing missing-environment diagnostic when a nonblank
|
||||
environment name was selected; and
|
||||
- use a concise required-key diagnostic when no environment name exists.
|
||||
Do not add an internal or public error type for this branch.
|
||||
4. Preserve request construction, endpoint validation, timeout behavior,
|
||||
response-body ownership, and direct-key redaction. Never serialize
|
||||
`APIKeyEnv`, `APIKeyRequired`, or a credential into the JSON request body.
|
||||
5. Allow an unauthenticated upstream non-2xx response to flow through the
|
||||
existing `ProviderHTTPError` and public `GenerationError` machinery without
|
||||
adding a credential-specific translation.
|
||||
|
||||
### Tests
|
||||
|
||||
Update `TestOpenAICompatibleClientAuthentication` in
|
||||
`internal/llm/openai_compatible_client_test.go` as the transport contract owner.
|
||||
Its table should cover only the meaningful credential states:
|
||||
|
||||
- a direct key takes precedence over a populated environment value;
|
||||
- a populated environment value supplies the bearer header when no direct key
|
||||
is present;
|
||||
- no configured source sends the request without `Authorization`;
|
||||
- a named but unset optional environment sends the request without
|
||||
`Authorization`;
|
||||
- a named but unset required environment fails with `ErrInvalidRequest` before
|
||||
transport; and
|
||||
- a required target with no source also fails before transport.
|
||||
|
||||
Use exact header assertions because presence or absence of `Authorization` is
|
||||
the wire contract. Do not repeat endpoint, body, timeout, or response-error
|
||||
matrices in this test.
|
||||
|
||||
### Verification
|
||||
|
||||
```sh
|
||||
go test ./internal/llm -run TestOpenAICompatibleClientAuthentication
|
||||
go test ./internal/llm
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Stage 2 is complete when the built-in transport sends unauthenticated requests
|
||||
for optional missing sources, still enforces explicit requirements, and all
|
||||
existing provider-response behavior remains green.
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Stage 3: Align the Public Contract, Durable Documentation, and Full Validation
|
||||
|
||||
### Objective
|
||||
|
||||
Prove the assembled consumer workflow, update canonical contract owners, and
|
||||
complete repository-wide validation.
|
||||
|
||||
### Public Contract And Tests
|
||||
|
||||
1. Update the GoDoc for `ErrAPIKeyEnvMissing` in `engine.go` so it applies only
|
||||
when an explicitly required credential names an unset or empty environment
|
||||
variable. Retain the exported value and its `ErrInvalidRequest` identity.
|
||||
2. Update credential GoDoc where the exact public semantics are exposed:
|
||||
- `Backend.APIKeyEnv` in `backends.go`;
|
||||
- `ExecutionTarget.APIKeyEnv` and
|
||||
`ExecutionTargetOverride.APIKeyEnv` in `types.go`; and
|
||||
- any nearby `APIKeyRequired` wording that would otherwise imply every
|
||||
named environment source is mandatory.
|
||||
State that optional missing values cause the built-in client to omit
|
||||
`Authorization`; do not promise that injected clients resolve environment
|
||||
variables identically.
|
||||
3. Replace `TestMissingCredentialsFailClearlyWhenProfileRequiresAuth` in
|
||||
`engine_test.go`, whose old assertion is intentionally obsolete, with one
|
||||
focused external-package contract test. Assemble a real engine with a
|
||||
controlled HTTP client or local server, select a backend or file profile
|
||||
whose `APIKeyEnv` is explicitly empty in the process environment, and have
|
||||
the controlled upstream return a structured authentication failure. Assert:
|
||||
- the request reaches upstream;
|
||||
- `Authorization` is absent;
|
||||
- the result is nil;
|
||||
- the error does not match `ErrInvalidRequest` or
|
||||
`ErrAPIKeyEnvMissing`; and
|
||||
- the error matches `ErrLLMGenerate` and is discoverable as a
|
||||
`*GenerationError` with the upstream status and representative provider
|
||||
detail.
|
||||
4. Keep this root test representative. The transport authentication table owns
|
||||
the full credential matrix, and the existing generation-error tests own
|
||||
envelope parsing, bounds, and formatting.
|
||||
|
||||
### Documentation
|
||||
|
||||
Update each canonical owner only for its topic:
|
||||
|
||||
1. In `docs/formats.md`, define profile, backend, and request `api_key_env`
|
||||
values as optional lookup sources. In the credentials section, distinguish
|
||||
them from `APIKeyRequired`, document header omission for a missing optional
|
||||
value, retain direct-key precedence, and state that required availability is
|
||||
validated during preparation and rechecked for prepared execution.
|
||||
2. In `docs/integrations/openai-compatible-chat.md`, document the outbound
|
||||
authentication wire behavior: a bearer header is sent only for a usable
|
||||
direct or environment credential; otherwise the header is omitted and the
|
||||
provider response is handled normally.
|
||||
3. In `docs/internal/llm.md`, update the internal flow and failure categories to
|
||||
distinguish optional omission from explicit required-key rejection. Preserve
|
||||
any unrelated edits already present in this file.
|
||||
4. In `docs/consumers/pkg-promptkit.md`, clarify near profile inspection or
|
||||
credential guidance that a reported `APIKeyEnv` is a configured optional
|
||||
source, while `APIKeyRequired` is the explicit local requirement. Link to
|
||||
the exact public GoDoc or format reference rather than reproducing the full
|
||||
precedence contract.
|
||||
5. Do not change architecture policy, backend IDs/default names, release notes,
|
||||
the README, or examples unless implementation uncovers a concrete
|
||||
inaccurate current-state statement in one of those owners.
|
||||
|
||||
### Final Validation
|
||||
|
||||
Run the complete maintainer workflow from
|
||||
[`docs/development.md#maintainer-validation`](../development.md#maintainer-validation):
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
go build ./...
|
||||
go run ./examples/go-library/prepare
|
||||
go run ./examples/go-library/run
|
||||
```
|
||||
|
||||
Also run the documented tracked-Go formatting check, local Markdown-link
|
||||
validation, repository-hygiene checks, ignored-file check, credential scan, and
|
||||
`git diff --check`. Review the complete diff and confirm that:
|
||||
|
||||
- optional absent, empty, and whitespace-only environment values omit
|
||||
authentication and reach transport;
|
||||
- direct and populated environment credentials still produce the correct
|
||||
bearer header;
|
||||
- explicit required flows still fail locally with the intended identities;
|
||||
- ordinary, prepared, built-in, injected-client, and repair paths follow their
|
||||
stated ownership boundaries;
|
||||
- upstream non-2xx responses remain structured generation errors rather than
|
||||
local credential errors;
|
||||
- no credential value is serialized, retained in public metadata, or exposed
|
||||
by formatting; and
|
||||
- durable documentation describes only the now-implemented behavior with one
|
||||
canonical owner per exact contract.
|
||||
|
||||
Stage 3 is complete when the public workflow, documentation, and full
|
||||
maintainer validation all match the target outcome.
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. Optional environment lookup, explicit requirement behavior, precedence,
|
||||
prepared-execution timing, outbound header semantics, error compatibility,
|
||||
documentation ownership, and test boundaries are fixed by this plan.
|
||||
@@ -1,260 +0,0 @@
|
||||
# Structured Generation Errors
|
||||
|
||||
## Purpose
|
||||
|
||||
Promptkit should give downstream applications actionable, machine-readable
|
||||
details when the built-in OpenAI-compatible client receives a non-success HTTP
|
||||
response. Today the client reports only the status code and discards the
|
||||
provider response body. This makes ordinary configuration failures, such as an
|
||||
unsupported strict JSON Schema keyword, unnecessarily difficult to diagnose.
|
||||
|
||||
This feature supplies bounded facts about the provider response. It does not
|
||||
make retry, presentation, or logging decisions for consumers.
|
||||
|
||||
## Target End State
|
||||
|
||||
Every non-2xx response received by Promptkit's built-in OpenAI-compatible
|
||||
client becomes a public typed generation error. A consumer can use
|
||||
`errors.As` to obtain the HTTP status and any safely extracted provider fields,
|
||||
and `errors.Is` continues to match `ErrLLMGenerate`.
|
||||
|
||||
The typed contract is available from both `Run` and `RunPrepared`. It is not
|
||||
produced during preparation, which performs no model request. Successful
|
||||
responses, transport failures before a response is received, cancellation,
|
||||
capacity failures, validation failures, and nil responses from injected model
|
||||
clients retain their existing categories and behavior.
|
||||
|
||||
An unusable response body never hides the known HTTP status. Empty, malformed,
|
||||
unrecognized, unreadable, or oversized bodies therefore produce the same typed
|
||||
error with status-only detail rather than falling back to an unstructured
|
||||
error or becoming a malformed-success response.
|
||||
|
||||
## Public Contract
|
||||
|
||||
The root package exposes an immutable `GenerationError` type with unexported
|
||||
state and these read-only accessors:
|
||||
|
||||
- `StatusCode() int` returns the received HTTP status code;
|
||||
- `ProviderCode() string` returns a normalized provider code, when present;
|
||||
- `ProviderType() string` returns a normalized provider error type, when
|
||||
present; and
|
||||
- `ProviderMessage() string` returns the bounded normalized diagnostic message,
|
||||
when present.
|
||||
|
||||
The engine returns a `*GenerationError`, so the idiomatic inspection form is:
|
||||
|
||||
```go
|
||||
var generationErr *promptkit.GenerationError
|
||||
if errors.As(err, &generationErr) {
|
||||
status := generationErr.StatusCode()
|
||||
message := generationErr.ProviderMessage()
|
||||
_, _ = status, message
|
||||
}
|
||||
```
|
||||
|
||||
There is no public constructor or mutation API. The type implements `error`,
|
||||
unwraps to `ErrLLMGenerate`, and provides safe ordinary and Go-syntax
|
||||
formatting. `Error()` and `GoString()` include the HTTP status but no provider-
|
||||
controlled code, type, or message. Consumers must use the accessors
|
||||
deliberately when they want provider details and must not classify failures by
|
||||
matching error text.
|
||||
|
||||
The zero value and a nil `*GenerationError` receiver are safe: accessors return
|
||||
zero or empty values, formatting returns a generic redacted generation-failure
|
||||
description, and unwrapping still identifies `ErrLLMGenerate`. Engine-produced
|
||||
values always have the non-2xx status received from the provider. The type has
|
||||
no stable JSON representation.
|
||||
|
||||
All provider-derived strings remain untrusted even after normalization. GoDoc
|
||||
must warn consumers that provider fields can contain sensitive request or
|
||||
schema fragments and must not be logged, displayed, or returned to another
|
||||
caller without an application-appropriate disclosure policy.
|
||||
|
||||
## Recognized Provider Envelope
|
||||
|
||||
Promptkit recognizes only the conventional OpenAI-compatible top-level error
|
||||
object:
|
||||
|
||||
```json
|
||||
{
|
||||
"error": {
|
||||
"message": "diagnostic text",
|
||||
"type": "invalid_request_error",
|
||||
"code": "unsupported_parameter"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The envelope must be one JSON document followed only by JSON whitespace. The
|
||||
top-level `error` value must be an object. Unknown top-level and error-object
|
||||
fields are ignored. The optional supported fields are interpreted
|
||||
independently:
|
||||
|
||||
- `message` and `type` must be JSON strings;
|
||||
- `code` may be a JSON string or number and is exposed as normalized text;
|
||||
numeric codes retain their validated JSON number text without floating-point
|
||||
coercion; and
|
||||
- `null`, booleans, arrays, objects, or otherwise invalid values are treated as
|
||||
absent for that field.
|
||||
|
||||
An invalid optional field does not discard other valid supported fields. An
|
||||
absent `error` object, malformed or multiply framed JSON, or an object with no
|
||||
usable supported fields simply leaves all provider accessors empty while
|
||||
preserving the typed status error.
|
||||
|
||||
Promptkit does not expose `param`, metadata objects, nested causes, headers, or
|
||||
provider-specific extensions in this feature.
|
||||
|
||||
## Bounded Reading And Normalization
|
||||
|
||||
Non-success bodies have a separate fixed limit of 64 KiB (65,536 bytes). This
|
||||
is intentionally much smaller than the successful completion-body limit while
|
||||
remaining large enough for useful schema diagnostics.
|
||||
|
||||
- A declared `Content-Length` above the limit is rejected without reading the
|
||||
body for detail extraction.
|
||||
- Otherwise Promptkit reads at most one byte beyond the limit so streamed,
|
||||
chunked, and underreported bodies are bounded.
|
||||
- A body over the limit contributes no provider fields; Promptkit does not
|
||||
parse or retain a prefix as though it were a complete envelope.
|
||||
- Read failures likewise discard provider fields while preserving the status.
|
||||
- The response body is closed on every outcome and is not drained beyond the
|
||||
bounded read.
|
||||
|
||||
Extracted strings are converted to valid UTF-8, trimmed, and made single-line:
|
||||
invalid UTF-8 is replaced, and runs of Unicode whitespace, control characters,
|
||||
and formatting controls are replaced with one ASCII space. Empty normalized
|
||||
values are treated as absent.
|
||||
|
||||
Normalized provider codes and types are retained only when they contain at
|
||||
most 256 Unicode code points. Longer values are omitted rather than truncated
|
||||
so consumers never classify on a fabricated partial identifier. A provider
|
||||
message is limited to 4,096 Unicode code points; a longer normalized message is
|
||||
truncated at a code-point boundary with a visible ellipsis inside that limit.
|
||||
The raw response body and pre-normalized strings are never exposed or retained
|
||||
in the public error.
|
||||
|
||||
## Error Propagation And Compatibility
|
||||
|
||||
- Every built-in-client non-2xx response matches `ErrLLMGenerate` and supports
|
||||
`errors.As` to `*GenerationError`, including status-only cases.
|
||||
- The internal model client retains its non-success-status identity for its
|
||||
own package tests. The use-case layer remains provider-neutral and continues
|
||||
to add only its generation category.
|
||||
- The root error boundary converts only the built-in transport's structured
|
||||
status error. It does not parse arbitrary error text, inspect consumer error
|
||||
fields, or fabricate HTTP details for an injected `LLMClient`.
|
||||
- Errors returned by injected clients remain in the chain exactly as today.
|
||||
If an injected client deliberately returns an existing `*GenerationError`,
|
||||
its identity may pass through ordinary wrapping, but Promptkit does not
|
||||
construct or enrich one on that client's behalf.
|
||||
- Existing cancellation and deadline identities, capacity errors, validation
|
||||
behavior, repair behavior, and successful response decoding remain
|
||||
unchanged.
|
||||
- This is an additive public API. Existing consumers that use
|
||||
`errors.Is(err, ErrLLMGenerate)` continue to work; consumers should not rely
|
||||
on the previous rendered wording of non-success errors.
|
||||
|
||||
## Architecture And Ownership
|
||||
|
||||
The provider-envelope parser and bounded body reader belong in `internal/llm`,
|
||||
which owns the OpenAI-compatible transport. The internal transport error owns
|
||||
only normalized status facts and continues to match the package's existing
|
||||
non-success-status sentinel.
|
||||
|
||||
The use-case package does not gain HTTP DTOs, status policy, or a provider-
|
||||
specific branch. Its existing wrapping carries the internal error to the root
|
||||
facade. The root error mapper recognizes the internal structured status error
|
||||
and constructs the public `GenerationError` without exposing an internal type
|
||||
or raw cause through public fields. No transport error is added to
|
||||
`internal/domain`.
|
||||
|
||||
The public type and its exact Go semantics are owned by its declaration and
|
||||
GoDoc. The
|
||||
[OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md)
|
||||
owns recognized wire shapes, limits, and observable response behavior. The
|
||||
[internal model-client document](../internal/llm.md) owns implementation flow,
|
||||
internal failure categories, and test ownership. Architecture policy does not
|
||||
need a new package or dependency rule for this feature.
|
||||
|
||||
## Documentation End State
|
||||
|
||||
Canonical documentation at the target state has these responsibilities:
|
||||
|
||||
- the `GenerationError` declaration and GoDoc define the exact public methods,
|
||||
formatting, unwrapping, zero-value behavior, and trust boundary;
|
||||
- `Engine.Run` and `Engine.RunPrepared` GoDoc identify the typed error without
|
||||
duplicating its accessor contract;
|
||||
- the consumer guide includes one short `errors.As` example and links to the
|
||||
public declaration;
|
||||
- the integration document replaces its status-only description with the
|
||||
bounded envelope contract; and
|
||||
- the internal model-client document describes parsing, conversion ownership,
|
||||
and narrow test owners.
|
||||
|
||||
The architecture policy, framework format reference, and built-in backend
|
||||
catalog do not duplicate this API or wire contract.
|
||||
|
||||
## Verification Expectations
|
||||
|
||||
Verification protects each behavior at its narrowest stable owner:
|
||||
|
||||
- internal model-client tests cover recognized string and numeric codes,
|
||||
independent optional-field handling, unknown fields, empty and malformed
|
||||
envelopes, single-document framing, read failures, declared and streamed
|
||||
size boundaries, body closure, normalization, field limits, and absence of
|
||||
raw provider content from rendered errors;
|
||||
- root error-boundary tests cover conversion to the immutable public type,
|
||||
every accessor, `errors.Is`, `errors.As`, and safe `%v`, `%+v`, and `%#v`
|
||||
formatting;
|
||||
- one representative ordinary run and one prepared run prove that the built-in
|
||||
transport contract crosses the assembled engine boundary, without repeating
|
||||
the complete parser matrix;
|
||||
- existing injected-client tests continue to prove preservation of consumer
|
||||
error identity without fabricated provider details; and
|
||||
- all tests use controlled transports or local servers and never contact a
|
||||
live or paid provider.
|
||||
|
||||
Security limits and their exact boundaries are contractual enough to warrant
|
||||
literal boundary tests. Higher-level tests should remain representative and
|
||||
must not duplicate the internal transport matrix.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A consumer can distinguish an HTTP 400 from other generation failures and
|
||||
deliberately obtain a bounded provider explanation when one is available.
|
||||
- The same typed error remains available through ordinary and prepared
|
||||
execution and still satisfies `errors.Is(err, ErrLLMGenerate)`.
|
||||
- Default and Go-syntax error formatting cannot disclose any provider-derived
|
||||
string or raw response content.
|
||||
- Empty, malformed, unreadable, unrecognized, and oversized bodies preserve a
|
||||
typed status-only error.
|
||||
- No read, retained field, or formatted representation can exceed its stated
|
||||
bound, and the body is closed on every outcome.
|
||||
- Existing success, cancellation, capacity, validation, repair, and injected-
|
||||
client contracts remain unchanged.
|
||||
- Current-state documentation changes only when the implementation exists and
|
||||
follows the repository's canonical ownership policy.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This feature does not add:
|
||||
|
||||
- retryability classification, retry loops, backoff, failover, or routing;
|
||||
- parsing of success bodies as errors or changes to successful-response limits;
|
||||
- provider-specific envelope variants beyond the conventional top-level
|
||||
`error` object;
|
||||
- response headers such as `Retry-After`, raw bodies, request data, endpoints,
|
||||
credentials, schema documents, generated content, or provider metadata;
|
||||
- logging, telemetry, redaction policy for downstream applications, HTTP status
|
||||
mapping for consumer servers, or user-facing presentation;
|
||||
- translation or enrichment of arbitrary injected-client errors; or
|
||||
- a new public package, public constructor, mutable error value, or transport
|
||||
type in the domain model.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The public type direction, accessor surface, formatting and error-chain
|
||||
behavior, envelope scope, normalization, safety limits, fallback behavior,
|
||||
layer ownership, compatibility boundaries, documentation ownership, and test
|
||||
boundaries are fixed by this roadmap.
|
||||
12
engine.go
12
engine.go
@@ -304,10 +304,12 @@ func WithFallbackProfileFS(fsys fs.FS, root string) Option {
|
||||
// WithProfiles configures in-memory profiles that take precedence over
|
||||
// ordinary configured, application fallback, and built-in profiles.
|
||||
//
|
||||
// NewEngine validates and copies every profile. IDs must be unique within one
|
||||
// call. An invalid profile, duplicate ID, or unsupported ExtraParams value
|
||||
// makes construction fail with ErrInvalidConfig. Repeating WithProfiles
|
||||
// replaces the complete earlier in-memory set rather than merging it.
|
||||
// NewEngine locally validates and copies every profile. IDs must be unique
|
||||
// within one call. An invalid local definition, duplicate ID, or unsupported
|
||||
// ExtraParams value makes construction fail with ErrInvalidConfig. A derived
|
||||
// profile's base reference and resolved target completeness are checked when it
|
||||
// is selected or inspected. Repeating WithProfiles replaces the complete
|
||||
// earlier in-memory set rather than merging it.
|
||||
func WithProfiles(profiles ...Profile) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
repo, err := newMemoryProfileRepository(profiles)
|
||||
@@ -458,7 +460,7 @@ func newProfileRepository(profileDir string, options engineOptions) profile.Repo
|
||||
repository = profile.NewOverlayRepository(options.memoryProfiles, repository)
|
||||
}
|
||||
|
||||
return repository
|
||||
return profile.NewResolvingRepository(repository)
|
||||
}
|
||||
|
||||
func fileSource(name string) (fs.FS, string, error) {
|
||||
|
||||
@@ -2018,6 +2018,25 @@ func TestWithProfilesRejectsDuplicateIDs(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithProfilesAcceptsDerivedDefinitionWithoutTargetFields(t *testing.T) {
|
||||
_, err := promptkit.NewEngine(promptkit.Config{PromptDir: frameworkPromptDir},
|
||||
promptkit.WithProfiles(promptkit.Profile{
|
||||
ID: " derived-profile ",
|
||||
BaseProfileID: " base-profile ",
|
||||
}),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("derived profile should be accepted during construction: %v", err)
|
||||
}
|
||||
|
||||
_, err = promptkit.NewEngine(promptkit.Config{PromptDir: frameworkPromptDir},
|
||||
promptkit.WithProfiles(promptkit.Profile{ID: "standalone-profile"}),
|
||||
)
|
||||
if !errors.Is(err, promptkit.ErrInvalidConfig) {
|
||||
t.Fatalf("standalone incomplete profile error = %v, want ErrInvalidConfig", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithProfilesRejectsInvalidExecutionSettings(t *testing.T) {
|
||||
type testCase struct {
|
||||
name string
|
||||
@@ -2170,6 +2189,7 @@ func TestOpenAICompatibleProfileMapsEveryField(t *testing.T) {
|
||||
extraParams := map[string]any{"provider_option": "distinct-extra-params"}
|
||||
got := promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||
ID: "distinct-id",
|
||||
BaseProfileID: "distinct-base",
|
||||
BackendID: "distinct-backend",
|
||||
Endpoint: "https://distinct.example/v1",
|
||||
Model: "distinct-model",
|
||||
@@ -2184,6 +2204,7 @@ func TestOpenAICompatibleProfileMapsEveryField(t *testing.T) {
|
||||
})
|
||||
want := promptkit.Profile{
|
||||
ID: "distinct-id",
|
||||
BaseProfileID: "distinct-base",
|
||||
BackendID: "distinct-backend",
|
||||
Endpoint: "https://distinct.example/v1",
|
||||
Model: "distinct-model",
|
||||
|
||||
@@ -192,6 +192,7 @@ type BackendCapacityPolicy struct {
|
||||
// ExecutionProfile describes how and where to execute a model.
|
||||
type ExecutionProfile struct {
|
||||
ID string `yaml:"id"`
|
||||
BaseProfileID string `yaml:"base_profile"`
|
||||
BackendID string `yaml:"backend"`
|
||||
Endpoint string `yaml:"endpoint"`
|
||||
Model string `yaml:"model"`
|
||||
|
||||
47
internal/profile/definition.go
Normal file
47
internal/profile/definition.go
Normal file
@@ -0,0 +1,47 @@
|
||||
package profile
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
|
||||
)
|
||||
|
||||
// NormalizeAndValidateDefinition normalizes and validates one source-local
|
||||
// profile definition without resolving a base profile.
|
||||
func NormalizeAndValidateDefinition(profile *domain.ExecutionProfile) error {
|
||||
if profile == nil {
|
||||
return errors.New("profile is required")
|
||||
}
|
||||
|
||||
profile.ID = strings.TrimSpace(profile.ID)
|
||||
profile.BaseProfileID = strings.TrimSpace(profile.BaseProfileID)
|
||||
profile.BackendID = strings.TrimSpace(profile.BackendID)
|
||||
profile.Endpoint = strings.TrimSpace(profile.Endpoint)
|
||||
|
||||
if profile.ID == "" {
|
||||
return errors.New("id is required")
|
||||
}
|
||||
if profile.Endpoint != "" {
|
||||
endpoint, err := domain.NormalizeOpenAICompatibleBaseEndpoint(profile.Endpoint)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
profile.Endpoint = endpoint
|
||||
}
|
||||
if profile.BaseProfileID == "" {
|
||||
if profile.BackendID == "" && profile.Endpoint == "" {
|
||||
return errors.New("backend or endpoint is required")
|
||||
}
|
||||
if strings.TrimSpace(profile.Model) == "" {
|
||||
return errors.New("model is required")
|
||||
}
|
||||
}
|
||||
|
||||
return domain.ValidateExecutionTargetSettings(domain.ExecutionTarget{
|
||||
Temperature: profile.Temperature,
|
||||
MaxTokens: profile.MaxTokens,
|
||||
TopP: profile.TopP,
|
||||
TimeoutSeconds: profile.TimeoutSeconds,
|
||||
})
|
||||
}
|
||||
@@ -127,12 +127,11 @@ func loadProfile(ctx context.Context, fsys fs.FS, root string, id string) (*doma
|
||||
if prof.ID != id {
|
||||
continue
|
||||
}
|
||||
prof.BackendID = strings.TrimSpace(prof.BackendID)
|
||||
prof.ExtraParams, err = jsonvalue.CopyMap(prof.ExtraParams)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, relPath, err)
|
||||
}
|
||||
if err := normalizeAndValidateProfile(prof); err != nil {
|
||||
if err := NormalizeAndValidateDefinition(prof); err != nil {
|
||||
if errors.Is(err, ErrRawAPIKeyNotAllowed) {
|
||||
return nil, fmt.Errorf("%w: %s", err, relPath)
|
||||
}
|
||||
@@ -255,30 +254,3 @@ func requireYAMLStreamEnd(decoder *yaml.Decoder) error {
|
||||
}
|
||||
return errors.New("profile file must contain exactly one YAML document")
|
||||
}
|
||||
|
||||
func normalizeAndValidateProfile(p *domain.ExecutionProfile) error {
|
||||
if strings.TrimSpace(p.ID) == "" {
|
||||
return errors.New("id is required")
|
||||
}
|
||||
p.Endpoint = strings.TrimSpace(p.Endpoint)
|
||||
if strings.TrimSpace(p.BackendID) == "" && p.Endpoint == "" {
|
||||
return errors.New("backend or endpoint is required")
|
||||
}
|
||||
if p.Endpoint != "" {
|
||||
endpoint, err := domain.NormalizeOpenAICompatibleBaseEndpoint(p.Endpoint)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
p.Endpoint = endpoint
|
||||
}
|
||||
if strings.TrimSpace(p.Model) == "" {
|
||||
return errors.New("model is required")
|
||||
}
|
||||
|
||||
return domain.ValidateExecutionTargetSettings(domain.ExecutionTarget{
|
||||
Temperature: p.Temperature,
|
||||
MaxTokens: p.MaxTokens,
|
||||
TopP: p.TopP,
|
||||
TimeoutSeconds: p.TimeoutSeconds,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -858,6 +858,107 @@ top_p: .inf
|
||||
})
|
||||
}
|
||||
|
||||
func TestProfileRepositoriesValidateDerivedDefinitions(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
files map[string]string
|
||||
wantError error
|
||||
wantBaseID string
|
||||
wantProfile bool
|
||||
}{
|
||||
{
|
||||
name: "alias is locally valid and normalizes base id",
|
||||
files: map[string]string{"alias.yaml": `
|
||||
id: selected-profile
|
||||
base_profile: " base-profile "
|
||||
`},
|
||||
wantBaseID: "base-profile",
|
||||
wantProfile: true,
|
||||
},
|
||||
{
|
||||
name: "derived endpoint remains valid",
|
||||
files: map[string]string{"invalid.yaml": `
|
||||
id: selected-profile
|
||||
base_profile: base-profile
|
||||
endpoint: /v1
|
||||
`},
|
||||
wantError: ErrInvalidProfile,
|
||||
},
|
||||
{
|
||||
name: "derived settings remain valid",
|
||||
files: map[string]string{"invalid.yaml": `
|
||||
id: selected-profile
|
||||
base_profile: base-profile
|
||||
top_p: 1.1
|
||||
`},
|
||||
wantError: ErrInvalidProfile,
|
||||
},
|
||||
{
|
||||
name: "derived extra params remain valid",
|
||||
files: map[string]string{"invalid.yaml": `
|
||||
id: selected-profile
|
||||
base_profile: base-profile
|
||||
extra_params:
|
||||
timestamp: 2026-08-11T12:34:56Z
|
||||
`},
|
||||
wantError: ErrInvalidProfile,
|
||||
},
|
||||
{
|
||||
name: "derived raw key remains prohibited",
|
||||
files: map[string]string{"invalid.yaml": `
|
||||
id: selected-profile
|
||||
base_profile: base-profile
|
||||
api_key: secret
|
||||
`},
|
||||
wantError: ErrRawAPIKeyNotAllowed,
|
||||
},
|
||||
{
|
||||
name: "derived duplicate id remains invalid",
|
||||
files: map[string]string{
|
||||
"first.yaml": "id: selected-profile\nbase_profile: first-base\n",
|
||||
"second.yaml": "id: selected-profile\nbase_profile: second-base\n",
|
||||
},
|
||||
wantError: ErrInvalidProfile,
|
||||
},
|
||||
{
|
||||
name: "derived extra document remains invalid",
|
||||
files: map[string]string{"invalid.yaml": `
|
||||
id: selected-profile
|
||||
base_profile: base-profile
|
||||
---
|
||||
id: other
|
||||
`},
|
||||
wantError: ErrInvalidYAML,
|
||||
},
|
||||
{
|
||||
name: "standalone profile remains complete",
|
||||
files: map[string]string{"invalid.yaml": "id: selected-profile\n"},
|
||||
wantError: ErrInvalidProfile,
|
||||
},
|
||||
}
|
||||
|
||||
for _, source := range profileRepositorySources() {
|
||||
for _, tc := range tests {
|
||||
t.Run(source.name+"/"+tc.name, func(t *testing.T) {
|
||||
repo := source.newRepository(t, tc.files)
|
||||
got, err := repo.GetProfile(context.Background(), "selected-profile")
|
||||
if tc.wantError != nil {
|
||||
if !errors.Is(err, tc.wantError) {
|
||||
t.Fatalf("error = %v, want %v", err, tc.wantError)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil || !tc.wantProfile {
|
||||
t.Fatalf("profile = %+v, error = %v, want valid derived definition", got, err)
|
||||
}
|
||||
if got.BaseProfileID != tc.wantBaseID {
|
||||
t.Fatalf("BaseProfileID = %q, want %q", got.BaseProfileID, tc.wantBaseID)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestOverlayRepository(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
primaryProfile := &domain.ExecutionProfile{ID: "shared", Endpoint: "http://primary", Model: "primary"}
|
||||
|
||||
160
internal/profile/resolving_repository.go
Normal file
160
internal/profile/resolving_repository.go
Normal file
@@ -0,0 +1,160 @@
|
||||
package profile
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/jsonvalue"
|
||||
)
|
||||
|
||||
const maximumProfileChainLength = 32
|
||||
|
||||
type resolvingRepository struct {
|
||||
source Repository
|
||||
}
|
||||
|
||||
// NewResolvingRepository resolves inherited profile definitions from source.
|
||||
func NewResolvingRepository(source Repository) Repository {
|
||||
return &resolvingRepository{source: source}
|
||||
}
|
||||
|
||||
func (r *resolvingRepository) GetProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) {
|
||||
if r == nil || r.source == nil {
|
||||
return nil, fmt.Errorf("%w: profile repository is required", ErrInvalidProfile)
|
||||
}
|
||||
requestedID := strings.TrimSpace(id)
|
||||
if requestedID == "" {
|
||||
return nil, fmt.Errorf("%w: profile id is required", ErrInvalidProfile)
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
profile, err := r.getRawProfile(ctx, requestedID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if profile == nil {
|
||||
return nil, fmt.Errorf("%w: selected profile %q is nil", ErrInvalidProfile, requestedID)
|
||||
}
|
||||
|
||||
chain := []*domain.ExecutionProfile{profile}
|
||||
chainIDs := []string{requestedID}
|
||||
visited := map[string]struct{}{requestedID: {}}
|
||||
current := profile
|
||||
|
||||
for {
|
||||
baseID := strings.TrimSpace(current.BaseProfileID)
|
||||
if baseID == "" {
|
||||
break
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if _, seen := visited[baseID]; seen {
|
||||
return nil, fmt.Errorf("%w: profile inheritance cycle %s", ErrInvalidProfile, joinProfileChain(chainIDs, baseID))
|
||||
}
|
||||
if len(chain) >= maximumProfileChainLength {
|
||||
return nil, fmt.Errorf("%w: profile inheritance chain exceeds %d profiles: %s", ErrInvalidProfile, maximumProfileChainLength, joinProfileChain(chainIDs, baseID))
|
||||
}
|
||||
|
||||
base, err := r.getRawProfile(ctx, baseID)
|
||||
if err != nil {
|
||||
if errors.Is(err, ErrProfileNotFound) {
|
||||
return nil, fmt.Errorf("%w: base profile %q is missing in chain %s", ErrInvalidProfile, baseID, joinProfileChain(chainIDs, baseID))
|
||||
}
|
||||
return nil, fmt.Errorf("%w: failed to load base profile %q in chain %s: %w", ErrInvalidProfile, baseID, joinProfileChain(chainIDs, baseID), err)
|
||||
}
|
||||
if base == nil {
|
||||
return nil, fmt.Errorf("%w: base profile %q is nil in chain %s", ErrInvalidProfile, baseID, joinProfileChain(chainIDs, baseID))
|
||||
}
|
||||
|
||||
chain = append(chain, base)
|
||||
chainIDs = append(chainIDs, baseID)
|
||||
visited[baseID] = struct{}{}
|
||||
current = base
|
||||
}
|
||||
|
||||
resolved, err := mergeProfileChain(chain)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: resolved profile chain %s: %w", ErrInvalidProfile, strings.Join(chainIDs, " -> "), err)
|
||||
}
|
||||
if err := validateResolvedProfile(resolved); err != nil {
|
||||
return nil, fmt.Errorf("%w: resolved profile chain %s: %w", ErrInvalidProfile, strings.Join(chainIDs, " -> "), err)
|
||||
}
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
func (r *resolvingRepository) getRawProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) {
|
||||
profile, err := r.source.GetProfile(ctx, id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return profile, nil
|
||||
}
|
||||
|
||||
func joinProfileChain(chain []string, next string) string {
|
||||
return strings.Join(append(append([]string(nil), chain...), next), " -> ")
|
||||
}
|
||||
|
||||
func mergeProfileChain(chain []*domain.ExecutionProfile) (*domain.ExecutionProfile, error) {
|
||||
resolved := &domain.ExecutionProfile{ID: chain[0].ID}
|
||||
for index := len(chain) - 1; index >= 0; index-- {
|
||||
definition := chain[index]
|
||||
if strings.TrimSpace(definition.BackendID) != "" {
|
||||
resolved.BackendID = definition.BackendID
|
||||
}
|
||||
if strings.TrimSpace(definition.Endpoint) != "" {
|
||||
resolved.Endpoint = definition.Endpoint
|
||||
}
|
||||
if strings.TrimSpace(definition.Model) != "" {
|
||||
resolved.Model = definition.Model
|
||||
}
|
||||
if definition.Temperature != 0 {
|
||||
resolved.Temperature = definition.Temperature
|
||||
}
|
||||
if definition.MaxTokens != 0 {
|
||||
resolved.MaxTokens = definition.MaxTokens
|
||||
}
|
||||
if definition.TopP != 0 {
|
||||
resolved.TopP = definition.TopP
|
||||
}
|
||||
if definition.TimeoutSeconds != 0 {
|
||||
resolved.TimeoutSeconds = definition.TimeoutSeconds
|
||||
}
|
||||
if strings.TrimSpace(definition.ServiceTier) != "" {
|
||||
resolved.ServiceTier = definition.ServiceTier
|
||||
}
|
||||
if strings.TrimSpace(definition.ReasoningEffort) != "" {
|
||||
resolved.ReasoningEffort = definition.ReasoningEffort
|
||||
}
|
||||
if strings.TrimSpace(definition.APIKeyEnv) != "" {
|
||||
resolved.APIKeyEnv = definition.APIKeyEnv
|
||||
}
|
||||
resolved.APIKeyRequired = resolved.APIKeyRequired || definition.APIKeyRequired
|
||||
if len(definition.ExtraParams) != 0 {
|
||||
extraParams, err := jsonvalue.CopyMap(definition.ExtraParams)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resolved.ExtraParams = extraParams
|
||||
}
|
||||
}
|
||||
resolved.ID = chain[0].ID
|
||||
resolved.BaseProfileID = ""
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
func validateResolvedProfile(profile *domain.ExecutionProfile) error {
|
||||
if profile == nil {
|
||||
return errors.New("resolved profile is required")
|
||||
}
|
||||
profile.BaseProfileID = ""
|
||||
return NormalizeAndValidateDefinition(profile)
|
||||
}
|
||||
418
internal/profile/resolving_repository_test.go
Normal file
418
internal/profile/resolving_repository_test.go
Normal file
@@ -0,0 +1,418 @@
|
||||
package profile
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
|
||||
)
|
||||
|
||||
func TestResolvingRepositoryMergesProfileChain(t *testing.T) {
|
||||
repo := &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
|
||||
"leaf": {
|
||||
ID: "leaf",
|
||||
BaseProfileID: "middle",
|
||||
BackendID: "leaf-backend",
|
||||
TopP: 0.8,
|
||||
TimeoutSeconds: 45,
|
||||
ReasoningEffort: "high",
|
||||
},
|
||||
"middle": {
|
||||
ID: "middle",
|
||||
BaseProfileID: "root",
|
||||
Endpoint: "https://middle.example/v1",
|
||||
Model: "middle-model",
|
||||
MaxTokens: 256,
|
||||
APIKeyEnv: "MIDDLE_API_KEY",
|
||||
APIKeyRequired: true,
|
||||
ExtraParams: map[string]any{"middle": map[string]any{"value": "middle"}},
|
||||
},
|
||||
"root": {
|
||||
ID: "root",
|
||||
BackendID: "root-backend",
|
||||
Endpoint: "https://root.example/v1",
|
||||
Model: "root-model",
|
||||
Temperature: 0.3,
|
||||
ServiceTier: "priority",
|
||||
ExtraParams: map[string]any{"root": "value"},
|
||||
},
|
||||
}}
|
||||
|
||||
got, err := NewResolvingRepository(repo).GetProfile(context.Background(), "leaf")
|
||||
if err != nil {
|
||||
t.Fatalf("resolve profile: %v", err)
|
||||
}
|
||||
want := &domain.ExecutionProfile{
|
||||
ID: "leaf",
|
||||
BackendID: "leaf-backend",
|
||||
Endpoint: "https://middle.example/v1",
|
||||
Model: "middle-model",
|
||||
Temperature: 0.3,
|
||||
MaxTokens: 256,
|
||||
TopP: 0.8,
|
||||
TimeoutSeconds: 45,
|
||||
ServiceTier: "priority",
|
||||
ReasoningEffort: "high",
|
||||
APIKeyEnv: "MIDDLE_API_KEY",
|
||||
APIKeyRequired: true,
|
||||
ExtraParams: map[string]any{"middle": map[string]any{"value": "middle"}},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("resolved profile:\n got %#v\nwant %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvingRepositoryRejectsMissingSourceAndProfileID(t *testing.T) {
|
||||
if _, err := NewResolvingRepository(nil).GetProfile(context.Background(), "profile"); !errors.Is(err, ErrInvalidProfile) {
|
||||
t.Fatalf("nil source error = %v, want ErrInvalidProfile", err)
|
||||
}
|
||||
|
||||
repo := &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{}}
|
||||
if _, err := NewResolvingRepository(repo).GetProfile(context.Background(), " \t "); !errors.Is(err, ErrInvalidProfile) {
|
||||
t.Fatalf("blank id error = %v, want ErrInvalidProfile", err)
|
||||
}
|
||||
if got := repo.callCount(" "); got != 0 {
|
||||
t.Fatalf("blank id looked up source %d times", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvingRepositoryCopiesExtraParams(t *testing.T) {
|
||||
baseParams := map[string]any{"nested": map[string]any{"value": "base"}}
|
||||
repo := &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
|
||||
"child": {ID: "child", BaseProfileID: "base"},
|
||||
"base": {
|
||||
ID: "base",
|
||||
Endpoint: "https://base.example/v1",
|
||||
Model: "model",
|
||||
ExtraParams: baseParams,
|
||||
},
|
||||
}}
|
||||
resolver := NewResolvingRepository(repo)
|
||||
|
||||
first, err := resolver.GetProfile(context.Background(), "child")
|
||||
if err != nil {
|
||||
t.Fatalf("resolve inherited map: %v", err)
|
||||
}
|
||||
first.ExtraParams["nested"].(map[string]any)["value"] = "mutated"
|
||||
second, err := resolver.GetProfile(context.Background(), "child")
|
||||
if err != nil {
|
||||
t.Fatalf("resolve inherited map again: %v", err)
|
||||
}
|
||||
if got := second.ExtraParams["nested"].(map[string]any)["value"]; got != "base" {
|
||||
t.Fatalf("later result retained mutation: %v", got)
|
||||
}
|
||||
if got := baseParams["nested"].(map[string]any)["value"]; got != "base" {
|
||||
t.Fatalf("source map retained mutation: %v", got)
|
||||
}
|
||||
|
||||
repo.set("child", &domain.ExecutionProfile{
|
||||
ID: "child",
|
||||
BaseProfileID: "base",
|
||||
ExtraParams: map[string]any{"child": "replacement"},
|
||||
})
|
||||
replaced, err := resolver.GetProfile(context.Background(), "child")
|
||||
if err != nil {
|
||||
t.Fatalf("resolve replacement map: %v", err)
|
||||
}
|
||||
if !reflect.DeepEqual(replaced.ExtraParams, map[string]any{"child": "replacement"}) {
|
||||
t.Fatalf("extra params = %#v, want complete child replacement", replaced.ExtraParams)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvingRepositoryUsesRawOverlayForEachLookup(t *testing.T) {
|
||||
leafSource := NewFSRepository(profileTestFS(map[string]string{
|
||||
"leaf.yaml": "id: leaf\nbase_profile: base\n",
|
||||
}), ".")
|
||||
fallback := NewFSRepository(profileTestFS(map[string]string{
|
||||
"base.yaml": "id: base\nendpoint: https://fallback.example/v1\nmodel: fallback-model\n",
|
||||
}), ".")
|
||||
overlay := NewOverlayRepository(leafSource, fallback)
|
||||
resolver := NewResolvingRepository(overlay)
|
||||
|
||||
got, err := resolver.GetProfile(context.Background(), "leaf")
|
||||
if err != nil {
|
||||
t.Fatalf("resolve fallback base: %v", err)
|
||||
}
|
||||
if got.Model != "fallback-model" {
|
||||
t.Fatalf("fallback base model = %q", got.Model)
|
||||
}
|
||||
|
||||
shadowing := NewOverlayRepository(NewFSRepository(profileTestFS(map[string]string{
|
||||
"leaf.yaml": "id: leaf\nbase_profile: base\n",
|
||||
"base.yaml": "id: base\nendpoint: https://primary.example/v1\nmodel: primary-model\n",
|
||||
}), "."), fallback)
|
||||
got, err = NewResolvingRepository(shadowing).GetProfile(context.Background(), "leaf")
|
||||
if err != nil {
|
||||
t.Fatalf("resolve shadowed base: %v", err)
|
||||
}
|
||||
if got.Model != "primary-model" || got.Endpoint != "https://primary.example/v1" {
|
||||
t.Fatalf("shadowed base = %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvingRepositoryReportsSafetyAndSourceErrors(t *testing.T) {
|
||||
sourceErr := errors.New("source failure")
|
||||
tests := []struct {
|
||||
name string
|
||||
repo *resolvingTestRepository
|
||||
id string
|
||||
want []error
|
||||
wantNot error
|
||||
contains []string
|
||||
}{
|
||||
{
|
||||
name: "missing selected profile preserves not found",
|
||||
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{}},
|
||||
id: "missing",
|
||||
want: []error{ErrProfileNotFound},
|
||||
wantNot: ErrInvalidProfile,
|
||||
},
|
||||
{
|
||||
name: "missing base is invalid but not not found",
|
||||
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
|
||||
"leaf": {ID: "leaf", BaseProfileID: "missing"},
|
||||
}},
|
||||
id: "leaf",
|
||||
want: []error{ErrInvalidProfile},
|
||||
wantNot: ErrProfileNotFound,
|
||||
contains: []string{"missing", "leaf -> missing"},
|
||||
},
|
||||
{
|
||||
name: "direct cycle",
|
||||
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
|
||||
"a": {ID: "a", BaseProfileID: "a"},
|
||||
}},
|
||||
id: "a",
|
||||
want: []error{ErrInvalidProfile},
|
||||
contains: []string{"a -> a"},
|
||||
},
|
||||
{
|
||||
name: "indirect cycle",
|
||||
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
|
||||
"a": {ID: "a", BaseProfileID: "b"},
|
||||
"b": {ID: "b", BaseProfileID: "c"},
|
||||
"c": {ID: "c", BaseProfileID: "a"},
|
||||
}},
|
||||
id: "a",
|
||||
want: []error{ErrInvalidProfile},
|
||||
contains: []string{"a -> b -> c -> a"},
|
||||
},
|
||||
{
|
||||
name: "nil result",
|
||||
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
|
||||
"leaf": nil,
|
||||
}},
|
||||
id: "leaf",
|
||||
want: []error{ErrInvalidProfile},
|
||||
},
|
||||
{
|
||||
name: "incomplete resolved profile",
|
||||
repo: &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
|
||||
"leaf": {ID: "leaf", BaseProfileID: "base"},
|
||||
"base": {ID: "base", Model: "model"},
|
||||
}},
|
||||
id: "leaf",
|
||||
want: []error{ErrInvalidProfile},
|
||||
},
|
||||
{
|
||||
name: "base source error is retained",
|
||||
repo: &resolvingTestRepository{
|
||||
profiles: map[string]*domain.ExecutionProfile{"leaf": {ID: "leaf", BaseProfileID: "base"}},
|
||||
errors: map[string]error{"base": sourceErr},
|
||||
},
|
||||
id: "leaf",
|
||||
want: []error{ErrInvalidProfile, sourceErr},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
_, err := NewResolvingRepository(tc.repo).GetProfile(context.Background(), tc.id)
|
||||
for _, want := range tc.want {
|
||||
if !errors.Is(err, want) {
|
||||
t.Fatalf("error = %v, want %v", err, want)
|
||||
}
|
||||
}
|
||||
if tc.wantNot != nil && errors.Is(err, tc.wantNot) {
|
||||
t.Fatalf("error = %v, must not match %v", err, tc.wantNot)
|
||||
}
|
||||
for _, fragment := range tc.contains {
|
||||
if !strings.Contains(err.Error(), fragment) {
|
||||
t.Fatalf("error = %v, want %q", err, fragment)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvingRepositoryEnforcesChainLength(t *testing.T) {
|
||||
for _, count := range []int{maximumProfileChainLength, maximumProfileChainLength + 1} {
|
||||
t.Run(fmt.Sprintf("%d profiles", count), func(t *testing.T) {
|
||||
profiles := make(map[string]*domain.ExecutionProfile, count)
|
||||
for index := 1; index <= count; index++ {
|
||||
id := fmt.Sprintf("profile-%d", index)
|
||||
definition := &domain.ExecutionProfile{ID: id}
|
||||
if index == count {
|
||||
definition.Endpoint = "https://root.example/v1"
|
||||
definition.Model = "model"
|
||||
} else {
|
||||
definition.BaseProfileID = fmt.Sprintf("profile-%d", index+1)
|
||||
}
|
||||
profiles[id] = definition
|
||||
}
|
||||
|
||||
got, err := NewResolvingRepository(&resolvingTestRepository{profiles: profiles}).GetProfile(context.Background(), "profile-1")
|
||||
if count == maximumProfileChainLength {
|
||||
if err != nil || got == nil {
|
||||
t.Fatalf("profile = %+v, error = %v, want accepted chain", got, err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if !errors.Is(err, ErrInvalidProfile) {
|
||||
t.Fatalf("error = %v, want ErrInvalidProfile", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvingRepositoryIsFreshAndCancellationAware(t *testing.T) {
|
||||
repo := &resolvingTestRepository{profiles: map[string]*domain.ExecutionProfile{
|
||||
"leaf": {ID: "leaf", BaseProfileID: "base"},
|
||||
"base": {ID: "base", Endpoint: "https://base.example/v1", Model: "first", ExtraParams: map[string]any{"nested": map[string]any{"value": "first"}}},
|
||||
}}
|
||||
resolver := NewResolvingRepository(repo)
|
||||
|
||||
first, err := resolver.GetProfile(context.Background(), "leaf")
|
||||
if err != nil || first.Model != "first" {
|
||||
t.Fatalf("first result=(%+v, %v)", first, err)
|
||||
}
|
||||
repo.set("base", &domain.ExecutionProfile{ID: "base", Endpoint: "https://base.example/v1", Model: "second", ExtraParams: map[string]any{"nested": map[string]any{"value": "second"}}})
|
||||
second, err := resolver.GetProfile(context.Background(), "leaf")
|
||||
if err != nil || second.Model != "second" {
|
||||
t.Fatalf("second result=(%+v, %v)", second, err)
|
||||
}
|
||||
|
||||
canceled, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if _, err := resolver.GetProfile(canceled, "leaf"); !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("canceled lookup error = %v", err)
|
||||
}
|
||||
if got := repo.callCount("leaf"); got != 2 {
|
||||
t.Fatalf("calls after canceled lookup = %d, want 2", got)
|
||||
}
|
||||
|
||||
duringTraversal, cancelDuringTraversal := context.WithCancel(context.Background())
|
||||
repo.afterGet = func(id string) {
|
||||
if id == "leaf" {
|
||||
cancelDuringTraversal()
|
||||
}
|
||||
}
|
||||
if _, err := resolver.GetProfile(duringTraversal, "leaf"); !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("during traversal error = %v", err)
|
||||
}
|
||||
if got := repo.callCount("base"); got != 2 {
|
||||
t.Fatalf("base calls after cancellation = %d, want 2", got)
|
||||
}
|
||||
|
||||
terminalLookup, cancelTerminalLookup := context.WithCancel(context.Background())
|
||||
repo.afterGet = func(id string) {
|
||||
if id == "base" {
|
||||
cancelTerminalLookup()
|
||||
}
|
||||
}
|
||||
if _, err := resolver.GetProfile(terminalLookup, "leaf"); !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("terminal lookup cancellation error = %v", err)
|
||||
}
|
||||
if got := repo.callCount("base"); got != 3 {
|
||||
t.Fatalf("base calls after terminal cancellation = %d, want 3", got)
|
||||
}
|
||||
|
||||
repo.afterGet = nil
|
||||
var wg sync.WaitGroup
|
||||
errors := make(chan error, 8)
|
||||
for index := 0; index < cap(errors); index++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
resolved, err := resolver.GetProfile(context.Background(), "leaf")
|
||||
if err != nil {
|
||||
errors <- err
|
||||
return
|
||||
}
|
||||
resolved.ExtraParams["nested"].(map[string]any)["value"] = "mutated"
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
close(errors)
|
||||
for err := range errors {
|
||||
t.Errorf("concurrent resolution: %v", err)
|
||||
}
|
||||
latest, err := resolver.GetProfile(context.Background(), "leaf")
|
||||
if err != nil || latest.ExtraParams["nested"].(map[string]any)["value"] != "second" {
|
||||
t.Fatalf("latest result=(%+v, %v)", latest, err)
|
||||
}
|
||||
}
|
||||
|
||||
type resolvingTestRepository struct {
|
||||
mu sync.Mutex
|
||||
profiles map[string]*domain.ExecutionProfile
|
||||
errors map[string]error
|
||||
calls map[string]int
|
||||
afterGet func(string)
|
||||
}
|
||||
|
||||
func (r *resolvingTestRepository) GetProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
r.mu.Lock()
|
||||
if r.calls == nil {
|
||||
r.calls = make(map[string]int)
|
||||
}
|
||||
r.calls[id]++
|
||||
err := r.errors[id]
|
||||
profile := r.profiles[id]
|
||||
afterGet := r.afterGet
|
||||
r.mu.Unlock()
|
||||
if afterGet != nil {
|
||||
afterGet(id)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if profile == nil {
|
||||
if _, exists := r.profiles[id]; exists {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, ErrProfileNotFound
|
||||
}
|
||||
copy := *profile
|
||||
return ©, nil
|
||||
}
|
||||
|
||||
func (r *resolvingTestRepository) set(id string, profile *domain.ExecutionProfile) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.profiles[id] = profile
|
||||
}
|
||||
|
||||
func (r *resolvingTestRepository) callCount(id string) int {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return r.calls[id]
|
||||
}
|
||||
|
||||
func profileTestFS(files map[string]string) fs.FS {
|
||||
fsys := make(fstest.MapFS, len(files))
|
||||
for name, content := range files {
|
||||
fsys[name] = profileMapFile(content)
|
||||
}
|
||||
return fsys
|
||||
}
|
||||
243
profile_inheritance_contract_test.go
Normal file
243
profile_inheritance_contract_test.go
Normal file
@@ -0,0 +1,243 @@
|
||||
package promptkit_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io/fs"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
)
|
||||
|
||||
func TestProfileInheritanceBuiltInAliasWorkflow(t *testing.T) {
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||
PromptDir: frameworkPromptDir,
|
||||
SchemaDir: frameworkSchemaDir,
|
||||
}, promptkit.WithProfiles(promptkit.Profile{
|
||||
ID: "weather-light",
|
||||
BaseProfileID: "deepseek-4-flash",
|
||||
ReasoningEffort: "high",
|
||||
TimeoutSeconds: 120,
|
||||
}))
|
||||
if err != nil {
|
||||
t.Fatalf("construct alias engine: %v", err)
|
||||
}
|
||||
|
||||
base, err := engine.InspectProfile(context.Background(), "deepseek-4-flash")
|
||||
if err != nil {
|
||||
t.Fatalf("inspect base: %v", err)
|
||||
}
|
||||
child, err := engine.InspectProfile(context.Background(), "weather-light")
|
||||
if err != nil {
|
||||
t.Fatalf("inspect alias: %v", err)
|
||||
}
|
||||
if child.ProfileID != "weather-light" ||
|
||||
child.EffectiveModelParams.BackendID != base.EffectiveModelParams.BackendID ||
|
||||
child.EffectiveModelParams.Model != base.EffectiveModelParams.Model ||
|
||||
child.EffectiveModelParams.ReasoningEffort != "high" ||
|
||||
child.EffectiveModelParams.TimeoutSeconds != 120 {
|
||||
t.Fatalf("alias inspection = %+v, base = %+v", child, base)
|
||||
}
|
||||
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||
PromptID: frameworkMarkdownSummaryPromptID,
|
||||
ProfileID: "weather-light",
|
||||
Inputs: map[string]promptkit.ArtifactRef{
|
||||
"transcript": promptkit.Inline("Rin opens the gate."),
|
||||
"glossary": promptkit.Inline("gate: A guarded passage."),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("prepare alias: %v", err)
|
||||
}
|
||||
if prepared.SelectedProfileID != "weather-light" {
|
||||
t.Fatalf("SelectedProfileID = %q", prepared.SelectedProfileID)
|
||||
}
|
||||
|
||||
timeout := 15
|
||||
reasoning := "low"
|
||||
overridden, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||
PromptID: frameworkMarkdownSummaryPromptID,
|
||||
ProfileID: "weather-light",
|
||||
Execution: &promptkit.ExecutionTargetOverride{
|
||||
TimeoutSeconds: &timeout,
|
||||
ReasoningEffort: &reasoning,
|
||||
},
|
||||
Inputs: map[string]promptkit.ArtifactRef{
|
||||
"transcript": promptkit.Inline("Rin opens the gate."),
|
||||
"glossary": promptkit.Inline("gate: A guarded passage."),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("prepare override: %v", err)
|
||||
}
|
||||
if overridden.EffectiveModelParams.TimeoutSeconds != timeout ||
|
||||
overridden.EffectiveModelParams.ReasoningEffort != reasoning {
|
||||
t.Fatalf("runtime override target = %+v", overridden.EffectiveModelParams)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProfileInheritanceYAMLAliasOfBuiltIn(t *testing.T) {
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(fstest.MapFS{}, "."),
|
||||
promptkit.WithProfileFS(fstest.MapFS{
|
||||
"alias.yaml": &fstest.MapFile{Data: []byte("id: yaml-alias\nbase_profile: deepseek-4-flash\n")},
|
||||
}, "."),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct YAML alias engine: %v", err)
|
||||
}
|
||||
inspection, err := engine.InspectProfile(context.Background(), "yaml-alias")
|
||||
if err != nil {
|
||||
t.Fatalf("inspect YAML alias: %v", err)
|
||||
}
|
||||
if inspection.ProfileID != "yaml-alias" || inspection.EffectiveModelParams.Model == "" {
|
||||
t.Fatalf("YAML alias inspection = %+v", inspection)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProfileInheritanceRetainsRequiredCredentialBehavior(t *testing.T) {
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(fstest.MapFS{}, "."),
|
||||
promptkit.WithBackend(promptkit.Backend{
|
||||
ID: "credential-backend",
|
||||
Endpoint: "https://credential.example/v1",
|
||||
APIKeyEnv: "OPTIONAL_BACKEND_KEY",
|
||||
}),
|
||||
promptkit.WithProfiles(
|
||||
promptkit.Profile{
|
||||
ID: "credential-base",
|
||||
BackendID: "credential-backend",
|
||||
Model: "model",
|
||||
APIKeyRequired: true,
|
||||
},
|
||||
promptkit.Profile{ID: "credential-child", BaseProfileID: "credential-base"},
|
||||
),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct credential inheritance engine: %v", err)
|
||||
}
|
||||
inspection, err := engine.InspectProfile(context.Background(), "credential-child")
|
||||
if err != nil {
|
||||
t.Fatalf("inspect credential child: %v", err)
|
||||
}
|
||||
if !inspection.APIKeyRequired || inspection.EffectiveModelParams.APIKeyEnv != "" {
|
||||
t.Fatalf("credential inspection = %+v", inspection)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProfileInheritancePreservesPublicErrorIdentities(t *testing.T) {
|
||||
newEngine := func(t *testing.T, profiles fs.FS) *promptkit.Engine {
|
||||
t.Helper()
|
||||
options := []promptkit.Option{promptkit.WithPromptFS(fstest.MapFS{}, ".")}
|
||||
if profiles != nil {
|
||||
options = append(options, promptkit.WithProfileFS(profiles, "."))
|
||||
}
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{}, options...)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
return engine
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
profiles fs.FS
|
||||
profile string
|
||||
contains []string
|
||||
want error
|
||||
wantNot error
|
||||
}{
|
||||
{
|
||||
name: "missing selected profile",
|
||||
profile: "missing",
|
||||
want: promptkit.ErrProfileNotFound,
|
||||
wantNot: promptkit.ErrProfileLoad,
|
||||
},
|
||||
{
|
||||
name: "missing base",
|
||||
profiles: fstest.MapFS{
|
||||
"child.yaml": &fstest.MapFile{Data: []byte("id: child\nbase_profile: missing\n")},
|
||||
},
|
||||
profile: "child",
|
||||
contains: []string{"child", "missing"},
|
||||
want: promptkit.ErrProfileLoad,
|
||||
wantNot: promptkit.ErrProfileNotFound,
|
||||
},
|
||||
{
|
||||
name: "cycle",
|
||||
profiles: fstest.MapFS{
|
||||
"a.yaml": &fstest.MapFile{Data: []byte("id: a\nbase_profile: b\n")},
|
||||
"b.yaml": &fstest.MapFile{Data: []byte("id: b\nbase_profile: a\n")},
|
||||
},
|
||||
profile: "a",
|
||||
contains: []string{"a", "b"},
|
||||
want: promptkit.ErrProfileLoad,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
result, err := newEngine(t, tc.profiles).InspectProfile(context.Background(), tc.profile)
|
||||
if result != nil || !errors.Is(err, tc.want) || (tc.wantNot != nil && errors.Is(err, tc.wantNot)) {
|
||||
t.Fatalf("inspection=(%+v, %v), want %v without %v", result, err, tc.want, tc.wantNot)
|
||||
}
|
||||
for _, fragment := range tc.contains {
|
||||
if !strings.Contains(err.Error(), fragment) {
|
||||
t.Fatalf("error = %v, want %q", err, fragment)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestProfileInheritanceFreezesPreparedExecution(t *testing.T) {
|
||||
profiles := &mutableInheritanceProfileFS{files: fstest.MapFS{
|
||||
"child.yaml": &fstest.MapFile{Data: []byte("id: child\nbase_profile: base\n")},
|
||||
"base.yaml": &fstest.MapFile{Data: []byte("id: base\nendpoint: https://base.example/v1\nmodel: first-model\n")},
|
||||
}}
|
||||
client := &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}}
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(contractPromptFS("prepared", "child", "content"), "."),
|
||||
promptkit.WithProfileFS(profiles, "."),
|
||||
promptkit.WithLLMClient(client),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
|
||||
prepared, err := engine.PrepareExecution(context.Background(), promptkit.RunRequest{PromptID: "prepared"})
|
||||
if err != nil {
|
||||
t.Fatalf("prepare execution: %v", err)
|
||||
}
|
||||
profiles.set("base.yaml", "id: base\nendpoint: https://base.example/v1\nmodel: second-model\n")
|
||||
|
||||
result, err := engine.RunPrepared(context.Background(), prepared)
|
||||
if err != nil || result == nil || len(client.requests) != 1 || client.requests[0].Target.Model != "first-model" {
|
||||
t.Fatalf("prepared execution=(%+v, %v), requests=%+v", result, err, client.requests)
|
||||
}
|
||||
inspection, err := engine.InspectProfile(context.Background(), "child")
|
||||
if err != nil || inspection.EffectiveModelParams.Model != "second-model" {
|
||||
t.Fatalf("fresh inspection=(%+v, %v)", inspection, err)
|
||||
}
|
||||
}
|
||||
|
||||
type mutableInheritanceProfileFS struct {
|
||||
mu sync.RWMutex
|
||||
files fstest.MapFS
|
||||
}
|
||||
|
||||
func (f *mutableInheritanceProfileFS) Open(name string) (fs.File, error) {
|
||||
f.mu.RLock()
|
||||
defer f.mu.RUnlock()
|
||||
return f.files.Open(name)
|
||||
}
|
||||
|
||||
func (f *mutableInheritanceProfileFS) set(name, content string) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.files[name] = &fstest.MapFile{Data: []byte(content)}
|
||||
}
|
||||
41
profiles.go
41
profiles.go
@@ -2,17 +2,16 @@ package promptkit
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/jsonvalue"
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/profile"
|
||||
)
|
||||
|
||||
// OpenAICompatibleProfile returns an ordinary in-memory Profile for an
|
||||
// OpenAI-compatible chat-completions endpoint.
|
||||
// OpenAICompatibleProfile returns an in-memory Profile for an OpenAI-compatible
|
||||
// chat-completions endpoint. A non-blank BaseProfileID permits its target
|
||||
// fields to be inherited when the profile is selected or inspected.
|
||||
//
|
||||
// It does not register global state, maintain a model catalog, or resolve
|
||||
// credentials. If APIKeyRequired is true, callers satisfy it with
|
||||
@@ -25,6 +24,7 @@ import (
|
||||
func OpenAICompatibleProfile(cfg OpenAICompatibleProfileConfig) Profile {
|
||||
return Profile{
|
||||
ID: cfg.ID,
|
||||
BaseProfileID: cfg.BaseProfileID,
|
||||
BackendID: cfg.BackendID,
|
||||
Endpoint: cfg.Endpoint,
|
||||
Model: cfg.Model,
|
||||
@@ -87,8 +87,9 @@ func toDomainProfile(publicProfile Profile) (domain.ExecutionProfile, error) {
|
||||
return domain.ExecutionProfile{}, err
|
||||
}
|
||||
prof := domain.ExecutionProfile{
|
||||
ID: strings.TrimSpace(publicProfile.ID),
|
||||
BackendID: strings.TrimSpace(publicProfile.BackendID),
|
||||
ID: publicProfile.ID,
|
||||
BaseProfileID: publicProfile.BaseProfileID,
|
||||
BackendID: publicProfile.BackendID,
|
||||
Endpoint: publicProfile.Endpoint,
|
||||
Model: publicProfile.Model,
|
||||
Temperature: publicProfile.Temperature,
|
||||
@@ -100,34 +101,8 @@ func toDomainProfile(publicProfile Profile) (domain.ExecutionProfile, error) {
|
||||
APIKeyRequired: publicProfile.APIKeyRequired,
|
||||
ExtraParams: extraParams,
|
||||
}
|
||||
if err := normalizeAndValidatePublicProfile(&prof); err != nil {
|
||||
if err := profile.NormalizeAndValidateDefinition(&prof); err != nil {
|
||||
return domain.ExecutionProfile{}, err
|
||||
}
|
||||
return prof, nil
|
||||
}
|
||||
|
||||
func normalizeAndValidatePublicProfile(prof *domain.ExecutionProfile) error {
|
||||
if strings.TrimSpace(prof.ID) == "" {
|
||||
return errors.New("id is required")
|
||||
}
|
||||
prof.Endpoint = strings.TrimSpace(prof.Endpoint)
|
||||
if strings.TrimSpace(prof.BackendID) == "" && prof.Endpoint == "" {
|
||||
return errors.New("backend or endpoint is required")
|
||||
}
|
||||
if prof.Endpoint != "" {
|
||||
endpoint, err := domain.NormalizeOpenAICompatibleBaseEndpoint(prof.Endpoint)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
prof.Endpoint = endpoint
|
||||
}
|
||||
if strings.TrimSpace(prof.Model) == "" {
|
||||
return errors.New("model is required")
|
||||
}
|
||||
return domain.ValidateExecutionTargetSettings(domain.ExecutionTarget{
|
||||
Temperature: prof.Temperature,
|
||||
MaxTokens: prof.MaxTokens,
|
||||
TopP: prof.TopP,
|
||||
TimeoutSeconds: prof.TimeoutSeconds,
|
||||
})
|
||||
}
|
||||
|
||||
47
types.go
47
types.go
@@ -443,31 +443,43 @@ type ExecutionTargetOverride struct {
|
||||
|
||||
// Profile is an in-memory execution profile for library consumers.
|
||||
//
|
||||
// It is equivalent to a loaded profile file after validation. Raw API keys do
|
||||
// not belong in profiles; use APIKeyRequired to require callers to provide a
|
||||
// RunRequest.APIKey or explicit request ExecutionTargetOverride.APIKeyEnv, or
|
||||
// use profile YAML api_key_env with file and FS profile sources. Profile has no
|
||||
// stable JSON representation.
|
||||
// A standalone Profile is equivalent to a loaded profile file after local
|
||||
// validation. A derived profile names BaseProfileID and can inherit target
|
||||
// fields when selected or inspected. Raw API keys do not belong in profiles;
|
||||
// use APIKeyRequired to require callers to provide a RunRequest.APIKey or
|
||||
// explicit request ExecutionTargetOverride.APIKeyEnv, or use profile YAML
|
||||
// api_key_env with file and FS profile sources. Profile has no stable JSON
|
||||
// representation.
|
||||
//
|
||||
// WithProfiles validates and copies Profile values during NewEngine. Zero
|
||||
// Temperature, MaxTokens, and TopP values and blank ServiceTier and
|
||||
// ReasoningEffort values leave those provider controls unspecified. A zero
|
||||
// TimeoutSeconds retains the framework deadline, while an empty ExtraParams map
|
||||
// inherits backend request defaults. Use ExecutionTargetOverride pointer fields
|
||||
// to request an explicit numeric zero.
|
||||
// WithProfiles locally validates and copies Profile values during NewEngine.
|
||||
// It checks base-reference existence and resolved target completeness when a
|
||||
// derived profile is selected or inspected. Zero Temperature, MaxTokens, and
|
||||
// TopP values and blank ServiceTier and ReasoningEffort values leave those
|
||||
// provider controls unspecified. A zero TimeoutSeconds retains the framework
|
||||
// deadline, while an empty ExtraParams map inherits backend request defaults.
|
||||
// Use ExecutionTargetOverride pointer fields to request an explicit numeric
|
||||
// zero.
|
||||
type Profile struct {
|
||||
// ID is the required non-blank profile identifier. WithProfiles trims it.
|
||||
ID string
|
||||
// BaseProfileID optionally names one base profile. WithProfiles trims it. A
|
||||
// non-blank value permits required target fields to be inherited when the
|
||||
// profile is selected or inspected, which is also when reference existence
|
||||
// and resolved completeness are checked. A blank value leaves this as a
|
||||
// standalone profile.
|
||||
BaseProfileID string
|
||||
// BackendID optionally selects an engine backend. WithProfiles trims it.
|
||||
// Backend membership is checked when a request selects the profile; an
|
||||
// unknown ID makes preparation fail with ErrProfileLoad.
|
||||
BackendID string
|
||||
// Endpoint is the model-provider base URL. It is required only when
|
||||
// BackendID is blank and otherwise overrides the backend endpoint when
|
||||
// Endpoint is the model-provider base URL. A standalone Profile requires an
|
||||
// endpoint when BackendID is blank; a derived Profile may inherit either
|
||||
// field. A non-blank endpoint overrides the backend endpoint when
|
||||
// non-blank. WithProfiles trims it and requires an absolute HTTP or HTTPS URL
|
||||
// with a host and no user information, query, or fragment.
|
||||
Endpoint string
|
||||
// Model is the required non-blank provider model identifier.
|
||||
// Model is the provider model identifier. It is required for a standalone
|
||||
// Profile and may be inherited by a derived Profile.
|
||||
Model string
|
||||
// Temperature is from 0 through 2. Zero leaves the provider control
|
||||
// unspecified.
|
||||
@@ -499,13 +511,16 @@ type Profile struct {
|
||||
// profile.
|
||||
//
|
||||
// It contains ordinary profile fields for OpenAI-compatible chat-completions
|
||||
// endpoints. APIKeyRequired follows Profile.APIKeyRequired. Raw API keys do not
|
||||
// belong in this config. OpenAICompatibleProfileConfig has no stable JSON
|
||||
// endpoints. BaseProfileID and APIKeyRequired follow Profile. Raw API keys do
|
||||
// not belong in this config. OpenAICompatibleProfileConfig has no stable JSON
|
||||
// representation and is not validated until its resulting Profile is supplied
|
||||
// through WithProfiles to NewEngine.
|
||||
type OpenAICompatibleProfileConfig struct {
|
||||
// ID becomes Profile.ID.
|
||||
ID string
|
||||
// BaseProfileID becomes Profile.BaseProfileID. A non-blank value permits the
|
||||
// resulting Profile to inherit target fields when it is selected or inspected.
|
||||
BaseProfileID string
|
||||
// BackendID becomes Profile.BackendID.
|
||||
BackendID string
|
||||
// Endpoint becomes Profile.Endpoint.
|
||||
|
||||
Reference in New Issue
Block a user