Resolve profiles through built-in backends

This commit is contained in:
2026-07-29 17:10:34 +00:00
parent 8d00354c59
commit 810f80e7c9
41 changed files with 538 additions and 173 deletions

View File

@@ -136,7 +136,7 @@ A profile supplies model execution settings:
```yaml
id: local-summary
endpoint: http://localhost:8000/v1
backend: openrouter
model: example-model
temperature: 0.2
max_tokens: 500
@@ -144,7 +144,6 @@ top_p: 0.95
timeout_seconds: 90
service_tier: flex
reasoning_effort: medium
api_key_env: EXAMPLE_API_KEY
extra_params:
provider_option: enabled
```
@@ -152,7 +151,8 @@ extra_params:
| Field | Required | Meaning |
| --- | --- | --- |
| `id` | yes | Non-empty profile identifier. IDs must be unique within one source. |
| `endpoint` | yes | Non-empty OpenAI-compatible base URL, including an API version path when required. |
| `backend` | unless `endpoint` is present | Backend registry ID. It is trimmed and registry membership is checked when the profile is prepared. |
| `endpoint` | unless `backend` is present | Non-empty OpenAI-compatible base URL, including an API version path when required. When both connection fields are present, this overrides the backend endpoint without changing backend identity. |
| `model` | yes | Non-empty provider model name. |
| `temperature` | no | Number from 0 through 2. |
| `max_tokens` | no | Integer zero or greater. |
@@ -166,6 +166,9 @@ extra_params:
Raw `api_key` is prohibited in profile YAML. Store only an environment
variable name in `api_key_env`.
Promptkit does not infer a backend from a model or endpoint. Endpoint-only
profiles remain supported and have no effective backend ID.
`extra_params` accepts null, booleans, finite numbers, strings, arrays, and
objects with string keys. Keys must be non-empty. With the built-in client,
they also cannot collide with the standard fields listed in the
@@ -176,8 +179,9 @@ they also cannot collide with the standard fields listed in the
Execution settings resolve in this order:
1. framework defaults;
2. the selected profile; and
3. request `ExecutionTargetOverride` values.
2. the selected backend, when the profile names one;
3. the selected profile; and
4. request `ExecutionTargetOverride` values.
The framework defaults are:
@@ -194,8 +198,10 @@ explicit zero is preserved. In particular, an explicit request
`timeout_seconds` of zero disables the per-generation deadline while leaving
the caller context and transport timeout intact.
Non-empty request strings replace profile strings. A non-empty request
`ExtraParams` map replaces the profile map rather than merging keys.
Non-empty profile strings replace backend defaults, and non-empty request
strings replace both. Backend identity is retained when either layer overrides
the endpoint. A non-empty `extra_params` map at each layer replaces the entire
lower-precedence map rather than merging keys.
The [outbound integration contract](integrations/openai-compatible-chat.md)
defines how the effective settings are serialized.
@@ -217,9 +223,11 @@ invalid matching profile is an error and does not fall back. In-memory
## Built-In Profile Catalog
Built-ins use the OpenRouter-compatible endpoint and
`OPENROUTER_API_KEY`. A custom or in-memory profile with the same ID takes
precedence.
Every built-in selects the `openrouter` backend. The engine's built-in backend
registry supplies `https://openrouter.ai/api/v1` and the environment-variable
name `OPENROUTER_API_KEY`, so individual profiles contain only model and
generation settings. A custom or in-memory profile with the same profile ID
takes precedence.
| Provider | ID | Model |
| --- | --- | --- |
@@ -270,6 +278,10 @@ prompt, profile, schema, or example files:
- a request can provide a direct `APIKey` or override `APIKeyEnv`; and
- a direct request key takes precedence over environment lookup.
After a direct request key, the credential-source precedence is request
`APIKeyEnv`, profile `api_key_env`, then the backend default. An in-memory
profile with `APIKeyRequired` clears an inherited backend environment name and
requires a direct key unless the request explicitly supplies `APIKeyEnv`.
Promptkit validates required credential availability during preparation.
Direct keys are excluded from JSON results and redacted by public string
formatters. Environment-variable names may appear in prepared metadata, but

View File

@@ -17,10 +17,11 @@ and override semantics consumed by the runner.
## Collaborators
`Runner` coordinates narrow internal interfaces for prompt definitions,
profiles, artifacts, rendering, model generation, and validation. Schema
documents are loaded through the validator's optional schema-loader interface.
An output repairer can be injected internally, but the ordinary runner
constructor does not enable one.
profiles, backend resolution, artifacts, rendering, model generation, and
validation. The root engine supplies an immutable built-in backend registry.
Schema documents are loaded through the validator's optional schema-loader
interface. An output repairer can be injected internally, but the ordinary
runner constructor does not enable one.
Each invocation carries its state in request, prepared-run, and result values.
The runner has no durable run or session store.
@@ -32,20 +33,25 @@ The runner has no durable run or session store.
1. validate the prompt selection and load the prompt definition;
2. hash the loaded definition;
3. select the request profile or the prompt's default profile;
4. resolve application-neutral defaults, profile values, and explicit request
overrides in that order;
5. validate endpoint, model, numeric overrides, and credential requirements;
6. resolve the output contract and load a structured-output schema when
4. resolve the profile's backend ID, when present;
5. resolve application-neutral defaults, backend defaults, profile values,
and explicit request overrides in that order;
6. validate endpoint, model, numeric overrides, and credential requirements;
7. resolve the output contract and load a structured-output schema when
required;
7. load and hash input artifacts;
8. render and hash the prompt; and
9. return the effective settings, source identities, messages, hashes, and
8. load and hash input artifacts;
9. render and hash the prompt; and
10. return the effective settings, source identities, messages, hashes, and
preparation timing.
Pointer-based numeric overrides preserve an explicit zero. Invalid negative or
out-of-range values fail as invalid requests. A direct API key takes
precedence over environment lookup for execution; secret values remain
excluded from serialized metadata.
out-of-range values fail as invalid requests. Endpoint overrides do not change
the selected backend identity. Non-empty extra-parameter maps replace whole
lower-precedence maps. A direct API key takes precedence over environment
lookup; otherwise request, profile, and backend environment-variable names
apply in that order. A profile requiring a direct key clears an inherited
backend environment name unless the request supplies its own name. Secret
values remain excluded from serialized metadata.
## Run Flow
@@ -60,8 +66,10 @@ target, validation errors, prior output, and structured-output specification.
This capability remains internal and is not a public option.
A successful result includes the output artifact and raw output, validation
state, prompt and rendered-prompt hashes, selected profile, effective settings,
input hashes, token usage, a generated run identifier, and UTC timing.
state, prompt and rendered-prompt hashes, selected profile and backend,
effective settings, input hashes, token usage, a generated run identifier, and
UTC timing. The same effective target, including backend identity, reaches
generation and any repair attempt.
## Failure Categories
@@ -71,13 +79,16 @@ validation failures. Wrapping preserves the package identities mapped by the
public facade and retains collaborator identities where they are part of the
internal contract. Context cancellation propagates through the invoked
collaborator and is classified by the owning operation.
An unknown selected backend, or a selected backend with no configured resolver,
is classified as a profile-load failure.
## Test Ownership And Changes
The [runner tests](../../internal/usecase/runner_test.go) own preparation order,
selection and override precedence, schema-before-generation behavior, hashing,
generation and validation outcomes, bounded repair, credentials and redaction,
error categories, artifact metadata, usage, and timing.
generation and validation outcomes, backend propagation, bounded repair,
credentials and redaction, error categories, artifact metadata, usage, and
timing.
Changes to orchestration should continue to use the existing package
interfaces, keep request state local to an invocation, and preserve `Run`'s use

View File

@@ -24,13 +24,19 @@ duplicate detection, and source containment:
`internal/profile` loads and validates execution profiles from an
operating-system filesystem or an `fs.FS`. It supports a primary repository
with fallback only when the primary reports that a profile is absent.
with fallback only when the primary reports that a profile is absent. Strict
YAML decoding recognizes the optional `backend` field, trims its value, and
requires a model plus at least one non-blank backend or endpoint. Loading does
not check registry membership because the available registry belongs to the
assembled engine; the runner checks membership during preparation.
`internal/profile/builtin` embeds the maintained built-in profile catalog and
can place a caller-selected repository ahead of that catalog. Profile behavior
is owned by the
can place a caller-selected repository ahead of that catalog. Every embedded
profile selects `openrouter` and inherits its endpoint and credential
environment-variable name from the built-in backend registry rather than
repeating those values. Profile behavior is owned by the
[profile repository tests](../../internal/profile/repository_test.go), while
catalog completeness, duplicate IDs, and overlay behavior are owned by the
catalog completeness, backend-selection invariant, duplicate IDs, and overlay behavior are owned by the
[built-in repository tests](../../internal/profile/builtin/repository_test.go).
## Ordinary Artifacts

View File

@@ -299,6 +299,8 @@ immutable and the final repository race suite exercises concurrent reads.
## Stage 2 — Profile Selection And Effective Resolution
**Status:** Complete.
### Goal
Make the built-in OpenRouter backend selectable by file and in-memory profiles,