Document profile inspection API
This commit is contained in:
@@ -135,6 +135,34 @@ For programmatic profiles,
|
||||
[`OpenAICompatibleProfile`](../../profiles.go) converts ordinary
|
||||
OpenAI-compatible settings into a value accepted by `WithProfiles`.
|
||||
|
||||
### Inspect A Profile Before Prompt Work
|
||||
|
||||
Use [`Engine.InspectProfile`](../../engine.go) to validate one configured
|
||||
profile without constructing a synthetic prompt or placeholder inputs. It
|
||||
resolves the profile's effective target but does not prepare or execute a
|
||||
prompt:
|
||||
|
||||
```go
|
||||
inspection, err := engine.InspectProfile(ctx, profileID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
target := inspection.EffectiveModelParams
|
||||
if target.APIKeyEnv != "" {
|
||||
// Apply application policy for the named environment variable.
|
||||
} else if inspection.APIKeyRequired {
|
||||
// Arrange a direct credential before later execution.
|
||||
}
|
||||
```
|
||||
|
||||
Use this configuration-time boundary when only the profile and its target need
|
||||
checking. Use `Prepare` when the application also needs prompt, input, schema,
|
||||
or rendering work; use prepared execution when that work must remain tied to a
|
||||
later execution. Inspection reports credential requirements but leaves the
|
||||
timing of credential enforcement to the application. The method's
|
||||
[GoDoc](../../engine.go) owns its exact result and error contract.
|
||||
|
||||
### Set A Per-Run Session And Reasoning
|
||||
|
||||
Supply a direct session ID when one prompt should be correlated with a
|
||||
|
||||
@@ -152,7 +152,7 @@ extra_params:
|
||||
| Field | Required | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `id` | yes | Non-empty profile identifier. IDs must be unique within one source. |
|
||||
| `backend` | unless `endpoint` is present | Backend registry ID. It is trimmed and registry membership is checked when the profile is prepared. |
|
||||
| `backend` | unless `endpoint` is present | Backend registry ID. It is trimmed and registry membership is checked when the profile is prepared or inspected. |
|
||||
| `endpoint` | unless `backend` is present | Non-empty OpenAI-compatible base URL, including an API version path when required. When both connection fields are present, this overrides the backend endpoint without changing backend identity. |
|
||||
| `model` | yes | Non-empty provider model name. |
|
||||
| `temperature` | no | Number from 0 through 2. |
|
||||
@@ -219,7 +219,9 @@ defines how the effective settings are serialized.
|
||||
### Source And Profile Precedence
|
||||
|
||||
An explicit request profile ID takes precedence over the prompt's
|
||||
`default_profile`. If neither is present, preparation fails.
|
||||
`default_profile`. If neither is present, preparation fails. Exact profile
|
||||
inspection instead takes one explicit profile ID and does not use a prompt
|
||||
default.
|
||||
|
||||
Profile sources resolve matching IDs in this order:
|
||||
|
||||
@@ -231,6 +233,7 @@ A higher-precedence source falls back only when the profile is absent. An
|
||||
invalid matching profile is an error and does not fall back. In-memory
|
||||
`Profile` values follow the same ranges as YAML profiles. They use
|
||||
`APIKeyRequired` for request-scoped credentials instead of `api_key_env`.
|
||||
Preparation and exact profile inspection use this same source precedence.
|
||||
|
||||
## Built-In Profile Catalog
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ contributor workflow and validation.
|
||||
|
||||
| Component | Implemented responsibility | References |
|
||||
| --- | --- | --- |
|
||||
| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request and result values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, public error mapping, and engine-local assembly. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
|
||||
| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, public error mapping, and engine-local assembly. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
|
||||
| `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
|
||||
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
|
||||
| `internal/backend` | Constructs each engine's immutable registry from the built-in OpenRouter definition and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) |
|
||||
@@ -27,7 +27,7 @@ contributor workflow and validation.
|
||||
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
|
||||
| `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates frozen validation plans for prepared execution. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
|
||||
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including response decoding, authentication, deadline handling, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
|
||||
| `internal/usecase` | Resolves backend, profile, and request settings and coordinates preparation, ordinary execution, and one-attempt prepared execution across internal sources, rendering, artifact loading, generation, validation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
|
||||
| `internal/usecase` | Resolves profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, generation, validation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
|
||||
|
||||
The root package assembles these internal components without exposing their
|
||||
representations. Consumers depend only on the root facade.
|
||||
|
||||
@@ -28,6 +28,21 @@ constructor does not enable one.
|
||||
Each invocation carries its state in request, prepared-run, and result values.
|
||||
The runner has no durable run or session store.
|
||||
|
||||
## Shared Profile Selection
|
||||
|
||||
The runner uses one profile-selection and target-resolution boundary for
|
||||
ordinary preparation and exact profile inspection. Preparation first selects a
|
||||
request profile or a prompt default; inspection begins with its required
|
||||
explicit profile ID. Both then apply the ordinary source precedence, resolve a
|
||||
named backend, and construct the effective target from framework, backend, and
|
||||
profile values.
|
||||
|
||||
Inspection stops after the resulting endpoint and model are structurally
|
||||
validated. It does not check credential availability or perform prompt,
|
||||
artifact, schema, rendering, admission, or model-client work. The root
|
||||
[`Engine.InspectProfile`](../../engine.go) GoDoc owns the public operation's
|
||||
exact contract.
|
||||
|
||||
## Shared Preparation Pipeline
|
||||
|
||||
`Prepare` and `Run` share one private preparation pipeline split at the point
|
||||
|
||||
@@ -28,7 +28,12 @@ with fallback only when the primary reports that a profile is absent. Strict
|
||||
YAML decoding recognizes the optional `backend` field, trims its value, and
|
||||
requires a model plus at least one non-blank backend or endpoint. Loading does
|
||||
not check registry membership because the available registry belongs to the
|
||||
assembled engine; the runner checks membership during preparation.
|
||||
assembled engine; the runner checks membership during preparation and exact
|
||||
profile inspection.
|
||||
|
||||
Exact profile inspection performs one point-in-time lookup through those
|
||||
profile sources and checks the resolved target without reading prompt, input,
|
||||
or schema sources. It does not retain that lookup for a later execution.
|
||||
|
||||
`internal/profile/builtin` embeds the maintained built-in profile catalog and
|
||||
can place a caller-selected repository ahead of that catalog. Every embedded
|
||||
@@ -79,7 +84,7 @@ captured root document.
|
||||
loading and hashing, session and message rendering, and target resolution.
|
||||
`RunPrepared` uses the retained source-derived state and validation plan; it
|
||||
does not reopen prompt, profile, input, or schema sources and does not rerender
|
||||
the request. By contrast, ordinary `Prepare` produces an inspection value only:
|
||||
the request. By contrast, ordinary `Prepare` produces a preparation value only:
|
||||
a later `Run` performs its own source resolution and preparation.
|
||||
|
||||
The [validator tests](../../internal/validate/standard_validator_test.go) own
|
||||
|
||||
@@ -33,10 +33,6 @@ consumers.
|
||||
|
||||
## Ideas
|
||||
|
||||
Prompt-independent profile inspection has been selected for active planning in
|
||||
the [focused feature roadmap](profile-inspection.md). The remaining ideas are
|
||||
still available for future selection.
|
||||
|
||||
### Prompt-definition inspection
|
||||
|
||||
Provide exact prompt-definition lookup without rendering, placeholder inputs,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Prompt-Independent Profile Inspection Implementation Plan
|
||||
|
||||
**Status:** Ready for implementation.
|
||||
**Status:** Complete.
|
||||
|
||||
## Purpose
|
||||
|
||||
|
||||
@@ -88,8 +88,9 @@ corresponds atomically to the actual execution.
|
||||
|
||||
## Priority 2: Prompt-Independent Profile Inspection
|
||||
|
||||
**Disposition:** Covered by the accepted
|
||||
[prompt-independent profile inspection](profile-inspection.md) roadmap.
|
||||
**Disposition:** Implemented as
|
||||
[`Engine.InspectProfile`](../../engine.go). See the
|
||||
[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work).
|
||||
|
||||
### Downstream need
|
||||
|
||||
@@ -104,62 +105,22 @@ needs to determine whether:
|
||||
|
||||
This validation should not require model generation.
|
||||
|
||||
### Current integration
|
||||
### Previous integration
|
||||
|
||||
Notarius constructs a synthetic prompt using `testing/fstest.MapFS`, supplies a
|
||||
dummy transcript, and calls `Engine.Prepare` solely to exercise profile and
|
||||
backend resolution. This works, but prompt preparation is serving as a
|
||||
substitute for a profile-inspection interface.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Add a prompt-independent profile-resolution API, for example:
|
||||
|
||||
```go
|
||||
type ResolvedProfile struct {
|
||||
ProfileID string
|
||||
BackendID string
|
||||
EffectiveTarget ExecutionTarget
|
||||
APIKeyEnv string
|
||||
}
|
||||
|
||||
func (e *Engine) ResolveProfile(
|
||||
ctx context.Context,
|
||||
profileID string,
|
||||
) (ResolvedProfile, error)
|
||||
```
|
||||
|
||||
The returned shape may differ, but it should provide enough information for a
|
||||
consumer to validate an explicit profile selection without inventing a prompt
|
||||
or supplying placeholder inputs.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Resolve built-in, file-backed, and programmatic profiles using normal
|
||||
PromptKit precedence.
|
||||
- Validate that a referenced backend registration exists.
|
||||
- Do not resolve, retain, or expose credential values.
|
||||
- Report credential requirements, such as an environment-variable name, so
|
||||
the consuming application can decide whether availability is required at
|
||||
configuration-validation time or only at execution time.
|
||||
- Return caller-owned values.
|
||||
- Preserve typed or sentinel error classification for missing and invalid
|
||||
profiles.
|
||||
- Consider accepting an `ExecutionTargetOverride` if consumers need to inspect
|
||||
the same effective target that a run-level override would produce.
|
||||
- Enumeration of all profiles is not required for the Notarius use case; exact
|
||||
lookup by ID is sufficient.
|
||||
Before profile inspection was available, Notarius constructed a synthetic
|
||||
prompt using `testing/fstest.MapFS`, supplied a dummy transcript, and called
|
||||
`Engine.Prepare` solely to exercise profile and backend resolution.
|
||||
|
||||
### Value to Notarius
|
||||
|
||||
This would eliminate a synthetic production-only prompt fixture and establish
|
||||
a direct, supported contract for configuration-time profile and backend
|
||||
validation.
|
||||
The implemented interface eliminates a synthetic production-only prompt
|
||||
fixture and establishes a direct, supported contract for configuration-time
|
||||
profile and backend validation.
|
||||
|
||||
## Priority 3: Semantic Execution-Target Fingerprints
|
||||
|
||||
**Disposition:** Deferred until prompt-independent profile inspection defines
|
||||
the resolved target whose configuration identity would be fingerprinted.
|
||||
**Disposition:** Deferred pending a separate semantic-equality design for
|
||||
resolved execution targets.
|
||||
|
||||
### Downstream need
|
||||
|
||||
@@ -348,11 +309,9 @@ If the PromptKit team chooses to pursue these ideas, the most useful order for
|
||||
Notarius would be:
|
||||
|
||||
1. Add atomic execution that returns prepared details and the completed result.
|
||||
2. Add prompt-independent profile inspection.
|
||||
3. Add a semantic execution-target digest, preferably as part of profile
|
||||
2. Add a semantic execution-target digest, preferably alongside profile
|
||||
inspection.
|
||||
4. Add a typed capacity error carrying backend identity.
|
||||
3. Add a typed capacity error carrying backend identity.
|
||||
|
||||
The first two address concrete workarounds in current Notarius code. The third
|
||||
would improve checkpoint correctness and reduce coupling. The fourth is
|
||||
operational polish.
|
||||
The first addresses a concrete execution workaround. The second would improve
|
||||
checkpoint correctness and reduce coupling. The third is operational polish.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Prompt-Independent Profile Inspection
|
||||
|
||||
**Status:** Accepted.
|
||||
**Status:** Complete.
|
||||
|
||||
## Purpose
|
||||
|
||||
|
||||
@@ -185,51 +185,23 @@ and move failures ahead of weather collection.
|
||||
|
||||
## Priority 3: Prompt-Independent Profile Inspection
|
||||
|
||||
**Disposition:** Covered by the accepted
|
||||
[prompt-independent profile inspection](profile-inspection.md) roadmap.
|
||||
**Disposition:** Implemented as
|
||||
[`Engine.InspectProfile`](../../engine.go). See the
|
||||
[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work).
|
||||
|
||||
### Downstream need
|
||||
|
||||
Weatherreporter will allow operators to select an external PromptKit profile
|
||||
source and may allow an explicit profile override. It should reject a missing
|
||||
profile, unknown backend, malformed execution target, or unsatisfied credential
|
||||
requirement before collecting weather data or writing report artifacts.
|
||||
|
||||
### Current integration option
|
||||
|
||||
Weatherreporter can validate an explicit profile by preparing one embedded
|
||||
prompt with fixture input. Prompts that use their own default profiles can be
|
||||
validated during their normal preparation stage.
|
||||
|
||||
This couples configuration validation to one prompt and requires placeholder
|
||||
input even when only profile and backend resolution are relevant.
|
||||
|
||||
### Requested capability
|
||||
|
||||
The prompt-independent `ResolveProfile` API proposed by the
|
||||
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
|
||||
would satisfy this need. It should resolve built-in, file-backed, and
|
||||
programmatic profiles, validate backend membership, report credential
|
||||
requirements without resolving credential values, and preserve typed error
|
||||
classification.
|
||||
|
||||
### Additional Weatherreporter considerations
|
||||
|
||||
- An explicit application profile override should be inspectable without
|
||||
selecting a report prompt.
|
||||
- A prompt-definition inspection result may expose its default profile ID so
|
||||
Weatherreporter can inspect that profile separately.
|
||||
- Inspection should distinguish structural profile validity from current
|
||||
credential availability so configuration validation can apply explicit
|
||||
application policy.
|
||||
- An optional execution-target override should be considered only if it
|
||||
describes the same target that a later run will use.
|
||||
source and may allow an explicit profile override. It needs to reject a missing
|
||||
profile, unknown backend, or malformed execution target before collecting
|
||||
weather data or writing report artifacts, and to apply its own policy to a
|
||||
reported credential requirement.
|
||||
|
||||
### Value to Weatherreporter
|
||||
|
||||
This would improve fail-fast configuration validation and give operator-facing
|
||||
errors direct profile and backend context. It is valuable but not required for
|
||||
the initial migration.
|
||||
The implemented interface improves fail-fast configuration validation and gives
|
||||
operator-facing errors direct profile and backend context. It remains optional
|
||||
for the initial migration.
|
||||
|
||||
## Priority 4: Eager Source Validation
|
||||
|
||||
@@ -368,8 +340,8 @@ It should not block adoption.
|
||||
|
||||
### Semantic Execution-Target Fingerprints
|
||||
|
||||
**Disposition:** Deferred until prompt-independent profile inspection defines
|
||||
the resolved target whose configuration identity would be fingerprinted.
|
||||
**Disposition:** Deferred pending a separate semantic-equality design for
|
||||
resolved execution targets.
|
||||
|
||||
The semantic target digest proposed by the
|
||||
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-3-semantic-execution-target-fingerprints)
|
||||
@@ -432,11 +404,10 @@ Weatherreporter would be:
|
||||
1. Add executable preparation handles, ideally sharing implementation with an
|
||||
atomic detailed-run API.
|
||||
2. Add prompt-definition inspection.
|
||||
3. Add prompt-independent profile inspection.
|
||||
4. Consider eager source validation after evaluating whether the two exact
|
||||
3. Consider eager source validation after evaluating whether the two exact
|
||||
inspection APIs are sufficient.
|
||||
5. Add structured generation errors.
|
||||
6. Add structured capacity errors and semantic execution-target fingerprints
|
||||
4. Add structured generation errors.
|
||||
5. Add structured capacity errors and semantic execution-target fingerprints
|
||||
as lower-priority operational improvements.
|
||||
|
||||
The first item removes the only material integration workaround. Prompt and
|
||||
|
||||
Reference in New Issue
Block a user