Document profile inspection API

This commit is contained in:
2026-07-30 19:57:07 +00:00
parent 242eace4a7
commit dde48a31fc
10 changed files with 90 additions and 113 deletions

View File

@@ -135,6 +135,34 @@ For programmatic profiles,
[`OpenAICompatibleProfile`](../../profiles.go) converts ordinary
OpenAI-compatible settings into a value accepted by `WithProfiles`.
### Inspect A Profile Before Prompt Work
Use [`Engine.InspectProfile`](../../engine.go) to validate one configured
profile without constructing a synthetic prompt or placeholder inputs. It
resolves the profile's effective target but does not prepare or execute a
prompt:
```go
inspection, err := engine.InspectProfile(ctx, profileID)
if err != nil {
return err
}
target := inspection.EffectiveModelParams
if target.APIKeyEnv != "" {
// Apply application policy for the named environment variable.
} else if inspection.APIKeyRequired {
// Arrange a direct credential before later execution.
}
```
Use this configuration-time boundary when only the profile and its target need
checking. Use `Prepare` when the application also needs prompt, input, schema,
or rendering work; use prepared execution when that work must remain tied to a
later execution. Inspection reports credential requirements but leaves the
timing of credential enforcement to the application. The method's
[GoDoc](../../engine.go) owns its exact result and error contract.
### Set A Per-Run Session And Reasoning
Supply a direct session ID when one prompt should be correlated with a

View File

@@ -152,7 +152,7 @@ extra_params:
| Field | Required | Meaning |
| --- | --- | --- |
| `id` | yes | Non-empty profile identifier. IDs must be unique within one source. |
| `backend` | unless `endpoint` is present | Backend registry ID. It is trimmed and registry membership is checked when the profile is prepared. |
| `backend` | unless `endpoint` is present | Backend registry ID. It is trimmed and registry membership is checked when the profile is prepared or inspected. |
| `endpoint` | unless `backend` is present | Non-empty OpenAI-compatible base URL, including an API version path when required. When both connection fields are present, this overrides the backend endpoint without changing backend identity. |
| `model` | yes | Non-empty provider model name. |
| `temperature` | no | Number from 0 through 2. |
@@ -219,7 +219,9 @@ defines how the effective settings are serialized.
### Source And Profile Precedence
An explicit request profile ID takes precedence over the prompt's
`default_profile`. If neither is present, preparation fails.
`default_profile`. If neither is present, preparation fails. Exact profile
inspection instead takes one explicit profile ID and does not use a prompt
default.
Profile sources resolve matching IDs in this order:
@@ -231,6 +233,7 @@ A higher-precedence source falls back only when the profile is absent. An
invalid matching profile is an error and does not fall back. In-memory
`Profile` values follow the same ranges as YAML profiles. They use
`APIKeyRequired` for request-scoped credentials instead of `api_key_env`.
Preparation and exact profile inspection use this same source precedence.
## Built-In Profile Catalog

View File

@@ -11,7 +11,7 @@ contributor workflow and validation.
| Component | Implemented responsibility | References |
| --- | --- | --- |
| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request and result values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, public error mapping, and engine-local assembly. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, public error mapping, and engine-local assembly. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
| `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
| `internal/backend` | Constructs each engine's immutable registry from the built-in OpenRouter definition and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) |
@@ -27,7 +27,7 @@ contributor workflow and validation.
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
| `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates frozen validation plans for prepared execution. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including response decoding, authentication, deadline handling, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
| `internal/usecase` | Resolves backend, profile, and request settings and coordinates preparation, ordinary execution, and one-attempt prepared execution across internal sources, rendering, artifact loading, generation, validation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
| `internal/usecase` | Resolves profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, generation, validation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
The root package assembles these internal components without exposing their
representations. Consumers depend only on the root facade.

View File

@@ -28,6 +28,21 @@ constructor does not enable one.
Each invocation carries its state in request, prepared-run, and result values.
The runner has no durable run or session store.
## Shared Profile Selection
The runner uses one profile-selection and target-resolution boundary for
ordinary preparation and exact profile inspection. Preparation first selects a
request profile or a prompt default; inspection begins with its required
explicit profile ID. Both then apply the ordinary source precedence, resolve a
named backend, and construct the effective target from framework, backend, and
profile values.
Inspection stops after the resulting endpoint and model are structurally
validated. It does not check credential availability or perform prompt,
artifact, schema, rendering, admission, or model-client work. The root
[`Engine.InspectProfile`](../../engine.go) GoDoc owns the public operation's
exact contract.
## Shared Preparation Pipeline
`Prepare` and `Run` share one private preparation pipeline split at the point

View File

@@ -28,7 +28,12 @@ with fallback only when the primary reports that a profile is absent. Strict
YAML decoding recognizes the optional `backend` field, trims its value, and
requires a model plus at least one non-blank backend or endpoint. Loading does
not check registry membership because the available registry belongs to the
assembled engine; the runner checks membership during preparation.
assembled engine; the runner checks membership during preparation and exact
profile inspection.
Exact profile inspection performs one point-in-time lookup through those
profile sources and checks the resolved target without reading prompt, input,
or schema sources. It does not retain that lookup for a later execution.
`internal/profile/builtin` embeds the maintained built-in profile catalog and
can place a caller-selected repository ahead of that catalog. Every embedded
@@ -79,7 +84,7 @@ captured root document.
loading and hashing, session and message rendering, and target resolution.
`RunPrepared` uses the retained source-derived state and validation plan; it
does not reopen prompt, profile, input, or schema sources and does not rerender
the request. By contrast, ordinary `Prepare` produces an inspection value only:
the request. By contrast, ordinary `Prepare` produces a preparation value only:
a later `Run` performs its own source resolution and preparation.
The [validator tests](../../internal/validate/standard_validator_test.go) own

View File

@@ -33,10 +33,6 @@ consumers.
## Ideas
Prompt-independent profile inspection has been selected for active planning in
the [focused feature roadmap](profile-inspection.md). The remaining ideas are
still available for future selection.
### Prompt-definition inspection
Provide exact prompt-definition lookup without rendering, placeholder inputs,

View File

@@ -1,6 +1,6 @@
# Prompt-Independent Profile Inspection Implementation Plan
**Status:** Ready for implementation.
**Status:** Complete.
## Purpose

View File

@@ -88,8 +88,9 @@ corresponds atomically to the actual execution.
## Priority 2: Prompt-Independent Profile Inspection
**Disposition:** Covered by the accepted
[prompt-independent profile inspection](profile-inspection.md) roadmap.
**Disposition:** Implemented as
[`Engine.InspectProfile`](../../engine.go). See the
[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work).
### Downstream need
@@ -104,62 +105,22 @@ needs to determine whether:
This validation should not require model generation.
### Current integration
### Previous integration
Notarius constructs a synthetic prompt using `testing/fstest.MapFS`, supplies a
dummy transcript, and calls `Engine.Prepare` solely to exercise profile and
backend resolution. This works, but prompt preparation is serving as a
substitute for a profile-inspection interface.
### Requested capability
Add a prompt-independent profile-resolution API, for example:
```go
type ResolvedProfile struct {
ProfileID string
BackendID string
EffectiveTarget ExecutionTarget
APIKeyEnv string
}
func (e *Engine) ResolveProfile(
ctx context.Context,
profileID string,
) (ResolvedProfile, error)
```
The returned shape may differ, but it should provide enough information for a
consumer to validate an explicit profile selection without inventing a prompt
or supplying placeholder inputs.
### Design considerations
- Resolve built-in, file-backed, and programmatic profiles using normal
PromptKit precedence.
- Validate that a referenced backend registration exists.
- Do not resolve, retain, or expose credential values.
- Report credential requirements, such as an environment-variable name, so
the consuming application can decide whether availability is required at
configuration-validation time or only at execution time.
- Return caller-owned values.
- Preserve typed or sentinel error classification for missing and invalid
profiles.
- Consider accepting an `ExecutionTargetOverride` if consumers need to inspect
the same effective target that a run-level override would produce.
- Enumeration of all profiles is not required for the Notarius use case; exact
lookup by ID is sufficient.
Before profile inspection was available, Notarius constructed a synthetic
prompt using `testing/fstest.MapFS`, supplied a dummy transcript, and called
`Engine.Prepare` solely to exercise profile and backend resolution.
### Value to Notarius
This would eliminate a synthetic production-only prompt fixture and establish
a direct, supported contract for configuration-time profile and backend
validation.
The implemented interface eliminates a synthetic production-only prompt
fixture and establishes a direct, supported contract for configuration-time
profile and backend validation.
## Priority 3: Semantic Execution-Target Fingerprints
**Disposition:** Deferred until prompt-independent profile inspection defines
the resolved target whose configuration identity would be fingerprinted.
**Disposition:** Deferred pending a separate semantic-equality design for
resolved execution targets.
### Downstream need
@@ -348,11 +309,9 @@ If the PromptKit team chooses to pursue these ideas, the most useful order for
Notarius would be:
1. Add atomic execution that returns prepared details and the completed result.
2. Add prompt-independent profile inspection.
3. Add a semantic execution-target digest, preferably as part of profile
2. Add a semantic execution-target digest, preferably alongside profile
inspection.
4. Add a typed capacity error carrying backend identity.
3. Add a typed capacity error carrying backend identity.
The first two address concrete workarounds in current Notarius code. The third
would improve checkpoint correctness and reduce coupling. The fourth is
operational polish.
The first addresses a concrete execution workaround. The second would improve
checkpoint correctness and reduce coupling. The third is operational polish.

View File

@@ -1,6 +1,6 @@
# Prompt-Independent Profile Inspection
**Status:** Accepted.
**Status:** Complete.
## Purpose

View File

@@ -185,51 +185,23 @@ and move failures ahead of weather collection.
## Priority 3: Prompt-Independent Profile Inspection
**Disposition:** Covered by the accepted
[prompt-independent profile inspection](profile-inspection.md) roadmap.
**Disposition:** Implemented as
[`Engine.InspectProfile`](../../engine.go). See the
[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work).
### Downstream need
Weatherreporter will allow operators to select an external PromptKit profile
source and may allow an explicit profile override. It should reject a missing
profile, unknown backend, malformed execution target, or unsatisfied credential
requirement before collecting weather data or writing report artifacts.
### Current integration option
Weatherreporter can validate an explicit profile by preparing one embedded
prompt with fixture input. Prompts that use their own default profiles can be
validated during their normal preparation stage.
This couples configuration validation to one prompt and requires placeholder
input even when only profile and backend resolution are relevant.
### Requested capability
The prompt-independent `ResolveProfile` API proposed by the
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
would satisfy this need. It should resolve built-in, file-backed, and
programmatic profiles, validate backend membership, report credential
requirements without resolving credential values, and preserve typed error
classification.
### Additional Weatherreporter considerations
- An explicit application profile override should be inspectable without
selecting a report prompt.
- A prompt-definition inspection result may expose its default profile ID so
Weatherreporter can inspect that profile separately.
- Inspection should distinguish structural profile validity from current
credential availability so configuration validation can apply explicit
application policy.
- An optional execution-target override should be considered only if it
describes the same target that a later run will use.
source and may allow an explicit profile override. It needs to reject a missing
profile, unknown backend, or malformed execution target before collecting
weather data or writing report artifacts, and to apply its own policy to a
reported credential requirement.
### Value to Weatherreporter
This would improve fail-fast configuration validation and give operator-facing
errors direct profile and backend context. It is valuable but not required for
the initial migration.
The implemented interface improves fail-fast configuration validation and gives
operator-facing errors direct profile and backend context. It remains optional
for the initial migration.
## Priority 4: Eager Source Validation
@@ -368,8 +340,8 @@ It should not block adoption.
### Semantic Execution-Target Fingerprints
**Disposition:** Deferred until prompt-independent profile inspection defines
the resolved target whose configuration identity would be fingerprinted.
**Disposition:** Deferred pending a separate semantic-equality design for
resolved execution targets.
The semantic target digest proposed by the
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-3-semantic-execution-target-fingerprints)
@@ -432,11 +404,10 @@ Weatherreporter would be:
1. Add executable preparation handles, ideally sharing implementation with an
atomic detailed-run API.
2. Add prompt-definition inspection.
3. Add prompt-independent profile inspection.
4. Consider eager source validation after evaluating whether the two exact
3. Consider eager source validation after evaluating whether the two exact
inspection APIs are sufficient.
5. Add structured generation errors.
6. Add structured capacity errors and semantic execution-target fingerprints
4. Add structured generation errors.
5. Add structured capacity errors and semantic execution-target fingerprints
as lower-priority operational improvements.
The first item removes the only material integration workaround. Prompt and