Add feature roadmap and implementation plan for profile inspection API

This commit is contained in:
2026-07-30 19:42:51 +00:00
parent 2ba0146e5d
commit 369ab5392d
5 changed files with 779 additions and 601 deletions

View File

@@ -33,26 +33,10 @@ consumers.
## Ideas
Executable preparation handles have been selected for active planning in the
[focused feature roadmap](prepared-execution.md). The remaining ideas are
Prompt-independent profile inspection has been selected for active planning in
the [focused feature roadmap](profile-inspection.md). The remaining ideas are
still available for future selection.
### Prompt-independent profile inspection
Provide exact profile lookup and structural resolution without requiring a
synthetic prompt, placeholder inputs, or model generation. This shared need is
described by
[Notarius](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
and
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-3-prompt-independent-profile-inspection).
- Apply ordinary built-in, file-backed, and programmatic profile precedence.
- Validate referenced backend membership and the structurally resolved
execution target.
- Report credential requirements and environment-variable names without
exposing credential values or requiring current credential availability.
- Support exact lookup by profile ID; enumeration is not required initially.
### Prompt-definition inspection
Provide exact prompt-definition lookup without rendering, placeholder inputs,

File diff suppressed because it is too large Load Diff

View File

@@ -88,8 +88,8 @@ corresponds atomically to the actual execution.
## Priority 2: Prompt-Independent Profile Inspection
**Disposition:** Accepted into the
[future catalog](future.md#prompt-independent-profile-inspection).
**Disposition:** Covered by the accepted
[prompt-independent profile inspection](profile-inspection.md) roadmap.
### Downstream need

View File

@@ -0,0 +1,246 @@
# Prompt-Independent Profile Inspection
**Status:** Accepted.
## Purpose
Allow consumers to look up one execution profile by ID and inspect its
structurally resolved model target without selecting a prompt, supplying
placeholder inputs, checking credential availability, or invoking a model.
This provides a direct configuration-validation boundary for
[Notarius](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
and
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-3-prompt-independent-profile-inspection).
## Motivation
Both downstream consumers need to reject invalid configured profile IDs before
starting application work. They currently have to construct a synthetic prompt
and call `Engine.Prepare` merely to exercise profile loading, backend lookup,
and execution-target resolution.
That workaround couples profile validation to unrelated prompt definitions,
fixture inputs, rendering, schema behavior, and current credential
availability. Promptkit already owns profile precedence, backend membership,
and target resolution, so it should expose that cohesive capability directly.
## Consumer Workflow
The target public workflow is:
```go
inspection, err := engine.InspectProfile(ctx, profileID)
if err != nil {
// Reject or report the configured profile.
return
}
target := inspection.EffectiveModelParams
if target.APIKeyEnv != "" {
// Apply application policy for the named environment variable.
}
```
The target public surface is:
```go
type ProfileInspection struct {
ProfileID string
EffectiveModelParams ExecutionTarget
APIKeyRequired bool
}
func (e *Engine) InspectProfile(
ctx context.Context,
profileID string,
) (*ProfileInspection, error)
```
The declarations and GoDoc will own the exact implemented contract. The
important public shape is exact lookup through the existing engine, one
caller-owned inspection value, the resolved `ExecutionTarget`, and an explicit
signal for a direct API-key requirement.
`EffectiveModelParams.BackendID` identifies the selected registered backend
and remains empty for endpoint-only profiles.
`EffectiveModelParams.APIKeyEnv` reports the effective credential environment
variable name. `APIKeyRequired` reports that the profile requires a direct
request credential instead. These states are mutually exclusive after normal
profile and backend precedence is applied.
`ProfileInspection` does not need a stable JSON representation. Consumers that
persist application configuration or diagnostics can select the fields their
own format requires.
## Lookup And Precedence
`InspectProfile` requires a non-blank explicit profile ID. It trims surrounding
whitespace and otherwise performs the same case-sensitive exact lookup used by
ordinary execution.
Lookup applies the engine's normal profile-source precedence:
- programmatic profiles supplied through `WithProfiles`;
- the configured file, directory, or `fs.FS` profile source; and
- the built-in profile catalog.
A valid higher-precedence match shadows a lower-precedence profile with the
same ID. A malformed or unreadable higher-precedence match fails rather than
silently falling back. The
[profile format reference](../formats.md#profile-definitions) remains the
canonical owner of profile-source and file-format behavior.
Inspection never derives a profile ID from a prompt's `default_profile`; the
caller is inspecting one explicitly named profile.
## Structural Resolution
Inspection loads and validates the selected profile, verifies that a named
backend exists in the engine's immutable backend registry, and applies normal
framework-default, backend, and profile precedence to produce the effective
target.
For the same engine state and profile ID, with no per-run execution override,
the inspected target must match the target that ordinary preparation would
resolve before applying request credentials and checking their availability.
This equivalence must use one shared resolution path rather than a second set
of precedence rules.
Structural resolution includes:
- backend routing identity;
- endpoint and model;
- sampling, token, timeout, service-tier, and reasoning settings;
- the effective credential environment-variable name or direct-key
requirement; and
- deeply copied provider-specific extra parameters.
Inspection returns the effective target rather than a raw profile definition.
This keeps framework and backend defaults visible to consumers without
creating a second public profile-loading interface.
The result does not include request-override presence because no
`ExecutionTargetOverride` participates in inspection.
## Credentials And Sensitive Data
Inspection reports credential requirements but never resolves, retains, or
returns a credential value.
The operation does not read the named environment variable and succeeds when
that variable is absent or blank. It accepts neither a direct API key nor an
API-key environment override. Consumers decide whether credential availability
must be enforced during application configuration, while `Prepare`,
`PrepareExecution`, `Run`, and `RunPrepared` retain their execution-time
credential contracts.
Error messages, formatting, and returned values must not expose environment
values or other resolved secrets. Existing restrictions against raw API keys
in profile sources remain unchanged.
## Ownership, Consistency, And Concurrency
Each successful call returns a caller-owned snapshot. Mutating the returned
target or any nested extra-parameter map or slice cannot affect the engine,
later inspection, or later execution.
Inspection is safe to call concurrently under the engine's existing immutable
registry and repository contracts. It does not mutate profile sources or
cache a result globally.
For filesystem-backed sources, an inspection describes the state observed by
that call. It does not freeze the profile for a later `Run`; a source may
change between operations. Consumers requiring an exact preflight-to-execution
snapshot should use the existing prepared-execution workflow.
## Errors And Cancellation
The operation uses existing public error categories:
- a blank profile ID matches `ErrInvalidRequest`;
- an absent exact ID matches `ErrProfileNotFound` and not `ErrProfileLoad`;
- read, decode, validation, and source-selection failures match
`ErrProfileLoad`; and
- an unknown referenced backend or an invalid structurally resolved target
matches `ErrProfileLoad`.
Errors should preserve useful underlying collaborator and context identities
through `errors.Is` where the existing facade does so, without exposing
internal package types. Context cancellation governs inspection and no partial
inspection result is returned on failure.
Missing credential values are not inspection errors. The operation cannot
return capacity or model-generation failures because it performs neither
backend admission nor generation.
## Compatibility And Boundaries
This feature is additive. Existing profile formats, source precedence,
backend registration, `Prepare`, prepared execution, and `Run` behavior remain
unchanged.
The method belongs on the root `Engine` facade. Profile repositories and the
backend registry remain internal implementation details, and no new public
repository interface is introduced.
Inspection does not require prompt lookup, rendering, artifact loading, schema
loading, validation, backend-capacity admission, or model-client access.
Engine construction retains its ordinary configuration requirements; this
feature does not introduce a separate profile-only engine.
## Documentation
The completed documentation set has these ownership boundaries:
- exported declarations and GoDoc own the exact method, result, ownership,
credential, error, and cancellation contracts;
- the promptkit consumer guide explains configuration-time profile inspection
and distinguishes it from `Prepare` and prepared execution;
- the profile format reference continues to own profile fields and source
precedence; and
- internal documentation describes shared profile and target resolution
without duplicating public contracts.
## Non-Goals
This work does not include:
- enumerating or searching profiles;
- returning raw profile definitions or profile source paths;
- accepting per-run execution overrides, direct API keys, or API-key
environment overrides;
- checking environment-variable contents or other credential availability;
- semantic execution-target fingerprints or profile hashes;
- prompt-definition inspection or prompt default-profile resolution;
- full-corpus validation across every profile source;
- freezing a filesystem-backed profile for later execution;
- exposing backend concurrency limits, queue state, or capacity policy;
- model generation, provider health checks, or endpoint connectivity tests;
- dynamic backend or profile registration after engine construction; or
- changing current profile, backend, prepared-execution, or stable JSON
contracts.
## Target End State
After this work:
- consumers can validate one configured profile without inventing a prompt or
placeholder inputs;
- lookup observes ordinary programmatic, configured-source, and built-in
precedence;
- a successful result proves that the profile exists, is valid, references a
registered backend when applicable, and resolves to a structurally valid
effective target;
- the inspected target matches ordinary preparation for the same profile and
engine state before per-run overrides and credential availability checks;
- credential requirements are visible without reading or exposing credential
values;
- returned targets and nested data are caller-owned;
- inspection performs no rendering, source loading unrelated to the profile,
capacity admission, or model work;
- existing execution workflows and compatibility contracts remain unchanged;
and
- Promptkit owns reusable profile validation while downstream applications
retain configuration policy, persistence, logging, and credential-timing
decisions.

View File

@@ -185,8 +185,8 @@ and move failures ahead of weather collection.
## Priority 3: Prompt-Independent Profile Inspection
**Disposition:** Accepted into the
[future catalog](future.md#prompt-independent-profile-inspection).
**Disposition:** Covered by the accepted
[prompt-independent profile inspection](profile-inspection.md) roadmap.
### Downstream need