Add feature roadmap and implementation plan for profile inspection API
This commit is contained in:
@@ -33,26 +33,10 @@ consumers.
|
|||||||
|
|
||||||
## Ideas
|
## Ideas
|
||||||
|
|
||||||
Executable preparation handles have been selected for active planning in the
|
Prompt-independent profile inspection has been selected for active planning in
|
||||||
[focused feature roadmap](prepared-execution.md). The remaining ideas are
|
the [focused feature roadmap](profile-inspection.md). The remaining ideas are
|
||||||
still available for future selection.
|
still available for future selection.
|
||||||
|
|
||||||
### Prompt-independent profile inspection
|
|
||||||
|
|
||||||
Provide exact profile lookup and structural resolution without requiring a
|
|
||||||
synthetic prompt, placeholder inputs, or model generation. This shared need is
|
|
||||||
described by
|
|
||||||
[Notarius](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
|
|
||||||
and
|
|
||||||
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-3-prompt-independent-profile-inspection).
|
|
||||||
|
|
||||||
- Apply ordinary built-in, file-backed, and programmatic profile precedence.
|
|
||||||
- Validate referenced backend membership and the structurally resolved
|
|
||||||
execution target.
|
|
||||||
- Report credential requirements and environment-variable names without
|
|
||||||
exposing credential values or requiring current credential availability.
|
|
||||||
- Support exact lookup by profile ID; enumeration is not required initially.
|
|
||||||
|
|
||||||
### Prompt-definition inspection
|
### Prompt-definition inspection
|
||||||
|
|
||||||
Provide exact prompt-definition lookup without rendering, placeholder inputs,
|
Provide exact prompt-definition lookup without rendering, placeholder inputs,
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -88,8 +88,8 @@ corresponds atomically to the actual execution.
|
|||||||
|
|
||||||
## Priority 2: Prompt-Independent Profile Inspection
|
## Priority 2: Prompt-Independent Profile Inspection
|
||||||
|
|
||||||
**Disposition:** Accepted into the
|
**Disposition:** Covered by the accepted
|
||||||
[future catalog](future.md#prompt-independent-profile-inspection).
|
[prompt-independent profile inspection](profile-inspection.md) roadmap.
|
||||||
|
|
||||||
### Downstream need
|
### Downstream need
|
||||||
|
|
||||||
|
|||||||
246
docs/roadmap/profile-inspection.md
Normal file
246
docs/roadmap/profile-inspection.md
Normal file
@@ -0,0 +1,246 @@
|
|||||||
|
# Prompt-Independent Profile Inspection
|
||||||
|
|
||||||
|
**Status:** Accepted.
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Allow consumers to look up one execution profile by ID and inspect its
|
||||||
|
structurally resolved model target without selecting a prompt, supplying
|
||||||
|
placeholder inputs, checking credential availability, or invoking a model.
|
||||||
|
|
||||||
|
This provides a direct configuration-validation boundary for
|
||||||
|
[Notarius](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
|
||||||
|
and
|
||||||
|
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-3-prompt-independent-profile-inspection).
|
||||||
|
|
||||||
|
## Motivation
|
||||||
|
|
||||||
|
Both downstream consumers need to reject invalid configured profile IDs before
|
||||||
|
starting application work. They currently have to construct a synthetic prompt
|
||||||
|
and call `Engine.Prepare` merely to exercise profile loading, backend lookup,
|
||||||
|
and execution-target resolution.
|
||||||
|
|
||||||
|
That workaround couples profile validation to unrelated prompt definitions,
|
||||||
|
fixture inputs, rendering, schema behavior, and current credential
|
||||||
|
availability. Promptkit already owns profile precedence, backend membership,
|
||||||
|
and target resolution, so it should expose that cohesive capability directly.
|
||||||
|
|
||||||
|
## Consumer Workflow
|
||||||
|
|
||||||
|
The target public workflow is:
|
||||||
|
|
||||||
|
```go
|
||||||
|
inspection, err := engine.InspectProfile(ctx, profileID)
|
||||||
|
if err != nil {
|
||||||
|
// Reject or report the configured profile.
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
target := inspection.EffectiveModelParams
|
||||||
|
if target.APIKeyEnv != "" {
|
||||||
|
// Apply application policy for the named environment variable.
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The target public surface is:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type ProfileInspection struct {
|
||||||
|
ProfileID string
|
||||||
|
EffectiveModelParams ExecutionTarget
|
||||||
|
APIKeyRequired bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *Engine) InspectProfile(
|
||||||
|
ctx context.Context,
|
||||||
|
profileID string,
|
||||||
|
) (*ProfileInspection, error)
|
||||||
|
```
|
||||||
|
|
||||||
|
The declarations and GoDoc will own the exact implemented contract. The
|
||||||
|
important public shape is exact lookup through the existing engine, one
|
||||||
|
caller-owned inspection value, the resolved `ExecutionTarget`, and an explicit
|
||||||
|
signal for a direct API-key requirement.
|
||||||
|
|
||||||
|
`EffectiveModelParams.BackendID` identifies the selected registered backend
|
||||||
|
and remains empty for endpoint-only profiles.
|
||||||
|
`EffectiveModelParams.APIKeyEnv` reports the effective credential environment
|
||||||
|
variable name. `APIKeyRequired` reports that the profile requires a direct
|
||||||
|
request credential instead. These states are mutually exclusive after normal
|
||||||
|
profile and backend precedence is applied.
|
||||||
|
|
||||||
|
`ProfileInspection` does not need a stable JSON representation. Consumers that
|
||||||
|
persist application configuration or diagnostics can select the fields their
|
||||||
|
own format requires.
|
||||||
|
|
||||||
|
## Lookup And Precedence
|
||||||
|
|
||||||
|
`InspectProfile` requires a non-blank explicit profile ID. It trims surrounding
|
||||||
|
whitespace and otherwise performs the same case-sensitive exact lookup used by
|
||||||
|
ordinary execution.
|
||||||
|
|
||||||
|
Lookup applies the engine's normal profile-source precedence:
|
||||||
|
|
||||||
|
- programmatic profiles supplied through `WithProfiles`;
|
||||||
|
- the configured file, directory, or `fs.FS` profile source; and
|
||||||
|
- the built-in profile catalog.
|
||||||
|
|
||||||
|
A valid higher-precedence match shadows a lower-precedence profile with the
|
||||||
|
same ID. A malformed or unreadable higher-precedence match fails rather than
|
||||||
|
silently falling back. The
|
||||||
|
[profile format reference](../formats.md#profile-definitions) remains the
|
||||||
|
canonical owner of profile-source and file-format behavior.
|
||||||
|
|
||||||
|
Inspection never derives a profile ID from a prompt's `default_profile`; the
|
||||||
|
caller is inspecting one explicitly named profile.
|
||||||
|
|
||||||
|
## Structural Resolution
|
||||||
|
|
||||||
|
Inspection loads and validates the selected profile, verifies that a named
|
||||||
|
backend exists in the engine's immutable backend registry, and applies normal
|
||||||
|
framework-default, backend, and profile precedence to produce the effective
|
||||||
|
target.
|
||||||
|
|
||||||
|
For the same engine state and profile ID, with no per-run execution override,
|
||||||
|
the inspected target must match the target that ordinary preparation would
|
||||||
|
resolve before applying request credentials and checking their availability.
|
||||||
|
This equivalence must use one shared resolution path rather than a second set
|
||||||
|
of precedence rules.
|
||||||
|
|
||||||
|
Structural resolution includes:
|
||||||
|
|
||||||
|
- backend routing identity;
|
||||||
|
- endpoint and model;
|
||||||
|
- sampling, token, timeout, service-tier, and reasoning settings;
|
||||||
|
- the effective credential environment-variable name or direct-key
|
||||||
|
requirement; and
|
||||||
|
- deeply copied provider-specific extra parameters.
|
||||||
|
|
||||||
|
Inspection returns the effective target rather than a raw profile definition.
|
||||||
|
This keeps framework and backend defaults visible to consumers without
|
||||||
|
creating a second public profile-loading interface.
|
||||||
|
|
||||||
|
The result does not include request-override presence because no
|
||||||
|
`ExecutionTargetOverride` participates in inspection.
|
||||||
|
|
||||||
|
## Credentials And Sensitive Data
|
||||||
|
|
||||||
|
Inspection reports credential requirements but never resolves, retains, or
|
||||||
|
returns a credential value.
|
||||||
|
|
||||||
|
The operation does not read the named environment variable and succeeds when
|
||||||
|
that variable is absent or blank. It accepts neither a direct API key nor an
|
||||||
|
API-key environment override. Consumers decide whether credential availability
|
||||||
|
must be enforced during application configuration, while `Prepare`,
|
||||||
|
`PrepareExecution`, `Run`, and `RunPrepared` retain their execution-time
|
||||||
|
credential contracts.
|
||||||
|
|
||||||
|
Error messages, formatting, and returned values must not expose environment
|
||||||
|
values or other resolved secrets. Existing restrictions against raw API keys
|
||||||
|
in profile sources remain unchanged.
|
||||||
|
|
||||||
|
## Ownership, Consistency, And Concurrency
|
||||||
|
|
||||||
|
Each successful call returns a caller-owned snapshot. Mutating the returned
|
||||||
|
target or any nested extra-parameter map or slice cannot affect the engine,
|
||||||
|
later inspection, or later execution.
|
||||||
|
|
||||||
|
Inspection is safe to call concurrently under the engine's existing immutable
|
||||||
|
registry and repository contracts. It does not mutate profile sources or
|
||||||
|
cache a result globally.
|
||||||
|
|
||||||
|
For filesystem-backed sources, an inspection describes the state observed by
|
||||||
|
that call. It does not freeze the profile for a later `Run`; a source may
|
||||||
|
change between operations. Consumers requiring an exact preflight-to-execution
|
||||||
|
snapshot should use the existing prepared-execution workflow.
|
||||||
|
|
||||||
|
## Errors And Cancellation
|
||||||
|
|
||||||
|
The operation uses existing public error categories:
|
||||||
|
|
||||||
|
- a blank profile ID matches `ErrInvalidRequest`;
|
||||||
|
- an absent exact ID matches `ErrProfileNotFound` and not `ErrProfileLoad`;
|
||||||
|
- read, decode, validation, and source-selection failures match
|
||||||
|
`ErrProfileLoad`; and
|
||||||
|
- an unknown referenced backend or an invalid structurally resolved target
|
||||||
|
matches `ErrProfileLoad`.
|
||||||
|
|
||||||
|
Errors should preserve useful underlying collaborator and context identities
|
||||||
|
through `errors.Is` where the existing facade does so, without exposing
|
||||||
|
internal package types. Context cancellation governs inspection and no partial
|
||||||
|
inspection result is returned on failure.
|
||||||
|
|
||||||
|
Missing credential values are not inspection errors. The operation cannot
|
||||||
|
return capacity or model-generation failures because it performs neither
|
||||||
|
backend admission nor generation.
|
||||||
|
|
||||||
|
## Compatibility And Boundaries
|
||||||
|
|
||||||
|
This feature is additive. Existing profile formats, source precedence,
|
||||||
|
backend registration, `Prepare`, prepared execution, and `Run` behavior remain
|
||||||
|
unchanged.
|
||||||
|
|
||||||
|
The method belongs on the root `Engine` facade. Profile repositories and the
|
||||||
|
backend registry remain internal implementation details, and no new public
|
||||||
|
repository interface is introduced.
|
||||||
|
|
||||||
|
Inspection does not require prompt lookup, rendering, artifact loading, schema
|
||||||
|
loading, validation, backend-capacity admission, or model-client access.
|
||||||
|
Engine construction retains its ordinary configuration requirements; this
|
||||||
|
feature does not introduce a separate profile-only engine.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
The completed documentation set has these ownership boundaries:
|
||||||
|
|
||||||
|
- exported declarations and GoDoc own the exact method, result, ownership,
|
||||||
|
credential, error, and cancellation contracts;
|
||||||
|
- the promptkit consumer guide explains configuration-time profile inspection
|
||||||
|
and distinguishes it from `Prepare` and prepared execution;
|
||||||
|
- the profile format reference continues to own profile fields and source
|
||||||
|
precedence; and
|
||||||
|
- internal documentation describes shared profile and target resolution
|
||||||
|
without duplicating public contracts.
|
||||||
|
|
||||||
|
## Non-Goals
|
||||||
|
|
||||||
|
This work does not include:
|
||||||
|
|
||||||
|
- enumerating or searching profiles;
|
||||||
|
- returning raw profile definitions or profile source paths;
|
||||||
|
- accepting per-run execution overrides, direct API keys, or API-key
|
||||||
|
environment overrides;
|
||||||
|
- checking environment-variable contents or other credential availability;
|
||||||
|
- semantic execution-target fingerprints or profile hashes;
|
||||||
|
- prompt-definition inspection or prompt default-profile resolution;
|
||||||
|
- full-corpus validation across every profile source;
|
||||||
|
- freezing a filesystem-backed profile for later execution;
|
||||||
|
- exposing backend concurrency limits, queue state, or capacity policy;
|
||||||
|
- model generation, provider health checks, or endpoint connectivity tests;
|
||||||
|
- dynamic backend or profile registration after engine construction; or
|
||||||
|
- changing current profile, backend, prepared-execution, or stable JSON
|
||||||
|
contracts.
|
||||||
|
|
||||||
|
## Target End State
|
||||||
|
|
||||||
|
After this work:
|
||||||
|
|
||||||
|
- consumers can validate one configured profile without inventing a prompt or
|
||||||
|
placeholder inputs;
|
||||||
|
- lookup observes ordinary programmatic, configured-source, and built-in
|
||||||
|
precedence;
|
||||||
|
- a successful result proves that the profile exists, is valid, references a
|
||||||
|
registered backend when applicable, and resolves to a structurally valid
|
||||||
|
effective target;
|
||||||
|
- the inspected target matches ordinary preparation for the same profile and
|
||||||
|
engine state before per-run overrides and credential availability checks;
|
||||||
|
- credential requirements are visible without reading or exposing credential
|
||||||
|
values;
|
||||||
|
- returned targets and nested data are caller-owned;
|
||||||
|
- inspection performs no rendering, source loading unrelated to the profile,
|
||||||
|
capacity admission, or model work;
|
||||||
|
- existing execution workflows and compatibility contracts remain unchanged;
|
||||||
|
and
|
||||||
|
- Promptkit owns reusable profile validation while downstream applications
|
||||||
|
retain configuration policy, persistence, logging, and credential-timing
|
||||||
|
decisions.
|
||||||
@@ -185,8 +185,8 @@ and move failures ahead of weather collection.
|
|||||||
|
|
||||||
## Priority 3: Prompt-Independent Profile Inspection
|
## Priority 3: Prompt-Independent Profile Inspection
|
||||||
|
|
||||||
**Disposition:** Accepted into the
|
**Disposition:** Covered by the accepted
|
||||||
[future catalog](future.md#prompt-independent-profile-inspection).
|
[prompt-independent profile inspection](profile-inspection.md) roadmap.
|
||||||
|
|
||||||
### Downstream need
|
### Downstream need
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user