Add feature roadmap and implementation plan for profile inspection API
This commit is contained in:
@@ -33,26 +33,10 @@ consumers.
|
||||
|
||||
## Ideas
|
||||
|
||||
Executable preparation handles have been selected for active planning in the
|
||||
[focused feature roadmap](prepared-execution.md). The remaining ideas are
|
||||
Prompt-independent profile inspection has been selected for active planning in
|
||||
the [focused feature roadmap](profile-inspection.md). The remaining ideas are
|
||||
still available for future selection.
|
||||
|
||||
### Prompt-independent profile inspection
|
||||
|
||||
Provide exact profile lookup and structural resolution without requiring a
|
||||
synthetic prompt, placeholder inputs, or model generation. This shared need is
|
||||
described by
|
||||
[Notarius](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
|
||||
and
|
||||
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-3-prompt-independent-profile-inspection).
|
||||
|
||||
- Apply ordinary built-in, file-backed, and programmatic profile precedence.
|
||||
- Validate referenced backend membership and the structurally resolved
|
||||
execution target.
|
||||
- Report credential requirements and environment-variable names without
|
||||
exposing credential values or requiring current credential availability.
|
||||
- Support exact lookup by profile ID; enumeration is not required initially.
|
||||
|
||||
### Prompt-definition inspection
|
||||
|
||||
Provide exact prompt-definition lookup without rendering, placeholder inputs,
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -88,8 +88,8 @@ corresponds atomically to the actual execution.
|
||||
|
||||
## Priority 2: Prompt-Independent Profile Inspection
|
||||
|
||||
**Disposition:** Accepted into the
|
||||
[future catalog](future.md#prompt-independent-profile-inspection).
|
||||
**Disposition:** Covered by the accepted
|
||||
[prompt-independent profile inspection](profile-inspection.md) roadmap.
|
||||
|
||||
### Downstream need
|
||||
|
||||
|
||||
246
docs/roadmap/profile-inspection.md
Normal file
246
docs/roadmap/profile-inspection.md
Normal file
@@ -0,0 +1,246 @@
|
||||
# Prompt-Independent Profile Inspection
|
||||
|
||||
**Status:** Accepted.
|
||||
|
||||
## Purpose
|
||||
|
||||
Allow consumers to look up one execution profile by ID and inspect its
|
||||
structurally resolved model target without selecting a prompt, supplying
|
||||
placeholder inputs, checking credential availability, or invoking a model.
|
||||
|
||||
This provides a direct configuration-validation boundary for
|
||||
[Notarius](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
|
||||
and
|
||||
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-3-prompt-independent-profile-inspection).
|
||||
|
||||
## Motivation
|
||||
|
||||
Both downstream consumers need to reject invalid configured profile IDs before
|
||||
starting application work. They currently have to construct a synthetic prompt
|
||||
and call `Engine.Prepare` merely to exercise profile loading, backend lookup,
|
||||
and execution-target resolution.
|
||||
|
||||
That workaround couples profile validation to unrelated prompt definitions,
|
||||
fixture inputs, rendering, schema behavior, and current credential
|
||||
availability. Promptkit already owns profile precedence, backend membership,
|
||||
and target resolution, so it should expose that cohesive capability directly.
|
||||
|
||||
## Consumer Workflow
|
||||
|
||||
The target public workflow is:
|
||||
|
||||
```go
|
||||
inspection, err := engine.InspectProfile(ctx, profileID)
|
||||
if err != nil {
|
||||
// Reject or report the configured profile.
|
||||
return
|
||||
}
|
||||
|
||||
target := inspection.EffectiveModelParams
|
||||
if target.APIKeyEnv != "" {
|
||||
// Apply application policy for the named environment variable.
|
||||
}
|
||||
```
|
||||
|
||||
The target public surface is:
|
||||
|
||||
```go
|
||||
type ProfileInspection struct {
|
||||
ProfileID string
|
||||
EffectiveModelParams ExecutionTarget
|
||||
APIKeyRequired bool
|
||||
}
|
||||
|
||||
func (e *Engine) InspectProfile(
|
||||
ctx context.Context,
|
||||
profileID string,
|
||||
) (*ProfileInspection, error)
|
||||
```
|
||||
|
||||
The declarations and GoDoc will own the exact implemented contract. The
|
||||
important public shape is exact lookup through the existing engine, one
|
||||
caller-owned inspection value, the resolved `ExecutionTarget`, and an explicit
|
||||
signal for a direct API-key requirement.
|
||||
|
||||
`EffectiveModelParams.BackendID` identifies the selected registered backend
|
||||
and remains empty for endpoint-only profiles.
|
||||
`EffectiveModelParams.APIKeyEnv` reports the effective credential environment
|
||||
variable name. `APIKeyRequired` reports that the profile requires a direct
|
||||
request credential instead. These states are mutually exclusive after normal
|
||||
profile and backend precedence is applied.
|
||||
|
||||
`ProfileInspection` does not need a stable JSON representation. Consumers that
|
||||
persist application configuration or diagnostics can select the fields their
|
||||
own format requires.
|
||||
|
||||
## Lookup And Precedence
|
||||
|
||||
`InspectProfile` requires a non-blank explicit profile ID. It trims surrounding
|
||||
whitespace and otherwise performs the same case-sensitive exact lookup used by
|
||||
ordinary execution.
|
||||
|
||||
Lookup applies the engine's normal profile-source precedence:
|
||||
|
||||
- programmatic profiles supplied through `WithProfiles`;
|
||||
- the configured file, directory, or `fs.FS` profile source; and
|
||||
- the built-in profile catalog.
|
||||
|
||||
A valid higher-precedence match shadows a lower-precedence profile with the
|
||||
same ID. A malformed or unreadable higher-precedence match fails rather than
|
||||
silently falling back. The
|
||||
[profile format reference](../formats.md#profile-definitions) remains the
|
||||
canonical owner of profile-source and file-format behavior.
|
||||
|
||||
Inspection never derives a profile ID from a prompt's `default_profile`; the
|
||||
caller is inspecting one explicitly named profile.
|
||||
|
||||
## Structural Resolution
|
||||
|
||||
Inspection loads and validates the selected profile, verifies that a named
|
||||
backend exists in the engine's immutable backend registry, and applies normal
|
||||
framework-default, backend, and profile precedence to produce the effective
|
||||
target.
|
||||
|
||||
For the same engine state and profile ID, with no per-run execution override,
|
||||
the inspected target must match the target that ordinary preparation would
|
||||
resolve before applying request credentials and checking their availability.
|
||||
This equivalence must use one shared resolution path rather than a second set
|
||||
of precedence rules.
|
||||
|
||||
Structural resolution includes:
|
||||
|
||||
- backend routing identity;
|
||||
- endpoint and model;
|
||||
- sampling, token, timeout, service-tier, and reasoning settings;
|
||||
- the effective credential environment-variable name or direct-key
|
||||
requirement; and
|
||||
- deeply copied provider-specific extra parameters.
|
||||
|
||||
Inspection returns the effective target rather than a raw profile definition.
|
||||
This keeps framework and backend defaults visible to consumers without
|
||||
creating a second public profile-loading interface.
|
||||
|
||||
The result does not include request-override presence because no
|
||||
`ExecutionTargetOverride` participates in inspection.
|
||||
|
||||
## Credentials And Sensitive Data
|
||||
|
||||
Inspection reports credential requirements but never resolves, retains, or
|
||||
returns a credential value.
|
||||
|
||||
The operation does not read the named environment variable and succeeds when
|
||||
that variable is absent or blank. It accepts neither a direct API key nor an
|
||||
API-key environment override. Consumers decide whether credential availability
|
||||
must be enforced during application configuration, while `Prepare`,
|
||||
`PrepareExecution`, `Run`, and `RunPrepared` retain their execution-time
|
||||
credential contracts.
|
||||
|
||||
Error messages, formatting, and returned values must not expose environment
|
||||
values or other resolved secrets. Existing restrictions against raw API keys
|
||||
in profile sources remain unchanged.
|
||||
|
||||
## Ownership, Consistency, And Concurrency
|
||||
|
||||
Each successful call returns a caller-owned snapshot. Mutating the returned
|
||||
target or any nested extra-parameter map or slice cannot affect the engine,
|
||||
later inspection, or later execution.
|
||||
|
||||
Inspection is safe to call concurrently under the engine's existing immutable
|
||||
registry and repository contracts. It does not mutate profile sources or
|
||||
cache a result globally.
|
||||
|
||||
For filesystem-backed sources, an inspection describes the state observed by
|
||||
that call. It does not freeze the profile for a later `Run`; a source may
|
||||
change between operations. Consumers requiring an exact preflight-to-execution
|
||||
snapshot should use the existing prepared-execution workflow.
|
||||
|
||||
## Errors And Cancellation
|
||||
|
||||
The operation uses existing public error categories:
|
||||
|
||||
- a blank profile ID matches `ErrInvalidRequest`;
|
||||
- an absent exact ID matches `ErrProfileNotFound` and not `ErrProfileLoad`;
|
||||
- read, decode, validation, and source-selection failures match
|
||||
`ErrProfileLoad`; and
|
||||
- an unknown referenced backend or an invalid structurally resolved target
|
||||
matches `ErrProfileLoad`.
|
||||
|
||||
Errors should preserve useful underlying collaborator and context identities
|
||||
through `errors.Is` where the existing facade does so, without exposing
|
||||
internal package types. Context cancellation governs inspection and no partial
|
||||
inspection result is returned on failure.
|
||||
|
||||
Missing credential values are not inspection errors. The operation cannot
|
||||
return capacity or model-generation failures because it performs neither
|
||||
backend admission nor generation.
|
||||
|
||||
## Compatibility And Boundaries
|
||||
|
||||
This feature is additive. Existing profile formats, source precedence,
|
||||
backend registration, `Prepare`, prepared execution, and `Run` behavior remain
|
||||
unchanged.
|
||||
|
||||
The method belongs on the root `Engine` facade. Profile repositories and the
|
||||
backend registry remain internal implementation details, and no new public
|
||||
repository interface is introduced.
|
||||
|
||||
Inspection does not require prompt lookup, rendering, artifact loading, schema
|
||||
loading, validation, backend-capacity admission, or model-client access.
|
||||
Engine construction retains its ordinary configuration requirements; this
|
||||
feature does not introduce a separate profile-only engine.
|
||||
|
||||
## Documentation
|
||||
|
||||
The completed documentation set has these ownership boundaries:
|
||||
|
||||
- exported declarations and GoDoc own the exact method, result, ownership,
|
||||
credential, error, and cancellation contracts;
|
||||
- the promptkit consumer guide explains configuration-time profile inspection
|
||||
and distinguishes it from `Prepare` and prepared execution;
|
||||
- the profile format reference continues to own profile fields and source
|
||||
precedence; and
|
||||
- internal documentation describes shared profile and target resolution
|
||||
without duplicating public contracts.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This work does not include:
|
||||
|
||||
- enumerating or searching profiles;
|
||||
- returning raw profile definitions or profile source paths;
|
||||
- accepting per-run execution overrides, direct API keys, or API-key
|
||||
environment overrides;
|
||||
- checking environment-variable contents or other credential availability;
|
||||
- semantic execution-target fingerprints or profile hashes;
|
||||
- prompt-definition inspection or prompt default-profile resolution;
|
||||
- full-corpus validation across every profile source;
|
||||
- freezing a filesystem-backed profile for later execution;
|
||||
- exposing backend concurrency limits, queue state, or capacity policy;
|
||||
- model generation, provider health checks, or endpoint connectivity tests;
|
||||
- dynamic backend or profile registration after engine construction; or
|
||||
- changing current profile, backend, prepared-execution, or stable JSON
|
||||
contracts.
|
||||
|
||||
## Target End State
|
||||
|
||||
After this work:
|
||||
|
||||
- consumers can validate one configured profile without inventing a prompt or
|
||||
placeholder inputs;
|
||||
- lookup observes ordinary programmatic, configured-source, and built-in
|
||||
precedence;
|
||||
- a successful result proves that the profile exists, is valid, references a
|
||||
registered backend when applicable, and resolves to a structurally valid
|
||||
effective target;
|
||||
- the inspected target matches ordinary preparation for the same profile and
|
||||
engine state before per-run overrides and credential availability checks;
|
||||
- credential requirements are visible without reading or exposing credential
|
||||
values;
|
||||
- returned targets and nested data are caller-owned;
|
||||
- inspection performs no rendering, source loading unrelated to the profile,
|
||||
capacity admission, or model work;
|
||||
- existing execution workflows and compatibility contracts remain unchanged;
|
||||
and
|
||||
- Promptkit owns reusable profile validation while downstream applications
|
||||
retain configuration policy, persistence, logging, and credential-timing
|
||||
decisions.
|
||||
@@ -185,8 +185,8 @@ and move failures ahead of weather collection.
|
||||
|
||||
## Priority 3: Prompt-Independent Profile Inspection
|
||||
|
||||
**Disposition:** Accepted into the
|
||||
[future catalog](future.md#prompt-independent-profile-inspection).
|
||||
**Disposition:** Covered by the accepted
|
||||
[prompt-independent profile inspection](profile-inspection.md) roadmap.
|
||||
|
||||
### Downstream need
|
||||
|
||||
|
||||
Reference in New Issue
Block a user