promptkit/docs/roadmap/profile-inspection.md

Prompt-Independent Profile Inspection

Status: Accepted.

Purpose

Allow consumers to look up one execution profile by ID and inspect its structurally resolved model target without selecting a prompt, supplying placeholder inputs, checking credential availability, or invoking a model.

This provides a direct configuration-validation boundary for Notarius and Weatherreporter.

Motivation

Both downstream consumers need to reject invalid configured profile IDs before starting application work. They currently have to construct a synthetic prompt and call Engine.Prepare merely to exercise profile loading, backend lookup, and execution-target resolution.

That workaround couples profile validation to unrelated prompt definitions, fixture inputs, rendering, schema behavior, and current credential availability. Promptkit already owns profile precedence, backend membership, and target resolution, so it should expose that cohesive capability directly.

Consumer Workflow

The target public workflow is:

inspection, err := engine.InspectProfile(ctx, profileID)
if err != nil {
	// Reject or report the configured profile.
	return
}

target := inspection.EffectiveModelParams
if target.APIKeyEnv != "" {
	// Apply application policy for the named environment variable.
}

The target public surface is:

type ProfileInspection struct {
	ProfileID           string
	EffectiveModelParams ExecutionTarget
	APIKeyRequired      bool
}

func (e *Engine) InspectProfile(
	ctx context.Context,
	profileID string,
) (*ProfileInspection, error)

The declarations and GoDoc will own the exact implemented contract. The important public shape is exact lookup through the existing engine, one caller-owned inspection value, the resolved ExecutionTarget, and an explicit signal for a direct API-key requirement.

EffectiveModelParams.BackendID identifies the selected registered backend and remains empty for endpoint-only profiles. EffectiveModelParams.APIKeyEnv reports the effective credential environment variable name. APIKeyRequired reports that the profile requires a direct request credential instead. These states are mutually exclusive after normal profile and backend precedence is applied.

ProfileInspection does not need a stable JSON representation. Consumers that persist application configuration or diagnostics can select the fields their own format requires.

Lookup And Precedence

InspectProfile requires a non-blank explicit profile ID. It trims surrounding whitespace and otherwise performs the same case-sensitive exact lookup used by ordinary execution.

Lookup applies the engine's normal profile-source precedence:

• programmatic profiles supplied through WithProfiles;
• the configured file, directory, or fs.FS profile source; and
• the built-in profile catalog.

A valid higher-precedence match shadows a lower-precedence profile with the same ID. A malformed or unreadable higher-precedence match fails rather than silently falling back. The profile format reference remains the canonical owner of profile-source and file-format behavior.

Inspection never derives a profile ID from a prompt's default_profile; the caller is inspecting one explicitly named profile.

Structural Resolution

Inspection loads and validates the selected profile, verifies that a named backend exists in the engine's immutable backend registry, and applies normal framework-default, backend, and profile precedence to produce the effective target.

For the same engine state and profile ID, with no per-run execution override, the inspected target must match the target that ordinary preparation would resolve before applying request credentials and checking their availability. This equivalence must use one shared resolution path rather than a second set of precedence rules.

Structural resolution includes:

• backend routing identity;
• endpoint and model;
• sampling, token, timeout, service-tier, and reasoning settings;
• the effective credential environment-variable name or direct-key requirement; and
• deeply copied provider-specific extra parameters.

Inspection returns the effective target rather than a raw profile definition. This keeps framework and backend defaults visible to consumers without creating a second public profile-loading interface.

The result does not include request-override presence because no ExecutionTargetOverride participates in inspection.

Credentials And Sensitive Data

Inspection reports credential requirements but never resolves, retains, or returns a credential value.

The operation does not read the named environment variable and succeeds when that variable is absent or blank. It accepts neither a direct API key nor an API-key environment override. Consumers decide whether credential availability must be enforced during application configuration, while Prepare, PrepareExecution, Run, and RunPrepared retain their execution-time credential contracts.

Error messages, formatting, and returned values must not expose environment values or other resolved secrets. Existing restrictions against raw API keys in profile sources remain unchanged.

Ownership, Consistency, And Concurrency

Each successful call returns a caller-owned snapshot. Mutating the returned target or any nested extra-parameter map or slice cannot affect the engine, later inspection, or later execution.

Inspection is safe to call concurrently under the engine's existing immutable registry and repository contracts. It does not mutate profile sources or cache a result globally.

For filesystem-backed sources, an inspection describes the state observed by that call. It does not freeze the profile for a later Run; a source may change between operations. Consumers requiring an exact preflight-to-execution snapshot should use the existing prepared-execution workflow.

Errors And Cancellation

The operation uses existing public error categories:

• a blank profile ID matches ErrInvalidRequest;
• an absent exact ID matches ErrProfileNotFound and not ErrProfileLoad;
• read, decode, validation, and source-selection failures match ErrProfileLoad; and
• an unknown referenced backend or an invalid structurally resolved target matches ErrProfileLoad.

Errors should preserve useful underlying collaborator and context identities through errors.Is where the existing facade does so, without exposing internal package types. Context cancellation governs inspection and no partial inspection result is returned on failure.

Missing credential values are not inspection errors. The operation cannot return capacity or model-generation failures because it performs neither backend admission nor generation.

Compatibility And Boundaries

This feature is additive. Existing profile formats, source precedence, backend registration, Prepare, prepared execution, and Run behavior remain unchanged.

The method belongs on the root Engine facade. Profile repositories and the backend registry remain internal implementation details, and no new public repository interface is introduced.

Inspection does not require prompt lookup, rendering, artifact loading, schema loading, validation, backend-capacity admission, or model-client access. Engine construction retains its ordinary configuration requirements; this feature does not introduce a separate profile-only engine.

Documentation

The completed documentation set has these ownership boundaries:

• exported declarations and GoDoc own the exact method, result, ownership, credential, error, and cancellation contracts;
• the promptkit consumer guide explains configuration-time profile inspection and distinguishes it from Prepare and prepared execution;
• the profile format reference continues to own profile fields and source precedence; and
• internal documentation describes shared profile and target resolution without duplicating public contracts.

Non-Goals

This work does not include:

• enumerating or searching profiles;
• returning raw profile definitions or profile source paths;
• accepting per-run execution overrides, direct API keys, or API-key environment overrides;
• checking environment-variable contents or other credential availability;
• semantic execution-target fingerprints or profile hashes;
• prompt-definition inspection or prompt default-profile resolution;
• full-corpus validation across every profile source;
• freezing a filesystem-backed profile for later execution;
• exposing backend concurrency limits, queue state, or capacity policy;
• model generation, provider health checks, or endpoint connectivity tests;
• dynamic backend or profile registration after engine construction; or
• changing current profile, backend, prepared-execution, or stable JSON contracts.

Target End State

After this work:

• consumers can validate one configured profile without inventing a prompt or placeholder inputs;
• lookup observes ordinary programmatic, configured-source, and built-in precedence;
• a successful result proves that the profile exists, is valid, references a registered backend when applicable, and resolves to a structurally valid effective target;
• the inspected target matches ordinary preparation for the same profile and engine state before per-run overrides and credential availability checks;
• credential requirements are visible without reading or exposing credential values;
• returned targets and nested data are caller-owned;
• inspection performs no rendering, source loading unrelated to the profile, capacity admission, or model work;
• existing execution workflows and compatibility contracts remain unchanged; and
• Promptkit owns reusable profile validation while downstream applications retain configuration policy, persistence, logging, and credential-timing decisions.