Document profile inspection API
This commit is contained in:
@@ -33,10 +33,6 @@ consumers.
|
||||
|
||||
## Ideas
|
||||
|
||||
Prompt-independent profile inspection has been selected for active planning in
|
||||
the [focused feature roadmap](profile-inspection.md). The remaining ideas are
|
||||
still available for future selection.
|
||||
|
||||
### Prompt-definition inspection
|
||||
|
||||
Provide exact prompt-definition lookup without rendering, placeholder inputs,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Prompt-Independent Profile Inspection Implementation Plan
|
||||
|
||||
**Status:** Ready for implementation.
|
||||
**Status:** Complete.
|
||||
|
||||
## Purpose
|
||||
|
||||
|
||||
@@ -88,8 +88,9 @@ corresponds atomically to the actual execution.
|
||||
|
||||
## Priority 2: Prompt-Independent Profile Inspection
|
||||
|
||||
**Disposition:** Covered by the accepted
|
||||
[prompt-independent profile inspection](profile-inspection.md) roadmap.
|
||||
**Disposition:** Implemented as
|
||||
[`Engine.InspectProfile`](../../engine.go). See the
|
||||
[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work).
|
||||
|
||||
### Downstream need
|
||||
|
||||
@@ -104,62 +105,22 @@ needs to determine whether:
|
||||
|
||||
This validation should not require model generation.
|
||||
|
||||
### Current integration
|
||||
### Previous integration
|
||||
|
||||
Notarius constructs a synthetic prompt using `testing/fstest.MapFS`, supplies a
|
||||
dummy transcript, and calls `Engine.Prepare` solely to exercise profile and
|
||||
backend resolution. This works, but prompt preparation is serving as a
|
||||
substitute for a profile-inspection interface.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Add a prompt-independent profile-resolution API, for example:
|
||||
|
||||
```go
|
||||
type ResolvedProfile struct {
|
||||
ProfileID string
|
||||
BackendID string
|
||||
EffectiveTarget ExecutionTarget
|
||||
APIKeyEnv string
|
||||
}
|
||||
|
||||
func (e *Engine) ResolveProfile(
|
||||
ctx context.Context,
|
||||
profileID string,
|
||||
) (ResolvedProfile, error)
|
||||
```
|
||||
|
||||
The returned shape may differ, but it should provide enough information for a
|
||||
consumer to validate an explicit profile selection without inventing a prompt
|
||||
or supplying placeholder inputs.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Resolve built-in, file-backed, and programmatic profiles using normal
|
||||
PromptKit precedence.
|
||||
- Validate that a referenced backend registration exists.
|
||||
- Do not resolve, retain, or expose credential values.
|
||||
- Report credential requirements, such as an environment-variable name, so
|
||||
the consuming application can decide whether availability is required at
|
||||
configuration-validation time or only at execution time.
|
||||
- Return caller-owned values.
|
||||
- Preserve typed or sentinel error classification for missing and invalid
|
||||
profiles.
|
||||
- Consider accepting an `ExecutionTargetOverride` if consumers need to inspect
|
||||
the same effective target that a run-level override would produce.
|
||||
- Enumeration of all profiles is not required for the Notarius use case; exact
|
||||
lookup by ID is sufficient.
|
||||
Before profile inspection was available, Notarius constructed a synthetic
|
||||
prompt using `testing/fstest.MapFS`, supplied a dummy transcript, and called
|
||||
`Engine.Prepare` solely to exercise profile and backend resolution.
|
||||
|
||||
### Value to Notarius
|
||||
|
||||
This would eliminate a synthetic production-only prompt fixture and establish
|
||||
a direct, supported contract for configuration-time profile and backend
|
||||
validation.
|
||||
The implemented interface eliminates a synthetic production-only prompt
|
||||
fixture and establishes a direct, supported contract for configuration-time
|
||||
profile and backend validation.
|
||||
|
||||
## Priority 3: Semantic Execution-Target Fingerprints
|
||||
|
||||
**Disposition:** Deferred until prompt-independent profile inspection defines
|
||||
the resolved target whose configuration identity would be fingerprinted.
|
||||
**Disposition:** Deferred pending a separate semantic-equality design for
|
||||
resolved execution targets.
|
||||
|
||||
### Downstream need
|
||||
|
||||
@@ -348,11 +309,9 @@ If the PromptKit team chooses to pursue these ideas, the most useful order for
|
||||
Notarius would be:
|
||||
|
||||
1. Add atomic execution that returns prepared details and the completed result.
|
||||
2. Add prompt-independent profile inspection.
|
||||
3. Add a semantic execution-target digest, preferably as part of profile
|
||||
2. Add a semantic execution-target digest, preferably alongside profile
|
||||
inspection.
|
||||
4. Add a typed capacity error carrying backend identity.
|
||||
3. Add a typed capacity error carrying backend identity.
|
||||
|
||||
The first two address concrete workarounds in current Notarius code. The third
|
||||
would improve checkpoint correctness and reduce coupling. The fourth is
|
||||
operational polish.
|
||||
The first addresses a concrete execution workaround. The second would improve
|
||||
checkpoint correctness and reduce coupling. The third is operational polish.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Prompt-Independent Profile Inspection
|
||||
|
||||
**Status:** Accepted.
|
||||
**Status:** Complete.
|
||||
|
||||
## Purpose
|
||||
|
||||
|
||||
@@ -185,51 +185,23 @@ and move failures ahead of weather collection.
|
||||
|
||||
## Priority 3: Prompt-Independent Profile Inspection
|
||||
|
||||
**Disposition:** Covered by the accepted
|
||||
[prompt-independent profile inspection](profile-inspection.md) roadmap.
|
||||
**Disposition:** Implemented as
|
||||
[`Engine.InspectProfile`](../../engine.go). See the
|
||||
[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work).
|
||||
|
||||
### Downstream need
|
||||
|
||||
Weatherreporter will allow operators to select an external PromptKit profile
|
||||
source and may allow an explicit profile override. It should reject a missing
|
||||
profile, unknown backend, malformed execution target, or unsatisfied credential
|
||||
requirement before collecting weather data or writing report artifacts.
|
||||
|
||||
### Current integration option
|
||||
|
||||
Weatherreporter can validate an explicit profile by preparing one embedded
|
||||
prompt with fixture input. Prompts that use their own default profiles can be
|
||||
validated during their normal preparation stage.
|
||||
|
||||
This couples configuration validation to one prompt and requires placeholder
|
||||
input even when only profile and backend resolution are relevant.
|
||||
|
||||
### Requested capability
|
||||
|
||||
The prompt-independent `ResolveProfile` API proposed by the
|
||||
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
|
||||
would satisfy this need. It should resolve built-in, file-backed, and
|
||||
programmatic profiles, validate backend membership, report credential
|
||||
requirements without resolving credential values, and preserve typed error
|
||||
classification.
|
||||
|
||||
### Additional Weatherreporter considerations
|
||||
|
||||
- An explicit application profile override should be inspectable without
|
||||
selecting a report prompt.
|
||||
- A prompt-definition inspection result may expose its default profile ID so
|
||||
Weatherreporter can inspect that profile separately.
|
||||
- Inspection should distinguish structural profile validity from current
|
||||
credential availability so configuration validation can apply explicit
|
||||
application policy.
|
||||
- An optional execution-target override should be considered only if it
|
||||
describes the same target that a later run will use.
|
||||
source and may allow an explicit profile override. It needs to reject a missing
|
||||
profile, unknown backend, or malformed execution target before collecting
|
||||
weather data or writing report artifacts, and to apply its own policy to a
|
||||
reported credential requirement.
|
||||
|
||||
### Value to Weatherreporter
|
||||
|
||||
This would improve fail-fast configuration validation and give operator-facing
|
||||
errors direct profile and backend context. It is valuable but not required for
|
||||
the initial migration.
|
||||
The implemented interface improves fail-fast configuration validation and gives
|
||||
operator-facing errors direct profile and backend context. It remains optional
|
||||
for the initial migration.
|
||||
|
||||
## Priority 4: Eager Source Validation
|
||||
|
||||
@@ -368,8 +340,8 @@ It should not block adoption.
|
||||
|
||||
### Semantic Execution-Target Fingerprints
|
||||
|
||||
**Disposition:** Deferred until prompt-independent profile inspection defines
|
||||
the resolved target whose configuration identity would be fingerprinted.
|
||||
**Disposition:** Deferred pending a separate semantic-equality design for
|
||||
resolved execution targets.
|
||||
|
||||
The semantic target digest proposed by the
|
||||
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-3-semantic-execution-target-fingerprints)
|
||||
@@ -432,11 +404,10 @@ Weatherreporter would be:
|
||||
1. Add executable preparation handles, ideally sharing implementation with an
|
||||
atomic detailed-run API.
|
||||
2. Add prompt-definition inspection.
|
||||
3. Add prompt-independent profile inspection.
|
||||
4. Consider eager source validation after evaluating whether the two exact
|
||||
3. Consider eager source validation after evaluating whether the two exact
|
||||
inspection APIs are sufficient.
|
||||
5. Add structured generation errors.
|
||||
6. Add structured capacity errors and semantic execution-target fingerprints
|
||||
4. Add structured generation errors.
|
||||
5. Add structured capacity errors and semantic execution-target fingerprints
|
||||
as lower-priority operational improvements.
|
||||
|
||||
The first item removes the only material integration workaround. Prompt and
|
||||
|
||||
Reference in New Issue
Block a user