Add feature roadmaps with wishlists from downstream consumers
This commit is contained in:
345
docs/roadmap/notarius-promptkit-wishlist.md
Normal file
345
docs/roadmap/notarius-promptkit-wishlist.md
Normal file
@@ -0,0 +1,345 @@
|
||||
# Notarius PromptKit Wishlist
|
||||
|
||||
## Purpose
|
||||
|
||||
This document records features and interface changes that would be useful
|
||||
additions to PromptKit from the perspective of the maintainers of Notarius, a
|
||||
downstream application that consumes PromptKit.
|
||||
|
||||
PromptKit v0.3.0 provides the capabilities Notarius currently needs. None of
|
||||
the ideas below blocks current Notarius development. They are opportunities to
|
||||
reduce downstream workarounds, improve integration correctness, and make
|
||||
PromptKit more ergonomic for applications with configuration validation,
|
||||
debugging, checkpointing, and operational-observability requirements.
|
||||
|
||||
The examples are API sketches intended to communicate the desired capability,
|
||||
not prescriptive names or finalized Go contracts.
|
||||
|
||||
## Priority 1: Atomic Execution With Prepared Details
|
||||
|
||||
### Downstream need
|
||||
|
||||
Notarius needs both:
|
||||
|
||||
- the completed `RunResult`; and
|
||||
- the rendered messages, effective output contract, hashes, and other
|
||||
preparation details exposed by `PreparedRun`.
|
||||
|
||||
Notarius uses the prepared details to construct redaction-aware debug bundles
|
||||
and retain enough information to diagnose model behavior.
|
||||
|
||||
### Current integration
|
||||
|
||||
Notarius currently calls `Engine.Prepare` and then `Engine.Run` with the same
|
||||
request. Because `Run` performs preparation internally, a successful request
|
||||
resolves and prepares the same work twice.
|
||||
|
||||
This duplicates profile resolution, input hashing, schema loading, and prompt
|
||||
rendering. It also creates a theoretical consistency window in which a
|
||||
filesystem-backed prompt, profile, schema, or input could change between the
|
||||
explicit preparation and the preparation performed by `Run`.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Add an opt-in execution method that prepares exactly once and returns both the
|
||||
prepared details and completed result:
|
||||
|
||||
```go
|
||||
type RunReport struct {
|
||||
Prepared PreparedRun
|
||||
Result RunResult
|
||||
}
|
||||
|
||||
func (e *Engine) RunDetailed(
|
||||
ctx context.Context,
|
||||
req RunRequest,
|
||||
) (*RunReport, error)
|
||||
```
|
||||
|
||||
The exact names are flexible. The important contract is that preparation
|
||||
occurs once and that the returned prepared state describes the execution that
|
||||
produced the returned result.
|
||||
|
||||
Existing `Prepare` and `Run` behavior should remain available for consumers
|
||||
that need only one side of the operation.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Keep this API additive and preserve the existing simple `Run` workflow.
|
||||
- Return caller-owned copies under PromptKit's existing ownership rules.
|
||||
- Define whether any prepared details are available after an operational
|
||||
generation or validation error. Notarius does not require partial results
|
||||
for the initial use case, but an explicit contract would be valuable.
|
||||
- Preserve cancellation and backend-admission semantics.
|
||||
- Do not add all prepared content directly to `RunResult`. Rendered prompt
|
||||
content can be large and sensitive, and consumers should opt in to receiving
|
||||
it.
|
||||
|
||||
### Value to Notarius
|
||||
|
||||
This is the highest-value wishlist item. It would remove duplicate work from
|
||||
every successful PromptKit-backed call and ensure that retained debug material
|
||||
corresponds atomically to the actual execution.
|
||||
|
||||
## Priority 2: Prompt-Independent Profile Inspection
|
||||
|
||||
### Downstream need
|
||||
|
||||
Notarius validates configured pipeline profile IDs before beginning a run. It
|
||||
needs to determine whether:
|
||||
|
||||
- a profile exists;
|
||||
- its referenced backend is registered;
|
||||
- its execution target can be resolved; and
|
||||
- it declares a credential requirement that the application may need to
|
||||
enforce.
|
||||
|
||||
This validation should not require model generation.
|
||||
|
||||
### Current integration
|
||||
|
||||
Notarius constructs a synthetic prompt using `testing/fstest.MapFS`, supplies a
|
||||
dummy transcript, and calls `Engine.Prepare` solely to exercise profile and
|
||||
backend resolution. This works, but prompt preparation is serving as a
|
||||
substitute for a profile-inspection interface.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Add a prompt-independent profile-resolution API, for example:
|
||||
|
||||
```go
|
||||
type ResolvedProfile struct {
|
||||
ProfileID string
|
||||
BackendID string
|
||||
EffectiveTarget ExecutionTarget
|
||||
APIKeyEnv string
|
||||
}
|
||||
|
||||
func (e *Engine) ResolveProfile(
|
||||
ctx context.Context,
|
||||
profileID string,
|
||||
) (ResolvedProfile, error)
|
||||
```
|
||||
|
||||
The returned shape may differ, but it should provide enough information for a
|
||||
consumer to validate an explicit profile selection without inventing a prompt
|
||||
or supplying placeholder inputs.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Resolve built-in, file-backed, and programmatic profiles using normal
|
||||
PromptKit precedence.
|
||||
- Validate that a referenced backend registration exists.
|
||||
- Do not resolve, retain, or expose credential values.
|
||||
- Report credential requirements, such as an environment-variable name, so
|
||||
the consuming application can decide whether availability is required at
|
||||
configuration-validation time or only at execution time.
|
||||
- Return caller-owned values.
|
||||
- Preserve typed or sentinel error classification for missing and invalid
|
||||
profiles.
|
||||
- Consider accepting an `ExecutionTargetOverride` if consumers need to inspect
|
||||
the same effective target that a run-level override would produce.
|
||||
- Enumeration of all profiles is not required for the Notarius use case; exact
|
||||
lookup by ID is sufficient.
|
||||
|
||||
### Value to Notarius
|
||||
|
||||
This would eliminate a synthetic production-only prompt fixture and establish
|
||||
a direct, supported contract for configuration-time profile and backend
|
||||
validation.
|
||||
|
||||
## Priority 3: Semantic Execution-Target Fingerprints
|
||||
|
||||
### Downstream need
|
||||
|
||||
Notarius checkpoints model-backed pipeline stages. A checkpoint must not be
|
||||
reused when generation-affecting PromptKit configuration changes.
|
||||
|
||||
Notarius therefore needs a stable equality signal for the effective profile
|
||||
and backend target used by a pipeline.
|
||||
|
||||
### Current integration
|
||||
|
||||
Notarius currently constructs this identity itself from:
|
||||
|
||||
- a manually maintained marker for the PromptKit release and built-in profile
|
||||
catalog;
|
||||
- raw hashes of configured profile files; and
|
||||
- a separate hash of the configured conventional local-backend endpoint.
|
||||
|
||||
This is safe but conservative and coupled to PromptKit details. Raw file
|
||||
hashing also invalidates checkpoints for semantically irrelevant YAML changes,
|
||||
such as comments or formatting.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Expose an opaque semantic digest for a resolved profile and its effective
|
||||
generation target. It could be returned by the proposed profile-resolution
|
||||
API:
|
||||
|
||||
```go
|
||||
type ResolvedProfile struct {
|
||||
ProfileID string
|
||||
BackendID string
|
||||
EffectiveTarget ExecutionTarget
|
||||
ExecutionDigest string
|
||||
}
|
||||
```
|
||||
|
||||
Alternatively, PromptKit could expose a dedicated method such as
|
||||
`ProfileExecutionDigest(profileID)`.
|
||||
|
||||
### Desired equality semantics
|
||||
|
||||
The digest should change when generation-affecting state changes, including:
|
||||
|
||||
- resolved model and endpoint;
|
||||
- backend routing identity;
|
||||
- backend request defaults and extra parameters;
|
||||
- profile generation parameters; and
|
||||
- the semantic identity of any selected built-in profile.
|
||||
|
||||
The digest should not incorporate:
|
||||
|
||||
- credential values;
|
||||
- concurrency or queue capacity;
|
||||
- filesystem source paths;
|
||||
- YAML comments or formatting; or
|
||||
- other settings that affect scheduling or source representation without
|
||||
changing the generation target.
|
||||
|
||||
The credential environment-variable name may need to participate if changing
|
||||
it can select a materially different provider account or target. PromptKit
|
||||
should define this deliberately while continuing to exclude the resolved
|
||||
secret value.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Treat the digest as an opaque equality value rather than a public encoding
|
||||
of internal structures.
|
||||
- Document which categories of change affect equality.
|
||||
- Include a versioned semantic marker internally so PromptKit can deliberately
|
||||
invalidate old digests when its resolution semantics change.
|
||||
- Prefer a per-profile digest over a digest of every profile known to an
|
||||
engine. Notarius generally knows which profiles a resolved pipeline uses.
|
||||
- Do not require consumers to know PromptKit's built-in catalog version.
|
||||
|
||||
### Value to Notarius
|
||||
|
||||
This would let Notarius remove its PromptKit release marker and raw
|
||||
profile-source fingerprinting, reduce unnecessary checkpoint invalidation, and
|
||||
delegate execution-target equality to the component that owns target
|
||||
resolution.
|
||||
|
||||
## Priority 4: Structured Capacity Errors
|
||||
|
||||
### Downstream need
|
||||
|
||||
Notarius translates PromptKit backend-capacity rejection into a
|
||||
provider-neutral application error. When multiple backends are active,
|
||||
operators would benefit from knowing which backend rejected admission without
|
||||
parsing an error string or exposing endpoint details.
|
||||
|
||||
### Current integration
|
||||
|
||||
PromptKit provides the useful `ErrCapacityExceeded` sentinel. Notarius can
|
||||
classify the failure reliably, but it retains only a sanitized diagnostic
|
||||
string as additional context.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Add a typed error that continues to match `ErrCapacityExceeded`:
|
||||
|
||||
```go
|
||||
type CapacityError struct {
|
||||
BackendID string
|
||||
}
|
||||
|
||||
func (e *CapacityError) Is(target error) bool {
|
||||
return target == ErrCapacityExceeded
|
||||
}
|
||||
```
|
||||
|
||||
The exact implementation may use `Unwrap` or another idiomatic mechanism. The
|
||||
important properties are compatibility with `errors.Is` and discoverability
|
||||
through `errors.As`.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Include the stable backend ID.
|
||||
- Do not expose the backend endpoint, credential environment, credential
|
||||
value, request content, or other sensitive configuration.
|
||||
- Consider including the configured concurrency and queue limits if they are
|
||||
useful and safe, but backend identity alone provides most of the downstream
|
||||
value.
|
||||
- Add a retry delay only if PromptKit can provide a meaningful value. A full
|
||||
queue does not necessarily imply a reliable `Retry-After` duration.
|
||||
- Keep retry and backoff policy with the consuming application. PromptKit
|
||||
should classify the admission failure rather than silently retry it.
|
||||
|
||||
### Value to Notarius
|
||||
|
||||
This would improve operational diagnostics and future metrics while preserving
|
||||
the provider-neutral error boundary used by Notarius.
|
||||
|
||||
## Capabilities PromptKit Already Provides Well
|
||||
|
||||
The current PromptKit boundary is sufficient for Notarius's implemented
|
||||
behavior. In particular, PromptKit already provides:
|
||||
|
||||
- filesystem, `fs.FS`, and programmatic prompt, profile, and schema sources;
|
||||
- offline preparation without model execution;
|
||||
- structured output and content validation;
|
||||
- direct session propagation;
|
||||
- tri-state per-run reasoning overrides;
|
||||
- selected profile, backend, model, endpoint, effective parameters, hashes, and
|
||||
token-usage provenance;
|
||||
- endpoint-only profiles;
|
||||
- the conventional `local` backend helper;
|
||||
- arbitrary engine-scoped `Backend` registrations;
|
||||
- backend authentication environment names, extra parameters, concurrency
|
||||
limits, and queue-capacity policies;
|
||||
- provider-client and artifact-reader extension interfaces;
|
||||
- context cancellation; and
|
||||
- useful public error sentinels, including profile absence and capacity
|
||||
exhaustion.
|
||||
|
||||
The wishlist does not imply that Notarius needs PromptKit to broaden its core
|
||||
responsibilities. It primarily asks for more direct access to information and
|
||||
operations that PromptKit already computes internally.
|
||||
|
||||
## Responsibilities That Should Remain In Notarius
|
||||
|
||||
The following concerns belong to the downstream application and should not
|
||||
move into PromptKit for the sake of Notarius:
|
||||
|
||||
- pipeline staging, dependencies, and generated references;
|
||||
- application-wide scheduling across providers and backends;
|
||||
- module and validation retry policy;
|
||||
- checkpoints, resume, and recomputation;
|
||||
- durable run artifacts and manifests;
|
||||
- D&D prompts, schemas, extractors, validators, and normalizers;
|
||||
- Notarius configuration-file parsing and precedence;
|
||||
- domain-specific prompt-cache prefix policy; and
|
||||
- application-specific redaction, retention, and debug-bundle policy.
|
||||
|
||||
PromptKit's complete `Backend` API already supports custom IDs, multiple local
|
||||
endpoints, authentication, extra parameters, and explicit queue policies.
|
||||
Whether Notarius exposes those capabilities in its own configuration is an
|
||||
application-policy decision, not an upstream PromptKit gap.
|
||||
|
||||
## Suggested Upstream Sequence
|
||||
|
||||
If the PromptKit team chooses to pursue these ideas, the most useful order for
|
||||
Notarius would be:
|
||||
|
||||
1. Add atomic execution that returns prepared details and the completed result.
|
||||
2. Add prompt-independent profile inspection.
|
||||
3. Add a semantic execution-target digest, preferably as part of profile
|
||||
inspection.
|
||||
4. Add a typed capacity error carrying backend identity.
|
||||
|
||||
The first two address concrete workarounds in current Notarius code. The third
|
||||
would improve checkpoint correctness and reduce coupling. The fourth is
|
||||
operational polish.
|
||||
|
||||
Reference in New Issue
Block a user