promptkit/docs/roadmap/notarius-promptkit-wishlist.md

Notarius PromptKit Wishlist

Purpose

This document records features and interface changes that would be useful additions to PromptKit from the perspective of the maintainers of Notarius, a downstream application that consumes PromptKit.

PromptKit v0.3.0 provides the capabilities Notarius currently needs. None of the ideas below blocks current Notarius development. They are opportunities to reduce downstream workarounds, improve integration correctness, and make PromptKit more ergonomic for applications with configuration validation, debugging, checkpointing, and operational-observability requirements.

The examples are API sketches intended to communicate the desired capability, not prescriptive names or finalized Go contracts.

Priority 1: Atomic Execution With Prepared Details

Disposition: Covered by the accepted executable preparation handles catalog entry. The shared two-phase capability should provide the required single-preparation consistency; a separate RunDetailed method is not cataloged initially.

Downstream need

Notarius needs both:

• the completed RunResult; and
• the rendered messages, effective output contract, hashes, and other preparation details exposed by PreparedRun.

Notarius uses the prepared details to construct redaction-aware debug bundles and retain enough information to diagnose model behavior.

Current integration

Notarius currently calls Engine.Prepare and then Engine.Run with the same request. Because Run performs preparation internally, a successful request resolves and prepares the same work twice.

This duplicates profile resolution, input hashing, schema loading, and prompt rendering. It also creates a theoretical consistency window in which a filesystem-backed prompt, profile, schema, or input could change between the explicit preparation and the preparation performed by Run.

Requested capability

Add an opt-in execution method that prepares exactly once and returns both the prepared details and completed result:

type RunReport struct {
	Prepared PreparedRun
	Result   RunResult
}

func (e *Engine) RunDetailed(
	ctx context.Context,
	req RunRequest,
) (*RunReport, error)

The exact names are flexible. The important contract is that preparation occurs once and that the returned prepared state describes the execution that produced the returned result.

Existing Prepare and Run behavior should remain available for consumers that need only one side of the operation.

Design considerations

• Keep this API additive and preserve the existing simple Run workflow.
• Return caller-owned copies under PromptKit's existing ownership rules.
• Define whether any prepared details are available after an operational generation or validation error. Notarius does not require partial results for the initial use case, but an explicit contract would be valuable.
• Preserve cancellation and backend-admission semantics.
• Do not add all prepared content directly to RunResult. Rendered prompt content can be large and sensitive, and consumers should opt in to receiving it.

Value to Notarius

This is the highest-value wishlist item. It would remove duplicate work from every successful PromptKit-backed call and ensure that retained debug material corresponds atomically to the actual execution.

Priority 2: Prompt-Independent Profile Inspection

Disposition: Accepted into the future catalog.

Downstream need

Notarius validates configured pipeline profile IDs before beginning a run. It needs to determine whether:

• a profile exists;
• its referenced backend is registered;
• its execution target can be resolved; and
• it declares a credential requirement that the application may need to enforce.

This validation should not require model generation.

Current integration

Notarius constructs a synthetic prompt using testing/fstest.MapFS, supplies a dummy transcript, and calls Engine.Prepare solely to exercise profile and backend resolution. This works, but prompt preparation is serving as a substitute for a profile-inspection interface.

Requested capability

Add a prompt-independent profile-resolution API, for example:

type ResolvedProfile struct {
	ProfileID       string
	BackendID       string
	EffectiveTarget ExecutionTarget
	APIKeyEnv       string
}

func (e *Engine) ResolveProfile(
	ctx context.Context,
	profileID string,
) (ResolvedProfile, error)

The returned shape may differ, but it should provide enough information for a consumer to validate an explicit profile selection without inventing a prompt or supplying placeholder inputs.

Design considerations

• Resolve built-in, file-backed, and programmatic profiles using normal PromptKit precedence.
• Validate that a referenced backend registration exists.
• Do not resolve, retain, or expose credential values.
• Report credential requirements, such as an environment-variable name, so the consuming application can decide whether availability is required at configuration-validation time or only at execution time.
• Return caller-owned values.
• Preserve typed or sentinel error classification for missing and invalid profiles.
• Consider accepting an ExecutionTargetOverride if consumers need to inspect the same effective target that a run-level override would produce.
• Enumeration of all profiles is not required for the Notarius use case; exact lookup by ID is sufficient.

Value to Notarius

This would eliminate a synthetic production-only prompt fixture and establish a direct, supported contract for configuration-time profile and backend validation.

Priority 3: Semantic Execution-Target Fingerprints

Disposition: Deferred until prompt-independent profile inspection defines the resolved target whose configuration identity would be fingerprinted.

Downstream need

Notarius checkpoints model-backed pipeline stages. A checkpoint must not be reused when generation-affecting PromptKit configuration changes.

Notarius therefore needs a stable equality signal for the effective profile and backend target used by a pipeline.

Current integration

Notarius currently constructs this identity itself from:

• a manually maintained marker for the PromptKit release and built-in profile catalog;
• raw hashes of configured profile files; and
• a separate hash of the configured conventional local-backend endpoint.

This is safe but conservative and coupled to PromptKit details. Raw file hashing also invalidates checkpoints for semantically irrelevant YAML changes, such as comments or formatting.

Requested capability

Expose an opaque semantic digest for a resolved profile and its effective generation target. It could be returned by the proposed profile-resolution API:

type ResolvedProfile struct {
	ProfileID       string
	BackendID       string
	EffectiveTarget ExecutionTarget
	ExecutionDigest string
}

Alternatively, PromptKit could expose a dedicated method such as ProfileExecutionDigest(profileID).

Desired equality semantics

The digest should change when generation-affecting state changes, including:

• resolved model and endpoint;
• backend routing identity;
• backend request defaults and extra parameters;
• profile generation parameters; and
• the semantic identity of any selected built-in profile.

The digest should not incorporate:

• credential values;
• concurrency or queue capacity;
• filesystem source paths;
• YAML comments or formatting; or
• other settings that affect scheduling or source representation without changing the generation target.

The credential environment-variable name may need to participate if changing it can select a materially different provider account or target. PromptKit should define this deliberately while continuing to exclude the resolved secret value.

Design considerations

• Treat the digest as an opaque equality value rather than a public encoding of internal structures.
• Document which categories of change affect equality.
• Include a versioned semantic marker internally so PromptKit can deliberately invalidate old digests when its resolution semantics change.
• Prefer a per-profile digest over a digest of every profile known to an engine. Notarius generally knows which profiles a resolved pipeline uses.
• Do not require consumers to know PromptKit's built-in catalog version.

Value to Notarius

This would let Notarius remove its PromptKit release marker and raw profile-source fingerprinting, reduce unnecessary checkpoint invalidation, and delegate execution-target equality to the component that owns target resolution.

Priority 4: Structured Capacity Errors

Disposition: Accepted into the future catalog.

Downstream need

Notarius translates PromptKit backend-capacity rejection into a provider-neutral application error. When multiple backends are active, operators would benefit from knowing which backend rejected admission without parsing an error string or exposing endpoint details.

Current integration

PromptKit provides the useful ErrCapacityExceeded sentinel. Notarius can classify the failure reliably, but it retains only a sanitized diagnostic string as additional context.

Requested capability

Add a typed error that continues to match ErrCapacityExceeded:

type CapacityError struct {
	BackendID string
}

func (e *CapacityError) Is(target error) bool {
	return target == ErrCapacityExceeded
}

The exact implementation may use Unwrap or another idiomatic mechanism. The important properties are compatibility with errors.Is and discoverability through errors.As.

Design considerations

• Include the stable backend ID.
• Do not expose the backend endpoint, credential environment, credential value, request content, or other sensitive configuration.
• Consider including the configured concurrency and queue limits if they are useful and safe, but backend identity alone provides most of the downstream value.
• Add a retry delay only if PromptKit can provide a meaningful value. A full queue does not necessarily imply a reliable Retry-After duration.
• Keep retry and backoff policy with the consuming application. PromptKit should classify the admission failure rather than silently retry it.

Value to Notarius

This would improve operational diagnostics and future metrics while preserving the provider-neutral error boundary used by Notarius.

Capabilities PromptKit Already Provides Well

The current PromptKit boundary is sufficient for Notarius's implemented behavior. In particular, PromptKit already provides:

• filesystem, fs.FS, and programmatic prompt, profile, and schema sources;
• offline preparation without model execution;
• structured output and content validation;
• direct session propagation;
• tri-state per-run reasoning overrides;
• selected profile, backend, model, endpoint, effective parameters, hashes, and token-usage provenance;
• endpoint-only profiles;
• the conventional local backend helper;
• arbitrary engine-scoped Backend registrations;
• backend authentication environment names, extra parameters, concurrency limits, and queue-capacity policies;
• provider-client and artifact-reader extension interfaces;
• context cancellation; and
• useful public error sentinels, including profile absence and capacity exhaustion.

The wishlist does not imply that Notarius needs PromptKit to broaden its core responsibilities. It primarily asks for more direct access to information and operations that PromptKit already computes internally.

Responsibilities That Should Remain In Notarius

The following concerns belong to the downstream application and should not move into PromptKit for the sake of Notarius:

• pipeline staging, dependencies, and generated references;
• application-wide scheduling across providers and backends;
• module and validation retry policy;
• checkpoints, resume, and recomputation;
• durable run artifacts and manifests;
• D&D prompts, schemas, extractors, validators, and normalizers;
• Notarius configuration-file parsing and precedence;
• domain-specific prompt-cache prefix policy; and
• application-specific redaction, retention, and debug-bundle policy.

PromptKit's complete Backend API already supports custom IDs, multiple local endpoints, authentication, extra parameters, and explicit queue policies. Whether Notarius exposes those capabilities in its own configuration is an application-policy decision, not an upstream PromptKit gap.

Suggested Upstream Sequence

If the PromptKit team chooses to pursue these ideas, the most useful order for Notarius would be:

1. Add atomic execution that returns prepared details and the completed result.
2. Add prompt-independent profile inspection.
3. Add a semantic execution-target digest, preferably as part of profile inspection.
4. Add a typed capacity error carrying backend identity.

The first two address concrete workarounds in current Notarius code. The third would improve checkpoint correctness and reduce coupling. The fourth is operational polish.