promptkit/docs/roadmap/notarius-promptkit-wishlist.md

Notarius PromptKit Wishlist

Purpose

This document records features and interface changes that would be useful additions to PromptKit from the perspective of the maintainers of Notarius, a downstream application that consumes PromptKit.

PromptKit now provides the capabilities Notarius currently needs. The remaining deferred ideas are optional opportunities to improve checkpointing and operational observability.

The examples are API sketches intended to communicate the desired capability, not prescriptive names or finalized Go contracts.

Priority 1: Atomic Execution With Prepared Details

Disposition: Implemented through Engine.PrepareExecution and Engine.RunPrepared. See the consumer guidance. A separate RunDetailed method is not cataloged.

Downstream need

Notarius needs both:

• the completed RunResult; and
• the rendered messages, effective output contract, hashes, and other preparation details exposed by PreparedRun.

Notarius uses the prepared details to construct redaction-aware debug bundles and retain enough information to diagnose model behavior.

Implemented behavior

Notarius can prepare one frozen execution snapshot, retain a caller-owned and credential-redacted Details value for its debug bundle, and execute the same snapshot through RunPrepared. The opaque handle is engine-bound and single-use; an unused handle can be released with Discard. The consumer guide and exported GoDoc own the exact lifecycle and failure contracts.

Value to Notarius

This removes duplicate work from PromptKit-backed calls and ensures that retained debug material corresponds atomically to the actual execution.

Priority 2: Prompt-Independent Profile Inspection

Disposition: Implemented as Engine.InspectProfile. See the consumer guidance.

Downstream need

Notarius validates configured pipeline profile IDs before beginning a run. It needs to determine whether:

• a profile exists;
• its referenced backend is registered;
• its execution target can be resolved; and
• it declares a credential requirement that the application may need to enforce.

This validation should not require model generation.

Previous integration

Before profile inspection was available, Notarius constructed a synthetic prompt using testing/fstest.MapFS, supplied a dummy transcript, and called Engine.Prepare solely to exercise profile and backend resolution.

Value to Notarius

The implemented interface eliminates a synthetic production-only prompt fixture and establishes a direct, supported contract for configuration-time profile and backend validation.

Priority 3: Semantic Execution-Target Fingerprints

Disposition: Deferred pending a separate semantic-equality design for resolved execution targets.

Downstream need

Notarius checkpoints model-backed pipeline stages. A checkpoint must not be reused when generation-affecting PromptKit configuration changes.

Notarius therefore needs a stable equality signal for the effective profile and backend target used by a pipeline.

Current integration

Notarius currently constructs this identity itself from:

• a manually maintained marker for the PromptKit release and built-in profile catalog;
• raw hashes of configured profile files; and
• a separate hash of the configured conventional local-backend endpoint.

This is safe but conservative and coupled to PromptKit details. Raw file hashing also invalidates checkpoints for semantically irrelevant YAML changes, such as comments or formatting.

Requested capability

Expose an opaque semantic digest for a resolved profile and its effective generation target. It could be returned by the proposed profile-resolution API:

type ResolvedProfile struct {
	ProfileID       string
	BackendID       string
	EffectiveTarget ExecutionTarget
	ExecutionDigest string
}

Alternatively, PromptKit could expose a dedicated method such as ProfileExecutionDigest(profileID).

Desired equality semantics

The digest should change when generation-affecting state changes, including:

• resolved model and endpoint;
• backend routing identity;
• backend request defaults and extra parameters;
• profile generation parameters; and
• the semantic identity of any selected built-in profile.

The digest should not incorporate:

• credential values;
• concurrency or queue capacity;
• filesystem source paths;
• YAML comments or formatting; or
• other settings that affect scheduling or source representation without changing the generation target.

The credential environment-variable name may need to participate if changing it can select a materially different provider account or target. PromptKit should define this deliberately while continuing to exclude the resolved secret value.

Design considerations

• Treat the digest as an opaque equality value rather than a public encoding of internal structures.
• Document which categories of change affect equality.
• Include a versioned semantic marker internally so PromptKit can deliberately invalidate old digests when its resolution semantics change.
• Prefer a per-profile digest over a digest of every profile known to an engine. Notarius generally knows which profiles a resolved pipeline uses.
• Do not require consumers to know PromptKit's built-in catalog version.

Value to Notarius

This would let Notarius remove its PromptKit release marker and raw profile-source fingerprinting, reduce unnecessary checkpoint invalidation, and delegate execution-target equality to the component that owns target resolution.

Priority 4: Structured Capacity Errors

Disposition: Implemented behavior. See the consumer guide's Handle Errors section.

Downstream need

Notarius translates PromptKit backend-capacity rejection into a provider-neutral application error. When multiple backends are active, operators would benefit from knowing which backend rejected admission without parsing an error string or exposing endpoint details.

Implemented behavior

PromptKit now retains the broad capacity classification while allowing Notarius to obtain the selected backend ID without parsing diagnostic text. The consumer guide owns the application workflow, including retry and backoff policy.

Value to Notarius

This improves operational diagnostics and future metrics while preserving the provider-neutral error boundary used by Notarius.

Capabilities PromptKit Already Provides Well

The current PromptKit boundary is sufficient for Notarius's implemented behavior. In particular, PromptKit already provides:

• filesystem, fs.FS, and programmatic prompt, profile, and schema sources;
• offline preparation without model execution;
• structured output and content validation;
• direct session propagation;
• tri-state per-run reasoning overrides;
• selected profile, backend, model, endpoint, effective parameters, hashes, and token-usage provenance;
• endpoint-only profiles;
• the conventional local backend helper;
• arbitrary engine-scoped Backend registrations;
• backend authentication environment names, extra parameters, concurrency limits, and queue-capacity policies;
• provider-client and artifact-reader extension interfaces;
• context cancellation; and
• useful public error sentinels, including profile absence and capacity exhaustion.

The wishlist does not imply that Notarius needs PromptKit to broaden its core responsibilities. It primarily asks for more direct access to information and operations that PromptKit already computes internally.

Responsibilities That Should Remain In Notarius

The following concerns belong to the downstream application and should not move into PromptKit for the sake of Notarius:

• pipeline staging, dependencies, and generated references;
• application-wide scheduling across providers and backends;
• module and validation retry policy;
• checkpoints, resume, and recomputation;
• durable run artifacts and manifests;
• D&D prompts, schemas, extractors, validators, and normalizers;
• Notarius configuration-file parsing and precedence;
• domain-specific prompt-cache prefix policy; and
• application-specific redaction, retention, and debug-bundle policy.

PromptKit's complete Backend API already supports custom IDs, multiple local endpoints, authentication, extra parameters, and explicit queue policies. Whether Notarius exposes those capabilities in its own configuration is an application-policy decision, not an upstream PromptKit gap.

Suggested Upstream Sequence

For downstream adoption and any remaining upstream work, the useful order is:

1. Adopt prepared execution for atomic details and results.
2. Add a semantic execution-target digest, preferably alongside profile inspection.
3. Use the implemented typed capacity error where backend admission diagnostics are needed.

The first removes the concrete execution workaround. The second would improve checkpoint correctness and reduce coupling. The third is operational polish.