# Notarius PromptKit Wishlist

## Purpose

This document records features and interface changes that would be useful
additions to PromptKit from the perspective of the maintainers of Notarius, a
downstream application that consumes PromptKit.

PromptKit now provides the capabilities Notarius currently needs. The
remaining deferred ideas are optional opportunities to improve checkpointing
and operational observability.

The examples are API sketches intended to communicate the desired capability,
not prescriptive names or finalized Go contracts.

## Priority 1: Atomic Execution With Prepared Details

**Disposition:** Implemented through [`Engine.PrepareExecution` and
`Engine.RunPrepared`](../../engine.go). See the
[consumer guidance](../consumers/pkg-promptkit.md#prepare-now-and-execute-the-same-snapshot-later).
A separate `RunDetailed` method is not cataloged.

### Downstream need

Notarius needs both:

- the completed `RunResult`; and
- the rendered messages, effective output contract, hashes, and other
  preparation details exposed by `PreparedRun`.

Notarius uses the prepared details to construct redaction-aware debug bundles
and retain enough information to diagnose model behavior.

### Implemented behavior

Notarius can prepare one frozen execution snapshot, retain a caller-owned and
credential-redacted `Details` value for its debug bundle, and execute the same
snapshot through `RunPrepared`. The opaque handle is engine-bound and
single-use; an unused handle can be released with `Discard`. The consumer
guide and exported GoDoc own the exact lifecycle and failure contracts.

### Value to Notarius

This removes duplicate work from PromptKit-backed calls and ensures that
retained debug material corresponds atomically to the actual execution.

## Priority 2: Prompt-Independent Profile Inspection

**Disposition:** Implemented as
[`Engine.InspectProfile`](../../engine.go). See the
[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work).

### Downstream need

Notarius validates configured pipeline profile IDs before beginning a run. It
needs to determine whether:

- a profile exists;
- its referenced backend is registered;
- its execution target can be resolved; and
- it declares a credential requirement that the application may need to
  enforce.

This validation should not require model generation.

### Previous integration

Before profile inspection was available, Notarius constructed a synthetic
prompt using `testing/fstest.MapFS`, supplied a dummy transcript, and called
`Engine.Prepare` solely to exercise profile and backend resolution.

### Value to Notarius

The implemented interface eliminates a synthetic production-only prompt
fixture and establishes a direct, supported contract for configuration-time
profile and backend validation.

## Priority 3: Semantic Execution-Target Fingerprints

**Disposition:** Deferred pending a separate semantic-equality design for
resolved execution targets.

### Downstream need

Notarius checkpoints model-backed pipeline stages. A checkpoint must not be
reused when generation-affecting PromptKit configuration changes.

Notarius therefore needs a stable equality signal for the effective profile
and backend target used by a pipeline.

### Current integration

Notarius currently constructs this identity itself from:

- a manually maintained marker for the PromptKit release and built-in profile
  catalog;
- raw hashes of configured profile files; and
- a separate hash of the configured conventional local-backend endpoint.

This is safe but conservative and coupled to PromptKit details. Raw file
hashing also invalidates checkpoints for semantically irrelevant YAML changes,
such as comments or formatting.

### Requested capability

Expose an opaque semantic digest for a resolved profile and its effective
generation target. It could be returned by the proposed profile-resolution
API:

```go
type ResolvedProfile struct {
	ProfileID       string
	BackendID       string
	EffectiveTarget ExecutionTarget
	ExecutionDigest string
}
```

Alternatively, PromptKit could expose a dedicated method such as
`ProfileExecutionDigest(profileID)`.

### Desired equality semantics

The digest should change when generation-affecting state changes, including:

- resolved model and endpoint;
- backend routing identity;
- backend request defaults and extra parameters;
- profile generation parameters; and
- the semantic identity of any selected built-in profile.

The digest should not incorporate:

- credential values;
- concurrency or queue capacity;
- filesystem source paths;
- YAML comments or formatting; or
- other settings that affect scheduling or source representation without
  changing the generation target.

The credential environment-variable name may need to participate if changing
it can select a materially different provider account or target. PromptKit
should define this deliberately while continuing to exclude the resolved
secret value.

### Design considerations

- Treat the digest as an opaque equality value rather than a public encoding
  of internal structures.
- Document which categories of change affect equality.
- Include a versioned semantic marker internally so PromptKit can deliberately
  invalidate old digests when its resolution semantics change.
- Prefer a per-profile digest over a digest of every profile known to an
  engine. Notarius generally knows which profiles a resolved pipeline uses.
- Do not require consumers to know PromptKit's built-in catalog version.

### Value to Notarius

This would let Notarius remove its PromptKit release marker and raw
profile-source fingerprinting, reduce unnecessary checkpoint invalidation, and
delegate execution-target equality to the component that owns target
resolution.

## Priority 4: Structured Capacity Errors

**Disposition:** Implemented behavior. See the consumer guide's
[Handle Errors](../consumers/pkg-promptkit.md#handle-errors) section.

### Downstream need

Notarius translates PromptKit backend-capacity rejection into a
provider-neutral application error. When multiple backends are active,
operators would benefit from knowing which backend rejected admission without
parsing an error string or exposing endpoint details.

### Implemented behavior

PromptKit now retains the broad capacity classification while allowing
Notarius to obtain the selected backend ID without parsing diagnostic text.
The consumer guide owns the application workflow, including retry and backoff
policy.

### Value to Notarius

This improves operational diagnostics and future metrics while preserving
the provider-neutral error boundary used by Notarius.

## Capabilities PromptKit Already Provides Well

The current PromptKit boundary is sufficient for Notarius's implemented
behavior. In particular, PromptKit already provides:

- filesystem, `fs.FS`, and programmatic prompt, profile, and schema sources;
- offline preparation without model execution;
- structured output and content validation;
- direct session propagation;
- tri-state per-run reasoning overrides;
- selected profile, backend, model, endpoint, effective parameters, hashes, and
  token-usage provenance;
- endpoint-only profiles;
- the conventional `local` backend helper;
- arbitrary engine-scoped `Backend` registrations;
- backend authentication environment names, extra parameters, concurrency
  limits, and queue-capacity policies;
- provider-client and artifact-reader extension interfaces;
- context cancellation; and
- useful public error sentinels, including profile absence and capacity
  exhaustion.

The wishlist does not imply that Notarius needs PromptKit to broaden its core
responsibilities. It primarily asks for more direct access to information and
operations that PromptKit already computes internally.

## Responsibilities That Should Remain In Notarius

The following concerns belong to the downstream application and should not
move into PromptKit for the sake of Notarius:

- pipeline staging, dependencies, and generated references;
- application-wide scheduling across providers and backends;
- module and validation retry policy;
- checkpoints, resume, and recomputation;
- durable run artifacts and manifests;
- D&D prompts, schemas, extractors, validators, and normalizers;
- Notarius configuration-file parsing and precedence;
- domain-specific prompt-cache prefix policy; and
- application-specific redaction, retention, and debug-bundle policy.

PromptKit's complete `Backend` API already supports custom IDs, multiple local
endpoints, authentication, extra parameters, and explicit queue policies.
Whether Notarius exposes those capabilities in its own configuration is an
application-policy decision, not an upstream PromptKit gap.

## Suggested Upstream Sequence

For downstream adoption and any remaining upstream work, the useful order is:

1. Adopt prepared execution for atomic details and results.
2. Add a semantic execution-target digest, preferably alongside profile
   inspection.
3. Use the implemented typed capacity error where backend admission diagnostics
   are needed.

The first removes the concrete execution workaround. The second would improve
checkpoint correctness and reduce coupling. The third is operational polish.