promptkit/docs/roadmap/notarius-promptkit-wishlist.md

# Notarius PromptKit Wishlist

## Purpose

This document records features and interface changes that would be useful
additions to PromptKit from the perspective of the maintainers of Notarius, a
downstream application that consumes PromptKit.

PromptKit v0.3.0 provides the capabilities Notarius currently needs. None of
the ideas below blocks current Notarius development. They are opportunities to
reduce downstream workarounds, improve integration correctness, and make
PromptKit more ergonomic for applications with configuration validation,
debugging, checkpointing, and operational-observability requirements.

The examples are API sketches intended to communicate the desired capability,
not prescriptive names or finalized Go contracts.

## Priority 1: Atomic Execution With Prepared Details

**Disposition:** Covered by the accepted
[executable preparation handles](prepared-execution.md) roadmap. The shared
two-phase capability should provide the required single-preparation
consistency; a separate `RunDetailed` method is not cataloged initially.

### Downstream need

Notarius needs both:

- the completed `RunResult`; and
- the rendered messages, effective output contract, hashes, and other
  preparation details exposed by `PreparedRun`.

Notarius uses the prepared details to construct redaction-aware debug bundles
and retain enough information to diagnose model behavior.

### Current integration

Notarius currently calls `Engine.Prepare` and then `Engine.Run` with the same
request. Because `Run` performs preparation internally, a successful request
resolves and prepares the same work twice.

This duplicates profile resolution, input hashing, schema loading, and prompt
rendering. It also creates a theoretical consistency window in which a
filesystem-backed prompt, profile, schema, or input could change between the
explicit preparation and the preparation performed by `Run`.

### Requested capability

Add an opt-in execution method that prepares exactly once and returns both the
prepared details and completed result:

```go
type RunReport struct {
	Prepared PreparedRun
	Result   RunResult
}

func (e *Engine) RunDetailed(
	ctx context.Context,
	req RunRequest,
) (*RunReport, error)
```

The exact names are flexible. The important contract is that preparation
occurs once and that the returned prepared state describes the execution that
produced the returned result.

Existing `Prepare` and `Run` behavior should remain available for consumers
that need only one side of the operation.

### Design considerations

- Keep this API additive and preserve the existing simple `Run` workflow.
- Return caller-owned copies under PromptKit's existing ownership rules.
- Define whether any prepared details are available after an operational
  generation or validation error. Notarius does not require partial results
  for the initial use case, but an explicit contract would be valuable.
- Preserve cancellation and backend-admission semantics.
- Do not add all prepared content directly to `RunResult`. Rendered prompt
  content can be large and sensitive, and consumers should opt in to receiving
  it.

### Value to Notarius

This is the highest-value wishlist item. It would remove duplicate work from
every successful PromptKit-backed call and ensure that retained debug material
corresponds atomically to the actual execution.

## Priority 2: Prompt-Independent Profile Inspection

**Disposition:** Implemented as
[`Engine.InspectProfile`](../../engine.go). See the
[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work).

### Downstream need

Notarius validates configured pipeline profile IDs before beginning a run. It
needs to determine whether:

- a profile exists;
- its referenced backend is registered;
- its execution target can be resolved; and
- it declares a credential requirement that the application may need to
  enforce.

This validation should not require model generation.

### Previous integration

Before profile inspection was available, Notarius constructed a synthetic
prompt using `testing/fstest.MapFS`, supplied a dummy transcript, and called
`Engine.Prepare` solely to exercise profile and backend resolution.

### Value to Notarius

The implemented interface eliminates a synthetic production-only prompt
fixture and establishes a direct, supported contract for configuration-time
profile and backend validation.

## Priority 3: Semantic Execution-Target Fingerprints

**Disposition:** Deferred pending a separate semantic-equality design for
resolved execution targets.

### Downstream need

Notarius checkpoints model-backed pipeline stages. A checkpoint must not be
reused when generation-affecting PromptKit configuration changes.

Notarius therefore needs a stable equality signal for the effective profile
and backend target used by a pipeline.

### Current integration

Notarius currently constructs this identity itself from:

- a manually maintained marker for the PromptKit release and built-in profile
  catalog;
- raw hashes of configured profile files; and
- a separate hash of the configured conventional local-backend endpoint.

This is safe but conservative and coupled to PromptKit details. Raw file
hashing also invalidates checkpoints for semantically irrelevant YAML changes,
such as comments or formatting.

### Requested capability

Expose an opaque semantic digest for a resolved profile and its effective
generation target. It could be returned by the proposed profile-resolution
API:

```go
type ResolvedProfile struct {
	ProfileID       string
	BackendID       string
	EffectiveTarget ExecutionTarget
	ExecutionDigest string
}
```

Alternatively, PromptKit could expose a dedicated method such as
`ProfileExecutionDigest(profileID)`.

### Desired equality semantics

The digest should change when generation-affecting state changes, including:

- resolved model and endpoint;
- backend routing identity;
- backend request defaults and extra parameters;
- profile generation parameters; and
- the semantic identity of any selected built-in profile.

The digest should not incorporate:

- credential values;
- concurrency or queue capacity;
- filesystem source paths;
- YAML comments or formatting; or
- other settings that affect scheduling or source representation without
  changing the generation target.

The credential environment-variable name may need to participate if changing
it can select a materially different provider account or target. PromptKit
should define this deliberately while continuing to exclude the resolved
secret value.

### Design considerations

- Treat the digest as an opaque equality value rather than a public encoding
  of internal structures.
- Document which categories of change affect equality.
- Include a versioned semantic marker internally so PromptKit can deliberately
  invalidate old digests when its resolution semantics change.
- Prefer a per-profile digest over a digest of every profile known to an
  engine. Notarius generally knows which profiles a resolved pipeline uses.
- Do not require consumers to know PromptKit's built-in catalog version.

### Value to Notarius

This would let Notarius remove its PromptKit release marker and raw
profile-source fingerprinting, reduce unnecessary checkpoint invalidation, and
delegate execution-target equality to the component that owns target
resolution.

## Priority 4: Structured Capacity Errors

**Disposition:** Accepted into the
[structured capacity errors](capacity-errors.md) feature roadmap.

### Downstream need

Notarius translates PromptKit backend-capacity rejection into a
provider-neutral application error. When multiple backends are active,
operators would benefit from knowing which backend rejected admission without
parsing an error string or exposing endpoint details.

### Current integration

PromptKit provides the useful `ErrCapacityExceeded` sentinel. Notarius can
classify the failure reliably, but it retains only a sanitized diagnostic
string as additional context.

### Requested capability

Add a typed error that continues to match `ErrCapacityExceeded`:

```go
type CapacityError struct {
	BackendID string
}

func (e *CapacityError) Is(target error) bool {
	return target == ErrCapacityExceeded
}
```

The exact implementation may use `Unwrap` or another idiomatic mechanism. The
important properties are compatibility with `errors.Is` and discoverability
through `errors.As`.

### Design considerations

- Include the stable backend ID.
- Do not expose the backend endpoint, credential environment, credential
  value, request content, or other sensitive configuration.
- Consider including the configured concurrency and queue limits if they are
  useful and safe, but backend identity alone provides most of the downstream
  value.
- Add a retry delay only if PromptKit can provide a meaningful value. A full
  queue does not necessarily imply a reliable `Retry-After` duration.
- Keep retry and backoff policy with the consuming application. PromptKit
  should classify the admission failure rather than silently retry it.

### Value to Notarius

This would improve operational diagnostics and future metrics while preserving
the provider-neutral error boundary used by Notarius.

## Capabilities PromptKit Already Provides Well

The current PromptKit boundary is sufficient for Notarius's implemented
behavior. In particular, PromptKit already provides:

- filesystem, `fs.FS`, and programmatic prompt, profile, and schema sources;
- offline preparation without model execution;
- structured output and content validation;
- direct session propagation;
- tri-state per-run reasoning overrides;
- selected profile, backend, model, endpoint, effective parameters, hashes, and
  token-usage provenance;
- endpoint-only profiles;
- the conventional `local` backend helper;
- arbitrary engine-scoped `Backend` registrations;
- backend authentication environment names, extra parameters, concurrency
  limits, and queue-capacity policies;
- provider-client and artifact-reader extension interfaces;
- context cancellation; and
- useful public error sentinels, including profile absence and capacity
  exhaustion.

The wishlist does not imply that Notarius needs PromptKit to broaden its core
responsibilities. It primarily asks for more direct access to information and
operations that PromptKit already computes internally.

## Responsibilities That Should Remain In Notarius

The following concerns belong to the downstream application and should not
move into PromptKit for the sake of Notarius:

- pipeline staging, dependencies, and generated references;
- application-wide scheduling across providers and backends;
- module and validation retry policy;
- checkpoints, resume, and recomputation;
- durable run artifacts and manifests;
- D&D prompts, schemas, extractors, validators, and normalizers;
- Notarius configuration-file parsing and precedence;
- domain-specific prompt-cache prefix policy; and
- application-specific redaction, retention, and debug-bundle policy.

PromptKit's complete `Backend` API already supports custom IDs, multiple local
endpoints, authentication, extra parameters, and explicit queue policies.
Whether Notarius exposes those capabilities in its own configuration is an
application-policy decision, not an upstream PromptKit gap.

## Suggested Upstream Sequence

If the PromptKit team chooses to pursue these ideas, the most useful order for
Notarius would be:

1. Add atomic execution that returns prepared details and the completed result.
2. Add a semantic execution-target digest, preferably alongside profile
   inspection.
3. Add a typed capacity error carrying backend identity.

The first addresses a concrete execution workaround. The second would improve
checkpoint correctness and reduce coupling. The third is operational polish.