246 lines
9.0 KiB
Markdown
246 lines
9.0 KiB
Markdown
# Notarius PromptKit Wishlist
|
|
|
|
## Purpose
|
|
|
|
This document records features and interface changes that would be useful
|
|
additions to PromptKit from the perspective of the maintainers of Notarius, a
|
|
downstream application that consumes PromptKit.
|
|
|
|
PromptKit now provides the capabilities Notarius currently needs. The
|
|
remaining deferred ideas are optional opportunities to improve checkpointing
|
|
and operational observability.
|
|
|
|
The examples are API sketches intended to communicate the desired capability,
|
|
not prescriptive names or finalized Go contracts.
|
|
|
|
## Priority 1: Atomic Execution With Prepared Details
|
|
|
|
**Disposition:** Implemented through [`Engine.PrepareExecution` and
|
|
`Engine.RunPrepared`](../../engine.go). See the
|
|
[consumer guidance](../consumers/pkg-promptkit.md#prepare-now-and-execute-the-same-snapshot-later).
|
|
A separate `RunDetailed` method is not cataloged.
|
|
|
|
### Downstream need
|
|
|
|
Notarius needs both:
|
|
|
|
- the completed `RunResult`; and
|
|
- the rendered messages, effective output contract, hashes, and other
|
|
preparation details exposed by `PreparedRun`.
|
|
|
|
Notarius uses the prepared details to construct redaction-aware debug bundles
|
|
and retain enough information to diagnose model behavior.
|
|
|
|
### Implemented behavior
|
|
|
|
Notarius can prepare one frozen execution snapshot, retain a caller-owned and
|
|
credential-redacted `Details` value for its debug bundle, and execute the same
|
|
snapshot through `RunPrepared`. The opaque handle is engine-bound and
|
|
single-use; an unused handle can be released with `Discard`. The consumer
|
|
guide and exported GoDoc own the exact lifecycle and failure contracts.
|
|
|
|
### Value to Notarius
|
|
|
|
This removes duplicate work from PromptKit-backed calls and ensures that
|
|
retained debug material corresponds atomically to the actual execution.
|
|
|
|
## Priority 2: Prompt-Independent Profile Inspection
|
|
|
|
**Disposition:** Implemented as
|
|
[`Engine.InspectProfile`](../../engine.go). See the
|
|
[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work).
|
|
|
|
### Downstream need
|
|
|
|
Notarius validates configured pipeline profile IDs before beginning a run. It
|
|
needs to determine whether:
|
|
|
|
- a profile exists;
|
|
- its referenced backend is registered;
|
|
- its execution target can be resolved; and
|
|
- it declares a credential requirement that the application may need to
|
|
enforce.
|
|
|
|
This validation should not require model generation.
|
|
|
|
### Previous integration
|
|
|
|
Before profile inspection was available, Notarius constructed a synthetic
|
|
prompt using `testing/fstest.MapFS`, supplied a dummy transcript, and called
|
|
`Engine.Prepare` solely to exercise profile and backend resolution.
|
|
|
|
### Value to Notarius
|
|
|
|
The implemented interface eliminates a synthetic production-only prompt
|
|
fixture and establishes a direct, supported contract for configuration-time
|
|
profile and backend validation.
|
|
|
|
## Priority 3: Semantic Execution-Target Fingerprints
|
|
|
|
**Disposition:** Deferred pending a separate semantic-equality design for
|
|
resolved execution targets.
|
|
|
|
### Downstream need
|
|
|
|
Notarius checkpoints model-backed pipeline stages. A checkpoint must not be
|
|
reused when generation-affecting PromptKit configuration changes.
|
|
|
|
Notarius therefore needs a stable equality signal for the effective profile
|
|
and backend target used by a pipeline.
|
|
|
|
### Current integration
|
|
|
|
Notarius currently constructs this identity itself from:
|
|
|
|
- a manually maintained marker for the PromptKit release and built-in profile
|
|
catalog;
|
|
- raw hashes of configured profile files; and
|
|
- a separate hash of the configured conventional local-backend endpoint.
|
|
|
|
This is safe but conservative and coupled to PromptKit details. Raw file
|
|
hashing also invalidates checkpoints for semantically irrelevant YAML changes,
|
|
such as comments or formatting.
|
|
|
|
### Requested capability
|
|
|
|
Expose an opaque semantic digest for a resolved profile and its effective
|
|
generation target. It could be returned by the proposed profile-resolution
|
|
API:
|
|
|
|
```go
|
|
type ResolvedProfile struct {
|
|
ProfileID string
|
|
BackendID string
|
|
EffectiveTarget ExecutionTarget
|
|
ExecutionDigest string
|
|
}
|
|
```
|
|
|
|
Alternatively, PromptKit could expose a dedicated method such as
|
|
`ProfileExecutionDigest(profileID)`.
|
|
|
|
### Desired equality semantics
|
|
|
|
The digest should change when generation-affecting state changes, including:
|
|
|
|
- resolved model and endpoint;
|
|
- backend routing identity;
|
|
- backend request defaults and extra parameters;
|
|
- profile generation parameters; and
|
|
- the semantic identity of any selected built-in profile.
|
|
|
|
The digest should not incorporate:
|
|
|
|
- credential values;
|
|
- concurrency or queue capacity;
|
|
- filesystem source paths;
|
|
- YAML comments or formatting; or
|
|
- other settings that affect scheduling or source representation without
|
|
changing the generation target.
|
|
|
|
The credential environment-variable name may need to participate if changing
|
|
it can select a materially different provider account or target. PromptKit
|
|
should define this deliberately while continuing to exclude the resolved
|
|
secret value.
|
|
|
|
### Design considerations
|
|
|
|
- Treat the digest as an opaque equality value rather than a public encoding
|
|
of internal structures.
|
|
- Document which categories of change affect equality.
|
|
- Include a versioned semantic marker internally so PromptKit can deliberately
|
|
invalidate old digests when its resolution semantics change.
|
|
- Prefer a per-profile digest over a digest of every profile known to an
|
|
engine. Notarius generally knows which profiles a resolved pipeline uses.
|
|
- Do not require consumers to know PromptKit's built-in catalog version.
|
|
|
|
### Value to Notarius
|
|
|
|
This would let Notarius remove its PromptKit release marker and raw
|
|
profile-source fingerprinting, reduce unnecessary checkpoint invalidation, and
|
|
delegate execution-target equality to the component that owns target
|
|
resolution.
|
|
|
|
## Priority 4: Structured Capacity Errors
|
|
|
|
**Disposition:** Implemented behavior. See the consumer guide's
|
|
[Handle Errors](../consumers/pkg-promptkit.md#handle-errors) section.
|
|
|
|
### Downstream need
|
|
|
|
Notarius translates PromptKit backend-capacity rejection into a
|
|
provider-neutral application error. When multiple backends are active,
|
|
operators would benefit from knowing which backend rejected admission without
|
|
parsing an error string or exposing endpoint details.
|
|
|
|
### Implemented behavior
|
|
|
|
PromptKit now retains the broad capacity classification while allowing
|
|
Notarius to obtain the selected backend ID without parsing diagnostic text.
|
|
The consumer guide owns the application workflow, including retry and backoff
|
|
policy.
|
|
|
|
### Value to Notarius
|
|
|
|
This improves operational diagnostics and future metrics while preserving
|
|
the provider-neutral error boundary used by Notarius.
|
|
|
|
## Capabilities PromptKit Already Provides Well
|
|
|
|
The current PromptKit boundary is sufficient for Notarius's implemented
|
|
behavior. In particular, PromptKit already provides:
|
|
|
|
- filesystem, `fs.FS`, and programmatic prompt, profile, and schema sources;
|
|
- offline preparation without model execution;
|
|
- structured output and content validation;
|
|
- direct session propagation;
|
|
- tri-state per-run reasoning overrides;
|
|
- selected profile, backend, model, endpoint, effective parameters, hashes, and
|
|
token-usage provenance;
|
|
- endpoint-only profiles;
|
|
- the conventional `local` backend helper;
|
|
- arbitrary engine-scoped `Backend` registrations;
|
|
- backend authentication environment names, extra parameters, concurrency
|
|
limits, and queue-capacity policies;
|
|
- provider-client and artifact-reader extension interfaces;
|
|
- context cancellation; and
|
|
- useful public error sentinels, including profile absence and capacity
|
|
exhaustion.
|
|
|
|
The wishlist does not imply that Notarius needs PromptKit to broaden its core
|
|
responsibilities. It primarily asks for more direct access to information and
|
|
operations that PromptKit already computes internally.
|
|
|
|
## Responsibilities That Should Remain In Notarius
|
|
|
|
The following concerns belong to the downstream application and should not
|
|
move into PromptKit for the sake of Notarius:
|
|
|
|
- pipeline staging, dependencies, and generated references;
|
|
- application-wide scheduling across providers and backends;
|
|
- module and validation retry policy;
|
|
- checkpoints, resume, and recomputation;
|
|
- durable run artifacts and manifests;
|
|
- D&D prompts, schemas, extractors, validators, and normalizers;
|
|
- Notarius configuration-file parsing and precedence;
|
|
- domain-specific prompt-cache prefix policy; and
|
|
- application-specific redaction, retention, and debug-bundle policy.
|
|
|
|
PromptKit's complete `Backend` API already supports custom IDs, multiple local
|
|
endpoints, authentication, extra parameters, and explicit queue policies.
|
|
Whether Notarius exposes those capabilities in its own configuration is an
|
|
application-policy decision, not an upstream PromptKit gap.
|
|
|
|
## Suggested Upstream Sequence
|
|
|
|
For downstream adoption and any remaining upstream work, the useful order is:
|
|
|
|
1. Adopt prepared execution for atomic details and results.
|
|
2. Add a semantic execution-target digest, preferably alongside profile
|
|
inspection.
|
|
3. Use the implemented typed capacity error where backend admission diagnostics
|
|
are needed.
|
|
|
|
The first removes the concrete execution workaround. The second would improve
|
|
checkpoint correctness and reduce coupling. The third is operational polish.
|