Prepare documentation for Promptkit v0.4.0
This commit is contained in:
@@ -6,21 +6,19 @@ This document records features and interface changes that would be useful
|
||||
additions to PromptKit from the perspective of the maintainers of Notarius, a
|
||||
downstream application that consumes PromptKit.
|
||||
|
||||
PromptKit v0.3.0 provides the capabilities Notarius currently needs. None of
|
||||
the ideas below blocks current Notarius development. They are opportunities to
|
||||
reduce downstream workarounds, improve integration correctness, and make
|
||||
PromptKit more ergonomic for applications with configuration validation,
|
||||
debugging, checkpointing, and operational-observability requirements.
|
||||
PromptKit now provides the capabilities Notarius currently needs. The
|
||||
remaining deferred ideas are optional opportunities to improve checkpointing
|
||||
and operational observability.
|
||||
|
||||
The examples are API sketches intended to communicate the desired capability,
|
||||
not prescriptive names or finalized Go contracts.
|
||||
|
||||
## Priority 1: Atomic Execution With Prepared Details
|
||||
|
||||
**Disposition:** Covered by the accepted
|
||||
[executable preparation handles](prepared-execution.md) roadmap. The shared
|
||||
two-phase capability should provide the required single-preparation
|
||||
consistency; a separate `RunDetailed` method is not cataloged initially.
|
||||
**Disposition:** Implemented through [`Engine.PrepareExecution` and
|
||||
`Engine.RunPrepared`](../../engine.go). See the
|
||||
[consumer guidance](../consumers/pkg-promptkit.md#prepare-now-and-execute-the-same-snapshot-later).
|
||||
A separate `RunDetailed` method is not cataloged.
|
||||
|
||||
### Downstream need
|
||||
|
||||
@@ -33,58 +31,18 @@ Notarius needs both:
|
||||
Notarius uses the prepared details to construct redaction-aware debug bundles
|
||||
and retain enough information to diagnose model behavior.
|
||||
|
||||
### Current integration
|
||||
### Implemented behavior
|
||||
|
||||
Notarius currently calls `Engine.Prepare` and then `Engine.Run` with the same
|
||||
request. Because `Run` performs preparation internally, a successful request
|
||||
resolves and prepares the same work twice.
|
||||
|
||||
This duplicates profile resolution, input hashing, schema loading, and prompt
|
||||
rendering. It also creates a theoretical consistency window in which a
|
||||
filesystem-backed prompt, profile, schema, or input could change between the
|
||||
explicit preparation and the preparation performed by `Run`.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Add an opt-in execution method that prepares exactly once and returns both the
|
||||
prepared details and completed result:
|
||||
|
||||
```go
|
||||
type RunReport struct {
|
||||
Prepared PreparedRun
|
||||
Result RunResult
|
||||
}
|
||||
|
||||
func (e *Engine) RunDetailed(
|
||||
ctx context.Context,
|
||||
req RunRequest,
|
||||
) (*RunReport, error)
|
||||
```
|
||||
|
||||
The exact names are flexible. The important contract is that preparation
|
||||
occurs once and that the returned prepared state describes the execution that
|
||||
produced the returned result.
|
||||
|
||||
Existing `Prepare` and `Run` behavior should remain available for consumers
|
||||
that need only one side of the operation.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Keep this API additive and preserve the existing simple `Run` workflow.
|
||||
- Return caller-owned copies under PromptKit's existing ownership rules.
|
||||
- Define whether any prepared details are available after an operational
|
||||
generation or validation error. Notarius does not require partial results
|
||||
for the initial use case, but an explicit contract would be valuable.
|
||||
- Preserve cancellation and backend-admission semantics.
|
||||
- Do not add all prepared content directly to `RunResult`. Rendered prompt
|
||||
content can be large and sensitive, and consumers should opt in to receiving
|
||||
it.
|
||||
Notarius can prepare one frozen execution snapshot, retain a caller-owned and
|
||||
credential-redacted `Details` value for its debug bundle, and execute the same
|
||||
snapshot through `RunPrepared`. The opaque handle is engine-bound and
|
||||
single-use; an unused handle can be released with `Discard`. The consumer
|
||||
guide and exported GoDoc own the exact lifecycle and failure contracts.
|
||||
|
||||
### Value to Notarius
|
||||
|
||||
This is the highest-value wishlist item. It would remove duplicate work from
|
||||
every successful PromptKit-backed call and ensure that retained debug material
|
||||
corresponds atomically to the actual execution.
|
||||
This removes duplicate work from PromptKit-backed calls and ensures that
|
||||
retained debug material corresponds atomically to the actual execution.
|
||||
|
||||
## Priority 2: Prompt-Independent Profile Inspection
|
||||
|
||||
@@ -275,14 +233,13 @@ application-policy decision, not an upstream PromptKit gap.
|
||||
|
||||
## Suggested Upstream Sequence
|
||||
|
||||
If the PromptKit team chooses to pursue these ideas, the most useful order for
|
||||
Notarius would be:
|
||||
For downstream adoption and any remaining upstream work, the useful order is:
|
||||
|
||||
1. Add atomic execution that returns prepared details and the completed result.
|
||||
1. Adopt prepared execution for atomic details and results.
|
||||
2. Add a semantic execution-target digest, preferably alongside profile
|
||||
inspection.
|
||||
3. Use the implemented typed capacity error where backend admission diagnostics
|
||||
are needed.
|
||||
|
||||
The first addresses a concrete execution workaround. The second would improve
|
||||
The first removes the concrete execution workaround. The second would improve
|
||||
checkpoint correctness and reduce coupling. The third is operational polish.
|
||||
|
||||
Reference in New Issue
Block a user