Compare commits
3 Commits
v0.3.0
...
c13e9710d9
| Author | SHA1 | Date | |
|---|---|---|---|
| c13e9710d9 | |||
| 87b5ec3d75 | |||
| cb4028a637 |
@@ -33,9 +33,52 @@ consumers.
|
|||||||
|
|
||||||
## Ideas
|
## Ideas
|
||||||
|
|
||||||
No ideas are currently cataloged. Backend-specific concurrency management has
|
Executable preparation handles have been selected for active planning in the
|
||||||
been selected for active planning in the
|
[focused feature roadmap](prepared-execution.md). The remaining ideas are
|
||||||
[focused concurrency roadmap](concurrency.md).
|
still available for future selection.
|
||||||
|
|
||||||
|
### Prompt-independent profile inspection
|
||||||
|
|
||||||
|
Provide exact profile lookup and structural resolution without requiring a
|
||||||
|
synthetic prompt, placeholder inputs, or model generation. This shared need is
|
||||||
|
described by
|
||||||
|
[Notarius](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
|
||||||
|
and
|
||||||
|
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-3-prompt-independent-profile-inspection).
|
||||||
|
|
||||||
|
- Apply ordinary built-in, file-backed, and programmatic profile precedence.
|
||||||
|
- Validate referenced backend membership and the structurally resolved
|
||||||
|
execution target.
|
||||||
|
- Report credential requirements and environment-variable names without
|
||||||
|
exposing credential values or requiring current credential availability.
|
||||||
|
- Support exact lookup by profile ID; enumeration is not required initially.
|
||||||
|
|
||||||
|
### Prompt-definition inspection
|
||||||
|
|
||||||
|
Provide exact prompt-definition lookup without rendering, placeholder inputs,
|
||||||
|
profile resolution, or model generation, as requested by
|
||||||
|
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-2-prompt-definition-inspection).
|
||||||
|
|
||||||
|
- Return caller-owned identity, version, input definitions, default-profile,
|
||||||
|
output-contract, and opaque definition-equality information.
|
||||||
|
- Apply ordinary prompt-source precedence and exact ID/version selection.
|
||||||
|
- Validate the selected definition and referenced prompt content
|
||||||
|
structurally, without returning source bodies or rendered messages.
|
||||||
|
- Leave complete cross-source corpus validation and enumeration outside the
|
||||||
|
initial inspection contract.
|
||||||
|
|
||||||
|
### Structured capacity errors
|
||||||
|
|
||||||
|
Add safe structured context to backend admission rejection, as requested by
|
||||||
|
[Notarius](notarius-promptkit-wishlist.md#priority-4-structured-capacity-errors)
|
||||||
|
and
|
||||||
|
[Weatherreporter](weatherreporter-promptkit-wishlist.md#structured-capacity-errors).
|
||||||
|
|
||||||
|
- Preserve compatibility with `errors.Is(err, ErrCapacityExceeded)`.
|
||||||
|
- Support `errors.As` to obtain the stable backend ID.
|
||||||
|
- Do not expose endpoints, credential configuration or values, request
|
||||||
|
content, or speculative retry timing.
|
||||||
|
- Keep retry and backoff policy with downstream consumers.
|
||||||
|
|
||||||
## Entry Format
|
## Entry Format
|
||||||
|
|
||||||
|
|||||||
@@ -1,315 +1,683 @@
|
|||||||
# Local Backend Convenience Implementation Plan
|
# Executable Preparation Handles Implementation Plan
|
||||||
|
|
||||||
**Status:** Complete.
|
**Status:** Accepted.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
This document is the decision-complete implementation plan for
|
This document is the decision-complete implementation plan for
|
||||||
[local backend convenience](local-backend.md). It is written for a coding
|
[executable preparation handles](prepared-execution.md). It is written for a
|
||||||
agent that will implement each stage in order.
|
coding agent that will implement each stage in order.
|
||||||
|
|
||||||
The feature roadmap owns the motivation, consumer paths, policy choices,
|
The feature roadmap owns the motivation, consumer workflow, policy choices,
|
||||||
compatibility requirements, non-goals, and target end state. This document
|
compatibility requirements, non-goals, and target end state. This document
|
||||||
owns the exact proposed API, file-level changes, implementation sequence, test
|
owns the concrete design, file-level changes, implementation sequence, test
|
||||||
ownership, documentation work, validation commands, and completion gates.
|
ownership, documentation work, validation commands, and completion gates.
|
||||||
|
|
||||||
## Implementation Rules
|
## Implementation Rules
|
||||||
|
|
||||||
- Complete the stages in order. Keep the root package compiling and its
|
- Complete the stages in order. Keep the repository compiling and the focused
|
||||||
focused tests passing at every stage boundary.
|
tests passing at every stage boundary.
|
||||||
- Preserve unrelated working-tree changes. The feature roadmap may already be
|
- Preserve unrelated working-tree changes. In particular, retain the accepted
|
||||||
uncommitted when implementation begins; retain it.
|
roadmap and the downstream wishlist and future-roadmap edits that may
|
||||||
|
already be present.
|
||||||
- Follow every policy under `docs/policy/`, the task-specific reading guide in
|
- Follow every policy under `docs/policy/`, the task-specific reading guide in
|
||||||
`docs/development.md`, and the accepted behavior in
|
`docs/development.md`, and the accepted behavior in
|
||||||
`local-backend.md`.
|
`prepared-execution.md`.
|
||||||
- Keep the API in the root `promptkit` package. Do not add a public or internal
|
- Keep the supported API in the root `promptkit` package. Internal lifecycle,
|
||||||
package for this feature.
|
validation, and execution state must remain below Go's `internal/`
|
||||||
- Implement the helper as a transparent constructor for the existing
|
boundary.
|
||||||
`Backend` type. Do not add a second backend representation or bypass
|
- Preserve the observable behavior and ordering of existing `Engine.Prepare`
|
||||||
`WithBackend`.
|
and `Engine.Run`. Do not implement the new workflow by redefining either
|
||||||
- Leave normalization, validation, copying, registry construction, capacity
|
existing method in terms of the new public methods.
|
||||||
defaulting, and duplicate detection in their existing owners.
|
- Reuse preparation and post-preparation execution mechanics where their
|
||||||
- Do not add environment-variable discovery, package-global registration,
|
ordering is genuinely shared. Preserve `Run`'s existing early admission
|
||||||
implicit local defaults, model selection, profile construction, or support
|
boundary while ensuring `PrepareExecution` performs complete preparation
|
||||||
for non-OpenAI-compatible transports.
|
without admission.
|
||||||
- Keep tests lean and behavior-focused. Do not duplicate the existing
|
- Do not add durable serialization, retries, provider failover, background
|
||||||
registry and concurrency test matrices merely because the constructor
|
workers, a task queue, capacity reservation during preparation, or
|
||||||
reaches those mechanisms.
|
consumer-specific persistence and redaction policy.
|
||||||
- Update exact GoDoc with the exported declarations. Update current-state
|
- Keep direct API-key values in private execution state only. Every public
|
||||||
consumer guidance only after the corresponding API is implemented.
|
details value, formatted handle, JSON value, result, hash, and error must
|
||||||
- Do not update release notes, create a release, change a module version, or
|
remain credential-free.
|
||||||
tag a commit as part of this work.
|
- Keep tests lean but give durable coverage to the public compatibility,
|
||||||
|
security, concurrency, one-attempt, source-snapshot, cancellation, and
|
||||||
|
capacity invariants. Use real internal collaborators where they are fast and
|
||||||
|
deterministic and fakes at model-generation and synchronization boundaries.
|
||||||
|
- Update exact GoDoc with exported declarations. Update current-state
|
||||||
|
consumer and internal documentation only after the corresponding behavior
|
||||||
|
exists.
|
||||||
|
- Do not add release notes, change a module version, create a release, or tag
|
||||||
|
a commit as part of this work.
|
||||||
|
|
||||||
## Fixed Design
|
## Fixed Design
|
||||||
|
|
||||||
### Exported API
|
### Public API
|
||||||
|
|
||||||
Add these declarations to `backends.go` in package `promptkit`:
|
Add an opaque root-package type and these methods:
|
||||||
|
|
||||||
```go
|
```go
|
||||||
// BackendLocal is the conventional ID used by LocalBackend. It is not a
|
type PreparedExecution struct {
|
||||||
// built-in or reserved backend and must be registered with WithBackend.
|
// Unexported Promptkit-owned state only.
|
||||||
const BackendLocal = "local"
|
}
|
||||||
|
|
||||||
// LocalBackend returns a Backend for a conventional local OpenAI-compatible
|
func (e *Engine) PrepareExecution(
|
||||||
// endpoint.
|
ctx context.Context,
|
||||||
func LocalBackend(endpoint string, concurrencyLimit int) Backend {
|
req RunRequest,
|
||||||
return Backend{
|
) (*PreparedExecution, error)
|
||||||
ID: BackendLocal,
|
|
||||||
Endpoint: endpoint,
|
func (p *PreparedExecution) Details() PreparedRun
|
||||||
ConcurrencyLimit: concurrencyLimit,
|
|
||||||
}
|
func (p *PreparedExecution) Discard()
|
||||||
|
|
||||||
|
func (e *Engine) RunPrepared(
|
||||||
|
ctx context.Context,
|
||||||
|
prepared *PreparedExecution,
|
||||||
|
) (*RunResult, error)
|
||||||
|
```
|
||||||
|
|
||||||
|
Place the public handle and its handle-local methods in a new root file named
|
||||||
|
`prepared_execution.go`. Keep the two engine operations in `engine.go` beside
|
||||||
|
`Prepare` and `Run` so all engine workflows remain discoverable together.
|
||||||
|
|
||||||
|
`PreparedExecution` must contain no exported fields. It wraps one pointer to
|
||||||
|
Promptkit-owned internal state; copying the public struct therefore preserves
|
||||||
|
one shared lifecycle rather than creating another execution opportunity.
|
||||||
|
|
||||||
|
Implement both:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func (p *PreparedExecution) String() string
|
||||||
|
func (p *PreparedExecution) GoString() string
|
||||||
|
```
|
||||||
|
|
||||||
|
with a constant credential- and content-free representation such as
|
||||||
|
`promptkit.PreparedExecution{opaque}`. These methods must not branch on,
|
||||||
|
inspect, or format retained execution state. Do not add a custom JSON
|
||||||
|
representation: with no exported fields, ordinary JSON encoding exposes no
|
||||||
|
state, and JSON is explicitly outside the handle contract.
|
||||||
|
|
||||||
|
The exact GoDoc must own these behaviors:
|
||||||
|
|
||||||
|
- `PrepareExecution` performs complete preparation but no model call and no
|
||||||
|
backend-capacity admission;
|
||||||
|
- the handle is bound to the creating engine and permits one
|
||||||
|
`RunPrepared` invocation;
|
||||||
|
- `Details` returns a fresh caller-owned, credential-redacted copy on every
|
||||||
|
call and remains usable after discard or execution;
|
||||||
|
- `Discard` is nil-safe and idempotent, invalidates an unclaimed handle, and
|
||||||
|
is not an execution-cancellation mechanism;
|
||||||
|
- `RunPrepared` atomically consumes the one attempt before credential
|
||||||
|
revalidation, admission, generation, or validation;
|
||||||
|
- lifecycle misuse matches `ErrInvalidRequest`, while a nil engine matches
|
||||||
|
`ErrInvalidConfig`;
|
||||||
|
- an operational execution error returns no partial `RunResult`;
|
||||||
|
- the `RunPrepared` context does not inherit from the preparation context and
|
||||||
|
governs the execution attempt; and
|
||||||
|
- handles are opaque in-process values, not serializable or restartable jobs.
|
||||||
|
|
||||||
|
For a nil or zero-value handle, `Details` returns a zero `PreparedRun` and
|
||||||
|
`Discard` does nothing. A nil, zero-value, foreign-engine, discarded,
|
||||||
|
claimed, or used handle passed to `RunPrepared` returns an error matching
|
||||||
|
`ErrInvalidRequest`.
|
||||||
|
|
||||||
|
### Internal Validation Snapshot
|
||||||
|
|
||||||
|
Extend `internal/validate/validator.go` with two internal interfaces:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// PreparedValidation validates artifacts against one frozen output contract.
|
||||||
|
type PreparedValidation interface {
|
||||||
|
Validate(
|
||||||
|
ctx context.Context,
|
||||||
|
artifact *domain.Artifact,
|
||||||
|
) (domain.ValidationResult, error)
|
||||||
|
|
||||||
|
// SchemaDocument returns the root JSON Schema document used for provider
|
||||||
|
// structured output, or nil for non-schema modes. The returned value is
|
||||||
|
// internal immutable state and must not be mutated by callers.
|
||||||
|
SchemaDocument() any
|
||||||
|
}
|
||||||
|
|
||||||
|
// ValidationPreparer freezes the validation resources for one output
|
||||||
|
// contract.
|
||||||
|
type ValidationPreparer interface {
|
||||||
|
PrepareValidation(
|
||||||
|
ctx context.Context,
|
||||||
|
contract domain.OutputContract,
|
||||||
|
) (PreparedValidation, error)
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
The exact GoDoc may be wrapped or expanded for clarity, but it must own and
|
The exact names may be adjusted only to avoid an actual Go naming conflict;
|
||||||
communicate all of these contract points:
|
the separation of responsibilities and method shapes are fixed.
|
||||||
|
`validate.Validator` and its existing `Validate` method remain unchanged so
|
||||||
|
the existing `Run` behavior and internal test fakes do not acquire a new
|
||||||
|
mandatory method.
|
||||||
|
|
||||||
- `BackendLocal` is the case-sensitive conventional ID `"local"`;
|
`StandardValidator` and `FSValidator` implement `ValidationPreparer`. Their
|
||||||
- it is neither built in nor reserved;
|
prepared-validation implementation:
|
||||||
- calling `LocalBackend` does not register anything;
|
|
||||||
- the returned value must be supplied through `WithBackend`;
|
|
||||||
- `endpoint` and `concurrencyLimit` are copied into the corresponding fields
|
|
||||||
without normalization or validation;
|
|
||||||
- `APIKeyEnv`, `ExtraParams`, and `QueueCapacity` retain their zero values; and
|
|
||||||
- normal `NewEngine` backend validation and concurrency semantics apply after
|
|
||||||
registration.
|
|
||||||
|
|
||||||
Keep `BackendOpenRouter` unchanged. `BackendLocal` must not alias an internal
|
- captures a value copy of the complete `OutputContract`;
|
||||||
registry constant because the internal registry has no special local-backend
|
- handles none, basic, and JSON validation without any source access;
|
||||||
identity or behavior.
|
- for JSON Schema, loads the root document, validates its dialect, registers
|
||||||
|
it with a new compiler, compiles it during `PrepareValidation`, and retains
|
||||||
|
the compiled schema;
|
||||||
|
- allows the compiler to resolve every transitive reference during that
|
||||||
|
compilation using the existing contained filesystem or `fs.FS` loader;
|
||||||
|
- returns the same loaded root document through `SchemaDocument`, so the
|
||||||
|
provider-facing structured-output constraint and later validation derive
|
||||||
|
from one root snapshot; and
|
||||||
|
- validates generated instances only against retained in-memory state. Its
|
||||||
|
`Validate` method must not reopen a path, call `fs.ReadFile`, resolve a
|
||||||
|
symlink, or invoke a schema loader after preparation succeeds.
|
||||||
|
|
||||||
Place `BackendLocal` near `BackendOpenRouter`, and place `LocalBackend` after
|
Reuse the existing JSON parsing, validation-result construction, schema
|
||||||
the `Backend` declaration and before `WithBackend`. This keeps the conventional
|
dialect checks, contained-path rules, compiler configuration, and validation
|
||||||
IDs, configured value, convenience constructor, and registration option
|
error wording. A prepared JSON Schema compilation or reference-loading failure
|
||||||
discoverable in one file.
|
is a preparation-time validation error. Existing direct calls to
|
||||||
|
`Validator.Validate`, including the existing `Run` path, retain their current
|
||||||
|
lazy compilation and source-loading behavior.
|
||||||
|
|
||||||
### Constructor Semantics
|
Do not expose the JSON Schema library's compiled type outside
|
||||||
|
`internal/validate`. Do not place compiled schemas or validation interfaces in
|
||||||
|
`internal/domain`.
|
||||||
|
|
||||||
`LocalBackend` is a pure struct constructor. Its complete behavior is
|
### Internal Prepared-Execution State
|
||||||
equivalent to the keyed literal shown above.
|
|
||||||
|
|
||||||
In particular, the constructor must not:
|
Add `internal/usecase/prepared_execution.go`. Define an internal
|
||||||
|
`PreparedExecution` owned by one `Runner`. It contains:
|
||||||
|
|
||||||
- trim or parse the endpoint;
|
- the immutable owner `*Runner`;
|
||||||
- reject blank endpoints or negative limits;
|
- a mutex;
|
||||||
- choose a default limit;
|
- a small private lifecycle state with at least ready, claimed, and discarded
|
||||||
- assign an API-key environment variable;
|
states;
|
||||||
- allocate an empty `ExtraParams` map;
|
- a credential-redacted prepared-details snapshot that survives every
|
||||||
- assign a queue-capacity pointer;
|
lifecycle transition; and
|
||||||
- read process environment;
|
- while ready, a pointer to a private payload containing the full
|
||||||
- mutate package-global or engine state; or
|
`domain.PreparedRun` and its `validate.PreparedValidation`.
|
||||||
- call `WithBackend` itself.
|
|
||||||
|
|
||||||
Deferred validation is intentional. It keeps one validation path for all
|
The payload is the only retained object containing `RunRequest.APIKey`.
|
||||||
`Backend` values: `WithBackend` copies the public value into engine options,
|
Construct the public-details snapshot as a deep copy of the prepared run and
|
||||||
and `NewEngine` constructs the validated immutable registry. A positive limit
|
clear its `EffectiveModelParams.APIKey`. Deep-copy every mutable nested value,
|
||||||
with a nil queue continues to select the existing default queue capacity of
|
including messages, maps, target extra parameters, and the structured-output
|
||||||
1024; zero continues to mean unlimited; a negative value continues to make
|
schema. Reuse the repository's JSON-compatible copy helpers rather than
|
||||||
`NewEngine` fail with `ErrInvalidConfig`.
|
maintaining an ad hoc reflection copier.
|
||||||
|
|
||||||
The returned `Backend` remains an ordinary caller-owned value. Consumers can
|
Add these internal operations:
|
||||||
modify it before passing it to `WithBackend`, although task-oriented
|
|
||||||
documentation should direct materially customized configurations to an
|
|
||||||
explicit keyed `Backend` literal.
|
|
||||||
|
|
||||||
### Identity And Compatibility
|
|
||||||
|
|
||||||
Do not add `"local"` to the internal built-in or reserved ID set. A consumer
|
|
||||||
must be able to register either:
|
|
||||||
|
|
||||||
```go
|
```go
|
||||||
promptkit.LocalBackend(endpoint, limit)
|
func (r *Runner) PrepareExecution(
|
||||||
|
ctx context.Context,
|
||||||
|
req domain.RunRequest,
|
||||||
|
) (*PreparedExecution, error)
|
||||||
|
|
||||||
|
func (p *PreparedExecution) Details() *domain.PreparedRun
|
||||||
|
|
||||||
|
func (p *PreparedExecution) Discard()
|
||||||
|
|
||||||
|
func (r *Runner) RunPrepared(
|
||||||
|
ctx context.Context,
|
||||||
|
prepared *PreparedExecution,
|
||||||
|
) (*domain.RunResult, error)
|
||||||
```
|
```
|
||||||
|
|
||||||
or:
|
Internal exported names are acceptable here because the root facade must call
|
||||||
|
them, but they remain inaccessible to downstream consumers through Go's
|
||||||
|
`internal` rule.
|
||||||
|
|
||||||
```go
|
`PrepareExecution` performs these operations in order:
|
||||||
promptkit.Backend{
|
|
||||||
ID: promptkit.BackendLocal,
|
|
||||||
Endpoint: endpoint,
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
through the existing `WithBackend` option. Both forms participate in ordinary
|
1. call the existing request-copy boundary in the root facade before entering
|
||||||
duplicate-ID detection. Existing consumers that already use the literal ID
|
the runner;
|
||||||
`"local"` remain source- and behavior-compatible.
|
2. run `resolvePreparation`, including the existing environment credential
|
||||||
|
availability check;
|
||||||
|
3. prepare a frozen validation plan through
|
||||||
|
`validate.ValidationPreparer`;
|
||||||
|
4. construct provider structured-output metadata from the plan's root schema
|
||||||
|
document for JSON Schema mode;
|
||||||
|
5. load and hash artifacts, render the session and messages, and assemble the
|
||||||
|
complete `domain.PreparedRun`; and
|
||||||
|
6. construct the ready internal handle with independent full and redacted
|
||||||
|
snapshots.
|
||||||
|
|
||||||
No existing `Backend`, `WithBackend`, profile, registry, execution-target, or
|
Extract the artifact-loading, rendering, hashing, timing, and
|
||||||
capacity semantics change. Do not modify `internal/backend`,
|
`domain.PreparedRun` assembly portion of `completePreparation` into a helper
|
||||||
`internal/capacity`, `internal/domain`, `engine.go`, `profiles.go`, or
|
that accepts an already-resolved `*domain.StructuredOutputSpec`.
|
||||||
`types.go` for this feature.
|
`completePreparation` must continue to call the existing
|
||||||
|
`resolveStructuredOutput` first and then use that helper. This preserves
|
||||||
|
existing `Prepare` and `Run` schema behavior. `PrepareExecution` uses the
|
||||||
|
frozen validation plan's schema document to build the same structured-output
|
||||||
|
shape without calling `resolveStructuredOutput`.
|
||||||
|
|
||||||
|
When the runner has no validator, preserve the existing runner semantics:
|
||||||
|
validation is skipped and a no-op prepared plan is sufficient. When it has a
|
||||||
|
non-nil validator that does not implement `ValidationPreparer`,
|
||||||
|
`PrepareExecution` fails with `ErrValidation`; do not fall back to a plan that
|
||||||
|
would reopen sources later. The root engine's standard and `fs.FS` validators
|
||||||
|
always implement the new interface.
|
||||||
|
|
||||||
|
### Claim, Cleanup, And Engine Binding
|
||||||
|
|
||||||
|
`RunPrepared` checks engine ownership before attempting a claim. A foreign
|
||||||
|
runner returns `ErrInvalidRequest` without changing the handle. The owning
|
||||||
|
runner then locks the state and atomically:
|
||||||
|
|
||||||
|
1. accepts only the ready state;
|
||||||
|
2. changes it permanently to claimed;
|
||||||
|
3. detaches the private payload from the handle; and
|
||||||
|
4. unlocks before doing any blocking or collaborator work.
|
||||||
|
|
||||||
|
Every valid owning-engine invocation consumes the attempt at this point. Run
|
||||||
|
ID creation, credential failure, pre-canceled context, capacity rejection,
|
||||||
|
generation failure, operational validation failure, and success all leave the
|
||||||
|
handle claimed and unusable.
|
||||||
|
|
||||||
|
The execution call holds the detached payload locally and defers cleanup on
|
||||||
|
every exit. Cleanup must at minimum:
|
||||||
|
|
||||||
|
- overwrite the direct `EffectiveModelParams.APIKey` field with an empty
|
||||||
|
string;
|
||||||
|
- drop the local prepared-run pointer;
|
||||||
|
- drop the frozen-validation reference; and
|
||||||
|
- leave the separate redacted details snapshot intact.
|
||||||
|
|
||||||
|
This is reference cleanup, not a promise of secure Go string-memory erasure.
|
||||||
|
|
||||||
|
`Discard` locks the same state. If ready, it changes the state to discarded,
|
||||||
|
detaches the payload, unlocks, and performs the same cleanup. If claimed or
|
||||||
|
already discarded, it is a no-op. Therefore a `RunPrepared`/`Discard` race has
|
||||||
|
one winner: discard prevents a claim only if it acquires the ready state
|
||||||
|
first; otherwise it does not cancel the running attempt.
|
||||||
|
|
||||||
|
`Details` locks only long enough to read the retained redacted snapshot and
|
||||||
|
returns a fresh deep copy. It must not return a pointer, slice, map, schema, or
|
||||||
|
extra-parameter tree shared with either the handle or another `Details` call.
|
||||||
|
|
||||||
|
### Prepared Execution Mechanics
|
||||||
|
|
||||||
|
Refactor the post-preparation portion of `Runner.Run` into a private execution
|
||||||
|
helper that accepts:
|
||||||
|
|
||||||
|
- a fully prepared `domain.PreparedRun`;
|
||||||
|
- a caller-supplied run ID and execution start time; and
|
||||||
|
- a validation function that accepts the artifact and repair-attempt count.
|
||||||
|
|
||||||
|
The helper owns model generation, artifact construction, content validation,
|
||||||
|
internal repair, final result assembly, and result end timing. The live
|
||||||
|
`Runner.Run` path supplies its existing `validateOutput` function. The
|
||||||
|
prepared path supplies a wrapper around the retained
|
||||||
|
`validate.PreparedValidation`. Both wrappers set the actual repair-attempt
|
||||||
|
count on the returned validation result.
|
||||||
|
|
||||||
|
Keep admission outside this shared helper:
|
||||||
|
|
||||||
|
- `Runner.Run` retains its current sequence of run-ID creation, start time,
|
||||||
|
target resolution, early admission, complete preparation, execution, and
|
||||||
|
release.
|
||||||
|
- `Runner.RunPrepared` claims the handle, creates the run ID and execution
|
||||||
|
start time, revalidates credential availability, admits the frozen backend,
|
||||||
|
executes the shared helper, and releases admission.
|
||||||
|
|
||||||
|
Extract a small private admission helper only if it preserves the exact
|
||||||
|
existing capacity wrapping and release behavior. The limited-backend lease
|
||||||
|
must cover generation, validation, and all internal repair calls and must be
|
||||||
|
released on every return.
|
||||||
|
|
||||||
|
Before admission, `RunPrepared` calls the existing `validateAPIKey` with the
|
||||||
|
frozen environment-variable name, retained direct key, and frozen requirement
|
||||||
|
flag. A missing environment credential returns an error wrapping
|
||||||
|
`ErrInvalidRequest` and `ErrAPIKeyEnvMissing`. Do not read and retain the
|
||||||
|
environment value; the model client continues to resolve it when generation
|
||||||
|
begins.
|
||||||
|
|
||||||
|
The result produced by the shared helper uses the frozen preparation values
|
||||||
|
for all provenance. For `RunPrepared`, `StartTime`, `EndTime`, and `Duration`
|
||||||
|
begin after the successful claim and exclude preparation and consumer-held
|
||||||
|
delay. For ordinary `Run`, retain the current timing boundary beginning before
|
||||||
|
preparation.
|
||||||
|
|
||||||
|
### Error Mapping And Compatibility
|
||||||
|
|
||||||
|
Do not add a new public sentinel. Internal lifecycle errors wrap
|
||||||
|
`usecase.ErrInvalidRequest`, which the existing root mapping exposes as
|
||||||
|
`promptkit.ErrInvalidRequest`.
|
||||||
|
|
||||||
|
`PrepareExecution` uses the same public categories as `Prepare`.
|
||||||
|
`RunPrepared` may return:
|
||||||
|
|
||||||
|
- `ErrInvalidConfig` for a nil engine;
|
||||||
|
- `ErrInvalidRequest` for handle misuse and invalid credentials;
|
||||||
|
- `ErrAPIKeyEnvMissing` together with `ErrInvalidRequest` when the frozen
|
||||||
|
environment name is no longer set and no direct key is retained;
|
||||||
|
- `ErrCapacityExceeded` for admission rejection;
|
||||||
|
- `ErrLLMGenerate` for model-generation failures; and
|
||||||
|
- `ErrValidation` for inability to validate or repair.
|
||||||
|
|
||||||
|
Preserve wrapped collaborator and context errors according to the existing
|
||||||
|
mapping rules. A content validation rejection remains a successful
|
||||||
|
`RunResult` with `ValidationFailed`, not an operational error.
|
||||||
|
|
||||||
|
Do not change stable JSON for `PreparedRun` or `RunResult`, public request and
|
||||||
|
result ownership, provider request bodies or headers, backend selection,
|
||||||
|
capacity policies, `LLMClient`, `ArtifactReader`, `Prepare`, or `Run`.
|
||||||
|
|
||||||
### Test Ownership
|
### Test Ownership
|
||||||
|
|
||||||
The root external-package contract suite in `public_contract_test.go` owns the
|
Tests must be split by the narrowest stable owner.
|
||||||
new public behavior. Add one focused test named:
|
|
||||||
|
|
||||||
```go
|
`internal/validate/standard_validator_test.go` owns frozen schema mechanics.
|
||||||
func TestLocalBackendConstructsAndRegistersConventionalBackend(t *testing.T)
|
Add focused tests for both configured schema-source forms:
|
||||||
```
|
|
||||||
|
|
||||||
The test must:
|
- a directory-backed `StandardValidator` test whose root schema references a
|
||||||
|
second file; after `PrepareValidation`, replace or remove both files and
|
||||||
|
prove that valid and invalid artifacts are judged by the original compiled
|
||||||
|
schema; and
|
||||||
|
- an `FSValidator` test using a mutable in-memory `fs.FS`; mutate its root and
|
||||||
|
referenced entries after `PrepareValidation` and prove the same invariant.
|
||||||
|
|
||||||
1. call `promptkit.LocalBackend` with a test endpoint and a positive,
|
Each test must also confirm that `SchemaDocument` is the original root
|
||||||
test-owned concurrency limit;
|
document used at preparation. Existing path-containment and dialect matrices
|
||||||
2. compare the returned value with this complete expected value:
|
remain the owners of those rules and must not be duplicated.
|
||||||
|
|
||||||
```go
|
`internal/usecase` tests own orchestration details that cannot be observed
|
||||||
promptkit.Backend{
|
cleanly at the facade:
|
||||||
ID: promptkit.BackendLocal,
|
|
||||||
Endpoint: endpoint,
|
|
||||||
ConcurrencyLimit: limit,
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
A whole-value comparison is appropriate here because the exact zero-value
|
- `PrepareExecution` completes source loading, artifact reading, rendering,
|
||||||
fields are part of this small public constructor's contract.
|
and validation-plan creation without calling admission or generation;
|
||||||
3. register that returned value with `WithBackend`;
|
- `RunPrepared` rechecks environment credential availability before admission;
|
||||||
4. add an in-memory profile whose `BackendID` is
|
- the prepared path supplies the frozen validation plan to initial validation
|
||||||
`promptkit.BackendLocal`;
|
and every repair attempt;
|
||||||
5. construct an engine through the existing contract-test prompt fixture;
|
- admission is released on success and each error exit; and
|
||||||
6. call `Prepare`; and
|
- the ordinary `Run` admission-before-complete-preparation ordering remains
|
||||||
7. assert that the prepared result exposes `BackendLocal` as the selected
|
unchanged after refactoring.
|
||||||
backend and the supplied endpoint as the effective endpoint.
|
|
||||||
|
|
||||||
This single test protects the realistic compatibility risks: accidental field
|
Use small test-controlled collaborators. Do not assert private enum values,
|
||||||
defaults, a changed conventional ID, failure to compose with `WithBackend`,
|
mutex layout, helper call graphs, or cleanup implementation details.
|
||||||
and accidental treatment of `"local"` as reserved. It also demonstrates that
|
|
||||||
the helper uses the existing backend/profile path.
|
|
||||||
|
|
||||||
Do not add separate tests for blank endpoints, malformed endpoints, negative
|
Add `prepared_execution_contract_test.go` in external package
|
||||||
limits, queue defaulting, duplicate IDs, engine isolation, runtime capacity,
|
`promptkit_test` for the public contract. Cover these distinct risks with the
|
||||||
or caller mutation. Those mechanisms are unchanged and already have tests at
|
smallest coherent set of tests:
|
||||||
their owning boundaries. Do not add internal-package tests for this root
|
|
||||||
facade constructor.
|
|
||||||
|
|
||||||
### Consumer Documentation
|
1. **Frozen execution and independent details.** Prepare a request, mutate
|
||||||
|
caller-owned request values, mutable prompt/profile/input/schema sources,
|
||||||
|
and one returned `PreparedRun`, then run the handle. Assert that the
|
||||||
|
captured `GenerateRequest`, validation, result provenance, and a second
|
||||||
|
`Details` call all retain the original prepared state. Assert that source
|
||||||
|
collaborators are not reopened during execution.
|
||||||
|
2. **Lifecycle and engine binding.** Prove that a foreign engine is rejected
|
||||||
|
without consuming the handle, the owner can then run it once, a second
|
||||||
|
call through either the original or a copied public handle is rejected, and
|
||||||
|
details remain available.
|
||||||
|
3. **Concurrent claim.** Race two owning-engine `RunPrepared` calls against a
|
||||||
|
blocking fake client. Exactly one reaches generation and the other matches
|
||||||
|
`ErrInvalidRequest`. Run this test under the race detector.
|
||||||
|
4. **Discard and formatting security.** Retain a distinctive direct API key,
|
||||||
|
verify that `Details`, ordinary JSON encoding, `%v`, `%+v`, `%#v`, and
|
||||||
|
lifecycle errors do not contain it or rendered content, discard the handle,
|
||||||
|
and verify idempotence, execution rejection, and retained details.
|
||||||
|
5. **Credential and capacity timing.** Prove that preparation does not occupy
|
||||||
|
admission; removing a required environment credential makes execution fail
|
||||||
|
before generation; a capacity rejection consumes the handle; and the
|
||||||
|
execution result's timing excludes a test-controlled delay between
|
||||||
|
preparation and execution.
|
||||||
|
|
||||||
Update `docs/consumers/pkg-promptkit.md` after the API exists. Keep Go
|
Reuse existing root contract fixtures and capacity fakes where practical.
|
||||||
declarations and GoDoc as the exact API owner; the guide should help consumers
|
Never put a real-looking credential in a fixture. Do not test JSON equality
|
||||||
choose a workflow and link to `backends.go` for precise semantics.
|
for the opaque handle, exact lifecycle error strings, private state, run-ID
|
||||||
|
format, or clock-duration precision.
|
||||||
|
|
||||||
Restructure the backend guidance to present these paths in increasing order of
|
### Documentation Ownership
|
||||||
configuration:
|
|
||||||
|
|
||||||
1. **Endpoint-only profile.** Show a small in-memory `Profile` with
|
After implementation:
|
||||||
`Endpoint` and `Model`. Explain that this is the simplest choice when only
|
|
||||||
one profile needs the endpoint and shared backend identity or capacity
|
|
||||||
policy is unnecessary.
|
|
||||||
2. **Local convenience constructor.** Show
|
|
||||||
`WithBackend(promptkit.LocalBackend("http://localhost:8000/v1", 2))`
|
|
||||||
together with a profile using
|
|
||||||
`BackendID: promptkit.BackendLocal`. Explain briefly that the helper is
|
|
||||||
explicit, is not pre-registered, does not read environment variables, and
|
|
||||||
leaves the queue capacity at the existing default for a positive limit.
|
|
||||||
3. **Complete backend value.** Preserve an advanced example using a keyed
|
|
||||||
`Backend` literal for needs such as `APIKeyEnv`, an explicit
|
|
||||||
`QueueCapacity`, extra parameters, a custom ID, or multiple local
|
|
||||||
endpoints. Use a custom ID other than `"local"` in that example so the
|
|
||||||
distinction from the conventional helper is clear.
|
|
||||||
|
|
||||||
Keep the existing backend-routing, selected-backend identity, concurrency,
|
- exported declarations and GoDoc in `prepared_execution.go`, `engine.go`, and
|
||||||
credential, and error guidance unless a small wording adjustment is required
|
`types.go` own the exact public lifecycle, ownership, security, timing,
|
||||||
to make the new decision path coherent. Avoid repeating the complete field
|
cancellation, and error contracts;
|
||||||
contract or registry validation rules from GoDoc.
|
- `docs/consumers/pkg-promptkit.md` explains when to choose `Prepare`, `Run`,
|
||||||
|
or `PrepareExecution` plus `RunPrepared`, and shows one concise two-phase
|
||||||
|
workflow using `defer prepared.Discard()`;
|
||||||
|
- `docs/internal/sources.md` explains that prepared execution freezes all
|
||||||
|
source-derived state, including transitive schema references, while
|
||||||
|
ordinary `Prepare` remains inspection-only;
|
||||||
|
- `docs/internal/llm.md` explains that prepared generation uses the retained
|
||||||
|
target and messages, rechecks environment credential availability, and
|
||||||
|
does not reopen sources;
|
||||||
|
- `docs/internal/capacity.md` explains that `PrepareExecution` performs no
|
||||||
|
admission and that `RunPrepared` acquires and holds the normal run lease
|
||||||
|
across generation, validation, and internal repair; and
|
||||||
|
- `docs/internal/overview.md` updates the validator and use-case inventory to
|
||||||
|
mention frozen validation plans and one-attempt prepared execution.
|
||||||
|
|
||||||
Do not add a new maintained example, README section, format-reference entry,
|
Keep these documents at their established abstraction levels. Link to exact
|
||||||
integration-contract change, internal-document change, or release note. The
|
Go declarations rather than reproducing field-by-field contracts. No README,
|
||||||
consumer-guide snippets are sufficient for this small convenience API, and
|
format reference, integration contract, maintained example, architecture
|
||||||
none of those other documents owns the affected task or contract.
|
policy, release note, or release-procedure update is required because the
|
||||||
|
feature changes none of their owned concerns.
|
||||||
|
|
||||||
## Stage 1: Add The Public Constructor And Contract Test
|
## Stage 1: Add Frozen Validation Plans
|
||||||
|
|
||||||
**Status:** Complete.
|
**Status:** Pending.
|
||||||
|
|
||||||
### Objective
|
### Objective
|
||||||
|
|
||||||
Add the smallest public API that expresses the accepted local-backend
|
Create an internal validation artifact that eagerly captures every schema
|
||||||
convention and protect its compatibility through the root public boundary.
|
resource required by later validation, without changing existing validation
|
||||||
|
or engine behavior.
|
||||||
|
|
||||||
### Implementation Prompt
|
### Implementation Prompt
|
||||||
|
|
||||||
1. Re-read `docs/development.md`, all files under `docs/policy/`,
|
1. Re-read `docs/development.md`, all files under `docs/policy/`,
|
||||||
`local-backend.md`, `backends.go`, the backend-related portion of
|
`prepared-execution.md`, `internal/validate/validator.go`,
|
||||||
`public_contract_test.go`, and the existing backend/concurrency GoDoc before
|
`internal/validate/standard_validator.go`, and the focused validator tests.
|
||||||
editing.
|
2. Confirm the working tree and preserve all existing roadmap and wishlist
|
||||||
2. Confirm the working tree and preserve the uncommitted roadmaps and any
|
changes.
|
||||||
unrelated consumer changes.
|
3. Add `PreparedValidation` and `ValidationPreparer` to
|
||||||
3. Add the untyped exported string constant `BackendLocal = "local"` to
|
`internal/validate/validator.go` with the exact responsibilities in the
|
||||||
`backends.go` without changing `BackendOpenRouter`.
|
fixed design. Leave `Validator` unchanged.
|
||||||
4. Add `LocalBackend(endpoint string, concurrencyLimit int) Backend` to
|
4. Implement `PrepareValidation` for `StandardValidator` and `FSValidator`.
|
||||||
`backends.go` using the exact keyed-literal implementation in the fixed
|
Reuse the existing root resolution, resource identifiers, contained
|
||||||
design.
|
loaders, dialect checks, compiler configuration, JSON parser, and
|
||||||
5. Write complete GoDoc for both declarations. Make their conventional,
|
validation-result builder.
|
||||||
explicit, non-built-in, non-reserved, and deferred-validation semantics
|
5. Ensure the root document is registered from the already-loaded value and
|
||||||
unambiguous.
|
every transitive reference is loaded during compilation. Retain only the
|
||||||
6. Add
|
compiled schema, root document, and copied contract required for later
|
||||||
`TestLocalBackendConstructsAndRegistersConventionalBackend` to
|
validation.
|
||||||
`public_contract_test.go` exactly as specified under Test Ownership. Reuse
|
6. Ensure the prepared validator checks its validation-time context and does
|
||||||
the existing contract prompt fixture rather than adding a fixture or test
|
not touch either source after successful preparation.
|
||||||
helper.
|
7. Add the two focused frozen-reference tests described under Test Ownership.
|
||||||
7. Do not edit internal packages. If the constructor appears to require an
|
Keep existing tests unchanged except for shared test setup that materially
|
||||||
internal change, stop and reconcile the implementation with the fixed
|
reduces duplication.
|
||||||
transparent-constructor design instead.
|
|
||||||
|
|
||||||
### Focused Validation
|
### Focused Validation
|
||||||
|
|
||||||
Run:
|
Run:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
gofmt -w backends.go public_contract_test.go
|
gofmt -w internal/validate/validator.go \
|
||||||
go test . -run '^TestLocalBackendConstructsAndRegistersConventionalBackend$'
|
internal/validate/standard_validator.go \
|
||||||
|
internal/validate/standard_validator_test.go
|
||||||
|
go test ./internal/validate
|
||||||
|
go test -race ./internal/validate
|
||||||
|
go vet ./internal/validate
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
### Completion Gate
|
||||||
|
|
||||||
|
Stage 1 is complete when:
|
||||||
|
|
||||||
|
- both built-in validator forms produce prepared plans;
|
||||||
|
- prepared schema validation works after root and referenced sources change or
|
||||||
|
disappear;
|
||||||
|
- provider schema metadata can use the exact root document captured by the
|
||||||
|
plan;
|
||||||
|
- existing `Validator.Validate` behavior remains unchanged; and
|
||||||
|
- focused tests, race tests, vet, formatting, and whitespace checks pass.
|
||||||
|
|
||||||
|
## Stage 2: Implement Internal Handle Lifecycle And Prepared Execution
|
||||||
|
|
||||||
|
**Status:** Pending.
|
||||||
|
|
||||||
|
### Objective
|
||||||
|
|
||||||
|
Add complete preparation, one-attempt lifecycle management, cleanup, and
|
||||||
|
source-free execution to the use-case layer while preserving existing
|
||||||
|
`Prepare` and `Run` behavior.
|
||||||
|
|
||||||
|
### Implementation Prompt
|
||||||
|
|
||||||
|
1. Re-read the completed Stage 1 declarations, `internal/usecase/runner.go`,
|
||||||
|
its focused tests, `internal/capacity`, and the execution flow described in
|
||||||
|
`prepared-execution.md`.
|
||||||
|
2. Add `internal/usecase/prepared_execution.go` with the internal handle,
|
||||||
|
payload, lifecycle transitions, deep-copying, redaction, cleanup, and
|
||||||
|
runner ownership described in the fixed design.
|
||||||
|
3. Extract the common final preparation helper that accepts resolved
|
||||||
|
structured-output metadata. Keep `completePreparation` and existing
|
||||||
|
`resolveStructuredOutput` behavior intact for `Prepare` and `Run`.
|
||||||
|
4. Implement `Runner.PrepareExecution` using
|
||||||
|
`validate.ValidationPreparer`. Build JSON Schema structured-output metadata
|
||||||
|
from the frozen plan's root document.
|
||||||
|
5. Refactor post-preparation generation, validation, repair, and result
|
||||||
|
assembly into the shared private helper. Preserve ordinary `Run` timing,
|
||||||
|
early admission, error ordering, and live validator behavior.
|
||||||
|
6. Implement `Runner.RunPrepared`: verify owner, claim once, create run ID and
|
||||||
|
execution start time, recheck credentials, acquire admission, execute
|
||||||
|
against the retained plan, release admission, and clean the detached
|
||||||
|
payload on every exit.
|
||||||
|
7. Implement nil-safe, idempotent discard and fresh redacted details copying.
|
||||||
|
Confirm the `RunPrepared`/`Discard` race is resolved solely by the shared
|
||||||
|
mutex transition.
|
||||||
|
8. Add the internal orchestration tests described under Test Ownership,
|
||||||
|
including regression coverage for ordinary `Run`'s admission ordering.
|
||||||
|
|
||||||
|
### Focused Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
gofmt -w internal/usecase/runner.go \
|
||||||
|
internal/usecase/prepared_execution.go \
|
||||||
|
internal/usecase/*_test.go
|
||||||
|
go test ./internal/usecase
|
||||||
|
go test -race ./internal/usecase
|
||||||
|
go test ./internal/validate
|
||||||
|
go vet ./internal/usecase ./internal/validate
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
### Completion Gate
|
||||||
|
|
||||||
|
Stage 2 is complete when:
|
||||||
|
|
||||||
|
- a runner can prepare and later execute without reopening any request source;
|
||||||
|
- one valid invocation consumes the handle on every outcome;
|
||||||
|
- foreign-runner rejection does not consume the handle;
|
||||||
|
- discard and claim are race-safe and cleanup drops secret-bearing state;
|
||||||
|
- prepared execution rechecks credentials and admits only at execution time;
|
||||||
|
- frozen validation is reused across initial output and repair attempts;
|
||||||
|
- existing `Prepare` and `Run` tests retain their observable ordering and
|
||||||
|
behavior; and
|
||||||
|
- focused tests, race tests, vet, formatting, and whitespace checks pass.
|
||||||
|
|
||||||
|
## Stage 3: Publish The Root Facade And Public Contract
|
||||||
|
|
||||||
|
**Status:** Pending.
|
||||||
|
|
||||||
|
### Objective
|
||||||
|
|
||||||
|
Expose the accepted opaque handle workflow through the root package and
|
||||||
|
protect its compatibility, security, and concurrency guarantees at the
|
||||||
|
consumer boundary.
|
||||||
|
|
||||||
|
### Implementation Prompt
|
||||||
|
|
||||||
|
1. Re-read the accepted public workflow, root conversion and error-mapping
|
||||||
|
helpers, existing `Prepare` and `Run` GoDoc, and the external-package
|
||||||
|
contract-test conventions.
|
||||||
|
2. Add root `prepared_execution.go` with the opaque wrapper, `Details`,
|
||||||
|
`Discard`, `String`, and `GoString`.
|
||||||
|
3. Add `Engine.PrepareExecution` and `Engine.RunPrepared` to `engine.go`.
|
||||||
|
Reuse `toDomainRunRequest`, `fromDomainPreparedRun`,
|
||||||
|
`fromDomainRunResult`, and `mapPublicError`.
|
||||||
|
4. Add or reuse one public prepared-run deep-copy helper so every `Details`
|
||||||
|
call returns fresh nested state. Do not expose the internal handle or
|
||||||
|
retain a public direct credential.
|
||||||
|
5. Write complete GoDoc for the type and methods. Update `RunRequest`,
|
||||||
|
`PreparedRun`, and `RunResult` GoDoc only where cross-references are needed
|
||||||
|
to distinguish the new workflow; do not change their stable JSON or
|
||||||
|
ownership contracts.
|
||||||
|
6. Add `prepared_execution_contract_test.go` and cover the five risk groups
|
||||||
|
under Test Ownership. Reuse existing fixtures and fakes rather than
|
||||||
|
duplicating backend, profile, schema, and capacity matrices.
|
||||||
|
7. Explicitly inspect formatted and JSON-encoded handles, lifecycle errors,
|
||||||
|
details, captured model requests, and results for the test sentinel direct
|
||||||
|
key.
|
||||||
|
|
||||||
|
### Focused Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
gofmt -w prepared_execution.go engine.go types.go convert.go \
|
||||||
|
prepared_execution_contract_test.go
|
||||||
|
go test . -run 'PreparedExecution|RunPrepared|PrepareExecution'
|
||||||
|
go test -race . -run 'PreparedExecution|RunPrepared|PrepareExecution'
|
||||||
go test .
|
go test .
|
||||||
go vet .
|
go vet .
|
||||||
go build .
|
go build .
|
||||||
git diff --check
|
git diff --check
|
||||||
```
|
```
|
||||||
|
|
||||||
Inspect the diff and confirm that this stage changes only `backends.go`,
|
|
||||||
`public_contract_test.go`, and the already-present roadmap files.
|
|
||||||
|
|
||||||
### Completion Gate
|
### Completion Gate
|
||||||
|
|
||||||
Stage 1 is complete when:
|
Stage 3 is complete when:
|
||||||
|
|
||||||
- the exported constant and constructor match the fixed API;
|
- the public API matches the accepted shape;
|
||||||
- the constructor returns only the three specified non-zero fields;
|
- details are fresh, stable, credential-redacted copies before and after every
|
||||||
- the helper registers through the ordinary `WithBackend` path;
|
lifecycle outcome;
|
||||||
- `"local"` remains a valid consumer registration rather than a reserved
|
- default formatting and JSON expose no retained request or credential data;
|
||||||
built-in;
|
- foreign, copied, concurrent, discarded, and reused handles obey the
|
||||||
- the focused public-contract test passes; and
|
one-attempt contract;
|
||||||
- the root package test, vet, build, formatting, and whitespace checks pass.
|
- execution uses exactly the frozen request and validation state;
|
||||||
|
- preparation performs no generation or admission;
|
||||||
|
- result provenance matches details and execution timing excludes preparation
|
||||||
|
and retention delay;
|
||||||
|
- public errors preserve the required `errors.Is` identities; and
|
||||||
|
- focused public, race, package, vet, build, formatting, and whitespace checks
|
||||||
|
pass.
|
||||||
|
|
||||||
## Stage 2: Publish Consumer Guidance And Validate The Repository
|
## Stage 4: Update Documentation And Validate The Repository
|
||||||
|
|
||||||
**Status:** Complete.
|
**Status:** Pending.
|
||||||
|
|
||||||
### Objective
|
### Objective
|
||||||
|
|
||||||
Make the simplest suitable local-endpoint configuration easy to discover,
|
Publish current-state guidance at the correct documentation owners, validate
|
||||||
confirm the complete change across the repository, and close the temporary
|
the complete repository, and mark the temporary roadmaps complete.
|
||||||
roadmaps.
|
|
||||||
|
|
||||||
### Implementation Prompt
|
### Implementation Prompt
|
||||||
|
|
||||||
1. Re-read the implemented declarations and GoDoc before describing them.
|
1. Re-read the implemented declarations and tests before documenting them.
|
||||||
2. Update `docs/consumers/pkg-promptkit.md` according to the three-path
|
2. Update `docs/consumers/pkg-promptkit.md`,
|
||||||
structure under Consumer Documentation.
|
`docs/internal/sources.md`, `docs/internal/llm.md`,
|
||||||
3. Keep examples illustrative, minimal, secret-free, and consistent with the
|
`docs/internal/capacity.md`, and `docs/internal/overview.md` according to
|
||||||
implemented declarations. Link precise semantics to `backends.go` rather
|
Documentation Ownership.
|
||||||
than duplicating its field-by-field contract.
|
3. Keep the consumer example concise, copyable, secret-free, and explicit
|
||||||
|
about `defer prepared.Discard()`. Link to declarations for exact lifecycle
|
||||||
|
and error semantics.
|
||||||
4. Follow every added or changed Markdown link and confirm its target exists.
|
4. Follow every added or changed Markdown link and confirm its target exists.
|
||||||
Confirm that all local repository-relative links in
|
Confirm that repository-relative links in both roadmap documents and all
|
||||||
`local-backend.md`, this plan, and the changed consumer guide resolve.
|
changed current-state documents resolve.
|
||||||
5. Run the complete maintainer validation sequence below.
|
5. Run the complete maintainer validation sequence below.
|
||||||
6. Inspect the final diff for accidental internal behavior changes,
|
6. Inspect the final diff for accidental provider-wire changes, source
|
||||||
generated artifacts, credentials, local workspace files, or module
|
reloading during prepared execution, stable-JSON changes, credentials,
|
||||||
replacements.
|
generated artifacts, local workspace files, module replacements, or
|
||||||
7. After every completion gate passes, set both stage statuses, this plan's
|
unrelated edits.
|
||||||
status, and the status in `local-backend.md` to `Complete`. Do not retire or
|
7. After every completion gate passes, set every stage status, this plan's
|
||||||
remove the roadmaps in the implementation change; roadmap retirement
|
status, and the status in `prepared-execution.md` to `Complete`. Do not
|
||||||
follows implementation review.
|
retire or remove the roadmaps in the implementation change; roadmap
|
||||||
|
retirement follows implementation review.
|
||||||
|
|
||||||
### Full Validation
|
### Full Validation
|
||||||
|
|
||||||
@@ -318,69 +686,37 @@ Run the complete sequence from `docs/development.md`:
|
|||||||
```sh
|
```sh
|
||||||
go test ./...
|
go test ./...
|
||||||
go test -race ./...
|
go test -race ./...
|
||||||
|
go run ./examples/go-library/prepare
|
||||||
go vet ./...
|
go vet ./...
|
||||||
go build ./...
|
go build ./...
|
||||||
go run ./examples/go-library/prepare
|
|
||||||
gofmt -l $(git ls-files '*.go')
|
gofmt -l $(git ls-files '*.go')
|
||||||
git diff --check
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The formatting command must produce no paths. Follow every added or changed
|
Then run the documented Markdown-link and repository-hygiene checks from
|
||||||
Markdown link and confirm its target and heading exist.
|
`docs/development.md`, followed by:
|
||||||
|
|
||||||
Also inspect:
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
|
git diff --check
|
||||||
git status --short
|
git status --short
|
||||||
git diff --stat
|
|
||||||
git diff
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Confirm that:
|
|
||||||
|
|
||||||
- production changes are limited to the root convenience API;
|
|
||||||
- no internal backend, registry, capacity, profile, or engine behavior
|
|
||||||
changed;
|
|
||||||
- `BackendLocal` is conventional and consumer-registerable, not built in or
|
|
||||||
reserved;
|
|
||||||
- `LocalBackend` performs no validation, normalization, environment lookup,
|
|
||||||
registration, allocation, or hidden default selection;
|
|
||||||
- the returned `Backend` leaves `QueueCapacity` nil so existing positive-limit
|
|
||||||
defaulting remains owned by the registry;
|
|
||||||
- existing endpoint-only profiles and complete custom backends remain
|
|
||||||
documented and supported;
|
|
||||||
- current-state documentation describes only the now-implemented API and
|
|
||||||
links to the canonical GoDoc for exact semantics;
|
|
||||||
- no `go.work`, `go.work.sum`, local module replacement, credential,
|
|
||||||
generated binary, or unrelated change was introduced; and
|
|
||||||
- the feature and implementation roadmaps contain no unresolved work marked
|
|
||||||
complete.
|
|
||||||
|
|
||||||
### Completion Gate
|
### Completion Gate
|
||||||
|
|
||||||
Stage 2 is complete when every target-end-state item in
|
Stage 4 and the feature are complete when:
|
||||||
`local-backend.md` is implemented, the consumer guide clearly presents all
|
|
||||||
three configuration paths, all complete validation commands pass, all changed
|
|
||||||
links resolve, and the roadmap statuses accurately report completion.
|
|
||||||
|
|
||||||
## Implementation Handoff
|
- exact public contracts and task-oriented guidance are documented at their
|
||||||
|
canonical owners;
|
||||||
The implementation handoff should report:
|
- internal source, validation, execution, credential, and capacity boundaries
|
||||||
|
are accurately described without duplicating public GoDoc;
|
||||||
- the new `BackendLocal` and `LocalBackend` public API;
|
- all changed links resolve and no documentation claims unimplemented
|
||||||
- that the helper remains explicit and composes with the ordinary backend
|
behavior;
|
||||||
registry;
|
- the complete ordinary and race test suites pass offline;
|
||||||
- the focused public-contract coverage added;
|
- the maintained preparation example, vet, build, formatting, link, hygiene,
|
||||||
- the consumer-guide decision path added;
|
and whitespace checks pass;
|
||||||
- the complete validation commands and results; and
|
- the final diff is limited to the feature, its tests, documentation, and
|
||||||
- any unrelated working-tree changes that were preserved.
|
already-present roadmap changes; and
|
||||||
|
- both roadmap documents accurately report completion.
|
||||||
Do not claim that a local backend is built in, pre-registered, configured from
|
|
||||||
environment variables, or assigned a default model or concurrency limit.
|
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
None. The feature roadmap and this plan fix the exported API, constructor
|
None.
|
||||||
semantics, identity treatment, compatibility behavior, test boundary,
|
|
||||||
documentation ownership, implementation sequence, validation gates, and
|
|
||||||
non-goals required for implementation.
|
|
||||||
|
|||||||
358
docs/roadmap/notarius-promptkit-wishlist.md
Normal file
358
docs/roadmap/notarius-promptkit-wishlist.md
Normal file
@@ -0,0 +1,358 @@
|
|||||||
|
# Notarius PromptKit Wishlist
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document records features and interface changes that would be useful
|
||||||
|
additions to PromptKit from the perspective of the maintainers of Notarius, a
|
||||||
|
downstream application that consumes PromptKit.
|
||||||
|
|
||||||
|
PromptKit v0.3.0 provides the capabilities Notarius currently needs. None of
|
||||||
|
the ideas below blocks current Notarius development. They are opportunities to
|
||||||
|
reduce downstream workarounds, improve integration correctness, and make
|
||||||
|
PromptKit more ergonomic for applications with configuration validation,
|
||||||
|
debugging, checkpointing, and operational-observability requirements.
|
||||||
|
|
||||||
|
The examples are API sketches intended to communicate the desired capability,
|
||||||
|
not prescriptive names or finalized Go contracts.
|
||||||
|
|
||||||
|
## Priority 1: Atomic Execution With Prepared Details
|
||||||
|
|
||||||
|
**Disposition:** Covered by the accepted
|
||||||
|
[executable preparation handles](prepared-execution.md) roadmap. The shared
|
||||||
|
two-phase capability should provide the required single-preparation
|
||||||
|
consistency; a separate `RunDetailed` method is not cataloged initially.
|
||||||
|
|
||||||
|
### Downstream need
|
||||||
|
|
||||||
|
Notarius needs both:
|
||||||
|
|
||||||
|
- the completed `RunResult`; and
|
||||||
|
- the rendered messages, effective output contract, hashes, and other
|
||||||
|
preparation details exposed by `PreparedRun`.
|
||||||
|
|
||||||
|
Notarius uses the prepared details to construct redaction-aware debug bundles
|
||||||
|
and retain enough information to diagnose model behavior.
|
||||||
|
|
||||||
|
### Current integration
|
||||||
|
|
||||||
|
Notarius currently calls `Engine.Prepare` and then `Engine.Run` with the same
|
||||||
|
request. Because `Run` performs preparation internally, a successful request
|
||||||
|
resolves and prepares the same work twice.
|
||||||
|
|
||||||
|
This duplicates profile resolution, input hashing, schema loading, and prompt
|
||||||
|
rendering. It also creates a theoretical consistency window in which a
|
||||||
|
filesystem-backed prompt, profile, schema, or input could change between the
|
||||||
|
explicit preparation and the preparation performed by `Run`.
|
||||||
|
|
||||||
|
### Requested capability
|
||||||
|
|
||||||
|
Add an opt-in execution method that prepares exactly once and returns both the
|
||||||
|
prepared details and completed result:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type RunReport struct {
|
||||||
|
Prepared PreparedRun
|
||||||
|
Result RunResult
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *Engine) RunDetailed(
|
||||||
|
ctx context.Context,
|
||||||
|
req RunRequest,
|
||||||
|
) (*RunReport, error)
|
||||||
|
```
|
||||||
|
|
||||||
|
The exact names are flexible. The important contract is that preparation
|
||||||
|
occurs once and that the returned prepared state describes the execution that
|
||||||
|
produced the returned result.
|
||||||
|
|
||||||
|
Existing `Prepare` and `Run` behavior should remain available for consumers
|
||||||
|
that need only one side of the operation.
|
||||||
|
|
||||||
|
### Design considerations
|
||||||
|
|
||||||
|
- Keep this API additive and preserve the existing simple `Run` workflow.
|
||||||
|
- Return caller-owned copies under PromptKit's existing ownership rules.
|
||||||
|
- Define whether any prepared details are available after an operational
|
||||||
|
generation or validation error. Notarius does not require partial results
|
||||||
|
for the initial use case, but an explicit contract would be valuable.
|
||||||
|
- Preserve cancellation and backend-admission semantics.
|
||||||
|
- Do not add all prepared content directly to `RunResult`. Rendered prompt
|
||||||
|
content can be large and sensitive, and consumers should opt in to receiving
|
||||||
|
it.
|
||||||
|
|
||||||
|
### Value to Notarius
|
||||||
|
|
||||||
|
This is the highest-value wishlist item. It would remove duplicate work from
|
||||||
|
every successful PromptKit-backed call and ensure that retained debug material
|
||||||
|
corresponds atomically to the actual execution.
|
||||||
|
|
||||||
|
## Priority 2: Prompt-Independent Profile Inspection
|
||||||
|
|
||||||
|
**Disposition:** Accepted into the
|
||||||
|
[future catalog](future.md#prompt-independent-profile-inspection).
|
||||||
|
|
||||||
|
### Downstream need
|
||||||
|
|
||||||
|
Notarius validates configured pipeline profile IDs before beginning a run. It
|
||||||
|
needs to determine whether:
|
||||||
|
|
||||||
|
- a profile exists;
|
||||||
|
- its referenced backend is registered;
|
||||||
|
- its execution target can be resolved; and
|
||||||
|
- it declares a credential requirement that the application may need to
|
||||||
|
enforce.
|
||||||
|
|
||||||
|
This validation should not require model generation.
|
||||||
|
|
||||||
|
### Current integration
|
||||||
|
|
||||||
|
Notarius constructs a synthetic prompt using `testing/fstest.MapFS`, supplies a
|
||||||
|
dummy transcript, and calls `Engine.Prepare` solely to exercise profile and
|
||||||
|
backend resolution. This works, but prompt preparation is serving as a
|
||||||
|
substitute for a profile-inspection interface.
|
||||||
|
|
||||||
|
### Requested capability
|
||||||
|
|
||||||
|
Add a prompt-independent profile-resolution API, for example:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type ResolvedProfile struct {
|
||||||
|
ProfileID string
|
||||||
|
BackendID string
|
||||||
|
EffectiveTarget ExecutionTarget
|
||||||
|
APIKeyEnv string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *Engine) ResolveProfile(
|
||||||
|
ctx context.Context,
|
||||||
|
profileID string,
|
||||||
|
) (ResolvedProfile, error)
|
||||||
|
```
|
||||||
|
|
||||||
|
The returned shape may differ, but it should provide enough information for a
|
||||||
|
consumer to validate an explicit profile selection without inventing a prompt
|
||||||
|
or supplying placeholder inputs.
|
||||||
|
|
||||||
|
### Design considerations
|
||||||
|
|
||||||
|
- Resolve built-in, file-backed, and programmatic profiles using normal
|
||||||
|
PromptKit precedence.
|
||||||
|
- Validate that a referenced backend registration exists.
|
||||||
|
- Do not resolve, retain, or expose credential values.
|
||||||
|
- Report credential requirements, such as an environment-variable name, so
|
||||||
|
the consuming application can decide whether availability is required at
|
||||||
|
configuration-validation time or only at execution time.
|
||||||
|
- Return caller-owned values.
|
||||||
|
- Preserve typed or sentinel error classification for missing and invalid
|
||||||
|
profiles.
|
||||||
|
- Consider accepting an `ExecutionTargetOverride` if consumers need to inspect
|
||||||
|
the same effective target that a run-level override would produce.
|
||||||
|
- Enumeration of all profiles is not required for the Notarius use case; exact
|
||||||
|
lookup by ID is sufficient.
|
||||||
|
|
||||||
|
### Value to Notarius
|
||||||
|
|
||||||
|
This would eliminate a synthetic production-only prompt fixture and establish
|
||||||
|
a direct, supported contract for configuration-time profile and backend
|
||||||
|
validation.
|
||||||
|
|
||||||
|
## Priority 3: Semantic Execution-Target Fingerprints
|
||||||
|
|
||||||
|
**Disposition:** Deferred until prompt-independent profile inspection defines
|
||||||
|
the resolved target whose configuration identity would be fingerprinted.
|
||||||
|
|
||||||
|
### Downstream need
|
||||||
|
|
||||||
|
Notarius checkpoints model-backed pipeline stages. A checkpoint must not be
|
||||||
|
reused when generation-affecting PromptKit configuration changes.
|
||||||
|
|
||||||
|
Notarius therefore needs a stable equality signal for the effective profile
|
||||||
|
and backend target used by a pipeline.
|
||||||
|
|
||||||
|
### Current integration
|
||||||
|
|
||||||
|
Notarius currently constructs this identity itself from:
|
||||||
|
|
||||||
|
- a manually maintained marker for the PromptKit release and built-in profile
|
||||||
|
catalog;
|
||||||
|
- raw hashes of configured profile files; and
|
||||||
|
- a separate hash of the configured conventional local-backend endpoint.
|
||||||
|
|
||||||
|
This is safe but conservative and coupled to PromptKit details. Raw file
|
||||||
|
hashing also invalidates checkpoints for semantically irrelevant YAML changes,
|
||||||
|
such as comments or formatting.
|
||||||
|
|
||||||
|
### Requested capability
|
||||||
|
|
||||||
|
Expose an opaque semantic digest for a resolved profile and its effective
|
||||||
|
generation target. It could be returned by the proposed profile-resolution
|
||||||
|
API:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type ResolvedProfile struct {
|
||||||
|
ProfileID string
|
||||||
|
BackendID string
|
||||||
|
EffectiveTarget ExecutionTarget
|
||||||
|
ExecutionDigest string
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Alternatively, PromptKit could expose a dedicated method such as
|
||||||
|
`ProfileExecutionDigest(profileID)`.
|
||||||
|
|
||||||
|
### Desired equality semantics
|
||||||
|
|
||||||
|
The digest should change when generation-affecting state changes, including:
|
||||||
|
|
||||||
|
- resolved model and endpoint;
|
||||||
|
- backend routing identity;
|
||||||
|
- backend request defaults and extra parameters;
|
||||||
|
- profile generation parameters; and
|
||||||
|
- the semantic identity of any selected built-in profile.
|
||||||
|
|
||||||
|
The digest should not incorporate:
|
||||||
|
|
||||||
|
- credential values;
|
||||||
|
- concurrency or queue capacity;
|
||||||
|
- filesystem source paths;
|
||||||
|
- YAML comments or formatting; or
|
||||||
|
- other settings that affect scheduling or source representation without
|
||||||
|
changing the generation target.
|
||||||
|
|
||||||
|
The credential environment-variable name may need to participate if changing
|
||||||
|
it can select a materially different provider account or target. PromptKit
|
||||||
|
should define this deliberately while continuing to exclude the resolved
|
||||||
|
secret value.
|
||||||
|
|
||||||
|
### Design considerations
|
||||||
|
|
||||||
|
- Treat the digest as an opaque equality value rather than a public encoding
|
||||||
|
of internal structures.
|
||||||
|
- Document which categories of change affect equality.
|
||||||
|
- Include a versioned semantic marker internally so PromptKit can deliberately
|
||||||
|
invalidate old digests when its resolution semantics change.
|
||||||
|
- Prefer a per-profile digest over a digest of every profile known to an
|
||||||
|
engine. Notarius generally knows which profiles a resolved pipeline uses.
|
||||||
|
- Do not require consumers to know PromptKit's built-in catalog version.
|
||||||
|
|
||||||
|
### Value to Notarius
|
||||||
|
|
||||||
|
This would let Notarius remove its PromptKit release marker and raw
|
||||||
|
profile-source fingerprinting, reduce unnecessary checkpoint invalidation, and
|
||||||
|
delegate execution-target equality to the component that owns target
|
||||||
|
resolution.
|
||||||
|
|
||||||
|
## Priority 4: Structured Capacity Errors
|
||||||
|
|
||||||
|
**Disposition:** Accepted into the
|
||||||
|
[future catalog](future.md#structured-capacity-errors).
|
||||||
|
|
||||||
|
### Downstream need
|
||||||
|
|
||||||
|
Notarius translates PromptKit backend-capacity rejection into a
|
||||||
|
provider-neutral application error. When multiple backends are active,
|
||||||
|
operators would benefit from knowing which backend rejected admission without
|
||||||
|
parsing an error string or exposing endpoint details.
|
||||||
|
|
||||||
|
### Current integration
|
||||||
|
|
||||||
|
PromptKit provides the useful `ErrCapacityExceeded` sentinel. Notarius can
|
||||||
|
classify the failure reliably, but it retains only a sanitized diagnostic
|
||||||
|
string as additional context.
|
||||||
|
|
||||||
|
### Requested capability
|
||||||
|
|
||||||
|
Add a typed error that continues to match `ErrCapacityExceeded`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type CapacityError struct {
|
||||||
|
BackendID string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *CapacityError) Is(target error) bool {
|
||||||
|
return target == ErrCapacityExceeded
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The exact implementation may use `Unwrap` or another idiomatic mechanism. The
|
||||||
|
important properties are compatibility with `errors.Is` and discoverability
|
||||||
|
through `errors.As`.
|
||||||
|
|
||||||
|
### Design considerations
|
||||||
|
|
||||||
|
- Include the stable backend ID.
|
||||||
|
- Do not expose the backend endpoint, credential environment, credential
|
||||||
|
value, request content, or other sensitive configuration.
|
||||||
|
- Consider including the configured concurrency and queue limits if they are
|
||||||
|
useful and safe, but backend identity alone provides most of the downstream
|
||||||
|
value.
|
||||||
|
- Add a retry delay only if PromptKit can provide a meaningful value. A full
|
||||||
|
queue does not necessarily imply a reliable `Retry-After` duration.
|
||||||
|
- Keep retry and backoff policy with the consuming application. PromptKit
|
||||||
|
should classify the admission failure rather than silently retry it.
|
||||||
|
|
||||||
|
### Value to Notarius
|
||||||
|
|
||||||
|
This would improve operational diagnostics and future metrics while preserving
|
||||||
|
the provider-neutral error boundary used by Notarius.
|
||||||
|
|
||||||
|
## Capabilities PromptKit Already Provides Well
|
||||||
|
|
||||||
|
The current PromptKit boundary is sufficient for Notarius's implemented
|
||||||
|
behavior. In particular, PromptKit already provides:
|
||||||
|
|
||||||
|
- filesystem, `fs.FS`, and programmatic prompt, profile, and schema sources;
|
||||||
|
- offline preparation without model execution;
|
||||||
|
- structured output and content validation;
|
||||||
|
- direct session propagation;
|
||||||
|
- tri-state per-run reasoning overrides;
|
||||||
|
- selected profile, backend, model, endpoint, effective parameters, hashes, and
|
||||||
|
token-usage provenance;
|
||||||
|
- endpoint-only profiles;
|
||||||
|
- the conventional `local` backend helper;
|
||||||
|
- arbitrary engine-scoped `Backend` registrations;
|
||||||
|
- backend authentication environment names, extra parameters, concurrency
|
||||||
|
limits, and queue-capacity policies;
|
||||||
|
- provider-client and artifact-reader extension interfaces;
|
||||||
|
- context cancellation; and
|
||||||
|
- useful public error sentinels, including profile absence and capacity
|
||||||
|
exhaustion.
|
||||||
|
|
||||||
|
The wishlist does not imply that Notarius needs PromptKit to broaden its core
|
||||||
|
responsibilities. It primarily asks for more direct access to information and
|
||||||
|
operations that PromptKit already computes internally.
|
||||||
|
|
||||||
|
## Responsibilities That Should Remain In Notarius
|
||||||
|
|
||||||
|
The following concerns belong to the downstream application and should not
|
||||||
|
move into PromptKit for the sake of Notarius:
|
||||||
|
|
||||||
|
- pipeline staging, dependencies, and generated references;
|
||||||
|
- application-wide scheduling across providers and backends;
|
||||||
|
- module and validation retry policy;
|
||||||
|
- checkpoints, resume, and recomputation;
|
||||||
|
- durable run artifacts and manifests;
|
||||||
|
- D&D prompts, schemas, extractors, validators, and normalizers;
|
||||||
|
- Notarius configuration-file parsing and precedence;
|
||||||
|
- domain-specific prompt-cache prefix policy; and
|
||||||
|
- application-specific redaction, retention, and debug-bundle policy.
|
||||||
|
|
||||||
|
PromptKit's complete `Backend` API already supports custom IDs, multiple local
|
||||||
|
endpoints, authentication, extra parameters, and explicit queue policies.
|
||||||
|
Whether Notarius exposes those capabilities in its own configuration is an
|
||||||
|
application-policy decision, not an upstream PromptKit gap.
|
||||||
|
|
||||||
|
## Suggested Upstream Sequence
|
||||||
|
|
||||||
|
If the PromptKit team chooses to pursue these ideas, the most useful order for
|
||||||
|
Notarius would be:
|
||||||
|
|
||||||
|
1. Add atomic execution that returns prepared details and the completed result.
|
||||||
|
2. Add prompt-independent profile inspection.
|
||||||
|
3. Add a semantic execution-target digest, preferably as part of profile
|
||||||
|
inspection.
|
||||||
|
4. Add a typed capacity error carrying backend identity.
|
||||||
|
|
||||||
|
The first two address concrete workarounds in current Notarius code. The third
|
||||||
|
would improve checkpoint correctness and reduce coupling. The fourth is
|
||||||
|
operational polish.
|
||||||
280
docs/roadmap/prepared-execution.md
Normal file
280
docs/roadmap/prepared-execution.md
Normal file
@@ -0,0 +1,280 @@
|
|||||||
|
# Executable Preparation Handles
|
||||||
|
|
||||||
|
**Status:** Accepted.
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Allow a consumer to prepare one exact Promptkit execution, inspect and retain
|
||||||
|
its credential-redacted public preparation details, and later execute that
|
||||||
|
already-prepared work without resolving or rendering the request again.
|
||||||
|
|
||||||
|
This provides a supported preflight-before-generation boundary for
|
||||||
|
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-1-executable-preparation-handles)
|
||||||
|
and removes the duplicate `Prepare`-then-`Run` workaround described by
|
||||||
|
[Notarius](notarius-promptkit-wishlist.md#priority-1-atomic-execution-with-prepared-details).
|
||||||
|
|
||||||
|
## Motivation
|
||||||
|
|
||||||
|
`Engine.Prepare` currently returns the provenance and rendered details that
|
||||||
|
consumers need for debugging, persistence, and preflight checks. `Engine.Run`
|
||||||
|
then performs its own preparation before generation. A consumer that needs
|
||||||
|
both values must therefore prepare the same logical request twice.
|
||||||
|
|
||||||
|
That workaround duplicates source loading, input hashing, schema work, and
|
||||||
|
template rendering. It also permits filesystem-backed prompts, profiles,
|
||||||
|
schemas, or inputs to change between the public preparation and the
|
||||||
|
preparation that actually produces the result.
|
||||||
|
|
||||||
|
Promptkit already owns the complete preparation and execution pipeline. An
|
||||||
|
opt-in prepared-execution handle should expose the missing boundary without
|
||||||
|
putting rendered content into every `RunResult` or moving persistence policy
|
||||||
|
into the library.
|
||||||
|
|
||||||
|
## Consumer Workflow
|
||||||
|
|
||||||
|
The target public workflow is:
|
||||||
|
|
||||||
|
```go
|
||||||
|
prepared, err := engine.PrepareExecution(ctx, request)
|
||||||
|
if err != nil {
|
||||||
|
// Handle preparation failure.
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer prepared.Discard()
|
||||||
|
|
||||||
|
details := prepared.Details()
|
||||||
|
// Persist or inspect a consumer-selected safe subset of details.
|
||||||
|
|
||||||
|
result, err := engine.RunPrepared(ctx, prepared)
|
||||||
|
```
|
||||||
|
|
||||||
|
The target public surface is:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type PreparedExecution struct {
|
||||||
|
// Opaque Promptkit-owned state.
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *Engine) PrepareExecution(
|
||||||
|
ctx context.Context,
|
||||||
|
req RunRequest,
|
||||||
|
) (*PreparedExecution, error)
|
||||||
|
|
||||||
|
func (p *PreparedExecution) Details() PreparedRun
|
||||||
|
|
||||||
|
func (p *PreparedExecution) Discard()
|
||||||
|
|
||||||
|
func (e *Engine) RunPrepared(
|
||||||
|
ctx context.Context,
|
||||||
|
prepared *PreparedExecution,
|
||||||
|
) (*RunResult, error)
|
||||||
|
```
|
||||||
|
|
||||||
|
The declarations and GoDoc will own the exact implemented contract. The
|
||||||
|
important public shape is an opaque handle, caller-owned `PreparedRun`
|
||||||
|
details, an explicit discard operation, and execution through the engine that
|
||||||
|
created the handle.
|
||||||
|
|
||||||
|
## Prepared Snapshot
|
||||||
|
|
||||||
|
`PrepareExecution` performs complete preparation without model generation or
|
||||||
|
backend-capacity admission. It applies the same request validation, source
|
||||||
|
precedence, backend and profile resolution, credential requirement checks,
|
||||||
|
output-contract resolution, artifact loading, hashing, schema loading,
|
||||||
|
session resolution, and rendering behavior as `Prepare`.
|
||||||
|
|
||||||
|
A successful handle freezes all source-derived state required for later
|
||||||
|
execution, including:
|
||||||
|
|
||||||
|
- the selected prompt definition, profile, and backend identity;
|
||||||
|
- the complete effective execution target and request-field presence;
|
||||||
|
- rendered messages and effective session ID;
|
||||||
|
- prompt, rendered-prompt, and input hashes;
|
||||||
|
- the effective output contract and provider-facing structured-output
|
||||||
|
constraint; and
|
||||||
|
- private validation state sufficient to validate generated output without
|
||||||
|
reopening schema files or `fs.FS` resources, including required schema
|
||||||
|
references.
|
||||||
|
|
||||||
|
After `PrepareExecution` succeeds, changes to prompt, profile, schema, input,
|
||||||
|
or request-owned data cannot change what `RunPrepared` sends to the model or
|
||||||
|
how it validates the generated output.
|
||||||
|
|
||||||
|
The handle retains an internal snapshot independent from values returned by
|
||||||
|
`Details`. Mutating a returned `PreparedRun`, its maps, slices, messages, or
|
||||||
|
schema values does not affect later execution. Each `Details` call returns a
|
||||||
|
fresh caller-owned copy under the existing `PreparedRun` ownership and stable
|
||||||
|
JSON rules.
|
||||||
|
|
||||||
|
## Handle Lifecycle
|
||||||
|
|
||||||
|
A `PreparedExecution` is:
|
||||||
|
|
||||||
|
- created only by a successful `PrepareExecution` call;
|
||||||
|
- bound to the exact `Engine` that created it;
|
||||||
|
- valid for one `RunPrepared` invocation;
|
||||||
|
- safe for repeated `Details` calls;
|
||||||
|
- intentionally opaque and without a supported JSON representation; and
|
||||||
|
- in-process state rather than a durable or restartable job.
|
||||||
|
|
||||||
|
`RunPrepared` atomically claims a valid handle before beginning the execution
|
||||||
|
attempt. A second or concurrent invocation fails without starting another
|
||||||
|
execution, including when the first invocation ended in cancellation, capacity
|
||||||
|
rejection, generation failure, or operational validation failure. Copies of
|
||||||
|
the public handle share the same one-attempt state and cannot bypass this rule.
|
||||||
|
|
||||||
|
A nil, zero-value, foreign-engine, discarded, already-claimed, or already-used
|
||||||
|
handle is invalid. `RunPrepared` reports these lifecycle errors through the
|
||||||
|
ordinary public invalid-request category. A failed foreign-engine invocation
|
||||||
|
does not consume a handle that remains valid for its owning engine.
|
||||||
|
|
||||||
|
`Discard` idempotently makes an unclaimed handle unavailable for execution and
|
||||||
|
drops Promptkit's references to secret-bearing or execution-only state.
|
||||||
|
`RunPrepared` performs the same cleanup automatically after claiming a handle.
|
||||||
|
Credential-redacted public preparation details remain available after discard,
|
||||||
|
success, or failure so consumers can retain diagnostics. Promptkit does not
|
||||||
|
promise secure erasure of Go string memory.
|
||||||
|
|
||||||
|
Lifecycle transitions are concurrency-safe. When `RunPrepared` and `Discard`
|
||||||
|
race, exactly one claims the ready handle. `Discard` is not an execution
|
||||||
|
cancellation mechanism and does not interrupt an attempt that has already
|
||||||
|
claimed the handle; consumers cancel that attempt through its context.
|
||||||
|
|
||||||
|
## Credentials And Sensitive Data
|
||||||
|
|
||||||
|
`Details` has the same security contract as `PreparedRun`: it can contain
|
||||||
|
rendered messages, schemas, identifiers, and hashes, but never a resolved API
|
||||||
|
key value. Consumers remain responsible for selecting, redacting, storing, and
|
||||||
|
retaining any persisted preparation material.
|
||||||
|
|
||||||
|
A direct `RunRequest.APIKey` is retained only in opaque execution state until
|
||||||
|
the handle is run or discarded. It is never added to details, hashes, JSON,
|
||||||
|
`String`, or `GoString` output.
|
||||||
|
|
||||||
|
An environment-variable name is frozen as part of the effective target, but
|
||||||
|
its credential value is not captured for the lifetime of the handle.
|
||||||
|
`PrepareExecution` applies the existing preparation-time availability check.
|
||||||
|
`RunPrepared` rechecks availability before admission, and the selected model
|
||||||
|
client uses the environment value visible during execution. This preserves
|
||||||
|
current secret ownership and avoids retaining an environment credential while
|
||||||
|
a consumer persists preflight material.
|
||||||
|
|
||||||
|
The opaque handle must not expose retained request data or credentials through
|
||||||
|
default formatting, JSON, or error messages.
|
||||||
|
|
||||||
|
## Admission, Cancellation, And Execution
|
||||||
|
|
||||||
|
`PrepareExecution` never reserves backend admission or an active-generation
|
||||||
|
permit. Its context governs preparation only; cancellation after it returns
|
||||||
|
does not invalidate the handle.
|
||||||
|
|
||||||
|
`RunPrepared` uses its own context for credential revalidation, backend
|
||||||
|
admission, active-generation waiting, model generation, output validation, and
|
||||||
|
any internal repair. Admission occurs when `RunPrepared` begins so a consumer
|
||||||
|
cannot occupy bounded capacity while inspecting or persisting preparation
|
||||||
|
details.
|
||||||
|
|
||||||
|
For a limited backend, the admission lease covers the complete prepared
|
||||||
|
execution attempt after admission: generation, validation, internal repair,
|
||||||
|
and every success or failure exit. Actual generation continues to use the
|
||||||
|
backend's FIFO active-generation permit. Existing capacity error identity,
|
||||||
|
cancellation behavior, and release guarantees remain in force.
|
||||||
|
|
||||||
|
Because a prepared handle is one-attempt, cancellation or capacity rejection
|
||||||
|
does not make it reusable. Retry and backoff policy remains with the consumer,
|
||||||
|
which may create a new prepared handle when another attempt is appropriate.
|
||||||
|
|
||||||
|
## Results And Failures
|
||||||
|
|
||||||
|
On success, `RunPrepared` returns the existing caller-owned `RunResult`. Its
|
||||||
|
source-derived provenance must match `Details`, including prompt identity and
|
||||||
|
hash, rendered-prompt hash, session ID, selected profile and backend,
|
||||||
|
effective target, and input hashes.
|
||||||
|
|
||||||
|
`RunResult.StartTime`, `EndTime`, and `Duration` describe the
|
||||||
|
`RunPrepared` execution attempt. They exclude preparation time and any delay
|
||||||
|
while the consumer retained the handle. Preparation timing remains in
|
||||||
|
`PreparedRun`.
|
||||||
|
|
||||||
|
Preparation failure returns no handle. After successful preparation,
|
||||||
|
`RunPrepared` retains the existing rule that an operational failure returns no
|
||||||
|
partial `RunResult`; the consumer already has independent preparation details.
|
||||||
|
A completed content-validation failure remains a successful result with
|
||||||
|
`ValidationFailed`.
|
||||||
|
|
||||||
|
`PrepareExecution` preserves the public error categories of `Prepare`.
|
||||||
|
`RunPrepared` preserves applicable invalid-request, credential, capacity,
|
||||||
|
generation, validation, collaborator, and cancellation identities without
|
||||||
|
reintroducing source-loading or rendering failures from frozen state.
|
||||||
|
|
||||||
|
## Compatibility And Existing Workflows
|
||||||
|
|
||||||
|
This feature is additive:
|
||||||
|
|
||||||
|
- `Prepare` remains the simple preparation-only operation;
|
||||||
|
- `Run` remains the simple prepare-and-execute operation with its current
|
||||||
|
early-admission and error-ordering behavior;
|
||||||
|
- `PreparedRun` and `RunResult` retain their existing stable JSON
|
||||||
|
representations;
|
||||||
|
- model-client and artifact-reader extension interfaces remain unchanged; and
|
||||||
|
- backend routing, concurrency limits, queue capacities, and provider wire
|
||||||
|
behavior remain unchanged.
|
||||||
|
|
||||||
|
The new workflow may share internal machinery with `Prepare` and `Run`, but it
|
||||||
|
must not change their observable behavior merely to simplify implementation.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
The completed documentation set has these ownership boundaries:
|
||||||
|
|
||||||
|
- exported declarations and GoDoc own the exact handle, method, lifecycle,
|
||||||
|
ownership, concurrency, credential, error, and cancellation contracts;
|
||||||
|
- the promptkit consumer guide explains when to use `Prepare`, `Run`, or the
|
||||||
|
two-phase prepared-execution workflow; and
|
||||||
|
- internal runner, source-validation, capacity, and model-client documentation
|
||||||
|
describe the implemented collaborator boundaries without duplicating public
|
||||||
|
contracts.
|
||||||
|
|
||||||
|
No release document is part of the feature implementation itself. Release
|
||||||
|
guidance is prepared only when the resulting public API is selected for
|
||||||
|
publication.
|
||||||
|
|
||||||
|
## Non-Goals
|
||||||
|
|
||||||
|
This work does not include:
|
||||||
|
|
||||||
|
- serializable, durable, resumable, or cross-process execution handles;
|
||||||
|
- reuse for multiple consumer-initiated executions;
|
||||||
|
- concurrent execution of one handle;
|
||||||
|
- capacity reservation during preparation;
|
||||||
|
- a background task queue, priorities, worker lifecycle, or job status;
|
||||||
|
- retry, backoff, or provider failover policy;
|
||||||
|
- a separate `RunDetailed` convenience method;
|
||||||
|
- adding preparation details or rendered messages to every `RunResult`;
|
||||||
|
- prompt or profile inspection APIs;
|
||||||
|
- structured capacity or generation errors;
|
||||||
|
- freezing environment-variable credential values for the handle lifetime;
|
||||||
|
- snapshotting provider state or mutable behavior inside an injected
|
||||||
|
`LLMClient`;
|
||||||
|
- allowing target, validation, session, variable, or input overrides after
|
||||||
|
preparation; or
|
||||||
|
- changing existing `Prepare`, `Run`, file-format, provider-request, or stable
|
||||||
|
JSON contracts.
|
||||||
|
|
||||||
|
## Target End State
|
||||||
|
|
||||||
|
After this work:
|
||||||
|
|
||||||
|
- consumers can perform and persist preflight before starting provider work;
|
||||||
|
- one prepared handle executes exactly the source-derived prompt, target,
|
||||||
|
schema, inputs, session, and messages described by its public details;
|
||||||
|
- execution never reloads or rerenders consumer sources;
|
||||||
|
- direct credentials remain confined to opaque, explicitly discardable state;
|
||||||
|
- environment credentials are not retained across the preflight boundary;
|
||||||
|
- backend capacity is reserved only when execution begins;
|
||||||
|
- one handle can start at most one execution attempt, including any internal
|
||||||
|
repair calls owned by that attempt;
|
||||||
|
- preparation details remain available after execution success or failure;
|
||||||
|
- existing simple `Prepare` and `Run` consumers remain unaffected; and
|
||||||
|
- Promptkit continues to own reusable execution mechanics without taking on
|
||||||
|
downstream persistence, redaction, retry, or job-management policy.
|
||||||
468
docs/roadmap/weatherreporter-promptkit-wishlist.md
Normal file
468
docs/roadmap/weatherreporter-promptkit-wishlist.md
Normal file
@@ -0,0 +1,468 @@
|
|||||||
|
# Weatherreporter PromptKit Wishlist
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document records features and interface changes that would be useful
|
||||||
|
additions to PromptKit from the perspective of the maintainers of
|
||||||
|
Weatherreporter, a downstream application planning to replace its Scriptorium
|
||||||
|
CLI integration with PromptKit.
|
||||||
|
|
||||||
|
PromptKit v0.3.0 provides the capabilities Weatherreporter needs for the
|
||||||
|
migration. None of the ideas below is a hard adoption requirement. They are
|
||||||
|
opportunities to avoid duplicate preparation, validate configuration earlier,
|
||||||
|
improve durable failure diagnostics, and make the integration more direct.
|
||||||
|
|
||||||
|
The examples are API sketches intended to communicate the desired capability,
|
||||||
|
not prescriptive names or finalized Go contracts. The related
|
||||||
|
[Notarius PromptKit wishlist](notarius-promptkit-wishlist.md) proposes several
|
||||||
|
overlapping features from another downstream consumer's perspective.
|
||||||
|
|
||||||
|
## Priority 1: Executable Preparation Handles
|
||||||
|
|
||||||
|
**Disposition:** Accepted into the
|
||||||
|
[executable preparation handles](prepared-execution.md) feature roadmap.
|
||||||
|
|
||||||
|
### Downstream need
|
||||||
|
|
||||||
|
Weatherreporter treats prompt preparation as a durable preflight boundary. It
|
||||||
|
needs to:
|
||||||
|
|
||||||
|
1. prepare the exact request that will be executed;
|
||||||
|
2. persist a safe preparation record before starting the provider call; and
|
||||||
|
3. execute without reloading or rerendering prompt, profile, schema, or input
|
||||||
|
sources.
|
||||||
|
|
||||||
|
Persisting preflight before generation leaves useful evidence when a provider
|
||||||
|
call fails or the process is interrupted during generation.
|
||||||
|
|
||||||
|
### Current integration option
|
||||||
|
|
||||||
|
With PromptKit v0.3.0, Weatherreporter can call `Engine.Prepare`, save selected
|
||||||
|
fields from the returned `PreparedRun`, and then call `Engine.Run` with the
|
||||||
|
same request. Because `Run` performs preparation internally, the work is
|
||||||
|
repeated.
|
||||||
|
|
||||||
|
Weatherreporter plans to use embedded prompt and schema files plus immutable
|
||||||
|
inline input bytes, which removes most of the consistency risk. An external
|
||||||
|
profile file or directory can still change between the two calls, and the
|
||||||
|
second preparation remains unnecessary work.
|
||||||
|
|
||||||
|
The atomic `RunDetailed` operation proposed by the
|
||||||
|
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-1-atomic-execution-with-prepared-details)
|
||||||
|
would guarantee that returned preparation details describe the completed
|
||||||
|
execution. However, returning those details only after generation would not
|
||||||
|
preserve Weatherreporter's preflight-before-generation persistence boundary.
|
||||||
|
|
||||||
|
### Requested capability
|
||||||
|
|
||||||
|
Add an opt-in two-phase API that returns a prepared execution handle:
|
||||||
|
|
||||||
|
```go
|
||||||
|
prepared, err := engine.PrepareExecution(ctx, request)
|
||||||
|
if err != nil {
|
||||||
|
// Handle preparation failure.
|
||||||
|
}
|
||||||
|
|
||||||
|
details := prepared.Details()
|
||||||
|
// Persist a consumer-selected safe preparation record.
|
||||||
|
|
||||||
|
result, err := engine.RunPrepared(ctx, prepared)
|
||||||
|
```
|
||||||
|
|
||||||
|
The exact names and shapes are flexible. The important contract is that
|
||||||
|
`RunPrepared` executes the already prepared prompt and does not reload or
|
||||||
|
rerender its prompt, profile, schema, or input sources.
|
||||||
|
|
||||||
|
`Details` should return the same caller-owned public preparation information
|
||||||
|
currently represented by `PreparedRun`. The execution handle may retain opaque
|
||||||
|
engine-owned state needed to invoke the model and validate the response.
|
||||||
|
|
||||||
|
### Design considerations
|
||||||
|
|
||||||
|
- Keep `Prepare` and `Run` available for consumers that do not need a
|
||||||
|
two-phase execution boundary.
|
||||||
|
- Bind a prepared handle to the engine that constructed it.
|
||||||
|
- Define whether a handle is one-shot, reusable, or safe for concurrent use.
|
||||||
|
A one-shot contract may be the safest initial design.
|
||||||
|
- Do not give the opaque handle a stable JSON representation.
|
||||||
|
- Do not expose or serialize resolved credential values through `Details`.
|
||||||
|
- Define how a direct request API key is retained and released when an opaque
|
||||||
|
handle must carry it until execution.
|
||||||
|
- Preserve caller-owned copies for all public details.
|
||||||
|
- Make context cancellation and backend admission timing explicit.
|
||||||
|
- Document whether profile credential environment values are resolved during
|
||||||
|
preparation or execution.
|
||||||
|
- Ensure an execution error does not invalidate the public details already
|
||||||
|
returned to the consumer.
|
||||||
|
- Consider whether an atomic `RunDetailed` can share the same internal
|
||||||
|
prepared-execution implementation.
|
||||||
|
|
||||||
|
### Value to Weatherreporter
|
||||||
|
|
||||||
|
This is the highest-value upstream addition. It would preserve
|
||||||
|
Weatherreporter's durable preflight behavior, remove duplicate work, eliminate
|
||||||
|
the remaining source-consistency window, and ensure that persisted provenance
|
||||||
|
describes the actual execution.
|
||||||
|
|
||||||
|
## Priority 2: Prompt-Definition Inspection
|
||||||
|
|
||||||
|
**Disposition:** Accepted into the
|
||||||
|
[future catalog](future.md#prompt-definition-inspection).
|
||||||
|
|
||||||
|
### Downstream need
|
||||||
|
|
||||||
|
Weatherreporter has a fixed registry of seven report definitions. Each report
|
||||||
|
selects a prompt ID and one of two output workflows:
|
||||||
|
|
||||||
|
- direct Markdown; or
|
||||||
|
- structured generated text followed by application-owned domain validation
|
||||||
|
and Markdown template rendering.
|
||||||
|
|
||||||
|
Weatherreporter will embed the PromptKit prompt definitions and private
|
||||||
|
response schemas that implement those reports. It needs to validate that the
|
||||||
|
report registry and embedded prompt corpus agree before weather collection or
|
||||||
|
provider execution.
|
||||||
|
|
||||||
|
### Current integration option
|
||||||
|
|
||||||
|
Weatherreporter can maintain synthetic data-package fixtures and call
|
||||||
|
`Engine.Prepare` for every report prompt during tests. Runtime validation can
|
||||||
|
also occur through the ordinary per-report preparation stage.
|
||||||
|
|
||||||
|
This works, but it requires complete placeholder inputs and profile resolution
|
||||||
|
when the application primarily wants to inspect prompt identity and declared
|
||||||
|
contracts.
|
||||||
|
|
||||||
|
### Requested capability
|
||||||
|
|
||||||
|
Add exact prompt-definition lookup without rendering or generation:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type PromptInfo struct {
|
||||||
|
PromptID string
|
||||||
|
PromptVersion string
|
||||||
|
PromptHash string
|
||||||
|
DefaultProfileID string
|
||||||
|
Inputs []InputDefinition
|
||||||
|
OutputContract OutputContract
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *Engine) ResolvePrompt(
|
||||||
|
ctx context.Context,
|
||||||
|
promptID string,
|
||||||
|
promptVersion string,
|
||||||
|
) (PromptInfo, error)
|
||||||
|
```
|
||||||
|
|
||||||
|
The exact returned shape may differ. Weatherreporter needs enough information
|
||||||
|
to verify prompt existence, version selection, declared inputs, default
|
||||||
|
profile identity, output format, validation mode, and schema selection without
|
||||||
|
supplying synthetic prompt input.
|
||||||
|
|
||||||
|
### Design considerations
|
||||||
|
|
||||||
|
- Use ordinary PromptKit prompt-source precedence and exact ID/version
|
||||||
|
selection.
|
||||||
|
- Fully load and structurally validate the selected prompt definition.
|
||||||
|
- Validate referenced prompt content files without rendering their templates.
|
||||||
|
- Resolve and validate the selected output contract and schema reference where
|
||||||
|
practical.
|
||||||
|
- Return an opaque prompt-definition equality value rather than raw source
|
||||||
|
bytes.
|
||||||
|
- Do not return rendered messages, schema bodies, profile credentials, or
|
||||||
|
another source of sensitive content.
|
||||||
|
- Preserve typed or sentinel errors for missing and invalid prompts.
|
||||||
|
- Return caller-owned values.
|
||||||
|
- Enumeration of all known prompts is not required for Weatherreporter; exact
|
||||||
|
lookup is sufficient.
|
||||||
|
|
||||||
|
### Value to Weatherreporter
|
||||||
|
|
||||||
|
This would let Weatherreporter directly verify that every report prompt
|
||||||
|
exists, requires the curated `data_package` input, and declares the expected
|
||||||
|
Markdown or JSON Schema output contract. It would reduce synthetic test setup
|
||||||
|
and move failures ahead of weather collection.
|
||||||
|
|
||||||
|
## Priority 3: Prompt-Independent Profile Inspection
|
||||||
|
|
||||||
|
**Disposition:** Accepted into the
|
||||||
|
[future catalog](future.md#prompt-independent-profile-inspection).
|
||||||
|
|
||||||
|
### Downstream need
|
||||||
|
|
||||||
|
Weatherreporter will allow operators to select an external PromptKit profile
|
||||||
|
source and may allow an explicit profile override. It should reject a missing
|
||||||
|
profile, unknown backend, malformed execution target, or unsatisfied credential
|
||||||
|
requirement before collecting weather data or writing report artifacts.
|
||||||
|
|
||||||
|
### Current integration option
|
||||||
|
|
||||||
|
Weatherreporter can validate an explicit profile by preparing one embedded
|
||||||
|
prompt with fixture input. Prompts that use their own default profiles can be
|
||||||
|
validated during their normal preparation stage.
|
||||||
|
|
||||||
|
This couples configuration validation to one prompt and requires placeholder
|
||||||
|
input even when only profile and backend resolution are relevant.
|
||||||
|
|
||||||
|
### Requested capability
|
||||||
|
|
||||||
|
The prompt-independent `ResolveProfile` API proposed by the
|
||||||
|
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
|
||||||
|
would satisfy this need. It should resolve built-in, file-backed, and
|
||||||
|
programmatic profiles, validate backend membership, report credential
|
||||||
|
requirements without resolving credential values, and preserve typed error
|
||||||
|
classification.
|
||||||
|
|
||||||
|
### Additional Weatherreporter considerations
|
||||||
|
|
||||||
|
- An explicit application profile override should be inspectable without
|
||||||
|
selecting a report prompt.
|
||||||
|
- A prompt-definition inspection result may expose its default profile ID so
|
||||||
|
Weatherreporter can inspect that profile separately.
|
||||||
|
- Inspection should distinguish structural profile validity from current
|
||||||
|
credential availability so configuration validation can apply explicit
|
||||||
|
application policy.
|
||||||
|
- An optional execution-target override should be considered only if it
|
||||||
|
describes the same target that a later run will use.
|
||||||
|
|
||||||
|
### Value to Weatherreporter
|
||||||
|
|
||||||
|
This would improve fail-fast configuration validation and give operator-facing
|
||||||
|
errors direct profile and backend context. It is valuable but not required for
|
||||||
|
the initial migration.
|
||||||
|
|
||||||
|
## Priority 4: Eager Source Validation
|
||||||
|
|
||||||
|
**Disposition:** Deferred until prompt and profile inspection have been used
|
||||||
|
to determine whether a broader engine-wide validation operation is still
|
||||||
|
needed.
|
||||||
|
|
||||||
|
### Downstream need
|
||||||
|
|
||||||
|
PromptKit deliberately defers reading and validating filesystem and `fs.FS`
|
||||||
|
prompt, profile, and schema content until a request needs it. Weatherreporter
|
||||||
|
has a small fixed embedded prompt corpus and one optional external profile
|
||||||
|
source. It would benefit from an explicit offline validation operation for
|
||||||
|
tests, startup diagnostics, and configuration checks.
|
||||||
|
|
||||||
|
### Current integration option
|
||||||
|
|
||||||
|
Weatherreporter can prepare every report prompt with fixture inputs and inspect
|
||||||
|
any explicit profiles individually. That provides strong coverage but requires
|
||||||
|
consumer-maintained traversal and synthetic material.
|
||||||
|
|
||||||
|
### Requested capability
|
||||||
|
|
||||||
|
Consider an opt-in source-validation operation:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type SourceValidationOptions struct {
|
||||||
|
RequireCredentials bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *Engine) ValidateSources(
|
||||||
|
ctx context.Context,
|
||||||
|
opts SourceValidationOptions,
|
||||||
|
) error
|
||||||
|
```
|
||||||
|
|
||||||
|
The operation should eagerly discover and structurally validate the configured
|
||||||
|
prompt, profile, and schema sources without model generation.
|
||||||
|
|
||||||
|
### Design considerations
|
||||||
|
|
||||||
|
- Keep deferred validation as the normal `NewEngine` behavior.
|
||||||
|
- Make eager validation an explicit consumer choice.
|
||||||
|
- Validate duplicate IDs and versions, strict YAML decoding, referenced content
|
||||||
|
files, profile/backend membership, schema syntax, and schema references.
|
||||||
|
- Distinguish structural credential declarations from current environment
|
||||||
|
availability.
|
||||||
|
- Do not read or expose credential values when credential availability is not
|
||||||
|
requested.
|
||||||
|
- Preserve source-specific public error identities and useful path context.
|
||||||
|
- Respect context cancellation during filesystem discovery and schema work.
|
||||||
|
- Consider whether exact prompt and profile inspection APIs already provide a
|
||||||
|
smaller sufficient surface before adding an engine-wide operation.
|
||||||
|
|
||||||
|
### Value to Weatherreporter
|
||||||
|
|
||||||
|
This would simplify offline corpus checks and catch malformed operator profile
|
||||||
|
sources before report work begins. It is helpful but lower priority than exact
|
||||||
|
prompt and profile inspection.
|
||||||
|
|
||||||
|
## Priority 5: Structured Generation Errors
|
||||||
|
|
||||||
|
**Disposition:** Deferred pending stronger downstream demand and a narrower
|
||||||
|
design that does not duplicate prepared provenance or impose HTTP-specific
|
||||||
|
fields on injected model clients.
|
||||||
|
|
||||||
|
### Downstream need
|
||||||
|
|
||||||
|
Weatherreporter preserves redacted, inspectable failure receipts for report
|
||||||
|
runs. When model generation fails operationally, it needs to classify the
|
||||||
|
failure and retain safe execution context without parsing error prose.
|
||||||
|
|
||||||
|
Prompt preparation already supplies selected profile, backend, and model
|
||||||
|
identity. Provider status classification would add useful operator context,
|
||||||
|
especially when the built-in OpenAI-compatible client receives a non-success
|
||||||
|
HTTP status.
|
||||||
|
|
||||||
|
### Current integration option
|
||||||
|
|
||||||
|
PromptKit exposes `ErrLLMGenerate` and preserves injected client errors through
|
||||||
|
`errors.Is`. Weatherreporter can reliably classify generation failure and use
|
||||||
|
its preparation record for profile, backend, and model provenance. Any further
|
||||||
|
diagnostic detail remains a redacted error string.
|
||||||
|
|
||||||
|
### Requested capability
|
||||||
|
|
||||||
|
Consider a typed generation error that continues to match `ErrLLMGenerate`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type GenerationError struct {
|
||||||
|
BackendID string
|
||||||
|
Model string
|
||||||
|
StatusCode int
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The exact fields may differ. The useful contract is safe structured context
|
||||||
|
available through `errors.As`, while `errors.Is(err, ErrLLMGenerate)` remains
|
||||||
|
compatible.
|
||||||
|
|
||||||
|
### Design considerations
|
||||||
|
|
||||||
|
- Include only fields that PromptKit knows reliably and can expose safely.
|
||||||
|
- Treat an HTTP status as optional because injected model clients may not use
|
||||||
|
HTTP.
|
||||||
|
- Do not expose provider response bodies, endpoints, credential environment
|
||||||
|
names, credential values, request content, or generated content.
|
||||||
|
- Do not make a structured error a second source of prompt/profile provenance
|
||||||
|
already present in a prepared execution.
|
||||||
|
- Preserve injected client error identity.
|
||||||
|
- Keep retry and backoff policy with the consuming application.
|
||||||
|
|
||||||
|
### Value to Weatherreporter
|
||||||
|
|
||||||
|
This would improve durable failure receipts and troubleshooting, particularly
|
||||||
|
for built-in transport failures. It is not required if preparation details and
|
||||||
|
the existing sentinel remain available.
|
||||||
|
|
||||||
|
## Lower-Priority Shared Wishlist Items
|
||||||
|
|
||||||
|
### Structured Capacity Errors
|
||||||
|
|
||||||
|
**Disposition:** Accepted into the
|
||||||
|
[future catalog](future.md#structured-capacity-errors).
|
||||||
|
|
||||||
|
The typed capacity error proposed by the
|
||||||
|
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-4-structured-capacity-errors)
|
||||||
|
would improve Weatherreporter diagnostics by exposing the stable backend ID
|
||||||
|
without parsing error text.
|
||||||
|
|
||||||
|
Weatherreporter currently generates batch reports sequentially and constructs
|
||||||
|
one engine per invocation, so engine-local capacity exhaustion is unlikely in
|
||||||
|
the initial design. The feature would become more valuable if report
|
||||||
|
generation later becomes concurrent or PromptKit engines become longer-lived.
|
||||||
|
It should not block adoption.
|
||||||
|
|
||||||
|
### Semantic Execution-Target Fingerprints
|
||||||
|
|
||||||
|
**Disposition:** Deferred until prompt-independent profile inspection defines
|
||||||
|
the resolved target whose configuration identity would be fingerprinted.
|
||||||
|
|
||||||
|
The semantic target digest proposed by the
|
||||||
|
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-3-semantic-execution-target-fingerprints)
|
||||||
|
would provide a compact equality signal for audit metadata.
|
||||||
|
|
||||||
|
Weatherreporter does not currently reuse LLM-dependent checkpoints. Its Recent
|
||||||
|
Changes behavior compares deterministic module snapshots rather than generated
|
||||||
|
reports, so the digest has no immediate cache-correctness role. Existing
|
||||||
|
PromptKit result metadata is sufficient for the initial integration. A digest
|
||||||
|
would still be useful provenance and future-proofing, but it is not a
|
||||||
|
migration priority.
|
||||||
|
|
||||||
|
## Capabilities PromptKit Already Provides Well
|
||||||
|
|
||||||
|
PromptKit v0.3.0 already provides the essential Weatherreporter integration
|
||||||
|
surface:
|
||||||
|
|
||||||
|
- importable in-process engine construction;
|
||||||
|
- filesystem, `fs.FS`, and programmatic prompt, profile, and schema sources;
|
||||||
|
- offline preparation without model execution;
|
||||||
|
- versioned prompt selection;
|
||||||
|
- text, Markdown, JSON, and JSON Schema output contracts;
|
||||||
|
- single-pass output validation with raw output retained after completed
|
||||||
|
validation failure;
|
||||||
|
- inline input artifacts with provenance URIs and input hashes;
|
||||||
|
- selected profile, backend, model, effective target, prompt hashes, timing,
|
||||||
|
and token-usage provenance;
|
||||||
|
- endpoint-only profiles and engine-scoped backend registration;
|
||||||
|
- injected model-client and artifact-reader interfaces;
|
||||||
|
- caller cancellation, generation timeout, and transport timeout behavior; and
|
||||||
|
- public error sentinels for configuration, prompt, profile, artifact,
|
||||||
|
validation, capacity, and generation failures.
|
||||||
|
|
||||||
|
These capabilities are sufficient for Weatherreporter to adopt PromptKit
|
||||||
|
without waiting for new upstream work.
|
||||||
|
|
||||||
|
## Responsibilities That Should Remain In Weatherreporter
|
||||||
|
|
||||||
|
The following concerns belong to Weatherreporter and should not move into
|
||||||
|
PromptKit:
|
||||||
|
|
||||||
|
- report definitions, valid periods, batches, and output naming;
|
||||||
|
- application prompt content and private report response schemas;
|
||||||
|
- deterministic weather facts, modules, and Recent Changes;
|
||||||
|
- curated `data_package` construction and persistence;
|
||||||
|
- generated-text domain validation and Markdown template rendering;
|
||||||
|
- managed artifact paths, atomic writes, metadata, and inspection commands;
|
||||||
|
- preparation, execution, raw-output, and failure-receipt schemas;
|
||||||
|
- CLI configuration loading and precedence;
|
||||||
|
- debug enablement, redaction, placement, sensitivity, and retention;
|
||||||
|
- distributor notification;
|
||||||
|
- batch continuation and any future retry policy; and
|
||||||
|
- application-level compatibility and migration policy.
|
||||||
|
|
||||||
|
## Suggested Upstream Sequence
|
||||||
|
|
||||||
|
If the PromptKit team chooses to pursue these ideas, the most useful order for
|
||||||
|
Weatherreporter would be:
|
||||||
|
|
||||||
|
1. Add executable preparation handles, ideally sharing implementation with an
|
||||||
|
atomic detailed-run API.
|
||||||
|
2. Add prompt-definition inspection.
|
||||||
|
3. Add prompt-independent profile inspection.
|
||||||
|
4. Consider eager source validation after evaluating whether the two exact
|
||||||
|
inspection APIs are sufficient.
|
||||||
|
5. Add structured generation errors.
|
||||||
|
6. Add structured capacity errors and semantic execution-target fingerprints
|
||||||
|
as lower-priority operational improvements.
|
||||||
|
|
||||||
|
The first item removes the only material integration workaround. Prompt and
|
||||||
|
profile inspection improve fail-fast validation. The remaining items improve
|
||||||
|
ergonomics and diagnostics.
|
||||||
|
|
||||||
|
## Adoption Sequencing
|
||||||
|
|
||||||
|
Weatherreporter should not wait for the complete wishlist. PromptKit v0.3.0 is
|
||||||
|
already sufficient when Weatherreporter:
|
||||||
|
|
||||||
|
- embeds immutable prompt and schema assets;
|
||||||
|
- supplies immutable inline data-package bytes;
|
||||||
|
- constructs one engine per CLI invocation;
|
||||||
|
- calls `Prepare` and `Run` with the same request; and
|
||||||
|
- keeps PromptKit behind a weatherreporter-owned adapter contract.
|
||||||
|
|
||||||
|
If executable preparation handles are scheduled for a near-term PromptKit
|
||||||
|
release, Weatherreporter may defer only its final adapter implementation to
|
||||||
|
avoid implementing and then removing duplicate preparation. Prompt corpus
|
||||||
|
retrieval, application-contract design, configuration work, embedded assets,
|
||||||
|
state contracts, and offline fixtures can proceed independently.
|
||||||
|
|
||||||
|
If the feature is not scheduled, Weatherreporter can adopt v0.3.0 and keep the
|
||||||
|
duplicate `Prepare` and `Run` sequence inside its adapter. A later PromptKit
|
||||||
|
upgrade would remain localized behind that neutral boundary.
|
||||||
|
|
||||||
|
Prompt inspection, profile inspection, source validation, structured errors,
|
||||||
|
capacity details, and semantic fingerprints should not gate adoption.
|
||||||
Reference in New Issue
Block a user