3 Commits

5 changed files with 1761 additions and 276 deletions

View File

@@ -33,9 +33,52 @@ consumers.
## Ideas ## Ideas
No ideas are currently cataloged. Backend-specific concurrency management has Executable preparation handles have been selected for active planning in the
been selected for active planning in the [focused feature roadmap](prepared-execution.md). The remaining ideas are
[focused concurrency roadmap](concurrency.md). still available for future selection.
### Prompt-independent profile inspection
Provide exact profile lookup and structural resolution without requiring a
synthetic prompt, placeholder inputs, or model generation. This shared need is
described by
[Notarius](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
and
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-3-prompt-independent-profile-inspection).
- Apply ordinary built-in, file-backed, and programmatic profile precedence.
- Validate referenced backend membership and the structurally resolved
execution target.
- Report credential requirements and environment-variable names without
exposing credential values or requiring current credential availability.
- Support exact lookup by profile ID; enumeration is not required initially.
### Prompt-definition inspection
Provide exact prompt-definition lookup without rendering, placeholder inputs,
profile resolution, or model generation, as requested by
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-2-prompt-definition-inspection).
- Return caller-owned identity, version, input definitions, default-profile,
output-contract, and opaque definition-equality information.
- Apply ordinary prompt-source precedence and exact ID/version selection.
- Validate the selected definition and referenced prompt content
structurally, without returning source bodies or rendered messages.
- Leave complete cross-source corpus validation and enumeration outside the
initial inspection contract.
### Structured capacity errors
Add safe structured context to backend admission rejection, as requested by
[Notarius](notarius-promptkit-wishlist.md#priority-4-structured-capacity-errors)
and
[Weatherreporter](weatherreporter-promptkit-wishlist.md#structured-capacity-errors).
- Preserve compatibility with `errors.Is(err, ErrCapacityExceeded)`.
- Support `errors.As` to obtain the stable backend ID.
- Do not expose endpoints, credential configuration or values, request
content, or speculative retry timing.
- Keep retry and backoff policy with downstream consumers.
## Entry Format ## Entry Format

View File

@@ -1,315 +1,683 @@
# Local Backend Convenience Implementation Plan # Executable Preparation Handles Implementation Plan
**Status:** Complete. **Status:** Accepted.
## Purpose ## Purpose
This document is the decision-complete implementation plan for This document is the decision-complete implementation plan for
[local backend convenience](local-backend.md). It is written for a coding [executable preparation handles](prepared-execution.md). It is written for a
agent that will implement each stage in order. coding agent that will implement each stage in order.
The feature roadmap owns the motivation, consumer paths, policy choices, The feature roadmap owns the motivation, consumer workflow, policy choices,
compatibility requirements, non-goals, and target end state. This document compatibility requirements, non-goals, and target end state. This document
owns the exact proposed API, file-level changes, implementation sequence, test owns the concrete design, file-level changes, implementation sequence, test
ownership, documentation work, validation commands, and completion gates. ownership, documentation work, validation commands, and completion gates.
## Implementation Rules ## Implementation Rules
- Complete the stages in order. Keep the root package compiling and its - Complete the stages in order. Keep the repository compiling and the focused
focused tests passing at every stage boundary. tests passing at every stage boundary.
- Preserve unrelated working-tree changes. The feature roadmap may already be - Preserve unrelated working-tree changes. In particular, retain the accepted
uncommitted when implementation begins; retain it. roadmap and the downstream wishlist and future-roadmap edits that may
already be present.
- Follow every policy under `docs/policy/`, the task-specific reading guide in - Follow every policy under `docs/policy/`, the task-specific reading guide in
`docs/development.md`, and the accepted behavior in `docs/development.md`, and the accepted behavior in
`local-backend.md`. `prepared-execution.md`.
- Keep the API in the root `promptkit` package. Do not add a public or internal - Keep the supported API in the root `promptkit` package. Internal lifecycle,
package for this feature. validation, and execution state must remain below Go's `internal/`
- Implement the helper as a transparent constructor for the existing boundary.
`Backend` type. Do not add a second backend representation or bypass - Preserve the observable behavior and ordering of existing `Engine.Prepare`
`WithBackend`. and `Engine.Run`. Do not implement the new workflow by redefining either
- Leave normalization, validation, copying, registry construction, capacity existing method in terms of the new public methods.
defaulting, and duplicate detection in their existing owners. - Reuse preparation and post-preparation execution mechanics where their
- Do not add environment-variable discovery, package-global registration, ordering is genuinely shared. Preserve `Run`'s existing early admission
implicit local defaults, model selection, profile construction, or support boundary while ensuring `PrepareExecution` performs complete preparation
for non-OpenAI-compatible transports. without admission.
- Keep tests lean and behavior-focused. Do not duplicate the existing - Do not add durable serialization, retries, provider failover, background
registry and concurrency test matrices merely because the constructor workers, a task queue, capacity reservation during preparation, or
reaches those mechanisms. consumer-specific persistence and redaction policy.
- Update exact GoDoc with the exported declarations. Update current-state - Keep direct API-key values in private execution state only. Every public
consumer guidance only after the corresponding API is implemented. details value, formatted handle, JSON value, result, hash, and error must
- Do not update release notes, create a release, change a module version, or remain credential-free.
tag a commit as part of this work. - Keep tests lean but give durable coverage to the public compatibility,
security, concurrency, one-attempt, source-snapshot, cancellation, and
capacity invariants. Use real internal collaborators where they are fast and
deterministic and fakes at model-generation and synchronization boundaries.
- Update exact GoDoc with exported declarations. Update current-state
consumer and internal documentation only after the corresponding behavior
exists.
- Do not add release notes, change a module version, create a release, or tag
a commit as part of this work.
## Fixed Design ## Fixed Design
### Exported API ### Public API
Add these declarations to `backends.go` in package `promptkit`: Add an opaque root-package type and these methods:
```go ```go
// BackendLocal is the conventional ID used by LocalBackend. It is not a type PreparedExecution struct {
// built-in or reserved backend and must be registered with WithBackend. // Unexported Promptkit-owned state only.
const BackendLocal = "local"
// LocalBackend returns a Backend for a conventional local OpenAI-compatible
// endpoint.
func LocalBackend(endpoint string, concurrencyLimit int) Backend {
return Backend{
ID: BackendLocal,
Endpoint: endpoint,
ConcurrencyLimit: concurrencyLimit,
} }
func (e *Engine) PrepareExecution(
ctx context.Context,
req RunRequest,
) (*PreparedExecution, error)
func (p *PreparedExecution) Details() PreparedRun
func (p *PreparedExecution) Discard()
func (e *Engine) RunPrepared(
ctx context.Context,
prepared *PreparedExecution,
) (*RunResult, error)
```
Place the public handle and its handle-local methods in a new root file named
`prepared_execution.go`. Keep the two engine operations in `engine.go` beside
`Prepare` and `Run` so all engine workflows remain discoverable together.
`PreparedExecution` must contain no exported fields. It wraps one pointer to
Promptkit-owned internal state; copying the public struct therefore preserves
one shared lifecycle rather than creating another execution opportunity.
Implement both:
```go
func (p *PreparedExecution) String() string
func (p *PreparedExecution) GoString() string
```
with a constant credential- and content-free representation such as
`promptkit.PreparedExecution{opaque}`. These methods must not branch on,
inspect, or format retained execution state. Do not add a custom JSON
representation: with no exported fields, ordinary JSON encoding exposes no
state, and JSON is explicitly outside the handle contract.
The exact GoDoc must own these behaviors:
- `PrepareExecution` performs complete preparation but no model call and no
backend-capacity admission;
- the handle is bound to the creating engine and permits one
`RunPrepared` invocation;
- `Details` returns a fresh caller-owned, credential-redacted copy on every
call and remains usable after discard or execution;
- `Discard` is nil-safe and idempotent, invalidates an unclaimed handle, and
is not an execution-cancellation mechanism;
- `RunPrepared` atomically consumes the one attempt before credential
revalidation, admission, generation, or validation;
- lifecycle misuse matches `ErrInvalidRequest`, while a nil engine matches
`ErrInvalidConfig`;
- an operational execution error returns no partial `RunResult`;
- the `RunPrepared` context does not inherit from the preparation context and
governs the execution attempt; and
- handles are opaque in-process values, not serializable or restartable jobs.
For a nil or zero-value handle, `Details` returns a zero `PreparedRun` and
`Discard` does nothing. A nil, zero-value, foreign-engine, discarded,
claimed, or used handle passed to `RunPrepared` returns an error matching
`ErrInvalidRequest`.
### Internal Validation Snapshot
Extend `internal/validate/validator.go` with two internal interfaces:
```go
// PreparedValidation validates artifacts against one frozen output contract.
type PreparedValidation interface {
Validate(
ctx context.Context,
artifact *domain.Artifact,
) (domain.ValidationResult, error)
// SchemaDocument returns the root JSON Schema document used for provider
// structured output, or nil for non-schema modes. The returned value is
// internal immutable state and must not be mutated by callers.
SchemaDocument() any
}
// ValidationPreparer freezes the validation resources for one output
// contract.
type ValidationPreparer interface {
PrepareValidation(
ctx context.Context,
contract domain.OutputContract,
) (PreparedValidation, error)
} }
``` ```
The exact GoDoc may be wrapped or expanded for clarity, but it must own and The exact names may be adjusted only to avoid an actual Go naming conflict;
communicate all of these contract points: the separation of responsibilities and method shapes are fixed.
`validate.Validator` and its existing `Validate` method remain unchanged so
the existing `Run` behavior and internal test fakes do not acquire a new
mandatory method.
- `BackendLocal` is the case-sensitive conventional ID `"local"`; `StandardValidator` and `FSValidator` implement `ValidationPreparer`. Their
- it is neither built in nor reserved; prepared-validation implementation:
- calling `LocalBackend` does not register anything;
- the returned value must be supplied through `WithBackend`;
- `endpoint` and `concurrencyLimit` are copied into the corresponding fields
without normalization or validation;
- `APIKeyEnv`, `ExtraParams`, and `QueueCapacity` retain their zero values; and
- normal `NewEngine` backend validation and concurrency semantics apply after
registration.
Keep `BackendOpenRouter` unchanged. `BackendLocal` must not alias an internal - captures a value copy of the complete `OutputContract`;
registry constant because the internal registry has no special local-backend - handles none, basic, and JSON validation without any source access;
identity or behavior. - for JSON Schema, loads the root document, validates its dialect, registers
it with a new compiler, compiles it during `PrepareValidation`, and retains
the compiled schema;
- allows the compiler to resolve every transitive reference during that
compilation using the existing contained filesystem or `fs.FS` loader;
- returns the same loaded root document through `SchemaDocument`, so the
provider-facing structured-output constraint and later validation derive
from one root snapshot; and
- validates generated instances only against retained in-memory state. Its
`Validate` method must not reopen a path, call `fs.ReadFile`, resolve a
symlink, or invoke a schema loader after preparation succeeds.
Place `BackendLocal` near `BackendOpenRouter`, and place `LocalBackend` after Reuse the existing JSON parsing, validation-result construction, schema
the `Backend` declaration and before `WithBackend`. This keeps the conventional dialect checks, contained-path rules, compiler configuration, and validation
IDs, configured value, convenience constructor, and registration option error wording. A prepared JSON Schema compilation or reference-loading failure
discoverable in one file. is a preparation-time validation error. Existing direct calls to
`Validator.Validate`, including the existing `Run` path, retain their current
lazy compilation and source-loading behavior.
### Constructor Semantics Do not expose the JSON Schema library's compiled type outside
`internal/validate`. Do not place compiled schemas or validation interfaces in
`internal/domain`.
`LocalBackend` is a pure struct constructor. Its complete behavior is ### Internal Prepared-Execution State
equivalent to the keyed literal shown above.
In particular, the constructor must not: Add `internal/usecase/prepared_execution.go`. Define an internal
`PreparedExecution` owned by one `Runner`. It contains:
- trim or parse the endpoint; - the immutable owner `*Runner`;
- reject blank endpoints or negative limits; - a mutex;
- choose a default limit; - a small private lifecycle state with at least ready, claimed, and discarded
- assign an API-key environment variable; states;
- allocate an empty `ExtraParams` map; - a credential-redacted prepared-details snapshot that survives every
- assign a queue-capacity pointer; lifecycle transition; and
- read process environment; - while ready, a pointer to a private payload containing the full
- mutate package-global or engine state; or `domain.PreparedRun` and its `validate.PreparedValidation`.
- call `WithBackend` itself.
Deferred validation is intentional. It keeps one validation path for all The payload is the only retained object containing `RunRequest.APIKey`.
`Backend` values: `WithBackend` copies the public value into engine options, Construct the public-details snapshot as a deep copy of the prepared run and
and `NewEngine` constructs the validated immutable registry. A positive limit clear its `EffectiveModelParams.APIKey`. Deep-copy every mutable nested value,
with a nil queue continues to select the existing default queue capacity of including messages, maps, target extra parameters, and the structured-output
1024; zero continues to mean unlimited; a negative value continues to make schema. Reuse the repository's JSON-compatible copy helpers rather than
`NewEngine` fail with `ErrInvalidConfig`. maintaining an ad hoc reflection copier.
The returned `Backend` remains an ordinary caller-owned value. Consumers can Add these internal operations:
modify it before passing it to `WithBackend`, although task-oriented
documentation should direct materially customized configurations to an
explicit keyed `Backend` literal.
### Identity And Compatibility
Do not add `"local"` to the internal built-in or reserved ID set. A consumer
must be able to register either:
```go ```go
promptkit.LocalBackend(endpoint, limit) func (r *Runner) PrepareExecution(
ctx context.Context,
req domain.RunRequest,
) (*PreparedExecution, error)
func (p *PreparedExecution) Details() *domain.PreparedRun
func (p *PreparedExecution) Discard()
func (r *Runner) RunPrepared(
ctx context.Context,
prepared *PreparedExecution,
) (*domain.RunResult, error)
``` ```
or: Internal exported names are acceptable here because the root facade must call
them, but they remain inaccessible to downstream consumers through Go's
`internal` rule.
```go `PrepareExecution` performs these operations in order:
promptkit.Backend{
ID: promptkit.BackendLocal,
Endpoint: endpoint,
}
```
through the existing `WithBackend` option. Both forms participate in ordinary 1. call the existing request-copy boundary in the root facade before entering
duplicate-ID detection. Existing consumers that already use the literal ID the runner;
`"local"` remain source- and behavior-compatible. 2. run `resolvePreparation`, including the existing environment credential
availability check;
3. prepare a frozen validation plan through
`validate.ValidationPreparer`;
4. construct provider structured-output metadata from the plan's root schema
document for JSON Schema mode;
5. load and hash artifacts, render the session and messages, and assemble the
complete `domain.PreparedRun`; and
6. construct the ready internal handle with independent full and redacted
snapshots.
No existing `Backend`, `WithBackend`, profile, registry, execution-target, or Extract the artifact-loading, rendering, hashing, timing, and
capacity semantics change. Do not modify `internal/backend`, `domain.PreparedRun` assembly portion of `completePreparation` into a helper
`internal/capacity`, `internal/domain`, `engine.go`, `profiles.go`, or that accepts an already-resolved `*domain.StructuredOutputSpec`.
`types.go` for this feature. `completePreparation` must continue to call the existing
`resolveStructuredOutput` first and then use that helper. This preserves
existing `Prepare` and `Run` schema behavior. `PrepareExecution` uses the
frozen validation plan's schema document to build the same structured-output
shape without calling `resolveStructuredOutput`.
When the runner has no validator, preserve the existing runner semantics:
validation is skipped and a no-op prepared plan is sufficient. When it has a
non-nil validator that does not implement `ValidationPreparer`,
`PrepareExecution` fails with `ErrValidation`; do not fall back to a plan that
would reopen sources later. The root engine's standard and `fs.FS` validators
always implement the new interface.
### Claim, Cleanup, And Engine Binding
`RunPrepared` checks engine ownership before attempting a claim. A foreign
runner returns `ErrInvalidRequest` without changing the handle. The owning
runner then locks the state and atomically:
1. accepts only the ready state;
2. changes it permanently to claimed;
3. detaches the private payload from the handle; and
4. unlocks before doing any blocking or collaborator work.
Every valid owning-engine invocation consumes the attempt at this point. Run
ID creation, credential failure, pre-canceled context, capacity rejection,
generation failure, operational validation failure, and success all leave the
handle claimed and unusable.
The execution call holds the detached payload locally and defers cleanup on
every exit. Cleanup must at minimum:
- overwrite the direct `EffectiveModelParams.APIKey` field with an empty
string;
- drop the local prepared-run pointer;
- drop the frozen-validation reference; and
- leave the separate redacted details snapshot intact.
This is reference cleanup, not a promise of secure Go string-memory erasure.
`Discard` locks the same state. If ready, it changes the state to discarded,
detaches the payload, unlocks, and performs the same cleanup. If claimed or
already discarded, it is a no-op. Therefore a `RunPrepared`/`Discard` race has
one winner: discard prevents a claim only if it acquires the ready state
first; otherwise it does not cancel the running attempt.
`Details` locks only long enough to read the retained redacted snapshot and
returns a fresh deep copy. It must not return a pointer, slice, map, schema, or
extra-parameter tree shared with either the handle or another `Details` call.
### Prepared Execution Mechanics
Refactor the post-preparation portion of `Runner.Run` into a private execution
helper that accepts:
- a fully prepared `domain.PreparedRun`;
- a caller-supplied run ID and execution start time; and
- a validation function that accepts the artifact and repair-attempt count.
The helper owns model generation, artifact construction, content validation,
internal repair, final result assembly, and result end timing. The live
`Runner.Run` path supplies its existing `validateOutput` function. The
prepared path supplies a wrapper around the retained
`validate.PreparedValidation`. Both wrappers set the actual repair-attempt
count on the returned validation result.
Keep admission outside this shared helper:
- `Runner.Run` retains its current sequence of run-ID creation, start time,
target resolution, early admission, complete preparation, execution, and
release.
- `Runner.RunPrepared` claims the handle, creates the run ID and execution
start time, revalidates credential availability, admits the frozen backend,
executes the shared helper, and releases admission.
Extract a small private admission helper only if it preserves the exact
existing capacity wrapping and release behavior. The limited-backend lease
must cover generation, validation, and all internal repair calls and must be
released on every return.
Before admission, `RunPrepared` calls the existing `validateAPIKey` with the
frozen environment-variable name, retained direct key, and frozen requirement
flag. A missing environment credential returns an error wrapping
`ErrInvalidRequest` and `ErrAPIKeyEnvMissing`. Do not read and retain the
environment value; the model client continues to resolve it when generation
begins.
The result produced by the shared helper uses the frozen preparation values
for all provenance. For `RunPrepared`, `StartTime`, `EndTime`, and `Duration`
begin after the successful claim and exclude preparation and consumer-held
delay. For ordinary `Run`, retain the current timing boundary beginning before
preparation.
### Error Mapping And Compatibility
Do not add a new public sentinel. Internal lifecycle errors wrap
`usecase.ErrInvalidRequest`, which the existing root mapping exposes as
`promptkit.ErrInvalidRequest`.
`PrepareExecution` uses the same public categories as `Prepare`.
`RunPrepared` may return:
- `ErrInvalidConfig` for a nil engine;
- `ErrInvalidRequest` for handle misuse and invalid credentials;
- `ErrAPIKeyEnvMissing` together with `ErrInvalidRequest` when the frozen
environment name is no longer set and no direct key is retained;
- `ErrCapacityExceeded` for admission rejection;
- `ErrLLMGenerate` for model-generation failures; and
- `ErrValidation` for inability to validate or repair.
Preserve wrapped collaborator and context errors according to the existing
mapping rules. A content validation rejection remains a successful
`RunResult` with `ValidationFailed`, not an operational error.
Do not change stable JSON for `PreparedRun` or `RunResult`, public request and
result ownership, provider request bodies or headers, backend selection,
capacity policies, `LLMClient`, `ArtifactReader`, `Prepare`, or `Run`.
### Test Ownership ### Test Ownership
The root external-package contract suite in `public_contract_test.go` owns the Tests must be split by the narrowest stable owner.
new public behavior. Add one focused test named:
```go `internal/validate/standard_validator_test.go` owns frozen schema mechanics.
func TestLocalBackendConstructsAndRegistersConventionalBackend(t *testing.T) Add focused tests for both configured schema-source forms:
```
The test must: - a directory-backed `StandardValidator` test whose root schema references a
second file; after `PrepareValidation`, replace or remove both files and
prove that valid and invalid artifacts are judged by the original compiled
schema; and
- an `FSValidator` test using a mutable in-memory `fs.FS`; mutate its root and
referenced entries after `PrepareValidation` and prove the same invariant.
1. call `promptkit.LocalBackend` with a test endpoint and a positive, Each test must also confirm that `SchemaDocument` is the original root
test-owned concurrency limit; document used at preparation. Existing path-containment and dialect matrices
2. compare the returned value with this complete expected value: remain the owners of those rules and must not be duplicated.
```go `internal/usecase` tests own orchestration details that cannot be observed
promptkit.Backend{ cleanly at the facade:
ID: promptkit.BackendLocal,
Endpoint: endpoint,
ConcurrencyLimit: limit,
}
```
A whole-value comparison is appropriate here because the exact zero-value - `PrepareExecution` completes source loading, artifact reading, rendering,
fields are part of this small public constructor's contract. and validation-plan creation without calling admission or generation;
3. register that returned value with `WithBackend`; - `RunPrepared` rechecks environment credential availability before admission;
4. add an in-memory profile whose `BackendID` is - the prepared path supplies the frozen validation plan to initial validation
`promptkit.BackendLocal`; and every repair attempt;
5. construct an engine through the existing contract-test prompt fixture; - admission is released on success and each error exit; and
6. call `Prepare`; and - the ordinary `Run` admission-before-complete-preparation ordering remains
7. assert that the prepared result exposes `BackendLocal` as the selected unchanged after refactoring.
backend and the supplied endpoint as the effective endpoint.
This single test protects the realistic compatibility risks: accidental field Use small test-controlled collaborators. Do not assert private enum values,
defaults, a changed conventional ID, failure to compose with `WithBackend`, mutex layout, helper call graphs, or cleanup implementation details.
and accidental treatment of `"local"` as reserved. It also demonstrates that
the helper uses the existing backend/profile path.
Do not add separate tests for blank endpoints, malformed endpoints, negative Add `prepared_execution_contract_test.go` in external package
limits, queue defaulting, duplicate IDs, engine isolation, runtime capacity, `promptkit_test` for the public contract. Cover these distinct risks with the
or caller mutation. Those mechanisms are unchanged and already have tests at smallest coherent set of tests:
their owning boundaries. Do not add internal-package tests for this root
facade constructor.
### Consumer Documentation 1. **Frozen execution and independent details.** Prepare a request, mutate
caller-owned request values, mutable prompt/profile/input/schema sources,
and one returned `PreparedRun`, then run the handle. Assert that the
captured `GenerateRequest`, validation, result provenance, and a second
`Details` call all retain the original prepared state. Assert that source
collaborators are not reopened during execution.
2. **Lifecycle and engine binding.** Prove that a foreign engine is rejected
without consuming the handle, the owner can then run it once, a second
call through either the original or a copied public handle is rejected, and
details remain available.
3. **Concurrent claim.** Race two owning-engine `RunPrepared` calls against a
blocking fake client. Exactly one reaches generation and the other matches
`ErrInvalidRequest`. Run this test under the race detector.
4. **Discard and formatting security.** Retain a distinctive direct API key,
verify that `Details`, ordinary JSON encoding, `%v`, `%+v`, `%#v`, and
lifecycle errors do not contain it or rendered content, discard the handle,
and verify idempotence, execution rejection, and retained details.
5. **Credential and capacity timing.** Prove that preparation does not occupy
admission; removing a required environment credential makes execution fail
before generation; a capacity rejection consumes the handle; and the
execution result's timing excludes a test-controlled delay between
preparation and execution.
Update `docs/consumers/pkg-promptkit.md` after the API exists. Keep Go Reuse existing root contract fixtures and capacity fakes where practical.
declarations and GoDoc as the exact API owner; the guide should help consumers Never put a real-looking credential in a fixture. Do not test JSON equality
choose a workflow and link to `backends.go` for precise semantics. for the opaque handle, exact lifecycle error strings, private state, run-ID
format, or clock-duration precision.
Restructure the backend guidance to present these paths in increasing order of ### Documentation Ownership
configuration:
1. **Endpoint-only profile.** Show a small in-memory `Profile` with After implementation:
`Endpoint` and `Model`. Explain that this is the simplest choice when only
one profile needs the endpoint and shared backend identity or capacity
policy is unnecessary.
2. **Local convenience constructor.** Show
`WithBackend(promptkit.LocalBackend("http://localhost:8000/v1", 2))`
together with a profile using
`BackendID: promptkit.BackendLocal`. Explain briefly that the helper is
explicit, is not pre-registered, does not read environment variables, and
leaves the queue capacity at the existing default for a positive limit.
3. **Complete backend value.** Preserve an advanced example using a keyed
`Backend` literal for needs such as `APIKeyEnv`, an explicit
`QueueCapacity`, extra parameters, a custom ID, or multiple local
endpoints. Use a custom ID other than `"local"` in that example so the
distinction from the conventional helper is clear.
Keep the existing backend-routing, selected-backend identity, concurrency, - exported declarations and GoDoc in `prepared_execution.go`, `engine.go`, and
credential, and error guidance unless a small wording adjustment is required `types.go` own the exact public lifecycle, ownership, security, timing,
to make the new decision path coherent. Avoid repeating the complete field cancellation, and error contracts;
contract or registry validation rules from GoDoc. - `docs/consumers/pkg-promptkit.md` explains when to choose `Prepare`, `Run`,
or `PrepareExecution` plus `RunPrepared`, and shows one concise two-phase
workflow using `defer prepared.Discard()`;
- `docs/internal/sources.md` explains that prepared execution freezes all
source-derived state, including transitive schema references, while
ordinary `Prepare` remains inspection-only;
- `docs/internal/llm.md` explains that prepared generation uses the retained
target and messages, rechecks environment credential availability, and
does not reopen sources;
- `docs/internal/capacity.md` explains that `PrepareExecution` performs no
admission and that `RunPrepared` acquires and holds the normal run lease
across generation, validation, and internal repair; and
- `docs/internal/overview.md` updates the validator and use-case inventory to
mention frozen validation plans and one-attempt prepared execution.
Do not add a new maintained example, README section, format-reference entry, Keep these documents at their established abstraction levels. Link to exact
integration-contract change, internal-document change, or release note. The Go declarations rather than reproducing field-by-field contracts. No README,
consumer-guide snippets are sufficient for this small convenience API, and format reference, integration contract, maintained example, architecture
none of those other documents owns the affected task or contract. policy, release note, or release-procedure update is required because the
feature changes none of their owned concerns.
## Stage 1: Add The Public Constructor And Contract Test ## Stage 1: Add Frozen Validation Plans
**Status:** Complete. **Status:** Pending.
### Objective ### Objective
Add the smallest public API that expresses the accepted local-backend Create an internal validation artifact that eagerly captures every schema
convention and protect its compatibility through the root public boundary. resource required by later validation, without changing existing validation
or engine behavior.
### Implementation Prompt ### Implementation Prompt
1. Re-read `docs/development.md`, all files under `docs/policy/`, 1. Re-read `docs/development.md`, all files under `docs/policy/`,
`local-backend.md`, `backends.go`, the backend-related portion of `prepared-execution.md`, `internal/validate/validator.go`,
`public_contract_test.go`, and the existing backend/concurrency GoDoc before `internal/validate/standard_validator.go`, and the focused validator tests.
editing. 2. Confirm the working tree and preserve all existing roadmap and wishlist
2. Confirm the working tree and preserve the uncommitted roadmaps and any changes.
unrelated consumer changes. 3. Add `PreparedValidation` and `ValidationPreparer` to
3. Add the untyped exported string constant `BackendLocal = "local"` to `internal/validate/validator.go` with the exact responsibilities in the
`backends.go` without changing `BackendOpenRouter`. fixed design. Leave `Validator` unchanged.
4. Add `LocalBackend(endpoint string, concurrencyLimit int) Backend` to 4. Implement `PrepareValidation` for `StandardValidator` and `FSValidator`.
`backends.go` using the exact keyed-literal implementation in the fixed Reuse the existing root resolution, resource identifiers, contained
design. loaders, dialect checks, compiler configuration, JSON parser, and
5. Write complete GoDoc for both declarations. Make their conventional, validation-result builder.
explicit, non-built-in, non-reserved, and deferred-validation semantics 5. Ensure the root document is registered from the already-loaded value and
unambiguous. every transitive reference is loaded during compilation. Retain only the
6. Add compiled schema, root document, and copied contract required for later
`TestLocalBackendConstructsAndRegistersConventionalBackend` to validation.
`public_contract_test.go` exactly as specified under Test Ownership. Reuse 6. Ensure the prepared validator checks its validation-time context and does
the existing contract prompt fixture rather than adding a fixture or test not touch either source after successful preparation.
helper. 7. Add the two focused frozen-reference tests described under Test Ownership.
7. Do not edit internal packages. If the constructor appears to require an Keep existing tests unchanged except for shared test setup that materially
internal change, stop and reconcile the implementation with the fixed reduces duplication.
transparent-constructor design instead.
### Focused Validation ### Focused Validation
Run: Run:
```sh ```sh
gofmt -w backends.go public_contract_test.go gofmt -w internal/validate/validator.go \
go test . -run '^TestLocalBackendConstructsAndRegistersConventionalBackend$' internal/validate/standard_validator.go \
internal/validate/standard_validator_test.go
go test ./internal/validate
go test -race ./internal/validate
go vet ./internal/validate
git diff --check
```
### Completion Gate
Stage 1 is complete when:
- both built-in validator forms produce prepared plans;
- prepared schema validation works after root and referenced sources change or
disappear;
- provider schema metadata can use the exact root document captured by the
plan;
- existing `Validator.Validate` behavior remains unchanged; and
- focused tests, race tests, vet, formatting, and whitespace checks pass.
## Stage 2: Implement Internal Handle Lifecycle And Prepared Execution
**Status:** Pending.
### Objective
Add complete preparation, one-attempt lifecycle management, cleanup, and
source-free execution to the use-case layer while preserving existing
`Prepare` and `Run` behavior.
### Implementation Prompt
1. Re-read the completed Stage 1 declarations, `internal/usecase/runner.go`,
its focused tests, `internal/capacity`, and the execution flow described in
`prepared-execution.md`.
2. Add `internal/usecase/prepared_execution.go` with the internal handle,
payload, lifecycle transitions, deep-copying, redaction, cleanup, and
runner ownership described in the fixed design.
3. Extract the common final preparation helper that accepts resolved
structured-output metadata. Keep `completePreparation` and existing
`resolveStructuredOutput` behavior intact for `Prepare` and `Run`.
4. Implement `Runner.PrepareExecution` using
`validate.ValidationPreparer`. Build JSON Schema structured-output metadata
from the frozen plan's root document.
5. Refactor post-preparation generation, validation, repair, and result
assembly into the shared private helper. Preserve ordinary `Run` timing,
early admission, error ordering, and live validator behavior.
6. Implement `Runner.RunPrepared`: verify owner, claim once, create run ID and
execution start time, recheck credentials, acquire admission, execute
against the retained plan, release admission, and clean the detached
payload on every exit.
7. Implement nil-safe, idempotent discard and fresh redacted details copying.
Confirm the `RunPrepared`/`Discard` race is resolved solely by the shared
mutex transition.
8. Add the internal orchestration tests described under Test Ownership,
including regression coverage for ordinary `Run`'s admission ordering.
### Focused Validation
Run:
```sh
gofmt -w internal/usecase/runner.go \
internal/usecase/prepared_execution.go \
internal/usecase/*_test.go
go test ./internal/usecase
go test -race ./internal/usecase
go test ./internal/validate
go vet ./internal/usecase ./internal/validate
git diff --check
```
### Completion Gate
Stage 2 is complete when:
- a runner can prepare and later execute without reopening any request source;
- one valid invocation consumes the handle on every outcome;
- foreign-runner rejection does not consume the handle;
- discard and claim are race-safe and cleanup drops secret-bearing state;
- prepared execution rechecks credentials and admits only at execution time;
- frozen validation is reused across initial output and repair attempts;
- existing `Prepare` and `Run` tests retain their observable ordering and
behavior; and
- focused tests, race tests, vet, formatting, and whitespace checks pass.
## Stage 3: Publish The Root Facade And Public Contract
**Status:** Pending.
### Objective
Expose the accepted opaque handle workflow through the root package and
protect its compatibility, security, and concurrency guarantees at the
consumer boundary.
### Implementation Prompt
1. Re-read the accepted public workflow, root conversion and error-mapping
helpers, existing `Prepare` and `Run` GoDoc, and the external-package
contract-test conventions.
2. Add root `prepared_execution.go` with the opaque wrapper, `Details`,
`Discard`, `String`, and `GoString`.
3. Add `Engine.PrepareExecution` and `Engine.RunPrepared` to `engine.go`.
Reuse `toDomainRunRequest`, `fromDomainPreparedRun`,
`fromDomainRunResult`, and `mapPublicError`.
4. Add or reuse one public prepared-run deep-copy helper so every `Details`
call returns fresh nested state. Do not expose the internal handle or
retain a public direct credential.
5. Write complete GoDoc for the type and methods. Update `RunRequest`,
`PreparedRun`, and `RunResult` GoDoc only where cross-references are needed
to distinguish the new workflow; do not change their stable JSON or
ownership contracts.
6. Add `prepared_execution_contract_test.go` and cover the five risk groups
under Test Ownership. Reuse existing fixtures and fakes rather than
duplicating backend, profile, schema, and capacity matrices.
7. Explicitly inspect formatted and JSON-encoded handles, lifecycle errors,
details, captured model requests, and results for the test sentinel direct
key.
### Focused Validation
Run:
```sh
gofmt -w prepared_execution.go engine.go types.go convert.go \
prepared_execution_contract_test.go
go test . -run 'PreparedExecution|RunPrepared|PrepareExecution'
go test -race . -run 'PreparedExecution|RunPrepared|PrepareExecution'
go test . go test .
go vet . go vet .
go build . go build .
git diff --check git diff --check
``` ```
Inspect the diff and confirm that this stage changes only `backends.go`,
`public_contract_test.go`, and the already-present roadmap files.
### Completion Gate ### Completion Gate
Stage 1 is complete when: Stage 3 is complete when:
- the exported constant and constructor match the fixed API; - the public API matches the accepted shape;
- the constructor returns only the three specified non-zero fields; - details are fresh, stable, credential-redacted copies before and after every
- the helper registers through the ordinary `WithBackend` path; lifecycle outcome;
- `"local"` remains a valid consumer registration rather than a reserved - default formatting and JSON expose no retained request or credential data;
built-in; - foreign, copied, concurrent, discarded, and reused handles obey the
- the focused public-contract test passes; and one-attempt contract;
- the root package test, vet, build, formatting, and whitespace checks pass. - execution uses exactly the frozen request and validation state;
- preparation performs no generation or admission;
- result provenance matches details and execution timing excludes preparation
and retention delay;
- public errors preserve the required `errors.Is` identities; and
- focused public, race, package, vet, build, formatting, and whitespace checks
pass.
## Stage 2: Publish Consumer Guidance And Validate The Repository ## Stage 4: Update Documentation And Validate The Repository
**Status:** Complete. **Status:** Pending.
### Objective ### Objective
Make the simplest suitable local-endpoint configuration easy to discover, Publish current-state guidance at the correct documentation owners, validate
confirm the complete change across the repository, and close the temporary the complete repository, and mark the temporary roadmaps complete.
roadmaps.
### Implementation Prompt ### Implementation Prompt
1. Re-read the implemented declarations and GoDoc before describing them. 1. Re-read the implemented declarations and tests before documenting them.
2. Update `docs/consumers/pkg-promptkit.md` according to the three-path 2. Update `docs/consumers/pkg-promptkit.md`,
structure under Consumer Documentation. `docs/internal/sources.md`, `docs/internal/llm.md`,
3. Keep examples illustrative, minimal, secret-free, and consistent with the `docs/internal/capacity.md`, and `docs/internal/overview.md` according to
implemented declarations. Link precise semantics to `backends.go` rather Documentation Ownership.
than duplicating its field-by-field contract. 3. Keep the consumer example concise, copyable, secret-free, and explicit
about `defer prepared.Discard()`. Link to declarations for exact lifecycle
and error semantics.
4. Follow every added or changed Markdown link and confirm its target exists. 4. Follow every added or changed Markdown link and confirm its target exists.
Confirm that all local repository-relative links in Confirm that repository-relative links in both roadmap documents and all
`local-backend.md`, this plan, and the changed consumer guide resolve. changed current-state documents resolve.
5. Run the complete maintainer validation sequence below. 5. Run the complete maintainer validation sequence below.
6. Inspect the final diff for accidental internal behavior changes, 6. Inspect the final diff for accidental provider-wire changes, source
generated artifacts, credentials, local workspace files, or module reloading during prepared execution, stable-JSON changes, credentials,
replacements. generated artifacts, local workspace files, module replacements, or
7. After every completion gate passes, set both stage statuses, this plan's unrelated edits.
status, and the status in `local-backend.md` to `Complete`. Do not retire or 7. After every completion gate passes, set every stage status, this plan's
remove the roadmaps in the implementation change; roadmap retirement status, and the status in `prepared-execution.md` to `Complete`. Do not
follows implementation review. retire or remove the roadmaps in the implementation change; roadmap
retirement follows implementation review.
### Full Validation ### Full Validation
@@ -318,69 +686,37 @@ Run the complete sequence from `docs/development.md`:
```sh ```sh
go test ./... go test ./...
go test -race ./... go test -race ./...
go run ./examples/go-library/prepare
go vet ./... go vet ./...
go build ./... go build ./...
go run ./examples/go-library/prepare
gofmt -l $(git ls-files '*.go') gofmt -l $(git ls-files '*.go')
git diff --check
``` ```
The formatting command must produce no paths. Follow every added or changed Then run the documented Markdown-link and repository-hygiene checks from
Markdown link and confirm its target and heading exist. `docs/development.md`, followed by:
Also inspect:
```sh ```sh
git diff --check
git status --short git status --short
git diff --stat
git diff
``` ```
Confirm that:
- production changes are limited to the root convenience API;
- no internal backend, registry, capacity, profile, or engine behavior
changed;
- `BackendLocal` is conventional and consumer-registerable, not built in or
reserved;
- `LocalBackend` performs no validation, normalization, environment lookup,
registration, allocation, or hidden default selection;
- the returned `Backend` leaves `QueueCapacity` nil so existing positive-limit
defaulting remains owned by the registry;
- existing endpoint-only profiles and complete custom backends remain
documented and supported;
- current-state documentation describes only the now-implemented API and
links to the canonical GoDoc for exact semantics;
- no `go.work`, `go.work.sum`, local module replacement, credential,
generated binary, or unrelated change was introduced; and
- the feature and implementation roadmaps contain no unresolved work marked
complete.
### Completion Gate ### Completion Gate
Stage 2 is complete when every target-end-state item in Stage 4 and the feature are complete when:
`local-backend.md` is implemented, the consumer guide clearly presents all
three configuration paths, all complete validation commands pass, all changed
links resolve, and the roadmap statuses accurately report completion.
## Implementation Handoff - exact public contracts and task-oriented guidance are documented at their
canonical owners;
The implementation handoff should report: - internal source, validation, execution, credential, and capacity boundaries
are accurately described without duplicating public GoDoc;
- the new `BackendLocal` and `LocalBackend` public API; - all changed links resolve and no documentation claims unimplemented
- that the helper remains explicit and composes with the ordinary backend behavior;
registry; - the complete ordinary and race test suites pass offline;
- the focused public-contract coverage added; - the maintained preparation example, vet, build, formatting, link, hygiene,
- the consumer-guide decision path added; and whitespace checks pass;
- the complete validation commands and results; and - the final diff is limited to the feature, its tests, documentation, and
- any unrelated working-tree changes that were preserved. already-present roadmap changes; and
- both roadmap documents accurately report completion.
Do not claim that a local backend is built in, pre-registered, configured from
environment variables, or assigned a default model or concurrency limit.
## Open Questions ## Open Questions
None. The feature roadmap and this plan fix the exported API, constructor None.
semantics, identity treatment, compatibility behavior, test boundary,
documentation ownership, implementation sequence, validation gates, and
non-goals required for implementation.

View File

@@ -0,0 +1,358 @@
# Notarius PromptKit Wishlist
## Purpose
This document records features and interface changes that would be useful
additions to PromptKit from the perspective of the maintainers of Notarius, a
downstream application that consumes PromptKit.
PromptKit v0.3.0 provides the capabilities Notarius currently needs. None of
the ideas below blocks current Notarius development. They are opportunities to
reduce downstream workarounds, improve integration correctness, and make
PromptKit more ergonomic for applications with configuration validation,
debugging, checkpointing, and operational-observability requirements.
The examples are API sketches intended to communicate the desired capability,
not prescriptive names or finalized Go contracts.
## Priority 1: Atomic Execution With Prepared Details
**Disposition:** Covered by the accepted
[executable preparation handles](prepared-execution.md) roadmap. The shared
two-phase capability should provide the required single-preparation
consistency; a separate `RunDetailed` method is not cataloged initially.
### Downstream need
Notarius needs both:
- the completed `RunResult`; and
- the rendered messages, effective output contract, hashes, and other
preparation details exposed by `PreparedRun`.
Notarius uses the prepared details to construct redaction-aware debug bundles
and retain enough information to diagnose model behavior.
### Current integration
Notarius currently calls `Engine.Prepare` and then `Engine.Run` with the same
request. Because `Run` performs preparation internally, a successful request
resolves and prepares the same work twice.
This duplicates profile resolution, input hashing, schema loading, and prompt
rendering. It also creates a theoretical consistency window in which a
filesystem-backed prompt, profile, schema, or input could change between the
explicit preparation and the preparation performed by `Run`.
### Requested capability
Add an opt-in execution method that prepares exactly once and returns both the
prepared details and completed result:
```go
type RunReport struct {
Prepared PreparedRun
Result RunResult
}
func (e *Engine) RunDetailed(
ctx context.Context,
req RunRequest,
) (*RunReport, error)
```
The exact names are flexible. The important contract is that preparation
occurs once and that the returned prepared state describes the execution that
produced the returned result.
Existing `Prepare` and `Run` behavior should remain available for consumers
that need only one side of the operation.
### Design considerations
- Keep this API additive and preserve the existing simple `Run` workflow.
- Return caller-owned copies under PromptKit's existing ownership rules.
- Define whether any prepared details are available after an operational
generation or validation error. Notarius does not require partial results
for the initial use case, but an explicit contract would be valuable.
- Preserve cancellation and backend-admission semantics.
- Do not add all prepared content directly to `RunResult`. Rendered prompt
content can be large and sensitive, and consumers should opt in to receiving
it.
### Value to Notarius
This is the highest-value wishlist item. It would remove duplicate work from
every successful PromptKit-backed call and ensure that retained debug material
corresponds atomically to the actual execution.
## Priority 2: Prompt-Independent Profile Inspection
**Disposition:** Accepted into the
[future catalog](future.md#prompt-independent-profile-inspection).
### Downstream need
Notarius validates configured pipeline profile IDs before beginning a run. It
needs to determine whether:
- a profile exists;
- its referenced backend is registered;
- its execution target can be resolved; and
- it declares a credential requirement that the application may need to
enforce.
This validation should not require model generation.
### Current integration
Notarius constructs a synthetic prompt using `testing/fstest.MapFS`, supplies a
dummy transcript, and calls `Engine.Prepare` solely to exercise profile and
backend resolution. This works, but prompt preparation is serving as a
substitute for a profile-inspection interface.
### Requested capability
Add a prompt-independent profile-resolution API, for example:
```go
type ResolvedProfile struct {
ProfileID string
BackendID string
EffectiveTarget ExecutionTarget
APIKeyEnv string
}
func (e *Engine) ResolveProfile(
ctx context.Context,
profileID string,
) (ResolvedProfile, error)
```
The returned shape may differ, but it should provide enough information for a
consumer to validate an explicit profile selection without inventing a prompt
or supplying placeholder inputs.
### Design considerations
- Resolve built-in, file-backed, and programmatic profiles using normal
PromptKit precedence.
- Validate that a referenced backend registration exists.
- Do not resolve, retain, or expose credential values.
- Report credential requirements, such as an environment-variable name, so
the consuming application can decide whether availability is required at
configuration-validation time or only at execution time.
- Return caller-owned values.
- Preserve typed or sentinel error classification for missing and invalid
profiles.
- Consider accepting an `ExecutionTargetOverride` if consumers need to inspect
the same effective target that a run-level override would produce.
- Enumeration of all profiles is not required for the Notarius use case; exact
lookup by ID is sufficient.
### Value to Notarius
This would eliminate a synthetic production-only prompt fixture and establish
a direct, supported contract for configuration-time profile and backend
validation.
## Priority 3: Semantic Execution-Target Fingerprints
**Disposition:** Deferred until prompt-independent profile inspection defines
the resolved target whose configuration identity would be fingerprinted.
### Downstream need
Notarius checkpoints model-backed pipeline stages. A checkpoint must not be
reused when generation-affecting PromptKit configuration changes.
Notarius therefore needs a stable equality signal for the effective profile
and backend target used by a pipeline.
### Current integration
Notarius currently constructs this identity itself from:
- a manually maintained marker for the PromptKit release and built-in profile
catalog;
- raw hashes of configured profile files; and
- a separate hash of the configured conventional local-backend endpoint.
This is safe but conservative and coupled to PromptKit details. Raw file
hashing also invalidates checkpoints for semantically irrelevant YAML changes,
such as comments or formatting.
### Requested capability
Expose an opaque semantic digest for a resolved profile and its effective
generation target. It could be returned by the proposed profile-resolution
API:
```go
type ResolvedProfile struct {
ProfileID string
BackendID string
EffectiveTarget ExecutionTarget
ExecutionDigest string
}
```
Alternatively, PromptKit could expose a dedicated method such as
`ProfileExecutionDigest(profileID)`.
### Desired equality semantics
The digest should change when generation-affecting state changes, including:
- resolved model and endpoint;
- backend routing identity;
- backend request defaults and extra parameters;
- profile generation parameters; and
- the semantic identity of any selected built-in profile.
The digest should not incorporate:
- credential values;
- concurrency or queue capacity;
- filesystem source paths;
- YAML comments or formatting; or
- other settings that affect scheduling or source representation without
changing the generation target.
The credential environment-variable name may need to participate if changing
it can select a materially different provider account or target. PromptKit
should define this deliberately while continuing to exclude the resolved
secret value.
### Design considerations
- Treat the digest as an opaque equality value rather than a public encoding
of internal structures.
- Document which categories of change affect equality.
- Include a versioned semantic marker internally so PromptKit can deliberately
invalidate old digests when its resolution semantics change.
- Prefer a per-profile digest over a digest of every profile known to an
engine. Notarius generally knows which profiles a resolved pipeline uses.
- Do not require consumers to know PromptKit's built-in catalog version.
### Value to Notarius
This would let Notarius remove its PromptKit release marker and raw
profile-source fingerprinting, reduce unnecessary checkpoint invalidation, and
delegate execution-target equality to the component that owns target
resolution.
## Priority 4: Structured Capacity Errors
**Disposition:** Accepted into the
[future catalog](future.md#structured-capacity-errors).
### Downstream need
Notarius translates PromptKit backend-capacity rejection into a
provider-neutral application error. When multiple backends are active,
operators would benefit from knowing which backend rejected admission without
parsing an error string or exposing endpoint details.
### Current integration
PromptKit provides the useful `ErrCapacityExceeded` sentinel. Notarius can
classify the failure reliably, but it retains only a sanitized diagnostic
string as additional context.
### Requested capability
Add a typed error that continues to match `ErrCapacityExceeded`:
```go
type CapacityError struct {
BackendID string
}
func (e *CapacityError) Is(target error) bool {
return target == ErrCapacityExceeded
}
```
The exact implementation may use `Unwrap` or another idiomatic mechanism. The
important properties are compatibility with `errors.Is` and discoverability
through `errors.As`.
### Design considerations
- Include the stable backend ID.
- Do not expose the backend endpoint, credential environment, credential
value, request content, or other sensitive configuration.
- Consider including the configured concurrency and queue limits if they are
useful and safe, but backend identity alone provides most of the downstream
value.
- Add a retry delay only if PromptKit can provide a meaningful value. A full
queue does not necessarily imply a reliable `Retry-After` duration.
- Keep retry and backoff policy with the consuming application. PromptKit
should classify the admission failure rather than silently retry it.
### Value to Notarius
This would improve operational diagnostics and future metrics while preserving
the provider-neutral error boundary used by Notarius.
## Capabilities PromptKit Already Provides Well
The current PromptKit boundary is sufficient for Notarius's implemented
behavior. In particular, PromptKit already provides:
- filesystem, `fs.FS`, and programmatic prompt, profile, and schema sources;
- offline preparation without model execution;
- structured output and content validation;
- direct session propagation;
- tri-state per-run reasoning overrides;
- selected profile, backend, model, endpoint, effective parameters, hashes, and
token-usage provenance;
- endpoint-only profiles;
- the conventional `local` backend helper;
- arbitrary engine-scoped `Backend` registrations;
- backend authentication environment names, extra parameters, concurrency
limits, and queue-capacity policies;
- provider-client and artifact-reader extension interfaces;
- context cancellation; and
- useful public error sentinels, including profile absence and capacity
exhaustion.
The wishlist does not imply that Notarius needs PromptKit to broaden its core
responsibilities. It primarily asks for more direct access to information and
operations that PromptKit already computes internally.
## Responsibilities That Should Remain In Notarius
The following concerns belong to the downstream application and should not
move into PromptKit for the sake of Notarius:
- pipeline staging, dependencies, and generated references;
- application-wide scheduling across providers and backends;
- module and validation retry policy;
- checkpoints, resume, and recomputation;
- durable run artifacts and manifests;
- D&D prompts, schemas, extractors, validators, and normalizers;
- Notarius configuration-file parsing and precedence;
- domain-specific prompt-cache prefix policy; and
- application-specific redaction, retention, and debug-bundle policy.
PromptKit's complete `Backend` API already supports custom IDs, multiple local
endpoints, authentication, extra parameters, and explicit queue policies.
Whether Notarius exposes those capabilities in its own configuration is an
application-policy decision, not an upstream PromptKit gap.
## Suggested Upstream Sequence
If the PromptKit team chooses to pursue these ideas, the most useful order for
Notarius would be:
1. Add atomic execution that returns prepared details and the completed result.
2. Add prompt-independent profile inspection.
3. Add a semantic execution-target digest, preferably as part of profile
inspection.
4. Add a typed capacity error carrying backend identity.
The first two address concrete workarounds in current Notarius code. The third
would improve checkpoint correctness and reduce coupling. The fourth is
operational polish.

View File

@@ -0,0 +1,280 @@
# Executable Preparation Handles
**Status:** Accepted.
## Purpose
Allow a consumer to prepare one exact Promptkit execution, inspect and retain
its credential-redacted public preparation details, and later execute that
already-prepared work without resolving or rendering the request again.
This provides a supported preflight-before-generation boundary for
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-1-executable-preparation-handles)
and removes the duplicate `Prepare`-then-`Run` workaround described by
[Notarius](notarius-promptkit-wishlist.md#priority-1-atomic-execution-with-prepared-details).
## Motivation
`Engine.Prepare` currently returns the provenance and rendered details that
consumers need for debugging, persistence, and preflight checks. `Engine.Run`
then performs its own preparation before generation. A consumer that needs
both values must therefore prepare the same logical request twice.
That workaround duplicates source loading, input hashing, schema work, and
template rendering. It also permits filesystem-backed prompts, profiles,
schemas, or inputs to change between the public preparation and the
preparation that actually produces the result.
Promptkit already owns the complete preparation and execution pipeline. An
opt-in prepared-execution handle should expose the missing boundary without
putting rendered content into every `RunResult` or moving persistence policy
into the library.
## Consumer Workflow
The target public workflow is:
```go
prepared, err := engine.PrepareExecution(ctx, request)
if err != nil {
// Handle preparation failure.
return
}
defer prepared.Discard()
details := prepared.Details()
// Persist or inspect a consumer-selected safe subset of details.
result, err := engine.RunPrepared(ctx, prepared)
```
The target public surface is:
```go
type PreparedExecution struct {
// Opaque Promptkit-owned state.
}
func (e *Engine) PrepareExecution(
ctx context.Context,
req RunRequest,
) (*PreparedExecution, error)
func (p *PreparedExecution) Details() PreparedRun
func (p *PreparedExecution) Discard()
func (e *Engine) RunPrepared(
ctx context.Context,
prepared *PreparedExecution,
) (*RunResult, error)
```
The declarations and GoDoc will own the exact implemented contract. The
important public shape is an opaque handle, caller-owned `PreparedRun`
details, an explicit discard operation, and execution through the engine that
created the handle.
## Prepared Snapshot
`PrepareExecution` performs complete preparation without model generation or
backend-capacity admission. It applies the same request validation, source
precedence, backend and profile resolution, credential requirement checks,
output-contract resolution, artifact loading, hashing, schema loading,
session resolution, and rendering behavior as `Prepare`.
A successful handle freezes all source-derived state required for later
execution, including:
- the selected prompt definition, profile, and backend identity;
- the complete effective execution target and request-field presence;
- rendered messages and effective session ID;
- prompt, rendered-prompt, and input hashes;
- the effective output contract and provider-facing structured-output
constraint; and
- private validation state sufficient to validate generated output without
reopening schema files or `fs.FS` resources, including required schema
references.
After `PrepareExecution` succeeds, changes to prompt, profile, schema, input,
or request-owned data cannot change what `RunPrepared` sends to the model or
how it validates the generated output.
The handle retains an internal snapshot independent from values returned by
`Details`. Mutating a returned `PreparedRun`, its maps, slices, messages, or
schema values does not affect later execution. Each `Details` call returns a
fresh caller-owned copy under the existing `PreparedRun` ownership and stable
JSON rules.
## Handle Lifecycle
A `PreparedExecution` is:
- created only by a successful `PrepareExecution` call;
- bound to the exact `Engine` that created it;
- valid for one `RunPrepared` invocation;
- safe for repeated `Details` calls;
- intentionally opaque and without a supported JSON representation; and
- in-process state rather than a durable or restartable job.
`RunPrepared` atomically claims a valid handle before beginning the execution
attempt. A second or concurrent invocation fails without starting another
execution, including when the first invocation ended in cancellation, capacity
rejection, generation failure, or operational validation failure. Copies of
the public handle share the same one-attempt state and cannot bypass this rule.
A nil, zero-value, foreign-engine, discarded, already-claimed, or already-used
handle is invalid. `RunPrepared` reports these lifecycle errors through the
ordinary public invalid-request category. A failed foreign-engine invocation
does not consume a handle that remains valid for its owning engine.
`Discard` idempotently makes an unclaimed handle unavailable for execution and
drops Promptkit's references to secret-bearing or execution-only state.
`RunPrepared` performs the same cleanup automatically after claiming a handle.
Credential-redacted public preparation details remain available after discard,
success, or failure so consumers can retain diagnostics. Promptkit does not
promise secure erasure of Go string memory.
Lifecycle transitions are concurrency-safe. When `RunPrepared` and `Discard`
race, exactly one claims the ready handle. `Discard` is not an execution
cancellation mechanism and does not interrupt an attempt that has already
claimed the handle; consumers cancel that attempt through its context.
## Credentials And Sensitive Data
`Details` has the same security contract as `PreparedRun`: it can contain
rendered messages, schemas, identifiers, and hashes, but never a resolved API
key value. Consumers remain responsible for selecting, redacting, storing, and
retaining any persisted preparation material.
A direct `RunRequest.APIKey` is retained only in opaque execution state until
the handle is run or discarded. It is never added to details, hashes, JSON,
`String`, or `GoString` output.
An environment-variable name is frozen as part of the effective target, but
its credential value is not captured for the lifetime of the handle.
`PrepareExecution` applies the existing preparation-time availability check.
`RunPrepared` rechecks availability before admission, and the selected model
client uses the environment value visible during execution. This preserves
current secret ownership and avoids retaining an environment credential while
a consumer persists preflight material.
The opaque handle must not expose retained request data or credentials through
default formatting, JSON, or error messages.
## Admission, Cancellation, And Execution
`PrepareExecution` never reserves backend admission or an active-generation
permit. Its context governs preparation only; cancellation after it returns
does not invalidate the handle.
`RunPrepared` uses its own context for credential revalidation, backend
admission, active-generation waiting, model generation, output validation, and
any internal repair. Admission occurs when `RunPrepared` begins so a consumer
cannot occupy bounded capacity while inspecting or persisting preparation
details.
For a limited backend, the admission lease covers the complete prepared
execution attempt after admission: generation, validation, internal repair,
and every success or failure exit. Actual generation continues to use the
backend's FIFO active-generation permit. Existing capacity error identity,
cancellation behavior, and release guarantees remain in force.
Because a prepared handle is one-attempt, cancellation or capacity rejection
does not make it reusable. Retry and backoff policy remains with the consumer,
which may create a new prepared handle when another attempt is appropriate.
## Results And Failures
On success, `RunPrepared` returns the existing caller-owned `RunResult`. Its
source-derived provenance must match `Details`, including prompt identity and
hash, rendered-prompt hash, session ID, selected profile and backend,
effective target, and input hashes.
`RunResult.StartTime`, `EndTime`, and `Duration` describe the
`RunPrepared` execution attempt. They exclude preparation time and any delay
while the consumer retained the handle. Preparation timing remains in
`PreparedRun`.
Preparation failure returns no handle. After successful preparation,
`RunPrepared` retains the existing rule that an operational failure returns no
partial `RunResult`; the consumer already has independent preparation details.
A completed content-validation failure remains a successful result with
`ValidationFailed`.
`PrepareExecution` preserves the public error categories of `Prepare`.
`RunPrepared` preserves applicable invalid-request, credential, capacity,
generation, validation, collaborator, and cancellation identities without
reintroducing source-loading or rendering failures from frozen state.
## Compatibility And Existing Workflows
This feature is additive:
- `Prepare` remains the simple preparation-only operation;
- `Run` remains the simple prepare-and-execute operation with its current
early-admission and error-ordering behavior;
- `PreparedRun` and `RunResult` retain their existing stable JSON
representations;
- model-client and artifact-reader extension interfaces remain unchanged; and
- backend routing, concurrency limits, queue capacities, and provider wire
behavior remain unchanged.
The new workflow may share internal machinery with `Prepare` and `Run`, but it
must not change their observable behavior merely to simplify implementation.
## Documentation
The completed documentation set has these ownership boundaries:
- exported declarations and GoDoc own the exact handle, method, lifecycle,
ownership, concurrency, credential, error, and cancellation contracts;
- the promptkit consumer guide explains when to use `Prepare`, `Run`, or the
two-phase prepared-execution workflow; and
- internal runner, source-validation, capacity, and model-client documentation
describe the implemented collaborator boundaries without duplicating public
contracts.
No release document is part of the feature implementation itself. Release
guidance is prepared only when the resulting public API is selected for
publication.
## Non-Goals
This work does not include:
- serializable, durable, resumable, or cross-process execution handles;
- reuse for multiple consumer-initiated executions;
- concurrent execution of one handle;
- capacity reservation during preparation;
- a background task queue, priorities, worker lifecycle, or job status;
- retry, backoff, or provider failover policy;
- a separate `RunDetailed` convenience method;
- adding preparation details or rendered messages to every `RunResult`;
- prompt or profile inspection APIs;
- structured capacity or generation errors;
- freezing environment-variable credential values for the handle lifetime;
- snapshotting provider state or mutable behavior inside an injected
`LLMClient`;
- allowing target, validation, session, variable, or input overrides after
preparation; or
- changing existing `Prepare`, `Run`, file-format, provider-request, or stable
JSON contracts.
## Target End State
After this work:
- consumers can perform and persist preflight before starting provider work;
- one prepared handle executes exactly the source-derived prompt, target,
schema, inputs, session, and messages described by its public details;
- execution never reloads or rerenders consumer sources;
- direct credentials remain confined to opaque, explicitly discardable state;
- environment credentials are not retained across the preflight boundary;
- backend capacity is reserved only when execution begins;
- one handle can start at most one execution attempt, including any internal
repair calls owned by that attempt;
- preparation details remain available after execution success or failure;
- existing simple `Prepare` and `Run` consumers remain unaffected; and
- Promptkit continues to own reusable execution mechanics without taking on
downstream persistence, redaction, retry, or job-management policy.

View File

@@ -0,0 +1,468 @@
# Weatherreporter PromptKit Wishlist
## Purpose
This document records features and interface changes that would be useful
additions to PromptKit from the perspective of the maintainers of
Weatherreporter, a downstream application planning to replace its Scriptorium
CLI integration with PromptKit.
PromptKit v0.3.0 provides the capabilities Weatherreporter needs for the
migration. None of the ideas below is a hard adoption requirement. They are
opportunities to avoid duplicate preparation, validate configuration earlier,
improve durable failure diagnostics, and make the integration more direct.
The examples are API sketches intended to communicate the desired capability,
not prescriptive names or finalized Go contracts. The related
[Notarius PromptKit wishlist](notarius-promptkit-wishlist.md) proposes several
overlapping features from another downstream consumer's perspective.
## Priority 1: Executable Preparation Handles
**Disposition:** Accepted into the
[executable preparation handles](prepared-execution.md) feature roadmap.
### Downstream need
Weatherreporter treats prompt preparation as a durable preflight boundary. It
needs to:
1. prepare the exact request that will be executed;
2. persist a safe preparation record before starting the provider call; and
3. execute without reloading or rerendering prompt, profile, schema, or input
sources.
Persisting preflight before generation leaves useful evidence when a provider
call fails or the process is interrupted during generation.
### Current integration option
With PromptKit v0.3.0, Weatherreporter can call `Engine.Prepare`, save selected
fields from the returned `PreparedRun`, and then call `Engine.Run` with the
same request. Because `Run` performs preparation internally, the work is
repeated.
Weatherreporter plans to use embedded prompt and schema files plus immutable
inline input bytes, which removes most of the consistency risk. An external
profile file or directory can still change between the two calls, and the
second preparation remains unnecessary work.
The atomic `RunDetailed` operation proposed by the
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-1-atomic-execution-with-prepared-details)
would guarantee that returned preparation details describe the completed
execution. However, returning those details only after generation would not
preserve Weatherreporter's preflight-before-generation persistence boundary.
### Requested capability
Add an opt-in two-phase API that returns a prepared execution handle:
```go
prepared, err := engine.PrepareExecution(ctx, request)
if err != nil {
// Handle preparation failure.
}
details := prepared.Details()
// Persist a consumer-selected safe preparation record.
result, err := engine.RunPrepared(ctx, prepared)
```
The exact names and shapes are flexible. The important contract is that
`RunPrepared` executes the already prepared prompt and does not reload or
rerender its prompt, profile, schema, or input sources.
`Details` should return the same caller-owned public preparation information
currently represented by `PreparedRun`. The execution handle may retain opaque
engine-owned state needed to invoke the model and validate the response.
### Design considerations
- Keep `Prepare` and `Run` available for consumers that do not need a
two-phase execution boundary.
- Bind a prepared handle to the engine that constructed it.
- Define whether a handle is one-shot, reusable, or safe for concurrent use.
A one-shot contract may be the safest initial design.
- Do not give the opaque handle a stable JSON representation.
- Do not expose or serialize resolved credential values through `Details`.
- Define how a direct request API key is retained and released when an opaque
handle must carry it until execution.
- Preserve caller-owned copies for all public details.
- Make context cancellation and backend admission timing explicit.
- Document whether profile credential environment values are resolved during
preparation or execution.
- Ensure an execution error does not invalidate the public details already
returned to the consumer.
- Consider whether an atomic `RunDetailed` can share the same internal
prepared-execution implementation.
### Value to Weatherreporter
This is the highest-value upstream addition. It would preserve
Weatherreporter's durable preflight behavior, remove duplicate work, eliminate
the remaining source-consistency window, and ensure that persisted provenance
describes the actual execution.
## Priority 2: Prompt-Definition Inspection
**Disposition:** Accepted into the
[future catalog](future.md#prompt-definition-inspection).
### Downstream need
Weatherreporter has a fixed registry of seven report definitions. Each report
selects a prompt ID and one of two output workflows:
- direct Markdown; or
- structured generated text followed by application-owned domain validation
and Markdown template rendering.
Weatherreporter will embed the PromptKit prompt definitions and private
response schemas that implement those reports. It needs to validate that the
report registry and embedded prompt corpus agree before weather collection or
provider execution.
### Current integration option
Weatherreporter can maintain synthetic data-package fixtures and call
`Engine.Prepare` for every report prompt during tests. Runtime validation can
also occur through the ordinary per-report preparation stage.
This works, but it requires complete placeholder inputs and profile resolution
when the application primarily wants to inspect prompt identity and declared
contracts.
### Requested capability
Add exact prompt-definition lookup without rendering or generation:
```go
type PromptInfo struct {
PromptID string
PromptVersion string
PromptHash string
DefaultProfileID string
Inputs []InputDefinition
OutputContract OutputContract
}
func (e *Engine) ResolvePrompt(
ctx context.Context,
promptID string,
promptVersion string,
) (PromptInfo, error)
```
The exact returned shape may differ. Weatherreporter needs enough information
to verify prompt existence, version selection, declared inputs, default
profile identity, output format, validation mode, and schema selection without
supplying synthetic prompt input.
### Design considerations
- Use ordinary PromptKit prompt-source precedence and exact ID/version
selection.
- Fully load and structurally validate the selected prompt definition.
- Validate referenced prompt content files without rendering their templates.
- Resolve and validate the selected output contract and schema reference where
practical.
- Return an opaque prompt-definition equality value rather than raw source
bytes.
- Do not return rendered messages, schema bodies, profile credentials, or
another source of sensitive content.
- Preserve typed or sentinel errors for missing and invalid prompts.
- Return caller-owned values.
- Enumeration of all known prompts is not required for Weatherreporter; exact
lookup is sufficient.
### Value to Weatherreporter
This would let Weatherreporter directly verify that every report prompt
exists, requires the curated `data_package` input, and declares the expected
Markdown or JSON Schema output contract. It would reduce synthetic test setup
and move failures ahead of weather collection.
## Priority 3: Prompt-Independent Profile Inspection
**Disposition:** Accepted into the
[future catalog](future.md#prompt-independent-profile-inspection).
### Downstream need
Weatherreporter will allow operators to select an external PromptKit profile
source and may allow an explicit profile override. It should reject a missing
profile, unknown backend, malformed execution target, or unsatisfied credential
requirement before collecting weather data or writing report artifacts.
### Current integration option
Weatherreporter can validate an explicit profile by preparing one embedded
prompt with fixture input. Prompts that use their own default profiles can be
validated during their normal preparation stage.
This couples configuration validation to one prompt and requires placeholder
input even when only profile and backend resolution are relevant.
### Requested capability
The prompt-independent `ResolveProfile` API proposed by the
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
would satisfy this need. It should resolve built-in, file-backed, and
programmatic profiles, validate backend membership, report credential
requirements without resolving credential values, and preserve typed error
classification.
### Additional Weatherreporter considerations
- An explicit application profile override should be inspectable without
selecting a report prompt.
- A prompt-definition inspection result may expose its default profile ID so
Weatherreporter can inspect that profile separately.
- Inspection should distinguish structural profile validity from current
credential availability so configuration validation can apply explicit
application policy.
- An optional execution-target override should be considered only if it
describes the same target that a later run will use.
### Value to Weatherreporter
This would improve fail-fast configuration validation and give operator-facing
errors direct profile and backend context. It is valuable but not required for
the initial migration.
## Priority 4: Eager Source Validation
**Disposition:** Deferred until prompt and profile inspection have been used
to determine whether a broader engine-wide validation operation is still
needed.
### Downstream need
PromptKit deliberately defers reading and validating filesystem and `fs.FS`
prompt, profile, and schema content until a request needs it. Weatherreporter
has a small fixed embedded prompt corpus and one optional external profile
source. It would benefit from an explicit offline validation operation for
tests, startup diagnostics, and configuration checks.
### Current integration option
Weatherreporter can prepare every report prompt with fixture inputs and inspect
any explicit profiles individually. That provides strong coverage but requires
consumer-maintained traversal and synthetic material.
### Requested capability
Consider an opt-in source-validation operation:
```go
type SourceValidationOptions struct {
RequireCredentials bool
}
func (e *Engine) ValidateSources(
ctx context.Context,
opts SourceValidationOptions,
) error
```
The operation should eagerly discover and structurally validate the configured
prompt, profile, and schema sources without model generation.
### Design considerations
- Keep deferred validation as the normal `NewEngine` behavior.
- Make eager validation an explicit consumer choice.
- Validate duplicate IDs and versions, strict YAML decoding, referenced content
files, profile/backend membership, schema syntax, and schema references.
- Distinguish structural credential declarations from current environment
availability.
- Do not read or expose credential values when credential availability is not
requested.
- Preserve source-specific public error identities and useful path context.
- Respect context cancellation during filesystem discovery and schema work.
- Consider whether exact prompt and profile inspection APIs already provide a
smaller sufficient surface before adding an engine-wide operation.
### Value to Weatherreporter
This would simplify offline corpus checks and catch malformed operator profile
sources before report work begins. It is helpful but lower priority than exact
prompt and profile inspection.
## Priority 5: Structured Generation Errors
**Disposition:** Deferred pending stronger downstream demand and a narrower
design that does not duplicate prepared provenance or impose HTTP-specific
fields on injected model clients.
### Downstream need
Weatherreporter preserves redacted, inspectable failure receipts for report
runs. When model generation fails operationally, it needs to classify the
failure and retain safe execution context without parsing error prose.
Prompt preparation already supplies selected profile, backend, and model
identity. Provider status classification would add useful operator context,
especially when the built-in OpenAI-compatible client receives a non-success
HTTP status.
### Current integration option
PromptKit exposes `ErrLLMGenerate` and preserves injected client errors through
`errors.Is`. Weatherreporter can reliably classify generation failure and use
its preparation record for profile, backend, and model provenance. Any further
diagnostic detail remains a redacted error string.
### Requested capability
Consider a typed generation error that continues to match `ErrLLMGenerate`:
```go
type GenerationError struct {
BackendID string
Model string
StatusCode int
}
```
The exact fields may differ. The useful contract is safe structured context
available through `errors.As`, while `errors.Is(err, ErrLLMGenerate)` remains
compatible.
### Design considerations
- Include only fields that PromptKit knows reliably and can expose safely.
- Treat an HTTP status as optional because injected model clients may not use
HTTP.
- Do not expose provider response bodies, endpoints, credential environment
names, credential values, request content, or generated content.
- Do not make a structured error a second source of prompt/profile provenance
already present in a prepared execution.
- Preserve injected client error identity.
- Keep retry and backoff policy with the consuming application.
### Value to Weatherreporter
This would improve durable failure receipts and troubleshooting, particularly
for built-in transport failures. It is not required if preparation details and
the existing sentinel remain available.
## Lower-Priority Shared Wishlist Items
### Structured Capacity Errors
**Disposition:** Accepted into the
[future catalog](future.md#structured-capacity-errors).
The typed capacity error proposed by the
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-4-structured-capacity-errors)
would improve Weatherreporter diagnostics by exposing the stable backend ID
without parsing error text.
Weatherreporter currently generates batch reports sequentially and constructs
one engine per invocation, so engine-local capacity exhaustion is unlikely in
the initial design. The feature would become more valuable if report
generation later becomes concurrent or PromptKit engines become longer-lived.
It should not block adoption.
### Semantic Execution-Target Fingerprints
**Disposition:** Deferred until prompt-independent profile inspection defines
the resolved target whose configuration identity would be fingerprinted.
The semantic target digest proposed by the
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-3-semantic-execution-target-fingerprints)
would provide a compact equality signal for audit metadata.
Weatherreporter does not currently reuse LLM-dependent checkpoints. Its Recent
Changes behavior compares deterministic module snapshots rather than generated
reports, so the digest has no immediate cache-correctness role. Existing
PromptKit result metadata is sufficient for the initial integration. A digest
would still be useful provenance and future-proofing, but it is not a
migration priority.
## Capabilities PromptKit Already Provides Well
PromptKit v0.3.0 already provides the essential Weatherreporter integration
surface:
- importable in-process engine construction;
- filesystem, `fs.FS`, and programmatic prompt, profile, and schema sources;
- offline preparation without model execution;
- versioned prompt selection;
- text, Markdown, JSON, and JSON Schema output contracts;
- single-pass output validation with raw output retained after completed
validation failure;
- inline input artifacts with provenance URIs and input hashes;
- selected profile, backend, model, effective target, prompt hashes, timing,
and token-usage provenance;
- endpoint-only profiles and engine-scoped backend registration;
- injected model-client and artifact-reader interfaces;
- caller cancellation, generation timeout, and transport timeout behavior; and
- public error sentinels for configuration, prompt, profile, artifact,
validation, capacity, and generation failures.
These capabilities are sufficient for Weatherreporter to adopt PromptKit
without waiting for new upstream work.
## Responsibilities That Should Remain In Weatherreporter
The following concerns belong to Weatherreporter and should not move into
PromptKit:
- report definitions, valid periods, batches, and output naming;
- application prompt content and private report response schemas;
- deterministic weather facts, modules, and Recent Changes;
- curated `data_package` construction and persistence;
- generated-text domain validation and Markdown template rendering;
- managed artifact paths, atomic writes, metadata, and inspection commands;
- preparation, execution, raw-output, and failure-receipt schemas;
- CLI configuration loading and precedence;
- debug enablement, redaction, placement, sensitivity, and retention;
- distributor notification;
- batch continuation and any future retry policy; and
- application-level compatibility and migration policy.
## Suggested Upstream Sequence
If the PromptKit team chooses to pursue these ideas, the most useful order for
Weatherreporter would be:
1. Add executable preparation handles, ideally sharing implementation with an
atomic detailed-run API.
2. Add prompt-definition inspection.
3. Add prompt-independent profile inspection.
4. Consider eager source validation after evaluating whether the two exact
inspection APIs are sufficient.
5. Add structured generation errors.
6. Add structured capacity errors and semantic execution-target fingerprints
as lower-priority operational improvements.
The first item removes the only material integration workaround. Prompt and
profile inspection improve fail-fast validation. The remaining items improve
ergonomics and diagnostics.
## Adoption Sequencing
Weatherreporter should not wait for the complete wishlist. PromptKit v0.3.0 is
already sufficient when Weatherreporter:
- embeds immutable prompt and schema assets;
- supplies immutable inline data-package bytes;
- constructs one engine per CLI invocation;
- calls `Prepare` and `Run` with the same request; and
- keeps PromptKit behind a weatherreporter-owned adapter contract.
If executable preparation handles are scheduled for a near-term PromptKit
release, Weatherreporter may defer only its final adapter implementation to
avoid implementing and then removing duplicate preparation. Prompt corpus
retrieval, application-contract design, configuration work, embedded assets,
state contracts, and offline fixtures can proceed independently.
If the feature is not scheduled, Weatherreporter can adopt v0.3.0 and keep the
duplicate `Prepare` and `Run` sequence inside its adapter. A later PromptKit
upgrade would remain localized behind that neutral boundary.
Prompt inspection, profile inspection, source validation, structured errors,
capacity details, and semantic fingerprints should not gate adoption.