Prepare documentation for Promptkit v0.4.0
This commit is contained in:
@@ -33,6 +33,9 @@ boundary and constraints that framework work must preserve.
|
||||
|
||||
## Release Guidance
|
||||
|
||||
Consumers upgrading from `v0.3.0` to `v0.4.0` should read the
|
||||
[v0.4.0 changelog and adoption guide](docs/releases/v0.4.0.md).
|
||||
|
||||
Consumers upgrading from `v0.2.0` to `v0.3.0` should read the
|
||||
[v0.3.0 changelog](docs/releases/v0.3.0.md).
|
||||
|
||||
|
||||
189
docs/releases/v0.4.0.md
Normal file
189
docs/releases/v0.4.0.md
Normal file
@@ -0,0 +1,189 @@
|
||||
# Promptkit v0.4.0
|
||||
|
||||
This supplemental changelog and adoption guide summarizes the consumer-facing
|
||||
changes from `v0.3.0` to `v0.4.0`. The annotated `v0.4.0` tag is the
|
||||
authoritative release record. Exact current contracts belong to the linked
|
||||
GoDoc and durable documentation.
|
||||
|
||||
## Summary
|
||||
|
||||
`v0.4.0` adds four complementary capabilities:
|
||||
|
||||
- opaque prepared-execution handles for preparing once, inspecting safe
|
||||
details, and executing the same frozen snapshot;
|
||||
- exact prompt-definition inspection without profile resolution or execution;
|
||||
- exact profile inspection without selecting a prompt or checking credential
|
||||
availability; and
|
||||
- structured backend identity on engine admission-capacity rejection.
|
||||
|
||||
These APIs let consumers perform more precise preflight work and retain useful
|
||||
operational context without reproducing Promptkit's internal resolution logic.
|
||||
|
||||
## Compatibility
|
||||
|
||||
The release is additive for `v0.3.0` consumers. Existing uses of `Prepare`,
|
||||
`Run`, backend registration, endpoint-only profiles, local-backend helpers,
|
||||
runtime overrides, public JSON values, and error sentinels continue to work
|
||||
without migration.
|
||||
|
||||
Capacity rejection now returns a structured error while continuing to match
|
||||
`ErrCapacityExceeded` through `errors.Is`. Error-string wording and direct
|
||||
sentinel equality were not public contracts.
|
||||
|
||||
The new inspection values, capacity error, and prepared-execution handle do not
|
||||
have stable JSON representations. `PreparedExecution.Details` returns the
|
||||
existing stable `PreparedRun` value.
|
||||
|
||||
## Upgrade
|
||||
|
||||
Update the module dependency with:
|
||||
|
||||
```sh
|
||||
go get gitea.maximumdirect.net/eric/promptkit@v0.4.0
|
||||
go mod tidy
|
||||
```
|
||||
|
||||
Run the consuming project's ordinary and race-enabled tests after upgrading.
|
||||
No source migration is required.
|
||||
|
||||
## Prepare Once And Execute The Same Snapshot
|
||||
|
||||
Consumers that need to persist preparation details before generation can now
|
||||
prepare an opaque, engine-bound execution:
|
||||
|
||||
```go
|
||||
prepared, err := engine.PrepareExecution(ctx, request)
|
||||
if err != nil {
|
||||
// Handle preparation failure.
|
||||
}
|
||||
defer prepared.Discard()
|
||||
|
||||
details := prepared.Details()
|
||||
// Persist a consumer-selected, appropriately protected preparation record.
|
||||
|
||||
result, err := engine.RunPrepared(ctx, prepared)
|
||||
```
|
||||
|
||||
Preparation freezes the selected sources, rendered messages, effective
|
||||
settings, input content, structured-output metadata, and validation resources
|
||||
needed by execution. `Details` returns a fresh, caller-owned,
|
||||
credential-redacted `PreparedRun`.
|
||||
|
||||
A handle belongs to its creating engine and permits one execution attempt.
|
||||
`RunPrepared` consumes that attempt on success and on operational failure.
|
||||
`Discard` is idempotent and releases an unclaimed handle's execution-only
|
||||
state. Consumers should discard handles they will not execute, particularly
|
||||
when a direct request API key may be retained privately until claim or
|
||||
discard.
|
||||
|
||||
Prepared execution does not reserve backend admission during preparation.
|
||||
Credential availability and backend admission are checked when execution
|
||||
begins. The execution context is independent of the preparation context.
|
||||
|
||||
See the
|
||||
[prepared-execution consumer guide](../consumers/pkg-promptkit.md#prepare-now-and-execute-the-same-snapshot-later),
|
||||
the [`PreparedExecution` GoDoc](../../prepared_execution.go), and the
|
||||
[`Engine.PrepareExecution` and `Engine.RunPrepared` GoDoc](../../engine.go)
|
||||
for the exact lifecycle, ownership, cancellation, capacity, timing, and
|
||||
failure contracts.
|
||||
|
||||
## Inspect A Prompt
|
||||
|
||||
`Engine.InspectPrompt` resolves one prompt ID and optional version through the
|
||||
engine's configured prompt source:
|
||||
|
||||
```go
|
||||
inspection, err := engine.InspectPrompt(ctx, "report.summary", "")
|
||||
```
|
||||
|
||||
The result includes prompt identity, the opaque prompt hash, declared default
|
||||
profile ID, declared input metadata, and normalized output contract. It
|
||||
structurally loads the selected definition and referenced message content but
|
||||
does not resolve a profile, load schemas or artifacts, render templates,
|
||||
reserve capacity, or contact a model.
|
||||
|
||||
Use inspection for exact configuration checks and metadata discovery. Use
|
||||
`PrepareExecution` rather than relying on a prior inspection when later
|
||||
execution must freeze one exact source state, because filesystem-backed
|
||||
inspection is only a point-in-time lookup.
|
||||
|
||||
See the
|
||||
[prompt-inspection consumer guide](../consumers/pkg-promptkit.md#inspect-a-prompt-before-preparation)
|
||||
and [`Engine.InspectPrompt` GoDoc](../../engine.go) for exact selection,
|
||||
ownership, and error behavior.
|
||||
|
||||
## Inspect A Profile
|
||||
|
||||
`Engine.InspectProfile` resolves one explicit profile independently of a
|
||||
prompt:
|
||||
|
||||
```go
|
||||
inspection, err := engine.InspectProfile(ctx, "report-production")
|
||||
```
|
||||
|
||||
The result includes the resolved effective execution target and whether a
|
||||
later request must provide a direct credential. Environment-variable names may
|
||||
be reported, but inspection does not read credential values or require the
|
||||
named variable to be populated.
|
||||
|
||||
Inspection applies the ordinary configured and built-in profile precedence and
|
||||
resolves any selected backend. It does not load a prompt, render content,
|
||||
reserve capacity, or contact a model.
|
||||
|
||||
See the
|
||||
[profile-inspection consumer guide](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work)
|
||||
and [`Engine.InspectProfile` GoDoc](../../engine.go) for the exact resolution,
|
||||
credential, ownership, and error contracts.
|
||||
|
||||
## Identify Capacity-Rejected Backends
|
||||
|
||||
Calls rejected at Promptkit's bounded engine admission boundary continue to
|
||||
match `ErrCapacityExceeded`. Consumers can additionally obtain the selected
|
||||
registered backend ID without parsing diagnostic text:
|
||||
|
||||
```go
|
||||
result, err := engine.Run(ctx, request)
|
||||
if errors.Is(err, promptkit.ErrCapacityExceeded) {
|
||||
var capacityErr *promptkit.CapacityError
|
||||
if errors.As(err, &capacityErr) {
|
||||
// Record capacityErr.BackendID using application-owned diagnostics.
|
||||
}
|
||||
|
||||
// Apply application-owned overload or retry policy.
|
||||
}
|
||||
```
|
||||
|
||||
The structured error applies to `Run` and `RunPrepared` admission rejection.
|
||||
It does not represent provider throttling, quota exhaustion, cancellation
|
||||
while waiting for generation capacity, or another model-client failure.
|
||||
Promptkit does not prescribe retry timing or transport status mapping.
|
||||
|
||||
See the
|
||||
[error-handling consumer guide](../consumers/pkg-promptkit.md#handle-errors),
|
||||
the [`CapacityError` GoDoc](../../capacity_error.go), and the
|
||||
[`ErrCapacityExceeded` GoDoc](../../engine.go) for the canonical contracts.
|
||||
|
||||
## Public API Additions
|
||||
|
||||
The release adds:
|
||||
|
||||
- `Engine.PrepareExecution`;
|
||||
- `Engine.RunPrepared`;
|
||||
- `PreparedExecution`, including `Details`, `Discard`, `String`, and
|
||||
`GoString`;
|
||||
- `Engine.InspectPrompt`;
|
||||
- `PromptInspection`;
|
||||
- `PromptInputDefinition`;
|
||||
- `Engine.InspectProfile`;
|
||||
- `ProfileInspection`; and
|
||||
- `CapacityError`.
|
||||
|
||||
No public API was removed.
|
||||
|
||||
## Consumer Action
|
||||
|
||||
None. Existing `v0.3.0` workflows may upgrade without adopting the new APIs.
|
||||
|
||||
Consumers that adopt prepared execution should discard unused handles.
|
||||
Consumers that need backend-specific capacity diagnostics may add an
|
||||
`errors.As` check while retaining their existing `errors.Is` classification.
|
||||
@@ -1,262 +0,0 @@
|
||||
# Structured Capacity Errors
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Purpose
|
||||
|
||||
Allow consumers to identify which registered backend rejected engine
|
||||
admission without parsing an error string, while preserving the existing
|
||||
`ErrCapacityExceeded` classification and Promptkit's provider-neutral error
|
||||
boundary.
|
||||
|
||||
This provides safe operational context requested by
|
||||
[Notarius](notarius-promptkit-wishlist.md#priority-4-structured-capacity-errors)
|
||||
and
|
||||
[Weatherreporter](weatherreporter-promptkit-wishlist.md#structured-capacity-errors).
|
||||
|
||||
## Motivation
|
||||
|
||||
Promptkit currently returns an error matching `ErrCapacityExceeded` when a
|
||||
limited backend's active and waiting admission capacity is full. That sentinel
|
||||
lets consumers classify the failure reliably, but the selected backend ID is
|
||||
available only in diagnostic text.
|
||||
|
||||
Applications using multiple backends need the stable backend ID for
|
||||
operator-facing diagnostics, metrics labels, failure receipts, and their own
|
||||
retry decisions. Parsing `Error()` text would turn non-contractual wording
|
||||
into an accidental API and risks retaining more diagnostic context than the
|
||||
application needs.
|
||||
|
||||
Promptkit already owns backend selection and the admission boundary. It should
|
||||
attach the selected backend's safe identity at that boundary and translate it
|
||||
into a small public typed error.
|
||||
|
||||
## Consumer Workflow
|
||||
|
||||
The target public workflow is:
|
||||
|
||||
```go
|
||||
result, err := engine.Run(ctx, request)
|
||||
if err != nil {
|
||||
var capacityErr *promptkit.CapacityError
|
||||
if errors.As(err, &capacityErr) {
|
||||
// Record capacityErr.BackendID or apply application retry policy.
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Existing classification remains valid:
|
||||
|
||||
```go
|
||||
if errors.Is(err, promptkit.ErrCapacityExceeded) {
|
||||
// Handle engine admission rejection without structured context.
|
||||
}
|
||||
```
|
||||
|
||||
The target public surface is:
|
||||
|
||||
```go
|
||||
type CapacityError struct {
|
||||
BackendID string
|
||||
}
|
||||
|
||||
func (e *CapacityError) Error() string
|
||||
func (e *CapacityError) Unwrap() error
|
||||
```
|
||||
|
||||
`Unwrap` returns `ErrCapacityExceeded`. Engine-produced capacity errors
|
||||
therefore support both `errors.As` to `*CapacityError` and
|
||||
`errors.Is(err, ErrCapacityExceeded)`, including through additional wrapping.
|
||||
|
||||
The exported declaration and GoDoc will own the exact implemented contract.
|
||||
`CapacityError` does not need a stable JSON representation, an exported
|
||||
constructor, or fields beyond `BackendID`.
|
||||
|
||||
## Error Boundary
|
||||
|
||||
A `CapacityError` represents one specific condition: `Run` or `RunPrepared`
|
||||
was rejected at Promptkit's engine-local bounded admission boundary because
|
||||
the selected limited backend had no remaining admitted slot.
|
||||
|
||||
It does not represent:
|
||||
|
||||
- waiting for an active-generation permit after admission;
|
||||
- caller cancellation or deadline expiration while acquiring or holding
|
||||
capacity;
|
||||
- provider rate limiting, quota exhaustion, HTTP 429 responses, or another
|
||||
model-client failure;
|
||||
- validation, output repair, or application-level queue rejection; or
|
||||
- an application's decision not to submit or retry work.
|
||||
|
||||
Provider and injected-client errors retain their existing generation error
|
||||
classification even when a provider describes its failure as capacity or rate
|
||||
limiting. Promptkit must not infer an engine admission error from provider
|
||||
status codes or error text.
|
||||
|
||||
Admission continues to check caller cancellation before reporting a full
|
||||
pool. A cancellation that wins that decision remains a context-related
|
||||
execution error and does not become a `CapacityError`.
|
||||
|
||||
## Backend Identity
|
||||
|
||||
`BackendID` is the normalized stable ID of the registered backend whose pool
|
||||
rejected admission. It is the same effective backend identity used for
|
||||
capacity routing and exposed by prepared and run metadata.
|
||||
|
||||
For ordinary `Run`, the ID comes from the backend selected by the resolved
|
||||
prompt, profile, and execution target. For `RunPrepared`, it comes from the
|
||||
backend identity frozen in the prepared execution.
|
||||
|
||||
An endpoint override does not change backend identity and therefore does not
|
||||
change the reported `BackendID`. Endpoint-only profiles and unlimited
|
||||
backends have no bounded admission pool and cannot produce an engine-generated
|
||||
`CapacityError`.
|
||||
|
||||
Every engine-produced `CapacityError` has a nonblank registered backend ID.
|
||||
The type does not report a profile ID, prompt ID, model, endpoint, run ID, or
|
||||
prepared-handle identity.
|
||||
|
||||
## Compatibility
|
||||
|
||||
`ErrCapacityExceeded` remains the stable sentinel for broad classification.
|
||||
This feature refines the error value returned by capacity-rejected `Run` and
|
||||
`RunPrepared` calls without replacing or removing that sentinel.
|
||||
|
||||
The following behaviors remain unchanged:
|
||||
|
||||
- `errors.Is(err, ErrCapacityExceeded)` succeeds for engine admission
|
||||
rejection;
|
||||
- capacity rejection remains distinct from `ErrInvalidRequest`,
|
||||
`ErrLLMGenerate`, and other public categories;
|
||||
- `Run` returns no partial result after admission rejection;
|
||||
- `RunPrepared` retains its existing one-attempt lifecycle, including that a
|
||||
claimed handle is consumed when admission is rejected;
|
||||
- admission ordering, limits, queue capacity, leases, release behavior, and
|
||||
backend independence are unchanged; and
|
||||
- `Prepare`, `PrepareExecution`, `InspectPrompt`, and `InspectProfile` perform
|
||||
no admission and cannot return this error.
|
||||
|
||||
Direct equality with the sentinel is not introduced as a contract. Consumers
|
||||
continue to use `errors.Is` for category checks and `errors.As` for structured
|
||||
context.
|
||||
|
||||
## Sensitive Data And Diagnostics
|
||||
|
||||
The typed error contains only `BackendID`. It must not expose or retain:
|
||||
|
||||
- backend endpoints;
|
||||
- credential environment-variable names or credential values;
|
||||
- model, profile, prompt, session, or request identifiers;
|
||||
- prompt, input, rendered, generated, or validation content;
|
||||
- concurrency limits, queue capacities, current counts, or queue depth;
|
||||
- request position or identities of other admitted work; or
|
||||
- retry timing, provider health, or speculative availability.
|
||||
|
||||
`Error()` may include the quoted backend ID and a concise admission-rejection
|
||||
description. Its wording is diagnostic and not a stable parsing contract.
|
||||
Default formatting, wrapping, and Go string formatting must not reveal
|
||||
anything beyond the public field and generic capacity classification.
|
||||
|
||||
Concurrency and queue configuration remain available where consumers define
|
||||
their backends; repeating them on each error would add stale or unnecessary
|
||||
operational detail. Current counts are race-prone observations rather than a
|
||||
durable explanation of when capacity will become available.
|
||||
|
||||
## Retry And Application Policy
|
||||
|
||||
Promptkit reports that admission was rejected but does not retry, wait for a
|
||||
new admission slot, calculate backoff, or declare an HTTP status.
|
||||
|
||||
A full in-process admission pool does not provide a meaningful retry delay:
|
||||
lease duration depends on consumer inputs, model latency, validation, repair,
|
||||
cancellation, and provider behavior. Consumers own whether to retry, which
|
||||
backend to use, how to schedule or back off, what to expose to operators, and
|
||||
how to translate the error into CLI or transport behavior.
|
||||
|
||||
The absence of retry metadata is deliberate. A future retry facility would
|
||||
require its own roadmap and application-neutral policy rather than extending
|
||||
this error speculatively.
|
||||
|
||||
## Ownership And Concurrency
|
||||
|
||||
Each rejected operation returns its own caller-owned `*CapacityError`. The
|
||||
error does not reference mutable pool state or retain the request. Reading it
|
||||
requires no engine access and remains safe after the rejected call returns.
|
||||
|
||||
Mutating the exported `BackendID` on a returned error affects only that error
|
||||
value. It cannot change the backend registry, capacity manager, another
|
||||
operation's error, or later routing and admission.
|
||||
|
||||
Constructing or returning the typed error requires no new process-global
|
||||
state, cache, goroutine, or synchronization beyond existing capacity
|
||||
management.
|
||||
|
||||
## Architectural Boundaries
|
||||
|
||||
The root `promptkit` package owns the exported error type, sentinel
|
||||
compatibility, and public translation. The engine's internal capacity and
|
||||
use-case components remain responsible for admission and for carrying the
|
||||
selected backend identity to the facade.
|
||||
|
||||
Internal error types or capacity-manager state must not cross the public
|
||||
boundary. The public facade must obtain structured identity through an
|
||||
internal typed or otherwise non-textual error contract; it must not parse the
|
||||
runner's diagnostic string.
|
||||
|
||||
The feature does not introduce a public capacity manager, backend registry,
|
||||
queue, scheduler, metrics collector, or transport-specific error. It remains a
|
||||
small refinement of the existing root error boundary.
|
||||
|
||||
## Documentation
|
||||
|
||||
When implemented, documentation should retain these ownership boundaries:
|
||||
|
||||
- the exported declaration and GoDoc own the exact type, field, `Error`,
|
||||
`Unwrap`, `errors.Is`, `errors.As`, and formatting contracts;
|
||||
- the consumer guide demonstrates broad classification and optional backend
|
||||
extraction without duplicating every error guarantee;
|
||||
- internal capacity and runner documentation describe where backend identity
|
||||
is attached and translated;
|
||||
- the backend API continues to own capacity configuration; and
|
||||
- the architecture and documentation policies continue to own library,
|
||||
application, and documentation boundaries.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This work does not include:
|
||||
|
||||
- changing concurrency limits, queue capacities, FIFO scheduling, or default
|
||||
capacity policy;
|
||||
- exposing configured limits, current counts, queue depth, or waiter
|
||||
positions;
|
||||
- reporting a retry delay, retryable boolean, HTTP status, CLI exit code, or
|
||||
backoff policy;
|
||||
- automatically retrying or rerouting rejected work;
|
||||
- unifying provider rate limits or quota failures with engine admission;
|
||||
- adding structured context to generation, validation, prompt, profile, or
|
||||
other error categories;
|
||||
- returning partial preparation or run results with the error;
|
||||
- changing prepared-execution handle lifecycle;
|
||||
- adding stable JSON for errors;
|
||||
- exposing internal capacity or use-case types; or
|
||||
- adding application logging, metrics emission, persistence, or failure
|
||||
receipts.
|
||||
|
||||
## Target End State
|
||||
|
||||
After this work:
|
||||
|
||||
- every engine admission rejection from `Run` or `RunPrepared` returns an
|
||||
error discoverable as `*CapacityError`;
|
||||
- the typed error contains the normalized selected backend ID and no other
|
||||
structured field;
|
||||
- existing `errors.Is(err, ErrCapacityExceeded)` handling continues to work;
|
||||
- `errors.As` obtains backend identity without parsing diagnostic text;
|
||||
- capacity rejection remains distinct from cancellation, model-client
|
||||
failures, provider throttling, and application scheduling;
|
||||
- endpoints, credentials, request content, capacity metrics, and speculative
|
||||
retry information remain absent;
|
||||
- retry, backoff, rerouting, metrics, persistence, and transport mapping remain
|
||||
consumer responsibilities; and
|
||||
- the existing capacity architecture and runtime behavior are otherwise
|
||||
unchanged.
|
||||
@@ -1,237 +0,0 @@
|
||||
# Backend-Specific Concurrency Management
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap defines the scope and target end state for engine-local,
|
||||
backend-specific concurrency management. It records the intended capability,
|
||||
consumer value, and important policy choices.
|
||||
|
||||
This document is planning material, not a description of current behavior.
|
||||
Current exported contracts remain owned by Go declarations and GoDoc, backend
|
||||
registration guidance by the
|
||||
[consumer guide](../consumers/pkg-promptkit.md#configure-a-local-openai-compatible-endpoint),
|
||||
and
|
||||
implemented orchestration by the
|
||||
[internal runner document](../internal/runner.md).
|
||||
|
||||
## Motivation
|
||||
|
||||
Different model backends can sustain very different request loads. A local
|
||||
network endpoint may need a small concurrency limit, while OpenRouter can
|
||||
usually accept substantially more simultaneous work. Requiring every consumer
|
||||
to build its own semaphores and queues would duplicate routing knowledge,
|
||||
create inconsistent cancellation behavior, and make it easy for one caller to
|
||||
bypass the intended backend limit.
|
||||
|
||||
Promptkit should own this coordination because it already resolves each run to
|
||||
an engine-scoped backend identity and owns every model-generation call made by
|
||||
the runner. Consumers should continue submitting ready-to-run requests through
|
||||
the synchronous API, including concurrently from multiple goroutines, without
|
||||
implementing their own backend scheduler.
|
||||
|
||||
The buffered queue is a safety boundary, not an ordinary throughput
|
||||
restriction. Its primary purpose is to prevent a bug or unintended submission
|
||||
loop from creating an unbounded in-memory backlog.
|
||||
|
||||
## Scope
|
||||
|
||||
The feature will add optional concurrency policy to registered backends and
|
||||
coordinate `Run` calls against independent per-backend capacity pools.
|
||||
|
||||
Each policy has two distinct controls:
|
||||
|
||||
- an active-generation limit, which protects the backend from too many
|
||||
simultaneous model requests; and
|
||||
- a bounded waiting capacity, which protects the process from admitting an
|
||||
unbounded backlog.
|
||||
|
||||
Concurrency policy belongs to a backend registration. It is not a profile
|
||||
model parameter and cannot be overridden per run. Profiles select the policy
|
||||
through their backend ID, while a profile or request endpoint override remains
|
||||
in the selected backend's pool.
|
||||
|
||||
`Prepare` does not call a model and will remain outside concurrency admission.
|
||||
|
||||
## Defaults And Configuration
|
||||
|
||||
The built-in OpenRouter backend will use:
|
||||
|
||||
- an active-generation limit of 16; and
|
||||
- a waiting capacity of 1024.
|
||||
|
||||
The waiting default is intentionally generous. Reaching it should indicate
|
||||
abnormal submission pressure rather than normal application behavior.
|
||||
|
||||
Consumer-registered backends will remain unlimited unless the consumer
|
||||
configures an active-generation limit. When a consumer enables a limit and
|
||||
does not specify waiting capacity, the waiting capacity will default to 1024.
|
||||
Consumers may configure a different bounded capacity, including zero when
|
||||
they want no admitted backlog beyond the active-limit-sized run set.
|
||||
|
||||
The public representation must distinguish an omitted waiting capacity from
|
||||
an explicit zero.
|
||||
|
||||
Endpoint-only profiles have no backend registration from which to obtain
|
||||
policy and will remain unlimited. A future engine-wide or endpoint-keyed
|
||||
policy can be considered separately if consumers demonstrate that need.
|
||||
|
||||
Invalid limits or capacities will fail engine construction as invalid
|
||||
configuration. Policy values will be copied into engine-owned immutable state
|
||||
along with the rest of the backend registration.
|
||||
|
||||
## Admission And Execution Behavior
|
||||
|
||||
`Run` remains a synchronous, wait-for-result operation. Concurrent callers may
|
||||
block inside `Run` while waiting for their selected backend, then receive the
|
||||
ordinary result or error from that invocation.
|
||||
|
||||
For a configured pool, the active-generation limit plus the waiting capacity
|
||||
defines the maximum number of concurrent `Run` invocations that Promptkit will
|
||||
accept for that backend. A waiting capacity of zero therefore accepts no more
|
||||
runs than the active limit. Admission is immediate: a call either reserves one
|
||||
of those bounded slots or receives the capacity error. An accepted run may
|
||||
then wait internally for active-generation capacity.
|
||||
|
||||
For a limited backend, Promptkit will bound the number of accepted runs before
|
||||
expensive artifact loading, prompt rendering, and large defensive copies where
|
||||
practical. Lightweight prompt, profile, and backend resolution may occur first
|
||||
when it is required to identify the correct capacity pool. This pre-admission
|
||||
resolution must not become a second execution-precedence path with behavior
|
||||
that can drift from `Prepare`.
|
||||
|
||||
An accepted run retains its admission until it completes or fails. Every
|
||||
actual model-generation call for that run must separately observe the
|
||||
backend's active-generation limit. This includes:
|
||||
|
||||
- the initial generation;
|
||||
- every output-repair generation; and
|
||||
- calls made through either the built-in or an injected model client.
|
||||
|
||||
Preparation and output validation should not hold an active-generation permit.
|
||||
A repair remains part of its already-admitted run, but reacquires active
|
||||
generation capacity so repairs cannot exceed the backend limit. It must not be
|
||||
rejected merely because new runs filled the waiting queue after its initial
|
||||
generation.
|
||||
|
||||
Within one backend pool, waiting generation calls should be served in FIFO
|
||||
order, subject to canceled calls being removed. Different backend pools make
|
||||
progress independently; a saturated local backend must not consume
|
||||
OpenRouter's active or waiting capacity.
|
||||
|
||||
The feature will not promise ordering across backend pools or completion order
|
||||
among admitted runs.
|
||||
|
||||
## Capacity Failure And Cancellation
|
||||
|
||||
When a backend's bounded waiting capacity is full, a new `Run` call will fail
|
||||
promptly rather than waiting outside the bounded admission system. The public
|
||||
API will expose a recognizable capacity-exhaustion error identity distinct
|
||||
from invalid configuration, invalid requests, and model-client failures.
|
||||
Rejected calls return no partial result and do not invoke the model client.
|
||||
|
||||
Waiting within the admitted backlog or for active-generation capacity must
|
||||
honor the caller's context. Cancellation or deadline expiry will:
|
||||
|
||||
- stop waiting promptly;
|
||||
- release any admission or generation capacity held by that invocation;
|
||||
- preserve the applicable context error identity; and
|
||||
- avoid invoking the model client if cancellation wins before generation
|
||||
starts.
|
||||
|
||||
Capacity must also be released after preparation, generation, validation,
|
||||
repair, or collaborator failure. One failed or canceled run must not reduce
|
||||
the backend's future usable capacity.
|
||||
|
||||
Elapsed `Run` timing will include time spent waiting after the call is
|
||||
accepted. `PreparedRun` timing will continue to describe preparation rather
|
||||
than queue waiting.
|
||||
|
||||
## Engine And Client Boundaries
|
||||
|
||||
All pools and queued state belong to one `Engine`. Separate engines do not
|
||||
share capacity, even when they register the same backend ID or endpoint. The
|
||||
feature introduces no process-global scheduler.
|
||||
|
||||
The engine will apply policy consistently to the built-in model client and an
|
||||
injected `LLMClient`. Consumers calling their own client outside Promptkit are
|
||||
outside this boundary. Injected clients remain responsible for their internal
|
||||
thread safety and cancellation behavior.
|
||||
|
||||
Backend policy is keyed by the resolved backend ID rather than endpoint text.
|
||||
This preserves stable routing when a selected backend's endpoint is overridden
|
||||
and avoids accidentally combining unrelated registrations that happen to use
|
||||
the same URL.
|
||||
|
||||
## Queue Lifetime And Observability
|
||||
|
||||
Admission state is buffered, ephemeral, and in-process. It is not persisted
|
||||
and has no survival guarantee across engine disposal or process termination.
|
||||
Promptkit will not introduce background job ownership or require consumers to
|
||||
start or stop workers.
|
||||
|
||||
The initial feature does not require public queue-depth metrics, callbacks, or
|
||||
inspection APIs. Capacity errors and ordinary call timing provide the
|
||||
consumer-visible behavior. Operational observability can be added later
|
||||
without coupling the scheduling mechanism to an application logging or
|
||||
metrics system.
|
||||
|
||||
## Compatibility
|
||||
|
||||
Consumer-registered backends and endpoint-only profiles remain unlimited
|
||||
unless concurrency is explicitly configured, preserving their existing
|
||||
behavior.
|
||||
|
||||
The built-in OpenRouter backend will change from unlimited concurrency to a
|
||||
limit of 16 with a bounded waiting capacity of 1024. Ordinary synchronous
|
||||
calls remain unchanged, while unusually high concurrent use may now wait or
|
||||
return the capacity error. This behavioral change must be identified in the
|
||||
release notes for the version that publishes it.
|
||||
|
||||
Adding backend policy fields and a public capacity error is otherwise
|
||||
additive. The change will use a pre-`v1` minor release under Promptkit's
|
||||
[release policy](../release.md#release-model).
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This scope does not include:
|
||||
|
||||
- asynchronous job handles, polling, or detached result delivery;
|
||||
- durable or cross-process queues;
|
||||
- persistence or recovery across engine or process shutdown;
|
||||
- priorities, scheduling weights, or consumer-defined fairness classes;
|
||||
- automatic retries, backoff, rate-limit interpretation, or provider quota
|
||||
discovery;
|
||||
- token-per-minute or request-per-minute rate limiting;
|
||||
- dynamic reconfiguration after engine construction;
|
||||
- per-profile or per-run concurrency overrides;
|
||||
- endpoint-keyed pooling for profiles without a backend ID;
|
||||
- process-global coordination across engines;
|
||||
- application worker lifecycle, logging, tracing, or metrics policy; or
|
||||
- changes to prompt, profile, schema, or model-provider wire formats.
|
||||
|
||||
## Target End State
|
||||
|
||||
This roadmap reaches its target end state when:
|
||||
|
||||
- each engine independently coordinates configured backend capacity;
|
||||
- the built-in OpenRouter backend allows 16 active generations and up to 1024
|
||||
waiting runs;
|
||||
- consumer backends can opt into their own active and waiting limits while
|
||||
remaining unlimited by default;
|
||||
- endpoint overrides retain the selected backend's capacity pool and
|
||||
endpoint-only profiles remain unlimited;
|
||||
- synchronous `Run` callers wait for and receive their ordinary result;
|
||||
- admission is bounded before expensive preparation work where practical;
|
||||
- every initial and repair generation observes the backend's active limit
|
||||
without serializing preparation or validation;
|
||||
- a full waiting queue returns a recognizable capacity error without invoking
|
||||
the model client;
|
||||
- cancellation and all failure paths promptly release capacity and preserve
|
||||
context error identity;
|
||||
- built-in and injected model clients receive the same scheduling behavior;
|
||||
- pools remain ephemeral, engine-scoped, and independent across backend IDs;
|
||||
and
|
||||
- current-state GoDoc, consumer, internal, and release documentation describe
|
||||
the implemented behavior once it lands.
|
||||
@@ -1,520 +0,0 @@
|
||||
# Structured Capacity Errors Implementation Plan
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Purpose
|
||||
|
||||
This document is the decision-complete implementation plan for
|
||||
[structured capacity errors](capacity-errors.md). It is written for a
|
||||
gpt-5.6-terra coding agent that will implement each stage in order.
|
||||
|
||||
The feature roadmap owns the motivation, public intent, compatibility policy,
|
||||
security boundary, non-goals, and target end state. This plan owns the fixed
|
||||
design, file-level work, implementation sequence, test ownership,
|
||||
documentation updates, validation commands, and completion gates.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
- Complete the stages in order. Keep the repository compiling and the focused
|
||||
tests passing at every stage boundary.
|
||||
- Preserve unrelated working-tree changes. In particular, retain the accepted
|
||||
feature roadmap and its existing future-catalog and downstream-wishlist
|
||||
edits.
|
||||
- Follow every policy under `docs/policy/`, the task-specific reading guide in
|
||||
`docs/development.md`, and the accepted behavior in
|
||||
`capacity-errors.md`.
|
||||
- Keep the supported error API in the root `promptkit` package. Internal
|
||||
capacity and use-case error values must not become consumer dependencies.
|
||||
- Do not parse error strings. Carry the selected backend ID in an internal
|
||||
typed error and translate it explicitly at the root facade.
|
||||
- Do not change capacity policy, admission ordering, limits, queueing, FIFO
|
||||
generation scheduling, lease lifetime, cancellation behavior, backend
|
||||
selection, prepared-handle lifecycle, or model-client classification.
|
||||
- Do not classify provider throttling, HTTP 429 responses, quota failures, or
|
||||
injected-client errors as engine admission rejection.
|
||||
- Preserve `errors.Is(err, ErrCapacityExceeded)` while adding discovery as
|
||||
`*CapacityError` through `errors.As`.
|
||||
- Return a fresh public typed error for each rejected operation. Do not retain
|
||||
a request, pool, registry, endpoint, credential, execution target, or live
|
||||
capacity state in either typed error.
|
||||
- Add only `BackendID` to the public type. Do not add limits, counts, queue
|
||||
depth, retry timing, retryability, transport status, provider state, or
|
||||
stable JSON.
|
||||
- Keep tests lean and behavior-focused. Extend the existing internal admission,
|
||||
public capacity, prepared-execution, and root error-boundary tests instead of
|
||||
creating a parallel test framework.
|
||||
- Update exact public contracts in declarations and GoDoc. Update current-state
|
||||
consumer and internal documentation only after the implementation exists.
|
||||
- Do not add release notes, change a module version, create a release, commit,
|
||||
or tag as part of this work.
|
||||
|
||||
## Fixed Design
|
||||
|
||||
### Internal Admission Error
|
||||
|
||||
Add `internal/usecase/capacity_error.go` with this internal boundary type:
|
||||
|
||||
```go
|
||||
// CapacityError identifies bounded admission rejected for one selected
|
||||
// backend.
|
||||
type CapacityError struct {
|
||||
BackendID string
|
||||
}
|
||||
|
||||
func (e *CapacityError) Error() string
|
||||
func (e *CapacityError) Unwrap() error
|
||||
```
|
||||
|
||||
Although exported from an `internal` package so the root facade can recognize
|
||||
it, this is not a public consumer API. Its methods have these fixed semantics:
|
||||
|
||||
- `Error` returns concise diagnostic wording that includes a quoted nonblank
|
||||
backend ID and the generic internal capacity classification;
|
||||
- a nil receiver or blank `BackendID` produces only the generic internal
|
||||
capacity wording and does not panic;
|
||||
- `Unwrap` always returns `capacity.ErrCapacityExceeded`, including for a nil
|
||||
receiver; and
|
||||
- the value contains no cause field, request reference, capacity-manager
|
||||
reference, or other structured data.
|
||||
|
||||
Update `Runner.admitRun` in `internal/usecase/runner.go`:
|
||||
|
||||
1. retain the existing nil-admitter unlimited fallback;
|
||||
2. call `RunAdmitter.Admit` exactly once with the selected backend ID;
|
||||
3. when the returned error matches `capacity.ErrCapacityExceeded` and
|
||||
`backendID` is nonblank, return a newly allocated
|
||||
`&CapacityError{BackendID: backendID}`;
|
||||
4. when a capacity error is returned for a blank backend ID by an invalid or
|
||||
test-only collaborator, pass that error through rather than manufacturing a
|
||||
structured value that violates the nonblank-ID guarantee;
|
||||
5. pass every non-capacity error through unchanged; and
|
||||
6. preserve the release function unchanged on success.
|
||||
|
||||
Do not move structured identity into `internal/capacity.Manager`. The use-case
|
||||
boundary already knows the effective selected backend used by both `Run` and
|
||||
`RunPrepared`, and it also normalizes any conforming `RunAdmitter`
|
||||
implementation into the same error contract. The capacity package continues
|
||||
to own only its generic internal sentinel and scheduling state.
|
||||
|
||||
Both ordinary and prepared execution already call `admitRun`; do not add
|
||||
separate wrapping logic to `Run` or `RunPrepared`.
|
||||
|
||||
### Public Error Type
|
||||
|
||||
Add a root file named `capacity_error.go` containing:
|
||||
|
||||
```go
|
||||
// CapacityError reports bounded admission rejected for a selected backend.
|
||||
type CapacityError struct {
|
||||
BackendID string
|
||||
}
|
||||
|
||||
func (e *CapacityError) Error() string
|
||||
func (e *CapacityError) Unwrap() error
|
||||
```
|
||||
|
||||
The declaration and method GoDoc must establish:
|
||||
|
||||
- engine-produced values identify only rejection at Promptkit's bounded
|
||||
`Run` or `RunPrepared` admission boundary;
|
||||
- `BackendID` is the normalized registered backend ID used for routing and
|
||||
capacity, and endpoint overrides do not change it;
|
||||
- every engine-produced value is nonnil and has a nonblank `BackendID`;
|
||||
- provider errors, active-generation waiting, and caller cancellation are not
|
||||
represented by this type;
|
||||
- `Error` wording is diagnostic and not a parsing contract;
|
||||
- `Unwrap` returns `ErrCapacityExceeded`, so `errors.Is` and `errors.As` can be
|
||||
used together;
|
||||
- a nil receiver and the zero value remain safe and unwrap to
|
||||
`ErrCapacityExceeded`, but a consumer-constructed value is not evidence that
|
||||
an engine rejected work;
|
||||
- the type and its default Go encoding have no stable JSON contract; and
|
||||
- consumers own returned values and may mutate `BackendID` without affecting
|
||||
engine state or another error.
|
||||
|
||||
Implement `Error` without exposing anything other than the public field and
|
||||
the generic sentinel wording. For a nil receiver or blank field, return
|
||||
`ErrCapacityExceeded.Error()`. Otherwise include the backend ID with `%q`.
|
||||
Implement `Unwrap` as an unconditional return of `ErrCapacityExceeded`.
|
||||
|
||||
Do not add an exported constructor, custom formatter, `Is` method, JSON tags,
|
||||
`MarshalJSON`, or `UnmarshalJSON`. Direct equality with
|
||||
`ErrCapacityExceeded` is not a supported contract.
|
||||
|
||||
### Root Translation
|
||||
|
||||
Update `mapPublicError` in `errors.go` before its general
|
||||
`publicErrorFor` mapping:
|
||||
|
||||
1. use `errors.As` to find an internal `*usecase.CapacityError`;
|
||||
2. require the matched pointer to be nonnil and its `BackendID` to be
|
||||
nonblank;
|
||||
3. return a newly allocated public
|
||||
`&CapacityError{BackendID: internalCapacityError.BackendID}` directly; and
|
||||
4. otherwise continue through the existing general mapping.
|
||||
|
||||
Returning the public value directly is deliberate. It prevents the internal
|
||||
typed error and internal sentinel from remaining in the returned error chain,
|
||||
while the public value's `Unwrap` supplies the supported public sentinel.
|
||||
Copy only the string field; do not retain the internal error.
|
||||
|
||||
Leave the existing `capacity.ErrCapacityExceeded` case in `publicErrorFor`.
|
||||
It remains a defensive compatibility fallback for an unstructured internal
|
||||
capacity error. No valid `Run` or `RunPrepared` capacity rejection should take
|
||||
that fallback after this feature is implemented.
|
||||
|
||||
Do not reorder unrelated error categories. In particular, generation failures
|
||||
remain `ErrLLMGenerate`, and the mapper must not infer admission rejection from
|
||||
public sentinel text, provider errors, status codes, or arbitrary errors that
|
||||
happen to expose a backend field.
|
||||
|
||||
### Operation Contracts
|
||||
|
||||
Update the existing declarations and GoDoc without duplicating the full type
|
||||
contract:
|
||||
|
||||
- the `ErrCapacityExceeded` GoDoc in `engine.go` remains the broad
|
||||
classification contract and points consumers to `CapacityError` for the
|
||||
selected backend ID;
|
||||
- `Engine.Run` states that engine admission rejection is discoverable as
|
||||
`*CapacityError` and still matches `ErrCapacityExceeded`;
|
||||
- `Engine.RunPrepared` states the same and retains its one-attempt handle
|
||||
semantics;
|
||||
- `doc.go` adds error values to its unstable-JSON category, lists
|
||||
`CapacityError` there, and notes that returned structured errors are
|
||||
caller-owned; and
|
||||
- no operation other than `Run` and `RunPrepared` claims it can return this
|
||||
type.
|
||||
|
||||
Do not change method signatures or add the type to any stable JSON list.
|
||||
|
||||
### Error And Identity Boundaries
|
||||
|
||||
The implementation must preserve all of these distinctions:
|
||||
|
||||
| Condition | `errors.Is` identity | `errors.As` to `*CapacityError` |
|
||||
| --- | --- | --- |
|
||||
| Limited selected backend has no admission slot | `ErrCapacityExceeded` | Yes, with selected backend ID |
|
||||
| Context is already done when admission checks it | Context error | No |
|
||||
| Waiting for an active generation permit is canceled | `ErrLLMGenerate` and context error | No |
|
||||
| Provider or injected client returns throttling or quota failure | `ErrLLMGenerate` and documented collaborator identity | No |
|
||||
| Invalid request, profile, credential, artifact, render, or validation failure | Existing category | No |
|
||||
| Unlimited backend or endpoint-only profile | No admission rejection | No |
|
||||
|
||||
For an ordinary `Run`, the ID is the effective registered backend selected
|
||||
during preparation. For `RunPrepared`, it is the backend frozen in the claimed
|
||||
handle. An endpoint override changes only the endpoint and must not change the
|
||||
reported ID.
|
||||
|
||||
### Test Ownership
|
||||
|
||||
Extend existing tests at their current ownership boundaries.
|
||||
|
||||
`internal/capacity/manager_test.go` continues to own admission mechanics.
|
||||
Strengthen `TestManagerAdmissionHonorsContextAndUnlimitedBackends` so the
|
||||
limited pool is full before the canceled admission attempt. This protects the
|
||||
existing rule that cancellation wins over capacity rejection without adding a
|
||||
new overlapping test.
|
||||
|
||||
`internal/usecase/runner_test.go` owns attachment of selected identity.
|
||||
Update `TestRunnerAdmissionFailureSkipsCompletionCollaborators` so:
|
||||
|
||||
- the capacity case uses `errors.As` to obtain the internal
|
||||
`*CapacityError`;
|
||||
- its `BackendID` is exactly `"custom"`;
|
||||
- the error still matches `capacity.ErrCapacityExceeded`;
|
||||
- cancellation does not produce an internal `*CapacityError`; and
|
||||
- the existing assertions about no partial result, no recategorization, the
|
||||
admitted backend ID, and skipped collaborators remain.
|
||||
|
||||
Replace the current string-content assertion with the structured assertion.
|
||||
Do not test exact diagnostic wording.
|
||||
|
||||
`errors_internal_test.go` owns root translation. Add a focused test that maps
|
||||
an internal `*usecase.CapacityError` and proves:
|
||||
|
||||
- the result is a public `*CapacityError` with the copied backend ID;
|
||||
- it matches public `ErrCapacityExceeded`;
|
||||
- it does not match unrelated public categories;
|
||||
- it no longer exposes the internal typed error through `errors.As`; and
|
||||
- mutating the source internal error after mapping does not alter the public
|
||||
value.
|
||||
|
||||
Retain the existing cancellation-preservation test.
|
||||
|
||||
`capacity_contract_test.go` owns assembled public `Run` behavior. Extend
|
||||
`TestEngineRejectsRunBeforeCompletionWhenAdmissionIsFull` to prove:
|
||||
|
||||
- no partial result is returned;
|
||||
- `errors.Is(err, promptkit.ErrCapacityExceeded)` still succeeds;
|
||||
- `errors.As` obtains a nonnil `*promptkit.CapacityError`;
|
||||
- `BackendID` is `"limited"` even though the rejected request overrides its
|
||||
endpoint;
|
||||
- unrelated public categories do not match;
|
||||
- mutating the returned error cannot alter a subsequent independently
|
||||
rejected call or its backend ID; and
|
||||
- no completion collaborator or model client is invoked for rejected calls.
|
||||
|
||||
In the same established full-pool setup, add a canceled-context assertion
|
||||
before releasing the admitted run. It must match `context.Canceled`, must not
|
||||
match `ErrCapacityExceeded`, and must not be discoverable as
|
||||
`*promptkit.CapacityError`. This is the root contract counterpart to the
|
||||
manager precedence test.
|
||||
|
||||
Extend `TestCapacityExceededSentinelContract` to cover the public type's zero
|
||||
and nil receiver behavior without asserting exact diagnostic wording:
|
||||
|
||||
- both safely match `ErrCapacityExceeded`;
|
||||
- an ordinary populated value is discoverable by `errors.As`; and
|
||||
- neither the sentinel nor the typed value matches unrelated categories.
|
||||
|
||||
`prepared_execution_contract_test.go` owns the public `RunPrepared` boundary.
|
||||
In the capacity-rejection section of
|
||||
`TestPreparedExecutionCredentialCapacityAndTimingBoundaries`, assert that
|
||||
`errors.As` obtains `*promptkit.CapacityError` with backend ID `"limited"`.
|
||||
Retain the existing assertions that the result is nil, the public sentinel
|
||||
matches, the handle is consumed, and details remain available.
|
||||
|
||||
Do not duplicate manager limit matrices, generation FIFO tests, backend
|
||||
registration tests, or provider-client failure matrices. Existing tests
|
||||
already own those behaviors.
|
||||
|
||||
### Documentation Ownership
|
||||
|
||||
After the code and contract tests pass:
|
||||
|
||||
- update `docs/consumers/pkg-promptkit.md` in **Handle Errors** with a concise
|
||||
`errors.As` example that extracts `BackendID` while retaining the existing
|
||||
`errors.Is` guidance and application-owned retry policy;
|
||||
- update `docs/internal/runner.md` to describe the internal typed attachment
|
||||
and root translation without reproducing the public API contract;
|
||||
- update `docs/internal/capacity.md` to clarify that the manager still emits
|
||||
only its internal sentinel, while the runner attaches selected identity and
|
||||
the facade translates it;
|
||||
- update `docs/internal/overview.md` only enough to include the root typed
|
||||
capacity error in the facade responsibility; and
|
||||
- update `doc.go` and `engine.go` with the exact public contract described
|
||||
above.
|
||||
|
||||
Do not update `docs/formats.md`, the OpenAI-compatible integration contract,
|
||||
backend configuration GoDoc, README, or release notes. This feature changes no
|
||||
file format, provider wire request, backend configuration, project
|
||||
orientation, or released-version record.
|
||||
|
||||
When implementation is complete:
|
||||
|
||||
- change `docs/roadmap/capacity-errors.md` to `**Status:** Complete.`;
|
||||
- change `docs/roadmap/future.md` so it no longer describes structured
|
||||
capacity errors as active planning and simply records that no ideas await
|
||||
selection;
|
||||
- change both downstream wishlist dispositions from accepted planning to
|
||||
implemented behavior, linking to the consumer guide's **Handle Errors**
|
||||
section rather than duplicating the contract; and
|
||||
- change this document to `**Status:** Complete.`
|
||||
|
||||
The feature roadmap already contains no staged or prompt-level implementation
|
||||
language. Do not add such language to it when changing its status.
|
||||
|
||||
## Stage 1: Carry Structured Identity Across Internal Admission
|
||||
|
||||
### Objective
|
||||
|
||||
Replace diagnostic-string-only backend context with a typed internal
|
||||
admission error while preserving capacity mechanics and cancellation
|
||||
precedence.
|
||||
|
||||
### Implementation Prompt
|
||||
|
||||
Implement only Stage 1 of
|
||||
`docs/roadmap/implementation.md`.
|
||||
|
||||
1. Add `internal/usecase/capacity_error.go` with the exact internal type and
|
||||
method semantics in **Fixed Design**.
|
||||
2. Update `Runner.admitRun` to create one fresh internal typed error for a
|
||||
nonblank selected backend when the admitter returns the internal capacity
|
||||
sentinel.
|
||||
3. Update the existing runner admission-failure test to assert typed identity
|
||||
instead of inspecting diagnostic text.
|
||||
4. Strengthen the existing capacity-manager context test so cancellation is
|
||||
checked while the limited pool is already full.
|
||||
5. Run the focused formatting and validation below.
|
||||
|
||||
Do not modify the root public API, root mapper, capacity-manager production
|
||||
code, public contract tests, or current-state documentation in this stage.
|
||||
|
||||
### Focused Validation
|
||||
|
||||
Run from the repository root:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/usecase/capacity_error.go \
|
||||
internal/usecase/runner.go \
|
||||
internal/usecase/runner_test.go \
|
||||
internal/capacity/manager_test.go
|
||||
go test ./internal/capacity ./internal/usecase
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Completion Gate
|
||||
|
||||
Stage 1 is complete only when:
|
||||
|
||||
- capacity rejection from `admitRun` is discoverable as the internal
|
||||
`*usecase.CapacityError`;
|
||||
- its ID comes from the effective backend passed to admission;
|
||||
- both `Run` and `RunPrepared` use the shared boundary without duplicate
|
||||
wrapping;
|
||||
- `errors.Is` still reaches `capacity.ErrCapacityExceeded`;
|
||||
- cancellation and other admission errors remain untyped and unchanged;
|
||||
- capacity scheduling production code is untouched; and
|
||||
- focused tests and whitespace checks pass.
|
||||
|
||||
## Stage 2: Expose And Protect The Public Error Contract
|
||||
|
||||
### Objective
|
||||
|
||||
Add the minimal public typed error, translate the internal value without
|
||||
leaking it, and protect ordinary and prepared consumer behavior.
|
||||
|
||||
### Implementation Prompt
|
||||
|
||||
Implement only Stage 2 of
|
||||
`docs/roadmap/implementation.md` after Stage 1 satisfies its completion gate.
|
||||
|
||||
1. Add root `capacity_error.go` with the exact public type, methods, and GoDoc
|
||||
in **Fixed Design**.
|
||||
2. Update `errors.go` to translate a nonblank internal typed error into a
|
||||
fresh public value before general sentinel mapping.
|
||||
3. Preserve the existing unstructured capacity fallback and all unrelated
|
||||
error-mapping order.
|
||||
4. Update `engine.go` and `doc.go` with the exact operation, ownership, and
|
||||
unstable-JSON contracts.
|
||||
5. Add the focused root translation test.
|
||||
6. Extend the existing external-package `Run`, sentinel, cancellation, and
|
||||
`RunPrepared` contract assertions described under **Test Ownership**.
|
||||
7. Run the focused formatting and validation below.
|
||||
|
||||
Do not change admission policy, add provider classification, introduce a
|
||||
constructor or serialization contract, or update durable prose documentation
|
||||
in this stage.
|
||||
|
||||
### Focused Validation
|
||||
|
||||
Run from the repository root:
|
||||
|
||||
```sh
|
||||
gofmt -w capacity_error.go errors.go engine.go doc.go \
|
||||
errors_internal_test.go capacity_contract_test.go \
|
||||
prepared_execution_contract_test.go
|
||||
go test . -run \
|
||||
'TestMapPublicError|TestEngineRejectsRunBeforeCompletionWhenAdmissionIsFull|TestCapacityExceededSentinelContract|TestPreparedExecutionCredentialCapacityAndTimingBoundaries'
|
||||
go test .
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Completion Gate
|
||||
|
||||
Stage 2 is complete only when:
|
||||
|
||||
- every assembled `Run` and `RunPrepared` admission rejection is discoverable
|
||||
as a public `*CapacityError` with the selected backend ID;
|
||||
- the public error unwraps only to `ErrCapacityExceeded` and does not retain
|
||||
the internal typed error;
|
||||
- existing broad `errors.Is` handling remains valid;
|
||||
- endpoint overrides do not change the reported ID;
|
||||
- cancellation at a full pool remains a context error and not a capacity
|
||||
error;
|
||||
- caller mutation cannot affect another rejection or engine state;
|
||||
- provider, generation, and unrelated error mappings remain unchanged; and
|
||||
- focused and complete root-package tests pass.
|
||||
|
||||
## Stage 3: Update Canonical Documentation And Validate
|
||||
|
||||
### Objective
|
||||
|
||||
Make the implemented structured error discoverable, reconcile temporary
|
||||
planning state, and complete repository-wide validation.
|
||||
|
||||
### Implementation Prompt
|
||||
|
||||
Implement only Stage 3 of
|
||||
`docs/roadmap/implementation.md` after Stages 1 and 2 satisfy their completion
|
||||
gates.
|
||||
|
||||
1. Update the consumer and internal documents listed under
|
||||
**Documentation Ownership**. Keep exact API guarantees in GoDoc and use
|
||||
prose documents for task guidance and implementation boundaries.
|
||||
2. Update the feature-roadmap, future-catalog, implementation-plan, and
|
||||
downstream-wishlist statuses and links exactly as specified above.
|
||||
3. Follow every added or changed Markdown link and confirm its file and
|
||||
heading target.
|
||||
4. Run the full validation sequence.
|
||||
5. Inspect the complete diff for scope, sensitive data, and repository
|
||||
hygiene.
|
||||
6. Only after every check passes, leave both roadmap statuses as `Complete`
|
||||
and rerun `git diff --check`.
|
||||
|
||||
Do not add a release document, new example, new public package, retry policy,
|
||||
transport mapping, or duplicate API reference.
|
||||
|
||||
### Full Validation
|
||||
|
||||
Run from the repository root:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/usecase/capacity_error.go \
|
||||
internal/usecase/runner.go \
|
||||
internal/usecase/runner_test.go \
|
||||
internal/capacity/manager_test.go \
|
||||
capacity_error.go errors.go engine.go doc.go \
|
||||
errors_internal_test.go capacity_contract_test.go \
|
||||
prepared_execution_contract_test.go
|
||||
gofmt -l $(git ls-files '*.go')
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
go build ./...
|
||||
go run ./examples/go-library/prepare
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
The `gofmt -l` command must print no paths. The maintained example must remain
|
||||
offline and require no real credential or provider.
|
||||
|
||||
Inspect the final state and confirm:
|
||||
|
||||
- only files required by this feature and pre-existing user changes are
|
||||
present;
|
||||
- no credential, private source content, endpoint, workspace file, local
|
||||
replacement, generated binary, or unrelated formatting change was added;
|
||||
- the root declaration and GoDoc own the exact public contract;
|
||||
- consumer guidance summarizes the workflow and links to the canonical API;
|
||||
- internal documents describe responsibility without redefining the public
|
||||
contract;
|
||||
- no current-state document claims Promptkit owns retry, backoff, transport,
|
||||
logging, or metrics policy;
|
||||
- no roadmap retains staged language outside this implementation plan; and
|
||||
- no release, commit, or tag was created.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
Implementation is complete only when:
|
||||
|
||||
- every Stage 1 and Stage 2 gate remains satisfied;
|
||||
- ordinary and race-enabled tests pass;
|
||||
- vet, build, formatting, the offline example, Markdown links, and whitespace
|
||||
checks pass;
|
||||
- public, consumer, and internal documentation agree on the implemented
|
||||
boundary;
|
||||
- future-catalog and downstream-wishlist dispositions no longer describe the
|
||||
feature as pending;
|
||||
- both roadmap statuses are `Complete`;
|
||||
- the working tree contains no unintended files or changes; and
|
||||
- the repository is ready for maintainer review without a commit or release
|
||||
having been created by this plan.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The accepted feature roadmap and fixed design above fully specify the
|
||||
implementation boundary.
|
||||
@@ -1,163 +0,0 @@
|
||||
# Local Backend Convenience
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Purpose
|
||||
|
||||
Make the common case of using a local OpenAI-compatible endpoint concise and
|
||||
easy to discover without introducing implicit configuration or a separate
|
||||
backend abstraction.
|
||||
|
||||
The existing `Backend` type and registry remain the canonical, fully
|
||||
configurable interface. A small convenience constructor will cover the usual
|
||||
local-network case, while improved consumer documentation will make it clear
|
||||
when an endpoint-only profile, the convenience constructor, or a complete
|
||||
`Backend` value is appropriate.
|
||||
|
||||
## Motivation
|
||||
|
||||
Consumers can already use a local endpoint by setting `Profile.Endpoint`, or
|
||||
register one as a backend with `WithBackend`. The first option is concise but
|
||||
does not provide shared backend-level concurrency control. The second supports
|
||||
the complete backend feature set but requires consumers to understand and
|
||||
populate several fields for a common configuration.
|
||||
|
||||
Most consumers adding a local backend need only:
|
||||
|
||||
- a stable backend ID;
|
||||
- an OpenAI-compatible endpoint; and
|
||||
- a concurrency limit appropriate for the local server.
|
||||
|
||||
Promptkit should provide a direct path for that case while keeping all
|
||||
configuration explicit and preserving the full registry interface for
|
||||
advanced needs.
|
||||
|
||||
## Consumer Paths
|
||||
|
||||
Documentation should present three progressively more configurable paths:
|
||||
|
||||
1. Set `Profile.Endpoint` when a profile only needs to target a local endpoint
|
||||
and does not need shared backend policy.
|
||||
2. Use the local-backend convenience constructor when profiles should share a
|
||||
named local endpoint and its concurrency limit.
|
||||
3. Construct a complete `Backend` value when the consumer needs a custom
|
||||
backend ID, authentication, extra request parameters, an explicit queue
|
||||
capacity, or multiple local backends.
|
||||
|
||||
These are complementary interfaces. The convenience constructor must return an
|
||||
ordinary `Backend`, so it does not create a second configuration model.
|
||||
|
||||
## Public Convenience API
|
||||
|
||||
The public package should expose:
|
||||
|
||||
```go
|
||||
const BackendLocal = "local"
|
||||
|
||||
func LocalBackend(endpoint string, concurrencyLimit int) Backend
|
||||
```
|
||||
|
||||
`LocalBackend` should return a `Backend` with:
|
||||
|
||||
- `ID` set to `BackendLocal`;
|
||||
- `Endpoint` set to the supplied endpoint;
|
||||
- `ConcurrencyLimit` set to the supplied limit; and
|
||||
- all other fields left at their zero values.
|
||||
|
||||
The returned value is passed to `WithBackend` and follows the same copying,
|
||||
normalization, validation, and registration rules as any consumer-constructed
|
||||
`Backend`.
|
||||
|
||||
The constructor should be a transparent value constructor. It should not read
|
||||
environment variables, mutate global state, register the backend, validate
|
||||
arguments independently, or create profiles. Consumers may inspect or modify
|
||||
the returned value before registration, although documentation should direct
|
||||
substantially customized configurations to the full `Backend` form.
|
||||
|
||||
## Identity and Registration
|
||||
|
||||
`BackendLocal` is a conventional ID used by the convenience constructor. It is
|
||||
not pre-registered and should not become a specially reserved registry ID.
|
||||
Consumers remain responsible for registering the returned backend with
|
||||
`WithBackend` and naming it from profiles through `BackendID`.
|
||||
|
||||
This distinction preserves compatibility with consumers that may already
|
||||
register their own backend using the ID `"local"`. Normal duplicate-ID rules
|
||||
still apply if a consumer attempts to register more than one backend with that
|
||||
ID.
|
||||
|
||||
Consumers that need multiple local endpoints should choose distinct IDs and
|
||||
use complete `Backend` values rather than the single conventional helper ID.
|
||||
|
||||
## Concurrency and Queue Semantics
|
||||
|
||||
The constructor must preserve the existing backend concurrency contract:
|
||||
|
||||
- a positive concurrency limit bounds simultaneous requests and uses the
|
||||
existing default queue capacity because `QueueCapacity` remains `nil`;
|
||||
- a zero concurrency limit leaves the backend unconstrained; and
|
||||
- a negative concurrency limit is rejected through the existing engine
|
||||
configuration validation path.
|
||||
|
||||
The constructor should not select a hidden default concurrency limit. Local
|
||||
servers vary substantially in capacity, so the consumer should make this
|
||||
choice explicitly.
|
||||
|
||||
## Documentation
|
||||
|
||||
The final documentation state has two canonical surfaces:
|
||||
|
||||
- Public Go documentation describes the exact contract of
|
||||
`BackendLocal` and `LocalBackend`, including their conventional,
|
||||
non-pre-registered nature.
|
||||
- The [promptkit consumer guide](../consumers/pkg-promptkit.md) includes
|
||||
a task-oriented local-endpoint section that shows the three consumer paths,
|
||||
explains the decision between them, and provides concise examples of the
|
||||
endpoint-only and convenience-constructor forms.
|
||||
|
||||
The consumer guide continues to document the full `Backend` interface as
|
||||
the advanced path rather than attempting to reproduce every configuration
|
||||
variation through convenience APIs.
|
||||
|
||||
## Compatibility
|
||||
|
||||
This feature is additive:
|
||||
|
||||
- existing endpoint-only profiles continue to work unchanged;
|
||||
- existing `Backend` values and `WithBackend` registrations remain the
|
||||
canonical general-purpose interface;
|
||||
- existing registrations using the literal ID `"local"` remain valid; and
|
||||
- OpenRouter defaults and all other backend behavior remain unchanged.
|
||||
|
||||
No consumer is required to adopt the convenience constructor.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This work does not include:
|
||||
|
||||
- pre-registering or implicitly enabling a local backend;
|
||||
- discovering a local endpoint, API key, or concurrency limit from environment
|
||||
variables;
|
||||
- adding local-backend fields to `Config`;
|
||||
- selecting a default local model or creating a profile automatically;
|
||||
- adding a combined backend-and-profile constructor;
|
||||
- adding convenience parameters for API keys, extra request parameters, or
|
||||
queue capacity;
|
||||
- replacing or redesigning the backend registry;
|
||||
- adding support for non-OpenAI-compatible local APIs; or
|
||||
- changing backend routing, scheduling, or queue behavior.
|
||||
|
||||
## Target End State
|
||||
|
||||
After this work:
|
||||
|
||||
- consumers with a simple one-profile local endpoint can continue to configure
|
||||
it directly on the profile;
|
||||
- consumers needing a shared local endpoint and concurrency policy can express
|
||||
it with one `LocalBackend` call and register the returned value normally;
|
||||
- consumers with advanced or multiple-local-backend requirements have a clear
|
||||
path to the complete `Backend` interface;
|
||||
- all local configuration remains explicit, inspectable, and compatible with
|
||||
dependency injection; and
|
||||
- canonical documentation makes the simplest suitable interface easy to find
|
||||
without obscuring the underlying registry model.
|
||||
@@ -6,21 +6,19 @@ This document records features and interface changes that would be useful
|
||||
additions to PromptKit from the perspective of the maintainers of Notarius, a
|
||||
downstream application that consumes PromptKit.
|
||||
|
||||
PromptKit v0.3.0 provides the capabilities Notarius currently needs. None of
|
||||
the ideas below blocks current Notarius development. They are opportunities to
|
||||
reduce downstream workarounds, improve integration correctness, and make
|
||||
PromptKit more ergonomic for applications with configuration validation,
|
||||
debugging, checkpointing, and operational-observability requirements.
|
||||
PromptKit now provides the capabilities Notarius currently needs. The
|
||||
remaining deferred ideas are optional opportunities to improve checkpointing
|
||||
and operational observability.
|
||||
|
||||
The examples are API sketches intended to communicate the desired capability,
|
||||
not prescriptive names or finalized Go contracts.
|
||||
|
||||
## Priority 1: Atomic Execution With Prepared Details
|
||||
|
||||
**Disposition:** Covered by the accepted
|
||||
[executable preparation handles](prepared-execution.md) roadmap. The shared
|
||||
two-phase capability should provide the required single-preparation
|
||||
consistency; a separate `RunDetailed` method is not cataloged initially.
|
||||
**Disposition:** Implemented through [`Engine.PrepareExecution` and
|
||||
`Engine.RunPrepared`](../../engine.go). See the
|
||||
[consumer guidance](../consumers/pkg-promptkit.md#prepare-now-and-execute-the-same-snapshot-later).
|
||||
A separate `RunDetailed` method is not cataloged.
|
||||
|
||||
### Downstream need
|
||||
|
||||
@@ -33,58 +31,18 @@ Notarius needs both:
|
||||
Notarius uses the prepared details to construct redaction-aware debug bundles
|
||||
and retain enough information to diagnose model behavior.
|
||||
|
||||
### Current integration
|
||||
### Implemented behavior
|
||||
|
||||
Notarius currently calls `Engine.Prepare` and then `Engine.Run` with the same
|
||||
request. Because `Run` performs preparation internally, a successful request
|
||||
resolves and prepares the same work twice.
|
||||
|
||||
This duplicates profile resolution, input hashing, schema loading, and prompt
|
||||
rendering. It also creates a theoretical consistency window in which a
|
||||
filesystem-backed prompt, profile, schema, or input could change between the
|
||||
explicit preparation and the preparation performed by `Run`.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Add an opt-in execution method that prepares exactly once and returns both the
|
||||
prepared details and completed result:
|
||||
|
||||
```go
|
||||
type RunReport struct {
|
||||
Prepared PreparedRun
|
||||
Result RunResult
|
||||
}
|
||||
|
||||
func (e *Engine) RunDetailed(
|
||||
ctx context.Context,
|
||||
req RunRequest,
|
||||
) (*RunReport, error)
|
||||
```
|
||||
|
||||
The exact names are flexible. The important contract is that preparation
|
||||
occurs once and that the returned prepared state describes the execution that
|
||||
produced the returned result.
|
||||
|
||||
Existing `Prepare` and `Run` behavior should remain available for consumers
|
||||
that need only one side of the operation.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Keep this API additive and preserve the existing simple `Run` workflow.
|
||||
- Return caller-owned copies under PromptKit's existing ownership rules.
|
||||
- Define whether any prepared details are available after an operational
|
||||
generation or validation error. Notarius does not require partial results
|
||||
for the initial use case, but an explicit contract would be valuable.
|
||||
- Preserve cancellation and backend-admission semantics.
|
||||
- Do not add all prepared content directly to `RunResult`. Rendered prompt
|
||||
content can be large and sensitive, and consumers should opt in to receiving
|
||||
it.
|
||||
Notarius can prepare one frozen execution snapshot, retain a caller-owned and
|
||||
credential-redacted `Details` value for its debug bundle, and execute the same
|
||||
snapshot through `RunPrepared`. The opaque handle is engine-bound and
|
||||
single-use; an unused handle can be released with `Discard`. The consumer
|
||||
guide and exported GoDoc own the exact lifecycle and failure contracts.
|
||||
|
||||
### Value to Notarius
|
||||
|
||||
This is the highest-value wishlist item. It would remove duplicate work from
|
||||
every successful PromptKit-backed call and ensure that retained debug material
|
||||
corresponds atomically to the actual execution.
|
||||
This removes duplicate work from PromptKit-backed calls and ensures that
|
||||
retained debug material corresponds atomically to the actual execution.
|
||||
|
||||
## Priority 2: Prompt-Independent Profile Inspection
|
||||
|
||||
@@ -275,14 +233,13 @@ application-policy decision, not an upstream PromptKit gap.
|
||||
|
||||
## Suggested Upstream Sequence
|
||||
|
||||
If the PromptKit team chooses to pursue these ideas, the most useful order for
|
||||
Notarius would be:
|
||||
For downstream adoption and any remaining upstream work, the useful order is:
|
||||
|
||||
1. Add atomic execution that returns prepared details and the completed result.
|
||||
1. Adopt prepared execution for atomic details and results.
|
||||
2. Add a semantic execution-target digest, preferably alongside profile
|
||||
inspection.
|
||||
3. Use the implemented typed capacity error where backend admission diagnostics
|
||||
are needed.
|
||||
|
||||
The first addresses a concrete execution workaround. The second would improve
|
||||
The first removes the concrete execution workaround. The second would improve
|
||||
checkpoint correctness and reduce coupling. The third is operational polish.
|
||||
|
||||
@@ -1,280 +0,0 @@
|
||||
# Executable Preparation Handles
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Purpose
|
||||
|
||||
Allow a consumer to prepare one exact Promptkit execution, inspect and retain
|
||||
its credential-redacted public preparation details, and later execute that
|
||||
already-prepared work without resolving or rendering the request again.
|
||||
|
||||
This provides a supported preflight-before-generation boundary for
|
||||
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-1-executable-preparation-handles)
|
||||
and removes the duplicate `Prepare`-then-`Run` workaround described by
|
||||
[Notarius](notarius-promptkit-wishlist.md#priority-1-atomic-execution-with-prepared-details).
|
||||
|
||||
## Motivation
|
||||
|
||||
`Engine.Prepare` currently returns the provenance and rendered details that
|
||||
consumers need for debugging, persistence, and preflight checks. `Engine.Run`
|
||||
then performs its own preparation before generation. A consumer that needs
|
||||
both values must therefore prepare the same logical request twice.
|
||||
|
||||
That workaround duplicates source loading, input hashing, schema work, and
|
||||
template rendering. It also permits filesystem-backed prompts, profiles,
|
||||
schemas, or inputs to change between the public preparation and the
|
||||
preparation that actually produces the result.
|
||||
|
||||
Promptkit already owns the complete preparation and execution pipeline. An
|
||||
opt-in prepared-execution handle should expose the missing boundary without
|
||||
putting rendered content into every `RunResult` or moving persistence policy
|
||||
into the library.
|
||||
|
||||
## Consumer Workflow
|
||||
|
||||
The target public workflow is:
|
||||
|
||||
```go
|
||||
prepared, err := engine.PrepareExecution(ctx, request)
|
||||
if err != nil {
|
||||
// Handle preparation failure.
|
||||
return
|
||||
}
|
||||
defer prepared.Discard()
|
||||
|
||||
details := prepared.Details()
|
||||
// Persist or inspect a consumer-selected safe subset of details.
|
||||
|
||||
result, err := engine.RunPrepared(ctx, prepared)
|
||||
```
|
||||
|
||||
The target public surface is:
|
||||
|
||||
```go
|
||||
type PreparedExecution struct {
|
||||
// Opaque Promptkit-owned state.
|
||||
}
|
||||
|
||||
func (e *Engine) PrepareExecution(
|
||||
ctx context.Context,
|
||||
req RunRequest,
|
||||
) (*PreparedExecution, error)
|
||||
|
||||
func (p *PreparedExecution) Details() PreparedRun
|
||||
|
||||
func (p *PreparedExecution) Discard()
|
||||
|
||||
func (e *Engine) RunPrepared(
|
||||
ctx context.Context,
|
||||
prepared *PreparedExecution,
|
||||
) (*RunResult, error)
|
||||
```
|
||||
|
||||
The declarations and GoDoc will own the exact implemented contract. The
|
||||
important public shape is an opaque handle, caller-owned `PreparedRun`
|
||||
details, an explicit discard operation, and execution through the engine that
|
||||
created the handle.
|
||||
|
||||
## Prepared Snapshot
|
||||
|
||||
`PrepareExecution` performs complete preparation without model generation or
|
||||
backend-capacity admission. It applies the same request validation, source
|
||||
precedence, backend and profile resolution, credential requirement checks,
|
||||
output-contract resolution, artifact loading, hashing, schema loading,
|
||||
session resolution, and rendering behavior as `Prepare`.
|
||||
|
||||
A successful handle freezes all source-derived state required for later
|
||||
execution, including:
|
||||
|
||||
- the selected prompt definition, profile, and backend identity;
|
||||
- the complete effective execution target and request-field presence;
|
||||
- rendered messages and effective session ID;
|
||||
- prompt, rendered-prompt, and input hashes;
|
||||
- the effective output contract and provider-facing structured-output
|
||||
constraint; and
|
||||
- private validation state sufficient to validate generated output without
|
||||
reopening schema files or `fs.FS` resources, including required schema
|
||||
references.
|
||||
|
||||
After `PrepareExecution` succeeds, changes to prompt, profile, schema, input,
|
||||
or request-owned data cannot change what `RunPrepared` sends to the model or
|
||||
how it validates the generated output.
|
||||
|
||||
The handle retains an internal snapshot independent from values returned by
|
||||
`Details`. Mutating a returned `PreparedRun`, its maps, slices, messages, or
|
||||
schema values does not affect later execution. Each `Details` call returns a
|
||||
fresh caller-owned copy under the existing `PreparedRun` ownership and stable
|
||||
JSON rules.
|
||||
|
||||
## Handle Lifecycle
|
||||
|
||||
A `PreparedExecution` is:
|
||||
|
||||
- created only by a successful `PrepareExecution` call;
|
||||
- bound to the exact `Engine` that created it;
|
||||
- valid for one `RunPrepared` invocation;
|
||||
- safe for repeated `Details` calls;
|
||||
- intentionally opaque and without a supported JSON representation; and
|
||||
- in-process state rather than a durable or restartable job.
|
||||
|
||||
`RunPrepared` atomically claims a valid handle before beginning the execution
|
||||
attempt. A second or concurrent invocation fails without starting another
|
||||
execution, including when the first invocation ended in cancellation, capacity
|
||||
rejection, generation failure, or operational validation failure. Copies of
|
||||
the public handle share the same one-attempt state and cannot bypass this rule.
|
||||
|
||||
A nil, zero-value, foreign-engine, discarded, already-claimed, or already-used
|
||||
handle is invalid. `RunPrepared` reports these lifecycle errors through the
|
||||
ordinary public invalid-request category. A failed foreign-engine invocation
|
||||
does not consume a handle that remains valid for its owning engine.
|
||||
|
||||
`Discard` idempotently makes an unclaimed handle unavailable for execution and
|
||||
drops Promptkit's references to secret-bearing or execution-only state.
|
||||
`RunPrepared` performs the same cleanup automatically after claiming a handle.
|
||||
Credential-redacted public preparation details remain available after discard,
|
||||
success, or failure so consumers can retain diagnostics. Promptkit does not
|
||||
promise secure erasure of Go string memory.
|
||||
|
||||
Lifecycle transitions are concurrency-safe. When `RunPrepared` and `Discard`
|
||||
race, exactly one claims the ready handle. `Discard` is not an execution
|
||||
cancellation mechanism and does not interrupt an attempt that has already
|
||||
claimed the handle; consumers cancel that attempt through its context.
|
||||
|
||||
## Credentials And Sensitive Data
|
||||
|
||||
`Details` has the same security contract as `PreparedRun`: it can contain
|
||||
rendered messages, schemas, identifiers, and hashes, but never a resolved API
|
||||
key value. Consumers remain responsible for selecting, redacting, storing, and
|
||||
retaining any persisted preparation material.
|
||||
|
||||
A direct `RunRequest.APIKey` is retained only in opaque execution state until
|
||||
the handle is run or discarded. It is never added to details, hashes, JSON,
|
||||
`String`, or `GoString` output.
|
||||
|
||||
An environment-variable name is frozen as part of the effective target, but
|
||||
its credential value is not captured for the lifetime of the handle.
|
||||
`PrepareExecution` applies the existing preparation-time availability check.
|
||||
`RunPrepared` rechecks availability before admission, and the selected model
|
||||
client uses the environment value visible during execution. This preserves
|
||||
current secret ownership and avoids retaining an environment credential while
|
||||
a consumer persists preflight material.
|
||||
|
||||
The opaque handle must not expose retained request data or credentials through
|
||||
default formatting, JSON, or error messages.
|
||||
|
||||
## Admission, Cancellation, And Execution
|
||||
|
||||
`PrepareExecution` never reserves backend admission or an active-generation
|
||||
permit. Its context governs preparation only; cancellation after it returns
|
||||
does not invalidate the handle.
|
||||
|
||||
`RunPrepared` uses its own context for credential revalidation, backend
|
||||
admission, active-generation waiting, model generation, output validation, and
|
||||
any internal repair. Admission occurs when `RunPrepared` begins so a consumer
|
||||
cannot occupy bounded capacity while inspecting or persisting preparation
|
||||
details.
|
||||
|
||||
For a limited backend, the admission lease covers the complete prepared
|
||||
execution attempt after admission: generation, validation, internal repair,
|
||||
and every success or failure exit. Actual generation continues to use the
|
||||
backend's FIFO active-generation permit. Existing capacity error identity,
|
||||
cancellation behavior, and release guarantees remain in force.
|
||||
|
||||
Because a prepared handle is one-attempt, cancellation or capacity rejection
|
||||
does not make it reusable. Retry and backoff policy remains with the consumer,
|
||||
which may create a new prepared handle when another attempt is appropriate.
|
||||
|
||||
## Results And Failures
|
||||
|
||||
On success, `RunPrepared` returns the existing caller-owned `RunResult`. Its
|
||||
source-derived provenance must match `Details`, including prompt identity and
|
||||
hash, rendered-prompt hash, session ID, selected profile and backend,
|
||||
effective target, and input hashes.
|
||||
|
||||
`RunResult.StartTime`, `EndTime`, and `Duration` describe the
|
||||
`RunPrepared` execution attempt. They exclude preparation time and any delay
|
||||
while the consumer retained the handle. Preparation timing remains in
|
||||
`PreparedRun`.
|
||||
|
||||
Preparation failure returns no handle. After successful preparation,
|
||||
`RunPrepared` retains the existing rule that an operational failure returns no
|
||||
partial `RunResult`; the consumer already has independent preparation details.
|
||||
A completed content-validation failure remains a successful result with
|
||||
`ValidationFailed`.
|
||||
|
||||
`PrepareExecution` preserves the public error categories of `Prepare`.
|
||||
`RunPrepared` preserves applicable invalid-request, credential, capacity,
|
||||
generation, validation, collaborator, and cancellation identities without
|
||||
reintroducing source-loading or rendering failures from frozen state.
|
||||
|
||||
## Compatibility And Existing Workflows
|
||||
|
||||
This feature is additive:
|
||||
|
||||
- `Prepare` remains the simple preparation-only operation;
|
||||
- `Run` remains the simple prepare-and-execute operation with its current
|
||||
early-admission and error-ordering behavior;
|
||||
- `PreparedRun` and `RunResult` retain their existing stable JSON
|
||||
representations;
|
||||
- model-client and artifact-reader extension interfaces remain unchanged; and
|
||||
- backend routing, concurrency limits, queue capacities, and provider wire
|
||||
behavior remain unchanged.
|
||||
|
||||
The new workflow may share internal machinery with `Prepare` and `Run`, but it
|
||||
must not change their observable behavior merely to simplify implementation.
|
||||
|
||||
## Documentation
|
||||
|
||||
The completed documentation set has these ownership boundaries:
|
||||
|
||||
- exported declarations and GoDoc own the exact handle, method, lifecycle,
|
||||
ownership, concurrency, credential, error, and cancellation contracts;
|
||||
- the promptkit consumer guide explains when to use `Prepare`, `Run`, or the
|
||||
two-phase prepared-execution workflow; and
|
||||
- internal runner, source-validation, capacity, and model-client documentation
|
||||
describe the implemented collaborator boundaries without duplicating public
|
||||
contracts.
|
||||
|
||||
No release document is part of the feature implementation itself. Release
|
||||
guidance is prepared only when the resulting public API is selected for
|
||||
publication.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This work does not include:
|
||||
|
||||
- serializable, durable, resumable, or cross-process execution handles;
|
||||
- reuse for multiple consumer-initiated executions;
|
||||
- concurrent execution of one handle;
|
||||
- capacity reservation during preparation;
|
||||
- a background task queue, priorities, worker lifecycle, or job status;
|
||||
- retry, backoff, or provider failover policy;
|
||||
- a separate `RunDetailed` convenience method;
|
||||
- adding preparation details or rendered messages to every `RunResult`;
|
||||
- prompt or profile inspection APIs;
|
||||
- structured capacity or generation errors;
|
||||
- freezing environment-variable credential values for the handle lifetime;
|
||||
- snapshotting provider state or mutable behavior inside an injected
|
||||
`LLMClient`;
|
||||
- allowing target, validation, session, variable, or input overrides after
|
||||
preparation; or
|
||||
- changing existing `Prepare`, `Run`, file-format, provider-request, or stable
|
||||
JSON contracts.
|
||||
|
||||
## Target End State
|
||||
|
||||
After this work:
|
||||
|
||||
- consumers can perform and persist preflight before starting provider work;
|
||||
- one prepared handle executes exactly the source-derived prompt, target,
|
||||
schema, inputs, session, and messages described by its public details;
|
||||
- execution never reloads or rerenders consumer sources;
|
||||
- direct credentials remain confined to opaque, explicitly discardable state;
|
||||
- environment credentials are not retained across the preflight boundary;
|
||||
- backend capacity is reserved only when execution begins;
|
||||
- one handle can start at most one execution attempt, including any internal
|
||||
repair calls owned by that attempt;
|
||||
- preparation details remain available after execution success or failure;
|
||||
- existing simple `Prepare` and `Run` consumers remain unaffected; and
|
||||
- Promptkit continues to own reusable execution mechanics without taking on
|
||||
downstream persistence, redaction, retry, or job-management policy.
|
||||
@@ -1,246 +0,0 @@
|
||||
# Prompt-Independent Profile Inspection
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Purpose
|
||||
|
||||
Allow consumers to look up one execution profile by ID and inspect its
|
||||
structurally resolved model target without selecting a prompt, supplying
|
||||
placeholder inputs, checking credential availability, or invoking a model.
|
||||
|
||||
This provides a direct configuration-validation boundary for
|
||||
[Notarius](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
|
||||
and
|
||||
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-3-prompt-independent-profile-inspection).
|
||||
|
||||
## Motivation
|
||||
|
||||
Both downstream consumers need to reject invalid configured profile IDs before
|
||||
starting application work. They currently have to construct a synthetic prompt
|
||||
and call `Engine.Prepare` merely to exercise profile loading, backend lookup,
|
||||
and execution-target resolution.
|
||||
|
||||
That workaround couples profile validation to unrelated prompt definitions,
|
||||
fixture inputs, rendering, schema behavior, and current credential
|
||||
availability. Promptkit already owns profile precedence, backend membership,
|
||||
and target resolution, so it should expose that cohesive capability directly.
|
||||
|
||||
## Consumer Workflow
|
||||
|
||||
The target public workflow is:
|
||||
|
||||
```go
|
||||
inspection, err := engine.InspectProfile(ctx, profileID)
|
||||
if err != nil {
|
||||
// Reject or report the configured profile.
|
||||
return
|
||||
}
|
||||
|
||||
target := inspection.EffectiveModelParams
|
||||
if target.APIKeyEnv != "" {
|
||||
// Apply application policy for the named environment variable.
|
||||
}
|
||||
```
|
||||
|
||||
The target public surface is:
|
||||
|
||||
```go
|
||||
type ProfileInspection struct {
|
||||
ProfileID string
|
||||
EffectiveModelParams ExecutionTarget
|
||||
APIKeyRequired bool
|
||||
}
|
||||
|
||||
func (e *Engine) InspectProfile(
|
||||
ctx context.Context,
|
||||
profileID string,
|
||||
) (*ProfileInspection, error)
|
||||
```
|
||||
|
||||
The declarations and GoDoc will own the exact implemented contract. The
|
||||
important public shape is exact lookup through the existing engine, one
|
||||
caller-owned inspection value, the resolved `ExecutionTarget`, and an explicit
|
||||
signal for a direct API-key requirement.
|
||||
|
||||
`EffectiveModelParams.BackendID` identifies the selected registered backend
|
||||
and remains empty for endpoint-only profiles.
|
||||
`EffectiveModelParams.APIKeyEnv` reports the effective credential environment
|
||||
variable name. `APIKeyRequired` reports that the profile requires a direct
|
||||
request credential instead. These states are mutually exclusive after normal
|
||||
profile and backend precedence is applied.
|
||||
|
||||
`ProfileInspection` does not need a stable JSON representation. Consumers that
|
||||
persist application configuration or diagnostics can select the fields their
|
||||
own format requires.
|
||||
|
||||
## Lookup And Precedence
|
||||
|
||||
`InspectProfile` requires a non-blank explicit profile ID. It trims surrounding
|
||||
whitespace and otherwise performs the same case-sensitive exact lookup used by
|
||||
ordinary execution.
|
||||
|
||||
Lookup applies the engine's normal profile-source precedence:
|
||||
|
||||
- programmatic profiles supplied through `WithProfiles`;
|
||||
- the configured file, directory, or `fs.FS` profile source; and
|
||||
- the built-in profile catalog.
|
||||
|
||||
A valid higher-precedence match shadows a lower-precedence profile with the
|
||||
same ID. A malformed or unreadable higher-precedence match fails rather than
|
||||
silently falling back. The
|
||||
[profile format reference](../formats.md#profile-definitions) remains the
|
||||
canonical owner of profile-source and file-format behavior.
|
||||
|
||||
Inspection never derives a profile ID from a prompt's `default_profile`; the
|
||||
caller is inspecting one explicitly named profile.
|
||||
|
||||
## Structural Resolution
|
||||
|
||||
Inspection loads and validates the selected profile, verifies that a named
|
||||
backend exists in the engine's immutable backend registry, and applies normal
|
||||
framework-default, backend, and profile precedence to produce the effective
|
||||
target.
|
||||
|
||||
For the same engine state and profile ID, with no per-run execution override,
|
||||
the inspected target must match the target that ordinary preparation would
|
||||
resolve before applying request credentials and checking their availability.
|
||||
This equivalence must use one shared resolution path rather than a second set
|
||||
of precedence rules.
|
||||
|
||||
Structural resolution includes:
|
||||
|
||||
- backend routing identity;
|
||||
- endpoint and model;
|
||||
- sampling, token, timeout, service-tier, and reasoning settings;
|
||||
- the effective credential environment-variable name or direct-key
|
||||
requirement; and
|
||||
- deeply copied provider-specific extra parameters.
|
||||
|
||||
Inspection returns the effective target rather than a raw profile definition.
|
||||
This keeps framework and backend defaults visible to consumers without
|
||||
creating a second public profile-loading interface.
|
||||
|
||||
The result does not include request-override presence because no
|
||||
`ExecutionTargetOverride` participates in inspection.
|
||||
|
||||
## Credentials And Sensitive Data
|
||||
|
||||
Inspection reports credential requirements but never resolves, retains, or
|
||||
returns a credential value.
|
||||
|
||||
The operation does not read the named environment variable and succeeds when
|
||||
that variable is absent or blank. It accepts neither a direct API key nor an
|
||||
API-key environment override. Consumers decide whether credential availability
|
||||
must be enforced during application configuration, while `Prepare`,
|
||||
`PrepareExecution`, `Run`, and `RunPrepared` retain their execution-time
|
||||
credential contracts.
|
||||
|
||||
Error messages, formatting, and returned values must not expose environment
|
||||
values or other resolved secrets. Existing restrictions against raw API keys
|
||||
in profile sources remain unchanged.
|
||||
|
||||
## Ownership, Consistency, And Concurrency
|
||||
|
||||
Each successful call returns a caller-owned snapshot. Mutating the returned
|
||||
target or any nested extra-parameter map or slice cannot affect the engine,
|
||||
later inspection, or later execution.
|
||||
|
||||
Inspection is safe to call concurrently under the engine's existing immutable
|
||||
registry and repository contracts. It does not mutate profile sources or
|
||||
cache a result globally.
|
||||
|
||||
For filesystem-backed sources, an inspection describes the state observed by
|
||||
that call. It does not freeze the profile for a later `Run`; a source may
|
||||
change between operations. Consumers requiring an exact preflight-to-execution
|
||||
snapshot should use the existing prepared-execution workflow.
|
||||
|
||||
## Errors And Cancellation
|
||||
|
||||
The operation uses existing public error categories:
|
||||
|
||||
- a blank profile ID matches `ErrInvalidRequest`;
|
||||
- an absent exact ID matches `ErrProfileNotFound` and not `ErrProfileLoad`;
|
||||
- read, decode, validation, and source-selection failures match
|
||||
`ErrProfileLoad`; and
|
||||
- an unknown referenced backend or an invalid structurally resolved target
|
||||
matches `ErrProfileLoad`.
|
||||
|
||||
Errors should preserve useful underlying collaborator and context identities
|
||||
through `errors.Is` where the existing facade does so, without exposing
|
||||
internal package types. Context cancellation governs inspection and no partial
|
||||
inspection result is returned on failure.
|
||||
|
||||
Missing credential values are not inspection errors. The operation cannot
|
||||
return capacity or model-generation failures because it performs neither
|
||||
backend admission nor generation.
|
||||
|
||||
## Compatibility And Boundaries
|
||||
|
||||
This feature is additive. Existing profile formats, source precedence,
|
||||
backend registration, `Prepare`, prepared execution, and `Run` behavior remain
|
||||
unchanged.
|
||||
|
||||
The method belongs on the root `Engine` facade. Profile repositories and the
|
||||
backend registry remain internal implementation details, and no new public
|
||||
repository interface is introduced.
|
||||
|
||||
Inspection does not require prompt lookup, rendering, artifact loading, schema
|
||||
loading, validation, backend-capacity admission, or model-client access.
|
||||
Engine construction retains its ordinary configuration requirements; this
|
||||
feature does not introduce a separate profile-only engine.
|
||||
|
||||
## Documentation
|
||||
|
||||
The completed documentation set has these ownership boundaries:
|
||||
|
||||
- exported declarations and GoDoc own the exact method, result, ownership,
|
||||
credential, error, and cancellation contracts;
|
||||
- the promptkit consumer guide explains configuration-time profile inspection
|
||||
and distinguishes it from `Prepare` and prepared execution;
|
||||
- the profile format reference continues to own profile fields and source
|
||||
precedence; and
|
||||
- internal documentation describes shared profile and target resolution
|
||||
without duplicating public contracts.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This work does not include:
|
||||
|
||||
- enumerating or searching profiles;
|
||||
- returning raw profile definitions or profile source paths;
|
||||
- accepting per-run execution overrides, direct API keys, or API-key
|
||||
environment overrides;
|
||||
- checking environment-variable contents or other credential availability;
|
||||
- semantic execution-target fingerprints or profile hashes;
|
||||
- prompt-definition inspection or prompt default-profile resolution;
|
||||
- full-corpus validation across every profile source;
|
||||
- freezing a filesystem-backed profile for later execution;
|
||||
- exposing backend concurrency limits, queue state, or capacity policy;
|
||||
- model generation, provider health checks, or endpoint connectivity tests;
|
||||
- dynamic backend or profile registration after engine construction; or
|
||||
- changing current profile, backend, prepared-execution, or stable JSON
|
||||
contracts.
|
||||
|
||||
## Target End State
|
||||
|
||||
After this work:
|
||||
|
||||
- consumers can validate one configured profile without inventing a prompt or
|
||||
placeholder inputs;
|
||||
- lookup observes ordinary programmatic, configured-source, and built-in
|
||||
precedence;
|
||||
- a successful result proves that the profile exists, is valid, references a
|
||||
registered backend when applicable, and resolves to a structurally valid
|
||||
effective target;
|
||||
- the inspected target matches ordinary preparation for the same profile and
|
||||
engine state before per-run overrides and credential availability checks;
|
||||
- credential requirements are visible without reading or exposing credential
|
||||
values;
|
||||
- returned targets and nested data are caller-owned;
|
||||
- inspection performs no rendering, source loading unrelated to the profile,
|
||||
capacity admission, or model work;
|
||||
- existing execution workflows and compatibility contracts remain unchanged;
|
||||
and
|
||||
- Promptkit owns reusable profile validation while downstream applications
|
||||
retain configuration policy, persistence, logging, and credential-timing
|
||||
decisions.
|
||||
@@ -1,268 +0,0 @@
|
||||
# Prompt-Definition Inspection
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Purpose
|
||||
|
||||
Allow consumers to look up one prompt definition by ID and optional version
|
||||
and inspect its declared interface without supplying placeholder inputs,
|
||||
resolving a profile, rendering templates, loading schemas, or invoking a
|
||||
model.
|
||||
|
||||
This provides a focused configuration-validation boundary for Weatherreporter,
|
||||
whose need is recorded in its
|
||||
[Promptkit wishlist](weatherreporter-promptkit-wishlist.md#priority-2-prompt-definition-inspection),
|
||||
while preserving Promptkit's deferred, request-oriented source model.
|
||||
|
||||
## Motivation
|
||||
|
||||
Weatherreporter has a fixed application-owned registry of report definitions.
|
||||
It needs to verify that every configured prompt exists and declares the
|
||||
expected inputs and output workflow before collecting weather data or starting
|
||||
provider work.
|
||||
|
||||
The current workaround is to construct synthetic artifacts and variables and
|
||||
call `Engine.Prepare`. That validates much more than the application needs:
|
||||
prompt inputs are loaded, templates and session IDs are rendered, a profile
|
||||
and backend are resolved, credentials are checked, and JSON Schemas may be
|
||||
loaded and compiled.
|
||||
|
||||
Promptkit already owns prompt source selection, exact ID and version lookup,
|
||||
strict definition decoding, referenced content-file resolution, and prompt
|
||||
hashing. It should expose that cohesive subset directly rather than requiring
|
||||
consumers to reproduce its rules or maintain placeholder execution fixtures.
|
||||
|
||||
## Consumer Workflow
|
||||
|
||||
The target public workflow is:
|
||||
|
||||
```go
|
||||
inspection, err := engine.InspectPrompt(ctx, promptID, promptVersion)
|
||||
if err != nil {
|
||||
// Reject or report the configured prompt.
|
||||
return
|
||||
}
|
||||
|
||||
for _, input := range inspection.Inputs {
|
||||
// Compare the declared prompt interface with application configuration.
|
||||
}
|
||||
```
|
||||
|
||||
The target public surface is:
|
||||
|
||||
```go
|
||||
type PromptInputDefinition struct {
|
||||
Name string
|
||||
Required bool
|
||||
ContentType string
|
||||
Description string
|
||||
}
|
||||
|
||||
type PromptInspection struct {
|
||||
PromptID string
|
||||
PromptVersion string
|
||||
PromptHash string
|
||||
DefaultProfileID string
|
||||
Inputs []PromptInputDefinition
|
||||
OutputContract OutputContract
|
||||
}
|
||||
|
||||
func (e *Engine) InspectPrompt(
|
||||
ctx context.Context,
|
||||
promptID string,
|
||||
promptVersion string,
|
||||
) (*PromptInspection, error)
|
||||
```
|
||||
|
||||
The exported declarations and GoDoc will own the exact implemented contract.
|
||||
The important public shape is exact lookup through an existing engine, one
|
||||
caller-owned inspection value, declared input metadata, the declared output
|
||||
contract, and the same opaque prompt equality value used by preparation.
|
||||
|
||||
`PromptInspection` and `PromptInputDefinition` do not need stable JSON
|
||||
representations. Consumers that persist application configuration or
|
||||
diagnostics can define their own format and select the fields they require.
|
||||
The existing stable representations of `OutputContract`, `OutputFormat`, and
|
||||
`ValidationMode` remain unchanged.
|
||||
|
||||
## Lookup And Source Selection
|
||||
|
||||
`InspectPrompt` requires a non-blank prompt ID and performs the same
|
||||
case-sensitive exact selection used by ordinary preparation. Inspection does
|
||||
not trim or otherwise canonicalize a non-blank ID or version independently of
|
||||
that execution path.
|
||||
|
||||
When `promptVersion` is non-empty, the configured prompt source must contain
|
||||
exactly one matching ID and version pair. When it is empty, the source must
|
||||
contain exactly one definition with the requested ID; multiple matching
|
||||
versions remain an ambiguous source-selection error.
|
||||
|
||||
Lookup uses the engine's normally selected prompt source:
|
||||
|
||||
- the last `WithPromptFS` or `WithPromptFile` option replaces earlier prompt
|
||||
source options and `Config.PromptDir`; or
|
||||
- `Config.PromptDir` supplies the source when no prompt source option is
|
||||
present.
|
||||
|
||||
Inspection does not merge prompt sources, fall back after a malformed match,
|
||||
or introduce a new public prompt repository. The
|
||||
[format reference](../formats.md#prompt-definitions) remains the canonical
|
||||
owner of prompt source and exact selection behavior.
|
||||
|
||||
## Structural Validation
|
||||
|
||||
A successful inspection proves that the selected definition can be loaded
|
||||
through the ordinary prompt repository and satisfies its source-level
|
||||
structural rules. This includes:
|
||||
|
||||
- deterministic YAML discovery and exact definition selection;
|
||||
- strict decoding and validation of required identity, version, messages, and
|
||||
output fields;
|
||||
- normalization and validation of input declarations, message roles, cache
|
||||
control, the default-profile identifier, and output-contract fields; and
|
||||
- contained, readable resolution of every referenced message
|
||||
`content_file`.
|
||||
|
||||
Referenced message content participates in validation and hashing but is not
|
||||
returned. Inspection does not parse or execute Go templates, resolve template
|
||||
variables or input helpers, or enforce the runtime presence and media type of
|
||||
declared inputs. Those checks remain part of rendering and preparation.
|
||||
|
||||
The returned `OutputContract` is the normalized contract declared by the
|
||||
prompt definition. For `json_schema` validation, structural inspection proves
|
||||
that a non-blank schema path is declared and returns that path. It does not
|
||||
open, resolve, compile, or return the schema document or its references.
|
||||
Schema-source validation remains part of executable preparation and any
|
||||
future explicit full-source validation feature.
|
||||
|
||||
Inspection returns `DefaultProfileID` as declared metadata. It does not look
|
||||
up that profile, resolve a backend or execution target, check credentials, or
|
||||
prove that the profile is executable. Consumers may call `InspectProfile`
|
||||
separately when their application policy requires that additional check.
|
||||
|
||||
## Result Semantics
|
||||
|
||||
The result exposes the complete declared input metadata in definition order:
|
||||
name, required status, content type, and human-readable description. It does
|
||||
not expose message templates, session-ID templates, cache-control details,
|
||||
prompt descriptions, raw YAML, source paths, rendered messages, or schema
|
||||
bodies.
|
||||
|
||||
`PromptID` and `PromptVersion` are the validated identity read from the
|
||||
selected definition. `PromptHash` is an opaque equality value for the complete
|
||||
loaded definition, including referenced message content. For the same selected
|
||||
definition and observed source state, it must equal the `PreparedRun.PromptHash`
|
||||
produced by ordinary preparation.
|
||||
|
||||
Consumers may compare `PromptHash` values but must not depend on their
|
||||
algorithm, encoding, length, or suitability as a security proof. Promptkit may
|
||||
change the representation in a future release together with the existing
|
||||
prepared-run hash contract.
|
||||
|
||||
## Ownership, Consistency, And Concurrency
|
||||
|
||||
Each successful call returns a caller-owned snapshot. Mutating the result or
|
||||
its input slice cannot affect the engine, later inspections, or later
|
||||
preparation.
|
||||
|
||||
Inspection is safe to call concurrently under the engine's existing
|
||||
repository contracts. It does not mutate prompt sources or install a global
|
||||
cache.
|
||||
|
||||
For filesystem-backed and mutable `fs.FS` sources, the result describes the
|
||||
state observed by that lookup. It does not freeze the definition for a later
|
||||
`Prepare`, `PrepareExecution`, or `Run`, and a source may change between
|
||||
operations. Consumers requiring an exact preflight-to-execution snapshot
|
||||
should use the existing prepared-execution workflow.
|
||||
|
||||
## Errors And Cancellation
|
||||
|
||||
The operation uses existing public error categories:
|
||||
|
||||
- a blank or whitespace-only prompt ID matches `ErrInvalidRequest`;
|
||||
- an absent exact ID or version matches `ErrPromptNotFound` and not
|
||||
`ErrPromptLoad`;
|
||||
- read, strict-decode, validation, duplicate, ambiguity, referenced-content,
|
||||
and prompt-hashing failures match `ErrPromptLoad`; and
|
||||
- a nil engine receiver matches `ErrInvalidConfig`.
|
||||
|
||||
Errors preserve useful underlying collaborator and context identities through
|
||||
`errors.Is` where the existing facade does so, without exposing internal
|
||||
package types. Context cancellation governs the complete lookup, and no
|
||||
partial inspection result is returned on failure.
|
||||
|
||||
Inspection cannot return profile, credential, artifact, rendering, schema,
|
||||
capacity, validation, or model-generation errors because it does not perform
|
||||
those operations.
|
||||
|
||||
## Compatibility And Boundaries
|
||||
|
||||
This feature is additive. Existing prompt formats, prompt source selection,
|
||||
`Prepare`, prepared execution, `Run`, profile inspection, and stable JSON
|
||||
contracts remain unchanged.
|
||||
|
||||
The method belongs on the root `Engine` facade. Prompt repositories and
|
||||
internal domain definitions remain implementation details, and consumers
|
||||
cannot use inspection to replace or mutate registered prompt definitions.
|
||||
Engine construction retains its ordinary configuration requirements; this
|
||||
feature does not introduce a separate prompt-only engine.
|
||||
|
||||
Inspection is intentionally definition-oriented rather than
|
||||
execution-oriented. It reports what the prompt declares, not an effective
|
||||
request after profile selection, per-run overrides, artifacts, variables,
|
||||
schema resolution, or rendering.
|
||||
|
||||
## Documentation
|
||||
|
||||
When implemented, documentation should retain these ownership boundaries:
|
||||
|
||||
- exported declarations and GoDoc own the exact method, result, ownership,
|
||||
hash, error, and cancellation contracts;
|
||||
- the Promptkit consumer guide explains configuration-time prompt inspection
|
||||
and distinguishes it from profile inspection, `Prepare`, and prepared
|
||||
execution;
|
||||
- the format reference continues to own prompt fields, source selection,
|
||||
content-file resolution, and version behavior; and
|
||||
- internal documentation describes shared prompt loading and hashing without
|
||||
duplicating the public contract.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This work does not include:
|
||||
|
||||
- enumerating, searching, or filtering prompt definitions;
|
||||
- validating every definition across one or more prompt sources;
|
||||
- returning raw definitions, YAML, source paths, message bodies, rendered
|
||||
messages, prompt descriptions, or session-ID templates;
|
||||
- parsing or rendering templates or accepting placeholder inputs and
|
||||
variables;
|
||||
- resolving the default profile, backend, effective execution target, or
|
||||
credentials;
|
||||
- loading, compiling, or returning JSON Schema documents or transitive
|
||||
references;
|
||||
- validating application-specific relationships among prompts;
|
||||
- freezing a mutable prompt source for later execution;
|
||||
- exposing or mutating an internal prompt repository;
|
||||
- adding a stable JSON representation for the new inspection values; or
|
||||
- changing current prompt selection, hashing, preparation, or execution
|
||||
behavior.
|
||||
|
||||
## Target End State
|
||||
|
||||
After this work:
|
||||
|
||||
- consumers can validate one configured prompt without inventing artifacts,
|
||||
variables, or an executable profile;
|
||||
- lookup uses ordinary prompt-source selection and exact ID/version semantics;
|
||||
- a successful result proves that the selected definition and its referenced
|
||||
message content are structurally loadable;
|
||||
- callers receive validated identity, complete declared input metadata, the
|
||||
default profile ID, the normalized output contract, and an opaque prompt
|
||||
equality value;
|
||||
- the equality value matches ordinary preparation for the same selected
|
||||
definition and observed source state;
|
||||
- no prompt body, rendered message, schema body, credential, or execution
|
||||
target is exposed;
|
||||
- returned values are caller-owned and safe to inspect concurrently; and
|
||||
- broader corpus validation, enumeration, schema validation, and executable
|
||||
preparation remain separate responsibilities.
|
||||
@@ -7,10 +7,9 @@ additions to PromptKit from the perspective of the maintainers of
|
||||
Weatherreporter, a downstream application planning to replace its Scriptorium
|
||||
CLI integration with PromptKit.
|
||||
|
||||
PromptKit v0.3.0 provides the capabilities Weatherreporter needs for the
|
||||
migration. None of the ideas below is a hard adoption requirement. They are
|
||||
opportunities to avoid duplicate preparation, validate configuration earlier,
|
||||
improve durable failure diagnostics, and make the integration more direct.
|
||||
PromptKit now provides the capabilities Weatherreporter needs for the
|
||||
migration. The remaining deferred ideas are optional opportunities to validate
|
||||
configuration earlier and improve durable failure diagnostics.
|
||||
|
||||
The examples are API sketches intended to communicate the desired capability,
|
||||
not prescriptive names or finalized Go contracts. The related
|
||||
@@ -19,8 +18,9 @@ overlapping features from another downstream consumer's perspective.
|
||||
|
||||
## Priority 1: Executable Preparation Handles
|
||||
|
||||
**Disposition:** Accepted into the
|
||||
[executable preparation handles](prepared-execution.md) feature roadmap.
|
||||
**Disposition:** Implemented as [`Engine.PrepareExecution` and
|
||||
`Engine.RunPrepared`](../../engine.go). See the
|
||||
[consumer guidance](../consumers/pkg-promptkit.md#prepare-now-and-execute-the-same-snapshot-later).
|
||||
|
||||
### Downstream need
|
||||
|
||||
@@ -35,74 +35,20 @@ needs to:
|
||||
Persisting preflight before generation leaves useful evidence when a provider
|
||||
call fails or the process is interrupted during generation.
|
||||
|
||||
### Current integration option
|
||||
### Implemented behavior
|
||||
|
||||
With PromptKit v0.3.0, Weatherreporter can call `Engine.Prepare`, save selected
|
||||
fields from the returned `PreparedRun`, and then call `Engine.Run` with the
|
||||
same request. Because `Run` performs preparation internally, the work is
|
||||
repeated.
|
||||
|
||||
Weatherreporter plans to use embedded prompt and schema files plus immutable
|
||||
inline input bytes, which removes most of the consistency risk. An external
|
||||
profile file or directory can still change between the two calls, and the
|
||||
second preparation remains unnecessary work.
|
||||
|
||||
The atomic `RunDetailed` operation proposed by the
|
||||
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-1-atomic-execution-with-prepared-details)
|
||||
would guarantee that returned preparation details describe the completed
|
||||
execution. However, returning those details only after generation would not
|
||||
preserve Weatherreporter's preflight-before-generation persistence boundary.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Add an opt-in two-phase API that returns a prepared execution handle:
|
||||
|
||||
```go
|
||||
prepared, err := engine.PrepareExecution(ctx, request)
|
||||
if err != nil {
|
||||
// Handle preparation failure.
|
||||
}
|
||||
|
||||
details := prepared.Details()
|
||||
// Persist a consumer-selected safe preparation record.
|
||||
|
||||
result, err := engine.RunPrepared(ctx, prepared)
|
||||
```
|
||||
|
||||
The exact names and shapes are flexible. The important contract is that
|
||||
`RunPrepared` executes the already prepared prompt and does not reload or
|
||||
rerender its prompt, profile, schema, or input sources.
|
||||
|
||||
`Details` should return the same caller-owned public preparation information
|
||||
currently represented by `PreparedRun`. The execution handle may retain opaque
|
||||
engine-owned state needed to invoke the model and validate the response.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Keep `Prepare` and `Run` available for consumers that do not need a
|
||||
two-phase execution boundary.
|
||||
- Bind a prepared handle to the engine that constructed it.
|
||||
- Define whether a handle is one-shot, reusable, or safe for concurrent use.
|
||||
A one-shot contract may be the safest initial design.
|
||||
- Do not give the opaque handle a stable JSON representation.
|
||||
- Do not expose or serialize resolved credential values through `Details`.
|
||||
- Define how a direct request API key is retained and released when an opaque
|
||||
handle must carry it until execution.
|
||||
- Preserve caller-owned copies for all public details.
|
||||
- Make context cancellation and backend admission timing explicit.
|
||||
- Document whether profile credential environment values are resolved during
|
||||
preparation or execution.
|
||||
- Ensure an execution error does not invalidate the public details already
|
||||
returned to the consumer.
|
||||
- Consider whether an atomic `RunDetailed` can share the same internal
|
||||
prepared-execution implementation.
|
||||
Weatherreporter can prepare one frozen execution snapshot, persist a
|
||||
caller-owned and credential-redacted `Details` value, and execute that same
|
||||
snapshot through `RunPrepared`. The opaque handle is engine-bound and
|
||||
single-use; an unused handle can be released with `Discard`. The consumer
|
||||
guide and exported GoDoc own the exact lifecycle, credential, cancellation,
|
||||
and capacity contracts.
|
||||
|
||||
### Value to Weatherreporter
|
||||
|
||||
This is the highest-value upstream addition. It would preserve
|
||||
Weatherreporter's durable preflight behavior, remove duplicate work, eliminate
|
||||
the remaining source-consistency window, and ensure that persisted provenance
|
||||
describes the actual execution.
|
||||
This preserves Weatherreporter's durable preflight behavior, removes duplicate
|
||||
work, eliminates the source-consistency window, and ensures that persisted
|
||||
provenance describes the actual execution.
|
||||
|
||||
## Priority 2: Prompt-Definition Inspection
|
||||
|
||||
@@ -312,12 +258,13 @@ migration priority.
|
||||
|
||||
## Capabilities PromptKit Already Provides Well
|
||||
|
||||
PromptKit v0.3.0 already provides the essential Weatherreporter integration
|
||||
surface:
|
||||
PromptKit already provides the essential Weatherreporter integration surface:
|
||||
|
||||
- importable in-process engine construction;
|
||||
- filesystem, `fs.FS`, and programmatic prompt, profile, and schema sources;
|
||||
- offline preparation without model execution;
|
||||
- prepared execution handles for a durable preflight boundary;
|
||||
- exact prompt and profile inspection;
|
||||
- versioned prompt selection;
|
||||
- text, Markdown, JSON, and JSON Schema output contracts;
|
||||
- single-pass output validation with raw output retained after completed
|
||||
@@ -329,7 +276,8 @@ surface:
|
||||
- injected model-client and artifact-reader interfaces;
|
||||
- caller cancellation, generation timeout, and transport timeout behavior; and
|
||||
- public error sentinels for configuration, prompt, profile, artifact,
|
||||
validation, capacity, and generation failures.
|
||||
validation, capacity, and generation failures; and
|
||||
- structured backend identity for capacity rejection.
|
||||
|
||||
These capabilities are sufficient for Weatherreporter to adopt PromptKit
|
||||
without waiting for new upstream work.
|
||||
@@ -354,41 +302,30 @@ PromptKit:
|
||||
|
||||
## Suggested Upstream Sequence
|
||||
|
||||
If the PromptKit team chooses to pursue these ideas, the most useful order for
|
||||
Weatherreporter would be:
|
||||
For downstream adoption and any remaining upstream work, the useful order is:
|
||||
|
||||
1. Add executable preparation handles, ideally sharing implementation with an
|
||||
atomic detailed-run API.
|
||||
1. Adopt the implemented executable preparation handles.
|
||||
2. Consider eager source validation after evaluating whether the two exact
|
||||
inspection APIs are sufficient.
|
||||
3. Add structured generation errors.
|
||||
4. Use structured capacity errors and consider semantic execution-target
|
||||
fingerprints as lower-priority operational improvements.
|
||||
|
||||
The first item removes the only material integration workaround. Prompt and
|
||||
profile inspection improve fail-fast validation. The remaining items improve
|
||||
ergonomics and diagnostics.
|
||||
The first item removes the material integration workaround. Prompt and profile
|
||||
inspection improve fail-fast validation. The remaining items are optional
|
||||
ergonomic and diagnostic improvements.
|
||||
|
||||
## Adoption Sequencing
|
||||
|
||||
Weatherreporter should not wait for the complete wishlist. PromptKit v0.3.0 is
|
||||
already sufficient when Weatherreporter:
|
||||
Weatherreporter should not wait for the deferred wishlist items. The current
|
||||
PromptKit interface is sufficient when Weatherreporter:
|
||||
|
||||
- embeds immutable prompt and schema assets;
|
||||
- supplies immutable inline data-package bytes;
|
||||
- constructs one engine per CLI invocation;
|
||||
- calls `Prepare` and `Run` with the same request; and
|
||||
- prepares an execution, persists selected `Details`, and calls
|
||||
`RunPrepared`; and
|
||||
- keeps PromptKit behind a weatherreporter-owned adapter contract.
|
||||
|
||||
If executable preparation handles are scheduled for a near-term PromptKit
|
||||
release, Weatherreporter may defer only its final adapter implementation to
|
||||
avoid implementing and then removing duplicate preparation. Prompt corpus
|
||||
retrieval, application-contract design, configuration work, embedded assets,
|
||||
state contracts, and offline fixtures can proceed independently.
|
||||
|
||||
If the feature is not scheduled, Weatherreporter can adopt v0.3.0 and keep the
|
||||
duplicate `Prepare` and `Run` sequence inside its adapter. A later PromptKit
|
||||
upgrade would remain localized behind that neutral boundary.
|
||||
|
||||
Prompt inspection, profile inspection, source validation, structured errors,
|
||||
capacity details, and semantic fingerprints should not gate adoption.
|
||||
|
||||
Reference in New Issue
Block a user