From bd6cffc9d0d97a26b521f89de3421836cd268507 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 30 Jul 2026 23:48:25 +0000 Subject: [PATCH] Prepare documentation for Promptkit v0.4.0 --- README.md | 3 + docs/releases/v0.4.0.md | 189 +++++++ docs/roadmap/capacity-errors.md | 262 --------- docs/roadmap/concurrency.md | 237 -------- docs/roadmap/implementation.md | 520 ------------------ docs/roadmap/local-backend.md | 163 ------ docs/roadmap/notarius-promptkit-wishlist.md | 79 +-- docs/roadmap/prepared-execution.md | 280 ---------- docs/roadmap/profile-inspection.md | 246 --------- docs/roadmap/prompt-inspection.md | 268 --------- .../weatherreporter-promptkit-wishlist.md | 123 +---- 11 files changed, 240 insertions(+), 2130 deletions(-) create mode 100644 docs/releases/v0.4.0.md delete mode 100644 docs/roadmap/capacity-errors.md delete mode 100644 docs/roadmap/concurrency.md delete mode 100644 docs/roadmap/implementation.md delete mode 100644 docs/roadmap/local-backend.md delete mode 100644 docs/roadmap/prepared-execution.md delete mode 100644 docs/roadmap/profile-inspection.md delete mode 100644 docs/roadmap/prompt-inspection.md diff --git a/README.md b/README.md index a17fffa..d95abf3 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,9 @@ boundary and constraints that framework work must preserve. ## Release Guidance +Consumers upgrading from `v0.3.0` to `v0.4.0` should read the +[v0.4.0 changelog and adoption guide](docs/releases/v0.4.0.md). + Consumers upgrading from `v0.2.0` to `v0.3.0` should read the [v0.3.0 changelog](docs/releases/v0.3.0.md). diff --git a/docs/releases/v0.4.0.md b/docs/releases/v0.4.0.md new file mode 100644 index 0000000..0e591ce --- /dev/null +++ b/docs/releases/v0.4.0.md @@ -0,0 +1,189 @@ +# Promptkit v0.4.0 + +This supplemental changelog and adoption guide summarizes the consumer-facing +changes from `v0.3.0` to `v0.4.0`. The annotated `v0.4.0` tag is the +authoritative release record. Exact current contracts belong to the linked +GoDoc and durable documentation. + +## Summary + +`v0.4.0` adds four complementary capabilities: + +- opaque prepared-execution handles for preparing once, inspecting safe + details, and executing the same frozen snapshot; +- exact prompt-definition inspection without profile resolution or execution; +- exact profile inspection without selecting a prompt or checking credential + availability; and +- structured backend identity on engine admission-capacity rejection. + +These APIs let consumers perform more precise preflight work and retain useful +operational context without reproducing Promptkit's internal resolution logic. + +## Compatibility + +The release is additive for `v0.3.0` consumers. Existing uses of `Prepare`, +`Run`, backend registration, endpoint-only profiles, local-backend helpers, +runtime overrides, public JSON values, and error sentinels continue to work +without migration. + +Capacity rejection now returns a structured error while continuing to match +`ErrCapacityExceeded` through `errors.Is`. Error-string wording and direct +sentinel equality were not public contracts. + +The new inspection values, capacity error, and prepared-execution handle do not +have stable JSON representations. `PreparedExecution.Details` returns the +existing stable `PreparedRun` value. + +## Upgrade + +Update the module dependency with: + +```sh +go get gitea.maximumdirect.net/eric/promptkit@v0.4.0 +go mod tidy +``` + +Run the consuming project's ordinary and race-enabled tests after upgrading. +No source migration is required. + +## Prepare Once And Execute The Same Snapshot + +Consumers that need to persist preparation details before generation can now +prepare an opaque, engine-bound execution: + +```go +prepared, err := engine.PrepareExecution(ctx, request) +if err != nil { + // Handle preparation failure. +} +defer prepared.Discard() + +details := prepared.Details() +// Persist a consumer-selected, appropriately protected preparation record. + +result, err := engine.RunPrepared(ctx, prepared) +``` + +Preparation freezes the selected sources, rendered messages, effective +settings, input content, structured-output metadata, and validation resources +needed by execution. `Details` returns a fresh, caller-owned, +credential-redacted `PreparedRun`. + +A handle belongs to its creating engine and permits one execution attempt. +`RunPrepared` consumes that attempt on success and on operational failure. +`Discard` is idempotent and releases an unclaimed handle's execution-only +state. Consumers should discard handles they will not execute, particularly +when a direct request API key may be retained privately until claim or +discard. + +Prepared execution does not reserve backend admission during preparation. +Credential availability and backend admission are checked when execution +begins. The execution context is independent of the preparation context. + +See the +[prepared-execution consumer guide](../consumers/pkg-promptkit.md#prepare-now-and-execute-the-same-snapshot-later), +the [`PreparedExecution` GoDoc](../../prepared_execution.go), and the +[`Engine.PrepareExecution` and `Engine.RunPrepared` GoDoc](../../engine.go) +for the exact lifecycle, ownership, cancellation, capacity, timing, and +failure contracts. + +## Inspect A Prompt + +`Engine.InspectPrompt` resolves one prompt ID and optional version through the +engine's configured prompt source: + +```go +inspection, err := engine.InspectPrompt(ctx, "report.summary", "") +``` + +The result includes prompt identity, the opaque prompt hash, declared default +profile ID, declared input metadata, and normalized output contract. It +structurally loads the selected definition and referenced message content but +does not resolve a profile, load schemas or artifacts, render templates, +reserve capacity, or contact a model. + +Use inspection for exact configuration checks and metadata discovery. Use +`PrepareExecution` rather than relying on a prior inspection when later +execution must freeze one exact source state, because filesystem-backed +inspection is only a point-in-time lookup. + +See the +[prompt-inspection consumer guide](../consumers/pkg-promptkit.md#inspect-a-prompt-before-preparation) +and [`Engine.InspectPrompt` GoDoc](../../engine.go) for exact selection, +ownership, and error behavior. + +## Inspect A Profile + +`Engine.InspectProfile` resolves one explicit profile independently of a +prompt: + +```go +inspection, err := engine.InspectProfile(ctx, "report-production") +``` + +The result includes the resolved effective execution target and whether a +later request must provide a direct credential. Environment-variable names may +be reported, but inspection does not read credential values or require the +named variable to be populated. + +Inspection applies the ordinary configured and built-in profile precedence and +resolves any selected backend. It does not load a prompt, render content, +reserve capacity, or contact a model. + +See the +[profile-inspection consumer guide](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work) +and [`Engine.InspectProfile` GoDoc](../../engine.go) for the exact resolution, +credential, ownership, and error contracts. + +## Identify Capacity-Rejected Backends + +Calls rejected at Promptkit's bounded engine admission boundary continue to +match `ErrCapacityExceeded`. Consumers can additionally obtain the selected +registered backend ID without parsing diagnostic text: + +```go +result, err := engine.Run(ctx, request) +if errors.Is(err, promptkit.ErrCapacityExceeded) { + var capacityErr *promptkit.CapacityError + if errors.As(err, &capacityErr) { + // Record capacityErr.BackendID using application-owned diagnostics. + } + + // Apply application-owned overload or retry policy. +} +``` + +The structured error applies to `Run` and `RunPrepared` admission rejection. +It does not represent provider throttling, quota exhaustion, cancellation +while waiting for generation capacity, or another model-client failure. +Promptkit does not prescribe retry timing or transport status mapping. + +See the +[error-handling consumer guide](../consumers/pkg-promptkit.md#handle-errors), +the [`CapacityError` GoDoc](../../capacity_error.go), and the +[`ErrCapacityExceeded` GoDoc](../../engine.go) for the canonical contracts. + +## Public API Additions + +The release adds: + +- `Engine.PrepareExecution`; +- `Engine.RunPrepared`; +- `PreparedExecution`, including `Details`, `Discard`, `String`, and + `GoString`; +- `Engine.InspectPrompt`; +- `PromptInspection`; +- `PromptInputDefinition`; +- `Engine.InspectProfile`; +- `ProfileInspection`; and +- `CapacityError`. + +No public API was removed. + +## Consumer Action + +None. Existing `v0.3.0` workflows may upgrade without adopting the new APIs. + +Consumers that adopt prepared execution should discard unused handles. +Consumers that need backend-specific capacity diagnostics may add an +`errors.As` check while retaining their existing `errors.Is` classification. diff --git a/docs/roadmap/capacity-errors.md b/docs/roadmap/capacity-errors.md deleted file mode 100644 index cc3d15e..0000000 --- a/docs/roadmap/capacity-errors.md +++ /dev/null @@ -1,262 +0,0 @@ -# Structured Capacity Errors - -**Status:** Complete. - -## Purpose - -Allow consumers to identify which registered backend rejected engine -admission without parsing an error string, while preserving the existing -`ErrCapacityExceeded` classification and Promptkit's provider-neutral error -boundary. - -This provides safe operational context requested by -[Notarius](notarius-promptkit-wishlist.md#priority-4-structured-capacity-errors) -and -[Weatherreporter](weatherreporter-promptkit-wishlist.md#structured-capacity-errors). - -## Motivation - -Promptkit currently returns an error matching `ErrCapacityExceeded` when a -limited backend's active and waiting admission capacity is full. That sentinel -lets consumers classify the failure reliably, but the selected backend ID is -available only in diagnostic text. - -Applications using multiple backends need the stable backend ID for -operator-facing diagnostics, metrics labels, failure receipts, and their own -retry decisions. Parsing `Error()` text would turn non-contractual wording -into an accidental API and risks retaining more diagnostic context than the -application needs. - -Promptkit already owns backend selection and the admission boundary. It should -attach the selected backend's safe identity at that boundary and translate it -into a small public typed error. - -## Consumer Workflow - -The target public workflow is: - -```go -result, err := engine.Run(ctx, request) -if err != nil { - var capacityErr *promptkit.CapacityError - if errors.As(err, &capacityErr) { - // Record capacityErr.BackendID or apply application retry policy. - } -} -``` - -Existing classification remains valid: - -```go -if errors.Is(err, promptkit.ErrCapacityExceeded) { - // Handle engine admission rejection without structured context. -} -``` - -The target public surface is: - -```go -type CapacityError struct { - BackendID string -} - -func (e *CapacityError) Error() string -func (e *CapacityError) Unwrap() error -``` - -`Unwrap` returns `ErrCapacityExceeded`. Engine-produced capacity errors -therefore support both `errors.As` to `*CapacityError` and -`errors.Is(err, ErrCapacityExceeded)`, including through additional wrapping. - -The exported declaration and GoDoc will own the exact implemented contract. -`CapacityError` does not need a stable JSON representation, an exported -constructor, or fields beyond `BackendID`. - -## Error Boundary - -A `CapacityError` represents one specific condition: `Run` or `RunPrepared` -was rejected at Promptkit's engine-local bounded admission boundary because -the selected limited backend had no remaining admitted slot. - -It does not represent: - -- waiting for an active-generation permit after admission; -- caller cancellation or deadline expiration while acquiring or holding - capacity; -- provider rate limiting, quota exhaustion, HTTP 429 responses, or another - model-client failure; -- validation, output repair, or application-level queue rejection; or -- an application's decision not to submit or retry work. - -Provider and injected-client errors retain their existing generation error -classification even when a provider describes its failure as capacity or rate -limiting. Promptkit must not infer an engine admission error from provider -status codes or error text. - -Admission continues to check caller cancellation before reporting a full -pool. A cancellation that wins that decision remains a context-related -execution error and does not become a `CapacityError`. - -## Backend Identity - -`BackendID` is the normalized stable ID of the registered backend whose pool -rejected admission. It is the same effective backend identity used for -capacity routing and exposed by prepared and run metadata. - -For ordinary `Run`, the ID comes from the backend selected by the resolved -prompt, profile, and execution target. For `RunPrepared`, it comes from the -backend identity frozen in the prepared execution. - -An endpoint override does not change backend identity and therefore does not -change the reported `BackendID`. Endpoint-only profiles and unlimited -backends have no bounded admission pool and cannot produce an engine-generated -`CapacityError`. - -Every engine-produced `CapacityError` has a nonblank registered backend ID. -The type does not report a profile ID, prompt ID, model, endpoint, run ID, or -prepared-handle identity. - -## Compatibility - -`ErrCapacityExceeded` remains the stable sentinel for broad classification. -This feature refines the error value returned by capacity-rejected `Run` and -`RunPrepared` calls without replacing or removing that sentinel. - -The following behaviors remain unchanged: - -- `errors.Is(err, ErrCapacityExceeded)` succeeds for engine admission - rejection; -- capacity rejection remains distinct from `ErrInvalidRequest`, - `ErrLLMGenerate`, and other public categories; -- `Run` returns no partial result after admission rejection; -- `RunPrepared` retains its existing one-attempt lifecycle, including that a - claimed handle is consumed when admission is rejected; -- admission ordering, limits, queue capacity, leases, release behavior, and - backend independence are unchanged; and -- `Prepare`, `PrepareExecution`, `InspectPrompt`, and `InspectProfile` perform - no admission and cannot return this error. - -Direct equality with the sentinel is not introduced as a contract. Consumers -continue to use `errors.Is` for category checks and `errors.As` for structured -context. - -## Sensitive Data And Diagnostics - -The typed error contains only `BackendID`. It must not expose or retain: - -- backend endpoints; -- credential environment-variable names or credential values; -- model, profile, prompt, session, or request identifiers; -- prompt, input, rendered, generated, or validation content; -- concurrency limits, queue capacities, current counts, or queue depth; -- request position or identities of other admitted work; or -- retry timing, provider health, or speculative availability. - -`Error()` may include the quoted backend ID and a concise admission-rejection -description. Its wording is diagnostic and not a stable parsing contract. -Default formatting, wrapping, and Go string formatting must not reveal -anything beyond the public field and generic capacity classification. - -Concurrency and queue configuration remain available where consumers define -their backends; repeating them on each error would add stale or unnecessary -operational detail. Current counts are race-prone observations rather than a -durable explanation of when capacity will become available. - -## Retry And Application Policy - -Promptkit reports that admission was rejected but does not retry, wait for a -new admission slot, calculate backoff, or declare an HTTP status. - -A full in-process admission pool does not provide a meaningful retry delay: -lease duration depends on consumer inputs, model latency, validation, repair, -cancellation, and provider behavior. Consumers own whether to retry, which -backend to use, how to schedule or back off, what to expose to operators, and -how to translate the error into CLI or transport behavior. - -The absence of retry metadata is deliberate. A future retry facility would -require its own roadmap and application-neutral policy rather than extending -this error speculatively. - -## Ownership And Concurrency - -Each rejected operation returns its own caller-owned `*CapacityError`. The -error does not reference mutable pool state or retain the request. Reading it -requires no engine access and remains safe after the rejected call returns. - -Mutating the exported `BackendID` on a returned error affects only that error -value. It cannot change the backend registry, capacity manager, another -operation's error, or later routing and admission. - -Constructing or returning the typed error requires no new process-global -state, cache, goroutine, or synchronization beyond existing capacity -management. - -## Architectural Boundaries - -The root `promptkit` package owns the exported error type, sentinel -compatibility, and public translation. The engine's internal capacity and -use-case components remain responsible for admission and for carrying the -selected backend identity to the facade. - -Internal error types or capacity-manager state must not cross the public -boundary. The public facade must obtain structured identity through an -internal typed or otherwise non-textual error contract; it must not parse the -runner's diagnostic string. - -The feature does not introduce a public capacity manager, backend registry, -queue, scheduler, metrics collector, or transport-specific error. It remains a -small refinement of the existing root error boundary. - -## Documentation - -When implemented, documentation should retain these ownership boundaries: - -- the exported declaration and GoDoc own the exact type, field, `Error`, - `Unwrap`, `errors.Is`, `errors.As`, and formatting contracts; -- the consumer guide demonstrates broad classification and optional backend - extraction without duplicating every error guarantee; -- internal capacity and runner documentation describe where backend identity - is attached and translated; -- the backend API continues to own capacity configuration; and -- the architecture and documentation policies continue to own library, - application, and documentation boundaries. - -## Non-Goals - -This work does not include: - -- changing concurrency limits, queue capacities, FIFO scheduling, or default - capacity policy; -- exposing configured limits, current counts, queue depth, or waiter - positions; -- reporting a retry delay, retryable boolean, HTTP status, CLI exit code, or - backoff policy; -- automatically retrying or rerouting rejected work; -- unifying provider rate limits or quota failures with engine admission; -- adding structured context to generation, validation, prompt, profile, or - other error categories; -- returning partial preparation or run results with the error; -- changing prepared-execution handle lifecycle; -- adding stable JSON for errors; -- exposing internal capacity or use-case types; or -- adding application logging, metrics emission, persistence, or failure - receipts. - -## Target End State - -After this work: - -- every engine admission rejection from `Run` or `RunPrepared` returns an - error discoverable as `*CapacityError`; -- the typed error contains the normalized selected backend ID and no other - structured field; -- existing `errors.Is(err, ErrCapacityExceeded)` handling continues to work; -- `errors.As` obtains backend identity without parsing diagnostic text; -- capacity rejection remains distinct from cancellation, model-client - failures, provider throttling, and application scheduling; -- endpoints, credentials, request content, capacity metrics, and speculative - retry information remain absent; -- retry, backoff, rerouting, metrics, persistence, and transport mapping remain - consumer responsibilities; and -- the existing capacity architecture and runtime behavior are otherwise - unchanged. diff --git a/docs/roadmap/concurrency.md b/docs/roadmap/concurrency.md deleted file mode 100644 index c6e9c33..0000000 --- a/docs/roadmap/concurrency.md +++ /dev/null @@ -1,237 +0,0 @@ -# Backend-Specific Concurrency Management - -**Status:** Complete. - -## Purpose - -This roadmap defines the scope and target end state for engine-local, -backend-specific concurrency management. It records the intended capability, -consumer value, and important policy choices. - -This document is planning material, not a description of current behavior. -Current exported contracts remain owned by Go declarations and GoDoc, backend -registration guidance by the -[consumer guide](../consumers/pkg-promptkit.md#configure-a-local-openai-compatible-endpoint), -and -implemented orchestration by the -[internal runner document](../internal/runner.md). - -## Motivation - -Different model backends can sustain very different request loads. A local -network endpoint may need a small concurrency limit, while OpenRouter can -usually accept substantially more simultaneous work. Requiring every consumer -to build its own semaphores and queues would duplicate routing knowledge, -create inconsistent cancellation behavior, and make it easy for one caller to -bypass the intended backend limit. - -Promptkit should own this coordination because it already resolves each run to -an engine-scoped backend identity and owns every model-generation call made by -the runner. Consumers should continue submitting ready-to-run requests through -the synchronous API, including concurrently from multiple goroutines, without -implementing their own backend scheduler. - -The buffered queue is a safety boundary, not an ordinary throughput -restriction. Its primary purpose is to prevent a bug or unintended submission -loop from creating an unbounded in-memory backlog. - -## Scope - -The feature will add optional concurrency policy to registered backends and -coordinate `Run` calls against independent per-backend capacity pools. - -Each policy has two distinct controls: - -- an active-generation limit, which protects the backend from too many - simultaneous model requests; and -- a bounded waiting capacity, which protects the process from admitting an - unbounded backlog. - -Concurrency policy belongs to a backend registration. It is not a profile -model parameter and cannot be overridden per run. Profiles select the policy -through their backend ID, while a profile or request endpoint override remains -in the selected backend's pool. - -`Prepare` does not call a model and will remain outside concurrency admission. - -## Defaults And Configuration - -The built-in OpenRouter backend will use: - -- an active-generation limit of 16; and -- a waiting capacity of 1024. - -The waiting default is intentionally generous. Reaching it should indicate -abnormal submission pressure rather than normal application behavior. - -Consumer-registered backends will remain unlimited unless the consumer -configures an active-generation limit. When a consumer enables a limit and -does not specify waiting capacity, the waiting capacity will default to 1024. -Consumers may configure a different bounded capacity, including zero when -they want no admitted backlog beyond the active-limit-sized run set. - -The public representation must distinguish an omitted waiting capacity from -an explicit zero. - -Endpoint-only profiles have no backend registration from which to obtain -policy and will remain unlimited. A future engine-wide or endpoint-keyed -policy can be considered separately if consumers demonstrate that need. - -Invalid limits or capacities will fail engine construction as invalid -configuration. Policy values will be copied into engine-owned immutable state -along with the rest of the backend registration. - -## Admission And Execution Behavior - -`Run` remains a synchronous, wait-for-result operation. Concurrent callers may -block inside `Run` while waiting for their selected backend, then receive the -ordinary result or error from that invocation. - -For a configured pool, the active-generation limit plus the waiting capacity -defines the maximum number of concurrent `Run` invocations that Promptkit will -accept for that backend. A waiting capacity of zero therefore accepts no more -runs than the active limit. Admission is immediate: a call either reserves one -of those bounded slots or receives the capacity error. An accepted run may -then wait internally for active-generation capacity. - -For a limited backend, Promptkit will bound the number of accepted runs before -expensive artifact loading, prompt rendering, and large defensive copies where -practical. Lightweight prompt, profile, and backend resolution may occur first -when it is required to identify the correct capacity pool. This pre-admission -resolution must not become a second execution-precedence path with behavior -that can drift from `Prepare`. - -An accepted run retains its admission until it completes or fails. Every -actual model-generation call for that run must separately observe the -backend's active-generation limit. This includes: - -- the initial generation; -- every output-repair generation; and -- calls made through either the built-in or an injected model client. - -Preparation and output validation should not hold an active-generation permit. -A repair remains part of its already-admitted run, but reacquires active -generation capacity so repairs cannot exceed the backend limit. It must not be -rejected merely because new runs filled the waiting queue after its initial -generation. - -Within one backend pool, waiting generation calls should be served in FIFO -order, subject to canceled calls being removed. Different backend pools make -progress independently; a saturated local backend must not consume -OpenRouter's active or waiting capacity. - -The feature will not promise ordering across backend pools or completion order -among admitted runs. - -## Capacity Failure And Cancellation - -When a backend's bounded waiting capacity is full, a new `Run` call will fail -promptly rather than waiting outside the bounded admission system. The public -API will expose a recognizable capacity-exhaustion error identity distinct -from invalid configuration, invalid requests, and model-client failures. -Rejected calls return no partial result and do not invoke the model client. - -Waiting within the admitted backlog or for active-generation capacity must -honor the caller's context. Cancellation or deadline expiry will: - -- stop waiting promptly; -- release any admission or generation capacity held by that invocation; -- preserve the applicable context error identity; and -- avoid invoking the model client if cancellation wins before generation - starts. - -Capacity must also be released after preparation, generation, validation, -repair, or collaborator failure. One failed or canceled run must not reduce -the backend's future usable capacity. - -Elapsed `Run` timing will include time spent waiting after the call is -accepted. `PreparedRun` timing will continue to describe preparation rather -than queue waiting. - -## Engine And Client Boundaries - -All pools and queued state belong to one `Engine`. Separate engines do not -share capacity, even when they register the same backend ID or endpoint. The -feature introduces no process-global scheduler. - -The engine will apply policy consistently to the built-in model client and an -injected `LLMClient`. Consumers calling their own client outside Promptkit are -outside this boundary. Injected clients remain responsible for their internal -thread safety and cancellation behavior. - -Backend policy is keyed by the resolved backend ID rather than endpoint text. -This preserves stable routing when a selected backend's endpoint is overridden -and avoids accidentally combining unrelated registrations that happen to use -the same URL. - -## Queue Lifetime And Observability - -Admission state is buffered, ephemeral, and in-process. It is not persisted -and has no survival guarantee across engine disposal or process termination. -Promptkit will not introduce background job ownership or require consumers to -start or stop workers. - -The initial feature does not require public queue-depth metrics, callbacks, or -inspection APIs. Capacity errors and ordinary call timing provide the -consumer-visible behavior. Operational observability can be added later -without coupling the scheduling mechanism to an application logging or -metrics system. - -## Compatibility - -Consumer-registered backends and endpoint-only profiles remain unlimited -unless concurrency is explicitly configured, preserving their existing -behavior. - -The built-in OpenRouter backend will change from unlimited concurrency to a -limit of 16 with a bounded waiting capacity of 1024. Ordinary synchronous -calls remain unchanged, while unusually high concurrent use may now wait or -return the capacity error. This behavioral change must be identified in the -release notes for the version that publishes it. - -Adding backend policy fields and a public capacity error is otherwise -additive. The change will use a pre-`v1` minor release under Promptkit's -[release policy](../release.md#release-model). - -## Non-Goals - -This scope does not include: - -- asynchronous job handles, polling, or detached result delivery; -- durable or cross-process queues; -- persistence or recovery across engine or process shutdown; -- priorities, scheduling weights, or consumer-defined fairness classes; -- automatic retries, backoff, rate-limit interpretation, or provider quota - discovery; -- token-per-minute or request-per-minute rate limiting; -- dynamic reconfiguration after engine construction; -- per-profile or per-run concurrency overrides; -- endpoint-keyed pooling for profiles without a backend ID; -- process-global coordination across engines; -- application worker lifecycle, logging, tracing, or metrics policy; or -- changes to prompt, profile, schema, or model-provider wire formats. - -## Target End State - -This roadmap reaches its target end state when: - -- each engine independently coordinates configured backend capacity; -- the built-in OpenRouter backend allows 16 active generations and up to 1024 - waiting runs; -- consumer backends can opt into their own active and waiting limits while - remaining unlimited by default; -- endpoint overrides retain the selected backend's capacity pool and - endpoint-only profiles remain unlimited; -- synchronous `Run` callers wait for and receive their ordinary result; -- admission is bounded before expensive preparation work where practical; -- every initial and repair generation observes the backend's active limit - without serializing preparation or validation; -- a full waiting queue returns a recognizable capacity error without invoking - the model client; -- cancellation and all failure paths promptly release capacity and preserve - context error identity; -- built-in and injected model clients receive the same scheduling behavior; -- pools remain ephemeral, engine-scoped, and independent across backend IDs; - and -- current-state GoDoc, consumer, internal, and release documentation describe - the implemented behavior once it lands. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 4d73b80..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,520 +0,0 @@ -# Structured Capacity Errors Implementation Plan - -**Status:** Complete. - -## Purpose - -This document is the decision-complete implementation plan for -[structured capacity errors](capacity-errors.md). It is written for a -gpt-5.6-terra coding agent that will implement each stage in order. - -The feature roadmap owns the motivation, public intent, compatibility policy, -security boundary, non-goals, and target end state. This plan owns the fixed -design, file-level work, implementation sequence, test ownership, -documentation updates, validation commands, and completion gates. - -## Implementation Rules - -- Complete the stages in order. Keep the repository compiling and the focused - tests passing at every stage boundary. -- Preserve unrelated working-tree changes. In particular, retain the accepted - feature roadmap and its existing future-catalog and downstream-wishlist - edits. -- Follow every policy under `docs/policy/`, the task-specific reading guide in - `docs/development.md`, and the accepted behavior in - `capacity-errors.md`. -- Keep the supported error API in the root `promptkit` package. Internal - capacity and use-case error values must not become consumer dependencies. -- Do not parse error strings. Carry the selected backend ID in an internal - typed error and translate it explicitly at the root facade. -- Do not change capacity policy, admission ordering, limits, queueing, FIFO - generation scheduling, lease lifetime, cancellation behavior, backend - selection, prepared-handle lifecycle, or model-client classification. -- Do not classify provider throttling, HTTP 429 responses, quota failures, or - injected-client errors as engine admission rejection. -- Preserve `errors.Is(err, ErrCapacityExceeded)` while adding discovery as - `*CapacityError` through `errors.As`. -- Return a fresh public typed error for each rejected operation. Do not retain - a request, pool, registry, endpoint, credential, execution target, or live - capacity state in either typed error. -- Add only `BackendID` to the public type. Do not add limits, counts, queue - depth, retry timing, retryability, transport status, provider state, or - stable JSON. -- Keep tests lean and behavior-focused. Extend the existing internal admission, - public capacity, prepared-execution, and root error-boundary tests instead of - creating a parallel test framework. -- Update exact public contracts in declarations and GoDoc. Update current-state - consumer and internal documentation only after the implementation exists. -- Do not add release notes, change a module version, create a release, commit, - or tag as part of this work. - -## Fixed Design - -### Internal Admission Error - -Add `internal/usecase/capacity_error.go` with this internal boundary type: - -```go -// CapacityError identifies bounded admission rejected for one selected -// backend. -type CapacityError struct { - BackendID string -} - -func (e *CapacityError) Error() string -func (e *CapacityError) Unwrap() error -``` - -Although exported from an `internal` package so the root facade can recognize -it, this is not a public consumer API. Its methods have these fixed semantics: - -- `Error` returns concise diagnostic wording that includes a quoted nonblank - backend ID and the generic internal capacity classification; -- a nil receiver or blank `BackendID` produces only the generic internal - capacity wording and does not panic; -- `Unwrap` always returns `capacity.ErrCapacityExceeded`, including for a nil - receiver; and -- the value contains no cause field, request reference, capacity-manager - reference, or other structured data. - -Update `Runner.admitRun` in `internal/usecase/runner.go`: - -1. retain the existing nil-admitter unlimited fallback; -2. call `RunAdmitter.Admit` exactly once with the selected backend ID; -3. when the returned error matches `capacity.ErrCapacityExceeded` and - `backendID` is nonblank, return a newly allocated - `&CapacityError{BackendID: backendID}`; -4. when a capacity error is returned for a blank backend ID by an invalid or - test-only collaborator, pass that error through rather than manufacturing a - structured value that violates the nonblank-ID guarantee; -5. pass every non-capacity error through unchanged; and -6. preserve the release function unchanged on success. - -Do not move structured identity into `internal/capacity.Manager`. The use-case -boundary already knows the effective selected backend used by both `Run` and -`RunPrepared`, and it also normalizes any conforming `RunAdmitter` -implementation into the same error contract. The capacity package continues -to own only its generic internal sentinel and scheduling state. - -Both ordinary and prepared execution already call `admitRun`; do not add -separate wrapping logic to `Run` or `RunPrepared`. - -### Public Error Type - -Add a root file named `capacity_error.go` containing: - -```go -// CapacityError reports bounded admission rejected for a selected backend. -type CapacityError struct { - BackendID string -} - -func (e *CapacityError) Error() string -func (e *CapacityError) Unwrap() error -``` - -The declaration and method GoDoc must establish: - -- engine-produced values identify only rejection at Promptkit's bounded - `Run` or `RunPrepared` admission boundary; -- `BackendID` is the normalized registered backend ID used for routing and - capacity, and endpoint overrides do not change it; -- every engine-produced value is nonnil and has a nonblank `BackendID`; -- provider errors, active-generation waiting, and caller cancellation are not - represented by this type; -- `Error` wording is diagnostic and not a parsing contract; -- `Unwrap` returns `ErrCapacityExceeded`, so `errors.Is` and `errors.As` can be - used together; -- a nil receiver and the zero value remain safe and unwrap to - `ErrCapacityExceeded`, but a consumer-constructed value is not evidence that - an engine rejected work; -- the type and its default Go encoding have no stable JSON contract; and -- consumers own returned values and may mutate `BackendID` without affecting - engine state or another error. - -Implement `Error` without exposing anything other than the public field and -the generic sentinel wording. For a nil receiver or blank field, return -`ErrCapacityExceeded.Error()`. Otherwise include the backend ID with `%q`. -Implement `Unwrap` as an unconditional return of `ErrCapacityExceeded`. - -Do not add an exported constructor, custom formatter, `Is` method, JSON tags, -`MarshalJSON`, or `UnmarshalJSON`. Direct equality with -`ErrCapacityExceeded` is not a supported contract. - -### Root Translation - -Update `mapPublicError` in `errors.go` before its general -`publicErrorFor` mapping: - -1. use `errors.As` to find an internal `*usecase.CapacityError`; -2. require the matched pointer to be nonnil and its `BackendID` to be - nonblank; -3. return a newly allocated public - `&CapacityError{BackendID: internalCapacityError.BackendID}` directly; and -4. otherwise continue through the existing general mapping. - -Returning the public value directly is deliberate. It prevents the internal -typed error and internal sentinel from remaining in the returned error chain, -while the public value's `Unwrap` supplies the supported public sentinel. -Copy only the string field; do not retain the internal error. - -Leave the existing `capacity.ErrCapacityExceeded` case in `publicErrorFor`. -It remains a defensive compatibility fallback for an unstructured internal -capacity error. No valid `Run` or `RunPrepared` capacity rejection should take -that fallback after this feature is implemented. - -Do not reorder unrelated error categories. In particular, generation failures -remain `ErrLLMGenerate`, and the mapper must not infer admission rejection from -public sentinel text, provider errors, status codes, or arbitrary errors that -happen to expose a backend field. - -### Operation Contracts - -Update the existing declarations and GoDoc without duplicating the full type -contract: - -- the `ErrCapacityExceeded` GoDoc in `engine.go` remains the broad - classification contract and points consumers to `CapacityError` for the - selected backend ID; -- `Engine.Run` states that engine admission rejection is discoverable as - `*CapacityError` and still matches `ErrCapacityExceeded`; -- `Engine.RunPrepared` states the same and retains its one-attempt handle - semantics; -- `doc.go` adds error values to its unstable-JSON category, lists - `CapacityError` there, and notes that returned structured errors are - caller-owned; and -- no operation other than `Run` and `RunPrepared` claims it can return this - type. - -Do not change method signatures or add the type to any stable JSON list. - -### Error And Identity Boundaries - -The implementation must preserve all of these distinctions: - -| Condition | `errors.Is` identity | `errors.As` to `*CapacityError` | -| --- | --- | --- | -| Limited selected backend has no admission slot | `ErrCapacityExceeded` | Yes, with selected backend ID | -| Context is already done when admission checks it | Context error | No | -| Waiting for an active generation permit is canceled | `ErrLLMGenerate` and context error | No | -| Provider or injected client returns throttling or quota failure | `ErrLLMGenerate` and documented collaborator identity | No | -| Invalid request, profile, credential, artifact, render, or validation failure | Existing category | No | -| Unlimited backend or endpoint-only profile | No admission rejection | No | - -For an ordinary `Run`, the ID is the effective registered backend selected -during preparation. For `RunPrepared`, it is the backend frozen in the claimed -handle. An endpoint override changes only the endpoint and must not change the -reported ID. - -### Test Ownership - -Extend existing tests at their current ownership boundaries. - -`internal/capacity/manager_test.go` continues to own admission mechanics. -Strengthen `TestManagerAdmissionHonorsContextAndUnlimitedBackends` so the -limited pool is full before the canceled admission attempt. This protects the -existing rule that cancellation wins over capacity rejection without adding a -new overlapping test. - -`internal/usecase/runner_test.go` owns attachment of selected identity. -Update `TestRunnerAdmissionFailureSkipsCompletionCollaborators` so: - -- the capacity case uses `errors.As` to obtain the internal - `*CapacityError`; -- its `BackendID` is exactly `"custom"`; -- the error still matches `capacity.ErrCapacityExceeded`; -- cancellation does not produce an internal `*CapacityError`; and -- the existing assertions about no partial result, no recategorization, the - admitted backend ID, and skipped collaborators remain. - -Replace the current string-content assertion with the structured assertion. -Do not test exact diagnostic wording. - -`errors_internal_test.go` owns root translation. Add a focused test that maps -an internal `*usecase.CapacityError` and proves: - -- the result is a public `*CapacityError` with the copied backend ID; -- it matches public `ErrCapacityExceeded`; -- it does not match unrelated public categories; -- it no longer exposes the internal typed error through `errors.As`; and -- mutating the source internal error after mapping does not alter the public - value. - -Retain the existing cancellation-preservation test. - -`capacity_contract_test.go` owns assembled public `Run` behavior. Extend -`TestEngineRejectsRunBeforeCompletionWhenAdmissionIsFull` to prove: - -- no partial result is returned; -- `errors.Is(err, promptkit.ErrCapacityExceeded)` still succeeds; -- `errors.As` obtains a nonnil `*promptkit.CapacityError`; -- `BackendID` is `"limited"` even though the rejected request overrides its - endpoint; -- unrelated public categories do not match; -- mutating the returned error cannot alter a subsequent independently - rejected call or its backend ID; and -- no completion collaborator or model client is invoked for rejected calls. - -In the same established full-pool setup, add a canceled-context assertion -before releasing the admitted run. It must match `context.Canceled`, must not -match `ErrCapacityExceeded`, and must not be discoverable as -`*promptkit.CapacityError`. This is the root contract counterpart to the -manager precedence test. - -Extend `TestCapacityExceededSentinelContract` to cover the public type's zero -and nil receiver behavior without asserting exact diagnostic wording: - -- both safely match `ErrCapacityExceeded`; -- an ordinary populated value is discoverable by `errors.As`; and -- neither the sentinel nor the typed value matches unrelated categories. - -`prepared_execution_contract_test.go` owns the public `RunPrepared` boundary. -In the capacity-rejection section of -`TestPreparedExecutionCredentialCapacityAndTimingBoundaries`, assert that -`errors.As` obtains `*promptkit.CapacityError` with backend ID `"limited"`. -Retain the existing assertions that the result is nil, the public sentinel -matches, the handle is consumed, and details remain available. - -Do not duplicate manager limit matrices, generation FIFO tests, backend -registration tests, or provider-client failure matrices. Existing tests -already own those behaviors. - -### Documentation Ownership - -After the code and contract tests pass: - -- update `docs/consumers/pkg-promptkit.md` in **Handle Errors** with a concise - `errors.As` example that extracts `BackendID` while retaining the existing - `errors.Is` guidance and application-owned retry policy; -- update `docs/internal/runner.md` to describe the internal typed attachment - and root translation without reproducing the public API contract; -- update `docs/internal/capacity.md` to clarify that the manager still emits - only its internal sentinel, while the runner attaches selected identity and - the facade translates it; -- update `docs/internal/overview.md` only enough to include the root typed - capacity error in the facade responsibility; and -- update `doc.go` and `engine.go` with the exact public contract described - above. - -Do not update `docs/formats.md`, the OpenAI-compatible integration contract, -backend configuration GoDoc, README, or release notes. This feature changes no -file format, provider wire request, backend configuration, project -orientation, or released-version record. - -When implementation is complete: - -- change `docs/roadmap/capacity-errors.md` to `**Status:** Complete.`; -- change `docs/roadmap/future.md` so it no longer describes structured - capacity errors as active planning and simply records that no ideas await - selection; -- change both downstream wishlist dispositions from accepted planning to - implemented behavior, linking to the consumer guide's **Handle Errors** - section rather than duplicating the contract; and -- change this document to `**Status:** Complete.` - -The feature roadmap already contains no staged or prompt-level implementation -language. Do not add such language to it when changing its status. - -## Stage 1: Carry Structured Identity Across Internal Admission - -### Objective - -Replace diagnostic-string-only backend context with a typed internal -admission error while preserving capacity mechanics and cancellation -precedence. - -### Implementation Prompt - -Implement only Stage 1 of -`docs/roadmap/implementation.md`. - -1. Add `internal/usecase/capacity_error.go` with the exact internal type and - method semantics in **Fixed Design**. -2. Update `Runner.admitRun` to create one fresh internal typed error for a - nonblank selected backend when the admitter returns the internal capacity - sentinel. -3. Update the existing runner admission-failure test to assert typed identity - instead of inspecting diagnostic text. -4. Strengthen the existing capacity-manager context test so cancellation is - checked while the limited pool is already full. -5. Run the focused formatting and validation below. - -Do not modify the root public API, root mapper, capacity-manager production -code, public contract tests, or current-state documentation in this stage. - -### Focused Validation - -Run from the repository root: - -```sh -gofmt -w internal/usecase/capacity_error.go \ - internal/usecase/runner.go \ - internal/usecase/runner_test.go \ - internal/capacity/manager_test.go -go test ./internal/capacity ./internal/usecase -git diff --check -``` - -### Completion Gate - -Stage 1 is complete only when: - -- capacity rejection from `admitRun` is discoverable as the internal - `*usecase.CapacityError`; -- its ID comes from the effective backend passed to admission; -- both `Run` and `RunPrepared` use the shared boundary without duplicate - wrapping; -- `errors.Is` still reaches `capacity.ErrCapacityExceeded`; -- cancellation and other admission errors remain untyped and unchanged; -- capacity scheduling production code is untouched; and -- focused tests and whitespace checks pass. - -## Stage 2: Expose And Protect The Public Error Contract - -### Objective - -Add the minimal public typed error, translate the internal value without -leaking it, and protect ordinary and prepared consumer behavior. - -### Implementation Prompt - -Implement only Stage 2 of -`docs/roadmap/implementation.md` after Stage 1 satisfies its completion gate. - -1. Add root `capacity_error.go` with the exact public type, methods, and GoDoc - in **Fixed Design**. -2. Update `errors.go` to translate a nonblank internal typed error into a - fresh public value before general sentinel mapping. -3. Preserve the existing unstructured capacity fallback and all unrelated - error-mapping order. -4. Update `engine.go` and `doc.go` with the exact operation, ownership, and - unstable-JSON contracts. -5. Add the focused root translation test. -6. Extend the existing external-package `Run`, sentinel, cancellation, and - `RunPrepared` contract assertions described under **Test Ownership**. -7. Run the focused formatting and validation below. - -Do not change admission policy, add provider classification, introduce a -constructor or serialization contract, or update durable prose documentation -in this stage. - -### Focused Validation - -Run from the repository root: - -```sh -gofmt -w capacity_error.go errors.go engine.go doc.go \ - errors_internal_test.go capacity_contract_test.go \ - prepared_execution_contract_test.go -go test . -run \ - 'TestMapPublicError|TestEngineRejectsRunBeforeCompletionWhenAdmissionIsFull|TestCapacityExceededSentinelContract|TestPreparedExecutionCredentialCapacityAndTimingBoundaries' -go test . -git diff --check -``` - -### Completion Gate - -Stage 2 is complete only when: - -- every assembled `Run` and `RunPrepared` admission rejection is discoverable - as a public `*CapacityError` with the selected backend ID; -- the public error unwraps only to `ErrCapacityExceeded` and does not retain - the internal typed error; -- existing broad `errors.Is` handling remains valid; -- endpoint overrides do not change the reported ID; -- cancellation at a full pool remains a context error and not a capacity - error; -- caller mutation cannot affect another rejection or engine state; -- provider, generation, and unrelated error mappings remain unchanged; and -- focused and complete root-package tests pass. - -## Stage 3: Update Canonical Documentation And Validate - -### Objective - -Make the implemented structured error discoverable, reconcile temporary -planning state, and complete repository-wide validation. - -### Implementation Prompt - -Implement only Stage 3 of -`docs/roadmap/implementation.md` after Stages 1 and 2 satisfy their completion -gates. - -1. Update the consumer and internal documents listed under - **Documentation Ownership**. Keep exact API guarantees in GoDoc and use - prose documents for task guidance and implementation boundaries. -2. Update the feature-roadmap, future-catalog, implementation-plan, and - downstream-wishlist statuses and links exactly as specified above. -3. Follow every added or changed Markdown link and confirm its file and - heading target. -4. Run the full validation sequence. -5. Inspect the complete diff for scope, sensitive data, and repository - hygiene. -6. Only after every check passes, leave both roadmap statuses as `Complete` - and rerun `git diff --check`. - -Do not add a release document, new example, new public package, retry policy, -transport mapping, or duplicate API reference. - -### Full Validation - -Run from the repository root: - -```sh -gofmt -w internal/usecase/capacity_error.go \ - internal/usecase/runner.go \ - internal/usecase/runner_test.go \ - internal/capacity/manager_test.go \ - capacity_error.go errors.go engine.go doc.go \ - errors_internal_test.go capacity_contract_test.go \ - prepared_execution_contract_test.go -gofmt -l $(git ls-files '*.go') -go test ./... -go test -race ./... -go vet ./... -go build ./... -go run ./examples/go-library/prepare -git diff --check -git status --short -``` - -The `gofmt -l` command must print no paths. The maintained example must remain -offline and require no real credential or provider. - -Inspect the final state and confirm: - -- only files required by this feature and pre-existing user changes are - present; -- no credential, private source content, endpoint, workspace file, local - replacement, generated binary, or unrelated formatting change was added; -- the root declaration and GoDoc own the exact public contract; -- consumer guidance summarizes the workflow and links to the canonical API; -- internal documents describe responsibility without redefining the public - contract; -- no current-state document claims Promptkit owns retry, backoff, transport, - logging, or metrics policy; -- no roadmap retains staged language outside this implementation plan; and -- no release, commit, or tag was created. - -### Completion Gate - -Implementation is complete only when: - -- every Stage 1 and Stage 2 gate remains satisfied; -- ordinary and race-enabled tests pass; -- vet, build, formatting, the offline example, Markdown links, and whitespace - checks pass; -- public, consumer, and internal documentation agree on the implemented - boundary; -- future-catalog and downstream-wishlist dispositions no longer describe the - feature as pending; -- both roadmap statuses are `Complete`; -- the working tree contains no unintended files or changes; and -- the repository is ready for maintainer review without a commit or release - having been created by this plan. - -## Open Questions - -None. The accepted feature roadmap and fixed design above fully specify the -implementation boundary. diff --git a/docs/roadmap/local-backend.md b/docs/roadmap/local-backend.md deleted file mode 100644 index 8d466e8..0000000 --- a/docs/roadmap/local-backend.md +++ /dev/null @@ -1,163 +0,0 @@ -# Local Backend Convenience - -**Status:** Complete. - -## Purpose - -Make the common case of using a local OpenAI-compatible endpoint concise and -easy to discover without introducing implicit configuration or a separate -backend abstraction. - -The existing `Backend` type and registry remain the canonical, fully -configurable interface. A small convenience constructor will cover the usual -local-network case, while improved consumer documentation will make it clear -when an endpoint-only profile, the convenience constructor, or a complete -`Backend` value is appropriate. - -## Motivation - -Consumers can already use a local endpoint by setting `Profile.Endpoint`, or -register one as a backend with `WithBackend`. The first option is concise but -does not provide shared backend-level concurrency control. The second supports -the complete backend feature set but requires consumers to understand and -populate several fields for a common configuration. - -Most consumers adding a local backend need only: - -- a stable backend ID; -- an OpenAI-compatible endpoint; and -- a concurrency limit appropriate for the local server. - -Promptkit should provide a direct path for that case while keeping all -configuration explicit and preserving the full registry interface for -advanced needs. - -## Consumer Paths - -Documentation should present three progressively more configurable paths: - -1. Set `Profile.Endpoint` when a profile only needs to target a local endpoint - and does not need shared backend policy. -2. Use the local-backend convenience constructor when profiles should share a - named local endpoint and its concurrency limit. -3. Construct a complete `Backend` value when the consumer needs a custom - backend ID, authentication, extra request parameters, an explicit queue - capacity, or multiple local backends. - -These are complementary interfaces. The convenience constructor must return an -ordinary `Backend`, so it does not create a second configuration model. - -## Public Convenience API - -The public package should expose: - -```go -const BackendLocal = "local" - -func LocalBackend(endpoint string, concurrencyLimit int) Backend -``` - -`LocalBackend` should return a `Backend` with: - -- `ID` set to `BackendLocal`; -- `Endpoint` set to the supplied endpoint; -- `ConcurrencyLimit` set to the supplied limit; and -- all other fields left at their zero values. - -The returned value is passed to `WithBackend` and follows the same copying, -normalization, validation, and registration rules as any consumer-constructed -`Backend`. - -The constructor should be a transparent value constructor. It should not read -environment variables, mutate global state, register the backend, validate -arguments independently, or create profiles. Consumers may inspect or modify -the returned value before registration, although documentation should direct -substantially customized configurations to the full `Backend` form. - -## Identity and Registration - -`BackendLocal` is a conventional ID used by the convenience constructor. It is -not pre-registered and should not become a specially reserved registry ID. -Consumers remain responsible for registering the returned backend with -`WithBackend` and naming it from profiles through `BackendID`. - -This distinction preserves compatibility with consumers that may already -register their own backend using the ID `"local"`. Normal duplicate-ID rules -still apply if a consumer attempts to register more than one backend with that -ID. - -Consumers that need multiple local endpoints should choose distinct IDs and -use complete `Backend` values rather than the single conventional helper ID. - -## Concurrency and Queue Semantics - -The constructor must preserve the existing backend concurrency contract: - -- a positive concurrency limit bounds simultaneous requests and uses the - existing default queue capacity because `QueueCapacity` remains `nil`; -- a zero concurrency limit leaves the backend unconstrained; and -- a negative concurrency limit is rejected through the existing engine - configuration validation path. - -The constructor should not select a hidden default concurrency limit. Local -servers vary substantially in capacity, so the consumer should make this -choice explicitly. - -## Documentation - -The final documentation state has two canonical surfaces: - -- Public Go documentation describes the exact contract of - `BackendLocal` and `LocalBackend`, including their conventional, - non-pre-registered nature. -- The [promptkit consumer guide](../consumers/pkg-promptkit.md) includes - a task-oriented local-endpoint section that shows the three consumer paths, - explains the decision between them, and provides concise examples of the - endpoint-only and convenience-constructor forms. - -The consumer guide continues to document the full `Backend` interface as -the advanced path rather than attempting to reproduce every configuration -variation through convenience APIs. - -## Compatibility - -This feature is additive: - -- existing endpoint-only profiles continue to work unchanged; -- existing `Backend` values and `WithBackend` registrations remain the - canonical general-purpose interface; -- existing registrations using the literal ID `"local"` remain valid; and -- OpenRouter defaults and all other backend behavior remain unchanged. - -No consumer is required to adopt the convenience constructor. - -## Non-Goals - -This work does not include: - -- pre-registering or implicitly enabling a local backend; -- discovering a local endpoint, API key, or concurrency limit from environment - variables; -- adding local-backend fields to `Config`; -- selecting a default local model or creating a profile automatically; -- adding a combined backend-and-profile constructor; -- adding convenience parameters for API keys, extra request parameters, or - queue capacity; -- replacing or redesigning the backend registry; -- adding support for non-OpenAI-compatible local APIs; or -- changing backend routing, scheduling, or queue behavior. - -## Target End State - -After this work: - -- consumers with a simple one-profile local endpoint can continue to configure - it directly on the profile; -- consumers needing a shared local endpoint and concurrency policy can express - it with one `LocalBackend` call and register the returned value normally; -- consumers with advanced or multiple-local-backend requirements have a clear - path to the complete `Backend` interface; -- all local configuration remains explicit, inspectable, and compatible with - dependency injection; and -- canonical documentation makes the simplest suitable interface easy to find - without obscuring the underlying registry model. diff --git a/docs/roadmap/notarius-promptkit-wishlist.md b/docs/roadmap/notarius-promptkit-wishlist.md index 4aa3e46..3613031 100644 --- a/docs/roadmap/notarius-promptkit-wishlist.md +++ b/docs/roadmap/notarius-promptkit-wishlist.md @@ -6,21 +6,19 @@ This document records features and interface changes that would be useful additions to PromptKit from the perspective of the maintainers of Notarius, a downstream application that consumes PromptKit. -PromptKit v0.3.0 provides the capabilities Notarius currently needs. None of -the ideas below blocks current Notarius development. They are opportunities to -reduce downstream workarounds, improve integration correctness, and make -PromptKit more ergonomic for applications with configuration validation, -debugging, checkpointing, and operational-observability requirements. +PromptKit now provides the capabilities Notarius currently needs. The +remaining deferred ideas are optional opportunities to improve checkpointing +and operational observability. The examples are API sketches intended to communicate the desired capability, not prescriptive names or finalized Go contracts. ## Priority 1: Atomic Execution With Prepared Details -**Disposition:** Covered by the accepted -[executable preparation handles](prepared-execution.md) roadmap. The shared -two-phase capability should provide the required single-preparation -consistency; a separate `RunDetailed` method is not cataloged initially. +**Disposition:** Implemented through [`Engine.PrepareExecution` and +`Engine.RunPrepared`](../../engine.go). See the +[consumer guidance](../consumers/pkg-promptkit.md#prepare-now-and-execute-the-same-snapshot-later). +A separate `RunDetailed` method is not cataloged. ### Downstream need @@ -33,58 +31,18 @@ Notarius needs both: Notarius uses the prepared details to construct redaction-aware debug bundles and retain enough information to diagnose model behavior. -### Current integration +### Implemented behavior -Notarius currently calls `Engine.Prepare` and then `Engine.Run` with the same -request. Because `Run` performs preparation internally, a successful request -resolves and prepares the same work twice. - -This duplicates profile resolution, input hashing, schema loading, and prompt -rendering. It also creates a theoretical consistency window in which a -filesystem-backed prompt, profile, schema, or input could change between the -explicit preparation and the preparation performed by `Run`. - -### Requested capability - -Add an opt-in execution method that prepares exactly once and returns both the -prepared details and completed result: - -```go -type RunReport struct { - Prepared PreparedRun - Result RunResult -} - -func (e *Engine) RunDetailed( - ctx context.Context, - req RunRequest, -) (*RunReport, error) -``` - -The exact names are flexible. The important contract is that preparation -occurs once and that the returned prepared state describes the execution that -produced the returned result. - -Existing `Prepare` and `Run` behavior should remain available for consumers -that need only one side of the operation. - -### Design considerations - -- Keep this API additive and preserve the existing simple `Run` workflow. -- Return caller-owned copies under PromptKit's existing ownership rules. -- Define whether any prepared details are available after an operational - generation or validation error. Notarius does not require partial results - for the initial use case, but an explicit contract would be valuable. -- Preserve cancellation and backend-admission semantics. -- Do not add all prepared content directly to `RunResult`. Rendered prompt - content can be large and sensitive, and consumers should opt in to receiving - it. +Notarius can prepare one frozen execution snapshot, retain a caller-owned and +credential-redacted `Details` value for its debug bundle, and execute the same +snapshot through `RunPrepared`. The opaque handle is engine-bound and +single-use; an unused handle can be released with `Discard`. The consumer +guide and exported GoDoc own the exact lifecycle and failure contracts. ### Value to Notarius -This is the highest-value wishlist item. It would remove duplicate work from -every successful PromptKit-backed call and ensure that retained debug material -corresponds atomically to the actual execution. +This removes duplicate work from PromptKit-backed calls and ensures that +retained debug material corresponds atomically to the actual execution. ## Priority 2: Prompt-Independent Profile Inspection @@ -275,14 +233,13 @@ application-policy decision, not an upstream PromptKit gap. ## Suggested Upstream Sequence -If the PromptKit team chooses to pursue these ideas, the most useful order for -Notarius would be: +For downstream adoption and any remaining upstream work, the useful order is: -1. Add atomic execution that returns prepared details and the completed result. +1. Adopt prepared execution for atomic details and results. 2. Add a semantic execution-target digest, preferably alongside profile inspection. 3. Use the implemented typed capacity error where backend admission diagnostics are needed. -The first addresses a concrete execution workaround. The second would improve +The first removes the concrete execution workaround. The second would improve checkpoint correctness and reduce coupling. The third is operational polish. diff --git a/docs/roadmap/prepared-execution.md b/docs/roadmap/prepared-execution.md deleted file mode 100644 index a0193c2..0000000 --- a/docs/roadmap/prepared-execution.md +++ /dev/null @@ -1,280 +0,0 @@ -# Executable Preparation Handles - -**Status:** Complete. - -## Purpose - -Allow a consumer to prepare one exact Promptkit execution, inspect and retain -its credential-redacted public preparation details, and later execute that -already-prepared work without resolving or rendering the request again. - -This provides a supported preflight-before-generation boundary for -[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-1-executable-preparation-handles) -and removes the duplicate `Prepare`-then-`Run` workaround described by -[Notarius](notarius-promptkit-wishlist.md#priority-1-atomic-execution-with-prepared-details). - -## Motivation - -`Engine.Prepare` currently returns the provenance and rendered details that -consumers need for debugging, persistence, and preflight checks. `Engine.Run` -then performs its own preparation before generation. A consumer that needs -both values must therefore prepare the same logical request twice. - -That workaround duplicates source loading, input hashing, schema work, and -template rendering. It also permits filesystem-backed prompts, profiles, -schemas, or inputs to change between the public preparation and the -preparation that actually produces the result. - -Promptkit already owns the complete preparation and execution pipeline. An -opt-in prepared-execution handle should expose the missing boundary without -putting rendered content into every `RunResult` or moving persistence policy -into the library. - -## Consumer Workflow - -The target public workflow is: - -```go -prepared, err := engine.PrepareExecution(ctx, request) -if err != nil { - // Handle preparation failure. - return -} -defer prepared.Discard() - -details := prepared.Details() -// Persist or inspect a consumer-selected safe subset of details. - -result, err := engine.RunPrepared(ctx, prepared) -``` - -The target public surface is: - -```go -type PreparedExecution struct { - // Opaque Promptkit-owned state. -} - -func (e *Engine) PrepareExecution( - ctx context.Context, - req RunRequest, -) (*PreparedExecution, error) - -func (p *PreparedExecution) Details() PreparedRun - -func (p *PreparedExecution) Discard() - -func (e *Engine) RunPrepared( - ctx context.Context, - prepared *PreparedExecution, -) (*RunResult, error) -``` - -The declarations and GoDoc will own the exact implemented contract. The -important public shape is an opaque handle, caller-owned `PreparedRun` -details, an explicit discard operation, and execution through the engine that -created the handle. - -## Prepared Snapshot - -`PrepareExecution` performs complete preparation without model generation or -backend-capacity admission. It applies the same request validation, source -precedence, backend and profile resolution, credential requirement checks, -output-contract resolution, artifact loading, hashing, schema loading, -session resolution, and rendering behavior as `Prepare`. - -A successful handle freezes all source-derived state required for later -execution, including: - -- the selected prompt definition, profile, and backend identity; -- the complete effective execution target and request-field presence; -- rendered messages and effective session ID; -- prompt, rendered-prompt, and input hashes; -- the effective output contract and provider-facing structured-output - constraint; and -- private validation state sufficient to validate generated output without - reopening schema files or `fs.FS` resources, including required schema - references. - -After `PrepareExecution` succeeds, changes to prompt, profile, schema, input, -or request-owned data cannot change what `RunPrepared` sends to the model or -how it validates the generated output. - -The handle retains an internal snapshot independent from values returned by -`Details`. Mutating a returned `PreparedRun`, its maps, slices, messages, or -schema values does not affect later execution. Each `Details` call returns a -fresh caller-owned copy under the existing `PreparedRun` ownership and stable -JSON rules. - -## Handle Lifecycle - -A `PreparedExecution` is: - -- created only by a successful `PrepareExecution` call; -- bound to the exact `Engine` that created it; -- valid for one `RunPrepared` invocation; -- safe for repeated `Details` calls; -- intentionally opaque and without a supported JSON representation; and -- in-process state rather than a durable or restartable job. - -`RunPrepared` atomically claims a valid handle before beginning the execution -attempt. A second or concurrent invocation fails without starting another -execution, including when the first invocation ended in cancellation, capacity -rejection, generation failure, or operational validation failure. Copies of -the public handle share the same one-attempt state and cannot bypass this rule. - -A nil, zero-value, foreign-engine, discarded, already-claimed, or already-used -handle is invalid. `RunPrepared` reports these lifecycle errors through the -ordinary public invalid-request category. A failed foreign-engine invocation -does not consume a handle that remains valid for its owning engine. - -`Discard` idempotently makes an unclaimed handle unavailable for execution and -drops Promptkit's references to secret-bearing or execution-only state. -`RunPrepared` performs the same cleanup automatically after claiming a handle. -Credential-redacted public preparation details remain available after discard, -success, or failure so consumers can retain diagnostics. Promptkit does not -promise secure erasure of Go string memory. - -Lifecycle transitions are concurrency-safe. When `RunPrepared` and `Discard` -race, exactly one claims the ready handle. `Discard` is not an execution -cancellation mechanism and does not interrupt an attempt that has already -claimed the handle; consumers cancel that attempt through its context. - -## Credentials And Sensitive Data - -`Details` has the same security contract as `PreparedRun`: it can contain -rendered messages, schemas, identifiers, and hashes, but never a resolved API -key value. Consumers remain responsible for selecting, redacting, storing, and -retaining any persisted preparation material. - -A direct `RunRequest.APIKey` is retained only in opaque execution state until -the handle is run or discarded. It is never added to details, hashes, JSON, -`String`, or `GoString` output. - -An environment-variable name is frozen as part of the effective target, but -its credential value is not captured for the lifetime of the handle. -`PrepareExecution` applies the existing preparation-time availability check. -`RunPrepared` rechecks availability before admission, and the selected model -client uses the environment value visible during execution. This preserves -current secret ownership and avoids retaining an environment credential while -a consumer persists preflight material. - -The opaque handle must not expose retained request data or credentials through -default formatting, JSON, or error messages. - -## Admission, Cancellation, And Execution - -`PrepareExecution` never reserves backend admission or an active-generation -permit. Its context governs preparation only; cancellation after it returns -does not invalidate the handle. - -`RunPrepared` uses its own context for credential revalidation, backend -admission, active-generation waiting, model generation, output validation, and -any internal repair. Admission occurs when `RunPrepared` begins so a consumer -cannot occupy bounded capacity while inspecting or persisting preparation -details. - -For a limited backend, the admission lease covers the complete prepared -execution attempt after admission: generation, validation, internal repair, -and every success or failure exit. Actual generation continues to use the -backend's FIFO active-generation permit. Existing capacity error identity, -cancellation behavior, and release guarantees remain in force. - -Because a prepared handle is one-attempt, cancellation or capacity rejection -does not make it reusable. Retry and backoff policy remains with the consumer, -which may create a new prepared handle when another attempt is appropriate. - -## Results And Failures - -On success, `RunPrepared` returns the existing caller-owned `RunResult`. Its -source-derived provenance must match `Details`, including prompt identity and -hash, rendered-prompt hash, session ID, selected profile and backend, -effective target, and input hashes. - -`RunResult.StartTime`, `EndTime`, and `Duration` describe the -`RunPrepared` execution attempt. They exclude preparation time and any delay -while the consumer retained the handle. Preparation timing remains in -`PreparedRun`. - -Preparation failure returns no handle. After successful preparation, -`RunPrepared` retains the existing rule that an operational failure returns no -partial `RunResult`; the consumer already has independent preparation details. -A completed content-validation failure remains a successful result with -`ValidationFailed`. - -`PrepareExecution` preserves the public error categories of `Prepare`. -`RunPrepared` preserves applicable invalid-request, credential, capacity, -generation, validation, collaborator, and cancellation identities without -reintroducing source-loading or rendering failures from frozen state. - -## Compatibility And Existing Workflows - -This feature is additive: - -- `Prepare` remains the simple preparation-only operation; -- `Run` remains the simple prepare-and-execute operation with its current - early-admission and error-ordering behavior; -- `PreparedRun` and `RunResult` retain their existing stable JSON - representations; -- model-client and artifact-reader extension interfaces remain unchanged; and -- backend routing, concurrency limits, queue capacities, and provider wire - behavior remain unchanged. - -The new workflow may share internal machinery with `Prepare` and `Run`, but it -must not change their observable behavior merely to simplify implementation. - -## Documentation - -The completed documentation set has these ownership boundaries: - -- exported declarations and GoDoc own the exact handle, method, lifecycle, - ownership, concurrency, credential, error, and cancellation contracts; -- the promptkit consumer guide explains when to use `Prepare`, `Run`, or the - two-phase prepared-execution workflow; and -- internal runner, source-validation, capacity, and model-client documentation - describe the implemented collaborator boundaries without duplicating public - contracts. - -No release document is part of the feature implementation itself. Release -guidance is prepared only when the resulting public API is selected for -publication. - -## Non-Goals - -This work does not include: - -- serializable, durable, resumable, or cross-process execution handles; -- reuse for multiple consumer-initiated executions; -- concurrent execution of one handle; -- capacity reservation during preparation; -- a background task queue, priorities, worker lifecycle, or job status; -- retry, backoff, or provider failover policy; -- a separate `RunDetailed` convenience method; -- adding preparation details or rendered messages to every `RunResult`; -- prompt or profile inspection APIs; -- structured capacity or generation errors; -- freezing environment-variable credential values for the handle lifetime; -- snapshotting provider state or mutable behavior inside an injected - `LLMClient`; -- allowing target, validation, session, variable, or input overrides after - preparation; or -- changing existing `Prepare`, `Run`, file-format, provider-request, or stable - JSON contracts. - -## Target End State - -After this work: - -- consumers can perform and persist preflight before starting provider work; -- one prepared handle executes exactly the source-derived prompt, target, - schema, inputs, session, and messages described by its public details; -- execution never reloads or rerenders consumer sources; -- direct credentials remain confined to opaque, explicitly discardable state; -- environment credentials are not retained across the preflight boundary; -- backend capacity is reserved only when execution begins; -- one handle can start at most one execution attempt, including any internal - repair calls owned by that attempt; -- preparation details remain available after execution success or failure; -- existing simple `Prepare` and `Run` consumers remain unaffected; and -- Promptkit continues to own reusable execution mechanics without taking on - downstream persistence, redaction, retry, or job-management policy. diff --git a/docs/roadmap/profile-inspection.md b/docs/roadmap/profile-inspection.md deleted file mode 100644 index 09827be..0000000 --- a/docs/roadmap/profile-inspection.md +++ /dev/null @@ -1,246 +0,0 @@ -# Prompt-Independent Profile Inspection - -**Status:** Complete. - -## Purpose - -Allow consumers to look up one execution profile by ID and inspect its -structurally resolved model target without selecting a prompt, supplying -placeholder inputs, checking credential availability, or invoking a model. - -This provides a direct configuration-validation boundary for -[Notarius](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection) -and -[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-3-prompt-independent-profile-inspection). - -## Motivation - -Both downstream consumers need to reject invalid configured profile IDs before -starting application work. They currently have to construct a synthetic prompt -and call `Engine.Prepare` merely to exercise profile loading, backend lookup, -and execution-target resolution. - -That workaround couples profile validation to unrelated prompt definitions, -fixture inputs, rendering, schema behavior, and current credential -availability. Promptkit already owns profile precedence, backend membership, -and target resolution, so it should expose that cohesive capability directly. - -## Consumer Workflow - -The target public workflow is: - -```go -inspection, err := engine.InspectProfile(ctx, profileID) -if err != nil { - // Reject or report the configured profile. - return -} - -target := inspection.EffectiveModelParams -if target.APIKeyEnv != "" { - // Apply application policy for the named environment variable. -} -``` - -The target public surface is: - -```go -type ProfileInspection struct { - ProfileID string - EffectiveModelParams ExecutionTarget - APIKeyRequired bool -} - -func (e *Engine) InspectProfile( - ctx context.Context, - profileID string, -) (*ProfileInspection, error) -``` - -The declarations and GoDoc will own the exact implemented contract. The -important public shape is exact lookup through the existing engine, one -caller-owned inspection value, the resolved `ExecutionTarget`, and an explicit -signal for a direct API-key requirement. - -`EffectiveModelParams.BackendID` identifies the selected registered backend -and remains empty for endpoint-only profiles. -`EffectiveModelParams.APIKeyEnv` reports the effective credential environment -variable name. `APIKeyRequired` reports that the profile requires a direct -request credential instead. These states are mutually exclusive after normal -profile and backend precedence is applied. - -`ProfileInspection` does not need a stable JSON representation. Consumers that -persist application configuration or diagnostics can select the fields their -own format requires. - -## Lookup And Precedence - -`InspectProfile` requires a non-blank explicit profile ID. It trims surrounding -whitespace and otherwise performs the same case-sensitive exact lookup used by -ordinary execution. - -Lookup applies the engine's normal profile-source precedence: - -- programmatic profiles supplied through `WithProfiles`; -- the configured file, directory, or `fs.FS` profile source; and -- the built-in profile catalog. - -A valid higher-precedence match shadows a lower-precedence profile with the -same ID. A malformed or unreadable higher-precedence match fails rather than -silently falling back. The -[profile format reference](../formats.md#profile-definitions) remains the -canonical owner of profile-source and file-format behavior. - -Inspection never derives a profile ID from a prompt's `default_profile`; the -caller is inspecting one explicitly named profile. - -## Structural Resolution - -Inspection loads and validates the selected profile, verifies that a named -backend exists in the engine's immutable backend registry, and applies normal -framework-default, backend, and profile precedence to produce the effective -target. - -For the same engine state and profile ID, with no per-run execution override, -the inspected target must match the target that ordinary preparation would -resolve before applying request credentials and checking their availability. -This equivalence must use one shared resolution path rather than a second set -of precedence rules. - -Structural resolution includes: - -- backend routing identity; -- endpoint and model; -- sampling, token, timeout, service-tier, and reasoning settings; -- the effective credential environment-variable name or direct-key - requirement; and -- deeply copied provider-specific extra parameters. - -Inspection returns the effective target rather than a raw profile definition. -This keeps framework and backend defaults visible to consumers without -creating a second public profile-loading interface. - -The result does not include request-override presence because no -`ExecutionTargetOverride` participates in inspection. - -## Credentials And Sensitive Data - -Inspection reports credential requirements but never resolves, retains, or -returns a credential value. - -The operation does not read the named environment variable and succeeds when -that variable is absent or blank. It accepts neither a direct API key nor an -API-key environment override. Consumers decide whether credential availability -must be enforced during application configuration, while `Prepare`, -`PrepareExecution`, `Run`, and `RunPrepared` retain their execution-time -credential contracts. - -Error messages, formatting, and returned values must not expose environment -values or other resolved secrets. Existing restrictions against raw API keys -in profile sources remain unchanged. - -## Ownership, Consistency, And Concurrency - -Each successful call returns a caller-owned snapshot. Mutating the returned -target or any nested extra-parameter map or slice cannot affect the engine, -later inspection, or later execution. - -Inspection is safe to call concurrently under the engine's existing immutable -registry and repository contracts. It does not mutate profile sources or -cache a result globally. - -For filesystem-backed sources, an inspection describes the state observed by -that call. It does not freeze the profile for a later `Run`; a source may -change between operations. Consumers requiring an exact preflight-to-execution -snapshot should use the existing prepared-execution workflow. - -## Errors And Cancellation - -The operation uses existing public error categories: - -- a blank profile ID matches `ErrInvalidRequest`; -- an absent exact ID matches `ErrProfileNotFound` and not `ErrProfileLoad`; -- read, decode, validation, and source-selection failures match - `ErrProfileLoad`; and -- an unknown referenced backend or an invalid structurally resolved target - matches `ErrProfileLoad`. - -Errors should preserve useful underlying collaborator and context identities -through `errors.Is` where the existing facade does so, without exposing -internal package types. Context cancellation governs inspection and no partial -inspection result is returned on failure. - -Missing credential values are not inspection errors. The operation cannot -return capacity or model-generation failures because it performs neither -backend admission nor generation. - -## Compatibility And Boundaries - -This feature is additive. Existing profile formats, source precedence, -backend registration, `Prepare`, prepared execution, and `Run` behavior remain -unchanged. - -The method belongs on the root `Engine` facade. Profile repositories and the -backend registry remain internal implementation details, and no new public -repository interface is introduced. - -Inspection does not require prompt lookup, rendering, artifact loading, schema -loading, validation, backend-capacity admission, or model-client access. -Engine construction retains its ordinary configuration requirements; this -feature does not introduce a separate profile-only engine. - -## Documentation - -The completed documentation set has these ownership boundaries: - -- exported declarations and GoDoc own the exact method, result, ownership, - credential, error, and cancellation contracts; -- the promptkit consumer guide explains configuration-time profile inspection - and distinguishes it from `Prepare` and prepared execution; -- the profile format reference continues to own profile fields and source - precedence; and -- internal documentation describes shared profile and target resolution - without duplicating public contracts. - -## Non-Goals - -This work does not include: - -- enumerating or searching profiles; -- returning raw profile definitions or profile source paths; -- accepting per-run execution overrides, direct API keys, or API-key - environment overrides; -- checking environment-variable contents or other credential availability; -- semantic execution-target fingerprints or profile hashes; -- prompt-definition inspection or prompt default-profile resolution; -- full-corpus validation across every profile source; -- freezing a filesystem-backed profile for later execution; -- exposing backend concurrency limits, queue state, or capacity policy; -- model generation, provider health checks, or endpoint connectivity tests; -- dynamic backend or profile registration after engine construction; or -- changing current profile, backend, prepared-execution, or stable JSON - contracts. - -## Target End State - -After this work: - -- consumers can validate one configured profile without inventing a prompt or - placeholder inputs; -- lookup observes ordinary programmatic, configured-source, and built-in - precedence; -- a successful result proves that the profile exists, is valid, references a - registered backend when applicable, and resolves to a structurally valid - effective target; -- the inspected target matches ordinary preparation for the same profile and - engine state before per-run overrides and credential availability checks; -- credential requirements are visible without reading or exposing credential - values; -- returned targets and nested data are caller-owned; -- inspection performs no rendering, source loading unrelated to the profile, - capacity admission, or model work; -- existing execution workflows and compatibility contracts remain unchanged; - and -- Promptkit owns reusable profile validation while downstream applications - retain configuration policy, persistence, logging, and credential-timing - decisions. diff --git a/docs/roadmap/prompt-inspection.md b/docs/roadmap/prompt-inspection.md deleted file mode 100644 index 9975b4c..0000000 --- a/docs/roadmap/prompt-inspection.md +++ /dev/null @@ -1,268 +0,0 @@ -# Prompt-Definition Inspection - -**Status:** Complete. - -## Purpose - -Allow consumers to look up one prompt definition by ID and optional version -and inspect its declared interface without supplying placeholder inputs, -resolving a profile, rendering templates, loading schemas, or invoking a -model. - -This provides a focused configuration-validation boundary for Weatherreporter, -whose need is recorded in its -[Promptkit wishlist](weatherreporter-promptkit-wishlist.md#priority-2-prompt-definition-inspection), -while preserving Promptkit's deferred, request-oriented source model. - -## Motivation - -Weatherreporter has a fixed application-owned registry of report definitions. -It needs to verify that every configured prompt exists and declares the -expected inputs and output workflow before collecting weather data or starting -provider work. - -The current workaround is to construct synthetic artifacts and variables and -call `Engine.Prepare`. That validates much more than the application needs: -prompt inputs are loaded, templates and session IDs are rendered, a profile -and backend are resolved, credentials are checked, and JSON Schemas may be -loaded and compiled. - -Promptkit already owns prompt source selection, exact ID and version lookup, -strict definition decoding, referenced content-file resolution, and prompt -hashing. It should expose that cohesive subset directly rather than requiring -consumers to reproduce its rules or maintain placeholder execution fixtures. - -## Consumer Workflow - -The target public workflow is: - -```go -inspection, err := engine.InspectPrompt(ctx, promptID, promptVersion) -if err != nil { - // Reject or report the configured prompt. - return -} - -for _, input := range inspection.Inputs { - // Compare the declared prompt interface with application configuration. -} -``` - -The target public surface is: - -```go -type PromptInputDefinition struct { - Name string - Required bool - ContentType string - Description string -} - -type PromptInspection struct { - PromptID string - PromptVersion string - PromptHash string - DefaultProfileID string - Inputs []PromptInputDefinition - OutputContract OutputContract -} - -func (e *Engine) InspectPrompt( - ctx context.Context, - promptID string, - promptVersion string, -) (*PromptInspection, error) -``` - -The exported declarations and GoDoc will own the exact implemented contract. -The important public shape is exact lookup through an existing engine, one -caller-owned inspection value, declared input metadata, the declared output -contract, and the same opaque prompt equality value used by preparation. - -`PromptInspection` and `PromptInputDefinition` do not need stable JSON -representations. Consumers that persist application configuration or -diagnostics can define their own format and select the fields they require. -The existing stable representations of `OutputContract`, `OutputFormat`, and -`ValidationMode` remain unchanged. - -## Lookup And Source Selection - -`InspectPrompt` requires a non-blank prompt ID and performs the same -case-sensitive exact selection used by ordinary preparation. Inspection does -not trim or otherwise canonicalize a non-blank ID or version independently of -that execution path. - -When `promptVersion` is non-empty, the configured prompt source must contain -exactly one matching ID and version pair. When it is empty, the source must -contain exactly one definition with the requested ID; multiple matching -versions remain an ambiguous source-selection error. - -Lookup uses the engine's normally selected prompt source: - -- the last `WithPromptFS` or `WithPromptFile` option replaces earlier prompt - source options and `Config.PromptDir`; or -- `Config.PromptDir` supplies the source when no prompt source option is - present. - -Inspection does not merge prompt sources, fall back after a malformed match, -or introduce a new public prompt repository. The -[format reference](../formats.md#prompt-definitions) remains the canonical -owner of prompt source and exact selection behavior. - -## Structural Validation - -A successful inspection proves that the selected definition can be loaded -through the ordinary prompt repository and satisfies its source-level -structural rules. This includes: - -- deterministic YAML discovery and exact definition selection; -- strict decoding and validation of required identity, version, messages, and - output fields; -- normalization and validation of input declarations, message roles, cache - control, the default-profile identifier, and output-contract fields; and -- contained, readable resolution of every referenced message - `content_file`. - -Referenced message content participates in validation and hashing but is not -returned. Inspection does not parse or execute Go templates, resolve template -variables or input helpers, or enforce the runtime presence and media type of -declared inputs. Those checks remain part of rendering and preparation. - -The returned `OutputContract` is the normalized contract declared by the -prompt definition. For `json_schema` validation, structural inspection proves -that a non-blank schema path is declared and returns that path. It does not -open, resolve, compile, or return the schema document or its references. -Schema-source validation remains part of executable preparation and any -future explicit full-source validation feature. - -Inspection returns `DefaultProfileID` as declared metadata. It does not look -up that profile, resolve a backend or execution target, check credentials, or -prove that the profile is executable. Consumers may call `InspectProfile` -separately when their application policy requires that additional check. - -## Result Semantics - -The result exposes the complete declared input metadata in definition order: -name, required status, content type, and human-readable description. It does -not expose message templates, session-ID templates, cache-control details, -prompt descriptions, raw YAML, source paths, rendered messages, or schema -bodies. - -`PromptID` and `PromptVersion` are the validated identity read from the -selected definition. `PromptHash` is an opaque equality value for the complete -loaded definition, including referenced message content. For the same selected -definition and observed source state, it must equal the `PreparedRun.PromptHash` -produced by ordinary preparation. - -Consumers may compare `PromptHash` values but must not depend on their -algorithm, encoding, length, or suitability as a security proof. Promptkit may -change the representation in a future release together with the existing -prepared-run hash contract. - -## Ownership, Consistency, And Concurrency - -Each successful call returns a caller-owned snapshot. Mutating the result or -its input slice cannot affect the engine, later inspections, or later -preparation. - -Inspection is safe to call concurrently under the engine's existing -repository contracts. It does not mutate prompt sources or install a global -cache. - -For filesystem-backed and mutable `fs.FS` sources, the result describes the -state observed by that lookup. It does not freeze the definition for a later -`Prepare`, `PrepareExecution`, or `Run`, and a source may change between -operations. Consumers requiring an exact preflight-to-execution snapshot -should use the existing prepared-execution workflow. - -## Errors And Cancellation - -The operation uses existing public error categories: - -- a blank or whitespace-only prompt ID matches `ErrInvalidRequest`; -- an absent exact ID or version matches `ErrPromptNotFound` and not - `ErrPromptLoad`; -- read, strict-decode, validation, duplicate, ambiguity, referenced-content, - and prompt-hashing failures match `ErrPromptLoad`; and -- a nil engine receiver matches `ErrInvalidConfig`. - -Errors preserve useful underlying collaborator and context identities through -`errors.Is` where the existing facade does so, without exposing internal -package types. Context cancellation governs the complete lookup, and no -partial inspection result is returned on failure. - -Inspection cannot return profile, credential, artifact, rendering, schema, -capacity, validation, or model-generation errors because it does not perform -those operations. - -## Compatibility And Boundaries - -This feature is additive. Existing prompt formats, prompt source selection, -`Prepare`, prepared execution, `Run`, profile inspection, and stable JSON -contracts remain unchanged. - -The method belongs on the root `Engine` facade. Prompt repositories and -internal domain definitions remain implementation details, and consumers -cannot use inspection to replace or mutate registered prompt definitions. -Engine construction retains its ordinary configuration requirements; this -feature does not introduce a separate prompt-only engine. - -Inspection is intentionally definition-oriented rather than -execution-oriented. It reports what the prompt declares, not an effective -request after profile selection, per-run overrides, artifacts, variables, -schema resolution, or rendering. - -## Documentation - -When implemented, documentation should retain these ownership boundaries: - -- exported declarations and GoDoc own the exact method, result, ownership, - hash, error, and cancellation contracts; -- the Promptkit consumer guide explains configuration-time prompt inspection - and distinguishes it from profile inspection, `Prepare`, and prepared - execution; -- the format reference continues to own prompt fields, source selection, - content-file resolution, and version behavior; and -- internal documentation describes shared prompt loading and hashing without - duplicating the public contract. - -## Non-Goals - -This work does not include: - -- enumerating, searching, or filtering prompt definitions; -- validating every definition across one or more prompt sources; -- returning raw definitions, YAML, source paths, message bodies, rendered - messages, prompt descriptions, or session-ID templates; -- parsing or rendering templates or accepting placeholder inputs and - variables; -- resolving the default profile, backend, effective execution target, or - credentials; -- loading, compiling, or returning JSON Schema documents or transitive - references; -- validating application-specific relationships among prompts; -- freezing a mutable prompt source for later execution; -- exposing or mutating an internal prompt repository; -- adding a stable JSON representation for the new inspection values; or -- changing current prompt selection, hashing, preparation, or execution - behavior. - -## Target End State - -After this work: - -- consumers can validate one configured prompt without inventing artifacts, - variables, or an executable profile; -- lookup uses ordinary prompt-source selection and exact ID/version semantics; -- a successful result proves that the selected definition and its referenced - message content are structurally loadable; -- callers receive validated identity, complete declared input metadata, the - default profile ID, the normalized output contract, and an opaque prompt - equality value; -- the equality value matches ordinary preparation for the same selected - definition and observed source state; -- no prompt body, rendered message, schema body, credential, or execution - target is exposed; -- returned values are caller-owned and safe to inspect concurrently; and -- broader corpus validation, enumeration, schema validation, and executable - preparation remain separate responsibilities. diff --git a/docs/roadmap/weatherreporter-promptkit-wishlist.md b/docs/roadmap/weatherreporter-promptkit-wishlist.md index c21c840..af45ed5 100644 --- a/docs/roadmap/weatherreporter-promptkit-wishlist.md +++ b/docs/roadmap/weatherreporter-promptkit-wishlist.md @@ -7,10 +7,9 @@ additions to PromptKit from the perspective of the maintainers of Weatherreporter, a downstream application planning to replace its Scriptorium CLI integration with PromptKit. -PromptKit v0.3.0 provides the capabilities Weatherreporter needs for the -migration. None of the ideas below is a hard adoption requirement. They are -opportunities to avoid duplicate preparation, validate configuration earlier, -improve durable failure diagnostics, and make the integration more direct. +PromptKit now provides the capabilities Weatherreporter needs for the +migration. The remaining deferred ideas are optional opportunities to validate +configuration earlier and improve durable failure diagnostics. The examples are API sketches intended to communicate the desired capability, not prescriptive names or finalized Go contracts. The related @@ -19,8 +18,9 @@ overlapping features from another downstream consumer's perspective. ## Priority 1: Executable Preparation Handles -**Disposition:** Accepted into the -[executable preparation handles](prepared-execution.md) feature roadmap. +**Disposition:** Implemented as [`Engine.PrepareExecution` and +`Engine.RunPrepared`](../../engine.go). See the +[consumer guidance](../consumers/pkg-promptkit.md#prepare-now-and-execute-the-same-snapshot-later). ### Downstream need @@ -35,74 +35,20 @@ needs to: Persisting preflight before generation leaves useful evidence when a provider call fails or the process is interrupted during generation. -### Current integration option +### Implemented behavior -With PromptKit v0.3.0, Weatherreporter can call `Engine.Prepare`, save selected -fields from the returned `PreparedRun`, and then call `Engine.Run` with the -same request. Because `Run` performs preparation internally, the work is -repeated. - -Weatherreporter plans to use embedded prompt and schema files plus immutable -inline input bytes, which removes most of the consistency risk. An external -profile file or directory can still change between the two calls, and the -second preparation remains unnecessary work. - -The atomic `RunDetailed` operation proposed by the -[Notarius wishlist](notarius-promptkit-wishlist.md#priority-1-atomic-execution-with-prepared-details) -would guarantee that returned preparation details describe the completed -execution. However, returning those details only after generation would not -preserve Weatherreporter's preflight-before-generation persistence boundary. - -### Requested capability - -Add an opt-in two-phase API that returns a prepared execution handle: - -```go -prepared, err := engine.PrepareExecution(ctx, request) -if err != nil { - // Handle preparation failure. -} - -details := prepared.Details() -// Persist a consumer-selected safe preparation record. - -result, err := engine.RunPrepared(ctx, prepared) -``` - -The exact names and shapes are flexible. The important contract is that -`RunPrepared` executes the already prepared prompt and does not reload or -rerender its prompt, profile, schema, or input sources. - -`Details` should return the same caller-owned public preparation information -currently represented by `PreparedRun`. The execution handle may retain opaque -engine-owned state needed to invoke the model and validate the response. - -### Design considerations - -- Keep `Prepare` and `Run` available for consumers that do not need a - two-phase execution boundary. -- Bind a prepared handle to the engine that constructed it. -- Define whether a handle is one-shot, reusable, or safe for concurrent use. - A one-shot contract may be the safest initial design. -- Do not give the opaque handle a stable JSON representation. -- Do not expose or serialize resolved credential values through `Details`. -- Define how a direct request API key is retained and released when an opaque - handle must carry it until execution. -- Preserve caller-owned copies for all public details. -- Make context cancellation and backend admission timing explicit. -- Document whether profile credential environment values are resolved during - preparation or execution. -- Ensure an execution error does not invalidate the public details already - returned to the consumer. -- Consider whether an atomic `RunDetailed` can share the same internal - prepared-execution implementation. +Weatherreporter can prepare one frozen execution snapshot, persist a +caller-owned and credential-redacted `Details` value, and execute that same +snapshot through `RunPrepared`. The opaque handle is engine-bound and +single-use; an unused handle can be released with `Discard`. The consumer +guide and exported GoDoc own the exact lifecycle, credential, cancellation, +and capacity contracts. ### Value to Weatherreporter -This is the highest-value upstream addition. It would preserve -Weatherreporter's durable preflight behavior, remove duplicate work, eliminate -the remaining source-consistency window, and ensure that persisted provenance -describes the actual execution. +This preserves Weatherreporter's durable preflight behavior, removes duplicate +work, eliminates the source-consistency window, and ensures that persisted +provenance describes the actual execution. ## Priority 2: Prompt-Definition Inspection @@ -312,12 +258,13 @@ migration priority. ## Capabilities PromptKit Already Provides Well -PromptKit v0.3.0 already provides the essential Weatherreporter integration -surface: +PromptKit already provides the essential Weatherreporter integration surface: - importable in-process engine construction; - filesystem, `fs.FS`, and programmatic prompt, profile, and schema sources; - offline preparation without model execution; +- prepared execution handles for a durable preflight boundary; +- exact prompt and profile inspection; - versioned prompt selection; - text, Markdown, JSON, and JSON Schema output contracts; - single-pass output validation with raw output retained after completed @@ -329,7 +276,8 @@ surface: - injected model-client and artifact-reader interfaces; - caller cancellation, generation timeout, and transport timeout behavior; and - public error sentinels for configuration, prompt, profile, artifact, - validation, capacity, and generation failures. + validation, capacity, and generation failures; and +- structured backend identity for capacity rejection. These capabilities are sufficient for Weatherreporter to adopt PromptKit without waiting for new upstream work. @@ -354,41 +302,30 @@ PromptKit: ## Suggested Upstream Sequence -If the PromptKit team chooses to pursue these ideas, the most useful order for -Weatherreporter would be: +For downstream adoption and any remaining upstream work, the useful order is: -1. Add executable preparation handles, ideally sharing implementation with an - atomic detailed-run API. +1. Adopt the implemented executable preparation handles. 2. Consider eager source validation after evaluating whether the two exact inspection APIs are sufficient. 3. Add structured generation errors. 4. Use structured capacity errors and consider semantic execution-target fingerprints as lower-priority operational improvements. -The first item removes the only material integration workaround. Prompt and -profile inspection improve fail-fast validation. The remaining items improve -ergonomics and diagnostics. +The first item removes the material integration workaround. Prompt and profile +inspection improve fail-fast validation. The remaining items are optional +ergonomic and diagnostic improvements. ## Adoption Sequencing -Weatherreporter should not wait for the complete wishlist. PromptKit v0.3.0 is -already sufficient when Weatherreporter: +Weatherreporter should not wait for the deferred wishlist items. The current +PromptKit interface is sufficient when Weatherreporter: - embeds immutable prompt and schema assets; - supplies immutable inline data-package bytes; - constructs one engine per CLI invocation; -- calls `Prepare` and `Run` with the same request; and +- prepares an execution, persists selected `Details`, and calls + `RunPrepared`; and - keeps PromptKit behind a weatherreporter-owned adapter contract. -If executable preparation handles are scheduled for a near-term PromptKit -release, Weatherreporter may defer only its final adapter implementation to -avoid implementing and then removing duplicate preparation. Prompt corpus -retrieval, application-contract design, configuration work, embedded assets, -state contracts, and offline fixtures can proceed independently. - -If the feature is not scheduled, Weatherreporter can adopt v0.3.0 and keep the -duplicate `Prepare` and `Run` sequence inside its adapter. A later PromptKit -upgrade would remain localized behind that neutral boundary. - Prompt inspection, profile inspection, source validation, structured errors, capacity details, and semantic fingerprints should not gate adoption.