Compare commits
3 Commits
v0.3.0
...
c13e9710d9
| Author | SHA1 | Date | |
|---|---|---|---|
| c13e9710d9 | |||
| 87b5ec3d75 | |||
| cb4028a637 |
@@ -33,9 +33,52 @@ consumers.
|
||||
|
||||
## Ideas
|
||||
|
||||
No ideas are currently cataloged. Backend-specific concurrency management has
|
||||
been selected for active planning in the
|
||||
[focused concurrency roadmap](concurrency.md).
|
||||
Executable preparation handles have been selected for active planning in the
|
||||
[focused feature roadmap](prepared-execution.md). The remaining ideas are
|
||||
still available for future selection.
|
||||
|
||||
### Prompt-independent profile inspection
|
||||
|
||||
Provide exact profile lookup and structural resolution without requiring a
|
||||
synthetic prompt, placeholder inputs, or model generation. This shared need is
|
||||
described by
|
||||
[Notarius](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
|
||||
and
|
||||
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-3-prompt-independent-profile-inspection).
|
||||
|
||||
- Apply ordinary built-in, file-backed, and programmatic profile precedence.
|
||||
- Validate referenced backend membership and the structurally resolved
|
||||
execution target.
|
||||
- Report credential requirements and environment-variable names without
|
||||
exposing credential values or requiring current credential availability.
|
||||
- Support exact lookup by profile ID; enumeration is not required initially.
|
||||
|
||||
### Prompt-definition inspection
|
||||
|
||||
Provide exact prompt-definition lookup without rendering, placeholder inputs,
|
||||
profile resolution, or model generation, as requested by
|
||||
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-2-prompt-definition-inspection).
|
||||
|
||||
- Return caller-owned identity, version, input definitions, default-profile,
|
||||
output-contract, and opaque definition-equality information.
|
||||
- Apply ordinary prompt-source precedence and exact ID/version selection.
|
||||
- Validate the selected definition and referenced prompt content
|
||||
structurally, without returning source bodies or rendered messages.
|
||||
- Leave complete cross-source corpus validation and enumeration outside the
|
||||
initial inspection contract.
|
||||
|
||||
### Structured capacity errors
|
||||
|
||||
Add safe structured context to backend admission rejection, as requested by
|
||||
[Notarius](notarius-promptkit-wishlist.md#priority-4-structured-capacity-errors)
|
||||
and
|
||||
[Weatherreporter](weatherreporter-promptkit-wishlist.md#structured-capacity-errors).
|
||||
|
||||
- Preserve compatibility with `errors.Is(err, ErrCapacityExceeded)`.
|
||||
- Support `errors.As` to obtain the stable backend ID.
|
||||
- Do not expose endpoints, credential configuration or values, request
|
||||
content, or speculative retry timing.
|
||||
- Keep retry and backoff policy with downstream consumers.
|
||||
|
||||
## Entry Format
|
||||
|
||||
|
||||
@@ -1,315 +1,683 @@
|
||||
# Local Backend Convenience Implementation Plan
|
||||
# Executable Preparation Handles Implementation Plan
|
||||
|
||||
**Status:** Complete.
|
||||
**Status:** Accepted.
|
||||
|
||||
## Purpose
|
||||
|
||||
This document is the decision-complete implementation plan for
|
||||
[local backend convenience](local-backend.md). It is written for a coding
|
||||
agent that will implement each stage in order.
|
||||
[executable preparation handles](prepared-execution.md). It is written for a
|
||||
coding agent that will implement each stage in order.
|
||||
|
||||
The feature roadmap owns the motivation, consumer paths, policy choices,
|
||||
The feature roadmap owns the motivation, consumer workflow, policy choices,
|
||||
compatibility requirements, non-goals, and target end state. This document
|
||||
owns the exact proposed API, file-level changes, implementation sequence, test
|
||||
owns the concrete design, file-level changes, implementation sequence, test
|
||||
ownership, documentation work, validation commands, and completion gates.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
- Complete the stages in order. Keep the root package compiling and its
|
||||
focused tests passing at every stage boundary.
|
||||
- Preserve unrelated working-tree changes. The feature roadmap may already be
|
||||
uncommitted when implementation begins; retain it.
|
||||
- Complete the stages in order. Keep the repository compiling and the focused
|
||||
tests passing at every stage boundary.
|
||||
- Preserve unrelated working-tree changes. In particular, retain the accepted
|
||||
roadmap and the downstream wishlist and future-roadmap edits that may
|
||||
already be present.
|
||||
- Follow every policy under `docs/policy/`, the task-specific reading guide in
|
||||
`docs/development.md`, and the accepted behavior in
|
||||
`local-backend.md`.
|
||||
- Keep the API in the root `promptkit` package. Do not add a public or internal
|
||||
package for this feature.
|
||||
- Implement the helper as a transparent constructor for the existing
|
||||
`Backend` type. Do not add a second backend representation or bypass
|
||||
`WithBackend`.
|
||||
- Leave normalization, validation, copying, registry construction, capacity
|
||||
defaulting, and duplicate detection in their existing owners.
|
||||
- Do not add environment-variable discovery, package-global registration,
|
||||
implicit local defaults, model selection, profile construction, or support
|
||||
for non-OpenAI-compatible transports.
|
||||
- Keep tests lean and behavior-focused. Do not duplicate the existing
|
||||
registry and concurrency test matrices merely because the constructor
|
||||
reaches those mechanisms.
|
||||
- Update exact GoDoc with the exported declarations. Update current-state
|
||||
consumer guidance only after the corresponding API is implemented.
|
||||
- Do not update release notes, create a release, change a module version, or
|
||||
tag a commit as part of this work.
|
||||
`prepared-execution.md`.
|
||||
- Keep the supported API in the root `promptkit` package. Internal lifecycle,
|
||||
validation, and execution state must remain below Go's `internal/`
|
||||
boundary.
|
||||
- Preserve the observable behavior and ordering of existing `Engine.Prepare`
|
||||
and `Engine.Run`. Do not implement the new workflow by redefining either
|
||||
existing method in terms of the new public methods.
|
||||
- Reuse preparation and post-preparation execution mechanics where their
|
||||
ordering is genuinely shared. Preserve `Run`'s existing early admission
|
||||
boundary while ensuring `PrepareExecution` performs complete preparation
|
||||
without admission.
|
||||
- Do not add durable serialization, retries, provider failover, background
|
||||
workers, a task queue, capacity reservation during preparation, or
|
||||
consumer-specific persistence and redaction policy.
|
||||
- Keep direct API-key values in private execution state only. Every public
|
||||
details value, formatted handle, JSON value, result, hash, and error must
|
||||
remain credential-free.
|
||||
- Keep tests lean but give durable coverage to the public compatibility,
|
||||
security, concurrency, one-attempt, source-snapshot, cancellation, and
|
||||
capacity invariants. Use real internal collaborators where they are fast and
|
||||
deterministic and fakes at model-generation and synchronization boundaries.
|
||||
- Update exact GoDoc with exported declarations. Update current-state
|
||||
consumer and internal documentation only after the corresponding behavior
|
||||
exists.
|
||||
- Do not add release notes, change a module version, create a release, or tag
|
||||
a commit as part of this work.
|
||||
|
||||
## Fixed Design
|
||||
|
||||
### Exported API
|
||||
### Public API
|
||||
|
||||
Add these declarations to `backends.go` in package `promptkit`:
|
||||
Add an opaque root-package type and these methods:
|
||||
|
||||
```go
|
||||
// BackendLocal is the conventional ID used by LocalBackend. It is not a
|
||||
// built-in or reserved backend and must be registered with WithBackend.
|
||||
const BackendLocal = "local"
|
||||
|
||||
// LocalBackend returns a Backend for a conventional local OpenAI-compatible
|
||||
// endpoint.
|
||||
func LocalBackend(endpoint string, concurrencyLimit int) Backend {
|
||||
return Backend{
|
||||
ID: BackendLocal,
|
||||
Endpoint: endpoint,
|
||||
ConcurrencyLimit: concurrencyLimit,
|
||||
type PreparedExecution struct {
|
||||
// Unexported Promptkit-owned state only.
|
||||
}
|
||||
|
||||
func (e *Engine) PrepareExecution(
|
||||
ctx context.Context,
|
||||
req RunRequest,
|
||||
) (*PreparedExecution, error)
|
||||
|
||||
func (p *PreparedExecution) Details() PreparedRun
|
||||
|
||||
func (p *PreparedExecution) Discard()
|
||||
|
||||
func (e *Engine) RunPrepared(
|
||||
ctx context.Context,
|
||||
prepared *PreparedExecution,
|
||||
) (*RunResult, error)
|
||||
```
|
||||
|
||||
Place the public handle and its handle-local methods in a new root file named
|
||||
`prepared_execution.go`. Keep the two engine operations in `engine.go` beside
|
||||
`Prepare` and `Run` so all engine workflows remain discoverable together.
|
||||
|
||||
`PreparedExecution` must contain no exported fields. It wraps one pointer to
|
||||
Promptkit-owned internal state; copying the public struct therefore preserves
|
||||
one shared lifecycle rather than creating another execution opportunity.
|
||||
|
||||
Implement both:
|
||||
|
||||
```go
|
||||
func (p *PreparedExecution) String() string
|
||||
func (p *PreparedExecution) GoString() string
|
||||
```
|
||||
|
||||
with a constant credential- and content-free representation such as
|
||||
`promptkit.PreparedExecution{opaque}`. These methods must not branch on,
|
||||
inspect, or format retained execution state. Do not add a custom JSON
|
||||
representation: with no exported fields, ordinary JSON encoding exposes no
|
||||
state, and JSON is explicitly outside the handle contract.
|
||||
|
||||
The exact GoDoc must own these behaviors:
|
||||
|
||||
- `PrepareExecution` performs complete preparation but no model call and no
|
||||
backend-capacity admission;
|
||||
- the handle is bound to the creating engine and permits one
|
||||
`RunPrepared` invocation;
|
||||
- `Details` returns a fresh caller-owned, credential-redacted copy on every
|
||||
call and remains usable after discard or execution;
|
||||
- `Discard` is nil-safe and idempotent, invalidates an unclaimed handle, and
|
||||
is not an execution-cancellation mechanism;
|
||||
- `RunPrepared` atomically consumes the one attempt before credential
|
||||
revalidation, admission, generation, or validation;
|
||||
- lifecycle misuse matches `ErrInvalidRequest`, while a nil engine matches
|
||||
`ErrInvalidConfig`;
|
||||
- an operational execution error returns no partial `RunResult`;
|
||||
- the `RunPrepared` context does not inherit from the preparation context and
|
||||
governs the execution attempt; and
|
||||
- handles are opaque in-process values, not serializable or restartable jobs.
|
||||
|
||||
For a nil or zero-value handle, `Details` returns a zero `PreparedRun` and
|
||||
`Discard` does nothing. A nil, zero-value, foreign-engine, discarded,
|
||||
claimed, or used handle passed to `RunPrepared` returns an error matching
|
||||
`ErrInvalidRequest`.
|
||||
|
||||
### Internal Validation Snapshot
|
||||
|
||||
Extend `internal/validate/validator.go` with two internal interfaces:
|
||||
|
||||
```go
|
||||
// PreparedValidation validates artifacts against one frozen output contract.
|
||||
type PreparedValidation interface {
|
||||
Validate(
|
||||
ctx context.Context,
|
||||
artifact *domain.Artifact,
|
||||
) (domain.ValidationResult, error)
|
||||
|
||||
// SchemaDocument returns the root JSON Schema document used for provider
|
||||
// structured output, or nil for non-schema modes. The returned value is
|
||||
// internal immutable state and must not be mutated by callers.
|
||||
SchemaDocument() any
|
||||
}
|
||||
|
||||
// ValidationPreparer freezes the validation resources for one output
|
||||
// contract.
|
||||
type ValidationPreparer interface {
|
||||
PrepareValidation(
|
||||
ctx context.Context,
|
||||
contract domain.OutputContract,
|
||||
) (PreparedValidation, error)
|
||||
}
|
||||
```
|
||||
|
||||
The exact GoDoc may be wrapped or expanded for clarity, but it must own and
|
||||
communicate all of these contract points:
|
||||
The exact names may be adjusted only to avoid an actual Go naming conflict;
|
||||
the separation of responsibilities and method shapes are fixed.
|
||||
`validate.Validator` and its existing `Validate` method remain unchanged so
|
||||
the existing `Run` behavior and internal test fakes do not acquire a new
|
||||
mandatory method.
|
||||
|
||||
- `BackendLocal` is the case-sensitive conventional ID `"local"`;
|
||||
- it is neither built in nor reserved;
|
||||
- calling `LocalBackend` does not register anything;
|
||||
- the returned value must be supplied through `WithBackend`;
|
||||
- `endpoint` and `concurrencyLimit` are copied into the corresponding fields
|
||||
without normalization or validation;
|
||||
- `APIKeyEnv`, `ExtraParams`, and `QueueCapacity` retain their zero values; and
|
||||
- normal `NewEngine` backend validation and concurrency semantics apply after
|
||||
registration.
|
||||
`StandardValidator` and `FSValidator` implement `ValidationPreparer`. Their
|
||||
prepared-validation implementation:
|
||||
|
||||
Keep `BackendOpenRouter` unchanged. `BackendLocal` must not alias an internal
|
||||
registry constant because the internal registry has no special local-backend
|
||||
identity or behavior.
|
||||
- captures a value copy of the complete `OutputContract`;
|
||||
- handles none, basic, and JSON validation without any source access;
|
||||
- for JSON Schema, loads the root document, validates its dialect, registers
|
||||
it with a new compiler, compiles it during `PrepareValidation`, and retains
|
||||
the compiled schema;
|
||||
- allows the compiler to resolve every transitive reference during that
|
||||
compilation using the existing contained filesystem or `fs.FS` loader;
|
||||
- returns the same loaded root document through `SchemaDocument`, so the
|
||||
provider-facing structured-output constraint and later validation derive
|
||||
from one root snapshot; and
|
||||
- validates generated instances only against retained in-memory state. Its
|
||||
`Validate` method must not reopen a path, call `fs.ReadFile`, resolve a
|
||||
symlink, or invoke a schema loader after preparation succeeds.
|
||||
|
||||
Place `BackendLocal` near `BackendOpenRouter`, and place `LocalBackend` after
|
||||
the `Backend` declaration and before `WithBackend`. This keeps the conventional
|
||||
IDs, configured value, convenience constructor, and registration option
|
||||
discoverable in one file.
|
||||
Reuse the existing JSON parsing, validation-result construction, schema
|
||||
dialect checks, contained-path rules, compiler configuration, and validation
|
||||
error wording. A prepared JSON Schema compilation or reference-loading failure
|
||||
is a preparation-time validation error. Existing direct calls to
|
||||
`Validator.Validate`, including the existing `Run` path, retain their current
|
||||
lazy compilation and source-loading behavior.
|
||||
|
||||
### Constructor Semantics
|
||||
Do not expose the JSON Schema library's compiled type outside
|
||||
`internal/validate`. Do not place compiled schemas or validation interfaces in
|
||||
`internal/domain`.
|
||||
|
||||
`LocalBackend` is a pure struct constructor. Its complete behavior is
|
||||
equivalent to the keyed literal shown above.
|
||||
### Internal Prepared-Execution State
|
||||
|
||||
In particular, the constructor must not:
|
||||
Add `internal/usecase/prepared_execution.go`. Define an internal
|
||||
`PreparedExecution` owned by one `Runner`. It contains:
|
||||
|
||||
- trim or parse the endpoint;
|
||||
- reject blank endpoints or negative limits;
|
||||
- choose a default limit;
|
||||
- assign an API-key environment variable;
|
||||
- allocate an empty `ExtraParams` map;
|
||||
- assign a queue-capacity pointer;
|
||||
- read process environment;
|
||||
- mutate package-global or engine state; or
|
||||
- call `WithBackend` itself.
|
||||
- the immutable owner `*Runner`;
|
||||
- a mutex;
|
||||
- a small private lifecycle state with at least ready, claimed, and discarded
|
||||
states;
|
||||
- a credential-redacted prepared-details snapshot that survives every
|
||||
lifecycle transition; and
|
||||
- while ready, a pointer to a private payload containing the full
|
||||
`domain.PreparedRun` and its `validate.PreparedValidation`.
|
||||
|
||||
Deferred validation is intentional. It keeps one validation path for all
|
||||
`Backend` values: `WithBackend` copies the public value into engine options,
|
||||
and `NewEngine` constructs the validated immutable registry. A positive limit
|
||||
with a nil queue continues to select the existing default queue capacity of
|
||||
1024; zero continues to mean unlimited; a negative value continues to make
|
||||
`NewEngine` fail with `ErrInvalidConfig`.
|
||||
The payload is the only retained object containing `RunRequest.APIKey`.
|
||||
Construct the public-details snapshot as a deep copy of the prepared run and
|
||||
clear its `EffectiveModelParams.APIKey`. Deep-copy every mutable nested value,
|
||||
including messages, maps, target extra parameters, and the structured-output
|
||||
schema. Reuse the repository's JSON-compatible copy helpers rather than
|
||||
maintaining an ad hoc reflection copier.
|
||||
|
||||
The returned `Backend` remains an ordinary caller-owned value. Consumers can
|
||||
modify it before passing it to `WithBackend`, although task-oriented
|
||||
documentation should direct materially customized configurations to an
|
||||
explicit keyed `Backend` literal.
|
||||
|
||||
### Identity And Compatibility
|
||||
|
||||
Do not add `"local"` to the internal built-in or reserved ID set. A consumer
|
||||
must be able to register either:
|
||||
Add these internal operations:
|
||||
|
||||
```go
|
||||
promptkit.LocalBackend(endpoint, limit)
|
||||
func (r *Runner) PrepareExecution(
|
||||
ctx context.Context,
|
||||
req domain.RunRequest,
|
||||
) (*PreparedExecution, error)
|
||||
|
||||
func (p *PreparedExecution) Details() *domain.PreparedRun
|
||||
|
||||
func (p *PreparedExecution) Discard()
|
||||
|
||||
func (r *Runner) RunPrepared(
|
||||
ctx context.Context,
|
||||
prepared *PreparedExecution,
|
||||
) (*domain.RunResult, error)
|
||||
```
|
||||
|
||||
or:
|
||||
Internal exported names are acceptable here because the root facade must call
|
||||
them, but they remain inaccessible to downstream consumers through Go's
|
||||
`internal` rule.
|
||||
|
||||
```go
|
||||
promptkit.Backend{
|
||||
ID: promptkit.BackendLocal,
|
||||
Endpoint: endpoint,
|
||||
}
|
||||
```
|
||||
`PrepareExecution` performs these operations in order:
|
||||
|
||||
through the existing `WithBackend` option. Both forms participate in ordinary
|
||||
duplicate-ID detection. Existing consumers that already use the literal ID
|
||||
`"local"` remain source- and behavior-compatible.
|
||||
1. call the existing request-copy boundary in the root facade before entering
|
||||
the runner;
|
||||
2. run `resolvePreparation`, including the existing environment credential
|
||||
availability check;
|
||||
3. prepare a frozen validation plan through
|
||||
`validate.ValidationPreparer`;
|
||||
4. construct provider structured-output metadata from the plan's root schema
|
||||
document for JSON Schema mode;
|
||||
5. load and hash artifacts, render the session and messages, and assemble the
|
||||
complete `domain.PreparedRun`; and
|
||||
6. construct the ready internal handle with independent full and redacted
|
||||
snapshots.
|
||||
|
||||
No existing `Backend`, `WithBackend`, profile, registry, execution-target, or
|
||||
capacity semantics change. Do not modify `internal/backend`,
|
||||
`internal/capacity`, `internal/domain`, `engine.go`, `profiles.go`, or
|
||||
`types.go` for this feature.
|
||||
Extract the artifact-loading, rendering, hashing, timing, and
|
||||
`domain.PreparedRun` assembly portion of `completePreparation` into a helper
|
||||
that accepts an already-resolved `*domain.StructuredOutputSpec`.
|
||||
`completePreparation` must continue to call the existing
|
||||
`resolveStructuredOutput` first and then use that helper. This preserves
|
||||
existing `Prepare` and `Run` schema behavior. `PrepareExecution` uses the
|
||||
frozen validation plan's schema document to build the same structured-output
|
||||
shape without calling `resolveStructuredOutput`.
|
||||
|
||||
When the runner has no validator, preserve the existing runner semantics:
|
||||
validation is skipped and a no-op prepared plan is sufficient. When it has a
|
||||
non-nil validator that does not implement `ValidationPreparer`,
|
||||
`PrepareExecution` fails with `ErrValidation`; do not fall back to a plan that
|
||||
would reopen sources later. The root engine's standard and `fs.FS` validators
|
||||
always implement the new interface.
|
||||
|
||||
### Claim, Cleanup, And Engine Binding
|
||||
|
||||
`RunPrepared` checks engine ownership before attempting a claim. A foreign
|
||||
runner returns `ErrInvalidRequest` without changing the handle. The owning
|
||||
runner then locks the state and atomically:
|
||||
|
||||
1. accepts only the ready state;
|
||||
2. changes it permanently to claimed;
|
||||
3. detaches the private payload from the handle; and
|
||||
4. unlocks before doing any blocking or collaborator work.
|
||||
|
||||
Every valid owning-engine invocation consumes the attempt at this point. Run
|
||||
ID creation, credential failure, pre-canceled context, capacity rejection,
|
||||
generation failure, operational validation failure, and success all leave the
|
||||
handle claimed and unusable.
|
||||
|
||||
The execution call holds the detached payload locally and defers cleanup on
|
||||
every exit. Cleanup must at minimum:
|
||||
|
||||
- overwrite the direct `EffectiveModelParams.APIKey` field with an empty
|
||||
string;
|
||||
- drop the local prepared-run pointer;
|
||||
- drop the frozen-validation reference; and
|
||||
- leave the separate redacted details snapshot intact.
|
||||
|
||||
This is reference cleanup, not a promise of secure Go string-memory erasure.
|
||||
|
||||
`Discard` locks the same state. If ready, it changes the state to discarded,
|
||||
detaches the payload, unlocks, and performs the same cleanup. If claimed or
|
||||
already discarded, it is a no-op. Therefore a `RunPrepared`/`Discard` race has
|
||||
one winner: discard prevents a claim only if it acquires the ready state
|
||||
first; otherwise it does not cancel the running attempt.
|
||||
|
||||
`Details` locks only long enough to read the retained redacted snapshot and
|
||||
returns a fresh deep copy. It must not return a pointer, slice, map, schema, or
|
||||
extra-parameter tree shared with either the handle or another `Details` call.
|
||||
|
||||
### Prepared Execution Mechanics
|
||||
|
||||
Refactor the post-preparation portion of `Runner.Run` into a private execution
|
||||
helper that accepts:
|
||||
|
||||
- a fully prepared `domain.PreparedRun`;
|
||||
- a caller-supplied run ID and execution start time; and
|
||||
- a validation function that accepts the artifact and repair-attempt count.
|
||||
|
||||
The helper owns model generation, artifact construction, content validation,
|
||||
internal repair, final result assembly, and result end timing. The live
|
||||
`Runner.Run` path supplies its existing `validateOutput` function. The
|
||||
prepared path supplies a wrapper around the retained
|
||||
`validate.PreparedValidation`. Both wrappers set the actual repair-attempt
|
||||
count on the returned validation result.
|
||||
|
||||
Keep admission outside this shared helper:
|
||||
|
||||
- `Runner.Run` retains its current sequence of run-ID creation, start time,
|
||||
target resolution, early admission, complete preparation, execution, and
|
||||
release.
|
||||
- `Runner.RunPrepared` claims the handle, creates the run ID and execution
|
||||
start time, revalidates credential availability, admits the frozen backend,
|
||||
executes the shared helper, and releases admission.
|
||||
|
||||
Extract a small private admission helper only if it preserves the exact
|
||||
existing capacity wrapping and release behavior. The limited-backend lease
|
||||
must cover generation, validation, and all internal repair calls and must be
|
||||
released on every return.
|
||||
|
||||
Before admission, `RunPrepared` calls the existing `validateAPIKey` with the
|
||||
frozen environment-variable name, retained direct key, and frozen requirement
|
||||
flag. A missing environment credential returns an error wrapping
|
||||
`ErrInvalidRequest` and `ErrAPIKeyEnvMissing`. Do not read and retain the
|
||||
environment value; the model client continues to resolve it when generation
|
||||
begins.
|
||||
|
||||
The result produced by the shared helper uses the frozen preparation values
|
||||
for all provenance. For `RunPrepared`, `StartTime`, `EndTime`, and `Duration`
|
||||
begin after the successful claim and exclude preparation and consumer-held
|
||||
delay. For ordinary `Run`, retain the current timing boundary beginning before
|
||||
preparation.
|
||||
|
||||
### Error Mapping And Compatibility
|
||||
|
||||
Do not add a new public sentinel. Internal lifecycle errors wrap
|
||||
`usecase.ErrInvalidRequest`, which the existing root mapping exposes as
|
||||
`promptkit.ErrInvalidRequest`.
|
||||
|
||||
`PrepareExecution` uses the same public categories as `Prepare`.
|
||||
`RunPrepared` may return:
|
||||
|
||||
- `ErrInvalidConfig` for a nil engine;
|
||||
- `ErrInvalidRequest` for handle misuse and invalid credentials;
|
||||
- `ErrAPIKeyEnvMissing` together with `ErrInvalidRequest` when the frozen
|
||||
environment name is no longer set and no direct key is retained;
|
||||
- `ErrCapacityExceeded` for admission rejection;
|
||||
- `ErrLLMGenerate` for model-generation failures; and
|
||||
- `ErrValidation` for inability to validate or repair.
|
||||
|
||||
Preserve wrapped collaborator and context errors according to the existing
|
||||
mapping rules. A content validation rejection remains a successful
|
||||
`RunResult` with `ValidationFailed`, not an operational error.
|
||||
|
||||
Do not change stable JSON for `PreparedRun` or `RunResult`, public request and
|
||||
result ownership, provider request bodies or headers, backend selection,
|
||||
capacity policies, `LLMClient`, `ArtifactReader`, `Prepare`, or `Run`.
|
||||
|
||||
### Test Ownership
|
||||
|
||||
The root external-package contract suite in `public_contract_test.go` owns the
|
||||
new public behavior. Add one focused test named:
|
||||
Tests must be split by the narrowest stable owner.
|
||||
|
||||
```go
|
||||
func TestLocalBackendConstructsAndRegistersConventionalBackend(t *testing.T)
|
||||
```
|
||||
`internal/validate/standard_validator_test.go` owns frozen schema mechanics.
|
||||
Add focused tests for both configured schema-source forms:
|
||||
|
||||
The test must:
|
||||
- a directory-backed `StandardValidator` test whose root schema references a
|
||||
second file; after `PrepareValidation`, replace or remove both files and
|
||||
prove that valid and invalid artifacts are judged by the original compiled
|
||||
schema; and
|
||||
- an `FSValidator` test using a mutable in-memory `fs.FS`; mutate its root and
|
||||
referenced entries after `PrepareValidation` and prove the same invariant.
|
||||
|
||||
1. call `promptkit.LocalBackend` with a test endpoint and a positive,
|
||||
test-owned concurrency limit;
|
||||
2. compare the returned value with this complete expected value:
|
||||
Each test must also confirm that `SchemaDocument` is the original root
|
||||
document used at preparation. Existing path-containment and dialect matrices
|
||||
remain the owners of those rules and must not be duplicated.
|
||||
|
||||
```go
|
||||
promptkit.Backend{
|
||||
ID: promptkit.BackendLocal,
|
||||
Endpoint: endpoint,
|
||||
ConcurrencyLimit: limit,
|
||||
}
|
||||
```
|
||||
`internal/usecase` tests own orchestration details that cannot be observed
|
||||
cleanly at the facade:
|
||||
|
||||
A whole-value comparison is appropriate here because the exact zero-value
|
||||
fields are part of this small public constructor's contract.
|
||||
3. register that returned value with `WithBackend`;
|
||||
4. add an in-memory profile whose `BackendID` is
|
||||
`promptkit.BackendLocal`;
|
||||
5. construct an engine through the existing contract-test prompt fixture;
|
||||
6. call `Prepare`; and
|
||||
7. assert that the prepared result exposes `BackendLocal` as the selected
|
||||
backend and the supplied endpoint as the effective endpoint.
|
||||
- `PrepareExecution` completes source loading, artifact reading, rendering,
|
||||
and validation-plan creation without calling admission or generation;
|
||||
- `RunPrepared` rechecks environment credential availability before admission;
|
||||
- the prepared path supplies the frozen validation plan to initial validation
|
||||
and every repair attempt;
|
||||
- admission is released on success and each error exit; and
|
||||
- the ordinary `Run` admission-before-complete-preparation ordering remains
|
||||
unchanged after refactoring.
|
||||
|
||||
This single test protects the realistic compatibility risks: accidental field
|
||||
defaults, a changed conventional ID, failure to compose with `WithBackend`,
|
||||
and accidental treatment of `"local"` as reserved. It also demonstrates that
|
||||
the helper uses the existing backend/profile path.
|
||||
Use small test-controlled collaborators. Do not assert private enum values,
|
||||
mutex layout, helper call graphs, or cleanup implementation details.
|
||||
|
||||
Do not add separate tests for blank endpoints, malformed endpoints, negative
|
||||
limits, queue defaulting, duplicate IDs, engine isolation, runtime capacity,
|
||||
or caller mutation. Those mechanisms are unchanged and already have tests at
|
||||
their owning boundaries. Do not add internal-package tests for this root
|
||||
facade constructor.
|
||||
Add `prepared_execution_contract_test.go` in external package
|
||||
`promptkit_test` for the public contract. Cover these distinct risks with the
|
||||
smallest coherent set of tests:
|
||||
|
||||
### Consumer Documentation
|
||||
1. **Frozen execution and independent details.** Prepare a request, mutate
|
||||
caller-owned request values, mutable prompt/profile/input/schema sources,
|
||||
and one returned `PreparedRun`, then run the handle. Assert that the
|
||||
captured `GenerateRequest`, validation, result provenance, and a second
|
||||
`Details` call all retain the original prepared state. Assert that source
|
||||
collaborators are not reopened during execution.
|
||||
2. **Lifecycle and engine binding.** Prove that a foreign engine is rejected
|
||||
without consuming the handle, the owner can then run it once, a second
|
||||
call through either the original or a copied public handle is rejected, and
|
||||
details remain available.
|
||||
3. **Concurrent claim.** Race two owning-engine `RunPrepared` calls against a
|
||||
blocking fake client. Exactly one reaches generation and the other matches
|
||||
`ErrInvalidRequest`. Run this test under the race detector.
|
||||
4. **Discard and formatting security.** Retain a distinctive direct API key,
|
||||
verify that `Details`, ordinary JSON encoding, `%v`, `%+v`, `%#v`, and
|
||||
lifecycle errors do not contain it or rendered content, discard the handle,
|
||||
and verify idempotence, execution rejection, and retained details.
|
||||
5. **Credential and capacity timing.** Prove that preparation does not occupy
|
||||
admission; removing a required environment credential makes execution fail
|
||||
before generation; a capacity rejection consumes the handle; and the
|
||||
execution result's timing excludes a test-controlled delay between
|
||||
preparation and execution.
|
||||
|
||||
Update `docs/consumers/pkg-promptkit.md` after the API exists. Keep Go
|
||||
declarations and GoDoc as the exact API owner; the guide should help consumers
|
||||
choose a workflow and link to `backends.go` for precise semantics.
|
||||
Reuse existing root contract fixtures and capacity fakes where practical.
|
||||
Never put a real-looking credential in a fixture. Do not test JSON equality
|
||||
for the opaque handle, exact lifecycle error strings, private state, run-ID
|
||||
format, or clock-duration precision.
|
||||
|
||||
Restructure the backend guidance to present these paths in increasing order of
|
||||
configuration:
|
||||
### Documentation Ownership
|
||||
|
||||
1. **Endpoint-only profile.** Show a small in-memory `Profile` with
|
||||
`Endpoint` and `Model`. Explain that this is the simplest choice when only
|
||||
one profile needs the endpoint and shared backend identity or capacity
|
||||
policy is unnecessary.
|
||||
2. **Local convenience constructor.** Show
|
||||
`WithBackend(promptkit.LocalBackend("http://localhost:8000/v1", 2))`
|
||||
together with a profile using
|
||||
`BackendID: promptkit.BackendLocal`. Explain briefly that the helper is
|
||||
explicit, is not pre-registered, does not read environment variables, and
|
||||
leaves the queue capacity at the existing default for a positive limit.
|
||||
3. **Complete backend value.** Preserve an advanced example using a keyed
|
||||
`Backend` literal for needs such as `APIKeyEnv`, an explicit
|
||||
`QueueCapacity`, extra parameters, a custom ID, or multiple local
|
||||
endpoints. Use a custom ID other than `"local"` in that example so the
|
||||
distinction from the conventional helper is clear.
|
||||
After implementation:
|
||||
|
||||
Keep the existing backend-routing, selected-backend identity, concurrency,
|
||||
credential, and error guidance unless a small wording adjustment is required
|
||||
to make the new decision path coherent. Avoid repeating the complete field
|
||||
contract or registry validation rules from GoDoc.
|
||||
- exported declarations and GoDoc in `prepared_execution.go`, `engine.go`, and
|
||||
`types.go` own the exact public lifecycle, ownership, security, timing,
|
||||
cancellation, and error contracts;
|
||||
- `docs/consumers/pkg-promptkit.md` explains when to choose `Prepare`, `Run`,
|
||||
or `PrepareExecution` plus `RunPrepared`, and shows one concise two-phase
|
||||
workflow using `defer prepared.Discard()`;
|
||||
- `docs/internal/sources.md` explains that prepared execution freezes all
|
||||
source-derived state, including transitive schema references, while
|
||||
ordinary `Prepare` remains inspection-only;
|
||||
- `docs/internal/llm.md` explains that prepared generation uses the retained
|
||||
target and messages, rechecks environment credential availability, and
|
||||
does not reopen sources;
|
||||
- `docs/internal/capacity.md` explains that `PrepareExecution` performs no
|
||||
admission and that `RunPrepared` acquires and holds the normal run lease
|
||||
across generation, validation, and internal repair; and
|
||||
- `docs/internal/overview.md` updates the validator and use-case inventory to
|
||||
mention frozen validation plans and one-attempt prepared execution.
|
||||
|
||||
Do not add a new maintained example, README section, format-reference entry,
|
||||
integration-contract change, internal-document change, or release note. The
|
||||
consumer-guide snippets are sufficient for this small convenience API, and
|
||||
none of those other documents owns the affected task or contract.
|
||||
Keep these documents at their established abstraction levels. Link to exact
|
||||
Go declarations rather than reproducing field-by-field contracts. No README,
|
||||
format reference, integration contract, maintained example, architecture
|
||||
policy, release note, or release-procedure update is required because the
|
||||
feature changes none of their owned concerns.
|
||||
|
||||
## Stage 1: Add The Public Constructor And Contract Test
|
||||
## Stage 1: Add Frozen Validation Plans
|
||||
|
||||
**Status:** Complete.
|
||||
**Status:** Pending.
|
||||
|
||||
### Objective
|
||||
|
||||
Add the smallest public API that expresses the accepted local-backend
|
||||
convention and protect its compatibility through the root public boundary.
|
||||
Create an internal validation artifact that eagerly captures every schema
|
||||
resource required by later validation, without changing existing validation
|
||||
or engine behavior.
|
||||
|
||||
### Implementation Prompt
|
||||
|
||||
1. Re-read `docs/development.md`, all files under `docs/policy/`,
|
||||
`local-backend.md`, `backends.go`, the backend-related portion of
|
||||
`public_contract_test.go`, and the existing backend/concurrency GoDoc before
|
||||
editing.
|
||||
2. Confirm the working tree and preserve the uncommitted roadmaps and any
|
||||
unrelated consumer changes.
|
||||
3. Add the untyped exported string constant `BackendLocal = "local"` to
|
||||
`backends.go` without changing `BackendOpenRouter`.
|
||||
4. Add `LocalBackend(endpoint string, concurrencyLimit int) Backend` to
|
||||
`backends.go` using the exact keyed-literal implementation in the fixed
|
||||
design.
|
||||
5. Write complete GoDoc for both declarations. Make their conventional,
|
||||
explicit, non-built-in, non-reserved, and deferred-validation semantics
|
||||
unambiguous.
|
||||
6. Add
|
||||
`TestLocalBackendConstructsAndRegistersConventionalBackend` to
|
||||
`public_contract_test.go` exactly as specified under Test Ownership. Reuse
|
||||
the existing contract prompt fixture rather than adding a fixture or test
|
||||
helper.
|
||||
7. Do not edit internal packages. If the constructor appears to require an
|
||||
internal change, stop and reconcile the implementation with the fixed
|
||||
transparent-constructor design instead.
|
||||
`prepared-execution.md`, `internal/validate/validator.go`,
|
||||
`internal/validate/standard_validator.go`, and the focused validator tests.
|
||||
2. Confirm the working tree and preserve all existing roadmap and wishlist
|
||||
changes.
|
||||
3. Add `PreparedValidation` and `ValidationPreparer` to
|
||||
`internal/validate/validator.go` with the exact responsibilities in the
|
||||
fixed design. Leave `Validator` unchanged.
|
||||
4. Implement `PrepareValidation` for `StandardValidator` and `FSValidator`.
|
||||
Reuse the existing root resolution, resource identifiers, contained
|
||||
loaders, dialect checks, compiler configuration, JSON parser, and
|
||||
validation-result builder.
|
||||
5. Ensure the root document is registered from the already-loaded value and
|
||||
every transitive reference is loaded during compilation. Retain only the
|
||||
compiled schema, root document, and copied contract required for later
|
||||
validation.
|
||||
6. Ensure the prepared validator checks its validation-time context and does
|
||||
not touch either source after successful preparation.
|
||||
7. Add the two focused frozen-reference tests described under Test Ownership.
|
||||
Keep existing tests unchanged except for shared test setup that materially
|
||||
reduces duplication.
|
||||
|
||||
### Focused Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w backends.go public_contract_test.go
|
||||
go test . -run '^TestLocalBackendConstructsAndRegistersConventionalBackend$'
|
||||
gofmt -w internal/validate/validator.go \
|
||||
internal/validate/standard_validator.go \
|
||||
internal/validate/standard_validator_test.go
|
||||
go test ./internal/validate
|
||||
go test -race ./internal/validate
|
||||
go vet ./internal/validate
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Completion Gate
|
||||
|
||||
Stage 1 is complete when:
|
||||
|
||||
- both built-in validator forms produce prepared plans;
|
||||
- prepared schema validation works after root and referenced sources change or
|
||||
disappear;
|
||||
- provider schema metadata can use the exact root document captured by the
|
||||
plan;
|
||||
- existing `Validator.Validate` behavior remains unchanged; and
|
||||
- focused tests, race tests, vet, formatting, and whitespace checks pass.
|
||||
|
||||
## Stage 2: Implement Internal Handle Lifecycle And Prepared Execution
|
||||
|
||||
**Status:** Pending.
|
||||
|
||||
### Objective
|
||||
|
||||
Add complete preparation, one-attempt lifecycle management, cleanup, and
|
||||
source-free execution to the use-case layer while preserving existing
|
||||
`Prepare` and `Run` behavior.
|
||||
|
||||
### Implementation Prompt
|
||||
|
||||
1. Re-read the completed Stage 1 declarations, `internal/usecase/runner.go`,
|
||||
its focused tests, `internal/capacity`, and the execution flow described in
|
||||
`prepared-execution.md`.
|
||||
2. Add `internal/usecase/prepared_execution.go` with the internal handle,
|
||||
payload, lifecycle transitions, deep-copying, redaction, cleanup, and
|
||||
runner ownership described in the fixed design.
|
||||
3. Extract the common final preparation helper that accepts resolved
|
||||
structured-output metadata. Keep `completePreparation` and existing
|
||||
`resolveStructuredOutput` behavior intact for `Prepare` and `Run`.
|
||||
4. Implement `Runner.PrepareExecution` using
|
||||
`validate.ValidationPreparer`. Build JSON Schema structured-output metadata
|
||||
from the frozen plan's root document.
|
||||
5. Refactor post-preparation generation, validation, repair, and result
|
||||
assembly into the shared private helper. Preserve ordinary `Run` timing,
|
||||
early admission, error ordering, and live validator behavior.
|
||||
6. Implement `Runner.RunPrepared`: verify owner, claim once, create run ID and
|
||||
execution start time, recheck credentials, acquire admission, execute
|
||||
against the retained plan, release admission, and clean the detached
|
||||
payload on every exit.
|
||||
7. Implement nil-safe, idempotent discard and fresh redacted details copying.
|
||||
Confirm the `RunPrepared`/`Discard` race is resolved solely by the shared
|
||||
mutex transition.
|
||||
8. Add the internal orchestration tests described under Test Ownership,
|
||||
including regression coverage for ordinary `Run`'s admission ordering.
|
||||
|
||||
### Focused Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/usecase/runner.go \
|
||||
internal/usecase/prepared_execution.go \
|
||||
internal/usecase/*_test.go
|
||||
go test ./internal/usecase
|
||||
go test -race ./internal/usecase
|
||||
go test ./internal/validate
|
||||
go vet ./internal/usecase ./internal/validate
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Completion Gate
|
||||
|
||||
Stage 2 is complete when:
|
||||
|
||||
- a runner can prepare and later execute without reopening any request source;
|
||||
- one valid invocation consumes the handle on every outcome;
|
||||
- foreign-runner rejection does not consume the handle;
|
||||
- discard and claim are race-safe and cleanup drops secret-bearing state;
|
||||
- prepared execution rechecks credentials and admits only at execution time;
|
||||
- frozen validation is reused across initial output and repair attempts;
|
||||
- existing `Prepare` and `Run` tests retain their observable ordering and
|
||||
behavior; and
|
||||
- focused tests, race tests, vet, formatting, and whitespace checks pass.
|
||||
|
||||
## Stage 3: Publish The Root Facade And Public Contract
|
||||
|
||||
**Status:** Pending.
|
||||
|
||||
### Objective
|
||||
|
||||
Expose the accepted opaque handle workflow through the root package and
|
||||
protect its compatibility, security, and concurrency guarantees at the
|
||||
consumer boundary.
|
||||
|
||||
### Implementation Prompt
|
||||
|
||||
1. Re-read the accepted public workflow, root conversion and error-mapping
|
||||
helpers, existing `Prepare` and `Run` GoDoc, and the external-package
|
||||
contract-test conventions.
|
||||
2. Add root `prepared_execution.go` with the opaque wrapper, `Details`,
|
||||
`Discard`, `String`, and `GoString`.
|
||||
3. Add `Engine.PrepareExecution` and `Engine.RunPrepared` to `engine.go`.
|
||||
Reuse `toDomainRunRequest`, `fromDomainPreparedRun`,
|
||||
`fromDomainRunResult`, and `mapPublicError`.
|
||||
4. Add or reuse one public prepared-run deep-copy helper so every `Details`
|
||||
call returns fresh nested state. Do not expose the internal handle or
|
||||
retain a public direct credential.
|
||||
5. Write complete GoDoc for the type and methods. Update `RunRequest`,
|
||||
`PreparedRun`, and `RunResult` GoDoc only where cross-references are needed
|
||||
to distinguish the new workflow; do not change their stable JSON or
|
||||
ownership contracts.
|
||||
6. Add `prepared_execution_contract_test.go` and cover the five risk groups
|
||||
under Test Ownership. Reuse existing fixtures and fakes rather than
|
||||
duplicating backend, profile, schema, and capacity matrices.
|
||||
7. Explicitly inspect formatted and JSON-encoded handles, lifecycle errors,
|
||||
details, captured model requests, and results for the test sentinel direct
|
||||
key.
|
||||
|
||||
### Focused Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w prepared_execution.go engine.go types.go convert.go \
|
||||
prepared_execution_contract_test.go
|
||||
go test . -run 'PreparedExecution|RunPrepared|PrepareExecution'
|
||||
go test -race . -run 'PreparedExecution|RunPrepared|PrepareExecution'
|
||||
go test .
|
||||
go vet .
|
||||
go build .
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Inspect the diff and confirm that this stage changes only `backends.go`,
|
||||
`public_contract_test.go`, and the already-present roadmap files.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
Stage 1 is complete when:
|
||||
Stage 3 is complete when:
|
||||
|
||||
- the exported constant and constructor match the fixed API;
|
||||
- the constructor returns only the three specified non-zero fields;
|
||||
- the helper registers through the ordinary `WithBackend` path;
|
||||
- `"local"` remains a valid consumer registration rather than a reserved
|
||||
built-in;
|
||||
- the focused public-contract test passes; and
|
||||
- the root package test, vet, build, formatting, and whitespace checks pass.
|
||||
- the public API matches the accepted shape;
|
||||
- details are fresh, stable, credential-redacted copies before and after every
|
||||
lifecycle outcome;
|
||||
- default formatting and JSON expose no retained request or credential data;
|
||||
- foreign, copied, concurrent, discarded, and reused handles obey the
|
||||
one-attempt contract;
|
||||
- execution uses exactly the frozen request and validation state;
|
||||
- preparation performs no generation or admission;
|
||||
- result provenance matches details and execution timing excludes preparation
|
||||
and retention delay;
|
||||
- public errors preserve the required `errors.Is` identities; and
|
||||
- focused public, race, package, vet, build, formatting, and whitespace checks
|
||||
pass.
|
||||
|
||||
## Stage 2: Publish Consumer Guidance And Validate The Repository
|
||||
## Stage 4: Update Documentation And Validate The Repository
|
||||
|
||||
**Status:** Complete.
|
||||
**Status:** Pending.
|
||||
|
||||
### Objective
|
||||
|
||||
Make the simplest suitable local-endpoint configuration easy to discover,
|
||||
confirm the complete change across the repository, and close the temporary
|
||||
roadmaps.
|
||||
Publish current-state guidance at the correct documentation owners, validate
|
||||
the complete repository, and mark the temporary roadmaps complete.
|
||||
|
||||
### Implementation Prompt
|
||||
|
||||
1. Re-read the implemented declarations and GoDoc before describing them.
|
||||
2. Update `docs/consumers/pkg-promptkit.md` according to the three-path
|
||||
structure under Consumer Documentation.
|
||||
3. Keep examples illustrative, minimal, secret-free, and consistent with the
|
||||
implemented declarations. Link precise semantics to `backends.go` rather
|
||||
than duplicating its field-by-field contract.
|
||||
1. Re-read the implemented declarations and tests before documenting them.
|
||||
2. Update `docs/consumers/pkg-promptkit.md`,
|
||||
`docs/internal/sources.md`, `docs/internal/llm.md`,
|
||||
`docs/internal/capacity.md`, and `docs/internal/overview.md` according to
|
||||
Documentation Ownership.
|
||||
3. Keep the consumer example concise, copyable, secret-free, and explicit
|
||||
about `defer prepared.Discard()`. Link to declarations for exact lifecycle
|
||||
and error semantics.
|
||||
4. Follow every added or changed Markdown link and confirm its target exists.
|
||||
Confirm that all local repository-relative links in
|
||||
`local-backend.md`, this plan, and the changed consumer guide resolve.
|
||||
Confirm that repository-relative links in both roadmap documents and all
|
||||
changed current-state documents resolve.
|
||||
5. Run the complete maintainer validation sequence below.
|
||||
6. Inspect the final diff for accidental internal behavior changes,
|
||||
generated artifacts, credentials, local workspace files, or module
|
||||
replacements.
|
||||
7. After every completion gate passes, set both stage statuses, this plan's
|
||||
status, and the status in `local-backend.md` to `Complete`. Do not retire or
|
||||
remove the roadmaps in the implementation change; roadmap retirement
|
||||
follows implementation review.
|
||||
6. Inspect the final diff for accidental provider-wire changes, source
|
||||
reloading during prepared execution, stable-JSON changes, credentials,
|
||||
generated artifacts, local workspace files, module replacements, or
|
||||
unrelated edits.
|
||||
7. After every completion gate passes, set every stage status, this plan's
|
||||
status, and the status in `prepared-execution.md` to `Complete`. Do not
|
||||
retire or remove the roadmaps in the implementation change; roadmap
|
||||
retirement follows implementation review.
|
||||
|
||||
### Full Validation
|
||||
|
||||
@@ -318,69 +686,37 @@ Run the complete sequence from `docs/development.md`:
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go run ./examples/go-library/prepare
|
||||
go vet ./...
|
||||
go build ./...
|
||||
go run ./examples/go-library/prepare
|
||||
gofmt -l $(git ls-files '*.go')
|
||||
git diff --check
|
||||
```
|
||||
|
||||
The formatting command must produce no paths. Follow every added or changed
|
||||
Markdown link and confirm its target and heading exist.
|
||||
|
||||
Also inspect:
|
||||
Then run the documented Markdown-link and repository-hygiene checks from
|
||||
`docs/development.md`, followed by:
|
||||
|
||||
```sh
|
||||
git diff --check
|
||||
git status --short
|
||||
git diff --stat
|
||||
git diff
|
||||
```
|
||||
|
||||
Confirm that:
|
||||
|
||||
- production changes are limited to the root convenience API;
|
||||
- no internal backend, registry, capacity, profile, or engine behavior
|
||||
changed;
|
||||
- `BackendLocal` is conventional and consumer-registerable, not built in or
|
||||
reserved;
|
||||
- `LocalBackend` performs no validation, normalization, environment lookup,
|
||||
registration, allocation, or hidden default selection;
|
||||
- the returned `Backend` leaves `QueueCapacity` nil so existing positive-limit
|
||||
defaulting remains owned by the registry;
|
||||
- existing endpoint-only profiles and complete custom backends remain
|
||||
documented and supported;
|
||||
- current-state documentation describes only the now-implemented API and
|
||||
links to the canonical GoDoc for exact semantics;
|
||||
- no `go.work`, `go.work.sum`, local module replacement, credential,
|
||||
generated binary, or unrelated change was introduced; and
|
||||
- the feature and implementation roadmaps contain no unresolved work marked
|
||||
complete.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
Stage 2 is complete when every target-end-state item in
|
||||
`local-backend.md` is implemented, the consumer guide clearly presents all
|
||||
three configuration paths, all complete validation commands pass, all changed
|
||||
links resolve, and the roadmap statuses accurately report completion.
|
||||
Stage 4 and the feature are complete when:
|
||||
|
||||
## Implementation Handoff
|
||||
|
||||
The implementation handoff should report:
|
||||
|
||||
- the new `BackendLocal` and `LocalBackend` public API;
|
||||
- that the helper remains explicit and composes with the ordinary backend
|
||||
registry;
|
||||
- the focused public-contract coverage added;
|
||||
- the consumer-guide decision path added;
|
||||
- the complete validation commands and results; and
|
||||
- any unrelated working-tree changes that were preserved.
|
||||
|
||||
Do not claim that a local backend is built in, pre-registered, configured from
|
||||
environment variables, or assigned a default model or concurrency limit.
|
||||
- exact public contracts and task-oriented guidance are documented at their
|
||||
canonical owners;
|
||||
- internal source, validation, execution, credential, and capacity boundaries
|
||||
are accurately described without duplicating public GoDoc;
|
||||
- all changed links resolve and no documentation claims unimplemented
|
||||
behavior;
|
||||
- the complete ordinary and race test suites pass offline;
|
||||
- the maintained preparation example, vet, build, formatting, link, hygiene,
|
||||
and whitespace checks pass;
|
||||
- the final diff is limited to the feature, its tests, documentation, and
|
||||
already-present roadmap changes; and
|
||||
- both roadmap documents accurately report completion.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The feature roadmap and this plan fix the exported API, constructor
|
||||
semantics, identity treatment, compatibility behavior, test boundary,
|
||||
documentation ownership, implementation sequence, validation gates, and
|
||||
non-goals required for implementation.
|
||||
None.
|
||||
|
||||
358
docs/roadmap/notarius-promptkit-wishlist.md
Normal file
358
docs/roadmap/notarius-promptkit-wishlist.md
Normal file
@@ -0,0 +1,358 @@
|
||||
# Notarius PromptKit Wishlist
|
||||
|
||||
## Purpose
|
||||
|
||||
This document records features and interface changes that would be useful
|
||||
additions to PromptKit from the perspective of the maintainers of Notarius, a
|
||||
downstream application that consumes PromptKit.
|
||||
|
||||
PromptKit v0.3.0 provides the capabilities Notarius currently needs. None of
|
||||
the ideas below blocks current Notarius development. They are opportunities to
|
||||
reduce downstream workarounds, improve integration correctness, and make
|
||||
PromptKit more ergonomic for applications with configuration validation,
|
||||
debugging, checkpointing, and operational-observability requirements.
|
||||
|
||||
The examples are API sketches intended to communicate the desired capability,
|
||||
not prescriptive names or finalized Go contracts.
|
||||
|
||||
## Priority 1: Atomic Execution With Prepared Details
|
||||
|
||||
**Disposition:** Covered by the accepted
|
||||
[executable preparation handles](prepared-execution.md) roadmap. The shared
|
||||
two-phase capability should provide the required single-preparation
|
||||
consistency; a separate `RunDetailed` method is not cataloged initially.
|
||||
|
||||
### Downstream need
|
||||
|
||||
Notarius needs both:
|
||||
|
||||
- the completed `RunResult`; and
|
||||
- the rendered messages, effective output contract, hashes, and other
|
||||
preparation details exposed by `PreparedRun`.
|
||||
|
||||
Notarius uses the prepared details to construct redaction-aware debug bundles
|
||||
and retain enough information to diagnose model behavior.
|
||||
|
||||
### Current integration
|
||||
|
||||
Notarius currently calls `Engine.Prepare` and then `Engine.Run` with the same
|
||||
request. Because `Run` performs preparation internally, a successful request
|
||||
resolves and prepares the same work twice.
|
||||
|
||||
This duplicates profile resolution, input hashing, schema loading, and prompt
|
||||
rendering. It also creates a theoretical consistency window in which a
|
||||
filesystem-backed prompt, profile, schema, or input could change between the
|
||||
explicit preparation and the preparation performed by `Run`.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Add an opt-in execution method that prepares exactly once and returns both the
|
||||
prepared details and completed result:
|
||||
|
||||
```go
|
||||
type RunReport struct {
|
||||
Prepared PreparedRun
|
||||
Result RunResult
|
||||
}
|
||||
|
||||
func (e *Engine) RunDetailed(
|
||||
ctx context.Context,
|
||||
req RunRequest,
|
||||
) (*RunReport, error)
|
||||
```
|
||||
|
||||
The exact names are flexible. The important contract is that preparation
|
||||
occurs once and that the returned prepared state describes the execution that
|
||||
produced the returned result.
|
||||
|
||||
Existing `Prepare` and `Run` behavior should remain available for consumers
|
||||
that need only one side of the operation.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Keep this API additive and preserve the existing simple `Run` workflow.
|
||||
- Return caller-owned copies under PromptKit's existing ownership rules.
|
||||
- Define whether any prepared details are available after an operational
|
||||
generation or validation error. Notarius does not require partial results
|
||||
for the initial use case, but an explicit contract would be valuable.
|
||||
- Preserve cancellation and backend-admission semantics.
|
||||
- Do not add all prepared content directly to `RunResult`. Rendered prompt
|
||||
content can be large and sensitive, and consumers should opt in to receiving
|
||||
it.
|
||||
|
||||
### Value to Notarius
|
||||
|
||||
This is the highest-value wishlist item. It would remove duplicate work from
|
||||
every successful PromptKit-backed call and ensure that retained debug material
|
||||
corresponds atomically to the actual execution.
|
||||
|
||||
## Priority 2: Prompt-Independent Profile Inspection
|
||||
|
||||
**Disposition:** Accepted into the
|
||||
[future catalog](future.md#prompt-independent-profile-inspection).
|
||||
|
||||
### Downstream need
|
||||
|
||||
Notarius validates configured pipeline profile IDs before beginning a run. It
|
||||
needs to determine whether:
|
||||
|
||||
- a profile exists;
|
||||
- its referenced backend is registered;
|
||||
- its execution target can be resolved; and
|
||||
- it declares a credential requirement that the application may need to
|
||||
enforce.
|
||||
|
||||
This validation should not require model generation.
|
||||
|
||||
### Current integration
|
||||
|
||||
Notarius constructs a synthetic prompt using `testing/fstest.MapFS`, supplies a
|
||||
dummy transcript, and calls `Engine.Prepare` solely to exercise profile and
|
||||
backend resolution. This works, but prompt preparation is serving as a
|
||||
substitute for a profile-inspection interface.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Add a prompt-independent profile-resolution API, for example:
|
||||
|
||||
```go
|
||||
type ResolvedProfile struct {
|
||||
ProfileID string
|
||||
BackendID string
|
||||
EffectiveTarget ExecutionTarget
|
||||
APIKeyEnv string
|
||||
}
|
||||
|
||||
func (e *Engine) ResolveProfile(
|
||||
ctx context.Context,
|
||||
profileID string,
|
||||
) (ResolvedProfile, error)
|
||||
```
|
||||
|
||||
The returned shape may differ, but it should provide enough information for a
|
||||
consumer to validate an explicit profile selection without inventing a prompt
|
||||
or supplying placeholder inputs.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Resolve built-in, file-backed, and programmatic profiles using normal
|
||||
PromptKit precedence.
|
||||
- Validate that a referenced backend registration exists.
|
||||
- Do not resolve, retain, or expose credential values.
|
||||
- Report credential requirements, such as an environment-variable name, so
|
||||
the consuming application can decide whether availability is required at
|
||||
configuration-validation time or only at execution time.
|
||||
- Return caller-owned values.
|
||||
- Preserve typed or sentinel error classification for missing and invalid
|
||||
profiles.
|
||||
- Consider accepting an `ExecutionTargetOverride` if consumers need to inspect
|
||||
the same effective target that a run-level override would produce.
|
||||
- Enumeration of all profiles is not required for the Notarius use case; exact
|
||||
lookup by ID is sufficient.
|
||||
|
||||
### Value to Notarius
|
||||
|
||||
This would eliminate a synthetic production-only prompt fixture and establish
|
||||
a direct, supported contract for configuration-time profile and backend
|
||||
validation.
|
||||
|
||||
## Priority 3: Semantic Execution-Target Fingerprints
|
||||
|
||||
**Disposition:** Deferred until prompt-independent profile inspection defines
|
||||
the resolved target whose configuration identity would be fingerprinted.
|
||||
|
||||
### Downstream need
|
||||
|
||||
Notarius checkpoints model-backed pipeline stages. A checkpoint must not be
|
||||
reused when generation-affecting PromptKit configuration changes.
|
||||
|
||||
Notarius therefore needs a stable equality signal for the effective profile
|
||||
and backend target used by a pipeline.
|
||||
|
||||
### Current integration
|
||||
|
||||
Notarius currently constructs this identity itself from:
|
||||
|
||||
- a manually maintained marker for the PromptKit release and built-in profile
|
||||
catalog;
|
||||
- raw hashes of configured profile files; and
|
||||
- a separate hash of the configured conventional local-backend endpoint.
|
||||
|
||||
This is safe but conservative and coupled to PromptKit details. Raw file
|
||||
hashing also invalidates checkpoints for semantically irrelevant YAML changes,
|
||||
such as comments or formatting.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Expose an opaque semantic digest for a resolved profile and its effective
|
||||
generation target. It could be returned by the proposed profile-resolution
|
||||
API:
|
||||
|
||||
```go
|
||||
type ResolvedProfile struct {
|
||||
ProfileID string
|
||||
BackendID string
|
||||
EffectiveTarget ExecutionTarget
|
||||
ExecutionDigest string
|
||||
}
|
||||
```
|
||||
|
||||
Alternatively, PromptKit could expose a dedicated method such as
|
||||
`ProfileExecutionDigest(profileID)`.
|
||||
|
||||
### Desired equality semantics
|
||||
|
||||
The digest should change when generation-affecting state changes, including:
|
||||
|
||||
- resolved model and endpoint;
|
||||
- backend routing identity;
|
||||
- backend request defaults and extra parameters;
|
||||
- profile generation parameters; and
|
||||
- the semantic identity of any selected built-in profile.
|
||||
|
||||
The digest should not incorporate:
|
||||
|
||||
- credential values;
|
||||
- concurrency or queue capacity;
|
||||
- filesystem source paths;
|
||||
- YAML comments or formatting; or
|
||||
- other settings that affect scheduling or source representation without
|
||||
changing the generation target.
|
||||
|
||||
The credential environment-variable name may need to participate if changing
|
||||
it can select a materially different provider account or target. PromptKit
|
||||
should define this deliberately while continuing to exclude the resolved
|
||||
secret value.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Treat the digest as an opaque equality value rather than a public encoding
|
||||
of internal structures.
|
||||
- Document which categories of change affect equality.
|
||||
- Include a versioned semantic marker internally so PromptKit can deliberately
|
||||
invalidate old digests when its resolution semantics change.
|
||||
- Prefer a per-profile digest over a digest of every profile known to an
|
||||
engine. Notarius generally knows which profiles a resolved pipeline uses.
|
||||
- Do not require consumers to know PromptKit's built-in catalog version.
|
||||
|
||||
### Value to Notarius
|
||||
|
||||
This would let Notarius remove its PromptKit release marker and raw
|
||||
profile-source fingerprinting, reduce unnecessary checkpoint invalidation, and
|
||||
delegate execution-target equality to the component that owns target
|
||||
resolution.
|
||||
|
||||
## Priority 4: Structured Capacity Errors
|
||||
|
||||
**Disposition:** Accepted into the
|
||||
[future catalog](future.md#structured-capacity-errors).
|
||||
|
||||
### Downstream need
|
||||
|
||||
Notarius translates PromptKit backend-capacity rejection into a
|
||||
provider-neutral application error. When multiple backends are active,
|
||||
operators would benefit from knowing which backend rejected admission without
|
||||
parsing an error string or exposing endpoint details.
|
||||
|
||||
### Current integration
|
||||
|
||||
PromptKit provides the useful `ErrCapacityExceeded` sentinel. Notarius can
|
||||
classify the failure reliably, but it retains only a sanitized diagnostic
|
||||
string as additional context.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Add a typed error that continues to match `ErrCapacityExceeded`:
|
||||
|
||||
```go
|
||||
type CapacityError struct {
|
||||
BackendID string
|
||||
}
|
||||
|
||||
func (e *CapacityError) Is(target error) bool {
|
||||
return target == ErrCapacityExceeded
|
||||
}
|
||||
```
|
||||
|
||||
The exact implementation may use `Unwrap` or another idiomatic mechanism. The
|
||||
important properties are compatibility with `errors.Is` and discoverability
|
||||
through `errors.As`.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Include the stable backend ID.
|
||||
- Do not expose the backend endpoint, credential environment, credential
|
||||
value, request content, or other sensitive configuration.
|
||||
- Consider including the configured concurrency and queue limits if they are
|
||||
useful and safe, but backend identity alone provides most of the downstream
|
||||
value.
|
||||
- Add a retry delay only if PromptKit can provide a meaningful value. A full
|
||||
queue does not necessarily imply a reliable `Retry-After` duration.
|
||||
- Keep retry and backoff policy with the consuming application. PromptKit
|
||||
should classify the admission failure rather than silently retry it.
|
||||
|
||||
### Value to Notarius
|
||||
|
||||
This would improve operational diagnostics and future metrics while preserving
|
||||
the provider-neutral error boundary used by Notarius.
|
||||
|
||||
## Capabilities PromptKit Already Provides Well
|
||||
|
||||
The current PromptKit boundary is sufficient for Notarius's implemented
|
||||
behavior. In particular, PromptKit already provides:
|
||||
|
||||
- filesystem, `fs.FS`, and programmatic prompt, profile, and schema sources;
|
||||
- offline preparation without model execution;
|
||||
- structured output and content validation;
|
||||
- direct session propagation;
|
||||
- tri-state per-run reasoning overrides;
|
||||
- selected profile, backend, model, endpoint, effective parameters, hashes, and
|
||||
token-usage provenance;
|
||||
- endpoint-only profiles;
|
||||
- the conventional `local` backend helper;
|
||||
- arbitrary engine-scoped `Backend` registrations;
|
||||
- backend authentication environment names, extra parameters, concurrency
|
||||
limits, and queue-capacity policies;
|
||||
- provider-client and artifact-reader extension interfaces;
|
||||
- context cancellation; and
|
||||
- useful public error sentinels, including profile absence and capacity
|
||||
exhaustion.
|
||||
|
||||
The wishlist does not imply that Notarius needs PromptKit to broaden its core
|
||||
responsibilities. It primarily asks for more direct access to information and
|
||||
operations that PromptKit already computes internally.
|
||||
|
||||
## Responsibilities That Should Remain In Notarius
|
||||
|
||||
The following concerns belong to the downstream application and should not
|
||||
move into PromptKit for the sake of Notarius:
|
||||
|
||||
- pipeline staging, dependencies, and generated references;
|
||||
- application-wide scheduling across providers and backends;
|
||||
- module and validation retry policy;
|
||||
- checkpoints, resume, and recomputation;
|
||||
- durable run artifacts and manifests;
|
||||
- D&D prompts, schemas, extractors, validators, and normalizers;
|
||||
- Notarius configuration-file parsing and precedence;
|
||||
- domain-specific prompt-cache prefix policy; and
|
||||
- application-specific redaction, retention, and debug-bundle policy.
|
||||
|
||||
PromptKit's complete `Backend` API already supports custom IDs, multiple local
|
||||
endpoints, authentication, extra parameters, and explicit queue policies.
|
||||
Whether Notarius exposes those capabilities in its own configuration is an
|
||||
application-policy decision, not an upstream PromptKit gap.
|
||||
|
||||
## Suggested Upstream Sequence
|
||||
|
||||
If the PromptKit team chooses to pursue these ideas, the most useful order for
|
||||
Notarius would be:
|
||||
|
||||
1. Add atomic execution that returns prepared details and the completed result.
|
||||
2. Add prompt-independent profile inspection.
|
||||
3. Add a semantic execution-target digest, preferably as part of profile
|
||||
inspection.
|
||||
4. Add a typed capacity error carrying backend identity.
|
||||
|
||||
The first two address concrete workarounds in current Notarius code. The third
|
||||
would improve checkpoint correctness and reduce coupling. The fourth is
|
||||
operational polish.
|
||||
280
docs/roadmap/prepared-execution.md
Normal file
280
docs/roadmap/prepared-execution.md
Normal file
@@ -0,0 +1,280 @@
|
||||
# Executable Preparation Handles
|
||||
|
||||
**Status:** Accepted.
|
||||
|
||||
## Purpose
|
||||
|
||||
Allow a consumer to prepare one exact Promptkit execution, inspect and retain
|
||||
its credential-redacted public preparation details, and later execute that
|
||||
already-prepared work without resolving or rendering the request again.
|
||||
|
||||
This provides a supported preflight-before-generation boundary for
|
||||
[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-1-executable-preparation-handles)
|
||||
and removes the duplicate `Prepare`-then-`Run` workaround described by
|
||||
[Notarius](notarius-promptkit-wishlist.md#priority-1-atomic-execution-with-prepared-details).
|
||||
|
||||
## Motivation
|
||||
|
||||
`Engine.Prepare` currently returns the provenance and rendered details that
|
||||
consumers need for debugging, persistence, and preflight checks. `Engine.Run`
|
||||
then performs its own preparation before generation. A consumer that needs
|
||||
both values must therefore prepare the same logical request twice.
|
||||
|
||||
That workaround duplicates source loading, input hashing, schema work, and
|
||||
template rendering. It also permits filesystem-backed prompts, profiles,
|
||||
schemas, or inputs to change between the public preparation and the
|
||||
preparation that actually produces the result.
|
||||
|
||||
Promptkit already owns the complete preparation and execution pipeline. An
|
||||
opt-in prepared-execution handle should expose the missing boundary without
|
||||
putting rendered content into every `RunResult` or moving persistence policy
|
||||
into the library.
|
||||
|
||||
## Consumer Workflow
|
||||
|
||||
The target public workflow is:
|
||||
|
||||
```go
|
||||
prepared, err := engine.PrepareExecution(ctx, request)
|
||||
if err != nil {
|
||||
// Handle preparation failure.
|
||||
return
|
||||
}
|
||||
defer prepared.Discard()
|
||||
|
||||
details := prepared.Details()
|
||||
// Persist or inspect a consumer-selected safe subset of details.
|
||||
|
||||
result, err := engine.RunPrepared(ctx, prepared)
|
||||
```
|
||||
|
||||
The target public surface is:
|
||||
|
||||
```go
|
||||
type PreparedExecution struct {
|
||||
// Opaque Promptkit-owned state.
|
||||
}
|
||||
|
||||
func (e *Engine) PrepareExecution(
|
||||
ctx context.Context,
|
||||
req RunRequest,
|
||||
) (*PreparedExecution, error)
|
||||
|
||||
func (p *PreparedExecution) Details() PreparedRun
|
||||
|
||||
func (p *PreparedExecution) Discard()
|
||||
|
||||
func (e *Engine) RunPrepared(
|
||||
ctx context.Context,
|
||||
prepared *PreparedExecution,
|
||||
) (*RunResult, error)
|
||||
```
|
||||
|
||||
The declarations and GoDoc will own the exact implemented contract. The
|
||||
important public shape is an opaque handle, caller-owned `PreparedRun`
|
||||
details, an explicit discard operation, and execution through the engine that
|
||||
created the handle.
|
||||
|
||||
## Prepared Snapshot
|
||||
|
||||
`PrepareExecution` performs complete preparation without model generation or
|
||||
backend-capacity admission. It applies the same request validation, source
|
||||
precedence, backend and profile resolution, credential requirement checks,
|
||||
output-contract resolution, artifact loading, hashing, schema loading,
|
||||
session resolution, and rendering behavior as `Prepare`.
|
||||
|
||||
A successful handle freezes all source-derived state required for later
|
||||
execution, including:
|
||||
|
||||
- the selected prompt definition, profile, and backend identity;
|
||||
- the complete effective execution target and request-field presence;
|
||||
- rendered messages and effective session ID;
|
||||
- prompt, rendered-prompt, and input hashes;
|
||||
- the effective output contract and provider-facing structured-output
|
||||
constraint; and
|
||||
- private validation state sufficient to validate generated output without
|
||||
reopening schema files or `fs.FS` resources, including required schema
|
||||
references.
|
||||
|
||||
After `PrepareExecution` succeeds, changes to prompt, profile, schema, input,
|
||||
or request-owned data cannot change what `RunPrepared` sends to the model or
|
||||
how it validates the generated output.
|
||||
|
||||
The handle retains an internal snapshot independent from values returned by
|
||||
`Details`. Mutating a returned `PreparedRun`, its maps, slices, messages, or
|
||||
schema values does not affect later execution. Each `Details` call returns a
|
||||
fresh caller-owned copy under the existing `PreparedRun` ownership and stable
|
||||
JSON rules.
|
||||
|
||||
## Handle Lifecycle
|
||||
|
||||
A `PreparedExecution` is:
|
||||
|
||||
- created only by a successful `PrepareExecution` call;
|
||||
- bound to the exact `Engine` that created it;
|
||||
- valid for one `RunPrepared` invocation;
|
||||
- safe for repeated `Details` calls;
|
||||
- intentionally opaque and without a supported JSON representation; and
|
||||
- in-process state rather than a durable or restartable job.
|
||||
|
||||
`RunPrepared` atomically claims a valid handle before beginning the execution
|
||||
attempt. A second or concurrent invocation fails without starting another
|
||||
execution, including when the first invocation ended in cancellation, capacity
|
||||
rejection, generation failure, or operational validation failure. Copies of
|
||||
the public handle share the same one-attempt state and cannot bypass this rule.
|
||||
|
||||
A nil, zero-value, foreign-engine, discarded, already-claimed, or already-used
|
||||
handle is invalid. `RunPrepared` reports these lifecycle errors through the
|
||||
ordinary public invalid-request category. A failed foreign-engine invocation
|
||||
does not consume a handle that remains valid for its owning engine.
|
||||
|
||||
`Discard` idempotently makes an unclaimed handle unavailable for execution and
|
||||
drops Promptkit's references to secret-bearing or execution-only state.
|
||||
`RunPrepared` performs the same cleanup automatically after claiming a handle.
|
||||
Credential-redacted public preparation details remain available after discard,
|
||||
success, or failure so consumers can retain diagnostics. Promptkit does not
|
||||
promise secure erasure of Go string memory.
|
||||
|
||||
Lifecycle transitions are concurrency-safe. When `RunPrepared` and `Discard`
|
||||
race, exactly one claims the ready handle. `Discard` is not an execution
|
||||
cancellation mechanism and does not interrupt an attempt that has already
|
||||
claimed the handle; consumers cancel that attempt through its context.
|
||||
|
||||
## Credentials And Sensitive Data
|
||||
|
||||
`Details` has the same security contract as `PreparedRun`: it can contain
|
||||
rendered messages, schemas, identifiers, and hashes, but never a resolved API
|
||||
key value. Consumers remain responsible for selecting, redacting, storing, and
|
||||
retaining any persisted preparation material.
|
||||
|
||||
A direct `RunRequest.APIKey` is retained only in opaque execution state until
|
||||
the handle is run or discarded. It is never added to details, hashes, JSON,
|
||||
`String`, or `GoString` output.
|
||||
|
||||
An environment-variable name is frozen as part of the effective target, but
|
||||
its credential value is not captured for the lifetime of the handle.
|
||||
`PrepareExecution` applies the existing preparation-time availability check.
|
||||
`RunPrepared` rechecks availability before admission, and the selected model
|
||||
client uses the environment value visible during execution. This preserves
|
||||
current secret ownership and avoids retaining an environment credential while
|
||||
a consumer persists preflight material.
|
||||
|
||||
The opaque handle must not expose retained request data or credentials through
|
||||
default formatting, JSON, or error messages.
|
||||
|
||||
## Admission, Cancellation, And Execution
|
||||
|
||||
`PrepareExecution` never reserves backend admission or an active-generation
|
||||
permit. Its context governs preparation only; cancellation after it returns
|
||||
does not invalidate the handle.
|
||||
|
||||
`RunPrepared` uses its own context for credential revalidation, backend
|
||||
admission, active-generation waiting, model generation, output validation, and
|
||||
any internal repair. Admission occurs when `RunPrepared` begins so a consumer
|
||||
cannot occupy bounded capacity while inspecting or persisting preparation
|
||||
details.
|
||||
|
||||
For a limited backend, the admission lease covers the complete prepared
|
||||
execution attempt after admission: generation, validation, internal repair,
|
||||
and every success or failure exit. Actual generation continues to use the
|
||||
backend's FIFO active-generation permit. Existing capacity error identity,
|
||||
cancellation behavior, and release guarantees remain in force.
|
||||
|
||||
Because a prepared handle is one-attempt, cancellation or capacity rejection
|
||||
does not make it reusable. Retry and backoff policy remains with the consumer,
|
||||
which may create a new prepared handle when another attempt is appropriate.
|
||||
|
||||
## Results And Failures
|
||||
|
||||
On success, `RunPrepared` returns the existing caller-owned `RunResult`. Its
|
||||
source-derived provenance must match `Details`, including prompt identity and
|
||||
hash, rendered-prompt hash, session ID, selected profile and backend,
|
||||
effective target, and input hashes.
|
||||
|
||||
`RunResult.StartTime`, `EndTime`, and `Duration` describe the
|
||||
`RunPrepared` execution attempt. They exclude preparation time and any delay
|
||||
while the consumer retained the handle. Preparation timing remains in
|
||||
`PreparedRun`.
|
||||
|
||||
Preparation failure returns no handle. After successful preparation,
|
||||
`RunPrepared` retains the existing rule that an operational failure returns no
|
||||
partial `RunResult`; the consumer already has independent preparation details.
|
||||
A completed content-validation failure remains a successful result with
|
||||
`ValidationFailed`.
|
||||
|
||||
`PrepareExecution` preserves the public error categories of `Prepare`.
|
||||
`RunPrepared` preserves applicable invalid-request, credential, capacity,
|
||||
generation, validation, collaborator, and cancellation identities without
|
||||
reintroducing source-loading or rendering failures from frozen state.
|
||||
|
||||
## Compatibility And Existing Workflows
|
||||
|
||||
This feature is additive:
|
||||
|
||||
- `Prepare` remains the simple preparation-only operation;
|
||||
- `Run` remains the simple prepare-and-execute operation with its current
|
||||
early-admission and error-ordering behavior;
|
||||
- `PreparedRun` and `RunResult` retain their existing stable JSON
|
||||
representations;
|
||||
- model-client and artifact-reader extension interfaces remain unchanged; and
|
||||
- backend routing, concurrency limits, queue capacities, and provider wire
|
||||
behavior remain unchanged.
|
||||
|
||||
The new workflow may share internal machinery with `Prepare` and `Run`, but it
|
||||
must not change their observable behavior merely to simplify implementation.
|
||||
|
||||
## Documentation
|
||||
|
||||
The completed documentation set has these ownership boundaries:
|
||||
|
||||
- exported declarations and GoDoc own the exact handle, method, lifecycle,
|
||||
ownership, concurrency, credential, error, and cancellation contracts;
|
||||
- the promptkit consumer guide explains when to use `Prepare`, `Run`, or the
|
||||
two-phase prepared-execution workflow; and
|
||||
- internal runner, source-validation, capacity, and model-client documentation
|
||||
describe the implemented collaborator boundaries without duplicating public
|
||||
contracts.
|
||||
|
||||
No release document is part of the feature implementation itself. Release
|
||||
guidance is prepared only when the resulting public API is selected for
|
||||
publication.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This work does not include:
|
||||
|
||||
- serializable, durable, resumable, or cross-process execution handles;
|
||||
- reuse for multiple consumer-initiated executions;
|
||||
- concurrent execution of one handle;
|
||||
- capacity reservation during preparation;
|
||||
- a background task queue, priorities, worker lifecycle, or job status;
|
||||
- retry, backoff, or provider failover policy;
|
||||
- a separate `RunDetailed` convenience method;
|
||||
- adding preparation details or rendered messages to every `RunResult`;
|
||||
- prompt or profile inspection APIs;
|
||||
- structured capacity or generation errors;
|
||||
- freezing environment-variable credential values for the handle lifetime;
|
||||
- snapshotting provider state or mutable behavior inside an injected
|
||||
`LLMClient`;
|
||||
- allowing target, validation, session, variable, or input overrides after
|
||||
preparation; or
|
||||
- changing existing `Prepare`, `Run`, file-format, provider-request, or stable
|
||||
JSON contracts.
|
||||
|
||||
## Target End State
|
||||
|
||||
After this work:
|
||||
|
||||
- consumers can perform and persist preflight before starting provider work;
|
||||
- one prepared handle executes exactly the source-derived prompt, target,
|
||||
schema, inputs, session, and messages described by its public details;
|
||||
- execution never reloads or rerenders consumer sources;
|
||||
- direct credentials remain confined to opaque, explicitly discardable state;
|
||||
- environment credentials are not retained across the preflight boundary;
|
||||
- backend capacity is reserved only when execution begins;
|
||||
- one handle can start at most one execution attempt, including any internal
|
||||
repair calls owned by that attempt;
|
||||
- preparation details remain available after execution success or failure;
|
||||
- existing simple `Prepare` and `Run` consumers remain unaffected; and
|
||||
- Promptkit continues to own reusable execution mechanics without taking on
|
||||
downstream persistence, redaction, retry, or job-management policy.
|
||||
468
docs/roadmap/weatherreporter-promptkit-wishlist.md
Normal file
468
docs/roadmap/weatherreporter-promptkit-wishlist.md
Normal file
@@ -0,0 +1,468 @@
|
||||
# Weatherreporter PromptKit Wishlist
|
||||
|
||||
## Purpose
|
||||
|
||||
This document records features and interface changes that would be useful
|
||||
additions to PromptKit from the perspective of the maintainers of
|
||||
Weatherreporter, a downstream application planning to replace its Scriptorium
|
||||
CLI integration with PromptKit.
|
||||
|
||||
PromptKit v0.3.0 provides the capabilities Weatherreporter needs for the
|
||||
migration. None of the ideas below is a hard adoption requirement. They are
|
||||
opportunities to avoid duplicate preparation, validate configuration earlier,
|
||||
improve durable failure diagnostics, and make the integration more direct.
|
||||
|
||||
The examples are API sketches intended to communicate the desired capability,
|
||||
not prescriptive names or finalized Go contracts. The related
|
||||
[Notarius PromptKit wishlist](notarius-promptkit-wishlist.md) proposes several
|
||||
overlapping features from another downstream consumer's perspective.
|
||||
|
||||
## Priority 1: Executable Preparation Handles
|
||||
|
||||
**Disposition:** Accepted into the
|
||||
[executable preparation handles](prepared-execution.md) feature roadmap.
|
||||
|
||||
### Downstream need
|
||||
|
||||
Weatherreporter treats prompt preparation as a durable preflight boundary. It
|
||||
needs to:
|
||||
|
||||
1. prepare the exact request that will be executed;
|
||||
2. persist a safe preparation record before starting the provider call; and
|
||||
3. execute without reloading or rerendering prompt, profile, schema, or input
|
||||
sources.
|
||||
|
||||
Persisting preflight before generation leaves useful evidence when a provider
|
||||
call fails or the process is interrupted during generation.
|
||||
|
||||
### Current integration option
|
||||
|
||||
With PromptKit v0.3.0, Weatherreporter can call `Engine.Prepare`, save selected
|
||||
fields from the returned `PreparedRun`, and then call `Engine.Run` with the
|
||||
same request. Because `Run` performs preparation internally, the work is
|
||||
repeated.
|
||||
|
||||
Weatherreporter plans to use embedded prompt and schema files plus immutable
|
||||
inline input bytes, which removes most of the consistency risk. An external
|
||||
profile file or directory can still change between the two calls, and the
|
||||
second preparation remains unnecessary work.
|
||||
|
||||
The atomic `RunDetailed` operation proposed by the
|
||||
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-1-atomic-execution-with-prepared-details)
|
||||
would guarantee that returned preparation details describe the completed
|
||||
execution. However, returning those details only after generation would not
|
||||
preserve Weatherreporter's preflight-before-generation persistence boundary.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Add an opt-in two-phase API that returns a prepared execution handle:
|
||||
|
||||
```go
|
||||
prepared, err := engine.PrepareExecution(ctx, request)
|
||||
if err != nil {
|
||||
// Handle preparation failure.
|
||||
}
|
||||
|
||||
details := prepared.Details()
|
||||
// Persist a consumer-selected safe preparation record.
|
||||
|
||||
result, err := engine.RunPrepared(ctx, prepared)
|
||||
```
|
||||
|
||||
The exact names and shapes are flexible. The important contract is that
|
||||
`RunPrepared` executes the already prepared prompt and does not reload or
|
||||
rerender its prompt, profile, schema, or input sources.
|
||||
|
||||
`Details` should return the same caller-owned public preparation information
|
||||
currently represented by `PreparedRun`. The execution handle may retain opaque
|
||||
engine-owned state needed to invoke the model and validate the response.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Keep `Prepare` and `Run` available for consumers that do not need a
|
||||
two-phase execution boundary.
|
||||
- Bind a prepared handle to the engine that constructed it.
|
||||
- Define whether a handle is one-shot, reusable, or safe for concurrent use.
|
||||
A one-shot contract may be the safest initial design.
|
||||
- Do not give the opaque handle a stable JSON representation.
|
||||
- Do not expose or serialize resolved credential values through `Details`.
|
||||
- Define how a direct request API key is retained and released when an opaque
|
||||
handle must carry it until execution.
|
||||
- Preserve caller-owned copies for all public details.
|
||||
- Make context cancellation and backend admission timing explicit.
|
||||
- Document whether profile credential environment values are resolved during
|
||||
preparation or execution.
|
||||
- Ensure an execution error does not invalidate the public details already
|
||||
returned to the consumer.
|
||||
- Consider whether an atomic `RunDetailed` can share the same internal
|
||||
prepared-execution implementation.
|
||||
|
||||
### Value to Weatherreporter
|
||||
|
||||
This is the highest-value upstream addition. It would preserve
|
||||
Weatherreporter's durable preflight behavior, remove duplicate work, eliminate
|
||||
the remaining source-consistency window, and ensure that persisted provenance
|
||||
describes the actual execution.
|
||||
|
||||
## Priority 2: Prompt-Definition Inspection
|
||||
|
||||
**Disposition:** Accepted into the
|
||||
[future catalog](future.md#prompt-definition-inspection).
|
||||
|
||||
### Downstream need
|
||||
|
||||
Weatherreporter has a fixed registry of seven report definitions. Each report
|
||||
selects a prompt ID and one of two output workflows:
|
||||
|
||||
- direct Markdown; or
|
||||
- structured generated text followed by application-owned domain validation
|
||||
and Markdown template rendering.
|
||||
|
||||
Weatherreporter will embed the PromptKit prompt definitions and private
|
||||
response schemas that implement those reports. It needs to validate that the
|
||||
report registry and embedded prompt corpus agree before weather collection or
|
||||
provider execution.
|
||||
|
||||
### Current integration option
|
||||
|
||||
Weatherreporter can maintain synthetic data-package fixtures and call
|
||||
`Engine.Prepare` for every report prompt during tests. Runtime validation can
|
||||
also occur through the ordinary per-report preparation stage.
|
||||
|
||||
This works, but it requires complete placeholder inputs and profile resolution
|
||||
when the application primarily wants to inspect prompt identity and declared
|
||||
contracts.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Add exact prompt-definition lookup without rendering or generation:
|
||||
|
||||
```go
|
||||
type PromptInfo struct {
|
||||
PromptID string
|
||||
PromptVersion string
|
||||
PromptHash string
|
||||
DefaultProfileID string
|
||||
Inputs []InputDefinition
|
||||
OutputContract OutputContract
|
||||
}
|
||||
|
||||
func (e *Engine) ResolvePrompt(
|
||||
ctx context.Context,
|
||||
promptID string,
|
||||
promptVersion string,
|
||||
) (PromptInfo, error)
|
||||
```
|
||||
|
||||
The exact returned shape may differ. Weatherreporter needs enough information
|
||||
to verify prompt existence, version selection, declared inputs, default
|
||||
profile identity, output format, validation mode, and schema selection without
|
||||
supplying synthetic prompt input.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Use ordinary PromptKit prompt-source precedence and exact ID/version
|
||||
selection.
|
||||
- Fully load and structurally validate the selected prompt definition.
|
||||
- Validate referenced prompt content files without rendering their templates.
|
||||
- Resolve and validate the selected output contract and schema reference where
|
||||
practical.
|
||||
- Return an opaque prompt-definition equality value rather than raw source
|
||||
bytes.
|
||||
- Do not return rendered messages, schema bodies, profile credentials, or
|
||||
another source of sensitive content.
|
||||
- Preserve typed or sentinel errors for missing and invalid prompts.
|
||||
- Return caller-owned values.
|
||||
- Enumeration of all known prompts is not required for Weatherreporter; exact
|
||||
lookup is sufficient.
|
||||
|
||||
### Value to Weatherreporter
|
||||
|
||||
This would let Weatherreporter directly verify that every report prompt
|
||||
exists, requires the curated `data_package` input, and declares the expected
|
||||
Markdown or JSON Schema output contract. It would reduce synthetic test setup
|
||||
and move failures ahead of weather collection.
|
||||
|
||||
## Priority 3: Prompt-Independent Profile Inspection
|
||||
|
||||
**Disposition:** Accepted into the
|
||||
[future catalog](future.md#prompt-independent-profile-inspection).
|
||||
|
||||
### Downstream need
|
||||
|
||||
Weatherreporter will allow operators to select an external PromptKit profile
|
||||
source and may allow an explicit profile override. It should reject a missing
|
||||
profile, unknown backend, malformed execution target, or unsatisfied credential
|
||||
requirement before collecting weather data or writing report artifacts.
|
||||
|
||||
### Current integration option
|
||||
|
||||
Weatherreporter can validate an explicit profile by preparing one embedded
|
||||
prompt with fixture input. Prompts that use their own default profiles can be
|
||||
validated during their normal preparation stage.
|
||||
|
||||
This couples configuration validation to one prompt and requires placeholder
|
||||
input even when only profile and backend resolution are relevant.
|
||||
|
||||
### Requested capability
|
||||
|
||||
The prompt-independent `ResolveProfile` API proposed by the
|
||||
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
|
||||
would satisfy this need. It should resolve built-in, file-backed, and
|
||||
programmatic profiles, validate backend membership, report credential
|
||||
requirements without resolving credential values, and preserve typed error
|
||||
classification.
|
||||
|
||||
### Additional Weatherreporter considerations
|
||||
|
||||
- An explicit application profile override should be inspectable without
|
||||
selecting a report prompt.
|
||||
- A prompt-definition inspection result may expose its default profile ID so
|
||||
Weatherreporter can inspect that profile separately.
|
||||
- Inspection should distinguish structural profile validity from current
|
||||
credential availability so configuration validation can apply explicit
|
||||
application policy.
|
||||
- An optional execution-target override should be considered only if it
|
||||
describes the same target that a later run will use.
|
||||
|
||||
### Value to Weatherreporter
|
||||
|
||||
This would improve fail-fast configuration validation and give operator-facing
|
||||
errors direct profile and backend context. It is valuable but not required for
|
||||
the initial migration.
|
||||
|
||||
## Priority 4: Eager Source Validation
|
||||
|
||||
**Disposition:** Deferred until prompt and profile inspection have been used
|
||||
to determine whether a broader engine-wide validation operation is still
|
||||
needed.
|
||||
|
||||
### Downstream need
|
||||
|
||||
PromptKit deliberately defers reading and validating filesystem and `fs.FS`
|
||||
prompt, profile, and schema content until a request needs it. Weatherreporter
|
||||
has a small fixed embedded prompt corpus and one optional external profile
|
||||
source. It would benefit from an explicit offline validation operation for
|
||||
tests, startup diagnostics, and configuration checks.
|
||||
|
||||
### Current integration option
|
||||
|
||||
Weatherreporter can prepare every report prompt with fixture inputs and inspect
|
||||
any explicit profiles individually. That provides strong coverage but requires
|
||||
consumer-maintained traversal and synthetic material.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Consider an opt-in source-validation operation:
|
||||
|
||||
```go
|
||||
type SourceValidationOptions struct {
|
||||
RequireCredentials bool
|
||||
}
|
||||
|
||||
func (e *Engine) ValidateSources(
|
||||
ctx context.Context,
|
||||
opts SourceValidationOptions,
|
||||
) error
|
||||
```
|
||||
|
||||
The operation should eagerly discover and structurally validate the configured
|
||||
prompt, profile, and schema sources without model generation.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Keep deferred validation as the normal `NewEngine` behavior.
|
||||
- Make eager validation an explicit consumer choice.
|
||||
- Validate duplicate IDs and versions, strict YAML decoding, referenced content
|
||||
files, profile/backend membership, schema syntax, and schema references.
|
||||
- Distinguish structural credential declarations from current environment
|
||||
availability.
|
||||
- Do not read or expose credential values when credential availability is not
|
||||
requested.
|
||||
- Preserve source-specific public error identities and useful path context.
|
||||
- Respect context cancellation during filesystem discovery and schema work.
|
||||
- Consider whether exact prompt and profile inspection APIs already provide a
|
||||
smaller sufficient surface before adding an engine-wide operation.
|
||||
|
||||
### Value to Weatherreporter
|
||||
|
||||
This would simplify offline corpus checks and catch malformed operator profile
|
||||
sources before report work begins. It is helpful but lower priority than exact
|
||||
prompt and profile inspection.
|
||||
|
||||
## Priority 5: Structured Generation Errors
|
||||
|
||||
**Disposition:** Deferred pending stronger downstream demand and a narrower
|
||||
design that does not duplicate prepared provenance or impose HTTP-specific
|
||||
fields on injected model clients.
|
||||
|
||||
### Downstream need
|
||||
|
||||
Weatherreporter preserves redacted, inspectable failure receipts for report
|
||||
runs. When model generation fails operationally, it needs to classify the
|
||||
failure and retain safe execution context without parsing error prose.
|
||||
|
||||
Prompt preparation already supplies selected profile, backend, and model
|
||||
identity. Provider status classification would add useful operator context,
|
||||
especially when the built-in OpenAI-compatible client receives a non-success
|
||||
HTTP status.
|
||||
|
||||
### Current integration option
|
||||
|
||||
PromptKit exposes `ErrLLMGenerate` and preserves injected client errors through
|
||||
`errors.Is`. Weatherreporter can reliably classify generation failure and use
|
||||
its preparation record for profile, backend, and model provenance. Any further
|
||||
diagnostic detail remains a redacted error string.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Consider a typed generation error that continues to match `ErrLLMGenerate`:
|
||||
|
||||
```go
|
||||
type GenerationError struct {
|
||||
BackendID string
|
||||
Model string
|
||||
StatusCode int
|
||||
}
|
||||
```
|
||||
|
||||
The exact fields may differ. The useful contract is safe structured context
|
||||
available through `errors.As`, while `errors.Is(err, ErrLLMGenerate)` remains
|
||||
compatible.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Include only fields that PromptKit knows reliably and can expose safely.
|
||||
- Treat an HTTP status as optional because injected model clients may not use
|
||||
HTTP.
|
||||
- Do not expose provider response bodies, endpoints, credential environment
|
||||
names, credential values, request content, or generated content.
|
||||
- Do not make a structured error a second source of prompt/profile provenance
|
||||
already present in a prepared execution.
|
||||
- Preserve injected client error identity.
|
||||
- Keep retry and backoff policy with the consuming application.
|
||||
|
||||
### Value to Weatherreporter
|
||||
|
||||
This would improve durable failure receipts and troubleshooting, particularly
|
||||
for built-in transport failures. It is not required if preparation details and
|
||||
the existing sentinel remain available.
|
||||
|
||||
## Lower-Priority Shared Wishlist Items
|
||||
|
||||
### Structured Capacity Errors
|
||||
|
||||
**Disposition:** Accepted into the
|
||||
[future catalog](future.md#structured-capacity-errors).
|
||||
|
||||
The typed capacity error proposed by the
|
||||
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-4-structured-capacity-errors)
|
||||
would improve Weatherreporter diagnostics by exposing the stable backend ID
|
||||
without parsing error text.
|
||||
|
||||
Weatherreporter currently generates batch reports sequentially and constructs
|
||||
one engine per invocation, so engine-local capacity exhaustion is unlikely in
|
||||
the initial design. The feature would become more valuable if report
|
||||
generation later becomes concurrent or PromptKit engines become longer-lived.
|
||||
It should not block adoption.
|
||||
|
||||
### Semantic Execution-Target Fingerprints
|
||||
|
||||
**Disposition:** Deferred until prompt-independent profile inspection defines
|
||||
the resolved target whose configuration identity would be fingerprinted.
|
||||
|
||||
The semantic target digest proposed by the
|
||||
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-3-semantic-execution-target-fingerprints)
|
||||
would provide a compact equality signal for audit metadata.
|
||||
|
||||
Weatherreporter does not currently reuse LLM-dependent checkpoints. Its Recent
|
||||
Changes behavior compares deterministic module snapshots rather than generated
|
||||
reports, so the digest has no immediate cache-correctness role. Existing
|
||||
PromptKit result metadata is sufficient for the initial integration. A digest
|
||||
would still be useful provenance and future-proofing, but it is not a
|
||||
migration priority.
|
||||
|
||||
## Capabilities PromptKit Already Provides Well
|
||||
|
||||
PromptKit v0.3.0 already provides the essential Weatherreporter integration
|
||||
surface:
|
||||
|
||||
- importable in-process engine construction;
|
||||
- filesystem, `fs.FS`, and programmatic prompt, profile, and schema sources;
|
||||
- offline preparation without model execution;
|
||||
- versioned prompt selection;
|
||||
- text, Markdown, JSON, and JSON Schema output contracts;
|
||||
- single-pass output validation with raw output retained after completed
|
||||
validation failure;
|
||||
- inline input artifacts with provenance URIs and input hashes;
|
||||
- selected profile, backend, model, effective target, prompt hashes, timing,
|
||||
and token-usage provenance;
|
||||
- endpoint-only profiles and engine-scoped backend registration;
|
||||
- injected model-client and artifact-reader interfaces;
|
||||
- caller cancellation, generation timeout, and transport timeout behavior; and
|
||||
- public error sentinels for configuration, prompt, profile, artifact,
|
||||
validation, capacity, and generation failures.
|
||||
|
||||
These capabilities are sufficient for Weatherreporter to adopt PromptKit
|
||||
without waiting for new upstream work.
|
||||
|
||||
## Responsibilities That Should Remain In Weatherreporter
|
||||
|
||||
The following concerns belong to Weatherreporter and should not move into
|
||||
PromptKit:
|
||||
|
||||
- report definitions, valid periods, batches, and output naming;
|
||||
- application prompt content and private report response schemas;
|
||||
- deterministic weather facts, modules, and Recent Changes;
|
||||
- curated `data_package` construction and persistence;
|
||||
- generated-text domain validation and Markdown template rendering;
|
||||
- managed artifact paths, atomic writes, metadata, and inspection commands;
|
||||
- preparation, execution, raw-output, and failure-receipt schemas;
|
||||
- CLI configuration loading and precedence;
|
||||
- debug enablement, redaction, placement, sensitivity, and retention;
|
||||
- distributor notification;
|
||||
- batch continuation and any future retry policy; and
|
||||
- application-level compatibility and migration policy.
|
||||
|
||||
## Suggested Upstream Sequence
|
||||
|
||||
If the PromptKit team chooses to pursue these ideas, the most useful order for
|
||||
Weatherreporter would be:
|
||||
|
||||
1. Add executable preparation handles, ideally sharing implementation with an
|
||||
atomic detailed-run API.
|
||||
2. Add prompt-definition inspection.
|
||||
3. Add prompt-independent profile inspection.
|
||||
4. Consider eager source validation after evaluating whether the two exact
|
||||
inspection APIs are sufficient.
|
||||
5. Add structured generation errors.
|
||||
6. Add structured capacity errors and semantic execution-target fingerprints
|
||||
as lower-priority operational improvements.
|
||||
|
||||
The first item removes the only material integration workaround. Prompt and
|
||||
profile inspection improve fail-fast validation. The remaining items improve
|
||||
ergonomics and diagnostics.
|
||||
|
||||
## Adoption Sequencing
|
||||
|
||||
Weatherreporter should not wait for the complete wishlist. PromptKit v0.3.0 is
|
||||
already sufficient when Weatherreporter:
|
||||
|
||||
- embeds immutable prompt and schema assets;
|
||||
- supplies immutable inline data-package bytes;
|
||||
- constructs one engine per CLI invocation;
|
||||
- calls `Prepare` and `Run` with the same request; and
|
||||
- keeps PromptKit behind a weatherreporter-owned adapter contract.
|
||||
|
||||
If executable preparation handles are scheduled for a near-term PromptKit
|
||||
release, Weatherreporter may defer only its final adapter implementation to
|
||||
avoid implementing and then removing duplicate preparation. Prompt corpus
|
||||
retrieval, application-contract design, configuration work, embedded assets,
|
||||
state contracts, and offline fixtures can proceed independently.
|
||||
|
||||
If the feature is not scheduled, Weatherreporter can adopt v0.3.0 and keep the
|
||||
duplicate `Prepare` and `Run` sequence inside its adapter. A later PromptKit
|
||||
upgrade would remain localized behind that neutral boundary.
|
||||
|
||||
Prompt inspection, profile inspection, source validation, structured errors,
|
||||
capacity details, and semantic fingerprints should not gate adoption.
|
||||
Reference in New Issue
Block a user