Compare commits
4 Commits
v0.1.0
...
b462153483
| Author | SHA1 | Date | |
|---|---|---|---|
| b462153483 | |||
| 086cf0fc86 | |||
| c1cecb1ee8 | |||
| bcb327f643 |
@@ -31,4 +31,9 @@ Contributors should start with the [development guide](docs/development.md).
|
||||
The [architecture policy](docs/policy/architecture.md) defines the library
|
||||
boundary and constraints that framework work must preserve.
|
||||
|
||||
## Related Project
|
||||
|
||||
[Scriptorium](https://gitea.maximumdirect.net/eric/scriptorium) is the CLI and
|
||||
HTTP application built on Promptkit.
|
||||
|
||||
Promptkit is licensed under the [GNU General Public License version 3](LICENSE).
|
||||
|
||||
@@ -20,7 +20,6 @@ func toDomainRunRequest(req RunRequest) (domain.RunRequest, error) {
|
||||
Vars: copyStringMap(req.Vars),
|
||||
Execution: execution,
|
||||
Validation: toDomainOutputContractPtr(req.Validation),
|
||||
Metadata: copyStringMap(req.Metadata),
|
||||
}, nil
|
||||
}
|
||||
|
||||
|
||||
54
doc.go
54
doc.go
@@ -1,8 +1,54 @@
|
||||
// Package promptkit provides an embeddable engine for preparing and executing
|
||||
// prompt-defined LLM workflows.
|
||||
//
|
||||
// Applications construct an Engine with NewEngine, select filesystem or
|
||||
// in-memory definition sources with options, and use Prepare or Run to execute
|
||||
// requests. Concrete repositories, validators, and outbound clients remain
|
||||
// internal implementation details.
|
||||
// Applications construct an [Engine] with [NewEngine], select filesystem or
|
||||
// in-memory sources with options, and call [Engine.Prepare] or [Engine.Run].
|
||||
// Concrete repositories, validators, and the built-in OpenAI-compatible client
|
||||
// remain internal implementation details.
|
||||
//
|
||||
// # Concurrency and ownership
|
||||
//
|
||||
// An Engine supports concurrent Prepare and Run calls. An injected [LLMClient]
|
||||
// or [ArtifactReader] can therefore receive concurrent calls and must be safe
|
||||
// for that use.
|
||||
//
|
||||
// NewEngine copies in-memory profiles. Prepare and Run copy request maps,
|
||||
// slices, pointer values, and JSON-compatible extra parameters before using
|
||||
// them. Returned values and values passed to extension interfaces are likewise
|
||||
// isolated from engine state. Callers own those copies and may mutate them
|
||||
// after the call that supplied or returned them.
|
||||
//
|
||||
// # Security and sensitive data
|
||||
//
|
||||
// The default artifact reader treats [File] paths as caller-selected operating
|
||||
// system paths. It does not restrict them to an application root or impose an
|
||||
// inbound request-size policy. Promptkit is not an inbound request or
|
||||
// untrusted-input security boundary. Applications must validate and restrict
|
||||
// untrusted input before constructing a request, or install an [ArtifactReader]
|
||||
// that enforces their filesystem, authorization, and size policies.
|
||||
//
|
||||
// Rendered messages, input and output [Artifact] bodies, [RunResult.RawOutput],
|
||||
// and [ValidationResult.Errors] may contain sensitive data. Credential
|
||||
// exclusion and redaction do not sanitize those values. Applications and
|
||||
// injected collaborators are responsible for access control, retention,
|
||||
// logging, and secret handling appropriate to their data.
|
||||
//
|
||||
// # JSON
|
||||
//
|
||||
// Stable JSON representations are provided for [PreparedRun], [RunResult],
|
||||
// [Artifact], [ExecutionTarget], [OutputContract], [ValidationResult],
|
||||
// [TokenUsage], [RenderedPrompt], [RenderedMessage], [CacheControl],
|
||||
// [StructuredOutputSpec], [StructuredOutputJSONSpec], [GenerateRequest],
|
||||
// [GenerateResponse], [ExecutionTargetPresence], and the string value types
|
||||
// used by those values.
|
||||
//
|
||||
// Construction values, including [Config], [RunRequest], [ArtifactRef],
|
||||
// [ExecutionTargetOverride], [Profile], and
|
||||
// [OpenAICompatibleProfileConfig], do not have stable JSON representations.
|
||||
// Direct API keys are nevertheless excluded from JSON for every public value.
|
||||
//
|
||||
// JSON timestamps use time.Time's RFC 3339 encoding and are omitted when zero.
|
||||
// PreparedRun and RunResult durations are encoded as integer milliseconds in
|
||||
// duration_ms and omitted when zero. Run IDs and all exposed hashes are opaque:
|
||||
// their spelling, length, character set, and algorithm are not API contracts.
|
||||
package promptkit
|
||||
|
||||
@@ -1,151 +1,147 @@
|
||||
# Package `promptkit`
|
||||
|
||||
Import path:
|
||||
## Purpose
|
||||
|
||||
This guide helps Go consumers assemble Promptkit and choose the main
|
||||
preparation or execution workflow. The declarations and GoDoc in the
|
||||
[root package](../../doc.go) own exact field, option, serialization,
|
||||
concurrency, ownership, failure, and cancellation semantics. The
|
||||
[framework format reference](../formats.md) owns prompt, profile, and schema
|
||||
file contracts.
|
||||
|
||||
Import the package as:
|
||||
|
||||
```go
|
||||
import "gitea.maximumdirect.net/eric/promptkit"
|
||||
```
|
||||
|
||||
Package `promptkit` is the supported Go contract for in-process prompt
|
||||
preparation and execution. The declarations and their GoDoc in the
|
||||
[root package](../../doc.go) own the exact API; this guide explains how the
|
||||
pieces are used together. The [framework format reference](../formats.md) owns
|
||||
prompt, profile, and schema file contracts.
|
||||
The following Go fragments are illustrative and omit surrounding package,
|
||||
import, and error-handling code. Use the maintained examples for complete
|
||||
programs.
|
||||
|
||||
## Engine Construction And Sources
|
||||
## Construct An Engine
|
||||
|
||||
Construct an engine with [`NewEngine`, `Config`, and
|
||||
`Option`](../../engine.go). `PromptDir` is required unless a prompt source
|
||||
option is supplied. `ProfileDir` optionally overlays built-in profiles, and an
|
||||
empty `SchemaDir` uses the current directory. `Timeout` is the transport-wide
|
||||
safety cap for the built-in OpenAI-compatible client. An optional `HTTPClient`
|
||||
is cloned; its positive timeout takes precedence.
|
||||
Create an engine with
|
||||
[`NewEngine`](../../engine.go). A directory-backed setup supplies a prompt
|
||||
directory and may supply profile and schema directories:
|
||||
|
||||
Nil options are ignored. Invalid construction, including a nil injected client
|
||||
or artifact reader, returns an error matching `ErrInvalidConfig`.
|
||||
```go
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||
PromptDir: "prompts",
|
||||
ProfileDir: "profiles",
|
||||
SchemaDir: "schemas",
|
||||
})
|
||||
```
|
||||
|
||||
The [source options](../../engine.go) replace their matching directory source:
|
||||
|
||||
- `WithPromptFS` and `WithPromptFile` select prompt definitions;
|
||||
- `WithProfileFS` and `WithProfileFile` overlay built-in profiles;
|
||||
- `WithProfiles` adds in-memory profiles ahead of file and built-in profiles;
|
||||
- `WithSchemaFS` and `WithSchemaFile` select JSON Schema documents;
|
||||
- `WithLLMClient` replaces the built-in model client; and
|
||||
- `WithArtifactReader` replaces the default reader for every input.
|
||||
|
||||
Source selection, path resolution, strict decoding, profile overlays, and
|
||||
file-to-request precedence are defined in the
|
||||
Options support single-file or `fs.FS` sources, in-memory profiles, and
|
||||
injected artifact or model clients. Consult the
|
||||
[constructor and option GoDoc](../../engine.go) for composition, precedence,
|
||||
validation, and default transport behavior. Source discovery, format
|
||||
validation, and profile precedence are defined by the
|
||||
[framework format reference](../formats.md).
|
||||
|
||||
Per-generation timeout values from profiles or requests are independent of
|
||||
the transport cap and caller context. An explicit request value of zero
|
||||
disables only the per-generation deadline. The
|
||||
[outbound integration contract](../integrations/openai-compatible-chat.md#timeout-and-cancellation)
|
||||
defines the complete timeout layering.
|
||||
## Prepare Without Model Execution
|
||||
|
||||
## Preparation And Execution
|
||||
[`Engine.Prepare`](../../engine.go) resolves the selected prompt and profile,
|
||||
loads inputs and any structured-output schema, and renders messages without
|
||||
calling a model client:
|
||||
|
||||
[`Engine.Prepare` and `Engine.Run`](../../engine.go) accept the public
|
||||
[`RunRequest`](../../types.go). `Prepare` resolves the prompt, profile, input
|
||||
artifacts, validation contract, and rendered messages without calling an LLM.
|
||||
`Run` performs the same preparation, calls the configured client, and validates
|
||||
the generated content. The maintained
|
||||
```go
|
||||
prepared, err := engine.Prepare(ctx, promptkit.RunRequest{
|
||||
PromptID: "meeting.summary",
|
||||
Inputs: map[string]promptkit.ArtifactRef{
|
||||
"note": promptkit.Inline("Synthetic meeting notes"),
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
The maintained
|
||||
[offline preparation example](../../examples/go-library/prepare/main.go)
|
||||
provides a complete runnable workflow using a prompt file, in-memory profile,
|
||||
and inline input.
|
||||
shows a complete runnable setup with a prompt file, in-memory profile, and
|
||||
inline input. Exact request requirements and prepared-result fields belong to
|
||||
the [`RunRequest` and `PreparedRun` GoDoc](../../types.go).
|
||||
|
||||
[`PreparedRun` and `RunResult`](../../types.go) expose copied public values.
|
||||
Preparation returns effective settings, hashes, rendered messages, selected
|
||||
profile, structured-output information, and timing without resolved secrets or
|
||||
model output. Execution adds the generated artifact and raw output, validation
|
||||
state, model metadata, usage, run ID, and duration.
|
||||
## Execute And Validate
|
||||
|
||||
A generated-content validation failure returns a result with
|
||||
`Validation.Status == ValidationFailed`. An inability to perform validation
|
||||
returns an error matching `ErrValidation`.
|
||||
[`Engine.Run`](../../engine.go) performs the same preparation, invokes the
|
||||
configured model client, classifies the generated artifact, and validates the
|
||||
content. A completed content check may return `ValidationFailed` in the result;
|
||||
an operational inability to validate returns an error.
|
||||
|
||||
## Requests, Inputs, And Overrides
|
||||
The maintained
|
||||
[offline execution example](../../examples/go-library/run/main.go) injects a
|
||||
deterministic model client and exercises `Run` without credentials, network
|
||||
access, or paid calls. It is intentionally separate from the preparation
|
||||
example so each workflow and its small prompt fixture can be copied and run on
|
||||
its own.
|
||||
|
||||
The [request and value declarations](../../types.go) own the available fields,
|
||||
serialized constants, and result shapes. Use `File`, `Inline`, or
|
||||
`InlineWithURI` to construct artifact references. The
|
||||
[framework format reference](../formats.md) defines declared inputs, template
|
||||
references, output contracts, and the relationship between file values and
|
||||
request overrides.
|
||||
Use the [`RunResult` and `ValidationResult` GoDoc](../../types.go) for the
|
||||
returned data and the `Engine.Run` GoDoc for failure and cancellation
|
||||
semantics. The
|
||||
[OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md)
|
||||
owns the built-in client's outbound HTTP behavior.
|
||||
|
||||
`ExecutionTargetOverride` uses pointers for numeric settings so an explicit
|
||||
zero remains distinct from no override. `ExtraParams` accepts JSON-compatible
|
||||
strings, booleans, finite numbers, string-keyed objects, arrays or slices, and
|
||||
nil. Unsupported values, non-string map keys, non-finite numbers, and cycles
|
||||
match `ErrInvalidConfig` in profiles or `ErrInvalidRequest` in request
|
||||
overrides.
|
||||
## Inputs, Profiles, And Overrides
|
||||
|
||||
Returned requests, profiles, prepared values, results, artifacts, maps, and
|
||||
slices are isolated from internal engine state. Consumers and injected
|
||||
extensions should not retain or mutate values owned by another caller.
|
||||
Use `File`, `Inline`, or `InlineWithURI` to construct request inputs. A request
|
||||
can select a profile explicitly or use the prompt's default profile, and can
|
||||
replace execution settings or the complete output contract.
|
||||
|
||||
## Profiles And Credentials
|
||||
The [public value GoDoc](../../types.go) defines nil, empty, zero, replacement,
|
||||
copy, and credential behavior. The
|
||||
[framework format reference](../formats.md) defines how those request values
|
||||
interact with prompt definitions, file-backed profiles, built-ins, schemas,
|
||||
and framework defaults.
|
||||
|
||||
[`OpenAICompatibleProfile`](../../profiles.go) constructs an ordinary
|
||||
in-memory profile for an OpenAI-compatible chat-completions endpoint.
|
||||
`WithProfiles` rejects duplicate IDs in one call and gives in-memory profiles
|
||||
precedence over explicit file sources and built-ins.
|
||||
For programmatic profiles,
|
||||
[`OpenAICompatibleProfile`](../../profiles.go) converts ordinary
|
||||
OpenAI-compatible settings into a value accepted by `WithProfiles`.
|
||||
|
||||
Raw API keys do not belong in profiles. File-backed profiles may name an
|
||||
environment variable, while an in-memory profile can require a request key.
|
||||
A direct `RunRequest.APIKey` is request-scoped and takes precedence over an
|
||||
environment lookup for the built-in client. Profile fields, ranges, built-ins,
|
||||
precedence, and credential rules are owned by the
|
||||
[framework format reference](../formats.md).
|
||||
## Credentials
|
||||
|
||||
API keys are excluded from JSON, prepared values, and results. The public
|
||||
`String` and `GoString` methods report only whether a direct key is present.
|
||||
Avoid reflection-based dumps of request structs, which can bypass that
|
||||
redaction.
|
||||
File-backed profiles name an environment variable; in-memory profiles can
|
||||
require a direct request key. Direct keys are request-scoped and are excluded
|
||||
from supported JSON values and the package's `String` and `GoString`
|
||||
summaries. The exact precedence and redaction guarantees belong to
|
||||
[`RunRequest`, `GenerateRequest`, and the profile GoDoc](../../types.go).
|
||||
|
||||
## Protect Files And Generated Data
|
||||
|
||||
The default artifact reader opens a `File` reference as a caller-selected
|
||||
operating-system path. It does not constrain paths to an application root,
|
||||
impose an inbound request-size policy, or establish an untrusted-input security
|
||||
boundary. Applications must validate and restrict untrusted paths and payloads
|
||||
before constructing a request, or inject an artifact reader that enforces
|
||||
their filesystem, authorization, and size policies.
|
||||
|
||||
Rendered messages, input and output artifact bodies, raw model output, and
|
||||
validation diagnostics can contain sensitive data. API-key redaction does not
|
||||
sanitize those values. Treat prepared values, results, collaborator requests,
|
||||
errors, and logs according to the application's data-access, retention, and
|
||||
secret-handling policies.
|
||||
|
||||
## Extension Interfaces
|
||||
|
||||
The [`LLMClient`, `GenerateRequest`, and
|
||||
`GenerateResponse`](../../types.go) boundary lets a consumer replace model
|
||||
generation. Injected clients receive copied rendered messages, effective
|
||||
settings, explicit numeric-setting presence, structured-output constraints,
|
||||
and the request-scoped key. They return generated content and token usage.
|
||||
Inject an [`LLMClient` or `ArtifactReader`](../../types.go) when the built-in
|
||||
behavior does not fit the application. Their GoDoc defines concurrent use,
|
||||
context handling, ownership of copied values, nil responses, and preservation
|
||||
of collaborator errors. Implementations must honor cancellation, safely manage
|
||||
copies they retain, avoid unsafe logging of content or credentials, and enforce
|
||||
the application policy that motivated the injection.
|
||||
|
||||
The [`ArtifactReader`](../../types.go) boundary replaces the default inline and
|
||||
file reader for every input. Readers provide artifact content and metadata; the
|
||||
engine fills an empty artifact name from the input-map key. A reader error
|
||||
matches `ErrArtifactLoad` while preserving the original identity for
|
||||
`errors.Is`. A nil artifact with a nil error is also an artifact-load failure.
|
||||
## Handle Errors
|
||||
|
||||
Extensions should honor context cancellation and avoid logging raw prompts,
|
||||
artifacts, or credentials.
|
||||
Use `errors.Is` with the
|
||||
[public error sentinels and operation GoDoc](../../engine.go). The declarations
|
||||
distinguish invalid construction, invalid requests, absent sources,
|
||||
source-loading failures, collaborator failures, and operational validation
|
||||
failures. Specific request conditions may also match the broader
|
||||
`ErrInvalidRequest`, and injected collaborator identities are preserved where
|
||||
documented.
|
||||
|
||||
## Errors
|
||||
|
||||
The [public error declarations](../../engine.go) and
|
||||
[mapping](../../errors.go) preserve these sentinel checks through `errors.Is`:
|
||||
|
||||
- `ErrInvalidConfig`
|
||||
- `ErrInvalidRequest`
|
||||
- `ErrPromptNotFound`
|
||||
- `ErrProfileNotFound`
|
||||
- `ErrProfileRequired`
|
||||
- `ErrPromptLoad`
|
||||
- `ErrProfileLoad`
|
||||
- `ErrAPIKeyEnvMissing`
|
||||
- `ErrArtifactLoad`
|
||||
- `ErrPromptRender`
|
||||
- `ErrLLMGenerate`
|
||||
- `ErrValidation`
|
||||
|
||||
`ErrProfileRequired` and `ErrAPIKeyEnvMissing` also match
|
||||
`ErrInvalidRequest`, allowing either broad request handling or a specific
|
||||
condition. Wrapped collaborator errors retain their identity where the public
|
||||
contract promises it.
|
||||
|
||||
## Consumer Boundary
|
||||
## Application Boundary
|
||||
|
||||
Promptkit is an importable library. It does not own a command, inbound HTTP
|
||||
API, process configuration, or deployment policy. Scriptorium is one
|
||||
downstream application that maps this root package contract into those
|
||||
application concerns.
|
||||
API, process configuration, or deployment policy. Applications map the root
|
||||
package's results and errors into those concerns, including inbound size and
|
||||
trust policy.
|
||||
|
||||
@@ -13,6 +13,7 @@ contributor workflow and validation.
|
||||
| --- | --- | --- |
|
||||
| Root `promptkit` package | Provides the supported engine facade, source and injection options, public request and result values, built-in profile construction, extension interfaces, value conversion, redacted formatting, and public error mapping. | [Package GoDoc](../../doc.go), [engine assembly](../../engine.go) |
|
||||
| `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
|
||||
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
|
||||
| `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, profiles, execution targets, rendering, generation, and validation. | [Domain declarations](../../internal/domain/domain.go) |
|
||||
| `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) |
|
||||
| `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |
|
||||
|
||||
@@ -41,9 +41,9 @@ The implemented internal components consist of:
|
||||
- `internal/usecase`, which coordinates preparation and execution across the
|
||||
internal framework components.
|
||||
|
||||
The `examples/go-library/prepare` package is a maintained downstream consumer
|
||||
of the root facade. It does not expose a library package or participate in
|
||||
internal assembly.
|
||||
The `examples/go-library/prepare` and `examples/go-library/run` packages are
|
||||
maintained downstream consumers of the root facade. They do not expose library
|
||||
packages or participate in internal assembly.
|
||||
|
||||
The root facade assembles the internal repositories, renderer, validator,
|
||||
outbound client, and use-case runner while translating public values and
|
||||
|
||||
254
docs/release.md
254
docs/release.md
@@ -7,27 +7,91 @@ tags. It does not publish runnable binaries or binary packages and does not
|
||||
currently use hosted CI. The release maintainer performs and records the
|
||||
required validation.
|
||||
|
||||
The first planned release is `v0.1.0`. Do not create that tag until the
|
||||
framework has been extracted and the resulting public library has passed this
|
||||
procedure. Later tags use the `vMAJOR.MINOR.PATCH` form. While Promptkit remains
|
||||
pre-`v1`, release notes must identify intentional public API changes and any
|
||||
consumer migration required by them.
|
||||
`v0.1.0` is the initial published release. Later releases use semantic
|
||||
`vMAJOR.MINOR.PATCH` tags. Before `v1`, minor releases may change the public
|
||||
API and patch releases preserve compatibility within their minor line. Every
|
||||
pre-`v1` release note must summarize compatibility, identify public API
|
||||
changes, and state any action required of consumers.
|
||||
|
||||
## Prepare The Release
|
||||
Promptkit releases are source-only. The annotated tag message is the release
|
||||
note; there is no separate hosted release or binary packaging step.
|
||||
|
||||
Work from a clean checkout of the intended release commit, outside any Go
|
||||
workspace and without a local module replacement. Confirm the source commit is
|
||||
already published through the normal branch workflow.
|
||||
## Establish The Candidate
|
||||
|
||||
From the Promptkit repository root, verify the checkout:
|
||||
Choose a version that has not been published and export it as
|
||||
`RELEASE_VERSION`. Run every command in this procedure from the Promptkit
|
||||
repository root in the same POSIX shell. Do not reuse `v0.1.0` or another
|
||||
existing version.
|
||||
|
||||
The following guard derives the release commit from `HEAD` and stops on a
|
||||
missing or malformed version, a checkout other than synchronized `main`,
|
||||
uncommitted changes, an active Go workspace, a module replacement, a vendor
|
||||
tree, or an existing local or remote tag:
|
||||
|
||||
```sh
|
||||
gowork=$(go env GOWORK)
|
||||
test -z "$gowork" || test "$gowork" = off
|
||||
test -z "$(git status --short)"
|
||||
git fetch --tags origin
|
||||
set -eu
|
||||
|
||||
: "${RELEASE_VERSION:?export an unpublished vMAJOR.MINOR.PATCH version}"
|
||||
if ! printf '%s\n' "$RELEASE_VERSION" |
|
||||
grep -Eq '^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$'
|
||||
then
|
||||
printf '%s\n' "invalid release version: $RELEASE_VERSION" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
RELEASE_COMMIT=$(git rev-parse --verify 'HEAD^{commit}')
|
||||
export RELEASE_COMMIT
|
||||
|
||||
check_release_candidate() {
|
||||
test "$(git branch --show-current)" = main
|
||||
test -z "$(git status --porcelain)"
|
||||
|
||||
gowork_value=$(go env GOWORK)
|
||||
case "$gowork_value" in
|
||||
''|off) ;;
|
||||
*)
|
||||
printf '%s\n' "active Go workspace: $gowork_value" >&2
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
|
||||
test -z "$(git ls-files go.work go.work.sum)"
|
||||
test ! -e vendor
|
||||
if grep -Eq '^[[:space:]]*replace([[:space:]]|\()' go.mod
|
||||
then
|
||||
printf '%s\n' 'go.mod contains a replacement' >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
git fetch origin main --tags
|
||||
test "$RELEASE_COMMIT" = \
|
||||
"$(git rev-parse --verify 'refs/remotes/origin/main^{commit}')"
|
||||
|
||||
if git show-ref --verify --quiet "refs/tags/$RELEASE_VERSION"
|
||||
then
|
||||
printf '%s\n' "local tag already exists: $RELEASE_VERSION" >&2
|
||||
return 1
|
||||
fi
|
||||
if test -n "$(
|
||||
git ls-remote --tags origin \
|
||||
"refs/tags/$RELEASE_VERSION" \
|
||||
"refs/tags/$RELEASE_VERSION^{}"
|
||||
)"
|
||||
then
|
||||
printf '%s\n' "remote tag already exists: $RELEASE_VERSION" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
check_release_candidate
|
||||
```
|
||||
|
||||
Do not continue unless the guard completes successfully. In particular, push
|
||||
the intended commit through the normal `main` branch workflow before release;
|
||||
the tag procedure is not a substitute for publishing the source commit.
|
||||
|
||||
## Validate The Candidate
|
||||
|
||||
Confirm the module and root package metadata:
|
||||
|
||||
```sh
|
||||
@@ -42,7 +106,7 @@ gitea.maximumdirect.net/eric/promptkit 1.25.5
|
||||
promptkit gitea.maximumdirect.net/eric/promptkit
|
||||
```
|
||||
|
||||
Run the same default Go validation required by the
|
||||
Run the complete maintainer validation required by the
|
||||
[development guide](development.md):
|
||||
|
||||
```sh
|
||||
@@ -53,84 +117,158 @@ go build ./...
|
||||
go run ./examples/go-library/prepare
|
||||
```
|
||||
|
||||
Check every tracked Go file and repository whitespace:
|
||||
Check every tracked Go file. This command must produce no output:
|
||||
|
||||
```sh
|
||||
gofmt -l $(git ls-files '*.go')
|
||||
unformatted=$(
|
||||
git ls-files '*.go' |
|
||||
while IFS= read -r go_file
|
||||
do
|
||||
gofmt -l "$go_file"
|
||||
done
|
||||
)
|
||||
test -z "$unformatted"
|
||||
```
|
||||
|
||||
Follow every maintained Markdown link and confirm that its local or published
|
||||
target exists. Review the repository for generated binaries, test or coverage
|
||||
output, credentials, template residue, downloaded assets, and other files that
|
||||
do not belong in source control.
|
||||
|
||||
Recheck module and repository hygiene, whitespace, and the clean checkout:
|
||||
|
||||
```sh
|
||||
test -z "$(git ls-files go.work go.work.sum)"
|
||||
test ! -e vendor
|
||||
if grep -Eq '^[[:space:]]*replace([[:space:]]|\()' go.mod
|
||||
then
|
||||
printf '%s\n' 'go.mod contains a replacement' >&2
|
||||
exit 1
|
||||
fi
|
||||
git diff --check
|
||||
test -z "$(git status --porcelain)"
|
||||
```
|
||||
|
||||
The formatting command must produce no paths. Follow every maintained Markdown
|
||||
link and confirm its target exists. Review the repository for generated
|
||||
binaries, test or coverage output, credentials, template residue, and other
|
||||
files that do not belong in source control.
|
||||
## Write The Release Note
|
||||
|
||||
Confirm that no workspace override is tracked and that `go.mod` contains no
|
||||
`replace` directive:
|
||||
Prepare a plain-text annotated-tag message outside the repository and export
|
||||
its path as `RELEASE_NOTES_FILE`. Use this form, replacing each summary with
|
||||
release-specific text; write `None.` when there are no public API changes or
|
||||
consumer actions:
|
||||
|
||||
```sh
|
||||
git ls-files go.work go.work.sum
|
||||
rg -n '^replace\b' go.mod
|
||||
```text
|
||||
Promptkit vMAJOR.MINOR.PATCH
|
||||
|
||||
Validated commit: full commit ID
|
||||
Compatibility: compatibility summary
|
||||
Public API changes: changes or None.
|
||||
Consumer action: required action or None.
|
||||
```
|
||||
|
||||
Both commands must produce no output. Re-run `git status --short` and require a
|
||||
clean result after every validation and review check.
|
||||
|
||||
## Create And Publish The Tag
|
||||
|
||||
Choose the semantic version from the intended compatibility change. Record the
|
||||
release commit before tagging:
|
||||
After writing it, require all release-note fields, the selected version, and
|
||||
the validated commit to be present:
|
||||
|
||||
```sh
|
||||
release_version=v0.1.0
|
||||
release_commit=$(git rev-parse HEAD)
|
||||
: "${RELEASE_NOTES_FILE:?export the path to the release-note file}"
|
||||
test -f "$RELEASE_NOTES_FILE"
|
||||
test -s "$RELEASE_NOTES_FILE"
|
||||
grep -F "Promptkit $RELEASE_VERSION" "$RELEASE_NOTES_FILE"
|
||||
grep -F "Validated commit: $RELEASE_COMMIT" "$RELEASE_NOTES_FILE"
|
||||
grep -F 'Compatibility:' "$RELEASE_NOTES_FILE"
|
||||
grep -F 'Public API changes:' "$RELEASE_NOTES_FILE"
|
||||
grep -F 'Consumer action:' "$RELEASE_NOTES_FILE"
|
||||
```
|
||||
|
||||
Replace the example version for later releases and keep both values in the same
|
||||
shell for the remaining commands. Confirm the tag does not already exist
|
||||
locally or remotely:
|
||||
Inspect the complete message and confirm that it accurately records the
|
||||
compatibility impact, public API changes, and required consumer action.
|
||||
|
||||
## Create And Inspect The Tag
|
||||
|
||||
Run the candidate guard again immediately before tag creation. This ensures
|
||||
that validation or release-note preparation did not change the checkout and
|
||||
that the commit is still published and untagged:
|
||||
|
||||
```sh
|
||||
test -z "$(git tag --list "$release_version")"
|
||||
test -z "$(git ls-remote --tags origin "refs/tags/$release_version")"
|
||||
check_release_candidate
|
||||
```
|
||||
|
||||
Create an annotated tag whose message identifies the release and records that
|
||||
the documented validation passed for the tagged commit:
|
||||
Create the annotated tag from the prepared release note and bind it explicitly
|
||||
to the validated commit:
|
||||
|
||||
```sh
|
||||
git tag --annotate "$release_version" \
|
||||
--message "Promptkit $release_version; documented validation passed for $release_commit"
|
||||
git tag --annotate "$RELEASE_VERSION" \
|
||||
--file "$RELEASE_NOTES_FILE" \
|
||||
"$RELEASE_COMMIT"
|
||||
```
|
||||
|
||||
Inspect the tag before publication:
|
||||
Inspect both the tag message and its source commit before publication:
|
||||
|
||||
```sh
|
||||
git show --no-patch --decorate "$release_version"
|
||||
test "$(git rev-list -n 1 "$release_version")" = "$release_commit"
|
||||
test "$(git cat-file -t "refs/tags/$RELEASE_VERSION")" = tag
|
||||
git show --no-patch --decorate "refs/tags/$RELEASE_VERSION"
|
||||
test "$(
|
||||
git rev-parse --verify "refs/tags/$RELEASE_VERSION^{commit}"
|
||||
)" = "$RELEASE_COMMIT"
|
||||
```
|
||||
|
||||
Publish the tag without relying on a hosting-provider-specific release
|
||||
interface:
|
||||
If inspection finds an error, delete the unpublished local tag, correct the
|
||||
release note or candidate, and repeat the guards. Never move or recreate a tag
|
||||
that has been published.
|
||||
|
||||
## Publish The Selected Tag
|
||||
|
||||
Push only the selected tag ref. Do not use `git push --tags`:
|
||||
|
||||
```sh
|
||||
git push origin "refs/tags/$release_version"
|
||||
git push origin \
|
||||
"refs/tags/$RELEASE_VERSION:refs/tags/$RELEASE_VERSION"
|
||||
```
|
||||
|
||||
## Verify Publication
|
||||
|
||||
Confirm that the remote tag object matches the local annotated tag and still
|
||||
resolves to the intended source commit:
|
||||
Compare the remote annotated-tag object with the local object, then compare the
|
||||
remote peeled source commit with the validated commit:
|
||||
|
||||
```sh
|
||||
remote_tag=$(git ls-remote --tags origin "refs/tags/$release_version" | awk '{print $1}')
|
||||
test "$remote_tag" = "$(git rev-parse "refs/tags/$release_version")"
|
||||
test "$(git rev-list -n 1 "refs/tags/$release_version")" = "$release_commit"
|
||||
remote_tag=$(
|
||||
git ls-remote --tags origin "refs/tags/$RELEASE_VERSION" |
|
||||
awk 'NR == 1 { print $1 }'
|
||||
)
|
||||
remote_commit=$(
|
||||
git ls-remote --tags origin "refs/tags/$RELEASE_VERSION^{}" |
|
||||
awk 'NR == 1 { print $1 }'
|
||||
)
|
||||
test -n "$remote_tag"
|
||||
test "$remote_tag" = \
|
||||
"$(git rev-parse --verify "refs/tags/$RELEASE_VERSION")"
|
||||
test "$remote_commit" = "$RELEASE_COMMIT"
|
||||
```
|
||||
|
||||
Promptkit must publish the required tag before Scriptorium or another consumer
|
||||
publishes a release that depends on that version. Released consumer modules
|
||||
must not use a local replacement or unpublished Promptkit revision.
|
||||
Finally, resolve the version as an ordinary Go module in a temporary module
|
||||
outside this repository and without a workspace or replacement:
|
||||
|
||||
```sh
|
||||
resolution_dir=$(mktemp -d)
|
||||
(
|
||||
trap 'rm -rf "$resolution_dir"' 0 1 2 15
|
||||
cd "$resolution_dir"
|
||||
GOWORK=off go mod init example.com/promptkit-release-check
|
||||
GOWORK=off go mod download \
|
||||
"gitea.maximumdirect.net/eric/promptkit@$RELEASE_VERSION"
|
||||
resolved_version=$(
|
||||
GOWORK=off go list -m -f '{{.Version}}' \
|
||||
"gitea.maximumdirect.net/eric/promptkit@$RELEASE_VERSION"
|
||||
)
|
||||
test "$resolved_version" = "$RELEASE_VERSION"
|
||||
)
|
||||
```
|
||||
|
||||
Promptkit must publish and verify the required version before Scriptorium or
|
||||
another consumer publishes a release that depends on it. This ordering does
|
||||
not replace the consumer project's own release procedure. Released consumers
|
||||
must select the published Promptkit tag through ordinary module resolution,
|
||||
without a workspace, replacement, vendored Promptkit source, or unpublished
|
||||
revision.
|
||||
|
||||
## Policy Changes
|
||||
|
||||
|
||||
222
docs/roadmap/documentation.md
Normal file
222
docs/roadmap/documentation.md
Normal file
@@ -0,0 +1,222 @@
|
||||
# Documentation Hardening Roadmap
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap coordinates a focused pass over Promptkit's documentation and
|
||||
public contract. The work should close the gaps identified by the documentation
|
||||
audit, strengthen canonical ownership, and leave consumers and contributors
|
||||
with guidance that is accurate, navigable, and proportionate to their needs.
|
||||
|
||||
This is planning material, not a description of implemented behavior. Follow
|
||||
the [documentation policy](../policy/documentation.md) throughout the work and
|
||||
update current-state documents only when their claims are supported by the
|
||||
implementation and tests.
|
||||
|
||||
## Scope And Principles
|
||||
|
||||
The work covers public GoDoc, consumer guidance, maintained examples, format
|
||||
and integration references, architecture ownership, roadmap lifecycle, and
|
||||
documentation validation.
|
||||
|
||||
- Resolve ambiguous behavior before documenting it as a contract.
|
||||
- Keep exact exported API semantics in Go declarations and GoDoc.
|
||||
- Keep task-oriented guidance in consumer documents and complete runnable
|
||||
artifacts in `examples/`.
|
||||
- Put security-relevant consumer responsibilities at the public boundary, not
|
||||
only in internal contributor documents.
|
||||
- Remove duplicated ownership instead of synchronizing parallel references.
|
||||
- Make documentation checks reproducible where practical.
|
||||
- Preserve existing behavior unless a stage explicitly selects and tests an
|
||||
API change.
|
||||
|
||||
The roadmap does not add new Promptkit features, redesign framework formats,
|
||||
or implement ideas from [the future feature catalog](future.md). If resolving
|
||||
an ambiguity requires a behavioral change, treat that change as a separately
|
||||
reviewable implementation unit and update its canonical documentation in the
|
||||
same unit.
|
||||
|
||||
## Stage 1: Resolve Public Contract Questions
|
||||
|
||||
Before expanding prose, decide the intended contract for exported behavior
|
||||
that is currently ambiguous.
|
||||
|
||||
- [x] Decide whether `RunRequest.Metadata` has a supported observable purpose.
|
||||
Define its propagation and ownership, or remove or deprecate it through an
|
||||
intentional public API change.
|
||||
- [x] Decide and test whether one `Engine` supports concurrent `Prepare` and
|
||||
`Run` calls.
|
||||
- [x] Decide how repeated options of the same category behave, including
|
||||
prompt, profile, schema, model-client, and artifact-reader options.
|
||||
- [x] Define which public values have supported JSON representations.
|
||||
- [x] Define JSON time units and omission behavior, including the relationship
|
||||
between prepared-run and run-result durations.
|
||||
- [x] Decide whether run IDs and exposed hashes have stable formats or must be
|
||||
treated as opaque values.
|
||||
- [x] Confirm the intended transport-timeout default and its zero or negative
|
||||
configuration semantics.
|
||||
- [x] Confirm the supported JSON Schema dialect and reference boundaries,
|
||||
including whether remote references are allowed.
|
||||
|
||||
The selected exported API contracts are implemented and tested. Their durable
|
||||
definitions now belong to the root package declarations and GoDoc.
|
||||
|
||||
One format-level decision remains here until Stage 5 moves it to the framework
|
||||
format reference: JSON Schema uses Draft 2020-12, with that dialect selected
|
||||
when `$schema` is omitted. Same-document fragments and relative references
|
||||
contained by a directory or `fs.FS` schema root are supported. A single-file
|
||||
source supports only references contained in that document. Absolute,
|
||||
escaping, and remote references are rejected.
|
||||
|
||||
**Gate:** Each question has an explicit answer backed by existing behavior or
|
||||
by an accepted implementation change and proportionate tests. No later stage
|
||||
should invent a contract merely to fill a documentation gap.
|
||||
|
||||
## Stage 2: Make GoDoc The Canonical Public Contract
|
||||
|
||||
Strengthen the root package declarations so `go doc` is sufficient to
|
||||
understand exact public behavior without relying on internal documents.
|
||||
|
||||
- [x] Add useful field-level GoDoc to configuration, request, profile,
|
||||
execution-target, result, artifact, validation, structured-output, and model
|
||||
client values.
|
||||
- [x] Document required fields and nil, empty, and zero-value semantics.
|
||||
- [x] Document override, replacement, profile-precedence, and copy-ownership
|
||||
behavior where it belongs to the exported API.
|
||||
- [x] Document credential inputs, redaction, and the values intentionally
|
||||
excluded from serialization.
|
||||
- [x] Give each public error sentinel an accurate comment and document the
|
||||
supported `errors.Is` relationships.
|
||||
- [x] Document engine concurrency and option-composition behavior selected in
|
||||
Stage 1.
|
||||
- [x] Document serialization, time, run-ID, and hash semantics selected in
|
||||
Stage 1.
|
||||
- [x] Review constructor, option, extension-interface, `Prepare`, and `Run`
|
||||
GoDoc for complete failure and cancellation expectations.
|
||||
|
||||
Update the [consumer guide](../consumers/pkg-promptkit.md) to summarize and link
|
||||
to these contracts instead of maintaining exhaustive copies of exported names
|
||||
or exact semantics.
|
||||
|
||||
**Gate:** `go doc -all .` presents a coherent public contract, exported
|
||||
declarations have accurate comments, and contract tests protect every newly
|
||||
documented behavior whose compatibility risk warrants durable coverage.
|
||||
|
||||
## Stage 3: Improve Consumer Safety And Executable Guidance
|
||||
|
||||
Move consumer-relevant security boundaries to the places where consumers will
|
||||
encounter them and add one representative execution workflow.
|
||||
|
||||
- [x] Explain in public GoDoc and the consumer guide that the default file
|
||||
artifact reader accepts unrestricted caller-selected paths.
|
||||
- [x] Make clear that Promptkit does not impose an application root, inbound
|
||||
request-size policy, or untrusted-input security boundary.
|
||||
- [x] Explain that rendered messages, artifact bodies, raw model output, and
|
||||
validation details may be sensitive even when credentials are redacted.
|
||||
- [x] Clarify the responsibilities of injected artifact readers and model
|
||||
clients for cancellation, copying, logging, and secret handling.
|
||||
- [x] Add a maintained offline `Run` example using an injected deterministic
|
||||
model client, without credentials, live network access, or paid calls.
|
||||
- [x] Link the consumer guide to the execution example and keep embedded
|
||||
snippets smaller than the maintained artifact.
|
||||
- [x] Decide whether the existing preparation example should remain separate
|
||||
or share reusable fixtures without obscuring either workflow.
|
||||
|
||||
The preparation and execution examples remain separate, self-contained
|
||||
workflows. Each keeps its own small prompt fixture so consumers can copy or run
|
||||
one example without depending on the other.
|
||||
|
||||
**Gate:** Both preparation and execution have complete, secret-free,
|
||||
deterministic consumer examples, and the consumer guide exposes the important
|
||||
filesystem and data-sensitivity boundaries without leaking internal mechanics.
|
||||
|
||||
## Stage 4: Restore Canonical Ownership
|
||||
|
||||
Remove parallel definitions and make navigation follow the ownership model in
|
||||
the documentation policy.
|
||||
|
||||
- [ ] Reduce the [architecture policy](../policy/architecture.md) to durable
|
||||
boundaries, layers, dependency direction, invariants, and non-goals.
|
||||
- [ ] Keep the exact implemented package and component inventory solely in the
|
||||
[internal component overview](../internal/overview.md).
|
||||
- [ ] Review the consumer guide's public error and option lists so they remain
|
||||
task-oriented summaries rather than duplicate API references.
|
||||
- [ ] Review internal documents for public-contract statements that should be
|
||||
links to GoDoc or the format and integration owners.
|
||||
- [ ] Reconcile the documentation policy's temporary-roadmap lifecycle with
|
||||
the continuing idea-catalog role of `docs/roadmap/future.md`.
|
||||
- [ ] Rephrase or link roadmap statements that depend on exact current API
|
||||
behavior, particularly the runtime reasoning entry.
|
||||
- [ ] Confirm that every document states its audience or purpose and links to
|
||||
the canonical owner of adjacent topics.
|
||||
|
||||
**Gate:** Every authoritative fact has one clear owner, package inventory
|
||||
changes no longer require edits to the architecture policy, and roadmaps
|
||||
cannot be mistaken for current-state references.
|
||||
|
||||
## Stage 5: Refine Format And Integration References
|
||||
|
||||
Close compatibility gaps in the documents that own file formats and outbound
|
||||
wire behavior.
|
||||
|
||||
- [ ] State or canonically link the exact session-ID limit enforced by the
|
||||
OpenAI-compatible client.
|
||||
- [ ] State the configured and default transport-timeout behavior without
|
||||
referring to an unnamed internal default.
|
||||
- [ ] Document the JSON Schema dialect and local, contained, and remote
|
||||
reference behavior selected in Stage 1.
|
||||
- [ ] Clarify structured-output naming and strictness when those values are
|
||||
part of the public or integration contract.
|
||||
- [ ] Add a caveat that built-in profiles are maintained configurations, not a
|
||||
guarantee of continuing third-party model availability.
|
||||
- [ ] Recheck every prompt, profile, schema, credential, request-body, response,
|
||||
timeout, and precedence statement against its owning implementation and
|
||||
tests.
|
||||
|
||||
**Gate:** A consumer can determine the supported file and wire compatibility
|
||||
boundaries without consulting internal source code or relying on unspecified
|
||||
defaults.
|
||||
|
||||
## Stage 6: Make Documentation Validation Reproducible
|
||||
|
||||
Align contributor and release procedures around a small, consistent set of
|
||||
checks.
|
||||
|
||||
- [ ] Use one robust command for checking every tracked Go file with `gofmt`.
|
||||
- [ ] Provide a repository-local or clearly documented command that validates
|
||||
local Markdown targets and heading fragments.
|
||||
- [ ] Decide how published external links are checked without making ordinary
|
||||
validation depend on mutable network services.
|
||||
- [ ] Reconcile the validation descriptions in the
|
||||
[development guide](../development.md),
|
||||
[testing policy](../policy/testing.md), and
|
||||
[release procedure](../release.md) so one document owns each requirement.
|
||||
- [ ] Ensure example validation covers every maintained example added by this
|
||||
roadmap.
|
||||
- [ ] Keep documentation-only validation proportionate while requiring full Go
|
||||
validation when commands, examples, generated output, or checked behavior
|
||||
changes.
|
||||
|
||||
**Gate:** A maintainer can run the documented formatting, link, example, Go,
|
||||
and repository-hygiene checks exactly as written, with no hidden manual
|
||||
procedure for local documentation.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
The roadmap is complete when:
|
||||
|
||||
- all Stage 1 contract questions are resolved;
|
||||
- GoDoc is the authoritative and sufficient exported API reference;
|
||||
- consumer guidance covers unrestricted file access and sensitive generated
|
||||
data;
|
||||
- maintained offline examples cover both `Prepare` and `Run`;
|
||||
- architecture, inventory, consumer, internal, format, integration, and
|
||||
roadmap documents follow their assigned ownership boundaries;
|
||||
- schema, session, timeout, structured-output, and built-in-profile
|
||||
compatibility statements are explicit;
|
||||
- documentation validation is reproducible and consistent across contributor
|
||||
and release workflows; and
|
||||
- the complete maintainer validation passes.
|
||||
|
||||
After completion, move any durable decisions to GoDoc, policy, format,
|
||||
integration, or ADR owners as appropriate. Remove this roadmap after incoming
|
||||
links are updated; do not retain it as a second current-state reference.
|
||||
122
docs/roadmap/future.md
Normal file
122
docs/roadmap/future.md
Normal file
@@ -0,0 +1,122 @@
|
||||
# Future Feature Ideas
|
||||
|
||||
## Purpose
|
||||
|
||||
This document catalogs reasonably specific ideas that may be useful in future
|
||||
Promptkit development. It is an idea pool, not a commitment, schedule, or
|
||||
description of current behavior.
|
||||
|
||||
Ideas belong here while they are worth retaining but have not been selected
|
||||
for active development. Keep each entry at the level of intended capability,
|
||||
consumer value, and important scope boundaries. Defer API design,
|
||||
implementation details, sequencing, and acceptance criteria until an idea is
|
||||
selected.
|
||||
|
||||
## Using This Catalog
|
||||
|
||||
- Add an idea when its purpose and likely value can be stated clearly.
|
||||
- Keep entries independent enough that maintainers can evaluate and select
|
||||
them individually.
|
||||
- Note significant dependencies or boundary concerns, but do not turn entries
|
||||
into implementation plans.
|
||||
- Treat inclusion as an invitation to evaluate, not as approval or priority.
|
||||
- When an idea is selected, move its active planning to a focused roadmap or,
|
||||
when it requires a durable architectural decision, an ADR. Update
|
||||
current-state documentation only when implementation lands.
|
||||
- Remove ideas that are no longer relevant. Retain a rejected idea only when
|
||||
its rationale is likely to prevent repeated reconsideration.
|
||||
|
||||
Future capabilities must continue to respect the
|
||||
[architecture policy](../policy/architecture.md), particularly Promptkit's
|
||||
role as an application-neutral library and its boundary with downstream
|
||||
consumers.
|
||||
|
||||
## Ideas
|
||||
|
||||
### Extensible LLM backend registry
|
||||
|
||||
Introduce a registry that separates backend-specific connection,
|
||||
authentication, and limited request defaults from model execution profiles.
|
||||
Initial support would cover OpenAI-compatible backends and include a small
|
||||
built-in catalog, potentially starting with OpenRouter. A profile could select
|
||||
a backend while optionally overriding its default endpoint, and each backend
|
||||
could name an optional environment variable for its API key without storing
|
||||
the credential itself. Downstream consumers could register additional,
|
||||
uniquely named backends, such as OpenAI or unauthenticated local-network
|
||||
services, but could not replace built-in IDs. Model selection and generation
|
||||
settings would remain profile concerns, and custom model clients would remain
|
||||
available for behavior outside the registry's supported protocol.
|
||||
|
||||
### Backend-specific concurrency management
|
||||
|
||||
Extend the proposed LLM backend registry with optional per-backend concurrency
|
||||
limits and bounded, buffered admission queues. Promptkit could then route
|
||||
simultaneous generation requests according to backend capacity while
|
||||
containing accidental runaway submission. Downstream consumers would continue
|
||||
invoking synchronous `Run` calls, including concurrently from multiple
|
||||
goroutines, and each admitted call would wait for and return its ordinary
|
||||
result.
|
||||
|
||||
- Scope limits to an engine instance rather than hidden process-global state.
|
||||
- Give different backend IDs independent capacity pools. A profile endpoint
|
||||
override would remain part of its selected backend's pool.
|
||||
- Configure active concurrency and waiting capacity separately. Concurrency
|
||||
protects the backend, while queue capacity protects the process from
|
||||
admitting an unbounded backlog.
|
||||
- Give queue capacity a generous, configurable bounded default intended as a
|
||||
safety ceiling for bugs or unintended loops rather than a routine
|
||||
application constraint. Select an exact default during implementation
|
||||
planning and measurement.
|
||||
- Reject a call with a recognizable capacity error when its backend queue is
|
||||
full rather than allowing it to wait outside the bounded queue.
|
||||
- Admit requests before expensive preparation and artifact copying where
|
||||
practical so queued work remains lightweight.
|
||||
- Apply a limit to each actual generation request, including repair attempts,
|
||||
without unnecessarily serializing prompt preparation.
|
||||
- Make queued and active waits respect caller cancellation and deadlines.
|
||||
- Treat concurrency as backend policy rather than a profile-level model
|
||||
setting.
|
||||
- Keep the queue ephemeral and in-process, with no survival guarantee across
|
||||
engine or process shutdown.
|
||||
- Preserve the existing execution model as far as practical. Durable jobs,
|
||||
polling, priorities, application worker lifecycle, retries, and
|
||||
cross-process coordination would be separate future capabilities.
|
||||
|
||||
### Explicit per-run session and reasoning controls
|
||||
|
||||
Allow consumers to associate a session ID with each run and to inherit,
|
||||
replace, or explicitly disable the reasoning effort configured by its selected
|
||||
profile. Prompt definitions can currently derive a session ID from a template,
|
||||
and a non-empty runtime `ReasoningEffort` can replace the profile value, but
|
||||
there is no direct request-level session ID and an empty reasoning value means
|
||||
that no override was supplied. These controls would let consumers reuse one
|
||||
prompt and model profile across sessions and reasoning levels without
|
||||
maintaining duplicate definitions.
|
||||
|
||||
- Preserve a prompt's session ID template and a profile's reasoning effort as
|
||||
reusable defaults.
|
||||
- Let a directly supplied per-run session ID take precedence over a rendered
|
||||
prompt default while retaining the existing validation limit and outbound
|
||||
representation.
|
||||
- Distinguish an omitted runtime reasoning choice from an explicit request to
|
||||
disable reasoning.
|
||||
- Ensure disabling reasoning omits the corresponding provider request setting
|
||||
rather than relying on a provider-specific magic value.
|
||||
- Keep the effective session ID and reasoning choice visible in prepared and
|
||||
run metadata and available to injected model clients without introducing
|
||||
additional prompt or profile selection mechanisms.
|
||||
|
||||
## Entry Format
|
||||
|
||||
Use a short heading followed by a concise summary. Add focused bullets when
|
||||
they help preserve important scope boundaries without becoming an
|
||||
implementation plan:
|
||||
|
||||
```markdown
|
||||
### Idea name
|
||||
|
||||
Describe the intended capability, who benefits, and the most important scope
|
||||
boundary or dependency.
|
||||
|
||||
- Optionally record an important behavior or boundary.
|
||||
```
|
||||
141
engine.go
141
engine.go
@@ -22,42 +22,93 @@ import (
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/validate"
|
||||
)
|
||||
|
||||
// ErrInvalidConfig indicates invalid public engine configuration.
|
||||
// ErrInvalidConfig identifies invalid engine construction, including missing
|
||||
// required configuration, invalid options, and a nil Engine receiver.
|
||||
var ErrInvalidConfig = errors.New("invalid engine configuration")
|
||||
|
||||
var (
|
||||
// ErrInvalidRequest identifies a request whose required values, overrides,
|
||||
// credentials, or effective settings are invalid.
|
||||
ErrInvalidRequest = errors.New("invalid run request")
|
||||
// ErrPromptNotFound identifies a requested prompt ID or version that is not
|
||||
// present in the selected prompt source. It does not also match
|
||||
// ErrPromptLoad.
|
||||
ErrPromptNotFound = errors.New("prompt not found")
|
||||
// ErrProfileNotFound identifies a selected profile ID that is absent from
|
||||
// every configured profile source. It does not also match ErrProfileLoad.
|
||||
ErrProfileNotFound = errors.New("profile not found")
|
||||
// ErrProfileRequired identifies a request for which neither RunRequest.ProfileID
|
||||
// nor the selected prompt's default profile is present. Such an error also
|
||||
// matches ErrInvalidRequest.
|
||||
ErrProfileRequired = errors.New("profile selection is required")
|
||||
// ErrPromptLoad identifies a failure to read, decode, validate, select, or
|
||||
// hash a prompt definition, except for the not-found case represented by
|
||||
// ErrPromptNotFound.
|
||||
ErrPromptLoad = errors.New("failed to load prompt definition")
|
||||
// ErrProfileLoad identifies a failure to read, decode, validate, or select
|
||||
// an execution profile, except for the not-found case represented by
|
||||
// ErrProfileNotFound.
|
||||
ErrProfileLoad = errors.New("failed to load execution profile")
|
||||
// ErrAPIKeyEnvMissing identifies an APIKeyEnv whose environment variable is
|
||||
// unset or empty when no direct RunRequest.APIKey takes precedence. Such an
|
||||
// error also matches ErrInvalidRequest.
|
||||
ErrAPIKeyEnvMissing = errors.New("api_key_env points to an unset environment variable")
|
||||
// ErrArtifactLoad identifies a failure to resolve an input artifact. Errors
|
||||
// returned by an injected ArtifactReader remain available through errors.Is.
|
||||
ErrArtifactLoad = errors.New("failed to load artifact")
|
||||
// ErrPromptRender identifies a failure to render prompt messages or the
|
||||
// session ID from the resolved inputs and variables.
|
||||
ErrPromptRender = errors.New("failed to render prompt")
|
||||
// ErrLLMGenerate identifies a model-client failure or a nil successful
|
||||
// response. Errors returned by an injected LLMClient remain available
|
||||
// through errors.Is.
|
||||
ErrLLMGenerate = errors.New("failed to generate output")
|
||||
// ErrValidation identifies an operational failure to load or compile a
|
||||
// schema or validate output. A completed validation whose Status is
|
||||
// ValidationFailed is returned in RunResult without this error.
|
||||
ErrValidation = errors.New("failed to validate output")
|
||||
)
|
||||
|
||||
// Engine prepares and runs Promptkit prompt requests.
|
||||
//
|
||||
// An Engine is safe for concurrent calls to [Engine.Prepare] and [Engine.Run].
|
||||
// Injected collaborators may consequently be invoked concurrently.
|
||||
type Engine struct {
|
||||
runner *usecase.Runner
|
||||
}
|
||||
|
||||
// Config configures a public Promptkit engine.
|
||||
// Config selects the directory-backed sources and built-in model-client
|
||||
// transport used by [NewEngine]. Config has no stable JSON representation.
|
||||
type Config struct {
|
||||
// PromptDir is the directory searched recursively for prompt definitions.
|
||||
// It is required unless a WithPromptFS or WithPromptFile option supplies the
|
||||
// prompt source.
|
||||
PromptDir string
|
||||
// ProfileDir is an optional directory whose profiles take precedence over
|
||||
// embedded built-in profiles. An empty value selects only built-ins unless
|
||||
// profile options are also supplied.
|
||||
ProfileDir string
|
||||
// SchemaDir is the root for JSON Schema files. An empty value uses the
|
||||
// current directory. WithSchemaFS or WithSchemaFile replaces this source.
|
||||
SchemaDir string
|
||||
// Timeout is the transport-wide safety cap for the built-in LLM client
|
||||
// when HTTPClient is absent or has a non-positive timeout.
|
||||
// when HTTPClient is absent or has a non-positive timeout. A zero or negative
|
||||
// value selects the 10-minute default.
|
||||
Timeout time.Duration
|
||||
// HTTPClient is cloned for the built-in LLM client. Its positive Timeout
|
||||
// takes precedence over Config.Timeout as the transport-wide safety cap.
|
||||
// takes precedence over Timeout. A zero or negative client Timeout inherits
|
||||
// Timeout or the 10-minute default. The supplied client is not mutated. This
|
||||
// field is ignored when WithLLMClient is used.
|
||||
HTTPClient *http.Client
|
||||
}
|
||||
|
||||
// Option customizes engine construction.
|
||||
//
|
||||
// NewEngine applies options in argument order and ignores nil options. Within
|
||||
// each prompt-source, profile-source, in-memory-profile, schema-source,
|
||||
// model-client, and artifact-reader category, the last non-nil valid option
|
||||
// replaces earlier options in that category. An invalid option fails
|
||||
// construction even if a later option would replace it.
|
||||
type Option interface {
|
||||
apply(*engineOptions) error
|
||||
}
|
||||
@@ -82,7 +133,10 @@ type engineOptions struct {
|
||||
artifactSource bool
|
||||
}
|
||||
|
||||
// WithLLMClient injects a custom LLM client for execution.
|
||||
// WithLLMClient replaces the built-in model client used by [Engine.Run].
|
||||
//
|
||||
// A nil client makes NewEngine fail with ErrInvalidConfig. The client may be
|
||||
// called concurrently and is not used by [Engine.Prepare].
|
||||
func WithLLMClient(client LLMClient) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
if client == nil {
|
||||
@@ -93,7 +147,11 @@ func WithLLMClient(client LLMClient) Option {
|
||||
})
|
||||
}
|
||||
|
||||
// WithArtifactReader injects a reader for every input artifact reference.
|
||||
// WithArtifactReader replaces the default reader for every input artifact
|
||||
// reference, regardless of its ArtifactRef.Type.
|
||||
//
|
||||
// A nil reader makes NewEngine fail with ErrInvalidConfig. The reader may be
|
||||
// called concurrently.
|
||||
func WithArtifactReader(reader ArtifactReader) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
if reader == nil {
|
||||
@@ -109,6 +167,9 @@ func WithArtifactReader(reader ArtifactReader) Option {
|
||||
//
|
||||
// The source uses the same strict prompt YAML rules as configured prompt
|
||||
// directories, and prompt content_file paths resolve within this source.
|
||||
// fsys must be non-nil and root must be non-empty; otherwise NewEngine fails
|
||||
// with ErrInvalidConfig. This option replaces Config.PromptDir and earlier
|
||||
// prompt-source options.
|
||||
func WithPromptFS(fsys fs.FS, root string) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
if fsys == nil {
|
||||
@@ -125,7 +186,9 @@ func WithPromptFS(fsys fs.FS, root string) Option {
|
||||
|
||||
// WithPromptFile loads prompt definitions from the single prompt file at path.
|
||||
//
|
||||
// Relative prompt content_file paths resolve from the file's directory.
|
||||
// Relative prompt content_file paths resolve from the file's directory. path
|
||||
// must name an existing non-directory file when NewEngine applies the option.
|
||||
// This option replaces Config.PromptDir and earlier prompt-source options.
|
||||
func WithPromptFile(path string) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
fsys, root, err := fileSource(path)
|
||||
@@ -142,6 +205,10 @@ func WithPromptFile(path string) Option {
|
||||
//
|
||||
// Profiles from this source overlay built-in profiles. Profile YAML must use
|
||||
// api_key_env for environment-based credentials; raw API keys are rejected.
|
||||
// fsys must be non-nil and root must be non-empty; otherwise NewEngine fails
|
||||
// with ErrInvalidConfig. This option replaces Config.ProfileDir and earlier
|
||||
// file or FS profile-source options, but remains below WithProfiles in
|
||||
// precedence.
|
||||
func WithProfileFS(fsys fs.FS, root string) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
if fsys == nil {
|
||||
@@ -159,7 +226,10 @@ func WithProfileFS(fsys fs.FS, root string) Option {
|
||||
// WithProfileFile loads execution profiles from the single profile file at path.
|
||||
//
|
||||
// The profile overlays built-in profiles. Profile YAML must use api_key_env for
|
||||
// environment-based credentials; raw API keys are rejected.
|
||||
// environment-based credentials; raw API keys are rejected. path must name an
|
||||
// existing non-directory file when NewEngine applies the option. This option
|
||||
// replaces Config.ProfileDir and earlier file or FS profile-source options,
|
||||
// but remains below WithProfiles in precedence.
|
||||
func WithProfileFile(path string) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
fsys, root, err := fileSource(path)
|
||||
@@ -174,6 +244,11 @@ func WithProfileFile(path string) Option {
|
||||
|
||||
// WithProfiles configures in-memory profiles that take precedence over
|
||||
// configured profile files and built-in profiles.
|
||||
//
|
||||
// NewEngine validates and copies every profile. IDs must be unique within one
|
||||
// call. An invalid profile, duplicate ID, or unsupported ExtraParams value
|
||||
// makes construction fail with ErrInvalidConfig. Repeating WithProfiles
|
||||
// replaces the complete earlier in-memory set rather than merging it.
|
||||
func WithProfiles(profiles ...Profile) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
repo, err := newMemoryProfileRepository(profiles)
|
||||
@@ -189,7 +264,9 @@ func WithProfiles(profiles ...Profile) Option {
|
||||
// WithSchemaFS loads JSON Schema documents from fsys under root.
|
||||
//
|
||||
// Prompt schema_path values resolve within this source when schema validation
|
||||
// or structured output is requested.
|
||||
// or structured output is requested. fsys must be non-nil and root must be
|
||||
// non-empty; otherwise NewEngine fails with ErrInvalidConfig. This option
|
||||
// replaces Config.SchemaDir and earlier schema-source options.
|
||||
func WithSchemaFS(fsys fs.FS, root string) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
if fsys == nil {
|
||||
@@ -206,7 +283,9 @@ func WithSchemaFS(fsys fs.FS, root string) Option {
|
||||
|
||||
// WithSchemaFile loads JSON Schema documents from the single schema file at path.
|
||||
//
|
||||
// Prompt schema_path values refer to the file's base name.
|
||||
// Prompt schema_path values refer to the file's base name. path must name an
|
||||
// existing non-directory file when NewEngine applies the option. This option
|
||||
// replaces Config.SchemaDir and earlier schema-source options.
|
||||
func WithSchemaFile(path string) Option {
|
||||
return optionFunc(func(options *engineOptions) error {
|
||||
fsys, root, err := fileSource(path)
|
||||
@@ -220,6 +299,15 @@ func WithSchemaFile(path string) Option {
|
||||
}
|
||||
|
||||
// NewEngine constructs an Engine from configuration and options.
|
||||
//
|
||||
// Options are applied in order according to [Option]. PromptDir is required
|
||||
// unless a prompt-source option is present. Construction validates option
|
||||
// arguments and in-memory profiles but defers reading and validating prompt,
|
||||
// file-backed profile, and schema contents until Prepare or Run needs them.
|
||||
//
|
||||
// NewEngine returns an error matching ErrInvalidConfig for invalid
|
||||
// configuration or options. It does not perform model requests or require
|
||||
// credentials.
|
||||
func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
|
||||
var options engineOptions
|
||||
for _, opt := range opts {
|
||||
@@ -305,7 +393,22 @@ func fileSource(name string) (fs.FS, string, error) {
|
||||
return os.DirFS(dir), filepath.ToSlash(base), nil
|
||||
}
|
||||
|
||||
// Prepare resolves a prompt request without calling an LLM.
|
||||
// Prepare resolves and renders a prompt request without calling an LLM.
|
||||
//
|
||||
// Prepare selects the prompt and profile, resolves effective execution
|
||||
// settings and the output contract, loads and hashes inputs, loads structured
|
||||
// output schema metadata when required, and renders the session ID and
|
||||
// messages. The returned PreparedRun is owned by the caller and never contains
|
||||
// a resolved API-key value, model output, or validation result.
|
||||
//
|
||||
// A nil Engine returns an error matching ErrInvalidConfig. Request and
|
||||
// preparation failures may match ErrInvalidRequest, ErrPromptNotFound,
|
||||
// ErrPromptLoad, ErrProfileNotFound, ErrProfileLoad, ErrProfileRequired,
|
||||
// ErrAPIKeyEnvMissing, ErrArtifactLoad, ErrPromptRender, or ErrValidation as
|
||||
// applicable. Cancellation is passed to the active collaborator and is
|
||||
// reported in the applicable operation category; no general errors.Is
|
||||
// relationship to ctx.Err is promised. Prepare returns no partial result on
|
||||
// error.
|
||||
func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, error) {
|
||||
if e == nil || e.runner == nil {
|
||||
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
|
||||
@@ -323,7 +426,21 @@ func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, err
|
||||
return fromDomainPreparedRun(prepared), nil
|
||||
}
|
||||
|
||||
// Run executes a prompt request and returns the generated artifact and metadata.
|
||||
// Run prepares a request, invokes the configured LLMClient, and validates the
|
||||
// generated output.
|
||||
//
|
||||
// A content-validation failure is a successful run whose
|
||||
// RunResult.Validation has Status ValidationFailed. An inability to perform
|
||||
// validation returns an error matching ErrValidation and no partial result.
|
||||
// The public Engine does not perform output repair, so validation is
|
||||
// single-pass even when OutputContract.RepairAttempts is positive.
|
||||
//
|
||||
// Run can return every error category documented by [Engine.Prepare], plus
|
||||
// ErrLLMGenerate. Errors from injected clients remain available through
|
||||
// errors.Is. Cancellation is passed through the active collaborator and is
|
||||
// reported in the applicable operation category; no general errors.Is
|
||||
// relationship to ctx.Err is promised. A nil Engine returns ErrInvalidConfig.
|
||||
// Run returns no partial result on error.
|
||||
func (e *Engine) Run(ctx context.Context, req RunRequest) (*RunResult, error) {
|
||||
if e == nil || e.runner == nil {
|
||||
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
|
||||
|
||||
@@ -1020,6 +1020,7 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
|
||||
client promptkit.LLMClient
|
||||
schemaDir string
|
||||
want error
|
||||
notWant error
|
||||
}{
|
||||
{
|
||||
name: "invalid request",
|
||||
@@ -1032,6 +1033,7 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
|
||||
req: promptkit.RunRequest{PromptID: "missing.prompt"},
|
||||
client: &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}},
|
||||
want: promptkit.ErrPromptNotFound,
|
||||
notWant: promptkit.ErrPromptLoad,
|
||||
},
|
||||
{
|
||||
name: "profile not found",
|
||||
@@ -1044,6 +1046,7 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
|
||||
},
|
||||
client: &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}},
|
||||
want: promptkit.ErrProfileNotFound,
|
||||
notWant: promptkit.ErrProfileLoad,
|
||||
},
|
||||
{
|
||||
name: "artifact load",
|
||||
@@ -1114,6 +1117,9 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
|
||||
if !errors.Is(err, tc.want) {
|
||||
t.Fatalf("expected errors.Is(%v), got %v", tc.want, err)
|
||||
}
|
||||
if tc.notWant != nil && errors.Is(err, tc.notWant) {
|
||||
t.Fatalf("did not expect errors.Is(%v), got %v", tc.notWant, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -1699,6 +1705,16 @@ func TestEngineRunLayersTransportAndGenerationTimeouts(t *testing.T) {
|
||||
configTimeout: 5 * time.Second,
|
||||
wantRemainingAtRequest: 5 * time.Second,
|
||||
},
|
||||
{
|
||||
name: "zero configuration uses ten minute transport default",
|
||||
wantRemainingAtRequest: 10 * time.Minute,
|
||||
},
|
||||
{
|
||||
name: "negative configuration uses ten minute transport default",
|
||||
configTimeout: -2 * time.Second,
|
||||
suppliedClientTimeout: -3 * time.Second,
|
||||
wantRemainingAtRequest: 10 * time.Minute,
|
||||
},
|
||||
{
|
||||
name: "profile deadline is shorter than transport cap",
|
||||
suppliedClientTimeout: 6 * time.Second,
|
||||
|
||||
81
examples/go-library/run/main.go
Normal file
81
examples/go-library/run/main.go
Normal file
@@ -0,0 +1,81 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
)
|
||||
|
||||
type deterministicClient struct{}
|
||||
|
||||
func (deterministicClient) Generate(
|
||||
ctx context.Context,
|
||||
_ promptkit.GenerateRequest,
|
||||
) (*promptkit.GenerateResponse, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &promptkit.GenerateResponse{
|
||||
Content: "Ada finished the migration review.",
|
||||
Usage: promptkit.TokenUsage{
|
||||
PromptTokens: 12,
|
||||
CompletionTokens: 6,
|
||||
TotalTokens: 18,
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type summary struct {
|
||||
Output string `json:"output"`
|
||||
ValidationStatus promptkit.ValidationStatus `json:"validation_status"`
|
||||
IsValid bool `json:"is_valid"`
|
||||
Model string `json:"model"`
|
||||
TotalTokens int `json:"total_tokens"`
|
||||
}
|
||||
|
||||
func main() {
|
||||
engine, err := promptkit.NewEngine(
|
||||
promptkit.Config{},
|
||||
promptkit.WithPromptFile("examples/go-library/run/prompt.yaml"),
|
||||
promptkit.WithProfiles(promptkit.Profile{
|
||||
ID: "offline-example",
|
||||
Endpoint: "https://example.invalid/v1",
|
||||
Model: "offline-model",
|
||||
}),
|
||||
promptkit.WithLLMClient(deterministicClient{}),
|
||||
)
|
||||
if err != nil {
|
||||
exit(err)
|
||||
}
|
||||
|
||||
result, err := engine.Run(context.Background(), promptkit.RunRequest{
|
||||
PromptID: "example.run",
|
||||
Inputs: map[string]promptkit.ArtifactRef{
|
||||
"note": promptkit.Inline("Ada finished the migration review."),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
exit(err)
|
||||
}
|
||||
|
||||
encoder := json.NewEncoder(os.Stdout)
|
||||
encoder.SetIndent("", " ")
|
||||
if err := encoder.Encode(summary{
|
||||
Output: result.RawOutput,
|
||||
ValidationStatus: result.Validation.Status,
|
||||
IsValid: result.Validation.IsValid,
|
||||
Model: result.ModelName,
|
||||
TotalTokens: result.Usage.TotalTokens,
|
||||
}); err != nil {
|
||||
exit(err)
|
||||
}
|
||||
}
|
||||
|
||||
func exit(err error) {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
16
examples/go-library/run/prompt.yaml
Normal file
16
examples/go-library/run/prompt.yaml
Normal file
@@ -0,0 +1,16 @@
|
||||
id: example.run
|
||||
version: "1.0.0"
|
||||
default_profile: offline-example
|
||||
description: Run a prompt with a deterministic injected model client.
|
||||
inputs:
|
||||
- name: note
|
||||
required: true
|
||||
content_type: text/plain
|
||||
messages:
|
||||
- role: system
|
||||
content: Summarize the note in one sentence.
|
||||
- role: user
|
||||
content: '{{input "note"}}'
|
||||
output:
|
||||
format: text
|
||||
validation_mode: basic
|
||||
@@ -2,19 +2,23 @@ package promptkit
|
||||
|
||||
import "fmt"
|
||||
|
||||
// String returns a concise request summary without exposing direct API keys.
|
||||
// String returns a concise request summary without exposing the direct API key
|
||||
// or input and variable contents. Reflection-based formatting does not carry
|
||||
// this guarantee.
|
||||
func (r RunRequest) String() string {
|
||||
return r.redactedString()
|
||||
}
|
||||
|
||||
// GoString returns a concise request summary without exposing direct API keys.
|
||||
// GoString returns a concise request summary without exposing the direct API
|
||||
// key or input and variable contents. Reflection-based formatting does not
|
||||
// carry this guarantee.
|
||||
func (r RunRequest) GoString() string {
|
||||
return r.redactedString()
|
||||
}
|
||||
|
||||
func (r RunRequest) redactedString() string {
|
||||
return fmt.Sprintf(
|
||||
"promptkit.RunRequest{PromptID:%q PromptVersion:%q ProfileID:%q APIKeySet:%t Inputs:%d Vars:%d ExecutionSet:%t ValidationSet:%t Metadata:%d}",
|
||||
"promptkit.RunRequest{PromptID:%q PromptVersion:%q ProfileID:%q APIKeySet:%t Inputs:%d Vars:%d ExecutionSet:%t ValidationSet:%t}",
|
||||
r.PromptID,
|
||||
r.PromptVersion,
|
||||
r.ProfileID,
|
||||
@@ -23,18 +27,19 @@ func (r RunRequest) redactedString() string {
|
||||
len(r.Vars),
|
||||
r.Execution != nil,
|
||||
r.Validation != nil,
|
||||
len(r.Metadata),
|
||||
)
|
||||
}
|
||||
|
||||
// String returns a concise request summary without exposing direct API keys or
|
||||
// rendered prompt content.
|
||||
// rendered prompt content. Reflection-based formatting does not carry this
|
||||
// guarantee.
|
||||
func (r GenerateRequest) String() string {
|
||||
return r.redactedString()
|
||||
}
|
||||
|
||||
// GoString returns a concise request summary without exposing direct API keys or
|
||||
// rendered prompt content.
|
||||
// rendered prompt content. Reflection-based formatting does not carry this
|
||||
// guarantee.
|
||||
func (r GenerateRequest) GoString() string {
|
||||
return r.redactedString()
|
||||
}
|
||||
|
||||
@@ -68,7 +68,6 @@ type RunRequest struct {
|
||||
Vars map[string]string
|
||||
Execution *ExecutionTargetOverride
|
||||
Validation *OutputContract
|
||||
Metadata map[string]string
|
||||
}
|
||||
|
||||
// RunResult represents the complete result of a prompt execution run.
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"net/url"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
@@ -16,6 +17,8 @@ import (
|
||||
"github.com/santhosh-tekuri/jsonschema/v6"
|
||||
)
|
||||
|
||||
const jsonSchemaDraft2020 = "https://json-schema.org/draft/2020-12/schema"
|
||||
|
||||
// StandardValidator provides basic, JSON, and JSON Schema output validation.
|
||||
type StandardValidator struct {
|
||||
schemaBaseDir string
|
||||
@@ -121,7 +124,11 @@ func (v *StandardValidator) validateJSONSchema(instance any, schemaPath string)
|
||||
return nil, err
|
||||
}
|
||||
|
||||
compiler := jsonschema.NewCompiler()
|
||||
schemaRoot, err := v.schemaRoot()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
compiler := newSchemaCompiler(standardSchemaLoader{root: schemaRoot})
|
||||
schema, err := compiler.Compile(resolvedSchemaPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", resolvedSchemaPath, err)
|
||||
@@ -140,7 +147,10 @@ func (v *FSValidator) validateJSONSchema(instance any, schemaPath string) ([]str
|
||||
}
|
||||
|
||||
resourceURL := fsSchemaResourceURL(schemaName)
|
||||
compiler := jsonschema.NewCompiler()
|
||||
if err := validateSchemaDialect(schemaDoc); err != nil {
|
||||
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", schemaName, err)
|
||||
}
|
||||
compiler := newSchemaCompiler(fsSchemaLoader{fsys: v.fsys, root: filecatalog.CleanFSRoot(v.root)})
|
||||
if err := compiler.AddResource(resourceURL, schemaDoc); err != nil {
|
||||
return nil, fmt.Errorf("failed to register JSON schema %q: %w", schemaName, err)
|
||||
}
|
||||
@@ -184,6 +194,9 @@ func (v *StandardValidator) LoadSchemaDocument(ctx context.Context, schemaPath s
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
return nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
|
||||
}
|
||||
if err := validateSchemaDialect(doc); err != nil {
|
||||
return nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
|
||||
}
|
||||
return doc, nil
|
||||
}
|
||||
|
||||
@@ -206,12 +219,14 @@ func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error)
|
||||
return "", errors.New("schema path is required for json_schema validation")
|
||||
}
|
||||
|
||||
resolved := schemaPath
|
||||
if !filepath.IsAbs(schemaPath) {
|
||||
resolved = filepath.Join(v.schemaBaseDir, schemaPath)
|
||||
root, err := v.schemaRoot()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
resolved, err := containedFilesystemPath(root, schemaPath)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
resolved = filepath.Clean(resolved)
|
||||
if _, err := os.Stat(resolved); err != nil {
|
||||
return "", fmt.Errorf("failed to access schema file %q: %w", resolved, err)
|
||||
}
|
||||
@@ -219,6 +234,22 @@ func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error)
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
func (v *StandardValidator) schemaRoot() (string, error) {
|
||||
root := v.schemaBaseDir
|
||||
if strings.TrimSpace(root) == "" {
|
||||
root = "."
|
||||
}
|
||||
absolute, err := filepath.Abs(root)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to resolve schema source %q: %w", root, err)
|
||||
}
|
||||
resolved, err := filepath.EvalSymlinks(absolute)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to access schema source %q: %w", root, err)
|
||||
}
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
func (v *FSValidator) loadSchemaDocument(schemaPath string) (string, any, error) {
|
||||
resolved, err := v.resolveSchemaPath(schemaPath)
|
||||
if err != nil {
|
||||
@@ -234,6 +265,9 @@ func (v *FSValidator) loadSchemaDocument(schemaPath string) (string, any, error)
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
|
||||
}
|
||||
if err := validateSchemaDialect(doc); err != nil {
|
||||
return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
|
||||
}
|
||||
return resolved, doc, nil
|
||||
}
|
||||
|
||||
@@ -290,3 +324,131 @@ func cleanSchemaFSPath(schemaPath string) (string, error) {
|
||||
func fsSchemaResourceURL(schemaName string) string {
|
||||
return "promptkit-schema:///" + strings.TrimPrefix(path.Clean(schemaName), "/")
|
||||
}
|
||||
|
||||
func newSchemaCompiler(loader jsonschema.URLLoader) *jsonschema.Compiler {
|
||||
compiler := jsonschema.NewCompiler()
|
||||
compiler.DefaultDraft(jsonschema.Draft2020)
|
||||
compiler.UseLoader(loader)
|
||||
return compiler
|
||||
}
|
||||
|
||||
func validateSchemaDialect(doc any) error {
|
||||
object, ok := doc.(map[string]any)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
value, ok := object["$schema"]
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
dialect, ok := value.(string)
|
||||
if !ok {
|
||||
return errors.New("$schema must be a string")
|
||||
}
|
||||
if dialect != jsonSchemaDraft2020 && dialect != jsonSchemaDraft2020+"#" {
|
||||
return fmt.Errorf("unsupported JSON Schema dialect %q; expected %q", dialect, jsonSchemaDraft2020)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type standardSchemaLoader struct {
|
||||
root string
|
||||
}
|
||||
|
||||
func (l standardSchemaLoader) Load(resourceURL string) (any, error) {
|
||||
fileName, err := (jsonschema.FileLoader{}).ToFile(resourceURL)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("schema reference %q is not a contained file reference: %w", resourceURL, err)
|
||||
}
|
||||
resolved, err := containedFilesystemPath(l.root, fileName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return loadJSONSchemaFile(resolved)
|
||||
}
|
||||
|
||||
func containedFilesystemPath(root, name string) (string, error) {
|
||||
candidate := name
|
||||
if !filepath.IsAbs(candidate) {
|
||||
candidate = filepath.Join(root, candidate)
|
||||
}
|
||||
candidate, err := filepath.Abs(candidate)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to resolve schema path %q: %w", name, err)
|
||||
}
|
||||
candidate, err = filepath.EvalSymlinks(candidate)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to access schema file %q: %w", candidate, err)
|
||||
}
|
||||
relative, err := filepath.Rel(root, candidate)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to compare schema path %q with source root: %w", candidate, err)
|
||||
}
|
||||
if relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
|
||||
return "", fmt.Errorf("schema path %q escapes source root", name)
|
||||
}
|
||||
return candidate, nil
|
||||
}
|
||||
|
||||
func loadJSONSchemaFile(name string) (any, error) {
|
||||
raw, err := os.ReadFile(name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var doc any
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := validateSchemaDialect(doc); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return doc, nil
|
||||
}
|
||||
|
||||
type fsSchemaLoader struct {
|
||||
fsys fs.FS
|
||||
root string
|
||||
}
|
||||
|
||||
func (l fsSchemaLoader) Load(resourceURL string) (any, error) {
|
||||
parsed, err := url.Parse(resourceURL)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid schema reference %q: %w", resourceURL, err)
|
||||
}
|
||||
if parsed.Scheme != "promptkit-schema" || parsed.Host != "" {
|
||||
return nil, fmt.Errorf("schema reference %q is not allowed", resourceURL)
|
||||
}
|
||||
name, err := url.PathUnescape(strings.TrimPrefix(parsed.Path, "/"))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid schema reference %q: %w", resourceURL, err)
|
||||
}
|
||||
name = path.Clean(name)
|
||||
if l.root == "." {
|
||||
if strings.HasPrefix(name, "../") || name == ".." {
|
||||
return nil, fmt.Errorf("schema reference %q escapes source root", resourceURL)
|
||||
}
|
||||
} else if name != l.root && !strings.HasPrefix(name, l.root+"/") {
|
||||
return nil, fmt.Errorf("schema reference %q escapes source root", resourceURL)
|
||||
}
|
||||
|
||||
rootInfo, err := fs.Stat(l.fsys, l.root)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !rootInfo.IsDir() && name != l.root {
|
||||
return nil, fmt.Errorf("schema reference %q is outside the configured schema file", resourceURL)
|
||||
}
|
||||
|
||||
raw, err := fs.ReadFile(l.fsys, name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var doc any
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := validateSchemaDialect(doc); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return doc, nil
|
||||
}
|
||||
|
||||
@@ -2,8 +2,10 @@ package validate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
@@ -411,3 +413,138 @@ func TestFSValidatorLoadSchemaDocument(t *testing.T) {
|
||||
t.Fatalf("unexpected schema document: %#v", doc)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStandardValidatorJSONSchemaReferenceBoundaries(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(root, "child.json"), []byte(`{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"type": "string"
|
||||
}`), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
reference string
|
||||
wantError string
|
||||
writeOuter bool
|
||||
}{
|
||||
{name: "contained relative reference", reference: "child.json"},
|
||||
{name: "remote reference", reference: "https://example.test/schema.json", wantError: "not a contained file reference"},
|
||||
{name: "escaping reference", reference: "../outside.json", wantError: "escapes source root", writeOuter: true},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if tc.writeOuter {
|
||||
if err := os.WriteFile(filepath.Join(filepath.Dir(root), "outside.json"), []byte(`{"type":"string"}`), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
schema := `{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$ref": ` + strconv.Quote(tc.reference) + `
|
||||
}`
|
||||
if err := os.WriteFile(filepath.Join(root, "root.json"), []byte(schema), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
v := NewStandardValidator(root)
|
||||
result, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`"value"`)}, domain.OutputContract{
|
||||
ValidationMode: domain.ValidationJSONSchema,
|
||||
SchemaPath: "root.json",
|
||||
})
|
||||
if tc.wantError == "" {
|
||||
if err != nil || !result.IsValid {
|
||||
t.Fatalf("expected contained reference to validate, got result=%#v error=%v", result, err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err == nil || !strings.Contains(err.Error(), tc.wantError) {
|
||||
t.Fatalf("expected error containing %q, got %v", tc.wantError, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFSValidatorJSONSchemaReferenceBoundaries(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
reference string
|
||||
wantError string
|
||||
}{
|
||||
{name: "same document fragment", reference: "#/$defs/value"},
|
||||
{name: "contained relative reference", reference: "child.json"},
|
||||
{name: "remote reference", reference: "https://example.test/schema.json", wantError: "is not allowed"},
|
||||
{name: "escaping reference", reference: "../outside.json", wantError: "escapes source root"},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
rootSchema := `{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$defs": {"value": {"type": "string"}},
|
||||
"$ref": ` + strconv.Quote(tc.reference) + `
|
||||
}`
|
||||
v := NewFSValidator(fstest.MapFS{
|
||||
"schemas/root.json": &fstest.MapFile{Data: []byte(rootSchema)},
|
||||
"schemas/child.json": &fstest.MapFile{Data: []byte(`{"type":"string"}`)},
|
||||
"outside.json": &fstest.MapFile{Data: []byte(`{"type":"string"}`)},
|
||||
}, "schemas")
|
||||
result, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`"value"`)}, domain.OutputContract{
|
||||
ValidationMode: domain.ValidationJSONSchema,
|
||||
SchemaPath: "root.json",
|
||||
})
|
||||
if tc.wantError == "" {
|
||||
if err != nil || !result.IsValid {
|
||||
t.Fatalf("expected supported reference to validate, got result=%#v error=%v", result, err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err == nil || !strings.Contains(err.Error(), tc.wantError) {
|
||||
t.Fatalf("expected error containing %q, got %v", tc.wantError, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestJSONSchemaDialectIsDraft2020(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
dialect string
|
||||
wantError bool
|
||||
}{
|
||||
{name: "omitted uses supported default"},
|
||||
{name: "draft 2020-12", dialect: "https://json-schema.org/draft/2020-12/schema"},
|
||||
{name: "draft 7 rejected", dialect: "http://json-schema.org/draft-07/schema#", wantError: true},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
schema := map[string]any{"type": "object"}
|
||||
if tc.dialect != "" {
|
||||
schema["$schema"] = tc.dialect
|
||||
}
|
||||
data, err := json.Marshal(schema)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
v := NewFSValidator(fstest.MapFS{
|
||||
"schema.json": &fstest.MapFile{Data: data},
|
||||
}, ".")
|
||||
_, err = v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{}`)}, domain.OutputContract{
|
||||
ValidationMode: domain.ValidationJSONSchema,
|
||||
SchemaPath: "schema.json",
|
||||
})
|
||||
if tc.wantError {
|
||||
if err == nil || !strings.Contains(err.Error(), "unsupported JSON Schema dialect") {
|
||||
t.Fatalf("expected unsupported-dialect error, got %v", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("expected supported dialect, got %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
154
json.go
Normal file
154
json.go
Normal file
@@ -0,0 +1,154 @@
|
||||
package promptkit
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"time"
|
||||
)
|
||||
|
||||
// MarshalJSON implements json.Marshaler for PreparedRun. It uses RFC 3339
|
||||
// timestamps, integer duration_ms, and omits zero timing values.
|
||||
func (r PreparedRun) MarshalJSON() ([]byte, error) {
|
||||
var startTime, endTime *time.Time
|
||||
if !r.StartTime.IsZero() {
|
||||
startTime = &r.StartTime
|
||||
}
|
||||
if !r.EndTime.IsZero() {
|
||||
endTime = &r.EndTime
|
||||
}
|
||||
|
||||
var durationMS *int64
|
||||
if r.DurationMS != 0 {
|
||||
durationMS = &r.DurationMS
|
||||
}
|
||||
|
||||
return json.Marshal(struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
PromptHash string `json:"prompt_hash,omitempty"`
|
||||
SelectedProfileID string `json:"selected_profile_id"`
|
||||
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
||||
OutputContract OutputContract `json:"output_contract"`
|
||||
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
|
||||
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
||||
Messages []RenderedMessage `json:"messages"`
|
||||
StartTime *time.Time `json:"start_time,omitempty"`
|
||||
EndTime *time.Time `json:"end_time,omitempty"`
|
||||
DurationMS *int64 `json:"duration_ms,omitempty"`
|
||||
}{
|
||||
PromptID: r.PromptID,
|
||||
PromptVersion: r.PromptVersion,
|
||||
PromptHash: r.PromptHash,
|
||||
SelectedProfileID: r.SelectedProfileID,
|
||||
EffectiveModelParams: r.EffectiveModelParams,
|
||||
OutputContract: r.OutputContract,
|
||||
StructuredOutput: r.StructuredOutput,
|
||||
InputHashes: r.InputHashes,
|
||||
SessionID: r.SessionID,
|
||||
RenderedPromptHash: r.RenderedPromptHash,
|
||||
Messages: r.Messages,
|
||||
StartTime: startTime,
|
||||
EndTime: endTime,
|
||||
DurationMS: durationMS,
|
||||
})
|
||||
}
|
||||
|
||||
// MarshalJSON implements json.Marshaler for RunResult. It encodes Duration as
|
||||
// integer milliseconds in duration_ms and omits zero timing values.
|
||||
func (r RunResult) MarshalJSON() ([]byte, error) {
|
||||
var startTime, endTime *time.Time
|
||||
if !r.StartTime.IsZero() {
|
||||
startTime = &r.StartTime
|
||||
}
|
||||
if !r.EndTime.IsZero() {
|
||||
endTime = &r.EndTime
|
||||
}
|
||||
|
||||
var durationMS *int64
|
||||
if r.Duration != 0 {
|
||||
value := r.Duration.Milliseconds()
|
||||
durationMS = &value
|
||||
}
|
||||
|
||||
return json.Marshal(runResultJSON{
|
||||
RunID: r.RunID,
|
||||
Artifact: r.Artifact,
|
||||
RawOutput: r.RawOutput,
|
||||
Validation: r.Validation,
|
||||
PromptID: r.PromptID,
|
||||
PromptVersion: r.PromptVersion,
|
||||
PromptHash: r.PromptHash,
|
||||
RenderedPromptHash: r.RenderedPromptHash,
|
||||
SelectedProfileID: r.SelectedProfileID,
|
||||
ModelName: r.ModelName,
|
||||
Endpoint: r.Endpoint,
|
||||
EffectiveModelParams: r.EffectiveModelParams,
|
||||
InputHashes: r.InputHashes,
|
||||
Usage: r.Usage,
|
||||
StartTime: startTime,
|
||||
EndTime: endTime,
|
||||
DurationMS: durationMS,
|
||||
})
|
||||
}
|
||||
|
||||
// UnmarshalJSON implements json.Unmarshaler for RunResult. It decodes
|
||||
// duration_ms into Duration with millisecond precision.
|
||||
func (r *RunResult) UnmarshalJSON(data []byte) error {
|
||||
var wire runResultJSON
|
||||
if err := json.Unmarshal(data, &wire); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
*r = RunResult{
|
||||
RunID: wire.RunID,
|
||||
Artifact: wire.Artifact,
|
||||
RawOutput: wire.RawOutput,
|
||||
Validation: wire.Validation,
|
||||
PromptID: wire.PromptID,
|
||||
PromptVersion: wire.PromptVersion,
|
||||
PromptHash: wire.PromptHash,
|
||||
RenderedPromptHash: wire.RenderedPromptHash,
|
||||
SelectedProfileID: wire.SelectedProfileID,
|
||||
ModelName: wire.ModelName,
|
||||
Endpoint: wire.Endpoint,
|
||||
EffectiveModelParams: wire.EffectiveModelParams,
|
||||
InputHashes: wire.InputHashes,
|
||||
Usage: wire.Usage,
|
||||
Duration: time.Duration(valueOrZero(wire.DurationMS)) * time.Millisecond,
|
||||
}
|
||||
if wire.StartTime != nil {
|
||||
r.StartTime = *wire.StartTime
|
||||
}
|
||||
if wire.EndTime != nil {
|
||||
r.EndTime = *wire.EndTime
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type runResultJSON struct {
|
||||
RunID string `json:"run_id"`
|
||||
Artifact Artifact `json:"artifact"`
|
||||
RawOutput string `json:"raw_output"`
|
||||
Validation ValidationResult `json:"validation"`
|
||||
PromptID string `json:"prompt_id"`
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
PromptHash string `json:"prompt_hash,omitempty"`
|
||||
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
||||
SelectedProfileID string `json:"selected_profile_id"`
|
||||
ModelName string `json:"model_name"`
|
||||
Endpoint string `json:"endpoint"`
|
||||
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
||||
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
||||
Usage TokenUsage `json:"usage"`
|
||||
StartTime *time.Time `json:"start_time,omitempty"`
|
||||
EndTime *time.Time `json:"end_time,omitempty"`
|
||||
DurationMS *int64 `json:"duration_ms,omitempty"`
|
||||
}
|
||||
|
||||
func valueOrZero(value *int64) int64 {
|
||||
if value == nil {
|
||||
return 0
|
||||
}
|
||||
return *value
|
||||
}
|
||||
@@ -16,6 +16,10 @@ import (
|
||||
// It does not register global state, maintain a model catalog, or resolve
|
||||
// credentials. If APIKeyRequired is true, callers satisfy it with
|
||||
// RunRequest.APIKey. Raw API keys do not belong in profiles.
|
||||
//
|
||||
// The function copies the ExtraParams map itself but does not recursively copy
|
||||
// nested values. Validation and a deep copy occur when NewEngine applies a
|
||||
// WithProfiles option containing the returned Profile.
|
||||
func OpenAICompatibleProfile(cfg OpenAICompatibleProfileConfig) Profile {
|
||||
return Profile{
|
||||
ID: cfg.ID,
|
||||
|
||||
377
public_contract_test.go
Normal file
377
public_contract_test.go
Normal file
@@ -0,0 +1,377 @@
|
||||
package promptkit_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
)
|
||||
|
||||
func TestPreparedRunJSONOmitsZeroTimingValues(t *testing.T) {
|
||||
payload, err := json.Marshal(promptkit.PreparedRun{})
|
||||
if err != nil {
|
||||
t.Fatalf("marshal prepared run: %v", err)
|
||||
}
|
||||
for _, field := range []string{"start_time", "end_time", "duration_ms"} {
|
||||
if strings.Contains(string(payload), `"`+field+`"`) {
|
||||
t.Fatalf("expected zero %s to be omitted, got %s", field, payload)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPreparedRunJSONTimingRoundTrips(t *testing.T) {
|
||||
start := time.Date(2026, time.July, 29, 12, 0, 0, 0, time.UTC)
|
||||
prepared := promptkit.PreparedRun{
|
||||
PromptID: "prompt",
|
||||
StartTime: start,
|
||||
EndTime: start.Add(1250 * time.Millisecond),
|
||||
DurationMS: 1250,
|
||||
}
|
||||
|
||||
payload, err := json.Marshal(prepared)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal prepared run: %v", err)
|
||||
}
|
||||
var decoded promptkit.PreparedRun
|
||||
if err := json.Unmarshal(payload, &decoded); err != nil {
|
||||
t.Fatalf("unmarshal prepared run: %v", err)
|
||||
}
|
||||
if decoded.DurationMS != prepared.DurationMS ||
|
||||
!decoded.StartTime.Equal(prepared.StartTime) ||
|
||||
!decoded.EndTime.Equal(prepared.EndTime) {
|
||||
t.Fatalf("timing values did not round trip: got %#v, want %#v", decoded, prepared)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunResultJSONUsesMillisecondsAndRoundTrips(t *testing.T) {
|
||||
start := time.Date(2026, time.July, 29, 12, 0, 0, 0, time.UTC)
|
||||
result := promptkit.RunResult{
|
||||
RunID: "opaque-run-id",
|
||||
Artifact: promptkit.Artifact{Name: "output", ContentType: "text/plain", Body: []byte("ok")},
|
||||
StartTime: start,
|
||||
EndTime: start.Add(1500 * time.Millisecond),
|
||||
Duration: 1500 * time.Millisecond,
|
||||
}
|
||||
|
||||
payload, err := json.Marshal(result)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal run result: %v", err)
|
||||
}
|
||||
var object map[string]any
|
||||
if err := json.Unmarshal(payload, &object); err != nil {
|
||||
t.Fatalf("decode run result JSON: %v", err)
|
||||
}
|
||||
if got := object["duration_ms"]; got != float64(1500) {
|
||||
t.Fatalf("expected duration_ms=1500, got %#v in %s", got, payload)
|
||||
}
|
||||
if _, exists := object["duration"]; exists {
|
||||
t.Fatalf("unexpected nanosecond duration field in %s", payload)
|
||||
}
|
||||
artifact, ok := object["artifact"].(map[string]any)
|
||||
if !ok || artifact["content_type"] != "text/plain" {
|
||||
t.Fatalf("expected stable artifact JSON fields, got %#v", object["artifact"])
|
||||
}
|
||||
|
||||
var decoded promptkit.RunResult
|
||||
if err := json.Unmarshal(payload, &decoded); err != nil {
|
||||
t.Fatalf("unmarshal run result: %v", err)
|
||||
}
|
||||
if decoded.Duration != result.Duration || !decoded.StartTime.Equal(result.StartTime) || !decoded.EndTime.Equal(result.EndTime) {
|
||||
t.Fatalf("timing values did not round trip: got %#v, want %#v", decoded, result)
|
||||
}
|
||||
|
||||
payload, err = json.Marshal(promptkit.RunResult{})
|
||||
if err != nil {
|
||||
t.Fatalf("marshal zero run result: %v", err)
|
||||
}
|
||||
for _, field := range []string{"start_time", "end_time", "duration_ms"} {
|
||||
if strings.Contains(string(payload), `"`+field+`"`) {
|
||||
t.Fatalf("expected zero %s to be omitted, got %s", field, payload)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestEngineValidationIsSinglePass(t *testing.T) {
|
||||
client := &fakeLLMClient{
|
||||
response: &promptkit.GenerateResponse{Content: "not-json"},
|
||||
}
|
||||
engine := newContractEngineWithOptions(t, frameworkSchemaDir, promptkit.WithLLMClient(client))
|
||||
|
||||
result, err := engine.Run(context.Background(), promptkit.RunRequest{
|
||||
PromptID: frameworkMarkdownSummaryPromptID,
|
||||
Inputs: map[string]promptkit.ArtifactRef{
|
||||
"transcript": promptkit.Inline("Rin opens the gate."),
|
||||
"glossary": promptkit.Inline("gate: A guarded passage."),
|
||||
},
|
||||
Validation: &promptkit.OutputContract{
|
||||
Format: promptkit.FormatJSON,
|
||||
ValidationMode: promptkit.ValidationJSON,
|
||||
RepairAttempts: 3,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("run with failed content validation: %v", err)
|
||||
}
|
||||
if result.Validation.Status != promptkit.ValidationFailed ||
|
||||
result.Validation.RepairAttempts != 0 {
|
||||
t.Fatalf("expected failed single-pass validation, got %#v", result.Validation)
|
||||
}
|
||||
if len(client.requests) != 1 {
|
||||
t.Fatalf("expected one model generation, got %d", len(client.requests))
|
||||
}
|
||||
}
|
||||
|
||||
func TestRepeatedOptionsUseLastValueInEachCategory(t *testing.T) {
|
||||
profile := promptkit.Profile{ID: "profile", Endpoint: "http://example.test/v1", Model: "model"}
|
||||
|
||||
t.Run("prompt source", func(t *testing.T) {
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(contractPromptFS("first", "profile", "first"), "."),
|
||||
promptkit.WithPromptFS(contractPromptFS("second", "profile", "second"), "."),
|
||||
promptkit.WithProfiles(profile),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "second"})
|
||||
if err != nil {
|
||||
t.Fatalf("prepare from last prompt source: %v", err)
|
||||
}
|
||||
if prepared.Messages[0].Content != "second" {
|
||||
t.Fatalf("expected last prompt source, got %#v", prepared.Messages)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("profile source", func(t *testing.T) {
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
|
||||
promptkit.WithProfileFS(contractProfileFS("profile", "first-model"), "."),
|
||||
promptkit.WithProfileFS(contractProfileFS("profile", "second-model"), "."),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
|
||||
if err != nil {
|
||||
t.Fatalf("prepare from last profile source: %v", err)
|
||||
}
|
||||
if prepared.EffectiveModelParams.Model != "second-model" {
|
||||
t.Fatalf("expected last profile source, got %q", prepared.EffectiveModelParams.Model)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("in-memory profiles", func(t *testing.T) {
|
||||
first := profile
|
||||
first.Model = "first-model"
|
||||
second := profile
|
||||
second.Model = "second-model"
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
|
||||
promptkit.WithProfiles(first),
|
||||
promptkit.WithProfiles(second),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
|
||||
if err != nil {
|
||||
t.Fatalf("prepare from last in-memory profile option: %v", err)
|
||||
}
|
||||
if prepared.EffectiveModelParams.Model != "second-model" {
|
||||
t.Fatalf("expected last in-memory profiles, got %q", prepared.EffectiveModelParams.Model)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("schema source", func(t *testing.T) {
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(contractSchemaPromptFS(), "."),
|
||||
promptkit.WithProfiles(profile),
|
||||
promptkit.WithSchemaFS(contractSchemaFS("first"), "."),
|
||||
promptkit.WithSchemaFS(contractSchemaFS("second"), "."),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "schema-prompt"})
|
||||
if err != nil {
|
||||
t.Fatalf("prepare from last schema source: %v", err)
|
||||
}
|
||||
schema := prepared.StructuredOutput.JSONSchema.Schema.(map[string]any)
|
||||
if schema["title"] != "second" {
|
||||
t.Fatalf("expected last schema source, got %#v", schema)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("model client", func(t *testing.T) {
|
||||
var firstCalls, secondCalls atomic.Int64
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
|
||||
promptkit.WithProfiles(profile),
|
||||
promptkit.WithLLMClient(countingLLMClient{calls: &firstCalls}),
|
||||
promptkit.WithLLMClient(countingLLMClient{calls: &secondCalls}),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
if _, err := engine.Run(context.Background(), promptkit.RunRequest{PromptID: "prompt"}); err != nil {
|
||||
t.Fatalf("run with last model client: %v", err)
|
||||
}
|
||||
if firstCalls.Load() != 0 || secondCalls.Load() != 1 {
|
||||
t.Fatalf("expected only last client call, got first=%d second=%d", firstCalls.Load(), secondCalls.Load())
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("artifact reader", func(t *testing.T) {
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(contractInputPromptFS(), "."),
|
||||
promptkit.WithProfiles(profile),
|
||||
promptkit.WithArtifactReader(fixedArtifactReader("first")),
|
||||
promptkit.WithArtifactReader(fixedArtifactReader("second")),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||
PromptID: "input-prompt",
|
||||
Inputs: map[string]promptkit.ArtifactRef{"input": promptkit.Inline("ignored")},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("prepare with last artifact reader: %v", err)
|
||||
}
|
||||
if prepared.Messages[0].Content != "second" {
|
||||
t.Fatalf("expected last artifact reader, got %#v", prepared.Messages)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestEngineSupportsConcurrentPrepareAndRun(t *testing.T) {
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
|
||||
promptkit.WithProfiles(promptkit.Profile{
|
||||
ID: "profile",
|
||||
Endpoint: "http://example.test/v1",
|
||||
Model: "model",
|
||||
}),
|
||||
promptkit.WithLLMClient(countingLLMClient{}),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
|
||||
const calls = 40
|
||||
errs := make(chan error, calls)
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < calls; i++ {
|
||||
wg.Add(1)
|
||||
go func(run bool) {
|
||||
defer wg.Done()
|
||||
request := promptkit.RunRequest{PromptID: "prompt"}
|
||||
if run {
|
||||
_, err := engine.Run(context.Background(), request)
|
||||
errs <- err
|
||||
return
|
||||
}
|
||||
_, err := engine.Prepare(context.Background(), request)
|
||||
errs <- err
|
||||
}(i%2 == 0)
|
||||
}
|
||||
wg.Wait()
|
||||
close(errs)
|
||||
for err := range errs {
|
||||
if err != nil {
|
||||
t.Fatalf("concurrent call failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type countingLLMClient struct {
|
||||
calls *atomic.Int64
|
||||
}
|
||||
|
||||
func (c countingLLMClient) Generate(context.Context, promptkit.GenerateRequest) (*promptkit.GenerateResponse, error) {
|
||||
if c.calls != nil {
|
||||
c.calls.Add(1)
|
||||
}
|
||||
return &promptkit.GenerateResponse{Content: "ok"}, nil
|
||||
}
|
||||
|
||||
type fixedArtifactReader string
|
||||
|
||||
func (r fixedArtifactReader) Read(context.Context, promptkit.ArtifactRef) (*promptkit.Artifact, error) {
|
||||
return &promptkit.Artifact{Body: []byte(r)}, nil
|
||||
}
|
||||
|
||||
func contractPromptFS(id, profileID, message string) fstest.MapFS {
|
||||
return fstest.MapFS{
|
||||
"prompt.yaml": &fstest.MapFile{Data: []byte(fmt.Sprintf(`id: %s
|
||||
version: "1"
|
||||
default_profile: %s
|
||||
messages:
|
||||
- role: user
|
||||
content: %q
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`, id, profileID, message))},
|
||||
}
|
||||
}
|
||||
|
||||
func contractInputPromptFS() fstest.MapFS {
|
||||
return fstest.MapFS{
|
||||
"prompt.yaml": &fstest.MapFile{Data: []byte(`id: input-prompt
|
||||
version: "1"
|
||||
default_profile: profile
|
||||
inputs:
|
||||
- name: input
|
||||
required: true
|
||||
messages:
|
||||
- role: user
|
||||
content: '{{input "input"}}'
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`)},
|
||||
}
|
||||
}
|
||||
|
||||
func contractProfileFS(id, model string) fstest.MapFS {
|
||||
return fstest.MapFS{
|
||||
"profile.yaml": &fstest.MapFile{Data: []byte(fmt.Sprintf(`id: %s
|
||||
endpoint: http://example.test/v1
|
||||
model: %s
|
||||
`, id, model))},
|
||||
}
|
||||
}
|
||||
|
||||
func contractSchemaPromptFS() fstest.MapFS {
|
||||
return fstest.MapFS{
|
||||
"prompt.yaml": &fstest.MapFile{Data: []byte(`id: schema-prompt
|
||||
version: "1"
|
||||
default_profile: profile
|
||||
messages:
|
||||
- role: user
|
||||
content: message
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: schema.json
|
||||
`)},
|
||||
}
|
||||
}
|
||||
|
||||
func contractSchemaFS(title string) fstest.MapFS {
|
||||
return fstest.MapFS{
|
||||
"schema.json": &fstest.MapFile{Data: []byte(fmt.Sprintf(`{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": %q,
|
||||
"type": "object"
|
||||
}`, title))},
|
||||
}
|
||||
}
|
||||
367
types.go
367
types.go
@@ -5,160 +5,318 @@ import (
|
||||
"time"
|
||||
)
|
||||
|
||||
// ArtifactRefType defines how an artifact is referenced.
|
||||
// ArtifactRefType identifies how an [ArtifactRef] supplies content.
|
||||
type ArtifactRefType string
|
||||
|
||||
const (
|
||||
// ArtifactRefInline selects ArtifactRef.Body as the content.
|
||||
ArtifactRefInline ArtifactRefType = "inline"
|
||||
// ArtifactRefFile selects the filesystem path in ArtifactRef.URI.
|
||||
ArtifactRefFile ArtifactRefType = "file"
|
||||
)
|
||||
|
||||
// OutputFormat defines the desired output format.
|
||||
// OutputFormat identifies the media format of generated output.
|
||||
// OutputFormat has a stable JSON string representation.
|
||||
type OutputFormat string
|
||||
|
||||
const (
|
||||
// FormatText identifies plain-text output.
|
||||
FormatText OutputFormat = "text"
|
||||
// FormatMarkdown identifies Markdown output.
|
||||
FormatMarkdown OutputFormat = "markdown"
|
||||
// FormatJSON identifies JSON output.
|
||||
FormatJSON OutputFormat = "json"
|
||||
)
|
||||
|
||||
// ValidationMode defines the output validation strategy.
|
||||
// ValidationMode identifies how generated output is checked.
|
||||
// ValidationMode has a stable JSON string representation.
|
||||
type ValidationMode string
|
||||
|
||||
const (
|
||||
// ValidationNone skips content validation.
|
||||
ValidationNone ValidationMode = "none"
|
||||
// ValidationBasic requires non-empty output.
|
||||
ValidationBasic ValidationMode = "basic"
|
||||
// ValidationJSON requires syntactically valid JSON.
|
||||
ValidationJSON ValidationMode = "json"
|
||||
// ValidationJSONSchema requires JSON that satisfies OutputContract.SchemaPath.
|
||||
ValidationJSONSchema ValidationMode = "json_schema"
|
||||
)
|
||||
|
||||
// ValidationStatus defines the result of a validation check.
|
||||
// ValidationStatus identifies the completed state of an output check.
|
||||
// ValidationStatus has a stable JSON string representation.
|
||||
type ValidationStatus string
|
||||
|
||||
const (
|
||||
// ValidationPassed means the generated output satisfied its contract.
|
||||
ValidationPassed ValidationStatus = "passed"
|
||||
// ValidationFailed means validation completed and rejected the generated
|
||||
// output. Engine.Run returns this status in a result, not as an error.
|
||||
ValidationFailed ValidationStatus = "failed"
|
||||
// ValidationSkipped means ValidationNone selected no content check.
|
||||
ValidationSkipped ValidationStatus = "skipped"
|
||||
)
|
||||
|
||||
// CacheControlType defines provider cache behavior for prompt content.
|
||||
// CacheControlType identifies provider cache behavior for prompt content.
|
||||
// CacheControlType has a stable JSON string representation.
|
||||
type CacheControlType string
|
||||
|
||||
const (
|
||||
// CacheControlEphemeral requests provider-defined ephemeral caching.
|
||||
CacheControlEphemeral CacheControlType = "ephemeral"
|
||||
)
|
||||
|
||||
// StructuredOutputType identifies provider-level structured output modes.
|
||||
// StructuredOutputType has a stable JSON string representation.
|
||||
type StructuredOutputType string
|
||||
|
||||
const (
|
||||
// StructuredOutputJSONSchema supplies JSON Schema response constraints.
|
||||
StructuredOutputJSONSchema StructuredOutputType = "json_schema"
|
||||
)
|
||||
|
||||
// RunRequest represents a request to prepare or run a single prompt.
|
||||
// RunRequest selects one prompt execution. It has no stable JSON
|
||||
// representation.
|
||||
//
|
||||
// Prepare and Run copy the request's maps, pointers, and nested
|
||||
// JSON-compatible values before using them. The caller may mutate the request
|
||||
// after either method returns.
|
||||
type RunRequest struct {
|
||||
// PromptID is the required non-empty prompt identifier.
|
||||
PromptID string
|
||||
// PromptVersion optionally selects one version of PromptID. When empty, the
|
||||
// prompt source must contain exactly one matching version.
|
||||
PromptVersion string
|
||||
// ProfileID selects an execution profile. When empty, the prompt's default
|
||||
// profile is used; if both are empty, the error matches ErrProfileRequired
|
||||
// and ErrInvalidRequest.
|
||||
ProfileID string
|
||||
// APIKey is a request-scoped direct credential. It takes precedence over
|
||||
// APIKeyEnv, is passed to the selected LLMClient, and is never included in
|
||||
// prepared values, results, hashes, JSON, String, or GoString output.
|
||||
APIKey string `json:"-"`
|
||||
// Inputs maps prompt input names to references. A nil or empty map is valid
|
||||
// only when the selected prompt and its templates require no inputs.
|
||||
Inputs map[string]ArtifactRef
|
||||
// Vars supplies Go-template data for messages and the session ID. Nil and
|
||||
// empty maps are equivalent.
|
||||
Vars map[string]string
|
||||
// Execution optionally overrides individual profile execution settings.
|
||||
// Nil uses the selected profile over framework defaults.
|
||||
Execution *ExecutionTargetOverride
|
||||
// Validation optionally replaces the prompt's complete output contract. It
|
||||
// does not merge individual fields. Nil uses the prompt contract.
|
||||
Validation *OutputContract
|
||||
Metadata map[string]string
|
||||
}
|
||||
|
||||
// PreparedRun contains prepared prompt execution state. It does not include
|
||||
// resolved API key values, model output, validation results, or internal target
|
||||
// presence metadata.
|
||||
// presence metadata. PreparedRun has a stable JSON representation.
|
||||
//
|
||||
// All maps, slices, pointers, and schema values are caller-owned copies. JSON
|
||||
// timestamps use RFC 3339 and zero timing values are omitted. Hash formats are
|
||||
// opaque.
|
||||
type PreparedRun struct {
|
||||
// PromptID is the selected prompt identifier.
|
||||
PromptID string `json:"prompt_id"`
|
||||
// PromptVersion is the selected prompt version.
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
// PromptHash is an opaque equality value for the selected definition.
|
||||
PromptHash string `json:"prompt_hash,omitempty"`
|
||||
// SelectedProfileID is the explicit request profile or prompt default that
|
||||
// supplied execution settings.
|
||||
SelectedProfileID string `json:"selected_profile_id"`
|
||||
// EffectiveModelParams contains framework defaults overlaid by the selected
|
||||
// profile and then request overrides. It excludes resolved API-key values.
|
||||
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
||||
// OutputContract is the complete effective output contract.
|
||||
OutputContract OutputContract `json:"output_contract"`
|
||||
// StructuredOutput is non-nil for JSON Schema validation and contains the
|
||||
// provider-facing response constraint passed to an LLM client.
|
||||
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
|
||||
// InputHashes maps every supplied input name to its opaque artifact hash.
|
||||
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
||||
// SessionID is the trimmed rendered session identifier, if any.
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
// RenderedPromptHash is an opaque equality value for SessionID and Messages.
|
||||
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
||||
// Messages are the rendered messages that Run passes to the LLM client.
|
||||
Messages []RenderedMessage `json:"messages"`
|
||||
// StartTime is the UTC time at which preparation began.
|
||||
StartTime time.Time `json:"start_time,omitempty"`
|
||||
// EndTime is the UTC time at which preparation completed.
|
||||
EndTime time.Time `json:"end_time,omitempty"`
|
||||
// DurationMS is preparation elapsed time in integer milliseconds. JSON uses
|
||||
// duration_ms and omits a zero value.
|
||||
DurationMS int64 `json:"duration_ms,omitempty"`
|
||||
}
|
||||
|
||||
// RunResult contains generated output, validation state, and run metadata.
|
||||
// RunResult has a stable JSON representation and round-trips its Duration
|
||||
// through the duration_ms JSON field.
|
||||
//
|
||||
// All maps, slices, and nested values are caller-owned copies. JSON timestamps
|
||||
// use RFC 3339 and zero timing values are omitted. Run IDs and hash formats are
|
||||
// opaque.
|
||||
type RunResult struct {
|
||||
// RunID is an opaque identifier for this invocation.
|
||||
RunID string `json:"run_id"`
|
||||
// Artifact contains the generated output and derived metadata.
|
||||
Artifact Artifact `json:"artifact"`
|
||||
// RawOutput is the exact generated content before artifact classification
|
||||
// and validation.
|
||||
RawOutput string `json:"raw_output"`
|
||||
// Validation records the completed content check.
|
||||
Validation ValidationResult `json:"validation"`
|
||||
// PromptID is the selected prompt identifier.
|
||||
PromptID string `json:"prompt_id"`
|
||||
// PromptVersion is the selected prompt version.
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
// PromptHash is the same opaque definition equality value exposed by
|
||||
// PreparedRun.
|
||||
PromptHash string `json:"prompt_hash,omitempty"`
|
||||
// RenderedPromptHash is the same opaque rendered-prompt equality value
|
||||
// computed during preparation.
|
||||
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
||||
// SelectedProfileID identifies the profile used for execution.
|
||||
SelectedProfileID string `json:"selected_profile_id"`
|
||||
// ModelName is the effective model name and equals
|
||||
// EffectiveModelParams.Model.
|
||||
ModelName string `json:"model_name"`
|
||||
// Endpoint is the effective base endpoint and equals
|
||||
// EffectiveModelParams.Endpoint.
|
||||
Endpoint string `json:"endpoint"`
|
||||
// EffectiveModelParams contains the settings supplied to the LLM client,
|
||||
// excluding resolved API-key values.
|
||||
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
||||
// InputHashes are the opaque input equality values computed during
|
||||
// preparation.
|
||||
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
||||
// Usage is the token accounting reported by the LLM client.
|
||||
Usage TokenUsage `json:"usage"`
|
||||
// StartTime is the UTC time immediately before preparation begins.
|
||||
StartTime time.Time `json:"start_time,omitempty"`
|
||||
// EndTime is the UTC time after generation and validation complete.
|
||||
EndTime time.Time `json:"end_time,omitempty"`
|
||||
Duration time.Duration `json:"duration,omitempty"`
|
||||
// Duration covers preparation, generation, and validation. JSON represents
|
||||
// it as integer milliseconds in duration_ms and omits a zero value.
|
||||
Duration time.Duration `json:"-"`
|
||||
}
|
||||
|
||||
// ArtifactRef represents a reference to prompt input content.
|
||||
// ArtifactRef identifies prompt input content. It has no stable JSON
|
||||
// representation. Prefer [File], [Inline], or [InlineWithURI] to construct one.
|
||||
type ArtifactRef struct {
|
||||
// Type must be ArtifactRefInline or ArtifactRefFile.
|
||||
Type ArtifactRefType
|
||||
// URI is the file path for ArtifactRefFile and optional provenance metadata
|
||||
// for ArtifactRefInline.
|
||||
URI string
|
||||
// Body is the content for ArtifactRefInline and is ignored for
|
||||
// ArtifactRefFile.
|
||||
Body string
|
||||
}
|
||||
|
||||
// Artifact represents loaded artifact content.
|
||||
// Artifact represents loaded or generated content and has a stable JSON
|
||||
// representation. Body uses encoding/json's base64 representation for []byte.
|
||||
type Artifact struct {
|
||||
Name string
|
||||
ContentType string
|
||||
Body []byte
|
||||
URI string
|
||||
Size int64
|
||||
Hash string
|
||||
// Name is artifact metadata. During input preparation the engine fills an
|
||||
// empty reader-supplied name with the request input-map key.
|
||||
Name string `json:"name"`
|
||||
// ContentType is the media type reported by the reader or derived for
|
||||
// generated output.
|
||||
ContentType string `json:"content_type"`
|
||||
// Body is the artifact content. Engine boundaries copy this slice.
|
||||
Body []byte `json:"body"`
|
||||
// URI is optional source or result provenance metadata.
|
||||
URI string `json:"uri"`
|
||||
// Size is content-size metadata in bytes.
|
||||
Size int64 `json:"size"`
|
||||
// Hash is an opaque content equality value when the producing reader
|
||||
// supplies one. Its format and algorithm are not API contracts.
|
||||
Hash string `json:"hash"`
|
||||
}
|
||||
|
||||
// ArtifactReader resolves a prompt input reference into its content.
|
||||
//
|
||||
// Readers are responsible for supplying artifact metadata. The engine assigns
|
||||
// an input-map name only when the returned artifact name is empty.
|
||||
// Read may be called concurrently. It must honor ctx cancellation to make
|
||||
// Prepare and Run responsive to cancellation. The engine passes a copied ref
|
||||
// and immediately copies the returned Artifact.Body; it does not retain either
|
||||
// value. Readers supply artifact metadata, and the engine assigns an input-map
|
||||
// name only when the returned artifact name is empty.
|
||||
//
|
||||
// An injected reader owns any application-specific path containment,
|
||||
// authorization, content-size, and content-type policy. It must protect
|
||||
// sensitive references and bodies in its logging and in any copies it retains.
|
||||
// It may reuse or mutate the returned artifact and body after Read returns.
|
||||
//
|
||||
// Returning a non-nil error makes the engine return an error matching
|
||||
// ErrArtifactLoad while preserving the reader error through errors.Is.
|
||||
// Returning a nil artifact with a nil error also produces ErrArtifactLoad.
|
||||
type ArtifactReader interface {
|
||||
Read(context.Context, ArtifactRef) (*Artifact, error)
|
||||
}
|
||||
|
||||
// ExecutionTarget represents effective model runtime settings.
|
||||
// ExecutionTarget represents effective model runtime settings and has a stable
|
||||
// JSON representation. It never exposes a resolved API-key value.
|
||||
type ExecutionTarget struct {
|
||||
// Endpoint is the model-provider base URL.
|
||||
Endpoint string `json:"endpoint"`
|
||||
// Model is the provider model identifier.
|
||||
Model string `json:"model"`
|
||||
// Temperature is the effective sampling temperature from 0 through 2.
|
||||
Temperature float64 `json:"temperature"`
|
||||
// MaxTokens is the non-negative effective output-token limit. Zero leaves
|
||||
// the limit unspecified to compatible providers unless it was an explicit
|
||||
// request override.
|
||||
MaxTokens int `json:"max_tokens"`
|
||||
// TopP is the effective nucleus-sampling value from 0 through 1.
|
||||
TopP float64 `json:"top_p"`
|
||||
// TimeoutSeconds is the non-negative per-generation deadline. Zero disables
|
||||
// this deadline without disabling caller cancellation or the transport cap.
|
||||
TimeoutSeconds int `json:"timeout_seconds"`
|
||||
// ServiceTier is an optional provider-specific request tier.
|
||||
ServiceTier string `json:"service_tier"`
|
||||
// ReasoningEffort is an optional provider-specific reasoning setting.
|
||||
ReasoningEffort string `json:"reasoning_effort"`
|
||||
// APIKeyEnv is an environment-variable name, not its credential value.
|
||||
APIKeyEnv string `json:"api_key_env"`
|
||||
// ExtraParams contains copied JSON-compatible provider parameters.
|
||||
ExtraParams map[string]any `json:"extra_params"`
|
||||
}
|
||||
|
||||
// ExecutionTargetOverride represents per-request runtime setting overrides.
|
||||
// ExecutionTargetOverride represents per-request runtime setting overrides and
|
||||
// has no stable JSON representation.
|
||||
//
|
||||
// Non-empty string fields replace profile values. Non-nil numeric pointers
|
||||
// replace profile values and preserve explicit zero. A non-empty ExtraParams
|
||||
// map replaces the complete profile map rather than merging keys. Empty string
|
||||
// fields, nil pointers, and a nil or empty ExtraParams map inherit the selected
|
||||
// profile over framework defaults.
|
||||
type ExecutionTargetOverride struct {
|
||||
// Endpoint replaces the profile endpoint when non-empty.
|
||||
Endpoint string
|
||||
// Model replaces the profile model when non-empty.
|
||||
Model string
|
||||
// Temperature, when non-nil, must point to a value from 0 through 2.
|
||||
Temperature *float64
|
||||
// MaxTokens, when non-nil, must point to a non-negative value.
|
||||
MaxTokens *int
|
||||
// TopP, when non-nil, must point to a value from 0 through 1.
|
||||
TopP *float64
|
||||
// TimeoutSeconds, when non-nil, must point to a non-negative value. A
|
||||
// pointed-to zero disables the per-generation deadline.
|
||||
TimeoutSeconds *int
|
||||
// ServiceTier replaces the profile value when non-blank.
|
||||
ServiceTier string
|
||||
// ReasoningEffort replaces the profile value when non-blank. An empty value
|
||||
// cannot clear a profile setting.
|
||||
ReasoningEffort string
|
||||
// APIKeyEnv replaces the profile environment-variable name when non-blank.
|
||||
// A direct RunRequest.APIKey still takes precedence over environment lookup.
|
||||
APIKeyEnv string
|
||||
// ExtraParams, when non-empty, replaces the profile map. Values must be
|
||||
// JSON-compatible: nil, booleans, finite numbers, strings, arrays or slices,
|
||||
// and maps with non-empty string keys. Cycles are invalid.
|
||||
ExtraParams map[string]any
|
||||
}
|
||||
|
||||
@@ -167,18 +325,38 @@ type ExecutionTargetOverride struct {
|
||||
// It is equivalent to a loaded profile file after validation. Raw API keys do
|
||||
// not belong in profiles; use APIKeyRequired to require callers to provide
|
||||
// RunRequest.APIKey for each request, or use profile YAML api_key_env with file
|
||||
// and FS profile sources.
|
||||
// and FS profile sources. Profile has no stable JSON representation.
|
||||
//
|
||||
// WithProfiles validates and copies Profile values during NewEngine. Numeric
|
||||
// zero, blank strings, and an empty ExtraParams map inherit framework defaults;
|
||||
// use ExecutionTargetOverride pointer fields to request explicit numeric zero.
|
||||
type Profile struct {
|
||||
// ID is the required non-blank profile identifier. WithProfiles trims it.
|
||||
ID string
|
||||
// Endpoint is the required non-blank model-provider base URL.
|
||||
Endpoint string
|
||||
// Model is the required non-blank provider model identifier.
|
||||
Model string
|
||||
// Temperature is from 0 through 2. Zero inherits the framework default.
|
||||
Temperature float64
|
||||
// MaxTokens is non-negative. Zero inherits the framework default.
|
||||
MaxTokens int
|
||||
// TopP is from 0 through 1. Zero inherits the framework default rather than
|
||||
// selecting an explicit zero.
|
||||
TopP float64
|
||||
// TimeoutSeconds is non-negative. Zero inherits the framework default.
|
||||
TimeoutSeconds int
|
||||
// ServiceTier is optional; a blank value inherits the framework default.
|
||||
ServiceTier string
|
||||
// ReasoningEffort is optional; a blank value inherits the framework
|
||||
// default.
|
||||
ReasoningEffort string
|
||||
// APIKeyRequired requires a non-blank RunRequest.APIKey. It does not store a
|
||||
// credential or enable environment lookup.
|
||||
APIKeyRequired bool
|
||||
// ExtraParams contains provider-specific JSON-compatible values. An empty
|
||||
// map inherits framework defaults. WithProfiles validates and deeply copies
|
||||
// it during NewEngine.
|
||||
ExtraParams map[string]any
|
||||
}
|
||||
|
||||
@@ -187,120 +365,225 @@ type Profile struct {
|
||||
//
|
||||
// It contains ordinary profile fields for OpenAI-compatible chat-completions
|
||||
// endpoints. APIKeyRequired is satisfied by RunRequest.APIKey. Raw API keys do
|
||||
// not belong in this config.
|
||||
// not belong in this config. OpenAICompatibleProfileConfig has no stable JSON
|
||||
// representation and is not validated until its resulting Profile is supplied
|
||||
// through WithProfiles to NewEngine.
|
||||
type OpenAICompatibleProfileConfig struct {
|
||||
// ID becomes Profile.ID.
|
||||
ID string
|
||||
// Endpoint becomes Profile.Endpoint.
|
||||
Endpoint string
|
||||
// Model becomes Profile.Model.
|
||||
Model string
|
||||
// APIKeyRequired becomes Profile.APIKeyRequired.
|
||||
APIKeyRequired bool
|
||||
// Temperature becomes Profile.Temperature.
|
||||
Temperature float64
|
||||
// MaxTokens becomes Profile.MaxTokens.
|
||||
MaxTokens int
|
||||
// TopP becomes Profile.TopP.
|
||||
TopP float64
|
||||
// TimeoutSeconds becomes Profile.TimeoutSeconds.
|
||||
TimeoutSeconds int
|
||||
// ServiceTier becomes Profile.ServiceTier.
|
||||
ServiceTier string
|
||||
// ReasoningEffort becomes Profile.ReasoningEffort.
|
||||
ReasoningEffort string
|
||||
// ExtraParams becomes a shallow-copied Profile.ExtraParams map. NewEngine
|
||||
// performs validation and a deep copy when WithProfiles applies the result.
|
||||
ExtraParams map[string]any
|
||||
}
|
||||
|
||||
// ExecutionTargetPresence tracks which numeric runtime settings were explicit
|
||||
// request overrides.
|
||||
// request overrides, including explicit zero values. It has a stable JSON
|
||||
// representation and is supplied to injected LLM clients so they can preserve
|
||||
// omission semantics.
|
||||
type ExecutionTargetPresence struct {
|
||||
Temperature bool
|
||||
MaxTokens bool
|
||||
TopP bool
|
||||
TimeoutSeconds bool
|
||||
// Temperature reports a non-nil ExecutionTargetOverride.Temperature.
|
||||
Temperature bool `json:"temperature"`
|
||||
// MaxTokens reports a non-nil ExecutionTargetOverride.MaxTokens.
|
||||
MaxTokens bool `json:"max_tokens"`
|
||||
// TopP reports a non-nil ExecutionTargetOverride.TopP.
|
||||
TopP bool `json:"top_p"`
|
||||
// TimeoutSeconds reports a non-nil ExecutionTargetOverride.TimeoutSeconds.
|
||||
TimeoutSeconds bool `json:"timeout_seconds"`
|
||||
}
|
||||
|
||||
// OutputContract defines output and validation requirements.
|
||||
// OutputContract defines output and validation requirements and has a stable
|
||||
// JSON representation.
|
||||
//
|
||||
// A non-nil RunRequest.Validation replaces the complete prompt contract. It
|
||||
// does not merge fields. The public Engine validates generated output once and
|
||||
// does not install an output repairer.
|
||||
type OutputContract struct {
|
||||
// Format selects generated artifact metadata. An empty effective value
|
||||
// defaults to FormatText.
|
||||
Format OutputFormat `json:"format"`
|
||||
// ValidationMode selects the content check. Use one of the declared
|
||||
// ValidationMode constants.
|
||||
ValidationMode ValidationMode `json:"validation_mode"`
|
||||
// SchemaPath is required when ValidationMode is ValidationJSONSchema and is
|
||||
// ignored by other modes.
|
||||
SchemaPath string `json:"schema_path"`
|
||||
// RepairAttempts is a requested repair limit. A non-positive value requests
|
||||
// no repairs. The public Engine performs no repairs even when this value is
|
||||
// positive, so its runs report zero attempts used.
|
||||
RepairAttempts int `json:"repair_attempts"`
|
||||
}
|
||||
|
||||
// ValidationResult represents output validation state.
|
||||
// ValidationResult represents a completed output check and has a stable JSON
|
||||
// representation. An operational inability to perform validation is returned
|
||||
// as ErrValidation instead of a ValidationResult.
|
||||
type ValidationResult struct {
|
||||
// Status is Passed, Failed, or Skipped.
|
||||
Status ValidationStatus `json:"status"`
|
||||
// Mode is the effective validation mode.
|
||||
Mode ValidationMode `json:"mode"`
|
||||
// Errors contains validation diagnostics when Status is ValidationFailed.
|
||||
Errors []string `json:"errors,omitempty"`
|
||||
// SchemaPath is the effective schema path for JSON Schema validation.
|
||||
SchemaPath string `json:"schema_path,omitempty"`
|
||||
// RepairAttempts is the number of repairs actually attempted. It is always
|
||||
// zero for the public Engine.
|
||||
RepairAttempts int `json:"repair_attempts"`
|
||||
// IsValid is true for ValidationPassed and ValidationSkipped and false for
|
||||
// ValidationFailed.
|
||||
IsValid bool `json:"is_valid"`
|
||||
}
|
||||
|
||||
// TokenUsage tracks token consumption.
|
||||
// TokenUsage contains model-client token accounting and has a stable JSON
|
||||
// representation. Promptkit preserves values reported by the client and does
|
||||
// not derive or reconcile them.
|
||||
type TokenUsage struct {
|
||||
// PromptTokens is the reported input-token count.
|
||||
PromptTokens int `json:"prompt_tokens"`
|
||||
// CompletionTokens is the reported generated-token count.
|
||||
CompletionTokens int `json:"completion_tokens"`
|
||||
// TotalTokens is the reported total-token count.
|
||||
TotalTokens int `json:"total_tokens"`
|
||||
// CachedTokens is the reported cached-input-token count.
|
||||
CachedTokens int `json:"cached_tokens"`
|
||||
// CacheWriteTokens is the reported cache-write-token count.
|
||||
CacheWriteTokens int `json:"cache_write_tokens"`
|
||||
}
|
||||
|
||||
// RenderedPrompt is the fully rendered prompt passed to an LLM client.
|
||||
// RenderedPrompt is the fully rendered prompt passed to an LLM client and has
|
||||
// a stable JSON representation.
|
||||
type RenderedPrompt struct {
|
||||
// SessionID is the optional trimmed session identifier rendered from the
|
||||
// prompt definition.
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
// Messages contains rendered messages in definition order.
|
||||
Messages []RenderedMessage `json:"messages"`
|
||||
}
|
||||
|
||||
// RenderedMessage is a rendered chat message.
|
||||
// RenderedMessage is a rendered chat message and has a stable JSON
|
||||
// representation.
|
||||
type RenderedMessage struct {
|
||||
// Role is the definition-supplied chat role.
|
||||
Role string `json:"role"`
|
||||
// Content is the rendered message text.
|
||||
Content string `json:"content"`
|
||||
// CacheControl is optional provider cache metadata.
|
||||
CacheControl *CacheControl `json:"cache_control,omitempty"`
|
||||
}
|
||||
|
||||
// CacheControl describes provider cache metadata attached to prompt content.
|
||||
// CacheControl describes provider cache metadata attached to prompt content
|
||||
// and has a stable JSON representation.
|
||||
type CacheControl struct {
|
||||
// Type identifies the cache behavior.
|
||||
Type CacheControlType `json:"type"`
|
||||
// TTL is an optional provider cache lifetime.
|
||||
TTL string `json:"ttl,omitempty"`
|
||||
}
|
||||
|
||||
// StructuredOutputSpec describes provider-level structured output.
|
||||
// StructuredOutputSpec describes provider-level structured output and has a
|
||||
// stable JSON representation.
|
||||
type StructuredOutputSpec struct {
|
||||
// Type identifies the structured-output mechanism.
|
||||
Type StructuredOutputType `json:"type"`
|
||||
// JSONSchema contains constraints when Type is StructuredOutputJSONSchema.
|
||||
JSONSchema *StructuredOutputJSONSpec `json:"json_schema,omitempty"`
|
||||
}
|
||||
|
||||
// StructuredOutputJSONSpec contains JSON Schema output constraints.
|
||||
// StructuredOutputJSONSpec contains provider-facing JSON Schema output
|
||||
// constraints and has a stable JSON representation.
|
||||
type StructuredOutputJSONSpec struct {
|
||||
// Name is the provider-facing schema name.
|
||||
Name string `json:"name"`
|
||||
// Strict requests strict provider enforcement of Schema.
|
||||
Strict bool `json:"strict"`
|
||||
// Schema is a caller-owned copy of the loaded JSON Schema document.
|
||||
Schema any `json:"schema"`
|
||||
}
|
||||
|
||||
// LLMClient executes rendered prompts for Engine.Run.
|
||||
// LLMClient executes rendered prompts for [Engine.Run].
|
||||
//
|
||||
// Generate may be called concurrently. It must honor context cancellation to
|
||||
// make Run responsive to cancellation. The request and all nested maps,
|
||||
// slices, and pointers are client-owned copies and may be mutated or retained
|
||||
// without affecting engine state.
|
||||
//
|
||||
// Generate receives rendered messages and may receive a direct API key. A
|
||||
// client must protect those values and any raw output in its logging, storage,
|
||||
// and retained copies. It is responsible for the cancellation behavior of any
|
||||
// work it starts and for synchronizing access to retained or shared data.
|
||||
//
|
||||
// A returned error makes Run return ErrLLMGenerate while preserving the client
|
||||
// error through errors.Is. A nil response with a nil error also produces
|
||||
// ErrLLMGenerate. Promptkit copies the non-nil response before returning from
|
||||
// Run.
|
||||
type LLMClient interface {
|
||||
Generate(context.Context, GenerateRequest) (*GenerateResponse, error)
|
||||
}
|
||||
|
||||
// GenerateRequest is passed to an injected LLM client.
|
||||
// GenerateRequest is passed to an injected LLM client and has a stable JSON
|
||||
// representation. Its String and GoString methods omit rendered content and
|
||||
// direct credentials.
|
||||
type GenerateRequest struct {
|
||||
// Prompt contains the rendered session ID and messages.
|
||||
Prompt RenderedPrompt `json:"prompt"`
|
||||
// Target contains effective model settings without the direct API key.
|
||||
Target ExecutionTarget `json:"target"`
|
||||
// TargetPresence distinguishes inherited numeric zeros from explicit
|
||||
// request overrides.
|
||||
TargetPresence ExecutionTargetPresence `json:"target_presence"`
|
||||
// StructuredOutput contains provider response constraints when requested.
|
||||
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
|
||||
// APIKey is the direct request-scoped credential, if any. It is excluded
|
||||
// from JSON, String, and GoString output.
|
||||
APIKey string `json:"-"`
|
||||
}
|
||||
|
||||
// GenerateResponse is returned by an injected LLM client.
|
||||
// GenerateResponse is returned by an injected LLM client and has a stable JSON
|
||||
// representation.
|
||||
type GenerateResponse struct {
|
||||
// Content is the generated output. It must be non-empty when using the
|
||||
// built-in client; injected clients may return empty content for Promptkit
|
||||
// validation to classify.
|
||||
Content string `json:"content"`
|
||||
// Usage is the client's token accounting.
|
||||
Usage TokenUsage `json:"usage"`
|
||||
}
|
||||
|
||||
// File returns a file-backed artifact reference.
|
||||
// File returns a file-backed artifact reference whose URI is path.
|
||||
//
|
||||
// The default artifact reader opens path as a caller-selected operating-system
|
||||
// path without restricting it to an application root or imposing a size limit.
|
||||
// Applications accepting untrusted paths must validate them before calling
|
||||
// Promptkit or use [WithArtifactReader] to enforce application policy.
|
||||
func File(path string) ArtifactRef {
|
||||
return ArtifactRef{Type: ArtifactRefFile, URI: path}
|
||||
}
|
||||
|
||||
// Inline returns an inline artifact reference.
|
||||
// Inline returns an inline artifact reference whose Body is body and whose URI
|
||||
// is empty.
|
||||
func Inline(body string) ArtifactRef {
|
||||
return ArtifactRef{Type: ArtifactRefInline, Body: body}
|
||||
}
|
||||
|
||||
// InlineWithURI returns an inline artifact reference with URI metadata.
|
||||
// InlineWithURI returns an inline artifact reference with body content and uri
|
||||
// provenance metadata.
|
||||
func InlineWithURI(uri string, body string) ArtifactRef {
|
||||
return ArtifactRef{Type: ArtifactRefInline, URI: uri, Body: body}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user