eric/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: eric:v0.1.0
Branches Tags
eric:main
eric:v0.9.0 eric:v0.8.0 eric:v0.7.0 eric:v0.6.0 eric:v0.5.0 eric:v0.4.0 eric:v0.3.0 eric:v0.2.0 eric:v0.1.0
...
compare: eric:b4621534838949622483431a4c8f973c7fd8c061
Branches Tags
eric:main
eric:v0.9.0 eric:v0.8.0 eric:v0.7.0 eric:v0.6.0 eric:v0.5.0 eric:v0.4.0 eric:v0.3.0 eric:v0.2.0 eric:v0.1.0

4 Commits
v0.1.0 ... b462153483

Author SHA1 Message Date
Eric Rakestraw
b462153483 Document consumer safety and offline execution 2026-07-29 14:13:52 +00:00
Eric Rakestraw
086cf0fc86 Make GoDoc the public API contract 2026-07-29 14:06:54 +00:00
Eric Rakestraw
c1cecb1ee8 Resolve public documentation contract questions 2026-07-29 13:53:33 +00:00
Eric Rakestraw
bcb327f643 Document the published Promptkit release process 2026-07-28 19:31:32 +00:00
21 changed files with 2262 additions and 382 deletions
Expand all files Collapse all files

5
README.md
View File

@@ -31,4 +31,9 @@ Contributors should start with the [development guide](docs/development.md).
The [architecture policy](docs/policy/architecture.md) defines the library
boundary and constraints that framework work must preserve.
## Related Project
[Scriptorium](https://gitea.maximumdirect.net/eric/scriptorium) is the CLI and
HTTP application built on Promptkit.
Promptkit is licensed under the [GNU General Public License version 3](LICENSE).

1
convert.go
View File

@@ -20,7 +20,6 @@ func toDomainRunRequest(req RunRequest) (domain.RunRequest, error) {
Vars: copyStringMap(req.Vars),
Execution: execution,
Validation: toDomainOutputContractPtr(req.Validation),
Metadata: copyStringMap(req.Metadata),
}, nil
}

54
doc.go
View File

@@ -1,8 +1,54 @@
// Package promptkit provides an embeddable engine for preparing and executing
// prompt-defined LLM workflows.
//
// Applications construct an Engine with NewEngine, select filesystem or
// in-memory definition sources with options, and use Prepare or Run to execute
// requests. Concrete repositories, validators, and outbound clients remain
// internal implementation details.
// Applications construct an [Engine] with [NewEngine], select filesystem or
// in-memory sources with options, and call [Engine.Prepare] or [Engine.Run].
// Concrete repositories, validators, and the built-in OpenAI-compatible client
// remain internal implementation details.
//
// # Concurrency and ownership
//
// An Engine supports concurrent Prepare and Run calls. An injected [LLMClient]
// or [ArtifactReader] can therefore receive concurrent calls and must be safe
// for that use.
//
// NewEngine copies in-memory profiles. Prepare and Run copy request maps,
// slices, pointer values, and JSON-compatible extra parameters before using
// them. Returned values and values passed to extension interfaces are likewise
// isolated from engine state. Callers own those copies and may mutate them
// after the call that supplied or returned them.
//
// # Security and sensitive data
//
// The default artifact reader treats [File] paths as caller-selected operating
// system paths. It does not restrict them to an application root or impose an
// inbound request-size policy. Promptkit is not an inbound request or
// untrusted-input security boundary. Applications must validate and restrict
// untrusted input before constructing a request, or install an [ArtifactReader]
// that enforces their filesystem, authorization, and size policies.
//
// Rendered messages, input and output [Artifact] bodies, [RunResult.RawOutput],
// and [ValidationResult.Errors] may contain sensitive data. Credential
// exclusion and redaction do not sanitize those values. Applications and
// injected collaborators are responsible for access control, retention,
// logging, and secret handling appropriate to their data.
//
// # JSON
//
// Stable JSON representations are provided for [PreparedRun], [RunResult],
// [Artifact], [ExecutionTarget], [OutputContract], [ValidationResult],
// [TokenUsage], [RenderedPrompt], [RenderedMessage], [CacheControl],
// [StructuredOutputSpec], [StructuredOutputJSONSpec], [GenerateRequest],
// [GenerateResponse], [ExecutionTargetPresence], and the string value types
// used by those values.
//
// Construction values, including [Config], [RunRequest], [ArtifactRef],
// [ExecutionTargetOverride], [Profile], and
// [OpenAICompatibleProfileConfig], do not have stable JSON representations.
// Direct API keys are nevertheless excluded from JSON for every public value.
//
// JSON timestamps use time.Time's RFC 3339 encoding and are omitted when zero.
// PreparedRun and RunResult durations are encoded as integer milliseconds in
// duration_ms and omitted when zero. Run IDs and all exposed hashes are opaque:
// their spelling, length, character set, and algorithm are not API contracts.
package promptkit

230
docs/consumers/pkg-promptkit.md
View File

@@ -1,151 +1,147 @@
# Package `promptkit`
Import path:
## Purpose
This guide helps Go consumers assemble Promptkit and choose the main
preparation or execution workflow. The declarations and GoDoc in the
[root package](../../doc.go) own exact field, option, serialization,
concurrency, ownership, failure, and cancellation semantics. The
[framework format reference](../formats.md) owns prompt, profile, and schema
file contracts.
Import the package as:
```go
import "gitea.maximumdirect.net/eric/promptkit"
```
Package `promptkit` is the supported Go contract for in-process prompt
preparation and execution. The declarations and their GoDoc in the
[root package](../../doc.go) own the exact API; this guide explains how the
pieces are used together. The [framework format reference](../formats.md) owns
prompt, profile, and schema file contracts.
The following Go fragments are illustrative and omit surrounding package,
import, and error-handling code. Use the maintained examples for complete
programs.
## Engine Construction And Sources
## Construct An Engine
Construct an engine with [`NewEngine`, `Config`, and
`Option`](../../engine.go). `PromptDir` is required unless a prompt source
option is supplied. `ProfileDir` optionally overlays built-in profiles, and an
empty `SchemaDir` uses the current directory. `Timeout` is the transport-wide
safety cap for the built-in OpenAI-compatible client. An optional `HTTPClient`
is cloned; its positive timeout takes precedence.
Create an engine with
[`NewEngine`](../../engine.go). A directory-backed setup supplies a prompt
directory and may supply profile and schema directories:
Nil options are ignored. Invalid construction, including a nil injected client
or artifact reader, returns an error matching `ErrInvalidConfig`.
```go
engine, err := promptkit.NewEngine(promptkit.Config{
PromptDir: "prompts",
ProfileDir: "profiles",
SchemaDir: "schemas",
})
```
The [source options](../../engine.go) replace their matching directory source:
- `WithPromptFS` and `WithPromptFile` select prompt definitions;
- `WithProfileFS` and `WithProfileFile` overlay built-in profiles;
- `WithProfiles` adds in-memory profiles ahead of file and built-in profiles;
- `WithSchemaFS` and `WithSchemaFile` select JSON Schema documents;
- `WithLLMClient` replaces the built-in model client; and
- `WithArtifactReader` replaces the default reader for every input.
Source selection, path resolution, strict decoding, profile overlays, and
file-to-request precedence are defined in the
Options support single-file or `fs.FS` sources, in-memory profiles, and
injected artifact or model clients. Consult the
[constructor and option GoDoc](../../engine.go) for composition, precedence,
validation, and default transport behavior. Source discovery, format
validation, and profile precedence are defined by the
[framework format reference](../formats.md).
Per-generation timeout values from profiles or requests are independent of
the transport cap and caller context. An explicit request value of zero
disables only the per-generation deadline. The
[outbound integration contract](../integrations/openai-compatible-chat.md#timeout-and-cancellation)
defines the complete timeout layering.
## Prepare Without Model Execution
## Preparation And Execution
[`Engine.Prepare`](../../engine.go) resolves the selected prompt and profile,
loads inputs and any structured-output schema, and renders messages without
calling a model client:
[`Engine.Prepare` and `Engine.Run`](../../engine.go) accept the public
[`RunRequest`](../../types.go). `Prepare` resolves the prompt, profile, input
artifacts, validation contract, and rendered messages without calling an LLM.
`Run` performs the same preparation, calls the configured client, and validates
the generated content. The maintained
```go
prepared, err := engine.Prepare(ctx, promptkit.RunRequest{
PromptID: "meeting.summary",
Inputs: map[string]promptkit.ArtifactRef{
"note": promptkit.Inline("Synthetic meeting notes"),
},
})
```
The maintained
[offline preparation example](../../examples/go-library/prepare/main.go)
provides a complete runnable workflow using a prompt file, in-memory profile,
and inline input.
shows a complete runnable setup with a prompt file, in-memory profile, and
inline input. Exact request requirements and prepared-result fields belong to
the [`RunRequest` and `PreparedRun` GoDoc](../../types.go).
[`PreparedRun` and `RunResult`](../../types.go) expose copied public values.
Preparation returns effective settings, hashes, rendered messages, selected
profile, structured-output information, and timing without resolved secrets or
model output. Execution adds the generated artifact and raw output, validation
state, model metadata, usage, run ID, and duration.
## Execute And Validate
A generated-content validation failure returns a result with
`Validation.Status == ValidationFailed`. An inability to perform validation
returns an error matching `ErrValidation`.
[`Engine.Run`](../../engine.go) performs the same preparation, invokes the
configured model client, classifies the generated artifact, and validates the
content. A completed content check may return `ValidationFailed` in the result;
an operational inability to validate returns an error.
## Requests, Inputs, And Overrides
The maintained
[offline execution example](../../examples/go-library/run/main.go) injects a
deterministic model client and exercises `Run` without credentials, network
access, or paid calls. It is intentionally separate from the preparation
example so each workflow and its small prompt fixture can be copied and run on
its own.
The [request and value declarations](../../types.go) own the available fields,
serialized constants, and result shapes. Use `File`, `Inline`, or
`InlineWithURI` to construct artifact references. The
[framework format reference](../formats.md) defines declared inputs, template
references, output contracts, and the relationship between file values and
request overrides.
Use the [`RunResult` and `ValidationResult` GoDoc](../../types.go) for the
returned data and the `Engine.Run` GoDoc for failure and cancellation
semantics. The
[OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md)
owns the built-in client's outbound HTTP behavior.
`ExecutionTargetOverride` uses pointers for numeric settings so an explicit
zero remains distinct from no override. `ExtraParams` accepts JSON-compatible
strings, booleans, finite numbers, string-keyed objects, arrays or slices, and
nil. Unsupported values, non-string map keys, non-finite numbers, and cycles
match `ErrInvalidConfig` in profiles or `ErrInvalidRequest` in request
overrides.
## Inputs, Profiles, And Overrides
Returned requests, profiles, prepared values, results, artifacts, maps, and
slices are isolated from internal engine state. Consumers and injected
extensions should not retain or mutate values owned by another caller.
Use `File`, `Inline`, or `InlineWithURI` to construct request inputs. A request
can select a profile explicitly or use the prompt's default profile, and can
replace execution settings or the complete output contract.
## Profiles And Credentials
The [public value GoDoc](../../types.go) defines nil, empty, zero, replacement,
copy, and credential behavior. The
[framework format reference](../formats.md) defines how those request values
interact with prompt definitions, file-backed profiles, built-ins, schemas,
and framework defaults.
[`OpenAICompatibleProfile`](../../profiles.go) constructs an ordinary
in-memory profile for an OpenAI-compatible chat-completions endpoint.
`WithProfiles` rejects duplicate IDs in one call and gives in-memory profiles
precedence over explicit file sources and built-ins.
For programmatic profiles,
[`OpenAICompatibleProfile`](../../profiles.go) converts ordinary
OpenAI-compatible settings into a value accepted by `WithProfiles`.
Raw API keys do not belong in profiles. File-backed profiles may name an
environment variable, while an in-memory profile can require a request key.
A direct `RunRequest.APIKey` is request-scoped and takes precedence over an
environment lookup for the built-in client. Profile fields, ranges, built-ins,
precedence, and credential rules are owned by the
[framework format reference](../formats.md).
## Credentials
API keys are excluded from JSON, prepared values, and results. The public
`String` and `GoString` methods report only whether a direct key is present.
Avoid reflection-based dumps of request structs, which can bypass that
redaction.
File-backed profiles name an environment variable; in-memory profiles can
require a direct request key. Direct keys are request-scoped and are excluded
from supported JSON values and the package's `String` and `GoString`
summaries. The exact precedence and redaction guarantees belong to
[`RunRequest`, `GenerateRequest`, and the profile GoDoc](../../types.go).
## Protect Files And Generated Data
The default artifact reader opens a `File` reference as a caller-selected
operating-system path. It does not constrain paths to an application root,
impose an inbound request-size policy, or establish an untrusted-input security
boundary. Applications must validate and restrict untrusted paths and payloads
before constructing a request, or inject an artifact reader that enforces
their filesystem, authorization, and size policies.
Rendered messages, input and output artifact bodies, raw model output, and
validation diagnostics can contain sensitive data. API-key redaction does not
sanitize those values. Treat prepared values, results, collaborator requests,
errors, and logs according to the application's data-access, retention, and
secret-handling policies.
## Extension Interfaces
The [`LLMClient`, `GenerateRequest`, and
`GenerateResponse`](../../types.go) boundary lets a consumer replace model
generation. Injected clients receive copied rendered messages, effective
settings, explicit numeric-setting presence, structured-output constraints,
and the request-scoped key. They return generated content and token usage.
Inject an [`LLMClient` or `ArtifactReader`](../../types.go) when the built-in
behavior does not fit the application. Their GoDoc defines concurrent use,
context handling, ownership of copied values, nil responses, and preservation
of collaborator errors. Implementations must honor cancellation, safely manage
copies they retain, avoid unsafe logging of content or credentials, and enforce
the application policy that motivated the injection.
The [`ArtifactReader`](../../types.go) boundary replaces the default inline and
file reader for every input. Readers provide artifact content and metadata; the
engine fills an empty artifact name from the input-map key. A reader error
matches `ErrArtifactLoad` while preserving the original identity for
`errors.Is`. A nil artifact with a nil error is also an artifact-load failure.
## Handle Errors
Extensions should honor context cancellation and avoid logging raw prompts,
artifacts, or credentials.
Use `errors.Is` with the
[public error sentinels and operation GoDoc](../../engine.go). The declarations
distinguish invalid construction, invalid requests, absent sources,
source-loading failures, collaborator failures, and operational validation
failures. Specific request conditions may also match the broader
`ErrInvalidRequest`, and injected collaborator identities are preserved where
documented.
## Errors
The [public error declarations](../../engine.go) and
[mapping](../../errors.go) preserve these sentinel checks through `errors.Is`:
- `ErrInvalidConfig`
- `ErrInvalidRequest`
- `ErrPromptNotFound`
- `ErrProfileNotFound`
- `ErrProfileRequired`
- `ErrPromptLoad`
- `ErrProfileLoad`
- `ErrAPIKeyEnvMissing`
- `ErrArtifactLoad`
- `ErrPromptRender`
- `ErrLLMGenerate`
- `ErrValidation`
`ErrProfileRequired` and `ErrAPIKeyEnvMissing` also match
`ErrInvalidRequest`, allowing either broad request handling or a specific
condition. Wrapped collaborator errors retain their identity where the public
contract promises it.
## Consumer Boundary
## Application Boundary
Promptkit is an importable library. It does not own a command, inbound HTTP
API, process configuration, or deployment policy. Scriptorium is one
downstream application that maps this root package contract into those
application concerns.
API, process configuration, or deployment policy. Applications map the root
package's results and errors into those concerns, including inbound size and
trust policy.

1
docs/internal/overview.md
View File

@@ -13,6 +13,7 @@ contributor workflow and validation.
| --- | --- | --- |
| Root `promptkit` package | Provides the supported engine facade, source and injection options, public request and result values, built-in profile construction, extension interfaces, value conversion, redacted formatting, and public error mapping. | [Package GoDoc](../../doc.go), [engine assembly](../../engine.go) |
| `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
| `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, profiles, execution targets, rendering, generation, and validation. | [Domain declarations](../../internal/domain/domain.go) |
| `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) |
| `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |

6
docs/policy/architecture.md
View File

@@ -41,9 +41,9 @@ The implemented internal components consist of:
- `internal/usecase`, which coordinates preparation and execution across the
internal framework components.
The `examples/go-library/prepare` package is a maintained downstream consumer
of the root facade. It does not expose a library package or participate in
internal assembly.
The `examples/go-library/prepare` and `examples/go-library/run` packages are
maintained downstream consumers of the root facade. They do not expose library
packages or participate in internal assembly.
The root facade assembles the internal repositories, renderer, validator,
outbound client, and use-case runner while translating public values and

254
docs/release.md
View File

@@ -7,27 +7,91 @@ tags. It does not publish runnable binaries or binary packages and does not
currently use hosted CI. The release maintainer performs and records the
required validation.
The first planned release is `v0.1.0`. Do not create that tag until the
framework has been extracted and the resulting public library has passed this
procedure. Later tags use the `vMAJOR.MINOR.PATCH` form. While Promptkit remains
pre-`v1`, release notes must identify intentional public API changes and any
consumer migration required by them.
`v0.1.0` is the initial published release. Later releases use semantic
`vMAJOR.MINOR.PATCH` tags. Before `v1`, minor releases may change the public
API and patch releases preserve compatibility within their minor line. Every
pre-`v1` release note must summarize compatibility, identify public API
changes, and state any action required of consumers.
## Prepare The Release
Promptkit releases are source-only. The annotated tag message is the release
note; there is no separate hosted release or binary packaging step.
Work from a clean checkout of the intended release commit, outside any Go
workspace and without a local module replacement. Confirm the source commit is
already published through the normal branch workflow.
## Establish The Candidate
From the Promptkit repository root, verify the checkout:
Choose a version that has not been published and export it as
`RELEASE_VERSION`. Run every command in this procedure from the Promptkit
repository root in the same POSIX shell. Do not reuse `v0.1.0` or another
existing version.
The following guard derives the release commit from `HEAD` and stops on a
missing or malformed version, a checkout other than synchronized `main`,
uncommitted changes, an active Go workspace, a module replacement, a vendor
tree, or an existing local or remote tag:
```sh
gowork=$(go env GOWORK)
test -z "$gowork" || test "$gowork" = off
test -z "$(git status --short)"
git fetch --tags origin
set -eu
: "${RELEASE_VERSION:?export an unpublished vMAJOR.MINOR.PATCH version}"
if ! printf '%s\n' "$RELEASE_VERSION" |
grep -Eq '^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$'
then
printf '%s\n' "invalid release version: $RELEASE_VERSION" >&2
exit 1
fi
RELEASE_COMMIT=$(git rev-parse --verify 'HEAD^{commit}')
export RELEASE_COMMIT
check_release_candidate() {
test "$(git branch --show-current)" = main
test -z "$(git status --porcelain)"
gowork_value=$(go env GOWORK)
case "$gowork_value" in
''|off) ;;
*)
printf '%s\n' "active Go workspace: $gowork_value" >&2
return 1
;;
esac
test -z "$(git ls-files go.work go.work.sum)"
test ! -e vendor
if grep -Eq '^[[:space:]]*replace([[:space:]]|\()' go.mod
then
printf '%s\n' 'go.mod contains a replacement' >&2
return 1
fi
git fetch origin main --tags
test "$RELEASE_COMMIT" = \
"$(git rev-parse --verify 'refs/remotes/origin/main^{commit}')"
if git show-ref --verify --quiet "refs/tags/$RELEASE_VERSION"
then
printf '%s\n' "local tag already exists: $RELEASE_VERSION" >&2
return 1
fi
if test -n "$(
git ls-remote --tags origin \
"refs/tags/$RELEASE_VERSION" \
"refs/tags/$RELEASE_VERSION^{}"
)"
then
printf '%s\n' "remote tag already exists: $RELEASE_VERSION" >&2
return 1
fi
}
check_release_candidate
```
Do not continue unless the guard completes successfully. In particular, push
the intended commit through the normal `main` branch workflow before release;
the tag procedure is not a substitute for publishing the source commit.
## Validate The Candidate
Confirm the module and root package metadata:
```sh
@@ -42,7 +106,7 @@ gitea.maximumdirect.net/eric/promptkit 1.25.5
promptkit gitea.maximumdirect.net/eric/promptkit
```
Run the same default Go validation required by the
Run the complete maintainer validation required by the
[development guide](development.md):
```sh
@@ -53,84 +117,158 @@ go build ./...
go run ./examples/go-library/prepare
```
Check every tracked Go file and repository whitespace:
Check every tracked Go file. This command must produce no output:
```sh
gofmt -l $(git ls-files '*.go')
unformatted=$(
git ls-files '*.go' |
while IFS= read -r go_file
do
gofmt -l "$go_file"
done
)
test -z "$unformatted"
```
Follow every maintained Markdown link and confirm that its local or published
target exists. Review the repository for generated binaries, test or coverage
output, credentials, template residue, downloaded assets, and other files that
do not belong in source control.
Recheck module and repository hygiene, whitespace, and the clean checkout:
```sh
test -z "$(git ls-files go.work go.work.sum)"
test ! -e vendor
if grep -Eq '^[[:space:]]*replace([[:space:]]|\()' go.mod
then
printf '%s\n' 'go.mod contains a replacement' >&2
exit 1
fi
git diff --check
test -z "$(git status --porcelain)"
```
The formatting command must produce no paths. Follow every maintained Markdown
link and confirm its target exists. Review the repository for generated
binaries, test or coverage output, credentials, template residue, and other
files that do not belong in source control.
## Write The Release Note
Confirm that no workspace override is tracked and that `go.mod` contains no
`replace` directive:
Prepare a plain-text annotated-tag message outside the repository and export
its path as `RELEASE_NOTES_FILE`. Use this form, replacing each summary with
release-specific text; write `None.` when there are no public API changes or
consumer actions:
```sh
git ls-files go.work go.work.sum
rg -n '^replace\b' go.mod
```text
Promptkit vMAJOR.MINOR.PATCH
Validated commit: full commit ID
Compatibility: compatibility summary
Public API changes: changes or None.
Consumer action: required action or None.
```
Both commands must produce no output. Re-run `git status --short` and require a
clean result after every validation and review check.
## Create And Publish The Tag
Choose the semantic version from the intended compatibility change. Record the
release commit before tagging:
After writing it, require all release-note fields, the selected version, and
the validated commit to be present:
```sh
release_version=v0.1.0
release_commit=$(git rev-parse HEAD)
: "${RELEASE_NOTES_FILE:?export the path to the release-note file}"
test -f "$RELEASE_NOTES_FILE"
test -s "$RELEASE_NOTES_FILE"
grep -F "Promptkit $RELEASE_VERSION" "$RELEASE_NOTES_FILE"
grep -F "Validated commit: $RELEASE_COMMIT" "$RELEASE_NOTES_FILE"
grep -F 'Compatibility:' "$RELEASE_NOTES_FILE"
grep -F 'Public API changes:' "$RELEASE_NOTES_FILE"
grep -F 'Consumer action:' "$RELEASE_NOTES_FILE"
```
Replace the example version for later releases and keep both values in the same
shell for the remaining commands. Confirm the tag does not already exist
locally or remotely:
Inspect the complete message and confirm that it accurately records the
compatibility impact, public API changes, and required consumer action.
## Create And Inspect The Tag
Run the candidate guard again immediately before tag creation. This ensures
that validation or release-note preparation did not change the checkout and
that the commit is still published and untagged:
```sh
test -z "$(git tag --list "$release_version")"
test -z "$(git ls-remote --tags origin "refs/tags/$release_version")"
check_release_candidate
```
Create an annotated tag whose message identifies the release and records that
the documented validation passed for the tagged commit:
Create the annotated tag from the prepared release note and bind it explicitly
to the validated commit:
```sh
git tag --annotate "$release_version" \
--message "Promptkit $release_version; documented validation passed for $release_commit"
git tag --annotate "$RELEASE_VERSION" \
--file "$RELEASE_NOTES_FILE" \
"$RELEASE_COMMIT"
```
Inspect the tag before publication:
Inspect both the tag message and its source commit before publication:
```sh
git show --no-patch --decorate "$release_version"
test "$(git rev-list -n 1 "$release_version")" = "$release_commit"
test "$(git cat-file -t "refs/tags/$RELEASE_VERSION")" = tag
git show --no-patch --decorate "refs/tags/$RELEASE_VERSION"
test "$(
git rev-parse --verify "refs/tags/$RELEASE_VERSION^{commit}"
)" = "$RELEASE_COMMIT"
```
Publish the tag without relying on a hosting-provider-specific release
interface:
If inspection finds an error, delete the unpublished local tag, correct the
release note or candidate, and repeat the guards. Never move or recreate a tag
that has been published.
## Publish The Selected Tag
Push only the selected tag ref. Do not use `git push --tags`:
```sh
git push origin "refs/tags/$release_version"
git push origin \
"refs/tags/$RELEASE_VERSION:refs/tags/$RELEASE_VERSION"
```
## Verify Publication
Confirm that the remote tag object matches the local annotated tag and still
resolves to the intended source commit:
Compare the remote annotated-tag object with the local object, then compare the
remote peeled source commit with the validated commit:
```sh
remote_tag=$(git ls-remote --tags origin "refs/tags/$release_version" | awk '{print $1}')
test "$remote_tag" = "$(git rev-parse "refs/tags/$release_version")"
test "$(git rev-list -n 1 "refs/tags/$release_version")" = "$release_commit"
remote_tag=$(
git ls-remote --tags origin "refs/tags/$RELEASE_VERSION" |
awk 'NR == 1 { print $1 }'
)
remote_commit=$(
git ls-remote --tags origin "refs/tags/$RELEASE_VERSION^{}" |
awk 'NR == 1 { print $1 }'
)
test -n "$remote_tag"
test "$remote_tag" = \
"$(git rev-parse --verify "refs/tags/$RELEASE_VERSION")"
test "$remote_commit" = "$RELEASE_COMMIT"
```
Promptkit must publish the required tag before Scriptorium or another consumer
publishes a release that depends on that version. Released consumer modules
must not use a local replacement or unpublished Promptkit revision.
Finally, resolve the version as an ordinary Go module in a temporary module
outside this repository and without a workspace or replacement:
```sh
resolution_dir=$(mktemp -d)
(
trap 'rm -rf "$resolution_dir"' 0 1 2 15
cd "$resolution_dir"
GOWORK=off go mod init example.com/promptkit-release-check
GOWORK=off go mod download \
"gitea.maximumdirect.net/eric/promptkit@$RELEASE_VERSION"
resolved_version=$(
GOWORK=off go list -m -f '{{.Version}}' \
"gitea.maximumdirect.net/eric/promptkit@$RELEASE_VERSION"
)
test "$resolved_version" = "$RELEASE_VERSION"
)
```
Promptkit must publish and verify the required version before Scriptorium or
another consumer publishes a release that depends on it. This ordering does
not replace the consumer project's own release procedure. Released consumers
must select the published Promptkit tag through ordinary module resolution,
without a workspace, replacement, vendored Promptkit source, or unpublished
revision.
## Policy Changes

222
docs/roadmap/documentation.md Normal file
View File

@@ -0,0 +1,222 @@
# Documentation Hardening Roadmap
## Purpose
This roadmap coordinates a focused pass over Promptkit's documentation and
public contract. The work should close the gaps identified by the documentation
audit, strengthen canonical ownership, and leave consumers and contributors
with guidance that is accurate, navigable, and proportionate to their needs.
This is planning material, not a description of implemented behavior. Follow
the [documentation policy](../policy/documentation.md) throughout the work and
update current-state documents only when their claims are supported by the
implementation and tests.
## Scope And Principles
The work covers public GoDoc, consumer guidance, maintained examples, format
and integration references, architecture ownership, roadmap lifecycle, and
documentation validation.
- Resolve ambiguous behavior before documenting it as a contract.
- Keep exact exported API semantics in Go declarations and GoDoc.
- Keep task-oriented guidance in consumer documents and complete runnable
artifacts in `examples/`.
- Put security-relevant consumer responsibilities at the public boundary, not
only in internal contributor documents.
- Remove duplicated ownership instead of synchronizing parallel references.
- Make documentation checks reproducible where practical.
- Preserve existing behavior unless a stage explicitly selects and tests an
API change.
The roadmap does not add new Promptkit features, redesign framework formats,
or implement ideas from [the future feature catalog](future.md). If resolving
an ambiguity requires a behavioral change, treat that change as a separately
reviewable implementation unit and update its canonical documentation in the
same unit.
## Stage 1: Resolve Public Contract Questions
Before expanding prose, decide the intended contract for exported behavior
that is currently ambiguous.
- [x] Decide whether `RunRequest.Metadata` has a supported observable purpose.
Define its propagation and ownership, or remove or deprecate it through an
intentional public API change.
- [x] Decide and test whether one `Engine` supports concurrent `Prepare` and
`Run` calls.
- [x] Decide how repeated options of the same category behave, including
prompt, profile, schema, model-client, and artifact-reader options.
- [x] Define which public values have supported JSON representations.
- [x] Define JSON time units and omission behavior, including the relationship
between prepared-run and run-result durations.
- [x] Decide whether run IDs and exposed hashes have stable formats or must be
treated as opaque values.
- [x] Confirm the intended transport-timeout default and its zero or negative
configuration semantics.
- [x] Confirm the supported JSON Schema dialect and reference boundaries,
including whether remote references are allowed.
The selected exported API contracts are implemented and tested. Their durable
definitions now belong to the root package declarations and GoDoc.
One format-level decision remains here until Stage 5 moves it to the framework
format reference: JSON Schema uses Draft 2020-12, with that dialect selected
when `$schema` is omitted. Same-document fragments and relative references
contained by a directory or `fs.FS` schema root are supported. A single-file
source supports only references contained in that document. Absolute,
escaping, and remote references are rejected.
**Gate:** Each question has an explicit answer backed by existing behavior or
by an accepted implementation change and proportionate tests. No later stage
should invent a contract merely to fill a documentation gap.
## Stage 2: Make GoDoc The Canonical Public Contract
Strengthen the root package declarations so `go doc` is sufficient to
understand exact public behavior without relying on internal documents.
- [x] Add useful field-level GoDoc to configuration, request, profile,
execution-target, result, artifact, validation, structured-output, and model
client values.
- [x] Document required fields and nil, empty, and zero-value semantics.
- [x] Document override, replacement, profile-precedence, and copy-ownership
behavior where it belongs to the exported API.
- [x] Document credential inputs, redaction, and the values intentionally
excluded from serialization.
- [x] Give each public error sentinel an accurate comment and document the
supported `errors.Is` relationships.
- [x] Document engine concurrency and option-composition behavior selected in
Stage 1.
- [x] Document serialization, time, run-ID, and hash semantics selected in
Stage 1.
- [x] Review constructor, option, extension-interface, `Prepare`, and `Run`
GoDoc for complete failure and cancellation expectations.
Update the [consumer guide](../consumers/pkg-promptkit.md) to summarize and link
to these contracts instead of maintaining exhaustive copies of exported names
or exact semantics.
**Gate:** `go doc -all .` presents a coherent public contract, exported
declarations have accurate comments, and contract tests protect every newly
documented behavior whose compatibility risk warrants durable coverage.
## Stage 3: Improve Consumer Safety And Executable Guidance
Move consumer-relevant security boundaries to the places where consumers will
encounter them and add one representative execution workflow.
- [x] Explain in public GoDoc and the consumer guide that the default file
artifact reader accepts unrestricted caller-selected paths.
- [x] Make clear that Promptkit does not impose an application root, inbound
request-size policy, or untrusted-input security boundary.
- [x] Explain that rendered messages, artifact bodies, raw model output, and
validation details may be sensitive even when credentials are redacted.
- [x] Clarify the responsibilities of injected artifact readers and model
clients for cancellation, copying, logging, and secret handling.
- [x] Add a maintained offline `Run` example using an injected deterministic
model client, without credentials, live network access, or paid calls.
- [x] Link the consumer guide to the execution example and keep embedded
snippets smaller than the maintained artifact.
- [x] Decide whether the existing preparation example should remain separate
or share reusable fixtures without obscuring either workflow.
The preparation and execution examples remain separate, self-contained
workflows. Each keeps its own small prompt fixture so consumers can copy or run
one example without depending on the other.
**Gate:** Both preparation and execution have complete, secret-free,
deterministic consumer examples, and the consumer guide exposes the important
filesystem and data-sensitivity boundaries without leaking internal mechanics.
## Stage 4: Restore Canonical Ownership
Remove parallel definitions and make navigation follow the ownership model in
the documentation policy.
- [ ] Reduce the [architecture policy](../policy/architecture.md) to durable
boundaries, layers, dependency direction, invariants, and non-goals.
- [ ] Keep the exact implemented package and component inventory solely in the
[internal component overview](../internal/overview.md).
- [ ] Review the consumer guide's public error and option lists so they remain
task-oriented summaries rather than duplicate API references.
- [ ] Review internal documents for public-contract statements that should be
links to GoDoc or the format and integration owners.
- [ ] Reconcile the documentation policy's temporary-roadmap lifecycle with
the continuing idea-catalog role of `docs/roadmap/future.md`.
- [ ] Rephrase or link roadmap statements that depend on exact current API
behavior, particularly the runtime reasoning entry.
- [ ] Confirm that every document states its audience or purpose and links to
the canonical owner of adjacent topics.
**Gate:** Every authoritative fact has one clear owner, package inventory
changes no longer require edits to the architecture policy, and roadmaps
cannot be mistaken for current-state references.
## Stage 5: Refine Format And Integration References
Close compatibility gaps in the documents that own file formats and outbound
wire behavior.
- [ ] State or canonically link the exact session-ID limit enforced by the
OpenAI-compatible client.
- [ ] State the configured and default transport-timeout behavior without
referring to an unnamed internal default.
- [ ] Document the JSON Schema dialect and local, contained, and remote
reference behavior selected in Stage 1.
- [ ] Clarify structured-output naming and strictness when those values are
part of the public or integration contract.
- [ ] Add a caveat that built-in profiles are maintained configurations, not a
guarantee of continuing third-party model availability.
- [ ] Recheck every prompt, profile, schema, credential, request-body, response,
timeout, and precedence statement against its owning implementation and
tests.
**Gate:** A consumer can determine the supported file and wire compatibility
boundaries without consulting internal source code or relying on unspecified
defaults.
## Stage 6: Make Documentation Validation Reproducible
Align contributor and release procedures around a small, consistent set of
checks.
- [ ] Use one robust command for checking every tracked Go file with `gofmt`.
- [ ] Provide a repository-local or clearly documented command that validates
local Markdown targets and heading fragments.
- [ ] Decide how published external links are checked without making ordinary
validation depend on mutable network services.
- [ ] Reconcile the validation descriptions in the
[development guide](../development.md),
[testing policy](../policy/testing.md), and
[release procedure](../release.md) so one document owns each requirement.
- [ ] Ensure example validation covers every maintained example added by this
roadmap.
- [ ] Keep documentation-only validation proportionate while requiring full Go
validation when commands, examples, generated output, or checked behavior
changes.
**Gate:** A maintainer can run the documented formatting, link, example, Go,
and repository-hygiene checks exactly as written, with no hidden manual
procedure for local documentation.
## Completion Criteria
The roadmap is complete when:
- all Stage 1 contract questions are resolved;
- GoDoc is the authoritative and sufficient exported API reference;
- consumer guidance covers unrestricted file access and sensitive generated
data;
- maintained offline examples cover both `Prepare` and `Run`;
- architecture, inventory, consumer, internal, format, integration, and
roadmap documents follow their assigned ownership boundaries;
- schema, session, timeout, structured-output, and built-in-profile
compatibility statements are explicit;
- documentation validation is reproducible and consistent across contributor
and release workflows; and
- the complete maintainer validation passes.
After completion, move any durable decisions to GoDoc, policy, format,
integration, or ADR owners as appropriate. Remove this roadmap after incoming
links are updated; do not retain it as a second current-state reference.

122
docs/roadmap/future.md Normal file
View File

@@ -0,0 +1,122 @@
# Future Feature Ideas
## Purpose
This document catalogs reasonably specific ideas that may be useful in future
Promptkit development. It is an idea pool, not a commitment, schedule, or
description of current behavior.
Ideas belong here while they are worth retaining but have not been selected
for active development. Keep each entry at the level of intended capability,
consumer value, and important scope boundaries. Defer API design,
implementation details, sequencing, and acceptance criteria until an idea is
selected.
## Using This Catalog
- Add an idea when its purpose and likely value can be stated clearly.
- Keep entries independent enough that maintainers can evaluate and select
them individually.
- Note significant dependencies or boundary concerns, but do not turn entries
into implementation plans.
- Treat inclusion as an invitation to evaluate, not as approval or priority.
- When an idea is selected, move its active planning to a focused roadmap or,
when it requires a durable architectural decision, an ADR. Update
current-state documentation only when implementation lands.
- Remove ideas that are no longer relevant. Retain a rejected idea only when
its rationale is likely to prevent repeated reconsideration.
Future capabilities must continue to respect the
[architecture policy](../policy/architecture.md), particularly Promptkit's
role as an application-neutral library and its boundary with downstream
consumers.
## Ideas
### Extensible LLM backend registry
Introduce a registry that separates backend-specific connection,
authentication, and limited request defaults from model execution profiles.
Initial support would cover OpenAI-compatible backends and include a small
built-in catalog, potentially starting with OpenRouter. A profile could select
a backend while optionally overriding its default endpoint, and each backend
could name an optional environment variable for its API key without storing
the credential itself. Downstream consumers could register additional,
uniquely named backends, such as OpenAI or unauthenticated local-network
services, but could not replace built-in IDs. Model selection and generation
settings would remain profile concerns, and custom model clients would remain
available for behavior outside the registry's supported protocol.
### Backend-specific concurrency management
Extend the proposed LLM backend registry with optional per-backend concurrency
limits and bounded, buffered admission queues. Promptkit could then route
simultaneous generation requests according to backend capacity while
containing accidental runaway submission. Downstream consumers would continue
invoking synchronous `Run` calls, including concurrently from multiple
goroutines, and each admitted call would wait for and return its ordinary
result.
- Scope limits to an engine instance rather than hidden process-global state.
- Give different backend IDs independent capacity pools. A profile endpoint
override would remain part of its selected backend's pool.
- Configure active concurrency and waiting capacity separately. Concurrency
protects the backend, while queue capacity protects the process from
admitting an unbounded backlog.
- Give queue capacity a generous, configurable bounded default intended as a
safety ceiling for bugs or unintended loops rather than a routine
application constraint. Select an exact default during implementation
planning and measurement.
- Reject a call with a recognizable capacity error when its backend queue is
full rather than allowing it to wait outside the bounded queue.
- Admit requests before expensive preparation and artifact copying where
practical so queued work remains lightweight.
- Apply a limit to each actual generation request, including repair attempts,
without unnecessarily serializing prompt preparation.
- Make queued and active waits respect caller cancellation and deadlines.
- Treat concurrency as backend policy rather than a profile-level model
setting.
- Keep the queue ephemeral and in-process, with no survival guarantee across
engine or process shutdown.
- Preserve the existing execution model as far as practical. Durable jobs,
polling, priorities, application worker lifecycle, retries, and
cross-process coordination would be separate future capabilities.
### Explicit per-run session and reasoning controls
Allow consumers to associate a session ID with each run and to inherit,
replace, or explicitly disable the reasoning effort configured by its selected
profile. Prompt definitions can currently derive a session ID from a template,
and a non-empty runtime `ReasoningEffort` can replace the profile value, but
there is no direct request-level session ID and an empty reasoning value means
that no override was supplied. These controls would let consumers reuse one
prompt and model profile across sessions and reasoning levels without
maintaining duplicate definitions.
- Preserve a prompt's session ID template and a profile's reasoning effort as
reusable defaults.
- Let a directly supplied per-run session ID take precedence over a rendered
prompt default while retaining the existing validation limit and outbound
representation.
- Distinguish an omitted runtime reasoning choice from an explicit request to
disable reasoning.
- Ensure disabling reasoning omits the corresponding provider request setting
rather than relying on a provider-specific magic value.
- Keep the effective session ID and reasoning choice visible in prepared and
run metadata and available to injected model clients without introducing
additional prompt or profile selection mechanisms.
## Entry Format
Use a short heading followed by a concise summary. Add focused bullets when
they help preserve important scope boundaries without becoming an
implementation plan:
```markdown
### Idea name
Describe the intended capability, who benefits, and the most important scope
boundary or dependency.
- Optionally record an important behavior or boundary.
```

165
engine.go
View File

@@ -22,42 +22,93 @@ import (
"gitea.maximumdirect.net/eric/promptkit/internal/validate"
)
// ErrInvalidConfig indicates invalid public engine configuration.
// ErrInvalidConfig identifies invalid engine construction, including missing
// required configuration, invalid options, and a nil Engine receiver.
var ErrInvalidConfig = errors.New("invalid engine configuration")
var (
ErrInvalidRequest = errors.New("invalid run request")
ErrPromptNotFound = errors.New("prompt not found")
ErrProfileNotFound = errors.New("profile not found")
ErrProfileRequired = errors.New("profile selection is required")
ErrPromptLoad = errors.New("failed to load prompt definition")
ErrProfileLoad = errors.New("failed to load execution profile")
// ErrInvalidRequest identifies a request whose required values, overrides,
// credentials, or effective settings are invalid.
ErrInvalidRequest = errors.New("invalid run request")
// ErrPromptNotFound identifies a requested prompt ID or version that is not
// present in the selected prompt source. It does not also match
// ErrPromptLoad.
ErrPromptNotFound = errors.New("prompt not found")
// ErrProfileNotFound identifies a selected profile ID that is absent from
// every configured profile source. It does not also match ErrProfileLoad.
ErrProfileNotFound = errors.New("profile not found")
// ErrProfileRequired identifies a request for which neither RunRequest.ProfileID
// nor the selected prompt's default profile is present. Such an error also
// matches ErrInvalidRequest.
ErrProfileRequired = errors.New("profile selection is required")
// ErrPromptLoad identifies a failure to read, decode, validate, select, or
// hash a prompt definition, except for the not-found case represented by
// ErrPromptNotFound.
ErrPromptLoad = errors.New("failed to load prompt definition")
// ErrProfileLoad identifies a failure to read, decode, validate, or select
// an execution profile, except for the not-found case represented by
// ErrProfileNotFound.
ErrProfileLoad = errors.New("failed to load execution profile")
// ErrAPIKeyEnvMissing identifies an APIKeyEnv whose environment variable is
// unset or empty when no direct RunRequest.APIKey takes precedence. Such an
// error also matches ErrInvalidRequest.
ErrAPIKeyEnvMissing = errors.New("api_key_env points to an unset environment variable")
ErrArtifactLoad = errors.New("failed to load artifact")
ErrPromptRender = errors.New("failed to render prompt")
ErrLLMGenerate = errors.New("failed to generate output")
ErrValidation = errors.New("failed to validate output")
// ErrArtifactLoad identifies a failure to resolve an input artifact. Errors
// returned by an injected ArtifactReader remain available through errors.Is.
ErrArtifactLoad = errors.New("failed to load artifact")
// ErrPromptRender identifies a failure to render prompt messages or the
// session ID from the resolved inputs and variables.
ErrPromptRender = errors.New("failed to render prompt")
// ErrLLMGenerate identifies a model-client failure or a nil successful
// response. Errors returned by an injected LLMClient remain available
// through errors.Is.
ErrLLMGenerate = errors.New("failed to generate output")
// ErrValidation identifies an operational failure to load or compile a
// schema or validate output. A completed validation whose Status is
// ValidationFailed is returned in RunResult without this error.
ErrValidation = errors.New("failed to validate output")
)
// Engine prepares and runs Promptkit prompt requests.
//
// An Engine is safe for concurrent calls to [Engine.Prepare] and [Engine.Run].
// Injected collaborators may consequently be invoked concurrently.
type Engine struct {
runner *usecase.Runner
}
// Config configures a public Promptkit engine.
// Config selects the directory-backed sources and built-in model-client
// transport used by [NewEngine]. Config has no stable JSON representation.
type Config struct {
PromptDir string
// PromptDir is the directory searched recursively for prompt definitions.
// It is required unless a WithPromptFS or WithPromptFile option supplies the
// prompt source.
PromptDir string
// ProfileDir is an optional directory whose profiles take precedence over
// embedded built-in profiles. An empty value selects only built-ins unless
// profile options are also supplied.
ProfileDir string
SchemaDir string
// SchemaDir is the root for JSON Schema files. An empty value uses the
// current directory. WithSchemaFS or WithSchemaFile replaces this source.
SchemaDir string
// Timeout is the transport-wide safety cap for the built-in LLM client
// when HTTPClient is absent or has a non-positive timeout.
// when HTTPClient is absent or has a non-positive timeout. A zero or negative
// value selects the 10-minute default.
Timeout time.Duration
// HTTPClient is cloned for the built-in LLM client. Its positive Timeout
// takes precedence over Config.Timeout as the transport-wide safety cap.
// takes precedence over Timeout. A zero or negative client Timeout inherits
// Timeout or the 10-minute default. The supplied client is not mutated. This
// field is ignored when WithLLMClient is used.
HTTPClient *http.Client
}
// Option customizes engine construction.
//
// NewEngine applies options in argument order and ignores nil options. Within
// each prompt-source, profile-source, in-memory-profile, schema-source,
// model-client, and artifact-reader category, the last non-nil valid option
// replaces earlier options in that category. An invalid option fails
// construction even if a later option would replace it.
type Option interface {
apply(*engineOptions) error
}
@@ -82,7 +133,10 @@ type engineOptions struct {
artifactSource bool
}
// WithLLMClient injects a custom LLM client for execution.
// WithLLMClient replaces the built-in model client used by [Engine.Run].
//
// A nil client makes NewEngine fail with ErrInvalidConfig. The client may be
// called concurrently and is not used by [Engine.Prepare].
func WithLLMClient(client LLMClient) Option {
return optionFunc(func(options *engineOptions) error {
if client == nil {
@@ -93,7 +147,11 @@ func WithLLMClient(client LLMClient) Option {
})
}
// WithArtifactReader injects a reader for every input artifact reference.
// WithArtifactReader replaces the default reader for every input artifact
// reference, regardless of its ArtifactRef.Type.
//
// A nil reader makes NewEngine fail with ErrInvalidConfig. The reader may be
// called concurrently.
func WithArtifactReader(reader ArtifactReader) Option {
return optionFunc(func(options *engineOptions) error {
if reader == nil {
@@ -109,6 +167,9 @@ func WithArtifactReader(reader ArtifactReader) Option {
//
// The source uses the same strict prompt YAML rules as configured prompt
// directories, and prompt content_file paths resolve within this source.
// fsys must be non-nil and root must be non-empty; otherwise NewEngine fails
// with ErrInvalidConfig. This option replaces Config.PromptDir and earlier
// prompt-source options.
func WithPromptFS(fsys fs.FS, root string) Option {
return optionFunc(func(options *engineOptions) error {
if fsys == nil {
@@ -125,7 +186,9 @@ func WithPromptFS(fsys fs.FS, root string) Option {
// WithPromptFile loads prompt definitions from the single prompt file at path.
//
// Relative prompt content_file paths resolve from the file's directory.
// Relative prompt content_file paths resolve from the file's directory. path
// must name an existing non-directory file when NewEngine applies the option.
// This option replaces Config.PromptDir and earlier prompt-source options.
func WithPromptFile(path string) Option {
return optionFunc(func(options *engineOptions) error {
fsys, root, err := fileSource(path)
@@ -142,6 +205,10 @@ func WithPromptFile(path string) Option {
//
// Profiles from this source overlay built-in profiles. Profile YAML must use
// api_key_env for environment-based credentials; raw API keys are rejected.
// fsys must be non-nil and root must be non-empty; otherwise NewEngine fails
// with ErrInvalidConfig. This option replaces Config.ProfileDir and earlier
// file or FS profile-source options, but remains below WithProfiles in
// precedence.
func WithProfileFS(fsys fs.FS, root string) Option {
return optionFunc(func(options *engineOptions) error {
if fsys == nil {
@@ -159,7 +226,10 @@ func WithProfileFS(fsys fs.FS, root string) Option {
// WithProfileFile loads execution profiles from the single profile file at path.
//
// The profile overlays built-in profiles. Profile YAML must use api_key_env for
// environment-based credentials; raw API keys are rejected.
// environment-based credentials; raw API keys are rejected. path must name an
// existing non-directory file when NewEngine applies the option. This option
// replaces Config.ProfileDir and earlier file or FS profile-source options,
// but remains below WithProfiles in precedence.
func WithProfileFile(path string) Option {
return optionFunc(func(options *engineOptions) error {
fsys, root, err := fileSource(path)
@@ -174,6 +244,11 @@ func WithProfileFile(path string) Option {
// WithProfiles configures in-memory profiles that take precedence over
// configured profile files and built-in profiles.
//
// NewEngine validates and copies every profile. IDs must be unique within one
// call. An invalid profile, duplicate ID, or unsupported ExtraParams value
// makes construction fail with ErrInvalidConfig. Repeating WithProfiles
// replaces the complete earlier in-memory set rather than merging it.
func WithProfiles(profiles ...Profile) Option {
return optionFunc(func(options *engineOptions) error {
repo, err := newMemoryProfileRepository(profiles)
@@ -189,7 +264,9 @@ func WithProfiles(profiles ...Profile) Option {
// WithSchemaFS loads JSON Schema documents from fsys under root.
//
// Prompt schema_path values resolve within this source when schema validation
// or structured output is requested.
// or structured output is requested. fsys must be non-nil and root must be
// non-empty; otherwise NewEngine fails with ErrInvalidConfig. This option
// replaces Config.SchemaDir and earlier schema-source options.
func WithSchemaFS(fsys fs.FS, root string) Option {
return optionFunc(func(options *engineOptions) error {
if fsys == nil {
@@ -206,7 +283,9 @@ func WithSchemaFS(fsys fs.FS, root string) Option {
// WithSchemaFile loads JSON Schema documents from the single schema file at path.
//
// Prompt schema_path values refer to the file's base name.
// Prompt schema_path values refer to the file's base name. path must name an
// existing non-directory file when NewEngine applies the option. This option
// replaces Config.SchemaDir and earlier schema-source options.
func WithSchemaFile(path string) Option {
return optionFunc(func(options *engineOptions) error {
fsys, root, err := fileSource(path)
@@ -220,6 +299,15 @@ func WithSchemaFile(path string) Option {
}
// NewEngine constructs an Engine from configuration and options.
//
// Options are applied in order according to [Option]. PromptDir is required
// unless a prompt-source option is present. Construction validates option
// arguments and in-memory profiles but defers reading and validating prompt,
// file-backed profile, and schema contents until Prepare or Run needs them.
//
// NewEngine returns an error matching ErrInvalidConfig for invalid
// configuration or options. It does not perform model requests or require
// credentials.
func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
var options engineOptions
for _, opt := range opts {
@@ -305,7 +393,22 @@ func fileSource(name string) (fs.FS, string, error) {
return os.DirFS(dir), filepath.ToSlash(base), nil
}
// Prepare resolves a prompt request without calling an LLM.
// Prepare resolves and renders a prompt request without calling an LLM.
//
// Prepare selects the prompt and profile, resolves effective execution
// settings and the output contract, loads and hashes inputs, loads structured
// output schema metadata when required, and renders the session ID and
// messages. The returned PreparedRun is owned by the caller and never contains
// a resolved API-key value, model output, or validation result.
//
// A nil Engine returns an error matching ErrInvalidConfig. Request and
// preparation failures may match ErrInvalidRequest, ErrPromptNotFound,
// ErrPromptLoad, ErrProfileNotFound, ErrProfileLoad, ErrProfileRequired,
// ErrAPIKeyEnvMissing, ErrArtifactLoad, ErrPromptRender, or ErrValidation as
// applicable. Cancellation is passed to the active collaborator and is
// reported in the applicable operation category; no general errors.Is
// relationship to ctx.Err is promised. Prepare returns no partial result on
// error.
func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, error) {
if e == nil || e.runner == nil {
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
@@ -323,7 +426,21 @@ func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, err
return fromDomainPreparedRun(prepared), nil
}
// Run executes a prompt request and returns the generated artifact and metadata.
// Run prepares a request, invokes the configured LLMClient, and validates the
// generated output.
//
// A content-validation failure is a successful run whose
// RunResult.Validation has Status ValidationFailed. An inability to perform
// validation returns an error matching ErrValidation and no partial result.
// The public Engine does not perform output repair, so validation is
// single-pass even when OutputContract.RepairAttempts is positive.
//
// Run can return every error category documented by [Engine.Prepare], plus
// ErrLLMGenerate. Errors from injected clients remain available through
// errors.Is. Cancellation is passed through the active collaborator and is
// reported in the applicable operation category; no general errors.Is
// relationship to ctx.Err is promised. A nil Engine returns ErrInvalidConfig.
// Run returns no partial result on error.
func (e *Engine) Run(ctx context.Context, req RunRequest) (*RunResult, error) {
if e == nil || e.runner == nil {
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)

28
engine_test.go
View File

@@ -1020,6 +1020,7 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
client promptkit.LLMClient
schemaDir string
want error
notWant error
}{
{
name: "invalid request",
@@ -1028,10 +1029,11 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
want: promptkit.ErrInvalidRequest,
},
{
name: "prompt not found",
req: promptkit.RunRequest{PromptID: "missing.prompt"},
client: &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}},
want: promptkit.ErrPromptNotFound,
name: "prompt not found",
req: promptkit.RunRequest{PromptID: "missing.prompt"},
client: &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}},
want: promptkit.ErrPromptNotFound,
notWant: promptkit.ErrPromptLoad,
},
{
name: "profile not found",
@@ -1042,8 +1044,9 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
"transcript": promptkit.Inline("Rin opens the gate."),
},
},
client: &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}},
want: promptkit.ErrProfileNotFound,
client: &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}},
want: promptkit.ErrProfileNotFound,
notWant: promptkit.ErrProfileLoad,
},
{
name: "artifact load",
@@ -1114,6 +1117,9 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
if !errors.Is(err, tc.want) {
t.Fatalf("expected errors.Is(%v), got %v", tc.want, err)
}
if tc.notWant != nil && errors.Is(err, tc.notWant) {
t.Fatalf("did not expect errors.Is(%v), got %v", tc.notWant, err)
}
})
}
}
@@ -1699,6 +1705,16 @@ func TestEngineRunLayersTransportAndGenerationTimeouts(t *testing.T) {
configTimeout: 5 * time.Second,
wantRemainingAtRequest: 5 * time.Second,
},
{
name: "zero configuration uses ten minute transport default",
wantRemainingAtRequest: 10 * time.Minute,
},
{
name: "negative configuration uses ten minute transport default",
configTimeout: -2 * time.Second,
suppliedClientTimeout: -3 * time.Second,
wantRemainingAtRequest: 10 * time.Minute,
},
{
name: "profile deadline is shorter than transport cap",
suppliedClientTimeout: 6 * time.Second,

81
examples/go-library/run/main.go Normal file
View File

@@ -0,0 +1,81 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"gitea.maximumdirect.net/eric/promptkit"
)
type deterministicClient struct{}
func (deterministicClient) Generate(
ctx context.Context,
_ promptkit.GenerateRequest,
) (*promptkit.GenerateResponse, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
return &promptkit.GenerateResponse{
Content: "Ada finished the migration review.",
Usage: promptkit.TokenUsage{
PromptTokens: 12,
CompletionTokens: 6,
TotalTokens: 18,
},
}, nil
}
type summary struct {
Output string `json:"output"`
ValidationStatus promptkit.ValidationStatus `json:"validation_status"`
IsValid bool `json:"is_valid"`
Model string `json:"model"`
TotalTokens int `json:"total_tokens"`
}
func main() {
engine, err := promptkit.NewEngine(
promptkit.Config{},
promptkit.WithPromptFile("examples/go-library/run/prompt.yaml"),
promptkit.WithProfiles(promptkit.Profile{
ID: "offline-example",
Endpoint: "https://example.invalid/v1",
Model: "offline-model",
}),
promptkit.WithLLMClient(deterministicClient{}),
)
if err != nil {
exit(err)
}
result, err := engine.Run(context.Background(), promptkit.RunRequest{
PromptID: "example.run",
Inputs: map[string]promptkit.ArtifactRef{
"note": promptkit.Inline("Ada finished the migration review."),
},
})
if err != nil {
exit(err)
}
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", " ")
if err := encoder.Encode(summary{
Output: result.RawOutput,
ValidationStatus: result.Validation.Status,
IsValid: result.Validation.IsValid,
Model: result.ModelName,
TotalTokens: result.Usage.TotalTokens,
}); err != nil {
exit(err)
}
}
func exit(err error) {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}

16
examples/go-library/run/prompt.yaml Normal file
View File

@@ -0,0 +1,16 @@
id: example.run
version: "1.0.0"
default_profile: offline-example
description: Run a prompt with a deterministic injected model client.
inputs:
- name: note
required: true
content_type: text/plain
messages:
- role: system
content: Summarize the note in one sentence.
- role: user
content: '{{input "note"}}'
output:
format: text
validation_mode: basic

17
formatting.go
View File

@@ -2,19 +2,23 @@ package promptkit
import "fmt"
// String returns a concise request summary without exposing direct API keys.
// String returns a concise request summary without exposing the direct API key
// or input and variable contents. Reflection-based formatting does not carry
// this guarantee.
func (r RunRequest) String() string {
return r.redactedString()
}
// GoString returns a concise request summary without exposing direct API keys.
// GoString returns a concise request summary without exposing the direct API
// key or input and variable contents. Reflection-based formatting does not
// carry this guarantee.
func (r RunRequest) GoString() string {
return r.redactedString()
}
func (r RunRequest) redactedString() string {
return fmt.Sprintf(
"promptkit.RunRequest{PromptID:%q PromptVersion:%q ProfileID:%q APIKeySet:%t Inputs:%d Vars:%d ExecutionSet:%t ValidationSet:%t Metadata:%d}",
"promptkit.RunRequest{PromptID:%q PromptVersion:%q ProfileID:%q APIKeySet:%t Inputs:%d Vars:%d ExecutionSet:%t ValidationSet:%t}",
r.PromptID,
r.PromptVersion,
r.ProfileID,
@@ -23,18 +27,19 @@ func (r RunRequest) redactedString() string {
len(r.Vars),
r.Execution != nil,
r.Validation != nil,
len(r.Metadata),
)
}
// String returns a concise request summary without exposing direct API keys or
// rendered prompt content.
// rendered prompt content. Reflection-based formatting does not carry this
// guarantee.
func (r GenerateRequest) String() string {
return r.redactedString()
}
// GoString returns a concise request summary without exposing direct API keys or
// rendered prompt content.
// rendered prompt content. Reflection-based formatting does not carry this
// guarantee.
func (r GenerateRequest) GoString() string {
return r.redactedString()
}

1
internal/domain/domain.go
View File

@@ -68,7 +68,6 @@ type RunRequest struct {
Vars map[string]string
Execution *ExecutionTargetOverride
Validation *OutputContract
Metadata map[string]string
}
// RunResult represents the complete result of a prompt execution run.

176
internal/validate/standard_validator.go
View File

@@ -6,6 +6,7 @@ import (
"errors"
"fmt"
"io/fs"
"net/url"
"os"
"path"
"path/filepath"
@@ -16,6 +17,8 @@ import (
"github.com/santhosh-tekuri/jsonschema/v6"
)
const jsonSchemaDraft2020 = "https://json-schema.org/draft/2020-12/schema"
// StandardValidator provides basic, JSON, and JSON Schema output validation.
type StandardValidator struct {
schemaBaseDir string
@@ -121,7 +124,11 @@ func (v *StandardValidator) validateJSONSchema(instance any, schemaPath string)
return nil, err
}
compiler := jsonschema.NewCompiler()
schemaRoot, err := v.schemaRoot()
if err != nil {
return nil, err
}
compiler := newSchemaCompiler(standardSchemaLoader{root: schemaRoot})
schema, err := compiler.Compile(resolvedSchemaPath)
if err != nil {
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", resolvedSchemaPath, err)
@@ -140,7 +147,10 @@ func (v *FSValidator) validateJSONSchema(instance any, schemaPath string) ([]str
}
resourceURL := fsSchemaResourceURL(schemaName)
compiler := jsonschema.NewCompiler()
if err := validateSchemaDialect(schemaDoc); err != nil {
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", schemaName, err)
}
compiler := newSchemaCompiler(fsSchemaLoader{fsys: v.fsys, root: filecatalog.CleanFSRoot(v.root)})
if err := compiler.AddResource(resourceURL, schemaDoc); err != nil {
return nil, fmt.Errorf("failed to register JSON schema %q: %w", schemaName, err)
}
@@ -184,6 +194,9 @@ func (v *StandardValidator) LoadSchemaDocument(ctx context.Context, schemaPath s
if err := json.Unmarshal(raw, &doc); err != nil {
return nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
}
if err := validateSchemaDialect(doc); err != nil {
return nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
}
return doc, nil
}
@@ -206,12 +219,14 @@ func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error)
return "", errors.New("schema path is required for json_schema validation")
}
resolved := schemaPath
if !filepath.IsAbs(schemaPath) {
resolved = filepath.Join(v.schemaBaseDir, schemaPath)
root, err := v.schemaRoot()
if err != nil {
return "", err
}
resolved, err := containedFilesystemPath(root, schemaPath)
if err != nil {
return "", err
}
resolved = filepath.Clean(resolved)
if _, err := os.Stat(resolved); err != nil {
return "", fmt.Errorf("failed to access schema file %q: %w", resolved, err)
}
@@ -219,6 +234,22 @@ func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error)
return resolved, nil
}
func (v *StandardValidator) schemaRoot() (string, error) {
root := v.schemaBaseDir
if strings.TrimSpace(root) == "" {
root = "."
}
absolute, err := filepath.Abs(root)
if err != nil {
return "", fmt.Errorf("failed to resolve schema source %q: %w", root, err)
}
resolved, err := filepath.EvalSymlinks(absolute)
if err != nil {
return "", fmt.Errorf("failed to access schema source %q: %w", root, err)
}
return resolved, nil
}
func (v *FSValidator) loadSchemaDocument(schemaPath string) (string, any, error) {
resolved, err := v.resolveSchemaPath(schemaPath)
if err != nil {
@@ -234,6 +265,9 @@ func (v *FSValidator) loadSchemaDocument(schemaPath string) (string, any, error)
if err := json.Unmarshal(raw, &doc); err != nil {
return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
}
if err := validateSchemaDialect(doc); err != nil {
return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
}
return resolved, doc, nil
}
@@ -290,3 +324,131 @@ func cleanSchemaFSPath(schemaPath string) (string, error) {
func fsSchemaResourceURL(schemaName string) string {
return "promptkit-schema:///" + strings.TrimPrefix(path.Clean(schemaName), "/")
}
func newSchemaCompiler(loader jsonschema.URLLoader) *jsonschema.Compiler {
compiler := jsonschema.NewCompiler()
compiler.DefaultDraft(jsonschema.Draft2020)
compiler.UseLoader(loader)
return compiler
}
func validateSchemaDialect(doc any) error {
object, ok := doc.(map[string]any)
if !ok {
return nil
}
value, ok := object["$schema"]
if !ok {
return nil
}
dialect, ok := value.(string)
if !ok {
return errors.New("$schema must be a string")
}
if dialect != jsonSchemaDraft2020 && dialect != jsonSchemaDraft2020+"#" {
return fmt.Errorf("unsupported JSON Schema dialect %q; expected %q", dialect, jsonSchemaDraft2020)
}
return nil
}
type standardSchemaLoader struct {
root string
}
func (l standardSchemaLoader) Load(resourceURL string) (any, error) {
fileName, err := (jsonschema.FileLoader{}).ToFile(resourceURL)
if err != nil {
return nil, fmt.Errorf("schema reference %q is not a contained file reference: %w", resourceURL, err)
}
resolved, err := containedFilesystemPath(l.root, fileName)
if err != nil {
return nil, err
}
return loadJSONSchemaFile(resolved)
}
func containedFilesystemPath(root, name string) (string, error) {
candidate := name
if !filepath.IsAbs(candidate) {
candidate = filepath.Join(root, candidate)
}
candidate, err := filepath.Abs(candidate)
if err != nil {
return "", fmt.Errorf("failed to resolve schema path %q: %w", name, err)
}
candidate, err = filepath.EvalSymlinks(candidate)
if err != nil {
return "", fmt.Errorf("failed to access schema file %q: %w", candidate, err)
}
relative, err := filepath.Rel(root, candidate)
if err != nil {
return "", fmt.Errorf("failed to compare schema path %q with source root: %w", candidate, err)
}
if relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
return "", fmt.Errorf("schema path %q escapes source root", name)
}
return candidate, nil
}
func loadJSONSchemaFile(name string) (any, error) {
raw, err := os.ReadFile(name)
if err != nil {
return nil, err
}
var doc any
if err := json.Unmarshal(raw, &doc); err != nil {
return nil, err
}
if err := validateSchemaDialect(doc); err != nil {
return nil, err
}
return doc, nil
}
type fsSchemaLoader struct {
fsys fs.FS
root string
}
func (l fsSchemaLoader) Load(resourceURL string) (any, error) {
parsed, err := url.Parse(resourceURL)
if err != nil {
return nil, fmt.Errorf("invalid schema reference %q: %w", resourceURL, err)
}
if parsed.Scheme != "promptkit-schema" || parsed.Host != "" {
return nil, fmt.Errorf("schema reference %q is not allowed", resourceURL)
}
name, err := url.PathUnescape(strings.TrimPrefix(parsed.Path, "/"))
if err != nil {
return nil, fmt.Errorf("invalid schema reference %q: %w", resourceURL, err)
}
name = path.Clean(name)
if l.root == "." {
if strings.HasPrefix(name, "../") || name == ".." {
return nil, fmt.Errorf("schema reference %q escapes source root", resourceURL)
}
} else if name != l.root && !strings.HasPrefix(name, l.root+"/") {
return nil, fmt.Errorf("schema reference %q escapes source root", resourceURL)
}
rootInfo, err := fs.Stat(l.fsys, l.root)
if err != nil {
return nil, err
}
if !rootInfo.IsDir() && name != l.root {
return nil, fmt.Errorf("schema reference %q is outside the configured schema file", resourceURL)
}
raw, err := fs.ReadFile(l.fsys, name)
if err != nil {
return nil, err
}
var doc any
if err := json.Unmarshal(raw, &doc); err != nil {
return nil, err
}
if err := validateSchemaDialect(doc); err != nil {
return nil, err
}
return doc, nil
}

137
internal/validate/standard_validator_test.go
View File

@@ -2,8 +2,10 @@ package validate
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"testing/fstest"
@@ -411,3 +413,138 @@ func TestFSValidatorLoadSchemaDocument(t *testing.T) {
t.Fatalf("unexpected schema document: %#v", doc)
}
}
func TestStandardValidatorJSONSchemaReferenceBoundaries(t *testing.T) {
root := t.TempDir()
if err := os.WriteFile(filepath.Join(root, "child.json"), []byte(`{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "string"
}`), 0o644); err != nil {
t.Fatal(err)
}
tests := []struct {
name string
reference string
wantError string
writeOuter bool
}{
{name: "contained relative reference", reference: "child.json"},
{name: "remote reference", reference: "https://example.test/schema.json", wantError: "not a contained file reference"},
{name: "escaping reference", reference: "../outside.json", wantError: "escapes source root", writeOuter: true},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if tc.writeOuter {
if err := os.WriteFile(filepath.Join(filepath.Dir(root), "outside.json"), []byte(`{"type":"string"}`), 0o644); err != nil {
t.Fatal(err)
}
}
schema := `{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$ref": ` + strconv.Quote(tc.reference) + `
}`
if err := os.WriteFile(filepath.Join(root, "root.json"), []byte(schema), 0o644); err != nil {
t.Fatal(err)
}
v := NewStandardValidator(root)
result, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`"value"`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "root.json",
})
if tc.wantError == "" {
if err != nil || !result.IsValid {
t.Fatalf("expected contained reference to validate, got result=%#v error=%v", result, err)
}
return
}
if err == nil || !strings.Contains(err.Error(), tc.wantError) {
t.Fatalf("expected error containing %q, got %v", tc.wantError, err)
}
})
}
}
func TestFSValidatorJSONSchemaReferenceBoundaries(t *testing.T) {
tests := []struct {
name string
reference string
wantError string
}{
{name: "same document fragment", reference: "#/$defs/value"},
{name: "contained relative reference", reference: "child.json"},
{name: "remote reference", reference: "https://example.test/schema.json", wantError: "is not allowed"},
{name: "escaping reference", reference: "../outside.json", wantError: "escapes source root"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
rootSchema := `{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$defs": {"value": {"type": "string"}},
"$ref": ` + strconv.Quote(tc.reference) + `
}`
v := NewFSValidator(fstest.MapFS{
"schemas/root.json": &fstest.MapFile{Data: []byte(rootSchema)},
"schemas/child.json": &fstest.MapFile{Data: []byte(`{"type":"string"}`)},
"outside.json": &fstest.MapFile{Data: []byte(`{"type":"string"}`)},
}, "schemas")
result, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`"value"`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "root.json",
})
if tc.wantError == "" {
if err != nil || !result.IsValid {
t.Fatalf("expected supported reference to validate, got result=%#v error=%v", result, err)
}
return
}
if err == nil || !strings.Contains(err.Error(), tc.wantError) {
t.Fatalf("expected error containing %q, got %v", tc.wantError, err)
}
})
}
}
func TestJSONSchemaDialectIsDraft2020(t *testing.T) {
tests := []struct {
name string
dialect string
wantError bool
}{
{name: "omitted uses supported default"},
{name: "draft 2020-12", dialect: "https://json-schema.org/draft/2020-12/schema"},
{name: "draft 7 rejected", dialect: "http://json-schema.org/draft-07/schema#", wantError: true},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
schema := map[string]any{"type": "object"}
if tc.dialect != "" {
schema["$schema"] = tc.dialect
}
data, err := json.Marshal(schema)
if err != nil {
t.Fatal(err)
}
v := NewFSValidator(fstest.MapFS{
"schema.json": &fstest.MapFile{Data: data},
}, ".")
_, err = v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{}`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "schema.json",
})
if tc.wantError {
if err == nil || !strings.Contains(err.Error(), "unsupported JSON Schema dialect") {
t.Fatalf("expected unsupported-dialect error, got %v", err)
}
return
}
if err != nil {
t.Fatalf("expected supported dialect, got %v", err)
}
})
}
}

154
json.go Normal file
View File

@@ -0,0 +1,154 @@
package promptkit
import (
"encoding/json"
"time"
)
// MarshalJSON implements json.Marshaler for PreparedRun. It uses RFC 3339
// timestamps, integer duration_ms, and omits zero timing values.
func (r PreparedRun) MarshalJSON() ([]byte, error) {
var startTime, endTime *time.Time
if !r.StartTime.IsZero() {
startTime = &r.StartTime
}
if !r.EndTime.IsZero() {
endTime = &r.EndTime
}
var durationMS *int64
if r.DurationMS != 0 {
durationMS = &r.DurationMS
}
return json.Marshal(struct {
PromptID string `json:"prompt_id"`
PromptVersion string `json:"prompt_version,omitempty"`
PromptHash string `json:"prompt_hash,omitempty"`
SelectedProfileID string `json:"selected_profile_id"`
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
OutputContract OutputContract `json:"output_contract"`
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
InputHashes map[string]string `json:"input_hashes,omitempty"`
SessionID string `json:"session_id,omitempty"`
RenderedPromptHash string `json:"rendered_prompt_hash"`
Messages []RenderedMessage `json:"messages"`
StartTime *time.Time `json:"start_time,omitempty"`
EndTime *time.Time `json:"end_time,omitempty"`
DurationMS *int64 `json:"duration_ms,omitempty"`
}{
PromptID: r.PromptID,
PromptVersion: r.PromptVersion,
PromptHash: r.PromptHash,
SelectedProfileID: r.SelectedProfileID,
EffectiveModelParams: r.EffectiveModelParams,
OutputContract: r.OutputContract,
StructuredOutput: r.StructuredOutput,
InputHashes: r.InputHashes,
SessionID: r.SessionID,
RenderedPromptHash: r.RenderedPromptHash,
Messages: r.Messages,
StartTime: startTime,
EndTime: endTime,
DurationMS: durationMS,
})
}
// MarshalJSON implements json.Marshaler for RunResult. It encodes Duration as
// integer milliseconds in duration_ms and omits zero timing values.
func (r RunResult) MarshalJSON() ([]byte, error) {
var startTime, endTime *time.Time
if !r.StartTime.IsZero() {
startTime = &r.StartTime
}
if !r.EndTime.IsZero() {
endTime = &r.EndTime
}
var durationMS *int64
if r.Duration != 0 {
value := r.Duration.Milliseconds()
durationMS = &value
}
return json.Marshal(runResultJSON{
RunID: r.RunID,
Artifact: r.Artifact,
RawOutput: r.RawOutput,
Validation: r.Validation,
PromptID: r.PromptID,
PromptVersion: r.PromptVersion,
PromptHash: r.PromptHash,
RenderedPromptHash: r.RenderedPromptHash,
SelectedProfileID: r.SelectedProfileID,
ModelName: r.ModelName,
Endpoint: r.Endpoint,
EffectiveModelParams: r.EffectiveModelParams,
InputHashes: r.InputHashes,
Usage: r.Usage,
StartTime: startTime,
EndTime: endTime,
DurationMS: durationMS,
})
}
// UnmarshalJSON implements json.Unmarshaler for RunResult. It decodes
// duration_ms into Duration with millisecond precision.
func (r *RunResult) UnmarshalJSON(data []byte) error {
var wire runResultJSON
if err := json.Unmarshal(data, &wire); err != nil {
return err
}
*r = RunResult{
RunID: wire.RunID,
Artifact: wire.Artifact,
RawOutput: wire.RawOutput,
Validation: wire.Validation,
PromptID: wire.PromptID,
PromptVersion: wire.PromptVersion,
PromptHash: wire.PromptHash,
RenderedPromptHash: wire.RenderedPromptHash,
SelectedProfileID: wire.SelectedProfileID,
ModelName: wire.ModelName,
Endpoint: wire.Endpoint,
EffectiveModelParams: wire.EffectiveModelParams,
InputHashes: wire.InputHashes,
Usage: wire.Usage,
Duration: time.Duration(valueOrZero(wire.DurationMS)) * time.Millisecond,
}
if wire.StartTime != nil {
r.StartTime = *wire.StartTime
}
if wire.EndTime != nil {
r.EndTime = *wire.EndTime
}
return nil
}
type runResultJSON struct {
RunID string `json:"run_id"`
Artifact Artifact `json:"artifact"`
RawOutput string `json:"raw_output"`
Validation ValidationResult `json:"validation"`
PromptID string `json:"prompt_id"`
PromptVersion string `json:"prompt_version,omitempty"`
PromptHash string `json:"prompt_hash,omitempty"`
RenderedPromptHash string `json:"rendered_prompt_hash"`
SelectedProfileID string `json:"selected_profile_id"`
ModelName string `json:"model_name"`
Endpoint string `json:"endpoint"`
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
InputHashes map[string]string `json:"input_hashes,omitempty"`
Usage TokenUsage `json:"usage"`
StartTime *time.Time `json:"start_time,omitempty"`
EndTime *time.Time `json:"end_time,omitempty"`
DurationMS *int64 `json:"duration_ms,omitempty"`
}
func valueOrZero(value *int64) int64 {
if value == nil {
return 0
}
return *value
}

4
profiles.go
View File

@@ -16,6 +16,10 @@ import (
// It does not register global state, maintain a model catalog, or resolve
// credentials. If APIKeyRequired is true, callers satisfy it with
// RunRequest.APIKey. Raw API keys do not belong in profiles.
//
// The function copies the ExtraParams map itself but does not recursively copy
// nested values. Validation and a deep copy occur when NewEngine applies a
// WithProfiles option containing the returned Profile.
func OpenAICompatibleProfile(cfg OpenAICompatibleProfileConfig) Profile {
return Profile{
ID: cfg.ID,

377
public_contract_test.go Normal file
View File

@@ -0,0 +1,377 @@
package promptkit_test
import (
"context"
"encoding/json"
"fmt"
"strings"
"sync"
"sync/atomic"
"testing"
"testing/fstest"
"time"
"gitea.maximumdirect.net/eric/promptkit"
)
func TestPreparedRunJSONOmitsZeroTimingValues(t *testing.T) {
payload, err := json.Marshal(promptkit.PreparedRun{})
if err != nil {
t.Fatalf("marshal prepared run: %v", err)
}
for _, field := range []string{"start_time", "end_time", "duration_ms"} {
if strings.Contains(string(payload), `"`+field+`"`) {
t.Fatalf("expected zero %s to be omitted, got %s", field, payload)
}
}
}
func TestPreparedRunJSONTimingRoundTrips(t *testing.T) {
start := time.Date(2026, time.July, 29, 12, 0, 0, 0, time.UTC)
prepared := promptkit.PreparedRun{
PromptID: "prompt",
StartTime: start,
EndTime: start.Add(1250 * time.Millisecond),
DurationMS: 1250,
}
payload, err := json.Marshal(prepared)
if err != nil {
t.Fatalf("marshal prepared run: %v", err)
}
var decoded promptkit.PreparedRun
if err := json.Unmarshal(payload, &decoded); err != nil {
t.Fatalf("unmarshal prepared run: %v", err)
}
if decoded.DurationMS != prepared.DurationMS ||
!decoded.StartTime.Equal(prepared.StartTime) ||
!decoded.EndTime.Equal(prepared.EndTime) {
t.Fatalf("timing values did not round trip: got %#v, want %#v", decoded, prepared)
}
}
func TestRunResultJSONUsesMillisecondsAndRoundTrips(t *testing.T) {
start := time.Date(2026, time.July, 29, 12, 0, 0, 0, time.UTC)
result := promptkit.RunResult{
RunID: "opaque-run-id",
Artifact: promptkit.Artifact{Name: "output", ContentType: "text/plain", Body: []byte("ok")},
StartTime: start,
EndTime: start.Add(1500 * time.Millisecond),
Duration: 1500 * time.Millisecond,
}
payload, err := json.Marshal(result)
if err != nil {
t.Fatalf("marshal run result: %v", err)
}
var object map[string]any
if err := json.Unmarshal(payload, &object); err != nil {
t.Fatalf("decode run result JSON: %v", err)
}
if got := object["duration_ms"]; got != float64(1500) {
t.Fatalf("expected duration_ms=1500, got %#v in %s", got, payload)
}
if _, exists := object["duration"]; exists {
t.Fatalf("unexpected nanosecond duration field in %s", payload)
}
artifact, ok := object["artifact"].(map[string]any)
if !ok || artifact["content_type"] != "text/plain" {
t.Fatalf("expected stable artifact JSON fields, got %#v", object["artifact"])
}
var decoded promptkit.RunResult
if err := json.Unmarshal(payload, &decoded); err != nil {
t.Fatalf("unmarshal run result: %v", err)
}
if decoded.Duration != result.Duration || !decoded.StartTime.Equal(result.StartTime) || !decoded.EndTime.Equal(result.EndTime) {
t.Fatalf("timing values did not round trip: got %#v, want %#v", decoded, result)
}
payload, err = json.Marshal(promptkit.RunResult{})
if err != nil {
t.Fatalf("marshal zero run result: %v", err)
}
for _, field := range []string{"start_time", "end_time", "duration_ms"} {
if strings.Contains(string(payload), `"`+field+`"`) {
t.Fatalf("expected zero %s to be omitted, got %s", field, payload)
}
}
}
func TestEngineValidationIsSinglePass(t *testing.T) {
client := &fakeLLMClient{
response: &promptkit.GenerateResponse{Content: "not-json"},
}
engine := newContractEngineWithOptions(t, frameworkSchemaDir, promptkit.WithLLMClient(client))
result, err := engine.Run(context.Background(), promptkit.RunRequest{
PromptID: frameworkMarkdownSummaryPromptID,
Inputs: map[string]promptkit.ArtifactRef{
"transcript": promptkit.Inline("Rin opens the gate."),
"glossary": promptkit.Inline("gate: A guarded passage."),
},
Validation: &promptkit.OutputContract{
Format: promptkit.FormatJSON,
ValidationMode: promptkit.ValidationJSON,
RepairAttempts: 3,
},
})
if err != nil {
t.Fatalf("run with failed content validation: %v", err)
}
if result.Validation.Status != promptkit.ValidationFailed ||
result.Validation.RepairAttempts != 0 {
t.Fatalf("expected failed single-pass validation, got %#v", result.Validation)
}
if len(client.requests) != 1 {
t.Fatalf("expected one model generation, got %d", len(client.requests))
}
}
func TestRepeatedOptionsUseLastValueInEachCategory(t *testing.T) {
profile := promptkit.Profile{ID: "profile", Endpoint: "http://example.test/v1", Model: "model"}
t.Run("prompt source", func(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("first", "profile", "first"), "."),
promptkit.WithPromptFS(contractPromptFS("second", "profile", "second"), "."),
promptkit.WithProfiles(profile),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "second"})
if err != nil {
t.Fatalf("prepare from last prompt source: %v", err)
}
if prepared.Messages[0].Content != "second" {
t.Fatalf("expected last prompt source, got %#v", prepared.Messages)
}
})
t.Run("profile source", func(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
promptkit.WithProfileFS(contractProfileFS("profile", "first-model"), "."),
promptkit.WithProfileFS(contractProfileFS("profile", "second-model"), "."),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
if err != nil {
t.Fatalf("prepare from last profile source: %v", err)
}
if prepared.EffectiveModelParams.Model != "second-model" {
t.Fatalf("expected last profile source, got %q", prepared.EffectiveModelParams.Model)
}
})
t.Run("in-memory profiles", func(t *testing.T) {
first := profile
first.Model = "first-model"
second := profile
second.Model = "second-model"
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
promptkit.WithProfiles(first),
promptkit.WithProfiles(second),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
if err != nil {
t.Fatalf("prepare from last in-memory profile option: %v", err)
}
if prepared.EffectiveModelParams.Model != "second-model" {
t.Fatalf("expected last in-memory profiles, got %q", prepared.EffectiveModelParams.Model)
}
})
t.Run("schema source", func(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractSchemaPromptFS(), "."),
promptkit.WithProfiles(profile),
promptkit.WithSchemaFS(contractSchemaFS("first"), "."),
promptkit.WithSchemaFS(contractSchemaFS("second"), "."),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "schema-prompt"})
if err != nil {
t.Fatalf("prepare from last schema source: %v", err)
}
schema := prepared.StructuredOutput.JSONSchema.Schema.(map[string]any)
if schema["title"] != "second" {
t.Fatalf("expected last schema source, got %#v", schema)
}
})
t.Run("model client", func(t *testing.T) {
var firstCalls, secondCalls atomic.Int64
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
promptkit.WithProfiles(profile),
promptkit.WithLLMClient(countingLLMClient{calls: &firstCalls}),
promptkit.WithLLMClient(countingLLMClient{calls: &secondCalls}),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
if _, err := engine.Run(context.Background(), promptkit.RunRequest{PromptID: "prompt"}); err != nil {
t.Fatalf("run with last model client: %v", err)
}
if firstCalls.Load() != 0 || secondCalls.Load() != 1 {
t.Fatalf("expected only last client call, got first=%d second=%d", firstCalls.Load(), secondCalls.Load())
}
})
t.Run("artifact reader", func(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractInputPromptFS(), "."),
promptkit.WithProfiles(profile),
promptkit.WithArtifactReader(fixedArtifactReader("first")),
promptkit.WithArtifactReader(fixedArtifactReader("second")),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
PromptID: "input-prompt",
Inputs: map[string]promptkit.ArtifactRef{"input": promptkit.Inline("ignored")},
})
if err != nil {
t.Fatalf("prepare with last artifact reader: %v", err)
}
if prepared.Messages[0].Content != "second" {
t.Fatalf("expected last artifact reader, got %#v", prepared.Messages)
}
})
}
func TestEngineSupportsConcurrentPrepareAndRun(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
promptkit.WithProfiles(promptkit.Profile{
ID: "profile",
Endpoint: "http://example.test/v1",
Model: "model",
}),
promptkit.WithLLMClient(countingLLMClient{}),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
const calls = 40
errs := make(chan error, calls)
var wg sync.WaitGroup
for i := 0; i < calls; i++ {
wg.Add(1)
go func(run bool) {
defer wg.Done()
request := promptkit.RunRequest{PromptID: "prompt"}
if run {
_, err := engine.Run(context.Background(), request)
errs <- err
return
}
_, err := engine.Prepare(context.Background(), request)
errs <- err
}(i%2 == 0)
}
wg.Wait()
close(errs)
for err := range errs {
if err != nil {
t.Fatalf("concurrent call failed: %v", err)
}
}
}
type countingLLMClient struct {
calls *atomic.Int64
}
func (c countingLLMClient) Generate(context.Context, promptkit.GenerateRequest) (*promptkit.GenerateResponse, error) {
if c.calls != nil {
c.calls.Add(1)
}
return &promptkit.GenerateResponse{Content: "ok"}, nil
}
type fixedArtifactReader string
func (r fixedArtifactReader) Read(context.Context, promptkit.ArtifactRef) (*promptkit.Artifact, error) {
return &promptkit.Artifact{Body: []byte(r)}, nil
}
func contractPromptFS(id, profileID, message string) fstest.MapFS {
return fstest.MapFS{
"prompt.yaml": &fstest.MapFile{Data: []byte(fmt.Sprintf(`id: %s
version: "1"
default_profile: %s
messages:
- role: user
content: %q
output:
format: text
validation_mode: none
`, id, profileID, message))},
}
}
func contractInputPromptFS() fstest.MapFS {
return fstest.MapFS{
"prompt.yaml": &fstest.MapFile{Data: []byte(`id: input-prompt
version: "1"
default_profile: profile
inputs:
- name: input
required: true
messages:
- role: user
content: '{{input "input"}}'
output:
format: text
validation_mode: none
`)},
}
}
func contractProfileFS(id, model string) fstest.MapFS {
return fstest.MapFS{
"profile.yaml": &fstest.MapFile{Data: []byte(fmt.Sprintf(`id: %s
endpoint: http://example.test/v1
model: %s
`, id, model))},
}
}
func contractSchemaPromptFS() fstest.MapFS {
return fstest.MapFS{
"prompt.yaml": &fstest.MapFile{Data: []byte(`id: schema-prompt
version: "1"
default_profile: profile
messages:
- role: user
content: message
output:
format: json
validation_mode: json_schema
schema_path: schema.json
`)},
}
}
func contractSchemaFS(title string) fstest.MapFS {
return fstest.MapFS{
"schema.json": &fstest.MapFile{Data: []byte(fmt.Sprintf(`{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": %q,
"type": "object"
}`, title))},
}
}

593
types.go
View File

@@ -5,161 +5,319 @@ import (
"time"
)
// ArtifactRefType defines how an artifact is referenced.
// ArtifactRefType identifies how an [ArtifactRef] supplies content.
type ArtifactRefType string
const (
// ArtifactRefInline selects ArtifactRef.Body as the content.
ArtifactRefInline ArtifactRefType = "inline"
ArtifactRefFile ArtifactRefType = "file"
// ArtifactRefFile selects the filesystem path in ArtifactRef.URI.
ArtifactRefFile ArtifactRefType = "file"
)
// OutputFormat defines the desired output format.
// OutputFormat identifies the media format of generated output.
// OutputFormat has a stable JSON string representation.
type OutputFormat string
const (
FormatText OutputFormat = "text"
// FormatText identifies plain-text output.
FormatText OutputFormat = "text"
// FormatMarkdown identifies Markdown output.
FormatMarkdown OutputFormat = "markdown"
FormatJSON OutputFormat = "json"
// FormatJSON identifies JSON output.
FormatJSON OutputFormat = "json"
)
// ValidationMode defines the output validation strategy.
// ValidationMode identifies how generated output is checked.
// ValidationMode has a stable JSON string representation.
type ValidationMode string
const (
ValidationNone ValidationMode = "none"
ValidationBasic ValidationMode = "basic"
ValidationJSON ValidationMode = "json"
// ValidationNone skips content validation.
ValidationNone ValidationMode = "none"
// ValidationBasic requires non-empty output.
ValidationBasic ValidationMode = "basic"
// ValidationJSON requires syntactically valid JSON.
ValidationJSON ValidationMode = "json"
// ValidationJSONSchema requires JSON that satisfies OutputContract.SchemaPath.
ValidationJSONSchema ValidationMode = "json_schema"
)
// ValidationStatus defines the result of a validation check.
// ValidationStatus identifies the completed state of an output check.
// ValidationStatus has a stable JSON string representation.
type ValidationStatus string
const (
ValidationPassed ValidationStatus = "passed"
ValidationFailed ValidationStatus = "failed"
// ValidationPassed means the generated output satisfied its contract.
ValidationPassed ValidationStatus = "passed"
// ValidationFailed means validation completed and rejected the generated
// output. Engine.Run returns this status in a result, not as an error.
ValidationFailed ValidationStatus = "failed"
// ValidationSkipped means ValidationNone selected no content check.
ValidationSkipped ValidationStatus = "skipped"
)
// CacheControlType defines provider cache behavior for prompt content.
// CacheControlType identifies provider cache behavior for prompt content.
// CacheControlType has a stable JSON string representation.
type CacheControlType string
const (
// CacheControlEphemeral requests provider-defined ephemeral caching.
CacheControlEphemeral CacheControlType = "ephemeral"
)
// StructuredOutputType identifies provider-level structured output modes.
// StructuredOutputType has a stable JSON string representation.
type StructuredOutputType string
const (
// StructuredOutputJSONSchema supplies JSON Schema response constraints.
StructuredOutputJSONSchema StructuredOutputType = "json_schema"
)
// RunRequest represents a request to prepare or run a single prompt.
// RunRequest selects one prompt execution. It has no stable JSON
// representation.
//
// Prepare and Run copy the request's maps, pointers, and nested
// JSON-compatible values before using them. The caller may mutate the request
// after either method returns.
type RunRequest struct {
PromptID string
// PromptID is the required non-empty prompt identifier.
PromptID string
// PromptVersion optionally selects one version of PromptID. When empty, the
// prompt source must contain exactly one matching version.
PromptVersion string
ProfileID string
APIKey string `json:"-"`
Inputs map[string]ArtifactRef
Vars map[string]string
Execution *ExecutionTargetOverride
Validation *OutputContract
Metadata map[string]string
// ProfileID selects an execution profile. When empty, the prompt's default
// profile is used; if both are empty, the error matches ErrProfileRequired
// and ErrInvalidRequest.
ProfileID string
// APIKey is a request-scoped direct credential. It takes precedence over
// APIKeyEnv, is passed to the selected LLMClient, and is never included in
// prepared values, results, hashes, JSON, String, or GoString output.
APIKey string `json:"-"`
// Inputs maps prompt input names to references. A nil or empty map is valid
// only when the selected prompt and its templates require no inputs.
Inputs map[string]ArtifactRef
// Vars supplies Go-template data for messages and the session ID. Nil and
// empty maps are equivalent.
Vars map[string]string
// Execution optionally overrides individual profile execution settings.
// Nil uses the selected profile over framework defaults.
Execution *ExecutionTargetOverride
// Validation optionally replaces the prompt's complete output contract. It
// does not merge individual fields. Nil uses the prompt contract.
Validation *OutputContract
}
// PreparedRun contains prepared prompt execution state. It does not include
// resolved API key values, model output, validation results, or internal target
// presence metadata.
// presence metadata. PreparedRun has a stable JSON representation.
//
// All maps, slices, pointers, and schema values are caller-owned copies. JSON
// timestamps use RFC 3339 and zero timing values are omitted. Hash formats are
// opaque.
type PreparedRun struct {
PromptID string `json:"prompt_id"`
PromptVersion string `json:"prompt_version,omitempty"`
PromptHash string `json:"prompt_hash,omitempty"`
SelectedProfileID string `json:"selected_profile_id"`
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
OutputContract OutputContract `json:"output_contract"`
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
InputHashes map[string]string `json:"input_hashes,omitempty"`
SessionID string `json:"session_id,omitempty"`
RenderedPromptHash string `json:"rendered_prompt_hash"`
Messages []RenderedMessage `json:"messages"`
StartTime time.Time `json:"start_time,omitempty"`
EndTime time.Time `json:"end_time,omitempty"`
DurationMS int64 `json:"duration_ms,omitempty"`
// PromptID is the selected prompt identifier.
PromptID string `json:"prompt_id"`
// PromptVersion is the selected prompt version.
PromptVersion string `json:"prompt_version,omitempty"`
// PromptHash is an opaque equality value for the selected definition.
PromptHash string `json:"prompt_hash,omitempty"`
// SelectedProfileID is the explicit request profile or prompt default that
// supplied execution settings.
SelectedProfileID string `json:"selected_profile_id"`
// EffectiveModelParams contains framework defaults overlaid by the selected
// profile and then request overrides. It excludes resolved API-key values.
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
// OutputContract is the complete effective output contract.
OutputContract OutputContract `json:"output_contract"`
// StructuredOutput is non-nil for JSON Schema validation and contains the
// provider-facing response constraint passed to an LLM client.
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
// InputHashes maps every supplied input name to its opaque artifact hash.
InputHashes map[string]string `json:"input_hashes,omitempty"`
// SessionID is the trimmed rendered session identifier, if any.
SessionID string `json:"session_id,omitempty"`
// RenderedPromptHash is an opaque equality value for SessionID and Messages.
RenderedPromptHash string `json:"rendered_prompt_hash"`
// Messages are the rendered messages that Run passes to the LLM client.
Messages []RenderedMessage `json:"messages"`
// StartTime is the UTC time at which preparation began.
StartTime time.Time `json:"start_time,omitempty"`
// EndTime is the UTC time at which preparation completed.
EndTime time.Time `json:"end_time,omitempty"`
// DurationMS is preparation elapsed time in integer milliseconds. JSON uses
// duration_ms and omits a zero value.
DurationMS int64 `json:"duration_ms,omitempty"`
}
// RunResult contains generated output, validation state, and run metadata.
// RunResult has a stable JSON representation and round-trips its Duration
// through the duration_ms JSON field.
//
// All maps, slices, and nested values are caller-owned copies. JSON timestamps
// use RFC 3339 and zero timing values are omitted. Run IDs and hash formats are
// opaque.
type RunResult struct {
RunID string `json:"run_id"`
Artifact Artifact `json:"artifact"`
RawOutput string `json:"raw_output"`
Validation ValidationResult `json:"validation"`
PromptID string `json:"prompt_id"`
PromptVersion string `json:"prompt_version,omitempty"`
PromptHash string `json:"prompt_hash,omitempty"`
RenderedPromptHash string `json:"rendered_prompt_hash"`
SelectedProfileID string `json:"selected_profile_id"`
ModelName string `json:"model_name"`
Endpoint string `json:"endpoint"`
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
InputHashes map[string]string `json:"input_hashes,omitempty"`
Usage TokenUsage `json:"usage"`
StartTime time.Time `json:"start_time,omitempty"`
EndTime time.Time `json:"end_time,omitempty"`
Duration time.Duration `json:"duration,omitempty"`
// RunID is an opaque identifier for this invocation.
RunID string `json:"run_id"`
// Artifact contains the generated output and derived metadata.
Artifact Artifact `json:"artifact"`
// RawOutput is the exact generated content before artifact classification
// and validation.
RawOutput string `json:"raw_output"`
// Validation records the completed content check.
Validation ValidationResult `json:"validation"`
// PromptID is the selected prompt identifier.
PromptID string `json:"prompt_id"`
// PromptVersion is the selected prompt version.
PromptVersion string `json:"prompt_version,omitempty"`
// PromptHash is the same opaque definition equality value exposed by
// PreparedRun.
PromptHash string `json:"prompt_hash,omitempty"`
// RenderedPromptHash is the same opaque rendered-prompt equality value
// computed during preparation.
RenderedPromptHash string `json:"rendered_prompt_hash"`
// SelectedProfileID identifies the profile used for execution.
SelectedProfileID string `json:"selected_profile_id"`
// ModelName is the effective model name and equals
// EffectiveModelParams.Model.
ModelName string `json:"model_name"`
// Endpoint is the effective base endpoint and equals
// EffectiveModelParams.Endpoint.
Endpoint string `json:"endpoint"`
// EffectiveModelParams contains the settings supplied to the LLM client,
// excluding resolved API-key values.
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
// InputHashes are the opaque input equality values computed during
// preparation.
InputHashes map[string]string `json:"input_hashes,omitempty"`
// Usage is the token accounting reported by the LLM client.
Usage TokenUsage `json:"usage"`
// StartTime is the UTC time immediately before preparation begins.
StartTime time.Time `json:"start_time,omitempty"`
// EndTime is the UTC time after generation and validation complete.
EndTime time.Time `json:"end_time,omitempty"`
// Duration covers preparation, generation, and validation. JSON represents
// it as integer milliseconds in duration_ms and omits a zero value.
Duration time.Duration `json:"-"`
}
// ArtifactRef represents a reference to prompt input content.
// ArtifactRef identifies prompt input content. It has no stable JSON
// representation. Prefer [File], [Inline], or [InlineWithURI] to construct one.
type ArtifactRef struct {
// Type must be ArtifactRefInline or ArtifactRefFile.
Type ArtifactRefType
URI string
// URI is the file path for ArtifactRefFile and optional provenance metadata
// for ArtifactRefInline.
URI string
// Body is the content for ArtifactRefInline and is ignored for
// ArtifactRefFile.
Body string
}
// Artifact represents loaded artifact content.
// Artifact represents loaded or generated content and has a stable JSON
// representation. Body uses encoding/json's base64 representation for []byte.
type Artifact struct {
Name string
ContentType string
Body []byte
URI string
Size int64
Hash string
// Name is artifact metadata. During input preparation the engine fills an
// empty reader-supplied name with the request input-map key.
Name string `json:"name"`
// ContentType is the media type reported by the reader or derived for
// generated output.
ContentType string `json:"content_type"`
// Body is the artifact content. Engine boundaries copy this slice.
Body []byte `json:"body"`
// URI is optional source or result provenance metadata.
URI string `json:"uri"`
// Size is content-size metadata in bytes.
Size int64 `json:"size"`
// Hash is an opaque content equality value when the producing reader
// supplies one. Its format and algorithm are not API contracts.
Hash string `json:"hash"`
}
// ArtifactReader resolves a prompt input reference into its content.
//
// Readers are responsible for supplying artifact metadata. The engine assigns
// an input-map name only when the returned artifact name is empty.
// Read may be called concurrently. It must honor ctx cancellation to make
// Prepare and Run responsive to cancellation. The engine passes a copied ref
// and immediately copies the returned Artifact.Body; it does not retain either
// value. Readers supply artifact metadata, and the engine assigns an input-map
// name only when the returned artifact name is empty.
//
// An injected reader owns any application-specific path containment,
// authorization, content-size, and content-type policy. It must protect
// sensitive references and bodies in its logging and in any copies it retains.
// It may reuse or mutate the returned artifact and body after Read returns.
//
// Returning a non-nil error makes the engine return an error matching
// ErrArtifactLoad while preserving the reader error through errors.Is.
// Returning a nil artifact with a nil error also produces ErrArtifactLoad.
type ArtifactReader interface {
Read(context.Context, ArtifactRef) (*Artifact, error)
}
// ExecutionTarget represents effective model runtime settings.
// ExecutionTarget represents effective model runtime settings and has a stable
// JSON representation. It never exposes a resolved API-key value.
type ExecutionTarget struct {
Endpoint string `json:"endpoint"`
Model string `json:"model"`
Temperature float64 `json:"temperature"`
MaxTokens int `json:"max_tokens"`
TopP float64 `json:"top_p"`
TimeoutSeconds int `json:"timeout_seconds"`
ServiceTier string `json:"service_tier"`
ReasoningEffort string `json:"reasoning_effort"`
APIKeyEnv string `json:"api_key_env"`
ExtraParams map[string]any `json:"extra_params"`
// Endpoint is the model-provider base URL.
Endpoint string `json:"endpoint"`
// Model is the provider model identifier.
Model string `json:"model"`
// Temperature is the effective sampling temperature from 0 through 2.
Temperature float64 `json:"temperature"`
// MaxTokens is the non-negative effective output-token limit. Zero leaves
// the limit unspecified to compatible providers unless it was an explicit
// request override.
MaxTokens int `json:"max_tokens"`
// TopP is the effective nucleus-sampling value from 0 through 1.
TopP float64 `json:"top_p"`
// TimeoutSeconds is the non-negative per-generation deadline. Zero disables
// this deadline without disabling caller cancellation or the transport cap.
TimeoutSeconds int `json:"timeout_seconds"`
// ServiceTier is an optional provider-specific request tier.
ServiceTier string `json:"service_tier"`
// ReasoningEffort is an optional provider-specific reasoning setting.
ReasoningEffort string `json:"reasoning_effort"`
// APIKeyEnv is an environment-variable name, not its credential value.
APIKeyEnv string `json:"api_key_env"`
// ExtraParams contains copied JSON-compatible provider parameters.
ExtraParams map[string]any `json:"extra_params"`
}
// ExecutionTargetOverride represents per-request runtime setting overrides.
// ExecutionTargetOverride represents per-request runtime setting overrides and
// has no stable JSON representation.
//
// Non-empty string fields replace profile values. Non-nil numeric pointers
// replace profile values and preserve explicit zero. A non-empty ExtraParams
// map replaces the complete profile map rather than merging keys. Empty string
// fields, nil pointers, and a nil or empty ExtraParams map inherit the selected
// profile over framework defaults.
type ExecutionTargetOverride struct {
Endpoint string
Model string
Temperature *float64
MaxTokens *int
TopP *float64
TimeoutSeconds *int
ServiceTier string
// Endpoint replaces the profile endpoint when non-empty.
Endpoint string
// Model replaces the profile model when non-empty.
Model string
// Temperature, when non-nil, must point to a value from 0 through 2.
Temperature *float64
// MaxTokens, when non-nil, must point to a non-negative value.
MaxTokens *int
// TopP, when non-nil, must point to a value from 0 through 1.
TopP *float64
// TimeoutSeconds, when non-nil, must point to a non-negative value. A
// pointed-to zero disables the per-generation deadline.
TimeoutSeconds *int
// ServiceTier replaces the profile value when non-blank.
ServiceTier string
// ReasoningEffort replaces the profile value when non-blank. An empty value
// cannot clear a profile setting.
ReasoningEffort string
APIKeyEnv string
ExtraParams map[string]any
// APIKeyEnv replaces the profile environment-variable name when non-blank.
// A direct RunRequest.APIKey still takes precedence over environment lookup.
APIKeyEnv string
// ExtraParams, when non-empty, replaces the profile map. Values must be
// JSON-compatible: nil, booleans, finite numbers, strings, arrays or slices,
// and maps with non-empty string keys. Cycles are invalid.
ExtraParams map[string]any
}
// Profile is an in-memory execution profile for library consumers.
@@ -167,19 +325,39 @@ type ExecutionTargetOverride struct {
// It is equivalent to a loaded profile file after validation. Raw API keys do
// not belong in profiles; use APIKeyRequired to require callers to provide
// RunRequest.APIKey for each request, or use profile YAML api_key_env with file
// and FS profile sources.
// and FS profile sources. Profile has no stable JSON representation.
//
// WithProfiles validates and copies Profile values during NewEngine. Numeric
// zero, blank strings, and an empty ExtraParams map inherit framework defaults;
// use ExecutionTargetOverride pointer fields to request explicit numeric zero.
type Profile struct {
ID string
Endpoint string
Model string
Temperature float64
MaxTokens int
TopP float64
TimeoutSeconds int
ServiceTier string
// ID is the required non-blank profile identifier. WithProfiles trims it.
ID string
// Endpoint is the required non-blank model-provider base URL.
Endpoint string
// Model is the required non-blank provider model identifier.
Model string
// Temperature is from 0 through 2. Zero inherits the framework default.
Temperature float64
// MaxTokens is non-negative. Zero inherits the framework default.
MaxTokens int
// TopP is from 0 through 1. Zero inherits the framework default rather than
// selecting an explicit zero.
TopP float64
// TimeoutSeconds is non-negative. Zero inherits the framework default.
TimeoutSeconds int
// ServiceTier is optional; a blank value inherits the framework default.
ServiceTier string
// ReasoningEffort is optional; a blank value inherits the framework
// default.
ReasoningEffort string
APIKeyRequired bool
ExtraParams map[string]any
// APIKeyRequired requires a non-blank RunRequest.APIKey. It does not store a
// credential or enable environment lookup.
APIKeyRequired bool
// ExtraParams contains provider-specific JSON-compatible values. An empty
// map inherits framework defaults. WithProfiles validates and deeply copies
// it during NewEngine.
ExtraParams map[string]any
}
// OpenAICompatibleProfileConfig configures an OpenAI-compatible in-memory
@@ -187,120 +365,225 @@ type Profile struct {
//
// It contains ordinary profile fields for OpenAI-compatible chat-completions
// endpoints. APIKeyRequired is satisfied by RunRequest.APIKey. Raw API keys do
// not belong in this config.
// not belong in this config. OpenAICompatibleProfileConfig has no stable JSON
// representation and is not validated until its resulting Profile is supplied
// through WithProfiles to NewEngine.
type OpenAICompatibleProfileConfig struct {
ID string
Endpoint string
Model string
APIKeyRequired bool
Temperature float64
MaxTokens int
TopP float64
TimeoutSeconds int
ServiceTier string
// ID becomes Profile.ID.
ID string
// Endpoint becomes Profile.Endpoint.
Endpoint string
// Model becomes Profile.Model.
Model string
// APIKeyRequired becomes Profile.APIKeyRequired.
APIKeyRequired bool
// Temperature becomes Profile.Temperature.
Temperature float64
// MaxTokens becomes Profile.MaxTokens.
MaxTokens int
// TopP becomes Profile.TopP.
TopP float64
// TimeoutSeconds becomes Profile.TimeoutSeconds.
TimeoutSeconds int
// ServiceTier becomes Profile.ServiceTier.
ServiceTier string
// ReasoningEffort becomes Profile.ReasoningEffort.
ReasoningEffort string
ExtraParams map[string]any
// ExtraParams becomes a shallow-copied Profile.ExtraParams map. NewEngine
// performs validation and a deep copy when WithProfiles applies the result.
ExtraParams map[string]any
}
// ExecutionTargetPresence tracks which numeric runtime settings were explicit
// request overrides.
// request overrides, including explicit zero values. It has a stable JSON
// representation and is supplied to injected LLM clients so they can preserve
// omission semantics.
type ExecutionTargetPresence struct {
Temperature bool
MaxTokens bool
TopP bool
TimeoutSeconds bool
// Temperature reports a non-nil ExecutionTargetOverride.Temperature.
Temperature bool `json:"temperature"`
// MaxTokens reports a non-nil ExecutionTargetOverride.MaxTokens.
MaxTokens bool `json:"max_tokens"`
// TopP reports a non-nil ExecutionTargetOverride.TopP.
TopP bool `json:"top_p"`
// TimeoutSeconds reports a non-nil ExecutionTargetOverride.TimeoutSeconds.
TimeoutSeconds bool `json:"timeout_seconds"`
}
// OutputContract defines output and validation requirements.
// OutputContract defines output and validation requirements and has a stable
// JSON representation.
//
// A non-nil RunRequest.Validation replaces the complete prompt contract. It
// does not merge fields. The public Engine validates generated output once and
// does not install an output repairer.
type OutputContract struct {
Format OutputFormat `json:"format"`
// Format selects generated artifact metadata. An empty effective value
// defaults to FormatText.
Format OutputFormat `json:"format"`
// ValidationMode selects the content check. Use one of the declared
// ValidationMode constants.
ValidationMode ValidationMode `json:"validation_mode"`
SchemaPath string `json:"schema_path"`
RepairAttempts int `json:"repair_attempts"`
// SchemaPath is required when ValidationMode is ValidationJSONSchema and is
// ignored by other modes.
SchemaPath string `json:"schema_path"`
// RepairAttempts is a requested repair limit. A non-positive value requests
// no repairs. The public Engine performs no repairs even when this value is
// positive, so its runs report zero attempts used.
RepairAttempts int `json:"repair_attempts"`
}
// ValidationResult represents output validation state.
// ValidationResult represents a completed output check and has a stable JSON
// representation. An operational inability to perform validation is returned
// as ErrValidation instead of a ValidationResult.
type ValidationResult struct {
Status ValidationStatus `json:"status"`
Mode ValidationMode `json:"mode"`
Errors []string `json:"errors,omitempty"`
SchemaPath string `json:"schema_path,omitempty"`
RepairAttempts int `json:"repair_attempts"`
IsValid bool `json:"is_valid"`
// Status is Passed, Failed, or Skipped.
Status ValidationStatus `json:"status"`
// Mode is the effective validation mode.
Mode ValidationMode `json:"mode"`
// Errors contains validation diagnostics when Status is ValidationFailed.
Errors []string `json:"errors,omitempty"`
// SchemaPath is the effective schema path for JSON Schema validation.
SchemaPath string `json:"schema_path,omitempty"`
// RepairAttempts is the number of repairs actually attempted. It is always
// zero for the public Engine.
RepairAttempts int `json:"repair_attempts"`
// IsValid is true for ValidationPassed and ValidationSkipped and false for
// ValidationFailed.
IsValid bool `json:"is_valid"`
}
// TokenUsage tracks token consumption.
// TokenUsage contains model-client token accounting and has a stable JSON
// representation. Promptkit preserves values reported by the client and does
// not derive or reconcile them.
type TokenUsage struct {
PromptTokens int `json:"prompt_tokens"`
// PromptTokens is the reported input-token count.
PromptTokens int `json:"prompt_tokens"`
// CompletionTokens is the reported generated-token count.
CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"`
CachedTokens int `json:"cached_tokens"`
// TotalTokens is the reported total-token count.
TotalTokens int `json:"total_tokens"`
// CachedTokens is the reported cached-input-token count.
CachedTokens int `json:"cached_tokens"`
// CacheWriteTokens is the reported cache-write-token count.
CacheWriteTokens int `json:"cache_write_tokens"`
}
// RenderedPrompt is the fully rendered prompt passed to an LLM client.
// RenderedPrompt is the fully rendered prompt passed to an LLM client and has
// a stable JSON representation.
type RenderedPrompt struct {
SessionID string `json:"session_id,omitempty"`
Messages []RenderedMessage `json:"messages"`
// SessionID is the optional trimmed session identifier rendered from the
// prompt definition.
SessionID string `json:"session_id,omitempty"`
// Messages contains rendered messages in definition order.
Messages []RenderedMessage `json:"messages"`
}
// RenderedMessage is a rendered chat message.
// RenderedMessage is a rendered chat message and has a stable JSON
// representation.
type RenderedMessage struct {
Role string `json:"role"`
Content string `json:"content"`
// Role is the definition-supplied chat role.
Role string `json:"role"`
// Content is the rendered message text.
Content string `json:"content"`
// CacheControl is optional provider cache metadata.
CacheControl *CacheControl `json:"cache_control,omitempty"`
}
// CacheControl describes provider cache metadata attached to prompt content.
// CacheControl describes provider cache metadata attached to prompt content
// and has a stable JSON representation.
type CacheControl struct {
// Type identifies the cache behavior.
Type CacheControlType `json:"type"`
TTL string `json:"ttl,omitempty"`
// TTL is an optional provider cache lifetime.
TTL string `json:"ttl,omitempty"`
}
// StructuredOutputSpec describes provider-level structured output.
// StructuredOutputSpec describes provider-level structured output and has a
// stable JSON representation.
type StructuredOutputSpec struct {
Type StructuredOutputType `json:"type"`
// Type identifies the structured-output mechanism.
Type StructuredOutputType `json:"type"`
// JSONSchema contains constraints when Type is StructuredOutputJSONSchema.
JSONSchema *StructuredOutputJSONSpec `json:"json_schema,omitempty"`
}
// StructuredOutputJSONSpec contains JSON Schema output constraints.
// StructuredOutputJSONSpec contains provider-facing JSON Schema output
// constraints and has a stable JSON representation.
type StructuredOutputJSONSpec struct {
Name string `json:"name"`
Strict bool `json:"strict"`
Schema any `json:"schema"`
// Name is the provider-facing schema name.
Name string `json:"name"`
// Strict requests strict provider enforcement of Schema.
Strict bool `json:"strict"`
// Schema is a caller-owned copy of the loaded JSON Schema document.
Schema any `json:"schema"`
}
// LLMClient executes rendered prompts for Engine.Run.
// LLMClient executes rendered prompts for [Engine.Run].
//
// Generate may be called concurrently. It must honor context cancellation to
// make Run responsive to cancellation. The request and all nested maps,
// slices, and pointers are client-owned copies and may be mutated or retained
// without affecting engine state.
//
// Generate receives rendered messages and may receive a direct API key. A
// client must protect those values and any raw output in its logging, storage,
// and retained copies. It is responsible for the cancellation behavior of any
// work it starts and for synchronizing access to retained or shared data.
//
// A returned error makes Run return ErrLLMGenerate while preserving the client
// error through errors.Is. A nil response with a nil error also produces
// ErrLLMGenerate. Promptkit copies the non-nil response before returning from
// Run.
type LLMClient interface {
Generate(context.Context, GenerateRequest) (*GenerateResponse, error)
}
// GenerateRequest is passed to an injected LLM client.
// GenerateRequest is passed to an injected LLM client and has a stable JSON
// representation. Its String and GoString methods omit rendered content and
// direct credentials.
type GenerateRequest struct {
Prompt RenderedPrompt `json:"prompt"`
Target ExecutionTarget `json:"target"`
TargetPresence ExecutionTargetPresence `json:"target_presence"`
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
APIKey string `json:"-"`
// Prompt contains the rendered session ID and messages.
Prompt RenderedPrompt `json:"prompt"`
// Target contains effective model settings without the direct API key.
Target ExecutionTarget `json:"target"`
// TargetPresence distinguishes inherited numeric zeros from explicit
// request overrides.
TargetPresence ExecutionTargetPresence `json:"target_presence"`
// StructuredOutput contains provider response constraints when requested.
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
// APIKey is the direct request-scoped credential, if any. It is excluded
// from JSON, String, and GoString output.
APIKey string `json:"-"`
}
// GenerateResponse is returned by an injected LLM client.
// GenerateResponse is returned by an injected LLM client and has a stable JSON
// representation.
type GenerateResponse struct {
Content string `json:"content"`
Usage TokenUsage `json:"usage"`
// Content is the generated output. It must be non-empty when using the
// built-in client; injected clients may return empty content for Promptkit
// validation to classify.
Content string `json:"content"`
// Usage is the client's token accounting.
Usage TokenUsage `json:"usage"`
}
// File returns a file-backed artifact reference.
// File returns a file-backed artifact reference whose URI is path.
//
// The default artifact reader opens path as a caller-selected operating-system
// path without restricting it to an application root or imposing a size limit.
// Applications accepting untrusted paths must validate them before calling
// Promptkit or use [WithArtifactReader] to enforce application policy.
func File(path string) ArtifactRef {
return ArtifactRef{Type: ArtifactRefFile, URI: path}
}
// Inline returns an inline artifact reference.
// Inline returns an inline artifact reference whose Body is body and whose URI
// is empty.
func Inline(body string) ArtifactRef {
return ArtifactRef{Type: ArtifactRefInline, Body: body}
}
// InlineWithURI returns an inline artifact reference with URI metadata.
// InlineWithURI returns an inline artifact reference with body content and uri
// provenance metadata.
func InlineWithURI(uri string, body string) ArtifactRef {
return ArtifactRef{Type: ArtifactRefInline, URI: uri, Body: body}
}