4 Commits

21 changed files with 2262 additions and 382 deletions

View File

@@ -31,4 +31,9 @@ Contributors should start with the [development guide](docs/development.md).
The [architecture policy](docs/policy/architecture.md) defines the library The [architecture policy](docs/policy/architecture.md) defines the library
boundary and constraints that framework work must preserve. boundary and constraints that framework work must preserve.
## Related Project
[Scriptorium](https://gitea.maximumdirect.net/eric/scriptorium) is the CLI and
HTTP application built on Promptkit.
Promptkit is licensed under the [GNU General Public License version 3](LICENSE). Promptkit is licensed under the [GNU General Public License version 3](LICENSE).

View File

@@ -20,7 +20,6 @@ func toDomainRunRequest(req RunRequest) (domain.RunRequest, error) {
Vars: copyStringMap(req.Vars), Vars: copyStringMap(req.Vars),
Execution: execution, Execution: execution,
Validation: toDomainOutputContractPtr(req.Validation), Validation: toDomainOutputContractPtr(req.Validation),
Metadata: copyStringMap(req.Metadata),
}, nil }, nil
} }

54
doc.go
View File

@@ -1,8 +1,54 @@
// Package promptkit provides an embeddable engine for preparing and executing // Package promptkit provides an embeddable engine for preparing and executing
// prompt-defined LLM workflows. // prompt-defined LLM workflows.
// //
// Applications construct an Engine with NewEngine, select filesystem or // Applications construct an [Engine] with [NewEngine], select filesystem or
// in-memory definition sources with options, and use Prepare or Run to execute // in-memory sources with options, and call [Engine.Prepare] or [Engine.Run].
// requests. Concrete repositories, validators, and outbound clients remain // Concrete repositories, validators, and the built-in OpenAI-compatible client
// internal implementation details. // remain internal implementation details.
//
// # Concurrency and ownership
//
// An Engine supports concurrent Prepare and Run calls. An injected [LLMClient]
// or [ArtifactReader] can therefore receive concurrent calls and must be safe
// for that use.
//
// NewEngine copies in-memory profiles. Prepare and Run copy request maps,
// slices, pointer values, and JSON-compatible extra parameters before using
// them. Returned values and values passed to extension interfaces are likewise
// isolated from engine state. Callers own those copies and may mutate them
// after the call that supplied or returned them.
//
// # Security and sensitive data
//
// The default artifact reader treats [File] paths as caller-selected operating
// system paths. It does not restrict them to an application root or impose an
// inbound request-size policy. Promptkit is not an inbound request or
// untrusted-input security boundary. Applications must validate and restrict
// untrusted input before constructing a request, or install an [ArtifactReader]
// that enforces their filesystem, authorization, and size policies.
//
// Rendered messages, input and output [Artifact] bodies, [RunResult.RawOutput],
// and [ValidationResult.Errors] may contain sensitive data. Credential
// exclusion and redaction do not sanitize those values. Applications and
// injected collaborators are responsible for access control, retention,
// logging, and secret handling appropriate to their data.
//
// # JSON
//
// Stable JSON representations are provided for [PreparedRun], [RunResult],
// [Artifact], [ExecutionTarget], [OutputContract], [ValidationResult],
// [TokenUsage], [RenderedPrompt], [RenderedMessage], [CacheControl],
// [StructuredOutputSpec], [StructuredOutputJSONSpec], [GenerateRequest],
// [GenerateResponse], [ExecutionTargetPresence], and the string value types
// used by those values.
//
// Construction values, including [Config], [RunRequest], [ArtifactRef],
// [ExecutionTargetOverride], [Profile], and
// [OpenAICompatibleProfileConfig], do not have stable JSON representations.
// Direct API keys are nevertheless excluded from JSON for every public value.
//
// JSON timestamps use time.Time's RFC 3339 encoding and are omitted when zero.
// PreparedRun and RunResult durations are encoded as integer milliseconds in
// duration_ms and omitted when zero. Run IDs and all exposed hashes are opaque:
// their spelling, length, character set, and algorithm are not API contracts.
package promptkit package promptkit

View File

@@ -1,151 +1,147 @@
# Package `promptkit` # Package `promptkit`
Import path: ## Purpose
This guide helps Go consumers assemble Promptkit and choose the main
preparation or execution workflow. The declarations and GoDoc in the
[root package](../../doc.go) own exact field, option, serialization,
concurrency, ownership, failure, and cancellation semantics. The
[framework format reference](../formats.md) owns prompt, profile, and schema
file contracts.
Import the package as:
```go ```go
import "gitea.maximumdirect.net/eric/promptkit" import "gitea.maximumdirect.net/eric/promptkit"
``` ```
Package `promptkit` is the supported Go contract for in-process prompt The following Go fragments are illustrative and omit surrounding package,
preparation and execution. The declarations and their GoDoc in the import, and error-handling code. Use the maintained examples for complete
[root package](../../doc.go) own the exact API; this guide explains how the programs.
pieces are used together. The [framework format reference](../formats.md) owns
prompt, profile, and schema file contracts.
## Engine Construction And Sources ## Construct An Engine
Construct an engine with [`NewEngine`, `Config`, and Create an engine with
`Option`](../../engine.go). `PromptDir` is required unless a prompt source [`NewEngine`](../../engine.go). A directory-backed setup supplies a prompt
option is supplied. `ProfileDir` optionally overlays built-in profiles, and an directory and may supply profile and schema directories:
empty `SchemaDir` uses the current directory. `Timeout` is the transport-wide
safety cap for the built-in OpenAI-compatible client. An optional `HTTPClient`
is cloned; its positive timeout takes precedence.
Nil options are ignored. Invalid construction, including a nil injected client ```go
or artifact reader, returns an error matching `ErrInvalidConfig`. engine, err := promptkit.NewEngine(promptkit.Config{
PromptDir: "prompts",
ProfileDir: "profiles",
SchemaDir: "schemas",
})
```
The [source options](../../engine.go) replace their matching directory source: Options support single-file or `fs.FS` sources, in-memory profiles, and
injected artifact or model clients. Consult the
- `WithPromptFS` and `WithPromptFile` select prompt definitions; [constructor and option GoDoc](../../engine.go) for composition, precedence,
- `WithProfileFS` and `WithProfileFile` overlay built-in profiles; validation, and default transport behavior. Source discovery, format
- `WithProfiles` adds in-memory profiles ahead of file and built-in profiles; validation, and profile precedence are defined by the
- `WithSchemaFS` and `WithSchemaFile` select JSON Schema documents;
- `WithLLMClient` replaces the built-in model client; and
- `WithArtifactReader` replaces the default reader for every input.
Source selection, path resolution, strict decoding, profile overlays, and
file-to-request precedence are defined in the
[framework format reference](../formats.md). [framework format reference](../formats.md).
Per-generation timeout values from profiles or requests are independent of ## Prepare Without Model Execution
the transport cap and caller context. An explicit request value of zero
disables only the per-generation deadline. The
[outbound integration contract](../integrations/openai-compatible-chat.md#timeout-and-cancellation)
defines the complete timeout layering.
## Preparation And Execution [`Engine.Prepare`](../../engine.go) resolves the selected prompt and profile,
loads inputs and any structured-output schema, and renders messages without
calling a model client:
[`Engine.Prepare` and `Engine.Run`](../../engine.go) accept the public ```go
[`RunRequest`](../../types.go). `Prepare` resolves the prompt, profile, input prepared, err := engine.Prepare(ctx, promptkit.RunRequest{
artifacts, validation contract, and rendered messages without calling an LLM. PromptID: "meeting.summary",
`Run` performs the same preparation, calls the configured client, and validates Inputs: map[string]promptkit.ArtifactRef{
the generated content. The maintained "note": promptkit.Inline("Synthetic meeting notes"),
},
})
```
The maintained
[offline preparation example](../../examples/go-library/prepare/main.go) [offline preparation example](../../examples/go-library/prepare/main.go)
provides a complete runnable workflow using a prompt file, in-memory profile, shows a complete runnable setup with a prompt file, in-memory profile, and
and inline input. inline input. Exact request requirements and prepared-result fields belong to
the [`RunRequest` and `PreparedRun` GoDoc](../../types.go).
[`PreparedRun` and `RunResult`](../../types.go) expose copied public values. ## Execute And Validate
Preparation returns effective settings, hashes, rendered messages, selected
profile, structured-output information, and timing without resolved secrets or
model output. Execution adds the generated artifact and raw output, validation
state, model metadata, usage, run ID, and duration.
A generated-content validation failure returns a result with [`Engine.Run`](../../engine.go) performs the same preparation, invokes the
`Validation.Status == ValidationFailed`. An inability to perform validation configured model client, classifies the generated artifact, and validates the
returns an error matching `ErrValidation`. content. A completed content check may return `ValidationFailed` in the result;
an operational inability to validate returns an error.
## Requests, Inputs, And Overrides The maintained
[offline execution example](../../examples/go-library/run/main.go) injects a
deterministic model client and exercises `Run` without credentials, network
access, or paid calls. It is intentionally separate from the preparation
example so each workflow and its small prompt fixture can be copied and run on
its own.
The [request and value declarations](../../types.go) own the available fields, Use the [`RunResult` and `ValidationResult` GoDoc](../../types.go) for the
serialized constants, and result shapes. Use `File`, `Inline`, or returned data and the `Engine.Run` GoDoc for failure and cancellation
`InlineWithURI` to construct artifact references. The semantics. The
[framework format reference](../formats.md) defines declared inputs, template [OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md)
references, output contracts, and the relationship between file values and owns the built-in client's outbound HTTP behavior.
request overrides.
`ExecutionTargetOverride` uses pointers for numeric settings so an explicit ## Inputs, Profiles, And Overrides
zero remains distinct from no override. `ExtraParams` accepts JSON-compatible
strings, booleans, finite numbers, string-keyed objects, arrays or slices, and
nil. Unsupported values, non-string map keys, non-finite numbers, and cycles
match `ErrInvalidConfig` in profiles or `ErrInvalidRequest` in request
overrides.
Returned requests, profiles, prepared values, results, artifacts, maps, and Use `File`, `Inline`, or `InlineWithURI` to construct request inputs. A request
slices are isolated from internal engine state. Consumers and injected can select a profile explicitly or use the prompt's default profile, and can
extensions should not retain or mutate values owned by another caller. replace execution settings or the complete output contract.
## Profiles And Credentials The [public value GoDoc](../../types.go) defines nil, empty, zero, replacement,
copy, and credential behavior. The
[framework format reference](../formats.md) defines how those request values
interact with prompt definitions, file-backed profiles, built-ins, schemas,
and framework defaults.
[`OpenAICompatibleProfile`](../../profiles.go) constructs an ordinary For programmatic profiles,
in-memory profile for an OpenAI-compatible chat-completions endpoint. [`OpenAICompatibleProfile`](../../profiles.go) converts ordinary
`WithProfiles` rejects duplicate IDs in one call and gives in-memory profiles OpenAI-compatible settings into a value accepted by `WithProfiles`.
precedence over explicit file sources and built-ins.
Raw API keys do not belong in profiles. File-backed profiles may name an ## Credentials
environment variable, while an in-memory profile can require a request key.
A direct `RunRequest.APIKey` is request-scoped and takes precedence over an
environment lookup for the built-in client. Profile fields, ranges, built-ins,
precedence, and credential rules are owned by the
[framework format reference](../formats.md).
API keys are excluded from JSON, prepared values, and results. The public File-backed profiles name an environment variable; in-memory profiles can
`String` and `GoString` methods report only whether a direct key is present. require a direct request key. Direct keys are request-scoped and are excluded
Avoid reflection-based dumps of request structs, which can bypass that from supported JSON values and the package's `String` and `GoString`
redaction. summaries. The exact precedence and redaction guarantees belong to
[`RunRequest`, `GenerateRequest`, and the profile GoDoc](../../types.go).
## Protect Files And Generated Data
The default artifact reader opens a `File` reference as a caller-selected
operating-system path. It does not constrain paths to an application root,
impose an inbound request-size policy, or establish an untrusted-input security
boundary. Applications must validate and restrict untrusted paths and payloads
before constructing a request, or inject an artifact reader that enforces
their filesystem, authorization, and size policies.
Rendered messages, input and output artifact bodies, raw model output, and
validation diagnostics can contain sensitive data. API-key redaction does not
sanitize those values. Treat prepared values, results, collaborator requests,
errors, and logs according to the application's data-access, retention, and
secret-handling policies.
## Extension Interfaces ## Extension Interfaces
The [`LLMClient`, `GenerateRequest`, and Inject an [`LLMClient` or `ArtifactReader`](../../types.go) when the built-in
`GenerateResponse`](../../types.go) boundary lets a consumer replace model behavior does not fit the application. Their GoDoc defines concurrent use,
generation. Injected clients receive copied rendered messages, effective context handling, ownership of copied values, nil responses, and preservation
settings, explicit numeric-setting presence, structured-output constraints, of collaborator errors. Implementations must honor cancellation, safely manage
and the request-scoped key. They return generated content and token usage. copies they retain, avoid unsafe logging of content or credentials, and enforce
the application policy that motivated the injection.
The [`ArtifactReader`](../../types.go) boundary replaces the default inline and ## Handle Errors
file reader for every input. Readers provide artifact content and metadata; the
engine fills an empty artifact name from the input-map key. A reader error
matches `ErrArtifactLoad` while preserving the original identity for
`errors.Is`. A nil artifact with a nil error is also an artifact-load failure.
Extensions should honor context cancellation and avoid logging raw prompts, Use `errors.Is` with the
artifacts, or credentials. [public error sentinels and operation GoDoc](../../engine.go). The declarations
distinguish invalid construction, invalid requests, absent sources,
source-loading failures, collaborator failures, and operational validation
failures. Specific request conditions may also match the broader
`ErrInvalidRequest`, and injected collaborator identities are preserved where
documented.
## Errors ## Application Boundary
The [public error declarations](../../engine.go) and
[mapping](../../errors.go) preserve these sentinel checks through `errors.Is`:
- `ErrInvalidConfig`
- `ErrInvalidRequest`
- `ErrPromptNotFound`
- `ErrProfileNotFound`
- `ErrProfileRequired`
- `ErrPromptLoad`
- `ErrProfileLoad`
- `ErrAPIKeyEnvMissing`
- `ErrArtifactLoad`
- `ErrPromptRender`
- `ErrLLMGenerate`
- `ErrValidation`
`ErrProfileRequired` and `ErrAPIKeyEnvMissing` also match
`ErrInvalidRequest`, allowing either broad request handling or a specific
condition. Wrapped collaborator errors retain their identity where the public
contract promises it.
## Consumer Boundary
Promptkit is an importable library. It does not own a command, inbound HTTP Promptkit is an importable library. It does not own a command, inbound HTTP
API, process configuration, or deployment policy. Scriptorium is one API, process configuration, or deployment policy. Applications map the root
downstream application that maps this root package contract into those package's results and errors into those concerns, including inbound size and
application concerns. trust policy.

View File

@@ -13,6 +13,7 @@ contributor workflow and validation.
| --- | --- | --- | | --- | --- | --- |
| Root `promptkit` package | Provides the supported engine facade, source and injection options, public request and result values, built-in profile construction, extension interfaces, value conversion, redacted formatting, and public error mapping. | [Package GoDoc](../../doc.go), [engine assembly](../../engine.go) | | Root `promptkit` package | Provides the supported engine facade, source and injection options, public request and result values, built-in profile construction, extension interfaces, value conversion, redacted formatting, and public error mapping. | [Package GoDoc](../../doc.go), [engine assembly](../../engine.go) |
| `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) | | `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
| `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, profiles, execution targets, rendering, generation, and validation. | [Domain declarations](../../internal/domain/domain.go) | | `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, profiles, execution targets, rendering, generation, and validation. | [Domain declarations](../../internal/domain/domain.go) |
| `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) | | `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) |
| `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) | | `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |

View File

@@ -41,9 +41,9 @@ The implemented internal components consist of:
- `internal/usecase`, which coordinates preparation and execution across the - `internal/usecase`, which coordinates preparation and execution across the
internal framework components. internal framework components.
The `examples/go-library/prepare` package is a maintained downstream consumer The `examples/go-library/prepare` and `examples/go-library/run` packages are
of the root facade. It does not expose a library package or participate in maintained downstream consumers of the root facade. They do not expose library
internal assembly. packages or participate in internal assembly.
The root facade assembles the internal repositories, renderer, validator, The root facade assembles the internal repositories, renderer, validator,
outbound client, and use-case runner while translating public values and outbound client, and use-case runner while translating public values and

View File

@@ -7,27 +7,91 @@ tags. It does not publish runnable binaries or binary packages and does not
currently use hosted CI. The release maintainer performs and records the currently use hosted CI. The release maintainer performs and records the
required validation. required validation.
The first planned release is `v0.1.0`. Do not create that tag until the `v0.1.0` is the initial published release. Later releases use semantic
framework has been extracted and the resulting public library has passed this `vMAJOR.MINOR.PATCH` tags. Before `v1`, minor releases may change the public
procedure. Later tags use the `vMAJOR.MINOR.PATCH` form. While Promptkit remains API and patch releases preserve compatibility within their minor line. Every
pre-`v1`, release notes must identify intentional public API changes and any pre-`v1` release note must summarize compatibility, identify public API
consumer migration required by them. changes, and state any action required of consumers.
## Prepare The Release Promptkit releases are source-only. The annotated tag message is the release
note; there is no separate hosted release or binary packaging step.
Work from a clean checkout of the intended release commit, outside any Go ## Establish The Candidate
workspace and without a local module replacement. Confirm the source commit is
already published through the normal branch workflow.
From the Promptkit repository root, verify the checkout: Choose a version that has not been published and export it as
`RELEASE_VERSION`. Run every command in this procedure from the Promptkit
repository root in the same POSIX shell. Do not reuse `v0.1.0` or another
existing version.
The following guard derives the release commit from `HEAD` and stops on a
missing or malformed version, a checkout other than synchronized `main`,
uncommitted changes, an active Go workspace, a module replacement, a vendor
tree, or an existing local or remote tag:
```sh ```sh
gowork=$(go env GOWORK) set -eu
test -z "$gowork" || test "$gowork" = off
test -z "$(git status --short)" : "${RELEASE_VERSION:?export an unpublished vMAJOR.MINOR.PATCH version}"
git fetch --tags origin if ! printf '%s\n' "$RELEASE_VERSION" |
grep -Eq '^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$'
then
printf '%s\n' "invalid release version: $RELEASE_VERSION" >&2
exit 1
fi
RELEASE_COMMIT=$(git rev-parse --verify 'HEAD^{commit}')
export RELEASE_COMMIT
check_release_candidate() {
test "$(git branch --show-current)" = main
test -z "$(git status --porcelain)"
gowork_value=$(go env GOWORK)
case "$gowork_value" in
''|off) ;;
*)
printf '%s\n' "active Go workspace: $gowork_value" >&2
return 1
;;
esac
test -z "$(git ls-files go.work go.work.sum)"
test ! -e vendor
if grep -Eq '^[[:space:]]*replace([[:space:]]|\()' go.mod
then
printf '%s\n' 'go.mod contains a replacement' >&2
return 1
fi
git fetch origin main --tags
test "$RELEASE_COMMIT" = \
"$(git rev-parse --verify 'refs/remotes/origin/main^{commit}')"
if git show-ref --verify --quiet "refs/tags/$RELEASE_VERSION"
then
printf '%s\n' "local tag already exists: $RELEASE_VERSION" >&2
return 1
fi
if test -n "$(
git ls-remote --tags origin \
"refs/tags/$RELEASE_VERSION" \
"refs/tags/$RELEASE_VERSION^{}"
)"
then
printf '%s\n' "remote tag already exists: $RELEASE_VERSION" >&2
return 1
fi
}
check_release_candidate
``` ```
Do not continue unless the guard completes successfully. In particular, push
the intended commit through the normal `main` branch workflow before release;
the tag procedure is not a substitute for publishing the source commit.
## Validate The Candidate
Confirm the module and root package metadata: Confirm the module and root package metadata:
```sh ```sh
@@ -42,7 +106,7 @@ gitea.maximumdirect.net/eric/promptkit 1.25.5
promptkit gitea.maximumdirect.net/eric/promptkit promptkit gitea.maximumdirect.net/eric/promptkit
``` ```
Run the same default Go validation required by the Run the complete maintainer validation required by the
[development guide](development.md): [development guide](development.md):
```sh ```sh
@@ -53,84 +117,158 @@ go build ./...
go run ./examples/go-library/prepare go run ./examples/go-library/prepare
``` ```
Check every tracked Go file and repository whitespace: Check every tracked Go file. This command must produce no output:
```sh ```sh
gofmt -l $(git ls-files '*.go') unformatted=$(
git ls-files '*.go' |
while IFS= read -r go_file
do
gofmt -l "$go_file"
done
)
test -z "$unformatted"
```
Follow every maintained Markdown link and confirm that its local or published
target exists. Review the repository for generated binaries, test or coverage
output, credentials, template residue, downloaded assets, and other files that
do not belong in source control.
Recheck module and repository hygiene, whitespace, and the clean checkout:
```sh
test -z "$(git ls-files go.work go.work.sum)"
test ! -e vendor
if grep -Eq '^[[:space:]]*replace([[:space:]]|\()' go.mod
then
printf '%s\n' 'go.mod contains a replacement' >&2
exit 1
fi
git diff --check git diff --check
test -z "$(git status --porcelain)"
``` ```
The formatting command must produce no paths. Follow every maintained Markdown ## Write The Release Note
link and confirm its target exists. Review the repository for generated
binaries, test or coverage output, credentials, template residue, and other
files that do not belong in source control.
Confirm that no workspace override is tracked and that `go.mod` contains no Prepare a plain-text annotated-tag message outside the repository and export
`replace` directive: its path as `RELEASE_NOTES_FILE`. Use this form, replacing each summary with
release-specific text; write `None.` when there are no public API changes or
consumer actions:
```sh ```text
git ls-files go.work go.work.sum Promptkit vMAJOR.MINOR.PATCH
rg -n '^replace\b' go.mod
Validated commit: full commit ID
Compatibility: compatibility summary
Public API changes: changes or None.
Consumer action: required action or None.
``` ```
Both commands must produce no output. Re-run `git status --short` and require a After writing it, require all release-note fields, the selected version, and
clean result after every validation and review check. the validated commit to be present:
## Create And Publish The Tag
Choose the semantic version from the intended compatibility change. Record the
release commit before tagging:
```sh ```sh
release_version=v0.1.0 : "${RELEASE_NOTES_FILE:?export the path to the release-note file}"
release_commit=$(git rev-parse HEAD) test -f "$RELEASE_NOTES_FILE"
test -s "$RELEASE_NOTES_FILE"
grep -F "Promptkit $RELEASE_VERSION" "$RELEASE_NOTES_FILE"
grep -F "Validated commit: $RELEASE_COMMIT" "$RELEASE_NOTES_FILE"
grep -F 'Compatibility:' "$RELEASE_NOTES_FILE"
grep -F 'Public API changes:' "$RELEASE_NOTES_FILE"
grep -F 'Consumer action:' "$RELEASE_NOTES_FILE"
``` ```
Replace the example version for later releases and keep both values in the same Inspect the complete message and confirm that it accurately records the
shell for the remaining commands. Confirm the tag does not already exist compatibility impact, public API changes, and required consumer action.
locally or remotely:
## Create And Inspect The Tag
Run the candidate guard again immediately before tag creation. This ensures
that validation or release-note preparation did not change the checkout and
that the commit is still published and untagged:
```sh ```sh
test -z "$(git tag --list "$release_version")" check_release_candidate
test -z "$(git ls-remote --tags origin "refs/tags/$release_version")"
``` ```
Create an annotated tag whose message identifies the release and records that Create the annotated tag from the prepared release note and bind it explicitly
the documented validation passed for the tagged commit: to the validated commit:
```sh ```sh
git tag --annotate "$release_version" \ git tag --annotate "$RELEASE_VERSION" \
--message "Promptkit $release_version; documented validation passed for $release_commit" --file "$RELEASE_NOTES_FILE" \
"$RELEASE_COMMIT"
``` ```
Inspect the tag before publication: Inspect both the tag message and its source commit before publication:
```sh ```sh
git show --no-patch --decorate "$release_version" test "$(git cat-file -t "refs/tags/$RELEASE_VERSION")" = tag
test "$(git rev-list -n 1 "$release_version")" = "$release_commit" git show --no-patch --decorate "refs/tags/$RELEASE_VERSION"
test "$(
git rev-parse --verify "refs/tags/$RELEASE_VERSION^{commit}"
)" = "$RELEASE_COMMIT"
``` ```
Publish the tag without relying on a hosting-provider-specific release If inspection finds an error, delete the unpublished local tag, correct the
interface: release note or candidate, and repeat the guards. Never move or recreate a tag
that has been published.
## Publish The Selected Tag
Push only the selected tag ref. Do not use `git push --tags`:
```sh ```sh
git push origin "refs/tags/$release_version" git push origin \
"refs/tags/$RELEASE_VERSION:refs/tags/$RELEASE_VERSION"
``` ```
## Verify Publication ## Verify Publication
Confirm that the remote tag object matches the local annotated tag and still Compare the remote annotated-tag object with the local object, then compare the
resolves to the intended source commit: remote peeled source commit with the validated commit:
```sh ```sh
remote_tag=$(git ls-remote --tags origin "refs/tags/$release_version" | awk '{print $1}') remote_tag=$(
test "$remote_tag" = "$(git rev-parse "refs/tags/$release_version")" git ls-remote --tags origin "refs/tags/$RELEASE_VERSION" |
test "$(git rev-list -n 1 "refs/tags/$release_version")" = "$release_commit" awk 'NR == 1 { print $1 }'
)
remote_commit=$(
git ls-remote --tags origin "refs/tags/$RELEASE_VERSION^{}" |
awk 'NR == 1 { print $1 }'
)
test -n "$remote_tag"
test "$remote_tag" = \
"$(git rev-parse --verify "refs/tags/$RELEASE_VERSION")"
test "$remote_commit" = "$RELEASE_COMMIT"
``` ```
Promptkit must publish the required tag before Scriptorium or another consumer Finally, resolve the version as an ordinary Go module in a temporary module
publishes a release that depends on that version. Released consumer modules outside this repository and without a workspace or replacement:
must not use a local replacement or unpublished Promptkit revision.
```sh
resolution_dir=$(mktemp -d)
(
trap 'rm -rf "$resolution_dir"' 0 1 2 15
cd "$resolution_dir"
GOWORK=off go mod init example.com/promptkit-release-check
GOWORK=off go mod download \
"gitea.maximumdirect.net/eric/promptkit@$RELEASE_VERSION"
resolved_version=$(
GOWORK=off go list -m -f '{{.Version}}' \
"gitea.maximumdirect.net/eric/promptkit@$RELEASE_VERSION"
)
test "$resolved_version" = "$RELEASE_VERSION"
)
```
Promptkit must publish and verify the required version before Scriptorium or
another consumer publishes a release that depends on it. This ordering does
not replace the consumer project's own release procedure. Released consumers
must select the published Promptkit tag through ordinary module resolution,
without a workspace, replacement, vendored Promptkit source, or unpublished
revision.
## Policy Changes ## Policy Changes

View File

@@ -0,0 +1,222 @@
# Documentation Hardening Roadmap
## Purpose
This roadmap coordinates a focused pass over Promptkit's documentation and
public contract. The work should close the gaps identified by the documentation
audit, strengthen canonical ownership, and leave consumers and contributors
with guidance that is accurate, navigable, and proportionate to their needs.
This is planning material, not a description of implemented behavior. Follow
the [documentation policy](../policy/documentation.md) throughout the work and
update current-state documents only when their claims are supported by the
implementation and tests.
## Scope And Principles
The work covers public GoDoc, consumer guidance, maintained examples, format
and integration references, architecture ownership, roadmap lifecycle, and
documentation validation.
- Resolve ambiguous behavior before documenting it as a contract.
- Keep exact exported API semantics in Go declarations and GoDoc.
- Keep task-oriented guidance in consumer documents and complete runnable
artifacts in `examples/`.
- Put security-relevant consumer responsibilities at the public boundary, not
only in internal contributor documents.
- Remove duplicated ownership instead of synchronizing parallel references.
- Make documentation checks reproducible where practical.
- Preserve existing behavior unless a stage explicitly selects and tests an
API change.
The roadmap does not add new Promptkit features, redesign framework formats,
or implement ideas from [the future feature catalog](future.md). If resolving
an ambiguity requires a behavioral change, treat that change as a separately
reviewable implementation unit and update its canonical documentation in the
same unit.
## Stage 1: Resolve Public Contract Questions
Before expanding prose, decide the intended contract for exported behavior
that is currently ambiguous.
- [x] Decide whether `RunRequest.Metadata` has a supported observable purpose.
Define its propagation and ownership, or remove or deprecate it through an
intentional public API change.
- [x] Decide and test whether one `Engine` supports concurrent `Prepare` and
`Run` calls.
- [x] Decide how repeated options of the same category behave, including
prompt, profile, schema, model-client, and artifact-reader options.
- [x] Define which public values have supported JSON representations.
- [x] Define JSON time units and omission behavior, including the relationship
between prepared-run and run-result durations.
- [x] Decide whether run IDs and exposed hashes have stable formats or must be
treated as opaque values.
- [x] Confirm the intended transport-timeout default and its zero or negative
configuration semantics.
- [x] Confirm the supported JSON Schema dialect and reference boundaries,
including whether remote references are allowed.
The selected exported API contracts are implemented and tested. Their durable
definitions now belong to the root package declarations and GoDoc.
One format-level decision remains here until Stage 5 moves it to the framework
format reference: JSON Schema uses Draft 2020-12, with that dialect selected
when `$schema` is omitted. Same-document fragments and relative references
contained by a directory or `fs.FS` schema root are supported. A single-file
source supports only references contained in that document. Absolute,
escaping, and remote references are rejected.
**Gate:** Each question has an explicit answer backed by existing behavior or
by an accepted implementation change and proportionate tests. No later stage
should invent a contract merely to fill a documentation gap.
## Stage 2: Make GoDoc The Canonical Public Contract
Strengthen the root package declarations so `go doc` is sufficient to
understand exact public behavior without relying on internal documents.
- [x] Add useful field-level GoDoc to configuration, request, profile,
execution-target, result, artifact, validation, structured-output, and model
client values.
- [x] Document required fields and nil, empty, and zero-value semantics.
- [x] Document override, replacement, profile-precedence, and copy-ownership
behavior where it belongs to the exported API.
- [x] Document credential inputs, redaction, and the values intentionally
excluded from serialization.
- [x] Give each public error sentinel an accurate comment and document the
supported `errors.Is` relationships.
- [x] Document engine concurrency and option-composition behavior selected in
Stage 1.
- [x] Document serialization, time, run-ID, and hash semantics selected in
Stage 1.
- [x] Review constructor, option, extension-interface, `Prepare`, and `Run`
GoDoc for complete failure and cancellation expectations.
Update the [consumer guide](../consumers/pkg-promptkit.md) to summarize and link
to these contracts instead of maintaining exhaustive copies of exported names
or exact semantics.
**Gate:** `go doc -all .` presents a coherent public contract, exported
declarations have accurate comments, and contract tests protect every newly
documented behavior whose compatibility risk warrants durable coverage.
## Stage 3: Improve Consumer Safety And Executable Guidance
Move consumer-relevant security boundaries to the places where consumers will
encounter them and add one representative execution workflow.
- [x] Explain in public GoDoc and the consumer guide that the default file
artifact reader accepts unrestricted caller-selected paths.
- [x] Make clear that Promptkit does not impose an application root, inbound
request-size policy, or untrusted-input security boundary.
- [x] Explain that rendered messages, artifact bodies, raw model output, and
validation details may be sensitive even when credentials are redacted.
- [x] Clarify the responsibilities of injected artifact readers and model
clients for cancellation, copying, logging, and secret handling.
- [x] Add a maintained offline `Run` example using an injected deterministic
model client, without credentials, live network access, or paid calls.
- [x] Link the consumer guide to the execution example and keep embedded
snippets smaller than the maintained artifact.
- [x] Decide whether the existing preparation example should remain separate
or share reusable fixtures without obscuring either workflow.
The preparation and execution examples remain separate, self-contained
workflows. Each keeps its own small prompt fixture so consumers can copy or run
one example without depending on the other.
**Gate:** Both preparation and execution have complete, secret-free,
deterministic consumer examples, and the consumer guide exposes the important
filesystem and data-sensitivity boundaries without leaking internal mechanics.
## Stage 4: Restore Canonical Ownership
Remove parallel definitions and make navigation follow the ownership model in
the documentation policy.
- [ ] Reduce the [architecture policy](../policy/architecture.md) to durable
boundaries, layers, dependency direction, invariants, and non-goals.
- [ ] Keep the exact implemented package and component inventory solely in the
[internal component overview](../internal/overview.md).
- [ ] Review the consumer guide's public error and option lists so they remain
task-oriented summaries rather than duplicate API references.
- [ ] Review internal documents for public-contract statements that should be
links to GoDoc or the format and integration owners.
- [ ] Reconcile the documentation policy's temporary-roadmap lifecycle with
the continuing idea-catalog role of `docs/roadmap/future.md`.
- [ ] Rephrase or link roadmap statements that depend on exact current API
behavior, particularly the runtime reasoning entry.
- [ ] Confirm that every document states its audience or purpose and links to
the canonical owner of adjacent topics.
**Gate:** Every authoritative fact has one clear owner, package inventory
changes no longer require edits to the architecture policy, and roadmaps
cannot be mistaken for current-state references.
## Stage 5: Refine Format And Integration References
Close compatibility gaps in the documents that own file formats and outbound
wire behavior.
- [ ] State or canonically link the exact session-ID limit enforced by the
OpenAI-compatible client.
- [ ] State the configured and default transport-timeout behavior without
referring to an unnamed internal default.
- [ ] Document the JSON Schema dialect and local, contained, and remote
reference behavior selected in Stage 1.
- [ ] Clarify structured-output naming and strictness when those values are
part of the public or integration contract.
- [ ] Add a caveat that built-in profiles are maintained configurations, not a
guarantee of continuing third-party model availability.
- [ ] Recheck every prompt, profile, schema, credential, request-body, response,
timeout, and precedence statement against its owning implementation and
tests.
**Gate:** A consumer can determine the supported file and wire compatibility
boundaries without consulting internal source code or relying on unspecified
defaults.
## Stage 6: Make Documentation Validation Reproducible
Align contributor and release procedures around a small, consistent set of
checks.
- [ ] Use one robust command for checking every tracked Go file with `gofmt`.
- [ ] Provide a repository-local or clearly documented command that validates
local Markdown targets and heading fragments.
- [ ] Decide how published external links are checked without making ordinary
validation depend on mutable network services.
- [ ] Reconcile the validation descriptions in the
[development guide](../development.md),
[testing policy](../policy/testing.md), and
[release procedure](../release.md) so one document owns each requirement.
- [ ] Ensure example validation covers every maintained example added by this
roadmap.
- [ ] Keep documentation-only validation proportionate while requiring full Go
validation when commands, examples, generated output, or checked behavior
changes.
**Gate:** A maintainer can run the documented formatting, link, example, Go,
and repository-hygiene checks exactly as written, with no hidden manual
procedure for local documentation.
## Completion Criteria
The roadmap is complete when:
- all Stage 1 contract questions are resolved;
- GoDoc is the authoritative and sufficient exported API reference;
- consumer guidance covers unrestricted file access and sensitive generated
data;
- maintained offline examples cover both `Prepare` and `Run`;
- architecture, inventory, consumer, internal, format, integration, and
roadmap documents follow their assigned ownership boundaries;
- schema, session, timeout, structured-output, and built-in-profile
compatibility statements are explicit;
- documentation validation is reproducible and consistent across contributor
and release workflows; and
- the complete maintainer validation passes.
After completion, move any durable decisions to GoDoc, policy, format,
integration, or ADR owners as appropriate. Remove this roadmap after incoming
links are updated; do not retain it as a second current-state reference.

122
docs/roadmap/future.md Normal file
View File

@@ -0,0 +1,122 @@
# Future Feature Ideas
## Purpose
This document catalogs reasonably specific ideas that may be useful in future
Promptkit development. It is an idea pool, not a commitment, schedule, or
description of current behavior.
Ideas belong here while they are worth retaining but have not been selected
for active development. Keep each entry at the level of intended capability,
consumer value, and important scope boundaries. Defer API design,
implementation details, sequencing, and acceptance criteria until an idea is
selected.
## Using This Catalog
- Add an idea when its purpose and likely value can be stated clearly.
- Keep entries independent enough that maintainers can evaluate and select
them individually.
- Note significant dependencies or boundary concerns, but do not turn entries
into implementation plans.
- Treat inclusion as an invitation to evaluate, not as approval or priority.
- When an idea is selected, move its active planning to a focused roadmap or,
when it requires a durable architectural decision, an ADR. Update
current-state documentation only when implementation lands.
- Remove ideas that are no longer relevant. Retain a rejected idea only when
its rationale is likely to prevent repeated reconsideration.
Future capabilities must continue to respect the
[architecture policy](../policy/architecture.md), particularly Promptkit's
role as an application-neutral library and its boundary with downstream
consumers.
## Ideas
### Extensible LLM backend registry
Introduce a registry that separates backend-specific connection,
authentication, and limited request defaults from model execution profiles.
Initial support would cover OpenAI-compatible backends and include a small
built-in catalog, potentially starting with OpenRouter. A profile could select
a backend while optionally overriding its default endpoint, and each backend
could name an optional environment variable for its API key without storing
the credential itself. Downstream consumers could register additional,
uniquely named backends, such as OpenAI or unauthenticated local-network
services, but could not replace built-in IDs. Model selection and generation
settings would remain profile concerns, and custom model clients would remain
available for behavior outside the registry's supported protocol.
### Backend-specific concurrency management
Extend the proposed LLM backend registry with optional per-backend concurrency
limits and bounded, buffered admission queues. Promptkit could then route
simultaneous generation requests according to backend capacity while
containing accidental runaway submission. Downstream consumers would continue
invoking synchronous `Run` calls, including concurrently from multiple
goroutines, and each admitted call would wait for and return its ordinary
result.
- Scope limits to an engine instance rather than hidden process-global state.
- Give different backend IDs independent capacity pools. A profile endpoint
override would remain part of its selected backend's pool.
- Configure active concurrency and waiting capacity separately. Concurrency
protects the backend, while queue capacity protects the process from
admitting an unbounded backlog.
- Give queue capacity a generous, configurable bounded default intended as a
safety ceiling for bugs or unintended loops rather than a routine
application constraint. Select an exact default during implementation
planning and measurement.
- Reject a call with a recognizable capacity error when its backend queue is
full rather than allowing it to wait outside the bounded queue.
- Admit requests before expensive preparation and artifact copying where
practical so queued work remains lightweight.
- Apply a limit to each actual generation request, including repair attempts,
without unnecessarily serializing prompt preparation.
- Make queued and active waits respect caller cancellation and deadlines.
- Treat concurrency as backend policy rather than a profile-level model
setting.
- Keep the queue ephemeral and in-process, with no survival guarantee across
engine or process shutdown.
- Preserve the existing execution model as far as practical. Durable jobs,
polling, priorities, application worker lifecycle, retries, and
cross-process coordination would be separate future capabilities.
### Explicit per-run session and reasoning controls
Allow consumers to associate a session ID with each run and to inherit,
replace, or explicitly disable the reasoning effort configured by its selected
profile. Prompt definitions can currently derive a session ID from a template,
and a non-empty runtime `ReasoningEffort` can replace the profile value, but
there is no direct request-level session ID and an empty reasoning value means
that no override was supplied. These controls would let consumers reuse one
prompt and model profile across sessions and reasoning levels without
maintaining duplicate definitions.
- Preserve a prompt's session ID template and a profile's reasoning effort as
reusable defaults.
- Let a directly supplied per-run session ID take precedence over a rendered
prompt default while retaining the existing validation limit and outbound
representation.
- Distinguish an omitted runtime reasoning choice from an explicit request to
disable reasoning.
- Ensure disabling reasoning omits the corresponding provider request setting
rather than relying on a provider-specific magic value.
- Keep the effective session ID and reasoning choice visible in prepared and
run metadata and available to injected model clients without introducing
additional prompt or profile selection mechanisms.
## Entry Format
Use a short heading followed by a concise summary. Add focused bullets when
they help preserve important scope boundaries without becoming an
implementation plan:
```markdown
### Idea name
Describe the intended capability, who benefits, and the most important scope
boundary or dependency.
- Optionally record an important behavior or boundary.
```

165
engine.go
View File

@@ -22,42 +22,93 @@ import (
"gitea.maximumdirect.net/eric/promptkit/internal/validate" "gitea.maximumdirect.net/eric/promptkit/internal/validate"
) )
// ErrInvalidConfig indicates invalid public engine configuration. // ErrInvalidConfig identifies invalid engine construction, including missing
// required configuration, invalid options, and a nil Engine receiver.
var ErrInvalidConfig = errors.New("invalid engine configuration") var ErrInvalidConfig = errors.New("invalid engine configuration")
var ( var (
ErrInvalidRequest = errors.New("invalid run request") // ErrInvalidRequest identifies a request whose required values, overrides,
ErrPromptNotFound = errors.New("prompt not found") // credentials, or effective settings are invalid.
ErrProfileNotFound = errors.New("profile not found") ErrInvalidRequest = errors.New("invalid run request")
ErrProfileRequired = errors.New("profile selection is required") // ErrPromptNotFound identifies a requested prompt ID or version that is not
ErrPromptLoad = errors.New("failed to load prompt definition") // present in the selected prompt source. It does not also match
ErrProfileLoad = errors.New("failed to load execution profile") // ErrPromptLoad.
ErrPromptNotFound = errors.New("prompt not found")
// ErrProfileNotFound identifies a selected profile ID that is absent from
// every configured profile source. It does not also match ErrProfileLoad.
ErrProfileNotFound = errors.New("profile not found")
// ErrProfileRequired identifies a request for which neither RunRequest.ProfileID
// nor the selected prompt's default profile is present. Such an error also
// matches ErrInvalidRequest.
ErrProfileRequired = errors.New("profile selection is required")
// ErrPromptLoad identifies a failure to read, decode, validate, select, or
// hash a prompt definition, except for the not-found case represented by
// ErrPromptNotFound.
ErrPromptLoad = errors.New("failed to load prompt definition")
// ErrProfileLoad identifies a failure to read, decode, validate, or select
// an execution profile, except for the not-found case represented by
// ErrProfileNotFound.
ErrProfileLoad = errors.New("failed to load execution profile")
// ErrAPIKeyEnvMissing identifies an APIKeyEnv whose environment variable is
// unset or empty when no direct RunRequest.APIKey takes precedence. Such an
// error also matches ErrInvalidRequest.
ErrAPIKeyEnvMissing = errors.New("api_key_env points to an unset environment variable") ErrAPIKeyEnvMissing = errors.New("api_key_env points to an unset environment variable")
ErrArtifactLoad = errors.New("failed to load artifact") // ErrArtifactLoad identifies a failure to resolve an input artifact. Errors
ErrPromptRender = errors.New("failed to render prompt") // returned by an injected ArtifactReader remain available through errors.Is.
ErrLLMGenerate = errors.New("failed to generate output") ErrArtifactLoad = errors.New("failed to load artifact")
ErrValidation = errors.New("failed to validate output") // ErrPromptRender identifies a failure to render prompt messages or the
// session ID from the resolved inputs and variables.
ErrPromptRender = errors.New("failed to render prompt")
// ErrLLMGenerate identifies a model-client failure or a nil successful
// response. Errors returned by an injected LLMClient remain available
// through errors.Is.
ErrLLMGenerate = errors.New("failed to generate output")
// ErrValidation identifies an operational failure to load or compile a
// schema or validate output. A completed validation whose Status is
// ValidationFailed is returned in RunResult without this error.
ErrValidation = errors.New("failed to validate output")
) )
// Engine prepares and runs Promptkit prompt requests. // Engine prepares and runs Promptkit prompt requests.
//
// An Engine is safe for concurrent calls to [Engine.Prepare] and [Engine.Run].
// Injected collaborators may consequently be invoked concurrently.
type Engine struct { type Engine struct {
runner *usecase.Runner runner *usecase.Runner
} }
// Config configures a public Promptkit engine. // Config selects the directory-backed sources and built-in model-client
// transport used by [NewEngine]. Config has no stable JSON representation.
type Config struct { type Config struct {
PromptDir string // PromptDir is the directory searched recursively for prompt definitions.
// It is required unless a WithPromptFS or WithPromptFile option supplies the
// prompt source.
PromptDir string
// ProfileDir is an optional directory whose profiles take precedence over
// embedded built-in profiles. An empty value selects only built-ins unless
// profile options are also supplied.
ProfileDir string ProfileDir string
SchemaDir string // SchemaDir is the root for JSON Schema files. An empty value uses the
// current directory. WithSchemaFS or WithSchemaFile replaces this source.
SchemaDir string
// Timeout is the transport-wide safety cap for the built-in LLM client // Timeout is the transport-wide safety cap for the built-in LLM client
// when HTTPClient is absent or has a non-positive timeout. // when HTTPClient is absent or has a non-positive timeout. A zero or negative
// value selects the 10-minute default.
Timeout time.Duration Timeout time.Duration
// HTTPClient is cloned for the built-in LLM client. Its positive Timeout // HTTPClient is cloned for the built-in LLM client. Its positive Timeout
// takes precedence over Config.Timeout as the transport-wide safety cap. // takes precedence over Timeout. A zero or negative client Timeout inherits
// Timeout or the 10-minute default. The supplied client is not mutated. This
// field is ignored when WithLLMClient is used.
HTTPClient *http.Client HTTPClient *http.Client
} }
// Option customizes engine construction. // Option customizes engine construction.
//
// NewEngine applies options in argument order and ignores nil options. Within
// each prompt-source, profile-source, in-memory-profile, schema-source,
// model-client, and artifact-reader category, the last non-nil valid option
// replaces earlier options in that category. An invalid option fails
// construction even if a later option would replace it.
type Option interface { type Option interface {
apply(*engineOptions) error apply(*engineOptions) error
} }
@@ -82,7 +133,10 @@ type engineOptions struct {
artifactSource bool artifactSource bool
} }
// WithLLMClient injects a custom LLM client for execution. // WithLLMClient replaces the built-in model client used by [Engine.Run].
//
// A nil client makes NewEngine fail with ErrInvalidConfig. The client may be
// called concurrently and is not used by [Engine.Prepare].
func WithLLMClient(client LLMClient) Option { func WithLLMClient(client LLMClient) Option {
return optionFunc(func(options *engineOptions) error { return optionFunc(func(options *engineOptions) error {
if client == nil { if client == nil {
@@ -93,7 +147,11 @@ func WithLLMClient(client LLMClient) Option {
}) })
} }
// WithArtifactReader injects a reader for every input artifact reference. // WithArtifactReader replaces the default reader for every input artifact
// reference, regardless of its ArtifactRef.Type.
//
// A nil reader makes NewEngine fail with ErrInvalidConfig. The reader may be
// called concurrently.
func WithArtifactReader(reader ArtifactReader) Option { func WithArtifactReader(reader ArtifactReader) Option {
return optionFunc(func(options *engineOptions) error { return optionFunc(func(options *engineOptions) error {
if reader == nil { if reader == nil {
@@ -109,6 +167,9 @@ func WithArtifactReader(reader ArtifactReader) Option {
// //
// The source uses the same strict prompt YAML rules as configured prompt // The source uses the same strict prompt YAML rules as configured prompt
// directories, and prompt content_file paths resolve within this source. // directories, and prompt content_file paths resolve within this source.
// fsys must be non-nil and root must be non-empty; otherwise NewEngine fails
// with ErrInvalidConfig. This option replaces Config.PromptDir and earlier
// prompt-source options.
func WithPromptFS(fsys fs.FS, root string) Option { func WithPromptFS(fsys fs.FS, root string) Option {
return optionFunc(func(options *engineOptions) error { return optionFunc(func(options *engineOptions) error {
if fsys == nil { if fsys == nil {
@@ -125,7 +186,9 @@ func WithPromptFS(fsys fs.FS, root string) Option {
// WithPromptFile loads prompt definitions from the single prompt file at path. // WithPromptFile loads prompt definitions from the single prompt file at path.
// //
// Relative prompt content_file paths resolve from the file's directory. // Relative prompt content_file paths resolve from the file's directory. path
// must name an existing non-directory file when NewEngine applies the option.
// This option replaces Config.PromptDir and earlier prompt-source options.
func WithPromptFile(path string) Option { func WithPromptFile(path string) Option {
return optionFunc(func(options *engineOptions) error { return optionFunc(func(options *engineOptions) error {
fsys, root, err := fileSource(path) fsys, root, err := fileSource(path)
@@ -142,6 +205,10 @@ func WithPromptFile(path string) Option {
// //
// Profiles from this source overlay built-in profiles. Profile YAML must use // Profiles from this source overlay built-in profiles. Profile YAML must use
// api_key_env for environment-based credentials; raw API keys are rejected. // api_key_env for environment-based credentials; raw API keys are rejected.
// fsys must be non-nil and root must be non-empty; otherwise NewEngine fails
// with ErrInvalidConfig. This option replaces Config.ProfileDir and earlier
// file or FS profile-source options, but remains below WithProfiles in
// precedence.
func WithProfileFS(fsys fs.FS, root string) Option { func WithProfileFS(fsys fs.FS, root string) Option {
return optionFunc(func(options *engineOptions) error { return optionFunc(func(options *engineOptions) error {
if fsys == nil { if fsys == nil {
@@ -159,7 +226,10 @@ func WithProfileFS(fsys fs.FS, root string) Option {
// WithProfileFile loads execution profiles from the single profile file at path. // WithProfileFile loads execution profiles from the single profile file at path.
// //
// The profile overlays built-in profiles. Profile YAML must use api_key_env for // The profile overlays built-in profiles. Profile YAML must use api_key_env for
// environment-based credentials; raw API keys are rejected. // environment-based credentials; raw API keys are rejected. path must name an
// existing non-directory file when NewEngine applies the option. This option
// replaces Config.ProfileDir and earlier file or FS profile-source options,
// but remains below WithProfiles in precedence.
func WithProfileFile(path string) Option { func WithProfileFile(path string) Option {
return optionFunc(func(options *engineOptions) error { return optionFunc(func(options *engineOptions) error {
fsys, root, err := fileSource(path) fsys, root, err := fileSource(path)
@@ -174,6 +244,11 @@ func WithProfileFile(path string) Option {
// WithProfiles configures in-memory profiles that take precedence over // WithProfiles configures in-memory profiles that take precedence over
// configured profile files and built-in profiles. // configured profile files and built-in profiles.
//
// NewEngine validates and copies every profile. IDs must be unique within one
// call. An invalid profile, duplicate ID, or unsupported ExtraParams value
// makes construction fail with ErrInvalidConfig. Repeating WithProfiles
// replaces the complete earlier in-memory set rather than merging it.
func WithProfiles(profiles ...Profile) Option { func WithProfiles(profiles ...Profile) Option {
return optionFunc(func(options *engineOptions) error { return optionFunc(func(options *engineOptions) error {
repo, err := newMemoryProfileRepository(profiles) repo, err := newMemoryProfileRepository(profiles)
@@ -189,7 +264,9 @@ func WithProfiles(profiles ...Profile) Option {
// WithSchemaFS loads JSON Schema documents from fsys under root. // WithSchemaFS loads JSON Schema documents from fsys under root.
// //
// Prompt schema_path values resolve within this source when schema validation // Prompt schema_path values resolve within this source when schema validation
// or structured output is requested. // or structured output is requested. fsys must be non-nil and root must be
// non-empty; otherwise NewEngine fails with ErrInvalidConfig. This option
// replaces Config.SchemaDir and earlier schema-source options.
func WithSchemaFS(fsys fs.FS, root string) Option { func WithSchemaFS(fsys fs.FS, root string) Option {
return optionFunc(func(options *engineOptions) error { return optionFunc(func(options *engineOptions) error {
if fsys == nil { if fsys == nil {
@@ -206,7 +283,9 @@ func WithSchemaFS(fsys fs.FS, root string) Option {
// WithSchemaFile loads JSON Schema documents from the single schema file at path. // WithSchemaFile loads JSON Schema documents from the single schema file at path.
// //
// Prompt schema_path values refer to the file's base name. // Prompt schema_path values refer to the file's base name. path must name an
// existing non-directory file when NewEngine applies the option. This option
// replaces Config.SchemaDir and earlier schema-source options.
func WithSchemaFile(path string) Option { func WithSchemaFile(path string) Option {
return optionFunc(func(options *engineOptions) error { return optionFunc(func(options *engineOptions) error {
fsys, root, err := fileSource(path) fsys, root, err := fileSource(path)
@@ -220,6 +299,15 @@ func WithSchemaFile(path string) Option {
} }
// NewEngine constructs an Engine from configuration and options. // NewEngine constructs an Engine from configuration and options.
//
// Options are applied in order according to [Option]. PromptDir is required
// unless a prompt-source option is present. Construction validates option
// arguments and in-memory profiles but defers reading and validating prompt,
// file-backed profile, and schema contents until Prepare or Run needs them.
//
// NewEngine returns an error matching ErrInvalidConfig for invalid
// configuration or options. It does not perform model requests or require
// credentials.
func NewEngine(cfg Config, opts ...Option) (*Engine, error) { func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
var options engineOptions var options engineOptions
for _, opt := range opts { for _, opt := range opts {
@@ -305,7 +393,22 @@ func fileSource(name string) (fs.FS, string, error) {
return os.DirFS(dir), filepath.ToSlash(base), nil return os.DirFS(dir), filepath.ToSlash(base), nil
} }
// Prepare resolves a prompt request without calling an LLM. // Prepare resolves and renders a prompt request without calling an LLM.
//
// Prepare selects the prompt and profile, resolves effective execution
// settings and the output contract, loads and hashes inputs, loads structured
// output schema metadata when required, and renders the session ID and
// messages. The returned PreparedRun is owned by the caller and never contains
// a resolved API-key value, model output, or validation result.
//
// A nil Engine returns an error matching ErrInvalidConfig. Request and
// preparation failures may match ErrInvalidRequest, ErrPromptNotFound,
// ErrPromptLoad, ErrProfileNotFound, ErrProfileLoad, ErrProfileRequired,
// ErrAPIKeyEnvMissing, ErrArtifactLoad, ErrPromptRender, or ErrValidation as
// applicable. Cancellation is passed to the active collaborator and is
// reported in the applicable operation category; no general errors.Is
// relationship to ctx.Err is promised. Prepare returns no partial result on
// error.
func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, error) { func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, error) {
if e == nil || e.runner == nil { if e == nil || e.runner == nil {
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig) return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
@@ -323,7 +426,21 @@ func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, err
return fromDomainPreparedRun(prepared), nil return fromDomainPreparedRun(prepared), nil
} }
// Run executes a prompt request and returns the generated artifact and metadata. // Run prepares a request, invokes the configured LLMClient, and validates the
// generated output.
//
// A content-validation failure is a successful run whose
// RunResult.Validation has Status ValidationFailed. An inability to perform
// validation returns an error matching ErrValidation and no partial result.
// The public Engine does not perform output repair, so validation is
// single-pass even when OutputContract.RepairAttempts is positive.
//
// Run can return every error category documented by [Engine.Prepare], plus
// ErrLLMGenerate. Errors from injected clients remain available through
// errors.Is. Cancellation is passed through the active collaborator and is
// reported in the applicable operation category; no general errors.Is
// relationship to ctx.Err is promised. A nil Engine returns ErrInvalidConfig.
// Run returns no partial result on error.
func (e *Engine) Run(ctx context.Context, req RunRequest) (*RunResult, error) { func (e *Engine) Run(ctx context.Context, req RunRequest) (*RunResult, error) {
if e == nil || e.runner == nil { if e == nil || e.runner == nil {
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig) return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)

View File

@@ -1020,6 +1020,7 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
client promptkit.LLMClient client promptkit.LLMClient
schemaDir string schemaDir string
want error want error
notWant error
}{ }{
{ {
name: "invalid request", name: "invalid request",
@@ -1028,10 +1029,11 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
want: promptkit.ErrInvalidRequest, want: promptkit.ErrInvalidRequest,
}, },
{ {
name: "prompt not found", name: "prompt not found",
req: promptkit.RunRequest{PromptID: "missing.prompt"}, req: promptkit.RunRequest{PromptID: "missing.prompt"},
client: &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}}, client: &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}},
want: promptkit.ErrPromptNotFound, want: promptkit.ErrPromptNotFound,
notWant: promptkit.ErrPromptLoad,
}, },
{ {
name: "profile not found", name: "profile not found",
@@ -1042,8 +1044,9 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
"transcript": promptkit.Inline("Rin opens the gate."), "transcript": promptkit.Inline("Rin opens the gate."),
}, },
}, },
client: &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}}, client: &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}},
want: promptkit.ErrProfileNotFound, want: promptkit.ErrProfileNotFound,
notWant: promptkit.ErrProfileLoad,
}, },
{ {
name: "artifact load", name: "artifact load",
@@ -1114,6 +1117,9 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
if !errors.Is(err, tc.want) { if !errors.Is(err, tc.want) {
t.Fatalf("expected errors.Is(%v), got %v", tc.want, err) t.Fatalf("expected errors.Is(%v), got %v", tc.want, err)
} }
if tc.notWant != nil && errors.Is(err, tc.notWant) {
t.Fatalf("did not expect errors.Is(%v), got %v", tc.notWant, err)
}
}) })
} }
} }
@@ -1699,6 +1705,16 @@ func TestEngineRunLayersTransportAndGenerationTimeouts(t *testing.T) {
configTimeout: 5 * time.Second, configTimeout: 5 * time.Second,
wantRemainingAtRequest: 5 * time.Second, wantRemainingAtRequest: 5 * time.Second,
}, },
{
name: "zero configuration uses ten minute transport default",
wantRemainingAtRequest: 10 * time.Minute,
},
{
name: "negative configuration uses ten minute transport default",
configTimeout: -2 * time.Second,
suppliedClientTimeout: -3 * time.Second,
wantRemainingAtRequest: 10 * time.Minute,
},
{ {
name: "profile deadline is shorter than transport cap", name: "profile deadline is shorter than transport cap",
suppliedClientTimeout: 6 * time.Second, suppliedClientTimeout: 6 * time.Second,

View File

@@ -0,0 +1,81 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"gitea.maximumdirect.net/eric/promptkit"
)
type deterministicClient struct{}
func (deterministicClient) Generate(
ctx context.Context,
_ promptkit.GenerateRequest,
) (*promptkit.GenerateResponse, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
return &promptkit.GenerateResponse{
Content: "Ada finished the migration review.",
Usage: promptkit.TokenUsage{
PromptTokens: 12,
CompletionTokens: 6,
TotalTokens: 18,
},
}, nil
}
type summary struct {
Output string `json:"output"`
ValidationStatus promptkit.ValidationStatus `json:"validation_status"`
IsValid bool `json:"is_valid"`
Model string `json:"model"`
TotalTokens int `json:"total_tokens"`
}
func main() {
engine, err := promptkit.NewEngine(
promptkit.Config{},
promptkit.WithPromptFile("examples/go-library/run/prompt.yaml"),
promptkit.WithProfiles(promptkit.Profile{
ID: "offline-example",
Endpoint: "https://example.invalid/v1",
Model: "offline-model",
}),
promptkit.WithLLMClient(deterministicClient{}),
)
if err != nil {
exit(err)
}
result, err := engine.Run(context.Background(), promptkit.RunRequest{
PromptID: "example.run",
Inputs: map[string]promptkit.ArtifactRef{
"note": promptkit.Inline("Ada finished the migration review."),
},
})
if err != nil {
exit(err)
}
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", " ")
if err := encoder.Encode(summary{
Output: result.RawOutput,
ValidationStatus: result.Validation.Status,
IsValid: result.Validation.IsValid,
Model: result.ModelName,
TotalTokens: result.Usage.TotalTokens,
}); err != nil {
exit(err)
}
}
func exit(err error) {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}

View File

@@ -0,0 +1,16 @@
id: example.run
version: "1.0.0"
default_profile: offline-example
description: Run a prompt with a deterministic injected model client.
inputs:
- name: note
required: true
content_type: text/plain
messages:
- role: system
content: Summarize the note in one sentence.
- role: user
content: '{{input "note"}}'
output:
format: text
validation_mode: basic

View File

@@ -2,19 +2,23 @@ package promptkit
import "fmt" import "fmt"
// String returns a concise request summary without exposing direct API keys. // String returns a concise request summary without exposing the direct API key
// or input and variable contents. Reflection-based formatting does not carry
// this guarantee.
func (r RunRequest) String() string { func (r RunRequest) String() string {
return r.redactedString() return r.redactedString()
} }
// GoString returns a concise request summary without exposing direct API keys. // GoString returns a concise request summary without exposing the direct API
// key or input and variable contents. Reflection-based formatting does not
// carry this guarantee.
func (r RunRequest) GoString() string { func (r RunRequest) GoString() string {
return r.redactedString() return r.redactedString()
} }
func (r RunRequest) redactedString() string { func (r RunRequest) redactedString() string {
return fmt.Sprintf( return fmt.Sprintf(
"promptkit.RunRequest{PromptID:%q PromptVersion:%q ProfileID:%q APIKeySet:%t Inputs:%d Vars:%d ExecutionSet:%t ValidationSet:%t Metadata:%d}", "promptkit.RunRequest{PromptID:%q PromptVersion:%q ProfileID:%q APIKeySet:%t Inputs:%d Vars:%d ExecutionSet:%t ValidationSet:%t}",
r.PromptID, r.PromptID,
r.PromptVersion, r.PromptVersion,
r.ProfileID, r.ProfileID,
@@ -23,18 +27,19 @@ func (r RunRequest) redactedString() string {
len(r.Vars), len(r.Vars),
r.Execution != nil, r.Execution != nil,
r.Validation != nil, r.Validation != nil,
len(r.Metadata),
) )
} }
// String returns a concise request summary without exposing direct API keys or // String returns a concise request summary without exposing direct API keys or
// rendered prompt content. // rendered prompt content. Reflection-based formatting does not carry this
// guarantee.
func (r GenerateRequest) String() string { func (r GenerateRequest) String() string {
return r.redactedString() return r.redactedString()
} }
// GoString returns a concise request summary without exposing direct API keys or // GoString returns a concise request summary without exposing direct API keys or
// rendered prompt content. // rendered prompt content. Reflection-based formatting does not carry this
// guarantee.
func (r GenerateRequest) GoString() string { func (r GenerateRequest) GoString() string {
return r.redactedString() return r.redactedString()
} }

View File

@@ -68,7 +68,6 @@ type RunRequest struct {
Vars map[string]string Vars map[string]string
Execution *ExecutionTargetOverride Execution *ExecutionTargetOverride
Validation *OutputContract Validation *OutputContract
Metadata map[string]string
} }
// RunResult represents the complete result of a prompt execution run. // RunResult represents the complete result of a prompt execution run.

View File

@@ -6,6 +6,7 @@ import (
"errors" "errors"
"fmt" "fmt"
"io/fs" "io/fs"
"net/url"
"os" "os"
"path" "path"
"path/filepath" "path/filepath"
@@ -16,6 +17,8 @@ import (
"github.com/santhosh-tekuri/jsonschema/v6" "github.com/santhosh-tekuri/jsonschema/v6"
) )
const jsonSchemaDraft2020 = "https://json-schema.org/draft/2020-12/schema"
// StandardValidator provides basic, JSON, and JSON Schema output validation. // StandardValidator provides basic, JSON, and JSON Schema output validation.
type StandardValidator struct { type StandardValidator struct {
schemaBaseDir string schemaBaseDir string
@@ -121,7 +124,11 @@ func (v *StandardValidator) validateJSONSchema(instance any, schemaPath string)
return nil, err return nil, err
} }
compiler := jsonschema.NewCompiler() schemaRoot, err := v.schemaRoot()
if err != nil {
return nil, err
}
compiler := newSchemaCompiler(standardSchemaLoader{root: schemaRoot})
schema, err := compiler.Compile(resolvedSchemaPath) schema, err := compiler.Compile(resolvedSchemaPath)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", resolvedSchemaPath, err) return nil, fmt.Errorf("failed to compile JSON schema %q: %w", resolvedSchemaPath, err)
@@ -140,7 +147,10 @@ func (v *FSValidator) validateJSONSchema(instance any, schemaPath string) ([]str
} }
resourceURL := fsSchemaResourceURL(schemaName) resourceURL := fsSchemaResourceURL(schemaName)
compiler := jsonschema.NewCompiler() if err := validateSchemaDialect(schemaDoc); err != nil {
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", schemaName, err)
}
compiler := newSchemaCompiler(fsSchemaLoader{fsys: v.fsys, root: filecatalog.CleanFSRoot(v.root)})
if err := compiler.AddResource(resourceURL, schemaDoc); err != nil { if err := compiler.AddResource(resourceURL, schemaDoc); err != nil {
return nil, fmt.Errorf("failed to register JSON schema %q: %w", schemaName, err) return nil, fmt.Errorf("failed to register JSON schema %q: %w", schemaName, err)
} }
@@ -184,6 +194,9 @@ func (v *StandardValidator) LoadSchemaDocument(ctx context.Context, schemaPath s
if err := json.Unmarshal(raw, &doc); err != nil { if err := json.Unmarshal(raw, &doc); err != nil {
return nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err) return nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
} }
if err := validateSchemaDialect(doc); err != nil {
return nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
}
return doc, nil return doc, nil
} }
@@ -206,12 +219,14 @@ func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error)
return "", errors.New("schema path is required for json_schema validation") return "", errors.New("schema path is required for json_schema validation")
} }
resolved := schemaPath root, err := v.schemaRoot()
if !filepath.IsAbs(schemaPath) { if err != nil {
resolved = filepath.Join(v.schemaBaseDir, schemaPath) return "", err
}
resolved, err := containedFilesystemPath(root, schemaPath)
if err != nil {
return "", err
} }
resolved = filepath.Clean(resolved)
if _, err := os.Stat(resolved); err != nil { if _, err := os.Stat(resolved); err != nil {
return "", fmt.Errorf("failed to access schema file %q: %w", resolved, err) return "", fmt.Errorf("failed to access schema file %q: %w", resolved, err)
} }
@@ -219,6 +234,22 @@ func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error)
return resolved, nil return resolved, nil
} }
func (v *StandardValidator) schemaRoot() (string, error) {
root := v.schemaBaseDir
if strings.TrimSpace(root) == "" {
root = "."
}
absolute, err := filepath.Abs(root)
if err != nil {
return "", fmt.Errorf("failed to resolve schema source %q: %w", root, err)
}
resolved, err := filepath.EvalSymlinks(absolute)
if err != nil {
return "", fmt.Errorf("failed to access schema source %q: %w", root, err)
}
return resolved, nil
}
func (v *FSValidator) loadSchemaDocument(schemaPath string) (string, any, error) { func (v *FSValidator) loadSchemaDocument(schemaPath string) (string, any, error) {
resolved, err := v.resolveSchemaPath(schemaPath) resolved, err := v.resolveSchemaPath(schemaPath)
if err != nil { if err != nil {
@@ -234,6 +265,9 @@ func (v *FSValidator) loadSchemaDocument(schemaPath string) (string, any, error)
if err := json.Unmarshal(raw, &doc); err != nil { if err := json.Unmarshal(raw, &doc); err != nil {
return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err) return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
} }
if err := validateSchemaDialect(doc); err != nil {
return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
}
return resolved, doc, nil return resolved, doc, nil
} }
@@ -290,3 +324,131 @@ func cleanSchemaFSPath(schemaPath string) (string, error) {
func fsSchemaResourceURL(schemaName string) string { func fsSchemaResourceURL(schemaName string) string {
return "promptkit-schema:///" + strings.TrimPrefix(path.Clean(schemaName), "/") return "promptkit-schema:///" + strings.TrimPrefix(path.Clean(schemaName), "/")
} }
func newSchemaCompiler(loader jsonschema.URLLoader) *jsonschema.Compiler {
compiler := jsonschema.NewCompiler()
compiler.DefaultDraft(jsonschema.Draft2020)
compiler.UseLoader(loader)
return compiler
}
func validateSchemaDialect(doc any) error {
object, ok := doc.(map[string]any)
if !ok {
return nil
}
value, ok := object["$schema"]
if !ok {
return nil
}
dialect, ok := value.(string)
if !ok {
return errors.New("$schema must be a string")
}
if dialect != jsonSchemaDraft2020 && dialect != jsonSchemaDraft2020+"#" {
return fmt.Errorf("unsupported JSON Schema dialect %q; expected %q", dialect, jsonSchemaDraft2020)
}
return nil
}
type standardSchemaLoader struct {
root string
}
func (l standardSchemaLoader) Load(resourceURL string) (any, error) {
fileName, err := (jsonschema.FileLoader{}).ToFile(resourceURL)
if err != nil {
return nil, fmt.Errorf("schema reference %q is not a contained file reference: %w", resourceURL, err)
}
resolved, err := containedFilesystemPath(l.root, fileName)
if err != nil {
return nil, err
}
return loadJSONSchemaFile(resolved)
}
func containedFilesystemPath(root, name string) (string, error) {
candidate := name
if !filepath.IsAbs(candidate) {
candidate = filepath.Join(root, candidate)
}
candidate, err := filepath.Abs(candidate)
if err != nil {
return "", fmt.Errorf("failed to resolve schema path %q: %w", name, err)
}
candidate, err = filepath.EvalSymlinks(candidate)
if err != nil {
return "", fmt.Errorf("failed to access schema file %q: %w", candidate, err)
}
relative, err := filepath.Rel(root, candidate)
if err != nil {
return "", fmt.Errorf("failed to compare schema path %q with source root: %w", candidate, err)
}
if relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
return "", fmt.Errorf("schema path %q escapes source root", name)
}
return candidate, nil
}
func loadJSONSchemaFile(name string) (any, error) {
raw, err := os.ReadFile(name)
if err != nil {
return nil, err
}
var doc any
if err := json.Unmarshal(raw, &doc); err != nil {
return nil, err
}
if err := validateSchemaDialect(doc); err != nil {
return nil, err
}
return doc, nil
}
type fsSchemaLoader struct {
fsys fs.FS
root string
}
func (l fsSchemaLoader) Load(resourceURL string) (any, error) {
parsed, err := url.Parse(resourceURL)
if err != nil {
return nil, fmt.Errorf("invalid schema reference %q: %w", resourceURL, err)
}
if parsed.Scheme != "promptkit-schema" || parsed.Host != "" {
return nil, fmt.Errorf("schema reference %q is not allowed", resourceURL)
}
name, err := url.PathUnescape(strings.TrimPrefix(parsed.Path, "/"))
if err != nil {
return nil, fmt.Errorf("invalid schema reference %q: %w", resourceURL, err)
}
name = path.Clean(name)
if l.root == "." {
if strings.HasPrefix(name, "../") || name == ".." {
return nil, fmt.Errorf("schema reference %q escapes source root", resourceURL)
}
} else if name != l.root && !strings.HasPrefix(name, l.root+"/") {
return nil, fmt.Errorf("schema reference %q escapes source root", resourceURL)
}
rootInfo, err := fs.Stat(l.fsys, l.root)
if err != nil {
return nil, err
}
if !rootInfo.IsDir() && name != l.root {
return nil, fmt.Errorf("schema reference %q is outside the configured schema file", resourceURL)
}
raw, err := fs.ReadFile(l.fsys, name)
if err != nil {
return nil, err
}
var doc any
if err := json.Unmarshal(raw, &doc); err != nil {
return nil, err
}
if err := validateSchemaDialect(doc); err != nil {
return nil, err
}
return doc, nil
}

View File

@@ -2,8 +2,10 @@ package validate
import ( import (
"context" "context"
"encoding/json"
"os" "os"
"path/filepath" "path/filepath"
"strconv"
"strings" "strings"
"testing" "testing"
"testing/fstest" "testing/fstest"
@@ -411,3 +413,138 @@ func TestFSValidatorLoadSchemaDocument(t *testing.T) {
t.Fatalf("unexpected schema document: %#v", doc) t.Fatalf("unexpected schema document: %#v", doc)
} }
} }
func TestStandardValidatorJSONSchemaReferenceBoundaries(t *testing.T) {
root := t.TempDir()
if err := os.WriteFile(filepath.Join(root, "child.json"), []byte(`{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "string"
}`), 0o644); err != nil {
t.Fatal(err)
}
tests := []struct {
name string
reference string
wantError string
writeOuter bool
}{
{name: "contained relative reference", reference: "child.json"},
{name: "remote reference", reference: "https://example.test/schema.json", wantError: "not a contained file reference"},
{name: "escaping reference", reference: "../outside.json", wantError: "escapes source root", writeOuter: true},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if tc.writeOuter {
if err := os.WriteFile(filepath.Join(filepath.Dir(root), "outside.json"), []byte(`{"type":"string"}`), 0o644); err != nil {
t.Fatal(err)
}
}
schema := `{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$ref": ` + strconv.Quote(tc.reference) + `
}`
if err := os.WriteFile(filepath.Join(root, "root.json"), []byte(schema), 0o644); err != nil {
t.Fatal(err)
}
v := NewStandardValidator(root)
result, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`"value"`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "root.json",
})
if tc.wantError == "" {
if err != nil || !result.IsValid {
t.Fatalf("expected contained reference to validate, got result=%#v error=%v", result, err)
}
return
}
if err == nil || !strings.Contains(err.Error(), tc.wantError) {
t.Fatalf("expected error containing %q, got %v", tc.wantError, err)
}
})
}
}
func TestFSValidatorJSONSchemaReferenceBoundaries(t *testing.T) {
tests := []struct {
name string
reference string
wantError string
}{
{name: "same document fragment", reference: "#/$defs/value"},
{name: "contained relative reference", reference: "child.json"},
{name: "remote reference", reference: "https://example.test/schema.json", wantError: "is not allowed"},
{name: "escaping reference", reference: "../outside.json", wantError: "escapes source root"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
rootSchema := `{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$defs": {"value": {"type": "string"}},
"$ref": ` + strconv.Quote(tc.reference) + `
}`
v := NewFSValidator(fstest.MapFS{
"schemas/root.json": &fstest.MapFile{Data: []byte(rootSchema)},
"schemas/child.json": &fstest.MapFile{Data: []byte(`{"type":"string"}`)},
"outside.json": &fstest.MapFile{Data: []byte(`{"type":"string"}`)},
}, "schemas")
result, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`"value"`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "root.json",
})
if tc.wantError == "" {
if err != nil || !result.IsValid {
t.Fatalf("expected supported reference to validate, got result=%#v error=%v", result, err)
}
return
}
if err == nil || !strings.Contains(err.Error(), tc.wantError) {
t.Fatalf("expected error containing %q, got %v", tc.wantError, err)
}
})
}
}
func TestJSONSchemaDialectIsDraft2020(t *testing.T) {
tests := []struct {
name string
dialect string
wantError bool
}{
{name: "omitted uses supported default"},
{name: "draft 2020-12", dialect: "https://json-schema.org/draft/2020-12/schema"},
{name: "draft 7 rejected", dialect: "http://json-schema.org/draft-07/schema#", wantError: true},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
schema := map[string]any{"type": "object"}
if tc.dialect != "" {
schema["$schema"] = tc.dialect
}
data, err := json.Marshal(schema)
if err != nil {
t.Fatal(err)
}
v := NewFSValidator(fstest.MapFS{
"schema.json": &fstest.MapFile{Data: data},
}, ".")
_, err = v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{}`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "schema.json",
})
if tc.wantError {
if err == nil || !strings.Contains(err.Error(), "unsupported JSON Schema dialect") {
t.Fatalf("expected unsupported-dialect error, got %v", err)
}
return
}
if err != nil {
t.Fatalf("expected supported dialect, got %v", err)
}
})
}
}

154
json.go Normal file
View File

@@ -0,0 +1,154 @@
package promptkit
import (
"encoding/json"
"time"
)
// MarshalJSON implements json.Marshaler for PreparedRun. It uses RFC 3339
// timestamps, integer duration_ms, and omits zero timing values.
func (r PreparedRun) MarshalJSON() ([]byte, error) {
var startTime, endTime *time.Time
if !r.StartTime.IsZero() {
startTime = &r.StartTime
}
if !r.EndTime.IsZero() {
endTime = &r.EndTime
}
var durationMS *int64
if r.DurationMS != 0 {
durationMS = &r.DurationMS
}
return json.Marshal(struct {
PromptID string `json:"prompt_id"`
PromptVersion string `json:"prompt_version,omitempty"`
PromptHash string `json:"prompt_hash,omitempty"`
SelectedProfileID string `json:"selected_profile_id"`
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
OutputContract OutputContract `json:"output_contract"`
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
InputHashes map[string]string `json:"input_hashes,omitempty"`
SessionID string `json:"session_id,omitempty"`
RenderedPromptHash string `json:"rendered_prompt_hash"`
Messages []RenderedMessage `json:"messages"`
StartTime *time.Time `json:"start_time,omitempty"`
EndTime *time.Time `json:"end_time,omitempty"`
DurationMS *int64 `json:"duration_ms,omitempty"`
}{
PromptID: r.PromptID,
PromptVersion: r.PromptVersion,
PromptHash: r.PromptHash,
SelectedProfileID: r.SelectedProfileID,
EffectiveModelParams: r.EffectiveModelParams,
OutputContract: r.OutputContract,
StructuredOutput: r.StructuredOutput,
InputHashes: r.InputHashes,
SessionID: r.SessionID,
RenderedPromptHash: r.RenderedPromptHash,
Messages: r.Messages,
StartTime: startTime,
EndTime: endTime,
DurationMS: durationMS,
})
}
// MarshalJSON implements json.Marshaler for RunResult. It encodes Duration as
// integer milliseconds in duration_ms and omits zero timing values.
func (r RunResult) MarshalJSON() ([]byte, error) {
var startTime, endTime *time.Time
if !r.StartTime.IsZero() {
startTime = &r.StartTime
}
if !r.EndTime.IsZero() {
endTime = &r.EndTime
}
var durationMS *int64
if r.Duration != 0 {
value := r.Duration.Milliseconds()
durationMS = &value
}
return json.Marshal(runResultJSON{
RunID: r.RunID,
Artifact: r.Artifact,
RawOutput: r.RawOutput,
Validation: r.Validation,
PromptID: r.PromptID,
PromptVersion: r.PromptVersion,
PromptHash: r.PromptHash,
RenderedPromptHash: r.RenderedPromptHash,
SelectedProfileID: r.SelectedProfileID,
ModelName: r.ModelName,
Endpoint: r.Endpoint,
EffectiveModelParams: r.EffectiveModelParams,
InputHashes: r.InputHashes,
Usage: r.Usage,
StartTime: startTime,
EndTime: endTime,
DurationMS: durationMS,
})
}
// UnmarshalJSON implements json.Unmarshaler for RunResult. It decodes
// duration_ms into Duration with millisecond precision.
func (r *RunResult) UnmarshalJSON(data []byte) error {
var wire runResultJSON
if err := json.Unmarshal(data, &wire); err != nil {
return err
}
*r = RunResult{
RunID: wire.RunID,
Artifact: wire.Artifact,
RawOutput: wire.RawOutput,
Validation: wire.Validation,
PromptID: wire.PromptID,
PromptVersion: wire.PromptVersion,
PromptHash: wire.PromptHash,
RenderedPromptHash: wire.RenderedPromptHash,
SelectedProfileID: wire.SelectedProfileID,
ModelName: wire.ModelName,
Endpoint: wire.Endpoint,
EffectiveModelParams: wire.EffectiveModelParams,
InputHashes: wire.InputHashes,
Usage: wire.Usage,
Duration: time.Duration(valueOrZero(wire.DurationMS)) * time.Millisecond,
}
if wire.StartTime != nil {
r.StartTime = *wire.StartTime
}
if wire.EndTime != nil {
r.EndTime = *wire.EndTime
}
return nil
}
type runResultJSON struct {
RunID string `json:"run_id"`
Artifact Artifact `json:"artifact"`
RawOutput string `json:"raw_output"`
Validation ValidationResult `json:"validation"`
PromptID string `json:"prompt_id"`
PromptVersion string `json:"prompt_version,omitempty"`
PromptHash string `json:"prompt_hash,omitempty"`
RenderedPromptHash string `json:"rendered_prompt_hash"`
SelectedProfileID string `json:"selected_profile_id"`
ModelName string `json:"model_name"`
Endpoint string `json:"endpoint"`
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
InputHashes map[string]string `json:"input_hashes,omitempty"`
Usage TokenUsage `json:"usage"`
StartTime *time.Time `json:"start_time,omitempty"`
EndTime *time.Time `json:"end_time,omitempty"`
DurationMS *int64 `json:"duration_ms,omitempty"`
}
func valueOrZero(value *int64) int64 {
if value == nil {
return 0
}
return *value
}

View File

@@ -16,6 +16,10 @@ import (
// It does not register global state, maintain a model catalog, or resolve // It does not register global state, maintain a model catalog, or resolve
// credentials. If APIKeyRequired is true, callers satisfy it with // credentials. If APIKeyRequired is true, callers satisfy it with
// RunRequest.APIKey. Raw API keys do not belong in profiles. // RunRequest.APIKey. Raw API keys do not belong in profiles.
//
// The function copies the ExtraParams map itself but does not recursively copy
// nested values. Validation and a deep copy occur when NewEngine applies a
// WithProfiles option containing the returned Profile.
func OpenAICompatibleProfile(cfg OpenAICompatibleProfileConfig) Profile { func OpenAICompatibleProfile(cfg OpenAICompatibleProfileConfig) Profile {
return Profile{ return Profile{
ID: cfg.ID, ID: cfg.ID,

377
public_contract_test.go Normal file
View File

@@ -0,0 +1,377 @@
package promptkit_test
import (
"context"
"encoding/json"
"fmt"
"strings"
"sync"
"sync/atomic"
"testing"
"testing/fstest"
"time"
"gitea.maximumdirect.net/eric/promptkit"
)
func TestPreparedRunJSONOmitsZeroTimingValues(t *testing.T) {
payload, err := json.Marshal(promptkit.PreparedRun{})
if err != nil {
t.Fatalf("marshal prepared run: %v", err)
}
for _, field := range []string{"start_time", "end_time", "duration_ms"} {
if strings.Contains(string(payload), `"`+field+`"`) {
t.Fatalf("expected zero %s to be omitted, got %s", field, payload)
}
}
}
func TestPreparedRunJSONTimingRoundTrips(t *testing.T) {
start := time.Date(2026, time.July, 29, 12, 0, 0, 0, time.UTC)
prepared := promptkit.PreparedRun{
PromptID: "prompt",
StartTime: start,
EndTime: start.Add(1250 * time.Millisecond),
DurationMS: 1250,
}
payload, err := json.Marshal(prepared)
if err != nil {
t.Fatalf("marshal prepared run: %v", err)
}
var decoded promptkit.PreparedRun
if err := json.Unmarshal(payload, &decoded); err != nil {
t.Fatalf("unmarshal prepared run: %v", err)
}
if decoded.DurationMS != prepared.DurationMS ||
!decoded.StartTime.Equal(prepared.StartTime) ||
!decoded.EndTime.Equal(prepared.EndTime) {
t.Fatalf("timing values did not round trip: got %#v, want %#v", decoded, prepared)
}
}
func TestRunResultJSONUsesMillisecondsAndRoundTrips(t *testing.T) {
start := time.Date(2026, time.July, 29, 12, 0, 0, 0, time.UTC)
result := promptkit.RunResult{
RunID: "opaque-run-id",
Artifact: promptkit.Artifact{Name: "output", ContentType: "text/plain", Body: []byte("ok")},
StartTime: start,
EndTime: start.Add(1500 * time.Millisecond),
Duration: 1500 * time.Millisecond,
}
payload, err := json.Marshal(result)
if err != nil {
t.Fatalf("marshal run result: %v", err)
}
var object map[string]any
if err := json.Unmarshal(payload, &object); err != nil {
t.Fatalf("decode run result JSON: %v", err)
}
if got := object["duration_ms"]; got != float64(1500) {
t.Fatalf("expected duration_ms=1500, got %#v in %s", got, payload)
}
if _, exists := object["duration"]; exists {
t.Fatalf("unexpected nanosecond duration field in %s", payload)
}
artifact, ok := object["artifact"].(map[string]any)
if !ok || artifact["content_type"] != "text/plain" {
t.Fatalf("expected stable artifact JSON fields, got %#v", object["artifact"])
}
var decoded promptkit.RunResult
if err := json.Unmarshal(payload, &decoded); err != nil {
t.Fatalf("unmarshal run result: %v", err)
}
if decoded.Duration != result.Duration || !decoded.StartTime.Equal(result.StartTime) || !decoded.EndTime.Equal(result.EndTime) {
t.Fatalf("timing values did not round trip: got %#v, want %#v", decoded, result)
}
payload, err = json.Marshal(promptkit.RunResult{})
if err != nil {
t.Fatalf("marshal zero run result: %v", err)
}
for _, field := range []string{"start_time", "end_time", "duration_ms"} {
if strings.Contains(string(payload), `"`+field+`"`) {
t.Fatalf("expected zero %s to be omitted, got %s", field, payload)
}
}
}
func TestEngineValidationIsSinglePass(t *testing.T) {
client := &fakeLLMClient{
response: &promptkit.GenerateResponse{Content: "not-json"},
}
engine := newContractEngineWithOptions(t, frameworkSchemaDir, promptkit.WithLLMClient(client))
result, err := engine.Run(context.Background(), promptkit.RunRequest{
PromptID: frameworkMarkdownSummaryPromptID,
Inputs: map[string]promptkit.ArtifactRef{
"transcript": promptkit.Inline("Rin opens the gate."),
"glossary": promptkit.Inline("gate: A guarded passage."),
},
Validation: &promptkit.OutputContract{
Format: promptkit.FormatJSON,
ValidationMode: promptkit.ValidationJSON,
RepairAttempts: 3,
},
})
if err != nil {
t.Fatalf("run with failed content validation: %v", err)
}
if result.Validation.Status != promptkit.ValidationFailed ||
result.Validation.RepairAttempts != 0 {
t.Fatalf("expected failed single-pass validation, got %#v", result.Validation)
}
if len(client.requests) != 1 {
t.Fatalf("expected one model generation, got %d", len(client.requests))
}
}
func TestRepeatedOptionsUseLastValueInEachCategory(t *testing.T) {
profile := promptkit.Profile{ID: "profile", Endpoint: "http://example.test/v1", Model: "model"}
t.Run("prompt source", func(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("first", "profile", "first"), "."),
promptkit.WithPromptFS(contractPromptFS("second", "profile", "second"), "."),
promptkit.WithProfiles(profile),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "second"})
if err != nil {
t.Fatalf("prepare from last prompt source: %v", err)
}
if prepared.Messages[0].Content != "second" {
t.Fatalf("expected last prompt source, got %#v", prepared.Messages)
}
})
t.Run("profile source", func(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
promptkit.WithProfileFS(contractProfileFS("profile", "first-model"), "."),
promptkit.WithProfileFS(contractProfileFS("profile", "second-model"), "."),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
if err != nil {
t.Fatalf("prepare from last profile source: %v", err)
}
if prepared.EffectiveModelParams.Model != "second-model" {
t.Fatalf("expected last profile source, got %q", prepared.EffectiveModelParams.Model)
}
})
t.Run("in-memory profiles", func(t *testing.T) {
first := profile
first.Model = "first-model"
second := profile
second.Model = "second-model"
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
promptkit.WithProfiles(first),
promptkit.WithProfiles(second),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
if err != nil {
t.Fatalf("prepare from last in-memory profile option: %v", err)
}
if prepared.EffectiveModelParams.Model != "second-model" {
t.Fatalf("expected last in-memory profiles, got %q", prepared.EffectiveModelParams.Model)
}
})
t.Run("schema source", func(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractSchemaPromptFS(), "."),
promptkit.WithProfiles(profile),
promptkit.WithSchemaFS(contractSchemaFS("first"), "."),
promptkit.WithSchemaFS(contractSchemaFS("second"), "."),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "schema-prompt"})
if err != nil {
t.Fatalf("prepare from last schema source: %v", err)
}
schema := prepared.StructuredOutput.JSONSchema.Schema.(map[string]any)
if schema["title"] != "second" {
t.Fatalf("expected last schema source, got %#v", schema)
}
})
t.Run("model client", func(t *testing.T) {
var firstCalls, secondCalls atomic.Int64
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
promptkit.WithProfiles(profile),
promptkit.WithLLMClient(countingLLMClient{calls: &firstCalls}),
promptkit.WithLLMClient(countingLLMClient{calls: &secondCalls}),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
if _, err := engine.Run(context.Background(), promptkit.RunRequest{PromptID: "prompt"}); err != nil {
t.Fatalf("run with last model client: %v", err)
}
if firstCalls.Load() != 0 || secondCalls.Load() != 1 {
t.Fatalf("expected only last client call, got first=%d second=%d", firstCalls.Load(), secondCalls.Load())
}
})
t.Run("artifact reader", func(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractInputPromptFS(), "."),
promptkit.WithProfiles(profile),
promptkit.WithArtifactReader(fixedArtifactReader("first")),
promptkit.WithArtifactReader(fixedArtifactReader("second")),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
PromptID: "input-prompt",
Inputs: map[string]promptkit.ArtifactRef{"input": promptkit.Inline("ignored")},
})
if err != nil {
t.Fatalf("prepare with last artifact reader: %v", err)
}
if prepared.Messages[0].Content != "second" {
t.Fatalf("expected last artifact reader, got %#v", prepared.Messages)
}
})
}
func TestEngineSupportsConcurrentPrepareAndRun(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
promptkit.WithProfiles(promptkit.Profile{
ID: "profile",
Endpoint: "http://example.test/v1",
Model: "model",
}),
promptkit.WithLLMClient(countingLLMClient{}),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
const calls = 40
errs := make(chan error, calls)
var wg sync.WaitGroup
for i := 0; i < calls; i++ {
wg.Add(1)
go func(run bool) {
defer wg.Done()
request := promptkit.RunRequest{PromptID: "prompt"}
if run {
_, err := engine.Run(context.Background(), request)
errs <- err
return
}
_, err := engine.Prepare(context.Background(), request)
errs <- err
}(i%2 == 0)
}
wg.Wait()
close(errs)
for err := range errs {
if err != nil {
t.Fatalf("concurrent call failed: %v", err)
}
}
}
type countingLLMClient struct {
calls *atomic.Int64
}
func (c countingLLMClient) Generate(context.Context, promptkit.GenerateRequest) (*promptkit.GenerateResponse, error) {
if c.calls != nil {
c.calls.Add(1)
}
return &promptkit.GenerateResponse{Content: "ok"}, nil
}
type fixedArtifactReader string
func (r fixedArtifactReader) Read(context.Context, promptkit.ArtifactRef) (*promptkit.Artifact, error) {
return &promptkit.Artifact{Body: []byte(r)}, nil
}
func contractPromptFS(id, profileID, message string) fstest.MapFS {
return fstest.MapFS{
"prompt.yaml": &fstest.MapFile{Data: []byte(fmt.Sprintf(`id: %s
version: "1"
default_profile: %s
messages:
- role: user
content: %q
output:
format: text
validation_mode: none
`, id, profileID, message))},
}
}
func contractInputPromptFS() fstest.MapFS {
return fstest.MapFS{
"prompt.yaml": &fstest.MapFile{Data: []byte(`id: input-prompt
version: "1"
default_profile: profile
inputs:
- name: input
required: true
messages:
- role: user
content: '{{input "input"}}'
output:
format: text
validation_mode: none
`)},
}
}
func contractProfileFS(id, model string) fstest.MapFS {
return fstest.MapFS{
"profile.yaml": &fstest.MapFile{Data: []byte(fmt.Sprintf(`id: %s
endpoint: http://example.test/v1
model: %s
`, id, model))},
}
}
func contractSchemaPromptFS() fstest.MapFS {
return fstest.MapFS{
"prompt.yaml": &fstest.MapFile{Data: []byte(`id: schema-prompt
version: "1"
default_profile: profile
messages:
- role: user
content: message
output:
format: json
validation_mode: json_schema
schema_path: schema.json
`)},
}
}
func contractSchemaFS(title string) fstest.MapFS {
return fstest.MapFS{
"schema.json": &fstest.MapFile{Data: []byte(fmt.Sprintf(`{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": %q,
"type": "object"
}`, title))},
}
}

593
types.go
View File

@@ -5,161 +5,319 @@ import (
"time" "time"
) )
// ArtifactRefType defines how an artifact is referenced. // ArtifactRefType identifies how an [ArtifactRef] supplies content.
type ArtifactRefType string type ArtifactRefType string
const ( const (
// ArtifactRefInline selects ArtifactRef.Body as the content.
ArtifactRefInline ArtifactRefType = "inline" ArtifactRefInline ArtifactRefType = "inline"
ArtifactRefFile ArtifactRefType = "file" // ArtifactRefFile selects the filesystem path in ArtifactRef.URI.
ArtifactRefFile ArtifactRefType = "file"
) )
// OutputFormat defines the desired output format. // OutputFormat identifies the media format of generated output.
// OutputFormat has a stable JSON string representation.
type OutputFormat string type OutputFormat string
const ( const (
FormatText OutputFormat = "text" // FormatText identifies plain-text output.
FormatText OutputFormat = "text"
// FormatMarkdown identifies Markdown output.
FormatMarkdown OutputFormat = "markdown" FormatMarkdown OutputFormat = "markdown"
FormatJSON OutputFormat = "json" // FormatJSON identifies JSON output.
FormatJSON OutputFormat = "json"
) )
// ValidationMode defines the output validation strategy. // ValidationMode identifies how generated output is checked.
// ValidationMode has a stable JSON string representation.
type ValidationMode string type ValidationMode string
const ( const (
ValidationNone ValidationMode = "none" // ValidationNone skips content validation.
ValidationBasic ValidationMode = "basic" ValidationNone ValidationMode = "none"
ValidationJSON ValidationMode = "json" // ValidationBasic requires non-empty output.
ValidationBasic ValidationMode = "basic"
// ValidationJSON requires syntactically valid JSON.
ValidationJSON ValidationMode = "json"
// ValidationJSONSchema requires JSON that satisfies OutputContract.SchemaPath.
ValidationJSONSchema ValidationMode = "json_schema" ValidationJSONSchema ValidationMode = "json_schema"
) )
// ValidationStatus defines the result of a validation check. // ValidationStatus identifies the completed state of an output check.
// ValidationStatus has a stable JSON string representation.
type ValidationStatus string type ValidationStatus string
const ( const (
ValidationPassed ValidationStatus = "passed" // ValidationPassed means the generated output satisfied its contract.
ValidationFailed ValidationStatus = "failed" ValidationPassed ValidationStatus = "passed"
// ValidationFailed means validation completed and rejected the generated
// output. Engine.Run returns this status in a result, not as an error.
ValidationFailed ValidationStatus = "failed"
// ValidationSkipped means ValidationNone selected no content check.
ValidationSkipped ValidationStatus = "skipped" ValidationSkipped ValidationStatus = "skipped"
) )
// CacheControlType defines provider cache behavior for prompt content. // CacheControlType identifies provider cache behavior for prompt content.
// CacheControlType has a stable JSON string representation.
type CacheControlType string type CacheControlType string
const ( const (
// CacheControlEphemeral requests provider-defined ephemeral caching.
CacheControlEphemeral CacheControlType = "ephemeral" CacheControlEphemeral CacheControlType = "ephemeral"
) )
// StructuredOutputType identifies provider-level structured output modes. // StructuredOutputType identifies provider-level structured output modes.
// StructuredOutputType has a stable JSON string representation.
type StructuredOutputType string type StructuredOutputType string
const ( const (
// StructuredOutputJSONSchema supplies JSON Schema response constraints.
StructuredOutputJSONSchema StructuredOutputType = "json_schema" StructuredOutputJSONSchema StructuredOutputType = "json_schema"
) )
// RunRequest represents a request to prepare or run a single prompt. // RunRequest selects one prompt execution. It has no stable JSON
// representation.
//
// Prepare and Run copy the request's maps, pointers, and nested
// JSON-compatible values before using them. The caller may mutate the request
// after either method returns.
type RunRequest struct { type RunRequest struct {
PromptID string // PromptID is the required non-empty prompt identifier.
PromptID string
// PromptVersion optionally selects one version of PromptID. When empty, the
// prompt source must contain exactly one matching version.
PromptVersion string PromptVersion string
ProfileID string // ProfileID selects an execution profile. When empty, the prompt's default
APIKey string `json:"-"` // profile is used; if both are empty, the error matches ErrProfileRequired
Inputs map[string]ArtifactRef // and ErrInvalidRequest.
Vars map[string]string ProfileID string
Execution *ExecutionTargetOverride // APIKey is a request-scoped direct credential. It takes precedence over
Validation *OutputContract // APIKeyEnv, is passed to the selected LLMClient, and is never included in
Metadata map[string]string // prepared values, results, hashes, JSON, String, or GoString output.
APIKey string `json:"-"`
// Inputs maps prompt input names to references. A nil or empty map is valid
// only when the selected prompt and its templates require no inputs.
Inputs map[string]ArtifactRef
// Vars supplies Go-template data for messages and the session ID. Nil and
// empty maps are equivalent.
Vars map[string]string
// Execution optionally overrides individual profile execution settings.
// Nil uses the selected profile over framework defaults.
Execution *ExecutionTargetOverride
// Validation optionally replaces the prompt's complete output contract. It
// does not merge individual fields. Nil uses the prompt contract.
Validation *OutputContract
} }
// PreparedRun contains prepared prompt execution state. It does not include // PreparedRun contains prepared prompt execution state. It does not include
// resolved API key values, model output, validation results, or internal target // resolved API key values, model output, validation results, or internal target
// presence metadata. // presence metadata. PreparedRun has a stable JSON representation.
//
// All maps, slices, pointers, and schema values are caller-owned copies. JSON
// timestamps use RFC 3339 and zero timing values are omitted. Hash formats are
// opaque.
type PreparedRun struct { type PreparedRun struct {
PromptID string `json:"prompt_id"` // PromptID is the selected prompt identifier.
PromptVersion string `json:"prompt_version,omitempty"` PromptID string `json:"prompt_id"`
PromptHash string `json:"prompt_hash,omitempty"` // PromptVersion is the selected prompt version.
SelectedProfileID string `json:"selected_profile_id"` PromptVersion string `json:"prompt_version,omitempty"`
EffectiveModelParams ExecutionTarget `json:"effective_model_params"` // PromptHash is an opaque equality value for the selected definition.
OutputContract OutputContract `json:"output_contract"` PromptHash string `json:"prompt_hash,omitempty"`
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"` // SelectedProfileID is the explicit request profile or prompt default that
InputHashes map[string]string `json:"input_hashes,omitempty"` // supplied execution settings.
SessionID string `json:"session_id,omitempty"` SelectedProfileID string `json:"selected_profile_id"`
RenderedPromptHash string `json:"rendered_prompt_hash"` // EffectiveModelParams contains framework defaults overlaid by the selected
Messages []RenderedMessage `json:"messages"` // profile and then request overrides. It excludes resolved API-key values.
StartTime time.Time `json:"start_time,omitempty"` EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
EndTime time.Time `json:"end_time,omitempty"` // OutputContract is the complete effective output contract.
DurationMS int64 `json:"duration_ms,omitempty"` OutputContract OutputContract `json:"output_contract"`
// StructuredOutput is non-nil for JSON Schema validation and contains the
// provider-facing response constraint passed to an LLM client.
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
// InputHashes maps every supplied input name to its opaque artifact hash.
InputHashes map[string]string `json:"input_hashes,omitempty"`
// SessionID is the trimmed rendered session identifier, if any.
SessionID string `json:"session_id,omitempty"`
// RenderedPromptHash is an opaque equality value for SessionID and Messages.
RenderedPromptHash string `json:"rendered_prompt_hash"`
// Messages are the rendered messages that Run passes to the LLM client.
Messages []RenderedMessage `json:"messages"`
// StartTime is the UTC time at which preparation began.
StartTime time.Time `json:"start_time,omitempty"`
// EndTime is the UTC time at which preparation completed.
EndTime time.Time `json:"end_time,omitempty"`
// DurationMS is preparation elapsed time in integer milliseconds. JSON uses
// duration_ms and omits a zero value.
DurationMS int64 `json:"duration_ms,omitempty"`
} }
// RunResult contains generated output, validation state, and run metadata. // RunResult contains generated output, validation state, and run metadata.
// RunResult has a stable JSON representation and round-trips its Duration
// through the duration_ms JSON field.
//
// All maps, slices, and nested values are caller-owned copies. JSON timestamps
// use RFC 3339 and zero timing values are omitted. Run IDs and hash formats are
// opaque.
type RunResult struct { type RunResult struct {
RunID string `json:"run_id"` // RunID is an opaque identifier for this invocation.
Artifact Artifact `json:"artifact"` RunID string `json:"run_id"`
RawOutput string `json:"raw_output"` // Artifact contains the generated output and derived metadata.
Validation ValidationResult `json:"validation"` Artifact Artifact `json:"artifact"`
PromptID string `json:"prompt_id"` // RawOutput is the exact generated content before artifact classification
PromptVersion string `json:"prompt_version,omitempty"` // and validation.
PromptHash string `json:"prompt_hash,omitempty"` RawOutput string `json:"raw_output"`
RenderedPromptHash string `json:"rendered_prompt_hash"` // Validation records the completed content check.
SelectedProfileID string `json:"selected_profile_id"` Validation ValidationResult `json:"validation"`
ModelName string `json:"model_name"` // PromptID is the selected prompt identifier.
Endpoint string `json:"endpoint"` PromptID string `json:"prompt_id"`
EffectiveModelParams ExecutionTarget `json:"effective_model_params"` // PromptVersion is the selected prompt version.
InputHashes map[string]string `json:"input_hashes,omitempty"` PromptVersion string `json:"prompt_version,omitempty"`
Usage TokenUsage `json:"usage"` // PromptHash is the same opaque definition equality value exposed by
StartTime time.Time `json:"start_time,omitempty"` // PreparedRun.
EndTime time.Time `json:"end_time,omitempty"` PromptHash string `json:"prompt_hash,omitempty"`
Duration time.Duration `json:"duration,omitempty"` // RenderedPromptHash is the same opaque rendered-prompt equality value
// computed during preparation.
RenderedPromptHash string `json:"rendered_prompt_hash"`
// SelectedProfileID identifies the profile used for execution.
SelectedProfileID string `json:"selected_profile_id"`
// ModelName is the effective model name and equals
// EffectiveModelParams.Model.
ModelName string `json:"model_name"`
// Endpoint is the effective base endpoint and equals
// EffectiveModelParams.Endpoint.
Endpoint string `json:"endpoint"`
// EffectiveModelParams contains the settings supplied to the LLM client,
// excluding resolved API-key values.
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
// InputHashes are the opaque input equality values computed during
// preparation.
InputHashes map[string]string `json:"input_hashes,omitempty"`
// Usage is the token accounting reported by the LLM client.
Usage TokenUsage `json:"usage"`
// StartTime is the UTC time immediately before preparation begins.
StartTime time.Time `json:"start_time,omitempty"`
// EndTime is the UTC time after generation and validation complete.
EndTime time.Time `json:"end_time,omitempty"`
// Duration covers preparation, generation, and validation. JSON represents
// it as integer milliseconds in duration_ms and omits a zero value.
Duration time.Duration `json:"-"`
} }
// ArtifactRef represents a reference to prompt input content. // ArtifactRef identifies prompt input content. It has no stable JSON
// representation. Prefer [File], [Inline], or [InlineWithURI] to construct one.
type ArtifactRef struct { type ArtifactRef struct {
// Type must be ArtifactRefInline or ArtifactRefFile.
Type ArtifactRefType Type ArtifactRefType
URI string // URI is the file path for ArtifactRefFile and optional provenance metadata
// for ArtifactRefInline.
URI string
// Body is the content for ArtifactRefInline and is ignored for
// ArtifactRefFile.
Body string Body string
} }
// Artifact represents loaded artifact content. // Artifact represents loaded or generated content and has a stable JSON
// representation. Body uses encoding/json's base64 representation for []byte.
type Artifact struct { type Artifact struct {
Name string // Name is artifact metadata. During input preparation the engine fills an
ContentType string // empty reader-supplied name with the request input-map key.
Body []byte Name string `json:"name"`
URI string // ContentType is the media type reported by the reader or derived for
Size int64 // generated output.
Hash string ContentType string `json:"content_type"`
// Body is the artifact content. Engine boundaries copy this slice.
Body []byte `json:"body"`
// URI is optional source or result provenance metadata.
URI string `json:"uri"`
// Size is content-size metadata in bytes.
Size int64 `json:"size"`
// Hash is an opaque content equality value when the producing reader
// supplies one. Its format and algorithm are not API contracts.
Hash string `json:"hash"`
} }
// ArtifactReader resolves a prompt input reference into its content. // ArtifactReader resolves a prompt input reference into its content.
// //
// Readers are responsible for supplying artifact metadata. The engine assigns // Read may be called concurrently. It must honor ctx cancellation to make
// an input-map name only when the returned artifact name is empty. // Prepare and Run responsive to cancellation. The engine passes a copied ref
// and immediately copies the returned Artifact.Body; it does not retain either
// value. Readers supply artifact metadata, and the engine assigns an input-map
// name only when the returned artifact name is empty.
//
// An injected reader owns any application-specific path containment,
// authorization, content-size, and content-type policy. It must protect
// sensitive references and bodies in its logging and in any copies it retains.
// It may reuse or mutate the returned artifact and body after Read returns.
//
// Returning a non-nil error makes the engine return an error matching
// ErrArtifactLoad while preserving the reader error through errors.Is.
// Returning a nil artifact with a nil error also produces ErrArtifactLoad.
type ArtifactReader interface { type ArtifactReader interface {
Read(context.Context, ArtifactRef) (*Artifact, error) Read(context.Context, ArtifactRef) (*Artifact, error)
} }
// ExecutionTarget represents effective model runtime settings. // ExecutionTarget represents effective model runtime settings and has a stable
// JSON representation. It never exposes a resolved API-key value.
type ExecutionTarget struct { type ExecutionTarget struct {
Endpoint string `json:"endpoint"` // Endpoint is the model-provider base URL.
Model string `json:"model"` Endpoint string `json:"endpoint"`
Temperature float64 `json:"temperature"` // Model is the provider model identifier.
MaxTokens int `json:"max_tokens"` Model string `json:"model"`
TopP float64 `json:"top_p"` // Temperature is the effective sampling temperature from 0 through 2.
TimeoutSeconds int `json:"timeout_seconds"` Temperature float64 `json:"temperature"`
ServiceTier string `json:"service_tier"` // MaxTokens is the non-negative effective output-token limit. Zero leaves
ReasoningEffort string `json:"reasoning_effort"` // the limit unspecified to compatible providers unless it was an explicit
APIKeyEnv string `json:"api_key_env"` // request override.
ExtraParams map[string]any `json:"extra_params"` MaxTokens int `json:"max_tokens"`
// TopP is the effective nucleus-sampling value from 0 through 1.
TopP float64 `json:"top_p"`
// TimeoutSeconds is the non-negative per-generation deadline. Zero disables
// this deadline without disabling caller cancellation or the transport cap.
TimeoutSeconds int `json:"timeout_seconds"`
// ServiceTier is an optional provider-specific request tier.
ServiceTier string `json:"service_tier"`
// ReasoningEffort is an optional provider-specific reasoning setting.
ReasoningEffort string `json:"reasoning_effort"`
// APIKeyEnv is an environment-variable name, not its credential value.
APIKeyEnv string `json:"api_key_env"`
// ExtraParams contains copied JSON-compatible provider parameters.
ExtraParams map[string]any `json:"extra_params"`
} }
// ExecutionTargetOverride represents per-request runtime setting overrides. // ExecutionTargetOverride represents per-request runtime setting overrides and
// has no stable JSON representation.
//
// Non-empty string fields replace profile values. Non-nil numeric pointers
// replace profile values and preserve explicit zero. A non-empty ExtraParams
// map replaces the complete profile map rather than merging keys. Empty string
// fields, nil pointers, and a nil or empty ExtraParams map inherit the selected
// profile over framework defaults.
type ExecutionTargetOverride struct { type ExecutionTargetOverride struct {
Endpoint string // Endpoint replaces the profile endpoint when non-empty.
Model string Endpoint string
Temperature *float64 // Model replaces the profile model when non-empty.
MaxTokens *int Model string
TopP *float64 // Temperature, when non-nil, must point to a value from 0 through 2.
TimeoutSeconds *int Temperature *float64
ServiceTier string // MaxTokens, when non-nil, must point to a non-negative value.
MaxTokens *int
// TopP, when non-nil, must point to a value from 0 through 1.
TopP *float64
// TimeoutSeconds, when non-nil, must point to a non-negative value. A
// pointed-to zero disables the per-generation deadline.
TimeoutSeconds *int
// ServiceTier replaces the profile value when non-blank.
ServiceTier string
// ReasoningEffort replaces the profile value when non-blank. An empty value
// cannot clear a profile setting.
ReasoningEffort string ReasoningEffort string
APIKeyEnv string // APIKeyEnv replaces the profile environment-variable name when non-blank.
ExtraParams map[string]any // A direct RunRequest.APIKey still takes precedence over environment lookup.
APIKeyEnv string
// ExtraParams, when non-empty, replaces the profile map. Values must be
// JSON-compatible: nil, booleans, finite numbers, strings, arrays or slices,
// and maps with non-empty string keys. Cycles are invalid.
ExtraParams map[string]any
} }
// Profile is an in-memory execution profile for library consumers. // Profile is an in-memory execution profile for library consumers.
@@ -167,19 +325,39 @@ type ExecutionTargetOverride struct {
// It is equivalent to a loaded profile file after validation. Raw API keys do // It is equivalent to a loaded profile file after validation. Raw API keys do
// not belong in profiles; use APIKeyRequired to require callers to provide // not belong in profiles; use APIKeyRequired to require callers to provide
// RunRequest.APIKey for each request, or use profile YAML api_key_env with file // RunRequest.APIKey for each request, or use profile YAML api_key_env with file
// and FS profile sources. // and FS profile sources. Profile has no stable JSON representation.
//
// WithProfiles validates and copies Profile values during NewEngine. Numeric
// zero, blank strings, and an empty ExtraParams map inherit framework defaults;
// use ExecutionTargetOverride pointer fields to request explicit numeric zero.
type Profile struct { type Profile struct {
ID string // ID is the required non-blank profile identifier. WithProfiles trims it.
Endpoint string ID string
Model string // Endpoint is the required non-blank model-provider base URL.
Temperature float64 Endpoint string
MaxTokens int // Model is the required non-blank provider model identifier.
TopP float64 Model string
TimeoutSeconds int // Temperature is from 0 through 2. Zero inherits the framework default.
ServiceTier string Temperature float64
// MaxTokens is non-negative. Zero inherits the framework default.
MaxTokens int
// TopP is from 0 through 1. Zero inherits the framework default rather than
// selecting an explicit zero.
TopP float64
// TimeoutSeconds is non-negative. Zero inherits the framework default.
TimeoutSeconds int
// ServiceTier is optional; a blank value inherits the framework default.
ServiceTier string
// ReasoningEffort is optional; a blank value inherits the framework
// default.
ReasoningEffort string ReasoningEffort string
APIKeyRequired bool // APIKeyRequired requires a non-blank RunRequest.APIKey. It does not store a
ExtraParams map[string]any // credential or enable environment lookup.
APIKeyRequired bool
// ExtraParams contains provider-specific JSON-compatible values. An empty
// map inherits framework defaults. WithProfiles validates and deeply copies
// it during NewEngine.
ExtraParams map[string]any
} }
// OpenAICompatibleProfileConfig configures an OpenAI-compatible in-memory // OpenAICompatibleProfileConfig configures an OpenAI-compatible in-memory
@@ -187,120 +365,225 @@ type Profile struct {
// //
// It contains ordinary profile fields for OpenAI-compatible chat-completions // It contains ordinary profile fields for OpenAI-compatible chat-completions
// endpoints. APIKeyRequired is satisfied by RunRequest.APIKey. Raw API keys do // endpoints. APIKeyRequired is satisfied by RunRequest.APIKey. Raw API keys do
// not belong in this config. // not belong in this config. OpenAICompatibleProfileConfig has no stable JSON
// representation and is not validated until its resulting Profile is supplied
// through WithProfiles to NewEngine.
type OpenAICompatibleProfileConfig struct { type OpenAICompatibleProfileConfig struct {
ID string // ID becomes Profile.ID.
Endpoint string ID string
Model string // Endpoint becomes Profile.Endpoint.
APIKeyRequired bool Endpoint string
Temperature float64 // Model becomes Profile.Model.
MaxTokens int Model string
TopP float64 // APIKeyRequired becomes Profile.APIKeyRequired.
TimeoutSeconds int APIKeyRequired bool
ServiceTier string // Temperature becomes Profile.Temperature.
Temperature float64
// MaxTokens becomes Profile.MaxTokens.
MaxTokens int
// TopP becomes Profile.TopP.
TopP float64
// TimeoutSeconds becomes Profile.TimeoutSeconds.
TimeoutSeconds int
// ServiceTier becomes Profile.ServiceTier.
ServiceTier string
// ReasoningEffort becomes Profile.ReasoningEffort.
ReasoningEffort string ReasoningEffort string
ExtraParams map[string]any // ExtraParams becomes a shallow-copied Profile.ExtraParams map. NewEngine
// performs validation and a deep copy when WithProfiles applies the result.
ExtraParams map[string]any
} }
// ExecutionTargetPresence tracks which numeric runtime settings were explicit // ExecutionTargetPresence tracks which numeric runtime settings were explicit
// request overrides. // request overrides, including explicit zero values. It has a stable JSON
// representation and is supplied to injected LLM clients so they can preserve
// omission semantics.
type ExecutionTargetPresence struct { type ExecutionTargetPresence struct {
Temperature bool // Temperature reports a non-nil ExecutionTargetOverride.Temperature.
MaxTokens bool Temperature bool `json:"temperature"`
TopP bool // MaxTokens reports a non-nil ExecutionTargetOverride.MaxTokens.
TimeoutSeconds bool MaxTokens bool `json:"max_tokens"`
// TopP reports a non-nil ExecutionTargetOverride.TopP.
TopP bool `json:"top_p"`
// TimeoutSeconds reports a non-nil ExecutionTargetOverride.TimeoutSeconds.
TimeoutSeconds bool `json:"timeout_seconds"`
} }
// OutputContract defines output and validation requirements. // OutputContract defines output and validation requirements and has a stable
// JSON representation.
//
// A non-nil RunRequest.Validation replaces the complete prompt contract. It
// does not merge fields. The public Engine validates generated output once and
// does not install an output repairer.
type OutputContract struct { type OutputContract struct {
Format OutputFormat `json:"format"` // Format selects generated artifact metadata. An empty effective value
// defaults to FormatText.
Format OutputFormat `json:"format"`
// ValidationMode selects the content check. Use one of the declared
// ValidationMode constants.
ValidationMode ValidationMode `json:"validation_mode"` ValidationMode ValidationMode `json:"validation_mode"`
SchemaPath string `json:"schema_path"` // SchemaPath is required when ValidationMode is ValidationJSONSchema and is
RepairAttempts int `json:"repair_attempts"` // ignored by other modes.
SchemaPath string `json:"schema_path"`
// RepairAttempts is a requested repair limit. A non-positive value requests
// no repairs. The public Engine performs no repairs even when this value is
// positive, so its runs report zero attempts used.
RepairAttempts int `json:"repair_attempts"`
} }
// ValidationResult represents output validation state. // ValidationResult represents a completed output check and has a stable JSON
// representation. An operational inability to perform validation is returned
// as ErrValidation instead of a ValidationResult.
type ValidationResult struct { type ValidationResult struct {
Status ValidationStatus `json:"status"` // Status is Passed, Failed, or Skipped.
Mode ValidationMode `json:"mode"` Status ValidationStatus `json:"status"`
Errors []string `json:"errors,omitempty"` // Mode is the effective validation mode.
SchemaPath string `json:"schema_path,omitempty"` Mode ValidationMode `json:"mode"`
RepairAttempts int `json:"repair_attempts"` // Errors contains validation diagnostics when Status is ValidationFailed.
IsValid bool `json:"is_valid"` Errors []string `json:"errors,omitempty"`
// SchemaPath is the effective schema path for JSON Schema validation.
SchemaPath string `json:"schema_path,omitempty"`
// RepairAttempts is the number of repairs actually attempted. It is always
// zero for the public Engine.
RepairAttempts int `json:"repair_attempts"`
// IsValid is true for ValidationPassed and ValidationSkipped and false for
// ValidationFailed.
IsValid bool `json:"is_valid"`
} }
// TokenUsage tracks token consumption. // TokenUsage contains model-client token accounting and has a stable JSON
// representation. Promptkit preserves values reported by the client and does
// not derive or reconcile them.
type TokenUsage struct { type TokenUsage struct {
PromptTokens int `json:"prompt_tokens"` // PromptTokens is the reported input-token count.
PromptTokens int `json:"prompt_tokens"`
// CompletionTokens is the reported generated-token count.
CompletionTokens int `json:"completion_tokens"` CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"` // TotalTokens is the reported total-token count.
CachedTokens int `json:"cached_tokens"` TotalTokens int `json:"total_tokens"`
// CachedTokens is the reported cached-input-token count.
CachedTokens int `json:"cached_tokens"`
// CacheWriteTokens is the reported cache-write-token count.
CacheWriteTokens int `json:"cache_write_tokens"` CacheWriteTokens int `json:"cache_write_tokens"`
} }
// RenderedPrompt is the fully rendered prompt passed to an LLM client. // RenderedPrompt is the fully rendered prompt passed to an LLM client and has
// a stable JSON representation.
type RenderedPrompt struct { type RenderedPrompt struct {
SessionID string `json:"session_id,omitempty"` // SessionID is the optional trimmed session identifier rendered from the
Messages []RenderedMessage `json:"messages"` // prompt definition.
SessionID string `json:"session_id,omitempty"`
// Messages contains rendered messages in definition order.
Messages []RenderedMessage `json:"messages"`
} }
// RenderedMessage is a rendered chat message. // RenderedMessage is a rendered chat message and has a stable JSON
// representation.
type RenderedMessage struct { type RenderedMessage struct {
Role string `json:"role"` // Role is the definition-supplied chat role.
Content string `json:"content"` Role string `json:"role"`
// Content is the rendered message text.
Content string `json:"content"`
// CacheControl is optional provider cache metadata.
CacheControl *CacheControl `json:"cache_control,omitempty"` CacheControl *CacheControl `json:"cache_control,omitempty"`
} }
// CacheControl describes provider cache metadata attached to prompt content. // CacheControl describes provider cache metadata attached to prompt content
// and has a stable JSON representation.
type CacheControl struct { type CacheControl struct {
// Type identifies the cache behavior.
Type CacheControlType `json:"type"` Type CacheControlType `json:"type"`
TTL string `json:"ttl,omitempty"` // TTL is an optional provider cache lifetime.
TTL string `json:"ttl,omitempty"`
} }
// StructuredOutputSpec describes provider-level structured output. // StructuredOutputSpec describes provider-level structured output and has a
// stable JSON representation.
type StructuredOutputSpec struct { type StructuredOutputSpec struct {
Type StructuredOutputType `json:"type"` // Type identifies the structured-output mechanism.
Type StructuredOutputType `json:"type"`
// JSONSchema contains constraints when Type is StructuredOutputJSONSchema.
JSONSchema *StructuredOutputJSONSpec `json:"json_schema,omitempty"` JSONSchema *StructuredOutputJSONSpec `json:"json_schema,omitempty"`
} }
// StructuredOutputJSONSpec contains JSON Schema output constraints. // StructuredOutputJSONSpec contains provider-facing JSON Schema output
// constraints and has a stable JSON representation.
type StructuredOutputJSONSpec struct { type StructuredOutputJSONSpec struct {
Name string `json:"name"` // Name is the provider-facing schema name.
Strict bool `json:"strict"` Name string `json:"name"`
Schema any `json:"schema"` // Strict requests strict provider enforcement of Schema.
Strict bool `json:"strict"`
// Schema is a caller-owned copy of the loaded JSON Schema document.
Schema any `json:"schema"`
} }
// LLMClient executes rendered prompts for Engine.Run. // LLMClient executes rendered prompts for [Engine.Run].
//
// Generate may be called concurrently. It must honor context cancellation to
// make Run responsive to cancellation. The request and all nested maps,
// slices, and pointers are client-owned copies and may be mutated or retained
// without affecting engine state.
//
// Generate receives rendered messages and may receive a direct API key. A
// client must protect those values and any raw output in its logging, storage,
// and retained copies. It is responsible for the cancellation behavior of any
// work it starts and for synchronizing access to retained or shared data.
//
// A returned error makes Run return ErrLLMGenerate while preserving the client
// error through errors.Is. A nil response with a nil error also produces
// ErrLLMGenerate. Promptkit copies the non-nil response before returning from
// Run.
type LLMClient interface { type LLMClient interface {
Generate(context.Context, GenerateRequest) (*GenerateResponse, error) Generate(context.Context, GenerateRequest) (*GenerateResponse, error)
} }
// GenerateRequest is passed to an injected LLM client. // GenerateRequest is passed to an injected LLM client and has a stable JSON
// representation. Its String and GoString methods omit rendered content and
// direct credentials.
type GenerateRequest struct { type GenerateRequest struct {
Prompt RenderedPrompt `json:"prompt"` // Prompt contains the rendered session ID and messages.
Target ExecutionTarget `json:"target"` Prompt RenderedPrompt `json:"prompt"`
TargetPresence ExecutionTargetPresence `json:"target_presence"` // Target contains effective model settings without the direct API key.
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"` Target ExecutionTarget `json:"target"`
APIKey string `json:"-"` // TargetPresence distinguishes inherited numeric zeros from explicit
// request overrides.
TargetPresence ExecutionTargetPresence `json:"target_presence"`
// StructuredOutput contains provider response constraints when requested.
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
// APIKey is the direct request-scoped credential, if any. It is excluded
// from JSON, String, and GoString output.
APIKey string `json:"-"`
} }
// GenerateResponse is returned by an injected LLM client. // GenerateResponse is returned by an injected LLM client and has a stable JSON
// representation.
type GenerateResponse struct { type GenerateResponse struct {
Content string `json:"content"` // Content is the generated output. It must be non-empty when using the
Usage TokenUsage `json:"usage"` // built-in client; injected clients may return empty content for Promptkit
// validation to classify.
Content string `json:"content"`
// Usage is the client's token accounting.
Usage TokenUsage `json:"usage"`
} }
// File returns a file-backed artifact reference. // File returns a file-backed artifact reference whose URI is path.
//
// The default artifact reader opens path as a caller-selected operating-system
// path without restricting it to an application root or imposing a size limit.
// Applications accepting untrusted paths must validate them before calling
// Promptkit or use [WithArtifactReader] to enforce application policy.
func File(path string) ArtifactRef { func File(path string) ArtifactRef {
return ArtifactRef{Type: ArtifactRefFile, URI: path} return ArtifactRef{Type: ArtifactRefFile, URI: path}
} }
// Inline returns an inline artifact reference. // Inline returns an inline artifact reference whose Body is body and whose URI
// is empty.
func Inline(body string) ArtifactRef { func Inline(body string) ArtifactRef {
return ArtifactRef{Type: ArtifactRefInline, Body: body} return ArtifactRef{Type: ArtifactRefInline, Body: body}
} }
// InlineWithURI returns an inline artifact reference with URI metadata. // InlineWithURI returns an inline artifact reference with body content and uri
// provenance metadata.
func InlineWithURI(uri string, body string) ArtifactRef { func InlineWithURI(uri string, body string) ArtifactRef {
return ArtifactRef{Type: ArtifactRefInline, URI: uri, Body: body} return ArtifactRef{Type: ArtifactRefInline, URI: uri, Body: body}
} }