Compare commits
4 Commits
v0.1.0
...
b462153483
| Author | SHA1 | Date | |
|---|---|---|---|
| b462153483 | |||
| 086cf0fc86 | |||
| c1cecb1ee8 | |||
| bcb327f643 |
@@ -31,4 +31,9 @@ Contributors should start with the [development guide](docs/development.md).
|
|||||||
The [architecture policy](docs/policy/architecture.md) defines the library
|
The [architecture policy](docs/policy/architecture.md) defines the library
|
||||||
boundary and constraints that framework work must preserve.
|
boundary and constraints that framework work must preserve.
|
||||||
|
|
||||||
|
## Related Project
|
||||||
|
|
||||||
|
[Scriptorium](https://gitea.maximumdirect.net/eric/scriptorium) is the CLI and
|
||||||
|
HTTP application built on Promptkit.
|
||||||
|
|
||||||
Promptkit is licensed under the [GNU General Public License version 3](LICENSE).
|
Promptkit is licensed under the [GNU General Public License version 3](LICENSE).
|
||||||
|
|||||||
@@ -20,7 +20,6 @@ func toDomainRunRequest(req RunRequest) (domain.RunRequest, error) {
|
|||||||
Vars: copyStringMap(req.Vars),
|
Vars: copyStringMap(req.Vars),
|
||||||
Execution: execution,
|
Execution: execution,
|
||||||
Validation: toDomainOutputContractPtr(req.Validation),
|
Validation: toDomainOutputContractPtr(req.Validation),
|
||||||
Metadata: copyStringMap(req.Metadata),
|
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
54
doc.go
54
doc.go
@@ -1,8 +1,54 @@
|
|||||||
// Package promptkit provides an embeddable engine for preparing and executing
|
// Package promptkit provides an embeddable engine for preparing and executing
|
||||||
// prompt-defined LLM workflows.
|
// prompt-defined LLM workflows.
|
||||||
//
|
//
|
||||||
// Applications construct an Engine with NewEngine, select filesystem or
|
// Applications construct an [Engine] with [NewEngine], select filesystem or
|
||||||
// in-memory definition sources with options, and use Prepare or Run to execute
|
// in-memory sources with options, and call [Engine.Prepare] or [Engine.Run].
|
||||||
// requests. Concrete repositories, validators, and outbound clients remain
|
// Concrete repositories, validators, and the built-in OpenAI-compatible client
|
||||||
// internal implementation details.
|
// remain internal implementation details.
|
||||||
|
//
|
||||||
|
// # Concurrency and ownership
|
||||||
|
//
|
||||||
|
// An Engine supports concurrent Prepare and Run calls. An injected [LLMClient]
|
||||||
|
// or [ArtifactReader] can therefore receive concurrent calls and must be safe
|
||||||
|
// for that use.
|
||||||
|
//
|
||||||
|
// NewEngine copies in-memory profiles. Prepare and Run copy request maps,
|
||||||
|
// slices, pointer values, and JSON-compatible extra parameters before using
|
||||||
|
// them. Returned values and values passed to extension interfaces are likewise
|
||||||
|
// isolated from engine state. Callers own those copies and may mutate them
|
||||||
|
// after the call that supplied or returned them.
|
||||||
|
//
|
||||||
|
// # Security and sensitive data
|
||||||
|
//
|
||||||
|
// The default artifact reader treats [File] paths as caller-selected operating
|
||||||
|
// system paths. It does not restrict them to an application root or impose an
|
||||||
|
// inbound request-size policy. Promptkit is not an inbound request or
|
||||||
|
// untrusted-input security boundary. Applications must validate and restrict
|
||||||
|
// untrusted input before constructing a request, or install an [ArtifactReader]
|
||||||
|
// that enforces their filesystem, authorization, and size policies.
|
||||||
|
//
|
||||||
|
// Rendered messages, input and output [Artifact] bodies, [RunResult.RawOutput],
|
||||||
|
// and [ValidationResult.Errors] may contain sensitive data. Credential
|
||||||
|
// exclusion and redaction do not sanitize those values. Applications and
|
||||||
|
// injected collaborators are responsible for access control, retention,
|
||||||
|
// logging, and secret handling appropriate to their data.
|
||||||
|
//
|
||||||
|
// # JSON
|
||||||
|
//
|
||||||
|
// Stable JSON representations are provided for [PreparedRun], [RunResult],
|
||||||
|
// [Artifact], [ExecutionTarget], [OutputContract], [ValidationResult],
|
||||||
|
// [TokenUsage], [RenderedPrompt], [RenderedMessage], [CacheControl],
|
||||||
|
// [StructuredOutputSpec], [StructuredOutputJSONSpec], [GenerateRequest],
|
||||||
|
// [GenerateResponse], [ExecutionTargetPresence], and the string value types
|
||||||
|
// used by those values.
|
||||||
|
//
|
||||||
|
// Construction values, including [Config], [RunRequest], [ArtifactRef],
|
||||||
|
// [ExecutionTargetOverride], [Profile], and
|
||||||
|
// [OpenAICompatibleProfileConfig], do not have stable JSON representations.
|
||||||
|
// Direct API keys are nevertheless excluded from JSON for every public value.
|
||||||
|
//
|
||||||
|
// JSON timestamps use time.Time's RFC 3339 encoding and are omitted when zero.
|
||||||
|
// PreparedRun and RunResult durations are encoded as integer milliseconds in
|
||||||
|
// duration_ms and omitted when zero. Run IDs and all exposed hashes are opaque:
|
||||||
|
// their spelling, length, character set, and algorithm are not API contracts.
|
||||||
package promptkit
|
package promptkit
|
||||||
|
|||||||
@@ -1,151 +1,147 @@
|
|||||||
# Package `promptkit`
|
# Package `promptkit`
|
||||||
|
|
||||||
Import path:
|
## Purpose
|
||||||
|
|
||||||
|
This guide helps Go consumers assemble Promptkit and choose the main
|
||||||
|
preparation or execution workflow. The declarations and GoDoc in the
|
||||||
|
[root package](../../doc.go) own exact field, option, serialization,
|
||||||
|
concurrency, ownership, failure, and cancellation semantics. The
|
||||||
|
[framework format reference](../formats.md) owns prompt, profile, and schema
|
||||||
|
file contracts.
|
||||||
|
|
||||||
|
Import the package as:
|
||||||
|
|
||||||
```go
|
```go
|
||||||
import "gitea.maximumdirect.net/eric/promptkit"
|
import "gitea.maximumdirect.net/eric/promptkit"
|
||||||
```
|
```
|
||||||
|
|
||||||
Package `promptkit` is the supported Go contract for in-process prompt
|
The following Go fragments are illustrative and omit surrounding package,
|
||||||
preparation and execution. The declarations and their GoDoc in the
|
import, and error-handling code. Use the maintained examples for complete
|
||||||
[root package](../../doc.go) own the exact API; this guide explains how the
|
programs.
|
||||||
pieces are used together. The [framework format reference](../formats.md) owns
|
|
||||||
prompt, profile, and schema file contracts.
|
|
||||||
|
|
||||||
## Engine Construction And Sources
|
## Construct An Engine
|
||||||
|
|
||||||
Construct an engine with [`NewEngine`, `Config`, and
|
Create an engine with
|
||||||
`Option`](../../engine.go). `PromptDir` is required unless a prompt source
|
[`NewEngine`](../../engine.go). A directory-backed setup supplies a prompt
|
||||||
option is supplied. `ProfileDir` optionally overlays built-in profiles, and an
|
directory and may supply profile and schema directories:
|
||||||
empty `SchemaDir` uses the current directory. `Timeout` is the transport-wide
|
|
||||||
safety cap for the built-in OpenAI-compatible client. An optional `HTTPClient`
|
|
||||||
is cloned; its positive timeout takes precedence.
|
|
||||||
|
|
||||||
Nil options are ignored. Invalid construction, including a nil injected client
|
```go
|
||||||
or artifact reader, returns an error matching `ErrInvalidConfig`.
|
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||||
|
PromptDir: "prompts",
|
||||||
|
ProfileDir: "profiles",
|
||||||
|
SchemaDir: "schemas",
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
The [source options](../../engine.go) replace their matching directory source:
|
Options support single-file or `fs.FS` sources, in-memory profiles, and
|
||||||
|
injected artifact or model clients. Consult the
|
||||||
- `WithPromptFS` and `WithPromptFile` select prompt definitions;
|
[constructor and option GoDoc](../../engine.go) for composition, precedence,
|
||||||
- `WithProfileFS` and `WithProfileFile` overlay built-in profiles;
|
validation, and default transport behavior. Source discovery, format
|
||||||
- `WithProfiles` adds in-memory profiles ahead of file and built-in profiles;
|
validation, and profile precedence are defined by the
|
||||||
- `WithSchemaFS` and `WithSchemaFile` select JSON Schema documents;
|
|
||||||
- `WithLLMClient` replaces the built-in model client; and
|
|
||||||
- `WithArtifactReader` replaces the default reader for every input.
|
|
||||||
|
|
||||||
Source selection, path resolution, strict decoding, profile overlays, and
|
|
||||||
file-to-request precedence are defined in the
|
|
||||||
[framework format reference](../formats.md).
|
[framework format reference](../formats.md).
|
||||||
|
|
||||||
Per-generation timeout values from profiles or requests are independent of
|
## Prepare Without Model Execution
|
||||||
the transport cap and caller context. An explicit request value of zero
|
|
||||||
disables only the per-generation deadline. The
|
|
||||||
[outbound integration contract](../integrations/openai-compatible-chat.md#timeout-and-cancellation)
|
|
||||||
defines the complete timeout layering.
|
|
||||||
|
|
||||||
## Preparation And Execution
|
[`Engine.Prepare`](../../engine.go) resolves the selected prompt and profile,
|
||||||
|
loads inputs and any structured-output schema, and renders messages without
|
||||||
|
calling a model client:
|
||||||
|
|
||||||
[`Engine.Prepare` and `Engine.Run`](../../engine.go) accept the public
|
```go
|
||||||
[`RunRequest`](../../types.go). `Prepare` resolves the prompt, profile, input
|
prepared, err := engine.Prepare(ctx, promptkit.RunRequest{
|
||||||
artifacts, validation contract, and rendered messages without calling an LLM.
|
PromptID: "meeting.summary",
|
||||||
`Run` performs the same preparation, calls the configured client, and validates
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
the generated content. The maintained
|
"note": promptkit.Inline("Synthetic meeting notes"),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
The maintained
|
||||||
[offline preparation example](../../examples/go-library/prepare/main.go)
|
[offline preparation example](../../examples/go-library/prepare/main.go)
|
||||||
provides a complete runnable workflow using a prompt file, in-memory profile,
|
shows a complete runnable setup with a prompt file, in-memory profile, and
|
||||||
and inline input.
|
inline input. Exact request requirements and prepared-result fields belong to
|
||||||
|
the [`RunRequest` and `PreparedRun` GoDoc](../../types.go).
|
||||||
|
|
||||||
[`PreparedRun` and `RunResult`](../../types.go) expose copied public values.
|
## Execute And Validate
|
||||||
Preparation returns effective settings, hashes, rendered messages, selected
|
|
||||||
profile, structured-output information, and timing without resolved secrets or
|
|
||||||
model output. Execution adds the generated artifact and raw output, validation
|
|
||||||
state, model metadata, usage, run ID, and duration.
|
|
||||||
|
|
||||||
A generated-content validation failure returns a result with
|
[`Engine.Run`](../../engine.go) performs the same preparation, invokes the
|
||||||
`Validation.Status == ValidationFailed`. An inability to perform validation
|
configured model client, classifies the generated artifact, and validates the
|
||||||
returns an error matching `ErrValidation`.
|
content. A completed content check may return `ValidationFailed` in the result;
|
||||||
|
an operational inability to validate returns an error.
|
||||||
|
|
||||||
## Requests, Inputs, And Overrides
|
The maintained
|
||||||
|
[offline execution example](../../examples/go-library/run/main.go) injects a
|
||||||
|
deterministic model client and exercises `Run` without credentials, network
|
||||||
|
access, or paid calls. It is intentionally separate from the preparation
|
||||||
|
example so each workflow and its small prompt fixture can be copied and run on
|
||||||
|
its own.
|
||||||
|
|
||||||
The [request and value declarations](../../types.go) own the available fields,
|
Use the [`RunResult` and `ValidationResult` GoDoc](../../types.go) for the
|
||||||
serialized constants, and result shapes. Use `File`, `Inline`, or
|
returned data and the `Engine.Run` GoDoc for failure and cancellation
|
||||||
`InlineWithURI` to construct artifact references. The
|
semantics. The
|
||||||
[framework format reference](../formats.md) defines declared inputs, template
|
[OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md)
|
||||||
references, output contracts, and the relationship between file values and
|
owns the built-in client's outbound HTTP behavior.
|
||||||
request overrides.
|
|
||||||
|
|
||||||
`ExecutionTargetOverride` uses pointers for numeric settings so an explicit
|
## Inputs, Profiles, And Overrides
|
||||||
zero remains distinct from no override. `ExtraParams` accepts JSON-compatible
|
|
||||||
strings, booleans, finite numbers, string-keyed objects, arrays or slices, and
|
|
||||||
nil. Unsupported values, non-string map keys, non-finite numbers, and cycles
|
|
||||||
match `ErrInvalidConfig` in profiles or `ErrInvalidRequest` in request
|
|
||||||
overrides.
|
|
||||||
|
|
||||||
Returned requests, profiles, prepared values, results, artifacts, maps, and
|
Use `File`, `Inline`, or `InlineWithURI` to construct request inputs. A request
|
||||||
slices are isolated from internal engine state. Consumers and injected
|
can select a profile explicitly or use the prompt's default profile, and can
|
||||||
extensions should not retain or mutate values owned by another caller.
|
replace execution settings or the complete output contract.
|
||||||
|
|
||||||
## Profiles And Credentials
|
The [public value GoDoc](../../types.go) defines nil, empty, zero, replacement,
|
||||||
|
copy, and credential behavior. The
|
||||||
|
[framework format reference](../formats.md) defines how those request values
|
||||||
|
interact with prompt definitions, file-backed profiles, built-ins, schemas,
|
||||||
|
and framework defaults.
|
||||||
|
|
||||||
[`OpenAICompatibleProfile`](../../profiles.go) constructs an ordinary
|
For programmatic profiles,
|
||||||
in-memory profile for an OpenAI-compatible chat-completions endpoint.
|
[`OpenAICompatibleProfile`](../../profiles.go) converts ordinary
|
||||||
`WithProfiles` rejects duplicate IDs in one call and gives in-memory profiles
|
OpenAI-compatible settings into a value accepted by `WithProfiles`.
|
||||||
precedence over explicit file sources and built-ins.
|
|
||||||
|
|
||||||
Raw API keys do not belong in profiles. File-backed profiles may name an
|
## Credentials
|
||||||
environment variable, while an in-memory profile can require a request key.
|
|
||||||
A direct `RunRequest.APIKey` is request-scoped and takes precedence over an
|
|
||||||
environment lookup for the built-in client. Profile fields, ranges, built-ins,
|
|
||||||
precedence, and credential rules are owned by the
|
|
||||||
[framework format reference](../formats.md).
|
|
||||||
|
|
||||||
API keys are excluded from JSON, prepared values, and results. The public
|
File-backed profiles name an environment variable; in-memory profiles can
|
||||||
`String` and `GoString` methods report only whether a direct key is present.
|
require a direct request key. Direct keys are request-scoped and are excluded
|
||||||
Avoid reflection-based dumps of request structs, which can bypass that
|
from supported JSON values and the package's `String` and `GoString`
|
||||||
redaction.
|
summaries. The exact precedence and redaction guarantees belong to
|
||||||
|
[`RunRequest`, `GenerateRequest`, and the profile GoDoc](../../types.go).
|
||||||
|
|
||||||
|
## Protect Files And Generated Data
|
||||||
|
|
||||||
|
The default artifact reader opens a `File` reference as a caller-selected
|
||||||
|
operating-system path. It does not constrain paths to an application root,
|
||||||
|
impose an inbound request-size policy, or establish an untrusted-input security
|
||||||
|
boundary. Applications must validate and restrict untrusted paths and payloads
|
||||||
|
before constructing a request, or inject an artifact reader that enforces
|
||||||
|
their filesystem, authorization, and size policies.
|
||||||
|
|
||||||
|
Rendered messages, input and output artifact bodies, raw model output, and
|
||||||
|
validation diagnostics can contain sensitive data. API-key redaction does not
|
||||||
|
sanitize those values. Treat prepared values, results, collaborator requests,
|
||||||
|
errors, and logs according to the application's data-access, retention, and
|
||||||
|
secret-handling policies.
|
||||||
|
|
||||||
## Extension Interfaces
|
## Extension Interfaces
|
||||||
|
|
||||||
The [`LLMClient`, `GenerateRequest`, and
|
Inject an [`LLMClient` or `ArtifactReader`](../../types.go) when the built-in
|
||||||
`GenerateResponse`](../../types.go) boundary lets a consumer replace model
|
behavior does not fit the application. Their GoDoc defines concurrent use,
|
||||||
generation. Injected clients receive copied rendered messages, effective
|
context handling, ownership of copied values, nil responses, and preservation
|
||||||
settings, explicit numeric-setting presence, structured-output constraints,
|
of collaborator errors. Implementations must honor cancellation, safely manage
|
||||||
and the request-scoped key. They return generated content and token usage.
|
copies they retain, avoid unsafe logging of content or credentials, and enforce
|
||||||
|
the application policy that motivated the injection.
|
||||||
|
|
||||||
The [`ArtifactReader`](../../types.go) boundary replaces the default inline and
|
## Handle Errors
|
||||||
file reader for every input. Readers provide artifact content and metadata; the
|
|
||||||
engine fills an empty artifact name from the input-map key. A reader error
|
|
||||||
matches `ErrArtifactLoad` while preserving the original identity for
|
|
||||||
`errors.Is`. A nil artifact with a nil error is also an artifact-load failure.
|
|
||||||
|
|
||||||
Extensions should honor context cancellation and avoid logging raw prompts,
|
Use `errors.Is` with the
|
||||||
artifacts, or credentials.
|
[public error sentinels and operation GoDoc](../../engine.go). The declarations
|
||||||
|
distinguish invalid construction, invalid requests, absent sources,
|
||||||
|
source-loading failures, collaborator failures, and operational validation
|
||||||
|
failures. Specific request conditions may also match the broader
|
||||||
|
`ErrInvalidRequest`, and injected collaborator identities are preserved where
|
||||||
|
documented.
|
||||||
|
|
||||||
## Errors
|
## Application Boundary
|
||||||
|
|
||||||
The [public error declarations](../../engine.go) and
|
|
||||||
[mapping](../../errors.go) preserve these sentinel checks through `errors.Is`:
|
|
||||||
|
|
||||||
- `ErrInvalidConfig`
|
|
||||||
- `ErrInvalidRequest`
|
|
||||||
- `ErrPromptNotFound`
|
|
||||||
- `ErrProfileNotFound`
|
|
||||||
- `ErrProfileRequired`
|
|
||||||
- `ErrPromptLoad`
|
|
||||||
- `ErrProfileLoad`
|
|
||||||
- `ErrAPIKeyEnvMissing`
|
|
||||||
- `ErrArtifactLoad`
|
|
||||||
- `ErrPromptRender`
|
|
||||||
- `ErrLLMGenerate`
|
|
||||||
- `ErrValidation`
|
|
||||||
|
|
||||||
`ErrProfileRequired` and `ErrAPIKeyEnvMissing` also match
|
|
||||||
`ErrInvalidRequest`, allowing either broad request handling or a specific
|
|
||||||
condition. Wrapped collaborator errors retain their identity where the public
|
|
||||||
contract promises it.
|
|
||||||
|
|
||||||
## Consumer Boundary
|
|
||||||
|
|
||||||
Promptkit is an importable library. It does not own a command, inbound HTTP
|
Promptkit is an importable library. It does not own a command, inbound HTTP
|
||||||
API, process configuration, or deployment policy. Scriptorium is one
|
API, process configuration, or deployment policy. Applications map the root
|
||||||
downstream application that maps this root package contract into those
|
package's results and errors into those concerns, including inbound size and
|
||||||
application concerns.
|
trust policy.
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ contributor workflow and validation.
|
|||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| Root `promptkit` package | Provides the supported engine facade, source and injection options, public request and result values, built-in profile construction, extension interfaces, value conversion, redacted formatting, and public error mapping. | [Package GoDoc](../../doc.go), [engine assembly](../../engine.go) |
|
| Root `promptkit` package | Provides the supported engine facade, source and injection options, public request and result values, built-in profile construction, extension interfaces, value conversion, redacted formatting, and public error mapping. | [Package GoDoc](../../doc.go), [engine assembly](../../engine.go) |
|
||||||
| `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
|
| `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
|
||||||
|
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
|
||||||
| `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, profiles, execution targets, rendering, generation, and validation. | [Domain declarations](../../internal/domain/domain.go) |
|
| `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, profiles, execution targets, rendering, generation, and validation. | [Domain declarations](../../internal/domain/domain.go) |
|
||||||
| `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) |
|
| `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) |
|
||||||
| `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |
|
| `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |
|
||||||
|
|||||||
@@ -41,9 +41,9 @@ The implemented internal components consist of:
|
|||||||
- `internal/usecase`, which coordinates preparation and execution across the
|
- `internal/usecase`, which coordinates preparation and execution across the
|
||||||
internal framework components.
|
internal framework components.
|
||||||
|
|
||||||
The `examples/go-library/prepare` package is a maintained downstream consumer
|
The `examples/go-library/prepare` and `examples/go-library/run` packages are
|
||||||
of the root facade. It does not expose a library package or participate in
|
maintained downstream consumers of the root facade. They do not expose library
|
||||||
internal assembly.
|
packages or participate in internal assembly.
|
||||||
|
|
||||||
The root facade assembles the internal repositories, renderer, validator,
|
The root facade assembles the internal repositories, renderer, validator,
|
||||||
outbound client, and use-case runner while translating public values and
|
outbound client, and use-case runner while translating public values and
|
||||||
|
|||||||
254
docs/release.md
254
docs/release.md
@@ -7,27 +7,91 @@ tags. It does not publish runnable binaries or binary packages and does not
|
|||||||
currently use hosted CI. The release maintainer performs and records the
|
currently use hosted CI. The release maintainer performs and records the
|
||||||
required validation.
|
required validation.
|
||||||
|
|
||||||
The first planned release is `v0.1.0`. Do not create that tag until the
|
`v0.1.0` is the initial published release. Later releases use semantic
|
||||||
framework has been extracted and the resulting public library has passed this
|
`vMAJOR.MINOR.PATCH` tags. Before `v1`, minor releases may change the public
|
||||||
procedure. Later tags use the `vMAJOR.MINOR.PATCH` form. While Promptkit remains
|
API and patch releases preserve compatibility within their minor line. Every
|
||||||
pre-`v1`, release notes must identify intentional public API changes and any
|
pre-`v1` release note must summarize compatibility, identify public API
|
||||||
consumer migration required by them.
|
changes, and state any action required of consumers.
|
||||||
|
|
||||||
## Prepare The Release
|
Promptkit releases are source-only. The annotated tag message is the release
|
||||||
|
note; there is no separate hosted release or binary packaging step.
|
||||||
|
|
||||||
Work from a clean checkout of the intended release commit, outside any Go
|
## Establish The Candidate
|
||||||
workspace and without a local module replacement. Confirm the source commit is
|
|
||||||
already published through the normal branch workflow.
|
|
||||||
|
|
||||||
From the Promptkit repository root, verify the checkout:
|
Choose a version that has not been published and export it as
|
||||||
|
`RELEASE_VERSION`. Run every command in this procedure from the Promptkit
|
||||||
|
repository root in the same POSIX shell. Do not reuse `v0.1.0` or another
|
||||||
|
existing version.
|
||||||
|
|
||||||
|
The following guard derives the release commit from `HEAD` and stops on a
|
||||||
|
missing or malformed version, a checkout other than synchronized `main`,
|
||||||
|
uncommitted changes, an active Go workspace, a module replacement, a vendor
|
||||||
|
tree, or an existing local or remote tag:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
gowork=$(go env GOWORK)
|
set -eu
|
||||||
test -z "$gowork" || test "$gowork" = off
|
|
||||||
test -z "$(git status --short)"
|
: "${RELEASE_VERSION:?export an unpublished vMAJOR.MINOR.PATCH version}"
|
||||||
git fetch --tags origin
|
if ! printf '%s\n' "$RELEASE_VERSION" |
|
||||||
|
grep -Eq '^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$'
|
||||||
|
then
|
||||||
|
printf '%s\n' "invalid release version: $RELEASE_VERSION" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
RELEASE_COMMIT=$(git rev-parse --verify 'HEAD^{commit}')
|
||||||
|
export RELEASE_COMMIT
|
||||||
|
|
||||||
|
check_release_candidate() {
|
||||||
|
test "$(git branch --show-current)" = main
|
||||||
|
test -z "$(git status --porcelain)"
|
||||||
|
|
||||||
|
gowork_value=$(go env GOWORK)
|
||||||
|
case "$gowork_value" in
|
||||||
|
''|off) ;;
|
||||||
|
*)
|
||||||
|
printf '%s\n' "active Go workspace: $gowork_value" >&2
|
||||||
|
return 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
test -z "$(git ls-files go.work go.work.sum)"
|
||||||
|
test ! -e vendor
|
||||||
|
if grep -Eq '^[[:space:]]*replace([[:space:]]|\()' go.mod
|
||||||
|
then
|
||||||
|
printf '%s\n' 'go.mod contains a replacement' >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
git fetch origin main --tags
|
||||||
|
test "$RELEASE_COMMIT" = \
|
||||||
|
"$(git rev-parse --verify 'refs/remotes/origin/main^{commit}')"
|
||||||
|
|
||||||
|
if git show-ref --verify --quiet "refs/tags/$RELEASE_VERSION"
|
||||||
|
then
|
||||||
|
printf '%s\n' "local tag already exists: $RELEASE_VERSION" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
if test -n "$(
|
||||||
|
git ls-remote --tags origin \
|
||||||
|
"refs/tags/$RELEASE_VERSION" \
|
||||||
|
"refs/tags/$RELEASE_VERSION^{}"
|
||||||
|
)"
|
||||||
|
then
|
||||||
|
printf '%s\n' "remote tag already exists: $RELEASE_VERSION" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
check_release_candidate
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Do not continue unless the guard completes successfully. In particular, push
|
||||||
|
the intended commit through the normal `main` branch workflow before release;
|
||||||
|
the tag procedure is not a substitute for publishing the source commit.
|
||||||
|
|
||||||
|
## Validate The Candidate
|
||||||
|
|
||||||
Confirm the module and root package metadata:
|
Confirm the module and root package metadata:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
@@ -42,7 +106,7 @@ gitea.maximumdirect.net/eric/promptkit 1.25.5
|
|||||||
promptkit gitea.maximumdirect.net/eric/promptkit
|
promptkit gitea.maximumdirect.net/eric/promptkit
|
||||||
```
|
```
|
||||||
|
|
||||||
Run the same default Go validation required by the
|
Run the complete maintainer validation required by the
|
||||||
[development guide](development.md):
|
[development guide](development.md):
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
@@ -53,84 +117,158 @@ go build ./...
|
|||||||
go run ./examples/go-library/prepare
|
go run ./examples/go-library/prepare
|
||||||
```
|
```
|
||||||
|
|
||||||
Check every tracked Go file and repository whitespace:
|
Check every tracked Go file. This command must produce no output:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
gofmt -l $(git ls-files '*.go')
|
unformatted=$(
|
||||||
|
git ls-files '*.go' |
|
||||||
|
while IFS= read -r go_file
|
||||||
|
do
|
||||||
|
gofmt -l "$go_file"
|
||||||
|
done
|
||||||
|
)
|
||||||
|
test -z "$unformatted"
|
||||||
|
```
|
||||||
|
|
||||||
|
Follow every maintained Markdown link and confirm that its local or published
|
||||||
|
target exists. Review the repository for generated binaries, test or coverage
|
||||||
|
output, credentials, template residue, downloaded assets, and other files that
|
||||||
|
do not belong in source control.
|
||||||
|
|
||||||
|
Recheck module and repository hygiene, whitespace, and the clean checkout:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
test -z "$(git ls-files go.work go.work.sum)"
|
||||||
|
test ! -e vendor
|
||||||
|
if grep -Eq '^[[:space:]]*replace([[:space:]]|\()' go.mod
|
||||||
|
then
|
||||||
|
printf '%s\n' 'go.mod contains a replacement' >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
git diff --check
|
git diff --check
|
||||||
|
test -z "$(git status --porcelain)"
|
||||||
```
|
```
|
||||||
|
|
||||||
The formatting command must produce no paths. Follow every maintained Markdown
|
## Write The Release Note
|
||||||
link and confirm its target exists. Review the repository for generated
|
|
||||||
binaries, test or coverage output, credentials, template residue, and other
|
|
||||||
files that do not belong in source control.
|
|
||||||
|
|
||||||
Confirm that no workspace override is tracked and that `go.mod` contains no
|
Prepare a plain-text annotated-tag message outside the repository and export
|
||||||
`replace` directive:
|
its path as `RELEASE_NOTES_FILE`. Use this form, replacing each summary with
|
||||||
|
release-specific text; write `None.` when there are no public API changes or
|
||||||
|
consumer actions:
|
||||||
|
|
||||||
```sh
|
```text
|
||||||
git ls-files go.work go.work.sum
|
Promptkit vMAJOR.MINOR.PATCH
|
||||||
rg -n '^replace\b' go.mod
|
|
||||||
|
Validated commit: full commit ID
|
||||||
|
Compatibility: compatibility summary
|
||||||
|
Public API changes: changes or None.
|
||||||
|
Consumer action: required action or None.
|
||||||
```
|
```
|
||||||
|
|
||||||
Both commands must produce no output. Re-run `git status --short` and require a
|
After writing it, require all release-note fields, the selected version, and
|
||||||
clean result after every validation and review check.
|
the validated commit to be present:
|
||||||
|
|
||||||
## Create And Publish The Tag
|
|
||||||
|
|
||||||
Choose the semantic version from the intended compatibility change. Record the
|
|
||||||
release commit before tagging:
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
release_version=v0.1.0
|
: "${RELEASE_NOTES_FILE:?export the path to the release-note file}"
|
||||||
release_commit=$(git rev-parse HEAD)
|
test -f "$RELEASE_NOTES_FILE"
|
||||||
|
test -s "$RELEASE_NOTES_FILE"
|
||||||
|
grep -F "Promptkit $RELEASE_VERSION" "$RELEASE_NOTES_FILE"
|
||||||
|
grep -F "Validated commit: $RELEASE_COMMIT" "$RELEASE_NOTES_FILE"
|
||||||
|
grep -F 'Compatibility:' "$RELEASE_NOTES_FILE"
|
||||||
|
grep -F 'Public API changes:' "$RELEASE_NOTES_FILE"
|
||||||
|
grep -F 'Consumer action:' "$RELEASE_NOTES_FILE"
|
||||||
```
|
```
|
||||||
|
|
||||||
Replace the example version for later releases and keep both values in the same
|
Inspect the complete message and confirm that it accurately records the
|
||||||
shell for the remaining commands. Confirm the tag does not already exist
|
compatibility impact, public API changes, and required consumer action.
|
||||||
locally or remotely:
|
|
||||||
|
## Create And Inspect The Tag
|
||||||
|
|
||||||
|
Run the candidate guard again immediately before tag creation. This ensures
|
||||||
|
that validation or release-note preparation did not change the checkout and
|
||||||
|
that the commit is still published and untagged:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
test -z "$(git tag --list "$release_version")"
|
check_release_candidate
|
||||||
test -z "$(git ls-remote --tags origin "refs/tags/$release_version")"
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Create an annotated tag whose message identifies the release and records that
|
Create the annotated tag from the prepared release note and bind it explicitly
|
||||||
the documented validation passed for the tagged commit:
|
to the validated commit:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
git tag --annotate "$release_version" \
|
git tag --annotate "$RELEASE_VERSION" \
|
||||||
--message "Promptkit $release_version; documented validation passed for $release_commit"
|
--file "$RELEASE_NOTES_FILE" \
|
||||||
|
"$RELEASE_COMMIT"
|
||||||
```
|
```
|
||||||
|
|
||||||
Inspect the tag before publication:
|
Inspect both the tag message and its source commit before publication:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
git show --no-patch --decorate "$release_version"
|
test "$(git cat-file -t "refs/tags/$RELEASE_VERSION")" = tag
|
||||||
test "$(git rev-list -n 1 "$release_version")" = "$release_commit"
|
git show --no-patch --decorate "refs/tags/$RELEASE_VERSION"
|
||||||
|
test "$(
|
||||||
|
git rev-parse --verify "refs/tags/$RELEASE_VERSION^{commit}"
|
||||||
|
)" = "$RELEASE_COMMIT"
|
||||||
```
|
```
|
||||||
|
|
||||||
Publish the tag without relying on a hosting-provider-specific release
|
If inspection finds an error, delete the unpublished local tag, correct the
|
||||||
interface:
|
release note or candidate, and repeat the guards. Never move or recreate a tag
|
||||||
|
that has been published.
|
||||||
|
|
||||||
|
## Publish The Selected Tag
|
||||||
|
|
||||||
|
Push only the selected tag ref. Do not use `git push --tags`:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
git push origin "refs/tags/$release_version"
|
git push origin \
|
||||||
|
"refs/tags/$RELEASE_VERSION:refs/tags/$RELEASE_VERSION"
|
||||||
```
|
```
|
||||||
|
|
||||||
## Verify Publication
|
## Verify Publication
|
||||||
|
|
||||||
Confirm that the remote tag object matches the local annotated tag and still
|
Compare the remote annotated-tag object with the local object, then compare the
|
||||||
resolves to the intended source commit:
|
remote peeled source commit with the validated commit:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
remote_tag=$(git ls-remote --tags origin "refs/tags/$release_version" | awk '{print $1}')
|
remote_tag=$(
|
||||||
test "$remote_tag" = "$(git rev-parse "refs/tags/$release_version")"
|
git ls-remote --tags origin "refs/tags/$RELEASE_VERSION" |
|
||||||
test "$(git rev-list -n 1 "refs/tags/$release_version")" = "$release_commit"
|
awk 'NR == 1 { print $1 }'
|
||||||
|
)
|
||||||
|
remote_commit=$(
|
||||||
|
git ls-remote --tags origin "refs/tags/$RELEASE_VERSION^{}" |
|
||||||
|
awk 'NR == 1 { print $1 }'
|
||||||
|
)
|
||||||
|
test -n "$remote_tag"
|
||||||
|
test "$remote_tag" = \
|
||||||
|
"$(git rev-parse --verify "refs/tags/$RELEASE_VERSION")"
|
||||||
|
test "$remote_commit" = "$RELEASE_COMMIT"
|
||||||
```
|
```
|
||||||
|
|
||||||
Promptkit must publish the required tag before Scriptorium or another consumer
|
Finally, resolve the version as an ordinary Go module in a temporary module
|
||||||
publishes a release that depends on that version. Released consumer modules
|
outside this repository and without a workspace or replacement:
|
||||||
must not use a local replacement or unpublished Promptkit revision.
|
|
||||||
|
```sh
|
||||||
|
resolution_dir=$(mktemp -d)
|
||||||
|
(
|
||||||
|
trap 'rm -rf "$resolution_dir"' 0 1 2 15
|
||||||
|
cd "$resolution_dir"
|
||||||
|
GOWORK=off go mod init example.com/promptkit-release-check
|
||||||
|
GOWORK=off go mod download \
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit@$RELEASE_VERSION"
|
||||||
|
resolved_version=$(
|
||||||
|
GOWORK=off go list -m -f '{{.Version}}' \
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit@$RELEASE_VERSION"
|
||||||
|
)
|
||||||
|
test "$resolved_version" = "$RELEASE_VERSION"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Promptkit must publish and verify the required version before Scriptorium or
|
||||||
|
another consumer publishes a release that depends on it. This ordering does
|
||||||
|
not replace the consumer project's own release procedure. Released consumers
|
||||||
|
must select the published Promptkit tag through ordinary module resolution,
|
||||||
|
without a workspace, replacement, vendored Promptkit source, or unpublished
|
||||||
|
revision.
|
||||||
|
|
||||||
## Policy Changes
|
## Policy Changes
|
||||||
|
|
||||||
|
|||||||
222
docs/roadmap/documentation.md
Normal file
222
docs/roadmap/documentation.md
Normal file
@@ -0,0 +1,222 @@
|
|||||||
|
# Documentation Hardening Roadmap
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This roadmap coordinates a focused pass over Promptkit's documentation and
|
||||||
|
public contract. The work should close the gaps identified by the documentation
|
||||||
|
audit, strengthen canonical ownership, and leave consumers and contributors
|
||||||
|
with guidance that is accurate, navigable, and proportionate to their needs.
|
||||||
|
|
||||||
|
This is planning material, not a description of implemented behavior. Follow
|
||||||
|
the [documentation policy](../policy/documentation.md) throughout the work and
|
||||||
|
update current-state documents only when their claims are supported by the
|
||||||
|
implementation and tests.
|
||||||
|
|
||||||
|
## Scope And Principles
|
||||||
|
|
||||||
|
The work covers public GoDoc, consumer guidance, maintained examples, format
|
||||||
|
and integration references, architecture ownership, roadmap lifecycle, and
|
||||||
|
documentation validation.
|
||||||
|
|
||||||
|
- Resolve ambiguous behavior before documenting it as a contract.
|
||||||
|
- Keep exact exported API semantics in Go declarations and GoDoc.
|
||||||
|
- Keep task-oriented guidance in consumer documents and complete runnable
|
||||||
|
artifacts in `examples/`.
|
||||||
|
- Put security-relevant consumer responsibilities at the public boundary, not
|
||||||
|
only in internal contributor documents.
|
||||||
|
- Remove duplicated ownership instead of synchronizing parallel references.
|
||||||
|
- Make documentation checks reproducible where practical.
|
||||||
|
- Preserve existing behavior unless a stage explicitly selects and tests an
|
||||||
|
API change.
|
||||||
|
|
||||||
|
The roadmap does not add new Promptkit features, redesign framework formats,
|
||||||
|
or implement ideas from [the future feature catalog](future.md). If resolving
|
||||||
|
an ambiguity requires a behavioral change, treat that change as a separately
|
||||||
|
reviewable implementation unit and update its canonical documentation in the
|
||||||
|
same unit.
|
||||||
|
|
||||||
|
## Stage 1: Resolve Public Contract Questions
|
||||||
|
|
||||||
|
Before expanding prose, decide the intended contract for exported behavior
|
||||||
|
that is currently ambiguous.
|
||||||
|
|
||||||
|
- [x] Decide whether `RunRequest.Metadata` has a supported observable purpose.
|
||||||
|
Define its propagation and ownership, or remove or deprecate it through an
|
||||||
|
intentional public API change.
|
||||||
|
- [x] Decide and test whether one `Engine` supports concurrent `Prepare` and
|
||||||
|
`Run` calls.
|
||||||
|
- [x] Decide how repeated options of the same category behave, including
|
||||||
|
prompt, profile, schema, model-client, and artifact-reader options.
|
||||||
|
- [x] Define which public values have supported JSON representations.
|
||||||
|
- [x] Define JSON time units and omission behavior, including the relationship
|
||||||
|
between prepared-run and run-result durations.
|
||||||
|
- [x] Decide whether run IDs and exposed hashes have stable formats or must be
|
||||||
|
treated as opaque values.
|
||||||
|
- [x] Confirm the intended transport-timeout default and its zero or negative
|
||||||
|
configuration semantics.
|
||||||
|
- [x] Confirm the supported JSON Schema dialect and reference boundaries,
|
||||||
|
including whether remote references are allowed.
|
||||||
|
|
||||||
|
The selected exported API contracts are implemented and tested. Their durable
|
||||||
|
definitions now belong to the root package declarations and GoDoc.
|
||||||
|
|
||||||
|
One format-level decision remains here until Stage 5 moves it to the framework
|
||||||
|
format reference: JSON Schema uses Draft 2020-12, with that dialect selected
|
||||||
|
when `$schema` is omitted. Same-document fragments and relative references
|
||||||
|
contained by a directory or `fs.FS` schema root are supported. A single-file
|
||||||
|
source supports only references contained in that document. Absolute,
|
||||||
|
escaping, and remote references are rejected.
|
||||||
|
|
||||||
|
**Gate:** Each question has an explicit answer backed by existing behavior or
|
||||||
|
by an accepted implementation change and proportionate tests. No later stage
|
||||||
|
should invent a contract merely to fill a documentation gap.
|
||||||
|
|
||||||
|
## Stage 2: Make GoDoc The Canonical Public Contract
|
||||||
|
|
||||||
|
Strengthen the root package declarations so `go doc` is sufficient to
|
||||||
|
understand exact public behavior without relying on internal documents.
|
||||||
|
|
||||||
|
- [x] Add useful field-level GoDoc to configuration, request, profile,
|
||||||
|
execution-target, result, artifact, validation, structured-output, and model
|
||||||
|
client values.
|
||||||
|
- [x] Document required fields and nil, empty, and zero-value semantics.
|
||||||
|
- [x] Document override, replacement, profile-precedence, and copy-ownership
|
||||||
|
behavior where it belongs to the exported API.
|
||||||
|
- [x] Document credential inputs, redaction, and the values intentionally
|
||||||
|
excluded from serialization.
|
||||||
|
- [x] Give each public error sentinel an accurate comment and document the
|
||||||
|
supported `errors.Is` relationships.
|
||||||
|
- [x] Document engine concurrency and option-composition behavior selected in
|
||||||
|
Stage 1.
|
||||||
|
- [x] Document serialization, time, run-ID, and hash semantics selected in
|
||||||
|
Stage 1.
|
||||||
|
- [x] Review constructor, option, extension-interface, `Prepare`, and `Run`
|
||||||
|
GoDoc for complete failure and cancellation expectations.
|
||||||
|
|
||||||
|
Update the [consumer guide](../consumers/pkg-promptkit.md) to summarize and link
|
||||||
|
to these contracts instead of maintaining exhaustive copies of exported names
|
||||||
|
or exact semantics.
|
||||||
|
|
||||||
|
**Gate:** `go doc -all .` presents a coherent public contract, exported
|
||||||
|
declarations have accurate comments, and contract tests protect every newly
|
||||||
|
documented behavior whose compatibility risk warrants durable coverage.
|
||||||
|
|
||||||
|
## Stage 3: Improve Consumer Safety And Executable Guidance
|
||||||
|
|
||||||
|
Move consumer-relevant security boundaries to the places where consumers will
|
||||||
|
encounter them and add one representative execution workflow.
|
||||||
|
|
||||||
|
- [x] Explain in public GoDoc and the consumer guide that the default file
|
||||||
|
artifact reader accepts unrestricted caller-selected paths.
|
||||||
|
- [x] Make clear that Promptkit does not impose an application root, inbound
|
||||||
|
request-size policy, or untrusted-input security boundary.
|
||||||
|
- [x] Explain that rendered messages, artifact bodies, raw model output, and
|
||||||
|
validation details may be sensitive even when credentials are redacted.
|
||||||
|
- [x] Clarify the responsibilities of injected artifact readers and model
|
||||||
|
clients for cancellation, copying, logging, and secret handling.
|
||||||
|
- [x] Add a maintained offline `Run` example using an injected deterministic
|
||||||
|
model client, without credentials, live network access, or paid calls.
|
||||||
|
- [x] Link the consumer guide to the execution example and keep embedded
|
||||||
|
snippets smaller than the maintained artifact.
|
||||||
|
- [x] Decide whether the existing preparation example should remain separate
|
||||||
|
or share reusable fixtures without obscuring either workflow.
|
||||||
|
|
||||||
|
The preparation and execution examples remain separate, self-contained
|
||||||
|
workflows. Each keeps its own small prompt fixture so consumers can copy or run
|
||||||
|
one example without depending on the other.
|
||||||
|
|
||||||
|
**Gate:** Both preparation and execution have complete, secret-free,
|
||||||
|
deterministic consumer examples, and the consumer guide exposes the important
|
||||||
|
filesystem and data-sensitivity boundaries without leaking internal mechanics.
|
||||||
|
|
||||||
|
## Stage 4: Restore Canonical Ownership
|
||||||
|
|
||||||
|
Remove parallel definitions and make navigation follow the ownership model in
|
||||||
|
the documentation policy.
|
||||||
|
|
||||||
|
- [ ] Reduce the [architecture policy](../policy/architecture.md) to durable
|
||||||
|
boundaries, layers, dependency direction, invariants, and non-goals.
|
||||||
|
- [ ] Keep the exact implemented package and component inventory solely in the
|
||||||
|
[internal component overview](../internal/overview.md).
|
||||||
|
- [ ] Review the consumer guide's public error and option lists so they remain
|
||||||
|
task-oriented summaries rather than duplicate API references.
|
||||||
|
- [ ] Review internal documents for public-contract statements that should be
|
||||||
|
links to GoDoc or the format and integration owners.
|
||||||
|
- [ ] Reconcile the documentation policy's temporary-roadmap lifecycle with
|
||||||
|
the continuing idea-catalog role of `docs/roadmap/future.md`.
|
||||||
|
- [ ] Rephrase or link roadmap statements that depend on exact current API
|
||||||
|
behavior, particularly the runtime reasoning entry.
|
||||||
|
- [ ] Confirm that every document states its audience or purpose and links to
|
||||||
|
the canonical owner of adjacent topics.
|
||||||
|
|
||||||
|
**Gate:** Every authoritative fact has one clear owner, package inventory
|
||||||
|
changes no longer require edits to the architecture policy, and roadmaps
|
||||||
|
cannot be mistaken for current-state references.
|
||||||
|
|
||||||
|
## Stage 5: Refine Format And Integration References
|
||||||
|
|
||||||
|
Close compatibility gaps in the documents that own file formats and outbound
|
||||||
|
wire behavior.
|
||||||
|
|
||||||
|
- [ ] State or canonically link the exact session-ID limit enforced by the
|
||||||
|
OpenAI-compatible client.
|
||||||
|
- [ ] State the configured and default transport-timeout behavior without
|
||||||
|
referring to an unnamed internal default.
|
||||||
|
- [ ] Document the JSON Schema dialect and local, contained, and remote
|
||||||
|
reference behavior selected in Stage 1.
|
||||||
|
- [ ] Clarify structured-output naming and strictness when those values are
|
||||||
|
part of the public or integration contract.
|
||||||
|
- [ ] Add a caveat that built-in profiles are maintained configurations, not a
|
||||||
|
guarantee of continuing third-party model availability.
|
||||||
|
- [ ] Recheck every prompt, profile, schema, credential, request-body, response,
|
||||||
|
timeout, and precedence statement against its owning implementation and
|
||||||
|
tests.
|
||||||
|
|
||||||
|
**Gate:** A consumer can determine the supported file and wire compatibility
|
||||||
|
boundaries without consulting internal source code or relying on unspecified
|
||||||
|
defaults.
|
||||||
|
|
||||||
|
## Stage 6: Make Documentation Validation Reproducible
|
||||||
|
|
||||||
|
Align contributor and release procedures around a small, consistent set of
|
||||||
|
checks.
|
||||||
|
|
||||||
|
- [ ] Use one robust command for checking every tracked Go file with `gofmt`.
|
||||||
|
- [ ] Provide a repository-local or clearly documented command that validates
|
||||||
|
local Markdown targets and heading fragments.
|
||||||
|
- [ ] Decide how published external links are checked without making ordinary
|
||||||
|
validation depend on mutable network services.
|
||||||
|
- [ ] Reconcile the validation descriptions in the
|
||||||
|
[development guide](../development.md),
|
||||||
|
[testing policy](../policy/testing.md), and
|
||||||
|
[release procedure](../release.md) so one document owns each requirement.
|
||||||
|
- [ ] Ensure example validation covers every maintained example added by this
|
||||||
|
roadmap.
|
||||||
|
- [ ] Keep documentation-only validation proportionate while requiring full Go
|
||||||
|
validation when commands, examples, generated output, or checked behavior
|
||||||
|
changes.
|
||||||
|
|
||||||
|
**Gate:** A maintainer can run the documented formatting, link, example, Go,
|
||||||
|
and repository-hygiene checks exactly as written, with no hidden manual
|
||||||
|
procedure for local documentation.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
The roadmap is complete when:
|
||||||
|
|
||||||
|
- all Stage 1 contract questions are resolved;
|
||||||
|
- GoDoc is the authoritative and sufficient exported API reference;
|
||||||
|
- consumer guidance covers unrestricted file access and sensitive generated
|
||||||
|
data;
|
||||||
|
- maintained offline examples cover both `Prepare` and `Run`;
|
||||||
|
- architecture, inventory, consumer, internal, format, integration, and
|
||||||
|
roadmap documents follow their assigned ownership boundaries;
|
||||||
|
- schema, session, timeout, structured-output, and built-in-profile
|
||||||
|
compatibility statements are explicit;
|
||||||
|
- documentation validation is reproducible and consistent across contributor
|
||||||
|
and release workflows; and
|
||||||
|
- the complete maintainer validation passes.
|
||||||
|
|
||||||
|
After completion, move any durable decisions to GoDoc, policy, format,
|
||||||
|
integration, or ADR owners as appropriate. Remove this roadmap after incoming
|
||||||
|
links are updated; do not retain it as a second current-state reference.
|
||||||
122
docs/roadmap/future.md
Normal file
122
docs/roadmap/future.md
Normal file
@@ -0,0 +1,122 @@
|
|||||||
|
# Future Feature Ideas
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document catalogs reasonably specific ideas that may be useful in future
|
||||||
|
Promptkit development. It is an idea pool, not a commitment, schedule, or
|
||||||
|
description of current behavior.
|
||||||
|
|
||||||
|
Ideas belong here while they are worth retaining but have not been selected
|
||||||
|
for active development. Keep each entry at the level of intended capability,
|
||||||
|
consumer value, and important scope boundaries. Defer API design,
|
||||||
|
implementation details, sequencing, and acceptance criteria until an idea is
|
||||||
|
selected.
|
||||||
|
|
||||||
|
## Using This Catalog
|
||||||
|
|
||||||
|
- Add an idea when its purpose and likely value can be stated clearly.
|
||||||
|
- Keep entries independent enough that maintainers can evaluate and select
|
||||||
|
them individually.
|
||||||
|
- Note significant dependencies or boundary concerns, but do not turn entries
|
||||||
|
into implementation plans.
|
||||||
|
- Treat inclusion as an invitation to evaluate, not as approval or priority.
|
||||||
|
- When an idea is selected, move its active planning to a focused roadmap or,
|
||||||
|
when it requires a durable architectural decision, an ADR. Update
|
||||||
|
current-state documentation only when implementation lands.
|
||||||
|
- Remove ideas that are no longer relevant. Retain a rejected idea only when
|
||||||
|
its rationale is likely to prevent repeated reconsideration.
|
||||||
|
|
||||||
|
Future capabilities must continue to respect the
|
||||||
|
[architecture policy](../policy/architecture.md), particularly Promptkit's
|
||||||
|
role as an application-neutral library and its boundary with downstream
|
||||||
|
consumers.
|
||||||
|
|
||||||
|
## Ideas
|
||||||
|
|
||||||
|
### Extensible LLM backend registry
|
||||||
|
|
||||||
|
Introduce a registry that separates backend-specific connection,
|
||||||
|
authentication, and limited request defaults from model execution profiles.
|
||||||
|
Initial support would cover OpenAI-compatible backends and include a small
|
||||||
|
built-in catalog, potentially starting with OpenRouter. A profile could select
|
||||||
|
a backend while optionally overriding its default endpoint, and each backend
|
||||||
|
could name an optional environment variable for its API key without storing
|
||||||
|
the credential itself. Downstream consumers could register additional,
|
||||||
|
uniquely named backends, such as OpenAI or unauthenticated local-network
|
||||||
|
services, but could not replace built-in IDs. Model selection and generation
|
||||||
|
settings would remain profile concerns, and custom model clients would remain
|
||||||
|
available for behavior outside the registry's supported protocol.
|
||||||
|
|
||||||
|
### Backend-specific concurrency management
|
||||||
|
|
||||||
|
Extend the proposed LLM backend registry with optional per-backend concurrency
|
||||||
|
limits and bounded, buffered admission queues. Promptkit could then route
|
||||||
|
simultaneous generation requests according to backend capacity while
|
||||||
|
containing accidental runaway submission. Downstream consumers would continue
|
||||||
|
invoking synchronous `Run` calls, including concurrently from multiple
|
||||||
|
goroutines, and each admitted call would wait for and return its ordinary
|
||||||
|
result.
|
||||||
|
|
||||||
|
- Scope limits to an engine instance rather than hidden process-global state.
|
||||||
|
- Give different backend IDs independent capacity pools. A profile endpoint
|
||||||
|
override would remain part of its selected backend's pool.
|
||||||
|
- Configure active concurrency and waiting capacity separately. Concurrency
|
||||||
|
protects the backend, while queue capacity protects the process from
|
||||||
|
admitting an unbounded backlog.
|
||||||
|
- Give queue capacity a generous, configurable bounded default intended as a
|
||||||
|
safety ceiling for bugs or unintended loops rather than a routine
|
||||||
|
application constraint. Select an exact default during implementation
|
||||||
|
planning and measurement.
|
||||||
|
- Reject a call with a recognizable capacity error when its backend queue is
|
||||||
|
full rather than allowing it to wait outside the bounded queue.
|
||||||
|
- Admit requests before expensive preparation and artifact copying where
|
||||||
|
practical so queued work remains lightweight.
|
||||||
|
- Apply a limit to each actual generation request, including repair attempts,
|
||||||
|
without unnecessarily serializing prompt preparation.
|
||||||
|
- Make queued and active waits respect caller cancellation and deadlines.
|
||||||
|
- Treat concurrency as backend policy rather than a profile-level model
|
||||||
|
setting.
|
||||||
|
- Keep the queue ephemeral and in-process, with no survival guarantee across
|
||||||
|
engine or process shutdown.
|
||||||
|
- Preserve the existing execution model as far as practical. Durable jobs,
|
||||||
|
polling, priorities, application worker lifecycle, retries, and
|
||||||
|
cross-process coordination would be separate future capabilities.
|
||||||
|
|
||||||
|
### Explicit per-run session and reasoning controls
|
||||||
|
|
||||||
|
Allow consumers to associate a session ID with each run and to inherit,
|
||||||
|
replace, or explicitly disable the reasoning effort configured by its selected
|
||||||
|
profile. Prompt definitions can currently derive a session ID from a template,
|
||||||
|
and a non-empty runtime `ReasoningEffort` can replace the profile value, but
|
||||||
|
there is no direct request-level session ID and an empty reasoning value means
|
||||||
|
that no override was supplied. These controls would let consumers reuse one
|
||||||
|
prompt and model profile across sessions and reasoning levels without
|
||||||
|
maintaining duplicate definitions.
|
||||||
|
|
||||||
|
- Preserve a prompt's session ID template and a profile's reasoning effort as
|
||||||
|
reusable defaults.
|
||||||
|
- Let a directly supplied per-run session ID take precedence over a rendered
|
||||||
|
prompt default while retaining the existing validation limit and outbound
|
||||||
|
representation.
|
||||||
|
- Distinguish an omitted runtime reasoning choice from an explicit request to
|
||||||
|
disable reasoning.
|
||||||
|
- Ensure disabling reasoning omits the corresponding provider request setting
|
||||||
|
rather than relying on a provider-specific magic value.
|
||||||
|
- Keep the effective session ID and reasoning choice visible in prepared and
|
||||||
|
run metadata and available to injected model clients without introducing
|
||||||
|
additional prompt or profile selection mechanisms.
|
||||||
|
|
||||||
|
## Entry Format
|
||||||
|
|
||||||
|
Use a short heading followed by a concise summary. Add focused bullets when
|
||||||
|
they help preserve important scope boundaries without becoming an
|
||||||
|
implementation plan:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
### Idea name
|
||||||
|
|
||||||
|
Describe the intended capability, who benefits, and the most important scope
|
||||||
|
boundary or dependency.
|
||||||
|
|
||||||
|
- Optionally record an important behavior or boundary.
|
||||||
|
```
|
||||||
141
engine.go
141
engine.go
@@ -22,42 +22,93 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/promptkit/internal/validate"
|
"gitea.maximumdirect.net/eric/promptkit/internal/validate"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ErrInvalidConfig indicates invalid public engine configuration.
|
// ErrInvalidConfig identifies invalid engine construction, including missing
|
||||||
|
// required configuration, invalid options, and a nil Engine receiver.
|
||||||
var ErrInvalidConfig = errors.New("invalid engine configuration")
|
var ErrInvalidConfig = errors.New("invalid engine configuration")
|
||||||
|
|
||||||
var (
|
var (
|
||||||
|
// ErrInvalidRequest identifies a request whose required values, overrides,
|
||||||
|
// credentials, or effective settings are invalid.
|
||||||
ErrInvalidRequest = errors.New("invalid run request")
|
ErrInvalidRequest = errors.New("invalid run request")
|
||||||
|
// ErrPromptNotFound identifies a requested prompt ID or version that is not
|
||||||
|
// present in the selected prompt source. It does not also match
|
||||||
|
// ErrPromptLoad.
|
||||||
ErrPromptNotFound = errors.New("prompt not found")
|
ErrPromptNotFound = errors.New("prompt not found")
|
||||||
|
// ErrProfileNotFound identifies a selected profile ID that is absent from
|
||||||
|
// every configured profile source. It does not also match ErrProfileLoad.
|
||||||
ErrProfileNotFound = errors.New("profile not found")
|
ErrProfileNotFound = errors.New("profile not found")
|
||||||
|
// ErrProfileRequired identifies a request for which neither RunRequest.ProfileID
|
||||||
|
// nor the selected prompt's default profile is present. Such an error also
|
||||||
|
// matches ErrInvalidRequest.
|
||||||
ErrProfileRequired = errors.New("profile selection is required")
|
ErrProfileRequired = errors.New("profile selection is required")
|
||||||
|
// ErrPromptLoad identifies a failure to read, decode, validate, select, or
|
||||||
|
// hash a prompt definition, except for the not-found case represented by
|
||||||
|
// ErrPromptNotFound.
|
||||||
ErrPromptLoad = errors.New("failed to load prompt definition")
|
ErrPromptLoad = errors.New("failed to load prompt definition")
|
||||||
|
// ErrProfileLoad identifies a failure to read, decode, validate, or select
|
||||||
|
// an execution profile, except for the not-found case represented by
|
||||||
|
// ErrProfileNotFound.
|
||||||
ErrProfileLoad = errors.New("failed to load execution profile")
|
ErrProfileLoad = errors.New("failed to load execution profile")
|
||||||
|
// ErrAPIKeyEnvMissing identifies an APIKeyEnv whose environment variable is
|
||||||
|
// unset or empty when no direct RunRequest.APIKey takes precedence. Such an
|
||||||
|
// error also matches ErrInvalidRequest.
|
||||||
ErrAPIKeyEnvMissing = errors.New("api_key_env points to an unset environment variable")
|
ErrAPIKeyEnvMissing = errors.New("api_key_env points to an unset environment variable")
|
||||||
|
// ErrArtifactLoad identifies a failure to resolve an input artifact. Errors
|
||||||
|
// returned by an injected ArtifactReader remain available through errors.Is.
|
||||||
ErrArtifactLoad = errors.New("failed to load artifact")
|
ErrArtifactLoad = errors.New("failed to load artifact")
|
||||||
|
// ErrPromptRender identifies a failure to render prompt messages or the
|
||||||
|
// session ID from the resolved inputs and variables.
|
||||||
ErrPromptRender = errors.New("failed to render prompt")
|
ErrPromptRender = errors.New("failed to render prompt")
|
||||||
|
// ErrLLMGenerate identifies a model-client failure or a nil successful
|
||||||
|
// response. Errors returned by an injected LLMClient remain available
|
||||||
|
// through errors.Is.
|
||||||
ErrLLMGenerate = errors.New("failed to generate output")
|
ErrLLMGenerate = errors.New("failed to generate output")
|
||||||
|
// ErrValidation identifies an operational failure to load or compile a
|
||||||
|
// schema or validate output. A completed validation whose Status is
|
||||||
|
// ValidationFailed is returned in RunResult without this error.
|
||||||
ErrValidation = errors.New("failed to validate output")
|
ErrValidation = errors.New("failed to validate output")
|
||||||
)
|
)
|
||||||
|
|
||||||
// Engine prepares and runs Promptkit prompt requests.
|
// Engine prepares and runs Promptkit prompt requests.
|
||||||
|
//
|
||||||
|
// An Engine is safe for concurrent calls to [Engine.Prepare] and [Engine.Run].
|
||||||
|
// Injected collaborators may consequently be invoked concurrently.
|
||||||
type Engine struct {
|
type Engine struct {
|
||||||
runner *usecase.Runner
|
runner *usecase.Runner
|
||||||
}
|
}
|
||||||
|
|
||||||
// Config configures a public Promptkit engine.
|
// Config selects the directory-backed sources and built-in model-client
|
||||||
|
// transport used by [NewEngine]. Config has no stable JSON representation.
|
||||||
type Config struct {
|
type Config struct {
|
||||||
|
// PromptDir is the directory searched recursively for prompt definitions.
|
||||||
|
// It is required unless a WithPromptFS or WithPromptFile option supplies the
|
||||||
|
// prompt source.
|
||||||
PromptDir string
|
PromptDir string
|
||||||
|
// ProfileDir is an optional directory whose profiles take precedence over
|
||||||
|
// embedded built-in profiles. An empty value selects only built-ins unless
|
||||||
|
// profile options are also supplied.
|
||||||
ProfileDir string
|
ProfileDir string
|
||||||
|
// SchemaDir is the root for JSON Schema files. An empty value uses the
|
||||||
|
// current directory. WithSchemaFS or WithSchemaFile replaces this source.
|
||||||
SchemaDir string
|
SchemaDir string
|
||||||
// Timeout is the transport-wide safety cap for the built-in LLM client
|
// Timeout is the transport-wide safety cap for the built-in LLM client
|
||||||
// when HTTPClient is absent or has a non-positive timeout.
|
// when HTTPClient is absent or has a non-positive timeout. A zero or negative
|
||||||
|
// value selects the 10-minute default.
|
||||||
Timeout time.Duration
|
Timeout time.Duration
|
||||||
// HTTPClient is cloned for the built-in LLM client. Its positive Timeout
|
// HTTPClient is cloned for the built-in LLM client. Its positive Timeout
|
||||||
// takes precedence over Config.Timeout as the transport-wide safety cap.
|
// takes precedence over Timeout. A zero or negative client Timeout inherits
|
||||||
|
// Timeout or the 10-minute default. The supplied client is not mutated. This
|
||||||
|
// field is ignored when WithLLMClient is used.
|
||||||
HTTPClient *http.Client
|
HTTPClient *http.Client
|
||||||
}
|
}
|
||||||
|
|
||||||
// Option customizes engine construction.
|
// Option customizes engine construction.
|
||||||
|
//
|
||||||
|
// NewEngine applies options in argument order and ignores nil options. Within
|
||||||
|
// each prompt-source, profile-source, in-memory-profile, schema-source,
|
||||||
|
// model-client, and artifact-reader category, the last non-nil valid option
|
||||||
|
// replaces earlier options in that category. An invalid option fails
|
||||||
|
// construction even if a later option would replace it.
|
||||||
type Option interface {
|
type Option interface {
|
||||||
apply(*engineOptions) error
|
apply(*engineOptions) error
|
||||||
}
|
}
|
||||||
@@ -82,7 +133,10 @@ type engineOptions struct {
|
|||||||
artifactSource bool
|
artifactSource bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// WithLLMClient injects a custom LLM client for execution.
|
// WithLLMClient replaces the built-in model client used by [Engine.Run].
|
||||||
|
//
|
||||||
|
// A nil client makes NewEngine fail with ErrInvalidConfig. The client may be
|
||||||
|
// called concurrently and is not used by [Engine.Prepare].
|
||||||
func WithLLMClient(client LLMClient) Option {
|
func WithLLMClient(client LLMClient) Option {
|
||||||
return optionFunc(func(options *engineOptions) error {
|
return optionFunc(func(options *engineOptions) error {
|
||||||
if client == nil {
|
if client == nil {
|
||||||
@@ -93,7 +147,11 @@ func WithLLMClient(client LLMClient) Option {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// WithArtifactReader injects a reader for every input artifact reference.
|
// WithArtifactReader replaces the default reader for every input artifact
|
||||||
|
// reference, regardless of its ArtifactRef.Type.
|
||||||
|
//
|
||||||
|
// A nil reader makes NewEngine fail with ErrInvalidConfig. The reader may be
|
||||||
|
// called concurrently.
|
||||||
func WithArtifactReader(reader ArtifactReader) Option {
|
func WithArtifactReader(reader ArtifactReader) Option {
|
||||||
return optionFunc(func(options *engineOptions) error {
|
return optionFunc(func(options *engineOptions) error {
|
||||||
if reader == nil {
|
if reader == nil {
|
||||||
@@ -109,6 +167,9 @@ func WithArtifactReader(reader ArtifactReader) Option {
|
|||||||
//
|
//
|
||||||
// The source uses the same strict prompt YAML rules as configured prompt
|
// The source uses the same strict prompt YAML rules as configured prompt
|
||||||
// directories, and prompt content_file paths resolve within this source.
|
// directories, and prompt content_file paths resolve within this source.
|
||||||
|
// fsys must be non-nil and root must be non-empty; otherwise NewEngine fails
|
||||||
|
// with ErrInvalidConfig. This option replaces Config.PromptDir and earlier
|
||||||
|
// prompt-source options.
|
||||||
func WithPromptFS(fsys fs.FS, root string) Option {
|
func WithPromptFS(fsys fs.FS, root string) Option {
|
||||||
return optionFunc(func(options *engineOptions) error {
|
return optionFunc(func(options *engineOptions) error {
|
||||||
if fsys == nil {
|
if fsys == nil {
|
||||||
@@ -125,7 +186,9 @@ func WithPromptFS(fsys fs.FS, root string) Option {
|
|||||||
|
|
||||||
// WithPromptFile loads prompt definitions from the single prompt file at path.
|
// WithPromptFile loads prompt definitions from the single prompt file at path.
|
||||||
//
|
//
|
||||||
// Relative prompt content_file paths resolve from the file's directory.
|
// Relative prompt content_file paths resolve from the file's directory. path
|
||||||
|
// must name an existing non-directory file when NewEngine applies the option.
|
||||||
|
// This option replaces Config.PromptDir and earlier prompt-source options.
|
||||||
func WithPromptFile(path string) Option {
|
func WithPromptFile(path string) Option {
|
||||||
return optionFunc(func(options *engineOptions) error {
|
return optionFunc(func(options *engineOptions) error {
|
||||||
fsys, root, err := fileSource(path)
|
fsys, root, err := fileSource(path)
|
||||||
@@ -142,6 +205,10 @@ func WithPromptFile(path string) Option {
|
|||||||
//
|
//
|
||||||
// Profiles from this source overlay built-in profiles. Profile YAML must use
|
// Profiles from this source overlay built-in profiles. Profile YAML must use
|
||||||
// api_key_env for environment-based credentials; raw API keys are rejected.
|
// api_key_env for environment-based credentials; raw API keys are rejected.
|
||||||
|
// fsys must be non-nil and root must be non-empty; otherwise NewEngine fails
|
||||||
|
// with ErrInvalidConfig. This option replaces Config.ProfileDir and earlier
|
||||||
|
// file or FS profile-source options, but remains below WithProfiles in
|
||||||
|
// precedence.
|
||||||
func WithProfileFS(fsys fs.FS, root string) Option {
|
func WithProfileFS(fsys fs.FS, root string) Option {
|
||||||
return optionFunc(func(options *engineOptions) error {
|
return optionFunc(func(options *engineOptions) error {
|
||||||
if fsys == nil {
|
if fsys == nil {
|
||||||
@@ -159,7 +226,10 @@ func WithProfileFS(fsys fs.FS, root string) Option {
|
|||||||
// WithProfileFile loads execution profiles from the single profile file at path.
|
// WithProfileFile loads execution profiles from the single profile file at path.
|
||||||
//
|
//
|
||||||
// The profile overlays built-in profiles. Profile YAML must use api_key_env for
|
// The profile overlays built-in profiles. Profile YAML must use api_key_env for
|
||||||
// environment-based credentials; raw API keys are rejected.
|
// environment-based credentials; raw API keys are rejected. path must name an
|
||||||
|
// existing non-directory file when NewEngine applies the option. This option
|
||||||
|
// replaces Config.ProfileDir and earlier file or FS profile-source options,
|
||||||
|
// but remains below WithProfiles in precedence.
|
||||||
func WithProfileFile(path string) Option {
|
func WithProfileFile(path string) Option {
|
||||||
return optionFunc(func(options *engineOptions) error {
|
return optionFunc(func(options *engineOptions) error {
|
||||||
fsys, root, err := fileSource(path)
|
fsys, root, err := fileSource(path)
|
||||||
@@ -174,6 +244,11 @@ func WithProfileFile(path string) Option {
|
|||||||
|
|
||||||
// WithProfiles configures in-memory profiles that take precedence over
|
// WithProfiles configures in-memory profiles that take precedence over
|
||||||
// configured profile files and built-in profiles.
|
// configured profile files and built-in profiles.
|
||||||
|
//
|
||||||
|
// NewEngine validates and copies every profile. IDs must be unique within one
|
||||||
|
// call. An invalid profile, duplicate ID, or unsupported ExtraParams value
|
||||||
|
// makes construction fail with ErrInvalidConfig. Repeating WithProfiles
|
||||||
|
// replaces the complete earlier in-memory set rather than merging it.
|
||||||
func WithProfiles(profiles ...Profile) Option {
|
func WithProfiles(profiles ...Profile) Option {
|
||||||
return optionFunc(func(options *engineOptions) error {
|
return optionFunc(func(options *engineOptions) error {
|
||||||
repo, err := newMemoryProfileRepository(profiles)
|
repo, err := newMemoryProfileRepository(profiles)
|
||||||
@@ -189,7 +264,9 @@ func WithProfiles(profiles ...Profile) Option {
|
|||||||
// WithSchemaFS loads JSON Schema documents from fsys under root.
|
// WithSchemaFS loads JSON Schema documents from fsys under root.
|
||||||
//
|
//
|
||||||
// Prompt schema_path values resolve within this source when schema validation
|
// Prompt schema_path values resolve within this source when schema validation
|
||||||
// or structured output is requested.
|
// or structured output is requested. fsys must be non-nil and root must be
|
||||||
|
// non-empty; otherwise NewEngine fails with ErrInvalidConfig. This option
|
||||||
|
// replaces Config.SchemaDir and earlier schema-source options.
|
||||||
func WithSchemaFS(fsys fs.FS, root string) Option {
|
func WithSchemaFS(fsys fs.FS, root string) Option {
|
||||||
return optionFunc(func(options *engineOptions) error {
|
return optionFunc(func(options *engineOptions) error {
|
||||||
if fsys == nil {
|
if fsys == nil {
|
||||||
@@ -206,7 +283,9 @@ func WithSchemaFS(fsys fs.FS, root string) Option {
|
|||||||
|
|
||||||
// WithSchemaFile loads JSON Schema documents from the single schema file at path.
|
// WithSchemaFile loads JSON Schema documents from the single schema file at path.
|
||||||
//
|
//
|
||||||
// Prompt schema_path values refer to the file's base name.
|
// Prompt schema_path values refer to the file's base name. path must name an
|
||||||
|
// existing non-directory file when NewEngine applies the option. This option
|
||||||
|
// replaces Config.SchemaDir and earlier schema-source options.
|
||||||
func WithSchemaFile(path string) Option {
|
func WithSchemaFile(path string) Option {
|
||||||
return optionFunc(func(options *engineOptions) error {
|
return optionFunc(func(options *engineOptions) error {
|
||||||
fsys, root, err := fileSource(path)
|
fsys, root, err := fileSource(path)
|
||||||
@@ -220,6 +299,15 @@ func WithSchemaFile(path string) Option {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// NewEngine constructs an Engine from configuration and options.
|
// NewEngine constructs an Engine from configuration and options.
|
||||||
|
//
|
||||||
|
// Options are applied in order according to [Option]. PromptDir is required
|
||||||
|
// unless a prompt-source option is present. Construction validates option
|
||||||
|
// arguments and in-memory profiles but defers reading and validating prompt,
|
||||||
|
// file-backed profile, and schema contents until Prepare or Run needs them.
|
||||||
|
//
|
||||||
|
// NewEngine returns an error matching ErrInvalidConfig for invalid
|
||||||
|
// configuration or options. It does not perform model requests or require
|
||||||
|
// credentials.
|
||||||
func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
|
func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
|
||||||
var options engineOptions
|
var options engineOptions
|
||||||
for _, opt := range opts {
|
for _, opt := range opts {
|
||||||
@@ -305,7 +393,22 @@ func fileSource(name string) (fs.FS, string, error) {
|
|||||||
return os.DirFS(dir), filepath.ToSlash(base), nil
|
return os.DirFS(dir), filepath.ToSlash(base), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// Prepare resolves a prompt request without calling an LLM.
|
// Prepare resolves and renders a prompt request without calling an LLM.
|
||||||
|
//
|
||||||
|
// Prepare selects the prompt and profile, resolves effective execution
|
||||||
|
// settings and the output contract, loads and hashes inputs, loads structured
|
||||||
|
// output schema metadata when required, and renders the session ID and
|
||||||
|
// messages. The returned PreparedRun is owned by the caller and never contains
|
||||||
|
// a resolved API-key value, model output, or validation result.
|
||||||
|
//
|
||||||
|
// A nil Engine returns an error matching ErrInvalidConfig. Request and
|
||||||
|
// preparation failures may match ErrInvalidRequest, ErrPromptNotFound,
|
||||||
|
// ErrPromptLoad, ErrProfileNotFound, ErrProfileLoad, ErrProfileRequired,
|
||||||
|
// ErrAPIKeyEnvMissing, ErrArtifactLoad, ErrPromptRender, or ErrValidation as
|
||||||
|
// applicable. Cancellation is passed to the active collaborator and is
|
||||||
|
// reported in the applicable operation category; no general errors.Is
|
||||||
|
// relationship to ctx.Err is promised. Prepare returns no partial result on
|
||||||
|
// error.
|
||||||
func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, error) {
|
func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, error) {
|
||||||
if e == nil || e.runner == nil {
|
if e == nil || e.runner == nil {
|
||||||
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
|
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
|
||||||
@@ -323,7 +426,21 @@ func (e *Engine) Prepare(ctx context.Context, req RunRequest) (*PreparedRun, err
|
|||||||
return fromDomainPreparedRun(prepared), nil
|
return fromDomainPreparedRun(prepared), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// Run executes a prompt request and returns the generated artifact and metadata.
|
// Run prepares a request, invokes the configured LLMClient, and validates the
|
||||||
|
// generated output.
|
||||||
|
//
|
||||||
|
// A content-validation failure is a successful run whose
|
||||||
|
// RunResult.Validation has Status ValidationFailed. An inability to perform
|
||||||
|
// validation returns an error matching ErrValidation and no partial result.
|
||||||
|
// The public Engine does not perform output repair, so validation is
|
||||||
|
// single-pass even when OutputContract.RepairAttempts is positive.
|
||||||
|
//
|
||||||
|
// Run can return every error category documented by [Engine.Prepare], plus
|
||||||
|
// ErrLLMGenerate. Errors from injected clients remain available through
|
||||||
|
// errors.Is. Cancellation is passed through the active collaborator and is
|
||||||
|
// reported in the applicable operation category; no general errors.Is
|
||||||
|
// relationship to ctx.Err is promised. A nil Engine returns ErrInvalidConfig.
|
||||||
|
// Run returns no partial result on error.
|
||||||
func (e *Engine) Run(ctx context.Context, req RunRequest) (*RunResult, error) {
|
func (e *Engine) Run(ctx context.Context, req RunRequest) (*RunResult, error) {
|
||||||
if e == nil || e.runner == nil {
|
if e == nil || e.runner == nil {
|
||||||
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
|
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
|
||||||
|
|||||||
@@ -1020,6 +1020,7 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
|
|||||||
client promptkit.LLMClient
|
client promptkit.LLMClient
|
||||||
schemaDir string
|
schemaDir string
|
||||||
want error
|
want error
|
||||||
|
notWant error
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "invalid request",
|
name: "invalid request",
|
||||||
@@ -1032,6 +1033,7 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
|
|||||||
req: promptkit.RunRequest{PromptID: "missing.prompt"},
|
req: promptkit.RunRequest{PromptID: "missing.prompt"},
|
||||||
client: &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}},
|
client: &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}},
|
||||||
want: promptkit.ErrPromptNotFound,
|
want: promptkit.ErrPromptNotFound,
|
||||||
|
notWant: promptkit.ErrPromptLoad,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "profile not found",
|
name: "profile not found",
|
||||||
@@ -1044,6 +1046,7 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
|
|||||||
},
|
},
|
||||||
client: &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}},
|
client: &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}},
|
||||||
want: promptkit.ErrProfileNotFound,
|
want: promptkit.ErrProfileNotFound,
|
||||||
|
notWant: promptkit.ErrProfileLoad,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "artifact load",
|
name: "artifact load",
|
||||||
@@ -1114,6 +1117,9 @@ func TestPublicErrorsSupportErrorsIs(t *testing.T) {
|
|||||||
if !errors.Is(err, tc.want) {
|
if !errors.Is(err, tc.want) {
|
||||||
t.Fatalf("expected errors.Is(%v), got %v", tc.want, err)
|
t.Fatalf("expected errors.Is(%v), got %v", tc.want, err)
|
||||||
}
|
}
|
||||||
|
if tc.notWant != nil && errors.Is(err, tc.notWant) {
|
||||||
|
t.Fatalf("did not expect errors.Is(%v), got %v", tc.notWant, err)
|
||||||
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1699,6 +1705,16 @@ func TestEngineRunLayersTransportAndGenerationTimeouts(t *testing.T) {
|
|||||||
configTimeout: 5 * time.Second,
|
configTimeout: 5 * time.Second,
|
||||||
wantRemainingAtRequest: 5 * time.Second,
|
wantRemainingAtRequest: 5 * time.Second,
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: "zero configuration uses ten minute transport default",
|
||||||
|
wantRemainingAtRequest: 10 * time.Minute,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "negative configuration uses ten minute transport default",
|
||||||
|
configTimeout: -2 * time.Second,
|
||||||
|
suppliedClientTimeout: -3 * time.Second,
|
||||||
|
wantRemainingAtRequest: 10 * time.Minute,
|
||||||
|
},
|
||||||
{
|
{
|
||||||
name: "profile deadline is shorter than transport cap",
|
name: "profile deadline is shorter than transport cap",
|
||||||
suppliedClientTimeout: 6 * time.Second,
|
suppliedClientTimeout: 6 * time.Second,
|
||||||
|
|||||||
81
examples/go-library/run/main.go
Normal file
81
examples/go-library/run/main.go
Normal file
@@ -0,0 +1,81 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
|
)
|
||||||
|
|
||||||
|
type deterministicClient struct{}
|
||||||
|
|
||||||
|
func (deterministicClient) Generate(
|
||||||
|
ctx context.Context,
|
||||||
|
_ promptkit.GenerateRequest,
|
||||||
|
) (*promptkit.GenerateResponse, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return &promptkit.GenerateResponse{
|
||||||
|
Content: "Ada finished the migration review.",
|
||||||
|
Usage: promptkit.TokenUsage{
|
||||||
|
PromptTokens: 12,
|
||||||
|
CompletionTokens: 6,
|
||||||
|
TotalTokens: 18,
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type summary struct {
|
||||||
|
Output string `json:"output"`
|
||||||
|
ValidationStatus promptkit.ValidationStatus `json:"validation_status"`
|
||||||
|
IsValid bool `json:"is_valid"`
|
||||||
|
Model string `json:"model"`
|
||||||
|
TotalTokens int `json:"total_tokens"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
engine, err := promptkit.NewEngine(
|
||||||
|
promptkit.Config{},
|
||||||
|
promptkit.WithPromptFile("examples/go-library/run/prompt.yaml"),
|
||||||
|
promptkit.WithProfiles(promptkit.Profile{
|
||||||
|
ID: "offline-example",
|
||||||
|
Endpoint: "https://example.invalid/v1",
|
||||||
|
Model: "offline-model",
|
||||||
|
}),
|
||||||
|
promptkit.WithLLMClient(deterministicClient{}),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
exit(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := engine.Run(context.Background(), promptkit.RunRequest{
|
||||||
|
PromptID: "example.run",
|
||||||
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
|
"note": promptkit.Inline("Ada finished the migration review."),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
exit(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
encoder := json.NewEncoder(os.Stdout)
|
||||||
|
encoder.SetIndent("", " ")
|
||||||
|
if err := encoder.Encode(summary{
|
||||||
|
Output: result.RawOutput,
|
||||||
|
ValidationStatus: result.Validation.Status,
|
||||||
|
IsValid: result.Validation.IsValid,
|
||||||
|
Model: result.ModelName,
|
||||||
|
TotalTokens: result.Usage.TotalTokens,
|
||||||
|
}); err != nil {
|
||||||
|
exit(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func exit(err error) {
|
||||||
|
fmt.Fprintln(os.Stderr, err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
16
examples/go-library/run/prompt.yaml
Normal file
16
examples/go-library/run/prompt.yaml
Normal file
@@ -0,0 +1,16 @@
|
|||||||
|
id: example.run
|
||||||
|
version: "1.0.0"
|
||||||
|
default_profile: offline-example
|
||||||
|
description: Run a prompt with a deterministic injected model client.
|
||||||
|
inputs:
|
||||||
|
- name: note
|
||||||
|
required: true
|
||||||
|
content_type: text/plain
|
||||||
|
messages:
|
||||||
|
- role: system
|
||||||
|
content: Summarize the note in one sentence.
|
||||||
|
- role: user
|
||||||
|
content: '{{input "note"}}'
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: basic
|
||||||
@@ -2,19 +2,23 @@ package promptkit
|
|||||||
|
|
||||||
import "fmt"
|
import "fmt"
|
||||||
|
|
||||||
// String returns a concise request summary without exposing direct API keys.
|
// String returns a concise request summary without exposing the direct API key
|
||||||
|
// or input and variable contents. Reflection-based formatting does not carry
|
||||||
|
// this guarantee.
|
||||||
func (r RunRequest) String() string {
|
func (r RunRequest) String() string {
|
||||||
return r.redactedString()
|
return r.redactedString()
|
||||||
}
|
}
|
||||||
|
|
||||||
// GoString returns a concise request summary without exposing direct API keys.
|
// GoString returns a concise request summary without exposing the direct API
|
||||||
|
// key or input and variable contents. Reflection-based formatting does not
|
||||||
|
// carry this guarantee.
|
||||||
func (r RunRequest) GoString() string {
|
func (r RunRequest) GoString() string {
|
||||||
return r.redactedString()
|
return r.redactedString()
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r RunRequest) redactedString() string {
|
func (r RunRequest) redactedString() string {
|
||||||
return fmt.Sprintf(
|
return fmt.Sprintf(
|
||||||
"promptkit.RunRequest{PromptID:%q PromptVersion:%q ProfileID:%q APIKeySet:%t Inputs:%d Vars:%d ExecutionSet:%t ValidationSet:%t Metadata:%d}",
|
"promptkit.RunRequest{PromptID:%q PromptVersion:%q ProfileID:%q APIKeySet:%t Inputs:%d Vars:%d ExecutionSet:%t ValidationSet:%t}",
|
||||||
r.PromptID,
|
r.PromptID,
|
||||||
r.PromptVersion,
|
r.PromptVersion,
|
||||||
r.ProfileID,
|
r.ProfileID,
|
||||||
@@ -23,18 +27,19 @@ func (r RunRequest) redactedString() string {
|
|||||||
len(r.Vars),
|
len(r.Vars),
|
||||||
r.Execution != nil,
|
r.Execution != nil,
|
||||||
r.Validation != nil,
|
r.Validation != nil,
|
||||||
len(r.Metadata),
|
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// String returns a concise request summary without exposing direct API keys or
|
// String returns a concise request summary without exposing direct API keys or
|
||||||
// rendered prompt content.
|
// rendered prompt content. Reflection-based formatting does not carry this
|
||||||
|
// guarantee.
|
||||||
func (r GenerateRequest) String() string {
|
func (r GenerateRequest) String() string {
|
||||||
return r.redactedString()
|
return r.redactedString()
|
||||||
}
|
}
|
||||||
|
|
||||||
// GoString returns a concise request summary without exposing direct API keys or
|
// GoString returns a concise request summary without exposing direct API keys or
|
||||||
// rendered prompt content.
|
// rendered prompt content. Reflection-based formatting does not carry this
|
||||||
|
// guarantee.
|
||||||
func (r GenerateRequest) GoString() string {
|
func (r GenerateRequest) GoString() string {
|
||||||
return r.redactedString()
|
return r.redactedString()
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -68,7 +68,6 @@ type RunRequest struct {
|
|||||||
Vars map[string]string
|
Vars map[string]string
|
||||||
Execution *ExecutionTargetOverride
|
Execution *ExecutionTargetOverride
|
||||||
Validation *OutputContract
|
Validation *OutputContract
|
||||||
Metadata map[string]string
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// RunResult represents the complete result of a prompt execution run.
|
// RunResult represents the complete result of a prompt execution run.
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io/fs"
|
"io/fs"
|
||||||
|
"net/url"
|
||||||
"os"
|
"os"
|
||||||
"path"
|
"path"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
@@ -16,6 +17,8 @@ import (
|
|||||||
"github.com/santhosh-tekuri/jsonschema/v6"
|
"github.com/santhosh-tekuri/jsonschema/v6"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
const jsonSchemaDraft2020 = "https://json-schema.org/draft/2020-12/schema"
|
||||||
|
|
||||||
// StandardValidator provides basic, JSON, and JSON Schema output validation.
|
// StandardValidator provides basic, JSON, and JSON Schema output validation.
|
||||||
type StandardValidator struct {
|
type StandardValidator struct {
|
||||||
schemaBaseDir string
|
schemaBaseDir string
|
||||||
@@ -121,7 +124,11 @@ func (v *StandardValidator) validateJSONSchema(instance any, schemaPath string)
|
|||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
compiler := jsonschema.NewCompiler()
|
schemaRoot, err := v.schemaRoot()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
compiler := newSchemaCompiler(standardSchemaLoader{root: schemaRoot})
|
||||||
schema, err := compiler.Compile(resolvedSchemaPath)
|
schema, err := compiler.Compile(resolvedSchemaPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", resolvedSchemaPath, err)
|
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", resolvedSchemaPath, err)
|
||||||
@@ -140,7 +147,10 @@ func (v *FSValidator) validateJSONSchema(instance any, schemaPath string) ([]str
|
|||||||
}
|
}
|
||||||
|
|
||||||
resourceURL := fsSchemaResourceURL(schemaName)
|
resourceURL := fsSchemaResourceURL(schemaName)
|
||||||
compiler := jsonschema.NewCompiler()
|
if err := validateSchemaDialect(schemaDoc); err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", schemaName, err)
|
||||||
|
}
|
||||||
|
compiler := newSchemaCompiler(fsSchemaLoader{fsys: v.fsys, root: filecatalog.CleanFSRoot(v.root)})
|
||||||
if err := compiler.AddResource(resourceURL, schemaDoc); err != nil {
|
if err := compiler.AddResource(resourceURL, schemaDoc); err != nil {
|
||||||
return nil, fmt.Errorf("failed to register JSON schema %q: %w", schemaName, err)
|
return nil, fmt.Errorf("failed to register JSON schema %q: %w", schemaName, err)
|
||||||
}
|
}
|
||||||
@@ -184,6 +194,9 @@ func (v *StandardValidator) LoadSchemaDocument(ctx context.Context, schemaPath s
|
|||||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||||
return nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
|
return nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
|
||||||
}
|
}
|
||||||
|
if err := validateSchemaDialect(doc); err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
|
||||||
|
}
|
||||||
return doc, nil
|
return doc, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -206,12 +219,14 @@ func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error)
|
|||||||
return "", errors.New("schema path is required for json_schema validation")
|
return "", errors.New("schema path is required for json_schema validation")
|
||||||
}
|
}
|
||||||
|
|
||||||
resolved := schemaPath
|
root, err := v.schemaRoot()
|
||||||
if !filepath.IsAbs(schemaPath) {
|
if err != nil {
|
||||||
resolved = filepath.Join(v.schemaBaseDir, schemaPath)
|
return "", err
|
||||||
|
}
|
||||||
|
resolved, err := containedFilesystemPath(root, schemaPath)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
}
|
}
|
||||||
|
|
||||||
resolved = filepath.Clean(resolved)
|
|
||||||
if _, err := os.Stat(resolved); err != nil {
|
if _, err := os.Stat(resolved); err != nil {
|
||||||
return "", fmt.Errorf("failed to access schema file %q: %w", resolved, err)
|
return "", fmt.Errorf("failed to access schema file %q: %w", resolved, err)
|
||||||
}
|
}
|
||||||
@@ -219,6 +234,22 @@ func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error)
|
|||||||
return resolved, nil
|
return resolved, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (v *StandardValidator) schemaRoot() (string, error) {
|
||||||
|
root := v.schemaBaseDir
|
||||||
|
if strings.TrimSpace(root) == "" {
|
||||||
|
root = "."
|
||||||
|
}
|
||||||
|
absolute, err := filepath.Abs(root)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("failed to resolve schema source %q: %w", root, err)
|
||||||
|
}
|
||||||
|
resolved, err := filepath.EvalSymlinks(absolute)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("failed to access schema source %q: %w", root, err)
|
||||||
|
}
|
||||||
|
return resolved, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (v *FSValidator) loadSchemaDocument(schemaPath string) (string, any, error) {
|
func (v *FSValidator) loadSchemaDocument(schemaPath string) (string, any, error) {
|
||||||
resolved, err := v.resolveSchemaPath(schemaPath)
|
resolved, err := v.resolveSchemaPath(schemaPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -234,6 +265,9 @@ func (v *FSValidator) loadSchemaDocument(schemaPath string) (string, any, error)
|
|||||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||||
return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
|
return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
|
||||||
}
|
}
|
||||||
|
if err := validateSchemaDialect(doc); err != nil {
|
||||||
|
return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
|
||||||
|
}
|
||||||
return resolved, doc, nil
|
return resolved, doc, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -290,3 +324,131 @@ func cleanSchemaFSPath(schemaPath string) (string, error) {
|
|||||||
func fsSchemaResourceURL(schemaName string) string {
|
func fsSchemaResourceURL(schemaName string) string {
|
||||||
return "promptkit-schema:///" + strings.TrimPrefix(path.Clean(schemaName), "/")
|
return "promptkit-schema:///" + strings.TrimPrefix(path.Clean(schemaName), "/")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func newSchemaCompiler(loader jsonschema.URLLoader) *jsonschema.Compiler {
|
||||||
|
compiler := jsonschema.NewCompiler()
|
||||||
|
compiler.DefaultDraft(jsonschema.Draft2020)
|
||||||
|
compiler.UseLoader(loader)
|
||||||
|
return compiler
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateSchemaDialect(doc any) error {
|
||||||
|
object, ok := doc.(map[string]any)
|
||||||
|
if !ok {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
value, ok := object["$schema"]
|
||||||
|
if !ok {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
dialect, ok := value.(string)
|
||||||
|
if !ok {
|
||||||
|
return errors.New("$schema must be a string")
|
||||||
|
}
|
||||||
|
if dialect != jsonSchemaDraft2020 && dialect != jsonSchemaDraft2020+"#" {
|
||||||
|
return fmt.Errorf("unsupported JSON Schema dialect %q; expected %q", dialect, jsonSchemaDraft2020)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type standardSchemaLoader struct {
|
||||||
|
root string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l standardSchemaLoader) Load(resourceURL string) (any, error) {
|
||||||
|
fileName, err := (jsonschema.FileLoader{}).ToFile(resourceURL)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("schema reference %q is not a contained file reference: %w", resourceURL, err)
|
||||||
|
}
|
||||||
|
resolved, err := containedFilesystemPath(l.root, fileName)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return loadJSONSchemaFile(resolved)
|
||||||
|
}
|
||||||
|
|
||||||
|
func containedFilesystemPath(root, name string) (string, error) {
|
||||||
|
candidate := name
|
||||||
|
if !filepath.IsAbs(candidate) {
|
||||||
|
candidate = filepath.Join(root, candidate)
|
||||||
|
}
|
||||||
|
candidate, err := filepath.Abs(candidate)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("failed to resolve schema path %q: %w", name, err)
|
||||||
|
}
|
||||||
|
candidate, err = filepath.EvalSymlinks(candidate)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("failed to access schema file %q: %w", candidate, err)
|
||||||
|
}
|
||||||
|
relative, err := filepath.Rel(root, candidate)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("failed to compare schema path %q with source root: %w", candidate, err)
|
||||||
|
}
|
||||||
|
if relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
|
||||||
|
return "", fmt.Errorf("schema path %q escapes source root", name)
|
||||||
|
}
|
||||||
|
return candidate, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadJSONSchemaFile(name string) (any, error) {
|
||||||
|
raw, err := os.ReadFile(name)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
var doc any
|
||||||
|
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := validateSchemaDialect(doc); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return doc, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type fsSchemaLoader struct {
|
||||||
|
fsys fs.FS
|
||||||
|
root string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l fsSchemaLoader) Load(resourceURL string) (any, error) {
|
||||||
|
parsed, err := url.Parse(resourceURL)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("invalid schema reference %q: %w", resourceURL, err)
|
||||||
|
}
|
||||||
|
if parsed.Scheme != "promptkit-schema" || parsed.Host != "" {
|
||||||
|
return nil, fmt.Errorf("schema reference %q is not allowed", resourceURL)
|
||||||
|
}
|
||||||
|
name, err := url.PathUnescape(strings.TrimPrefix(parsed.Path, "/"))
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("invalid schema reference %q: %w", resourceURL, err)
|
||||||
|
}
|
||||||
|
name = path.Clean(name)
|
||||||
|
if l.root == "." {
|
||||||
|
if strings.HasPrefix(name, "../") || name == ".." {
|
||||||
|
return nil, fmt.Errorf("schema reference %q escapes source root", resourceURL)
|
||||||
|
}
|
||||||
|
} else if name != l.root && !strings.HasPrefix(name, l.root+"/") {
|
||||||
|
return nil, fmt.Errorf("schema reference %q escapes source root", resourceURL)
|
||||||
|
}
|
||||||
|
|
||||||
|
rootInfo, err := fs.Stat(l.fsys, l.root)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if !rootInfo.IsDir() && name != l.root {
|
||||||
|
return nil, fmt.Errorf("schema reference %q is outside the configured schema file", resourceURL)
|
||||||
|
}
|
||||||
|
|
||||||
|
raw, err := fs.ReadFile(l.fsys, name)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
var doc any
|
||||||
|
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := validateSchemaDialect(doc); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return doc, nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -2,8 +2,10 @@ package validate
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"encoding/json"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"testing/fstest"
|
"testing/fstest"
|
||||||
@@ -411,3 +413,138 @@ func TestFSValidatorLoadSchemaDocument(t *testing.T) {
|
|||||||
t.Fatalf("unexpected schema document: %#v", doc)
|
t.Fatalf("unexpected schema document: %#v", doc)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestStandardValidatorJSONSchemaReferenceBoundaries(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
if err := os.WriteFile(filepath.Join(root, "child.json"), []byte(`{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"type": "string"
|
||||||
|
}`), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
reference string
|
||||||
|
wantError string
|
||||||
|
writeOuter bool
|
||||||
|
}{
|
||||||
|
{name: "contained relative reference", reference: "child.json"},
|
||||||
|
{name: "remote reference", reference: "https://example.test/schema.json", wantError: "not a contained file reference"},
|
||||||
|
{name: "escaping reference", reference: "../outside.json", wantError: "escapes source root", writeOuter: true},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
if tc.writeOuter {
|
||||||
|
if err := os.WriteFile(filepath.Join(filepath.Dir(root), "outside.json"), []byte(`{"type":"string"}`), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
schema := `{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"$ref": ` + strconv.Quote(tc.reference) + `
|
||||||
|
}`
|
||||||
|
if err := os.WriteFile(filepath.Join(root, "root.json"), []byte(schema), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
v := NewStandardValidator(root)
|
||||||
|
result, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`"value"`)}, domain.OutputContract{
|
||||||
|
ValidationMode: domain.ValidationJSONSchema,
|
||||||
|
SchemaPath: "root.json",
|
||||||
|
})
|
||||||
|
if tc.wantError == "" {
|
||||||
|
if err != nil || !result.IsValid {
|
||||||
|
t.Fatalf("expected contained reference to validate, got result=%#v error=%v", result, err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err == nil || !strings.Contains(err.Error(), tc.wantError) {
|
||||||
|
t.Fatalf("expected error containing %q, got %v", tc.wantError, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFSValidatorJSONSchemaReferenceBoundaries(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
reference string
|
||||||
|
wantError string
|
||||||
|
}{
|
||||||
|
{name: "same document fragment", reference: "#/$defs/value"},
|
||||||
|
{name: "contained relative reference", reference: "child.json"},
|
||||||
|
{name: "remote reference", reference: "https://example.test/schema.json", wantError: "is not allowed"},
|
||||||
|
{name: "escaping reference", reference: "../outside.json", wantError: "escapes source root"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
rootSchema := `{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"$defs": {"value": {"type": "string"}},
|
||||||
|
"$ref": ` + strconv.Quote(tc.reference) + `
|
||||||
|
}`
|
||||||
|
v := NewFSValidator(fstest.MapFS{
|
||||||
|
"schemas/root.json": &fstest.MapFile{Data: []byte(rootSchema)},
|
||||||
|
"schemas/child.json": &fstest.MapFile{Data: []byte(`{"type":"string"}`)},
|
||||||
|
"outside.json": &fstest.MapFile{Data: []byte(`{"type":"string"}`)},
|
||||||
|
}, "schemas")
|
||||||
|
result, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`"value"`)}, domain.OutputContract{
|
||||||
|
ValidationMode: domain.ValidationJSONSchema,
|
||||||
|
SchemaPath: "root.json",
|
||||||
|
})
|
||||||
|
if tc.wantError == "" {
|
||||||
|
if err != nil || !result.IsValid {
|
||||||
|
t.Fatalf("expected supported reference to validate, got result=%#v error=%v", result, err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err == nil || !strings.Contains(err.Error(), tc.wantError) {
|
||||||
|
t.Fatalf("expected error containing %q, got %v", tc.wantError, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestJSONSchemaDialectIsDraft2020(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
dialect string
|
||||||
|
wantError bool
|
||||||
|
}{
|
||||||
|
{name: "omitted uses supported default"},
|
||||||
|
{name: "draft 2020-12", dialect: "https://json-schema.org/draft/2020-12/schema"},
|
||||||
|
{name: "draft 7 rejected", dialect: "http://json-schema.org/draft-07/schema#", wantError: true},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
schema := map[string]any{"type": "object"}
|
||||||
|
if tc.dialect != "" {
|
||||||
|
schema["$schema"] = tc.dialect
|
||||||
|
}
|
||||||
|
data, err := json.Marshal(schema)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
v := NewFSValidator(fstest.MapFS{
|
||||||
|
"schema.json": &fstest.MapFile{Data: data},
|
||||||
|
}, ".")
|
||||||
|
_, err = v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{}`)}, domain.OutputContract{
|
||||||
|
ValidationMode: domain.ValidationJSONSchema,
|
||||||
|
SchemaPath: "schema.json",
|
||||||
|
})
|
||||||
|
if tc.wantError {
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "unsupported JSON Schema dialect") {
|
||||||
|
t.Fatalf("expected unsupported-dialect error, got %v", err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("expected supported dialect, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
154
json.go
Normal file
154
json.go
Normal file
@@ -0,0 +1,154 @@
|
|||||||
|
package promptkit
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// MarshalJSON implements json.Marshaler for PreparedRun. It uses RFC 3339
|
||||||
|
// timestamps, integer duration_ms, and omits zero timing values.
|
||||||
|
func (r PreparedRun) MarshalJSON() ([]byte, error) {
|
||||||
|
var startTime, endTime *time.Time
|
||||||
|
if !r.StartTime.IsZero() {
|
||||||
|
startTime = &r.StartTime
|
||||||
|
}
|
||||||
|
if !r.EndTime.IsZero() {
|
||||||
|
endTime = &r.EndTime
|
||||||
|
}
|
||||||
|
|
||||||
|
var durationMS *int64
|
||||||
|
if r.DurationMS != 0 {
|
||||||
|
durationMS = &r.DurationMS
|
||||||
|
}
|
||||||
|
|
||||||
|
return json.Marshal(struct {
|
||||||
|
PromptID string `json:"prompt_id"`
|
||||||
|
PromptVersion string `json:"prompt_version,omitempty"`
|
||||||
|
PromptHash string `json:"prompt_hash,omitempty"`
|
||||||
|
SelectedProfileID string `json:"selected_profile_id"`
|
||||||
|
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
||||||
|
OutputContract OutputContract `json:"output_contract"`
|
||||||
|
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
|
||||||
|
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
||||||
|
SessionID string `json:"session_id,omitempty"`
|
||||||
|
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
||||||
|
Messages []RenderedMessage `json:"messages"`
|
||||||
|
StartTime *time.Time `json:"start_time,omitempty"`
|
||||||
|
EndTime *time.Time `json:"end_time,omitempty"`
|
||||||
|
DurationMS *int64 `json:"duration_ms,omitempty"`
|
||||||
|
}{
|
||||||
|
PromptID: r.PromptID,
|
||||||
|
PromptVersion: r.PromptVersion,
|
||||||
|
PromptHash: r.PromptHash,
|
||||||
|
SelectedProfileID: r.SelectedProfileID,
|
||||||
|
EffectiveModelParams: r.EffectiveModelParams,
|
||||||
|
OutputContract: r.OutputContract,
|
||||||
|
StructuredOutput: r.StructuredOutput,
|
||||||
|
InputHashes: r.InputHashes,
|
||||||
|
SessionID: r.SessionID,
|
||||||
|
RenderedPromptHash: r.RenderedPromptHash,
|
||||||
|
Messages: r.Messages,
|
||||||
|
StartTime: startTime,
|
||||||
|
EndTime: endTime,
|
||||||
|
DurationMS: durationMS,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// MarshalJSON implements json.Marshaler for RunResult. It encodes Duration as
|
||||||
|
// integer milliseconds in duration_ms and omits zero timing values.
|
||||||
|
func (r RunResult) MarshalJSON() ([]byte, error) {
|
||||||
|
var startTime, endTime *time.Time
|
||||||
|
if !r.StartTime.IsZero() {
|
||||||
|
startTime = &r.StartTime
|
||||||
|
}
|
||||||
|
if !r.EndTime.IsZero() {
|
||||||
|
endTime = &r.EndTime
|
||||||
|
}
|
||||||
|
|
||||||
|
var durationMS *int64
|
||||||
|
if r.Duration != 0 {
|
||||||
|
value := r.Duration.Milliseconds()
|
||||||
|
durationMS = &value
|
||||||
|
}
|
||||||
|
|
||||||
|
return json.Marshal(runResultJSON{
|
||||||
|
RunID: r.RunID,
|
||||||
|
Artifact: r.Artifact,
|
||||||
|
RawOutput: r.RawOutput,
|
||||||
|
Validation: r.Validation,
|
||||||
|
PromptID: r.PromptID,
|
||||||
|
PromptVersion: r.PromptVersion,
|
||||||
|
PromptHash: r.PromptHash,
|
||||||
|
RenderedPromptHash: r.RenderedPromptHash,
|
||||||
|
SelectedProfileID: r.SelectedProfileID,
|
||||||
|
ModelName: r.ModelName,
|
||||||
|
Endpoint: r.Endpoint,
|
||||||
|
EffectiveModelParams: r.EffectiveModelParams,
|
||||||
|
InputHashes: r.InputHashes,
|
||||||
|
Usage: r.Usage,
|
||||||
|
StartTime: startTime,
|
||||||
|
EndTime: endTime,
|
||||||
|
DurationMS: durationMS,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// UnmarshalJSON implements json.Unmarshaler for RunResult. It decodes
|
||||||
|
// duration_ms into Duration with millisecond precision.
|
||||||
|
func (r *RunResult) UnmarshalJSON(data []byte) error {
|
||||||
|
var wire runResultJSON
|
||||||
|
if err := json.Unmarshal(data, &wire); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
*r = RunResult{
|
||||||
|
RunID: wire.RunID,
|
||||||
|
Artifact: wire.Artifact,
|
||||||
|
RawOutput: wire.RawOutput,
|
||||||
|
Validation: wire.Validation,
|
||||||
|
PromptID: wire.PromptID,
|
||||||
|
PromptVersion: wire.PromptVersion,
|
||||||
|
PromptHash: wire.PromptHash,
|
||||||
|
RenderedPromptHash: wire.RenderedPromptHash,
|
||||||
|
SelectedProfileID: wire.SelectedProfileID,
|
||||||
|
ModelName: wire.ModelName,
|
||||||
|
Endpoint: wire.Endpoint,
|
||||||
|
EffectiveModelParams: wire.EffectiveModelParams,
|
||||||
|
InputHashes: wire.InputHashes,
|
||||||
|
Usage: wire.Usage,
|
||||||
|
Duration: time.Duration(valueOrZero(wire.DurationMS)) * time.Millisecond,
|
||||||
|
}
|
||||||
|
if wire.StartTime != nil {
|
||||||
|
r.StartTime = *wire.StartTime
|
||||||
|
}
|
||||||
|
if wire.EndTime != nil {
|
||||||
|
r.EndTime = *wire.EndTime
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type runResultJSON struct {
|
||||||
|
RunID string `json:"run_id"`
|
||||||
|
Artifact Artifact `json:"artifact"`
|
||||||
|
RawOutput string `json:"raw_output"`
|
||||||
|
Validation ValidationResult `json:"validation"`
|
||||||
|
PromptID string `json:"prompt_id"`
|
||||||
|
PromptVersion string `json:"prompt_version,omitempty"`
|
||||||
|
PromptHash string `json:"prompt_hash,omitempty"`
|
||||||
|
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
||||||
|
SelectedProfileID string `json:"selected_profile_id"`
|
||||||
|
ModelName string `json:"model_name"`
|
||||||
|
Endpoint string `json:"endpoint"`
|
||||||
|
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
||||||
|
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
||||||
|
Usage TokenUsage `json:"usage"`
|
||||||
|
StartTime *time.Time `json:"start_time,omitempty"`
|
||||||
|
EndTime *time.Time `json:"end_time,omitempty"`
|
||||||
|
DurationMS *int64 `json:"duration_ms,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func valueOrZero(value *int64) int64 {
|
||||||
|
if value == nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return *value
|
||||||
|
}
|
||||||
@@ -16,6 +16,10 @@ import (
|
|||||||
// It does not register global state, maintain a model catalog, or resolve
|
// It does not register global state, maintain a model catalog, or resolve
|
||||||
// credentials. If APIKeyRequired is true, callers satisfy it with
|
// credentials. If APIKeyRequired is true, callers satisfy it with
|
||||||
// RunRequest.APIKey. Raw API keys do not belong in profiles.
|
// RunRequest.APIKey. Raw API keys do not belong in profiles.
|
||||||
|
//
|
||||||
|
// The function copies the ExtraParams map itself but does not recursively copy
|
||||||
|
// nested values. Validation and a deep copy occur when NewEngine applies a
|
||||||
|
// WithProfiles option containing the returned Profile.
|
||||||
func OpenAICompatibleProfile(cfg OpenAICompatibleProfileConfig) Profile {
|
func OpenAICompatibleProfile(cfg OpenAICompatibleProfileConfig) Profile {
|
||||||
return Profile{
|
return Profile{
|
||||||
ID: cfg.ID,
|
ID: cfg.ID,
|
||||||
|
|||||||
377
public_contract_test.go
Normal file
377
public_contract_test.go
Normal file
@@ -0,0 +1,377 @@
|
|||||||
|
package promptkit_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"testing/fstest"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestPreparedRunJSONOmitsZeroTimingValues(t *testing.T) {
|
||||||
|
payload, err := json.Marshal(promptkit.PreparedRun{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("marshal prepared run: %v", err)
|
||||||
|
}
|
||||||
|
for _, field := range []string{"start_time", "end_time", "duration_ms"} {
|
||||||
|
if strings.Contains(string(payload), `"`+field+`"`) {
|
||||||
|
t.Fatalf("expected zero %s to be omitted, got %s", field, payload)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPreparedRunJSONTimingRoundTrips(t *testing.T) {
|
||||||
|
start := time.Date(2026, time.July, 29, 12, 0, 0, 0, time.UTC)
|
||||||
|
prepared := promptkit.PreparedRun{
|
||||||
|
PromptID: "prompt",
|
||||||
|
StartTime: start,
|
||||||
|
EndTime: start.Add(1250 * time.Millisecond),
|
||||||
|
DurationMS: 1250,
|
||||||
|
}
|
||||||
|
|
||||||
|
payload, err := json.Marshal(prepared)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("marshal prepared run: %v", err)
|
||||||
|
}
|
||||||
|
var decoded promptkit.PreparedRun
|
||||||
|
if err := json.Unmarshal(payload, &decoded); err != nil {
|
||||||
|
t.Fatalf("unmarshal prepared run: %v", err)
|
||||||
|
}
|
||||||
|
if decoded.DurationMS != prepared.DurationMS ||
|
||||||
|
!decoded.StartTime.Equal(prepared.StartTime) ||
|
||||||
|
!decoded.EndTime.Equal(prepared.EndTime) {
|
||||||
|
t.Fatalf("timing values did not round trip: got %#v, want %#v", decoded, prepared)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunResultJSONUsesMillisecondsAndRoundTrips(t *testing.T) {
|
||||||
|
start := time.Date(2026, time.July, 29, 12, 0, 0, 0, time.UTC)
|
||||||
|
result := promptkit.RunResult{
|
||||||
|
RunID: "opaque-run-id",
|
||||||
|
Artifact: promptkit.Artifact{Name: "output", ContentType: "text/plain", Body: []byte("ok")},
|
||||||
|
StartTime: start,
|
||||||
|
EndTime: start.Add(1500 * time.Millisecond),
|
||||||
|
Duration: 1500 * time.Millisecond,
|
||||||
|
}
|
||||||
|
|
||||||
|
payload, err := json.Marshal(result)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("marshal run result: %v", err)
|
||||||
|
}
|
||||||
|
var object map[string]any
|
||||||
|
if err := json.Unmarshal(payload, &object); err != nil {
|
||||||
|
t.Fatalf("decode run result JSON: %v", err)
|
||||||
|
}
|
||||||
|
if got := object["duration_ms"]; got != float64(1500) {
|
||||||
|
t.Fatalf("expected duration_ms=1500, got %#v in %s", got, payload)
|
||||||
|
}
|
||||||
|
if _, exists := object["duration"]; exists {
|
||||||
|
t.Fatalf("unexpected nanosecond duration field in %s", payload)
|
||||||
|
}
|
||||||
|
artifact, ok := object["artifact"].(map[string]any)
|
||||||
|
if !ok || artifact["content_type"] != "text/plain" {
|
||||||
|
t.Fatalf("expected stable artifact JSON fields, got %#v", object["artifact"])
|
||||||
|
}
|
||||||
|
|
||||||
|
var decoded promptkit.RunResult
|
||||||
|
if err := json.Unmarshal(payload, &decoded); err != nil {
|
||||||
|
t.Fatalf("unmarshal run result: %v", err)
|
||||||
|
}
|
||||||
|
if decoded.Duration != result.Duration || !decoded.StartTime.Equal(result.StartTime) || !decoded.EndTime.Equal(result.EndTime) {
|
||||||
|
t.Fatalf("timing values did not round trip: got %#v, want %#v", decoded, result)
|
||||||
|
}
|
||||||
|
|
||||||
|
payload, err = json.Marshal(promptkit.RunResult{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("marshal zero run result: %v", err)
|
||||||
|
}
|
||||||
|
for _, field := range []string{"start_time", "end_time", "duration_ms"} {
|
||||||
|
if strings.Contains(string(payload), `"`+field+`"`) {
|
||||||
|
t.Fatalf("expected zero %s to be omitted, got %s", field, payload)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEngineValidationIsSinglePass(t *testing.T) {
|
||||||
|
client := &fakeLLMClient{
|
||||||
|
response: &promptkit.GenerateResponse{Content: "not-json"},
|
||||||
|
}
|
||||||
|
engine := newContractEngineWithOptions(t, frameworkSchemaDir, promptkit.WithLLMClient(client))
|
||||||
|
|
||||||
|
result, err := engine.Run(context.Background(), promptkit.RunRequest{
|
||||||
|
PromptID: frameworkMarkdownSummaryPromptID,
|
||||||
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
|
"transcript": promptkit.Inline("Rin opens the gate."),
|
||||||
|
"glossary": promptkit.Inline("gate: A guarded passage."),
|
||||||
|
},
|
||||||
|
Validation: &promptkit.OutputContract{
|
||||||
|
Format: promptkit.FormatJSON,
|
||||||
|
ValidationMode: promptkit.ValidationJSON,
|
||||||
|
RepairAttempts: 3,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("run with failed content validation: %v", err)
|
||||||
|
}
|
||||||
|
if result.Validation.Status != promptkit.ValidationFailed ||
|
||||||
|
result.Validation.RepairAttempts != 0 {
|
||||||
|
t.Fatalf("expected failed single-pass validation, got %#v", result.Validation)
|
||||||
|
}
|
||||||
|
if len(client.requests) != 1 {
|
||||||
|
t.Fatalf("expected one model generation, got %d", len(client.requests))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRepeatedOptionsUseLastValueInEachCategory(t *testing.T) {
|
||||||
|
profile := promptkit.Profile{ID: "profile", Endpoint: "http://example.test/v1", Model: "model"}
|
||||||
|
|
||||||
|
t.Run("prompt source", func(t *testing.T) {
|
||||||
|
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||||
|
promptkit.WithPromptFS(contractPromptFS("first", "profile", "first"), "."),
|
||||||
|
promptkit.WithPromptFS(contractPromptFS("second", "profile", "second"), "."),
|
||||||
|
promptkit.WithProfiles(profile),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("construct engine: %v", err)
|
||||||
|
}
|
||||||
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "second"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("prepare from last prompt source: %v", err)
|
||||||
|
}
|
||||||
|
if prepared.Messages[0].Content != "second" {
|
||||||
|
t.Fatalf("expected last prompt source, got %#v", prepared.Messages)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("profile source", func(t *testing.T) {
|
||||||
|
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||||
|
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
|
||||||
|
promptkit.WithProfileFS(contractProfileFS("profile", "first-model"), "."),
|
||||||
|
promptkit.WithProfileFS(contractProfileFS("profile", "second-model"), "."),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("construct engine: %v", err)
|
||||||
|
}
|
||||||
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("prepare from last profile source: %v", err)
|
||||||
|
}
|
||||||
|
if prepared.EffectiveModelParams.Model != "second-model" {
|
||||||
|
t.Fatalf("expected last profile source, got %q", prepared.EffectiveModelParams.Model)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("in-memory profiles", func(t *testing.T) {
|
||||||
|
first := profile
|
||||||
|
first.Model = "first-model"
|
||||||
|
second := profile
|
||||||
|
second.Model = "second-model"
|
||||||
|
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||||
|
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
|
||||||
|
promptkit.WithProfiles(first),
|
||||||
|
promptkit.WithProfiles(second),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("construct engine: %v", err)
|
||||||
|
}
|
||||||
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("prepare from last in-memory profile option: %v", err)
|
||||||
|
}
|
||||||
|
if prepared.EffectiveModelParams.Model != "second-model" {
|
||||||
|
t.Fatalf("expected last in-memory profiles, got %q", prepared.EffectiveModelParams.Model)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("schema source", func(t *testing.T) {
|
||||||
|
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||||
|
promptkit.WithPromptFS(contractSchemaPromptFS(), "."),
|
||||||
|
promptkit.WithProfiles(profile),
|
||||||
|
promptkit.WithSchemaFS(contractSchemaFS("first"), "."),
|
||||||
|
promptkit.WithSchemaFS(contractSchemaFS("second"), "."),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("construct engine: %v", err)
|
||||||
|
}
|
||||||
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "schema-prompt"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("prepare from last schema source: %v", err)
|
||||||
|
}
|
||||||
|
schema := prepared.StructuredOutput.JSONSchema.Schema.(map[string]any)
|
||||||
|
if schema["title"] != "second" {
|
||||||
|
t.Fatalf("expected last schema source, got %#v", schema)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("model client", func(t *testing.T) {
|
||||||
|
var firstCalls, secondCalls atomic.Int64
|
||||||
|
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||||
|
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
|
||||||
|
promptkit.WithProfiles(profile),
|
||||||
|
promptkit.WithLLMClient(countingLLMClient{calls: &firstCalls}),
|
||||||
|
promptkit.WithLLMClient(countingLLMClient{calls: &secondCalls}),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("construct engine: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := engine.Run(context.Background(), promptkit.RunRequest{PromptID: "prompt"}); err != nil {
|
||||||
|
t.Fatalf("run with last model client: %v", err)
|
||||||
|
}
|
||||||
|
if firstCalls.Load() != 0 || secondCalls.Load() != 1 {
|
||||||
|
t.Fatalf("expected only last client call, got first=%d second=%d", firstCalls.Load(), secondCalls.Load())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("artifact reader", func(t *testing.T) {
|
||||||
|
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||||
|
promptkit.WithPromptFS(contractInputPromptFS(), "."),
|
||||||
|
promptkit.WithProfiles(profile),
|
||||||
|
promptkit.WithArtifactReader(fixedArtifactReader("first")),
|
||||||
|
promptkit.WithArtifactReader(fixedArtifactReader("second")),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("construct engine: %v", err)
|
||||||
|
}
|
||||||
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
|
PromptID: "input-prompt",
|
||||||
|
Inputs: map[string]promptkit.ArtifactRef{"input": promptkit.Inline("ignored")},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("prepare with last artifact reader: %v", err)
|
||||||
|
}
|
||||||
|
if prepared.Messages[0].Content != "second" {
|
||||||
|
t.Fatalf("expected last artifact reader, got %#v", prepared.Messages)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEngineSupportsConcurrentPrepareAndRun(t *testing.T) {
|
||||||
|
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||||
|
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
|
||||||
|
promptkit.WithProfiles(promptkit.Profile{
|
||||||
|
ID: "profile",
|
||||||
|
Endpoint: "http://example.test/v1",
|
||||||
|
Model: "model",
|
||||||
|
}),
|
||||||
|
promptkit.WithLLMClient(countingLLMClient{}),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("construct engine: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
const calls = 40
|
||||||
|
errs := make(chan error, calls)
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := 0; i < calls; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func(run bool) {
|
||||||
|
defer wg.Done()
|
||||||
|
request := promptkit.RunRequest{PromptID: "prompt"}
|
||||||
|
if run {
|
||||||
|
_, err := engine.Run(context.Background(), request)
|
||||||
|
errs <- err
|
||||||
|
return
|
||||||
|
}
|
||||||
|
_, err := engine.Prepare(context.Background(), request)
|
||||||
|
errs <- err
|
||||||
|
}(i%2 == 0)
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
close(errs)
|
||||||
|
for err := range errs {
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("concurrent call failed: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type countingLLMClient struct {
|
||||||
|
calls *atomic.Int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c countingLLMClient) Generate(context.Context, promptkit.GenerateRequest) (*promptkit.GenerateResponse, error) {
|
||||||
|
if c.calls != nil {
|
||||||
|
c.calls.Add(1)
|
||||||
|
}
|
||||||
|
return &promptkit.GenerateResponse{Content: "ok"}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type fixedArtifactReader string
|
||||||
|
|
||||||
|
func (r fixedArtifactReader) Read(context.Context, promptkit.ArtifactRef) (*promptkit.Artifact, error) {
|
||||||
|
return &promptkit.Artifact{Body: []byte(r)}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func contractPromptFS(id, profileID, message string) fstest.MapFS {
|
||||||
|
return fstest.MapFS{
|
||||||
|
"prompt.yaml": &fstest.MapFile{Data: []byte(fmt.Sprintf(`id: %s
|
||||||
|
version: "1"
|
||||||
|
default_profile: %s
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: %q
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`, id, profileID, message))},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func contractInputPromptFS() fstest.MapFS {
|
||||||
|
return fstest.MapFS{
|
||||||
|
"prompt.yaml": &fstest.MapFile{Data: []byte(`id: input-prompt
|
||||||
|
version: "1"
|
||||||
|
default_profile: profile
|
||||||
|
inputs:
|
||||||
|
- name: input
|
||||||
|
required: true
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: '{{input "input"}}'
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`)},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func contractProfileFS(id, model string) fstest.MapFS {
|
||||||
|
return fstest.MapFS{
|
||||||
|
"profile.yaml": &fstest.MapFile{Data: []byte(fmt.Sprintf(`id: %s
|
||||||
|
endpoint: http://example.test/v1
|
||||||
|
model: %s
|
||||||
|
`, id, model))},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func contractSchemaPromptFS() fstest.MapFS {
|
||||||
|
return fstest.MapFS{
|
||||||
|
"prompt.yaml": &fstest.MapFile{Data: []byte(`id: schema-prompt
|
||||||
|
version: "1"
|
||||||
|
default_profile: profile
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: message
|
||||||
|
output:
|
||||||
|
format: json
|
||||||
|
validation_mode: json_schema
|
||||||
|
schema_path: schema.json
|
||||||
|
`)},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func contractSchemaFS(title string) fstest.MapFS {
|
||||||
|
return fstest.MapFS{
|
||||||
|
"schema.json": &fstest.MapFile{Data: []byte(fmt.Sprintf(`{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"title": %q,
|
||||||
|
"type": "object"
|
||||||
|
}`, title))},
|
||||||
|
}
|
||||||
|
}
|
||||||
367
types.go
367
types.go
@@ -5,160 +5,318 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ArtifactRefType defines how an artifact is referenced.
|
// ArtifactRefType identifies how an [ArtifactRef] supplies content.
|
||||||
type ArtifactRefType string
|
type ArtifactRefType string
|
||||||
|
|
||||||
const (
|
const (
|
||||||
|
// ArtifactRefInline selects ArtifactRef.Body as the content.
|
||||||
ArtifactRefInline ArtifactRefType = "inline"
|
ArtifactRefInline ArtifactRefType = "inline"
|
||||||
|
// ArtifactRefFile selects the filesystem path in ArtifactRef.URI.
|
||||||
ArtifactRefFile ArtifactRefType = "file"
|
ArtifactRefFile ArtifactRefType = "file"
|
||||||
)
|
)
|
||||||
|
|
||||||
// OutputFormat defines the desired output format.
|
// OutputFormat identifies the media format of generated output.
|
||||||
|
// OutputFormat has a stable JSON string representation.
|
||||||
type OutputFormat string
|
type OutputFormat string
|
||||||
|
|
||||||
const (
|
const (
|
||||||
|
// FormatText identifies plain-text output.
|
||||||
FormatText OutputFormat = "text"
|
FormatText OutputFormat = "text"
|
||||||
|
// FormatMarkdown identifies Markdown output.
|
||||||
FormatMarkdown OutputFormat = "markdown"
|
FormatMarkdown OutputFormat = "markdown"
|
||||||
|
// FormatJSON identifies JSON output.
|
||||||
FormatJSON OutputFormat = "json"
|
FormatJSON OutputFormat = "json"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ValidationMode defines the output validation strategy.
|
// ValidationMode identifies how generated output is checked.
|
||||||
|
// ValidationMode has a stable JSON string representation.
|
||||||
type ValidationMode string
|
type ValidationMode string
|
||||||
|
|
||||||
const (
|
const (
|
||||||
|
// ValidationNone skips content validation.
|
||||||
ValidationNone ValidationMode = "none"
|
ValidationNone ValidationMode = "none"
|
||||||
|
// ValidationBasic requires non-empty output.
|
||||||
ValidationBasic ValidationMode = "basic"
|
ValidationBasic ValidationMode = "basic"
|
||||||
|
// ValidationJSON requires syntactically valid JSON.
|
||||||
ValidationJSON ValidationMode = "json"
|
ValidationJSON ValidationMode = "json"
|
||||||
|
// ValidationJSONSchema requires JSON that satisfies OutputContract.SchemaPath.
|
||||||
ValidationJSONSchema ValidationMode = "json_schema"
|
ValidationJSONSchema ValidationMode = "json_schema"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ValidationStatus defines the result of a validation check.
|
// ValidationStatus identifies the completed state of an output check.
|
||||||
|
// ValidationStatus has a stable JSON string representation.
|
||||||
type ValidationStatus string
|
type ValidationStatus string
|
||||||
|
|
||||||
const (
|
const (
|
||||||
|
// ValidationPassed means the generated output satisfied its contract.
|
||||||
ValidationPassed ValidationStatus = "passed"
|
ValidationPassed ValidationStatus = "passed"
|
||||||
|
// ValidationFailed means validation completed and rejected the generated
|
||||||
|
// output. Engine.Run returns this status in a result, not as an error.
|
||||||
ValidationFailed ValidationStatus = "failed"
|
ValidationFailed ValidationStatus = "failed"
|
||||||
|
// ValidationSkipped means ValidationNone selected no content check.
|
||||||
ValidationSkipped ValidationStatus = "skipped"
|
ValidationSkipped ValidationStatus = "skipped"
|
||||||
)
|
)
|
||||||
|
|
||||||
// CacheControlType defines provider cache behavior for prompt content.
|
// CacheControlType identifies provider cache behavior for prompt content.
|
||||||
|
// CacheControlType has a stable JSON string representation.
|
||||||
type CacheControlType string
|
type CacheControlType string
|
||||||
|
|
||||||
const (
|
const (
|
||||||
|
// CacheControlEphemeral requests provider-defined ephemeral caching.
|
||||||
CacheControlEphemeral CacheControlType = "ephemeral"
|
CacheControlEphemeral CacheControlType = "ephemeral"
|
||||||
)
|
)
|
||||||
|
|
||||||
// StructuredOutputType identifies provider-level structured output modes.
|
// StructuredOutputType identifies provider-level structured output modes.
|
||||||
|
// StructuredOutputType has a stable JSON string representation.
|
||||||
type StructuredOutputType string
|
type StructuredOutputType string
|
||||||
|
|
||||||
const (
|
const (
|
||||||
|
// StructuredOutputJSONSchema supplies JSON Schema response constraints.
|
||||||
StructuredOutputJSONSchema StructuredOutputType = "json_schema"
|
StructuredOutputJSONSchema StructuredOutputType = "json_schema"
|
||||||
)
|
)
|
||||||
|
|
||||||
// RunRequest represents a request to prepare or run a single prompt.
|
// RunRequest selects one prompt execution. It has no stable JSON
|
||||||
|
// representation.
|
||||||
|
//
|
||||||
|
// Prepare and Run copy the request's maps, pointers, and nested
|
||||||
|
// JSON-compatible values before using them. The caller may mutate the request
|
||||||
|
// after either method returns.
|
||||||
type RunRequest struct {
|
type RunRequest struct {
|
||||||
|
// PromptID is the required non-empty prompt identifier.
|
||||||
PromptID string
|
PromptID string
|
||||||
|
// PromptVersion optionally selects one version of PromptID. When empty, the
|
||||||
|
// prompt source must contain exactly one matching version.
|
||||||
PromptVersion string
|
PromptVersion string
|
||||||
|
// ProfileID selects an execution profile. When empty, the prompt's default
|
||||||
|
// profile is used; if both are empty, the error matches ErrProfileRequired
|
||||||
|
// and ErrInvalidRequest.
|
||||||
ProfileID string
|
ProfileID string
|
||||||
|
// APIKey is a request-scoped direct credential. It takes precedence over
|
||||||
|
// APIKeyEnv, is passed to the selected LLMClient, and is never included in
|
||||||
|
// prepared values, results, hashes, JSON, String, or GoString output.
|
||||||
APIKey string `json:"-"`
|
APIKey string `json:"-"`
|
||||||
|
// Inputs maps prompt input names to references. A nil or empty map is valid
|
||||||
|
// only when the selected prompt and its templates require no inputs.
|
||||||
Inputs map[string]ArtifactRef
|
Inputs map[string]ArtifactRef
|
||||||
|
// Vars supplies Go-template data for messages and the session ID. Nil and
|
||||||
|
// empty maps are equivalent.
|
||||||
Vars map[string]string
|
Vars map[string]string
|
||||||
|
// Execution optionally overrides individual profile execution settings.
|
||||||
|
// Nil uses the selected profile over framework defaults.
|
||||||
Execution *ExecutionTargetOverride
|
Execution *ExecutionTargetOverride
|
||||||
|
// Validation optionally replaces the prompt's complete output contract. It
|
||||||
|
// does not merge individual fields. Nil uses the prompt contract.
|
||||||
Validation *OutputContract
|
Validation *OutputContract
|
||||||
Metadata map[string]string
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// PreparedRun contains prepared prompt execution state. It does not include
|
// PreparedRun contains prepared prompt execution state. It does not include
|
||||||
// resolved API key values, model output, validation results, or internal target
|
// resolved API key values, model output, validation results, or internal target
|
||||||
// presence metadata.
|
// presence metadata. PreparedRun has a stable JSON representation.
|
||||||
|
//
|
||||||
|
// All maps, slices, pointers, and schema values are caller-owned copies. JSON
|
||||||
|
// timestamps use RFC 3339 and zero timing values are omitted. Hash formats are
|
||||||
|
// opaque.
|
||||||
type PreparedRun struct {
|
type PreparedRun struct {
|
||||||
|
// PromptID is the selected prompt identifier.
|
||||||
PromptID string `json:"prompt_id"`
|
PromptID string `json:"prompt_id"`
|
||||||
|
// PromptVersion is the selected prompt version.
|
||||||
PromptVersion string `json:"prompt_version,omitempty"`
|
PromptVersion string `json:"prompt_version,omitempty"`
|
||||||
|
// PromptHash is an opaque equality value for the selected definition.
|
||||||
PromptHash string `json:"prompt_hash,omitempty"`
|
PromptHash string `json:"prompt_hash,omitempty"`
|
||||||
|
// SelectedProfileID is the explicit request profile or prompt default that
|
||||||
|
// supplied execution settings.
|
||||||
SelectedProfileID string `json:"selected_profile_id"`
|
SelectedProfileID string `json:"selected_profile_id"`
|
||||||
|
// EffectiveModelParams contains framework defaults overlaid by the selected
|
||||||
|
// profile and then request overrides. It excludes resolved API-key values.
|
||||||
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
||||||
|
// OutputContract is the complete effective output contract.
|
||||||
OutputContract OutputContract `json:"output_contract"`
|
OutputContract OutputContract `json:"output_contract"`
|
||||||
|
// StructuredOutput is non-nil for JSON Schema validation and contains the
|
||||||
|
// provider-facing response constraint passed to an LLM client.
|
||||||
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
|
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
|
||||||
|
// InputHashes maps every supplied input name to its opaque artifact hash.
|
||||||
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
||||||
|
// SessionID is the trimmed rendered session identifier, if any.
|
||||||
SessionID string `json:"session_id,omitempty"`
|
SessionID string `json:"session_id,omitempty"`
|
||||||
|
// RenderedPromptHash is an opaque equality value for SessionID and Messages.
|
||||||
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
||||||
|
// Messages are the rendered messages that Run passes to the LLM client.
|
||||||
Messages []RenderedMessage `json:"messages"`
|
Messages []RenderedMessage `json:"messages"`
|
||||||
|
// StartTime is the UTC time at which preparation began.
|
||||||
StartTime time.Time `json:"start_time,omitempty"`
|
StartTime time.Time `json:"start_time,omitempty"`
|
||||||
|
// EndTime is the UTC time at which preparation completed.
|
||||||
EndTime time.Time `json:"end_time,omitempty"`
|
EndTime time.Time `json:"end_time,omitempty"`
|
||||||
|
// DurationMS is preparation elapsed time in integer milliseconds. JSON uses
|
||||||
|
// duration_ms and omits a zero value.
|
||||||
DurationMS int64 `json:"duration_ms,omitempty"`
|
DurationMS int64 `json:"duration_ms,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// RunResult contains generated output, validation state, and run metadata.
|
// RunResult contains generated output, validation state, and run metadata.
|
||||||
|
// RunResult has a stable JSON representation and round-trips its Duration
|
||||||
|
// through the duration_ms JSON field.
|
||||||
|
//
|
||||||
|
// All maps, slices, and nested values are caller-owned copies. JSON timestamps
|
||||||
|
// use RFC 3339 and zero timing values are omitted. Run IDs and hash formats are
|
||||||
|
// opaque.
|
||||||
type RunResult struct {
|
type RunResult struct {
|
||||||
|
// RunID is an opaque identifier for this invocation.
|
||||||
RunID string `json:"run_id"`
|
RunID string `json:"run_id"`
|
||||||
|
// Artifact contains the generated output and derived metadata.
|
||||||
Artifact Artifact `json:"artifact"`
|
Artifact Artifact `json:"artifact"`
|
||||||
|
// RawOutput is the exact generated content before artifact classification
|
||||||
|
// and validation.
|
||||||
RawOutput string `json:"raw_output"`
|
RawOutput string `json:"raw_output"`
|
||||||
|
// Validation records the completed content check.
|
||||||
Validation ValidationResult `json:"validation"`
|
Validation ValidationResult `json:"validation"`
|
||||||
|
// PromptID is the selected prompt identifier.
|
||||||
PromptID string `json:"prompt_id"`
|
PromptID string `json:"prompt_id"`
|
||||||
|
// PromptVersion is the selected prompt version.
|
||||||
PromptVersion string `json:"prompt_version,omitempty"`
|
PromptVersion string `json:"prompt_version,omitempty"`
|
||||||
|
// PromptHash is the same opaque definition equality value exposed by
|
||||||
|
// PreparedRun.
|
||||||
PromptHash string `json:"prompt_hash,omitempty"`
|
PromptHash string `json:"prompt_hash,omitempty"`
|
||||||
|
// RenderedPromptHash is the same opaque rendered-prompt equality value
|
||||||
|
// computed during preparation.
|
||||||
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
||||||
|
// SelectedProfileID identifies the profile used for execution.
|
||||||
SelectedProfileID string `json:"selected_profile_id"`
|
SelectedProfileID string `json:"selected_profile_id"`
|
||||||
|
// ModelName is the effective model name and equals
|
||||||
|
// EffectiveModelParams.Model.
|
||||||
ModelName string `json:"model_name"`
|
ModelName string `json:"model_name"`
|
||||||
|
// Endpoint is the effective base endpoint and equals
|
||||||
|
// EffectiveModelParams.Endpoint.
|
||||||
Endpoint string `json:"endpoint"`
|
Endpoint string `json:"endpoint"`
|
||||||
|
// EffectiveModelParams contains the settings supplied to the LLM client,
|
||||||
|
// excluding resolved API-key values.
|
||||||
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
||||||
|
// InputHashes are the opaque input equality values computed during
|
||||||
|
// preparation.
|
||||||
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
||||||
|
// Usage is the token accounting reported by the LLM client.
|
||||||
Usage TokenUsage `json:"usage"`
|
Usage TokenUsage `json:"usage"`
|
||||||
|
// StartTime is the UTC time immediately before preparation begins.
|
||||||
StartTime time.Time `json:"start_time,omitempty"`
|
StartTime time.Time `json:"start_time,omitempty"`
|
||||||
|
// EndTime is the UTC time after generation and validation complete.
|
||||||
EndTime time.Time `json:"end_time,omitempty"`
|
EndTime time.Time `json:"end_time,omitempty"`
|
||||||
Duration time.Duration `json:"duration,omitempty"`
|
// Duration covers preparation, generation, and validation. JSON represents
|
||||||
|
// it as integer milliseconds in duration_ms and omits a zero value.
|
||||||
|
Duration time.Duration `json:"-"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// ArtifactRef represents a reference to prompt input content.
|
// ArtifactRef identifies prompt input content. It has no stable JSON
|
||||||
|
// representation. Prefer [File], [Inline], or [InlineWithURI] to construct one.
|
||||||
type ArtifactRef struct {
|
type ArtifactRef struct {
|
||||||
|
// Type must be ArtifactRefInline or ArtifactRefFile.
|
||||||
Type ArtifactRefType
|
Type ArtifactRefType
|
||||||
|
// URI is the file path for ArtifactRefFile and optional provenance metadata
|
||||||
|
// for ArtifactRefInline.
|
||||||
URI string
|
URI string
|
||||||
|
// Body is the content for ArtifactRefInline and is ignored for
|
||||||
|
// ArtifactRefFile.
|
||||||
Body string
|
Body string
|
||||||
}
|
}
|
||||||
|
|
||||||
// Artifact represents loaded artifact content.
|
// Artifact represents loaded or generated content and has a stable JSON
|
||||||
|
// representation. Body uses encoding/json's base64 representation for []byte.
|
||||||
type Artifact struct {
|
type Artifact struct {
|
||||||
Name string
|
// Name is artifact metadata. During input preparation the engine fills an
|
||||||
ContentType string
|
// empty reader-supplied name with the request input-map key.
|
||||||
Body []byte
|
Name string `json:"name"`
|
||||||
URI string
|
// ContentType is the media type reported by the reader or derived for
|
||||||
Size int64
|
// generated output.
|
||||||
Hash string
|
ContentType string `json:"content_type"`
|
||||||
|
// Body is the artifact content. Engine boundaries copy this slice.
|
||||||
|
Body []byte `json:"body"`
|
||||||
|
// URI is optional source or result provenance metadata.
|
||||||
|
URI string `json:"uri"`
|
||||||
|
// Size is content-size metadata in bytes.
|
||||||
|
Size int64 `json:"size"`
|
||||||
|
// Hash is an opaque content equality value when the producing reader
|
||||||
|
// supplies one. Its format and algorithm are not API contracts.
|
||||||
|
Hash string `json:"hash"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// ArtifactReader resolves a prompt input reference into its content.
|
// ArtifactReader resolves a prompt input reference into its content.
|
||||||
//
|
//
|
||||||
// Readers are responsible for supplying artifact metadata. The engine assigns
|
// Read may be called concurrently. It must honor ctx cancellation to make
|
||||||
// an input-map name only when the returned artifact name is empty.
|
// Prepare and Run responsive to cancellation. The engine passes a copied ref
|
||||||
|
// and immediately copies the returned Artifact.Body; it does not retain either
|
||||||
|
// value. Readers supply artifact metadata, and the engine assigns an input-map
|
||||||
|
// name only when the returned artifact name is empty.
|
||||||
|
//
|
||||||
|
// An injected reader owns any application-specific path containment,
|
||||||
|
// authorization, content-size, and content-type policy. It must protect
|
||||||
|
// sensitive references and bodies in its logging and in any copies it retains.
|
||||||
|
// It may reuse or mutate the returned artifact and body after Read returns.
|
||||||
|
//
|
||||||
|
// Returning a non-nil error makes the engine return an error matching
|
||||||
|
// ErrArtifactLoad while preserving the reader error through errors.Is.
|
||||||
|
// Returning a nil artifact with a nil error also produces ErrArtifactLoad.
|
||||||
type ArtifactReader interface {
|
type ArtifactReader interface {
|
||||||
Read(context.Context, ArtifactRef) (*Artifact, error)
|
Read(context.Context, ArtifactRef) (*Artifact, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
// ExecutionTarget represents effective model runtime settings.
|
// ExecutionTarget represents effective model runtime settings and has a stable
|
||||||
|
// JSON representation. It never exposes a resolved API-key value.
|
||||||
type ExecutionTarget struct {
|
type ExecutionTarget struct {
|
||||||
|
// Endpoint is the model-provider base URL.
|
||||||
Endpoint string `json:"endpoint"`
|
Endpoint string `json:"endpoint"`
|
||||||
|
// Model is the provider model identifier.
|
||||||
Model string `json:"model"`
|
Model string `json:"model"`
|
||||||
|
// Temperature is the effective sampling temperature from 0 through 2.
|
||||||
Temperature float64 `json:"temperature"`
|
Temperature float64 `json:"temperature"`
|
||||||
|
// MaxTokens is the non-negative effective output-token limit. Zero leaves
|
||||||
|
// the limit unspecified to compatible providers unless it was an explicit
|
||||||
|
// request override.
|
||||||
MaxTokens int `json:"max_tokens"`
|
MaxTokens int `json:"max_tokens"`
|
||||||
|
// TopP is the effective nucleus-sampling value from 0 through 1.
|
||||||
TopP float64 `json:"top_p"`
|
TopP float64 `json:"top_p"`
|
||||||
|
// TimeoutSeconds is the non-negative per-generation deadline. Zero disables
|
||||||
|
// this deadline without disabling caller cancellation or the transport cap.
|
||||||
TimeoutSeconds int `json:"timeout_seconds"`
|
TimeoutSeconds int `json:"timeout_seconds"`
|
||||||
|
// ServiceTier is an optional provider-specific request tier.
|
||||||
ServiceTier string `json:"service_tier"`
|
ServiceTier string `json:"service_tier"`
|
||||||
|
// ReasoningEffort is an optional provider-specific reasoning setting.
|
||||||
ReasoningEffort string `json:"reasoning_effort"`
|
ReasoningEffort string `json:"reasoning_effort"`
|
||||||
|
// APIKeyEnv is an environment-variable name, not its credential value.
|
||||||
APIKeyEnv string `json:"api_key_env"`
|
APIKeyEnv string `json:"api_key_env"`
|
||||||
|
// ExtraParams contains copied JSON-compatible provider parameters.
|
||||||
ExtraParams map[string]any `json:"extra_params"`
|
ExtraParams map[string]any `json:"extra_params"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// ExecutionTargetOverride represents per-request runtime setting overrides.
|
// ExecutionTargetOverride represents per-request runtime setting overrides and
|
||||||
|
// has no stable JSON representation.
|
||||||
|
//
|
||||||
|
// Non-empty string fields replace profile values. Non-nil numeric pointers
|
||||||
|
// replace profile values and preserve explicit zero. A non-empty ExtraParams
|
||||||
|
// map replaces the complete profile map rather than merging keys. Empty string
|
||||||
|
// fields, nil pointers, and a nil or empty ExtraParams map inherit the selected
|
||||||
|
// profile over framework defaults.
|
||||||
type ExecutionTargetOverride struct {
|
type ExecutionTargetOverride struct {
|
||||||
|
// Endpoint replaces the profile endpoint when non-empty.
|
||||||
Endpoint string
|
Endpoint string
|
||||||
|
// Model replaces the profile model when non-empty.
|
||||||
Model string
|
Model string
|
||||||
|
// Temperature, when non-nil, must point to a value from 0 through 2.
|
||||||
Temperature *float64
|
Temperature *float64
|
||||||
|
// MaxTokens, when non-nil, must point to a non-negative value.
|
||||||
MaxTokens *int
|
MaxTokens *int
|
||||||
|
// TopP, when non-nil, must point to a value from 0 through 1.
|
||||||
TopP *float64
|
TopP *float64
|
||||||
|
// TimeoutSeconds, when non-nil, must point to a non-negative value. A
|
||||||
|
// pointed-to zero disables the per-generation deadline.
|
||||||
TimeoutSeconds *int
|
TimeoutSeconds *int
|
||||||
|
// ServiceTier replaces the profile value when non-blank.
|
||||||
ServiceTier string
|
ServiceTier string
|
||||||
|
// ReasoningEffort replaces the profile value when non-blank. An empty value
|
||||||
|
// cannot clear a profile setting.
|
||||||
ReasoningEffort string
|
ReasoningEffort string
|
||||||
|
// APIKeyEnv replaces the profile environment-variable name when non-blank.
|
||||||
|
// A direct RunRequest.APIKey still takes precedence over environment lookup.
|
||||||
APIKeyEnv string
|
APIKeyEnv string
|
||||||
|
// ExtraParams, when non-empty, replaces the profile map. Values must be
|
||||||
|
// JSON-compatible: nil, booleans, finite numbers, strings, arrays or slices,
|
||||||
|
// and maps with non-empty string keys. Cycles are invalid.
|
||||||
ExtraParams map[string]any
|
ExtraParams map[string]any
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -167,18 +325,38 @@ type ExecutionTargetOverride struct {
|
|||||||
// It is equivalent to a loaded profile file after validation. Raw API keys do
|
// It is equivalent to a loaded profile file after validation. Raw API keys do
|
||||||
// not belong in profiles; use APIKeyRequired to require callers to provide
|
// not belong in profiles; use APIKeyRequired to require callers to provide
|
||||||
// RunRequest.APIKey for each request, or use profile YAML api_key_env with file
|
// RunRequest.APIKey for each request, or use profile YAML api_key_env with file
|
||||||
// and FS profile sources.
|
// and FS profile sources. Profile has no stable JSON representation.
|
||||||
|
//
|
||||||
|
// WithProfiles validates and copies Profile values during NewEngine. Numeric
|
||||||
|
// zero, blank strings, and an empty ExtraParams map inherit framework defaults;
|
||||||
|
// use ExecutionTargetOverride pointer fields to request explicit numeric zero.
|
||||||
type Profile struct {
|
type Profile struct {
|
||||||
|
// ID is the required non-blank profile identifier. WithProfiles trims it.
|
||||||
ID string
|
ID string
|
||||||
|
// Endpoint is the required non-blank model-provider base URL.
|
||||||
Endpoint string
|
Endpoint string
|
||||||
|
// Model is the required non-blank provider model identifier.
|
||||||
Model string
|
Model string
|
||||||
|
// Temperature is from 0 through 2. Zero inherits the framework default.
|
||||||
Temperature float64
|
Temperature float64
|
||||||
|
// MaxTokens is non-negative. Zero inherits the framework default.
|
||||||
MaxTokens int
|
MaxTokens int
|
||||||
|
// TopP is from 0 through 1. Zero inherits the framework default rather than
|
||||||
|
// selecting an explicit zero.
|
||||||
TopP float64
|
TopP float64
|
||||||
|
// TimeoutSeconds is non-negative. Zero inherits the framework default.
|
||||||
TimeoutSeconds int
|
TimeoutSeconds int
|
||||||
|
// ServiceTier is optional; a blank value inherits the framework default.
|
||||||
ServiceTier string
|
ServiceTier string
|
||||||
|
// ReasoningEffort is optional; a blank value inherits the framework
|
||||||
|
// default.
|
||||||
ReasoningEffort string
|
ReasoningEffort string
|
||||||
|
// APIKeyRequired requires a non-blank RunRequest.APIKey. It does not store a
|
||||||
|
// credential or enable environment lookup.
|
||||||
APIKeyRequired bool
|
APIKeyRequired bool
|
||||||
|
// ExtraParams contains provider-specific JSON-compatible values. An empty
|
||||||
|
// map inherits framework defaults. WithProfiles validates and deeply copies
|
||||||
|
// it during NewEngine.
|
||||||
ExtraParams map[string]any
|
ExtraParams map[string]any
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -187,120 +365,225 @@ type Profile struct {
|
|||||||
//
|
//
|
||||||
// It contains ordinary profile fields for OpenAI-compatible chat-completions
|
// It contains ordinary profile fields for OpenAI-compatible chat-completions
|
||||||
// endpoints. APIKeyRequired is satisfied by RunRequest.APIKey. Raw API keys do
|
// endpoints. APIKeyRequired is satisfied by RunRequest.APIKey. Raw API keys do
|
||||||
// not belong in this config.
|
// not belong in this config. OpenAICompatibleProfileConfig has no stable JSON
|
||||||
|
// representation and is not validated until its resulting Profile is supplied
|
||||||
|
// through WithProfiles to NewEngine.
|
||||||
type OpenAICompatibleProfileConfig struct {
|
type OpenAICompatibleProfileConfig struct {
|
||||||
|
// ID becomes Profile.ID.
|
||||||
ID string
|
ID string
|
||||||
|
// Endpoint becomes Profile.Endpoint.
|
||||||
Endpoint string
|
Endpoint string
|
||||||
|
// Model becomes Profile.Model.
|
||||||
Model string
|
Model string
|
||||||
|
// APIKeyRequired becomes Profile.APIKeyRequired.
|
||||||
APIKeyRequired bool
|
APIKeyRequired bool
|
||||||
|
// Temperature becomes Profile.Temperature.
|
||||||
Temperature float64
|
Temperature float64
|
||||||
|
// MaxTokens becomes Profile.MaxTokens.
|
||||||
MaxTokens int
|
MaxTokens int
|
||||||
|
// TopP becomes Profile.TopP.
|
||||||
TopP float64
|
TopP float64
|
||||||
|
// TimeoutSeconds becomes Profile.TimeoutSeconds.
|
||||||
TimeoutSeconds int
|
TimeoutSeconds int
|
||||||
|
// ServiceTier becomes Profile.ServiceTier.
|
||||||
ServiceTier string
|
ServiceTier string
|
||||||
|
// ReasoningEffort becomes Profile.ReasoningEffort.
|
||||||
ReasoningEffort string
|
ReasoningEffort string
|
||||||
|
// ExtraParams becomes a shallow-copied Profile.ExtraParams map. NewEngine
|
||||||
|
// performs validation and a deep copy when WithProfiles applies the result.
|
||||||
ExtraParams map[string]any
|
ExtraParams map[string]any
|
||||||
}
|
}
|
||||||
|
|
||||||
// ExecutionTargetPresence tracks which numeric runtime settings were explicit
|
// ExecutionTargetPresence tracks which numeric runtime settings were explicit
|
||||||
// request overrides.
|
// request overrides, including explicit zero values. It has a stable JSON
|
||||||
|
// representation and is supplied to injected LLM clients so they can preserve
|
||||||
|
// omission semantics.
|
||||||
type ExecutionTargetPresence struct {
|
type ExecutionTargetPresence struct {
|
||||||
Temperature bool
|
// Temperature reports a non-nil ExecutionTargetOverride.Temperature.
|
||||||
MaxTokens bool
|
Temperature bool `json:"temperature"`
|
||||||
TopP bool
|
// MaxTokens reports a non-nil ExecutionTargetOverride.MaxTokens.
|
||||||
TimeoutSeconds bool
|
MaxTokens bool `json:"max_tokens"`
|
||||||
|
// TopP reports a non-nil ExecutionTargetOverride.TopP.
|
||||||
|
TopP bool `json:"top_p"`
|
||||||
|
// TimeoutSeconds reports a non-nil ExecutionTargetOverride.TimeoutSeconds.
|
||||||
|
TimeoutSeconds bool `json:"timeout_seconds"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// OutputContract defines output and validation requirements.
|
// OutputContract defines output and validation requirements and has a stable
|
||||||
|
// JSON representation.
|
||||||
|
//
|
||||||
|
// A non-nil RunRequest.Validation replaces the complete prompt contract. It
|
||||||
|
// does not merge fields. The public Engine validates generated output once and
|
||||||
|
// does not install an output repairer.
|
||||||
type OutputContract struct {
|
type OutputContract struct {
|
||||||
|
// Format selects generated artifact metadata. An empty effective value
|
||||||
|
// defaults to FormatText.
|
||||||
Format OutputFormat `json:"format"`
|
Format OutputFormat `json:"format"`
|
||||||
|
// ValidationMode selects the content check. Use one of the declared
|
||||||
|
// ValidationMode constants.
|
||||||
ValidationMode ValidationMode `json:"validation_mode"`
|
ValidationMode ValidationMode `json:"validation_mode"`
|
||||||
|
// SchemaPath is required when ValidationMode is ValidationJSONSchema and is
|
||||||
|
// ignored by other modes.
|
||||||
SchemaPath string `json:"schema_path"`
|
SchemaPath string `json:"schema_path"`
|
||||||
|
// RepairAttempts is a requested repair limit. A non-positive value requests
|
||||||
|
// no repairs. The public Engine performs no repairs even when this value is
|
||||||
|
// positive, so its runs report zero attempts used.
|
||||||
RepairAttempts int `json:"repair_attempts"`
|
RepairAttempts int `json:"repair_attempts"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// ValidationResult represents output validation state.
|
// ValidationResult represents a completed output check and has a stable JSON
|
||||||
|
// representation. An operational inability to perform validation is returned
|
||||||
|
// as ErrValidation instead of a ValidationResult.
|
||||||
type ValidationResult struct {
|
type ValidationResult struct {
|
||||||
|
// Status is Passed, Failed, or Skipped.
|
||||||
Status ValidationStatus `json:"status"`
|
Status ValidationStatus `json:"status"`
|
||||||
|
// Mode is the effective validation mode.
|
||||||
Mode ValidationMode `json:"mode"`
|
Mode ValidationMode `json:"mode"`
|
||||||
|
// Errors contains validation diagnostics when Status is ValidationFailed.
|
||||||
Errors []string `json:"errors,omitempty"`
|
Errors []string `json:"errors,omitempty"`
|
||||||
|
// SchemaPath is the effective schema path for JSON Schema validation.
|
||||||
SchemaPath string `json:"schema_path,omitempty"`
|
SchemaPath string `json:"schema_path,omitempty"`
|
||||||
|
// RepairAttempts is the number of repairs actually attempted. It is always
|
||||||
|
// zero for the public Engine.
|
||||||
RepairAttempts int `json:"repair_attempts"`
|
RepairAttempts int `json:"repair_attempts"`
|
||||||
|
// IsValid is true for ValidationPassed and ValidationSkipped and false for
|
||||||
|
// ValidationFailed.
|
||||||
IsValid bool `json:"is_valid"`
|
IsValid bool `json:"is_valid"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// TokenUsage tracks token consumption.
|
// TokenUsage contains model-client token accounting and has a stable JSON
|
||||||
|
// representation. Promptkit preserves values reported by the client and does
|
||||||
|
// not derive or reconcile them.
|
||||||
type TokenUsage struct {
|
type TokenUsage struct {
|
||||||
|
// PromptTokens is the reported input-token count.
|
||||||
PromptTokens int `json:"prompt_tokens"`
|
PromptTokens int `json:"prompt_tokens"`
|
||||||
|
// CompletionTokens is the reported generated-token count.
|
||||||
CompletionTokens int `json:"completion_tokens"`
|
CompletionTokens int `json:"completion_tokens"`
|
||||||
|
// TotalTokens is the reported total-token count.
|
||||||
TotalTokens int `json:"total_tokens"`
|
TotalTokens int `json:"total_tokens"`
|
||||||
|
// CachedTokens is the reported cached-input-token count.
|
||||||
CachedTokens int `json:"cached_tokens"`
|
CachedTokens int `json:"cached_tokens"`
|
||||||
|
// CacheWriteTokens is the reported cache-write-token count.
|
||||||
CacheWriteTokens int `json:"cache_write_tokens"`
|
CacheWriteTokens int `json:"cache_write_tokens"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// RenderedPrompt is the fully rendered prompt passed to an LLM client.
|
// RenderedPrompt is the fully rendered prompt passed to an LLM client and has
|
||||||
|
// a stable JSON representation.
|
||||||
type RenderedPrompt struct {
|
type RenderedPrompt struct {
|
||||||
|
// SessionID is the optional trimmed session identifier rendered from the
|
||||||
|
// prompt definition.
|
||||||
SessionID string `json:"session_id,omitempty"`
|
SessionID string `json:"session_id,omitempty"`
|
||||||
|
// Messages contains rendered messages in definition order.
|
||||||
Messages []RenderedMessage `json:"messages"`
|
Messages []RenderedMessage `json:"messages"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// RenderedMessage is a rendered chat message.
|
// RenderedMessage is a rendered chat message and has a stable JSON
|
||||||
|
// representation.
|
||||||
type RenderedMessage struct {
|
type RenderedMessage struct {
|
||||||
|
// Role is the definition-supplied chat role.
|
||||||
Role string `json:"role"`
|
Role string `json:"role"`
|
||||||
|
// Content is the rendered message text.
|
||||||
Content string `json:"content"`
|
Content string `json:"content"`
|
||||||
|
// CacheControl is optional provider cache metadata.
|
||||||
CacheControl *CacheControl `json:"cache_control,omitempty"`
|
CacheControl *CacheControl `json:"cache_control,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// CacheControl describes provider cache metadata attached to prompt content.
|
// CacheControl describes provider cache metadata attached to prompt content
|
||||||
|
// and has a stable JSON representation.
|
||||||
type CacheControl struct {
|
type CacheControl struct {
|
||||||
|
// Type identifies the cache behavior.
|
||||||
Type CacheControlType `json:"type"`
|
Type CacheControlType `json:"type"`
|
||||||
|
// TTL is an optional provider cache lifetime.
|
||||||
TTL string `json:"ttl,omitempty"`
|
TTL string `json:"ttl,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// StructuredOutputSpec describes provider-level structured output.
|
// StructuredOutputSpec describes provider-level structured output and has a
|
||||||
|
// stable JSON representation.
|
||||||
type StructuredOutputSpec struct {
|
type StructuredOutputSpec struct {
|
||||||
|
// Type identifies the structured-output mechanism.
|
||||||
Type StructuredOutputType `json:"type"`
|
Type StructuredOutputType `json:"type"`
|
||||||
|
// JSONSchema contains constraints when Type is StructuredOutputJSONSchema.
|
||||||
JSONSchema *StructuredOutputJSONSpec `json:"json_schema,omitempty"`
|
JSONSchema *StructuredOutputJSONSpec `json:"json_schema,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// StructuredOutputJSONSpec contains JSON Schema output constraints.
|
// StructuredOutputJSONSpec contains provider-facing JSON Schema output
|
||||||
|
// constraints and has a stable JSON representation.
|
||||||
type StructuredOutputJSONSpec struct {
|
type StructuredOutputJSONSpec struct {
|
||||||
|
// Name is the provider-facing schema name.
|
||||||
Name string `json:"name"`
|
Name string `json:"name"`
|
||||||
|
// Strict requests strict provider enforcement of Schema.
|
||||||
Strict bool `json:"strict"`
|
Strict bool `json:"strict"`
|
||||||
|
// Schema is a caller-owned copy of the loaded JSON Schema document.
|
||||||
Schema any `json:"schema"`
|
Schema any `json:"schema"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// LLMClient executes rendered prompts for Engine.Run.
|
// LLMClient executes rendered prompts for [Engine.Run].
|
||||||
|
//
|
||||||
|
// Generate may be called concurrently. It must honor context cancellation to
|
||||||
|
// make Run responsive to cancellation. The request and all nested maps,
|
||||||
|
// slices, and pointers are client-owned copies and may be mutated or retained
|
||||||
|
// without affecting engine state.
|
||||||
|
//
|
||||||
|
// Generate receives rendered messages and may receive a direct API key. A
|
||||||
|
// client must protect those values and any raw output in its logging, storage,
|
||||||
|
// and retained copies. It is responsible for the cancellation behavior of any
|
||||||
|
// work it starts and for synchronizing access to retained or shared data.
|
||||||
|
//
|
||||||
|
// A returned error makes Run return ErrLLMGenerate while preserving the client
|
||||||
|
// error through errors.Is. A nil response with a nil error also produces
|
||||||
|
// ErrLLMGenerate. Promptkit copies the non-nil response before returning from
|
||||||
|
// Run.
|
||||||
type LLMClient interface {
|
type LLMClient interface {
|
||||||
Generate(context.Context, GenerateRequest) (*GenerateResponse, error)
|
Generate(context.Context, GenerateRequest) (*GenerateResponse, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
// GenerateRequest is passed to an injected LLM client.
|
// GenerateRequest is passed to an injected LLM client and has a stable JSON
|
||||||
|
// representation. Its String and GoString methods omit rendered content and
|
||||||
|
// direct credentials.
|
||||||
type GenerateRequest struct {
|
type GenerateRequest struct {
|
||||||
|
// Prompt contains the rendered session ID and messages.
|
||||||
Prompt RenderedPrompt `json:"prompt"`
|
Prompt RenderedPrompt `json:"prompt"`
|
||||||
|
// Target contains effective model settings without the direct API key.
|
||||||
Target ExecutionTarget `json:"target"`
|
Target ExecutionTarget `json:"target"`
|
||||||
|
// TargetPresence distinguishes inherited numeric zeros from explicit
|
||||||
|
// request overrides.
|
||||||
TargetPresence ExecutionTargetPresence `json:"target_presence"`
|
TargetPresence ExecutionTargetPresence `json:"target_presence"`
|
||||||
|
// StructuredOutput contains provider response constraints when requested.
|
||||||
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
|
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
|
||||||
|
// APIKey is the direct request-scoped credential, if any. It is excluded
|
||||||
|
// from JSON, String, and GoString output.
|
||||||
APIKey string `json:"-"`
|
APIKey string `json:"-"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// GenerateResponse is returned by an injected LLM client.
|
// GenerateResponse is returned by an injected LLM client and has a stable JSON
|
||||||
|
// representation.
|
||||||
type GenerateResponse struct {
|
type GenerateResponse struct {
|
||||||
|
// Content is the generated output. It must be non-empty when using the
|
||||||
|
// built-in client; injected clients may return empty content for Promptkit
|
||||||
|
// validation to classify.
|
||||||
Content string `json:"content"`
|
Content string `json:"content"`
|
||||||
|
// Usage is the client's token accounting.
|
||||||
Usage TokenUsage `json:"usage"`
|
Usage TokenUsage `json:"usage"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// File returns a file-backed artifact reference.
|
// File returns a file-backed artifact reference whose URI is path.
|
||||||
|
//
|
||||||
|
// The default artifact reader opens path as a caller-selected operating-system
|
||||||
|
// path without restricting it to an application root or imposing a size limit.
|
||||||
|
// Applications accepting untrusted paths must validate them before calling
|
||||||
|
// Promptkit or use [WithArtifactReader] to enforce application policy.
|
||||||
func File(path string) ArtifactRef {
|
func File(path string) ArtifactRef {
|
||||||
return ArtifactRef{Type: ArtifactRefFile, URI: path}
|
return ArtifactRef{Type: ArtifactRefFile, URI: path}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Inline returns an inline artifact reference.
|
// Inline returns an inline artifact reference whose Body is body and whose URI
|
||||||
|
// is empty.
|
||||||
func Inline(body string) ArtifactRef {
|
func Inline(body string) ArtifactRef {
|
||||||
return ArtifactRef{Type: ArtifactRefInline, Body: body}
|
return ArtifactRef{Type: ArtifactRefInline, Body: body}
|
||||||
}
|
}
|
||||||
|
|
||||||
// InlineWithURI returns an inline artifact reference with URI metadata.
|
// InlineWithURI returns an inline artifact reference with body content and uri
|
||||||
|
// provenance metadata.
|
||||||
func InlineWithURI(uri string, body string) ArtifactRef {
|
func InlineWithURI(uri string, body string) ArtifactRef {
|
||||||
return ArtifactRef{Type: ArtifactRefInline, URI: uri, Body: body}
|
return ArtifactRef{Type: ArtifactRefInline, URI: uri, Body: body}
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user