43 Commits
v0.5.0 ... main

Author SHA1 Message Date
4ca3be2c14 Finish audit remediation and prepare v0.6.0 2026-08-12 13:01:05 +00:00
227fb35f99 Centralize the maintainer validation workflow 2026-08-12 00:04:48 +00:00
e291b8bfe9 Consolidate provider transport test scaffolding 2026-08-11 23:58:31 +00:00
2b6a7f83c4 Bound and strictly decode provider responses 2026-08-11 23:46:16 +00:00
3a43550f70 Validate and compose provider endpoints 2026-08-11 23:38:45 +00:00
c281f721bc Preserve transport error identities 2026-08-11 23:28:00 +00:00
350b0e76d9 Preserve repair settings and cumulative usage 2026-08-11 23:21:34 +00:00
e43350fd0d Make validation cancellation authoritative 2026-08-11 23:14:01 +00:00
20d3e3b5ee Escape schema resources and reuse compiled plans 2026-08-11 23:05:11 +00:00
a93b799236 Preserve exact JSON validation semantics 2026-08-11 22:55:17 +00:00
e83a3ce179 Honor cancellation while rendering prompts 2026-08-11 22:48:20 +00:00
a04a3bbc5f Correct artifact file and empty input handling 2026-08-11 22:41:37 +00:00
731b66cff5 Avoid decoding unrelated profiles 2026-08-11 22:32:33 +00:00
70e0ea0cf0 Correct profile source validation and identity 2026-08-11 22:28:48 +00:00
d45c474c1e Unify prompt repository source handling 2026-08-11 22:18:50 +00:00
25f1ba0b30 Correct prompt definition selection and decoding 2026-08-11 22:11:39 +00:00
a718762da1 Contain prompt content paths within source roots 2026-08-11 22:03:32 +00:00
58ac3ce298 Correct engine construction edge cases 2026-08-11 21:54:47 +00:00
57f2ce1ce4 Harden public ownership and diagnostic contracts 2026-08-11 21:45:54 +00:00
c8b6d5c490 Consolidate public JSON serialization 2026-08-11 21:40:12 +00:00
abeb50b525 Bound JSON-compatible value copying 2026-08-11 21:32:24 +00:00
1cb07c7d91 Centralize output contract validation 2026-08-11 21:21:33 +00:00
8cfc71c351 Centralize execution setting and session validation 2026-08-11 21:12:56 +00:00
5ccfa4a345 Plan the codebase audit remediation 2026-08-11 20:10:14 +00:00
14e03f19d0 Consolidate and close the codebase audit 2026-08-11 17:21:29 +00:00
7c562a9374 Document cross-cutting architecture audit findings 2026-08-11 17:10:59 +00:00
ef97d85ac9 Document repository-wide test strategy audit 2026-08-11 17:00:59 +00:00
805e48c873 Document capacity scheduling audit results 2026-08-11 16:51:28 +00:00
e9e126dcba Document OpenAI transport audit findings 2026-08-11 16:42:55 +00:00
32e7a3557c Document prepared execution lifecycle audit 2026-08-11 16:31:05 +00:00
1d1b04e2e0 Record ordinary execution and repair audit findings 2026-08-11 16:22:09 +00:00
9748897751 Document inspection and target resolution audit findings 2026-08-11 16:09:50 +00:00
9d020039d5 Record output validation audit findings 2026-08-11 15:59:46 +00:00
5247ce0b73 Record artifact loading and prompt rendering audit findings 2026-08-11 15:41:57 +00:00
c434aa1dae Record profile source audit findings 2026-08-11 15:29:23 +00:00
ac9b3f3d80 Record prompt source audit findings 2026-08-11 15:18:17 +00:00
4f12a89a1b Record backend registry and defaults audit findings 2026-08-11 15:05:15 +00:00
df31e7f58e Record domain and JSON value audit findings 2026-08-11 14:54:56 +00:00
0678d242b9 Record engine operation audit findings 2026-08-11 14:43:20 +00:00
3b4ea21208 Record engine construction audit findings 2026-08-11 14:36:33 +00:00
1430e85147 Record configuration and adapter audit findings 2026-08-11 14:28:07 +00:00
34d7a19da5 Record public value and error audit findings 2026-08-11 14:17:55 +00:00
ebf1602635 Prepare the codebase audit plan 2026-08-11 14:05:30 +00:00
71 changed files with 8377 additions and 2536 deletions

View File

@@ -33,7 +33,10 @@ boundary and constraints that framework work must preserve.
## Release Guidance
Consumers upgrading from `v0.4.0` to `v0.5.0` should read the
Consumers upgrading from `v0.5.0` to `v0.6.0` should read the
[v0.6.0 changelog and migration guide](docs/releases/v0.6.0.md).
Earlier adopters can consult the
[v0.5.0 changelog and migration guide](docs/releases/v0.5.0.md).
Consumers upgrading from `v0.3.0` to `v0.4.0` should read the

View File

@@ -34,7 +34,8 @@ type Backend struct {
// JSON-compatible, finite, acyclic, and keyed by non-empty strings. Keys
// must not be model, session_id, messages, temperature, max_tokens, top_p,
// service_tier, reasoning_effort, or response_format. An empty map supplies
// no defaults. NewEngine deeply copies the map.
// no defaults. NewEngine deeply copies the map and rejects excessively deep
// or large values for safety.
ExtraParams map[string]any
// ConcurrencyLimit is the maximum number of simultaneous model-generation
// calls allowed for this backend within one Engine. Zero leaves the backend

View File

@@ -44,3 +44,190 @@ Start with:
For cross-cutting changes, follow every applicable row. Do not create
placeholder documents for packages, APIs, or integrations that do not yet
exist.
## Maintainer Validation
This section is the canonical local validation workflow for Promptkit. Run
every command from the repository root before accepting a change. The test
suite and maintained examples are deterministic, offline, and require no real
provider credentials.
### Tests, Analysis, Build, And Examples
Run the ordinary and race-enabled suites, static analysis, the build, and both
maintained consumer examples:
```sh
go test ./...
go test -race ./...
go vet ./...
go build ./...
go run ./examples/go-library/prepare
go run ./examples/go-library/run
```
Both examples must exit successfully. Review their JSON output: preparation
must report the selected offline prompt, profile, model, and message count;
execution must report the deterministic generated output, successful
validation, selected offline model, and usage. Neither command may contact a
provider or require credentials.
### Go Formatting
Check every tracked Go file. The final command must succeed and the captured
list must be empty:
```sh
unformatted=$(
git ls-files '*.go' |
while IFS= read -r go_file
do
gofmt -l "$go_file"
done
)
test -z "$unformatted"
```
### Local Markdown Links
Use the Python standard library to verify every repository-relative Markdown
target and local heading fragment. The check is offline and prints nothing on
success:
```sh
python3 - <<'PY'
from pathlib import Path
import re
import subprocess
import sys
from urllib.parse import unquote
root = Path.cwd().resolve()
markdown_files = [
root / name
for name in subprocess.check_output(
["git", "ls-files", "*.md"], text=True
).splitlines()
]
link_pattern = re.compile(r"!?\[[^]]*\]\(([^)]+)\)")
heading_pattern = re.compile(r"^#{1,6}\s+(.+?)\s*#*\s*$")
scheme_pattern = re.compile(r"^[a-z][a-z0-9+.-]*:", re.IGNORECASE)
def markdown_lines(path):
in_fence = False
fence = ""
for line in path.read_text(encoding="utf-8").splitlines():
stripped = line.lstrip()
marker = stripped[:3]
if marker in {"```", "~~~"}:
if not in_fence:
in_fence = True
fence = marker
elif marker == fence:
in_fence = False
fence = ""
continue
if not in_fence:
yield line
anchor_cache = {}
def anchors(path):
if path in anchor_cache:
return anchor_cache[path]
found = set()
counts = {}
for line in markdown_lines(path):
match = heading_pattern.match(line)
if not match:
continue
heading = re.sub(r"<[^>]+>", "", match.group(1)).replace("`", "")
base = re.sub(r"[^\w\- ]", "", heading.lower()).replace(" ", "-")
count = counts.get(base, 0)
counts[base] = count + 1
found.add(base if count == 0 else f"{base}-{count}")
anchor_cache[path] = found
return found
failures = []
for source in markdown_files:
text = "\n".join(markdown_lines(source))
for match in link_pattern.finditer(text):
target = match.group(1).strip()
if target.startswith("<") and target.endswith(">"):
target = target[1:-1]
if scheme_pattern.match(target) or target.startswith("//"):
continue
path_text, separator, fragment = target.partition("#")
destination = source if not path_text else source.parent / unquote(path_text)
try:
destination = destination.resolve()
destination.relative_to(root)
except ValueError:
failures.append(f"{source.relative_to(root)}: escapes repository: {target}")
continue
if not destination.exists():
failures.append(f"{source.relative_to(root)}: missing target: {target}")
continue
if separator and destination.suffix.lower() == ".md":
fragment = unquote(fragment).lower()
if fragment not in anchors(destination):
failures.append(f"{source.relative_to(root)}: missing anchor: {target}")
if failures:
print("\n".join(failures), file=sys.stderr)
raise SystemExit(1)
PY
```
### Repository Hygiene And Review
Reject an active Go workspace, tracked workspace files, a vendor tree, or a
module replacement:
```sh
case "$(go env GOWORK)" in
''|off) ;;
*) printf '%s\n' 'an active Go workspace is not allowed' >&2; exit 1 ;;
esac
test -z "$(git ls-files go.work go.work.sum)"
test ! -e vendor
if grep -Eq '^[[:space:]]*replace([[:space:]]|\()' go.mod
then
printf '%s\n' 'go.mod contains a replacement' >&2
exit 1
fi
```
Check whitespace in both unstaged and staged changes. List ignored files and
scan tracked content for common credential forms:
```sh
git diff --check
git diff --cached --check
test -z "$(git ls-files --others --ignored --exclude-standard)"
credential_pattern='-----BEGIN ([A-Z0-9]+ )?PRIV''ATE KEY-----|AKI''A[0-9A-Z]{16}|gh[pousr]_[A-Za-z0-9]{36,}|sk-[A-Za-z0-9]{32,}'
if git grep -nEI -e "$credential_pattern" -- .
then
printf '%s\n' 'possible credential found' >&2
exit 1
fi
```
Inspect `git status --short --untracked-files=all` and the complete diff before
accepting a change. The status may contain only the intended source changes
during development. Reject credentials, private keys, environment files,
generated binaries, test or coverage output, downloaded assets, template
residue, and any other artifact that does not belong in source control. The
credential scan catches common forms but does not replace inspection of the
actual change.
After committing the accepted change, require a clean candidate:
```sh
test -z "$(git status --porcelain)"
```

View File

@@ -9,9 +9,10 @@ explains how to select these sources and invoke the engine. The
owns the resulting outbound wire behavior.
Prompt and profile sources recursively discover files ending in `.yaml` or
`.yml`. YAML decoding is strict: unknown fields are errors for the selected
definition. Definitions are selected by their YAML `id`, not their file name
or directory.
`.yml`. Each prompt-definition and profile file contains exactly one YAML
document; comments and trailing whitespace are allowed. YAML decoding is
strict: unknown fields are errors for the selected definition. Definitions are
selected by their YAML `id`, not their file name or directory.
## Prompt Definitions
@@ -86,9 +87,16 @@ Each message has a non-empty `role` and exactly one of:
- `content`, containing an inline Go template; or
- `content_file`, naming a file whose contents are the Go template.
For directory and `fs.FS` prompt sources, `content_file` resolves relative to
the prompt file and remains within the source root. `WithPromptFile` also
resolves it relative to that file.
`content_file` must be a relative path. It resolves from the directory that
contains the prompt file and must remain within the configured prompt source
root; parent components are allowed only when the resolved target remains
inside that root. Absolute paths and paths that escape the root are rejected.
Operating-system directory and single-file sources also reject symlink targets
outside the root, while injected `fs.FS` sources apply containment in that
filesystem's relative path namespace. For `WithPromptFile`, the source root is
the directory containing the selected prompt file. Promptkit uses the parsed
path text exactly after checking separately that it is not blank, so leading
and trailing whitespace can name real filesystem entries.
Request variables are the template data, so a variable named `audience` is
referenced as `{{.audience}}`. The `{{input "note"}}` helper renders the body
@@ -156,9 +164,9 @@ extra_params:
| Field | Required | Meaning |
| --- | --- | --- |
| `id` | yes | Non-empty profile identifier. IDs must be unique within one source. |
| `id` | yes | Profile identifier, trimmed before selection and publication. It must be non-empty after trimming and unique within one source after normalization. |
| `backend` | unless `endpoint` is present | Backend registry ID. It is trimmed and registry membership is checked when the profile is prepared or inspected. |
| `endpoint` | unless `backend` is present | Non-empty OpenAI-compatible base URL, including an API version path when required. When both connection fields are present, this overrides the backend endpoint without changing backend identity. |
| `endpoint` | unless `backend` is present | OpenAI-compatible base URL, including an API version path when required. A nonempty value is trimmed and must be absolute HTTP or HTTPS with a host and without user information, a query, or a fragment. When both connection fields are present, this overrides the backend endpoint without changing backend identity. |
| `model` | yes | Non-empty provider model name. |
| `temperature` | no | Number from 0 through 2. |
| `max_tokens` | no | Integer zero or greater. |
@@ -183,6 +191,7 @@ GoDoc.
objects with string keys. Keys must be non-empty. With the built-in client,
they also cannot collide with the standard fields listed in the
[outbound request contract](integrations/openai-compatible-chat.md#request-body).
Excessively deep or large JSON-shaped values are rejected for safety.
### Defaults And Overrides

View File

@@ -14,10 +14,16 @@ that produce these outbound settings.
Generation sends an HTTP `POST` with `Content-Type: application/json`.
Before the client is called, the engine resolves framework, backend, profile,
and request values into one execution target. A non-empty endpoint from that
target overrides the client's configured base URL. After trailing slashes are
removed, `/chat/completions` is appended. Generation fails before sending when
neither source supplies an endpoint.
and request values into one execution target. Endpoint configuration is trimmed
and must be an absolute HTTP or HTTPS URL with a host and without user
information, a query, or a fragment. A non-empty endpoint from the target
overrides the client's configured base URL. The final selected endpoint is
validated again before transport.
The completion URL is composed through parsed URL path operations. Nested base
paths are retained, repeated trailing slashes are normalized, and the result
has exactly one appended `/chat/completions` suffix. Generation fails before
sending when neither source supplies a valid endpoint.
The target's backend ID is routing metadata for prepared values, results, and
injected clients. The built-in client does not derive the URL from that ID and
@@ -82,13 +88,29 @@ request fields.
## Response Handling
Any 2xx response is decoded as an OpenAI-compatible chat response. The client
returns the first choice's non-empty message content and maps prompt,
completion, total, cached, and cache-write token counts.
Any 2xx response body is limited to 16 MiB (16,777,216 bytes). A larger
declared `Content-Length` is rejected before the body is read, and streamed,
chunked, or underreported bodies are read through the same bound with at most
one additional byte used to detect overflow. A body exactly at the limit is
allowed. The body is closed on every outcome and an oversized stream is not
drained.
Invalid JSON, absent choices, and empty first-choice content are malformed
responses. For a non-2xx status, the error includes the status code but never
the provider response body.
The bounded body must contain exactly one OpenAI-compatible JSON response
object followed only by JSON whitespace and EOF. The client returns the first
choice's non-empty message content and maps prompt, completion, total, cached,
and cache-write token counts. Invalid or truncated JSON, trailing non-whitespace
data, a second JSON value, absent choices, empty first-choice content, and size
overflow are malformed responses and return no partial result.
For a non-2xx status, the error includes the status code but never the provider
response body. Promptkit does not yet parse provider error envelopes; bounded
non-success parsing belongs to the
[structured-generation-error roadmap](../roadmap/structured-generation-errors.md).
An outbound `http.Client.Do` failure retains both Promptkit's request-failure
identity and the exact transport error for `errors.Is` and `errors.As` checks.
The rendered error does not include the selected endpoint, request headers,
request content, credentials, or provider body.
## Timeout And Cancellation
@@ -103,5 +125,7 @@ Timeouts are layered:
timeout when the supplied value is not positive.
The earliest applicable caller, generation, or transport deadline controls the
request. Constructing the internal client does not mutate a supplied
`http.Client`.
request. Caller cancellation retains `context.Canceled`; caller, generation,
and whole-request timeout failures retain `context.DeadlineExceeded`, together
with the request-failure identity. Constructing the internal client does not
mutate a supplied `http.Client`.

View File

@@ -56,7 +56,10 @@ reserve another bounded slot.
`NewClient` wraps the engine's selected internal model client after public
client adaptation or built-in client construction. Initial generation and the
default repairer receive the same wrapper.
default repairer receive the same wrapper. Their requests retain the same
effective backend, credential, numeric-presence metadata, and structured-output
settings, so scheduling does not change provider omission semantics between
calls.
For each `Generate` call, the wrapper selects a pool from the request's
effective backend ID. An unlimited call passes directly to the next client. A
@@ -71,7 +74,9 @@ other backend IDs.
The wrapper passes generation requests, responses, and collaborator errors
through unchanged. It owns scheduling only; the concrete model client remains
responsible for provider transport behavior.
responsible for provider transport behavior. The runner, rather than the
capacity layer, sums all five usage fields from the initial response and every
completed repair response into the successful run result.
## Cancellation And Release

View File

@@ -25,16 +25,20 @@ and request precedence. The client uses its endpoint, credential metadata,
generation fields, and extra parameters. `BackendID` remains routing metadata
for the generation boundary and is not mapped into the provider payload.
Construction validates the configured base URL and clones any supplied
`http.Client` so Promptkit can apply its timeout default without mutating the
caller's client. Generation then:
Construction trims and validates a nonempty configured base URL and clones any
supplied `http.Client` so Promptkit can apply its timeout default without
mutating the caller's client. An empty configured base remains valid because a
resolved request target may supply the endpoint. Generation then:
1. validates request-level timeout and endpoint requirements;
1. validates shared execution-setting invariants and the final selected base
endpoint;
2. maps the internal request into the OpenAI-compatible chat payload;
3. validates and merges extra parameters;
4. resolves authentication;
5. performs the outbound request under the applicable deadlines; and
6. decodes the first response choice and token usage.
4. composes `/chat/completions` through parsed URL path operations;
5. resolves authentication;
6. performs the outbound request under the applicable deadlines; and
7. decodes one strictly framed, size-bounded response object and maps its first
choice and token usage.
`internal/llm` owns the set of reserved OpenAI-compatible request fields used
when validating extra parameters. Backend registration consumes the same rule
@@ -65,9 +69,28 @@ configuration, invalid generation requests, request execution failures,
non-success provider statuses, and malformed successful responses. Provider
response bodies are not included in non-success errors.
Caller cancellation and deadline failures during the outbound request are
reported as request execution failures. The runner classifies these identities
without depending on HTTP status mapping.
Invalid nonempty configured endpoints are configuration failures. A missing or
invalid final selected endpoint is an invalid generation request and is
rejected before transport.
Successful response bodies have a fixed 16 MiB limit enforced by declared
length and by reading at most one byte beyond the boundary. The decoder accepts
exactly one JSON object plus trailing whitespace and EOF. Size overflow,
truncation, malformed JSON, trailing data, and a second value are malformed
responses with no partial result or provider content in the error. Every body
is closed, and an unbounded oversized stream is not drained. Non-success
responses remain status-only; bounded provider error-envelope parsing belongs
to the
[structured-generation-error roadmap](../roadmap/structured-generation-errors.md).
An `http.Client.Do` failure is represented by a redacting multi-cause error:
the package request-failure sentinel and the exact returned transport error are
both available through `errors.Is` and `errors.As`, while the rendered text
does not expose the endpoint, headers, request content, credential, transport
detail, or provider body. Caller cancellation retains `context.Canceled`;
caller deadlines, generation deadlines, and whole-request client timeouts
retain `context.DeadlineExceeded`. The runner adds its generation category
without discarding those identities or depending on HTTP status mapping.
## Test Ownership
@@ -75,7 +98,11 @@ The
[OpenAI-compatible client tests](../../internal/llm/openai_compatible_client_test.go)
own configuration, client cloning, deterministic deadline precedence,
authentication, request and response mapping, malformed data, error identity,
cancellation, and response-body suppression. The root transport contract test
also verifies that resolved backend settings reach this client without
serializing backend identity. All use local test servers or test transports;
the default suite makes no live or paid provider requests.
cancellation, endpoint selection and composition, pre-transport rejection, and
bounded single-document response framing, closure, and response-body
suppression. The root
transport contract tests also verify that resolved backend settings reach this
client without serializing backend identity and that ordinary-run cancellation
retains its public generation and context identities. All use local test
servers or controlled test transports; the default suite makes no live or paid
provider requests.

View File

@@ -16,18 +16,18 @@ contributor workflow and validation.
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
| `internal/backend` | Constructs each engine's immutable registry from the built-in OpenRouter definition and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) |
| `internal/capacity` | Owns engine-local bounded execution admission and FIFO model-generation permits for limited backend IDs, including cancellation-safe waiter removal and client wrapping. | [Internal capacity management](capacity.md) |
| `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, profiles, execution targets, rendering, generation, and validation. | [Domain declarations](../../internal/domain/domain.go) |
| `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, profiles, execution targets, rendering, generation, and validation, and owns source-neutral invariants for shared execution settings, OpenAI-compatible base endpoints, session identifiers, and output contracts. Source parsing, required fields, other source-specific normalization, defaulting, and boundary-specific error classification remain with their callers. | [Domain declarations](../../internal/domain/domain.go), [endpoint invariant](../../internal/domain/endpoint.go) |
| `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) |
| `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |
| `internal/jsonvalue` | Validates and deeply copies JSON-compatible extra-parameter and prepared-schema trees while preserving supported concrete value types. | [JSON values](../../internal/jsonvalue/jsonvalue.go) |
| `internal/jsonvalue` | Validates and deeply copies bounded JSON-compatible extra-parameter and prepared-schema trees while preserving supported concrete value types and rejecting cycles or excessive depth and work. | [JSON values](../../internal/jsonvalue/jsonvalue.go) |
| `internal/promptdef` | Loads strictly decoded, validated prompt definitions from filesystem and `fs.FS` sources, including version selection and contained file-backed message content. | [Framework formats](../formats.md), [prompt-definition repository](../../internal/promptdef/filesystem_repository.go) |
| `internal/profile` | Loads strictly decoded, validated execution profiles, including backend selection, from filesystem and `fs.FS` sources and composes repositories with error-preserving fallback. | [Framework formats](../formats.md), [profile repositories](../../internal/profile/filesystem_repository.go) |
| `internal/profile/builtin` | Embeds the built-in profile catalog, whose entries select OpenRouter. | [Built-in catalog](../formats.md#built-in-profile-catalog), [repository](../../internal/profile/builtin/repository.go) |
| `internal/prompt` | Renders prompt messages from Go templates with artifact, variable, session, and cache-control data. | [Go-template renderer](../../internal/prompt/go_renderer.go) |
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
| `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates frozen validation plans for prepared execution. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
| `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates operation-local validation plans with canonical contained schema resources. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including response decoding, authentication, deadline handling, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
| `internal/usecase` | Resolves prompt definitions and hashes, profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, generation, validation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
| `internal/usecase` | Resolves prompt definitions and hashes, profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, operation-local validation plans, generation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
The root package assembles these internal components without exposing their
representations. Consumers depend only on the root facade.

View File

@@ -20,8 +20,9 @@ and override semantics consumed by the runner.
profiles, backend resolution, artifacts, rendering, model generation, and
validation. The root engine supplies one immutable registry containing the
built-in backend and validated consumer additions, one engine-local run
admitter, and a model client wrapped by the same capacity manager. Schema
documents are loaded through the validator's optional schema-loader interface.
admitter, and a model client wrapped by the same capacity manager. Validation
plans and provider-facing schema metadata come from the validator's preparation
interface.
An output repairer can be injected internally, but the ordinary runner
constructor does not enable one.
@@ -69,14 +70,16 @@ performs only the work needed to validate routing and admission:
5. resolve application-neutral defaults, backend defaults, profile values,
and explicit request overrides in that order;
6. validate endpoint, model, numeric overrides, and credential requirements;
7. resolve the effective output contract without loading its schema; and
7. resolve and validate the effective output contract without loading its
schema; and
8. retain the definition, source identities, effective settings, output
contract, and preparation start time in invocation-local state.
The completion phase consumes that state without reloading the prompt,
profile, or backend:
1. load structured-output schema metadata when required;
1. create one operation-local validation plan and derive structured-output
schema metadata from it when required;
2. load and hash input artifacts;
3. render messages and the prompt-defined session;
4. apply any direct session ID;
@@ -87,7 +90,10 @@ profile, or backend:
`Run` performs backend admission between the phases. This structure preserves
one execution-precedence and error-ordering implementation while allowing a
full backend pool to reject work before expensive schema, artifact, and
rendering operations.
rendering operations. `Prepare` discards the plan after returning its public
metadata. `Run` retains the plan through initial and repaired-output validation
and discards it when the operation ends. Prepared execution stores the same
kind of plan only in its private payload.
Pointer-based numeric overrides preserve an explicit zero. Invalid negative or
out-of-range values fail as invalid requests. Endpoint overrides do not change
@@ -118,8 +124,17 @@ its `RunAdmitter` to reserve capacity for the effective backend ID. A nil
admitter is an internal unlimited fallback. After successful admission, `Run`
immediately defers the returned release function, performs the completion
phase, makes one initial generation call, builds the named output artifact,
and validates that artifact. Invalid generated content remains a validation
result; an inability to perform validation is an operational error.
and validates that artifact with the plan compiled during completion. Invalid
generated content remains a validation result; an inability to perform
validation is an operational error.
Validation preparation and execution honor cancellation at every
Promptkit-controlled boundary and do not publish a partial plan or result.
Schema reads are bounded and context-checked between chunks; JSON decoding,
schema compilation, and schema execution are checked immediately before and
after their synchronous calls. Promptkit does not move arbitrary filesystem or
JSON Schema work to background goroutines, so an already-blocked dependency
method must return before cancellation can take precedence over its outcome.
The admission lease covers completion-phase preparation, initial generation,
validation, every repair, and every exit. It bounds accepted work without
@@ -129,20 +144,27 @@ each actual generation call.
When an internal repairer is present, a JSON or JSON Schema content failure can
trigger bounded repair attempts. Repair receives the effective execution
target and session ID, validation errors, prior output, and structured-output
specification. The default repairer uses the same wrapped client as initial
generation, so each repair reacquires the selected backend's active permit
while remaining inside its original admission lease. Repair never performs a
second bounded admission. This capability remains internal and is not a public
option.
target, explicit numeric-presence bits, credential, backend identity, session
ID, validation errors, prior output, and structured-output specification. One
request constructor supplies those common fields to initial and repair
generation while their rendered prompts remain intentionally distinct. The
default repairer uses the same wrapped client as initial generation, so each
repair reacquires the selected backend's active permit while remaining inside
its original admission lease. Repair never performs a second bounded
admission, and repaired outputs use the operation's existing validation plan.
This capability remains internal and is not a public option.
A successful result includes the output artifact and raw output, validation
state, effective session ID, prompt and rendered-prompt hashes, selected
profile and backend, effective settings, input hashes, token usage, a generated
run identifier, and UTC timing. The same effective session reaches initial
generation and any repair attempt through the rendered prompt. The same
effective target, including backend identity, reaches generation and any
repair attempt.
effective target and presence metadata, including backend identity and direct
credential during execution, reaches generation and every repair attempt.
Result usage is the field-wise sum of all five usage values from the initial
response and every completed repair response. Final raw output, artifact, and
validation state still come from the last candidate. A repair error returns no
partial run result or partial usage.
## Failure Categories
@@ -165,7 +187,8 @@ category. Deferred release restores the admission lease on preparation,
generation, validation, repair, and cancellation failures.
Other context cancellation propagates through the invoked collaborator and is
classified by the owning operation.
classified by the owning operation. In particular, cancellation observed by
validation retains the context identity through the validation error category.
An overlong direct session is an invalid request before source loading, while
an invalid or overlong prompt session template remains a prompt-render failure.
An unknown selected backend, or a selected backend with no configured resolver,
@@ -177,8 +200,9 @@ The [runner tests](../../internal/usecase/runner_test.go) own preparation order,
selection and override precedence, the two-phase boundary, early admission,
lease lifetime and release, direct-session resolution, schema-before-generation
behavior, hashing, generation and validation outcomes, backend propagation,
bounded repair, shared initial/repair capacity, credentials and redaction,
error categories, artifact metadata, usage, and timing. The
bounded repair progression, initial/repair request parity, cumulative usage,
shared initial/repair capacity, credentials and redaction, error categories,
artifact metadata, and timing. The
[capacity subsystem document](capacity.md) identifies the focused pool,
waiter, and wrapped-client tests.

View File

@@ -12,9 +12,21 @@ validation modes, built-in catalog, and source precedence.
## Prompt Definitions
`internal/promptdef` discovers YAML deterministically, decodes and validates
definitions, selects an ID and optional version, and resolves file-backed
message content within the selected operating-system or `fs.FS` source.
`internal/promptdef` uses one source-neutral flow for prompt selection and
normalization. That flow scans normalized YAML ID and version metadata,
requires one strictly decoded document per file, classifies errors for the
selected definition, detects duplicates, and normalizes the exact match.
Small operating-system and `fs.FS` adapters own discovery, byte reads, display
paths, content opening, and root containment. Each lookup remains a
point-in-time scan: definitions and catalogs are not cached, and file-backed
message content is opened only for the exact selected candidate.
Operating-system sources enforce containment against canonical roots and
targets so symlinks cannot escape. Injected `fs.FS` sources enforce containment
in their clean relative path namespace. A single-file source uses the selected
prompt file's containing directory as its root. Every content path must be
relative and is opened from its exact parsed text after a separate blank check;
contained parent components and whitespace-bearing names remain valid.
Exact prompt inspection performs one point-in-time lookup through that same
repository and validates referenced message content before returning declared
@@ -28,13 +40,24 @@ duplicate detection, and source containment:
## Profiles And Built-Ins
`internal/profile` loads and validates execution profiles from an
operating-system filesystem or an `fs.FS`. Its overlay repository consults the
next repository only when the higher-precedence repository reports that a
profile is absent. Strict YAML decoding recognizes the optional `backend`
field, trims its value, and requires a model plus at least one non-blank
backend or endpoint. Loading does not check registry membership because the
available registry belongs to the assembled engine; the runner checks
membership during preparation and exact profile inspection.
operating-system filesystem or an `fs.FS`. A file contains exactly one YAML
document and its trimmed YAML `id` is its only selection identity; filenames do
not confer authority. Each point lookup reads discovered files once for their
metadata and reuses the selected file's bytes for strict decoding; unrelated
profiles are not fully decoded. Strict selected decoding recognizes the
optional `backend` field, trims its value, and requires a model plus at least
one non-blank backend or endpoint. File-backed `extra_params` values are
validated and defensively copied through the shared bounded JSON-value owner
before a profile is published. OpenAI-compatible reserved-field policy remains
with the model-client and backend-registry owners.
The overlay repository consults the next repository only when the
higher-precedence repository reports that a profile is absent. A reliably
selected malformed profile stops fallback, while an unrelated malformed file
does not become authoritative through its filename. Loading does not check
backend registry membership because the available registry belongs to the
assembled engine; the runner checks membership during preparation and exact
profile inspection.
The root engine assembles profile repositories in precedence order: in-memory
profiles, one ordinary configured source, an application fallback source, then
@@ -56,10 +79,17 @@ are owned by the
## Ordinary Artifacts
`internal/artifact` resolves inline references and unrestricted,
caller-selected file paths. It copies content into an artifact, records
metadata and a content hash, applies a content-type fallback, and honors
context cancellation.
`internal/artifact` accepts explicitly typed inline references even when their
body is empty. It also resolves unrestricted, caller-selected paths only when
they identify regular operating-system files, checking that condition before
and after opening the file. It copies content into an artifact, records
metadata and an opaque content-equality value, and applies a content-type
fallback.
Regular files are read synchronously in bounded chunks. Cancellation is
checked before opening, before and after every read, and before publishing the
artifact, so a canceled read never publishes partial content. The ordinary
reader does not detach file reads into background goroutines.
This ordinary reader does not implement an inbound HTTP security boundary. In
particular, it does not constrain files to an application root or impose an
@@ -71,8 +101,17 @@ implemented reader behavior and failures.
## Rendering
`internal/prompt` renders definition messages as Go templates using named
artifacts and variables. It carries message roles, session IDs, and cache
control into the rendered prompt. The
artifacts and variables. Within one render, each referenced artifact body is
converted to text lazily and cached by input name for reuse across the session
and every message; the cache is not shared across renders. Conversion uses
bounded chunks and preserves the artifact bytes exactly.
Session and message parsing and execution remain synchronous. The renderer
checks cancellation before and after each parse and execution boundary,
between artifact conversion chunks, around each message, and before publishing
the complete prompt. It cannot interrupt template work already in progress and
never publishes a partial prompt after observing cancellation. It carries
message roles, session IDs, and cache control into the rendered prompt. The
[renderer tests](../../internal/prompt/renderer_test.go) own rendering behavior.
## Schemas And Output Validation
@@ -82,22 +121,35 @@ filesystem or an `fs.FS`. Invalid generated content is returned as a validation
result; inability to load, register, or compile a schema is an operational
error.
For executable preparation, the built-in validators create a frozen validation
plan. None, basic, and JSON modes retain the effective output contract without
source access. JSON Schema mode loads the root document, resolves and compiles
every transitive reference during preparation, and retains the compiled
validator. The provider-facing structured-output metadata uses that same
captured root document.
Every preparation operation creates one operation-local validation plan. None,
basic, and JSON modes retain the effective output contract without source
access. JSON Schema mode loads the root document once, resolves and compiles
each transitive reference, and retains the compiled validator. Schema compiler
resources use canonical escaped file or private-scheme URLs; loaders decode
their paths once and enforce the configured source boundary. The
provider-facing structured-output metadata uses the root document captured by
the same plan.
`PrepareExecution` also completes prompt and profile selection, artifact
loading and hashing, session and message rendering, and target resolution.
`RunPrepared` uses the retained source-derived state and validation plan; it
does not reopen prompt, profile, input, or schema sources and does not rerender
the request. By contrast, ordinary `Prepare` produces a preparation value only:
a later `Run` performs its own source resolution and preparation.
Schema preparation and execution remain synchronous. Promptkit checks
cancellation before and after source resolution, JSON decoding, compilation,
and validation, and between bounded schema-read chunks. Once cancellation is
observed it returns the context error without publishing a partial plan or
validation result, even when a compiler or validator has just returned a
different error or a successful result. An `fs.FS` method or JSON Schema
dependency call already in progress cannot be preempted; Promptkit waits for
that call to return and then gives cancellation precedence. Validation does
not detach dependency work into background goroutines.
`Prepare` discards its validation plan after returning metadata. `Run` retains
its plan for initial and repaired-output validation, then discards it with the
operation. `PrepareExecution` retains the plan in its private frozen payload;
`RunPrepared` uses that plan without reopening prompt, profile, input, or
schema sources or rerendering the request. A later ordinary `Run` always
performs fresh source resolution and preparation.
The [validator tests](../../internal/validate/standard_validator_test.go) own
basic, JSON, JSON Schema, source resolution, schema loading, compilation,
frozen-reference behavior, and content-failure behavior. Prepared execution
frozen-reference behavior, content-failure behavior, and the synchronous
cancellation boundary. Prepared execution
orchestration is owned by the
[use-case tests](../../internal/usecase/prepared_execution_test.go).

View File

@@ -19,8 +19,8 @@ results, public values, extension interfaces, profiles, and error sentinels.
The implemented internal components consist of:
- `internal/domain`, which owns framework data values shared by later internal
components;
- `internal/domain`, which owns framework data values and source-neutral
invariants shared by later internal components;
- `internal/backend`, which owns validated immutable OpenAI-compatible backend
definitions and the built-in OpenRouter definition;
- `internal/capacity`, which owns engine-local bounded run admission and
@@ -92,6 +92,13 @@ coordinates internal components and adapts the supported public extension
interfaces to narrow internal abstractions. Internal components must not depend
on consumers or on Scriptorium.
`internal/domain` owns source-neutral invariants for values shared across
multiple input and execution boundaries, including execution-setting bounds,
OpenAI-compatible base endpoints, session identifiers, and output-contract
legality. Callers retain source parsing, required-field rules, other
source-specific normalization, defaulting, error classification, and policy
specific to their own boundary.
## Repository And Consumer Boundary
Scriptorium is a downstream application that consumes Promptkit through

View File

@@ -50,19 +50,18 @@ Examples of appropriate seams include clocks, randomness, subprocesses, remote A
## Test execution requirements
Promptkit currently uses maintainer-run validation rather than hosted CI.
Maintainers run the repository-documented test, vet, build, formatting,
documentation-link, and repository-hygiene checks before accepting changes.
Maintainers run the complete local workflow in the
[development guide](../development.md#maintainer-validation) before accepting
changes. That guide is the canonical owner of exact commands, formatting,
documentation-link validation, and repository-hygiene checks.
Introducing hosted CI later would supplement, not silently redefine, this
documented validation model.
The complete test sequence includes ordinary and race-enabled package tests.
The maintained offline consumer workflow is also run from the repository root:
```sh
go test ./...
go test -race ./...
go run ./examples/go-library/prepare
```
Maintainer validation must include ordinary and race-enabled package tests,
static analysis, a complete build, and execution of both maintained offline
consumer examples. The preparation example protects assembled preparation and
inspection behavior. The execution example separately protects assembled
`Run`, injected-client, validation, usage, and result behavior.
Tests in the default suite must be deterministic, offline, and independent of
real credentials. They must not invoke paid APIs, use live network
@@ -88,8 +87,9 @@ Use each test type where it protects a distinct risk:
interaction, while replacing live or nondeterministic external boundaries.
- External-package root tests exercise the public facade as a Go consumer,
while internal package tests own focused implementation behavior.
- The maintained offline preparation example protects one representative
assembled consumer workflow without contacting a model provider.
- The maintained offline preparation and execution examples protect distinct
representative assembled consumer workflows without contacting a model
provider.
- Fixtures should be minimal, synthetic, versioned with the behavior they
exercise, and free of credentials or private data.
- Golden files are appropriate only when the complete output is intentionally

View File

@@ -106,48 +106,12 @@ gitea.maximumdirect.net/eric/promptkit 1.25.5
promptkit gitea.maximumdirect.net/eric/promptkit
```
Run the complete maintainer validation required by the
[development guide](development.md):
```sh
go test ./...
go test -race ./...
go vet ./...
go build ./...
go run ./examples/go-library/prepare
```
Check every tracked Go file. This command must produce no output:
```sh
unformatted=$(
git ls-files '*.go' |
while IFS= read -r go_file
do
gofmt -l "$go_file"
done
)
test -z "$unformatted"
```
Follow every maintained Markdown link and confirm that its local or published
target exists. Review the repository for generated binaries, test or coverage
output, credentials, template residue, downloaded assets, and other files that
do not belong in source control.
Recheck module and repository hygiene, whitespace, and the clean checkout:
```sh
test -z "$(git ls-files go.work go.work.sum)"
test ! -e vendor
if grep -Eq '^[[:space:]]*replace([[:space:]]|\()' go.mod
then
printf '%s\n' 'go.mod contains a replacement' >&2
exit 1
fi
git diff --check
test -z "$(git status --porcelain)"
```
As a release prerequisite, run the complete
[maintainer validation workflow](development.md#maintainer-validation) against
the clean candidate. Do not substitute a partial command list: the development
guide owns the tests, race checks, analysis, build, both offline examples,
formatting, Markdown links, generated-output and credential review, and
repository hygiene. Record the successful workflow result with the candidate.
## Write The Release Note

131
docs/releases/v0.6.0.md Normal file
View File

@@ -0,0 +1,131 @@
# Promptkit v0.6.0
This supplemental changelog and migration guide summarizes the consumer-facing
changes from `v0.5.0` to `v0.6.0`. The annotated `v0.6.0` tag is the
authoritative release record. Exact current contracts belong to the linked
GoDoc and durable documentation.
## Summary
`v0.6.0` is a broad correctness, safety, efficiency, and maintainability
release. It does not add or remove public declarations. The release:
- centralizes shared execution-setting, output-contract, endpoint, and
JSON-compatible-value rules;
- unifies prompt repository behavior and avoids unnecessary prompt and profile
decoding;
- bounds consumer-controlled JSON trees and successful provider responses;
- hardens prompt content paths, artifact files, provider URLs, JSON framing,
and error propagation;
- reuses compiled schema plans and rendered artifact text within an operation;
and
- improves cancellation behavior, prepared-value ownership, deterministic
transport testing, and maintainer validation.
## Compatibility
No public declaration was added, removed, or changed. Ordinary valid `v0.5.0`
configurations and requests should continue to compile and behave as before.
The release intentionally rejects or reports several inputs that were
previously accepted, altered, or misclassified:
- execution settings must be finite, within their documented ranges, and safe
to convert to Go durations;
- output formats, validation modes, repair counts, and JSON Schema dependencies
are validated consistently;
- file-backed prompt and profile identity comes from normalized YAML metadata,
not filenames;
- prompt `content_file` values must be exact relative paths contained by their
configured source root;
- built-in file artifacts must resolve to regular files;
- selected provider endpoints must be absolute HTTP or HTTPS URLs without user
information, query strings, or fragments;
- JSON documents and successful provider responses must contain exactly one
value, and successful provider bodies are limited to 16 MiB; and
- excessively deep or expansive JSON-compatible values fail with ordinary
validation errors.
These are compatibility corrections and safety boundaries rather than new
consumer configuration requirements. Consumers relying on an invalid or
ambiguous input should correct that input before upgrading.
## Upgrade
Update the module dependency with:
```sh
go get gitea.maximumdirect.net/eric/promptkit@v0.6.0
go mod tidy
```
Run the consuming project's ordinary and race-enabled tests after upgrading.
Applications with custom prompt/profile sources, local provider endpoints,
unusual artifact paths, or assertions over provider error identities should
pay particular attention to the compatibility notes below.
## Source Loading And Identity
Prompt definitions now share one source-neutral selection and normalization
flow across operating-system and `fs.FS` sources. YAML `id` and `version`
metadata are authoritative; filenames do not create a second identity system.
Only selected content bodies are loaded, malformed unrelated definitions do
not shadow valid exact matches, and per-file read failures are reported as
prompt-load failures rather than false absence.
File-backed profiles likewise use normalized YAML IDs, reuse their metadata
read for selected strict decoding, and avoid fully decoding unrelated files.
Selected malformed definitions remain authoritative and do not silently fall
through to a lower-precedence source.
Prompt `content_file` paths are opened exactly as declared after a separate
blank check. They must remain relative to and contained by the configured
prompt source root, including across operating-system symlinks.
See the [framework source and identity reference](../formats.md) and
[internal source overview](../internal/sources.md) for the current contracts.
## Validation, Cancellation, And Efficiency
JSON Schema documents preserve exact JSON-number representations. Schema
resource URLs safely escape legal filesystem names, and each operation loads
and compiles its schema graph once. `Run` and prepared execution reuse that
operation-local plan; Promptkit does not introduce a cross-operation cache.
Artifact reading, rendering, schema loading, compilation, and validation now
check cancellation at the synchronous boundaries Promptkit controls. Rendering
memoizes each artifact's text within one render operation, while plain JSON
validation avoids materializing an unnecessary generic tree.
The shared JSON-compatible-value owner now limits nesting and produced work so
unsafe consumer-controlled structures return errors instead of risking
unbounded recursion or allocation. See the
[architecture policy](../policy/architecture.md) for invariant ownership and
the [format reference](../formats.md) for validation behavior.
## Provider Transport Hardening
OpenAI-compatible endpoints are parsed and composed structurally, including
nested base paths. Underlying transport cancellation and deadline errors remain
discoverable with `errors.Is` through Promptkit's generation error category.
Successful provider bodies are read with a fixed 16 MiB bound and must contain
exactly one JSON response object followed only by whitespace. Oversized,
truncated, malformed, or multiply framed responses fail without returning a
partial result. See the
[OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md)
for the canonical request, endpoint, error, and response behavior.
## Public API Changes
None.
## Consumer Action
- Correct any configuration or request that depends on the formerly permissive
cases described under Compatibility.
- Confirm custom local provider endpoints are absolute HTTP or HTTPS base URLs
without credentials, queries, or fragments.
- Confirm prompt content paths remain within their configured source root and
file artifacts resolve to regular files.
- Run ordinary and race-enabled consumer tests after updating the dependency.

View File

@@ -59,24 +59,3 @@ if supported at all, should be an explicit option and must not expose secrets.
Reconsider this idea after downstream use of `InspectPrompt`,
`InspectProfile`, and fixture-based preparation demonstrates a concrete gap.
### Structured Generation Errors
**Reason for deferral:** Existing `ErrLLMGenerate` classification, preserved
injected-client errors, and prepared execution details currently provide the
necessary failure boundary. A typed error should wait for stronger downstream
demand and a transport-neutral field design.
Promptkit could expose safe structured generation context through
`errors.As` while preserving `errors.Is(err, ErrLLMGenerate)`. Potential
fields include the selected backend ID and model plus an optional HTTP status
when the built-in OpenAI-compatible transport supplies one.
The design must not expose provider response bodies, endpoints, credential
environment names or values, request content, or generated content. It should
not duplicate prompt and profile provenance already available from a prepared
execution, and it must preserve the identity of errors returned by injected
clients. Retry and backoff policy remains a consumer responsibility.
Reconsider this idea when consumers need structured generation diagnostics
beyond the existing sentinel, wrapped client error, and preparation record.

View File

@@ -38,7 +38,33 @@ consumers.
## Ideas
No ideas currently await selection.
### Public bounded output repair
After the codebase-audit remediations are complete, Promptkit should make its
bounded output-repair capability available through the public engine. A
consumer should be able to request a limited number of corrective generation
attempts when JSON or JSON Schema output fails content validation, without
having to reproduce Promptkit's generation, validation, capacity, and result-
accounting orchestration.
- Repair is validation recovery, not a general provider retry, failover, or
backoff policy. Transport failures, cancellation, and operational schema or
validation errors must retain their ordinary error behavior.
- Repair must stop after the first valid result or the configured attempt
bound. Exhausting the bound should preserve the final invalid result and its
validation diagnostics rather than inventing success.
- Initial generation and every repair attempt must use the same resolved
backend, effective execution settings and presence semantics, session,
credential boundary, structured-output contract, and backend-capacity
policy.
- Results should report the number of repair attempts and cumulative usage for
every model call made by the run.
- Ordinary and prepared execution should expose coherent behavior, including
cancellation, frozen prepared state, error identity, and capacity lifetime.
Select this work only after the accepted audit findings affecting shared
execution invariants, validation, orchestration, transport, and repair
internals have been remediated.
## Entry Format

View File

@@ -0,0 +1,71 @@
# Structured Generation Errors
## Purpose
Promptkit should give downstream applications actionable, machine-readable
details when the built-in OpenAI-compatible client receives a non-success HTTP
response. Today the client reports only the status code and discards the
provider response body. This makes ordinary configuration failures—such as an
unsupported strict JSON Schema keyword—unnecessarily difficult to diagnose.
## Target End State
Failures from the built-in transport are available through a public typed error
that works with `errors.As` while continuing to match `ErrLLMGenerate` through
`errors.Is`. The error should expose:
- the HTTP status code;
- a normalized provider error code or type when supplied; and
- a bounded provider message extracted from a recognized OpenAI-compatible
JSON error envelope.
The ordinary `Error()` string should remain safe and concise: it should include
the status and provider code or type, but not automatically include the
provider message. Consumers that deliberately want the provider's diagnostic
text can retrieve it from the typed error and apply their own disclosure and
logging policy.
This contract should be available for both ordinary and prepared execution.
Errors returned by injected model clients must continue to preserve their own
identity and should not be converted into fabricated HTTP details.
## Safety And Compatibility Boundaries
- Never expose the raw response body, response headers, endpoint, credentials,
request messages, schema document, or generated content through this API.
- Read only a small fixed maximum response body, reject malformed or
unrecognized envelopes, normalize invalid UTF-8 and control characters, and
cap every retained diagnostic field independently.
- Treat the extracted provider message as untrusted and potentially sensitive:
its GoDoc must tell consumers not to log or display it without applying their
own policy.
- Preserve the existing generic behavior when a response is empty, non-JSON,
oversized, or does not match a recognized error envelope.
- Do not assign retryability from an HTTP status. Promptkit supplies facts;
downstream applications retain retry and presentation policy.
## Recommended API Direction
Prefer one immutable public `GenerationError` value, constructed internally and
carrying accessors for HTTP status, provider code or type, and provider message.
This keeps the exact representation evolvable while giving consumers an
idiomatic `errors.As` contract. Public Go declarations and GoDoc should own the
final exact names and semantics.
The internal OpenAI-compatible client should parse only the conventional
top-level `error` envelope and pass normalized details through the use-case and
public error-mapping layers. The integration documentation should continue to
own wire behavior; the public declarations should own the consumer contract.
## Acceptance Criteria
- A downstream consumer can distinguish a provider HTTP 400 from other
generation failures and obtain a bounded provider explanation when present.
- The typed error still satisfies `errors.Is(err, ErrLLMGenerate)`.
- Existing cancellation, capacity, validation, and injected-client error
identities remain unchanged.
- Tests cover recognized string and numeric provider codes, absent and malformed
envelopes, oversized bodies and fields, control characters, and error-chain
behavior without making live provider requests.
- Current-state GoDoc and the OpenAI-compatible integration and internal-client
documents are updated only when the implementation lands.

View File

@@ -219,7 +219,7 @@ func WithPromptFile(path string) Option {
if err != nil {
return err
}
options.promptDefs = promptdef.NewFSRepository(fsys, root)
options.promptDefs = promptdef.NewFileRepository(fsys, root, filepath.Dir(path))
options.promptSource = true
return nil
})
@@ -461,21 +461,20 @@ func newProfileRepository(profileDir string, options engineOptions) profile.Repo
}
func fileSource(name string) (fs.FS, string, error) {
cleanName := strings.TrimSpace(name)
if cleanName == "" {
if strings.TrimSpace(name) == "" {
return nil, "", ErrInvalidConfig
}
dir := filepath.Dir(cleanName)
base := filepath.Base(cleanName)
if base == "." || base == string(filepath.Separator) || strings.TrimSpace(base) == "" {
dir := filepath.Dir(name)
base := filepath.Base(name)
if base == "." || base == string(filepath.Separator) {
return nil, "", ErrInvalidConfig
}
info, err := os.Stat(cleanName)
info, err := os.Stat(name)
if err != nil {
return nil, "", fmt.Errorf("%w: failed to access source file %q: %v", ErrInvalidConfig, cleanName, err)
return nil, "", fmt.Errorf("%w: failed to access source file %q: %v", ErrInvalidConfig, name, err)
}
if info.IsDir() {
return nil, "", fmt.Errorf("%w: source path %q must be a file", ErrInvalidConfig, cleanName)
return nil, "", fmt.Errorf("%w: source path %q must be a file", ErrInvalidConfig, name)
}
return os.DirFS(dir), filepath.ToSlash(base), nil
}

File diff suppressed because it is too large Load Diff

View File

@@ -16,10 +16,12 @@ import (
var (
ErrUnsupportedRefType = errors.New("unsupported artifact reference type")
ErrMissingInlineBody = errors.New("missing body for inline artifact")
ErrMissingFilePath = errors.New("missing file path for file artifact")
ErrUnsupportedFile = errors.New("file artifact path is not a regular file")
)
const fileReadChunkSize = 64 * 1024
// Reader resolves artifact references into actual artifacts.
type Reader interface {
Read(ctx context.Context, ref domain.ArtifactRef) (*domain.Artifact, error)
@@ -34,7 +36,7 @@ type CompositeReader struct {
func NewCompositeReader() Reader {
return &CompositeReader{
inlineReader: &inlineReader{},
fileReader: &fileReader{},
fileReader: &fileReader{open: openArtifactFile},
}
}
@@ -64,10 +66,6 @@ func (r *inlineReader) Read(ctx context.Context, ref domain.ArtifactRef) (*domai
default:
}
if ref.Body == "" {
return nil, ErrMissingInlineBody
}
body := []byte(ref.Body)
return &domain.Artifact{
ContentType: defaults.ContentTypeTextPlain,
@@ -78,7 +76,15 @@ func (r *inlineReader) Read(ctx context.Context, ref domain.ArtifactRef) (*domai
}, nil
}
type fileReader struct{}
type artifactFile interface {
Read([]byte) (int, error)
Stat() (os.FileInfo, error)
Close() error
}
type fileReader struct {
open func(string) (artifactFile, error)
}
func (r *fileReader) Read(ctx context.Context, ref domain.ArtifactRef) (*domain.Artifact, error) {
select {
@@ -91,25 +97,71 @@ func (r *fileReader) Read(ctx context.Context, ref domain.ArtifactRef) (*domain.
return nil, ErrMissingFilePath
}
return readFileArtifact(ref.URI)
return readFileArtifact(ctx, ref.URI, r.open)
}
func readFileArtifact(path string) (*domain.Artifact, error) {
file, err := os.Open(path)
func openArtifactFile(path string) (artifactFile, error) {
return os.Open(path)
}
func readFileArtifact(ctx context.Context, path string, open func(string) (artifactFile, error)) (*domain.Artifact, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
info, err := os.Stat(path)
if err != nil {
return nil, fmt.Errorf("failed to read file %s: %w", path, err)
}
if !info.Mode().IsRegular() {
return nil, fmt.Errorf("%w: %s", ErrUnsupportedFile, path)
}
if err := ctx.Err(); err != nil {
return nil, err
}
file, err := open(path)
if err != nil {
return nil, fmt.Errorf("failed to read file %s: %w", path, err)
}
defer file.Close()
data, err := io.ReadAll(file)
openedInfo, err := file.Stat()
if err != nil {
return nil, fmt.Errorf("failed to read file %s: %w", path, err)
return nil, fmt.Errorf("failed to inspect opened file %s: %w", path, err)
}
if !openedInfo.Mode().IsRegular() {
return nil, fmt.Errorf("%w: %s", ErrUnsupportedFile, path)
}
data := make([]byte, 0)
chunk := make([]byte, fileReadChunkSize)
for {
if err := ctx.Err(); err != nil {
return nil, err
}
n, readErr := file.Read(chunk)
if n > 0 {
data = append(data, chunk[:n]...)
}
if err := ctx.Err(); err != nil {
return nil, err
}
if errors.Is(readErr, io.EOF) {
break
}
if readErr != nil {
return nil, fmt.Errorf("failed to read file %s: %w", path, readErr)
}
}
contentType := mime.TypeByExtension(filepath.Ext(path))
if contentType == "" {
contentType = defaults.ContentTypeTextPlain
}
hash := fmt.Sprintf("%x", sha256.Sum256(data))
if err := ctx.Err(); err != nil {
return nil, err
}
return &domain.Artifact{
Name: filepath.Base(path),
@@ -117,6 +169,6 @@ func readFileArtifact(path string) (*domain.Artifact, error) {
Body: data,
URI: path,
Size: int64(len(data)),
Hash: fmt.Sprintf("%x", sha256.Sum256(data)),
Hash: hash,
}, nil
}

View File

@@ -0,0 +1,43 @@
//go:build linux
package artifact
import (
"context"
"errors"
"path/filepath"
"syscall"
"testing"
"time"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
)
func TestFileReaderRejectsFIFOBeforeOpen(t *testing.T) {
path := filepath.Join(t.TempDir(), "artifact.fifo")
if err := syscall.Mkfifo(path, 0o600); err != nil {
t.Fatalf("create fifo: %v", err)
}
type result struct {
artifact *domain.Artifact
err error
}
done := make(chan result, 1)
go func() {
artifact, err := NewCompositeReader().Read(context.Background(), domain.ArtifactRef{
Type: domain.ArtifactRefFile,
URI: path,
})
done <- result{artifact: artifact, err: err}
}()
select {
case got := <-done:
if got.artifact != nil || !errors.Is(got.err, ErrUnsupportedFile) {
t.Fatalf("artifact=%#v err=%v, want nil/ErrUnsupportedFile", got.artifact, got.err)
}
case <-time.After(time.Second):
t.Fatal("FIFO read blocked instead of rejecting the non-regular file")
}
}

View File

@@ -1,6 +1,7 @@
package artifact
import (
"bytes"
"context"
"errors"
"os"
@@ -11,54 +12,97 @@ import (
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
)
func TestCompositeReader_Read(t *testing.T) {
func TestCompositeReaderRejectsUnsupportedReferences(t *testing.T) {
_, err := NewCompositeReader().Read(context.Background(), domain.ArtifactRef{
Type: domain.ArtifactRefType("unsupported"),
URI: "unsupported://bucket/key",
})
if !errors.Is(err, ErrUnsupportedRefType) {
t.Fatalf("expected ErrUnsupportedRefType, got %v", err)
}
}
func TestCompositeReaderSourceParityAndOpaqueHashes(t *testing.T) {
reader := NewCompositeReader()
ctx := context.Background()
hashes := make(map[string]string)
tests := []struct {
name string
content string
}{
{name: "empty", content: ""},
{name: "ordinary", content: "same content"},
{name: "changed", content: "changed content"},
}
t.Run("inline artifact", func(t *testing.T) {
ref := domain.ArtifactRef{
Type: domain.ArtifactRefInline,
Body: "hello world",
}
art, err := reader.Read(ctx, ref)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if string(art.Body) != "hello world" {
t.Errorf("expected 'hello world', got %s", string(art.Body))
}
if art.ContentType != "text/plain" {
t.Errorf("expected text/plain content type, got %q", art.ContentType)
}
if art.Hash != "b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9" {
t.Errorf("unexpected hash: %s", art.Hash)
}
if art.Size != int64(len(ref.Body)) {
t.Errorf("expected size %d, got %d", len(ref.Body), art.Size)
}
})
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
filePath := filepath.Join(t.TempDir(), "artifact.txt")
if err := os.WriteFile(filePath, []byte(tc.content), 0o600); err != nil {
t.Fatal(err)
}
t.Run("inline artifact missing body", func(t *testing.T) {
ref := domain.ArtifactRef{
Type: domain.ArtifactRefInline,
Body: "",
}
_, err := reader.Read(ctx, ref)
if !errors.Is(err, ErrMissingInlineBody) {
t.Errorf("expected ErrMissingInlineBody, got %v", err)
}
})
sources := []struct {
name string
ref domain.ArtifactRef
wantURI string
}{
{
name: "inline",
ref: domain.ArtifactRef{Type: domain.ArtifactRefInline, Body: tc.content},
},
{
name: "inline with uri",
ref: domain.ArtifactRef{Type: domain.ArtifactRefInline, URI: "memory://input", Body: tc.content},
wantURI: "memory://input",
},
{
name: "file",
ref: domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: filePath},
wantURI: filePath,
},
}
t.Run("unsupported ref type", func(t *testing.T) {
ref := domain.ArtifactRef{
Type: domain.ArtifactRefType("unsupported"),
URI: "unsupported://bucket/key",
}
_, err := reader.Read(ctx, ref)
if !errors.Is(err, ErrUnsupportedRefType) {
t.Error("expected error for unsupported type")
}
})
var sourceHash string
for _, source := range sources {
t.Run(source.name, func(t *testing.T) {
first, err := reader.Read(context.Background(), source.ref)
if err != nil {
t.Fatalf("first read: %v", err)
}
second, err := reader.Read(context.Background(), source.ref)
if err != nil {
t.Fatalf("second read: %v", err)
}
if string(first.Body) != tc.content || first.Size != int64(len(tc.content)) {
t.Fatalf("body=%q size=%d, want %q/%d", first.Body, first.Size, tc.content, len(tc.content))
}
if first.URI != source.wantURI {
t.Fatalf("URI = %q, want %q", first.URI, source.wantURI)
}
if first.Hash == "" || first.Hash != second.Hash {
t.Fatalf("hashes are not non-empty and stable: %q/%q", first.Hash, second.Hash)
}
if sourceHash == "" {
sourceHash = first.Hash
} else if first.Hash != sourceHash {
t.Fatalf("equal content hashes differ: %q/%q", sourceHash, first.Hash)
}
if source.ref.Type == domain.ArtifactRefFile {
if first.Name != filepath.Base(filePath) || !strings.HasPrefix(first.ContentType, "text/plain") {
t.Fatalf("unexpected file metadata: %+v", first)
}
} else if first.ContentType != "text/plain" {
t.Fatalf("inline content type = %q", first.ContentType)
}
})
}
hashes[tc.name] = sourceHash
})
}
if hashes["empty"] == hashes["ordinary"] || hashes["ordinary"] == hashes["changed"] {
t.Fatalf("changed content did not change opaque hash: %#v", hashes)
}
}
func TestCompositeReaderCopiesInlineData(t *testing.T) {
@@ -87,94 +131,154 @@ func TestCompositeReaderCopiesInlineData(t *testing.T) {
}
}
func TestCompositeReaderHonorsCancellation(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
func TestCompositeReaderHonorsPreCancellation(t *testing.T) {
filePath := filepath.Join(t.TempDir(), "artifact.txt")
if err := os.WriteFile(filePath, []byte("ignored"), 0o600); err != nil {
t.Fatal(err)
}
tests := []struct {
name string
ref domain.ArtifactRef
}{
{name: "inline", ref: domain.ArtifactRef{Type: domain.ArtifactRefInline, Body: "ignored"}},
{name: "file", ref: domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: filePath}},
}
_, err := NewCompositeReader().Read(ctx, domain.ArtifactRef{
Type: domain.ArtifactRefInline,
Body: "ignored",
})
if !errors.Is(err, context.Canceled) {
t.Fatalf("expected context cancellation, got %v", err)
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
artifact, err := NewCompositeReader().Read(ctx, tc.ref)
if artifact != nil || !errors.Is(err, context.Canceled) {
t.Fatalf("artifact=%#v err=%v, want nil/context.Canceled", artifact, err)
}
})
}
}
func TestFileReader_Read(t *testing.T) {
content := []byte("test file content")
filePath := filepath.Join(t.TempDir(), "artifact.txt")
if err := os.WriteFile(filePath, content, 0o600); err != nil {
t.Fatal(err)
}
func TestFileReaderFailuresAndMetadata(t *testing.T) {
reader := NewCompositeReader()
ctx := context.Background()
t.Run("file artifact loading", func(t *testing.T) {
ref := domain.ArtifactRef{
Type: domain.ArtifactRefFile,
URI: filePath,
}
art, err := reader.Read(ctx, ref)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if string(art.Body) != string(content) {
t.Errorf("expected %s, got %s", string(content), string(art.Body))
}
if art.Name != filepath.Base(filePath) {
t.Errorf("expected name %q, got %q", filepath.Base(filePath), art.Name)
}
if !strings.HasPrefix(art.ContentType, "text/plain") {
t.Errorf("expected text content type, got %q", art.ContentType)
}
if art.URI != filePath {
t.Errorf("expected URI %q, got %q", filePath, art.URI)
}
if art.Size != int64(len(content)) {
t.Errorf("expected size %d, got %d", len(content), art.Size)
}
if art.Hash != "60f5237ed4049f0382661ef009d2bc42e48c3ceb3edb6600f7024e7ab3b838f3" {
t.Errorf("unexpected hash: %s", art.Hash)
}
})
t.Run("missing file path", func(t *testing.T) {
ref := domain.ArtifactRef{
Type: domain.ArtifactRefFile,
URI: "",
}
_, err := reader.Read(ctx, ref)
_, err := reader.Read(context.Background(), domain.ArtifactRef{Type: domain.ArtifactRefFile})
if !errors.Is(err, ErrMissingFilePath) {
t.Errorf("expected ErrMissingFilePath, got %v", err)
t.Fatalf("expected ErrMissingFilePath, got %v", err)
}
})
t.Run("missing file", func(t *testing.T) {
ref := domain.ArtifactRef{
_, err := reader.Read(context.Background(), domain.ArtifactRef{
Type: domain.ArtifactRefFile,
URI: filepath.Join(t.TempDir(), "missing.txt"),
}
if _, err := reader.Read(ctx, ref); err == nil {
})
if err == nil {
t.Fatal("expected missing file error")
}
})
t.Run("directory rejected before open", func(t *testing.T) {
artifact, err := reader.Read(context.Background(), domain.ArtifactRef{
Type: domain.ArtifactRefFile,
URI: t.TempDir(),
})
if artifact != nil || !errors.Is(err, ErrUnsupportedFile) {
t.Fatalf("artifact=%#v err=%v, want nil/ErrUnsupportedFile", artifact, err)
}
})
t.Run("non-regular opened target rejected", func(t *testing.T) {
filePath := filepath.Join(t.TempDir(), "artifact.txt")
if err := os.WriteFile(filePath, []byte("content"), 0o600); err != nil {
t.Fatal(err)
}
directoryInfo, err := os.Stat(t.TempDir())
if err != nil {
t.Fatal(err)
}
fileReader := &fileReader{open: func(path string) (artifactFile, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
return &reportedInfoFile{artifactFile: file, info: directoryInfo}, nil
}}
artifact, err := fileReader.Read(context.Background(), domain.ArtifactRef{
Type: domain.ArtifactRefFile,
URI: filePath,
})
if artifact != nil || !errors.Is(err, ErrUnsupportedFile) {
t.Fatalf("artifact=%#v err=%v, want nil/ErrUnsupportedFile", artifact, err)
}
})
t.Run("unknown extension uses text fallback", func(t *testing.T) {
path := filepath.Join(t.TempDir(), "artifact.unknownextension")
if err := os.WriteFile(path, content, 0o600); err != nil {
filePath := filepath.Join(t.TempDir(), "artifact.unknownextension")
if err := os.WriteFile(filePath, []byte("content"), 0o600); err != nil {
t.Fatal(err)
}
art, err := reader.Read(ctx, domain.ArtifactRef{
artifact, err := reader.Read(context.Background(), domain.ArtifactRef{
Type: domain.ArtifactRefFile,
URI: path,
URI: filePath,
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
t.Fatalf("read artifact: %v", err)
}
if art.ContentType != "text/plain" {
t.Errorf("expected text/plain fallback, got %q", art.ContentType)
if artifact.ContentType != "text/plain" {
t.Fatalf("content type = %q", artifact.ContentType)
}
})
}
func TestFileReaderCancelsAfterReadProgress(t *testing.T) {
filePath := filepath.Join(t.TempDir(), "artifact.bin")
content := bytes.Repeat([]byte("x"), fileReadChunkSize*2)
if err := os.WriteFile(filePath, content, 0o600); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
var opened *cancelAfterProgressFile
reader := &fileReader{open: func(path string) (artifactFile, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
opened = &cancelAfterProgressFile{artifactFile: file, cancel: cancel}
return opened, nil
}}
artifact, err := reader.Read(ctx, domain.ArtifactRef{Type: domain.ArtifactRefFile, URI: filePath})
if artifact != nil || !errors.Is(err, context.Canceled) {
t.Fatalf("artifact=%#v err=%v, want nil/context.Canceled", artifact, err)
}
if opened == nil || opened.reads != 1 {
t.Fatalf("read count = %v, want one progressing read", opened)
}
}
type reportedInfoFile struct {
artifactFile
info os.FileInfo
}
func (f *reportedInfoFile) Stat() (os.FileInfo, error) {
return f.info, nil
}
type cancelAfterProgressFile struct {
artifactFile
cancel context.CancelFunc
reads int
}
func (f *cancelAfterProgressFile) Read(buffer []byte) (int, error) {
n, err := f.artifactFile.Read(buffer)
if n > 0 {
f.reads++
f.cancel()
}
return n, err
}

View File

@@ -5,7 +5,6 @@ package backend
import (
"errors"
"fmt"
"net/url"
"regexp"
"sort"
"strings"
@@ -109,10 +108,11 @@ func (r *Registry) CapacityPolicies() map[string]domain.BackendCapacityPolicy {
}
func normalizeBackend(definition domain.Backend) (domain.Backend, error) {
definition.Endpoint = strings.TrimSpace(definition.Endpoint)
if err := validateEndpoint(definition.Endpoint); err != nil {
endpoint, err := domain.NormalizeOpenAICompatibleBaseEndpoint(definition.Endpoint)
if err != nil {
return domain.Backend{}, fmt.Errorf("backend %q endpoint: %w", definition.ID, err)
}
definition.Endpoint = endpoint
definition.APIKeyEnv = strings.TrimSpace(definition.APIKeyEnv)
if definition.APIKeyEnv != "" && !environmentVariableName.MatchString(definition.APIKeyEnv) {
@@ -182,31 +182,3 @@ func normalizeBackend(definition domain.Backend) (domain.Backend, error) {
definition.ExtraParams = extraParams
return definition, nil
}
func validateEndpoint(endpoint string) error {
if endpoint == "" {
return errors.New("must not be blank")
}
if strings.Contains(endpoint, "#") {
return errors.New("must not contain a fragment")
}
parsed, err := url.Parse(endpoint)
if err != nil {
return fmt.Errorf("must be a valid URL: %w", err)
}
scheme := strings.ToLower(parsed.Scheme)
if scheme != "http" && scheme != "https" {
return errors.New("must use http or https")
}
if !parsed.IsAbs() || parsed.Hostname() == "" {
return errors.New("must be absolute and include a host")
}
if parsed.User != nil {
return errors.New("must not contain user information")
}
if parsed.RawQuery != "" || parsed.ForceQuery {
return errors.New("must not contain a query string")
}
return nil
}

View File

@@ -15,10 +15,7 @@ const (
OpenAIChatCompletionsPath = "/chat/completions"
ExecutionDefaultTimeoutSeconds = 600
)
var (
LLMRequestTimeoutDefault = 10 * time.Minute
LLMRequestTimeoutDefault = 10 * time.Minute
)
func ExecutionTargetDefault() domain.ExecutionTarget {

View File

@@ -97,22 +97,22 @@ type RunResult struct {
// PreparedRun contains pre-LLM execution state from the prepare/render phase.
// It must never include resolved API key values, model output, or validation data.
type PreparedRun struct {
PromptID string `json:"prompt_id"`
PromptVersion string `json:"prompt_version,omitempty"`
PromptHash string `json:"prompt_hash,omitempty"`
SelectedProfileID string `json:"selected_profile_id"`
SelectedBackendID string `json:"selected_backend_id,omitempty"`
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
TargetPresence ExecutionTargetPresence `json:"-"`
OutputContract OutputContract `json:"output_contract"`
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
InputHashes map[string]string `json:"input_hashes,omitempty"`
SessionID string `json:"session_id,omitempty"`
RenderedPromptHash string `json:"rendered_prompt_hash"`
Messages []RenderedMessage `json:"messages"`
StartTime time.Time `json:"start_time,omitempty"`
EndTime time.Time `json:"end_time,omitempty"`
DurationMS int64 `json:"duration_ms,omitempty"`
PromptID string
PromptVersion string
PromptHash string
SelectedProfileID string
SelectedBackendID string
EffectiveModelParams ExecutionTarget
TargetPresence ExecutionTargetPresence
OutputContract OutputContract
StructuredOutput *StructuredOutputSpec
InputHashes map[string]string
SessionID string
RenderedPromptHash string
Messages []RenderedMessage
StartTime time.Time
EndTime time.Time
DurationMS int64
}
// ArtifactRef represents a reference to an input artifact.

View File

@@ -0,0 +1,39 @@
package domain
import (
"errors"
"net/url"
"strings"
)
// NormalizeOpenAICompatibleBaseEndpoint trims and validates a source-neutral
// OpenAI-compatible provider base endpoint.
func NormalizeOpenAICompatibleBaseEndpoint(endpoint string) (string, error) {
endpoint = strings.TrimSpace(endpoint)
if endpoint == "" {
return "", errors.New("endpoint must not be blank")
}
if strings.Contains(endpoint, "#") {
return "", errors.New("endpoint must not contain a fragment")
}
parsed, err := url.Parse(endpoint)
if err != nil {
return "", errors.New("endpoint must be a valid URL")
}
parsed.Scheme = strings.ToLower(parsed.Scheme)
if parsed.Scheme != "http" && parsed.Scheme != "https" {
return "", errors.New("endpoint must use http or https")
}
if !parsed.IsAbs() || parsed.Hostname() == "" {
return "", errors.New("endpoint must be absolute and include a host")
}
if parsed.User != nil {
return "", errors.New("endpoint must not contain user information")
}
if parsed.RawQuery != "" || parsed.ForceQuery {
return "", errors.New("endpoint must not contain a query string")
}
return parsed.String(), nil
}

View File

@@ -0,0 +1,47 @@
package domain
import "testing"
func TestNormalizeOpenAICompatibleBaseEndpoint(t *testing.T) {
tests := []struct {
name string
endpoint string
want string
wantErr bool
}{
{name: "http host", endpoint: "http://provider.example", want: "http://provider.example"},
{name: "https nested path and whitespace", endpoint: " HTTPS://provider.example/api/openai/v1 ", want: "https://provider.example/api/openai/v1"},
{name: "IPv4 host and port", endpoint: "http://127.0.0.1:8080/v1", want: "http://127.0.0.1:8080/v1"},
{name: "IPv6 host and port", endpoint: "https://[::1]:8443/v1", want: "https://[::1]:8443/v1"},
{name: "repeated trailing slashes", endpoint: "https://provider.example/v1///", want: "https://provider.example/v1///"},
{name: "blank", endpoint: " \t\n ", wantErr: true},
{name: "relative path", endpoint: "/api/v1", wantErr: true},
{name: "scheme relative", endpoint: "//provider.example/v1", wantErr: true},
{name: "missing host", endpoint: "https:///v1", wantErr: true},
{name: "unsupported scheme", endpoint: "ftp://provider.example/v1", wantErr: true},
{name: "user information", endpoint: "https://user:secret@provider.example/v1", wantErr: true},
{name: "query", endpoint: "https://provider.example/v1?mode=chat", wantErr: true},
{name: "empty query", endpoint: "https://provider.example/v1?", wantErr: true},
{name: "fragment", endpoint: "https://provider.example/v1#chat", wantErr: true},
{name: "empty fragment", endpoint: "https://provider.example/v1#", wantErr: true},
{name: "malformed URL", endpoint: "https://provider.example/%zz", wantErr: true},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
got, err := NormalizeOpenAICompatibleBaseEndpoint(tc.endpoint)
if tc.wantErr {
if err == nil {
t.Fatalf("expected endpoint error, got %q", got)
}
return
}
if err != nil {
t.Fatalf("normalize endpoint: %v", err)
}
if got != tc.want {
t.Fatalf("normalized endpoint = %q, want %q", got, tc.want)
}
})
}
}

View File

@@ -0,0 +1,34 @@
package domain
import (
"errors"
"math"
"time"
)
const maxExecutionTimeoutSeconds int64 = math.MaxInt64 / int64(time.Second)
// ValidateExecutionTargetSettings validates source-neutral execution-setting
// invariants on a resolved target.
func ValidateExecutionTargetSettings(target ExecutionTarget) error {
if !isFinite(target.Temperature) || target.Temperature < 0 || target.Temperature > 2 {
return errors.New("temperature must be finite and between 0 and 2")
}
if target.MaxTokens < 0 {
return errors.New("max_tokens must be greater than or equal to 0")
}
if !isFinite(target.TopP) || target.TopP < 0 || target.TopP > 1 {
return errors.New("top_p must be finite and between 0 and 1")
}
if target.TimeoutSeconds < 0 {
return errors.New("timeout_seconds must be greater than or equal to 0")
}
if int64(target.TimeoutSeconds) > maxExecutionTimeoutSeconds {
return errors.New("timeout_seconds exceeds the maximum supported duration")
}
return nil
}
func isFinite(value float64) bool {
return !math.IsNaN(value) && !math.IsInf(value, 0)
}

View File

@@ -0,0 +1,73 @@
package domain
import (
"math"
"strconv"
"strings"
"testing"
)
func TestValidateExecutionTargetSettings(t *testing.T) {
valid := ExecutionTarget{
Temperature: 1,
MaxTokens: 1,
TopP: 0.5,
TimeoutSeconds: 1,
}
type testCase struct {
name string
change func(*ExecutionTarget)
wantErr string
}
tests := []testCase{
{name: "temperature lower boundary", change: func(v *ExecutionTarget) { v.Temperature = 0 }},
{name: "temperature finite lower neighbor", change: func(v *ExecutionTarget) { v.Temperature = math.Nextafter(0, 1) }},
{name: "temperature finite upper neighbor", change: func(v *ExecutionTarget) { v.Temperature = math.Nextafter(2, 0) }},
{name: "temperature upper boundary", change: func(v *ExecutionTarget) { v.Temperature = 2 }},
{name: "temperature below lower boundary", change: func(v *ExecutionTarget) { v.Temperature = math.Nextafter(0, math.Inf(-1)) }, wantErr: "temperature"},
{name: "temperature above upper boundary", change: func(v *ExecutionTarget) { v.Temperature = math.Nextafter(2, math.Inf(1)) }, wantErr: "temperature"},
{name: "temperature NaN", change: func(v *ExecutionTarget) { v.Temperature = math.NaN() }, wantErr: "temperature"},
{name: "temperature positive infinity", change: func(v *ExecutionTarget) { v.Temperature = math.Inf(1) }, wantErr: "temperature"},
{name: "temperature negative infinity", change: func(v *ExecutionTarget) { v.Temperature = math.Inf(-1) }, wantErr: "temperature"},
{name: "max tokens lower boundary", change: func(v *ExecutionTarget) { v.MaxTokens = 0 }},
{name: "max tokens finite neighbor", change: func(v *ExecutionTarget) { v.MaxTokens = 1 }},
{name: "max tokens below lower boundary", change: func(v *ExecutionTarget) { v.MaxTokens = -1 }, wantErr: "max_tokens"},
{name: "top p lower boundary", change: func(v *ExecutionTarget) { v.TopP = 0 }},
{name: "top p finite lower neighbor", change: func(v *ExecutionTarget) { v.TopP = math.Nextafter(0, 1) }},
{name: "top p finite upper neighbor", change: func(v *ExecutionTarget) { v.TopP = math.Nextafter(1, 0) }},
{name: "top p upper boundary", change: func(v *ExecutionTarget) { v.TopP = 1 }},
{name: "top p below lower boundary", change: func(v *ExecutionTarget) { v.TopP = math.Nextafter(0, math.Inf(-1)) }, wantErr: "top_p"},
{name: "top p above upper boundary", change: func(v *ExecutionTarget) { v.TopP = math.Nextafter(1, math.Inf(1)) }, wantErr: "top_p"},
{name: "top p NaN", change: func(v *ExecutionTarget) { v.TopP = math.NaN() }, wantErr: "top_p"},
{name: "top p positive infinity", change: func(v *ExecutionTarget) { v.TopP = math.Inf(1) }, wantErr: "top_p"},
{name: "top p negative infinity", change: func(v *ExecutionTarget) { v.TopP = math.Inf(-1) }, wantErr: "top_p"},
{name: "timeout lower boundary", change: func(v *ExecutionTarget) { v.TimeoutSeconds = 0 }},
{name: "timeout finite neighbor", change: func(v *ExecutionTarget) { v.TimeoutSeconds = 1 }},
{name: "timeout below lower boundary", change: func(v *ExecutionTarget) { v.TimeoutSeconds = -1 }, wantErr: "timeout_seconds"},
}
if strconv.IntSize == 64 {
durationLimit := maxExecutionTimeoutSeconds
tests = append(tests,
testCase{name: "timeout duration boundary", change: func(v *ExecutionTarget) { v.TimeoutSeconds = int(durationLimit) }},
testCase{name: "timeout above duration boundary", change: func(v *ExecutionTarget) { v.TimeoutSeconds = int(durationLimit) + 1 }, wantErr: "timeout_seconds"},
)
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
target := valid
tt.change(&target)
err := ValidateExecutionTargetSettings(target)
if tt.wantErr == "" {
if err != nil {
t.Fatalf("validate execution settings: %v", err)
}
return
}
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("error = %v, want diagnostic containing %q", err, tt.wantErr)
}
})
}
}

View File

@@ -0,0 +1,30 @@
package domain
import (
"errors"
"fmt"
"strings"
)
// ValidateOutputContract validates source-neutral output-contract invariants.
func ValidateOutputContract(contract OutputContract) error {
switch contract.Format {
case FormatText, FormatMarkdown, FormatJSON:
default:
return fmt.Errorf("invalid output format: %q", contract.Format)
}
switch contract.ValidationMode {
case ValidationNone, ValidationBasic, ValidationJSON, ValidationJSONSchema:
default:
return fmt.Errorf("invalid validation mode: %q", contract.ValidationMode)
}
if contract.ValidationMode == ValidationJSONSchema && strings.TrimSpace(contract.SchemaPath) == "" {
return errors.New("schema_path is required when validation_mode is json_schema")
}
if contract.RepairAttempts < 0 {
return errors.New("repair_attempts must be greater than or equal to 0")
}
return nil
}

View File

@@ -0,0 +1,68 @@
package domain
import (
"strings"
"testing"
)
func TestValidateOutputContract(t *testing.T) {
valid := OutputContract{
Format: FormatText,
ValidationMode: ValidationNone,
}
tests := []struct {
name string
change func(*OutputContract)
wantErr string
}{
{name: "text format", change: func(c *OutputContract) { c.Format = FormatText }},
{name: "markdown format", change: func(c *OutputContract) { c.Format = FormatMarkdown }},
{name: "json format", change: func(c *OutputContract) { c.Format = FormatJSON }},
{name: "empty format", change: func(c *OutputContract) { c.Format = "" }, wantErr: "format"},
{name: "unsupported format", change: func(c *OutputContract) { c.Format = OutputFormat("binary") }, wantErr: "format"},
{name: "none validation", change: func(c *OutputContract) { c.ValidationMode = ValidationNone }},
{name: "basic validation", change: func(c *OutputContract) { c.ValidationMode = ValidationBasic }},
{name: "json validation", change: func(c *OutputContract) { c.ValidationMode = ValidationJSON }},
{name: "json schema validation", change: func(c *OutputContract) {
c.ValidationMode = ValidationJSONSchema
c.SchemaPath = "schema.json"
}},
{name: "empty validation mode", change: func(c *OutputContract) { c.ValidationMode = "" }, wantErr: "validation mode"},
{name: "unsupported validation mode", change: func(c *OutputContract) { c.ValidationMode = ValidationMode("unknown") }, wantErr: "validation mode"},
{name: "negative repair attempts", change: func(c *OutputContract) { c.RepairAttempts = -1 }, wantErr: "repair_attempts"},
{name: "zero repair attempts", change: func(c *OutputContract) { c.RepairAttempts = 0 }},
{name: "positive repair attempts", change: func(c *OutputContract) { c.RepairAttempts = 1 }},
{name: "json schema empty path", change: func(c *OutputContract) {
c.ValidationMode = ValidationJSONSchema
c.SchemaPath = ""
}, wantErr: "schema_path"},
{name: "json schema whitespace path", change: func(c *OutputContract) {
c.ValidationMode = ValidationJSONSchema
c.SchemaPath = " \t "
}, wantErr: "schema_path"},
{name: "json schema nonblank path", change: func(c *OutputContract) {
c.ValidationMode = ValidationJSONSchema
c.SchemaPath = " schema.json "
}},
{name: "non-schema empty path", change: func(c *OutputContract) { c.SchemaPath = "" }},
{name: "non-schema populated path", change: func(c *OutputContract) { c.SchemaPath = "ignored.json" }},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
contract := valid
tt.change(&contract)
err := ValidateOutputContract(contract)
if tt.wantErr == "" {
if err != nil {
t.Fatalf("validate output contract: %v", err)
}
return
}
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("error = %v, want diagnostic containing %q", err, tt.wantErr)
}
})
}
}

View File

@@ -1,141 +0,0 @@
package domain
import (
"encoding/json"
"strings"
"testing"
)
func TestPreparedRunJSONDoesNotIncludeSecretValues(t *testing.T) {
const envName = "PROMPTKIT_TEST_API_KEY"
const secret = "super-secret-value"
t.Setenv(envName, secret)
prepared := PreparedRun{
PromptID: "prompt.id",
PromptVersion: "v1",
PromptHash: "prompt-hash",
SelectedProfileID: "local-fast",
EffectiveModelParams: ExecutionTarget{
Endpoint: "http://llm/v1",
Model: "gpt-test",
APIKeyEnv: envName,
APIKey: secret,
},
InputHashes: map[string]string{"transcript": "hash-1"},
RenderedPromptHash: "rendered-hash",
Messages: []RenderedMessage{
{Role: "system", Content: "You are helpful."},
{Role: "user", Content: "Summarize this."},
},
}
b, err := json.Marshal(prepared)
if err != nil {
t.Fatalf("marshal failed: %v", err)
}
out := string(b)
if strings.Contains(out, secret) {
t.Fatalf("prepared run JSON unexpectedly contains secret value: %s", out)
}
if !strings.Contains(out, `"api_key_env":"`+envName+`"`) {
t.Fatalf("prepared run JSON should include api_key_env name: %s", out)
}
var top map[string]any
if err := json.Unmarshal(b, &top); err != nil {
t.Fatalf("unmarshal failed: %v", err)
}
for _, forbidden := range []string{"raw_output", "validation", "artifact"} {
if _, ok := top[forbidden]; ok {
t.Fatalf("prepared run JSON should not include %q", forbidden)
}
}
}
func TestPreparedRunJSONIncludesMessageCacheControlOnlyWhenPresent(t *testing.T) {
prepared := PreparedRun{
PromptID: "prompt.id",
SelectedProfileID: "local-fast",
EffectiveModelParams: ExecutionTarget{
Endpoint: "http://llm/v1",
Model: "gpt-test",
},
RenderedPromptHash: "rendered-hash",
Messages: []RenderedMessage{
{
Role: "system",
Content: "You are helpful.",
CacheControl: &CacheControl{
Type: CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "Summarize this."},
},
}
b, err := json.Marshal(prepared)
if err != nil {
t.Fatalf("marshal failed: %v", err)
}
var decoded struct {
Messages []map[string]any `json:"messages"`
}
if err := json.Unmarshal(b, &decoded); err != nil {
t.Fatalf("unmarshal failed: %v", err)
}
if len(decoded.Messages) != 2 {
t.Fatalf("expected 2 messages, got %d", len(decoded.Messages))
}
cacheControl, ok := decoded.Messages[0]["cache_control"].(map[string]any)
if !ok {
t.Fatalf("expected cache_control on first message, got %#v", decoded.Messages[0])
}
if cacheControl["type"] != string(CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
}
if _, ok := decoded.Messages[1]["cache_control"]; ok {
t.Fatalf("expected second message to omit cache_control, got %#v", decoded.Messages[1])
}
}
func TestPreparedRunJSONIncludesSessionIDOnlyWhenPresent(t *testing.T) {
prepared := PreparedRun{
PromptID: "prompt.id",
SelectedProfileID: "local-fast",
EffectiveModelParams: ExecutionTarget{
Endpoint: "http://llm/v1",
Model: "gpt-test",
},
SessionID: "session-123",
RenderedPromptHash: "rendered-hash",
Messages: []RenderedMessage{{Role: "user", Content: "Summarize this."}},
}
b, err := json.Marshal(prepared)
if err != nil {
t.Fatalf("marshal failed: %v", err)
}
var decoded map[string]any
if err := json.Unmarshal(b, &decoded); err != nil {
t.Fatalf("unmarshal failed: %v", err)
}
if decoded["session_id"] != "session-123" {
t.Fatalf("expected session_id in prepared run JSON, got %#v", decoded["session_id"])
}
prepared.SessionID = ""
b, err = json.Marshal(prepared)
if err != nil {
t.Fatalf("marshal failed: %v", err)
}
if strings.Contains(string(b), "session_id") {
t.Fatalf("expected empty session_id to be omitted, got %s", b)
}
}

View File

@@ -8,6 +8,9 @@ import (
// NormalizeSessionID applies the shared session identifier rule.
func NormalizeSessionID(raw string) (string, error) {
if !utf8.ValidString(raw) {
return "", fmt.Errorf("session_id must contain valid UTF-8")
}
normalized := strings.TrimSpace(raw)
if normalized == "" {
return "", nil

View File

@@ -7,10 +7,10 @@ import (
func TestNormalizeSessionID(t *testing.T) {
tests := []struct {
name string
raw string
want string
wantErr bool
name string
raw string
want string
wantErrContains string
}{
{
name: "trims surrounding Unicode whitespace",
@@ -28,21 +28,24 @@ func TestNormalizeSessionID(t *testing.T) {
want: strings.Repeat("界", SessionIDMaxLength),
},
{
name: "one Unicode code point over maximum is rejected",
raw: strings.Repeat("界", SessionIDMaxLength+1),
wantErr: true,
name: "one Unicode code point over maximum is rejected",
raw: strings.Repeat("界", SessionIDMaxLength+1),
wantErrContains: "exceeds maximum",
},
{name: "invalid UTF-8 before valid content", raw: string([]byte{0xff}) + "session", wantErrContains: "valid UTF-8"},
{name: "invalid UTF-8 within valid content", raw: "ses" + string([]byte{0xff}) + "sion", wantErrContains: "valid UTF-8"},
{name: "invalid UTF-8 after valid content", raw: "session" + string([]byte{0xff}), wantErrContains: "valid UTF-8"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := NormalizeSessionID(tt.raw)
if tt.wantErr {
if tt.wantErrContains != "" {
if err == nil {
t.Fatal("expected normalization error")
}
if !strings.Contains(err.Error(), "exceeds maximum") {
t.Fatalf("expected useful length diagnostic, got %v", err)
if !strings.Contains(err.Error(), tt.wantErrContains) {
t.Fatalf("expected diagnostic containing %q, got %v", tt.wantErrContains, err)
}
return
}

View File

@@ -36,7 +36,8 @@ func FindYAMLFiles(ctx context.Context, root string) ([]string, error) {
return files, err
}
// FindFSYAMLFiles returns sorted paths for .yaml and .yml files under root in fsys.
// FindFSYAMLFiles returns root itself when it names a file. For a directory
// root, it returns sorted paths for .yaml and .yml files beneath that root.
func FindFSYAMLFiles(ctx context.Context, fsys fs.FS, root string) ([]string, error) {
cleanRoot := CleanFSRoot(root)
var files []string
@@ -52,6 +53,10 @@ func FindFSYAMLFiles(ctx context.Context, fsys fs.FS, root string) ([]string, er
if d.IsDir() {
return nil
}
if name == cleanRoot {
files = append(files, name)
return nil
}
if !IsYAMLFile(d.Name()) {
return nil
}
@@ -71,10 +76,10 @@ func RelativePath(root string, filePath string) string {
return filepath.Clean(rel)
}
// CleanFSRoot normalizes a root path for use with fs.FS.
// CleanFSRoot normalizes a root path for use with fs.FS while preserving
// nonblank leading and trailing whitespace.
func CleanFSRoot(root string) string {
root = strings.TrimSpace(root)
if root == "" || root == "." {
if strings.TrimSpace(root) == "" || root == "." {
return "."
}
return path.Clean(root)
@@ -97,22 +102,21 @@ func DisplayPath(root string, name string) string {
// ResolveFSPath resolves userPath from baseDir and keeps it inside root.
func ResolveFSPath(root string, baseDir string, userPath string) (string, string, error) {
cleanRoot := CleanFSRoot(root)
cleanBase := path.Clean(strings.TrimSpace(baseDir))
if cleanBase == "" {
cleanBase := path.Clean(baseDir)
if strings.TrimSpace(baseDir) == "" {
cleanBase = cleanRoot
}
if !containsFSPath(cleanRoot, cleanBase) {
return "", "", fmt.Errorf("base path %q is outside source root %q", cleanBase, cleanRoot)
}
cleanUserPath := strings.TrimSpace(userPath)
if cleanUserPath == "" {
if strings.TrimSpace(userPath) == "" {
return "", "", fmt.Errorf("path is required")
}
cleanUserPath = path.Clean(cleanUserPath)
if path.IsAbs(cleanUserPath) {
if path.IsAbs(userPath) {
return "", "", fmt.Errorf("path %q must be relative", userPath)
}
cleanUserPath := path.Clean(userPath)
resolved := path.Clean(path.Join(cleanBase, cleanUserPath))
if !containsFSPath(cleanRoot, resolved) {
@@ -130,13 +134,6 @@ func containsFSPath(root string, name string) bool {
return name == root || strings.HasPrefix(name, strings.TrimSuffix(root, "/")+"/")
}
// Stem strips .yaml or .yml from a file name.
func Stem(name string) string {
name = strings.TrimSuffix(name, ".yaml")
name = strings.TrimSuffix(name, ".yml")
return name
}
func IsYAMLFile(name string) bool {
return strings.HasSuffix(name, ".yaml") || strings.HasSuffix(name, ".yml")
}

View File

@@ -54,7 +54,7 @@ func TestFindFSYAMLFilesNestedSortedAndFiltered(t *testing.T) {
"other/ignored.yaml": &fstest.MapFile{Data: []byte("id: ignored")},
}
got, err := FindFSYAMLFiles(context.Background(), fsys, " prompts ")
got, err := FindFSYAMLFiles(context.Background(), fsys, "prompts")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
@@ -98,8 +98,11 @@ func TestCleanFSRoot(t *testing.T) {
want string
}{
{name: "empty", root: "", want: "."},
{name: "whitespace only", root: " \t ", want: "."},
{name: "dot", root: ".", want: "."},
{name: "trimmed", root: " prompts/../profiles ", want: "profiles"},
{name: "cleaned", root: "prompts/../profiles", want: "profiles"},
{name: "leading whitespace preserved", root: " profiles", want: " profiles"},
{name: "trailing whitespace preserved", root: "profiles ", want: "profiles "},
}
for _, tc := range tests {
@@ -158,6 +161,22 @@ func TestResolveFSPath(t *testing.T) {
wantPath: "prompts/shared/user.tmpl",
wantDisplay: "shared/user.tmpl",
},
{
name: "leading whitespace preserved",
root: "prompts",
baseDir: "prompts/nested",
userPath: " user.tmpl",
wantPath: "prompts/nested/ user.tmpl",
wantDisplay: "nested/ user.tmpl",
},
{
name: "trailing whitespace preserved",
root: "prompts",
baseDir: "prompts/nested",
userPath: "user.tmpl ",
wantPath: "prompts/nested/user.tmpl ",
wantDisplay: "nested/user.tmpl ",
},
{
name: "escape rejected",
root: "prompts",
@@ -218,26 +237,6 @@ func TestResolveFSPath(t *testing.T) {
}
}
func TestStemStripsYAMLExtensions(t *testing.T) {
tests := []struct {
name string
in string
want string
}{
{name: "yaml", in: "prompt.yaml", want: "prompt"},
{name: "yml", in: "profile.yml", want: "profile"},
{name: "other", in: "file.txt", want: "file.txt"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if got := Stem(tc.in); got != tc.want {
t.Fatalf("expected %q, got %q", tc.want, got)
}
})
}
}
func TestIsYAMLFile(t *testing.T) {
tests := []struct {
name string

View File

@@ -1,5 +1,5 @@
// Package jsonvalue validates and defensively copies JSON-compatible value
// trees used by configuration, request, and prepared-state boundaries.
// Package jsonvalue validates and defensively copies bounded JSON-compatible
// value trees used by configuration, request, and prepared-state boundaries.
package jsonvalue
import (
@@ -8,29 +8,39 @@ import (
"math"
"reflect"
"sort"
"strconv"
)
const maxSafeJSONInteger = 1<<53 - 1
const (
maxContainerDepth = 100
maxProducedNodes = 100_000
)
type visit struct {
typ reflect.Type
ptr uintptr
}
type traversalState struct {
active map[visit]struct{}
producedNodes int
}
// Copy validates and deeply copies a JSON-compatible value while preserving
// compatible concrete map, slice, array, scalar, and number types.
// compatible concrete map, slice, array, scalar, and number types. It rejects
// cycles and values that exceed the package's traversal limits.
func Copy(src any) (any, error) {
return copyValue(reflect.ValueOf(src), "value", make(map[visit]struct{}), true)
return copyValue(reflect.ValueOf(src), "value", newTraversalState(), true, 0)
}
// CopyMap validates and deeply copies an extra-parameter map while preserving
// compatible concrete map, slice, array, scalar, and number types.
// compatible concrete map, slice, array, scalar, and number types. It rejects
// empty object keys, cycles, and values that exceed the package's traversal
// limits.
func CopyMap(src map[string]any) (map[string]any, error) {
if src == nil {
return nil, nil
}
copied, err := copyValue(reflect.ValueOf(src), "extra_params", make(map[visit]struct{}), false)
copied, err := copyValue(reflect.ValueOf(src), "extra_params", newTraversalState(), false, 0)
if err != nil {
return nil, err
}
@@ -44,71 +54,90 @@ func CopyMap(src map[string]any) (map[string]any, error) {
func copyValue(
value reflect.Value,
path string,
seen map[visit]struct{},
state *traversalState,
allowEmptyMapKeys bool,
containerDepth int,
) (any, error) {
if !value.IsValid() {
resolved, cleanup, isNull, err := state.resolveIndirection(value, path)
if err != nil {
return nil, err
}
defer cleanup()
if isNull {
if err := state.produceNode(path); err != nil {
return nil, err
}
return nil, nil
}
if value.Kind() == reflect.Interface {
if value.IsNil() {
return nil, nil
}
return copyValue(value.Elem(), path, seen, allowEmptyMapKeys)
}
value = resolved
if !value.CanInterface() {
return nil, fmt.Errorf("%s: value cannot be copied", path)
}
if number, ok := value.Interface().(json.Number); ok {
if _, err := json.Marshal(number); err != nil {
if !validJSONNumber(number) {
return nil, fmt.Errorf("%s: invalid JSON number", path)
}
f, err := strconv.ParseFloat(number.String(), 64)
if err != nil || math.IsNaN(f) || math.IsInf(f, 0) {
return nil, fmt.Errorf("%s: invalid JSON number", path)
if err := state.produceNode(path); err != nil {
return nil, err
}
return number, nil
}
switch value.Kind() {
case reflect.Bool, reflect.String:
if err := state.produceNode(path); err != nil {
return nil, err
}
return value.Interface(), nil
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
if value.Int() < -maxSafeJSONInteger || value.Int() > maxSafeJSONInteger {
return nil, fmt.Errorf("%s: integer is outside the JSON-safe range", path)
if err := state.produceNode(path); err != nil {
return nil, err
}
return value.Interface(), nil
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
if value.Uint() > maxSafeJSONInteger {
return nil, fmt.Errorf("%s: integer is outside the JSON-safe range", path)
if err := state.produceNode(path); err != nil {
return nil, err
}
return value.Interface(), nil
case reflect.Float32, reflect.Float64:
number := value.Convert(reflect.TypeOf(float64(0))).Float()
number := value.Float()
if math.IsNaN(number) || math.IsInf(number, 0) {
return nil, fmt.Errorf("%s: floating-point value must be finite", path)
}
if err := state.produceNode(path); err != nil {
return nil, err
}
return value.Interface(), nil
case reflect.Pointer:
case reflect.Map:
if value.IsNil() {
if err := state.produceNode(path); err != nil {
return nil, err
}
return nil, nil
}
current := visit{typ: value.Type(), ptr: value.Pointer()}
if _, ok := seen[current]; ok {
return nil, fmt.Errorf("%s: cyclic value is not supported", path)
nextDepth, err := state.enterContainer(path, containerDepth)
if err != nil {
return nil, err
}
seen[current] = struct{}{}
defer delete(seen, current)
return copyValue(value.Elem(), path, seen, allowEmptyMapKeys)
case reflect.Map:
return copyMapValue(value, path, seen, allowEmptyMapKeys)
return copyMapValue(value, path, state, allowEmptyMapKeys, nextDepth)
case reflect.Slice:
if value.IsNil() {
if err := state.produceNode(path); err != nil {
return nil, err
}
return nil, nil
}
return copySequenceValue(value, path, seen, allowEmptyMapKeys)
nextDepth, err := state.enterContainer(path, containerDepth)
if err != nil {
return nil, err
}
return copySequenceValue(value, path, state, allowEmptyMapKeys, nextDepth)
case reflect.Array:
return copySequenceValue(value, path, seen, allowEmptyMapKeys)
nextDepth, err := state.enterContainer(path, containerDepth)
if err != nil {
return nil, err
}
return copySequenceValue(value, path, state, allowEmptyMapKeys, nextDepth)
default:
return nil, fmt.Errorf("%s: unsupported JSON value type %s", path, value.Type())
}
@@ -117,22 +146,26 @@ func copyValue(
func copyMapValue(
value reflect.Value,
path string,
seen map[visit]struct{},
state *traversalState,
allowEmptyMapKeys bool,
containerDepth int,
) (any, error) {
if value.IsNil() {
return nil, nil
}
if value.Type().Key().Kind() != reflect.String {
return nil, fmt.Errorf("%s: map key type %s is not supported", path, value.Type().Key())
}
if err := state.produceNode(path); err != nil {
return nil, err
}
if err := state.ensureChildCapacity(path, value.Len()); err != nil {
return nil, err
}
current := visit{typ: value.Type(), ptr: value.Pointer()}
if _, ok := seen[current]; ok {
if _, ok := state.active[current]; ok {
return nil, fmt.Errorf("%s: cyclic value is not supported", path)
}
seen[current] = struct{}{}
defer delete(seen, current)
state.active[current] = struct{}{}
defer delete(state.active, current)
keys := value.MapKeys()
sort.Slice(keys, func(i, j int) bool {
@@ -152,7 +185,13 @@ func copyMapValue(
if name == "" && !allowEmptyMapKeys {
return nil, fmt.Errorf("%s: map key must not be empty", path)
}
copied, err := copyValue(value.MapIndex(key), path+"."+name, seen, allowEmptyMapKeys)
copied, err := copyValue(
value.MapIndex(key),
path+"."+name,
state,
allowEmptyMapKeys,
containerDepth,
)
if err != nil {
return nil, err
}
@@ -190,17 +229,25 @@ func copyMapValue(
func copySequenceValue(
value reflect.Value,
path string,
seen map[visit]struct{},
state *traversalState,
allowEmptyMapKeys bool,
containerDepth int,
) (any, error) {
if err := state.produceNode(path); err != nil {
return nil, err
}
if err := state.ensureChildCapacity(path, value.Len()); err != nil {
return nil, err
}
var current visit
if value.Kind() == reflect.Slice {
current = visit{typ: value.Type(), ptr: value.Pointer()}
if _, ok := seen[current]; ok {
if _, ok := state.active[current]; ok {
return nil, fmt.Errorf("%s: cyclic value is not supported", path)
}
seen[current] = struct{}{}
defer delete(seen, current)
state.active[current] = struct{}{}
defer delete(state.active, current)
}
values := make([]any, value.Len())
@@ -210,8 +257,9 @@ func copySequenceValue(
copied, err := copyValue(
value.Index(i),
fmt.Sprintf("%s[%d]", path, i),
seen,
state,
allowEmptyMapKeys,
containerDepth,
)
if err != nil {
return nil, err
@@ -248,6 +296,73 @@ func copySequenceValue(
return out, nil
}
func validJSONNumber(number json.Number) bool {
var parsed json.Number
if err := json.Unmarshal([]byte(number.String()), &parsed); err != nil {
return false
}
return parsed.String() == number.String()
}
func newTraversalState() *traversalState {
return &traversalState{active: make(map[visit]struct{})}
}
func (state *traversalState) produceNode(path string) error {
if state.producedNodes >= maxProducedNodes {
return fmt.Errorf("%s: JSON value work limit exceeded", path)
}
state.producedNodes++
return nil
}
func (state *traversalState) enterContainer(path string, depth int) (int, error) {
depth++
if depth > maxContainerDepth {
return 0, fmt.Errorf("%s: JSON container depth limit exceeded", path)
}
return depth, nil
}
func (state *traversalState) ensureChildCapacity(path string, count int) error {
if count > maxProducedNodes-state.producedNodes {
return fmt.Errorf("%s: JSON value work limit exceeded", path)
}
return nil
}
func (state *traversalState) resolveIndirection(
value reflect.Value,
path string,
) (reflect.Value, func(), bool, error) {
var visits []visit
cleanup := func() {
for _, current := range visits {
delete(state.active, current)
}
}
for value.IsValid() && (value.Kind() == reflect.Interface || value.Kind() == reflect.Pointer) {
if value.IsNil() {
return reflect.Value{}, cleanup, true, nil
}
if value.Kind() == reflect.Pointer {
current := visit{typ: value.Type(), ptr: value.Pointer()}
if _, ok := state.active[current]; ok {
cleanup()
return reflect.Value{}, nil, false, fmt.Errorf("%s: cyclic value is not supported", path)
}
state.active[current] = struct{}{}
visits = append(visits, current)
}
value = value.Elem()
}
if !value.IsValid() {
return reflect.Value{}, cleanup, true, nil
}
return value, cleanup, false, nil
}
func canAssignNil(typ reflect.Type) bool {
switch typ.Kind() {
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:

View File

@@ -1,112 +1,332 @@
package jsonvalue_test
package jsonvalue
import (
"encoding/json"
"math"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/promptkit/internal/jsonvalue"
)
func TestCopyMapPreservesTypesAndIsolatesMutations(t *testing.T) {
nested := map[string]int{"limit": 2}
sequence := []string{"one", "two"}
input := map[string]any{
"count": int64(7),
"number": json.Number("-1.25e+2"),
"nested": nested,
"sequence": sequence,
}
copied, err := jsonvalue.CopyMap(input)
if err != nil {
t.Fatalf("copy map: %v", err)
}
nested["limit"] = 99
sequence[0] = "changed"
input["added"] = true
if got, ok := copied["count"].(int64); !ok || got != 7 {
t.Fatalf("integer type or value changed: %#v", copied["count"])
}
if got, ok := copied["number"].(json.Number); !ok || got != "-1.25e+2" {
t.Fatalf("JSON number type or value changed: %#v", copied["number"])
}
if got := copied["nested"].(map[string]int)["limit"]; got != 2 {
t.Fatalf("nested map was not isolated: %d", got)
}
if got := copied["sequence"].([]string)[0]; got != "one" {
t.Fatalf("sequence was not isolated: %q", got)
}
if _, ok := copied["added"]; ok {
t.Fatalf("top-level map was not isolated: %#v", copied)
}
}
func TestCopyAllowsEmptyObjectKeysAndIsolatesMutations(t *testing.T) {
nested := map[string]any{"": []any{"original"}}
copiedValue, err := jsonvalue.Copy(nested)
if err != nil {
t.Fatalf("copy value: %v", err)
}
nested[""].([]any)[0] = "changed"
copied := copiedValue.(map[string]any)
if got := copied[""].([]any)[0]; got != "original" {
t.Fatalf("copied value was not isolated: %v", got)
}
}
func TestCopyMapRejectsInvalidValues(t *testing.T) {
cyclicMap := map[string]any{}
cyclicMap["self"] = cyclicMap
cyclicSlice := []any{nil}
cyclicSlice[0] = cyclicSlice
type (
namedBool bool
namedString string
namedInt64 int64
namedUint64 uint64
namedFloat32 float32
namedFloat64 float64
namedKey string
namedMap map[namedKey]namedInt64
namedSlice []namedString
namedArray [1]map[string]int
)
func TestCopyPreservesSupportedScalarAndNumberTypes(t *testing.T) {
maxInt := int(^uint(0) >> 1)
minInt := -maxInt - 1
tests := []struct {
name string
value any
}{
{name: "empty nested key", value: map[string]int{"": 1}},
{name: "non-string map key", value: map[int]string{1: "one"}},
{name: "unsupported value", value: make(chan int)},
{name: "cyclic map", value: cyclicMap},
{name: "cyclic slice", value: cyclicSlice},
{name: "NaN", value: math.NaN()},
{name: "positive infinity", value: math.Inf(1)},
{name: "unsafe signed integer", value: int64(1 << 53)},
{name: "unsafe unsigned integer", value: uint64(1 << 53)},
{name: "bool", value: true},
{name: "named bool", value: namedBool(true)},
{name: "string", value: "value"},
{name: "named string", value: namedString("value")},
{name: "int", value: minInt},
{name: "int8", value: int8(-1 << 7)},
{name: "int16", value: int16(-1 << 15)},
{name: "int32", value: int32(-1 << 31)},
{name: "int64", value: int64(-1 << 63)},
{name: "named int64", value: namedInt64(1<<63 - 1)},
{name: "uint", value: ^uint(0)},
{name: "uint8", value: ^uint8(0)},
{name: "uint16", value: ^uint16(0)},
{name: "uint32", value: ^uint32(0)},
{name: "uint64", value: ^uint64(0)},
{name: "uintptr", value: ^uintptr(0)},
{name: "named uint64", value: namedUint64(^uint64(0))},
{name: "float32", value: float32(1.25)},
{name: "float64", value: float64(-2.5e100)},
{name: "named float32", value: namedFloat32(3.5)},
{name: "named float64", value: namedFloat64(-4.5e200)},
{name: "JSON number integer", value: json.Number("18446744073709551615")},
{name: "JSON number fraction", value: json.Number("-1.25e+2")},
{name: "JSON number beyond float64", value: json.Number("1e9999")},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if _, err := jsonvalue.CopyMap(map[string]any{"value": tc.value}); err == nil {
got, err := Copy(tc.value)
if err != nil {
t.Fatalf("copy value: %v", err)
}
if !reflect.DeepEqual(got, tc.value) {
t.Fatalf("value or concrete type changed: got %#v (%T), want %#v (%T)", got, got, tc.value, tc.value)
}
})
}
}
func TestCopyRejectsInvalidNumbers(t *testing.T) {
tests := []struct {
name string
value any
}{
{name: "float32 NaN", value: float32(math.NaN())},
{name: "float64 NaN", value: math.NaN()},
{name: "named float NaN", value: namedFloat64(math.NaN())},
{name: "positive infinity", value: math.Inf(1)},
{name: "negative infinity", value: math.Inf(-1)},
{name: "empty JSON number", value: json.Number("")},
{name: "leading zero JSON number", value: json.Number("01")},
{name: "leading plus JSON number", value: json.Number("+1")},
{name: "trailing decimal JSON number", value: json.Number("1.")},
{name: "leading decimal JSON number", value: json.Number(".1")},
{name: "non-number JSON number", value: json.Number("NaN")},
{name: "spaced JSON number", value: json.Number(" 1")},
{name: "quoted JSON number", value: json.Number(`"1"`)},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if _, err := Copy(tc.value); err == nil {
t.Fatal("expected validation error")
}
})
}
}
func TestCopyMapValidatesJSONNumberSyntaxAndRange(t *testing.T) {
for _, number := range []json.Number{"0", "-1", "1.25", "-1.25e+2"} {
t.Run("valid "+number.String(), func(t *testing.T) {
got, err := jsonvalue.CopyMap(map[string]any{"value": number})
func TestCopyPreservesCompatibleCollectionsAndNilEmptyDistinctions(t *testing.T) {
collections := []struct {
name string
value any
}{
{name: "unnamed map", value: map[string]int{"limit": 2}},
{name: "named map", value: namedMap{"limit": 2}},
{name: "unnamed slice", value: []string{"one", "two"}},
{name: "named slice", value: namedSlice{"one", "two"}},
{name: "unnamed array", value: [2]int{1, 2}},
{name: "named array", value: namedArray{{"limit": 2}}},
{name: "empty map", value: map[string]int{}},
{name: "empty named map", value: namedMap{}},
{name: "empty slice", value: []string{}},
{name: "empty named slice", value: namedSlice{}},
{name: "empty array", value: [0]string{}},
}
for _, tc := range collections {
t.Run(tc.name, func(t *testing.T) {
got, err := Copy(tc.value)
if err != nil {
t.Fatalf("copy valid JSON number: %v", err)
t.Fatalf("copy collection: %v", err)
}
if !reflect.DeepEqual(got["value"], number) {
t.Fatalf("JSON number changed: got %#v want %#v", got["value"], number)
if !reflect.DeepEqual(got, tc.value) || reflect.TypeOf(got) != reflect.TypeOf(tc.value) {
t.Fatalf("collection changed: got %#v (%T), want %#v (%T)", got, got, tc.value, tc.value)
}
kind := reflect.ValueOf(got).Kind()
if (kind == reflect.Map || kind == reflect.Slice) && reflect.ValueOf(got).IsNil() {
t.Fatal("non-nil collection became nil")
}
})
}
for _, number := range []json.Number{"", "01", "+1", "1.", ".1", "1e9999", "not-a-number"} {
t.Run("invalid "+number.String(), func(t *testing.T) {
if _, err := jsonvalue.CopyMap(map[string]any{"value": number}); err == nil {
t.Fatal("expected invalid JSON number error")
var nilMap map[string]int
var nilSlice []string
var nilPointer *namedInt64
for _, value := range []any{nil, nilMap, nilSlice, nilPointer} {
got, err := Copy(value)
if err != nil {
t.Fatalf("copy null value: %v", err)
}
if got != nil {
t.Fatalf("null value became %#v (%T)", got, got)
}
}
gotNil, err := CopyMap(nil)
if err != nil || gotNil != nil {
t.Fatalf("nil CopyMap result = %#v, %v", gotNil, err)
}
gotEmpty, err := CopyMap(map[string]any{})
if err != nil || gotEmpty == nil || len(gotEmpty) != 0 {
t.Fatalf("empty CopyMap result = %#v, %v", gotEmpty, err)
}
}
func TestCopyHandlesIndirectionAndIsolatesNestedMutations(t *testing.T) {
integer := namedInt64(7)
nestedMap := namedMap{"limit": 2}
nestedSlice := namedSlice{"original"}
nestedArray := namedArray{{"limit": 3}}
shared := []any{map[string]int{"value": 4}}
input := map[string]any{
"integer": &integer,
"map": nestedMap,
"slice": nestedSlice,
"array": nestedArray,
"first": shared,
"second": shared,
}
copiedValue, err := Copy(input)
if err != nil {
t.Fatalf("copy mixed tree: %v", err)
}
copied := copiedValue.(map[string]any)
nestedMap["limit"] = 20
nestedSlice[0] = "changed"
nestedArray[0]["limit"] = 30
shared[0].(map[string]int)["value"] = 40
if got, ok := copied["integer"].(namedInt64); !ok || got != 7 {
t.Fatalf("pointer target changed: %#v", copied["integer"])
}
if got := copied["map"].(namedMap)["limit"]; got != 2 {
t.Fatalf("nested map aliased input: %d", got)
}
if got := copied["slice"].(namedSlice)[0]; got != "original" {
t.Fatalf("nested slice aliased input: %q", got)
}
if got := copied["array"].(namedArray)[0]["limit"]; got != 3 {
t.Fatalf("nested array aliased input: %d", got)
}
first := copied["first"].([]any)
second := copied["second"].([]any)
if got := first[0].(map[string]int)["value"]; got != 4 {
t.Fatalf("shared child aliased input: %d", got)
}
first[0].(map[string]int)["value"] = 99
if got := second[0].(map[string]int)["value"]; got != 4 {
t.Fatalf("repeated acyclic value shared copied output: %d", got)
}
}
func TestCopyAndCopyMapApplyDistinctEmptyKeyRules(t *testing.T) {
nested := map[string]any{"": []any{"original"}}
copiedValue, err := Copy(nested)
if err != nil {
t.Fatalf("Copy rejected empty schema key: %v", err)
}
nested[""].([]any)[0] = "changed"
if got := copiedValue.(map[string]any)[""].([]any)[0]; got != "original" {
t.Fatalf("copied schema value was not isolated: %v", got)
}
_, err = CopyMap(map[string]any{"nested": map[string]any{"": true}})
if err == nil || !strings.Contains(err.Error(), "extra_params.nested") {
t.Fatalf("CopyMap empty-key error = %v", err)
}
}
func TestCopyRejectsUnsupportedValuesAndActiveCycles(t *testing.T) {
cyclicMap := map[string]any{}
cyclicMap["self"] = cyclicMap
cyclicSlice := []any{nil}
cyclicSlice[0] = cyclicSlice
var cyclicPointer any
cyclicPointer = &cyclicPointer
tests := []struct {
name string
value any
wantPath string
}{
{name: "non-string map key", value: map[int]string{1: "one"}, wantPath: "value"},
{name: "unsupported channel", value: make(chan int), wantPath: "value"},
{name: "deterministic map path", value: map[string]any{"z": make(chan int), "a": make(chan int)}, wantPath: "value.a"},
{name: "cyclic map", value: cyclicMap, wantPath: "value.self"},
{name: "cyclic slice", value: cyclicSlice, wantPath: "value[0]"},
{name: "cyclic pointer", value: cyclicPointer, wantPath: "value"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
_, err := Copy(tc.value)
if err == nil || !strings.Contains(err.Error(), tc.wantPath) {
t.Fatalf("error = %v, want structural path %q", err, tc.wantPath)
}
})
}
}
func TestCopyEnforcesContainerDepth(t *testing.T) {
tests := []struct {
name string
depth int
wantErr bool
}{
{name: "just below", depth: maxContainerDepth - 1},
{name: "at limit", depth: maxContainerDepth},
{name: "over limit", depth: maxContainerDepth + 1, wantErr: true},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
_, err := Copy(alternatingContainers(tc.depth))
if tc.wantErr {
if err == nil || !strings.HasPrefix(err.Error(), "value") || !strings.Contains(err.Error(), "container depth limit") {
t.Fatalf("depth error = %v", err)
}
return
}
if err != nil {
t.Fatalf("copy depth %d: %v", tc.depth, err)
}
})
}
}
func TestCopyEnforcesProducedNodeBudgetForRepeatedAcyclicValues(t *testing.T) {
shared := []any{true}
sharedOccurrences := (maxProducedNodes - 2) / 2
justBelow := repeatedValues(shared, sharedOccurrences, 0)
atLimit := repeatedValues(shared, sharedOccurrences, 1)
overLimit := repeatedValues(shared, sharedOccurrences, 2)
for name, value := range map[string]any{
"just below": justBelow,
"at limit": atLimit,
} {
t.Run(name, func(t *testing.T) {
if _, err := Copy(value); err != nil {
t.Fatalf("copy value within work budget: %v", err)
}
})
}
_, err := Copy(overLimit)
if err == nil || !strings.HasPrefix(err.Error(), "value[") || !strings.Contains(err.Error(), "value work limit") {
t.Fatalf("work-budget error = %v", err)
}
_, err = Copy(make([]any, maxProducedNodes))
if err == nil || !strings.HasPrefix(err.Error(), "value:") || !strings.Contains(err.Error(), "value work limit") {
t.Fatalf("flat work-budget error = %v", err)
}
}
func alternatingContainers(depth int) any {
var value any = true
for level := 0; level < depth; level++ {
switch level % 3 {
case 0:
value = map[string]any{"child": value}
case 1:
value = []any{value}
default:
value = [1]any{value}
}
}
return value
}
func repeatedValues(shared []any, occurrences, leadingScalars int) []any {
values := make([]any, 0, leadingScalars+occurrences)
for i := 0; i < leadingScalars; i++ {
values = append(values, false)
}
for i := 0; i < occurrences; i++ {
values = append(values, shared)
}
return values
}

View File

@@ -25,6 +25,20 @@ var (
ErrMalformedResponse = errors.New("malformed llm response")
)
const maxOpenAIChatResponseBytes int64 = 16 << 20
type requestFailedError struct {
cause error
}
func (e *requestFailedError) Error() string {
return ErrRequestFailed.Error()
}
func (e *requestFailedError) Unwrap() []error {
return []error{ErrRequestFailed, e.cause}
}
type OpenAICompatibleConfig struct {
BaseURL string
Model string
@@ -39,9 +53,11 @@ type OpenAICompatibleClient struct {
}
func NewOpenAICompatibleClient(cfg OpenAICompatibleConfig) (*OpenAICompatibleClient, error) {
baseURL := strings.TrimSpace(cfg.BaseURL)
if baseURL != "" {
if _, err := url.ParseRequestURI(baseURL); err != nil {
baseURL := ""
if strings.TrimSpace(cfg.BaseURL) != "" {
var err error
baseURL, err = domain.NormalizeOpenAICompatibleBaseEndpoint(cfg.BaseURL)
if err != nil {
return nil, fmt.Errorf("%w: invalid base URL: %v", ErrInvalidConfig, err)
}
}
@@ -63,25 +79,29 @@ func NewOpenAICompatibleClient(cfg OpenAICompatibleConfig) (*OpenAICompatibleCli
}
return &OpenAICompatibleClient{
baseURL: strings.TrimRight(baseURL, "/"),
baseURL: baseURL,
defaultModel: cfg.Model,
httpClient: client,
}, nil
}
func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.GenerateRequest) (*domain.GenerateResponse, error) {
if req.Target.TimeoutSeconds < 0 {
return nil, fmt.Errorf("%w: timeout_seconds must be greater than or equal to 0", ErrInvalidRequest)
if err := domain.ValidateExecutionTargetSettings(req.Target); err != nil {
return nil, fmt.Errorf("%w: %v", ErrInvalidRequest, err)
}
endpoint := strings.TrimSpace(req.Target.Endpoint)
if endpoint == "" {
endpoint = c.baseURL
selectedEndpoint := req.Target.Endpoint
if strings.TrimSpace(selectedEndpoint) == "" {
selectedEndpoint = c.baseURL
}
if endpoint == "" {
return nil, fmt.Errorf("%w: endpoint is required", ErrInvalidRequest)
endpoint, err := domain.NormalizeOpenAICompatibleBaseEndpoint(selectedEndpoint)
if err != nil {
return nil, fmt.Errorf("%w: invalid endpoint: %v", ErrInvalidRequest, err)
}
endpoint, err = url.JoinPath(endpoint, defaults.OpenAIChatCompletionsPath)
if err != nil {
return nil, fmt.Errorf("%w: invalid endpoint path: %v", ErrInvalidRequest, err)
}
endpoint = strings.TrimRight(endpoint, "/") + defaults.OpenAIChatCompletionsPath
wireReq, err := openAIChatRequestFromGenerateRequest(req, c.defaultModel)
if err != nil {
@@ -130,7 +150,7 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
httpResp, err := httpClient.Do(httpReq)
if err != nil {
return nil, fmt.Errorf("%w: %v", ErrRequestFailed, err)
return nil, &requestFailedError{cause: err}
}
defer httpResp.Body.Close()
@@ -138,10 +158,13 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
_, _ = io.Copy(io.Discard, io.LimitReader(httpResp.Body, 4096))
return nil, fmt.Errorf("%w: status=%d", ErrUnexpectedStatus, httpResp.StatusCode)
}
if httpResp.ContentLength > maxOpenAIChatResponseBytes {
return nil, openAIChatResponseTooLargeError()
}
var wireResp openAIChatResponse
if err := json.NewDecoder(httpResp.Body).Decode(&wireResp); err != nil {
return nil, fmt.Errorf("%w: failed to decode response: %v", ErrMalformedResponse, err)
wireResp, err := decodeOpenAIChatResponse(httpResp.Body)
if err != nil {
return nil, err
}
if len(wireResp.Choices) == 0 {
@@ -164,6 +187,46 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
}, nil
}
func decodeOpenAIChatResponse(body io.Reader) (openAIChatResponse, error) {
limited := &io.LimitedReader{
R: body,
N: maxOpenAIChatResponseBytes + 1,
}
decoder := json.NewDecoder(limited)
var response openAIChatResponse
if err := decoder.Decode(&response); err != nil {
if limited.N == 0 {
return openAIChatResponse{}, openAIChatResponseTooLargeError()
}
return openAIChatResponse{}, fmt.Errorf("%w: failed to decode response", ErrMalformedResponse)
}
if limited.N == 0 {
return openAIChatResponse{}, openAIChatResponseTooLargeError()
}
var trailing any
if err := decoder.Decode(&trailing); !errors.Is(err, io.EOF) {
if limited.N == 0 {
return openAIChatResponse{}, openAIChatResponseTooLargeError()
}
return openAIChatResponse{}, fmt.Errorf("%w: response contains trailing data", ErrMalformedResponse)
}
if limited.N == 0 {
return openAIChatResponse{}, openAIChatResponseTooLargeError()
}
return response, nil
}
func openAIChatResponseTooLargeError() error {
return fmt.Errorf(
"%w: response exceeds %d-byte limit",
ErrMalformedResponse,
maxOpenAIChatResponseBytes,
)
}
func openAIChatRequestFromGenerateRequest(req domain.GenerateRequest, defaultModel string) (openAIChatRequest, error) {
model := strings.TrimSpace(req.Target.Model)
if model == "" {

File diff suppressed because it is too large Load Diff

View File

@@ -5,13 +5,14 @@ import (
"context"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path"
"strings"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
"gitea.maximumdirect.net/eric/promptkit/internal/filecatalog"
"gitea.maximumdirect.net/eric/promptkit/internal/jsonvalue"
"gopkg.in/yaml.v3"
)
@@ -73,7 +74,8 @@ func (r *overlayRepository) GetProfile(ctx context.Context, id string) (*domain.
}
func loadProfile(ctx context.Context, fsys fs.FS, root string, id string) (*domain.ExecutionProfile, error) {
if strings.TrimSpace(id) == "" {
id = strings.TrimSpace(id)
if id == "" {
return nil, fmt.Errorf("%w: profile id is required", ErrInvalidProfile)
}
if fsys == nil {
@@ -94,42 +96,50 @@ func loadProfile(ctx context.Context, fsys fs.FS, root string, id string) (*doma
}
relPath := filecatalog.DisplayPath(root, fullPath)
fileMatch := filecatalog.Stem(path.Base(fullPath)) == id
data, err := fs.ReadFile(fsys, fullPath)
if err != nil {
return nil, fmt.Errorf("failed to read profile file %s: %w", relPath, err)
}
metadata := readProfileFileMetadata(data)
idMatch := fileMatch || metadata.id == id
metadata, metadataErr := readProfileFileMetadata(data)
idMatch := metadata.matchesID(id)
if metadataErr != nil {
if idMatch {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, metadataErr)
}
continue
}
if metadata.hasRawAPIKey {
if idMatch {
return nil, fmt.Errorf("%w: %s", ErrRawAPIKeyNotAllowed, relPath)
}
continue
}
var prof domain.ExecutionProfile
decoder := yaml.NewDecoder(bytes.NewReader(data))
decoder.KnownFields(true)
if err := decoder.Decode(&prof); err != nil {
if idMatch {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
}
if !idMatch {
continue
}
prof, err := decodeProfile(data)
if err != nil {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
}
prof.ID = strings.TrimSpace(prof.ID)
if prof.ID != id {
continue
}
prof.BackendID = strings.TrimSpace(prof.BackendID)
if err := validateProfile(&prof); err != nil {
prof.ExtraParams, err = jsonvalue.CopyMap(prof.ExtraParams)
if err != nil {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, relPath, err)
}
if err := normalizeAndValidateProfile(prof); err != nil {
if errors.Is(err, ErrRawAPIKeyNotAllowed) {
return nil, fmt.Errorf("%w: %s", err, relPath)
}
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidProfile, relPath, err)
}
matches = append(matches, profileMatch{
profile: &prof,
profile: prof,
path: relPath,
})
}
@@ -155,15 +165,36 @@ type profileMatch struct {
}
type profileFileMetadata struct {
id string
ids []string
hasRawAPIKey bool
}
func readProfileFileMetadata(data []byte) profileFileMetadata {
func readProfileFileMetadata(data []byte) (profileFileMetadata, error) {
decoder := yaml.NewDecoder(bytes.NewReader(data))
var node yaml.Node
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&node); err != nil {
return profileFileMetadata{}
if err := decoder.Decode(&node); err != nil {
return profileFileMetadata{}, err
}
metadata := profileMetadataFromNode(&node)
documentCount := 1
for {
var trailing yaml.Node
err := decoder.Decode(&trailing)
if errors.Is(err, io.EOF) {
if documentCount == 1 {
return metadata, nil
}
return metadata, errors.New("profile file must contain exactly one YAML document")
}
if err != nil {
return metadata, err
}
documentCount++
metadata.merge(profileMetadataFromNode(&trailing))
}
}
func profileMetadataFromNode(node *yaml.Node) profileFileMetadata {
if node.Kind != yaml.DocumentNode || len(node.Content) == 0 {
return profileFileMetadata{}
}
@@ -178,7 +209,7 @@ func readProfileFileMetadata(data []byte) profileFileMetadata {
value := mapping.Content[i+1]
switch key.Value {
case "id":
metadata.id = strings.TrimSpace(value.Value)
metadata.ids = append(metadata.ids, strings.TrimSpace(value.Value))
case "api_key":
metadata.hasRawAPIKey = true
}
@@ -186,29 +217,68 @@ func readProfileFileMetadata(data []byte) profileFileMetadata {
return metadata
}
func validateProfile(p *domain.ExecutionProfile) error {
func (m profileFileMetadata) matchesID(id string) bool {
for _, candidate := range m.ids {
if candidate == id {
return true
}
}
return false
}
func (m *profileFileMetadata) merge(other profileFileMetadata) {
m.ids = append(m.ids, other.ids...)
m.hasRawAPIKey = m.hasRawAPIKey || other.hasRawAPIKey
}
func decodeProfile(data []byte) (*domain.ExecutionProfile, error) {
var prof domain.ExecutionProfile
decoder := yaml.NewDecoder(bytes.NewReader(data))
decoder.KnownFields(true)
if err := decoder.Decode(&prof); err != nil {
return nil, err
}
if err := requireYAMLStreamEnd(decoder); err != nil {
return nil, err
}
return &prof, nil
}
func requireYAMLStreamEnd(decoder *yaml.Decoder) error {
var trailing yaml.Node
err := decoder.Decode(&trailing)
if errors.Is(err, io.EOF) {
return nil
}
if err != nil {
return err
}
return errors.New("profile file must contain exactly one YAML document")
}
func normalizeAndValidateProfile(p *domain.ExecutionProfile) error {
if strings.TrimSpace(p.ID) == "" {
return errors.New("id is required")
}
if strings.TrimSpace(p.BackendID) == "" && strings.TrimSpace(p.Endpoint) == "" {
p.Endpoint = strings.TrimSpace(p.Endpoint)
if strings.TrimSpace(p.BackendID) == "" && p.Endpoint == "" {
return errors.New("backend or endpoint is required")
}
if p.Endpoint != "" {
endpoint, err := domain.NormalizeOpenAICompatibleBaseEndpoint(p.Endpoint)
if err != nil {
return err
}
p.Endpoint = endpoint
}
if strings.TrimSpace(p.Model) == "" {
return errors.New("model is required")
}
if p.Temperature < 0 || p.Temperature > 2 {
return errors.New("temperature must be between 0 and 2")
}
if p.MaxTokens < 0 {
return errors.New("max_tokens must be greater than or equal to 0")
}
if p.TopP < 0 || p.TopP > 1 {
return errors.New("top_p must be between 0 and 1")
}
if p.TimeoutSeconds < 0 {
return errors.New("timeout_seconds must be greater than or equal to 0")
}
return nil
return domain.ValidateExecutionTargetSettings(domain.ExecutionTarget{
Temperature: p.Temperature,
MaxTokens: p.MaxTokens,
TopP: p.TopP,
TimeoutSeconds: p.TimeoutSeconds,
})
}

View File

@@ -0,0 +1,58 @@
package profile
import (
"context"
"fmt"
"testing"
"testing/fstest"
)
func BenchmarkProfileRepositoryLookup(b *testing.B) {
for _, size := range []int{10, 1000} {
b.Run(fmt.Sprintf("catalog-%d", size), func(b *testing.B) {
files := fstest.MapFS{
"target.yaml": profileMapFile(`
id: target
endpoint: http://localhost:8000/v1
model: target-model
extra_params:
selected: true
`),
}
metadataNames := []string{"target.yaml"}
for i := 1; i < size; i++ {
name := fmt.Sprintf("profile-%04d.yaml", i)
files[name] = profileMapFile(fmt.Sprintf(`
id: profile-%04d
endpoint: http://localhost:8000/v1
model: unrelated-model
temperature: 0.5
max_tokens: 500
extra_params:
provider:
order:
- first
- second
`, i))
metadataNames = append(metadataNames, name)
}
fsys := &recordingProfileFS{FS: files}
repo := NewFSRepository(fsys, ".")
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if _, err := repo.GetProfile(context.Background(), "target"); err != nil {
b.Fatal(err)
}
}
b.StopTimer()
for _, name := range metadataNames {
if got := fsys.openCount(name); got != b.N {
b.Fatalf("metadata %q opens = %d, want %d", name, got, b.N)
}
}
})
}
}

View File

@@ -2,11 +2,13 @@ package profile
import (
"context"
"encoding/json"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"testing/fstest"
@@ -61,7 +63,7 @@ func TestFilesystemRepository_GetProfile(t *testing.T) {
wantErr bool
}{
{name: "backend only", connection: "backend: ' openrouter '", wantBackend: "openrouter"},
{name: "endpoint only", connection: "endpoint: http://localhost:8000/v1", wantEndpoint: "http://localhost:8000/v1"},
{name: "endpoint only", connection: "endpoint: ' https://localhost:8000/nested/v1 '", wantEndpoint: "https://localhost:8000/nested/v1"},
{name: "both", connection: "backend: openrouter\nendpoint: http://localhost:8000/v1", wantBackend: "openrouter", wantEndpoint: "http://localhost:8000/v1"},
{name: "neither", wantErr: true},
{name: "blank backend", connection: "backend: ' '", wantErr: true},
@@ -126,63 +128,6 @@ temperature: 0.1
}
})
t.Run("valid profile with JSON-compatible extra params", func(t *testing.T) {
writeProfileTestFile(t, filepath.Join(tmpDir, "json-extra-params.yaml"), `
id: json-extra-params
endpoint: http://localhost:8000/v1
model: nested-model
extra_params:
string_value: enabled
number_value: 42
boolean_value: true
object_value:
nested: value
count: 2
array_value:
- first
- 3
- false
`)
p, err := repo.GetProfile(ctx, "json-extra-params")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
var got map[string]any
encoded, err := json.Marshal(p.ExtraParams)
if err != nil {
t.Fatalf("expected extra_params to marshal as JSON, got %v", err)
}
if err := json.Unmarshal(encoded, &got); err != nil {
t.Fatalf("expected extra_params JSON to decode, got %v", err)
}
if got["string_value"] != "enabled" {
t.Fatalf("unexpected string extra param: %#v", got["string_value"])
}
if got["number_value"] != float64(42) {
t.Fatalf("unexpected number extra param: %#v", got["number_value"])
}
if got["boolean_value"] != true {
t.Fatalf("unexpected boolean extra param: %#v", got["boolean_value"])
}
objectValue, ok := got["object_value"].(map[string]any)
if !ok {
t.Fatalf("expected object extra param, got %#v", got["object_value"])
}
if objectValue["nested"] != "value" || objectValue["count"] != float64(2) {
t.Fatalf("unexpected object extra param: %#v", objectValue)
}
arrayValue, ok := got["array_value"].([]any)
if !ok {
t.Fatalf("expected array extra param, got %#v", got["array_value"])
}
if len(arrayValue) != 3 || arrayValue[0] != "first" || arrayValue[1] != float64(3) || arrayValue[2] != false {
t.Fatalf("unexpected array extra param: %#v", arrayValue)
}
})
t.Run("duplicate profile IDs fail as ambiguous", func(t *testing.T) {
writeProfileTestFile(t, filepath.Join(tmpDir, "duplicate-profile-a.yaml"), `
id: duplicate-profile
@@ -245,10 +190,10 @@ api_key: secret
}
})
t.Run("invalid yaml", func(t *testing.T) {
t.Run("unidentifiable invalid yaml is unrelated", func(t *testing.T) {
_, err := repo.GetProfile(ctx, "invalid_yaml")
if !errors.Is(err, ErrInvalidYAML) {
t.Fatalf("expected ErrInvalidYAML, got %v", err)
if !errors.Is(err, ErrProfileNotFound) {
t.Fatalf("expected ErrProfileNotFound, got %v", err)
}
})
@@ -274,14 +219,14 @@ api_key: secret
})
t.Run("unknown field", func(t *testing.T) {
_, err := repo.GetProfile(ctx, "unknown_field")
_, err := repo.GetProfile(ctx, "unknown-field")
if !errors.Is(err, ErrInvalidYAML) {
t.Fatalf("expected ErrInvalidYAML for strict decode unknown field, got %v", err)
}
})
t.Run("raw api_key rejected", func(t *testing.T) {
_, err := repo.GetProfile(ctx, "raw_api_key")
_, err := repo.GetProfile(ctx, "raw-api-key")
if !errors.Is(err, ErrRawAPIKeyNotAllowed) {
t.Fatalf("expected ErrRawAPIKeyNotAllowed, got %v", err)
}
@@ -406,6 +351,513 @@ model: second
})
}
func TestProfileRepositoriesRejectInvalidEndpoints(t *testing.T) {
tests := []struct {
name string
endpoint string
withBackend bool
}{
{name: "relative", endpoint: "/v1"},
{name: "missing host", endpoint: "https:///v1"},
{name: "unsupported scheme", endpoint: "ftp://provider.example/v1"},
{name: "user information", endpoint: "https://user@provider.example/v1"},
{name: "query", endpoint: "https://provider.example/v1?mode=chat"},
{name: "fragment", endpoint: "https://provider.example/v1#chat"},
{name: "backend with invalid override", endpoint: "/v1", withBackend: true},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
backend := ""
if tc.withBackend {
backend = "backend: openrouter\n"
}
repo := NewFSRepository(fstest.MapFS{
"profiles/invalid.yaml": profileMapFile(fmt.Sprintf(
"id: invalid-endpoint\nmodel: model\n%sendpoint: %q\n",
backend,
tc.endpoint,
)),
}, "profiles")
_, err := repo.GetProfile(context.Background(), "invalid-endpoint")
if !errors.Is(err, ErrInvalidProfile) {
t.Fatalf("expected ErrInvalidProfile, got %v", err)
}
})
}
}
func TestProfileRepositoriesValidateExtraParams(t *testing.T) {
const validProfile = `
id: selected-profile
endpoint: http://localhost:8000/v1
model: model
extra_params:
string_value: enabled
object_value:
nested: true
array_value:
- first
- 3
`
tests := []struct {
name string
definition string
wantErr bool
diagnostics []string
}{
{name: "valid nested values", definition: validProfile},
{
name: "empty key",
definition: `
id: selected-profile
endpoint: http://localhost:8000/v1
model: model
extra_params:
"": value
`,
wantErr: true,
diagnostics: []string{"extra_params", "key must not be empty"},
},
{
name: "non-finite value",
definition: `
id: selected-profile
endpoint: http://localhost:8000/v1
model: model
extra_params:
invalid: .nan
`,
wantErr: true,
diagnostics: []string{"extra_params.invalid", "must be finite"},
},
{
name: "nested non-finite value",
definition: `
id: selected-profile
endpoint: http://localhost:8000/v1
model: model
extra_params:
outer:
invalid: .inf
`,
wantErr: true,
diagnostics: []string{"extra_params.outer.invalid", "must be finite"},
},
{
name: "unsupported decoded value",
definition: `
id: selected-profile
endpoint: http://localhost:8000/v1
model: model
extra_params:
timestamp: 2026-08-11T12:34:56Z
`,
wantErr: true,
diagnostics: []string{"extra_params.timestamp", "unsupported JSON value type"},
},
{
name: "excessive nesting",
definition: deeplyNestedExtraParamsProfile(101),
wantErr: true,
diagnostics: []string{"extra_params", "JSON container depth limit exceeded"},
},
}
for _, source := range profileRepositorySources() {
for _, tc := range tests {
t.Run(source.name+"/"+tc.name, func(t *testing.T) {
repo := source.newRepository(t, map[string]string{"selected.yaml": tc.definition})
got, err := repo.GetProfile(context.Background(), "selected-profile")
if tc.wantErr {
if !errors.Is(err, ErrInvalidProfile) {
t.Fatalf("expected ErrInvalidProfile, got %v", err)
}
if !strings.Contains(err.Error(), "selected.yaml") {
t.Fatalf("expected source path in error, got %v", err)
}
for _, diagnostic := range tc.diagnostics {
if !strings.Contains(err.Error(), diagnostic) {
t.Fatalf("expected error to contain %q, got %v", diagnostic, err)
}
}
return
}
if err != nil {
t.Fatalf("load valid profile: %v", err)
}
if got.ExtraParams["string_value"] != "enabled" {
t.Fatalf("unexpected copied extra params: %#v", got.ExtraParams)
}
objectValue, objectOK := got.ExtraParams["object_value"].(map[string]any)
arrayValue, arrayOK := got.ExtraParams["array_value"].([]any)
if !objectOK || objectValue["nested"] != true ||
!arrayOK || len(arrayValue) != 2 || arrayValue[0] != "first" || arrayValue[1] != 3 {
t.Fatalf("unexpected copied nested extra params: %#v", got.ExtraParams)
}
})
}
}
}
func TestProfileRepositoriesSelectCanonicalYAMLID(t *testing.T) {
const validProfile = `
id: selected-profile
endpoint: http://localhost:8000/v1
model: selected-model
`
tests := []struct {
name string
files map[string]string
wantErr error
diagnostics []string
}{
{
name: "same stem unknown field with different id is unrelated",
files: map[string]string{
"selected-profile.yaml": `
id: unrelated-profile
endpoint: http://localhost:8000/v1
model: unrelated
unknown: true
`,
"valid.yaml": validProfile,
},
},
{
name: "same stem unidentifiable yaml is unrelated",
files: map[string]string{
"selected-profile.yaml": "id: [",
"valid.yaml": validProfile,
},
},
{
name: "same stem raw key with different id is unrelated",
files: map[string]string{
"selected-profile.yaml": `
id: unrelated-profile
endpoint: http://localhost:8000/v1
model: unrelated
api_key: secret
`,
"valid.yaml": validProfile,
},
},
{
name: "leading and trailing whitespace is normalized",
files: map[string]string{
"padded.yaml": `
id: " selected-profile "
endpoint: http://localhost:8000/v1
model: selected-model
`,
},
},
{
name: "blank id is unrelated",
files: map[string]string{
"selected-profile.yaml": `
id: " "
endpoint: http://localhost:8000/v1
model: unrelated
`,
},
wantErr: ErrProfileNotFound,
},
{
name: "normalized duplicates are ambiguous",
files: map[string]string{
"first.yaml": validProfile,
"nested/second.yaml": `
id: " selected-profile "
endpoint: http://localhost:8000/v1
model: duplicate
`,
},
wantErr: ErrInvalidProfile,
diagnostics: []string{"duplicate execution profile id", "first.yaml", "nested/second.yaml"},
},
{
name: "selected unknown field is authoritative",
files: map[string]string{
"malformed.yaml": `
id: selected-profile
endpoint: http://localhost:8000/v1
model: selected-model
unknown: true
`,
},
wantErr: ErrInvalidYAML,
diagnostics: []string{"malformed.yaml"},
},
{
name: "selected raw key is authoritative",
files: map[string]string{
"insecure.yaml": `
id: selected-profile
endpoint: http://localhost:8000/v1
model: selected-model
api_key: secret
`,
},
wantErr: ErrRawAPIKeyNotAllowed,
diagnostics: []string{"insecure.yaml"},
},
{
name: "selected identity in an additional document is authoritative",
files: map[string]string{
"additional-document.yaml": `
---
---
id: selected-profile
endpoint: http://localhost:8000/v1
model: selected-model
`,
},
wantErr: ErrInvalidYAML,
diagnostics: []string{"additional-document.yaml", "exactly one YAML document"},
},
}
for _, source := range profileRepositorySources() {
for _, tc := range tests {
t.Run(source.name+"/"+tc.name, func(t *testing.T) {
repo := source.newRepository(t, tc.files)
got, err := repo.GetProfile(context.Background(), " selected-profile ")
if tc.wantErr != nil {
if !errors.Is(err, tc.wantErr) {
t.Fatalf("expected %v, got %v", tc.wantErr, err)
}
for _, diagnostic := range tc.diagnostics {
if !strings.Contains(err.Error(), diagnostic) {
t.Fatalf("expected error to contain %q, got %v", diagnostic, err)
}
}
return
}
if err != nil {
t.Fatalf("load selected profile: %v", err)
}
if got.ID != "selected-profile" || got.Model != "selected-model" {
t.Fatalf("unexpected selected profile: %+v", got)
}
})
}
}
}
func TestProfileRepositoriesRequireOneYAMLDocument(t *testing.T) {
const profile = `
id: selected-profile
endpoint: http://localhost:8000/v1
model: selected-model
`
tests := []struct {
name string
suffix string
wantErr bool
}{
{name: "comments and trailing whitespace", suffix: "\n# trailing comment\n\n"},
{name: "second populated document", suffix: "\n---\nid: another\n", wantErr: true},
{name: "second empty document", suffix: "\n---\n", wantErr: true},
{name: "malformed trailing yaml", suffix: "\n---\n[", wantErr: true},
{name: "raw key in trailing document", suffix: "\n---\napi_key: secret\n", wantErr: true},
}
for _, source := range profileRepositorySources() {
for _, tc := range tests {
t.Run(source.name+"/"+tc.name, func(t *testing.T) {
repo := source.newRepository(t, map[string]string{"definition.yaml": profile + tc.suffix})
got, err := repo.GetProfile(context.Background(), "selected-profile")
if tc.wantErr {
if !errors.Is(err, ErrInvalidYAML) {
t.Fatalf("expected ErrInvalidYAML, got %v", err)
}
if !strings.Contains(err.Error(), "definition.yaml") {
t.Fatalf("expected source path in error, got %v", err)
}
return
}
if err != nil {
t.Fatalf("load one-document profile: %v", err)
}
if got.ID != "selected-profile" {
t.Fatalf("unexpected profile: %+v", got)
}
})
}
}
}
func TestProfileRepositoriesPreserveOverlayFallbackRules(t *testing.T) {
fallback := staticProfileRepo{profiles: map[string]*domain.ExecutionProfile{
"selected-profile": {ID: "selected-profile", Endpoint: "http://fallback", Model: "fallback-model"},
}}
tests := []struct {
name string
files map[string]string
wantModel string
wantErr error
}{
{
name: "same stem malformed different id falls back",
files: map[string]string{
"selected-profile.yaml": `
id: unrelated-profile
endpoint: http://localhost:8000/v1
model: unrelated
unknown: true
`,
},
wantModel: "fallback-model",
},
{
name: "blank id falls back",
files: map[string]string{
"selected-profile.yaml": `
id: " "
endpoint: http://localhost:8000/v1
model: unrelated
`,
},
wantModel: "fallback-model",
},
{
name: "selected malformed profile stops fallback",
files: map[string]string{
"other-name.yaml": `
id: selected-profile
endpoint: http://localhost:8000/v1
model: selected
unknown: true
`,
},
wantErr: ErrInvalidYAML,
},
}
for _, source := range profileRepositorySources() {
for _, tc := range tests {
t.Run(source.name+"/"+tc.name, func(t *testing.T) {
primary := source.newRepository(t, tc.files)
got, err := NewOverlayRepository(primary, fallback).GetProfile(context.Background(), "selected-profile")
if tc.wantErr != nil {
if !errors.Is(err, tc.wantErr) {
t.Fatalf("expected %v, got %v", tc.wantErr, err)
}
return
}
if err != nil {
t.Fatalf("load fallback profile: %v", err)
}
if got.Model != tc.wantModel {
t.Fatalf("model = %q, want %q", got.Model, tc.wantModel)
}
})
}
}
}
func TestProfileRepositoryReadsSourcesFreshOnEveryLookup(t *testing.T) {
newSource := func() (*recordingProfileFS, Repository) {
fsys := &recordingProfileFS{FS: fstest.MapFS{
"target.yaml": profileMapFile(`
id: target
endpoint: http://localhost:8000/v1
model: target-model
`),
"unrelated.yaml": profileMapFile(`
id: unrelated
endpoint: http://localhost:8000/v1
model: unrelated-model
`),
}}
return fsys, NewFSRepository(fsys, ".")
}
t.Run("selected source", func(t *testing.T) {
fsys, repo := newSource()
for lookup := 1; lookup <= 2; lookup++ {
got, err := repo.GetProfile(context.Background(), "target")
if err != nil {
t.Fatalf("lookup %d: %v", lookup, err)
}
if got.Model != "target-model" {
t.Fatalf("lookup %d model = %q", lookup, got.Model)
}
for _, name := range []string{"target.yaml", "unrelated.yaml"} {
if count := fsys.openCount(name); count != lookup {
t.Fatalf("%s opens after lookup %d = %d, want %d", name, lookup, count, lookup)
}
}
}
})
t.Run("overlay fallthrough", func(t *testing.T) {
primaryFS := &recordingProfileFS{FS: fstest.MapFS{
"unrelated.yaml": profileMapFile(`
id: unrelated
endpoint: http://localhost:8000/v1
model: unrelated-model
`),
}}
fallbackFS, fallback := newSource()
repo := NewOverlayRepository(NewFSRepository(primaryFS, "."), fallback)
for lookup := 1; lookup <= 2; lookup++ {
got, err := repo.GetProfile(context.Background(), "target")
if err != nil {
t.Fatalf("lookup %d: %v", lookup, err)
}
if got.Model != "target-model" {
t.Fatalf("lookup %d model = %q", lookup, got.Model)
}
if count := primaryFS.openCount("unrelated.yaml"); count != lookup {
t.Fatalf("primary opens after lookup %d = %d, want %d", lookup, count, lookup)
}
if count := fallbackFS.openCount("target.yaml"); count != lookup {
t.Fatalf("fallback opens after lookup %d = %d, want %d", lookup, count, lookup)
}
}
})
}
func TestProfileRepositoriesRejectInvalidExecutionSettings(t *testing.T) {
ctx := context.Background()
t.Run("operating-system filesystem", func(t *testing.T) {
dir := t.TempDir()
writeProfileTestFile(t, filepath.Join(dir, "invalid.yaml"), `
id: invalid
endpoint: http://localhost:8000/v1
model: model
temperature: .nan
`)
_, err := NewFilesystemRepository(dir).GetProfile(ctx, "invalid")
if !errors.Is(err, ErrInvalidProfile) {
t.Fatalf("expected ErrInvalidProfile, got %v", err)
}
})
t.Run("fs.FS", func(t *testing.T) {
repo := NewFSRepository(fstest.MapFS{
"profiles/invalid.yaml": profileMapFile(`
id: invalid
endpoint: http://localhost:8000/v1
model: model
top_p: .inf
`),
}, "profiles")
_, err := repo.GetProfile(ctx, "invalid")
if !errors.Is(err, ErrInvalidProfile) {
t.Fatalf("expected ErrInvalidProfile, got %v", err)
}
})
}
func TestOverlayRepository(t *testing.T) {
ctx := context.Background()
primaryProfile := &domain.ExecutionProfile{ID: "shared", Endpoint: "http://primary", Model: "primary"}
@@ -495,6 +947,77 @@ func TestOverlayRepository(t *testing.T) {
})
}
type profileRepositorySource struct {
name string
newRepository func(t *testing.T, files map[string]string) Repository
}
type recordingProfileFS struct {
fs.FS
mu sync.Mutex
opened []string
}
func (f *recordingProfileFS) Open(name string) (fs.File, error) {
f.mu.Lock()
f.opened = append(f.opened, name)
f.mu.Unlock()
return f.FS.Open(name)
}
func (f *recordingProfileFS) openCount(name string) int {
f.mu.Lock()
defer f.mu.Unlock()
count := 0
for _, opened := range f.opened {
if opened == name {
count++
}
}
return count
}
func profileRepositorySources() []profileRepositorySource {
return []profileRepositorySource{
{
name: "operating system",
newRepository: func(t *testing.T, files map[string]string) Repository {
t.Helper()
root := t.TempDir()
for name, content := range files {
filePath := filepath.Join(root, filepath.FromSlash(name))
if err := os.MkdirAll(filepath.Dir(filePath), 0o755); err != nil {
t.Fatalf("create profile directory: %v", err)
}
writeProfileTestFile(t, filePath, content)
}
return NewFilesystemRepository(root)
},
},
{
name: "filesystem",
newRepository: func(t *testing.T, files map[string]string) Repository {
t.Helper()
fsys := make(fstest.MapFS, len(files))
for name, content := range files {
fsys[name] = profileMapFile(content)
}
return NewFSRepository(fsys, ".")
},
},
}
}
func deeplyNestedExtraParamsProfile(depth int) string {
var definition strings.Builder
definition.WriteString("id: selected-profile\nendpoint: http://localhost:8000/v1\nmodel: model\nextra_params:\n")
for level := 0; level < depth; level++ {
fmt.Fprintf(&definition, "%slevel_%d:\n", strings.Repeat(" ", level+1), level)
}
fmt.Fprintf(&definition, "%svalue: true\n", strings.Repeat(" ", depth+1))
return definition.String()
}
func profileMapFile(content string) *fstest.MapFile {
return &fstest.MapFile{Data: []byte(strings.TrimLeft(content, "\n"))}
}

View File

@@ -5,6 +5,7 @@ import (
"context"
"errors"
"fmt"
"strings"
"text/template"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
@@ -18,6 +19,8 @@ var (
ErrInvalidMessageRole = errors.New("invalid or empty message role")
)
const artifactTextChunkSize = 64 * 1024
type goRenderer struct{}
func NewGoRenderer() Renderer {
@@ -25,11 +28,13 @@ func NewGoRenderer() Renderer {
}
func (r *goRenderer) Render(ctx context.Context, definition *domain.PromptDefinition, inputs map[string]*domain.Artifact, vars map[string]string) (*domain.RenderedPrompt, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
if definition == nil {
return nil, fmt.Errorf("%w: nil prompt definition", ErrRenderFailure)
}
// 1. Verify required inputs
for _, in := range definition.Inputs {
if !in.Required {
continue
@@ -40,44 +45,54 @@ func (r *goRenderer) Render(ctx context.Context, definition *domain.PromptDefini
}
}
// 2. Setup template functions
resolver := newArtifactTextResolver(ctx, inputs)
funcs := template.FuncMap{
"input": func(name string) (string, error) {
art, ok := inputs[name]
if !ok || art == nil {
return "", fmt.Errorf("%w: %s", ErrUnknownInput, name)
}
return string(art.Body), nil
},
"input": resolver.resolve,
}
sessionID, err := renderSessionID(definition.SessionID, funcs, vars)
if err := ctx.Err(); err != nil {
return nil, err
}
sessionID, err := renderSessionID(ctx, definition.SessionID, funcs, vars)
if err != nil {
return nil, err
}
if err := ctx.Err(); err != nil {
return nil, err
}
var renderedMessages []domain.RenderedMessage
renderedMessages := make([]domain.RenderedMessage, 0, len(definition.Templates))
for i, tmplMsg := range definition.Templates {
select {
case <-ctx.Done():
return nil, ctx.Err()
default:
if err := ctx.Err(); err != nil {
return nil, err
}
if tmplMsg.Role == "" {
return nil, fmt.Errorf("%w: message %d", ErrInvalidMessageRole, i)
}
// Parse and execute template
tmpl, err := template.New(fmt.Sprintf("msg_%d", i)).Funcs(funcs).Option("missingkey=error").Parse(tmplMsg.Content)
if err != nil {
return nil, fmt.Errorf("%w: message %d: %v", ErrInvalidTemplate, i, err)
if err := ctx.Err(); err != nil {
return nil, err
}
tmpl, parseErr := template.New(fmt.Sprintf("msg_%d", i)).Funcs(funcs).Option("missingkey=error").Parse(tmplMsg.Content)
if err := ctx.Err(); err != nil {
return nil, err
}
if parseErr != nil {
return nil, fmt.Errorf("%w: message %d: %v", ErrInvalidTemplate, i, parseErr)
}
if err := ctx.Err(); err != nil {
return nil, err
}
var buf bytes.Buffer
if err := tmpl.Execute(&buf, vars); err != nil {
return nil, fmt.Errorf("%w: message %d: %w", ErrRenderFailure, i, err)
executeErr := tmpl.Execute(&buf, vars)
if err := ctx.Err(); err != nil {
return nil, err
}
if executeErr != nil {
return nil, fmt.Errorf("%w: message %d: %w", ErrRenderFailure, i, executeErr)
}
renderedMessages = append(renderedMessages, domain.RenderedMessage{
@@ -85,6 +100,13 @@ func (r *goRenderer) Render(ctx context.Context, definition *domain.PromptDefini
Content: buf.String(),
CacheControl: cloneCacheControl(tmplMsg.CacheControl),
})
if err := ctx.Err(); err != nil {
return nil, err
}
}
if err := ctx.Err(); err != nil {
return nil, err
}
return &domain.RenderedPrompt{
@@ -93,15 +115,75 @@ func (r *goRenderer) Render(ctx context.Context, definition *domain.PromptDefini
}, nil
}
func renderSessionID(raw string, funcs template.FuncMap, vars map[string]string) (string, error) {
tmpl, err := template.New("session_id").Funcs(funcs).Option("missingkey=error").Parse(raw)
if err != nil {
return "", fmt.Errorf("%w: session_id: %v", ErrInvalidTemplate, err)
type artifactTextResolver struct {
ctx context.Context
inputs map[string]*domain.Artifact
textByName map[string]string
}
func newArtifactTextResolver(ctx context.Context, inputs map[string]*domain.Artifact) *artifactTextResolver {
return &artifactTextResolver{
ctx: ctx,
inputs: inputs,
textByName: make(map[string]string),
}
}
func (r *artifactTextResolver) resolve(name string) (string, error) {
if err := r.ctx.Err(); err != nil {
return "", err
}
artifact, ok := r.inputs[name]
if !ok || artifact == nil {
return "", fmt.Errorf("%w: %s", ErrUnknownInput, name)
}
if text, ok := r.textByName[name]; ok {
return text, nil
}
var builder strings.Builder
builder.Grow(len(artifact.Body))
for start := 0; start < len(artifact.Body); start += artifactTextChunkSize {
if err := r.ctx.Err(); err != nil {
return "", err
}
end := min(start+artifactTextChunkSize, len(artifact.Body))
_, _ = builder.Write(artifact.Body[start:end])
if err := r.ctx.Err(); err != nil {
return "", err
}
}
if err := r.ctx.Err(); err != nil {
return "", err
}
text := builder.String()
r.textByName[name] = text
return text, nil
}
func renderSessionID(ctx context.Context, raw string, funcs template.FuncMap, vars map[string]string) (string, error) {
if err := ctx.Err(); err != nil {
return "", err
}
tmpl, parseErr := template.New("session_id").Funcs(funcs).Option("missingkey=error").Parse(raw)
if err := ctx.Err(); err != nil {
return "", err
}
if parseErr != nil {
return "", fmt.Errorf("%w: session_id: %v", ErrInvalidTemplate, parseErr)
}
if err := ctx.Err(); err != nil {
return "", err
}
var buf bytes.Buffer
if err := tmpl.Execute(&buf, vars); err != nil {
return "", fmt.Errorf("%w: session_id: %w", ErrRenderFailure, err)
executeErr := tmpl.Execute(&buf, vars)
if err := ctx.Err(); err != nil {
return "", err
}
if executeErr != nil {
return "", fmt.Errorf("%w: session_id: %w", ErrRenderFailure, executeErr)
}
sessionID, err := domain.NormalizeSessionID(buf.String())

View File

@@ -1,6 +1,7 @@
package prompt
import (
"bytes"
"context"
"errors"
"strings"
@@ -228,6 +229,24 @@ func TestGoRenderer_Render(t *testing.T) {
}
})
t.Run("malformed rendered session id fails rendering", func(t *testing.T) {
def := &domain.PromptDefinition{
SessionID: "{{ .session_id }}",
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
Templates: []domain.PromptMessageTemplate{
{Role: "system", Content: "Speak in a {{.tone}} tone."},
},
}
_, err := renderer.Render(ctx, def, inputs, map[string]string{
"tone": "concise",
"session_id": "session" + string([]byte{0xff}),
})
if !errors.Is(err, ErrRenderFailure) {
t.Fatalf("expected ErrRenderFailure, got %v", err)
}
})
t.Run("inserting required input artifact", func(t *testing.T) {
def := &domain.PromptDefinition{
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
@@ -343,3 +362,186 @@ func TestGoRenderer_Render(t *testing.T) {
}
})
}
func TestGoRendererCancellation(t *testing.T) {
t.Run("before session parsing", func(t *testing.T) {
definition := &domain.PromptDefinition{
SessionID: "{{ malformed",
Templates: []domain.PromptMessageTemplate{
{Role: "user", Content: "not rendered"},
},
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
result, err := NewGoRenderer().Render(ctx, definition, nil, nil)
if result != nil || !errors.Is(err, context.Canceled) {
t.Fatalf("result=%#v err=%v, want nil/context.Canceled", result, err)
}
if errors.Is(err, ErrInvalidTemplate) {
t.Fatalf("pre-canceled render parsed the malformed session: %v", err)
}
result, err = NewGoRenderer().Render(context.Background(), definition, nil, nil)
if result != nil || !errors.Is(err, ErrInvalidTemplate) {
t.Fatalf("active render result=%#v err=%v, want nil/ErrInvalidTemplate", result, err)
}
})
t.Run("during artifact text conversion", func(t *testing.T) {
ctx := newCancelOnCheckContext(3)
body := bytes.Repeat([]byte("x"), artifactTextChunkSize*2)
original := append([]byte(nil), body...)
resolver := newArtifactTextResolver(ctx, map[string]*domain.Artifact{
"document": {Body: body},
})
text, err := resolver.resolve("document")
if text != "" || !errors.Is(err, context.Canceled) {
t.Fatalf("text length=%d err=%v, want empty/context.Canceled", len(text), err)
}
if _, published := resolver.textByName["document"]; published {
t.Fatal("canceled conversion published partial artifact text")
}
if !bytes.Equal(body, original) {
t.Fatal("resolver mutated the artifact body")
}
})
t.Run("after final message execution", func(t *testing.T) {
definition := &domain.PromptDefinition{
Templates: []domain.PromptMessageTemplate{
{Role: "user", Content: "fully rendered"},
},
}
counter := &checkCountingContext{Context: context.Background()}
if _, err := NewGoRenderer().Render(counter, definition, nil, nil); err != nil {
t.Fatalf("count render checkpoints: %v", err)
}
// The final three checks occur after template execution, after the
// message is assembled, and immediately before publication.
ctx := newCancelOnCheckContext(counter.checks - 2)
result, err := NewGoRenderer().Render(ctx, definition, nil, nil)
if result != nil || !errors.Is(err, context.Canceled) {
t.Fatalf("result=%#v err=%v, want nil/context.Canceled", result, err)
}
})
}
func TestGoRendererArtifactTextLifecycle(t *testing.T) {
body := []byte{'a', 0xff, 'b', 0xfe}
original := append([]byte(nil), body...)
artifact := &domain.Artifact{Body: body}
inputs := map[string]*domain.Artifact{"document": artifact}
definition := &domain.PromptDefinition{
Templates: []domain.PromptMessageTemplate{
{Role: "user", Content: "{{input \"document\"}}|{{input \"document\"}}"},
},
}
first, err := NewGoRenderer().Render(context.Background(), definition, inputs, nil)
if err != nil {
t.Fatalf("first render: %v", err)
}
wantFirst := append(append(append([]byte(nil), body...), '|'), body...)
if !bytes.Equal([]byte(first.Messages[0].Content), wantFirst) {
t.Fatalf("rendered bytes=%v, want %v", []byte(first.Messages[0].Content), wantFirst)
}
if !bytes.Equal(body, original) {
t.Fatalf("renderer mutated artifact body: got %v want %v", body, original)
}
body[0] = 'z'
if bytes.Equal([]byte(first.Messages[0].Content), append(append(append([]byte(nil), body...), '|'), body...)) {
t.Fatal("completed render aliases the artifact body")
}
second, err := NewGoRenderer().Render(context.Background(), definition, inputs, nil)
if err != nil {
t.Fatalf("second render: %v", err)
}
wantSecond := append(append(append([]byte(nil), body...), '|'), body...)
if !bytes.Equal([]byte(second.Messages[0].Content), wantSecond) {
t.Fatalf("second render reused text from another call: got %v want %v", []byte(second.Messages[0].Content), wantSecond)
}
nilInputs := map[string]*domain.Artifact{"document": nil}
result, err := NewGoRenderer().Render(context.Background(), definition, nilInputs, nil)
if result != nil || !errors.Is(err, ErrUnknownInput) || !errors.Is(err, ErrRenderFailure) {
t.Fatalf("nil input result=%#v err=%v, want ErrUnknownInput and ErrRenderFailure", result, err)
}
}
func BenchmarkGoRendererArtifactReferences(b *testing.B) {
body := bytes.Repeat([]byte("document content "), (artifactTextChunkSize*4)/len("document content "))
inputs := map[string]*domain.Artifact{
"document": {Body: body},
}
tests := []struct {
name string
definition *domain.PromptDefinition
}{
{
name: "one reference",
definition: &domain.PromptDefinition{
Templates: []domain.PromptMessageTemplate{
{Role: "user", Content: "{{input \"document\"}}"},
},
},
},
{
name: "repeated across session and messages",
definition: &domain.PromptDefinition{
SessionID: "document-{{len (input \"document\")}}",
Templates: []domain.PromptMessageTemplate{
{Role: "system", Content: "{{input \"document\"}}"},
{Role: "user", Content: "{{input \"document\"}} {{input \"document\"}}"},
},
},
},
}
for _, tc := range tests {
b.Run(tc.name, func(b *testing.B) {
renderer := NewGoRenderer()
b.ReportAllocs()
b.SetBytes(int64(len(body)))
for range b.N {
if _, err := renderer.Render(context.Background(), tc.definition, inputs, nil); err != nil {
b.Fatal(err)
}
}
})
}
}
type checkCountingContext struct {
context.Context
checks int
}
func (c *checkCountingContext) Err() error {
c.checks++
return c.Context.Err()
}
type cancelOnCheckContext struct {
context.Context
cancel context.CancelFunc
remaining int
}
func newCancelOnCheckContext(checks int) *cancelOnCheckContext {
ctx, cancel := context.WithCancel(context.Background())
return &cancelOnCheckContext{Context: ctx, cancel: cancel, remaining: checks}
}
func (c *cancelOnCheckContext) Err() error {
if c.Context.Err() == nil {
c.remaining--
if c.remaining == 0 {
c.cancel()
}
}
return c.Context.Err()
}

View File

@@ -0,0 +1,98 @@
package promptdef
import (
"fmt"
"io/fs"
"os"
"path"
"path/filepath"
"strings"
"gitea.maximumdirect.net/eric/promptkit/internal/filecatalog"
)
type contentSourceRoot interface {
readContentFile(sourcePath string, contentFile string) (string, string, error)
}
type osContentSourceRoot struct {
root string
sourcePathsRelative bool
}
func (r osContentSourceRoot) readContentFile(sourcePath string, contentFile string) (string, string, error) {
if strings.TrimSpace(contentFile) == "" {
return "", "", fmt.Errorf("path is required")
}
if filepath.IsAbs(contentFile) {
return "", "", fmt.Errorf("path %q must be relative", contentFile)
}
root, err := filepath.Abs(r.root)
if err != nil {
return "", "", fmt.Errorf("resolve source root %q: %w", r.root, err)
}
canonicalRoot, err := filepath.EvalSymlinks(root)
if err != nil {
return "", "", fmt.Errorf("resolve source root %q: %w", r.root, err)
}
promptPath := sourcePath
if r.sourcePathsRelative && !filepath.IsAbs(promptPath) {
promptPath = filepath.Join(root, filepath.FromSlash(promptPath))
} else {
promptPath, err = filepath.Abs(promptPath)
if err != nil {
return "", "", fmt.Errorf("resolve prompt source %q: %w", sourcePath, err)
}
}
resolvedPath := filepath.Clean(filepath.Join(filepath.Dir(promptPath), contentFile))
if !containsOSPath(root, resolvedPath) {
return "", "", fmt.Errorf("path %q escapes source root %q", contentFile, r.root)
}
canonicalPath, err := filepath.EvalSymlinks(resolvedPath)
if err != nil {
return "", "", err
}
if !containsOSPath(canonicalRoot, canonicalPath) {
return "", "", fmt.Errorf("path %q escapes source root %q", contentFile, r.root)
}
body, err := os.ReadFile(canonicalPath)
if err != nil {
return "", "", err
}
return string(body), resolvedPath, nil
}
type fsContentSourceRoot struct {
fsys fs.FS
root string
}
func (r fsContentSourceRoot) readContentFile(sourcePath string, contentFile string) (string, string, error) {
root := filecatalog.CleanFSRoot(r.root)
cleanSourcePath := path.Clean(sourcePath)
if cleanSourcePath == root {
root = path.Dir(root)
}
resolvedPath, _, err := filecatalog.ResolveFSPath(root, path.Dir(cleanSourcePath), contentFile)
if err != nil {
return "", "", err
}
body, err := fs.ReadFile(r.fsys, resolvedPath)
if err != nil {
return "", "", err
}
return string(body), resolvedPath, nil
}
func containsOSPath(root string, name string) bool {
relative, err := filepath.Rel(root, name)
if err != nil || filepath.IsAbs(relative) {
return false
}
return relative != ".." && !strings.HasPrefix(relative, ".."+string(filepath.Separator))
}

View File

@@ -5,14 +5,11 @@ import (
"context"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path"
"path/filepath"
"strings"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
"gitea.maximumdirect.net/eric/promptkit/internal/filecatalog"
"gopkg.in/yaml.v3"
)
@@ -22,13 +19,8 @@ var (
ErrInvalidPromptDefinition = errors.New("invalid prompt definition configuration")
)
type filesystemRepository struct {
dir string
}
type fsRepository struct {
fsys fs.FS
root string
type sourceRepository struct {
source promptDefinitionSource
}
type promptDefinitionFile struct {
@@ -69,19 +61,47 @@ type promptOutputContractFile struct {
}
func NewFilesystemRepository(dir string) Repository {
return &filesystemRepository{dir: dir}
return &sourceRepository{
source: osPromptSource{
root: dir,
contentRoot: osContentSourceRoot{root: dir},
},
}
}
func NewFSRepository(fsys fs.FS, root string) Repository {
return &fsRepository{fsys: fsys, root: root}
return &sourceRepository{
source: fsPromptSource{
fsys: fsys,
root: root,
contentRoot: fsContentSourceRoot{fsys: fsys, root: root},
},
}
}
func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error) {
// NewFileRepository constructs a repository for one operating-system prompt file.
func NewFileRepository(fsys fs.FS, file string, sourceDir string) Repository {
return &sourceRepository{
source: fsPromptSource{
fsys: fsys,
root: file,
contentRoot: osContentSourceRoot{
root: sourceDir,
sourcePathsRelative: true,
},
},
}
}
func (r *sourceRepository) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error) {
if strings.TrimSpace(id) == "" {
return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition)
}
if r == nil || r.source == nil {
return nil, errors.New("failed to read prompt definition directory: source is nil")
}
files, err := filecatalog.FindYAMLFiles(ctx, r.dir)
files, err := r.source.findYAMLFiles(ctx)
if err != nil {
return nil, fmt.Errorf("failed to read prompt definition directory: %w", err)
}
@@ -94,34 +114,25 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
default:
}
relPath := filecatalog.RelativePath(r.dir, fullPath)
fileMatch := filecatalog.Stem(filepath.Base(fullPath)) == id
raw, err := loadPromptDefinitionFile(fullPath)
relPath := r.source.displayPath(fullPath)
data, err := r.source.readDefinition(fullPath)
if err != nil {
if fileMatch || promptDefinitionFileHasID(fullPath, id) {
return nil, fmt.Errorf("failed to read prompt definition file %s: %w", relPath, err)
}
raw, err := decodePromptDefinition(data)
if err != nil {
if promptDefinitionDataMatches(data, id, version) {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
}
continue
}
def, err := normalizePromptDefinition(raw, fullPath)
if err != nil {
if fileMatch || strings.TrimSpace(raw.ID) == id {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, relPath, err)
}
continue
}
if def.ID != id {
continue
}
if version != "" && def.Version != version {
if !promptDefinitionMatches(raw, id, version) {
continue
}
matches = append(matches, promptDefinitionMatch{
def: def,
path: relPath,
raw: raw,
sourcePath: fullPath,
path: relPath,
})
}
@@ -137,130 +148,21 @@ func (r *filesystemRepository) GetPromptDefinition(ctx context.Context, id strin
}
if len(matches) == 1 {
return matches[0].def, nil
match := matches[0]
def, err := normalizePromptDefinition(match.raw, r.source, match.sourcePath)
if err != nil {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, match.path, err)
}
return def, nil
}
return nil, ErrPromptDefinitionNotFound
}
func (r *fsRepository) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error) {
return loadPromptDefinition(ctx, r.fsys, r.root, id, version)
}
type promptDefinitionMatch struct {
def *domain.PromptDefinition
path string
}
func loadPromptDefinitionFile(path string) (*promptDefinitionFile, error) {
data, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("failed to read prompt definition file: %w", err)
}
var raw promptDefinitionFile
decoder := yaml.NewDecoder(bytes.NewReader(data))
decoder.KnownFields(true)
if err := decoder.Decode(&raw); err != nil {
return nil, err
}
return &raw, nil
}
func promptDefinitionFileHasID(path string, id string) bool {
data, err := os.ReadFile(path)
if err != nil {
return false
}
var raw struct {
ID string `yaml:"id"`
}
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&raw); err != nil {
return false
}
return strings.TrimSpace(raw.ID) == id
}
func loadPromptDefinition(ctx context.Context, fsys fs.FS, root string, id string, version string) (*domain.PromptDefinition, error) {
if strings.TrimSpace(id) == "" {
return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidPromptDefinition)
}
if fsys == nil {
return nil, fmt.Errorf("failed to read prompt definition directory: filesystem is nil")
}
files, err := filecatalog.FindFSYAMLFiles(ctx, fsys, root)
if err != nil {
return nil, fmt.Errorf("failed to read prompt definition directory: %w", err)
}
cleanRoot := filecatalog.CleanFSRoot(root)
rootInfo, err := fs.Stat(fsys, cleanRoot)
if err != nil {
return nil, fmt.Errorf("failed to read prompt definition directory: %w", err)
}
var matches []promptDefinitionMatch
for _, fullPath := range files {
select {
case <-ctx.Done():
return nil, ctx.Err()
default:
}
relPath := filecatalog.DisplayPath(root, fullPath)
fileMatch := filecatalog.Stem(path.Base(fullPath)) == id
data, err := fs.ReadFile(fsys, fullPath)
if err != nil {
if fileMatch {
return nil, fmt.Errorf("%w: %s: failed to read prompt definition file: %v", ErrInvalidYAML, relPath, err)
}
continue
}
raw, err := decodePromptDefinition(data)
if err != nil {
if fileMatch || promptDefinitionDataHasID(data, id) {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidYAML, relPath, err)
}
continue
}
def, err := normalizePromptDefinitionFromFS(raw, fsys, root, fullPath, rootInfo.IsDir())
if err != nil {
if fileMatch || strings.TrimSpace(raw.ID) == id {
return nil, fmt.Errorf("%w: %s: %v", ErrInvalidPromptDefinition, relPath, err)
}
continue
}
if def.ID != id {
continue
}
if version != "" && def.Version != version {
continue
}
matches = append(matches, promptDefinitionMatch{
def: def,
path: relPath,
})
}
if len(matches) > 1 {
paths := make([]string, 0, len(matches))
for _, match := range matches {
paths = append(paths, match.path)
}
if version != "" {
return nil, fmt.Errorf("%w: duplicate prompt definition id %q version %q found in: %s", ErrInvalidPromptDefinition, id, version, strings.Join(paths, ", "))
}
return nil, fmt.Errorf("%w: duplicate prompt definition id %q found in: %s", ErrInvalidPromptDefinition, id, strings.Join(paths, ", "))
}
if len(matches) == 1 {
return matches[0].def, nil
}
return nil, ErrPromptDefinitionNotFound
raw *promptDefinitionFile
sourcePath string
path string
}
func decodePromptDefinition(data []byte) (*promptDefinitionFile, error) {
@@ -270,59 +172,44 @@ func decodePromptDefinition(data []byte) (*promptDefinitionFile, error) {
if err := decoder.Decode(&raw); err != nil {
return nil, err
}
var additional yaml.Node
if err := decoder.Decode(&additional); err != io.EOF {
if err != nil {
return nil, err
}
return nil, errors.New("prompt definition file must contain exactly one YAML document")
}
return &raw, nil
}
func promptDefinitionDataHasID(data []byte, id string) bool {
func promptDefinitionDataMatches(data []byte, id string, version string) bool {
var raw struct {
ID string `yaml:"id"`
ID string `yaml:"id"`
Version string `yaml:"version"`
}
if err := yaml.NewDecoder(bytes.NewReader(data)).Decode(&raw); err != nil {
return false
}
return strings.TrimSpace(raw.ID) == id
return promptSelectorMatches(raw.ID, raw.Version, id, version)
}
func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*domain.PromptDefinition, error) {
promptDir := filepath.Dir(sourcePath)
return normalizePromptDefinitionWithContent(raw, func(contentFile string) (string, string, error) {
resolvedPath := strings.TrimSpace(contentFile)
if !filepath.IsAbs(resolvedPath) {
resolvedPath = filepath.Join(promptDir, resolvedPath)
}
resolvedPath = filepath.Clean(resolvedPath)
body, err := os.ReadFile(resolvedPath)
if err != nil {
return "", "", err
}
return string(body), resolvedPath, nil
})
func promptDefinitionMatches(raw *promptDefinitionFile, id string, version string) bool {
if raw == nil {
return false
}
return promptSelectorMatches(raw.ID, raw.Version, id, version)
}
func normalizePromptDefinitionFromFS(raw *promptDefinitionFile, fsys fs.FS, root string, sourcePath string, rootIsDir bool) (*domain.PromptDefinition, error) {
promptDir := path.Dir(sourcePath)
return normalizePromptDefinitionWithContent(raw, func(contentFile string) (string, string, error) {
var resolvedPath string
if rootIsDir {
var err error
resolvedPath, _, err = filecatalog.ResolveFSPath(root, promptDir, contentFile)
if err != nil {
return "", "", err
}
} else {
resolvedPath = strings.TrimSpace(contentFile)
if !path.IsAbs(resolvedPath) {
resolvedPath = path.Join(promptDir, resolvedPath)
}
resolvedPath = strings.TrimPrefix(path.Clean(resolvedPath), "/")
}
func promptSelectorMatches(rawID string, rawVersion string, id string, version string) bool {
if strings.TrimSpace(rawID) != id {
return false
}
return version == "" || strings.TrimSpace(rawVersion) == version
}
body, err := fs.ReadFile(fsys, resolvedPath)
if err != nil {
return "", "", err
}
return string(body), resolvedPath, nil
func normalizePromptDefinition(raw *promptDefinitionFile, sourceRoot contentSourceRoot, sourcePath string) (*domain.PromptDefinition, error) {
return normalizePromptDefinitionWithContent(raw, func(contentFile string) (string, string, error) {
return sourceRoot.readContentFile(sourcePath, contentFile)
})
}
@@ -402,17 +289,14 @@ func normalizePromptDefinitionWithContent(raw *promptDefinitionFile, readContent
})
}
if !isValidOutputFormat(raw.Output.Format) {
return nil, fmt.Errorf("invalid output format: %q", raw.Output.Format)
outputContract := domain.OutputContract{
Format: raw.Output.Format,
ValidationMode: raw.Output.ValidationMode,
SchemaPath: strings.TrimSpace(raw.Output.SchemaPath),
RepairAttempts: raw.Output.RepairAttempts,
}
if !isValidValidationMode(raw.Output.ValidationMode) {
return nil, fmt.Errorf("invalid validation mode: %q", raw.Output.ValidationMode)
}
if raw.Output.ValidationMode == domain.ValidationJSONSchema && strings.TrimSpace(raw.Output.SchemaPath) == "" {
return nil, errors.New("output.schema_path is required when output.validation_mode is json_schema")
}
if raw.Output.RepairAttempts < 0 {
return nil, errors.New("output.repair_attempts must be greater than or equal to 0")
if err := domain.ValidateOutputContract(outputContract); err != nil {
return nil, fmt.Errorf("output: %w", err)
}
defaultProfile := ""
@@ -432,12 +316,7 @@ func normalizePromptDefinitionWithContent(raw *promptDefinitionFile, readContent
Inputs: inputs,
Templates: templates,
OutputFormat: raw.Output.Format,
Validation: domain.OutputContract{
Format: raw.Output.Format,
ValidationMode: raw.Output.ValidationMode,
SchemaPath: strings.TrimSpace(raw.Output.SchemaPath),
RepairAttempts: raw.Output.RepairAttempts,
},
Validation: outputContract,
}, nil
}
@@ -464,21 +343,3 @@ func normalizeCacheControl(raw *cacheControlFile) (*domain.CacheControl, error)
TTL: ttl,
}, nil
}
func isValidOutputFormat(f domain.OutputFormat) bool {
switch f {
case domain.FormatText, domain.FormatMarkdown, domain.FormatJSON:
return true
default:
return false
}
}
func isValidValidationMode(m domain.ValidationMode) bool {
switch m {
case domain.ValidationNone, domain.ValidationBasic, domain.ValidationJSON, domain.ValidationJSONSchema:
return true
default:
return false
}
}

View File

@@ -3,23 +3,51 @@ package promptdef
import (
"context"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"testing/fstest"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
)
func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
func TestPromptRepositoryDefinitionFixtures(t *testing.T) {
sources := []struct {
name string
newRepository func(string) Repository
contentPathsAreFull bool
}{
{
name: "operating system",
newRepository: NewFilesystemRepository,
contentPathsAreFull: true,
},
{
name: "filesystem",
newRepository: func(root string) Repository {
return NewFSRepository(os.DirFS(root), ".")
},
},
}
for _, source := range sources {
t.Run(source.name, func(t *testing.T) {
testPromptRepositoryDefinitionFixtures(t, source.newRepository, source.contentPathsAreFull)
})
}
}
func testPromptRepositoryDefinitionFixtures(t *testing.T, newRepository func(string) Repository, contentPathsAreFull bool) {
tmpDir := t.TempDir()
if err := copyTree("testdata", tmpDir); err != nil {
t.Fatalf("failed to copy testdata: %v", err)
}
repo := NewFilesystemRepository(tmpDir)
repo := newRepository(tmpDir)
ctx := context.Background()
t.Run("valid inline prompt", func(t *testing.T) {
@@ -64,8 +92,8 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
if p.Templates[1].ContentFile == "" {
t.Fatal("expected ContentFile source metadata to be preserved")
}
if !filepath.IsAbs(p.Templates[1].ContentFile) {
t.Fatalf("expected resolved content_file path to be absolute, got %q", p.Templates[1].ContentFile)
if filepath.IsAbs(p.Templates[1].ContentFile) != contentPathsAreFull {
t.Fatalf("unexpected content_file path representation: %q", p.Templates[1].ContentFile)
}
})
@@ -287,20 +315,20 @@ output:
targetErr error
errSubstrs []string
}{
{name: "invalid YAML", id: "invalid_yaml", targetErr: ErrInvalidYAML},
{name: "missing id", id: "missing_id", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"id is required"}},
{name: "no messages", id: "no_messages", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"at least one message is required"}},
{name: "both content and content_file", id: "both_content_and_content_file", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"exactly one"}},
{name: "neither content nor content_file", id: "neither_content_nor_content_file", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"exactly one"}},
{name: "missing content_file", id: "missing_content_file", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"failed to read content_file"}},
{name: "duplicate input names", id: "duplicate_input_names", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"duplicate input name"}},
{name: "invalid validation mode", id: "invalid_validation_mode", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"invalid validation mode"}},
{name: "json_schema without schema_path", id: "json_schema_without_schema_path", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"schema_path"}},
{name: "unknown input field", id: "unknown_input_field", targetErr: ErrInvalidYAML, errSubstrs: []string{"field unknown_input_setting not found"}},
{name: "empty cache control type", id: "empty_cache_control_type", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"cache_control", "type is required"}},
{name: "unsupported cache control type", id: "unsupported_cache_control_type", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"cache_control", "unsupported type"}},
{name: "unsupported cache control ttl", id: "unsupported_cache_control_ttl", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"cache_control", "unsupported ttl"}},
{name: "unknown cache control field", id: "unknown_cache_control_field", targetErr: ErrInvalidYAML, errSubstrs: []string{"field unexpected not found"}},
{name: "unidentifiable invalid YAML is unrelated", id: "invalid-yaml", targetErr: ErrPromptDefinitionNotFound},
{name: "missing id is not selected by filename", id: "missing_id", targetErr: ErrPromptDefinitionNotFound},
{name: "no messages", id: "no-messages", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"at least one message is required"}},
{name: "both content and content_file", id: "both-content-and-content-file", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"exactly one"}},
{name: "neither content nor content_file", id: "neither-content-nor-content-file", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"exactly one"}},
{name: "missing content_file", id: "missing-content-file", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"failed to read content_file"}},
{name: "duplicate input names", id: "duplicate-input-names", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"duplicate input name"}},
{name: "invalid validation mode", id: "invalid-validation-mode", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"invalid validation mode"}},
{name: "json_schema without schema_path", id: "json-schema-without-schema-path", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"schema_path"}},
{name: "unknown input field", id: "unknown-input-field", targetErr: ErrInvalidYAML, errSubstrs: []string{"field unknown_input_setting not found"}},
{name: "empty cache control type", id: "empty-cache-control-type", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"cache_control", "type is required"}},
{name: "unsupported cache control type", id: "unsupported-cache-control-type", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"cache_control", "unsupported type"}},
{name: "unsupported cache control ttl", id: "unsupported-cache-control-ttl", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"cache_control", "unsupported ttl"}},
{name: "unknown cache control field", id: "unknown-cache-control-field", targetErr: ErrInvalidYAML, errSubstrs: []string{"field unexpected not found"}},
}
for _, tc := range cases {
@@ -396,7 +424,7 @@ output:
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
repo := NewFSRepository(fstest.MapFS{
fsys := &recordingFS{FS: fstest.MapFS{
"prompts/prompt.yaml": &fstest.MapFile{Data: []byte(`
id: fs-escaped-prompt
version: "1.0.0"
@@ -409,7 +437,8 @@ output:
repair_attempts: 0
`)},
"outside.tmpl": &fstest.MapFile{Data: []byte(`Outside root.`)},
}, "prompts")
}}
repo := NewFSRepository(fsys, "prompts")
_, err := repo.GetPromptDefinition(context.Background(), "fs-escaped-prompt", "")
if !errors.Is(err, ErrInvalidPromptDefinition) {
@@ -418,64 +447,611 @@ output:
if !strings.Contains(err.Error(), tc.wantErr) {
t.Fatalf("expected error to contain %q, got %v", tc.wantErr, err)
}
if fsys.wasOpened("outside.tmpl") {
t.Fatal("rejected content path opened the outside file")
}
})
}
}
func TestFSRepositoryRejectsDuplicatePromptIDs(t *testing.T) {
repo := NewFSRepository(fstest.MapFS{
"one.yaml": &fstest.MapFile{Data: []byte(`
id: duplicate-fs-prompt
version: "1.0.0"
messages:
- role: user
content: First.
output:
format: text
validation_mode: none
repair_attempts: 0
`)},
"nested/two.yaml": &fstest.MapFile{Data: []byte(`
id: duplicate-fs-prompt
version: "1.0.0"
messages:
- role: user
content: Second.
output:
format: text
validation_mode: none
repair_attempts: 0
`)},
}, ".")
type recordingFS struct {
fs.FS
mu sync.Mutex
opened []string
}
_, err := repo.GetPromptDefinition(context.Background(), "duplicate-fs-prompt", "")
if !errors.Is(err, ErrInvalidPromptDefinition) {
t.Fatalf("expected ErrInvalidPromptDefinition, got %v", err)
func TestPromptRepositoryReturnsDefinitionReadFailures(t *testing.T) {
readErr := errors.New("definition read failed")
fsys := &definitionReadFailureFS{
FS: fstest.MapFS{
"prompts/target.yaml": &fstest.MapFile{Data: []byte("unread")},
},
target: "prompts/target.yaml",
err: readErr,
}
if !strings.Contains(err.Error(), "one.yaml") || !strings.Contains(err.Error(), "nested/two.yaml") {
t.Fatalf("expected duplicate paths in error, got %v", err)
repo := NewFSRepository(fsys, "prompts")
definition, err := repo.GetPromptDefinition(context.Background(), "target", "1")
if definition != nil || !errors.Is(err, readErr) {
t.Fatalf("GetPromptDefinition() = (%#v, %v), want nil and definition read error", definition, err)
}
if errors.Is(err, ErrPromptDefinitionNotFound) {
t.Fatalf("definition read error was classified as absence: %v", err)
}
if !strings.Contains(err.Error(), "target.yaml") {
t.Fatalf("definition read error lacks source context: %v", err)
}
}
func TestFSRepositoryRejectsUnknownYAMLFields(t *testing.T) {
repo := NewFSRepository(fstest.MapFS{
"not_named_like_id.yaml": &fstest.MapFile{Data: []byte(`
id: strict-fs-prompt
version: "1.0.0"
unknown: true
type definitionReadFailureFS struct {
fs.FS
target string
err error
}
func (f *definitionReadFailureFS) Open(name string) (fs.File, error) {
if name == f.target {
return nil, f.err
}
return f.FS.Open(name)
}
func (f *recordingFS) Open(name string) (fs.File, error) {
f.mu.Lock()
f.opened = append(f.opened, name)
f.mu.Unlock()
return f.FS.Open(name)
}
func (f *recordingFS) wasOpened(name string) bool {
return f.openCount(name) > 0
}
func (f *recordingFS) openCount(name string) int {
f.mu.Lock()
defer f.mu.Unlock()
count := 0
for _, opened := range f.opened {
if opened == name {
count++
}
}
return count
}
func TestPromptRepositorySelectionUsesYAMLMetadata(t *testing.T) {
const validDefinition = `
id: selected-prompt
version: "1"
messages:
- role: user
content: Invalid.
content: selected
output:
format: text
validation_mode: none
repair_attempts: 0
`)},
}, ".")
`
tests := []struct {
name string
files map[string]string
wantErr error
diagnostics []string
wantContent string
}{
{
name: "same-stem strict error with different YAML ID is unrelated",
files: map[string]string{
"selected-prompt.yaml": `
id: another-prompt
version: "1"
unknown: true
`,
"valid.yaml": validDefinition,
},
},
{
name: "unidentifiable same-stem YAML is unrelated",
files: map[string]string{
"selected-prompt.yaml": "id: [",
"valid.yaml": validDefinition,
},
},
{
name: "same ID invalid different version is unrelated",
files: map[string]string{
"invalid-version.yaml": `
id: selected-prompt
version: "2"
output:
format: text
validation_mode: none
`,
"valid.yaml": validDefinition,
},
},
{
name: "selected content is resolved relative to its definition",
files: map[string]string{
"nested/selected.yaml": `
id: selected-prompt
version: "1"
messages:
- role: user
content_file: ./content/selected.tmpl
output:
format: text
validation_mode: none
`,
"nested/content/selected.tmpl": "selected from file",
},
wantContent: "selected from file",
},
{
name: "duplicate selected definitions are ambiguous",
files: map[string]string{
"selected-a.yaml": validDefinition,
"nested/selected-b.yaml": `
id: selected-prompt
version: "1"
messages:
- role: user
content: duplicate
output:
format: text
validation_mode: none
`,
},
wantErr: ErrInvalidPromptDefinition,
diagnostics: []string{"duplicate prompt definition id", "selected-a.yaml", "nested/selected-b.yaml"},
},
{
name: "selected strict error is authoritative",
files: map[string]string{
"selected-strict.yaml": `
id: selected-prompt
version: "1"
unknown: true
`,
},
wantErr: ErrInvalidYAML,
diagnostics: []string{"selected-strict.yaml"},
},
{
name: "selected semantic error is authoritative",
files: map[string]string{
"selected-invalid.yaml": `
id: selected-prompt
version: "1"
output:
format: text
validation_mode: none
`,
},
wantErr: ErrInvalidPromptDefinition,
diagnostics: []string{"selected-invalid.yaml", "at least one message"},
},
{
name: "selected content error includes definition context",
files: map[string]string{
"selected-missing-content.yaml": `
id: selected-prompt
version: "1"
messages:
- role: user
content_file: missing.tmpl
output:
format: text
validation_mode: none
`,
},
wantErr: ErrInvalidPromptDefinition,
diagnostics: []string{"selected-missing-content.yaml", "failed to read content_file", "missing.tmpl"},
},
}
_, err := repo.GetPromptDefinition(context.Background(), "strict-fs-prompt", "")
if !errors.Is(err, ErrInvalidYAML) {
t.Fatalf("expected ErrInvalidYAML, got %v", err)
for _, source := range promptRepositorySources() {
for _, tc := range tests {
t.Run(source.name+"/"+tc.name, func(t *testing.T) {
repo := source.newRepository(t, tc.files)
got, err := repo.GetPromptDefinition(context.Background(), "selected-prompt", "1")
if tc.wantErr != nil {
if !errors.Is(err, tc.wantErr) {
t.Fatalf("expected %v, got %v", tc.wantErr, err)
}
for _, diagnostic := range tc.diagnostics {
if !strings.Contains(err.Error(), diagnostic) {
t.Fatalf("expected error to contain %q, got %v", diagnostic, err)
}
}
return
}
if err != nil {
t.Fatalf("load selected prompt: %v", err)
}
wantContent := tc.wantContent
if wantContent == "" {
wantContent = "selected"
}
if got.ID != "selected-prompt" || got.Version != "1" || got.Templates[0].Content != wantContent {
t.Fatalf("unexpected selected definition: %+v", got)
}
})
}
}
}
func TestPromptRepositoryHonorsCancellation(t *testing.T) {
for _, source := range promptRepositorySources() {
t.Run(source.name, func(t *testing.T) {
repo := source.newRepository(t, map[string]string{
"definition.yaml": `
id: cancelled-prompt
version: "1"
messages:
- role: user
content: selected
output:
format: text
validation_mode: none
`,
})
ctx, cancel := context.WithCancel(context.Background())
cancel()
_, err := repo.GetPromptDefinition(ctx, "cancelled-prompt", "1")
if !errors.Is(err, context.Canceled) {
t.Fatalf("expected context cancellation, got %v", err)
}
})
}
}
func TestPromptRepositoryRequiresOneYAMLDocument(t *testing.T) {
const definition = `
id: one-document
version: "1"
messages:
- role: user
content: selected
output:
format: text
validation_mode: none
`
tests := []struct {
name string
suffix string
wantErr bool
}{
{name: "comments and trailing whitespace", suffix: "\n# trailing comment\n\n"},
{name: "second populated document", suffix: "\n---\nid: another\n", wantErr: true},
{name: "second empty document", suffix: "\n---\n", wantErr: true},
{name: "malformed trailing YAML", suffix: "\n---\n[", wantErr: true},
}
for _, source := range promptRepositorySources() {
for _, tc := range tests {
t.Run(source.name+"/"+tc.name, func(t *testing.T) {
repo := source.newRepository(t, map[string]string{"definition.yaml": definition + tc.suffix})
_, err := repo.GetPromptDefinition(context.Background(), "one-document", "1")
if tc.wantErr {
if !errors.Is(err, ErrInvalidYAML) {
t.Fatalf("expected ErrInvalidYAML, got %v", err)
}
if !strings.Contains(err.Error(), "definition.yaml") {
t.Fatalf("expected source path in error, got %v", err)
}
return
}
if err != nil {
t.Fatalf("load one document: %v", err)
}
})
}
}
}
func TestPromptRepositoryReadsOnlySelectedContent(t *testing.T) {
fsys := &recordingFS{FS: fstest.MapFS{
"target.yaml": &fstest.MapFile{Data: []byte(`
id: target
version: "1"
messages:
- role: user
content_file: target.tmpl
output:
format: text
validation_mode: none
`)},
"target.tmpl": &fstest.MapFile{Data: []byte("selected")},
"unrelated.yaml": &fstest.MapFile{Data: []byte(`
id: unrelated
version: "1"
messages:
- role: user
content_file: unrelated.tmpl
output:
format: text
validation_mode: none
`)},
"unrelated.tmpl": &fstest.MapFile{Data: []byte("unrelated")},
"other-version.yaml": &fstest.MapFile{Data: []byte(`
id: target
version: "2"
messages:
- role: user
content_file: other-version.tmpl
output:
format: text
validation_mode: none
`)},
"other-version.tmpl": &fstest.MapFile{Data: []byte("other version")},
}}
repo := NewFSRepository(fsys, ".")
for lookup := 1; lookup <= 2; lookup++ {
got, err := repo.GetPromptDefinition(context.Background(), "target", "1")
if err != nil {
t.Fatalf("lookup %d: %v", lookup, err)
}
if got.Templates[0].Content != "selected" {
t.Fatalf("lookup %d content = %q", lookup, got.Templates[0].Content)
}
if count := fsys.openCount("target.tmpl"); count != lookup {
t.Fatalf("selected content opens after lookup %d = %d, want %d", lookup, count, lookup)
}
for _, name := range []string{"unrelated.tmpl", "other-version.tmpl"} {
if count := fsys.openCount(name); count != 0 {
t.Fatalf("unselected content %q opened %d times", name, count)
}
}
for _, name := range []string{"target.yaml", "unrelated.yaml", "other-version.yaml"} {
if count := fsys.openCount(name); count != lookup {
t.Fatalf("metadata %q opens after lookup %d = %d, want %d", name, lookup, count, lookup)
}
}
}
}
func TestPromptDefinitionNormalizationRules(t *testing.T) {
tests := []struct {
name string
definition string
wantErr bool
wantDiagnostic string
wantSchemaPath string
}{
{
name: "missing version",
definition: `
id: normalization-rule
messages:
- role: user
content: test
output:
format: text
validation_mode: none
`,
wantErr: true,
wantDiagnostic: "version",
},
{
name: "blank input name",
definition: `
id: normalization-rule
version: "1"
inputs:
- name: " "
messages:
- role: user
content: test
output:
format: text
validation_mode: none
`,
wantErr: true,
wantDiagnostic: "input 0",
},
{
name: "blank message role",
definition: `
id: normalization-rule
version: "1"
messages:
- role: " "
content: test
output:
format: text
validation_mode: none
`,
wantErr: true,
wantDiagnostic: "role",
},
{
name: "invalid output format",
definition: `
id: normalization-rule
version: "1"
messages:
- role: user
content: test
output:
format: binary
validation_mode: none
`,
wantErr: true,
wantDiagnostic: "format",
},
{
name: "negative repair attempts",
definition: `
id: normalization-rule
version: "1"
messages:
- role: user
content: test
output:
format: text
validation_mode: none
repair_attempts: -1
`,
wantErr: true,
wantDiagnostic: "repair_attempts",
},
{
name: "explicit blank default profile",
definition: `
id: normalization-rule
version: "1"
default_profile: " "
messages:
- role: user
content: test
output:
format: text
validation_mode: none
`,
wantErr: true,
wantDiagnostic: "default_profile",
},
{
name: "schema path normalization",
definition: `
id: normalization-rule
version: "1"
messages:
- role: user
content: test
output:
format: json
validation_mode: json_schema
schema_path: ' schema.json '
`,
wantSchemaPath: "schema.json",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
repo := NewFSRepository(fstest.MapFS{
"definition.yaml": &fstest.MapFile{Data: []byte(tt.definition)},
}, ".")
got, err := repo.GetPromptDefinition(context.Background(), "normalization-rule", "")
if tt.wantErr {
if !errors.Is(err, ErrInvalidPromptDefinition) {
t.Fatalf("expected ErrInvalidPromptDefinition, got %v", err)
}
if !strings.Contains(err.Error(), tt.wantDiagnostic) {
t.Fatalf("expected error containing %q, got %v", tt.wantDiagnostic, err)
}
return
}
if err != nil {
t.Fatalf("load prompt definition: %v", err)
}
if got.Validation.SchemaPath != tt.wantSchemaPath {
t.Fatalf("schema path = %q, want %q", got.Validation.SchemaPath, tt.wantSchemaPath)
}
})
}
}
type promptRepositorySource struct {
name string
newRepository func(t *testing.T, files map[string]string) Repository
}
func promptRepositorySources() []promptRepositorySource {
return []promptRepositorySource{
{
name: "operating system",
newRepository: func(t *testing.T, files map[string]string) Repository {
t.Helper()
root := t.TempDir()
for name, content := range files {
filePath := filepath.Join(root, filepath.FromSlash(name))
if err := os.MkdirAll(filepath.Dir(filePath), 0o755); err != nil {
t.Fatalf("create prompt directory: %v", err)
}
writePromptTestFile(t, filePath, content)
}
return NewFilesystemRepository(root)
},
},
{
name: "filesystem",
newRepository: func(t *testing.T, files map[string]string) Repository {
t.Helper()
fsys := make(fstest.MapFS, len(files))
for name, content := range files {
fsys[name] = &fstest.MapFile{Data: []byte(strings.TrimLeft(content, "\n"))}
}
return NewFSRepository(fsys, ".")
},
},
}
}
func BenchmarkPromptRepositoryLookup(b *testing.B) {
for _, size := range []int{10, 1000} {
b.Run(fmt.Sprintf("catalog-%d", size), func(b *testing.B) {
files := fstest.MapFS{
"target.yaml": &fstest.MapFile{Data: []byte(`
id: target
version: "1"
messages:
- role: user
content_file: target.tmpl
output:
format: text
validation_mode: none
`)},
"target.tmpl": &fstest.MapFile{Data: []byte("selected")},
}
metadataNames := []string{"target.yaml"}
contentNames := make([]string, 0, size-1)
for i := 1; i < size; i++ {
definitionName := fmt.Sprintf("prompt-%04d.yaml", i)
contentName := fmt.Sprintf("prompt-%04d.tmpl", i)
files[definitionName] = &fstest.MapFile{Data: []byte(fmt.Sprintf(`
id: prompt-%04d
version: "1"
messages:
- role: user
content_file: %s
output:
format: text
validation_mode: none
`, i, contentName))}
files[contentName] = &fstest.MapFile{Data: []byte("unrelated")}
metadataNames = append(metadataNames, definitionName)
contentNames = append(contentNames, contentName)
}
fsys := &recordingFS{FS: files}
repo := NewFSRepository(fsys, ".")
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if _, err := repo.GetPromptDefinition(context.Background(), "target", "1"); err != nil {
b.Fatal(err)
}
}
b.StopTimer()
if got := fsys.openCount("target.tmpl"); got != b.N {
b.Fatalf("selected content opens = %d, want %d", got, b.N)
}
for _, name := range contentNames {
if got := fsys.openCount(name); got != 0 {
b.Fatalf("unrelated content %q opened %d times", name, got)
}
}
for _, name := range metadataNames {
if got := fsys.openCount(name); got != b.N {
b.Fatalf("metadata %q opens = %d, want %d", name, got, b.N)
}
}
})
}
}

View File

@@ -0,0 +1,63 @@
package promptdef
import (
"context"
"errors"
"io/fs"
"os"
"gitea.maximumdirect.net/eric/promptkit/internal/filecatalog"
)
type promptDefinitionSource interface {
contentSourceRoot
findYAMLFiles(context.Context) ([]string, error)
readDefinition(string) ([]byte, error)
displayPath(string) string
}
type osPromptSource struct {
root string
contentRoot osContentSourceRoot
}
func (s osPromptSource) findYAMLFiles(ctx context.Context) ([]string, error) {
return filecatalog.FindYAMLFiles(ctx, s.root)
}
func (s osPromptSource) readDefinition(name string) ([]byte, error) {
return os.ReadFile(name)
}
func (s osPromptSource) displayPath(name string) string {
return filecatalog.RelativePath(s.root, name)
}
func (s osPromptSource) readContentFile(sourcePath string, contentFile string) (string, string, error) {
return s.contentRoot.readContentFile(sourcePath, contentFile)
}
type fsPromptSource struct {
fsys fs.FS
root string
contentRoot contentSourceRoot
}
func (s fsPromptSource) findYAMLFiles(ctx context.Context) ([]string, error) {
if s.fsys == nil {
return nil, errors.New("filesystem is nil")
}
return filecatalog.FindFSYAMLFiles(ctx, s.fsys, s.root)
}
func (s fsPromptSource) readDefinition(name string) ([]byte, error) {
return fs.ReadFile(s.fsys, name)
}
func (s fsPromptSource) displayPath(name string) string {
return filecatalog.DisplayPath(s.root, name)
}
func (s fsPromptSource) readContentFile(sourcePath string, contentFile string) (string, string, error) {
return s.contentRoot.readContentFile(sourcePath, contentFile)
}

View File

@@ -0,0 +1,83 @@
package usecase
import (
"context"
"errors"
"math"
"testing"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
)
func TestRunnerPrepareExecutionRejectsInvalidExecutionSettings(t *testing.T) {
runner := NewRunner(
&fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}},
nil,
defaultArtifactReader(),
defaultRenderer(),
&fakeLLM{forbid: true},
nil,
nil,
)
_, err := runner.PrepareExecution(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
Execution: &domain.ExecutionTargetOverride{TopP: float64Ptr(math.Inf(-1))},
})
if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected ErrInvalidRequest, got %v", err)
}
}
func TestRunnerPrepareExecutionValidatesAndNormalizesRequestEndpoints(t *testing.T) {
newRunner := func() *Runner {
return NewRunner(
&fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}},
nil,
defaultArtifactReader(),
defaultRenderer(),
&fakeLLM{forbid: true},
nil,
nil,
)
}
invalidEndpoints := []string{
"/v1",
"https:///v1",
"ftp://provider.example/v1",
"https://user@provider.example/v1",
"https://provider.example/v1?mode=chat",
"https://provider.example/v1#chat",
}
for _, endpoint := range invalidEndpoints {
t.Run(endpoint, func(t *testing.T) {
_, err := newRunner().PrepareExecution(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
Execution: &domain.ExecutionTargetOverride{Endpoint: endpoint},
})
if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected ErrInvalidRequest, got %v", err)
}
})
}
prepared, err := newRunner().PrepareExecution(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
Execution: &domain.ExecutionTargetOverride{Endpoint: " https://provider.example/nested/v1 "},
})
if err != nil {
t.Fatalf("prepare normalized endpoint: %v", err)
}
if got := prepared.Details().EffectiveModelParams.Endpoint; got != "https://provider.example/nested/v1" {
t.Fatalf("effective endpoint = %q", got)
}
}

View File

@@ -0,0 +1,19 @@
package usecase
import "gitea.maximumdirect.net/eric/promptkit/internal/domain"
func newGenerationRequest(
prompt domain.RenderedPrompt,
sessionID string,
target domain.ExecutionTarget,
targetPresence domain.ExecutionTargetPresence,
structuredOutput *domain.StructuredOutputSpec,
) domain.GenerateRequest {
prompt.SessionID = sessionID
return domain.GenerateRequest{
Prompt: prompt,
Target: target,
TargetPresence: targetPresence,
StructuredOutput: structuredOutput,
}
}

View File

@@ -0,0 +1,185 @@
package usecase
import (
"context"
"errors"
"testing"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
)
type outputContractTestCollaborators struct {
artifacts *fakeArtifactReader
renderer *fakeRenderer
llm *fakeLLM
validator *recordingValidationPreparer
admitter *fakeRunAdmitter
}
func newOutputContractTestRunner() (*Runner, outputContractTestCollaborators) {
collaborators := outputContractTestCollaborators{
artifacts: defaultArtifactReader(),
renderer: defaultRenderer(),
llm: &fakeLLM{forbid: true},
validator: &recordingValidationPreparer{plan: &recordingPreparedValidation{}},
admitter: &fakeRunAdmitter{},
}
return NewRunner(
&fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}},
nil,
collaborators.artifacts,
collaborators.renderer,
collaborators.llm,
collaborators.validator,
collaborators.admitter,
), collaborators
}
func TestRunnerPreparationNormalizesOutputContractConsistently(t *testing.T) {
tests := []struct {
name string
override domain.OutputContract
want domain.OutputContract
}{
{
name: "empty replacement format defaults to text",
override: domain.OutputContract{ValidationMode: domain.ValidationNone},
want: domain.OutputContract{Format: domain.FormatText, ValidationMode: domain.ValidationNone},
},
{
name: "markdown basic replacement",
override: domain.OutputContract{Format: domain.FormatMarkdown, ValidationMode: domain.ValidationBasic},
want: domain.OutputContract{Format: domain.FormatMarkdown, ValidationMode: domain.ValidationBasic},
},
{
name: "json replacement preserves non-schema fields",
override: domain.OutputContract{
Format: domain.FormatJSON,
ValidationMode: domain.ValidationJSON,
SchemaPath: "ignored.json",
RepairAttempts: 2,
},
want: domain.OutputContract{
Format: domain.FormatJSON,
ValidationMode: domain.ValidationJSON,
SchemaPath: "ignored.json",
RepairAttempts: 2,
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
runner, _ := newOutputContractTestRunner()
req := domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
Validation: &tt.override,
}
prepared, err := runner.Prepare(context.Background(), req)
if err != nil {
t.Fatalf("prepare: %v", err)
}
preparedExecution, err := runner.PrepareExecution(context.Background(), req)
if err != nil {
t.Fatalf("prepare execution: %v", err)
}
details := preparedExecution.Details()
if details == nil {
t.Fatal("prepared execution returned nil details")
}
if prepared.OutputContract != tt.want || details.OutputContract != tt.want {
t.Fatalf("output contracts = (%+v, %+v), want %+v", prepared.OutputContract, details.OutputContract, tt.want)
}
})
}
}
func TestRunnerPreparationRejectsInvalidOutputContractsBeforeCompletion(t *testing.T) {
tests := []struct {
name string
override domain.OutputContract
}{
{name: "unsupported format", override: domain.OutputContract{Format: "binary", ValidationMode: domain.ValidationNone}},
{name: "empty validation mode", override: domain.OutputContract{Format: domain.FormatText}},
{name: "negative repair attempts", override: domain.OutputContract{Format: domain.FormatText, ValidationMode: domain.ValidationNone, RepairAttempts: -1}},
{name: "json schema without path", override: domain.OutputContract{Format: domain.FormatJSON, ValidationMode: domain.ValidationJSONSchema}},
}
for _, tt := range tests {
for _, operation := range []string{"Prepare", "PrepareExecution"} {
t.Run(tt.name+"/"+operation, func(t *testing.T) {
runner, collaborators := newOutputContractTestRunner()
req := domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
Validation: &tt.override,
}
var err error
switch operation {
case "Prepare":
var prepared *domain.PreparedRun
prepared, err = runner.Prepare(context.Background(), req)
if prepared != nil {
t.Fatalf("expected no partial prepared run, got %+v", prepared)
}
case "PrepareExecution":
var prepared *PreparedExecution
prepared, err = runner.PrepareExecution(context.Background(), req)
if prepared != nil {
t.Fatalf("expected no partial prepared execution, got %+v", prepared)
}
default:
t.Fatalf("unknown operation %q", operation)
}
if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected ErrInvalidRequest, got %v", err)
}
assertOutputContractCompletionSkipped(t, collaborators)
})
}
}
}
func TestRunnerRunRejectsInvalidOutputContractBeforeAdmission(t *testing.T) {
runner, collaborators := newOutputContractTestRunner()
result, err := runner.Run(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
Validation: &domain.OutputContract{
Format: domain.OutputFormat("binary"),
ValidationMode: domain.ValidationNone,
},
})
if result != nil {
t.Fatalf("expected no partial result, got %+v", result)
}
if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected ErrInvalidRequest, got %v", err)
}
assertOutputContractCompletionSkipped(t, collaborators)
}
func assertOutputContractCompletionSkipped(t *testing.T, collaborators outputContractTestCollaborators) {
t.Helper()
if collaborators.artifacts.calls != 0 || collaborators.renderer.calls != 0 ||
collaborators.validator.prepareCalls != 0 || collaborators.validator.directValidateCalls != 0 ||
len(collaborators.admitter.backendIDs) != 0 || collaborators.llm.calls != 0 {
t.Fatalf(
"invalid output contract reached downstream work: artifacts=%d renderer=%d prepare_validation=%d validation=%d admissions=%d generation=%d",
collaborators.artifacts.calls,
collaborators.renderer.calls,
collaborators.validator.prepareCalls,
collaborators.validator.directValidateCalls,
len(collaborators.admitter.backendIDs),
collaborators.llm.calls,
)
}
}

View File

@@ -42,25 +42,12 @@ func (r *Runner) PrepareExecution(ctx context.Context, req domain.RunRequest) (*
return nil, err
}
validationPlan, err := r.prepareValidation(ctx, state.effectiveContract)
operation, err := r.completePreparation(ctx, req, state)
if err != nil {
return nil, err
}
structuredOutput, err := r.structuredOutputFromValidationPlan(
state.definition,
state.effectiveContract,
validationPlan,
)
if err != nil {
return nil, err
}
prepared, err := r.completePreparationWithStructuredOutput(ctx, req, state, structuredOutput)
if err != nil {
return nil, err
}
executionSnapshot, err := clonePreparedRun(prepared)
executionSnapshot, err := clonePreparedRun(operation.run)
if err != nil {
return nil, fmt.Errorf("%w: failed to copy prepared execution: %v", ErrInvalidRequest, err)
}
@@ -76,52 +63,12 @@ func (r *Runner) PrepareExecution(ctx context.Context, req domain.RunRequest) (*
details: details,
payload: &preparedExecutionPayload{
prepared: executionSnapshot,
validation: validationPlan,
validation: operation.validation,
directKey: state.effectiveModel.APIKey,
},
}, nil
}
func (r *Runner) prepareValidation(
ctx context.Context,
contract domain.OutputContract,
) (validate.PreparedValidation, error) {
if r.validator == nil {
return noOpPreparedValidation{contract: contract}, nil
}
preparer, ok := r.validator.(validate.ValidationPreparer)
if !ok {
return nil, fmt.Errorf("%w: validator does not support prepared validation", ErrValidation)
}
plan, err := preparer.PrepareValidation(ctx, contract)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrValidation, err)
}
if plan == nil {
return nil, fmt.Errorf("%w: validator returned nil prepared validation", ErrValidation)
}
return plan, nil
}
func (r *Runner) structuredOutputFromValidationPlan(
def *domain.PromptDefinition,
contract domain.OutputContract,
plan validate.PreparedValidation,
) (*domain.StructuredOutputSpec, error) {
if contract.ValidationMode != domain.ValidationJSONSchema {
return nil, nil
}
schemaDocument := plan.SchemaDocument()
if schemaDocument == nil {
if r.validator == nil {
return nil, nil
}
return nil, fmt.Errorf("%w: prepared json_schema validation has no schema document", ErrValidation)
}
return structuredOutputSpec(def, schemaDocument), nil
}
// Details returns a fresh credential-redacted copy of the prepared run.
func (p *PreparedExecution) Details() *domain.PreparedRun {
if p == nil {

View File

@@ -52,6 +52,12 @@ type recordingValidationPreparer struct {
directValidateCalls int
}
type validationOnly struct{}
func (validationOnly) Validate(context.Context, *domain.Artifact, domain.OutputContract) (domain.ValidationResult, error) {
return domain.ValidationResult{}, nil
}
func (v *recordingValidationPreparer) Validate(
context.Context,
*domain.Artifact,
@@ -164,6 +170,77 @@ func TestRunnerPrepareExecutionCompletesWithoutAdmissionOrGeneration(t *testing.
}
}
func TestRunnerPreparationRejectsExcessivelyDeepPreparedSchema(t *testing.T) {
operations := []struct {
name string
run func(*Runner, domain.RunRequest) error
}{
{
name: "Prepare",
run: func(runner *Runner, request domain.RunRequest) error {
_, err := runner.Prepare(context.Background(), request)
return err
},
},
{
name: "Run",
run: func(runner *Runner, request domain.RunRequest) error {
_, err := runner.Run(context.Background(), request)
return err
},
},
{
name: "PrepareExecution",
run: func(runner *Runner, request domain.RunRequest) error {
_, err := runner.PrepareExecution(context.Background(), request)
return err
},
},
}
for _, operation := range operations {
t.Run(operation.name, func(t *testing.T) {
def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 0)
def.Validation.SchemaPath = "schema.json"
llmClient := &fakeLLM{forbid: true}
validator := &recordingValidationPreparer{
plan: &recordingPreparedValidation{schemaDocument: excessivelyDeepPreparedJSONValue()},
}
runner := NewRunner(
&fakePromptRepo{def: def},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}},
nil,
defaultArtifactReader(),
defaultRenderer(),
llmClient,
validator,
nil,
)
err := operation.run(runner, domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
})
if !errors.Is(err, ErrValidation) {
t.Fatalf("expected ErrValidation, got %v", err)
}
if llmClient.calls != 0 {
t.Fatalf("invalid prepared schema reached generation: %d calls", llmClient.calls)
}
})
}
}
func excessivelyDeepPreparedJSONValue() any {
const clearlyUnsafeContainerDepth = 1_000
var value any = true
for level := 0; level < clearlyUnsafeContainerDepth; level++ {
value = map[string]any{"child": value}
}
return value
}
func TestRunnerRunPreparedRechecksEnvironmentCredentialBeforeAdmission(t *testing.T) {
const environmentName = "PROMPTKIT_PREPARED_EXECUTION_TEST_KEY"
t.Setenv(environmentName, "available-during-preparation")
@@ -271,7 +348,13 @@ func TestRunnerRunPreparedUsesFrozenValidationForInitialAndRepairOutputs(t *test
},
}
repairer := &fakeRepairer{
responses: []*domain.GenerateResponse{{Content: `{"repaired":true}`}},
responses: []*domain.GenerateResponse{{
Content: `{"repaired":true}`,
Usage: domain.TokenUsage{
PromptTokens: 2, CompletionTokens: 3, TotalTokens: 5,
CachedTokens: 7, CacheWriteTokens: 11,
},
}},
}
admitter := &fakeRunAdmitter{}
reader := defaultArtifactReader()
@@ -282,7 +365,13 @@ func TestRunnerRunPreparedUsesFrozenValidationForInitialAndRepairOutputs(t *test
nil,
reader,
renderer,
&fakeLLM{resp: &domain.GenerateResponse{Content: `{"broken":true}`}},
&fakeLLM{resp: &domain.GenerateResponse{
Content: `{"broken":true}`,
Usage: domain.TokenUsage{
PromptTokens: 13, CompletionTokens: 17, TotalTokens: 19,
CachedTokens: 23, CacheWriteTokens: 29,
},
}},
validator,
repairer,
admitter,
@@ -309,6 +398,13 @@ func TestRunnerRunPreparedUsesFrozenValidationForInitialAndRepairOutputs(t *test
if result.Validation.Status != domain.ValidationPassed || result.Validation.RepairAttempts != 1 {
t.Fatalf("unexpected repaired validation result: %+v", result.Validation)
}
wantUsage := domain.TokenUsage{
PromptTokens: 15, CompletionTokens: 20, TotalTokens: 24,
CachedTokens: 30, CacheWriteTokens: 40,
}
if result.Usage != wantUsage {
t.Fatalf("prepared cumulative usage = %+v, want %+v", result.Usage, wantUsage)
}
if admitter.releaseCalls != 1 {
t.Fatalf("admission releases=%d, want 1", admitter.releaseCalls)
}
@@ -492,7 +588,7 @@ func TestRunnerPrepareExecutionRequiresValidationPreparer(t *testing.T) {
reader,
defaultRenderer(),
&fakeLLM{forbid: true},
&fakeValidator{},
validationOnly{},
nil,
)

View File

@@ -57,14 +57,19 @@ func (r *Runner) resolveProfileSelection(
}, nil
}
func validateResolvedExecutionTarget(target domain.ExecutionTarget) error {
if strings.TrimSpace(target.Endpoint) == "" {
return errors.New("execution endpoint is required")
func normalizeResolvedExecutionTarget(target domain.ExecutionTarget) (domain.ExecutionTarget, error) {
endpoint, err := domain.NormalizeOpenAICompatibleBaseEndpoint(target.Endpoint)
if err != nil {
return domain.ExecutionTarget{}, fmt.Errorf("execution endpoint: %w", err)
}
target.Endpoint = endpoint
if strings.TrimSpace(target.Model) == "" {
return errors.New("execution model is required")
return domain.ExecutionTarget{}, errors.New("execution model is required")
}
return nil
if err := domain.ValidateExecutionTargetSettings(target); err != nil {
return domain.ExecutionTarget{}, err
}
return target, nil
}
// InspectProfile resolves one explicit profile without prompt or execution work.
@@ -86,13 +91,11 @@ func (r *Runner) InspectProfile(
if err != nil {
return nil, err
}
target, _, err := resolveExecutionTarget(selection.backend, selection.profile, nil)
target, _ := resolveExecutionTarget(selection.backend, selection.profile, nil)
target, err = normalizeResolvedExecutionTarget(target)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrProfileLoad, err)
}
if err := validateResolvedExecutionTarget(target); err != nil {
return nil, fmt.Errorf("%w: %w", ErrProfileLoad, err)
}
target.APIKey = ""
return &domain.ProfileInspection{

View File

@@ -19,6 +19,7 @@ type RepairRequest struct {
ValidationErrors []string
SessionID string
Target domain.ExecutionTarget
TargetPresence domain.ExecutionTargetPresence
StructuredOutput *domain.StructuredOutputSpec
Attempt int
MaxAttempts int
@@ -44,7 +45,6 @@ func (r *defaultOutputRepairer) Repair(ctx context.Context, req RepairRequest) (
}
prompt := domain.RenderedPrompt{
SessionID: req.SessionID,
Messages: []domain.RenderedMessage{
{
Role: "system",
@@ -64,11 +64,13 @@ func (r *defaultOutputRepairer) Repair(ctx context.Context, req RepairRequest) (
},
}
resp, err := r.llm.Generate(ctx, domain.GenerateRequest{
Prompt: prompt,
Target: req.Target,
StructuredOutput: req.StructuredOutput,
})
resp, err := r.llm.Generate(ctx, newGenerationRequest(
prompt,
req.SessionID,
req.Target,
req.TargetPresence,
req.StructuredOutput,
))
if err != nil {
return nil, err
}

View File

@@ -17,6 +17,7 @@ import (
"gitea.maximumdirect.net/eric/promptkit/internal/capacity"
"gitea.maximumdirect.net/eric/promptkit/internal/defaults"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
"gitea.maximumdirect.net/eric/promptkit/internal/jsonvalue"
"gitea.maximumdirect.net/eric/promptkit/internal/llm"
"gitea.maximumdirect.net/eric/promptkit/internal/profile"
"gitea.maximumdirect.net/eric/promptkit/internal/prompt"
@@ -71,6 +72,11 @@ type preparationState struct {
start time.Time
}
type preparedOperation struct {
run *domain.PreparedRun
validation validate.PreparedValidation
}
func NewRunner(
promptDefs promptdef.Repository,
profiles profile.Repository,
@@ -137,18 +143,20 @@ func (r *Runner) Run(ctx context.Context, req domain.RunRequest) (*domain.RunRes
}
defer release()
prepared, err := r.completePreparation(ctx, req, state)
operation, err := r.completePreparation(ctx, req, state)
if err != nil {
return nil, err
}
directAPIKey := state.effectiveModel.APIKey
return r.executePreparedRun(ctx, prepared, directAPIKey, runID, start, func(
return r.executePreparedRun(ctx, operation.run, directAPIKey, runID, start, func(
ctx context.Context,
artifact *domain.Artifact,
attemptsUsed int,
) (domain.ValidationResult, error) {
return r.validateOutput(ctx, artifact, prepared.OutputContract, attemptsUsed)
result, err := operation.validation.Validate(ctx, artifact)
result.RepairAttempts = attemptsUsed
return result, err
})
}
@@ -168,18 +176,20 @@ func (r *Runner) executePreparedRun(
) (*domain.RunResult, error) {
executionTarget := prepared.EffectiveModelParams
executionTarget.APIKey = directAPIKey
genResp, err := r.llm.Generate(ctx, domain.GenerateRequest{
Prompt: domain.RenderedPrompt{SessionID: prepared.SessionID, Messages: prepared.Messages},
Target: executionTarget,
TargetPresence: prepared.TargetPresence,
StructuredOutput: prepared.StructuredOutput,
})
genResp, err := r.llm.Generate(ctx, newGenerationRequest(
domain.RenderedPrompt{Messages: prepared.Messages},
prepared.SessionID,
executionTarget,
prepared.TargetPresence,
prepared.StructuredOutput,
))
if err != nil {
if errors.Is(err, llm.ErrInvalidRequest) {
return nil, fmt.Errorf("%w: %w", ErrInvalidRequest, err)
}
return nil, fmt.Errorf("%w: %w", ErrLLMGenerate, err)
}
usage := genResp.Usage
outputArtifact := buildOutputArtifact(genResp.Content, prepared.OutputContract.Format)
validationResult, err := validateArtifact(ctx, &outputArtifact, 0)
@@ -197,6 +207,7 @@ func (r *Runner) executePreparedRun(
ValidationErrors: validationResult.Errors,
SessionID: prepared.SessionID,
Target: executionTarget,
TargetPresence: prepared.TargetPresence,
StructuredOutput: prepared.StructuredOutput,
Attempt: attemptsUsed,
MaxAttempts: prepared.OutputContract.RepairAttempts,
@@ -210,6 +221,7 @@ func (r *Runner) executePreparedRun(
}
genResp = repairResp
usage = addTokenUsage(usage, repairResp.Usage)
outputArtifact = buildOutputArtifact(genResp.Content, prepared.OutputContract.Format)
validationResult, err = validateArtifact(ctx, &outputArtifact, attemptsUsed)
@@ -238,19 +250,33 @@ func (r *Runner) executePreparedRun(
Endpoint: prepared.EffectiveModelParams.Endpoint,
EffectiveModelParams: executionTarget,
InputHashes: prepared.InputHashes,
Usage: genResp.Usage,
Usage: usage,
StartTime: start,
EndTime: end,
Duration: end.Sub(start),
}, nil
}
func addTokenUsage(total, next domain.TokenUsage) domain.TokenUsage {
return domain.TokenUsage{
PromptTokens: total.PromptTokens + next.PromptTokens,
CompletionTokens: total.CompletionTokens + next.CompletionTokens,
TotalTokens: total.TotalTokens + next.TotalTokens,
CachedTokens: total.CachedTokens + next.CachedTokens,
CacheWriteTokens: total.CacheWriteTokens + next.CacheWriteTokens,
}
}
func (r *Runner) Prepare(ctx context.Context, req domain.RunRequest) (*domain.PreparedRun, error) {
state, err := r.resolvePreparation(ctx, req, time.Now().UTC())
if err != nil {
return nil, err
}
return r.completePreparation(ctx, req, state)
operation, err := r.completePreparation(ctx, req, state)
if err != nil {
return nil, err
}
return operation.run, nil
}
func (r *Runner) resolvePreparation(
@@ -272,6 +298,10 @@ func (r *Runner) resolvePreparation(
}
def := promptSelection.definition
promptDefinitionHash := promptSelection.hash
effectiveContract, err := resolveOutputContract(def, req.Validation)
if err != nil {
return nil, fmt.Errorf("%w: output contract: %v", ErrInvalidRequest, err)
}
selectedProfileID := strings.TrimSpace(req.ProfileID)
if selectedProfileID == "" {
@@ -286,19 +316,16 @@ func (r *Runner) resolvePreparation(
return nil, err
}
effectiveModel, targetPresence, err := resolveExecutionTarget(selection.backend, selection.profile, req.Execution)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrInvalidRequest, err)
}
effectiveModel, targetPresence := resolveExecutionTarget(selection.backend, selection.profile, req.Execution)
effectiveModel.APIKey = req.APIKey
if err := validateResolvedExecutionTarget(effectiveModel); err != nil {
effectiveModel, err = normalizeResolvedExecutionTarget(effectiveModel)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrInvalidRequest, err)
}
if err := validateAPIKey(effectiveModel.APIKeyEnv, effectiveModel.APIKey, effectiveModel.APIKeyRequired); err != nil {
return nil, fmt.Errorf("%w: %w", ErrInvalidRequest, err)
}
effectiveContract := resolveOutputContract(def, req.Validation)
return &preparationState{
definition: def,
directSessionID: directSessionID,
@@ -315,16 +342,68 @@ func (r *Runner) completePreparation(
ctx context.Context,
req domain.RunRequest,
state *preparationState,
) (*domain.PreparedRun, error) {
structuredOutput, err := r.resolveStructuredOutput(
ctx,
) (*preparedOperation, error) {
validationPlan, err := r.prepareValidation(ctx, state.effectiveContract)
if err != nil {
return nil, err
}
structuredOutput, err := r.structuredOutputFromValidationPlan(
state.definition,
state.effectiveContract,
validationPlan,
)
if err != nil {
return nil, err
}
return r.completePreparationWithStructuredOutput(ctx, req, state, structuredOutput)
prepared, err := r.completePreparationWithStructuredOutput(ctx, req, state, structuredOutput)
if err != nil {
return nil, err
}
return &preparedOperation{run: prepared, validation: validationPlan}, nil
}
func (r *Runner) prepareValidation(
ctx context.Context,
contract domain.OutputContract,
) (validate.PreparedValidation, error) {
if r.validator == nil {
return noOpPreparedValidation{contract: contract}, nil
}
preparer, ok := r.validator.(validate.ValidationPreparer)
if !ok {
return nil, fmt.Errorf("%w: validator does not support prepared validation", ErrValidation)
}
plan, err := preparer.PrepareValidation(ctx, contract)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrValidation, err)
}
if plan == nil {
return nil, fmt.Errorf("%w: validator returned nil prepared validation", ErrValidation)
}
return plan, nil
}
func (r *Runner) structuredOutputFromValidationPlan(
def *domain.PromptDefinition,
contract domain.OutputContract,
plan validate.PreparedValidation,
) (*domain.StructuredOutputSpec, error) {
if contract.ValidationMode != domain.ValidationJSONSchema {
return nil, nil
}
schemaDocument := plan.SchemaDocument()
if schemaDocument == nil {
if r.validator == nil {
return nil, nil
}
return nil, fmt.Errorf("%w: prepared json_schema validation has no schema document", ErrValidation)
}
schemaDocument, err := jsonvalue.Copy(schemaDocument)
if err != nil {
return nil, fmt.Errorf("%w: invalid prepared json_schema schema document: %v", ErrValidation, err)
}
return structuredOutputSpec(def, schemaDocument), nil
}
func (r *Runner) completePreparationWithStructuredOutput(
@@ -398,24 +477,6 @@ func (r *Runner) admitRun(ctx context.Context, backendID string) (func(), error)
return release, nil
}
func (r *Runner) resolveStructuredOutput(ctx context.Context, def *domain.PromptDefinition, contract domain.OutputContract) (*domain.StructuredOutputSpec, error) {
if contract.ValidationMode != domain.ValidationJSONSchema {
return nil, nil
}
loader, ok := r.validator.(validate.SchemaDocumentLoader)
if !ok || loader == nil {
return nil, fmt.Errorf("%w: json_schema output requires schema document loader", ErrValidation)
}
schemaDoc, err := loader.LoadSchemaDocument(ctx, contract.SchemaPath)
if err != nil {
return nil, fmt.Errorf("%w: failed to load json schema for structured output: %v", ErrValidation, err)
}
return structuredOutputSpec(def, schemaDoc), nil
}
func structuredOutputSpec(def *domain.PromptDefinition, schemaDocument any) *domain.StructuredOutputSpec {
return &domain.StructuredOutputSpec{
Type: domain.StructuredOutputJSONSchema,
@@ -453,25 +514,6 @@ func deriveStructuredSchemaName(promptID string, promptVersion string) string {
return name
}
func (r *Runner) validateOutput(ctx context.Context, artifact *domain.Artifact, contract domain.OutputContract, attemptsUsed int) (domain.ValidationResult, error) {
if r.validator == nil || contract.ValidationMode == domain.ValidationNone {
return domain.ValidationResult{
Status: domain.ValidationSkipped,
Mode: contract.ValidationMode,
SchemaPath: contract.SchemaPath,
RepairAttempts: attemptsUsed,
IsValid: true,
}, nil
}
res, err := r.validator.Validate(ctx, artifact, contract)
if err != nil {
return domain.ValidationResult{}, err
}
res.RepairAttempts = attemptsUsed
return res, nil
}
func (r *Runner) shouldAttemptRepair(contract domain.OutputContract, validationResult domain.ValidationResult) bool {
if r.repairer == nil {
return false
@@ -527,7 +569,7 @@ func mergeExecutionTarget(base domain.ExecutionTarget, override domain.Execution
return out
}
func mergeExecutionTargetOverride(base domain.ExecutionTarget, override domain.ExecutionTargetOverride) (domain.ExecutionTarget, domain.ExecutionTargetPresence, error) {
func mergeExecutionTargetOverride(base domain.ExecutionTarget, override domain.ExecutionTargetOverride) (domain.ExecutionTarget, domain.ExecutionTargetPresence) {
out := base
var presence domain.ExecutionTargetPresence
if override.Endpoint != "" {
@@ -537,30 +579,18 @@ func mergeExecutionTargetOverride(base domain.ExecutionTarget, override domain.E
out.Model = override.Model
}
if override.Temperature != nil {
if *override.Temperature < 0 || *override.Temperature > 2 {
return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("temperature must be between 0 and 2")
}
out.Temperature = *override.Temperature
presence.Temperature = true
}
if override.MaxTokens != nil {
if *override.MaxTokens < 0 {
return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("max_tokens must be greater than or equal to 0")
}
out.MaxTokens = *override.MaxTokens
presence.MaxTokens = true
}
if override.TopP != nil {
if *override.TopP < 0 || *override.TopP > 1 {
return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("top_p must be between 0 and 1")
}
out.TopP = *override.TopP
presence.TopP = true
}
if override.TimeoutSeconds != nil {
if *override.TimeoutSeconds < 0 {
return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("timeout_seconds must be greater than or equal to 0")
}
out.TimeoutSeconds = *override.TimeoutSeconds
presence.TimeoutSeconds = true
}
@@ -576,22 +606,18 @@ func mergeExecutionTargetOverride(base domain.ExecutionTarget, override domain.E
if len(override.ExtraParams) > 0 {
out.ExtraParams = copyExtraParams(override.ExtraParams)
}
return out, presence, nil
return out, presence
}
func resolveExecutionTarget(backendValue *domain.Backend, profileValue *domain.ExecutionProfile, override *domain.ExecutionTargetOverride) (domain.ExecutionTarget, domain.ExecutionTargetPresence, error) {
func resolveExecutionTarget(backendValue *domain.Backend, profileValue *domain.ExecutionProfile, override *domain.ExecutionTargetOverride) (domain.ExecutionTarget, domain.ExecutionTargetPresence) {
out := defaults.ExecutionTargetDefault()
out = mergeExecutionTarget(out, backendToTarget(backendValue))
out = mergeExecutionTarget(out, executionProfileToTarget(profileValue))
var presence domain.ExecutionTargetPresence
if override != nil {
var err error
out, presence, err = mergeExecutionTargetOverride(out, *override)
if err != nil {
return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, err
}
out, presence = mergeExecutionTargetOverride(out, *override)
}
return out, presence, nil
return out, presence
}
func validateAPIKey(apiKeyEnv string, apiKey string, apiKeyRequired bool) error {
@@ -658,18 +684,21 @@ func copyExtraParams(src map[string]any) map[string]any {
return cp
}
func resolveOutputContract(def *domain.PromptDefinition, override *domain.OutputContract) domain.OutputContract {
func resolveOutputContract(def *domain.PromptDefinition, override *domain.OutputContract) (domain.OutputContract, error) {
contract := def.Validation
if contract.Format == "" {
contract.Format = def.OutputFormat
}
if override != nil {
contract = *override
if contract.Format == "" {
contract.Format = domain.FormatText
}
}
if contract.Format == "" {
contract.Format = domain.FormatText
if err := domain.ValidateOutputContract(contract); err != nil {
return domain.OutputContract{}, err
}
return contract
return contract, nil
}
func hashRenderedPrompt(p domain.RenderedPrompt) string {

View File

@@ -6,9 +6,11 @@ import (
"encoding/hex"
"errors"
"fmt"
"math"
"path/filepath"
"reflect"
"regexp"
"strconv"
"strings"
"sync"
"testing"
@@ -189,16 +191,34 @@ func (f *fakeValidator) Validate(ctx context.Context, artifact *domain.Artifact,
return f.result, nil
}
func (f *fakeValidator) LoadSchemaDocument(ctx context.Context, schemaPath string) (any, error) {
f.schemaLoads++
f.schemaLoadPath = schemaPath
if f.schemaErr != nil {
return nil, f.schemaErr
func (f *fakeValidator) PrepareValidation(_ context.Context, contract domain.OutputContract) (validate.PreparedValidation, error) {
var schemaDocument any
if contract.ValidationMode == domain.ValidationJSONSchema {
f.schemaLoads++
f.schemaLoadPath = contract.SchemaPath
if f.schemaErr != nil {
return nil, f.schemaErr
}
schemaDocument = f.schemaDoc
if schemaDocument == nil {
schemaDocument = map[string]any{"type": "object"}
}
}
if f.schemaDoc != nil {
return f.schemaDoc, nil
}
return map[string]any{"type": "object"}, nil
return &fakePreparedValidator{validator: f, contract: contract, schemaDocument: schemaDocument}, nil
}
type fakePreparedValidator struct {
validator *fakeValidator
contract domain.OutputContract
schemaDocument any
}
func (p *fakePreparedValidator) Validate(ctx context.Context, artifact *domain.Artifact) (domain.ValidationResult, error) {
return p.validator.Validate(ctx, artifact, p.contract)
}
func (p *fakePreparedValidator) SchemaDocument() any {
return p.schemaDocument
}
type fakeRepairer struct {
@@ -208,6 +228,30 @@ type fakeRepairer struct {
reqs []RepairRequest
}
type sequenceLLM struct {
responses []*domain.GenerateResponse
requests []domain.GenerateRequest
}
func (c *sequenceLLM) Generate(_ context.Context, req domain.GenerateRequest) (*domain.GenerateResponse, error) {
c.requests = append(c.requests, req)
index := len(c.requests) - 1
if index >= len(c.responses) {
return nil, errors.New("no generation response configured")
}
return c.responses[index], nil
}
type recordingRepairer struct {
next OutputRepairer
reqs []RepairRequest
}
func (r *recordingRepairer) Repair(ctx context.Context, req RepairRequest) (*domain.GenerateResponse, error) {
r.reqs = append(r.reqs, req)
return r.next.Repair(ctx, req)
}
type fakeRunAdmitter struct {
backendIDs []string
err error
@@ -502,6 +546,35 @@ func TestRunnerDirectSessionResolution(t *testing.T) {
t.Fatalf("invalid direct session invoked generation %d times", llmClient.calls)
}
})
t.Run("malformed direct value fails before loading or generation", func(t *testing.T) {
promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)}
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "unexpected"}}
runner := NewRunner(
promptRepo,
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}},
nil,
defaultArtifactReader(),
defaultRenderer(),
llmClient,
nil, nil)
_, err := runner.Run(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
SessionID: "session" + string([]byte{0xff}),
Inputs: singleInputRef(),
})
if !errors.Is(err, ErrInvalidRequest) {
t.Fatalf("expected ErrInvalidRequest, got %v", err)
}
if promptRepo.lastID != "" {
t.Fatalf("invalid direct session loaded prompt %q", promptRepo.lastID)
}
if llmClient.calls != 0 {
t.Fatalf("invalid direct session invoked generation %d times", llmClient.calls)
}
})
}
func TestRunnerPrepareUsesPromptDefaultProfileWhenNoExplicitProfileID(t *testing.T) {
@@ -704,16 +777,23 @@ func TestRunnerPrepareRequestNumericOverridePresence(t *testing.T) {
}
func TestRunnerPrepareInvalidRequestNumericOverridesFail(t *testing.T) {
tests := []struct {
type testCase struct {
name string
override *domain.ExecutionTargetOverride
}{
}
tests := []testCase{
{name: "temperature below range", override: &domain.ExecutionTargetOverride{Temperature: float64Ptr(-0.1)}},
{name: "temperature above range", override: &domain.ExecutionTargetOverride{Temperature: float64Ptr(2.1)}},
{name: "max tokens below range", override: &domain.ExecutionTargetOverride{MaxTokens: intPtr(-1)}},
{name: "top p below range", override: &domain.ExecutionTargetOverride{TopP: float64Ptr(-0.1)}},
{name: "top p above range", override: &domain.ExecutionTargetOverride{TopP: float64Ptr(1.1)}},
{name: "timeout below range", override: &domain.ExecutionTargetOverride{TimeoutSeconds: intPtr(-1)}},
{name: "temperature is not finite", override: &domain.ExecutionTargetOverride{Temperature: float64Ptr(math.NaN())}},
{name: "top p is not finite", override: &domain.ExecutionTargetOverride{TopP: float64Ptr(math.Inf(1))}},
}
if strconv.IntSize == 64 {
durationLimit := int64(math.MaxInt64 / int64(time.Second))
tests = append(tests, testCase{name: "timeout cannot be represented as a duration", override: &domain.ExecutionTargetOverride{TimeoutSeconds: intPtr(int(durationLimit) + 1)}})
}
for _, tc := range tests {
@@ -2014,50 +2094,255 @@ func TestRunnerRunValidationStillWorks(t *testing.T) {
}
}
func TestRunnerRunStructuredRepairRemainsBoundedAndUsesEffectiveModelSettings(t *testing.T) {
repairer := &fakeRepairer{responses: []*domain.GenerateResponse{{Content: `{"broken":`}, {Content: `{"still":`}}}
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: `{"initial":`}}
func TestRunnerRepairStateMachine(t *testing.T) {
failed := func(mode domain.ValidationMode, diagnostic string) domain.ValidationResult {
return domain.ValidationResult{
Status: domain.ValidationFailed,
Mode: mode,
Errors: []string{diagnostic},
IsValid: false,
}
}
passed := func(mode domain.ValidationMode) domain.ValidationResult {
return domain.ValidationResult{
Status: domain.ValidationPassed,
Mode: mode,
IsValid: true,
}
}
responses := func(count int) []*domain.GenerateResponse {
values := make([]*domain.GenerateResponse, count)
for index := range values {
unit := index + 1
values[index] = &domain.GenerateResponse{
Content: fmt.Sprintf(`{"candidate":%d}`, index),
Usage: domain.TokenUsage{
PromptTokens: unit,
CompletionTokens: unit * 10,
TotalTokens: unit * 100,
CachedTokens: unit * 1000,
CacheWriteTokens: unit * 10000,
},
}
}
return values
}
zeroOverrides := &domain.ExecutionTargetOverride{
Temperature: float64Ptr(0),
MaxTokens: intPtr(0),
TopP: float64Ptr(0),
TimeoutSeconds: intPtr(0),
}
allPresent := domain.ExecutionTargetPresence{
Temperature: true,
MaxTokens: true,
TopP: true,
TimeoutSeconds: true,
}
runner := NewRunnerWithRepairer(
&fakePromptRepo{def: promptDef(domain.FormatJSON, domain.ValidationJSON, 1)},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{
"exec": {ID: "exec", BackendID: "custom", Model: "profile-model", TimeoutSeconds: 55},
}}, fakeBackendResolver{backends: map[string]domain.Backend{
"custom": {ID: "custom", Endpoint: "http://backend/v1"},
}},
tests := []struct {
name string
mode domain.ValidationMode
budget int
validationResults []domain.ValidationResult
responses []*domain.GenerateResponse
execution *domain.ExecutionTargetOverride
wantPresence domain.ExecutionTargetPresence
wantRepairs int
wantStatus domain.ValidationStatus
structured bool
}{
{
name: "initial success does not repair",
mode: domain.ValidationJSON,
budget: 3,
validationResults: []domain.ValidationResult{passed(domain.ValidationJSON)},
responses: responses(1),
wantStatus: domain.ValidationPassed,
},
{
name: "basic failure is ineligible despite budget",
mode: domain.ValidationBasic,
budget: 3,
validationResults: []domain.ValidationResult{failed(domain.ValidationBasic, "empty output")},
responses: responses(1),
wantStatus: domain.ValidationFailed,
},
{
name: "inherited numeric values remain absent",
mode: domain.ValidationJSON,
budget: 1,
validationResults: []domain.ValidationResult{
failed(domain.ValidationJSON, "initial syntax"),
passed(domain.ValidationJSON),
},
responses: responses(2),
wantRepairs: 1,
wantStatus: domain.ValidationPassed,
},
{
name: "explicit numeric zeros remain present",
mode: domain.ValidationJSONSchema,
budget: 1,
execution: zeroOverrides,
structured: true,
validationResults: []domain.ValidationResult{
failed(domain.ValidationJSONSchema, "initial schema mismatch"),
passed(domain.ValidationJSONSchema),
},
responses: responses(2),
wantPresence: allPresent,
wantRepairs: 1,
wantStatus: domain.ValidationPassed,
},
{
name: "successful repair stops below larger budget",
mode: domain.ValidationJSON,
budget: 4,
validationResults: []domain.ValidationResult{
failed(domain.ValidationJSON, "candidate zero"),
failed(domain.ValidationJSON, "candidate one"),
passed(domain.ValidationJSON),
},
responses: responses(3),
wantRepairs: 2,
wantStatus: domain.ValidationPassed,
},
{
name: "failed repairs exhaust exact larger budget",
mode: domain.ValidationJSON,
budget: 3,
validationResults: []domain.ValidationResult{
failed(domain.ValidationJSON, "candidate zero"),
failed(domain.ValidationJSON, "candidate one"),
failed(domain.ValidationJSON, "candidate two"),
failed(domain.ValidationJSON, "candidate three"),
},
responses: responses(4),
wantRepairs: 3,
wantStatus: domain.ValidationFailed,
},
}
defaultArtifactReader(),
defaultRenderer(),
llmClient,
validate.NewStandardValidator("."),
repairer, nil)
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
definition := promptDef(domain.FormatJSON, tc.mode, tc.budget)
plan := &recordingPreparedValidation{results: tc.validationResults}
if tc.structured {
definition.Validation.SchemaPath = "schema.json"
plan.schemaDocument = map[string]any{"type": "object"}
}
validator := &recordingValidationPreparer{plan: plan}
client := &sequenceLLM{responses: tc.responses}
repairer := &recordingRepairer{next: NewDefaultOutputRepairer(client)}
runner := NewRunnerWithRepairer(
&fakePromptRepo{def: definition},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{
"exec": {ID: "exec", BackendID: "custom", Model: "profile-model"},
}},
fakeBackendResolver{backends: map[string]domain.Backend{
"custom": {ID: "custom", Endpoint: "http://backend.example/v1"},
}},
defaultArtifactReader(),
defaultRenderer(),
client,
validator,
repairer,
nil,
)
res, err := runner.Run(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
Execution: &domain.ExecutionTargetOverride{Endpoint: "http://override/v1", Model: "override-model", TimeoutSeconds: intPtr(22)},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if repairer.calls != 1 || res.Validation.RepairAttempts != 1 {
t.Fatalf("expected one bounded repair, calls=%d attempts=%d", repairer.calls, res.Validation.RepairAttempts)
}
if len(repairer.reqs) != 1 {
t.Fatalf("expected one repair request, got %d", len(repairer.reqs))
}
if repairer.reqs[0].Target.Endpoint != "http://override/v1" || repairer.reqs[0].Target.Model != "override-model" {
t.Fatalf("expected repair to use effective target, got %+v", repairer.reqs[0].Target)
}
if repairer.reqs[0].Target.TimeoutSeconds != 22 {
t.Fatalf("expected repair to use effective timeout, got %d", repairer.reqs[0].Target.TimeoutSeconds)
}
if llmClient.lastReq.Target.BackendID != "custom" ||
repairer.reqs[0].Target.BackendID != "custom" ||
res.SelectedBackendID != "custom" {
t.Fatalf("expected backend identity in generation, repair, and result: generate=%q repair=%q result=%q",
llmClient.lastReq.Target.BackendID, repairer.reqs[0].Target.BackendID, res.SelectedBackendID)
result, err := runner.Run(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
SessionID: " repair-session ",
APIKey: "direct-secret",
Inputs: singleInputRef(),
Execution: tc.execution,
})
if err != nil {
t.Fatalf("run: %v", err)
}
if len(client.requests) != tc.wantRepairs+1 || len(repairer.reqs) != tc.wantRepairs {
t.Fatalf(
"generation/repair calls = (%d, %d), want (%d, %d)",
len(client.requests), len(repairer.reqs), tc.wantRepairs+1, tc.wantRepairs,
)
}
if len(plan.artifacts) != tc.wantRepairs+1 {
t.Fatalf("validation calls = %d, want %d", len(plan.artifacts), tc.wantRepairs+1)
}
initialRequest := client.requests[0]
if initialRequest.TargetPresence != tc.wantPresence {
t.Fatalf("initial target presence = %+v, want %+v", initialRequest.TargetPresence, tc.wantPresence)
}
if initialRequest.Target.APIKey != "direct-secret" || initialRequest.Target.BackendID != "custom" ||
initialRequest.Prompt.SessionID != "repair-session" {
t.Fatalf("initial common request fields = %+v", initialRequest)
}
if initialRequest.Target.Endpoint != "http://backend.example/v1" ||
initialRequest.Target.Model != "profile-model" ||
initialRequest.Target.Temperature != 0 || initialRequest.Target.MaxTokens != 0 || initialRequest.Target.TopP != 0 {
t.Fatalf("initial effective target = %+v", initialRequest.Target)
}
if tc.structured != (initialRequest.StructuredOutput != nil) {
t.Fatalf("initial structured output = %+v, want present %v", initialRequest.StructuredOutput, tc.structured)
}
if tc.structured && (initialRequest.StructuredOutput.JSONSchema == nil ||
initialRequest.StructuredOutput.JSONSchema.Name != "p_1") {
t.Fatalf("initial JSON Schema metadata = %+v, want derived schema name p_1", initialRequest.StructuredOutput)
}
for index, req := range repairer.reqs {
if req.Attempt != index+1 || req.MaxAttempts != tc.budget || req.Mode != tc.mode {
t.Fatalf("repair request %d progression = %+v", index, req)
}
if req.PreviousOutput != tc.responses[index].Content ||
!reflect.DeepEqual(req.ValidationErrors, tc.validationResults[index].Errors) {
t.Fatalf("repair request %d prior state = %+v", index, req)
}
if req.TargetPresence != tc.wantPresence || !reflect.DeepEqual(req.Target, initialRequest.Target) ||
req.SessionID != initialRequest.Prompt.SessionID ||
!reflect.DeepEqual(req.StructuredOutput, initialRequest.StructuredOutput) {
t.Fatalf("repair request %d common fields drifted: %+v", index, req)
}
generated := client.requests[index+1]
if generated.TargetPresence != initialRequest.TargetPresence ||
!reflect.DeepEqual(generated.Target, initialRequest.Target) ||
generated.Prompt.SessionID != initialRequest.Prompt.SessionID ||
!reflect.DeepEqual(generated.StructuredOutput, initialRequest.StructuredOutput) {
t.Fatalf("repair generation request %d common fields drifted: %+v", index, generated)
}
if reflect.DeepEqual(generated.Prompt.Messages, initialRequest.Prompt.Messages) {
t.Fatalf("repair generation request %d reused the initial prompt", index)
}
}
lastResponse := tc.responses[tc.wantRepairs]
if result.RawOutput != lastResponse.Content || string(result.Artifact.Body) != lastResponse.Content {
t.Fatalf("final output = (%q, %q), want %q", result.RawOutput, result.Artifact.Body, lastResponse.Content)
}
if result.Validation.Status != tc.wantStatus || result.Validation.RepairAttempts != tc.wantRepairs {
t.Fatalf("final validation = %+v, want status %q and %d repairs", result.Validation, tc.wantStatus, tc.wantRepairs)
}
if result.SelectedBackendID != "custom" || result.SessionID != "repair-session" ||
result.EffectiveModelParams.APIKey != "" {
t.Fatalf("result execution metadata = %+v", result)
}
var wantUsage domain.TokenUsage
for _, response := range tc.responses[:tc.wantRepairs+1] {
wantUsage.PromptTokens += response.Usage.PromptTokens
wantUsage.CompletionTokens += response.Usage.CompletionTokens
wantUsage.TotalTokens += response.Usage.TotalTokens
wantUsage.CachedTokens += response.Usage.CachedTokens
wantUsage.CacheWriteTokens += response.Usage.CacheWriteTokens
}
if result.Usage != wantUsage {
t.Fatalf("cumulative usage = %+v, want %+v", result.Usage, wantUsage)
}
})
}
}
@@ -2152,96 +2437,6 @@ func TestRunnerSchedulesInitialAndRepairGenerationThroughOneBackendPool(t *testi
}
}
func TestRunnerRunRepairCarriesEffectiveSessionID(t *testing.T) {
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: `{"broken":`}}
runner := NewRunnerWithRepairer(
&fakePromptRepo{def: promptDef(domain.FormatJSON, domain.ValidationJSON, 1)},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{
"exec": {ID: "exec", Endpoint: "http://example.test/v1", Model: "model"},
}},
nil,
defaultArtifactReader(),
defaultRenderer(),
llmClient,
validate.NewStandardValidator("."),
NewDefaultOutputRepairer(llmClient), nil)
result, err := runner.Run(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
SessionID: " repair-session ",
Inputs: singleInputRef(),
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if llmClient.calls != 2 {
t.Fatalf("expected initial generation and one repair, got %d calls", llmClient.calls)
}
if llmClient.lastReq.Prompt.SessionID != "repair-session" {
t.Fatalf("expected repair generation to retain effective session, got %q", llmClient.lastReq.Prompt.SessionID)
}
if result.SessionID != "repair-session" {
t.Fatalf("expected result to retain effective session, got %q", result.SessionID)
}
}
func TestRunnerRunJSONSchemaRepairCarriesStructuredOutputSpec(t *testing.T) {
def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 1)
def.Validation.SchemaPath = "events.schema.json"
validator := &fakeValidator{
result: domain.ValidationResult{
Status: domain.ValidationFailed,
Mode: domain.ValidationJSONSchema,
Errors: []string{"schema mismatch"},
IsValid: false,
},
schemaDoc: map[string]any{
"type": "object",
"properties": map[string]any{
"events": map[string]any{"type": "array"},
},
},
}
repairer := &fakeRepairer{
responses: []*domain.GenerateResponse{
{Content: `{"events":[]}`},
},
}
llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: `{"events":[1]}`}}
runner := NewRunnerWithRepairer(
&fakePromptRepo{def: def},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, nil,
defaultArtifactReader(),
defaultRenderer(),
llmClient,
validator,
repairer, nil)
_, err := runner.Run(context.Background(), domain.RunRequest{
PromptID: "p",
ProfileID: "exec",
Inputs: singleInputRef(),
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if llmClient.lastReq.StructuredOutput == nil || llmClient.lastReq.StructuredOutput.JSONSchema == nil {
t.Fatalf("expected initial llm request to include structured output, got %+v", llmClient.lastReq.StructuredOutput)
}
if len(repairer.reqs) != 1 {
t.Fatalf("expected one repair request, got %d", len(repairer.reqs))
}
if repairer.reqs[0].StructuredOutput == nil || repairer.reqs[0].StructuredOutput.JSONSchema == nil {
t.Fatalf("expected repair request structured output, got %+v", repairer.reqs[0].StructuredOutput)
}
if repairer.reqs[0].StructuredOutput.JSONSchema.Name != "p_1" {
t.Fatalf("expected derived schema name p_1, got %q", repairer.reqs[0].StructuredOutput.JSONSchema.Name)
}
}
func TestExecutionProfileToTargetPopulatesAllFieldsAndCopiesExtraParams(t *testing.T) {
src := &domain.ExecutionProfile{
ID: "exec",
@@ -2300,10 +2495,7 @@ func TestResolveExecutionTargetProfileValuesPopulateAllSupportedFields(t *testin
},
}
target, presence, err := resolveExecutionTarget(nil, profileValue, nil)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
target, presence := resolveExecutionTarget(nil, profileValue, nil)
if presence != (domain.ExecutionTargetPresence{}) {
t.Fatalf("expected no request override presence, got %+v", presence)
}
@@ -2355,10 +2547,7 @@ func TestResolveExecutionTargetRuntimeOverridesBeatProfileForAllOverrideableFiel
},
}
target, presence, err := resolveExecutionTarget(nil, profileValue, override)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
target, presence := resolveExecutionTarget(nil, profileValue, override)
if presence != (domain.ExecutionTargetPresence{Temperature: true, MaxTokens: true, TopP: true, TimeoutSeconds: true}) {
t.Fatalf("unexpected override presence: %+v", presence)
}
@@ -2405,12 +2594,9 @@ func TestResolveExecutionTargetReasoningOverrideStates(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
target, _, err := resolveExecutionTarget(nil, profileValue, &domain.ExecutionTargetOverride{
target, _ := resolveExecutionTarget(nil, profileValue, &domain.ExecutionTargetOverride{
ReasoningEffort: tt.override,
})
if err != nil {
t.Fatalf("resolve execution target: %v", err)
}
if target.ReasoningEffort != tt.want {
t.Fatalf("reasoning effort = %q, want %q", target.ReasoningEffort, tt.want)
}
@@ -2539,10 +2725,7 @@ func TestResolveExecutionTargetUsesBackendProfileAndRequestPrecedence(t *testing
ExtraParams: map[string]any{"request": true},
}
target, _, err := resolveExecutionTarget(backendValue, profileValue, override)
if err != nil {
t.Fatalf("resolve target: %v", err)
}
target, _ := resolveExecutionTarget(backendValue, profileValue, override)
if target.BackendID != "custom" {
t.Fatalf("endpoint override changed backend identity: %+v", target)
}
@@ -2553,12 +2736,9 @@ func TestResolveExecutionTargetUsesBackendProfileAndRequestPrecedence(t *testing
t.Fatalf("expected whole-map request replacement, got %#v", target.ExtraParams)
}
target, _, err = resolveExecutionTarget(backendValue, &domain.ExecutionProfile{
target, _ = resolveExecutionTarget(backendValue, &domain.ExecutionProfile{
ID: "exec", BackendID: "custom", Model: "profile-model",
}, nil)
if err != nil {
t.Fatalf("resolve backend defaults: %v", err)
}
if target.Endpoint != backendValue.Endpoint ||
target.APIKeyEnv != backendValue.APIKeyEnv ||
!reflect.DeepEqual(target.ExtraParams, backendValue.ExtraParams) {

View File

@@ -0,0 +1,311 @@
package validate
import (
"context"
"errors"
"io/fs"
"strings"
"sync"
"sync/atomic"
"testing"
"testing/fstest"
"time"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
"github.com/santhosh-tekuri/jsonschema/v6"
)
func TestValidationCancellationBeforeWorkDoesNotOpenSchemaSource(t *testing.T) {
source := &countingSchemaFS{FS: fstest.MapFS{
"schema.json": {Data: []byte(`{"type":"object"}`)},
}}
validator := NewFSValidator(source, ".").(ValidationPreparer)
ctx, cancel := context.WithCancel(context.Background())
cancel()
plan, err := validator.PrepareValidation(ctx, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "schema.json",
})
if plan != nil || !errors.Is(err, context.Canceled) {
t.Fatalf("PrepareValidation() = (%v, %v), want nil plan and context cancellation", plan, err)
}
if opens := source.opens.Load(); opens != 0 {
t.Fatalf("schema source opens = %d, want 0", opens)
}
}
func TestValidationCancellationBetweenReferencedSchemaReadChunks(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
source := &controlledSchemaFS{
FS: fstest.MapFS{
"root.json": {Data: []byte(`{"$ref":"child.json"}`)},
"child.json": {Data: []byte(`{"type":"string"}` + strings.Repeat(" ", schemaReadChunkSize*2))},
},
target: "child.json",
cancel: cancel,
}
validator := NewFSValidator(source, ".").(ValidationPreparer)
plan, err := validator.PrepareValidation(ctx, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "root.json",
})
if plan != nil || !errors.Is(err, context.Canceled) {
t.Fatalf("PrepareValidation() = (%v, %v), want nil plan and context cancellation", plan, err)
}
if reads := source.reads.Load(); reads != 1 {
t.Fatalf("controlled child reads = %d, want 1", reads)
}
if closes := source.closes.Load(); closes != 1 {
t.Fatalf("controlled child closes = %d, want 1", closes)
}
}
func TestDecodeCancellationAfterSynchronousCallWins(t *testing.T) {
ctx := newCheckpointContext(2)
value, err := decodeJSONValue(ctx, []byte(`{"value":1}`))
if value != nil || !errors.Is(err, context.Canceled) {
t.Fatalf("decodeJSONValue() = (%v, %v), want nil value and context cancellation", value, err)
}
}
func TestCompileCancellationAfterSynchronousCallWins(t *testing.T) {
for _, dependencyErr := range []error{nil, errors.New("compile failed")} {
ctx, cancel := context.WithCancel(context.Background())
compiled := &jsonschema.Schema{}
schema, err := compileJSONSchema(ctx, "promptkit-schema:/root.json", func(string) (*jsonschema.Schema, error) {
cancel()
return compiled, dependencyErr
})
if schema != nil || !errors.Is(err, context.Canceled) {
t.Fatalf("compileJSONSchema() = (%v, %v), want authoritative context cancellation", schema, err)
}
if dependencyErr != nil && errors.Is(err, dependencyErr) {
t.Fatalf("compileJSONSchema() error = %v, dependency error should not win", err)
}
}
}
func TestExecutionCancellationAfterSynchronousCallWins(t *testing.T) {
for _, dependencyErr := range []error{nil, errors.New("schema mismatch")} {
ctx, cancel := context.WithCancel(context.Background())
executor := &controlledSchemaExecutor{cancel: cancel, err: dependencyErr}
validationErrors, err := executeJSONSchema(ctx, executor, map[string]any{"ok": true})
if validationErrors != nil || !errors.Is(err, context.Canceled) {
t.Fatalf("executeJSONSchema() = (%v, %v), want no result and context cancellation", validationErrors, err)
}
if calls := executor.calls.Load(); calls != 1 {
t.Fatalf("schema execution calls = %d, want 1", calls)
}
}
}
func TestCancellationDoesNotDetachBlockedSchemaRead(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
started := make(chan struct{})
release := make(chan struct{})
source := &controlledSchemaFS{
FS: fstest.MapFS{
"schema.json": {Data: []byte(`{"type":"object"}`)},
},
target: "schema.json",
started: started,
release: release,
}
validator := NewFSValidator(source, ".").(ValidationPreparer)
type outcome struct {
plan PreparedValidation
err error
}
result := make(chan outcome, 1)
go func() {
plan, err := validator.PrepareValidation(ctx, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "schema.json",
})
result <- outcome{plan: plan, err: err}
}()
waitForSignal(t, started, "schema read to start")
cancel()
select {
case got := <-result:
t.Fatalf("blocked dependency returned before release: (%v, %v)", got.plan, got.err)
default:
}
close(release)
got := waitForOutcome(t, result)
if got.plan != nil || !errors.Is(got.err, context.Canceled) {
t.Fatalf("PrepareValidation() after release = (%v, %v), want nil plan and context cancellation", got.plan, got.err)
}
if reads := source.reads.Load(); reads != 1 {
t.Fatalf("blocked reads = %d, want 1 completed read", reads)
}
}
func TestCancellationDoesNotDetachBlockedSchemaExecution(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
started := make(chan struct{})
release := make(chan struct{})
executor := &controlledSchemaExecutor{started: started, release: release}
result := make(chan error, 1)
go func() {
_, err := executeJSONSchema(ctx, executor, map[string]any{"ok": true})
result <- err
}()
waitForSignal(t, started, "schema execution to start")
cancel()
select {
case err := <-result:
t.Fatalf("blocked dependency returned before release: %v", err)
default:
}
close(release)
select {
case err := <-result:
if !errors.Is(err, context.Canceled) {
t.Fatalf("executeJSONSchema() after release = %v, want context cancellation", err)
}
case <-time.After(2 * time.Second):
t.Fatal("timed out waiting for released schema execution")
}
if calls := executor.calls.Load(); calls != 1 {
t.Fatalf("schema execution calls = %d, want 1 completed call", calls)
}
}
type countingSchemaFS struct {
fs.FS
opens atomic.Int32
}
func (f *countingSchemaFS) Open(name string) (fs.File, error) {
f.opens.Add(1)
return f.FS.Open(name)
}
type controlledSchemaFS struct {
fs.FS
target string
cancel context.CancelFunc
started chan struct{}
release chan struct{}
once sync.Once
reads atomic.Int32
closes atomic.Int32
}
func (f *controlledSchemaFS) Open(name string) (fs.File, error) {
file, err := f.FS.Open(name)
if err != nil || name != f.target {
return file, err
}
return &controlledSchemaFile{File: file, owner: f}, nil
}
type controlledSchemaFile struct {
fs.File
owner *controlledSchemaFS
}
func (f *controlledSchemaFile) Read(buffer []byte) (int, error) {
f.owner.once.Do(func() {
if f.owner.started != nil {
close(f.owner.started)
}
if f.owner.release != nil {
<-f.owner.release
}
})
n, err := f.File.Read(buffer)
f.owner.reads.Add(1)
if f.owner.cancel != nil {
f.owner.cancel()
}
return n, err
}
func (f *controlledSchemaFile) Close() error {
f.owner.closes.Add(1)
return f.File.Close()
}
type controlledSchemaExecutor struct {
cancel context.CancelFunc
err error
started chan struct{}
release chan struct{}
calls atomic.Int32
}
func (e *controlledSchemaExecutor) Validate(any) error {
e.calls.Add(1)
if e.started != nil {
close(e.started)
}
if e.release != nil {
<-e.release
}
if e.cancel != nil {
e.cancel()
}
return e.err
}
type checkpointContext struct {
context.Context
mu sync.Mutex
remaining int
canceled bool
done chan struct{}
}
func newCheckpointContext(checksUntilCancel int) *checkpointContext {
return &checkpointContext{
Context: context.Background(),
remaining: checksUntilCancel,
done: make(chan struct{}),
}
}
func (c *checkpointContext) Done() <-chan struct{} {
return c.done
}
func (c *checkpointContext) Err() error {
c.mu.Lock()
defer c.mu.Unlock()
if c.canceled {
return context.Canceled
}
c.remaining--
if c.remaining == 0 {
c.canceled = true
close(c.done)
return context.Canceled
}
return nil
}
func waitForSignal(t *testing.T, signal <-chan struct{}, description string) {
t.Helper()
select {
case <-signal:
case <-time.After(2 * time.Second):
t.Fatalf("timed out waiting for %s", description)
}
}
func waitForOutcome[T any](t *testing.T, result <-chan T) T {
t.Helper()
select {
case value := <-result:
return value
case <-time.After(2 * time.Second):
t.Fatal("timed out waiting for released dependency")
var zero T
return zero
}
}

View File

@@ -1,15 +1,18 @@
package validate
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"net/url"
"os"
"path"
"path/filepath"
"runtime"
"strings"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
@@ -19,6 +22,8 @@ import (
const jsonSchemaDraft2020 = "https://json-schema.org/draft/2020-12/schema"
const schemaReadChunkSize = 64 * 1024
// StandardValidator provides basic, JSON, and JSON Schema output validation.
type StandardValidator struct {
schemaBaseDir string
@@ -48,7 +53,7 @@ func (v *FSValidator) Validate(ctx context.Context, artifact *domain.Artifact, c
type preparedValidation struct {
contract domain.OutputContract
schemaDocument any
schema *jsonschema.Schema
schema schemaExecutor
}
func (p *preparedValidation) Validate(ctx context.Context, artifact *domain.Artifact) (domain.ValidationResult, error) {
@@ -59,14 +64,11 @@ func (p *preparedValidation) SchemaDocument() any {
return p.schemaDocument
}
func (p *preparedValidation) validateJSONSchema(instance any, _ string) ([]string, error) {
func (p *preparedValidation) validateJSONSchema(ctx context.Context, instance any, _ string) ([]string, error) {
if p.schema == nil {
return nil, errors.New("prepared JSON schema is unavailable")
}
if err := p.schema.Validate(instance); err != nil {
return []string{fmt.Sprintf("json schema validation failed: %v", err)}, nil
}
return nil, nil
return executeJSONSchema(ctx, p.schema, instance)
}
func (v *StandardValidator) PrepareValidation(ctx context.Context, contract domain.OutputContract) (PreparedValidation, error) {
@@ -79,11 +81,11 @@ func (v *StandardValidator) PrepareValidation(ctx context.Context, contract doma
return prepared, nil
}
resolvedSchemaPath, err := v.resolveSchemaPath(contract.SchemaPath)
resolvedSchemaPath, err := v.resolveSchemaPath(ctx, contract.SchemaPath)
if err != nil {
return nil, err
}
schemaDocument, err := loadJSONSchemaFile(resolvedSchemaPath)
schemaDocument, err := loadJSONSchemaFile(ctx, resolvedSchemaPath)
if err != nil {
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", resolvedSchemaPath, err)
}
@@ -91,11 +93,24 @@ func (v *StandardValidator) PrepareValidation(ctx context.Context, contract doma
if err != nil {
return nil, err
}
compiler := newSchemaCompiler(standardSchemaLoader{root: schemaRoot})
if err := compiler.AddResource(resolvedSchemaPath, schemaDocument); err != nil {
if err := ctx.Err(); err != nil {
return nil, err
}
compiler := newSchemaCompiler(standardSchemaLoader{ctx: ctx, root: schemaRoot})
resourceURL := fileSchemaResourceURL(resolvedSchemaPath)
if err := ctx.Err(); err != nil {
return nil, err
}
if err := compiler.AddResource(resourceURL.String(), schemaDocument); err != nil {
if contextErr := ctx.Err(); contextErr != nil {
return nil, contextErr
}
return nil, fmt.Errorf("failed to register JSON schema %q: %w", resolvedSchemaPath, err)
}
schema, err := compiler.Compile(resolvedSchemaPath)
if err := ctx.Err(); err != nil {
return nil, err
}
schema, err := compileJSONSchema(ctx, resourceURL.String(), compiler.Compile)
if err != nil {
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", resolvedSchemaPath, err)
}
@@ -118,16 +133,25 @@ func (v *FSValidator) PrepareValidation(ctx context.Context, contract domain.Out
return prepared, nil
}
schemaName, schemaDocument, err := v.loadSchemaDocument(contract.SchemaPath)
schemaName, schemaDocument, err := v.loadSchemaDocument(ctx, contract.SchemaPath)
if err != nil {
return nil, err
}
resourceURL := fsSchemaResourceURL(schemaName)
compiler := newSchemaCompiler(fsSchemaLoader{fsys: v.fsys, root: filecatalog.CleanFSRoot(v.root)})
if err := compiler.AddResource(resourceURL, schemaDocument); err != nil {
compiler := newSchemaCompiler(fsSchemaLoader{ctx: ctx, fsys: v.fsys, root: filecatalog.CleanFSRoot(v.root)})
if err := ctx.Err(); err != nil {
return nil, err
}
if err := compiler.AddResource(resourceURL.String(), schemaDocument); err != nil {
if contextErr := ctx.Err(); contextErr != nil {
return nil, contextErr
}
return nil, fmt.Errorf("failed to register JSON schema %q: %w", schemaName, err)
}
schema, err := compiler.Compile(resourceURL)
if err := ctx.Err(); err != nil {
return nil, err
}
schema, err := compileJSONSchema(ctx, resourceURL.String(), compiler.Compile)
if err != nil {
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", schemaName, err)
}
@@ -140,7 +164,7 @@ func (v *FSValidator) PrepareValidation(ctx context.Context, contract domain.Out
return prepared, nil
}
type schemaValidatorFunc func(instance any, schemaPath string) ([]string, error)
type schemaValidatorFunc func(ctx context.Context, instance any, schemaPath string) ([]string, error)
func validateArtifact(ctx context.Context, artifact *domain.Artifact, contract domain.OutputContract, validateSchema schemaValidatorFunc) (domain.ValidationResult, error) {
select {
@@ -165,7 +189,11 @@ func validateArtifact(ctx context.Context, artifact *domain.Artifact, contract d
res.IsValid = true
return res, nil
case domain.ValidationBasic:
if strings.TrimSpace(string(artifact.Body)) == "" {
empty := strings.TrimSpace(string(artifact.Body)) == ""
if err := ctx.Err(); err != nil {
return domain.ValidationResult{}, err
}
if empty {
res.Status = domain.ValidationFailed
res.IsValid = false
res.Errors = []string{"output is empty"}
@@ -175,26 +203,32 @@ func validateArtifact(ctx context.Context, artifact *domain.Artifact, contract d
res.IsValid = true
return res, nil
case domain.ValidationJSON:
_, jsonErr := parseJSON(artifact.Body)
if jsonErr != nil {
valid := json.Valid(artifact.Body)
if err := ctx.Err(); err != nil {
return domain.ValidationResult{}, err
}
if !valid {
res.Status = domain.ValidationFailed
res.IsValid = false
res.Errors = []string{fmt.Sprintf("invalid JSON: %v", jsonErr)}
res.Errors = []string{"invalid JSON"}
return res, nil
}
res.Status = domain.ValidationPassed
res.IsValid = true
return res, nil
case domain.ValidationJSONSchema:
instance, jsonErr := parseJSON(artifact.Body)
instance, jsonErr := decodeJSONValue(ctx, artifact.Body)
if jsonErr != nil {
if contextErr := ctx.Err(); contextErr != nil {
return domain.ValidationResult{}, contextErr
}
res.Status = domain.ValidationFailed
res.IsValid = false
res.Errors = []string{fmt.Sprintf("invalid JSON: %v", jsonErr)}
return res, nil
}
validationErrors, err := validateSchema(instance, contract.SchemaPath)
validationErrors, err := validateSchema(ctx, instance, contract.SchemaPath)
if err != nil {
return domain.ValidationResult{}, err
}
@@ -213,8 +247,8 @@ func validateArtifact(ctx context.Context, artifact *domain.Artifact, contract d
}
}
func (v *StandardValidator) validateJSONSchema(instance any, schemaPath string) ([]string, error) {
resolvedSchemaPath, err := v.resolveSchemaPath(schemaPath)
func (v *StandardValidator) validateJSONSchema(ctx context.Context, instance any, schemaPath string) ([]string, error) {
resolvedSchemaPath, err := v.resolveSchemaPath(ctx, schemaPath)
if err != nil {
return nil, err
}
@@ -223,20 +257,21 @@ func (v *StandardValidator) validateJSONSchema(instance any, schemaPath string)
if err != nil {
return nil, err
}
compiler := newSchemaCompiler(standardSchemaLoader{root: schemaRoot})
schema, err := compiler.Compile(resolvedSchemaPath)
if err := ctx.Err(); err != nil {
return nil, err
}
compiler := newSchemaCompiler(standardSchemaLoader{ctx: ctx, root: schemaRoot})
resourceURL := fileSchemaResourceURL(resolvedSchemaPath)
schema, err := compileJSONSchema(ctx, resourceURL.String(), compiler.Compile)
if err != nil {
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", resolvedSchemaPath, err)
}
if err := schema.Validate(instance); err != nil {
return []string{fmt.Sprintf("json schema validation failed: %v", err)}, nil
}
return nil, nil
return executeJSONSchema(ctx, schema, instance)
}
func (v *FSValidator) validateJSONSchema(instance any, schemaPath string) ([]string, error) {
schemaName, schemaDoc, err := v.loadSchemaDocument(schemaPath)
func (v *FSValidator) validateJSONSchema(ctx context.Context, instance any, schemaPath string) ([]string, error) {
schemaName, schemaDoc, err := v.loadSchemaDocument(ctx, schemaPath)
if err != nil {
return nil, err
}
@@ -245,71 +280,60 @@ func (v *FSValidator) validateJSONSchema(instance any, schemaPath string) ([]str
if err := validateSchemaDialect(schemaDoc); err != nil {
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", schemaName, err)
}
compiler := newSchemaCompiler(fsSchemaLoader{fsys: v.fsys, root: filecatalog.CleanFSRoot(v.root)})
if err := compiler.AddResource(resourceURL, schemaDoc); err != nil {
compiler := newSchemaCompiler(fsSchemaLoader{ctx: ctx, fsys: v.fsys, root: filecatalog.CleanFSRoot(v.root)})
if err := ctx.Err(); err != nil {
return nil, err
}
if err := compiler.AddResource(resourceURL.String(), schemaDoc); err != nil {
if contextErr := ctx.Err(); contextErr != nil {
return nil, contextErr
}
return nil, fmt.Errorf("failed to register JSON schema %q: %w", schemaName, err)
}
schema, err := compiler.Compile(resourceURL)
if err := ctx.Err(); err != nil {
return nil, err
}
schema, err := compileJSONSchema(ctx, resourceURL.String(), compiler.Compile)
if err != nil {
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", schemaName, err)
}
if err := schema.Validate(instance); err != nil {
return []string{fmt.Sprintf("json schema validation failed: %v", err)}, nil
}
return nil, nil
return executeJSONSchema(ctx, schema, instance)
}
func parseJSON(body []byte) (any, error) {
var v any
if err := json.Unmarshal(body, &v); err != nil {
func decodeJSONValue(ctx context.Context, body []byte) (any, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
return v, nil
}
decoder := json.NewDecoder(bytes.NewReader(body))
decoder.UseNumber()
func (v *StandardValidator) LoadSchemaDocument(ctx context.Context, schemaPath string) (any, error) {
select {
case <-ctx.Done():
return nil, ctx.Err()
default:
}
resolved, err := v.resolveSchemaPath(schemaPath)
if err != nil {
var value any
decodeErr := decoder.Decode(&value)
if err := ctx.Err(); err != nil {
return nil, err
}
raw, err := os.ReadFile(resolved)
if err != nil {
return nil, fmt.Errorf("failed to read schema file %q: %w", resolved, err)
if decodeErr != nil {
return nil, decodeErr
}
var doc any
if err := json.Unmarshal(raw, &doc); err != nil {
return nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
}
if err := validateSchemaDialect(doc); err != nil {
return nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
}
return doc, nil
}
func (v *FSValidator) LoadSchemaDocument(ctx context.Context, schemaPath string) (any, error) {
select {
case <-ctx.Done():
return nil, ctx.Err()
default:
}
_, doc, err := v.loadSchemaDocument(schemaPath)
if err != nil {
var trailing any
trailingErr := decoder.Decode(&trailing)
if err := ctx.Err(); err != nil {
return nil, err
}
return doc, nil
if errors.Is(trailingErr, io.EOF) {
return value, nil
} else if trailingErr != nil {
return nil, trailingErr
}
return nil, errors.New("multiple JSON values")
}
func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error) {
func (v *StandardValidator) resolveSchemaPath(ctx context.Context, schemaPath string) (string, error) {
if err := ctx.Err(); err != nil {
return "", err
}
if strings.TrimSpace(schemaPath) == "" {
return "", errors.New("schema path is required for json_schema validation")
}
@@ -318,13 +342,25 @@ func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error)
if err != nil {
return "", err
}
if err := ctx.Err(); err != nil {
return "", err
}
resolved, err := containedFilesystemPath(root, schemaPath)
if err != nil {
return "", err
}
if err := ctx.Err(); err != nil {
return "", err
}
if _, err := os.Stat(resolved); err != nil {
if contextErr := ctx.Err(); contextErr != nil {
return "", contextErr
}
return "", fmt.Errorf("failed to access schema file %q: %w", resolved, err)
}
if err := ctx.Err(); err != nil {
return "", err
}
return resolved, nil
}
@@ -345,28 +381,39 @@ func (v *StandardValidator) schemaRoot() (string, error) {
return resolved, nil
}
func (v *FSValidator) loadSchemaDocument(schemaPath string) (string, any, error) {
resolved, err := v.resolveSchemaPath(schemaPath)
func (v *FSValidator) loadSchemaDocument(ctx context.Context, schemaPath string) (string, any, error) {
resolved, err := v.resolveSchemaPath(ctx, schemaPath)
if err != nil {
return "", nil, err
}
raw, err := fs.ReadFile(v.fsys, resolved)
raw, err := readSchemaFile(ctx, func() (fs.File, error) {
return v.fsys.Open(resolved)
})
if err != nil {
return "", nil, fmt.Errorf("failed to read schema file %q: %w", resolved, err)
}
var doc any
if err := json.Unmarshal(raw, &doc); err != nil {
doc, err := decodeJSONValue(ctx, raw)
if err != nil {
return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
}
if err := validateSchemaDialect(doc); err != nil {
if contextErr := ctx.Err(); contextErr != nil {
return "", nil, contextErr
}
return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
}
if err := ctx.Err(); err != nil {
return "", nil, err
}
return resolved, doc, nil
}
func (v *FSValidator) resolveSchemaPath(schemaPath string) (string, error) {
func (v *FSValidator) resolveSchemaPath(ctx context.Context, schemaPath string) (string, error) {
if err := ctx.Err(); err != nil {
return "", err
}
if strings.TrimSpace(schemaPath) == "" {
return "", errors.New("schema path is required for json_schema validation")
}
@@ -377,8 +424,14 @@ func (v *FSValidator) resolveSchemaPath(schemaPath string) (string, error) {
cleanRoot := filecatalog.CleanFSRoot(v.root)
rootInfo, err := fs.Stat(v.fsys, cleanRoot)
if err != nil {
if contextErr := ctx.Err(); contextErr != nil {
return "", contextErr
}
return "", fmt.Errorf("failed to access schema source %q: %w", cleanRoot, err)
}
if err := ctx.Err(); err != nil {
return "", err
}
var resolved string
if rootInfo.IsDir() {
@@ -392,15 +445,21 @@ func (v *FSValidator) resolveSchemaPath(schemaPath string) (string, error) {
if err != nil {
return "", err
}
if cleanSchemaPath != path.Base(cleanRoot) {
if cleanSchemaPath != strings.TrimSpace(path.Base(cleanRoot)) {
return "", fmt.Errorf("schema path %q does not match schema file %q", cleanSchemaPath, path.Base(cleanRoot))
}
resolved = cleanRoot
}
if _, err := fs.Stat(v.fsys, resolved); err != nil {
if contextErr := ctx.Err(); contextErr != nil {
return "", contextErr
}
return "", fmt.Errorf("failed to access schema file %q: %w", resolved, err)
}
if err := ctx.Err(); err != nil {
return "", err
}
return resolved, nil
}
@@ -416,8 +475,19 @@ func cleanSchemaFSPath(schemaPath string) (string, error) {
return cleaned, nil
}
func fsSchemaResourceURL(schemaName string) string {
return "promptkit-schema:///" + strings.TrimPrefix(path.Clean(schemaName), "/")
func fileSchemaResourceURL(schemaName string) *url.URL {
filePath := filepath.ToSlash(schemaName)
if runtime.GOOS == "windows" && !strings.HasPrefix(filePath, "/") {
filePath = "/" + filePath
}
return &url.URL{Scheme: "file", Path: filePath}
}
func fsSchemaResourceURL(schemaName string) *url.URL {
return &url.URL{
Scheme: "promptkit-schema",
Path: "/" + strings.TrimPrefix(path.Clean(schemaName), "/"),
}
}
func newSchemaCompiler(loader jsonschema.URLLoader) *jsonschema.Compiler {
@@ -427,6 +497,35 @@ func newSchemaCompiler(loader jsonschema.URLLoader) *jsonschema.Compiler {
return compiler
}
type schemaExecutor interface {
Validate(instance any) error
}
func compileJSONSchema(ctx context.Context, resourceURL string, compile func(string) (*jsonschema.Schema, error)) (*jsonschema.Schema, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
schema, compileErr := compile(resourceURL)
if err := ctx.Err(); err != nil {
return nil, err
}
return schema, compileErr
}
func executeJSONSchema(ctx context.Context, schema schemaExecutor, instance any) ([]string, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
validationErr := schema.Validate(instance)
if err := ctx.Err(); err != nil {
return nil, err
}
if validationErr != nil {
return []string{fmt.Sprintf("json schema validation failed: %v", validationErr)}, nil
}
return nil, nil
}
func validateSchemaDialect(doc any) error {
object, ok := doc.(map[string]any)
if !ok {
@@ -447,19 +546,33 @@ func validateSchemaDialect(doc any) error {
}
type standardSchemaLoader struct {
ctx context.Context
root string
}
func (l standardSchemaLoader) Load(resourceURL string) (any, error) {
if err := l.ctx.Err(); err != nil {
return nil, err
}
parsed, err := url.Parse(resourceURL)
if err != nil || parsed.Scheme != "file" || parsed.Host != "" || parsed.RawQuery != "" || parsed.Opaque != "" {
return nil, fmt.Errorf("schema reference %q is not a contained file reference", resourceURL)
}
fileName, err := (jsonschema.FileLoader{}).ToFile(resourceURL)
if err != nil {
return nil, fmt.Errorf("schema reference %q is not a contained file reference: %w", resourceURL, err)
}
resolved, err := containedFilesystemPath(l.root, fileName)
if err != nil {
if contextErr := l.ctx.Err(); contextErr != nil {
return nil, contextErr
}
return nil, err
}
return loadJSONSchemaFile(resolved)
if err := l.ctx.Err(); err != nil {
return nil, err
}
return loadJSONSchemaFile(l.ctx, resolved)
}
func containedFilesystemPath(root, name string) (string, error) {
@@ -485,38 +598,94 @@ func containedFilesystemPath(root, name string) (string, error) {
return candidate, nil
}
func loadJSONSchemaFile(name string) (any, error) {
raw, err := os.ReadFile(name)
func loadJSONSchemaFile(ctx context.Context, name string) (any, error) {
raw, err := readSchemaFile(ctx, func() (fs.File, error) {
return os.Open(name)
})
if err != nil {
return nil, err
}
var doc any
if err := json.Unmarshal(raw, &doc); err != nil {
doc, err := decodeJSONValue(ctx, raw)
if err != nil {
return nil, err
}
if err := validateSchemaDialect(doc); err != nil {
if contextErr := ctx.Err(); contextErr != nil {
return nil, contextErr
}
return nil, err
}
if err := ctx.Err(); err != nil {
return nil, err
}
return doc, nil
}
func readSchemaFile(ctx context.Context, open func() (fs.File, error)) ([]byte, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
file, openErr := open()
if err := ctx.Err(); err != nil {
if file != nil {
_ = file.Close()
}
return nil, err
}
if openErr != nil {
if file != nil {
_ = file.Close()
}
return nil, openErr
}
if file == nil {
return nil, errors.New("schema source returned a nil file")
}
defer file.Close()
var contents []byte
chunk := make([]byte, schemaReadChunkSize)
for {
if err := ctx.Err(); err != nil {
return nil, err
}
n, readErr := file.Read(chunk)
if n > 0 {
contents = append(contents, chunk[:n]...)
}
if err := ctx.Err(); err != nil {
return nil, err
}
if errors.Is(readErr, io.EOF) {
return contents, nil
}
if readErr != nil {
return nil, readErr
}
if n == 0 {
return nil, io.ErrNoProgress
}
}
}
type fsSchemaLoader struct {
ctx context.Context
fsys fs.FS
root string
}
func (l fsSchemaLoader) Load(resourceURL string) (any, error) {
if err := l.ctx.Err(); err != nil {
return nil, err
}
parsed, err := url.Parse(resourceURL)
if err != nil {
return nil, fmt.Errorf("invalid schema reference %q: %w", resourceURL, err)
}
if parsed.Scheme != "promptkit-schema" || parsed.Host != "" {
if parsed.Scheme != "promptkit-schema" || parsed.Host != "" || parsed.RawQuery != "" || parsed.Opaque != "" {
return nil, fmt.Errorf("schema reference %q is not allowed", resourceURL)
}
name, err := url.PathUnescape(strings.TrimPrefix(parsed.Path, "/"))
if err != nil {
return nil, fmt.Errorf("invalid schema reference %q: %w", resourceURL, err)
}
name := strings.TrimPrefix(parsed.Path, "/")
name = path.Clean(name)
if l.root == "." {
if strings.HasPrefix(name, "../") || name == ".." {
@@ -528,21 +697,35 @@ func (l fsSchemaLoader) Load(resourceURL string) (any, error) {
rootInfo, err := fs.Stat(l.fsys, l.root)
if err != nil {
if contextErr := l.ctx.Err(); contextErr != nil {
return nil, contextErr
}
return nil, err
}
if err := l.ctx.Err(); err != nil {
return nil, err
}
if !rootInfo.IsDir() && name != l.root {
return nil, fmt.Errorf("schema reference %q is outside the configured schema file", resourceURL)
}
raw, err := fs.ReadFile(l.fsys, name)
raw, err := readSchemaFile(l.ctx, func() (fs.File, error) {
return l.fsys.Open(name)
})
if err != nil {
return nil, err
}
var doc any
if err := json.Unmarshal(raw, &doc); err != nil {
doc, err := decodeJSONValue(l.ctx, raw)
if err != nil {
return nil, err
}
if err := validateSchemaDialect(doc); err != nil {
if contextErr := l.ctx.Err(); contextErr != nil {
return nil, contextErr
}
return nil, err
}
if err := l.ctx.Err(); err != nil {
return nil, err
}
return doc, nil

View File

@@ -3,9 +3,11 @@ package validate
import (
"context"
"encoding/json"
"net/url"
"os"
"path/filepath"
"reflect"
"runtime"
"strconv"
"strings"
"testing"
@@ -93,6 +95,51 @@ func TestStandardValidatorJSONFailure(t *testing.T) {
}
}
func TestJSONValidationChecksCompleteSyntaxWithoutChangingArtifact(t *testing.T) {
tests := []struct {
name string
body string
wantValid bool
}{
{name: "ordinary object", body: `{"count":2,"ok":true}`, wantValid: true},
{name: "integer at exact float boundary", body: `9007199254740992`, wantValid: true},
{name: "integer beyond exact float boundary", body: `9007199254740993`, wantValid: true},
{name: "large exponent", body: `1e400`, wantValid: true},
{name: "precise decimal", body: `0.123456789012345678901234567890`, wantValid: true},
{name: "surrounding whitespace", body: " \n [1,2,3] \t", wantValid: true},
{name: "malformed document", body: `{"count":`, wantValid: false},
{name: "trailing value", body: `1 2`, wantValid: false},
}
validator := NewStandardValidator("")
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
body := []byte(tc.body)
before := append([]byte(nil), body...)
artifact := &domain.Artifact{Body: body}
result, err := validator.Validate(context.Background(), artifact, domain.OutputContract{
ValidationMode: domain.ValidationJSON,
})
if err != nil {
t.Fatalf("validate JSON: %v", err)
}
if result.IsValid != tc.wantValid {
t.Fatalf("valid = %v, want %v; result=%+v", result.IsValid, tc.wantValid, result)
}
if tc.wantValid && result.Status != domain.ValidationPassed {
t.Fatalf("status = %q, want %q", result.Status, domain.ValidationPassed)
}
if !tc.wantValid && result.Status != domain.ValidationFailed {
t.Fatalf("status = %q, want %q", result.Status, domain.ValidationFailed)
}
if !reflect.DeepEqual(artifact.Body, before) {
t.Fatalf("artifact body changed: got %q, want %q", artifact.Body, before)
}
})
}
}
func TestStandardValidatorJSONSchemaSuccess(t *testing.T) {
tmp := t.TempDir()
schemaPath := filepath.Join(tmp, "schema.json")
@@ -285,55 +332,6 @@ func TestStandardValidatorJSONSchemaCompilationError(t *testing.T) {
}
}
func TestStandardValidatorLoadSchemaDocumentSuccess(t *testing.T) {
tmp := t.TempDir()
if err := os.WriteFile(filepath.Join(tmp, "schema.json"), []byte(`{
"type": "object",
"properties": {
"name": {"type": "string"}
}
}`), 0644); err != nil {
t.Fatal(err)
}
v := NewStandardValidator(tmp)
loader, ok := v.(SchemaDocumentLoader)
if !ok {
t.Fatal("standard validator must implement SchemaDocumentLoader")
}
doc, err := loader.LoadSchemaDocument(context.Background(), "schema.json")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
obj, ok := doc.(map[string]any)
if !ok {
t.Fatalf("expected object document, got %#v", doc)
}
if obj["type"] != "object" {
t.Fatalf("expected schema type=object, got %#v", obj["type"])
}
}
func TestStandardValidatorLoadSchemaDocumentInvalidJSON(t *testing.T) {
tmp := t.TempDir()
if err := os.WriteFile(filepath.Join(tmp, "schema.json"), []byte(`{`), 0644); err != nil {
t.Fatal(err)
}
v := NewStandardValidator(tmp)
loader, ok := v.(SchemaDocumentLoader)
if !ok {
t.Fatal("standard validator must implement SchemaDocumentLoader")
}
_, err := loader.LoadSchemaDocument(context.Background(), "schema.json")
if err == nil {
t.Fatal("expected decode error")
}
}
func TestFSValidatorJSONSchemaSuccess(t *testing.T) {
v := NewFSValidator(fstest.MapFS{
"schemas/events.schema.json": &fstest.MapFile{Data: []byte(`{
@@ -416,17 +414,70 @@ func TestFSValidatorPreparedSchemaSurvivesSourceMutation(t *testing.T) {
}
}
func TestFSValidatorJSONSchemaRegistrationError(t *testing.T) {
v := NewFSValidator(fstest.MapFS{
"schemas/%zz.json": &fstest.MapFile{Data: []byte(`{"type":"object"}`)},
}, "schemas")
func TestFSValidatorEscapesSchemaResourcePath(t *testing.T) {
for _, schemaName := range []string{"%zz.json", "space name.json", "hash#.json", "query?.json", "rún.json"} {
t.Run(schemaName, func(t *testing.T) {
v := NewFSValidator(fstest.MapFS{
"schemas/" + schemaName: &fstest.MapFile{Data: []byte(`{"type":"object"}`)},
}, "schemas")
res, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{}`)}, domain.OutputContract{
res, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{}`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: schemaName,
})
if err != nil || !res.IsValid {
t.Fatalf("validate schema %q: result=%#v error=%v", schemaName, res, err)
}
})
}
}
func TestSchemaReferencesPreserveEscapedFilenames(t *testing.T) {
for _, name := range []string{"%2F.json", "space name.json", "hash#.json", "query?.json", "rún.json"} {
t.Run(name, func(t *testing.T) {
rootName := "root-" + name
childName := "child-" + name
childPath := "nested/" + childName
reference := (&url.URL{Path: childPath}).EscapedPath()
rootSchema := []byte(`{"$ref":` + strconv.Quote(reference) + `}`)
childSchema := []byte(`{"type":"integer","minimum":2}`)
t.Run("fs.FS", func(t *testing.T) {
validator := NewFSValidator(fstest.MapFS{
"schemas/" + rootName: &fstest.MapFile{Data: rootSchema},
"schemas/" + childPath: &fstest.MapFile{Data: childSchema},
}, "schemas")
assertSchemaValidation(t, validator, rootName)
})
t.Run("operating system files", func(t *testing.T) {
if runtime.GOOS == "windows" && strings.ContainsAny(rootName+childName, `<>:"/\|?*`) {
t.Skip("filename is not legal on Windows")
}
root := t.TempDir()
if err := os.Mkdir(filepath.Join(root, "nested"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(root, rootName), rootSchema, 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(root, filepath.FromSlash(childPath)), childSchema, 0o644); err != nil {
t.Fatal(err)
}
assertSchemaValidation(t, NewStandardValidator(root), rootName)
})
})
}
}
func assertSchemaValidation(t *testing.T, validator Validator, schemaPath string) {
t.Helper()
result, err := validator.Validate(context.Background(), &domain.Artifact{Body: []byte(`2`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "%zz.json",
SchemaPath: schemaPath,
})
if err == nil || !strings.Contains(err.Error(), "failed to register JSON schema") {
t.Fatalf("expected schema registration error, got result=%#v error=%v", res, err)
if err != nil || !result.IsValid {
t.Fatalf("validate schema %q: result=%#v error=%v", schemaPath, result, err)
}
}
@@ -516,25 +567,6 @@ func TestFSValidatorSingleSchemaFileUsesBaseName(t *testing.T) {
}
}
func TestFSValidatorLoadSchemaDocument(t *testing.T) {
v := NewFSValidator(fstest.MapFS{
"schemas/schema.json": &fstest.MapFile{Data: []byte(`{"type":"object"}`)},
}, "schemas")
loader, ok := v.(SchemaDocumentLoader)
if !ok {
t.Fatal("fs validator must implement SchemaDocumentLoader")
}
doc, err := loader.LoadSchemaDocument(context.Background(), "schema.json")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
obj, ok := doc.(map[string]any)
if !ok || obj["type"] != "object" {
t.Fatalf("unexpected schema document: %#v", doc)
}
}
func TestStandardValidatorJSONSchemaReferenceBoundaries(t *testing.T) {
root := t.TempDir()
if err := os.WriteFile(filepath.Join(root, "child.json"), []byte(`{
@@ -629,6 +661,225 @@ func TestFSValidatorJSONSchemaReferenceBoundaries(t *testing.T) {
}
}
func TestJSONSchemaNumericConstraintsRetainJSONPrecision(t *testing.T) {
tests := []struct {
name string
schema string
instance string
wantValid bool
}{
{
name: "const distinguishes adjacent large integers",
schema: `{"const":9007199254740993}`,
instance: `9007199254740993`,
wantValid: true,
},
{
name: "const rejects adjacent large integer",
schema: `{"const":9007199254740993}`,
instance: `9007199254740992`,
wantValid: false,
},
{
name: "const accepts exponent beyond float range",
schema: `{"const":1e400}`,
instance: `1e400`,
wantValid: true,
},
{
name: "minimum accepts precise decimal boundary",
schema: `{"type":"number","minimum":0.123456789012345678901234567890}`,
instance: `0.123456789012345678901234567890`,
wantValid: true,
},
{
name: "minimum rejects lower precise decimal",
schema: `{"type":"number","minimum":0.123456789012345678901234567890}`,
instance: `0.123456789012345678901234567889`,
wantValid: false,
},
{
name: "maximum distinguishes adjacent large integers",
schema: `{"type":"number","maximum":9007199254740992}`,
instance: `9007199254740993`,
wantValid: false,
},
{
name: "multiple of accepts exact decimal multiple",
schema: `{"type":"number","multipleOf":0.0000000000000000001}`,
instance: `0.0000000000000000003`,
wantValid: true,
},
{
name: "multiple of rejects inexact decimal multiple",
schema: `{"type":"number","multipleOf":0.0000000000000000001}`,
instance: `0.00000000000000000031`,
wantValid: false,
},
{
name: "ordinary number remains supported",
schema: `{"type":"number","minimum":1,"maximum":3}`,
instance: `2`,
wantValid: true,
},
}
for _, source := range jsonSchemaValidatorSources() {
t.Run(source.name, func(t *testing.T) {
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
validator := source.new(t, []byte(tc.schema))
result, err := validator.Validate(context.Background(), &domain.Artifact{Body: []byte(tc.instance)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "schema.json",
})
if err != nil {
t.Fatalf("validate JSON Schema instance: %v", err)
}
if result.IsValid != tc.wantValid {
t.Fatalf("valid = %v, want %v; result=%+v", result.IsValid, tc.wantValid, result)
}
})
}
})
}
}
func TestJSONSchemaDecodingRequiresOneCompleteDocument(t *testing.T) {
for _, source := range jsonSchemaValidatorSources() {
t.Run(source.name, func(t *testing.T) {
for _, schema := range []string{`{"type":`, `{} {}`} {
validator := source.new(t, []byte(schema))
_, err := validator.Validate(context.Background(), &domain.Artifact{Body: []byte(`1`)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "schema.json",
})
if err == nil {
t.Fatalf("schema %q: expected decoding error", schema)
}
}
validator := source.new(t, []byte(`{}`))
for _, instance := range []string{`{"value":`, `1 2`} {
result, err := validator.Validate(context.Background(), &domain.Artifact{Body: []byte(instance)}, domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "schema.json",
})
if err != nil {
t.Fatalf("instance %q: expected completed validation, got %v", instance, err)
}
if result.Status != domain.ValidationFailed || result.IsValid || len(result.Errors) == 0 {
t.Fatalf("instance %q: expected failed validation, got %+v", instance, result)
}
}
})
}
}
func TestPreparedSchemaDocumentsRetainExactNumbers(t *testing.T) {
const schema = `{
"const": 9007199254740993,
"minimum": 0.123456789012345678901234567890,
"maximum": 1e400,
"multipleOf": 0.0000000000000000001
}`
want := map[string]string{
"const": "9007199254740993",
"minimum": "0.123456789012345678901234567890",
"maximum": "1e400",
"multipleOf": "0.0000000000000000001",
}
for _, source := range jsonSchemaValidatorSources() {
t.Run(source.name, func(t *testing.T) {
validator := source.new(t, []byte(schema))
preparer, ok := validator.(ValidationPreparer)
if !ok {
t.Fatal("validator does not support validation preparation")
}
prepared, err := preparer.PrepareValidation(context.Background(), domain.OutputContract{
ValidationMode: domain.ValidationJSONSchema,
SchemaPath: "schema.json",
})
if err != nil {
t.Fatalf("prepare validation: %v", err)
}
document, ok := prepared.SchemaDocument().(map[string]any)
if !ok {
t.Fatalf("schema document = %#v, want object", prepared.SchemaDocument())
}
for name, wantNumber := range want {
got, ok := document[name].(json.Number)
if !ok {
t.Fatalf("schema field %q = %#v, want json.Number", name, document[name])
}
if got.String() != wantNumber {
t.Fatalf("schema field %q = %q, want %q", name, got, wantNumber)
}
}
})
}
}
type jsonSchemaValidatorSource struct {
name string
new func(*testing.T, []byte) Validator
}
func jsonSchemaValidatorSources() []jsonSchemaValidatorSource {
return []jsonSchemaValidatorSource{
{
name: "operating system files",
new: func(t *testing.T, schema []byte) Validator {
t.Helper()
root := t.TempDir()
if err := os.WriteFile(filepath.Join(root, "schema.json"), schema, 0o644); err != nil {
t.Fatalf("write schema: %v", err)
}
return NewStandardValidator(root)
},
},
{
name: "fs.FS",
new: func(t *testing.T, schema []byte) Validator {
t.Helper()
return NewFSValidator(fstest.MapFS{
"schema.json": &fstest.MapFile{Data: schema},
}, ".")
},
},
}
}
func BenchmarkJSONValidation(b *testing.B) {
largeArray := []byte(`[` + strings.Repeat(`12345678901234567890,`, 32*1024) + `0]`)
benchmarks := []struct {
name string
body []byte
}{
{name: "scalar", body: []byte(`1e400`)},
{name: "object", body: []byte(`{"name":"eris","count":9007199254740993,"enabled":true}`)},
{name: "large array", body: largeArray},
}
validator := NewStandardValidator("")
contract := domain.OutputContract{ValidationMode: domain.ValidationJSON}
for _, benchmark := range benchmarks {
b.Run(benchmark.name, func(b *testing.B) {
artifact := &domain.Artifact{Body: benchmark.body}
b.ReportAllocs()
b.SetBytes(int64(len(benchmark.body)))
b.ResetTimer()
for range b.N {
result, err := validator.Validate(context.Background(), artifact, contract)
if err != nil || !result.IsValid {
b.Fatalf("validate JSON: result=%+v error=%v", result, err)
}
}
})
}
}
func TestJSONSchemaDialectIsDraft2020(t *testing.T) {
tests := []struct {
name string
@@ -673,8 +924,8 @@ func TestJSONSchemaDialectIsDraft2020(t *testing.T) {
func assertSchemaDocument(t *testing.T, got any, expectedJSON []byte) {
t.Helper()
var expected any
if err := json.Unmarshal(expectedJSON, &expected); err != nil {
expected, err := decodeJSONValue(context.Background(), expectedJSON)
if err != nil {
t.Fatalf("decode expected schema document: %v", err)
}
if !reflect.DeepEqual(got, expected) {

View File

@@ -7,11 +7,16 @@ import (
)
// Validator validates the generated artifact based on the output contract.
// Validation checks ctx around Promptkit-controlled work and synchronous
// dependency calls. A dependency call already in progress cannot be preempted;
// after it returns, cancellation takes precedence over its result.
type Validator interface {
Validate(ctx context.Context, artifact *domain.Artifact, contract domain.OutputContract) (domain.ValidationResult, error)
}
// PreparedValidation validates artifacts against one frozen output contract.
// Its cancellation boundary is synchronous: Validate does not detach schema
// execution, and an observed context error prevents publication of a result.
type PreparedValidation interface {
Validate(ctx context.Context, artifact *domain.Artifact) (domain.ValidationResult, error)
// SchemaDocument returns the root JSON Schema document used for provider
@@ -21,11 +26,9 @@ type PreparedValidation interface {
}
// ValidationPreparer freezes validation resources for one output contract.
// Preparation reads schemas in context-checked chunks and checks ctx around
// decoding and compilation. Filesystem and compiler calls remain synchronous,
// so cancellation becomes authoritative when an in-progress call returns.
type ValidationPreparer interface {
PrepareValidation(ctx context.Context, contract domain.OutputContract) (PreparedValidation, error)
}
// SchemaDocumentLoader loads JSON schema documents using validator path semantics.
type SchemaDocumentLoader interface {
LoadSchemaDocument(ctx context.Context, schemaPath string) (any, error)
}

142
json.go
View File

@@ -2,9 +2,16 @@ package promptkit
import (
"encoding/json"
"fmt"
"math"
"time"
)
const (
minDurationMilliseconds = int64(time.Duration(math.MinInt64) / time.Millisecond)
maxDurationMilliseconds = int64(time.Duration(math.MaxInt64) / time.Millisecond)
)
// MarshalJSON implements json.Marshaler for PreparedRun. It uses RFC 3339
// timestamps, integer duration_ms, and omits zero timing values.
func (r PreparedRun) MarshalJSON() ([]byte, error) {
@@ -21,38 +28,11 @@ func (r PreparedRun) MarshalJSON() ([]byte, error) {
durationMS = &r.DurationMS
}
return json.Marshal(struct {
PromptID string `json:"prompt_id"`
PromptVersion string `json:"prompt_version,omitempty"`
PromptHash string `json:"prompt_hash,omitempty"`
SelectedProfileID string `json:"selected_profile_id"`
SelectedBackendID string `json:"selected_backend_id,omitempty"`
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
OutputContract OutputContract `json:"output_contract"`
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
InputHashes map[string]string `json:"input_hashes,omitempty"`
SessionID string `json:"session_id,omitempty"`
RenderedPromptHash string `json:"rendered_prompt_hash"`
Messages []RenderedMessage `json:"messages"`
StartTime *time.Time `json:"start_time,omitempty"`
EndTime *time.Time `json:"end_time,omitempty"`
DurationMS *int64 `json:"duration_ms,omitempty"`
}{
PromptID: r.PromptID,
PromptVersion: r.PromptVersion,
PromptHash: r.PromptHash,
SelectedProfileID: r.SelectedProfileID,
SelectedBackendID: r.SelectedBackendID,
EffectiveModelParams: r.EffectiveModelParams,
OutputContract: r.OutputContract,
StructuredOutput: r.StructuredOutput,
InputHashes: r.InputHashes,
SessionID: r.SessionID,
RenderedPromptHash: r.RenderedPromptHash,
Messages: r.Messages,
StartTime: startTime,
EndTime: endTime,
DurationMS: durationMS,
return json.Marshal(preparedRunJSON{
preparedRunJSONFields: preparedRunJSONFields(r),
StartTime: startTime,
EndTime: endTime,
DurationMS: durationMS,
})
}
@@ -74,89 +54,57 @@ func (r RunResult) MarshalJSON() ([]byte, error) {
}
return json.Marshal(runResultJSON{
RunID: r.RunID,
Artifact: r.Artifact,
RawOutput: r.RawOutput,
Validation: r.Validation,
PromptID: r.PromptID,
PromptVersion: r.PromptVersion,
PromptHash: r.PromptHash,
SessionID: r.SessionID,
RenderedPromptHash: r.RenderedPromptHash,
SelectedProfileID: r.SelectedProfileID,
SelectedBackendID: r.SelectedBackendID,
ModelName: r.ModelName,
Endpoint: r.Endpoint,
EffectiveModelParams: r.EffectiveModelParams,
InputHashes: r.InputHashes,
Usage: r.Usage,
StartTime: startTime,
EndTime: endTime,
DurationMS: durationMS,
runResultJSONFields: runResultJSONFields(r),
StartTime: startTime,
EndTime: endTime,
DurationMS: durationMS,
})
}
// UnmarshalJSON implements json.Unmarshaler for RunResult. It decodes
// duration_ms into Duration with millisecond precision.
// duration_ms into Duration with millisecond precision. A duration_ms outside
// the range representable by time.Duration returns an error without changing
// the receiver.
func (r *RunResult) UnmarshalJSON(data []byte) error {
var wire runResultJSON
if err := json.Unmarshal(data, &wire); err != nil {
return err
}
*r = RunResult{
RunID: wire.RunID,
Artifact: wire.Artifact,
RawOutput: wire.RawOutput,
Validation: wire.Validation,
PromptID: wire.PromptID,
PromptVersion: wire.PromptVersion,
PromptHash: wire.PromptHash,
SessionID: wire.SessionID,
RenderedPromptHash: wire.RenderedPromptHash,
SelectedProfileID: wire.SelectedProfileID,
SelectedBackendID: wire.SelectedBackendID,
ModelName: wire.ModelName,
Endpoint: wire.Endpoint,
EffectiveModelParams: wire.EffectiveModelParams,
InputHashes: wire.InputHashes,
Usage: wire.Usage,
Duration: time.Duration(valueOrZero(wire.DurationMS)) * time.Millisecond,
result := RunResult(wire.runResultJSONFields)
if wire.DurationMS != nil {
if *wire.DurationMS < minDurationMilliseconds || *wire.DurationMS > maxDurationMilliseconds {
return fmt.Errorf(
"decode RunResult duration_ms: %d cannot be represented as time.Duration",
*wire.DurationMS,
)
}
result.Duration = time.Duration(*wire.DurationMS) * time.Millisecond
}
if wire.StartTime != nil {
r.StartTime = *wire.StartTime
result.StartTime = *wire.StartTime
}
if wire.EndTime != nil {
r.EndTime = *wire.EndTime
result.EndTime = *wire.EndTime
}
*r = result
return nil
}
type runResultJSON struct {
RunID string `json:"run_id"`
Artifact Artifact `json:"artifact"`
RawOutput string `json:"raw_output"`
Validation ValidationResult `json:"validation"`
PromptID string `json:"prompt_id"`
PromptVersion string `json:"prompt_version,omitempty"`
PromptHash string `json:"prompt_hash,omitempty"`
SessionID string `json:"session_id,omitempty"`
RenderedPromptHash string `json:"rendered_prompt_hash"`
SelectedProfileID string `json:"selected_profile_id"`
SelectedBackendID string `json:"selected_backend_id,omitempty"`
ModelName string `json:"model_name"`
Endpoint string `json:"endpoint"`
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
InputHashes map[string]string `json:"input_hashes,omitempty"`
Usage TokenUsage `json:"usage"`
StartTime *time.Time `json:"start_time,omitempty"`
EndTime *time.Time `json:"end_time,omitempty"`
DurationMS *int64 `json:"duration_ms,omitempty"`
type preparedRunJSONFields PreparedRun
type preparedRunJSON struct {
preparedRunJSONFields
StartTime *time.Time `json:"start_time,omitempty"`
EndTime *time.Time `json:"end_time,omitempty"`
DurationMS *int64 `json:"duration_ms,omitempty"`
}
func valueOrZero(value *int64) int64 {
if value == nil {
return 0
}
return *value
type runResultJSONFields RunResult
type runResultJSON struct {
runResultJSONFields
StartTime *time.Time `json:"start_time,omitempty"`
EndTime *time.Time `json:"end_time,omitempty"`
DurationMS *int64 `json:"duration_ms,omitempty"`
}

328
json_contract_test.go Normal file
View File

@@ -0,0 +1,328 @@
package promptkit_test
import (
"encoding/json"
"math"
"reflect"
"strconv"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/promptkit"
)
func TestPreparedRunJSONContractRoundTripsAllFields(t *testing.T) {
value := fullyPopulatedPreparedRun()
payload, err := json.Marshal(value)
if err != nil {
t.Fatalf("marshal PreparedRun: %v", err)
}
object := decodeJSONObject(t, payload)
assertJSONFields(t, object,
"prompt_id",
"prompt_version",
"prompt_hash",
"selected_profile_id",
"selected_backend_id",
"effective_model_params",
"output_contract",
"structured_output",
"input_hashes",
"session_id",
"rendered_prompt_hash",
"messages",
"start_time",
"end_time",
"duration_ms",
)
var messages []map[string]json.RawMessage
if err := json.Unmarshal(object["messages"], &messages); err != nil {
t.Fatalf("decode PreparedRun messages: %v", err)
}
if len(messages) != 2 {
t.Fatalf("message count = %d, want 2", len(messages))
}
if _, ok := messages[0]["cache_control"]; !ok {
t.Fatalf("first message omitted cache_control: %s", object["messages"])
}
if _, ok := messages[1]["cache_control"]; ok {
t.Fatalf("second message included empty cache_control: %s", object["messages"])
}
var decoded promptkit.PreparedRun
if err := json.Unmarshal(payload, &decoded); err != nil {
t.Fatalf("unmarshal PreparedRun: %v", err)
}
if !reflect.DeepEqual(decoded, value) {
t.Fatalf("PreparedRun did not round trip:\ngot %#v\nwant %#v", decoded, value)
}
zeroPayload, err := json.Marshal(promptkit.PreparedRun{})
if err != nil {
t.Fatalf("marshal zero PreparedRun: %v", err)
}
zeroObject := decodeJSONObject(t, zeroPayload)
for _, field := range []string{
"prompt_version",
"prompt_hash",
"selected_backend_id",
"structured_output",
"input_hashes",
"session_id",
"start_time",
"end_time",
"duration_ms",
} {
if _, ok := zeroObject[field]; ok {
t.Fatalf("zero PreparedRun included %q: %s", field, zeroPayload)
}
}
}
func TestRunResultJSONContractRoundTripsAllFields(t *testing.T) {
value := fullyPopulatedRunResult()
payload, err := json.Marshal(value)
if err != nil {
t.Fatalf("marshal RunResult: %v", err)
}
object := decodeJSONObject(t, payload)
assertJSONFields(t, object,
"run_id",
"artifact",
"raw_output",
"validation",
"prompt_id",
"prompt_version",
"prompt_hash",
"session_id",
"rendered_prompt_hash",
"selected_profile_id",
"selected_backend_id",
"model_name",
"endpoint",
"effective_model_params",
"input_hashes",
"usage",
"start_time",
"end_time",
"duration_ms",
)
if _, ok := object["duration"]; ok {
t.Fatalf("RunResult included nanosecond duration field: %s", payload)
}
var decoded promptkit.RunResult
if err := json.Unmarshal(payload, &decoded); err != nil {
t.Fatalf("unmarshal RunResult: %v", err)
}
if !reflect.DeepEqual(decoded, value) {
t.Fatalf("RunResult did not round trip:\ngot %#v\nwant %#v", decoded, value)
}
zeroPayload, err := json.Marshal(promptkit.RunResult{})
if err != nil {
t.Fatalf("marshal zero RunResult: %v", err)
}
zeroObject := decodeJSONObject(t, zeroPayload)
for _, field := range []string{
"prompt_version",
"prompt_hash",
"session_id",
"selected_backend_id",
"input_hashes",
"start_time",
"end_time",
"duration_ms",
"duration",
} {
if _, ok := zeroObject[field]; ok {
t.Fatalf("zero RunResult included %q: %s", field, zeroPayload)
}
}
}
func TestRunResultJSONDurationMillisecondBoundaries(t *testing.T) {
maxMilliseconds := int64(time.Duration(math.MaxInt64) / time.Millisecond)
minMilliseconds := int64(time.Duration(math.MinInt64) / time.Millisecond)
for _, milliseconds := range []int64{minMilliseconds, maxMilliseconds} {
t.Run(strconv.FormatInt(milliseconds, 10), func(t *testing.T) {
var decoded promptkit.RunResult
if err := json.Unmarshal(durationPayload(milliseconds), &decoded); err != nil {
t.Fatalf("decode representable duration_ms: %v", err)
}
want := time.Duration(milliseconds) * time.Millisecond
if decoded.Duration != want {
t.Fatalf("Duration = %v, want %v", decoded.Duration, want)
}
})
}
for _, milliseconds := range []int64{
minMilliseconds - 1,
maxMilliseconds + 1,
math.MinInt64,
math.MaxInt64,
} {
t.Run(strconv.FormatInt(milliseconds, 10), func(t *testing.T) {
original := fullyPopulatedRunResult()
decoded := original
err := json.Unmarshal(durationPayload(milliseconds), &decoded)
if err == nil || !strings.Contains(err.Error(), "duration_ms") || !strings.Contains(err.Error(), "time.Duration") {
t.Fatalf("overflow error = %v", err)
}
if !reflect.DeepEqual(decoded, original) {
t.Fatalf("failed decode partially updated receiver:\ngot %#v\nwant %#v", decoded, original)
}
})
}
}
func fullyPopulatedPreparedRun() promptkit.PreparedRun {
start := time.Date(2026, time.August, 11, 12, 13, 14, 150_000_000, time.UTC)
return promptkit.PreparedRun{
PromptID: "prompt.prepared",
PromptVersion: "2.1.0",
PromptHash: "prompt-hash",
SelectedProfileID: "profile-prepared",
SelectedBackendID: "backend-prepared",
EffectiveModelParams: jsonContractExecutionTarget(),
OutputContract: promptkit.OutputContract{
Format: promptkit.FormatJSON,
ValidationMode: promptkit.ValidationJSONSchema,
SchemaPath: "schemas/prepared.json",
RepairAttempts: 2,
},
StructuredOutput: &promptkit.StructuredOutputSpec{
Type: promptkit.StructuredOutputJSONSchema,
JSONSchema: &promptkit.StructuredOutputJSONSpec{
Name: "prepared_schema",
Strict: true,
Schema: map[string]any{
"type": "object",
"required": []any{"value"},
"properties": map[string]any{
"value": map[string]any{"minimum": float64(1)},
},
},
},
},
InputHashes: map[string]string{"first": "hash-1", "second": "hash-2"},
SessionID: "session-prepared",
RenderedPromptHash: "rendered-hash",
Messages: []promptkit.RenderedMessage{
{
Role: "system",
Content: "System message",
CacheControl: &promptkit.CacheControl{
Type: promptkit.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "User message"},
},
StartTime: start,
EndTime: start.Add(1501 * time.Millisecond),
DurationMS: 1501,
}
}
func fullyPopulatedRunResult() promptkit.RunResult {
start := time.Date(2026, time.August, 11, 15, 16, 17, 250_000_000, time.UTC)
return promptkit.RunResult{
RunID: "run-id",
Artifact: promptkit.Artifact{
Name: "result.json",
ContentType: "application/json",
Body: []byte(`{"value":2}`),
URI: "memory://result.json",
Size: 11,
Hash: "artifact-hash",
},
RawOutput: `{"value":2}`,
Validation: promptkit.ValidationResult{
Status: promptkit.ValidationFailed,
Mode: promptkit.ValidationJSONSchema,
Errors: []string{"first error", "second error"},
SchemaPath: "schemas/result.json",
RepairAttempts: 2,
IsValid: false,
},
PromptID: "prompt.result",
PromptVersion: "3.2.1",
PromptHash: "result-prompt-hash",
SessionID: "session-result",
RenderedPromptHash: "result-rendered-hash",
SelectedProfileID: "profile-result",
SelectedBackendID: "backend-result",
ModelName: "model-result",
Endpoint: "https://result.example/v1",
EffectiveModelParams: jsonContractExecutionTarget(),
InputHashes: map[string]string{"input": "input-hash"},
Usage: promptkit.TokenUsage{
PromptTokens: 101,
CompletionTokens: 202,
TotalTokens: 303,
CachedTokens: 44,
CacheWriteTokens: 55,
},
StartTime: start,
EndTime: start.Add(1750 * time.Millisecond),
Duration: 1750 * time.Millisecond,
}
}
func jsonContractExecutionTarget() promptkit.ExecutionTarget {
return promptkit.ExecutionTarget{
BackendID: "backend-target",
Endpoint: "https://target.example/v1",
Model: "model-target",
Temperature: 0.75,
MaxTokens: 321,
TopP: 0.875,
TimeoutSeconds: 43,
ServiceTier: "priority",
ReasoningEffort: "high",
APIKeyEnv: "PROMPTKIT_JSON_CONTRACT_KEY",
ExtraParams: map[string]any{
"enabled": true,
"weight": float64(1.25),
"nested": map[string]any{"name": "value"},
},
}
}
func durationPayload(milliseconds int64) []byte {
return []byte(`{"duration_ms":` + strconv.FormatInt(milliseconds, 10) + `}`)
}
func decodeJSONObject(t *testing.T, payload []byte) map[string]json.RawMessage {
t.Helper()
var object map[string]json.RawMessage
if err := json.Unmarshal(payload, &object); err != nil {
t.Fatalf("decode JSON object: %v", err)
}
return object
}
func assertJSONFields(t *testing.T, object map[string]json.RawMessage, fields ...string) {
t.Helper()
want := make(map[string]struct{}, len(fields))
for _, field := range fields {
want[field] = struct{}{}
}
for field := range object {
if _, ok := want[field]; !ok {
t.Errorf("unexpected JSON field %q", field)
}
}
for field := range want {
if _, ok := object[field]; !ok {
t.Errorf("missing JSON field %q", field)
}
}
}

View File

@@ -0,0 +1,96 @@
package promptkit
import (
"context"
"reflect"
"strconv"
"sync"
"testing"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
)
func TestPublicLLMClientAdapterGivesClientOwnedNestedValues(t *testing.T) {
source := adapterOwnershipRequest()
want := adapterOwnershipRequest()
client := &retainingMutatingLLMClient{}
adapter := publicLLMClientAdapter{client: client}
if _, err := adapter.Generate(context.Background(), source); err != nil {
t.Fatalf("generate: %v", err)
}
if !reflect.DeepEqual(source, want) {
t.Fatalf("client mutation changed prepared source:\ngot %#v\nwant %#v", source, want)
}
var mutations sync.WaitGroup
mutations.Add(1)
go func() {
defer mutations.Done()
for i := 0; i < 10_000; i++ {
mutateGenerateRequest(&client.retained, strconv.Itoa(i))
}
}()
for i := 0; i < 10_000; i++ {
laterRequest := fromDomainGenerateRequest(source)
if laterRequest.Prompt.Messages[0].Content != "source-message" ||
laterRequest.Prompt.Messages[0].CacheControl.TTL != "source-ttl" ||
laterRequest.Target.ExtraParams["nested"].([]any)[0] != "source-extra" ||
laterRequest.StructuredOutput.JSONSchema.Schema.(map[string]any)["enum"].([]any)[0] != "source-schema" {
t.Fatal("retained client mutation reached a later execution request")
}
}
mutations.Wait()
if !reflect.DeepEqual(source, want) {
t.Fatalf("retained client mutation changed prepared source:\ngot %#v\nwant %#v", source, want)
}
}
type retainingMutatingLLMClient struct {
retained GenerateRequest
}
func (c *retainingMutatingLLMClient) Generate(_ context.Context, request GenerateRequest) (*GenerateResponse, error) {
c.retained = request
mutateGenerateRequest(&c.retained, "client-mutation")
return &GenerateResponse{Content: "generated"}, nil
}
func adapterOwnershipRequest() domain.GenerateRequest {
return domain.GenerateRequest{
Prompt: domain.RenderedPrompt{
SessionID: "source-session",
Messages: []domain.RenderedMessage{{
Role: "user",
Content: "source-message",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
TTL: "source-ttl",
},
}},
},
Target: domain.ExecutionTarget{
Model: "source-model",
APIKey: "source-api-key",
ExtraParams: map[string]any{
"nested": []any{"source-extra"},
},
},
StructuredOutput: &domain.StructuredOutputSpec{
Type: domain.StructuredOutputJSONSchema,
JSONSchema: &domain.StructuredOutputJSONSpec{
Name: "source-schema-name",
Strict: true,
Schema: map[string]any{"enum": []any{"source-schema"}},
},
},
}
}
func mutateGenerateRequest(request *GenerateRequest, value string) {
request.Prompt.Messages[0].Content = value
request.Prompt.Messages[0].CacheControl.TTL = value
request.Target.ExtraParams["nested"].([]any)[0] = value
request.StructuredOutput.JSONSchema.Schema.(map[string]any)["enum"].([]any)[0] = value
}

View File

@@ -0,0 +1,48 @@
package promptkit_test
import (
"context"
"errors"
"testing"
"gitea.maximumdirect.net/eric/promptkit"
)
func TestPreparationRejectsInvalidOutputContractWithPublicError(t *testing.T) {
engine, err := promptkit.NewEngine(
promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
promptkit.WithProfiles(promptkit.Profile{
ID: "profile",
Endpoint: "http://example.test/v1",
Model: "model",
}),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
req := promptkit.RunRequest{
PromptID: "prompt",
Validation: &promptkit.OutputContract{
Format: promptkit.OutputFormat("binary"),
ValidationMode: promptkit.ValidationNone,
},
}
prepared, err := engine.Prepare(context.Background(), req)
if prepared != nil {
t.Fatalf("expected no partial prepared run, got %+v", prepared)
}
if !errors.Is(err, promptkit.ErrInvalidRequest) {
t.Fatalf("prepare error = %v, want ErrInvalidRequest", err)
}
preparedExecution, err := engine.PrepareExecution(context.Background(), req)
if preparedExecution != nil {
t.Fatalf("expected no partial prepared execution, got %+v", preparedExecution)
}
if !errors.Is(err, promptkit.ErrInvalidRequest) {
t.Fatalf("prepare execution error = %v, want ErrInvalidRequest", err)
}
}

View File

@@ -44,12 +44,12 @@ func (p *PreparedExecution) Discard() {
// String returns a constant representation that exposes no retained request,
// rendered content, or credential data.
func (p *PreparedExecution) String() string {
func (p PreparedExecution) String() string {
return preparedExecutionString
}
// GoString returns a constant Go-syntax representation that exposes no
// retained request, rendered content, or credential data.
func (p *PreparedExecution) GoString() string {
func (p PreparedExecution) GoString() string {
return preparedExecutionString
}

View File

@@ -409,14 +409,35 @@ func TestPreparedExecutionDiscardAndFormattingDoNotExposePrivateState(t *testing
t.Fatalf("prepare execution: %v", err)
}
formattedValues := []string{
fmt.Sprint(prepared),
fmt.Sprintf("%+v", prepared),
fmt.Sprintf("%#v", prepared),
copied := *prepared
zeroValue := promptkit.PreparedExecution{}
var nilHandle *promptkit.PreparedExecution
for name, value := range map[string]any{
"original pointer": prepared,
"copied value": copied,
"zero value": zeroValue,
"zero pointer": &zeroValue,
} {
for format, formatted := range map[string]string{
"String": fmt.Sprintf("%s", value),
"GoString": fmt.Sprintf("%#v", value),
"v": fmt.Sprintf("%v", value),
"+v": fmt.Sprintf("%+v", value),
} {
if formatted != "promptkit.PreparedExecution{opaque}" {
t.Fatalf("%s %s formatting = %q, want opaque representation", name, format, formatted)
}
assertPreparedPrivateValuesAbsent(t, formatted, directCredential, renderedContent)
}
}
for _, formatted := range formattedValues {
if formatted != "promptkit.PreparedExecution{opaque}" {
t.Fatalf("unexpected opaque formatting: %q", formatted)
for format, formatted := range map[string]string{
"String": fmt.Sprintf("%s", nilHandle),
"GoString": fmt.Sprintf("%#v", nilHandle),
"v": fmt.Sprintf("%v", nilHandle),
"+v": fmt.Sprintf("%+v", nilHandle),
} {
if formatted != "<nil>" {
t.Fatalf("nil pointer %s formatting = %q, want <nil>", format, formatted)
}
assertPreparedPrivateValuesAbsent(t, formatted, directCredential, renderedContent)
}
@@ -433,27 +454,9 @@ func TestPreparedExecutionDiscardAndFormattingDoNotExposePrivateState(t *testing
}
assertPreparedPrivateValuesAbsent(t, string(detailsJSON), directCredential)
prepared.Discard()
prepared.Discard()
result, lifecycleErr := engine.RunPrepared(context.Background(), prepared)
if result != nil || !errors.Is(lifecycleErr, promptkit.ErrInvalidRequest) {
t.Fatalf("discarded execution result=(%+v, %v), want ErrInvalidRequest", result, lifecycleErr)
}
assertPreparedPrivateValuesAbsent(t, lifecycleErr.Error(), directCredential, renderedContent)
if !reflect.DeepEqual(prepared.Details(), detailsBefore) {
t.Fatal("details changed after discard")
}
executed, err := engine.PrepareExecution(context.Background(), promptkit.RunRequest{
PromptID: "prepared",
APIKey: directCredential,
})
executionResult, err := engine.RunPrepared(context.Background(), &copied)
if err != nil {
t.Fatalf("prepare execution for request inspection: %v", err)
}
executionResult, err := engine.RunPrepared(context.Background(), executed)
if err != nil {
t.Fatalf("run execution for request inspection: %v", err)
t.Fatalf("run copied execution after formatting: %v", err)
}
requests := client.snapshot()
if len(requests) != 1 || requests[0].APIKey != directCredential {
@@ -478,16 +481,34 @@ func TestPreparedExecutionDiscardAndFormattingDoNotExposePrivateState(t *testing
}
assertPreparedPrivateValuesAbsent(t, string(resultJSON), directCredential)
var nilHandle *promptkit.PreparedExecution
nilHandle.Discard()
if !reflect.DeepEqual(nilHandle.Details(), promptkit.PreparedRun{}) {
t.Fatalf("nil handle details=%+v, want zero value", nilHandle.Details())
}
zeroHandle := &promptkit.PreparedExecution{}
zeroHandle := &zeroValue
zeroHandle.Discard()
if !reflect.DeepEqual(zeroHandle.Details(), promptkit.PreparedRun{}) {
t.Fatalf("zero handle details=%+v, want zero value", zeroHandle.Details())
}
discarded, err := engine.PrepareExecution(context.Background(), promptkit.RunRequest{
PromptID: "prepared",
APIKey: directCredential,
})
if err != nil {
t.Fatalf("prepare execution for discard: %v", err)
}
discardedDetails := discarded.Details()
discarded.Discard()
discarded.Discard()
result, lifecycleErr := engine.RunPrepared(context.Background(), discarded)
if result != nil || !errors.Is(lifecycleErr, promptkit.ErrInvalidRequest) {
t.Fatalf("discarded execution result=(%+v, %v), want ErrInvalidRequest", result, lifecycleErr)
}
assertPreparedPrivateValuesAbsent(t, lifecycleErr.Error(), directCredential, renderedContent)
if !reflect.DeepEqual(discarded.Details(), discardedDetails) {
t.Fatal("details changed after discard")
}
}
func TestPreparedExecutionCredentialCapacityAndTimingBoundaries(t *testing.T) {

View File

@@ -100,33 +100,34 @@ func toDomainProfile(publicProfile Profile) (domain.ExecutionProfile, error) {
APIKeyRequired: publicProfile.APIKeyRequired,
ExtraParams: extraParams,
}
if err := validatePublicProfile(prof); err != nil {
if err := normalizeAndValidatePublicProfile(&prof); err != nil {
return domain.ExecutionProfile{}, err
}
return prof, nil
}
func validatePublicProfile(prof domain.ExecutionProfile) error {
func normalizeAndValidatePublicProfile(prof *domain.ExecutionProfile) error {
if strings.TrimSpace(prof.ID) == "" {
return errors.New("id is required")
}
if strings.TrimSpace(prof.BackendID) == "" && strings.TrimSpace(prof.Endpoint) == "" {
prof.Endpoint = strings.TrimSpace(prof.Endpoint)
if strings.TrimSpace(prof.BackendID) == "" && prof.Endpoint == "" {
return errors.New("backend or endpoint is required")
}
if prof.Endpoint != "" {
endpoint, err := domain.NormalizeOpenAICompatibleBaseEndpoint(prof.Endpoint)
if err != nil {
return err
}
prof.Endpoint = endpoint
}
if strings.TrimSpace(prof.Model) == "" {
return errors.New("model is required")
}
if prof.Temperature < 0 || prof.Temperature > 2 {
return errors.New("temperature must be between 0 and 2")
}
if prof.MaxTokens < 0 {
return errors.New("max_tokens must be greater than or equal to 0")
}
if prof.TopP < 0 || prof.TopP > 1 {
return errors.New("top_p must be between 0 and 1")
}
if prof.TimeoutSeconds < 0 {
return errors.New("timeout_seconds must be greater than or equal to 0")
}
return nil
return domain.ValidateExecutionTargetSettings(domain.ExecutionTarget{
Temperature: prof.Temperature,
MaxTokens: prof.MaxTokens,
TopP: prof.TopP,
TimeoutSeconds: prof.TimeoutSeconds,
})
}

View File

@@ -12,7 +12,6 @@ import (
"sync/atomic"
"testing"
"testing/fstest"
"time"
"gitea.maximumdirect.net/eric/promptkit"
)
@@ -78,6 +77,41 @@ api_key_env: PROMPTKIT_INSPECTION_ABSENT_KEY
}
}
func TestFileProfileNormalizedIDMatchesInspectionAndPreparation(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prompt", "normalized-profile", "message"), "."),
promptkit.WithProfileFS(fstest.MapFS{
"profile.yaml": &fstest.MapFile{Data: []byte(`
id: " normalized-profile "
endpoint: http://profile.example/v1
model: normalized-model
`)},
}, "."),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
inspection, err := engine.InspectProfile(context.Background(), " normalized-profile ")
if err != nil {
t.Fatalf("inspect normalized profile: %v", err)
}
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
PromptID: "prompt",
ProfileID: " normalized-profile ",
})
if err != nil {
t.Fatalf("prepare with normalized profile: %v", err)
}
if inspection.ProfileID != "normalized-profile" ||
prepared.SelectedProfileID != inspection.ProfileID ||
inspection.EffectiveModelParams.Model != "normalized-model" ||
prepared.EffectiveModelParams.Model != inspection.EffectiveModelParams.Model {
t.Fatalf("inspection=%#v prepared=%#v", inspection, prepared)
}
}
func TestInspectProfilePreservesPublicErrorIdentities(t *testing.T) {
newEngine := func(t *testing.T, options ...promptkit.Option) *promptkit.Engine {
t.Helper()
@@ -110,7 +144,7 @@ func TestInspectProfilePreservesPublicErrorIdentities(t *testing.T) {
}
malformed := newEngine(t, promptkit.WithProfileFS(fstest.MapFS{
"broken.yaml": &fstest.MapFile{Data: []byte("id: broken\nendpoint: http://broken.example/v1\nmodel: model\nunknown: value\n")},
"broken.yaml": &fstest.MapFile{Data: []byte("id: broken\nendpoint: http://broken.example/v1\nmodel: model\nextra_params:\n invalid: .nan\n")},
}, "."))
if result, err := malformed.InspectProfile(context.Background(), "broken"); result != nil ||
!errors.Is(err, promptkit.ErrProfileLoad) {
@@ -391,18 +425,6 @@ output:
}
}
func TestPreparedRunJSONOmitsZeroTimingValues(t *testing.T) {
payload, err := json.Marshal(promptkit.PreparedRun{})
if err != nil {
t.Fatalf("marshal prepared run: %v", err)
}
for _, field := range []string{"start_time", "end_time", "duration_ms"} {
if strings.Contains(string(payload), `"`+field+`"`) {
t.Fatalf("expected zero %s to be omitted, got %s", field, payload)
}
}
}
func TestBackendIdentityJSONNamesAndOmission(t *testing.T) {
t.Run("execution target round trip", func(t *testing.T) {
value := promptkit.ExecutionTarget{BackendID: promptkit.BackendOpenRouter}
@@ -757,6 +779,7 @@ func TestBackendRegistrationRejectsInvalidAndDuplicateDefinitions(t *testing.T)
{name: "reserved extra parameter", backends: []promptkit.Backend{{ID: "custom", Endpoint: "http://example.test/v1", ExtraParams: map[string]any{"model": "override"}}}},
{name: "cyclic extra parameter", backends: []promptkit.Backend{{ID: "custom", Endpoint: "http://example.test/v1", ExtraParams: cycle}}},
{name: "malformed JSON number", backends: []promptkit.Backend{{ID: "custom", Endpoint: "http://example.test/v1", ExtraParams: map[string]any{"value": json.Number("01")}}}},
{name: "excessively deep extra parameter", backends: []promptkit.Backend{{ID: "custom", Endpoint: "http://example.test/v1", ExtraParams: map[string]any{"value": excessivelyDeepJSONValue()}}}},
{name: "duplicate consumer id", backends: []promptkit.Backend{
{ID: " custom ", Endpoint: "http://one.example/v1"},
{ID: "custom", Endpoint: "http://two.example/v1"},
@@ -818,92 +841,6 @@ func TestBackendExtraParamsAreDeeplyCopiedAtConstructionAndLookup(t *testing.T)
}
}
func TestPreparedRunJSONTimingRoundTrips(t *testing.T) {
start := time.Date(2026, time.July, 29, 12, 0, 0, 0, time.UTC)
prepared := promptkit.PreparedRun{
PromptID: "prompt",
StartTime: start,
EndTime: start.Add(1250 * time.Millisecond),
DurationMS: 1250,
}
payload, err := json.Marshal(prepared)
if err != nil {
t.Fatalf("marshal prepared run: %v", err)
}
var decoded promptkit.PreparedRun
if err := json.Unmarshal(payload, &decoded); err != nil {
t.Fatalf("unmarshal prepared run: %v", err)
}
if decoded.DurationMS != prepared.DurationMS ||
!decoded.StartTime.Equal(prepared.StartTime) ||
!decoded.EndTime.Equal(prepared.EndTime) {
t.Fatalf("timing values did not round trip: got %#v, want %#v", decoded, prepared)
}
}
func TestRunResultJSONUsesMillisecondsAndRoundTrips(t *testing.T) {
start := time.Date(2026, time.July, 29, 12, 0, 0, 0, time.UTC)
result := promptkit.RunResult{
RunID: "opaque-run-id",
Artifact: promptkit.Artifact{Name: "output", ContentType: "text/plain", Body: []byte("ok")},
SessionID: "session-123",
StartTime: start,
EndTime: start.Add(1500 * time.Millisecond),
Duration: 1500 * time.Millisecond,
}
payload, err := json.Marshal(result)
if err != nil {
t.Fatalf("marshal run result: %v", err)
}
var object map[string]any
if err := json.Unmarshal(payload, &object); err != nil {
t.Fatalf("decode run result JSON: %v", err)
}
if got := object["duration_ms"]; got != float64(1500) {
t.Fatalf("expected duration_ms=1500, got %#v in %s", got, payload)
}
if _, exists := object["duration"]; exists {
t.Fatalf("unexpected nanosecond duration field in %s", payload)
}
if got := object["session_id"]; got != result.SessionID {
t.Fatalf("expected session_id=%q, got %#v in %s", result.SessionID, got, payload)
}
artifact, ok := object["artifact"].(map[string]any)
if !ok || artifact["content_type"] != "text/plain" {
t.Fatalf("expected stable artifact JSON fields, got %#v", object["artifact"])
}
var decoded promptkit.RunResult
if err := json.Unmarshal(payload, &decoded); err != nil {
t.Fatalf("unmarshal run result: %v", err)
}
if decoded.SessionID != result.SessionID ||
decoded.Duration != result.Duration ||
!decoded.StartTime.Equal(result.StartTime) ||
!decoded.EndTime.Equal(result.EndTime) {
t.Fatalf("timing values did not round trip: got %#v, want %#v", decoded, result)
}
payload, err = json.Marshal(promptkit.RunResult{})
if err != nil {
t.Fatalf("marshal zero run result: %v", err)
}
for _, field := range []string{"session_id", "start_time", "end_time", "duration_ms"} {
if strings.Contains(string(payload), `"`+field+`"`) {
t.Fatalf("expected zero %s to be omitted, got %s", field, payload)
}
}
var decodedEmpty promptkit.RunResult
if err := json.Unmarshal(payload, &decodedEmpty); err != nil {
t.Fatalf("unmarshal run result without session_id: %v", err)
}
if decodedEmpty.SessionID != "" {
t.Fatalf("expected absent session_id to decode empty, got %q", decodedEmpty.SessionID)
}
}
func TestEngineValidationIsSinglePass(t *testing.T) {
client := &fakeLLMClient{
response: &promptkit.GenerateResponse{Content: "not-json"},
@@ -1090,9 +1027,9 @@ func TestFallbackProfileSourcePrecedence(t *testing.T) {
t.Run("in-memory profiles override ordinary and fallback profiles", func(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prompt", profileID, "message"), "."),
promptkit.WithFallbackProfileFS(contractProfileFS(profileID, "fallback-model"), "."),
promptkit.WithProfileFS(contractProfileFS(profileID, "ordinary-model"), "."),
promptkit.WithProfiles(promptkit.Profile{ID: profileID, Endpoint: "http://example.test/v1", Model: "memory-model"}),
promptkit.WithProfileFS(contractProfileFS(profileID, "ordinary-model"), "."),
promptkit.WithFallbackProfileFS(contractProfileFS(profileID, "fallback-model"), "."),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
@@ -1105,8 +1042,8 @@ func TestFallbackProfileSourcePrecedence(t *testing.T) {
t.Run("ordinary filesystem source overrides fallback profile", func(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(contractPromptFS("prompt", profileID, "message"), "."),
promptkit.WithFallbackProfileFS(contractProfileFS(profileID, "fallback-model"), "."),
promptkit.WithProfileFS(contractProfileFS(profileID, "ordinary-model"), "."),
promptkit.WithFallbackProfileFS(contractProfileFS(profileID, "fallback-model"), "."),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
@@ -1116,6 +1053,21 @@ func TestFallbackProfileSourcePrecedence(t *testing.T) {
}
})
t.Run("ordinary option replaces configured directory", func(t *testing.T) {
profileDir := t.TempDir()
writePublicProfileFile(t, profileDir, profileID, "http://example.test/v1", "directory-model")
engine, err := promptkit.NewEngine(promptkit.Config{ProfileDir: profileDir},
promptkit.WithPromptFS(contractPromptFS("prompt", profileID, "message"), "."),
promptkit.WithProfileFS(contractProfileFS(profileID, "option-model"), "."),
)
if err != nil {
t.Fatalf("construct engine: %v", err)
}
if model := prepareModel(t, engine, "prompt"); model != "option-model" {
t.Fatalf("expected ordinary option profile, got %q", model)
}
})
t.Run("configured directory overrides fallback profile", func(t *testing.T) {
profileDir := t.TempDir()
writePublicProfileFile(t, profileDir, profileID, "http://example.test/v1", "directory-model")

View File

@@ -82,7 +82,8 @@ const (
// Prepare, PrepareExecution, and Run copy the request's maps, pointers, and
// nested JSON-compatible values before using them. The caller may mutate the
// request after any method returns. A successful PrepareExecution retains its
// own private execution snapshot for RunPrepared.
// own private execution snapshot for RunPrepared. Excessively deep or large
// JSON-shaped values are rejected for safety.
type RunRequest struct {
// PromptID is the required non-empty prompt identifier.
PromptID string
@@ -242,8 +243,8 @@ type ArtifactRef struct {
// URI is the file path for ArtifactRefFile and optional provenance metadata
// for ArtifactRefInline.
URI string
// Body is the content for ArtifactRefInline and is ignored for
// ArtifactRefFile.
// Body is the content for ArtifactRefInline, where an empty value is valid,
// and is ignored for ArtifactRefFile.
Body string
}
@@ -296,7 +297,8 @@ type ExecutionTarget struct {
// empty for endpoint-only profiles. It is supplied to injected LLMClient
// implementations as part of the effective target.
BackendID string `json:"backend_id,omitempty"`
// Endpoint is the model-provider base URL.
// Endpoint is the normalized absolute HTTP or HTTPS model-provider base URL.
// It has a host and no user information, query, or fragment.
Endpoint string `json:"endpoint"`
// Model is the provider model identifier.
Model string `json:"model"`
@@ -395,7 +397,9 @@ type PromptInspection struct {
// framework deadline when no higher-precedence value is present.
type ExecutionTargetOverride struct {
// Endpoint replaces the profile or backend endpoint when non-empty without
// changing the effective BackendID.
// changing the effective BackendID. Preparation trims it and requires an
// absolute HTTP or HTTPS URL with a host and no user information, query, or
// fragment.
Endpoint string
// Model replaces the profile model when non-empty.
Model string
@@ -428,7 +432,8 @@ type ExecutionTargetOverride struct {
APIKeyEnv string
// ExtraParams, when non-empty, replaces the complete profile or backend map.
// Values must be JSON-compatible: nil, booleans, finite numbers, strings,
// arrays or slices, and maps with non-empty string keys. Cycles are invalid.
// arrays or slices, and maps with non-empty string keys. Cycles and
// excessively deep or large values are invalid.
ExtraParams map[string]any
}
@@ -455,7 +460,8 @@ type Profile struct {
BackendID string
// Endpoint is the model-provider base URL. It is required only when
// BackendID is blank and otherwise overrides the backend endpoint when
// non-blank.
// non-blank. WithProfiles trims it and requires an absolute HTTP or HTTPS URL
// with a host and no user information, query, or fragment.
Endpoint string
// Model is the required non-blank provider model identifier.
Model string
@@ -479,7 +485,8 @@ type Profile struct {
APIKeyRequired bool
// ExtraParams contains provider-specific JSON-compatible values. An empty
// map inherits backend request defaults, when any. WithProfiles validates
// and deeply copies it during NewEngine.
// and deeply copies it during NewEngine. Excessively deep or large values
// are rejected for safety.
ExtraParams map[string]any
}
@@ -541,8 +548,8 @@ type ExecutionTargetPresence struct {
// does not merge fields. The public Engine validates generated output once and
// does not install an output repairer.
type OutputContract struct {
// Format selects generated artifact metadata. An empty effective value
// defaults to FormatText.
// Format selects generated artifact metadata. An empty value in a non-nil
// request replacement defaults to FormatText.
Format OutputFormat `json:"format"`
// ValidationMode selects the content check. Use one of the declared
// ValidationMode constants.
@@ -550,8 +557,8 @@ type OutputContract struct {
// SchemaPath is required when ValidationMode is ValidationJSONSchema and is
// ignored by other modes.
SchemaPath string `json:"schema_path"`
// RepairAttempts is a requested repair limit. A non-positive value requests
// no repairs. The public Engine performs no repairs even when this value is
// RepairAttempts is a non-negative requested repair limit. Zero requests no
// repairs. The public Engine performs no repairs even when this value is
// positive, so its runs report zero attempts used.
RepairAttempts int `json:"repair_attempts"`
}
@@ -697,8 +704,12 @@ type GenerateResponse struct {
// File returns a file-backed artifact reference whose URI is path.
//
// The default artifact reader opens path as a caller-selected operating-system
// path without restricting it to an application root or imposing a size limit.
// The default artifact reader accepts path only when it resolves to a regular
// operating-system file, checking that condition before and after opening it.
// It reads synchronously in bounded chunks and checks context cancellation
// before opening, before and after each read, and before returning the
// artifact; it cannot interrupt a filesystem operation already in progress.
// It does not restrict path to an application root or impose a size limit.
// Applications accepting untrusted paths must validate them before calling
// Promptkit or use [WithArtifactReader] to enforce application policy.
func File(path string) ArtifactRef {
@@ -706,13 +717,13 @@ func File(path string) ArtifactRef {
}
// Inline returns an inline artifact reference whose Body is body and whose URI
// is empty.
// is empty. An empty body is a valid, explicitly supplied input.
func Inline(body string) ArtifactRef {
return ArtifactRef{Type: ArtifactRefInline, Body: body}
}
// InlineWithURI returns an inline artifact reference with body content and uri
// provenance metadata.
// provenance metadata. An empty body is a valid, explicitly supplied input.
func InlineWithURI(uri string, body string) ArtifactRef {
return ArtifactRef{Type: ArtifactRefInline, URI: uri, Body: body}
}