Compare commits
7 Commits
a16dcdfa52
...
b5aaeb1c78
| Author | SHA1 | Date | |
|---|---|---|---|
| b5aaeb1c78 | |||
| 9171b66a41 | |||
| b4363b3b73 | |||
| 241e9d2a89 | |||
| 715fff7b72 | |||
| d627b91b4f | |||
| a67b3aa76d |
@@ -46,7 +46,7 @@ other than **version** is optional.
|
||||
| Field | Type | Default | Rules |
|
||||
| --- | --- | --- | --- |
|
||||
| **version** | integer | none | Required; must be 4. |
|
||||
| **promptkit** | object | none | Profile source configuration. |
|
||||
| **promptkit** | object | none | Profile source and optional local-backend configuration. |
|
||||
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
|
||||
| **concurrency** | object | see below | Global LLM and extraction limits. |
|
||||
| **output** | object | see below | Published output settings. |
|
||||
@@ -71,27 +71,58 @@ per-user root. An explicit empty output or debug directory is invalid.
|
||||
|
||||
## PromptKit Profiles
|
||||
|
||||
The optional **promptkit** object selects one source of profile definitions:
|
||||
The optional **promptkit** object selects one source of profile definitions and
|
||||
may register one conventional local OpenAI-compatible backend:
|
||||
|
||||
~~~yaml
|
||||
version: 4
|
||||
|
||||
promptkit:
|
||||
profile_dir: /path/to/profiles
|
||||
# profile_file: /path/to/profiles.yml
|
||||
profile_dir: ./profiles
|
||||
# profile_file: ./profiles.yml
|
||||
local_backend:
|
||||
endpoint: http://localhost:8000/v1
|
||||
concurrency_limit: 2
|
||||
~~~
|
||||
|
||||
| Field | Type | Rules |
|
||||
| --- | --- | --- |
|
||||
| **profile_dir** | string | Non-empty directory containing profile files. |
|
||||
| **profile_file** | string | Non-empty profile file. |
|
||||
| **local_backend** | object | Optional registration for the conventional PromptKit backend ID **local**. |
|
||||
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
|
||||
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
|
||||
|
||||
Set at most one of these fields. Profile IDs used by a binding must be available
|
||||
from the selected PromptKit profile source when the pipeline is resolved.
|
||||
Keep credentials out of this file: configure a profile to read its credential
|
||||
from an environment variable, then set that environment variable only in the
|
||||
run environment. PromptKit owns the profile-file format; see the
|
||||
[PromptKit upstream boundary](integrations/pkg-promptkit.md) for the pinned
|
||||
package and canonical format reference.
|
||||
Set at most one of **profile_dir** and **profile_file**. Profile IDs used by a
|
||||
binding must be available from the selected PromptKit profile source when the
|
||||
pipeline is resolved. The optional local registration may coexist with either
|
||||
profile source or with PromptKit's built-in profiles.
|
||||
|
||||
When **local_backend** is present, its endpoint is trimmed and must use HTTP or
|
||||
HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are
|
||||
allowed. User information, queries, and fragments are rejected. A zero
|
||||
**concurrency_limit** leaves the local backend unrestricted inside PromptKit;
|
||||
a positive value limits simultaneous local generations. The application-wide
|
||||
**concurrency.total_llm** limit still applies in both cases. Neither local
|
||||
backend field has an environment override. Omitting **local_backend** registers
|
||||
nothing and preserves existing built-in and endpoint-only profile behavior.
|
||||
|
||||
A file-backed PromptKit profile selects the registration by its case-sensitive
|
||||
backend ID:
|
||||
|
||||
~~~yaml
|
||||
id: local-summary
|
||||
backend: local
|
||||
model: example-model
|
||||
~~~
|
||||
|
||||
Keep credentials out of the local-backend object. A PromptKit profile may name
|
||||
its credential environment variable through `api_key_env`; set that variable
|
||||
only in the run environment. PromptKit owns the
|
||||
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/formats.md).
|
||||
The [PromptKit upstream boundary](integrations/pkg-promptkit.md) identifies the
|
||||
supported package API, and [Operations](operations.md#operational-limits)
|
||||
describes the effective concurrency layers.
|
||||
|
||||
## Migrating Version 3 Configuration
|
||||
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
# PromptKit Integration
|
||||
|
||||
Notarius pins
|
||||
[`gitea.maximumdirect.net/eric/promptkit` v0.2.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0)
|
||||
[`gitea.maximumdirect.net/eric/promptkit` v0.3.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0)
|
||||
as its in-process prompt engine. The upstream
|
||||
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0/docs/consumers/pkg-promptkit.md)
|
||||
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/consumers/pkg-promptkit.md)
|
||||
owns the public engine API, and the upstream
|
||||
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0/docs/formats.md)
|
||||
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/formats.md)
|
||||
owns prompt, profile, and schema file contracts.
|
||||
|
||||
## Supported Boundary
|
||||
@@ -18,10 +18,16 @@ Notarius relies on the root `promptkit` package to:
|
||||
a direct session ID, prompt identity, and profile selection;
|
||||
- return rendered debug material, validated structured output, selected
|
||||
profile, backend, effective model metadata, and token usage;
|
||||
- register the optional conventional `local` backend through `BackendLocal`,
|
||||
`LocalBackend`, and `WithBackend`;
|
||||
- distinguish structured-output validation failure from execution failure; and
|
||||
- identify a missing explicit profile through `ErrProfileNotFound` and backend
|
||||
admission exhaustion through `ErrCapacityExceeded`.
|
||||
|
||||
The pinned
|
||||
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/backends.go)
|
||||
owns the registration and backend-capacity contract.
|
||||
|
||||
Notarius does not use PromptKit's optional `ArtifactReader`. It materializes
|
||||
source and reference content itself and supplies owned inline artifacts at the
|
||||
adapter boundary. It also retains responsibility for pipeline retries,
|
||||
@@ -38,8 +44,9 @@ Notarius records PromptKit's selected backend ID and effective reasoning
|
||||
setting as optional run-manifest provenance. Endpoint-only profiles have no
|
||||
backend ID. Debug prompt material also retains the selected backend ID and
|
||||
PromptKit's stable lower-case `effective_model_params` JSON, which may include
|
||||
`backend_id`. Notarius production configuration does not expose user-defined
|
||||
PromptKit backend registration.
|
||||
`backend_id`. Notarius production configuration exposes one optional
|
||||
conventional `local` registration. It does not expose a general user-defined
|
||||
PromptKit backend registry. Endpoint-only profiles remain supported unchanged.
|
||||
|
||||
Notarius retains its application-wide scheduled client around the PromptKit
|
||||
adapter. PromptKit may apply a narrower limit for the selected backend;
|
||||
@@ -55,7 +62,8 @@ module assets, maps its transport-neutral completion contract, prepares and
|
||||
executes requests, validates output, records provenance, captures debug
|
||||
material, redacts errors, and preserves timeout ownership.
|
||||
[Configuration](../config.md#promptkit-profiles) defines how a Notarius
|
||||
configuration selects one PromptKit profile source.
|
||||
configuration selects one PromptKit profile source and optionally registers
|
||||
the conventional local backend.
|
||||
|
||||
PromptKit API or format changes outside this boundary are not implicitly
|
||||
supported. Updating the pinned version requires reviewing the adapter and
|
||||
|
||||
@@ -39,14 +39,20 @@ This establishes the public precedence order without giving environment input a
|
||||
second file schema. Loading and application reject malformed YAML, unsupported
|
||||
file versions, unknown fields, invalid values, and identifiers that are empty
|
||||
or collide after whitespace normalization. The file application also makes the
|
||||
effective extraction-worker default follow the effective LLM limit.
|
||||
effective extraction-worker default follow the effective LLM limit. A present
|
||||
PromptKit local-backend object requires and trims its endpoint, defaults its
|
||||
omitted concurrency limit to zero, and is copied so the parsed file model
|
||||
cannot alias the populated **Config**.
|
||||
|
||||
**Config.Validate** checks configuration-only invariants before resolution. It
|
||||
rejects incompatible profile sources, invalid state-surface values, unsupported
|
||||
concurrency settings, malformed bindings and references, invalid retries, and
|
||||
invalid pipeline, step, or lane structure. Its errors retain the closest known
|
||||
pipeline, lane, and binding context. It deliberately does not require modules
|
||||
to be registered: that requires a catalog and belongs to resolution.
|
||||
invalid pipeline, step, or lane structure. PromptKit local-backend validation
|
||||
accepts only an absolute HTTP or HTTPS endpoint with a host and no user
|
||||
information, query, or fragment, and rejects a negative local concurrency
|
||||
limit. Its errors retain the closest known pipeline, lane, and binding context.
|
||||
It deliberately does not require modules to be registered: that requires a
|
||||
catalog and belongs to resolution.
|
||||
|
||||
The exact user-selectable values and validation rules are defined in
|
||||
[Configuration](../config.md). Keep additions to the file model, an
|
||||
@@ -71,7 +77,8 @@ options, and returns the fixed ordered pipeline shape. The resulting
|
||||
changes, a clone of the input configuration, and the resolved pipeline.
|
||||
Callers may therefore retain or modify their input slices and maps without
|
||||
changing the resolved result, and later consumers cannot mutate the original
|
||||
configuration through the effective value.
|
||||
configuration through the effective value. This ownership includes the nested
|
||||
PromptKit local-backend value.
|
||||
|
||||
Resolution failures stop before module construction and source parsing. They
|
||||
include an error path for an unconfigured pipeline, missing module, missing
|
||||
@@ -94,9 +101,12 @@ Configuration summaries must use **Redacted**, **RedactedSummaryPayload**, or
|
||||
Those methods copy every binding and nested option container, replace values
|
||||
whose key is credential-shaped with **[REDACTED]**, and omit materialized
|
||||
reference content while retaining safe binding and reference provenance. The
|
||||
payload must not alias the source configuration or resolved pipeline. This
|
||||
redaction is deliberately narrow: it protects configuration summaries and does
|
||||
not authorize recording arbitrary environment values or provider requests.
|
||||
payload must not alias the source configuration or resolved pipeline.
|
||||
PromptKit's local endpoint and concurrency limit are preserved as non-secret
|
||||
configuration metadata in the independently owned summary; the object contains
|
||||
no credential value. This redaction is deliberately narrow: it protects
|
||||
configuration summaries and does not authorize recording arbitrary environment
|
||||
values or provider requests.
|
||||
|
||||
## Invariants To Preserve
|
||||
|
||||
|
||||
@@ -56,6 +56,13 @@ distinct and deterministically ordered. Endpoint-only profiles retain an empty
|
||||
backend ID, which the published JSON omits. Successful completion responses and
|
||||
recorded profile manifests identify the adapter provider as `promptkit`.
|
||||
|
||||
The CLI's preparation-only engine and the production adapter use the same
|
||||
conversion helper to register the optional conventional `local` backend.
|
||||
Preflight therefore resolves the same backend membership as runtime without
|
||||
performing generation. When the registration is absent, a profile selecting
|
||||
`backend: local` fails preparation instead of falling back to a built-in or
|
||||
endpoint-only target.
|
||||
|
||||
Before execution, the adapter also contributes a non-secret checkpoint
|
||||
fingerprint for the effective PromptKit profile source. It combines the
|
||||
identity of PromptKit's compiled-in profile catalog with a deterministic digest
|
||||
@@ -65,7 +72,11 @@ paths. It covers both explicit binding profiles and prompt-selected defaults,
|
||||
so changing a model or other profile setting cannot reuse checkpoints created
|
||||
under the prior profile source. This cache identity is independent of durable
|
||||
profile provenance: run manifests continue to list only profiles actually
|
||||
observed during LLM calls.
|
||||
observed during LLM calls. When the local backend is registered, a second
|
||||
fingerprint hashes its trimmed endpoint behind a stable marker. Changing that
|
||||
semantic execution target invalidates checkpoint reuse. The raw endpoint is not
|
||||
stored in checkpoint identity, and the local concurrency limit is excluded
|
||||
because it changes scheduling rather than execution semantics.
|
||||
|
||||
## Shared Provider-Call Limit
|
||||
|
||||
@@ -85,8 +96,10 @@ PromptKit applies a second, independent admission limit when the selected
|
||||
profile names a limited backend. It sits beneath the Notarius scheduled client,
|
||||
so it may narrow but cannot expand the application-wide limit. Built-in
|
||||
OpenRouter profiles select PromptKit's reserved backend and its upstream
|
||||
capacity policy. Endpoint-only profiles do not select a PromptKit backend and
|
||||
remain limited only by the Notarius scheduler.
|
||||
capacity policy. A positive configured local-backend limit bounds active local
|
||||
generations inside PromptKit; zero leaves that backend unlimited there.
|
||||
Endpoint-only profiles do not select a PromptKit backend and remain limited
|
||||
only by the Notarius scheduler.
|
||||
|
||||
## Prompt And Schema Assets
|
||||
|
||||
|
||||
@@ -118,8 +118,10 @@ disabled. Without **--resume**, a recording-enabled run executes normally and
|
||||
does not load checkpoint state. Compatibility includes the resolved pipeline,
|
||||
input, selected lanes, runtime overrides, reference provenance, LLM-profile
|
||||
provenance, the effective PromptKit profile-source fingerprint, and
|
||||
prepared-component fingerprints. Changing profile content causes a cold miss
|
||||
even when its profile ID is unchanged. A changed identity produces a cold miss;
|
||||
prepared-component fingerprints. When a local PromptKit backend is configured,
|
||||
compatibility also includes a non-secret fingerprint of its endpoint. Changing
|
||||
profile content or the local endpoint causes a cold miss; changing only the
|
||||
local concurrency limit does not. A changed identity produces a cold miss;
|
||||
Notarius does not migrate, rewrite, or delete older checkpoint directories.
|
||||
Reasoning-effort inheritance, replacement, and explicit clearing are distinct
|
||||
runtime identities, so checkpoints created under one state are not reused by
|
||||
@@ -217,7 +219,7 @@ Provider execution settings and the generation timeout come from the selected
|
||||
PromptKit profile. The invocation-only **--reasoning-effort** and
|
||||
**--clear-reasoning-effort** controls may replace or clear that profile setting
|
||||
for all LLM-backed calls in one run without changing the profile. PromptKit
|
||||
v0.2.0 does not add a provider retry loop. Notarius binding retries rerun the
|
||||
v0.3.0 does not add a provider retry loop. Notarius binding retries rerun the
|
||||
complete module operation and validation chain as defined by
|
||||
[module bindings](config.md#module-bindings-and-validators).
|
||||
|
||||
@@ -234,13 +236,19 @@ and validators. PromptKit may impose a narrower admission limit for the
|
||||
selected backend. The effective active-generation bound is the intersection of
|
||||
both limits and can therefore be lower than **total_llm**. Built-in OpenRouter
|
||||
profiles use PromptKit's upstream backend limit; endpoint-only profiles have no
|
||||
PromptKit backend limit and remain bounded by Notarius.
|
||||
PromptKit backend limit and remain bounded by Notarius. For the configured
|
||||
local backend, a zero **concurrency_limit** leaves only the Notarius scheduler
|
||||
as a call limit. A positive value makes the effective active local-generation
|
||||
bound the smaller of **total_llm** and that local limit.
|
||||
|
||||
When a PromptKit backend has admitted all active and queued work, a new call
|
||||
fails as capacity exhaustion before generation. The adapter does not retry it.
|
||||
The calling stage's configured retry policy applies normally, and the run fails
|
||||
if those attempts are exhausted. Caller cancellation remains authoritative.
|
||||
Configuration contracts are documented under
|
||||
For a positive local limit, PromptKit owns its default waiting capacity and
|
||||
admission behavior. When a PromptKit backend has admitted all active and queued
|
||||
work, a new call fails as capacity exhaustion before generation. The adapter
|
||||
maps that failure to Notarius's existing provider-neutral capacity error and
|
||||
does not retry it. The calling stage's configured retry policy applies
|
||||
normally, and the run fails if those attempts are exhausted. Caller
|
||||
cancellation remains authoritative. Configuration contracts are documented
|
||||
under [PromptKit profiles](config.md#promptkit-profiles) and
|
||||
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
|
||||
limits and actual provider-call limits are independent. Notarius writes local
|
||||
filesystem state only; remote storage, archival, and retention automation are
|
||||
|
||||
@@ -53,6 +53,120 @@ not as committed release dates.
|
||||
spell, combat, interaction, and scene-description lanes after real-world use.
|
||||
Add more complex chunking only in response to demonstrated failures.
|
||||
|
||||
## Cross-Cutting LLM Runtime
|
||||
|
||||
### Deterministic Prompt Session Identity
|
||||
|
||||
- Replace the source-document-ID default for prompt sessions with one
|
||||
predictable, procedurally generated session ID for the complete
|
||||
source-processing workload.
|
||||
- Preserve an explicit non-empty `--session-id` as the highest-precedence
|
||||
override. Otherwise, derive the default only from the effective input module
|
||||
identity and the exact raw input bytes.
|
||||
- Use a versioned, bounded representation such as
|
||||
`notarius:v1:<sha256(input-module + NUL + raw-input)>`. The exact encoding
|
||||
must fit PromptKit's session length contract and must not embed source
|
||||
content.
|
||||
- Keep the derived session stable across runs, pipelines, selected lanes,
|
||||
ordered steps, retries, resume, recomputation, LLM profiles, reasoning
|
||||
overrides, and output, debug, or cache settings.
|
||||
- Do not include file-backed references, generated references, reference
|
||||
contents, or the composition of a reference bundle in session derivation.
|
||||
References may change between prompt calls within one pipeline without
|
||||
changing routing affinity.
|
||||
- Resolve the authoritative session before checkpoint construction and use the
|
||||
same value for checkpoint runtime identity, every prompt-facing module,
|
||||
PromptKit's direct session field, the compatibility `session_id` prompt
|
||||
variable, run-manifest metadata, and debug metadata.
|
||||
- Keep routing identity separate from cache and checkpoint content identity.
|
||||
Exact prompt prefixes, reference contents, model settings, and other
|
||||
generation-affecting inputs must continue to participate in their existing
|
||||
hashes and checkpoint fingerprints even though they do not change the
|
||||
session.
|
||||
- Treat the generated value as a provider-visible, stable pseudonymous
|
||||
correlation identifier. Do not introduce an installation-specific HMAC or
|
||||
secret unless a concrete multi-tenant or privacy requirement justifies
|
||||
sacrificing deterministic identity across installations.
|
||||
|
||||
### Pipeline-Level LLM Profile Defaults
|
||||
|
||||
- Add an optional pipeline-level `llm_profile` default so an operator can
|
||||
select one PromptKit execution policy for the pipeline without repeating the
|
||||
same profile ID on every LLM-backed module binding.
|
||||
- Apply the following precedence consistently: an explicit run-wide
|
||||
`--llm-profile` override, then a binding-specific `llm_profile`, then the
|
||||
pipeline-level default, then the prompt definition's embedded
|
||||
`default_profile`.
|
||||
- Apply inheritance only to bindings whose resolved modules are LLM-backed,
|
||||
including applicable chunk, extraction, merge, normalization, and validation
|
||||
bindings. Do not attach an inherited profile to deterministic modules or
|
||||
weaken existing validation that rejects profiles where generation is not
|
||||
supported.
|
||||
- Resolve inherited profiles before effective-pipeline validation, digest and
|
||||
checkpoint construction, execution, and provenance capture. Validate every
|
||||
resulting explicit profile ID against the selected PromptKit profile source,
|
||||
and ensure manifests and debug output report the profile actually selected
|
||||
for each generation target.
|
||||
- Preserve binding-specific profiles as intentional exceptions for modules
|
||||
that require a different quality, latency, cost, provider, or reasoning
|
||||
policy. Preserve `--llm-profile` as the convenient highest-precedence
|
||||
experiment or incident-response override for an entire run.
|
||||
- Treat PromptKit profiles as execution-policy configuration and prefer stable,
|
||||
workload-oriented IDs such as `dnd-extraction` over model names. A pipeline
|
||||
should express the kind of work it performs rather than encode a particular
|
||||
provider, model, or deployment environment.
|
||||
- Establish deployment-managed PromptKit profile files as the recommended
|
||||
environment-specific configuration mechanism. Production, development, and
|
||||
local deployments may each define the same logical `dnd-extraction` ID with
|
||||
different model and generation settings, while retaining one unchanged
|
||||
Notarius pipeline definition.
|
||||
- Keep deployment profiles distinct from both Notarius prompt assets embedded
|
||||
in the application binary and PromptKit's built-in profile catalog. Document
|
||||
that `promptkit.profile_dir` or `promptkit.profile_file` selects an external
|
||||
filesystem source whose definitions overlay PromptKit built-ins, and
|
||||
recommend application-owned IDs rather than silently replacing built-in
|
||||
profile IDs.
|
||||
- Document an operator-friendly layout in which the Notarius configuration and
|
||||
a profile subdirectory are deployed together. Until profile paths are
|
||||
explicitly resolved relative to the configuration file, clearly state that
|
||||
relative paths use the process working directory and recommend absolute
|
||||
paths for services and containers.
|
||||
- Update the configuration reference, operations guide, PromptKit integration
|
||||
boundary, relevant internal configuration and pipeline documentation, and
|
||||
maintained D&D examples together. Include one concrete external-profile
|
||||
example and explain the profile precedence and environment-neutral pipeline
|
||||
pattern without duplicating PromptKit's complete profile-format reference.
|
||||
|
||||
### Raise The Default Application-Wide LLM Limit
|
||||
|
||||
- Raise the default `concurrency.total_llm` value from 1 to 16 so ordinary
|
||||
single-backend runs can use PromptKit's expected OpenRouter capacity and
|
||||
lower-capacity local backends without an unnecessarily narrower Notarius
|
||||
limit.
|
||||
- Keep the Notarius application-wide scheduler mandatory and require
|
||||
`total_llm` to remain a positive integer. Do not make the default unlimited:
|
||||
endpoint-only profiles, an unrestricted local backend, injected clients, and
|
||||
aggregate work across several backends may have no narrower PromptKit limit.
|
||||
- Continue defaulting `concurrency.stage_workers.extract` to the effective
|
||||
`total_llm`, making its default 16 as part of the same change. Preserve an
|
||||
explicit lower extract-worker setting when an operator wants less queued or
|
||||
concurrent extraction work.
|
||||
- Define effective provider concurrency as the intersection of the Notarius
|
||||
application-wide limit, the selected PromptKit backend limit when present,
|
||||
and the work made available by stage execution. A Notarius limit of 16 does
|
||||
not narrow a backend already limited to 16, while a local backend limited to
|
||||
4 remains bounded at 4.
|
||||
- Treat the default as an application-wide safety ceiling across profiles,
|
||||
backends, modules, retries, and validators. A run that intentionally needs
|
||||
the combined capacity of several backends may configure a higher
|
||||
`total_llm` and an appropriate extract-worker count explicitly.
|
||||
- Retain the existing configuration and environment override surfaces. Update
|
||||
canonical configuration, operations, and internal documentation together
|
||||
when the default changes.
|
||||
- Reconsider decoupling the extract-worker default from `total_llm` only after
|
||||
mixed-backend workloads demonstrate a need for a high global emergency
|
||||
ceiling with a lower default work-production rate.
|
||||
|
||||
## Shared Normalization And Quality Work
|
||||
|
||||
### Generic LLM-Assisted Deduplication
|
||||
|
||||
2
go.mod
2
go.mod
@@ -3,7 +3,7 @@ module gitea.maximumdirect.net/eric/notarius
|
||||
go 1.25.5
|
||||
|
||||
require (
|
||||
gitea.maximumdirect.net/eric/promptkit v0.2.0
|
||||
gitea.maximumdirect.net/eric/promptkit v0.3.0
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
|
||||
4
go.sum
4
go.sum
@@ -1,5 +1,5 @@
|
||||
gitea.maximumdirect.net/eric/promptkit v0.2.0 h1:6bUATKnVgLlHSbKjGb8EnCt2r7jV5Xw6t+pX7G9S/lU=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.2.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.3.0 h1:P5/GJ6fVIVsBLJx8NxoaGV34ZWeAkU8lm4Ht6aVMOFg=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.3.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
||||
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
||||
|
||||
@@ -171,6 +171,7 @@ func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID
|
||||
client, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{
|
||||
ProfileDir: cfg.PromptKit.ProfileDir,
|
||||
ProfileFile: cfg.PromptKit.ProfileFile,
|
||||
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
|
||||
Assets: assets,
|
||||
Recorder: recorder,
|
||||
ReasoningEffort: overrides.ReasoningEffort,
|
||||
|
||||
@@ -6,6 +6,8 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
@@ -13,7 +15,9 @@ import (
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||
@@ -34,6 +38,7 @@ import (
|
||||
itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents"
|
||||
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
)
|
||||
|
||||
func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
|
||||
@@ -406,6 +411,89 @@ func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestProductionLLMClientFactoryUsesConfiguredLocalBackend(t *testing.T) {
|
||||
var providerCalls atomic.Int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
providerCalls.Add(1)
|
||||
if r.URL.Path != "/v1/chat/completions" {
|
||||
t.Errorf("provider path = %q, want /v1/chat/completions", r.URL.Path)
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{
|
||||
"choices": [{"message": {"role": "assistant", "content": "{\"ok\":true}"}}],
|
||||
"usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7}
|
||||
}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
|
||||
if err := os.WriteFile(profilePath, []byte(`id: local-profile
|
||||
backend: local
|
||||
model: local-model
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
assets := llm.NewAssetRegistry()
|
||||
if err := assets.RegisterPromptFS(fstest.MapFS{
|
||||
"production.local.yaml": {Data: []byte(`id: production.local
|
||||
version: "v1"
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
messages:
|
||||
- role: user
|
||||
content: '{{ input "transcript" }}'
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json
|
||||
`)},
|
||||
}, "."); err != nil {
|
||||
t.Fatalf("register prompt assets: %v", err)
|
||||
}
|
||||
|
||||
cfg := config.Default()
|
||||
cfg.PromptKit.ProfileFile = profilePath
|
||||
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
|
||||
Endpoint: server.URL + "/v1",
|
||||
ConcurrencyLimit: 2,
|
||||
}
|
||||
client, manifests, err := productionLLMClientFactoryWithAssets(assets)(
|
||||
context.Background(),
|
||||
cfg,
|
||||
"local-profile",
|
||||
LLMRuntimeOverrides{},
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("build production LLM runtime: %v", err)
|
||||
}
|
||||
if len(manifests) != 0 {
|
||||
t.Fatalf("eager profile manifests = %#v, want none", manifests)
|
||||
}
|
||||
|
||||
var out map[string]any
|
||||
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
PromptID: "production.local",
|
||||
ProfileID: "local-profile",
|
||||
Inputs: contracts.LLMInputSet{
|
||||
"transcript": contracts.NewLLMInputMaterial("transcript", "text/plain", []byte("local request"), "", ""),
|
||||
},
|
||||
}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
||||
}
|
||||
if providerCalls.Load() != 1 {
|
||||
t.Fatalf("provider calls = %d, want 1", providerCalls.Load())
|
||||
}
|
||||
provider, ok := client.(contracts.LLMProfileManifestProvider)
|
||||
if !ok {
|
||||
t.Fatalf("production client %T does not provide profile manifests", client)
|
||||
}
|
||||
recorded := provider.LLMProfileManifests()
|
||||
if len(recorded) != 1 || recorded[0].BackendID != promptkit.BackendLocal {
|
||||
t.Fatalf("production profile manifests = %#v, want local backend", recorded)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
|
||||
t.Run("canceled context", func(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"testing/fstest"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
)
|
||||
|
||||
@@ -61,8 +62,21 @@ func newProfileValidationEngine(cfg config.Config) (*promptkit.Engine, error) {
|
||||
if cfg.PromptKit.ProfileFile != "" {
|
||||
opts = append(opts, promptkit.WithProfileFile(cfg.PromptKit.ProfileFile))
|
||||
}
|
||||
if localBackend := mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend); localBackend != nil {
|
||||
opts = append(opts, llm.PromptKitLocalBackendOption(*localBackend))
|
||||
}
|
||||
return promptkit.NewEngine(promptkit.Config{
|
||||
PromptDir: "unused",
|
||||
ProfileDir: cfg.PromptKit.ProfileDir,
|
||||
}, opts...)
|
||||
}
|
||||
|
||||
func mapPromptKitLocalBackend(cfg *config.PromptKitLocalBackendConfig) *llm.PromptKitLocalBackendConfig {
|
||||
if cfg == nil {
|
||||
return nil
|
||||
}
|
||||
return &llm.PromptKitLocalBackendConfig{
|
||||
Endpoint: cfg.Endpoint,
|
||||
ConcurrencyLimit: cfg.ConcurrencyLimit,
|
||||
}
|
||||
}
|
||||
|
||||
54
internal/cli/promptkit_profiles_test.go
Normal file
54
internal/cli/promptkit_profiles_test.go
Normal file
@@ -0,0 +1,54 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
)
|
||||
|
||||
func TestExplicitPromptKitProfileValidationUsesConfiguredLocalBackendWithoutGeneration(t *testing.T) {
|
||||
var providerCalls atomic.Int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
|
||||
providerCalls.Add(1)
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
|
||||
if err := os.WriteFile(profilePath, []byte(`id: local-profile
|
||||
backend: local
|
||||
model: local-model
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfg := config.Default()
|
||||
cfg.PromptKit.ProfileFile = profilePath
|
||||
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
|
||||
Endpoint: server.URL + "/v1",
|
||||
ConcurrencyLimit: 2,
|
||||
}
|
||||
if err := validateExplicitPromptKitProfiles(context.Background(), cfg, []string{"local-profile"}); err != nil {
|
||||
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
|
||||
}
|
||||
if providerCalls.Load() != 0 {
|
||||
t.Fatalf("provider calls during configured profile validation = %d, want 0", providerCalls.Load())
|
||||
}
|
||||
|
||||
cfg.PromptKit.LocalBackend = nil
|
||||
err := validateExplicitPromptKitProfiles(context.Background(), cfg, []string{"local-profile"})
|
||||
if err == nil ||
|
||||
!strings.Contains(err.Error(), `validate PromptKit profile "local-profile"`) ||
|
||||
!strings.Contains(err.Error(), promptkit.BackendLocal) {
|
||||
t.Fatalf("validation without registration error = %v, want profile and local backend context", err)
|
||||
}
|
||||
if providerCalls.Load() != 0 {
|
||||
t.Fatalf("provider calls after missing-registration validation = %d, want 0", providerCalls.Load())
|
||||
}
|
||||
}
|
||||
@@ -16,8 +16,14 @@ type Config struct {
|
||||
}
|
||||
|
||||
type PromptKitConfig struct {
|
||||
ProfileDir string `json:"profile_dir,omitempty"`
|
||||
ProfileFile string `json:"profile_file,omitempty"`
|
||||
ProfileDir string `json:"profile_dir,omitempty"`
|
||||
ProfileFile string `json:"profile_file,omitempty"`
|
||||
LocalBackend *PromptKitLocalBackendConfig `json:"local_backend,omitempty"`
|
||||
}
|
||||
|
||||
type PromptKitLocalBackendConfig struct {
|
||||
Endpoint string `json:"endpoint"`
|
||||
ConcurrencyLimit int `json:"concurrency_limit"`
|
||||
}
|
||||
|
||||
type ConcurrencyConfig struct {
|
||||
@@ -66,6 +72,10 @@ func Default() Config {
|
||||
|
||||
func cloneConfig(in Config) Config {
|
||||
out := in
|
||||
if in.PromptKit.LocalBackend != nil {
|
||||
localBackend := *in.PromptKit.LocalBackend
|
||||
out.PromptKit.LocalBackend = &localBackend
|
||||
}
|
||||
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
|
||||
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
|
||||
for key, profile := range in.Pipelines {
|
||||
|
||||
@@ -107,6 +107,33 @@ func TestEffectiveConfigPreservesPromptKitProfileSource(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestEffectiveConfigOwnsPromptKitLocalBackend(t *testing.T) {
|
||||
cfg := configForEffectiveTests(t, effectiveProfile())
|
||||
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
|
||||
Endpoint: "http://localhost:8000/v1",
|
||||
ConcurrencyLimit: 2,
|
||||
}
|
||||
effective, err := cfg.Resolve(ResolveInput{PipelineID: "main", Catalog: effectiveCatalog(t)})
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve() error = %v", err)
|
||||
}
|
||||
if effective.Config.PromptKit.LocalBackend == nil {
|
||||
t.Fatal("effective local backend = nil")
|
||||
}
|
||||
if effective.Config.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend {
|
||||
t.Fatal("effective local backend aliases input config")
|
||||
}
|
||||
|
||||
cfg.PromptKit.LocalBackend.Endpoint = "http://changed-input.example/v1"
|
||||
if effective.Config.PromptKit.LocalBackend.Endpoint != "http://localhost:8000/v1" {
|
||||
t.Fatalf("input mutation changed effective config: %#v", effective.Config.PromptKit.LocalBackend)
|
||||
}
|
||||
effective.Config.PromptKit.LocalBackend.ConcurrencyLimit = 9
|
||||
if cfg.PromptKit.LocalBackend.ConcurrencyLimit != 2 {
|
||||
t.Fatalf("effective mutation changed input config: %#v", cfg.PromptKit.LocalBackend)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
|
||||
@@ -23,8 +23,14 @@ type FileConfig struct {
|
||||
}
|
||||
|
||||
type FilePromptKitConfig struct {
|
||||
ProfileDir *string `yaml:"profile_dir,omitempty"`
|
||||
ProfileFile *string `yaml:"profile_file,omitempty"`
|
||||
ProfileDir *string `yaml:"profile_dir,omitempty"`
|
||||
ProfileFile *string `yaml:"profile_file,omitempty"`
|
||||
LocalBackend *FilePromptKitLocalBackendConfig `yaml:"local_backend,omitempty"`
|
||||
}
|
||||
|
||||
type FilePromptKitLocalBackendConfig struct {
|
||||
Endpoint *string `yaml:"endpoint,omitempty"`
|
||||
ConcurrencyLimit *int `yaml:"concurrency_limit,omitempty"`
|
||||
}
|
||||
|
||||
type FilePipelineProfile struct {
|
||||
@@ -468,6 +474,20 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
||||
}
|
||||
c.PromptKit.ProfileFile = value
|
||||
}
|
||||
if fileCfg.PromptKit.LocalBackend != nil {
|
||||
if fileCfg.PromptKit.LocalBackend.Endpoint == nil {
|
||||
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
|
||||
}
|
||||
endpoint := strings.TrimSpace(*fileCfg.PromptKit.LocalBackend.Endpoint)
|
||||
if endpoint == "" {
|
||||
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
|
||||
}
|
||||
localBackend := PromptKitLocalBackendConfig{Endpoint: endpoint}
|
||||
if fileCfg.PromptKit.LocalBackend.ConcurrencyLimit != nil {
|
||||
localBackend.ConcurrencyLimit = *fileCfg.PromptKit.LocalBackend.ConcurrencyLimit
|
||||
}
|
||||
c.PromptKit.LocalBackend = &localBackend
|
||||
}
|
||||
}
|
||||
|
||||
for _, pipelineID := range pipelineIDs {
|
||||
|
||||
@@ -97,6 +97,100 @@ func TestFilePromptKitProfileSourcesSurviveConfigBoundaries(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilePromptKitLocalBackendSurvivesConfigBoundaries(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
concurrencyYAML string
|
||||
wantConcurrency int
|
||||
}{
|
||||
{name: "omitted concurrency defaults to zero"},
|
||||
{name: "positive concurrency is preserved", concurrencyYAML: " concurrency_limit: 2\n", wantConcurrency: 2},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
file := parseFileConfig(t, "version: 4\npromptkit:\n local_backend:\n endpoint: ' http://localhost:8000/v1 '\n"+tt.concurrencyYAML)
|
||||
cfg := Default()
|
||||
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||
t.Fatalf("ApplyFileConfig() error = %v", err)
|
||||
}
|
||||
want := PromptKitLocalBackendConfig{
|
||||
Endpoint: "http://localhost:8000/v1",
|
||||
ConcurrencyLimit: tt.wantConcurrency,
|
||||
}
|
||||
if cfg.PromptKit.LocalBackend == nil || *cfg.PromptKit.LocalBackend != want {
|
||||
t.Fatalf("local backend config = %#v, want %#v", cfg.PromptKit.LocalBackend, want)
|
||||
}
|
||||
|
||||
*file.PromptKit.LocalBackend.Endpoint = "http://changed.example/v1"
|
||||
if file.PromptKit.LocalBackend.ConcurrencyLimit != nil {
|
||||
*file.PromptKit.LocalBackend.ConcurrencyLimit = 99
|
||||
}
|
||||
if *cfg.PromptKit.LocalBackend != want {
|
||||
t.Fatalf("effective config aliases parsed file model: %#v", cfg.PromptKit.LocalBackend)
|
||||
}
|
||||
|
||||
cloned := cloneConfig(cfg)
|
||||
if cloned.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *cloned.PromptKit.LocalBackend != want {
|
||||
t.Fatalf("cloned local backend = %#v, want detached %#v", cloned.PromptKit.LocalBackend, want)
|
||||
}
|
||||
cloned.PromptKit.LocalBackend.Endpoint = "http://clone.example/v1"
|
||||
if *cfg.PromptKit.LocalBackend != want {
|
||||
t.Fatalf("mutating clone changed source config: %#v", cfg.PromptKit.LocalBackend)
|
||||
}
|
||||
|
||||
redacted := cfg.Redacted()
|
||||
if redacted.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *redacted.PromptKit.LocalBackend != want {
|
||||
t.Fatalf("redacted local backend = %#v, want detached %#v", redacted.PromptKit.LocalBackend, want)
|
||||
}
|
||||
|
||||
data, err := json.Marshal(cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("json.Marshal() error = %v", err)
|
||||
}
|
||||
var payload struct {
|
||||
PromptKit map[string]json.RawMessage `json:"promptkit"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &payload); err != nil {
|
||||
t.Fatalf("json.Unmarshal() error = %v", err)
|
||||
}
|
||||
localJSON, ok := payload.PromptKit["local_backend"]
|
||||
if !ok {
|
||||
t.Fatalf("runtime PromptKit JSON keys = %v, want local_backend", payload.PromptKit)
|
||||
}
|
||||
var localPayload map[string]json.RawMessage
|
||||
if err := json.Unmarshal(localJSON, &localPayload); err != nil {
|
||||
t.Fatalf("unmarshal local_backend JSON: %v", err)
|
||||
}
|
||||
if _, ok := localPayload["endpoint"]; !ok {
|
||||
t.Fatalf("runtime local_backend JSON keys = %v, want endpoint", localPayload)
|
||||
}
|
||||
if _, ok := localPayload["concurrency_limit"]; !ok {
|
||||
t.Fatalf("runtime local_backend JSON keys = %v, want concurrency_limit", localPayload)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilePromptKitLocalBackendRequiresEndpoint(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
yaml string
|
||||
}{
|
||||
{name: "missing", yaml: "version: 4\npromptkit:\n local_backend: {}\n"},
|
||||
{name: "empty", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ''\n"},
|
||||
{name: "blank", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ' '\n"},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
file := parseFileConfig(t, tt.yaml)
|
||||
cfg := Default()
|
||||
err := cfg.ApplyFileConfig(file)
|
||||
if err == nil || !strings.Contains(err.Error(), "promptkit.local_backend.endpoint") {
|
||||
t.Fatalf("ApplyFileConfig() error = %v, want endpoint field context", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilePromptKitExplicitEmptyProfileSourcesAreRejected(t *testing.T) {
|
||||
for _, field := range []string{"profile_dir", "profile_file"} {
|
||||
t.Run(field, func(t *testing.T) {
|
||||
@@ -188,6 +282,11 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
||||
yaml: "version: 4\ncache:\n checkpoints:\n enabled: definitely\n",
|
||||
want: "cannot unmarshal",
|
||||
},
|
||||
{
|
||||
name: "local backend field",
|
||||
yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: http://localhost:8000/v1\n unknown: true\n",
|
||||
want: "field unknown not found",
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
|
||||
@@ -2,6 +2,7 @@ package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/url"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
@@ -53,6 +54,27 @@ func validatePromptKit(cfg PromptKitConfig) error {
|
||||
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
||||
return fmt.Errorf("promptkit profile_dir and profile_file are mutually exclusive")
|
||||
}
|
||||
if cfg.LocalBackend == nil {
|
||||
return nil
|
||||
}
|
||||
endpoint := strings.TrimSpace(cfg.LocalBackend.Endpoint)
|
||||
if endpoint == "" {
|
||||
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
|
||||
}
|
||||
parsed, err := url.Parse(endpoint)
|
||||
if err != nil ||
|
||||
(!strings.EqualFold(parsed.Scheme, "http") && !strings.EqualFold(parsed.Scheme, "https")) ||
|
||||
!parsed.IsAbs() ||
|
||||
parsed.Hostname() == "" ||
|
||||
parsed.User != nil ||
|
||||
parsed.RawQuery != "" ||
|
||||
parsed.ForceQuery ||
|
||||
strings.Contains(endpoint, "#") {
|
||||
return fmt.Errorf("promptkit.local_backend.endpoint must be an absolute HTTP or HTTPS URL with a host and no user information, query, or fragment")
|
||||
}
|
||||
if cfg.LocalBackend.ConcurrencyLimit < 0 {
|
||||
return fmt.Errorf("promptkit.local_backend.concurrency_limit must not be negative")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -93,6 +93,73 @@ func TestValidatePromptKitSourcesAreMutuallyExclusive(t *testing.T) {
|
||||
assertValidationContains(t, cfg, "promptkit profile_dir and profile_file are mutually exclusive")
|
||||
}
|
||||
|
||||
func TestValidatePromptKitLocalBackendEndpoints(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
endpoint string
|
||||
profileSource PromptKitConfig
|
||||
}{
|
||||
{
|
||||
name: "HTTP endpoint with path and profile directory",
|
||||
endpoint: "http://localhost:8000/v1",
|
||||
profileSource: PromptKitConfig{ProfileDir: "./profiles"},
|
||||
},
|
||||
{
|
||||
name: "case-insensitive HTTPS endpoint and profile file",
|
||||
endpoint: "HTTPS://inference.example.test/api",
|
||||
profileSource: PromptKitConfig{ProfileFile: "./profiles.yml"},
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
cfg := Default()
|
||||
cfg.PromptKit = tt.profileSource
|
||||
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
|
||||
Endpoint: tt.endpoint,
|
||||
ConcurrencyLimit: 2,
|
||||
}
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Fatalf("Validate() error = %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidatePromptKitLocalBackendRejectsInvalidValues(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
endpoint string
|
||||
concurrencyLimit int
|
||||
want string
|
||||
}{
|
||||
{name: "blank endpoint", endpoint: " ", want: "promptkit.local_backend.endpoint"},
|
||||
{name: "relative URL", endpoint: "localhost:8000/v1", want: "promptkit.local_backend.endpoint"},
|
||||
{name: "unsupported scheme", endpoint: "ftp://localhost/model", want: "promptkit.local_backend.endpoint"},
|
||||
{name: "missing host", endpoint: "http:///v1", want: "promptkit.local_backend.endpoint"},
|
||||
{name: "user information", endpoint: "http://user:secret@localhost/v1", want: "promptkit.local_backend.endpoint"},
|
||||
{name: "query", endpoint: "http://localhost/v1?model=example", want: "promptkit.local_backend.endpoint"},
|
||||
{name: "empty query", endpoint: "http://localhost/v1?", want: "promptkit.local_backend.endpoint"},
|
||||
{name: "fragment", endpoint: "http://localhost/v1#model", want: "promptkit.local_backend.endpoint"},
|
||||
{name: "empty fragment", endpoint: "http://localhost/v1#", want: "promptkit.local_backend.endpoint"},
|
||||
{
|
||||
name: "negative concurrency",
|
||||
endpoint: "http://localhost:8000/v1",
|
||||
concurrencyLimit: -1,
|
||||
want: "promptkit.local_backend.concurrency_limit",
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
cfg := Default()
|
||||
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
|
||||
Endpoint: tt.endpoint,
|
||||
ConcurrencyLimit: tt.concurrencyLimit,
|
||||
}
|
||||
assertValidationContains(t, cfg, tt.want)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateStateSurfaceRules(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
|
||||
@@ -19,9 +19,15 @@ import (
|
||||
|
||||
const promptKitProviderName = "promptkit"
|
||||
|
||||
type PromptKitLocalBackendConfig struct {
|
||||
Endpoint string
|
||||
ConcurrencyLimit int
|
||||
}
|
||||
|
||||
type PromptKitClientConfig struct {
|
||||
ProfileDir string
|
||||
ProfileFile string
|
||||
LocalBackend *PromptKitLocalBackendConfig
|
||||
Assets *AssetRegistry
|
||||
Timeout time.Duration
|
||||
HTTPClient *http.Client
|
||||
@@ -35,6 +41,7 @@ type PromptKitClient struct {
|
||||
recorder *LLMProfileRecorder
|
||||
profileDir string
|
||||
profileFile string
|
||||
localEndpoint string
|
||||
reasoningEffort *string
|
||||
}
|
||||
|
||||
@@ -46,6 +53,10 @@ type LLMProfileRecorder struct {
|
||||
var _ contracts.StructuredLLMClient = (*PromptKitClient)(nil)
|
||||
var _ contracts.LLMProfileManifestProvider = (*PromptKitClient)(nil)
|
||||
|
||||
func PromptKitLocalBackendOption(cfg PromptKitLocalBackendConfig) promptkit.Option {
|
||||
return promptkit.WithBackend(promptkit.LocalBackend(cfg.Endpoint, cfg.ConcurrencyLimit))
|
||||
}
|
||||
|
||||
func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
||||
if cfg.Assets == nil {
|
||||
return nil, fmt.Errorf("PromptKit client assets must not be nil")
|
||||
@@ -60,6 +71,13 @@ func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
||||
if profileFile := strings.TrimSpace(cfg.ProfileFile); profileFile != "" {
|
||||
options = append(options, promptkit.WithProfileFile(profileFile))
|
||||
}
|
||||
var localEndpoint string
|
||||
if cfg.LocalBackend != nil {
|
||||
localBackend := *cfg.LocalBackend
|
||||
localBackend.Endpoint = strings.TrimSpace(localBackend.Endpoint)
|
||||
localEndpoint = localBackend.Endpoint
|
||||
options = append(options, PromptKitLocalBackendOption(localBackend))
|
||||
}
|
||||
options = append(options, cfg.EngineOptions...)
|
||||
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||
@@ -84,6 +102,7 @@ func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
||||
recorder: recorder,
|
||||
profileDir: strings.TrimSpace(cfg.ProfileDir),
|
||||
profileFile: strings.TrimSpace(cfg.ProfileFile),
|
||||
localEndpoint: localEndpoint,
|
||||
reasoningEffort: reasoningEffort,
|
||||
}, nil
|
||||
}
|
||||
@@ -302,7 +321,11 @@ func (c *PromptKitClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint,
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return []CheckpointFingerprint{fingerprint}, nil
|
||||
fingerprints := []CheckpointFingerprint{fingerprint}
|
||||
if c.localEndpoint != "" {
|
||||
fingerprints = append(fingerprints, promptKitLocalBackendFingerprint(c.localEndpoint))
|
||||
}
|
||||
return fingerprints, nil
|
||||
}
|
||||
|
||||
func NewLLMProfileRecorder() *LLMProfileRecorder {
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
@@ -305,12 +306,89 @@ func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
|
||||
if fresh[0].Value == "mutated" {
|
||||
t.Fatal("LLMCheckpointFingerprints exposed mutable backing storage")
|
||||
}
|
||||
const wantBuiltinFingerprint = "sha256:4b48cd1ef010b587cd8d56bc13b73a0cdd84da1414a8de9f9a2adc695dfcf0a7"
|
||||
const wantBuiltinFingerprint = "sha256:5218b1dec48f5fdd46836826e0b25906c33efbdf943c5f18085b8d82467e0276"
|
||||
if fresh[0].Value != wantBuiltinFingerprint {
|
||||
t.Fatalf("built-in profile fingerprint = %q, want %q", fresh[0].Value, wantBuiltinFingerprint)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientCheckpointFingerprintTracksLocalBackendTarget(t *testing.T) {
|
||||
const (
|
||||
firstEndpoint = "http://localhost:8000/v1"
|
||||
secondEndpoint = "https://inference.example.test/v1"
|
||||
)
|
||||
fingerprintsFor := func(endpoint string, concurrencyLimit int) []CheckpointFingerprint {
|
||||
t.Helper()
|
||||
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||
Assets: newTestPromptKitAssets(t),
|
||||
LocalBackend: &PromptKitLocalBackendConfig{
|
||||
Endpoint: endpoint,
|
||||
ConcurrencyLimit: concurrencyLimit,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
values, err := client.LLMCheckpointFingerprints()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return values
|
||||
}
|
||||
|
||||
baseline := fingerprintsFor(firstEndpoint, 0)
|
||||
if len(baseline) != 2 ||
|
||||
baseline[0].Name != promptKitProfileFingerprintName ||
|
||||
baseline[1].Name != promptKitLocalBackendFingerprintName {
|
||||
t.Fatalf("checkpoint fingerprints = %#v, want profile source then local backend target", baseline)
|
||||
}
|
||||
endpointChanged := fingerprintsFor(secondEndpoint, 0)
|
||||
if baseline[0] != endpointChanged[0] || baseline[1] == endpointChanged[1] {
|
||||
t.Fatalf("endpoint fingerprints = %#v and %#v, want only local target to change", baseline, endpointChanged)
|
||||
}
|
||||
concurrencyChanged := fingerprintsFor(firstEndpoint, 4)
|
||||
if !reflect.DeepEqual(baseline, concurrencyChanged) {
|
||||
t.Fatalf("concurrency fingerprints = %#v, want %#v", concurrencyChanged, baseline)
|
||||
}
|
||||
for _, values := range [][]CheckpointFingerprint{baseline, endpointChanged} {
|
||||
for _, fingerprint := range values {
|
||||
if strings.Contains(fingerprint.Value, firstEndpoint) ||
|
||||
strings.Contains(fingerprint.Value, secondEndpoint) {
|
||||
t.Fatalf("checkpoint fingerprint exposes endpoint: %#v", fingerprint)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
localBackend := &PromptKitLocalBackendConfig{
|
||||
Endpoint: " " + firstEndpoint + " ",
|
||||
ConcurrencyLimit: 0,
|
||||
}
|
||||
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||
Assets: newTestPromptKitAssets(t),
|
||||
LocalBackend: localBackend,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
localBackend.Endpoint = secondEndpoint
|
||||
copy, err := client.LLMCheckpointFingerprints()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !reflect.DeepEqual(copy, baseline) {
|
||||
t.Fatalf("fingerprints after input mutation = %#v, want retained target %#v", copy, baseline)
|
||||
}
|
||||
copy[0].Value = "mutated-profile"
|
||||
copy[1].Value = "mutated-target"
|
||||
fresh, err := client.LLMCheckpointFingerprints()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !reflect.DeepEqual(fresh, baseline) {
|
||||
t.Fatalf("fingerprints after returned-slice mutation = %#v, want %#v", fresh, baseline)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) {
|
||||
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||
client := newTestPromptKitClient(t, fake)
|
||||
@@ -334,6 +412,88 @@ func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testi
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientUsesConfiguredLocalBackend(t *testing.T) {
|
||||
var providerCalls atomic.Int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
providerCalls.Add(1)
|
||||
if r.URL.Path != "/v1/chat/completions" {
|
||||
t.Errorf("provider path = %q, want /v1/chat/completions", r.URL.Path)
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{
|
||||
"choices": [{"message": {"role": "assistant", "content": "{\"ok\":true}"}}],
|
||||
"usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7}
|
||||
}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
|
||||
if err := os.WriteFile(profilePath, []byte(`id: local-profile
|
||||
backend: local
|
||||
model: local-model
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
localBackend := &PromptKitLocalBackendConfig{
|
||||
Endpoint: " " + server.URL + "/v1 ",
|
||||
ConcurrencyLimit: 2,
|
||||
}
|
||||
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||
Assets: newTestPromptKitAssets(t),
|
||||
ProfileFile: profilePath,
|
||||
LocalBackend: localBackend,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
|
||||
}
|
||||
localBackend.Endpoint = "http://127.0.0.1:1/v1"
|
||||
|
||||
request := contracts.StructuredCompletionRequest{
|
||||
PromptID: "adapter.direct-session",
|
||||
ProfileID: "local-profile",
|
||||
Inputs: contracts.LLMInputSet{
|
||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||
},
|
||||
Vars: map[string]any{"custom": "value"},
|
||||
}
|
||||
var out map[string]any
|
||||
response, err := client.CompleteStructured(context.Background(), request, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
||||
}
|
||||
if providerCalls.Load() != 1 {
|
||||
t.Fatalf("provider calls = %d, want 1", providerCalls.Load())
|
||||
}
|
||||
if response.ProfileID != "local-profile" || response.Model != "local-model" {
|
||||
t.Fatalf("response metadata = %#v", response)
|
||||
}
|
||||
if response.Debug == nil || response.Debug.Prompt == nil ||
|
||||
response.Debug.Prompt.SelectedBackendID != promptkit.BackendLocal {
|
||||
t.Fatalf("response debug prompt = %#v, want local backend", response.Debug)
|
||||
}
|
||||
manifests := client.LLMProfileManifests()
|
||||
if len(manifests) != 1 || manifests[0].BackendID != promptkit.BackendLocal {
|
||||
t.Fatalf("profile manifests = %#v, want local backend", manifests)
|
||||
}
|
||||
|
||||
clientWithoutRegistration, err := NewPromptKitClient(PromptKitClientConfig{
|
||||
Assets: newTestPromptKitAssets(t),
|
||||
ProfileFile: profilePath,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewPromptKitClient() without local registration error = %v, want nil", err)
|
||||
}
|
||||
_, err = clientWithoutRegistration.CompleteStructured(context.Background(), request, &out)
|
||||
if err == nil ||
|
||||
!strings.Contains(err.Error(), "prepare PromptKit prompt") ||
|
||||
!strings.Contains(err.Error(), promptkit.BackendLocal) {
|
||||
t.Fatalf("CompleteStructured() without registration error = %v, want preparation failure with local backend context", err)
|
||||
}
|
||||
if providerCalls.Load() != 1 {
|
||||
t.Fatalf("provider calls after missing-registration failure = %d, want 1", providerCalls.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestLLMProfileRecorderDistinguishesEffectiveTargets(t *testing.T) {
|
||||
recorder := NewLLMProfileRecorder()
|
||||
for _, profile := range []artifacts.LLMProfileManifest{
|
||||
|
||||
@@ -12,10 +12,12 @@ import (
|
||||
)
|
||||
|
||||
const (
|
||||
promptKitProfileFingerprintName = "promptkit_profile_source"
|
||||
promptKitProfileFingerprintName = "promptkit_profile_source"
|
||||
promptKitLocalBackendFingerprintName = "promptkit_local_backend_target"
|
||||
promptKitLocalBackendMarker = "notarius:promptkit-local-backend:v1"
|
||||
// The built-in profile catalog is compiled into this pinned PromptKit
|
||||
// release. Update this identity when the dependency is upgraded.
|
||||
promptKitBuiltinProfileCatalogID = "promptkit:v0.2.0:builtin-profiles"
|
||||
promptKitBuiltinProfileCatalogID = "promptkit:v0.3.0:builtin-profiles"
|
||||
)
|
||||
|
||||
func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFingerprint, error) {
|
||||
@@ -45,6 +47,16 @@ func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFing
|
||||
}, nil
|
||||
}
|
||||
|
||||
func promptKitLocalBackendFingerprint(endpoint string) CheckpointFingerprint {
|
||||
hasher := sha256.New()
|
||||
writeFingerprintPart(hasher, []byte(promptKitLocalBackendMarker))
|
||||
writeFingerprintPart(hasher, []byte(strings.TrimSpace(endpoint)))
|
||||
return CheckpointFingerprint{
|
||||
Name: promptKitLocalBackendFingerprintName,
|
||||
Value: "sha256:" + hex.EncodeToString(hasher.Sum(nil)),
|
||||
}
|
||||
}
|
||||
|
||||
func promptKitProfileFileDigests(root string) ([][]byte, error) {
|
||||
var digests [][]byte
|
||||
err := filepath.WalkDir(root, func(name string, entry fs.DirEntry, walkErr error) error {
|
||||
|
||||
@@ -81,7 +81,10 @@ func TestScheduledClientPreservesCheckpointFingerprints(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inner := &fingerprintedStructuredClient{
|
||||
fingerprints: []CheckpointFingerprint{{Name: "profile_source", Value: "sha256:one"}},
|
||||
fingerprints: []CheckpointFingerprint{
|
||||
{Name: "profile_source", Value: "sha256:one"},
|
||||
{Name: "backend_target", Value: "sha256:two"},
|
||||
},
|
||||
}
|
||||
client := NewScheduledClient(inner, scheduler)
|
||||
provider, ok := client.(CheckpointFingerprintProvider)
|
||||
@@ -92,7 +95,9 @@ func TestScheduledClientPreservesCheckpointFingerprints(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 1 || got[0] != inner.fingerprints[0] {
|
||||
if len(got) != len(inner.fingerprints) ||
|
||||
got[0] != inner.fingerprints[0] ||
|
||||
got[1] != inner.fingerprints[1] {
|
||||
t.Fatalf("checkpoint fingerprints = %#v, want %#v", got, inner.fingerprints)
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user