Compare commits

..

7 Commits

22 changed files with 836 additions and 50 deletions

View File

@@ -46,7 +46,7 @@ other than **version** is optional.
| Field | Type | Default | Rules | | Field | Type | Default | Rules |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| **version** | integer | none | Required; must be 4. | | **version** | integer | none | Required; must be 4. |
| **promptkit** | object | none | Profile source configuration. | | **promptkit** | object | none | Profile source and optional local-backend configuration. |
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. | | **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
| **concurrency** | object | see below | Global LLM and extraction limits. | | **concurrency** | object | see below | Global LLM and extraction limits. |
| **output** | object | see below | Published output settings. | | **output** | object | see below | Published output settings. |
@@ -71,27 +71,58 @@ per-user root. An explicit empty output or debug directory is invalid.
## PromptKit Profiles ## PromptKit Profiles
The optional **promptkit** object selects one source of profile definitions: The optional **promptkit** object selects one source of profile definitions and
may register one conventional local OpenAI-compatible backend:
~~~yaml ~~~yaml
version: 4 version: 4
promptkit: promptkit:
profile_dir: /path/to/profiles profile_dir: ./profiles
# profile_file: /path/to/profiles.yml # profile_file: ./profiles.yml
local_backend:
endpoint: http://localhost:8000/v1
concurrency_limit: 2
~~~ ~~~
| Field | Type | Rules | | Field | Type | Rules |
| --- | --- | --- | | --- | --- | --- |
| **profile_dir** | string | Non-empty directory containing profile files. | | **profile_dir** | string | Non-empty directory containing profile files. |
| **profile_file** | string | Non-empty profile file. | | **profile_file** | string | Non-empty profile file. |
| **local_backend** | object | Optional registration for the conventional PromptKit backend ID **local**. |
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
Set at most one of these fields. Profile IDs used by a binding must be available Set at most one of **profile_dir** and **profile_file**. Profile IDs used by a
from the selected PromptKit profile source when the pipeline is resolved. binding must be available from the selected PromptKit profile source when the
Keep credentials out of this file: configure a profile to read its credential pipeline is resolved. The optional local registration may coexist with either
from an environment variable, then set that environment variable only in the profile source or with PromptKit's built-in profiles.
run environment. PromptKit owns the profile-file format; see the
[PromptKit upstream boundary](integrations/pkg-promptkit.md) for the pinned When **local_backend** is present, its endpoint is trimmed and must use HTTP or
package and canonical format reference. HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are
allowed. User information, queries, and fragments are rejected. A zero
**concurrency_limit** leaves the local backend unrestricted inside PromptKit;
a positive value limits simultaneous local generations. The application-wide
**concurrency.total_llm** limit still applies in both cases. Neither local
backend field has an environment override. Omitting **local_backend** registers
nothing and preserves existing built-in and endpoint-only profile behavior.
A file-backed PromptKit profile selects the registration by its case-sensitive
backend ID:
~~~yaml
id: local-summary
backend: local
model: example-model
~~~
Keep credentials out of the local-backend object. A PromptKit profile may name
its credential environment variable through `api_key_env`; set that variable
only in the run environment. PromptKit owns the
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/formats.md).
The [PromptKit upstream boundary](integrations/pkg-promptkit.md) identifies the
supported package API, and [Operations](operations.md#operational-limits)
describes the effective concurrency layers.
## Migrating Version 3 Configuration ## Migrating Version 3 Configuration

View File

@@ -1,11 +1,11 @@
# PromptKit Integration # PromptKit Integration
Notarius pins Notarius pins
[`gitea.maximumdirect.net/eric/promptkit` v0.2.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0) [`gitea.maximumdirect.net/eric/promptkit` v0.3.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0)
as its in-process prompt engine. The upstream as its in-process prompt engine. The upstream
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0/docs/consumers/pkg-promptkit.md) [Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/consumers/pkg-promptkit.md)
owns the public engine API, and the upstream owns the public engine API, and the upstream
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0/docs/formats.md) [format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/formats.md)
owns prompt, profile, and schema file contracts. owns prompt, profile, and schema file contracts.
## Supported Boundary ## Supported Boundary
@@ -18,10 +18,16 @@ Notarius relies on the root `promptkit` package to:
a direct session ID, prompt identity, and profile selection; a direct session ID, prompt identity, and profile selection;
- return rendered debug material, validated structured output, selected - return rendered debug material, validated structured output, selected
profile, backend, effective model metadata, and token usage; profile, backend, effective model metadata, and token usage;
- register the optional conventional `local` backend through `BackendLocal`,
`LocalBackend`, and `WithBackend`;
- distinguish structured-output validation failure from execution failure; and - distinguish structured-output validation failure from execution failure; and
- identify a missing explicit profile through `ErrProfileNotFound` and backend - identify a missing explicit profile through `ErrProfileNotFound` and backend
admission exhaustion through `ErrCapacityExceeded`. admission exhaustion through `ErrCapacityExceeded`.
The pinned
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/backends.go)
owns the registration and backend-capacity contract.
Notarius does not use PromptKit's optional `ArtifactReader`. It materializes Notarius does not use PromptKit's optional `ArtifactReader`. It materializes
source and reference content itself and supplies owned inline artifacts at the source and reference content itself and supplies owned inline artifacts at the
adapter boundary. It also retains responsibility for pipeline retries, adapter boundary. It also retains responsibility for pipeline retries,
@@ -38,8 +44,9 @@ Notarius records PromptKit's selected backend ID and effective reasoning
setting as optional run-manifest provenance. Endpoint-only profiles have no setting as optional run-manifest provenance. Endpoint-only profiles have no
backend ID. Debug prompt material also retains the selected backend ID and backend ID. Debug prompt material also retains the selected backend ID and
PromptKit's stable lower-case `effective_model_params` JSON, which may include PromptKit's stable lower-case `effective_model_params` JSON, which may include
`backend_id`. Notarius production configuration does not expose user-defined `backend_id`. Notarius production configuration exposes one optional
PromptKit backend registration. conventional `local` registration. It does not expose a general user-defined
PromptKit backend registry. Endpoint-only profiles remain supported unchanged.
Notarius retains its application-wide scheduled client around the PromptKit Notarius retains its application-wide scheduled client around the PromptKit
adapter. PromptKit may apply a narrower limit for the selected backend; adapter. PromptKit may apply a narrower limit for the selected backend;
@@ -55,7 +62,8 @@ module assets, maps its transport-neutral completion contract, prepares and
executes requests, validates output, records provenance, captures debug executes requests, validates output, records provenance, captures debug
material, redacts errors, and preserves timeout ownership. material, redacts errors, and preserves timeout ownership.
[Configuration](../config.md#promptkit-profiles) defines how a Notarius [Configuration](../config.md#promptkit-profiles) defines how a Notarius
configuration selects one PromptKit profile source. configuration selects one PromptKit profile source and optionally registers
the conventional local backend.
PromptKit API or format changes outside this boundary are not implicitly PromptKit API or format changes outside this boundary are not implicitly
supported. Updating the pinned version requires reviewing the adapter and supported. Updating the pinned version requires reviewing the adapter and

View File

@@ -39,14 +39,20 @@ This establishes the public precedence order without giving environment input a
second file schema. Loading and application reject malformed YAML, unsupported second file schema. Loading and application reject malformed YAML, unsupported
file versions, unknown fields, invalid values, and identifiers that are empty file versions, unknown fields, invalid values, and identifiers that are empty
or collide after whitespace normalization. The file application also makes the or collide after whitespace normalization. The file application also makes the
effective extraction-worker default follow the effective LLM limit. effective extraction-worker default follow the effective LLM limit. A present
PromptKit local-backend object requires and trims its endpoint, defaults its
omitted concurrency limit to zero, and is copied so the parsed file model
cannot alias the populated **Config**.
**Config.Validate** checks configuration-only invariants before resolution. It **Config.Validate** checks configuration-only invariants before resolution. It
rejects incompatible profile sources, invalid state-surface values, unsupported rejects incompatible profile sources, invalid state-surface values, unsupported
concurrency settings, malformed bindings and references, invalid retries, and concurrency settings, malformed bindings and references, invalid retries, and
invalid pipeline, step, or lane structure. Its errors retain the closest known invalid pipeline, step, or lane structure. PromptKit local-backend validation
pipeline, lane, and binding context. It deliberately does not require modules accepts only an absolute HTTP or HTTPS endpoint with a host and no user
to be registered: that requires a catalog and belongs to resolution. information, query, or fragment, and rejects a negative local concurrency
limit. Its errors retain the closest known pipeline, lane, and binding context.
It deliberately does not require modules to be registered: that requires a
catalog and belongs to resolution.
The exact user-selectable values and validation rules are defined in The exact user-selectable values and validation rules are defined in
[Configuration](../config.md). Keep additions to the file model, an [Configuration](../config.md). Keep additions to the file model, an
@@ -71,7 +77,8 @@ options, and returns the fixed ordered pipeline shape. The resulting
changes, a clone of the input configuration, and the resolved pipeline. changes, a clone of the input configuration, and the resolved pipeline.
Callers may therefore retain or modify their input slices and maps without Callers may therefore retain or modify their input slices and maps without
changing the resolved result, and later consumers cannot mutate the original changing the resolved result, and later consumers cannot mutate the original
configuration through the effective value. configuration through the effective value. This ownership includes the nested
PromptKit local-backend value.
Resolution failures stop before module construction and source parsing. They Resolution failures stop before module construction and source parsing. They
include an error path for an unconfigured pipeline, missing module, missing include an error path for an unconfigured pipeline, missing module, missing
@@ -94,9 +101,12 @@ Configuration summaries must use **Redacted**, **RedactedSummaryPayload**, or
Those methods copy every binding and nested option container, replace values Those methods copy every binding and nested option container, replace values
whose key is credential-shaped with **[REDACTED]**, and omit materialized whose key is credential-shaped with **[REDACTED]**, and omit materialized
reference content while retaining safe binding and reference provenance. The reference content while retaining safe binding and reference provenance. The
payload must not alias the source configuration or resolved pipeline. This payload must not alias the source configuration or resolved pipeline.
redaction is deliberately narrow: it protects configuration summaries and does PromptKit's local endpoint and concurrency limit are preserved as non-secret
not authorize recording arbitrary environment values or provider requests. configuration metadata in the independently owned summary; the object contains
no credential value. This redaction is deliberately narrow: it protects
configuration summaries and does not authorize recording arbitrary environment
values or provider requests.
## Invariants To Preserve ## Invariants To Preserve

View File

@@ -56,6 +56,13 @@ distinct and deterministically ordered. Endpoint-only profiles retain an empty
backend ID, which the published JSON omits. Successful completion responses and backend ID, which the published JSON omits. Successful completion responses and
recorded profile manifests identify the adapter provider as `promptkit`. recorded profile manifests identify the adapter provider as `promptkit`.
The CLI's preparation-only engine and the production adapter use the same
conversion helper to register the optional conventional `local` backend.
Preflight therefore resolves the same backend membership as runtime without
performing generation. When the registration is absent, a profile selecting
`backend: local` fails preparation instead of falling back to a built-in or
endpoint-only target.
Before execution, the adapter also contributes a non-secret checkpoint Before execution, the adapter also contributes a non-secret checkpoint
fingerprint for the effective PromptKit profile source. It combines the fingerprint for the effective PromptKit profile source. It combines the
identity of PromptKit's compiled-in profile catalog with a deterministic digest identity of PromptKit's compiled-in profile catalog with a deterministic digest
@@ -65,7 +72,11 @@ paths. It covers both explicit binding profiles and prompt-selected defaults,
so changing a model or other profile setting cannot reuse checkpoints created so changing a model or other profile setting cannot reuse checkpoints created
under the prior profile source. This cache identity is independent of durable under the prior profile source. This cache identity is independent of durable
profile provenance: run manifests continue to list only profiles actually profile provenance: run manifests continue to list only profiles actually
observed during LLM calls. observed during LLM calls. When the local backend is registered, a second
fingerprint hashes its trimmed endpoint behind a stable marker. Changing that
semantic execution target invalidates checkpoint reuse. The raw endpoint is not
stored in checkpoint identity, and the local concurrency limit is excluded
because it changes scheduling rather than execution semantics.
## Shared Provider-Call Limit ## Shared Provider-Call Limit
@@ -85,8 +96,10 @@ PromptKit applies a second, independent admission limit when the selected
profile names a limited backend. It sits beneath the Notarius scheduled client, profile names a limited backend. It sits beneath the Notarius scheduled client,
so it may narrow but cannot expand the application-wide limit. Built-in so it may narrow but cannot expand the application-wide limit. Built-in
OpenRouter profiles select PromptKit's reserved backend and its upstream OpenRouter profiles select PromptKit's reserved backend and its upstream
capacity policy. Endpoint-only profiles do not select a PromptKit backend and capacity policy. A positive configured local-backend limit bounds active local
remain limited only by the Notarius scheduler. generations inside PromptKit; zero leaves that backend unlimited there.
Endpoint-only profiles do not select a PromptKit backend and remain limited
only by the Notarius scheduler.
## Prompt And Schema Assets ## Prompt And Schema Assets

View File

@@ -118,8 +118,10 @@ disabled. Without **--resume**, a recording-enabled run executes normally and
does not load checkpoint state. Compatibility includes the resolved pipeline, does not load checkpoint state. Compatibility includes the resolved pipeline,
input, selected lanes, runtime overrides, reference provenance, LLM-profile input, selected lanes, runtime overrides, reference provenance, LLM-profile
provenance, the effective PromptKit profile-source fingerprint, and provenance, the effective PromptKit profile-source fingerprint, and
prepared-component fingerprints. Changing profile content causes a cold miss prepared-component fingerprints. When a local PromptKit backend is configured,
even when its profile ID is unchanged. A changed identity produces a cold miss; compatibility also includes a non-secret fingerprint of its endpoint. Changing
profile content or the local endpoint causes a cold miss; changing only the
local concurrency limit does not. A changed identity produces a cold miss;
Notarius does not migrate, rewrite, or delete older checkpoint directories. Notarius does not migrate, rewrite, or delete older checkpoint directories.
Reasoning-effort inheritance, replacement, and explicit clearing are distinct Reasoning-effort inheritance, replacement, and explicit clearing are distinct
runtime identities, so checkpoints created under one state are not reused by runtime identities, so checkpoints created under one state are not reused by
@@ -217,7 +219,7 @@ Provider execution settings and the generation timeout come from the selected
PromptKit profile. The invocation-only **--reasoning-effort** and PromptKit profile. The invocation-only **--reasoning-effort** and
**--clear-reasoning-effort** controls may replace or clear that profile setting **--clear-reasoning-effort** controls may replace or clear that profile setting
for all LLM-backed calls in one run without changing the profile. PromptKit for all LLM-backed calls in one run without changing the profile. PromptKit
v0.2.0 does not add a provider retry loop. Notarius binding retries rerun the v0.3.0 does not add a provider retry loop. Notarius binding retries rerun the
complete module operation and validation chain as defined by complete module operation and validation chain as defined by
[module bindings](config.md#module-bindings-and-validators). [module bindings](config.md#module-bindings-and-validators).
@@ -234,13 +236,19 @@ and validators. PromptKit may impose a narrower admission limit for the
selected backend. The effective active-generation bound is the intersection of selected backend. The effective active-generation bound is the intersection of
both limits and can therefore be lower than **total_llm**. Built-in OpenRouter both limits and can therefore be lower than **total_llm**. Built-in OpenRouter
profiles use PromptKit's upstream backend limit; endpoint-only profiles have no profiles use PromptKit's upstream backend limit; endpoint-only profiles have no
PromptKit backend limit and remain bounded by Notarius. PromptKit backend limit and remain bounded by Notarius. For the configured
local backend, a zero **concurrency_limit** leaves only the Notarius scheduler
as a call limit. A positive value makes the effective active local-generation
bound the smaller of **total_llm** and that local limit.
When a PromptKit backend has admitted all active and queued work, a new call For a positive local limit, PromptKit owns its default waiting capacity and
fails as capacity exhaustion before generation. The adapter does not retry it. admission behavior. When a PromptKit backend has admitted all active and queued
The calling stage's configured retry policy applies normally, and the run fails work, a new call fails as capacity exhaustion before generation. The adapter
if those attempts are exhausted. Caller cancellation remains authoritative. maps that failure to Notarius's existing provider-neutral capacity error and
Configuration contracts are documented under does not retry it. The calling stage's configured retry policy applies
normally, and the run fails if those attempts are exhausted. Caller
cancellation remains authoritative. Configuration contracts are documented
under [PromptKit profiles](config.md#promptkit-profiles) and
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker [concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
limits and actual provider-call limits are independent. Notarius writes local limits and actual provider-call limits are independent. Notarius writes local
filesystem state only; remote storage, archival, and retention automation are filesystem state only; remote storage, archival, and retention automation are

View File

@@ -53,6 +53,120 @@ not as committed release dates.
spell, combat, interaction, and scene-description lanes after real-world use. spell, combat, interaction, and scene-description lanes after real-world use.
Add more complex chunking only in response to demonstrated failures. Add more complex chunking only in response to demonstrated failures.
## Cross-Cutting LLM Runtime
### Deterministic Prompt Session Identity
- Replace the source-document-ID default for prompt sessions with one
predictable, procedurally generated session ID for the complete
source-processing workload.
- Preserve an explicit non-empty `--session-id` as the highest-precedence
override. Otherwise, derive the default only from the effective input module
identity and the exact raw input bytes.
- Use a versioned, bounded representation such as
`notarius:v1:<sha256(input-module + NUL + raw-input)>`. The exact encoding
must fit PromptKit's session length contract and must not embed source
content.
- Keep the derived session stable across runs, pipelines, selected lanes,
ordered steps, retries, resume, recomputation, LLM profiles, reasoning
overrides, and output, debug, or cache settings.
- Do not include file-backed references, generated references, reference
contents, or the composition of a reference bundle in session derivation.
References may change between prompt calls within one pipeline without
changing routing affinity.
- Resolve the authoritative session before checkpoint construction and use the
same value for checkpoint runtime identity, every prompt-facing module,
PromptKit's direct session field, the compatibility `session_id` prompt
variable, run-manifest metadata, and debug metadata.
- Keep routing identity separate from cache and checkpoint content identity.
Exact prompt prefixes, reference contents, model settings, and other
generation-affecting inputs must continue to participate in their existing
hashes and checkpoint fingerprints even though they do not change the
session.
- Treat the generated value as a provider-visible, stable pseudonymous
correlation identifier. Do not introduce an installation-specific HMAC or
secret unless a concrete multi-tenant or privacy requirement justifies
sacrificing deterministic identity across installations.
### Pipeline-Level LLM Profile Defaults
- Add an optional pipeline-level `llm_profile` default so an operator can
select one PromptKit execution policy for the pipeline without repeating the
same profile ID on every LLM-backed module binding.
- Apply the following precedence consistently: an explicit run-wide
`--llm-profile` override, then a binding-specific `llm_profile`, then the
pipeline-level default, then the prompt definition's embedded
`default_profile`.
- Apply inheritance only to bindings whose resolved modules are LLM-backed,
including applicable chunk, extraction, merge, normalization, and validation
bindings. Do not attach an inherited profile to deterministic modules or
weaken existing validation that rejects profiles where generation is not
supported.
- Resolve inherited profiles before effective-pipeline validation, digest and
checkpoint construction, execution, and provenance capture. Validate every
resulting explicit profile ID against the selected PromptKit profile source,
and ensure manifests and debug output report the profile actually selected
for each generation target.
- Preserve binding-specific profiles as intentional exceptions for modules
that require a different quality, latency, cost, provider, or reasoning
policy. Preserve `--llm-profile` as the convenient highest-precedence
experiment or incident-response override for an entire run.
- Treat PromptKit profiles as execution-policy configuration and prefer stable,
workload-oriented IDs such as `dnd-extraction` over model names. A pipeline
should express the kind of work it performs rather than encode a particular
provider, model, or deployment environment.
- Establish deployment-managed PromptKit profile files as the recommended
environment-specific configuration mechanism. Production, development, and
local deployments may each define the same logical `dnd-extraction` ID with
different model and generation settings, while retaining one unchanged
Notarius pipeline definition.
- Keep deployment profiles distinct from both Notarius prompt assets embedded
in the application binary and PromptKit's built-in profile catalog. Document
that `promptkit.profile_dir` or `promptkit.profile_file` selects an external
filesystem source whose definitions overlay PromptKit built-ins, and
recommend application-owned IDs rather than silently replacing built-in
profile IDs.
- Document an operator-friendly layout in which the Notarius configuration and
a profile subdirectory are deployed together. Until profile paths are
explicitly resolved relative to the configuration file, clearly state that
relative paths use the process working directory and recommend absolute
paths for services and containers.
- Update the configuration reference, operations guide, PromptKit integration
boundary, relevant internal configuration and pipeline documentation, and
maintained D&D examples together. Include one concrete external-profile
example and explain the profile precedence and environment-neutral pipeline
pattern without duplicating PromptKit's complete profile-format reference.
### Raise The Default Application-Wide LLM Limit
- Raise the default `concurrency.total_llm` value from 1 to 16 so ordinary
single-backend runs can use PromptKit's expected OpenRouter capacity and
lower-capacity local backends without an unnecessarily narrower Notarius
limit.
- Keep the Notarius application-wide scheduler mandatory and require
`total_llm` to remain a positive integer. Do not make the default unlimited:
endpoint-only profiles, an unrestricted local backend, injected clients, and
aggregate work across several backends may have no narrower PromptKit limit.
- Continue defaulting `concurrency.stage_workers.extract` to the effective
`total_llm`, making its default 16 as part of the same change. Preserve an
explicit lower extract-worker setting when an operator wants less queued or
concurrent extraction work.
- Define effective provider concurrency as the intersection of the Notarius
application-wide limit, the selected PromptKit backend limit when present,
and the work made available by stage execution. A Notarius limit of 16 does
not narrow a backend already limited to 16, while a local backend limited to
4 remains bounded at 4.
- Treat the default as an application-wide safety ceiling across profiles,
backends, modules, retries, and validators. A run that intentionally needs
the combined capacity of several backends may configure a higher
`total_llm` and an appropriate extract-worker count explicitly.
- Retain the existing configuration and environment override surfaces. Update
canonical configuration, operations, and internal documentation together
when the default changes.
- Reconsider decoupling the extract-worker default from `total_llm` only after
mixed-backend workloads demonstrate a need for a high global emergency
ceiling with a lower default work-production rate.
## Shared Normalization And Quality Work ## Shared Normalization And Quality Work
### Generic LLM-Assisted Deduplication ### Generic LLM-Assisted Deduplication

2
go.mod
View File

@@ -3,7 +3,7 @@ module gitea.maximumdirect.net/eric/notarius
go 1.25.5 go 1.25.5
require ( require (
gitea.maximumdirect.net/eric/promptkit v0.2.0 gitea.maximumdirect.net/eric/promptkit v0.3.0
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
gopkg.in/yaml.v3 v3.0.1 gopkg.in/yaml.v3 v3.0.1
) )

4
go.sum
View File

@@ -1,5 +1,5 @@
gitea.maximumdirect.net/eric/promptkit v0.2.0 h1:6bUATKnVgLlHSbKjGb8EnCt2r7jV5Xw6t+pX7G9S/lU= gitea.maximumdirect.net/eric/promptkit v0.3.0 h1:P5/GJ6fVIVsBLJx8NxoaGV34ZWeAkU8lm4Ht6aVMOFg=
gitea.maximumdirect.net/eric/promptkit v0.2.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ= gitea.maximumdirect.net/eric/promptkit v0.3.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI= github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8= github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ= github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=

View File

@@ -171,6 +171,7 @@ func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID
client, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{ client, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{
ProfileDir: cfg.PromptKit.ProfileDir, ProfileDir: cfg.PromptKit.ProfileDir,
ProfileFile: cfg.PromptKit.ProfileFile, ProfileFile: cfg.PromptKit.ProfileFile,
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
Assets: assets, Assets: assets,
Recorder: recorder, Recorder: recorder,
ReasoningEffort: overrides.ReasoningEffort, ReasoningEffort: overrides.ReasoningEffort,

View File

@@ -6,6 +6,8 @@ import (
"errors" "errors"
"fmt" "fmt"
"io/fs" "io/fs"
"net/http"
"net/http/httptest"
"os" "os"
"path/filepath" "path/filepath"
"reflect" "reflect"
@@ -13,7 +15,9 @@ import (
"sort" "sort"
"strings" "strings"
"sync" "sync"
"sync/atomic"
"testing" "testing"
"testing/fstest"
"time" "time"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts" "gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
@@ -34,6 +38,7 @@ import (
itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents" itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents"
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells" spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop" "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
"gitea.maximumdirect.net/eric/promptkit"
) )
func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) { func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
@@ -406,6 +411,89 @@ func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) {
} }
} }
func TestProductionLLMClientFactoryUsesConfiguredLocalBackend(t *testing.T) {
var providerCalls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
providerCalls.Add(1)
if r.URL.Path != "/v1/chat/completions" {
t.Errorf("provider path = %q, want /v1/chat/completions", r.URL.Path)
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{
"choices": [{"message": {"role": "assistant", "content": "{\"ok\":true}"}}],
"usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7}
}`))
}))
defer server.Close()
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
if err := os.WriteFile(profilePath, []byte(`id: local-profile
backend: local
model: local-model
`), 0o600); err != nil {
t.Fatal(err)
}
assets := llm.NewAssetRegistry()
if err := assets.RegisterPromptFS(fstest.MapFS{
"production.local.yaml": {Data: []byte(`id: production.local
version: "v1"
inputs:
- name: transcript
required: true
messages:
- role: user
content: '{{ input "transcript" }}'
output:
format: json
validation_mode: json
`)},
}, "."); err != nil {
t.Fatalf("register prompt assets: %v", err)
}
cfg := config.Default()
cfg.PromptKit.ProfileFile = profilePath
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
Endpoint: server.URL + "/v1",
ConcurrencyLimit: 2,
}
client, manifests, err := productionLLMClientFactoryWithAssets(assets)(
context.Background(),
cfg,
"local-profile",
LLMRuntimeOverrides{},
)
if err != nil {
t.Fatalf("build production LLM runtime: %v", err)
}
if len(manifests) != 0 {
t.Fatalf("eager profile manifests = %#v, want none", manifests)
}
var out map[string]any
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
PromptID: "production.local",
ProfileID: "local-profile",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "text/plain", []byte("local request"), "", ""),
},
}, &out)
if err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", err)
}
if providerCalls.Load() != 1 {
t.Fatalf("provider calls = %d, want 1", providerCalls.Load())
}
provider, ok := client.(contracts.LLMProfileManifestProvider)
if !ok {
t.Fatalf("production client %T does not provide profile manifests", client)
}
recorded := provider.LLMProfileManifests()
if len(recorded) != 1 || recorded[0].BackendID != promptkit.BackendLocal {
t.Fatalf("production profile manifests = %#v, want local backend", recorded)
}
}
func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) { func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
t.Run("canceled context", func(t *testing.T) { t.Run("canceled context", func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())

View File

@@ -7,6 +7,7 @@ import (
"testing/fstest" "testing/fstest"
"gitea.maximumdirect.net/eric/notarius/internal/core/config" "gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/promptkit" "gitea.maximumdirect.net/eric/promptkit"
) )
@@ -61,8 +62,21 @@ func newProfileValidationEngine(cfg config.Config) (*promptkit.Engine, error) {
if cfg.PromptKit.ProfileFile != "" { if cfg.PromptKit.ProfileFile != "" {
opts = append(opts, promptkit.WithProfileFile(cfg.PromptKit.ProfileFile)) opts = append(opts, promptkit.WithProfileFile(cfg.PromptKit.ProfileFile))
} }
if localBackend := mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend); localBackend != nil {
opts = append(opts, llm.PromptKitLocalBackendOption(*localBackend))
}
return promptkit.NewEngine(promptkit.Config{ return promptkit.NewEngine(promptkit.Config{
PromptDir: "unused", PromptDir: "unused",
ProfileDir: cfg.PromptKit.ProfileDir, ProfileDir: cfg.PromptKit.ProfileDir,
}, opts...) }, opts...)
} }
func mapPromptKitLocalBackend(cfg *config.PromptKitLocalBackendConfig) *llm.PromptKitLocalBackendConfig {
if cfg == nil {
return nil
}
return &llm.PromptKitLocalBackendConfig{
Endpoint: cfg.Endpoint,
ConcurrencyLimit: cfg.ConcurrencyLimit,
}
}

View File

@@ -0,0 +1,54 @@
package cli
import (
"context"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync/atomic"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/promptkit"
)
func TestExplicitPromptKitProfileValidationUsesConfiguredLocalBackendWithoutGeneration(t *testing.T) {
var providerCalls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
providerCalls.Add(1)
}))
defer server.Close()
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
if err := os.WriteFile(profilePath, []byte(`id: local-profile
backend: local
model: local-model
`), 0o600); err != nil {
t.Fatal(err)
}
cfg := config.Default()
cfg.PromptKit.ProfileFile = profilePath
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
Endpoint: server.URL + "/v1",
ConcurrencyLimit: 2,
}
if err := validateExplicitPromptKitProfiles(context.Background(), cfg, []string{"local-profile"}); err != nil {
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
}
if providerCalls.Load() != 0 {
t.Fatalf("provider calls during configured profile validation = %d, want 0", providerCalls.Load())
}
cfg.PromptKit.LocalBackend = nil
err := validateExplicitPromptKitProfiles(context.Background(), cfg, []string{"local-profile"})
if err == nil ||
!strings.Contains(err.Error(), `validate PromptKit profile "local-profile"`) ||
!strings.Contains(err.Error(), promptkit.BackendLocal) {
t.Fatalf("validation without registration error = %v, want profile and local backend context", err)
}
if providerCalls.Load() != 0 {
t.Fatalf("provider calls after missing-registration validation = %d, want 0", providerCalls.Load())
}
}

View File

@@ -16,8 +16,14 @@ type Config struct {
} }
type PromptKitConfig struct { type PromptKitConfig struct {
ProfileDir string `json:"profile_dir,omitempty"` ProfileDir string `json:"profile_dir,omitempty"`
ProfileFile string `json:"profile_file,omitempty"` ProfileFile string `json:"profile_file,omitempty"`
LocalBackend *PromptKitLocalBackendConfig `json:"local_backend,omitempty"`
}
type PromptKitLocalBackendConfig struct {
Endpoint string `json:"endpoint"`
ConcurrencyLimit int `json:"concurrency_limit"`
} }
type ConcurrencyConfig struct { type ConcurrencyConfig struct {
@@ -66,6 +72,10 @@ func Default() Config {
func cloneConfig(in Config) Config { func cloneConfig(in Config) Config {
out := in out := in
if in.PromptKit.LocalBackend != nil {
localBackend := *in.PromptKit.LocalBackend
out.PromptKit.LocalBackend = &localBackend
}
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers) out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines)) out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
for key, profile := range in.Pipelines { for key, profile := range in.Pipelines {

View File

@@ -107,6 +107,33 @@ func TestEffectiveConfigPreservesPromptKitProfileSource(t *testing.T) {
} }
} }
func TestEffectiveConfigOwnsPromptKitLocalBackend(t *testing.T) {
cfg := configForEffectiveTests(t, effectiveProfile())
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
Endpoint: "http://localhost:8000/v1",
ConcurrencyLimit: 2,
}
effective, err := cfg.Resolve(ResolveInput{PipelineID: "main", Catalog: effectiveCatalog(t)})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if effective.Config.PromptKit.LocalBackend == nil {
t.Fatal("effective local backend = nil")
}
if effective.Config.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend {
t.Fatal("effective local backend aliases input config")
}
cfg.PromptKit.LocalBackend.Endpoint = "http://changed-input.example/v1"
if effective.Config.PromptKit.LocalBackend.Endpoint != "http://localhost:8000/v1" {
t.Fatalf("input mutation changed effective config: %#v", effective.Config.PromptKit.LocalBackend)
}
effective.Config.PromptKit.LocalBackend.ConcurrencyLimit = 9
if cfg.PromptKit.LocalBackend.ConcurrencyLimit != 2 {
t.Fatalf("effective mutation changed input config: %#v", cfg.PromptKit.LocalBackend)
}
}
func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) { func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
tests := []struct { tests := []struct {
name string name string

View File

@@ -23,8 +23,14 @@ type FileConfig struct {
} }
type FilePromptKitConfig struct { type FilePromptKitConfig struct {
ProfileDir *string `yaml:"profile_dir,omitempty"` ProfileDir *string `yaml:"profile_dir,omitempty"`
ProfileFile *string `yaml:"profile_file,omitempty"` ProfileFile *string `yaml:"profile_file,omitempty"`
LocalBackend *FilePromptKitLocalBackendConfig `yaml:"local_backend,omitempty"`
}
type FilePromptKitLocalBackendConfig struct {
Endpoint *string `yaml:"endpoint,omitempty"`
ConcurrencyLimit *int `yaml:"concurrency_limit,omitempty"`
} }
type FilePipelineProfile struct { type FilePipelineProfile struct {
@@ -468,6 +474,20 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
} }
c.PromptKit.ProfileFile = value c.PromptKit.ProfileFile = value
} }
if fileCfg.PromptKit.LocalBackend != nil {
if fileCfg.PromptKit.LocalBackend.Endpoint == nil {
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
}
endpoint := strings.TrimSpace(*fileCfg.PromptKit.LocalBackend.Endpoint)
if endpoint == "" {
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
}
localBackend := PromptKitLocalBackendConfig{Endpoint: endpoint}
if fileCfg.PromptKit.LocalBackend.ConcurrencyLimit != nil {
localBackend.ConcurrencyLimit = *fileCfg.PromptKit.LocalBackend.ConcurrencyLimit
}
c.PromptKit.LocalBackend = &localBackend
}
} }
for _, pipelineID := range pipelineIDs { for _, pipelineID := range pipelineIDs {

View File

@@ -97,6 +97,100 @@ func TestFilePromptKitProfileSourcesSurviveConfigBoundaries(t *testing.T) {
} }
} }
func TestFilePromptKitLocalBackendSurvivesConfigBoundaries(t *testing.T) {
tests := []struct {
name string
concurrencyYAML string
wantConcurrency int
}{
{name: "omitted concurrency defaults to zero"},
{name: "positive concurrency is preserved", concurrencyYAML: " concurrency_limit: 2\n", wantConcurrency: 2},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
file := parseFileConfig(t, "version: 4\npromptkit:\n local_backend:\n endpoint: ' http://localhost:8000/v1 '\n"+tt.concurrencyYAML)
cfg := Default()
if err := cfg.ApplyFileConfig(file); err != nil {
t.Fatalf("ApplyFileConfig() error = %v", err)
}
want := PromptKitLocalBackendConfig{
Endpoint: "http://localhost:8000/v1",
ConcurrencyLimit: tt.wantConcurrency,
}
if cfg.PromptKit.LocalBackend == nil || *cfg.PromptKit.LocalBackend != want {
t.Fatalf("local backend config = %#v, want %#v", cfg.PromptKit.LocalBackend, want)
}
*file.PromptKit.LocalBackend.Endpoint = "http://changed.example/v1"
if file.PromptKit.LocalBackend.ConcurrencyLimit != nil {
*file.PromptKit.LocalBackend.ConcurrencyLimit = 99
}
if *cfg.PromptKit.LocalBackend != want {
t.Fatalf("effective config aliases parsed file model: %#v", cfg.PromptKit.LocalBackend)
}
cloned := cloneConfig(cfg)
if cloned.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *cloned.PromptKit.LocalBackend != want {
t.Fatalf("cloned local backend = %#v, want detached %#v", cloned.PromptKit.LocalBackend, want)
}
cloned.PromptKit.LocalBackend.Endpoint = "http://clone.example/v1"
if *cfg.PromptKit.LocalBackend != want {
t.Fatalf("mutating clone changed source config: %#v", cfg.PromptKit.LocalBackend)
}
redacted := cfg.Redacted()
if redacted.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *redacted.PromptKit.LocalBackend != want {
t.Fatalf("redacted local backend = %#v, want detached %#v", redacted.PromptKit.LocalBackend, want)
}
data, err := json.Marshal(cfg)
if err != nil {
t.Fatalf("json.Marshal() error = %v", err)
}
var payload struct {
PromptKit map[string]json.RawMessage `json:"promptkit"`
}
if err := json.Unmarshal(data, &payload); err != nil {
t.Fatalf("json.Unmarshal() error = %v", err)
}
localJSON, ok := payload.PromptKit["local_backend"]
if !ok {
t.Fatalf("runtime PromptKit JSON keys = %v, want local_backend", payload.PromptKit)
}
var localPayload map[string]json.RawMessage
if err := json.Unmarshal(localJSON, &localPayload); err != nil {
t.Fatalf("unmarshal local_backend JSON: %v", err)
}
if _, ok := localPayload["endpoint"]; !ok {
t.Fatalf("runtime local_backend JSON keys = %v, want endpoint", localPayload)
}
if _, ok := localPayload["concurrency_limit"]; !ok {
t.Fatalf("runtime local_backend JSON keys = %v, want concurrency_limit", localPayload)
}
})
}
}
func TestFilePromptKitLocalBackendRequiresEndpoint(t *testing.T) {
for _, tt := range []struct {
name string
yaml string
}{
{name: "missing", yaml: "version: 4\npromptkit:\n local_backend: {}\n"},
{name: "empty", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ''\n"},
{name: "blank", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ' '\n"},
} {
t.Run(tt.name, func(t *testing.T) {
file := parseFileConfig(t, tt.yaml)
cfg := Default()
err := cfg.ApplyFileConfig(file)
if err == nil || !strings.Contains(err.Error(), "promptkit.local_backend.endpoint") {
t.Fatalf("ApplyFileConfig() error = %v, want endpoint field context", err)
}
})
}
}
func TestFilePromptKitExplicitEmptyProfileSourcesAreRejected(t *testing.T) { func TestFilePromptKitExplicitEmptyProfileSourcesAreRejected(t *testing.T) {
for _, field := range []string{"profile_dir", "profile_file"} { for _, field := range []string{"profile_dir", "profile_file"} {
t.Run(field, func(t *testing.T) { t.Run(field, func(t *testing.T) {
@@ -188,6 +282,11 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
yaml: "version: 4\ncache:\n checkpoints:\n enabled: definitely\n", yaml: "version: 4\ncache:\n checkpoints:\n enabled: definitely\n",
want: "cannot unmarshal", want: "cannot unmarshal",
}, },
{
name: "local backend field",
yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: http://localhost:8000/v1\n unknown: true\n",
want: "field unknown not found",
},
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {

View File

@@ -2,6 +2,7 @@ package config
import ( import (
"fmt" "fmt"
"net/url"
"sort" "sort"
"strings" "strings"
@@ -53,6 +54,27 @@ func validatePromptKit(cfg PromptKitConfig) error {
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" { if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
return fmt.Errorf("promptkit profile_dir and profile_file are mutually exclusive") return fmt.Errorf("promptkit profile_dir and profile_file are mutually exclusive")
} }
if cfg.LocalBackend == nil {
return nil
}
endpoint := strings.TrimSpace(cfg.LocalBackend.Endpoint)
if endpoint == "" {
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
}
parsed, err := url.Parse(endpoint)
if err != nil ||
(!strings.EqualFold(parsed.Scheme, "http") && !strings.EqualFold(parsed.Scheme, "https")) ||
!parsed.IsAbs() ||
parsed.Hostname() == "" ||
parsed.User != nil ||
parsed.RawQuery != "" ||
parsed.ForceQuery ||
strings.Contains(endpoint, "#") {
return fmt.Errorf("promptkit.local_backend.endpoint must be an absolute HTTP or HTTPS URL with a host and no user information, query, or fragment")
}
if cfg.LocalBackend.ConcurrencyLimit < 0 {
return fmt.Errorf("promptkit.local_backend.concurrency_limit must not be negative")
}
return nil return nil
} }

View File

@@ -93,6 +93,73 @@ func TestValidatePromptKitSourcesAreMutuallyExclusive(t *testing.T) {
assertValidationContains(t, cfg, "promptkit profile_dir and profile_file are mutually exclusive") assertValidationContains(t, cfg, "promptkit profile_dir and profile_file are mutually exclusive")
} }
func TestValidatePromptKitLocalBackendEndpoints(t *testing.T) {
tests := []struct {
name string
endpoint string
profileSource PromptKitConfig
}{
{
name: "HTTP endpoint with path and profile directory",
endpoint: "http://localhost:8000/v1",
profileSource: PromptKitConfig{ProfileDir: "./profiles"},
},
{
name: "case-insensitive HTTPS endpoint and profile file",
endpoint: "HTTPS://inference.example.test/api",
profileSource: PromptKitConfig{ProfileFile: "./profiles.yml"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := Default()
cfg.PromptKit = tt.profileSource
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
Endpoint: tt.endpoint,
ConcurrencyLimit: 2,
}
if err := cfg.Validate(); err != nil {
t.Fatalf("Validate() error = %v", err)
}
})
}
}
func TestValidatePromptKitLocalBackendRejectsInvalidValues(t *testing.T) {
tests := []struct {
name string
endpoint string
concurrencyLimit int
want string
}{
{name: "blank endpoint", endpoint: " ", want: "promptkit.local_backend.endpoint"},
{name: "relative URL", endpoint: "localhost:8000/v1", want: "promptkit.local_backend.endpoint"},
{name: "unsupported scheme", endpoint: "ftp://localhost/model", want: "promptkit.local_backend.endpoint"},
{name: "missing host", endpoint: "http:///v1", want: "promptkit.local_backend.endpoint"},
{name: "user information", endpoint: "http://user:secret@localhost/v1", want: "promptkit.local_backend.endpoint"},
{name: "query", endpoint: "http://localhost/v1?model=example", want: "promptkit.local_backend.endpoint"},
{name: "empty query", endpoint: "http://localhost/v1?", want: "promptkit.local_backend.endpoint"},
{name: "fragment", endpoint: "http://localhost/v1#model", want: "promptkit.local_backend.endpoint"},
{name: "empty fragment", endpoint: "http://localhost/v1#", want: "promptkit.local_backend.endpoint"},
{
name: "negative concurrency",
endpoint: "http://localhost:8000/v1",
concurrencyLimit: -1,
want: "promptkit.local_backend.concurrency_limit",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := Default()
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
Endpoint: tt.endpoint,
ConcurrencyLimit: tt.concurrencyLimit,
}
assertValidationContains(t, cfg, tt.want)
})
}
}
func TestValidateStateSurfaceRules(t *testing.T) { func TestValidateStateSurfaceRules(t *testing.T) {
tests := []struct { tests := []struct {
name string name string

View File

@@ -19,9 +19,15 @@ import (
const promptKitProviderName = "promptkit" const promptKitProviderName = "promptkit"
type PromptKitLocalBackendConfig struct {
Endpoint string
ConcurrencyLimit int
}
type PromptKitClientConfig struct { type PromptKitClientConfig struct {
ProfileDir string ProfileDir string
ProfileFile string ProfileFile string
LocalBackend *PromptKitLocalBackendConfig
Assets *AssetRegistry Assets *AssetRegistry
Timeout time.Duration Timeout time.Duration
HTTPClient *http.Client HTTPClient *http.Client
@@ -35,6 +41,7 @@ type PromptKitClient struct {
recorder *LLMProfileRecorder recorder *LLMProfileRecorder
profileDir string profileDir string
profileFile string profileFile string
localEndpoint string
reasoningEffort *string reasoningEffort *string
} }
@@ -46,6 +53,10 @@ type LLMProfileRecorder struct {
var _ contracts.StructuredLLMClient = (*PromptKitClient)(nil) var _ contracts.StructuredLLMClient = (*PromptKitClient)(nil)
var _ contracts.LLMProfileManifestProvider = (*PromptKitClient)(nil) var _ contracts.LLMProfileManifestProvider = (*PromptKitClient)(nil)
func PromptKitLocalBackendOption(cfg PromptKitLocalBackendConfig) promptkit.Option {
return promptkit.WithBackend(promptkit.LocalBackend(cfg.Endpoint, cfg.ConcurrencyLimit))
}
func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) { func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
if cfg.Assets == nil { if cfg.Assets == nil {
return nil, fmt.Errorf("PromptKit client assets must not be nil") return nil, fmt.Errorf("PromptKit client assets must not be nil")
@@ -60,6 +71,13 @@ func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
if profileFile := strings.TrimSpace(cfg.ProfileFile); profileFile != "" { if profileFile := strings.TrimSpace(cfg.ProfileFile); profileFile != "" {
options = append(options, promptkit.WithProfileFile(profileFile)) options = append(options, promptkit.WithProfileFile(profileFile))
} }
var localEndpoint string
if cfg.LocalBackend != nil {
localBackend := *cfg.LocalBackend
localBackend.Endpoint = strings.TrimSpace(localBackend.Endpoint)
localEndpoint = localBackend.Endpoint
options = append(options, PromptKitLocalBackendOption(localBackend))
}
options = append(options, cfg.EngineOptions...) options = append(options, cfg.EngineOptions...)
engine, err := promptkit.NewEngine(promptkit.Config{ engine, err := promptkit.NewEngine(promptkit.Config{
@@ -84,6 +102,7 @@ func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
recorder: recorder, recorder: recorder,
profileDir: strings.TrimSpace(cfg.ProfileDir), profileDir: strings.TrimSpace(cfg.ProfileDir),
profileFile: strings.TrimSpace(cfg.ProfileFile), profileFile: strings.TrimSpace(cfg.ProfileFile),
localEndpoint: localEndpoint,
reasoningEffort: reasoningEffort, reasoningEffort: reasoningEffort,
}, nil }, nil
} }
@@ -302,7 +321,11 @@ func (c *PromptKitClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint,
if err != nil { if err != nil {
return nil, err return nil, err
} }
return []CheckpointFingerprint{fingerprint}, nil fingerprints := []CheckpointFingerprint{fingerprint}
if c.localEndpoint != "" {
fingerprints = append(fingerprints, promptKitLocalBackendFingerprint(c.localEndpoint))
}
return fingerprints, nil
} }
func NewLLMProfileRecorder() *LLMProfileRecorder { func NewLLMProfileRecorder() *LLMProfileRecorder {

View File

@@ -6,6 +6,7 @@ import (
"errors" "errors"
"io" "io"
"net/http" "net/http"
"net/http/httptest"
"os" "os"
"path/filepath" "path/filepath"
"reflect" "reflect"
@@ -305,12 +306,89 @@ func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
if fresh[0].Value == "mutated" { if fresh[0].Value == "mutated" {
t.Fatal("LLMCheckpointFingerprints exposed mutable backing storage") t.Fatal("LLMCheckpointFingerprints exposed mutable backing storage")
} }
const wantBuiltinFingerprint = "sha256:4b48cd1ef010b587cd8d56bc13b73a0cdd84da1414a8de9f9a2adc695dfcf0a7" const wantBuiltinFingerprint = "sha256:5218b1dec48f5fdd46836826e0b25906c33efbdf943c5f18085b8d82467e0276"
if fresh[0].Value != wantBuiltinFingerprint { if fresh[0].Value != wantBuiltinFingerprint {
t.Fatalf("built-in profile fingerprint = %q, want %q", fresh[0].Value, wantBuiltinFingerprint) t.Fatalf("built-in profile fingerprint = %q, want %q", fresh[0].Value, wantBuiltinFingerprint)
} }
} }
func TestPromptKitClientCheckpointFingerprintTracksLocalBackendTarget(t *testing.T) {
const (
firstEndpoint = "http://localhost:8000/v1"
secondEndpoint = "https://inference.example.test/v1"
)
fingerprintsFor := func(endpoint string, concurrencyLimit int) []CheckpointFingerprint {
t.Helper()
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
LocalBackend: &PromptKitLocalBackendConfig{
Endpoint: endpoint,
ConcurrencyLimit: concurrencyLimit,
},
})
if err != nil {
t.Fatal(err)
}
values, err := client.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
return values
}
baseline := fingerprintsFor(firstEndpoint, 0)
if len(baseline) != 2 ||
baseline[0].Name != promptKitProfileFingerprintName ||
baseline[1].Name != promptKitLocalBackendFingerprintName {
t.Fatalf("checkpoint fingerprints = %#v, want profile source then local backend target", baseline)
}
endpointChanged := fingerprintsFor(secondEndpoint, 0)
if baseline[0] != endpointChanged[0] || baseline[1] == endpointChanged[1] {
t.Fatalf("endpoint fingerprints = %#v and %#v, want only local target to change", baseline, endpointChanged)
}
concurrencyChanged := fingerprintsFor(firstEndpoint, 4)
if !reflect.DeepEqual(baseline, concurrencyChanged) {
t.Fatalf("concurrency fingerprints = %#v, want %#v", concurrencyChanged, baseline)
}
for _, values := range [][]CheckpointFingerprint{baseline, endpointChanged} {
for _, fingerprint := range values {
if strings.Contains(fingerprint.Value, firstEndpoint) ||
strings.Contains(fingerprint.Value, secondEndpoint) {
t.Fatalf("checkpoint fingerprint exposes endpoint: %#v", fingerprint)
}
}
}
localBackend := &PromptKitLocalBackendConfig{
Endpoint: " " + firstEndpoint + " ",
ConcurrencyLimit: 0,
}
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
LocalBackend: localBackend,
})
if err != nil {
t.Fatal(err)
}
localBackend.Endpoint = secondEndpoint
copy, err := client.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(copy, baseline) {
t.Fatalf("fingerprints after input mutation = %#v, want retained target %#v", copy, baseline)
}
copy[0].Value = "mutated-profile"
copy[1].Value = "mutated-target"
fresh, err := client.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(fresh, baseline) {
t.Fatalf("fingerprints after returned-slice mutation = %#v, want %#v", fresh, baseline)
}
}
func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) { func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) {
fake := &fakePromptKitLLM{content: `{"ok":true}`} fake := &fakePromptKitLLM{content: `{"ok":true}`}
client := newTestPromptKitClient(t, fake) client := newTestPromptKitClient(t, fake)
@@ -334,6 +412,88 @@ func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testi
} }
} }
func TestPromptKitClientUsesConfiguredLocalBackend(t *testing.T) {
var providerCalls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
providerCalls.Add(1)
if r.URL.Path != "/v1/chat/completions" {
t.Errorf("provider path = %q, want /v1/chat/completions", r.URL.Path)
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{
"choices": [{"message": {"role": "assistant", "content": "{\"ok\":true}"}}],
"usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7}
}`))
}))
defer server.Close()
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
if err := os.WriteFile(profilePath, []byte(`id: local-profile
backend: local
model: local-model
`), 0o600); err != nil {
t.Fatal(err)
}
localBackend := &PromptKitLocalBackendConfig{
Endpoint: " " + server.URL + "/v1 ",
ConcurrencyLimit: 2,
}
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
ProfileFile: profilePath,
LocalBackend: localBackend,
})
if err != nil {
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
}
localBackend.Endpoint = "http://127.0.0.1:1/v1"
request := contracts.StructuredCompletionRequest{
PromptID: "adapter.direct-session",
ProfileID: "local-profile",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
Vars: map[string]any{"custom": "value"},
}
var out map[string]any
response, err := client.CompleteStructured(context.Background(), request, &out)
if err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", err)
}
if providerCalls.Load() != 1 {
t.Fatalf("provider calls = %d, want 1", providerCalls.Load())
}
if response.ProfileID != "local-profile" || response.Model != "local-model" {
t.Fatalf("response metadata = %#v", response)
}
if response.Debug == nil || response.Debug.Prompt == nil ||
response.Debug.Prompt.SelectedBackendID != promptkit.BackendLocal {
t.Fatalf("response debug prompt = %#v, want local backend", response.Debug)
}
manifests := client.LLMProfileManifests()
if len(manifests) != 1 || manifests[0].BackendID != promptkit.BackendLocal {
t.Fatalf("profile manifests = %#v, want local backend", manifests)
}
clientWithoutRegistration, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
ProfileFile: profilePath,
})
if err != nil {
t.Fatalf("NewPromptKitClient() without local registration error = %v, want nil", err)
}
_, err = clientWithoutRegistration.CompleteStructured(context.Background(), request, &out)
if err == nil ||
!strings.Contains(err.Error(), "prepare PromptKit prompt") ||
!strings.Contains(err.Error(), promptkit.BackendLocal) {
t.Fatalf("CompleteStructured() without registration error = %v, want preparation failure with local backend context", err)
}
if providerCalls.Load() != 1 {
t.Fatalf("provider calls after missing-registration failure = %d, want 1", providerCalls.Load())
}
}
func TestLLMProfileRecorderDistinguishesEffectiveTargets(t *testing.T) { func TestLLMProfileRecorderDistinguishesEffectiveTargets(t *testing.T) {
recorder := NewLLMProfileRecorder() recorder := NewLLMProfileRecorder()
for _, profile := range []artifacts.LLMProfileManifest{ for _, profile := range []artifacts.LLMProfileManifest{

View File

@@ -12,10 +12,12 @@ import (
) )
const ( const (
promptKitProfileFingerprintName = "promptkit_profile_source" promptKitProfileFingerprintName = "promptkit_profile_source"
promptKitLocalBackendFingerprintName = "promptkit_local_backend_target"
promptKitLocalBackendMarker = "notarius:promptkit-local-backend:v1"
// The built-in profile catalog is compiled into this pinned PromptKit // The built-in profile catalog is compiled into this pinned PromptKit
// release. Update this identity when the dependency is upgraded. // release. Update this identity when the dependency is upgraded.
promptKitBuiltinProfileCatalogID = "promptkit:v0.2.0:builtin-profiles" promptKitBuiltinProfileCatalogID = "promptkit:v0.3.0:builtin-profiles"
) )
func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFingerprint, error) { func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFingerprint, error) {
@@ -45,6 +47,16 @@ func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFing
}, nil }, nil
} }
func promptKitLocalBackendFingerprint(endpoint string) CheckpointFingerprint {
hasher := sha256.New()
writeFingerprintPart(hasher, []byte(promptKitLocalBackendMarker))
writeFingerprintPart(hasher, []byte(strings.TrimSpace(endpoint)))
return CheckpointFingerprint{
Name: promptKitLocalBackendFingerprintName,
Value: "sha256:" + hex.EncodeToString(hasher.Sum(nil)),
}
}
func promptKitProfileFileDigests(root string) ([][]byte, error) { func promptKitProfileFileDigests(root string) ([][]byte, error) {
var digests [][]byte var digests [][]byte
err := filepath.WalkDir(root, func(name string, entry fs.DirEntry, walkErr error) error { err := filepath.WalkDir(root, func(name string, entry fs.DirEntry, walkErr error) error {

View File

@@ -81,7 +81,10 @@ func TestScheduledClientPreservesCheckpointFingerprints(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
inner := &fingerprintedStructuredClient{ inner := &fingerprintedStructuredClient{
fingerprints: []CheckpointFingerprint{{Name: "profile_source", Value: "sha256:one"}}, fingerprints: []CheckpointFingerprint{
{Name: "profile_source", Value: "sha256:one"},
{Name: "backend_target", Value: "sha256:two"},
},
} }
client := NewScheduledClient(inner, scheduler) client := NewScheduledClient(inner, scheduler)
provider, ok := client.(CheckpointFingerprintProvider) provider, ok := client.(CheckpointFingerprintProvider)
@@ -92,7 +95,9 @@ func TestScheduledClientPreservesCheckpointFingerprints(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if len(got) != 1 || got[0] != inner.fingerprints[0] { if len(got) != len(inner.fingerprints) ||
got[0] != inner.fingerprints[0] ||
got[1] != inner.fingerprints[1] {
t.Fatalf("checkpoint fingerprints = %#v, want %#v", got, inner.fingerprints) t.Fatalf("checkpoint fingerprints = %#v, want %#v", got, inner.fingerprints)
} }
} }