Compare commits
7 Commits
a16dcdfa52
...
b5aaeb1c78
| Author | SHA1 | Date | |
|---|---|---|---|
| b5aaeb1c78 | |||
| 9171b66a41 | |||
| b4363b3b73 | |||
| 241e9d2a89 | |||
| 715fff7b72 | |||
| d627b91b4f | |||
| a67b3aa76d |
@@ -46,7 +46,7 @@ other than **version** is optional.
|
|||||||
| Field | Type | Default | Rules |
|
| Field | Type | Default | Rules |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| **version** | integer | none | Required; must be 4. |
|
| **version** | integer | none | Required; must be 4. |
|
||||||
| **promptkit** | object | none | Profile source configuration. |
|
| **promptkit** | object | none | Profile source and optional local-backend configuration. |
|
||||||
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
|
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
|
||||||
| **concurrency** | object | see below | Global LLM and extraction limits. |
|
| **concurrency** | object | see below | Global LLM and extraction limits. |
|
||||||
| **output** | object | see below | Published output settings. |
|
| **output** | object | see below | Published output settings. |
|
||||||
@@ -71,27 +71,58 @@ per-user root. An explicit empty output or debug directory is invalid.
|
|||||||
|
|
||||||
## PromptKit Profiles
|
## PromptKit Profiles
|
||||||
|
|
||||||
The optional **promptkit** object selects one source of profile definitions:
|
The optional **promptkit** object selects one source of profile definitions and
|
||||||
|
may register one conventional local OpenAI-compatible backend:
|
||||||
|
|
||||||
~~~yaml
|
~~~yaml
|
||||||
version: 4
|
version: 4
|
||||||
|
|
||||||
promptkit:
|
promptkit:
|
||||||
profile_dir: /path/to/profiles
|
profile_dir: ./profiles
|
||||||
# profile_file: /path/to/profiles.yml
|
# profile_file: ./profiles.yml
|
||||||
|
local_backend:
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
concurrency_limit: 2
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
| Field | Type | Rules |
|
| Field | Type | Rules |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| **profile_dir** | string | Non-empty directory containing profile files. |
|
| **profile_dir** | string | Non-empty directory containing profile files. |
|
||||||
| **profile_file** | string | Non-empty profile file. |
|
| **profile_file** | string | Non-empty profile file. |
|
||||||
|
| **local_backend** | object | Optional registration for the conventional PromptKit backend ID **local**. |
|
||||||
|
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
|
||||||
|
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
|
||||||
|
|
||||||
Set at most one of these fields. Profile IDs used by a binding must be available
|
Set at most one of **profile_dir** and **profile_file**. Profile IDs used by a
|
||||||
from the selected PromptKit profile source when the pipeline is resolved.
|
binding must be available from the selected PromptKit profile source when the
|
||||||
Keep credentials out of this file: configure a profile to read its credential
|
pipeline is resolved. The optional local registration may coexist with either
|
||||||
from an environment variable, then set that environment variable only in the
|
profile source or with PromptKit's built-in profiles.
|
||||||
run environment. PromptKit owns the profile-file format; see the
|
|
||||||
[PromptKit upstream boundary](integrations/pkg-promptkit.md) for the pinned
|
When **local_backend** is present, its endpoint is trimmed and must use HTTP or
|
||||||
package and canonical format reference.
|
HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are
|
||||||
|
allowed. User information, queries, and fragments are rejected. A zero
|
||||||
|
**concurrency_limit** leaves the local backend unrestricted inside PromptKit;
|
||||||
|
a positive value limits simultaneous local generations. The application-wide
|
||||||
|
**concurrency.total_llm** limit still applies in both cases. Neither local
|
||||||
|
backend field has an environment override. Omitting **local_backend** registers
|
||||||
|
nothing and preserves existing built-in and endpoint-only profile behavior.
|
||||||
|
|
||||||
|
A file-backed PromptKit profile selects the registration by its case-sensitive
|
||||||
|
backend ID:
|
||||||
|
|
||||||
|
~~~yaml
|
||||||
|
id: local-summary
|
||||||
|
backend: local
|
||||||
|
model: example-model
|
||||||
|
~~~
|
||||||
|
|
||||||
|
Keep credentials out of the local-backend object. A PromptKit profile may name
|
||||||
|
its credential environment variable through `api_key_env`; set that variable
|
||||||
|
only in the run environment. PromptKit owns the
|
||||||
|
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/formats.md).
|
||||||
|
The [PromptKit upstream boundary](integrations/pkg-promptkit.md) identifies the
|
||||||
|
supported package API, and [Operations](operations.md#operational-limits)
|
||||||
|
describes the effective concurrency layers.
|
||||||
|
|
||||||
## Migrating Version 3 Configuration
|
## Migrating Version 3 Configuration
|
||||||
|
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
# PromptKit Integration
|
# PromptKit Integration
|
||||||
|
|
||||||
Notarius pins
|
Notarius pins
|
||||||
[`gitea.maximumdirect.net/eric/promptkit` v0.2.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0)
|
[`gitea.maximumdirect.net/eric/promptkit` v0.3.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0)
|
||||||
as its in-process prompt engine. The upstream
|
as its in-process prompt engine. The upstream
|
||||||
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0/docs/consumers/pkg-promptkit.md)
|
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/consumers/pkg-promptkit.md)
|
||||||
owns the public engine API, and the upstream
|
owns the public engine API, and the upstream
|
||||||
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0/docs/formats.md)
|
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/formats.md)
|
||||||
owns prompt, profile, and schema file contracts.
|
owns prompt, profile, and schema file contracts.
|
||||||
|
|
||||||
## Supported Boundary
|
## Supported Boundary
|
||||||
@@ -18,10 +18,16 @@ Notarius relies on the root `promptkit` package to:
|
|||||||
a direct session ID, prompt identity, and profile selection;
|
a direct session ID, prompt identity, and profile selection;
|
||||||
- return rendered debug material, validated structured output, selected
|
- return rendered debug material, validated structured output, selected
|
||||||
profile, backend, effective model metadata, and token usage;
|
profile, backend, effective model metadata, and token usage;
|
||||||
|
- register the optional conventional `local` backend through `BackendLocal`,
|
||||||
|
`LocalBackend`, and `WithBackend`;
|
||||||
- distinguish structured-output validation failure from execution failure; and
|
- distinguish structured-output validation failure from execution failure; and
|
||||||
- identify a missing explicit profile through `ErrProfileNotFound` and backend
|
- identify a missing explicit profile through `ErrProfileNotFound` and backend
|
||||||
admission exhaustion through `ErrCapacityExceeded`.
|
admission exhaustion through `ErrCapacityExceeded`.
|
||||||
|
|
||||||
|
The pinned
|
||||||
|
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/backends.go)
|
||||||
|
owns the registration and backend-capacity contract.
|
||||||
|
|
||||||
Notarius does not use PromptKit's optional `ArtifactReader`. It materializes
|
Notarius does not use PromptKit's optional `ArtifactReader`. It materializes
|
||||||
source and reference content itself and supplies owned inline artifacts at the
|
source and reference content itself and supplies owned inline artifacts at the
|
||||||
adapter boundary. It also retains responsibility for pipeline retries,
|
adapter boundary. It also retains responsibility for pipeline retries,
|
||||||
@@ -38,8 +44,9 @@ Notarius records PromptKit's selected backend ID and effective reasoning
|
|||||||
setting as optional run-manifest provenance. Endpoint-only profiles have no
|
setting as optional run-manifest provenance. Endpoint-only profiles have no
|
||||||
backend ID. Debug prompt material also retains the selected backend ID and
|
backend ID. Debug prompt material also retains the selected backend ID and
|
||||||
PromptKit's stable lower-case `effective_model_params` JSON, which may include
|
PromptKit's stable lower-case `effective_model_params` JSON, which may include
|
||||||
`backend_id`. Notarius production configuration does not expose user-defined
|
`backend_id`. Notarius production configuration exposes one optional
|
||||||
PromptKit backend registration.
|
conventional `local` registration. It does not expose a general user-defined
|
||||||
|
PromptKit backend registry. Endpoint-only profiles remain supported unchanged.
|
||||||
|
|
||||||
Notarius retains its application-wide scheduled client around the PromptKit
|
Notarius retains its application-wide scheduled client around the PromptKit
|
||||||
adapter. PromptKit may apply a narrower limit for the selected backend;
|
adapter. PromptKit may apply a narrower limit for the selected backend;
|
||||||
@@ -55,7 +62,8 @@ module assets, maps its transport-neutral completion contract, prepares and
|
|||||||
executes requests, validates output, records provenance, captures debug
|
executes requests, validates output, records provenance, captures debug
|
||||||
material, redacts errors, and preserves timeout ownership.
|
material, redacts errors, and preserves timeout ownership.
|
||||||
[Configuration](../config.md#promptkit-profiles) defines how a Notarius
|
[Configuration](../config.md#promptkit-profiles) defines how a Notarius
|
||||||
configuration selects one PromptKit profile source.
|
configuration selects one PromptKit profile source and optionally registers
|
||||||
|
the conventional local backend.
|
||||||
|
|
||||||
PromptKit API or format changes outside this boundary are not implicitly
|
PromptKit API or format changes outside this boundary are not implicitly
|
||||||
supported. Updating the pinned version requires reviewing the adapter and
|
supported. Updating the pinned version requires reviewing the adapter and
|
||||||
|
|||||||
@@ -39,14 +39,20 @@ This establishes the public precedence order without giving environment input a
|
|||||||
second file schema. Loading and application reject malformed YAML, unsupported
|
second file schema. Loading and application reject malformed YAML, unsupported
|
||||||
file versions, unknown fields, invalid values, and identifiers that are empty
|
file versions, unknown fields, invalid values, and identifiers that are empty
|
||||||
or collide after whitespace normalization. The file application also makes the
|
or collide after whitespace normalization. The file application also makes the
|
||||||
effective extraction-worker default follow the effective LLM limit.
|
effective extraction-worker default follow the effective LLM limit. A present
|
||||||
|
PromptKit local-backend object requires and trims its endpoint, defaults its
|
||||||
|
omitted concurrency limit to zero, and is copied so the parsed file model
|
||||||
|
cannot alias the populated **Config**.
|
||||||
|
|
||||||
**Config.Validate** checks configuration-only invariants before resolution. It
|
**Config.Validate** checks configuration-only invariants before resolution. It
|
||||||
rejects incompatible profile sources, invalid state-surface values, unsupported
|
rejects incompatible profile sources, invalid state-surface values, unsupported
|
||||||
concurrency settings, malformed bindings and references, invalid retries, and
|
concurrency settings, malformed bindings and references, invalid retries, and
|
||||||
invalid pipeline, step, or lane structure. Its errors retain the closest known
|
invalid pipeline, step, or lane structure. PromptKit local-backend validation
|
||||||
pipeline, lane, and binding context. It deliberately does not require modules
|
accepts only an absolute HTTP or HTTPS endpoint with a host and no user
|
||||||
to be registered: that requires a catalog and belongs to resolution.
|
information, query, or fragment, and rejects a negative local concurrency
|
||||||
|
limit. Its errors retain the closest known pipeline, lane, and binding context.
|
||||||
|
It deliberately does not require modules to be registered: that requires a
|
||||||
|
catalog and belongs to resolution.
|
||||||
|
|
||||||
The exact user-selectable values and validation rules are defined in
|
The exact user-selectable values and validation rules are defined in
|
||||||
[Configuration](../config.md). Keep additions to the file model, an
|
[Configuration](../config.md). Keep additions to the file model, an
|
||||||
@@ -71,7 +77,8 @@ options, and returns the fixed ordered pipeline shape. The resulting
|
|||||||
changes, a clone of the input configuration, and the resolved pipeline.
|
changes, a clone of the input configuration, and the resolved pipeline.
|
||||||
Callers may therefore retain or modify their input slices and maps without
|
Callers may therefore retain or modify their input slices and maps without
|
||||||
changing the resolved result, and later consumers cannot mutate the original
|
changing the resolved result, and later consumers cannot mutate the original
|
||||||
configuration through the effective value.
|
configuration through the effective value. This ownership includes the nested
|
||||||
|
PromptKit local-backend value.
|
||||||
|
|
||||||
Resolution failures stop before module construction and source parsing. They
|
Resolution failures stop before module construction and source parsing. They
|
||||||
include an error path for an unconfigured pipeline, missing module, missing
|
include an error path for an unconfigured pipeline, missing module, missing
|
||||||
@@ -94,9 +101,12 @@ Configuration summaries must use **Redacted**, **RedactedSummaryPayload**, or
|
|||||||
Those methods copy every binding and nested option container, replace values
|
Those methods copy every binding and nested option container, replace values
|
||||||
whose key is credential-shaped with **[REDACTED]**, and omit materialized
|
whose key is credential-shaped with **[REDACTED]**, and omit materialized
|
||||||
reference content while retaining safe binding and reference provenance. The
|
reference content while retaining safe binding and reference provenance. The
|
||||||
payload must not alias the source configuration or resolved pipeline. This
|
payload must not alias the source configuration or resolved pipeline.
|
||||||
redaction is deliberately narrow: it protects configuration summaries and does
|
PromptKit's local endpoint and concurrency limit are preserved as non-secret
|
||||||
not authorize recording arbitrary environment values or provider requests.
|
configuration metadata in the independently owned summary; the object contains
|
||||||
|
no credential value. This redaction is deliberately narrow: it protects
|
||||||
|
configuration summaries and does not authorize recording arbitrary environment
|
||||||
|
values or provider requests.
|
||||||
|
|
||||||
## Invariants To Preserve
|
## Invariants To Preserve
|
||||||
|
|
||||||
|
|||||||
@@ -56,6 +56,13 @@ distinct and deterministically ordered. Endpoint-only profiles retain an empty
|
|||||||
backend ID, which the published JSON omits. Successful completion responses and
|
backend ID, which the published JSON omits. Successful completion responses and
|
||||||
recorded profile manifests identify the adapter provider as `promptkit`.
|
recorded profile manifests identify the adapter provider as `promptkit`.
|
||||||
|
|
||||||
|
The CLI's preparation-only engine and the production adapter use the same
|
||||||
|
conversion helper to register the optional conventional `local` backend.
|
||||||
|
Preflight therefore resolves the same backend membership as runtime without
|
||||||
|
performing generation. When the registration is absent, a profile selecting
|
||||||
|
`backend: local` fails preparation instead of falling back to a built-in or
|
||||||
|
endpoint-only target.
|
||||||
|
|
||||||
Before execution, the adapter also contributes a non-secret checkpoint
|
Before execution, the adapter also contributes a non-secret checkpoint
|
||||||
fingerprint for the effective PromptKit profile source. It combines the
|
fingerprint for the effective PromptKit profile source. It combines the
|
||||||
identity of PromptKit's compiled-in profile catalog with a deterministic digest
|
identity of PromptKit's compiled-in profile catalog with a deterministic digest
|
||||||
@@ -65,7 +72,11 @@ paths. It covers both explicit binding profiles and prompt-selected defaults,
|
|||||||
so changing a model or other profile setting cannot reuse checkpoints created
|
so changing a model or other profile setting cannot reuse checkpoints created
|
||||||
under the prior profile source. This cache identity is independent of durable
|
under the prior profile source. This cache identity is independent of durable
|
||||||
profile provenance: run manifests continue to list only profiles actually
|
profile provenance: run manifests continue to list only profiles actually
|
||||||
observed during LLM calls.
|
observed during LLM calls. When the local backend is registered, a second
|
||||||
|
fingerprint hashes its trimmed endpoint behind a stable marker. Changing that
|
||||||
|
semantic execution target invalidates checkpoint reuse. The raw endpoint is not
|
||||||
|
stored in checkpoint identity, and the local concurrency limit is excluded
|
||||||
|
because it changes scheduling rather than execution semantics.
|
||||||
|
|
||||||
## Shared Provider-Call Limit
|
## Shared Provider-Call Limit
|
||||||
|
|
||||||
@@ -85,8 +96,10 @@ PromptKit applies a second, independent admission limit when the selected
|
|||||||
profile names a limited backend. It sits beneath the Notarius scheduled client,
|
profile names a limited backend. It sits beneath the Notarius scheduled client,
|
||||||
so it may narrow but cannot expand the application-wide limit. Built-in
|
so it may narrow but cannot expand the application-wide limit. Built-in
|
||||||
OpenRouter profiles select PromptKit's reserved backend and its upstream
|
OpenRouter profiles select PromptKit's reserved backend and its upstream
|
||||||
capacity policy. Endpoint-only profiles do not select a PromptKit backend and
|
capacity policy. A positive configured local-backend limit bounds active local
|
||||||
remain limited only by the Notarius scheduler.
|
generations inside PromptKit; zero leaves that backend unlimited there.
|
||||||
|
Endpoint-only profiles do not select a PromptKit backend and remain limited
|
||||||
|
only by the Notarius scheduler.
|
||||||
|
|
||||||
## Prompt And Schema Assets
|
## Prompt And Schema Assets
|
||||||
|
|
||||||
|
|||||||
@@ -118,8 +118,10 @@ disabled. Without **--resume**, a recording-enabled run executes normally and
|
|||||||
does not load checkpoint state. Compatibility includes the resolved pipeline,
|
does not load checkpoint state. Compatibility includes the resolved pipeline,
|
||||||
input, selected lanes, runtime overrides, reference provenance, LLM-profile
|
input, selected lanes, runtime overrides, reference provenance, LLM-profile
|
||||||
provenance, the effective PromptKit profile-source fingerprint, and
|
provenance, the effective PromptKit profile-source fingerprint, and
|
||||||
prepared-component fingerprints. Changing profile content causes a cold miss
|
prepared-component fingerprints. When a local PromptKit backend is configured,
|
||||||
even when its profile ID is unchanged. A changed identity produces a cold miss;
|
compatibility also includes a non-secret fingerprint of its endpoint. Changing
|
||||||
|
profile content or the local endpoint causes a cold miss; changing only the
|
||||||
|
local concurrency limit does not. A changed identity produces a cold miss;
|
||||||
Notarius does not migrate, rewrite, or delete older checkpoint directories.
|
Notarius does not migrate, rewrite, or delete older checkpoint directories.
|
||||||
Reasoning-effort inheritance, replacement, and explicit clearing are distinct
|
Reasoning-effort inheritance, replacement, and explicit clearing are distinct
|
||||||
runtime identities, so checkpoints created under one state are not reused by
|
runtime identities, so checkpoints created under one state are not reused by
|
||||||
@@ -217,7 +219,7 @@ Provider execution settings and the generation timeout come from the selected
|
|||||||
PromptKit profile. The invocation-only **--reasoning-effort** and
|
PromptKit profile. The invocation-only **--reasoning-effort** and
|
||||||
**--clear-reasoning-effort** controls may replace or clear that profile setting
|
**--clear-reasoning-effort** controls may replace or clear that profile setting
|
||||||
for all LLM-backed calls in one run without changing the profile. PromptKit
|
for all LLM-backed calls in one run without changing the profile. PromptKit
|
||||||
v0.2.0 does not add a provider retry loop. Notarius binding retries rerun the
|
v0.3.0 does not add a provider retry loop. Notarius binding retries rerun the
|
||||||
complete module operation and validation chain as defined by
|
complete module operation and validation chain as defined by
|
||||||
[module bindings](config.md#module-bindings-and-validators).
|
[module bindings](config.md#module-bindings-and-validators).
|
||||||
|
|
||||||
@@ -234,13 +236,19 @@ and validators. PromptKit may impose a narrower admission limit for the
|
|||||||
selected backend. The effective active-generation bound is the intersection of
|
selected backend. The effective active-generation bound is the intersection of
|
||||||
both limits and can therefore be lower than **total_llm**. Built-in OpenRouter
|
both limits and can therefore be lower than **total_llm**. Built-in OpenRouter
|
||||||
profiles use PromptKit's upstream backend limit; endpoint-only profiles have no
|
profiles use PromptKit's upstream backend limit; endpoint-only profiles have no
|
||||||
PromptKit backend limit and remain bounded by Notarius.
|
PromptKit backend limit and remain bounded by Notarius. For the configured
|
||||||
|
local backend, a zero **concurrency_limit** leaves only the Notarius scheduler
|
||||||
|
as a call limit. A positive value makes the effective active local-generation
|
||||||
|
bound the smaller of **total_llm** and that local limit.
|
||||||
|
|
||||||
When a PromptKit backend has admitted all active and queued work, a new call
|
For a positive local limit, PromptKit owns its default waiting capacity and
|
||||||
fails as capacity exhaustion before generation. The adapter does not retry it.
|
admission behavior. When a PromptKit backend has admitted all active and queued
|
||||||
The calling stage's configured retry policy applies normally, and the run fails
|
work, a new call fails as capacity exhaustion before generation. The adapter
|
||||||
if those attempts are exhausted. Caller cancellation remains authoritative.
|
maps that failure to Notarius's existing provider-neutral capacity error and
|
||||||
Configuration contracts are documented under
|
does not retry it. The calling stage's configured retry policy applies
|
||||||
|
normally, and the run fails if those attempts are exhausted. Caller
|
||||||
|
cancellation remains authoritative. Configuration contracts are documented
|
||||||
|
under [PromptKit profiles](config.md#promptkit-profiles) and
|
||||||
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
|
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
|
||||||
limits and actual provider-call limits are independent. Notarius writes local
|
limits and actual provider-call limits are independent. Notarius writes local
|
||||||
filesystem state only; remote storage, archival, and retention automation are
|
filesystem state only; remote storage, archival, and retention automation are
|
||||||
|
|||||||
@@ -53,6 +53,120 @@ not as committed release dates.
|
|||||||
spell, combat, interaction, and scene-description lanes after real-world use.
|
spell, combat, interaction, and scene-description lanes after real-world use.
|
||||||
Add more complex chunking only in response to demonstrated failures.
|
Add more complex chunking only in response to demonstrated failures.
|
||||||
|
|
||||||
|
## Cross-Cutting LLM Runtime
|
||||||
|
|
||||||
|
### Deterministic Prompt Session Identity
|
||||||
|
|
||||||
|
- Replace the source-document-ID default for prompt sessions with one
|
||||||
|
predictable, procedurally generated session ID for the complete
|
||||||
|
source-processing workload.
|
||||||
|
- Preserve an explicit non-empty `--session-id` as the highest-precedence
|
||||||
|
override. Otherwise, derive the default only from the effective input module
|
||||||
|
identity and the exact raw input bytes.
|
||||||
|
- Use a versioned, bounded representation such as
|
||||||
|
`notarius:v1:<sha256(input-module + NUL + raw-input)>`. The exact encoding
|
||||||
|
must fit PromptKit's session length contract and must not embed source
|
||||||
|
content.
|
||||||
|
- Keep the derived session stable across runs, pipelines, selected lanes,
|
||||||
|
ordered steps, retries, resume, recomputation, LLM profiles, reasoning
|
||||||
|
overrides, and output, debug, or cache settings.
|
||||||
|
- Do not include file-backed references, generated references, reference
|
||||||
|
contents, or the composition of a reference bundle in session derivation.
|
||||||
|
References may change between prompt calls within one pipeline without
|
||||||
|
changing routing affinity.
|
||||||
|
- Resolve the authoritative session before checkpoint construction and use the
|
||||||
|
same value for checkpoint runtime identity, every prompt-facing module,
|
||||||
|
PromptKit's direct session field, the compatibility `session_id` prompt
|
||||||
|
variable, run-manifest metadata, and debug metadata.
|
||||||
|
- Keep routing identity separate from cache and checkpoint content identity.
|
||||||
|
Exact prompt prefixes, reference contents, model settings, and other
|
||||||
|
generation-affecting inputs must continue to participate in their existing
|
||||||
|
hashes and checkpoint fingerprints even though they do not change the
|
||||||
|
session.
|
||||||
|
- Treat the generated value as a provider-visible, stable pseudonymous
|
||||||
|
correlation identifier. Do not introduce an installation-specific HMAC or
|
||||||
|
secret unless a concrete multi-tenant or privacy requirement justifies
|
||||||
|
sacrificing deterministic identity across installations.
|
||||||
|
|
||||||
|
### Pipeline-Level LLM Profile Defaults
|
||||||
|
|
||||||
|
- Add an optional pipeline-level `llm_profile` default so an operator can
|
||||||
|
select one PromptKit execution policy for the pipeline without repeating the
|
||||||
|
same profile ID on every LLM-backed module binding.
|
||||||
|
- Apply the following precedence consistently: an explicit run-wide
|
||||||
|
`--llm-profile` override, then a binding-specific `llm_profile`, then the
|
||||||
|
pipeline-level default, then the prompt definition's embedded
|
||||||
|
`default_profile`.
|
||||||
|
- Apply inheritance only to bindings whose resolved modules are LLM-backed,
|
||||||
|
including applicable chunk, extraction, merge, normalization, and validation
|
||||||
|
bindings. Do not attach an inherited profile to deterministic modules or
|
||||||
|
weaken existing validation that rejects profiles where generation is not
|
||||||
|
supported.
|
||||||
|
- Resolve inherited profiles before effective-pipeline validation, digest and
|
||||||
|
checkpoint construction, execution, and provenance capture. Validate every
|
||||||
|
resulting explicit profile ID against the selected PromptKit profile source,
|
||||||
|
and ensure manifests and debug output report the profile actually selected
|
||||||
|
for each generation target.
|
||||||
|
- Preserve binding-specific profiles as intentional exceptions for modules
|
||||||
|
that require a different quality, latency, cost, provider, or reasoning
|
||||||
|
policy. Preserve `--llm-profile` as the convenient highest-precedence
|
||||||
|
experiment or incident-response override for an entire run.
|
||||||
|
- Treat PromptKit profiles as execution-policy configuration and prefer stable,
|
||||||
|
workload-oriented IDs such as `dnd-extraction` over model names. A pipeline
|
||||||
|
should express the kind of work it performs rather than encode a particular
|
||||||
|
provider, model, or deployment environment.
|
||||||
|
- Establish deployment-managed PromptKit profile files as the recommended
|
||||||
|
environment-specific configuration mechanism. Production, development, and
|
||||||
|
local deployments may each define the same logical `dnd-extraction` ID with
|
||||||
|
different model and generation settings, while retaining one unchanged
|
||||||
|
Notarius pipeline definition.
|
||||||
|
- Keep deployment profiles distinct from both Notarius prompt assets embedded
|
||||||
|
in the application binary and PromptKit's built-in profile catalog. Document
|
||||||
|
that `promptkit.profile_dir` or `promptkit.profile_file` selects an external
|
||||||
|
filesystem source whose definitions overlay PromptKit built-ins, and
|
||||||
|
recommend application-owned IDs rather than silently replacing built-in
|
||||||
|
profile IDs.
|
||||||
|
- Document an operator-friendly layout in which the Notarius configuration and
|
||||||
|
a profile subdirectory are deployed together. Until profile paths are
|
||||||
|
explicitly resolved relative to the configuration file, clearly state that
|
||||||
|
relative paths use the process working directory and recommend absolute
|
||||||
|
paths for services and containers.
|
||||||
|
- Update the configuration reference, operations guide, PromptKit integration
|
||||||
|
boundary, relevant internal configuration and pipeline documentation, and
|
||||||
|
maintained D&D examples together. Include one concrete external-profile
|
||||||
|
example and explain the profile precedence and environment-neutral pipeline
|
||||||
|
pattern without duplicating PromptKit's complete profile-format reference.
|
||||||
|
|
||||||
|
### Raise The Default Application-Wide LLM Limit
|
||||||
|
|
||||||
|
- Raise the default `concurrency.total_llm` value from 1 to 16 so ordinary
|
||||||
|
single-backend runs can use PromptKit's expected OpenRouter capacity and
|
||||||
|
lower-capacity local backends without an unnecessarily narrower Notarius
|
||||||
|
limit.
|
||||||
|
- Keep the Notarius application-wide scheduler mandatory and require
|
||||||
|
`total_llm` to remain a positive integer. Do not make the default unlimited:
|
||||||
|
endpoint-only profiles, an unrestricted local backend, injected clients, and
|
||||||
|
aggregate work across several backends may have no narrower PromptKit limit.
|
||||||
|
- Continue defaulting `concurrency.stage_workers.extract` to the effective
|
||||||
|
`total_llm`, making its default 16 as part of the same change. Preserve an
|
||||||
|
explicit lower extract-worker setting when an operator wants less queued or
|
||||||
|
concurrent extraction work.
|
||||||
|
- Define effective provider concurrency as the intersection of the Notarius
|
||||||
|
application-wide limit, the selected PromptKit backend limit when present,
|
||||||
|
and the work made available by stage execution. A Notarius limit of 16 does
|
||||||
|
not narrow a backend already limited to 16, while a local backend limited to
|
||||||
|
4 remains bounded at 4.
|
||||||
|
- Treat the default as an application-wide safety ceiling across profiles,
|
||||||
|
backends, modules, retries, and validators. A run that intentionally needs
|
||||||
|
the combined capacity of several backends may configure a higher
|
||||||
|
`total_llm` and an appropriate extract-worker count explicitly.
|
||||||
|
- Retain the existing configuration and environment override surfaces. Update
|
||||||
|
canonical configuration, operations, and internal documentation together
|
||||||
|
when the default changes.
|
||||||
|
- Reconsider decoupling the extract-worker default from `total_llm` only after
|
||||||
|
mixed-backend workloads demonstrate a need for a high global emergency
|
||||||
|
ceiling with a lower default work-production rate.
|
||||||
|
|
||||||
## Shared Normalization And Quality Work
|
## Shared Normalization And Quality Work
|
||||||
|
|
||||||
### Generic LLM-Assisted Deduplication
|
### Generic LLM-Assisted Deduplication
|
||||||
|
|||||||
2
go.mod
2
go.mod
@@ -3,7 +3,7 @@ module gitea.maximumdirect.net/eric/notarius
|
|||||||
go 1.25.5
|
go 1.25.5
|
||||||
|
|
||||||
require (
|
require (
|
||||||
gitea.maximumdirect.net/eric/promptkit v0.2.0
|
gitea.maximumdirect.net/eric/promptkit v0.3.0
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
||||||
gopkg.in/yaml.v3 v3.0.1
|
gopkg.in/yaml.v3 v3.0.1
|
||||||
)
|
)
|
||||||
|
|||||||
4
go.sum
4
go.sum
@@ -1,5 +1,5 @@
|
|||||||
gitea.maximumdirect.net/eric/promptkit v0.2.0 h1:6bUATKnVgLlHSbKjGb8EnCt2r7jV5Xw6t+pX7G9S/lU=
|
gitea.maximumdirect.net/eric/promptkit v0.3.0 h1:P5/GJ6fVIVsBLJx8NxoaGV34ZWeAkU8lm4Ht6aVMOFg=
|
||||||
gitea.maximumdirect.net/eric/promptkit v0.2.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
gitea.maximumdirect.net/eric/promptkit v0.3.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||||
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
||||||
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
||||||
|
|||||||
@@ -171,6 +171,7 @@ func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID
|
|||||||
client, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{
|
client, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{
|
||||||
ProfileDir: cfg.PromptKit.ProfileDir,
|
ProfileDir: cfg.PromptKit.ProfileDir,
|
||||||
ProfileFile: cfg.PromptKit.ProfileFile,
|
ProfileFile: cfg.PromptKit.ProfileFile,
|
||||||
|
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
|
||||||
Assets: assets,
|
Assets: assets,
|
||||||
Recorder: recorder,
|
Recorder: recorder,
|
||||||
ReasoningEffort: overrides.ReasoningEffort,
|
ReasoningEffort: overrides.ReasoningEffort,
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io/fs"
|
"io/fs"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"reflect"
|
"reflect"
|
||||||
@@ -13,7 +15,9 @@ import (
|
|||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
|
"testing/fstest"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
@@ -34,6 +38,7 @@ import (
|
|||||||
itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents"
|
itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents"
|
||||||
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
|
func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
|
||||||
@@ -406,6 +411,89 @@ func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestProductionLLMClientFactoryUsesConfiguredLocalBackend(t *testing.T) {
|
||||||
|
var providerCalls atomic.Int32
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
providerCalls.Add(1)
|
||||||
|
if r.URL.Path != "/v1/chat/completions" {
|
||||||
|
t.Errorf("provider path = %q, want /v1/chat/completions", r.URL.Path)
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_, _ = w.Write([]byte(`{
|
||||||
|
"choices": [{"message": {"role": "assistant", "content": "{\"ok\":true}"}}],
|
||||||
|
"usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7}
|
||||||
|
}`))
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte(`id: local-profile
|
||||||
|
backend: local
|
||||||
|
model: local-model
|
||||||
|
`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
assets := llm.NewAssetRegistry()
|
||||||
|
if err := assets.RegisterPromptFS(fstest.MapFS{
|
||||||
|
"production.local.yaml": {Data: []byte(`id: production.local
|
||||||
|
version: "v1"
|
||||||
|
inputs:
|
||||||
|
- name: transcript
|
||||||
|
required: true
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: '{{ input "transcript" }}'
|
||||||
|
output:
|
||||||
|
format: json
|
||||||
|
validation_mode: json
|
||||||
|
`)},
|
||||||
|
}, "."); err != nil {
|
||||||
|
t.Fatalf("register prompt assets: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg := config.Default()
|
||||||
|
cfg.PromptKit.ProfileFile = profilePath
|
||||||
|
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: server.URL + "/v1",
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
client, manifests, err := productionLLMClientFactoryWithAssets(assets)(
|
||||||
|
context.Background(),
|
||||||
|
cfg,
|
||||||
|
"local-profile",
|
||||||
|
LLMRuntimeOverrides{},
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("build production LLM runtime: %v", err)
|
||||||
|
}
|
||||||
|
if len(manifests) != 0 {
|
||||||
|
t.Fatalf("eager profile manifests = %#v, want none", manifests)
|
||||||
|
}
|
||||||
|
|
||||||
|
var out map[string]any
|
||||||
|
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||||
|
PromptID: "production.local",
|
||||||
|
ProfileID: "local-profile",
|
||||||
|
Inputs: contracts.LLMInputSet{
|
||||||
|
"transcript": contracts.NewLLMInputMaterial("transcript", "text/plain", []byte("local request"), "", ""),
|
||||||
|
},
|
||||||
|
}, &out)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if providerCalls.Load() != 1 {
|
||||||
|
t.Fatalf("provider calls = %d, want 1", providerCalls.Load())
|
||||||
|
}
|
||||||
|
provider, ok := client.(contracts.LLMProfileManifestProvider)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("production client %T does not provide profile manifests", client)
|
||||||
|
}
|
||||||
|
recorded := provider.LLMProfileManifests()
|
||||||
|
if len(recorded) != 1 || recorded[0].BackendID != promptkit.BackendLocal {
|
||||||
|
t.Fatalf("production profile manifests = %#v, want local backend", recorded)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
|
func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
|
||||||
t.Run("canceled context", func(t *testing.T) {
|
t.Run("canceled context", func(t *testing.T) {
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"testing/fstest"
|
"testing/fstest"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/promptkit"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -61,8 +62,21 @@ func newProfileValidationEngine(cfg config.Config) (*promptkit.Engine, error) {
|
|||||||
if cfg.PromptKit.ProfileFile != "" {
|
if cfg.PromptKit.ProfileFile != "" {
|
||||||
opts = append(opts, promptkit.WithProfileFile(cfg.PromptKit.ProfileFile))
|
opts = append(opts, promptkit.WithProfileFile(cfg.PromptKit.ProfileFile))
|
||||||
}
|
}
|
||||||
|
if localBackend := mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend); localBackend != nil {
|
||||||
|
opts = append(opts, llm.PromptKitLocalBackendOption(*localBackend))
|
||||||
|
}
|
||||||
return promptkit.NewEngine(promptkit.Config{
|
return promptkit.NewEngine(promptkit.Config{
|
||||||
PromptDir: "unused",
|
PromptDir: "unused",
|
||||||
ProfileDir: cfg.PromptKit.ProfileDir,
|
ProfileDir: cfg.PromptKit.ProfileDir,
|
||||||
}, opts...)
|
}, opts...)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func mapPromptKitLocalBackend(cfg *config.PromptKitLocalBackendConfig) *llm.PromptKitLocalBackendConfig {
|
||||||
|
if cfg == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return &llm.PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: cfg.Endpoint,
|
||||||
|
ConcurrencyLimit: cfg.ConcurrencyLimit,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
54
internal/cli/promptkit_profiles_test.go
Normal file
54
internal/cli/promptkit_profiles_test.go
Normal file
@@ -0,0 +1,54 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestExplicitPromptKitProfileValidationUsesConfiguredLocalBackendWithoutGeneration(t *testing.T) {
|
||||||
|
var providerCalls atomic.Int32
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
|
||||||
|
providerCalls.Add(1)
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte(`id: local-profile
|
||||||
|
backend: local
|
||||||
|
model: local-model
|
||||||
|
`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
cfg := config.Default()
|
||||||
|
cfg.PromptKit.ProfileFile = profilePath
|
||||||
|
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: server.URL + "/v1",
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
if err := validateExplicitPromptKitProfiles(context.Background(), cfg, []string{"local-profile"}); err != nil {
|
||||||
|
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if providerCalls.Load() != 0 {
|
||||||
|
t.Fatalf("provider calls during configured profile validation = %d, want 0", providerCalls.Load())
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg.PromptKit.LocalBackend = nil
|
||||||
|
err := validateExplicitPromptKitProfiles(context.Background(), cfg, []string{"local-profile"})
|
||||||
|
if err == nil ||
|
||||||
|
!strings.Contains(err.Error(), `validate PromptKit profile "local-profile"`) ||
|
||||||
|
!strings.Contains(err.Error(), promptkit.BackendLocal) {
|
||||||
|
t.Fatalf("validation without registration error = %v, want profile and local backend context", err)
|
||||||
|
}
|
||||||
|
if providerCalls.Load() != 0 {
|
||||||
|
t.Fatalf("provider calls after missing-registration validation = %d, want 0", providerCalls.Load())
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -18,6 +18,12 @@ type Config struct {
|
|||||||
type PromptKitConfig struct {
|
type PromptKitConfig struct {
|
||||||
ProfileDir string `json:"profile_dir,omitempty"`
|
ProfileDir string `json:"profile_dir,omitempty"`
|
||||||
ProfileFile string `json:"profile_file,omitempty"`
|
ProfileFile string `json:"profile_file,omitempty"`
|
||||||
|
LocalBackend *PromptKitLocalBackendConfig `json:"local_backend,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitLocalBackendConfig struct {
|
||||||
|
Endpoint string `json:"endpoint"`
|
||||||
|
ConcurrencyLimit int `json:"concurrency_limit"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type ConcurrencyConfig struct {
|
type ConcurrencyConfig struct {
|
||||||
@@ -66,6 +72,10 @@ func Default() Config {
|
|||||||
|
|
||||||
func cloneConfig(in Config) Config {
|
func cloneConfig(in Config) Config {
|
||||||
out := in
|
out := in
|
||||||
|
if in.PromptKit.LocalBackend != nil {
|
||||||
|
localBackend := *in.PromptKit.LocalBackend
|
||||||
|
out.PromptKit.LocalBackend = &localBackend
|
||||||
|
}
|
||||||
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
|
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
|
||||||
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
|
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
|
||||||
for key, profile := range in.Pipelines {
|
for key, profile := range in.Pipelines {
|
||||||
|
|||||||
@@ -107,6 +107,33 @@ func TestEffectiveConfigPreservesPromptKitProfileSource(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestEffectiveConfigOwnsPromptKitLocalBackend(t *testing.T) {
|
||||||
|
cfg := configForEffectiveTests(t, effectiveProfile())
|
||||||
|
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: "http://localhost:8000/v1",
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
effective, err := cfg.Resolve(ResolveInput{PipelineID: "main", Catalog: effectiveCatalog(t)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v", err)
|
||||||
|
}
|
||||||
|
if effective.Config.PromptKit.LocalBackend == nil {
|
||||||
|
t.Fatal("effective local backend = nil")
|
||||||
|
}
|
||||||
|
if effective.Config.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend {
|
||||||
|
t.Fatal("effective local backend aliases input config")
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg.PromptKit.LocalBackend.Endpoint = "http://changed-input.example/v1"
|
||||||
|
if effective.Config.PromptKit.LocalBackend.Endpoint != "http://localhost:8000/v1" {
|
||||||
|
t.Fatalf("input mutation changed effective config: %#v", effective.Config.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
effective.Config.PromptKit.LocalBackend.ConcurrencyLimit = 9
|
||||||
|
if cfg.PromptKit.LocalBackend.ConcurrencyLimit != 2 {
|
||||||
|
t.Fatalf("effective mutation changed input config: %#v", cfg.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
|
func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
|
|||||||
@@ -25,6 +25,12 @@ type FileConfig struct {
|
|||||||
type FilePromptKitConfig struct {
|
type FilePromptKitConfig struct {
|
||||||
ProfileDir *string `yaml:"profile_dir,omitempty"`
|
ProfileDir *string `yaml:"profile_dir,omitempty"`
|
||||||
ProfileFile *string `yaml:"profile_file,omitempty"`
|
ProfileFile *string `yaml:"profile_file,omitempty"`
|
||||||
|
LocalBackend *FilePromptKitLocalBackendConfig `yaml:"local_backend,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type FilePromptKitLocalBackendConfig struct {
|
||||||
|
Endpoint *string `yaml:"endpoint,omitempty"`
|
||||||
|
ConcurrencyLimit *int `yaml:"concurrency_limit,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type FilePipelineProfile struct {
|
type FilePipelineProfile struct {
|
||||||
@@ -468,6 +474,20 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
|||||||
}
|
}
|
||||||
c.PromptKit.ProfileFile = value
|
c.PromptKit.ProfileFile = value
|
||||||
}
|
}
|
||||||
|
if fileCfg.PromptKit.LocalBackend != nil {
|
||||||
|
if fileCfg.PromptKit.LocalBackend.Endpoint == nil {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
|
||||||
|
}
|
||||||
|
endpoint := strings.TrimSpace(*fileCfg.PromptKit.LocalBackend.Endpoint)
|
||||||
|
if endpoint == "" {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
|
||||||
|
}
|
||||||
|
localBackend := PromptKitLocalBackendConfig{Endpoint: endpoint}
|
||||||
|
if fileCfg.PromptKit.LocalBackend.ConcurrencyLimit != nil {
|
||||||
|
localBackend.ConcurrencyLimit = *fileCfg.PromptKit.LocalBackend.ConcurrencyLimit
|
||||||
|
}
|
||||||
|
c.PromptKit.LocalBackend = &localBackend
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, pipelineID := range pipelineIDs {
|
for _, pipelineID := range pipelineIDs {
|
||||||
|
|||||||
@@ -97,6 +97,100 @@ func TestFilePromptKitProfileSourcesSurviveConfigBoundaries(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitLocalBackendSurvivesConfigBoundaries(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
concurrencyYAML string
|
||||||
|
wantConcurrency int
|
||||||
|
}{
|
||||||
|
{name: "omitted concurrency defaults to zero"},
|
||||||
|
{name: "positive concurrency is preserved", concurrencyYAML: " concurrency_limit: 2\n", wantConcurrency: 2},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, "version: 4\npromptkit:\n local_backend:\n endpoint: ' http://localhost:8000/v1 '\n"+tt.concurrencyYAML)
|
||||||
|
cfg := Default()
|
||||||
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v", err)
|
||||||
|
}
|
||||||
|
want := PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: "http://localhost:8000/v1",
|
||||||
|
ConcurrencyLimit: tt.wantConcurrency,
|
||||||
|
}
|
||||||
|
if cfg.PromptKit.LocalBackend == nil || *cfg.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("local backend config = %#v, want %#v", cfg.PromptKit.LocalBackend, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
*file.PromptKit.LocalBackend.Endpoint = "http://changed.example/v1"
|
||||||
|
if file.PromptKit.LocalBackend.ConcurrencyLimit != nil {
|
||||||
|
*file.PromptKit.LocalBackend.ConcurrencyLimit = 99
|
||||||
|
}
|
||||||
|
if *cfg.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("effective config aliases parsed file model: %#v", cfg.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
|
||||||
|
cloned := cloneConfig(cfg)
|
||||||
|
if cloned.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *cloned.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("cloned local backend = %#v, want detached %#v", cloned.PromptKit.LocalBackend, want)
|
||||||
|
}
|
||||||
|
cloned.PromptKit.LocalBackend.Endpoint = "http://clone.example/v1"
|
||||||
|
if *cfg.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("mutating clone changed source config: %#v", cfg.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
|
||||||
|
redacted := cfg.Redacted()
|
||||||
|
if redacted.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *redacted.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("redacted local backend = %#v, want detached %#v", redacted.PromptKit.LocalBackend, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := json.Marshal(cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("json.Marshal() error = %v", err)
|
||||||
|
}
|
||||||
|
var payload struct {
|
||||||
|
PromptKit map[string]json.RawMessage `json:"promptkit"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(data, &payload); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal() error = %v", err)
|
||||||
|
}
|
||||||
|
localJSON, ok := payload.PromptKit["local_backend"]
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("runtime PromptKit JSON keys = %v, want local_backend", payload.PromptKit)
|
||||||
|
}
|
||||||
|
var localPayload map[string]json.RawMessage
|
||||||
|
if err := json.Unmarshal(localJSON, &localPayload); err != nil {
|
||||||
|
t.Fatalf("unmarshal local_backend JSON: %v", err)
|
||||||
|
}
|
||||||
|
if _, ok := localPayload["endpoint"]; !ok {
|
||||||
|
t.Fatalf("runtime local_backend JSON keys = %v, want endpoint", localPayload)
|
||||||
|
}
|
||||||
|
if _, ok := localPayload["concurrency_limit"]; !ok {
|
||||||
|
t.Fatalf("runtime local_backend JSON keys = %v, want concurrency_limit", localPayload)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitLocalBackendRequiresEndpoint(t *testing.T) {
|
||||||
|
for _, tt := range []struct {
|
||||||
|
name string
|
||||||
|
yaml string
|
||||||
|
}{
|
||||||
|
{name: "missing", yaml: "version: 4\npromptkit:\n local_backend: {}\n"},
|
||||||
|
{name: "empty", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ''\n"},
|
||||||
|
{name: "blank", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ' '\n"},
|
||||||
|
} {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, tt.yaml)
|
||||||
|
cfg := Default()
|
||||||
|
err := cfg.ApplyFileConfig(file)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "promptkit.local_backend.endpoint") {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v, want endpoint field context", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestFilePromptKitExplicitEmptyProfileSourcesAreRejected(t *testing.T) {
|
func TestFilePromptKitExplicitEmptyProfileSourcesAreRejected(t *testing.T) {
|
||||||
for _, field := range []string{"profile_dir", "profile_file"} {
|
for _, field := range []string{"profile_dir", "profile_file"} {
|
||||||
t.Run(field, func(t *testing.T) {
|
t.Run(field, func(t *testing.T) {
|
||||||
@@ -188,6 +282,11 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
|||||||
yaml: "version: 4\ncache:\n checkpoints:\n enabled: definitely\n",
|
yaml: "version: 4\ncache:\n checkpoints:\n enabled: definitely\n",
|
||||||
want: "cannot unmarshal",
|
want: "cannot unmarshal",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: "local backend field",
|
||||||
|
yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: http://localhost:8000/v1\n unknown: true\n",
|
||||||
|
want: "field unknown not found",
|
||||||
|
},
|
||||||
}
|
}
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package config
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"net/url"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
@@ -53,6 +54,27 @@ func validatePromptKit(cfg PromptKitConfig) error {
|
|||||||
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
||||||
return fmt.Errorf("promptkit profile_dir and profile_file are mutually exclusive")
|
return fmt.Errorf("promptkit profile_dir and profile_file are mutually exclusive")
|
||||||
}
|
}
|
||||||
|
if cfg.LocalBackend == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
endpoint := strings.TrimSpace(cfg.LocalBackend.Endpoint)
|
||||||
|
if endpoint == "" {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
|
||||||
|
}
|
||||||
|
parsed, err := url.Parse(endpoint)
|
||||||
|
if err != nil ||
|
||||||
|
(!strings.EqualFold(parsed.Scheme, "http") && !strings.EqualFold(parsed.Scheme, "https")) ||
|
||||||
|
!parsed.IsAbs() ||
|
||||||
|
parsed.Hostname() == "" ||
|
||||||
|
parsed.User != nil ||
|
||||||
|
parsed.RawQuery != "" ||
|
||||||
|
parsed.ForceQuery ||
|
||||||
|
strings.Contains(endpoint, "#") {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must be an absolute HTTP or HTTPS URL with a host and no user information, query, or fragment")
|
||||||
|
}
|
||||||
|
if cfg.LocalBackend.ConcurrencyLimit < 0 {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.concurrency_limit must not be negative")
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -93,6 +93,73 @@ func TestValidatePromptKitSourcesAreMutuallyExclusive(t *testing.T) {
|
|||||||
assertValidationContains(t, cfg, "promptkit profile_dir and profile_file are mutually exclusive")
|
assertValidationContains(t, cfg, "promptkit profile_dir and profile_file are mutually exclusive")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestValidatePromptKitLocalBackendEndpoints(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
endpoint string
|
||||||
|
profileSource PromptKitConfig
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "HTTP endpoint with path and profile directory",
|
||||||
|
endpoint: "http://localhost:8000/v1",
|
||||||
|
profileSource: PromptKitConfig{ProfileDir: "./profiles"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "case-insensitive HTTPS endpoint and profile file",
|
||||||
|
endpoint: "HTTPS://inference.example.test/api",
|
||||||
|
profileSource: PromptKitConfig{ProfileFile: "./profiles.yml"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := Default()
|
||||||
|
cfg.PromptKit = tt.profileSource
|
||||||
|
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: tt.endpoint,
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
if err := cfg.Validate(); err != nil {
|
||||||
|
t.Fatalf("Validate() error = %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatePromptKitLocalBackendRejectsInvalidValues(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
endpoint string
|
||||||
|
concurrencyLimit int
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "blank endpoint", endpoint: " ", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "relative URL", endpoint: "localhost:8000/v1", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "unsupported scheme", endpoint: "ftp://localhost/model", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "missing host", endpoint: "http:///v1", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "user information", endpoint: "http://user:secret@localhost/v1", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "query", endpoint: "http://localhost/v1?model=example", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "empty query", endpoint: "http://localhost/v1?", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "fragment", endpoint: "http://localhost/v1#model", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "empty fragment", endpoint: "http://localhost/v1#", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{
|
||||||
|
name: "negative concurrency",
|
||||||
|
endpoint: "http://localhost:8000/v1",
|
||||||
|
concurrencyLimit: -1,
|
||||||
|
want: "promptkit.local_backend.concurrency_limit",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := Default()
|
||||||
|
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: tt.endpoint,
|
||||||
|
ConcurrencyLimit: tt.concurrencyLimit,
|
||||||
|
}
|
||||||
|
assertValidationContains(t, cfg, tt.want)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestValidateStateSurfaceRules(t *testing.T) {
|
func TestValidateStateSurfaceRules(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
|
|||||||
@@ -19,9 +19,15 @@ import (
|
|||||||
|
|
||||||
const promptKitProviderName = "promptkit"
|
const promptKitProviderName = "promptkit"
|
||||||
|
|
||||||
|
type PromptKitLocalBackendConfig struct {
|
||||||
|
Endpoint string
|
||||||
|
ConcurrencyLimit int
|
||||||
|
}
|
||||||
|
|
||||||
type PromptKitClientConfig struct {
|
type PromptKitClientConfig struct {
|
||||||
ProfileDir string
|
ProfileDir string
|
||||||
ProfileFile string
|
ProfileFile string
|
||||||
|
LocalBackend *PromptKitLocalBackendConfig
|
||||||
Assets *AssetRegistry
|
Assets *AssetRegistry
|
||||||
Timeout time.Duration
|
Timeout time.Duration
|
||||||
HTTPClient *http.Client
|
HTTPClient *http.Client
|
||||||
@@ -35,6 +41,7 @@ type PromptKitClient struct {
|
|||||||
recorder *LLMProfileRecorder
|
recorder *LLMProfileRecorder
|
||||||
profileDir string
|
profileDir string
|
||||||
profileFile string
|
profileFile string
|
||||||
|
localEndpoint string
|
||||||
reasoningEffort *string
|
reasoningEffort *string
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -46,6 +53,10 @@ type LLMProfileRecorder struct {
|
|||||||
var _ contracts.StructuredLLMClient = (*PromptKitClient)(nil)
|
var _ contracts.StructuredLLMClient = (*PromptKitClient)(nil)
|
||||||
var _ contracts.LLMProfileManifestProvider = (*PromptKitClient)(nil)
|
var _ contracts.LLMProfileManifestProvider = (*PromptKitClient)(nil)
|
||||||
|
|
||||||
|
func PromptKitLocalBackendOption(cfg PromptKitLocalBackendConfig) promptkit.Option {
|
||||||
|
return promptkit.WithBackend(promptkit.LocalBackend(cfg.Endpoint, cfg.ConcurrencyLimit))
|
||||||
|
}
|
||||||
|
|
||||||
func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
||||||
if cfg.Assets == nil {
|
if cfg.Assets == nil {
|
||||||
return nil, fmt.Errorf("PromptKit client assets must not be nil")
|
return nil, fmt.Errorf("PromptKit client assets must not be nil")
|
||||||
@@ -60,6 +71,13 @@ func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
|||||||
if profileFile := strings.TrimSpace(cfg.ProfileFile); profileFile != "" {
|
if profileFile := strings.TrimSpace(cfg.ProfileFile); profileFile != "" {
|
||||||
options = append(options, promptkit.WithProfileFile(profileFile))
|
options = append(options, promptkit.WithProfileFile(profileFile))
|
||||||
}
|
}
|
||||||
|
var localEndpoint string
|
||||||
|
if cfg.LocalBackend != nil {
|
||||||
|
localBackend := *cfg.LocalBackend
|
||||||
|
localBackend.Endpoint = strings.TrimSpace(localBackend.Endpoint)
|
||||||
|
localEndpoint = localBackend.Endpoint
|
||||||
|
options = append(options, PromptKitLocalBackendOption(localBackend))
|
||||||
|
}
|
||||||
options = append(options, cfg.EngineOptions...)
|
options = append(options, cfg.EngineOptions...)
|
||||||
|
|
||||||
engine, err := promptkit.NewEngine(promptkit.Config{
|
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||||
@@ -84,6 +102,7 @@ func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
|||||||
recorder: recorder,
|
recorder: recorder,
|
||||||
profileDir: strings.TrimSpace(cfg.ProfileDir),
|
profileDir: strings.TrimSpace(cfg.ProfileDir),
|
||||||
profileFile: strings.TrimSpace(cfg.ProfileFile),
|
profileFile: strings.TrimSpace(cfg.ProfileFile),
|
||||||
|
localEndpoint: localEndpoint,
|
||||||
reasoningEffort: reasoningEffort,
|
reasoningEffort: reasoningEffort,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
@@ -302,7 +321,11 @@ func (c *PromptKitClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint,
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
return []CheckpointFingerprint{fingerprint}, nil
|
fingerprints := []CheckpointFingerprint{fingerprint}
|
||||||
|
if c.localEndpoint != "" {
|
||||||
|
fingerprints = append(fingerprints, promptKitLocalBackendFingerprint(c.localEndpoint))
|
||||||
|
}
|
||||||
|
return fingerprints, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewLLMProfileRecorder() *LLMProfileRecorder {
|
func NewLLMProfileRecorder() *LLMProfileRecorder {
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"reflect"
|
"reflect"
|
||||||
@@ -305,12 +306,89 @@ func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
|
|||||||
if fresh[0].Value == "mutated" {
|
if fresh[0].Value == "mutated" {
|
||||||
t.Fatal("LLMCheckpointFingerprints exposed mutable backing storage")
|
t.Fatal("LLMCheckpointFingerprints exposed mutable backing storage")
|
||||||
}
|
}
|
||||||
const wantBuiltinFingerprint = "sha256:4b48cd1ef010b587cd8d56bc13b73a0cdd84da1414a8de9f9a2adc695dfcf0a7"
|
const wantBuiltinFingerprint = "sha256:5218b1dec48f5fdd46836826e0b25906c33efbdf943c5f18085b8d82467e0276"
|
||||||
if fresh[0].Value != wantBuiltinFingerprint {
|
if fresh[0].Value != wantBuiltinFingerprint {
|
||||||
t.Fatalf("built-in profile fingerprint = %q, want %q", fresh[0].Value, wantBuiltinFingerprint)
|
t.Fatalf("built-in profile fingerprint = %q, want %q", fresh[0].Value, wantBuiltinFingerprint)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestPromptKitClientCheckpointFingerprintTracksLocalBackendTarget(t *testing.T) {
|
||||||
|
const (
|
||||||
|
firstEndpoint = "http://localhost:8000/v1"
|
||||||
|
secondEndpoint = "https://inference.example.test/v1"
|
||||||
|
)
|
||||||
|
fingerprintsFor := func(endpoint string, concurrencyLimit int) []CheckpointFingerprint {
|
||||||
|
t.Helper()
|
||||||
|
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||||
|
Assets: newTestPromptKitAssets(t),
|
||||||
|
LocalBackend: &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: endpoint,
|
||||||
|
ConcurrencyLimit: concurrencyLimit,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
values, err := client.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return values
|
||||||
|
}
|
||||||
|
|
||||||
|
baseline := fingerprintsFor(firstEndpoint, 0)
|
||||||
|
if len(baseline) != 2 ||
|
||||||
|
baseline[0].Name != promptKitProfileFingerprintName ||
|
||||||
|
baseline[1].Name != promptKitLocalBackendFingerprintName {
|
||||||
|
t.Fatalf("checkpoint fingerprints = %#v, want profile source then local backend target", baseline)
|
||||||
|
}
|
||||||
|
endpointChanged := fingerprintsFor(secondEndpoint, 0)
|
||||||
|
if baseline[0] != endpointChanged[0] || baseline[1] == endpointChanged[1] {
|
||||||
|
t.Fatalf("endpoint fingerprints = %#v and %#v, want only local target to change", baseline, endpointChanged)
|
||||||
|
}
|
||||||
|
concurrencyChanged := fingerprintsFor(firstEndpoint, 4)
|
||||||
|
if !reflect.DeepEqual(baseline, concurrencyChanged) {
|
||||||
|
t.Fatalf("concurrency fingerprints = %#v, want %#v", concurrencyChanged, baseline)
|
||||||
|
}
|
||||||
|
for _, values := range [][]CheckpointFingerprint{baseline, endpointChanged} {
|
||||||
|
for _, fingerprint := range values {
|
||||||
|
if strings.Contains(fingerprint.Value, firstEndpoint) ||
|
||||||
|
strings.Contains(fingerprint.Value, secondEndpoint) {
|
||||||
|
t.Fatalf("checkpoint fingerprint exposes endpoint: %#v", fingerprint)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
localBackend := &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: " " + firstEndpoint + " ",
|
||||||
|
ConcurrencyLimit: 0,
|
||||||
|
}
|
||||||
|
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||||
|
Assets: newTestPromptKitAssets(t),
|
||||||
|
LocalBackend: localBackend,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
localBackend.Endpoint = secondEndpoint
|
||||||
|
copy, err := client.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(copy, baseline) {
|
||||||
|
t.Fatalf("fingerprints after input mutation = %#v, want retained target %#v", copy, baseline)
|
||||||
|
}
|
||||||
|
copy[0].Value = "mutated-profile"
|
||||||
|
copy[1].Value = "mutated-target"
|
||||||
|
fresh, err := client.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(fresh, baseline) {
|
||||||
|
t.Fatalf("fingerprints after returned-slice mutation = %#v, want %#v", fresh, baseline)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) {
|
func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) {
|
||||||
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||||
client := newTestPromptKitClient(t, fake)
|
client := newTestPromptKitClient(t, fake)
|
||||||
@@ -334,6 +412,88 @@ func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testi
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestPromptKitClientUsesConfiguredLocalBackend(t *testing.T) {
|
||||||
|
var providerCalls atomic.Int32
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
providerCalls.Add(1)
|
||||||
|
if r.URL.Path != "/v1/chat/completions" {
|
||||||
|
t.Errorf("provider path = %q, want /v1/chat/completions", r.URL.Path)
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_, _ = w.Write([]byte(`{
|
||||||
|
"choices": [{"message": {"role": "assistant", "content": "{\"ok\":true}"}}],
|
||||||
|
"usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7}
|
||||||
|
}`))
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte(`id: local-profile
|
||||||
|
backend: local
|
||||||
|
model: local-model
|
||||||
|
`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
localBackend := &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: " " + server.URL + "/v1 ",
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||||
|
Assets: newTestPromptKitAssets(t),
|
||||||
|
ProfileFile: profilePath,
|
||||||
|
LocalBackend: localBackend,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
localBackend.Endpoint = "http://127.0.0.1:1/v1"
|
||||||
|
|
||||||
|
request := contracts.StructuredCompletionRequest{
|
||||||
|
PromptID: "adapter.direct-session",
|
||||||
|
ProfileID: "local-profile",
|
||||||
|
Inputs: contracts.LLMInputSet{
|
||||||
|
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||||
|
},
|
||||||
|
Vars: map[string]any{"custom": "value"},
|
||||||
|
}
|
||||||
|
var out map[string]any
|
||||||
|
response, err := client.CompleteStructured(context.Background(), request, &out)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if providerCalls.Load() != 1 {
|
||||||
|
t.Fatalf("provider calls = %d, want 1", providerCalls.Load())
|
||||||
|
}
|
||||||
|
if response.ProfileID != "local-profile" || response.Model != "local-model" {
|
||||||
|
t.Fatalf("response metadata = %#v", response)
|
||||||
|
}
|
||||||
|
if response.Debug == nil || response.Debug.Prompt == nil ||
|
||||||
|
response.Debug.Prompt.SelectedBackendID != promptkit.BackendLocal {
|
||||||
|
t.Fatalf("response debug prompt = %#v, want local backend", response.Debug)
|
||||||
|
}
|
||||||
|
manifests := client.LLMProfileManifests()
|
||||||
|
if len(manifests) != 1 || manifests[0].BackendID != promptkit.BackendLocal {
|
||||||
|
t.Fatalf("profile manifests = %#v, want local backend", manifests)
|
||||||
|
}
|
||||||
|
|
||||||
|
clientWithoutRegistration, err := NewPromptKitClient(PromptKitClientConfig{
|
||||||
|
Assets: newTestPromptKitAssets(t),
|
||||||
|
ProfileFile: profilePath,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewPromptKitClient() without local registration error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
_, err = clientWithoutRegistration.CompleteStructured(context.Background(), request, &out)
|
||||||
|
if err == nil ||
|
||||||
|
!strings.Contains(err.Error(), "prepare PromptKit prompt") ||
|
||||||
|
!strings.Contains(err.Error(), promptkit.BackendLocal) {
|
||||||
|
t.Fatalf("CompleteStructured() without registration error = %v, want preparation failure with local backend context", err)
|
||||||
|
}
|
||||||
|
if providerCalls.Load() != 1 {
|
||||||
|
t.Fatalf("provider calls after missing-registration failure = %d, want 1", providerCalls.Load())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestLLMProfileRecorderDistinguishesEffectiveTargets(t *testing.T) {
|
func TestLLMProfileRecorderDistinguishesEffectiveTargets(t *testing.T) {
|
||||||
recorder := NewLLMProfileRecorder()
|
recorder := NewLLMProfileRecorder()
|
||||||
for _, profile := range []artifacts.LLMProfileManifest{
|
for _, profile := range []artifacts.LLMProfileManifest{
|
||||||
|
|||||||
@@ -13,9 +13,11 @@ import (
|
|||||||
|
|
||||||
const (
|
const (
|
||||||
promptKitProfileFingerprintName = "promptkit_profile_source"
|
promptKitProfileFingerprintName = "promptkit_profile_source"
|
||||||
|
promptKitLocalBackendFingerprintName = "promptkit_local_backend_target"
|
||||||
|
promptKitLocalBackendMarker = "notarius:promptkit-local-backend:v1"
|
||||||
// The built-in profile catalog is compiled into this pinned PromptKit
|
// The built-in profile catalog is compiled into this pinned PromptKit
|
||||||
// release. Update this identity when the dependency is upgraded.
|
// release. Update this identity when the dependency is upgraded.
|
||||||
promptKitBuiltinProfileCatalogID = "promptkit:v0.2.0:builtin-profiles"
|
promptKitBuiltinProfileCatalogID = "promptkit:v0.3.0:builtin-profiles"
|
||||||
)
|
)
|
||||||
|
|
||||||
func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFingerprint, error) {
|
func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFingerprint, error) {
|
||||||
@@ -45,6 +47,16 @@ func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFing
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func promptKitLocalBackendFingerprint(endpoint string) CheckpointFingerprint {
|
||||||
|
hasher := sha256.New()
|
||||||
|
writeFingerprintPart(hasher, []byte(promptKitLocalBackendMarker))
|
||||||
|
writeFingerprintPart(hasher, []byte(strings.TrimSpace(endpoint)))
|
||||||
|
return CheckpointFingerprint{
|
||||||
|
Name: promptKitLocalBackendFingerprintName,
|
||||||
|
Value: "sha256:" + hex.EncodeToString(hasher.Sum(nil)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func promptKitProfileFileDigests(root string) ([][]byte, error) {
|
func promptKitProfileFileDigests(root string) ([][]byte, error) {
|
||||||
var digests [][]byte
|
var digests [][]byte
|
||||||
err := filepath.WalkDir(root, func(name string, entry fs.DirEntry, walkErr error) error {
|
err := filepath.WalkDir(root, func(name string, entry fs.DirEntry, walkErr error) error {
|
||||||
|
|||||||
@@ -81,7 +81,10 @@ func TestScheduledClientPreservesCheckpointFingerprints(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
inner := &fingerprintedStructuredClient{
|
inner := &fingerprintedStructuredClient{
|
||||||
fingerprints: []CheckpointFingerprint{{Name: "profile_source", Value: "sha256:one"}},
|
fingerprints: []CheckpointFingerprint{
|
||||||
|
{Name: "profile_source", Value: "sha256:one"},
|
||||||
|
{Name: "backend_target", Value: "sha256:two"},
|
||||||
|
},
|
||||||
}
|
}
|
||||||
client := NewScheduledClient(inner, scheduler)
|
client := NewScheduledClient(inner, scheduler)
|
||||||
provider, ok := client.(CheckpointFingerprintProvider)
|
provider, ok := client.(CheckpointFingerprintProvider)
|
||||||
@@ -92,7 +95,9 @@ func TestScheduledClientPreservesCheckpointFingerprints(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if len(got) != 1 || got[0] != inner.fingerprints[0] {
|
if len(got) != len(inner.fingerprints) ||
|
||||||
|
got[0] != inner.fingerprints[0] ||
|
||||||
|
got[1] != inner.fingerprints[1] {
|
||||||
t.Fatalf("checkpoint fingerprints = %#v, want %#v", got, inner.fingerprints)
|
t.Fatalf("checkpoint fingerprints = %#v, want %#v", got, inner.fingerprints)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user