Compare commits

..

7 Commits

29 changed files with 868 additions and 189 deletions

View File

@@ -41,14 +41,21 @@ pipeline ID and **--input** are required.
| **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. |
| **--llm-profile id** | Override effective LLM-capable module bindings with one configured profile. |
| **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. |
| **--reasoning-effort value** | Replace the selected PromptKit profile's reasoning effort for every LLM-backed call in this run. The value must be non-empty and the flag may be specified only once. |
| **--clear-reasoning-effort** | Clear reasoning effort inherited from the selected PromptKit profile for every LLM-backed call in this run. |
| **--reference selector=path** | Add or replace a file reference binding. Repeatable. |
| **--without-reference selector** | Remove a configured optional reference binding. Repeatable. |
**--chunk_cache** accepts only **auto**, **bypass**, or **refresh**.
**--debug-dir**, **--output-dir**, **--session-id**, and
**--recompute-step** reject explicit empty values. **--recompute-step**
requires **--resume**; checkpoint requirements and reuse behavior are
documented in [Operations](operations.md).
**--reasoning-effort**, and **--recompute-step** reject explicit empty values.
**--reasoning-effort** and **--clear-reasoning-effort** are mutually exclusive.
When neither is present, reasoning effort comes from the selected PromptKit
profile. These controls apply to the shared run client, including retries and
LLM-backed validators, and do not modify configuration or profile files.
Persistent reasoning settings remain a PromptKit profile concern.
**--recompute-step** requires **--resume**; checkpoint requirements and reuse
behavior are documented in [Operations](operations.md).
### Reference selectors

View File

@@ -98,6 +98,21 @@ summarize results without embedding lane payload bytes. A chunk-plan summary is
provenance for the plan used by this run; cache records, debug artifacts, and
other operational state are not published as bundle files.
Each `llm_profiles` entry identifies effective, non-secret LLM execution
provenance:
| Field | Required | Meaning |
| --- | --- | --- |
| `id` | Yes | Selected PromptKit profile identifier. |
| `provider` | No | Notarius adapter provider identifier. |
| `model` | No | Effective provider model identifier. |
| `backend_id` | No | Effective PromptKit backend registration identifier. Endpoint-only profiles omit it. |
| `reasoning_effort` | No | Effective opaque provider reasoning setting. An empty or explicitly cleared setting is omitted. |
These values describe observed execution; they are not a backend-registration
interface. Entries that differ by backend or effective reasoning remain
distinct even when their profile, provider, and model are otherwise equal.
## Rejections And Warnings
`rejected.json` is always an object with a `rejected` array. Each entry has

View File

@@ -1,11 +1,11 @@
# PromptKit Integration
Notarius pins
[`gitea.maximumdirect.net/eric/promptkit` v0.1.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0)
[`gitea.maximumdirect.net/eric/promptkit` v0.2.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0)
as its in-process prompt engine. The upstream
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/consumers/pkg-promptkit.md)
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0/docs/consumers/pkg-promptkit.md)
owns the public engine API, and the upstream
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/formats.md)
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0/docs/formats.md)
owns prompt, profile, and schema file contracts.
## Supported Boundary
@@ -15,11 +15,12 @@ Notarius relies on the root `promptkit` package to:
- construct an `Engine` with filesystem-backed prompt, schema, and optional
profile sources;
- prepare and run a `RunRequest` with named inline artifacts, variables,
metadata, prompt identity, and profile selection;
a direct session ID, prompt identity, and profile selection;
- return rendered debug material, validated structured output, selected
profile and model metadata, and token usage;
profile, backend, effective model metadata, and token usage;
- distinguish structured-output validation failure from execution failure; and
- identify a missing explicit profile through `ErrProfileNotFound`.
- identify a missing explicit profile through `ErrProfileNotFound` and backend
admission exhaustion through `ErrCapacityExceeded`.
Notarius does not use PromptKit's optional `ArtifactReader`. It materializes
source and reference content itself and supplies owned inline artifacts at the
@@ -27,6 +28,26 @@ adapter boundary. It also retains responsibility for pipeline retries,
scheduling, debug persistence, redaction, profile provenance, and conversion
from private model responses into durable domain artifacts.
Notarius sends its trimmed run session through PromptKit's direct session
field, which is authoritative for provider session behavior. It also retains
the same value as the `session_id` prompt variable for maintained prompt
compatibility. Session IDs are stable, non-secret correlation identifiers and
may be exposed to providers and provider observability.
Notarius records PromptKit's selected backend ID and effective reasoning
setting as optional run-manifest provenance. Endpoint-only profiles have no
backend ID. Debug prompt material also retains the selected backend ID and
PromptKit's stable lower-case `effective_model_params` JSON, which may include
`backend_id`. Notarius production configuration does not expose user-defined
PromptKit backend registration.
Notarius retains its application-wide scheduled client around the PromptKit
adapter. PromptKit may apply a narrower limit for the selected backend;
endpoint-only profiles have no such backend limit. The adapter translates
PromptKit capacity rejection into the provider-neutral Notarius
`ErrLLMCapacityExceeded` contract and leaves retries to the calling pipeline
stage.
## Notarius Ownership
[LLM Runtime Internals](../internal/llm.md) describes how Notarius mounts

View File

@@ -61,6 +61,16 @@ construction errors are returned before a pipeline is prepared. Configuration
field definitions remain in [Configuration](../config.md#promptkit-profiles);
the adapter mechanics remain in [LLM Runtime](llm.md).
The factory also accepts `LLMRuntimeOverrides`, whose reasoning pointer
preserves inherit, replace, and clear states across the composition boundary.
Run orchestration constructs this value from the mutually exclusive
`--reasoning-effort` and `--clear-reasoning-effort` controls. Absence preserves
a nil pointer, replacement is trimmed, and clear uses a non-nil empty string.
The same override reaches the one shared production client, checkpoint
identity, and debug invocation metadata. Persistent reasoning configuration
remains owned by PromptKit profiles; Notarius configuration has no reasoning
field.
## Run Orchestration
After parsing and validating a run invocation, the CLI performs this ordered

View File

@@ -26,20 +26,35 @@ durable schemas. Those responsibilities remain with the module and its
`PromptKitClient` validates the request target and prompt identity, maps each
named material to a PromptKit inline artifact while preserving its origin URI,
maps the request session to the existing `session_id` prompt variable, forwards
profile selection, then prepares and runs the prompt. PromptKit v0.1.0 has no
direct request-level session field. The adapter returns PromptKits validated
raw bytes rather than re-encoding the decoded target. An empty optional
material is represented as one space so its named input is retained by
PromptKit.
maps the trimmed request session to PromptKit's direct per-run session field,
retains the same value as the `session_id` prompt variable for maintained
prompt compatibility, forwards profile selection, then prepares and runs the
prompt. The direct field is authoritative for provider session behavior. A
session ID is a stable, non-secret correlation identifier and may be exposed
to providers and provider observability. The adapter returns PromptKits
validated raw bytes rather than re-encoding the decoded target. An empty
optional material is represented as one space so its named input is retained
by PromptKit.
Client construction may also receive a run-wide reasoning-effort override from
the CLI factory boundary. The adapter copies the caller-owned pointer and
creates a fresh PromptKit execution override for each request: a nil pointer
inherits the selected profile, a non-empty value replaces it, and an empty
value clears inherited reasoning. The CLI's mutually exclusive
`--reasoning-effort` and `--clear-reasoning-effort` controls select those
states. With neither flag, profile behavior remains unchanged. Because
production constructs one shared client, the selected state applies uniformly
to module calls, retries, and LLM-backed validators for the whole run.
An empty request profile lets the prompt select its configured default. The CLI
prepares every explicitly selected binding profile before a run begins, so a
missing explicit profile fails before stage execution. Calls record the profile
actually selected by PromptKit; the recorder deduplicates non-secret profile
identity, provider, and model values for manifest use. Successful completion
responses and recorded profile manifests identify the adapter provider as
`promptkit`.
actually selected by PromptKit. The recorder trims and deduplicates non-secret
profile identity, provider, model, selected backend ID, and effective reasoning
values for manifest use. Entries that differ in backend or reasoning remain
distinct and deterministically ordered. Endpoint-only profiles retain an empty
backend ID, which the published JSON omits. Successful completion responses and
recorded profile manifests identify the adapter provider as `promptkit`.
Before execution, the adapter also contributes a non-secret checkpoint
fingerprint for the effective PromptKit profile source. It combines the
@@ -66,6 +81,13 @@ worker counts cannot exceed the configured LLM limit. The configuration field
and its effective default are owned by
[Configuration](../config.md#concurrency-output-cache-and-debug).
PromptKit applies a second, independent admission limit when the selected
profile names a limited backend. It sits beneath the Notarius scheduled client,
so it may narrow but cannot expand the application-wide limit. Built-in
OpenRouter profiles select PromptKit's reserved backend and its upstream
capacity policy. Endpoint-only profiles do not select a PromptKit backend and
remain limited only by the Notarius scheduler.
## Prompt And Schema Assets
An `AssetRegistry` collects prompt and schema filesystems from production module
@@ -117,6 +139,14 @@ failure, empty structured body, or decode failure as
material when they exist. Provider failures remain operational errors rather
than output-validation failures.
When PromptKit rejects backend admission before generation, the adapter maps
`promptkit.ErrCapacityExceeded` to
`contracts.ErrLLMCapacityExceeded`, retaining prompt context and a redacted
upstream diagnostic without exposing the PromptKit sentinel as a framework
contract. A canceled caller context takes precedence. The adapter does not
retry capacity failures; the pipeline's existing binding attempt policy sees
the operational error and decides whether to rerun the complete operation.
Prompt-declared repair is executed within PromptKits structured-output flow.
The current production D&D prompt manifests set repair attempts to zero. That
setting does not replace pipeline retry behavior: a bindings configured retry
@@ -136,7 +166,7 @@ does not install another timeout wrapper around PromptKit.
The selected PromptKit profile owns generation settings. Notarius binding
retries remain outside the adapter and repeat the complete module operation
and validation chain. PromptKit v0.1.0 does not add a provider retry loop.
and validation chain. PromptKit does not add a provider retry loop.
Operator-facing behavior is summarized in
[Operations](../operations.md#operational-limits), and the pinned upstream
contract is identified in
@@ -146,18 +176,22 @@ contract is identified in
When debug recording is enabled, the pipeline decorates the shared client. The
wrapper records prepared prompt and response material, timing, selected profile
and model, and call identifiers in the runs debug bundle, including material
available from a failed structured completion. For a successful completion, a
debug-write failure is surfaced; when the completion already failed, its call
error remains the result. Debug-bundle location, retention, and handling are
operational concerns documented in [Operations](../operations.md#debug-bundles).
and backend, effective model parameters, and call identifiers in the runs
debug bundle, including material available from a failed structured completion.
Effective parameters use PromptKit's stable lower-case JSON field names and may
include `backend_id`. For a successful completion, a debug-write failure is
surfaced; when the completion already failed, its call error remains the
result. Debug-bundle location, retention, and handling are operational concerns
documented in [Operations](../operations.md#debug-bundles).
Run manifests receive selected profile summaries and component identities, not
prompt, schema, source, reference, or response content. Provider error text is
wrapped with prompt context and bearer credentials are redacted before it
crosses the runtime boundary. Known-secret redaction is available to other
runtime collaborators; it does not make prompt or response contents safe for
general logging.
Run manifests receive selected profile summaries, including optional effective
backend and reasoning provenance, and component identities—not prompt, schema,
source, reference, or response content. The published field semantics belong
to the [JSON output contract](../integrations/json-output.md#manifestjson).
Provider error text is wrapped with prompt context and bearer credentials are
redacted before it crosses the runtime boundary. Known-secret redaction is
available to other runtime collaborators; it does not make prompt or response
contents safe for general logging.
## Failure Boundaries
@@ -165,6 +199,8 @@ general logging.
sources, invalid asset registration, or a non-positive scheduler limit.
- Preparation failures, unavailable explicit profiles, provider failures, and
context cancellation propagate to the calling stage with context.
- Backend admission exhaustion is a provider-neutral operational error and is
not classified as invalid structured output or validator rejection.
- Malformed or schema-invalid provider output is classified separately as
invalid structured output so the module or pipeline can apply its own retry
and rejection policy.

View File

@@ -121,6 +121,9 @@ provenance, the effective PromptKit profile-source fingerprint, and
prepared-component fingerprints. Changing profile content causes a cold miss
even when its profile ID is unchanged. A changed identity produces a cold miss;
Notarius does not migrate, rewrite, or delete older checkpoint directories.
Reasoning-effort inheritance, replacement, and explicit clearing are distinct
runtime identities, so checkpoints created under one state are not reused by
either of the others.
Checkpoint state is confined below an identity-specific path:
@@ -180,7 +183,9 @@ warning, checkpoint, chunk-plan, and terminal reporting artifacts. The trace
contains allowlisted application diagnostic records and can include source or
derived application data. Neither surface is a cache input. Do not treat a
debug bundle as safe to share merely because its configuration summary is
redacted.
redacted. Invocation metadata omits reasoning effort when it is inherited,
records the replacement value when one is supplied, and records an empty value
when inherited reasoning was explicitly cleared.
Notarius never creates debug state without an explicit request and never
automatically deletes a requested bundle. If allocation succeeds, the command
@@ -209,9 +214,12 @@ or automatic cleanup command.
## Operational Limits
Provider execution settings and the generation timeout come from the selected
PromptKit profile. PromptKit v0.1.0 does not add a provider retry loop;
Notarius binding retries rerun the complete module operation and validation
chain as defined by [module bindings](config.md#module-bindings-and-validators).
PromptKit profile. The invocation-only **--reasoning-effort** and
**--clear-reasoning-effort** controls may replace or clear that profile setting
for all LLM-backed calls in one run without changing the profile. PromptKit
v0.2.0 does not add a provider retry loop. Notarius binding retries rerun the
complete module operation and validation chain as defined by
[module bindings](config.md#module-bindings-and-validators).
Timeouts are layered. Caller cancellation is the outer authority. A positive
effective generation timeout adds an inner request deadline, while zero
@@ -220,7 +228,19 @@ transport-wide cap. Notarius does not add another timeout around PromptKit.
The pinned upstream boundary and profile-format links are in
[PromptKit Integration](integrations/pkg-promptkit.md).
Concurrency limits are configuration contracts; see
Concurrency has two independent layers. Notarius **total_llm** is the
application-wide provider-call limit shared by all backends, modules, retries,
and validators. PromptKit may impose a narrower admission limit for the
selected backend. The effective active-generation bound is the intersection of
both limits and can therefore be lower than **total_llm**. Built-in OpenRouter
profiles use PromptKit's upstream backend limit; endpoint-only profiles have no
PromptKit backend limit and remain bounded by Notarius.
When a PromptKit backend has admitted all active and queued work, a new call
fails as capacity exhaustion before generation. The adapter does not retry it.
The calling stage's configured retry policy applies normally, and the run fails
if those attempts are exhausted. Caller cancellation remains authoritative.
Configuration contracts are documented under
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
limits and actual provider-call limits are independent. Notarius writes local
filesystem state only; remote storage, archival, and retention automation are

View File

@@ -177,7 +177,8 @@ individual modules.
The application-wide LLM scheduler bounds actual provider calls independently
of framework worker limits. Every LLM-backed module, retry, and validator uses
the single injected scheduled client, including work performed by overlapping
lanes.
lanes. Provider runtime adapters may enforce a narrower backend-specific limit
beneath this mandatory application-wide scheduler.
## Configuration And Provenance

View File

@@ -91,31 +91,6 @@ safety checks, and deterministic application of accepted changes.
- Add media-type validators when non-JSON artifact representations are
introduced.
## LLM Runtime Evolution
### Native Session Propagation
- Once upstream PromptKit exposes a direct request-level session identifier,
propagate the existing `StructuredCompletionRequest.SessionID` through the
PromptKit adapter's native session field.
- Preserve the current `--session-id` invocation contract and its run-wide
propagation to every prompt-facing module and validator. Do not introduce a
second session configuration surface.
- Retain session identity in checkpoint provenance so runs with different
sessions cannot reuse one another's LLM-derived checkpoints.
- Define the upstream compatibility and prompt-variable transition explicitly:
native provider session behavior must not silently remove a `session_id`
prompt variable while maintained prompts still consume it.
- Add adapter and assembled-run coverage for exact forwarding, trimming,
concurrent-run isolation, and unsupported-provider behavior once the
upstream contract is available.
This work is blocked because PromptKit v0.1.0 does not expose the required
direct request-level session field. Notarius already carries a run-scoped
session ID through its CLI, pipeline requests, checkpoint identity, and a
`session_id` prompt variable; that prompt-variable propagation is not native
provider session support.
## Further Reference Evolution
- Make prior-run artifacts easier to bind as references without changing the

2
go.mod
View File

@@ -3,7 +3,7 @@ module gitea.maximumdirect.net/eric/notarius
go 1.25.5
require (
gitea.maximumdirect.net/eric/promptkit v0.1.0
gitea.maximumdirect.net/eric/promptkit v0.2.0
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
gopkg.in/yaml.v3 v3.0.1
)

4
go.sum
View File

@@ -1,5 +1,5 @@
gitea.maximumdirect.net/eric/promptkit v0.1.0 h1:vuKeBxkiY8E54LRFbLQFjlJJCiOfMvB1++DYBCrD/ug=
gitea.maximumdirect.net/eric/promptkit v0.1.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
gitea.maximumdirect.net/eric/promptkit v0.2.0 h1:6bUATKnVgLlHSbKjGb8EnCt2r7jV5Xw6t+pX7G9S/lU=
gitea.maximumdirect.net/eric/promptkit v0.2.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=

View File

@@ -143,7 +143,7 @@ func isEmptyRegistries(registries pipeline.Registries) bool {
registries.Outputs == nil
}
func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
if err := ctx.Err(); err != nil {
return nil, nil, err
}
@@ -151,16 +151,16 @@ func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileI
if err != nil {
return nil, nil, err
}
return buildProductionLLMClient(ctx, cfg, profileID, assets)
return buildProductionLLMClient(ctx, cfg, profileID, overrides, assets)
}
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
return func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
return buildProductionLLMClient(ctx, cfg, profileID, assets)
return func(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
return buildProductionLLMClient(ctx, cfg, profileID, overrides, assets)
}
}
func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
if err := ctx.Err(); err != nil {
return nil, nil, err
}
@@ -173,6 +173,7 @@ func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID
ProfileFile: cfg.PromptKit.ProfileFile,
Assets: assets,
Recorder: recorder,
ReasoningEffort: overrides.ReasoningEffort,
})
if err != nil {
return nil, nil, fmt.Errorf("create PromptKit-backed LLM client: %w", err)

View File

@@ -51,7 +51,7 @@ pipelines:
options := Options{
Catalog: catalogFromRegistries(components.registries),
Registries: components.registries,
LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
llmConstructed = true
return nil, nil, errors.New("LLM client must not be constructed")
},

View File

@@ -322,7 +322,7 @@ func TestProductionSpellNormalizerRejectsInvalidCatalogReferencesBeforeExecution
options := Options{
Catalog: catalogFromRegistries(components.registries),
Registries: components.registries,
LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
llmConstructed = true
return nil, nil, errors.New("LLM client must not be constructed")
},
@@ -381,7 +381,7 @@ func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) {
}
for _, tt := range factories {
t.Run(tt.name, func(t *testing.T) {
client, manifests, err := tt.factory(context.Background(), config.Default(), "test-profile")
client, manifests, err := tt.factory(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
if err != nil {
t.Fatalf("build production LLM runtime: %v", err)
}
@@ -410,14 +410,14 @@ func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
t.Run("canceled context", func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
client, manifests, err := productionLLMClientFactory(ctx, config.Default(), "test-profile")
client, manifests, err := productionLLMClientFactory(ctx, config.Default(), "test-profile", LLMRuntimeOverrides{})
if !errors.Is(err, context.Canceled) || client != nil || len(manifests) != 0 {
t.Fatalf("client=%T manifests=%#v error=%v, want canceled construction", client, manifests, err)
}
})
t.Run("nil assets", func(t *testing.T) {
client, manifests, err := productionLLMClientFactoryWithAssets(nil)(context.Background(), config.Default(), "test-profile")
client, manifests, err := productionLLMClientFactoryWithAssets(nil)(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
if err == nil || !strings.Contains(err.Error(), "asset registry must not be nil") || client != nil || len(manifests) != 0 {
t.Fatalf("client=%T manifests=%#v error=%v, want nil-assets failure", client, manifests, err)
}
@@ -427,7 +427,7 @@ func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
components := productionTestComponents(t)
cfg := config.Default()
cfg.Concurrency.TotalLLM = 0
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), cfg, "test-profile")
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), cfg, "test-profile", LLMRuntimeOverrides{})
if err == nil || !strings.Contains(err.Error(), "create LLM scheduler") || !strings.Contains(err.Error(), "greater than zero") || client != nil || len(manifests) != 0 {
t.Fatalf("client=%T manifests=%#v error=%v, want scheduler-construction failure", client, manifests, err)
}
@@ -682,7 +682,7 @@ func productionRunOptions(t *testing.T, fake *productionFakeLLMClient) Options {
options.Now = func() time.Time { return time.Unix(1700000000, 0).UTC() }
options.RunIDGenerator = func(time.Time) (string, error) { return productionRunID, nil }
options.UserCacheDir = func() (string, error) { return "", errors.New("user cache must not be used") }
options.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
options.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
return fake, nil, nil
}
return options

View File

@@ -30,7 +30,7 @@ import (
const defaultConfigPath = "/usr/local/etc/notarius/config.yml"
const usage = `Usage:
notarius help
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--output-dir path] [--json] [--chunk_cache auto|bypass|refresh] [--resume] [--recompute-step step-id] [--debug [--debug-dir path]] [--only lane-a,lane-b] [--session-id id] [--reference selector=path] [--without-reference selector]
notarius run <pipeline-id> --input path/to/source.json [--json] [flags]
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list --config path/to/config.yml [--json]
`
@@ -48,7 +48,11 @@ type Options struct {
DebugTerminalFactory func(*debugbundle.SummaryWriter) DebugTerminalWriter
}
type LLMClientFactory func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error)
type LLMRuntimeOverrides struct {
ReasoningEffort *string
}
type LLMClientFactory func(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error)
// Run executes the command-line interface and returns a process exit code.
func Run(args []string, stdout, stderr io.Writer) int {
@@ -138,13 +142,16 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
debug := fs.Bool("debug", false, "write a debug bundle")
debugDir := fs.String("debug-dir", "", "debug bundle directory")
llmProfile := fs.String("llm-profile", "", "LLM profile override")
reasoningEffort := singleValueFlag{name: "--reasoning-effort"}
clearReasoningEffort := fs.Bool("clear-reasoning-effort", false, "clear the LLM profile reasoning effort")
resume := fs.Bool("resume", false, "reuse compatible recorded checkpoints")
recomputeStep := singleValueFlag{}
recomputeStep := singleValueFlag{name: "--recompute-step"}
chunkCache := chunkCacheFlag{}
sessionID := sessionIDFlag{}
referenceFlags := stringListFlag{}
withoutReferenceFlags := stringListFlag{}
fs.Var(&sessionID, "session-id", "prompt session identifier")
fs.Var(&reasoningEffort, "reasoning-effort", "reasoning effort override")
fs.Var(&chunkCache, "chunk_cache", "chunk plan cache mode: auto, bypass, or refresh")
fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, merge.slot=path, lane.slot=path, lane.extract.slot=path, lane.merge.slot=path, or lane.normalize.slot=path")
fs.Var(&withoutReferenceFlags, "without-reference", "unbind a reference, using the same selector forms as --reference")
@@ -190,6 +197,22 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
fmt.Fprintln(stderr, "notarius: --session-id must not be empty")
return 2
}
if reasoningEffort.set && *clearReasoningEffort {
fmt.Fprintln(stderr, "notarius: --reasoning-effort cannot be combined with --clear-reasoning-effort")
return 2
}
if reasoningEffort.set && strings.TrimSpace(reasoningEffort.value) == "" {
fmt.Fprintln(stderr, "notarius: --reasoning-effort must not be empty")
return 2
}
runtimeOverrides := LLMRuntimeOverrides{}
if reasoningEffort.set {
value := strings.TrimSpace(reasoningEffort.value)
runtimeOverrides.ReasoningEffort = &value
} else if *clearReasoningEffort {
value := ""
runtimeOverrides.ReasoningEffort = &value
}
only, err := parseOnly(*onlyRaw)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
@@ -287,6 +310,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
ConfigSource: configSource(*configPath),
OnlyLanes: append([]string(nil), only...),
ChunkCacheOverride: chunkCache.explicitValue(),
ReasoningEffortOverride: runtimeOverrides.ReasoningEffort,
Resume: *resume,
RecomputeStep: strings.TrimSpace(recomputeStep.value),
RunID: runID,
@@ -364,7 +388,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
if len(profileIDs) == 1 {
factoryProfileID = profileIDs[0]
}
llmClient, llmProfiles, err := opts.LLMClientFactory(ctx, effective.Config, factoryProfileID)
llmClient, llmProfiles, err := opts.LLMClientFactory(ctx, effective.Config, factoryProfileID, runtimeOverrides)
if err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("create LLM client for profile %q: %w", factoryProfileID, err))
}
@@ -388,7 +412,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
if err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, err)
}
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(effective.Config.Cache.Checkpoints, opts, effective.ResolvedPipeline, prepared.CheckpointFingerprints(), llmFingerprints, rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), *resume)
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(effective.Config.Cache.Checkpoints, opts, effective.ResolvedPipeline, prepared.CheckpointFingerprints(), llmFingerprints, rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), runtimeOverrides, *resume)
if err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, err)
}
@@ -495,6 +519,7 @@ func checkpointHandlersForRun(
llmProfiles []artifacts.LLMProfileManifest,
llmProfileOverride string,
sessionID string,
runtimeOverrides LLMRuntimeOverrides,
resume bool,
) (pipeline.CheckpointRecorder, pipeline.CheckpointLoader, error) {
if !settings.Enabled {
@@ -508,7 +533,7 @@ func checkpointHandlersForRun(
InputKey: resolved.Input.Module,
RawInputDigest: rawInputDigest(rawInput),
SelectedLanes: only,
RuntimeOverrides: runtimeOverrideFingerprints(llmProfileOverride, sessionID),
RuntimeOverrides: runtimeOverrideFingerprints(llmProfileOverride, sessionID, runtimeOverrides),
References: pipeline.ReferenceProvenance(resolved),
ProvenanceFingerprints: combineCheckpointFingerprints(
llmProfileFingerprints(llmProfiles),
@@ -669,7 +694,7 @@ func rawInputDigest(data []byte) string {
return "sha256:" + hex.EncodeToString(sum[:])
}
func runtimeOverrideFingerprints(llmProfileOverride string, sessionID string) []checkpoint.Fingerprint {
func runtimeOverrideFingerprints(llmProfileOverride string, sessionID string, runtimeOverrides LLMRuntimeOverrides) []checkpoint.Fingerprint {
var values []checkpoint.Fingerprint
if strings.TrimSpace(llmProfileOverride) != "" {
values = append(values, checkpoint.Fingerprint{Name: "llm_profile_override", Value: strings.TrimSpace(llmProfileOverride)})
@@ -677,6 +702,13 @@ func runtimeOverrideFingerprints(llmProfileOverride string, sessionID string) []
if strings.TrimSpace(sessionID) != "" {
values = append(values, checkpoint.Fingerprint{Name: "session_id", Value: strings.TrimSpace(sessionID)})
}
if runtimeOverrides.ReasoningEffort != nil {
value := strings.TrimSpace(*runtimeOverrides.ReasoningEffort)
if value == "" {
value = "<cleared>"
}
values = append(values, checkpoint.Fingerprint{Name: "reasoning_effort_override", Value: value})
}
return values
}
@@ -833,7 +865,7 @@ func reorderRunArgs(args []string) []string {
func runFlagTakesValue(arg string) bool {
switch arg {
case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--chunk_cache", "--reference", "--without-reference", "--recompute-step":
case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--reasoning-effort", "--chunk_cache", "--reference", "--without-reference", "--recompute-step":
return true
default:
return false
@@ -893,11 +925,11 @@ func chunkPlanStoreForRun(cfg config.ChunkPlanCacheConfig, opts Options) (pipeli
func validateRunFlagValues(args []string) error {
for i, arg := range args {
if arg != "--session-id" {
if arg != "--session-id" && arg != "--reasoning-effort" {
continue
}
if i+1 >= len(args) || strings.HasPrefix(args[i+1], "-") {
return fmt.Errorf("flag needs an argument: --session-id")
return fmt.Errorf("flag needs an argument: %s", arg)
}
}
return nil
@@ -1181,6 +1213,7 @@ type sessionIDFlag struct {
}
type singleValueFlag struct {
name string
value string
set bool
}
@@ -1194,7 +1227,7 @@ func (flag *singleValueFlag) String() string {
func (flag *singleValueFlag) Set(value string) error {
if flag.set {
return fmt.Errorf("--recompute-step may be specified only once")
return fmt.Errorf("%s may be specified only once", flag.name)
}
flag.value = value
flag.set = true

View File

@@ -12,6 +12,7 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/checkpoint"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
@@ -245,8 +246,10 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
harness := newStateTestHarness()
var factoryProfiles []string
opts := harness.options()
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
var factoryOverrides []LLMRuntimeOverrides
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
factoryProfiles = append(factoryProfiles, profileID)
factoryOverrides = append(factoryOverrides, overrides)
return nil, nil, nil
}
var stdout, stderr bytes.Buffer
@@ -257,6 +260,9 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" {
t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles)
}
if len(factoryOverrides) != 1 || factoryOverrides[0].ReasoningEffort != nil {
t.Fatalf("factory overrides = %#v, want inherited reasoning", factoryOverrides)
}
harness.mu.Lock()
profiles := append([]string(nil), harness.moduleProfiles...)
harness.mu.Unlock()
@@ -279,7 +285,7 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
opts := harness.options()
registerRunContractValidator(t, &opts, &validatorProfiles)
factoryProfiles := []string{}
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string, _ LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
factoryProfiles = append(factoryProfiles, profileID)
return nil, nil, nil
}
@@ -302,7 +308,7 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
factoryCalls := 0
opts := newStateTestHarness().options()
opts.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
factoryCalls++
return nil, nil, nil
}
@@ -314,6 +320,129 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
})
}
func TestRunReasoningEffortOverrideReachesFactory(t *testing.T) {
tests := []struct {
name string
flags []string
wantValue string
wantSet bool
}{
{name: "inherit"},
{name: "replace", flags: []string{"--reasoning-effort", " focused "}, wantValue: "focused", wantSet: true},
{name: "clear", flags: []string{"--clear-reasoning-effort"}, wantSet: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
roots := newStateTestRoots(t)
opts := newStateTestHarness().options()
var got []LLMRuntimeOverrides
opts.LLMClientFactory = func(_ context.Context, _ config.Config, _ string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
got = append(got, overrides)
return nil, nil, nil
}
args := append([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, tt.flags...)
var stdout, stderr bytes.Buffer
if code := RunWithOptions(args, &stdout, &stderr, opts); code != 0 || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
if len(got) != 1 {
t.Fatalf("factory overrides = %#v, want one call", got)
}
if !tt.wantSet {
if got[0].ReasoningEffort != nil {
t.Fatalf("reasoning effort = %q, want inherit", *got[0].ReasoningEffort)
}
return
}
if got[0].ReasoningEffort == nil || *got[0].ReasoningEffort != tt.wantValue {
t.Fatalf("reasoning effort = %#v, want %q", got[0].ReasoningEffort, tt.wantValue)
}
})
}
}
func TestRunReasoningEffortOverrideRejectsInvalidSyntax(t *testing.T) {
tests := []struct {
name string
flags []string
wantError string
}{
{
name: "mutually exclusive controls",
flags: []string{"--reasoning-effort", "focused", "--clear-reasoning-effort"},
wantError: "cannot be combined",
},
{
name: "empty replacement",
flags: []string{"--reasoning-effort", " "},
wantError: "must not be empty",
},
{
name: "duplicate replacement",
flags: []string{"--reasoning-effort", "low", "--reasoning-effort", "high"},
wantError: "may be specified only once",
},
{
name: "missing replacement",
flags: []string{"--reasoning-effort"},
wantError: "flag needs an argument",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
roots := newStateTestRoots(t)
args := append([]string{"run", "sample", "--config", roots.config, "--input", roots.input}, tt.flags...)
var stdout, stderr bytes.Buffer
code := RunWithOptions(args, &stdout, &stderr, newStateTestHarness().options())
if code != 2 || stdout.Len() != 0 || !strings.Contains(stderr.String(), tt.wantError) {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
assertNoRunState(t, roots)
})
}
}
func TestReasoningEffortOverrideSeparatesCheckpointIdentities(t *testing.T) {
replacement := " focused "
cleared := ""
states := []struct {
name string
overrides LLMRuntimeOverrides
wantValue string
wantSet bool
}{
{name: "inherit"},
{name: "replace", overrides: LLMRuntimeOverrides{ReasoningEffort: &replacement}, wantValue: "focused", wantSet: true},
{name: "clear", overrides: LLMRuntimeOverrides{ReasoningEffort: &cleared}, wantValue: "<cleared>", wantSet: true},
}
digests := make(map[string]string, len(states))
for _, state := range states {
fingerprints := runtimeOverrideFingerprints("", "", state.overrides)
var value string
var found bool
for _, fingerprint := range fingerprints {
if fingerprint.Name == "reasoning_effort_override" {
value, found = fingerprint.Value, true
}
}
if found != state.wantSet || (found && value != state.wantValue) {
t.Fatalf("%s fingerprint found=%t value=%q, want found=%t value=%q", state.name, found, value, state.wantSet, state.wantValue)
}
identity, err := checkpoint.NewIdentity(checkpoint.IdentityInput{
Pipeline: pipeline.ResolvedPipeline{ID: "sample", Digest: "sha256:pipeline", Input: pipeline.Binding("test/input")},
RawInputDigest: "sha256:input",
RuntimeOverrides: fingerprints,
})
if err != nil {
t.Fatal(err)
}
digests[state.name] = identity.Digest
}
if digests["inherit"] == digests["replace"] || digests["inherit"] == digests["clear"] || digests["replace"] == digests["clear"] {
t.Fatalf("checkpoint identity digests are not distinct: %#v", digests)
}
}
func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
resolved := pipeline.ResolvedPipeline{
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
@@ -376,7 +505,7 @@ func TestRunFactoryAndPreparationFailuresAreProcessFailures(t *testing.T) {
t.Run("LLM factory", func(t *testing.T) {
roots := newStateTestRoots(t)
opts := newStateTestHarness().options()
opts.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
return nil, nil, errors.New("injected LLM factory failure")
}
var stdout, stderr bytes.Buffer

View File

@@ -216,7 +216,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
fingerprints := prepared.CheckpointFingerprints()
llmFingerprints := []checkpoint.Fingerprint{{Name: "promptkit_profile_source", Value: "sha256:profile-source-one"}}
settings := config.CheckpointCacheConfig{Enabled: true, Directory: t.TempDir()}
recorder, _, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", false)
recorder, _, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, false)
if err != nil {
t.Fatal(err)
}
@@ -242,7 +242,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
t.Fatal(err)
}
_, sameLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", true)
_, sameLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
if err != nil {
t.Fatal(err)
}
@@ -254,7 +254,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
}
changed := replaceCheckpointFingerprintValue(t, fingerprints, normalizeSpellCatalogFingerprintName(), "sha256:changed-effective-catalog")
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changed, normalizeSpellCatalogFingerprintName())
_, changedLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changed, llmFingerprints, []byte("same input"), nil, nil, "", "", true)
_, changedLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changed, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
if err != nil {
t.Fatal(err)
}
@@ -266,7 +266,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
}
changedMapping := replaceCheckpointFingerprintValue(t, fingerprints, extractSpellMappingFingerprintName(), "dnd.spells.extract_mapping.v3")
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changedMapping, extractSpellMappingFingerprintName())
_, mappingLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changedMapping, llmFingerprints, []byte("same input"), nil, nil, "", "", true)
_, mappingLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changedMapping, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
if err != nil {
t.Fatal(err)
}
@@ -275,7 +275,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
}
changedLLMFingerprints := []checkpoint.Fingerprint{{Name: "promptkit_profile_source", Value: "sha256:profile-source-two"}}
_, profileLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, changedLLMFingerprints, []byte("same input"), nil, nil, "", "", true)
_, profileLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, changedLLMFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
if err != nil {
t.Fatal(err)
}

View File

@@ -848,7 +848,7 @@ func (h *stateTestHarness) options() Options {
defer h.mu.Unlock()
h.runIDCalls++
return fmt.Sprintf("run-%d-%032x", startedAt.UnixNano(), h.runIDCalls), nil
}, UserCacheDir: func() (string, error) { return "", errors.New("unexpected user cache lookup") }, LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
}, UserCacheDir: func() (string, error) { return "", errors.New("unexpected user cache lookup") }, LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
return nil, nil, nil
}}
}

View File

@@ -1,6 +1,7 @@
package artifacts
import (
"strings"
"time"
)
@@ -29,6 +30,29 @@ type LLMProfileManifest struct {
ID string `json:"id"`
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
BackendID string `json:"backend_id,omitempty"`
ReasoningEffort string `json:"reasoning_effort,omitempty"`
}
// Normalized returns the canonical representation used for manifest identity
// and publication.
func (profile LLMProfileManifest) Normalized() LLMProfileManifest {
profile.ID = strings.TrimSpace(profile.ID)
profile.Provider = strings.TrimSpace(profile.Provider)
profile.Model = strings.TrimSpace(profile.Model)
profile.BackendID = strings.TrimSpace(profile.BackendID)
profile.ReasoningEffort = strings.TrimSpace(profile.ReasoningEffort)
return profile
}
// IdentityKey returns an opaque, deterministic key for the effective profile.
func (profile LLMProfileManifest) IdentityKey() string {
profile = profile.Normalized()
return profile.ID + "\x00" +
profile.Provider + "\x00" +
profile.Model + "\x00" +
profile.BackendID + "\x00" +
profile.ReasoningEffort
}
type ReferenceProvenance struct {

View File

@@ -53,7 +53,13 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
PipelineID: "pipeline-1",
PipelineDigest: "sha256:abc123",
LLMProfiles: []LLMProfileManifest{
{ID: "default", Provider: "promptkit", Model: "model-a"},
{
ID: "default",
Provider: "promptkit",
Model: "model-a",
BackendID: "openrouter",
ReasoningEffort: "high",
},
},
ArtifactLanes: []ArtifactLaneManifest{
{
@@ -101,10 +107,13 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
if !ok {
t.Fatalf("llm_profiles[0] = %#v, want object", profiles[0])
}
assertHasKeys(t, profile, "id", "provider", "model")
assertHasKeys(t, profile, "id", "provider", "model", "backend_id", "reasoning_effort")
if profile["provider"] != "promptkit" {
t.Fatalf("llm_profiles[0].provider = %#v, want promptkit", profile["provider"])
}
if profile["backend_id"] != "openrouter" || profile["reasoning_effort"] != "high" {
t.Fatalf("llm_profiles[0] = %#v, want backend and reasoning provenance", profile)
}
lanes, ok := got["artifact_lanes"].([]any)
if !ok {

View File

@@ -2,6 +2,7 @@ package debugbundle
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"strings"
@@ -137,6 +138,47 @@ func TestSummaryWriterWritesEverySummaryArtifact(t *testing.T) {
}
}
}
func TestWriteInvocationPreservesReasoningEffortOverrideStates(t *testing.T) {
replacement := "focused"
cleared := ""
tests := []struct {
name string
override *string
wantValue string
wantSet bool
}{
{name: "inherit"},
{name: "replace", override: &replacement, wantValue: "focused", wantSet: true},
{name: "clear", override: &cleared, wantSet: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
bundle, err := Allocate(t.TempDir(), testBundleRunID, time.Unix(0, 42))
if err != nil {
t.Fatal(err)
}
if err := bundle.Summary().WriteInvocation(Invocation{
Operation: "run",
ReasoningEffortOverride: tt.override,
}); err != nil {
t.Fatal(err)
}
data, err := os.ReadFile(filepath.Join(bundle.SummaryRoot(), ArtifactInvocationMetadata))
if err != nil {
t.Fatal(err)
}
var payload map[string]any
if err := json.Unmarshal(data, &payload); err != nil {
t.Fatal(err)
}
value, found := payload["reasoning_effort_override"]
if found != tt.wantSet || (found && value != tt.wantValue) {
t.Fatalf("reasoning override found=%t value=%#v, want found=%t value=%q; JSON=%s", found, value, tt.wantSet, tt.wantValue, data)
}
})
}
}
func TestSummaryWriterInternalWritesConfineArtifacts(t *testing.T) {
bundle, err := Allocate(t.TempDir(), testBundleRunID, time.Unix(0, 42))
if err != nil {

View File

@@ -39,6 +39,7 @@ type Invocation struct {
ConfigSource string `json:"config_source,omitempty"`
OnlyLanes []string `json:"only_lanes,omitempty"`
ChunkCacheOverride string `json:"chunk_cache_override,omitempty"`
ReasoningEffortOverride *string `json:"reasoning_effort_override,omitempty"`
RunID string `json:"run_id"`
StartedAt time.Time `json:"started_at"`
}
@@ -68,6 +69,10 @@ func (w *SummaryWriter) WriteInvocation(payload Invocation) error {
if payload.StartedAt.IsZero() {
payload.StartedAt = w.createdAt
}
if payload.ReasoningEffortOverride != nil {
value := *payload.ReasoningEffortOverride
payload.ReasoningEffortOverride = &value
}
return w.writeJSON(ArtifactInvocationMetadata, payload)
}
func (w *SummaryWriter) WriteRedactedEffectiveConfig(payload RedactedSummaryPayload) error {

View File

@@ -43,6 +43,7 @@ type LLMDebugPrompt struct {
PromptVersion string `json:"prompt_version,omitempty"`
PromptHash string `json:"prompt_hash,omitempty"`
SelectedProfileID string `json:"selected_profile_id,omitempty"`
SelectedBackendID string `json:"selected_backend_id,omitempty"`
SessionID string `json:"session_id,omitempty"`
RenderedPromptHash string `json:"rendered_prompt_hash,omitempty"`
Messages []LLMDebugMessage `json:"messages,omitempty"`

View File

@@ -5,3 +5,7 @@ import "errors"
// ErrInvalidStructuredOutput identifies a provider response that cannot satisfy
// the caller's declared structured-output contract.
var ErrInvalidStructuredOutput = errors.New("invalid structured output")
// ErrLLMCapacityExceeded identifies backend admission exhaustion before model
// generation begins.
var ErrLLMCapacityExceeded = errors.New("LLM capacity exceeded")

View File

@@ -3,6 +3,7 @@ package llm
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"regexp"
@@ -26,6 +27,7 @@ type PromptKitClientConfig struct {
HTTPClient *http.Client
EngineOptions []promptkit.Option
Recorder *LLMProfileRecorder
ReasoningEffort *string
}
type PromptKitClient struct {
@@ -33,6 +35,7 @@ type PromptKitClient struct {
recorder *LLMProfileRecorder
profileDir string
profileFile string
reasoningEffort *string
}
type LLMProfileRecorder struct {
@@ -71,11 +74,17 @@ func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
if recorder == nil {
recorder = NewLLMProfileRecorder()
}
var reasoningEffort *string
if cfg.ReasoningEffort != nil {
value := *cfg.ReasoningEffort
reasoningEffort = &value
}
return &PromptKitClient{
engine: engine,
recorder: recorder,
profileDir: strings.TrimSpace(cfg.ProfileDir),
profileFile: strings.TrimSpace(cfg.ProfileFile),
reasoningEffort: reasoningEffort,
}, nil
}
@@ -93,14 +102,23 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
if promptID == "" {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion prompt_id must not be empty")
}
sessionID := strings.TrimSpace(req.SessionID)
var execution *promptkit.ExecutionTargetOverride
if c.reasoningEffort != nil {
reasoningEffort := *c.reasoningEffort
execution = &promptkit.ExecutionTargetOverride{
ReasoningEffort: &reasoningEffort,
}
}
runReq := promptkit.RunRequest{
PromptID: promptID,
PromptVersion: strings.TrimSpace(req.PromptVersion),
ProfileID: strings.TrimSpace(req.ProfileID),
SessionID: sessionID,
Inputs: promptKitInputs(req.Inputs),
Vars: promptKitVars(req),
Metadata: promptKitMetadata(req),
Vars: promptKitVars(req, sessionID),
Execution: execution,
}
prepared, err := c.engine.Prepare(ctx, runReq)
if err != nil {
@@ -114,6 +132,14 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
if ctxErr := ctx.Err(); ctxErr != nil {
return contracts.StructuredCompletionResponse{}, ctxErr
}
if errors.Is(err, promptkit.ErrCapacityExceeded) {
return contracts.StructuredCompletionResponse{}, fmt.Errorf(
"run PromptKit prompt %q: %w: %v",
promptID,
contracts.ErrLLMCapacityExceeded,
redactPromptKitError(err),
)
}
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w", promptID, redactPromptKitError(err))
}
if result == nil {
@@ -141,6 +167,8 @@ func (c *PromptKitClient) responseFromResult(result *promptkit.RunResult, prepar
ID: strings.TrimSpace(result.SelectedProfileID),
Provider: promptKitProviderName,
Model: firstNonEmpty(result.ModelName, result.EffectiveModelParams.Model),
BackendID: strings.TrimSpace(result.SelectedBackendID),
ReasoningEffort: strings.TrimSpace(result.EffectiveModelParams.ReasoningEffort),
}
if c.recorder != nil {
c.recorder.Record(profile)
@@ -188,6 +216,7 @@ func promptKitDebugPrompt(prepared *promptkit.PreparedRun) *contracts.LLMDebugPr
PromptVersion: prepared.PromptVersion,
PromptHash: prepared.PromptHash,
SelectedProfileID: prepared.SelectedProfileID,
SelectedBackendID: prepared.SelectedBackendID,
SessionID: prepared.SessionID,
RenderedPromptHash: prepared.RenderedPromptHash,
Messages: messages,
@@ -284,10 +313,8 @@ func (r *LLMProfileRecorder) Record(profile artifacts.LLMProfileManifest) {
if r == nil {
return
}
profile.ID = strings.TrimSpace(profile.ID)
profile.Provider = strings.TrimSpace(profile.Provider)
profile.Model = strings.TrimSpace(profile.Model)
key := profile.ID + "\x00" + profile.Provider + "\x00" + profile.Model
profile = profile.Normalized()
key := profile.IdentityKey()
r.mu.Lock()
defer r.mu.Unlock()
if r.profiles == nil {
@@ -343,7 +370,7 @@ func promptKitInputs(inputs contracts.LLMInputSet) map[string]promptkit.Artifact
return out
}
func promptKitVars(req contracts.StructuredCompletionRequest) map[string]string {
func promptKitVars(req contracts.StructuredCompletionRequest, sessionID string) map[string]string {
vars := make(map[string]string, len(req.Vars)+1)
for key, value := range req.Vars {
name := strings.TrimSpace(key)
@@ -352,7 +379,7 @@ func promptKitVars(req contracts.StructuredCompletionRequest) map[string]string
}
vars[name] = fmt.Sprint(value)
}
if sessionID := strings.TrimSpace(req.SessionID); sessionID != "" {
if sessionID != "" {
vars["session_id"] = sessionID
}
if len(vars) == 0 {
@@ -361,17 +388,6 @@ func promptKitVars(req contracts.StructuredCompletionRequest) map[string]string
return vars
}
func promptKitMetadata(req contracts.StructuredCompletionRequest) map[string]string {
metadata := map[string]string{}
if stageName := strings.TrimSpace(req.StageName); stageName != "" {
metadata["stage_name"] = stageName
}
if len(metadata) == 0 {
return nil
}
return metadata
}
var bearerTokenPattern = regexp.MustCompile(`(?i)Bearer\s+[A-Za-z0-9._~+/=-]+`)
func redactPromptKitError(err error) error {

View File

@@ -8,6 +8,7 @@ import (
"net/http"
"os"
"path/filepath"
"reflect"
"strings"
"sync"
"sync/atomic"
@@ -15,6 +16,7 @@ import (
"testing/fstest"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/promptkit"
)
@@ -28,7 +30,7 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
}
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
StageName: "test-stage",
PromptID: "adapter.test",
PromptID: "adapter.direct-session",
PromptVersion: "v1",
ProfileID: "explicit-profile",
SessionID: " session-123 ",
@@ -52,8 +54,15 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
if resp.Debug == nil || resp.Debug.Prompt == nil {
t.Fatalf("debug prompt = nil, want prepared prompt material")
}
if resp.Debug.Prompt.PromptID != "adapter.test" || resp.Debug.Prompt.SelectedProfileID != "explicit-profile" {
t.Fatalf("debug prompt metadata = %#v, want prompt/profile", resp.Debug.Prompt)
if resp.Debug.Prompt.PromptID != "adapter.direct-session" ||
resp.Debug.Prompt.SelectedProfileID != "explicit-profile" ||
resp.Debug.Prompt.SelectedBackendID != "test-backend" ||
resp.Debug.Prompt.SessionID != "session-123" {
t.Fatalf("debug prompt metadata = %#v, want prompt/profile/backend/session", resp.Debug.Prompt)
}
if resp.Debug.Prompt.EffectiveModelParams["backend_id"] != "test-backend" ||
resp.Debug.Prompt.EffectiveModelParams["reasoning_effort"] != "profile-reasoning" {
t.Fatalf("debug effective model params = %#v, want backend and reasoning", resp.Debug.Prompt.EffectiveModelParams)
}
if len(resp.Debug.Prompt.Messages) != 1 || !strings.Contains(resp.Debug.Prompt.Messages[0].Content, `{"source":true}`) {
t.Fatalf("debug prompt messages = %#v, want rendered input content", resp.Debug.Prompt.Messages)
@@ -64,6 +73,10 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
if resp.Debug.Response.Usage.CachedTokens != 5 || resp.Debug.Response.Usage.CacheWriteTokens != 3 {
t.Fatalf("debug usage = %#v, want cached token counts", resp.Debug.Response.Usage)
}
if resp.Debug.Response.EffectiveModelParams["backend_id"] != "test-backend" ||
resp.Debug.Response.EffectiveModelParams["reasoning_effort"] != "profile-reasoning" {
t.Fatalf("debug response effective model params = %#v, want backend and reasoning", resp.Debug.Response.EffectiveModelParams)
}
debugJSON, err := json.Marshal(resp.Debug)
if err != nil {
t.Fatalf("marshal debug material: %v", err)
@@ -78,6 +91,9 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
if gotReq.Target.Model != "explicit-model" {
t.Fatalf("model = %q, want explicit-model", gotReq.Target.Model)
}
if gotReq.Target.BackendID != "test-backend" {
t.Fatalf("backend id = %q, want test-backend", gotReq.Target.BackendID)
}
if len(gotReq.Prompt.Messages) != 1 ||
!strings.Contains(gotReq.Prompt.Messages[0].Content, `{"source":true}`) ||
!strings.Contains(gotReq.Prompt.Messages[0].Content, "value") {
@@ -90,11 +106,124 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
if len(manifests) != 1 ||
manifests[0].ID != "explicit-profile" ||
manifests[0].Provider != "promptkit" ||
manifests[0].Model != "explicit-model" {
manifests[0].Model != "explicit-model" ||
manifests[0].BackendID != "test-backend" ||
manifests[0].ReasoningEffort != "profile-reasoning" {
t.Fatalf("profile manifests = %#v", manifests)
}
}
func TestPromptKitClientRetainsSessionPromptVariable(t *testing.T) {
fake := &fakePromptKitLLM{content: `{"ok":true}`}
client := newTestPromptKitClient(t, fake)
var out map[string]any
_, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
PromptID: "adapter.test",
SessionID: " canonical-session ",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
Vars: map[string]any{
"custom": "value",
"session_id": "caller-session",
},
}, &out)
if err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", err)
}
gotReq := fake.lastRequest()
if gotReq.Prompt.SessionID != "canonical-session" {
t.Fatalf("session id = %q, want canonical-session", gotReq.Prompt.SessionID)
}
if len(gotReq.Prompt.Messages) != 1 ||
!strings.Contains(gotReq.Prompt.Messages[0].Content, "Session: canonical-session") ||
strings.Contains(gotReq.Prompt.Messages[0].Content, "caller-session") {
t.Fatalf("rendered messages = %#v, want canonical session compatibility variable", gotReq.Prompt.Messages)
}
}
func TestPromptKitClientDoesNotInventDirectSession(t *testing.T) {
fake := &fakePromptKitLLM{content: `{"ok":true}`}
client := newTestPromptKitClient(t, fake)
var out map[string]any
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
PromptID: "adapter.direct-session",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
Vars: map[string]any{"custom": "value"},
}, &out)
if err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", err)
}
if got := fake.lastRequest().Prompt.SessionID; got != "" {
t.Fatalf("session id = %q, want empty", got)
}
if resp.Debug == nil || resp.Debug.Prompt == nil || resp.Debug.Prompt.SessionID != "" {
t.Fatalf("debug prompt = %#v, want no effective session", resp.Debug)
}
}
func TestPromptKitClientAppliesReasoningEffortOverride(t *testing.T) {
tests := []struct {
name string
override func() *string
mutateAfterCreate bool
want string
}{
{
name: "inherit",
want: "profile-reasoning",
},
{
name: "replace",
override: func() *string { value := "focused"; return &value },
want: "focused",
},
{
name: "clear",
override: func() *string { value := ""; return &value },
want: "",
},
{
name: "defensive copy",
override: func() *string { value := "original"; return &value },
mutateAfterCreate: true,
want: "original",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
fake := &fakePromptKitLLM{content: `{"ok":true}`}
var override *string
if tt.override != nil {
override = tt.override()
}
client := newTestPromptKitClientWithReasoning(t, fake, override)
if tt.mutateAfterCreate {
*override = "mutated"
}
var out map[string]any
_, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
PromptID: "adapter.direct-session",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
Vars: map[string]any{"custom": "value"},
}, &out)
if err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", err)
}
if got := fake.lastRequest().Target.ReasoningEffort; got != tt.want {
t.Fatalf("reasoning effort = %q, want %q", got, tt.want)
}
})
}
}
func TestNewPromptKitClientReportsAssetAndEngineConstructionFailures(t *testing.T) {
t.Run("assets", func(t *testing.T) {
registry := NewAssetRegistry()
@@ -176,6 +305,10 @@ func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
if fresh[0].Value == "mutated" {
t.Fatal("LLMCheckpointFingerprints exposed mutable backing storage")
}
const wantBuiltinFingerprint = "sha256:4b48cd1ef010b587cd8d56bc13b73a0cdd84da1414a8de9f9a2adc695dfcf0a7"
if fresh[0].Value != wantBuiltinFingerprint {
t.Fatalf("built-in profile fingerprint = %q, want %q", fresh[0].Value, wantBuiltinFingerprint)
}
}
func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) {
@@ -195,6 +328,30 @@ func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testi
if got := fake.lastRequest().Target.Model; got != "default-model" {
t.Fatalf("model = %q, want prompt default profile model", got)
}
manifests := client.LLMProfileManifests()
if len(manifests) != 1 || manifests[0].BackendID != "" || manifests[0].ReasoningEffort != "profile-reasoning" {
t.Fatalf("endpoint-only profile manifests = %#v, want omitted backend and effective reasoning", manifests)
}
}
func TestLLMProfileRecorderDistinguishesEffectiveTargets(t *testing.T) {
recorder := NewLLMProfileRecorder()
for _, profile := range []artifacts.LLMProfileManifest{
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-b", ReasoningEffort: "low"},
{ID: " profile ", Provider: " promptkit ", Model: " model ", BackendID: " backend-a ", ReasoningEffort: " low "},
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "high"},
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "low"},
} {
recorder.Record(profile)
}
want := []artifacts.LLMProfileManifest{
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "high"},
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "low"},
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-b", ReasoningEffort: "low"},
}
if got := recorder.Manifests(); !reflect.DeepEqual(got, want) {
t.Fatalf("profile manifests = %#v, want %#v", got, want)
}
}
func TestPromptKitClientValidationFailureReturnsError(t *testing.T) {
@@ -272,6 +429,89 @@ func TestPromptKitClientProviderFailureIncludesContextAndRedactsBearerToken(t *t
}
}
func TestPromptKitClientTranslatesBackendCapacityExhaustion(t *testing.T) {
queueCapacity := 0
fake := &fakePromptKitLLM{
err: errors.New("provider failed with Bearer secret-token"),
block: make(chan struct{}),
}
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
EngineOptions: []promptkit.Option{
promptkit.WithBackend(promptkit.Backend{
ID: "limited-backend",
Endpoint: "http://127.0.0.1:1/v1",
ConcurrencyLimit: 1,
QueueCapacity: &queueCapacity,
}),
promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
ID: "limited-profile",
BackendID: "limited-backend",
Model: "limited-model",
})),
promptkit.WithLLMClient(fake),
},
})
if err != nil {
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
}
defer func() {
select {
case <-fake.block:
default:
close(fake.block)
}
}()
request := contracts.StructuredCompletionRequest{
PromptID: "adapter.direct-session",
ProfileID: "limited-profile",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
Vars: map[string]any{"custom": "value"},
}
firstResult := make(chan error, 1)
go func() {
var out map[string]any
_, callErr := client.CompleteStructured(context.Background(), request, &out)
firstResult <- callErr
}()
waitForAtomicAtLeast(t, &fake.calls, 1)
var out map[string]any
response, capacityErr := client.CompleteStructured(context.Background(), request, &out)
if len(response.Content) != 0 {
t.Fatalf("capacity response = %#v, want empty", response)
}
if !errors.Is(capacityErr, contracts.ErrLLMCapacityExceeded) {
t.Fatalf("capacity error = %v, want ErrLLMCapacityExceeded", capacityErr)
}
if errors.Is(capacityErr, contracts.ErrInvalidStructuredOutput) {
t.Fatalf("capacity error = %v, must not be invalid structured output", capacityErr)
}
if errors.Is(capacityErr, promptkit.ErrCapacityExceeded) {
t.Fatalf("capacity error exposes PromptKit sentinel: %v", capacityErr)
}
if !strings.Contains(capacityErr.Error(), `run PromptKit prompt "adapter.direct-session"`) ||
!strings.Contains(capacityErr.Error(), "backend capacity exceeded") {
t.Fatalf("capacity error = %q, want prompt context and upstream diagnostic", capacityErr)
}
if calls := atomic.LoadInt32(&fake.calls); calls != 1 {
t.Fatalf("provider calls after capacity rejection = %d, want 1", calls)
}
close(fake.block)
firstErr := <-firstResult
if firstErr == nil || strings.Contains(firstErr.Error(), "secret-token") ||
!strings.Contains(firstErr.Error(), "Bearer [REDACTED]") {
t.Fatalf("admitted provider error = %v, want redacted diagnostic", firstErr)
}
if calls := atomic.LoadInt32(&fake.calls); calls != 1 {
t.Fatalf("provider calls after release = %d, want no adapter retry", calls)
}
}
func TestPromptKitClientContextCancellationIsRespected(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
@@ -405,21 +645,32 @@ func TestPromptKitClientValidatesRequest(t *testing.T) {
}
func newTestPromptKitClient(t *testing.T, fake *fakePromptKitLLM) *PromptKitClient {
return newTestPromptKitClientWithReasoning(t, fake, nil)
}
func newTestPromptKitClientWithReasoning(t *testing.T, fake *fakePromptKitLLM, reasoningEffort *string) *PromptKitClient {
t.Helper()
registry := newTestPromptKitAssets(t)
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: registry,
ReasoningEffort: reasoningEffort,
EngineOptions: []promptkit.Option{
promptkit.WithBackend(promptkit.Backend{
ID: "test-backend",
Endpoint: "http://127.0.0.1:1/v1",
}),
promptkit.WithProfiles(
promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
ID: "default-profile",
Endpoint: "http://127.0.0.1:1/v1",
Model: "default-model",
ReasoningEffort: "profile-reasoning",
}),
promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
ID: "explicit-profile",
Endpoint: "http://127.0.0.1:1/v1",
BackendID: "test-backend",
Model: "explicit-model",
ReasoningEffort: "profile-reasoning",
}),
),
promptkit.WithLLMClient(fake),
@@ -439,6 +690,22 @@ func newTestPromptKitAssets(t *testing.T) *AssetRegistry {
version: "v1"
default_profile: default-profile
session_id: "{{ .session_id }}"
inputs:
- name: transcript
required: true
content_type: application/json
messages:
- role: user
content: "Transcript: {{ input \"transcript\" }} Custom: {{ index . \"custom\" }} Session: {{ .session_id }}"
output:
format: json
validation_mode: json_schema
schema_path: adapter.schema.json
repair_attempts: 0
`)},
"adapter.direct-session.yaml": {Data: []byte(`id: adapter.direct-session
version: "v1"
default_profile: default-profile
inputs:
- name: transcript
required: true

View File

@@ -15,7 +15,7 @@ const (
promptKitProfileFingerprintName = "promptkit_profile_source"
// The built-in profile catalog is compiled into this pinned PromptKit
// release. Update this identity when the dependency is upgraded.
promptKitBuiltinProfileCatalogID = "promptkit:v0.1.0:builtin-profiles"
promptKitBuiltinProfileCatalogID = "promptkit:v0.2.0:builtin-profiles"
)
func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFingerprint, error) {

View File

@@ -829,18 +829,12 @@ func mergeLLMProfileManifests(sources ...[]artifacts.LLMProfileManifest) []artif
merged := make(map[string]artifacts.LLMProfileManifest)
for _, source := range sources {
for _, profile := range source {
id := strings.TrimSpace(profile.ID)
provider := strings.TrimSpace(profile.Provider)
model := strings.TrimSpace(profile.Model)
key := id + "\x00" + provider + "\x00" + model
profile = profile.Normalized()
key := profile.IdentityKey()
if _, exists := merged[key]; exists {
continue
}
merged[key] = artifacts.LLMProfileManifest{
ID: id,
Provider: provider,
Model: model,
}
merged[key] = profile
}
}
if len(merged) == 0 {

View File

@@ -0,0 +1,29 @@
package pipeline
import (
"reflect"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
)
func TestMergeLLMProfileManifestsDistinguishesEffectiveTargets(t *testing.T) {
got := mergeLLMProfileManifests(
[]artifacts.LLMProfileManifest{
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-b", ReasoningEffort: "low"},
{ID: " profile ", Provider: " promptkit ", Model: " model ", BackendID: " backend-a ", ReasoningEffort: " low "},
},
[]artifacts.LLMProfileManifest{
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "high"},
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "low"},
},
)
want := []artifacts.LLMProfileManifest{
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "high"},
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "low"},
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-b", ReasoningEffort: "low"},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("merged profiles = %#v, want %#v", got, want)
}
}

View File

@@ -449,6 +449,46 @@ func TestEncodePrettyPrintsJSON(t *testing.T) {
}
}
func TestEncodePublishesLLMProfileProvenance(t *testing.T) {
result, err := New().Encode(context.Background(), contracts.OutputRequest{
Manifest: artifacts.RunManifest{
RunID: "run-1",
LLMProfiles: []artifacts.LLMProfileManifest{
{
ID: "endpoint-profile",
Provider: "promptkit",
Model: "endpoint-model",
ReasoningEffort: "low",
},
{
ID: "profile",
Provider: "promptkit",
Model: "model",
BackendID: "openrouter",
ReasoningEffort: "high",
},
},
},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
manifest := decodeObject(t, fileBytes(t, result.Files, "manifest.json"))
profiles := manifest["llm_profiles"].([]any)
if len(profiles) != 2 {
t.Fatalf("llm_profiles = %#v, want two entries", profiles)
}
endpointProfile := profiles[0].(map[string]any)
if _, exists := endpointProfile["backend_id"]; exists || endpointProfile["reasoning_effort"] != "low" {
t.Fatalf("endpoint-only LLM profile = %#v, want omitted backend and published reasoning", endpointProfile)
}
backendProfile := profiles[1].(map[string]any)
if backendProfile["backend_id"] != "openrouter" || backendProfile["reasoning_effort"] != "high" {
t.Fatalf("backend LLM profile = %#v, want published backend and reasoning fields", backendProfile)
}
}
func TestEncodeIncludesManifestReferences(t *testing.T) {
result, err := New().Encode(context.Background(), contracts.OutputRequest{
Manifest: artifacts.RunManifest{