Compare commits
7 Commits
de046a8f13
...
a16dcdfa52
| Author | SHA1 | Date | |
|---|---|---|---|
| a16dcdfa52 | |||
| 46e4466d28 | |||
| 71a004bfc8 | |||
| f8333f2c15 | |||
| f603f7ac64 | |||
| 7a00e7049c | |||
| 2a9db9a957 |
13
docs/cli.md
13
docs/cli.md
@@ -41,14 +41,21 @@ pipeline ID and **--input** are required.
|
||||
| **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. |
|
||||
| **--llm-profile id** | Override effective LLM-capable module bindings with one configured profile. |
|
||||
| **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. |
|
||||
| **--reasoning-effort value** | Replace the selected PromptKit profile's reasoning effort for every LLM-backed call in this run. The value must be non-empty and the flag may be specified only once. |
|
||||
| **--clear-reasoning-effort** | Clear reasoning effort inherited from the selected PromptKit profile for every LLM-backed call in this run. |
|
||||
| **--reference selector=path** | Add or replace a file reference binding. Repeatable. |
|
||||
| **--without-reference selector** | Remove a configured optional reference binding. Repeatable. |
|
||||
|
||||
**--chunk_cache** accepts only **auto**, **bypass**, or **refresh**.
|
||||
**--debug-dir**, **--output-dir**, **--session-id**, and
|
||||
**--recompute-step** reject explicit empty values. **--recompute-step**
|
||||
requires **--resume**; checkpoint requirements and reuse behavior are
|
||||
documented in [Operations](operations.md).
|
||||
**--reasoning-effort**, and **--recompute-step** reject explicit empty values.
|
||||
**--reasoning-effort** and **--clear-reasoning-effort** are mutually exclusive.
|
||||
When neither is present, reasoning effort comes from the selected PromptKit
|
||||
profile. These controls apply to the shared run client, including retries and
|
||||
LLM-backed validators, and do not modify configuration or profile files.
|
||||
Persistent reasoning settings remain a PromptKit profile concern.
|
||||
**--recompute-step** requires **--resume**; checkpoint requirements and reuse
|
||||
behavior are documented in [Operations](operations.md).
|
||||
|
||||
### Reference selectors
|
||||
|
||||
|
||||
@@ -98,6 +98,21 @@ summarize results without embedding lane payload bytes. A chunk-plan summary is
|
||||
provenance for the plan used by this run; cache records, debug artifacts, and
|
||||
other operational state are not published as bundle files.
|
||||
|
||||
Each `llm_profiles` entry identifies effective, non-secret LLM execution
|
||||
provenance:
|
||||
|
||||
| Field | Required | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `id` | Yes | Selected PromptKit profile identifier. |
|
||||
| `provider` | No | Notarius adapter provider identifier. |
|
||||
| `model` | No | Effective provider model identifier. |
|
||||
| `backend_id` | No | Effective PromptKit backend registration identifier. Endpoint-only profiles omit it. |
|
||||
| `reasoning_effort` | No | Effective opaque provider reasoning setting. An empty or explicitly cleared setting is omitted. |
|
||||
|
||||
These values describe observed execution; they are not a backend-registration
|
||||
interface. Entries that differ by backend or effective reasoning remain
|
||||
distinct even when their profile, provider, and model are otherwise equal.
|
||||
|
||||
## Rejections And Warnings
|
||||
|
||||
`rejected.json` is always an object with a `rejected` array. Each entry has
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
# PromptKit Integration
|
||||
|
||||
Notarius pins
|
||||
[`gitea.maximumdirect.net/eric/promptkit` v0.1.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0)
|
||||
[`gitea.maximumdirect.net/eric/promptkit` v0.2.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0)
|
||||
as its in-process prompt engine. The upstream
|
||||
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/consumers/pkg-promptkit.md)
|
||||
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0/docs/consumers/pkg-promptkit.md)
|
||||
owns the public engine API, and the upstream
|
||||
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/formats.md)
|
||||
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0/docs/formats.md)
|
||||
owns prompt, profile, and schema file contracts.
|
||||
|
||||
## Supported Boundary
|
||||
@@ -15,11 +15,12 @@ Notarius relies on the root `promptkit` package to:
|
||||
- construct an `Engine` with filesystem-backed prompt, schema, and optional
|
||||
profile sources;
|
||||
- prepare and run a `RunRequest` with named inline artifacts, variables,
|
||||
metadata, prompt identity, and profile selection;
|
||||
a direct session ID, prompt identity, and profile selection;
|
||||
- return rendered debug material, validated structured output, selected
|
||||
profile and model metadata, and token usage;
|
||||
profile, backend, effective model metadata, and token usage;
|
||||
- distinguish structured-output validation failure from execution failure; and
|
||||
- identify a missing explicit profile through `ErrProfileNotFound`.
|
||||
- identify a missing explicit profile through `ErrProfileNotFound` and backend
|
||||
admission exhaustion through `ErrCapacityExceeded`.
|
||||
|
||||
Notarius does not use PromptKit's optional `ArtifactReader`. It materializes
|
||||
source and reference content itself and supplies owned inline artifacts at the
|
||||
@@ -27,6 +28,26 @@ adapter boundary. It also retains responsibility for pipeline retries,
|
||||
scheduling, debug persistence, redaction, profile provenance, and conversion
|
||||
from private model responses into durable domain artifacts.
|
||||
|
||||
Notarius sends its trimmed run session through PromptKit's direct session
|
||||
field, which is authoritative for provider session behavior. It also retains
|
||||
the same value as the `session_id` prompt variable for maintained prompt
|
||||
compatibility. Session IDs are stable, non-secret correlation identifiers and
|
||||
may be exposed to providers and provider observability.
|
||||
|
||||
Notarius records PromptKit's selected backend ID and effective reasoning
|
||||
setting as optional run-manifest provenance. Endpoint-only profiles have no
|
||||
backend ID. Debug prompt material also retains the selected backend ID and
|
||||
PromptKit's stable lower-case `effective_model_params` JSON, which may include
|
||||
`backend_id`. Notarius production configuration does not expose user-defined
|
||||
PromptKit backend registration.
|
||||
|
||||
Notarius retains its application-wide scheduled client around the PromptKit
|
||||
adapter. PromptKit may apply a narrower limit for the selected backend;
|
||||
endpoint-only profiles have no such backend limit. The adapter translates
|
||||
PromptKit capacity rejection into the provider-neutral Notarius
|
||||
`ErrLLMCapacityExceeded` contract and leaves retries to the calling pipeline
|
||||
stage.
|
||||
|
||||
## Notarius Ownership
|
||||
|
||||
[LLM Runtime Internals](../internal/llm.md) describes how Notarius mounts
|
||||
|
||||
@@ -61,6 +61,16 @@ construction errors are returned before a pipeline is prepared. Configuration
|
||||
field definitions remain in [Configuration](../config.md#promptkit-profiles);
|
||||
the adapter mechanics remain in [LLM Runtime](llm.md).
|
||||
|
||||
The factory also accepts `LLMRuntimeOverrides`, whose reasoning pointer
|
||||
preserves inherit, replace, and clear states across the composition boundary.
|
||||
Run orchestration constructs this value from the mutually exclusive
|
||||
`--reasoning-effort` and `--clear-reasoning-effort` controls. Absence preserves
|
||||
a nil pointer, replacement is trimmed, and clear uses a non-nil empty string.
|
||||
The same override reaches the one shared production client, checkpoint
|
||||
identity, and debug invocation metadata. Persistent reasoning configuration
|
||||
remains owned by PromptKit profiles; Notarius configuration has no reasoning
|
||||
field.
|
||||
|
||||
## Run Orchestration
|
||||
|
||||
After parsing and validating a run invocation, the CLI performs this ordered
|
||||
|
||||
@@ -26,20 +26,35 @@ durable schemas. Those responsibilities remain with the module and its
|
||||
|
||||
`PromptKitClient` validates the request target and prompt identity, maps each
|
||||
named material to a PromptKit inline artifact while preserving its origin URI,
|
||||
maps the request session to the existing `session_id` prompt variable, forwards
|
||||
profile selection, then prepares and runs the prompt. PromptKit v0.1.0 has no
|
||||
direct request-level session field. The adapter returns PromptKit’s validated
|
||||
raw bytes rather than re-encoding the decoded target. An empty optional
|
||||
material is represented as one space so its named input is retained by
|
||||
PromptKit.
|
||||
maps the trimmed request session to PromptKit's direct per-run session field,
|
||||
retains the same value as the `session_id` prompt variable for maintained
|
||||
prompt compatibility, forwards profile selection, then prepares and runs the
|
||||
prompt. The direct field is authoritative for provider session behavior. A
|
||||
session ID is a stable, non-secret correlation identifier and may be exposed
|
||||
to providers and provider observability. The adapter returns PromptKit’s
|
||||
validated raw bytes rather than re-encoding the decoded target. An empty
|
||||
optional material is represented as one space so its named input is retained
|
||||
by PromptKit.
|
||||
|
||||
Client construction may also receive a run-wide reasoning-effort override from
|
||||
the CLI factory boundary. The adapter copies the caller-owned pointer and
|
||||
creates a fresh PromptKit execution override for each request: a nil pointer
|
||||
inherits the selected profile, a non-empty value replaces it, and an empty
|
||||
value clears inherited reasoning. The CLI's mutually exclusive
|
||||
`--reasoning-effort` and `--clear-reasoning-effort` controls select those
|
||||
states. With neither flag, profile behavior remains unchanged. Because
|
||||
production constructs one shared client, the selected state applies uniformly
|
||||
to module calls, retries, and LLM-backed validators for the whole run.
|
||||
|
||||
An empty request profile lets the prompt select its configured default. The CLI
|
||||
prepares every explicitly selected binding profile before a run begins, so a
|
||||
missing explicit profile fails before stage execution. Calls record the profile
|
||||
actually selected by PromptKit; the recorder deduplicates non-secret profile
|
||||
identity, provider, and model values for manifest use. Successful completion
|
||||
responses and recorded profile manifests identify the adapter provider as
|
||||
`promptkit`.
|
||||
actually selected by PromptKit. The recorder trims and deduplicates non-secret
|
||||
profile identity, provider, model, selected backend ID, and effective reasoning
|
||||
values for manifest use. Entries that differ in backend or reasoning remain
|
||||
distinct and deterministically ordered. Endpoint-only profiles retain an empty
|
||||
backend ID, which the published JSON omits. Successful completion responses and
|
||||
recorded profile manifests identify the adapter provider as `promptkit`.
|
||||
|
||||
Before execution, the adapter also contributes a non-secret checkpoint
|
||||
fingerprint for the effective PromptKit profile source. It combines the
|
||||
@@ -66,6 +81,13 @@ worker counts cannot exceed the configured LLM limit. The configuration field
|
||||
and its effective default are owned by
|
||||
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||||
|
||||
PromptKit applies a second, independent admission limit when the selected
|
||||
profile names a limited backend. It sits beneath the Notarius scheduled client,
|
||||
so it may narrow but cannot expand the application-wide limit. Built-in
|
||||
OpenRouter profiles select PromptKit's reserved backend and its upstream
|
||||
capacity policy. Endpoint-only profiles do not select a PromptKit backend and
|
||||
remain limited only by the Notarius scheduler.
|
||||
|
||||
## Prompt And Schema Assets
|
||||
|
||||
An `AssetRegistry` collects prompt and schema filesystems from production module
|
||||
@@ -117,6 +139,14 @@ failure, empty structured body, or decode failure as
|
||||
material when they exist. Provider failures remain operational errors rather
|
||||
than output-validation failures.
|
||||
|
||||
When PromptKit rejects backend admission before generation, the adapter maps
|
||||
`promptkit.ErrCapacityExceeded` to
|
||||
`contracts.ErrLLMCapacityExceeded`, retaining prompt context and a redacted
|
||||
upstream diagnostic without exposing the PromptKit sentinel as a framework
|
||||
contract. A canceled caller context takes precedence. The adapter does not
|
||||
retry capacity failures; the pipeline's existing binding attempt policy sees
|
||||
the operational error and decides whether to rerun the complete operation.
|
||||
|
||||
Prompt-declared repair is executed within PromptKit’s structured-output flow.
|
||||
The current production D&D prompt manifests set repair attempts to zero. That
|
||||
setting does not replace pipeline retry behavior: a binding’s configured retry
|
||||
@@ -136,7 +166,7 @@ does not install another timeout wrapper around PromptKit.
|
||||
|
||||
The selected PromptKit profile owns generation settings. Notarius binding
|
||||
retries remain outside the adapter and repeat the complete module operation
|
||||
and validation chain. PromptKit v0.1.0 does not add a provider retry loop.
|
||||
and validation chain. PromptKit does not add a provider retry loop.
|
||||
Operator-facing behavior is summarized in
|
||||
[Operations](../operations.md#operational-limits), and the pinned upstream
|
||||
contract is identified in
|
||||
@@ -146,18 +176,22 @@ contract is identified in
|
||||
|
||||
When debug recording is enabled, the pipeline decorates the shared client. The
|
||||
wrapper records prepared prompt and response material, timing, selected profile
|
||||
and model, and call identifiers in the run’s debug bundle, including material
|
||||
available from a failed structured completion. For a successful completion, a
|
||||
debug-write failure is surfaced; when the completion already failed, its call
|
||||
error remains the result. Debug-bundle location, retention, and handling are
|
||||
operational concerns documented in [Operations](../operations.md#debug-bundles).
|
||||
and backend, effective model parameters, and call identifiers in the run’s
|
||||
debug bundle, including material available from a failed structured completion.
|
||||
Effective parameters use PromptKit's stable lower-case JSON field names and may
|
||||
include `backend_id`. For a successful completion, a debug-write failure is
|
||||
surfaced; when the completion already failed, its call error remains the
|
||||
result. Debug-bundle location, retention, and handling are operational concerns
|
||||
documented in [Operations](../operations.md#debug-bundles).
|
||||
|
||||
Run manifests receive selected profile summaries and component identities, not
|
||||
prompt, schema, source, reference, or response content. Provider error text is
|
||||
wrapped with prompt context and bearer credentials are redacted before it
|
||||
crosses the runtime boundary. Known-secret redaction is available to other
|
||||
runtime collaborators; it does not make prompt or response contents safe for
|
||||
general logging.
|
||||
Run manifests receive selected profile summaries, including optional effective
|
||||
backend and reasoning provenance, and component identities—not prompt, schema,
|
||||
source, reference, or response content. The published field semantics belong
|
||||
to the [JSON output contract](../integrations/json-output.md#manifestjson).
|
||||
Provider error text is wrapped with prompt context and bearer credentials are
|
||||
redacted before it crosses the runtime boundary. Known-secret redaction is
|
||||
available to other runtime collaborators; it does not make prompt or response
|
||||
contents safe for general logging.
|
||||
|
||||
## Failure Boundaries
|
||||
|
||||
@@ -165,6 +199,8 @@ general logging.
|
||||
sources, invalid asset registration, or a non-positive scheduler limit.
|
||||
- Preparation failures, unavailable explicit profiles, provider failures, and
|
||||
context cancellation propagate to the calling stage with context.
|
||||
- Backend admission exhaustion is a provider-neutral operational error and is
|
||||
not classified as invalid structured output or validator rejection.
|
||||
- Malformed or schema-invalid provider output is classified separately as
|
||||
invalid structured output so the module or pipeline can apply its own retry
|
||||
and rejection policy.
|
||||
|
||||
@@ -121,6 +121,9 @@ provenance, the effective PromptKit profile-source fingerprint, and
|
||||
prepared-component fingerprints. Changing profile content causes a cold miss
|
||||
even when its profile ID is unchanged. A changed identity produces a cold miss;
|
||||
Notarius does not migrate, rewrite, or delete older checkpoint directories.
|
||||
Reasoning-effort inheritance, replacement, and explicit clearing are distinct
|
||||
runtime identities, so checkpoints created under one state are not reused by
|
||||
either of the others.
|
||||
|
||||
Checkpoint state is confined below an identity-specific path:
|
||||
|
||||
@@ -180,7 +183,9 @@ warning, checkpoint, chunk-plan, and terminal reporting artifacts. The trace
|
||||
contains allowlisted application diagnostic records and can include source or
|
||||
derived application data. Neither surface is a cache input. Do not treat a
|
||||
debug bundle as safe to share merely because its configuration summary is
|
||||
redacted.
|
||||
redacted. Invocation metadata omits reasoning effort when it is inherited,
|
||||
records the replacement value when one is supplied, and records an empty value
|
||||
when inherited reasoning was explicitly cleared.
|
||||
|
||||
Notarius never creates debug state without an explicit request and never
|
||||
automatically deletes a requested bundle. If allocation succeeds, the command
|
||||
@@ -209,9 +214,12 @@ or automatic cleanup command.
|
||||
## Operational Limits
|
||||
|
||||
Provider execution settings and the generation timeout come from the selected
|
||||
PromptKit profile. PromptKit v0.1.0 does not add a provider retry loop;
|
||||
Notarius binding retries rerun the complete module operation and validation
|
||||
chain as defined by [module bindings](config.md#module-bindings-and-validators).
|
||||
PromptKit profile. The invocation-only **--reasoning-effort** and
|
||||
**--clear-reasoning-effort** controls may replace or clear that profile setting
|
||||
for all LLM-backed calls in one run without changing the profile. PromptKit
|
||||
v0.2.0 does not add a provider retry loop. Notarius binding retries rerun the
|
||||
complete module operation and validation chain as defined by
|
||||
[module bindings](config.md#module-bindings-and-validators).
|
||||
|
||||
Timeouts are layered. Caller cancellation is the outer authority. A positive
|
||||
effective generation timeout adds an inner request deadline, while zero
|
||||
@@ -220,7 +228,19 @@ transport-wide cap. Notarius does not add another timeout around PromptKit.
|
||||
The pinned upstream boundary and profile-format links are in
|
||||
[PromptKit Integration](integrations/pkg-promptkit.md).
|
||||
|
||||
Concurrency limits are configuration contracts; see
|
||||
Concurrency has two independent layers. Notarius **total_llm** is the
|
||||
application-wide provider-call limit shared by all backends, modules, retries,
|
||||
and validators. PromptKit may impose a narrower admission limit for the
|
||||
selected backend. The effective active-generation bound is the intersection of
|
||||
both limits and can therefore be lower than **total_llm**. Built-in OpenRouter
|
||||
profiles use PromptKit's upstream backend limit; endpoint-only profiles have no
|
||||
PromptKit backend limit and remain bounded by Notarius.
|
||||
|
||||
When a PromptKit backend has admitted all active and queued work, a new call
|
||||
fails as capacity exhaustion before generation. The adapter does not retry it.
|
||||
The calling stage's configured retry policy applies normally, and the run fails
|
||||
if those attempts are exhausted. Caller cancellation remains authoritative.
|
||||
Configuration contracts are documented under
|
||||
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
|
||||
limits and actual provider-call limits are independent. Notarius writes local
|
||||
filesystem state only; remote storage, archival, and retention automation are
|
||||
|
||||
@@ -177,7 +177,8 @@ individual modules.
|
||||
The application-wide LLM scheduler bounds actual provider calls independently
|
||||
of framework worker limits. Every LLM-backed module, retry, and validator uses
|
||||
the single injected scheduled client, including work performed by overlapping
|
||||
lanes.
|
||||
lanes. Provider runtime adapters may enforce a narrower backend-specific limit
|
||||
beneath this mandatory application-wide scheduler.
|
||||
|
||||
## Configuration And Provenance
|
||||
|
||||
|
||||
@@ -91,31 +91,6 @@ safety checks, and deterministic application of accepted changes.
|
||||
- Add media-type validators when non-JSON artifact representations are
|
||||
introduced.
|
||||
|
||||
## LLM Runtime Evolution
|
||||
|
||||
### Native Session Propagation
|
||||
|
||||
- Once upstream PromptKit exposes a direct request-level session identifier,
|
||||
propagate the existing `StructuredCompletionRequest.SessionID` through the
|
||||
PromptKit adapter's native session field.
|
||||
- Preserve the current `--session-id` invocation contract and its run-wide
|
||||
propagation to every prompt-facing module and validator. Do not introduce a
|
||||
second session configuration surface.
|
||||
- Retain session identity in checkpoint provenance so runs with different
|
||||
sessions cannot reuse one another's LLM-derived checkpoints.
|
||||
- Define the upstream compatibility and prompt-variable transition explicitly:
|
||||
native provider session behavior must not silently remove a `session_id`
|
||||
prompt variable while maintained prompts still consume it.
|
||||
- Add adapter and assembled-run coverage for exact forwarding, trimming,
|
||||
concurrent-run isolation, and unsupported-provider behavior once the
|
||||
upstream contract is available.
|
||||
|
||||
This work is blocked because PromptKit v0.1.0 does not expose the required
|
||||
direct request-level session field. Notarius already carries a run-scoped
|
||||
session ID through its CLI, pipeline requests, checkpoint identity, and a
|
||||
`session_id` prompt variable; that prompt-variable propagation is not native
|
||||
provider session support.
|
||||
|
||||
## Further Reference Evolution
|
||||
|
||||
- Make prior-run artifacts easier to bind as references without changing the
|
||||
|
||||
2
go.mod
2
go.mod
@@ -3,7 +3,7 @@ module gitea.maximumdirect.net/eric/notarius
|
||||
go 1.25.5
|
||||
|
||||
require (
|
||||
gitea.maximumdirect.net/eric/promptkit v0.1.0
|
||||
gitea.maximumdirect.net/eric/promptkit v0.2.0
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
|
||||
4
go.sum
4
go.sum
@@ -1,5 +1,5 @@
|
||||
gitea.maximumdirect.net/eric/promptkit v0.1.0 h1:vuKeBxkiY8E54LRFbLQFjlJJCiOfMvB1++DYBCrD/ug=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.1.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.2.0 h1:6bUATKnVgLlHSbKjGb8EnCt2r7jV5Xw6t+pX7G9S/lU=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.2.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
||||
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
||||
|
||||
@@ -143,7 +143,7 @@ func isEmptyRegistries(registries pipeline.Registries) bool {
|
||||
registries.Outputs == nil
|
||||
}
|
||||
|
||||
func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
@@ -151,16 +151,16 @@ func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileI
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
return buildProductionLLMClient(ctx, cfg, profileID, assets)
|
||||
return buildProductionLLMClient(ctx, cfg, profileID, overrides, assets)
|
||||
}
|
||||
|
||||
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
|
||||
return func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
return buildProductionLLMClient(ctx, cfg, profileID, assets)
|
||||
return func(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
return buildProductionLLMClient(ctx, cfg, profileID, overrides, assets)
|
||||
}
|
||||
}
|
||||
|
||||
func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
@@ -173,6 +173,7 @@ func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID
|
||||
ProfileFile: cfg.PromptKit.ProfileFile,
|
||||
Assets: assets,
|
||||
Recorder: recorder,
|
||||
ReasoningEffort: overrides.ReasoningEffort,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("create PromptKit-backed LLM client: %w", err)
|
||||
|
||||
@@ -51,7 +51,7 @@ pipelines:
|
||||
options := Options{
|
||||
Catalog: catalogFromRegistries(components.registries),
|
||||
Registries: components.registries,
|
||||
LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
llmConstructed = true
|
||||
return nil, nil, errors.New("LLM client must not be constructed")
|
||||
},
|
||||
|
||||
@@ -322,7 +322,7 @@ func TestProductionSpellNormalizerRejectsInvalidCatalogReferencesBeforeExecution
|
||||
options := Options{
|
||||
Catalog: catalogFromRegistries(components.registries),
|
||||
Registries: components.registries,
|
||||
LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
llmConstructed = true
|
||||
return nil, nil, errors.New("LLM client must not be constructed")
|
||||
},
|
||||
@@ -381,7 +381,7 @@ func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) {
|
||||
}
|
||||
for _, tt := range factories {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
client, manifests, err := tt.factory(context.Background(), config.Default(), "test-profile")
|
||||
client, manifests, err := tt.factory(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
|
||||
if err != nil {
|
||||
t.Fatalf("build production LLM runtime: %v", err)
|
||||
}
|
||||
@@ -410,14 +410,14 @@ func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
|
||||
t.Run("canceled context", func(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
client, manifests, err := productionLLMClientFactory(ctx, config.Default(), "test-profile")
|
||||
client, manifests, err := productionLLMClientFactory(ctx, config.Default(), "test-profile", LLMRuntimeOverrides{})
|
||||
if !errors.Is(err, context.Canceled) || client != nil || len(manifests) != 0 {
|
||||
t.Fatalf("client=%T manifests=%#v error=%v, want canceled construction", client, manifests, err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("nil assets", func(t *testing.T) {
|
||||
client, manifests, err := productionLLMClientFactoryWithAssets(nil)(context.Background(), config.Default(), "test-profile")
|
||||
client, manifests, err := productionLLMClientFactoryWithAssets(nil)(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
|
||||
if err == nil || !strings.Contains(err.Error(), "asset registry must not be nil") || client != nil || len(manifests) != 0 {
|
||||
t.Fatalf("client=%T manifests=%#v error=%v, want nil-assets failure", client, manifests, err)
|
||||
}
|
||||
@@ -427,7 +427,7 @@ func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
|
||||
components := productionTestComponents(t)
|
||||
cfg := config.Default()
|
||||
cfg.Concurrency.TotalLLM = 0
|
||||
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), cfg, "test-profile")
|
||||
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), cfg, "test-profile", LLMRuntimeOverrides{})
|
||||
if err == nil || !strings.Contains(err.Error(), "create LLM scheduler") || !strings.Contains(err.Error(), "greater than zero") || client != nil || len(manifests) != 0 {
|
||||
t.Fatalf("client=%T manifests=%#v error=%v, want scheduler-construction failure", client, manifests, err)
|
||||
}
|
||||
@@ -682,7 +682,7 @@ func productionRunOptions(t *testing.T, fake *productionFakeLLMClient) Options {
|
||||
options.Now = func() time.Time { return time.Unix(1700000000, 0).UTC() }
|
||||
options.RunIDGenerator = func(time.Time) (string, error) { return productionRunID, nil }
|
||||
options.UserCacheDir = func() (string, error) { return "", errors.New("user cache must not be used") }
|
||||
options.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
options.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
return fake, nil, nil
|
||||
}
|
||||
return options
|
||||
|
||||
@@ -30,7 +30,7 @@ import (
|
||||
const defaultConfigPath = "/usr/local/etc/notarius/config.yml"
|
||||
const usage = `Usage:
|
||||
notarius help
|
||||
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--output-dir path] [--json] [--chunk_cache auto|bypass|refresh] [--resume] [--recompute-step step-id] [--debug [--debug-dir path]] [--only lane-a,lane-b] [--session-id id] [--reference selector=path] [--without-reference selector]
|
||||
notarius run <pipeline-id> --input path/to/source.json [--json] [flags]
|
||||
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
|
||||
notarius pipelines list --config path/to/config.yml [--json]
|
||||
`
|
||||
@@ -48,7 +48,11 @@ type Options struct {
|
||||
DebugTerminalFactory func(*debugbundle.SummaryWriter) DebugTerminalWriter
|
||||
}
|
||||
|
||||
type LLMClientFactory func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error)
|
||||
type LLMRuntimeOverrides struct {
|
||||
ReasoningEffort *string
|
||||
}
|
||||
|
||||
type LLMClientFactory func(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error)
|
||||
|
||||
// Run executes the command-line interface and returns a process exit code.
|
||||
func Run(args []string, stdout, stderr io.Writer) int {
|
||||
@@ -138,13 +142,16 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
debug := fs.Bool("debug", false, "write a debug bundle")
|
||||
debugDir := fs.String("debug-dir", "", "debug bundle directory")
|
||||
llmProfile := fs.String("llm-profile", "", "LLM profile override")
|
||||
reasoningEffort := singleValueFlag{name: "--reasoning-effort"}
|
||||
clearReasoningEffort := fs.Bool("clear-reasoning-effort", false, "clear the LLM profile reasoning effort")
|
||||
resume := fs.Bool("resume", false, "reuse compatible recorded checkpoints")
|
||||
recomputeStep := singleValueFlag{}
|
||||
recomputeStep := singleValueFlag{name: "--recompute-step"}
|
||||
chunkCache := chunkCacheFlag{}
|
||||
sessionID := sessionIDFlag{}
|
||||
referenceFlags := stringListFlag{}
|
||||
withoutReferenceFlags := stringListFlag{}
|
||||
fs.Var(&sessionID, "session-id", "prompt session identifier")
|
||||
fs.Var(&reasoningEffort, "reasoning-effort", "reasoning effort override")
|
||||
fs.Var(&chunkCache, "chunk_cache", "chunk plan cache mode: auto, bypass, or refresh")
|
||||
fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, merge.slot=path, lane.slot=path, lane.extract.slot=path, lane.merge.slot=path, or lane.normalize.slot=path")
|
||||
fs.Var(&withoutReferenceFlags, "without-reference", "unbind a reference, using the same selector forms as --reference")
|
||||
@@ -190,6 +197,22 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
fmt.Fprintln(stderr, "notarius: --session-id must not be empty")
|
||||
return 2
|
||||
}
|
||||
if reasoningEffort.set && *clearReasoningEffort {
|
||||
fmt.Fprintln(stderr, "notarius: --reasoning-effort cannot be combined with --clear-reasoning-effort")
|
||||
return 2
|
||||
}
|
||||
if reasoningEffort.set && strings.TrimSpace(reasoningEffort.value) == "" {
|
||||
fmt.Fprintln(stderr, "notarius: --reasoning-effort must not be empty")
|
||||
return 2
|
||||
}
|
||||
runtimeOverrides := LLMRuntimeOverrides{}
|
||||
if reasoningEffort.set {
|
||||
value := strings.TrimSpace(reasoningEffort.value)
|
||||
runtimeOverrides.ReasoningEffort = &value
|
||||
} else if *clearReasoningEffort {
|
||||
value := ""
|
||||
runtimeOverrides.ReasoningEffort = &value
|
||||
}
|
||||
only, err := parseOnly(*onlyRaw)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||
@@ -287,6 +310,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
ConfigSource: configSource(*configPath),
|
||||
OnlyLanes: append([]string(nil), only...),
|
||||
ChunkCacheOverride: chunkCache.explicitValue(),
|
||||
ReasoningEffortOverride: runtimeOverrides.ReasoningEffort,
|
||||
Resume: *resume,
|
||||
RecomputeStep: strings.TrimSpace(recomputeStep.value),
|
||||
RunID: runID,
|
||||
@@ -364,7 +388,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
if len(profileIDs) == 1 {
|
||||
factoryProfileID = profileIDs[0]
|
||||
}
|
||||
llmClient, llmProfiles, err := opts.LLMClientFactory(ctx, effective.Config, factoryProfileID)
|
||||
llmClient, llmProfiles, err := opts.LLMClientFactory(ctx, effective.Config, factoryProfileID, runtimeOverrides)
|
||||
if err != nil {
|
||||
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("create LLM client for profile %q: %w", factoryProfileID, err))
|
||||
}
|
||||
@@ -388,7 +412,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
if err != nil {
|
||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||
}
|
||||
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(effective.Config.Cache.Checkpoints, opts, effective.ResolvedPipeline, prepared.CheckpointFingerprints(), llmFingerprints, rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), *resume)
|
||||
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(effective.Config.Cache.Checkpoints, opts, effective.ResolvedPipeline, prepared.CheckpointFingerprints(), llmFingerprints, rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), runtimeOverrides, *resume)
|
||||
if err != nil {
|
||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||
}
|
||||
@@ -495,6 +519,7 @@ func checkpointHandlersForRun(
|
||||
llmProfiles []artifacts.LLMProfileManifest,
|
||||
llmProfileOverride string,
|
||||
sessionID string,
|
||||
runtimeOverrides LLMRuntimeOverrides,
|
||||
resume bool,
|
||||
) (pipeline.CheckpointRecorder, pipeline.CheckpointLoader, error) {
|
||||
if !settings.Enabled {
|
||||
@@ -508,7 +533,7 @@ func checkpointHandlersForRun(
|
||||
InputKey: resolved.Input.Module,
|
||||
RawInputDigest: rawInputDigest(rawInput),
|
||||
SelectedLanes: only,
|
||||
RuntimeOverrides: runtimeOverrideFingerprints(llmProfileOverride, sessionID),
|
||||
RuntimeOverrides: runtimeOverrideFingerprints(llmProfileOverride, sessionID, runtimeOverrides),
|
||||
References: pipeline.ReferenceProvenance(resolved),
|
||||
ProvenanceFingerprints: combineCheckpointFingerprints(
|
||||
llmProfileFingerprints(llmProfiles),
|
||||
@@ -669,7 +694,7 @@ func rawInputDigest(data []byte) string {
|
||||
return "sha256:" + hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
func runtimeOverrideFingerprints(llmProfileOverride string, sessionID string) []checkpoint.Fingerprint {
|
||||
func runtimeOverrideFingerprints(llmProfileOverride string, sessionID string, runtimeOverrides LLMRuntimeOverrides) []checkpoint.Fingerprint {
|
||||
var values []checkpoint.Fingerprint
|
||||
if strings.TrimSpace(llmProfileOverride) != "" {
|
||||
values = append(values, checkpoint.Fingerprint{Name: "llm_profile_override", Value: strings.TrimSpace(llmProfileOverride)})
|
||||
@@ -677,6 +702,13 @@ func runtimeOverrideFingerprints(llmProfileOverride string, sessionID string) []
|
||||
if strings.TrimSpace(sessionID) != "" {
|
||||
values = append(values, checkpoint.Fingerprint{Name: "session_id", Value: strings.TrimSpace(sessionID)})
|
||||
}
|
||||
if runtimeOverrides.ReasoningEffort != nil {
|
||||
value := strings.TrimSpace(*runtimeOverrides.ReasoningEffort)
|
||||
if value == "" {
|
||||
value = "<cleared>"
|
||||
}
|
||||
values = append(values, checkpoint.Fingerprint{Name: "reasoning_effort_override", Value: value})
|
||||
}
|
||||
return values
|
||||
}
|
||||
|
||||
@@ -833,7 +865,7 @@ func reorderRunArgs(args []string) []string {
|
||||
|
||||
func runFlagTakesValue(arg string) bool {
|
||||
switch arg {
|
||||
case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--chunk_cache", "--reference", "--without-reference", "--recompute-step":
|
||||
case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--reasoning-effort", "--chunk_cache", "--reference", "--without-reference", "--recompute-step":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
@@ -893,11 +925,11 @@ func chunkPlanStoreForRun(cfg config.ChunkPlanCacheConfig, opts Options) (pipeli
|
||||
|
||||
func validateRunFlagValues(args []string) error {
|
||||
for i, arg := range args {
|
||||
if arg != "--session-id" {
|
||||
if arg != "--session-id" && arg != "--reasoning-effort" {
|
||||
continue
|
||||
}
|
||||
if i+1 >= len(args) || strings.HasPrefix(args[i+1], "-") {
|
||||
return fmt.Errorf("flag needs an argument: --session-id")
|
||||
return fmt.Errorf("flag needs an argument: %s", arg)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
@@ -1181,6 +1213,7 @@ type sessionIDFlag struct {
|
||||
}
|
||||
|
||||
type singleValueFlag struct {
|
||||
name string
|
||||
value string
|
||||
set bool
|
||||
}
|
||||
@@ -1194,7 +1227,7 @@ func (flag *singleValueFlag) String() string {
|
||||
|
||||
func (flag *singleValueFlag) Set(value string) error {
|
||||
if flag.set {
|
||||
return fmt.Errorf("--recompute-step may be specified only once")
|
||||
return fmt.Errorf("%s may be specified only once", flag.name)
|
||||
}
|
||||
flag.value = value
|
||||
flag.set = true
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/checkpoint"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
)
|
||||
@@ -245,8 +246,10 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
||||
harness := newStateTestHarness()
|
||||
var factoryProfiles []string
|
||||
opts := harness.options()
|
||||
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
var factoryOverrides []LLMRuntimeOverrides
|
||||
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
factoryProfiles = append(factoryProfiles, profileID)
|
||||
factoryOverrides = append(factoryOverrides, overrides)
|
||||
return nil, nil, nil
|
||||
}
|
||||
var stdout, stderr bytes.Buffer
|
||||
@@ -257,6 +260,9 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
||||
if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" {
|
||||
t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles)
|
||||
}
|
||||
if len(factoryOverrides) != 1 || factoryOverrides[0].ReasoningEffort != nil {
|
||||
t.Fatalf("factory overrides = %#v, want inherited reasoning", factoryOverrides)
|
||||
}
|
||||
harness.mu.Lock()
|
||||
profiles := append([]string(nil), harness.moduleProfiles...)
|
||||
harness.mu.Unlock()
|
||||
@@ -279,7 +285,7 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
||||
opts := harness.options()
|
||||
registerRunContractValidator(t, &opts, &validatorProfiles)
|
||||
factoryProfiles := []string{}
|
||||
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string, _ LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
factoryProfiles = append(factoryProfiles, profileID)
|
||||
return nil, nil, nil
|
||||
}
|
||||
@@ -302,7 +308,7 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
||||
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||
factoryCalls := 0
|
||||
opts := newStateTestHarness().options()
|
||||
opts.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
factoryCalls++
|
||||
return nil, nil, nil
|
||||
}
|
||||
@@ -314,6 +320,129 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
||||
})
|
||||
}
|
||||
|
||||
func TestRunReasoningEffortOverrideReachesFactory(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
flags []string
|
||||
wantValue string
|
||||
wantSet bool
|
||||
}{
|
||||
{name: "inherit"},
|
||||
{name: "replace", flags: []string{"--reasoning-effort", " focused "}, wantValue: "focused", wantSet: true},
|
||||
{name: "clear", flags: []string{"--clear-reasoning-effort"}, wantSet: true},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
roots := newStateTestRoots(t)
|
||||
opts := newStateTestHarness().options()
|
||||
var got []LLMRuntimeOverrides
|
||||
opts.LLMClientFactory = func(_ context.Context, _ config.Config, _ string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
got = append(got, overrides)
|
||||
return nil, nil, nil
|
||||
}
|
||||
args := append([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, tt.flags...)
|
||||
var stdout, stderr bytes.Buffer
|
||||
if code := RunWithOptions(args, &stdout, &stderr, opts); code != 0 || stderr.Len() != 0 {
|
||||
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||
}
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("factory overrides = %#v, want one call", got)
|
||||
}
|
||||
if !tt.wantSet {
|
||||
if got[0].ReasoningEffort != nil {
|
||||
t.Fatalf("reasoning effort = %q, want inherit", *got[0].ReasoningEffort)
|
||||
}
|
||||
return
|
||||
}
|
||||
if got[0].ReasoningEffort == nil || *got[0].ReasoningEffort != tt.wantValue {
|
||||
t.Fatalf("reasoning effort = %#v, want %q", got[0].ReasoningEffort, tt.wantValue)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunReasoningEffortOverrideRejectsInvalidSyntax(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
flags []string
|
||||
wantError string
|
||||
}{
|
||||
{
|
||||
name: "mutually exclusive controls",
|
||||
flags: []string{"--reasoning-effort", "focused", "--clear-reasoning-effort"},
|
||||
wantError: "cannot be combined",
|
||||
},
|
||||
{
|
||||
name: "empty replacement",
|
||||
flags: []string{"--reasoning-effort", " "},
|
||||
wantError: "must not be empty",
|
||||
},
|
||||
{
|
||||
name: "duplicate replacement",
|
||||
flags: []string{"--reasoning-effort", "low", "--reasoning-effort", "high"},
|
||||
wantError: "may be specified only once",
|
||||
},
|
||||
{
|
||||
name: "missing replacement",
|
||||
flags: []string{"--reasoning-effort"},
|
||||
wantError: "flag needs an argument",
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
roots := newStateTestRoots(t)
|
||||
args := append([]string{"run", "sample", "--config", roots.config, "--input", roots.input}, tt.flags...)
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := RunWithOptions(args, &stdout, &stderr, newStateTestHarness().options())
|
||||
if code != 2 || stdout.Len() != 0 || !strings.Contains(stderr.String(), tt.wantError) {
|
||||
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||
}
|
||||
assertNoRunState(t, roots)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestReasoningEffortOverrideSeparatesCheckpointIdentities(t *testing.T) {
|
||||
replacement := " focused "
|
||||
cleared := ""
|
||||
states := []struct {
|
||||
name string
|
||||
overrides LLMRuntimeOverrides
|
||||
wantValue string
|
||||
wantSet bool
|
||||
}{
|
||||
{name: "inherit"},
|
||||
{name: "replace", overrides: LLMRuntimeOverrides{ReasoningEffort: &replacement}, wantValue: "focused", wantSet: true},
|
||||
{name: "clear", overrides: LLMRuntimeOverrides{ReasoningEffort: &cleared}, wantValue: "<cleared>", wantSet: true},
|
||||
}
|
||||
digests := make(map[string]string, len(states))
|
||||
for _, state := range states {
|
||||
fingerprints := runtimeOverrideFingerprints("", "", state.overrides)
|
||||
var value string
|
||||
var found bool
|
||||
for _, fingerprint := range fingerprints {
|
||||
if fingerprint.Name == "reasoning_effort_override" {
|
||||
value, found = fingerprint.Value, true
|
||||
}
|
||||
}
|
||||
if found != state.wantSet || (found && value != state.wantValue) {
|
||||
t.Fatalf("%s fingerprint found=%t value=%q, want found=%t value=%q", state.name, found, value, state.wantSet, state.wantValue)
|
||||
}
|
||||
identity, err := checkpoint.NewIdentity(checkpoint.IdentityInput{
|
||||
Pipeline: pipeline.ResolvedPipeline{ID: "sample", Digest: "sha256:pipeline", Input: pipeline.Binding("test/input")},
|
||||
RawInputDigest: "sha256:input",
|
||||
RuntimeOverrides: fingerprints,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
digests[state.name] = identity.Digest
|
||||
}
|
||||
if digests["inherit"] == digests["replace"] || digests["inherit"] == digests["clear"] || digests["replace"] == digests["clear"] {
|
||||
t.Fatalf("checkpoint identity digests are not distinct: %#v", digests)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
|
||||
resolved := pipeline.ResolvedPipeline{
|
||||
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
|
||||
@@ -376,7 +505,7 @@ func TestRunFactoryAndPreparationFailuresAreProcessFailures(t *testing.T) {
|
||||
t.Run("LLM factory", func(t *testing.T) {
|
||||
roots := newStateTestRoots(t)
|
||||
opts := newStateTestHarness().options()
|
||||
opts.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
return nil, nil, errors.New("injected LLM factory failure")
|
||||
}
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
@@ -216,7 +216,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
||||
fingerprints := prepared.CheckpointFingerprints()
|
||||
llmFingerprints := []checkpoint.Fingerprint{{Name: "promptkit_profile_source", Value: "sha256:profile-source-one"}}
|
||||
settings := config.CheckpointCacheConfig{Enabled: true, Directory: t.TempDir()}
|
||||
recorder, _, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", false)
|
||||
recorder, _, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -242,7 +242,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, sameLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", true)
|
||||
_, sameLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -254,7 +254,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
||||
}
|
||||
changed := replaceCheckpointFingerprintValue(t, fingerprints, normalizeSpellCatalogFingerprintName(), "sha256:changed-effective-catalog")
|
||||
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changed, normalizeSpellCatalogFingerprintName())
|
||||
_, changedLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changed, llmFingerprints, []byte("same input"), nil, nil, "", "", true)
|
||||
_, changedLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changed, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -266,7 +266,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
||||
}
|
||||
changedMapping := replaceCheckpointFingerprintValue(t, fingerprints, extractSpellMappingFingerprintName(), "dnd.spells.extract_mapping.v3")
|
||||
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changedMapping, extractSpellMappingFingerprintName())
|
||||
_, mappingLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changedMapping, llmFingerprints, []byte("same input"), nil, nil, "", "", true)
|
||||
_, mappingLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changedMapping, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -275,7 +275,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
||||
}
|
||||
|
||||
changedLLMFingerprints := []checkpoint.Fingerprint{{Name: "promptkit_profile_source", Value: "sha256:profile-source-two"}}
|
||||
_, profileLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, changedLLMFingerprints, []byte("same input"), nil, nil, "", "", true)
|
||||
_, profileLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, changedLLMFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
@@ -848,7 +848,7 @@ func (h *stateTestHarness) options() Options {
|
||||
defer h.mu.Unlock()
|
||||
h.runIDCalls++
|
||||
return fmt.Sprintf("run-%d-%032x", startedAt.UnixNano(), h.runIDCalls), nil
|
||||
}, UserCacheDir: func() (string, error) { return "", errors.New("unexpected user cache lookup") }, LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
}, UserCacheDir: func() (string, error) { return "", errors.New("unexpected user cache lookup") }, LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
return nil, nil, nil
|
||||
}}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
@@ -29,6 +30,29 @@ type LLMProfileManifest struct {
|
||||
ID string `json:"id"`
|
||||
Provider string `json:"provider,omitempty"`
|
||||
Model string `json:"model,omitempty"`
|
||||
BackendID string `json:"backend_id,omitempty"`
|
||||
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||
}
|
||||
|
||||
// Normalized returns the canonical representation used for manifest identity
|
||||
// and publication.
|
||||
func (profile LLMProfileManifest) Normalized() LLMProfileManifest {
|
||||
profile.ID = strings.TrimSpace(profile.ID)
|
||||
profile.Provider = strings.TrimSpace(profile.Provider)
|
||||
profile.Model = strings.TrimSpace(profile.Model)
|
||||
profile.BackendID = strings.TrimSpace(profile.BackendID)
|
||||
profile.ReasoningEffort = strings.TrimSpace(profile.ReasoningEffort)
|
||||
return profile
|
||||
}
|
||||
|
||||
// IdentityKey returns an opaque, deterministic key for the effective profile.
|
||||
func (profile LLMProfileManifest) IdentityKey() string {
|
||||
profile = profile.Normalized()
|
||||
return profile.ID + "\x00" +
|
||||
profile.Provider + "\x00" +
|
||||
profile.Model + "\x00" +
|
||||
profile.BackendID + "\x00" +
|
||||
profile.ReasoningEffort
|
||||
}
|
||||
|
||||
type ReferenceProvenance struct {
|
||||
|
||||
@@ -53,7 +53,13 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
|
||||
PipelineID: "pipeline-1",
|
||||
PipelineDigest: "sha256:abc123",
|
||||
LLMProfiles: []LLMProfileManifest{
|
||||
{ID: "default", Provider: "promptkit", Model: "model-a"},
|
||||
{
|
||||
ID: "default",
|
||||
Provider: "promptkit",
|
||||
Model: "model-a",
|
||||
BackendID: "openrouter",
|
||||
ReasoningEffort: "high",
|
||||
},
|
||||
},
|
||||
ArtifactLanes: []ArtifactLaneManifest{
|
||||
{
|
||||
@@ -101,10 +107,13 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
|
||||
if !ok {
|
||||
t.Fatalf("llm_profiles[0] = %#v, want object", profiles[0])
|
||||
}
|
||||
assertHasKeys(t, profile, "id", "provider", "model")
|
||||
assertHasKeys(t, profile, "id", "provider", "model", "backend_id", "reasoning_effort")
|
||||
if profile["provider"] != "promptkit" {
|
||||
t.Fatalf("llm_profiles[0].provider = %#v, want promptkit", profile["provider"])
|
||||
}
|
||||
if profile["backend_id"] != "openrouter" || profile["reasoning_effort"] != "high" {
|
||||
t.Fatalf("llm_profiles[0] = %#v, want backend and reasoning provenance", profile)
|
||||
}
|
||||
|
||||
lanes, ok := got["artifact_lanes"].([]any)
|
||||
if !ok {
|
||||
|
||||
@@ -2,6 +2,7 @@ package debugbundle
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -137,6 +138,47 @@ func TestSummaryWriterWritesEverySummaryArtifact(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteInvocationPreservesReasoningEffortOverrideStates(t *testing.T) {
|
||||
replacement := "focused"
|
||||
cleared := ""
|
||||
tests := []struct {
|
||||
name string
|
||||
override *string
|
||||
wantValue string
|
||||
wantSet bool
|
||||
}{
|
||||
{name: "inherit"},
|
||||
{name: "replace", override: &replacement, wantValue: "focused", wantSet: true},
|
||||
{name: "clear", override: &cleared, wantSet: true},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
bundle, err := Allocate(t.TempDir(), testBundleRunID, time.Unix(0, 42))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := bundle.Summary().WriteInvocation(Invocation{
|
||||
Operation: "run",
|
||||
ReasoningEffortOverride: tt.override,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
data, err := os.ReadFile(filepath.Join(bundle.SummaryRoot(), ArtifactInvocationMetadata))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var payload map[string]any
|
||||
if err := json.Unmarshal(data, &payload); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
value, found := payload["reasoning_effort_override"]
|
||||
if found != tt.wantSet || (found && value != tt.wantValue) {
|
||||
t.Fatalf("reasoning override found=%t value=%#v, want found=%t value=%q; JSON=%s", found, value, tt.wantSet, tt.wantValue, data)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
func TestSummaryWriterInternalWritesConfineArtifacts(t *testing.T) {
|
||||
bundle, err := Allocate(t.TempDir(), testBundleRunID, time.Unix(0, 42))
|
||||
if err != nil {
|
||||
|
||||
@@ -39,6 +39,7 @@ type Invocation struct {
|
||||
ConfigSource string `json:"config_source,omitempty"`
|
||||
OnlyLanes []string `json:"only_lanes,omitempty"`
|
||||
ChunkCacheOverride string `json:"chunk_cache_override,omitempty"`
|
||||
ReasoningEffortOverride *string `json:"reasoning_effort_override,omitempty"`
|
||||
RunID string `json:"run_id"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
}
|
||||
@@ -68,6 +69,10 @@ func (w *SummaryWriter) WriteInvocation(payload Invocation) error {
|
||||
if payload.StartedAt.IsZero() {
|
||||
payload.StartedAt = w.createdAt
|
||||
}
|
||||
if payload.ReasoningEffortOverride != nil {
|
||||
value := *payload.ReasoningEffortOverride
|
||||
payload.ReasoningEffortOverride = &value
|
||||
}
|
||||
return w.writeJSON(ArtifactInvocationMetadata, payload)
|
||||
}
|
||||
func (w *SummaryWriter) WriteRedactedEffectiveConfig(payload RedactedSummaryPayload) error {
|
||||
|
||||
@@ -43,6 +43,7 @@ type LLMDebugPrompt struct {
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
PromptHash string `json:"prompt_hash,omitempty"`
|
||||
SelectedProfileID string `json:"selected_profile_id,omitempty"`
|
||||
SelectedBackendID string `json:"selected_backend_id,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
RenderedPromptHash string `json:"rendered_prompt_hash,omitempty"`
|
||||
Messages []LLMDebugMessage `json:"messages,omitempty"`
|
||||
|
||||
@@ -5,3 +5,7 @@ import "errors"
|
||||
// ErrInvalidStructuredOutput identifies a provider response that cannot satisfy
|
||||
// the caller's declared structured-output contract.
|
||||
var ErrInvalidStructuredOutput = errors.New("invalid structured output")
|
||||
|
||||
// ErrLLMCapacityExceeded identifies backend admission exhaustion before model
|
||||
// generation begins.
|
||||
var ErrLLMCapacityExceeded = errors.New("LLM capacity exceeded")
|
||||
|
||||
@@ -3,6 +3,7 @@ package llm
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"regexp"
|
||||
@@ -26,6 +27,7 @@ type PromptKitClientConfig struct {
|
||||
HTTPClient *http.Client
|
||||
EngineOptions []promptkit.Option
|
||||
Recorder *LLMProfileRecorder
|
||||
ReasoningEffort *string
|
||||
}
|
||||
|
||||
type PromptKitClient struct {
|
||||
@@ -33,6 +35,7 @@ type PromptKitClient struct {
|
||||
recorder *LLMProfileRecorder
|
||||
profileDir string
|
||||
profileFile string
|
||||
reasoningEffort *string
|
||||
}
|
||||
|
||||
type LLMProfileRecorder struct {
|
||||
@@ -71,11 +74,17 @@ func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
||||
if recorder == nil {
|
||||
recorder = NewLLMProfileRecorder()
|
||||
}
|
||||
var reasoningEffort *string
|
||||
if cfg.ReasoningEffort != nil {
|
||||
value := *cfg.ReasoningEffort
|
||||
reasoningEffort = &value
|
||||
}
|
||||
return &PromptKitClient{
|
||||
engine: engine,
|
||||
recorder: recorder,
|
||||
profileDir: strings.TrimSpace(cfg.ProfileDir),
|
||||
profileFile: strings.TrimSpace(cfg.ProfileFile),
|
||||
reasoningEffort: reasoningEffort,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -93,14 +102,23 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
|
||||
if promptID == "" {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion prompt_id must not be empty")
|
||||
}
|
||||
sessionID := strings.TrimSpace(req.SessionID)
|
||||
var execution *promptkit.ExecutionTargetOverride
|
||||
if c.reasoningEffort != nil {
|
||||
reasoningEffort := *c.reasoningEffort
|
||||
execution = &promptkit.ExecutionTargetOverride{
|
||||
ReasoningEffort: &reasoningEffort,
|
||||
}
|
||||
}
|
||||
|
||||
runReq := promptkit.RunRequest{
|
||||
PromptID: promptID,
|
||||
PromptVersion: strings.TrimSpace(req.PromptVersion),
|
||||
ProfileID: strings.TrimSpace(req.ProfileID),
|
||||
SessionID: sessionID,
|
||||
Inputs: promptKitInputs(req.Inputs),
|
||||
Vars: promptKitVars(req),
|
||||
Metadata: promptKitMetadata(req),
|
||||
Vars: promptKitVars(req, sessionID),
|
||||
Execution: execution,
|
||||
}
|
||||
prepared, err := c.engine.Prepare(ctx, runReq)
|
||||
if err != nil {
|
||||
@@ -114,6 +132,14 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
|
||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||
return contracts.StructuredCompletionResponse{}, ctxErr
|
||||
}
|
||||
if errors.Is(err, promptkit.ErrCapacityExceeded) {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf(
|
||||
"run PromptKit prompt %q: %w: %v",
|
||||
promptID,
|
||||
contracts.ErrLLMCapacityExceeded,
|
||||
redactPromptKitError(err),
|
||||
)
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w", promptID, redactPromptKitError(err))
|
||||
}
|
||||
if result == nil {
|
||||
@@ -141,6 +167,8 @@ func (c *PromptKitClient) responseFromResult(result *promptkit.RunResult, prepar
|
||||
ID: strings.TrimSpace(result.SelectedProfileID),
|
||||
Provider: promptKitProviderName,
|
||||
Model: firstNonEmpty(result.ModelName, result.EffectiveModelParams.Model),
|
||||
BackendID: strings.TrimSpace(result.SelectedBackendID),
|
||||
ReasoningEffort: strings.TrimSpace(result.EffectiveModelParams.ReasoningEffort),
|
||||
}
|
||||
if c.recorder != nil {
|
||||
c.recorder.Record(profile)
|
||||
@@ -188,6 +216,7 @@ func promptKitDebugPrompt(prepared *promptkit.PreparedRun) *contracts.LLMDebugPr
|
||||
PromptVersion: prepared.PromptVersion,
|
||||
PromptHash: prepared.PromptHash,
|
||||
SelectedProfileID: prepared.SelectedProfileID,
|
||||
SelectedBackendID: prepared.SelectedBackendID,
|
||||
SessionID: prepared.SessionID,
|
||||
RenderedPromptHash: prepared.RenderedPromptHash,
|
||||
Messages: messages,
|
||||
@@ -284,10 +313,8 @@ func (r *LLMProfileRecorder) Record(profile artifacts.LLMProfileManifest) {
|
||||
if r == nil {
|
||||
return
|
||||
}
|
||||
profile.ID = strings.TrimSpace(profile.ID)
|
||||
profile.Provider = strings.TrimSpace(profile.Provider)
|
||||
profile.Model = strings.TrimSpace(profile.Model)
|
||||
key := profile.ID + "\x00" + profile.Provider + "\x00" + profile.Model
|
||||
profile = profile.Normalized()
|
||||
key := profile.IdentityKey()
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if r.profiles == nil {
|
||||
@@ -343,7 +370,7 @@ func promptKitInputs(inputs contracts.LLMInputSet) map[string]promptkit.Artifact
|
||||
return out
|
||||
}
|
||||
|
||||
func promptKitVars(req contracts.StructuredCompletionRequest) map[string]string {
|
||||
func promptKitVars(req contracts.StructuredCompletionRequest, sessionID string) map[string]string {
|
||||
vars := make(map[string]string, len(req.Vars)+1)
|
||||
for key, value := range req.Vars {
|
||||
name := strings.TrimSpace(key)
|
||||
@@ -352,7 +379,7 @@ func promptKitVars(req contracts.StructuredCompletionRequest) map[string]string
|
||||
}
|
||||
vars[name] = fmt.Sprint(value)
|
||||
}
|
||||
if sessionID := strings.TrimSpace(req.SessionID); sessionID != "" {
|
||||
if sessionID != "" {
|
||||
vars["session_id"] = sessionID
|
||||
}
|
||||
if len(vars) == 0 {
|
||||
@@ -361,17 +388,6 @@ func promptKitVars(req contracts.StructuredCompletionRequest) map[string]string
|
||||
return vars
|
||||
}
|
||||
|
||||
func promptKitMetadata(req contracts.StructuredCompletionRequest) map[string]string {
|
||||
metadata := map[string]string{}
|
||||
if stageName := strings.TrimSpace(req.StageName); stageName != "" {
|
||||
metadata["stage_name"] = stageName
|
||||
}
|
||||
if len(metadata) == 0 {
|
||||
return nil
|
||||
}
|
||||
return metadata
|
||||
}
|
||||
|
||||
var bearerTokenPattern = regexp.MustCompile(`(?i)Bearer\s+[A-Za-z0-9._~+/=-]+`)
|
||||
|
||||
func redactPromptKitError(err error) error {
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
@@ -15,6 +16,7 @@ import (
|
||||
"testing/fstest"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
)
|
||||
@@ -28,10 +30,10 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
|
||||
}
|
||||
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
StageName: "test-stage",
|
||||
PromptID: "adapter.test",
|
||||
PromptID: "adapter.direct-session",
|
||||
PromptVersion: "v1",
|
||||
ProfileID: "explicit-profile",
|
||||
SessionID: "session-123",
|
||||
SessionID: " session-123 ",
|
||||
Inputs: contracts.LLMInputSet{
|
||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "sha256:source", "file:///source.json"),
|
||||
},
|
||||
@@ -52,8 +54,15 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
|
||||
if resp.Debug == nil || resp.Debug.Prompt == nil {
|
||||
t.Fatalf("debug prompt = nil, want prepared prompt material")
|
||||
}
|
||||
if resp.Debug.Prompt.PromptID != "adapter.test" || resp.Debug.Prompt.SelectedProfileID != "explicit-profile" {
|
||||
t.Fatalf("debug prompt metadata = %#v, want prompt/profile", resp.Debug.Prompt)
|
||||
if resp.Debug.Prompt.PromptID != "adapter.direct-session" ||
|
||||
resp.Debug.Prompt.SelectedProfileID != "explicit-profile" ||
|
||||
resp.Debug.Prompt.SelectedBackendID != "test-backend" ||
|
||||
resp.Debug.Prompt.SessionID != "session-123" {
|
||||
t.Fatalf("debug prompt metadata = %#v, want prompt/profile/backend/session", resp.Debug.Prompt)
|
||||
}
|
||||
if resp.Debug.Prompt.EffectiveModelParams["backend_id"] != "test-backend" ||
|
||||
resp.Debug.Prompt.EffectiveModelParams["reasoning_effort"] != "profile-reasoning" {
|
||||
t.Fatalf("debug effective model params = %#v, want backend and reasoning", resp.Debug.Prompt.EffectiveModelParams)
|
||||
}
|
||||
if len(resp.Debug.Prompt.Messages) != 1 || !strings.Contains(resp.Debug.Prompt.Messages[0].Content, `{"source":true}`) {
|
||||
t.Fatalf("debug prompt messages = %#v, want rendered input content", resp.Debug.Prompt.Messages)
|
||||
@@ -64,6 +73,10 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
|
||||
if resp.Debug.Response.Usage.CachedTokens != 5 || resp.Debug.Response.Usage.CacheWriteTokens != 3 {
|
||||
t.Fatalf("debug usage = %#v, want cached token counts", resp.Debug.Response.Usage)
|
||||
}
|
||||
if resp.Debug.Response.EffectiveModelParams["backend_id"] != "test-backend" ||
|
||||
resp.Debug.Response.EffectiveModelParams["reasoning_effort"] != "profile-reasoning" {
|
||||
t.Fatalf("debug response effective model params = %#v, want backend and reasoning", resp.Debug.Response.EffectiveModelParams)
|
||||
}
|
||||
debugJSON, err := json.Marshal(resp.Debug)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal debug material: %v", err)
|
||||
@@ -78,6 +91,9 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
|
||||
if gotReq.Target.Model != "explicit-model" {
|
||||
t.Fatalf("model = %q, want explicit-model", gotReq.Target.Model)
|
||||
}
|
||||
if gotReq.Target.BackendID != "test-backend" {
|
||||
t.Fatalf("backend id = %q, want test-backend", gotReq.Target.BackendID)
|
||||
}
|
||||
if len(gotReq.Prompt.Messages) != 1 ||
|
||||
!strings.Contains(gotReq.Prompt.Messages[0].Content, `{"source":true}`) ||
|
||||
!strings.Contains(gotReq.Prompt.Messages[0].Content, "value") {
|
||||
@@ -90,11 +106,124 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
|
||||
if len(manifests) != 1 ||
|
||||
manifests[0].ID != "explicit-profile" ||
|
||||
manifests[0].Provider != "promptkit" ||
|
||||
manifests[0].Model != "explicit-model" {
|
||||
manifests[0].Model != "explicit-model" ||
|
||||
manifests[0].BackendID != "test-backend" ||
|
||||
manifests[0].ReasoningEffort != "profile-reasoning" {
|
||||
t.Fatalf("profile manifests = %#v", manifests)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientRetainsSessionPromptVariable(t *testing.T) {
|
||||
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||
client := newTestPromptKitClient(t, fake)
|
||||
|
||||
var out map[string]any
|
||||
_, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
PromptID: "adapter.test",
|
||||
SessionID: " canonical-session ",
|
||||
Inputs: contracts.LLMInputSet{
|
||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||
},
|
||||
Vars: map[string]any{
|
||||
"custom": "value",
|
||||
"session_id": "caller-session",
|
||||
},
|
||||
}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
||||
}
|
||||
gotReq := fake.lastRequest()
|
||||
if gotReq.Prompt.SessionID != "canonical-session" {
|
||||
t.Fatalf("session id = %q, want canonical-session", gotReq.Prompt.SessionID)
|
||||
}
|
||||
if len(gotReq.Prompt.Messages) != 1 ||
|
||||
!strings.Contains(gotReq.Prompt.Messages[0].Content, "Session: canonical-session") ||
|
||||
strings.Contains(gotReq.Prompt.Messages[0].Content, "caller-session") {
|
||||
t.Fatalf("rendered messages = %#v, want canonical session compatibility variable", gotReq.Prompt.Messages)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientDoesNotInventDirectSession(t *testing.T) {
|
||||
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||
client := newTestPromptKitClient(t, fake)
|
||||
|
||||
var out map[string]any
|
||||
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
PromptID: "adapter.direct-session",
|
||||
Inputs: contracts.LLMInputSet{
|
||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||
},
|
||||
Vars: map[string]any{"custom": "value"},
|
||||
}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
||||
}
|
||||
if got := fake.lastRequest().Prompt.SessionID; got != "" {
|
||||
t.Fatalf("session id = %q, want empty", got)
|
||||
}
|
||||
if resp.Debug == nil || resp.Debug.Prompt == nil || resp.Debug.Prompt.SessionID != "" {
|
||||
t.Fatalf("debug prompt = %#v, want no effective session", resp.Debug)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientAppliesReasoningEffortOverride(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
override func() *string
|
||||
mutateAfterCreate bool
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "inherit",
|
||||
want: "profile-reasoning",
|
||||
},
|
||||
{
|
||||
name: "replace",
|
||||
override: func() *string { value := "focused"; return &value },
|
||||
want: "focused",
|
||||
},
|
||||
{
|
||||
name: "clear",
|
||||
override: func() *string { value := ""; return &value },
|
||||
want: "",
|
||||
},
|
||||
{
|
||||
name: "defensive copy",
|
||||
override: func() *string { value := "original"; return &value },
|
||||
mutateAfterCreate: true,
|
||||
want: "original",
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||
var override *string
|
||||
if tt.override != nil {
|
||||
override = tt.override()
|
||||
}
|
||||
client := newTestPromptKitClientWithReasoning(t, fake, override)
|
||||
if tt.mutateAfterCreate {
|
||||
*override = "mutated"
|
||||
}
|
||||
|
||||
var out map[string]any
|
||||
_, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
PromptID: "adapter.direct-session",
|
||||
Inputs: contracts.LLMInputSet{
|
||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||
},
|
||||
Vars: map[string]any{"custom": "value"},
|
||||
}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
||||
}
|
||||
if got := fake.lastRequest().Target.ReasoningEffort; got != tt.want {
|
||||
t.Fatalf("reasoning effort = %q, want %q", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewPromptKitClientReportsAssetAndEngineConstructionFailures(t *testing.T) {
|
||||
t.Run("assets", func(t *testing.T) {
|
||||
registry := NewAssetRegistry()
|
||||
@@ -176,6 +305,10 @@ func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
|
||||
if fresh[0].Value == "mutated" {
|
||||
t.Fatal("LLMCheckpointFingerprints exposed mutable backing storage")
|
||||
}
|
||||
const wantBuiltinFingerprint = "sha256:4b48cd1ef010b587cd8d56bc13b73a0cdd84da1414a8de9f9a2adc695dfcf0a7"
|
||||
if fresh[0].Value != wantBuiltinFingerprint {
|
||||
t.Fatalf("built-in profile fingerprint = %q, want %q", fresh[0].Value, wantBuiltinFingerprint)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) {
|
||||
@@ -195,6 +328,30 @@ func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testi
|
||||
if got := fake.lastRequest().Target.Model; got != "default-model" {
|
||||
t.Fatalf("model = %q, want prompt default profile model", got)
|
||||
}
|
||||
manifests := client.LLMProfileManifests()
|
||||
if len(manifests) != 1 || manifests[0].BackendID != "" || manifests[0].ReasoningEffort != "profile-reasoning" {
|
||||
t.Fatalf("endpoint-only profile manifests = %#v, want omitted backend and effective reasoning", manifests)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLLMProfileRecorderDistinguishesEffectiveTargets(t *testing.T) {
|
||||
recorder := NewLLMProfileRecorder()
|
||||
for _, profile := range []artifacts.LLMProfileManifest{
|
||||
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-b", ReasoningEffort: "low"},
|
||||
{ID: " profile ", Provider: " promptkit ", Model: " model ", BackendID: " backend-a ", ReasoningEffort: " low "},
|
||||
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "high"},
|
||||
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "low"},
|
||||
} {
|
||||
recorder.Record(profile)
|
||||
}
|
||||
want := []artifacts.LLMProfileManifest{
|
||||
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "high"},
|
||||
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "low"},
|
||||
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-b", ReasoningEffort: "low"},
|
||||
}
|
||||
if got := recorder.Manifests(); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("profile manifests = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientValidationFailureReturnsError(t *testing.T) {
|
||||
@@ -272,6 +429,89 @@ func TestPromptKitClientProviderFailureIncludesContextAndRedactsBearerToken(t *t
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientTranslatesBackendCapacityExhaustion(t *testing.T) {
|
||||
queueCapacity := 0
|
||||
fake := &fakePromptKitLLM{
|
||||
err: errors.New("provider failed with Bearer secret-token"),
|
||||
block: make(chan struct{}),
|
||||
}
|
||||
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||
Assets: newTestPromptKitAssets(t),
|
||||
EngineOptions: []promptkit.Option{
|
||||
promptkit.WithBackend(promptkit.Backend{
|
||||
ID: "limited-backend",
|
||||
Endpoint: "http://127.0.0.1:1/v1",
|
||||
ConcurrencyLimit: 1,
|
||||
QueueCapacity: &queueCapacity,
|
||||
}),
|
||||
promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||
ID: "limited-profile",
|
||||
BackendID: "limited-backend",
|
||||
Model: "limited-model",
|
||||
})),
|
||||
promptkit.WithLLMClient(fake),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
|
||||
}
|
||||
defer func() {
|
||||
select {
|
||||
case <-fake.block:
|
||||
default:
|
||||
close(fake.block)
|
||||
}
|
||||
}()
|
||||
|
||||
request := contracts.StructuredCompletionRequest{
|
||||
PromptID: "adapter.direct-session",
|
||||
ProfileID: "limited-profile",
|
||||
Inputs: contracts.LLMInputSet{
|
||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||
},
|
||||
Vars: map[string]any{"custom": "value"},
|
||||
}
|
||||
firstResult := make(chan error, 1)
|
||||
go func() {
|
||||
var out map[string]any
|
||||
_, callErr := client.CompleteStructured(context.Background(), request, &out)
|
||||
firstResult <- callErr
|
||||
}()
|
||||
waitForAtomicAtLeast(t, &fake.calls, 1)
|
||||
|
||||
var out map[string]any
|
||||
response, capacityErr := client.CompleteStructured(context.Background(), request, &out)
|
||||
if len(response.Content) != 0 {
|
||||
t.Fatalf("capacity response = %#v, want empty", response)
|
||||
}
|
||||
if !errors.Is(capacityErr, contracts.ErrLLMCapacityExceeded) {
|
||||
t.Fatalf("capacity error = %v, want ErrLLMCapacityExceeded", capacityErr)
|
||||
}
|
||||
if errors.Is(capacityErr, contracts.ErrInvalidStructuredOutput) {
|
||||
t.Fatalf("capacity error = %v, must not be invalid structured output", capacityErr)
|
||||
}
|
||||
if errors.Is(capacityErr, promptkit.ErrCapacityExceeded) {
|
||||
t.Fatalf("capacity error exposes PromptKit sentinel: %v", capacityErr)
|
||||
}
|
||||
if !strings.Contains(capacityErr.Error(), `run PromptKit prompt "adapter.direct-session"`) ||
|
||||
!strings.Contains(capacityErr.Error(), "backend capacity exceeded") {
|
||||
t.Fatalf("capacity error = %q, want prompt context and upstream diagnostic", capacityErr)
|
||||
}
|
||||
if calls := atomic.LoadInt32(&fake.calls); calls != 1 {
|
||||
t.Fatalf("provider calls after capacity rejection = %d, want 1", calls)
|
||||
}
|
||||
|
||||
close(fake.block)
|
||||
firstErr := <-firstResult
|
||||
if firstErr == nil || strings.Contains(firstErr.Error(), "secret-token") ||
|
||||
!strings.Contains(firstErr.Error(), "Bearer [REDACTED]") {
|
||||
t.Fatalf("admitted provider error = %v, want redacted diagnostic", firstErr)
|
||||
}
|
||||
if calls := atomic.LoadInt32(&fake.calls); calls != 1 {
|
||||
t.Fatalf("provider calls after release = %d, want no adapter retry", calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientContextCancellationIsRespected(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
@@ -405,21 +645,32 @@ func TestPromptKitClientValidatesRequest(t *testing.T) {
|
||||
}
|
||||
|
||||
func newTestPromptKitClient(t *testing.T, fake *fakePromptKitLLM) *PromptKitClient {
|
||||
return newTestPromptKitClientWithReasoning(t, fake, nil)
|
||||
}
|
||||
|
||||
func newTestPromptKitClientWithReasoning(t *testing.T, fake *fakePromptKitLLM, reasoningEffort *string) *PromptKitClient {
|
||||
t.Helper()
|
||||
registry := newTestPromptKitAssets(t)
|
||||
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||
Assets: registry,
|
||||
ReasoningEffort: reasoningEffort,
|
||||
EngineOptions: []promptkit.Option{
|
||||
promptkit.WithBackend(promptkit.Backend{
|
||||
ID: "test-backend",
|
||||
Endpoint: "http://127.0.0.1:1/v1",
|
||||
}),
|
||||
promptkit.WithProfiles(
|
||||
promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||
ID: "default-profile",
|
||||
Endpoint: "http://127.0.0.1:1/v1",
|
||||
Model: "default-model",
|
||||
ReasoningEffort: "profile-reasoning",
|
||||
}),
|
||||
promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||
ID: "explicit-profile",
|
||||
Endpoint: "http://127.0.0.1:1/v1",
|
||||
BackendID: "test-backend",
|
||||
Model: "explicit-model",
|
||||
ReasoningEffort: "profile-reasoning",
|
||||
}),
|
||||
),
|
||||
promptkit.WithLLMClient(fake),
|
||||
@@ -439,6 +690,22 @@ func newTestPromptKitAssets(t *testing.T) *AssetRegistry {
|
||||
version: "v1"
|
||||
default_profile: default-profile
|
||||
session_id: "{{ .session_id }}"
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: application/json
|
||||
messages:
|
||||
- role: user
|
||||
content: "Transcript: {{ input \"transcript\" }} Custom: {{ index . \"custom\" }} Session: {{ .session_id }}"
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: adapter.schema.json
|
||||
repair_attempts: 0
|
||||
`)},
|
||||
"adapter.direct-session.yaml": {Data: []byte(`id: adapter.direct-session
|
||||
version: "v1"
|
||||
default_profile: default-profile
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
|
||||
@@ -15,7 +15,7 @@ const (
|
||||
promptKitProfileFingerprintName = "promptkit_profile_source"
|
||||
// The built-in profile catalog is compiled into this pinned PromptKit
|
||||
// release. Update this identity when the dependency is upgraded.
|
||||
promptKitBuiltinProfileCatalogID = "promptkit:v0.1.0:builtin-profiles"
|
||||
promptKitBuiltinProfileCatalogID = "promptkit:v0.2.0:builtin-profiles"
|
||||
)
|
||||
|
||||
func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFingerprint, error) {
|
||||
|
||||
@@ -829,18 +829,12 @@ func mergeLLMProfileManifests(sources ...[]artifacts.LLMProfileManifest) []artif
|
||||
merged := make(map[string]artifacts.LLMProfileManifest)
|
||||
for _, source := range sources {
|
||||
for _, profile := range source {
|
||||
id := strings.TrimSpace(profile.ID)
|
||||
provider := strings.TrimSpace(profile.Provider)
|
||||
model := strings.TrimSpace(profile.Model)
|
||||
key := id + "\x00" + provider + "\x00" + model
|
||||
profile = profile.Normalized()
|
||||
key := profile.IdentityKey()
|
||||
if _, exists := merged[key]; exists {
|
||||
continue
|
||||
}
|
||||
merged[key] = artifacts.LLMProfileManifest{
|
||||
ID: id,
|
||||
Provider: provider,
|
||||
Model: model,
|
||||
}
|
||||
merged[key] = profile
|
||||
}
|
||||
}
|
||||
if len(merged) == 0 {
|
||||
|
||||
29
internal/framework/pipeline/runner_manifest_test.go
Normal file
29
internal/framework/pipeline/runner_manifest_test.go
Normal file
@@ -0,0 +1,29 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||
)
|
||||
|
||||
func TestMergeLLMProfileManifestsDistinguishesEffectiveTargets(t *testing.T) {
|
||||
got := mergeLLMProfileManifests(
|
||||
[]artifacts.LLMProfileManifest{
|
||||
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-b", ReasoningEffort: "low"},
|
||||
{ID: " profile ", Provider: " promptkit ", Model: " model ", BackendID: " backend-a ", ReasoningEffort: " low "},
|
||||
},
|
||||
[]artifacts.LLMProfileManifest{
|
||||
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "high"},
|
||||
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "low"},
|
||||
},
|
||||
)
|
||||
want := []artifacts.LLMProfileManifest{
|
||||
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "high"},
|
||||
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "low"},
|
||||
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-b", ReasoningEffort: "low"},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("merged profiles = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
@@ -449,6 +449,46 @@ func TestEncodePrettyPrintsJSON(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestEncodePublishesLLMProfileProvenance(t *testing.T) {
|
||||
result, err := New().Encode(context.Background(), contracts.OutputRequest{
|
||||
Manifest: artifacts.RunManifest{
|
||||
RunID: "run-1",
|
||||
LLMProfiles: []artifacts.LLMProfileManifest{
|
||||
{
|
||||
ID: "endpoint-profile",
|
||||
Provider: "promptkit",
|
||||
Model: "endpoint-model",
|
||||
ReasoningEffort: "low",
|
||||
},
|
||||
{
|
||||
ID: "profile",
|
||||
Provider: "promptkit",
|
||||
Model: "model",
|
||||
BackendID: "openrouter",
|
||||
ReasoningEffort: "high",
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Encode() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
manifest := decodeObject(t, fileBytes(t, result.Files, "manifest.json"))
|
||||
profiles := manifest["llm_profiles"].([]any)
|
||||
if len(profiles) != 2 {
|
||||
t.Fatalf("llm_profiles = %#v, want two entries", profiles)
|
||||
}
|
||||
endpointProfile := profiles[0].(map[string]any)
|
||||
if _, exists := endpointProfile["backend_id"]; exists || endpointProfile["reasoning_effort"] != "low" {
|
||||
t.Fatalf("endpoint-only LLM profile = %#v, want omitted backend and published reasoning", endpointProfile)
|
||||
}
|
||||
backendProfile := profiles[1].(map[string]any)
|
||||
if backendProfile["backend_id"] != "openrouter" || backendProfile["reasoning_effort"] != "high" {
|
||||
t.Fatalf("backend LLM profile = %#v, want published backend and reasoning fields", backendProfile)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEncodeIncludesManifestReferences(t *testing.T) {
|
||||
result, err := New().Encode(context.Background(), contracts.OutputRequest{
|
||||
Manifest: artifacts.RunManifest{
|
||||
|
||||
Reference in New Issue
Block a user