28 Commits

Author SHA1 Message Date
39388e96d4 Make PromptKit profile handling safer and more consistent 2026-08-03 18:35:40 +00:00
12ac25bd63 Sanitize PromptKit profile fingerprint errors 2026-08-03 17:25:44 +00:00
394278e1f2 Document workload-oriented LLM profile deployment 2026-08-03 17:21:42 +00:00
5cd7f8e737 Expose pipeline LLM profile defaults 2026-08-03 17:15:07 +00:00
bf3fadf9ae Resolve pipeline LLM profile defaults 2026-08-03 17:09:16 +00:00
58815aaf33 Require explicit module execution classes 2026-08-03 17:00:08 +00:00
ce857966f1 Add execution metadata to module specifications 2026-08-03 16:49:51 +00:00
a3bd0c1867 Add D&D extraction fallback profile 2026-08-03 16:39:45 +00:00
b05634ee86 Add fallback PromptKit profile assets 2026-08-03 16:34:48 +00:00
4829f94157 Inspect PromptKit profiles during preflight 2026-08-03 16:26:15 +00:00
67b315099d Execute PromptKit requests from prepared snapshots 2026-08-03 16:17:40 +00:00
b5c86de4d7 Upgrade PromptKit to version 0.5.0 2026-08-03 16:12:29 +00:00
2eeca2ed5a Plan the PromptKit upgrade and profile workflow 2026-08-03 16:07:37 +00:00
b5aaeb1c78 Update future roadmap document with ideas for new feature developments 2026-07-30 16:27:36 +00:00
9171b66a41 Clarify PromptKit configuration and retire the completed plan 2026-07-30 15:38:31 +00:00
b4363b3b73 Document local PromptKit backend support 2026-07-30 05:22:41 +00:00
241e9d2a89 Include local backend target in checkpoint identity 2026-07-30 05:17:39 +00:00
715fff7b72 Register configured local PromptKit backend 2026-07-30 05:14:22 +00:00
d627b91b4f Add local PromptKit backend configuration 2026-07-30 05:09:54 +00:00
a67b3aa76d Upgrade PromptKit to version 0.3.0 2026-07-30 05:06:32 +00:00
a16dcdfa52 Clean up the PromptKit upgrade 2026-07-30 02:58:25 +00:00
46e4466d28 Translate LLM backend capacity failures 2026-07-30 02:23:51 +00:00
71a004bfc8 Publish effective LLM backend provenance 2026-07-30 02:18:14 +00:00
f8333f2c15 Expose run-wide reasoning effort controls 2026-07-30 02:11:35 +00:00
f603f7ac64 Support internal reasoning effort overrides 2026-07-30 02:03:04 +00:00
7a00e7049c Forward sessions through PromptKit requests 2026-07-30 01:58:40 +00:00
2a9db9a957 Upgrade PromptKit to version 0.2.0 2026-07-30 01:54:47 +00:00
de046a8f13 Keep checkpoints aligned with PromptKit profiles 2026-07-28 13:36:01 -05:00
124 changed files with 5153 additions and 1064 deletions

View File

@@ -0,0 +1,49 @@
# ADR-0010: Use workload-oriented LLM profile defaults
**Status:** Accepted
**Date:** 2026-08-03
## Context
LLM-backed D&D operations share an execution-policy choice, but repeating a
provider or model-named profile on every module binding ties pipeline structure
to a deployment decision. Different environments may require different model,
backend, timeout, or reasoning settings while retaining the same workload.
Notarius also needs a usable default for maintained D&D prompts without making
an operator profile mandatory. That default must remain owned by the D&D
family, while generic LLM infrastructure stays unaware of domain-specific
policy.
## Decision
Pipelines may name one workload-oriented default profile, inherited only by
selected LLM-backed bindings and validators. Binding-level profile IDs remain
intentional exceptions, and the run-wide CLI profile override has highest
precedence.
The D&D family owns an embedded fallback profile named `dnd-extraction`.
Operators may provide a complete profile with the same ID through a PromptKit
filesystem source. PromptKit selects the higher-precedence matching definition;
Notarius does not merge profile documents. Production, development, and local
deployments can therefore use different execution policy behind one unchanged
pipeline ID.
## Alternatives considered
- Repeat a model-named profile on every binding. This makes routine deployment
policy changes noisy and obscures the shared workload intent.
- Require every deployment to install a profile file. This adds configuration
friction and leaves maintained D&D prompts without an application-owned
fallback.
- Put D&D profile policy in generic LLM infrastructure. This breaks domain
ownership and makes generic code depend on one workload.
## Consequences
Pipeline configuration expresses workload intent rather than a specific
provider or model. Operators can replace the complete execution policy without
editing bindings, while binding-level and run-wide exceptions remain available.
Profile changes affect resolved pipeline and checkpoint identity, so they may
intentionally cause work to be recomputed. The D&D fallback becomes a
maintained application execution-policy asset.

View File

@@ -39,16 +39,23 @@ pipeline ID and **--input** are required.
| **--debug** | Retain a debug bundle for this run. | | **--debug** | Retain a debug bundle for this run. |
| **--debug-dir path** | Override the debug-bundle root. Requires **--debug**. | | **--debug-dir path** | Override the debug-bundle root. Requires **--debug**. |
| **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. | | **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. |
| **--llm-profile id** | Override effective LLM-capable module bindings with one configured profile. | | **--llm-profile id** | Highest-precedence configured profile for selected LLM-backed bindings and validators; it replaces binding and [pipeline](config.md#pipelines) defaults. |
| **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. | | **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. |
| **--reasoning-effort value** | Replace the selected PromptKit profile's reasoning effort for every LLM-backed call in this run. The value must be non-empty and the flag may be specified only once. |
| **--clear-reasoning-effort** | Clear reasoning effort inherited from the selected PromptKit profile for every LLM-backed call in this run. |
| **--reference selector=path** | Add or replace a file reference binding. Repeatable. | | **--reference selector=path** | Add or replace a file reference binding. Repeatable. |
| **--without-reference selector** | Remove a configured optional reference binding. Repeatable. | | **--without-reference selector** | Remove a configured optional reference binding. Repeatable. |
**--chunk_cache** accepts only **auto**, **bypass**, or **refresh**. **--chunk_cache** accepts only **auto**, **bypass**, or **refresh**.
**--debug-dir**, **--output-dir**, **--session-id**, and **--debug-dir**, **--output-dir**, **--session-id**, and
**--recompute-step** reject explicit empty values. **--recompute-step** **--reasoning-effort**, and **--recompute-step** reject explicit empty values.
requires **--resume**; checkpoint requirements and reuse behavior are **--reasoning-effort** and **--clear-reasoning-effort** are mutually exclusive.
documented in [Operations](operations.md). When neither is present, reasoning effort comes from the selected PromptKit
profile. These controls apply to the shared run client, including retries and
LLM-backed validators, and do not modify configuration or profile files.
Persistent reasoning settings remain a PromptKit profile concern.
**--recompute-step** requires **--resume**; checkpoint requirements and reuse
behavior are documented in [Operations](operations.md).
### Reference selectors ### Reference selectors

View File

@@ -32,7 +32,8 @@ override the fields listed below.
single-lane Seriatim-to-spell pipeline. single-lane Seriatim-to-spell pipeline.
- [Complete D&D configuration](../examples/dnd-complete.config.yml) uses - [Complete D&D configuration](../examples/dnd-complete.config.yml) uses
ordered steps, all implemented D&D lanes, generated references, state ordered steps, all implemented D&D lanes, generated references, state
settings, and bounded LLM concurrency. settings, bounded LLM concurrency, and the maintained
[operator profile](../examples/profiles/dnd-extraction.yml).
Use these complete files as starting points rather than combining the Use these complete files as starting points rather than combining the
illustrative fragments in this reference. illustrative fragments in this reference.
@@ -46,7 +47,7 @@ other than **version** is optional.
| Field | Type | Default | Rules | | Field | Type | Default | Rules |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| **version** | integer | none | Required; must be 4. | | **version** | integer | none | Required; must be 4. |
| **promptkit** | object | none | Profile source configuration. | | **promptkit** | object | none | Profile source and optional local-backend configuration. |
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. | | **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
| **concurrency** | object | see below | Global LLM and extraction limits. | | **concurrency** | object | see below | Global LLM and extraction limits. |
| **output** | object | see below | Published output settings. | | **output** | object | see below | Published output settings. |
@@ -71,27 +72,76 @@ per-user root. An explicit empty output or debug directory is invalid.
## PromptKit Profiles ## PromptKit Profiles
The optional **promptkit** object selects one source of profile definitions: The optional **promptkit** object selects one source of profile definitions and
may register one conventional local OpenAI-compatible backend:
~~~yaml ~~~yaml
version: 4 version: 4
promptkit: promptkit:
profile_dir: /path/to/profiles profile_dir: ./profiles
# profile_file: /path/to/profiles.yml # profile_file: ./profiles.yml
local_backend:
endpoint: http://localhost:8000/v1
concurrency_limit: 2
~~~ ~~~
| Field | Type | Rules | | Field | Type | Rules |
| --- | --- | --- | | --- | --- | --- |
| **profile_dir** | string | Non-empty directory containing profile files. | | **profile_dir** | string | Non-empty directory containing profile files. |
| **profile_file** | string | Non-empty profile file. | | **profile_file** | string | Non-empty profile file. |
| **local_backend** | object | Optional registration for the conventional PromptKit backend ID **local**. |
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
Set at most one of these fields. Profile IDs used by a binding must be available Set at most one of **profile_dir** and **profile_file**. Relative values use
from the selected PromptKit profile source when the pipeline is resolved. the process working directory, not the configuration file's directory. The
Keep credentials out of this file: configure a profile to read its credential complete example's `./examples/profiles/dnd-extraction.yml` value is therefore
from an environment variable, then set that environment variable only in the valid when Notarius is launched from the repository root; use an absolute path
run environment. PromptKit owns the profile-file format; see the for services and containers.
[PromptKit upstream boundary](integrations/pkg-promptkit.md) for the pinned
package and canonical format reference. An operator source is optional. For a requested ID, PromptKit checks the
configured operator source first, then Notarius's embedded fallback profiles,
then its own built-in catalog. A matching profile is complete: it replaces a
lower-precedence definition rather than merging with it. The maintained
[`dnd-extraction` operator profile](../examples/profiles/dnd-extraction.yml)
is a secret-free deployment artifact; production, development, and local
deployments can each provide a complete definition with that same workload ID.
Use workload-oriented IDs for new profiles instead of model names.
[Operations](operations.md#promptkit-profile-deployment) owns the deployment
workflow and credential-handling guidance.
When **local_backend** is present, its endpoint is trimmed and must use HTTP or
HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are
allowed. User information, queries, and fragments are rejected. A zero
**concurrency_limit** leaves the local backend unrestricted inside PromptKit;
a positive value limits simultaneous local generations. The application-wide
**concurrency.total_llm** limit still applies in both cases. Neither local
backend field has an environment override. Omitting **local_backend** registers
nothing and preserves existing built-in and endpoint-only profile behavior.
A file-backed PromptKit profile selects the registration by its case-sensitive
backend ID:
~~~yaml
id: local-summary
backend: local
model: example-model
~~~
Keep credentials out of the local-backend object. A PromptKit profile may name
its credential environment variable through `api_key_env`; set that variable
only in the run environment. PromptKit owns the
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md).
The [PromptKit upstream boundary](integrations/pkg-promptkit.md) identifies the
supported package API, and [Operations](operations.md#operational-limits)
describes the effective concurrency layers.
`notarius config validate --pipeline <id>` resolves the selected pipeline and
inspects every explicit effective profile without contacting a provider or
requiring credential values. It rejects absent, malformed, or incompatible
profiles before a run prepares modules. Credential availability is checked only
when a generation is prepared.
## Migrating Version 3 Configuration ## Migrating Version 3 Configuration
@@ -154,6 +204,7 @@ Each **pipelines** entry has a unique, non-empty ID and the following shape:
~~~yaml ~~~yaml
pipelines: pipelines:
dnd-session: dnd-session:
llm_profile: dnd-extraction
input: seriatim input: seriatim
chunk: generic chunk: generic
output: json output: json
@@ -166,6 +217,7 @@ pipelines:
| Field | Type | Default | Rules | | Field | Type | Default | Rules |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| **llm_profile** | string | none | Optional non-empty default PromptKit profile ID for selected LLM-backed bindings and validators. An explicitly present blank value is invalid. |
| **input** | module binding | none | Required. | | **input** | module binding | none | Required. |
| **chunk** | module binding | **generic** | Optional. | | **chunk** | module binding | **generic** | Optional. |
| **output** | module binding | **json** | Optional. | | **output** | module binding | **json** | Optional. |
@@ -179,6 +231,12 @@ needs a unique non-empty **id**, an **artifacts** map, and may have
**references**. A lane ID must not appear more than once in a pipeline, **references**. A lane ID must not appear more than once in a pipeline,
including across explicit steps. including across explicit steps.
For each selected LLM-backed binding or validator, profile selection occurs
after module, validator, and `--only` lane selection. It uses the
run-level **--llm-profile** value first, then the binding's **llm_profile**,
then the pipeline's **llm_profile**, and finally the PromptKit default.
Deterministic bindings do not receive these defaults or run overrides.
A lane has these fields: A lane has these fields:
| Field | Type | Default | Rules | | Field | Type | Default | Rules |
@@ -206,7 +264,7 @@ Use an object for fields:
~~~yaml ~~~yaml
extract: extract:
module: dnd/spells module: dnd/spells
llm_profile: gemini-2-flash llm_profile: dnd-extraction
retries: 2 retries: 2
references: references:
spell_catalog: ./dnd-spell-catalog.json spell_catalog: ./dnd-spell-catalog.json
@@ -215,7 +273,7 @@ extract:
| Binding field | Type | Default | Rules | | Binding field | Type | Default | Rules |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| **module** | string | none | Required for an object binding. Must be a registered compatible key. | | **module** | string | none | Required for an object binding. Must be a registered compatible key. |
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID. | | **llm_profile** | string | none | Optional non-empty PromptKit profile ID for an LLM-backed binding. It overrides the pipeline default unless the run supplies **--llm-profile**. |
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. | | **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. |
| **options** | object | none | Must satisfy the selected module. | | **options** | object | none | Must satisfy the selected module. |
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. | | **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
@@ -225,7 +283,8 @@ Omitting **validators** uses the registered chain. **validators: []** selects
an empty chain; a non-empty list replaces the chain in the listed order. an empty chain; a non-empty list replaces the chain in the listed order.
Validator bindings accept only **module**, **llm_profile**, and **options**. Validator bindings accept only **module**, **llm_profile**, and **options**.
They reject **references**, **retries**, and nested **validators**. Deterministic They reject **references**, **retries**, and nested **validators**. Deterministic
validators reject an explicit **llm_profile**. validators reject an explicit **llm_profile**. Deterministic module bindings
also reject an explicit **llm_profile**.
The **json** output module accepts optional **include_chunk_map** and The **json** output module accepts optional **include_chunk_map** and
**evidence_context** settings: **evidence_context** settings:

View File

@@ -98,6 +98,21 @@ summarize results without embedding lane payload bytes. A chunk-plan summary is
provenance for the plan used by this run; cache records, debug artifacts, and provenance for the plan used by this run; cache records, debug artifacts, and
other operational state are not published as bundle files. other operational state are not published as bundle files.
Each `llm_profiles` entry identifies effective, non-secret LLM execution
provenance:
| Field | Required | Meaning |
| --- | --- | --- |
| `id` | Yes | Selected PromptKit profile identifier. |
| `provider` | No | Notarius adapter provider identifier. |
| `model` | No | Effective provider model identifier. |
| `backend_id` | No | Effective PromptKit backend registration identifier. Endpoint-only profiles omit it. |
| `reasoning_effort` | No | Effective opaque provider reasoning setting. An empty or explicitly cleared setting is omitted. |
These values describe observed execution; they are not a backend-registration
interface. Entries that differ by backend or effective reasoning remain
distinct even when their profile, provider, and model are otherwise equal.
## Rejections And Warnings ## Rejections And Warnings
`rejected.json` is always an object with a `rejected` array. Each entry has `rejected.json` is always an object with a `rejected` array. Each entry has

View File

@@ -1,11 +1,11 @@
# PromptKit Integration # PromptKit Integration
Notarius pins Notarius pins
[`gitea.maximumdirect.net/eric/promptkit` v0.1.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0) [`gitea.maximumdirect.net/eric/promptkit` v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0)
as its in-process prompt engine. The upstream as its in-process prompt engine. The upstream
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/consumers/pkg-promptkit.md) [Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/consumers/pkg-promptkit.md)
owns the public engine API, and the upstream owns the public engine API, and the upstream
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/formats.md) [format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
owns prompt, profile, and schema file contracts. owns prompt, profile, and schema file contracts.
## Supported Boundary ## Supported Boundary
@@ -13,13 +13,34 @@ owns prompt, profile, and schema file contracts.
Notarius relies on the root `promptkit` package to: Notarius relies on the root `promptkit` package to:
- construct an `Engine` with filesystem-backed prompt, schema, and optional - construct an `Engine` with filesystem-backed prompt, schema, and optional
profile sources; operator and application-fallback profile sources;
- prepare and run a `RunRequest` with named inline artifacts, variables, - prepare one frozen execution from a `RunRequest` with named inline artifacts,
metadata, prompt identity, and profile selection; variables, a direct session ID, prompt identity, and profile selection, then
record credential-redacted details and run that exact execution;
- return rendered debug material, validated structured output, selected - return rendered debug material, validated structured output, selected
profile and model metadata, and token usage; profile, backend, effective model metadata, and token usage;
- register the optional conventional `local` backend through `BackendLocal`,
`LocalBackend`, and `WithBackend`;
- distinguish structured-output validation failure from execution failure; and - distinguish structured-output validation failure from execution failure; and
- identify a missing explicit profile through `ErrProfileNotFound`. - identify a missing explicit profile through `ErrProfileNotFound` and backend
admission exhaustion through `ErrCapacityExceeded`.
The pinned
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/backends.go)
owns the registration and backend-capacity contract.
For one completion, the adapter calls `PrepareExecution`, takes a
caller-owned `Details` snapshot, and calls `RunPrepared` for that same opaque
prepared execution. It defers `Discard` for every unexecuted handle. Explicit
profile preflight uses `Engine.InspectProfile`; it does not prepare a synthetic
prompt. PromptKit's prepared handle, inspection result, and capacity-error
types stay inside the Notarius LLM adapter.
When a PromptKit profile and runtime override leave `temperature`, `max_tokens`,
or `top_p` unset, Notarius leaves that control unset as well. Compatible
providers therefore apply their own defaults; an operator that requires a
specific sampling value must select it explicitly in the profile or runtime
override.
Notarius does not use PromptKit's optional `ArtifactReader`. It materializes Notarius does not use PromptKit's optional `ArtifactReader`. It materializes
source and reference content itself and supplies owned inline artifacts at the source and reference content itself and supplies owned inline artifacts at the
@@ -27,14 +48,58 @@ adapter boundary. It also retains responsibility for pipeline retries,
scheduling, debug persistence, redaction, profile provenance, and conversion scheduling, debug persistence, redaction, profile provenance, and conversion
from private model responses into durable domain artifacts. from private model responses into durable domain artifacts.
Notarius sends its trimmed run session through PromptKit's direct session
field, which is authoritative for provider session behavior. It also retains
the same value as the `session_id` prompt variable for maintained prompt
compatibility. Session IDs are stable, non-secret correlation identifiers and
may be exposed to providers and provider observability.
Notarius records PromptKit's selected backend ID and effective reasoning
setting as optional run-manifest provenance. Endpoint-only profiles have no
backend ID. Debug prompt material also retains the selected backend ID and
PromptKit's stable lower-case `effective_model_params` JSON, which may include
`backend_id`. Notarius production configuration exposes one optional
conventional `local` registration. It does not expose a general user-defined
PromptKit backend registry. Endpoint-only profiles remain supported unchanged.
Notarius retains its application-wide scheduled client around the PromptKit
adapter. PromptKit may apply a narrower limit for the selected backend;
endpoint-only profiles have no such backend limit. The adapter translates
PromptKit capacity rejection into the provider-neutral Notarius
`ErrLLMCapacityExceeded` contract. It may include the normalized selected
backend ID in safe diagnostic context, without exposing PromptKit's capacity
error type, and leaves retries to the calling pipeline stage.
## Profile Sources And Compatibility
Notarius gives PromptKit the configured operator profile source, registered
application fallback profile assets, and optional backend registration through
the same construction path for inspection and execution. PromptKit owns the
resulting source precedence and strict profile parsing: a matching operator
profile is a complete replacement for a fallback or built-in profile, while an
invalid matching document fails instead of falling through. The operator
configuration and deployment workflow are defined in
[Configuration](../config.md#promptkit-profiles) and
[Operations](../operations.md#promptkit-profile-deployment).
Notarius supports this boundary against PromptKit v0.5.0. Its fallback source,
prepared-execution, inspection, and typed capacity APIs are used as public
upstream contracts; other PromptKit APIs or file-format behavior are not
implicitly supported. A dependency upgrade requires reviewing the adapter,
profile-source construction, and this compatibility statement against the
pinned upstream documentation.
## Notarius Ownership ## Notarius Ownership
[LLM Runtime Internals](../internal/llm.md) describes how Notarius mounts [LLM Runtime Internals](../internal/llm.md) describes how Notarius mounts
module assets, maps its transport-neutral completion contract, prepares and module assets, maps its transport-neutral completion contract, prepares and
executes requests, validates output, records provenance, captures debug executes requests, validates output, records provenance, captures debug
material, redacts errors, and preserves timeout ownership. material, redacts errors, and preserves timeout ownership.
[D&D Module Internals](../internal/dnd.md) owns the embedded
`dnd-extraction` fallback profile and the maintained D&D prompt defaults.
[Configuration](../config.md#promptkit-profiles) defines how a Notarius [Configuration](../config.md#promptkit-profiles) defines how a Notarius
configuration selects one PromptKit profile source. configuration selects one PromptKit profile source and optionally registers
the conventional local backend.
PromptKit API or format changes outside this boundary are not implicitly PromptKit API or format changes outside this boundary are not implicitly
supported. Updating the pinned version requires reviewing the adapter and supported. Updating the pinned version requires reviewing the adapter and

View File

@@ -38,11 +38,14 @@ in [Configuration Internals](configuration.md).
Configuration validation without a selected pipeline checks structural Configuration validation without a selected pipeline checks structural
configuration only. Validation with a selected pipeline also builds the configuration only. Validation with a selected pipeline also builds the
effective catalog, resolves the pipeline, and verifies explicitly selected effective catalog, resolves the pipeline, and verifies every explicit effective
PromptKit profiles. Each explicit binding or validator profile is prepared PromptKit profile. Selected LLM-backed input, chunk, lane, output, and validator
against the configured PromptKit source without performing generation, so an profiles are inspected
unknown profile fails before pipeline preparation. Pipeline listing validates against the configured PromptKit source and backend registrations without
configuration before returning normalized, sorted identifiers. loading a prompt or performing generation, so an unknown or invalid profile
fails before pipeline preparation. Credential availability remains an
execution-time concern. Pipeline listing validates configuration before
returning normalized, sorted identifiers.
## Production Composition ## Production Composition
@@ -59,7 +62,18 @@ profile-provenance recorder, creates one scheduler from the effective global
LLM limit, and wraps the client before it reaches modules. Registration and LLM LLM limit, and wraps the client before it reaches modules. Registration and LLM
construction errors are returned before a pipeline is prepared. Configuration construction errors are returned before a pipeline is prepared. Configuration
field definitions remain in [Configuration](../config.md#promptkit-profiles); field definitions remain in [Configuration](../config.md#promptkit-profiles);
the adapter mechanics remain in [LLM Runtime](llm.md). the D&D registrar's fallback profile assets and the adapter mechanics remain in
[LLM Runtime](llm.md).
The factory also accepts `LLMRuntimeOverrides`, whose reasoning pointer
preserves inherit, replace, and clear states across the composition boundary.
Run orchestration constructs this value from the mutually exclusive
`--reasoning-effort` and `--clear-reasoning-effort` controls. Absence preserves
a nil pointer, replacement is trimmed, and clear uses a non-nil empty string.
The same override reaches the one shared production client, checkpoint
identity, and debug invocation metadata. Persistent reasoning configuration
remains owned by PromptKit profiles; Notarius configuration has no reasoning
field.
## Run Orchestration ## Run Orchestration
@@ -71,7 +85,8 @@ handoff:
2. create and validate a safe run identity, then allocate a debug bundle only 2. create and validate a safe run identity, then allocate a debug bundle only
when requested; when requested;
3. build the effective catalog, resolve requested reference changes, resolve 3. build the effective catalog, resolve requested reference changes, resolve
the effective pipeline, and verify explicit PromptKit profiles; the effective pipeline, and inspect its explicit effective PromptKit
profiles;
4. materialize external or generated references and record redacted invocation 4. materialize external or generated references and record redacted invocation
and resolution provenance when debug capture is enabled; and resolution provenance when debug capture is enabled;
5. construct registries, the scheduled LLM client, prepared modules, and the 5. construct registries, the scheduled LLM client, prepared modules, and the

View File

@@ -39,14 +39,22 @@ This establishes the public precedence order without giving environment input a
second file schema. Loading and application reject malformed YAML, unsupported second file schema. Loading and application reject malformed YAML, unsupported
file versions, unknown fields, invalid values, and identifiers that are empty file versions, unknown fields, invalid values, and identifiers that are empty
or collide after whitespace normalization. The file application also makes the or collide after whitespace normalization. The file application also makes the
effective extraction-worker default follow the effective LLM limit. effective extraction-worker default follow the effective LLM limit. A present
PromptKit local-backend object requires and trims its endpoint, defaults its
omitted concurrency limit to zero, and is copied so the parsed file model
cannot alias the populated **Config**. A pipeline `llm_profile` is
presence-aware: omission remains empty, while a present blank value is
rejected and a non-empty file value is trimmed before it reaches **Config**.
**Config.Validate** checks configuration-only invariants before resolution. It **Config.Validate** checks configuration-only invariants before resolution. It
rejects incompatible profile sources, invalid state-surface values, unsupported rejects incompatible profile sources, invalid state-surface values, unsupported
concurrency settings, malformed bindings and references, invalid retries, and concurrency settings, malformed bindings and references, invalid retries, and
invalid pipeline, step, or lane structure. Its errors retain the closest known invalid pipeline, step, or lane structure. PromptKit local-backend validation
pipeline, lane, and binding context. It deliberately does not require modules accepts only an absolute HTTP or HTTPS endpoint with a host and no user
to be registered: that requires a catalog and belongs to resolution. information, query, or fragment, and rejects a negative local concurrency
limit. Its errors retain the closest known pipeline, lane, and binding context.
It deliberately does not require modules to be registered: that requires a
catalog and belongs to resolution.
The exact user-selectable values and validation rules are defined in The exact user-selectable values and validation rules are defined in
[Configuration](../config.md). Keep additions to the file model, an [Configuration](../config.md). Keep additions to the file model, an
@@ -56,13 +64,15 @@ environment override, its validation, and that reference in the same change.
**Config.Resolve** first recomputes derived concurrency defaults and validates **Config.Resolve** first recomputes derived concurrency defaults and validates
the configuration. It normalizes the requested pipeline ID, copies the selected the configuration. It normalizes the requested pipeline ID, copies the selected
profile, applies a non-empty command-level LLM profile override to the profile, and passes the non-empty command-level LLM profile override, requested
LLM-capable stage bindings, and calls the framework resolver with the requested lane selection, and reference changes to the framework resolver.
lane selection and reference changes.
The command-level override does not replace an explicitly selected validator After module and validator selection, the resolver applies the effective
profile. Validator bindings remain part of the resolved validator chain and profile policy to LLM-backed bindings only: command override, binding profile,
are resolved under their own declared configuration. pipeline profile, then the prompt default. Deterministic bindings remain
profile-free, and no second inheritance decision occurs during execution. The
public field definitions and precedence are owned by
[Configuration](../config.md#pipelines).
The framework resolver supplies defaults, selects lanes, resolves validator The framework resolver supplies defaults, selects lanes, resolves validator
chains, checks registered module and artifact compatibility, validates module chains, checks registered module and artifact compatibility, validates module
@@ -71,7 +81,8 @@ options, and returns the fixed ordered pipeline shape. The resulting
changes, a clone of the input configuration, and the resolved pipeline. changes, a clone of the input configuration, and the resolved pipeline.
Callers may therefore retain or modify their input slices and maps without Callers may therefore retain or modify their input slices and maps without
changing the resolved result, and later consumers cannot mutate the original changing the resolved result, and later consumers cannot mutate the original
configuration through the effective value. configuration through the effective value. This ownership includes the nested
PromptKit local-backend value.
Resolution failures stop before module construction and source parsing. They Resolution failures stop before module construction and source parsing. They
include an error path for an unconfigured pipeline, missing module, missing include an error path for an unconfigured pipeline, missing module, missing
@@ -83,7 +94,8 @@ runtime error class described in the [CLI reference](../cli.md#output-streams-an
The framework assigns the resolved pipeline a deterministic SHA-256 digest The framework assigns the resolved pipeline a deterministic SHA-256 digest
after defaults, lane selection, module bindings, reference bindings, validator after defaults, lane selection, module bindings, reference bindings, validator
chains, and artifact schema identity have been resolved. The digest excludes chains, effective LLM profiles, and artifact schema identity have been
resolved. The digest excludes
its own stored value. It identifies resolved composition rather than raw YAML its own stored value. It identifies resolved composition rather than raw YAML
bytes, a debug payload, or all runtime state. The CLI records it as invocation bytes, a debug payload, or all runtime state. The CLI records it as invocation
provenance before execution; cache and checkpoint identity have additional provenance before execution; cache and checkpoint identity have additional
@@ -94,9 +106,12 @@ Configuration summaries must use **Redacted**, **RedactedSummaryPayload**, or
Those methods copy every binding and nested option container, replace values Those methods copy every binding and nested option container, replace values
whose key is credential-shaped with **[REDACTED]**, and omit materialized whose key is credential-shaped with **[REDACTED]**, and omit materialized
reference content while retaining safe binding and reference provenance. The reference content while retaining safe binding and reference provenance. The
payload must not alias the source configuration or resolved pipeline. This payload must not alias the source configuration or resolved pipeline.
redaction is deliberately narrow: it protects configuration summaries and does PromptKit's local endpoint and concurrency limit are preserved as non-secret
not authorize recording arbitrary environment values or provider requests. configuration metadata in the independently owned summary; the object contains
no credential value. This redaction is deliberately narrow: it protects
configuration summaries and does not authorize recording arbitrary environment
values or provider requests.
## Invariants To Preserve ## Invariants To Preserve

View File

@@ -22,10 +22,15 @@ does not repeat their JSON shapes or schemas.
## Family Composition ## Family Composition
The D&D registrar registers the familys artifact codecs, extractors, typed The D&D registrar registers the familys artifact codecs, extractors, typed
append-order mergers, normalizers, validators, prompt assets, and default append-order mergers, normalizers, validators, prompt assets, fallback LLM
validator chains. Each extractor and normalizer has a stable module spec, profile asset, and default validator chains. Each extractor and normalizer has
strict option decoding, and a typed builder. Configuration remains the a stable module spec, explicit execution class, strict option decoding, and a
canonical owner of the exact keys and validator order. typed builder. Scene chunking, every extractor, and NPC normalization are
registered as `llm_backed`; the remaining current D&D mergers and normalizers
are `deterministic`. The metadata is available to catalog inspection and
resolved-pipeline debug data and determines which selected bindings inherit the
pipeline profile. Configuration remains the canonical owner of the exact keys,
profile precedence, and validator order.
Private structured-LLM response schemas are deliberately minimal. They reject Private structured-LLM response schemas are deliberately minimal. They reject
invalid JSON structure, missing required fields, incompatible types, and invalid JSON structure, missing required fields, incompatible types, and
@@ -41,6 +46,15 @@ reference, and transcript assets instead of copying their text into individual
modules. A manifests declared sequence, including cache-control placement, is modules. A manifests declared sequence, including cache-control placement, is
part of the prompt behavior. part of the prompt behavior.
Every maintained D&D LLM prompt selects `dnd-extraction` as its default
profile. The D&D registrar embeds that fallback profile with the maintained
OpenRouter model, timeout, and service-tier policy. An operator may provide a
complete profile with the same ID through the configured PromptKit source; that
definition replaces the fallback rather than merging with it. The fallback
leaves reasoning and optional sampling controls unspecified. Deployment profile
selection and the maintained operator example are documented in
[Configuration](../config.md#promptkit-profiles).
All extraction prompts share this four-message rendered prefix: the system All extraction prompts share this four-message rendered prefix: the system
message without cache control, the identity message without cache control, the message without cache control, the identity message without cache control, the
campaign-reference message with ephemeral cache control, and the chunk campaign-reference message with ephemeral cache control, and the chunk

View File

@@ -26,20 +26,72 @@ durable schemas. Those responsibilities remain with the module and its
`PromptKitClient` validates the request target and prompt identity, maps each `PromptKitClient` validates the request target and prompt identity, maps each
named material to a PromptKit inline artifact while preserving its origin URI, named material to a PromptKit inline artifact while preserving its origin URI,
maps the request session to the existing `session_id` prompt variable, forwards maps the trimmed request session to PromptKit's direct per-run session field,
profile selection, then prepares and runs the prompt. PromptKit v0.1.0 has no retains the same value as the `session_id` prompt variable for maintained
direct request-level session field. The adapter returns PromptKits validated prompt compatibility, and forwards profile selection. It then creates one
raw bytes rather than re-encoding the decoded target. An empty optional frozen prepared execution, captures its caller-owned credential-redacted
material is represented as one space so its named input is retained by details for debug material, and executes that exact snapshot through
PromptKit. PromptKit's prepared-execution boundary. The direct field
is authoritative for provider session behavior. A session ID is a stable,
non-secret correlation identifier and may be exposed to providers and provider
observability. The adapter returns PromptKits validated raw bytes rather than
re-encoding the decoded target. An empty optional material is represented as
one space so its named input is retained by PromptKit.
An empty request profile lets the prompt select its configured default. The CLI Client construction may also receive a run-wide reasoning-effort override from
prepares every explicitly selected binding profile before a run begins, so a the CLI factory boundary. The adapter copies the caller-owned pointer and
missing explicit profile fails before stage execution. Calls record the profile creates a fresh PromptKit execution override for each request: a nil pointer
actually selected by PromptKit; the recorder deduplicates non-secret profile inherits the selected profile, a non-empty value replaces it, and an empty
identity, provider, and model values for manifest use. Successful completion value clears inherited reasoning. The CLI's mutually exclusive
responses and recorded profile manifests identify the adapter provider as `--reasoning-effort` and `--clear-reasoning-effort` controls select those
`promptkit`. states. With neither flag, profile behavior remains unchanged. Because
production constructs one shared client, the selected state applies uniformly
to module calls, retries, and LLM-backed validators for the whole run.
An empty request profile lets the prompt select its configured default. Before a
run begins, the CLI asks the adapter to inspect every explicit profile on the
resolved selected LLM-backed bindings and validators, including inherited
pipeline profiles. Inspection resolves the profile and its selected backend and
target without loading a prompt, reading credentials, admitting capacity, or
contacting a provider, so a missing or invalid explicit profile fails before
stage execution while a valid `api_key_env` may remain unset. Calls record the
profile actually selected by PromptKit. The recorder trims and deduplicates
non-secret profile identity, provider, model, selected backend ID, and
effective reasoning values for manifest use. Entries that differ in backend or
reasoning remain distinct and deterministically ordered. Endpoint-only profiles
retain an empty backend ID, which the published JSON omits. Successful
completion responses and recorded profile manifests identify the adapter
provider as `promptkit`.
The CLI's profile-inspection engine and the production adapter use the same
profile-source construction to apply the configured profile directory or file,
the optional registered fallback profile assets, and the optional conventional
`local` backend. Preflight therefore resolves the same profile sources and
backend membership as runtime without performing generation. Fallback assets
are mounted only when at least one source is registered. The production D&D
registrar contributes its `dnd-extraction` fallback, and the maintained D&D
prompts select that logical ID by default. PromptKit owns source precedence and
profile parsing: an operator-provided matching profile takes precedence over a
fallback profile without Notarius merging either document.
When the registration is absent, a profile selecting `backend: local` fails
inspection instead of falling back to a built-in or endpoint-only target.
Before execution, the adapter also contributes a non-secret checkpoint
fingerprint for the effective PromptKit profile source. It combines the
identity of PromptKit's compiled-in profile catalog with a deterministic digest
of every YAML profile in the configured profile directory, or of the configured
profile file, and a deterministic digest of the flattened fallback profile
assets. The fingerprint contains neither profile content nor source paths. It
covers inherited pipeline profiles, explicit binding profiles, and
prompt-selected defaults, so changing a model or other profile setting cannot
reuse checkpoints created under the
prior profile source. This cache identity is independent of durable
profile provenance: run manifests continue to list only profiles actually
observed during LLM calls. When the local backend is registered, a second
fingerprint hashes its trimmed endpoint behind a stable marker. Changing that
semantic execution target invalidates checkpoint reuse. The raw endpoint is not
stored in checkpoint identity, and the local concurrency limit is excluded
because it changes scheduling rather than execution semantics.
## Shared Provider-Call Limit ## Shared Provider-Call Limit
@@ -55,12 +107,23 @@ worker counts cannot exceed the configured LLM limit. The configuration field
and its effective default are owned by and its effective default are owned by
[Configuration](../config.md#concurrency-output-cache-and-debug). [Configuration](../config.md#concurrency-output-cache-and-debug).
PromptKit applies a second, independent admission limit when the selected
profile names a limited backend. It sits beneath the Notarius scheduled client,
so it may narrow but cannot expand the application-wide limit. Built-in
OpenRouter profiles select PromptKit's reserved backend and its upstream
capacity policy. A positive configured local-backend limit bounds active local
generations inside PromptKit; zero leaves that backend unlimited there.
Endpoint-only profiles do not select a PromptKit backend and remain limited
only by the Notarius scheduler.
## Prompt And Schema Assets ## Prompt And Schema Assets
An `AssetRegistry` collects prompt and schema filesystems from production module An `AssetRegistry` collects prompt, schema, and optional fallback-profile
families. It flattens registered roots into the PromptKit filesystems and filesystems from production module families. It flattens registered roots into
rejects invalid roots, unreadable assets, duplicate paths, and missing prompt the corresponding PromptKit filesystems and rejects invalid roots, unreadable
or schema files during preparation. The frameworks `promptfs` helper combines assets, duplicate paths, and missing prompt or schema files during preparation.
Fallback assets receive a safe content digest for checkpoint identity; raw
paths and bytes are never included. The frameworks `promptfs` helper combines
module-owned prompt files with reusable domain fragments without making the module-owned prompt files with reusable domain fragments without making the
framework depend on D&D content. framework depend on D&D content.
@@ -106,6 +169,16 @@ failure, empty structured body, or decode failure as
material when they exist. Provider failures remain operational errors rather material when they exist. Provider failures remain operational errors rather
than output-validation failures. than output-validation failures.
When PromptKit rejects backend admission before generation, the adapter maps
`promptkit.ErrCapacityExceeded` to
`contracts.ErrLLMCapacityExceeded`, retaining prompt context and a redacted
upstream diagnostic without exposing the PromptKit sentinel or capacity-error
type as a framework contract. When supplied, the normalized selected backend
ID appears only in that safe application-owned diagnostic context. A canceled
caller context takes precedence. The adapter does not retry capacity failures;
the pipeline's existing binding attempt policy sees the operational error and
decides whether to rerun the complete operation.
Prompt-declared repair is executed within PromptKits structured-output flow. Prompt-declared repair is executed within PromptKits structured-output flow.
The current production D&D prompt manifests set repair attempts to zero. That The current production D&D prompt manifests set repair attempts to zero. That
setting does not replace pipeline retry behavior: a bindings configured retry setting does not replace pipeline retry behavior: a bindings configured retry
@@ -125,7 +198,7 @@ does not install another timeout wrapper around PromptKit.
The selected PromptKit profile owns generation settings. Notarius binding The selected PromptKit profile owns generation settings. Notarius binding
retries remain outside the adapter and repeat the complete module operation retries remain outside the adapter and repeat the complete module operation
and validation chain. PromptKit v0.1.0 does not add a provider retry loop. and validation chain. PromptKit does not add a provider retry loop.
Operator-facing behavior is summarized in Operator-facing behavior is summarized in
[Operations](../operations.md#operational-limits), and the pinned upstream [Operations](../operations.md#operational-limits), and the pinned upstream
contract is identified in contract is identified in
@@ -135,18 +208,22 @@ contract is identified in
When debug recording is enabled, the pipeline decorates the shared client. The When debug recording is enabled, the pipeline decorates the shared client. The
wrapper records prepared prompt and response material, timing, selected profile wrapper records prepared prompt and response material, timing, selected profile
and model, and call identifiers in the runs debug bundle, including material and backend, effective model parameters, and call identifiers in the runs
available from a failed structured completion. For a successful completion, a debug bundle, including material available from a failed structured completion.
debug-write failure is surfaced; when the completion already failed, its call Effective parameters use PromptKit's stable lower-case JSON field names and may
error remains the result. Debug-bundle location, retention, and handling are include `backend_id`. For a successful completion, a debug-write failure is
operational concerns documented in [Operations](../operations.md#debug-bundles). surfaced; when the completion already failed, its call error remains the
result. Debug-bundle location, retention, and handling are operational concerns
documented in [Operations](../operations.md#debug-bundles).
Run manifests receive selected profile summaries and component identities, not Run manifests receive selected profile summaries, including optional effective
prompt, schema, source, reference, or response content. Provider error text is backend and reasoning provenance, and component identities—not prompt, schema,
wrapped with prompt context and bearer credentials are redacted before it source, reference, or response content. The published field semantics belong
crosses the runtime boundary. Known-secret redaction is available to other to the [JSON output contract](../integrations/json-output.md#manifestjson).
runtime collaborators; it does not make prompt or response contents safe for Provider error text is wrapped with prompt context and bearer credentials are
general logging. redacted before it crosses the runtime boundary. Known-secret redaction is
available to other runtime collaborators; it does not make prompt or response
contents safe for general logging.
## Failure Boundaries ## Failure Boundaries
@@ -154,6 +231,8 @@ general logging.
sources, invalid asset registration, or a non-positive scheduler limit. sources, invalid asset registration, or a non-positive scheduler limit.
- Preparation failures, unavailable explicit profiles, provider failures, and - Preparation failures, unavailable explicit profiles, provider failures, and
context cancellation propagate to the calling stage with context. context cancellation propagate to the calling stage with context.
- Backend admission exhaustion is a provider-neutral operational error and is
not classified as invalid structured output or validator rejection.
- Malformed or schema-invalid provider output is classified separately as - Malformed or schema-invalid provider output is classified separately as
invalid structured output so the module or pipeline can apply its own retry invalid structured output so the module or pipeline can apply its own retry
and rejection policy. and rejection policy.

View File

@@ -12,9 +12,15 @@ exceptions. See [D&D Module Internals](dnd.md) rather than adding them here.
A module is a typed implementation registered for one pipeline stage. Its A module is a typed implementation registered for one pipeline stage. Its
`ModuleSpec` is the public-to-the-framework declaration of its stable key, `ModuleSpec` is the public-to-the-framework declaration of its stable key,
stage, required and provided capabilities, artifact kind, and accepted stage, execution class, required and provided capabilities, artifact kind, and
reference slots. The framework uses that declaration to resolve a configured accepted reference slots. The execution class states whether a module is
binding before it builds the implementation. `deterministic` or `llm_backed`; registries retain it for catalog inspection and
resolved-pipeline debug data without constructing the module. The framework
uses the declaration to resolve a configured binding before it builds the
implementation. After selection, the resolver applies profile inheritance only
to bindings whose declared execution class is `llm_backed` and rejects a
binding-specific profile on a deterministic module. The user-facing precedence
contract belongs in [Configuration](../config.md#pipelines).
Implementations that accept options must provide both an option validator and Implementations that accept options must provide both an option validator and
a builder. The validator is used while resolving configuration; the builder a builder. The validator is used while resolving configuration; the builder
@@ -47,7 +53,7 @@ Production composition is intentionally split by family:
- The Seriatim registrar provides the transcript input adapter. Its external - The Seriatim registrar provides the transcript input adapter. Its external
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md). input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
- The D&D registrar provides its codecs, extractors, mergers, normalizers, - The D&D registrar provides its codecs, extractors, mergers, normalizers,
validators, prompt assets, and default chains. Its behavioral conventions validators, prompt assets, fallback profile asset, and default chains. Its behavioral conventions
are documented in [D&D Module Internals](dnd.md). are documented in [D&D Module Internals](dnd.md).
The CLI owns the composition that invokes these registrars. A module package The CLI owns the composition that invokes these registrars. A module package
@@ -59,10 +65,10 @@ packages depend on production extensions.
1. Choose the pipeline stage and the typed artifact boundary. Put external 1. Choose the pipeline stage and the typed artifact boundary. Put external
input or durable artifact formats in the relevant integration contract, input or durable artifact formats in the relevant integration contract,
not in this guide or in a private LLM response type. not in this guide or in a private LLM response type.
2. Define a stable `ModuleSpec` with the exact capabilities and reference 2. Define a stable `ModuleSpec` with an explicit execution class, the exact
slots needed for the operation. Model a producer/consumer handoff as an capabilities, and reference slots needed for the operation. Model a
artifact-compatible slot; configuration then chooses an external file or a producer/consumer handoff as an artifact-compatible slot; configuration
generated binding. then chooses an external file or a generated binding.
3. Implement strict option decoding, construction, and the typed stage 3. Implement strict option decoding, construction, and the typed stage
interface. Preserve caller ownership: do not retain mutable request data interface. Preserve caller ownership: do not retain mutable request data
and return defensive copies where an implementation exposes stored data. and return defensive copies where an implementation exposes stored data.

View File

@@ -34,6 +34,11 @@ requested lanes where that is supported, resolves validator chains, checks
module capabilities and typed artifact compatibility, validates options, and module capabilities and typed artifact compatibility, validates options, and
assigns a deterministic resolved-composition digest. The resolved pipeline assigns a deterministic resolved-composition digest. The resolved pipeline
contains bindings and declared reference targets, not external reference bytes. contains bindings and declared reference targets, not external reference bytes.
After selection, the resolver applies command, binding, and pipeline profile
precedence to LLM-backed bindings and validators only; prompt defaults remain
an empty resolved binding profile. Deterministic bindings remain profile-free.
These effective values are part of the digest, so execution and checkpoint
consumers do not repeat profile inheritance.
Configuration resolution supplies the selected profile and catalog; see Configuration resolution supplies the selected profile and catalog; see
[Configuration Internals](configuration.md). [Configuration Internals](configuration.md).

View File

@@ -39,7 +39,9 @@ codecs, loader, and recorder. The CLI constructs a recorder whenever checkpoint
recording is enabled and constructs a loader only for a `--resume` invocation. recording is enabled and constructs a loader only for a `--resume` invocation.
Identity incorporates explicit stable semantic fingerprints collected from Identity incorporates explicit stable semantic fingerprints collected from
prepared modules and validators in addition to configuration, input, prepared modules and validators in addition to configuration, input,
references, runtime overrides, and LLM profiles. references, runtime overrides, observed LLM profiles, and the LLM runtime's
non-secret effective profile-source identity. A profile source change therefore
causes a cold miss even when the configured profile ID remains unchanged.
The serialized The serialized
`workspace_schema_version` identifiers are frozen wire-compatibility fields; `workspace_schema_version` identifiers are frozen wire-compatibility fields;
they do not describe a current public state surface. they do not describe a current public state surface.

View File

@@ -42,6 +42,42 @@ evidence publication. Apply an appropriate umask and output-root access policy
before enabling that option; the requested output modes alone may not be before enabling that option; the requested output modes alone may not be
suitable for transcript-bearing bundles. suitable for transcript-bearing bundles.
## PromptKit Profile Deployment
Profile deployment has four distinct layers:
| Layer | Owner | Operational role |
| --- | --- | --- |
| Prompts and schemas | Notarius module families | Embedded request and structured-output definitions. They are not deployment profile files. |
| Fallback profiles | Notarius module families | Embedded application defaults, including D&D's `dnd-extraction` profile. |
| Built-in profiles | PromptKit | Upstream catalog entries available when no higher-precedence source defines an ID. |
| Operator profiles | Deployment filesystem | Complete environment-specific definitions selected by `promptkit.profile_file` or `promptkit.profile_dir`. |
The maintained D&D pipeline uses the workload ID `dnd-extraction`. The
embedded fallback makes that ID usable without an operator file. Production,
development, and local deployments can each install a different complete
definition for the same ID, retaining the pipeline while choosing their own
model, backend, timeout, or reasoning policy. An operator definition wins over
the fallback; it is not merged with it. The configuration field and full
precedence rules are owned by [Configuration](config.md#promptkit-profiles).
Use a profile source owned by the service account, keep it readable only by
the intended operator, and supply provider credentials through the service
environment—not in the Notarius configuration or profile YAML. The maintained
[operator profile](../examples/profiles/dnd-extraction.yml) is secret-free and
can be copied as a format starting point. Validate a deployment without a
provider call or credentials:
~~~sh
notarius config validate --config /etc/notarius/config.yml --pipeline dnd-session
~~~
Profile paths are currently resolved from the process working directory, not
from the configuration file. The complete example's
`./examples/profiles/dnd-extraction.yml` path is valid for a repository-root
invocation only. Use absolute paths such as
`/etc/notarius/profiles/dnd-extraction.yml` for services and containers.
## Run Lifecycle ## Run Lifecycle
Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads
@@ -117,9 +153,15 @@ compatible recorded work. A resume request fails when checkpoint recording is
disabled. Without **--resume**, a recording-enabled run executes normally and disabled. Without **--resume**, a recording-enabled run executes normally and
does not load checkpoint state. Compatibility includes the resolved pipeline, does not load checkpoint state. Compatibility includes the resolved pipeline,
input, selected lanes, runtime overrides, reference provenance, LLM-profile input, selected lanes, runtime overrides, reference provenance, LLM-profile
provenance, and prepared-component fingerprints. A changed identity produces a provenance, the effective PromptKit profile-source fingerprint, and
cold miss; Notarius does not migrate, rewrite, or delete older checkpoint prepared-component fingerprints. When a local PromptKit backend is configured,
directories. compatibility also includes a non-secret fingerprint of its endpoint. Changing
profile content or the local endpoint causes a cold miss; changing only the
local concurrency limit does not. A changed identity produces a cold miss;
Notarius does not migrate, rewrite, or delete older checkpoint directories.
Reasoning-effort inheritance, replacement, and explicit clearing are distinct
runtime identities, so checkpoints created under one state are not reused by
either of the others.
Checkpoint state is confined below an identity-specific path: Checkpoint state is confined below an identity-specific path:
@@ -179,7 +221,9 @@ warning, checkpoint, chunk-plan, and terminal reporting artifacts. The trace
contains allowlisted application diagnostic records and can include source or contains allowlisted application diagnostic records and can include source or
derived application data. Neither surface is a cache input. Do not treat a derived application data. Neither surface is a cache input. Do not treat a
debug bundle as safe to share merely because its configuration summary is debug bundle as safe to share merely because its configuration summary is
redacted. redacted. Invocation metadata omits reasoning effort when it is inherited,
records the replacement value when one is supplied, and records an empty value
when inherited reasoning was explicitly cleared.
Notarius never creates debug state without an explicit request and never Notarius never creates debug state without an explicit request and never
automatically deletes a requested bundle. If allocation succeeds, the command automatically deletes a requested bundle. If allocation succeeds, the command
@@ -208,9 +252,12 @@ or automatic cleanup command.
## Operational Limits ## Operational Limits
Provider execution settings and the generation timeout come from the selected Provider execution settings and the generation timeout come from the selected
PromptKit profile. PromptKit v0.1.0 does not add a provider retry loop; PromptKit profile. The invocation-only **--reasoning-effort** and
Notarius binding retries rerun the complete module operation and validation **--clear-reasoning-effort** controls may replace or clear that profile setting
chain as defined by [module bindings](config.md#module-bindings-and-validators). for all LLM-backed calls in one run without changing the profile. PromptKit
v0.5.0 does not add a provider retry loop. Notarius binding retries rerun the
complete module operation and validation chain as defined by
[module bindings](config.md#module-bindings-and-validators).
Timeouts are layered. Caller cancellation is the outer authority. A positive Timeouts are layered. Caller cancellation is the outer authority. A positive
effective generation timeout adds an inner request deadline, while zero effective generation timeout adds an inner request deadline, while zero
@@ -219,7 +266,25 @@ transport-wide cap. Notarius does not add another timeout around PromptKit.
The pinned upstream boundary and profile-format links are in The pinned upstream boundary and profile-format links are in
[PromptKit Integration](integrations/pkg-promptkit.md). [PromptKit Integration](integrations/pkg-promptkit.md).
Concurrency limits are configuration contracts; see Concurrency has two independent layers. Notarius **total_llm** is the
application-wide provider-call limit shared by all backends, modules, retries,
and validators. PromptKit may impose a narrower admission limit for the
selected backend. The effective active-generation bound is the intersection of
both limits and can therefore be lower than **total_llm**. Built-in OpenRouter
profiles use PromptKit's upstream backend limit; endpoint-only profiles have no
PromptKit backend limit and remain bounded by Notarius. For the configured
local backend, a zero **concurrency_limit** leaves only the Notarius scheduler
as a call limit. A positive value makes the effective active local-generation
bound the smaller of **total_llm** and that local limit.
For a positive local limit, PromptKit owns its default waiting capacity and
admission behavior. When a PromptKit backend has admitted all active and queued
work, a new call fails as capacity exhaustion before generation. The adapter
maps that failure to Notarius's existing provider-neutral capacity error and
does not retry it. The calling stage's configured retry policy applies
normally, and the run fails if those attempts are exhausted. Caller
cancellation remains authoritative. Configuration contracts are documented
under [PromptKit profiles](config.md#promptkit-profiles) and
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker [concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
limits and actual provider-call limits are independent. Notarius writes local limits and actual provider-call limits are independent. Notarius writes local
filesystem state only; remote storage, archival, and retention automation are filesystem state only; remote storage, archival, and retention automation are

View File

@@ -177,7 +177,8 @@ individual modules.
The application-wide LLM scheduler bounds actual provider calls independently The application-wide LLM scheduler bounds actual provider calls independently
of framework worker limits. Every LLM-backed module, retry, and validator uses of framework worker limits. Every LLM-backed module, retry, and validator uses
the single injected scheduled client, including work performed by overlapping the single injected scheduled client, including work performed by overlapping
lanes. lanes. Provider runtime adapters may enforce a narrower backend-specific limit
beneath this mandatory application-wide scheduler.
## Configuration And Provenance ## Configuration And Provenance

View File

@@ -53,6 +53,71 @@ not as committed release dates.
spell, combat, interaction, and scene-description lanes after real-world use. spell, combat, interaction, and scene-description lanes after real-world use.
Add more complex chunking only in response to demonstrated failures. Add more complex chunking only in response to demonstrated failures.
## Cross-Cutting LLM Runtime
### Deterministic Prompt Session Identity
- Replace the source-document-ID default for prompt sessions with one
predictable, procedurally generated session ID for the complete
source-processing workload.
- Preserve an explicit non-empty `--session-id` as the highest-precedence
override. Otherwise, derive the default only from the effective input module
identity and the exact raw input bytes.
- Use a versioned, bounded representation such as
`notarius:v1:<sha256(input-module + NUL + raw-input)>`. The exact encoding
must fit PromptKit's session length contract and must not embed source
content.
- Keep the derived session stable across runs, pipelines, selected lanes,
ordered steps, retries, resume, recomputation, LLM profiles, reasoning
overrides, and output, debug, or cache settings.
- Do not include file-backed references, generated references, reference
contents, or the composition of a reference bundle in session derivation.
References may change between prompt calls within one pipeline without
changing routing affinity.
- Resolve the authoritative session before checkpoint construction and use the
same value for checkpoint runtime identity, every prompt-facing module,
PromptKit's direct session field, the compatibility `session_id` prompt
variable, run-manifest metadata, and debug metadata.
- Keep routing identity separate from cache and checkpoint content identity.
Exact prompt prefixes, reference contents, model settings, and other
generation-affecting inputs must continue to participate in their existing
hashes and checkpoint fingerprints even though they do not change the
session.
- Treat the generated value as a provider-visible, stable pseudonymous
correlation identifier. Do not introduce an installation-specific HMAC or
secret unless a concrete multi-tenant or privacy requirement justifies
sacrificing deterministic identity across installations.
### Raise The Default Application-Wide LLM Limit
- Raise the default `concurrency.total_llm` value from 1 to 16 so ordinary
single-backend runs can use PromptKit's expected OpenRouter capacity and
lower-capacity local backends without an unnecessarily narrower Notarius
limit.
- Keep the Notarius application-wide scheduler mandatory and require
`total_llm` to remain a positive integer. Do not make the default unlimited:
endpoint-only profiles, an unrestricted local backend, injected clients, and
aggregate work across several backends may have no narrower PromptKit limit.
- Continue defaulting `concurrency.stage_workers.extract` to the effective
`total_llm`, making its default 16 as part of the same change. Preserve an
explicit lower extract-worker setting when an operator wants less queued or
concurrent extraction work.
- Define effective provider concurrency as the intersection of the Notarius
application-wide limit, the selected PromptKit backend limit when present,
and the work made available by stage execution. A Notarius limit of 16 does
not narrow a backend already limited to 16, while a local backend limited to
4 remains bounded at 4.
- Treat the default as an application-wide safety ceiling across profiles,
backends, modules, retries, and validators. A run that intentionally needs
the combined capacity of several backends may configure a higher
`total_llm` and an appropriate extract-worker count explicitly.
- Retain the existing configuration and environment override surfaces. Update
canonical configuration, operations, and internal documentation together
when the default changes.
- Reconsider decoupling the extract-worker default from `total_llm` only after
mixed-backend workloads demonstrate a need for a high global emergency
ceiling with a lower default work-production rate.
## Shared Normalization And Quality Work ## Shared Normalization And Quality Work
### Generic LLM-Assisted Deduplication ### Generic LLM-Assisted Deduplication
@@ -91,31 +156,6 @@ safety checks, and deterministic application of accepted changes.
- Add media-type validators when non-JSON artifact representations are - Add media-type validators when non-JSON artifact representations are
introduced. introduced.
## LLM Runtime Evolution
### Native Session Propagation
- Once upstream PromptKit exposes a direct request-level session identifier,
propagate the existing `StructuredCompletionRequest.SessionID` through the
PromptKit adapter's native session field.
- Preserve the current `--session-id` invocation contract and its run-wide
propagation to every prompt-facing module and validator. Do not introduce a
second session configuration surface.
- Retain session identity in checkpoint provenance so runs with different
sessions cannot reuse one another's LLM-derived checkpoints.
- Define the upstream compatibility and prompt-variable transition explicitly:
native provider session behavior must not silently remove a `session_id`
prompt variable while maintained prompts still consume it.
- Add adapter and assembled-run coverage for exact forwarding, trimming,
concurrent-run isolation, and unsupported-provider behavior once the
upstream contract is available.
This work is blocked because PromptKit v0.1.0 does not expose the required
direct request-level session field. Notarius already carries a run-scoped
session ID through its CLI, pipeline requests, checkpoint identity, and a
`session_id` prompt variable; that prompt-variable propagation is not native
provider session support.
## Further Reference Evolution ## Further Reference Evolution
- Make prior-run artifacts easier to bind as references without changing the - Make prior-run artifacts easier to bind as references without changing the

View File

@@ -1,376 +1,586 @@
# PromptKit Dependency Migration Implementation Plan # PromptKit v0.5 Implementation Plan
## Status
In progress. The dependency, framework adapter, version 4 PromptKit
configuration migration, provider-neutral module prompt-asset support,
provenance alignment, and canonical documentation are implemented; final
repository verification is still planned.
## Objective ## Objective
Replace Notarius's dependency on Implement the target state in
`gitea.maximumdirect.net/eric/scriptorium v0.11.1` with [PromptKit v0.5 Integration And LLM Profile Policy](promptkit.md). Each numbered
`gitea.maximumdirect.net/eric/promptkit v0.1.0`, and consistently adopt the stage is intended to be one implementation prompt for a GPT-5.6-Terra coding
PromptKit name at the adapter, configuration, provenance, documentation, and agent. Complete stages in order and leave the repository buildable, tested, and
module-support boundaries. internally coherent after every stage.
The migration is a clean break. Notarius has no production compatibility Follow [Architecture](../policy/architecture.md),
requirement for the existing Scriptorium-named configuration or development [Testing Policy](../policy/testing.md), and
artifacts, so the implementation must not add legacy configuration aliases, [Documentation Policy](../policy/documentation.md) throughout. Preserve
deprecated Go APIs, schema migrations, or compatibility shims. unrelated user changes. Use `apply_patch` for source and documentation edits,
run `gofmt` on changed Go files, and add only tests that protect the behaviors
and risks assigned to that stage.
PromptKit's public engine contract is source-compatible with the subset Do not implement the separate deterministic session-ID or default-concurrency
Notarius currently consumes. Preserve current prompt content, prompt and schema roadmap items as part of this plan. Do not perform paid or credentialed LLM
identities, structured-completion behavior, LLM scheduling, retry behavior, calls.
debug capture, redaction, and durable output shapes except for the intentional
configuration-version and LLM-provider provenance changes described below.
## Governing Decisions ## Background Summary
- Pin PromptKit at `v0.1.0` and remove Scriptorium from `go.mod` and `go.sum`. Notarius currently pins PromptKit v0.3.0, calls `Prepare` and then `Run` for one
- Rename the top-level configuration section from `scriptorium` to completion, validates profiles through a synthetic prompt, has no application
`promptkit`. fallback profile source, and accepts LLM profiles only at individual bindings
- Bump the strict file-configuration version from 3 to 4. Version 4 accepts or through the run-wide CLI override. PromptKit v0.5.0 is source-compatible
only the new `promptkit` key; it does not recognize `scriptorium`. with the current tree; a temporary v0.5.0 module override has already passed
- Rename the internal adapter and its exported-within-`internal` Go API to `go test ./...`.
PromptKit terminology without transitional aliases.
- Use provider-neutral names for module-owned prompt asset helpers. Those
helpers describe Notarius assets rather than the library that loads them.
- Record `promptkit`, not `scriptorium`, as the LLM adapter/provider value in
completion responses and run manifest profile provenance.
- Do not add the PromptKit package version to checkpoint identity. A library
implementation version is not itself a semantic pipeline input. Existing
prompt, schema, module, reference, profile, runtime-override, and prepared
component fingerprints remain responsible for semantic invalidation.
- Do not adopt PromptKit's optional `ArtifactReader` extension in this work.
Notarius retains ownership of materializing and supplying prompt inputs.
- Do not add native session propagation in this work. PromptKit `v0.1.0` does
not expose the desired request-level session identifier; retain the existing
prompt-variable behavior and update the future roadmap terminology only.
- Accept PromptKit's documented timeout layering: caller context cancellation
is the outer authority, a positive generation timeout supplies an inner
request deadline, zero disables only that generation deadline, and the HTTP
client timeout remains a transport-wide cap.
- PromptKit and Notarius are both GPL-3.0 licensed, so the dependency change
requires no Notarius licensing change.
## Non-Goals The implementation must nevertheless treat the upstream optional-parameter
change as intentional: unset `temperature`, `max_tokens`, and `top_p` remain
unset and are omitted from compatible provider requests. Do not restore the old
implicit `top_p: 1` default.
- Changing prompt text, message ordering, prompt IDs, schema IDs, or embedded ## Stage 1: Upgrade The PromptKit Dependency
asset paths.
- Changing public artifact schemas, output bundle contents, run-result
receipts, checkpoint formats, or filesystem layouts.
- Adding new LLM profiles, changing profile precedence, or changing credential
handling.
- Refactoring the transport-neutral `StructuredLLMClient` contract.
- Adding PromptKit features that Notarius does not currently need.
- Preserving version 3 configuration compatibility.
## Stage 1: Replace the Dependency and Adapter ### Goal
Update the Go dependency and the framework adapter as one compilable change. Establish a clean PromptKit v0.5.0 baseline before adopting its new APIs.
### Implementation ### Work
- Replace the Scriptorium requirement with - Update `go.mod` and `go.sum` from PromptKit v0.3.0 to v0.5.0 and run
`gitea.maximumdirect.net/eric/promptkit v0.1.0`, then run `go mod tidy` so `go mod tidy`.
`go.mod` and `go.sum` contain no obsolete Scriptorium module entries. - Change the PromptKit built-in profile-catalog marker in
- In `internal/framework/llm`, replace imports of `internal/framework/llm/promptkit_profile_fingerprint.go` to identify
`gitea.maximumdirect.net/eric/scriptorium` with PromptKit and rename: v0.5.0. This deliberately invalidates LLM checkpoints tied to the prior
- `scriptorium_client.go` to `promptkit_client.go`; catalog identity.
- `ScriptoriumClientConfig` to `PromptKitClientConfig`; - Review PromptKit-facing compile errors or test failures against the v0.4.0
- `ScriptoriumClient` to `PromptKitClient`; and v0.5.0 release guides. Do not adopt prepared execution, inspection, or
- `NewScriptoriumClient` to `NewPromptKitClient`; fallback profiles in this stage.
- `AssetRegistry.ScriptoriumOptions` to - Replace the existing test assertion for one exact built-in fingerprint hash
`AssetRegistry.PromptKitOptions`; and with durable assertions that the fingerprint is deterministic, non-empty,
- Scriptorium-named private input, variable, metadata, debug, response, non-secret, and changes when a semantic profile source changes. Do not add a
validation, and error-redaction helpers to PromptKit terminology. new version-constant or exact-hash change detector.
- Do not retain aliases for the old types, constructor, method, filenames, or - Update `docs/integrations/pkg-promptkit.md` to pin and link v0.5.0 and state
private helpers. the implemented dependency-level behavior: unset optional sampling controls
- Preserve the existing adapter sequence: are provider defaults. Do not document later stages as implemented.
1. validate the transport-neutral request and output target; - Update any other canonical text that explicitly claims the dependency is
2. map inputs, variables, metadata, prompt identity, profile identity, and v0.3.0, but defer descriptions of unimplemented v0.5 APIs.
session prompt variable into a PromptKit `RunRequest`;
3. call `Prepare` to capture rendered debug material;
4. call `Run`;
5. translate PromptKit validation failure into
`contracts.ErrInvalidStructuredOutput`;
6. decode the structured artifact into the caller's target; and
7. return transport-neutral response, usage, profile, and debug data.
- Continue wrapping and redacting upstream errors at the same trust boundary.
Update safe wrapper text from Scriptorium to PromptKit without exposing
prompt content, credentials, artifact content, filesystem paths, or raw
upstream payloads.
- Rename the adapter provenance constant and change its value from
`scriptorium` to `promptkit`.
- Preserve PromptKit's cancellation and timeout semantics. Do not introduce
another timeout wrapper in Notarius.
- Update the production composition root in `internal/cli/catalog.go` to
construct `PromptKitClientConfig` and `NewPromptKitClient`.
### Tests ### Tests And Validation
- Rename and adapt the existing adapter and asset-registry tests; do not add - `go test ./internal/framework/llm ./internal/cli`
tests whose only purpose is to enforce private symbol or filename choices. - `go test ./...`
- Through the adapter's stable behavior, retain coverage for: - `go vet ./...`
- engine and asset construction failures; - `go build ./cmd/notarius`
- prompt, schema, input, variable, metadata, and profile forwarding; - `rg -n 'promptkit v0\.3\.0|promptkit@v0\.3\.0|PromptKit v0\.3\.0' .`
- structured-output success and validation failure; - `git diff --check`
- malformed or empty generated output;
- caller cancellation and configured timeout behavior;
- credential and content redaction;
- raw response and prepared-prompt debug capture;
- token usage, including cached and cache-write token fields; and
- profile manifest recording.
- Use injected PromptKit clients or local deterministic HTTP test servers.
Default tests must remain offline and must not require credentials or paid
model calls.
### Completion Criteria ### Completion Criteria
- The framework and production CLI compile against PromptKit only. - The repository directly pins v0.5.0 and all default offline checks pass.
- No Go import of the Scriptorium module remains. - The profile-source fingerprint identifies the new upstream catalog without a
- Existing adapter behavior is preserved except that response and manifest brittle literal-hash test.
provenance now identify `promptkit`. - Current documentation no longer identifies v0.3.0 as the supported version.
## Stage 2: Introduce Version 4 PromptKit Configuration ## Stage 2: Execute One Frozen Prepared Snapshot
Make the user-visible configuration terminology agree with the dependency and ### Goal
adapter.
### Implementation Make Notarius debug details and generation use one exact PromptKit preparation.
- Change `SupportedFileConfigVersion` from 3 to 4. ### Work
- In `internal/core/config`, rename:
- `ScriptoriumConfig` to `PromptKitConfig`;
- `FileScriptoriumConfig` to `FilePromptKitConfig`;
- `Config.Scriptorium` to `Config.PromptKit`;
- `FileConfig.Scriptorium` to `FileConfig.PromptKit`; and
- Scriptorium-named validation and application helpers to PromptKit
terminology.
- Change the runtime JSON and file YAML section name from `scriptorium` to
`promptkit`.
- Preserve the two optional fields and their meaning:
```yaml - Refactor `PromptKitClient.CompleteStructured` to call
version: 4 `PrepareExecution`, immediately defer `Discard`, obtain a caller-owned
promptkit: `Details` value, and execute with `RunPrepared`.
profile_dir: /path/to/profiles - Preserve the existing Notarius request mapping, cancellation precedence,
# profile_file: /path/to/profiles.yml validation classification, raw structured bytes, response decoding,
``` profile recording, usage reporting, and credential redaction.
- Ensure every preparation, execution, validation, empty-result, and decode
error retains useful Notarius prompt context without exposing prepared handle
state or secrets.
- Use `errors.As` to obtain `*promptkit.CapacityError` on admission rejection.
Preserve `contracts.ErrLLMCapacityExceeded` as the stable classification and
add a nonblank backend ID only to safe application-owned diagnostic context.
Do not expose `promptkit.CapacityError` outside the LLM adapter.
- Update `docs/internal/llm.md` and the implemented-mechanics portion of
`docs/integrations/pkg-promptkit.md` to describe the single frozen execution
snapshot and structured capacity adaptation.
- Preserve trimming, clone, apply, effective-configuration, and redaction ### Tests And Validation
behavior for `profile_dir` and `profile_file`.
- Preserve the rule that `profile_dir` and `profile_file` are mutually
exclusive and that an explicitly supplied value must not be empty.
- Update production client construction and explicit-profile validation to
read `cfg.PromptKit`.
- Rename `internal/cli/scriptorium_profiles.go` and its functions to PromptKit
terminology. Preserve the existing profile validation timing and
`errors.Is`-based handling of PromptKit's `ErrProfileNotFound`.
- Keep YAML decoding strict. A version 4 file containing `scriptorium` must
fail as an unknown field.
- Add a targeted version 3 migration diagnostic that instructs users to:
1. change `version: 3` to `version: 4`; and
2. rename `scriptorium:` to `promptkit:`.
Do not attempt to decode or automatically rewrite version 3 files.
- Update maintained configuration examples to version 4 and use `promptkit`
wherever profile sources are demonstrated.
### Tests - Adapt existing PromptKit client tests to the prepared-execution path.
- Retain or add one behavioral test proving that the debug prompt details match
- Update configuration contract tests to establish: the request actually passed to generation when a backing prompt source could
- a minimal version 4 file applies over defaults; otherwise change between independent preparations. Test the resulting
- explicit PromptKit profile directory and profile file values decode and snapshot consistency, not a private helper call count.
survive apply, clone, and effective configuration; - Retain capacity tests proving `errors.Is` reaches
- the two profile sources remain mutually exclusive; `contracts.ErrLLMCapacityExceeded`, the selected backend can appear in safe
- explicit empty values remain invalid; diagnostic context, and provider calls are not made after rejected
- unknown fields remain rejected by strict decoding; admission.
- version 4 rejects the removed `scriptorium` key; and - Run `go test ./internal/framework/llm` and
- version 3 produces the actionable migration classification. `go test -race ./internal/framework/llm`.
- Update CLI contract tests to demonstrate that configured and explicitly - Run `go test ./...` and `git diff --check`.
selected PromptKit profiles are validated before pipeline preparation and
that invalid profiles retain the existing process-failure behavior.
- Test configuration behavior at the parser/configuration and CLI composition
boundaries. Do not reproduce PromptKit's own profile parser test matrix.
### Completion Criteria ### Completion Criteria
- All runtime and file configuration code uses PromptKit terminology. - `CompleteStructured` no longer calls independent `Prepare` and `Run`
- Maintained examples are valid version 4 files. operations for one request.
- The obsolete section is rejected rather than silently accepted or ignored. - Debug prompt material and generation result originate from the same frozen
PromptKit snapshot.
- Capacity remains a provider-neutral Notarius error classification.
## Stage 3: Make Module Prompt-Asset Support Provider-Neutral ## Stage 3: Replace Synthetic Profile Validation With Inspection
Remove dependency-brand terminology from module-owned prompt asset ### Goal
registration without changing the assets themselves.
### Implementation Validate profiles through PromptKit's exact profile-inspection boundary and
centralize engine profile-source construction.
- Across the D&D scene chunker, extractors, NPC normalizer, and registration ### Work
support, rename each `scriptorium_assets.go` and corresponding test file to
`prompt_assets.go` and `prompt_assets_test.go`.
- Rename private helpers and values such as `scriptoriumPromptRoot`,
`scriptoriumPromptMetadata`, and equivalent schema registration names to
provider-neutral forms such as `promptAssetRoot` and
`promptAssetMetadata`.
- Retain PromptKit terminology only where code directly calls a PromptKit API,
such as producing `promptkit.Option` values at the framework asset registry
boundary.
- Do not change:
- prompt or schema contents;
- prompt, schema, or asset IDs and versions;
- embedded filesystem paths;
- message ordering or cache-control placement;
- module keys or declared prompt inputs;
- metadata values or content fingerprint algorithms; or
- extraction, normalization, or validation behavior.
### Tests - Introduce a small provider-adapter-owned profile inspection or validation
function in `internal/framework/llm`. Its public internal signature must use
Notarius-owned configuration and result/error types rather than returning
PromptKit types to the CLI.
- Share the code that applies `profile_dir`, `profile_file`, and registered
backend options between the production PromptKit engine and the inspection
engine. Preserve the mutual-exclusion and local-backend rules.
- Change CLI explicit-profile preflight to use `Engine.InspectProfile` through
that LLM boundary.
- Remove `profileCheckPromptID`, `profileCheckPromptFS`, the `testing/fstest`
production dependency, and the synthetic `Prepare` request.
- Preserve distinct, useful errors for an absent profile, invalid profile,
unknown backend registration, cancellation, and invalid profile source.
- Do not require `api_key_env` to be populated during configuration validation.
Inspection may report credential requirements internally, but actual
preparation remains responsible for credential availability before a model
call.
- Update current-behavior sections in `docs/internal/cli.md` and
`docs/internal/llm.md`. Keep field definitions in `docs/config.md`.
- Update existing module prompt-preparation and registration tests to compile ### Tests And Validation
through the renamed support code.
- Retain the centralized behavioral coverage that all registered production - Replace synthetic-prompt tests with profile inspection tests covering:
prompts and schemas can be mounted and prepared with their declared inputs. configured local backend success; missing local backend failure; absent
- Retain prompt ordering and cache-prefix contract coverage where ordering is profile; malformed profile; and an otherwise valid profile whose credential
semantically significant. environment variable is intentionally unset.
- Do not add word-presence tests, rename detectors, or private-helper tests. - Prove validation performs no provider HTTP call and remains offline.
- Run `go test ./internal/framework/llm ./internal/cli` and `go test ./...`.
- Run `git diff --check`.
### Completion Criteria ### Completion Criteria
- Module-owned code no longer describes its prompt assets as Scriptorium - No production synthetic profile-check prompt remains.
assets. - Profile validation uses the same ordinary profile source and backend
- Prompt and schema fingerprints remain unchanged from the pre-migration registrations as execution.
source content. - Configuration validation succeeds for structurally valid profiles without
reading credential values.
## Stage 4: Align Provenance and Checkpoint Behavior ## Stage 4: Add Application Fallback Profile Asset Plumbing
Make the intentional public provenance change explicit while leaving cache ### Goal
identity tied to semantic inputs.
### Implementation Allow module families to register application-owned fallback profile YAML
without placing domain policy in generic LLM code.
- Record `Provider: "promptkit"` in every newly observed ### Work
`LLMProfileManifest` and `StructuredCompletionResponse` produced by the
production adapter.
- Update manifest, debug, artifact, and assembled-run expectations that
currently identify Scriptorium.
- Do not change checkpoint schemas, layouts, or the
`CheckpointFingerprintProvider` contract.
- Do not introduce a fingerprint containing the PromptKit package name or
version. Dependency implementation identity is not a stable semantic
identity.
- Verify that no accidental prompt, schema, component, reference, profile, or
runtime-override fingerprint changes result from the provider-neutral file
and helper renames.
- If an existing test fixture contains recorded LLM provenance, update only
the adapter/provider value and preserve the profile ID, model, usage, and
remaining run data.
### Tests - Extend `internal/framework/llm.AssetRegistry` with a separate fallback
profile source collection, registration method, flattened filesystem, and
safe content digest.
- Reuse the existing asset-source path validation and flattening behavior where
appropriate. Reject invalid roots, unreadable assets, and duplicate flattened
paths. Do not parse PromptKit profile YAML in Notarius.
- Add `promptkit.WithFallbackProfileFS` to production engine options only when
at least one fallback profile source is registered.
- Supply the identical assembled fallback source to the profile-inspection
engine. Adjust CLI composition so pipeline-aware profile validation can use
the production LLM asset registry without exposing PromptKit types.
- Extend profile-source checkpoint identity to include the exact fallback
profile asset digest in addition to the PromptKit catalog marker and operator
source. Keep the resulting fingerprint hash-only and path/content/credential
free.
- Keep operator source precedence owned by PromptKit. Do not implement profile
merging or duplicate PromptKit source resolution in Notarius.
- Update `docs/internal/llm.md` only for the new implemented generic asset and
fingerprint mechanics. No domain fallback exists until Stage 5.
- At the adapter boundary, assert that a successful completion and recorded ### Tests And Validation
profile manifest report `promptkit`.
- At one assembled run boundary, confirm that the finalized manifest carries - Add focused AssetRegistry tests for successful flattening, invalid roots,
the PromptKit profile provenance observed by the client. duplicate paths, and hash changes when fallback bytes change.
- Retain existing checkpoint identity tests for semantic prompt, module, - Add adapter-level tests showing that the fallback filesystem reaches both
profile, reference, and runtime changes. Do not add a test coupled only to execution construction and inspection construction.
the absence of a dependency-version fingerprint. - Extend checkpoint tests to prove fallback content changes profile-source
identity without exposing raw YAML or paths. Use relational comparisons, not
a fixed hash literal.
- Run `go test ./internal/framework/llm ./internal/cli` and `go test ./...`.
- Run `git diff --check`.
### Completion Criteria ### Completion Criteria
- New durable and debug provenance consistently identifies PromptKit. - Generic plumbing can carry application fallback profiles while remaining
- No checkpoint wire-format or layout change has been introduced. unaware of D&D IDs or model settings.
- Inspection, execution, and checkpoint identity use the same fallback asset
source.
## Stage 5: Update Canonical Documentation ## Stage 5: Adopt The D&D `dnd-extraction` Fallback
Update documentation in the same change that implements the behavior, following ### Goal
the repository's canonical ownership rules.
### Implementation Give the D&D module family one stable embedded workload profile that operators
can replace.
- Update `docs/config.md` to own: ### Work
- configuration version 4;
- the `promptkit` section and its fields; - Add a D&D-owned embedded PromptKit profile asset with ID `dnd-extraction`
- mutual exclusion and validation rules; and under `internal/modules/dnd`. Use the exact baseline defined in
- the version 3-to-4 migration instruction. `promptkit.md`: OpenRouter, `openai/gpt-5.6-luna`, no explicit reasoning
- Update `docs/operations.md` to describe provider retries and timeouts as effort, a 240-second timeout, flex service tier, and no selected temperature,
PromptKit profile behavior and accurately summarize the timeout layers. token limit, or `top_p`. The omitted reasoning value intentionally allows
- Update `docs/internal/llm.md` to describe: OpenAI's backend to apply its `medium` default.
- the PromptKit-backed adapter; - Register the profile filesystem from the D&D registrar through the generic
- transport-neutral request and response mapping; fallback profile asset boundary. Keep D&D policy out of
- asset mounting; `internal/framework/llm` and the CLI composition root.
- preparation, execution, validation, redaction, and debug behavior; - Change every maintained D&D LLM prompt definition—including scene chunking,
- profile provenance; and all D&D extractors, and NPC normalization—from the model-named default to
- caller, generation, and transport timeout ownership. `default_profile: dnd-extraction`.
- Update `docs/internal/cli.md` to describe PromptKit profile validation and - Add an integration-level profile-resolution test proving that:
production client composition. - the fallback resolves when no operator source defines the ID;
- Update `docs/development.md` so its task-reading guide refers to the - a valid operator profile with the same ID wins completely; and
PromptKit integration. - an invalid matching operator profile fails rather than falling through.
- Update any maintained examples and nearby navigation links. Configuration - Test through Notarius's assembled production assets and PromptKit boundary;
definitions and defaults remain canonical in `docs/config.md`; other do not duplicate every upstream source-precedence case.
documents should summarize and link rather than repeat them. - Update the implemented profile ownership and prompt-default behavior in
- Replace the copied `docs/integrations/pkg-promptkit.md` guide with a concise `docs/internal/dnd.md`, `docs/internal/llm.md`, and
Notarius-owned upstream boundary document. It must: `docs/integrations/pkg-promptkit.md`. Defer the complete operator walkthrough
- identify the pinned PromptKit package and supported public boundary used and examples to Stage 10.
by Notarius;
- link to PromptKit's canonical upstream package and format documentation; ### Tests And Validation
- describe only integration facts that Notarius relies upon;
- point to `docs/internal/llm.md` for Notarius implementation behavior; and - Run focused D&D prompt preparation tests and the production composition
- contain no copied relative links that resolve only inside the PromptKit tests.
source repository. - Run `go test ./internal/modules/dnd/... ./internal/framework/llm
- Update `docs/roadmap/future.md` to replace Scriptorium terminology in the ./internal/cli`.
native-session item with PromptKit. Keep the work deferred and state - Run `go test ./...`.
accurately that PromptKit `v0.1.0` does not yet provide the desired direct - Verify `rg -n 'default_profile: gemini-2-flash' internal/modules/dnd`
request-level session field. Do not imply that existing prompt-variable returns no matches.
propagation is native provider session support. - Run `git diff --check`.
- Search all maintained Go, Markdown, YAML, JSON, and module files for
remaining `Scriptorium` or `scriptorium` occurrences. Retain the old name
only where necessary to explain the one-time version 3 migration or
historical context.
- Validate affected relative documentation links.
### Completion Criteria ### Completion Criteria
- Current-behavior documentation outside `docs/roadmap` describes the - All maintained D&D prompts use the application-owned logical profile ID.
implemented PromptKit integration only. - The fallback works without an operator profile and remains authoritatively
- Configuration and operational facts have one canonical owner. overridable by a matching valid operator definition.
- The integration document is Notarius-specific and does not duplicate or
impersonate upstream package documentation.
## Stage 6: Repository-Wide Verification ## Stage 6: Introduce Module Execution-Class Metadata
Complete the migration with focused and repository-wide validation. ### Goal
### Required Checks Make each production module's ability to use an LLM statically discoverable
without yet changing profile inheritance.
Run: ### Work
- Add `ExecutionClass contracts.ExecutionClass` to `pipeline.ModuleSpec` and
preserve it through normalization, cloning, catalogs, registries, JSON/debug
views, and lookup helpers.
- In this transitional stage only, allow an omitted execution class to
normalize to deterministic so existing test-only fixtures can be migrated in
Stage 7 without breaking the repository midway.
- Explicitly classify every production module:
- D&D scene chunking, every D&D extractor, and D&D NPC normalization as
`llm_backed`;
- all other current production input, chunk, merge, normalize, and output
modules as `deterministic`.
- Update production module specification tests and production catalog tests to
assert the semantic class alongside stage, artifact kind, and capabilities.
- Add catalog lookup support needed by later resolution to retrieve a selected
module's execution class by stage and key without constructing it.
- Do not implement pipeline-level profile inheritance or reject deterministic
profiles yet.
- Update `docs/internal/modules.md` and `docs/internal/dnd.md` to identify
execution class as registered module metadata, while noting only implemented
uses.
### Tests And Validation
- Run module registration/spec tests across generic, Seriatim, and D&D
families.
- Run `go test ./internal/framework/pipeline ./internal/modules/...`.
- Run `go test ./...` and `git diff --check`.
### Completion Criteria
- Every production module has an explicit correct execution class.
- Catalog consumers can retrieve that class without a concrete module
instance.
- Test-only omitted classes remain the only temporary compatibility behavior.
## Stage 7: Enforce Execution Metadata And Remove Runtime Probing
### Goal
Finish the execution-class contract so missing metadata cannot cause future
profile drift.
### Work
- Update every framework, CLI, and integration test module specification to
declare an explicit execution class appropriate to the fake behavior.
- Change module-spec validation so an empty or unsupported execution class is a
registration error. Remove the transitional deterministic default from
Stage 6.
- Replace the chunk runner's special `ChunkExecutionClassProvider` probe with
specification-derived behavior. Remove the now-redundant provider interface,
implementation methods, and tests when they have no remaining consumer.
- Ensure chunk producer provenance remains unchanged: it records a non-empty
effective binding profile for an LLM-backed chunker, while a deterministic
chunker records no profile. A profile selected only through the prompt
default remains represented by PromptKit's actual-profile manifest rather
than being invented as an explicit chunk binding.
- Review helper constructors and fixtures for opportunities to set execution
class once without obscuring the class under test. Do not introduce an
elaborate test-spec framework.
- Update internal documentation if the removal changes any described runtime
mechanics.
### Tests And Validation
- Add or retain focused registration tests for missing and invalid execution
classes.
- Retain chunk-plan provenance tests for LLM-backed and deterministic
chunkers.
- Run `go test ./internal/framework/pipeline ./internal/modules/...`.
- Run `go test ./...`, `go vet ./...`, and `git diff --check`.
### Completion Criteria
- No registered module specification relies on an implicit execution class.
- Pipeline metadata, not a concrete runtime type assertion, owns module
execution classification.
## Stage 8: Resolve Programmatic Pipeline Profile Defaults
### Goal
Implement profile inheritance and precedence inside the pipeline resolver
before exposing the field through YAML configuration.
### Work
- Add an optional trimmed `LLMProfile` field to
`pipeline.PipelineProfile`. Add a non-empty runtime override field to
`pipeline.ResolveOptions` so all precedence decisions occur in the resolver
rather than through pre-resolution mutation.
- After module selection, `--only` filtering, default validator-chain
selection, and validator compatibility resolution, apply effective profiles
to every selected input, chunk, extract, merge, normalize, output, and
validator binding according to the precedence in `promptkit.md`.
- Apply profiles only when the selected module or validator execution class is
`llm_backed`.
- Reject a binding-specific `llm_profile` on any deterministic module or
validator. Do not reject or inspect an unused pipeline default when no
selected LLM-backed binding consumes it.
- Leave an LLM-backed binding empty when no CLI, binding, or pipeline profile is
selected so PromptKit can use the prompt's `default_profile`.
- Store the effective values on resolved bindings before digest construction.
Do not add a second inheritance decision to execution.
- Ensure semantically equivalent repeated binding profiles and one inherited
default produce the same resolved pipeline digest. Ensure any changed
effective profile changes the digest.
- Do not modify file configuration or CLI parsing in this stage.
### Tests And Validation
- Add pipeline package tests for the complete precedence matrix:
runtime override; binding-specific exception; pipeline default; prompt
fallback; and deterministic bindings.
- Cover default and explicitly configured validator chains, all relevant stage
categories, `--only` lane selection, unused defaults, deterministic-profile
rejection, and semantic digest equivalence.
- Prefer table-driven package-level tests over assertions on private traversal
helpers.
- Run `go test ./internal/framework/pipeline` and `go test ./...`.
- Run `git diff --check`.
### Completion Criteria
- Programmatic pipelines resolve one canonical effective profile policy.
- Only LLM-backed resolved bindings can contain a profile.
- Runtime override, binding, pipeline, and prompt precedence is unambiguous and
digest-stable.
## Stage 9: Expose Pipeline Defaults Through Configuration And CLI
### Goal
Make the profile-default workflow available to operators while preserving
validation and override behavior.
### Work
- Add optional `pipelines.<id>.llm_profile` support to the version 4 file
configuration model. Use presence-aware decoding so an explicitly set blank
value is rejected, while omission remains valid.
- Preserve the field through file application, configuration cloning,
effective configuration, and programmatic profile copies without aliasing or
trimming drift.
- Remove `applyLLMProfileOverride`. Pass the CLI override through the resolver's
runtime-override input so deterministic bindings are never populated.
- Update effective profile-ID collection to cover every selected LLM-backed
module stage and LLM-backed validator, including future LLM-backed input and
output modules. Do not inspect deterministic or unselected profiles.
- Ensure `run`, `config validate --pipeline`, resume/checkpoint identity, and
relevant dry preflight paths all use the same resolved effective profiles.
- Preserve `--llm-profile` as the highest-precedence non-empty run-wide
override and preserve binding-specific profiles as exceptions when no CLI
override is present.
- Do not increment the configuration version.
- Update current configuration and CLI contracts in `docs/config.md` and
`docs/cli.md` in the same stage. Link to operations for the deployment
workflow rather than duplicating it prematurely.
### Tests And Validation
- Add file-config tests for omission, trimming, explicit blank rejection,
unknown-key behavior, cloning, and round-trip application.
- Add effective-config and CLI contract tests for precedence, LLM-only
application, inherited-profile inspection failure before factory execution,
`--only`, and digest changes.
- Retain offline operation and do not require credentials for
`config validate --pipeline`.
- Run `go test ./internal/core/config ./internal/framework/pipeline
./internal/cli`.
- Run `go test ./...`, `go vet ./...`, and `git diff --check`.
### Completion Criteria
- Operators can select `dnd-extraction` once per pipeline.
- Configuration and CLI paths share the resolver's precedence policy.
- Unknown effective profiles fail preflight, while deterministic and unused
profiles do not cause spurious inspection.
## Stage 10: Complete Operator Documentation, Examples, And Decision Record
### Goal
Make the implemented workflow understandable, copyable, and maintainable
without duplicating canonical facts.
### Work
- Create an ADR using the next sequential number for the durable decision to
use workload-oriented pipeline defaults with operator-overridable application
fallback profiles. Record context, decision, alternatives, and consequences;
do not turn the ADR into a field reference or implementation log.
- Complete `docs/config.md` as the canonical owner of profile-source fields,
`pipelines.<id>.llm_profile`, validation, and precedence.
- Complete `docs/operations.md` with an operator workflow that distinguishes
Notarius embedded prompts, Notarius fallback profiles, PromptKit built-ins,
and deployment filesystem profiles. Include production/development/local use
of the same `dnd-extraction` ID, credential handling, absolute-path guidance,
and the fact that current relative profile paths use the process working
directory rather than the configuration file's directory.
- Complete `docs/integrations/pkg-promptkit.md` with the v0.5.0 boundary,
prepared execution, inspection, fallback and ordinary source precedence,
optional provider controls, capacity adaptation, and compatibility policy.
- Update `docs/internal/configuration.md`, `docs/internal/pipeline.md`,
`docs/internal/cli.md`, `docs/internal/llm.md`, `docs/internal/modules.md`, and
`docs/internal/dnd.md` only for their owned implementation details. Link to
canonical configuration, operations, and upstream format contracts rather
than restating them.
- Keep exactly the existing two D&D configuration examples. Add
`llm_profile: dnd-extraction` to the minimal and complete pipelines and remove
the now-redundant model-named binding override from the complete example.
- Add one secret-free maintained operator profile at
`examples/profiles/dnd-extraction.yml`. It should be a complete valid profile
for the same logical ID and may mirror the embedded baseline; its purpose is
to demonstrate file ownership and format, not claim automatic environment
detection. Link it from the configuration and operations documentation.
- If the complete example selects the external profile file, use a path that
is valid for the documented repository-root invocation and explicitly note
the working-directory rule. Keep the minimal example dependent only on the
embedded fallback.
- Add or extend maintained-example validation so both configuration examples
and the profile YAML are checked without generation or credentials.
- Remove the now-implemented `Pipeline-Level LLM Profile Defaults` section from
`docs/roadmap/future.md`. Preserve the unrelated deterministic session and
concurrency items.
- Do not delete `promptkit.md` or this implementation plan during the feature
implementation; retire them only after post-implementation review.
### Tests And Validation
- Run maintained example/configuration tests and relevant CLI help/parser
tests.
- Run `go test ./...`.
- Run `rg -n 'gemini-2-flash' examples docs` and review every remaining match
for intentional model-policy or historical context.
- Run `rg -n 'v0\.3\.0|profileCheckPrompt|applyLLMProfileOverride' .` and resolve
stale production or current-documentation matches.
- Verify all new links and `git diff --check`.
### Completion Criteria
- Every current fact has one canonical documentation owner.
- Operators can distinguish and deploy all profile layers without reading Go
source.
- Both maintained configurations and the maintained external profile are valid,
secret-free, and tested offline.
- Implemented profile work no longer remains in `future.md`.
## Stage 11: Final Verification And Quality Review
### Goal
Verify the complete migration as one integrated change and correct only defects
or omissions found during that review.
### Work
- Review the final diff against every acceptance criterion in `promptkit.md`.
- Confirm provider-specific PromptKit types remain inside the LLM integration
boundary and D&D policy remains inside the D&D module family.
- Confirm execution and inspection receive identical ordinary, fallback, and
backend configuration.
- Confirm no paths, profile YAML, endpoints, credentials, or prepared handle
state leak into fingerprints or ordinary diagnostics.
- Confirm all production module specs have explicit correct execution classes
and every resolved deterministic binding is profile-free.
- Confirm prompt default, pipeline default, binding override, and CLI override
behavior through representative assembled configurations.
- Review tests for redundancy and remove obsolete synthetic-prompt,
runtime-probe, exact-hash, or duplicated upstream-behavior tests superseded by
stronger contract tests.
- Perform an optional manual D&D quality comparison if credentials and an
evaluation transcript are deliberately supplied. Record no private input or
credential material, and do not make this comparison a completion gate.
### Validation Commands
```sh ```sh
git diff --check gofmt -w <changed-go-files>
go mod tidy
go test ./... go test ./...
go test -race ./internal/framework/llm ./internal/core/config ./internal/framework/pipeline ./internal/cli
go vet ./... go vet ./...
go build ./cmd/notarius go build ./cmd/notarius
go test -race ./internal/framework/llm ./internal/cli ./internal/modules/dnd/... git diff --check
``` ```
Also verify: Also run focused stale-contract searches:
- `go.mod` and `go.sum` contain PromptKit and no Scriptorium dependency; ```sh
- the repository contains no obsolete Scriptorium Go identifiers or imports; rg -n 'gitea.maximumdirect.net/eric/promptkit v0\.3\.0|PromptKit v0\.3\.0' .
- any remaining textual use of Scriptorium is limited to intentional migration rg -n 'default_profile: gemini-2-flash|profileCheckPrompt|applyLLMProfileOverride' internal docs examples
or historical explanation; ```
- maintained configuration examples parse successfully as version 4;
- tests remain deterministic, offline, and independent of real credentials; Review any matches rather than deleting intentional historical references
- no prompt, schema, artifact, or checkpoint format changed unintentionally; blindly.
and
- no secrets, prompt payloads, source content, or private infrastructure paths ### Completion Criteria
were introduced into code, errors, tests, or documentation.
- All automated checks pass offline and without real credentials.
- The implemented behavior matches `promptkit.md` with no known architecture,
provenance, checkpoint, profile-precedence, or documentation gap.
- Any optional live evaluation is clearly separate from correctness testing.
## Open Questions ## Open Questions
None. The dependency version, compatibility policy, configuration migration, None. The roadmap decisions are sufficient to implement every stage without an
internal naming, provenance value, checkpoint treatment, session scope, additional product or architecture choice.
documentation ownership, and verification boundaries are decided above.

318
docs/roadmap/promptkit.md Normal file
View File

@@ -0,0 +1,318 @@
# PromptKit v0.5 Integration And LLM Profile Policy
## Purpose
This roadmap defines the target state for upgrading Notarius from PromptKit
v0.3.0 to v0.5.0 and adopting the upstream runtime and profile facilities that
directly improve Notarius. It also defines the application policy for stable,
domain-oriented LLM profile names, operator overrides, pipeline inheritance,
profile validation, provider defaults, checkpoint identity, and documentation.
The ordered work needed to reach this state belongs in
[the implementation plan](implementation.md). Current behavior remains defined
by the canonical documentation outside `docs/roadmap/` until the corresponding
work is implemented.
## Background
Notarius currently pins PromptKit v0.3.0. Its adapter prepares a request once
for debug material and then independently runs the original request, causing
PromptKit to prepare the same logical call a second time. The CLI validates an
explicit profile by preparing a synthetic prompt. PromptKit profile selection
can be repeated on individual module bindings or replaced for one invocation
with `--llm-profile`, but a configured pipeline cannot yet declare one inherited
profile policy.
PromptKit v0.4.0 and v0.5.0 add the upstream boundaries needed to improve these
areas:
- [v0.4.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.4.0.md)
adds opaque prepared executions, exact profile and prompt inspection, and a
typed backend-capacity error;
- [v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.5.0.md)
adds application fallback profile filesystems and stops sending unset
optional sampling controls as framework-selected provider values; and
- the [v0.5.0 format contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
defines the resulting profile-source and execution-setting precedence.
A source-compatibility test of the current Notarius repository against
PromptKit v0.5.0 completed successfully. The work is therefore primarily an
intentional runtime and configuration migration rather than a repair for a
breaking Go API change.
## Goals
- Pin and document PromptKit v0.5.0 as Notarius's supported upstream contract.
- Execute the exact prepared request snapshot whose safe details are recorded
in Notarius debug material.
- Validate configured PromptKit profiles through the upstream inspection API
without synthetic prompts, provider calls, or credential-value access.
- Give Notarius an application-owned, operator-overridable
`dnd-extraction` profile fallback.
- Let a pipeline choose one default LLM profile without repeating that ID on
every LLM-backed binding.
- Apply profile inheritance and run-wide overrides only where the resolved
module or validator can use an LLM.
- Preserve accurate checkpoint invalidation, effective profile provenance,
redaction, cancellation, concurrency, and provider-neutral module contracts.
- Provide operators with one clear deployment pattern for production,
development, and local profile definitions.
## Target End State
### PromptKit Runtime Boundary
Notarius depends on PromptKit v0.5.0 and uses its public APIs rather than
reimplementing source or execution resolution.
For each structured completion, the adapter:
1. builds one PromptKit run request from the provider-neutral Notarius request;
2. calls `PrepareExecution` once;
3. immediately arranges an idempotent `Discard` for every unexecuted handle;
4. obtains credential-redacted `Details` for debug and response metadata; and
5. calls `RunPrepared` so generation uses that exact frozen snapshot.
The debug prompt and successful result therefore describe the same selected
profile, rendered messages, input bytes, session, output contract, and effective
settings even when a filesystem-backed source changes concurrently. PromptKit
handle types remain private to `internal/framework/llm`.
PromptKit admission failures continue to match Notarius's provider-neutral
`ErrLLMCapacityExceeded` contract. When PromptKit supplies a `CapacityError`,
the adapter obtains the normalized backend ID through `errors.As` and may add it
to safe application-owned diagnostics without parsing upstream error wording.
The backend ID does not become a provider-specific module contract.
### Optional Provider Controls
Notarius accepts PromptKit v0.5.0's new behavior for `temperature`,
`max_tokens`, and `top_p`: an unset setting is omitted from compatible provider
requests and the provider chooses its own default. Notarius does not restore
PromptKit's former implicit `top_p: 1` value globally.
An operator who requires a particular value specifies it in the selected
PromptKit profile. The application fallback described below intentionally
leaves these controls unset. A human-reviewed D&D extraction comparison should
be performed after the upgrade, but paid or nondeterministic model output is
not part of the default automated test suite.
### Profile Inspection
Pipeline-aware configuration validation uses `Engine.InspectProfile` for every
effective explicit profile ID. It verifies that the profile exists, parses and
validates, resolves its backend and target, and is compatible with the engine's
registered backends. It does not create a synthetic prompt, load prompt inputs,
contact a provider, or require credential values to exist in the validation
process environment.
Credential availability is execution-time state. PromptKit preparation still
enforces the selected profile's credential contract before generation. This
keeps `notarius config validate` useful in build and deployment validation
environments where secrets are deliberately absent.
PromptKit construction for inspection and execution uses one shared internal
profile-source and backend-option path. The CLI does not expose PromptKit public
types across the Notarius LLM boundary merely to perform inspection.
`InspectPrompt` is not adopted merely because it exists. It remains available
for a later, separately defined module-to-prompt interface preflight if a
concrete validation requirement justifies that additional contract.
### Application And Operator Profile Sources
Notarius embeds one ordinary PromptKit YAML profile with the stable ID
`dnd-extraction`. It is an application fallback registered through
`WithFallbackProfileFS`, is owned by the D&D module family, and initially
preserves the current effective D&D baseline:
- backend: PromptKit's built-in `openrouter` backend;
- model: `openai/gpt-5.6-luna`;
- reasoning effort: unset, allowing OpenAI's backend to apply its default of
`medium`;
- generation timeout: 240 seconds;
- service tier: `flex`; and
- no application-selected `temperature`, `max_tokens`, or `top_p`.
All maintained D&D LLM prompt definitions use `dnd-extraction` as their
`default_profile`. The ID communicates workload intent rather than a provider,
model, or environment. Changing the embedded fallback is an intentional
Notarius execution-policy change and participates in checkpoint identity.
Effective profile definitions resolve in PromptKit's order:
1. programmatic in-memory profiles used by tests or explicit consumers;
2. the operator source configured by `promptkit.profile_file` or
`promptkit.profile_dir`;
3. the Notarius application fallback source; and
4. PromptKit's embedded built-in catalog.
Only an absent ID falls through to the next source. A matching profile is a
complete definition: fields are not merged with a lower-precedence definition,
and a malformed matching operator profile fails rather than silently selecting
the application fallback.
Production, development, and local deployments should normally provide
different complete definitions for the same `dnd-extraction` ID. An operator
source is optional because the application fallback keeps the maintained D&D
workflow usable, but a deployment that needs an intentional model or backend
policy should configure its own definition.
### Domain Ownership And Asset Assembly
The D&D fallback profile remains under `internal/modules/dnd` and is registered
by the D&D registrar, consistent with ADR-0004. Generic LLM plumbing knows how
to collect and flatten application fallback profile filesystems but contains no
D&D model or policy knowledge.
The shared asset registry detects invalid roots, unreadable sources, and
duplicate flattened paths. PromptKit remains responsible for strict profile
YAML parsing, duplicate profile-ID detection, source precedence, and effective
target resolution. The same assembled fallback source is supplied to runtime
execution and CLI profile inspection.
### Explicit Module Execution Metadata
Every registered input, chunk, extract, merge, normalize, and output module
declares one required execution class: `deterministic` or `llm_backed`.
Validator registrations continue to declare the same distinction through their
validator specifications.
The registered specification is authoritative for configuration resolution.
Current production classifications are:
- the D&D scene chunker, all D&D extractors, and the D&D NPC normalizer are
LLM-backed;
- the Seriatim input adapter, generic chunker, all current mergers, all other
current normalizers, and the JSON output encoder are deterministic; and
- current validators retain their declared classifications.
Missing or unsupported execution metadata is a registration error. Explicitly
assigning `llm_profile` to a deterministic module or validator is a pipeline
resolution error. The framework does not infer execution class by inspecting
domain package names or concrete implementation types at runtime.
The module specification replaces the chunk runner's special runtime
execution-class probe. Effective resolved bindings already express the result:
only LLM-backed bindings may retain a non-empty profile.
### Pipeline-Level Profile Default
Configuration version 4 gains one optional non-empty pipeline field:
```yaml
pipelines:
dnd-session:
llm_profile: dnd-extraction
```
No configuration-version increment is required because the field is additive
and existing files remain valid. An explicitly present blank value is invalid.
For every selected LLM-backed module and validator, the effective profile uses
this precedence:
1. non-empty run-wide `--llm-profile` override;
2. binding-specific `llm_profile`;
3. pipeline-level `llm_profile`; and
4. the prompt definition's `default_profile`, represented by an empty effective
Notarius binding profile.
The run-wide override and inherited pipeline default never attach to a
deterministic binding. Binding-specific exceptions remain available when one
operation needs a different cost, latency, quality, backend, or reasoning
policy.
Inheritance is resolved after module and validator selection, including
`--only` lane filtering, but before effective-pipeline validation, digest
construction, explicit-profile inspection, checkpoint construction,
preparation, execution, or provenance capture. Only profiles used by selected
LLM-backed bindings are inspected. An unused pipeline default in a pipeline
with no selected LLM-backed work does not require an otherwise unused profile
to exist.
The resolved pipeline contains effective binding profiles rather than a second
runtime inheritance mechanism. Two pipelines that differ only by spelling the
same effective policy once as a pipeline default and once on every LLM-backed
binding have the same semantic resolved digest. Changing an effective profile
changes the digest and applicable checkpoint identity.
### Provenance And Checkpoints
The PromptKit profile-source checkpoint fingerprint covers:
- the PromptKit v0.5.0 built-in profile catalog identity;
- exact application fallback profile asset content; and
- exact configured operator profile YAML content, when present.
The existing local-backend target fingerprint remains separate and continues
to exclude scheduling-only concurrency limits. Fingerprints contain hashes and
stable markers, not profile contents, filesystem paths, endpoints, credentials,
or other secrets.
Changing the PromptKit version, application fallback, operator profile, or
effective pipeline profile makes incompatible LLM checkpoints ineligible for
reuse. The dependency upgrade is expected to invalidate checkpoints produced
under v0.3.0.
Successful run manifests continue to record only profiles actually selected by
PromptKit, including their effective model, backend, and reasoning metadata.
Debug output reports the same effective execution snapshot used for generation.
### Operator Documentation And Examples
Canonical documentation clearly distinguishes:
- Notarius prompt and schema assets embedded in the application;
- Notarius application fallback profiles embedded in the application;
- PromptKit's own embedded built-in profiles; and
- operator profile files on the deployment filesystem.
The configuration reference owns the pipeline field, profile-source fields,
validation rules, and precedence. Operations owns deployment layout, working
directory behavior, credentials, and environment-specific profile management.
The PromptKit integration document owns the pinned upstream contract and
source-precedence boundary. Internal documents describe asset registration,
resolution, inspection, prepared execution, fingerprinting, and tests without
duplicating user-facing field definitions.
The maintained examples continue to include only the minimal and complete D&D
configurations. They use the stable `dnd-extraction` policy, and one maintained
PromptKit profile file under `examples/` demonstrates an operator override.
Examples remain secret-free and are validated without live provider calls.
## Out Of Scope
- Implementing the separate deterministic prompt-session identity roadmap
item.
- Changing the default `concurrency.total_llm` value; PromptKit's retained
OpenRouter capacity of 16 remains relevant to that separate item.
- Adding model evaluation as a deterministic or CI correctness gate.
- Automatically selecting production, development, or local environments.
Deployment configuration chooses the operator profile source.
- Profile inheritance, partial profile merging, or cross-profile aliases.
- Exposing PromptKit types to modules, validators, durable output contracts, or
public configuration structures.
- Adopting `InspectPrompt` without a separately justified prompt-interface
validation contract.
## Acceptance Criteria
- Notarius builds and its offline test suite passes with PromptKit v0.5.0.
- Every structured completion executes the exact snapshot used for safe debug
prompt details.
- Profile preflight uses profile inspection and no synthetic prompt.
- The embedded `dnd-extraction` fallback resolves without an operator source,
and a matching valid operator profile replaces it completely.
- Every production module has explicit, correct execution metadata.
- Pipeline, binding, CLI, and prompt-default precedence behaves as defined for
modules and validators, while deterministic bindings remain profile-free.
- Effective profiles participate in pipeline digests, profile inspection,
checkpoint identity, debug records, and run provenance at the appropriate
boundaries.
- The dependency and application fallback changes invalidate incompatible old
checkpoints without exposing profile or credential content.
- Canonical documentation and maintained examples accurately describe and
exercise the implemented operator workflow.
- Default tests remain deterministic, offline, credential-free, and focused on
Notarius-owned behavior rather than duplicating PromptKit's upstream suite.

View File

@@ -1,4 +1,6 @@
version: 4 version: 4
promptkit:
profile_file: ./examples/profiles/dnd-extraction.yml
concurrency: concurrency:
total_llm: 2 total_llm: 2
stage_workers: stage_workers:
@@ -16,6 +18,7 @@ debug:
directory: ./notarius-debug directory: ./notarius-debug
pipelines: pipelines:
dnd-session: dnd-session:
llm_profile: dnd-extraction
input: seriatim input: seriatim
# Stable campaign context is shared by every module that accepts these slots. # Stable campaign context is shared by every module that accepts these slots.
references: references:
@@ -54,7 +57,6 @@ pipelines:
merge: appendorder merge: appendorder
normalize: normalize:
module: dnd/npcs module: dnd/npcs
llm_profile: gemini-2-flash
retries: 2 retries: 2
scene-descriptions: scene-descriptions:
extract: extract:

View File

@@ -1,6 +1,7 @@
version: 4 version: 4
pipelines: pipelines:
dnd-session: dnd-session:
llm_profile: dnd-extraction
input: seriatim input: seriatim
artifacts: artifacts:
spells: spells:

View File

@@ -0,0 +1,5 @@
id: dnd-extraction
backend: openrouter
model: openai/gpt-5.6-luna
timeout_seconds: 240
service_tier: flex

2
go.mod
View File

@@ -3,7 +3,7 @@ module gitea.maximumdirect.net/eric/notarius
go 1.25.5 go 1.25.5
require ( require (
gitea.maximumdirect.net/eric/promptkit v0.1.0 gitea.maximumdirect.net/eric/promptkit v0.5.0
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
gopkg.in/yaml.v3 v3.0.1 gopkg.in/yaml.v3 v3.0.1
) )

4
go.sum
View File

@@ -1,5 +1,5 @@
gitea.maximumdirect.net/eric/promptkit v0.1.0 h1:vuKeBxkiY8E54LRFbLQFjlJJCiOfMvB1++DYBCrD/ug= gitea.maximumdirect.net/eric/promptkit v0.5.0 h1:jnpazLyyNhWrB2xzwwtUkNUfktkTdkENTwuSPnKiYrc=
gitea.maximumdirect.net/eric/promptkit v0.1.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ= gitea.maximumdirect.net/eric/promptkit v0.5.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI= github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8= github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ= github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=

View File

@@ -215,6 +215,7 @@ func assembledSpellPipeline(t *testing.T, options assembledSpellPipelineOptions)
if err := pipeline.RegisterExtractor[dnd.SpellList](components.registries.Extractors, pipeline.ModuleSpec{ if err := pipeline.RegisterExtractor[dnd.SpellList](components.registries.Extractors, pipeline.ModuleSpec{
Key: assembledSpellExtractorKey, Key: assembledSpellExtractorKey,
Stage: pipeline.StageExtract, Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunks", "source.transcript"}, Requires: []string{"chunks", "source.transcript"},
Provides: []string{"dnd.spell_casts"}, Provides: []string{"dnd.spell_casts"},
ArtifactKind: dnd.SpellListKind, ArtifactKind: dnd.SpellListKind,

View File

@@ -143,24 +143,13 @@ func isEmptyRegistries(registries pipeline.Registries) bool {
registries.Outputs == nil registries.Outputs == nil
} }
func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
if err := ctx.Err(); err != nil {
return nil, nil, err
}
assets, err := productionPromptAssets()
if err != nil {
return nil, nil, err
}
return buildProductionLLMClient(ctx, cfg, profileID, assets)
}
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory { func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
return func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) { return func(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
return buildProductionLLMClient(ctx, cfg, profileID, assets) return buildProductionLLMClient(ctx, cfg, profileID, overrides, assets)
} }
} }
func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) { func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
if err := ctx.Err(); err != nil { if err := ctx.Err(); err != nil {
return nil, nil, err return nil, nil, err
} }
@@ -171,8 +160,10 @@ func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID
client, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{ client, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{
ProfileDir: cfg.PromptKit.ProfileDir, ProfileDir: cfg.PromptKit.ProfileDir,
ProfileFile: cfg.PromptKit.ProfileFile, ProfileFile: cfg.PromptKit.ProfileFile,
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
Assets: assets, Assets: assets,
Recorder: recorder, Recorder: recorder,
ReasoningEffort: overrides.ReasoningEffort,
}) })
if err != nil { if err != nil {
return nil, nil, fmt.Errorf("create PromptKit-backed LLM client: %w", err) return nil, nil, fmt.Errorf("create PromptKit-backed LLM client: %w", err)

View File

@@ -154,6 +154,23 @@ func TestConfigValidateResolvesPipelineAndChecksSelection(t *testing.T) {
} }
} }
func TestConfigValidatePipelineDefaultProfileIsOffline(t *testing.T) {
configPath := writeCommandConfigContent(t, `version: 4
pipelines:
demo:
llm_profile: dnd-extraction
input: seriatim
artifacts:
spells:
extract: dnd/spells
`)
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "demo"}, &stdout, &stderr, Options{})
if code != 0 || !strings.Contains(stdout.String(), "valid for pipeline \"demo\"") || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
}
func TestPipelinesListSortsNormalizedIDsInTextAndJSON(t *testing.T) { func TestPipelinesListSortsNormalizedIDsInTextAndJSON(t *testing.T) {
configPath := writeCommandConfig(t, " zeta ", "alpha") configPath := writeCommandConfig(t, " zeta ", "alpha")
options := commandContractOptions(t) options := commandContractOptions(t)

View File

@@ -94,6 +94,37 @@ func TestMaintainedConfigurationExampleSet(t *testing.T) {
if got := strings.Join(names, ","); got != "dnd-complete.config.yml,dnd-minimal.config.yml" { if got := strings.Join(names, ","); got != "dnd-complete.config.yml,dnd-minimal.config.yml" {
t.Fatalf("maintained configuration examples = %q, want only the minimal and complete D&D examples", got) t.Fatalf("maintained configuration examples = %q, want only the minimal and complete D&D examples", got)
} }
profileEntries, err := os.ReadDir(repositoryPath("examples", "profiles"))
if err != nil {
t.Fatal(err)
}
names = names[:0]
for _, entry := range profileEntries {
if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".yml") {
names = append(names, entry.Name())
}
}
sort.Strings(names)
if got := strings.Join(names, ","); got != "dnd-extraction.yml" {
t.Fatalf("maintained operator profiles = %q, want dnd-extraction.yml", got)
}
}
func TestMaintainedExamplesValidateEffectiveProfilesOffline(t *testing.T) {
t.Chdir(repositoryPath())
t.Setenv("OPENROUTER_API_KEY", "")
for _, example := range maintainedExampleFiles(t) {
t.Run(example.name, func(t *testing.T) {
var stdout, stderr strings.Builder
code := RunWithOptions([]string{
"config", "validate", "--config", example.path, "--pipeline", "dnd-session",
}, &stdout, &stderr, Options{})
if code != 0 || stderr.Len() != 0 || !strings.Contains(stdout.String(), `valid for pipeline "dnd-session"`) {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})
}
} }
func exampleStepLaneIDs(resolved pipeline.ResolvedPipeline) []string { func exampleStepLaneIDs(resolved pipeline.ResolvedPipeline) []string {

View File

@@ -51,7 +51,7 @@ pipelines:
options := Options{ options := Options{
Catalog: catalogFromRegistries(components.registries), Catalog: catalogFromRegistries(components.registries),
Registries: components.registries, Registries: components.registries,
LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) { LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
llmConstructed = true llmConstructed = true
return nil, nil, errors.New("LLM client must not be constructed") return nil, nil, errors.New("LLM client must not be constructed")
}, },

View File

@@ -6,6 +6,8 @@ import (
"errors" "errors"
"fmt" "fmt"
"io/fs" "io/fs"
"net/http"
"net/http/httptest"
"os" "os"
"path/filepath" "path/filepath"
"reflect" "reflect"
@@ -13,13 +15,16 @@ import (
"sort" "sort"
"strings" "strings"
"sync" "sync"
"sync/atomic"
"testing" "testing"
"testing/fstest"
"time" "time"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts" "gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config" "gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkmap" "gitea.maximumdirect.net/eric/notarius/internal/framework/chunkmap"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts" "gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline" "gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd" "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes" "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
@@ -33,6 +38,7 @@ import (
itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents" itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents"
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells" spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop" "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
"gitea.maximumdirect.net/eric/promptkit"
) )
func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) { func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
@@ -160,6 +166,35 @@ func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
assertProductionContains(t, "production prompt assets", assetNames, requiredAssets) assertProductionContains(t, "production prompt assets", assetNames, requiredAssets)
catalog := catalogFromRegistries(registries) catalog := catalogFromRegistries(registries)
for _, test := range []struct {
stage pipeline.ModuleStage
key string
want contracts.ExecutionClass
}{
{stage: pipeline.StageInput, key: "seriatim", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageChunk, key: "generic", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageChunk, key: "dnd/scenes", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageExtract, key: "dnd/spells", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageExtract, key: "dnd/npcs", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageExtract, key: "dnd/combat-turns", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageExtract, key: "dnd/item-events", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageExtract, key: "dnd/npc-interactions", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageExtract, key: "dnd/scene-descriptions", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageMerge, key: "appendorder", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageNormalize, key: "noop", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageNormalize, key: "dnd/spells", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageNormalize, key: "dnd/npcs", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageNormalize, key: "dnd/combat-turns", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageNormalize, key: "dnd/item-events", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageNormalize, key: "dnd/npc-interactions", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageNormalize, key: "dnd/scene-descriptions", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageOutput, key: "json", want: contracts.ExecutionClassDeterministic},
} {
got, ok := catalog.ExecutionClass(test.stage, test.key)
if !ok || got != test.want {
t.Fatalf("production execution class for %s/%s = %q, %t; want %q, true", test.stage, test.key, got, ok, test.want)
}
}
converted := registriesFromCatalog(catalog) converted := registriesFromCatalog(catalog)
if converted.ArtifactCodecs != registries.ArtifactCodecs || converted.ArtifactEvidence != registries.ArtifactEvidence || converted.ValidatorChains != registries.ValidatorChains { if converted.ArtifactCodecs != registries.ArtifactCodecs || converted.ArtifactEvidence != registries.ArtifactEvidence || converted.ValidatorChains != registries.ValidatorChains {
t.Fatal("catalog/registry conversion did not preserve artifact and validator registries") t.Fatal("catalog/registry conversion did not preserve artifact and validator registries")
@@ -195,6 +230,76 @@ func TestDefaultCLICompositionValidatesRepresentativeConfiguration(t *testing.T)
} }
} }
func TestProductionAssetsResolveDNDExtractionProfile(t *testing.T) {
components := productionTestComponents(t)
newEngine := func(profileFile string) (*promptkit.Engine, error) {
t.Helper()
options, err := components.assets.PromptKitOptions()
if err != nil {
return nil, err
}
if profileFile != "" {
options = append(options, promptkit.WithProfileFile(profileFile))
}
return promptkit.NewEngine(promptkit.Config{}, options...)
}
t.Run("fallback", func(t *testing.T) {
engine, err := newEngine("")
if err != nil {
t.Fatal(err)
}
inspection, err := engine.InspectProfile(context.Background(), "dnd-extraction")
if err != nil {
t.Fatalf("InspectProfile() error = %v, want fallback profile", err)
}
params := inspection.EffectiveModelParams
if params.BackendID != "openrouter" || params.Model != "openai/gpt-5.6-luna" || params.TimeoutSeconds != 240 || params.ServiceTier != "flex" {
t.Fatalf("fallback profile parameters = %#v", params)
}
if params.ReasoningEffort != "" || params.Temperature != 0 || params.MaxTokens != 0 || params.TopP != 0 {
t.Fatalf("fallback profile selected optional provider controls: %#v", params)
}
})
t.Run("valid operator profile wins", func(t *testing.T) {
profilePath := filepath.Join(t.TempDir(), "profiles.yaml")
if err := os.WriteFile(profilePath, []byte(`id: dnd-extraction
endpoint: http://operator.example.test/v1
model: operator-model
timeout_seconds: 75
`), 0o600); err != nil {
t.Fatal(err)
}
engine, err := newEngine(profilePath)
if err != nil {
t.Fatal(err)
}
inspection, err := engine.InspectProfile(context.Background(), "dnd-extraction")
if err != nil {
t.Fatalf("InspectProfile() error = %v, want operator profile", err)
}
params := inspection.EffectiveModelParams
if params.BackendID != "" || params.Endpoint != "http://operator.example.test/v1" || params.Model != "operator-model" || params.TimeoutSeconds != 75 || params.ServiceTier != "" {
t.Fatalf("operator profile parameters = %#v, want complete replacement", params)
}
})
t.Run("invalid operator profile does not fall through", func(t *testing.T) {
profilePath := filepath.Join(t.TempDir(), "profiles.yaml")
if err := os.WriteFile(profilePath, []byte("id: dnd-extraction\nendpoint: http://operator.example.test/v1\nmodel: operator-model\nunknown: value\n"), 0o600); err != nil {
t.Fatal(err)
}
engine, err := newEngine(profilePath)
if err == nil {
_, err = engine.InspectProfile(context.Background(), "dnd-extraction")
}
if err == nil {
t.Fatal("operator profile error = nil, want failure instead of fallback")
}
})
}
func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) { func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) {
components := productionTestComponents(t) components := productionTestComponents(t)
cfg := config.Default() cfg := config.Default()
@@ -321,7 +426,7 @@ func TestProductionSpellNormalizerRejectsInvalidCatalogReferencesBeforeExecution
options := Options{ options := Options{
Catalog: catalogFromRegistries(components.registries), Catalog: catalogFromRegistries(components.registries),
Registries: components.registries, Registries: components.registries,
LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) { LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
llmConstructed = true llmConstructed = true
return nil, nil, errors.New("LLM client must not be constructed") return nil, nil, errors.New("LLM client must not be constructed")
}, },
@@ -369,18 +474,9 @@ func setNormalizeSpellCatalogSource(t *testing.T, resolved *pipeline.ResolvedPip
resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings = bindings resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings = bindings
} }
func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) { func TestProductionLLMClientFactoryBuildsOfflineRuntime(t *testing.T) {
components := productionTestComponents(t) components := productionTestComponents(t)
factories := []struct { client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
name string
factory LLMClientFactory
}{
{name: "default production assets", factory: productionLLMClientFactory},
{name: "provided production assets", factory: productionLLMClientFactoryWithAssets(components.assets)},
}
for _, tt := range factories {
t.Run(tt.name, func(t *testing.T) {
client, manifests, err := tt.factory(context.Background(), config.Default(), "test-profile")
if err != nil { if err != nil {
t.Fatalf("build production LLM runtime: %v", err) t.Fatalf("build production LLM runtime: %v", err)
} }
@@ -390,10 +486,138 @@ func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) {
if len(manifests) != 0 { if len(manifests) != 0 {
t.Fatalf("eager profile manifests = %#v, want none", manifests) t.Fatalf("eager profile manifests = %#v, want none", manifests)
} }
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
if !ok {
t.Fatalf("production LLM client %T does not provide one profile-source checkpoint fingerprint", client)
}
fingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
if err != nil || len(fingerprints) != 1 {
t.Fatalf("production LLM checkpoint fingerprints = %#v, error = %v, want one profile-source identity", fingerprints, err)
}
if _, ok := client.(contracts.LLMProfileManifestProvider); !ok { if _, ok := client.(contracts.LLMProfileManifestProvider); !ok {
t.Fatalf("production LLM client %T does not provide profile manifests", client) t.Fatalf("production LLM client %T does not provide profile manifests", client)
} }
}
func TestNormalizeOptionsSharesProductionProfileAssetsWithDefaultRuntime(t *testing.T) {
opts, err := normalizeOptions(Options{
Catalog: pipeline.ModuleCatalog{Inputs: pipeline.NewInputAdapterRegistry()},
}) })
if err != nil {
t.Fatal(err)
}
if opts.promptKitAssets == nil || opts.LLMClientFactory == nil {
t.Fatalf("normalized options = %#v, want shared profile assets and default runtime factory", opts)
}
if err := validateExplicitPromptKitProfiles(context.Background(), config.Default(), []string{"dnd-extraction"}, opts.promptKitAssets); err != nil {
t.Fatalf("inspect application fallback profile: %v", err)
}
client, _, err := opts.LLMClientFactory(context.Background(), config.Default(), "dnd-extraction", LLMRuntimeOverrides{})
if err != nil {
t.Fatalf("build default runtime: %v", err)
}
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
if !ok {
t.Fatalf("default runtime client %T does not provide checkpoint fingerprints", client)
}
runtimeFingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
directClient, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{Assets: opts.promptKitAssets})
if err != nil {
t.Fatal(err)
}
inspectionFingerprints, err := directClient.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(runtimeFingerprints, inspectionFingerprints) {
t.Fatalf("runtime profile fingerprints = %#v, inspection profile fingerprints = %#v", runtimeFingerprints, inspectionFingerprints)
}
}
func TestProductionLLMClientFactoryUsesConfiguredLocalBackend(t *testing.T) {
var providerCalls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
providerCalls.Add(1)
if r.URL.Path != "/v1/chat/completions" {
t.Errorf("provider path = %q, want /v1/chat/completions", r.URL.Path)
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{
"choices": [{"message": {"role": "assistant", "content": "{\"ok\":true}"}}],
"usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7}
}`))
}))
defer server.Close()
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
if err := os.WriteFile(profilePath, []byte(`id: local-profile
backend: local
model: local-model
`), 0o600); err != nil {
t.Fatal(err)
}
assets := llm.NewAssetRegistry()
if err := assets.RegisterPromptFS(fstest.MapFS{
"production.local.yaml": {Data: []byte(`id: production.local
version: "v1"
inputs:
- name: transcript
required: true
messages:
- role: user
content: '{{ input "transcript" }}'
output:
format: json
validation_mode: json
`)},
}, "."); err != nil {
t.Fatalf("register prompt assets: %v", err)
}
cfg := config.Default()
cfg.PromptKit.ProfileFile = profilePath
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
Endpoint: server.URL + "/v1",
ConcurrencyLimit: 2,
}
client, manifests, err := productionLLMClientFactoryWithAssets(assets)(
context.Background(),
cfg,
"local-profile",
LLMRuntimeOverrides{},
)
if err != nil {
t.Fatalf("build production LLM runtime: %v", err)
}
if len(manifests) != 0 {
t.Fatalf("eager profile manifests = %#v, want none", manifests)
}
var out map[string]any
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
PromptID: "production.local",
ProfileID: "local-profile",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "text/plain", []byte("local request"), "", ""),
},
}, &out)
if err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", err)
}
if providerCalls.Load() != 1 {
t.Fatalf("provider calls = %d, want 1", providerCalls.Load())
}
provider, ok := client.(contracts.LLMProfileManifestProvider)
if !ok {
t.Fatalf("production client %T does not provide profile manifests", client)
}
recorded := provider.LLMProfileManifests()
if len(recorded) != 1 || recorded[0].BackendID != promptkit.BackendLocal {
t.Fatalf("production profile manifests = %#v, want local backend", recorded)
} }
} }
@@ -401,14 +625,15 @@ func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
t.Run("canceled context", func(t *testing.T) { t.Run("canceled context", func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
cancel() cancel()
client, manifests, err := productionLLMClientFactory(ctx, config.Default(), "test-profile") components := productionTestComponents(t)
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(ctx, config.Default(), "test-profile", LLMRuntimeOverrides{})
if !errors.Is(err, context.Canceled) || client != nil || len(manifests) != 0 { if !errors.Is(err, context.Canceled) || client != nil || len(manifests) != 0 {
t.Fatalf("client=%T manifests=%#v error=%v, want canceled construction", client, manifests, err) t.Fatalf("client=%T manifests=%#v error=%v, want canceled construction", client, manifests, err)
} }
}) })
t.Run("nil assets", func(t *testing.T) { t.Run("nil assets", func(t *testing.T) {
client, manifests, err := productionLLMClientFactoryWithAssets(nil)(context.Background(), config.Default(), "test-profile") client, manifests, err := productionLLMClientFactoryWithAssets(nil)(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
if err == nil || !strings.Contains(err.Error(), "asset registry must not be nil") || client != nil || len(manifests) != 0 { if err == nil || !strings.Contains(err.Error(), "asset registry must not be nil") || client != nil || len(manifests) != 0 {
t.Fatalf("client=%T manifests=%#v error=%v, want nil-assets failure", client, manifests, err) t.Fatalf("client=%T manifests=%#v error=%v, want nil-assets failure", client, manifests, err)
} }
@@ -418,7 +643,7 @@ func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
components := productionTestComponents(t) components := productionTestComponents(t)
cfg := config.Default() cfg := config.Default()
cfg.Concurrency.TotalLLM = 0 cfg.Concurrency.TotalLLM = 0
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), cfg, "test-profile") client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), cfg, "test-profile", LLMRuntimeOverrides{})
if err == nil || !strings.Contains(err.Error(), "create LLM scheduler") || !strings.Contains(err.Error(), "greater than zero") || client != nil || len(manifests) != 0 { if err == nil || !strings.Contains(err.Error(), "create LLM scheduler") || !strings.Contains(err.Error(), "greater than zero") || client != nil || len(manifests) != 0 {
t.Fatalf("client=%T manifests=%#v error=%v, want scheduler-construction failure", client, manifests, err) t.Fatalf("client=%T manifests=%#v error=%v, want scheduler-construction failure", client, manifests, err)
} }
@@ -651,6 +876,7 @@ func productionOptionsFromComponents(components productionComponents) Options {
Catalog: catalogFromRegistries(components.registries), Catalog: catalogFromRegistries(components.registries),
Registries: components.registries, Registries: components.registries,
LookupEnv: emptyLookup, LookupEnv: emptyLookup,
promptKitAssets: components.assets,
} }
} }
@@ -673,7 +899,7 @@ func productionRunOptions(t *testing.T, fake *productionFakeLLMClient) Options {
options.Now = func() time.Time { return time.Unix(1700000000, 0).UTC() } options.Now = func() time.Time { return time.Unix(1700000000, 0).UTC() }
options.RunIDGenerator = func(time.Time) (string, error) { return productionRunID, nil } options.RunIDGenerator = func(time.Time) (string, error) { return productionRunID, nil }
options.UserCacheDir = func() (string, error) { return "", errors.New("user cache must not be used") } options.UserCacheDir = func() (string, error) { return "", errors.New("user cache must not be used") }
options.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) { options.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
return fake, nil, nil return fake, nil, nil
} }
return options return options

View File

@@ -2,67 +2,45 @@ package cli
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"testing/fstest"
"gitea.maximumdirect.net/eric/notarius/internal/core/config" "gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/promptkit" "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
) )
const profileCheckPromptID = "notarius.profile.check" func validateExplicitPromptKitProfiles(ctx context.Context, cfg config.Config, profileIDs []string, assets *llm.AssetRegistry) error {
var profileCheckPromptFS = fstest.MapFS{
"prompts/profile-check.yaml": &fstest.MapFile{Data: []byte(`id: notarius.profile.check
version: "1.0.0"
default_profile: mistral-small-3
inputs:
- name: transcript
required: true
messages:
- role: user
content: "{{input \"transcript\"}}"
output:
format: text
validation_mode: none
repair_attempts: 0
`)},
}
func validateExplicitPromptKitProfiles(ctx context.Context, cfg config.Config, profileIDs []string) error {
if len(profileIDs) == 0 { if len(profileIDs) == 0 {
return nil return nil
} }
engine, err := newProfileValidationEngine(cfg) inspector, err := llm.NewPromptKitProfileInspector(llm.PromptKitProfileInspectorConfig{
Source: promptKitProfileSourceConfig(cfg),
Assets: assets,
})
if err != nil { if err != nil {
return fmt.Errorf("load PromptKit profiles: %w", err) return fmt.Errorf("load PromptKit profiles: %w", err)
} }
for _, profileID := range profileIDs { for _, profileID := range profileIDs {
if _, err := engine.Prepare(ctx, promptkit.RunRequest{ if _, err := inspector.InspectProfile(ctx, profileID); err != nil {
PromptID: profileCheckPromptID, return err
ProfileID: profileID,
Inputs: map[string]promptkit.ArtifactRef{
"transcript": promptkit.Inline("profile check"),
},
}); err != nil {
if errors.Is(err, promptkit.ErrProfileNotFound) {
return fmt.Errorf("PromptKit profile %q is not configured", profileID)
}
return fmt.Errorf("validate PromptKit profile %q: %w", profileID, err)
} }
} }
return nil return nil
} }
func newProfileValidationEngine(cfg config.Config) (*promptkit.Engine, error) { func promptKitProfileSourceConfig(cfg config.Config) llm.PromptKitProfileSourceConfig {
opts := []promptkit.Option{ return llm.PromptKitProfileSourceConfig{
promptkit.WithPromptFS(profileCheckPromptFS, "prompts"),
}
if cfg.PromptKit.ProfileFile != "" {
opts = append(opts, promptkit.WithProfileFile(cfg.PromptKit.ProfileFile))
}
return promptkit.NewEngine(promptkit.Config{
PromptDir: "unused",
ProfileDir: cfg.PromptKit.ProfileDir, ProfileDir: cfg.PromptKit.ProfileDir,
}, opts...) ProfileFile: cfg.PromptKit.ProfileFile,
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
}
}
func mapPromptKitLocalBackend(cfg *config.PromptKitLocalBackendConfig) *llm.PromptKitLocalBackendConfig {
if cfg == nil {
return nil
}
return &llm.PromptKitLocalBackendConfig{
Endpoint: cfg.Endpoint,
ConcurrencyLimit: cfg.ConcurrencyLimit,
}
} }

View File

@@ -0,0 +1,159 @@
package cli
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync/atomic"
"testing"
"testing/fstest"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
)
func TestExplicitPromptKitProfileValidationInspectsProfilesWithoutGeneration(t *testing.T) {
var providerCalls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
providerCalls.Add(1)
}))
defer server.Close()
writeProfile := func(t *testing.T, name, content string) string {
t.Helper()
profilePath := filepath.Join(t.TempDir(), name+".yaml")
if err := os.WriteFile(profilePath, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
return profilePath
}
localProfile := "id: local-profile\nbackend: local\nmodel: local-model\n"
credentialProfile := `id: credential-profile
endpoint: ` + server.URL + `/v1
model: credential-model
api_key_env: NOTARIUS_PROMPTKIT_PROFILE_INSPECTION_TEST_KEY
`
t.Setenv("NOTARIUS_PROMPTKIT_PROFILE_INSPECTION_TEST_KEY", "")
tests := []struct {
name string
profilePath string
profileID string
profileDir bool
localBackend bool
canceled bool
wantErr []string
rejectErr []string
}{
{
name: "configured local backend",
profilePath: writeProfile(t, "local-profile", localProfile),
profileID: "local-profile",
profileDir: true,
localBackend: true,
},
{
name: "missing local backend registration",
profilePath: writeProfile(t, "local-profile", localProfile),
profileID: "local-profile",
wantErr: []string{`PromptKit profile "local-profile" is invalid or unreadable`},
},
{
name: "absent profile",
profilePath: writeProfile(t, "local-profile", localProfile),
profileID: "absent-profile",
localBackend: true,
wantErr: []string{`PromptKit profile "absent-profile" is not configured`},
},
{
name: "malformed profile",
profilePath: writeProfile(t, "malformed-profile", "id: malformed-profile\nbackend: [\n"),
profileID: "malformed-profile",
wantErr: []string{`PromptKit profile "malformed-profile" is invalid or unreadable`},
rejectErr: []string{"malformed-profile.yaml", "backend: ["},
},
{
name: "invalid profile source",
profilePath: filepath.Join(t.TempDir(), "missing-profile.yaml"),
profileID: "missing-profile",
wantErr: []string{"load PromptKit profiles", "profile configuration is invalid or unreadable"},
},
{
name: "credential environment intentionally unset",
profilePath: writeProfile(t, "credential-profile", credentialProfile),
profileID: "credential-profile",
},
{
name: "canceled inspection",
profilePath: writeProfile(t, "local-profile", localProfile),
profileID: "local-profile",
localBackend: true,
canceled: true,
wantErr: []string{"context canceled"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := config.Default()
if tt.profileDir {
cfg.PromptKit.ProfileDir = filepath.Dir(tt.profilePath)
} else {
cfg.PromptKit.ProfileFile = tt.profilePath
}
if tt.localBackend {
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
Endpoint: server.URL + "/v1",
ConcurrencyLimit: 2,
}
}
ctx := context.Background()
if tt.canceled {
var cancel context.CancelFunc
ctx, cancel = context.WithCancel(ctx)
cancel()
}
err := validateExplicitPromptKitProfiles(ctx, cfg, []string{tt.profileID}, nil)
if len(tt.wantErr) == 0 {
if err != nil {
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
}
return
}
if err == nil {
t.Fatal("validateExplicitPromptKitProfiles() error = nil, want failure")
}
if tt.canceled && !errors.Is(err, context.Canceled) {
t.Fatalf("canceled inspection error = %v, want context canceled", err)
}
for _, want := range tt.wantErr {
if !strings.Contains(err.Error(), want) {
t.Fatalf("validation error = %q, want %q", err, want)
}
}
for _, rejected := range append(tt.rejectErr, tt.profilePath) {
if rejected != "" && strings.Contains(err.Error(), rejected) {
t.Fatalf("validation error = %q, must not expose %q", err, rejected)
}
}
})
}
if providerCalls.Load() != 0 {
t.Fatalf("provider calls during profile inspection = %d, want 0", providerCalls.Load())
}
}
func TestExplicitPromptKitProfileValidationUsesFallbackAssets(t *testing.T) {
assets := llm.NewAssetRegistry()
if err := assets.RegisterFallbackProfileFS(fstest.MapFS{
"profiles/fallback.yaml": {Data: []byte("id: fallback-profile\nendpoint: http://promptkit.test/v1\nmodel: fallback-model\n")},
}, "profiles"); err != nil {
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
}
if err := validateExplicitPromptKitProfiles(context.Background(), config.Default(), []string{"fallback-profile"}, assets); err != nil {
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
}
}

View File

@@ -116,7 +116,7 @@ func (h *recomputeTestHarness) options() Options {
for _, key := range []string{"test/extract/producer", "test/extract/unrelated", "test/extract/middle", "test/extract/dependent"} { for _, key := range []string{"test/extract/producer", "test/extract/unrelated", "test/extract/middle", "test/extract/dependent"} {
moduleKey := key moduleKey := key
spec := pipeline.ModuleSpec{ spec := pipeline.ModuleSpec{
Key: moduleKey, Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind, Key: moduleKey, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind,
ReferenceSlots: []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}},
} }
if err := pipeline.RegisterExtractor(opts.Registries.Extractors, spec, func() (contracts.Extractor[stateTestArtifact], error) { if err := pipeline.RegisterExtractor(opts.Registries.Extractors, spec, func() (contracts.Extractor[stateTestArtifact], error) {
@@ -125,7 +125,7 @@ func (h *recomputeTestHarness) options() Options {
panic(err) panic(err)
} }
} }
if err := opts.Registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/recompute-output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { if err := opts.Registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/recompute-output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
return recomputeTestOutput{}, nil return recomputeTestOutput{}, nil
}); err != nil { }); err != nil {
panic(err) panic(err)

View File

@@ -364,21 +364,21 @@ func referenceContractCatalog(t *testing.T, includeBetaMerger, includeBetaNormal
t.Fatal(err) t.Fatal(err)
} }
} }
register(registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "reference/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil })) register(registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "reference/input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }))
register(registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/chunk", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-slot"}, {Name: "required-chunk", Required: true}}}, func() (contracts.Chunker, error) { return stateTestChunker{}, nil })) register(registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-slot"}, {Name: "required-chunk", Required: true}}}, func() (contracts.Chunker, error) { return stateTestChunker{}, nil }))
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecA{})) register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecA{}))
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecB{})) register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecB{}))
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-alpha", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil })) register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-alpha", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-beta", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil })) register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-beta", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil })) register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
if includeBetaMerger { if includeBetaMerger {
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil })) register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
} }
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil })) register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
if includeBetaNormalizer { if includeBetaNormalizer {
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil })) register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
} }
register(registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { return stateTestOutput{}, nil })) register(registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { return stateTestOutput{}, nil }))
return catalogFromRegistries(registries) return catalogFromRegistries(registries)
} }

View File

@@ -23,13 +23,14 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkplan" "gitea.maximumdirect.net/eric/notarius/internal/framework/chunkplan"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts" "gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
frameworkdebug "gitea.maximumdirect.net/eric/notarius/internal/framework/debug" frameworkdebug "gitea.maximumdirect.net/eric/notarius/internal/framework/debug"
frameworkllm "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline" "gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
) )
const defaultConfigPath = "/usr/local/etc/notarius/config.yml" const defaultConfigPath = "/usr/local/etc/notarius/config.yml"
const usage = `Usage: const usage = `Usage:
notarius help notarius help
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--output-dir path] [--json] [--chunk_cache auto|bypass|refresh] [--resume] [--recompute-step step-id] [--debug [--debug-dir path]] [--only lane-a,lane-b] [--session-id id] [--reference selector=path] [--without-reference selector] notarius run <pipeline-id> --input path/to/source.json [--json] [flags]
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b] notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list --config path/to/config.yml [--json] notarius pipelines list --config path/to/config.yml [--json]
` `
@@ -45,9 +46,14 @@ type Options struct {
ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory
DebugRecorderFactory func(string) (pipeline.DebugRecorder, error) DebugRecorderFactory func(string) (pipeline.DebugRecorder, error)
DebugTerminalFactory func(*debugbundle.SummaryWriter) DebugTerminalWriter DebugTerminalFactory func(*debugbundle.SummaryWriter) DebugTerminalWriter
promptKitAssets *frameworkllm.AssetRegistry
} }
type LLMClientFactory func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) type LLMRuntimeOverrides struct {
ReasoningEffort *string
}
type LLMClientFactory func(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error)
// Run executes the command-line interface and returns a process exit code. // Run executes the command-line interface and returns a process exit code.
func Run(args []string, stdout, stderr io.Writer) int { func Run(args []string, stdout, stderr io.Writer) int {
@@ -116,12 +122,17 @@ func normalizeOptions(opts Options) (Options, error) {
} }
opts.Registries = components.registries opts.Registries = components.registries
opts.Catalog = catalogFromRegistries(components.registries) opts.Catalog = catalogFromRegistries(components.registries)
if opts.LLMClientFactory == nil { opts.promptKitAssets = components.assets
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(components.assets)
}
} }
if opts.LLMClientFactory == nil { if opts.LLMClientFactory == nil {
opts.LLMClientFactory = productionLLMClientFactory if opts.promptKitAssets == nil {
assets, err := productionPromptAssets()
if err != nil {
return Options{}, err
}
opts.promptKitAssets = assets
}
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(opts.promptKitAssets)
} }
return opts, nil return opts, nil
} }
@@ -137,13 +148,16 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
debug := fs.Bool("debug", false, "write a debug bundle") debug := fs.Bool("debug", false, "write a debug bundle")
debugDir := fs.String("debug-dir", "", "debug bundle directory") debugDir := fs.String("debug-dir", "", "debug bundle directory")
llmProfile := fs.String("llm-profile", "", "LLM profile override") llmProfile := fs.String("llm-profile", "", "LLM profile override")
reasoningEffort := singleValueFlag{name: "--reasoning-effort"}
clearReasoningEffort := fs.Bool("clear-reasoning-effort", false, "clear the LLM profile reasoning effort")
resume := fs.Bool("resume", false, "reuse compatible recorded checkpoints") resume := fs.Bool("resume", false, "reuse compatible recorded checkpoints")
recomputeStep := singleValueFlag{} recomputeStep := singleValueFlag{name: "--recompute-step"}
chunkCache := chunkCacheFlag{} chunkCache := chunkCacheFlag{}
sessionID := sessionIDFlag{} sessionID := sessionIDFlag{}
referenceFlags := stringListFlag{} referenceFlags := stringListFlag{}
withoutReferenceFlags := stringListFlag{} withoutReferenceFlags := stringListFlag{}
fs.Var(&sessionID, "session-id", "prompt session identifier") fs.Var(&sessionID, "session-id", "prompt session identifier")
fs.Var(&reasoningEffort, "reasoning-effort", "reasoning effort override")
fs.Var(&chunkCache, "chunk_cache", "chunk plan cache mode: auto, bypass, or refresh") fs.Var(&chunkCache, "chunk_cache", "chunk plan cache mode: auto, bypass, or refresh")
fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, merge.slot=path, lane.slot=path, lane.extract.slot=path, lane.merge.slot=path, or lane.normalize.slot=path") fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, merge.slot=path, lane.slot=path, lane.extract.slot=path, lane.merge.slot=path, or lane.normalize.slot=path")
fs.Var(&withoutReferenceFlags, "without-reference", "unbind a reference, using the same selector forms as --reference") fs.Var(&withoutReferenceFlags, "without-reference", "unbind a reference, using the same selector forms as --reference")
@@ -189,6 +203,22 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
fmt.Fprintln(stderr, "notarius: --session-id must not be empty") fmt.Fprintln(stderr, "notarius: --session-id must not be empty")
return 2 return 2
} }
if reasoningEffort.set && *clearReasoningEffort {
fmt.Fprintln(stderr, "notarius: --reasoning-effort cannot be combined with --clear-reasoning-effort")
return 2
}
if reasoningEffort.set && strings.TrimSpace(reasoningEffort.value) == "" {
fmt.Fprintln(stderr, "notarius: --reasoning-effort must not be empty")
return 2
}
runtimeOverrides := LLMRuntimeOverrides{}
if reasoningEffort.set {
value := strings.TrimSpace(reasoningEffort.value)
runtimeOverrides.ReasoningEffort = &value
} else if *clearReasoningEffort {
value := ""
runtimeOverrides.ReasoningEffort = &value
}
only, err := parseOnly(*onlyRaw) only, err := parseOnly(*onlyRaw)
if err != nil { if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err) fmt.Fprintf(stderr, "notarius: %v\n", err)
@@ -286,6 +316,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
ConfigSource: configSource(*configPath), ConfigSource: configSource(*configPath),
OnlyLanes: append([]string(nil), only...), OnlyLanes: append([]string(nil), only...),
ChunkCacheOverride: chunkCache.explicitValue(), ChunkCacheOverride: chunkCache.explicitValue(),
ReasoningEffortOverride: runtimeOverrides.ReasoningEffort,
Resume: *resume, Resume: *resume,
RecomputeStep: strings.TrimSpace(recomputeStep.value), RecomputeStep: strings.TrimSpace(recomputeStep.value),
RunID: runID, RunID: runID,
@@ -315,7 +346,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
return failPipelineCommand(stderr, commandState, terminalWriter, err) return failPipelineCommand(stderr, commandState, terminalWriter, err)
} }
profileIDs := effectiveLLMProfileIDs(effective.ResolvedPipeline) profileIDs := effectiveLLMProfileIDs(effective.ResolvedPipeline)
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, profileIDs); err != nil { if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, profileIDs, opts.promptKitAssets); err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, err) return failPipelineCommand(stderr, commandState, terminalWriter, err)
} }
workingDir, err := os.Getwd() workingDir, err := os.Getwd()
@@ -363,10 +394,17 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
if len(profileIDs) == 1 { if len(profileIDs) == 1 {
factoryProfileID = profileIDs[0] factoryProfileID = profileIDs[0]
} }
llmClient, llmProfiles, err := opts.LLMClientFactory(ctx, effective.Config, factoryProfileID) llmClient, llmProfiles, err := opts.LLMClientFactory(ctx, effective.Config, factoryProfileID, runtimeOverrides)
if err != nil { if err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("create LLM client for profile %q: %w", factoryProfileID, err)) return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("create LLM client for profile %q: %w", factoryProfileID, err))
} }
var llmFingerprints []checkpoint.Fingerprint
if effective.Config.Cache.Checkpoints.Enabled {
llmFingerprints, err = llmCheckpointFingerprints(llmClient)
if err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("prepare LLM checkpoint identity: %w", err))
}
}
llmClient = pipeline.WithDebugLLMRecording(llmClient, debugRecorder) llmClient = pipeline.WithDebugLLMRecording(llmClient, debugRecorder)
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, registries, pipeline.ModuleDependencies{LLM: llmClient}) prepared, err := pipeline.Prepare(effective.ResolvedPipeline, registries, pipeline.ModuleDependencies{LLM: llmClient})
if err != nil { if err != nil {
@@ -380,7 +418,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
if err != nil { if err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, err) return failPipelineCommand(stderr, commandState, terminalWriter, err)
} }
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(effective.Config.Cache.Checkpoints, opts, effective.ResolvedPipeline, prepared.CheckpointFingerprints(), rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), *resume) checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(effective.Config.Cache.Checkpoints, opts, effective.ResolvedPipeline, prepared.CheckpointFingerprints(), llmFingerprints, rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), runtimeOverrides, *resume)
if err != nil { if err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, err) return failPipelineCommand(stderr, commandState, terminalWriter, err)
} }
@@ -481,11 +519,13 @@ func checkpointHandlersForRun(
opts Options, opts Options,
resolved pipeline.ResolvedPipeline, resolved pipeline.ResolvedPipeline,
componentFingerprints []pipeline.CheckpointFingerprint, componentFingerprints []pipeline.CheckpointFingerprint,
llmFingerprints []checkpoint.Fingerprint,
rawInput []byte, rawInput []byte,
only []string, only []string,
llmProfiles []artifacts.LLMProfileManifest, llmProfiles []artifacts.LLMProfileManifest,
llmProfileOverride string, llmProfileOverride string,
sessionID string, sessionID string,
runtimeOverrides LLMRuntimeOverrides,
resume bool, resume bool,
) (pipeline.CheckpointRecorder, pipeline.CheckpointLoader, error) { ) (pipeline.CheckpointRecorder, pipeline.CheckpointLoader, error) {
if !settings.Enabled { if !settings.Enabled {
@@ -499,9 +539,13 @@ func checkpointHandlersForRun(
InputKey: resolved.Input.Module, InputKey: resolved.Input.Module,
RawInputDigest: rawInputDigest(rawInput), RawInputDigest: rawInputDigest(rawInput),
SelectedLanes: only, SelectedLanes: only,
RuntimeOverrides: runtimeOverrideFingerprints(llmProfileOverride, sessionID), RuntimeOverrides: runtimeOverrideFingerprints(llmProfileOverride, sessionID, runtimeOverrides),
References: pipeline.ReferenceProvenance(resolved), References: pipeline.ReferenceProvenance(resolved),
ProvenanceFingerprints: append(llmProfileFingerprints(llmProfiles), checkpointIdentityFingerprints(componentFingerprints)...), ProvenanceFingerprints: combineCheckpointFingerprints(
llmProfileFingerprints(llmProfiles),
llmFingerprints,
checkpointIdentityFingerprints(componentFingerprints),
),
}) })
if err != nil { if err != nil {
return nil, nil, fmt.Errorf("create checkpoint identity: %w", err) return nil, nil, fmt.Errorf("create checkpoint identity: %w", err)
@@ -527,6 +571,30 @@ func checkpointHandlersForRun(
return recorder, loader, nil return recorder, loader, nil
} }
func llmCheckpointFingerprints(client contracts.StructuredLLMClient) ([]checkpoint.Fingerprint, error) {
provider, ok := client.(frameworkllm.CheckpointFingerprintProvider)
if !ok {
return nil, nil
}
values, err := provider.LLMCheckpointFingerprints()
if err != nil {
return nil, err
}
out := make([]checkpoint.Fingerprint, 0, len(values))
for _, value := range values {
out = append(out, checkpoint.Fingerprint{Name: value.Name, Value: value.Value})
}
return out, nil
}
func combineCheckpointFingerprints(sources ...[]checkpoint.Fingerprint) []checkpoint.Fingerprint {
var out []checkpoint.Fingerprint
for _, source := range sources {
out = append(out, source...)
}
return out
}
func recomputePolicy(resolved pipeline.ResolvedPipeline, requestedStep string) (pipeline.CheckpointExecutionPolicy, error) { func recomputePolicy(resolved pipeline.ResolvedPipeline, requestedStep string) (pipeline.CheckpointExecutionPolicy, error) {
requestedStep = strings.TrimSpace(requestedStep) requestedStep = strings.TrimSpace(requestedStep)
if requestedStep == "" { if requestedStep == "" {
@@ -632,7 +700,7 @@ func rawInputDigest(data []byte) string {
return "sha256:" + hex.EncodeToString(sum[:]) return "sha256:" + hex.EncodeToString(sum[:])
} }
func runtimeOverrideFingerprints(llmProfileOverride string, sessionID string) []checkpoint.Fingerprint { func runtimeOverrideFingerprints(llmProfileOverride string, sessionID string, runtimeOverrides LLMRuntimeOverrides) []checkpoint.Fingerprint {
var values []checkpoint.Fingerprint var values []checkpoint.Fingerprint
if strings.TrimSpace(llmProfileOverride) != "" { if strings.TrimSpace(llmProfileOverride) != "" {
values = append(values, checkpoint.Fingerprint{Name: "llm_profile_override", Value: strings.TrimSpace(llmProfileOverride)}) values = append(values, checkpoint.Fingerprint{Name: "llm_profile_override", Value: strings.TrimSpace(llmProfileOverride)})
@@ -640,6 +708,13 @@ func runtimeOverrideFingerprints(llmProfileOverride string, sessionID string) []
if strings.TrimSpace(sessionID) != "" { if strings.TrimSpace(sessionID) != "" {
values = append(values, checkpoint.Fingerprint{Name: "session_id", Value: strings.TrimSpace(sessionID)}) values = append(values, checkpoint.Fingerprint{Name: "session_id", Value: strings.TrimSpace(sessionID)})
} }
if runtimeOverrides.ReasoningEffort != nil {
value := strings.TrimSpace(*runtimeOverrides.ReasoningEffort)
if value == "" {
value = "<cleared>"
}
values = append(values, checkpoint.Fingerprint{Name: "reasoning_effort_override", Value: value})
}
return values return values
} }
@@ -796,7 +871,7 @@ func reorderRunArgs(args []string) []string {
func runFlagTakesValue(arg string) bool { func runFlagTakesValue(arg string) bool {
switch arg { switch arg {
case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--chunk_cache", "--reference", "--without-reference", "--recompute-step": case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--reasoning-effort", "--chunk_cache", "--reference", "--without-reference", "--recompute-step":
return true return true
default: default:
return false return false
@@ -856,11 +931,11 @@ func chunkPlanStoreForRun(cfg config.ChunkPlanCacheConfig, opts Options) (pipeli
func validateRunFlagValues(args []string) error { func validateRunFlagValues(args []string) error {
for i, arg := range args { for i, arg := range args {
if arg != "--session-id" { if arg != "--session-id" && arg != "--reasoning-effort" {
continue continue
} }
if i+1 >= len(args) || strings.HasPrefix(args[i+1], "-") { if i+1 >= len(args) || strings.HasPrefix(args[i+1], "-") {
return fmt.Errorf("flag needs an argument: --session-id") return fmt.Errorf("flag needs an argument: %s", arg)
} }
} }
return nil return nil
@@ -883,12 +958,23 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
seen[id] = struct{}{} seen[id] = struct{}{}
} }
} }
if resolved.InputExecutionClass == contracts.ExecutionClassLLMBacked {
add(resolved.Input)
}
if resolved.ChunkExecutionClass == contracts.ExecutionClassLLMBacked {
add(resolved.Chunk) add(resolved.Chunk)
}
for _, lane := range resolved.AllArtifactLanes() { for _, lane := range resolved.AllArtifactLanes() {
if lane.ExtractExecutionClass == contracts.ExecutionClassLLMBacked {
add(lane.Extract) add(lane.Extract)
}
if lane.MergeExecutionClass == contracts.ExecutionClassLLMBacked {
add(lane.Merge) add(lane.Merge)
}
if lane.NormalizeExecutionClass == contracts.ExecutionClassLLMBacked {
add(lane.Normalize) add(lane.Normalize)
} }
}
for _, chain := range resolved.ValidatorChains { for _, chain := range resolved.ValidatorChains {
for _, validator := range chain.Validators { for _, validator := range chain.Validators {
if validator.ExecutionClass == contracts.ExecutionClassLLMBacked { if validator.ExecutionClass == contracts.ExecutionClassLLMBacked {
@@ -896,6 +982,9 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
} }
} }
} }
if resolved.OutputExecutionClass == contracts.ExecutionClassLLMBacked {
add(resolved.Output)
}
ids := make([]string, 0, len(seen)) ids := make([]string, 0, len(seen))
for id := range seen { for id := range seen {
ids = append(ids, id) ids = append(ids, id)
@@ -979,7 +1068,7 @@ func runConfigValidate(args []string, stdout, stderr io.Writer, opts Options) in
fmt.Fprintf(stderr, "notarius: %v\n", err) fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1 return 1
} }
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline)); err != nil { if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline), opts.promptKitAssets); err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err) fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1 return 1
} }
@@ -1144,6 +1233,7 @@ type sessionIDFlag struct {
} }
type singleValueFlag struct { type singleValueFlag struct {
name string
value string value string
set bool set bool
} }
@@ -1157,7 +1247,7 @@ func (flag *singleValueFlag) String() string {
func (flag *singleValueFlag) Set(value string) error { func (flag *singleValueFlag) Set(value string) error {
if flag.set { if flag.set {
return fmt.Errorf("--recompute-step may be specified only once") return fmt.Errorf("%s may be specified only once", flag.name)
} }
flag.value = value flag.value = value
flag.set = true flag.set = true

View File

@@ -12,6 +12,7 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts" "gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config" "gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/checkpoint"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts" "gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline" "gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
) )
@@ -245,8 +246,10 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
harness := newStateTestHarness() harness := newStateTestHarness()
var factoryProfiles []string var factoryProfiles []string
opts := harness.options() opts := harness.options()
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) { var factoryOverrides []LLMRuntimeOverrides
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
factoryProfiles = append(factoryProfiles, profileID) factoryProfiles = append(factoryProfiles, profileID)
factoryOverrides = append(factoryOverrides, overrides)
return nil, nil, nil return nil, nil, nil
} }
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
@@ -257,6 +260,9 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" { if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" {
t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles) t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles)
} }
if len(factoryOverrides) != 1 || factoryOverrides[0].ReasoningEffort != nil {
t.Fatalf("factory overrides = %#v, want inherited reasoning", factoryOverrides)
}
harness.mu.Lock() harness.mu.Lock()
profiles := append([]string(nil), harness.moduleProfiles...) profiles := append([]string(nil), harness.moduleProfiles...)
harness.mu.Unlock() harness.mu.Unlock()
@@ -270,7 +276,7 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
} }
}) })
t.Run("validator profile remains distinct", func(t *testing.T) { t.Run("runtime override applies to validators", func(t *testing.T) {
roots := newStateTestRoots(t) roots := newStateTestRoots(t)
profileDir := writeRunContractProfiles(t, "override-profile", "validator-profile") profileDir := writeRunContractProfiles(t, "override-profile", "validator-profile")
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir)) prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
@@ -279,7 +285,7 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
opts := harness.options() opts := harness.options()
registerRunContractValidator(t, &opts, &validatorProfiles) registerRunContractValidator(t, &opts, &validatorProfiles)
factoryProfiles := []string{} factoryProfiles := []string{}
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) { opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string, _ LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
factoryProfiles = append(factoryProfiles, profileID) factoryProfiles = append(factoryProfiles, profileID)
return nil, nil, nil return nil, nil, nil
} }
@@ -288,11 +294,11 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
if code != 0 || stderr.Len() != 0 { if code != 0 || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String()) t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
} }
if len(factoryProfiles) != 1 || factoryProfiles[0] != "" { if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" {
t.Fatalf("factory profiles = %#v, want one call without a unique profile", factoryProfiles) t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles)
} }
if len(validatorProfiles) != 1 || validatorProfiles[0] != "validator-profile" { if len(validatorProfiles) != 1 || validatorProfiles[0] != "override-profile" {
t.Fatalf("validator profiles = %#v, want configured validator profile", validatorProfiles) t.Fatalf("validator profiles = %#v, want runtime override", validatorProfiles)
} }
}) })
@@ -302,7 +308,7 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir)) prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
factoryCalls := 0 factoryCalls := 0
opts := newStateTestHarness().options() opts := newStateTestHarness().options()
opts.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) { opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
factoryCalls++ factoryCalls++
return nil, nil, nil return nil, nil, nil
} }
@@ -312,18 +318,164 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls) t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
} }
}) })
t.Run("pipeline default is rejected before factory access", func(t *testing.T) {
roots := newStateTestRoots(t)
profileDir := writeRunContractProfiles(t, "configured-profile")
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
replaceStateTestConfigLine(t, roots.config, " sample:\n", " sample:\n llm_profile: missing-profile\n")
factoryCalls := 0
opts := newStateTestHarness().options()
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
factoryCalls++
return nil, nil, nil
}
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, &stdout, &stderr, opts)
if code != 1 || !strings.Contains(stderr.String(), "not configured") || factoryCalls != 0 || stdout.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
}
})
}
func TestRunReasoningEffortOverrideReachesFactory(t *testing.T) {
tests := []struct {
name string
flags []string
wantValue string
wantSet bool
}{
{name: "inherit"},
{name: "replace", flags: []string{"--reasoning-effort", " focused "}, wantValue: "focused", wantSet: true},
{name: "clear", flags: []string{"--clear-reasoning-effort"}, wantSet: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
roots := newStateTestRoots(t)
opts := newStateTestHarness().options()
var got []LLMRuntimeOverrides
opts.LLMClientFactory = func(_ context.Context, _ config.Config, _ string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
got = append(got, overrides)
return nil, nil, nil
}
args := append([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, tt.flags...)
var stdout, stderr bytes.Buffer
if code := RunWithOptions(args, &stdout, &stderr, opts); code != 0 || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
if len(got) != 1 {
t.Fatalf("factory overrides = %#v, want one call", got)
}
if !tt.wantSet {
if got[0].ReasoningEffort != nil {
t.Fatalf("reasoning effort = %q, want inherit", *got[0].ReasoningEffort)
}
return
}
if got[0].ReasoningEffort == nil || *got[0].ReasoningEffort != tt.wantValue {
t.Fatalf("reasoning effort = %#v, want %q", got[0].ReasoningEffort, tt.wantValue)
}
})
}
}
func TestRunReasoningEffortOverrideRejectsInvalidSyntax(t *testing.T) {
tests := []struct {
name string
flags []string
wantError string
}{
{
name: "mutually exclusive controls",
flags: []string{"--reasoning-effort", "focused", "--clear-reasoning-effort"},
wantError: "cannot be combined",
},
{
name: "empty replacement",
flags: []string{"--reasoning-effort", " "},
wantError: "must not be empty",
},
{
name: "duplicate replacement",
flags: []string{"--reasoning-effort", "low", "--reasoning-effort", "high"},
wantError: "may be specified only once",
},
{
name: "missing replacement",
flags: []string{"--reasoning-effort"},
wantError: "flag needs an argument",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
roots := newStateTestRoots(t)
args := append([]string{"run", "sample", "--config", roots.config, "--input", roots.input}, tt.flags...)
var stdout, stderr bytes.Buffer
code := RunWithOptions(args, &stdout, &stderr, newStateTestHarness().options())
if code != 2 || stdout.Len() != 0 || !strings.Contains(stderr.String(), tt.wantError) {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
assertNoRunState(t, roots)
})
}
}
func TestReasoningEffortOverrideSeparatesCheckpointIdentities(t *testing.T) {
replacement := " focused "
cleared := ""
states := []struct {
name string
overrides LLMRuntimeOverrides
wantValue string
wantSet bool
}{
{name: "inherit"},
{name: "replace", overrides: LLMRuntimeOverrides{ReasoningEffort: &replacement}, wantValue: "focused", wantSet: true},
{name: "clear", overrides: LLMRuntimeOverrides{ReasoningEffort: &cleared}, wantValue: "<cleared>", wantSet: true},
}
digests := make(map[string]string, len(states))
for _, state := range states {
fingerprints := runtimeOverrideFingerprints("", "", state.overrides)
var value string
var found bool
for _, fingerprint := range fingerprints {
if fingerprint.Name == "reasoning_effort_override" {
value, found = fingerprint.Value, true
}
}
if found != state.wantSet || (found && value != state.wantValue) {
t.Fatalf("%s fingerprint found=%t value=%q, want found=%t value=%q", state.name, found, value, state.wantSet, state.wantValue)
}
identity, err := checkpoint.NewIdentity(checkpoint.IdentityInput{
Pipeline: pipeline.ResolvedPipeline{ID: "sample", Digest: "sha256:pipeline", Input: pipeline.Binding("test/input")},
RawInputDigest: "sha256:input",
RuntimeOverrides: fingerprints,
})
if err != nil {
t.Fatal(err)
}
digests[state.name] = identity.Digest
}
if digests["inherit"] == digests["replace"] || digests["inherit"] == digests["clear"] || digests["replace"] == digests["clear"] {
t.Fatalf("checkpoint identity digests are not distinct: %#v", digests)
}
} }
func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) { func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
resolved := pipeline.ResolvedPipeline{ resolved := pipeline.ResolvedPipeline{
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"}, Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
InputExecutionClass: contracts.ExecutionClassLLMBacked,
Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "}, Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "},
ChunkExecutionClass: contracts.ExecutionClassLLMBacked,
Steps: []pipeline.ResolvedPipelineStep{{ Steps: []pipeline.ResolvedPipelineStep{{
ID: "default", ID: "default",
ArtifactLanes: []pipeline.ResolvedArtifactLane{{ ArtifactLanes: []pipeline.ResolvedArtifactLane{{
Extract: pipeline.ModuleBinding{LLMProfile: "alpha"}, Extract: pipeline.ModuleBinding{LLMProfile: "alpha"},
Merge: pipeline.ModuleBinding{LLMProfile: "zeta"}, ExtractExecutionClass: contracts.ExecutionClassLLMBacked,
Merge: pipeline.ModuleBinding{LLMProfile: "deterministic-merge"},
MergeExecutionClass: contracts.ExecutionClassDeterministic,
Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "}, Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "},
NormalizeExecutionClass: contracts.ExecutionClassLLMBacked,
}}, }},
}}, }},
ValidatorChains: []pipeline.ResolvedValidatorChain{{Validators: []pipeline.ResolvedValidator{ ValidatorChains: []pipeline.ResolvedValidatorChain{{Validators: []pipeline.ResolvedValidator{
@@ -331,9 +483,10 @@ func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
{Binding: pipeline.ModuleBinding{LLMProfile: "beta"}, ExecutionClass: contracts.ExecutionClassLLMBacked}, {Binding: pipeline.ModuleBinding{LLMProfile: "beta"}, ExecutionClass: contracts.ExecutionClassLLMBacked},
}}}, }}},
Output: pipeline.ModuleBinding{LLMProfile: "output-profile"}, Output: pipeline.ModuleBinding{LLMProfile: "output-profile"},
OutputExecutionClass: contracts.ExecutionClassLLMBacked,
} }
got := effectiveLLMProfileIDs(resolved) got := effectiveLLMProfileIDs(resolved)
want := []string{"alpha", "beta", "gamma", "zeta"} want := []string{"alpha", "beta", "gamma", "input-profile", "output-profile", "zeta"}
if strings.Join(got, ",") != strings.Join(want, ",") { if strings.Join(got, ",") != strings.Join(want, ",") {
t.Fatalf("effective profiles = %#v, want %#v", got, want) t.Fatalf("effective profiles = %#v, want %#v", got, want)
} }
@@ -376,7 +529,7 @@ func TestRunFactoryAndPreparationFailuresAreProcessFailures(t *testing.T) {
t.Run("LLM factory", func(t *testing.T) { t.Run("LLM factory", func(t *testing.T) {
roots := newStateTestRoots(t) roots := newStateTestRoots(t)
opts := newStateTestHarness().options() opts := newStateTestHarness().options()
opts.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) { opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
return nil, nil, errors.New("injected LLM factory failure") return nil, nil, errors.New("injected LLM factory failure")
} }
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
@@ -397,7 +550,7 @@ func TestRunFactoryAndPreparationFailuresAreProcessFailures(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
opts := newStateTestHarness().options() opts := newStateTestHarness().options()
if err := pipeline.RegisterExtractorBuilder(opts.Registries.Extractors, pipeline.ModuleSpec{Key: "test/failing-extract", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Extractor[stateTestArtifact], error) { if err := pipeline.RegisterExtractorBuilder(opts.Registries.Extractors, pipeline.ModuleSpec{Key: "test/failing-extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Extractor[stateTestArtifact], error) {
return nil, errors.New("injected extractor construction failure") return nil, errors.New("injected extractor construction failure")
}); err != nil { }); err != nil {
t.Fatal(err) t.Fatal(err)

View File

@@ -214,8 +214,9 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
t.Fatal(err) t.Fatal(err)
} }
fingerprints := prepared.CheckpointFingerprints() fingerprints := prepared.CheckpointFingerprints()
llmFingerprints := []checkpoint.Fingerprint{{Name: "promptkit_profile_source", Value: "sha256:profile-source-one"}}
settings := config.CheckpointCacheConfig{Enabled: true, Directory: t.TempDir()} settings := config.CheckpointCacheConfig{Enabled: true, Directory: t.TempDir()}
recorder, _, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, []byte("same input"), nil, nil, "", "", false) recorder, _, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, false)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -241,7 +242,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
t.Fatal(err) t.Fatal(err)
} }
_, sameLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, []byte("same input"), nil, nil, "", "", true) _, sameLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -253,7 +254,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
} }
changed := replaceCheckpointFingerprintValue(t, fingerprints, normalizeSpellCatalogFingerprintName(), "sha256:changed-effective-catalog") changed := replaceCheckpointFingerprintValue(t, fingerprints, normalizeSpellCatalogFingerprintName(), "sha256:changed-effective-catalog")
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changed, normalizeSpellCatalogFingerprintName()) assertOnlyCheckpointFingerprintChanged(t, fingerprints, changed, normalizeSpellCatalogFingerprintName())
_, changedLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changed, []byte("same input"), nil, nil, "", "", true) _, changedLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changed, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -265,13 +266,25 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
} }
changedMapping := replaceCheckpointFingerprintValue(t, fingerprints, extractSpellMappingFingerprintName(), "dnd.spells.extract_mapping.v3") changedMapping := replaceCheckpointFingerprintValue(t, fingerprints, extractSpellMappingFingerprintName(), "dnd.spells.extract_mapping.v3")
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changedMapping, extractSpellMappingFingerprintName()) assertOnlyCheckpointFingerprintChanged(t, fingerprints, changedMapping, extractSpellMappingFingerprintName())
_, mappingLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changedMapping, []byte("same input"), nil, nil, "", "", true) _, mappingLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changedMapping, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if _, decision := mappingLoader.Source(materialized.Input.Module); decision.Reused { if _, decision := mappingLoader.Source(materialized.Input.Module); decision.Reused {
t.Fatalf("changed mapping policy decision = %#v, want cold miss", decision) t.Fatalf("changed mapping policy decision = %#v, want cold miss", decision)
} }
changedLLMFingerprints := []checkpoint.Fingerprint{{Name: "promptkit_profile_source", Value: "sha256:profile-source-two"}}
_, profileLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, changedLLMFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
if err != nil {
t.Fatal(err)
}
if _, decision := profileLoader.Source(materialized.Input.Module); decision.Reused {
t.Fatalf("changed PromptKit profile source decision = %#v, want cold miss", decision)
}
if _, decision := profileLoader.Normalize("spells", spellnormalize.Key, normalizeDependencies); decision.Reused {
t.Fatalf("changed PromptKit profile normalize decision = %#v, want cold miss", decision)
}
} }
func normalizeSpellCatalogFingerprintName() string { func normalizeSpellCatalogFingerprintName() string {

View File

@@ -823,22 +823,22 @@ func (h *stateTestHarness) options() Options {
if err := pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, stateTestCodec{}); err != nil { if err := pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, stateTestCodec{}); err != nil {
panic(err) panic(err)
} }
if err := registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }); err != nil { if err := registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }); err != nil {
panic(err) panic(err)
} }
if err := registries.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/chunk", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "cache-reference"}}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Chunker, error) { return stateTestChunker{h}, nil }); err != nil { if err := registries.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "cache-reference"}}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Chunker, error) { return stateTestChunker{h}, nil }); err != nil {
panic(err) panic(err)
} }
if err := pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "test/extract", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{h}, nil }); err != nil { if err := pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "test/extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{h}, nil }); err != nil {
panic(err) panic(err)
} }
if err := pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "test/merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{harness: h}, nil }); err != nil { if err := pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "test/merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{harness: h}, nil }); err != nil {
panic(err) panic(err)
} }
if err := pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "test/normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{harness: h}, nil }); err != nil { if err := pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "test/normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{harness: h}, nil }); err != nil {
panic(err) panic(err)
} }
if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
return stateTestOutput{harness: h, includeWarnings: h.includeWarnings}, nil return stateTestOutput{harness: h, includeWarnings: h.includeWarnings}, nil
}); err != nil { }); err != nil {
panic(err) panic(err)
@@ -848,7 +848,7 @@ func (h *stateTestHarness) options() Options {
defer h.mu.Unlock() defer h.mu.Unlock()
h.runIDCalls++ h.runIDCalls++
return fmt.Sprintf("run-%d-%032x", startedAt.UnixNano(), h.runIDCalls), nil return fmt.Sprintf("run-%d-%032x", startedAt.UnixNano(), h.runIDCalls), nil
}, UserCacheDir: func() (string, error) { return "", errors.New("unexpected user cache lookup") }, LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) { }, UserCacheDir: func() (string, error) { return "", errors.New("unexpected user cache lookup") }, LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
return nil, nil, nil return nil, nil, nil
}} }}
} }

View File

@@ -1,6 +1,7 @@
package artifacts package artifacts
import ( import (
"strings"
"time" "time"
) )
@@ -29,6 +30,29 @@ type LLMProfileManifest struct {
ID string `json:"id"` ID string `json:"id"`
Provider string `json:"provider,omitempty"` Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"` Model string `json:"model,omitempty"`
BackendID string `json:"backend_id,omitempty"`
ReasoningEffort string `json:"reasoning_effort,omitempty"`
}
// Normalized returns the canonical representation used for manifest identity
// and publication.
func (profile LLMProfileManifest) Normalized() LLMProfileManifest {
profile.ID = strings.TrimSpace(profile.ID)
profile.Provider = strings.TrimSpace(profile.Provider)
profile.Model = strings.TrimSpace(profile.Model)
profile.BackendID = strings.TrimSpace(profile.BackendID)
profile.ReasoningEffort = strings.TrimSpace(profile.ReasoningEffort)
return profile
}
// IdentityKey returns an opaque, deterministic key for the effective profile.
func (profile LLMProfileManifest) IdentityKey() string {
profile = profile.Normalized()
return profile.ID + "\x00" +
profile.Provider + "\x00" +
profile.Model + "\x00" +
profile.BackendID + "\x00" +
profile.ReasoningEffort
} }
type ReferenceProvenance struct { type ReferenceProvenance struct {

View File

@@ -53,7 +53,13 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
PipelineID: "pipeline-1", PipelineID: "pipeline-1",
PipelineDigest: "sha256:abc123", PipelineDigest: "sha256:abc123",
LLMProfiles: []LLMProfileManifest{ LLMProfiles: []LLMProfileManifest{
{ID: "default", Provider: "promptkit", Model: "model-a"}, {
ID: "default",
Provider: "promptkit",
Model: "model-a",
BackendID: "openrouter",
ReasoningEffort: "high",
},
}, },
ArtifactLanes: []ArtifactLaneManifest{ ArtifactLanes: []ArtifactLaneManifest{
{ {
@@ -101,10 +107,13 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
if !ok { if !ok {
t.Fatalf("llm_profiles[0] = %#v, want object", profiles[0]) t.Fatalf("llm_profiles[0] = %#v, want object", profiles[0])
} }
assertHasKeys(t, profile, "id", "provider", "model") assertHasKeys(t, profile, "id", "provider", "model", "backend_id", "reasoning_effort")
if profile["provider"] != "promptkit" { if profile["provider"] != "promptkit" {
t.Fatalf("llm_profiles[0].provider = %#v, want promptkit", profile["provider"]) t.Fatalf("llm_profiles[0].provider = %#v, want promptkit", profile["provider"])
} }
if profile["backend_id"] != "openrouter" || profile["reasoning_effort"] != "high" {
t.Fatalf("llm_profiles[0] = %#v, want backend and reasoning provenance", profile)
}
lanes, ok := got["artifact_lanes"].([]any) lanes, ok := got["artifact_lanes"].([]any)
if !ok { if !ok {

View File

@@ -18,6 +18,12 @@ type Config struct {
type PromptKitConfig struct { type PromptKitConfig struct {
ProfileDir string `json:"profile_dir,omitempty"` ProfileDir string `json:"profile_dir,omitempty"`
ProfileFile string `json:"profile_file,omitempty"` ProfileFile string `json:"profile_file,omitempty"`
LocalBackend *PromptKitLocalBackendConfig `json:"local_backend,omitempty"`
}
type PromptKitLocalBackendConfig struct {
Endpoint string `json:"endpoint"`
ConcurrencyLimit int `json:"concurrency_limit"`
} }
type ConcurrencyConfig struct { type ConcurrencyConfig struct {
@@ -66,6 +72,10 @@ func Default() Config {
func cloneConfig(in Config) Config { func cloneConfig(in Config) Config {
out := in out := in
if in.PromptKit.LocalBackend != nil {
localBackend := *in.PromptKit.LocalBackend
out.PromptKit.LocalBackend = &localBackend
}
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers) out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines)) out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
for key, profile := range in.Pipelines { for key, profile := range in.Pipelines {

View File

@@ -42,12 +42,10 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
} }
profile = clonePipelineProfile(profile) profile = clonePipelineProfile(profile)
profile.ID = pipelineID profile.ID = pipelineID
if override := strings.TrimSpace(input.LLMProfileOverride); override != "" {
applyLLMProfileOverride(&profile, override)
}
resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{ resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{
Only: input.Only, Only: input.Only,
LLMProfileOverride: input.LLMProfileOverride,
ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...), ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...),
ReferenceUnbinds: append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...), ReferenceUnbinds: append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...),
}, input.Catalog) }, input.Catalog)
@@ -65,22 +63,6 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
}, nil }, nil
} }
func applyLLMProfileOverride(profile *pipeline.PipelineProfile, profileID string) {
profile.Chunk.LLMProfile = profileID
apply := func(artifacts map[string]pipeline.ArtifactLaneProfile) {
for laneID, lane := range artifacts {
lane.Extract.LLMProfile = profileID
lane.Merge.LLMProfile = profileID
lane.Normalize.LLMProfile = profileID
artifacts[laneID] = lane
}
}
apply(profile.Artifacts)
for index := range profile.Steps {
apply(profile.Steps[index].Artifacts)
}
}
func lookupPipelineProfile(profiles map[string]pipeline.PipelineProfile, pipelineID string) (pipeline.PipelineProfile, bool) { func lookupPipelineProfile(profiles map[string]pipeline.PipelineProfile, pipelineID string) (pipeline.PipelineProfile, bool) {
pipelineID = strings.TrimSpace(pipelineID) pipelineID = strings.TrimSpace(pipelineID)
for rawID, profile := range profiles { for rawID, profile := range profiles {

View File

@@ -107,6 +107,33 @@ func TestEffectiveConfigPreservesPromptKitProfileSource(t *testing.T) {
} }
} }
func TestEffectiveConfigOwnsPromptKitLocalBackend(t *testing.T) {
cfg := configForEffectiveTests(t, effectiveProfile())
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
Endpoint: "http://localhost:8000/v1",
ConcurrencyLimit: 2,
}
effective, err := cfg.Resolve(ResolveInput{PipelineID: "main", Catalog: effectiveCatalog(t)})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if effective.Config.PromptKit.LocalBackend == nil {
t.Fatal("effective local backend = nil")
}
if effective.Config.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend {
t.Fatal("effective local backend aliases input config")
}
cfg.PromptKit.LocalBackend.Endpoint = "http://changed-input.example/v1"
if effective.Config.PromptKit.LocalBackend.Endpoint != "http://localhost:8000/v1" {
t.Fatalf("input mutation changed effective config: %#v", effective.Config.PromptKit.LocalBackend)
}
effective.Config.PromptKit.LocalBackend.ConcurrencyLimit = 9
if cfg.PromptKit.LocalBackend.ConcurrencyLimit != 2 {
t.Fatalf("effective mutation changed input config: %#v", cfg.PromptKit.LocalBackend)
}
}
func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) { func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
@@ -177,7 +204,7 @@ func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
} }
} }
func TestEffectiveConfigLLMProfileOverrideChangesDigestWithoutOverridingValidators(t *testing.T) { func TestEffectiveConfigLLMProfileOverrideChangesDigestAndOverridesValidators(t *testing.T) {
profile := effectiveProfile() profile := effectiveProfile()
profile.Chunk.LLMProfile = "chunk-profile" profile.Chunk.LLMProfile = "chunk-profile"
lane := profile.Artifacts["lane"] lane := profile.Artifacts["lane"]
@@ -210,8 +237,27 @@ func TestEffectiveConfigLLMProfileOverrideChangesDigestWithoutOverridingValidato
t.Fatalf("pipeline profile override was not applied: %#v", resolved) t.Fatalf("pipeline profile override was not applied: %#v", resolved)
} }
validators := findEffectiveValidatorChain(resolved, pipeline.StageExtract, "lane") validators := findEffectiveValidatorChain(resolved, pipeline.StageExtract, "lane")
if len(validators.Validators) != 1 || validators.Validators[0].Binding.LLMProfile != "validator-profile" { if len(validators.Validators) != 1 || validators.Validators[0].Binding.LLMProfile != "override-profile" {
t.Fatalf("validator profile was overridden: %#v", validators) t.Fatalf("validator profile = %#v, want runtime override", validators)
}
}
func TestEffectiveConfigPipelineLLMProfileIsInheritedWithoutMutatingConfig(t *testing.T) {
profile := effectiveProfile()
profile.LLMProfile = " configured-profile "
effective, err := resolveEffectiveProfile(t, profile, ResolveInput{})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if got := effective.Config.Pipelines["main"].LLMProfile; got != " configured-profile " {
t.Fatalf("effective config pipeline llm profile = %q, want preserved programmatic value", got)
}
resolved := effective.ResolvedPipeline
if got := resolved.Chunk.LLMProfile; got != "configured-profile" {
t.Fatalf("resolved chunk profile = %q, want inherited profile", got)
}
if got := resolved.Steps[0].ArtifactLanes[0].Extract.LLMProfile; got != "configured-profile" {
t.Fatalf("resolved extract profile = %q, want inherited profile", got)
} }
} }
@@ -458,7 +504,7 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
if err := pipeline.RegisterArtifactCodec(catalog.ArtifactCodecs, effectiveCodec{}); err != nil { if err := pipeline.RegisterArtifactCodec(catalog.ArtifactCodecs, effectiveCodec{}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := catalog.Inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) { if err := catalog.Inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
return effectiveInput{key: "input"}, nil return effectiveInput{key: "input"}, nil
}); err != nil { }); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -466,6 +512,7 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
chunkSpec := pipeline.ModuleSpec{ chunkSpec := pipeline.ModuleSpec{
Key: pipeline.DefaultChunkModule, Key: pipeline.DefaultChunkModule,
Stage: pipeline.StageChunk, Stage: pipeline.StageChunk,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: []string{"source"}, Requires: []string{"source"},
Provides: []string{"chunk"}, Provides: []string{"chunk"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-ref"}}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-ref"}},
@@ -476,32 +523,32 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
}); err != nil { }); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := catalog.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "needs-capability", Stage: pipeline.StageChunk, Requires: []string{"missing"}}, chunkOptions, func(pipeline.BuildRequest) (contracts.Chunker, error) { if err := catalog.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "needs-capability", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}}, chunkOptions, func(pipeline.BuildRequest) (contracts.Chunker, error) {
return effectiveChunker{key: "needs-capability"}, nil return effectiveChunker{key: "needs-capability"}, nil
}); err != nil { }); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := pipeline.RegisterExtractor(catalog.Extractors, pipeline.ModuleSpec{Key: "extract", Stage: pipeline.StageExtract, ArtifactKind: effectiveArtifactKind, Requires: []string{"chunk"}, Provides: []string{"candidate"}}, func() (contracts.Extractor[effectiveArtifact], error) { if err := pipeline.RegisterExtractor(catalog.Extractors, pipeline.ModuleSpec{Key: "extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"chunk"}, Provides: []string{"candidate"}}, func() (contracts.Extractor[effectiveArtifact], error) {
return effectiveExtractor{key: "extract"}, nil return effectiveExtractor{key: "extract"}, nil
}); err != nil { }); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: pipeline.DefaultMergeModule, Stage: pipeline.StageMerge, ArtifactKind: effectiveArtifactKind, Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) { if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: pipeline.DefaultMergeModule, Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
return effectiveMerger{key: pipeline.DefaultMergeModule}, nil return effectiveMerger{key: pipeline.DefaultMergeModule}, nil
}); err != nil { }); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: "other-merge", Stage: pipeline.StageMerge, ArtifactKind: "other-kind", Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) { if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: "other-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "other-kind", Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
return effectiveMerger{key: "other-merge"}, nil return effectiveMerger{key: "other-merge"}, nil
}); err != nil { }); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := pipeline.RegisterNormalizer(catalog.Normalizers, pipeline.ModuleSpec{Key: pipeline.DefaultNormalizeModule, Stage: pipeline.StageNormalize, ArtifactKind: effectiveArtifactKind, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer[effectiveArtifact], error) { if err := pipeline.RegisterNormalizer(catalog.Normalizers, pipeline.ModuleSpec{Key: pipeline.DefaultNormalizeModule, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer[effectiveArtifact], error) {
return effectiveNormalizer{key: pipeline.DefaultNormalizeModule}, nil return effectiveNormalizer{key: pipeline.DefaultNormalizeModule}, nil
}); err != nil { }); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := catalog.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: pipeline.DefaultOutputModule, Stage: pipeline.StageOutput, Requires: []string{"normalized"}}, func() (contracts.OutputEncoder, error) { if err := catalog.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: pipeline.DefaultOutputModule, Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}}, func() (contracts.OutputEncoder, error) {
return effectiveOutput{key: pipeline.DefaultOutputModule}, nil return effectiveOutput{key: pipeline.DefaultOutputModule}, nil
}); err != nil { }); err != nil {
t.Fatal(err) t.Fatal(err)

View File

@@ -25,9 +25,16 @@ type FileConfig struct {
type FilePromptKitConfig struct { type FilePromptKitConfig struct {
ProfileDir *string `yaml:"profile_dir,omitempty"` ProfileDir *string `yaml:"profile_dir,omitempty"`
ProfileFile *string `yaml:"profile_file,omitempty"` ProfileFile *string `yaml:"profile_file,omitempty"`
LocalBackend *FilePromptKitLocalBackendConfig `yaml:"local_backend,omitempty"`
}
type FilePromptKitLocalBackendConfig struct {
Endpoint *string `yaml:"endpoint,omitempty"`
ConcurrencyLimit *int `yaml:"concurrency_limit,omitempty"`
} }
type FilePipelineProfile struct { type FilePipelineProfile struct {
LLMProfile *string `yaml:"llm_profile,omitempty"`
Input fileModuleBinding `yaml:"input"` Input fileModuleBinding `yaml:"input"`
Chunk *fileModuleBinding `yaml:"chunk,omitempty"` Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"` Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
@@ -36,13 +43,14 @@ type FilePipelineProfile struct {
References map[string]fileReferenceSource `yaml:"references,omitempty"` References map[string]fileReferenceSource `yaml:"references,omitempty"`
artifactsSet bool `yaml:"-"` artifactsSet bool `yaml:"-"`
stepsSet bool `yaml:"-"` stepsSet bool `yaml:"-"`
llmProfileSet bool `yaml:"-"`
} }
func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error { func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
type plainFilePipelineProfile FilePipelineProfile type plainFilePipelineProfile FilePipelineProfile
var decoded plainFilePipelineProfile var decoded plainFilePipelineProfile
seen, err := decodeKnownMapping(node, &decoded, map[string]struct{}{ seen, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
"input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {}, "llm_profile": {}, "input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
}, "pipeline profile") }, "pipeline profile")
if err != nil { if err != nil {
return err return err
@@ -50,6 +58,7 @@ func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
*p = FilePipelineProfile(decoded) *p = FilePipelineProfile(decoded)
_, p.artifactsSet = seen["artifacts"] _, p.artifactsSet = seen["artifacts"]
_, p.stepsSet = seen["steps"] _, p.stepsSet = seen["steps"]
_, p.llmProfileSet = seen["llm_profile"]
return nil return nil
} }
@@ -251,6 +260,9 @@ func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
return err return err
} }
b.LLMProfile = strings.TrimSpace(llmProfile) b.LLMProfile = strings.TrimSpace(llmProfile)
if b.LLMProfile == "" {
return fmt.Errorf("llm_profile must not be empty when set")
}
case "retries": case "retries":
var retries int var retries int
if err := valueNode.Decode(&retries); err != nil { if err := valueNode.Decode(&retries); err != nil {
@@ -468,10 +480,31 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
} }
c.PromptKit.ProfileFile = value c.PromptKit.ProfileFile = value
} }
if fileCfg.PromptKit.LocalBackend != nil {
if fileCfg.PromptKit.LocalBackend.Endpoint == nil {
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
}
endpoint := strings.TrimSpace(*fileCfg.PromptKit.LocalBackend.Endpoint)
if endpoint == "" {
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
}
localBackend := PromptKitLocalBackendConfig{Endpoint: endpoint}
if fileCfg.PromptKit.LocalBackend.ConcurrencyLimit != nil {
localBackend.ConcurrencyLimit = *fileCfg.PromptKit.LocalBackend.ConcurrencyLimit
}
c.PromptKit.LocalBackend = &localBackend
}
} }
for _, pipelineID := range pipelineIDs { for _, pipelineID := range pipelineIDs {
filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]] filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]]
llmProfile := ""
if filePipeline.llmProfileSet || filePipeline.LLMProfile != nil {
if filePipeline.LLMProfile == nil || strings.TrimSpace(*filePipeline.LLMProfile) == "" {
return fmt.Errorf("pipeline %q llm_profile must not be empty when set", pipelineID)
}
llmProfile = strings.TrimSpace(*filePipeline.LLMProfile)
}
hasSteps := filePipeline.stepsSet || filePipeline.Steps != nil hasSteps := filePipeline.stepsSet || filePipeline.Steps != nil
laneIDs, rawLaneIDs, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID)) laneIDs, rawLaneIDs, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID))
if err != nil { if err != nil {
@@ -479,6 +512,7 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
} }
profile := pipeline.PipelineProfile{ profile := pipeline.PipelineProfile{
ID: pipelineID, ID: pipelineID,
LLMProfile: llmProfile,
Input: filePipeline.Input.toPipelineBinding(), Input: filePipeline.Input.toPipelineBinding(),
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)), Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
References: fileReferenceSourcesToPipeline(filePipeline.References), References: fileReferenceSourcesToPipeline(filePipeline.References),

View File

@@ -2,6 +2,7 @@ package config
import ( import (
"encoding/json" "encoding/json"
"fmt"
"os" "os"
"path/filepath" "path/filepath"
"reflect" "reflect"
@@ -49,6 +50,93 @@ func TestFileConfigMinimalVersion4AppliesOverDefaults(t *testing.T) {
} }
} }
func TestFilePipelineLLMProfileIsPresenceAwareAndDetached(t *testing.T) {
const pipelineYAML = `version: 4
pipelines:
main:
%s
input: input
artifacts:
lane:
extract: extract
`
t.Run("omitted", func(t *testing.T) {
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, ""))
if file.Pipelines["main"].LLMProfile != nil || file.Pipelines["main"].llmProfileSet {
t.Fatalf("parsed pipeline profile = %#v, want omitted llm profile", file.Pipelines["main"])
}
cfg := Default()
if err := cfg.ApplyFileConfig(file); err != nil {
t.Fatal(err)
}
if got := cfg.Pipelines["main"].LLMProfile; got != "" {
t.Fatalf("pipeline llm profile = %q, want empty", got)
}
})
t.Run("trimmed and detached", func(t *testing.T) {
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, "llm_profile: ' configured-profile '"))
cfg := Default()
if err := cfg.ApplyFileConfig(file); err != nil {
t.Fatal(err)
}
if got := cfg.Pipelines["main"].LLMProfile; got != "configured-profile" {
t.Fatalf("pipeline llm profile = %q, want trimmed value", got)
}
*file.Pipelines["main"].LLMProfile = "changed-profile"
if got := cfg.Pipelines["main"].LLMProfile; got != "configured-profile" {
t.Fatalf("effective config aliases parsed file: %q", got)
}
if got := cloneConfig(cfg).Pipelines["main"].LLMProfile; got != "configured-profile" {
t.Fatalf("cloned pipeline llm profile = %q", got)
}
data, err := json.Marshal(cfg)
if err != nil {
t.Fatal(err)
}
var roundTripped Config
if err := json.Unmarshal(data, &roundTripped); err != nil {
t.Fatal(err)
}
if got := roundTripped.Pipelines["main"].LLMProfile; got != "configured-profile" {
t.Fatalf("round-tripped pipeline llm profile = %q", got)
}
})
for _, value := range []string{"''", "' '", "null"} {
t.Run("explicit empty "+value, func(t *testing.T) {
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, "llm_profile: "+value))
cfg := Default()
err := cfg.ApplyFileConfig(file)
if err == nil || !strings.Contains(err.Error(), `pipeline "main" llm_profile must not be empty`) {
t.Fatalf("ApplyFileConfig() error = %v, want explicit-empty rejection", err)
}
})
}
}
func TestFileModuleBindingRejectsExplicitEmptyLLMProfile(t *testing.T) {
const configYAML = `version: 4
pipelines:
main:
input:
module: input
llm_profile: %s
artifacts:
lane:
extract: extract
`
for _, value := range []string{"''", "' '", "null"} {
t.Run(value, func(t *testing.T) {
_, err := ParseFileConfigYAML([]byte(fmt.Sprintf(configYAML, value)))
if err == nil || !strings.Contains(err.Error(), "llm_profile must not be empty when set") {
t.Fatalf("ParseFileConfigYAML() error = %v, want explicit-empty binding profile rejection", err)
}
})
}
}
func TestFilePromptKitProfileSourcesSurviveConfigBoundaries(t *testing.T) { func TestFilePromptKitProfileSourcesSurviveConfigBoundaries(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
@@ -97,6 +185,100 @@ func TestFilePromptKitProfileSourcesSurviveConfigBoundaries(t *testing.T) {
} }
} }
func TestFilePromptKitLocalBackendSurvivesConfigBoundaries(t *testing.T) {
tests := []struct {
name string
concurrencyYAML string
wantConcurrency int
}{
{name: "omitted concurrency defaults to zero"},
{name: "positive concurrency is preserved", concurrencyYAML: " concurrency_limit: 2\n", wantConcurrency: 2},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
file := parseFileConfig(t, "version: 4\npromptkit:\n local_backend:\n endpoint: ' http://localhost:8000/v1 '\n"+tt.concurrencyYAML)
cfg := Default()
if err := cfg.ApplyFileConfig(file); err != nil {
t.Fatalf("ApplyFileConfig() error = %v", err)
}
want := PromptKitLocalBackendConfig{
Endpoint: "http://localhost:8000/v1",
ConcurrencyLimit: tt.wantConcurrency,
}
if cfg.PromptKit.LocalBackend == nil || *cfg.PromptKit.LocalBackend != want {
t.Fatalf("local backend config = %#v, want %#v", cfg.PromptKit.LocalBackend, want)
}
*file.PromptKit.LocalBackend.Endpoint = "http://changed.example/v1"
if file.PromptKit.LocalBackend.ConcurrencyLimit != nil {
*file.PromptKit.LocalBackend.ConcurrencyLimit = 99
}
if *cfg.PromptKit.LocalBackend != want {
t.Fatalf("effective config aliases parsed file model: %#v", cfg.PromptKit.LocalBackend)
}
cloned := cloneConfig(cfg)
if cloned.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *cloned.PromptKit.LocalBackend != want {
t.Fatalf("cloned local backend = %#v, want detached %#v", cloned.PromptKit.LocalBackend, want)
}
cloned.PromptKit.LocalBackend.Endpoint = "http://clone.example/v1"
if *cfg.PromptKit.LocalBackend != want {
t.Fatalf("mutating clone changed source config: %#v", cfg.PromptKit.LocalBackend)
}
redacted := cfg.Redacted()
if redacted.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *redacted.PromptKit.LocalBackend != want {
t.Fatalf("redacted local backend = %#v, want detached %#v", redacted.PromptKit.LocalBackend, want)
}
data, err := json.Marshal(cfg)
if err != nil {
t.Fatalf("json.Marshal() error = %v", err)
}
var payload struct {
PromptKit map[string]json.RawMessage `json:"promptkit"`
}
if err := json.Unmarshal(data, &payload); err != nil {
t.Fatalf("json.Unmarshal() error = %v", err)
}
localJSON, ok := payload.PromptKit["local_backend"]
if !ok {
t.Fatalf("runtime PromptKit JSON keys = %v, want local_backend", payload.PromptKit)
}
var localPayload map[string]json.RawMessage
if err := json.Unmarshal(localJSON, &localPayload); err != nil {
t.Fatalf("unmarshal local_backend JSON: %v", err)
}
if _, ok := localPayload["endpoint"]; !ok {
t.Fatalf("runtime local_backend JSON keys = %v, want endpoint", localPayload)
}
if _, ok := localPayload["concurrency_limit"]; !ok {
t.Fatalf("runtime local_backend JSON keys = %v, want concurrency_limit", localPayload)
}
})
}
}
func TestFilePromptKitLocalBackendRequiresEndpoint(t *testing.T) {
for _, tt := range []struct {
name string
yaml string
}{
{name: "missing", yaml: "version: 4\npromptkit:\n local_backend: {}\n"},
{name: "empty", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ''\n"},
{name: "blank", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ' '\n"},
} {
t.Run(tt.name, func(t *testing.T) {
file := parseFileConfig(t, tt.yaml)
cfg := Default()
err := cfg.ApplyFileConfig(file)
if err == nil || !strings.Contains(err.Error(), "promptkit.local_backend.endpoint") {
t.Fatalf("ApplyFileConfig() error = %v, want endpoint field context", err)
}
})
}
}
func TestFilePromptKitExplicitEmptyProfileSourcesAreRejected(t *testing.T) { func TestFilePromptKitExplicitEmptyProfileSourcesAreRejected(t *testing.T) {
for _, field := range []string{"profile_dir", "profile_file"} { for _, field := range []string{"profile_dir", "profile_file"} {
t.Run(field, func(t *testing.T) { t.Run(field, func(t *testing.T) {
@@ -188,6 +370,11 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
yaml: "version: 4\ncache:\n checkpoints:\n enabled: definitely\n", yaml: "version: 4\ncache:\n checkpoints:\n enabled: definitely\n",
want: "cannot unmarshal", want: "cannot unmarshal",
}, },
{
name: "local backend field",
yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: http://localhost:8000/v1\n unknown: true\n",
want: "field unknown not found",
},
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {

View File

@@ -22,7 +22,9 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
ID: "redaction-test", ID: "redaction-test",
Digest: "sha256:safe-digest", Digest: "sha256:safe-digest",
Input: bindings["input"], Input: bindings["input"],
InputExecutionClass: contracts.ExecutionClassDeterministic,
Chunk: bindings["chunk"], Chunk: bindings["chunk"],
ChunkExecutionClass: contracts.ExecutionClassLLMBacked,
ChunkReferences: redactionTestReferenceTarget(pipeline.StageChunk, "", "chunk-reference-content"), ChunkReferences: redactionTestReferenceTarget(pipeline.StageChunk, "", "chunk-reference-content"),
Steps: []pipeline.ResolvedPipelineStep{{ Steps: []pipeline.ResolvedPipelineStep{{
ID: "default", ID: "default",
@@ -30,8 +32,11 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
ID: "safe-lane", ID: "safe-lane",
ArtifactKind: "safe/artifact", ArtifactKind: "safe/artifact",
Extract: bindings["extract"], Extract: bindings["extract"],
ExtractExecutionClass: contracts.ExecutionClassLLMBacked,
Merge: bindings["merge"], Merge: bindings["merge"],
MergeExecutionClass: contracts.ExecutionClassDeterministic,
Normalize: bindings["normalize"], Normalize: bindings["normalize"],
NormalizeExecutionClass: contracts.ExecutionClassLLMBacked,
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]}, Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"), ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"),
MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"), MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"),
@@ -50,6 +55,7 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
}}, }},
}}, }},
Output: bindings["output"], Output: bindings["output"],
OutputExecutionClass: contracts.ExecutionClassDeterministic,
} }
effective := EffectiveConfig{ effective := EffectiveConfig{
Config: Config{Pipelines: map[string]pipeline.PipelineProfile{ Config: Config{Pipelines: map[string]pipeline.PipelineProfile{
@@ -88,6 +94,11 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
t.Fatalf("resolved pipeline summary does not retain %q: %s", safe, text) t.Fatalf("resolved pipeline summary does not retain %q: %s", safe, text)
} }
} }
for _, executionClass := range []string{"input_execution_class\":\"deterministic", "chunk_execution_class\":\"llm_backed", "extract_execution_class\":\"llm_backed", "merge_execution_class\":\"deterministic", "normalize_execution_class\":\"llm_backed", "output_execution_class\":\"deterministic"} {
if !strings.Contains(text, executionClass) {
t.Fatalf("resolved pipeline summary does not retain %q: %s", executionClass, text)
}
}
payload.Input.Options["safe"] = "mutated" payload.Input.Options["safe"] = "mutated"
nested := payload.Input.Options["nested"].([]any)[0].([]any)[0].(map[string]any) nested := payload.Input.Options["nested"].([]any)[0].([]any)[0].(map[string]any)

View File

@@ -2,6 +2,7 @@ package config
import ( import (
"fmt" "fmt"
"net/url"
"sort" "sort"
"strings" "strings"
@@ -53,6 +54,27 @@ func validatePromptKit(cfg PromptKitConfig) error {
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" { if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
return fmt.Errorf("promptkit profile_dir and profile_file are mutually exclusive") return fmt.Errorf("promptkit profile_dir and profile_file are mutually exclusive")
} }
if cfg.LocalBackend == nil {
return nil
}
endpoint := strings.TrimSpace(cfg.LocalBackend.Endpoint)
if endpoint == "" {
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
}
parsed, err := url.Parse(endpoint)
if err != nil ||
(!strings.EqualFold(parsed.Scheme, "http") && !strings.EqualFold(parsed.Scheme, "https")) ||
!parsed.IsAbs() ||
parsed.Hostname() == "" ||
parsed.User != nil ||
parsed.RawQuery != "" ||
parsed.ForceQuery ||
strings.Contains(endpoint, "#") {
return fmt.Errorf("promptkit.local_backend.endpoint must be an absolute HTTP or HTTPS URL with a host and no user information, query, or fragment")
}
if cfg.LocalBackend.ConcurrencyLimit < 0 {
return fmt.Errorf("promptkit.local_backend.concurrency_limit must not be negative")
}
return nil return nil
} }

View File

@@ -93,6 +93,73 @@ func TestValidatePromptKitSourcesAreMutuallyExclusive(t *testing.T) {
assertValidationContains(t, cfg, "promptkit profile_dir and profile_file are mutually exclusive") assertValidationContains(t, cfg, "promptkit profile_dir and profile_file are mutually exclusive")
} }
func TestValidatePromptKitLocalBackendEndpoints(t *testing.T) {
tests := []struct {
name string
endpoint string
profileSource PromptKitConfig
}{
{
name: "HTTP endpoint with path and profile directory",
endpoint: "http://localhost:8000/v1",
profileSource: PromptKitConfig{ProfileDir: "./profiles"},
},
{
name: "case-insensitive HTTPS endpoint and profile file",
endpoint: "HTTPS://inference.example.test/api",
profileSource: PromptKitConfig{ProfileFile: "./profiles.yml"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := Default()
cfg.PromptKit = tt.profileSource
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
Endpoint: tt.endpoint,
ConcurrencyLimit: 2,
}
if err := cfg.Validate(); err != nil {
t.Fatalf("Validate() error = %v", err)
}
})
}
}
func TestValidatePromptKitLocalBackendRejectsInvalidValues(t *testing.T) {
tests := []struct {
name string
endpoint string
concurrencyLimit int
want string
}{
{name: "blank endpoint", endpoint: " ", want: "promptkit.local_backend.endpoint"},
{name: "relative URL", endpoint: "localhost:8000/v1", want: "promptkit.local_backend.endpoint"},
{name: "unsupported scheme", endpoint: "ftp://localhost/model", want: "promptkit.local_backend.endpoint"},
{name: "missing host", endpoint: "http:///v1", want: "promptkit.local_backend.endpoint"},
{name: "user information", endpoint: "http://user:secret@localhost/v1", want: "promptkit.local_backend.endpoint"},
{name: "query", endpoint: "http://localhost/v1?model=example", want: "promptkit.local_backend.endpoint"},
{name: "empty query", endpoint: "http://localhost/v1?", want: "promptkit.local_backend.endpoint"},
{name: "fragment", endpoint: "http://localhost/v1#model", want: "promptkit.local_backend.endpoint"},
{name: "empty fragment", endpoint: "http://localhost/v1#", want: "promptkit.local_backend.endpoint"},
{
name: "negative concurrency",
endpoint: "http://localhost:8000/v1",
concurrencyLimit: -1,
want: "promptkit.local_backend.concurrency_limit",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := Default()
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
Endpoint: tt.endpoint,
ConcurrencyLimit: tt.concurrencyLimit,
}
assertValidationContains(t, cfg, tt.want)
})
}
}
func TestValidateStateSurfaceRules(t *testing.T) { func TestValidateStateSurfaceRules(t *testing.T) {
tests := []struct { tests := []struct {
name string name string

View File

@@ -2,6 +2,7 @@ package debugbundle
import ( import (
"bytes" "bytes"
"encoding/json"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -137,6 +138,47 @@ func TestSummaryWriterWritesEverySummaryArtifact(t *testing.T) {
} }
} }
} }
func TestWriteInvocationPreservesReasoningEffortOverrideStates(t *testing.T) {
replacement := "focused"
cleared := ""
tests := []struct {
name string
override *string
wantValue string
wantSet bool
}{
{name: "inherit"},
{name: "replace", override: &replacement, wantValue: "focused", wantSet: true},
{name: "clear", override: &cleared, wantSet: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
bundle, err := Allocate(t.TempDir(), testBundleRunID, time.Unix(0, 42))
if err != nil {
t.Fatal(err)
}
if err := bundle.Summary().WriteInvocation(Invocation{
Operation: "run",
ReasoningEffortOverride: tt.override,
}); err != nil {
t.Fatal(err)
}
data, err := os.ReadFile(filepath.Join(bundle.SummaryRoot(), ArtifactInvocationMetadata))
if err != nil {
t.Fatal(err)
}
var payload map[string]any
if err := json.Unmarshal(data, &payload); err != nil {
t.Fatal(err)
}
value, found := payload["reasoning_effort_override"]
if found != tt.wantSet || (found && value != tt.wantValue) {
t.Fatalf("reasoning override found=%t value=%#v, want found=%t value=%q; JSON=%s", found, value, tt.wantSet, tt.wantValue, data)
}
})
}
}
func TestSummaryWriterInternalWritesConfineArtifacts(t *testing.T) { func TestSummaryWriterInternalWritesConfineArtifacts(t *testing.T) {
bundle, err := Allocate(t.TempDir(), testBundleRunID, time.Unix(0, 42)) bundle, err := Allocate(t.TempDir(), testBundleRunID, time.Unix(0, 42))
if err != nil { if err != nil {

View File

@@ -39,6 +39,7 @@ type Invocation struct {
ConfigSource string `json:"config_source,omitempty"` ConfigSource string `json:"config_source,omitempty"`
OnlyLanes []string `json:"only_lanes,omitempty"` OnlyLanes []string `json:"only_lanes,omitempty"`
ChunkCacheOverride string `json:"chunk_cache_override,omitempty"` ChunkCacheOverride string `json:"chunk_cache_override,omitempty"`
ReasoningEffortOverride *string `json:"reasoning_effort_override,omitempty"`
RunID string `json:"run_id"` RunID string `json:"run_id"`
StartedAt time.Time `json:"started_at"` StartedAt time.Time `json:"started_at"`
} }
@@ -68,6 +69,10 @@ func (w *SummaryWriter) WriteInvocation(payload Invocation) error {
if payload.StartedAt.IsZero() { if payload.StartedAt.IsZero() {
payload.StartedAt = w.createdAt payload.StartedAt = w.createdAt
} }
if payload.ReasoningEffortOverride != nil {
value := *payload.ReasoningEffortOverride
payload.ReasoningEffortOverride = &value
}
return w.writeJSON(ArtifactInvocationMetadata, payload) return w.writeJSON(ArtifactInvocationMetadata, payload)
} }
func (w *SummaryWriter) WriteRedactedEffectiveConfig(payload RedactedSummaryPayload) error { func (w *SummaryWriter) WriteRedactedEffectiveConfig(payload RedactedSummaryPayload) error {

View File

@@ -43,6 +43,7 @@ type LLMDebugPrompt struct {
PromptVersion string `json:"prompt_version,omitempty"` PromptVersion string `json:"prompt_version,omitempty"`
PromptHash string `json:"prompt_hash,omitempty"` PromptHash string `json:"prompt_hash,omitempty"`
SelectedProfileID string `json:"selected_profile_id,omitempty"` SelectedProfileID string `json:"selected_profile_id,omitempty"`
SelectedBackendID string `json:"selected_backend_id,omitempty"`
SessionID string `json:"session_id,omitempty"` SessionID string `json:"session_id,omitempty"`
RenderedPromptHash string `json:"rendered_prompt_hash,omitempty"` RenderedPromptHash string `json:"rendered_prompt_hash,omitempty"`
Messages []LLMDebugMessage `json:"messages,omitempty"` Messages []LLMDebugMessage `json:"messages,omitempty"`
@@ -266,10 +267,6 @@ const (
ExecutionClassLLMBacked ExecutionClass = "llm_backed" ExecutionClassLLMBacked ExecutionClass = "llm_backed"
) )
type ChunkExecutionClassProvider interface {
ExecutionClass() ExecutionClass
}
type ValidationResult struct { type ValidationResult struct {
Approved bool `json:"approved"` Approved bool `json:"approved"`
ReasonCode string `json:"reason_code,omitempty"` ReasonCode string `json:"reason_code,omitempty"`

View File

@@ -5,3 +5,7 @@ import "errors"
// ErrInvalidStructuredOutput identifies a provider response that cannot satisfy // ErrInvalidStructuredOutput identifies a provider response that cannot satisfy
// the caller's declared structured-output contract. // the caller's declared structured-output contract.
var ErrInvalidStructuredOutput = errors.New("invalid structured output") var ErrInvalidStructuredOutput = errors.New("invalid structured output")
// ErrLLMCapacityExceeded identifies backend admission exhaustion before model
// generation begins.
var ErrLLMCapacityExceeded = errors.New("LLM capacity exceeded")

View File

@@ -23,6 +23,7 @@ type AssetSource struct {
type AssetRegistry struct { type AssetRegistry struct {
prompts []AssetSource prompts []AssetSource
schemas []AssetSource schemas []AssetSource
fallbackProfiles []AssetSource
} }
type AssetHashPart struct { type AssetHashPart struct {
@@ -58,6 +59,20 @@ func (r *AssetRegistry) RegisterSchemaFS(fsys fs.FS, root string) error {
return nil return nil
} }
// RegisterFallbackProfileFS registers profile assets that PromptKit uses only
// when an operator-configured source does not provide a matching profile.
func (r *AssetRegistry) RegisterFallbackProfileFS(fsys fs.FS, root string) error {
if r == nil {
return fmt.Errorf("asset registry must not be nil")
}
source, err := newAssetSource(fsys, root)
if err != nil {
return fmt.Errorf("register fallback profile assets: %w", err)
}
r.fallbackProfiles = append(r.fallbackProfiles, source)
return nil
}
func (r *AssetRegistry) PromptFS() (fs.FS, error) { func (r *AssetRegistry) PromptFS() (fs.FS, error) {
if r == nil { if r == nil {
return nil, fmt.Errorf("asset registry must not be nil") return nil, fmt.Errorf("asset registry must not be nil")
@@ -72,19 +87,76 @@ func (r *AssetRegistry) SchemaFS() (fs.FS, error) {
return flattenAssetSources(r.schemas) return flattenAssetSources(r.schemas)
} }
func (r *AssetRegistry) FallbackProfileFS() (fs.FS, error) {
if r == nil {
return nil, fmt.Errorf("asset registry must not be nil")
}
return flattenAssetSources(r.fallbackProfiles)
}
// FallbackProfileDigest returns a deterministic, non-secret identity for the
// flattened fallback profile assets.
func (r *AssetRegistry) FallbackProfileDigest() (string, error) {
_, digest, _, err := r.fallbackProfileAssets()
return digest, err
}
func (r *AssetRegistry) PromptKitOptions() ([]promptkit.Option, error) { func (r *AssetRegistry) PromptKitOptions() ([]promptkit.Option, error) {
options, _, err := r.promptKitOptions()
return options, err
}
func (r *AssetRegistry) promptKitOptions() ([]promptkit.Option, string, error) {
promptFS, err := r.PromptFS() promptFS, err := r.PromptFS()
if err != nil { if err != nil {
return nil, fmt.Errorf("prepare prompt assets: %w", err) return nil, "", fmt.Errorf("prepare prompt assets: %w", err)
} }
schemaFS, err := r.SchemaFS() schemaFS, err := r.SchemaFS()
if err != nil { if err != nil {
return nil, fmt.Errorf("prepare schema assets: %w", err) return nil, "", fmt.Errorf("prepare schema assets: %w", err)
} }
return []promptkit.Option{ options := []promptkit.Option{
promptkit.WithPromptFS(promptFS, "."), promptkit.WithPromptFS(promptFS, "."),
promptkit.WithSchemaFS(schemaFS, "."), promptkit.WithSchemaFS(schemaFS, "."),
}, nil }
fallbackFS, fallbackDigest, hasFallback, err := r.fallbackProfileAssets()
if err != nil {
return nil, "", err
}
if hasFallback {
options = append(options, promptkit.WithFallbackProfileFS(fallbackFS, "."))
}
return options, fallbackDigest, nil
}
func (r *AssetRegistry) promptKitFallbackProfileOption() (promptkit.Option, bool, error) {
fallbackFS, _, hasFallback, err := r.fallbackProfileAssets()
if err != nil {
return nil, false, err
}
if !hasFallback {
return nil, false, nil
}
return promptkit.WithFallbackProfileFS(fallbackFS, "."), true, nil
}
func (r *AssetRegistry) fallbackProfileAssets() (fs.FS, string, bool, error) {
if r == nil {
return nil, "", false, fmt.Errorf("asset registry must not be nil")
}
if len(r.fallbackProfiles) == 0 {
empty := sha256.Sum256([]byte("notarius:fallback-profile-assets:empty"))
return nil, "sha256:" + hex.EncodeToString(empty[:]), false, nil
}
fallbackFS, err := r.FallbackProfileFS()
if err != nil {
return nil, "", false, fmt.Errorf("prepare fallback profile assets: %w", err)
}
digest, err := hashAssetFilesystem(fallbackFS)
if err != nil {
return nil, "", false, err
}
return fallbackFS, digest, true, nil
} }
func HashAssets(parts []AssetHashPart) (string, error) { func HashAssets(parts []AssetHashPart) (string, error) {
@@ -117,6 +189,28 @@ func HashAssets(parts []AssetHashPart) (string, error) {
return "sha256:" + hex.EncodeToString(hash.Sum(nil)), nil return "sha256:" + hex.EncodeToString(hash.Sum(nil)), nil
} }
func hashAssetFilesystem(fsys fs.FS) (string, error) {
var parts []AssetHashPart
err := fs.WalkDir(fsys, ".", func(name string, entry fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if entry.IsDir() {
return nil
}
parts = append(parts, AssetHashPart{FS: fsys, Path: name})
return nil
})
if err != nil {
return "", fmt.Errorf("walk assets for digest: %w", err)
}
if len(parts) > 0 {
return HashAssets(parts)
}
empty := sha256.Sum256([]byte("notarius:fallback-profile-assets:empty"))
return "sha256:" + hex.EncodeToString(empty[:]), nil
}
func newAssetSource(fsys fs.FS, root string) (AssetSource, error) { func newAssetSource(fsys fs.FS, root string) (AssetSource, error) {
if fsys == nil { if fsys == nil {
return AssetSource{}, fmt.Errorf("filesystem must not be nil") return AssetSource{}, fmt.Errorf("filesystem must not be nil")
@@ -248,6 +342,9 @@ func (m assetMapFS) dirEntries(dir string) []fs.DirEntry {
children[childName] = entry children[childName] = entry
} }
if len(children) == 0 { if len(children) == 0 {
if dir == "." {
return []fs.DirEntry{}
}
return nil return nil
} }
names := make([]string, 0, len(children)) names := make([]string, 0, len(children))

View File

@@ -2,6 +2,7 @@ package llm
import ( import (
"context" "context"
"io/fs"
"strings" "strings"
"testing" "testing"
"testing/fstest" "testing/fstest"
@@ -98,6 +99,89 @@ func TestAssetRegistryRejectsDuplicateAssetPaths(t *testing.T) {
} }
} }
func TestAssetRegistryCombinesFallbackProfileSources(t *testing.T) {
registry := NewAssetRegistry()
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
"first/profiles/one.yaml": {Data: []byte("id: one\nmodel: first\n")},
}, "first/profiles"); err != nil {
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
}
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
"second/two.yaml": {Data: []byte("id: two\nmodel: second\n")},
}, "second"); err != nil {
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
}
fallbackFS, err := registry.FallbackProfileFS()
if err != nil {
t.Fatalf("FallbackProfileFS() error = %v, want nil", err)
}
for _, name := range []string{"one.yaml", "two.yaml"} {
if _, err := fs.ReadFile(fallbackFS, name); err != nil {
t.Fatalf("FallbackProfileFS().ReadFile(%q) error = %v, want nil", name, err)
}
}
}
func TestAssetRegistryRejectsInvalidFallbackProfileRoot(t *testing.T) {
registry := NewAssetRegistry()
err := registry.RegisterFallbackProfileFS(fstest.MapFS{}, "../profiles")
if err == nil || !strings.Contains(err.Error(), "invalid path") {
t.Fatalf("RegisterFallbackProfileFS() error = %v, want invalid root error", err)
}
}
func TestAssetRegistryRejectsUnreadableFallbackProfileAssets(t *testing.T) {
registry := NewAssetRegistry()
if err := registry.RegisterFallbackProfileFS(unreadableAssetFS{}, "."); err != nil {
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
}
_, err := registry.FallbackProfileFS()
if err == nil || !strings.Contains(err.Error(), "permission denied") {
t.Fatalf("FallbackProfileFS() error = %v, want unreadable asset error", err)
}
}
func TestAssetRegistryRejectsDuplicateFallbackProfilePaths(t *testing.T) {
registry := NewAssetRegistry()
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{"first/profile.yaml": {Data: []byte("id: first\n")}}, "first"); err != nil {
t.Fatal(err)
}
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{"second/profile.yaml": {Data: []byte("id: second\n")}}, "second"); err != nil {
t.Fatal(err)
}
_, err := registry.FallbackProfileFS()
if err == nil || !strings.Contains(err.Error(), "duplicate asset path") {
t.Fatalf("FallbackProfileFS() error = %v, want duplicate path error", err)
}
}
func TestAssetRegistryFallbackProfileDigestTracksContentWithoutLeakingIt(t *testing.T) {
digestFor := func(content string) string {
t.Helper()
registry := NewAssetRegistry()
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
"profiles/profile.yaml": {Data: []byte(content)},
}, "profiles"); err != nil {
t.Fatal(err)
}
digest, err := registry.FallbackProfileDigest()
if err != nil {
t.Fatal(err)
}
return digest
}
first := digestFor("id: fallback\nmodel: model-one\n")
second := digestFor("id: fallback\nmodel: model-two\n")
if first == second {
t.Fatalf("fallback profile digests = %q and %q, want content change", first, second)
}
if !strings.HasPrefix(first, "sha256:") || strings.Contains(first, "model-one") || strings.Contains(first, "profile.yaml") {
t.Fatalf("fallback profile digest leaked source details: %q", first)
}
}
func TestAssetRegistryCombinesNamespacedPromptSources(t *testing.T) { func TestAssetRegistryCombinesNamespacedPromptSources(t *testing.T) {
registry := NewAssetRegistry() registry := NewAssetRegistry()
mustRegisterPromptFS(t, registry, fstest.MapFS{ mustRegisterPromptFS(t, registry, fstest.MapFS{
@@ -196,3 +280,9 @@ output:
repair_attempts: 0 repair_attempts: 0
` `
} }
type unreadableAssetFS struct{}
func (unreadableAssetFS) Open(name string) (fs.File, error) {
return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrPermission}
}

View File

@@ -0,0 +1,14 @@
package llm
// CheckpointFingerprint is a stable, non-secret semantic identity contributed
// by the LLM runtime before pipeline execution.
type CheckpointFingerprint struct {
Name string
Value string
}
// CheckpointFingerprintProvider exposes LLM-runtime identities that must
// participate in checkpoint composition.
type CheckpointFingerprintProvider interface {
LLMCheckpointFingerprints() ([]CheckpointFingerprint, error)
}

View File

@@ -3,6 +3,7 @@ package llm
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"net/http" "net/http"
"regexp" "regexp"
@@ -18,19 +19,31 @@ import (
const promptKitProviderName = "promptkit" const promptKitProviderName = "promptkit"
type PromptKitLocalBackendConfig struct {
Endpoint string
ConcurrencyLimit int
}
type PromptKitClientConfig struct { type PromptKitClientConfig struct {
ProfileDir string ProfileDir string
ProfileFile string ProfileFile string
LocalBackend *PromptKitLocalBackendConfig
Assets *AssetRegistry Assets *AssetRegistry
Timeout time.Duration Timeout time.Duration
HTTPClient *http.Client HTTPClient *http.Client
EngineOptions []promptkit.Option EngineOptions []promptkit.Option
Recorder *LLMProfileRecorder Recorder *LLMProfileRecorder
ReasoningEffort *string
} }
type PromptKitClient struct { type PromptKitClient struct {
engine *promptkit.Engine engine *promptkit.Engine
recorder *LLMProfileRecorder recorder *LLMProfileRecorder
profileDir string
profileFile string
localEndpoint string
fallbackProfileDigest string
reasoningEffort *string
} }
type LLMProfileRecorder struct { type LLMProfileRecorder struct {
@@ -41,24 +54,31 @@ type LLMProfileRecorder struct {
var _ contracts.StructuredLLMClient = (*PromptKitClient)(nil) var _ contracts.StructuredLLMClient = (*PromptKitClient)(nil)
var _ contracts.LLMProfileManifestProvider = (*PromptKitClient)(nil) var _ contracts.LLMProfileManifestProvider = (*PromptKitClient)(nil)
func PromptKitLocalBackendOption(cfg PromptKitLocalBackendConfig) promptkit.Option {
return promptkit.WithBackend(promptkit.LocalBackend(cfg.Endpoint, cfg.ConcurrencyLimit))
}
func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) { func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
if cfg.Assets == nil { if cfg.Assets == nil {
return nil, fmt.Errorf("PromptKit client assets must not be nil") return nil, fmt.Errorf("PromptKit client assets must not be nil")
} }
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" { profileSource, profileOptions, err := promptKitProfileSourceEngineOptions(PromptKitProfileSourceConfig{
return nil, fmt.Errorf("PromptKit profile_dir and profile_file are mutually exclusive") ProfileDir: cfg.ProfileDir,
} ProfileFile: cfg.ProfileFile,
options, err := cfg.Assets.PromptKitOptions() LocalBackend: cfg.LocalBackend,
})
if err != nil { if err != nil {
return nil, err return nil, err
} }
if profileFile := strings.TrimSpace(cfg.ProfileFile); profileFile != "" { options, fallbackProfileDigest, err := cfg.Assets.promptKitOptions()
options = append(options, promptkit.WithProfileFile(profileFile)) if err != nil {
return nil, err
} }
options = append(options, profileOptions...)
options = append(options, cfg.EngineOptions...) options = append(options, cfg.EngineOptions...)
engine, err := promptkit.NewEngine(promptkit.Config{ engine, err := promptkit.NewEngine(promptkit.Config{
ProfileDir: strings.TrimSpace(cfg.ProfileDir), ProfileDir: profileSource.ProfileDir,
Timeout: cfg.Timeout, Timeout: cfg.Timeout,
HTTPClient: cfg.HTTPClient, HTTPClient: cfg.HTTPClient,
}, options...) }, options...)
@@ -69,9 +89,19 @@ func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
if recorder == nil { if recorder == nil {
recorder = NewLLMProfileRecorder() recorder = NewLLMProfileRecorder()
} }
var reasoningEffort *string
if cfg.ReasoningEffort != nil {
value := *cfg.ReasoningEffort
reasoningEffort = &value
}
return &PromptKitClient{ return &PromptKitClient{
engine: engine, engine: engine,
recorder: recorder, recorder: recorder,
profileDir: profileSource.ProfileDir,
profileFile: profileSource.ProfileFile,
localEndpoint: profileSource.localEndpoint(),
fallbackProfileDigest: fallbackProfileDigest,
reasoningEffort: reasoningEffort,
}, nil }, nil
} }
@@ -89,33 +119,62 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
if promptID == "" { if promptID == "" {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion prompt_id must not be empty") return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion prompt_id must not be empty")
} }
sessionID := strings.TrimSpace(req.SessionID)
var execution *promptkit.ExecutionTargetOverride
if c.reasoningEffort != nil {
reasoningEffort := *c.reasoningEffort
execution = &promptkit.ExecutionTargetOverride{
ReasoningEffort: &reasoningEffort,
}
}
runReq := promptkit.RunRequest{ runReq := promptkit.RunRequest{
PromptID: promptID, PromptID: promptID,
PromptVersion: strings.TrimSpace(req.PromptVersion), PromptVersion: strings.TrimSpace(req.PromptVersion),
ProfileID: strings.TrimSpace(req.ProfileID), ProfileID: strings.TrimSpace(req.ProfileID),
SessionID: sessionID,
Inputs: promptKitInputs(req.Inputs), Inputs: promptKitInputs(req.Inputs),
Vars: promptKitVars(req), Vars: promptKitVars(req, sessionID),
Metadata: promptKitMetadata(req), Execution: execution,
} }
prepared, err := c.engine.Prepare(ctx, runReq) prepared, err := c.engine.PrepareExecution(ctx, runReq)
if err != nil { if err != nil {
if ctxErr := ctx.Err(); ctxErr != nil { if ctxErr := ctx.Err(); ctxErr != nil {
return contracts.StructuredCompletionResponse{}, ctxErr return contracts.StructuredCompletionResponse{}, ctxErr
} }
return contracts.StructuredCompletionResponse{}, fmt.Errorf("prepare PromptKit prompt %q: %w", promptID, redactPromptKitError(err)) return contracts.StructuredCompletionResponse{}, fmt.Errorf("prepare PromptKit prompt %q: %w", promptID, redactPromptKitError(err))
} }
result, err := c.engine.Run(ctx, runReq) defer prepared.Discard()
preparedDetails := prepared.Details()
result, err := c.engine.RunPrepared(ctx, prepared)
if err != nil { if err != nil {
if ctxErr := ctx.Err(); ctxErr != nil { if ctxErr := ctx.Err(); ctxErr != nil {
return contracts.StructuredCompletionResponse{}, ctxErr return contracts.StructuredCompletionResponse{}, ctxErr
} }
if errors.Is(err, promptkit.ErrCapacityExceeded) {
var capacityErr *promptkit.CapacityError
if errors.As(err, &capacityErr) && strings.TrimSpace(capacityErr.BackendID) != "" {
return contracts.StructuredCompletionResponse{}, fmt.Errorf(
"run PromptKit prompt %q on backend %q: %w: %v",
promptID,
strings.TrimSpace(capacityErr.BackendID),
contracts.ErrLLMCapacityExceeded,
redactPromptKitError(err),
)
}
return contracts.StructuredCompletionResponse{}, fmt.Errorf(
"run PromptKit prompt %q: %w: %v",
promptID,
contracts.ErrLLMCapacityExceeded,
redactPromptKitError(err),
)
}
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w", promptID, redactPromptKitError(err)) return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w", promptID, redactPromptKitError(err))
} }
if result == nil { if result == nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w: empty result", promptID, contracts.ErrInvalidStructuredOutput) return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w: empty result", promptID, contracts.ErrInvalidStructuredOutput)
} }
response := c.responseFromResult(result, prepared) response := c.responseFromResult(result, &preparedDetails)
if result.Validation.Status == promptkit.ValidationFailed || !result.Validation.IsValid { if result.Validation.Status == promptkit.ValidationFailed || !result.Validation.IsValid {
return response, fmt.Errorf("run PromptKit prompt %q: %w: validation failed: %s", promptID, contracts.ErrInvalidStructuredOutput, strings.Join(result.Validation.Errors, "; ")) return response, fmt.Errorf("run PromptKit prompt %q: %w: validation failed: %s", promptID, contracts.ErrInvalidStructuredOutput, strings.Join(result.Validation.Errors, "; "))
} }
@@ -137,6 +196,8 @@ func (c *PromptKitClient) responseFromResult(result *promptkit.RunResult, prepar
ID: strings.TrimSpace(result.SelectedProfileID), ID: strings.TrimSpace(result.SelectedProfileID),
Provider: promptKitProviderName, Provider: promptKitProviderName,
Model: firstNonEmpty(result.ModelName, result.EffectiveModelParams.Model), Model: firstNonEmpty(result.ModelName, result.EffectiveModelParams.Model),
BackendID: strings.TrimSpace(result.SelectedBackendID),
ReasoningEffort: strings.TrimSpace(result.EffectiveModelParams.ReasoningEffort),
} }
if c.recorder != nil { if c.recorder != nil {
c.recorder.Record(profile) c.recorder.Record(profile)
@@ -184,6 +245,7 @@ func promptKitDebugPrompt(prepared *promptkit.PreparedRun) *contracts.LLMDebugPr
PromptVersion: prepared.PromptVersion, PromptVersion: prepared.PromptVersion,
PromptHash: prepared.PromptHash, PromptHash: prepared.PromptHash,
SelectedProfileID: prepared.SelectedProfileID, SelectedProfileID: prepared.SelectedProfileID,
SelectedBackendID: prepared.SelectedBackendID,
SessionID: prepared.SessionID, SessionID: prepared.SessionID,
RenderedPromptHash: prepared.RenderedPromptHash, RenderedPromptHash: prepared.RenderedPromptHash,
Messages: messages, Messages: messages,
@@ -261,6 +323,21 @@ func (c *PromptKitClient) LLMProfileManifests() []artifacts.LLMProfileManifest {
return c.recorder.Manifests() return c.recorder.Manifests()
} }
func (c *PromptKitClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint, error) {
if c == nil {
return nil, nil
}
fingerprint, err := promptKitProfileFingerprint(c.profileDir, c.profileFile, c.fallbackProfileDigest)
if err != nil {
return nil, err
}
fingerprints := []CheckpointFingerprint{fingerprint}
if c.localEndpoint != "" {
fingerprints = append(fingerprints, promptKitLocalBackendFingerprint(c.localEndpoint))
}
return fingerprints, nil
}
func NewLLMProfileRecorder() *LLMProfileRecorder { func NewLLMProfileRecorder() *LLMProfileRecorder {
return &LLMProfileRecorder{profiles: map[string]artifacts.LLMProfileManifest{}} return &LLMProfileRecorder{profiles: map[string]artifacts.LLMProfileManifest{}}
} }
@@ -269,10 +346,8 @@ func (r *LLMProfileRecorder) Record(profile artifacts.LLMProfileManifest) {
if r == nil { if r == nil {
return return
} }
profile.ID = strings.TrimSpace(profile.ID) profile = profile.Normalized()
profile.Provider = strings.TrimSpace(profile.Provider) key := profile.IdentityKey()
profile.Model = strings.TrimSpace(profile.Model)
key := profile.ID + "\x00" + profile.Provider + "\x00" + profile.Model
r.mu.Lock() r.mu.Lock()
defer r.mu.Unlock() defer r.mu.Unlock()
if r.profiles == nil { if r.profiles == nil {
@@ -328,7 +403,7 @@ func promptKitInputs(inputs contracts.LLMInputSet) map[string]promptkit.Artifact
return out return out
} }
func promptKitVars(req contracts.StructuredCompletionRequest) map[string]string { func promptKitVars(req contracts.StructuredCompletionRequest, sessionID string) map[string]string {
vars := make(map[string]string, len(req.Vars)+1) vars := make(map[string]string, len(req.Vars)+1)
for key, value := range req.Vars { for key, value := range req.Vars {
name := strings.TrimSpace(key) name := strings.TrimSpace(key)
@@ -337,7 +412,7 @@ func promptKitVars(req contracts.StructuredCompletionRequest) map[string]string
} }
vars[name] = fmt.Sprint(value) vars[name] = fmt.Sprint(value)
} }
if sessionID := strings.TrimSpace(req.SessionID); sessionID != "" { if sessionID != "" {
vars["session_id"] = sessionID vars["session_id"] = sessionID
} }
if len(vars) == 0 { if len(vars) == 0 {
@@ -346,17 +421,6 @@ func promptKitVars(req contracts.StructuredCompletionRequest) map[string]string
return vars return vars
} }
func promptKitMetadata(req contracts.StructuredCompletionRequest) map[string]string {
metadata := map[string]string{}
if stageName := strings.TrimSpace(req.StageName); stageName != "" {
metadata["stage_name"] = stageName
}
if len(metadata) == 0 {
return nil
}
return metadata
}
var bearerTokenPattern = regexp.MustCompile(`(?i)Bearer\s+[A-Za-z0-9._~+/=-]+`) var bearerTokenPattern = regexp.MustCompile(`(?i)Bearer\s+[A-Za-z0-9._~+/=-]+`)
func redactPromptKitError(err error) error { func redactPromptKitError(err error) error {

View File

@@ -5,7 +5,12 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"io" "io"
"io/fs"
"net/http" "net/http"
"net/http/httptest"
"os"
"path/filepath"
"reflect"
"strings" "strings"
"sync" "sync"
"sync/atomic" "sync/atomic"
@@ -13,6 +18,7 @@ import (
"testing/fstest" "testing/fstest"
"time" "time"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts" "gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/promptkit" "gitea.maximumdirect.net/eric/promptkit"
) )
@@ -26,7 +32,7 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
} }
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{ resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
StageName: "test-stage", StageName: "test-stage",
PromptID: "adapter.test", PromptID: "adapter.direct-session",
PromptVersion: "v1", PromptVersion: "v1",
ProfileID: "explicit-profile", ProfileID: "explicit-profile",
SessionID: " session-123 ", SessionID: " session-123 ",
@@ -50,8 +56,15 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
if resp.Debug == nil || resp.Debug.Prompt == nil { if resp.Debug == nil || resp.Debug.Prompt == nil {
t.Fatalf("debug prompt = nil, want prepared prompt material") t.Fatalf("debug prompt = nil, want prepared prompt material")
} }
if resp.Debug.Prompt.PromptID != "adapter.test" || resp.Debug.Prompt.SelectedProfileID != "explicit-profile" { if resp.Debug.Prompt.PromptID != "adapter.direct-session" ||
t.Fatalf("debug prompt metadata = %#v, want prompt/profile", resp.Debug.Prompt) resp.Debug.Prompt.SelectedProfileID != "explicit-profile" ||
resp.Debug.Prompt.SelectedBackendID != "test-backend" ||
resp.Debug.Prompt.SessionID != "session-123" {
t.Fatalf("debug prompt metadata = %#v, want prompt/profile/backend/session", resp.Debug.Prompt)
}
if resp.Debug.Prompt.EffectiveModelParams["backend_id"] != "test-backend" ||
resp.Debug.Prompt.EffectiveModelParams["reasoning_effort"] != "profile-reasoning" {
t.Fatalf("debug effective model params = %#v, want backend and reasoning", resp.Debug.Prompt.EffectiveModelParams)
} }
if len(resp.Debug.Prompt.Messages) != 1 || !strings.Contains(resp.Debug.Prompt.Messages[0].Content, `{"source":true}`) { if len(resp.Debug.Prompt.Messages) != 1 || !strings.Contains(resp.Debug.Prompt.Messages[0].Content, `{"source":true}`) {
t.Fatalf("debug prompt messages = %#v, want rendered input content", resp.Debug.Prompt.Messages) t.Fatalf("debug prompt messages = %#v, want rendered input content", resp.Debug.Prompt.Messages)
@@ -62,6 +75,10 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
if resp.Debug.Response.Usage.CachedTokens != 5 || resp.Debug.Response.Usage.CacheWriteTokens != 3 { if resp.Debug.Response.Usage.CachedTokens != 5 || resp.Debug.Response.Usage.CacheWriteTokens != 3 {
t.Fatalf("debug usage = %#v, want cached token counts", resp.Debug.Response.Usage) t.Fatalf("debug usage = %#v, want cached token counts", resp.Debug.Response.Usage)
} }
if resp.Debug.Response.EffectiveModelParams["backend_id"] != "test-backend" ||
resp.Debug.Response.EffectiveModelParams["reasoning_effort"] != "profile-reasoning" {
t.Fatalf("debug response effective model params = %#v, want backend and reasoning", resp.Debug.Response.EffectiveModelParams)
}
debugJSON, err := json.Marshal(resp.Debug) debugJSON, err := json.Marshal(resp.Debug)
if err != nil { if err != nil {
t.Fatalf("marshal debug material: %v", err) t.Fatalf("marshal debug material: %v", err)
@@ -76,6 +93,9 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
if gotReq.Target.Model != "explicit-model" { if gotReq.Target.Model != "explicit-model" {
t.Fatalf("model = %q, want explicit-model", gotReq.Target.Model) t.Fatalf("model = %q, want explicit-model", gotReq.Target.Model)
} }
if gotReq.Target.BackendID != "test-backend" {
t.Fatalf("backend id = %q, want test-backend", gotReq.Target.BackendID)
}
if len(gotReq.Prompt.Messages) != 1 || if len(gotReq.Prompt.Messages) != 1 ||
!strings.Contains(gotReq.Prompt.Messages[0].Content, `{"source":true}`) || !strings.Contains(gotReq.Prompt.Messages[0].Content, `{"source":true}`) ||
!strings.Contains(gotReq.Prompt.Messages[0].Content, "value") { !strings.Contains(gotReq.Prompt.Messages[0].Content, "value") {
@@ -88,11 +108,222 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
if len(manifests) != 1 || if len(manifests) != 1 ||
manifests[0].ID != "explicit-profile" || manifests[0].ID != "explicit-profile" ||
manifests[0].Provider != "promptkit" || manifests[0].Provider != "promptkit" ||
manifests[0].Model != "explicit-model" { manifests[0].Model != "explicit-model" ||
manifests[0].BackendID != "test-backend" ||
manifests[0].ReasoningEffort != "profile-reasoning" {
t.Fatalf("profile manifests = %#v", manifests) t.Fatalf("profile manifests = %#v", manifests)
} }
} }
func TestPromptKitClientUsesOnePreparedSnapshotForDebugAndGeneration(t *testing.T) {
const initialPrompt = `id: snapshot.test
version: "v1"
default_profile: snapshot-profile
inputs:
- name: transcript
required: true
content_type: application/json
messages:
- role: user
content: "Snapshot A: {{ input \"transcript\" }}"
output:
format: json
validation_mode: json_schema
schema_path: adapter.schema.json
repair_attempts: 0
`
const updatedPrompt = `id: snapshot.test
version: "v1"
default_profile: snapshot-profile
inputs:
- name: transcript
required: true
content_type: application/json
messages:
- role: user
content: "Snapshot B: {{ input \"transcript\" }}"
output:
format: json
validation_mode: json_schema
schema_path: adapter.schema.json
repair_attempts: 0
`
source := &switchingPromptFS{files: fstest.MapFS{
"snapshot.test.yaml": {Data: []byte(initialPrompt)},
}}
fake := &fakePromptKitLLM{content: `{"ok":true}`}
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
EngineOptions: []promptkit.Option{
promptkit.WithPromptFS(source, "."),
promptkit.WithBackend(promptkit.Backend{ID: "snapshot-backend", Endpoint: "http://promptkit.test/v1"}),
promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
ID: "snapshot-profile",
BackendID: "snapshot-backend",
Model: "snapshot-model",
})),
promptkit.WithLLMClient(fake),
},
})
if err != nil {
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
}
opened := source.holdNextPromptRead()
defer source.resumePromptRead()
type completion struct {
response contracts.StructuredCompletionResponse
err error
}
completed := make(chan completion, 1)
go func() {
var out struct {
OK bool `json:"ok"`
}
response, callErr := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
PromptID: "snapshot.test",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
}, &out)
completed <- completion{response: response, err: callErr}
}()
select {
case <-opened:
case <-time.After(time.Second):
t.Fatal("PromptKit did not read the prompt source")
}
source.replacePrompt([]byte(updatedPrompt))
source.resumePromptRead()
result := <-completed
if result.err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", result.err)
}
if result.response.Debug == nil || result.response.Debug.Prompt == nil || len(result.response.Debug.Prompt.Messages) != 1 {
t.Fatalf("debug prompt = %#v, want one prepared message", result.response.Debug)
}
debugContent := result.response.Debug.Prompt.Messages[0].Content
generatedContent := fake.lastRequest().Prompt.Messages[0].Content
if debugContent != generatedContent {
t.Fatalf("debug content = %q, generation content = %q, want one snapshot", debugContent, generatedContent)
}
if !strings.Contains(debugContent, "Snapshot A") || strings.Contains(debugContent, "Snapshot B") {
t.Fatalf("snapshot content = %q, want the source read before it changed", debugContent)
}
}
func TestPromptKitClientRetainsSessionPromptVariable(t *testing.T) {
fake := &fakePromptKitLLM{content: `{"ok":true}`}
client := newTestPromptKitClient(t, fake)
var out map[string]any
_, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
PromptID: "adapter.test",
SessionID: " canonical-session ",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
Vars: map[string]any{
"custom": "value",
"session_id": "caller-session",
},
}, &out)
if err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", err)
}
gotReq := fake.lastRequest()
if gotReq.Prompt.SessionID != "canonical-session" {
t.Fatalf("session id = %q, want canonical-session", gotReq.Prompt.SessionID)
}
if len(gotReq.Prompt.Messages) != 1 ||
!strings.Contains(gotReq.Prompt.Messages[0].Content, "Session: canonical-session") ||
strings.Contains(gotReq.Prompt.Messages[0].Content, "caller-session") {
t.Fatalf("rendered messages = %#v, want canonical session compatibility variable", gotReq.Prompt.Messages)
}
}
func TestPromptKitClientDoesNotInventDirectSession(t *testing.T) {
fake := &fakePromptKitLLM{content: `{"ok":true}`}
client := newTestPromptKitClient(t, fake)
var out map[string]any
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
PromptID: "adapter.direct-session",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
Vars: map[string]any{"custom": "value"},
}, &out)
if err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", err)
}
if got := fake.lastRequest().Prompt.SessionID; got != "" {
t.Fatalf("session id = %q, want empty", got)
}
if resp.Debug == nil || resp.Debug.Prompt == nil || resp.Debug.Prompt.SessionID != "" {
t.Fatalf("debug prompt = %#v, want no effective session", resp.Debug)
}
}
func TestPromptKitClientAppliesReasoningEffortOverride(t *testing.T) {
tests := []struct {
name string
override func() *string
mutateAfterCreate bool
want string
}{
{
name: "inherit",
want: "profile-reasoning",
},
{
name: "replace",
override: func() *string { value := "focused"; return &value },
want: "focused",
},
{
name: "clear",
override: func() *string { value := ""; return &value },
want: "",
},
{
name: "defensive copy",
override: func() *string { value := "original"; return &value },
mutateAfterCreate: true,
want: "original",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
fake := &fakePromptKitLLM{content: `{"ok":true}`}
var override *string
if tt.override != nil {
override = tt.override()
}
client := newTestPromptKitClientWithReasoning(t, fake, override)
if tt.mutateAfterCreate {
*override = "mutated"
}
var out map[string]any
_, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
PromptID: "adapter.direct-session",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
Vars: map[string]any{"custom": "value"},
}, &out)
if err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", err)
}
if got := fake.lastRequest().Target.ReasoningEffort; got != tt.want {
t.Fatalf("reasoning effort = %q, want %q", got, tt.want)
}
})
}
}
func TestNewPromptKitClientReportsAssetAndEngineConstructionFailures(t *testing.T) { func TestNewPromptKitClientReportsAssetAndEngineConstructionFailures(t *testing.T) {
t.Run("assets", func(t *testing.T) { t.Run("assets", func(t *testing.T) {
registry := NewAssetRegistry() registry := NewAssetRegistry()
@@ -119,6 +350,297 @@ func TestNewPromptKitClientReportsAssetAndEngineConstructionFailures(t *testing.
}) })
} }
func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
const credentialEnvironment = "PROMPTKIT_TEST_API_KEY"
writeProfile := func(model string) {
t.Helper()
content := "id: checkpoint-profile\nendpoint: http://promptkit.test/v1\nmodel: " + model + "\napi_key_env: " + credentialEnvironment + "\n"
if err := os.WriteFile(profilePath, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
}
fingerprintFor := func() CheckpointFingerprint {
t.Helper()
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
ProfileFile: profilePath,
})
if err != nil {
t.Fatal(err)
}
values, err := client.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
if len(values) != 1 || values[0].Name != promptKitProfileFingerprintName {
t.Fatalf("checkpoint fingerprints = %#v, want one profile-source identity", values)
}
return values[0]
}
writeProfile("model-one")
first := fingerprintFor()
repeated := fingerprintFor()
if first != repeated {
t.Fatalf("profile-source fingerprint = %#v then %#v for unchanged source", first, repeated)
}
writeProfile("model-two")
second := fingerprintFor()
if first == second {
t.Fatalf("profile-source fingerprint = %#v for both profile models", first)
}
if strings.TrimSpace(first.Value) == "" {
t.Fatal("profile-source fingerprint is empty")
}
if strings.Contains(first.Value, profilePath) ||
strings.Contains(first.Value, "model-one") ||
strings.Contains(first.Value, credentialEnvironment) {
t.Fatalf("profile-source fingerprint exposes source details: %#v", first)
}
client, err := NewPromptKitClient(PromptKitClientConfig{Assets: newTestPromptKitAssets(t)})
if err != nil {
t.Fatal(err)
}
copy, err := client.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
copy[0].Value = "mutated"
fresh, err := client.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
if fresh[0].Value == "mutated" {
t.Fatal("LLMCheckpointFingerprints exposed mutable backing storage")
}
secondClient, err := NewPromptKitClient(PromptKitClientConfig{Assets: newTestPromptKitAssets(t)})
if err != nil {
t.Fatal(err)
}
secondBuiltin, err := secondClient.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
if len(secondBuiltin) != 1 || fresh[0] != secondBuiltin[0] {
t.Fatalf("built-in profile fingerprints = %#v and %#v, want deterministic identity", fresh, secondBuiltin)
}
if strings.TrimSpace(fresh[0].Value) == "" {
t.Fatal("built-in profile fingerprint is empty")
}
t.Run("directory layout", func(t *testing.T) {
profileDir := t.TempDir()
firstPath := filepath.Join(profileDir, "first-profile.yaml")
secondPath := filepath.Join(profileDir, "second-profile.yaml")
content := []byte("id: directory-profile\nendpoint: http://promptkit.test/v1\nmodel: directory-model\n")
if err := os.WriteFile(firstPath, content, 0o600); err != nil {
t.Fatal(err)
}
first, err := promptKitProfileFingerprint(profileDir, "", "")
if err != nil {
t.Fatal(err)
}
if err := os.Rename(firstPath, secondPath); err != nil {
t.Fatal(err)
}
second, err := promptKitProfileFingerprint(profileDir, "", "")
if err != nil {
t.Fatal(err)
}
if first == second {
t.Fatalf("profile-source fingerprint = %#v after source filename changed", first)
}
if strings.Contains(first.Value, firstPath) || strings.Contains(second.Value, secondPath) {
t.Fatalf("profile-source fingerprint exposes source path: %#v, %#v", first, second)
}
})
}
func TestPromptKitProfileFingerprintReadErrorsDoNotExposeSourcePaths(t *testing.T) {
for _, source := range []struct {
name string
profileDir string
profileFile string
}{
{
name: "file",
profileFile: filepath.Join(t.TempDir(), "missing-profile.yaml"),
},
{
name: "directory",
profileDir: filepath.Join(t.TempDir(), "missing-profiles"),
},
} {
t.Run(source.name, func(t *testing.T) {
_, err := promptKitProfileFingerprint(source.profileDir, source.profileFile, "")
if err == nil {
t.Fatal("promptKitProfileFingerprint() error = nil, want source read failure")
}
if (source.profileDir != "" && strings.Contains(err.Error(), source.profileDir)) ||
(source.profileFile != "" && strings.Contains(err.Error(), source.profileFile)) {
t.Fatalf("fingerprint error exposes profile source path: %q", err)
}
})
}
}
func TestPromptKitClientUsesFallbackProfilesForExecutionAndInspection(t *testing.T) {
assets := newTestPromptKitAssets(t)
const profileID = "fallback-profile"
if err := assets.RegisterFallbackProfileFS(fstest.MapFS{
"profiles/fallback.yaml": {Data: []byte("id: " + profileID + "\nendpoint: http://promptkit.test/v1\nmodel: fallback-model\n")},
}, "profiles"); err != nil {
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
}
fake := &fakePromptKitLLM{content: `{"ok":true}`}
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: assets,
EngineOptions: []promptkit.Option{promptkit.WithLLMClient(fake)},
})
if err != nil {
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
}
var out map[string]any
response, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
PromptID: "adapter.test",
ProfileID: profileID,
SessionID: "fallback-profile-test",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
}, &out)
if err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", err)
}
if response.ProfileID != profileID || response.Model != "fallback-model" {
t.Fatalf("completion response = %#v, want fallback profile", response)
}
inspector, err := NewPromptKitProfileInspector(PromptKitProfileInspectorConfig{Assets: assets})
if err != nil {
t.Fatalf("NewPromptKitProfileInspector() error = %v, want nil", err)
}
inspection, err := inspector.InspectProfile(context.Background(), profileID)
if err != nil {
t.Fatalf("InspectProfile() error = %v, want nil", err)
}
if inspection.ProfileID != profileID || inspection.Model != "fallback-model" {
t.Fatalf("profile inspection = %#v, want fallback profile", inspection)
}
}
func TestPromptKitClientCheckpointFingerprintTracksFallbackProfileAssets(t *testing.T) {
fingerprintFor := func(content string) CheckpointFingerprint {
t.Helper()
assets := newTestPromptKitAssets(t)
if err := assets.RegisterFallbackProfileFS(fstest.MapFS{
"profiles/fallback.yaml": {Data: []byte(content)},
}, "profiles"); err != nil {
t.Fatal(err)
}
client, err := NewPromptKitClient(PromptKitClientConfig{Assets: assets})
if err != nil {
t.Fatal(err)
}
fingerprints, err := client.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
if len(fingerprints) != 1 || fingerprints[0].Name != promptKitProfileFingerprintName {
t.Fatalf("checkpoint fingerprints = %#v, want profile source identity", fingerprints)
}
return fingerprints[0]
}
first := fingerprintFor("id: fallback\nendpoint: http://promptkit.test/v1\nmodel: model-one\n")
second := fingerprintFor("id: fallback\nendpoint: http://promptkit.test/v1\nmodel: model-two\n")
if first == second {
t.Fatalf("checkpoint fingerprints = %#v and %#v, want fallback asset change", first, second)
}
if strings.Contains(first.Value, "model-one") || strings.Contains(first.Value, "fallback.yaml") {
t.Fatalf("checkpoint fingerprint leaked fallback source details: %#v", first)
}
}
func TestPromptKitClientCheckpointFingerprintTracksLocalBackendTarget(t *testing.T) {
const (
firstEndpoint = "http://localhost:8000/v1"
secondEndpoint = "https://inference.example.test/v1"
)
fingerprintsFor := func(endpoint string, concurrencyLimit int) []CheckpointFingerprint {
t.Helper()
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
LocalBackend: &PromptKitLocalBackendConfig{
Endpoint: endpoint,
ConcurrencyLimit: concurrencyLimit,
},
})
if err != nil {
t.Fatal(err)
}
values, err := client.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
return values
}
baseline := fingerprintsFor(firstEndpoint, 0)
if len(baseline) != 2 ||
baseline[0].Name != promptKitProfileFingerprintName ||
baseline[1].Name != promptKitLocalBackendFingerprintName {
t.Fatalf("checkpoint fingerprints = %#v, want profile source then local backend target", baseline)
}
endpointChanged := fingerprintsFor(secondEndpoint, 0)
if baseline[0] != endpointChanged[0] || baseline[1] == endpointChanged[1] {
t.Fatalf("endpoint fingerprints = %#v and %#v, want only local target to change", baseline, endpointChanged)
}
concurrencyChanged := fingerprintsFor(firstEndpoint, 4)
if !reflect.DeepEqual(baseline, concurrencyChanged) {
t.Fatalf("concurrency fingerprints = %#v, want %#v", concurrencyChanged, baseline)
}
for _, values := range [][]CheckpointFingerprint{baseline, endpointChanged} {
for _, fingerprint := range values {
if strings.Contains(fingerprint.Value, firstEndpoint) ||
strings.Contains(fingerprint.Value, secondEndpoint) {
t.Fatalf("checkpoint fingerprint exposes endpoint: %#v", fingerprint)
}
}
}
localBackend := &PromptKitLocalBackendConfig{
Endpoint: " " + firstEndpoint + " ",
ConcurrencyLimit: 0,
}
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
LocalBackend: localBackend,
})
if err != nil {
t.Fatal(err)
}
localBackend.Endpoint = secondEndpoint
copy, err := client.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(copy, baseline) {
t.Fatalf("fingerprints after input mutation = %#v, want retained target %#v", copy, baseline)
}
copy[0].Value = "mutated-profile"
copy[1].Value = "mutated-target"
fresh, err := client.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(fresh, baseline) {
t.Fatalf("fingerprints after returned-slice mutation = %#v, want %#v", fresh, baseline)
}
}
func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) { func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) {
fake := &fakePromptKitLLM{content: `{"ok":true}`} fake := &fakePromptKitLLM{content: `{"ok":true}`}
client := newTestPromptKitClient(t, fake) client := newTestPromptKitClient(t, fake)
@@ -136,6 +658,112 @@ func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testi
if got := fake.lastRequest().Target.Model; got != "default-model" { if got := fake.lastRequest().Target.Model; got != "default-model" {
t.Fatalf("model = %q, want prompt default profile model", got) t.Fatalf("model = %q, want prompt default profile model", got)
} }
manifests := client.LLMProfileManifests()
if len(manifests) != 1 || manifests[0].BackendID != "" || manifests[0].ReasoningEffort != "profile-reasoning" {
t.Fatalf("endpoint-only profile manifests = %#v, want omitted backend and effective reasoning", manifests)
}
}
func TestPromptKitClientUsesConfiguredLocalBackend(t *testing.T) {
var providerCalls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
providerCalls.Add(1)
if r.URL.Path != "/v1/chat/completions" {
t.Errorf("provider path = %q, want /v1/chat/completions", r.URL.Path)
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{
"choices": [{"message": {"role": "assistant", "content": "{\"ok\":true}"}}],
"usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7}
}`))
}))
defer server.Close()
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
if err := os.WriteFile(profilePath, []byte(`id: local-profile
backend: local
model: local-model
`), 0o600); err != nil {
t.Fatal(err)
}
localBackend := &PromptKitLocalBackendConfig{
Endpoint: " " + server.URL + "/v1 ",
ConcurrencyLimit: 2,
}
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
ProfileFile: profilePath,
LocalBackend: localBackend,
})
if err != nil {
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
}
localBackend.Endpoint = "http://127.0.0.1:1/v1"
request := contracts.StructuredCompletionRequest{
PromptID: "adapter.direct-session",
ProfileID: "local-profile",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
Vars: map[string]any{"custom": "value"},
}
var out map[string]any
response, err := client.CompleteStructured(context.Background(), request, &out)
if err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", err)
}
if providerCalls.Load() != 1 {
t.Fatalf("provider calls = %d, want 1", providerCalls.Load())
}
if response.ProfileID != "local-profile" || response.Model != "local-model" {
t.Fatalf("response metadata = %#v", response)
}
if response.Debug == nil || response.Debug.Prompt == nil ||
response.Debug.Prompt.SelectedBackendID != promptkit.BackendLocal {
t.Fatalf("response debug prompt = %#v, want local backend", response.Debug)
}
manifests := client.LLMProfileManifests()
if len(manifests) != 1 || manifests[0].BackendID != promptkit.BackendLocal {
t.Fatalf("profile manifests = %#v, want local backend", manifests)
}
clientWithoutRegistration, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
ProfileFile: profilePath,
})
if err != nil {
t.Fatalf("NewPromptKitClient() without local registration error = %v, want nil", err)
}
_, err = clientWithoutRegistration.CompleteStructured(context.Background(), request, &out)
if err == nil ||
!strings.Contains(err.Error(), "prepare PromptKit prompt") ||
!strings.Contains(err.Error(), promptkit.BackendLocal) {
t.Fatalf("CompleteStructured() without registration error = %v, want preparation failure with local backend context", err)
}
if providerCalls.Load() != 1 {
t.Fatalf("provider calls after missing-registration failure = %d, want 1", providerCalls.Load())
}
}
func TestLLMProfileRecorderDistinguishesEffectiveTargets(t *testing.T) {
recorder := NewLLMProfileRecorder()
for _, profile := range []artifacts.LLMProfileManifest{
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-b", ReasoningEffort: "low"},
{ID: " profile ", Provider: " promptkit ", Model: " model ", BackendID: " backend-a ", ReasoningEffort: " low "},
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "high"},
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "low"},
} {
recorder.Record(profile)
}
want := []artifacts.LLMProfileManifest{
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "high"},
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "low"},
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-b", ReasoningEffort: "low"},
}
if got := recorder.Manifests(); !reflect.DeepEqual(got, want) {
t.Fatalf("profile manifests = %#v, want %#v", got, want)
}
} }
func TestPromptKitClientValidationFailureReturnsError(t *testing.T) { func TestPromptKitClientValidationFailureReturnsError(t *testing.T) {
@@ -213,6 +841,94 @@ func TestPromptKitClientProviderFailureIncludesContextAndRedactsBearerToken(t *t
} }
} }
func TestPromptKitClientTranslatesBackendCapacityExhaustion(t *testing.T) {
queueCapacity := 0
fake := &fakePromptKitLLM{
err: errors.New("provider failed with Bearer secret-token"),
block: make(chan struct{}),
}
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
EngineOptions: []promptkit.Option{
promptkit.WithBackend(promptkit.Backend{
ID: "limited-backend",
Endpoint: "http://127.0.0.1:1/v1",
ConcurrencyLimit: 1,
QueueCapacity: &queueCapacity,
}),
promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
ID: "limited-profile",
BackendID: "limited-backend",
Model: "limited-model",
})),
promptkit.WithLLMClient(fake),
},
})
if err != nil {
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
}
defer func() {
select {
case <-fake.block:
default:
close(fake.block)
}
}()
request := contracts.StructuredCompletionRequest{
PromptID: "adapter.direct-session",
ProfileID: "limited-profile",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
Vars: map[string]any{"custom": "value"},
}
firstResult := make(chan error, 1)
go func() {
var out map[string]any
_, callErr := client.CompleteStructured(context.Background(), request, &out)
firstResult <- callErr
}()
waitForAtomicAtLeast(t, &fake.calls, 1)
var out map[string]any
response, capacityErr := client.CompleteStructured(context.Background(), request, &out)
if len(response.Content) != 0 {
t.Fatalf("capacity response = %#v, want empty", response)
}
if !errors.Is(capacityErr, contracts.ErrLLMCapacityExceeded) {
t.Fatalf("capacity error = %v, want ErrLLMCapacityExceeded", capacityErr)
}
if errors.Is(capacityErr, contracts.ErrInvalidStructuredOutput) {
t.Fatalf("capacity error = %v, must not be invalid structured output", capacityErr)
}
if errors.Is(capacityErr, promptkit.ErrCapacityExceeded) {
t.Fatalf("capacity error exposes PromptKit sentinel: %v", capacityErr)
}
if !strings.Contains(capacityErr.Error(), `run PromptKit prompt "adapter.direct-session"`) ||
!strings.Contains(capacityErr.Error(), `backend "limited-backend"`) ||
!strings.Contains(capacityErr.Error(), "backend capacity exceeded") {
t.Fatalf("capacity error = %q, want prompt, backend, and upstream diagnostic context", capacityErr)
}
var upstreamCapacityErr *promptkit.CapacityError
if errors.As(capacityErr, &upstreamCapacityErr) {
t.Fatalf("capacity error exposes PromptKit capacity type: %v", capacityErr)
}
if calls := atomic.LoadInt32(&fake.calls); calls != 1 {
t.Fatalf("provider calls after capacity rejection = %d, want 1", calls)
}
close(fake.block)
firstErr := <-firstResult
if firstErr == nil || strings.Contains(firstErr.Error(), "secret-token") ||
!strings.Contains(firstErr.Error(), "Bearer [REDACTED]") {
t.Fatalf("admitted provider error = %v, want redacted diagnostic", firstErr)
}
if calls := atomic.LoadInt32(&fake.calls); calls != 1 {
t.Fatalf("provider calls after release = %d, want no adapter retry", calls)
}
}
func TestPromptKitClientContextCancellationIsRespected(t *testing.T) { func TestPromptKitClientContextCancellationIsRespected(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
cancel() cancel()
@@ -346,21 +1062,32 @@ func TestPromptKitClientValidatesRequest(t *testing.T) {
} }
func newTestPromptKitClient(t *testing.T, fake *fakePromptKitLLM) *PromptKitClient { func newTestPromptKitClient(t *testing.T, fake *fakePromptKitLLM) *PromptKitClient {
return newTestPromptKitClientWithReasoning(t, fake, nil)
}
func newTestPromptKitClientWithReasoning(t *testing.T, fake *fakePromptKitLLM, reasoningEffort *string) *PromptKitClient {
t.Helper() t.Helper()
registry := newTestPromptKitAssets(t) registry := newTestPromptKitAssets(t)
client, err := NewPromptKitClient(PromptKitClientConfig{ client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: registry, Assets: registry,
ReasoningEffort: reasoningEffort,
EngineOptions: []promptkit.Option{ EngineOptions: []promptkit.Option{
promptkit.WithBackend(promptkit.Backend{
ID: "test-backend",
Endpoint: "http://127.0.0.1:1/v1",
}),
promptkit.WithProfiles( promptkit.WithProfiles(
promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{ promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
ID: "default-profile", ID: "default-profile",
Endpoint: "http://127.0.0.1:1/v1", Endpoint: "http://127.0.0.1:1/v1",
Model: "default-model", Model: "default-model",
ReasoningEffort: "profile-reasoning",
}), }),
promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{ promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
ID: "explicit-profile", ID: "explicit-profile",
Endpoint: "http://127.0.0.1:1/v1", BackendID: "test-backend",
Model: "explicit-model", Model: "explicit-model",
ReasoningEffort: "profile-reasoning",
}), }),
), ),
promptkit.WithLLMClient(fake), promptkit.WithLLMClient(fake),
@@ -380,6 +1107,22 @@ func newTestPromptKitAssets(t *testing.T) *AssetRegistry {
version: "v1" version: "v1"
default_profile: default-profile default_profile: default-profile
session_id: "{{ .session_id }}" session_id: "{{ .session_id }}"
inputs:
- name: transcript
required: true
content_type: application/json
messages:
- role: user
content: "Transcript: {{ input \"transcript\" }} Custom: {{ index . \"custom\" }} Session: {{ .session_id }}"
output:
format: json
validation_mode: json_schema
schema_path: adapter.schema.json
repair_attempts: 0
`)},
"adapter.direct-session.yaml": {Data: []byte(`id: adapter.direct-session
version: "v1"
default_profile: default-profile
inputs: inputs:
- name: transcript - name: transcript
required: true required: true
@@ -404,6 +1147,61 @@ output:
return registry return registry
} }
type switchingPromptFS struct {
mu sync.Mutex
files fstest.MapFS
pauseNextOpen bool
opened chan struct{}
resume chan struct{}
}
func (f *switchingPromptFS) Open(name string) (fs.File, error) {
f.mu.Lock()
file, err := f.files.Open(name)
pause := f.pauseNextOpen && name == "snapshot.test.yaml"
resume := f.resume
if pause {
f.pauseNextOpen = false
close(f.opened)
}
f.mu.Unlock()
if pause {
<-resume
}
return file, err
}
func (f *switchingPromptFS) ReadDir(name string) ([]fs.DirEntry, error) {
f.mu.Lock()
defer f.mu.Unlock()
return f.files.ReadDir(name)
}
func (f *switchingPromptFS) holdNextPromptRead() <-chan struct{} {
f.mu.Lock()
defer f.mu.Unlock()
f.pauseNextOpen = true
f.opened = make(chan struct{})
f.resume = make(chan struct{})
return f.opened
}
func (f *switchingPromptFS) replacePrompt(content []byte) {
f.mu.Lock()
defer f.mu.Unlock()
f.files["snapshot.test.yaml"] = &fstest.MapFile{Data: append([]byte(nil), content...)}
}
func (f *switchingPromptFS) resumePromptRead() {
f.mu.Lock()
resume := f.resume
f.resume = nil
f.mu.Unlock()
if resume != nil {
close(resume)
}
}
type fakePromptKitLLM struct { type fakePromptKitLLM struct {
content string content string
allowEmpty bool allowEmpty bool

View File

@@ -0,0 +1,110 @@
package llm
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"io/fs"
"os"
"path/filepath"
"sort"
"strings"
)
const (
promptKitProfileFingerprintName = "promptkit_profile_source"
promptKitLocalBackendFingerprintName = "promptkit_local_backend_target"
promptKitLocalBackendMarker = "notarius:promptkit-local-backend:v1"
// The built-in profile catalog is compiled into this pinned PromptKit
// release. Update this identity when the dependency is upgraded.
promptKitBuiltinProfileCatalogID = "promptkit:v0.5.0:builtin-profiles"
)
func promptKitProfileFingerprint(profileDir, profileFile, fallbackProfileDigest string) (CheckpointFingerprint, error) {
hasher := sha256.New()
writeFingerprintPart(hasher, []byte(promptKitBuiltinProfileCatalogID))
writeFingerprintPart(hasher, []byte(strings.TrimSpace(fallbackProfileDigest)))
switch {
case strings.TrimSpace(profileFile) != "":
cleanProfileFile := strings.TrimSpace(profileFile)
data, err := os.ReadFile(cleanProfileFile)
if err != nil {
return CheckpointFingerprint{}, fmt.Errorf("read PromptKit profile file for checkpoint identity")
}
writeFingerprintPart(hasher, []byte(filepath.ToSlash(filepath.Base(cleanProfileFile))))
writeFingerprintPart(hasher, data)
case strings.TrimSpace(profileDir) != "":
files, err := promptKitProfileFiles(strings.TrimSpace(profileDir))
if err != nil {
return CheckpointFingerprint{}, err
}
for _, file := range files {
writeFingerprintPart(hasher, []byte(file.path))
writeFingerprintPart(hasher, file.digest)
}
}
return CheckpointFingerprint{
Name: promptKitProfileFingerprintName,
Value: "sha256:" + hex.EncodeToString(hasher.Sum(nil)),
}, nil
}
func promptKitLocalBackendFingerprint(endpoint string) CheckpointFingerprint {
hasher := sha256.New()
writeFingerprintPart(hasher, []byte(promptKitLocalBackendMarker))
writeFingerprintPart(hasher, []byte(strings.TrimSpace(endpoint)))
return CheckpointFingerprint{
Name: promptKitLocalBackendFingerprintName,
Value: "sha256:" + hex.EncodeToString(hasher.Sum(nil)),
}
}
type promptKitProfileFile struct {
path string
digest []byte
}
func promptKitProfileFiles(root string) ([]promptKitProfileFile, error) {
var files []promptKitProfileFile
err := filepath.WalkDir(root, func(name string, entry fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if entry.IsDir() {
return nil
}
extension := filepath.Ext(entry.Name())
if extension != ".yaml" && extension != ".yml" {
return nil
}
data, err := os.ReadFile(name)
if err != nil {
return err
}
relativePath, err := filepath.Rel(root, name)
if err != nil {
return err
}
sum := sha256.Sum256(data)
files = append(files, promptKitProfileFile{
path: filepath.ToSlash(relativePath),
digest: append([]byte(nil), sum[:]...),
})
return nil
})
if err != nil {
return nil, fmt.Errorf("read PromptKit profile directory for checkpoint identity")
}
sort.Slice(files, func(i, j int) bool {
return files[i].path < files[j].path
})
return files, nil
}
func writeFingerprintPart(hasher interface{ Write([]byte) (int, error) }, value []byte) {
length := []byte(fmt.Sprintf("%d:", len(value)))
_, _ = hasher.Write(length)
_, _ = hasher.Write(value)
}

View File

@@ -0,0 +1,146 @@
package llm
import (
"context"
"errors"
"fmt"
"strings"
"gitea.maximumdirect.net/eric/promptkit"
)
type PromptKitProfileSourceConfig struct {
ProfileDir string
ProfileFile string
LocalBackend *PromptKitLocalBackendConfig
}
type PromptKitProfileInspectorConfig struct {
Source PromptKitProfileSourceConfig
Assets *AssetRegistry
}
func (c PromptKitProfileSourceConfig) localEndpoint() string {
if c.LocalBackend == nil {
return ""
}
return c.LocalBackend.Endpoint
}
type PromptKitProfileInspector struct {
engine *promptkit.Engine
}
type PromptKitProfileInspection struct {
ProfileID string
BackendID string
Model string
CredentialEnvironment string
CredentialRequired bool
}
type PromptKitProfileInspectionError struct {
ProfileID string
err error
}
func (e *PromptKitProfileInspectionError) Error() string {
switch {
case errors.Is(e.err, promptkit.ErrProfileNotFound):
return fmt.Sprintf("PromptKit profile %q is not configured", e.ProfileID)
case errors.Is(e.err, promptkit.ErrInvalidRequest):
return fmt.Sprintf("PromptKit profile ID %q is invalid", e.ProfileID)
case errors.Is(e.err, promptkit.ErrProfileLoad):
return fmt.Sprintf("PromptKit profile %q is invalid or unreadable", e.ProfileID)
default:
return fmt.Sprintf("PromptKit profile %q could not be inspected", e.ProfileID)
}
}
func (e *PromptKitProfileInspectionError) Unwrap() error {
return e.err
}
type promptKitProfileConfigurationError struct {
err error
}
func (e *promptKitProfileConfigurationError) Error() string {
return "PromptKit profile configuration is invalid or unreadable"
}
func (e *promptKitProfileConfigurationError) Unwrap() error {
return e.err
}
func NewPromptKitProfileInspector(cfg PromptKitProfileInspectorConfig) (*PromptKitProfileInspector, error) {
source, options, err := promptKitProfileSourceEngineOptions(cfg.Source)
if err != nil {
return nil, err
}
if cfg.Assets != nil {
fallbackOption, hasFallback, err := cfg.Assets.promptKitFallbackProfileOption()
if err != nil {
return nil, err
}
if hasFallback {
options = append(options, fallbackOption)
}
}
engine, err := promptkit.NewEngine(promptkit.Config{
PromptDir: ".",
ProfileDir: source.ProfileDir,
}, options...)
if err != nil {
return nil, &promptKitProfileConfigurationError{err: err}
}
return &PromptKitProfileInspector{engine: engine}, nil
}
func (i *PromptKitProfileInspector) InspectProfile(ctx context.Context, profileID string) (PromptKitProfileInspection, error) {
if i == nil || i.engine == nil {
return PromptKitProfileInspection{}, fmt.Errorf("PromptKit profile inspector must not be nil")
}
profileID = strings.TrimSpace(profileID)
inspection, err := i.engine.InspectProfile(ctx, profileID)
if err != nil {
if ctxErr := ctx.Err(); ctxErr != nil {
return PromptKitProfileInspection{}, ctxErr
}
return PromptKitProfileInspection{}, &PromptKitProfileInspectionError{
ProfileID: profileID,
err: err,
}
}
return PromptKitProfileInspection{
ProfileID: inspection.ProfileID,
BackendID: strings.TrimSpace(inspection.EffectiveModelParams.BackendID),
Model: strings.TrimSpace(inspection.EffectiveModelParams.Model),
CredentialEnvironment: strings.TrimSpace(inspection.EffectiveModelParams.APIKeyEnv),
CredentialRequired: inspection.APIKeyRequired,
}, nil
}
func promptKitProfileSourceEngineOptions(cfg PromptKitProfileSourceConfig) (PromptKitProfileSourceConfig, []promptkit.Option, error) {
source := PromptKitProfileSourceConfig{
ProfileDir: strings.TrimSpace(cfg.ProfileDir),
ProfileFile: strings.TrimSpace(cfg.ProfileFile),
}
if source.ProfileDir != "" && source.ProfileFile != "" {
return PromptKitProfileSourceConfig{}, nil, fmt.Errorf("PromptKit profile_dir and profile_file are mutually exclusive")
}
if cfg.LocalBackend != nil {
localBackend := *cfg.LocalBackend
localBackend.Endpoint = strings.TrimSpace(localBackend.Endpoint)
source.LocalBackend = &localBackend
}
var options []promptkit.Option
if source.ProfileFile != "" {
options = append(options, promptkit.WithProfileFile(source.ProfileFile))
}
if source.LocalBackend != nil {
options = append(options, PromptKitLocalBackendOption(*source.LocalBackend))
}
return source, options, nil
}

View File

@@ -53,3 +53,14 @@ func (c *scheduledClient) LLMProfileManifests() []artifacts.LLMProfileManifest {
} }
return provider.LLMProfileManifests() return provider.LLMProfileManifests()
} }
func (c *scheduledClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint, error) {
if c == nil || c.client == nil {
return nil, nil
}
provider, ok := c.client.(CheckpointFingerprintProvider)
if !ok {
return nil, nil
}
return provider.LLMCheckpointFingerprints()
}

View File

@@ -75,6 +75,33 @@ func TestScheduledClientPropagatesSchedulerError(t *testing.T) {
} }
} }
func TestScheduledClientPreservesCheckpointFingerprints(t *testing.T) {
scheduler, err := NewScheduler(1)
if err != nil {
t.Fatal(err)
}
inner := &fingerprintedStructuredClient{
fingerprints: []CheckpointFingerprint{
{Name: "profile_source", Value: "sha256:one"},
{Name: "backend_target", Value: "sha256:two"},
},
}
client := NewScheduledClient(inner, scheduler)
provider, ok := client.(CheckpointFingerprintProvider)
if !ok {
t.Fatalf("scheduled client %T does not preserve checkpoint fingerprints", client)
}
got, err := provider.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
if len(got) != len(inner.fingerprints) ||
got[0] != inner.fingerprints[0] ||
got[1] != inner.fingerprints[1] {
t.Fatalf("checkpoint fingerprints = %#v, want %#v", got, inner.fingerprints)
}
}
type blockingStructuredClient struct { type blockingStructuredClient struct {
release chan struct{} release chan struct{}
inFlight int32 inFlight int32
@@ -110,6 +137,15 @@ type errorStructuredClient struct {
err error err error
} }
type fingerprintedStructuredClient struct {
errorStructuredClient
fingerprints []CheckpointFingerprint
}
func (c *fingerprintedStructuredClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint, error) {
return append([]CheckpointFingerprint(nil), c.fingerprints...), nil
}
func (c *errorStructuredClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) { func (c *errorStructuredClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
return contracts.StructuredCompletionResponse{}, c.err return contracts.StructuredCompletionResponse{}, c.err
} }

View File

@@ -24,10 +24,6 @@ func NewChunkerRegistry() *ChunkerRegistry {
} }
} }
func (r *ChunkerRegistry) Register(key string, constructor ChunkerConstructor) error {
return r.RegisterWithSpec(defaultModuleSpec(key, StageChunk), constructor)
}
func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerConstructor) error { func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerConstructor) error {
if constructor == nil { if constructor == nil {
return fmt.Errorf("chunker constructor for %q must not be nil", strings.TrimSpace(spec.Key)) return fmt.Errorf("chunker constructor for %q must not be nil", strings.TrimSpace(spec.Key))

View File

@@ -29,6 +29,10 @@ type registryBehaviorCase[M any] struct {
moduleKey func(M) string moduleKey func(M) string
} }
func testModuleSpec(key string, stage ModuleStage) ModuleSpec {
return ModuleSpec{Key: key, Stage: stage, ExecutionClass: contracts.ExecutionClassDeterministic}
}
func TestChunkerRegistryBehavior(t *testing.T) { func TestChunkerRegistryBehavior(t *testing.T) {
runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Chunker]{ runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Chunker]{
name: "ChunkerRegistry", name: "ChunkerRegistry",
@@ -39,7 +43,9 @@ func TestChunkerRegistryBehavior(t *testing.T) {
return NewChunkerRegistry() return NewChunkerRegistry()
}, },
register: func(registry any, key string, constructor func() (contracts.Chunker, error)) error { register: func(registry any, key string, constructor func() (contracts.Chunker, error)) error {
return registry.(*ChunkerRegistry).Register(key, constructor) return registry.(*ChunkerRegistry).RegisterWithSpec(ModuleSpec{
Key: key, Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic,
}, constructor)
}, },
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Chunker, error)) error { registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Chunker, error)) error {
return registry.(*ChunkerRegistry).RegisterWithSpec(spec, constructor) return registry.(*ChunkerRegistry).RegisterWithSpec(spec, constructor)
@@ -55,7 +61,9 @@ func TestChunkerRegistryBehavior(t *testing.T) {
}, },
nilRegister: func(key string, constructor func() (contracts.Chunker, error)) error { nilRegister: func(key string, constructor func() (contracts.Chunker, error)) error {
var registry *ChunkerRegistry var registry *ChunkerRegistry
return registry.Register(key, constructor) return registry.RegisterWithSpec(ModuleSpec{
Key: key, Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic,
}, constructor)
}, },
nilBuild: func(key string) (contracts.Chunker, error) { nilBuild: func(key string) (contracts.Chunker, error) {
var registry *ChunkerRegistry var registry *ChunkerRegistry
@@ -103,6 +111,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
spec := ModuleSpec{ spec := ModuleSpec{
Key: " " + testCase.key + " ", Key: " " + testCase.key + " ",
Stage: testCase.stage, Stage: testCase.stage,
ExecutionClass: contracts.ExecutionClassDeterministic,
Provides: []string{" beta ", "alpha", "", "beta"}, Provides: []string{" beta ", "alpha", "", "beta"},
Requires: []string{" source ", "source", ""}, Requires: []string{" source ", "source", ""},
} }
@@ -117,6 +126,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
want := ModuleSpec{ want := ModuleSpec{
Key: testCase.key, Key: testCase.key,
Stage: testCase.stage, Stage: testCase.stage,
ExecutionClass: contracts.ExecutionClassDeterministic,
Provides: []string{"alpha", "beta"}, Provides: []string{"alpha", "beta"},
Requires: []string{"source"}, Requires: []string{"source"},
} }
@@ -134,7 +144,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
} }
}) })
t.Run(testCase.name+"/default spec from register", func(t *testing.T) { t.Run(testCase.name+"/minimal explicit spec registration", func(t *testing.T) {
registry := testCase.newRegistry() registry := testCase.newRegistry()
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil { if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
t.Fatalf("Register() error = %v, want nil", err) t.Fatalf("Register() error = %v, want nil", err)
@@ -144,7 +154,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
if !ok { if !ok {
t.Fatal("Spec() ok = false, want true") t.Fatal("Spec() ok = false, want true")
} }
want := ModuleSpec{Key: testCase.key, Stage: testCase.stage} want := ModuleSpec{Key: testCase.key, Stage: testCase.stage, ExecutionClass: contracts.ExecutionClassDeterministic}
if !reflect.DeepEqual(spec, want) { if !reflect.DeepEqual(spec, want) {
t.Fatalf("Spec() = %#v, want %#v", spec, want) t.Fatalf("Spec() = %#v, want %#v", spec, want)
} }
@@ -152,7 +162,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
t.Run(testCase.name+"/wrong stage rejection", func(t *testing.T) { t.Run(testCase.name+"/wrong stage rejection", func(t *testing.T) {
registry := testCase.newRegistry() registry := testCase.newRegistry()
err := testCase.registerWithSpec(registry, ModuleSpec{Key: testCase.key, Stage: testCase.wrongStage}, testCase.constructor(testCase.key)) err := testCase.registerWithSpec(registry, ModuleSpec{Key: testCase.key, Stage: testCase.wrongStage, ExecutionClass: contracts.ExecutionClassDeterministic}, testCase.constructor(testCase.key))
if err == nil { if err == nil {
t.Fatal("RegisterWithSpec() error = nil, want error") t.Fatal("RegisterWithSpec() error = nil, want error")
} }
@@ -161,6 +171,22 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
} }
}) })
t.Run(testCase.name+"/execution class rejection", func(t *testing.T) {
for _, spec := range []ModuleSpec{
{Key: testCase.key, Stage: testCase.stage},
{Key: testCase.key, Stage: testCase.stage, ExecutionClass: contracts.ExecutionClass("unsupported")},
} {
registry := testCase.newRegistry()
err := testCase.registerWithSpec(registry, spec, testCase.constructor(testCase.key))
if err == nil {
t.Fatal("RegisterWithSpec() error = nil, want execution class error")
}
if !strings.Contains(err.Error(), "execution class") {
t.Fatalf("RegisterWithSpec() error = %q, want execution class error", err.Error())
}
}
})
t.Run(testCase.name+"/key trimming", func(t *testing.T) { t.Run(testCase.name+"/key trimming", func(t *testing.T) {
registry := testCase.newRegistry() registry := testCase.newRegistry()
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil { if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {

View File

@@ -108,7 +108,7 @@ func registerTestEvidenceOutput(t *testing.T, registries *Registries, policy Evi
func registerTestEvidenceOutputWithProfileValidation(t *testing.T, registries *Registries, policy EvidenceContextPolicy, validateProfile OutputProfileOptionValidator) { func registerTestEvidenceOutputWithProfileValidation(t *testing.T, registries *Registries, policy EvidenceContextPolicy, validateProfile OutputProfileOptionValidator) {
t.Helper() t.Helper()
registry := NewOutputEncoderRegistry() registry := NewOutputEncoderRegistry()
if err := registry.RegisterBuilderWithProfileValidation(defaultModuleSpec("output", StageOutput), func(options map[string]any) error { if err := registry.RegisterBuilderWithProfileValidation(testModuleSpec("output", StageOutput), func(options map[string]any) error {
return RejectUnknownOptions(options, "known") return RejectUnknownOptions(options, "known")
}, validateProfile, func(BuildRequest) (contracts.OutputEncoder, error) { }, validateProfile, func(BuildRequest) (contracts.OutputEncoder, error) {
return testEvidenceOutput{policy: cloneEvidenceContextPolicy(policy)}, nil return testEvidenceOutput{policy: cloneEvidenceContextPolicy(policy)}, nil

View File

@@ -24,10 +24,6 @@ func NewInputAdapterRegistry() *InputAdapterRegistry {
} }
} }
func (r *InputAdapterRegistry) Register(key string, constructor InputAdapterConstructor) error {
return r.RegisterWithSpec(defaultModuleSpec(key, StageInput), constructor)
}
func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor InputAdapterConstructor) error { func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor InputAdapterConstructor) error {
if constructor == nil { if constructor == nil {
return fmt.Errorf("input adapter constructor for %q must not be nil", strings.TrimSpace(spec.Key)) return fmt.Errorf("input adapter constructor for %q must not be nil", strings.TrimSpace(spec.Key))

View File

@@ -11,10 +11,14 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts" "gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
) )
func registerTestInput(registry *InputAdapterRegistry, key string, constructor InputAdapterConstructor) error {
return registry.RegisterWithSpec(testModuleSpec(key, StageInput), constructor)
}
func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) { func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) {
registry := NewInputAdapterRegistry() registry := NewInputAdapterRegistry()
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err != nil { if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
t.Fatalf("Register() error = %v, want nil", err) t.Fatalf("Register() error = %v, want nil", err)
} }
@@ -30,7 +34,7 @@ func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) {
func TestInputAdapterRegistryRegisterAndBuildTrimKeys(t *testing.T) { func TestInputAdapterRegistryRegisterAndBuildTrimKeys(t *testing.T) {
registry := NewInputAdapterRegistry() registry := NewInputAdapterRegistry()
if err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil { if err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
t.Fatalf("Register() error = %v, want nil", err) t.Fatalf("Register() error = %v, want nil", err)
} }
@@ -48,6 +52,7 @@ func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
spec := ModuleSpec{ spec := ModuleSpec{
Key: " generic-input ", Key: " generic-input ",
Stage: StageInput, Stage: StageInput,
ExecutionClass: contracts.ExecutionClassDeterministic,
Provides: []string{" parsed-source ", "source-document", "parsed-source", ""}, Provides: []string{" parsed-source ", "source-document", "parsed-source", ""},
Requires: []string{" raw-bytes ", "raw-bytes", ""}, Requires: []string{" raw-bytes ", "raw-bytes", ""},
} }
@@ -63,6 +68,7 @@ func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
want := ModuleSpec{ want := ModuleSpec{
Key: "generic-input", Key: "generic-input",
Stage: StageInput, Stage: StageInput,
ExecutionClass: contracts.ExecutionClassDeterministic,
Provides: []string{"parsed-source", "source-document"}, Provides: []string{"parsed-source", "source-document"},
Requires: []string{"raw-bytes"}, Requires: []string{"raw-bytes"},
} }
@@ -80,10 +86,10 @@ func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
} }
} }
func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) { func TestInputAdapterRegistryRegisterWithSpecStoresMinimalMetadata(t *testing.T) {
registry := NewInputAdapterRegistry() registry := NewInputAdapterRegistry()
if err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil { if err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
t.Fatalf("Register() error = %v, want nil", err) t.Fatalf("Register() error = %v, want nil", err)
} }
@@ -91,7 +97,7 @@ func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
if !ok { if !ok {
t.Fatal("Spec() ok = false, want true") t.Fatal("Spec() ok = false, want true")
} }
want := ModuleSpec{Key: "generic-input", Stage: StageInput} want := ModuleSpec{Key: "generic-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic}
if !reflect.DeepEqual(got, want) { if !reflect.DeepEqual(got, want) {
t.Fatalf("Spec() = %#v, want %#v", got, want) t.Fatalf("Spec() = %#v, want %#v", got, want)
} }
@@ -100,7 +106,7 @@ func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
func TestInputAdapterRegistryRegisterWithSpecRejectsWrongStage(t *testing.T) { func TestInputAdapterRegistryRegisterWithSpecRejectsWrongStage(t *testing.T) {
registry := NewInputAdapterRegistry() registry := NewInputAdapterRegistry()
err := registry.RegisterWithSpec(ModuleSpec{Key: "generic-input", Stage: StageExtract}, fakeInputAdapterConstructor("generic-input")) err := registry.RegisterWithSpec(ModuleSpec{Key: "generic-input", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic}, fakeInputAdapterConstructor("generic-input"))
if err == nil { if err == nil {
t.Fatal("RegisterWithSpec() error = nil, want error") t.Fatal("RegisterWithSpec() error = nil, want error")
@@ -121,7 +127,7 @@ func TestInputAdapterRegistrySpecRejectsUnknownKey(t *testing.T) {
func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) { func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) {
registry := NewInputAdapterRegistry() registry := NewInputAdapterRegistry()
err := registry.Register(" \t", fakeInputAdapterConstructor("generic-input")) err := registerTestInput(registry, " \t", fakeInputAdapterConstructor("generic-input"))
if err == nil { if err == nil {
t.Fatal("Register() error = nil, want error") t.Fatal("Register() error = nil, want error")
@@ -133,11 +139,11 @@ func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) {
func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) { func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) {
registry := NewInputAdapterRegistry() registry := NewInputAdapterRegistry()
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err != nil { if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
t.Fatalf("Register() error = %v, want nil", err) t.Fatalf("Register() error = %v, want nil", err)
} }
err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input")) err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input"))
if err == nil { if err == nil {
t.Fatal("Register() error = nil, want error") t.Fatal("Register() error = nil, want error")
@@ -150,7 +156,7 @@ func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) {
func TestInputAdapterRegistryRegisterRejectsNilConstructor(t *testing.T) { func TestInputAdapterRegistryRegisterRejectsNilConstructor(t *testing.T) {
registry := NewInputAdapterRegistry() registry := NewInputAdapterRegistry()
err := registry.Register("generic-input", nil) err := registerTestInput(registry, "generic-input", nil)
if err == nil { if err == nil {
t.Fatal("Register() error = nil, want error") t.Fatal("Register() error = nil, want error")
@@ -176,7 +182,7 @@ func TestInputAdapterRegistryBuildRejectsUnknownKey(t *testing.T) {
func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) { func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) {
registry := NewInputAdapterRegistry() registry := NewInputAdapterRegistry()
constructorErr := errors.New("constructor failed") constructorErr := errors.New("constructor failed")
if err := registry.Register("generic-input", func() (contracts.InputAdapter, error) { if err := registerTestInput(registry, "generic-input", func() (contracts.InputAdapter, error) {
return nil, constructorErr return nil, constructorErr
}); err != nil { }); err != nil {
t.Fatalf("Register() error = %v, want nil", err) t.Fatalf("Register() error = %v, want nil", err)
@@ -197,7 +203,7 @@ func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) {
func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) { func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) {
registry := NewInputAdapterRegistry() registry := NewInputAdapterRegistry()
if err := registry.Register("generic-input", func() (contracts.InputAdapter, error) { if err := registerTestInput(registry, "generic-input", func() (contracts.InputAdapter, error) {
return nil, nil return nil, nil
}); err != nil { }); err != nil {
t.Fatalf("Register() error = %v, want nil", err) t.Fatalf("Register() error = %v, want nil", err)
@@ -215,7 +221,7 @@ func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) {
func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) { func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) {
registry := NewInputAdapterRegistry() registry := NewInputAdapterRegistry()
if err := registry.Register("generic-input", fakeInputAdapterConstructor("other-input")); err != nil { if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("other-input")); err != nil {
t.Fatalf("Register() error = %v, want nil", err) t.Fatalf("Register() error = %v, want nil", err)
} }
@@ -232,7 +238,7 @@ func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) {
func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) { func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
registry := NewInputAdapterRegistry() registry := NewInputAdapterRegistry()
for _, key := range []string{"zeta", "alpha", "middle"} { for _, key := range []string{"zeta", "alpha", "middle"} {
if err := registry.Register(key, fakeInputAdapterConstructor(key)); err != nil { if err := registerTestInput(registry, key, fakeInputAdapterConstructor(key)); err != nil {
t.Fatalf("Register(%q) error = %v, want nil", key, err) t.Fatalf("Register(%q) error = %v, want nil", key, err)
} }
} }
@@ -253,7 +259,7 @@ func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
func TestInputAdapterRegistryNilRegistryBehavior(t *testing.T) { func TestInputAdapterRegistryNilRegistryBehavior(t *testing.T) {
var registry *InputAdapterRegistry var registry *InputAdapterRegistry
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err == nil { if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err == nil {
t.Fatal("Register() error = nil, want error") t.Fatal("Register() error = nil, want error")
} }
if _, err := registry.Build("generic-input"); err == nil { if _, err := registry.Build("generic-input"); err == nil {

View File

@@ -23,23 +23,19 @@ const (
type ModuleSpec struct { type ModuleSpec struct {
Key string Key string
Stage ModuleStage Stage ModuleStage
ExecutionClass contracts.ExecutionClass
ArtifactKind contracts.ArtifactKind ArtifactKind contracts.ArtifactKind
Provides []string Provides []string
Requires []string Requires []string
ReferenceSlots []contracts.ReferenceSlot ReferenceSlots []contracts.ReferenceSlot
} }
func defaultModuleSpec(key string, stage ModuleStage) ModuleSpec {
return ModuleSpec{
Key: key,
Stage: stage,
}
}
func normalizeModuleSpec(spec ModuleSpec) ModuleSpec { func normalizeModuleSpec(spec ModuleSpec) ModuleSpec {
executionClass := contracts.ExecutionClass(strings.TrimSpace(string(spec.ExecutionClass)))
return ModuleSpec{ return ModuleSpec{
Key: strings.TrimSpace(spec.Key), Key: strings.TrimSpace(spec.Key),
Stage: spec.Stage, Stage: spec.Stage,
ExecutionClass: executionClass,
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind), ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
Provides: normalizeCapabilities(spec.Provides), Provides: normalizeCapabilities(spec.Provides),
Requires: normalizeCapabilities(spec.Requires), Requires: normalizeCapabilities(spec.Requires),
@@ -76,6 +72,7 @@ func cloneModuleSpec(spec ModuleSpec) ModuleSpec {
return ModuleSpec{ return ModuleSpec{
Key: spec.Key, Key: spec.Key,
Stage: spec.Stage, Stage: spec.Stage,
ExecutionClass: spec.ExecutionClass,
ArtifactKind: spec.ArtifactKind, ArtifactKind: spec.ArtifactKind,
Provides: append([]string(nil), spec.Provides...), Provides: append([]string(nil), spec.Provides...),
Requires: append([]string(nil), spec.Requires...), Requires: append([]string(nil), spec.Requires...),
@@ -90,6 +87,12 @@ func validateModuleSpec(kind string, expectedStage ModuleStage, spec ModuleSpec)
if spec.Stage != expectedStage { if spec.Stage != expectedStage {
return fmt.Errorf("%s %q must use %q stage, got %q", kind, spec.Key, expectedStage, spec.Stage) return fmt.Errorf("%s %q must use %q stage, got %q", kind, spec.Key, expectedStage, spec.Stage)
} }
if spec.ExecutionClass == "" {
return fmt.Errorf("%s %q execution class must not be empty", kind, spec.Key)
}
if spec.ExecutionClass != contracts.ExecutionClassDeterministic && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
return fmt.Errorf("%s %q has unsupported execution class %q", kind, spec.Key, spec.ExecutionClass)
}
if spec.ArtifactKind != "" && spec.Stage != StageExtract && spec.Stage != StageMerge && spec.Stage != StageNormalize { if spec.ArtifactKind != "" && spec.Stage != StageExtract && spec.Stage != StageMerge && spec.Stage != StageNormalize {
return fmt.Errorf("%s %q must not declare an artifact kind", kind, spec.Key) return fmt.Errorf("%s %q must not declare an artifact kind", kind, spec.Key)
} }

View File

@@ -1,12 +1,45 @@
package pipeline package pipeline
import ( import (
"reflect"
"strings" "strings"
"testing" "testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts" "gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
) )
func TestValidateModuleSpecRequiresSupportedExecutionClass(t *testing.T) {
for _, test := range []struct {
name string
class contracts.ExecutionClass
want string
}{
{name: "missing", want: "execution class"},
{name: "unsupported", class: "remote", want: "unsupported"},
{name: "deterministic", class: contracts.ExecutionClassDeterministic},
{name: "llm backed", class: contracts.ExecutionClassLLMBacked},
} {
t.Run(test.name, func(t *testing.T) {
spec := normalizeModuleSpec(ModuleSpec{Key: "module", Stage: StageChunk, ExecutionClass: test.class})
err := validateModuleSpec("chunker", StageChunk, spec)
if test.want == "" && err != nil {
t.Fatalf("validateModuleSpec() error = %v, want nil", err)
}
if test.want != "" && (err == nil || !strings.Contains(err.Error(), test.want)) {
t.Fatalf("validateModuleSpec() error = %v, want %q", err, test.want)
}
})
}
}
func TestCloneModuleSpecPreservesExecutionClass(t *testing.T) {
spec := normalizeModuleSpec(ModuleSpec{Key: " module ", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked})
cloned := cloneModuleSpec(spec)
if !reflect.DeepEqual(cloned, spec) {
t.Fatalf("cloneModuleSpec() = %#v, want %#v", cloned, spec)
}
}
func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) { func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
@@ -24,6 +57,7 @@ func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
spec := normalizeModuleSpec(ModuleSpec{ spec := normalizeModuleSpec(ModuleSpec{
Key: "module", Key: "module",
Stage: test.stage, Stage: test.stage,
ExecutionClass: contracts.ExecutionClassDeterministic,
ReferenceSlots: []contracts.ReferenceSlot{ ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster", Description: "Character roster", MaxBytes: 1024}, {Name: "roster", Description: "Character roster", MaxBytes: 1024},
}, },
@@ -52,6 +86,7 @@ func TestValidateModuleSpecRejectsReferenceSlotsForIneligibleStages(t *testing.T
spec := normalizeModuleSpec(ModuleSpec{ spec := normalizeModuleSpec(ModuleSpec{
Key: "module", Key: "module",
Stage: test.stage, Stage: test.stage,
ExecutionClass: contracts.ExecutionClassDeterministic,
ReferenceSlots: []contracts.ReferenceSlot{ ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster"}, {Name: "roster"},
}, },
@@ -109,6 +144,7 @@ func TestValidateModuleSpecRejectsInvalidReferenceSlotsForEligibleStages(t *test
spec := normalizeModuleSpec(ModuleSpec{ spec := normalizeModuleSpec(ModuleSpec{
Key: "module", Key: "module",
Stage: stage.stage, Stage: stage.stage,
ExecutionClass: contracts.ExecutionClassDeterministic,
ReferenceSlots: invalid.slots, ReferenceSlots: invalid.slots,
}) })
err := validateModuleSpec(stage.kind, stage.stage, spec) err := validateModuleSpec(stage.kind, stage.stage, spec)

View File

@@ -26,7 +26,7 @@ func TestNormalizerRegistryErasureClonesRetryDirective(t *testing.T) {
FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "omitted", Message: "fallback warning"}}, FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "omitted", Message: "fallback warning"}},
} }
registry := NewNormalizerRegistry() registry := NewNormalizerRegistry()
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) { if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
return retryingNotesNormalizer{warnings: warnings, retry: retry}, nil return retryingNotesNormalizer{warnings: warnings, retry: retry}, nil
}); err != nil { }); err != nil {
t.Fatalf("RegisterNormalizer() error = %v", err) t.Fatalf("RegisterNormalizer() error = %v", err)

View File

@@ -32,10 +32,6 @@ func NewOutputEncoderRegistry() *OutputEncoderRegistry {
} }
} }
func (r *OutputEncoderRegistry) Register(key string, constructor OutputEncoderConstructor) error {
return r.RegisterWithSpec(defaultModuleSpec(key, StageOutput), constructor)
}
func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor OutputEncoderConstructor) error { func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor OutputEncoderConstructor) error {
if constructor == nil { if constructor == nil {
return fmt.Errorf("output encoder constructor for %q must not be nil", strings.TrimSpace(spec.Key)) return fmt.Errorf("output encoder constructor for %q must not be nil", strings.TrimSpace(spec.Key))

View File

@@ -17,7 +17,9 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
return NewOutputEncoderRegistry() return NewOutputEncoderRegistry()
}, },
register: func(registry any, key string, constructor func() (contracts.OutputEncoder, error)) error { register: func(registry any, key string, constructor func() (contracts.OutputEncoder, error)) error {
return registry.(*OutputEncoderRegistry).Register(key, constructor) return registry.(*OutputEncoderRegistry).RegisterWithSpec(ModuleSpec{
Key: key, Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic,
}, constructor)
}, },
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.OutputEncoder, error)) error { registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.OutputEncoder, error)) error {
return registry.(*OutputEncoderRegistry).RegisterWithSpec(spec, constructor) return registry.(*OutputEncoderRegistry).RegisterWithSpec(spec, constructor)
@@ -33,7 +35,9 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
}, },
nilRegister: func(key string, constructor func() (contracts.OutputEncoder, error)) error { nilRegister: func(key string, constructor func() (contracts.OutputEncoder, error)) error {
var registry *OutputEncoderRegistry var registry *OutputEncoderRegistry
return registry.Register(key, constructor) return registry.RegisterWithSpec(ModuleSpec{
Key: key, Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic,
}, constructor)
}, },
nilBuild: func(key string) (contracts.OutputEncoder, error) { nilBuild: func(key string) (contracts.OutputEncoder, error) {
var registry *OutputEncoderRegistry var registry *OutputEncoderRegistry
@@ -60,7 +64,7 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
func TestOutputProfileValidationReceivesOwnedOptionsAndLaneIDs(t *testing.T) { func TestOutputProfileValidationReceivesOwnedOptionsAndLaneIDs(t *testing.T) {
registry := NewOutputEncoderRegistry() registry := NewOutputEncoderRegistry()
if err := registry.RegisterBuilderWithProfileValidation(defaultModuleSpec("profile-output", StageOutput), func(options map[string]any) error { if err := registry.RegisterBuilderWithProfileValidation(testModuleSpec("profile-output", StageOutput), func(options map[string]any) error {
options["nested"].(map[string]any)["value"] = "changed" options["nested"].(map[string]any)["value"] = "changed"
return nil return nil
}, func(context OutputProfileOptionContext, options map[string]any) error { }, func(context OutputProfileOptionContext, options map[string]any) error {

View File

@@ -480,19 +480,19 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
if err := RegisterArtifactCodec(registries.ArtifactCodecs, notesCodec()); err != nil { if err := RegisterArtifactCodec(registries.ArtifactCodecs, notesCodec()); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := registries.Inputs.RegisterBuilderWithSpec(defaultModuleSpec("input", StageInput), strict, func(request BuildRequest) (contracts.InputAdapter, error) { if err := registries.Inputs.RegisterBuilderWithSpec(testModuleSpec("input", StageInput), strict, func(request BuildRequest) (contracts.InputAdapter, error) {
record("input", &request) record("input", &request)
return input, nil return input, nil
}); err != nil { }); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := registries.Chunkers.RegisterBuilderWithSpec(defaultModuleSpec("chunk", StageChunk), strict, func(request BuildRequest) (contracts.Chunker, error) { if err := registries.Chunkers.RegisterBuilderWithSpec(testModuleSpec("chunk", StageChunk), strict, func(request BuildRequest) (contracts.Chunker, error) {
record("chunk", &request) record("chunk", &request)
return &typedTestChunker{key: "chunk"}, nil return &typedTestChunker{key: "chunk"}, nil
}); err != nil { }); err != nil {
t.Fatal(err) t.Fatal(err)
} }
extractSpec := defaultModuleSpec("extract", StageExtract) extractSpec := testModuleSpec("extract", StageExtract)
extractSpec.ArtifactKind = "test/notes" extractSpec.ArtifactKind = "test/notes"
if err := RegisterExtractorBuilder(registries.Extractors, extractSpec, strict, func(request BuildRequest) (contracts.Extractor[codecNotes], error) { if err := RegisterExtractorBuilder(registries.Extractors, extractSpec, strict, func(request BuildRequest) (contracts.Extractor[codecNotes], error) {
record("extract", &request) record("extract", &request)
@@ -503,7 +503,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
}); err != nil { }); err != nil {
t.Fatal(err) t.Fatal(err)
} }
mergeSpec := defaultModuleSpec("merge", StageMerge) mergeSpec := testModuleSpec("merge", StageMerge)
mergeSpec.ArtifactKind = "test/notes" mergeSpec.ArtifactKind = "test/notes"
if err := RegisterMergerBuilder(registries.Mergers, mergeSpec, strict, func(request BuildRequest) (contracts.Merger[codecNotes], error) { if err := RegisterMergerBuilder(registries.Mergers, mergeSpec, strict, func(request BuildRequest) (contracts.Merger[codecNotes], error) {
record("merge", &request) record("merge", &request)
@@ -511,7 +511,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
}); err != nil { }); err != nil {
t.Fatal(err) t.Fatal(err)
} }
normalizeSpec := defaultModuleSpec("normalize", StageNormalize) normalizeSpec := testModuleSpec("normalize", StageNormalize)
normalizeSpec.ArtifactKind = "test/notes" normalizeSpec.ArtifactKind = "test/notes"
if err := RegisterNormalizerBuilder(registries.Normalizers, normalizeSpec, strict, func(request BuildRequest) (contracts.Normalizer[codecNotes], error) { if err := RegisterNormalizerBuilder(registries.Normalizers, normalizeSpec, strict, func(request BuildRequest) (contracts.Normalizer[codecNotes], error) {
record("normalize", &request) record("normalize", &request)
@@ -532,7 +532,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
}); err != nil { }); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := registries.Outputs.RegisterBuilderWithSpec(defaultModuleSpec("output", StageOutput), strict, func(request BuildRequest) (contracts.OutputEncoder, error) { if err := registries.Outputs.RegisterBuilderWithSpec(testModuleSpec("output", StageOutput), strict, func(request BuildRequest) (contracts.OutputEncoder, error) {
record("output", &request) record("output", &request)
if failure.output != nil { if failure.output != nil {
return nil, failure.output return nil, failure.output

View File

@@ -117,6 +117,7 @@ type PipelineStepProfile struct {
type PipelineProfile struct { type PipelineProfile struct {
ID string `json:"id"` ID string `json:"id"`
LLMProfile string `json:"llm_profile,omitempty"`
Input ModuleBinding `json:"input"` Input ModuleBinding `json:"input"`
Chunk ModuleBinding `json:"chunk,omitempty"` Chunk ModuleBinding `json:"chunk,omitempty"`
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"` Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
@@ -127,6 +128,7 @@ type PipelineProfile struct {
type ResolveOptions struct { type ResolveOptions struct {
Only []string Only []string
LLMProfileOverride string
ReferenceOverrides []ReferenceBinding ReferenceOverrides []ReferenceBinding
ReferenceUnbinds []ReferenceUnbind ReferenceUnbinds []ReferenceUnbind
} }
@@ -164,8 +166,11 @@ type ResolvedArtifactLane struct {
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"` ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"` ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
Extract ModuleBinding Extract ModuleBinding
ExtractExecutionClass contracts.ExecutionClass `json:"extract_execution_class"`
Merge ModuleBinding Merge ModuleBinding
MergeExecutionClass contracts.ExecutionClass `json:"merge_execution_class"`
Normalize ModuleBinding Normalize ModuleBinding
NormalizeExecutionClass contracts.ExecutionClass `json:"normalize_execution_class"`
Validators []ModuleBinding Validators []ModuleBinding
ExtractReferences ResolvedReferenceTarget `json:"extract_references"` ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
MergeReferences ResolvedReferenceTarget `json:"merge_references"` MergeReferences ResolvedReferenceTarget `json:"merge_references"`
@@ -195,11 +200,14 @@ type ResolvedPipeline struct {
ID string ID string
Digest string Digest string
Input ModuleBinding Input ModuleBinding
InputExecutionClass contracts.ExecutionClass `json:"input_execution_class"`
Chunk ModuleBinding Chunk ModuleBinding
ChunkExecutionClass contracts.ExecutionClass `json:"chunk_execution_class"`
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"` ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
Steps []ResolvedPipelineStep Steps []ResolvedPipelineStep
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"` ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
Output ModuleBinding Output ModuleBinding
OutputExecutionClass contracts.ExecutionClass `json:"output_execution_class"`
} }
// AllArtifactLanes returns lanes in deterministic step order for read-only // AllArtifactLanes returns lanes in deterministic step order for read-only
@@ -225,6 +233,46 @@ type ModuleCatalog struct {
Outputs *OutputEncoderRegistry Outputs *OutputEncoderRegistry
} }
// ExecutionClass returns the registered execution class for a module selected
// by stage and key without constructing the module.
func (catalog ModuleCatalog) ExecutionClass(stage ModuleStage, key string) (contracts.ExecutionClass, bool) {
var executionClass contracts.ExecutionClass
var ok bool
switch stage {
case StageInput:
var spec ModuleSpec
spec, ok = catalog.Inputs.Spec(key)
executionClass = spec.ExecutionClass
case StageChunk:
var spec ModuleSpec
spec, ok = catalog.Chunkers.Spec(key)
executionClass = spec.ExecutionClass
case StageExtract:
var spec ModuleSpec
spec, ok = catalog.Extractors.Spec(key)
executionClass = spec.ExecutionClass
case StageMerge:
var spec ModuleSpec
spec, ok = catalog.Mergers.Spec(key)
executionClass = spec.ExecutionClass
case StageNormalize:
var spec ModuleSpec
spec, ok = catalog.Normalizers.Spec(key)
executionClass = spec.ExecutionClass
case StageValidate:
var spec ValidatorSpec
spec, ok = catalog.Validators.Spec(key)
executionClass = spec.ExecutionClass
case StageOutput:
var spec ModuleSpec
spec, ok = catalog.Outputs.Spec(key)
executionClass = spec.ExecutionClass
}
return executionClass, ok
}
func Binding(module string) ModuleBinding { func Binding(module string) ModuleBinding {
return ModuleBinding{Module: strings.TrimSpace(module)} return ModuleBinding{Module: strings.TrimSpace(module)}
} }
@@ -318,7 +366,9 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
resolved := ResolvedPipeline{ resolved := ResolvedPipeline{
ID: pipelineID, ID: pipelineID,
Input: input, Input: input,
InputExecutionClass: inputModuleSpec.ExecutionClass,
Chunk: chunk, Chunk: chunk,
ChunkExecutionClass: chunkSpec.ExecutionClass,
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences), ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
Output: resolveBinding(profile.Output, DefaultOutputModule), Output: resolveBinding(profile.Output, DefaultOutputModule),
} }
@@ -386,6 +436,10 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
if missing, ok := outputCapabilities.missing(outputSpec.Requires); ok { if missing, ok := outputCapabilities.missing(outputSpec.Requires); ok {
return ResolvedPipeline{}, capabilityError(pipelineID, "", StageOutput, resolved.Output.Module, missing) return ResolvedPipeline{}, capabilityError(pipelineID, "", StageOutput, resolved.Output.Module, missing)
} }
resolved.OutputExecutionClass = outputSpec.ExecutionClass
if err := applyEffectiveLLMProfiles(&resolved, profile.LLMProfile, options.LLMProfileOverride); err != nil {
return ResolvedPipeline{}, err
}
if err := validateResolvedOptions(resolved, catalog, configuredLaneIDs); err != nil { if err := validateResolvedOptions(resolved, catalog, configuredLaneIDs); err != nil {
return ResolvedPipeline{}, err return ResolvedPipeline{}, err
} }
@@ -475,6 +529,7 @@ func resolveArtifactLane(
} }
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references) lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
lane.ExtractReferences.StepID = strings.TrimSpace(stepID) lane.ExtractReferences.StepID = strings.TrimSpace(stepID)
lane.ExtractExecutionClass = extractSpec.ExecutionClass
capabilities.add(extractSpec.Provides...) capabilities.add(extractSpec.Provides...)
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType) mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
@@ -499,6 +554,7 @@ func resolveArtifactLane(
} }
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences) lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
lane.MergeReferences.StepID = strings.TrimSpace(stepID) lane.MergeReferences.StepID = strings.TrimSpace(stepID)
lane.MergeExecutionClass = mergeSpec.ExecutionClass
capabilities.add(mergeSpec.Provides...) capabilities.add(mergeSpec.Provides...)
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType) normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
@@ -523,6 +579,7 @@ func resolveArtifactLane(
} }
lane.NormalizeReferences = referenceTarget(StageNormalize, laneID, lane.Normalize.Module, normalizeReferences) lane.NormalizeReferences = referenceTarget(StageNormalize, laneID, lane.Normalize.Module, normalizeReferences)
lane.NormalizeReferences.StepID = strings.TrimSpace(stepID) lane.NormalizeReferences.StepID = strings.TrimSpace(stepID)
lane.NormalizeExecutionClass = normalizeSpec.ExecutionClass
capabilities.add(normalizeSpec.Provides...) capabilities.add(normalizeSpec.Provides...)
if len(lane.Validators) > 0 { if len(lane.Validators) > 0 {
@@ -784,9 +841,6 @@ func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage,
if err != nil { if err != nil {
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q: %w", pipelineID, stage, chain.ModuleKey, err) return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q: %w", pipelineID, stage, chain.ModuleKey, err)
} }
if strings.TrimSpace(validator.LLMProfile) != "" && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q assigns llm_profile to deterministic validator %q", pipelineID, stage, chain.ModuleKey, validator.Module)
}
chain.Validators = append(chain.Validators, ResolvedValidator{ chain.Validators = append(chain.Validators, ResolvedValidator{
Binding: cloneModuleBinding(validator), Binding: cloneModuleBinding(validator),
ExecutionClass: spec.ExecutionClass, ExecutionClass: spec.ExecutionClass,
@@ -1198,6 +1252,66 @@ func resolveBinding(binding ModuleBinding, defaultModule string) ModuleBinding {
} }
} }
func applyEffectiveLLMProfiles(resolved *ResolvedPipeline, pipelineProfile, overrideProfile string) error {
pipelineProfile = strings.TrimSpace(pipelineProfile)
overrideProfile = strings.TrimSpace(overrideProfile)
apply := func(stage ModuleStage, laneID, module string, binding *ModuleBinding, executionClass contracts.ExecutionClass, kind string) error {
binding.LLMProfile = strings.TrimSpace(binding.LLMProfile)
if executionClass != contracts.ExecutionClassLLMBacked {
if binding.LLMProfile != "" {
if laneID == "" {
return fmt.Errorf("pipeline %q %s %q assigns llm_profile to deterministic %s %q", resolved.ID, stage, module, kind, binding.Module)
}
return fmt.Errorf("pipeline %q lane %q %s %q assigns llm_profile to deterministic %s %q", resolved.ID, laneID, stage, module, kind, binding.Module)
}
return nil
}
if overrideProfile != "" {
binding.LLMProfile = overrideProfile
return nil
}
if binding.LLMProfile == "" {
binding.LLMProfile = pipelineProfile
}
return nil
}
if err := apply(StageInput, "", resolved.Input.Module, &resolved.Input, resolved.InputExecutionClass, "module"); err != nil {
return err
}
if err := apply(StageChunk, "", resolved.Chunk.Module, &resolved.Chunk, resolved.ChunkExecutionClass, "module"); err != nil {
return err
}
for stepIndex := range resolved.Steps {
for laneIndex := range resolved.Steps[stepIndex].ArtifactLanes {
lane := &resolved.Steps[stepIndex].ArtifactLanes[laneIndex]
if err := apply(StageExtract, lane.ID, lane.Extract.Module, &lane.Extract, lane.ExtractExecutionClass, "module"); err != nil {
return err
}
if err := apply(StageMerge, lane.ID, lane.Merge.Module, &lane.Merge, lane.MergeExecutionClass, "module"); err != nil {
return err
}
if err := apply(StageNormalize, lane.ID, lane.Normalize.Module, &lane.Normalize, lane.NormalizeExecutionClass, "module"); err != nil {
return err
}
}
}
if err := apply(StageOutput, "", resolved.Output.Module, &resolved.Output, resolved.OutputExecutionClass, "module"); err != nil {
return err
}
for chainIndex := range resolved.ValidatorChains {
chain := &resolved.ValidatorChains[chainIndex]
for validatorIndex := range chain.Validators {
validator := &chain.Validators[validatorIndex]
if err := apply(chain.Stage, chain.LaneID, chain.ModuleKey, &validator.Binding, validator.ExecutionClass, "validator"); err != nil {
return err
}
}
}
return nil
}
func resolveBindings(bindings []ModuleBinding, defaultModule string) []ModuleBinding { func resolveBindings(bindings []ModuleBinding, defaultModule string) []ModuleBinding {
if len(bindings) == 0 { if len(bindings) == 0 {
return nil return nil
@@ -1309,19 +1423,25 @@ func resolvedPipelineDigest(resolved ResolvedPipeline) (string, error) {
withoutDigest := struct { withoutDigest := struct {
ID string ID string
Input ModuleBinding Input ModuleBinding
InputExecutionClass contracts.ExecutionClass
Chunk ModuleBinding Chunk ModuleBinding
ChunkExecutionClass contracts.ExecutionClass
ChunkReferences ResolvedReferenceTarget ChunkReferences ResolvedReferenceTarget
Steps []ResolvedPipelineStep Steps []ResolvedPipelineStep
ValidatorChains []ResolvedValidatorChain ValidatorChains []ResolvedValidatorChain
Output ModuleBinding Output ModuleBinding
OutputExecutionClass contracts.ExecutionClass
}{ }{
ID: resolved.ID, ID: resolved.ID,
Input: resolved.Input, Input: resolved.Input,
InputExecutionClass: resolved.InputExecutionClass,
Chunk: resolved.Chunk, Chunk: resolved.Chunk,
ChunkExecutionClass: resolved.ChunkExecutionClass,
ChunkReferences: resolved.ChunkReferences, ChunkReferences: resolved.ChunkReferences,
Steps: resolved.Steps, Steps: resolved.Steps,
ValidatorChains: resolved.ValidatorChains, ValidatorChains: resolved.ValidatorChains,
Output: resolved.Output, Output: resolved.Output,
OutputExecutionClass: resolved.OutputExecutionClass,
} }
encoded, err := json.Marshal(withoutDigest) encoded, err := json.Marshal(withoutDigest)
if err != nil { if err != nil {

View File

@@ -15,22 +15,22 @@ import (
func TestResolvePipelineWithExplicitModules(t *testing.T) { func TestResolvePipelineWithExplicitModules(t *testing.T) {
catalog := newProfileCatalog(t) catalog := newProfileCatalog(t)
registerProfileSpecs(t, catalog, registerProfileSpecs(t, catalog,
ModuleSpec{Key: "window", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}}, ModuleSpec{Key: "window", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}},
ModuleSpec{Key: "record-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}}, ModuleSpec{Key: "record-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "dedupe", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}}, ModuleSpec{Key: "dedupe", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"candidate"}, Provides: []string{"merged"}},
ModuleSpec{Key: "canonical", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}}, ModuleSpec{Key: "canonical", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}},
ModuleSpec{Key: "ndjson", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}}, ModuleSpec{Key: "ndjson", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
) )
resolved, err := ResolvePipeline(PipelineProfile{ resolved, err := ResolvePipeline(PipelineProfile{
ID: " campaign ", ID: " campaign ",
Input: ModuleBinding{Module: " text ", LLMProfile: " fast "}, Input: Binding(" text "),
Chunk: ModuleBinding{Module: " window ", Options: map[string]any{ Chunk: ModuleBinding{Module: " window ", Options: map[string]any{
"size": 10, "size": 10,
}}, }},
Artifacts: map[string]ArtifactLaneProfile{ Artifacts: map[string]ArtifactLaneProfile{
" records ": { " records ": {
Extract: ModuleBinding{Module: " record-extractor ", LLMProfile: " careful "}, Extract: Binding(" record-extractor "),
Merge: Binding(" dedupe "), Merge: Binding(" dedupe "),
Normalize: Binding(" canonical "), Normalize: Binding(" canonical "),
}, },
@@ -44,15 +44,21 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
if resolved.ID != "campaign" { if resolved.ID != "campaign" {
t.Fatalf("ID = %q, want campaign", resolved.ID) t.Fatalf("ID = %q, want campaign", resolved.ID)
} }
if !reflect.DeepEqual(resolved.Input, ModuleBinding{Module: "text", LLMProfile: "fast"}) { if !reflect.DeepEqual(resolved.Input, ModuleBinding{Module: "text"}) {
t.Fatalf("Input = %#v, want trimmed explicit input", resolved.Input) t.Fatalf("Input = %#v, want trimmed explicit input", resolved.Input)
} }
if resolved.InputExecutionClass != contracts.ExecutionClassDeterministic {
t.Fatalf("InputExecutionClass = %q, want deterministic", resolved.InputExecutionClass)
}
if resolved.Chunk.Module != "window" || resolved.Chunk.LLMProfile != "" { if resolved.Chunk.Module != "window" || resolved.Chunk.LLMProfile != "" {
t.Fatalf("Chunk = %#v, want explicit module and empty LLM profile", resolved.Chunk) t.Fatalf("Chunk = %#v, want explicit module and empty LLM profile", resolved.Chunk)
} }
if resolved.Chunk.Options["size"] != 10 { if resolved.Chunk.Options["size"] != 10 {
t.Fatalf("Chunk.Options = %#v, want size option", resolved.Chunk.Options) t.Fatalf("Chunk.Options = %#v, want size option", resolved.Chunk.Options)
} }
if resolved.ChunkExecutionClass != contracts.ExecutionClassDeterministic {
t.Fatalf("ChunkExecutionClass = %q, want deterministic", resolved.ChunkExecutionClass)
}
if len(resolved.Steps) != 1 || resolved.Steps[0].ID != "default" || len(resolved.Steps[0].ArtifactLanes) != 1 { if len(resolved.Steps) != 1 || resolved.Steps[0].ID != "default" || len(resolved.Steps[0].ArtifactLanes) != 1 {
t.Fatalf("resolved steps = %#v, want one default step with one lane", resolved.Steps) t.Fatalf("resolved steps = %#v, want one default step with one lane", resolved.Steps)
} }
@@ -60,9 +66,12 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
if lane.ID != "records" { if lane.ID != "records" {
t.Fatalf("lane.ID = %q, want records", lane.ID) t.Fatalf("lane.ID = %q, want records", lane.ID)
} }
if !reflect.DeepEqual(lane.Extract, ModuleBinding{Module: "record-extractor", LLMProfile: "careful"}) { if !reflect.DeepEqual(lane.Extract, ModuleBinding{Module: "record-extractor"}) {
t.Fatalf("lane.Extract = %#v, want explicit extractor", lane.Extract) t.Fatalf("lane.Extract = %#v, want explicit extractor", lane.Extract)
} }
if lane.ExtractExecutionClass != contracts.ExecutionClassDeterministic || lane.MergeExecutionClass != contracts.ExecutionClassDeterministic || lane.NormalizeExecutionClass != contracts.ExecutionClassDeterministic {
t.Fatalf("lane execution classes = %q/%q/%q, want deterministic", lane.ExtractExecutionClass, lane.MergeExecutionClass, lane.NormalizeExecutionClass)
}
if lane.Merge.Module != "dedupe" || lane.Normalize.Module != "canonical" { if lane.Merge.Module != "dedupe" || lane.Normalize.Module != "canonical" {
t.Fatalf("lane merge/normalize = %#v/%#v, want explicit modules", lane.Merge, lane.Normalize) t.Fatalf("lane merge/normalize = %#v/%#v, want explicit modules", lane.Merge, lane.Normalize)
} }
@@ -72,11 +81,51 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
if resolved.Output.Module != "ndjson" { if resolved.Output.Module != "ndjson" {
t.Fatalf("Output.Module = %q, want ndjson", resolved.Output.Module) t.Fatalf("Output.Module = %q, want ndjson", resolved.Output.Module)
} }
if resolved.OutputExecutionClass != contracts.ExecutionClassDeterministic {
t.Fatalf("OutputExecutionClass = %q, want deterministic", resolved.OutputExecutionClass)
}
if !strings.HasPrefix(resolved.Digest, "sha256:") { if !strings.HasPrefix(resolved.Digest, "sha256:") {
t.Fatalf("Digest = %q, want sha256 digest", resolved.Digest) t.Fatalf("Digest = %q, want sha256 digest", resolved.Digest)
} }
} }
func TestModuleCatalogExecutionClassLooksUpRegisteredMetadata(t *testing.T) {
catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked},
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked},
ModuleSpec{Key: "llm-extract", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked},
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked},
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked},
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked},
)
registerProfileValidatorSpec(t, catalog, ValidatorSpec{Key: "llm-validator", ExecutionClass: contracts.ExecutionClassLLMBacked})
for _, test := range []struct {
stage ModuleStage
key string
want contracts.ExecutionClass
}{
{stage: StageInput, key: "llm-input", want: contracts.ExecutionClassLLMBacked},
{stage: StageChunk, key: "llm-chunk", want: contracts.ExecutionClassLLMBacked},
{stage: StageExtract, key: "llm-extract", want: contracts.ExecutionClassLLMBacked},
{stage: StageMerge, key: "llm-merge", want: contracts.ExecutionClassLLMBacked},
{stage: StageNormalize, key: "llm-normalize", want: contracts.ExecutionClassLLMBacked},
{stage: StageValidate, key: "llm-validator", want: contracts.ExecutionClassLLMBacked},
{stage: StageOutput, key: "llm-output", want: contracts.ExecutionClassLLMBacked},
} {
t.Run(string(test.stage), func(t *testing.T) {
got, ok := catalog.ExecutionClass(test.stage, test.key)
if !ok || got != test.want {
t.Fatalf("ExecutionClass(%q, %q) = %q, %t; want %q, true", test.stage, test.key, got, ok, test.want)
}
})
}
if _, ok := catalog.ExecutionClass(StageExtract, "missing"); ok {
t.Fatal("ExecutionClass() found an unregistered module")
}
}
func TestResolvePipelineAppliesDefaults(t *testing.T) { func TestResolvePipelineAppliesDefaults(t *testing.T) {
resolved, err := ResolvePipeline(PipelineProfile{ resolved, err := ResolvePipeline(PipelineProfile{
ID: "defaulted", ID: "defaulted",
@@ -110,6 +159,176 @@ func TestResolvePipelineAppliesDefaults(t *testing.T) {
} }
} }
func TestResolvePipelineAppliesEffectiveLLMProfiles(t *testing.T) {
for _, test := range []struct {
name string
profile PipelineProfile
options ResolveOptions
want map[string]string
}{
{
name: "runtime override",
profile: func() PipelineProfile {
profile := llmProfilePipeline()
profile.LLMProfile = " pipeline "
profile.Chunk.LLMProfile = "binding"
return profile
}(),
options: ResolveOptions{LLMProfileOverride: " runtime "},
want: llmProfileValues("runtime"),
},
{
name: "binding exception",
profile: func() PipelineProfile {
profile := llmProfilePipeline()
profile.LLMProfile = "pipeline"
lane := profile.Artifacts["events"]
lane.Extract.LLMProfile = "extract"
lane.Extract.Validators = ValidatorOverride{
Set: true,
Validators: []ModuleBinding{{Module: "llm-validator", LLMProfile: "validator"}},
}
profile.Artifacts["events"] = lane
return profile
}(),
want: func() map[string]string {
values := llmProfileValues("pipeline")
values["extract"] = "extract"
values["validator:extract:events"] = "validator"
return values
}(),
},
{
name: "pipeline default",
profile: func() PipelineProfile {
profile := llmProfilePipeline()
profile.LLMProfile = "pipeline"
return profile
}(),
want: llmProfileValues("pipeline"),
},
{
name: "prompt fallback",
profile: llmProfilePipeline(),
want: llmProfileValues(""),
},
} {
t.Run(test.name, func(t *testing.T) {
resolved, err := ResolvePipeline(test.profile, test.options, llmProfileCatalog(t))
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
if got := resolvedLLMProfileValues(resolved); !reflect.DeepEqual(got, test.want) {
t.Fatalf("resolved profiles = %#v, want %#v", got, test.want)
}
})
}
}
func TestResolvePipelineAppliesProfilesOnlyToSelectedLLMBackedBindings(t *testing.T) {
profile := llmProfilePipeline()
profile.LLMProfile = "pipeline"
profile.Artifacts["notes"] = ArtifactLaneProfile{Extract: Binding("llm-extractor")}
resolved, err := ResolvePipeline(profile, ResolveOptions{Only: []string{"events"}}, llmProfileCatalog(t))
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
if got := laneIDs(resolved.Steps[0].ArtifactLanes); !reflect.DeepEqual(got, []string{"events"}) {
t.Fatalf("selected lanes = %#v, want events only", got)
}
if got := resolvedLLMProfileValues(resolved); !reflect.DeepEqual(got, llmProfileValues("pipeline")) {
t.Fatalf("resolved profiles = %#v, want selected LLM bindings only", got)
}
}
func TestResolvePipelineLeavesUnusedProfilesOffDeterministicBindings(t *testing.T) {
profile := baselineProfile()
profile.LLMProfile = "unused"
resolved, err := ResolvePipeline(profile, ResolveOptions{LLMProfileOverride: "also-unused"}, newProfileCatalog(t))
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
if resolved.Input.LLMProfile != "" || resolved.Chunk.LLMProfile != "" || resolved.Output.LLMProfile != "" {
t.Fatalf("deterministic pipeline profiles = input %q chunk %q output %q, want empty", resolved.Input.LLMProfile, resolved.Chunk.LLMProfile, resolved.Output.LLMProfile)
}
lane := resolved.Steps[0].ArtifactLanes[0]
if lane.Extract.LLMProfile != "" || lane.Merge.LLMProfile != "" || lane.Normalize.LLMProfile != "" {
t.Fatalf("deterministic lane profiles = extract %q merge %q normalize %q, want empty", lane.Extract.LLMProfile, lane.Merge.LLMProfile, lane.Normalize.LLMProfile)
}
}
func TestResolvePipelineRejectsLLMProfileForDeterministicModule(t *testing.T) {
for _, test := range []struct {
name string
mutate func(*PipelineProfile)
}{
{name: "input", mutate: func(profile *PipelineProfile) { profile.Input.LLMProfile = "invalid" }},
{name: "chunk", mutate: func(profile *PipelineProfile) { profile.Chunk.LLMProfile = "invalid" }},
{name: "extract", mutate: func(profile *PipelineProfile) {
lane := profile.Artifacts["events"]
lane.Extract.LLMProfile = "invalid"
profile.Artifacts["events"] = lane
}},
{name: "merge", mutate: func(profile *PipelineProfile) {
lane := profile.Artifacts["events"]
lane.Merge.LLMProfile = "invalid"
profile.Artifacts["events"] = lane
}},
{name: "normalize", mutate: func(profile *PipelineProfile) {
lane := profile.Artifacts["events"]
lane.Normalize.LLMProfile = "invalid"
profile.Artifacts["events"] = lane
}},
{name: "output", mutate: func(profile *PipelineProfile) { profile.Output.LLMProfile = "invalid" }},
} {
t.Run(test.name, func(t *testing.T) {
profile := baselineProfile()
test.mutate(&profile)
_, err := ResolvePipeline(profile, ResolveOptions{}, newProfileCatalog(t))
if err == nil || !strings.Contains(err.Error(), "llm_profile") || !strings.Contains(err.Error(), "deterministic") {
t.Fatalf("ResolvePipeline() error = %v, want deterministic profile rejection", err)
}
})
}
}
func TestResolvePipelineDigestUsesEffectiveLLMProfiles(t *testing.T) {
inherited := llmProfilePipeline()
inherited.LLMProfile = "shared"
explicit := llmProfilePipeline()
explicit.Input.LLMProfile = "shared"
explicit.Chunk.LLMProfile = "shared"
explicit.Output.LLMProfile = "shared"
lane := explicit.Artifacts["events"]
lane.Extract.LLMProfile = "shared"
lane.Merge.LLMProfile = "shared"
lane.Normalize.LLMProfile = "shared"
explicit.Artifacts["events"] = lane
inheritedResolved, err := ResolvePipeline(inherited, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
if err != nil {
t.Fatalf("ResolvePipeline(inherited) error = %v, want nil", err)
}
explicitResolved, err := ResolvePipeline(explicit, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
if err != nil {
t.Fatalf("ResolvePipeline(explicit) error = %v, want nil", err)
}
if inheritedResolved.Digest != explicitResolved.Digest {
t.Fatalf("effective profile digests differ: %q != %q", inheritedResolved.Digest, explicitResolved.Digest)
}
explicit.Chunk.LLMProfile = "different"
changedResolved, err := ResolvePipeline(explicit, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
if err != nil {
t.Fatalf("ResolvePipeline(changed) error = %v, want nil", err)
}
if explicitResolved.Digest == changedResolved.Digest {
t.Fatalf("digest = %q after effective profile change, want different", explicitResolved.Digest)
}
}
func TestResolvePipelineRecordsValidatorChains(t *testing.T) { func TestResolvePipelineRecordsValidatorChains(t *testing.T) {
catalog := newProfileCatalog(t) catalog := newProfileCatalog(t)
if err := catalog.ValidatorChains.Register(ValidatorChainMapping{ if err := catalog.ValidatorChains.Register(ValidatorChainMapping{
@@ -337,6 +556,7 @@ func TestResolvePipelinePreservesOrderedStepsAndExpandsGeneratedBindings(t *test
catalog := newProfileCatalogWithOverrides(t, catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{ ModuleSpec{
Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes",
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"}, Provides: []string{"candidate"}, Requires: []string{"chunk"}, Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}, AcceptedMediaTypes: []string{"application/json"}}}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}, AcceptedMediaTypes: []string{"application/json"}}},
}) })
@@ -365,7 +585,7 @@ func TestResolvePipelinePreservesOrderedStepsAndExpandsGeneratedBindings(t *test
func TestResolvePipelineRejectsGeneratedBindingOrderingAndKind(t *testing.T) { func TestResolvePipelineRejectsGeneratedBindingOrderingAndKind(t *testing.T) {
catalog := newProfileCatalogWithOverrides(t, catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/other"}}}}, ModuleSpec{Key: "note-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/other"}}}},
) )
_, err := ResolvePipeline(PipelineProfile{ _, err := ResolvePipeline(PipelineProfile{
ID: "invalid-order", Input: Binding("text"), Steps: []PipelineStepProfile{ ID: "invalid-order", Input: Binding("text"), Steps: []PipelineStepProfile{
@@ -402,6 +622,7 @@ func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
catalog := newProfileCatalogWithOverride(t, ModuleSpec{ catalog := newProfileCatalogWithOverride(t, ModuleSpec{
Key: "event-extractor", Key: "event-extractor",
Stage: StageExtract, Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"}, Requires: []string{"chunk"},
Provides: []string{"candidate"}, Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{ ReferenceSlots: []contracts.ReferenceSlot{
@@ -446,6 +667,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToChunkTarget(t *testing.
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "generic", Key: "generic",
Stage: StageChunk, Stage: StageChunk,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"source"}, Requires: []string{"source"},
Provides: []string{"chunk"}, Provides: []string{"chunk"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide"}}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide"}},
@@ -474,6 +696,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToExtractorTarget(t *test
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "event-extractor", Key: "event-extractor",
Stage: StageExtract, Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"}, Requires: []string{"chunk"},
Provides: []string{"candidate"}, Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
@@ -502,6 +725,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToNormalizerTarget(t *tes
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "noop", Key: "noop",
Stage: StageNormalize, Stage: StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"merged"}, Requires: []string{"merged"},
Provides: []string{"normalized"}, Provides: []string{"normalized"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
@@ -530,6 +754,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToMergeTarget(t *testing.
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "appendorder", Key: "appendorder",
Stage: StageMerge, Stage: StageMerge,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"candidate"}, Requires: []string{"candidate"},
Provides: []string{"merged"}, Provides: []string{"merged"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "merge_notes"}}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "merge_notes"}},
@@ -559,10 +784,10 @@ func TestResolvePipelineAppliesOnePipelineReferenceDefaultToMultipleTargets(t *t
profile := baselineProfile() profile := baselineProfile()
profile.References = ExternalReferenceMap(map[string]string{"context": "./context.md"}) profile.References = ExternalReferenceMap(map[string]string{"context": "./context.md"})
catalog := newProfileCatalogWithOverrides(t, catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}}, ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}}, ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}}, ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"candidate"}, Provides: []string{"merged"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}}, ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
) )
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog) resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
@@ -582,6 +807,7 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedLane(t *t
catalog := newProfileCatalogWithOverride(t, ModuleSpec{ catalog := newProfileCatalogWithOverride(t, ModuleSpec{
Key: "note-extractor", Key: "note-extractor",
Stage: StageExtract, Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"}, Requires: []string{"chunk"},
Provides: []string{"candidate"}, Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{ ReferenceSlots: []contracts.ReferenceSlot{
@@ -607,6 +833,7 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedNormalize
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "note-normalizer", Key: "note-normalizer",
Stage: StageNormalize, Stage: StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"merged"}, Requires: []string{"merged"},
Provides: []string{"normalized"}, Provides: []string{"normalized"},
ReferenceSlots: []contracts.ReferenceSlot{ ReferenceSlots: []contracts.ReferenceSlot{
@@ -658,6 +885,7 @@ func TestResolvePipelineRejectsExtractLocalReferenceDeclaredOnlyByNormalizer(t *
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "noop", Key: "noop",
Stage: StageNormalize, Stage: StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"merged"}, Requires: []string{"merged"},
Provides: []string{"normalized"}, Provides: []string{"normalized"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
@@ -678,6 +906,7 @@ func TestResolvePipelineRejectsMergeLocalReferenceDeclaredOnlyByNormalizer(t *te
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "noop", Key: "noop",
Stage: StageNormalize, Stage: StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"merged"}, Requires: []string{"merged"},
Provides: []string{"normalized"}, Provides: []string{"normalized"},
ReferenceSlots: []contracts.ReferenceSlot{ ReferenceSlots: []contracts.ReferenceSlot{
@@ -700,6 +929,7 @@ func TestResolvePipelineRejectsNormalizeLocalReferenceDeclaredOnlyByExtractor(t
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "event-extractor", Key: "event-extractor",
Stage: StageExtract, Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"}, Requires: []string{"chunk"},
Provides: []string{"candidate"}, Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
@@ -716,6 +946,7 @@ func TestResolvePipelineRequiresBoundChunkReference(t *testing.T) {
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "generic", Key: "generic",
Stage: StageChunk, Stage: StageChunk,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"source"}, Requires: []string{"source"},
Provides: []string{"chunk"}, Provides: []string{"chunk"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide", Required: true}}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide", Required: true}},
@@ -732,6 +963,7 @@ func TestResolvePipelineRequiresBoundNormalizeReference(t *testing.T) {
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "noop", Key: "noop",
Stage: StageNormalize, Stage: StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"merged"}, Requires: []string{"merged"},
Provides: []string{"normalized"}, Provides: []string{"normalized"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes", Required: true}}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes", Required: true}},
@@ -757,9 +989,9 @@ func TestResolvePipelineLocalReferencesOverridePipelineDefaultsForEligibleTarget
lane.Normalize.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./local-normalize.md"}) lane.Normalize.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./local-normalize.md"})
profile.Artifacts["events"] = lane profile.Artifacts["events"] = lane
catalog := newProfileCatalogWithOverrides(t, catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}}, ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}}}, ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}}},
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}}}, ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}}},
) )
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog) resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
@@ -776,6 +1008,7 @@ func TestResolvePipelineRequiresBoundReferenceSlotsForSelectedLanes(t *testing.T
catalog := newProfileCatalogWithOverride(t, ModuleSpec{ catalog := newProfileCatalogWithOverride(t, ModuleSpec{
Key: "event-extractor", Key: "event-extractor",
Stage: StageExtract, Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"}, Requires: []string{"chunk"},
Provides: []string{"candidate"}, Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{ ReferenceSlots: []contracts.ReferenceSlot{
@@ -800,6 +1033,7 @@ func TestResolvePipelineReferenceUnbindCanLeaveRequiredSlotMissing(t *testing.T)
catalog := newProfileCatalogWithOverride(t, ModuleSpec{ catalog := newProfileCatalogWithOverride(t, ModuleSpec{
Key: "event-extractor", Key: "event-extractor",
Stage: StageExtract, Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"}, Requires: []string{"chunk"},
Provides: []string{"candidate"}, Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{ ReferenceSlots: []contracts.ReferenceSlot{
@@ -829,6 +1063,7 @@ func TestResolvePipelineUsesReferenceSlotsFromSpecWithoutConstructingExtractor(t
if err := RegisterExtractor[codecNotes](catalog.Extractors, ModuleSpec{ if err := RegisterExtractor[codecNotes](catalog.Extractors, ModuleSpec{
Key: "event-extractor", Key: "event-extractor",
Stage: StageExtract, Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
ArtifactKind: "test/notes", ArtifactKind: "test/notes",
Requires: []string{"chunk"}, Requires: []string{"chunk"},
Provides: []string{"candidate"}, Provides: []string{"candidate"},
@@ -989,32 +1224,32 @@ func TestResolvePipelineRejectsMissingCapabilities(t *testing.T) {
}{ }{
{ {
name: "input", name: "input",
spec: ModuleSpec{Key: "text", Stage: StageInput, Requires: []string{"raw"}}, spec: ModuleSpec{Key: "text", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"raw"}},
want: []string{"baseline", "input", "text", "raw"}, want: []string{"baseline", "input", "text", "raw"},
}, },
{ {
name: "chunk", name: "chunk",
spec: ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"missing"}}, spec: ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
want: []string{"baseline", "chunk", "generic", "missing"}, want: []string{"baseline", "chunk", "generic", "missing"},
}, },
{ {
name: "extract", name: "extract",
spec: ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"missing"}}, spec: ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
want: []string{"baseline", "events", "extract", "event-extractor", "missing"}, want: []string{"baseline", "events", "extract", "event-extractor", "missing"},
}, },
{ {
name: "merge", name: "merge",
spec: ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"missing"}}, spec: ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
want: []string{"baseline", "events", "merge", "appendorder", "missing"}, want: []string{"baseline", "events", "merge", "appendorder", "missing"},
}, },
{ {
name: "normalize", name: "normalize",
spec: ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"missing"}}, spec: ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
want: []string{"baseline", "events", "normalize", "noop", "missing"}, want: []string{"baseline", "events", "normalize", "noop", "missing"},
}, },
{ {
name: "output", name: "output",
spec: ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"missing"}}, spec: ModuleSpec{Key: "json", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
want: []string{"baseline", "output", "json", "missing"}, want: []string{"baseline", "output", "json", "missing"},
}, },
} }
@@ -1100,7 +1335,7 @@ func TestResolvePipelineDigestChangesWhenBindingChanges(t *testing.T) {
right := baselineProfile() right := baselineProfile()
right.Chunk = Binding("window") right.Chunk = Binding("window")
catalog := newProfileCatalog(t) catalog := newProfileCatalog(t)
registerProfileSpecs(t, catalog, ModuleSpec{Key: "window", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}}) registerProfileSpecs(t, catalog, ModuleSpec{Key: "window", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}})
leftResolved, err := ResolvePipeline(left, ResolveOptions{}, catalog) leftResolved, err := ResolvePipeline(left, ResolveOptions{}, catalog)
if err != nil { if err != nil {
@@ -1467,17 +1702,103 @@ func emptyProfileCatalog() ModuleCatalog {
func defaultProfileSpecs() []ModuleSpec { func defaultProfileSpecs() []ModuleSpec {
return []ModuleSpec{ return []ModuleSpec{
ModuleSpec{Key: "text", Stage: StageInput, Provides: []string{"source"}}, ModuleSpec{Key: "text", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}},
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}}, ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}},
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}}, ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}}, ModuleSpec{Key: "note-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "appendorder", Stage: StageMerge, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}}, ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
ModuleSpec{Key: "noop", Stage: StageNormalize, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}}, ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
ModuleSpec{Key: "grounded", Stage: StageValidate, Requires: []string{"normalized"}, Provides: []string{"validated"}}, ModuleSpec{Key: "grounded", Stage: StageValidate, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"validated"}},
ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}}, ModuleSpec{Key: "json", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
} }
} }
func llmProfileCatalog(t *testing.T) ModuleCatalog {
t.Helper()
catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked, Provides: []string{"source"}},
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"source"}, Provides: []string{"chunk"}},
ModuleSpec{Key: "llm-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
)
if err := RegisterChunkValidator(catalog.Validators, ValidatorSpec{Key: "llm-chunk-validator", ExecutionClass: contracts.ExecutionClassLLMBacked}, func() (contracts.ChunkValidator, error) {
return typedTestChunkValidator{key: "llm-chunk-validator"}, nil
}); err != nil {
t.Fatalf("register chunk validator: %v", err)
}
registerProfileValidatorSpec(t, catalog, ValidatorSpec{Key: "llm-validator", ExecutionClass: contracts.ExecutionClassLLMBacked})
for _, mapping := range []ValidatorChainMapping{
{Stage: StageChunk, Module: "llm-chunk", Validators: []ModuleBinding{Binding("llm-chunk-validator")}},
{Stage: StageExtract, Module: "llm-extractor", Validators: []ModuleBinding{Binding("llm-validator")}},
{Stage: StageMerge, Module: "llm-merge", Validators: []ModuleBinding{Binding("llm-validator")}},
{Stage: StageNormalize, Module: "llm-normalize", Validators: []ModuleBinding{Binding("llm-validator")}},
} {
if err := catalog.ValidatorChains.Register(mapping); err != nil {
t.Fatalf("register validator chain %#v: %v", mapping, err)
}
}
return catalog
}
func llmProfileCatalogWithoutValidatorChains(t *testing.T) ModuleCatalog {
catalog := llmProfileCatalog(t)
catalog.ValidatorChains = NewValidatorChainRegistry()
return catalog
}
func llmProfilePipeline() PipelineProfile {
return PipelineProfile{
ID: "llm-profile",
Input: Binding("llm-input"),
Chunk: Binding("llm-chunk"),
Output: Binding("llm-output"),
Artifacts: map[string]ArtifactLaneProfile{
"events": {
Extract: Binding("llm-extractor"),
Merge: Binding("llm-merge"),
Normalize: Binding("llm-normalize"),
},
},
}
}
func llmProfileValues(profile string) map[string]string {
return map[string]string{
"input": profile,
"chunk": profile,
"extract": profile,
"merge": profile,
"normalize": profile,
"output": profile,
"validator:chunk:": profile,
"validator:extract:events": profile,
"validator:merge:events": profile,
"validator:normalize:events": profile,
}
}
func resolvedLLMProfileValues(resolved ResolvedPipeline) map[string]string {
values := map[string]string{
"input": resolved.Input.LLMProfile,
"chunk": resolved.Chunk.LLMProfile,
"output": resolved.Output.LLMProfile,
}
lane := resolved.Steps[0].ArtifactLanes[0]
values["extract"] = lane.Extract.LLMProfile
values["merge"] = lane.Merge.LLMProfile
values["normalize"] = lane.Normalize.LLMProfile
for _, chain := range resolved.ValidatorChains {
for _, validator := range chain.Validators {
if validator.ExecutionClass == contracts.ExecutionClassLLMBacked {
values["validator:"+string(chain.Stage)+":"+chain.LaneID] = validator.Binding.LLMProfile
}
}
}
return values
}
func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSpec) { func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSpec) {
t.Helper() t.Helper()
@@ -1566,7 +1887,13 @@ func TestResolvedPipelineCanMarshalToCanonicalJSON(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err) t.Fatalf("ResolvePipeline() error = %v, want nil", err)
} }
if _, err := json.Marshal(resolved); err != nil { encoded, err := json.Marshal(resolved)
if err != nil {
t.Fatalf("json.Marshal(resolved) error = %v, want nil", err) t.Fatalf("json.Marshal(resolved) error = %v, want nil", err)
} }
for _, field := range []string{"input_execution_class", "chunk_execution_class", "extract_execution_class", "merge_execution_class", "normalize_execution_class", "output_execution_class"} {
if !strings.Contains(string(encoded), `"`+field+`":"deterministic"`) {
t.Fatalf("resolved JSON does not retain %q: %s", field, encoded)
}
}
} }

View File

@@ -189,12 +189,12 @@ func TestMaterializeReferencesUsesLaneArtifactVariant(t *testing.T) {
{kind: "test/beta", mergeSlot: "beta_merge", normalizeSlot: "beta_normalize"}, {kind: "test/beta", mergeSlot: "beta_merge", normalizeSlot: "beta_normalize"},
{kind: "test/alpha", mergeSlot: "alpha_merge", normalizeSlot: "alpha_normalize"}, {kind: "test/alpha", mergeSlot: "alpha_merge", normalizeSlot: "alpha_normalize"},
} { } {
if err := RegisterMerger(mergers, ModuleSpec{Key: "shared/merge", Stage: StageMerge, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.mergeSlot}}}, func() (contracts.Merger[codecNotes], error) { if err := RegisterMerger(mergers, ModuleSpec{Key: "shared/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.mergeSlot}}}, func() (contracts.Merger[codecNotes], error) {
return typedTestMerger[codecNotes]{key: "shared/merge"}, nil return typedTestMerger[codecNotes]{key: "shared/merge"}, nil
}); err != nil { }); err != nil {
t.Fatalf("RegisterMerger(%s): %v", item.kind, err) t.Fatalf("RegisterMerger(%s): %v", item.kind, err)
} }
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "shared/normalize", Stage: StageNormalize, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.normalizeSlot}}}, func() (contracts.Normalizer[codecNotes], error) { if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "shared/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.normalizeSlot}}}, func() (contracts.Normalizer[codecNotes], error) {
return typedTestNormalizer[codecNotes]{key: "shared/normalize"}, nil return typedTestNormalizer[codecNotes]{key: "shared/normalize"}, nil
}); err != nil { }); err != nil {
t.Fatalf("RegisterNormalizer(%s): %v", item.kind, err) t.Fatalf("RegisterNormalizer(%s): %v", item.kind, err)
@@ -531,6 +531,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
ModuleSpec{ ModuleSpec{
Key: "generic", Key: "generic",
Stage: StageChunk, Stage: StageChunk,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"source"}, Requires: []string{"source"},
Provides: []string{"chunk"}, Provides: []string{"chunk"},
ReferenceSlots: chunkSlots, ReferenceSlots: chunkSlots,
@@ -538,6 +539,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
ModuleSpec{ ModuleSpec{
Key: "event-extractor", Key: "event-extractor",
Stage: StageExtract, Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"}, Requires: []string{"chunk"},
Provides: []string{"candidate"}, Provides: []string{"candidate"},
ReferenceSlots: extractSlots, ReferenceSlots: extractSlots,
@@ -545,6 +547,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
ModuleSpec{ ModuleSpec{
Key: "appendorder", Key: "appendorder",
Stage: StageMerge, Stage: StageMerge,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"candidate"}, Requires: []string{"candidate"},
Provides: []string{"merged"}, Provides: []string{"merged"},
ReferenceSlots: mergeSlots, ReferenceSlots: mergeSlots,
@@ -552,6 +555,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
ModuleSpec{ ModuleSpec{
Key: "noop", Key: "noop",
Stage: StageNormalize, Stage: StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"merged"}, Requires: []string{"merged"},
Provides: []string{"normalized"}, Provides: []string{"normalized"},
ReferenceSlots: normalizeSlots, ReferenceSlots: normalizeSlots,

View File

@@ -829,18 +829,12 @@ func mergeLLMProfileManifests(sources ...[]artifacts.LLMProfileManifest) []artif
merged := make(map[string]artifacts.LLMProfileManifest) merged := make(map[string]artifacts.LLMProfileManifest)
for _, source := range sources { for _, source := range sources {
for _, profile := range source { for _, profile := range source {
id := strings.TrimSpace(profile.ID) profile = profile.Normalized()
provider := strings.TrimSpace(profile.Provider) key := profile.IdentityKey()
model := strings.TrimSpace(profile.Model)
key := id + "\x00" + provider + "\x00" + model
if _, exists := merged[key]; exists { if _, exists := merged[key]; exists {
continue continue
} }
merged[key] = artifacts.LLMProfileManifest{ merged[key] = profile
ID: id,
Provider: provider,
Model: model,
}
} }
} }
if len(merged) == 0 { if len(merged) == 0 {

View File

@@ -116,7 +116,7 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
return false, nil, terminal.record(nil, fmt.Errorf("clone chunker manifest metadata: %w", metadataErr)) return false, nil, terminal.record(nil, fmt.Errorf("clone chunker manifest metadata: %w", metadataErr))
} }
profile := "" profile := ""
if provider, ok := chunker.(contracts.ChunkExecutionClassProvider); ok && provider.ExecutionClass() == contracts.ExecutionClassLLMBacked { if input.pipeline.ChunkExecutionClass == contracts.ExecutionClassLLMBacked {
profile = input.pipeline.Chunk.LLMProfile profile = input.pipeline.Chunk.LLMProfile
} }
candidate := ChunkPlanRecord{ candidate := ChunkPlanRecord{

View File

@@ -67,30 +67,17 @@ type manifestChunker struct {
} }
func (c manifestChunker) ManifestMetadata() map[string]any { return c.metadata } func (c manifestChunker) ManifestMetadata() map[string]any { return c.metadata }
func (manifestChunker) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassLLMBacked
}
type llmCountingChunker struct { type llmCountingChunker struct {
terminalChunker terminalChunker
llmCalls *int llmCalls *int
} }
func (llmCountingChunker) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassLLMBacked
}
func (c llmCountingChunker) Plan(ctx context.Context, request contracts.ChunkRequest) (contracts.ChunkPlanResult, error) { func (c llmCountingChunker) Plan(ctx context.Context, request contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
(*c.llmCalls)++ (*c.llmCalls)++
return c.terminalChunker.Plan(ctx, request) return c.terminalChunker.Plan(ctx, request)
} }
type deterministicChunker struct{ terminalChunker }
func (deterministicChunker) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassDeterministic
}
type retryingChunker struct { type retryingChunker struct {
key string key string
plan source.ChunkPlan plan source.ChunkPlan
@@ -535,6 +522,7 @@ func TestRunnerPublishesOnlyAcceptedGeneratedPlans(t *testing.T) {
func TestRunnerStoresProducerProvenanceAndProducerWarnings(t *testing.T) { func TestRunnerStoresProducerProvenanceAndProducerWarnings(t *testing.T) {
prepared, plan := preparedTerminalDebugPipeline(t) prepared, plan := preparedTerminalDebugPipeline(t)
prepared.resolved.Chunk.LLMProfile = "chunk-profile" prepared.resolved.Chunk.LLMProfile = "chunk-profile"
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
prepared.resolved.ChunkReferences = ResolvedReferenceTarget{ prepared.resolved.ChunkReferences = ResolvedReferenceTarget{
Stage: StageChunk, Stage: StageChunk,
ReferenceSet: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{ ReferenceSet: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
@@ -577,7 +565,8 @@ func TestRunnerRejectsUncloneableModuleManifestMetadata(t *testing.T) {
func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) { func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
prepared, plan := preparedTerminalDebugPipeline(t) prepared, plan := preparedTerminalDebugPipeline(t)
prepared.resolved.Chunk.LLMProfile = "configured-but-unused" prepared.resolved.Chunk.LLMProfile = "configured-but-unused"
prepared.chunker = deterministicChunker{terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}} prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassDeterministic
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}
store := &recordingChunkPlanStore{} store := &recordingChunkPlanStore{}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store}) output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store})
if err != nil { if err != nil {
@@ -588,6 +577,22 @@ func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
} }
} }
func TestRunnerDoesNotInventProducerProfileForLLMChunker(t *testing.T) {
prepared, plan := preparedTerminalDebugPipeline(t)
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
prepared.resolved.Chunk.LLMProfile = ""
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}
store := &recordingChunkPlanStore{}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store})
if err != nil {
t.Fatal(err)
}
if store.saved.Producer.LLMProfile != "" || output.Manifest.ChunkPlan.ProducerLLMProfile != "" {
t.Fatalf("LLM producer profile = stored %q manifest %q, want empty", store.saved.Producer.LLMProfile, output.Manifest.ChunkPlan.ProducerLLMProfile)
}
}
func TestRunnerRefreshChangesDownstreamChunkFingerprint(t *testing.T) { func TestRunnerRefreshChangesDownstreamChunkFingerprint(t *testing.T) {
doc := typedTestDocumentWithUnits(2) doc := typedTestDocumentWithUnits(2)
prepared := preparedConcurrentPipeline(t, 1) prepared := preparedConcurrentPipeline(t, 1)

View File

@@ -0,0 +1,29 @@
package pipeline
import (
"reflect"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
)
func TestMergeLLMProfileManifestsDistinguishesEffectiveTargets(t *testing.T) {
got := mergeLLMProfileManifests(
[]artifacts.LLMProfileManifest{
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-b", ReasoningEffort: "low"},
{ID: " profile ", Provider: " promptkit ", Model: " model ", BackendID: " backend-a ", ReasoningEffort: " low "},
},
[]artifacts.LLMProfileManifest{
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "high"},
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "low"},
},
)
want := []artifacts.LLMProfileManifest{
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "high"},
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "low"},
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-b", ReasoningEffort: "low"},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("merged profiles = %#v, want %#v", got, want)
}
}

View File

@@ -229,7 +229,7 @@ func TestResolveTypedLaneRejectsIncompatibleComposition(t *testing.T) {
func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) { func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) {
registry := NewMergerRegistry() registry := NewMergerRegistry()
spec := ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"} spec := ModuleSpec{Key: "typed/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}
constructor := func() (contracts.Merger[codecNotes], error) { constructor := func() (contracts.Merger[codecNotes], error) {
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
} }
@@ -243,7 +243,7 @@ func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) {
func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) { func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
extractors := NewExtractorRegistry() extractors := NewExtractorRegistry()
if err := RegisterExtractor(extractors, ModuleSpec{Key: "typed/extract", Stage: StageExtract, ArtifactKind: "test/notes"}, func() (contracts.Extractor[codecNotes], error) { if err := RegisterExtractor(extractors, ModuleSpec{Key: "typed/extract", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Extractor[codecNotes], error) {
return typedTestExtractor[codecNotes]{key: "typed/extract"}, nil return typedTestExtractor[codecNotes]{key: "typed/extract"}, nil
}); err != nil { }); err != nil {
t.Fatalf("RegisterExtractor() error = %v", err) t.Fatalf("RegisterExtractor() error = %v", err)
@@ -253,7 +253,7 @@ func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
} }
mergers := NewMergerRegistry() mergers := NewMergerRegistry()
if err := RegisterMerger(mergers, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"}, func() (contracts.Merger[codecNotes], error) { if err := RegisterMerger(mergers, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Merger[codecNotes], error) {
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
}); err != nil { }); err != nil {
t.Fatalf("RegisterMerger() error = %v", err) t.Fatalf("RegisterMerger() error = %v", err)
@@ -263,7 +263,7 @@ func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
} }
normalizers := NewNormalizerRegistry() normalizers := NewNormalizerRegistry()
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) { if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
return typedTestNormalizer[codecNotes]{key: "typed/normalize"}, nil return typedTestNormalizer[codecNotes]{key: "typed/normalize"}, nil
}); err != nil { }); err != nil {
t.Fatalf("RegisterNormalizer() error = %v", err) t.Fatalf("RegisterNormalizer() error = %v", err)
@@ -298,7 +298,7 @@ func TestTypedRegistrySpecLookupUsesArtifactKindAndStableCatalogOrder(t *testing
mergers := NewMergerRegistry() mergers := NewMergerRegistry()
normalizers := NewNormalizerRegistry() normalizers := NewNormalizerRegistry()
for _, item := range order { for _, item := range order {
base := ModuleSpec{Key: "typed/shared", ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.slot}}} base := ModuleSpec{Key: "typed/shared", ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.slot}}}
mergeSpec := base mergeSpec := base
mergeSpec.Stage = StageMerge mergeSpec.Stage = StageMerge
normalizeSpec := base normalizeSpec := base
@@ -474,18 +474,18 @@ func typedResolutionCatalog(t *testing.T, options typedCatalogOptions) ModuleCat
func mustRegisterTypedTestBase(t *testing.T, catalog ModuleCatalog) { func mustRegisterTypedTestBase(t *testing.T, catalog ModuleCatalog) {
t.Helper() t.Helper()
if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{Key: "typed/input", Stage: StageInput}, func() (contracts.InputAdapter, error) { if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{Key: "typed/input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.InputAdapter, error) {
return &typedTestInput{key: "typed/input", doc: typedTestDocument()}, nil return &typedTestInput{key: "typed/input", doc: typedTestDocument()}, nil
}); err != nil { }); err != nil {
t.Fatalf("register input: %v", err) t.Fatalf("register input: %v", err)
} }
if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{Key: "typed/chunk", Stage: StageChunk}, func() (contracts.Chunker, error) { if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{Key: "typed/chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.Chunker, error) {
doc := typedTestDocument() doc := typedTestDocument()
return &typedTestChunker{key: "typed/chunk", plan: typedTestPlan(doc)}, nil return &typedTestChunker{key: "typed/chunk", plan: typedTestPlan(doc)}, nil
}); err != nil { }); err != nil {
t.Fatalf("register chunker: %v", err) t.Fatalf("register chunker: %v", err)
} }
if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{Key: "typed/output", Stage: StageOutput}, func() (contracts.OutputEncoder, error) { if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{Key: "typed/output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.OutputEncoder, error) {
return &typedTestOutput{key: "typed/output"}, nil return &typedTestOutput{key: "typed/output"}, nil
}); err != nil { }); err != nil {
t.Fatalf("register output: %v", err) t.Fatalf("register output: %v", err)
@@ -509,21 +509,21 @@ func mustRegisterArtifactCodec[T any](t *testing.T, registry *ArtifactCodecRegis
func mustRegisterTypedExtractor[T any](t *testing.T, registry *ExtractorRegistry, key string, kind contracts.ArtifactKind, extractor contracts.Extractor[T]) { func mustRegisterTypedExtractor[T any](t *testing.T, registry *ExtractorRegistry, key string, kind contracts.ArtifactKind, extractor contracts.Extractor[T]) {
t.Helper() t.Helper()
if err := RegisterExtractor(registry, ModuleSpec{Key: key, Stage: StageExtract, ArtifactKind: kind}, func() (contracts.Extractor[T], error) { return extractor, nil }); err != nil { if err := RegisterExtractor(registry, ModuleSpec{Key: key, Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: kind}, func() (contracts.Extractor[T], error) { return extractor, nil }); err != nil {
t.Fatalf("RegisterExtractor() error = %v", err) t.Fatalf("RegisterExtractor() error = %v", err)
} }
} }
func mustRegisterTypedMerger[T any](t *testing.T, registry *MergerRegistry, kind contracts.ArtifactKind, merger contracts.Merger[T]) { func mustRegisterTypedMerger[T any](t *testing.T, registry *MergerRegistry, kind contracts.ArtifactKind, merger contracts.Merger[T]) {
t.Helper() t.Helper()
if err := RegisterMerger(registry, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: kind}, func() (contracts.Merger[T], error) { return merger, nil }); err != nil { if err := RegisterMerger(registry, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: kind}, func() (contracts.Merger[T], error) { return merger, nil }); err != nil {
t.Fatalf("RegisterMerger() error = %v", err) t.Fatalf("RegisterMerger() error = %v", err)
} }
} }
func mustRegisterTypedNormalizer[T any](t *testing.T, registry *NormalizerRegistry, kind contracts.ArtifactKind, normalizer contracts.Normalizer[T]) { func mustRegisterTypedNormalizer[T any](t *testing.T, registry *NormalizerRegistry, kind contracts.ArtifactKind, normalizer contracts.Normalizer[T]) {
t.Helper() t.Helper()
if err := RegisterNormalizer(registry, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: kind}, func() (contracts.Normalizer[T], error) { return normalizer, nil }); err != nil { if err := RegisterNormalizer(registry, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: kind}, func() (contracts.Normalizer[T], error) { return normalizer, nil }); err != nil {
t.Fatalf("RegisterNormalizer() error = %v", err) t.Fatalf("RegisterNormalizer() error = %v", err)
} }
} }

View File

@@ -11,15 +11,15 @@ import (
"time" "time"
) )
// ModulePromptFile maps a module-owned embedded prompt file into the // ModulePromptFile maps a module-owned embedded prompt file into the registered
// PromptKit-visible module prompt directory. // module prompt directory.
type ModulePromptFile struct { type ModulePromptFile struct {
Name string Name string
Path string Path string
} }
// SharedPromptFile maps a caller-owned shared prompt file into a module's // SharedPromptFile maps a caller-owned shared prompt file into a module's
// PromptKit-visible sharedassets prompt subdirectory. // registered sharedassets prompt subdirectory.
type SharedPromptFile struct { type SharedPromptFile struct {
Name string Name string
FS fs.FS FS fs.FS

View File

@@ -1,6 +1,6 @@
id: dnd.scenes id: dnd.scenes
version: "v1" version: "v1"
default_profile: gemini-2-flash default_profile: dnd-extraction
inputs: inputs:
- name: transcript - name: transcript
required: true required: true

View File

@@ -47,10 +47,6 @@ func (c *Chunker) Key() string {
return Key return Key
} }
func (*Chunker) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassLLMBacked
}
func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot { func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
return shared.ReferenceSlots(referenceSlotDescriptions) return shared.ReferenceSlots(referenceSlotDescriptions)
} }
@@ -120,6 +116,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{ return pipeline.ModuleSpec{
Key: Key, Key: Key,
Stage: pipeline.StageChunk, Stage: pipeline.StageChunk,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...), Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...), Provides: append([]string(nil), providedCapabilities...),
ReferenceSlots: shared.ReferenceSlots(referenceSlotDescriptions), ReferenceSlots: shared.ReferenceSlots(referenceSlotDescriptions),

View File

@@ -24,6 +24,7 @@ func TestNewModuleSpecAndRegister(t *testing.T) {
want := pipeline.ModuleSpec{ want := pipeline.ModuleSpec{
Key: Key, Key: Key,
Stage: pipeline.StageChunk, Stage: pipeline.StageChunk,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: []string{"source.transcript"}, Requires: []string{"source.transcript"},
Provides: []string{"chunks"}, Provides: []string{"chunks"},
ReferenceSlots: wantReferenceSlots(), ReferenceSlots: wantReferenceSlots(),

View File

@@ -1,6 +1,6 @@
id: dnd.combat_turns id: dnd.combat_turns
version: "v1" version: "v1"
default_profile: gemini-2-flash default_profile: dnd-extraction
inputs: inputs:
- name: transcript - name: transcript
required: true required: true

View File

@@ -226,6 +226,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{ return pipeline.ModuleSpec{
Key: Key, Key: Key,
Stage: pipeline.StageExtract, Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...), Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...), Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.CombatTurnListKind, ArtifactKind: dnd.CombatTurnListKind,

View File

@@ -1,6 +1,6 @@
id: dnd.item_events id: dnd.item_events
version: "v1" version: "v1"
default_profile: gemini-2-flash default_profile: dnd-extraction
inputs: inputs:
- name: transcript - name: transcript
required: true required: true

View File

@@ -129,6 +129,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{ return pipeline.ModuleSpec{
Key: Key, Key: Key,
Stage: pipeline.StageExtract, Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...), Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...), Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.ItemEventListKind, ArtifactKind: dnd.ItemEventListKind,

View File

@@ -18,7 +18,7 @@ func TestConstructorSpecOptionsAndMetadata(t *testing.T) {
t.Fatalf("New() error = %v", err) t.Fatalf("New() error = %v", err)
} }
want := pipeline.ModuleSpec{ want := pipeline.ModuleSpec{
Key: Key, Stage: pipeline.StageExtract, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.item_events"}, ArtifactKind: dnd.ItemEventListKind, Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.item_events"}, ArtifactKind: dnd.ItemEventListKind,
ReferenceSlots: []contracts.ReferenceSlot{ ReferenceSlots: []contracts.ReferenceSlot{
{Name: "glossary", Description: referenceSlotDescriptions.Glossary, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}}, {Name: "glossary", Description: referenceSlotDescriptions.Glossary, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
{Name: "party", Description: referenceSlotDescriptions.Party, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}}, {Name: "party", Description: referenceSlotDescriptions.Party, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},

View File

@@ -1,6 +1,6 @@
id: dnd.npc_interactions id: dnd.npc_interactions
version: "v1" version: "v1"
default_profile: gemini-2-flash default_profile: dnd-extraction
inputs: inputs:
- name: transcript - name: transcript
required: true required: true

View File

@@ -177,6 +177,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{ return pipeline.ModuleSpec{
Key: Key, Key: Key,
Stage: pipeline.StageExtract, Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...), Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...), Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.NPCInteractionListKind, ArtifactKind: dnd.NPCInteractionListKind,

View File

@@ -1,6 +1,6 @@
id: dnd.npcs id: dnd.npcs
version: "v1" version: "v1"
default_profile: gemini-2-flash default_profile: dnd-extraction
inputs: inputs:
- name: transcript - name: transcript
required: true required: true

View File

@@ -135,6 +135,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{ return pipeline.ModuleSpec{
Key: Key, Key: Key,
Stage: pipeline.StageExtract, Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...), Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...), Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.NPCListKind, ArtifactKind: dnd.NPCListKind,

View File

@@ -27,6 +27,7 @@ func TestModuleSpecAndReferenceSlots(t *testing.T) {
want := pipeline.ModuleSpec{ want := pipeline.ModuleSpec{
Key: Key, Key: Key,
Stage: pipeline.StageExtract, Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: []string{"chunks", "source.transcript"}, Requires: []string{"chunks", "source.transcript"},
Provides: []string{"dnd.npcs"}, Provides: []string{"dnd.npcs"},
ArtifactKind: dnd.NPCListKind, ArtifactKind: dnd.NPCListKind,

View File

@@ -1,6 +1,6 @@
id: dnd.scene_descriptions id: dnd.scene_descriptions
version: "v1" version: "v1"
default_profile: gemini-2-flash default_profile: dnd-extraction
inputs: inputs:
- name: transcript - name: transcript
required: true required: true

View File

@@ -146,6 +146,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{ return pipeline.ModuleSpec{
Key: Key, Key: Key,
Stage: pipeline.StageExtract, Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...), Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...), Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.SceneDescriptionListKind, ArtifactKind: dnd.SceneDescriptionListKind,

View File

@@ -24,7 +24,7 @@ func TestNewRequiresLLMClientAndRejectsAmbiguousReferences(t *testing.T) {
func TestModuleSpecAndReferenceSlots(t *testing.T) { func TestModuleSpecAndReferenceSlots(t *testing.T) {
want := pipeline.ModuleSpec{ want := pipeline.ModuleSpec{
Key: Key, Stage: pipeline.StageExtract, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.scene_descriptions"}, ArtifactKind: dnd.SceneDescriptionListKind, Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.scene_descriptions"}, ArtifactKind: dnd.SceneDescriptionListKind,
ReferenceSlots: []contracts.ReferenceSlot{ ReferenceSlots: []contracts.ReferenceSlot{
{Name: "glossary", Description: "Optional campaign glossary reference material used only to disambiguate scene descriptions.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}}, {Name: "glossary", Description: "Optional campaign glossary reference material used only to disambiguate scene descriptions.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
{Name: "party", Description: "Optional party roster reference material used only to disambiguate scene descriptions.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}}, {Name: "party", Description: "Optional party roster reference material used only to disambiguate scene descriptions.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},

View File

@@ -1,6 +1,6 @@
id: dnd.spells id: dnd.spells
version: "v1" version: "v1"
default_profile: gemini-2-flash default_profile: dnd-extraction
inputs: inputs:
- name: transcript - name: transcript
required: true required: true

View File

@@ -200,6 +200,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{ return pipeline.ModuleSpec{
Key: Key, Key: Key,
Stage: pipeline.StageExtract, Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...), Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...), Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.SpellListKind, ArtifactKind: dnd.SpellListKind,

View File

@@ -25,6 +25,7 @@ func TestModuleSpec(t *testing.T) {
want := pipeline.ModuleSpec{ want := pipeline.ModuleSpec{
Key: Key, Key: Key,
Stage: pipeline.StageExtract, Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: []string{ Requires: []string{
"chunks", "chunks",
"source.transcript", "source.transcript",

View File

@@ -336,6 +336,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{ return pipeline.ModuleSpec{
Key: Key, Key: Key,
Stage: pipeline.StageNormalize, Stage: pipeline.StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: append([]string(nil), requiredCapabilities...), Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...), Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.CombatTurnListKind, ArtifactKind: dnd.CombatTurnListKind,

View File

@@ -224,6 +224,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{ return pipeline.ModuleSpec{
Key: Key, Key: Key,
Stage: pipeline.StageNormalize, Stage: pipeline.StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: append([]string(nil), requiredCapabilities...), Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...), Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.ItemEventListKind, ArtifactKind: dnd.ItemEventListKind,

Some files were not shown because too many files have changed in this diff Show More