Compare commits
20 Commits
a16dcdfa52
...
v0.2.0
| Author | SHA1 | Date | |
|---|---|---|---|
| 39388e96d4 | |||
| 12ac25bd63 | |||
| 394278e1f2 | |||
| 5cd7f8e737 | |||
| bf3fadf9ae | |||
| 58815aaf33 | |||
| ce857966f1 | |||
| a3bd0c1867 | |||
| b05634ee86 | |||
| 4829f94157 | |||
| 67b315099d | |||
| b5c86de4d7 | |||
| 2eeca2ed5a | |||
| b5aaeb1c78 | |||
| 9171b66a41 | |||
| b4363b3b73 | |||
| 241e9d2a89 | |||
| 715fff7b72 | |||
| d627b91b4f | |||
| a67b3aa76d |
49
docs/adr/0010-workload-oriented-llm-profile-defaults.md
Normal file
49
docs/adr/0010-workload-oriented-llm-profile-defaults.md
Normal file
@@ -0,0 +1,49 @@
|
|||||||
|
# ADR-0010: Use workload-oriented LLM profile defaults
|
||||||
|
|
||||||
|
**Status:** Accepted
|
||||||
|
**Date:** 2026-08-03
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
LLM-backed D&D operations share an execution-policy choice, but repeating a
|
||||||
|
provider or model-named profile on every module binding ties pipeline structure
|
||||||
|
to a deployment decision. Different environments may require different model,
|
||||||
|
backend, timeout, or reasoning settings while retaining the same workload.
|
||||||
|
|
||||||
|
Notarius also needs a usable default for maintained D&D prompts without making
|
||||||
|
an operator profile mandatory. That default must remain owned by the D&D
|
||||||
|
family, while generic LLM infrastructure stays unaware of domain-specific
|
||||||
|
policy.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Pipelines may name one workload-oriented default profile, inherited only by
|
||||||
|
selected LLM-backed bindings and validators. Binding-level profile IDs remain
|
||||||
|
intentional exceptions, and the run-wide CLI profile override has highest
|
||||||
|
precedence.
|
||||||
|
|
||||||
|
The D&D family owns an embedded fallback profile named `dnd-extraction`.
|
||||||
|
Operators may provide a complete profile with the same ID through a PromptKit
|
||||||
|
filesystem source. PromptKit selects the higher-precedence matching definition;
|
||||||
|
Notarius does not merge profile documents. Production, development, and local
|
||||||
|
deployments can therefore use different execution policy behind one unchanged
|
||||||
|
pipeline ID.
|
||||||
|
|
||||||
|
## Alternatives considered
|
||||||
|
|
||||||
|
- Repeat a model-named profile on every binding. This makes routine deployment
|
||||||
|
policy changes noisy and obscures the shared workload intent.
|
||||||
|
- Require every deployment to install a profile file. This adds configuration
|
||||||
|
friction and leaves maintained D&D prompts without an application-owned
|
||||||
|
fallback.
|
||||||
|
- Put D&D profile policy in generic LLM infrastructure. This breaks domain
|
||||||
|
ownership and makes generic code depend on one workload.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
Pipeline configuration expresses workload intent rather than a specific
|
||||||
|
provider or model. Operators can replace the complete execution policy without
|
||||||
|
editing bindings, while binding-level and run-wide exceptions remain available.
|
||||||
|
Profile changes affect resolved pipeline and checkpoint identity, so they may
|
||||||
|
intentionally cause work to be recomputed. The D&D fallback becomes a
|
||||||
|
maintained application execution-policy asset.
|
||||||
@@ -39,7 +39,7 @@ pipeline ID and **--input** are required.
|
|||||||
| **--debug** | Retain a debug bundle for this run. |
|
| **--debug** | Retain a debug bundle for this run. |
|
||||||
| **--debug-dir path** | Override the debug-bundle root. Requires **--debug**. |
|
| **--debug-dir path** | Override the debug-bundle root. Requires **--debug**. |
|
||||||
| **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. |
|
| **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. |
|
||||||
| **--llm-profile id** | Override effective LLM-capable module bindings with one configured profile. |
|
| **--llm-profile id** | Highest-precedence configured profile for selected LLM-backed bindings and validators; it replaces binding and [pipeline](config.md#pipelines) defaults. |
|
||||||
| **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. |
|
| **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. |
|
||||||
| **--reasoning-effort value** | Replace the selected PromptKit profile's reasoning effort for every LLM-backed call in this run. The value must be non-empty and the flag may be specified only once. |
|
| **--reasoning-effort value** | Replace the selected PromptKit profile's reasoning effort for every LLM-backed call in this run. The value must be non-empty and the flag may be specified only once. |
|
||||||
| **--clear-reasoning-effort** | Clear reasoning effort inherited from the selected PromptKit profile for every LLM-backed call in this run. |
|
| **--clear-reasoning-effort** | Clear reasoning effort inherited from the selected PromptKit profile for every LLM-backed call in this run. |
|
||||||
|
|||||||
@@ -32,7 +32,8 @@ override the fields listed below.
|
|||||||
single-lane Seriatim-to-spell pipeline.
|
single-lane Seriatim-to-spell pipeline.
|
||||||
- [Complete D&D configuration](../examples/dnd-complete.config.yml) uses
|
- [Complete D&D configuration](../examples/dnd-complete.config.yml) uses
|
||||||
ordered steps, all implemented D&D lanes, generated references, state
|
ordered steps, all implemented D&D lanes, generated references, state
|
||||||
settings, and bounded LLM concurrency.
|
settings, bounded LLM concurrency, and the maintained
|
||||||
|
[operator profile](../examples/profiles/dnd-extraction.yml).
|
||||||
|
|
||||||
Use these complete files as starting points rather than combining the
|
Use these complete files as starting points rather than combining the
|
||||||
illustrative fragments in this reference.
|
illustrative fragments in this reference.
|
||||||
@@ -46,7 +47,7 @@ other than **version** is optional.
|
|||||||
| Field | Type | Default | Rules |
|
| Field | Type | Default | Rules |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| **version** | integer | none | Required; must be 4. |
|
| **version** | integer | none | Required; must be 4. |
|
||||||
| **promptkit** | object | none | Profile source configuration. |
|
| **promptkit** | object | none | Profile source and optional local-backend configuration. |
|
||||||
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
|
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
|
||||||
| **concurrency** | object | see below | Global LLM and extraction limits. |
|
| **concurrency** | object | see below | Global LLM and extraction limits. |
|
||||||
| **output** | object | see below | Published output settings. |
|
| **output** | object | see below | Published output settings. |
|
||||||
@@ -71,27 +72,76 @@ per-user root. An explicit empty output or debug directory is invalid.
|
|||||||
|
|
||||||
## PromptKit Profiles
|
## PromptKit Profiles
|
||||||
|
|
||||||
The optional **promptkit** object selects one source of profile definitions:
|
The optional **promptkit** object selects one source of profile definitions and
|
||||||
|
may register one conventional local OpenAI-compatible backend:
|
||||||
|
|
||||||
~~~yaml
|
~~~yaml
|
||||||
version: 4
|
version: 4
|
||||||
|
|
||||||
promptkit:
|
promptkit:
|
||||||
profile_dir: /path/to/profiles
|
profile_dir: ./profiles
|
||||||
# profile_file: /path/to/profiles.yml
|
# profile_file: ./profiles.yml
|
||||||
|
local_backend:
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
concurrency_limit: 2
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
| Field | Type | Rules |
|
| Field | Type | Rules |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| **profile_dir** | string | Non-empty directory containing profile files. |
|
| **profile_dir** | string | Non-empty directory containing profile files. |
|
||||||
| **profile_file** | string | Non-empty profile file. |
|
| **profile_file** | string | Non-empty profile file. |
|
||||||
|
| **local_backend** | object | Optional registration for the conventional PromptKit backend ID **local**. |
|
||||||
|
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
|
||||||
|
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
|
||||||
|
|
||||||
Set at most one of these fields. Profile IDs used by a binding must be available
|
Set at most one of **profile_dir** and **profile_file**. Relative values use
|
||||||
from the selected PromptKit profile source when the pipeline is resolved.
|
the process working directory, not the configuration file's directory. The
|
||||||
Keep credentials out of this file: configure a profile to read its credential
|
complete example's `./examples/profiles/dnd-extraction.yml` value is therefore
|
||||||
from an environment variable, then set that environment variable only in the
|
valid when Notarius is launched from the repository root; use an absolute path
|
||||||
run environment. PromptKit owns the profile-file format; see the
|
for services and containers.
|
||||||
[PromptKit upstream boundary](integrations/pkg-promptkit.md) for the pinned
|
|
||||||
package and canonical format reference.
|
An operator source is optional. For a requested ID, PromptKit checks the
|
||||||
|
configured operator source first, then Notarius's embedded fallback profiles,
|
||||||
|
then its own built-in catalog. A matching profile is complete: it replaces a
|
||||||
|
lower-precedence definition rather than merging with it. The maintained
|
||||||
|
[`dnd-extraction` operator profile](../examples/profiles/dnd-extraction.yml)
|
||||||
|
is a secret-free deployment artifact; production, development, and local
|
||||||
|
deployments can each provide a complete definition with that same workload ID.
|
||||||
|
Use workload-oriented IDs for new profiles instead of model names.
|
||||||
|
[Operations](operations.md#promptkit-profile-deployment) owns the deployment
|
||||||
|
workflow and credential-handling guidance.
|
||||||
|
|
||||||
|
When **local_backend** is present, its endpoint is trimmed and must use HTTP or
|
||||||
|
HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are
|
||||||
|
allowed. User information, queries, and fragments are rejected. A zero
|
||||||
|
**concurrency_limit** leaves the local backend unrestricted inside PromptKit;
|
||||||
|
a positive value limits simultaneous local generations. The application-wide
|
||||||
|
**concurrency.total_llm** limit still applies in both cases. Neither local
|
||||||
|
backend field has an environment override. Omitting **local_backend** registers
|
||||||
|
nothing and preserves existing built-in and endpoint-only profile behavior.
|
||||||
|
|
||||||
|
A file-backed PromptKit profile selects the registration by its case-sensitive
|
||||||
|
backend ID:
|
||||||
|
|
||||||
|
~~~yaml
|
||||||
|
id: local-summary
|
||||||
|
backend: local
|
||||||
|
model: example-model
|
||||||
|
~~~
|
||||||
|
|
||||||
|
Keep credentials out of the local-backend object. A PromptKit profile may name
|
||||||
|
its credential environment variable through `api_key_env`; set that variable
|
||||||
|
only in the run environment. PromptKit owns the
|
||||||
|
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md).
|
||||||
|
The [PromptKit upstream boundary](integrations/pkg-promptkit.md) identifies the
|
||||||
|
supported package API, and [Operations](operations.md#operational-limits)
|
||||||
|
describes the effective concurrency layers.
|
||||||
|
|
||||||
|
`notarius config validate --pipeline <id>` resolves the selected pipeline and
|
||||||
|
inspects every explicit effective profile without contacting a provider or
|
||||||
|
requiring credential values. It rejects absent, malformed, or incompatible
|
||||||
|
profiles before a run prepares modules. Credential availability is checked only
|
||||||
|
when a generation is prepared.
|
||||||
|
|
||||||
## Migrating Version 3 Configuration
|
## Migrating Version 3 Configuration
|
||||||
|
|
||||||
@@ -154,6 +204,7 @@ Each **pipelines** entry has a unique, non-empty ID and the following shape:
|
|||||||
~~~yaml
|
~~~yaml
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input: seriatim
|
||||||
chunk: generic
|
chunk: generic
|
||||||
output: json
|
output: json
|
||||||
@@ -166,6 +217,7 @@ pipelines:
|
|||||||
|
|
||||||
| Field | Type | Default | Rules |
|
| Field | Type | Default | Rules |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
|
| **llm_profile** | string | none | Optional non-empty default PromptKit profile ID for selected LLM-backed bindings and validators. An explicitly present blank value is invalid. |
|
||||||
| **input** | module binding | none | Required. |
|
| **input** | module binding | none | Required. |
|
||||||
| **chunk** | module binding | **generic** | Optional. |
|
| **chunk** | module binding | **generic** | Optional. |
|
||||||
| **output** | module binding | **json** | Optional. |
|
| **output** | module binding | **json** | Optional. |
|
||||||
@@ -179,6 +231,12 @@ needs a unique non-empty **id**, an **artifacts** map, and may have
|
|||||||
**references**. A lane ID must not appear more than once in a pipeline,
|
**references**. A lane ID must not appear more than once in a pipeline,
|
||||||
including across explicit steps.
|
including across explicit steps.
|
||||||
|
|
||||||
|
For each selected LLM-backed binding or validator, profile selection occurs
|
||||||
|
after module, validator, and `--only` lane selection. It uses the
|
||||||
|
run-level **--llm-profile** value first, then the binding's **llm_profile**,
|
||||||
|
then the pipeline's **llm_profile**, and finally the PromptKit default.
|
||||||
|
Deterministic bindings do not receive these defaults or run overrides.
|
||||||
|
|
||||||
A lane has these fields:
|
A lane has these fields:
|
||||||
|
|
||||||
| Field | Type | Default | Rules |
|
| Field | Type | Default | Rules |
|
||||||
@@ -206,7 +264,7 @@ Use an object for fields:
|
|||||||
~~~yaml
|
~~~yaml
|
||||||
extract:
|
extract:
|
||||||
module: dnd/spells
|
module: dnd/spells
|
||||||
llm_profile: gemini-2-flash
|
llm_profile: dnd-extraction
|
||||||
retries: 2
|
retries: 2
|
||||||
references:
|
references:
|
||||||
spell_catalog: ./dnd-spell-catalog.json
|
spell_catalog: ./dnd-spell-catalog.json
|
||||||
@@ -215,7 +273,7 @@ extract:
|
|||||||
| Binding field | Type | Default | Rules |
|
| Binding field | Type | Default | Rules |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| **module** | string | none | Required for an object binding. Must be a registered compatible key. |
|
| **module** | string | none | Required for an object binding. Must be a registered compatible key. |
|
||||||
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID. |
|
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID for an LLM-backed binding. It overrides the pipeline default unless the run supplies **--llm-profile**. |
|
||||||
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. |
|
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. |
|
||||||
| **options** | object | none | Must satisfy the selected module. |
|
| **options** | object | none | Must satisfy the selected module. |
|
||||||
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
|
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
|
||||||
@@ -225,7 +283,8 @@ Omitting **validators** uses the registered chain. **validators: []** selects
|
|||||||
an empty chain; a non-empty list replaces the chain in the listed order.
|
an empty chain; a non-empty list replaces the chain in the listed order.
|
||||||
Validator bindings accept only **module**, **llm_profile**, and **options**.
|
Validator bindings accept only **module**, **llm_profile**, and **options**.
|
||||||
They reject **references**, **retries**, and nested **validators**. Deterministic
|
They reject **references**, **retries**, and nested **validators**. Deterministic
|
||||||
validators reject an explicit **llm_profile**.
|
validators reject an explicit **llm_profile**. Deterministic module bindings
|
||||||
|
also reject an explicit **llm_profile**.
|
||||||
|
|
||||||
The **json** output module accepts optional **include_chunk_map** and
|
The **json** output module accepts optional **include_chunk_map** and
|
||||||
**evidence_context** settings:
|
**evidence_context** settings:
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
# PromptKit Integration
|
# PromptKit Integration
|
||||||
|
|
||||||
Notarius pins
|
Notarius pins
|
||||||
[`gitea.maximumdirect.net/eric/promptkit` v0.2.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0)
|
[`gitea.maximumdirect.net/eric/promptkit` v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0)
|
||||||
as its in-process prompt engine. The upstream
|
as its in-process prompt engine. The upstream
|
||||||
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0/docs/consumers/pkg-promptkit.md)
|
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/consumers/pkg-promptkit.md)
|
||||||
owns the public engine API, and the upstream
|
owns the public engine API, and the upstream
|
||||||
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0/docs/formats.md)
|
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
|
||||||
owns prompt, profile, and schema file contracts.
|
owns prompt, profile, and schema file contracts.
|
||||||
|
|
||||||
## Supported Boundary
|
## Supported Boundary
|
||||||
@@ -13,15 +13,35 @@ owns prompt, profile, and schema file contracts.
|
|||||||
Notarius relies on the root `promptkit` package to:
|
Notarius relies on the root `promptkit` package to:
|
||||||
|
|
||||||
- construct an `Engine` with filesystem-backed prompt, schema, and optional
|
- construct an `Engine` with filesystem-backed prompt, schema, and optional
|
||||||
profile sources;
|
operator and application-fallback profile sources;
|
||||||
- prepare and run a `RunRequest` with named inline artifacts, variables,
|
- prepare one frozen execution from a `RunRequest` with named inline artifacts,
|
||||||
a direct session ID, prompt identity, and profile selection;
|
variables, a direct session ID, prompt identity, and profile selection, then
|
||||||
|
record credential-redacted details and run that exact execution;
|
||||||
- return rendered debug material, validated structured output, selected
|
- return rendered debug material, validated structured output, selected
|
||||||
profile, backend, effective model metadata, and token usage;
|
profile, backend, effective model metadata, and token usage;
|
||||||
|
- register the optional conventional `local` backend through `BackendLocal`,
|
||||||
|
`LocalBackend`, and `WithBackend`;
|
||||||
- distinguish structured-output validation failure from execution failure; and
|
- distinguish structured-output validation failure from execution failure; and
|
||||||
- identify a missing explicit profile through `ErrProfileNotFound` and backend
|
- identify a missing explicit profile through `ErrProfileNotFound` and backend
|
||||||
admission exhaustion through `ErrCapacityExceeded`.
|
admission exhaustion through `ErrCapacityExceeded`.
|
||||||
|
|
||||||
|
The pinned
|
||||||
|
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/backends.go)
|
||||||
|
owns the registration and backend-capacity contract.
|
||||||
|
|
||||||
|
For one completion, the adapter calls `PrepareExecution`, takes a
|
||||||
|
caller-owned `Details` snapshot, and calls `RunPrepared` for that same opaque
|
||||||
|
prepared execution. It defers `Discard` for every unexecuted handle. Explicit
|
||||||
|
profile preflight uses `Engine.InspectProfile`; it does not prepare a synthetic
|
||||||
|
prompt. PromptKit's prepared handle, inspection result, and capacity-error
|
||||||
|
types stay inside the Notarius LLM adapter.
|
||||||
|
|
||||||
|
When a PromptKit profile and runtime override leave `temperature`, `max_tokens`,
|
||||||
|
or `top_p` unset, Notarius leaves that control unset as well. Compatible
|
||||||
|
providers therefore apply their own defaults; an operator that requires a
|
||||||
|
specific sampling value must select it explicitly in the profile or runtime
|
||||||
|
override.
|
||||||
|
|
||||||
Notarius does not use PromptKit's optional `ArtifactReader`. It materializes
|
Notarius does not use PromptKit's optional `ArtifactReader`. It materializes
|
||||||
source and reference content itself and supplies owned inline artifacts at the
|
source and reference content itself and supplies owned inline artifacts at the
|
||||||
adapter boundary. It also retains responsibility for pipeline retries,
|
adapter boundary. It also retains responsibility for pipeline retries,
|
||||||
@@ -38,15 +58,36 @@ Notarius records PromptKit's selected backend ID and effective reasoning
|
|||||||
setting as optional run-manifest provenance. Endpoint-only profiles have no
|
setting as optional run-manifest provenance. Endpoint-only profiles have no
|
||||||
backend ID. Debug prompt material also retains the selected backend ID and
|
backend ID. Debug prompt material also retains the selected backend ID and
|
||||||
PromptKit's stable lower-case `effective_model_params` JSON, which may include
|
PromptKit's stable lower-case `effective_model_params` JSON, which may include
|
||||||
`backend_id`. Notarius production configuration does not expose user-defined
|
`backend_id`. Notarius production configuration exposes one optional
|
||||||
PromptKit backend registration.
|
conventional `local` registration. It does not expose a general user-defined
|
||||||
|
PromptKit backend registry. Endpoint-only profiles remain supported unchanged.
|
||||||
|
|
||||||
Notarius retains its application-wide scheduled client around the PromptKit
|
Notarius retains its application-wide scheduled client around the PromptKit
|
||||||
adapter. PromptKit may apply a narrower limit for the selected backend;
|
adapter. PromptKit may apply a narrower limit for the selected backend;
|
||||||
endpoint-only profiles have no such backend limit. The adapter translates
|
endpoint-only profiles have no such backend limit. The adapter translates
|
||||||
PromptKit capacity rejection into the provider-neutral Notarius
|
PromptKit capacity rejection into the provider-neutral Notarius
|
||||||
`ErrLLMCapacityExceeded` contract and leaves retries to the calling pipeline
|
`ErrLLMCapacityExceeded` contract. It may include the normalized selected
|
||||||
stage.
|
backend ID in safe diagnostic context, without exposing PromptKit's capacity
|
||||||
|
error type, and leaves retries to the calling pipeline stage.
|
||||||
|
|
||||||
|
## Profile Sources And Compatibility
|
||||||
|
|
||||||
|
Notarius gives PromptKit the configured operator profile source, registered
|
||||||
|
application fallback profile assets, and optional backend registration through
|
||||||
|
the same construction path for inspection and execution. PromptKit owns the
|
||||||
|
resulting source precedence and strict profile parsing: a matching operator
|
||||||
|
profile is a complete replacement for a fallback or built-in profile, while an
|
||||||
|
invalid matching document fails instead of falling through. The operator
|
||||||
|
configuration and deployment workflow are defined in
|
||||||
|
[Configuration](../config.md#promptkit-profiles) and
|
||||||
|
[Operations](../operations.md#promptkit-profile-deployment).
|
||||||
|
|
||||||
|
Notarius supports this boundary against PromptKit v0.5.0. Its fallback source,
|
||||||
|
prepared-execution, inspection, and typed capacity APIs are used as public
|
||||||
|
upstream contracts; other PromptKit APIs or file-format behavior are not
|
||||||
|
implicitly supported. A dependency upgrade requires reviewing the adapter,
|
||||||
|
profile-source construction, and this compatibility statement against the
|
||||||
|
pinned upstream documentation.
|
||||||
|
|
||||||
## Notarius Ownership
|
## Notarius Ownership
|
||||||
|
|
||||||
@@ -54,8 +95,11 @@ stage.
|
|||||||
module assets, maps its transport-neutral completion contract, prepares and
|
module assets, maps its transport-neutral completion contract, prepares and
|
||||||
executes requests, validates output, records provenance, captures debug
|
executes requests, validates output, records provenance, captures debug
|
||||||
material, redacts errors, and preserves timeout ownership.
|
material, redacts errors, and preserves timeout ownership.
|
||||||
|
[D&D Module Internals](../internal/dnd.md) owns the embedded
|
||||||
|
`dnd-extraction` fallback profile and the maintained D&D prompt defaults.
|
||||||
[Configuration](../config.md#promptkit-profiles) defines how a Notarius
|
[Configuration](../config.md#promptkit-profiles) defines how a Notarius
|
||||||
configuration selects one PromptKit profile source.
|
configuration selects one PromptKit profile source and optionally registers
|
||||||
|
the conventional local backend.
|
||||||
|
|
||||||
PromptKit API or format changes outside this boundary are not implicitly
|
PromptKit API or format changes outside this boundary are not implicitly
|
||||||
supported. Updating the pinned version requires reviewing the adapter and
|
supported. Updating the pinned version requires reviewing the adapter and
|
||||||
|
|||||||
@@ -38,11 +38,14 @@ in [Configuration Internals](configuration.md).
|
|||||||
|
|
||||||
Configuration validation without a selected pipeline checks structural
|
Configuration validation without a selected pipeline checks structural
|
||||||
configuration only. Validation with a selected pipeline also builds the
|
configuration only. Validation with a selected pipeline also builds the
|
||||||
effective catalog, resolves the pipeline, and verifies explicitly selected
|
effective catalog, resolves the pipeline, and verifies every explicit effective
|
||||||
PromptKit profiles. Each explicit binding or validator profile is prepared
|
PromptKit profile. Selected LLM-backed input, chunk, lane, output, and validator
|
||||||
against the configured PromptKit source without performing generation, so an
|
profiles are inspected
|
||||||
unknown profile fails before pipeline preparation. Pipeline listing validates
|
against the configured PromptKit source and backend registrations without
|
||||||
configuration before returning normalized, sorted identifiers.
|
loading a prompt or performing generation, so an unknown or invalid profile
|
||||||
|
fails before pipeline preparation. Credential availability remains an
|
||||||
|
execution-time concern. Pipeline listing validates configuration before
|
||||||
|
returning normalized, sorted identifiers.
|
||||||
|
|
||||||
## Production Composition
|
## Production Composition
|
||||||
|
|
||||||
@@ -59,7 +62,8 @@ profile-provenance recorder, creates one scheduler from the effective global
|
|||||||
LLM limit, and wraps the client before it reaches modules. Registration and LLM
|
LLM limit, and wraps the client before it reaches modules. Registration and LLM
|
||||||
construction errors are returned before a pipeline is prepared. Configuration
|
construction errors are returned before a pipeline is prepared. Configuration
|
||||||
field definitions remain in [Configuration](../config.md#promptkit-profiles);
|
field definitions remain in [Configuration](../config.md#promptkit-profiles);
|
||||||
the adapter mechanics remain in [LLM Runtime](llm.md).
|
the D&D registrar's fallback profile assets and the adapter mechanics remain in
|
||||||
|
[LLM Runtime](llm.md).
|
||||||
|
|
||||||
The factory also accepts `LLMRuntimeOverrides`, whose reasoning pointer
|
The factory also accepts `LLMRuntimeOverrides`, whose reasoning pointer
|
||||||
preserves inherit, replace, and clear states across the composition boundary.
|
preserves inherit, replace, and clear states across the composition boundary.
|
||||||
@@ -81,7 +85,8 @@ handoff:
|
|||||||
2. create and validate a safe run identity, then allocate a debug bundle only
|
2. create and validate a safe run identity, then allocate a debug bundle only
|
||||||
when requested;
|
when requested;
|
||||||
3. build the effective catalog, resolve requested reference changes, resolve
|
3. build the effective catalog, resolve requested reference changes, resolve
|
||||||
the effective pipeline, and verify explicit PromptKit profiles;
|
the effective pipeline, and inspect its explicit effective PromptKit
|
||||||
|
profiles;
|
||||||
4. materialize external or generated references and record redacted invocation
|
4. materialize external or generated references and record redacted invocation
|
||||||
and resolution provenance when debug capture is enabled;
|
and resolution provenance when debug capture is enabled;
|
||||||
5. construct registries, the scheduled LLM client, prepared modules, and the
|
5. construct registries, the scheduled LLM client, prepared modules, and the
|
||||||
|
|||||||
@@ -39,14 +39,22 @@ This establishes the public precedence order without giving environment input a
|
|||||||
second file schema. Loading and application reject malformed YAML, unsupported
|
second file schema. Loading and application reject malformed YAML, unsupported
|
||||||
file versions, unknown fields, invalid values, and identifiers that are empty
|
file versions, unknown fields, invalid values, and identifiers that are empty
|
||||||
or collide after whitespace normalization. The file application also makes the
|
or collide after whitespace normalization. The file application also makes the
|
||||||
effective extraction-worker default follow the effective LLM limit.
|
effective extraction-worker default follow the effective LLM limit. A present
|
||||||
|
PromptKit local-backend object requires and trims its endpoint, defaults its
|
||||||
|
omitted concurrency limit to zero, and is copied so the parsed file model
|
||||||
|
cannot alias the populated **Config**. A pipeline `llm_profile` is
|
||||||
|
presence-aware: omission remains empty, while a present blank value is
|
||||||
|
rejected and a non-empty file value is trimmed before it reaches **Config**.
|
||||||
|
|
||||||
**Config.Validate** checks configuration-only invariants before resolution. It
|
**Config.Validate** checks configuration-only invariants before resolution. It
|
||||||
rejects incompatible profile sources, invalid state-surface values, unsupported
|
rejects incompatible profile sources, invalid state-surface values, unsupported
|
||||||
concurrency settings, malformed bindings and references, invalid retries, and
|
concurrency settings, malformed bindings and references, invalid retries, and
|
||||||
invalid pipeline, step, or lane structure. Its errors retain the closest known
|
invalid pipeline, step, or lane structure. PromptKit local-backend validation
|
||||||
pipeline, lane, and binding context. It deliberately does not require modules
|
accepts only an absolute HTTP or HTTPS endpoint with a host and no user
|
||||||
to be registered: that requires a catalog and belongs to resolution.
|
information, query, or fragment, and rejects a negative local concurrency
|
||||||
|
limit. Its errors retain the closest known pipeline, lane, and binding context.
|
||||||
|
It deliberately does not require modules to be registered: that requires a
|
||||||
|
catalog and belongs to resolution.
|
||||||
|
|
||||||
The exact user-selectable values and validation rules are defined in
|
The exact user-selectable values and validation rules are defined in
|
||||||
[Configuration](../config.md). Keep additions to the file model, an
|
[Configuration](../config.md). Keep additions to the file model, an
|
||||||
@@ -56,13 +64,15 @@ environment override, its validation, and that reference in the same change.
|
|||||||
|
|
||||||
**Config.Resolve** first recomputes derived concurrency defaults and validates
|
**Config.Resolve** first recomputes derived concurrency defaults and validates
|
||||||
the configuration. It normalizes the requested pipeline ID, copies the selected
|
the configuration. It normalizes the requested pipeline ID, copies the selected
|
||||||
profile, applies a non-empty command-level LLM profile override to the
|
profile, and passes the non-empty command-level LLM profile override, requested
|
||||||
LLM-capable stage bindings, and calls the framework resolver with the requested
|
lane selection, and reference changes to the framework resolver.
|
||||||
lane selection and reference changes.
|
|
||||||
|
|
||||||
The command-level override does not replace an explicitly selected validator
|
After module and validator selection, the resolver applies the effective
|
||||||
profile. Validator bindings remain part of the resolved validator chain and
|
profile policy to LLM-backed bindings only: command override, binding profile,
|
||||||
are resolved under their own declared configuration.
|
pipeline profile, then the prompt default. Deterministic bindings remain
|
||||||
|
profile-free, and no second inheritance decision occurs during execution. The
|
||||||
|
public field definitions and precedence are owned by
|
||||||
|
[Configuration](../config.md#pipelines).
|
||||||
|
|
||||||
The framework resolver supplies defaults, selects lanes, resolves validator
|
The framework resolver supplies defaults, selects lanes, resolves validator
|
||||||
chains, checks registered module and artifact compatibility, validates module
|
chains, checks registered module and artifact compatibility, validates module
|
||||||
@@ -71,7 +81,8 @@ options, and returns the fixed ordered pipeline shape. The resulting
|
|||||||
changes, a clone of the input configuration, and the resolved pipeline.
|
changes, a clone of the input configuration, and the resolved pipeline.
|
||||||
Callers may therefore retain or modify their input slices and maps without
|
Callers may therefore retain or modify their input slices and maps without
|
||||||
changing the resolved result, and later consumers cannot mutate the original
|
changing the resolved result, and later consumers cannot mutate the original
|
||||||
configuration through the effective value.
|
configuration through the effective value. This ownership includes the nested
|
||||||
|
PromptKit local-backend value.
|
||||||
|
|
||||||
Resolution failures stop before module construction and source parsing. They
|
Resolution failures stop before module construction and source parsing. They
|
||||||
include an error path for an unconfigured pipeline, missing module, missing
|
include an error path for an unconfigured pipeline, missing module, missing
|
||||||
@@ -83,7 +94,8 @@ runtime error class described in the [CLI reference](../cli.md#output-streams-an
|
|||||||
|
|
||||||
The framework assigns the resolved pipeline a deterministic SHA-256 digest
|
The framework assigns the resolved pipeline a deterministic SHA-256 digest
|
||||||
after defaults, lane selection, module bindings, reference bindings, validator
|
after defaults, lane selection, module bindings, reference bindings, validator
|
||||||
chains, and artifact schema identity have been resolved. The digest excludes
|
chains, effective LLM profiles, and artifact schema identity have been
|
||||||
|
resolved. The digest excludes
|
||||||
its own stored value. It identifies resolved composition rather than raw YAML
|
its own stored value. It identifies resolved composition rather than raw YAML
|
||||||
bytes, a debug payload, or all runtime state. The CLI records it as invocation
|
bytes, a debug payload, or all runtime state. The CLI records it as invocation
|
||||||
provenance before execution; cache and checkpoint identity have additional
|
provenance before execution; cache and checkpoint identity have additional
|
||||||
@@ -94,9 +106,12 @@ Configuration summaries must use **Redacted**, **RedactedSummaryPayload**, or
|
|||||||
Those methods copy every binding and nested option container, replace values
|
Those methods copy every binding and nested option container, replace values
|
||||||
whose key is credential-shaped with **[REDACTED]**, and omit materialized
|
whose key is credential-shaped with **[REDACTED]**, and omit materialized
|
||||||
reference content while retaining safe binding and reference provenance. The
|
reference content while retaining safe binding and reference provenance. The
|
||||||
payload must not alias the source configuration or resolved pipeline. This
|
payload must not alias the source configuration or resolved pipeline.
|
||||||
redaction is deliberately narrow: it protects configuration summaries and does
|
PromptKit's local endpoint and concurrency limit are preserved as non-secret
|
||||||
not authorize recording arbitrary environment values or provider requests.
|
configuration metadata in the independently owned summary; the object contains
|
||||||
|
no credential value. This redaction is deliberately narrow: it protects
|
||||||
|
configuration summaries and does not authorize recording arbitrary environment
|
||||||
|
values or provider requests.
|
||||||
|
|
||||||
## Invariants To Preserve
|
## Invariants To Preserve
|
||||||
|
|
||||||
|
|||||||
@@ -22,10 +22,15 @@ does not repeat their JSON shapes or schemas.
|
|||||||
## Family Composition
|
## Family Composition
|
||||||
|
|
||||||
The D&D registrar registers the family’s artifact codecs, extractors, typed
|
The D&D registrar registers the family’s artifact codecs, extractors, typed
|
||||||
append-order mergers, normalizers, validators, prompt assets, and default
|
append-order mergers, normalizers, validators, prompt assets, fallback LLM
|
||||||
validator chains. Each extractor and normalizer has a stable module spec,
|
profile asset, and default validator chains. Each extractor and normalizer has
|
||||||
strict option decoding, and a typed builder. Configuration remains the
|
a stable module spec, explicit execution class, strict option decoding, and a
|
||||||
canonical owner of the exact keys and validator order.
|
typed builder. Scene chunking, every extractor, and NPC normalization are
|
||||||
|
registered as `llm_backed`; the remaining current D&D mergers and normalizers
|
||||||
|
are `deterministic`. The metadata is available to catalog inspection and
|
||||||
|
resolved-pipeline debug data and determines which selected bindings inherit the
|
||||||
|
pipeline profile. Configuration remains the canonical owner of the exact keys,
|
||||||
|
profile precedence, and validator order.
|
||||||
|
|
||||||
Private structured-LLM response schemas are deliberately minimal. They reject
|
Private structured-LLM response schemas are deliberately minimal. They reject
|
||||||
invalid JSON structure, missing required fields, incompatible types, and
|
invalid JSON structure, missing required fields, incompatible types, and
|
||||||
@@ -41,6 +46,15 @@ reference, and transcript assets instead of copying their text into individual
|
|||||||
modules. A manifest’s declared sequence, including cache-control placement, is
|
modules. A manifest’s declared sequence, including cache-control placement, is
|
||||||
part of the prompt behavior.
|
part of the prompt behavior.
|
||||||
|
|
||||||
|
Every maintained D&D LLM prompt selects `dnd-extraction` as its default
|
||||||
|
profile. The D&D registrar embeds that fallback profile with the maintained
|
||||||
|
OpenRouter model, timeout, and service-tier policy. An operator may provide a
|
||||||
|
complete profile with the same ID through the configured PromptKit source; that
|
||||||
|
definition replaces the fallback rather than merging with it. The fallback
|
||||||
|
leaves reasoning and optional sampling controls unspecified. Deployment profile
|
||||||
|
selection and the maintained operator example are documented in
|
||||||
|
[Configuration](../config.md#promptkit-profiles).
|
||||||
|
|
||||||
All extraction prompts share this four-message rendered prefix: the system
|
All extraction prompts share this four-message rendered prefix: the system
|
||||||
message without cache control, the identity message without cache control, the
|
message without cache control, the identity message without cache control, the
|
||||||
campaign-reference message with ephemeral cache control, and the chunk
|
campaign-reference message with ephemeral cache control, and the chunk
|
||||||
|
|||||||
@@ -28,13 +28,15 @@ durable schemas. Those responsibilities remain with the module and its
|
|||||||
named material to a PromptKit inline artifact while preserving its origin URI,
|
named material to a PromptKit inline artifact while preserving its origin URI,
|
||||||
maps the trimmed request session to PromptKit's direct per-run session field,
|
maps the trimmed request session to PromptKit's direct per-run session field,
|
||||||
retains the same value as the `session_id` prompt variable for maintained
|
retains the same value as the `session_id` prompt variable for maintained
|
||||||
prompt compatibility, forwards profile selection, then prepares and runs the
|
prompt compatibility, and forwards profile selection. It then creates one
|
||||||
prompt. The direct field is authoritative for provider session behavior. A
|
frozen prepared execution, captures its caller-owned credential-redacted
|
||||||
session ID is a stable, non-secret correlation identifier and may be exposed
|
details for debug material, and executes that exact snapshot through
|
||||||
to providers and provider observability. The adapter returns PromptKit’s
|
PromptKit's prepared-execution boundary. The direct field
|
||||||
validated raw bytes rather than re-encoding the decoded target. An empty
|
is authoritative for provider session behavior. A session ID is a stable,
|
||||||
optional material is represented as one space so its named input is retained
|
non-secret correlation identifier and may be exposed to providers and provider
|
||||||
by PromptKit.
|
observability. The adapter returns PromptKit’s validated raw bytes rather than
|
||||||
|
re-encoding the decoded target. An empty optional material is represented as
|
||||||
|
one space so its named input is retained by PromptKit.
|
||||||
|
|
||||||
Client construction may also receive a run-wide reasoning-effort override from
|
Client construction may also receive a run-wide reasoning-effort override from
|
||||||
the CLI factory boundary. The adapter copies the caller-owned pointer and
|
the CLI factory boundary. The adapter copies the caller-owned pointer and
|
||||||
@@ -46,26 +48,50 @@ states. With neither flag, profile behavior remains unchanged. Because
|
|||||||
production constructs one shared client, the selected state applies uniformly
|
production constructs one shared client, the selected state applies uniformly
|
||||||
to module calls, retries, and LLM-backed validators for the whole run.
|
to module calls, retries, and LLM-backed validators for the whole run.
|
||||||
|
|
||||||
An empty request profile lets the prompt select its configured default. The CLI
|
An empty request profile lets the prompt select its configured default. Before a
|
||||||
prepares every explicitly selected binding profile before a run begins, so a
|
run begins, the CLI asks the adapter to inspect every explicit profile on the
|
||||||
missing explicit profile fails before stage execution. Calls record the profile
|
resolved selected LLM-backed bindings and validators, including inherited
|
||||||
actually selected by PromptKit. The recorder trims and deduplicates non-secret
|
pipeline profiles. Inspection resolves the profile and its selected backend and
|
||||||
profile identity, provider, model, selected backend ID, and effective reasoning
|
target without loading a prompt, reading credentials, admitting capacity, or
|
||||||
values for manifest use. Entries that differ in backend or reasoning remain
|
contacting a provider, so a missing or invalid explicit profile fails before
|
||||||
distinct and deterministically ordered. Endpoint-only profiles retain an empty
|
stage execution while a valid `api_key_env` may remain unset. Calls record the
|
||||||
backend ID, which the published JSON omits. Successful completion responses and
|
profile actually selected by PromptKit. The recorder trims and deduplicates
|
||||||
recorded profile manifests identify the adapter provider as `promptkit`.
|
non-secret profile identity, provider, model, selected backend ID, and
|
||||||
|
effective reasoning values for manifest use. Entries that differ in backend or
|
||||||
|
reasoning remain distinct and deterministically ordered. Endpoint-only profiles
|
||||||
|
retain an empty backend ID, which the published JSON omits. Successful
|
||||||
|
completion responses and recorded profile manifests identify the adapter
|
||||||
|
provider as `promptkit`.
|
||||||
|
|
||||||
|
The CLI's profile-inspection engine and the production adapter use the same
|
||||||
|
profile-source construction to apply the configured profile directory or file,
|
||||||
|
the optional registered fallback profile assets, and the optional conventional
|
||||||
|
`local` backend. Preflight therefore resolves the same profile sources and
|
||||||
|
backend membership as runtime without performing generation. Fallback assets
|
||||||
|
are mounted only when at least one source is registered. The production D&D
|
||||||
|
registrar contributes its `dnd-extraction` fallback, and the maintained D&D
|
||||||
|
prompts select that logical ID by default. PromptKit owns source precedence and
|
||||||
|
profile parsing: an operator-provided matching profile takes precedence over a
|
||||||
|
fallback profile without Notarius merging either document.
|
||||||
|
When the registration is absent, a profile selecting `backend: local` fails
|
||||||
|
inspection instead of falling back to a built-in or endpoint-only target.
|
||||||
|
|
||||||
Before execution, the adapter also contributes a non-secret checkpoint
|
Before execution, the adapter also contributes a non-secret checkpoint
|
||||||
fingerprint for the effective PromptKit profile source. It combines the
|
fingerprint for the effective PromptKit profile source. It combines the
|
||||||
identity of PromptKit's compiled-in profile catalog with a deterministic digest
|
identity of PromptKit's compiled-in profile catalog with a deterministic digest
|
||||||
of every YAML profile in the configured profile directory, or of the configured
|
of every YAML profile in the configured profile directory, or of the configured
|
||||||
profile file. The fingerprint contains neither profile content nor source
|
profile file, and a deterministic digest of the flattened fallback profile
|
||||||
paths. It covers both explicit binding profiles and prompt-selected defaults,
|
assets. The fingerprint contains neither profile content nor source paths. It
|
||||||
so changing a model or other profile setting cannot reuse checkpoints created
|
covers inherited pipeline profiles, explicit binding profiles, and
|
||||||
under the prior profile source. This cache identity is independent of durable
|
prompt-selected defaults, so changing a model or other profile setting cannot
|
||||||
|
reuse checkpoints created under the
|
||||||
|
prior profile source. This cache identity is independent of durable
|
||||||
profile provenance: run manifests continue to list only profiles actually
|
profile provenance: run manifests continue to list only profiles actually
|
||||||
observed during LLM calls.
|
observed during LLM calls. When the local backend is registered, a second
|
||||||
|
fingerprint hashes its trimmed endpoint behind a stable marker. Changing that
|
||||||
|
semantic execution target invalidates checkpoint reuse. The raw endpoint is not
|
||||||
|
stored in checkpoint identity, and the local concurrency limit is excluded
|
||||||
|
because it changes scheduling rather than execution semantics.
|
||||||
|
|
||||||
## Shared Provider-Call Limit
|
## Shared Provider-Call Limit
|
||||||
|
|
||||||
@@ -85,15 +111,19 @@ PromptKit applies a second, independent admission limit when the selected
|
|||||||
profile names a limited backend. It sits beneath the Notarius scheduled client,
|
profile names a limited backend. It sits beneath the Notarius scheduled client,
|
||||||
so it may narrow but cannot expand the application-wide limit. Built-in
|
so it may narrow but cannot expand the application-wide limit. Built-in
|
||||||
OpenRouter profiles select PromptKit's reserved backend and its upstream
|
OpenRouter profiles select PromptKit's reserved backend and its upstream
|
||||||
capacity policy. Endpoint-only profiles do not select a PromptKit backend and
|
capacity policy. A positive configured local-backend limit bounds active local
|
||||||
remain limited only by the Notarius scheduler.
|
generations inside PromptKit; zero leaves that backend unlimited there.
|
||||||
|
Endpoint-only profiles do not select a PromptKit backend and remain limited
|
||||||
|
only by the Notarius scheduler.
|
||||||
|
|
||||||
## Prompt And Schema Assets
|
## Prompt And Schema Assets
|
||||||
|
|
||||||
An `AssetRegistry` collects prompt and schema filesystems from production module
|
An `AssetRegistry` collects prompt, schema, and optional fallback-profile
|
||||||
families. It flattens registered roots into the PromptKit filesystems and
|
filesystems from production module families. It flattens registered roots into
|
||||||
rejects invalid roots, unreadable assets, duplicate paths, and missing prompt
|
the corresponding PromptKit filesystems and rejects invalid roots, unreadable
|
||||||
or schema files during preparation. The framework’s `promptfs` helper combines
|
assets, duplicate paths, and missing prompt or schema files during preparation.
|
||||||
|
Fallback assets receive a safe content digest for checkpoint identity; raw
|
||||||
|
paths and bytes are never included. The framework’s `promptfs` helper combines
|
||||||
module-owned prompt files with reusable domain fragments without making the
|
module-owned prompt files with reusable domain fragments without making the
|
||||||
framework depend on D&D content.
|
framework depend on D&D content.
|
||||||
|
|
||||||
@@ -142,10 +172,12 @@ than output-validation failures.
|
|||||||
When PromptKit rejects backend admission before generation, the adapter maps
|
When PromptKit rejects backend admission before generation, the adapter maps
|
||||||
`promptkit.ErrCapacityExceeded` to
|
`promptkit.ErrCapacityExceeded` to
|
||||||
`contracts.ErrLLMCapacityExceeded`, retaining prompt context and a redacted
|
`contracts.ErrLLMCapacityExceeded`, retaining prompt context and a redacted
|
||||||
upstream diagnostic without exposing the PromptKit sentinel as a framework
|
upstream diagnostic without exposing the PromptKit sentinel or capacity-error
|
||||||
contract. A canceled caller context takes precedence. The adapter does not
|
type as a framework contract. When supplied, the normalized selected backend
|
||||||
retry capacity failures; the pipeline's existing binding attempt policy sees
|
ID appears only in that safe application-owned diagnostic context. A canceled
|
||||||
the operational error and decides whether to rerun the complete operation.
|
caller context takes precedence. The adapter does not retry capacity failures;
|
||||||
|
the pipeline's existing binding attempt policy sees the operational error and
|
||||||
|
decides whether to rerun the complete operation.
|
||||||
|
|
||||||
Prompt-declared repair is executed within PromptKit’s structured-output flow.
|
Prompt-declared repair is executed within PromptKit’s structured-output flow.
|
||||||
The current production D&D prompt manifests set repair attempts to zero. That
|
The current production D&D prompt manifests set repair attempts to zero. That
|
||||||
|
|||||||
@@ -12,9 +12,15 @@ exceptions. See [D&D Module Internals](dnd.md) rather than adding them here.
|
|||||||
|
|
||||||
A module is a typed implementation registered for one pipeline stage. Its
|
A module is a typed implementation registered for one pipeline stage. Its
|
||||||
`ModuleSpec` is the public-to-the-framework declaration of its stable key,
|
`ModuleSpec` is the public-to-the-framework declaration of its stable key,
|
||||||
stage, required and provided capabilities, artifact kind, and accepted
|
stage, execution class, required and provided capabilities, artifact kind, and
|
||||||
reference slots. The framework uses that declaration to resolve a configured
|
accepted reference slots. The execution class states whether a module is
|
||||||
binding before it builds the implementation.
|
`deterministic` or `llm_backed`; registries retain it for catalog inspection and
|
||||||
|
resolved-pipeline debug data without constructing the module. The framework
|
||||||
|
uses the declaration to resolve a configured binding before it builds the
|
||||||
|
implementation. After selection, the resolver applies profile inheritance only
|
||||||
|
to bindings whose declared execution class is `llm_backed` and rejects a
|
||||||
|
binding-specific profile on a deterministic module. The user-facing precedence
|
||||||
|
contract belongs in [Configuration](../config.md#pipelines).
|
||||||
|
|
||||||
Implementations that accept options must provide both an option validator and
|
Implementations that accept options must provide both an option validator and
|
||||||
a builder. The validator is used while resolving configuration; the builder
|
a builder. The validator is used while resolving configuration; the builder
|
||||||
@@ -47,7 +53,7 @@ Production composition is intentionally split by family:
|
|||||||
- The Seriatim registrar provides the transcript input adapter. Its external
|
- The Seriatim registrar provides the transcript input adapter. Its external
|
||||||
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
|
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
|
||||||
- The D&D registrar provides its codecs, extractors, mergers, normalizers,
|
- The D&D registrar provides its codecs, extractors, mergers, normalizers,
|
||||||
validators, prompt assets, and default chains. Its behavioral conventions
|
validators, prompt assets, fallback profile asset, and default chains. Its behavioral conventions
|
||||||
are documented in [D&D Module Internals](dnd.md).
|
are documented in [D&D Module Internals](dnd.md).
|
||||||
|
|
||||||
The CLI owns the composition that invokes these registrars. A module package
|
The CLI owns the composition that invokes these registrars. A module package
|
||||||
@@ -59,10 +65,10 @@ packages depend on production extensions.
|
|||||||
1. Choose the pipeline stage and the typed artifact boundary. Put external
|
1. Choose the pipeline stage and the typed artifact boundary. Put external
|
||||||
input or durable artifact formats in the relevant integration contract,
|
input or durable artifact formats in the relevant integration contract,
|
||||||
not in this guide or in a private LLM response type.
|
not in this guide or in a private LLM response type.
|
||||||
2. Define a stable `ModuleSpec` with the exact capabilities and reference
|
2. Define a stable `ModuleSpec` with an explicit execution class, the exact
|
||||||
slots needed for the operation. Model a producer/consumer handoff as an
|
capabilities, and reference slots needed for the operation. Model a
|
||||||
artifact-compatible slot; configuration then chooses an external file or a
|
producer/consumer handoff as an artifact-compatible slot; configuration
|
||||||
generated binding.
|
then chooses an external file or a generated binding.
|
||||||
3. Implement strict option decoding, construction, and the typed stage
|
3. Implement strict option decoding, construction, and the typed stage
|
||||||
interface. Preserve caller ownership: do not retain mutable request data
|
interface. Preserve caller ownership: do not retain mutable request data
|
||||||
and return defensive copies where an implementation exposes stored data.
|
and return defensive copies where an implementation exposes stored data.
|
||||||
|
|||||||
@@ -34,6 +34,11 @@ requested lanes where that is supported, resolves validator chains, checks
|
|||||||
module capabilities and typed artifact compatibility, validates options, and
|
module capabilities and typed artifact compatibility, validates options, and
|
||||||
assigns a deterministic resolved-composition digest. The resolved pipeline
|
assigns a deterministic resolved-composition digest. The resolved pipeline
|
||||||
contains bindings and declared reference targets, not external reference bytes.
|
contains bindings and declared reference targets, not external reference bytes.
|
||||||
|
After selection, the resolver applies command, binding, and pipeline profile
|
||||||
|
precedence to LLM-backed bindings and validators only; prompt defaults remain
|
||||||
|
an empty resolved binding profile. Deterministic bindings remain profile-free.
|
||||||
|
These effective values are part of the digest, so execution and checkpoint
|
||||||
|
consumers do not repeat profile inheritance.
|
||||||
Configuration resolution supplies the selected profile and catalog; see
|
Configuration resolution supplies the selected profile and catalog; see
|
||||||
[Configuration Internals](configuration.md).
|
[Configuration Internals](configuration.md).
|
||||||
|
|
||||||
|
|||||||
@@ -42,6 +42,42 @@ evidence publication. Apply an appropriate umask and output-root access policy
|
|||||||
before enabling that option; the requested output modes alone may not be
|
before enabling that option; the requested output modes alone may not be
|
||||||
suitable for transcript-bearing bundles.
|
suitable for transcript-bearing bundles.
|
||||||
|
|
||||||
|
## PromptKit Profile Deployment
|
||||||
|
|
||||||
|
Profile deployment has four distinct layers:
|
||||||
|
|
||||||
|
| Layer | Owner | Operational role |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Prompts and schemas | Notarius module families | Embedded request and structured-output definitions. They are not deployment profile files. |
|
||||||
|
| Fallback profiles | Notarius module families | Embedded application defaults, including D&D's `dnd-extraction` profile. |
|
||||||
|
| Built-in profiles | PromptKit | Upstream catalog entries available when no higher-precedence source defines an ID. |
|
||||||
|
| Operator profiles | Deployment filesystem | Complete environment-specific definitions selected by `promptkit.profile_file` or `promptkit.profile_dir`. |
|
||||||
|
|
||||||
|
The maintained D&D pipeline uses the workload ID `dnd-extraction`. The
|
||||||
|
embedded fallback makes that ID usable without an operator file. Production,
|
||||||
|
development, and local deployments can each install a different complete
|
||||||
|
definition for the same ID, retaining the pipeline while choosing their own
|
||||||
|
model, backend, timeout, or reasoning policy. An operator definition wins over
|
||||||
|
the fallback; it is not merged with it. The configuration field and full
|
||||||
|
precedence rules are owned by [Configuration](config.md#promptkit-profiles).
|
||||||
|
|
||||||
|
Use a profile source owned by the service account, keep it readable only by
|
||||||
|
the intended operator, and supply provider credentials through the service
|
||||||
|
environment—not in the Notarius configuration or profile YAML. The maintained
|
||||||
|
[operator profile](../examples/profiles/dnd-extraction.yml) is secret-free and
|
||||||
|
can be copied as a format starting point. Validate a deployment without a
|
||||||
|
provider call or credentials:
|
||||||
|
|
||||||
|
~~~sh
|
||||||
|
notarius config validate --config /etc/notarius/config.yml --pipeline dnd-session
|
||||||
|
~~~
|
||||||
|
|
||||||
|
Profile paths are currently resolved from the process working directory, not
|
||||||
|
from the configuration file. The complete example's
|
||||||
|
`./examples/profiles/dnd-extraction.yml` path is valid for a repository-root
|
||||||
|
invocation only. Use absolute paths such as
|
||||||
|
`/etc/notarius/profiles/dnd-extraction.yml` for services and containers.
|
||||||
|
|
||||||
## Run Lifecycle
|
## Run Lifecycle
|
||||||
|
|
||||||
Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads
|
Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads
|
||||||
@@ -118,8 +154,10 @@ disabled. Without **--resume**, a recording-enabled run executes normally and
|
|||||||
does not load checkpoint state. Compatibility includes the resolved pipeline,
|
does not load checkpoint state. Compatibility includes the resolved pipeline,
|
||||||
input, selected lanes, runtime overrides, reference provenance, LLM-profile
|
input, selected lanes, runtime overrides, reference provenance, LLM-profile
|
||||||
provenance, the effective PromptKit profile-source fingerprint, and
|
provenance, the effective PromptKit profile-source fingerprint, and
|
||||||
prepared-component fingerprints. Changing profile content causes a cold miss
|
prepared-component fingerprints. When a local PromptKit backend is configured,
|
||||||
even when its profile ID is unchanged. A changed identity produces a cold miss;
|
compatibility also includes a non-secret fingerprint of its endpoint. Changing
|
||||||
|
profile content or the local endpoint causes a cold miss; changing only the
|
||||||
|
local concurrency limit does not. A changed identity produces a cold miss;
|
||||||
Notarius does not migrate, rewrite, or delete older checkpoint directories.
|
Notarius does not migrate, rewrite, or delete older checkpoint directories.
|
||||||
Reasoning-effort inheritance, replacement, and explicit clearing are distinct
|
Reasoning-effort inheritance, replacement, and explicit clearing are distinct
|
||||||
runtime identities, so checkpoints created under one state are not reused by
|
runtime identities, so checkpoints created under one state are not reused by
|
||||||
@@ -217,7 +255,7 @@ Provider execution settings and the generation timeout come from the selected
|
|||||||
PromptKit profile. The invocation-only **--reasoning-effort** and
|
PromptKit profile. The invocation-only **--reasoning-effort** and
|
||||||
**--clear-reasoning-effort** controls may replace or clear that profile setting
|
**--clear-reasoning-effort** controls may replace or clear that profile setting
|
||||||
for all LLM-backed calls in one run without changing the profile. PromptKit
|
for all LLM-backed calls in one run without changing the profile. PromptKit
|
||||||
v0.2.0 does not add a provider retry loop. Notarius binding retries rerun the
|
v0.5.0 does not add a provider retry loop. Notarius binding retries rerun the
|
||||||
complete module operation and validation chain as defined by
|
complete module operation and validation chain as defined by
|
||||||
[module bindings](config.md#module-bindings-and-validators).
|
[module bindings](config.md#module-bindings-and-validators).
|
||||||
|
|
||||||
@@ -234,13 +272,19 @@ and validators. PromptKit may impose a narrower admission limit for the
|
|||||||
selected backend. The effective active-generation bound is the intersection of
|
selected backend. The effective active-generation bound is the intersection of
|
||||||
both limits and can therefore be lower than **total_llm**. Built-in OpenRouter
|
both limits and can therefore be lower than **total_llm**. Built-in OpenRouter
|
||||||
profiles use PromptKit's upstream backend limit; endpoint-only profiles have no
|
profiles use PromptKit's upstream backend limit; endpoint-only profiles have no
|
||||||
PromptKit backend limit and remain bounded by Notarius.
|
PromptKit backend limit and remain bounded by Notarius. For the configured
|
||||||
|
local backend, a zero **concurrency_limit** leaves only the Notarius scheduler
|
||||||
|
as a call limit. A positive value makes the effective active local-generation
|
||||||
|
bound the smaller of **total_llm** and that local limit.
|
||||||
|
|
||||||
When a PromptKit backend has admitted all active and queued work, a new call
|
For a positive local limit, PromptKit owns its default waiting capacity and
|
||||||
fails as capacity exhaustion before generation. The adapter does not retry it.
|
admission behavior. When a PromptKit backend has admitted all active and queued
|
||||||
The calling stage's configured retry policy applies normally, and the run fails
|
work, a new call fails as capacity exhaustion before generation. The adapter
|
||||||
if those attempts are exhausted. Caller cancellation remains authoritative.
|
maps that failure to Notarius's existing provider-neutral capacity error and
|
||||||
Configuration contracts are documented under
|
does not retry it. The calling stage's configured retry policy applies
|
||||||
|
normally, and the run fails if those attempts are exhausted. Caller
|
||||||
|
cancellation remains authoritative. Configuration contracts are documented
|
||||||
|
under [PromptKit profiles](config.md#promptkit-profiles) and
|
||||||
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
|
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
|
||||||
limits and actual provider-call limits are independent. Notarius writes local
|
limits and actual provider-call limits are independent. Notarius writes local
|
||||||
filesystem state only; remote storage, archival, and retention automation are
|
filesystem state only; remote storage, archival, and retention automation are
|
||||||
|
|||||||
@@ -53,6 +53,71 @@ not as committed release dates.
|
|||||||
spell, combat, interaction, and scene-description lanes after real-world use.
|
spell, combat, interaction, and scene-description lanes after real-world use.
|
||||||
Add more complex chunking only in response to demonstrated failures.
|
Add more complex chunking only in response to demonstrated failures.
|
||||||
|
|
||||||
|
## Cross-Cutting LLM Runtime
|
||||||
|
|
||||||
|
### Deterministic Prompt Session Identity
|
||||||
|
|
||||||
|
- Replace the source-document-ID default for prompt sessions with one
|
||||||
|
predictable, procedurally generated session ID for the complete
|
||||||
|
source-processing workload.
|
||||||
|
- Preserve an explicit non-empty `--session-id` as the highest-precedence
|
||||||
|
override. Otherwise, derive the default only from the effective input module
|
||||||
|
identity and the exact raw input bytes.
|
||||||
|
- Use a versioned, bounded representation such as
|
||||||
|
`notarius:v1:<sha256(input-module + NUL + raw-input)>`. The exact encoding
|
||||||
|
must fit PromptKit's session length contract and must not embed source
|
||||||
|
content.
|
||||||
|
- Keep the derived session stable across runs, pipelines, selected lanes,
|
||||||
|
ordered steps, retries, resume, recomputation, LLM profiles, reasoning
|
||||||
|
overrides, and output, debug, or cache settings.
|
||||||
|
- Do not include file-backed references, generated references, reference
|
||||||
|
contents, or the composition of a reference bundle in session derivation.
|
||||||
|
References may change between prompt calls within one pipeline without
|
||||||
|
changing routing affinity.
|
||||||
|
- Resolve the authoritative session before checkpoint construction and use the
|
||||||
|
same value for checkpoint runtime identity, every prompt-facing module,
|
||||||
|
PromptKit's direct session field, the compatibility `session_id` prompt
|
||||||
|
variable, run-manifest metadata, and debug metadata.
|
||||||
|
- Keep routing identity separate from cache and checkpoint content identity.
|
||||||
|
Exact prompt prefixes, reference contents, model settings, and other
|
||||||
|
generation-affecting inputs must continue to participate in their existing
|
||||||
|
hashes and checkpoint fingerprints even though they do not change the
|
||||||
|
session.
|
||||||
|
- Treat the generated value as a provider-visible, stable pseudonymous
|
||||||
|
correlation identifier. Do not introduce an installation-specific HMAC or
|
||||||
|
secret unless a concrete multi-tenant or privacy requirement justifies
|
||||||
|
sacrificing deterministic identity across installations.
|
||||||
|
|
||||||
|
### Raise The Default Application-Wide LLM Limit
|
||||||
|
|
||||||
|
- Raise the default `concurrency.total_llm` value from 1 to 16 so ordinary
|
||||||
|
single-backend runs can use PromptKit's expected OpenRouter capacity and
|
||||||
|
lower-capacity local backends without an unnecessarily narrower Notarius
|
||||||
|
limit.
|
||||||
|
- Keep the Notarius application-wide scheduler mandatory and require
|
||||||
|
`total_llm` to remain a positive integer. Do not make the default unlimited:
|
||||||
|
endpoint-only profiles, an unrestricted local backend, injected clients, and
|
||||||
|
aggregate work across several backends may have no narrower PromptKit limit.
|
||||||
|
- Continue defaulting `concurrency.stage_workers.extract` to the effective
|
||||||
|
`total_llm`, making its default 16 as part of the same change. Preserve an
|
||||||
|
explicit lower extract-worker setting when an operator wants less queued or
|
||||||
|
concurrent extraction work.
|
||||||
|
- Define effective provider concurrency as the intersection of the Notarius
|
||||||
|
application-wide limit, the selected PromptKit backend limit when present,
|
||||||
|
and the work made available by stage execution. A Notarius limit of 16 does
|
||||||
|
not narrow a backend already limited to 16, while a local backend limited to
|
||||||
|
4 remains bounded at 4.
|
||||||
|
- Treat the default as an application-wide safety ceiling across profiles,
|
||||||
|
backends, modules, retries, and validators. A run that intentionally needs
|
||||||
|
the combined capacity of several backends may configure a higher
|
||||||
|
`total_llm` and an appropriate extract-worker count explicitly.
|
||||||
|
- Retain the existing configuration and environment override surfaces. Update
|
||||||
|
canonical configuration, operations, and internal documentation together
|
||||||
|
when the default changes.
|
||||||
|
- Reconsider decoupling the extract-worker default from `total_llm` only after
|
||||||
|
mixed-backend workloads demonstrate a need for a high global emergency
|
||||||
|
ceiling with a lower default work-production rate.
|
||||||
|
|
||||||
## Shared Normalization And Quality Work
|
## Shared Normalization And Quality Work
|
||||||
|
|
||||||
### Generic LLM-Assisted Deduplication
|
### Generic LLM-Assisted Deduplication
|
||||||
|
|||||||
586
docs/roadmap/implementation.md
Normal file
586
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,586 @@
|
|||||||
|
# PromptKit v0.5 Implementation Plan
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Implement the target state in
|
||||||
|
[PromptKit v0.5 Integration And LLM Profile Policy](promptkit.md). Each numbered
|
||||||
|
stage is intended to be one implementation prompt for a GPT-5.6-Terra coding
|
||||||
|
agent. Complete stages in order and leave the repository buildable, tested, and
|
||||||
|
internally coherent after every stage.
|
||||||
|
|
||||||
|
Follow [Architecture](../policy/architecture.md),
|
||||||
|
[Testing Policy](../policy/testing.md), and
|
||||||
|
[Documentation Policy](../policy/documentation.md) throughout. Preserve
|
||||||
|
unrelated user changes. Use `apply_patch` for source and documentation edits,
|
||||||
|
run `gofmt` on changed Go files, and add only tests that protect the behaviors
|
||||||
|
and risks assigned to that stage.
|
||||||
|
|
||||||
|
Do not implement the separate deterministic session-ID or default-concurrency
|
||||||
|
roadmap items as part of this plan. Do not perform paid or credentialed LLM
|
||||||
|
calls.
|
||||||
|
|
||||||
|
## Background Summary
|
||||||
|
|
||||||
|
Notarius currently pins PromptKit v0.3.0, calls `Prepare` and then `Run` for one
|
||||||
|
completion, validates profiles through a synthetic prompt, has no application
|
||||||
|
fallback profile source, and accepts LLM profiles only at individual bindings
|
||||||
|
or through the run-wide CLI override. PromptKit v0.5.0 is source-compatible
|
||||||
|
with the current tree; a temporary v0.5.0 module override has already passed
|
||||||
|
`go test ./...`.
|
||||||
|
|
||||||
|
The implementation must nevertheless treat the upstream optional-parameter
|
||||||
|
change as intentional: unset `temperature`, `max_tokens`, and `top_p` remain
|
||||||
|
unset and are omitted from compatible provider requests. Do not restore the old
|
||||||
|
implicit `top_p: 1` default.
|
||||||
|
|
||||||
|
## Stage 1: Upgrade The PromptKit Dependency
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Establish a clean PromptKit v0.5.0 baseline before adopting its new APIs.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Update `go.mod` and `go.sum` from PromptKit v0.3.0 to v0.5.0 and run
|
||||||
|
`go mod tidy`.
|
||||||
|
- Change the PromptKit built-in profile-catalog marker in
|
||||||
|
`internal/framework/llm/promptkit_profile_fingerprint.go` to identify
|
||||||
|
v0.5.0. This deliberately invalidates LLM checkpoints tied to the prior
|
||||||
|
catalog identity.
|
||||||
|
- Review PromptKit-facing compile errors or test failures against the v0.4.0
|
||||||
|
and v0.5.0 release guides. Do not adopt prepared execution, inspection, or
|
||||||
|
fallback profiles in this stage.
|
||||||
|
- Replace the existing test assertion for one exact built-in fingerprint hash
|
||||||
|
with durable assertions that the fingerprint is deterministic, non-empty,
|
||||||
|
non-secret, and changes when a semantic profile source changes. Do not add a
|
||||||
|
new version-constant or exact-hash change detector.
|
||||||
|
- Update `docs/integrations/pkg-promptkit.md` to pin and link v0.5.0 and state
|
||||||
|
the implemented dependency-level behavior: unset optional sampling controls
|
||||||
|
are provider defaults. Do not document later stages as implemented.
|
||||||
|
- Update any other canonical text that explicitly claims the dependency is
|
||||||
|
v0.3.0, but defer descriptions of unimplemented v0.5 APIs.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- `go test ./internal/framework/llm ./internal/cli`
|
||||||
|
- `go test ./...`
|
||||||
|
- `go vet ./...`
|
||||||
|
- `go build ./cmd/notarius`
|
||||||
|
- `rg -n 'promptkit v0\.3\.0|promptkit@v0\.3\.0|PromptKit v0\.3\.0' .`
|
||||||
|
- `git diff --check`
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- The repository directly pins v0.5.0 and all default offline checks pass.
|
||||||
|
- The profile-source fingerprint identifies the new upstream catalog without a
|
||||||
|
brittle literal-hash test.
|
||||||
|
- Current documentation no longer identifies v0.3.0 as the supported version.
|
||||||
|
|
||||||
|
## Stage 2: Execute One Frozen Prepared Snapshot
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Make Notarius debug details and generation use one exact PromptKit preparation.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Refactor `PromptKitClient.CompleteStructured` to call
|
||||||
|
`PrepareExecution`, immediately defer `Discard`, obtain a caller-owned
|
||||||
|
`Details` value, and execute with `RunPrepared`.
|
||||||
|
- Preserve the existing Notarius request mapping, cancellation precedence,
|
||||||
|
validation classification, raw structured bytes, response decoding,
|
||||||
|
profile recording, usage reporting, and credential redaction.
|
||||||
|
- Ensure every preparation, execution, validation, empty-result, and decode
|
||||||
|
error retains useful Notarius prompt context without exposing prepared handle
|
||||||
|
state or secrets.
|
||||||
|
- Use `errors.As` to obtain `*promptkit.CapacityError` on admission rejection.
|
||||||
|
Preserve `contracts.ErrLLMCapacityExceeded` as the stable classification and
|
||||||
|
add a nonblank backend ID only to safe application-owned diagnostic context.
|
||||||
|
Do not expose `promptkit.CapacityError` outside the LLM adapter.
|
||||||
|
- Update `docs/internal/llm.md` and the implemented-mechanics portion of
|
||||||
|
`docs/integrations/pkg-promptkit.md` to describe the single frozen execution
|
||||||
|
snapshot and structured capacity adaptation.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Adapt existing PromptKit client tests to the prepared-execution path.
|
||||||
|
- Retain or add one behavioral test proving that the debug prompt details match
|
||||||
|
the request actually passed to generation when a backing prompt source could
|
||||||
|
otherwise change between independent preparations. Test the resulting
|
||||||
|
snapshot consistency, not a private helper call count.
|
||||||
|
- Retain capacity tests proving `errors.Is` reaches
|
||||||
|
`contracts.ErrLLMCapacityExceeded`, the selected backend can appear in safe
|
||||||
|
diagnostic context, and provider calls are not made after rejected
|
||||||
|
admission.
|
||||||
|
- Run `go test ./internal/framework/llm` and
|
||||||
|
`go test -race ./internal/framework/llm`.
|
||||||
|
- Run `go test ./...` and `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- `CompleteStructured` no longer calls independent `Prepare` and `Run`
|
||||||
|
operations for one request.
|
||||||
|
- Debug prompt material and generation result originate from the same frozen
|
||||||
|
PromptKit snapshot.
|
||||||
|
- Capacity remains a provider-neutral Notarius error classification.
|
||||||
|
|
||||||
|
## Stage 3: Replace Synthetic Profile Validation With Inspection
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Validate profiles through PromptKit's exact profile-inspection boundary and
|
||||||
|
centralize engine profile-source construction.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Introduce a small provider-adapter-owned profile inspection or validation
|
||||||
|
function in `internal/framework/llm`. Its public internal signature must use
|
||||||
|
Notarius-owned configuration and result/error types rather than returning
|
||||||
|
PromptKit types to the CLI.
|
||||||
|
- Share the code that applies `profile_dir`, `profile_file`, and registered
|
||||||
|
backend options between the production PromptKit engine and the inspection
|
||||||
|
engine. Preserve the mutual-exclusion and local-backend rules.
|
||||||
|
- Change CLI explicit-profile preflight to use `Engine.InspectProfile` through
|
||||||
|
that LLM boundary.
|
||||||
|
- Remove `profileCheckPromptID`, `profileCheckPromptFS`, the `testing/fstest`
|
||||||
|
production dependency, and the synthetic `Prepare` request.
|
||||||
|
- Preserve distinct, useful errors for an absent profile, invalid profile,
|
||||||
|
unknown backend registration, cancellation, and invalid profile source.
|
||||||
|
- Do not require `api_key_env` to be populated during configuration validation.
|
||||||
|
Inspection may report credential requirements internally, but actual
|
||||||
|
preparation remains responsible for credential availability before a model
|
||||||
|
call.
|
||||||
|
- Update current-behavior sections in `docs/internal/cli.md` and
|
||||||
|
`docs/internal/llm.md`. Keep field definitions in `docs/config.md`.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Replace synthetic-prompt tests with profile inspection tests covering:
|
||||||
|
configured local backend success; missing local backend failure; absent
|
||||||
|
profile; malformed profile; and an otherwise valid profile whose credential
|
||||||
|
environment variable is intentionally unset.
|
||||||
|
- Prove validation performs no provider HTTP call and remains offline.
|
||||||
|
- Run `go test ./internal/framework/llm ./internal/cli` and `go test ./...`.
|
||||||
|
- Run `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- No production synthetic profile-check prompt remains.
|
||||||
|
- Profile validation uses the same ordinary profile source and backend
|
||||||
|
registrations as execution.
|
||||||
|
- Configuration validation succeeds for structurally valid profiles without
|
||||||
|
reading credential values.
|
||||||
|
|
||||||
|
## Stage 4: Add Application Fallback Profile Asset Plumbing
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Allow module families to register application-owned fallback profile YAML
|
||||||
|
without placing domain policy in generic LLM code.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Extend `internal/framework/llm.AssetRegistry` with a separate fallback
|
||||||
|
profile source collection, registration method, flattened filesystem, and
|
||||||
|
safe content digest.
|
||||||
|
- Reuse the existing asset-source path validation and flattening behavior where
|
||||||
|
appropriate. Reject invalid roots, unreadable assets, and duplicate flattened
|
||||||
|
paths. Do not parse PromptKit profile YAML in Notarius.
|
||||||
|
- Add `promptkit.WithFallbackProfileFS` to production engine options only when
|
||||||
|
at least one fallback profile source is registered.
|
||||||
|
- Supply the identical assembled fallback source to the profile-inspection
|
||||||
|
engine. Adjust CLI composition so pipeline-aware profile validation can use
|
||||||
|
the production LLM asset registry without exposing PromptKit types.
|
||||||
|
- Extend profile-source checkpoint identity to include the exact fallback
|
||||||
|
profile asset digest in addition to the PromptKit catalog marker and operator
|
||||||
|
source. Keep the resulting fingerprint hash-only and path/content/credential
|
||||||
|
free.
|
||||||
|
- Keep operator source precedence owned by PromptKit. Do not implement profile
|
||||||
|
merging or duplicate PromptKit source resolution in Notarius.
|
||||||
|
- Update `docs/internal/llm.md` only for the new implemented generic asset and
|
||||||
|
fingerprint mechanics. No domain fallback exists until Stage 5.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Add focused AssetRegistry tests for successful flattening, invalid roots,
|
||||||
|
duplicate paths, and hash changes when fallback bytes change.
|
||||||
|
- Add adapter-level tests showing that the fallback filesystem reaches both
|
||||||
|
execution construction and inspection construction.
|
||||||
|
- Extend checkpoint tests to prove fallback content changes profile-source
|
||||||
|
identity without exposing raw YAML or paths. Use relational comparisons, not
|
||||||
|
a fixed hash literal.
|
||||||
|
- Run `go test ./internal/framework/llm ./internal/cli` and `go test ./...`.
|
||||||
|
- Run `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- Generic plumbing can carry application fallback profiles while remaining
|
||||||
|
unaware of D&D IDs or model settings.
|
||||||
|
- Inspection, execution, and checkpoint identity use the same fallback asset
|
||||||
|
source.
|
||||||
|
|
||||||
|
## Stage 5: Adopt The D&D `dnd-extraction` Fallback
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Give the D&D module family one stable embedded workload profile that operators
|
||||||
|
can replace.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Add a D&D-owned embedded PromptKit profile asset with ID `dnd-extraction`
|
||||||
|
under `internal/modules/dnd`. Use the exact baseline defined in
|
||||||
|
`promptkit.md`: OpenRouter, `openai/gpt-5.6-luna`, no explicit reasoning
|
||||||
|
effort, a 240-second timeout, flex service tier, and no selected temperature,
|
||||||
|
token limit, or `top_p`. The omitted reasoning value intentionally allows
|
||||||
|
OpenAI's backend to apply its `medium` default.
|
||||||
|
- Register the profile filesystem from the D&D registrar through the generic
|
||||||
|
fallback profile asset boundary. Keep D&D policy out of
|
||||||
|
`internal/framework/llm` and the CLI composition root.
|
||||||
|
- Change every maintained D&D LLM prompt definition—including scene chunking,
|
||||||
|
all D&D extractors, and NPC normalization—from the model-named default to
|
||||||
|
`default_profile: dnd-extraction`.
|
||||||
|
- Add an integration-level profile-resolution test proving that:
|
||||||
|
- the fallback resolves when no operator source defines the ID;
|
||||||
|
- a valid operator profile with the same ID wins completely; and
|
||||||
|
- an invalid matching operator profile fails rather than falling through.
|
||||||
|
- Test through Notarius's assembled production assets and PromptKit boundary;
|
||||||
|
do not duplicate every upstream source-precedence case.
|
||||||
|
- Update the implemented profile ownership and prompt-default behavior in
|
||||||
|
`docs/internal/dnd.md`, `docs/internal/llm.md`, and
|
||||||
|
`docs/integrations/pkg-promptkit.md`. Defer the complete operator walkthrough
|
||||||
|
and examples to Stage 10.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Run focused D&D prompt preparation tests and the production composition
|
||||||
|
tests.
|
||||||
|
- Run `go test ./internal/modules/dnd/... ./internal/framework/llm
|
||||||
|
./internal/cli`.
|
||||||
|
- Run `go test ./...`.
|
||||||
|
- Verify `rg -n 'default_profile: gemini-2-flash' internal/modules/dnd`
|
||||||
|
returns no matches.
|
||||||
|
- Run `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- All maintained D&D prompts use the application-owned logical profile ID.
|
||||||
|
- The fallback works without an operator profile and remains authoritatively
|
||||||
|
overridable by a matching valid operator definition.
|
||||||
|
|
||||||
|
## Stage 6: Introduce Module Execution-Class Metadata
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Make each production module's ability to use an LLM statically discoverable
|
||||||
|
without yet changing profile inheritance.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Add `ExecutionClass contracts.ExecutionClass` to `pipeline.ModuleSpec` and
|
||||||
|
preserve it through normalization, cloning, catalogs, registries, JSON/debug
|
||||||
|
views, and lookup helpers.
|
||||||
|
- In this transitional stage only, allow an omitted execution class to
|
||||||
|
normalize to deterministic so existing test-only fixtures can be migrated in
|
||||||
|
Stage 7 without breaking the repository midway.
|
||||||
|
- Explicitly classify every production module:
|
||||||
|
- D&D scene chunking, every D&D extractor, and D&D NPC normalization as
|
||||||
|
`llm_backed`;
|
||||||
|
- all other current production input, chunk, merge, normalize, and output
|
||||||
|
modules as `deterministic`.
|
||||||
|
- Update production module specification tests and production catalog tests to
|
||||||
|
assert the semantic class alongside stage, artifact kind, and capabilities.
|
||||||
|
- Add catalog lookup support needed by later resolution to retrieve a selected
|
||||||
|
module's execution class by stage and key without constructing it.
|
||||||
|
- Do not implement pipeline-level profile inheritance or reject deterministic
|
||||||
|
profiles yet.
|
||||||
|
- Update `docs/internal/modules.md` and `docs/internal/dnd.md` to identify
|
||||||
|
execution class as registered module metadata, while noting only implemented
|
||||||
|
uses.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Run module registration/spec tests across generic, Seriatim, and D&D
|
||||||
|
families.
|
||||||
|
- Run `go test ./internal/framework/pipeline ./internal/modules/...`.
|
||||||
|
- Run `go test ./...` and `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- Every production module has an explicit correct execution class.
|
||||||
|
- Catalog consumers can retrieve that class without a concrete module
|
||||||
|
instance.
|
||||||
|
- Test-only omitted classes remain the only temporary compatibility behavior.
|
||||||
|
|
||||||
|
## Stage 7: Enforce Execution Metadata And Remove Runtime Probing
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Finish the execution-class contract so missing metadata cannot cause future
|
||||||
|
profile drift.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Update every framework, CLI, and integration test module specification to
|
||||||
|
declare an explicit execution class appropriate to the fake behavior.
|
||||||
|
- Change module-spec validation so an empty or unsupported execution class is a
|
||||||
|
registration error. Remove the transitional deterministic default from
|
||||||
|
Stage 6.
|
||||||
|
- Replace the chunk runner's special `ChunkExecutionClassProvider` probe with
|
||||||
|
specification-derived behavior. Remove the now-redundant provider interface,
|
||||||
|
implementation methods, and tests when they have no remaining consumer.
|
||||||
|
- Ensure chunk producer provenance remains unchanged: it records a non-empty
|
||||||
|
effective binding profile for an LLM-backed chunker, while a deterministic
|
||||||
|
chunker records no profile. A profile selected only through the prompt
|
||||||
|
default remains represented by PromptKit's actual-profile manifest rather
|
||||||
|
than being invented as an explicit chunk binding.
|
||||||
|
- Review helper constructors and fixtures for opportunities to set execution
|
||||||
|
class once without obscuring the class under test. Do not introduce an
|
||||||
|
elaborate test-spec framework.
|
||||||
|
- Update internal documentation if the removal changes any described runtime
|
||||||
|
mechanics.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Add or retain focused registration tests for missing and invalid execution
|
||||||
|
classes.
|
||||||
|
- Retain chunk-plan provenance tests for LLM-backed and deterministic
|
||||||
|
chunkers.
|
||||||
|
- Run `go test ./internal/framework/pipeline ./internal/modules/...`.
|
||||||
|
- Run `go test ./...`, `go vet ./...`, and `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- No registered module specification relies on an implicit execution class.
|
||||||
|
- Pipeline metadata, not a concrete runtime type assertion, owns module
|
||||||
|
execution classification.
|
||||||
|
|
||||||
|
## Stage 8: Resolve Programmatic Pipeline Profile Defaults
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Implement profile inheritance and precedence inside the pipeline resolver
|
||||||
|
before exposing the field through YAML configuration.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Add an optional trimmed `LLMProfile` field to
|
||||||
|
`pipeline.PipelineProfile`. Add a non-empty runtime override field to
|
||||||
|
`pipeline.ResolveOptions` so all precedence decisions occur in the resolver
|
||||||
|
rather than through pre-resolution mutation.
|
||||||
|
- After module selection, `--only` filtering, default validator-chain
|
||||||
|
selection, and validator compatibility resolution, apply effective profiles
|
||||||
|
to every selected input, chunk, extract, merge, normalize, output, and
|
||||||
|
validator binding according to the precedence in `promptkit.md`.
|
||||||
|
- Apply profiles only when the selected module or validator execution class is
|
||||||
|
`llm_backed`.
|
||||||
|
- Reject a binding-specific `llm_profile` on any deterministic module or
|
||||||
|
validator. Do not reject or inspect an unused pipeline default when no
|
||||||
|
selected LLM-backed binding consumes it.
|
||||||
|
- Leave an LLM-backed binding empty when no CLI, binding, or pipeline profile is
|
||||||
|
selected so PromptKit can use the prompt's `default_profile`.
|
||||||
|
- Store the effective values on resolved bindings before digest construction.
|
||||||
|
Do not add a second inheritance decision to execution.
|
||||||
|
- Ensure semantically equivalent repeated binding profiles and one inherited
|
||||||
|
default produce the same resolved pipeline digest. Ensure any changed
|
||||||
|
effective profile changes the digest.
|
||||||
|
- Do not modify file configuration or CLI parsing in this stage.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Add pipeline package tests for the complete precedence matrix:
|
||||||
|
runtime override; binding-specific exception; pipeline default; prompt
|
||||||
|
fallback; and deterministic bindings.
|
||||||
|
- Cover default and explicitly configured validator chains, all relevant stage
|
||||||
|
categories, `--only` lane selection, unused defaults, deterministic-profile
|
||||||
|
rejection, and semantic digest equivalence.
|
||||||
|
- Prefer table-driven package-level tests over assertions on private traversal
|
||||||
|
helpers.
|
||||||
|
- Run `go test ./internal/framework/pipeline` and `go test ./...`.
|
||||||
|
- Run `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- Programmatic pipelines resolve one canonical effective profile policy.
|
||||||
|
- Only LLM-backed resolved bindings can contain a profile.
|
||||||
|
- Runtime override, binding, pipeline, and prompt precedence is unambiguous and
|
||||||
|
digest-stable.
|
||||||
|
|
||||||
|
## Stage 9: Expose Pipeline Defaults Through Configuration And CLI
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Make the profile-default workflow available to operators while preserving
|
||||||
|
validation and override behavior.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Add optional `pipelines.<id>.llm_profile` support to the version 4 file
|
||||||
|
configuration model. Use presence-aware decoding so an explicitly set blank
|
||||||
|
value is rejected, while omission remains valid.
|
||||||
|
- Preserve the field through file application, configuration cloning,
|
||||||
|
effective configuration, and programmatic profile copies without aliasing or
|
||||||
|
trimming drift.
|
||||||
|
- Remove `applyLLMProfileOverride`. Pass the CLI override through the resolver's
|
||||||
|
runtime-override input so deterministic bindings are never populated.
|
||||||
|
- Update effective profile-ID collection to cover every selected LLM-backed
|
||||||
|
module stage and LLM-backed validator, including future LLM-backed input and
|
||||||
|
output modules. Do not inspect deterministic or unselected profiles.
|
||||||
|
- Ensure `run`, `config validate --pipeline`, resume/checkpoint identity, and
|
||||||
|
relevant dry preflight paths all use the same resolved effective profiles.
|
||||||
|
- Preserve `--llm-profile` as the highest-precedence non-empty run-wide
|
||||||
|
override and preserve binding-specific profiles as exceptions when no CLI
|
||||||
|
override is present.
|
||||||
|
- Do not increment the configuration version.
|
||||||
|
- Update current configuration and CLI contracts in `docs/config.md` and
|
||||||
|
`docs/cli.md` in the same stage. Link to operations for the deployment
|
||||||
|
workflow rather than duplicating it prematurely.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Add file-config tests for omission, trimming, explicit blank rejection,
|
||||||
|
unknown-key behavior, cloning, and round-trip application.
|
||||||
|
- Add effective-config and CLI contract tests for precedence, LLM-only
|
||||||
|
application, inherited-profile inspection failure before factory execution,
|
||||||
|
`--only`, and digest changes.
|
||||||
|
- Retain offline operation and do not require credentials for
|
||||||
|
`config validate --pipeline`.
|
||||||
|
- Run `go test ./internal/core/config ./internal/framework/pipeline
|
||||||
|
./internal/cli`.
|
||||||
|
- Run `go test ./...`, `go vet ./...`, and `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- Operators can select `dnd-extraction` once per pipeline.
|
||||||
|
- Configuration and CLI paths share the resolver's precedence policy.
|
||||||
|
- Unknown effective profiles fail preflight, while deterministic and unused
|
||||||
|
profiles do not cause spurious inspection.
|
||||||
|
|
||||||
|
## Stage 10: Complete Operator Documentation, Examples, And Decision Record
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Make the implemented workflow understandable, copyable, and maintainable
|
||||||
|
without duplicating canonical facts.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Create an ADR using the next sequential number for the durable decision to
|
||||||
|
use workload-oriented pipeline defaults with operator-overridable application
|
||||||
|
fallback profiles. Record context, decision, alternatives, and consequences;
|
||||||
|
do not turn the ADR into a field reference or implementation log.
|
||||||
|
- Complete `docs/config.md` as the canonical owner of profile-source fields,
|
||||||
|
`pipelines.<id>.llm_profile`, validation, and precedence.
|
||||||
|
- Complete `docs/operations.md` with an operator workflow that distinguishes
|
||||||
|
Notarius embedded prompts, Notarius fallback profiles, PromptKit built-ins,
|
||||||
|
and deployment filesystem profiles. Include production/development/local use
|
||||||
|
of the same `dnd-extraction` ID, credential handling, absolute-path guidance,
|
||||||
|
and the fact that current relative profile paths use the process working
|
||||||
|
directory rather than the configuration file's directory.
|
||||||
|
- Complete `docs/integrations/pkg-promptkit.md` with the v0.5.0 boundary,
|
||||||
|
prepared execution, inspection, fallback and ordinary source precedence,
|
||||||
|
optional provider controls, capacity adaptation, and compatibility policy.
|
||||||
|
- Update `docs/internal/configuration.md`, `docs/internal/pipeline.md`,
|
||||||
|
`docs/internal/cli.md`, `docs/internal/llm.md`, `docs/internal/modules.md`, and
|
||||||
|
`docs/internal/dnd.md` only for their owned implementation details. Link to
|
||||||
|
canonical configuration, operations, and upstream format contracts rather
|
||||||
|
than restating them.
|
||||||
|
- Keep exactly the existing two D&D configuration examples. Add
|
||||||
|
`llm_profile: dnd-extraction` to the minimal and complete pipelines and remove
|
||||||
|
the now-redundant model-named binding override from the complete example.
|
||||||
|
- Add one secret-free maintained operator profile at
|
||||||
|
`examples/profiles/dnd-extraction.yml`. It should be a complete valid profile
|
||||||
|
for the same logical ID and may mirror the embedded baseline; its purpose is
|
||||||
|
to demonstrate file ownership and format, not claim automatic environment
|
||||||
|
detection. Link it from the configuration and operations documentation.
|
||||||
|
- If the complete example selects the external profile file, use a path that
|
||||||
|
is valid for the documented repository-root invocation and explicitly note
|
||||||
|
the working-directory rule. Keep the minimal example dependent only on the
|
||||||
|
embedded fallback.
|
||||||
|
- Add or extend maintained-example validation so both configuration examples
|
||||||
|
and the profile YAML are checked without generation or credentials.
|
||||||
|
- Remove the now-implemented `Pipeline-Level LLM Profile Defaults` section from
|
||||||
|
`docs/roadmap/future.md`. Preserve the unrelated deterministic session and
|
||||||
|
concurrency items.
|
||||||
|
- Do not delete `promptkit.md` or this implementation plan during the feature
|
||||||
|
implementation; retire them only after post-implementation review.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Run maintained example/configuration tests and relevant CLI help/parser
|
||||||
|
tests.
|
||||||
|
- Run `go test ./...`.
|
||||||
|
- Run `rg -n 'gemini-2-flash' examples docs` and review every remaining match
|
||||||
|
for intentional model-policy or historical context.
|
||||||
|
- Run `rg -n 'v0\.3\.0|profileCheckPrompt|applyLLMProfileOverride' .` and resolve
|
||||||
|
stale production or current-documentation matches.
|
||||||
|
- Verify all new links and `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- Every current fact has one canonical documentation owner.
|
||||||
|
- Operators can distinguish and deploy all profile layers without reading Go
|
||||||
|
source.
|
||||||
|
- Both maintained configurations and the maintained external profile are valid,
|
||||||
|
secret-free, and tested offline.
|
||||||
|
- Implemented profile work no longer remains in `future.md`.
|
||||||
|
|
||||||
|
## Stage 11: Final Verification And Quality Review
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Verify the complete migration as one integrated change and correct only defects
|
||||||
|
or omissions found during that review.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Review the final diff against every acceptance criterion in `promptkit.md`.
|
||||||
|
- Confirm provider-specific PromptKit types remain inside the LLM integration
|
||||||
|
boundary and D&D policy remains inside the D&D module family.
|
||||||
|
- Confirm execution and inspection receive identical ordinary, fallback, and
|
||||||
|
backend configuration.
|
||||||
|
- Confirm no paths, profile YAML, endpoints, credentials, or prepared handle
|
||||||
|
state leak into fingerprints or ordinary diagnostics.
|
||||||
|
- Confirm all production module specs have explicit correct execution classes
|
||||||
|
and every resolved deterministic binding is profile-free.
|
||||||
|
- Confirm prompt default, pipeline default, binding override, and CLI override
|
||||||
|
behavior through representative assembled configurations.
|
||||||
|
- Review tests for redundancy and remove obsolete synthetic-prompt,
|
||||||
|
runtime-probe, exact-hash, or duplicated upstream-behavior tests superseded by
|
||||||
|
stronger contract tests.
|
||||||
|
- Perform an optional manual D&D quality comparison if credentials and an
|
||||||
|
evaluation transcript are deliberately supplied. Record no private input or
|
||||||
|
credential material, and do not make this comparison a completion gate.
|
||||||
|
|
||||||
|
### Validation Commands
|
||||||
|
|
||||||
|
```sh
|
||||||
|
gofmt -w <changed-go-files>
|
||||||
|
go test ./...
|
||||||
|
go test -race ./internal/framework/llm ./internal/core/config ./internal/framework/pipeline ./internal/cli
|
||||||
|
go vet ./...
|
||||||
|
go build ./cmd/notarius
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
Also run focused stale-contract searches:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
rg -n 'gitea.maximumdirect.net/eric/promptkit v0\.3\.0|PromptKit v0\.3\.0' .
|
||||||
|
rg -n 'default_profile: gemini-2-flash|profileCheckPrompt|applyLLMProfileOverride' internal docs examples
|
||||||
|
```
|
||||||
|
|
||||||
|
Review any matches rather than deleting intentional historical references
|
||||||
|
blindly.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- All automated checks pass offline and without real credentials.
|
||||||
|
- The implemented behavior matches `promptkit.md` with no known architecture,
|
||||||
|
provenance, checkpoint, profile-precedence, or documentation gap.
|
||||||
|
- Any optional live evaluation is clearly separate from correctness testing.
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
None. The roadmap decisions are sufficient to implement every stage without an
|
||||||
|
additional product or architecture choice.
|
||||||
318
docs/roadmap/promptkit.md
Normal file
318
docs/roadmap/promptkit.md
Normal file
@@ -0,0 +1,318 @@
|
|||||||
|
# PromptKit v0.5 Integration And LLM Profile Policy
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This roadmap defines the target state for upgrading Notarius from PromptKit
|
||||||
|
v0.3.0 to v0.5.0 and adopting the upstream runtime and profile facilities that
|
||||||
|
directly improve Notarius. It also defines the application policy for stable,
|
||||||
|
domain-oriented LLM profile names, operator overrides, pipeline inheritance,
|
||||||
|
profile validation, provider defaults, checkpoint identity, and documentation.
|
||||||
|
|
||||||
|
The ordered work needed to reach this state belongs in
|
||||||
|
[the implementation plan](implementation.md). Current behavior remains defined
|
||||||
|
by the canonical documentation outside `docs/roadmap/` until the corresponding
|
||||||
|
work is implemented.
|
||||||
|
|
||||||
|
## Background
|
||||||
|
|
||||||
|
Notarius currently pins PromptKit v0.3.0. Its adapter prepares a request once
|
||||||
|
for debug material and then independently runs the original request, causing
|
||||||
|
PromptKit to prepare the same logical call a second time. The CLI validates an
|
||||||
|
explicit profile by preparing a synthetic prompt. PromptKit profile selection
|
||||||
|
can be repeated on individual module bindings or replaced for one invocation
|
||||||
|
with `--llm-profile`, but a configured pipeline cannot yet declare one inherited
|
||||||
|
profile policy.
|
||||||
|
|
||||||
|
PromptKit v0.4.0 and v0.5.0 add the upstream boundaries needed to improve these
|
||||||
|
areas:
|
||||||
|
|
||||||
|
- [v0.4.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.4.0.md)
|
||||||
|
adds opaque prepared executions, exact profile and prompt inspection, and a
|
||||||
|
typed backend-capacity error;
|
||||||
|
- [v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.5.0.md)
|
||||||
|
adds application fallback profile filesystems and stops sending unset
|
||||||
|
optional sampling controls as framework-selected provider values; and
|
||||||
|
- the [v0.5.0 format contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
|
||||||
|
defines the resulting profile-source and execution-setting precedence.
|
||||||
|
|
||||||
|
A source-compatibility test of the current Notarius repository against
|
||||||
|
PromptKit v0.5.0 completed successfully. The work is therefore primarily an
|
||||||
|
intentional runtime and configuration migration rather than a repair for a
|
||||||
|
breaking Go API change.
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
- Pin and document PromptKit v0.5.0 as Notarius's supported upstream contract.
|
||||||
|
- Execute the exact prepared request snapshot whose safe details are recorded
|
||||||
|
in Notarius debug material.
|
||||||
|
- Validate configured PromptKit profiles through the upstream inspection API
|
||||||
|
without synthetic prompts, provider calls, or credential-value access.
|
||||||
|
- Give Notarius an application-owned, operator-overridable
|
||||||
|
`dnd-extraction` profile fallback.
|
||||||
|
- Let a pipeline choose one default LLM profile without repeating that ID on
|
||||||
|
every LLM-backed binding.
|
||||||
|
- Apply profile inheritance and run-wide overrides only where the resolved
|
||||||
|
module or validator can use an LLM.
|
||||||
|
- Preserve accurate checkpoint invalidation, effective profile provenance,
|
||||||
|
redaction, cancellation, concurrency, and provider-neutral module contracts.
|
||||||
|
- Provide operators with one clear deployment pattern for production,
|
||||||
|
development, and local profile definitions.
|
||||||
|
|
||||||
|
## Target End State
|
||||||
|
|
||||||
|
### PromptKit Runtime Boundary
|
||||||
|
|
||||||
|
Notarius depends on PromptKit v0.5.0 and uses its public APIs rather than
|
||||||
|
reimplementing source or execution resolution.
|
||||||
|
|
||||||
|
For each structured completion, the adapter:
|
||||||
|
|
||||||
|
1. builds one PromptKit run request from the provider-neutral Notarius request;
|
||||||
|
2. calls `PrepareExecution` once;
|
||||||
|
3. immediately arranges an idempotent `Discard` for every unexecuted handle;
|
||||||
|
4. obtains credential-redacted `Details` for debug and response metadata; and
|
||||||
|
5. calls `RunPrepared` so generation uses that exact frozen snapshot.
|
||||||
|
|
||||||
|
The debug prompt and successful result therefore describe the same selected
|
||||||
|
profile, rendered messages, input bytes, session, output contract, and effective
|
||||||
|
settings even when a filesystem-backed source changes concurrently. PromptKit
|
||||||
|
handle types remain private to `internal/framework/llm`.
|
||||||
|
|
||||||
|
PromptKit admission failures continue to match Notarius's provider-neutral
|
||||||
|
`ErrLLMCapacityExceeded` contract. When PromptKit supplies a `CapacityError`,
|
||||||
|
the adapter obtains the normalized backend ID through `errors.As` and may add it
|
||||||
|
to safe application-owned diagnostics without parsing upstream error wording.
|
||||||
|
The backend ID does not become a provider-specific module contract.
|
||||||
|
|
||||||
|
### Optional Provider Controls
|
||||||
|
|
||||||
|
Notarius accepts PromptKit v0.5.0's new behavior for `temperature`,
|
||||||
|
`max_tokens`, and `top_p`: an unset setting is omitted from compatible provider
|
||||||
|
requests and the provider chooses its own default. Notarius does not restore
|
||||||
|
PromptKit's former implicit `top_p: 1` value globally.
|
||||||
|
|
||||||
|
An operator who requires a particular value specifies it in the selected
|
||||||
|
PromptKit profile. The application fallback described below intentionally
|
||||||
|
leaves these controls unset. A human-reviewed D&D extraction comparison should
|
||||||
|
be performed after the upgrade, but paid or nondeterministic model output is
|
||||||
|
not part of the default automated test suite.
|
||||||
|
|
||||||
|
### Profile Inspection
|
||||||
|
|
||||||
|
Pipeline-aware configuration validation uses `Engine.InspectProfile` for every
|
||||||
|
effective explicit profile ID. It verifies that the profile exists, parses and
|
||||||
|
validates, resolves its backend and target, and is compatible with the engine's
|
||||||
|
registered backends. It does not create a synthetic prompt, load prompt inputs,
|
||||||
|
contact a provider, or require credential values to exist in the validation
|
||||||
|
process environment.
|
||||||
|
|
||||||
|
Credential availability is execution-time state. PromptKit preparation still
|
||||||
|
enforces the selected profile's credential contract before generation. This
|
||||||
|
keeps `notarius config validate` useful in build and deployment validation
|
||||||
|
environments where secrets are deliberately absent.
|
||||||
|
|
||||||
|
PromptKit construction for inspection and execution uses one shared internal
|
||||||
|
profile-source and backend-option path. The CLI does not expose PromptKit public
|
||||||
|
types across the Notarius LLM boundary merely to perform inspection.
|
||||||
|
|
||||||
|
`InspectPrompt` is not adopted merely because it exists. It remains available
|
||||||
|
for a later, separately defined module-to-prompt interface preflight if a
|
||||||
|
concrete validation requirement justifies that additional contract.
|
||||||
|
|
||||||
|
### Application And Operator Profile Sources
|
||||||
|
|
||||||
|
Notarius embeds one ordinary PromptKit YAML profile with the stable ID
|
||||||
|
`dnd-extraction`. It is an application fallback registered through
|
||||||
|
`WithFallbackProfileFS`, is owned by the D&D module family, and initially
|
||||||
|
preserves the current effective D&D baseline:
|
||||||
|
|
||||||
|
- backend: PromptKit's built-in `openrouter` backend;
|
||||||
|
- model: `openai/gpt-5.6-luna`;
|
||||||
|
- reasoning effort: unset, allowing OpenAI's backend to apply its default of
|
||||||
|
`medium`;
|
||||||
|
- generation timeout: 240 seconds;
|
||||||
|
- service tier: `flex`; and
|
||||||
|
- no application-selected `temperature`, `max_tokens`, or `top_p`.
|
||||||
|
|
||||||
|
All maintained D&D LLM prompt definitions use `dnd-extraction` as their
|
||||||
|
`default_profile`. The ID communicates workload intent rather than a provider,
|
||||||
|
model, or environment. Changing the embedded fallback is an intentional
|
||||||
|
Notarius execution-policy change and participates in checkpoint identity.
|
||||||
|
|
||||||
|
Effective profile definitions resolve in PromptKit's order:
|
||||||
|
|
||||||
|
1. programmatic in-memory profiles used by tests or explicit consumers;
|
||||||
|
2. the operator source configured by `promptkit.profile_file` or
|
||||||
|
`promptkit.profile_dir`;
|
||||||
|
3. the Notarius application fallback source; and
|
||||||
|
4. PromptKit's embedded built-in catalog.
|
||||||
|
|
||||||
|
Only an absent ID falls through to the next source. A matching profile is a
|
||||||
|
complete definition: fields are not merged with a lower-precedence definition,
|
||||||
|
and a malformed matching operator profile fails rather than silently selecting
|
||||||
|
the application fallback.
|
||||||
|
|
||||||
|
Production, development, and local deployments should normally provide
|
||||||
|
different complete definitions for the same `dnd-extraction` ID. An operator
|
||||||
|
source is optional because the application fallback keeps the maintained D&D
|
||||||
|
workflow usable, but a deployment that needs an intentional model or backend
|
||||||
|
policy should configure its own definition.
|
||||||
|
|
||||||
|
### Domain Ownership And Asset Assembly
|
||||||
|
|
||||||
|
The D&D fallback profile remains under `internal/modules/dnd` and is registered
|
||||||
|
by the D&D registrar, consistent with ADR-0004. Generic LLM plumbing knows how
|
||||||
|
to collect and flatten application fallback profile filesystems but contains no
|
||||||
|
D&D model or policy knowledge.
|
||||||
|
|
||||||
|
The shared asset registry detects invalid roots, unreadable sources, and
|
||||||
|
duplicate flattened paths. PromptKit remains responsible for strict profile
|
||||||
|
YAML parsing, duplicate profile-ID detection, source precedence, and effective
|
||||||
|
target resolution. The same assembled fallback source is supplied to runtime
|
||||||
|
execution and CLI profile inspection.
|
||||||
|
|
||||||
|
### Explicit Module Execution Metadata
|
||||||
|
|
||||||
|
Every registered input, chunk, extract, merge, normalize, and output module
|
||||||
|
declares one required execution class: `deterministic` or `llm_backed`.
|
||||||
|
Validator registrations continue to declare the same distinction through their
|
||||||
|
validator specifications.
|
||||||
|
|
||||||
|
The registered specification is authoritative for configuration resolution.
|
||||||
|
Current production classifications are:
|
||||||
|
|
||||||
|
- the D&D scene chunker, all D&D extractors, and the D&D NPC normalizer are
|
||||||
|
LLM-backed;
|
||||||
|
- the Seriatim input adapter, generic chunker, all current mergers, all other
|
||||||
|
current normalizers, and the JSON output encoder are deterministic; and
|
||||||
|
- current validators retain their declared classifications.
|
||||||
|
|
||||||
|
Missing or unsupported execution metadata is a registration error. Explicitly
|
||||||
|
assigning `llm_profile` to a deterministic module or validator is a pipeline
|
||||||
|
resolution error. The framework does not infer execution class by inspecting
|
||||||
|
domain package names or concrete implementation types at runtime.
|
||||||
|
|
||||||
|
The module specification replaces the chunk runner's special runtime
|
||||||
|
execution-class probe. Effective resolved bindings already express the result:
|
||||||
|
only LLM-backed bindings may retain a non-empty profile.
|
||||||
|
|
||||||
|
### Pipeline-Level Profile Default
|
||||||
|
|
||||||
|
Configuration version 4 gains one optional non-empty pipeline field:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
pipelines:
|
||||||
|
dnd-session:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
|
```
|
||||||
|
|
||||||
|
No configuration-version increment is required because the field is additive
|
||||||
|
and existing files remain valid. An explicitly present blank value is invalid.
|
||||||
|
|
||||||
|
For every selected LLM-backed module and validator, the effective profile uses
|
||||||
|
this precedence:
|
||||||
|
|
||||||
|
1. non-empty run-wide `--llm-profile` override;
|
||||||
|
2. binding-specific `llm_profile`;
|
||||||
|
3. pipeline-level `llm_profile`; and
|
||||||
|
4. the prompt definition's `default_profile`, represented by an empty effective
|
||||||
|
Notarius binding profile.
|
||||||
|
|
||||||
|
The run-wide override and inherited pipeline default never attach to a
|
||||||
|
deterministic binding. Binding-specific exceptions remain available when one
|
||||||
|
operation needs a different cost, latency, quality, backend, or reasoning
|
||||||
|
policy.
|
||||||
|
|
||||||
|
Inheritance is resolved after module and validator selection, including
|
||||||
|
`--only` lane filtering, but before effective-pipeline validation, digest
|
||||||
|
construction, explicit-profile inspection, checkpoint construction,
|
||||||
|
preparation, execution, or provenance capture. Only profiles used by selected
|
||||||
|
LLM-backed bindings are inspected. An unused pipeline default in a pipeline
|
||||||
|
with no selected LLM-backed work does not require an otherwise unused profile
|
||||||
|
to exist.
|
||||||
|
|
||||||
|
The resolved pipeline contains effective binding profiles rather than a second
|
||||||
|
runtime inheritance mechanism. Two pipelines that differ only by spelling the
|
||||||
|
same effective policy once as a pipeline default and once on every LLM-backed
|
||||||
|
binding have the same semantic resolved digest. Changing an effective profile
|
||||||
|
changes the digest and applicable checkpoint identity.
|
||||||
|
|
||||||
|
### Provenance And Checkpoints
|
||||||
|
|
||||||
|
The PromptKit profile-source checkpoint fingerprint covers:
|
||||||
|
|
||||||
|
- the PromptKit v0.5.0 built-in profile catalog identity;
|
||||||
|
- exact application fallback profile asset content; and
|
||||||
|
- exact configured operator profile YAML content, when present.
|
||||||
|
|
||||||
|
The existing local-backend target fingerprint remains separate and continues
|
||||||
|
to exclude scheduling-only concurrency limits. Fingerprints contain hashes and
|
||||||
|
stable markers, not profile contents, filesystem paths, endpoints, credentials,
|
||||||
|
or other secrets.
|
||||||
|
|
||||||
|
Changing the PromptKit version, application fallback, operator profile, or
|
||||||
|
effective pipeline profile makes incompatible LLM checkpoints ineligible for
|
||||||
|
reuse. The dependency upgrade is expected to invalidate checkpoints produced
|
||||||
|
under v0.3.0.
|
||||||
|
|
||||||
|
Successful run manifests continue to record only profiles actually selected by
|
||||||
|
PromptKit, including their effective model, backend, and reasoning metadata.
|
||||||
|
Debug output reports the same effective execution snapshot used for generation.
|
||||||
|
|
||||||
|
### Operator Documentation And Examples
|
||||||
|
|
||||||
|
Canonical documentation clearly distinguishes:
|
||||||
|
|
||||||
|
- Notarius prompt and schema assets embedded in the application;
|
||||||
|
- Notarius application fallback profiles embedded in the application;
|
||||||
|
- PromptKit's own embedded built-in profiles; and
|
||||||
|
- operator profile files on the deployment filesystem.
|
||||||
|
|
||||||
|
The configuration reference owns the pipeline field, profile-source fields,
|
||||||
|
validation rules, and precedence. Operations owns deployment layout, working
|
||||||
|
directory behavior, credentials, and environment-specific profile management.
|
||||||
|
The PromptKit integration document owns the pinned upstream contract and
|
||||||
|
source-precedence boundary. Internal documents describe asset registration,
|
||||||
|
resolution, inspection, prepared execution, fingerprinting, and tests without
|
||||||
|
duplicating user-facing field definitions.
|
||||||
|
|
||||||
|
The maintained examples continue to include only the minimal and complete D&D
|
||||||
|
configurations. They use the stable `dnd-extraction` policy, and one maintained
|
||||||
|
PromptKit profile file under `examples/` demonstrates an operator override.
|
||||||
|
Examples remain secret-free and are validated without live provider calls.
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
- Implementing the separate deterministic prompt-session identity roadmap
|
||||||
|
item.
|
||||||
|
- Changing the default `concurrency.total_llm` value; PromptKit's retained
|
||||||
|
OpenRouter capacity of 16 remains relevant to that separate item.
|
||||||
|
- Adding model evaluation as a deterministic or CI correctness gate.
|
||||||
|
- Automatically selecting production, development, or local environments.
|
||||||
|
Deployment configuration chooses the operator profile source.
|
||||||
|
- Profile inheritance, partial profile merging, or cross-profile aliases.
|
||||||
|
- Exposing PromptKit types to modules, validators, durable output contracts, or
|
||||||
|
public configuration structures.
|
||||||
|
- Adopting `InspectPrompt` without a separately justified prompt-interface
|
||||||
|
validation contract.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- Notarius builds and its offline test suite passes with PromptKit v0.5.0.
|
||||||
|
- Every structured completion executes the exact snapshot used for safe debug
|
||||||
|
prompt details.
|
||||||
|
- Profile preflight uses profile inspection and no synthetic prompt.
|
||||||
|
- The embedded `dnd-extraction` fallback resolves without an operator source,
|
||||||
|
and a matching valid operator profile replaces it completely.
|
||||||
|
- Every production module has explicit, correct execution metadata.
|
||||||
|
- Pipeline, binding, CLI, and prompt-default precedence behaves as defined for
|
||||||
|
modules and validators, while deterministic bindings remain profile-free.
|
||||||
|
- Effective profiles participate in pipeline digests, profile inspection,
|
||||||
|
checkpoint identity, debug records, and run provenance at the appropriate
|
||||||
|
boundaries.
|
||||||
|
- The dependency and application fallback changes invalidate incompatible old
|
||||||
|
checkpoints without exposing profile or credential content.
|
||||||
|
- Canonical documentation and maintained examples accurately describe and
|
||||||
|
exercise the implemented operator workflow.
|
||||||
|
- Default tests remain deterministic, offline, credential-free, and focused on
|
||||||
|
Notarius-owned behavior rather than duplicating PromptKit's upstream suite.
|
||||||
@@ -1,4 +1,6 @@
|
|||||||
version: 4
|
version: 4
|
||||||
|
promptkit:
|
||||||
|
profile_file: ./examples/profiles/dnd-extraction.yml
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 2
|
total_llm: 2
|
||||||
stage_workers:
|
stage_workers:
|
||||||
@@ -16,6 +18,7 @@ debug:
|
|||||||
directory: ./notarius-debug
|
directory: ./notarius-debug
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input: seriatim
|
||||||
# Stable campaign context is shared by every module that accepts these slots.
|
# Stable campaign context is shared by every module that accepts these slots.
|
||||||
references:
|
references:
|
||||||
@@ -54,7 +57,6 @@ pipelines:
|
|||||||
merge: appendorder
|
merge: appendorder
|
||||||
normalize:
|
normalize:
|
||||||
module: dnd/npcs
|
module: dnd/npcs
|
||||||
llm_profile: gemini-2-flash
|
|
||||||
retries: 2
|
retries: 2
|
||||||
scene-descriptions:
|
scene-descriptions:
|
||||||
extract:
|
extract:
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
version: 4
|
version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input: seriatim
|
||||||
artifacts:
|
artifacts:
|
||||||
spells:
|
spells:
|
||||||
|
|||||||
5
examples/profiles/dnd-extraction.yml
Normal file
5
examples/profiles/dnd-extraction.yml
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
id: dnd-extraction
|
||||||
|
backend: openrouter
|
||||||
|
model: openai/gpt-5.6-luna
|
||||||
|
timeout_seconds: 240
|
||||||
|
service_tier: flex
|
||||||
2
go.mod
2
go.mod
@@ -3,7 +3,7 @@ module gitea.maximumdirect.net/eric/notarius
|
|||||||
go 1.25.5
|
go 1.25.5
|
||||||
|
|
||||||
require (
|
require (
|
||||||
gitea.maximumdirect.net/eric/promptkit v0.2.0
|
gitea.maximumdirect.net/eric/promptkit v0.5.0
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
||||||
gopkg.in/yaml.v3 v3.0.1
|
gopkg.in/yaml.v3 v3.0.1
|
||||||
)
|
)
|
||||||
|
|||||||
4
go.sum
4
go.sum
@@ -1,5 +1,5 @@
|
|||||||
gitea.maximumdirect.net/eric/promptkit v0.2.0 h1:6bUATKnVgLlHSbKjGb8EnCt2r7jV5Xw6t+pX7G9S/lU=
|
gitea.maximumdirect.net/eric/promptkit v0.5.0 h1:jnpazLyyNhWrB2xzwwtUkNUfktkTdkENTwuSPnKiYrc=
|
||||||
gitea.maximumdirect.net/eric/promptkit v0.2.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
gitea.maximumdirect.net/eric/promptkit v0.5.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||||
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
||||||
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
||||||
|
|||||||
@@ -213,11 +213,12 @@ func assembledSpellPipeline(t *testing.T, options assembledSpellPipelineOptions)
|
|||||||
components := productionTestComponents(t)
|
components := productionTestComponents(t)
|
||||||
extractor := &assembledSpellExtractor{unknownSpell: options.unknownSpell}
|
extractor := &assembledSpellExtractor{unknownSpell: options.unknownSpell}
|
||||||
if err := pipeline.RegisterExtractor[dnd.SpellList](components.registries.Extractors, pipeline.ModuleSpec{
|
if err := pipeline.RegisterExtractor[dnd.SpellList](components.registries.Extractors, pipeline.ModuleSpec{
|
||||||
Key: assembledSpellExtractorKey,
|
Key: assembledSpellExtractorKey,
|
||||||
Stage: pipeline.StageExtract,
|
Stage: pipeline.StageExtract,
|
||||||
Requires: []string{"chunks", "source.transcript"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"dnd.spell_casts"},
|
Requires: []string{"chunks", "source.transcript"},
|
||||||
ArtifactKind: dnd.SpellListKind,
|
Provides: []string{"dnd.spell_casts"},
|
||||||
|
ArtifactKind: dnd.SpellListKind,
|
||||||
}, func() (contracts.Extractor[dnd.SpellList], error) {
|
}, func() (contracts.Extractor[dnd.SpellList], error) {
|
||||||
return extractor, nil
|
return extractor, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
|
|||||||
@@ -143,17 +143,6 @@ func isEmptyRegistries(registries pipeline.Registries) bool {
|
|||||||
registries.Outputs == nil
|
registries.Outputs == nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
|
||||||
if err := ctx.Err(); err != nil {
|
|
||||||
return nil, nil, err
|
|
||||||
}
|
|
||||||
assets, err := productionPromptAssets()
|
|
||||||
if err != nil {
|
|
||||||
return nil, nil, err
|
|
||||||
}
|
|
||||||
return buildProductionLLMClient(ctx, cfg, profileID, overrides, assets)
|
|
||||||
}
|
|
||||||
|
|
||||||
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
|
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
|
||||||
return func(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
return func(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
return buildProductionLLMClient(ctx, cfg, profileID, overrides, assets)
|
return buildProductionLLMClient(ctx, cfg, profileID, overrides, assets)
|
||||||
@@ -171,6 +160,7 @@ func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID
|
|||||||
client, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{
|
client, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{
|
||||||
ProfileDir: cfg.PromptKit.ProfileDir,
|
ProfileDir: cfg.PromptKit.ProfileDir,
|
||||||
ProfileFile: cfg.PromptKit.ProfileFile,
|
ProfileFile: cfg.PromptKit.ProfileFile,
|
||||||
|
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
|
||||||
Assets: assets,
|
Assets: assets,
|
||||||
Recorder: recorder,
|
Recorder: recorder,
|
||||||
ReasoningEffort: overrides.ReasoningEffort,
|
ReasoningEffort: overrides.ReasoningEffort,
|
||||||
|
|||||||
@@ -154,6 +154,23 @@ func TestConfigValidateResolvesPipelineAndChecksSelection(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestConfigValidatePipelineDefaultProfileIsOffline(t *testing.T) {
|
||||||
|
configPath := writeCommandConfigContent(t, `version: 4
|
||||||
|
pipelines:
|
||||||
|
demo:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
|
input: seriatim
|
||||||
|
artifacts:
|
||||||
|
spells:
|
||||||
|
extract: dnd/spells
|
||||||
|
`)
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
code := RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "demo"}, &stdout, &stderr, Options{})
|
||||||
|
if code != 0 || !strings.Contains(stdout.String(), "valid for pipeline \"demo\"") || stderr.Len() != 0 {
|
||||||
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestPipelinesListSortsNormalizedIDsInTextAndJSON(t *testing.T) {
|
func TestPipelinesListSortsNormalizedIDsInTextAndJSON(t *testing.T) {
|
||||||
configPath := writeCommandConfig(t, " zeta ", "alpha")
|
configPath := writeCommandConfig(t, " zeta ", "alpha")
|
||||||
options := commandContractOptions(t)
|
options := commandContractOptions(t)
|
||||||
|
|||||||
@@ -94,6 +94,37 @@ func TestMaintainedConfigurationExampleSet(t *testing.T) {
|
|||||||
if got := strings.Join(names, ","); got != "dnd-complete.config.yml,dnd-minimal.config.yml" {
|
if got := strings.Join(names, ","); got != "dnd-complete.config.yml,dnd-minimal.config.yml" {
|
||||||
t.Fatalf("maintained configuration examples = %q, want only the minimal and complete D&D examples", got)
|
t.Fatalf("maintained configuration examples = %q, want only the minimal and complete D&D examples", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
profileEntries, err := os.ReadDir(repositoryPath("examples", "profiles"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
names = names[:0]
|
||||||
|
for _, entry := range profileEntries {
|
||||||
|
if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".yml") {
|
||||||
|
names = append(names, entry.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
if got := strings.Join(names, ","); got != "dnd-extraction.yml" {
|
||||||
|
t.Fatalf("maintained operator profiles = %q, want dnd-extraction.yml", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMaintainedExamplesValidateEffectiveProfilesOffline(t *testing.T) {
|
||||||
|
t.Chdir(repositoryPath())
|
||||||
|
t.Setenv("OPENROUTER_API_KEY", "")
|
||||||
|
for _, example := range maintainedExampleFiles(t) {
|
||||||
|
t.Run(example.name, func(t *testing.T) {
|
||||||
|
var stdout, stderr strings.Builder
|
||||||
|
code := RunWithOptions([]string{
|
||||||
|
"config", "validate", "--config", example.path, "--pipeline", "dnd-session",
|
||||||
|
}, &stdout, &stderr, Options{})
|
||||||
|
if code != 0 || stderr.Len() != 0 || !strings.Contains(stdout.String(), `valid for pipeline "dnd-session"`) {
|
||||||
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func exampleStepLaneIDs(resolved pipeline.ResolvedPipeline) []string {
|
func exampleStepLaneIDs(resolved pipeline.ResolvedPipeline) []string {
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io/fs"
|
"io/fs"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"reflect"
|
"reflect"
|
||||||
@@ -13,7 +15,9 @@ import (
|
|||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
|
"testing/fstest"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
@@ -34,6 +38,7 @@ import (
|
|||||||
itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents"
|
itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents"
|
||||||
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
|
func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
|
||||||
@@ -161,6 +166,35 @@ func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
|
|||||||
assertProductionContains(t, "production prompt assets", assetNames, requiredAssets)
|
assertProductionContains(t, "production prompt assets", assetNames, requiredAssets)
|
||||||
|
|
||||||
catalog := catalogFromRegistries(registries)
|
catalog := catalogFromRegistries(registries)
|
||||||
|
for _, test := range []struct {
|
||||||
|
stage pipeline.ModuleStage
|
||||||
|
key string
|
||||||
|
want contracts.ExecutionClass
|
||||||
|
}{
|
||||||
|
{stage: pipeline.StageInput, key: "seriatim", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageChunk, key: "generic", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageChunk, key: "dnd/scenes", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/spells", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/npcs", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/combat-turns", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/item-events", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/npc-interactions", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/scene-descriptions", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageMerge, key: "appendorder", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "noop", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/spells", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/npcs", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/combat-turns", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/item-events", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/npc-interactions", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/scene-descriptions", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageOutput, key: "json", want: contracts.ExecutionClassDeterministic},
|
||||||
|
} {
|
||||||
|
got, ok := catalog.ExecutionClass(test.stage, test.key)
|
||||||
|
if !ok || got != test.want {
|
||||||
|
t.Fatalf("production execution class for %s/%s = %q, %t; want %q, true", test.stage, test.key, got, ok, test.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
converted := registriesFromCatalog(catalog)
|
converted := registriesFromCatalog(catalog)
|
||||||
if converted.ArtifactCodecs != registries.ArtifactCodecs || converted.ArtifactEvidence != registries.ArtifactEvidence || converted.ValidatorChains != registries.ValidatorChains {
|
if converted.ArtifactCodecs != registries.ArtifactCodecs || converted.ArtifactEvidence != registries.ArtifactEvidence || converted.ValidatorChains != registries.ValidatorChains {
|
||||||
t.Fatal("catalog/registry conversion did not preserve artifact and validator registries")
|
t.Fatal("catalog/registry conversion did not preserve artifact and validator registries")
|
||||||
@@ -196,6 +230,76 @@ func TestDefaultCLICompositionValidatesRepresentativeConfiguration(t *testing.T)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestProductionAssetsResolveDNDExtractionProfile(t *testing.T) {
|
||||||
|
components := productionTestComponents(t)
|
||||||
|
newEngine := func(profileFile string) (*promptkit.Engine, error) {
|
||||||
|
t.Helper()
|
||||||
|
options, err := components.assets.PromptKitOptions()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if profileFile != "" {
|
||||||
|
options = append(options, promptkit.WithProfileFile(profileFile))
|
||||||
|
}
|
||||||
|
return promptkit.NewEngine(promptkit.Config{}, options...)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Run("fallback", func(t *testing.T) {
|
||||||
|
engine, err := newEngine("")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
inspection, err := engine.InspectProfile(context.Background(), "dnd-extraction")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("InspectProfile() error = %v, want fallback profile", err)
|
||||||
|
}
|
||||||
|
params := inspection.EffectiveModelParams
|
||||||
|
if params.BackendID != "openrouter" || params.Model != "openai/gpt-5.6-luna" || params.TimeoutSeconds != 240 || params.ServiceTier != "flex" {
|
||||||
|
t.Fatalf("fallback profile parameters = %#v", params)
|
||||||
|
}
|
||||||
|
if params.ReasoningEffort != "" || params.Temperature != 0 || params.MaxTokens != 0 || params.TopP != 0 {
|
||||||
|
t.Fatalf("fallback profile selected optional provider controls: %#v", params)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("valid operator profile wins", func(t *testing.T) {
|
||||||
|
profilePath := filepath.Join(t.TempDir(), "profiles.yaml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte(`id: dnd-extraction
|
||||||
|
endpoint: http://operator.example.test/v1
|
||||||
|
model: operator-model
|
||||||
|
timeout_seconds: 75
|
||||||
|
`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
engine, err := newEngine(profilePath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
inspection, err := engine.InspectProfile(context.Background(), "dnd-extraction")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("InspectProfile() error = %v, want operator profile", err)
|
||||||
|
}
|
||||||
|
params := inspection.EffectiveModelParams
|
||||||
|
if params.BackendID != "" || params.Endpoint != "http://operator.example.test/v1" || params.Model != "operator-model" || params.TimeoutSeconds != 75 || params.ServiceTier != "" {
|
||||||
|
t.Fatalf("operator profile parameters = %#v, want complete replacement", params)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("invalid operator profile does not fall through", func(t *testing.T) {
|
||||||
|
profilePath := filepath.Join(t.TempDir(), "profiles.yaml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte("id: dnd-extraction\nendpoint: http://operator.example.test/v1\nmodel: operator-model\nunknown: value\n"), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
engine, err := newEngine(profilePath)
|
||||||
|
if err == nil {
|
||||||
|
_, err = engine.InspectProfile(context.Background(), "dnd-extraction")
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("operator profile error = nil, want failure instead of fallback")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) {
|
func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) {
|
||||||
components := productionTestComponents(t)
|
components := productionTestComponents(t)
|
||||||
cfg := config.Default()
|
cfg := config.Default()
|
||||||
@@ -370,39 +474,150 @@ func setNormalizeSpellCatalogSource(t *testing.T, resolved *pipeline.ResolvedPip
|
|||||||
resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings = bindings
|
resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings = bindings
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) {
|
func TestProductionLLMClientFactoryBuildsOfflineRuntime(t *testing.T) {
|
||||||
components := productionTestComponents(t)
|
components := productionTestComponents(t)
|
||||||
factories := []struct {
|
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
|
||||||
name string
|
if err != nil {
|
||||||
factory LLMClientFactory
|
t.Fatalf("build production LLM runtime: %v", err)
|
||||||
}{
|
|
||||||
{name: "default production assets", factory: productionLLMClientFactory},
|
|
||||||
{name: "provided production assets", factory: productionLLMClientFactoryWithAssets(components.assets)},
|
|
||||||
}
|
}
|
||||||
for _, tt := range factories {
|
if client == nil {
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Fatal("production LLM runtime returned a nil client")
|
||||||
client, manifests, err := tt.factory(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
|
}
|
||||||
if err != nil {
|
if len(manifests) != 0 {
|
||||||
t.Fatalf("build production LLM runtime: %v", err)
|
t.Fatalf("eager profile manifests = %#v, want none", manifests)
|
||||||
}
|
}
|
||||||
if client == nil {
|
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
|
||||||
t.Fatal("production LLM runtime returned a nil client")
|
if !ok {
|
||||||
}
|
t.Fatalf("production LLM client %T does not provide one profile-source checkpoint fingerprint", client)
|
||||||
if len(manifests) != 0 {
|
}
|
||||||
t.Fatalf("eager profile manifests = %#v, want none", manifests)
|
fingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
|
||||||
}
|
if err != nil || len(fingerprints) != 1 {
|
||||||
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
|
t.Fatalf("production LLM checkpoint fingerprints = %#v, error = %v, want one profile-source identity", fingerprints, err)
|
||||||
if !ok {
|
}
|
||||||
t.Fatalf("production LLM client %T does not provide one profile-source checkpoint fingerprint", client)
|
if _, ok := client.(contracts.LLMProfileManifestProvider); !ok {
|
||||||
}
|
t.Fatalf("production LLM client %T does not provide profile manifests", client)
|
||||||
fingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
|
}
|
||||||
if err != nil || len(fingerprints) != 1 {
|
}
|
||||||
t.Fatalf("production LLM checkpoint fingerprints = %#v, error = %v, want one profile-source identity", fingerprints, err)
|
|
||||||
}
|
func TestNormalizeOptionsSharesProductionProfileAssetsWithDefaultRuntime(t *testing.T) {
|
||||||
if _, ok := client.(contracts.LLMProfileManifestProvider); !ok {
|
opts, err := normalizeOptions(Options{
|
||||||
t.Fatalf("production LLM client %T does not provide profile manifests", client)
|
Catalog: pipeline.ModuleCatalog{Inputs: pipeline.NewInputAdapterRegistry()},
|
||||||
}
|
})
|
||||||
})
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if opts.promptKitAssets == nil || opts.LLMClientFactory == nil {
|
||||||
|
t.Fatalf("normalized options = %#v, want shared profile assets and default runtime factory", opts)
|
||||||
|
}
|
||||||
|
if err := validateExplicitPromptKitProfiles(context.Background(), config.Default(), []string{"dnd-extraction"}, opts.promptKitAssets); err != nil {
|
||||||
|
t.Fatalf("inspect application fallback profile: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
client, _, err := opts.LLMClientFactory(context.Background(), config.Default(), "dnd-extraction", LLMRuntimeOverrides{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("build default runtime: %v", err)
|
||||||
|
}
|
||||||
|
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("default runtime client %T does not provide checkpoint fingerprints", client)
|
||||||
|
}
|
||||||
|
runtimeFingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
directClient, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{Assets: opts.promptKitAssets})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
inspectionFingerprints, err := directClient.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(runtimeFingerprints, inspectionFingerprints) {
|
||||||
|
t.Fatalf("runtime profile fingerprints = %#v, inspection profile fingerprints = %#v", runtimeFingerprints, inspectionFingerprints)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProductionLLMClientFactoryUsesConfiguredLocalBackend(t *testing.T) {
|
||||||
|
var providerCalls atomic.Int32
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
providerCalls.Add(1)
|
||||||
|
if r.URL.Path != "/v1/chat/completions" {
|
||||||
|
t.Errorf("provider path = %q, want /v1/chat/completions", r.URL.Path)
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_, _ = w.Write([]byte(`{
|
||||||
|
"choices": [{"message": {"role": "assistant", "content": "{\"ok\":true}"}}],
|
||||||
|
"usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7}
|
||||||
|
}`))
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte(`id: local-profile
|
||||||
|
backend: local
|
||||||
|
model: local-model
|
||||||
|
`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
assets := llm.NewAssetRegistry()
|
||||||
|
if err := assets.RegisterPromptFS(fstest.MapFS{
|
||||||
|
"production.local.yaml": {Data: []byte(`id: production.local
|
||||||
|
version: "v1"
|
||||||
|
inputs:
|
||||||
|
- name: transcript
|
||||||
|
required: true
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: '{{ input "transcript" }}'
|
||||||
|
output:
|
||||||
|
format: json
|
||||||
|
validation_mode: json
|
||||||
|
`)},
|
||||||
|
}, "."); err != nil {
|
||||||
|
t.Fatalf("register prompt assets: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg := config.Default()
|
||||||
|
cfg.PromptKit.ProfileFile = profilePath
|
||||||
|
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: server.URL + "/v1",
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
client, manifests, err := productionLLMClientFactoryWithAssets(assets)(
|
||||||
|
context.Background(),
|
||||||
|
cfg,
|
||||||
|
"local-profile",
|
||||||
|
LLMRuntimeOverrides{},
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("build production LLM runtime: %v", err)
|
||||||
|
}
|
||||||
|
if len(manifests) != 0 {
|
||||||
|
t.Fatalf("eager profile manifests = %#v, want none", manifests)
|
||||||
|
}
|
||||||
|
|
||||||
|
var out map[string]any
|
||||||
|
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||||
|
PromptID: "production.local",
|
||||||
|
ProfileID: "local-profile",
|
||||||
|
Inputs: contracts.LLMInputSet{
|
||||||
|
"transcript": contracts.NewLLMInputMaterial("transcript", "text/plain", []byte("local request"), "", ""),
|
||||||
|
},
|
||||||
|
}, &out)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if providerCalls.Load() != 1 {
|
||||||
|
t.Fatalf("provider calls = %d, want 1", providerCalls.Load())
|
||||||
|
}
|
||||||
|
provider, ok := client.(contracts.LLMProfileManifestProvider)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("production client %T does not provide profile manifests", client)
|
||||||
|
}
|
||||||
|
recorded := provider.LLMProfileManifests()
|
||||||
|
if len(recorded) != 1 || recorded[0].BackendID != promptkit.BackendLocal {
|
||||||
|
t.Fatalf("production profile manifests = %#v, want local backend", recorded)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -410,7 +625,8 @@ func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
|
|||||||
t.Run("canceled context", func(t *testing.T) {
|
t.Run("canceled context", func(t *testing.T) {
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
cancel()
|
cancel()
|
||||||
client, manifests, err := productionLLMClientFactory(ctx, config.Default(), "test-profile", LLMRuntimeOverrides{})
|
components := productionTestComponents(t)
|
||||||
|
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(ctx, config.Default(), "test-profile", LLMRuntimeOverrides{})
|
||||||
if !errors.Is(err, context.Canceled) || client != nil || len(manifests) != 0 {
|
if !errors.Is(err, context.Canceled) || client != nil || len(manifests) != 0 {
|
||||||
t.Fatalf("client=%T manifests=%#v error=%v, want canceled construction", client, manifests, err)
|
t.Fatalf("client=%T manifests=%#v error=%v, want canceled construction", client, manifests, err)
|
||||||
}
|
}
|
||||||
@@ -657,9 +873,10 @@ func productionCLIOptions(t *testing.T) Options {
|
|||||||
|
|
||||||
func productionOptionsFromComponents(components productionComponents) Options {
|
func productionOptionsFromComponents(components productionComponents) Options {
|
||||||
return Options{
|
return Options{
|
||||||
Catalog: catalogFromRegistries(components.registries),
|
Catalog: catalogFromRegistries(components.registries),
|
||||||
Registries: components.registries,
|
Registries: components.registries,
|
||||||
LookupEnv: emptyLookup,
|
LookupEnv: emptyLookup,
|
||||||
|
promptKitAssets: components.assets,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -2,67 +2,45 @@ package cli
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
|
||||||
"fmt"
|
"fmt"
|
||||||
"testing/fstest"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
"gitea.maximumdirect.net/eric/promptkit"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
)
|
)
|
||||||
|
|
||||||
const profileCheckPromptID = "notarius.profile.check"
|
func validateExplicitPromptKitProfiles(ctx context.Context, cfg config.Config, profileIDs []string, assets *llm.AssetRegistry) error {
|
||||||
|
|
||||||
var profileCheckPromptFS = fstest.MapFS{
|
|
||||||
"prompts/profile-check.yaml": &fstest.MapFile{Data: []byte(`id: notarius.profile.check
|
|
||||||
version: "1.0.0"
|
|
||||||
default_profile: mistral-small-3
|
|
||||||
inputs:
|
|
||||||
- name: transcript
|
|
||||||
required: true
|
|
||||||
messages:
|
|
||||||
- role: user
|
|
||||||
content: "{{input \"transcript\"}}"
|
|
||||||
output:
|
|
||||||
format: text
|
|
||||||
validation_mode: none
|
|
||||||
repair_attempts: 0
|
|
||||||
`)},
|
|
||||||
}
|
|
||||||
|
|
||||||
func validateExplicitPromptKitProfiles(ctx context.Context, cfg config.Config, profileIDs []string) error {
|
|
||||||
if len(profileIDs) == 0 {
|
if len(profileIDs) == 0 {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
engine, err := newProfileValidationEngine(cfg)
|
inspector, err := llm.NewPromptKitProfileInspector(llm.PromptKitProfileInspectorConfig{
|
||||||
|
Source: promptKitProfileSourceConfig(cfg),
|
||||||
|
Assets: assets,
|
||||||
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("load PromptKit profiles: %w", err)
|
return fmt.Errorf("load PromptKit profiles: %w", err)
|
||||||
}
|
}
|
||||||
for _, profileID := range profileIDs {
|
for _, profileID := range profileIDs {
|
||||||
if _, err := engine.Prepare(ctx, promptkit.RunRequest{
|
if _, err := inspector.InspectProfile(ctx, profileID); err != nil {
|
||||||
PromptID: profileCheckPromptID,
|
return err
|
||||||
ProfileID: profileID,
|
|
||||||
Inputs: map[string]promptkit.ArtifactRef{
|
|
||||||
"transcript": promptkit.Inline("profile check"),
|
|
||||||
},
|
|
||||||
}); err != nil {
|
|
||||||
if errors.Is(err, promptkit.ErrProfileNotFound) {
|
|
||||||
return fmt.Errorf("PromptKit profile %q is not configured", profileID)
|
|
||||||
}
|
|
||||||
return fmt.Errorf("validate PromptKit profile %q: %w", profileID, err)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func newProfileValidationEngine(cfg config.Config) (*promptkit.Engine, error) {
|
func promptKitProfileSourceConfig(cfg config.Config) llm.PromptKitProfileSourceConfig {
|
||||||
opts := []promptkit.Option{
|
return llm.PromptKitProfileSourceConfig{
|
||||||
promptkit.WithPromptFS(profileCheckPromptFS, "prompts"),
|
ProfileDir: cfg.PromptKit.ProfileDir,
|
||||||
|
ProfileFile: cfg.PromptKit.ProfileFile,
|
||||||
|
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mapPromptKitLocalBackend(cfg *config.PromptKitLocalBackendConfig) *llm.PromptKitLocalBackendConfig {
|
||||||
|
if cfg == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return &llm.PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: cfg.Endpoint,
|
||||||
|
ConcurrencyLimit: cfg.ConcurrencyLimit,
|
||||||
}
|
}
|
||||||
if cfg.PromptKit.ProfileFile != "" {
|
|
||||||
opts = append(opts, promptkit.WithProfileFile(cfg.PromptKit.ProfileFile))
|
|
||||||
}
|
|
||||||
return promptkit.NewEngine(promptkit.Config{
|
|
||||||
PromptDir: "unused",
|
|
||||||
ProfileDir: cfg.PromptKit.ProfileDir,
|
|
||||||
}, opts...)
|
|
||||||
}
|
}
|
||||||
|
|||||||
159
internal/cli/promptkit_profiles_test.go
Normal file
159
internal/cli/promptkit_profiles_test.go
Normal file
@@ -0,0 +1,159 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"testing/fstest"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestExplicitPromptKitProfileValidationInspectsProfilesWithoutGeneration(t *testing.T) {
|
||||||
|
var providerCalls atomic.Int32
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
|
||||||
|
providerCalls.Add(1)
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
writeProfile := func(t *testing.T, name, content string) string {
|
||||||
|
t.Helper()
|
||||||
|
profilePath := filepath.Join(t.TempDir(), name+".yaml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte(content), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return profilePath
|
||||||
|
}
|
||||||
|
localProfile := "id: local-profile\nbackend: local\nmodel: local-model\n"
|
||||||
|
credentialProfile := `id: credential-profile
|
||||||
|
endpoint: ` + server.URL + `/v1
|
||||||
|
model: credential-model
|
||||||
|
api_key_env: NOTARIUS_PROMPTKIT_PROFILE_INSPECTION_TEST_KEY
|
||||||
|
`
|
||||||
|
t.Setenv("NOTARIUS_PROMPTKIT_PROFILE_INSPECTION_TEST_KEY", "")
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
profilePath string
|
||||||
|
profileID string
|
||||||
|
profileDir bool
|
||||||
|
localBackend bool
|
||||||
|
canceled bool
|
||||||
|
wantErr []string
|
||||||
|
rejectErr []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "configured local backend",
|
||||||
|
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||||
|
profileID: "local-profile",
|
||||||
|
profileDir: true,
|
||||||
|
localBackend: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "missing local backend registration",
|
||||||
|
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||||
|
profileID: "local-profile",
|
||||||
|
wantErr: []string{`PromptKit profile "local-profile" is invalid or unreadable`},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "absent profile",
|
||||||
|
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||||
|
profileID: "absent-profile",
|
||||||
|
localBackend: true,
|
||||||
|
wantErr: []string{`PromptKit profile "absent-profile" is not configured`},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "malformed profile",
|
||||||
|
profilePath: writeProfile(t, "malformed-profile", "id: malformed-profile\nbackend: [\n"),
|
||||||
|
profileID: "malformed-profile",
|
||||||
|
wantErr: []string{`PromptKit profile "malformed-profile" is invalid or unreadable`},
|
||||||
|
rejectErr: []string{"malformed-profile.yaml", "backend: ["},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "invalid profile source",
|
||||||
|
profilePath: filepath.Join(t.TempDir(), "missing-profile.yaml"),
|
||||||
|
profileID: "missing-profile",
|
||||||
|
wantErr: []string{"load PromptKit profiles", "profile configuration is invalid or unreadable"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "credential environment intentionally unset",
|
||||||
|
profilePath: writeProfile(t, "credential-profile", credentialProfile),
|
||||||
|
profileID: "credential-profile",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "canceled inspection",
|
||||||
|
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||||
|
profileID: "local-profile",
|
||||||
|
localBackend: true,
|
||||||
|
canceled: true,
|
||||||
|
wantErr: []string{"context canceled"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := config.Default()
|
||||||
|
if tt.profileDir {
|
||||||
|
cfg.PromptKit.ProfileDir = filepath.Dir(tt.profilePath)
|
||||||
|
} else {
|
||||||
|
cfg.PromptKit.ProfileFile = tt.profilePath
|
||||||
|
}
|
||||||
|
if tt.localBackend {
|
||||||
|
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: server.URL + "/v1",
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
if tt.canceled {
|
||||||
|
var cancel context.CancelFunc
|
||||||
|
ctx, cancel = context.WithCancel(ctx)
|
||||||
|
cancel()
|
||||||
|
}
|
||||||
|
err := validateExplicitPromptKitProfiles(ctx, cfg, []string{tt.profileID}, nil)
|
||||||
|
if len(tt.wantErr) == 0 {
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("validateExplicitPromptKitProfiles() error = nil, want failure")
|
||||||
|
}
|
||||||
|
if tt.canceled && !errors.Is(err, context.Canceled) {
|
||||||
|
t.Fatalf("canceled inspection error = %v, want context canceled", err)
|
||||||
|
}
|
||||||
|
for _, want := range tt.wantErr {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("validation error = %q, want %q", err, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, rejected := range append(tt.rejectErr, tt.profilePath) {
|
||||||
|
if rejected != "" && strings.Contains(err.Error(), rejected) {
|
||||||
|
t.Fatalf("validation error = %q, must not expose %q", err, rejected)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if providerCalls.Load() != 0 {
|
||||||
|
t.Fatalf("provider calls during profile inspection = %d, want 0", providerCalls.Load())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExplicitPromptKitProfileValidationUsesFallbackAssets(t *testing.T) {
|
||||||
|
assets := llm.NewAssetRegistry()
|
||||||
|
if err := assets.RegisterFallbackProfileFS(fstest.MapFS{
|
||||||
|
"profiles/fallback.yaml": {Data: []byte("id: fallback-profile\nendpoint: http://promptkit.test/v1\nmodel: fallback-model\n")},
|
||||||
|
}, "profiles"); err != nil {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if err := validateExplicitPromptKitProfiles(context.Background(), config.Default(), []string{"fallback-profile"}, assets); err != nil {
|
||||||
|
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -116,7 +116,7 @@ func (h *recomputeTestHarness) options() Options {
|
|||||||
for _, key := range []string{"test/extract/producer", "test/extract/unrelated", "test/extract/middle", "test/extract/dependent"} {
|
for _, key := range []string{"test/extract/producer", "test/extract/unrelated", "test/extract/middle", "test/extract/dependent"} {
|
||||||
moduleKey := key
|
moduleKey := key
|
||||||
spec := pipeline.ModuleSpec{
|
spec := pipeline.ModuleSpec{
|
||||||
Key: moduleKey, Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind,
|
Key: moduleKey, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind,
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}},
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterExtractor(opts.Registries.Extractors, spec, func() (contracts.Extractor[stateTestArtifact], error) {
|
if err := pipeline.RegisterExtractor(opts.Registries.Extractors, spec, func() (contracts.Extractor[stateTestArtifact], error) {
|
||||||
@@ -125,7 +125,7 @@ func (h *recomputeTestHarness) options() Options {
|
|||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if err := opts.Registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/recompute-output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
if err := opts.Registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/recompute-output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
||||||
return recomputeTestOutput{}, nil
|
return recomputeTestOutput{}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
|
|||||||
@@ -364,21 +364,21 @@ func referenceContractCatalog(t *testing.T, includeBetaMerger, includeBetaNormal
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
register(registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "reference/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }))
|
register(registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "reference/input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }))
|
||||||
register(registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/chunk", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-slot"}, {Name: "required-chunk", Required: true}}}, func() (contracts.Chunker, error) { return stateTestChunker{}, nil }))
|
register(registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-slot"}, {Name: "required-chunk", Required: true}}}, func() (contracts.Chunker, error) { return stateTestChunker{}, nil }))
|
||||||
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecA{}))
|
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecA{}))
|
||||||
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecB{}))
|
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecB{}))
|
||||||
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-alpha", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-alpha", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
||||||
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-beta", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-beta", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
||||||
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
||||||
if includeBetaMerger {
|
if includeBetaMerger {
|
||||||
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
||||||
}
|
}
|
||||||
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
||||||
if includeBetaNormalizer {
|
if includeBetaNormalizer {
|
||||||
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
||||||
}
|
}
|
||||||
register(registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { return stateTestOutput{}, nil }))
|
register(registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { return stateTestOutput{}, nil }))
|
||||||
return catalogFromRegistries(registries)
|
return catalogFromRegistries(registries)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -46,6 +46,7 @@ type Options struct {
|
|||||||
ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory
|
ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory
|
||||||
DebugRecorderFactory func(string) (pipeline.DebugRecorder, error)
|
DebugRecorderFactory func(string) (pipeline.DebugRecorder, error)
|
||||||
DebugTerminalFactory func(*debugbundle.SummaryWriter) DebugTerminalWriter
|
DebugTerminalFactory func(*debugbundle.SummaryWriter) DebugTerminalWriter
|
||||||
|
promptKitAssets *frameworkllm.AssetRegistry
|
||||||
}
|
}
|
||||||
|
|
||||||
type LLMRuntimeOverrides struct {
|
type LLMRuntimeOverrides struct {
|
||||||
@@ -121,12 +122,17 @@ func normalizeOptions(opts Options) (Options, error) {
|
|||||||
}
|
}
|
||||||
opts.Registries = components.registries
|
opts.Registries = components.registries
|
||||||
opts.Catalog = catalogFromRegistries(components.registries)
|
opts.Catalog = catalogFromRegistries(components.registries)
|
||||||
if opts.LLMClientFactory == nil {
|
opts.promptKitAssets = components.assets
|
||||||
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(components.assets)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if opts.LLMClientFactory == nil {
|
if opts.LLMClientFactory == nil {
|
||||||
opts.LLMClientFactory = productionLLMClientFactory
|
if opts.promptKitAssets == nil {
|
||||||
|
assets, err := productionPromptAssets()
|
||||||
|
if err != nil {
|
||||||
|
return Options{}, err
|
||||||
|
}
|
||||||
|
opts.promptKitAssets = assets
|
||||||
|
}
|
||||||
|
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(opts.promptKitAssets)
|
||||||
}
|
}
|
||||||
return opts, nil
|
return opts, nil
|
||||||
}
|
}
|
||||||
@@ -340,7 +346,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||||
}
|
}
|
||||||
profileIDs := effectiveLLMProfileIDs(effective.ResolvedPipeline)
|
profileIDs := effectiveLLMProfileIDs(effective.ResolvedPipeline)
|
||||||
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, profileIDs); err != nil {
|
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, profileIDs, opts.promptKitAssets); err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||||
}
|
}
|
||||||
workingDir, err := os.Getwd()
|
workingDir, err := os.Getwd()
|
||||||
@@ -952,11 +958,22 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
|
|||||||
seen[id] = struct{}{}
|
seen[id] = struct{}{}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
add(resolved.Chunk)
|
if resolved.InputExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
|
add(resolved.Input)
|
||||||
|
}
|
||||||
|
if resolved.ChunkExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
|
add(resolved.Chunk)
|
||||||
|
}
|
||||||
for _, lane := range resolved.AllArtifactLanes() {
|
for _, lane := range resolved.AllArtifactLanes() {
|
||||||
add(lane.Extract)
|
if lane.ExtractExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
add(lane.Merge)
|
add(lane.Extract)
|
||||||
add(lane.Normalize)
|
}
|
||||||
|
if lane.MergeExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
|
add(lane.Merge)
|
||||||
|
}
|
||||||
|
if lane.NormalizeExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
|
add(lane.Normalize)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
for _, chain := range resolved.ValidatorChains {
|
for _, chain := range resolved.ValidatorChains {
|
||||||
for _, validator := range chain.Validators {
|
for _, validator := range chain.Validators {
|
||||||
@@ -965,6 +982,9 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if resolved.OutputExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
|
add(resolved.Output)
|
||||||
|
}
|
||||||
ids := make([]string, 0, len(seen))
|
ids := make([]string, 0, len(seen))
|
||||||
for id := range seen {
|
for id := range seen {
|
||||||
ids = append(ids, id)
|
ids = append(ids, id)
|
||||||
@@ -1048,7 +1068,7 @@ func runConfigValidate(args []string, stdout, stderr io.Writer, opts Options) in
|
|||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline)); err != nil {
|
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline), opts.promptKitAssets); err != nil {
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -276,7 +276,7 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
t.Run("validator profile remains distinct", func(t *testing.T) {
|
t.Run("runtime override applies to validators", func(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
profileDir := writeRunContractProfiles(t, "override-profile", "validator-profile")
|
profileDir := writeRunContractProfiles(t, "override-profile", "validator-profile")
|
||||||
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||||
@@ -294,11 +294,11 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
if code != 0 || stderr.Len() != 0 {
|
if code != 0 || stderr.Len() != 0 {
|
||||||
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
}
|
}
|
||||||
if len(factoryProfiles) != 1 || factoryProfiles[0] != "" {
|
if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" {
|
||||||
t.Fatalf("factory profiles = %#v, want one call without a unique profile", factoryProfiles)
|
t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles)
|
||||||
}
|
}
|
||||||
if len(validatorProfiles) != 1 || validatorProfiles[0] != "validator-profile" {
|
if len(validatorProfiles) != 1 || validatorProfiles[0] != "override-profile" {
|
||||||
t.Fatalf("validator profiles = %#v, want configured validator profile", validatorProfiles)
|
t.Fatalf("validator profiles = %#v, want runtime override", validatorProfiles)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -318,6 +318,24 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
|
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run("pipeline default is rejected before factory access", func(t *testing.T) {
|
||||||
|
roots := newStateTestRoots(t)
|
||||||
|
profileDir := writeRunContractProfiles(t, "configured-profile")
|
||||||
|
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||||
|
replaceStateTestConfigLine(t, roots.config, " sample:\n", " sample:\n llm_profile: missing-profile\n")
|
||||||
|
factoryCalls := 0
|
||||||
|
opts := newStateTestHarness().options()
|
||||||
|
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
|
factoryCalls++
|
||||||
|
return nil, nil, nil
|
||||||
|
}
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, &stdout, &stderr, opts)
|
||||||
|
if code != 1 || !strings.Contains(stderr.String(), "not configured") || factoryCalls != 0 || stdout.Len() != 0 {
|
||||||
|
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
|
||||||
|
}
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunReasoningEffortOverrideReachesFactory(t *testing.T) {
|
func TestRunReasoningEffortOverrideReachesFactory(t *testing.T) {
|
||||||
@@ -445,24 +463,30 @@ func TestReasoningEffortOverrideSeparatesCheckpointIdentities(t *testing.T) {
|
|||||||
|
|
||||||
func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
|
func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
|
||||||
resolved := pipeline.ResolvedPipeline{
|
resolved := pipeline.ResolvedPipeline{
|
||||||
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
|
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
|
||||||
Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "},
|
InputExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
|
Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "},
|
||||||
|
ChunkExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Steps: []pipeline.ResolvedPipelineStep{{
|
Steps: []pipeline.ResolvedPipelineStep{{
|
||||||
ID: "default",
|
ID: "default",
|
||||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
||||||
Extract: pipeline.ModuleBinding{LLMProfile: "alpha"},
|
Extract: pipeline.ModuleBinding{LLMProfile: "alpha"},
|
||||||
Merge: pipeline.ModuleBinding{LLMProfile: "zeta"},
|
ExtractExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "},
|
Merge: pipeline.ModuleBinding{LLMProfile: "deterministic-merge"},
|
||||||
|
MergeExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "},
|
||||||
|
NormalizeExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
}},
|
}},
|
||||||
}},
|
}},
|
||||||
ValidatorChains: []pipeline.ResolvedValidatorChain{{Validators: []pipeline.ResolvedValidator{
|
ValidatorChains: []pipeline.ResolvedValidatorChain{{Validators: []pipeline.ResolvedValidator{
|
||||||
{Binding: pipeline.ModuleBinding{LLMProfile: "deterministic-profile"}, ExecutionClass: contracts.ExecutionClassDeterministic},
|
{Binding: pipeline.ModuleBinding{LLMProfile: "deterministic-profile"}, ExecutionClass: contracts.ExecutionClassDeterministic},
|
||||||
{Binding: pipeline.ModuleBinding{LLMProfile: "beta"}, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
{Binding: pipeline.ModuleBinding{LLMProfile: "beta"}, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
}}},
|
}}},
|
||||||
Output: pipeline.ModuleBinding{LLMProfile: "output-profile"},
|
Output: pipeline.ModuleBinding{LLMProfile: "output-profile"},
|
||||||
|
OutputExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
}
|
}
|
||||||
got := effectiveLLMProfileIDs(resolved)
|
got := effectiveLLMProfileIDs(resolved)
|
||||||
want := []string{"alpha", "beta", "gamma", "zeta"}
|
want := []string{"alpha", "beta", "gamma", "input-profile", "output-profile", "zeta"}
|
||||||
if strings.Join(got, ",") != strings.Join(want, ",") {
|
if strings.Join(got, ",") != strings.Join(want, ",") {
|
||||||
t.Fatalf("effective profiles = %#v, want %#v", got, want)
|
t.Fatalf("effective profiles = %#v, want %#v", got, want)
|
||||||
}
|
}
|
||||||
@@ -526,7 +550,7 @@ func TestRunFactoryAndPreparationFailuresAreProcessFailures(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
opts := newStateTestHarness().options()
|
opts := newStateTestHarness().options()
|
||||||
if err := pipeline.RegisterExtractorBuilder(opts.Registries.Extractors, pipeline.ModuleSpec{Key: "test/failing-extract", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Extractor[stateTestArtifact], error) {
|
if err := pipeline.RegisterExtractorBuilder(opts.Registries.Extractors, pipeline.ModuleSpec{Key: "test/failing-extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Extractor[stateTestArtifact], error) {
|
||||||
return nil, errors.New("injected extractor construction failure")
|
return nil, errors.New("injected extractor construction failure")
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
|
|||||||
@@ -823,22 +823,22 @@ func (h *stateTestHarness) options() Options {
|
|||||||
if err := pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, stateTestCodec{}); err != nil {
|
if err := pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, stateTestCodec{}); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }); err != nil {
|
if err := registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := registries.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/chunk", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "cache-reference"}}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Chunker, error) { return stateTestChunker{h}, nil }); err != nil {
|
if err := registries.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "cache-reference"}}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Chunker, error) { return stateTestChunker{h}, nil }); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "test/extract", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{h}, nil }); err != nil {
|
if err := pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "test/extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{h}, nil }); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "test/merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{harness: h}, nil }); err != nil {
|
if err := pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "test/merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{harness: h}, nil }); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "test/normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{harness: h}, nil }); err != nil {
|
if err := pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "test/normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{harness: h}, nil }); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
||||||
return stateTestOutput{harness: h, includeWarnings: h.includeWarnings}, nil
|
return stateTestOutput{harness: h, includeWarnings: h.includeWarnings}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
|
|||||||
@@ -16,8 +16,14 @@ type Config struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type PromptKitConfig struct {
|
type PromptKitConfig struct {
|
||||||
ProfileDir string `json:"profile_dir,omitempty"`
|
ProfileDir string `json:"profile_dir,omitempty"`
|
||||||
ProfileFile string `json:"profile_file,omitempty"`
|
ProfileFile string `json:"profile_file,omitempty"`
|
||||||
|
LocalBackend *PromptKitLocalBackendConfig `json:"local_backend,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitLocalBackendConfig struct {
|
||||||
|
Endpoint string `json:"endpoint"`
|
||||||
|
ConcurrencyLimit int `json:"concurrency_limit"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type ConcurrencyConfig struct {
|
type ConcurrencyConfig struct {
|
||||||
@@ -66,6 +72,10 @@ func Default() Config {
|
|||||||
|
|
||||||
func cloneConfig(in Config) Config {
|
func cloneConfig(in Config) Config {
|
||||||
out := in
|
out := in
|
||||||
|
if in.PromptKit.LocalBackend != nil {
|
||||||
|
localBackend := *in.PromptKit.LocalBackend
|
||||||
|
out.PromptKit.LocalBackend = &localBackend
|
||||||
|
}
|
||||||
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
|
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
|
||||||
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
|
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
|
||||||
for key, profile := range in.Pipelines {
|
for key, profile := range in.Pipelines {
|
||||||
|
|||||||
@@ -42,12 +42,10 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
|
|||||||
}
|
}
|
||||||
profile = clonePipelineProfile(profile)
|
profile = clonePipelineProfile(profile)
|
||||||
profile.ID = pipelineID
|
profile.ID = pipelineID
|
||||||
if override := strings.TrimSpace(input.LLMProfileOverride); override != "" {
|
|
||||||
applyLLMProfileOverride(&profile, override)
|
|
||||||
}
|
|
||||||
|
|
||||||
resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{
|
resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{
|
||||||
Only: input.Only,
|
Only: input.Only,
|
||||||
|
LLMProfileOverride: input.LLMProfileOverride,
|
||||||
ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...),
|
ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...),
|
||||||
ReferenceUnbinds: append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...),
|
ReferenceUnbinds: append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...),
|
||||||
}, input.Catalog)
|
}, input.Catalog)
|
||||||
@@ -65,22 +63,6 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func applyLLMProfileOverride(profile *pipeline.PipelineProfile, profileID string) {
|
|
||||||
profile.Chunk.LLMProfile = profileID
|
|
||||||
apply := func(artifacts map[string]pipeline.ArtifactLaneProfile) {
|
|
||||||
for laneID, lane := range artifacts {
|
|
||||||
lane.Extract.LLMProfile = profileID
|
|
||||||
lane.Merge.LLMProfile = profileID
|
|
||||||
lane.Normalize.LLMProfile = profileID
|
|
||||||
artifacts[laneID] = lane
|
|
||||||
}
|
|
||||||
}
|
|
||||||
apply(profile.Artifacts)
|
|
||||||
for index := range profile.Steps {
|
|
||||||
apply(profile.Steps[index].Artifacts)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func lookupPipelineProfile(profiles map[string]pipeline.PipelineProfile, pipelineID string) (pipeline.PipelineProfile, bool) {
|
func lookupPipelineProfile(profiles map[string]pipeline.PipelineProfile, pipelineID string) (pipeline.PipelineProfile, bool) {
|
||||||
pipelineID = strings.TrimSpace(pipelineID)
|
pipelineID = strings.TrimSpace(pipelineID)
|
||||||
for rawID, profile := range profiles {
|
for rawID, profile := range profiles {
|
||||||
|
|||||||
@@ -107,6 +107,33 @@ func TestEffectiveConfigPreservesPromptKitProfileSource(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestEffectiveConfigOwnsPromptKitLocalBackend(t *testing.T) {
|
||||||
|
cfg := configForEffectiveTests(t, effectiveProfile())
|
||||||
|
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: "http://localhost:8000/v1",
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
effective, err := cfg.Resolve(ResolveInput{PipelineID: "main", Catalog: effectiveCatalog(t)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v", err)
|
||||||
|
}
|
||||||
|
if effective.Config.PromptKit.LocalBackend == nil {
|
||||||
|
t.Fatal("effective local backend = nil")
|
||||||
|
}
|
||||||
|
if effective.Config.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend {
|
||||||
|
t.Fatal("effective local backend aliases input config")
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg.PromptKit.LocalBackend.Endpoint = "http://changed-input.example/v1"
|
||||||
|
if effective.Config.PromptKit.LocalBackend.Endpoint != "http://localhost:8000/v1" {
|
||||||
|
t.Fatalf("input mutation changed effective config: %#v", effective.Config.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
effective.Config.PromptKit.LocalBackend.ConcurrencyLimit = 9
|
||||||
|
if cfg.PromptKit.LocalBackend.ConcurrencyLimit != 2 {
|
||||||
|
t.Fatalf("effective mutation changed input config: %#v", cfg.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
|
func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -177,7 +204,7 @@ func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestEffectiveConfigLLMProfileOverrideChangesDigestWithoutOverridingValidators(t *testing.T) {
|
func TestEffectiveConfigLLMProfileOverrideChangesDigestAndOverridesValidators(t *testing.T) {
|
||||||
profile := effectiveProfile()
|
profile := effectiveProfile()
|
||||||
profile.Chunk.LLMProfile = "chunk-profile"
|
profile.Chunk.LLMProfile = "chunk-profile"
|
||||||
lane := profile.Artifacts["lane"]
|
lane := profile.Artifacts["lane"]
|
||||||
@@ -210,8 +237,27 @@ func TestEffectiveConfigLLMProfileOverrideChangesDigestWithoutOverridingValidato
|
|||||||
t.Fatalf("pipeline profile override was not applied: %#v", resolved)
|
t.Fatalf("pipeline profile override was not applied: %#v", resolved)
|
||||||
}
|
}
|
||||||
validators := findEffectiveValidatorChain(resolved, pipeline.StageExtract, "lane")
|
validators := findEffectiveValidatorChain(resolved, pipeline.StageExtract, "lane")
|
||||||
if len(validators.Validators) != 1 || validators.Validators[0].Binding.LLMProfile != "validator-profile" {
|
if len(validators.Validators) != 1 || validators.Validators[0].Binding.LLMProfile != "override-profile" {
|
||||||
t.Fatalf("validator profile was overridden: %#v", validators)
|
t.Fatalf("validator profile = %#v, want runtime override", validators)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEffectiveConfigPipelineLLMProfileIsInheritedWithoutMutatingConfig(t *testing.T) {
|
||||||
|
profile := effectiveProfile()
|
||||||
|
profile.LLMProfile = " configured-profile "
|
||||||
|
effective, err := resolveEffectiveProfile(t, profile, ResolveInput{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v", err)
|
||||||
|
}
|
||||||
|
if got := effective.Config.Pipelines["main"].LLMProfile; got != " configured-profile " {
|
||||||
|
t.Fatalf("effective config pipeline llm profile = %q, want preserved programmatic value", got)
|
||||||
|
}
|
||||||
|
resolved := effective.ResolvedPipeline
|
||||||
|
if got := resolved.Chunk.LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("resolved chunk profile = %q, want inherited profile", got)
|
||||||
|
}
|
||||||
|
if got := resolved.Steps[0].ArtifactLanes[0].Extract.LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("resolved extract profile = %q, want inherited profile", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -458,7 +504,7 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
|
|||||||
if err := pipeline.RegisterArtifactCodec(catalog.ArtifactCodecs, effectiveCodec{}); err != nil {
|
if err := pipeline.RegisterArtifactCodec(catalog.ArtifactCodecs, effectiveCodec{}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := catalog.Inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
|
if err := catalog.Inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
|
||||||
return effectiveInput{key: "input"}, nil
|
return effectiveInput{key: "input"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
@@ -466,6 +512,7 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
|
|||||||
chunkSpec := pipeline.ModuleSpec{
|
chunkSpec := pipeline.ModuleSpec{
|
||||||
Key: pipeline.DefaultChunkModule,
|
Key: pipeline.DefaultChunkModule,
|
||||||
Stage: pipeline.StageChunk,
|
Stage: pipeline.StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: []string{"source"},
|
Requires: []string{"source"},
|
||||||
Provides: []string{"chunk"},
|
Provides: []string{"chunk"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-ref"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-ref"}},
|
||||||
@@ -476,32 +523,32 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := catalog.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "needs-capability", Stage: pipeline.StageChunk, Requires: []string{"missing"}}, chunkOptions, func(pipeline.BuildRequest) (contracts.Chunker, error) {
|
if err := catalog.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "needs-capability", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}}, chunkOptions, func(pipeline.BuildRequest) (contracts.Chunker, error) {
|
||||||
return effectiveChunker{key: "needs-capability"}, nil
|
return effectiveChunker{key: "needs-capability"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterExtractor(catalog.Extractors, pipeline.ModuleSpec{Key: "extract", Stage: pipeline.StageExtract, ArtifactKind: effectiveArtifactKind, Requires: []string{"chunk"}, Provides: []string{"candidate"}}, func() (contracts.Extractor[effectiveArtifact], error) {
|
if err := pipeline.RegisterExtractor(catalog.Extractors, pipeline.ModuleSpec{Key: "extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"chunk"}, Provides: []string{"candidate"}}, func() (contracts.Extractor[effectiveArtifact], error) {
|
||||||
return effectiveExtractor{key: "extract"}, nil
|
return effectiveExtractor{key: "extract"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: pipeline.DefaultMergeModule, Stage: pipeline.StageMerge, ArtifactKind: effectiveArtifactKind, Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: pipeline.DefaultMergeModule, Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
||||||
return effectiveMerger{key: pipeline.DefaultMergeModule}, nil
|
return effectiveMerger{key: pipeline.DefaultMergeModule}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: "other-merge", Stage: pipeline.StageMerge, ArtifactKind: "other-kind", Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: "other-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "other-kind", Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
||||||
return effectiveMerger{key: "other-merge"}, nil
|
return effectiveMerger{key: "other-merge"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterNormalizer(catalog.Normalizers, pipeline.ModuleSpec{Key: pipeline.DefaultNormalizeModule, Stage: pipeline.StageNormalize, ArtifactKind: effectiveArtifactKind, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer[effectiveArtifact], error) {
|
if err := pipeline.RegisterNormalizer(catalog.Normalizers, pipeline.ModuleSpec{Key: pipeline.DefaultNormalizeModule, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer[effectiveArtifact], error) {
|
||||||
return effectiveNormalizer{key: pipeline.DefaultNormalizeModule}, nil
|
return effectiveNormalizer{key: pipeline.DefaultNormalizeModule}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := catalog.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: pipeline.DefaultOutputModule, Stage: pipeline.StageOutput, Requires: []string{"normalized"}}, func() (contracts.OutputEncoder, error) {
|
if err := catalog.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: pipeline.DefaultOutputModule, Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}}, func() (contracts.OutputEncoder, error) {
|
||||||
return effectiveOutput{key: pipeline.DefaultOutputModule}, nil
|
return effectiveOutput{key: pipeline.DefaultOutputModule}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
|
|||||||
@@ -23,26 +23,34 @@ type FileConfig struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type FilePromptKitConfig struct {
|
type FilePromptKitConfig struct {
|
||||||
ProfileDir *string `yaml:"profile_dir,omitempty"`
|
ProfileDir *string `yaml:"profile_dir,omitempty"`
|
||||||
ProfileFile *string `yaml:"profile_file,omitempty"`
|
ProfileFile *string `yaml:"profile_file,omitempty"`
|
||||||
|
LocalBackend *FilePromptKitLocalBackendConfig `yaml:"local_backend,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type FilePromptKitLocalBackendConfig struct {
|
||||||
|
Endpoint *string `yaml:"endpoint,omitempty"`
|
||||||
|
ConcurrencyLimit *int `yaml:"concurrency_limit,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type FilePipelineProfile struct {
|
type FilePipelineProfile struct {
|
||||||
Input fileModuleBinding `yaml:"input"`
|
LLMProfile *string `yaml:"llm_profile,omitempty"`
|
||||||
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
|
Input fileModuleBinding `yaml:"input"`
|
||||||
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
|
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
|
||||||
Steps []FilePipelineStepProfile `yaml:"steps,omitempty"`
|
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
|
||||||
Output *fileModuleBinding `yaml:"output,omitempty"`
|
Steps []FilePipelineStepProfile `yaml:"steps,omitempty"`
|
||||||
References map[string]fileReferenceSource `yaml:"references,omitempty"`
|
Output *fileModuleBinding `yaml:"output,omitempty"`
|
||||||
artifactsSet bool `yaml:"-"`
|
References map[string]fileReferenceSource `yaml:"references,omitempty"`
|
||||||
stepsSet bool `yaml:"-"`
|
artifactsSet bool `yaml:"-"`
|
||||||
|
stepsSet bool `yaml:"-"`
|
||||||
|
llmProfileSet bool `yaml:"-"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
|
func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
|
||||||
type plainFilePipelineProfile FilePipelineProfile
|
type plainFilePipelineProfile FilePipelineProfile
|
||||||
var decoded plainFilePipelineProfile
|
var decoded plainFilePipelineProfile
|
||||||
seen, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
|
seen, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
|
||||||
"input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
|
"llm_profile": {}, "input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
|
||||||
}, "pipeline profile")
|
}, "pipeline profile")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -50,6 +58,7 @@ func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
|
|||||||
*p = FilePipelineProfile(decoded)
|
*p = FilePipelineProfile(decoded)
|
||||||
_, p.artifactsSet = seen["artifacts"]
|
_, p.artifactsSet = seen["artifacts"]
|
||||||
_, p.stepsSet = seen["steps"]
|
_, p.stepsSet = seen["steps"]
|
||||||
|
_, p.llmProfileSet = seen["llm_profile"]
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -251,6 +260,9 @@ func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
b.LLMProfile = strings.TrimSpace(llmProfile)
|
b.LLMProfile = strings.TrimSpace(llmProfile)
|
||||||
|
if b.LLMProfile == "" {
|
||||||
|
return fmt.Errorf("llm_profile must not be empty when set")
|
||||||
|
}
|
||||||
case "retries":
|
case "retries":
|
||||||
var retries int
|
var retries int
|
||||||
if err := valueNode.Decode(&retries); err != nil {
|
if err := valueNode.Decode(&retries); err != nil {
|
||||||
@@ -468,10 +480,31 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
|||||||
}
|
}
|
||||||
c.PromptKit.ProfileFile = value
|
c.PromptKit.ProfileFile = value
|
||||||
}
|
}
|
||||||
|
if fileCfg.PromptKit.LocalBackend != nil {
|
||||||
|
if fileCfg.PromptKit.LocalBackend.Endpoint == nil {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
|
||||||
|
}
|
||||||
|
endpoint := strings.TrimSpace(*fileCfg.PromptKit.LocalBackend.Endpoint)
|
||||||
|
if endpoint == "" {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
|
||||||
|
}
|
||||||
|
localBackend := PromptKitLocalBackendConfig{Endpoint: endpoint}
|
||||||
|
if fileCfg.PromptKit.LocalBackend.ConcurrencyLimit != nil {
|
||||||
|
localBackend.ConcurrencyLimit = *fileCfg.PromptKit.LocalBackend.ConcurrencyLimit
|
||||||
|
}
|
||||||
|
c.PromptKit.LocalBackend = &localBackend
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, pipelineID := range pipelineIDs {
|
for _, pipelineID := range pipelineIDs {
|
||||||
filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]]
|
filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]]
|
||||||
|
llmProfile := ""
|
||||||
|
if filePipeline.llmProfileSet || filePipeline.LLMProfile != nil {
|
||||||
|
if filePipeline.LLMProfile == nil || strings.TrimSpace(*filePipeline.LLMProfile) == "" {
|
||||||
|
return fmt.Errorf("pipeline %q llm_profile must not be empty when set", pipelineID)
|
||||||
|
}
|
||||||
|
llmProfile = strings.TrimSpace(*filePipeline.LLMProfile)
|
||||||
|
}
|
||||||
hasSteps := filePipeline.stepsSet || filePipeline.Steps != nil
|
hasSteps := filePipeline.stepsSet || filePipeline.Steps != nil
|
||||||
laneIDs, rawLaneIDs, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID))
|
laneIDs, rawLaneIDs, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -479,6 +512,7 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
|||||||
}
|
}
|
||||||
profile := pipeline.PipelineProfile{
|
profile := pipeline.PipelineProfile{
|
||||||
ID: pipelineID,
|
ID: pipelineID,
|
||||||
|
LLMProfile: llmProfile,
|
||||||
Input: filePipeline.Input.toPipelineBinding(),
|
Input: filePipeline.Input.toPipelineBinding(),
|
||||||
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
|
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
|
||||||
References: fileReferenceSourcesToPipeline(filePipeline.References),
|
References: fileReferenceSourcesToPipeline(filePipeline.References),
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package config
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"reflect"
|
"reflect"
|
||||||
@@ -49,6 +50,93 @@ func TestFileConfigMinimalVersion4AppliesOverDefaults(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFilePipelineLLMProfileIsPresenceAwareAndDetached(t *testing.T) {
|
||||||
|
const pipelineYAML = `version: 4
|
||||||
|
pipelines:
|
||||||
|
main:
|
||||||
|
%s
|
||||||
|
input: input
|
||||||
|
artifacts:
|
||||||
|
lane:
|
||||||
|
extract: extract
|
||||||
|
`
|
||||||
|
|
||||||
|
t.Run("omitted", func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, ""))
|
||||||
|
if file.Pipelines["main"].LLMProfile != nil || file.Pipelines["main"].llmProfileSet {
|
||||||
|
t.Fatalf("parsed pipeline profile = %#v, want omitted llm profile", file.Pipelines["main"])
|
||||||
|
}
|
||||||
|
cfg := Default()
|
||||||
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got := cfg.Pipelines["main"].LLMProfile; got != "" {
|
||||||
|
t.Fatalf("pipeline llm profile = %q, want empty", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("trimmed and detached", func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, "llm_profile: ' configured-profile '"))
|
||||||
|
cfg := Default()
|
||||||
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got := cfg.Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("pipeline llm profile = %q, want trimmed value", got)
|
||||||
|
}
|
||||||
|
*file.Pipelines["main"].LLMProfile = "changed-profile"
|
||||||
|
if got := cfg.Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("effective config aliases parsed file: %q", got)
|
||||||
|
}
|
||||||
|
if got := cloneConfig(cfg).Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("cloned pipeline llm profile = %q", got)
|
||||||
|
}
|
||||||
|
data, err := json.Marshal(cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var roundTripped Config
|
||||||
|
if err := json.Unmarshal(data, &roundTripped); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got := roundTripped.Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("round-tripped pipeline llm profile = %q", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
for _, value := range []string{"''", "' '", "null"} {
|
||||||
|
t.Run("explicit empty "+value, func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, "llm_profile: "+value))
|
||||||
|
cfg := Default()
|
||||||
|
err := cfg.ApplyFileConfig(file)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), `pipeline "main" llm_profile must not be empty`) {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v, want explicit-empty rejection", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFileModuleBindingRejectsExplicitEmptyLLMProfile(t *testing.T) {
|
||||||
|
const configYAML = `version: 4
|
||||||
|
pipelines:
|
||||||
|
main:
|
||||||
|
input:
|
||||||
|
module: input
|
||||||
|
llm_profile: %s
|
||||||
|
artifacts:
|
||||||
|
lane:
|
||||||
|
extract: extract
|
||||||
|
`
|
||||||
|
for _, value := range []string{"''", "' '", "null"} {
|
||||||
|
t.Run(value, func(t *testing.T) {
|
||||||
|
_, err := ParseFileConfigYAML([]byte(fmt.Sprintf(configYAML, value)))
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "llm_profile must not be empty when set") {
|
||||||
|
t.Fatalf("ParseFileConfigYAML() error = %v, want explicit-empty binding profile rejection", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestFilePromptKitProfileSourcesSurviveConfigBoundaries(t *testing.T) {
|
func TestFilePromptKitProfileSourcesSurviveConfigBoundaries(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -97,6 +185,100 @@ func TestFilePromptKitProfileSourcesSurviveConfigBoundaries(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitLocalBackendSurvivesConfigBoundaries(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
concurrencyYAML string
|
||||||
|
wantConcurrency int
|
||||||
|
}{
|
||||||
|
{name: "omitted concurrency defaults to zero"},
|
||||||
|
{name: "positive concurrency is preserved", concurrencyYAML: " concurrency_limit: 2\n", wantConcurrency: 2},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, "version: 4\npromptkit:\n local_backend:\n endpoint: ' http://localhost:8000/v1 '\n"+tt.concurrencyYAML)
|
||||||
|
cfg := Default()
|
||||||
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v", err)
|
||||||
|
}
|
||||||
|
want := PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: "http://localhost:8000/v1",
|
||||||
|
ConcurrencyLimit: tt.wantConcurrency,
|
||||||
|
}
|
||||||
|
if cfg.PromptKit.LocalBackend == nil || *cfg.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("local backend config = %#v, want %#v", cfg.PromptKit.LocalBackend, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
*file.PromptKit.LocalBackend.Endpoint = "http://changed.example/v1"
|
||||||
|
if file.PromptKit.LocalBackend.ConcurrencyLimit != nil {
|
||||||
|
*file.PromptKit.LocalBackend.ConcurrencyLimit = 99
|
||||||
|
}
|
||||||
|
if *cfg.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("effective config aliases parsed file model: %#v", cfg.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
|
||||||
|
cloned := cloneConfig(cfg)
|
||||||
|
if cloned.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *cloned.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("cloned local backend = %#v, want detached %#v", cloned.PromptKit.LocalBackend, want)
|
||||||
|
}
|
||||||
|
cloned.PromptKit.LocalBackend.Endpoint = "http://clone.example/v1"
|
||||||
|
if *cfg.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("mutating clone changed source config: %#v", cfg.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
|
||||||
|
redacted := cfg.Redacted()
|
||||||
|
if redacted.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *redacted.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("redacted local backend = %#v, want detached %#v", redacted.PromptKit.LocalBackend, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := json.Marshal(cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("json.Marshal() error = %v", err)
|
||||||
|
}
|
||||||
|
var payload struct {
|
||||||
|
PromptKit map[string]json.RawMessage `json:"promptkit"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(data, &payload); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal() error = %v", err)
|
||||||
|
}
|
||||||
|
localJSON, ok := payload.PromptKit["local_backend"]
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("runtime PromptKit JSON keys = %v, want local_backend", payload.PromptKit)
|
||||||
|
}
|
||||||
|
var localPayload map[string]json.RawMessage
|
||||||
|
if err := json.Unmarshal(localJSON, &localPayload); err != nil {
|
||||||
|
t.Fatalf("unmarshal local_backend JSON: %v", err)
|
||||||
|
}
|
||||||
|
if _, ok := localPayload["endpoint"]; !ok {
|
||||||
|
t.Fatalf("runtime local_backend JSON keys = %v, want endpoint", localPayload)
|
||||||
|
}
|
||||||
|
if _, ok := localPayload["concurrency_limit"]; !ok {
|
||||||
|
t.Fatalf("runtime local_backend JSON keys = %v, want concurrency_limit", localPayload)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitLocalBackendRequiresEndpoint(t *testing.T) {
|
||||||
|
for _, tt := range []struct {
|
||||||
|
name string
|
||||||
|
yaml string
|
||||||
|
}{
|
||||||
|
{name: "missing", yaml: "version: 4\npromptkit:\n local_backend: {}\n"},
|
||||||
|
{name: "empty", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ''\n"},
|
||||||
|
{name: "blank", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ' '\n"},
|
||||||
|
} {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, tt.yaml)
|
||||||
|
cfg := Default()
|
||||||
|
err := cfg.ApplyFileConfig(file)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "promptkit.local_backend.endpoint") {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v, want endpoint field context", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestFilePromptKitExplicitEmptyProfileSourcesAreRejected(t *testing.T) {
|
func TestFilePromptKitExplicitEmptyProfileSourcesAreRejected(t *testing.T) {
|
||||||
for _, field := range []string{"profile_dir", "profile_file"} {
|
for _, field := range []string{"profile_dir", "profile_file"} {
|
||||||
t.Run(field, func(t *testing.T) {
|
t.Run(field, func(t *testing.T) {
|
||||||
@@ -188,6 +370,11 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
|||||||
yaml: "version: 4\ncache:\n checkpoints:\n enabled: definitely\n",
|
yaml: "version: 4\ncache:\n checkpoints:\n enabled: definitely\n",
|
||||||
want: "cannot unmarshal",
|
want: "cannot unmarshal",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: "local backend field",
|
||||||
|
yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: http://localhost:8000/v1\n unknown: true\n",
|
||||||
|
want: "field unknown not found",
|
||||||
|
},
|
||||||
}
|
}
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
|||||||
@@ -19,23 +19,28 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
resolved := pipeline.ResolvedPipeline{
|
resolved := pipeline.ResolvedPipeline{
|
||||||
ID: "redaction-test",
|
ID: "redaction-test",
|
||||||
Digest: "sha256:safe-digest",
|
Digest: "sha256:safe-digest",
|
||||||
Input: bindings["input"],
|
Input: bindings["input"],
|
||||||
Chunk: bindings["chunk"],
|
InputExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
ChunkReferences: redactionTestReferenceTarget(pipeline.StageChunk, "", "chunk-reference-content"),
|
Chunk: bindings["chunk"],
|
||||||
|
ChunkExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
|
ChunkReferences: redactionTestReferenceTarget(pipeline.StageChunk, "", "chunk-reference-content"),
|
||||||
Steps: []pipeline.ResolvedPipelineStep{{
|
Steps: []pipeline.ResolvedPipelineStep{{
|
||||||
ID: "default",
|
ID: "default",
|
||||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
||||||
ID: "safe-lane",
|
ID: "safe-lane",
|
||||||
ArtifactKind: "safe/artifact",
|
ArtifactKind: "safe/artifact",
|
||||||
Extract: bindings["extract"],
|
Extract: bindings["extract"],
|
||||||
Merge: bindings["merge"],
|
ExtractExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Normalize: bindings["normalize"],
|
Merge: bindings["merge"],
|
||||||
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
|
MergeExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"),
|
Normalize: bindings["normalize"],
|
||||||
MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"),
|
NormalizeExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
NormalizeReferences: redactionTestReferenceTarget(pipeline.StageNormalize, "safe-lane", "normalize-reference-content"),
|
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
|
||||||
|
ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"),
|
||||||
|
MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"),
|
||||||
|
NormalizeReferences: redactionTestReferenceTarget(pipeline.StageNormalize, "safe-lane", "normalize-reference-content"),
|
||||||
}},
|
}},
|
||||||
}},
|
}},
|
||||||
ValidatorChains: []pipeline.ResolvedValidatorChain{{
|
ValidatorChains: []pipeline.ResolvedValidatorChain{{
|
||||||
@@ -49,7 +54,8 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
|
|||||||
ArtifactKind: "safe/artifact",
|
ArtifactKind: "safe/artifact",
|
||||||
}},
|
}},
|
||||||
}},
|
}},
|
||||||
Output: bindings["output"],
|
Output: bindings["output"],
|
||||||
|
OutputExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
}
|
}
|
||||||
effective := EffectiveConfig{
|
effective := EffectiveConfig{
|
||||||
Config: Config{Pipelines: map[string]pipeline.PipelineProfile{
|
Config: Config{Pipelines: map[string]pipeline.PipelineProfile{
|
||||||
@@ -88,6 +94,11 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
|
|||||||
t.Fatalf("resolved pipeline summary does not retain %q: %s", safe, text)
|
t.Fatalf("resolved pipeline summary does not retain %q: %s", safe, text)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
for _, executionClass := range []string{"input_execution_class\":\"deterministic", "chunk_execution_class\":\"llm_backed", "extract_execution_class\":\"llm_backed", "merge_execution_class\":\"deterministic", "normalize_execution_class\":\"llm_backed", "output_execution_class\":\"deterministic"} {
|
||||||
|
if !strings.Contains(text, executionClass) {
|
||||||
|
t.Fatalf("resolved pipeline summary does not retain %q: %s", executionClass, text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
payload.Input.Options["safe"] = "mutated"
|
payload.Input.Options["safe"] = "mutated"
|
||||||
nested := payload.Input.Options["nested"].([]any)[0].([]any)[0].(map[string]any)
|
nested := payload.Input.Options["nested"].([]any)[0].([]any)[0].(map[string]any)
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package config
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"net/url"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
@@ -53,6 +54,27 @@ func validatePromptKit(cfg PromptKitConfig) error {
|
|||||||
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
||||||
return fmt.Errorf("promptkit profile_dir and profile_file are mutually exclusive")
|
return fmt.Errorf("promptkit profile_dir and profile_file are mutually exclusive")
|
||||||
}
|
}
|
||||||
|
if cfg.LocalBackend == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
endpoint := strings.TrimSpace(cfg.LocalBackend.Endpoint)
|
||||||
|
if endpoint == "" {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
|
||||||
|
}
|
||||||
|
parsed, err := url.Parse(endpoint)
|
||||||
|
if err != nil ||
|
||||||
|
(!strings.EqualFold(parsed.Scheme, "http") && !strings.EqualFold(parsed.Scheme, "https")) ||
|
||||||
|
!parsed.IsAbs() ||
|
||||||
|
parsed.Hostname() == "" ||
|
||||||
|
parsed.User != nil ||
|
||||||
|
parsed.RawQuery != "" ||
|
||||||
|
parsed.ForceQuery ||
|
||||||
|
strings.Contains(endpoint, "#") {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must be an absolute HTTP or HTTPS URL with a host and no user information, query, or fragment")
|
||||||
|
}
|
||||||
|
if cfg.LocalBackend.ConcurrencyLimit < 0 {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.concurrency_limit must not be negative")
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -93,6 +93,73 @@ func TestValidatePromptKitSourcesAreMutuallyExclusive(t *testing.T) {
|
|||||||
assertValidationContains(t, cfg, "promptkit profile_dir and profile_file are mutually exclusive")
|
assertValidationContains(t, cfg, "promptkit profile_dir and profile_file are mutually exclusive")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestValidatePromptKitLocalBackendEndpoints(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
endpoint string
|
||||||
|
profileSource PromptKitConfig
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "HTTP endpoint with path and profile directory",
|
||||||
|
endpoint: "http://localhost:8000/v1",
|
||||||
|
profileSource: PromptKitConfig{ProfileDir: "./profiles"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "case-insensitive HTTPS endpoint and profile file",
|
||||||
|
endpoint: "HTTPS://inference.example.test/api",
|
||||||
|
profileSource: PromptKitConfig{ProfileFile: "./profiles.yml"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := Default()
|
||||||
|
cfg.PromptKit = tt.profileSource
|
||||||
|
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: tt.endpoint,
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
if err := cfg.Validate(); err != nil {
|
||||||
|
t.Fatalf("Validate() error = %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatePromptKitLocalBackendRejectsInvalidValues(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
endpoint string
|
||||||
|
concurrencyLimit int
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "blank endpoint", endpoint: " ", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "relative URL", endpoint: "localhost:8000/v1", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "unsupported scheme", endpoint: "ftp://localhost/model", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "missing host", endpoint: "http:///v1", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "user information", endpoint: "http://user:secret@localhost/v1", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "query", endpoint: "http://localhost/v1?model=example", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "empty query", endpoint: "http://localhost/v1?", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "fragment", endpoint: "http://localhost/v1#model", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "empty fragment", endpoint: "http://localhost/v1#", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{
|
||||||
|
name: "negative concurrency",
|
||||||
|
endpoint: "http://localhost:8000/v1",
|
||||||
|
concurrencyLimit: -1,
|
||||||
|
want: "promptkit.local_backend.concurrency_limit",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := Default()
|
||||||
|
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: tt.endpoint,
|
||||||
|
ConcurrencyLimit: tt.concurrencyLimit,
|
||||||
|
}
|
||||||
|
assertValidationContains(t, cfg, tt.want)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestValidateStateSurfaceRules(t *testing.T) {
|
func TestValidateStateSurfaceRules(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
|
|||||||
@@ -267,10 +267,6 @@ const (
|
|||||||
ExecutionClassLLMBacked ExecutionClass = "llm_backed"
|
ExecutionClassLLMBacked ExecutionClass = "llm_backed"
|
||||||
)
|
)
|
||||||
|
|
||||||
type ChunkExecutionClassProvider interface {
|
|
||||||
ExecutionClass() ExecutionClass
|
|
||||||
}
|
|
||||||
|
|
||||||
type ValidationResult struct {
|
type ValidationResult struct {
|
||||||
Approved bool `json:"approved"`
|
Approved bool `json:"approved"`
|
||||||
ReasonCode string `json:"reason_code,omitempty"`
|
ReasonCode string `json:"reason_code,omitempty"`
|
||||||
|
|||||||
@@ -21,8 +21,9 @@ type AssetSource struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type AssetRegistry struct {
|
type AssetRegistry struct {
|
||||||
prompts []AssetSource
|
prompts []AssetSource
|
||||||
schemas []AssetSource
|
schemas []AssetSource
|
||||||
|
fallbackProfiles []AssetSource
|
||||||
}
|
}
|
||||||
|
|
||||||
type AssetHashPart struct {
|
type AssetHashPart struct {
|
||||||
@@ -58,6 +59,20 @@ func (r *AssetRegistry) RegisterSchemaFS(fsys fs.FS, root string) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RegisterFallbackProfileFS registers profile assets that PromptKit uses only
|
||||||
|
// when an operator-configured source does not provide a matching profile.
|
||||||
|
func (r *AssetRegistry) RegisterFallbackProfileFS(fsys fs.FS, root string) error {
|
||||||
|
if r == nil {
|
||||||
|
return fmt.Errorf("asset registry must not be nil")
|
||||||
|
}
|
||||||
|
source, err := newAssetSource(fsys, root)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("register fallback profile assets: %w", err)
|
||||||
|
}
|
||||||
|
r.fallbackProfiles = append(r.fallbackProfiles, source)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func (r *AssetRegistry) PromptFS() (fs.FS, error) {
|
func (r *AssetRegistry) PromptFS() (fs.FS, error) {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return nil, fmt.Errorf("asset registry must not be nil")
|
return nil, fmt.Errorf("asset registry must not be nil")
|
||||||
@@ -72,19 +87,76 @@ func (r *AssetRegistry) SchemaFS() (fs.FS, error) {
|
|||||||
return flattenAssetSources(r.schemas)
|
return flattenAssetSources(r.schemas)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (r *AssetRegistry) FallbackProfileFS() (fs.FS, error) {
|
||||||
|
if r == nil {
|
||||||
|
return nil, fmt.Errorf("asset registry must not be nil")
|
||||||
|
}
|
||||||
|
return flattenAssetSources(r.fallbackProfiles)
|
||||||
|
}
|
||||||
|
|
||||||
|
// FallbackProfileDigest returns a deterministic, non-secret identity for the
|
||||||
|
// flattened fallback profile assets.
|
||||||
|
func (r *AssetRegistry) FallbackProfileDigest() (string, error) {
|
||||||
|
_, digest, _, err := r.fallbackProfileAssets()
|
||||||
|
return digest, err
|
||||||
|
}
|
||||||
|
|
||||||
func (r *AssetRegistry) PromptKitOptions() ([]promptkit.Option, error) {
|
func (r *AssetRegistry) PromptKitOptions() ([]promptkit.Option, error) {
|
||||||
|
options, _, err := r.promptKitOptions()
|
||||||
|
return options, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *AssetRegistry) promptKitOptions() ([]promptkit.Option, string, error) {
|
||||||
promptFS, err := r.PromptFS()
|
promptFS, err := r.PromptFS()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("prepare prompt assets: %w", err)
|
return nil, "", fmt.Errorf("prepare prompt assets: %w", err)
|
||||||
}
|
}
|
||||||
schemaFS, err := r.SchemaFS()
|
schemaFS, err := r.SchemaFS()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("prepare schema assets: %w", err)
|
return nil, "", fmt.Errorf("prepare schema assets: %w", err)
|
||||||
}
|
}
|
||||||
return []promptkit.Option{
|
options := []promptkit.Option{
|
||||||
promptkit.WithPromptFS(promptFS, "."),
|
promptkit.WithPromptFS(promptFS, "."),
|
||||||
promptkit.WithSchemaFS(schemaFS, "."),
|
promptkit.WithSchemaFS(schemaFS, "."),
|
||||||
}, nil
|
}
|
||||||
|
fallbackFS, fallbackDigest, hasFallback, err := r.fallbackProfileAssets()
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", err
|
||||||
|
}
|
||||||
|
if hasFallback {
|
||||||
|
options = append(options, promptkit.WithFallbackProfileFS(fallbackFS, "."))
|
||||||
|
}
|
||||||
|
return options, fallbackDigest, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *AssetRegistry) promptKitFallbackProfileOption() (promptkit.Option, bool, error) {
|
||||||
|
fallbackFS, _, hasFallback, err := r.fallbackProfileAssets()
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
if !hasFallback {
|
||||||
|
return nil, false, nil
|
||||||
|
}
|
||||||
|
return promptkit.WithFallbackProfileFS(fallbackFS, "."), true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *AssetRegistry) fallbackProfileAssets() (fs.FS, string, bool, error) {
|
||||||
|
if r == nil {
|
||||||
|
return nil, "", false, fmt.Errorf("asset registry must not be nil")
|
||||||
|
}
|
||||||
|
if len(r.fallbackProfiles) == 0 {
|
||||||
|
empty := sha256.Sum256([]byte("notarius:fallback-profile-assets:empty"))
|
||||||
|
return nil, "sha256:" + hex.EncodeToString(empty[:]), false, nil
|
||||||
|
}
|
||||||
|
fallbackFS, err := r.FallbackProfileFS()
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", false, fmt.Errorf("prepare fallback profile assets: %w", err)
|
||||||
|
}
|
||||||
|
digest, err := hashAssetFilesystem(fallbackFS)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", false, err
|
||||||
|
}
|
||||||
|
return fallbackFS, digest, true, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func HashAssets(parts []AssetHashPart) (string, error) {
|
func HashAssets(parts []AssetHashPart) (string, error) {
|
||||||
@@ -117,6 +189,28 @@ func HashAssets(parts []AssetHashPart) (string, error) {
|
|||||||
return "sha256:" + hex.EncodeToString(hash.Sum(nil)), nil
|
return "sha256:" + hex.EncodeToString(hash.Sum(nil)), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func hashAssetFilesystem(fsys fs.FS) (string, error) {
|
||||||
|
var parts []AssetHashPart
|
||||||
|
err := fs.WalkDir(fsys, ".", func(name string, entry fs.DirEntry, walkErr error) error {
|
||||||
|
if walkErr != nil {
|
||||||
|
return walkErr
|
||||||
|
}
|
||||||
|
if entry.IsDir() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
parts = append(parts, AssetHashPart{FS: fsys, Path: name})
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("walk assets for digest: %w", err)
|
||||||
|
}
|
||||||
|
if len(parts) > 0 {
|
||||||
|
return HashAssets(parts)
|
||||||
|
}
|
||||||
|
empty := sha256.Sum256([]byte("notarius:fallback-profile-assets:empty"))
|
||||||
|
return "sha256:" + hex.EncodeToString(empty[:]), nil
|
||||||
|
}
|
||||||
|
|
||||||
func newAssetSource(fsys fs.FS, root string) (AssetSource, error) {
|
func newAssetSource(fsys fs.FS, root string) (AssetSource, error) {
|
||||||
if fsys == nil {
|
if fsys == nil {
|
||||||
return AssetSource{}, fmt.Errorf("filesystem must not be nil")
|
return AssetSource{}, fmt.Errorf("filesystem must not be nil")
|
||||||
@@ -248,6 +342,9 @@ func (m assetMapFS) dirEntries(dir string) []fs.DirEntry {
|
|||||||
children[childName] = entry
|
children[childName] = entry
|
||||||
}
|
}
|
||||||
if len(children) == 0 {
|
if len(children) == 0 {
|
||||||
|
if dir == "." {
|
||||||
|
return []fs.DirEntry{}
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
names := make([]string, 0, len(children))
|
names := make([]string, 0, len(children))
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package llm
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"io/fs"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"testing/fstest"
|
"testing/fstest"
|
||||||
@@ -98,6 +99,89 @@ func TestAssetRegistryRejectsDuplicateAssetPaths(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestAssetRegistryCombinesFallbackProfileSources(t *testing.T) {
|
||||||
|
registry := NewAssetRegistry()
|
||||||
|
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
|
||||||
|
"first/profiles/one.yaml": {Data: []byte("id: one\nmodel: first\n")},
|
||||||
|
}, "first/profiles"); err != nil {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
|
||||||
|
"second/two.yaml": {Data: []byte("id: two\nmodel: second\n")},
|
||||||
|
}, "second"); err != nil {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
fallbackFS, err := registry.FallbackProfileFS()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("FallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
for _, name := range []string{"one.yaml", "two.yaml"} {
|
||||||
|
if _, err := fs.ReadFile(fallbackFS, name); err != nil {
|
||||||
|
t.Fatalf("FallbackProfileFS().ReadFile(%q) error = %v, want nil", name, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAssetRegistryRejectsInvalidFallbackProfileRoot(t *testing.T) {
|
||||||
|
registry := NewAssetRegistry()
|
||||||
|
err := registry.RegisterFallbackProfileFS(fstest.MapFS{}, "../profiles")
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "invalid path") {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want invalid root error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAssetRegistryRejectsUnreadableFallbackProfileAssets(t *testing.T) {
|
||||||
|
registry := NewAssetRegistry()
|
||||||
|
if err := registry.RegisterFallbackProfileFS(unreadableAssetFS{}, "."); err != nil {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
_, err := registry.FallbackProfileFS()
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "permission denied") {
|
||||||
|
t.Fatalf("FallbackProfileFS() error = %v, want unreadable asset error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAssetRegistryRejectsDuplicateFallbackProfilePaths(t *testing.T) {
|
||||||
|
registry := NewAssetRegistry()
|
||||||
|
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{"first/profile.yaml": {Data: []byte("id: first\n")}}, "first"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{"second/profile.yaml": {Data: []byte("id: second\n")}}, "second"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
_, err := registry.FallbackProfileFS()
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "duplicate asset path") {
|
||||||
|
t.Fatalf("FallbackProfileFS() error = %v, want duplicate path error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAssetRegistryFallbackProfileDigestTracksContentWithoutLeakingIt(t *testing.T) {
|
||||||
|
digestFor := func(content string) string {
|
||||||
|
t.Helper()
|
||||||
|
registry := NewAssetRegistry()
|
||||||
|
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
|
||||||
|
"profiles/profile.yaml": {Data: []byte(content)},
|
||||||
|
}, "profiles"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
digest, err := registry.FallbackProfileDigest()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return digest
|
||||||
|
}
|
||||||
|
|
||||||
|
first := digestFor("id: fallback\nmodel: model-one\n")
|
||||||
|
second := digestFor("id: fallback\nmodel: model-two\n")
|
||||||
|
if first == second {
|
||||||
|
t.Fatalf("fallback profile digests = %q and %q, want content change", first, second)
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(first, "sha256:") || strings.Contains(first, "model-one") || strings.Contains(first, "profile.yaml") {
|
||||||
|
t.Fatalf("fallback profile digest leaked source details: %q", first)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestAssetRegistryCombinesNamespacedPromptSources(t *testing.T) {
|
func TestAssetRegistryCombinesNamespacedPromptSources(t *testing.T) {
|
||||||
registry := NewAssetRegistry()
|
registry := NewAssetRegistry()
|
||||||
mustRegisterPromptFS(t, registry, fstest.MapFS{
|
mustRegisterPromptFS(t, registry, fstest.MapFS{
|
||||||
@@ -196,3 +280,9 @@ output:
|
|||||||
repair_attempts: 0
|
repair_attempts: 0
|
||||||
`
|
`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type unreadableAssetFS struct{}
|
||||||
|
|
||||||
|
func (unreadableAssetFS) Open(name string) (fs.File, error) {
|
||||||
|
return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrPermission}
|
||||||
|
}
|
||||||
|
|||||||
@@ -19,9 +19,15 @@ import (
|
|||||||
|
|
||||||
const promptKitProviderName = "promptkit"
|
const promptKitProviderName = "promptkit"
|
||||||
|
|
||||||
|
type PromptKitLocalBackendConfig struct {
|
||||||
|
Endpoint string
|
||||||
|
ConcurrencyLimit int
|
||||||
|
}
|
||||||
|
|
||||||
type PromptKitClientConfig struct {
|
type PromptKitClientConfig struct {
|
||||||
ProfileDir string
|
ProfileDir string
|
||||||
ProfileFile string
|
ProfileFile string
|
||||||
|
LocalBackend *PromptKitLocalBackendConfig
|
||||||
Assets *AssetRegistry
|
Assets *AssetRegistry
|
||||||
Timeout time.Duration
|
Timeout time.Duration
|
||||||
HTTPClient *http.Client
|
HTTPClient *http.Client
|
||||||
@@ -31,11 +37,13 @@ type PromptKitClientConfig struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type PromptKitClient struct {
|
type PromptKitClient struct {
|
||||||
engine *promptkit.Engine
|
engine *promptkit.Engine
|
||||||
recorder *LLMProfileRecorder
|
recorder *LLMProfileRecorder
|
||||||
profileDir string
|
profileDir string
|
||||||
profileFile string
|
profileFile string
|
||||||
reasoningEffort *string
|
localEndpoint string
|
||||||
|
fallbackProfileDigest string
|
||||||
|
reasoningEffort *string
|
||||||
}
|
}
|
||||||
|
|
||||||
type LLMProfileRecorder struct {
|
type LLMProfileRecorder struct {
|
||||||
@@ -46,24 +54,31 @@ type LLMProfileRecorder struct {
|
|||||||
var _ contracts.StructuredLLMClient = (*PromptKitClient)(nil)
|
var _ contracts.StructuredLLMClient = (*PromptKitClient)(nil)
|
||||||
var _ contracts.LLMProfileManifestProvider = (*PromptKitClient)(nil)
|
var _ contracts.LLMProfileManifestProvider = (*PromptKitClient)(nil)
|
||||||
|
|
||||||
|
func PromptKitLocalBackendOption(cfg PromptKitLocalBackendConfig) promptkit.Option {
|
||||||
|
return promptkit.WithBackend(promptkit.LocalBackend(cfg.Endpoint, cfg.ConcurrencyLimit))
|
||||||
|
}
|
||||||
|
|
||||||
func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
||||||
if cfg.Assets == nil {
|
if cfg.Assets == nil {
|
||||||
return nil, fmt.Errorf("PromptKit client assets must not be nil")
|
return nil, fmt.Errorf("PromptKit client assets must not be nil")
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
profileSource, profileOptions, err := promptKitProfileSourceEngineOptions(PromptKitProfileSourceConfig{
|
||||||
return nil, fmt.Errorf("PromptKit profile_dir and profile_file are mutually exclusive")
|
ProfileDir: cfg.ProfileDir,
|
||||||
}
|
ProfileFile: cfg.ProfileFile,
|
||||||
options, err := cfg.Assets.PromptKitOptions()
|
LocalBackend: cfg.LocalBackend,
|
||||||
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
if profileFile := strings.TrimSpace(cfg.ProfileFile); profileFile != "" {
|
options, fallbackProfileDigest, err := cfg.Assets.promptKitOptions()
|
||||||
options = append(options, promptkit.WithProfileFile(profileFile))
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
}
|
}
|
||||||
|
options = append(options, profileOptions...)
|
||||||
options = append(options, cfg.EngineOptions...)
|
options = append(options, cfg.EngineOptions...)
|
||||||
|
|
||||||
engine, err := promptkit.NewEngine(promptkit.Config{
|
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||||
ProfileDir: strings.TrimSpace(cfg.ProfileDir),
|
ProfileDir: profileSource.ProfileDir,
|
||||||
Timeout: cfg.Timeout,
|
Timeout: cfg.Timeout,
|
||||||
HTTPClient: cfg.HTTPClient,
|
HTTPClient: cfg.HTTPClient,
|
||||||
}, options...)
|
}, options...)
|
||||||
@@ -80,11 +95,13 @@ func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
|||||||
reasoningEffort = &value
|
reasoningEffort = &value
|
||||||
}
|
}
|
||||||
return &PromptKitClient{
|
return &PromptKitClient{
|
||||||
engine: engine,
|
engine: engine,
|
||||||
recorder: recorder,
|
recorder: recorder,
|
||||||
profileDir: strings.TrimSpace(cfg.ProfileDir),
|
profileDir: profileSource.ProfileDir,
|
||||||
profileFile: strings.TrimSpace(cfg.ProfileFile),
|
profileFile: profileSource.ProfileFile,
|
||||||
reasoningEffort: reasoningEffort,
|
localEndpoint: profileSource.localEndpoint(),
|
||||||
|
fallbackProfileDigest: fallbackProfileDigest,
|
||||||
|
reasoningEffort: reasoningEffort,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -120,19 +137,31 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
|
|||||||
Vars: promptKitVars(req, sessionID),
|
Vars: promptKitVars(req, sessionID),
|
||||||
Execution: execution,
|
Execution: execution,
|
||||||
}
|
}
|
||||||
prepared, err := c.engine.Prepare(ctx, runReq)
|
prepared, err := c.engine.PrepareExecution(ctx, runReq)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, ctxErr
|
return contracts.StructuredCompletionResponse{}, ctxErr
|
||||||
}
|
}
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("prepare PromptKit prompt %q: %w", promptID, redactPromptKitError(err))
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("prepare PromptKit prompt %q: %w", promptID, redactPromptKitError(err))
|
||||||
}
|
}
|
||||||
result, err := c.engine.Run(ctx, runReq)
|
defer prepared.Discard()
|
||||||
|
preparedDetails := prepared.Details()
|
||||||
|
result, err := c.engine.RunPrepared(ctx, prepared)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, ctxErr
|
return contracts.StructuredCompletionResponse{}, ctxErr
|
||||||
}
|
}
|
||||||
if errors.Is(err, promptkit.ErrCapacityExceeded) {
|
if errors.Is(err, promptkit.ErrCapacityExceeded) {
|
||||||
|
var capacityErr *promptkit.CapacityError
|
||||||
|
if errors.As(err, &capacityErr) && strings.TrimSpace(capacityErr.BackendID) != "" {
|
||||||
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf(
|
||||||
|
"run PromptKit prompt %q on backend %q: %w: %v",
|
||||||
|
promptID,
|
||||||
|
strings.TrimSpace(capacityErr.BackendID),
|
||||||
|
contracts.ErrLLMCapacityExceeded,
|
||||||
|
redactPromptKitError(err),
|
||||||
|
)
|
||||||
|
}
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf(
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf(
|
||||||
"run PromptKit prompt %q: %w: %v",
|
"run PromptKit prompt %q: %w: %v",
|
||||||
promptID,
|
promptID,
|
||||||
@@ -145,7 +174,7 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
|
|||||||
if result == nil {
|
if result == nil {
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w: empty result", promptID, contracts.ErrInvalidStructuredOutput)
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w: empty result", promptID, contracts.ErrInvalidStructuredOutput)
|
||||||
}
|
}
|
||||||
response := c.responseFromResult(result, prepared)
|
response := c.responseFromResult(result, &preparedDetails)
|
||||||
if result.Validation.Status == promptkit.ValidationFailed || !result.Validation.IsValid {
|
if result.Validation.Status == promptkit.ValidationFailed || !result.Validation.IsValid {
|
||||||
return response, fmt.Errorf("run PromptKit prompt %q: %w: validation failed: %s", promptID, contracts.ErrInvalidStructuredOutput, strings.Join(result.Validation.Errors, "; "))
|
return response, fmt.Errorf("run PromptKit prompt %q: %w: validation failed: %s", promptID, contracts.ErrInvalidStructuredOutput, strings.Join(result.Validation.Errors, "; "))
|
||||||
}
|
}
|
||||||
@@ -298,11 +327,15 @@ func (c *PromptKitClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint,
|
|||||||
if c == nil {
|
if c == nil {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
}
|
}
|
||||||
fingerprint, err := promptKitProfileFingerprint(c.profileDir, c.profileFile)
|
fingerprint, err := promptKitProfileFingerprint(c.profileDir, c.profileFile, c.fallbackProfileDigest)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
return []CheckpointFingerprint{fingerprint}, nil
|
fingerprints := []CheckpointFingerprint{fingerprint}
|
||||||
|
if c.localEndpoint != "" {
|
||||||
|
fingerprints = append(fingerprints, promptKitLocalBackendFingerprint(c.localEndpoint))
|
||||||
|
}
|
||||||
|
return fingerprints, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewLLMProfileRecorder() *LLMProfileRecorder {
|
func NewLLMProfileRecorder() *LLMProfileRecorder {
|
||||||
|
|||||||
@@ -5,7 +5,9 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
|
"io/fs"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"reflect"
|
"reflect"
|
||||||
@@ -113,6 +115,104 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestPromptKitClientUsesOnePreparedSnapshotForDebugAndGeneration(t *testing.T) {
|
||||||
|
const initialPrompt = `id: snapshot.test
|
||||||
|
version: "v1"
|
||||||
|
default_profile: snapshot-profile
|
||||||
|
inputs:
|
||||||
|
- name: transcript
|
||||||
|
required: true
|
||||||
|
content_type: application/json
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "Snapshot A: {{ input \"transcript\" }}"
|
||||||
|
output:
|
||||||
|
format: json
|
||||||
|
validation_mode: json_schema
|
||||||
|
schema_path: adapter.schema.json
|
||||||
|
repair_attempts: 0
|
||||||
|
`
|
||||||
|
const updatedPrompt = `id: snapshot.test
|
||||||
|
version: "v1"
|
||||||
|
default_profile: snapshot-profile
|
||||||
|
inputs:
|
||||||
|
- name: transcript
|
||||||
|
required: true
|
||||||
|
content_type: application/json
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "Snapshot B: {{ input \"transcript\" }}"
|
||||||
|
output:
|
||||||
|
format: json
|
||||||
|
validation_mode: json_schema
|
||||||
|
schema_path: adapter.schema.json
|
||||||
|
repair_attempts: 0
|
||||||
|
`
|
||||||
|
source := &switchingPromptFS{files: fstest.MapFS{
|
||||||
|
"snapshot.test.yaml": {Data: []byte(initialPrompt)},
|
||||||
|
}}
|
||||||
|
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||||
|
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||||
|
Assets: newTestPromptKitAssets(t),
|
||||||
|
EngineOptions: []promptkit.Option{
|
||||||
|
promptkit.WithPromptFS(source, "."),
|
||||||
|
promptkit.WithBackend(promptkit.Backend{ID: "snapshot-backend", Endpoint: "http://promptkit.test/v1"}),
|
||||||
|
promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
|
ID: "snapshot-profile",
|
||||||
|
BackendID: "snapshot-backend",
|
||||||
|
Model: "snapshot-model",
|
||||||
|
})),
|
||||||
|
promptkit.WithLLMClient(fake),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
opened := source.holdNextPromptRead()
|
||||||
|
defer source.resumePromptRead()
|
||||||
|
type completion struct {
|
||||||
|
response contracts.StructuredCompletionResponse
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
completed := make(chan completion, 1)
|
||||||
|
go func() {
|
||||||
|
var out struct {
|
||||||
|
OK bool `json:"ok"`
|
||||||
|
}
|
||||||
|
response, callErr := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||||
|
PromptID: "snapshot.test",
|
||||||
|
Inputs: contracts.LLMInputSet{
|
||||||
|
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||||
|
},
|
||||||
|
}, &out)
|
||||||
|
completed <- completion{response: response, err: callErr}
|
||||||
|
}()
|
||||||
|
select {
|
||||||
|
case <-opened:
|
||||||
|
case <-time.After(time.Second):
|
||||||
|
t.Fatal("PromptKit did not read the prompt source")
|
||||||
|
}
|
||||||
|
source.replacePrompt([]byte(updatedPrompt))
|
||||||
|
source.resumePromptRead()
|
||||||
|
|
||||||
|
result := <-completed
|
||||||
|
if result.err != nil {
|
||||||
|
t.Fatalf("CompleteStructured() error = %v, want nil", result.err)
|
||||||
|
}
|
||||||
|
if result.response.Debug == nil || result.response.Debug.Prompt == nil || len(result.response.Debug.Prompt.Messages) != 1 {
|
||||||
|
t.Fatalf("debug prompt = %#v, want one prepared message", result.response.Debug)
|
||||||
|
}
|
||||||
|
debugContent := result.response.Debug.Prompt.Messages[0].Content
|
||||||
|
generatedContent := fake.lastRequest().Prompt.Messages[0].Content
|
||||||
|
if debugContent != generatedContent {
|
||||||
|
t.Fatalf("debug content = %q, generation content = %q, want one snapshot", debugContent, generatedContent)
|
||||||
|
}
|
||||||
|
if !strings.Contains(debugContent, "Snapshot A") || strings.Contains(debugContent, "Snapshot B") {
|
||||||
|
t.Fatalf("snapshot content = %q, want the source read before it changed", debugContent)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestPromptKitClientRetainsSessionPromptVariable(t *testing.T) {
|
func TestPromptKitClientRetainsSessionPromptVariable(t *testing.T) {
|
||||||
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||||
client := newTestPromptKitClient(t, fake)
|
client := newTestPromptKitClient(t, fake)
|
||||||
@@ -252,9 +352,10 @@ func TestNewPromptKitClientReportsAssetAndEngineConstructionFailures(t *testing.
|
|||||||
|
|
||||||
func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
|
func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
|
||||||
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
|
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
|
||||||
|
const credentialEnvironment = "PROMPTKIT_TEST_API_KEY"
|
||||||
writeProfile := func(model string) {
|
writeProfile := func(model string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
content := "id: checkpoint-profile\nendpoint: http://promptkit.test/v1\nmodel: " + model + "\n"
|
content := "id: checkpoint-profile\nendpoint: http://promptkit.test/v1\nmodel: " + model + "\napi_key_env: " + credentialEnvironment + "\n"
|
||||||
if err := os.WriteFile(profilePath, []byte(content), 0o600); err != nil {
|
if err := os.WriteFile(profilePath, []byte(content), 0o600); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -280,12 +381,21 @@ func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
|
|||||||
|
|
||||||
writeProfile("model-one")
|
writeProfile("model-one")
|
||||||
first := fingerprintFor()
|
first := fingerprintFor()
|
||||||
|
repeated := fingerprintFor()
|
||||||
|
if first != repeated {
|
||||||
|
t.Fatalf("profile-source fingerprint = %#v then %#v for unchanged source", first, repeated)
|
||||||
|
}
|
||||||
writeProfile("model-two")
|
writeProfile("model-two")
|
||||||
second := fingerprintFor()
|
second := fingerprintFor()
|
||||||
if first == second {
|
if first == second {
|
||||||
t.Fatalf("profile-source fingerprint = %#v for both profile models", first)
|
t.Fatalf("profile-source fingerprint = %#v for both profile models", first)
|
||||||
}
|
}
|
||||||
if strings.Contains(first.Value, profilePath) || strings.Contains(first.Value, "model-one") {
|
if strings.TrimSpace(first.Value) == "" {
|
||||||
|
t.Fatal("profile-source fingerprint is empty")
|
||||||
|
}
|
||||||
|
if strings.Contains(first.Value, profilePath) ||
|
||||||
|
strings.Contains(first.Value, "model-one") ||
|
||||||
|
strings.Contains(first.Value, credentialEnvironment) {
|
||||||
t.Fatalf("profile-source fingerprint exposes source details: %#v", first)
|
t.Fatalf("profile-source fingerprint exposes source details: %#v", first)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -305,9 +415,229 @@ func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
|
|||||||
if fresh[0].Value == "mutated" {
|
if fresh[0].Value == "mutated" {
|
||||||
t.Fatal("LLMCheckpointFingerprints exposed mutable backing storage")
|
t.Fatal("LLMCheckpointFingerprints exposed mutable backing storage")
|
||||||
}
|
}
|
||||||
const wantBuiltinFingerprint = "sha256:4b48cd1ef010b587cd8d56bc13b73a0cdd84da1414a8de9f9a2adc695dfcf0a7"
|
secondClient, err := NewPromptKitClient(PromptKitClientConfig{Assets: newTestPromptKitAssets(t)})
|
||||||
if fresh[0].Value != wantBuiltinFingerprint {
|
if err != nil {
|
||||||
t.Fatalf("built-in profile fingerprint = %q, want %q", fresh[0].Value, wantBuiltinFingerprint)
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
secondBuiltin, err := secondClient.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(secondBuiltin) != 1 || fresh[0] != secondBuiltin[0] {
|
||||||
|
t.Fatalf("built-in profile fingerprints = %#v and %#v, want deterministic identity", fresh, secondBuiltin)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(fresh[0].Value) == "" {
|
||||||
|
t.Fatal("built-in profile fingerprint is empty")
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Run("directory layout", func(t *testing.T) {
|
||||||
|
profileDir := t.TempDir()
|
||||||
|
firstPath := filepath.Join(profileDir, "first-profile.yaml")
|
||||||
|
secondPath := filepath.Join(profileDir, "second-profile.yaml")
|
||||||
|
content := []byte("id: directory-profile\nendpoint: http://promptkit.test/v1\nmodel: directory-model\n")
|
||||||
|
if err := os.WriteFile(firstPath, content, 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
first, err := promptKitProfileFingerprint(profileDir, "", "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.Rename(firstPath, secondPath); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
second, err := promptKitProfileFingerprint(profileDir, "", "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if first == second {
|
||||||
|
t.Fatalf("profile-source fingerprint = %#v after source filename changed", first)
|
||||||
|
}
|
||||||
|
if strings.Contains(first.Value, firstPath) || strings.Contains(second.Value, secondPath) {
|
||||||
|
t.Fatalf("profile-source fingerprint exposes source path: %#v, %#v", first, second)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptKitProfileFingerprintReadErrorsDoNotExposeSourcePaths(t *testing.T) {
|
||||||
|
for _, source := range []struct {
|
||||||
|
name string
|
||||||
|
profileDir string
|
||||||
|
profileFile string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "file",
|
||||||
|
profileFile: filepath.Join(t.TempDir(), "missing-profile.yaml"),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "directory",
|
||||||
|
profileDir: filepath.Join(t.TempDir(), "missing-profiles"),
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(source.name, func(t *testing.T) {
|
||||||
|
_, err := promptKitProfileFingerprint(source.profileDir, source.profileFile, "")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("promptKitProfileFingerprint() error = nil, want source read failure")
|
||||||
|
}
|
||||||
|
if (source.profileDir != "" && strings.Contains(err.Error(), source.profileDir)) ||
|
||||||
|
(source.profileFile != "" && strings.Contains(err.Error(), source.profileFile)) {
|
||||||
|
t.Fatalf("fingerprint error exposes profile source path: %q", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptKitClientUsesFallbackProfilesForExecutionAndInspection(t *testing.T) {
|
||||||
|
assets := newTestPromptKitAssets(t)
|
||||||
|
const profileID = "fallback-profile"
|
||||||
|
if err := assets.RegisterFallbackProfileFS(fstest.MapFS{
|
||||||
|
"profiles/fallback.yaml": {Data: []byte("id: " + profileID + "\nendpoint: http://promptkit.test/v1\nmodel: fallback-model\n")},
|
||||||
|
}, "profiles"); err != nil {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||||
|
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||||
|
Assets: assets,
|
||||||
|
EngineOptions: []promptkit.Option{promptkit.WithLLMClient(fake)},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
var out map[string]any
|
||||||
|
response, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||||
|
PromptID: "adapter.test",
|
||||||
|
ProfileID: profileID,
|
||||||
|
SessionID: "fallback-profile-test",
|
||||||
|
Inputs: contracts.LLMInputSet{
|
||||||
|
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||||
|
},
|
||||||
|
}, &out)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if response.ProfileID != profileID || response.Model != "fallback-model" {
|
||||||
|
t.Fatalf("completion response = %#v, want fallback profile", response)
|
||||||
|
}
|
||||||
|
|
||||||
|
inspector, err := NewPromptKitProfileInspector(PromptKitProfileInspectorConfig{Assets: assets})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewPromptKitProfileInspector() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
inspection, err := inspector.InspectProfile(context.Background(), profileID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("InspectProfile() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if inspection.ProfileID != profileID || inspection.Model != "fallback-model" {
|
||||||
|
t.Fatalf("profile inspection = %#v, want fallback profile", inspection)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptKitClientCheckpointFingerprintTracksFallbackProfileAssets(t *testing.T) {
|
||||||
|
fingerprintFor := func(content string) CheckpointFingerprint {
|
||||||
|
t.Helper()
|
||||||
|
assets := newTestPromptKitAssets(t)
|
||||||
|
if err := assets.RegisterFallbackProfileFS(fstest.MapFS{
|
||||||
|
"profiles/fallback.yaml": {Data: []byte(content)},
|
||||||
|
}, "profiles"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
client, err := NewPromptKitClient(PromptKitClientConfig{Assets: assets})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
fingerprints, err := client.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(fingerprints) != 1 || fingerprints[0].Name != promptKitProfileFingerprintName {
|
||||||
|
t.Fatalf("checkpoint fingerprints = %#v, want profile source identity", fingerprints)
|
||||||
|
}
|
||||||
|
return fingerprints[0]
|
||||||
|
}
|
||||||
|
|
||||||
|
first := fingerprintFor("id: fallback\nendpoint: http://promptkit.test/v1\nmodel: model-one\n")
|
||||||
|
second := fingerprintFor("id: fallback\nendpoint: http://promptkit.test/v1\nmodel: model-two\n")
|
||||||
|
if first == second {
|
||||||
|
t.Fatalf("checkpoint fingerprints = %#v and %#v, want fallback asset change", first, second)
|
||||||
|
}
|
||||||
|
if strings.Contains(first.Value, "model-one") || strings.Contains(first.Value, "fallback.yaml") {
|
||||||
|
t.Fatalf("checkpoint fingerprint leaked fallback source details: %#v", first)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptKitClientCheckpointFingerprintTracksLocalBackendTarget(t *testing.T) {
|
||||||
|
const (
|
||||||
|
firstEndpoint = "http://localhost:8000/v1"
|
||||||
|
secondEndpoint = "https://inference.example.test/v1"
|
||||||
|
)
|
||||||
|
fingerprintsFor := func(endpoint string, concurrencyLimit int) []CheckpointFingerprint {
|
||||||
|
t.Helper()
|
||||||
|
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||||
|
Assets: newTestPromptKitAssets(t),
|
||||||
|
LocalBackend: &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: endpoint,
|
||||||
|
ConcurrencyLimit: concurrencyLimit,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
values, err := client.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return values
|
||||||
|
}
|
||||||
|
|
||||||
|
baseline := fingerprintsFor(firstEndpoint, 0)
|
||||||
|
if len(baseline) != 2 ||
|
||||||
|
baseline[0].Name != promptKitProfileFingerprintName ||
|
||||||
|
baseline[1].Name != promptKitLocalBackendFingerprintName {
|
||||||
|
t.Fatalf("checkpoint fingerprints = %#v, want profile source then local backend target", baseline)
|
||||||
|
}
|
||||||
|
endpointChanged := fingerprintsFor(secondEndpoint, 0)
|
||||||
|
if baseline[0] != endpointChanged[0] || baseline[1] == endpointChanged[1] {
|
||||||
|
t.Fatalf("endpoint fingerprints = %#v and %#v, want only local target to change", baseline, endpointChanged)
|
||||||
|
}
|
||||||
|
concurrencyChanged := fingerprintsFor(firstEndpoint, 4)
|
||||||
|
if !reflect.DeepEqual(baseline, concurrencyChanged) {
|
||||||
|
t.Fatalf("concurrency fingerprints = %#v, want %#v", concurrencyChanged, baseline)
|
||||||
|
}
|
||||||
|
for _, values := range [][]CheckpointFingerprint{baseline, endpointChanged} {
|
||||||
|
for _, fingerprint := range values {
|
||||||
|
if strings.Contains(fingerprint.Value, firstEndpoint) ||
|
||||||
|
strings.Contains(fingerprint.Value, secondEndpoint) {
|
||||||
|
t.Fatalf("checkpoint fingerprint exposes endpoint: %#v", fingerprint)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
localBackend := &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: " " + firstEndpoint + " ",
|
||||||
|
ConcurrencyLimit: 0,
|
||||||
|
}
|
||||||
|
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||||
|
Assets: newTestPromptKitAssets(t),
|
||||||
|
LocalBackend: localBackend,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
localBackend.Endpoint = secondEndpoint
|
||||||
|
copy, err := client.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(copy, baseline) {
|
||||||
|
t.Fatalf("fingerprints after input mutation = %#v, want retained target %#v", copy, baseline)
|
||||||
|
}
|
||||||
|
copy[0].Value = "mutated-profile"
|
||||||
|
copy[1].Value = "mutated-target"
|
||||||
|
fresh, err := client.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(fresh, baseline) {
|
||||||
|
t.Fatalf("fingerprints after returned-slice mutation = %#v, want %#v", fresh, baseline)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -334,6 +664,88 @@ func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testi
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestPromptKitClientUsesConfiguredLocalBackend(t *testing.T) {
|
||||||
|
var providerCalls atomic.Int32
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
providerCalls.Add(1)
|
||||||
|
if r.URL.Path != "/v1/chat/completions" {
|
||||||
|
t.Errorf("provider path = %q, want /v1/chat/completions", r.URL.Path)
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_, _ = w.Write([]byte(`{
|
||||||
|
"choices": [{"message": {"role": "assistant", "content": "{\"ok\":true}"}}],
|
||||||
|
"usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7}
|
||||||
|
}`))
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte(`id: local-profile
|
||||||
|
backend: local
|
||||||
|
model: local-model
|
||||||
|
`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
localBackend := &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: " " + server.URL + "/v1 ",
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||||
|
Assets: newTestPromptKitAssets(t),
|
||||||
|
ProfileFile: profilePath,
|
||||||
|
LocalBackend: localBackend,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
localBackend.Endpoint = "http://127.0.0.1:1/v1"
|
||||||
|
|
||||||
|
request := contracts.StructuredCompletionRequest{
|
||||||
|
PromptID: "adapter.direct-session",
|
||||||
|
ProfileID: "local-profile",
|
||||||
|
Inputs: contracts.LLMInputSet{
|
||||||
|
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||||
|
},
|
||||||
|
Vars: map[string]any{"custom": "value"},
|
||||||
|
}
|
||||||
|
var out map[string]any
|
||||||
|
response, err := client.CompleteStructured(context.Background(), request, &out)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if providerCalls.Load() != 1 {
|
||||||
|
t.Fatalf("provider calls = %d, want 1", providerCalls.Load())
|
||||||
|
}
|
||||||
|
if response.ProfileID != "local-profile" || response.Model != "local-model" {
|
||||||
|
t.Fatalf("response metadata = %#v", response)
|
||||||
|
}
|
||||||
|
if response.Debug == nil || response.Debug.Prompt == nil ||
|
||||||
|
response.Debug.Prompt.SelectedBackendID != promptkit.BackendLocal {
|
||||||
|
t.Fatalf("response debug prompt = %#v, want local backend", response.Debug)
|
||||||
|
}
|
||||||
|
manifests := client.LLMProfileManifests()
|
||||||
|
if len(manifests) != 1 || manifests[0].BackendID != promptkit.BackendLocal {
|
||||||
|
t.Fatalf("profile manifests = %#v, want local backend", manifests)
|
||||||
|
}
|
||||||
|
|
||||||
|
clientWithoutRegistration, err := NewPromptKitClient(PromptKitClientConfig{
|
||||||
|
Assets: newTestPromptKitAssets(t),
|
||||||
|
ProfileFile: profilePath,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewPromptKitClient() without local registration error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
_, err = clientWithoutRegistration.CompleteStructured(context.Background(), request, &out)
|
||||||
|
if err == nil ||
|
||||||
|
!strings.Contains(err.Error(), "prepare PromptKit prompt") ||
|
||||||
|
!strings.Contains(err.Error(), promptkit.BackendLocal) {
|
||||||
|
t.Fatalf("CompleteStructured() without registration error = %v, want preparation failure with local backend context", err)
|
||||||
|
}
|
||||||
|
if providerCalls.Load() != 1 {
|
||||||
|
t.Fatalf("provider calls after missing-registration failure = %d, want 1", providerCalls.Load())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestLLMProfileRecorderDistinguishesEffectiveTargets(t *testing.T) {
|
func TestLLMProfileRecorderDistinguishesEffectiveTargets(t *testing.T) {
|
||||||
recorder := NewLLMProfileRecorder()
|
recorder := NewLLMProfileRecorder()
|
||||||
for _, profile := range []artifacts.LLMProfileManifest{
|
for _, profile := range []artifacts.LLMProfileManifest{
|
||||||
@@ -494,8 +906,13 @@ func TestPromptKitClientTranslatesBackendCapacityExhaustion(t *testing.T) {
|
|||||||
t.Fatalf("capacity error exposes PromptKit sentinel: %v", capacityErr)
|
t.Fatalf("capacity error exposes PromptKit sentinel: %v", capacityErr)
|
||||||
}
|
}
|
||||||
if !strings.Contains(capacityErr.Error(), `run PromptKit prompt "adapter.direct-session"`) ||
|
if !strings.Contains(capacityErr.Error(), `run PromptKit prompt "adapter.direct-session"`) ||
|
||||||
|
!strings.Contains(capacityErr.Error(), `backend "limited-backend"`) ||
|
||||||
!strings.Contains(capacityErr.Error(), "backend capacity exceeded") {
|
!strings.Contains(capacityErr.Error(), "backend capacity exceeded") {
|
||||||
t.Fatalf("capacity error = %q, want prompt context and upstream diagnostic", capacityErr)
|
t.Fatalf("capacity error = %q, want prompt, backend, and upstream diagnostic context", capacityErr)
|
||||||
|
}
|
||||||
|
var upstreamCapacityErr *promptkit.CapacityError
|
||||||
|
if errors.As(capacityErr, &upstreamCapacityErr) {
|
||||||
|
t.Fatalf("capacity error exposes PromptKit capacity type: %v", capacityErr)
|
||||||
}
|
}
|
||||||
if calls := atomic.LoadInt32(&fake.calls); calls != 1 {
|
if calls := atomic.LoadInt32(&fake.calls); calls != 1 {
|
||||||
t.Fatalf("provider calls after capacity rejection = %d, want 1", calls)
|
t.Fatalf("provider calls after capacity rejection = %d, want 1", calls)
|
||||||
@@ -730,6 +1147,61 @@ output:
|
|||||||
return registry
|
return registry
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type switchingPromptFS struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
files fstest.MapFS
|
||||||
|
pauseNextOpen bool
|
||||||
|
opened chan struct{}
|
||||||
|
resume chan struct{}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *switchingPromptFS) Open(name string) (fs.File, error) {
|
||||||
|
f.mu.Lock()
|
||||||
|
file, err := f.files.Open(name)
|
||||||
|
pause := f.pauseNextOpen && name == "snapshot.test.yaml"
|
||||||
|
resume := f.resume
|
||||||
|
if pause {
|
||||||
|
f.pauseNextOpen = false
|
||||||
|
close(f.opened)
|
||||||
|
}
|
||||||
|
f.mu.Unlock()
|
||||||
|
if pause {
|
||||||
|
<-resume
|
||||||
|
}
|
||||||
|
return file, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *switchingPromptFS) ReadDir(name string) ([]fs.DirEntry, error) {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
return f.files.ReadDir(name)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *switchingPromptFS) holdNextPromptRead() <-chan struct{} {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
f.pauseNextOpen = true
|
||||||
|
f.opened = make(chan struct{})
|
||||||
|
f.resume = make(chan struct{})
|
||||||
|
return f.opened
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *switchingPromptFS) replacePrompt(content []byte) {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
f.files["snapshot.test.yaml"] = &fstest.MapFile{Data: append([]byte(nil), content...)}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *switchingPromptFS) resumePromptRead() {
|
||||||
|
f.mu.Lock()
|
||||||
|
resume := f.resume
|
||||||
|
f.resume = nil
|
||||||
|
f.mu.Unlock()
|
||||||
|
if resume != nil {
|
||||||
|
close(resume)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
type fakePromptKitLLM struct {
|
type fakePromptKitLLM struct {
|
||||||
content string
|
content string
|
||||||
allowEmpty bool
|
allowEmpty bool
|
||||||
|
|||||||
@@ -12,30 +12,36 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
promptKitProfileFingerprintName = "promptkit_profile_source"
|
promptKitProfileFingerprintName = "promptkit_profile_source"
|
||||||
|
promptKitLocalBackendFingerprintName = "promptkit_local_backend_target"
|
||||||
|
promptKitLocalBackendMarker = "notarius:promptkit-local-backend:v1"
|
||||||
// The built-in profile catalog is compiled into this pinned PromptKit
|
// The built-in profile catalog is compiled into this pinned PromptKit
|
||||||
// release. Update this identity when the dependency is upgraded.
|
// release. Update this identity when the dependency is upgraded.
|
||||||
promptKitBuiltinProfileCatalogID = "promptkit:v0.2.0:builtin-profiles"
|
promptKitBuiltinProfileCatalogID = "promptkit:v0.5.0:builtin-profiles"
|
||||||
)
|
)
|
||||||
|
|
||||||
func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFingerprint, error) {
|
func promptKitProfileFingerprint(profileDir, profileFile, fallbackProfileDigest string) (CheckpointFingerprint, error) {
|
||||||
hasher := sha256.New()
|
hasher := sha256.New()
|
||||||
writeFingerprintPart(hasher, []byte(promptKitBuiltinProfileCatalogID))
|
writeFingerprintPart(hasher, []byte(promptKitBuiltinProfileCatalogID))
|
||||||
|
writeFingerprintPart(hasher, []byte(strings.TrimSpace(fallbackProfileDigest)))
|
||||||
|
|
||||||
switch {
|
switch {
|
||||||
case strings.TrimSpace(profileFile) != "":
|
case strings.TrimSpace(profileFile) != "":
|
||||||
data, err := os.ReadFile(strings.TrimSpace(profileFile))
|
cleanProfileFile := strings.TrimSpace(profileFile)
|
||||||
|
data, err := os.ReadFile(cleanProfileFile)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return CheckpointFingerprint{}, fmt.Errorf("read PromptKit profile file for checkpoint identity: %w", err)
|
return CheckpointFingerprint{}, fmt.Errorf("read PromptKit profile file for checkpoint identity")
|
||||||
}
|
}
|
||||||
|
writeFingerprintPart(hasher, []byte(filepath.ToSlash(filepath.Base(cleanProfileFile))))
|
||||||
writeFingerprintPart(hasher, data)
|
writeFingerprintPart(hasher, data)
|
||||||
case strings.TrimSpace(profileDir) != "":
|
case strings.TrimSpace(profileDir) != "":
|
||||||
digests, err := promptKitProfileFileDigests(strings.TrimSpace(profileDir))
|
files, err := promptKitProfileFiles(strings.TrimSpace(profileDir))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return CheckpointFingerprint{}, err
|
return CheckpointFingerprint{}, err
|
||||||
}
|
}
|
||||||
for _, digest := range digests {
|
for _, file := range files {
|
||||||
writeFingerprintPart(hasher, digest)
|
writeFingerprintPart(hasher, []byte(file.path))
|
||||||
|
writeFingerprintPart(hasher, file.digest)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -45,8 +51,23 @@ func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFing
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func promptKitProfileFileDigests(root string) ([][]byte, error) {
|
func promptKitLocalBackendFingerprint(endpoint string) CheckpointFingerprint {
|
||||||
var digests [][]byte
|
hasher := sha256.New()
|
||||||
|
writeFingerprintPart(hasher, []byte(promptKitLocalBackendMarker))
|
||||||
|
writeFingerprintPart(hasher, []byte(strings.TrimSpace(endpoint)))
|
||||||
|
return CheckpointFingerprint{
|
||||||
|
Name: promptKitLocalBackendFingerprintName,
|
||||||
|
Value: "sha256:" + hex.EncodeToString(hasher.Sum(nil)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type promptKitProfileFile struct {
|
||||||
|
path string
|
||||||
|
digest []byte
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptKitProfileFiles(root string) ([]promptKitProfileFile, error) {
|
||||||
|
var files []promptKitProfileFile
|
||||||
err := filepath.WalkDir(root, func(name string, entry fs.DirEntry, walkErr error) error {
|
err := filepath.WalkDir(root, func(name string, entry fs.DirEntry, walkErr error) error {
|
||||||
if walkErr != nil {
|
if walkErr != nil {
|
||||||
return walkErr
|
return walkErr
|
||||||
@@ -62,17 +83,24 @@ func promptKitProfileFileDigests(root string) ([][]byte, error) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
relativePath, err := filepath.Rel(root, name)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
sum := sha256.Sum256(data)
|
sum := sha256.Sum256(data)
|
||||||
digests = append(digests, append([]byte(nil), sum[:]...))
|
files = append(files, promptKitProfileFile{
|
||||||
|
path: filepath.ToSlash(relativePath),
|
||||||
|
digest: append([]byte(nil), sum[:]...),
|
||||||
|
})
|
||||||
return nil
|
return nil
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("read PromptKit profile directory for checkpoint identity: %w", err)
|
return nil, fmt.Errorf("read PromptKit profile directory for checkpoint identity")
|
||||||
}
|
}
|
||||||
sort.Slice(digests, func(i, j int) bool {
|
sort.Slice(files, func(i, j int) bool {
|
||||||
return string(digests[i]) < string(digests[j])
|
return files[i].path < files[j].path
|
||||||
})
|
})
|
||||||
return digests, nil
|
return files, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeFingerprintPart(hasher interface{ Write([]byte) (int, error) }, value []byte) {
|
func writeFingerprintPart(hasher interface{ Write([]byte) (int, error) }, value []byte) {
|
||||||
|
|||||||
146
internal/framework/llm/promptkit_profiles.go
Normal file
146
internal/framework/llm/promptkit_profiles.go
Normal file
@@ -0,0 +1,146 @@
|
|||||||
|
package llm
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
|
)
|
||||||
|
|
||||||
|
type PromptKitProfileSourceConfig struct {
|
||||||
|
ProfileDir string
|
||||||
|
ProfileFile string
|
||||||
|
LocalBackend *PromptKitLocalBackendConfig
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitProfileInspectorConfig struct {
|
||||||
|
Source PromptKitProfileSourceConfig
|
||||||
|
Assets *AssetRegistry
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c PromptKitProfileSourceConfig) localEndpoint() string {
|
||||||
|
if c.LocalBackend == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return c.LocalBackend.Endpoint
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitProfileInspector struct {
|
||||||
|
engine *promptkit.Engine
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitProfileInspection struct {
|
||||||
|
ProfileID string
|
||||||
|
BackendID string
|
||||||
|
Model string
|
||||||
|
CredentialEnvironment string
|
||||||
|
CredentialRequired bool
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitProfileInspectionError struct {
|
||||||
|
ProfileID string
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *PromptKitProfileInspectionError) Error() string {
|
||||||
|
switch {
|
||||||
|
case errors.Is(e.err, promptkit.ErrProfileNotFound):
|
||||||
|
return fmt.Sprintf("PromptKit profile %q is not configured", e.ProfileID)
|
||||||
|
case errors.Is(e.err, promptkit.ErrInvalidRequest):
|
||||||
|
return fmt.Sprintf("PromptKit profile ID %q is invalid", e.ProfileID)
|
||||||
|
case errors.Is(e.err, promptkit.ErrProfileLoad):
|
||||||
|
return fmt.Sprintf("PromptKit profile %q is invalid or unreadable", e.ProfileID)
|
||||||
|
default:
|
||||||
|
return fmt.Sprintf("PromptKit profile %q could not be inspected", e.ProfileID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *PromptKitProfileInspectionError) Unwrap() error {
|
||||||
|
return e.err
|
||||||
|
}
|
||||||
|
|
||||||
|
type promptKitProfileConfigurationError struct {
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *promptKitProfileConfigurationError) Error() string {
|
||||||
|
return "PromptKit profile configuration is invalid or unreadable"
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *promptKitProfileConfigurationError) Unwrap() error {
|
||||||
|
return e.err
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewPromptKitProfileInspector(cfg PromptKitProfileInspectorConfig) (*PromptKitProfileInspector, error) {
|
||||||
|
source, options, err := promptKitProfileSourceEngineOptions(cfg.Source)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if cfg.Assets != nil {
|
||||||
|
fallbackOption, hasFallback, err := cfg.Assets.promptKitFallbackProfileOption()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if hasFallback {
|
||||||
|
options = append(options, fallbackOption)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||||
|
PromptDir: ".",
|
||||||
|
ProfileDir: source.ProfileDir,
|
||||||
|
}, options...)
|
||||||
|
if err != nil {
|
||||||
|
return nil, &promptKitProfileConfigurationError{err: err}
|
||||||
|
}
|
||||||
|
return &PromptKitProfileInspector{engine: engine}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (i *PromptKitProfileInspector) InspectProfile(ctx context.Context, profileID string) (PromptKitProfileInspection, error) {
|
||||||
|
if i == nil || i.engine == nil {
|
||||||
|
return PromptKitProfileInspection{}, fmt.Errorf("PromptKit profile inspector must not be nil")
|
||||||
|
}
|
||||||
|
profileID = strings.TrimSpace(profileID)
|
||||||
|
inspection, err := i.engine.InspectProfile(ctx, profileID)
|
||||||
|
if err != nil {
|
||||||
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
|
return PromptKitProfileInspection{}, ctxErr
|
||||||
|
}
|
||||||
|
return PromptKitProfileInspection{}, &PromptKitProfileInspectionError{
|
||||||
|
ProfileID: profileID,
|
||||||
|
err: err,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return PromptKitProfileInspection{
|
||||||
|
ProfileID: inspection.ProfileID,
|
||||||
|
BackendID: strings.TrimSpace(inspection.EffectiveModelParams.BackendID),
|
||||||
|
Model: strings.TrimSpace(inspection.EffectiveModelParams.Model),
|
||||||
|
CredentialEnvironment: strings.TrimSpace(inspection.EffectiveModelParams.APIKeyEnv),
|
||||||
|
CredentialRequired: inspection.APIKeyRequired,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptKitProfileSourceEngineOptions(cfg PromptKitProfileSourceConfig) (PromptKitProfileSourceConfig, []promptkit.Option, error) {
|
||||||
|
source := PromptKitProfileSourceConfig{
|
||||||
|
ProfileDir: strings.TrimSpace(cfg.ProfileDir),
|
||||||
|
ProfileFile: strings.TrimSpace(cfg.ProfileFile),
|
||||||
|
}
|
||||||
|
if source.ProfileDir != "" && source.ProfileFile != "" {
|
||||||
|
return PromptKitProfileSourceConfig{}, nil, fmt.Errorf("PromptKit profile_dir and profile_file are mutually exclusive")
|
||||||
|
}
|
||||||
|
if cfg.LocalBackend != nil {
|
||||||
|
localBackend := *cfg.LocalBackend
|
||||||
|
localBackend.Endpoint = strings.TrimSpace(localBackend.Endpoint)
|
||||||
|
source.LocalBackend = &localBackend
|
||||||
|
}
|
||||||
|
|
||||||
|
var options []promptkit.Option
|
||||||
|
if source.ProfileFile != "" {
|
||||||
|
options = append(options, promptkit.WithProfileFile(source.ProfileFile))
|
||||||
|
}
|
||||||
|
if source.LocalBackend != nil {
|
||||||
|
options = append(options, PromptKitLocalBackendOption(*source.LocalBackend))
|
||||||
|
}
|
||||||
|
return source, options, nil
|
||||||
|
}
|
||||||
@@ -81,7 +81,10 @@ func TestScheduledClientPreservesCheckpointFingerprints(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
inner := &fingerprintedStructuredClient{
|
inner := &fingerprintedStructuredClient{
|
||||||
fingerprints: []CheckpointFingerprint{{Name: "profile_source", Value: "sha256:one"}},
|
fingerprints: []CheckpointFingerprint{
|
||||||
|
{Name: "profile_source", Value: "sha256:one"},
|
||||||
|
{Name: "backend_target", Value: "sha256:two"},
|
||||||
|
},
|
||||||
}
|
}
|
||||||
client := NewScheduledClient(inner, scheduler)
|
client := NewScheduledClient(inner, scheduler)
|
||||||
provider, ok := client.(CheckpointFingerprintProvider)
|
provider, ok := client.(CheckpointFingerprintProvider)
|
||||||
@@ -92,7 +95,9 @@ func TestScheduledClientPreservesCheckpointFingerprints(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if len(got) != 1 || got[0] != inner.fingerprints[0] {
|
if len(got) != len(inner.fingerprints) ||
|
||||||
|
got[0] != inner.fingerprints[0] ||
|
||||||
|
got[1] != inner.fingerprints[1] {
|
||||||
t.Fatalf("checkpoint fingerprints = %#v, want %#v", got, inner.fingerprints)
|
t.Fatalf("checkpoint fingerprints = %#v, want %#v", got, inner.fingerprints)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -24,10 +24,6 @@ func NewChunkerRegistry() *ChunkerRegistry {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *ChunkerRegistry) Register(key string, constructor ChunkerConstructor) error {
|
|
||||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageChunk), constructor)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerConstructor) error {
|
func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerConstructor) error {
|
||||||
if constructor == nil {
|
if constructor == nil {
|
||||||
return fmt.Errorf("chunker constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
return fmt.Errorf("chunker constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
|
|||||||
@@ -29,6 +29,10 @@ type registryBehaviorCase[M any] struct {
|
|||||||
moduleKey func(M) string
|
moduleKey func(M) string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func testModuleSpec(key string, stage ModuleStage) ModuleSpec {
|
||||||
|
return ModuleSpec{Key: key, Stage: stage, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
|
}
|
||||||
|
|
||||||
func TestChunkerRegistryBehavior(t *testing.T) {
|
func TestChunkerRegistryBehavior(t *testing.T) {
|
||||||
runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Chunker]{
|
runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Chunker]{
|
||||||
name: "ChunkerRegistry",
|
name: "ChunkerRegistry",
|
||||||
@@ -39,7 +43,9 @@ func TestChunkerRegistryBehavior(t *testing.T) {
|
|||||||
return NewChunkerRegistry()
|
return NewChunkerRegistry()
|
||||||
},
|
},
|
||||||
register: func(registry any, key string, constructor func() (contracts.Chunker, error)) error {
|
register: func(registry any, key string, constructor func() (contracts.Chunker, error)) error {
|
||||||
return registry.(*ChunkerRegistry).Register(key, constructor)
|
return registry.(*ChunkerRegistry).RegisterWithSpec(ModuleSpec{
|
||||||
|
Key: key, Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
}, constructor)
|
||||||
},
|
},
|
||||||
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Chunker, error)) error {
|
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Chunker, error)) error {
|
||||||
return registry.(*ChunkerRegistry).RegisterWithSpec(spec, constructor)
|
return registry.(*ChunkerRegistry).RegisterWithSpec(spec, constructor)
|
||||||
@@ -55,7 +61,9 @@ func TestChunkerRegistryBehavior(t *testing.T) {
|
|||||||
},
|
},
|
||||||
nilRegister: func(key string, constructor func() (contracts.Chunker, error)) error {
|
nilRegister: func(key string, constructor func() (contracts.Chunker, error)) error {
|
||||||
var registry *ChunkerRegistry
|
var registry *ChunkerRegistry
|
||||||
return registry.Register(key, constructor)
|
return registry.RegisterWithSpec(ModuleSpec{
|
||||||
|
Key: key, Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
}, constructor)
|
||||||
},
|
},
|
||||||
nilBuild: func(key string) (contracts.Chunker, error) {
|
nilBuild: func(key string) (contracts.Chunker, error) {
|
||||||
var registry *ChunkerRegistry
|
var registry *ChunkerRegistry
|
||||||
@@ -101,10 +109,11 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
t.Run(testCase.name+"/metadata registration and lookup", func(t *testing.T) {
|
t.Run(testCase.name+"/metadata registration and lookup", func(t *testing.T) {
|
||||||
registry := testCase.newRegistry()
|
registry := testCase.newRegistry()
|
||||||
spec := ModuleSpec{
|
spec := ModuleSpec{
|
||||||
Key: " " + testCase.key + " ",
|
Key: " " + testCase.key + " ",
|
||||||
Stage: testCase.stage,
|
Stage: testCase.stage,
|
||||||
Provides: []string{" beta ", "alpha", "", "beta"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{" source ", "source", ""},
|
Provides: []string{" beta ", "alpha", "", "beta"},
|
||||||
|
Requires: []string{" source ", "source", ""},
|
||||||
}
|
}
|
||||||
if err := testCase.registerWithSpec(registry, spec, testCase.constructor(testCase.key)); err != nil {
|
if err := testCase.registerWithSpec(registry, spec, testCase.constructor(testCase.key)); err != nil {
|
||||||
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
||||||
@@ -115,10 +124,11 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
t.Fatal("Spec() ok = false, want true")
|
t.Fatal("Spec() ok = false, want true")
|
||||||
}
|
}
|
||||||
want := ModuleSpec{
|
want := ModuleSpec{
|
||||||
Key: testCase.key,
|
Key: testCase.key,
|
||||||
Stage: testCase.stage,
|
Stage: testCase.stage,
|
||||||
Provides: []string{"alpha", "beta"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"source"},
|
Provides: []string{"alpha", "beta"},
|
||||||
|
Requires: []string{"source"},
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(got, want) {
|
if !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||||
@@ -134,7 +144,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
t.Run(testCase.name+"/default spec from register", func(t *testing.T) {
|
t.Run(testCase.name+"/minimal explicit spec registration", func(t *testing.T) {
|
||||||
registry := testCase.newRegistry()
|
registry := testCase.newRegistry()
|
||||||
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
|
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
@@ -144,7 +154,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
if !ok {
|
if !ok {
|
||||||
t.Fatal("Spec() ok = false, want true")
|
t.Fatal("Spec() ok = false, want true")
|
||||||
}
|
}
|
||||||
want := ModuleSpec{Key: testCase.key, Stage: testCase.stage}
|
want := ModuleSpec{Key: testCase.key, Stage: testCase.stage, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
if !reflect.DeepEqual(spec, want) {
|
if !reflect.DeepEqual(spec, want) {
|
||||||
t.Fatalf("Spec() = %#v, want %#v", spec, want)
|
t.Fatalf("Spec() = %#v, want %#v", spec, want)
|
||||||
}
|
}
|
||||||
@@ -152,7 +162,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
|
|
||||||
t.Run(testCase.name+"/wrong stage rejection", func(t *testing.T) {
|
t.Run(testCase.name+"/wrong stage rejection", func(t *testing.T) {
|
||||||
registry := testCase.newRegistry()
|
registry := testCase.newRegistry()
|
||||||
err := testCase.registerWithSpec(registry, ModuleSpec{Key: testCase.key, Stage: testCase.wrongStage}, testCase.constructor(testCase.key))
|
err := testCase.registerWithSpec(registry, ModuleSpec{Key: testCase.key, Stage: testCase.wrongStage, ExecutionClass: contracts.ExecutionClassDeterministic}, testCase.constructor(testCase.key))
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("RegisterWithSpec() error = nil, want error")
|
t.Fatal("RegisterWithSpec() error = nil, want error")
|
||||||
}
|
}
|
||||||
@@ -161,6 +171,22 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run(testCase.name+"/execution class rejection", func(t *testing.T) {
|
||||||
|
for _, spec := range []ModuleSpec{
|
||||||
|
{Key: testCase.key, Stage: testCase.stage},
|
||||||
|
{Key: testCase.key, Stage: testCase.stage, ExecutionClass: contracts.ExecutionClass("unsupported")},
|
||||||
|
} {
|
||||||
|
registry := testCase.newRegistry()
|
||||||
|
err := testCase.registerWithSpec(registry, spec, testCase.constructor(testCase.key))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("RegisterWithSpec() error = nil, want execution class error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "execution class") {
|
||||||
|
t.Fatalf("RegisterWithSpec() error = %q, want execution class error", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
t.Run(testCase.name+"/key trimming", func(t *testing.T) {
|
t.Run(testCase.name+"/key trimming", func(t *testing.T) {
|
||||||
registry := testCase.newRegistry()
|
registry := testCase.newRegistry()
|
||||||
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
|
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
|
||||||
|
|||||||
@@ -108,7 +108,7 @@ func registerTestEvidenceOutput(t *testing.T, registries *Registries, policy Evi
|
|||||||
func registerTestEvidenceOutputWithProfileValidation(t *testing.T, registries *Registries, policy EvidenceContextPolicy, validateProfile OutputProfileOptionValidator) {
|
func registerTestEvidenceOutputWithProfileValidation(t *testing.T, registries *Registries, policy EvidenceContextPolicy, validateProfile OutputProfileOptionValidator) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
registry := NewOutputEncoderRegistry()
|
registry := NewOutputEncoderRegistry()
|
||||||
if err := registry.RegisterBuilderWithProfileValidation(defaultModuleSpec("output", StageOutput), func(options map[string]any) error {
|
if err := registry.RegisterBuilderWithProfileValidation(testModuleSpec("output", StageOutput), func(options map[string]any) error {
|
||||||
return RejectUnknownOptions(options, "known")
|
return RejectUnknownOptions(options, "known")
|
||||||
}, validateProfile, func(BuildRequest) (contracts.OutputEncoder, error) {
|
}, validateProfile, func(BuildRequest) (contracts.OutputEncoder, error) {
|
||||||
return testEvidenceOutput{policy: cloneEvidenceContextPolicy(policy)}, nil
|
return testEvidenceOutput{policy: cloneEvidenceContextPolicy(policy)}, nil
|
||||||
|
|||||||
@@ -24,10 +24,6 @@ func NewInputAdapterRegistry() *InputAdapterRegistry {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *InputAdapterRegistry) Register(key string, constructor InputAdapterConstructor) error {
|
|
||||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageInput), constructor)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor InputAdapterConstructor) error {
|
func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor InputAdapterConstructor) error {
|
||||||
if constructor == nil {
|
if constructor == nil {
|
||||||
return fmt.Errorf("input adapter constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
return fmt.Errorf("input adapter constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
|
|||||||
@@ -11,10 +11,14 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
func registerTestInput(registry *InputAdapterRegistry, key string, constructor InputAdapterConstructor) error {
|
||||||
|
return registry.RegisterWithSpec(testModuleSpec(key, StageInput), constructor)
|
||||||
|
}
|
||||||
|
|
||||||
func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) {
|
func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -30,7 +34,7 @@ func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisterAndBuildTrimKeys(t *testing.T) {
|
func TestInputAdapterRegistryRegisterAndBuildTrimKeys(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
if err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
if err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -46,10 +50,11 @@ func TestInputAdapterRegistryRegisterAndBuildTrimKeys(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
spec := ModuleSpec{
|
spec := ModuleSpec{
|
||||||
Key: " generic-input ",
|
Key: " generic-input ",
|
||||||
Stage: StageInput,
|
Stage: StageInput,
|
||||||
Provides: []string{" parsed-source ", "source-document", "parsed-source", ""},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{" raw-bytes ", "raw-bytes", ""},
|
Provides: []string{" parsed-source ", "source-document", "parsed-source", ""},
|
||||||
|
Requires: []string{" raw-bytes ", "raw-bytes", ""},
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := registry.RegisterWithSpec(spec, fakeInputAdapterConstructor("generic-input")); err != nil {
|
if err := registry.RegisterWithSpec(spec, fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||||
@@ -61,10 +66,11 @@ func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
|||||||
t.Fatal("Spec() ok = false, want true")
|
t.Fatal("Spec() ok = false, want true")
|
||||||
}
|
}
|
||||||
want := ModuleSpec{
|
want := ModuleSpec{
|
||||||
Key: "generic-input",
|
Key: "generic-input",
|
||||||
Stage: StageInput,
|
Stage: StageInput,
|
||||||
Provides: []string{"parsed-source", "source-document"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"raw-bytes"},
|
Provides: []string{"parsed-source", "source-document"},
|
||||||
|
Requires: []string{"raw-bytes"},
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(got, want) {
|
if !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||||
@@ -80,10 +86,10 @@ func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
|
func TestInputAdapterRegistryRegisterWithSpecStoresMinimalMetadata(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
if err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
if err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -91,7 +97,7 @@ func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
|
|||||||
if !ok {
|
if !ok {
|
||||||
t.Fatal("Spec() ok = false, want true")
|
t.Fatal("Spec() ok = false, want true")
|
||||||
}
|
}
|
||||||
want := ModuleSpec{Key: "generic-input", Stage: StageInput}
|
want := ModuleSpec{Key: "generic-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
if !reflect.DeepEqual(got, want) {
|
if !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||||
}
|
}
|
||||||
@@ -100,7 +106,7 @@ func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisterWithSpecRejectsWrongStage(t *testing.T) {
|
func TestInputAdapterRegistryRegisterWithSpecRejectsWrongStage(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
err := registry.RegisterWithSpec(ModuleSpec{Key: "generic-input", Stage: StageExtract}, fakeInputAdapterConstructor("generic-input"))
|
err := registry.RegisterWithSpec(ModuleSpec{Key: "generic-input", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic}, fakeInputAdapterConstructor("generic-input"))
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("RegisterWithSpec() error = nil, want error")
|
t.Fatal("RegisterWithSpec() error = nil, want error")
|
||||||
@@ -121,7 +127,7 @@ func TestInputAdapterRegistrySpecRejectsUnknownKey(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) {
|
func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
err := registry.Register(" \t", fakeInputAdapterConstructor("generic-input"))
|
err := registerTestInput(registry, " \t", fakeInputAdapterConstructor("generic-input"))
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("Register() error = nil, want error")
|
t.Fatal("Register() error = nil, want error")
|
||||||
@@ -133,11 +139,11 @@ func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) {
|
|||||||
|
|
||||||
func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) {
|
func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input"))
|
err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input"))
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("Register() error = nil, want error")
|
t.Fatal("Register() error = nil, want error")
|
||||||
@@ -150,7 +156,7 @@ func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisterRejectsNilConstructor(t *testing.T) {
|
func TestInputAdapterRegistryRegisterRejectsNilConstructor(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
err := registry.Register("generic-input", nil)
|
err := registerTestInput(registry, "generic-input", nil)
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("Register() error = nil, want error")
|
t.Fatal("Register() error = nil, want error")
|
||||||
@@ -176,7 +182,7 @@ func TestInputAdapterRegistryBuildRejectsUnknownKey(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) {
|
func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
constructorErr := errors.New("constructor failed")
|
constructorErr := errors.New("constructor failed")
|
||||||
if err := registry.Register("generic-input", func() (contracts.InputAdapter, error) {
|
if err := registerTestInput(registry, "generic-input", func() (contracts.InputAdapter, error) {
|
||||||
return nil, constructorErr
|
return nil, constructorErr
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
@@ -197,7 +203,7 @@ func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) {
|
|||||||
|
|
||||||
func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) {
|
func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
if err := registry.Register("generic-input", func() (contracts.InputAdapter, error) {
|
if err := registerTestInput(registry, "generic-input", func() (contracts.InputAdapter, error) {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
@@ -215,7 +221,7 @@ func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) {
|
|||||||
|
|
||||||
func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) {
|
func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("other-input")); err != nil {
|
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("other-input")); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -232,7 +238,7 @@ func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
|
func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
for _, key := range []string{"zeta", "alpha", "middle"} {
|
for _, key := range []string{"zeta", "alpha", "middle"} {
|
||||||
if err := registry.Register(key, fakeInputAdapterConstructor(key)); err != nil {
|
if err := registerTestInput(registry, key, fakeInputAdapterConstructor(key)); err != nil {
|
||||||
t.Fatalf("Register(%q) error = %v, want nil", key, err)
|
t.Fatalf("Register(%q) error = %v, want nil", key, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -253,7 +259,7 @@ func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryNilRegistryBehavior(t *testing.T) {
|
func TestInputAdapterRegistryNilRegistryBehavior(t *testing.T) {
|
||||||
var registry *InputAdapterRegistry
|
var registry *InputAdapterRegistry
|
||||||
|
|
||||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err == nil {
|
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err == nil {
|
||||||
t.Fatal("Register() error = nil, want error")
|
t.Fatal("Register() error = nil, want error")
|
||||||
}
|
}
|
||||||
if _, err := registry.Build("generic-input"); err == nil {
|
if _, err := registry.Build("generic-input"); err == nil {
|
||||||
|
|||||||
@@ -23,23 +23,19 @@ const (
|
|||||||
type ModuleSpec struct {
|
type ModuleSpec struct {
|
||||||
Key string
|
Key string
|
||||||
Stage ModuleStage
|
Stage ModuleStage
|
||||||
|
ExecutionClass contracts.ExecutionClass
|
||||||
ArtifactKind contracts.ArtifactKind
|
ArtifactKind contracts.ArtifactKind
|
||||||
Provides []string
|
Provides []string
|
||||||
Requires []string
|
Requires []string
|
||||||
ReferenceSlots []contracts.ReferenceSlot
|
ReferenceSlots []contracts.ReferenceSlot
|
||||||
}
|
}
|
||||||
|
|
||||||
func defaultModuleSpec(key string, stage ModuleStage) ModuleSpec {
|
|
||||||
return ModuleSpec{
|
|
||||||
Key: key,
|
|
||||||
Stage: stage,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func normalizeModuleSpec(spec ModuleSpec) ModuleSpec {
|
func normalizeModuleSpec(spec ModuleSpec) ModuleSpec {
|
||||||
|
executionClass := contracts.ExecutionClass(strings.TrimSpace(string(spec.ExecutionClass)))
|
||||||
return ModuleSpec{
|
return ModuleSpec{
|
||||||
Key: strings.TrimSpace(spec.Key),
|
Key: strings.TrimSpace(spec.Key),
|
||||||
Stage: spec.Stage,
|
Stage: spec.Stage,
|
||||||
|
ExecutionClass: executionClass,
|
||||||
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
|
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
|
||||||
Provides: normalizeCapabilities(spec.Provides),
|
Provides: normalizeCapabilities(spec.Provides),
|
||||||
Requires: normalizeCapabilities(spec.Requires),
|
Requires: normalizeCapabilities(spec.Requires),
|
||||||
@@ -76,6 +72,7 @@ func cloneModuleSpec(spec ModuleSpec) ModuleSpec {
|
|||||||
return ModuleSpec{
|
return ModuleSpec{
|
||||||
Key: spec.Key,
|
Key: spec.Key,
|
||||||
Stage: spec.Stage,
|
Stage: spec.Stage,
|
||||||
|
ExecutionClass: spec.ExecutionClass,
|
||||||
ArtifactKind: spec.ArtifactKind,
|
ArtifactKind: spec.ArtifactKind,
|
||||||
Provides: append([]string(nil), spec.Provides...),
|
Provides: append([]string(nil), spec.Provides...),
|
||||||
Requires: append([]string(nil), spec.Requires...),
|
Requires: append([]string(nil), spec.Requires...),
|
||||||
@@ -90,6 +87,12 @@ func validateModuleSpec(kind string, expectedStage ModuleStage, spec ModuleSpec)
|
|||||||
if spec.Stage != expectedStage {
|
if spec.Stage != expectedStage {
|
||||||
return fmt.Errorf("%s %q must use %q stage, got %q", kind, spec.Key, expectedStage, spec.Stage)
|
return fmt.Errorf("%s %q must use %q stage, got %q", kind, spec.Key, expectedStage, spec.Stage)
|
||||||
}
|
}
|
||||||
|
if spec.ExecutionClass == "" {
|
||||||
|
return fmt.Errorf("%s %q execution class must not be empty", kind, spec.Key)
|
||||||
|
}
|
||||||
|
if spec.ExecutionClass != contracts.ExecutionClassDeterministic && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
|
||||||
|
return fmt.Errorf("%s %q has unsupported execution class %q", kind, spec.Key, spec.ExecutionClass)
|
||||||
|
}
|
||||||
if spec.ArtifactKind != "" && spec.Stage != StageExtract && spec.Stage != StageMerge && spec.Stage != StageNormalize {
|
if spec.ArtifactKind != "" && spec.Stage != StageExtract && spec.Stage != StageMerge && spec.Stage != StageNormalize {
|
||||||
return fmt.Errorf("%s %q must not declare an artifact kind", kind, spec.Key)
|
return fmt.Errorf("%s %q must not declare an artifact kind", kind, spec.Key)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,12 +1,45 @@
|
|||||||
package pipeline
|
package pipeline
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
func TestValidateModuleSpecRequiresSupportedExecutionClass(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
class contracts.ExecutionClass
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "missing", want: "execution class"},
|
||||||
|
{name: "unsupported", class: "remote", want: "unsupported"},
|
||||||
|
{name: "deterministic", class: contracts.ExecutionClassDeterministic},
|
||||||
|
{name: "llm backed", class: contracts.ExecutionClassLLMBacked},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
spec := normalizeModuleSpec(ModuleSpec{Key: "module", Stage: StageChunk, ExecutionClass: test.class})
|
||||||
|
err := validateModuleSpec("chunker", StageChunk, spec)
|
||||||
|
if test.want == "" && err != nil {
|
||||||
|
t.Fatalf("validateModuleSpec() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if test.want != "" && (err == nil || !strings.Contains(err.Error(), test.want)) {
|
||||||
|
t.Fatalf("validateModuleSpec() error = %v, want %q", err, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCloneModuleSpecPreservesExecutionClass(t *testing.T) {
|
||||||
|
spec := normalizeModuleSpec(ModuleSpec{Key: " module ", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked})
|
||||||
|
cloned := cloneModuleSpec(spec)
|
||||||
|
if !reflect.DeepEqual(cloned, spec) {
|
||||||
|
t.Fatalf("cloneModuleSpec() = %#v, want %#v", cloned, spec)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
|
func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -22,8 +55,9 @@ func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
|
|||||||
for _, test := range tests {
|
for _, test := range tests {
|
||||||
t.Run(test.name, func(t *testing.T) {
|
t.Run(test.name, func(t *testing.T) {
|
||||||
spec := normalizeModuleSpec(ModuleSpec{
|
spec := normalizeModuleSpec(ModuleSpec{
|
||||||
Key: "module",
|
Key: "module",
|
||||||
Stage: test.stage,
|
Stage: test.stage,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster", Description: "Character roster", MaxBytes: 1024},
|
{Name: "roster", Description: "Character roster", MaxBytes: 1024},
|
||||||
},
|
},
|
||||||
@@ -50,8 +84,9 @@ func TestValidateModuleSpecRejectsReferenceSlotsForIneligibleStages(t *testing.T
|
|||||||
for _, test := range tests {
|
for _, test := range tests {
|
||||||
t.Run(test.name, func(t *testing.T) {
|
t.Run(test.name, func(t *testing.T) {
|
||||||
spec := normalizeModuleSpec(ModuleSpec{
|
spec := normalizeModuleSpec(ModuleSpec{
|
||||||
Key: "module",
|
Key: "module",
|
||||||
Stage: test.stage,
|
Stage: test.stage,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster"},
|
{Name: "roster"},
|
||||||
},
|
},
|
||||||
@@ -109,6 +144,7 @@ func TestValidateModuleSpecRejectsInvalidReferenceSlotsForEligibleStages(t *test
|
|||||||
spec := normalizeModuleSpec(ModuleSpec{
|
spec := normalizeModuleSpec(ModuleSpec{
|
||||||
Key: "module",
|
Key: "module",
|
||||||
Stage: stage.stage,
|
Stage: stage.stage,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
ReferenceSlots: invalid.slots,
|
ReferenceSlots: invalid.slots,
|
||||||
})
|
})
|
||||||
err := validateModuleSpec(stage.kind, stage.stage, spec)
|
err := validateModuleSpec(stage.kind, stage.stage, spec)
|
||||||
|
|||||||
@@ -26,7 +26,7 @@ func TestNormalizerRegistryErasureClonesRetryDirective(t *testing.T) {
|
|||||||
FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "omitted", Message: "fallback warning"}},
|
FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "omitted", Message: "fallback warning"}},
|
||||||
}
|
}
|
||||||
registry := NewNormalizerRegistry()
|
registry := NewNormalizerRegistry()
|
||||||
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
||||||
return retryingNotesNormalizer{warnings: warnings, retry: retry}, nil
|
return retryingNotesNormalizer{warnings: warnings, retry: retry}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterNormalizer() error = %v", err)
|
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||||
|
|||||||
@@ -32,10 +32,6 @@ func NewOutputEncoderRegistry() *OutputEncoderRegistry {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *OutputEncoderRegistry) Register(key string, constructor OutputEncoderConstructor) error {
|
|
||||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageOutput), constructor)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor OutputEncoderConstructor) error {
|
func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor OutputEncoderConstructor) error {
|
||||||
if constructor == nil {
|
if constructor == nil {
|
||||||
return fmt.Errorf("output encoder constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
return fmt.Errorf("output encoder constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
|
|||||||
@@ -17,7 +17,9 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
|
|||||||
return NewOutputEncoderRegistry()
|
return NewOutputEncoderRegistry()
|
||||||
},
|
},
|
||||||
register: func(registry any, key string, constructor func() (contracts.OutputEncoder, error)) error {
|
register: func(registry any, key string, constructor func() (contracts.OutputEncoder, error)) error {
|
||||||
return registry.(*OutputEncoderRegistry).Register(key, constructor)
|
return registry.(*OutputEncoderRegistry).RegisterWithSpec(ModuleSpec{
|
||||||
|
Key: key, Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
}, constructor)
|
||||||
},
|
},
|
||||||
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.OutputEncoder, error)) error {
|
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.OutputEncoder, error)) error {
|
||||||
return registry.(*OutputEncoderRegistry).RegisterWithSpec(spec, constructor)
|
return registry.(*OutputEncoderRegistry).RegisterWithSpec(spec, constructor)
|
||||||
@@ -33,7 +35,9 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
|
|||||||
},
|
},
|
||||||
nilRegister: func(key string, constructor func() (contracts.OutputEncoder, error)) error {
|
nilRegister: func(key string, constructor func() (contracts.OutputEncoder, error)) error {
|
||||||
var registry *OutputEncoderRegistry
|
var registry *OutputEncoderRegistry
|
||||||
return registry.Register(key, constructor)
|
return registry.RegisterWithSpec(ModuleSpec{
|
||||||
|
Key: key, Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
}, constructor)
|
||||||
},
|
},
|
||||||
nilBuild: func(key string) (contracts.OutputEncoder, error) {
|
nilBuild: func(key string) (contracts.OutputEncoder, error) {
|
||||||
var registry *OutputEncoderRegistry
|
var registry *OutputEncoderRegistry
|
||||||
@@ -60,7 +64,7 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
|
|||||||
|
|
||||||
func TestOutputProfileValidationReceivesOwnedOptionsAndLaneIDs(t *testing.T) {
|
func TestOutputProfileValidationReceivesOwnedOptionsAndLaneIDs(t *testing.T) {
|
||||||
registry := NewOutputEncoderRegistry()
|
registry := NewOutputEncoderRegistry()
|
||||||
if err := registry.RegisterBuilderWithProfileValidation(defaultModuleSpec("profile-output", StageOutput), func(options map[string]any) error {
|
if err := registry.RegisterBuilderWithProfileValidation(testModuleSpec("profile-output", StageOutput), func(options map[string]any) error {
|
||||||
options["nested"].(map[string]any)["value"] = "changed"
|
options["nested"].(map[string]any)["value"] = "changed"
|
||||||
return nil
|
return nil
|
||||||
}, func(context OutputProfileOptionContext, options map[string]any) error {
|
}, func(context OutputProfileOptionContext, options map[string]any) error {
|
||||||
|
|||||||
@@ -480,19 +480,19 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
|||||||
if err := RegisterArtifactCodec(registries.ArtifactCodecs, notesCodec()); err != nil {
|
if err := RegisterArtifactCodec(registries.ArtifactCodecs, notesCodec()); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := registries.Inputs.RegisterBuilderWithSpec(defaultModuleSpec("input", StageInput), strict, func(request BuildRequest) (contracts.InputAdapter, error) {
|
if err := registries.Inputs.RegisterBuilderWithSpec(testModuleSpec("input", StageInput), strict, func(request BuildRequest) (contracts.InputAdapter, error) {
|
||||||
record("input", &request)
|
record("input", &request)
|
||||||
return input, nil
|
return input, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := registries.Chunkers.RegisterBuilderWithSpec(defaultModuleSpec("chunk", StageChunk), strict, func(request BuildRequest) (contracts.Chunker, error) {
|
if err := registries.Chunkers.RegisterBuilderWithSpec(testModuleSpec("chunk", StageChunk), strict, func(request BuildRequest) (contracts.Chunker, error) {
|
||||||
record("chunk", &request)
|
record("chunk", &request)
|
||||||
return &typedTestChunker{key: "chunk"}, nil
|
return &typedTestChunker{key: "chunk"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
extractSpec := defaultModuleSpec("extract", StageExtract)
|
extractSpec := testModuleSpec("extract", StageExtract)
|
||||||
extractSpec.ArtifactKind = "test/notes"
|
extractSpec.ArtifactKind = "test/notes"
|
||||||
if err := RegisterExtractorBuilder(registries.Extractors, extractSpec, strict, func(request BuildRequest) (contracts.Extractor[codecNotes], error) {
|
if err := RegisterExtractorBuilder(registries.Extractors, extractSpec, strict, func(request BuildRequest) (contracts.Extractor[codecNotes], error) {
|
||||||
record("extract", &request)
|
record("extract", &request)
|
||||||
@@ -503,7 +503,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
mergeSpec := defaultModuleSpec("merge", StageMerge)
|
mergeSpec := testModuleSpec("merge", StageMerge)
|
||||||
mergeSpec.ArtifactKind = "test/notes"
|
mergeSpec.ArtifactKind = "test/notes"
|
||||||
if err := RegisterMergerBuilder(registries.Mergers, mergeSpec, strict, func(request BuildRequest) (contracts.Merger[codecNotes], error) {
|
if err := RegisterMergerBuilder(registries.Mergers, mergeSpec, strict, func(request BuildRequest) (contracts.Merger[codecNotes], error) {
|
||||||
record("merge", &request)
|
record("merge", &request)
|
||||||
@@ -511,7 +511,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
normalizeSpec := defaultModuleSpec("normalize", StageNormalize)
|
normalizeSpec := testModuleSpec("normalize", StageNormalize)
|
||||||
normalizeSpec.ArtifactKind = "test/notes"
|
normalizeSpec.ArtifactKind = "test/notes"
|
||||||
if err := RegisterNormalizerBuilder(registries.Normalizers, normalizeSpec, strict, func(request BuildRequest) (contracts.Normalizer[codecNotes], error) {
|
if err := RegisterNormalizerBuilder(registries.Normalizers, normalizeSpec, strict, func(request BuildRequest) (contracts.Normalizer[codecNotes], error) {
|
||||||
record("normalize", &request)
|
record("normalize", &request)
|
||||||
@@ -532,7 +532,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := registries.Outputs.RegisterBuilderWithSpec(defaultModuleSpec("output", StageOutput), strict, func(request BuildRequest) (contracts.OutputEncoder, error) {
|
if err := registries.Outputs.RegisterBuilderWithSpec(testModuleSpec("output", StageOutput), strict, func(request BuildRequest) (contracts.OutputEncoder, error) {
|
||||||
record("output", &request)
|
record("output", &request)
|
||||||
if failure.output != nil {
|
if failure.output != nil {
|
||||||
return nil, failure.output
|
return nil, failure.output
|
||||||
|
|||||||
@@ -117,6 +117,7 @@ type PipelineStepProfile struct {
|
|||||||
|
|
||||||
type PipelineProfile struct {
|
type PipelineProfile struct {
|
||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
|
LLMProfile string `json:"llm_profile,omitempty"`
|
||||||
Input ModuleBinding `json:"input"`
|
Input ModuleBinding `json:"input"`
|
||||||
Chunk ModuleBinding `json:"chunk,omitempty"`
|
Chunk ModuleBinding `json:"chunk,omitempty"`
|
||||||
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
|
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
|
||||||
@@ -127,6 +128,7 @@ type PipelineProfile struct {
|
|||||||
|
|
||||||
type ResolveOptions struct {
|
type ResolveOptions struct {
|
||||||
Only []string
|
Only []string
|
||||||
|
LLMProfileOverride string
|
||||||
ReferenceOverrides []ReferenceBinding
|
ReferenceOverrides []ReferenceBinding
|
||||||
ReferenceUnbinds []ReferenceUnbind
|
ReferenceUnbinds []ReferenceUnbind
|
||||||
}
|
}
|
||||||
@@ -156,20 +158,23 @@ type ResolvedReferenceTarget struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type ResolvedArtifactLane struct {
|
type ResolvedArtifactLane struct {
|
||||||
StepID string
|
StepID string
|
||||||
ID string
|
ID string
|
||||||
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
|
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
|
||||||
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
|
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
|
||||||
ArtifactSchemaName string `json:"artifact_schema_name,omitempty"`
|
ArtifactSchemaName string `json:"artifact_schema_name,omitempty"`
|
||||||
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
|
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
|
||||||
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
|
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
|
||||||
Extract ModuleBinding
|
Extract ModuleBinding
|
||||||
Merge ModuleBinding
|
ExtractExecutionClass contracts.ExecutionClass `json:"extract_execution_class"`
|
||||||
Normalize ModuleBinding
|
Merge ModuleBinding
|
||||||
Validators []ModuleBinding
|
MergeExecutionClass contracts.ExecutionClass `json:"merge_execution_class"`
|
||||||
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
|
Normalize ModuleBinding
|
||||||
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
|
NormalizeExecutionClass contracts.ExecutionClass `json:"normalize_execution_class"`
|
||||||
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
|
Validators []ModuleBinding
|
||||||
|
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
|
||||||
|
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
|
||||||
|
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type ResolvedPipelineStep struct {
|
type ResolvedPipelineStep struct {
|
||||||
@@ -192,14 +197,17 @@ type ResolvedValidator struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type ResolvedPipeline struct {
|
type ResolvedPipeline struct {
|
||||||
ID string
|
ID string
|
||||||
Digest string
|
Digest string
|
||||||
Input ModuleBinding
|
Input ModuleBinding
|
||||||
Chunk ModuleBinding
|
InputExecutionClass contracts.ExecutionClass `json:"input_execution_class"`
|
||||||
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
|
Chunk ModuleBinding
|
||||||
Steps []ResolvedPipelineStep
|
ChunkExecutionClass contracts.ExecutionClass `json:"chunk_execution_class"`
|
||||||
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
|
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
|
||||||
Output ModuleBinding
|
Steps []ResolvedPipelineStep
|
||||||
|
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
|
||||||
|
Output ModuleBinding
|
||||||
|
OutputExecutionClass contracts.ExecutionClass `json:"output_execution_class"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// AllArtifactLanes returns lanes in deterministic step order for read-only
|
// AllArtifactLanes returns lanes in deterministic step order for read-only
|
||||||
@@ -225,6 +233,46 @@ type ModuleCatalog struct {
|
|||||||
Outputs *OutputEncoderRegistry
|
Outputs *OutputEncoderRegistry
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ExecutionClass returns the registered execution class for a module selected
|
||||||
|
// by stage and key without constructing the module.
|
||||||
|
func (catalog ModuleCatalog) ExecutionClass(stage ModuleStage, key string) (contracts.ExecutionClass, bool) {
|
||||||
|
var executionClass contracts.ExecutionClass
|
||||||
|
var ok bool
|
||||||
|
|
||||||
|
switch stage {
|
||||||
|
case StageInput:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Inputs.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageChunk:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Chunkers.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageExtract:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Extractors.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageMerge:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Mergers.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageNormalize:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Normalizers.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageValidate:
|
||||||
|
var spec ValidatorSpec
|
||||||
|
spec, ok = catalog.Validators.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageOutput:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Outputs.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
}
|
||||||
|
|
||||||
|
return executionClass, ok
|
||||||
|
}
|
||||||
|
|
||||||
func Binding(module string) ModuleBinding {
|
func Binding(module string) ModuleBinding {
|
||||||
return ModuleBinding{Module: strings.TrimSpace(module)}
|
return ModuleBinding{Module: strings.TrimSpace(module)}
|
||||||
}
|
}
|
||||||
@@ -316,11 +364,13 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
|||||||
return ResolvedPipeline{}, err
|
return ResolvedPipeline{}, err
|
||||||
}
|
}
|
||||||
resolved := ResolvedPipeline{
|
resolved := ResolvedPipeline{
|
||||||
ID: pipelineID,
|
ID: pipelineID,
|
||||||
Input: input,
|
Input: input,
|
||||||
Chunk: chunk,
|
InputExecutionClass: inputModuleSpec.ExecutionClass,
|
||||||
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
|
Chunk: chunk,
|
||||||
Output: resolveBinding(profile.Output, DefaultOutputModule),
|
ChunkExecutionClass: chunkSpec.ExecutionClass,
|
||||||
|
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
|
||||||
|
Output: resolveBinding(profile.Output, DefaultOutputModule),
|
||||||
}
|
}
|
||||||
chunkValidatorChain, err := resolveValidatorChain(pipelineID, "", StageChunk, chunk.Module, chunk.Validators, "", nil, catalog)
|
chunkValidatorChain, err := resolveValidatorChain(pipelineID, "", StageChunk, chunk.Module, chunk.Validators, "", nil, catalog)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -386,6 +436,10 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
|||||||
if missing, ok := outputCapabilities.missing(outputSpec.Requires); ok {
|
if missing, ok := outputCapabilities.missing(outputSpec.Requires); ok {
|
||||||
return ResolvedPipeline{}, capabilityError(pipelineID, "", StageOutput, resolved.Output.Module, missing)
|
return ResolvedPipeline{}, capabilityError(pipelineID, "", StageOutput, resolved.Output.Module, missing)
|
||||||
}
|
}
|
||||||
|
resolved.OutputExecutionClass = outputSpec.ExecutionClass
|
||||||
|
if err := applyEffectiveLLMProfiles(&resolved, profile.LLMProfile, options.LLMProfileOverride); err != nil {
|
||||||
|
return ResolvedPipeline{}, err
|
||||||
|
}
|
||||||
if err := validateResolvedOptions(resolved, catalog, configuredLaneIDs); err != nil {
|
if err := validateResolvedOptions(resolved, catalog, configuredLaneIDs); err != nil {
|
||||||
return ResolvedPipeline{}, err
|
return ResolvedPipeline{}, err
|
||||||
}
|
}
|
||||||
@@ -475,6 +529,7 @@ func resolveArtifactLane(
|
|||||||
}
|
}
|
||||||
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
|
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
|
||||||
lane.ExtractReferences.StepID = strings.TrimSpace(stepID)
|
lane.ExtractReferences.StepID = strings.TrimSpace(stepID)
|
||||||
|
lane.ExtractExecutionClass = extractSpec.ExecutionClass
|
||||||
capabilities.add(extractSpec.Provides...)
|
capabilities.add(extractSpec.Provides...)
|
||||||
|
|
||||||
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
|
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
|
||||||
@@ -499,6 +554,7 @@ func resolveArtifactLane(
|
|||||||
}
|
}
|
||||||
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
|
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
|
||||||
lane.MergeReferences.StepID = strings.TrimSpace(stepID)
|
lane.MergeReferences.StepID = strings.TrimSpace(stepID)
|
||||||
|
lane.MergeExecutionClass = mergeSpec.ExecutionClass
|
||||||
capabilities.add(mergeSpec.Provides...)
|
capabilities.add(mergeSpec.Provides...)
|
||||||
|
|
||||||
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
|
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
|
||||||
@@ -523,6 +579,7 @@ func resolveArtifactLane(
|
|||||||
}
|
}
|
||||||
lane.NormalizeReferences = referenceTarget(StageNormalize, laneID, lane.Normalize.Module, normalizeReferences)
|
lane.NormalizeReferences = referenceTarget(StageNormalize, laneID, lane.Normalize.Module, normalizeReferences)
|
||||||
lane.NormalizeReferences.StepID = strings.TrimSpace(stepID)
|
lane.NormalizeReferences.StepID = strings.TrimSpace(stepID)
|
||||||
|
lane.NormalizeExecutionClass = normalizeSpec.ExecutionClass
|
||||||
capabilities.add(normalizeSpec.Provides...)
|
capabilities.add(normalizeSpec.Provides...)
|
||||||
|
|
||||||
if len(lane.Validators) > 0 {
|
if len(lane.Validators) > 0 {
|
||||||
@@ -784,9 +841,6 @@ func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage,
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q: %w", pipelineID, stage, chain.ModuleKey, err)
|
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q: %w", pipelineID, stage, chain.ModuleKey, err)
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(validator.LLMProfile) != "" && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
|
|
||||||
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q assigns llm_profile to deterministic validator %q", pipelineID, stage, chain.ModuleKey, validator.Module)
|
|
||||||
}
|
|
||||||
chain.Validators = append(chain.Validators, ResolvedValidator{
|
chain.Validators = append(chain.Validators, ResolvedValidator{
|
||||||
Binding: cloneModuleBinding(validator),
|
Binding: cloneModuleBinding(validator),
|
||||||
ExecutionClass: spec.ExecutionClass,
|
ExecutionClass: spec.ExecutionClass,
|
||||||
@@ -1198,6 +1252,66 @@ func resolveBinding(binding ModuleBinding, defaultModule string) ModuleBinding {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func applyEffectiveLLMProfiles(resolved *ResolvedPipeline, pipelineProfile, overrideProfile string) error {
|
||||||
|
pipelineProfile = strings.TrimSpace(pipelineProfile)
|
||||||
|
overrideProfile = strings.TrimSpace(overrideProfile)
|
||||||
|
|
||||||
|
apply := func(stage ModuleStage, laneID, module string, binding *ModuleBinding, executionClass contracts.ExecutionClass, kind string) error {
|
||||||
|
binding.LLMProfile = strings.TrimSpace(binding.LLMProfile)
|
||||||
|
if executionClass != contracts.ExecutionClassLLMBacked {
|
||||||
|
if binding.LLMProfile != "" {
|
||||||
|
if laneID == "" {
|
||||||
|
return fmt.Errorf("pipeline %q %s %q assigns llm_profile to deterministic %s %q", resolved.ID, stage, module, kind, binding.Module)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("pipeline %q lane %q %s %q assigns llm_profile to deterministic %s %q", resolved.ID, laneID, stage, module, kind, binding.Module)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if overrideProfile != "" {
|
||||||
|
binding.LLMProfile = overrideProfile
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if binding.LLMProfile == "" {
|
||||||
|
binding.LLMProfile = pipelineProfile
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := apply(StageInput, "", resolved.Input.Module, &resolved.Input, resolved.InputExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := apply(StageChunk, "", resolved.Chunk.Module, &resolved.Chunk, resolved.ChunkExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for stepIndex := range resolved.Steps {
|
||||||
|
for laneIndex := range resolved.Steps[stepIndex].ArtifactLanes {
|
||||||
|
lane := &resolved.Steps[stepIndex].ArtifactLanes[laneIndex]
|
||||||
|
if err := apply(StageExtract, lane.ID, lane.Extract.Module, &lane.Extract, lane.ExtractExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := apply(StageMerge, lane.ID, lane.Merge.Module, &lane.Merge, lane.MergeExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := apply(StageNormalize, lane.ID, lane.Normalize.Module, &lane.Normalize, lane.NormalizeExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := apply(StageOutput, "", resolved.Output.Module, &resolved.Output, resolved.OutputExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for chainIndex := range resolved.ValidatorChains {
|
||||||
|
chain := &resolved.ValidatorChains[chainIndex]
|
||||||
|
for validatorIndex := range chain.Validators {
|
||||||
|
validator := &chain.Validators[validatorIndex]
|
||||||
|
if err := apply(chain.Stage, chain.LaneID, chain.ModuleKey, &validator.Binding, validator.ExecutionClass, "validator"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func resolveBindings(bindings []ModuleBinding, defaultModule string) []ModuleBinding {
|
func resolveBindings(bindings []ModuleBinding, defaultModule string) []ModuleBinding {
|
||||||
if len(bindings) == 0 {
|
if len(bindings) == 0 {
|
||||||
return nil
|
return nil
|
||||||
@@ -1307,21 +1421,27 @@ func selectedArtifactLanes(pipelineID string, artifacts map[string]ArtifactLaneP
|
|||||||
|
|
||||||
func resolvedPipelineDigest(resolved ResolvedPipeline) (string, error) {
|
func resolvedPipelineDigest(resolved ResolvedPipeline) (string, error) {
|
||||||
withoutDigest := struct {
|
withoutDigest := struct {
|
||||||
ID string
|
ID string
|
||||||
Input ModuleBinding
|
Input ModuleBinding
|
||||||
Chunk ModuleBinding
|
InputExecutionClass contracts.ExecutionClass
|
||||||
ChunkReferences ResolvedReferenceTarget
|
Chunk ModuleBinding
|
||||||
Steps []ResolvedPipelineStep
|
ChunkExecutionClass contracts.ExecutionClass
|
||||||
ValidatorChains []ResolvedValidatorChain
|
ChunkReferences ResolvedReferenceTarget
|
||||||
Output ModuleBinding
|
Steps []ResolvedPipelineStep
|
||||||
|
ValidatorChains []ResolvedValidatorChain
|
||||||
|
Output ModuleBinding
|
||||||
|
OutputExecutionClass contracts.ExecutionClass
|
||||||
}{
|
}{
|
||||||
ID: resolved.ID,
|
ID: resolved.ID,
|
||||||
Input: resolved.Input,
|
Input: resolved.Input,
|
||||||
Chunk: resolved.Chunk,
|
InputExecutionClass: resolved.InputExecutionClass,
|
||||||
ChunkReferences: resolved.ChunkReferences,
|
Chunk: resolved.Chunk,
|
||||||
Steps: resolved.Steps,
|
ChunkExecutionClass: resolved.ChunkExecutionClass,
|
||||||
ValidatorChains: resolved.ValidatorChains,
|
ChunkReferences: resolved.ChunkReferences,
|
||||||
Output: resolved.Output,
|
Steps: resolved.Steps,
|
||||||
|
ValidatorChains: resolved.ValidatorChains,
|
||||||
|
Output: resolved.Output,
|
||||||
|
OutputExecutionClass: resolved.OutputExecutionClass,
|
||||||
}
|
}
|
||||||
encoded, err := json.Marshal(withoutDigest)
|
encoded, err := json.Marshal(withoutDigest)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -15,22 +15,22 @@ import (
|
|||||||
func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
||||||
catalog := newProfileCatalog(t)
|
catalog := newProfileCatalog(t)
|
||||||
registerProfileSpecs(t, catalog,
|
registerProfileSpecs(t, catalog,
|
||||||
ModuleSpec{Key: "window", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
ModuleSpec{Key: "window", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||||
ModuleSpec{Key: "record-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
ModuleSpec{Key: "record-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||||
ModuleSpec{Key: "dedupe", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
ModuleSpec{Key: "dedupe", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||||
ModuleSpec{Key: "canonical", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
ModuleSpec{Key: "canonical", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||||
ModuleSpec{Key: "ndjson", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
ModuleSpec{Key: "ndjson", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||||
)
|
)
|
||||||
|
|
||||||
resolved, err := ResolvePipeline(PipelineProfile{
|
resolved, err := ResolvePipeline(PipelineProfile{
|
||||||
ID: " campaign ",
|
ID: " campaign ",
|
||||||
Input: ModuleBinding{Module: " text ", LLMProfile: " fast "},
|
Input: Binding(" text "),
|
||||||
Chunk: ModuleBinding{Module: " window ", Options: map[string]any{
|
Chunk: ModuleBinding{Module: " window ", Options: map[string]any{
|
||||||
"size": 10,
|
"size": 10,
|
||||||
}},
|
}},
|
||||||
Artifacts: map[string]ArtifactLaneProfile{
|
Artifacts: map[string]ArtifactLaneProfile{
|
||||||
" records ": {
|
" records ": {
|
||||||
Extract: ModuleBinding{Module: " record-extractor ", LLMProfile: " careful "},
|
Extract: Binding(" record-extractor "),
|
||||||
Merge: Binding(" dedupe "),
|
Merge: Binding(" dedupe "),
|
||||||
Normalize: Binding(" canonical "),
|
Normalize: Binding(" canonical "),
|
||||||
},
|
},
|
||||||
@@ -44,15 +44,21 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
|||||||
if resolved.ID != "campaign" {
|
if resolved.ID != "campaign" {
|
||||||
t.Fatalf("ID = %q, want campaign", resolved.ID)
|
t.Fatalf("ID = %q, want campaign", resolved.ID)
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(resolved.Input, ModuleBinding{Module: "text", LLMProfile: "fast"}) {
|
if !reflect.DeepEqual(resolved.Input, ModuleBinding{Module: "text"}) {
|
||||||
t.Fatalf("Input = %#v, want trimmed explicit input", resolved.Input)
|
t.Fatalf("Input = %#v, want trimmed explicit input", resolved.Input)
|
||||||
}
|
}
|
||||||
|
if resolved.InputExecutionClass != contracts.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("InputExecutionClass = %q, want deterministic", resolved.InputExecutionClass)
|
||||||
|
}
|
||||||
if resolved.Chunk.Module != "window" || resolved.Chunk.LLMProfile != "" {
|
if resolved.Chunk.Module != "window" || resolved.Chunk.LLMProfile != "" {
|
||||||
t.Fatalf("Chunk = %#v, want explicit module and empty LLM profile", resolved.Chunk)
|
t.Fatalf("Chunk = %#v, want explicit module and empty LLM profile", resolved.Chunk)
|
||||||
}
|
}
|
||||||
if resolved.Chunk.Options["size"] != 10 {
|
if resolved.Chunk.Options["size"] != 10 {
|
||||||
t.Fatalf("Chunk.Options = %#v, want size option", resolved.Chunk.Options)
|
t.Fatalf("Chunk.Options = %#v, want size option", resolved.Chunk.Options)
|
||||||
}
|
}
|
||||||
|
if resolved.ChunkExecutionClass != contracts.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("ChunkExecutionClass = %q, want deterministic", resolved.ChunkExecutionClass)
|
||||||
|
}
|
||||||
if len(resolved.Steps) != 1 || resolved.Steps[0].ID != "default" || len(resolved.Steps[0].ArtifactLanes) != 1 {
|
if len(resolved.Steps) != 1 || resolved.Steps[0].ID != "default" || len(resolved.Steps[0].ArtifactLanes) != 1 {
|
||||||
t.Fatalf("resolved steps = %#v, want one default step with one lane", resolved.Steps)
|
t.Fatalf("resolved steps = %#v, want one default step with one lane", resolved.Steps)
|
||||||
}
|
}
|
||||||
@@ -60,9 +66,12 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
|||||||
if lane.ID != "records" {
|
if lane.ID != "records" {
|
||||||
t.Fatalf("lane.ID = %q, want records", lane.ID)
|
t.Fatalf("lane.ID = %q, want records", lane.ID)
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(lane.Extract, ModuleBinding{Module: "record-extractor", LLMProfile: "careful"}) {
|
if !reflect.DeepEqual(lane.Extract, ModuleBinding{Module: "record-extractor"}) {
|
||||||
t.Fatalf("lane.Extract = %#v, want explicit extractor", lane.Extract)
|
t.Fatalf("lane.Extract = %#v, want explicit extractor", lane.Extract)
|
||||||
}
|
}
|
||||||
|
if lane.ExtractExecutionClass != contracts.ExecutionClassDeterministic || lane.MergeExecutionClass != contracts.ExecutionClassDeterministic || lane.NormalizeExecutionClass != contracts.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("lane execution classes = %q/%q/%q, want deterministic", lane.ExtractExecutionClass, lane.MergeExecutionClass, lane.NormalizeExecutionClass)
|
||||||
|
}
|
||||||
if lane.Merge.Module != "dedupe" || lane.Normalize.Module != "canonical" {
|
if lane.Merge.Module != "dedupe" || lane.Normalize.Module != "canonical" {
|
||||||
t.Fatalf("lane merge/normalize = %#v/%#v, want explicit modules", lane.Merge, lane.Normalize)
|
t.Fatalf("lane merge/normalize = %#v/%#v, want explicit modules", lane.Merge, lane.Normalize)
|
||||||
}
|
}
|
||||||
@@ -72,11 +81,51 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
|||||||
if resolved.Output.Module != "ndjson" {
|
if resolved.Output.Module != "ndjson" {
|
||||||
t.Fatalf("Output.Module = %q, want ndjson", resolved.Output.Module)
|
t.Fatalf("Output.Module = %q, want ndjson", resolved.Output.Module)
|
||||||
}
|
}
|
||||||
|
if resolved.OutputExecutionClass != contracts.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("OutputExecutionClass = %q, want deterministic", resolved.OutputExecutionClass)
|
||||||
|
}
|
||||||
if !strings.HasPrefix(resolved.Digest, "sha256:") {
|
if !strings.HasPrefix(resolved.Digest, "sha256:") {
|
||||||
t.Fatalf("Digest = %q, want sha256 digest", resolved.Digest)
|
t.Fatalf("Digest = %q, want sha256 digest", resolved.Digest)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestModuleCatalogExecutionClassLooksUpRegisteredMetadata(t *testing.T) {
|
||||||
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
|
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
ModuleSpec{Key: "llm-extract", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
)
|
||||||
|
registerProfileValidatorSpec(t, catalog, ValidatorSpec{Key: "llm-validator", ExecutionClass: contracts.ExecutionClassLLMBacked})
|
||||||
|
|
||||||
|
for _, test := range []struct {
|
||||||
|
stage ModuleStage
|
||||||
|
key string
|
||||||
|
want contracts.ExecutionClass
|
||||||
|
}{
|
||||||
|
{stage: StageInput, key: "llm-input", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageChunk, key: "llm-chunk", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageExtract, key: "llm-extract", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageMerge, key: "llm-merge", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageNormalize, key: "llm-normalize", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageValidate, key: "llm-validator", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageOutput, key: "llm-output", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
} {
|
||||||
|
t.Run(string(test.stage), func(t *testing.T) {
|
||||||
|
got, ok := catalog.ExecutionClass(test.stage, test.key)
|
||||||
|
if !ok || got != test.want {
|
||||||
|
t.Fatalf("ExecutionClass(%q, %q) = %q, %t; want %q, true", test.stage, test.key, got, ok, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, ok := catalog.ExecutionClass(StageExtract, "missing"); ok {
|
||||||
|
t.Fatal("ExecutionClass() found an unregistered module")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestResolvePipelineAppliesDefaults(t *testing.T) {
|
func TestResolvePipelineAppliesDefaults(t *testing.T) {
|
||||||
resolved, err := ResolvePipeline(PipelineProfile{
|
resolved, err := ResolvePipeline(PipelineProfile{
|
||||||
ID: "defaulted",
|
ID: "defaulted",
|
||||||
@@ -110,6 +159,176 @@ func TestResolvePipelineAppliesDefaults(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestResolvePipelineAppliesEffectiveLLMProfiles(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
profile PipelineProfile
|
||||||
|
options ResolveOptions
|
||||||
|
want map[string]string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "runtime override",
|
||||||
|
profile: func() PipelineProfile {
|
||||||
|
profile := llmProfilePipeline()
|
||||||
|
profile.LLMProfile = " pipeline "
|
||||||
|
profile.Chunk.LLMProfile = "binding"
|
||||||
|
return profile
|
||||||
|
}(),
|
||||||
|
options: ResolveOptions{LLMProfileOverride: " runtime "},
|
||||||
|
want: llmProfileValues("runtime"),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "binding exception",
|
||||||
|
profile: func() PipelineProfile {
|
||||||
|
profile := llmProfilePipeline()
|
||||||
|
profile.LLMProfile = "pipeline"
|
||||||
|
lane := profile.Artifacts["events"]
|
||||||
|
lane.Extract.LLMProfile = "extract"
|
||||||
|
lane.Extract.Validators = ValidatorOverride{
|
||||||
|
Set: true,
|
||||||
|
Validators: []ModuleBinding{{Module: "llm-validator", LLMProfile: "validator"}},
|
||||||
|
}
|
||||||
|
profile.Artifacts["events"] = lane
|
||||||
|
return profile
|
||||||
|
}(),
|
||||||
|
want: func() map[string]string {
|
||||||
|
values := llmProfileValues("pipeline")
|
||||||
|
values["extract"] = "extract"
|
||||||
|
values["validator:extract:events"] = "validator"
|
||||||
|
return values
|
||||||
|
}(),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "pipeline default",
|
||||||
|
profile: func() PipelineProfile {
|
||||||
|
profile := llmProfilePipeline()
|
||||||
|
profile.LLMProfile = "pipeline"
|
||||||
|
return profile
|
||||||
|
}(),
|
||||||
|
want: llmProfileValues("pipeline"),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "prompt fallback",
|
||||||
|
profile: llmProfilePipeline(),
|
||||||
|
want: llmProfileValues(""),
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
resolved, err := ResolvePipeline(test.profile, test.options, llmProfileCatalog(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if got := resolvedLLMProfileValues(resolved); !reflect.DeepEqual(got, test.want) {
|
||||||
|
t.Fatalf("resolved profiles = %#v, want %#v", got, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePipelineAppliesProfilesOnlyToSelectedLLMBackedBindings(t *testing.T) {
|
||||||
|
profile := llmProfilePipeline()
|
||||||
|
profile.LLMProfile = "pipeline"
|
||||||
|
profile.Artifacts["notes"] = ArtifactLaneProfile{Extract: Binding("llm-extractor")}
|
||||||
|
|
||||||
|
resolved, err := ResolvePipeline(profile, ResolveOptions{Only: []string{"events"}}, llmProfileCatalog(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if got := laneIDs(resolved.Steps[0].ArtifactLanes); !reflect.DeepEqual(got, []string{"events"}) {
|
||||||
|
t.Fatalf("selected lanes = %#v, want events only", got)
|
||||||
|
}
|
||||||
|
if got := resolvedLLMProfileValues(resolved); !reflect.DeepEqual(got, llmProfileValues("pipeline")) {
|
||||||
|
t.Fatalf("resolved profiles = %#v, want selected LLM bindings only", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePipelineLeavesUnusedProfilesOffDeterministicBindings(t *testing.T) {
|
||||||
|
profile := baselineProfile()
|
||||||
|
profile.LLMProfile = "unused"
|
||||||
|
|
||||||
|
resolved, err := ResolvePipeline(profile, ResolveOptions{LLMProfileOverride: "also-unused"}, newProfileCatalog(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if resolved.Input.LLMProfile != "" || resolved.Chunk.LLMProfile != "" || resolved.Output.LLMProfile != "" {
|
||||||
|
t.Fatalf("deterministic pipeline profiles = input %q chunk %q output %q, want empty", resolved.Input.LLMProfile, resolved.Chunk.LLMProfile, resolved.Output.LLMProfile)
|
||||||
|
}
|
||||||
|
lane := resolved.Steps[0].ArtifactLanes[0]
|
||||||
|
if lane.Extract.LLMProfile != "" || lane.Merge.LLMProfile != "" || lane.Normalize.LLMProfile != "" {
|
||||||
|
t.Fatalf("deterministic lane profiles = extract %q merge %q normalize %q, want empty", lane.Extract.LLMProfile, lane.Merge.LLMProfile, lane.Normalize.LLMProfile)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePipelineRejectsLLMProfileForDeterministicModule(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
mutate func(*PipelineProfile)
|
||||||
|
}{
|
||||||
|
{name: "input", mutate: func(profile *PipelineProfile) { profile.Input.LLMProfile = "invalid" }},
|
||||||
|
{name: "chunk", mutate: func(profile *PipelineProfile) { profile.Chunk.LLMProfile = "invalid" }},
|
||||||
|
{name: "extract", mutate: func(profile *PipelineProfile) {
|
||||||
|
lane := profile.Artifacts["events"]
|
||||||
|
lane.Extract.LLMProfile = "invalid"
|
||||||
|
profile.Artifacts["events"] = lane
|
||||||
|
}},
|
||||||
|
{name: "merge", mutate: func(profile *PipelineProfile) {
|
||||||
|
lane := profile.Artifacts["events"]
|
||||||
|
lane.Merge.LLMProfile = "invalid"
|
||||||
|
profile.Artifacts["events"] = lane
|
||||||
|
}},
|
||||||
|
{name: "normalize", mutate: func(profile *PipelineProfile) {
|
||||||
|
lane := profile.Artifacts["events"]
|
||||||
|
lane.Normalize.LLMProfile = "invalid"
|
||||||
|
profile.Artifacts["events"] = lane
|
||||||
|
}},
|
||||||
|
{name: "output", mutate: func(profile *PipelineProfile) { profile.Output.LLMProfile = "invalid" }},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
profile := baselineProfile()
|
||||||
|
test.mutate(&profile)
|
||||||
|
_, err := ResolvePipeline(profile, ResolveOptions{}, newProfileCatalog(t))
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "llm_profile") || !strings.Contains(err.Error(), "deterministic") {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want deterministic profile rejection", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePipelineDigestUsesEffectiveLLMProfiles(t *testing.T) {
|
||||||
|
inherited := llmProfilePipeline()
|
||||||
|
inherited.LLMProfile = "shared"
|
||||||
|
explicit := llmProfilePipeline()
|
||||||
|
explicit.Input.LLMProfile = "shared"
|
||||||
|
explicit.Chunk.LLMProfile = "shared"
|
||||||
|
explicit.Output.LLMProfile = "shared"
|
||||||
|
lane := explicit.Artifacts["events"]
|
||||||
|
lane.Extract.LLMProfile = "shared"
|
||||||
|
lane.Merge.LLMProfile = "shared"
|
||||||
|
lane.Normalize.LLMProfile = "shared"
|
||||||
|
explicit.Artifacts["events"] = lane
|
||||||
|
|
||||||
|
inheritedResolved, err := ResolvePipeline(inherited, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline(inherited) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
explicitResolved, err := ResolvePipeline(explicit, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline(explicit) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if inheritedResolved.Digest != explicitResolved.Digest {
|
||||||
|
t.Fatalf("effective profile digests differ: %q != %q", inheritedResolved.Digest, explicitResolved.Digest)
|
||||||
|
}
|
||||||
|
|
||||||
|
explicit.Chunk.LLMProfile = "different"
|
||||||
|
changedResolved, err := ResolvePipeline(explicit, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline(changed) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if explicitResolved.Digest == changedResolved.Digest {
|
||||||
|
t.Fatalf("digest = %q after effective profile change, want different", explicitResolved.Digest)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestResolvePipelineRecordsValidatorChains(t *testing.T) {
|
func TestResolvePipelineRecordsValidatorChains(t *testing.T) {
|
||||||
catalog := newProfileCatalog(t)
|
catalog := newProfileCatalog(t)
|
||||||
if err := catalog.ValidatorChains.Register(ValidatorChainMapping{
|
if err := catalog.ValidatorChains.Register(ValidatorChainMapping{
|
||||||
@@ -337,7 +556,8 @@ func TestResolvePipelinePreservesOrderedStepsAndExpandsGeneratedBindings(t *test
|
|||||||
catalog := newProfileCatalogWithOverrides(t,
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
ModuleSpec{
|
ModuleSpec{
|
||||||
Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes",
|
Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes",
|
||||||
Requires: []string{"chunk"}, Provides: []string{"candidate"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
Requires: []string{"chunk"}, Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}, AcceptedMediaTypes: []string{"application/json"}}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}, AcceptedMediaTypes: []string{"application/json"}}},
|
||||||
})
|
})
|
||||||
resolved, err := ResolvePipeline(PipelineProfile{
|
resolved, err := ResolvePipeline(PipelineProfile{
|
||||||
@@ -365,7 +585,7 @@ func TestResolvePipelinePreservesOrderedStepsAndExpandsGeneratedBindings(t *test
|
|||||||
|
|
||||||
func TestResolvePipelineRejectsGeneratedBindingOrderingAndKind(t *testing.T) {
|
func TestResolvePipelineRejectsGeneratedBindingOrderingAndKind(t *testing.T) {
|
||||||
catalog := newProfileCatalogWithOverrides(t,
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/other"}}}},
|
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/other"}}}},
|
||||||
)
|
)
|
||||||
_, err := ResolvePipeline(PipelineProfile{
|
_, err := ResolvePipeline(PipelineProfile{
|
||||||
ID: "invalid-order", Input: Binding("text"), Steps: []PipelineStepProfile{
|
ID: "invalid-order", Input: Binding("text"), Steps: []PipelineStepProfile{
|
||||||
@@ -400,10 +620,11 @@ func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
|
|||||||
profile.Artifacts["events"] = lane
|
profile.Artifacts["events"] = lane
|
||||||
|
|
||||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
Requires: []string{"chunk"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"candidate"},
|
Requires: []string{"chunk"},
|
||||||
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster", Required: true},
|
{Name: "roster", Required: true},
|
||||||
{Name: "lore"},
|
{Name: "lore"},
|
||||||
@@ -446,6 +667,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToChunkTarget(t *testing.
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "generic",
|
Key: "generic",
|
||||||
Stage: StageChunk,
|
Stage: StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"source"},
|
Requires: []string{"source"},
|
||||||
Provides: []string{"chunk"},
|
Provides: []string{"chunk"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide"}},
|
||||||
@@ -474,6 +696,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToExtractorTarget(t *test
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"},
|
Requires: []string{"chunk"},
|
||||||
Provides: []string{"candidate"},
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
|
||||||
@@ -502,6 +725,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToNormalizerTarget(t *tes
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "noop",
|
Key: "noop",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"merged"},
|
Requires: []string{"merged"},
|
||||||
Provides: []string{"normalized"},
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
||||||
@@ -530,6 +754,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToMergeTarget(t *testing.
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "appendorder",
|
Key: "appendorder",
|
||||||
Stage: StageMerge,
|
Stage: StageMerge,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"candidate"},
|
Requires: []string{"candidate"},
|
||||||
Provides: []string{"merged"},
|
Provides: []string{"merged"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "merge_notes"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "merge_notes"}},
|
||||||
@@ -559,10 +784,10 @@ func TestResolvePipelineAppliesOnePipelineReferenceDefaultToMultipleTargets(t *t
|
|||||||
profile := baselineProfile()
|
profile := baselineProfile()
|
||||||
profile.References = ExternalReferenceMap(map[string]string{"context": "./context.md"})
|
profile.References = ExternalReferenceMap(map[string]string{"context": "./context.md"})
|
||||||
catalog := newProfileCatalogWithOverrides(t,
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||||
ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"candidate"}, Provides: []string{"merged"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||||
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||||
)
|
)
|
||||||
|
|
||||||
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
||||||
@@ -580,10 +805,11 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedLane(t *t
|
|||||||
profile := multiLaneProfile()
|
profile := multiLaneProfile()
|
||||||
profile.References = ExternalReferenceMap(map[string]string{"notes_context": "./notes.md"})
|
profile.References = ExternalReferenceMap(map[string]string{"notes_context": "./notes.md"})
|
||||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||||
Key: "note-extractor",
|
Key: "note-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
Requires: []string{"chunk"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"candidate"},
|
Requires: []string{"chunk"},
|
||||||
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "notes_context"},
|
{Name: "notes_context"},
|
||||||
},
|
},
|
||||||
@@ -605,10 +831,11 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedNormalize
|
|||||||
lane.Normalize = Binding("note-normalizer")
|
lane.Normalize = Binding("note-normalizer")
|
||||||
profile.Artifacts["notes"] = lane
|
profile.Artifacts["notes"] = lane
|
||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "note-normalizer",
|
Key: "note-normalizer",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
Requires: []string{"merged"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"normalized"},
|
Requires: []string{"merged"},
|
||||||
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "notes_context"},
|
{Name: "notes_context"},
|
||||||
},
|
},
|
||||||
@@ -658,6 +885,7 @@ func TestResolvePipelineRejectsExtractLocalReferenceDeclaredOnlyByNormalizer(t *
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "noop",
|
Key: "noop",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"merged"},
|
Requires: []string{"merged"},
|
||||||
Provides: []string{"normalized"},
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
||||||
@@ -676,10 +904,11 @@ func TestResolvePipelineRejectsMergeLocalReferenceDeclaredOnlyByNormalizer(t *te
|
|||||||
lane.Merge.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./normalize.md"})
|
lane.Merge.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./normalize.md"})
|
||||||
profile.Artifacts["events"] = lane
|
profile.Artifacts["events"] = lane
|
||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "noop",
|
Key: "noop",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
Requires: []string{"merged"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"normalized"},
|
Requires: []string{"merged"},
|
||||||
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "normalization_notes"},
|
{Name: "normalization_notes"},
|
||||||
},
|
},
|
||||||
@@ -700,6 +929,7 @@ func TestResolvePipelineRejectsNormalizeLocalReferenceDeclaredOnlyByExtractor(t
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"},
|
Requires: []string{"chunk"},
|
||||||
Provides: []string{"candidate"},
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
|
||||||
@@ -716,6 +946,7 @@ func TestResolvePipelineRequiresBoundChunkReference(t *testing.T) {
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "generic",
|
Key: "generic",
|
||||||
Stage: StageChunk,
|
Stage: StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"source"},
|
Requires: []string{"source"},
|
||||||
Provides: []string{"chunk"},
|
Provides: []string{"chunk"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide", Required: true}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide", Required: true}},
|
||||||
@@ -732,6 +963,7 @@ func TestResolvePipelineRequiresBoundNormalizeReference(t *testing.T) {
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "noop",
|
Key: "noop",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"merged"},
|
Requires: []string{"merged"},
|
||||||
Provides: []string{"normalized"},
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes", Required: true}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes", Required: true}},
|
||||||
@@ -757,9 +989,9 @@ func TestResolvePipelineLocalReferencesOverridePipelineDefaultsForEligibleTarget
|
|||||||
lane.Normalize.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./local-normalize.md"})
|
lane.Normalize.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./local-normalize.md"})
|
||||||
profile.Artifacts["events"] = lane
|
profile.Artifacts["events"] = lane
|
||||||
catalog := newProfileCatalogWithOverrides(t,
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}}},
|
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}}},
|
||||||
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}}},
|
ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}}},
|
||||||
)
|
)
|
||||||
|
|
||||||
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
||||||
@@ -774,10 +1006,11 @@ func TestResolvePipelineLocalReferencesOverridePipelineDefaultsForEligibleTarget
|
|||||||
|
|
||||||
func TestResolvePipelineRequiresBoundReferenceSlotsForSelectedLanes(t *testing.T) {
|
func TestResolvePipelineRequiresBoundReferenceSlotsForSelectedLanes(t *testing.T) {
|
||||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
Requires: []string{"chunk"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"candidate"},
|
Requires: []string{"chunk"},
|
||||||
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster", Required: true},
|
{Name: "roster", Required: true},
|
||||||
},
|
},
|
||||||
@@ -798,10 +1031,11 @@ func TestResolvePipelineReferenceUnbindCanLeaveRequiredSlotMissing(t *testing.T)
|
|||||||
profile := baselineProfile()
|
profile := baselineProfile()
|
||||||
profile.References = ExternalReferenceMap(map[string]string{"roster": "./roster.yml"})
|
profile.References = ExternalReferenceMap(map[string]string{"roster": "./roster.yml"})
|
||||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
Requires: []string{"chunk"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"candidate"},
|
Requires: []string{"chunk"},
|
||||||
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster", Required: true},
|
{Name: "roster", Required: true},
|
||||||
},
|
},
|
||||||
@@ -827,11 +1061,12 @@ func TestResolvePipelineUsesReferenceSlotsFromSpecWithoutConstructingExtractor(t
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
if err := RegisterExtractor[codecNotes](catalog.Extractors, ModuleSpec{
|
if err := RegisterExtractor[codecNotes](catalog.Extractors, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
ArtifactKind: "test/notes",
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"},
|
ArtifactKind: "test/notes",
|
||||||
Provides: []string{"candidate"},
|
Requires: []string{"chunk"},
|
||||||
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster", Required: true},
|
{Name: "roster", Required: true},
|
||||||
},
|
},
|
||||||
@@ -989,32 +1224,32 @@ func TestResolvePipelineRejectsMissingCapabilities(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "input",
|
name: "input",
|
||||||
spec: ModuleSpec{Key: "text", Stage: StageInput, Requires: []string{"raw"}},
|
spec: ModuleSpec{Key: "text", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"raw"}},
|
||||||
want: []string{"baseline", "input", "text", "raw"},
|
want: []string{"baseline", "input", "text", "raw"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "chunk",
|
name: "chunk",
|
||||||
spec: ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"missing"}},
|
spec: ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||||
want: []string{"baseline", "chunk", "generic", "missing"},
|
want: []string{"baseline", "chunk", "generic", "missing"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "extract",
|
name: "extract",
|
||||||
spec: ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"missing"}},
|
spec: ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||||
want: []string{"baseline", "events", "extract", "event-extractor", "missing"},
|
want: []string{"baseline", "events", "extract", "event-extractor", "missing"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "merge",
|
name: "merge",
|
||||||
spec: ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"missing"}},
|
spec: ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||||
want: []string{"baseline", "events", "merge", "appendorder", "missing"},
|
want: []string{"baseline", "events", "merge", "appendorder", "missing"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "normalize",
|
name: "normalize",
|
||||||
spec: ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"missing"}},
|
spec: ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||||
want: []string{"baseline", "events", "normalize", "noop", "missing"},
|
want: []string{"baseline", "events", "normalize", "noop", "missing"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "output",
|
name: "output",
|
||||||
spec: ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"missing"}},
|
spec: ModuleSpec{Key: "json", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||||
want: []string{"baseline", "output", "json", "missing"},
|
want: []string{"baseline", "output", "json", "missing"},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -1100,7 +1335,7 @@ func TestResolvePipelineDigestChangesWhenBindingChanges(t *testing.T) {
|
|||||||
right := baselineProfile()
|
right := baselineProfile()
|
||||||
right.Chunk = Binding("window")
|
right.Chunk = Binding("window")
|
||||||
catalog := newProfileCatalog(t)
|
catalog := newProfileCatalog(t)
|
||||||
registerProfileSpecs(t, catalog, ModuleSpec{Key: "window", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}})
|
registerProfileSpecs(t, catalog, ModuleSpec{Key: "window", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}})
|
||||||
|
|
||||||
leftResolved, err := ResolvePipeline(left, ResolveOptions{}, catalog)
|
leftResolved, err := ResolvePipeline(left, ResolveOptions{}, catalog)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1467,17 +1702,103 @@ func emptyProfileCatalog() ModuleCatalog {
|
|||||||
|
|
||||||
func defaultProfileSpecs() []ModuleSpec {
|
func defaultProfileSpecs() []ModuleSpec {
|
||||||
return []ModuleSpec{
|
return []ModuleSpec{
|
||||||
ModuleSpec{Key: "text", Stage: StageInput, Provides: []string{"source"}},
|
ModuleSpec{Key: "text", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}},
|
||||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||||
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||||
ModuleSpec{Key: "appendorder", Stage: StageMerge, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||||
ModuleSpec{Key: "noop", Stage: StageNormalize, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||||
ModuleSpec{Key: "grounded", Stage: StageValidate, Requires: []string{"normalized"}, Provides: []string{"validated"}},
|
ModuleSpec{Key: "grounded", Stage: StageValidate, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"validated"}},
|
||||||
ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
ModuleSpec{Key: "json", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func llmProfileCatalog(t *testing.T) ModuleCatalog {
|
||||||
|
t.Helper()
|
||||||
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
|
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked, Provides: []string{"source"}},
|
||||||
|
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||||
|
ModuleSpec{Key: "llm-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||||
|
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||||
|
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||||
|
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||||
|
)
|
||||||
|
if err := RegisterChunkValidator(catalog.Validators, ValidatorSpec{Key: "llm-chunk-validator", ExecutionClass: contracts.ExecutionClassLLMBacked}, func() (contracts.ChunkValidator, error) {
|
||||||
|
return typedTestChunkValidator{key: "llm-chunk-validator"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register chunk validator: %v", err)
|
||||||
|
}
|
||||||
|
registerProfileValidatorSpec(t, catalog, ValidatorSpec{Key: "llm-validator", ExecutionClass: contracts.ExecutionClassLLMBacked})
|
||||||
|
for _, mapping := range []ValidatorChainMapping{
|
||||||
|
{Stage: StageChunk, Module: "llm-chunk", Validators: []ModuleBinding{Binding("llm-chunk-validator")}},
|
||||||
|
{Stage: StageExtract, Module: "llm-extractor", Validators: []ModuleBinding{Binding("llm-validator")}},
|
||||||
|
{Stage: StageMerge, Module: "llm-merge", Validators: []ModuleBinding{Binding("llm-validator")}},
|
||||||
|
{Stage: StageNormalize, Module: "llm-normalize", Validators: []ModuleBinding{Binding("llm-validator")}},
|
||||||
|
} {
|
||||||
|
if err := catalog.ValidatorChains.Register(mapping); err != nil {
|
||||||
|
t.Fatalf("register validator chain %#v: %v", mapping, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return catalog
|
||||||
|
}
|
||||||
|
|
||||||
|
func llmProfileCatalogWithoutValidatorChains(t *testing.T) ModuleCatalog {
|
||||||
|
catalog := llmProfileCatalog(t)
|
||||||
|
catalog.ValidatorChains = NewValidatorChainRegistry()
|
||||||
|
return catalog
|
||||||
|
}
|
||||||
|
|
||||||
|
func llmProfilePipeline() PipelineProfile {
|
||||||
|
return PipelineProfile{
|
||||||
|
ID: "llm-profile",
|
||||||
|
Input: Binding("llm-input"),
|
||||||
|
Chunk: Binding("llm-chunk"),
|
||||||
|
Output: Binding("llm-output"),
|
||||||
|
Artifacts: map[string]ArtifactLaneProfile{
|
||||||
|
"events": {
|
||||||
|
Extract: Binding("llm-extractor"),
|
||||||
|
Merge: Binding("llm-merge"),
|
||||||
|
Normalize: Binding("llm-normalize"),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func llmProfileValues(profile string) map[string]string {
|
||||||
|
return map[string]string{
|
||||||
|
"input": profile,
|
||||||
|
"chunk": profile,
|
||||||
|
"extract": profile,
|
||||||
|
"merge": profile,
|
||||||
|
"normalize": profile,
|
||||||
|
"output": profile,
|
||||||
|
"validator:chunk:": profile,
|
||||||
|
"validator:extract:events": profile,
|
||||||
|
"validator:merge:events": profile,
|
||||||
|
"validator:normalize:events": profile,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolvedLLMProfileValues(resolved ResolvedPipeline) map[string]string {
|
||||||
|
values := map[string]string{
|
||||||
|
"input": resolved.Input.LLMProfile,
|
||||||
|
"chunk": resolved.Chunk.LLMProfile,
|
||||||
|
"output": resolved.Output.LLMProfile,
|
||||||
|
}
|
||||||
|
lane := resolved.Steps[0].ArtifactLanes[0]
|
||||||
|
values["extract"] = lane.Extract.LLMProfile
|
||||||
|
values["merge"] = lane.Merge.LLMProfile
|
||||||
|
values["normalize"] = lane.Normalize.LLMProfile
|
||||||
|
for _, chain := range resolved.ValidatorChains {
|
||||||
|
for _, validator := range chain.Validators {
|
||||||
|
if validator.ExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
|
values["validator:"+string(chain.Stage)+":"+chain.LaneID] = validator.Binding.LLMProfile
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return values
|
||||||
|
}
|
||||||
|
|
||||||
func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSpec) {
|
func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSpec) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -1566,7 +1887,13 @@ func TestResolvedPipelineCanMarshalToCanonicalJSON(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
if _, err := json.Marshal(resolved); err != nil {
|
encoded, err := json.Marshal(resolved)
|
||||||
|
if err != nil {
|
||||||
t.Fatalf("json.Marshal(resolved) error = %v, want nil", err)
|
t.Fatalf("json.Marshal(resolved) error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
for _, field := range []string{"input_execution_class", "chunk_execution_class", "extract_execution_class", "merge_execution_class", "normalize_execution_class", "output_execution_class"} {
|
||||||
|
if !strings.Contains(string(encoded), `"`+field+`":"deterministic"`) {
|
||||||
|
t.Fatalf("resolved JSON does not retain %q: %s", field, encoded)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -189,12 +189,12 @@ func TestMaterializeReferencesUsesLaneArtifactVariant(t *testing.T) {
|
|||||||
{kind: "test/beta", mergeSlot: "beta_merge", normalizeSlot: "beta_normalize"},
|
{kind: "test/beta", mergeSlot: "beta_merge", normalizeSlot: "beta_normalize"},
|
||||||
{kind: "test/alpha", mergeSlot: "alpha_merge", normalizeSlot: "alpha_normalize"},
|
{kind: "test/alpha", mergeSlot: "alpha_merge", normalizeSlot: "alpha_normalize"},
|
||||||
} {
|
} {
|
||||||
if err := RegisterMerger(mergers, ModuleSpec{Key: "shared/merge", Stage: StageMerge, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.mergeSlot}}}, func() (contracts.Merger[codecNotes], error) {
|
if err := RegisterMerger(mergers, ModuleSpec{Key: "shared/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.mergeSlot}}}, func() (contracts.Merger[codecNotes], error) {
|
||||||
return typedTestMerger[codecNotes]{key: "shared/merge"}, nil
|
return typedTestMerger[codecNotes]{key: "shared/merge"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterMerger(%s): %v", item.kind, err)
|
t.Fatalf("RegisterMerger(%s): %v", item.kind, err)
|
||||||
}
|
}
|
||||||
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "shared/normalize", Stage: StageNormalize, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.normalizeSlot}}}, func() (contracts.Normalizer[codecNotes], error) {
|
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "shared/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.normalizeSlot}}}, func() (contracts.Normalizer[codecNotes], error) {
|
||||||
return typedTestNormalizer[codecNotes]{key: "shared/normalize"}, nil
|
return typedTestNormalizer[codecNotes]{key: "shared/normalize"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterNormalizer(%s): %v", item.kind, err)
|
t.Fatalf("RegisterNormalizer(%s): %v", item.kind, err)
|
||||||
@@ -531,6 +531,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
|||||||
ModuleSpec{
|
ModuleSpec{
|
||||||
Key: "generic",
|
Key: "generic",
|
||||||
Stage: StageChunk,
|
Stage: StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"source"},
|
Requires: []string{"source"},
|
||||||
Provides: []string{"chunk"},
|
Provides: []string{"chunk"},
|
||||||
ReferenceSlots: chunkSlots,
|
ReferenceSlots: chunkSlots,
|
||||||
@@ -538,6 +539,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
|||||||
ModuleSpec{
|
ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"},
|
Requires: []string{"chunk"},
|
||||||
Provides: []string{"candidate"},
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: extractSlots,
|
ReferenceSlots: extractSlots,
|
||||||
@@ -545,6 +547,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
|||||||
ModuleSpec{
|
ModuleSpec{
|
||||||
Key: "appendorder",
|
Key: "appendorder",
|
||||||
Stage: StageMerge,
|
Stage: StageMerge,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"candidate"},
|
Requires: []string{"candidate"},
|
||||||
Provides: []string{"merged"},
|
Provides: []string{"merged"},
|
||||||
ReferenceSlots: mergeSlots,
|
ReferenceSlots: mergeSlots,
|
||||||
@@ -552,6 +555,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
|||||||
ModuleSpec{
|
ModuleSpec{
|
||||||
Key: "noop",
|
Key: "noop",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"merged"},
|
Requires: []string{"merged"},
|
||||||
Provides: []string{"normalized"},
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: normalizeSlots,
|
ReferenceSlots: normalizeSlots,
|
||||||
|
|||||||
@@ -116,7 +116,7 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
|
|||||||
return false, nil, terminal.record(nil, fmt.Errorf("clone chunker manifest metadata: %w", metadataErr))
|
return false, nil, terminal.record(nil, fmt.Errorf("clone chunker manifest metadata: %w", metadataErr))
|
||||||
}
|
}
|
||||||
profile := ""
|
profile := ""
|
||||||
if provider, ok := chunker.(contracts.ChunkExecutionClassProvider); ok && provider.ExecutionClass() == contracts.ExecutionClassLLMBacked {
|
if input.pipeline.ChunkExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
profile = input.pipeline.Chunk.LLMProfile
|
profile = input.pipeline.Chunk.LLMProfile
|
||||||
}
|
}
|
||||||
candidate := ChunkPlanRecord{
|
candidate := ChunkPlanRecord{
|
||||||
|
|||||||
@@ -67,30 +67,17 @@ type manifestChunker struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (c manifestChunker) ManifestMetadata() map[string]any { return c.metadata }
|
func (c manifestChunker) ManifestMetadata() map[string]any { return c.metadata }
|
||||||
func (manifestChunker) ExecutionClass() contracts.ExecutionClass {
|
|
||||||
return contracts.ExecutionClassLLMBacked
|
|
||||||
}
|
|
||||||
|
|
||||||
type llmCountingChunker struct {
|
type llmCountingChunker struct {
|
||||||
terminalChunker
|
terminalChunker
|
||||||
llmCalls *int
|
llmCalls *int
|
||||||
}
|
}
|
||||||
|
|
||||||
func (llmCountingChunker) ExecutionClass() contracts.ExecutionClass {
|
|
||||||
return contracts.ExecutionClassLLMBacked
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c llmCountingChunker) Plan(ctx context.Context, request contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
|
func (c llmCountingChunker) Plan(ctx context.Context, request contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
|
||||||
(*c.llmCalls)++
|
(*c.llmCalls)++
|
||||||
return c.terminalChunker.Plan(ctx, request)
|
return c.terminalChunker.Plan(ctx, request)
|
||||||
}
|
}
|
||||||
|
|
||||||
type deterministicChunker struct{ terminalChunker }
|
|
||||||
|
|
||||||
func (deterministicChunker) ExecutionClass() contracts.ExecutionClass {
|
|
||||||
return contracts.ExecutionClassDeterministic
|
|
||||||
}
|
|
||||||
|
|
||||||
type retryingChunker struct {
|
type retryingChunker struct {
|
||||||
key string
|
key string
|
||||||
plan source.ChunkPlan
|
plan source.ChunkPlan
|
||||||
@@ -535,6 +522,7 @@ func TestRunnerPublishesOnlyAcceptedGeneratedPlans(t *testing.T) {
|
|||||||
func TestRunnerStoresProducerProvenanceAndProducerWarnings(t *testing.T) {
|
func TestRunnerStoresProducerProvenanceAndProducerWarnings(t *testing.T) {
|
||||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||||
prepared.resolved.Chunk.LLMProfile = "chunk-profile"
|
prepared.resolved.Chunk.LLMProfile = "chunk-profile"
|
||||||
|
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
|
||||||
prepared.resolved.ChunkReferences = ResolvedReferenceTarget{
|
prepared.resolved.ChunkReferences = ResolvedReferenceTarget{
|
||||||
Stage: StageChunk,
|
Stage: StageChunk,
|
||||||
ReferenceSet: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
ReferenceSet: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||||
@@ -577,7 +565,8 @@ func TestRunnerRejectsUncloneableModuleManifestMetadata(t *testing.T) {
|
|||||||
func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
|
func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
|
||||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||||
prepared.resolved.Chunk.LLMProfile = "configured-but-unused"
|
prepared.resolved.Chunk.LLMProfile = "configured-but-unused"
|
||||||
prepared.chunker = deterministicChunker{terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}}
|
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassDeterministic
|
||||||
|
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}
|
||||||
store := &recordingChunkPlanStore{}
|
store := &recordingChunkPlanStore{}
|
||||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store})
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -588,6 +577,22 @@ func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunnerDoesNotInventProducerProfileForLLMChunker(t *testing.T) {
|
||||||
|
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||||
|
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
|
||||||
|
prepared.resolved.Chunk.LLMProfile = ""
|
||||||
|
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}
|
||||||
|
store := &recordingChunkPlanStore{}
|
||||||
|
|
||||||
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if store.saved.Producer.LLMProfile != "" || output.Manifest.ChunkPlan.ProducerLLMProfile != "" {
|
||||||
|
t.Fatalf("LLM producer profile = stored %q manifest %q, want empty", store.saved.Producer.LLMProfile, output.Manifest.ChunkPlan.ProducerLLMProfile)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunnerRefreshChangesDownstreamChunkFingerprint(t *testing.T) {
|
func TestRunnerRefreshChangesDownstreamChunkFingerprint(t *testing.T) {
|
||||||
doc := typedTestDocumentWithUnits(2)
|
doc := typedTestDocumentWithUnits(2)
|
||||||
prepared := preparedConcurrentPipeline(t, 1)
|
prepared := preparedConcurrentPipeline(t, 1)
|
||||||
|
|||||||
@@ -229,7 +229,7 @@ func TestResolveTypedLaneRejectsIncompatibleComposition(t *testing.T) {
|
|||||||
|
|
||||||
func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) {
|
func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) {
|
||||||
registry := NewMergerRegistry()
|
registry := NewMergerRegistry()
|
||||||
spec := ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"}
|
spec := ModuleSpec{Key: "typed/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}
|
||||||
constructor := func() (contracts.Merger[codecNotes], error) {
|
constructor := func() (contracts.Merger[codecNotes], error) {
|
||||||
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
|
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
|
||||||
}
|
}
|
||||||
@@ -243,7 +243,7 @@ func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) {
|
|||||||
|
|
||||||
func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
|
func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
|
||||||
extractors := NewExtractorRegistry()
|
extractors := NewExtractorRegistry()
|
||||||
if err := RegisterExtractor(extractors, ModuleSpec{Key: "typed/extract", Stage: StageExtract, ArtifactKind: "test/notes"}, func() (contracts.Extractor[codecNotes], error) {
|
if err := RegisterExtractor(extractors, ModuleSpec{Key: "typed/extract", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Extractor[codecNotes], error) {
|
||||||
return typedTestExtractor[codecNotes]{key: "typed/extract"}, nil
|
return typedTestExtractor[codecNotes]{key: "typed/extract"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterExtractor() error = %v", err)
|
t.Fatalf("RegisterExtractor() error = %v", err)
|
||||||
@@ -253,7 +253,7 @@ func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
mergers := NewMergerRegistry()
|
mergers := NewMergerRegistry()
|
||||||
if err := RegisterMerger(mergers, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"}, func() (contracts.Merger[codecNotes], error) {
|
if err := RegisterMerger(mergers, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Merger[codecNotes], error) {
|
||||||
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
|
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterMerger() error = %v", err)
|
t.Fatalf("RegisterMerger() error = %v", err)
|
||||||
@@ -263,7 +263,7 @@ func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
normalizers := NewNormalizerRegistry()
|
normalizers := NewNormalizerRegistry()
|
||||||
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
||||||
return typedTestNormalizer[codecNotes]{key: "typed/normalize"}, nil
|
return typedTestNormalizer[codecNotes]{key: "typed/normalize"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterNormalizer() error = %v", err)
|
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||||
@@ -298,7 +298,7 @@ func TestTypedRegistrySpecLookupUsesArtifactKindAndStableCatalogOrder(t *testing
|
|||||||
mergers := NewMergerRegistry()
|
mergers := NewMergerRegistry()
|
||||||
normalizers := NewNormalizerRegistry()
|
normalizers := NewNormalizerRegistry()
|
||||||
for _, item := range order {
|
for _, item := range order {
|
||||||
base := ModuleSpec{Key: "typed/shared", ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.slot}}}
|
base := ModuleSpec{Key: "typed/shared", ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.slot}}}
|
||||||
mergeSpec := base
|
mergeSpec := base
|
||||||
mergeSpec.Stage = StageMerge
|
mergeSpec.Stage = StageMerge
|
||||||
normalizeSpec := base
|
normalizeSpec := base
|
||||||
@@ -474,18 +474,18 @@ func typedResolutionCatalog(t *testing.T, options typedCatalogOptions) ModuleCat
|
|||||||
|
|
||||||
func mustRegisterTypedTestBase(t *testing.T, catalog ModuleCatalog) {
|
func mustRegisterTypedTestBase(t *testing.T, catalog ModuleCatalog) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{Key: "typed/input", Stage: StageInput}, func() (contracts.InputAdapter, error) {
|
if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{Key: "typed/input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.InputAdapter, error) {
|
||||||
return &typedTestInput{key: "typed/input", doc: typedTestDocument()}, nil
|
return &typedTestInput{key: "typed/input", doc: typedTestDocument()}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("register input: %v", err)
|
t.Fatalf("register input: %v", err)
|
||||||
}
|
}
|
||||||
if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{Key: "typed/chunk", Stage: StageChunk}, func() (contracts.Chunker, error) {
|
if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{Key: "typed/chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.Chunker, error) {
|
||||||
doc := typedTestDocument()
|
doc := typedTestDocument()
|
||||||
return &typedTestChunker{key: "typed/chunk", plan: typedTestPlan(doc)}, nil
|
return &typedTestChunker{key: "typed/chunk", plan: typedTestPlan(doc)}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("register chunker: %v", err)
|
t.Fatalf("register chunker: %v", err)
|
||||||
}
|
}
|
||||||
if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{Key: "typed/output", Stage: StageOutput}, func() (contracts.OutputEncoder, error) {
|
if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{Key: "typed/output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.OutputEncoder, error) {
|
||||||
return &typedTestOutput{key: "typed/output"}, nil
|
return &typedTestOutput{key: "typed/output"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("register output: %v", err)
|
t.Fatalf("register output: %v", err)
|
||||||
@@ -509,21 +509,21 @@ func mustRegisterArtifactCodec[T any](t *testing.T, registry *ArtifactCodecRegis
|
|||||||
|
|
||||||
func mustRegisterTypedExtractor[T any](t *testing.T, registry *ExtractorRegistry, key string, kind contracts.ArtifactKind, extractor contracts.Extractor[T]) {
|
func mustRegisterTypedExtractor[T any](t *testing.T, registry *ExtractorRegistry, key string, kind contracts.ArtifactKind, extractor contracts.Extractor[T]) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := RegisterExtractor(registry, ModuleSpec{Key: key, Stage: StageExtract, ArtifactKind: kind}, func() (contracts.Extractor[T], error) { return extractor, nil }); err != nil {
|
if err := RegisterExtractor(registry, ModuleSpec{Key: key, Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: kind}, func() (contracts.Extractor[T], error) { return extractor, nil }); err != nil {
|
||||||
t.Fatalf("RegisterExtractor() error = %v", err)
|
t.Fatalf("RegisterExtractor() error = %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func mustRegisterTypedMerger[T any](t *testing.T, registry *MergerRegistry, kind contracts.ArtifactKind, merger contracts.Merger[T]) {
|
func mustRegisterTypedMerger[T any](t *testing.T, registry *MergerRegistry, kind contracts.ArtifactKind, merger contracts.Merger[T]) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := RegisterMerger(registry, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: kind}, func() (contracts.Merger[T], error) { return merger, nil }); err != nil {
|
if err := RegisterMerger(registry, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: kind}, func() (contracts.Merger[T], error) { return merger, nil }); err != nil {
|
||||||
t.Fatalf("RegisterMerger() error = %v", err)
|
t.Fatalf("RegisterMerger() error = %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func mustRegisterTypedNormalizer[T any](t *testing.T, registry *NormalizerRegistry, kind contracts.ArtifactKind, normalizer contracts.Normalizer[T]) {
|
func mustRegisterTypedNormalizer[T any](t *testing.T, registry *NormalizerRegistry, kind contracts.ArtifactKind, normalizer contracts.Normalizer[T]) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := RegisterNormalizer(registry, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: kind}, func() (contracts.Normalizer[T], error) { return normalizer, nil }); err != nil {
|
if err := RegisterNormalizer(registry, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: kind}, func() (contracts.Normalizer[T], error) { return normalizer, nil }); err != nil {
|
||||||
t.Fatalf("RegisterNormalizer() error = %v", err)
|
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
id: dnd.scenes
|
id: dnd.scenes
|
||||||
version: "v1"
|
version: "v1"
|
||||||
default_profile: gemini-2-flash
|
default_profile: dnd-extraction
|
||||||
inputs:
|
inputs:
|
||||||
- name: transcript
|
- name: transcript
|
||||||
required: true
|
required: true
|
||||||
|
|||||||
@@ -47,10 +47,6 @@ func (c *Chunker) Key() string {
|
|||||||
return Key
|
return Key
|
||||||
}
|
}
|
||||||
|
|
||||||
func (*Chunker) ExecutionClass() contracts.ExecutionClass {
|
|
||||||
return contracts.ExecutionClassLLMBacked
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
|
func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
|
||||||
return shared.ReferenceSlots(referenceSlotDescriptions)
|
return shared.ReferenceSlots(referenceSlotDescriptions)
|
||||||
}
|
}
|
||||||
@@ -120,6 +116,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
|||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageChunk,
|
Stage: pipeline.StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
ReferenceSlots: shared.ReferenceSlots(referenceSlotDescriptions),
|
ReferenceSlots: shared.ReferenceSlots(referenceSlotDescriptions),
|
||||||
|
|||||||
@@ -24,6 +24,7 @@ func TestNewModuleSpecAndRegister(t *testing.T) {
|
|||||||
want := pipeline.ModuleSpec{
|
want := pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageChunk,
|
Stage: pipeline.StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: []string{"source.transcript"},
|
Requires: []string{"source.transcript"},
|
||||||
Provides: []string{"chunks"},
|
Provides: []string{"chunks"},
|
||||||
ReferenceSlots: wantReferenceSlots(),
|
ReferenceSlots: wantReferenceSlots(),
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
id: dnd.combat_turns
|
id: dnd.combat_turns
|
||||||
version: "v1"
|
version: "v1"
|
||||||
default_profile: gemini-2-flash
|
default_profile: dnd-extraction
|
||||||
inputs:
|
inputs:
|
||||||
- name: transcript
|
- name: transcript
|
||||||
required: true
|
required: true
|
||||||
|
|||||||
@@ -226,6 +226,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
|||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageExtract,
|
Stage: pipeline.StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
ArtifactKind: dnd.CombatTurnListKind,
|
ArtifactKind: dnd.CombatTurnListKind,
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
id: dnd.item_events
|
id: dnd.item_events
|
||||||
version: "v1"
|
version: "v1"
|
||||||
default_profile: gemini-2-flash
|
default_profile: dnd-extraction
|
||||||
inputs:
|
inputs:
|
||||||
- name: transcript
|
- name: transcript
|
||||||
required: true
|
required: true
|
||||||
|
|||||||
@@ -129,6 +129,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
|||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageExtract,
|
Stage: pipeline.StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
ArtifactKind: dnd.ItemEventListKind,
|
ArtifactKind: dnd.ItemEventListKind,
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ func TestConstructorSpecOptionsAndMetadata(t *testing.T) {
|
|||||||
t.Fatalf("New() error = %v", err)
|
t.Fatalf("New() error = %v", err)
|
||||||
}
|
}
|
||||||
want := pipeline.ModuleSpec{
|
want := pipeline.ModuleSpec{
|
||||||
Key: Key, Stage: pipeline.StageExtract, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.item_events"}, ArtifactKind: dnd.ItemEventListKind,
|
Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.item_events"}, ArtifactKind: dnd.ItemEventListKind,
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "glossary", Description: referenceSlotDescriptions.Glossary, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
{Name: "glossary", Description: referenceSlotDescriptions.Glossary, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||||
{Name: "party", Description: referenceSlotDescriptions.Party, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
{Name: "party", Description: referenceSlotDescriptions.Party, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
id: dnd.npc_interactions
|
id: dnd.npc_interactions
|
||||||
version: "v1"
|
version: "v1"
|
||||||
default_profile: gemini-2-flash
|
default_profile: dnd-extraction
|
||||||
inputs:
|
inputs:
|
||||||
- name: transcript
|
- name: transcript
|
||||||
required: true
|
required: true
|
||||||
|
|||||||
@@ -177,6 +177,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
|||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageExtract,
|
Stage: pipeline.StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
ArtifactKind: dnd.NPCInteractionListKind,
|
ArtifactKind: dnd.NPCInteractionListKind,
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
id: dnd.npcs
|
id: dnd.npcs
|
||||||
version: "v1"
|
version: "v1"
|
||||||
default_profile: gemini-2-flash
|
default_profile: dnd-extraction
|
||||||
inputs:
|
inputs:
|
||||||
- name: transcript
|
- name: transcript
|
||||||
required: true
|
required: true
|
||||||
|
|||||||
@@ -135,6 +135,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
|||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageExtract,
|
Stage: pipeline.StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
ArtifactKind: dnd.NPCListKind,
|
ArtifactKind: dnd.NPCListKind,
|
||||||
|
|||||||
@@ -25,11 +25,12 @@ func TestNewRequiresLLMClientAndRejectsAmbiguousReferences(t *testing.T) {
|
|||||||
func TestModuleSpecAndReferenceSlots(t *testing.T) {
|
func TestModuleSpecAndReferenceSlots(t *testing.T) {
|
||||||
got := ModuleSpec()
|
got := ModuleSpec()
|
||||||
want := pipeline.ModuleSpec{
|
want := pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageExtract,
|
Stage: pipeline.StageExtract,
|
||||||
Requires: []string{"chunks", "source.transcript"},
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Provides: []string{"dnd.npcs"},
|
Requires: []string{"chunks", "source.transcript"},
|
||||||
ArtifactKind: dnd.NPCListKind,
|
Provides: []string{"dnd.npcs"},
|
||||||
|
ArtifactKind: dnd.NPCListKind,
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "glossary", Description: "Optional campaign glossary reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
{Name: "glossary", Description: "Optional campaign glossary reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||||
{Name: "party", Description: "Optional party roster reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
{Name: "party", Description: "Optional party roster reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
id: dnd.scene_descriptions
|
id: dnd.scene_descriptions
|
||||||
version: "v1"
|
version: "v1"
|
||||||
default_profile: gemini-2-flash
|
default_profile: dnd-extraction
|
||||||
inputs:
|
inputs:
|
||||||
- name: transcript
|
- name: transcript
|
||||||
required: true
|
required: true
|
||||||
|
|||||||
@@ -146,6 +146,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
|||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageExtract,
|
Stage: pipeline.StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
ArtifactKind: dnd.SceneDescriptionListKind,
|
ArtifactKind: dnd.SceneDescriptionListKind,
|
||||||
|
|||||||
@@ -24,7 +24,7 @@ func TestNewRequiresLLMClientAndRejectsAmbiguousReferences(t *testing.T) {
|
|||||||
|
|
||||||
func TestModuleSpecAndReferenceSlots(t *testing.T) {
|
func TestModuleSpecAndReferenceSlots(t *testing.T) {
|
||||||
want := pipeline.ModuleSpec{
|
want := pipeline.ModuleSpec{
|
||||||
Key: Key, Stage: pipeline.StageExtract, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.scene_descriptions"}, ArtifactKind: dnd.SceneDescriptionListKind,
|
Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.scene_descriptions"}, ArtifactKind: dnd.SceneDescriptionListKind,
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "glossary", Description: "Optional campaign glossary reference material used only to disambiguate scene descriptions.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
{Name: "glossary", Description: "Optional campaign glossary reference material used only to disambiguate scene descriptions.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||||
{Name: "party", Description: "Optional party roster reference material used only to disambiguate scene descriptions.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
{Name: "party", Description: "Optional party roster reference material used only to disambiguate scene descriptions.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
id: dnd.spells
|
id: dnd.spells
|
||||||
version: "v1"
|
version: "v1"
|
||||||
default_profile: gemini-2-flash
|
default_profile: dnd-extraction
|
||||||
inputs:
|
inputs:
|
||||||
- name: transcript
|
- name: transcript
|
||||||
required: true
|
required: true
|
||||||
|
|||||||
@@ -200,6 +200,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
|||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageExtract,
|
Stage: pipeline.StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
ArtifactKind: dnd.SpellListKind,
|
ArtifactKind: dnd.SpellListKind,
|
||||||
|
|||||||
@@ -23,8 +23,9 @@ func TestNewRequiresLLMClientAndReturnsExtractor(t *testing.T) {
|
|||||||
func TestModuleSpec(t *testing.T) {
|
func TestModuleSpec(t *testing.T) {
|
||||||
got := ModuleSpec()
|
got := ModuleSpec()
|
||||||
want := pipeline.ModuleSpec{
|
want := pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageExtract,
|
Stage: pipeline.StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: []string{
|
Requires: []string{
|
||||||
"chunks",
|
"chunks",
|
||||||
"source.transcript",
|
"source.transcript",
|
||||||
|
|||||||
@@ -336,6 +336,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
|||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageNormalize,
|
Stage: pipeline.StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
ArtifactKind: dnd.CombatTurnListKind,
|
ArtifactKind: dnd.CombatTurnListKind,
|
||||||
|
|||||||
@@ -222,11 +222,12 @@ func eventScope(index int) string { return fmt.Sprintf("events[%d]", index) }
|
|||||||
|
|
||||||
func ModuleSpec() pipeline.ModuleSpec {
|
func ModuleSpec() pipeline.ModuleSpec {
|
||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageNormalize,
|
Stage: pipeline.StageNormalize,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
ArtifactKind: dnd.ItemEventListKind,
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
|
ArtifactKind: dnd.ItemEventListKind,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -278,6 +278,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
|||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageNormalize,
|
Stage: pipeline.StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
ArtifactKind: dnd.NPCInteractionListKind,
|
ArtifactKind: dnd.NPCInteractionListKind,
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
id: dnd.npcs.normalize
|
id: dnd.npcs.normalize
|
||||||
version: "v1"
|
version: "v1"
|
||||||
default_profile: gemini-2-flash
|
default_profile: dnd-extraction
|
||||||
inputs:
|
inputs:
|
||||||
- name: candidates
|
- name: candidates
|
||||||
required: true
|
required: true
|
||||||
|
|||||||
@@ -360,7 +360,7 @@ func duplicateWarning(retainedIndex int, removed []int) contracts.Warning {
|
|||||||
func npcScope(index int) string { return fmt.Sprintf("npcs[%d]", index) }
|
func npcScope(index int) string { return fmt.Sprintf("npcs[%d]", index) }
|
||||||
|
|
||||||
func ModuleSpec() pipeline.ModuleSpec {
|
func ModuleSpec() pipeline.ModuleSpec {
|
||||||
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, Requires: append([]string(nil), requiredCapabilities...), Provides: append([]string(nil), providedCapabilities...), ArtifactKind: dnd.NPCListKind}
|
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: append([]string(nil), requiredCapabilities...), Provides: append([]string(nil), providedCapabilities...), ArtifactKind: dnd.NPCListKind}
|
||||||
}
|
}
|
||||||
|
|
||||||
func Register(registry *pipeline.NormalizerRegistry) error {
|
func Register(registry *pipeline.NormalizerRegistry) error {
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ func TestModuleContractAndIdentity(t *testing.T) {
|
|||||||
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
|
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
|
||||||
t.Fatal("DecodeOptions() accepted unknown option")
|
t.Fatal("DecodeOptions() accepted unknown option")
|
||||||
}
|
}
|
||||||
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: dnd.NPCListKind}
|
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: dnd.NPCListKind}
|
||||||
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -138,11 +138,12 @@ func sourceRefLabel(ref source.SourceRef) string {
|
|||||||
|
|
||||||
func ModuleSpec() pipeline.ModuleSpec {
|
func ModuleSpec() pipeline.ModuleSpec {
|
||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageNormalize,
|
Stage: pipeline.StageNormalize,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
ArtifactKind: dnd.SceneDescriptionListKind,
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
|
ArtifactKind: dnd.SceneDescriptionListKind,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -90,7 +90,7 @@ func TestNormalizerContractAndCancellation(t *testing.T) {
|
|||||||
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
|
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
|
||||||
t.Fatal("DecodeOptions() accepted unknown options")
|
t.Fatal("DecodeOptions() accepted unknown options")
|
||||||
}
|
}
|
||||||
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: dnd.SceneDescriptionListKind}
|
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: dnd.SceneDescriptionListKind}
|
||||||
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -297,6 +297,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
|||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageNormalize,
|
Stage: pipeline.StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
ArtifactKind: dnd.SpellListKind,
|
ArtifactKind: dnd.SpellListKind,
|
||||||
|
|||||||
@@ -24,11 +24,12 @@ func TestModuleContractAndStrictOptions(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
want := pipeline.ModuleSpec{
|
want := pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageNormalize,
|
Stage: pipeline.StageNormalize,
|
||||||
Requires: []string{"merged"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"normalized"},
|
Requires: []string{"merged"},
|
||||||
ArtifactKind: dnd.SpellListKind,
|
Provides: []string{"normalized"},
|
||||||
|
ArtifactKind: dnd.SpellListKind,
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{
|
ReferenceSlots: []contracts.ReferenceSlot{{
|
||||||
Name: spellcatalog.SpellCatalogReferenceSlot,
|
Name: spellcatalog.SpellCatalogReferenceSlot,
|
||||||
Description: "Optional canonical spell-name catalog used for normalization and duplicate identity.",
|
Description: "Optional canonical spell-name catalog used for normalization and duplicate identity.",
|
||||||
|
|||||||
@@ -0,0 +1,5 @@
|
|||||||
|
id: dnd-extraction
|
||||||
|
backend: openrouter
|
||||||
|
model: openai/gpt-5.6-luna
|
||||||
|
timeout_seconds: 240
|
||||||
|
service_tier: flex
|
||||||
14
internal/modules/dnd/register/profiles.go
Normal file
14
internal/modules/dnd/register/profiles.go
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
package register
|
||||||
|
|
||||||
|
import (
|
||||||
|
"embed"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
|
)
|
||||||
|
|
||||||
|
//go:embed assets/profiles/*.yaml
|
||||||
|
var embeddedProfileAssets embed.FS
|
||||||
|
|
||||||
|
func registerFallbackProfiles(assets *llm.AssetRegistry) error {
|
||||||
|
return assets.RegisterFallbackProfileFS(embeddedProfileAssets, "assets/profiles")
|
||||||
|
}
|
||||||
@@ -30,6 +30,9 @@ func Register(registries pipeline.Registries, assets *llm.AssetRegistry) error {
|
|||||||
if err := registerPromptAssets(assets); err != nil {
|
if err := registerPromptAssets(assets); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
if err := registerFallbackProfiles(assets); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
return registerDefaultChains(registries.ValidatorChains)
|
return registerDefaultChains(registries.ValidatorChains)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -39,6 +39,31 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
|
|||||||
if _, err := fs.ReadFile(promptFS, "dnd.npcs.normalize/dnd.npcs.normalize.yaml"); err != nil {
|
if _, err := fs.ReadFile(promptFS, "dnd.npcs.normalize/dnd.npcs.normalize.yaml"); err != nil {
|
||||||
t.Fatalf("normalization prompt asset = %v, want registered private prompt", err)
|
t.Fatalf("normalization prompt asset = %v, want registered private prompt", err)
|
||||||
}
|
}
|
||||||
|
for _, name := range []string{
|
||||||
|
"dnd.scenes/dnd.scenes.yaml",
|
||||||
|
"dnd.spells/dnd.spells.yaml",
|
||||||
|
"dnd.npcs/dnd.npcs.yaml",
|
||||||
|
"dnd.combat_turns/dnd.combat_turns.yaml",
|
||||||
|
"dnd.item_events/dnd.item_events.yaml",
|
||||||
|
"dnd.npc_interactions/dnd.npc_interactions.yaml",
|
||||||
|
"dnd.scene_descriptions/dnd.scene_descriptions.yaml",
|
||||||
|
"dnd.npcs.normalize/dnd.npcs.normalize.yaml",
|
||||||
|
} {
|
||||||
|
content, err := fs.ReadFile(promptFS, name)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read prompt asset %q: %v", name, err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(content), "default_profile: dnd-extraction") {
|
||||||
|
t.Fatalf("prompt asset %q does not select dnd-extraction", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fallbackFS, err := assets.FallbackProfileFS()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("FallbackProfileFS() error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := fs.ReadFile(fallbackFS, "dnd-extraction.yaml"); err != nil {
|
||||||
|
t.Fatalf("fallback profile asset = %v, want registered D&D profile", err)
|
||||||
|
}
|
||||||
schemaFS, err := assets.SchemaFS()
|
schemaFS, err := assets.SchemaFS()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("SchemaFS() error = %v", err)
|
t.Fatalf("SchemaFS() error = %v", err)
|
||||||
|
|||||||
@@ -38,10 +38,6 @@ func (c *Chunker) Key() string {
|
|||||||
return Key
|
return Key
|
||||||
}
|
}
|
||||||
|
|
||||||
func (*Chunker) ExecutionClass() contracts.ExecutionClass {
|
|
||||||
return contracts.ExecutionClassDeterministic
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
|
func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -90,9 +86,10 @@ func (c *Chunker) Plan(ctx context.Context, req contracts.ChunkRequest) (contrac
|
|||||||
|
|
||||||
func ModuleSpec() pipeline.ModuleSpec {
|
func ModuleSpec() pipeline.ModuleSpec {
|
||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageChunk,
|
Stage: pipeline.StageChunk,
|
||||||
Provides: []string{"chunks"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
Provides: []string{"chunks"},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -15,9 +15,10 @@ import (
|
|||||||
|
|
||||||
func TestModuleSpecAndRegister(t *testing.T) {
|
func TestModuleSpecAndRegister(t *testing.T) {
|
||||||
want := pipeline.ModuleSpec{
|
want := pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageChunk,
|
Stage: pipeline.StageChunk,
|
||||||
Provides: []string{"chunks"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
Provides: []string{"chunks"},
|
||||||
}
|
}
|
||||||
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||||
|
|||||||
@@ -3,13 +3,14 @@ package appendorder
|
|||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
)
|
)
|
||||||
|
|
||||||
const Key = "appendorder"
|
const Key = "appendorder"
|
||||||
|
|
||||||
func ModuleSpec() pipeline.ModuleSpec {
|
func ModuleSpec() pipeline.ModuleSpec {
|
||||||
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageMerge, Provides: []string{"merged"}}
|
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"merged"}}
|
||||||
}
|
}
|
||||||
|
|
||||||
func mergerErrorf(format string, args ...any) error {
|
func mergerErrorf(format string, args ...any) error {
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user