Compare commits
35 Commits
de8ed41b34
...
v0.2.0
| Author | SHA1 | Date | |
|---|---|---|---|
| 39388e96d4 | |||
| 12ac25bd63 | |||
| 394278e1f2 | |||
| 5cd7f8e737 | |||
| bf3fadf9ae | |||
| 58815aaf33 | |||
| ce857966f1 | |||
| a3bd0c1867 | |||
| b05634ee86 | |||
| 4829f94157 | |||
| 67b315099d | |||
| b5c86de4d7 | |||
| 2eeca2ed5a | |||
| b5aaeb1c78 | |||
| 9171b66a41 | |||
| b4363b3b73 | |||
| 241e9d2a89 | |||
| 715fff7b72 | |||
| d627b91b4f | |||
| a67b3aa76d | |||
| a16dcdfa52 | |||
| 46e4466d28 | |||
| 71a004bfc8 | |||
| f8333f2c15 | |||
| f603f7ac64 | |||
| 7a00e7049c | |||
| 2a9db9a957 | |||
| de046a8f13 | |||
| f1a6574013 | |||
| 4bca6d3103 | |||
| 7c569a3d8c | |||
| 8e04ef9e2b | |||
| 53a330587b | |||
| 7cfab8ada0 | |||
| 5c82b62856 |
49
docs/adr/0010-workload-oriented-llm-profile-defaults.md
Normal file
49
docs/adr/0010-workload-oriented-llm-profile-defaults.md
Normal file
@@ -0,0 +1,49 @@
|
|||||||
|
# ADR-0010: Use workload-oriented LLM profile defaults
|
||||||
|
|
||||||
|
**Status:** Accepted
|
||||||
|
**Date:** 2026-08-03
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
LLM-backed D&D operations share an execution-policy choice, but repeating a
|
||||||
|
provider or model-named profile on every module binding ties pipeline structure
|
||||||
|
to a deployment decision. Different environments may require different model,
|
||||||
|
backend, timeout, or reasoning settings while retaining the same workload.
|
||||||
|
|
||||||
|
Notarius also needs a usable default for maintained D&D prompts without making
|
||||||
|
an operator profile mandatory. That default must remain owned by the D&D
|
||||||
|
family, while generic LLM infrastructure stays unaware of domain-specific
|
||||||
|
policy.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Pipelines may name one workload-oriented default profile, inherited only by
|
||||||
|
selected LLM-backed bindings and validators. Binding-level profile IDs remain
|
||||||
|
intentional exceptions, and the run-wide CLI profile override has highest
|
||||||
|
precedence.
|
||||||
|
|
||||||
|
The D&D family owns an embedded fallback profile named `dnd-extraction`.
|
||||||
|
Operators may provide a complete profile with the same ID through a PromptKit
|
||||||
|
filesystem source. PromptKit selects the higher-precedence matching definition;
|
||||||
|
Notarius does not merge profile documents. Production, development, and local
|
||||||
|
deployments can therefore use different execution policy behind one unchanged
|
||||||
|
pipeline ID.
|
||||||
|
|
||||||
|
## Alternatives considered
|
||||||
|
|
||||||
|
- Repeat a model-named profile on every binding. This makes routine deployment
|
||||||
|
policy changes noisy and obscures the shared workload intent.
|
||||||
|
- Require every deployment to install a profile file. This adds configuration
|
||||||
|
friction and leaves maintained D&D prompts without an application-owned
|
||||||
|
fallback.
|
||||||
|
- Put D&D profile policy in generic LLM infrastructure. This breaks domain
|
||||||
|
ownership and makes generic code depend on one workload.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
Pipeline configuration expresses workload intent rather than a specific
|
||||||
|
provider or model. Operators can replace the complete execution policy without
|
||||||
|
editing bindings, while binding-level and run-wide exceptions remain available.
|
||||||
|
Profile changes affect resolved pipeline and checkpoint identity, so they may
|
||||||
|
intentionally cause work to be recomputed. The D&D fallback becomes a
|
||||||
|
maintained application execution-policy asset.
|
||||||
15
docs/cli.md
15
docs/cli.md
@@ -39,16 +39,23 @@ pipeline ID and **--input** are required.
|
|||||||
| **--debug** | Retain a debug bundle for this run. |
|
| **--debug** | Retain a debug bundle for this run. |
|
||||||
| **--debug-dir path** | Override the debug-bundle root. Requires **--debug**. |
|
| **--debug-dir path** | Override the debug-bundle root. Requires **--debug**. |
|
||||||
| **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. |
|
| **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. |
|
||||||
| **--llm-profile id** | Override effective LLM-capable module bindings with one configured profile. |
|
| **--llm-profile id** | Highest-precedence configured profile for selected LLM-backed bindings and validators; it replaces binding and [pipeline](config.md#pipelines) defaults. |
|
||||||
| **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. |
|
| **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. |
|
||||||
|
| **--reasoning-effort value** | Replace the selected PromptKit profile's reasoning effort for every LLM-backed call in this run. The value must be non-empty and the flag may be specified only once. |
|
||||||
|
| **--clear-reasoning-effort** | Clear reasoning effort inherited from the selected PromptKit profile for every LLM-backed call in this run. |
|
||||||
| **--reference selector=path** | Add or replace a file reference binding. Repeatable. |
|
| **--reference selector=path** | Add or replace a file reference binding. Repeatable. |
|
||||||
| **--without-reference selector** | Remove a configured optional reference binding. Repeatable. |
|
| **--without-reference selector** | Remove a configured optional reference binding. Repeatable. |
|
||||||
|
|
||||||
**--chunk_cache** accepts only **auto**, **bypass**, or **refresh**.
|
**--chunk_cache** accepts only **auto**, **bypass**, or **refresh**.
|
||||||
**--debug-dir**, **--output-dir**, **--session-id**, and
|
**--debug-dir**, **--output-dir**, **--session-id**, and
|
||||||
**--recompute-step** reject explicit empty values. **--recompute-step**
|
**--reasoning-effort**, and **--recompute-step** reject explicit empty values.
|
||||||
requires **--resume**; checkpoint requirements and reuse behavior are
|
**--reasoning-effort** and **--clear-reasoning-effort** are mutually exclusive.
|
||||||
documented in [Operations](operations.md).
|
When neither is present, reasoning effort comes from the selected PromptKit
|
||||||
|
profile. These controls apply to the shared run client, including retries and
|
||||||
|
LLM-backed validators, and do not modify configuration or profile files.
|
||||||
|
Persistent reasoning settings remain a PromptKit profile concern.
|
||||||
|
**--recompute-step** requires **--resume**; checkpoint requirements and reuse
|
||||||
|
behavior are documented in [Operations](operations.md).
|
||||||
|
|
||||||
### Reference selectors
|
### Reference selectors
|
||||||
|
|
||||||
|
|||||||
103
docs/config.md
103
docs/config.md
@@ -1,7 +1,7 @@
|
|||||||
# Configuration
|
# Configuration
|
||||||
|
|
||||||
This is the canonical reference for Notarius configuration. Configuration files
|
This is the canonical reference for Notarius configuration. Configuration files
|
||||||
are YAML and must declare version 3. They select pipelines and their modules;
|
are YAML and must declare version 4. They select pipelines and their modules;
|
||||||
the [CLI reference](cli.md) owns invocation syntax, and
|
the [CLI reference](cli.md) owns invocation syntax, and
|
||||||
[Operations](operations.md) owns run-state procedures.
|
[Operations](operations.md) owns run-state procedures.
|
||||||
|
|
||||||
@@ -32,7 +32,8 @@ override the fields listed below.
|
|||||||
single-lane Seriatim-to-spell pipeline.
|
single-lane Seriatim-to-spell pipeline.
|
||||||
- [Complete D&D configuration](../examples/dnd-complete.config.yml) uses
|
- [Complete D&D configuration](../examples/dnd-complete.config.yml) uses
|
||||||
ordered steps, all implemented D&D lanes, generated references, state
|
ordered steps, all implemented D&D lanes, generated references, state
|
||||||
settings, and bounded LLM concurrency.
|
settings, bounded LLM concurrency, and the maintained
|
||||||
|
[operator profile](../examples/profiles/dnd-extraction.yml).
|
||||||
|
|
||||||
Use these complete files as starting points rather than combining the
|
Use these complete files as starting points rather than combining the
|
||||||
illustrative fragments in this reference.
|
illustrative fragments in this reference.
|
||||||
@@ -45,8 +46,8 @@ other than **version** is optional.
|
|||||||
|
|
||||||
| Field | Type | Default | Rules |
|
| Field | Type | Default | Rules |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| **version** | integer | none | Required; must be 3. |
|
| **version** | integer | none | Required; must be 4. |
|
||||||
| **scriptorium** | object | none | Profile source configuration. |
|
| **promptkit** | object | none | Profile source and optional local-backend configuration. |
|
||||||
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
|
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
|
||||||
| **concurrency** | object | see below | Global LLM and extraction limits. |
|
| **concurrency** | object | see below | Global LLM and extraction limits. |
|
||||||
| **output** | object | see below | Published output settings. |
|
| **output** | object | see below | Published output settings. |
|
||||||
@@ -69,20 +70,85 @@ Built-in defaults are:
|
|||||||
An empty cache directory in YAML deliberately selects the corresponding
|
An empty cache directory in YAML deliberately selects the corresponding
|
||||||
per-user root. An explicit empty output or debug directory is invalid.
|
per-user root. An explicit empty output or debug directory is invalid.
|
||||||
|
|
||||||
## Scriptorium Profiles
|
## PromptKit Profiles
|
||||||
|
|
||||||
The optional **scriptorium** object selects one source of profile definitions:
|
The optional **promptkit** object selects one source of profile definitions and
|
||||||
|
may register one conventional local OpenAI-compatible backend:
|
||||||
|
|
||||||
|
~~~yaml
|
||||||
|
version: 4
|
||||||
|
|
||||||
|
promptkit:
|
||||||
|
profile_dir: ./profiles
|
||||||
|
# profile_file: ./profiles.yml
|
||||||
|
local_backend:
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
concurrency_limit: 2
|
||||||
|
~~~
|
||||||
|
|
||||||
| Field | Type | Rules |
|
| Field | Type | Rules |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| **profile_dir** | string | Non-empty directory containing profile files. |
|
| **profile_dir** | string | Non-empty directory containing profile files. |
|
||||||
| **profile_file** | string | Non-empty profile file. |
|
| **profile_file** | string | Non-empty profile file. |
|
||||||
|
| **local_backend** | object | Optional registration for the conventional PromptKit backend ID **local**. |
|
||||||
|
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
|
||||||
|
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
|
||||||
|
|
||||||
Set at most one of these fields. Profile IDs used by a binding must be available
|
Set at most one of **profile_dir** and **profile_file**. Relative values use
|
||||||
from the selected Scriptorium profile source when the pipeline is resolved.
|
the process working directory, not the configuration file's directory. The
|
||||||
Keep credentials out of this file: configure a profile to read its credential
|
complete example's `./examples/profiles/dnd-extraction.yml` value is therefore
|
||||||
from an environment variable, then set that environment variable only in the
|
valid when Notarius is launched from the repository root; use an absolute path
|
||||||
run environment.
|
for services and containers.
|
||||||
|
|
||||||
|
An operator source is optional. For a requested ID, PromptKit checks the
|
||||||
|
configured operator source first, then Notarius's embedded fallback profiles,
|
||||||
|
then its own built-in catalog. A matching profile is complete: it replaces a
|
||||||
|
lower-precedence definition rather than merging with it. The maintained
|
||||||
|
[`dnd-extraction` operator profile](../examples/profiles/dnd-extraction.yml)
|
||||||
|
is a secret-free deployment artifact; production, development, and local
|
||||||
|
deployments can each provide a complete definition with that same workload ID.
|
||||||
|
Use workload-oriented IDs for new profiles instead of model names.
|
||||||
|
[Operations](operations.md#promptkit-profile-deployment) owns the deployment
|
||||||
|
workflow and credential-handling guidance.
|
||||||
|
|
||||||
|
When **local_backend** is present, its endpoint is trimmed and must use HTTP or
|
||||||
|
HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are
|
||||||
|
allowed. User information, queries, and fragments are rejected. A zero
|
||||||
|
**concurrency_limit** leaves the local backend unrestricted inside PromptKit;
|
||||||
|
a positive value limits simultaneous local generations. The application-wide
|
||||||
|
**concurrency.total_llm** limit still applies in both cases. Neither local
|
||||||
|
backend field has an environment override. Omitting **local_backend** registers
|
||||||
|
nothing and preserves existing built-in and endpoint-only profile behavior.
|
||||||
|
|
||||||
|
A file-backed PromptKit profile selects the registration by its case-sensitive
|
||||||
|
backend ID:
|
||||||
|
|
||||||
|
~~~yaml
|
||||||
|
id: local-summary
|
||||||
|
backend: local
|
||||||
|
model: example-model
|
||||||
|
~~~
|
||||||
|
|
||||||
|
Keep credentials out of the local-backend object. A PromptKit profile may name
|
||||||
|
its credential environment variable through `api_key_env`; set that variable
|
||||||
|
only in the run environment. PromptKit owns the
|
||||||
|
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md).
|
||||||
|
The [PromptKit upstream boundary](integrations/pkg-promptkit.md) identifies the
|
||||||
|
supported package API, and [Operations](operations.md#operational-limits)
|
||||||
|
describes the effective concurrency layers.
|
||||||
|
|
||||||
|
`notarius config validate --pipeline <id>` resolves the selected pipeline and
|
||||||
|
inspects every explicit effective profile without contacting a provider or
|
||||||
|
requiring credential values. It rejects absent, malformed, or incompatible
|
||||||
|
profiles before a run prepares modules. Credential availability is checked only
|
||||||
|
when a generation is prepared.
|
||||||
|
|
||||||
|
## Migrating Version 3 Configuration
|
||||||
|
|
||||||
|
Version 3 files are not decoded or rewritten. Change **version: 3** to
|
||||||
|
**version: 4** and rename the top-level **scriptorium:** section to
|
||||||
|
**promptkit:**. Version 4 decoding is strict, so a remaining **scriptorium**
|
||||||
|
field is rejected as unknown.
|
||||||
|
|
||||||
## Operational Environment Variables
|
## Operational Environment Variables
|
||||||
|
|
||||||
@@ -138,6 +204,7 @@ Each **pipelines** entry has a unique, non-empty ID and the following shape:
|
|||||||
~~~yaml
|
~~~yaml
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input: seriatim
|
||||||
chunk: generic
|
chunk: generic
|
||||||
output: json
|
output: json
|
||||||
@@ -150,6 +217,7 @@ pipelines:
|
|||||||
|
|
||||||
| Field | Type | Default | Rules |
|
| Field | Type | Default | Rules |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
|
| **llm_profile** | string | none | Optional non-empty default PromptKit profile ID for selected LLM-backed bindings and validators. An explicitly present blank value is invalid. |
|
||||||
| **input** | module binding | none | Required. |
|
| **input** | module binding | none | Required. |
|
||||||
| **chunk** | module binding | **generic** | Optional. |
|
| **chunk** | module binding | **generic** | Optional. |
|
||||||
| **output** | module binding | **json** | Optional. |
|
| **output** | module binding | **json** | Optional. |
|
||||||
@@ -163,6 +231,12 @@ needs a unique non-empty **id**, an **artifacts** map, and may have
|
|||||||
**references**. A lane ID must not appear more than once in a pipeline,
|
**references**. A lane ID must not appear more than once in a pipeline,
|
||||||
including across explicit steps.
|
including across explicit steps.
|
||||||
|
|
||||||
|
For each selected LLM-backed binding or validator, profile selection occurs
|
||||||
|
after module, validator, and `--only` lane selection. It uses the
|
||||||
|
run-level **--llm-profile** value first, then the binding's **llm_profile**,
|
||||||
|
then the pipeline's **llm_profile**, and finally the PromptKit default.
|
||||||
|
Deterministic bindings do not receive these defaults or run overrides.
|
||||||
|
|
||||||
A lane has these fields:
|
A lane has these fields:
|
||||||
|
|
||||||
| Field | Type | Default | Rules |
|
| Field | Type | Default | Rules |
|
||||||
@@ -190,7 +264,7 @@ Use an object for fields:
|
|||||||
~~~yaml
|
~~~yaml
|
||||||
extract:
|
extract:
|
||||||
module: dnd/spells
|
module: dnd/spells
|
||||||
llm_profile: gemini-2-flash
|
llm_profile: dnd-extraction
|
||||||
retries: 2
|
retries: 2
|
||||||
references:
|
references:
|
||||||
spell_catalog: ./dnd-spell-catalog.json
|
spell_catalog: ./dnd-spell-catalog.json
|
||||||
@@ -199,7 +273,7 @@ extract:
|
|||||||
| Binding field | Type | Default | Rules |
|
| Binding field | Type | Default | Rules |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| **module** | string | none | Required for an object binding. Must be a registered compatible key. |
|
| **module** | string | none | Required for an object binding. Must be a registered compatible key. |
|
||||||
| **llm_profile** | string | none | Optional non-empty Scriptorium profile ID. |
|
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID for an LLM-backed binding. It overrides the pipeline default unless the run supplies **--llm-profile**. |
|
||||||
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. |
|
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. |
|
||||||
| **options** | object | none | Must satisfy the selected module. |
|
| **options** | object | none | Must satisfy the selected module. |
|
||||||
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
|
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
|
||||||
@@ -209,7 +283,8 @@ Omitting **validators** uses the registered chain. **validators: []** selects
|
|||||||
an empty chain; a non-empty list replaces the chain in the listed order.
|
an empty chain; a non-empty list replaces the chain in the listed order.
|
||||||
Validator bindings accept only **module**, **llm_profile**, and **options**.
|
Validator bindings accept only **module**, **llm_profile**, and **options**.
|
||||||
They reject **references**, **retries**, and nested **validators**. Deterministic
|
They reject **references**, **retries**, and nested **validators**. Deterministic
|
||||||
validators reject an explicit **llm_profile**.
|
validators reject an explicit **llm_profile**. Deterministic module bindings
|
||||||
|
also reject an explicit **llm_profile**.
|
||||||
|
|
||||||
The **json** output module accepts optional **include_chunk_map** and
|
The **json** output module accepts optional **include_chunk_map** and
|
||||||
**evidence_context** settings:
|
**evidence_context** settings:
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ implemented component map.
|
|||||||
| Configuration loading, resolution, or user-visible configuration behavior | [Configuration Internals](internal/configuration.md) and [Configuration](config.md) | The internal guide owns loading and resolution mechanics; the reference owns the configuration contract. |
|
| Configuration loading, resolution, or user-visible configuration behavior | [Configuration Internals](internal/configuration.md) and [Configuration](config.md) | The internal guide owns loading and resolution mechanics; the reference owns the configuration contract. |
|
||||||
| Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
|
| Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
|
||||||
| Production modules or validators | [Module Internals](internal/modules.md), [D&D Module Internals](internal/dnd.md), and [D&D integration contracts](integrations/) | The generic guide owns extension mechanics, the D&D guide owns shared family conventions, and the contracts own durable output shapes. |
|
| Production modules or validators | [Module Internals](internal/modules.md), [D&D Module Internals](internal/dnd.md), and [D&D integration contracts](integrations/) | The generic guide owns extension mechanics, the D&D guide owns shared family conventions, and the contracts own durable output shapes. |
|
||||||
| LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and Scriptorium integration. |
|
| LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and PromptKit integration. |
|
||||||
| Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. |
|
| Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. |
|
||||||
| External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. |
|
| External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. |
|
||||||
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
|
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
|
||||||
|
|||||||
@@ -98,6 +98,21 @@ summarize results without embedding lane payload bytes. A chunk-plan summary is
|
|||||||
provenance for the plan used by this run; cache records, debug artifacts, and
|
provenance for the plan used by this run; cache records, debug artifacts, and
|
||||||
other operational state are not published as bundle files.
|
other operational state are not published as bundle files.
|
||||||
|
|
||||||
|
Each `llm_profiles` entry identifies effective, non-secret LLM execution
|
||||||
|
provenance:
|
||||||
|
|
||||||
|
| Field | Required | Meaning |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `id` | Yes | Selected PromptKit profile identifier. |
|
||||||
|
| `provider` | No | Notarius adapter provider identifier. |
|
||||||
|
| `model` | No | Effective provider model identifier. |
|
||||||
|
| `backend_id` | No | Effective PromptKit backend registration identifier. Endpoint-only profiles omit it. |
|
||||||
|
| `reasoning_effort` | No | Effective opaque provider reasoning setting. An empty or explicitly cleared setting is omitted. |
|
||||||
|
|
||||||
|
These values describe observed execution; they are not a backend-registration
|
||||||
|
interface. Entries that differ by backend or effective reasoning remain
|
||||||
|
distinct even when their profile, provider, and model are otherwise equal.
|
||||||
|
|
||||||
## Rejections And Warnings
|
## Rejections And Warnings
|
||||||
|
|
||||||
`rejected.json` is always an object with a `rejected` array. Each entry has
|
`rejected.json` is always an object with a `rejected` array. Each entry has
|
||||||
|
|||||||
106
docs/integrations/pkg-promptkit.md
Normal file
106
docs/integrations/pkg-promptkit.md
Normal file
@@ -0,0 +1,106 @@
|
|||||||
|
# PromptKit Integration
|
||||||
|
|
||||||
|
Notarius pins
|
||||||
|
[`gitea.maximumdirect.net/eric/promptkit` v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0)
|
||||||
|
as its in-process prompt engine. The upstream
|
||||||
|
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/consumers/pkg-promptkit.md)
|
||||||
|
owns the public engine API, and the upstream
|
||||||
|
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
|
||||||
|
owns prompt, profile, and schema file contracts.
|
||||||
|
|
||||||
|
## Supported Boundary
|
||||||
|
|
||||||
|
Notarius relies on the root `promptkit` package to:
|
||||||
|
|
||||||
|
- construct an `Engine` with filesystem-backed prompt, schema, and optional
|
||||||
|
operator and application-fallback profile sources;
|
||||||
|
- prepare one frozen execution from a `RunRequest` with named inline artifacts,
|
||||||
|
variables, a direct session ID, prompt identity, and profile selection, then
|
||||||
|
record credential-redacted details and run that exact execution;
|
||||||
|
- return rendered debug material, validated structured output, selected
|
||||||
|
profile, backend, effective model metadata, and token usage;
|
||||||
|
- register the optional conventional `local` backend through `BackendLocal`,
|
||||||
|
`LocalBackend`, and `WithBackend`;
|
||||||
|
- distinguish structured-output validation failure from execution failure; and
|
||||||
|
- identify a missing explicit profile through `ErrProfileNotFound` and backend
|
||||||
|
admission exhaustion through `ErrCapacityExceeded`.
|
||||||
|
|
||||||
|
The pinned
|
||||||
|
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/backends.go)
|
||||||
|
owns the registration and backend-capacity contract.
|
||||||
|
|
||||||
|
For one completion, the adapter calls `PrepareExecution`, takes a
|
||||||
|
caller-owned `Details` snapshot, and calls `RunPrepared` for that same opaque
|
||||||
|
prepared execution. It defers `Discard` for every unexecuted handle. Explicit
|
||||||
|
profile preflight uses `Engine.InspectProfile`; it does not prepare a synthetic
|
||||||
|
prompt. PromptKit's prepared handle, inspection result, and capacity-error
|
||||||
|
types stay inside the Notarius LLM adapter.
|
||||||
|
|
||||||
|
When a PromptKit profile and runtime override leave `temperature`, `max_tokens`,
|
||||||
|
or `top_p` unset, Notarius leaves that control unset as well. Compatible
|
||||||
|
providers therefore apply their own defaults; an operator that requires a
|
||||||
|
specific sampling value must select it explicitly in the profile or runtime
|
||||||
|
override.
|
||||||
|
|
||||||
|
Notarius does not use PromptKit's optional `ArtifactReader`. It materializes
|
||||||
|
source and reference content itself and supplies owned inline artifacts at the
|
||||||
|
adapter boundary. It also retains responsibility for pipeline retries,
|
||||||
|
scheduling, debug persistence, redaction, profile provenance, and conversion
|
||||||
|
from private model responses into durable domain artifacts.
|
||||||
|
|
||||||
|
Notarius sends its trimmed run session through PromptKit's direct session
|
||||||
|
field, which is authoritative for provider session behavior. It also retains
|
||||||
|
the same value as the `session_id` prompt variable for maintained prompt
|
||||||
|
compatibility. Session IDs are stable, non-secret correlation identifiers and
|
||||||
|
may be exposed to providers and provider observability.
|
||||||
|
|
||||||
|
Notarius records PromptKit's selected backend ID and effective reasoning
|
||||||
|
setting as optional run-manifest provenance. Endpoint-only profiles have no
|
||||||
|
backend ID. Debug prompt material also retains the selected backend ID and
|
||||||
|
PromptKit's stable lower-case `effective_model_params` JSON, which may include
|
||||||
|
`backend_id`. Notarius production configuration exposes one optional
|
||||||
|
conventional `local` registration. It does not expose a general user-defined
|
||||||
|
PromptKit backend registry. Endpoint-only profiles remain supported unchanged.
|
||||||
|
|
||||||
|
Notarius retains its application-wide scheduled client around the PromptKit
|
||||||
|
adapter. PromptKit may apply a narrower limit for the selected backend;
|
||||||
|
endpoint-only profiles have no such backend limit. The adapter translates
|
||||||
|
PromptKit capacity rejection into the provider-neutral Notarius
|
||||||
|
`ErrLLMCapacityExceeded` contract. It may include the normalized selected
|
||||||
|
backend ID in safe diagnostic context, without exposing PromptKit's capacity
|
||||||
|
error type, and leaves retries to the calling pipeline stage.
|
||||||
|
|
||||||
|
## Profile Sources And Compatibility
|
||||||
|
|
||||||
|
Notarius gives PromptKit the configured operator profile source, registered
|
||||||
|
application fallback profile assets, and optional backend registration through
|
||||||
|
the same construction path for inspection and execution. PromptKit owns the
|
||||||
|
resulting source precedence and strict profile parsing: a matching operator
|
||||||
|
profile is a complete replacement for a fallback or built-in profile, while an
|
||||||
|
invalid matching document fails instead of falling through. The operator
|
||||||
|
configuration and deployment workflow are defined in
|
||||||
|
[Configuration](../config.md#promptkit-profiles) and
|
||||||
|
[Operations](../operations.md#promptkit-profile-deployment).
|
||||||
|
|
||||||
|
Notarius supports this boundary against PromptKit v0.5.0. Its fallback source,
|
||||||
|
prepared-execution, inspection, and typed capacity APIs are used as public
|
||||||
|
upstream contracts; other PromptKit APIs or file-format behavior are not
|
||||||
|
implicitly supported. A dependency upgrade requires reviewing the adapter,
|
||||||
|
profile-source construction, and this compatibility statement against the
|
||||||
|
pinned upstream documentation.
|
||||||
|
|
||||||
|
## Notarius Ownership
|
||||||
|
|
||||||
|
[LLM Runtime Internals](../internal/llm.md) describes how Notarius mounts
|
||||||
|
module assets, maps its transport-neutral completion contract, prepares and
|
||||||
|
executes requests, validates output, records provenance, captures debug
|
||||||
|
material, redacts errors, and preserves timeout ownership.
|
||||||
|
[D&D Module Internals](../internal/dnd.md) owns the embedded
|
||||||
|
`dnd-extraction` fallback profile and the maintained D&D prompt defaults.
|
||||||
|
[Configuration](../config.md#promptkit-profiles) defines how a Notarius
|
||||||
|
configuration selects one PromptKit profile source and optionally registers
|
||||||
|
the conventional local backend.
|
||||||
|
|
||||||
|
PromptKit API or format changes outside this boundary are not implicitly
|
||||||
|
supported. Updating the pinned version requires reviewing the adapter and
|
||||||
|
profile/configuration contracts against the upstream documentation.
|
||||||
@@ -38,9 +38,14 @@ in [Configuration Internals](configuration.md).
|
|||||||
|
|
||||||
Configuration validation without a selected pipeline checks structural
|
Configuration validation without a selected pipeline checks structural
|
||||||
configuration only. Validation with a selected pipeline also builds the
|
configuration only. Validation with a selected pipeline also builds the
|
||||||
effective catalog, resolves the pipeline, and verifies explicitly selected
|
effective catalog, resolves the pipeline, and verifies every explicit effective
|
||||||
Scriptorium profiles. Pipeline listing validates configuration before returning
|
PromptKit profile. Selected LLM-backed input, chunk, lane, output, and validator
|
||||||
normalized, sorted identifiers.
|
profiles are inspected
|
||||||
|
against the configured PromptKit source and backend registrations without
|
||||||
|
loading a prompt or performing generation, so an unknown or invalid profile
|
||||||
|
fails before pipeline preparation. Credential availability remains an
|
||||||
|
execution-time concern. Pipeline listing validates configuration before
|
||||||
|
returning normalized, sorted identifiers.
|
||||||
|
|
||||||
## Production Composition
|
## Production Composition
|
||||||
|
|
||||||
@@ -51,12 +56,24 @@ catalog used for resolution and the concrete constructors used for preparation.
|
|||||||
Tests may provide a catalog or registries instead; production code must not
|
Tests may provide a catalog or registries instead; production code must not
|
||||||
silently merge an injected partial catalog with production registrations.
|
silently merge an injected partial catalog with production registrations.
|
||||||
|
|
||||||
The production LLM factory builds the Scriptorium-backed client from resolved
|
The production LLM factory builds one PromptKit-backed client from the resolved
|
||||||
configuration, creates one scheduler from the effective global LLM limit, and
|
**promptkit.profile_dir** or **promptkit.profile_file** source, attaches the
|
||||||
wraps the client before it reaches modules. Registration and LLM construction
|
profile-provenance recorder, creates one scheduler from the effective global
|
||||||
errors are returned before a pipeline is prepared. Concrete module keys and
|
LLM limit, and wraps the client before it reaches modules. Registration and LLM
|
||||||
validator chains are public configuration choices and remain documented in
|
construction errors are returned before a pipeline is prepared. Configuration
|
||||||
[Configuration](../config.md).
|
field definitions remain in [Configuration](../config.md#promptkit-profiles);
|
||||||
|
the D&D registrar's fallback profile assets and the adapter mechanics remain in
|
||||||
|
[LLM Runtime](llm.md).
|
||||||
|
|
||||||
|
The factory also accepts `LLMRuntimeOverrides`, whose reasoning pointer
|
||||||
|
preserves inherit, replace, and clear states across the composition boundary.
|
||||||
|
Run orchestration constructs this value from the mutually exclusive
|
||||||
|
`--reasoning-effort` and `--clear-reasoning-effort` controls. Absence preserves
|
||||||
|
a nil pointer, replacement is trimmed, and clear uses a non-nil empty string.
|
||||||
|
The same override reaches the one shared production client, checkpoint
|
||||||
|
identity, and debug invocation metadata. Persistent reasoning configuration
|
||||||
|
remains owned by PromptKit profiles; Notarius configuration has no reasoning
|
||||||
|
field.
|
||||||
|
|
||||||
## Run Orchestration
|
## Run Orchestration
|
||||||
|
|
||||||
@@ -68,7 +85,8 @@ handoff:
|
|||||||
2. create and validate a safe run identity, then allocate a debug bundle only
|
2. create and validate a safe run identity, then allocate a debug bundle only
|
||||||
when requested;
|
when requested;
|
||||||
3. build the effective catalog, resolve requested reference changes, resolve
|
3. build the effective catalog, resolve requested reference changes, resolve
|
||||||
the effective pipeline, and verify explicit Scriptorium profiles;
|
the effective pipeline, and inspect its explicit effective PromptKit
|
||||||
|
profiles;
|
||||||
4. materialize external or generated references and record redacted invocation
|
4. materialize external or generated references and record redacted invocation
|
||||||
and resolution provenance when debug capture is enabled;
|
and resolution provenance when debug capture is enabled;
|
||||||
5. construct registries, the scheduled LLM client, prepared modules, and the
|
5. construct registries, the scheduled LLM client, prepared modules, and the
|
||||||
|
|||||||
@@ -39,14 +39,22 @@ This establishes the public precedence order without giving environment input a
|
|||||||
second file schema. Loading and application reject malformed YAML, unsupported
|
second file schema. Loading and application reject malformed YAML, unsupported
|
||||||
file versions, unknown fields, invalid values, and identifiers that are empty
|
file versions, unknown fields, invalid values, and identifiers that are empty
|
||||||
or collide after whitespace normalization. The file application also makes the
|
or collide after whitespace normalization. The file application also makes the
|
||||||
effective extraction-worker default follow the effective LLM limit.
|
effective extraction-worker default follow the effective LLM limit. A present
|
||||||
|
PromptKit local-backend object requires and trims its endpoint, defaults its
|
||||||
|
omitted concurrency limit to zero, and is copied so the parsed file model
|
||||||
|
cannot alias the populated **Config**. A pipeline `llm_profile` is
|
||||||
|
presence-aware: omission remains empty, while a present blank value is
|
||||||
|
rejected and a non-empty file value is trimmed before it reaches **Config**.
|
||||||
|
|
||||||
**Config.Validate** checks configuration-only invariants before resolution. It
|
**Config.Validate** checks configuration-only invariants before resolution. It
|
||||||
rejects incompatible profile sources, invalid state-surface values, unsupported
|
rejects incompatible profile sources, invalid state-surface values, unsupported
|
||||||
concurrency settings, malformed bindings and references, invalid retries, and
|
concurrency settings, malformed bindings and references, invalid retries, and
|
||||||
invalid pipeline, step, or lane structure. Its errors retain the closest known
|
invalid pipeline, step, or lane structure. PromptKit local-backend validation
|
||||||
pipeline, lane, and binding context. It deliberately does not require modules
|
accepts only an absolute HTTP or HTTPS endpoint with a host and no user
|
||||||
to be registered: that requires a catalog and belongs to resolution.
|
information, query, or fragment, and rejects a negative local concurrency
|
||||||
|
limit. Its errors retain the closest known pipeline, lane, and binding context.
|
||||||
|
It deliberately does not require modules to be registered: that requires a
|
||||||
|
catalog and belongs to resolution.
|
||||||
|
|
||||||
The exact user-selectable values and validation rules are defined in
|
The exact user-selectable values and validation rules are defined in
|
||||||
[Configuration](../config.md). Keep additions to the file model, an
|
[Configuration](../config.md). Keep additions to the file model, an
|
||||||
@@ -56,13 +64,15 @@ environment override, its validation, and that reference in the same change.
|
|||||||
|
|
||||||
**Config.Resolve** first recomputes derived concurrency defaults and validates
|
**Config.Resolve** first recomputes derived concurrency defaults and validates
|
||||||
the configuration. It normalizes the requested pipeline ID, copies the selected
|
the configuration. It normalizes the requested pipeline ID, copies the selected
|
||||||
profile, applies a non-empty command-level LLM profile override to the
|
profile, and passes the non-empty command-level LLM profile override, requested
|
||||||
LLM-capable stage bindings, and calls the framework resolver with the requested
|
lane selection, and reference changes to the framework resolver.
|
||||||
lane selection and reference changes.
|
|
||||||
|
|
||||||
The command-level override does not replace an explicitly selected validator
|
After module and validator selection, the resolver applies the effective
|
||||||
profile. Validator bindings remain part of the resolved validator chain and
|
profile policy to LLM-backed bindings only: command override, binding profile,
|
||||||
are resolved under their own declared configuration.
|
pipeline profile, then the prompt default. Deterministic bindings remain
|
||||||
|
profile-free, and no second inheritance decision occurs during execution. The
|
||||||
|
public field definitions and precedence are owned by
|
||||||
|
[Configuration](../config.md#pipelines).
|
||||||
|
|
||||||
The framework resolver supplies defaults, selects lanes, resolves validator
|
The framework resolver supplies defaults, selects lanes, resolves validator
|
||||||
chains, checks registered module and artifact compatibility, validates module
|
chains, checks registered module and artifact compatibility, validates module
|
||||||
@@ -71,7 +81,8 @@ options, and returns the fixed ordered pipeline shape. The resulting
|
|||||||
changes, a clone of the input configuration, and the resolved pipeline.
|
changes, a clone of the input configuration, and the resolved pipeline.
|
||||||
Callers may therefore retain or modify their input slices and maps without
|
Callers may therefore retain or modify their input slices and maps without
|
||||||
changing the resolved result, and later consumers cannot mutate the original
|
changing the resolved result, and later consumers cannot mutate the original
|
||||||
configuration through the effective value.
|
configuration through the effective value. This ownership includes the nested
|
||||||
|
PromptKit local-backend value.
|
||||||
|
|
||||||
Resolution failures stop before module construction and source parsing. They
|
Resolution failures stop before module construction and source parsing. They
|
||||||
include an error path for an unconfigured pipeline, missing module, missing
|
include an error path for an unconfigured pipeline, missing module, missing
|
||||||
@@ -83,7 +94,8 @@ runtime error class described in the [CLI reference](../cli.md#output-streams-an
|
|||||||
|
|
||||||
The framework assigns the resolved pipeline a deterministic SHA-256 digest
|
The framework assigns the resolved pipeline a deterministic SHA-256 digest
|
||||||
after defaults, lane selection, module bindings, reference bindings, validator
|
after defaults, lane selection, module bindings, reference bindings, validator
|
||||||
chains, and artifact schema identity have been resolved. The digest excludes
|
chains, effective LLM profiles, and artifact schema identity have been
|
||||||
|
resolved. The digest excludes
|
||||||
its own stored value. It identifies resolved composition rather than raw YAML
|
its own stored value. It identifies resolved composition rather than raw YAML
|
||||||
bytes, a debug payload, or all runtime state. The CLI records it as invocation
|
bytes, a debug payload, or all runtime state. The CLI records it as invocation
|
||||||
provenance before execution; cache and checkpoint identity have additional
|
provenance before execution; cache and checkpoint identity have additional
|
||||||
@@ -94,9 +106,12 @@ Configuration summaries must use **Redacted**, **RedactedSummaryPayload**, or
|
|||||||
Those methods copy every binding and nested option container, replace values
|
Those methods copy every binding and nested option container, replace values
|
||||||
whose key is credential-shaped with **[REDACTED]**, and omit materialized
|
whose key is credential-shaped with **[REDACTED]**, and omit materialized
|
||||||
reference content while retaining safe binding and reference provenance. The
|
reference content while retaining safe binding and reference provenance. The
|
||||||
payload must not alias the source configuration or resolved pipeline. This
|
payload must not alias the source configuration or resolved pipeline.
|
||||||
redaction is deliberately narrow: it protects configuration summaries and does
|
PromptKit's local endpoint and concurrency limit are preserved as non-secret
|
||||||
not authorize recording arbitrary environment values or provider requests.
|
configuration metadata in the independently owned summary; the object contains
|
||||||
|
no credential value. This redaction is deliberately narrow: it protects
|
||||||
|
configuration summaries and does not authorize recording arbitrary environment
|
||||||
|
values or provider requests.
|
||||||
|
|
||||||
## Invariants To Preserve
|
## Invariants To Preserve
|
||||||
|
|
||||||
|
|||||||
@@ -22,10 +22,15 @@ does not repeat their JSON shapes or schemas.
|
|||||||
## Family Composition
|
## Family Composition
|
||||||
|
|
||||||
The D&D registrar registers the family’s artifact codecs, extractors, typed
|
The D&D registrar registers the family’s artifact codecs, extractors, typed
|
||||||
append-order mergers, normalizers, validators, prompt assets, and default
|
append-order mergers, normalizers, validators, prompt assets, fallback LLM
|
||||||
validator chains. Each extractor and normalizer has a stable module spec,
|
profile asset, and default validator chains. Each extractor and normalizer has
|
||||||
strict option decoding, and a typed builder. Configuration remains the
|
a stable module spec, explicit execution class, strict option decoding, and a
|
||||||
canonical owner of the exact keys and validator order.
|
typed builder. Scene chunking, every extractor, and NPC normalization are
|
||||||
|
registered as `llm_backed`; the remaining current D&D mergers and normalizers
|
||||||
|
are `deterministic`. The metadata is available to catalog inspection and
|
||||||
|
resolved-pipeline debug data and determines which selected bindings inherit the
|
||||||
|
pipeline profile. Configuration remains the canonical owner of the exact keys,
|
||||||
|
profile precedence, and validator order.
|
||||||
|
|
||||||
Private structured-LLM response schemas are deliberately minimal. They reject
|
Private structured-LLM response schemas are deliberately minimal. They reject
|
||||||
invalid JSON structure, missing required fields, incompatible types, and
|
invalid JSON structure, missing required fields, incompatible types, and
|
||||||
@@ -41,6 +46,15 @@ reference, and transcript assets instead of copying their text into individual
|
|||||||
modules. A manifest’s declared sequence, including cache-control placement, is
|
modules. A manifest’s declared sequence, including cache-control placement, is
|
||||||
part of the prompt behavior.
|
part of the prompt behavior.
|
||||||
|
|
||||||
|
Every maintained D&D LLM prompt selects `dnd-extraction` as its default
|
||||||
|
profile. The D&D registrar embeds that fallback profile with the maintained
|
||||||
|
OpenRouter model, timeout, and service-tier policy. An operator may provide a
|
||||||
|
complete profile with the same ID through the configured PromptKit source; that
|
||||||
|
definition replaces the fallback rather than merging with it. The fallback
|
||||||
|
leaves reasoning and optional sampling controls unspecified. Deployment profile
|
||||||
|
selection and the maintained operator example are documented in
|
||||||
|
[Configuration](../config.md#promptkit-profiles).
|
||||||
|
|
||||||
All extraction prompts share this four-message rendered prefix: the system
|
All extraction prompts share this four-message rendered prefix: the system
|
||||||
message without cache control, the identity message without cache control, the
|
message without cache control, the identity message without cache control, the
|
||||||
campaign-reference message with ephemeral cache control, and the chunk
|
campaign-reference message with ephemeral cache control, and the chunk
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
# LLM Runtime Internals
|
# LLM Runtime Internals
|
||||||
|
|
||||||
`internal/framework/llm` is Notarius’s provider-independent structured
|
`internal/framework/llm` is Notarius’s provider-independent structured
|
||||||
completion boundary. It adapts framework requests to Scriptorium, bounds
|
completion boundary. It adapts framework requests to PromptKit, bounds
|
||||||
provider calls, assembles registered prompt and schema assets, records selected
|
provider calls, assembles registered prompt and schema assets, records selected
|
||||||
profiles, and redacts provider errors. The architectural boundary is defined in
|
profiles, and redacts provider errors. The architectural boundary is defined in
|
||||||
[Architecture](../policy/architecture.md#llm-boundary); profile sources,
|
[Architecture](../policy/architecture.md#llm-boundary); profile sources,
|
||||||
credentials, and concurrency settings belong in
|
credentials, and concurrency settings belong in
|
||||||
[Configuration](../config.md#scriptorium-profiles) and
|
[Configuration](../config.md#promptkit-profiles) and
|
||||||
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||||||
|
|
||||||
## Structured Completion Boundary
|
## Structured Completion Boundary
|
||||||
@@ -24,22 +24,78 @@ adapter does not own source evidence, artifact conversion, normalization, or
|
|||||||
durable schemas. Those responsibilities remain with the module and its
|
durable schemas. Those responsibilities remain with the module and its
|
||||||
[integration contract](../integrations/).
|
[integration contract](../integrations/).
|
||||||
|
|
||||||
`ScriptoriumClient` validates the request target and prompt identity, maps each
|
`PromptKitClient` validates the request target and prompt identity, maps each
|
||||||
named material to a Scriptorium inline artifact while preserving its origin URI,
|
named material to a PromptKit inline artifact while preserving its origin URI,
|
||||||
forwards session and profile selection, then prepares and runs the prompt. It
|
maps the trimmed request session to PromptKit's direct per-run session field,
|
||||||
returns Scriptorium’s validated raw bytes rather than re-encoding the decoded
|
retains the same value as the `session_id` prompt variable for maintained
|
||||||
target. An empty optional material is represented as one space so its named
|
prompt compatibility, and forwards profile selection. It then creates one
|
||||||
input is retained by Scriptorium.
|
frozen prepared execution, captures its caller-owned credential-redacted
|
||||||
|
details for debug material, and executes that exact snapshot through
|
||||||
|
PromptKit's prepared-execution boundary. The direct field
|
||||||
|
is authoritative for provider session behavior. A session ID is a stable,
|
||||||
|
non-secret correlation identifier and may be exposed to providers and provider
|
||||||
|
observability. The adapter returns PromptKit’s validated raw bytes rather than
|
||||||
|
re-encoding the decoded target. An empty optional material is represented as
|
||||||
|
one space so its named input is retained by PromptKit.
|
||||||
|
|
||||||
An empty request profile lets the prompt select its configured default. The CLI
|
Client construction may also receive a run-wide reasoning-effort override from
|
||||||
prepares every explicitly selected binding profile before a run begins, so a
|
the CLI factory boundary. The adapter copies the caller-owned pointer and
|
||||||
missing explicit profile fails before stage execution. Calls record the profile
|
creates a fresh PromptKit execution override for each request: a nil pointer
|
||||||
actually selected by Scriptorium; the recorder deduplicates non-secret profile
|
inherits the selected profile, a non-empty value replaces it, and an empty
|
||||||
identity, provider, and model values for manifest use.
|
value clears inherited reasoning. The CLI's mutually exclusive
|
||||||
|
`--reasoning-effort` and `--clear-reasoning-effort` controls select those
|
||||||
|
states. With neither flag, profile behavior remains unchanged. Because
|
||||||
|
production constructs one shared client, the selected state applies uniformly
|
||||||
|
to module calls, retries, and LLM-backed validators for the whole run.
|
||||||
|
|
||||||
|
An empty request profile lets the prompt select its configured default. Before a
|
||||||
|
run begins, the CLI asks the adapter to inspect every explicit profile on the
|
||||||
|
resolved selected LLM-backed bindings and validators, including inherited
|
||||||
|
pipeline profiles. Inspection resolves the profile and its selected backend and
|
||||||
|
target without loading a prompt, reading credentials, admitting capacity, or
|
||||||
|
contacting a provider, so a missing or invalid explicit profile fails before
|
||||||
|
stage execution while a valid `api_key_env` may remain unset. Calls record the
|
||||||
|
profile actually selected by PromptKit. The recorder trims and deduplicates
|
||||||
|
non-secret profile identity, provider, model, selected backend ID, and
|
||||||
|
effective reasoning values for manifest use. Entries that differ in backend or
|
||||||
|
reasoning remain distinct and deterministically ordered. Endpoint-only profiles
|
||||||
|
retain an empty backend ID, which the published JSON omits. Successful
|
||||||
|
completion responses and recorded profile manifests identify the adapter
|
||||||
|
provider as `promptkit`.
|
||||||
|
|
||||||
|
The CLI's profile-inspection engine and the production adapter use the same
|
||||||
|
profile-source construction to apply the configured profile directory or file,
|
||||||
|
the optional registered fallback profile assets, and the optional conventional
|
||||||
|
`local` backend. Preflight therefore resolves the same profile sources and
|
||||||
|
backend membership as runtime without performing generation. Fallback assets
|
||||||
|
are mounted only when at least one source is registered. The production D&D
|
||||||
|
registrar contributes its `dnd-extraction` fallback, and the maintained D&D
|
||||||
|
prompts select that logical ID by default. PromptKit owns source precedence and
|
||||||
|
profile parsing: an operator-provided matching profile takes precedence over a
|
||||||
|
fallback profile without Notarius merging either document.
|
||||||
|
When the registration is absent, a profile selecting `backend: local` fails
|
||||||
|
inspection instead of falling back to a built-in or endpoint-only target.
|
||||||
|
|
||||||
|
Before execution, the adapter also contributes a non-secret checkpoint
|
||||||
|
fingerprint for the effective PromptKit profile source. It combines the
|
||||||
|
identity of PromptKit's compiled-in profile catalog with a deterministic digest
|
||||||
|
of every YAML profile in the configured profile directory, or of the configured
|
||||||
|
profile file, and a deterministic digest of the flattened fallback profile
|
||||||
|
assets. The fingerprint contains neither profile content nor source paths. It
|
||||||
|
covers inherited pipeline profiles, explicit binding profiles, and
|
||||||
|
prompt-selected defaults, so changing a model or other profile setting cannot
|
||||||
|
reuse checkpoints created under the
|
||||||
|
prior profile source. This cache identity is independent of durable
|
||||||
|
profile provenance: run manifests continue to list only profiles actually
|
||||||
|
observed during LLM calls. When the local backend is registered, a second
|
||||||
|
fingerprint hashes its trimmed endpoint behind a stable marker. Changing that
|
||||||
|
semantic execution target invalidates checkpoint reuse. The raw endpoint is not
|
||||||
|
stored in checkpoint identity, and the local concurrency limit is excluded
|
||||||
|
because it changes scheduling rather than execution semantics.
|
||||||
|
|
||||||
## Shared Provider-Call Limit
|
## Shared Provider-Call Limit
|
||||||
|
|
||||||
Production construction creates one Scriptorium client and wraps it in one
|
Production construction creates one PromptKit client and wraps it in one
|
||||||
scheduled client. The scheduler has a fixed, positive permit limit, serves
|
scheduled client. The scheduler has a fixed, positive permit limit, serves
|
||||||
queued calls in FIFO order, and removes a queued call when its context is
|
queued calls in FIFO order, and removes a queued call when its context is
|
||||||
cancelled. A granted permit is released exactly once on every completion path.
|
cancelled. A granted permit is released exactly once on every completion path.
|
||||||
@@ -51,12 +107,23 @@ worker counts cannot exceed the configured LLM limit. The configuration field
|
|||||||
and its effective default are owned by
|
and its effective default are owned by
|
||||||
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||||||
|
|
||||||
|
PromptKit applies a second, independent admission limit when the selected
|
||||||
|
profile names a limited backend. It sits beneath the Notarius scheduled client,
|
||||||
|
so it may narrow but cannot expand the application-wide limit. Built-in
|
||||||
|
OpenRouter profiles select PromptKit's reserved backend and its upstream
|
||||||
|
capacity policy. A positive configured local-backend limit bounds active local
|
||||||
|
generations inside PromptKit; zero leaves that backend unlimited there.
|
||||||
|
Endpoint-only profiles do not select a PromptKit backend and remain limited
|
||||||
|
only by the Notarius scheduler.
|
||||||
|
|
||||||
## Prompt And Schema Assets
|
## Prompt And Schema Assets
|
||||||
|
|
||||||
An `AssetRegistry` collects prompt and schema filesystems from production module
|
An `AssetRegistry` collects prompt, schema, and optional fallback-profile
|
||||||
families. It flattens registered roots into the Scriptorium filesystems and
|
filesystems from production module families. It flattens registered roots into
|
||||||
rejects invalid roots, unreadable assets, duplicate paths, and missing prompt
|
the corresponding PromptKit filesystems and rejects invalid roots, unreadable
|
||||||
or schema files during preparation. The framework’s `promptfs` helper combines
|
assets, duplicate paths, and missing prompt or schema files during preparation.
|
||||||
|
Fallback assets receive a safe content digest for checkpoint identity; raw
|
||||||
|
paths and bytes are never included. The framework’s `promptfs` helper combines
|
||||||
module-owned prompt files with reusable domain fragments without making the
|
module-owned prompt files with reusable domain fragments without making the
|
||||||
framework depend on D&D content.
|
framework depend on D&D content.
|
||||||
|
|
||||||
@@ -95,14 +162,24 @@ meaningful input placement and cache controls of the prompt being changed.
|
|||||||
|
|
||||||
## Validation, Repair, And Retries
|
## Validation, Repair, And Retries
|
||||||
|
|
||||||
Scriptorium performs prompt rendering, provider execution, and the prompt’s
|
PromptKit performs prompt rendering, provider execution, and the prompt’s
|
||||||
structured-output validation. The adapter reports an empty result, validation
|
structured-output validation. The adapter reports an empty result, validation
|
||||||
failure, empty structured body, or decode failure as
|
failure, empty structured body, or decode failure as
|
||||||
`ErrInvalidStructuredOutput`, while retaining the returned raw bytes and debug
|
`ErrInvalidStructuredOutput`, while retaining the returned raw bytes and debug
|
||||||
material when they exist. Provider failures remain operational errors rather
|
material when they exist. Provider failures remain operational errors rather
|
||||||
than output-validation failures.
|
than output-validation failures.
|
||||||
|
|
||||||
Prompt-declared repair is executed within Scriptorium’s structured-output flow.
|
When PromptKit rejects backend admission before generation, the adapter maps
|
||||||
|
`promptkit.ErrCapacityExceeded` to
|
||||||
|
`contracts.ErrLLMCapacityExceeded`, retaining prompt context and a redacted
|
||||||
|
upstream diagnostic without exposing the PromptKit sentinel or capacity-error
|
||||||
|
type as a framework contract. When supplied, the normalized selected backend
|
||||||
|
ID appears only in that safe application-owned diagnostic context. A canceled
|
||||||
|
caller context takes precedence. The adapter does not retry capacity failures;
|
||||||
|
the pipeline's existing binding attempt policy sees the operational error and
|
||||||
|
decides whether to rerun the complete operation.
|
||||||
|
|
||||||
|
Prompt-declared repair is executed within PromptKit’s structured-output flow.
|
||||||
The current production D&D prompt manifests set repair attempts to zero. That
|
The current production D&D prompt manifests set repair attempts to zero. That
|
||||||
setting does not replace pipeline retry behavior: a binding’s configured retry
|
setting does not replace pipeline retry behavior: a binding’s configured retry
|
||||||
count reruns its stage attempt after an error or rejection, and an exhausted
|
count reruns its stage attempt after an error or rejection, and an exhausted
|
||||||
@@ -111,22 +188,42 @@ attempt lifecycle, validation chains, and retry diagnostics; see
|
|||||||
[Pipeline Internals](pipeline.md#validation-retries-and-output) and the
|
[Pipeline Internals](pipeline.md#validation-retries-and-output) and the
|
||||||
[binding reference](../config.md#module-bindings-and-validators).
|
[binding reference](../config.md#module-bindings-and-validators).
|
||||||
|
|
||||||
|
## Timeout Ownership
|
||||||
|
|
||||||
|
The caller context remains the outer cancellation authority. PromptKit applies
|
||||||
|
a positive effective generation timeout as an inner request deadline; an
|
||||||
|
explicit zero disables only that generation deadline. The HTTP client timeout
|
||||||
|
is a separate transport-wide cap. Notarius forwards the caller context and
|
||||||
|
does not install another timeout wrapper around PromptKit.
|
||||||
|
|
||||||
|
The selected PromptKit profile owns generation settings. Notarius binding
|
||||||
|
retries remain outside the adapter and repeat the complete module operation
|
||||||
|
and validation chain. PromptKit does not add a provider retry loop.
|
||||||
|
Operator-facing behavior is summarized in
|
||||||
|
[Operations](../operations.md#operational-limits), and the pinned upstream
|
||||||
|
contract is identified in
|
||||||
|
[PromptKit Integration](../integrations/pkg-promptkit.md).
|
||||||
|
|
||||||
## Observability And Redaction
|
## Observability And Redaction
|
||||||
|
|
||||||
When debug recording is enabled, the pipeline decorates the shared client. The
|
When debug recording is enabled, the pipeline decorates the shared client. The
|
||||||
wrapper records prepared prompt and response material, timing, selected profile
|
wrapper records prepared prompt and response material, timing, selected profile
|
||||||
and model, and call identifiers in the run’s debug bundle, including material
|
and backend, effective model parameters, and call identifiers in the run’s
|
||||||
available from a failed structured completion. For a successful completion, a
|
debug bundle, including material available from a failed structured completion.
|
||||||
debug-write failure is surfaced; when the completion already failed, its call
|
Effective parameters use PromptKit's stable lower-case JSON field names and may
|
||||||
error remains the result. Debug-bundle location, retention, and handling are
|
include `backend_id`. For a successful completion, a debug-write failure is
|
||||||
operational concerns documented in [Operations](../operations.md#debug-bundles).
|
surfaced; when the completion already failed, its call error remains the
|
||||||
|
result. Debug-bundle location, retention, and handling are operational concerns
|
||||||
|
documented in [Operations](../operations.md#debug-bundles).
|
||||||
|
|
||||||
Run manifests receive selected profile summaries and component identities, not
|
Run manifests receive selected profile summaries, including optional effective
|
||||||
prompt, schema, source, reference, or response content. Provider error text is
|
backend and reasoning provenance, and component identities—not prompt, schema,
|
||||||
wrapped with prompt context and bearer credentials are redacted before it
|
source, reference, or response content. The published field semantics belong
|
||||||
crosses the runtime boundary. Known-secret redaction is available to other
|
to the [JSON output contract](../integrations/json-output.md#manifestjson).
|
||||||
runtime collaborators; it does not make prompt or response contents safe for
|
Provider error text is wrapped with prompt context and bearer credentials are
|
||||||
general logging.
|
redacted before it crosses the runtime boundary. Known-secret redaction is
|
||||||
|
available to other runtime collaborators; it does not make prompt or response
|
||||||
|
contents safe for general logging.
|
||||||
|
|
||||||
## Failure Boundaries
|
## Failure Boundaries
|
||||||
|
|
||||||
@@ -134,6 +231,8 @@ general logging.
|
|||||||
sources, invalid asset registration, or a non-positive scheduler limit.
|
sources, invalid asset registration, or a non-positive scheduler limit.
|
||||||
- Preparation failures, unavailable explicit profiles, provider failures, and
|
- Preparation failures, unavailable explicit profiles, provider failures, and
|
||||||
context cancellation propagate to the calling stage with context.
|
context cancellation propagate to the calling stage with context.
|
||||||
|
- Backend admission exhaustion is a provider-neutral operational error and is
|
||||||
|
not classified as invalid structured output or validator rejection.
|
||||||
- Malformed or schema-invalid provider output is classified separately as
|
- Malformed or schema-invalid provider output is classified separately as
|
||||||
invalid structured output so the module or pipeline can apply its own retry
|
invalid structured output so the module or pipeline can apply its own retry
|
||||||
and rejection policy.
|
and rejection policy.
|
||||||
|
|||||||
@@ -12,9 +12,15 @@ exceptions. See [D&D Module Internals](dnd.md) rather than adding them here.
|
|||||||
|
|
||||||
A module is a typed implementation registered for one pipeline stage. Its
|
A module is a typed implementation registered for one pipeline stage. Its
|
||||||
`ModuleSpec` is the public-to-the-framework declaration of its stable key,
|
`ModuleSpec` is the public-to-the-framework declaration of its stable key,
|
||||||
stage, required and provided capabilities, artifact kind, and accepted
|
stage, execution class, required and provided capabilities, artifact kind, and
|
||||||
reference slots. The framework uses that declaration to resolve a configured
|
accepted reference slots. The execution class states whether a module is
|
||||||
binding before it builds the implementation.
|
`deterministic` or `llm_backed`; registries retain it for catalog inspection and
|
||||||
|
resolved-pipeline debug data without constructing the module. The framework
|
||||||
|
uses the declaration to resolve a configured binding before it builds the
|
||||||
|
implementation. After selection, the resolver applies profile inheritance only
|
||||||
|
to bindings whose declared execution class is `llm_backed` and rejects a
|
||||||
|
binding-specific profile on a deterministic module. The user-facing precedence
|
||||||
|
contract belongs in [Configuration](../config.md#pipelines).
|
||||||
|
|
||||||
Implementations that accept options must provide both an option validator and
|
Implementations that accept options must provide both an option validator and
|
||||||
a builder. The validator is used while resolving configuration; the builder
|
a builder. The validator is used while resolving configuration; the builder
|
||||||
@@ -47,7 +53,7 @@ Production composition is intentionally split by family:
|
|||||||
- The Seriatim registrar provides the transcript input adapter. Its external
|
- The Seriatim registrar provides the transcript input adapter. Its external
|
||||||
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
|
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
|
||||||
- The D&D registrar provides its codecs, extractors, mergers, normalizers,
|
- The D&D registrar provides its codecs, extractors, mergers, normalizers,
|
||||||
validators, prompt assets, and default chains. Its behavioral conventions
|
validators, prompt assets, fallback profile asset, and default chains. Its behavioral conventions
|
||||||
are documented in [D&D Module Internals](dnd.md).
|
are documented in [D&D Module Internals](dnd.md).
|
||||||
|
|
||||||
The CLI owns the composition that invokes these registrars. A module package
|
The CLI owns the composition that invokes these registrars. A module package
|
||||||
@@ -59,10 +65,10 @@ packages depend on production extensions.
|
|||||||
1. Choose the pipeline stage and the typed artifact boundary. Put external
|
1. Choose the pipeline stage and the typed artifact boundary. Put external
|
||||||
input or durable artifact formats in the relevant integration contract,
|
input or durable artifact formats in the relevant integration contract,
|
||||||
not in this guide or in a private LLM response type.
|
not in this guide or in a private LLM response type.
|
||||||
2. Define a stable `ModuleSpec` with the exact capabilities and reference
|
2. Define a stable `ModuleSpec` with an explicit execution class, the exact
|
||||||
slots needed for the operation. Model a producer/consumer handoff as an
|
capabilities, and reference slots needed for the operation. Model a
|
||||||
artifact-compatible slot; configuration then chooses an external file or a
|
producer/consumer handoff as an artifact-compatible slot; configuration
|
||||||
generated binding.
|
then chooses an external file or a generated binding.
|
||||||
3. Implement strict option decoding, construction, and the typed stage
|
3. Implement strict option decoding, construction, and the typed stage
|
||||||
interface. Preserve caller ownership: do not retain mutable request data
|
interface. Preserve caller ownership: do not retain mutable request data
|
||||||
and return defensive copies where an implementation exposes stored data.
|
and return defensive copies where an implementation exposes stored data.
|
||||||
|
|||||||
@@ -34,6 +34,11 @@ requested lanes where that is supported, resolves validator chains, checks
|
|||||||
module capabilities and typed artifact compatibility, validates options, and
|
module capabilities and typed artifact compatibility, validates options, and
|
||||||
assigns a deterministic resolved-composition digest. The resolved pipeline
|
assigns a deterministic resolved-composition digest. The resolved pipeline
|
||||||
contains bindings and declared reference targets, not external reference bytes.
|
contains bindings and declared reference targets, not external reference bytes.
|
||||||
|
After selection, the resolver applies command, binding, and pipeline profile
|
||||||
|
precedence to LLM-backed bindings and validators only; prompt defaults remain
|
||||||
|
an empty resolved binding profile. Deterministic bindings remain profile-free.
|
||||||
|
These effective values are part of the digest, so execution and checkpoint
|
||||||
|
consumers do not repeat profile inheritance.
|
||||||
Configuration resolution supplies the selected profile and catalog; see
|
Configuration resolution supplies the selected profile and catalog; see
|
||||||
[Configuration Internals](configuration.md).
|
[Configuration Internals](configuration.md).
|
||||||
|
|
||||||
|
|||||||
@@ -39,7 +39,9 @@ codecs, loader, and recorder. The CLI constructs a recorder whenever checkpoint
|
|||||||
recording is enabled and constructs a loader only for a `--resume` invocation.
|
recording is enabled and constructs a loader only for a `--resume` invocation.
|
||||||
Identity incorporates explicit stable semantic fingerprints collected from
|
Identity incorporates explicit stable semantic fingerprints collected from
|
||||||
prepared modules and validators in addition to configuration, input,
|
prepared modules and validators in addition to configuration, input,
|
||||||
references, runtime overrides, and LLM profiles.
|
references, runtime overrides, observed LLM profiles, and the LLM runtime's
|
||||||
|
non-secret effective profile-source identity. A profile source change therefore
|
||||||
|
causes a cold miss even when the configured profile ID remains unchanged.
|
||||||
The serialized
|
The serialized
|
||||||
`workspace_schema_version` identifiers are frozen wire-compatibility fields;
|
`workspace_schema_version` identifiers are frozen wire-compatibility fields;
|
||||||
they do not describe a current public state surface.
|
they do not describe a current public state surface.
|
||||||
|
|||||||
@@ -42,6 +42,42 @@ evidence publication. Apply an appropriate umask and output-root access policy
|
|||||||
before enabling that option; the requested output modes alone may not be
|
before enabling that option; the requested output modes alone may not be
|
||||||
suitable for transcript-bearing bundles.
|
suitable for transcript-bearing bundles.
|
||||||
|
|
||||||
|
## PromptKit Profile Deployment
|
||||||
|
|
||||||
|
Profile deployment has four distinct layers:
|
||||||
|
|
||||||
|
| Layer | Owner | Operational role |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Prompts and schemas | Notarius module families | Embedded request and structured-output definitions. They are not deployment profile files. |
|
||||||
|
| Fallback profiles | Notarius module families | Embedded application defaults, including D&D's `dnd-extraction` profile. |
|
||||||
|
| Built-in profiles | PromptKit | Upstream catalog entries available when no higher-precedence source defines an ID. |
|
||||||
|
| Operator profiles | Deployment filesystem | Complete environment-specific definitions selected by `promptkit.profile_file` or `promptkit.profile_dir`. |
|
||||||
|
|
||||||
|
The maintained D&D pipeline uses the workload ID `dnd-extraction`. The
|
||||||
|
embedded fallback makes that ID usable without an operator file. Production,
|
||||||
|
development, and local deployments can each install a different complete
|
||||||
|
definition for the same ID, retaining the pipeline while choosing their own
|
||||||
|
model, backend, timeout, or reasoning policy. An operator definition wins over
|
||||||
|
the fallback; it is not merged with it. The configuration field and full
|
||||||
|
precedence rules are owned by [Configuration](config.md#promptkit-profiles).
|
||||||
|
|
||||||
|
Use a profile source owned by the service account, keep it readable only by
|
||||||
|
the intended operator, and supply provider credentials through the service
|
||||||
|
environment—not in the Notarius configuration or profile YAML. The maintained
|
||||||
|
[operator profile](../examples/profiles/dnd-extraction.yml) is secret-free and
|
||||||
|
can be copied as a format starting point. Validate a deployment without a
|
||||||
|
provider call or credentials:
|
||||||
|
|
||||||
|
~~~sh
|
||||||
|
notarius config validate --config /etc/notarius/config.yml --pipeline dnd-session
|
||||||
|
~~~
|
||||||
|
|
||||||
|
Profile paths are currently resolved from the process working directory, not
|
||||||
|
from the configuration file. The complete example's
|
||||||
|
`./examples/profiles/dnd-extraction.yml` path is valid for a repository-root
|
||||||
|
invocation only. Use absolute paths such as
|
||||||
|
`/etc/notarius/profiles/dnd-extraction.yml` for services and containers.
|
||||||
|
|
||||||
## Run Lifecycle
|
## Run Lifecycle
|
||||||
|
|
||||||
Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads
|
Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads
|
||||||
@@ -117,9 +153,15 @@ compatible recorded work. A resume request fails when checkpoint recording is
|
|||||||
disabled. Without **--resume**, a recording-enabled run executes normally and
|
disabled. Without **--resume**, a recording-enabled run executes normally and
|
||||||
does not load checkpoint state. Compatibility includes the resolved pipeline,
|
does not load checkpoint state. Compatibility includes the resolved pipeline,
|
||||||
input, selected lanes, runtime overrides, reference provenance, LLM-profile
|
input, selected lanes, runtime overrides, reference provenance, LLM-profile
|
||||||
provenance, and prepared-component fingerprints. A changed identity produces a
|
provenance, the effective PromptKit profile-source fingerprint, and
|
||||||
cold miss; Notarius does not migrate, rewrite, or delete older checkpoint
|
prepared-component fingerprints. When a local PromptKit backend is configured,
|
||||||
directories.
|
compatibility also includes a non-secret fingerprint of its endpoint. Changing
|
||||||
|
profile content or the local endpoint causes a cold miss; changing only the
|
||||||
|
local concurrency limit does not. A changed identity produces a cold miss;
|
||||||
|
Notarius does not migrate, rewrite, or delete older checkpoint directories.
|
||||||
|
Reasoning-effort inheritance, replacement, and explicit clearing are distinct
|
||||||
|
runtime identities, so checkpoints created under one state are not reused by
|
||||||
|
either of the others.
|
||||||
|
|
||||||
Checkpoint state is confined below an identity-specific path:
|
Checkpoint state is confined below an identity-specific path:
|
||||||
|
|
||||||
@@ -179,7 +221,9 @@ warning, checkpoint, chunk-plan, and terminal reporting artifacts. The trace
|
|||||||
contains allowlisted application diagnostic records and can include source or
|
contains allowlisted application diagnostic records and can include source or
|
||||||
derived application data. Neither surface is a cache input. Do not treat a
|
derived application data. Neither surface is a cache input. Do not treat a
|
||||||
debug bundle as safe to share merely because its configuration summary is
|
debug bundle as safe to share merely because its configuration summary is
|
||||||
redacted.
|
redacted. Invocation metadata omits reasoning effort when it is inherited,
|
||||||
|
records the replacement value when one is supplied, and records an empty value
|
||||||
|
when inherited reasoning was explicitly cleared.
|
||||||
|
|
||||||
Notarius never creates debug state without an explicit request and never
|
Notarius never creates debug state without an explicit request and never
|
||||||
automatically deletes a requested bundle. If allocation succeeds, the command
|
automatically deletes a requested bundle. If allocation succeeds, the command
|
||||||
@@ -207,9 +251,40 @@ or automatic cleanup command.
|
|||||||
|
|
||||||
## Operational Limits
|
## Operational Limits
|
||||||
|
|
||||||
Provider retries and timeouts are supplied by the selected Scriptorium profile.
|
Provider execution settings and the generation timeout come from the selected
|
||||||
Module retry settings and concurrency limits are configuration contracts; see
|
PromptKit profile. The invocation-only **--reasoning-effort** and
|
||||||
[module bindings](config.md#module-bindings-and-validators) and
|
**--clear-reasoning-effort** controls may replace or clear that profile setting
|
||||||
|
for all LLM-backed calls in one run without changing the profile. PromptKit
|
||||||
|
v0.5.0 does not add a provider retry loop. Notarius binding retries rerun the
|
||||||
|
complete module operation and validation chain as defined by
|
||||||
|
[module bindings](config.md#module-bindings-and-validators).
|
||||||
|
|
||||||
|
Timeouts are layered. Caller cancellation is the outer authority. A positive
|
||||||
|
effective generation timeout adds an inner request deadline, while zero
|
||||||
|
disables only that generation deadline. The HTTP client timeout remains a
|
||||||
|
transport-wide cap. Notarius does not add another timeout around PromptKit.
|
||||||
|
The pinned upstream boundary and profile-format links are in
|
||||||
|
[PromptKit Integration](integrations/pkg-promptkit.md).
|
||||||
|
|
||||||
|
Concurrency has two independent layers. Notarius **total_llm** is the
|
||||||
|
application-wide provider-call limit shared by all backends, modules, retries,
|
||||||
|
and validators. PromptKit may impose a narrower admission limit for the
|
||||||
|
selected backend. The effective active-generation bound is the intersection of
|
||||||
|
both limits and can therefore be lower than **total_llm**. Built-in OpenRouter
|
||||||
|
profiles use PromptKit's upstream backend limit; endpoint-only profiles have no
|
||||||
|
PromptKit backend limit and remain bounded by Notarius. For the configured
|
||||||
|
local backend, a zero **concurrency_limit** leaves only the Notarius scheduler
|
||||||
|
as a call limit. A positive value makes the effective active local-generation
|
||||||
|
bound the smaller of **total_llm** and that local limit.
|
||||||
|
|
||||||
|
For a positive local limit, PromptKit owns its default waiting capacity and
|
||||||
|
admission behavior. When a PromptKit backend has admitted all active and queued
|
||||||
|
work, a new call fails as capacity exhaustion before generation. The adapter
|
||||||
|
maps that failure to Notarius's existing provider-neutral capacity error and
|
||||||
|
does not retry it. The calling stage's configured retry policy applies
|
||||||
|
normally, and the run fails if those attempts are exhausted. Caller
|
||||||
|
cancellation remains authoritative. Configuration contracts are documented
|
||||||
|
under [PromptKit profiles](config.md#promptkit-profiles) and
|
||||||
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
|
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
|
||||||
limits and actual provider-call limits are independent. Notarius writes local
|
limits and actual provider-call limits are independent. Notarius writes local
|
||||||
filesystem state only; remote storage, archival, and retention automation are
|
filesystem state only; remote storage, archival, and retention automation are
|
||||||
|
|||||||
@@ -177,7 +177,8 @@ individual modules.
|
|||||||
The application-wide LLM scheduler bounds actual provider calls independently
|
The application-wide LLM scheduler bounds actual provider calls independently
|
||||||
of framework worker limits. Every LLM-backed module, retry, and validator uses
|
of framework worker limits. Every LLM-backed module, retry, and validator uses
|
||||||
the single injected scheduled client, including work performed by overlapping
|
the single injected scheduled client, including work performed by overlapping
|
||||||
lanes.
|
lanes. Provider runtime adapters may enforce a narrower backend-specific limit
|
||||||
|
beneath this mandatory application-wide scheduler.
|
||||||
|
|
||||||
## Configuration And Provenance
|
## Configuration And Provenance
|
||||||
|
|
||||||
|
|||||||
@@ -1,90 +0,0 @@
|
|||||||
# D&D Prompt Cache Ordering
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
Arrange maintained D&D prompts so sibling extraction lanes share the longest
|
|
||||||
useful byte-identical prefix, including the transcript chunk, and can therefore
|
|
||||||
reuse provider-side prompt caches effectively.
|
|
||||||
|
|
||||||
This policy concerns rendered message order and cache boundaries. It does not
|
|
||||||
change extraction semantics, artifact contracts, module inputs, durable output,
|
|
||||||
or the responsibilities of deterministic validators and normalizers.
|
|
||||||
|
|
||||||
## Extraction Prompt Policy
|
|
||||||
|
|
||||||
Every D&D extraction prompt shares this rendered prefix:
|
|
||||||
|
|
||||||
1. the shared D&D system prompt;
|
|
||||||
2. the shared D&D identity prompt;
|
|
||||||
3. the shared campaign-reference prompt; and
|
|
||||||
4. the shared transcript-chunk prompt.
|
|
||||||
|
|
||||||
The messages must render identically, including roles, content, and cache
|
|
||||||
metadata, when two lanes receive the same transcript and common references.
|
|
||||||
The campaign-reference and transcript messages establish ephemeral cache
|
|
||||||
boundaries.
|
|
||||||
|
|
||||||
Only material used by every extraction lane belongs before the transcript.
|
|
||||||
Shared wording that applies to only a subset of lanes is placed after the
|
|
||||||
transcript rather than being added to unrelated prompts solely to lengthen the
|
|
||||||
common prefix. Module instructions, optional generated references, catalogs,
|
|
||||||
and other lane-specific context also follow the transcript.
|
|
||||||
|
|
||||||
The lane-specific suffixes are:
|
|
||||||
|
|
||||||
- NPCs: shared extraction-evidence policy, task, then instructions.
|
|
||||||
- Item events: shared extraction-evidence policy, task, then instructions.
|
|
||||||
- Scene descriptions: task, then instructions.
|
|
||||||
- Combat turns: shared extraction-evidence policy, NPC registry, task, then
|
|
||||||
instructions.
|
|
||||||
- NPC interactions: shared extraction-evidence policy, NPC registry, task,
|
|
||||||
then instructions.
|
|
||||||
- Spells: shared extraction-evidence policy, NPC registry, spell catalog, task,
|
|
||||||
then instructions.
|
|
||||||
|
|
||||||
The final instructions message establishes an ephemeral cache boundary.
|
|
||||||
Intermediate cache markers on the shared identity message or lane-specific
|
|
||||||
reference messages are unnecessary.
|
|
||||||
|
|
||||||
## Purpose-Specific Prompt Families
|
|
||||||
|
|
||||||
The extraction prefix is a sibling-lane policy, not a universal ordering rule
|
|
||||||
for every D&D LLM call.
|
|
||||||
|
|
||||||
Scene chunking has no sibling lane with which to share a transcript prefix. It
|
|
||||||
orders its prompt as system, common campaign references, task, instructions,
|
|
||||||
and full transcript. The campaign-reference and transcript messages establish
|
|
||||||
ephemeral cache boundaries.
|
|
||||||
|
|
||||||
NPC normalization orders its prompt as system, task, instructions, candidate
|
|
||||||
NPCs, and transcript windows. The instructions and transcript-window messages
|
|
||||||
establish ephemeral cache boundaries. Candidate artifacts remain ahead of the
|
|
||||||
evidence windows needed to evaluate them.
|
|
||||||
|
|
||||||
Future D&D prompt families should identify their actual reuse boundary rather
|
|
||||||
than mechanically copying either exception or the extraction sequence.
|
|
||||||
|
|
||||||
## Compatibility And Observability
|
|
||||||
|
|
||||||
Existing prompt IDs, prompt versions, response schemas, and module contracts
|
|
||||||
remain unchanged. Prompt-content fingerprints already make the reordered
|
|
||||||
assets part of checkpoint identity, so old development checkpoints may become
|
|
||||||
cold misses without a checkpoint-format migration.
|
|
||||||
|
|
||||||
Prompt-order verification should exercise rendered messages with unique input
|
|
||||||
sentinels. Tests must protect roles, cache metadata, input isolation, and the
|
|
||||||
shared-prefix invariant without requiring particular prose, words, or phrases
|
|
||||||
to remain in prompt assets.
|
|
||||||
|
|
||||||
## Desired End State
|
|
||||||
|
|
||||||
- All six D&D extraction prompts render the same four-message prefix for
|
|
||||||
equivalent common inputs.
|
|
||||||
- The transcript chunk is included in that common prefix and is followed only
|
|
||||||
by lane-specific material.
|
|
||||||
- Cache-control hints identify useful prefix boundaries without redundant
|
|
||||||
intermediate markers.
|
|
||||||
- Scene chunking and NPC normalization retain orderings suited to their
|
|
||||||
distinct inputs and reuse opportunities.
|
|
||||||
- Internal documentation explains both the general provider-cache principle
|
|
||||||
and the concrete D&D prompt-family policy.
|
|
||||||
@@ -7,6 +7,37 @@ not as committed release dates.
|
|||||||
|
|
||||||
## Near-Term D&D Pipeline
|
## Near-Term D&D Pipeline
|
||||||
|
|
||||||
|
### Combat Enemy Ledger
|
||||||
|
|
||||||
|
- Add a D&D artifact that identifies enemies faced during combat and supports
|
||||||
|
an end-of-session encounter ledger.
|
||||||
|
- Track each enemy's observed state using a small controlled vocabulary such as
|
||||||
|
`active`, `killed`, `fled`, `captured`, or `incapacitated`, while preserving
|
||||||
|
an explicit unresolved state when the transcript does not establish an
|
||||||
|
outcome.
|
||||||
|
- Preserve the evidence for enemy participation and state changes rather than
|
||||||
|
inferring a terminal outcome from combat ending or an enemy disappearing
|
||||||
|
from the conversation.
|
||||||
|
- Define how repeated mentions, groups of unnamed enemies, summoned or allied
|
||||||
|
creatures, and the same enemy appearing in multiple combats affect identity
|
||||||
|
and ledger entries.
|
||||||
|
- Evaluate whether the ledger should be extracted directly, derived from
|
||||||
|
combat-turn artifacts, or use a sequential pipeline that consumes combat
|
||||||
|
turns and the normalized NPC registry as grounding references.
|
||||||
|
|
||||||
|
### Location Extraction
|
||||||
|
|
||||||
|
- Add a D&D artifact for locations visited by the party or otherwise mentioned
|
||||||
|
in the transcript.
|
||||||
|
- Distinguish observed visits from references, plans, recalled places, and
|
||||||
|
uncertain or inferred locations so a mention alone is not reported as a
|
||||||
|
visit.
|
||||||
|
- Preserve transcript evidence for each visit or mention and reconcile aliases,
|
||||||
|
nested places, and repeated appearances without collapsing distinct
|
||||||
|
locations that share a generic name.
|
||||||
|
- Define how the location artifact should ground later narrative reports and
|
||||||
|
whether future event artifacts should retain canonical location identities.
|
||||||
|
|
||||||
### Evaluate Spell Extraction And Normalization
|
### Evaluate Spell Extraction And Normalization
|
||||||
|
|
||||||
- Evaluate ordinary extraction retries and the completed normalization path
|
- Evaluate ordinary extraction retries and the completed normalization path
|
||||||
@@ -22,6 +53,71 @@ not as committed release dates.
|
|||||||
spell, combat, interaction, and scene-description lanes after real-world use.
|
spell, combat, interaction, and scene-description lanes after real-world use.
|
||||||
Add more complex chunking only in response to demonstrated failures.
|
Add more complex chunking only in response to demonstrated failures.
|
||||||
|
|
||||||
|
## Cross-Cutting LLM Runtime
|
||||||
|
|
||||||
|
### Deterministic Prompt Session Identity
|
||||||
|
|
||||||
|
- Replace the source-document-ID default for prompt sessions with one
|
||||||
|
predictable, procedurally generated session ID for the complete
|
||||||
|
source-processing workload.
|
||||||
|
- Preserve an explicit non-empty `--session-id` as the highest-precedence
|
||||||
|
override. Otherwise, derive the default only from the effective input module
|
||||||
|
identity and the exact raw input bytes.
|
||||||
|
- Use a versioned, bounded representation such as
|
||||||
|
`notarius:v1:<sha256(input-module + NUL + raw-input)>`. The exact encoding
|
||||||
|
must fit PromptKit's session length contract and must not embed source
|
||||||
|
content.
|
||||||
|
- Keep the derived session stable across runs, pipelines, selected lanes,
|
||||||
|
ordered steps, retries, resume, recomputation, LLM profiles, reasoning
|
||||||
|
overrides, and output, debug, or cache settings.
|
||||||
|
- Do not include file-backed references, generated references, reference
|
||||||
|
contents, or the composition of a reference bundle in session derivation.
|
||||||
|
References may change between prompt calls within one pipeline without
|
||||||
|
changing routing affinity.
|
||||||
|
- Resolve the authoritative session before checkpoint construction and use the
|
||||||
|
same value for checkpoint runtime identity, every prompt-facing module,
|
||||||
|
PromptKit's direct session field, the compatibility `session_id` prompt
|
||||||
|
variable, run-manifest metadata, and debug metadata.
|
||||||
|
- Keep routing identity separate from cache and checkpoint content identity.
|
||||||
|
Exact prompt prefixes, reference contents, model settings, and other
|
||||||
|
generation-affecting inputs must continue to participate in their existing
|
||||||
|
hashes and checkpoint fingerprints even though they do not change the
|
||||||
|
session.
|
||||||
|
- Treat the generated value as a provider-visible, stable pseudonymous
|
||||||
|
correlation identifier. Do not introduce an installation-specific HMAC or
|
||||||
|
secret unless a concrete multi-tenant or privacy requirement justifies
|
||||||
|
sacrificing deterministic identity across installations.
|
||||||
|
|
||||||
|
### Raise The Default Application-Wide LLM Limit
|
||||||
|
|
||||||
|
- Raise the default `concurrency.total_llm` value from 1 to 16 so ordinary
|
||||||
|
single-backend runs can use PromptKit's expected OpenRouter capacity and
|
||||||
|
lower-capacity local backends without an unnecessarily narrower Notarius
|
||||||
|
limit.
|
||||||
|
- Keep the Notarius application-wide scheduler mandatory and require
|
||||||
|
`total_llm` to remain a positive integer. Do not make the default unlimited:
|
||||||
|
endpoint-only profiles, an unrestricted local backend, injected clients, and
|
||||||
|
aggregate work across several backends may have no narrower PromptKit limit.
|
||||||
|
- Continue defaulting `concurrency.stage_workers.extract` to the effective
|
||||||
|
`total_llm`, making its default 16 as part of the same change. Preserve an
|
||||||
|
explicit lower extract-worker setting when an operator wants less queued or
|
||||||
|
concurrent extraction work.
|
||||||
|
- Define effective provider concurrency as the intersection of the Notarius
|
||||||
|
application-wide limit, the selected PromptKit backend limit when present,
|
||||||
|
and the work made available by stage execution. A Notarius limit of 16 does
|
||||||
|
not narrow a backend already limited to 16, while a local backend limited to
|
||||||
|
4 remains bounded at 4.
|
||||||
|
- Treat the default as an application-wide safety ceiling across profiles,
|
||||||
|
backends, modules, retries, and validators. A run that intentionally needs
|
||||||
|
the combined capacity of several backends may configure a higher
|
||||||
|
`total_llm` and an appropriate extract-worker count explicitly.
|
||||||
|
- Retain the existing configuration and environment override surfaces. Update
|
||||||
|
canonical configuration, operations, and internal documentation together
|
||||||
|
when the default changes.
|
||||||
|
- Reconsider decoupling the extract-worker default from `total_llm` only after
|
||||||
|
mixed-backend workloads demonstrate a need for a high global emergency
|
||||||
|
ceiling with a lower default work-production rate.
|
||||||
|
|
||||||
## Shared Normalization And Quality Work
|
## Shared Normalization And Quality Work
|
||||||
|
|
||||||
### Generic LLM-Assisted Deduplication
|
### Generic LLM-Assisted Deduplication
|
||||||
@@ -60,30 +156,6 @@ safety checks, and deterministic application of accepted changes.
|
|||||||
- Add media-type validators when non-JSON artifact representations are
|
- Add media-type validators when non-JSON artifact representations are
|
||||||
introduced.
|
introduced.
|
||||||
|
|
||||||
## LLM Runtime Evolution
|
|
||||||
|
|
||||||
### Native Session Propagation
|
|
||||||
|
|
||||||
- Once upstream Scriptorium exposes a session identifier on its run request,
|
|
||||||
propagate the existing `StructuredCompletionRequest.SessionID` through the
|
|
||||||
Scriptorium adapter's native session field.
|
|
||||||
- Preserve the current `--session-id` invocation contract and its run-wide
|
|
||||||
propagation to every prompt-facing module and validator. Do not introduce a
|
|
||||||
second session configuration surface.
|
|
||||||
- Retain session identity in checkpoint provenance so runs with different
|
|
||||||
sessions cannot reuse one another's LLM-derived checkpoints.
|
|
||||||
- Define the upstream compatibility and prompt-variable transition explicitly:
|
|
||||||
native provider session behavior must not silently remove a `session_id`
|
|
||||||
prompt variable while maintained prompts still consume it.
|
|
||||||
- Add adapter and assembled-run coverage for exact forwarding, trimming,
|
|
||||||
concurrent-run isolation, and unsupported-provider behavior once the
|
|
||||||
upstream contract is available.
|
|
||||||
|
|
||||||
This work is blocked on native session support in the upstream Scriptorium
|
|
||||||
package. Notarius already carries a run-scoped session ID through its CLI,
|
|
||||||
pipeline requests, checkpoint identity, and prompt variables; the missing
|
|
||||||
capability is native propagation across the LLM adapter boundary.
|
|
||||||
|
|
||||||
## Further Reference Evolution
|
## Further Reference Evolution
|
||||||
|
|
||||||
- Make prior-run artifacts easier to bind as references without changing the
|
- Make prior-run artifacts easier to bind as references without changing the
|
||||||
|
|||||||
@@ -1,257 +1,586 @@
|
|||||||
# D&D Prompt Cache Ordering Implementation Plan
|
# PromptKit v0.5 Implementation Plan
|
||||||
|
|
||||||
## Summary
|
## Objective
|
||||||
|
|
||||||
Implement the target state in
|
Implement the target state in
|
||||||
[D&D Prompt Cache Ordering](dnd-prompt-cache-ordering.md): give all six D&D
|
[PromptKit v0.5 Integration And LLM Profile Policy](promptkit.md). Each numbered
|
||||||
extraction prompts a byte-identical rendered prefix through the transcript
|
stage is intended to be one implementation prompt for a GPT-5.6-Terra coding
|
||||||
chunk, retain purpose-specific ordering for scene chunking and NPC
|
agent. Complete stages in order and leave the repository buildable, tested, and
|
||||||
normalization, consolidate tests around durable prompt-order invariants, and
|
internally coherent after every stage.
|
||||||
replace the current transcript-last documentation.
|
|
||||||
|
|
||||||
This is a prompt-asset and documentation change. Do not change Go module APIs,
|
Follow [Architecture](../policy/architecture.md),
|
||||||
LLM request contracts, prompt IDs or versions, response schemas, artifact
|
[Testing Policy](../policy/testing.md), and
|
||||||
schemas, reference declarations, extraction behavior, retry behavior, or
|
[Documentation Policy](../policy/documentation.md) throughout. Preserve
|
||||||
durable output. Existing prompt fingerprints must continue to provide
|
unrelated user changes. Use `apply_patch` for source and documentation edits,
|
||||||
checkpoint invalidation; no checkpoint migration or cleanup is required.
|
run `gofmt` on changed Go files, and add only tests that protect the behaviors
|
||||||
|
and risks assigned to that stage.
|
||||||
|
|
||||||
## Stage 1: Reorder The Six Extraction Prompts
|
Do not implement the separate deterministic session-ID or default-concurrency
|
||||||
|
roadmap items as part of this plan. Do not perform paid or credentialed LLM
|
||||||
|
calls.
|
||||||
|
|
||||||
Update these manifests:
|
## Background Summary
|
||||||
|
|
||||||
- `internal/modules/dnd/extract/npcs/assets/prompts/dnd.npcs.yaml`
|
Notarius currently pins PromptKit v0.3.0, calls `Prepare` and then `Run` for one
|
||||||
- `internal/modules/dnd/extract/itemevents/assets/prompts/dnd.item_events.yaml`
|
completion, validates profiles through a synthetic prompt, has no application
|
||||||
- `internal/modules/dnd/extract/scenedescriptions/assets/prompts/dnd.scene_descriptions.yaml`
|
fallback profile source, and accepts LLM profiles only at individual bindings
|
||||||
- `internal/modules/dnd/extract/combatturns/assets/prompts/dnd.combat_turns.yaml`
|
or through the run-wide CLI override. PromptKit v0.5.0 is source-compatible
|
||||||
- `internal/modules/dnd/extract/npcinteractions/assets/prompts/dnd.npc_interactions.yaml`
|
with the current tree; a temporary v0.5.0 module override has already passed
|
||||||
- `internal/modules/dnd/extract/spells/assets/prompts/dnd.spells.yaml`
|
`go test ./...`.
|
||||||
|
|
||||||
Give every manifest this exact leading message sequence:
|
The implementation must nevertheless treat the upstream optional-parameter
|
||||||
|
change as intentional: unset `temperature`, `max_tokens`, and `top_p` remain
|
||||||
|
unset and are omitted from compatible provider requests. Do not restore the old
|
||||||
|
implicit `top_p: 1` default.
|
||||||
|
|
||||||
1. `common-dnd-system.md`, with role `system` and no cache control;
|
## Stage 1: Upgrade The PromptKit Dependency
|
||||||
2. `common-dnd-identity.md`, with role `user` and no cache control;
|
|
||||||
3. `common-dnd-references.md`, with role `user` and ephemeral cache control;
|
|
||||||
4. `common-dnd-transcript.md`, with role `user` and ephemeral cache control.
|
|
||||||
|
|
||||||
The common prefix must contain no extraction-evidence policy, NPC registry,
|
### Goal
|
||||||
spell catalog, task, or module instructions. Those messages vary across lanes
|
|
||||||
and would prevent the transcript from participating in a shared cache prefix.
|
|
||||||
|
|
||||||
Append the following lane-specific sequences after the transcript:
|
Establish a clean PromptKit v0.5.0 baseline before adopting its new APIs.
|
||||||
|
|
||||||
| Prompt | Post-transcript sequence |
|
### Work
|
||||||
| --- | --- |
|
|
||||||
| `dnd.npcs` | `common-dnd-extraction-evidence.md`, `task.md`, `instructions.md` |
|
|
||||||
| `dnd.item_events` | `common-dnd-extraction-evidence.md`, `task.md`, `instructions.md` |
|
|
||||||
| `dnd.scene_descriptions` | `task.md`, `instructions.md` |
|
|
||||||
| `dnd.combat_turns` | `common-dnd-extraction-evidence.md`, `common-dnd-npcs.md`, `task.md`, `instructions.md` |
|
|
||||||
| `dnd.npc_interactions` | `common-dnd-extraction-evidence.md`, `common-dnd-npcs.md`, `task.md`, `instructions.md` |
|
|
||||||
| `dnd.spells` | `common-dnd-extraction-evidence.md`, `common-dnd-npcs.md`, `catalog.md`, `task.md`, `instructions.md` |
|
|
||||||
|
|
||||||
For every extraction prompt:
|
- Update `go.mod` and `go.sum` from PromptKit v0.3.0 to v0.5.0 and run
|
||||||
|
`go mod tidy`.
|
||||||
|
- Change the PromptKit built-in profile-catalog marker in
|
||||||
|
`internal/framework/llm/promptkit_profile_fingerprint.go` to identify
|
||||||
|
v0.5.0. This deliberately invalidates LLM checkpoints tied to the prior
|
||||||
|
catalog identity.
|
||||||
|
- Review PromptKit-facing compile errors or test failures against the v0.4.0
|
||||||
|
and v0.5.0 release guides. Do not adopt prepared execution, inspection, or
|
||||||
|
fallback profiles in this stage.
|
||||||
|
- Replace the existing test assertion for one exact built-in fingerprint hash
|
||||||
|
with durable assertions that the fingerprint is deterministic, non-empty,
|
||||||
|
non-secret, and changes when a semantic profile source changes. Do not add a
|
||||||
|
new version-constant or exact-hash change detector.
|
||||||
|
- Update `docs/integrations/pkg-promptkit.md` to pin and link v0.5.0 and state
|
||||||
|
the implemented dependency-level behavior: unset optional sampling controls
|
||||||
|
are provider defaults. Do not document later stages as implemented.
|
||||||
|
- Update any other canonical text that explicitly claims the dependency is
|
||||||
|
v0.3.0, but defer descriptions of unimplemented v0.5 APIs.
|
||||||
|
|
||||||
- apply ephemeral cache control to the final `instructions.md` message;
|
### Tests And Validation
|
||||||
- do not apply cache control to the extraction-evidence, NPC-registry, catalog,
|
|
||||||
or task messages;
|
|
||||||
- remove the existing cache control from `common-dnd-identity.md` and
|
|
||||||
`common-dnd-npcs.md`; and
|
|
||||||
- retain the existing inputs, requiredness, media types, output contract,
|
|
||||||
profile, repair setting, prompt identity, and schema path.
|
|
||||||
|
|
||||||
Do not add the extraction-evidence asset to the scene-description prompt merely
|
- `go test ./internal/framework/llm ./internal/cli`
|
||||||
to make it resemble the other lanes. It is not universal shared context and
|
- `go test ./...`
|
||||||
therefore belongs outside the common-prefix contract.
|
- `go vet ./...`
|
||||||
|
- `go build ./cmd/notarius`
|
||||||
|
- `rg -n 'promptkit v0\.3\.0|promptkit@v0\.3\.0|PromptKit v0\.3\.0' .`
|
||||||
|
- `git diff --check`
|
||||||
|
|
||||||
### Stage 1 completion criteria
|
### Completion Criteria
|
||||||
|
|
||||||
- Equivalent transcript, player, party, and glossary inputs render the same
|
- The repository directly pins v0.5.0 and all default offline checks pass.
|
||||||
first four messages for all six extraction prompts, including identical
|
- The profile-source fingerprint identifies the new upstream catalog without a
|
||||||
roles, content, and cache metadata.
|
brittle literal-hash test.
|
||||||
- Each transcript input is rendered exactly once.
|
- Current documentation no longer identifies v0.3.0 as the supported version.
|
||||||
- Every lane-specific message follows the transcript.
|
|
||||||
- Prompt preparation and schema wiring remain successful for every lane.
|
|
||||||
|
|
||||||
## Stage 2: Apply Purpose-Specific Ordering To Other D&D LLM Prompts
|
## Stage 2: Execute One Frozen Prepared Snapshot
|
||||||
|
|
||||||
Update
|
### Goal
|
||||||
`internal/modules/dnd/chunk/scenes/assets/prompts/dnd.scenes.yaml` to use:
|
|
||||||
|
|
||||||
1. `common-dnd-system.md`, without cache control;
|
Make Notarius debug details and generation use one exact PromptKit preparation.
|
||||||
2. `common-dnd-references.md`, with ephemeral cache control;
|
|
||||||
3. `task.md`, without cache control;
|
|
||||||
4. `instructions.md`, without cache control; and
|
|
||||||
5. `common-dnd-transcript.md`, with ephemeral cache control.
|
|
||||||
|
|
||||||
The scene chunker consumes the full transcript once and has no sibling lanes
|
### Work
|
||||||
that can share a transcript prefix. Stable task instructions therefore remain
|
|
||||||
before the changing full transcript.
|
|
||||||
|
|
||||||
Update
|
- Refactor `PromptKitClient.CompleteStructured` to call
|
||||||
`internal/modules/dnd/normalize/npcs/assets/prompts/dnd.npcs.normalize.yaml` to
|
`PrepareExecution`, immediately defer `Discard`, obtain a caller-owned
|
||||||
use:
|
`Details` value, and execute with `RunPrepared`.
|
||||||
|
- Preserve the existing Notarius request mapping, cancellation precedence,
|
||||||
|
validation classification, raw structured bytes, response decoding,
|
||||||
|
profile recording, usage reporting, and credential redaction.
|
||||||
|
- Ensure every preparation, execution, validation, empty-result, and decode
|
||||||
|
error retains useful Notarius prompt context without exposing prepared handle
|
||||||
|
state or secrets.
|
||||||
|
- Use `errors.As` to obtain `*promptkit.CapacityError` on admission rejection.
|
||||||
|
Preserve `contracts.ErrLLMCapacityExceeded` as the stable classification and
|
||||||
|
add a nonblank backend ID only to safe application-owned diagnostic context.
|
||||||
|
Do not expose `promptkit.CapacityError` outside the LLM adapter.
|
||||||
|
- Update `docs/internal/llm.md` and the implemented-mechanics portion of
|
||||||
|
`docs/integrations/pkg-promptkit.md` to describe the single frozen execution
|
||||||
|
snapshot and structured capacity adaptation.
|
||||||
|
|
||||||
1. `common-dnd-system.md`, without cache control;
|
### Tests And Validation
|
||||||
2. `task.md`, without cache control;
|
|
||||||
3. `instructions.md`, with ephemeral cache control;
|
|
||||||
4. `candidates.md`, without cache control; and
|
|
||||||
5. `common-dnd-transcript.md`, with ephemeral cache control.
|
|
||||||
|
|
||||||
Remove the normalizer system message's current cache control. Preserve the
|
- Adapt existing PromptKit client tests to the prepared-execution path.
|
||||||
candidate-before-evidence ordering: the transcript windows support evaluation
|
- Retain or add one behavioral test proving that the debug prompt details match
|
||||||
of the candidate collection and are not a cross-lane transcript prefix.
|
the request actually passed to generation when a backing prompt source could
|
||||||
|
otherwise change between independent preparations. Test the resulting
|
||||||
|
snapshot consistency, not a private helper call count.
|
||||||
|
- Retain capacity tests proving `errors.Is` reaches
|
||||||
|
`contracts.ErrLLMCapacityExceeded`, the selected backend can appear in safe
|
||||||
|
diagnostic context, and provider calls are not made after rejected
|
||||||
|
admission.
|
||||||
|
- Run `go test ./internal/framework/llm` and
|
||||||
|
`go test -race ./internal/framework/llm`.
|
||||||
|
- Run `go test ./...` and `git diff --check`.
|
||||||
|
|
||||||
Do not change input material construction, transcript windowing, prompt
|
### Completion Criteria
|
||||||
fragments, module implementations, or response handling in either package.
|
|
||||||
|
|
||||||
### Stage 2 completion criteria
|
- `CompleteStructured` no longer calls independent `Prepare` and `Run`
|
||||||
|
operations for one request.
|
||||||
|
- Debug prompt material and generation result originate from the same frozen
|
||||||
|
PromptKit snapshot.
|
||||||
|
- Capacity remains a provider-neutral Notarius error classification.
|
||||||
|
|
||||||
- Scene chunking renders references separately before the task and the full
|
## Stage 3: Replace Synthetic Profile Validation With Inspection
|
||||||
transcript exactly once at the end.
|
|
||||||
- NPC normalization renders candidates and transcript windows in separate
|
|
||||||
messages, in that order, with the agreed cache boundaries.
|
|
||||||
- Both prompts retain their current identities, schemas, inputs, and runtime
|
|
||||||
behavior.
|
|
||||||
|
|
||||||
## Stage 3: Consolidate Prompt-Ordering Verification
|
### Goal
|
||||||
|
|
||||||
Read and follow `docs/policy/testing.md` before modifying tests. The meaningful
|
Validate profiles through PromptKit's exact profile-inspection boundary and
|
||||||
risk is loss of the shared rendered prefix or incorrect placement of
|
centralize engine profile-source construction.
|
||||||
request-specific material, not edits to prompt prose.
|
|
||||||
|
|
||||||
Add one family-level extraction invariant test in
|
### Work
|
||||||
`internal/modules/dnd/register/prompt_cache_test.go` (or an equivalently scoped
|
|
||||||
new register-package test file):
|
|
||||||
|
|
||||||
- register the complete D&D prompt asset family in a real in-memory
|
- Introduce a small provider-adapter-owned profile inspection or validation
|
||||||
`llm.AssetRegistry`;
|
function in `internal/framework/llm`. Its public internal signature must use
|
||||||
- construct a Scriptorium engine with a local, non-networked test profile;
|
Notarius-owned configuration and result/error types rather than returning
|
||||||
- prepare all six extraction prompts using their exported prompt IDs and
|
PromptKit types to the CLI.
|
||||||
versions;
|
- Share the code that applies `profile_dir`, `profile_file`, and registered
|
||||||
- supply the same unique transcript, players, party, and glossary sentinels to
|
backend options between the production PromptKit engine and the inspection
|
||||||
every prompt, plus only the lane-specific required inputs such as the NPC
|
engine. Preserve the mutual-exclusion and local-backend rules.
|
||||||
registry and spell catalog;
|
- Change CLI explicit-profile preflight to use `Engine.InspectProfile` through
|
||||||
- locate the one rendered message containing the transcript sentinel rather
|
that LLM boundary.
|
||||||
than locating it through prompt prose or a hard-coded phrase;
|
- Remove `profileCheckPromptID`, `profileCheckPromptFS`, the `testing/fstest`
|
||||||
- assert that the transcript sentinel occurs exactly once in each prepared
|
production dependency, and the synthetic `Prepare` request.
|
||||||
prompt;
|
- Preserve distinct, useful errors for an absent profile, invalid profile,
|
||||||
- compare the rendered messages from the start through the transcript message
|
unknown backend registration, cancellation, and invalid profile source.
|
||||||
across all six prompts, including role, full rendered content, and complete
|
- Do not require `api_key_env` to be populated during configuration validation.
|
||||||
cache-control metadata;
|
Inspection may report credential requirements internally, but actual
|
||||||
- assert that the shared prefix contains exactly the four agreed messages and
|
preparation remains responsible for credential availability before a model
|
||||||
that at least one lane-specific message follows it in every extraction
|
call.
|
||||||
prompt; and
|
- Update current-behavior sections in `docs/internal/cli.md` and
|
||||||
- assert that NPC-registry and spell-catalog sentinels, where supplied, occur
|
`docs/internal/llm.md`. Keep field definitions in `docs/config.md`.
|
||||||
only after the transcript.
|
|
||||||
|
|
||||||
Use this family-level test as the single owner of the cross-lane prefix policy.
|
### Tests And Validation
|
||||||
Do not duplicate the same prefix assertion in every extractor package.
|
|
||||||
|
|
||||||
Revise the existing prompt-asset tests in:
|
- Replace synthetic-prompt tests with profile inspection tests covering:
|
||||||
|
configured local backend success; missing local backend failure; absent
|
||||||
|
profile; malformed profile; and an otherwise valid profile whose credential
|
||||||
|
environment variable is intentionally unset.
|
||||||
|
- Prove validation performs no provider HTTP call and remains offline.
|
||||||
|
- Run `go test ./internal/framework/llm ./internal/cli` and `go test ./...`.
|
||||||
|
- Run `git diff --check`.
|
||||||
|
|
||||||
- `internal/modules/dnd/extract/npcs/scriptorium_assets_test.go`
|
### Completion Criteria
|
||||||
- `internal/modules/dnd/extract/itemevents/scriptorium_assets_test.go`
|
|
||||||
- `internal/modules/dnd/extract/scenedescriptions/scriptorium_assets_test.go`
|
|
||||||
- `internal/modules/dnd/extract/combatturns/scriptorium_assets_test.go`
|
|
||||||
- `internal/modules/dnd/extract/npcinteractions/scriptorium_assets_test.go`
|
|
||||||
- `internal/modules/dnd/extract/spells/scriptorium_assets_test.go`
|
|
||||||
- `internal/modules/dnd/chunk/scenes/scriptorium_assets_test.go`
|
|
||||||
- `internal/modules/dnd/normalize/npcs/scriptorium_assets_test.go`
|
|
||||||
|
|
||||||
Remove transcript-last expectations that contradict the new extraction policy.
|
- No production synthetic profile-check prompt remains.
|
||||||
Keep package-local coverage only where it protects package-owned behavior, such
|
- Profile validation uses the same ordinary profile source and backend
|
||||||
as prompt/schema registration, declared shared assets, isolation and placement
|
registrations as execution.
|
||||||
of package-specific input material, cache boundaries unique to the scene
|
- Configuration validation succeeds for structurally valid profiles without
|
||||||
chunker or NPC normalizer, and non-disclosure properties. Consolidate or remove
|
reading credential values.
|
||||||
positional and prose-marker checks already covered by the family-level
|
|
||||||
invariant.
|
|
||||||
|
|
||||||
Tests may search for unique input sentinels created by the test. They must not
|
## Stage 4: Add Application Fallback Profile Asset Plumbing
|
||||||
require the presence of specific words, phrases, sentences, or other prose in a
|
|
||||||
prompt asset. Do not introduce golden snapshots of complete prompts or a test
|
|
||||||
that treats harmless wording changes as failures.
|
|
||||||
|
|
||||||
### Stage 3 completion criteria
|
### Goal
|
||||||
|
|
||||||
- One durable family-level test fails if any extraction lane moves
|
Allow module families to register application-owned fallback profile YAML
|
||||||
lane-specific material ahead of the transcript or changes the common
|
without placing domain policy in generic LLM code.
|
||||||
rendered prefix.
|
|
||||||
- Local asset tests protect only nonredundant package behavior and the two
|
|
||||||
purpose-specific prompt sequences.
|
|
||||||
- The test suite contains no prompt-language change detector.
|
|
||||||
- All prompt tests remain deterministic, offline, and independent of
|
|
||||||
credentials.
|
|
||||||
|
|
||||||
## Stage 4: Update Canonical Internal Documentation
|
### Work
|
||||||
|
|
||||||
Update `docs/internal/llm.md` as the generic owner of prompt-cache mechanics:
|
- Extend `internal/framework/llm.AssetRegistry` with a separate fallback
|
||||||
|
profile source collection, registration method, flattened filesystem, and
|
||||||
|
safe content digest.
|
||||||
|
- Reuse the existing asset-source path validation and flattening behavior where
|
||||||
|
appropriate. Reject invalid roots, unreadable assets, and duplicate flattened
|
||||||
|
paths. Do not parse PromptKit profile YAML in Notarius.
|
||||||
|
- Add `promptkit.WithFallbackProfileFS` to production engine options only when
|
||||||
|
at least one fallback profile source is registered.
|
||||||
|
- Supply the identical assembled fallback source to the profile-inspection
|
||||||
|
engine. Adjust CLI composition so pipeline-aware profile validation can use
|
||||||
|
the production LLM asset registry without exposing PromptKit types.
|
||||||
|
- Extend profile-source checkpoint identity to include the exact fallback
|
||||||
|
profile asset digest in addition to the PromptKit catalog marker and operator
|
||||||
|
source. Keep the resulting fingerprint hash-only and path/content/credential
|
||||||
|
free.
|
||||||
|
- Keep operator source precedence owned by PromptKit. Do not implement profile
|
||||||
|
merging or duplicate PromptKit source resolution in Notarius.
|
||||||
|
- Update `docs/internal/llm.md` only for the new implemented generic asset and
|
||||||
|
fingerprint mechanics. No domain fallback exists until Stage 5.
|
||||||
|
|
||||||
- explain that backend reuse depends on identical preceding roles, rendered
|
### Tests And Validation
|
||||||
bytes, and cache metadata, not merely equivalent semantics;
|
|
||||||
- state the general sibling-prompt ordering rule: universal shared context,
|
|
||||||
request source material, then module-specific suffixes;
|
|
||||||
- explain that a cache boundary should be placed at a useful reusable prefix
|
|
||||||
and that redundant intermediate boundaries add no value;
|
|
||||||
- retain the rule that prompt-family owners may choose a different sequence
|
|
||||||
when their inputs and reuse pattern differ; and
|
|
||||||
- replace the current statement that D&D extraction puts transcripts last with
|
|
||||||
a link to the D&D-specific policy.
|
|
||||||
|
|
||||||
Update `docs/internal/dnd.md` as the concrete owner of maintained D&D prompt
|
- Add focused AssetRegistry tests for successful flattening, invalid roots,
|
||||||
composition:
|
duplicate paths, and hash changes when fallback bytes change.
|
||||||
|
- Add adapter-level tests showing that the fallback filesystem reaches both
|
||||||
|
execution construction and inspection construction.
|
||||||
|
- Extend checkpoint tests to prove fallback content changes profile-source
|
||||||
|
identity without exposing raw YAML or paths. Use relational comparisons, not
|
||||||
|
a fixed hash literal.
|
||||||
|
- Run `go test ./internal/framework/llm ./internal/cli` and `go test ./...`.
|
||||||
|
- Run `git diff --check`.
|
||||||
|
|
||||||
- document the four-message extraction prefix and its cache boundaries;
|
### Completion Criteria
|
||||||
- state that extraction-evidence policy, generated NPC registries, catalogs,
|
|
||||||
tasks, and instructions follow the transcript because they are not universal
|
|
||||||
across all extraction lanes;
|
|
||||||
- summarize the scene-chunking and NPC-normalization exceptions and their
|
|
||||||
rationale; and
|
|
||||||
- retain existing rules about shared-asset reuse, canonical reference input,
|
|
||||||
prompt fingerprints, and prompt behavior.
|
|
||||||
|
|
||||||
Do not update the README, configuration reference, architecture policy, CLI
|
- Generic plumbing can carry application fallback profiles while remaining
|
||||||
documentation, or integration contracts. Prompt message order is an internal
|
unaware of D&D IDs or model settings.
|
||||||
runtime policy and has no user-visible configuration or wire-contract change.
|
- Inspection, execution, and checkpoint identity use the same fallback asset
|
||||||
|
source.
|
||||||
|
|
||||||
### Stage 4 completion criteria
|
## Stage 5: Adopt The D&D `dnd-extraction` Fallback
|
||||||
|
|
||||||
- No current-behavior document still says that D&D extraction transcripts are
|
### Goal
|
||||||
final messages.
|
|
||||||
- Generic cache mechanics and D&D-specific ordering each have one canonical
|
|
||||||
owner.
|
|
||||||
- Documentation describes implemented behavior once the prompt changes land
|
|
||||||
and does not duplicate volatile manifest inventories unnecessarily.
|
|
||||||
|
|
||||||
## Stage 5: Verification
|
Give the D&D module family one stable embedded workload profile that operators
|
||||||
|
can replace.
|
||||||
|
|
||||||
Inspect representative prepared prompts to confirm that the test inputs produce
|
### Work
|
||||||
the intended shared prefix and lane-specific suffixes. Then run:
|
|
||||||
|
- Add a D&D-owned embedded PromptKit profile asset with ID `dnd-extraction`
|
||||||
|
under `internal/modules/dnd`. Use the exact baseline defined in
|
||||||
|
`promptkit.md`: OpenRouter, `openai/gpt-5.6-luna`, no explicit reasoning
|
||||||
|
effort, a 240-second timeout, flex service tier, and no selected temperature,
|
||||||
|
token limit, or `top_p`. The omitted reasoning value intentionally allows
|
||||||
|
OpenAI's backend to apply its `medium` default.
|
||||||
|
- Register the profile filesystem from the D&D registrar through the generic
|
||||||
|
fallback profile asset boundary. Keep D&D policy out of
|
||||||
|
`internal/framework/llm` and the CLI composition root.
|
||||||
|
- Change every maintained D&D LLM prompt definition—including scene chunking,
|
||||||
|
all D&D extractors, and NPC normalization—from the model-named default to
|
||||||
|
`default_profile: dnd-extraction`.
|
||||||
|
- Add an integration-level profile-resolution test proving that:
|
||||||
|
- the fallback resolves when no operator source defines the ID;
|
||||||
|
- a valid operator profile with the same ID wins completely; and
|
||||||
|
- an invalid matching operator profile fails rather than falling through.
|
||||||
|
- Test through Notarius's assembled production assets and PromptKit boundary;
|
||||||
|
do not duplicate every upstream source-precedence case.
|
||||||
|
- Update the implemented profile ownership and prompt-default behavior in
|
||||||
|
`docs/internal/dnd.md`, `docs/internal/llm.md`, and
|
||||||
|
`docs/integrations/pkg-promptkit.md`. Defer the complete operator walkthrough
|
||||||
|
and examples to Stage 10.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Run focused D&D prompt preparation tests and the production composition
|
||||||
|
tests.
|
||||||
|
- Run `go test ./internal/modules/dnd/... ./internal/framework/llm
|
||||||
|
./internal/cli`.
|
||||||
|
- Run `go test ./...`.
|
||||||
|
- Verify `rg -n 'default_profile: gemini-2-flash' internal/modules/dnd`
|
||||||
|
returns no matches.
|
||||||
|
- Run `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- All maintained D&D prompts use the application-owned logical profile ID.
|
||||||
|
- The fallback works without an operator profile and remains authoritatively
|
||||||
|
overridable by a matching valid operator definition.
|
||||||
|
|
||||||
|
## Stage 6: Introduce Module Execution-Class Metadata
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Make each production module's ability to use an LLM statically discoverable
|
||||||
|
without yet changing profile inheritance.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Add `ExecutionClass contracts.ExecutionClass` to `pipeline.ModuleSpec` and
|
||||||
|
preserve it through normalization, cloning, catalogs, registries, JSON/debug
|
||||||
|
views, and lookup helpers.
|
||||||
|
- In this transitional stage only, allow an omitted execution class to
|
||||||
|
normalize to deterministic so existing test-only fixtures can be migrated in
|
||||||
|
Stage 7 without breaking the repository midway.
|
||||||
|
- Explicitly classify every production module:
|
||||||
|
- D&D scene chunking, every D&D extractor, and D&D NPC normalization as
|
||||||
|
`llm_backed`;
|
||||||
|
- all other current production input, chunk, merge, normalize, and output
|
||||||
|
modules as `deterministic`.
|
||||||
|
- Update production module specification tests and production catalog tests to
|
||||||
|
assert the semantic class alongside stage, artifact kind, and capabilities.
|
||||||
|
- Add catalog lookup support needed by later resolution to retrieve a selected
|
||||||
|
module's execution class by stage and key without constructing it.
|
||||||
|
- Do not implement pipeline-level profile inheritance or reject deterministic
|
||||||
|
profiles yet.
|
||||||
|
- Update `docs/internal/modules.md` and `docs/internal/dnd.md` to identify
|
||||||
|
execution class as registered module metadata, while noting only implemented
|
||||||
|
uses.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Run module registration/spec tests across generic, Seriatim, and D&D
|
||||||
|
families.
|
||||||
|
- Run `go test ./internal/framework/pipeline ./internal/modules/...`.
|
||||||
|
- Run `go test ./...` and `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- Every production module has an explicit correct execution class.
|
||||||
|
- Catalog consumers can retrieve that class without a concrete module
|
||||||
|
instance.
|
||||||
|
- Test-only omitted classes remain the only temporary compatibility behavior.
|
||||||
|
|
||||||
|
## Stage 7: Enforce Execution Metadata And Remove Runtime Probing
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Finish the execution-class contract so missing metadata cannot cause future
|
||||||
|
profile drift.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Update every framework, CLI, and integration test module specification to
|
||||||
|
declare an explicit execution class appropriate to the fake behavior.
|
||||||
|
- Change module-spec validation so an empty or unsupported execution class is a
|
||||||
|
registration error. Remove the transitional deterministic default from
|
||||||
|
Stage 6.
|
||||||
|
- Replace the chunk runner's special `ChunkExecutionClassProvider` probe with
|
||||||
|
specification-derived behavior. Remove the now-redundant provider interface,
|
||||||
|
implementation methods, and tests when they have no remaining consumer.
|
||||||
|
- Ensure chunk producer provenance remains unchanged: it records a non-empty
|
||||||
|
effective binding profile for an LLM-backed chunker, while a deterministic
|
||||||
|
chunker records no profile. A profile selected only through the prompt
|
||||||
|
default remains represented by PromptKit's actual-profile manifest rather
|
||||||
|
than being invented as an explicit chunk binding.
|
||||||
|
- Review helper constructors and fixtures for opportunities to set execution
|
||||||
|
class once without obscuring the class under test. Do not introduce an
|
||||||
|
elaborate test-spec framework.
|
||||||
|
- Update internal documentation if the removal changes any described runtime
|
||||||
|
mechanics.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Add or retain focused registration tests for missing and invalid execution
|
||||||
|
classes.
|
||||||
|
- Retain chunk-plan provenance tests for LLM-backed and deterministic
|
||||||
|
chunkers.
|
||||||
|
- Run `go test ./internal/framework/pipeline ./internal/modules/...`.
|
||||||
|
- Run `go test ./...`, `go vet ./...`, and `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- No registered module specification relies on an implicit execution class.
|
||||||
|
- Pipeline metadata, not a concrete runtime type assertion, owns module
|
||||||
|
execution classification.
|
||||||
|
|
||||||
|
## Stage 8: Resolve Programmatic Pipeline Profile Defaults
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Implement profile inheritance and precedence inside the pipeline resolver
|
||||||
|
before exposing the field through YAML configuration.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Add an optional trimmed `LLMProfile` field to
|
||||||
|
`pipeline.PipelineProfile`. Add a non-empty runtime override field to
|
||||||
|
`pipeline.ResolveOptions` so all precedence decisions occur in the resolver
|
||||||
|
rather than through pre-resolution mutation.
|
||||||
|
- After module selection, `--only` filtering, default validator-chain
|
||||||
|
selection, and validator compatibility resolution, apply effective profiles
|
||||||
|
to every selected input, chunk, extract, merge, normalize, output, and
|
||||||
|
validator binding according to the precedence in `promptkit.md`.
|
||||||
|
- Apply profiles only when the selected module or validator execution class is
|
||||||
|
`llm_backed`.
|
||||||
|
- Reject a binding-specific `llm_profile` on any deterministic module or
|
||||||
|
validator. Do not reject or inspect an unused pipeline default when no
|
||||||
|
selected LLM-backed binding consumes it.
|
||||||
|
- Leave an LLM-backed binding empty when no CLI, binding, or pipeline profile is
|
||||||
|
selected so PromptKit can use the prompt's `default_profile`.
|
||||||
|
- Store the effective values on resolved bindings before digest construction.
|
||||||
|
Do not add a second inheritance decision to execution.
|
||||||
|
- Ensure semantically equivalent repeated binding profiles and one inherited
|
||||||
|
default produce the same resolved pipeline digest. Ensure any changed
|
||||||
|
effective profile changes the digest.
|
||||||
|
- Do not modify file configuration or CLI parsing in this stage.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Add pipeline package tests for the complete precedence matrix:
|
||||||
|
runtime override; binding-specific exception; pipeline default; prompt
|
||||||
|
fallback; and deterministic bindings.
|
||||||
|
- Cover default and explicitly configured validator chains, all relevant stage
|
||||||
|
categories, `--only` lane selection, unused defaults, deterministic-profile
|
||||||
|
rejection, and semantic digest equivalence.
|
||||||
|
- Prefer table-driven package-level tests over assertions on private traversal
|
||||||
|
helpers.
|
||||||
|
- Run `go test ./internal/framework/pipeline` and `go test ./...`.
|
||||||
|
- Run `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- Programmatic pipelines resolve one canonical effective profile policy.
|
||||||
|
- Only LLM-backed resolved bindings can contain a profile.
|
||||||
|
- Runtime override, binding, pipeline, and prompt precedence is unambiguous and
|
||||||
|
digest-stable.
|
||||||
|
|
||||||
|
## Stage 9: Expose Pipeline Defaults Through Configuration And CLI
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Make the profile-default workflow available to operators while preserving
|
||||||
|
validation and override behavior.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Add optional `pipelines.<id>.llm_profile` support to the version 4 file
|
||||||
|
configuration model. Use presence-aware decoding so an explicitly set blank
|
||||||
|
value is rejected, while omission remains valid.
|
||||||
|
- Preserve the field through file application, configuration cloning,
|
||||||
|
effective configuration, and programmatic profile copies without aliasing or
|
||||||
|
trimming drift.
|
||||||
|
- Remove `applyLLMProfileOverride`. Pass the CLI override through the resolver's
|
||||||
|
runtime-override input so deterministic bindings are never populated.
|
||||||
|
- Update effective profile-ID collection to cover every selected LLM-backed
|
||||||
|
module stage and LLM-backed validator, including future LLM-backed input and
|
||||||
|
output modules. Do not inspect deterministic or unselected profiles.
|
||||||
|
- Ensure `run`, `config validate --pipeline`, resume/checkpoint identity, and
|
||||||
|
relevant dry preflight paths all use the same resolved effective profiles.
|
||||||
|
- Preserve `--llm-profile` as the highest-precedence non-empty run-wide
|
||||||
|
override and preserve binding-specific profiles as exceptions when no CLI
|
||||||
|
override is present.
|
||||||
|
- Do not increment the configuration version.
|
||||||
|
- Update current configuration and CLI contracts in `docs/config.md` and
|
||||||
|
`docs/cli.md` in the same stage. Link to operations for the deployment
|
||||||
|
workflow rather than duplicating it prematurely.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Add file-config tests for omission, trimming, explicit blank rejection,
|
||||||
|
unknown-key behavior, cloning, and round-trip application.
|
||||||
|
- Add effective-config and CLI contract tests for precedence, LLM-only
|
||||||
|
application, inherited-profile inspection failure before factory execution,
|
||||||
|
`--only`, and digest changes.
|
||||||
|
- Retain offline operation and do not require credentials for
|
||||||
|
`config validate --pipeline`.
|
||||||
|
- Run `go test ./internal/core/config ./internal/framework/pipeline
|
||||||
|
./internal/cli`.
|
||||||
|
- Run `go test ./...`, `go vet ./...`, and `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- Operators can select `dnd-extraction` once per pipeline.
|
||||||
|
- Configuration and CLI paths share the resolver's precedence policy.
|
||||||
|
- Unknown effective profiles fail preflight, while deterministic and unused
|
||||||
|
profiles do not cause spurious inspection.
|
||||||
|
|
||||||
|
## Stage 10: Complete Operator Documentation, Examples, And Decision Record
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Make the implemented workflow understandable, copyable, and maintainable
|
||||||
|
without duplicating canonical facts.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Create an ADR using the next sequential number for the durable decision to
|
||||||
|
use workload-oriented pipeline defaults with operator-overridable application
|
||||||
|
fallback profiles. Record context, decision, alternatives, and consequences;
|
||||||
|
do not turn the ADR into a field reference or implementation log.
|
||||||
|
- Complete `docs/config.md` as the canonical owner of profile-source fields,
|
||||||
|
`pipelines.<id>.llm_profile`, validation, and precedence.
|
||||||
|
- Complete `docs/operations.md` with an operator workflow that distinguishes
|
||||||
|
Notarius embedded prompts, Notarius fallback profiles, PromptKit built-ins,
|
||||||
|
and deployment filesystem profiles. Include production/development/local use
|
||||||
|
of the same `dnd-extraction` ID, credential handling, absolute-path guidance,
|
||||||
|
and the fact that current relative profile paths use the process working
|
||||||
|
directory rather than the configuration file's directory.
|
||||||
|
- Complete `docs/integrations/pkg-promptkit.md` with the v0.5.0 boundary,
|
||||||
|
prepared execution, inspection, fallback and ordinary source precedence,
|
||||||
|
optional provider controls, capacity adaptation, and compatibility policy.
|
||||||
|
- Update `docs/internal/configuration.md`, `docs/internal/pipeline.md`,
|
||||||
|
`docs/internal/cli.md`, `docs/internal/llm.md`, `docs/internal/modules.md`, and
|
||||||
|
`docs/internal/dnd.md` only for their owned implementation details. Link to
|
||||||
|
canonical configuration, operations, and upstream format contracts rather
|
||||||
|
than restating them.
|
||||||
|
- Keep exactly the existing two D&D configuration examples. Add
|
||||||
|
`llm_profile: dnd-extraction` to the minimal and complete pipelines and remove
|
||||||
|
the now-redundant model-named binding override from the complete example.
|
||||||
|
- Add one secret-free maintained operator profile at
|
||||||
|
`examples/profiles/dnd-extraction.yml`. It should be a complete valid profile
|
||||||
|
for the same logical ID and may mirror the embedded baseline; its purpose is
|
||||||
|
to demonstrate file ownership and format, not claim automatic environment
|
||||||
|
detection. Link it from the configuration and operations documentation.
|
||||||
|
- If the complete example selects the external profile file, use a path that
|
||||||
|
is valid for the documented repository-root invocation and explicitly note
|
||||||
|
the working-directory rule. Keep the minimal example dependent only on the
|
||||||
|
embedded fallback.
|
||||||
|
- Add or extend maintained-example validation so both configuration examples
|
||||||
|
and the profile YAML are checked without generation or credentials.
|
||||||
|
- Remove the now-implemented `Pipeline-Level LLM Profile Defaults` section from
|
||||||
|
`docs/roadmap/future.md`. Preserve the unrelated deterministic session and
|
||||||
|
concurrency items.
|
||||||
|
- Do not delete `promptkit.md` or this implementation plan during the feature
|
||||||
|
implementation; retire them only after post-implementation review.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Run maintained example/configuration tests and relevant CLI help/parser
|
||||||
|
tests.
|
||||||
|
- Run `go test ./...`.
|
||||||
|
- Run `rg -n 'gemini-2-flash' examples docs` and review every remaining match
|
||||||
|
for intentional model-policy or historical context.
|
||||||
|
- Run `rg -n 'v0\.3\.0|profileCheckPrompt|applyLLMProfileOverride' .` and resolve
|
||||||
|
stale production or current-documentation matches.
|
||||||
|
- Verify all new links and `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- Every current fact has one canonical documentation owner.
|
||||||
|
- Operators can distinguish and deploy all profile layers without reading Go
|
||||||
|
source.
|
||||||
|
- Both maintained configurations and the maintained external profile are valid,
|
||||||
|
secret-free, and tested offline.
|
||||||
|
- Implemented profile work no longer remains in `future.md`.
|
||||||
|
|
||||||
|
## Stage 11: Final Verification And Quality Review
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Verify the complete migration as one integrated change and correct only defects
|
||||||
|
or omissions found during that review.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Review the final diff against every acceptance criterion in `promptkit.md`.
|
||||||
|
- Confirm provider-specific PromptKit types remain inside the LLM integration
|
||||||
|
boundary and D&D policy remains inside the D&D module family.
|
||||||
|
- Confirm execution and inspection receive identical ordinary, fallback, and
|
||||||
|
backend configuration.
|
||||||
|
- Confirm no paths, profile YAML, endpoints, credentials, or prepared handle
|
||||||
|
state leak into fingerprints or ordinary diagnostics.
|
||||||
|
- Confirm all production module specs have explicit correct execution classes
|
||||||
|
and every resolved deterministic binding is profile-free.
|
||||||
|
- Confirm prompt default, pipeline default, binding override, and CLI override
|
||||||
|
behavior through representative assembled configurations.
|
||||||
|
- Review tests for redundancy and remove obsolete synthetic-prompt,
|
||||||
|
runtime-probe, exact-hash, or duplicated upstream-behavior tests superseded by
|
||||||
|
stronger contract tests.
|
||||||
|
- Perform an optional manual D&D quality comparison if credentials and an
|
||||||
|
evaluation transcript are deliberately supplied. Record no private input or
|
||||||
|
credential material, and do not make this comparison a completion gate.
|
||||||
|
|
||||||
|
### Validation Commands
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
git diff --check
|
gofmt -w <changed-go-files>
|
||||||
go test ./internal/modules/dnd/chunk/scenes
|
|
||||||
go test ./internal/modules/dnd/extract/...
|
|
||||||
go test ./internal/modules/dnd/normalize/npcs
|
|
||||||
go test ./internal/modules/dnd/register
|
|
||||||
go test ./internal/modules/dnd/...
|
|
||||||
go test ./internal/modules/integration/...
|
|
||||||
go test ./...
|
go test ./...
|
||||||
|
go test -race ./internal/framework/llm ./internal/core/config ./internal/framework/pipeline ./internal/cli
|
||||||
go vet ./...
|
go vet ./...
|
||||||
go build ./cmd/notarius
|
go build ./cmd/notarius
|
||||||
|
git diff --check
|
||||||
```
|
```
|
||||||
|
|
||||||
The focused commands should be used while implementing their respective
|
Also run focused stale-contract searches:
|
||||||
stages; the repository-wide commands are the final acceptance gate.
|
|
||||||
|
|
||||||
## Expected Compatibility Effects
|
```sh
|
||||||
|
rg -n 'gitea.maximumdirect.net/eric/promptkit v0\.3\.0|PromptKit v0\.3\.0' .
|
||||||
|
rg -n 'default_profile: gemini-2-flash|profileCheckPrompt|applyLLMProfileOverride' internal docs examples
|
||||||
|
```
|
||||||
|
|
||||||
- Reordered prompt content changes prompt fingerprints and may intentionally
|
Review any matches rather than deleting intentional historical references
|
||||||
cause one-time cold misses for affected development checkpoints.
|
blindly.
|
||||||
- Prompt IDs, prompt versions, schema identities, checkpoint format, durable
|
|
||||||
artifacts, and public behavior do not change.
|
### Completion Criteria
|
||||||
- Backend cache reuse remains provider-dependent, but Notarius supplies a
|
|
||||||
longer common extraction prefix that includes the transcript chunk.
|
- All automated checks pass offline and without real credentials.
|
||||||
|
- The implemented behavior matches `promptkit.md` with no known architecture,
|
||||||
|
provenance, checkpoint, profile-precedence, or documentation gap.
|
||||||
|
- Any optional live evaluation is clearly separate from correctness testing.
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
None. The message sequences, cache boundaries, test ownership, documentation
|
None. The roadmap decisions are sufficient to implement every stage without an
|
||||||
owners, and compatibility policy are specified above.
|
additional product or architecture choice.
|
||||||
|
|||||||
318
docs/roadmap/promptkit.md
Normal file
318
docs/roadmap/promptkit.md
Normal file
@@ -0,0 +1,318 @@
|
|||||||
|
# PromptKit v0.5 Integration And LLM Profile Policy
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This roadmap defines the target state for upgrading Notarius from PromptKit
|
||||||
|
v0.3.0 to v0.5.0 and adopting the upstream runtime and profile facilities that
|
||||||
|
directly improve Notarius. It also defines the application policy for stable,
|
||||||
|
domain-oriented LLM profile names, operator overrides, pipeline inheritance,
|
||||||
|
profile validation, provider defaults, checkpoint identity, and documentation.
|
||||||
|
|
||||||
|
The ordered work needed to reach this state belongs in
|
||||||
|
[the implementation plan](implementation.md). Current behavior remains defined
|
||||||
|
by the canonical documentation outside `docs/roadmap/` until the corresponding
|
||||||
|
work is implemented.
|
||||||
|
|
||||||
|
## Background
|
||||||
|
|
||||||
|
Notarius currently pins PromptKit v0.3.0. Its adapter prepares a request once
|
||||||
|
for debug material and then independently runs the original request, causing
|
||||||
|
PromptKit to prepare the same logical call a second time. The CLI validates an
|
||||||
|
explicit profile by preparing a synthetic prompt. PromptKit profile selection
|
||||||
|
can be repeated on individual module bindings or replaced for one invocation
|
||||||
|
with `--llm-profile`, but a configured pipeline cannot yet declare one inherited
|
||||||
|
profile policy.
|
||||||
|
|
||||||
|
PromptKit v0.4.0 and v0.5.0 add the upstream boundaries needed to improve these
|
||||||
|
areas:
|
||||||
|
|
||||||
|
- [v0.4.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.4.0.md)
|
||||||
|
adds opaque prepared executions, exact profile and prompt inspection, and a
|
||||||
|
typed backend-capacity error;
|
||||||
|
- [v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.5.0.md)
|
||||||
|
adds application fallback profile filesystems and stops sending unset
|
||||||
|
optional sampling controls as framework-selected provider values; and
|
||||||
|
- the [v0.5.0 format contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
|
||||||
|
defines the resulting profile-source and execution-setting precedence.
|
||||||
|
|
||||||
|
A source-compatibility test of the current Notarius repository against
|
||||||
|
PromptKit v0.5.0 completed successfully. The work is therefore primarily an
|
||||||
|
intentional runtime and configuration migration rather than a repair for a
|
||||||
|
breaking Go API change.
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
- Pin and document PromptKit v0.5.0 as Notarius's supported upstream contract.
|
||||||
|
- Execute the exact prepared request snapshot whose safe details are recorded
|
||||||
|
in Notarius debug material.
|
||||||
|
- Validate configured PromptKit profiles through the upstream inspection API
|
||||||
|
without synthetic prompts, provider calls, or credential-value access.
|
||||||
|
- Give Notarius an application-owned, operator-overridable
|
||||||
|
`dnd-extraction` profile fallback.
|
||||||
|
- Let a pipeline choose one default LLM profile without repeating that ID on
|
||||||
|
every LLM-backed binding.
|
||||||
|
- Apply profile inheritance and run-wide overrides only where the resolved
|
||||||
|
module or validator can use an LLM.
|
||||||
|
- Preserve accurate checkpoint invalidation, effective profile provenance,
|
||||||
|
redaction, cancellation, concurrency, and provider-neutral module contracts.
|
||||||
|
- Provide operators with one clear deployment pattern for production,
|
||||||
|
development, and local profile definitions.
|
||||||
|
|
||||||
|
## Target End State
|
||||||
|
|
||||||
|
### PromptKit Runtime Boundary
|
||||||
|
|
||||||
|
Notarius depends on PromptKit v0.5.0 and uses its public APIs rather than
|
||||||
|
reimplementing source or execution resolution.
|
||||||
|
|
||||||
|
For each structured completion, the adapter:
|
||||||
|
|
||||||
|
1. builds one PromptKit run request from the provider-neutral Notarius request;
|
||||||
|
2. calls `PrepareExecution` once;
|
||||||
|
3. immediately arranges an idempotent `Discard` for every unexecuted handle;
|
||||||
|
4. obtains credential-redacted `Details` for debug and response metadata; and
|
||||||
|
5. calls `RunPrepared` so generation uses that exact frozen snapshot.
|
||||||
|
|
||||||
|
The debug prompt and successful result therefore describe the same selected
|
||||||
|
profile, rendered messages, input bytes, session, output contract, and effective
|
||||||
|
settings even when a filesystem-backed source changes concurrently. PromptKit
|
||||||
|
handle types remain private to `internal/framework/llm`.
|
||||||
|
|
||||||
|
PromptKit admission failures continue to match Notarius's provider-neutral
|
||||||
|
`ErrLLMCapacityExceeded` contract. When PromptKit supplies a `CapacityError`,
|
||||||
|
the adapter obtains the normalized backend ID through `errors.As` and may add it
|
||||||
|
to safe application-owned diagnostics without parsing upstream error wording.
|
||||||
|
The backend ID does not become a provider-specific module contract.
|
||||||
|
|
||||||
|
### Optional Provider Controls
|
||||||
|
|
||||||
|
Notarius accepts PromptKit v0.5.0's new behavior for `temperature`,
|
||||||
|
`max_tokens`, and `top_p`: an unset setting is omitted from compatible provider
|
||||||
|
requests and the provider chooses its own default. Notarius does not restore
|
||||||
|
PromptKit's former implicit `top_p: 1` value globally.
|
||||||
|
|
||||||
|
An operator who requires a particular value specifies it in the selected
|
||||||
|
PromptKit profile. The application fallback described below intentionally
|
||||||
|
leaves these controls unset. A human-reviewed D&D extraction comparison should
|
||||||
|
be performed after the upgrade, but paid or nondeterministic model output is
|
||||||
|
not part of the default automated test suite.
|
||||||
|
|
||||||
|
### Profile Inspection
|
||||||
|
|
||||||
|
Pipeline-aware configuration validation uses `Engine.InspectProfile` for every
|
||||||
|
effective explicit profile ID. It verifies that the profile exists, parses and
|
||||||
|
validates, resolves its backend and target, and is compatible with the engine's
|
||||||
|
registered backends. It does not create a synthetic prompt, load prompt inputs,
|
||||||
|
contact a provider, or require credential values to exist in the validation
|
||||||
|
process environment.
|
||||||
|
|
||||||
|
Credential availability is execution-time state. PromptKit preparation still
|
||||||
|
enforces the selected profile's credential contract before generation. This
|
||||||
|
keeps `notarius config validate` useful in build and deployment validation
|
||||||
|
environments where secrets are deliberately absent.
|
||||||
|
|
||||||
|
PromptKit construction for inspection and execution uses one shared internal
|
||||||
|
profile-source and backend-option path. The CLI does not expose PromptKit public
|
||||||
|
types across the Notarius LLM boundary merely to perform inspection.
|
||||||
|
|
||||||
|
`InspectPrompt` is not adopted merely because it exists. It remains available
|
||||||
|
for a later, separately defined module-to-prompt interface preflight if a
|
||||||
|
concrete validation requirement justifies that additional contract.
|
||||||
|
|
||||||
|
### Application And Operator Profile Sources
|
||||||
|
|
||||||
|
Notarius embeds one ordinary PromptKit YAML profile with the stable ID
|
||||||
|
`dnd-extraction`. It is an application fallback registered through
|
||||||
|
`WithFallbackProfileFS`, is owned by the D&D module family, and initially
|
||||||
|
preserves the current effective D&D baseline:
|
||||||
|
|
||||||
|
- backend: PromptKit's built-in `openrouter` backend;
|
||||||
|
- model: `openai/gpt-5.6-luna`;
|
||||||
|
- reasoning effort: unset, allowing OpenAI's backend to apply its default of
|
||||||
|
`medium`;
|
||||||
|
- generation timeout: 240 seconds;
|
||||||
|
- service tier: `flex`; and
|
||||||
|
- no application-selected `temperature`, `max_tokens`, or `top_p`.
|
||||||
|
|
||||||
|
All maintained D&D LLM prompt definitions use `dnd-extraction` as their
|
||||||
|
`default_profile`. The ID communicates workload intent rather than a provider,
|
||||||
|
model, or environment. Changing the embedded fallback is an intentional
|
||||||
|
Notarius execution-policy change and participates in checkpoint identity.
|
||||||
|
|
||||||
|
Effective profile definitions resolve in PromptKit's order:
|
||||||
|
|
||||||
|
1. programmatic in-memory profiles used by tests or explicit consumers;
|
||||||
|
2. the operator source configured by `promptkit.profile_file` or
|
||||||
|
`promptkit.profile_dir`;
|
||||||
|
3. the Notarius application fallback source; and
|
||||||
|
4. PromptKit's embedded built-in catalog.
|
||||||
|
|
||||||
|
Only an absent ID falls through to the next source. A matching profile is a
|
||||||
|
complete definition: fields are not merged with a lower-precedence definition,
|
||||||
|
and a malformed matching operator profile fails rather than silently selecting
|
||||||
|
the application fallback.
|
||||||
|
|
||||||
|
Production, development, and local deployments should normally provide
|
||||||
|
different complete definitions for the same `dnd-extraction` ID. An operator
|
||||||
|
source is optional because the application fallback keeps the maintained D&D
|
||||||
|
workflow usable, but a deployment that needs an intentional model or backend
|
||||||
|
policy should configure its own definition.
|
||||||
|
|
||||||
|
### Domain Ownership And Asset Assembly
|
||||||
|
|
||||||
|
The D&D fallback profile remains under `internal/modules/dnd` and is registered
|
||||||
|
by the D&D registrar, consistent with ADR-0004. Generic LLM plumbing knows how
|
||||||
|
to collect and flatten application fallback profile filesystems but contains no
|
||||||
|
D&D model or policy knowledge.
|
||||||
|
|
||||||
|
The shared asset registry detects invalid roots, unreadable sources, and
|
||||||
|
duplicate flattened paths. PromptKit remains responsible for strict profile
|
||||||
|
YAML parsing, duplicate profile-ID detection, source precedence, and effective
|
||||||
|
target resolution. The same assembled fallback source is supplied to runtime
|
||||||
|
execution and CLI profile inspection.
|
||||||
|
|
||||||
|
### Explicit Module Execution Metadata
|
||||||
|
|
||||||
|
Every registered input, chunk, extract, merge, normalize, and output module
|
||||||
|
declares one required execution class: `deterministic` or `llm_backed`.
|
||||||
|
Validator registrations continue to declare the same distinction through their
|
||||||
|
validator specifications.
|
||||||
|
|
||||||
|
The registered specification is authoritative for configuration resolution.
|
||||||
|
Current production classifications are:
|
||||||
|
|
||||||
|
- the D&D scene chunker, all D&D extractors, and the D&D NPC normalizer are
|
||||||
|
LLM-backed;
|
||||||
|
- the Seriatim input adapter, generic chunker, all current mergers, all other
|
||||||
|
current normalizers, and the JSON output encoder are deterministic; and
|
||||||
|
- current validators retain their declared classifications.
|
||||||
|
|
||||||
|
Missing or unsupported execution metadata is a registration error. Explicitly
|
||||||
|
assigning `llm_profile` to a deterministic module or validator is a pipeline
|
||||||
|
resolution error. The framework does not infer execution class by inspecting
|
||||||
|
domain package names or concrete implementation types at runtime.
|
||||||
|
|
||||||
|
The module specification replaces the chunk runner's special runtime
|
||||||
|
execution-class probe. Effective resolved bindings already express the result:
|
||||||
|
only LLM-backed bindings may retain a non-empty profile.
|
||||||
|
|
||||||
|
### Pipeline-Level Profile Default
|
||||||
|
|
||||||
|
Configuration version 4 gains one optional non-empty pipeline field:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
pipelines:
|
||||||
|
dnd-session:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
|
```
|
||||||
|
|
||||||
|
No configuration-version increment is required because the field is additive
|
||||||
|
and existing files remain valid. An explicitly present blank value is invalid.
|
||||||
|
|
||||||
|
For every selected LLM-backed module and validator, the effective profile uses
|
||||||
|
this precedence:
|
||||||
|
|
||||||
|
1. non-empty run-wide `--llm-profile` override;
|
||||||
|
2. binding-specific `llm_profile`;
|
||||||
|
3. pipeline-level `llm_profile`; and
|
||||||
|
4. the prompt definition's `default_profile`, represented by an empty effective
|
||||||
|
Notarius binding profile.
|
||||||
|
|
||||||
|
The run-wide override and inherited pipeline default never attach to a
|
||||||
|
deterministic binding. Binding-specific exceptions remain available when one
|
||||||
|
operation needs a different cost, latency, quality, backend, or reasoning
|
||||||
|
policy.
|
||||||
|
|
||||||
|
Inheritance is resolved after module and validator selection, including
|
||||||
|
`--only` lane filtering, but before effective-pipeline validation, digest
|
||||||
|
construction, explicit-profile inspection, checkpoint construction,
|
||||||
|
preparation, execution, or provenance capture. Only profiles used by selected
|
||||||
|
LLM-backed bindings are inspected. An unused pipeline default in a pipeline
|
||||||
|
with no selected LLM-backed work does not require an otherwise unused profile
|
||||||
|
to exist.
|
||||||
|
|
||||||
|
The resolved pipeline contains effective binding profiles rather than a second
|
||||||
|
runtime inheritance mechanism. Two pipelines that differ only by spelling the
|
||||||
|
same effective policy once as a pipeline default and once on every LLM-backed
|
||||||
|
binding have the same semantic resolved digest. Changing an effective profile
|
||||||
|
changes the digest and applicable checkpoint identity.
|
||||||
|
|
||||||
|
### Provenance And Checkpoints
|
||||||
|
|
||||||
|
The PromptKit profile-source checkpoint fingerprint covers:
|
||||||
|
|
||||||
|
- the PromptKit v0.5.0 built-in profile catalog identity;
|
||||||
|
- exact application fallback profile asset content; and
|
||||||
|
- exact configured operator profile YAML content, when present.
|
||||||
|
|
||||||
|
The existing local-backend target fingerprint remains separate and continues
|
||||||
|
to exclude scheduling-only concurrency limits. Fingerprints contain hashes and
|
||||||
|
stable markers, not profile contents, filesystem paths, endpoints, credentials,
|
||||||
|
or other secrets.
|
||||||
|
|
||||||
|
Changing the PromptKit version, application fallback, operator profile, or
|
||||||
|
effective pipeline profile makes incompatible LLM checkpoints ineligible for
|
||||||
|
reuse. The dependency upgrade is expected to invalidate checkpoints produced
|
||||||
|
under v0.3.0.
|
||||||
|
|
||||||
|
Successful run manifests continue to record only profiles actually selected by
|
||||||
|
PromptKit, including their effective model, backend, and reasoning metadata.
|
||||||
|
Debug output reports the same effective execution snapshot used for generation.
|
||||||
|
|
||||||
|
### Operator Documentation And Examples
|
||||||
|
|
||||||
|
Canonical documentation clearly distinguishes:
|
||||||
|
|
||||||
|
- Notarius prompt and schema assets embedded in the application;
|
||||||
|
- Notarius application fallback profiles embedded in the application;
|
||||||
|
- PromptKit's own embedded built-in profiles; and
|
||||||
|
- operator profile files on the deployment filesystem.
|
||||||
|
|
||||||
|
The configuration reference owns the pipeline field, profile-source fields,
|
||||||
|
validation rules, and precedence. Operations owns deployment layout, working
|
||||||
|
directory behavior, credentials, and environment-specific profile management.
|
||||||
|
The PromptKit integration document owns the pinned upstream contract and
|
||||||
|
source-precedence boundary. Internal documents describe asset registration,
|
||||||
|
resolution, inspection, prepared execution, fingerprinting, and tests without
|
||||||
|
duplicating user-facing field definitions.
|
||||||
|
|
||||||
|
The maintained examples continue to include only the minimal and complete D&D
|
||||||
|
configurations. They use the stable `dnd-extraction` policy, and one maintained
|
||||||
|
PromptKit profile file under `examples/` demonstrates an operator override.
|
||||||
|
Examples remain secret-free and are validated without live provider calls.
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
- Implementing the separate deterministic prompt-session identity roadmap
|
||||||
|
item.
|
||||||
|
- Changing the default `concurrency.total_llm` value; PromptKit's retained
|
||||||
|
OpenRouter capacity of 16 remains relevant to that separate item.
|
||||||
|
- Adding model evaluation as a deterministic or CI correctness gate.
|
||||||
|
- Automatically selecting production, development, or local environments.
|
||||||
|
Deployment configuration chooses the operator profile source.
|
||||||
|
- Profile inheritance, partial profile merging, or cross-profile aliases.
|
||||||
|
- Exposing PromptKit types to modules, validators, durable output contracts, or
|
||||||
|
public configuration structures.
|
||||||
|
- Adopting `InspectPrompt` without a separately justified prompt-interface
|
||||||
|
validation contract.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- Notarius builds and its offline test suite passes with PromptKit v0.5.0.
|
||||||
|
- Every structured completion executes the exact snapshot used for safe debug
|
||||||
|
prompt details.
|
||||||
|
- Profile preflight uses profile inspection and no synthetic prompt.
|
||||||
|
- The embedded `dnd-extraction` fallback resolves without an operator source,
|
||||||
|
and a matching valid operator profile replaces it completely.
|
||||||
|
- Every production module has explicit, correct execution metadata.
|
||||||
|
- Pipeline, binding, CLI, and prompt-default precedence behaves as defined for
|
||||||
|
modules and validators, while deterministic bindings remain profile-free.
|
||||||
|
- Effective profiles participate in pipeline digests, profile inspection,
|
||||||
|
checkpoint identity, debug records, and run provenance at the appropriate
|
||||||
|
boundaries.
|
||||||
|
- The dependency and application fallback changes invalidate incompatible old
|
||||||
|
checkpoints without exposing profile or credential content.
|
||||||
|
- Canonical documentation and maintained examples accurately describe and
|
||||||
|
exercise the implemented operator workflow.
|
||||||
|
- Default tests remain deterministic, offline, credential-free, and focused on
|
||||||
|
Notarius-owned behavior rather than duplicating PromptKit's upstream suite.
|
||||||
@@ -1,4 +1,6 @@
|
|||||||
version: 3
|
version: 4
|
||||||
|
promptkit:
|
||||||
|
profile_file: ./examples/profiles/dnd-extraction.yml
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 2
|
total_llm: 2
|
||||||
stage_workers:
|
stage_workers:
|
||||||
@@ -16,6 +18,7 @@ debug:
|
|||||||
directory: ./notarius-debug
|
directory: ./notarius-debug
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input: seriatim
|
||||||
# Stable campaign context is shared by every module that accepts these slots.
|
# Stable campaign context is shared by every module that accepts these slots.
|
||||||
references:
|
references:
|
||||||
@@ -54,7 +57,6 @@ pipelines:
|
|||||||
merge: appendorder
|
merge: appendorder
|
||||||
normalize:
|
normalize:
|
||||||
module: dnd/npcs
|
module: dnd/npcs
|
||||||
llm_profile: gemini-2-flash
|
|
||||||
retries: 2
|
retries: 2
|
||||||
scene-descriptions:
|
scene-descriptions:
|
||||||
extract:
|
extract:
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
version: 3
|
version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input: seriatim
|
||||||
artifacts:
|
artifacts:
|
||||||
spells:
|
spells:
|
||||||
|
|||||||
5
examples/profiles/dnd-extraction.yml
Normal file
5
examples/profiles/dnd-extraction.yml
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
id: dnd-extraction
|
||||||
|
backend: openrouter
|
||||||
|
model: openai/gpt-5.6-luna
|
||||||
|
timeout_seconds: 240
|
||||||
|
service_tier: flex
|
||||||
2
go.mod
2
go.mod
@@ -3,7 +3,7 @@ module gitea.maximumdirect.net/eric/notarius
|
|||||||
go 1.25.5
|
go 1.25.5
|
||||||
|
|
||||||
require (
|
require (
|
||||||
gitea.maximumdirect.net/eric/scriptorium v0.11.1
|
gitea.maximumdirect.net/eric/promptkit v0.5.0
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
||||||
gopkg.in/yaml.v3 v3.0.1
|
gopkg.in/yaml.v3 v3.0.1
|
||||||
)
|
)
|
||||||
|
|||||||
8
go.sum
8
go.sum
@@ -1,13 +1,9 @@
|
|||||||
gitea.maximumdirect.net/eric/scriptorium v0.11.0 h1:rjvbt9FTaWHxYlHq7QlUzmMVUt3QdbTmeCkmH81N//o=
|
gitea.maximumdirect.net/eric/promptkit v0.5.0 h1:jnpazLyyNhWrB2xzwwtUkNUfktkTdkENTwuSPnKiYrc=
|
||||||
gitea.maximumdirect.net/eric/scriptorium v0.11.0/go.mod h1:FQ5lEuNxmrQyNgIomkpZdxvfTC0jWjbXYuq3tbJWF64=
|
gitea.maximumdirect.net/eric/promptkit v0.5.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||||
gitea.maximumdirect.net/eric/scriptorium v0.11.1 h1:zBKtB3+fP8FcHGI8DJD99CiTL6crAGitBhWtE+xYJHc=
|
|
||||||
gitea.maximumdirect.net/eric/scriptorium v0.11.1/go.mod h1:FQ5lEuNxmrQyNgIomkpZdxvfTC0jWjbXYuq3tbJWF64=
|
|
||||||
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
||||||
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
|
||||||
golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
|
|
||||||
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
|
||||||
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
||||||
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||||
|
|||||||
@@ -213,11 +213,12 @@ func assembledSpellPipeline(t *testing.T, options assembledSpellPipelineOptions)
|
|||||||
components := productionTestComponents(t)
|
components := productionTestComponents(t)
|
||||||
extractor := &assembledSpellExtractor{unknownSpell: options.unknownSpell}
|
extractor := &assembledSpellExtractor{unknownSpell: options.unknownSpell}
|
||||||
if err := pipeline.RegisterExtractor[dnd.SpellList](components.registries.Extractors, pipeline.ModuleSpec{
|
if err := pipeline.RegisterExtractor[dnd.SpellList](components.registries.Extractors, pipeline.ModuleSpec{
|
||||||
Key: assembledSpellExtractorKey,
|
Key: assembledSpellExtractorKey,
|
||||||
Stage: pipeline.StageExtract,
|
Stage: pipeline.StageExtract,
|
||||||
Requires: []string{"chunks", "source.transcript"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"dnd.spell_casts"},
|
Requires: []string{"chunks", "source.transcript"},
|
||||||
ArtifactKind: dnd.SpellListKind,
|
Provides: []string{"dnd.spell_casts"},
|
||||||
|
ArtifactKind: dnd.SpellListKind,
|
||||||
}, func() (contracts.Extractor[dnd.SpellList], error) {
|
}, func() (contracts.Extractor[dnd.SpellList], error) {
|
||||||
return extractor, nil
|
return extractor, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
|
|||||||
@@ -143,24 +143,13 @@ func isEmptyRegistries(registries pipeline.Registries) bool {
|
|||||||
registries.Outputs == nil
|
registries.Outputs == nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
|
||||||
if err := ctx.Err(); err != nil {
|
|
||||||
return nil, nil, err
|
|
||||||
}
|
|
||||||
assets, err := productionPromptAssets()
|
|
||||||
if err != nil {
|
|
||||||
return nil, nil, err
|
|
||||||
}
|
|
||||||
return buildProductionLLMClient(ctx, cfg, profileID, assets)
|
|
||||||
}
|
|
||||||
|
|
||||||
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
|
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
|
||||||
return func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
return func(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
return buildProductionLLMClient(ctx, cfg, profileID, assets)
|
return buildProductionLLMClient(ctx, cfg, profileID, overrides, assets)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
if err := ctx.Err(); err != nil {
|
if err := ctx.Err(); err != nil {
|
||||||
return nil, nil, err
|
return nil, nil, err
|
||||||
}
|
}
|
||||||
@@ -168,14 +157,16 @@ func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID
|
|||||||
return nil, nil, fmt.Errorf("production asset registry must not be nil")
|
return nil, nil, fmt.Errorf("production asset registry must not be nil")
|
||||||
}
|
}
|
||||||
recorder := llm.NewLLMProfileRecorder()
|
recorder := llm.NewLLMProfileRecorder()
|
||||||
client, err := llm.NewScriptoriumClient(llm.ScriptoriumClientConfig{
|
client, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{
|
||||||
ProfileDir: cfg.Scriptorium.ProfileDir,
|
ProfileDir: cfg.PromptKit.ProfileDir,
|
||||||
ProfileFile: cfg.Scriptorium.ProfileFile,
|
ProfileFile: cfg.PromptKit.ProfileFile,
|
||||||
Assets: assets,
|
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
|
||||||
Recorder: recorder,
|
Assets: assets,
|
||||||
|
Recorder: recorder,
|
||||||
|
ReasoningEffort: overrides.ReasoningEffort,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, nil, fmt.Errorf("create Scriptorium-backed LLM client: %w", err)
|
return nil, nil, fmt.Errorf("create PromptKit-backed LLM client: %w", err)
|
||||||
}
|
}
|
||||||
scheduler, err := llm.NewScheduler(cfg.Concurrency.TotalLLM)
|
scheduler, err := llm.NewScheduler(cfg.Concurrency.TotalLLM)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -154,6 +154,23 @@ func TestConfigValidateResolvesPipelineAndChecksSelection(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestConfigValidatePipelineDefaultProfileIsOffline(t *testing.T) {
|
||||||
|
configPath := writeCommandConfigContent(t, `version: 4
|
||||||
|
pipelines:
|
||||||
|
demo:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
|
input: seriatim
|
||||||
|
artifacts:
|
||||||
|
spells:
|
||||||
|
extract: dnd/spells
|
||||||
|
`)
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
code := RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "demo"}, &stdout, &stderr, Options{})
|
||||||
|
if code != 0 || !strings.Contains(stdout.String(), "valid for pipeline \"demo\"") || stderr.Len() != 0 {
|
||||||
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestPipelinesListSortsNormalizedIDsInTextAndJSON(t *testing.T) {
|
func TestPipelinesListSortsNormalizedIDsInTextAndJSON(t *testing.T) {
|
||||||
configPath := writeCommandConfig(t, " zeta ", "alpha")
|
configPath := writeCommandConfig(t, " zeta ", "alpha")
|
||||||
options := commandContractOptions(t)
|
options := commandContractOptions(t)
|
||||||
@@ -214,13 +231,13 @@ func commandContractOptionsWithLookup(t *testing.T, lookup func(string) (string,
|
|||||||
|
|
||||||
func writeCommandConfig(t *testing.T, firstID, secondID string) string {
|
func writeCommandConfig(t *testing.T, firstID, secondID string) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
content := fmt.Sprintf("version: 3\npipelines:\n %q:\n input: seriatim\n %q:\n input: seriatim\n", firstID, secondID)
|
content := fmt.Sprintf("version: 4\npipelines:\n %q:\n input: seriatim\n %q:\n input: seriatim\n", firstID, secondID)
|
||||||
return writeCommandConfigContent(t, content)
|
return writeCommandConfigContent(t, content)
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeResolvableCommandConfig(t *testing.T) string {
|
func writeResolvableCommandConfig(t *testing.T) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
return writeCommandConfigContent(t, `version: 3
|
return writeCommandConfigContent(t, `version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
demo:
|
demo:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
|
|||||||
@@ -6,8 +6,10 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"reflect"
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
@@ -28,7 +30,7 @@ func TestProductionSceneDescriptionWorkflow(t *testing.T) {
|
|||||||
Output: pipeline.Binding("json"),
|
Output: pipeline.Binding("json"),
|
||||||
Artifacts: map[string]pipeline.ArtifactLaneProfile{
|
Artifacts: map[string]pipeline.ArtifactLaneProfile{
|
||||||
"scene-descriptions": {
|
"scene-descriptions": {
|
||||||
Extract: pipeline.Binding(sceneextract.Key),
|
Extract: pipeline.ModuleBinding{Module: sceneextract.Key, LLMProfile: "scene-description-profile"},
|
||||||
Normalize: pipeline.Binding(scenenormalize.Key),
|
Normalize: pipeline.Binding(scenenormalize.Key),
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
@@ -45,7 +47,8 @@ func TestProductionSceneDescriptionWorkflow(t *testing.T) {
|
|||||||
t.Fatalf("resolved references = %#v / %#v, want no generated or required references", lane.ExtractReferences, lane.NormalizeReferences)
|
t.Fatalf("resolved references = %#v / %#v, want no generated or required references", lane.ExtractReferences, lane.NormalizeReferences)
|
||||||
}
|
}
|
||||||
|
|
||||||
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, components.registries, pipeline.ModuleDependencies{LLM: sceneDescriptionLLM{}})
|
llmClient := &sceneDescriptionLLM{}
|
||||||
|
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, components.registries, pipeline.ModuleDependencies{LLM: llmClient})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Prepare() error = %v", err)
|
t.Fatalf("Prepare() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -60,6 +63,14 @@ func TestProductionSceneDescriptionWorkflow(t *testing.T) {
|
|||||||
if output.Manifest.ValidationStatus != "approved" || len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 1 {
|
if output.Manifest.ValidationStatus != "approved" || len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 1 {
|
||||||
t.Fatalf("run output = %#v, want one approved normalized artifact", output)
|
t.Fatalf("run output = %#v, want one approved normalized artifact", output)
|
||||||
}
|
}
|
||||||
|
wantProfiles := []artifacts.LLMProfileManifest{{
|
||||||
|
ID: "scene-description-profile",
|
||||||
|
Provider: "promptkit",
|
||||||
|
Model: "deterministic",
|
||||||
|
}}
|
||||||
|
if !reflect.DeepEqual(output.Manifest.LLMProfiles, wantProfiles) {
|
||||||
|
t.Fatalf("manifest LLM profiles = %#v, want %#v", output.Manifest.LLMProfiles, wantProfiles)
|
||||||
|
}
|
||||||
normalizedOutput := output.NormalizeOutputs[0]
|
normalizedOutput := output.NormalizeOutputs[0]
|
||||||
if normalizedOutput.NormalizerKey != scenenormalize.Key || normalizedOutput.Artifact.Kind != dnd.SceneDescriptionListKind || normalizedOutput.Artifact.Schema.ID != scenecodec.SchemaID || normalizedOutput.Artifact.Schema.Name != scenecodec.SchemaName || normalizedOutput.Artifact.Schema.Version != scenecodec.SchemaVersion {
|
if normalizedOutput.NormalizerKey != scenenormalize.Key || normalizedOutput.Artifact.Kind != dnd.SceneDescriptionListKind || normalizedOutput.Artifact.Schema.ID != scenecodec.SchemaID || normalizedOutput.Artifact.Schema.Name != scenecodec.SchemaName || normalizedOutput.Artifact.Schema.Version != scenecodec.SchemaVersion {
|
||||||
t.Fatalf("normalized output = %#v, want registered durable scene-description schema", normalizedOutput)
|
t.Fatalf("normalized output = %#v, want registered durable scene-description schema", normalizedOutput)
|
||||||
@@ -85,9 +96,12 @@ func TestProductionSceneDescriptionWorkflow(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
type sceneDescriptionLLM struct{}
|
type sceneDescriptionLLM struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
profile *artifacts.LLMProfileManifest
|
||||||
|
}
|
||||||
|
|
||||||
func (sceneDescriptionLLM) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
func (client *sceneDescriptionLLM) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
if err := ctx.Err(); err != nil {
|
if err := ctx.Err(); err != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, err
|
return contracts.StructuredCompletionResponse{}, err
|
||||||
}
|
}
|
||||||
@@ -107,5 +121,27 @@ func (sceneDescriptionLLM) CompleteStructured(ctx context.Context, req contracts
|
|||||||
if err := json.Unmarshal([]byte(content), out); err != nil {
|
if err := json.Unmarshal([]byte(content), out); err != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate structured response: %w", err)
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate structured response: %w", err)
|
||||||
}
|
}
|
||||||
return contracts.StructuredCompletionResponse{Content: []byte(content), Provider: "test", Model: "deterministic", ProfileID: req.ProfileID}, nil
|
profile := artifacts.LLMProfileManifest{
|
||||||
|
ID: req.ProfileID,
|
||||||
|
Provider: "promptkit",
|
||||||
|
Model: "deterministic",
|
||||||
|
}
|
||||||
|
client.mu.Lock()
|
||||||
|
client.profile = &profile
|
||||||
|
client.mu.Unlock()
|
||||||
|
return contracts.StructuredCompletionResponse{
|
||||||
|
Content: []byte(content),
|
||||||
|
Provider: profile.Provider,
|
||||||
|
Model: profile.Model,
|
||||||
|
ProfileID: profile.ID,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (client *sceneDescriptionLLM) LLMProfileManifests() []artifacts.LLMProfileManifest {
|
||||||
|
client.mu.Lock()
|
||||||
|
defer client.mu.Unlock()
|
||||||
|
if client.profile == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return []artifacts.LLMProfileManifest{*client.profile}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -94,6 +94,37 @@ func TestMaintainedConfigurationExampleSet(t *testing.T) {
|
|||||||
if got := strings.Join(names, ","); got != "dnd-complete.config.yml,dnd-minimal.config.yml" {
|
if got := strings.Join(names, ","); got != "dnd-complete.config.yml,dnd-minimal.config.yml" {
|
||||||
t.Fatalf("maintained configuration examples = %q, want only the minimal and complete D&D examples", got)
|
t.Fatalf("maintained configuration examples = %q, want only the minimal and complete D&D examples", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
profileEntries, err := os.ReadDir(repositoryPath("examples", "profiles"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
names = names[:0]
|
||||||
|
for _, entry := range profileEntries {
|
||||||
|
if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".yml") {
|
||||||
|
names = append(names, entry.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
if got := strings.Join(names, ","); got != "dnd-extraction.yml" {
|
||||||
|
t.Fatalf("maintained operator profiles = %q, want dnd-extraction.yml", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMaintainedExamplesValidateEffectiveProfilesOffline(t *testing.T) {
|
||||||
|
t.Chdir(repositoryPath())
|
||||||
|
t.Setenv("OPENROUTER_API_KEY", "")
|
||||||
|
for _, example := range maintainedExampleFiles(t) {
|
||||||
|
t.Run(example.name, func(t *testing.T) {
|
||||||
|
var stdout, stderr strings.Builder
|
||||||
|
code := RunWithOptions([]string{
|
||||||
|
"config", "validate", "--config", example.path, "--pipeline", "dnd-session",
|
||||||
|
}, &stdout, &stderr, Options{})
|
||||||
|
if code != 0 || stderr.Len() != 0 || !strings.Contains(stdout.String(), `valid for pipeline "dnd-session"`) {
|
||||||
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func exampleStepLaneIDs(resolved pipeline.ResolvedPipeline) []string {
|
func exampleStepLaneIDs(resolved pipeline.ResolvedPipeline) []string {
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ func TestOversizedNPCRegistryFailsBeforeRuntimeAndCheckpointConstruction(t *test
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
checkpointRoot := filepath.Join(t.TempDir(), "checkpoints")
|
checkpointRoot := filepath.Join(t.TempDir(), "checkpoints")
|
||||||
content := fmt.Sprintf(`version: 3
|
content := fmt.Sprintf(`version: 4
|
||||||
cache:
|
cache:
|
||||||
chunk_plans:
|
chunk_plans:
|
||||||
mode: bypass
|
mode: bypass
|
||||||
@@ -51,7 +51,7 @@ pipelines:
|
|||||||
options := Options{
|
options := Options{
|
||||||
Catalog: catalogFromRegistries(components.registries),
|
Catalog: catalogFromRegistries(components.registries),
|
||||||
Registries: components.registries,
|
Registries: components.registries,
|
||||||
LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
llmConstructed = true
|
llmConstructed = true
|
||||||
return nil, nil, errors.New("LLM client must not be constructed")
|
return nil, nil, errors.New("LLM client must not be constructed")
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io/fs"
|
"io/fs"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"reflect"
|
"reflect"
|
||||||
@@ -13,13 +15,16 @@ import (
|
|||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
|
"testing/fstest"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkmap"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkmap"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
|
||||||
@@ -33,6 +38,7 @@ import (
|
|||||||
itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents"
|
itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents"
|
||||||
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
|
func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
|
||||||
@@ -160,6 +166,35 @@ func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
|
|||||||
assertProductionContains(t, "production prompt assets", assetNames, requiredAssets)
|
assertProductionContains(t, "production prompt assets", assetNames, requiredAssets)
|
||||||
|
|
||||||
catalog := catalogFromRegistries(registries)
|
catalog := catalogFromRegistries(registries)
|
||||||
|
for _, test := range []struct {
|
||||||
|
stage pipeline.ModuleStage
|
||||||
|
key string
|
||||||
|
want contracts.ExecutionClass
|
||||||
|
}{
|
||||||
|
{stage: pipeline.StageInput, key: "seriatim", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageChunk, key: "generic", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageChunk, key: "dnd/scenes", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/spells", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/npcs", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/combat-turns", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/item-events", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/npc-interactions", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/scene-descriptions", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageMerge, key: "appendorder", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "noop", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/spells", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/npcs", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/combat-turns", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/item-events", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/npc-interactions", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/scene-descriptions", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageOutput, key: "json", want: contracts.ExecutionClassDeterministic},
|
||||||
|
} {
|
||||||
|
got, ok := catalog.ExecutionClass(test.stage, test.key)
|
||||||
|
if !ok || got != test.want {
|
||||||
|
t.Fatalf("production execution class for %s/%s = %q, %t; want %q, true", test.stage, test.key, got, ok, test.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
converted := registriesFromCatalog(catalog)
|
converted := registriesFromCatalog(catalog)
|
||||||
if converted.ArtifactCodecs != registries.ArtifactCodecs || converted.ArtifactEvidence != registries.ArtifactEvidence || converted.ValidatorChains != registries.ValidatorChains {
|
if converted.ArtifactCodecs != registries.ArtifactCodecs || converted.ArtifactEvidence != registries.ArtifactEvidence || converted.ValidatorChains != registries.ValidatorChains {
|
||||||
t.Fatal("catalog/registry conversion did not preserve artifact and validator registries")
|
t.Fatal("catalog/registry conversion did not preserve artifact and validator registries")
|
||||||
@@ -195,6 +230,76 @@ func TestDefaultCLICompositionValidatesRepresentativeConfiguration(t *testing.T)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestProductionAssetsResolveDNDExtractionProfile(t *testing.T) {
|
||||||
|
components := productionTestComponents(t)
|
||||||
|
newEngine := func(profileFile string) (*promptkit.Engine, error) {
|
||||||
|
t.Helper()
|
||||||
|
options, err := components.assets.PromptKitOptions()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if profileFile != "" {
|
||||||
|
options = append(options, promptkit.WithProfileFile(profileFile))
|
||||||
|
}
|
||||||
|
return promptkit.NewEngine(promptkit.Config{}, options...)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Run("fallback", func(t *testing.T) {
|
||||||
|
engine, err := newEngine("")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
inspection, err := engine.InspectProfile(context.Background(), "dnd-extraction")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("InspectProfile() error = %v, want fallback profile", err)
|
||||||
|
}
|
||||||
|
params := inspection.EffectiveModelParams
|
||||||
|
if params.BackendID != "openrouter" || params.Model != "openai/gpt-5.6-luna" || params.TimeoutSeconds != 240 || params.ServiceTier != "flex" {
|
||||||
|
t.Fatalf("fallback profile parameters = %#v", params)
|
||||||
|
}
|
||||||
|
if params.ReasoningEffort != "" || params.Temperature != 0 || params.MaxTokens != 0 || params.TopP != 0 {
|
||||||
|
t.Fatalf("fallback profile selected optional provider controls: %#v", params)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("valid operator profile wins", func(t *testing.T) {
|
||||||
|
profilePath := filepath.Join(t.TempDir(), "profiles.yaml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte(`id: dnd-extraction
|
||||||
|
endpoint: http://operator.example.test/v1
|
||||||
|
model: operator-model
|
||||||
|
timeout_seconds: 75
|
||||||
|
`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
engine, err := newEngine(profilePath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
inspection, err := engine.InspectProfile(context.Background(), "dnd-extraction")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("InspectProfile() error = %v, want operator profile", err)
|
||||||
|
}
|
||||||
|
params := inspection.EffectiveModelParams
|
||||||
|
if params.BackendID != "" || params.Endpoint != "http://operator.example.test/v1" || params.Model != "operator-model" || params.TimeoutSeconds != 75 || params.ServiceTier != "" {
|
||||||
|
t.Fatalf("operator profile parameters = %#v, want complete replacement", params)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("invalid operator profile does not fall through", func(t *testing.T) {
|
||||||
|
profilePath := filepath.Join(t.TempDir(), "profiles.yaml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte("id: dnd-extraction\nendpoint: http://operator.example.test/v1\nmodel: operator-model\nunknown: value\n"), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
engine, err := newEngine(profilePath)
|
||||||
|
if err == nil {
|
||||||
|
_, err = engine.InspectProfile(context.Background(), "dnd-extraction")
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("operator profile error = nil, want failure instead of fallback")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) {
|
func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) {
|
||||||
components := productionTestComponents(t)
|
components := productionTestComponents(t)
|
||||||
cfg := config.Default()
|
cfg := config.Default()
|
||||||
@@ -321,7 +426,7 @@ func TestProductionSpellNormalizerRejectsInvalidCatalogReferencesBeforeExecution
|
|||||||
options := Options{
|
options := Options{
|
||||||
Catalog: catalogFromRegistries(components.registries),
|
Catalog: catalogFromRegistries(components.registries),
|
||||||
Registries: components.registries,
|
Registries: components.registries,
|
||||||
LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
llmConstructed = true
|
llmConstructed = true
|
||||||
return nil, nil, errors.New("LLM client must not be constructed")
|
return nil, nil, errors.New("LLM client must not be constructed")
|
||||||
},
|
},
|
||||||
@@ -369,31 +474,150 @@ func setNormalizeSpellCatalogSource(t *testing.T, resolved *pipeline.ResolvedPip
|
|||||||
resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings = bindings
|
resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings = bindings
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) {
|
func TestProductionLLMClientFactoryBuildsOfflineRuntime(t *testing.T) {
|
||||||
components := productionTestComponents(t)
|
components := productionTestComponents(t)
|
||||||
factories := []struct {
|
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
|
||||||
name string
|
if err != nil {
|
||||||
factory LLMClientFactory
|
t.Fatalf("build production LLM runtime: %v", err)
|
||||||
}{
|
|
||||||
{name: "default production assets", factory: productionLLMClientFactory},
|
|
||||||
{name: "provided production assets", factory: productionLLMClientFactoryWithAssets(components.assets)},
|
|
||||||
}
|
}
|
||||||
for _, tt := range factories {
|
if client == nil {
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Fatal("production LLM runtime returned a nil client")
|
||||||
client, manifests, err := tt.factory(context.Background(), config.Default(), "test-profile")
|
}
|
||||||
if err != nil {
|
if len(manifests) != 0 {
|
||||||
t.Fatalf("build production LLM runtime: %v", err)
|
t.Fatalf("eager profile manifests = %#v, want none", manifests)
|
||||||
}
|
}
|
||||||
if client == nil {
|
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
|
||||||
t.Fatal("production LLM runtime returned a nil client")
|
if !ok {
|
||||||
}
|
t.Fatalf("production LLM client %T does not provide one profile-source checkpoint fingerprint", client)
|
||||||
if len(manifests) != 0 {
|
}
|
||||||
t.Fatalf("eager profile manifests = %#v, want none", manifests)
|
fingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
|
||||||
}
|
if err != nil || len(fingerprints) != 1 {
|
||||||
if _, ok := client.(contracts.LLMProfileManifestProvider); !ok {
|
t.Fatalf("production LLM checkpoint fingerprints = %#v, error = %v, want one profile-source identity", fingerprints, err)
|
||||||
t.Fatalf("production LLM client %T does not provide profile manifests", client)
|
}
|
||||||
}
|
if _, ok := client.(contracts.LLMProfileManifestProvider); !ok {
|
||||||
})
|
t.Fatalf("production LLM client %T does not provide profile manifests", client)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNormalizeOptionsSharesProductionProfileAssetsWithDefaultRuntime(t *testing.T) {
|
||||||
|
opts, err := normalizeOptions(Options{
|
||||||
|
Catalog: pipeline.ModuleCatalog{Inputs: pipeline.NewInputAdapterRegistry()},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if opts.promptKitAssets == nil || opts.LLMClientFactory == nil {
|
||||||
|
t.Fatalf("normalized options = %#v, want shared profile assets and default runtime factory", opts)
|
||||||
|
}
|
||||||
|
if err := validateExplicitPromptKitProfiles(context.Background(), config.Default(), []string{"dnd-extraction"}, opts.promptKitAssets); err != nil {
|
||||||
|
t.Fatalf("inspect application fallback profile: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
client, _, err := opts.LLMClientFactory(context.Background(), config.Default(), "dnd-extraction", LLMRuntimeOverrides{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("build default runtime: %v", err)
|
||||||
|
}
|
||||||
|
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("default runtime client %T does not provide checkpoint fingerprints", client)
|
||||||
|
}
|
||||||
|
runtimeFingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
directClient, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{Assets: opts.promptKitAssets})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
inspectionFingerprints, err := directClient.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(runtimeFingerprints, inspectionFingerprints) {
|
||||||
|
t.Fatalf("runtime profile fingerprints = %#v, inspection profile fingerprints = %#v", runtimeFingerprints, inspectionFingerprints)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProductionLLMClientFactoryUsesConfiguredLocalBackend(t *testing.T) {
|
||||||
|
var providerCalls atomic.Int32
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
providerCalls.Add(1)
|
||||||
|
if r.URL.Path != "/v1/chat/completions" {
|
||||||
|
t.Errorf("provider path = %q, want /v1/chat/completions", r.URL.Path)
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_, _ = w.Write([]byte(`{
|
||||||
|
"choices": [{"message": {"role": "assistant", "content": "{\"ok\":true}"}}],
|
||||||
|
"usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7}
|
||||||
|
}`))
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte(`id: local-profile
|
||||||
|
backend: local
|
||||||
|
model: local-model
|
||||||
|
`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
assets := llm.NewAssetRegistry()
|
||||||
|
if err := assets.RegisterPromptFS(fstest.MapFS{
|
||||||
|
"production.local.yaml": {Data: []byte(`id: production.local
|
||||||
|
version: "v1"
|
||||||
|
inputs:
|
||||||
|
- name: transcript
|
||||||
|
required: true
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: '{{ input "transcript" }}'
|
||||||
|
output:
|
||||||
|
format: json
|
||||||
|
validation_mode: json
|
||||||
|
`)},
|
||||||
|
}, "."); err != nil {
|
||||||
|
t.Fatalf("register prompt assets: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg := config.Default()
|
||||||
|
cfg.PromptKit.ProfileFile = profilePath
|
||||||
|
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: server.URL + "/v1",
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
client, manifests, err := productionLLMClientFactoryWithAssets(assets)(
|
||||||
|
context.Background(),
|
||||||
|
cfg,
|
||||||
|
"local-profile",
|
||||||
|
LLMRuntimeOverrides{},
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("build production LLM runtime: %v", err)
|
||||||
|
}
|
||||||
|
if len(manifests) != 0 {
|
||||||
|
t.Fatalf("eager profile manifests = %#v, want none", manifests)
|
||||||
|
}
|
||||||
|
|
||||||
|
var out map[string]any
|
||||||
|
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||||
|
PromptID: "production.local",
|
||||||
|
ProfileID: "local-profile",
|
||||||
|
Inputs: contracts.LLMInputSet{
|
||||||
|
"transcript": contracts.NewLLMInputMaterial("transcript", "text/plain", []byte("local request"), "", ""),
|
||||||
|
},
|
||||||
|
}, &out)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if providerCalls.Load() != 1 {
|
||||||
|
t.Fatalf("provider calls = %d, want 1", providerCalls.Load())
|
||||||
|
}
|
||||||
|
provider, ok := client.(contracts.LLMProfileManifestProvider)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("production client %T does not provide profile manifests", client)
|
||||||
|
}
|
||||||
|
recorded := provider.LLMProfileManifests()
|
||||||
|
if len(recorded) != 1 || recorded[0].BackendID != promptkit.BackendLocal {
|
||||||
|
t.Fatalf("production profile manifests = %#v, want local backend", recorded)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -401,14 +625,15 @@ func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
|
|||||||
t.Run("canceled context", func(t *testing.T) {
|
t.Run("canceled context", func(t *testing.T) {
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
cancel()
|
cancel()
|
||||||
client, manifests, err := productionLLMClientFactory(ctx, config.Default(), "test-profile")
|
components := productionTestComponents(t)
|
||||||
|
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(ctx, config.Default(), "test-profile", LLMRuntimeOverrides{})
|
||||||
if !errors.Is(err, context.Canceled) || client != nil || len(manifests) != 0 {
|
if !errors.Is(err, context.Canceled) || client != nil || len(manifests) != 0 {
|
||||||
t.Fatalf("client=%T manifests=%#v error=%v, want canceled construction", client, manifests, err)
|
t.Fatalf("client=%T manifests=%#v error=%v, want canceled construction", client, manifests, err)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
t.Run("nil assets", func(t *testing.T) {
|
t.Run("nil assets", func(t *testing.T) {
|
||||||
client, manifests, err := productionLLMClientFactoryWithAssets(nil)(context.Background(), config.Default(), "test-profile")
|
client, manifests, err := productionLLMClientFactoryWithAssets(nil)(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
|
||||||
if err == nil || !strings.Contains(err.Error(), "asset registry must not be nil") || client != nil || len(manifests) != 0 {
|
if err == nil || !strings.Contains(err.Error(), "asset registry must not be nil") || client != nil || len(manifests) != 0 {
|
||||||
t.Fatalf("client=%T manifests=%#v error=%v, want nil-assets failure", client, manifests, err)
|
t.Fatalf("client=%T manifests=%#v error=%v, want nil-assets failure", client, manifests, err)
|
||||||
}
|
}
|
||||||
@@ -418,7 +643,7 @@ func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
|
|||||||
components := productionTestComponents(t)
|
components := productionTestComponents(t)
|
||||||
cfg := config.Default()
|
cfg := config.Default()
|
||||||
cfg.Concurrency.TotalLLM = 0
|
cfg.Concurrency.TotalLLM = 0
|
||||||
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), cfg, "test-profile")
|
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), cfg, "test-profile", LLMRuntimeOverrides{})
|
||||||
if err == nil || !strings.Contains(err.Error(), "create LLM scheduler") || !strings.Contains(err.Error(), "greater than zero") || client != nil || len(manifests) != 0 {
|
if err == nil || !strings.Contains(err.Error(), "create LLM scheduler") || !strings.Contains(err.Error(), "greater than zero") || client != nil || len(manifests) != 0 {
|
||||||
t.Fatalf("client=%T manifests=%#v error=%v, want scheduler-construction failure", client, manifests, err)
|
t.Fatalf("client=%T manifests=%#v error=%v, want scheduler-construction failure", client, manifests, err)
|
||||||
}
|
}
|
||||||
@@ -583,7 +808,7 @@ func maintainedExampleFiles(t *testing.T) []maintainedExample {
|
|||||||
|
|
||||||
func productionSpellCatalogContractConfig(t *testing.T) string {
|
func productionSpellCatalogContractConfig(t *testing.T) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
return fmt.Sprintf(`version: 3
|
return fmt.Sprintf(`version: 4
|
||||||
cache:
|
cache:
|
||||||
chunk_plans:
|
chunk_plans:
|
||||||
mode: bypass
|
mode: bypass
|
||||||
@@ -648,9 +873,10 @@ func productionCLIOptions(t *testing.T) Options {
|
|||||||
|
|
||||||
func productionOptionsFromComponents(components productionComponents) Options {
|
func productionOptionsFromComponents(components productionComponents) Options {
|
||||||
return Options{
|
return Options{
|
||||||
Catalog: catalogFromRegistries(components.registries),
|
Catalog: catalogFromRegistries(components.registries),
|
||||||
Registries: components.registries,
|
Registries: components.registries,
|
||||||
LookupEnv: emptyLookup,
|
LookupEnv: emptyLookup,
|
||||||
|
promptKitAssets: components.assets,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -673,14 +899,14 @@ func productionRunOptions(t *testing.T, fake *productionFakeLLMClient) Options {
|
|||||||
options.Now = func() time.Time { return time.Unix(1700000000, 0).UTC() }
|
options.Now = func() time.Time { return time.Unix(1700000000, 0).UTC() }
|
||||||
options.RunIDGenerator = func(time.Time) (string, error) { return productionRunID, nil }
|
options.RunIDGenerator = func(time.Time) (string, error) { return productionRunID, nil }
|
||||||
options.UserCacheDir = func() (string, error) { return "", errors.New("user cache must not be used") }
|
options.UserCacheDir = func() (string, error) { return "", errors.New("user cache must not be used") }
|
||||||
options.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
options.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
return fake, nil, nil
|
return fake, nil, nil
|
||||||
}
|
}
|
||||||
return options
|
return options
|
||||||
}
|
}
|
||||||
|
|
||||||
func productionRunConfig(outputRoot, chunkModule string) string {
|
func productionRunConfig(outputRoot, chunkModule string) string {
|
||||||
return fmt.Sprintf(`version: 3
|
return fmt.Sprintf(`version: 4
|
||||||
output:
|
output:
|
||||||
directory: %q
|
directory: %q
|
||||||
cache:
|
cache:
|
||||||
|
|||||||
46
internal/cli/promptkit_profiles.go
Normal file
46
internal/cli/promptkit_profiles.go
Normal file
@@ -0,0 +1,46 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
|
)
|
||||||
|
|
||||||
|
func validateExplicitPromptKitProfiles(ctx context.Context, cfg config.Config, profileIDs []string, assets *llm.AssetRegistry) error {
|
||||||
|
if len(profileIDs) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
inspector, err := llm.NewPromptKitProfileInspector(llm.PromptKitProfileInspectorConfig{
|
||||||
|
Source: promptKitProfileSourceConfig(cfg),
|
||||||
|
Assets: assets,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("load PromptKit profiles: %w", err)
|
||||||
|
}
|
||||||
|
for _, profileID := range profileIDs {
|
||||||
|
if _, err := inspector.InspectProfile(ctx, profileID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptKitProfileSourceConfig(cfg config.Config) llm.PromptKitProfileSourceConfig {
|
||||||
|
return llm.PromptKitProfileSourceConfig{
|
||||||
|
ProfileDir: cfg.PromptKit.ProfileDir,
|
||||||
|
ProfileFile: cfg.PromptKit.ProfileFile,
|
||||||
|
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mapPromptKitLocalBackend(cfg *config.PromptKitLocalBackendConfig) *llm.PromptKitLocalBackendConfig {
|
||||||
|
if cfg == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return &llm.PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: cfg.Endpoint,
|
||||||
|
ConcurrencyLimit: cfg.ConcurrencyLimit,
|
||||||
|
}
|
||||||
|
}
|
||||||
159
internal/cli/promptkit_profiles_test.go
Normal file
159
internal/cli/promptkit_profiles_test.go
Normal file
@@ -0,0 +1,159 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"testing/fstest"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestExplicitPromptKitProfileValidationInspectsProfilesWithoutGeneration(t *testing.T) {
|
||||||
|
var providerCalls atomic.Int32
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
|
||||||
|
providerCalls.Add(1)
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
writeProfile := func(t *testing.T, name, content string) string {
|
||||||
|
t.Helper()
|
||||||
|
profilePath := filepath.Join(t.TempDir(), name+".yaml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte(content), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return profilePath
|
||||||
|
}
|
||||||
|
localProfile := "id: local-profile\nbackend: local\nmodel: local-model\n"
|
||||||
|
credentialProfile := `id: credential-profile
|
||||||
|
endpoint: ` + server.URL + `/v1
|
||||||
|
model: credential-model
|
||||||
|
api_key_env: NOTARIUS_PROMPTKIT_PROFILE_INSPECTION_TEST_KEY
|
||||||
|
`
|
||||||
|
t.Setenv("NOTARIUS_PROMPTKIT_PROFILE_INSPECTION_TEST_KEY", "")
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
profilePath string
|
||||||
|
profileID string
|
||||||
|
profileDir bool
|
||||||
|
localBackend bool
|
||||||
|
canceled bool
|
||||||
|
wantErr []string
|
||||||
|
rejectErr []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "configured local backend",
|
||||||
|
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||||
|
profileID: "local-profile",
|
||||||
|
profileDir: true,
|
||||||
|
localBackend: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "missing local backend registration",
|
||||||
|
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||||
|
profileID: "local-profile",
|
||||||
|
wantErr: []string{`PromptKit profile "local-profile" is invalid or unreadable`},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "absent profile",
|
||||||
|
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||||
|
profileID: "absent-profile",
|
||||||
|
localBackend: true,
|
||||||
|
wantErr: []string{`PromptKit profile "absent-profile" is not configured`},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "malformed profile",
|
||||||
|
profilePath: writeProfile(t, "malformed-profile", "id: malformed-profile\nbackend: [\n"),
|
||||||
|
profileID: "malformed-profile",
|
||||||
|
wantErr: []string{`PromptKit profile "malformed-profile" is invalid or unreadable`},
|
||||||
|
rejectErr: []string{"malformed-profile.yaml", "backend: ["},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "invalid profile source",
|
||||||
|
profilePath: filepath.Join(t.TempDir(), "missing-profile.yaml"),
|
||||||
|
profileID: "missing-profile",
|
||||||
|
wantErr: []string{"load PromptKit profiles", "profile configuration is invalid or unreadable"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "credential environment intentionally unset",
|
||||||
|
profilePath: writeProfile(t, "credential-profile", credentialProfile),
|
||||||
|
profileID: "credential-profile",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "canceled inspection",
|
||||||
|
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||||
|
profileID: "local-profile",
|
||||||
|
localBackend: true,
|
||||||
|
canceled: true,
|
||||||
|
wantErr: []string{"context canceled"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := config.Default()
|
||||||
|
if tt.profileDir {
|
||||||
|
cfg.PromptKit.ProfileDir = filepath.Dir(tt.profilePath)
|
||||||
|
} else {
|
||||||
|
cfg.PromptKit.ProfileFile = tt.profilePath
|
||||||
|
}
|
||||||
|
if tt.localBackend {
|
||||||
|
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: server.URL + "/v1",
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
if tt.canceled {
|
||||||
|
var cancel context.CancelFunc
|
||||||
|
ctx, cancel = context.WithCancel(ctx)
|
||||||
|
cancel()
|
||||||
|
}
|
||||||
|
err := validateExplicitPromptKitProfiles(ctx, cfg, []string{tt.profileID}, nil)
|
||||||
|
if len(tt.wantErr) == 0 {
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("validateExplicitPromptKitProfiles() error = nil, want failure")
|
||||||
|
}
|
||||||
|
if tt.canceled && !errors.Is(err, context.Canceled) {
|
||||||
|
t.Fatalf("canceled inspection error = %v, want context canceled", err)
|
||||||
|
}
|
||||||
|
for _, want := range tt.wantErr {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("validation error = %q, want %q", err, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, rejected := range append(tt.rejectErr, tt.profilePath) {
|
||||||
|
if rejected != "" && strings.Contains(err.Error(), rejected) {
|
||||||
|
t.Fatalf("validation error = %q, must not expose %q", err, rejected)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if providerCalls.Load() != 0 {
|
||||||
|
t.Fatalf("provider calls during profile inspection = %d, want 0", providerCalls.Load())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExplicitPromptKitProfileValidationUsesFallbackAssets(t *testing.T) {
|
||||||
|
assets := llm.NewAssetRegistry()
|
||||||
|
if err := assets.RegisterFallbackProfileFS(fstest.MapFS{
|
||||||
|
"profiles/fallback.yaml": {Data: []byte("id: fallback-profile\nendpoint: http://promptkit.test/v1\nmodel: fallback-model\n")},
|
||||||
|
}, "profiles"); err != nil {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if err := validateExplicitPromptKitProfiles(context.Background(), config.Default(), []string{"fallback-profile"}, assets); err != nil {
|
||||||
|
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -116,7 +116,7 @@ func (h *recomputeTestHarness) options() Options {
|
|||||||
for _, key := range []string{"test/extract/producer", "test/extract/unrelated", "test/extract/middle", "test/extract/dependent"} {
|
for _, key := range []string{"test/extract/producer", "test/extract/unrelated", "test/extract/middle", "test/extract/dependent"} {
|
||||||
moduleKey := key
|
moduleKey := key
|
||||||
spec := pipeline.ModuleSpec{
|
spec := pipeline.ModuleSpec{
|
||||||
Key: moduleKey, Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind,
|
Key: moduleKey, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind,
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}},
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterExtractor(opts.Registries.Extractors, spec, func() (contracts.Extractor[stateTestArtifact], error) {
|
if err := pipeline.RegisterExtractor(opts.Registries.Extractors, spec, func() (contracts.Extractor[stateTestArtifact], error) {
|
||||||
@@ -125,7 +125,7 @@ func (h *recomputeTestHarness) options() Options {
|
|||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if err := opts.Registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/recompute-output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
if err := opts.Registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/recompute-output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
||||||
return recomputeTestOutput{}, nil
|
return recomputeTestOutput{}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
@@ -194,7 +194,7 @@ func (recomputeTestOutput) Encode(_ context.Context, req contracts.OutputRequest
|
|||||||
func newRecomputeTestRoots(t *testing.T) stateTestRoots {
|
func newRecomputeTestRoots(t *testing.T) stateTestRoots {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
config := fmt.Sprintf(`version: 3
|
config := fmt.Sprintf(`version: 4
|
||||||
output:
|
output:
|
||||||
directory: %q
|
directory: %q
|
||||||
cache:
|
cache:
|
||||||
|
|||||||
@@ -217,7 +217,7 @@ func TestReferenceMaterializationSeparatesCLIAndConfigPathOrigins(t *testing.T)
|
|||||||
workingDir := t.TempDir()
|
workingDir := t.TempDir()
|
||||||
cfg := referenceContractConfig()
|
cfg := referenceContractConfig()
|
||||||
configPath := filepath.Join(configDir, "config.yml")
|
configPath := filepath.Join(configDir, "config.yml")
|
||||||
if err := os.WriteFile(configPath, []byte("version: 3\n"), 0o600); err != nil {
|
if err := os.WriteFile(configPath, []byte("version: 4\n"), 0o600); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(filepath.Join(configDir, "required.txt"), []byte("config reference"), 0o600); err != nil {
|
if err := os.WriteFile(filepath.Join(configDir, "required.txt"), []byte("config reference"), 0o600); err != nil {
|
||||||
@@ -364,21 +364,21 @@ func referenceContractCatalog(t *testing.T, includeBetaMerger, includeBetaNormal
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
register(registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "reference/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }))
|
register(registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "reference/input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }))
|
||||||
register(registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/chunk", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-slot"}, {Name: "required-chunk", Required: true}}}, func() (contracts.Chunker, error) { return stateTestChunker{}, nil }))
|
register(registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-slot"}, {Name: "required-chunk", Required: true}}}, func() (contracts.Chunker, error) { return stateTestChunker{}, nil }))
|
||||||
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecA{}))
|
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecA{}))
|
||||||
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecB{}))
|
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecB{}))
|
||||||
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-alpha", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-alpha", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
||||||
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-beta", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-beta", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
||||||
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
||||||
if includeBetaMerger {
|
if includeBetaMerger {
|
||||||
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
||||||
}
|
}
|
||||||
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
||||||
if includeBetaNormalizer {
|
if includeBetaNormalizer {
|
||||||
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
||||||
}
|
}
|
||||||
register(registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { return stateTestOutput{}, nil }))
|
register(registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { return stateTestOutput{}, nil }))
|
||||||
return catalogFromRegistries(registries)
|
return catalogFromRegistries(registries)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -23,13 +23,14 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkplan"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkplan"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
frameworkdebug "gitea.maximumdirect.net/eric/notarius/internal/framework/debug"
|
frameworkdebug "gitea.maximumdirect.net/eric/notarius/internal/framework/debug"
|
||||||
|
frameworkllm "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
)
|
)
|
||||||
|
|
||||||
const defaultConfigPath = "/usr/local/etc/notarius/config.yml"
|
const defaultConfigPath = "/usr/local/etc/notarius/config.yml"
|
||||||
const usage = `Usage:
|
const usage = `Usage:
|
||||||
notarius help
|
notarius help
|
||||||
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--output-dir path] [--json] [--chunk_cache auto|bypass|refresh] [--resume] [--recompute-step step-id] [--debug [--debug-dir path]] [--only lane-a,lane-b] [--session-id id] [--reference selector=path] [--without-reference selector]
|
notarius run <pipeline-id> --input path/to/source.json [--json] [flags]
|
||||||
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
|
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
|
||||||
notarius pipelines list --config path/to/config.yml [--json]
|
notarius pipelines list --config path/to/config.yml [--json]
|
||||||
`
|
`
|
||||||
@@ -45,9 +46,14 @@ type Options struct {
|
|||||||
ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory
|
ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory
|
||||||
DebugRecorderFactory func(string) (pipeline.DebugRecorder, error)
|
DebugRecorderFactory func(string) (pipeline.DebugRecorder, error)
|
||||||
DebugTerminalFactory func(*debugbundle.SummaryWriter) DebugTerminalWriter
|
DebugTerminalFactory func(*debugbundle.SummaryWriter) DebugTerminalWriter
|
||||||
|
promptKitAssets *frameworkllm.AssetRegistry
|
||||||
}
|
}
|
||||||
|
|
||||||
type LLMClientFactory func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error)
|
type LLMRuntimeOverrides struct {
|
||||||
|
ReasoningEffort *string
|
||||||
|
}
|
||||||
|
|
||||||
|
type LLMClientFactory func(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error)
|
||||||
|
|
||||||
// Run executes the command-line interface and returns a process exit code.
|
// Run executes the command-line interface and returns a process exit code.
|
||||||
func Run(args []string, stdout, stderr io.Writer) int {
|
func Run(args []string, stdout, stderr io.Writer) int {
|
||||||
@@ -116,12 +122,17 @@ func normalizeOptions(opts Options) (Options, error) {
|
|||||||
}
|
}
|
||||||
opts.Registries = components.registries
|
opts.Registries = components.registries
|
||||||
opts.Catalog = catalogFromRegistries(components.registries)
|
opts.Catalog = catalogFromRegistries(components.registries)
|
||||||
if opts.LLMClientFactory == nil {
|
opts.promptKitAssets = components.assets
|
||||||
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(components.assets)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if opts.LLMClientFactory == nil {
|
if opts.LLMClientFactory == nil {
|
||||||
opts.LLMClientFactory = productionLLMClientFactory
|
if opts.promptKitAssets == nil {
|
||||||
|
assets, err := productionPromptAssets()
|
||||||
|
if err != nil {
|
||||||
|
return Options{}, err
|
||||||
|
}
|
||||||
|
opts.promptKitAssets = assets
|
||||||
|
}
|
||||||
|
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(opts.promptKitAssets)
|
||||||
}
|
}
|
||||||
return opts, nil
|
return opts, nil
|
||||||
}
|
}
|
||||||
@@ -137,13 +148,16 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
debug := fs.Bool("debug", false, "write a debug bundle")
|
debug := fs.Bool("debug", false, "write a debug bundle")
|
||||||
debugDir := fs.String("debug-dir", "", "debug bundle directory")
|
debugDir := fs.String("debug-dir", "", "debug bundle directory")
|
||||||
llmProfile := fs.String("llm-profile", "", "LLM profile override")
|
llmProfile := fs.String("llm-profile", "", "LLM profile override")
|
||||||
|
reasoningEffort := singleValueFlag{name: "--reasoning-effort"}
|
||||||
|
clearReasoningEffort := fs.Bool("clear-reasoning-effort", false, "clear the LLM profile reasoning effort")
|
||||||
resume := fs.Bool("resume", false, "reuse compatible recorded checkpoints")
|
resume := fs.Bool("resume", false, "reuse compatible recorded checkpoints")
|
||||||
recomputeStep := singleValueFlag{}
|
recomputeStep := singleValueFlag{name: "--recompute-step"}
|
||||||
chunkCache := chunkCacheFlag{}
|
chunkCache := chunkCacheFlag{}
|
||||||
sessionID := sessionIDFlag{}
|
sessionID := sessionIDFlag{}
|
||||||
referenceFlags := stringListFlag{}
|
referenceFlags := stringListFlag{}
|
||||||
withoutReferenceFlags := stringListFlag{}
|
withoutReferenceFlags := stringListFlag{}
|
||||||
fs.Var(&sessionID, "session-id", "prompt session identifier")
|
fs.Var(&sessionID, "session-id", "prompt session identifier")
|
||||||
|
fs.Var(&reasoningEffort, "reasoning-effort", "reasoning effort override")
|
||||||
fs.Var(&chunkCache, "chunk_cache", "chunk plan cache mode: auto, bypass, or refresh")
|
fs.Var(&chunkCache, "chunk_cache", "chunk plan cache mode: auto, bypass, or refresh")
|
||||||
fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, merge.slot=path, lane.slot=path, lane.extract.slot=path, lane.merge.slot=path, or lane.normalize.slot=path")
|
fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, merge.slot=path, lane.slot=path, lane.extract.slot=path, lane.merge.slot=path, or lane.normalize.slot=path")
|
||||||
fs.Var(&withoutReferenceFlags, "without-reference", "unbind a reference, using the same selector forms as --reference")
|
fs.Var(&withoutReferenceFlags, "without-reference", "unbind a reference, using the same selector forms as --reference")
|
||||||
@@ -189,6 +203,22 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
fmt.Fprintln(stderr, "notarius: --session-id must not be empty")
|
fmt.Fprintln(stderr, "notarius: --session-id must not be empty")
|
||||||
return 2
|
return 2
|
||||||
}
|
}
|
||||||
|
if reasoningEffort.set && *clearReasoningEffort {
|
||||||
|
fmt.Fprintln(stderr, "notarius: --reasoning-effort cannot be combined with --clear-reasoning-effort")
|
||||||
|
return 2
|
||||||
|
}
|
||||||
|
if reasoningEffort.set && strings.TrimSpace(reasoningEffort.value) == "" {
|
||||||
|
fmt.Fprintln(stderr, "notarius: --reasoning-effort must not be empty")
|
||||||
|
return 2
|
||||||
|
}
|
||||||
|
runtimeOverrides := LLMRuntimeOverrides{}
|
||||||
|
if reasoningEffort.set {
|
||||||
|
value := strings.TrimSpace(reasoningEffort.value)
|
||||||
|
runtimeOverrides.ReasoningEffort = &value
|
||||||
|
} else if *clearReasoningEffort {
|
||||||
|
value := ""
|
||||||
|
runtimeOverrides.ReasoningEffort = &value
|
||||||
|
}
|
||||||
only, err := parseOnly(*onlyRaw)
|
only, err := parseOnly(*onlyRaw)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||||
@@ -279,17 +309,18 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
debugRecorder = pipeline.SynchronizedDebugRecorder(debugRecorder)
|
debugRecorder = pipeline.SynchronizedDebugRecorder(debugRecorder)
|
||||||
}
|
}
|
||||||
invocation := debugbundle.Invocation{
|
invocation := debugbundle.Invocation{
|
||||||
Operation: "run",
|
Operation: "run",
|
||||||
PipelineID: pipelineID,
|
PipelineID: pipelineID,
|
||||||
InputPath: strings.TrimSpace(*inputPath),
|
InputPath: strings.TrimSpace(*inputPath),
|
||||||
ConfigPath: loadedConfigPath,
|
ConfigPath: loadedConfigPath,
|
||||||
ConfigSource: configSource(*configPath),
|
ConfigSource: configSource(*configPath),
|
||||||
OnlyLanes: append([]string(nil), only...),
|
OnlyLanes: append([]string(nil), only...),
|
||||||
ChunkCacheOverride: chunkCache.explicitValue(),
|
ChunkCacheOverride: chunkCache.explicitValue(),
|
||||||
Resume: *resume,
|
ReasoningEffortOverride: runtimeOverrides.ReasoningEffort,
|
||||||
RecomputeStep: strings.TrimSpace(recomputeStep.value),
|
Resume: *resume,
|
||||||
RunID: runID,
|
RecomputeStep: strings.TrimSpace(recomputeStep.value),
|
||||||
StartedAt: startedAt,
|
RunID: runID,
|
||||||
|
StartedAt: startedAt,
|
||||||
}
|
}
|
||||||
if err := writeSummary(summary, func() error { return summary.WriteInvocation(invocation) }); err != nil {
|
if err := writeSummary(summary, func() error { return summary.WriteInvocation(invocation) }); err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug invocation metadata: %w", err))
|
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug invocation metadata: %w", err))
|
||||||
@@ -315,7 +346,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||||
}
|
}
|
||||||
profileIDs := effectiveLLMProfileIDs(effective.ResolvedPipeline)
|
profileIDs := effectiveLLMProfileIDs(effective.ResolvedPipeline)
|
||||||
if err := validateExplicitScriptoriumProfiles(context.Background(), effective.Config, profileIDs); err != nil {
|
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, profileIDs, opts.promptKitAssets); err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||||
}
|
}
|
||||||
workingDir, err := os.Getwd()
|
workingDir, err := os.Getwd()
|
||||||
@@ -363,10 +394,17 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
if len(profileIDs) == 1 {
|
if len(profileIDs) == 1 {
|
||||||
factoryProfileID = profileIDs[0]
|
factoryProfileID = profileIDs[0]
|
||||||
}
|
}
|
||||||
llmClient, llmProfiles, err := opts.LLMClientFactory(ctx, effective.Config, factoryProfileID)
|
llmClient, llmProfiles, err := opts.LLMClientFactory(ctx, effective.Config, factoryProfileID, runtimeOverrides)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("create LLM client for profile %q: %w", factoryProfileID, err))
|
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("create LLM client for profile %q: %w", factoryProfileID, err))
|
||||||
}
|
}
|
||||||
|
var llmFingerprints []checkpoint.Fingerprint
|
||||||
|
if effective.Config.Cache.Checkpoints.Enabled {
|
||||||
|
llmFingerprints, err = llmCheckpointFingerprints(llmClient)
|
||||||
|
if err != nil {
|
||||||
|
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("prepare LLM checkpoint identity: %w", err))
|
||||||
|
}
|
||||||
|
}
|
||||||
llmClient = pipeline.WithDebugLLMRecording(llmClient, debugRecorder)
|
llmClient = pipeline.WithDebugLLMRecording(llmClient, debugRecorder)
|
||||||
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, registries, pipeline.ModuleDependencies{LLM: llmClient})
|
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, registries, pipeline.ModuleDependencies{LLM: llmClient})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -380,7 +418,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||||
}
|
}
|
||||||
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(effective.Config.Cache.Checkpoints, opts, effective.ResolvedPipeline, prepared.CheckpointFingerprints(), rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), *resume)
|
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(effective.Config.Cache.Checkpoints, opts, effective.ResolvedPipeline, prepared.CheckpointFingerprints(), llmFingerprints, rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), runtimeOverrides, *resume)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||||
}
|
}
|
||||||
@@ -481,11 +519,13 @@ func checkpointHandlersForRun(
|
|||||||
opts Options,
|
opts Options,
|
||||||
resolved pipeline.ResolvedPipeline,
|
resolved pipeline.ResolvedPipeline,
|
||||||
componentFingerprints []pipeline.CheckpointFingerprint,
|
componentFingerprints []pipeline.CheckpointFingerprint,
|
||||||
|
llmFingerprints []checkpoint.Fingerprint,
|
||||||
rawInput []byte,
|
rawInput []byte,
|
||||||
only []string,
|
only []string,
|
||||||
llmProfiles []artifacts.LLMProfileManifest,
|
llmProfiles []artifacts.LLMProfileManifest,
|
||||||
llmProfileOverride string,
|
llmProfileOverride string,
|
||||||
sessionID string,
|
sessionID string,
|
||||||
|
runtimeOverrides LLMRuntimeOverrides,
|
||||||
resume bool,
|
resume bool,
|
||||||
) (pipeline.CheckpointRecorder, pipeline.CheckpointLoader, error) {
|
) (pipeline.CheckpointRecorder, pipeline.CheckpointLoader, error) {
|
||||||
if !settings.Enabled {
|
if !settings.Enabled {
|
||||||
@@ -495,13 +535,17 @@ func checkpointHandlersForRun(
|
|||||||
return pipeline.NoopCheckpointRecorder(), pipeline.NoopCheckpointLoader(), nil
|
return pipeline.NoopCheckpointRecorder(), pipeline.NoopCheckpointLoader(), nil
|
||||||
}
|
}
|
||||||
identity, err := checkpoint.NewIdentity(checkpoint.IdentityInput{
|
identity, err := checkpoint.NewIdentity(checkpoint.IdentityInput{
|
||||||
Pipeline: resolved,
|
Pipeline: resolved,
|
||||||
InputKey: resolved.Input.Module,
|
InputKey: resolved.Input.Module,
|
||||||
RawInputDigest: rawInputDigest(rawInput),
|
RawInputDigest: rawInputDigest(rawInput),
|
||||||
SelectedLanes: only,
|
SelectedLanes: only,
|
||||||
RuntimeOverrides: runtimeOverrideFingerprints(llmProfileOverride, sessionID),
|
RuntimeOverrides: runtimeOverrideFingerprints(llmProfileOverride, sessionID, runtimeOverrides),
|
||||||
References: pipeline.ReferenceProvenance(resolved),
|
References: pipeline.ReferenceProvenance(resolved),
|
||||||
ProvenanceFingerprints: append(llmProfileFingerprints(llmProfiles), checkpointIdentityFingerprints(componentFingerprints)...),
|
ProvenanceFingerprints: combineCheckpointFingerprints(
|
||||||
|
llmProfileFingerprints(llmProfiles),
|
||||||
|
llmFingerprints,
|
||||||
|
checkpointIdentityFingerprints(componentFingerprints),
|
||||||
|
),
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, nil, fmt.Errorf("create checkpoint identity: %w", err)
|
return nil, nil, fmt.Errorf("create checkpoint identity: %w", err)
|
||||||
@@ -527,6 +571,30 @@ func checkpointHandlersForRun(
|
|||||||
return recorder, loader, nil
|
return recorder, loader, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func llmCheckpointFingerprints(client contracts.StructuredLLMClient) ([]checkpoint.Fingerprint, error) {
|
||||||
|
provider, ok := client.(frameworkllm.CheckpointFingerprintProvider)
|
||||||
|
if !ok {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
values, err := provider.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out := make([]checkpoint.Fingerprint, 0, len(values))
|
||||||
|
for _, value := range values {
|
||||||
|
out = append(out, checkpoint.Fingerprint{Name: value.Name, Value: value.Value})
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func combineCheckpointFingerprints(sources ...[]checkpoint.Fingerprint) []checkpoint.Fingerprint {
|
||||||
|
var out []checkpoint.Fingerprint
|
||||||
|
for _, source := range sources {
|
||||||
|
out = append(out, source...)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
func recomputePolicy(resolved pipeline.ResolvedPipeline, requestedStep string) (pipeline.CheckpointExecutionPolicy, error) {
|
func recomputePolicy(resolved pipeline.ResolvedPipeline, requestedStep string) (pipeline.CheckpointExecutionPolicy, error) {
|
||||||
requestedStep = strings.TrimSpace(requestedStep)
|
requestedStep = strings.TrimSpace(requestedStep)
|
||||||
if requestedStep == "" {
|
if requestedStep == "" {
|
||||||
@@ -632,7 +700,7 @@ func rawInputDigest(data []byte) string {
|
|||||||
return "sha256:" + hex.EncodeToString(sum[:])
|
return "sha256:" + hex.EncodeToString(sum[:])
|
||||||
}
|
}
|
||||||
|
|
||||||
func runtimeOverrideFingerprints(llmProfileOverride string, sessionID string) []checkpoint.Fingerprint {
|
func runtimeOverrideFingerprints(llmProfileOverride string, sessionID string, runtimeOverrides LLMRuntimeOverrides) []checkpoint.Fingerprint {
|
||||||
var values []checkpoint.Fingerprint
|
var values []checkpoint.Fingerprint
|
||||||
if strings.TrimSpace(llmProfileOverride) != "" {
|
if strings.TrimSpace(llmProfileOverride) != "" {
|
||||||
values = append(values, checkpoint.Fingerprint{Name: "llm_profile_override", Value: strings.TrimSpace(llmProfileOverride)})
|
values = append(values, checkpoint.Fingerprint{Name: "llm_profile_override", Value: strings.TrimSpace(llmProfileOverride)})
|
||||||
@@ -640,6 +708,13 @@ func runtimeOverrideFingerprints(llmProfileOverride string, sessionID string) []
|
|||||||
if strings.TrimSpace(sessionID) != "" {
|
if strings.TrimSpace(sessionID) != "" {
|
||||||
values = append(values, checkpoint.Fingerprint{Name: "session_id", Value: strings.TrimSpace(sessionID)})
|
values = append(values, checkpoint.Fingerprint{Name: "session_id", Value: strings.TrimSpace(sessionID)})
|
||||||
}
|
}
|
||||||
|
if runtimeOverrides.ReasoningEffort != nil {
|
||||||
|
value := strings.TrimSpace(*runtimeOverrides.ReasoningEffort)
|
||||||
|
if value == "" {
|
||||||
|
value = "<cleared>"
|
||||||
|
}
|
||||||
|
values = append(values, checkpoint.Fingerprint{Name: "reasoning_effort_override", Value: value})
|
||||||
|
}
|
||||||
return values
|
return values
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -796,7 +871,7 @@ func reorderRunArgs(args []string) []string {
|
|||||||
|
|
||||||
func runFlagTakesValue(arg string) bool {
|
func runFlagTakesValue(arg string) bool {
|
||||||
switch arg {
|
switch arg {
|
||||||
case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--chunk_cache", "--reference", "--without-reference", "--recompute-step":
|
case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--reasoning-effort", "--chunk_cache", "--reference", "--without-reference", "--recompute-step":
|
||||||
return true
|
return true
|
||||||
default:
|
default:
|
||||||
return false
|
return false
|
||||||
@@ -856,11 +931,11 @@ func chunkPlanStoreForRun(cfg config.ChunkPlanCacheConfig, opts Options) (pipeli
|
|||||||
|
|
||||||
func validateRunFlagValues(args []string) error {
|
func validateRunFlagValues(args []string) error {
|
||||||
for i, arg := range args {
|
for i, arg := range args {
|
||||||
if arg != "--session-id" {
|
if arg != "--session-id" && arg != "--reasoning-effort" {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if i+1 >= len(args) || strings.HasPrefix(args[i+1], "-") {
|
if i+1 >= len(args) || strings.HasPrefix(args[i+1], "-") {
|
||||||
return fmt.Errorf("flag needs an argument: --session-id")
|
return fmt.Errorf("flag needs an argument: %s", arg)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
@@ -883,11 +958,22 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
|
|||||||
seen[id] = struct{}{}
|
seen[id] = struct{}{}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
add(resolved.Chunk)
|
if resolved.InputExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
|
add(resolved.Input)
|
||||||
|
}
|
||||||
|
if resolved.ChunkExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
|
add(resolved.Chunk)
|
||||||
|
}
|
||||||
for _, lane := range resolved.AllArtifactLanes() {
|
for _, lane := range resolved.AllArtifactLanes() {
|
||||||
add(lane.Extract)
|
if lane.ExtractExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
add(lane.Merge)
|
add(lane.Extract)
|
||||||
add(lane.Normalize)
|
}
|
||||||
|
if lane.MergeExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
|
add(lane.Merge)
|
||||||
|
}
|
||||||
|
if lane.NormalizeExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
|
add(lane.Normalize)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
for _, chain := range resolved.ValidatorChains {
|
for _, chain := range resolved.ValidatorChains {
|
||||||
for _, validator := range chain.Validators {
|
for _, validator := range chain.Validators {
|
||||||
@@ -896,6 +982,9 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if resolved.OutputExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
|
add(resolved.Output)
|
||||||
|
}
|
||||||
ids := make([]string, 0, len(seen))
|
ids := make([]string, 0, len(seen))
|
||||||
for id := range seen {
|
for id := range seen {
|
||||||
ids = append(ids, id)
|
ids = append(ids, id)
|
||||||
@@ -979,7 +1068,7 @@ func runConfigValidate(args []string, stdout, stderr io.Writer, opts Options) in
|
|||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
if err := validateExplicitScriptoriumProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline)); err != nil {
|
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline), opts.promptKitAssets); err != nil {
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
@@ -1144,6 +1233,7 @@ type sessionIDFlag struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type singleValueFlag struct {
|
type singleValueFlag struct {
|
||||||
|
name string
|
||||||
value string
|
value string
|
||||||
set bool
|
set bool
|
||||||
}
|
}
|
||||||
@@ -1157,7 +1247,7 @@ func (flag *singleValueFlag) String() string {
|
|||||||
|
|
||||||
func (flag *singleValueFlag) Set(value string) error {
|
func (flag *singleValueFlag) Set(value string) error {
|
||||||
if flag.set {
|
if flag.set {
|
||||||
return fmt.Errorf("--recompute-step may be specified only once")
|
return fmt.Errorf("%s may be specified only once", flag.name)
|
||||||
}
|
}
|
||||||
flag.value = value
|
flag.value = value
|
||||||
flag.set = true
|
flag.set = true
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ import (
|
|||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/checkpoint"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
)
|
)
|
||||||
@@ -241,12 +242,14 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
t.Run("one effective profile reaches the factory and modules", func(t *testing.T) {
|
t.Run("one effective profile reaches the factory and modules", func(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
profileDir := writeRunContractProfiles(t, "override-profile")
|
profileDir := writeRunContractProfiles(t, "override-profile")
|
||||||
prependRunContractConfig(t, roots, fmt.Sprintf("scriptorium:\n profile_dir: %q\n", profileDir))
|
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||||
harness := newStateTestHarness()
|
harness := newStateTestHarness()
|
||||||
var factoryProfiles []string
|
var factoryProfiles []string
|
||||||
opts := harness.options()
|
opts := harness.options()
|
||||||
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
var factoryOverrides []LLMRuntimeOverrides
|
||||||
|
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
factoryProfiles = append(factoryProfiles, profileID)
|
factoryProfiles = append(factoryProfiles, profileID)
|
||||||
|
factoryOverrides = append(factoryOverrides, overrides)
|
||||||
return nil, nil, nil
|
return nil, nil, nil
|
||||||
}
|
}
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
@@ -257,6 +260,9 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" {
|
if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" {
|
||||||
t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles)
|
t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles)
|
||||||
}
|
}
|
||||||
|
if len(factoryOverrides) != 1 || factoryOverrides[0].ReasoningEffort != nil {
|
||||||
|
t.Fatalf("factory overrides = %#v, want inherited reasoning", factoryOverrides)
|
||||||
|
}
|
||||||
harness.mu.Lock()
|
harness.mu.Lock()
|
||||||
profiles := append([]string(nil), harness.moduleProfiles...)
|
profiles := append([]string(nil), harness.moduleProfiles...)
|
||||||
harness.mu.Unlock()
|
harness.mu.Unlock()
|
||||||
@@ -270,16 +276,16 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
t.Run("validator profile remains distinct", func(t *testing.T) {
|
t.Run("runtime override applies to validators", func(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
profileDir := writeRunContractProfiles(t, "override-profile", "validator-profile")
|
profileDir := writeRunContractProfiles(t, "override-profile", "validator-profile")
|
||||||
prependRunContractConfig(t, roots, fmt.Sprintf("scriptorium:\n profile_dir: %q\n", profileDir))
|
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||||
harness := newStateTestHarness()
|
harness := newStateTestHarness()
|
||||||
var validatorProfiles []string
|
var validatorProfiles []string
|
||||||
opts := harness.options()
|
opts := harness.options()
|
||||||
registerRunContractValidator(t, &opts, &validatorProfiles)
|
registerRunContractValidator(t, &opts, &validatorProfiles)
|
||||||
factoryProfiles := []string{}
|
factoryProfiles := []string{}
|
||||||
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string, _ LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
factoryProfiles = append(factoryProfiles, profileID)
|
factoryProfiles = append(factoryProfiles, profileID)
|
||||||
return nil, nil, nil
|
return nil, nil, nil
|
||||||
}
|
}
|
||||||
@@ -288,21 +294,21 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
if code != 0 || stderr.Len() != 0 {
|
if code != 0 || stderr.Len() != 0 {
|
||||||
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
}
|
}
|
||||||
if len(factoryProfiles) != 1 || factoryProfiles[0] != "" {
|
if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" {
|
||||||
t.Fatalf("factory profiles = %#v, want one call without a unique profile", factoryProfiles)
|
t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles)
|
||||||
}
|
}
|
||||||
if len(validatorProfiles) != 1 || validatorProfiles[0] != "validator-profile" {
|
if len(validatorProfiles) != 1 || validatorProfiles[0] != "override-profile" {
|
||||||
t.Fatalf("validator profiles = %#v, want configured validator profile", validatorProfiles)
|
t.Fatalf("validator profiles = %#v, want runtime override", validatorProfiles)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
t.Run("unknown profile is rejected without factory access", func(t *testing.T) {
|
t.Run("unknown profile is rejected without factory access", func(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
profileDir := writeRunContractProfiles(t, "override-profile")
|
profileDir := writeRunContractProfiles(t, "override-profile")
|
||||||
prependRunContractConfig(t, roots, fmt.Sprintf("scriptorium:\n profile_dir: %q\n", profileDir))
|
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||||
factoryCalls := 0
|
factoryCalls := 0
|
||||||
opts := newStateTestHarness().options()
|
opts := newStateTestHarness().options()
|
||||||
opts.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
factoryCalls++
|
factoryCalls++
|
||||||
return nil, nil, nil
|
return nil, nil, nil
|
||||||
}
|
}
|
||||||
@@ -312,28 +318,175 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
|
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run("pipeline default is rejected before factory access", func(t *testing.T) {
|
||||||
|
roots := newStateTestRoots(t)
|
||||||
|
profileDir := writeRunContractProfiles(t, "configured-profile")
|
||||||
|
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||||
|
replaceStateTestConfigLine(t, roots.config, " sample:\n", " sample:\n llm_profile: missing-profile\n")
|
||||||
|
factoryCalls := 0
|
||||||
|
opts := newStateTestHarness().options()
|
||||||
|
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
|
factoryCalls++
|
||||||
|
return nil, nil, nil
|
||||||
|
}
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, &stdout, &stderr, opts)
|
||||||
|
if code != 1 || !strings.Contains(stderr.String(), "not configured") || factoryCalls != 0 || stdout.Len() != 0 {
|
||||||
|
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunReasoningEffortOverrideReachesFactory(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
flags []string
|
||||||
|
wantValue string
|
||||||
|
wantSet bool
|
||||||
|
}{
|
||||||
|
{name: "inherit"},
|
||||||
|
{name: "replace", flags: []string{"--reasoning-effort", " focused "}, wantValue: "focused", wantSet: true},
|
||||||
|
{name: "clear", flags: []string{"--clear-reasoning-effort"}, wantSet: true},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
roots := newStateTestRoots(t)
|
||||||
|
opts := newStateTestHarness().options()
|
||||||
|
var got []LLMRuntimeOverrides
|
||||||
|
opts.LLMClientFactory = func(_ context.Context, _ config.Config, _ string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
|
got = append(got, overrides)
|
||||||
|
return nil, nil, nil
|
||||||
|
}
|
||||||
|
args := append([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, tt.flags...)
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
if code := RunWithOptions(args, &stdout, &stderr, opts); code != 0 || stderr.Len() != 0 {
|
||||||
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
|
}
|
||||||
|
if len(got) != 1 {
|
||||||
|
t.Fatalf("factory overrides = %#v, want one call", got)
|
||||||
|
}
|
||||||
|
if !tt.wantSet {
|
||||||
|
if got[0].ReasoningEffort != nil {
|
||||||
|
t.Fatalf("reasoning effort = %q, want inherit", *got[0].ReasoningEffort)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if got[0].ReasoningEffort == nil || *got[0].ReasoningEffort != tt.wantValue {
|
||||||
|
t.Fatalf("reasoning effort = %#v, want %q", got[0].ReasoningEffort, tt.wantValue)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunReasoningEffortOverrideRejectsInvalidSyntax(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
flags []string
|
||||||
|
wantError string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "mutually exclusive controls",
|
||||||
|
flags: []string{"--reasoning-effort", "focused", "--clear-reasoning-effort"},
|
||||||
|
wantError: "cannot be combined",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "empty replacement",
|
||||||
|
flags: []string{"--reasoning-effort", " "},
|
||||||
|
wantError: "must not be empty",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "duplicate replacement",
|
||||||
|
flags: []string{"--reasoning-effort", "low", "--reasoning-effort", "high"},
|
||||||
|
wantError: "may be specified only once",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "missing replacement",
|
||||||
|
flags: []string{"--reasoning-effort"},
|
||||||
|
wantError: "flag needs an argument",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
roots := newStateTestRoots(t)
|
||||||
|
args := append([]string{"run", "sample", "--config", roots.config, "--input", roots.input}, tt.flags...)
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
code := RunWithOptions(args, &stdout, &stderr, newStateTestHarness().options())
|
||||||
|
if code != 2 || stdout.Len() != 0 || !strings.Contains(stderr.String(), tt.wantError) {
|
||||||
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
|
}
|
||||||
|
assertNoRunState(t, roots)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReasoningEffortOverrideSeparatesCheckpointIdentities(t *testing.T) {
|
||||||
|
replacement := " focused "
|
||||||
|
cleared := ""
|
||||||
|
states := []struct {
|
||||||
|
name string
|
||||||
|
overrides LLMRuntimeOverrides
|
||||||
|
wantValue string
|
||||||
|
wantSet bool
|
||||||
|
}{
|
||||||
|
{name: "inherit"},
|
||||||
|
{name: "replace", overrides: LLMRuntimeOverrides{ReasoningEffort: &replacement}, wantValue: "focused", wantSet: true},
|
||||||
|
{name: "clear", overrides: LLMRuntimeOverrides{ReasoningEffort: &cleared}, wantValue: "<cleared>", wantSet: true},
|
||||||
|
}
|
||||||
|
digests := make(map[string]string, len(states))
|
||||||
|
for _, state := range states {
|
||||||
|
fingerprints := runtimeOverrideFingerprints("", "", state.overrides)
|
||||||
|
var value string
|
||||||
|
var found bool
|
||||||
|
for _, fingerprint := range fingerprints {
|
||||||
|
if fingerprint.Name == "reasoning_effort_override" {
|
||||||
|
value, found = fingerprint.Value, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if found != state.wantSet || (found && value != state.wantValue) {
|
||||||
|
t.Fatalf("%s fingerprint found=%t value=%q, want found=%t value=%q", state.name, found, value, state.wantSet, state.wantValue)
|
||||||
|
}
|
||||||
|
identity, err := checkpoint.NewIdentity(checkpoint.IdentityInput{
|
||||||
|
Pipeline: pipeline.ResolvedPipeline{ID: "sample", Digest: "sha256:pipeline", Input: pipeline.Binding("test/input")},
|
||||||
|
RawInputDigest: "sha256:input",
|
||||||
|
RuntimeOverrides: fingerprints,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
digests[state.name] = identity.Digest
|
||||||
|
}
|
||||||
|
if digests["inherit"] == digests["replace"] || digests["inherit"] == digests["clear"] || digests["replace"] == digests["clear"] {
|
||||||
|
t.Fatalf("checkpoint identity digests are not distinct: %#v", digests)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
|
func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
|
||||||
resolved := pipeline.ResolvedPipeline{
|
resolved := pipeline.ResolvedPipeline{
|
||||||
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
|
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
|
||||||
Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "},
|
InputExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
|
Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "},
|
||||||
|
ChunkExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Steps: []pipeline.ResolvedPipelineStep{{
|
Steps: []pipeline.ResolvedPipelineStep{{
|
||||||
ID: "default",
|
ID: "default",
|
||||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
||||||
Extract: pipeline.ModuleBinding{LLMProfile: "alpha"},
|
Extract: pipeline.ModuleBinding{LLMProfile: "alpha"},
|
||||||
Merge: pipeline.ModuleBinding{LLMProfile: "zeta"},
|
ExtractExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "},
|
Merge: pipeline.ModuleBinding{LLMProfile: "deterministic-merge"},
|
||||||
|
MergeExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "},
|
||||||
|
NormalizeExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
}},
|
}},
|
||||||
}},
|
}},
|
||||||
ValidatorChains: []pipeline.ResolvedValidatorChain{{Validators: []pipeline.ResolvedValidator{
|
ValidatorChains: []pipeline.ResolvedValidatorChain{{Validators: []pipeline.ResolvedValidator{
|
||||||
{Binding: pipeline.ModuleBinding{LLMProfile: "deterministic-profile"}, ExecutionClass: contracts.ExecutionClassDeterministic},
|
{Binding: pipeline.ModuleBinding{LLMProfile: "deterministic-profile"}, ExecutionClass: contracts.ExecutionClassDeterministic},
|
||||||
{Binding: pipeline.ModuleBinding{LLMProfile: "beta"}, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
{Binding: pipeline.ModuleBinding{LLMProfile: "beta"}, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
}}},
|
}}},
|
||||||
Output: pipeline.ModuleBinding{LLMProfile: "output-profile"},
|
Output: pipeline.ModuleBinding{LLMProfile: "output-profile"},
|
||||||
|
OutputExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
}
|
}
|
||||||
got := effectiveLLMProfileIDs(resolved)
|
got := effectiveLLMProfileIDs(resolved)
|
||||||
want := []string{"alpha", "beta", "gamma", "zeta"}
|
want := []string{"alpha", "beta", "gamma", "input-profile", "output-profile", "zeta"}
|
||||||
if strings.Join(got, ",") != strings.Join(want, ",") {
|
if strings.Join(got, ",") != strings.Join(want, ",") {
|
||||||
t.Fatalf("effective profiles = %#v, want %#v", got, want)
|
t.Fatalf("effective profiles = %#v, want %#v", got, want)
|
||||||
}
|
}
|
||||||
@@ -376,7 +529,7 @@ func TestRunFactoryAndPreparationFailuresAreProcessFailures(t *testing.T) {
|
|||||||
t.Run("LLM factory", func(t *testing.T) {
|
t.Run("LLM factory", func(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
opts := newStateTestHarness().options()
|
opts := newStateTestHarness().options()
|
||||||
opts.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
return nil, nil, errors.New("injected LLM factory failure")
|
return nil, nil, errors.New("injected LLM factory failure")
|
||||||
}
|
}
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
@@ -397,7 +550,7 @@ func TestRunFactoryAndPreparationFailuresAreProcessFailures(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
opts := newStateTestHarness().options()
|
opts := newStateTestHarness().options()
|
||||||
if err := pipeline.RegisterExtractorBuilder(opts.Registries.Extractors, pipeline.ModuleSpec{Key: "test/failing-extract", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Extractor[stateTestArtifact], error) {
|
if err := pipeline.RegisterExtractorBuilder(opts.Registries.Extractors, pipeline.ModuleSpec{Key: "test/failing-extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Extractor[stateTestArtifact], error) {
|
||||||
return nil, errors.New("injected extractor construction failure")
|
return nil, errors.New("injected extractor construction failure")
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
|
|||||||
@@ -1,68 +0,0 @@
|
|||||||
package cli
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"testing/fstest"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
|
||||||
)
|
|
||||||
|
|
||||||
const profileCheckPromptID = "notarius.profile.check"
|
|
||||||
|
|
||||||
var profileCheckPromptFS = fstest.MapFS{
|
|
||||||
"prompts/profile-check.yaml": &fstest.MapFile{Data: []byte(`id: notarius.profile.check
|
|
||||||
version: "1.0.0"
|
|
||||||
default_profile: mistral-small-3
|
|
||||||
inputs:
|
|
||||||
- name: transcript
|
|
||||||
required: true
|
|
||||||
messages:
|
|
||||||
- role: user
|
|
||||||
content: "{{input \"transcript\"}}"
|
|
||||||
output:
|
|
||||||
format: text
|
|
||||||
validation_mode: none
|
|
||||||
repair_attempts: 0
|
|
||||||
`)},
|
|
||||||
}
|
|
||||||
|
|
||||||
func validateExplicitScriptoriumProfiles(ctx context.Context, cfg config.Config, profileIDs []string) error {
|
|
||||||
if len(profileIDs) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
engine, err := newProfileValidationEngine(cfg)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("load Scriptorium profiles: %w", err)
|
|
||||||
}
|
|
||||||
for _, profileID := range profileIDs {
|
|
||||||
if _, err := engine.Prepare(ctx, scriptorium.RunRequest{
|
|
||||||
PromptID: profileCheckPromptID,
|
|
||||||
ProfileID: profileID,
|
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
|
||||||
"transcript": scriptorium.Inline("profile check"),
|
|
||||||
},
|
|
||||||
}); err != nil {
|
|
||||||
if errors.Is(err, scriptorium.ErrProfileNotFound) {
|
|
||||||
return fmt.Errorf("Scriptorium profile %q is not configured", profileID)
|
|
||||||
}
|
|
||||||
return fmt.Errorf("validate Scriptorium profile %q: %w", profileID, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func newProfileValidationEngine(cfg config.Config) (*scriptorium.Engine, error) {
|
|
||||||
opts := []scriptorium.Option{
|
|
||||||
scriptorium.WithPromptFS(profileCheckPromptFS, "prompts"),
|
|
||||||
}
|
|
||||||
if cfg.Scriptorium.ProfileFile != "" {
|
|
||||||
opts = append(opts, scriptorium.WithProfileFile(cfg.Scriptorium.ProfileFile))
|
|
||||||
}
|
|
||||||
return scriptorium.NewEngine(scriptorium.Config{
|
|
||||||
PromptDir: "unused",
|
|
||||||
ProfileDir: cfg.Scriptorium.ProfileDir,
|
|
||||||
}, opts...)
|
|
||||||
}
|
|
||||||
@@ -214,8 +214,9 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
fingerprints := prepared.CheckpointFingerprints()
|
fingerprints := prepared.CheckpointFingerprints()
|
||||||
|
llmFingerprints := []checkpoint.Fingerprint{{Name: "promptkit_profile_source", Value: "sha256:profile-source-one"}}
|
||||||
settings := config.CheckpointCacheConfig{Enabled: true, Directory: t.TempDir()}
|
settings := config.CheckpointCacheConfig{Enabled: true, Directory: t.TempDir()}
|
||||||
recorder, _, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, []byte("same input"), nil, nil, "", "", false)
|
recorder, _, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -241,7 +242,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
_, sameLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, []byte("same input"), nil, nil, "", "", true)
|
_, sameLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -253,7 +254,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
|||||||
}
|
}
|
||||||
changed := replaceCheckpointFingerprintValue(t, fingerprints, normalizeSpellCatalogFingerprintName(), "sha256:changed-effective-catalog")
|
changed := replaceCheckpointFingerprintValue(t, fingerprints, normalizeSpellCatalogFingerprintName(), "sha256:changed-effective-catalog")
|
||||||
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changed, normalizeSpellCatalogFingerprintName())
|
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changed, normalizeSpellCatalogFingerprintName())
|
||||||
_, changedLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changed, []byte("same input"), nil, nil, "", "", true)
|
_, changedLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changed, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -265,13 +266,25 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
|||||||
}
|
}
|
||||||
changedMapping := replaceCheckpointFingerprintValue(t, fingerprints, extractSpellMappingFingerprintName(), "dnd.spells.extract_mapping.v3")
|
changedMapping := replaceCheckpointFingerprintValue(t, fingerprints, extractSpellMappingFingerprintName(), "dnd.spells.extract_mapping.v3")
|
||||||
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changedMapping, extractSpellMappingFingerprintName())
|
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changedMapping, extractSpellMappingFingerprintName())
|
||||||
_, mappingLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changedMapping, []byte("same input"), nil, nil, "", "", true)
|
_, mappingLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changedMapping, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, decision := mappingLoader.Source(materialized.Input.Module); decision.Reused {
|
if _, decision := mappingLoader.Source(materialized.Input.Module); decision.Reused {
|
||||||
t.Fatalf("changed mapping policy decision = %#v, want cold miss", decision)
|
t.Fatalf("changed mapping policy decision = %#v, want cold miss", decision)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
changedLLMFingerprints := []checkpoint.Fingerprint{{Name: "promptkit_profile_source", Value: "sha256:profile-source-two"}}
|
||||||
|
_, profileLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, changedLLMFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, decision := profileLoader.Source(materialized.Input.Module); decision.Reused {
|
||||||
|
t.Fatalf("changed PromptKit profile source decision = %#v, want cold miss", decision)
|
||||||
|
}
|
||||||
|
if _, decision := profileLoader.Normalize("spells", spellnormalize.Key, normalizeDependencies); decision.Reused {
|
||||||
|
t.Fatalf("changed PromptKit profile normalize decision = %#v, want cold miss", decision)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeSpellCatalogFingerprintName() string {
|
func normalizeSpellCatalogFingerprintName() string {
|
||||||
|
|||||||
@@ -635,7 +635,7 @@ func newStateTestRoots(t *testing.T) stateTestRoots {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
roots.config = filepath.Join(base, "config.yml")
|
roots.config = filepath.Join(base, "config.yml")
|
||||||
config := fmt.Sprintf("version: 3\noutput:\n directory: %q\ncache:\n chunk_plans:\n directory: %q\n mode: auto\n checkpoints:\n enabled: true\n directory: %q\ndebug:\n directory: %q\npipelines:\n sample:\n input: test/input\n chunk: test/chunk\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n output: test/output\n", roots.output, roots.plans, roots.checkpoints, roots.debug)
|
config := fmt.Sprintf("version: 4\noutput:\n directory: %q\ncache:\n chunk_plans:\n directory: %q\n mode: auto\n checkpoints:\n enabled: true\n directory: %q\ndebug:\n directory: %q\npipelines:\n sample:\n input: test/input\n chunk: test/chunk\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n output: test/output\n", roots.output, roots.plans, roots.checkpoints, roots.debug)
|
||||||
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
|
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -823,22 +823,22 @@ func (h *stateTestHarness) options() Options {
|
|||||||
if err := pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, stateTestCodec{}); err != nil {
|
if err := pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, stateTestCodec{}); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }); err != nil {
|
if err := registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := registries.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/chunk", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "cache-reference"}}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Chunker, error) { return stateTestChunker{h}, nil }); err != nil {
|
if err := registries.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "cache-reference"}}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Chunker, error) { return stateTestChunker{h}, nil }); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "test/extract", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{h}, nil }); err != nil {
|
if err := pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "test/extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{h}, nil }); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "test/merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{harness: h}, nil }); err != nil {
|
if err := pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "test/merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{harness: h}, nil }); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "test/normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{harness: h}, nil }); err != nil {
|
if err := pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "test/normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{harness: h}, nil }); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
||||||
return stateTestOutput{harness: h, includeWarnings: h.includeWarnings}, nil
|
return stateTestOutput{harness: h, includeWarnings: h.includeWarnings}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
@@ -848,7 +848,7 @@ func (h *stateTestHarness) options() Options {
|
|||||||
defer h.mu.Unlock()
|
defer h.mu.Unlock()
|
||||||
h.runIDCalls++
|
h.runIDCalls++
|
||||||
return fmt.Sprintf("run-%d-%032x", startedAt.UnixNano(), h.runIDCalls), nil
|
return fmt.Sprintf("run-%d-%032x", startedAt.UnixNano(), h.runIDCalls), nil
|
||||||
}, UserCacheDir: func() (string, error) { return "", errors.New("unexpected user cache lookup") }, LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
}, UserCacheDir: func() (string, error) { return "", errors.New("unexpected user cache lookup") }, LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
return nil, nil, nil
|
return nil, nil, nil
|
||||||
}}
|
}}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package artifacts
|
package artifacts
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -26,9 +27,32 @@ type ValidatorManifest struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type LLMProfileManifest struct {
|
type LLMProfileManifest struct {
|
||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
Provider string `json:"provider,omitempty"`
|
Provider string `json:"provider,omitempty"`
|
||||||
Model string `json:"model,omitempty"`
|
Model string `json:"model,omitempty"`
|
||||||
|
BackendID string `json:"backend_id,omitempty"`
|
||||||
|
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Normalized returns the canonical representation used for manifest identity
|
||||||
|
// and publication.
|
||||||
|
func (profile LLMProfileManifest) Normalized() LLMProfileManifest {
|
||||||
|
profile.ID = strings.TrimSpace(profile.ID)
|
||||||
|
profile.Provider = strings.TrimSpace(profile.Provider)
|
||||||
|
profile.Model = strings.TrimSpace(profile.Model)
|
||||||
|
profile.BackendID = strings.TrimSpace(profile.BackendID)
|
||||||
|
profile.ReasoningEffort = strings.TrimSpace(profile.ReasoningEffort)
|
||||||
|
return profile
|
||||||
|
}
|
||||||
|
|
||||||
|
// IdentityKey returns an opaque, deterministic key for the effective profile.
|
||||||
|
func (profile LLMProfileManifest) IdentityKey() string {
|
||||||
|
profile = profile.Normalized()
|
||||||
|
return profile.ID + "\x00" +
|
||||||
|
profile.Provider + "\x00" +
|
||||||
|
profile.Model + "\x00" +
|
||||||
|
profile.BackendID + "\x00" +
|
||||||
|
profile.ReasoningEffort
|
||||||
}
|
}
|
||||||
|
|
||||||
type ReferenceProvenance struct {
|
type ReferenceProvenance struct {
|
||||||
|
|||||||
@@ -53,7 +53,13 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
|
|||||||
PipelineID: "pipeline-1",
|
PipelineID: "pipeline-1",
|
||||||
PipelineDigest: "sha256:abc123",
|
PipelineDigest: "sha256:abc123",
|
||||||
LLMProfiles: []LLMProfileManifest{
|
LLMProfiles: []LLMProfileManifest{
|
||||||
{ID: "default", Provider: "scriptorium", Model: "model-a"},
|
{
|
||||||
|
ID: "default",
|
||||||
|
Provider: "promptkit",
|
||||||
|
Model: "model-a",
|
||||||
|
BackendID: "openrouter",
|
||||||
|
ReasoningEffort: "high",
|
||||||
|
},
|
||||||
},
|
},
|
||||||
ArtifactLanes: []ArtifactLaneManifest{
|
ArtifactLanes: []ArtifactLaneManifest{
|
||||||
{
|
{
|
||||||
@@ -101,7 +107,13 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
|
|||||||
if !ok {
|
if !ok {
|
||||||
t.Fatalf("llm_profiles[0] = %#v, want object", profiles[0])
|
t.Fatalf("llm_profiles[0] = %#v, want object", profiles[0])
|
||||||
}
|
}
|
||||||
assertHasKeys(t, profile, "id", "provider", "model")
|
assertHasKeys(t, profile, "id", "provider", "model", "backend_id", "reasoning_effort")
|
||||||
|
if profile["provider"] != "promptkit" {
|
||||||
|
t.Fatalf("llm_profiles[0].provider = %#v, want promptkit", profile["provider"])
|
||||||
|
}
|
||||||
|
if profile["backend_id"] != "openrouter" || profile["reasoning_effort"] != "high" {
|
||||||
|
t.Fatalf("llm_profiles[0] = %#v, want backend and reasoning provenance", profile)
|
||||||
|
}
|
||||||
|
|
||||||
lanes, ok := got["artifact_lanes"].([]any)
|
lanes, ok := got["artifact_lanes"].([]any)
|
||||||
if !ok {
|
if !ok {
|
||||||
|
|||||||
@@ -4,10 +4,10 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
)
|
)
|
||||||
|
|
||||||
const SupportedFileConfigVersion = 3
|
const SupportedFileConfigVersion = 4
|
||||||
|
|
||||||
type Config struct {
|
type Config struct {
|
||||||
Scriptorium ScriptoriumConfig `json:"scriptorium,omitempty"`
|
PromptKit PromptKitConfig `json:"promptkit,omitempty"`
|
||||||
Pipelines map[string]pipeline.PipelineProfile `json:"pipelines"`
|
Pipelines map[string]pipeline.PipelineProfile `json:"pipelines"`
|
||||||
Concurrency ConcurrencyConfig `json:"concurrency"`
|
Concurrency ConcurrencyConfig `json:"concurrency"`
|
||||||
Output OutputConfig `json:"output"`
|
Output OutputConfig `json:"output"`
|
||||||
@@ -15,9 +15,15 @@ type Config struct {
|
|||||||
Debug DebugConfig `json:"debug"`
|
Debug DebugConfig `json:"debug"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type ScriptoriumConfig struct {
|
type PromptKitConfig struct {
|
||||||
ProfileDir string `json:"profile_dir,omitempty"`
|
ProfileDir string `json:"profile_dir,omitempty"`
|
||||||
ProfileFile string `json:"profile_file,omitempty"`
|
ProfileFile string `json:"profile_file,omitempty"`
|
||||||
|
LocalBackend *PromptKitLocalBackendConfig `json:"local_backend,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitLocalBackendConfig struct {
|
||||||
|
Endpoint string `json:"endpoint"`
|
||||||
|
ConcurrencyLimit int `json:"concurrency_limit"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type ConcurrencyConfig struct {
|
type ConcurrencyConfig struct {
|
||||||
@@ -66,6 +72,10 @@ func Default() Config {
|
|||||||
|
|
||||||
func cloneConfig(in Config) Config {
|
func cloneConfig(in Config) Config {
|
||||||
out := in
|
out := in
|
||||||
|
if in.PromptKit.LocalBackend != nil {
|
||||||
|
localBackend := *in.PromptKit.LocalBackend
|
||||||
|
out.PromptKit.LocalBackend = &localBackend
|
||||||
|
}
|
||||||
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
|
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
|
||||||
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
|
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
|
||||||
for key, profile := range in.Pipelines {
|
for key, profile := range in.Pipelines {
|
||||||
|
|||||||
@@ -42,12 +42,10 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
|
|||||||
}
|
}
|
||||||
profile = clonePipelineProfile(profile)
|
profile = clonePipelineProfile(profile)
|
||||||
profile.ID = pipelineID
|
profile.ID = pipelineID
|
||||||
if override := strings.TrimSpace(input.LLMProfileOverride); override != "" {
|
|
||||||
applyLLMProfileOverride(&profile, override)
|
|
||||||
}
|
|
||||||
|
|
||||||
resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{
|
resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{
|
||||||
Only: input.Only,
|
Only: input.Only,
|
||||||
|
LLMProfileOverride: input.LLMProfileOverride,
|
||||||
ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...),
|
ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...),
|
||||||
ReferenceUnbinds: append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...),
|
ReferenceUnbinds: append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...),
|
||||||
}, input.Catalog)
|
}, input.Catalog)
|
||||||
@@ -65,22 +63,6 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func applyLLMProfileOverride(profile *pipeline.PipelineProfile, profileID string) {
|
|
||||||
profile.Chunk.LLMProfile = profileID
|
|
||||||
apply := func(artifacts map[string]pipeline.ArtifactLaneProfile) {
|
|
||||||
for laneID, lane := range artifacts {
|
|
||||||
lane.Extract.LLMProfile = profileID
|
|
||||||
lane.Merge.LLMProfile = profileID
|
|
||||||
lane.Normalize.LLMProfile = profileID
|
|
||||||
artifacts[laneID] = lane
|
|
||||||
}
|
|
||||||
}
|
|
||||||
apply(profile.Artifacts)
|
|
||||||
for index := range profile.Steps {
|
|
||||||
apply(profile.Steps[index].Artifacts)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func lookupPipelineProfile(profiles map[string]pipeline.PipelineProfile, pipelineID string) (pipeline.PipelineProfile, bool) {
|
func lookupPipelineProfile(profiles map[string]pipeline.PipelineProfile, pipelineID string) (pipeline.PipelineProfile, bool) {
|
||||||
pipelineID = strings.TrimSpace(pipelineID)
|
pipelineID = strings.TrimSpace(pipelineID)
|
||||||
for rawID, profile := range profiles {
|
for rawID, profile := range profiles {
|
||||||
|
|||||||
@@ -84,6 +84,56 @@ func TestEffectiveConfigMaterializesDefaultBindingsThroughCatalog(t *testing.T)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestEffectiveConfigPreservesPromptKitProfileSource(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
profileSource PromptKitConfig
|
||||||
|
}{
|
||||||
|
{name: "profile directory", profileSource: PromptKitConfig{ProfileDir: "./profiles"}},
|
||||||
|
{name: "profile file", profileSource: PromptKitConfig{ProfileFile: "./profiles.yml"}},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := configForEffectiveTests(t, effectiveProfile())
|
||||||
|
cfg.PromptKit = tt.profileSource
|
||||||
|
effective, err := cfg.Resolve(ResolveInput{PipelineID: "main", Catalog: effectiveCatalog(t)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v", err)
|
||||||
|
}
|
||||||
|
if effective.Config.PromptKit != cfg.PromptKit {
|
||||||
|
t.Fatalf("effective PromptKit config = %#v, want %#v", effective.Config.PromptKit, cfg.PromptKit)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEffectiveConfigOwnsPromptKitLocalBackend(t *testing.T) {
|
||||||
|
cfg := configForEffectiveTests(t, effectiveProfile())
|
||||||
|
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: "http://localhost:8000/v1",
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
effective, err := cfg.Resolve(ResolveInput{PipelineID: "main", Catalog: effectiveCatalog(t)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v", err)
|
||||||
|
}
|
||||||
|
if effective.Config.PromptKit.LocalBackend == nil {
|
||||||
|
t.Fatal("effective local backend = nil")
|
||||||
|
}
|
||||||
|
if effective.Config.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend {
|
||||||
|
t.Fatal("effective local backend aliases input config")
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg.PromptKit.LocalBackend.Endpoint = "http://changed-input.example/v1"
|
||||||
|
if effective.Config.PromptKit.LocalBackend.Endpoint != "http://localhost:8000/v1" {
|
||||||
|
t.Fatalf("input mutation changed effective config: %#v", effective.Config.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
effective.Config.PromptKit.LocalBackend.ConcurrencyLimit = 9
|
||||||
|
if cfg.PromptKit.LocalBackend.ConcurrencyLimit != 2 {
|
||||||
|
t.Fatalf("effective mutation changed input config: %#v", cfg.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
|
func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -154,7 +204,7 @@ func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestEffectiveConfigLLMProfileOverrideChangesDigestWithoutOverridingValidators(t *testing.T) {
|
func TestEffectiveConfigLLMProfileOverrideChangesDigestAndOverridesValidators(t *testing.T) {
|
||||||
profile := effectiveProfile()
|
profile := effectiveProfile()
|
||||||
profile.Chunk.LLMProfile = "chunk-profile"
|
profile.Chunk.LLMProfile = "chunk-profile"
|
||||||
lane := profile.Artifacts["lane"]
|
lane := profile.Artifacts["lane"]
|
||||||
@@ -187,8 +237,27 @@ func TestEffectiveConfigLLMProfileOverrideChangesDigestWithoutOverridingValidato
|
|||||||
t.Fatalf("pipeline profile override was not applied: %#v", resolved)
|
t.Fatalf("pipeline profile override was not applied: %#v", resolved)
|
||||||
}
|
}
|
||||||
validators := findEffectiveValidatorChain(resolved, pipeline.StageExtract, "lane")
|
validators := findEffectiveValidatorChain(resolved, pipeline.StageExtract, "lane")
|
||||||
if len(validators.Validators) != 1 || validators.Validators[0].Binding.LLMProfile != "validator-profile" {
|
if len(validators.Validators) != 1 || validators.Validators[0].Binding.LLMProfile != "override-profile" {
|
||||||
t.Fatalf("validator profile was overridden: %#v", validators)
|
t.Fatalf("validator profile = %#v, want runtime override", validators)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEffectiveConfigPipelineLLMProfileIsInheritedWithoutMutatingConfig(t *testing.T) {
|
||||||
|
profile := effectiveProfile()
|
||||||
|
profile.LLMProfile = " configured-profile "
|
||||||
|
effective, err := resolveEffectiveProfile(t, profile, ResolveInput{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v", err)
|
||||||
|
}
|
||||||
|
if got := effective.Config.Pipelines["main"].LLMProfile; got != " configured-profile " {
|
||||||
|
t.Fatalf("effective config pipeline llm profile = %q, want preserved programmatic value", got)
|
||||||
|
}
|
||||||
|
resolved := effective.ResolvedPipeline
|
||||||
|
if got := resolved.Chunk.LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("resolved chunk profile = %q, want inherited profile", got)
|
||||||
|
}
|
||||||
|
if got := resolved.Steps[0].ArtifactLanes[0].Extract.LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("resolved extract profile = %q, want inherited profile", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -435,7 +504,7 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
|
|||||||
if err := pipeline.RegisterArtifactCodec(catalog.ArtifactCodecs, effectiveCodec{}); err != nil {
|
if err := pipeline.RegisterArtifactCodec(catalog.ArtifactCodecs, effectiveCodec{}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := catalog.Inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
|
if err := catalog.Inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
|
||||||
return effectiveInput{key: "input"}, nil
|
return effectiveInput{key: "input"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
@@ -443,6 +512,7 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
|
|||||||
chunkSpec := pipeline.ModuleSpec{
|
chunkSpec := pipeline.ModuleSpec{
|
||||||
Key: pipeline.DefaultChunkModule,
|
Key: pipeline.DefaultChunkModule,
|
||||||
Stage: pipeline.StageChunk,
|
Stage: pipeline.StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: []string{"source"},
|
Requires: []string{"source"},
|
||||||
Provides: []string{"chunk"},
|
Provides: []string{"chunk"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-ref"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-ref"}},
|
||||||
@@ -453,32 +523,32 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := catalog.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "needs-capability", Stage: pipeline.StageChunk, Requires: []string{"missing"}}, chunkOptions, func(pipeline.BuildRequest) (contracts.Chunker, error) {
|
if err := catalog.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "needs-capability", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}}, chunkOptions, func(pipeline.BuildRequest) (contracts.Chunker, error) {
|
||||||
return effectiveChunker{key: "needs-capability"}, nil
|
return effectiveChunker{key: "needs-capability"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterExtractor(catalog.Extractors, pipeline.ModuleSpec{Key: "extract", Stage: pipeline.StageExtract, ArtifactKind: effectiveArtifactKind, Requires: []string{"chunk"}, Provides: []string{"candidate"}}, func() (contracts.Extractor[effectiveArtifact], error) {
|
if err := pipeline.RegisterExtractor(catalog.Extractors, pipeline.ModuleSpec{Key: "extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"chunk"}, Provides: []string{"candidate"}}, func() (contracts.Extractor[effectiveArtifact], error) {
|
||||||
return effectiveExtractor{key: "extract"}, nil
|
return effectiveExtractor{key: "extract"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: pipeline.DefaultMergeModule, Stage: pipeline.StageMerge, ArtifactKind: effectiveArtifactKind, Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: pipeline.DefaultMergeModule, Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
||||||
return effectiveMerger{key: pipeline.DefaultMergeModule}, nil
|
return effectiveMerger{key: pipeline.DefaultMergeModule}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: "other-merge", Stage: pipeline.StageMerge, ArtifactKind: "other-kind", Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: "other-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "other-kind", Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
||||||
return effectiveMerger{key: "other-merge"}, nil
|
return effectiveMerger{key: "other-merge"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterNormalizer(catalog.Normalizers, pipeline.ModuleSpec{Key: pipeline.DefaultNormalizeModule, Stage: pipeline.StageNormalize, ArtifactKind: effectiveArtifactKind, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer[effectiveArtifact], error) {
|
if err := pipeline.RegisterNormalizer(catalog.Normalizers, pipeline.ModuleSpec{Key: pipeline.DefaultNormalizeModule, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer[effectiveArtifact], error) {
|
||||||
return effectiveNormalizer{key: pipeline.DefaultNormalizeModule}, nil
|
return effectiveNormalizer{key: pipeline.DefaultNormalizeModule}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := catalog.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: pipeline.DefaultOutputModule, Stage: pipeline.StageOutput, Requires: []string{"normalized"}}, func() (contracts.OutputEncoder, error) {
|
if err := catalog.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: pipeline.DefaultOutputModule, Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}}, func() (contracts.OutputEncoder, error) {
|
||||||
return effectiveOutput{key: pipeline.DefaultOutputModule}, nil
|
return effectiveOutput{key: pipeline.DefaultOutputModule}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
func TestPrecedenceFileValuesOverrideBuiltInDefaults(t *testing.T) {
|
func TestPrecedenceFileValuesOverrideBuiltInDefaults(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 4
|
total_llm: 4
|
||||||
stage_workers:
|
stage_workers:
|
||||||
@@ -35,7 +35,7 @@ debug:
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestPrecedenceOperationalEnvironmentOverridesFileValues(t *testing.T) {
|
func TestPrecedenceOperationalEnvironmentOverridesFileValues(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 2
|
total_llm: 2
|
||||||
stage_workers:
|
stage_workers:
|
||||||
@@ -81,21 +81,21 @@ func TestPrecedenceExtractWorkersFollowEffectiveConcurrencyUnlessExplicit(t *tes
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "default follows environment total",
|
name: "default follows environment total",
|
||||||
file: "version: 3\n",
|
file: "version: 4\n",
|
||||||
env: map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "5"},
|
env: map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "5"},
|
||||||
wantTotal: 5,
|
wantTotal: 5,
|
||||||
wantWorker: 5,
|
wantWorker: 5,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "file worker is retained",
|
name: "file worker is retained",
|
||||||
file: "version: 3\nconcurrency:\n total_llm: 3\n stage_workers:\n extract: 2\n",
|
file: "version: 4\nconcurrency:\n total_llm: 3\n stage_workers:\n extract: 2\n",
|
||||||
env: map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6"},
|
env: map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6"},
|
||||||
wantTotal: 6,
|
wantTotal: 6,
|
||||||
wantWorker: 2,
|
wantWorker: 2,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "environment worker is retained",
|
name: "environment worker is retained",
|
||||||
file: "version: 3\nconcurrency:\n total_llm: 2\n",
|
file: "version: 4\nconcurrency:\n total_llm: 2\n",
|
||||||
env: map[string]string{
|
env: map[string]string{
|
||||||
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6",
|
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6",
|
||||||
"NOTARIUS_STAGE_WORKERS_EXTRACT": "4",
|
"NOTARIUS_STAGE_WORKERS_EXTRACT": "4",
|
||||||
@@ -118,7 +118,7 @@ func TestPrecedenceExtractWorkersFollowEffectiveConcurrencyUnlessExplicit(t *tes
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestPrecedenceEmptyFileCacheDirectoriesDeferPerUserResolution(t *testing.T) {
|
func TestPrecedenceEmptyFileCacheDirectoriesDeferPerUserResolution(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
cache:
|
cache:
|
||||||
chunk_plans:
|
chunk_plans:
|
||||||
directory: ""
|
directory: ""
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ import (
|
|||||||
|
|
||||||
type FileConfig struct {
|
type FileConfig struct {
|
||||||
Version int `yaml:"version"`
|
Version int `yaml:"version"`
|
||||||
Scriptorium *FileScriptoriumConfig `yaml:"scriptorium,omitempty"`
|
PromptKit *FilePromptKitConfig `yaml:"promptkit,omitempty"`
|
||||||
Pipelines map[string]FilePipelineProfile `yaml:"pipelines,omitempty"`
|
Pipelines map[string]FilePipelineProfile `yaml:"pipelines,omitempty"`
|
||||||
Concurrency *FileConcurrencyConfig `yaml:"concurrency,omitempty"`
|
Concurrency *FileConcurrencyConfig `yaml:"concurrency,omitempty"`
|
||||||
Output *FileOutputConfig `yaml:"output,omitempty"`
|
Output *FileOutputConfig `yaml:"output,omitempty"`
|
||||||
@@ -22,27 +22,35 @@ type FileConfig struct {
|
|||||||
Debug *FileDebugConfig `yaml:"debug,omitempty"`
|
Debug *FileDebugConfig `yaml:"debug,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type FileScriptoriumConfig struct {
|
type FilePromptKitConfig struct {
|
||||||
ProfileDir *string `yaml:"profile_dir,omitempty"`
|
ProfileDir *string `yaml:"profile_dir,omitempty"`
|
||||||
ProfileFile *string `yaml:"profile_file,omitempty"`
|
ProfileFile *string `yaml:"profile_file,omitempty"`
|
||||||
|
LocalBackend *FilePromptKitLocalBackendConfig `yaml:"local_backend,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type FilePromptKitLocalBackendConfig struct {
|
||||||
|
Endpoint *string `yaml:"endpoint,omitempty"`
|
||||||
|
ConcurrencyLimit *int `yaml:"concurrency_limit,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type FilePipelineProfile struct {
|
type FilePipelineProfile struct {
|
||||||
Input fileModuleBinding `yaml:"input"`
|
LLMProfile *string `yaml:"llm_profile,omitempty"`
|
||||||
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
|
Input fileModuleBinding `yaml:"input"`
|
||||||
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
|
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
|
||||||
Steps []FilePipelineStepProfile `yaml:"steps,omitempty"`
|
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
|
||||||
Output *fileModuleBinding `yaml:"output,omitempty"`
|
Steps []FilePipelineStepProfile `yaml:"steps,omitempty"`
|
||||||
References map[string]fileReferenceSource `yaml:"references,omitempty"`
|
Output *fileModuleBinding `yaml:"output,omitempty"`
|
||||||
artifactsSet bool `yaml:"-"`
|
References map[string]fileReferenceSource `yaml:"references,omitempty"`
|
||||||
stepsSet bool `yaml:"-"`
|
artifactsSet bool `yaml:"-"`
|
||||||
|
stepsSet bool `yaml:"-"`
|
||||||
|
llmProfileSet bool `yaml:"-"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
|
func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
|
||||||
type plainFilePipelineProfile FilePipelineProfile
|
type plainFilePipelineProfile FilePipelineProfile
|
||||||
var decoded plainFilePipelineProfile
|
var decoded plainFilePipelineProfile
|
||||||
seen, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
|
seen, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
|
||||||
"input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
|
"llm_profile": {}, "input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
|
||||||
}, "pipeline profile")
|
}, "pipeline profile")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -50,6 +58,7 @@ func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
|
|||||||
*p = FilePipelineProfile(decoded)
|
*p = FilePipelineProfile(decoded)
|
||||||
_, p.artifactsSet = seen["artifacts"]
|
_, p.artifactsSet = seen["artifacts"]
|
||||||
_, p.stepsSet = seen["steps"]
|
_, p.stepsSet = seen["steps"]
|
||||||
|
_, p.llmProfileSet = seen["llm_profile"]
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -251,6 +260,9 @@ func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
b.LLMProfile = strings.TrimSpace(llmProfile)
|
b.LLMProfile = strings.TrimSpace(llmProfile)
|
||||||
|
if b.LLMProfile == "" {
|
||||||
|
return fmt.Errorf("llm_profile must not be empty when set")
|
||||||
|
}
|
||||||
case "retries":
|
case "retries":
|
||||||
var retries int
|
var retries int
|
||||||
if err := valueNode.Decode(&retries); err != nil {
|
if err := valueNode.Decode(&retries); err != nil {
|
||||||
@@ -325,6 +337,9 @@ func ParseFileConfigYAML(data []byte) (FileConfig, error) {
|
|||||||
if header.Version == 2 {
|
if header.Version == 2 {
|
||||||
return FileConfig{}, fmt.Errorf("config version 2 is no longer supported; migrate the file using the version 2-to-3 migration in docs/config.md")
|
return FileConfig{}, fmt.Errorf("config version 2 is no longer supported; migrate the file using the version 2-to-3 migration in docs/config.md")
|
||||||
}
|
}
|
||||||
|
if header.Version == 3 {
|
||||||
|
return FileConfig{}, fmt.Errorf("config version 3 is no longer supported; change \"version: 3\" to \"version: 4\" and rename \"scriptorium:\" to \"promptkit:\"")
|
||||||
|
}
|
||||||
if header.Version != SupportedFileConfigVersion {
|
if header.Version != SupportedFileConfigVersion {
|
||||||
return FileConfig{}, fmt.Errorf("unsupported config version %d (supported version is %d)", header.Version, SupportedFileConfigVersion)
|
return FileConfig{}, fmt.Errorf("unsupported config version %d (supported version is %d)", header.Version, SupportedFileConfigVersion)
|
||||||
}
|
}
|
||||||
@@ -450,25 +465,46 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if fileCfg.Scriptorium != nil {
|
if fileCfg.PromptKit != nil {
|
||||||
if fileCfg.Scriptorium.ProfileDir != nil {
|
if fileCfg.PromptKit.ProfileDir != nil {
|
||||||
value := strings.TrimSpace(*fileCfg.Scriptorium.ProfileDir)
|
value := strings.TrimSpace(*fileCfg.PromptKit.ProfileDir)
|
||||||
if value == "" {
|
if value == "" {
|
||||||
return fmt.Errorf("scriptorium.profile_dir must not be empty when set")
|
return fmt.Errorf("promptkit.profile_dir must not be empty when set")
|
||||||
}
|
}
|
||||||
c.Scriptorium.ProfileDir = value
|
c.PromptKit.ProfileDir = value
|
||||||
}
|
}
|
||||||
if fileCfg.Scriptorium.ProfileFile != nil {
|
if fileCfg.PromptKit.ProfileFile != nil {
|
||||||
value := strings.TrimSpace(*fileCfg.Scriptorium.ProfileFile)
|
value := strings.TrimSpace(*fileCfg.PromptKit.ProfileFile)
|
||||||
if value == "" {
|
if value == "" {
|
||||||
return fmt.Errorf("scriptorium.profile_file must not be empty when set")
|
return fmt.Errorf("promptkit.profile_file must not be empty when set")
|
||||||
}
|
}
|
||||||
c.Scriptorium.ProfileFile = value
|
c.PromptKit.ProfileFile = value
|
||||||
|
}
|
||||||
|
if fileCfg.PromptKit.LocalBackend != nil {
|
||||||
|
if fileCfg.PromptKit.LocalBackend.Endpoint == nil {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
|
||||||
|
}
|
||||||
|
endpoint := strings.TrimSpace(*fileCfg.PromptKit.LocalBackend.Endpoint)
|
||||||
|
if endpoint == "" {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
|
||||||
|
}
|
||||||
|
localBackend := PromptKitLocalBackendConfig{Endpoint: endpoint}
|
||||||
|
if fileCfg.PromptKit.LocalBackend.ConcurrencyLimit != nil {
|
||||||
|
localBackend.ConcurrencyLimit = *fileCfg.PromptKit.LocalBackend.ConcurrencyLimit
|
||||||
|
}
|
||||||
|
c.PromptKit.LocalBackend = &localBackend
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, pipelineID := range pipelineIDs {
|
for _, pipelineID := range pipelineIDs {
|
||||||
filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]]
|
filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]]
|
||||||
|
llmProfile := ""
|
||||||
|
if filePipeline.llmProfileSet || filePipeline.LLMProfile != nil {
|
||||||
|
if filePipeline.LLMProfile == nil || strings.TrimSpace(*filePipeline.LLMProfile) == "" {
|
||||||
|
return fmt.Errorf("pipeline %q llm_profile must not be empty when set", pipelineID)
|
||||||
|
}
|
||||||
|
llmProfile = strings.TrimSpace(*filePipeline.LLMProfile)
|
||||||
|
}
|
||||||
hasSteps := filePipeline.stepsSet || filePipeline.Steps != nil
|
hasSteps := filePipeline.stepsSet || filePipeline.Steps != nil
|
||||||
laneIDs, rawLaneIDs, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID))
|
laneIDs, rawLaneIDs, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -476,6 +512,7 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
|||||||
}
|
}
|
||||||
profile := pipeline.PipelineProfile{
|
profile := pipeline.PipelineProfile{
|
||||||
ID: pipelineID,
|
ID: pipelineID,
|
||||||
|
LLMProfile: llmProfile,
|
||||||
Input: filePipeline.Input.toPipelineBinding(),
|
Input: filePipeline.Input.toPipelineBinding(),
|
||||||
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
|
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
|
||||||
References: fileReferenceSourcesToPipeline(filePipeline.References),
|
References: fileReferenceSourcesToPipeline(filePipeline.References),
|
||||||
|
|||||||
@@ -1,6 +1,8 @@
|
|||||||
package config
|
package config
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"reflect"
|
"reflect"
|
||||||
@@ -34,8 +36,8 @@ func TestDefaultReturnsDocumentedValuesAndIndependentMaps(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigMinimalVersion3AppliesOverDefaults(t *testing.T) {
|
func TestFileConfigMinimalVersion4AppliesOverDefaults(t *testing.T) {
|
||||||
file := parseFileConfig(t, "version: 3\n")
|
file := parseFileConfig(t, "version: 4\n")
|
||||||
cfg := Default()
|
cfg := Default()
|
||||||
if err := cfg.ApplyFileConfig(file); err != nil {
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
@@ -48,6 +50,259 @@ func TestFileConfigMinimalVersion3AppliesOverDefaults(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFilePipelineLLMProfileIsPresenceAwareAndDetached(t *testing.T) {
|
||||||
|
const pipelineYAML = `version: 4
|
||||||
|
pipelines:
|
||||||
|
main:
|
||||||
|
%s
|
||||||
|
input: input
|
||||||
|
artifacts:
|
||||||
|
lane:
|
||||||
|
extract: extract
|
||||||
|
`
|
||||||
|
|
||||||
|
t.Run("omitted", func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, ""))
|
||||||
|
if file.Pipelines["main"].LLMProfile != nil || file.Pipelines["main"].llmProfileSet {
|
||||||
|
t.Fatalf("parsed pipeline profile = %#v, want omitted llm profile", file.Pipelines["main"])
|
||||||
|
}
|
||||||
|
cfg := Default()
|
||||||
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got := cfg.Pipelines["main"].LLMProfile; got != "" {
|
||||||
|
t.Fatalf("pipeline llm profile = %q, want empty", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("trimmed and detached", func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, "llm_profile: ' configured-profile '"))
|
||||||
|
cfg := Default()
|
||||||
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got := cfg.Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("pipeline llm profile = %q, want trimmed value", got)
|
||||||
|
}
|
||||||
|
*file.Pipelines["main"].LLMProfile = "changed-profile"
|
||||||
|
if got := cfg.Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("effective config aliases parsed file: %q", got)
|
||||||
|
}
|
||||||
|
if got := cloneConfig(cfg).Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("cloned pipeline llm profile = %q", got)
|
||||||
|
}
|
||||||
|
data, err := json.Marshal(cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var roundTripped Config
|
||||||
|
if err := json.Unmarshal(data, &roundTripped); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got := roundTripped.Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("round-tripped pipeline llm profile = %q", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
for _, value := range []string{"''", "' '", "null"} {
|
||||||
|
t.Run("explicit empty "+value, func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, "llm_profile: "+value))
|
||||||
|
cfg := Default()
|
||||||
|
err := cfg.ApplyFileConfig(file)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), `pipeline "main" llm_profile must not be empty`) {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v, want explicit-empty rejection", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFileModuleBindingRejectsExplicitEmptyLLMProfile(t *testing.T) {
|
||||||
|
const configYAML = `version: 4
|
||||||
|
pipelines:
|
||||||
|
main:
|
||||||
|
input:
|
||||||
|
module: input
|
||||||
|
llm_profile: %s
|
||||||
|
artifacts:
|
||||||
|
lane:
|
||||||
|
extract: extract
|
||||||
|
`
|
||||||
|
for _, value := range []string{"''", "' '", "null"} {
|
||||||
|
t.Run(value, func(t *testing.T) {
|
||||||
|
_, err := ParseFileConfigYAML([]byte(fmt.Sprintf(configYAML, value)))
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "llm_profile must not be empty when set") {
|
||||||
|
t.Fatalf("ParseFileConfigYAML() error = %v, want explicit-empty binding profile rejection", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitProfileSourcesSurviveConfigBoundaries(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
yaml string
|
||||||
|
want PromptKitConfig
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "profile directory",
|
||||||
|
yaml: "version: 4\npromptkit:\n profile_dir: ' ./profiles '\n",
|
||||||
|
want: PromptKitConfig{ProfileDir: "./profiles"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "profile file",
|
||||||
|
yaml: "version: 4\npromptkit:\n profile_file: ' ./profiles.yml '\n",
|
||||||
|
want: PromptKitConfig{ProfileFile: "./profiles.yml"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := applyFileConfig(t, tt.yaml)
|
||||||
|
if cfg.PromptKit != tt.want {
|
||||||
|
t.Fatalf("PromptKit config = %#v, want %#v", cfg.PromptKit, tt.want)
|
||||||
|
}
|
||||||
|
if got := cloneConfig(cfg).PromptKit; got != tt.want {
|
||||||
|
t.Fatalf("cloned PromptKit config = %#v, want %#v", got, tt.want)
|
||||||
|
}
|
||||||
|
if got := cfg.Redacted().PromptKit; got != tt.want {
|
||||||
|
t.Fatalf("redacted PromptKit config = %#v, want %#v", got, tt.want)
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := json.Marshal(cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("json.Marshal() error = %v", err)
|
||||||
|
}
|
||||||
|
var payload map[string]json.RawMessage
|
||||||
|
if err := json.Unmarshal(data, &payload); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal() error = %v", err)
|
||||||
|
}
|
||||||
|
if _, ok := payload["promptkit"]; !ok {
|
||||||
|
t.Fatalf("runtime JSON keys = %v, want promptkit", payload)
|
||||||
|
}
|
||||||
|
if _, ok := payload["scriptorium"]; ok {
|
||||||
|
t.Fatalf("runtime JSON keys = %v, must not contain removed section", payload)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitLocalBackendSurvivesConfigBoundaries(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
concurrencyYAML string
|
||||||
|
wantConcurrency int
|
||||||
|
}{
|
||||||
|
{name: "omitted concurrency defaults to zero"},
|
||||||
|
{name: "positive concurrency is preserved", concurrencyYAML: " concurrency_limit: 2\n", wantConcurrency: 2},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, "version: 4\npromptkit:\n local_backend:\n endpoint: ' http://localhost:8000/v1 '\n"+tt.concurrencyYAML)
|
||||||
|
cfg := Default()
|
||||||
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v", err)
|
||||||
|
}
|
||||||
|
want := PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: "http://localhost:8000/v1",
|
||||||
|
ConcurrencyLimit: tt.wantConcurrency,
|
||||||
|
}
|
||||||
|
if cfg.PromptKit.LocalBackend == nil || *cfg.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("local backend config = %#v, want %#v", cfg.PromptKit.LocalBackend, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
*file.PromptKit.LocalBackend.Endpoint = "http://changed.example/v1"
|
||||||
|
if file.PromptKit.LocalBackend.ConcurrencyLimit != nil {
|
||||||
|
*file.PromptKit.LocalBackend.ConcurrencyLimit = 99
|
||||||
|
}
|
||||||
|
if *cfg.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("effective config aliases parsed file model: %#v", cfg.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
|
||||||
|
cloned := cloneConfig(cfg)
|
||||||
|
if cloned.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *cloned.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("cloned local backend = %#v, want detached %#v", cloned.PromptKit.LocalBackend, want)
|
||||||
|
}
|
||||||
|
cloned.PromptKit.LocalBackend.Endpoint = "http://clone.example/v1"
|
||||||
|
if *cfg.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("mutating clone changed source config: %#v", cfg.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
|
||||||
|
redacted := cfg.Redacted()
|
||||||
|
if redacted.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *redacted.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("redacted local backend = %#v, want detached %#v", redacted.PromptKit.LocalBackend, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := json.Marshal(cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("json.Marshal() error = %v", err)
|
||||||
|
}
|
||||||
|
var payload struct {
|
||||||
|
PromptKit map[string]json.RawMessage `json:"promptkit"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(data, &payload); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal() error = %v", err)
|
||||||
|
}
|
||||||
|
localJSON, ok := payload.PromptKit["local_backend"]
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("runtime PromptKit JSON keys = %v, want local_backend", payload.PromptKit)
|
||||||
|
}
|
||||||
|
var localPayload map[string]json.RawMessage
|
||||||
|
if err := json.Unmarshal(localJSON, &localPayload); err != nil {
|
||||||
|
t.Fatalf("unmarshal local_backend JSON: %v", err)
|
||||||
|
}
|
||||||
|
if _, ok := localPayload["endpoint"]; !ok {
|
||||||
|
t.Fatalf("runtime local_backend JSON keys = %v, want endpoint", localPayload)
|
||||||
|
}
|
||||||
|
if _, ok := localPayload["concurrency_limit"]; !ok {
|
||||||
|
t.Fatalf("runtime local_backend JSON keys = %v, want concurrency_limit", localPayload)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitLocalBackendRequiresEndpoint(t *testing.T) {
|
||||||
|
for _, tt := range []struct {
|
||||||
|
name string
|
||||||
|
yaml string
|
||||||
|
}{
|
||||||
|
{name: "missing", yaml: "version: 4\npromptkit:\n local_backend: {}\n"},
|
||||||
|
{name: "empty", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ''\n"},
|
||||||
|
{name: "blank", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ' '\n"},
|
||||||
|
} {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, tt.yaml)
|
||||||
|
cfg := Default()
|
||||||
|
err := cfg.ApplyFileConfig(file)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "promptkit.local_backend.endpoint") {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v, want endpoint field context", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitExplicitEmptyProfileSourcesAreRejected(t *testing.T) {
|
||||||
|
for _, field := range []string{"profile_dir", "profile_file"} {
|
||||||
|
t.Run(field, func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, "version: 4\npromptkit:\n "+field+": ''\n")
|
||||||
|
cfg := Default()
|
||||||
|
err := cfg.ApplyFileConfig(file)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "promptkit."+field+" must not be empty") {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v, want explicit-empty rejection", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitProfileSourcesRemainMutuallyExclusive(t *testing.T) {
|
||||||
|
cfg := applyFileConfig(t, `version: 4
|
||||||
|
promptkit:
|
||||||
|
profile_dir: ./profiles
|
||||||
|
profile_file: ./profiles.yml
|
||||||
|
`)
|
||||||
|
if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "promptkit profile_dir and profile_file are mutually exclusive") {
|
||||||
|
t.Fatalf("Validate() error = %v, want mutually exclusive profile sources", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestFileConfigMissingVersionIsReportedBeforeFieldDecoding(t *testing.T) {
|
func TestFileConfigMissingVersionIsReportedBeforeFieldDecoding(t *testing.T) {
|
||||||
_, err := ParseFileConfigYAML([]byte("workspace:\n directory: /tmp/old\n"))
|
_, err := ParseFileConfigYAML([]byte("workspace:\n directory: /tmp/old\n"))
|
||||||
if err == nil || !strings.Contains(err.Error(), "config version is required") {
|
if err == nil || !strings.Contains(err.Error(), "config version is required") {
|
||||||
@@ -55,6 +310,15 @@ func TestFileConfigMissingVersionIsReportedBeforeFieldDecoding(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFileConfigVersion3ReportsPromptKitMigration(t *testing.T) {
|
||||||
|
_, err := ParseFileConfigYAML([]byte("version: 3\nscriptorium:\n profile_dir: ./profiles\n"))
|
||||||
|
if err == nil ||
|
||||||
|
!strings.Contains(err.Error(), `change "version: 3" to "version: 4"`) ||
|
||||||
|
!strings.Contains(err.Error(), `rename "scriptorium:" to "promptkit:"`) {
|
||||||
|
t.Fatalf("version 3 error = %v, want actionable version and section migration", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -63,14 +327,19 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "removed diagnostics",
|
name: "removed diagnostics",
|
||||||
yaml: "version: 3\ndiagnostics: {}\n",
|
yaml: "version: 4\ndiagnostics: {}\n",
|
||||||
want: "field diagnostics not found",
|
want: "field diagnostics not found",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "removed llm profiles",
|
name: "removed llm profiles",
|
||||||
yaml: "version: 3\nllm_profiles: {}\n",
|
yaml: "version: 4\nllm_profiles: {}\n",
|
||||||
want: "field llm_profiles not found",
|
want: "field llm_profiles not found",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: "removed scriptorium section",
|
||||||
|
yaml: "version: 4\nscriptorium: {}\n",
|
||||||
|
want: "field scriptorium not found",
|
||||||
|
},
|
||||||
{
|
{
|
||||||
name: "version 2 migration",
|
name: "version 2 migration",
|
||||||
yaml: "version: 2\nworkspace:\n directory: /tmp/old\n",
|
yaml: "version: 2\nworkspace:\n directory: /tmp/old\n",
|
||||||
@@ -78,29 +347,34 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "pipeline field",
|
name: "pipeline field",
|
||||||
yaml: "version: 3\npipelines:\n main:\n unknown: true\n",
|
yaml: "version: 4\npipelines:\n main:\n unknown: true\n",
|
||||||
want: "field unknown not found",
|
want: "field unknown not found",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "lane field",
|
name: "lane field",
|
||||||
yaml: "version: 3\npipelines:\n main:\n artifacts:\n spells:\n unknown: true\n",
|
yaml: "version: 4\npipelines:\n main:\n artifacts:\n spells:\n unknown: true\n",
|
||||||
want: "field unknown not found",
|
want: "field unknown not found",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "module binding field",
|
name: "module binding field",
|
||||||
yaml: "version: 3\npipelines:\n main:\n input:\n module: seriatim\n unknown: true\n",
|
yaml: "version: 4\npipelines:\n main:\n input:\n module: seriatim\n unknown: true\n",
|
||||||
want: "field unknown not found in module binding",
|
want: "field unknown not found in module binding",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "checkpoint field",
|
name: "checkpoint field",
|
||||||
yaml: "version: 3\ncache:\n checkpoints:\n unknown: true\n",
|
yaml: "version: 4\ncache:\n checkpoints:\n unknown: true\n",
|
||||||
want: "field unknown not found",
|
want: "field unknown not found",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "checkpoint enabled type",
|
name: "checkpoint enabled type",
|
||||||
yaml: "version: 3\ncache:\n checkpoints:\n enabled: definitely\n",
|
yaml: "version: 4\ncache:\n checkpoints:\n enabled: definitely\n",
|
||||||
want: "cannot unmarshal",
|
want: "cannot unmarshal",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: "local backend field",
|
||||||
|
yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: http://localhost:8000/v1\n unknown: true\n",
|
||||||
|
want: "field unknown not found",
|
||||||
|
},
|
||||||
}
|
}
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
@@ -113,7 +387,7 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigModuleBindingsPreserveFormsAndValidatorPresence(t *testing.T) {
|
func TestFileConfigModuleBindingsPreserveFormsAndValidatorPresence(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
main:
|
main:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
@@ -160,7 +434,7 @@ pipelines:
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigReferencePrecedenceIsRetained(t *testing.T) {
|
func TestFileConfigReferencePrecedenceIsRetained(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
main:
|
main:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
@@ -224,7 +498,7 @@ pipelines:
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigStageLocalValidatorsPreserveOrderAndFields(t *testing.T) {
|
func TestFileConfigStageLocalValidatorsPreserveOrderAndFields(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
main:
|
main:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
@@ -277,8 +551,8 @@ pipelines:
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigStateSectionsApplyIndependently(t *testing.T) {
|
func TestFileConfigStateSectionsApplyIndependently(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
scriptorium:
|
promptkit:
|
||||||
profile_dir: ./profiles
|
profile_dir: ./profiles
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 7
|
total_llm: 7
|
||||||
@@ -294,15 +568,15 @@ cache:
|
|||||||
debug:
|
debug:
|
||||||
directory: ./debug
|
directory: ./debug
|
||||||
`)
|
`)
|
||||||
if cfg.Scriptorium.ProfileDir != "./profiles" || cfg.Scriptorium.ProfileFile != "" {
|
if cfg.PromptKit.ProfileDir != "./profiles" || cfg.PromptKit.ProfileFile != "" {
|
||||||
t.Fatalf("scriptorium = %#v", cfg.Scriptorium)
|
t.Fatalf("promptkit = %#v", cfg.PromptKit)
|
||||||
}
|
}
|
||||||
if cfg.Concurrency.TotalLLM != 7 || cfg.Concurrency.StageWorkers["extract"] != 7 {
|
if cfg.Concurrency.TotalLLM != 7 || cfg.Concurrency.StageWorkers["extract"] != 7 {
|
||||||
t.Fatalf("concurrency = %#v", cfg.Concurrency)
|
t.Fatalf("concurrency = %#v", cfg.Concurrency)
|
||||||
}
|
}
|
||||||
if cfg.Output.Directory != "./output" || cfg.Cache.ChunkPlans.Directory != "plans" || cfg.Cache.ChunkPlans.Mode != pipeline.ChunkCacheBypass ||
|
if cfg.Output.Directory != "./output" || cfg.Cache.ChunkPlans.Directory != "plans" || cfg.Cache.ChunkPlans.Mode != pipeline.ChunkCacheBypass ||
|
||||||
!cfg.Cache.Checkpoints.Enabled || cfg.Cache.Checkpoints.Directory != "checkpoints" || cfg.Debug.Directory != "./debug" {
|
!cfg.Cache.Checkpoints.Enabled || cfg.Cache.Checkpoints.Directory != "checkpoints" || cfg.Debug.Directory != "./debug" {
|
||||||
t.Fatalf("state sections = %#v, %#v, %#v, %#v", cfg.Output, cfg.Cache, cfg.Debug, cfg.Scriptorium)
|
t.Fatalf("state sections = %#v, %#v, %#v, %#v", cfg.Output, cfg.Cache, cfg.Debug, cfg.PromptKit)
|
||||||
}
|
}
|
||||||
if cfg.Output.Directory == cfg.Cache.ChunkPlans.Directory || cfg.Cache.ChunkPlans.Directory == cfg.Cache.Checkpoints.Directory || cfg.Cache.Checkpoints.Directory == cfg.Debug.Directory {
|
if cfg.Output.Directory == cfg.Cache.ChunkPlans.Directory || cfg.Cache.ChunkPlans.Directory == cfg.Cache.Checkpoints.Directory || cfg.Cache.Checkpoints.Directory == cfg.Debug.Directory {
|
||||||
t.Fatal("state roots were coupled")
|
t.Fatal("state roots were coupled")
|
||||||
@@ -310,11 +584,11 @@ debug:
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigCheckpointEnabledCanBeExplicitlyDisabled(t *testing.T) {
|
func TestFileConfigCheckpointEnabledCanBeExplicitlyDisabled(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, "version: 3\ncache:\n checkpoints:\n enabled: true\n")
|
cfg := applyFileConfig(t, "version: 4\ncache:\n checkpoints:\n enabled: true\n")
|
||||||
if !cfg.Cache.Checkpoints.Enabled || !cloneConfig(cfg).Cache.Checkpoints.Enabled {
|
if !cfg.Cache.Checkpoints.Enabled || !cloneConfig(cfg).Cache.Checkpoints.Enabled {
|
||||||
t.Fatalf("enabled checkpoint config was not retained: %#v", cfg.Cache.Checkpoints)
|
t.Fatalf("enabled checkpoint config was not retained: %#v", cfg.Cache.Checkpoints)
|
||||||
}
|
}
|
||||||
file := parseFileConfig(t, "version: 3\ncache:\n checkpoints:\n enabled: false\n")
|
file := parseFileConfig(t, "version: 4\ncache:\n checkpoints:\n enabled: false\n")
|
||||||
if err := cfg.ApplyFileConfig(file); err != nil {
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -331,17 +605,17 @@ func TestFileConfigRejectsTrimmedKeyCollisions(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "pipeline ids",
|
name: "pipeline ids",
|
||||||
yaml: "version: 3\npipelines:\n main: {}\n ' main ': {}\n",
|
yaml: "version: 4\npipelines:\n main: {}\n ' main ': {}\n",
|
||||||
want: "pipeline id \"main\" is duplicated after trimming",
|
want: "pipeline id \"main\" is duplicated after trimming",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "lane ids",
|
name: "lane ids",
|
||||||
yaml: "version: 3\npipelines:\n main:\n artifacts:\n spells: {}\n ' spells ': {}\n",
|
yaml: "version: 4\npipelines:\n main:\n artifacts:\n spells: {}\n ' spells ': {}\n",
|
||||||
want: "artifact lane id \"spells\" is duplicated after trimming",
|
want: "artifact lane id \"spells\" is duplicated after trimming",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "reference slots",
|
name: "reference slots",
|
||||||
yaml: "version: 3\npipelines:\n main:\n references:\n slot: ./one.txt\n ' slot ': ./two.txt\n",
|
yaml: "version: 4\npipelines:\n main:\n references:\n slot: ./one.txt\n ' slot ': ./two.txt\n",
|
||||||
want: "reference slot \"slot\" is duplicated after trimming",
|
want: "reference slot \"slot\" is duplicated after trimming",
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -358,7 +632,7 @@ func TestFileConfigRejectsTrimmedKeyCollisions(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigParsesOrderedStepsAndReferenceSources(t *testing.T) {
|
func TestFileConfigParsesOrderedStepsAndReferenceSources(t *testing.T) {
|
||||||
file := parseFileConfig(t, `version: 3
|
file := parseFileConfig(t, `version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
session:
|
session:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
@@ -397,7 +671,7 @@ func TestFileConfigRejectsAmbiguousReferenceSourceForms(t *testing.T) {
|
|||||||
"artifact: {step: 1, lane: b}",
|
"artifact: {step: 1, lane: b}",
|
||||||
"1",
|
"1",
|
||||||
} {
|
} {
|
||||||
_, err := ParseFileConfigYAML([]byte("version: 3\npipelines:\n p:\n input: text\n references:\n slot: " + source + "\n"))
|
_, err := ParseFileConfigYAML([]byte("version: 4\npipelines:\n p:\n input: text\n references:\n slot: " + source + "\n"))
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatalf("ParseFileConfigYAML(%q) error = nil", source)
|
t.Fatalf("ParseFileConfigYAML(%q) error = nil", source)
|
||||||
}
|
}
|
||||||
@@ -412,12 +686,12 @@ func TestFileConfigRejectsEmptyAndAmbiguousPipelineShapes(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "empty steps",
|
name: "empty steps",
|
||||||
yaml: "version: 3\npipelines:\n p:\n input: text\n steps: []\n",
|
yaml: "version: 4\npipelines:\n p:\n input: text\n steps: []\n",
|
||||||
want: "at least one ordered step",
|
want: "at least one ordered step",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "both forms",
|
name: "both forms",
|
||||||
yaml: "version: 3\npipelines:\n p:\n input: text\n artifacts: {}\n steps: []\n",
|
yaml: "version: 4\npipelines:\n p:\n input: text\n artifacts: {}\n steps: []\n",
|
||||||
want: "both artifacts and steps",
|
want: "both artifacts and steps",
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -19,23 +19,28 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
resolved := pipeline.ResolvedPipeline{
|
resolved := pipeline.ResolvedPipeline{
|
||||||
ID: "redaction-test",
|
ID: "redaction-test",
|
||||||
Digest: "sha256:safe-digest",
|
Digest: "sha256:safe-digest",
|
||||||
Input: bindings["input"],
|
Input: bindings["input"],
|
||||||
Chunk: bindings["chunk"],
|
InputExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
ChunkReferences: redactionTestReferenceTarget(pipeline.StageChunk, "", "chunk-reference-content"),
|
Chunk: bindings["chunk"],
|
||||||
|
ChunkExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
|
ChunkReferences: redactionTestReferenceTarget(pipeline.StageChunk, "", "chunk-reference-content"),
|
||||||
Steps: []pipeline.ResolvedPipelineStep{{
|
Steps: []pipeline.ResolvedPipelineStep{{
|
||||||
ID: "default",
|
ID: "default",
|
||||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
||||||
ID: "safe-lane",
|
ID: "safe-lane",
|
||||||
ArtifactKind: "safe/artifact",
|
ArtifactKind: "safe/artifact",
|
||||||
Extract: bindings["extract"],
|
Extract: bindings["extract"],
|
||||||
Merge: bindings["merge"],
|
ExtractExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Normalize: bindings["normalize"],
|
Merge: bindings["merge"],
|
||||||
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
|
MergeExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"),
|
Normalize: bindings["normalize"],
|
||||||
MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"),
|
NormalizeExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
NormalizeReferences: redactionTestReferenceTarget(pipeline.StageNormalize, "safe-lane", "normalize-reference-content"),
|
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
|
||||||
|
ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"),
|
||||||
|
MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"),
|
||||||
|
NormalizeReferences: redactionTestReferenceTarget(pipeline.StageNormalize, "safe-lane", "normalize-reference-content"),
|
||||||
}},
|
}},
|
||||||
}},
|
}},
|
||||||
ValidatorChains: []pipeline.ResolvedValidatorChain{{
|
ValidatorChains: []pipeline.ResolvedValidatorChain{{
|
||||||
@@ -49,7 +54,8 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
|
|||||||
ArtifactKind: "safe/artifact",
|
ArtifactKind: "safe/artifact",
|
||||||
}},
|
}},
|
||||||
}},
|
}},
|
||||||
Output: bindings["output"],
|
Output: bindings["output"],
|
||||||
|
OutputExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
}
|
}
|
||||||
effective := EffectiveConfig{
|
effective := EffectiveConfig{
|
||||||
Config: Config{Pipelines: map[string]pipeline.PipelineProfile{
|
Config: Config{Pipelines: map[string]pipeline.PipelineProfile{
|
||||||
@@ -88,6 +94,11 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
|
|||||||
t.Fatalf("resolved pipeline summary does not retain %q: %s", safe, text)
|
t.Fatalf("resolved pipeline summary does not retain %q: %s", safe, text)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
for _, executionClass := range []string{"input_execution_class\":\"deterministic", "chunk_execution_class\":\"llm_backed", "extract_execution_class\":\"llm_backed", "merge_execution_class\":\"deterministic", "normalize_execution_class\":\"llm_backed", "output_execution_class\":\"deterministic"} {
|
||||||
|
if !strings.Contains(text, executionClass) {
|
||||||
|
t.Fatalf("resolved pipeline summary does not retain %q: %s", executionClass, text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
payload.Input.Options["safe"] = "mutated"
|
payload.Input.Options["safe"] = "mutated"
|
||||||
nested := payload.Input.Options["nested"].([]any)[0].([]any)[0].(map[string]any)
|
nested := payload.Input.Options["nested"].([]any)[0].([]any)[0].(map[string]any)
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package config
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"net/url"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
@@ -10,7 +11,7 @@ import (
|
|||||||
|
|
||||||
func (c Config) Validate() error {
|
func (c Config) Validate() error {
|
||||||
c.Concurrency.recomputeStageWorkerDefaults()
|
c.Concurrency.recomputeStageWorkerDefaults()
|
||||||
if err := validateScriptorium(c.Scriptorium); err != nil {
|
if err := validatePromptKit(c.PromptKit); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if err := validateStateSurfaces(c); err != nil {
|
if err := validateStateSurfaces(c); err != nil {
|
||||||
@@ -49,9 +50,30 @@ func validateStageWorkers(cfg ConcurrencyConfig) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func validateScriptorium(cfg ScriptoriumConfig) error {
|
func validatePromptKit(cfg PromptKitConfig) error {
|
||||||
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
||||||
return fmt.Errorf("scriptorium profile_dir and profile_file are mutually exclusive")
|
return fmt.Errorf("promptkit profile_dir and profile_file are mutually exclusive")
|
||||||
|
}
|
||||||
|
if cfg.LocalBackend == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
endpoint := strings.TrimSpace(cfg.LocalBackend.Endpoint)
|
||||||
|
if endpoint == "" {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
|
||||||
|
}
|
||||||
|
parsed, err := url.Parse(endpoint)
|
||||||
|
if err != nil ||
|
||||||
|
(!strings.EqualFold(parsed.Scheme, "http") && !strings.EqualFold(parsed.Scheme, "https")) ||
|
||||||
|
!parsed.IsAbs() ||
|
||||||
|
parsed.Hostname() == "" ||
|
||||||
|
parsed.User != nil ||
|
||||||
|
parsed.RawQuery != "" ||
|
||||||
|
parsed.ForceQuery ||
|
||||||
|
strings.Contains(endpoint, "#") {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must be an absolute HTTP or HTTPS URL with a host and no user information, query, or fragment")
|
||||||
|
}
|
||||||
|
if cfg.LocalBackend.ConcurrencyLimit < 0 {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.concurrency_limit must not be negative")
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -87,10 +87,77 @@ func TestValidateConcurrencyRules(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestValidateScriptoriumSourcesAreMutuallyExclusive(t *testing.T) {
|
func TestValidatePromptKitSourcesAreMutuallyExclusive(t *testing.T) {
|
||||||
cfg := Default()
|
cfg := Default()
|
||||||
cfg.Scriptorium = ScriptoriumConfig{ProfileDir: "./profiles", ProfileFile: "./profile.yml"}
|
cfg.PromptKit = PromptKitConfig{ProfileDir: "./profiles", ProfileFile: "./profile.yml"}
|
||||||
assertValidationContains(t, cfg, "scriptorium profile_dir and profile_file are mutually exclusive")
|
assertValidationContains(t, cfg, "promptkit profile_dir and profile_file are mutually exclusive")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatePromptKitLocalBackendEndpoints(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
endpoint string
|
||||||
|
profileSource PromptKitConfig
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "HTTP endpoint with path and profile directory",
|
||||||
|
endpoint: "http://localhost:8000/v1",
|
||||||
|
profileSource: PromptKitConfig{ProfileDir: "./profiles"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "case-insensitive HTTPS endpoint and profile file",
|
||||||
|
endpoint: "HTTPS://inference.example.test/api",
|
||||||
|
profileSource: PromptKitConfig{ProfileFile: "./profiles.yml"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := Default()
|
||||||
|
cfg.PromptKit = tt.profileSource
|
||||||
|
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: tt.endpoint,
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
if err := cfg.Validate(); err != nil {
|
||||||
|
t.Fatalf("Validate() error = %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatePromptKitLocalBackendRejectsInvalidValues(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
endpoint string
|
||||||
|
concurrencyLimit int
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "blank endpoint", endpoint: " ", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "relative URL", endpoint: "localhost:8000/v1", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "unsupported scheme", endpoint: "ftp://localhost/model", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "missing host", endpoint: "http:///v1", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "user information", endpoint: "http://user:secret@localhost/v1", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "query", endpoint: "http://localhost/v1?model=example", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "empty query", endpoint: "http://localhost/v1?", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "fragment", endpoint: "http://localhost/v1#model", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "empty fragment", endpoint: "http://localhost/v1#", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{
|
||||||
|
name: "negative concurrency",
|
||||||
|
endpoint: "http://localhost:8000/v1",
|
||||||
|
concurrencyLimit: -1,
|
||||||
|
want: "promptkit.local_backend.concurrency_limit",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := Default()
|
||||||
|
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: tt.endpoint,
|
||||||
|
ConcurrencyLimit: tt.concurrencyLimit,
|
||||||
|
}
|
||||||
|
assertValidationContains(t, cfg, tt.want)
|
||||||
|
})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestValidateStateSurfaceRules(t *testing.T) {
|
func TestValidateStateSurfaceRules(t *testing.T) {
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package debugbundle
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -137,6 +138,47 @@ func TestSummaryWriterWritesEverySummaryArtifact(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestWriteInvocationPreservesReasoningEffortOverrideStates(t *testing.T) {
|
||||||
|
replacement := "focused"
|
||||||
|
cleared := ""
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
override *string
|
||||||
|
wantValue string
|
||||||
|
wantSet bool
|
||||||
|
}{
|
||||||
|
{name: "inherit"},
|
||||||
|
{name: "replace", override: &replacement, wantValue: "focused", wantSet: true},
|
||||||
|
{name: "clear", override: &cleared, wantSet: true},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
bundle, err := Allocate(t.TempDir(), testBundleRunID, time.Unix(0, 42))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := bundle.Summary().WriteInvocation(Invocation{
|
||||||
|
Operation: "run",
|
||||||
|
ReasoningEffortOverride: tt.override,
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
data, err := os.ReadFile(filepath.Join(bundle.SummaryRoot(), ArtifactInvocationMetadata))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var payload map[string]any
|
||||||
|
if err := json.Unmarshal(data, &payload); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
value, found := payload["reasoning_effort_override"]
|
||||||
|
if found != tt.wantSet || (found && value != tt.wantValue) {
|
||||||
|
t.Fatalf("reasoning override found=%t value=%#v, want found=%t value=%q; JSON=%s", found, value, tt.wantSet, tt.wantValue, data)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
func TestSummaryWriterInternalWritesConfineArtifacts(t *testing.T) {
|
func TestSummaryWriterInternalWritesConfineArtifacts(t *testing.T) {
|
||||||
bundle, err := Allocate(t.TempDir(), testBundleRunID, time.Unix(0, 42))
|
bundle, err := Allocate(t.TempDir(), testBundleRunID, time.Unix(0, 42))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -29,18 +29,19 @@ type RedactedResolvedPipelinePayload interface {
|
|||||||
RedactedResolvedPipelinePayload() pipeline.ResolvedPipeline
|
RedactedResolvedPipelinePayload() pipeline.ResolvedPipeline
|
||||||
}
|
}
|
||||||
type Invocation struct {
|
type Invocation struct {
|
||||||
Operation string `json:"operation"`
|
Operation string `json:"operation"`
|
||||||
PipelineID string `json:"pipeline_id,omitempty"`
|
PipelineID string `json:"pipeline_id,omitempty"`
|
||||||
PipelineDigest string `json:"pipeline_digest,omitempty"`
|
PipelineDigest string `json:"pipeline_digest,omitempty"`
|
||||||
Resume bool `json:"resume,omitempty"`
|
Resume bool `json:"resume,omitempty"`
|
||||||
RecomputeStep string `json:"recompute_step,omitempty"`
|
RecomputeStep string `json:"recompute_step,omitempty"`
|
||||||
InputPath string `json:"input_path,omitempty"`
|
InputPath string `json:"input_path,omitempty"`
|
||||||
ConfigPath string `json:"config_path,omitempty"`
|
ConfigPath string `json:"config_path,omitempty"`
|
||||||
ConfigSource string `json:"config_source,omitempty"`
|
ConfigSource string `json:"config_source,omitempty"`
|
||||||
OnlyLanes []string `json:"only_lanes,omitempty"`
|
OnlyLanes []string `json:"only_lanes,omitempty"`
|
||||||
ChunkCacheOverride string `json:"chunk_cache_override,omitempty"`
|
ChunkCacheOverride string `json:"chunk_cache_override,omitempty"`
|
||||||
RunID string `json:"run_id"`
|
ReasoningEffortOverride *string `json:"reasoning_effort_override,omitempty"`
|
||||||
StartedAt time.Time `json:"started_at"`
|
RunID string `json:"run_id"`
|
||||||
|
StartedAt time.Time `json:"started_at"`
|
||||||
}
|
}
|
||||||
type RunReport struct {
|
type RunReport struct {
|
||||||
RunID string `json:"run_id"`
|
RunID string `json:"run_id"`
|
||||||
@@ -68,6 +69,10 @@ func (w *SummaryWriter) WriteInvocation(payload Invocation) error {
|
|||||||
if payload.StartedAt.IsZero() {
|
if payload.StartedAt.IsZero() {
|
||||||
payload.StartedAt = w.createdAt
|
payload.StartedAt = w.createdAt
|
||||||
}
|
}
|
||||||
|
if payload.ReasoningEffortOverride != nil {
|
||||||
|
value := *payload.ReasoningEffortOverride
|
||||||
|
payload.ReasoningEffortOverride = &value
|
||||||
|
}
|
||||||
return w.writeJSON(ArtifactInvocationMetadata, payload)
|
return w.writeJSON(ArtifactInvocationMetadata, payload)
|
||||||
}
|
}
|
||||||
func (w *SummaryWriter) WriteRedactedEffectiveConfig(payload RedactedSummaryPayload) error {
|
func (w *SummaryWriter) WriteRedactedEffectiveConfig(payload RedactedSummaryPayload) error {
|
||||||
|
|||||||
@@ -43,6 +43,7 @@ type LLMDebugPrompt struct {
|
|||||||
PromptVersion string `json:"prompt_version,omitempty"`
|
PromptVersion string `json:"prompt_version,omitempty"`
|
||||||
PromptHash string `json:"prompt_hash,omitempty"`
|
PromptHash string `json:"prompt_hash,omitempty"`
|
||||||
SelectedProfileID string `json:"selected_profile_id,omitempty"`
|
SelectedProfileID string `json:"selected_profile_id,omitempty"`
|
||||||
|
SelectedBackendID string `json:"selected_backend_id,omitempty"`
|
||||||
SessionID string `json:"session_id,omitempty"`
|
SessionID string `json:"session_id,omitempty"`
|
||||||
RenderedPromptHash string `json:"rendered_prompt_hash,omitempty"`
|
RenderedPromptHash string `json:"rendered_prompt_hash,omitempty"`
|
||||||
Messages []LLMDebugMessage `json:"messages,omitempty"`
|
Messages []LLMDebugMessage `json:"messages,omitempty"`
|
||||||
@@ -266,10 +267,6 @@ const (
|
|||||||
ExecutionClassLLMBacked ExecutionClass = "llm_backed"
|
ExecutionClassLLMBacked ExecutionClass = "llm_backed"
|
||||||
)
|
)
|
||||||
|
|
||||||
type ChunkExecutionClassProvider interface {
|
|
||||||
ExecutionClass() ExecutionClass
|
|
||||||
}
|
|
||||||
|
|
||||||
type ValidationResult struct {
|
type ValidationResult struct {
|
||||||
Approved bool `json:"approved"`
|
Approved bool `json:"approved"`
|
||||||
ReasonCode string `json:"reason_code,omitempty"`
|
ReasonCode string `json:"reason_code,omitempty"`
|
||||||
|
|||||||
@@ -5,3 +5,7 @@ import "errors"
|
|||||||
// ErrInvalidStructuredOutput identifies a provider response that cannot satisfy
|
// ErrInvalidStructuredOutput identifies a provider response that cannot satisfy
|
||||||
// the caller's declared structured-output contract.
|
// the caller's declared structured-output contract.
|
||||||
var ErrInvalidStructuredOutput = errors.New("invalid structured output")
|
var ErrInvalidStructuredOutput = errors.New("invalid structured output")
|
||||||
|
|
||||||
|
// ErrLLMCapacityExceeded identifies backend admission exhaustion before model
|
||||||
|
// generation begins.
|
||||||
|
var ErrLLMCapacityExceeded = errors.New("LLM capacity exceeded")
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
type AssetSource struct {
|
type AssetSource struct {
|
||||||
@@ -21,8 +21,9 @@ type AssetSource struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type AssetRegistry struct {
|
type AssetRegistry struct {
|
||||||
prompts []AssetSource
|
prompts []AssetSource
|
||||||
schemas []AssetSource
|
schemas []AssetSource
|
||||||
|
fallbackProfiles []AssetSource
|
||||||
}
|
}
|
||||||
|
|
||||||
type AssetHashPart struct {
|
type AssetHashPart struct {
|
||||||
@@ -58,6 +59,20 @@ func (r *AssetRegistry) RegisterSchemaFS(fsys fs.FS, root string) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RegisterFallbackProfileFS registers profile assets that PromptKit uses only
|
||||||
|
// when an operator-configured source does not provide a matching profile.
|
||||||
|
func (r *AssetRegistry) RegisterFallbackProfileFS(fsys fs.FS, root string) error {
|
||||||
|
if r == nil {
|
||||||
|
return fmt.Errorf("asset registry must not be nil")
|
||||||
|
}
|
||||||
|
source, err := newAssetSource(fsys, root)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("register fallback profile assets: %w", err)
|
||||||
|
}
|
||||||
|
r.fallbackProfiles = append(r.fallbackProfiles, source)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func (r *AssetRegistry) PromptFS() (fs.FS, error) {
|
func (r *AssetRegistry) PromptFS() (fs.FS, error) {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return nil, fmt.Errorf("asset registry must not be nil")
|
return nil, fmt.Errorf("asset registry must not be nil")
|
||||||
@@ -72,19 +87,76 @@ func (r *AssetRegistry) SchemaFS() (fs.FS, error) {
|
|||||||
return flattenAssetSources(r.schemas)
|
return flattenAssetSources(r.schemas)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *AssetRegistry) ScriptoriumOptions() ([]scriptorium.Option, error) {
|
func (r *AssetRegistry) FallbackProfileFS() (fs.FS, error) {
|
||||||
|
if r == nil {
|
||||||
|
return nil, fmt.Errorf("asset registry must not be nil")
|
||||||
|
}
|
||||||
|
return flattenAssetSources(r.fallbackProfiles)
|
||||||
|
}
|
||||||
|
|
||||||
|
// FallbackProfileDigest returns a deterministic, non-secret identity for the
|
||||||
|
// flattened fallback profile assets.
|
||||||
|
func (r *AssetRegistry) FallbackProfileDigest() (string, error) {
|
||||||
|
_, digest, _, err := r.fallbackProfileAssets()
|
||||||
|
return digest, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *AssetRegistry) PromptKitOptions() ([]promptkit.Option, error) {
|
||||||
|
options, _, err := r.promptKitOptions()
|
||||||
|
return options, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *AssetRegistry) promptKitOptions() ([]promptkit.Option, string, error) {
|
||||||
promptFS, err := r.PromptFS()
|
promptFS, err := r.PromptFS()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("prepare prompt assets: %w", err)
|
return nil, "", fmt.Errorf("prepare prompt assets: %w", err)
|
||||||
}
|
}
|
||||||
schemaFS, err := r.SchemaFS()
|
schemaFS, err := r.SchemaFS()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("prepare schema assets: %w", err)
|
return nil, "", fmt.Errorf("prepare schema assets: %w", err)
|
||||||
}
|
}
|
||||||
return []scriptorium.Option{
|
options := []promptkit.Option{
|
||||||
scriptorium.WithPromptFS(promptFS, "."),
|
promptkit.WithPromptFS(promptFS, "."),
|
||||||
scriptorium.WithSchemaFS(schemaFS, "."),
|
promptkit.WithSchemaFS(schemaFS, "."),
|
||||||
}, nil
|
}
|
||||||
|
fallbackFS, fallbackDigest, hasFallback, err := r.fallbackProfileAssets()
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", err
|
||||||
|
}
|
||||||
|
if hasFallback {
|
||||||
|
options = append(options, promptkit.WithFallbackProfileFS(fallbackFS, "."))
|
||||||
|
}
|
||||||
|
return options, fallbackDigest, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *AssetRegistry) promptKitFallbackProfileOption() (promptkit.Option, bool, error) {
|
||||||
|
fallbackFS, _, hasFallback, err := r.fallbackProfileAssets()
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
if !hasFallback {
|
||||||
|
return nil, false, nil
|
||||||
|
}
|
||||||
|
return promptkit.WithFallbackProfileFS(fallbackFS, "."), true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *AssetRegistry) fallbackProfileAssets() (fs.FS, string, bool, error) {
|
||||||
|
if r == nil {
|
||||||
|
return nil, "", false, fmt.Errorf("asset registry must not be nil")
|
||||||
|
}
|
||||||
|
if len(r.fallbackProfiles) == 0 {
|
||||||
|
empty := sha256.Sum256([]byte("notarius:fallback-profile-assets:empty"))
|
||||||
|
return nil, "sha256:" + hex.EncodeToString(empty[:]), false, nil
|
||||||
|
}
|
||||||
|
fallbackFS, err := r.FallbackProfileFS()
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", false, fmt.Errorf("prepare fallback profile assets: %w", err)
|
||||||
|
}
|
||||||
|
digest, err := hashAssetFilesystem(fallbackFS)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", false, err
|
||||||
|
}
|
||||||
|
return fallbackFS, digest, true, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func HashAssets(parts []AssetHashPart) (string, error) {
|
func HashAssets(parts []AssetHashPart) (string, error) {
|
||||||
@@ -117,6 +189,28 @@ func HashAssets(parts []AssetHashPart) (string, error) {
|
|||||||
return "sha256:" + hex.EncodeToString(hash.Sum(nil)), nil
|
return "sha256:" + hex.EncodeToString(hash.Sum(nil)), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func hashAssetFilesystem(fsys fs.FS) (string, error) {
|
||||||
|
var parts []AssetHashPart
|
||||||
|
err := fs.WalkDir(fsys, ".", func(name string, entry fs.DirEntry, walkErr error) error {
|
||||||
|
if walkErr != nil {
|
||||||
|
return walkErr
|
||||||
|
}
|
||||||
|
if entry.IsDir() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
parts = append(parts, AssetHashPart{FS: fsys, Path: name})
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("walk assets for digest: %w", err)
|
||||||
|
}
|
||||||
|
if len(parts) > 0 {
|
||||||
|
return HashAssets(parts)
|
||||||
|
}
|
||||||
|
empty := sha256.Sum256([]byte("notarius:fallback-profile-assets:empty"))
|
||||||
|
return "sha256:" + hex.EncodeToString(empty[:]), nil
|
||||||
|
}
|
||||||
|
|
||||||
func newAssetSource(fsys fs.FS, root string) (AssetSource, error) {
|
func newAssetSource(fsys fs.FS, root string) (AssetSource, error) {
|
||||||
if fsys == nil {
|
if fsys == nil {
|
||||||
return AssetSource{}, fmt.Errorf("filesystem must not be nil")
|
return AssetSource{}, fmt.Errorf("filesystem must not be nil")
|
||||||
@@ -248,6 +342,9 @@ func (m assetMapFS) dirEntries(dir string) []fs.DirEntry {
|
|||||||
children[childName] = entry
|
children[childName] = entry
|
||||||
}
|
}
|
||||||
if len(children) == 0 {
|
if len(children) == 0 {
|
||||||
|
if dir == "." {
|
||||||
|
return []fs.DirEntry{}
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
names := make([]string, 0, len(children))
|
names := make([]string, 0, len(children))
|
||||||
|
|||||||
@@ -2,12 +2,13 @@ package llm
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"io/fs"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"testing/fstest"
|
"testing/fstest"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestAssetRegistryCombinesPromptAndSchemaSources(t *testing.T) {
|
func TestAssetRegistryCombinesPromptAndSchemaSources(t *testing.T) {
|
||||||
@@ -22,12 +23,12 @@ func TestAssetRegistryCombinesPromptAndSchemaSources(t *testing.T) {
|
|||||||
"root/schemas/out.json": {Data: []byte(`{"type":"object"}`)},
|
"root/schemas/out.json": {Data: []byte(`{"type":"object"}`)},
|
||||||
}, "root")
|
}, "root")
|
||||||
|
|
||||||
engine := newAssetTestEngine(t, registry)
|
engine := newPromptKitAssetTestEngine(t, registry)
|
||||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: "asset.test",
|
PromptID: "asset.test",
|
||||||
ProfileID: "asset-test-profile",
|
ProfileID: "asset-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.Inline(`{"ok":true}`),
|
"transcript": promptkit.Inline(`{"ok":true}`),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -50,12 +51,12 @@ func TestAssetRegistryPrepareFailsForMissingPromptAsset(t *testing.T) {
|
|||||||
"out.json": {Data: []byte(`{"type":"object"}`)},
|
"out.json": {Data: []byte(`{"type":"object"}`)},
|
||||||
}, ".")
|
}, ".")
|
||||||
|
|
||||||
engine := newAssetTestEngine(t, registry)
|
engine := newPromptKitAssetTestEngine(t, registry)
|
||||||
_, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
_, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: "asset.test",
|
PromptID: "asset.test",
|
||||||
ProfileID: "asset-test-profile",
|
ProfileID: "asset-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.Inline(`{"ok":true}`),
|
"transcript": promptkit.Inline(`{"ok":true}`),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err == nil || !strings.Contains(err.Error(), "content_file") {
|
if err == nil || !strings.Contains(err.Error(), "content_file") {
|
||||||
@@ -74,12 +75,12 @@ func TestAssetRegistryPrepareFailsForMissingSchemaAsset(t *testing.T) {
|
|||||||
"present.json": {Data: []byte(`{"type":"object"}`)},
|
"present.json": {Data: []byte(`{"type":"object"}`)},
|
||||||
}, ".")
|
}, ".")
|
||||||
|
|
||||||
engine := newAssetTestEngine(t, registry)
|
engine := newPromptKitAssetTestEngine(t, registry)
|
||||||
_, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
_, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: "asset.test",
|
PromptID: "asset.test",
|
||||||
ProfileID: "asset-test-profile",
|
ProfileID: "asset-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.Inline(`{"ok":true}`),
|
"transcript": promptkit.Inline(`{"ok":true}`),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err == nil || !strings.Contains(err.Error(), "missing.json") {
|
if err == nil || !strings.Contains(err.Error(), "missing.json") {
|
||||||
@@ -98,6 +99,89 @@ func TestAssetRegistryRejectsDuplicateAssetPaths(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestAssetRegistryCombinesFallbackProfileSources(t *testing.T) {
|
||||||
|
registry := NewAssetRegistry()
|
||||||
|
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
|
||||||
|
"first/profiles/one.yaml": {Data: []byte("id: one\nmodel: first\n")},
|
||||||
|
}, "first/profiles"); err != nil {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
|
||||||
|
"second/two.yaml": {Data: []byte("id: two\nmodel: second\n")},
|
||||||
|
}, "second"); err != nil {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
fallbackFS, err := registry.FallbackProfileFS()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("FallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
for _, name := range []string{"one.yaml", "two.yaml"} {
|
||||||
|
if _, err := fs.ReadFile(fallbackFS, name); err != nil {
|
||||||
|
t.Fatalf("FallbackProfileFS().ReadFile(%q) error = %v, want nil", name, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAssetRegistryRejectsInvalidFallbackProfileRoot(t *testing.T) {
|
||||||
|
registry := NewAssetRegistry()
|
||||||
|
err := registry.RegisterFallbackProfileFS(fstest.MapFS{}, "../profiles")
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "invalid path") {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want invalid root error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAssetRegistryRejectsUnreadableFallbackProfileAssets(t *testing.T) {
|
||||||
|
registry := NewAssetRegistry()
|
||||||
|
if err := registry.RegisterFallbackProfileFS(unreadableAssetFS{}, "."); err != nil {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
_, err := registry.FallbackProfileFS()
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "permission denied") {
|
||||||
|
t.Fatalf("FallbackProfileFS() error = %v, want unreadable asset error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAssetRegistryRejectsDuplicateFallbackProfilePaths(t *testing.T) {
|
||||||
|
registry := NewAssetRegistry()
|
||||||
|
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{"first/profile.yaml": {Data: []byte("id: first\n")}}, "first"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{"second/profile.yaml": {Data: []byte("id: second\n")}}, "second"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
_, err := registry.FallbackProfileFS()
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "duplicate asset path") {
|
||||||
|
t.Fatalf("FallbackProfileFS() error = %v, want duplicate path error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAssetRegistryFallbackProfileDigestTracksContentWithoutLeakingIt(t *testing.T) {
|
||||||
|
digestFor := func(content string) string {
|
||||||
|
t.Helper()
|
||||||
|
registry := NewAssetRegistry()
|
||||||
|
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
|
||||||
|
"profiles/profile.yaml": {Data: []byte(content)},
|
||||||
|
}, "profiles"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
digest, err := registry.FallbackProfileDigest()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return digest
|
||||||
|
}
|
||||||
|
|
||||||
|
first := digestFor("id: fallback\nmodel: model-one\n")
|
||||||
|
second := digestFor("id: fallback\nmodel: model-two\n")
|
||||||
|
if first == second {
|
||||||
|
t.Fatalf("fallback profile digests = %q and %q, want content change", first, second)
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(first, "sha256:") || strings.Contains(first, "model-one") || strings.Contains(first, "profile.yaml") {
|
||||||
|
t.Fatalf("fallback profile digest leaked source details: %q", first)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestAssetRegistryCombinesNamespacedPromptSources(t *testing.T) {
|
func TestAssetRegistryCombinesNamespacedPromptSources(t *testing.T) {
|
||||||
registry := NewAssetRegistry()
|
registry := NewAssetRegistry()
|
||||||
mustRegisterPromptFS(t, registry, fstest.MapFS{
|
mustRegisterPromptFS(t, registry, fstest.MapFS{
|
||||||
@@ -145,18 +229,18 @@ func TestHashAssetsOmitsRawAssetContent(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func newAssetTestEngine(t *testing.T, registry *AssetRegistry) *scriptorium.Engine {
|
func newPromptKitAssetTestEngine(t *testing.T, registry *AssetRegistry) *promptkit.Engine {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
options, err := registry.ScriptoriumOptions()
|
options, err := registry.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ScriptoriumOptions() error = %v, want nil", err)
|
t.Fatalf("PromptKitOptions() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
ID: "asset-test-profile",
|
ID: "asset-test-profile",
|
||||||
Endpoint: "http://127.0.0.1:1/v1",
|
Endpoint: "http://127.0.0.1:1/v1",
|
||||||
Model: "asset-test-model",
|
Model: "asset-test-model",
|
||||||
})))
|
})))
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("NewEngine() error = %v, want nil", err)
|
t.Fatalf("NewEngine() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
@@ -196,3 +280,9 @@ output:
|
|||||||
repair_attempts: 0
|
repair_attempts: 0
|
||||||
`
|
`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type unreadableAssetFS struct{}
|
||||||
|
|
||||||
|
func (unreadableAssetFS) Open(name string) (fs.File, error) {
|
||||||
|
return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrPermission}
|
||||||
|
}
|
||||||
|
|||||||
14
internal/framework/llm/checkpoint_fingerprint.go
Normal file
14
internal/framework/llm/checkpoint_fingerprint.go
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
package llm
|
||||||
|
|
||||||
|
// CheckpointFingerprint is a stable, non-secret semantic identity contributed
|
||||||
|
// by the LLM runtime before pipeline execution.
|
||||||
|
type CheckpointFingerprint struct {
|
||||||
|
Name string
|
||||||
|
Value string
|
||||||
|
}
|
||||||
|
|
||||||
|
// CheckpointFingerprintProvider exposes LLM-runtime identities that must
|
||||||
|
// participate in checkpoint composition.
|
||||||
|
type CheckpointFingerprintProvider interface {
|
||||||
|
LLMCheckpointFingerprints() ([]CheckpointFingerprint, error)
|
||||||
|
}
|
||||||
@@ -3,6 +3,7 @@ package llm
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"net/http"
|
"net/http"
|
||||||
"regexp"
|
"regexp"
|
||||||
@@ -13,24 +14,36 @@ import (
|
|||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
const scriptoriumProviderName = "scriptorium"
|
const promptKitProviderName = "promptkit"
|
||||||
|
|
||||||
type ScriptoriumClientConfig struct {
|
type PromptKitLocalBackendConfig struct {
|
||||||
ProfileDir string
|
Endpoint string
|
||||||
ProfileFile string
|
ConcurrencyLimit int
|
||||||
Assets *AssetRegistry
|
|
||||||
Timeout time.Duration
|
|
||||||
HTTPClient *http.Client
|
|
||||||
EngineOptions []scriptorium.Option
|
|
||||||
Recorder *LLMProfileRecorder
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type ScriptoriumClient struct {
|
type PromptKitClientConfig struct {
|
||||||
engine *scriptorium.Engine
|
ProfileDir string
|
||||||
recorder *LLMProfileRecorder
|
ProfileFile string
|
||||||
|
LocalBackend *PromptKitLocalBackendConfig
|
||||||
|
Assets *AssetRegistry
|
||||||
|
Timeout time.Duration
|
||||||
|
HTTPClient *http.Client
|
||||||
|
EngineOptions []promptkit.Option
|
||||||
|
Recorder *LLMProfileRecorder
|
||||||
|
ReasoningEffort *string
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitClient struct {
|
||||||
|
engine *promptkit.Engine
|
||||||
|
recorder *LLMProfileRecorder
|
||||||
|
profileDir string
|
||||||
|
profileFile string
|
||||||
|
localEndpoint string
|
||||||
|
fallbackProfileDigest string
|
||||||
|
reasoningEffort *string
|
||||||
}
|
}
|
||||||
|
|
||||||
type LLMProfileRecorder struct {
|
type LLMProfileRecorder struct {
|
||||||
@@ -38,49 +51,66 @@ type LLMProfileRecorder struct {
|
|||||||
profiles map[string]artifacts.LLMProfileManifest
|
profiles map[string]artifacts.LLMProfileManifest
|
||||||
}
|
}
|
||||||
|
|
||||||
var _ contracts.StructuredLLMClient = (*ScriptoriumClient)(nil)
|
var _ contracts.StructuredLLMClient = (*PromptKitClient)(nil)
|
||||||
var _ contracts.LLMProfileManifestProvider = (*ScriptoriumClient)(nil)
|
var _ contracts.LLMProfileManifestProvider = (*PromptKitClient)(nil)
|
||||||
|
|
||||||
func NewScriptoriumClient(cfg ScriptoriumClientConfig) (*ScriptoriumClient, error) {
|
func PromptKitLocalBackendOption(cfg PromptKitLocalBackendConfig) promptkit.Option {
|
||||||
|
return promptkit.WithBackend(promptkit.LocalBackend(cfg.Endpoint, cfg.ConcurrencyLimit))
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
||||||
if cfg.Assets == nil {
|
if cfg.Assets == nil {
|
||||||
return nil, fmt.Errorf("scriptorium client assets must not be nil")
|
return nil, fmt.Errorf("PromptKit client assets must not be nil")
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
profileSource, profileOptions, err := promptKitProfileSourceEngineOptions(PromptKitProfileSourceConfig{
|
||||||
return nil, fmt.Errorf("scriptorium profile_dir and profile_file are mutually exclusive")
|
ProfileDir: cfg.ProfileDir,
|
||||||
}
|
ProfileFile: cfg.ProfileFile,
|
||||||
options, err := cfg.Assets.ScriptoriumOptions()
|
LocalBackend: cfg.LocalBackend,
|
||||||
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
if profileFile := strings.TrimSpace(cfg.ProfileFile); profileFile != "" {
|
options, fallbackProfileDigest, err := cfg.Assets.promptKitOptions()
|
||||||
options = append(options, scriptorium.WithProfileFile(profileFile))
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
}
|
}
|
||||||
|
options = append(options, profileOptions...)
|
||||||
options = append(options, cfg.EngineOptions...)
|
options = append(options, cfg.EngineOptions...)
|
||||||
|
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{
|
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||||
ProfileDir: strings.TrimSpace(cfg.ProfileDir),
|
ProfileDir: profileSource.ProfileDir,
|
||||||
Timeout: cfg.Timeout,
|
Timeout: cfg.Timeout,
|
||||||
HTTPClient: cfg.HTTPClient,
|
HTTPClient: cfg.HTTPClient,
|
||||||
}, options...)
|
}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("create Scriptorium engine: %w", err)
|
return nil, fmt.Errorf("create PromptKit engine: %w", err)
|
||||||
}
|
}
|
||||||
recorder := cfg.Recorder
|
recorder := cfg.Recorder
|
||||||
if recorder == nil {
|
if recorder == nil {
|
||||||
recorder = NewLLMProfileRecorder()
|
recorder = NewLLMProfileRecorder()
|
||||||
}
|
}
|
||||||
return &ScriptoriumClient{
|
var reasoningEffort *string
|
||||||
engine: engine,
|
if cfg.ReasoningEffort != nil {
|
||||||
recorder: recorder,
|
value := *cfg.ReasoningEffort
|
||||||
|
reasoningEffort = &value
|
||||||
|
}
|
||||||
|
return &PromptKitClient{
|
||||||
|
engine: engine,
|
||||||
|
recorder: recorder,
|
||||||
|
profileDir: profileSource.ProfileDir,
|
||||||
|
profileFile: profileSource.ProfileFile,
|
||||||
|
localEndpoint: profileSource.localEndpoint(),
|
||||||
|
fallbackProfileDigest: fallbackProfileDigest,
|
||||||
|
reasoningEffort: reasoningEffort,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *ScriptoriumClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
if c == nil {
|
if c == nil {
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("scriptorium client must not be nil")
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("PromptKit client must not be nil")
|
||||||
}
|
}
|
||||||
if c.engine == nil {
|
if c.engine == nil {
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("scriptorium client engine must not be nil")
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("PromptKit client engine must not be nil")
|
||||||
}
|
}
|
||||||
if err := validateOutputTarget(out); err != nil {
|
if err := validateOutputTarget(out); err != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, err
|
return contracts.StructuredCompletionResponse{}, err
|
||||||
@@ -89,54 +119,85 @@ func (c *ScriptoriumClient) CompleteStructured(ctx context.Context, req contract
|
|||||||
if promptID == "" {
|
if promptID == "" {
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion prompt_id must not be empty")
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion prompt_id must not be empty")
|
||||||
}
|
}
|
||||||
|
sessionID := strings.TrimSpace(req.SessionID)
|
||||||
|
var execution *promptkit.ExecutionTargetOverride
|
||||||
|
if c.reasoningEffort != nil {
|
||||||
|
reasoningEffort := *c.reasoningEffort
|
||||||
|
execution = &promptkit.ExecutionTargetOverride{
|
||||||
|
ReasoningEffort: &reasoningEffort,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
runReq := scriptorium.RunRequest{
|
runReq := promptkit.RunRequest{
|
||||||
PromptID: promptID,
|
PromptID: promptID,
|
||||||
PromptVersion: strings.TrimSpace(req.PromptVersion),
|
PromptVersion: strings.TrimSpace(req.PromptVersion),
|
||||||
ProfileID: strings.TrimSpace(req.ProfileID),
|
ProfileID: strings.TrimSpace(req.ProfileID),
|
||||||
Inputs: scriptoriumInputs(req.Inputs),
|
SessionID: sessionID,
|
||||||
Vars: scriptoriumVars(req),
|
Inputs: promptKitInputs(req.Inputs),
|
||||||
Metadata: scriptoriumMetadata(req),
|
Vars: promptKitVars(req, sessionID),
|
||||||
|
Execution: execution,
|
||||||
}
|
}
|
||||||
prepared, err := c.engine.Prepare(ctx, runReq)
|
prepared, err := c.engine.PrepareExecution(ctx, runReq)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, ctxErr
|
return contracts.StructuredCompletionResponse{}, ctxErr
|
||||||
}
|
}
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("prepare Scriptorium prompt %q: %w", promptID, redactScriptoriumError(err))
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("prepare PromptKit prompt %q: %w", promptID, redactPromptKitError(err))
|
||||||
}
|
}
|
||||||
result, err := c.engine.Run(ctx, runReq)
|
defer prepared.Discard()
|
||||||
|
preparedDetails := prepared.Details()
|
||||||
|
result, err := c.engine.RunPrepared(ctx, prepared)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, ctxErr
|
return contracts.StructuredCompletionResponse{}, ctxErr
|
||||||
}
|
}
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run Scriptorium prompt %q: %w", promptID, redactScriptoriumError(err))
|
if errors.Is(err, promptkit.ErrCapacityExceeded) {
|
||||||
|
var capacityErr *promptkit.CapacityError
|
||||||
|
if errors.As(err, &capacityErr) && strings.TrimSpace(capacityErr.BackendID) != "" {
|
||||||
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf(
|
||||||
|
"run PromptKit prompt %q on backend %q: %w: %v",
|
||||||
|
promptID,
|
||||||
|
strings.TrimSpace(capacityErr.BackendID),
|
||||||
|
contracts.ErrLLMCapacityExceeded,
|
||||||
|
redactPromptKitError(err),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf(
|
||||||
|
"run PromptKit prompt %q: %w: %v",
|
||||||
|
promptID,
|
||||||
|
contracts.ErrLLMCapacityExceeded,
|
||||||
|
redactPromptKitError(err),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w", promptID, redactPromptKitError(err))
|
||||||
}
|
}
|
||||||
if result == nil {
|
if result == nil {
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run Scriptorium prompt %q: %w: empty result", promptID, contracts.ErrInvalidStructuredOutput)
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w: empty result", promptID, contracts.ErrInvalidStructuredOutput)
|
||||||
}
|
}
|
||||||
response := c.responseFromResult(result, prepared)
|
response := c.responseFromResult(result, &preparedDetails)
|
||||||
if result.Validation.Status == scriptorium.ValidationFailed || !result.Validation.IsValid {
|
if result.Validation.Status == promptkit.ValidationFailed || !result.Validation.IsValid {
|
||||||
return response, fmt.Errorf("run Scriptorium prompt %q: %w: validation failed: %s", promptID, contracts.ErrInvalidStructuredOutput, strings.Join(result.Validation.Errors, "; "))
|
return response, fmt.Errorf("run PromptKit prompt %q: %w: validation failed: %s", promptID, contracts.ErrInvalidStructuredOutput, strings.Join(result.Validation.Errors, "; "))
|
||||||
}
|
}
|
||||||
if len(strings.TrimSpace(string(response.Content))) == 0 {
|
if len(strings.TrimSpace(string(response.Content))) == 0 {
|
||||||
return response, fmt.Errorf("run Scriptorium prompt %q: %w: empty structured output", promptID, contracts.ErrInvalidStructuredOutput)
|
return response, fmt.Errorf("run PromptKit prompt %q: %w: empty structured output", promptID, contracts.ErrInvalidStructuredOutput)
|
||||||
}
|
}
|
||||||
if err := json.Unmarshal(response.Content, out); err != nil {
|
if err := json.Unmarshal(response.Content, out); err != nil {
|
||||||
return response, fmt.Errorf("decode Scriptorium structured output for prompt %q: %w: %w", promptID, contracts.ErrInvalidStructuredOutput, err)
|
return response, fmt.Errorf("decode PromptKit structured output for prompt %q: %w: %w", promptID, contracts.ErrInvalidStructuredOutput, err)
|
||||||
}
|
}
|
||||||
return response, nil
|
return response, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *ScriptoriumClient) responseFromResult(result *scriptorium.RunResult, prepared *scriptorium.PreparedRun) contracts.StructuredCompletionResponse {
|
func (c *PromptKitClient) responseFromResult(result *promptkit.RunResult, prepared *promptkit.PreparedRun) contracts.StructuredCompletionResponse {
|
||||||
content := result.Artifact.Body
|
content := result.Artifact.Body
|
||||||
if len(content) == 0 {
|
if len(content) == 0 {
|
||||||
content = []byte(result.RawOutput)
|
content = []byte(result.RawOutput)
|
||||||
}
|
}
|
||||||
profile := artifacts.LLMProfileManifest{
|
profile := artifacts.LLMProfileManifest{
|
||||||
ID: strings.TrimSpace(result.SelectedProfileID),
|
ID: strings.TrimSpace(result.SelectedProfileID),
|
||||||
Provider: scriptoriumProviderName,
|
Provider: promptKitProviderName,
|
||||||
Model: firstNonEmpty(result.ModelName, result.EffectiveModelParams.Model),
|
Model: firstNonEmpty(result.ModelName, result.EffectiveModelParams.Model),
|
||||||
|
BackendID: strings.TrimSpace(result.SelectedBackendID),
|
||||||
|
ReasoningEffort: strings.TrimSpace(result.EffectiveModelParams.ReasoningEffort),
|
||||||
}
|
}
|
||||||
if c.recorder != nil {
|
if c.recorder != nil {
|
||||||
c.recorder.Record(profile)
|
c.recorder.Record(profile)
|
||||||
@@ -149,17 +210,17 @@ func (c *ScriptoriumClient) responseFromResult(result *scriptorium.RunResult, pr
|
|||||||
PromptTokens: result.Usage.PromptTokens,
|
PromptTokens: result.Usage.PromptTokens,
|
||||||
CompletionTokens: result.Usage.CompletionTokens,
|
CompletionTokens: result.Usage.CompletionTokens,
|
||||||
TotalTokens: result.Usage.TotalTokens,
|
TotalTokens: result.Usage.TotalTokens,
|
||||||
Debug: scriptoriumDebugMaterial(prepared, result),
|
Debug: promptKitDebugMaterial(prepared, result),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumDebugMaterial(prepared *scriptorium.PreparedRun, result *scriptorium.RunResult) *contracts.LLMDebugMaterial {
|
func promptKitDebugMaterial(prepared *promptkit.PreparedRun, result *promptkit.RunResult) *contracts.LLMDebugMaterial {
|
||||||
material := &contracts.LLMDebugMaterial{}
|
material := &contracts.LLMDebugMaterial{}
|
||||||
if prepared != nil {
|
if prepared != nil {
|
||||||
material.Prompt = scriptoriumDebugPrompt(prepared)
|
material.Prompt = promptKitDebugPrompt(prepared)
|
||||||
}
|
}
|
||||||
if result != nil {
|
if result != nil {
|
||||||
material.Response = scriptoriumDebugResponse(result)
|
material.Response = promptKitDebugResponse(result)
|
||||||
}
|
}
|
||||||
if material.Prompt == nil && material.Response == nil {
|
if material.Prompt == nil && material.Response == nil {
|
||||||
return nil
|
return nil
|
||||||
@@ -167,7 +228,7 @@ func scriptoriumDebugMaterial(prepared *scriptorium.PreparedRun, result *scripto
|
|||||||
return material
|
return material
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumDebugPrompt(prepared *scriptorium.PreparedRun) *contracts.LLMDebugPrompt {
|
func promptKitDebugPrompt(prepared *promptkit.PreparedRun) *contracts.LLMDebugPrompt {
|
||||||
if prepared == nil {
|
if prepared == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -184,6 +245,7 @@ func scriptoriumDebugPrompt(prepared *scriptorium.PreparedRun) *contracts.LLMDeb
|
|||||||
PromptVersion: prepared.PromptVersion,
|
PromptVersion: prepared.PromptVersion,
|
||||||
PromptHash: prepared.PromptHash,
|
PromptHash: prepared.PromptHash,
|
||||||
SelectedProfileID: prepared.SelectedProfileID,
|
SelectedProfileID: prepared.SelectedProfileID,
|
||||||
|
SelectedBackendID: prepared.SelectedBackendID,
|
||||||
SessionID: prepared.SessionID,
|
SessionID: prepared.SessionID,
|
||||||
RenderedPromptHash: prepared.RenderedPromptHash,
|
RenderedPromptHash: prepared.RenderedPromptHash,
|
||||||
Messages: messages,
|
Messages: messages,
|
||||||
@@ -194,7 +256,7 @@ func scriptoriumDebugPrompt(prepared *scriptorium.PreparedRun) *contracts.LLMDeb
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumDebugResponse(result *scriptorium.RunResult) *contracts.LLMDebugResponse {
|
func promptKitDebugResponse(result *promptkit.RunResult) *contracts.LLMDebugResponse {
|
||||||
if result == nil {
|
if result == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -254,13 +316,28 @@ func cloneStringMap(values map[string]string) map[string]string {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *ScriptoriumClient) LLMProfileManifests() []artifacts.LLMProfileManifest {
|
func (c *PromptKitClient) LLMProfileManifests() []artifacts.LLMProfileManifest {
|
||||||
if c == nil || c.recorder == nil {
|
if c == nil || c.recorder == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return c.recorder.Manifests()
|
return c.recorder.Manifests()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (c *PromptKitClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint, error) {
|
||||||
|
if c == nil {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
fingerprint, err := promptKitProfileFingerprint(c.profileDir, c.profileFile, c.fallbackProfileDigest)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
fingerprints := []CheckpointFingerprint{fingerprint}
|
||||||
|
if c.localEndpoint != "" {
|
||||||
|
fingerprints = append(fingerprints, promptKitLocalBackendFingerprint(c.localEndpoint))
|
||||||
|
}
|
||||||
|
return fingerprints, nil
|
||||||
|
}
|
||||||
|
|
||||||
func NewLLMProfileRecorder() *LLMProfileRecorder {
|
func NewLLMProfileRecorder() *LLMProfileRecorder {
|
||||||
return &LLMProfileRecorder{profiles: map[string]artifacts.LLMProfileManifest{}}
|
return &LLMProfileRecorder{profiles: map[string]artifacts.LLMProfileManifest{}}
|
||||||
}
|
}
|
||||||
@@ -269,10 +346,8 @@ func (r *LLMProfileRecorder) Record(profile artifacts.LLMProfileManifest) {
|
|||||||
if r == nil {
|
if r == nil {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
profile.ID = strings.TrimSpace(profile.ID)
|
profile = profile.Normalized()
|
||||||
profile.Provider = strings.TrimSpace(profile.Provider)
|
key := profile.IdentityKey()
|
||||||
profile.Model = strings.TrimSpace(profile.Model)
|
|
||||||
key := profile.ID + "\x00" + profile.Provider + "\x00" + profile.Model
|
|
||||||
r.mu.Lock()
|
r.mu.Lock()
|
||||||
defer r.mu.Unlock()
|
defer r.mu.Unlock()
|
||||||
if r.profiles == nil {
|
if r.profiles == nil {
|
||||||
@@ -302,11 +377,11 @@ func (r *LLMProfileRecorder) Manifests() []artifacts.LLMProfileManifest {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumInputs(inputs contracts.LLMInputSet) map[string]scriptorium.ArtifactRef {
|
func promptKitInputs(inputs contracts.LLMInputSet) map[string]promptkit.ArtifactRef {
|
||||||
if len(inputs) == 0 {
|
if len(inputs) == 0 {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
out := make(map[string]scriptorium.ArtifactRef, len(inputs))
|
out := make(map[string]promptkit.ArtifactRef, len(inputs))
|
||||||
for key, material := range inputs {
|
for key, material := range inputs {
|
||||||
name := strings.TrimSpace(key)
|
name := strings.TrimSpace(key)
|
||||||
if name == "" {
|
if name == "" {
|
||||||
@@ -320,15 +395,15 @@ func scriptoriumInputs(inputs contracts.LLMInputSet) map[string]scriptorium.Arti
|
|||||||
body = " "
|
body = " "
|
||||||
}
|
}
|
||||||
if origin := strings.TrimSpace(material.OriginURI); origin != "" {
|
if origin := strings.TrimSpace(material.OriginURI); origin != "" {
|
||||||
out[name] = scriptorium.InlineWithURI(origin, body)
|
out[name] = promptkit.InlineWithURI(origin, body)
|
||||||
} else {
|
} else {
|
||||||
out[name] = scriptorium.Inline(body)
|
out[name] = promptkit.Inline(body)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumVars(req contracts.StructuredCompletionRequest) map[string]string {
|
func promptKitVars(req contracts.StructuredCompletionRequest, sessionID string) map[string]string {
|
||||||
vars := make(map[string]string, len(req.Vars)+1)
|
vars := make(map[string]string, len(req.Vars)+1)
|
||||||
for key, value := range req.Vars {
|
for key, value := range req.Vars {
|
||||||
name := strings.TrimSpace(key)
|
name := strings.TrimSpace(key)
|
||||||
@@ -337,7 +412,7 @@ func scriptoriumVars(req contracts.StructuredCompletionRequest) map[string]strin
|
|||||||
}
|
}
|
||||||
vars[name] = fmt.Sprint(value)
|
vars[name] = fmt.Sprint(value)
|
||||||
}
|
}
|
||||||
if sessionID := strings.TrimSpace(req.SessionID); sessionID != "" {
|
if sessionID != "" {
|
||||||
vars["session_id"] = sessionID
|
vars["session_id"] = sessionID
|
||||||
}
|
}
|
||||||
if len(vars) == 0 {
|
if len(vars) == 0 {
|
||||||
@@ -346,20 +421,9 @@ func scriptoriumVars(req contracts.StructuredCompletionRequest) map[string]strin
|
|||||||
return vars
|
return vars
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumMetadata(req contracts.StructuredCompletionRequest) map[string]string {
|
|
||||||
metadata := map[string]string{}
|
|
||||||
if stageName := strings.TrimSpace(req.StageName); stageName != "" {
|
|
||||||
metadata["stage_name"] = stageName
|
|
||||||
}
|
|
||||||
if len(metadata) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
return metadata
|
|
||||||
}
|
|
||||||
|
|
||||||
var bearerTokenPattern = regexp.MustCompile(`(?i)Bearer\s+[A-Za-z0-9._~+/=-]+`)
|
var bearerTokenPattern = regexp.MustCompile(`(?i)Bearer\s+[A-Za-z0-9._~+/=-]+`)
|
||||||
|
|
||||||
func redactScriptoriumError(err error) error {
|
func redactPromptKitError(err error) error {
|
||||||
if err == nil {
|
if err == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
1269
internal/framework/llm/promptkit_client_test.go
Normal file
1269
internal/framework/llm/promptkit_client_test.go
Normal file
File diff suppressed because it is too large
Load Diff
110
internal/framework/llm/promptkit_profile_fingerprint.go
Normal file
110
internal/framework/llm/promptkit_profile_fingerprint.go
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
package llm
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"fmt"
|
||||||
|
"io/fs"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
promptKitProfileFingerprintName = "promptkit_profile_source"
|
||||||
|
promptKitLocalBackendFingerprintName = "promptkit_local_backend_target"
|
||||||
|
promptKitLocalBackendMarker = "notarius:promptkit-local-backend:v1"
|
||||||
|
// The built-in profile catalog is compiled into this pinned PromptKit
|
||||||
|
// release. Update this identity when the dependency is upgraded.
|
||||||
|
promptKitBuiltinProfileCatalogID = "promptkit:v0.5.0:builtin-profiles"
|
||||||
|
)
|
||||||
|
|
||||||
|
func promptKitProfileFingerprint(profileDir, profileFile, fallbackProfileDigest string) (CheckpointFingerprint, error) {
|
||||||
|
hasher := sha256.New()
|
||||||
|
writeFingerprintPart(hasher, []byte(promptKitBuiltinProfileCatalogID))
|
||||||
|
writeFingerprintPart(hasher, []byte(strings.TrimSpace(fallbackProfileDigest)))
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case strings.TrimSpace(profileFile) != "":
|
||||||
|
cleanProfileFile := strings.TrimSpace(profileFile)
|
||||||
|
data, err := os.ReadFile(cleanProfileFile)
|
||||||
|
if err != nil {
|
||||||
|
return CheckpointFingerprint{}, fmt.Errorf("read PromptKit profile file for checkpoint identity")
|
||||||
|
}
|
||||||
|
writeFingerprintPart(hasher, []byte(filepath.ToSlash(filepath.Base(cleanProfileFile))))
|
||||||
|
writeFingerprintPart(hasher, data)
|
||||||
|
case strings.TrimSpace(profileDir) != "":
|
||||||
|
files, err := promptKitProfileFiles(strings.TrimSpace(profileDir))
|
||||||
|
if err != nil {
|
||||||
|
return CheckpointFingerprint{}, err
|
||||||
|
}
|
||||||
|
for _, file := range files {
|
||||||
|
writeFingerprintPart(hasher, []byte(file.path))
|
||||||
|
writeFingerprintPart(hasher, file.digest)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return CheckpointFingerprint{
|
||||||
|
Name: promptKitProfileFingerprintName,
|
||||||
|
Value: "sha256:" + hex.EncodeToString(hasher.Sum(nil)),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptKitLocalBackendFingerprint(endpoint string) CheckpointFingerprint {
|
||||||
|
hasher := sha256.New()
|
||||||
|
writeFingerprintPart(hasher, []byte(promptKitLocalBackendMarker))
|
||||||
|
writeFingerprintPart(hasher, []byte(strings.TrimSpace(endpoint)))
|
||||||
|
return CheckpointFingerprint{
|
||||||
|
Name: promptKitLocalBackendFingerprintName,
|
||||||
|
Value: "sha256:" + hex.EncodeToString(hasher.Sum(nil)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type promptKitProfileFile struct {
|
||||||
|
path string
|
||||||
|
digest []byte
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptKitProfileFiles(root string) ([]promptKitProfileFile, error) {
|
||||||
|
var files []promptKitProfileFile
|
||||||
|
err := filepath.WalkDir(root, func(name string, entry fs.DirEntry, walkErr error) error {
|
||||||
|
if walkErr != nil {
|
||||||
|
return walkErr
|
||||||
|
}
|
||||||
|
if entry.IsDir() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
extension := filepath.Ext(entry.Name())
|
||||||
|
if extension != ".yaml" && extension != ".yml" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
data, err := os.ReadFile(name)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
relativePath, err := filepath.Rel(root, name)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
sum := sha256.Sum256(data)
|
||||||
|
files = append(files, promptKitProfileFile{
|
||||||
|
path: filepath.ToSlash(relativePath),
|
||||||
|
digest: append([]byte(nil), sum[:]...),
|
||||||
|
})
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read PromptKit profile directory for checkpoint identity")
|
||||||
|
}
|
||||||
|
sort.Slice(files, func(i, j int) bool {
|
||||||
|
return files[i].path < files[j].path
|
||||||
|
})
|
||||||
|
return files, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeFingerprintPart(hasher interface{ Write([]byte) (int, error) }, value []byte) {
|
||||||
|
length := []byte(fmt.Sprintf("%d:", len(value)))
|
||||||
|
_, _ = hasher.Write(length)
|
||||||
|
_, _ = hasher.Write(value)
|
||||||
|
}
|
||||||
146
internal/framework/llm/promptkit_profiles.go
Normal file
146
internal/framework/llm/promptkit_profiles.go
Normal file
@@ -0,0 +1,146 @@
|
|||||||
|
package llm
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
|
)
|
||||||
|
|
||||||
|
type PromptKitProfileSourceConfig struct {
|
||||||
|
ProfileDir string
|
||||||
|
ProfileFile string
|
||||||
|
LocalBackend *PromptKitLocalBackendConfig
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitProfileInspectorConfig struct {
|
||||||
|
Source PromptKitProfileSourceConfig
|
||||||
|
Assets *AssetRegistry
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c PromptKitProfileSourceConfig) localEndpoint() string {
|
||||||
|
if c.LocalBackend == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return c.LocalBackend.Endpoint
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitProfileInspector struct {
|
||||||
|
engine *promptkit.Engine
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitProfileInspection struct {
|
||||||
|
ProfileID string
|
||||||
|
BackendID string
|
||||||
|
Model string
|
||||||
|
CredentialEnvironment string
|
||||||
|
CredentialRequired bool
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitProfileInspectionError struct {
|
||||||
|
ProfileID string
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *PromptKitProfileInspectionError) Error() string {
|
||||||
|
switch {
|
||||||
|
case errors.Is(e.err, promptkit.ErrProfileNotFound):
|
||||||
|
return fmt.Sprintf("PromptKit profile %q is not configured", e.ProfileID)
|
||||||
|
case errors.Is(e.err, promptkit.ErrInvalidRequest):
|
||||||
|
return fmt.Sprintf("PromptKit profile ID %q is invalid", e.ProfileID)
|
||||||
|
case errors.Is(e.err, promptkit.ErrProfileLoad):
|
||||||
|
return fmt.Sprintf("PromptKit profile %q is invalid or unreadable", e.ProfileID)
|
||||||
|
default:
|
||||||
|
return fmt.Sprintf("PromptKit profile %q could not be inspected", e.ProfileID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *PromptKitProfileInspectionError) Unwrap() error {
|
||||||
|
return e.err
|
||||||
|
}
|
||||||
|
|
||||||
|
type promptKitProfileConfigurationError struct {
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *promptKitProfileConfigurationError) Error() string {
|
||||||
|
return "PromptKit profile configuration is invalid or unreadable"
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *promptKitProfileConfigurationError) Unwrap() error {
|
||||||
|
return e.err
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewPromptKitProfileInspector(cfg PromptKitProfileInspectorConfig) (*PromptKitProfileInspector, error) {
|
||||||
|
source, options, err := promptKitProfileSourceEngineOptions(cfg.Source)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if cfg.Assets != nil {
|
||||||
|
fallbackOption, hasFallback, err := cfg.Assets.promptKitFallbackProfileOption()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if hasFallback {
|
||||||
|
options = append(options, fallbackOption)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||||
|
PromptDir: ".",
|
||||||
|
ProfileDir: source.ProfileDir,
|
||||||
|
}, options...)
|
||||||
|
if err != nil {
|
||||||
|
return nil, &promptKitProfileConfigurationError{err: err}
|
||||||
|
}
|
||||||
|
return &PromptKitProfileInspector{engine: engine}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (i *PromptKitProfileInspector) InspectProfile(ctx context.Context, profileID string) (PromptKitProfileInspection, error) {
|
||||||
|
if i == nil || i.engine == nil {
|
||||||
|
return PromptKitProfileInspection{}, fmt.Errorf("PromptKit profile inspector must not be nil")
|
||||||
|
}
|
||||||
|
profileID = strings.TrimSpace(profileID)
|
||||||
|
inspection, err := i.engine.InspectProfile(ctx, profileID)
|
||||||
|
if err != nil {
|
||||||
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
|
return PromptKitProfileInspection{}, ctxErr
|
||||||
|
}
|
||||||
|
return PromptKitProfileInspection{}, &PromptKitProfileInspectionError{
|
||||||
|
ProfileID: profileID,
|
||||||
|
err: err,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return PromptKitProfileInspection{
|
||||||
|
ProfileID: inspection.ProfileID,
|
||||||
|
BackendID: strings.TrimSpace(inspection.EffectiveModelParams.BackendID),
|
||||||
|
Model: strings.TrimSpace(inspection.EffectiveModelParams.Model),
|
||||||
|
CredentialEnvironment: strings.TrimSpace(inspection.EffectiveModelParams.APIKeyEnv),
|
||||||
|
CredentialRequired: inspection.APIKeyRequired,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptKitProfileSourceEngineOptions(cfg PromptKitProfileSourceConfig) (PromptKitProfileSourceConfig, []promptkit.Option, error) {
|
||||||
|
source := PromptKitProfileSourceConfig{
|
||||||
|
ProfileDir: strings.TrimSpace(cfg.ProfileDir),
|
||||||
|
ProfileFile: strings.TrimSpace(cfg.ProfileFile),
|
||||||
|
}
|
||||||
|
if source.ProfileDir != "" && source.ProfileFile != "" {
|
||||||
|
return PromptKitProfileSourceConfig{}, nil, fmt.Errorf("PromptKit profile_dir and profile_file are mutually exclusive")
|
||||||
|
}
|
||||||
|
if cfg.LocalBackend != nil {
|
||||||
|
localBackend := *cfg.LocalBackend
|
||||||
|
localBackend.Endpoint = strings.TrimSpace(localBackend.Endpoint)
|
||||||
|
source.LocalBackend = &localBackend
|
||||||
|
}
|
||||||
|
|
||||||
|
var options []promptkit.Option
|
||||||
|
if source.ProfileFile != "" {
|
||||||
|
options = append(options, promptkit.WithProfileFile(source.ProfileFile))
|
||||||
|
}
|
||||||
|
if source.LocalBackend != nil {
|
||||||
|
options = append(options, PromptKitLocalBackendOption(*source.LocalBackend))
|
||||||
|
}
|
||||||
|
return source, options, nil
|
||||||
|
}
|
||||||
@@ -53,3 +53,14 @@ func (c *scheduledClient) LLMProfileManifests() []artifacts.LLMProfileManifest {
|
|||||||
}
|
}
|
||||||
return provider.LLMProfileManifests()
|
return provider.LLMProfileManifests()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (c *scheduledClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint, error) {
|
||||||
|
if c == nil || c.client == nil {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
provider, ok := c.client.(CheckpointFingerprintProvider)
|
||||||
|
if !ok {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return provider.LLMCheckpointFingerprints()
|
||||||
|
}
|
||||||
|
|||||||
@@ -75,6 +75,33 @@ func TestScheduledClientPropagatesSchedulerError(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestScheduledClientPreservesCheckpointFingerprints(t *testing.T) {
|
||||||
|
scheduler, err := NewScheduler(1)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
inner := &fingerprintedStructuredClient{
|
||||||
|
fingerprints: []CheckpointFingerprint{
|
||||||
|
{Name: "profile_source", Value: "sha256:one"},
|
||||||
|
{Name: "backend_target", Value: "sha256:two"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
client := NewScheduledClient(inner, scheduler)
|
||||||
|
provider, ok := client.(CheckpointFingerprintProvider)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("scheduled client %T does not preserve checkpoint fingerprints", client)
|
||||||
|
}
|
||||||
|
got, err := provider.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(got) != len(inner.fingerprints) ||
|
||||||
|
got[0] != inner.fingerprints[0] ||
|
||||||
|
got[1] != inner.fingerprints[1] {
|
||||||
|
t.Fatalf("checkpoint fingerprints = %#v, want %#v", got, inner.fingerprints)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
type blockingStructuredClient struct {
|
type blockingStructuredClient struct {
|
||||||
release chan struct{}
|
release chan struct{}
|
||||||
inFlight int32
|
inFlight int32
|
||||||
@@ -110,6 +137,15 @@ type errorStructuredClient struct {
|
|||||||
err error
|
err error
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type fingerprintedStructuredClient struct {
|
||||||
|
errorStructuredClient
|
||||||
|
fingerprints []CheckpointFingerprint
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *fingerprintedStructuredClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint, error) {
|
||||||
|
return append([]CheckpointFingerprint(nil), c.fingerprints...), nil
|
||||||
|
}
|
||||||
|
|
||||||
func (c *errorStructuredClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
func (c *errorStructuredClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
return contracts.StructuredCompletionResponse{}, c.err
|
return contracts.StructuredCompletionResponse{}, c.err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,372 +0,0 @@
|
|||||||
package llm
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"strings"
|
|
||||||
"sync"
|
|
||||||
"sync/atomic"
|
|
||||||
"testing"
|
|
||||||
"testing/fstest"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestScriptoriumClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
|
|
||||||
fake := &fakeScriptoriumLLM{content: `{"ok":true}`}
|
|
||||||
client := newTestScriptoriumClient(t, fake)
|
|
||||||
|
|
||||||
var out struct {
|
|
||||||
OK bool `json:"ok"`
|
|
||||||
}
|
|
||||||
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
|
||||||
StageName: "test-stage",
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
PromptVersion: "v1",
|
|
||||||
ProfileID: "explicit-profile",
|
|
||||||
SessionID: "session-123",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "sha256:source", "file:///source.json"),
|
|
||||||
},
|
|
||||||
Vars: map[string]any{"custom": "value"},
|
|
||||||
}, &out)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
if !out.OK {
|
|
||||||
t.Fatalf("decoded output OK = false, want true")
|
|
||||||
}
|
|
||||||
if resp.Provider != scriptoriumProviderName || resp.Model != "explicit-model" || resp.ProfileID != "explicit-profile" {
|
|
||||||
t.Fatalf("response metadata = %#v", resp)
|
|
||||||
}
|
|
||||||
if resp.PromptTokens != 11 || resp.CompletionTokens != 7 || resp.TotalTokens != 18 {
|
|
||||||
t.Fatalf("usage = %#v, want mapped token counts", resp)
|
|
||||||
}
|
|
||||||
if resp.Debug == nil || resp.Debug.Prompt == nil {
|
|
||||||
t.Fatalf("debug prompt = nil, want prepared prompt material")
|
|
||||||
}
|
|
||||||
if resp.Debug.Prompt.PromptID != "adapter.test" || resp.Debug.Prompt.SelectedProfileID != "explicit-profile" {
|
|
||||||
t.Fatalf("debug prompt metadata = %#v, want prompt/profile", resp.Debug.Prompt)
|
|
||||||
}
|
|
||||||
if len(resp.Debug.Prompt.Messages) != 1 || !strings.Contains(resp.Debug.Prompt.Messages[0].Content, `{"source":true}`) {
|
|
||||||
t.Fatalf("debug prompt messages = %#v, want rendered input content", resp.Debug.Prompt.Messages)
|
|
||||||
}
|
|
||||||
if resp.Debug.Response == nil || resp.Debug.Response.Content != `{"ok":true}` {
|
|
||||||
t.Fatalf("debug response = %#v, want raw response content", resp.Debug.Response)
|
|
||||||
}
|
|
||||||
debugJSON, err := json.Marshal(resp.Debug)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("marshal debug material: %v", err)
|
|
||||||
}
|
|
||||||
if strings.Contains(string(debugJSON), "secret-token") || strings.Contains(string(debugJSON), "sk-") {
|
|
||||||
t.Fatalf("debug material contains secret material: %s", debugJSON)
|
|
||||||
}
|
|
||||||
gotReq := fake.lastRequest()
|
|
||||||
if gotReq.Prompt.SessionID != "session-123" {
|
|
||||||
t.Fatalf("session id = %q, want session-123", gotReq.Prompt.SessionID)
|
|
||||||
}
|
|
||||||
if gotReq.Target.Model != "explicit-model" {
|
|
||||||
t.Fatalf("model = %q, want explicit-model", gotReq.Target.Model)
|
|
||||||
}
|
|
||||||
if len(gotReq.Prompt.Messages) != 1 || !strings.Contains(gotReq.Prompt.Messages[0].Content, `{"source":true}`) {
|
|
||||||
t.Fatalf("rendered messages = %#v, want transcript input content", gotReq.Prompt.Messages)
|
|
||||||
}
|
|
||||||
if gotReq.StructuredOutput == nil {
|
|
||||||
t.Fatalf("structured output = nil, want JSON schema")
|
|
||||||
}
|
|
||||||
manifests := client.LLMProfileManifests()
|
|
||||||
if len(manifests) != 1 || manifests[0].ID != "explicit-profile" || manifests[0].Model != "explicit-model" {
|
|
||||||
t.Fatalf("profile manifests = %#v", manifests)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScriptoriumClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) {
|
|
||||||
fake := &fakeScriptoriumLLM{content: `{"ok":true}`}
|
|
||||||
client := newTestScriptoriumClient(t, fake)
|
|
||||||
|
|
||||||
var out map[string]any
|
|
||||||
if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
SessionID: "session-123",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
|
||||||
},
|
|
||||||
}, &out); err != nil {
|
|
||||||
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
if got := fake.lastRequest().Target.Model; got != "default-model" {
|
|
||||||
t.Fatalf("model = %q, want prompt default profile model", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScriptoriumClientValidationFailureReturnsError(t *testing.T) {
|
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"bad":true}`})
|
|
||||||
|
|
||||||
var out map[string]any
|
|
||||||
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
SessionID: "session-123",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
|
||||||
},
|
|
||||||
}, &out)
|
|
||||||
if err == nil || !errors.Is(err, contracts.ErrInvalidStructuredOutput) || !strings.Contains(err.Error(), "validation failed") {
|
|
||||||
t.Fatalf("CompleteStructured() error = %v, want validation failure", err)
|
|
||||||
}
|
|
||||||
if got := string(resp.Content); got != `{"bad":true}` {
|
|
||||||
t.Fatalf("response content = %q, want raw failed output", got)
|
|
||||||
}
|
|
||||||
if resp.Debug == nil || resp.Debug.Response == nil || resp.Debug.Response.Content != `{"bad":true}` {
|
|
||||||
t.Fatalf("debug response = %#v, want raw failed output", resp.Debug)
|
|
||||||
}
|
|
||||||
if resp.Debug.Prompt == nil || len(resp.Debug.Prompt.Messages) == 0 {
|
|
||||||
t.Fatalf("debug prompt = %#v, want prepared prompt material", resp.Debug.Prompt)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScriptoriumClientDecodeFailureReturnsRawResponse(t *testing.T) {
|
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"ok":true}`})
|
|
||||||
|
|
||||||
var out []any
|
|
||||||
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
SessionID: "session-123",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
|
||||||
},
|
|
||||||
}, &out)
|
|
||||||
if err == nil || !errors.Is(err, contracts.ErrInvalidStructuredOutput) || !strings.Contains(err.Error(), "decode Scriptorium structured output") {
|
|
||||||
t.Fatalf("CompleteStructured() error = %v, want decode failure", err)
|
|
||||||
}
|
|
||||||
if got := string(resp.Content); got != `{"ok":true}` {
|
|
||||||
t.Fatalf("response content = %q, want raw decode-failed output", got)
|
|
||||||
}
|
|
||||||
if resp.Debug == nil || resp.Debug.Response == nil || resp.Debug.Response.Content != `{"ok":true}` {
|
|
||||||
t.Fatalf("debug response = %#v, want raw decode-failed output", resp.Debug)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScriptoriumClientProviderFailureIncludesContextAndRedactsBearerToken(t *testing.T) {
|
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{err: errors.New("provider failed with Bearer secret-token")})
|
|
||||||
|
|
||||||
var out map[string]any
|
|
||||||
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
SessionID: "session-123",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
|
||||||
},
|
|
||||||
}, &out)
|
|
||||||
if err == nil {
|
|
||||||
t.Fatalf("CompleteStructured() error = nil, want provider error")
|
|
||||||
}
|
|
||||||
if errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
|
||||||
t.Fatalf("provider error = %v, must not be classified as invalid structured output", err)
|
|
||||||
}
|
|
||||||
if !strings.Contains(err.Error(), `run Scriptorium prompt "adapter.test"`) {
|
|
||||||
t.Fatalf("error = %q, want operation context", err.Error())
|
|
||||||
}
|
|
||||||
if strings.Contains(err.Error(), "secret-token") || !strings.Contains(err.Error(), "Bearer [REDACTED]") {
|
|
||||||
t.Fatalf("error = %q, want redacted bearer token", err.Error())
|
|
||||||
}
|
|
||||||
if resp.Debug != nil {
|
|
||||||
t.Fatalf("debug material = %#v, want none for provider failure without result", resp.Debug)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScriptoriumClientContextCancellationIsRespected(t *testing.T) {
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
cancel()
|
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"ok":true}`})
|
|
||||||
|
|
||||||
var out map[string]any
|
|
||||||
_, err := client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
|
||||||
},
|
|
||||||
}, &out)
|
|
||||||
if !errors.Is(err, context.Canceled) || errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
|
||||||
t.Fatalf("CompleteStructured() error = %v, want context canceled", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScriptoriumClientClassifiesEmptyStructuredCompletion(t *testing.T) {
|
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{allowEmpty: true})
|
|
||||||
|
|
||||||
var out map[string]any
|
|
||||||
_, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
SessionID: "session-123",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
|
||||||
},
|
|
||||||
}, &out)
|
|
||||||
if !errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
|
||||||
t.Fatalf("CompleteStructured() error = %v, want invalid structured output", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScheduledScriptoriumClientBoundsConcurrentCalls(t *testing.T) {
|
|
||||||
fake := &fakeScriptoriumLLM{
|
|
||||||
content: `{"ok":true}`,
|
|
||||||
block: make(chan struct{}),
|
|
||||||
}
|
|
||||||
client := newTestScriptoriumClient(t, fake)
|
|
||||||
scheduler, err := NewScheduler(1)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("NewScheduler() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
scheduled := NewScheduledClient(client, scheduler)
|
|
||||||
|
|
||||||
var wg sync.WaitGroup
|
|
||||||
for i := 0; i < 3; i++ {
|
|
||||||
wg.Add(1)
|
|
||||||
go func() {
|
|
||||||
defer wg.Done()
|
|
||||||
var out map[string]any
|
|
||||||
_, callErr := scheduled.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
SessionID: "session-123",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
|
||||||
},
|
|
||||||
}, &out)
|
|
||||||
if callErr != nil {
|
|
||||||
t.Errorf("CompleteStructured() error = %v, want nil", callErr)
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
}
|
|
||||||
waitForAtomicAtLeast(t, &fake.calls, 1)
|
|
||||||
time.Sleep(20 * time.Millisecond)
|
|
||||||
if got := atomic.LoadInt32(&fake.maxInFlight); got > 1 {
|
|
||||||
t.Fatalf("max in-flight calls = %d, want <= 1", got)
|
|
||||||
}
|
|
||||||
close(fake.block)
|
|
||||||
wg.Wait()
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScriptoriumClientValidatesRequest(t *testing.T) {
|
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"ok":true}`})
|
|
||||||
var out map[string]any
|
|
||||||
if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{}, &out); err == nil || !strings.Contains(err.Error(), "prompt_id") {
|
|
||||||
t.Fatalf("missing prompt id error = %v, want prompt_id validation", err)
|
|
||||||
}
|
|
||||||
if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{PromptID: "adapter.test"}, nil); err == nil || !strings.Contains(err.Error(), "non-nil pointer") {
|
|
||||||
t.Fatalf("nil output error = %v, want output validation", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func newTestScriptoriumClient(t *testing.T, fake *fakeScriptoriumLLM) *ScriptoriumClient {
|
|
||||||
t.Helper()
|
|
||||||
registry := NewAssetRegistry()
|
|
||||||
if err := registry.RegisterPromptFS(fstest.MapFS{
|
|
||||||
"adapter.test.yaml": {Data: []byte(`id: adapter.test
|
|
||||||
version: "v1"
|
|
||||||
default_profile: default-profile
|
|
||||||
session_id: "{{ .session_id }}"
|
|
||||||
inputs:
|
|
||||||
- name: transcript
|
|
||||||
required: true
|
|
||||||
content_type: application/json
|
|
||||||
messages:
|
|
||||||
- role: user
|
|
||||||
content: "Transcript: {{ input \"transcript\" }}"
|
|
||||||
output:
|
|
||||||
format: json
|
|
||||||
validation_mode: json_schema
|
|
||||||
schema_path: adapter.schema.json
|
|
||||||
repair_attempts: 0
|
|
||||||
`)},
|
|
||||||
}, "."); err != nil {
|
|
||||||
t.Fatalf("RegisterPromptFS() error = %v", err)
|
|
||||||
}
|
|
||||||
if err := registry.RegisterSchemaFS(fstest.MapFS{
|
|
||||||
"adapter.schema.json": {Data: []byte(`{"type":"object","required":["ok"],"properties":{"ok":{"type":"boolean"}}}`)},
|
|
||||||
}, "."); err != nil {
|
|
||||||
t.Fatalf("RegisterSchemaFS() error = %v", err)
|
|
||||||
}
|
|
||||||
client, err := NewScriptoriumClient(ScriptoriumClientConfig{
|
|
||||||
Assets: registry,
|
|
||||||
EngineOptions: []scriptorium.Option{
|
|
||||||
scriptorium.WithProfiles(
|
|
||||||
scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
|
||||||
ID: "default-profile",
|
|
||||||
Endpoint: "http://127.0.0.1:1/v1",
|
|
||||||
Model: "default-model",
|
|
||||||
}),
|
|
||||||
scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
|
||||||
ID: "explicit-profile",
|
|
||||||
Endpoint: "http://127.0.0.1:1/v1",
|
|
||||||
Model: "explicit-model",
|
|
||||||
}),
|
|
||||||
),
|
|
||||||
scriptorium.WithLLMClient(fake),
|
|
||||||
},
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("NewScriptoriumClient() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
return client
|
|
||||||
}
|
|
||||||
|
|
||||||
type fakeScriptoriumLLM struct {
|
|
||||||
content string
|
|
||||||
allowEmpty bool
|
|
||||||
err error
|
|
||||||
block chan struct{}
|
|
||||||
mu sync.Mutex
|
|
||||||
last scriptorium.GenerateRequest
|
|
||||||
calls int32
|
|
||||||
inFlight int32
|
|
||||||
maxInFlight int32
|
|
||||||
}
|
|
||||||
|
|
||||||
func (f *fakeScriptoriumLLM) Generate(ctx context.Context, req scriptorium.GenerateRequest) (*scriptorium.GenerateResponse, error) {
|
|
||||||
f.mu.Lock()
|
|
||||||
f.last = req
|
|
||||||
f.mu.Unlock()
|
|
||||||
atomic.AddInt32(&f.calls, 1)
|
|
||||||
current := atomic.AddInt32(&f.inFlight, 1)
|
|
||||||
for {
|
|
||||||
seen := atomic.LoadInt32(&f.maxInFlight)
|
|
||||||
if current <= seen || atomic.CompareAndSwapInt32(&f.maxInFlight, seen, current) {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
defer atomic.AddInt32(&f.inFlight, -1)
|
|
||||||
if f.block != nil {
|
|
||||||
select {
|
|
||||||
case <-f.block:
|
|
||||||
case <-ctx.Done():
|
|
||||||
return nil, ctx.Err()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if f.err != nil {
|
|
||||||
return nil, f.err
|
|
||||||
}
|
|
||||||
content := f.content
|
|
||||||
if content == "" && !f.allowEmpty {
|
|
||||||
content = `{"ok":true}`
|
|
||||||
}
|
|
||||||
if !f.allowEmpty && !json.Valid([]byte(content)) {
|
|
||||||
return nil, errors.New("test fake must return JSON content")
|
|
||||||
}
|
|
||||||
return &scriptorium.GenerateResponse{
|
|
||||||
Content: content,
|
|
||||||
Usage: scriptorium.TokenUsage{
|
|
||||||
PromptTokens: 11,
|
|
||||||
CompletionTokens: 7,
|
|
||||||
TotalTokens: 18,
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (f *fakeScriptoriumLLM) lastRequest() scriptorium.GenerateRequest {
|
|
||||||
f.mu.Lock()
|
|
||||||
defer f.mu.Unlock()
|
|
||||||
return f.last
|
|
||||||
}
|
|
||||||
@@ -24,10 +24,6 @@ func NewChunkerRegistry() *ChunkerRegistry {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *ChunkerRegistry) Register(key string, constructor ChunkerConstructor) error {
|
|
||||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageChunk), constructor)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerConstructor) error {
|
func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerConstructor) error {
|
||||||
if constructor == nil {
|
if constructor == nil {
|
||||||
return fmt.Errorf("chunker constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
return fmt.Errorf("chunker constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
|
|||||||
@@ -29,6 +29,10 @@ type registryBehaviorCase[M any] struct {
|
|||||||
moduleKey func(M) string
|
moduleKey func(M) string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func testModuleSpec(key string, stage ModuleStage) ModuleSpec {
|
||||||
|
return ModuleSpec{Key: key, Stage: stage, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
|
}
|
||||||
|
|
||||||
func TestChunkerRegistryBehavior(t *testing.T) {
|
func TestChunkerRegistryBehavior(t *testing.T) {
|
||||||
runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Chunker]{
|
runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Chunker]{
|
||||||
name: "ChunkerRegistry",
|
name: "ChunkerRegistry",
|
||||||
@@ -39,7 +43,9 @@ func TestChunkerRegistryBehavior(t *testing.T) {
|
|||||||
return NewChunkerRegistry()
|
return NewChunkerRegistry()
|
||||||
},
|
},
|
||||||
register: func(registry any, key string, constructor func() (contracts.Chunker, error)) error {
|
register: func(registry any, key string, constructor func() (contracts.Chunker, error)) error {
|
||||||
return registry.(*ChunkerRegistry).Register(key, constructor)
|
return registry.(*ChunkerRegistry).RegisterWithSpec(ModuleSpec{
|
||||||
|
Key: key, Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
}, constructor)
|
||||||
},
|
},
|
||||||
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Chunker, error)) error {
|
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Chunker, error)) error {
|
||||||
return registry.(*ChunkerRegistry).RegisterWithSpec(spec, constructor)
|
return registry.(*ChunkerRegistry).RegisterWithSpec(spec, constructor)
|
||||||
@@ -55,7 +61,9 @@ func TestChunkerRegistryBehavior(t *testing.T) {
|
|||||||
},
|
},
|
||||||
nilRegister: func(key string, constructor func() (contracts.Chunker, error)) error {
|
nilRegister: func(key string, constructor func() (contracts.Chunker, error)) error {
|
||||||
var registry *ChunkerRegistry
|
var registry *ChunkerRegistry
|
||||||
return registry.Register(key, constructor)
|
return registry.RegisterWithSpec(ModuleSpec{
|
||||||
|
Key: key, Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
}, constructor)
|
||||||
},
|
},
|
||||||
nilBuild: func(key string) (contracts.Chunker, error) {
|
nilBuild: func(key string) (contracts.Chunker, error) {
|
||||||
var registry *ChunkerRegistry
|
var registry *ChunkerRegistry
|
||||||
@@ -101,10 +109,11 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
t.Run(testCase.name+"/metadata registration and lookup", func(t *testing.T) {
|
t.Run(testCase.name+"/metadata registration and lookup", func(t *testing.T) {
|
||||||
registry := testCase.newRegistry()
|
registry := testCase.newRegistry()
|
||||||
spec := ModuleSpec{
|
spec := ModuleSpec{
|
||||||
Key: " " + testCase.key + " ",
|
Key: " " + testCase.key + " ",
|
||||||
Stage: testCase.stage,
|
Stage: testCase.stage,
|
||||||
Provides: []string{" beta ", "alpha", "", "beta"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{" source ", "source", ""},
|
Provides: []string{" beta ", "alpha", "", "beta"},
|
||||||
|
Requires: []string{" source ", "source", ""},
|
||||||
}
|
}
|
||||||
if err := testCase.registerWithSpec(registry, spec, testCase.constructor(testCase.key)); err != nil {
|
if err := testCase.registerWithSpec(registry, spec, testCase.constructor(testCase.key)); err != nil {
|
||||||
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
||||||
@@ -115,10 +124,11 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
t.Fatal("Spec() ok = false, want true")
|
t.Fatal("Spec() ok = false, want true")
|
||||||
}
|
}
|
||||||
want := ModuleSpec{
|
want := ModuleSpec{
|
||||||
Key: testCase.key,
|
Key: testCase.key,
|
||||||
Stage: testCase.stage,
|
Stage: testCase.stage,
|
||||||
Provides: []string{"alpha", "beta"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"source"},
|
Provides: []string{"alpha", "beta"},
|
||||||
|
Requires: []string{"source"},
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(got, want) {
|
if !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||||
@@ -134,7 +144,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
t.Run(testCase.name+"/default spec from register", func(t *testing.T) {
|
t.Run(testCase.name+"/minimal explicit spec registration", func(t *testing.T) {
|
||||||
registry := testCase.newRegistry()
|
registry := testCase.newRegistry()
|
||||||
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
|
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
@@ -144,7 +154,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
if !ok {
|
if !ok {
|
||||||
t.Fatal("Spec() ok = false, want true")
|
t.Fatal("Spec() ok = false, want true")
|
||||||
}
|
}
|
||||||
want := ModuleSpec{Key: testCase.key, Stage: testCase.stage}
|
want := ModuleSpec{Key: testCase.key, Stage: testCase.stage, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
if !reflect.DeepEqual(spec, want) {
|
if !reflect.DeepEqual(spec, want) {
|
||||||
t.Fatalf("Spec() = %#v, want %#v", spec, want)
|
t.Fatalf("Spec() = %#v, want %#v", spec, want)
|
||||||
}
|
}
|
||||||
@@ -152,7 +162,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
|
|
||||||
t.Run(testCase.name+"/wrong stage rejection", func(t *testing.T) {
|
t.Run(testCase.name+"/wrong stage rejection", func(t *testing.T) {
|
||||||
registry := testCase.newRegistry()
|
registry := testCase.newRegistry()
|
||||||
err := testCase.registerWithSpec(registry, ModuleSpec{Key: testCase.key, Stage: testCase.wrongStage}, testCase.constructor(testCase.key))
|
err := testCase.registerWithSpec(registry, ModuleSpec{Key: testCase.key, Stage: testCase.wrongStage, ExecutionClass: contracts.ExecutionClassDeterministic}, testCase.constructor(testCase.key))
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("RegisterWithSpec() error = nil, want error")
|
t.Fatal("RegisterWithSpec() error = nil, want error")
|
||||||
}
|
}
|
||||||
@@ -161,6 +171,22 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run(testCase.name+"/execution class rejection", func(t *testing.T) {
|
||||||
|
for _, spec := range []ModuleSpec{
|
||||||
|
{Key: testCase.key, Stage: testCase.stage},
|
||||||
|
{Key: testCase.key, Stage: testCase.stage, ExecutionClass: contracts.ExecutionClass("unsupported")},
|
||||||
|
} {
|
||||||
|
registry := testCase.newRegistry()
|
||||||
|
err := testCase.registerWithSpec(registry, spec, testCase.constructor(testCase.key))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("RegisterWithSpec() error = nil, want execution class error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "execution class") {
|
||||||
|
t.Fatalf("RegisterWithSpec() error = %q, want execution class error", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
t.Run(testCase.name+"/key trimming", func(t *testing.T) {
|
t.Run(testCase.name+"/key trimming", func(t *testing.T) {
|
||||||
registry := testCase.newRegistry()
|
registry := testCase.newRegistry()
|
||||||
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
|
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
|
||||||
|
|||||||
@@ -108,7 +108,7 @@ func registerTestEvidenceOutput(t *testing.T, registries *Registries, policy Evi
|
|||||||
func registerTestEvidenceOutputWithProfileValidation(t *testing.T, registries *Registries, policy EvidenceContextPolicy, validateProfile OutputProfileOptionValidator) {
|
func registerTestEvidenceOutputWithProfileValidation(t *testing.T, registries *Registries, policy EvidenceContextPolicy, validateProfile OutputProfileOptionValidator) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
registry := NewOutputEncoderRegistry()
|
registry := NewOutputEncoderRegistry()
|
||||||
if err := registry.RegisterBuilderWithProfileValidation(defaultModuleSpec("output", StageOutput), func(options map[string]any) error {
|
if err := registry.RegisterBuilderWithProfileValidation(testModuleSpec("output", StageOutput), func(options map[string]any) error {
|
||||||
return RejectUnknownOptions(options, "known")
|
return RejectUnknownOptions(options, "known")
|
||||||
}, validateProfile, func(BuildRequest) (contracts.OutputEncoder, error) {
|
}, validateProfile, func(BuildRequest) (contracts.OutputEncoder, error) {
|
||||||
return testEvidenceOutput{policy: cloneEvidenceContextPolicy(policy)}, nil
|
return testEvidenceOutput{policy: cloneEvidenceContextPolicy(policy)}, nil
|
||||||
|
|||||||
@@ -24,10 +24,6 @@ func NewInputAdapterRegistry() *InputAdapterRegistry {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *InputAdapterRegistry) Register(key string, constructor InputAdapterConstructor) error {
|
|
||||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageInput), constructor)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor InputAdapterConstructor) error {
|
func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor InputAdapterConstructor) error {
|
||||||
if constructor == nil {
|
if constructor == nil {
|
||||||
return fmt.Errorf("input adapter constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
return fmt.Errorf("input adapter constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
|
|||||||
@@ -11,10 +11,14 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
func registerTestInput(registry *InputAdapterRegistry, key string, constructor InputAdapterConstructor) error {
|
||||||
|
return registry.RegisterWithSpec(testModuleSpec(key, StageInput), constructor)
|
||||||
|
}
|
||||||
|
|
||||||
func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) {
|
func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -30,7 +34,7 @@ func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisterAndBuildTrimKeys(t *testing.T) {
|
func TestInputAdapterRegistryRegisterAndBuildTrimKeys(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
if err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
if err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -46,10 +50,11 @@ func TestInputAdapterRegistryRegisterAndBuildTrimKeys(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
spec := ModuleSpec{
|
spec := ModuleSpec{
|
||||||
Key: " generic-input ",
|
Key: " generic-input ",
|
||||||
Stage: StageInput,
|
Stage: StageInput,
|
||||||
Provides: []string{" parsed-source ", "source-document", "parsed-source", ""},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{" raw-bytes ", "raw-bytes", ""},
|
Provides: []string{" parsed-source ", "source-document", "parsed-source", ""},
|
||||||
|
Requires: []string{" raw-bytes ", "raw-bytes", ""},
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := registry.RegisterWithSpec(spec, fakeInputAdapterConstructor("generic-input")); err != nil {
|
if err := registry.RegisterWithSpec(spec, fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||||
@@ -61,10 +66,11 @@ func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
|||||||
t.Fatal("Spec() ok = false, want true")
|
t.Fatal("Spec() ok = false, want true")
|
||||||
}
|
}
|
||||||
want := ModuleSpec{
|
want := ModuleSpec{
|
||||||
Key: "generic-input",
|
Key: "generic-input",
|
||||||
Stage: StageInput,
|
Stage: StageInput,
|
||||||
Provides: []string{"parsed-source", "source-document"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"raw-bytes"},
|
Provides: []string{"parsed-source", "source-document"},
|
||||||
|
Requires: []string{"raw-bytes"},
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(got, want) {
|
if !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||||
@@ -80,10 +86,10 @@ func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
|
func TestInputAdapterRegistryRegisterWithSpecStoresMinimalMetadata(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
if err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
if err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -91,7 +97,7 @@ func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
|
|||||||
if !ok {
|
if !ok {
|
||||||
t.Fatal("Spec() ok = false, want true")
|
t.Fatal("Spec() ok = false, want true")
|
||||||
}
|
}
|
||||||
want := ModuleSpec{Key: "generic-input", Stage: StageInput}
|
want := ModuleSpec{Key: "generic-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
if !reflect.DeepEqual(got, want) {
|
if !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||||
}
|
}
|
||||||
@@ -100,7 +106,7 @@ func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisterWithSpecRejectsWrongStage(t *testing.T) {
|
func TestInputAdapterRegistryRegisterWithSpecRejectsWrongStage(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
err := registry.RegisterWithSpec(ModuleSpec{Key: "generic-input", Stage: StageExtract}, fakeInputAdapterConstructor("generic-input"))
|
err := registry.RegisterWithSpec(ModuleSpec{Key: "generic-input", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic}, fakeInputAdapterConstructor("generic-input"))
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("RegisterWithSpec() error = nil, want error")
|
t.Fatal("RegisterWithSpec() error = nil, want error")
|
||||||
@@ -121,7 +127,7 @@ func TestInputAdapterRegistrySpecRejectsUnknownKey(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) {
|
func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
err := registry.Register(" \t", fakeInputAdapterConstructor("generic-input"))
|
err := registerTestInput(registry, " \t", fakeInputAdapterConstructor("generic-input"))
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("Register() error = nil, want error")
|
t.Fatal("Register() error = nil, want error")
|
||||||
@@ -133,11 +139,11 @@ func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) {
|
|||||||
|
|
||||||
func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) {
|
func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input"))
|
err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input"))
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("Register() error = nil, want error")
|
t.Fatal("Register() error = nil, want error")
|
||||||
@@ -150,7 +156,7 @@ func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisterRejectsNilConstructor(t *testing.T) {
|
func TestInputAdapterRegistryRegisterRejectsNilConstructor(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
err := registry.Register("generic-input", nil)
|
err := registerTestInput(registry, "generic-input", nil)
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("Register() error = nil, want error")
|
t.Fatal("Register() error = nil, want error")
|
||||||
@@ -176,7 +182,7 @@ func TestInputAdapterRegistryBuildRejectsUnknownKey(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) {
|
func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
constructorErr := errors.New("constructor failed")
|
constructorErr := errors.New("constructor failed")
|
||||||
if err := registry.Register("generic-input", func() (contracts.InputAdapter, error) {
|
if err := registerTestInput(registry, "generic-input", func() (contracts.InputAdapter, error) {
|
||||||
return nil, constructorErr
|
return nil, constructorErr
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
@@ -197,7 +203,7 @@ func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) {
|
|||||||
|
|
||||||
func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) {
|
func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
if err := registry.Register("generic-input", func() (contracts.InputAdapter, error) {
|
if err := registerTestInput(registry, "generic-input", func() (contracts.InputAdapter, error) {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
@@ -215,7 +221,7 @@ func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) {
|
|||||||
|
|
||||||
func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) {
|
func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("other-input")); err != nil {
|
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("other-input")); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -232,7 +238,7 @@ func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
|
func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
for _, key := range []string{"zeta", "alpha", "middle"} {
|
for _, key := range []string{"zeta", "alpha", "middle"} {
|
||||||
if err := registry.Register(key, fakeInputAdapterConstructor(key)); err != nil {
|
if err := registerTestInput(registry, key, fakeInputAdapterConstructor(key)); err != nil {
|
||||||
t.Fatalf("Register(%q) error = %v, want nil", key, err)
|
t.Fatalf("Register(%q) error = %v, want nil", key, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -253,7 +259,7 @@ func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryNilRegistryBehavior(t *testing.T) {
|
func TestInputAdapterRegistryNilRegistryBehavior(t *testing.T) {
|
||||||
var registry *InputAdapterRegistry
|
var registry *InputAdapterRegistry
|
||||||
|
|
||||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err == nil {
|
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err == nil {
|
||||||
t.Fatal("Register() error = nil, want error")
|
t.Fatal("Register() error = nil, want error")
|
||||||
}
|
}
|
||||||
if _, err := registry.Build("generic-input"); err == nil {
|
if _, err := registry.Build("generic-input"); err == nil {
|
||||||
|
|||||||
@@ -23,23 +23,19 @@ const (
|
|||||||
type ModuleSpec struct {
|
type ModuleSpec struct {
|
||||||
Key string
|
Key string
|
||||||
Stage ModuleStage
|
Stage ModuleStage
|
||||||
|
ExecutionClass contracts.ExecutionClass
|
||||||
ArtifactKind contracts.ArtifactKind
|
ArtifactKind contracts.ArtifactKind
|
||||||
Provides []string
|
Provides []string
|
||||||
Requires []string
|
Requires []string
|
||||||
ReferenceSlots []contracts.ReferenceSlot
|
ReferenceSlots []contracts.ReferenceSlot
|
||||||
}
|
}
|
||||||
|
|
||||||
func defaultModuleSpec(key string, stage ModuleStage) ModuleSpec {
|
|
||||||
return ModuleSpec{
|
|
||||||
Key: key,
|
|
||||||
Stage: stage,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func normalizeModuleSpec(spec ModuleSpec) ModuleSpec {
|
func normalizeModuleSpec(spec ModuleSpec) ModuleSpec {
|
||||||
|
executionClass := contracts.ExecutionClass(strings.TrimSpace(string(spec.ExecutionClass)))
|
||||||
return ModuleSpec{
|
return ModuleSpec{
|
||||||
Key: strings.TrimSpace(spec.Key),
|
Key: strings.TrimSpace(spec.Key),
|
||||||
Stage: spec.Stage,
|
Stage: spec.Stage,
|
||||||
|
ExecutionClass: executionClass,
|
||||||
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
|
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
|
||||||
Provides: normalizeCapabilities(spec.Provides),
|
Provides: normalizeCapabilities(spec.Provides),
|
||||||
Requires: normalizeCapabilities(spec.Requires),
|
Requires: normalizeCapabilities(spec.Requires),
|
||||||
@@ -76,6 +72,7 @@ func cloneModuleSpec(spec ModuleSpec) ModuleSpec {
|
|||||||
return ModuleSpec{
|
return ModuleSpec{
|
||||||
Key: spec.Key,
|
Key: spec.Key,
|
||||||
Stage: spec.Stage,
|
Stage: spec.Stage,
|
||||||
|
ExecutionClass: spec.ExecutionClass,
|
||||||
ArtifactKind: spec.ArtifactKind,
|
ArtifactKind: spec.ArtifactKind,
|
||||||
Provides: append([]string(nil), spec.Provides...),
|
Provides: append([]string(nil), spec.Provides...),
|
||||||
Requires: append([]string(nil), spec.Requires...),
|
Requires: append([]string(nil), spec.Requires...),
|
||||||
@@ -90,6 +87,12 @@ func validateModuleSpec(kind string, expectedStage ModuleStage, spec ModuleSpec)
|
|||||||
if spec.Stage != expectedStage {
|
if spec.Stage != expectedStage {
|
||||||
return fmt.Errorf("%s %q must use %q stage, got %q", kind, spec.Key, expectedStage, spec.Stage)
|
return fmt.Errorf("%s %q must use %q stage, got %q", kind, spec.Key, expectedStage, spec.Stage)
|
||||||
}
|
}
|
||||||
|
if spec.ExecutionClass == "" {
|
||||||
|
return fmt.Errorf("%s %q execution class must not be empty", kind, spec.Key)
|
||||||
|
}
|
||||||
|
if spec.ExecutionClass != contracts.ExecutionClassDeterministic && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
|
||||||
|
return fmt.Errorf("%s %q has unsupported execution class %q", kind, spec.Key, spec.ExecutionClass)
|
||||||
|
}
|
||||||
if spec.ArtifactKind != "" && spec.Stage != StageExtract && spec.Stage != StageMerge && spec.Stage != StageNormalize {
|
if spec.ArtifactKind != "" && spec.Stage != StageExtract && spec.Stage != StageMerge && spec.Stage != StageNormalize {
|
||||||
return fmt.Errorf("%s %q must not declare an artifact kind", kind, spec.Key)
|
return fmt.Errorf("%s %q must not declare an artifact kind", kind, spec.Key)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,12 +1,45 @@
|
|||||||
package pipeline
|
package pipeline
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
func TestValidateModuleSpecRequiresSupportedExecutionClass(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
class contracts.ExecutionClass
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "missing", want: "execution class"},
|
||||||
|
{name: "unsupported", class: "remote", want: "unsupported"},
|
||||||
|
{name: "deterministic", class: contracts.ExecutionClassDeterministic},
|
||||||
|
{name: "llm backed", class: contracts.ExecutionClassLLMBacked},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
spec := normalizeModuleSpec(ModuleSpec{Key: "module", Stage: StageChunk, ExecutionClass: test.class})
|
||||||
|
err := validateModuleSpec("chunker", StageChunk, spec)
|
||||||
|
if test.want == "" && err != nil {
|
||||||
|
t.Fatalf("validateModuleSpec() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if test.want != "" && (err == nil || !strings.Contains(err.Error(), test.want)) {
|
||||||
|
t.Fatalf("validateModuleSpec() error = %v, want %q", err, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCloneModuleSpecPreservesExecutionClass(t *testing.T) {
|
||||||
|
spec := normalizeModuleSpec(ModuleSpec{Key: " module ", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked})
|
||||||
|
cloned := cloneModuleSpec(spec)
|
||||||
|
if !reflect.DeepEqual(cloned, spec) {
|
||||||
|
t.Fatalf("cloneModuleSpec() = %#v, want %#v", cloned, spec)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
|
func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -22,8 +55,9 @@ func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
|
|||||||
for _, test := range tests {
|
for _, test := range tests {
|
||||||
t.Run(test.name, func(t *testing.T) {
|
t.Run(test.name, func(t *testing.T) {
|
||||||
spec := normalizeModuleSpec(ModuleSpec{
|
spec := normalizeModuleSpec(ModuleSpec{
|
||||||
Key: "module",
|
Key: "module",
|
||||||
Stage: test.stage,
|
Stage: test.stage,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster", Description: "Character roster", MaxBytes: 1024},
|
{Name: "roster", Description: "Character roster", MaxBytes: 1024},
|
||||||
},
|
},
|
||||||
@@ -50,8 +84,9 @@ func TestValidateModuleSpecRejectsReferenceSlotsForIneligibleStages(t *testing.T
|
|||||||
for _, test := range tests {
|
for _, test := range tests {
|
||||||
t.Run(test.name, func(t *testing.T) {
|
t.Run(test.name, func(t *testing.T) {
|
||||||
spec := normalizeModuleSpec(ModuleSpec{
|
spec := normalizeModuleSpec(ModuleSpec{
|
||||||
Key: "module",
|
Key: "module",
|
||||||
Stage: test.stage,
|
Stage: test.stage,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster"},
|
{Name: "roster"},
|
||||||
},
|
},
|
||||||
@@ -109,6 +144,7 @@ func TestValidateModuleSpecRejectsInvalidReferenceSlotsForEligibleStages(t *test
|
|||||||
spec := normalizeModuleSpec(ModuleSpec{
|
spec := normalizeModuleSpec(ModuleSpec{
|
||||||
Key: "module",
|
Key: "module",
|
||||||
Stage: stage.stage,
|
Stage: stage.stage,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
ReferenceSlots: invalid.slots,
|
ReferenceSlots: invalid.slots,
|
||||||
})
|
})
|
||||||
err := validateModuleSpec(stage.kind, stage.stage, spec)
|
err := validateModuleSpec(stage.kind, stage.stage, spec)
|
||||||
|
|||||||
@@ -26,7 +26,7 @@ func TestNormalizerRegistryErasureClonesRetryDirective(t *testing.T) {
|
|||||||
FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "omitted", Message: "fallback warning"}},
|
FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "omitted", Message: "fallback warning"}},
|
||||||
}
|
}
|
||||||
registry := NewNormalizerRegistry()
|
registry := NewNormalizerRegistry()
|
||||||
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
||||||
return retryingNotesNormalizer{warnings: warnings, retry: retry}, nil
|
return retryingNotesNormalizer{warnings: warnings, retry: retry}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterNormalizer() error = %v", err)
|
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||||
|
|||||||
@@ -32,10 +32,6 @@ func NewOutputEncoderRegistry() *OutputEncoderRegistry {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *OutputEncoderRegistry) Register(key string, constructor OutputEncoderConstructor) error {
|
|
||||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageOutput), constructor)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor OutputEncoderConstructor) error {
|
func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor OutputEncoderConstructor) error {
|
||||||
if constructor == nil {
|
if constructor == nil {
|
||||||
return fmt.Errorf("output encoder constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
return fmt.Errorf("output encoder constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
|
|||||||
@@ -17,7 +17,9 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
|
|||||||
return NewOutputEncoderRegistry()
|
return NewOutputEncoderRegistry()
|
||||||
},
|
},
|
||||||
register: func(registry any, key string, constructor func() (contracts.OutputEncoder, error)) error {
|
register: func(registry any, key string, constructor func() (contracts.OutputEncoder, error)) error {
|
||||||
return registry.(*OutputEncoderRegistry).Register(key, constructor)
|
return registry.(*OutputEncoderRegistry).RegisterWithSpec(ModuleSpec{
|
||||||
|
Key: key, Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
}, constructor)
|
||||||
},
|
},
|
||||||
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.OutputEncoder, error)) error {
|
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.OutputEncoder, error)) error {
|
||||||
return registry.(*OutputEncoderRegistry).RegisterWithSpec(spec, constructor)
|
return registry.(*OutputEncoderRegistry).RegisterWithSpec(spec, constructor)
|
||||||
@@ -33,7 +35,9 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
|
|||||||
},
|
},
|
||||||
nilRegister: func(key string, constructor func() (contracts.OutputEncoder, error)) error {
|
nilRegister: func(key string, constructor func() (contracts.OutputEncoder, error)) error {
|
||||||
var registry *OutputEncoderRegistry
|
var registry *OutputEncoderRegistry
|
||||||
return registry.Register(key, constructor)
|
return registry.RegisterWithSpec(ModuleSpec{
|
||||||
|
Key: key, Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
}, constructor)
|
||||||
},
|
},
|
||||||
nilBuild: func(key string) (contracts.OutputEncoder, error) {
|
nilBuild: func(key string) (contracts.OutputEncoder, error) {
|
||||||
var registry *OutputEncoderRegistry
|
var registry *OutputEncoderRegistry
|
||||||
@@ -60,7 +64,7 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
|
|||||||
|
|
||||||
func TestOutputProfileValidationReceivesOwnedOptionsAndLaneIDs(t *testing.T) {
|
func TestOutputProfileValidationReceivesOwnedOptionsAndLaneIDs(t *testing.T) {
|
||||||
registry := NewOutputEncoderRegistry()
|
registry := NewOutputEncoderRegistry()
|
||||||
if err := registry.RegisterBuilderWithProfileValidation(defaultModuleSpec("profile-output", StageOutput), func(options map[string]any) error {
|
if err := registry.RegisterBuilderWithProfileValidation(testModuleSpec("profile-output", StageOutput), func(options map[string]any) error {
|
||||||
options["nested"].(map[string]any)["value"] = "changed"
|
options["nested"].(map[string]any)["value"] = "changed"
|
||||||
return nil
|
return nil
|
||||||
}, func(context OutputProfileOptionContext, options map[string]any) error {
|
}, func(context OutputProfileOptionContext, options map[string]any) error {
|
||||||
|
|||||||
@@ -480,19 +480,19 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
|||||||
if err := RegisterArtifactCodec(registries.ArtifactCodecs, notesCodec()); err != nil {
|
if err := RegisterArtifactCodec(registries.ArtifactCodecs, notesCodec()); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := registries.Inputs.RegisterBuilderWithSpec(defaultModuleSpec("input", StageInput), strict, func(request BuildRequest) (contracts.InputAdapter, error) {
|
if err := registries.Inputs.RegisterBuilderWithSpec(testModuleSpec("input", StageInput), strict, func(request BuildRequest) (contracts.InputAdapter, error) {
|
||||||
record("input", &request)
|
record("input", &request)
|
||||||
return input, nil
|
return input, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := registries.Chunkers.RegisterBuilderWithSpec(defaultModuleSpec("chunk", StageChunk), strict, func(request BuildRequest) (contracts.Chunker, error) {
|
if err := registries.Chunkers.RegisterBuilderWithSpec(testModuleSpec("chunk", StageChunk), strict, func(request BuildRequest) (contracts.Chunker, error) {
|
||||||
record("chunk", &request)
|
record("chunk", &request)
|
||||||
return &typedTestChunker{key: "chunk"}, nil
|
return &typedTestChunker{key: "chunk"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
extractSpec := defaultModuleSpec("extract", StageExtract)
|
extractSpec := testModuleSpec("extract", StageExtract)
|
||||||
extractSpec.ArtifactKind = "test/notes"
|
extractSpec.ArtifactKind = "test/notes"
|
||||||
if err := RegisterExtractorBuilder(registries.Extractors, extractSpec, strict, func(request BuildRequest) (contracts.Extractor[codecNotes], error) {
|
if err := RegisterExtractorBuilder(registries.Extractors, extractSpec, strict, func(request BuildRequest) (contracts.Extractor[codecNotes], error) {
|
||||||
record("extract", &request)
|
record("extract", &request)
|
||||||
@@ -503,7 +503,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
mergeSpec := defaultModuleSpec("merge", StageMerge)
|
mergeSpec := testModuleSpec("merge", StageMerge)
|
||||||
mergeSpec.ArtifactKind = "test/notes"
|
mergeSpec.ArtifactKind = "test/notes"
|
||||||
if err := RegisterMergerBuilder(registries.Mergers, mergeSpec, strict, func(request BuildRequest) (contracts.Merger[codecNotes], error) {
|
if err := RegisterMergerBuilder(registries.Mergers, mergeSpec, strict, func(request BuildRequest) (contracts.Merger[codecNotes], error) {
|
||||||
record("merge", &request)
|
record("merge", &request)
|
||||||
@@ -511,7 +511,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
normalizeSpec := defaultModuleSpec("normalize", StageNormalize)
|
normalizeSpec := testModuleSpec("normalize", StageNormalize)
|
||||||
normalizeSpec.ArtifactKind = "test/notes"
|
normalizeSpec.ArtifactKind = "test/notes"
|
||||||
if err := RegisterNormalizerBuilder(registries.Normalizers, normalizeSpec, strict, func(request BuildRequest) (contracts.Normalizer[codecNotes], error) {
|
if err := RegisterNormalizerBuilder(registries.Normalizers, normalizeSpec, strict, func(request BuildRequest) (contracts.Normalizer[codecNotes], error) {
|
||||||
record("normalize", &request)
|
record("normalize", &request)
|
||||||
@@ -532,7 +532,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := registries.Outputs.RegisterBuilderWithSpec(defaultModuleSpec("output", StageOutput), strict, func(request BuildRequest) (contracts.OutputEncoder, error) {
|
if err := registries.Outputs.RegisterBuilderWithSpec(testModuleSpec("output", StageOutput), strict, func(request BuildRequest) (contracts.OutputEncoder, error) {
|
||||||
record("output", &request)
|
record("output", &request)
|
||||||
if failure.output != nil {
|
if failure.output != nil {
|
||||||
return nil, failure.output
|
return nil, failure.output
|
||||||
|
|||||||
@@ -117,6 +117,7 @@ type PipelineStepProfile struct {
|
|||||||
|
|
||||||
type PipelineProfile struct {
|
type PipelineProfile struct {
|
||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
|
LLMProfile string `json:"llm_profile,omitempty"`
|
||||||
Input ModuleBinding `json:"input"`
|
Input ModuleBinding `json:"input"`
|
||||||
Chunk ModuleBinding `json:"chunk,omitempty"`
|
Chunk ModuleBinding `json:"chunk,omitempty"`
|
||||||
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
|
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
|
||||||
@@ -127,6 +128,7 @@ type PipelineProfile struct {
|
|||||||
|
|
||||||
type ResolveOptions struct {
|
type ResolveOptions struct {
|
||||||
Only []string
|
Only []string
|
||||||
|
LLMProfileOverride string
|
||||||
ReferenceOverrides []ReferenceBinding
|
ReferenceOverrides []ReferenceBinding
|
||||||
ReferenceUnbinds []ReferenceUnbind
|
ReferenceUnbinds []ReferenceUnbind
|
||||||
}
|
}
|
||||||
@@ -156,20 +158,23 @@ type ResolvedReferenceTarget struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type ResolvedArtifactLane struct {
|
type ResolvedArtifactLane struct {
|
||||||
StepID string
|
StepID string
|
||||||
ID string
|
ID string
|
||||||
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
|
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
|
||||||
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
|
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
|
||||||
ArtifactSchemaName string `json:"artifact_schema_name,omitempty"`
|
ArtifactSchemaName string `json:"artifact_schema_name,omitempty"`
|
||||||
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
|
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
|
||||||
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
|
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
|
||||||
Extract ModuleBinding
|
Extract ModuleBinding
|
||||||
Merge ModuleBinding
|
ExtractExecutionClass contracts.ExecutionClass `json:"extract_execution_class"`
|
||||||
Normalize ModuleBinding
|
Merge ModuleBinding
|
||||||
Validators []ModuleBinding
|
MergeExecutionClass contracts.ExecutionClass `json:"merge_execution_class"`
|
||||||
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
|
Normalize ModuleBinding
|
||||||
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
|
NormalizeExecutionClass contracts.ExecutionClass `json:"normalize_execution_class"`
|
||||||
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
|
Validators []ModuleBinding
|
||||||
|
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
|
||||||
|
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
|
||||||
|
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type ResolvedPipelineStep struct {
|
type ResolvedPipelineStep struct {
|
||||||
@@ -192,14 +197,17 @@ type ResolvedValidator struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type ResolvedPipeline struct {
|
type ResolvedPipeline struct {
|
||||||
ID string
|
ID string
|
||||||
Digest string
|
Digest string
|
||||||
Input ModuleBinding
|
Input ModuleBinding
|
||||||
Chunk ModuleBinding
|
InputExecutionClass contracts.ExecutionClass `json:"input_execution_class"`
|
||||||
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
|
Chunk ModuleBinding
|
||||||
Steps []ResolvedPipelineStep
|
ChunkExecutionClass contracts.ExecutionClass `json:"chunk_execution_class"`
|
||||||
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
|
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
|
||||||
Output ModuleBinding
|
Steps []ResolvedPipelineStep
|
||||||
|
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
|
||||||
|
Output ModuleBinding
|
||||||
|
OutputExecutionClass contracts.ExecutionClass `json:"output_execution_class"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// AllArtifactLanes returns lanes in deterministic step order for read-only
|
// AllArtifactLanes returns lanes in deterministic step order for read-only
|
||||||
@@ -225,6 +233,46 @@ type ModuleCatalog struct {
|
|||||||
Outputs *OutputEncoderRegistry
|
Outputs *OutputEncoderRegistry
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ExecutionClass returns the registered execution class for a module selected
|
||||||
|
// by stage and key without constructing the module.
|
||||||
|
func (catalog ModuleCatalog) ExecutionClass(stage ModuleStage, key string) (contracts.ExecutionClass, bool) {
|
||||||
|
var executionClass contracts.ExecutionClass
|
||||||
|
var ok bool
|
||||||
|
|
||||||
|
switch stage {
|
||||||
|
case StageInput:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Inputs.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageChunk:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Chunkers.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageExtract:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Extractors.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageMerge:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Mergers.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageNormalize:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Normalizers.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageValidate:
|
||||||
|
var spec ValidatorSpec
|
||||||
|
spec, ok = catalog.Validators.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageOutput:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Outputs.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
}
|
||||||
|
|
||||||
|
return executionClass, ok
|
||||||
|
}
|
||||||
|
|
||||||
func Binding(module string) ModuleBinding {
|
func Binding(module string) ModuleBinding {
|
||||||
return ModuleBinding{Module: strings.TrimSpace(module)}
|
return ModuleBinding{Module: strings.TrimSpace(module)}
|
||||||
}
|
}
|
||||||
@@ -316,11 +364,13 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
|||||||
return ResolvedPipeline{}, err
|
return ResolvedPipeline{}, err
|
||||||
}
|
}
|
||||||
resolved := ResolvedPipeline{
|
resolved := ResolvedPipeline{
|
||||||
ID: pipelineID,
|
ID: pipelineID,
|
||||||
Input: input,
|
Input: input,
|
||||||
Chunk: chunk,
|
InputExecutionClass: inputModuleSpec.ExecutionClass,
|
||||||
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
|
Chunk: chunk,
|
||||||
Output: resolveBinding(profile.Output, DefaultOutputModule),
|
ChunkExecutionClass: chunkSpec.ExecutionClass,
|
||||||
|
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
|
||||||
|
Output: resolveBinding(profile.Output, DefaultOutputModule),
|
||||||
}
|
}
|
||||||
chunkValidatorChain, err := resolveValidatorChain(pipelineID, "", StageChunk, chunk.Module, chunk.Validators, "", nil, catalog)
|
chunkValidatorChain, err := resolveValidatorChain(pipelineID, "", StageChunk, chunk.Module, chunk.Validators, "", nil, catalog)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -386,6 +436,10 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
|||||||
if missing, ok := outputCapabilities.missing(outputSpec.Requires); ok {
|
if missing, ok := outputCapabilities.missing(outputSpec.Requires); ok {
|
||||||
return ResolvedPipeline{}, capabilityError(pipelineID, "", StageOutput, resolved.Output.Module, missing)
|
return ResolvedPipeline{}, capabilityError(pipelineID, "", StageOutput, resolved.Output.Module, missing)
|
||||||
}
|
}
|
||||||
|
resolved.OutputExecutionClass = outputSpec.ExecutionClass
|
||||||
|
if err := applyEffectiveLLMProfiles(&resolved, profile.LLMProfile, options.LLMProfileOverride); err != nil {
|
||||||
|
return ResolvedPipeline{}, err
|
||||||
|
}
|
||||||
if err := validateResolvedOptions(resolved, catalog, configuredLaneIDs); err != nil {
|
if err := validateResolvedOptions(resolved, catalog, configuredLaneIDs); err != nil {
|
||||||
return ResolvedPipeline{}, err
|
return ResolvedPipeline{}, err
|
||||||
}
|
}
|
||||||
@@ -475,6 +529,7 @@ func resolveArtifactLane(
|
|||||||
}
|
}
|
||||||
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
|
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
|
||||||
lane.ExtractReferences.StepID = strings.TrimSpace(stepID)
|
lane.ExtractReferences.StepID = strings.TrimSpace(stepID)
|
||||||
|
lane.ExtractExecutionClass = extractSpec.ExecutionClass
|
||||||
capabilities.add(extractSpec.Provides...)
|
capabilities.add(extractSpec.Provides...)
|
||||||
|
|
||||||
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
|
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
|
||||||
@@ -499,6 +554,7 @@ func resolveArtifactLane(
|
|||||||
}
|
}
|
||||||
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
|
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
|
||||||
lane.MergeReferences.StepID = strings.TrimSpace(stepID)
|
lane.MergeReferences.StepID = strings.TrimSpace(stepID)
|
||||||
|
lane.MergeExecutionClass = mergeSpec.ExecutionClass
|
||||||
capabilities.add(mergeSpec.Provides...)
|
capabilities.add(mergeSpec.Provides...)
|
||||||
|
|
||||||
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
|
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
|
||||||
@@ -523,6 +579,7 @@ func resolveArtifactLane(
|
|||||||
}
|
}
|
||||||
lane.NormalizeReferences = referenceTarget(StageNormalize, laneID, lane.Normalize.Module, normalizeReferences)
|
lane.NormalizeReferences = referenceTarget(StageNormalize, laneID, lane.Normalize.Module, normalizeReferences)
|
||||||
lane.NormalizeReferences.StepID = strings.TrimSpace(stepID)
|
lane.NormalizeReferences.StepID = strings.TrimSpace(stepID)
|
||||||
|
lane.NormalizeExecutionClass = normalizeSpec.ExecutionClass
|
||||||
capabilities.add(normalizeSpec.Provides...)
|
capabilities.add(normalizeSpec.Provides...)
|
||||||
|
|
||||||
if len(lane.Validators) > 0 {
|
if len(lane.Validators) > 0 {
|
||||||
@@ -784,9 +841,6 @@ func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage,
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q: %w", pipelineID, stage, chain.ModuleKey, err)
|
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q: %w", pipelineID, stage, chain.ModuleKey, err)
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(validator.LLMProfile) != "" && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
|
|
||||||
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q assigns llm_profile to deterministic validator %q", pipelineID, stage, chain.ModuleKey, validator.Module)
|
|
||||||
}
|
|
||||||
chain.Validators = append(chain.Validators, ResolvedValidator{
|
chain.Validators = append(chain.Validators, ResolvedValidator{
|
||||||
Binding: cloneModuleBinding(validator),
|
Binding: cloneModuleBinding(validator),
|
||||||
ExecutionClass: spec.ExecutionClass,
|
ExecutionClass: spec.ExecutionClass,
|
||||||
@@ -1198,6 +1252,66 @@ func resolveBinding(binding ModuleBinding, defaultModule string) ModuleBinding {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func applyEffectiveLLMProfiles(resolved *ResolvedPipeline, pipelineProfile, overrideProfile string) error {
|
||||||
|
pipelineProfile = strings.TrimSpace(pipelineProfile)
|
||||||
|
overrideProfile = strings.TrimSpace(overrideProfile)
|
||||||
|
|
||||||
|
apply := func(stage ModuleStage, laneID, module string, binding *ModuleBinding, executionClass contracts.ExecutionClass, kind string) error {
|
||||||
|
binding.LLMProfile = strings.TrimSpace(binding.LLMProfile)
|
||||||
|
if executionClass != contracts.ExecutionClassLLMBacked {
|
||||||
|
if binding.LLMProfile != "" {
|
||||||
|
if laneID == "" {
|
||||||
|
return fmt.Errorf("pipeline %q %s %q assigns llm_profile to deterministic %s %q", resolved.ID, stage, module, kind, binding.Module)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("pipeline %q lane %q %s %q assigns llm_profile to deterministic %s %q", resolved.ID, laneID, stage, module, kind, binding.Module)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if overrideProfile != "" {
|
||||||
|
binding.LLMProfile = overrideProfile
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if binding.LLMProfile == "" {
|
||||||
|
binding.LLMProfile = pipelineProfile
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := apply(StageInput, "", resolved.Input.Module, &resolved.Input, resolved.InputExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := apply(StageChunk, "", resolved.Chunk.Module, &resolved.Chunk, resolved.ChunkExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for stepIndex := range resolved.Steps {
|
||||||
|
for laneIndex := range resolved.Steps[stepIndex].ArtifactLanes {
|
||||||
|
lane := &resolved.Steps[stepIndex].ArtifactLanes[laneIndex]
|
||||||
|
if err := apply(StageExtract, lane.ID, lane.Extract.Module, &lane.Extract, lane.ExtractExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := apply(StageMerge, lane.ID, lane.Merge.Module, &lane.Merge, lane.MergeExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := apply(StageNormalize, lane.ID, lane.Normalize.Module, &lane.Normalize, lane.NormalizeExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := apply(StageOutput, "", resolved.Output.Module, &resolved.Output, resolved.OutputExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for chainIndex := range resolved.ValidatorChains {
|
||||||
|
chain := &resolved.ValidatorChains[chainIndex]
|
||||||
|
for validatorIndex := range chain.Validators {
|
||||||
|
validator := &chain.Validators[validatorIndex]
|
||||||
|
if err := apply(chain.Stage, chain.LaneID, chain.ModuleKey, &validator.Binding, validator.ExecutionClass, "validator"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func resolveBindings(bindings []ModuleBinding, defaultModule string) []ModuleBinding {
|
func resolveBindings(bindings []ModuleBinding, defaultModule string) []ModuleBinding {
|
||||||
if len(bindings) == 0 {
|
if len(bindings) == 0 {
|
||||||
return nil
|
return nil
|
||||||
@@ -1307,21 +1421,27 @@ func selectedArtifactLanes(pipelineID string, artifacts map[string]ArtifactLaneP
|
|||||||
|
|
||||||
func resolvedPipelineDigest(resolved ResolvedPipeline) (string, error) {
|
func resolvedPipelineDigest(resolved ResolvedPipeline) (string, error) {
|
||||||
withoutDigest := struct {
|
withoutDigest := struct {
|
||||||
ID string
|
ID string
|
||||||
Input ModuleBinding
|
Input ModuleBinding
|
||||||
Chunk ModuleBinding
|
InputExecutionClass contracts.ExecutionClass
|
||||||
ChunkReferences ResolvedReferenceTarget
|
Chunk ModuleBinding
|
||||||
Steps []ResolvedPipelineStep
|
ChunkExecutionClass contracts.ExecutionClass
|
||||||
ValidatorChains []ResolvedValidatorChain
|
ChunkReferences ResolvedReferenceTarget
|
||||||
Output ModuleBinding
|
Steps []ResolvedPipelineStep
|
||||||
|
ValidatorChains []ResolvedValidatorChain
|
||||||
|
Output ModuleBinding
|
||||||
|
OutputExecutionClass contracts.ExecutionClass
|
||||||
}{
|
}{
|
||||||
ID: resolved.ID,
|
ID: resolved.ID,
|
||||||
Input: resolved.Input,
|
Input: resolved.Input,
|
||||||
Chunk: resolved.Chunk,
|
InputExecutionClass: resolved.InputExecutionClass,
|
||||||
ChunkReferences: resolved.ChunkReferences,
|
Chunk: resolved.Chunk,
|
||||||
Steps: resolved.Steps,
|
ChunkExecutionClass: resolved.ChunkExecutionClass,
|
||||||
ValidatorChains: resolved.ValidatorChains,
|
ChunkReferences: resolved.ChunkReferences,
|
||||||
Output: resolved.Output,
|
Steps: resolved.Steps,
|
||||||
|
ValidatorChains: resolved.ValidatorChains,
|
||||||
|
Output: resolved.Output,
|
||||||
|
OutputExecutionClass: resolved.OutputExecutionClass,
|
||||||
}
|
}
|
||||||
encoded, err := json.Marshal(withoutDigest)
|
encoded, err := json.Marshal(withoutDigest)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -15,22 +15,22 @@ import (
|
|||||||
func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
||||||
catalog := newProfileCatalog(t)
|
catalog := newProfileCatalog(t)
|
||||||
registerProfileSpecs(t, catalog,
|
registerProfileSpecs(t, catalog,
|
||||||
ModuleSpec{Key: "window", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
ModuleSpec{Key: "window", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||||
ModuleSpec{Key: "record-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
ModuleSpec{Key: "record-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||||
ModuleSpec{Key: "dedupe", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
ModuleSpec{Key: "dedupe", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||||
ModuleSpec{Key: "canonical", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
ModuleSpec{Key: "canonical", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||||
ModuleSpec{Key: "ndjson", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
ModuleSpec{Key: "ndjson", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||||
)
|
)
|
||||||
|
|
||||||
resolved, err := ResolvePipeline(PipelineProfile{
|
resolved, err := ResolvePipeline(PipelineProfile{
|
||||||
ID: " campaign ",
|
ID: " campaign ",
|
||||||
Input: ModuleBinding{Module: " text ", LLMProfile: " fast "},
|
Input: Binding(" text "),
|
||||||
Chunk: ModuleBinding{Module: " window ", Options: map[string]any{
|
Chunk: ModuleBinding{Module: " window ", Options: map[string]any{
|
||||||
"size": 10,
|
"size": 10,
|
||||||
}},
|
}},
|
||||||
Artifacts: map[string]ArtifactLaneProfile{
|
Artifacts: map[string]ArtifactLaneProfile{
|
||||||
" records ": {
|
" records ": {
|
||||||
Extract: ModuleBinding{Module: " record-extractor ", LLMProfile: " careful "},
|
Extract: Binding(" record-extractor "),
|
||||||
Merge: Binding(" dedupe "),
|
Merge: Binding(" dedupe "),
|
||||||
Normalize: Binding(" canonical "),
|
Normalize: Binding(" canonical "),
|
||||||
},
|
},
|
||||||
@@ -44,15 +44,21 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
|||||||
if resolved.ID != "campaign" {
|
if resolved.ID != "campaign" {
|
||||||
t.Fatalf("ID = %q, want campaign", resolved.ID)
|
t.Fatalf("ID = %q, want campaign", resolved.ID)
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(resolved.Input, ModuleBinding{Module: "text", LLMProfile: "fast"}) {
|
if !reflect.DeepEqual(resolved.Input, ModuleBinding{Module: "text"}) {
|
||||||
t.Fatalf("Input = %#v, want trimmed explicit input", resolved.Input)
|
t.Fatalf("Input = %#v, want trimmed explicit input", resolved.Input)
|
||||||
}
|
}
|
||||||
|
if resolved.InputExecutionClass != contracts.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("InputExecutionClass = %q, want deterministic", resolved.InputExecutionClass)
|
||||||
|
}
|
||||||
if resolved.Chunk.Module != "window" || resolved.Chunk.LLMProfile != "" {
|
if resolved.Chunk.Module != "window" || resolved.Chunk.LLMProfile != "" {
|
||||||
t.Fatalf("Chunk = %#v, want explicit module and empty LLM profile", resolved.Chunk)
|
t.Fatalf("Chunk = %#v, want explicit module and empty LLM profile", resolved.Chunk)
|
||||||
}
|
}
|
||||||
if resolved.Chunk.Options["size"] != 10 {
|
if resolved.Chunk.Options["size"] != 10 {
|
||||||
t.Fatalf("Chunk.Options = %#v, want size option", resolved.Chunk.Options)
|
t.Fatalf("Chunk.Options = %#v, want size option", resolved.Chunk.Options)
|
||||||
}
|
}
|
||||||
|
if resolved.ChunkExecutionClass != contracts.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("ChunkExecutionClass = %q, want deterministic", resolved.ChunkExecutionClass)
|
||||||
|
}
|
||||||
if len(resolved.Steps) != 1 || resolved.Steps[0].ID != "default" || len(resolved.Steps[0].ArtifactLanes) != 1 {
|
if len(resolved.Steps) != 1 || resolved.Steps[0].ID != "default" || len(resolved.Steps[0].ArtifactLanes) != 1 {
|
||||||
t.Fatalf("resolved steps = %#v, want one default step with one lane", resolved.Steps)
|
t.Fatalf("resolved steps = %#v, want one default step with one lane", resolved.Steps)
|
||||||
}
|
}
|
||||||
@@ -60,9 +66,12 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
|||||||
if lane.ID != "records" {
|
if lane.ID != "records" {
|
||||||
t.Fatalf("lane.ID = %q, want records", lane.ID)
|
t.Fatalf("lane.ID = %q, want records", lane.ID)
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(lane.Extract, ModuleBinding{Module: "record-extractor", LLMProfile: "careful"}) {
|
if !reflect.DeepEqual(lane.Extract, ModuleBinding{Module: "record-extractor"}) {
|
||||||
t.Fatalf("lane.Extract = %#v, want explicit extractor", lane.Extract)
|
t.Fatalf("lane.Extract = %#v, want explicit extractor", lane.Extract)
|
||||||
}
|
}
|
||||||
|
if lane.ExtractExecutionClass != contracts.ExecutionClassDeterministic || lane.MergeExecutionClass != contracts.ExecutionClassDeterministic || lane.NormalizeExecutionClass != contracts.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("lane execution classes = %q/%q/%q, want deterministic", lane.ExtractExecutionClass, lane.MergeExecutionClass, lane.NormalizeExecutionClass)
|
||||||
|
}
|
||||||
if lane.Merge.Module != "dedupe" || lane.Normalize.Module != "canonical" {
|
if lane.Merge.Module != "dedupe" || lane.Normalize.Module != "canonical" {
|
||||||
t.Fatalf("lane merge/normalize = %#v/%#v, want explicit modules", lane.Merge, lane.Normalize)
|
t.Fatalf("lane merge/normalize = %#v/%#v, want explicit modules", lane.Merge, lane.Normalize)
|
||||||
}
|
}
|
||||||
@@ -72,11 +81,51 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
|||||||
if resolved.Output.Module != "ndjson" {
|
if resolved.Output.Module != "ndjson" {
|
||||||
t.Fatalf("Output.Module = %q, want ndjson", resolved.Output.Module)
|
t.Fatalf("Output.Module = %q, want ndjson", resolved.Output.Module)
|
||||||
}
|
}
|
||||||
|
if resolved.OutputExecutionClass != contracts.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("OutputExecutionClass = %q, want deterministic", resolved.OutputExecutionClass)
|
||||||
|
}
|
||||||
if !strings.HasPrefix(resolved.Digest, "sha256:") {
|
if !strings.HasPrefix(resolved.Digest, "sha256:") {
|
||||||
t.Fatalf("Digest = %q, want sha256 digest", resolved.Digest)
|
t.Fatalf("Digest = %q, want sha256 digest", resolved.Digest)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestModuleCatalogExecutionClassLooksUpRegisteredMetadata(t *testing.T) {
|
||||||
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
|
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
ModuleSpec{Key: "llm-extract", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
)
|
||||||
|
registerProfileValidatorSpec(t, catalog, ValidatorSpec{Key: "llm-validator", ExecutionClass: contracts.ExecutionClassLLMBacked})
|
||||||
|
|
||||||
|
for _, test := range []struct {
|
||||||
|
stage ModuleStage
|
||||||
|
key string
|
||||||
|
want contracts.ExecutionClass
|
||||||
|
}{
|
||||||
|
{stage: StageInput, key: "llm-input", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageChunk, key: "llm-chunk", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageExtract, key: "llm-extract", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageMerge, key: "llm-merge", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageNormalize, key: "llm-normalize", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageValidate, key: "llm-validator", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageOutput, key: "llm-output", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
} {
|
||||||
|
t.Run(string(test.stage), func(t *testing.T) {
|
||||||
|
got, ok := catalog.ExecutionClass(test.stage, test.key)
|
||||||
|
if !ok || got != test.want {
|
||||||
|
t.Fatalf("ExecutionClass(%q, %q) = %q, %t; want %q, true", test.stage, test.key, got, ok, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, ok := catalog.ExecutionClass(StageExtract, "missing"); ok {
|
||||||
|
t.Fatal("ExecutionClass() found an unregistered module")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestResolvePipelineAppliesDefaults(t *testing.T) {
|
func TestResolvePipelineAppliesDefaults(t *testing.T) {
|
||||||
resolved, err := ResolvePipeline(PipelineProfile{
|
resolved, err := ResolvePipeline(PipelineProfile{
|
||||||
ID: "defaulted",
|
ID: "defaulted",
|
||||||
@@ -110,6 +159,176 @@ func TestResolvePipelineAppliesDefaults(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestResolvePipelineAppliesEffectiveLLMProfiles(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
profile PipelineProfile
|
||||||
|
options ResolveOptions
|
||||||
|
want map[string]string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "runtime override",
|
||||||
|
profile: func() PipelineProfile {
|
||||||
|
profile := llmProfilePipeline()
|
||||||
|
profile.LLMProfile = " pipeline "
|
||||||
|
profile.Chunk.LLMProfile = "binding"
|
||||||
|
return profile
|
||||||
|
}(),
|
||||||
|
options: ResolveOptions{LLMProfileOverride: " runtime "},
|
||||||
|
want: llmProfileValues("runtime"),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "binding exception",
|
||||||
|
profile: func() PipelineProfile {
|
||||||
|
profile := llmProfilePipeline()
|
||||||
|
profile.LLMProfile = "pipeline"
|
||||||
|
lane := profile.Artifacts["events"]
|
||||||
|
lane.Extract.LLMProfile = "extract"
|
||||||
|
lane.Extract.Validators = ValidatorOverride{
|
||||||
|
Set: true,
|
||||||
|
Validators: []ModuleBinding{{Module: "llm-validator", LLMProfile: "validator"}},
|
||||||
|
}
|
||||||
|
profile.Artifacts["events"] = lane
|
||||||
|
return profile
|
||||||
|
}(),
|
||||||
|
want: func() map[string]string {
|
||||||
|
values := llmProfileValues("pipeline")
|
||||||
|
values["extract"] = "extract"
|
||||||
|
values["validator:extract:events"] = "validator"
|
||||||
|
return values
|
||||||
|
}(),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "pipeline default",
|
||||||
|
profile: func() PipelineProfile {
|
||||||
|
profile := llmProfilePipeline()
|
||||||
|
profile.LLMProfile = "pipeline"
|
||||||
|
return profile
|
||||||
|
}(),
|
||||||
|
want: llmProfileValues("pipeline"),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "prompt fallback",
|
||||||
|
profile: llmProfilePipeline(),
|
||||||
|
want: llmProfileValues(""),
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
resolved, err := ResolvePipeline(test.profile, test.options, llmProfileCatalog(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if got := resolvedLLMProfileValues(resolved); !reflect.DeepEqual(got, test.want) {
|
||||||
|
t.Fatalf("resolved profiles = %#v, want %#v", got, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePipelineAppliesProfilesOnlyToSelectedLLMBackedBindings(t *testing.T) {
|
||||||
|
profile := llmProfilePipeline()
|
||||||
|
profile.LLMProfile = "pipeline"
|
||||||
|
profile.Artifacts["notes"] = ArtifactLaneProfile{Extract: Binding("llm-extractor")}
|
||||||
|
|
||||||
|
resolved, err := ResolvePipeline(profile, ResolveOptions{Only: []string{"events"}}, llmProfileCatalog(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if got := laneIDs(resolved.Steps[0].ArtifactLanes); !reflect.DeepEqual(got, []string{"events"}) {
|
||||||
|
t.Fatalf("selected lanes = %#v, want events only", got)
|
||||||
|
}
|
||||||
|
if got := resolvedLLMProfileValues(resolved); !reflect.DeepEqual(got, llmProfileValues("pipeline")) {
|
||||||
|
t.Fatalf("resolved profiles = %#v, want selected LLM bindings only", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePipelineLeavesUnusedProfilesOffDeterministicBindings(t *testing.T) {
|
||||||
|
profile := baselineProfile()
|
||||||
|
profile.LLMProfile = "unused"
|
||||||
|
|
||||||
|
resolved, err := ResolvePipeline(profile, ResolveOptions{LLMProfileOverride: "also-unused"}, newProfileCatalog(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if resolved.Input.LLMProfile != "" || resolved.Chunk.LLMProfile != "" || resolved.Output.LLMProfile != "" {
|
||||||
|
t.Fatalf("deterministic pipeline profiles = input %q chunk %q output %q, want empty", resolved.Input.LLMProfile, resolved.Chunk.LLMProfile, resolved.Output.LLMProfile)
|
||||||
|
}
|
||||||
|
lane := resolved.Steps[0].ArtifactLanes[0]
|
||||||
|
if lane.Extract.LLMProfile != "" || lane.Merge.LLMProfile != "" || lane.Normalize.LLMProfile != "" {
|
||||||
|
t.Fatalf("deterministic lane profiles = extract %q merge %q normalize %q, want empty", lane.Extract.LLMProfile, lane.Merge.LLMProfile, lane.Normalize.LLMProfile)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePipelineRejectsLLMProfileForDeterministicModule(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
mutate func(*PipelineProfile)
|
||||||
|
}{
|
||||||
|
{name: "input", mutate: func(profile *PipelineProfile) { profile.Input.LLMProfile = "invalid" }},
|
||||||
|
{name: "chunk", mutate: func(profile *PipelineProfile) { profile.Chunk.LLMProfile = "invalid" }},
|
||||||
|
{name: "extract", mutate: func(profile *PipelineProfile) {
|
||||||
|
lane := profile.Artifacts["events"]
|
||||||
|
lane.Extract.LLMProfile = "invalid"
|
||||||
|
profile.Artifacts["events"] = lane
|
||||||
|
}},
|
||||||
|
{name: "merge", mutate: func(profile *PipelineProfile) {
|
||||||
|
lane := profile.Artifacts["events"]
|
||||||
|
lane.Merge.LLMProfile = "invalid"
|
||||||
|
profile.Artifacts["events"] = lane
|
||||||
|
}},
|
||||||
|
{name: "normalize", mutate: func(profile *PipelineProfile) {
|
||||||
|
lane := profile.Artifacts["events"]
|
||||||
|
lane.Normalize.LLMProfile = "invalid"
|
||||||
|
profile.Artifacts["events"] = lane
|
||||||
|
}},
|
||||||
|
{name: "output", mutate: func(profile *PipelineProfile) { profile.Output.LLMProfile = "invalid" }},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
profile := baselineProfile()
|
||||||
|
test.mutate(&profile)
|
||||||
|
_, err := ResolvePipeline(profile, ResolveOptions{}, newProfileCatalog(t))
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "llm_profile") || !strings.Contains(err.Error(), "deterministic") {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want deterministic profile rejection", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePipelineDigestUsesEffectiveLLMProfiles(t *testing.T) {
|
||||||
|
inherited := llmProfilePipeline()
|
||||||
|
inherited.LLMProfile = "shared"
|
||||||
|
explicit := llmProfilePipeline()
|
||||||
|
explicit.Input.LLMProfile = "shared"
|
||||||
|
explicit.Chunk.LLMProfile = "shared"
|
||||||
|
explicit.Output.LLMProfile = "shared"
|
||||||
|
lane := explicit.Artifacts["events"]
|
||||||
|
lane.Extract.LLMProfile = "shared"
|
||||||
|
lane.Merge.LLMProfile = "shared"
|
||||||
|
lane.Normalize.LLMProfile = "shared"
|
||||||
|
explicit.Artifacts["events"] = lane
|
||||||
|
|
||||||
|
inheritedResolved, err := ResolvePipeline(inherited, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline(inherited) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
explicitResolved, err := ResolvePipeline(explicit, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline(explicit) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if inheritedResolved.Digest != explicitResolved.Digest {
|
||||||
|
t.Fatalf("effective profile digests differ: %q != %q", inheritedResolved.Digest, explicitResolved.Digest)
|
||||||
|
}
|
||||||
|
|
||||||
|
explicit.Chunk.LLMProfile = "different"
|
||||||
|
changedResolved, err := ResolvePipeline(explicit, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline(changed) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if explicitResolved.Digest == changedResolved.Digest {
|
||||||
|
t.Fatalf("digest = %q after effective profile change, want different", explicitResolved.Digest)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestResolvePipelineRecordsValidatorChains(t *testing.T) {
|
func TestResolvePipelineRecordsValidatorChains(t *testing.T) {
|
||||||
catalog := newProfileCatalog(t)
|
catalog := newProfileCatalog(t)
|
||||||
if err := catalog.ValidatorChains.Register(ValidatorChainMapping{
|
if err := catalog.ValidatorChains.Register(ValidatorChainMapping{
|
||||||
@@ -337,7 +556,8 @@ func TestResolvePipelinePreservesOrderedStepsAndExpandsGeneratedBindings(t *test
|
|||||||
catalog := newProfileCatalogWithOverrides(t,
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
ModuleSpec{
|
ModuleSpec{
|
||||||
Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes",
|
Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes",
|
||||||
Requires: []string{"chunk"}, Provides: []string{"candidate"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
Requires: []string{"chunk"}, Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}, AcceptedMediaTypes: []string{"application/json"}}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}, AcceptedMediaTypes: []string{"application/json"}}},
|
||||||
})
|
})
|
||||||
resolved, err := ResolvePipeline(PipelineProfile{
|
resolved, err := ResolvePipeline(PipelineProfile{
|
||||||
@@ -365,7 +585,7 @@ func TestResolvePipelinePreservesOrderedStepsAndExpandsGeneratedBindings(t *test
|
|||||||
|
|
||||||
func TestResolvePipelineRejectsGeneratedBindingOrderingAndKind(t *testing.T) {
|
func TestResolvePipelineRejectsGeneratedBindingOrderingAndKind(t *testing.T) {
|
||||||
catalog := newProfileCatalogWithOverrides(t,
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/other"}}}},
|
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/other"}}}},
|
||||||
)
|
)
|
||||||
_, err := ResolvePipeline(PipelineProfile{
|
_, err := ResolvePipeline(PipelineProfile{
|
||||||
ID: "invalid-order", Input: Binding("text"), Steps: []PipelineStepProfile{
|
ID: "invalid-order", Input: Binding("text"), Steps: []PipelineStepProfile{
|
||||||
@@ -400,10 +620,11 @@ func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
|
|||||||
profile.Artifacts["events"] = lane
|
profile.Artifacts["events"] = lane
|
||||||
|
|
||||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
Requires: []string{"chunk"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"candidate"},
|
Requires: []string{"chunk"},
|
||||||
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster", Required: true},
|
{Name: "roster", Required: true},
|
||||||
{Name: "lore"},
|
{Name: "lore"},
|
||||||
@@ -446,6 +667,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToChunkTarget(t *testing.
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "generic",
|
Key: "generic",
|
||||||
Stage: StageChunk,
|
Stage: StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"source"},
|
Requires: []string{"source"},
|
||||||
Provides: []string{"chunk"},
|
Provides: []string{"chunk"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide"}},
|
||||||
@@ -474,6 +696,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToExtractorTarget(t *test
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"},
|
Requires: []string{"chunk"},
|
||||||
Provides: []string{"candidate"},
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
|
||||||
@@ -502,6 +725,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToNormalizerTarget(t *tes
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "noop",
|
Key: "noop",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"merged"},
|
Requires: []string{"merged"},
|
||||||
Provides: []string{"normalized"},
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
||||||
@@ -530,6 +754,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToMergeTarget(t *testing.
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "appendorder",
|
Key: "appendorder",
|
||||||
Stage: StageMerge,
|
Stage: StageMerge,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"candidate"},
|
Requires: []string{"candidate"},
|
||||||
Provides: []string{"merged"},
|
Provides: []string{"merged"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "merge_notes"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "merge_notes"}},
|
||||||
@@ -559,10 +784,10 @@ func TestResolvePipelineAppliesOnePipelineReferenceDefaultToMultipleTargets(t *t
|
|||||||
profile := baselineProfile()
|
profile := baselineProfile()
|
||||||
profile.References = ExternalReferenceMap(map[string]string{"context": "./context.md"})
|
profile.References = ExternalReferenceMap(map[string]string{"context": "./context.md"})
|
||||||
catalog := newProfileCatalogWithOverrides(t,
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||||
ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"candidate"}, Provides: []string{"merged"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||||
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||||
)
|
)
|
||||||
|
|
||||||
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
||||||
@@ -580,10 +805,11 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedLane(t *t
|
|||||||
profile := multiLaneProfile()
|
profile := multiLaneProfile()
|
||||||
profile.References = ExternalReferenceMap(map[string]string{"notes_context": "./notes.md"})
|
profile.References = ExternalReferenceMap(map[string]string{"notes_context": "./notes.md"})
|
||||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||||
Key: "note-extractor",
|
Key: "note-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
Requires: []string{"chunk"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"candidate"},
|
Requires: []string{"chunk"},
|
||||||
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "notes_context"},
|
{Name: "notes_context"},
|
||||||
},
|
},
|
||||||
@@ -605,10 +831,11 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedNormalize
|
|||||||
lane.Normalize = Binding("note-normalizer")
|
lane.Normalize = Binding("note-normalizer")
|
||||||
profile.Artifacts["notes"] = lane
|
profile.Artifacts["notes"] = lane
|
||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "note-normalizer",
|
Key: "note-normalizer",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
Requires: []string{"merged"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"normalized"},
|
Requires: []string{"merged"},
|
||||||
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "notes_context"},
|
{Name: "notes_context"},
|
||||||
},
|
},
|
||||||
@@ -658,6 +885,7 @@ func TestResolvePipelineRejectsExtractLocalReferenceDeclaredOnlyByNormalizer(t *
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "noop",
|
Key: "noop",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"merged"},
|
Requires: []string{"merged"},
|
||||||
Provides: []string{"normalized"},
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
||||||
@@ -676,10 +904,11 @@ func TestResolvePipelineRejectsMergeLocalReferenceDeclaredOnlyByNormalizer(t *te
|
|||||||
lane.Merge.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./normalize.md"})
|
lane.Merge.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./normalize.md"})
|
||||||
profile.Artifacts["events"] = lane
|
profile.Artifacts["events"] = lane
|
||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "noop",
|
Key: "noop",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
Requires: []string{"merged"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"normalized"},
|
Requires: []string{"merged"},
|
||||||
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "normalization_notes"},
|
{Name: "normalization_notes"},
|
||||||
},
|
},
|
||||||
@@ -700,6 +929,7 @@ func TestResolvePipelineRejectsNormalizeLocalReferenceDeclaredOnlyByExtractor(t
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"},
|
Requires: []string{"chunk"},
|
||||||
Provides: []string{"candidate"},
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
|
||||||
@@ -716,6 +946,7 @@ func TestResolvePipelineRequiresBoundChunkReference(t *testing.T) {
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "generic",
|
Key: "generic",
|
||||||
Stage: StageChunk,
|
Stage: StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"source"},
|
Requires: []string{"source"},
|
||||||
Provides: []string{"chunk"},
|
Provides: []string{"chunk"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide", Required: true}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide", Required: true}},
|
||||||
@@ -732,6 +963,7 @@ func TestResolvePipelineRequiresBoundNormalizeReference(t *testing.T) {
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "noop",
|
Key: "noop",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"merged"},
|
Requires: []string{"merged"},
|
||||||
Provides: []string{"normalized"},
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes", Required: true}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes", Required: true}},
|
||||||
@@ -757,9 +989,9 @@ func TestResolvePipelineLocalReferencesOverridePipelineDefaultsForEligibleTarget
|
|||||||
lane.Normalize.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./local-normalize.md"})
|
lane.Normalize.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./local-normalize.md"})
|
||||||
profile.Artifacts["events"] = lane
|
profile.Artifacts["events"] = lane
|
||||||
catalog := newProfileCatalogWithOverrides(t,
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}}},
|
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}}},
|
||||||
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}}},
|
ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}}},
|
||||||
)
|
)
|
||||||
|
|
||||||
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
||||||
@@ -774,10 +1006,11 @@ func TestResolvePipelineLocalReferencesOverridePipelineDefaultsForEligibleTarget
|
|||||||
|
|
||||||
func TestResolvePipelineRequiresBoundReferenceSlotsForSelectedLanes(t *testing.T) {
|
func TestResolvePipelineRequiresBoundReferenceSlotsForSelectedLanes(t *testing.T) {
|
||||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
Requires: []string{"chunk"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"candidate"},
|
Requires: []string{"chunk"},
|
||||||
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster", Required: true},
|
{Name: "roster", Required: true},
|
||||||
},
|
},
|
||||||
@@ -798,10 +1031,11 @@ func TestResolvePipelineReferenceUnbindCanLeaveRequiredSlotMissing(t *testing.T)
|
|||||||
profile := baselineProfile()
|
profile := baselineProfile()
|
||||||
profile.References = ExternalReferenceMap(map[string]string{"roster": "./roster.yml"})
|
profile.References = ExternalReferenceMap(map[string]string{"roster": "./roster.yml"})
|
||||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
Requires: []string{"chunk"},
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"candidate"},
|
Requires: []string{"chunk"},
|
||||||
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster", Required: true},
|
{Name: "roster", Required: true},
|
||||||
},
|
},
|
||||||
@@ -827,11 +1061,12 @@ func TestResolvePipelineUsesReferenceSlotsFromSpecWithoutConstructingExtractor(t
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
if err := RegisterExtractor[codecNotes](catalog.Extractors, ModuleSpec{
|
if err := RegisterExtractor[codecNotes](catalog.Extractors, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
ArtifactKind: "test/notes",
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"},
|
ArtifactKind: "test/notes",
|
||||||
Provides: []string{"candidate"},
|
Requires: []string{"chunk"},
|
||||||
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster", Required: true},
|
{Name: "roster", Required: true},
|
||||||
},
|
},
|
||||||
@@ -989,32 +1224,32 @@ func TestResolvePipelineRejectsMissingCapabilities(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "input",
|
name: "input",
|
||||||
spec: ModuleSpec{Key: "text", Stage: StageInput, Requires: []string{"raw"}},
|
spec: ModuleSpec{Key: "text", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"raw"}},
|
||||||
want: []string{"baseline", "input", "text", "raw"},
|
want: []string{"baseline", "input", "text", "raw"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "chunk",
|
name: "chunk",
|
||||||
spec: ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"missing"}},
|
spec: ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||||
want: []string{"baseline", "chunk", "generic", "missing"},
|
want: []string{"baseline", "chunk", "generic", "missing"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "extract",
|
name: "extract",
|
||||||
spec: ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"missing"}},
|
spec: ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||||
want: []string{"baseline", "events", "extract", "event-extractor", "missing"},
|
want: []string{"baseline", "events", "extract", "event-extractor", "missing"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "merge",
|
name: "merge",
|
||||||
spec: ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"missing"}},
|
spec: ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||||
want: []string{"baseline", "events", "merge", "appendorder", "missing"},
|
want: []string{"baseline", "events", "merge", "appendorder", "missing"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "normalize",
|
name: "normalize",
|
||||||
spec: ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"missing"}},
|
spec: ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||||
want: []string{"baseline", "events", "normalize", "noop", "missing"},
|
want: []string{"baseline", "events", "normalize", "noop", "missing"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "output",
|
name: "output",
|
||||||
spec: ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"missing"}},
|
spec: ModuleSpec{Key: "json", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||||
want: []string{"baseline", "output", "json", "missing"},
|
want: []string{"baseline", "output", "json", "missing"},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -1100,7 +1335,7 @@ func TestResolvePipelineDigestChangesWhenBindingChanges(t *testing.T) {
|
|||||||
right := baselineProfile()
|
right := baselineProfile()
|
||||||
right.Chunk = Binding("window")
|
right.Chunk = Binding("window")
|
||||||
catalog := newProfileCatalog(t)
|
catalog := newProfileCatalog(t)
|
||||||
registerProfileSpecs(t, catalog, ModuleSpec{Key: "window", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}})
|
registerProfileSpecs(t, catalog, ModuleSpec{Key: "window", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}})
|
||||||
|
|
||||||
leftResolved, err := ResolvePipeline(left, ResolveOptions{}, catalog)
|
leftResolved, err := ResolvePipeline(left, ResolveOptions{}, catalog)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1467,17 +1702,103 @@ func emptyProfileCatalog() ModuleCatalog {
|
|||||||
|
|
||||||
func defaultProfileSpecs() []ModuleSpec {
|
func defaultProfileSpecs() []ModuleSpec {
|
||||||
return []ModuleSpec{
|
return []ModuleSpec{
|
||||||
ModuleSpec{Key: "text", Stage: StageInput, Provides: []string{"source"}},
|
ModuleSpec{Key: "text", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}},
|
||||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||||
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||||
ModuleSpec{Key: "appendorder", Stage: StageMerge, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||||
ModuleSpec{Key: "noop", Stage: StageNormalize, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||||
ModuleSpec{Key: "grounded", Stage: StageValidate, Requires: []string{"normalized"}, Provides: []string{"validated"}},
|
ModuleSpec{Key: "grounded", Stage: StageValidate, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"validated"}},
|
||||||
ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
ModuleSpec{Key: "json", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func llmProfileCatalog(t *testing.T) ModuleCatalog {
|
||||||
|
t.Helper()
|
||||||
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
|
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked, Provides: []string{"source"}},
|
||||||
|
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||||
|
ModuleSpec{Key: "llm-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||||
|
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||||
|
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||||
|
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||||
|
)
|
||||||
|
if err := RegisterChunkValidator(catalog.Validators, ValidatorSpec{Key: "llm-chunk-validator", ExecutionClass: contracts.ExecutionClassLLMBacked}, func() (contracts.ChunkValidator, error) {
|
||||||
|
return typedTestChunkValidator{key: "llm-chunk-validator"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register chunk validator: %v", err)
|
||||||
|
}
|
||||||
|
registerProfileValidatorSpec(t, catalog, ValidatorSpec{Key: "llm-validator", ExecutionClass: contracts.ExecutionClassLLMBacked})
|
||||||
|
for _, mapping := range []ValidatorChainMapping{
|
||||||
|
{Stage: StageChunk, Module: "llm-chunk", Validators: []ModuleBinding{Binding("llm-chunk-validator")}},
|
||||||
|
{Stage: StageExtract, Module: "llm-extractor", Validators: []ModuleBinding{Binding("llm-validator")}},
|
||||||
|
{Stage: StageMerge, Module: "llm-merge", Validators: []ModuleBinding{Binding("llm-validator")}},
|
||||||
|
{Stage: StageNormalize, Module: "llm-normalize", Validators: []ModuleBinding{Binding("llm-validator")}},
|
||||||
|
} {
|
||||||
|
if err := catalog.ValidatorChains.Register(mapping); err != nil {
|
||||||
|
t.Fatalf("register validator chain %#v: %v", mapping, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return catalog
|
||||||
|
}
|
||||||
|
|
||||||
|
func llmProfileCatalogWithoutValidatorChains(t *testing.T) ModuleCatalog {
|
||||||
|
catalog := llmProfileCatalog(t)
|
||||||
|
catalog.ValidatorChains = NewValidatorChainRegistry()
|
||||||
|
return catalog
|
||||||
|
}
|
||||||
|
|
||||||
|
func llmProfilePipeline() PipelineProfile {
|
||||||
|
return PipelineProfile{
|
||||||
|
ID: "llm-profile",
|
||||||
|
Input: Binding("llm-input"),
|
||||||
|
Chunk: Binding("llm-chunk"),
|
||||||
|
Output: Binding("llm-output"),
|
||||||
|
Artifacts: map[string]ArtifactLaneProfile{
|
||||||
|
"events": {
|
||||||
|
Extract: Binding("llm-extractor"),
|
||||||
|
Merge: Binding("llm-merge"),
|
||||||
|
Normalize: Binding("llm-normalize"),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func llmProfileValues(profile string) map[string]string {
|
||||||
|
return map[string]string{
|
||||||
|
"input": profile,
|
||||||
|
"chunk": profile,
|
||||||
|
"extract": profile,
|
||||||
|
"merge": profile,
|
||||||
|
"normalize": profile,
|
||||||
|
"output": profile,
|
||||||
|
"validator:chunk:": profile,
|
||||||
|
"validator:extract:events": profile,
|
||||||
|
"validator:merge:events": profile,
|
||||||
|
"validator:normalize:events": profile,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolvedLLMProfileValues(resolved ResolvedPipeline) map[string]string {
|
||||||
|
values := map[string]string{
|
||||||
|
"input": resolved.Input.LLMProfile,
|
||||||
|
"chunk": resolved.Chunk.LLMProfile,
|
||||||
|
"output": resolved.Output.LLMProfile,
|
||||||
|
}
|
||||||
|
lane := resolved.Steps[0].ArtifactLanes[0]
|
||||||
|
values["extract"] = lane.Extract.LLMProfile
|
||||||
|
values["merge"] = lane.Merge.LLMProfile
|
||||||
|
values["normalize"] = lane.Normalize.LLMProfile
|
||||||
|
for _, chain := range resolved.ValidatorChains {
|
||||||
|
for _, validator := range chain.Validators {
|
||||||
|
if validator.ExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
|
values["validator:"+string(chain.Stage)+":"+chain.LaneID] = validator.Binding.LLMProfile
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return values
|
||||||
|
}
|
||||||
|
|
||||||
func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSpec) {
|
func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSpec) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -1566,7 +1887,13 @@ func TestResolvedPipelineCanMarshalToCanonicalJSON(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
if _, err := json.Marshal(resolved); err != nil {
|
encoded, err := json.Marshal(resolved)
|
||||||
|
if err != nil {
|
||||||
t.Fatalf("json.Marshal(resolved) error = %v, want nil", err)
|
t.Fatalf("json.Marshal(resolved) error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
for _, field := range []string{"input_execution_class", "chunk_execution_class", "extract_execution_class", "merge_execution_class", "normalize_execution_class", "output_execution_class"} {
|
||||||
|
if !strings.Contains(string(encoded), `"`+field+`":"deterministic"`) {
|
||||||
|
t.Fatalf("resolved JSON does not retain %q: %s", field, encoded)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -189,12 +189,12 @@ func TestMaterializeReferencesUsesLaneArtifactVariant(t *testing.T) {
|
|||||||
{kind: "test/beta", mergeSlot: "beta_merge", normalizeSlot: "beta_normalize"},
|
{kind: "test/beta", mergeSlot: "beta_merge", normalizeSlot: "beta_normalize"},
|
||||||
{kind: "test/alpha", mergeSlot: "alpha_merge", normalizeSlot: "alpha_normalize"},
|
{kind: "test/alpha", mergeSlot: "alpha_merge", normalizeSlot: "alpha_normalize"},
|
||||||
} {
|
} {
|
||||||
if err := RegisterMerger(mergers, ModuleSpec{Key: "shared/merge", Stage: StageMerge, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.mergeSlot}}}, func() (contracts.Merger[codecNotes], error) {
|
if err := RegisterMerger(mergers, ModuleSpec{Key: "shared/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.mergeSlot}}}, func() (contracts.Merger[codecNotes], error) {
|
||||||
return typedTestMerger[codecNotes]{key: "shared/merge"}, nil
|
return typedTestMerger[codecNotes]{key: "shared/merge"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterMerger(%s): %v", item.kind, err)
|
t.Fatalf("RegisterMerger(%s): %v", item.kind, err)
|
||||||
}
|
}
|
||||||
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "shared/normalize", Stage: StageNormalize, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.normalizeSlot}}}, func() (contracts.Normalizer[codecNotes], error) {
|
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "shared/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.normalizeSlot}}}, func() (contracts.Normalizer[codecNotes], error) {
|
||||||
return typedTestNormalizer[codecNotes]{key: "shared/normalize"}, nil
|
return typedTestNormalizer[codecNotes]{key: "shared/normalize"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterNormalizer(%s): %v", item.kind, err)
|
t.Fatalf("RegisterNormalizer(%s): %v", item.kind, err)
|
||||||
@@ -531,6 +531,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
|||||||
ModuleSpec{
|
ModuleSpec{
|
||||||
Key: "generic",
|
Key: "generic",
|
||||||
Stage: StageChunk,
|
Stage: StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"source"},
|
Requires: []string{"source"},
|
||||||
Provides: []string{"chunk"},
|
Provides: []string{"chunk"},
|
||||||
ReferenceSlots: chunkSlots,
|
ReferenceSlots: chunkSlots,
|
||||||
@@ -538,6 +539,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
|||||||
ModuleSpec{
|
ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"},
|
Requires: []string{"chunk"},
|
||||||
Provides: []string{"candidate"},
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: extractSlots,
|
ReferenceSlots: extractSlots,
|
||||||
@@ -545,6 +547,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
|||||||
ModuleSpec{
|
ModuleSpec{
|
||||||
Key: "appendorder",
|
Key: "appendorder",
|
||||||
Stage: StageMerge,
|
Stage: StageMerge,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"candidate"},
|
Requires: []string{"candidate"},
|
||||||
Provides: []string{"merged"},
|
Provides: []string{"merged"},
|
||||||
ReferenceSlots: mergeSlots,
|
ReferenceSlots: mergeSlots,
|
||||||
@@ -552,6 +555,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
|||||||
ModuleSpec{
|
ModuleSpec{
|
||||||
Key: "noop",
|
Key: "noop",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"merged"},
|
Requires: []string{"merged"},
|
||||||
Provides: []string{"normalized"},
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: normalizeSlots,
|
ReferenceSlots: normalizeSlots,
|
||||||
|
|||||||
@@ -829,18 +829,12 @@ func mergeLLMProfileManifests(sources ...[]artifacts.LLMProfileManifest) []artif
|
|||||||
merged := make(map[string]artifacts.LLMProfileManifest)
|
merged := make(map[string]artifacts.LLMProfileManifest)
|
||||||
for _, source := range sources {
|
for _, source := range sources {
|
||||||
for _, profile := range source {
|
for _, profile := range source {
|
||||||
id := strings.TrimSpace(profile.ID)
|
profile = profile.Normalized()
|
||||||
provider := strings.TrimSpace(profile.Provider)
|
key := profile.IdentityKey()
|
||||||
model := strings.TrimSpace(profile.Model)
|
|
||||||
key := id + "\x00" + provider + "\x00" + model
|
|
||||||
if _, exists := merged[key]; exists {
|
if _, exists := merged[key]; exists {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
merged[key] = artifacts.LLMProfileManifest{
|
merged[key] = profile
|
||||||
ID: id,
|
|
||||||
Provider: provider,
|
|
||||||
Model: model,
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if len(merged) == 0 {
|
if len(merged) == 0 {
|
||||||
|
|||||||
@@ -116,7 +116,7 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
|
|||||||
return false, nil, terminal.record(nil, fmt.Errorf("clone chunker manifest metadata: %w", metadataErr))
|
return false, nil, terminal.record(nil, fmt.Errorf("clone chunker manifest metadata: %w", metadataErr))
|
||||||
}
|
}
|
||||||
profile := ""
|
profile := ""
|
||||||
if provider, ok := chunker.(contracts.ChunkExecutionClassProvider); ok && provider.ExecutionClass() == contracts.ExecutionClassLLMBacked {
|
if input.pipeline.ChunkExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
profile = input.pipeline.Chunk.LLMProfile
|
profile = input.pipeline.Chunk.LLMProfile
|
||||||
}
|
}
|
||||||
candidate := ChunkPlanRecord{
|
candidate := ChunkPlanRecord{
|
||||||
|
|||||||
@@ -67,30 +67,17 @@ type manifestChunker struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (c manifestChunker) ManifestMetadata() map[string]any { return c.metadata }
|
func (c manifestChunker) ManifestMetadata() map[string]any { return c.metadata }
|
||||||
func (manifestChunker) ExecutionClass() contracts.ExecutionClass {
|
|
||||||
return contracts.ExecutionClassLLMBacked
|
|
||||||
}
|
|
||||||
|
|
||||||
type llmCountingChunker struct {
|
type llmCountingChunker struct {
|
||||||
terminalChunker
|
terminalChunker
|
||||||
llmCalls *int
|
llmCalls *int
|
||||||
}
|
}
|
||||||
|
|
||||||
func (llmCountingChunker) ExecutionClass() contracts.ExecutionClass {
|
|
||||||
return contracts.ExecutionClassLLMBacked
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c llmCountingChunker) Plan(ctx context.Context, request contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
|
func (c llmCountingChunker) Plan(ctx context.Context, request contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
|
||||||
(*c.llmCalls)++
|
(*c.llmCalls)++
|
||||||
return c.terminalChunker.Plan(ctx, request)
|
return c.terminalChunker.Plan(ctx, request)
|
||||||
}
|
}
|
||||||
|
|
||||||
type deterministicChunker struct{ terminalChunker }
|
|
||||||
|
|
||||||
func (deterministicChunker) ExecutionClass() contracts.ExecutionClass {
|
|
||||||
return contracts.ExecutionClassDeterministic
|
|
||||||
}
|
|
||||||
|
|
||||||
type retryingChunker struct {
|
type retryingChunker struct {
|
||||||
key string
|
key string
|
||||||
plan source.ChunkPlan
|
plan source.ChunkPlan
|
||||||
@@ -535,6 +522,7 @@ func TestRunnerPublishesOnlyAcceptedGeneratedPlans(t *testing.T) {
|
|||||||
func TestRunnerStoresProducerProvenanceAndProducerWarnings(t *testing.T) {
|
func TestRunnerStoresProducerProvenanceAndProducerWarnings(t *testing.T) {
|
||||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||||
prepared.resolved.Chunk.LLMProfile = "chunk-profile"
|
prepared.resolved.Chunk.LLMProfile = "chunk-profile"
|
||||||
|
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
|
||||||
prepared.resolved.ChunkReferences = ResolvedReferenceTarget{
|
prepared.resolved.ChunkReferences = ResolvedReferenceTarget{
|
||||||
Stage: StageChunk,
|
Stage: StageChunk,
|
||||||
ReferenceSet: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
ReferenceSet: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||||
@@ -577,7 +565,8 @@ func TestRunnerRejectsUncloneableModuleManifestMetadata(t *testing.T) {
|
|||||||
func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
|
func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
|
||||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||||
prepared.resolved.Chunk.LLMProfile = "configured-but-unused"
|
prepared.resolved.Chunk.LLMProfile = "configured-but-unused"
|
||||||
prepared.chunker = deterministicChunker{terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}}
|
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassDeterministic
|
||||||
|
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}
|
||||||
store := &recordingChunkPlanStore{}
|
store := &recordingChunkPlanStore{}
|
||||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store})
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -588,6 +577,22 @@ func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunnerDoesNotInventProducerProfileForLLMChunker(t *testing.T) {
|
||||||
|
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||||
|
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
|
||||||
|
prepared.resolved.Chunk.LLMProfile = ""
|
||||||
|
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}
|
||||||
|
store := &recordingChunkPlanStore{}
|
||||||
|
|
||||||
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if store.saved.Producer.LLMProfile != "" || output.Manifest.ChunkPlan.ProducerLLMProfile != "" {
|
||||||
|
t.Fatalf("LLM producer profile = stored %q manifest %q, want empty", store.saved.Producer.LLMProfile, output.Manifest.ChunkPlan.ProducerLLMProfile)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunnerRefreshChangesDownstreamChunkFingerprint(t *testing.T) {
|
func TestRunnerRefreshChangesDownstreamChunkFingerprint(t *testing.T) {
|
||||||
doc := typedTestDocumentWithUnits(2)
|
doc := typedTestDocumentWithUnits(2)
|
||||||
prepared := preparedConcurrentPipeline(t, 1)
|
prepared := preparedConcurrentPipeline(t, 1)
|
||||||
|
|||||||
29
internal/framework/pipeline/runner_manifest_test.go
Normal file
29
internal/framework/pipeline/runner_manifest_test.go
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"reflect"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestMergeLLMProfileManifestsDistinguishesEffectiveTargets(t *testing.T) {
|
||||||
|
got := mergeLLMProfileManifests(
|
||||||
|
[]artifacts.LLMProfileManifest{
|
||||||
|
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-b", ReasoningEffort: "low"},
|
||||||
|
{ID: " profile ", Provider: " promptkit ", Model: " model ", BackendID: " backend-a ", ReasoningEffort: " low "},
|
||||||
|
},
|
||||||
|
[]artifacts.LLMProfileManifest{
|
||||||
|
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "high"},
|
||||||
|
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "low"},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
want := []artifacts.LLMProfileManifest{
|
||||||
|
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "high"},
|
||||||
|
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-a", ReasoningEffort: "low"},
|
||||||
|
{ID: "profile", Provider: "promptkit", Model: "model", BackendID: "backend-b", ReasoningEffort: "low"},
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("merged profiles = %#v, want %#v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -229,7 +229,7 @@ func TestResolveTypedLaneRejectsIncompatibleComposition(t *testing.T) {
|
|||||||
|
|
||||||
func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) {
|
func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) {
|
||||||
registry := NewMergerRegistry()
|
registry := NewMergerRegistry()
|
||||||
spec := ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"}
|
spec := ModuleSpec{Key: "typed/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}
|
||||||
constructor := func() (contracts.Merger[codecNotes], error) {
|
constructor := func() (contracts.Merger[codecNotes], error) {
|
||||||
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
|
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
|
||||||
}
|
}
|
||||||
@@ -243,7 +243,7 @@ func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) {
|
|||||||
|
|
||||||
func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
|
func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
|
||||||
extractors := NewExtractorRegistry()
|
extractors := NewExtractorRegistry()
|
||||||
if err := RegisterExtractor(extractors, ModuleSpec{Key: "typed/extract", Stage: StageExtract, ArtifactKind: "test/notes"}, func() (contracts.Extractor[codecNotes], error) {
|
if err := RegisterExtractor(extractors, ModuleSpec{Key: "typed/extract", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Extractor[codecNotes], error) {
|
||||||
return typedTestExtractor[codecNotes]{key: "typed/extract"}, nil
|
return typedTestExtractor[codecNotes]{key: "typed/extract"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterExtractor() error = %v", err)
|
t.Fatalf("RegisterExtractor() error = %v", err)
|
||||||
@@ -253,7 +253,7 @@ func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
mergers := NewMergerRegistry()
|
mergers := NewMergerRegistry()
|
||||||
if err := RegisterMerger(mergers, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"}, func() (contracts.Merger[codecNotes], error) {
|
if err := RegisterMerger(mergers, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Merger[codecNotes], error) {
|
||||||
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
|
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterMerger() error = %v", err)
|
t.Fatalf("RegisterMerger() error = %v", err)
|
||||||
@@ -263,7 +263,7 @@ func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
normalizers := NewNormalizerRegistry()
|
normalizers := NewNormalizerRegistry()
|
||||||
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
||||||
return typedTestNormalizer[codecNotes]{key: "typed/normalize"}, nil
|
return typedTestNormalizer[codecNotes]{key: "typed/normalize"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterNormalizer() error = %v", err)
|
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||||
@@ -298,7 +298,7 @@ func TestTypedRegistrySpecLookupUsesArtifactKindAndStableCatalogOrder(t *testing
|
|||||||
mergers := NewMergerRegistry()
|
mergers := NewMergerRegistry()
|
||||||
normalizers := NewNormalizerRegistry()
|
normalizers := NewNormalizerRegistry()
|
||||||
for _, item := range order {
|
for _, item := range order {
|
||||||
base := ModuleSpec{Key: "typed/shared", ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.slot}}}
|
base := ModuleSpec{Key: "typed/shared", ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.slot}}}
|
||||||
mergeSpec := base
|
mergeSpec := base
|
||||||
mergeSpec.Stage = StageMerge
|
mergeSpec.Stage = StageMerge
|
||||||
normalizeSpec := base
|
normalizeSpec := base
|
||||||
@@ -474,18 +474,18 @@ func typedResolutionCatalog(t *testing.T, options typedCatalogOptions) ModuleCat
|
|||||||
|
|
||||||
func mustRegisterTypedTestBase(t *testing.T, catalog ModuleCatalog) {
|
func mustRegisterTypedTestBase(t *testing.T, catalog ModuleCatalog) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{Key: "typed/input", Stage: StageInput}, func() (contracts.InputAdapter, error) {
|
if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{Key: "typed/input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.InputAdapter, error) {
|
||||||
return &typedTestInput{key: "typed/input", doc: typedTestDocument()}, nil
|
return &typedTestInput{key: "typed/input", doc: typedTestDocument()}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("register input: %v", err)
|
t.Fatalf("register input: %v", err)
|
||||||
}
|
}
|
||||||
if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{Key: "typed/chunk", Stage: StageChunk}, func() (contracts.Chunker, error) {
|
if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{Key: "typed/chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.Chunker, error) {
|
||||||
doc := typedTestDocument()
|
doc := typedTestDocument()
|
||||||
return &typedTestChunker{key: "typed/chunk", plan: typedTestPlan(doc)}, nil
|
return &typedTestChunker{key: "typed/chunk", plan: typedTestPlan(doc)}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("register chunker: %v", err)
|
t.Fatalf("register chunker: %v", err)
|
||||||
}
|
}
|
||||||
if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{Key: "typed/output", Stage: StageOutput}, func() (contracts.OutputEncoder, error) {
|
if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{Key: "typed/output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.OutputEncoder, error) {
|
||||||
return &typedTestOutput{key: "typed/output"}, nil
|
return &typedTestOutput{key: "typed/output"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("register output: %v", err)
|
t.Fatalf("register output: %v", err)
|
||||||
@@ -509,21 +509,21 @@ func mustRegisterArtifactCodec[T any](t *testing.T, registry *ArtifactCodecRegis
|
|||||||
|
|
||||||
func mustRegisterTypedExtractor[T any](t *testing.T, registry *ExtractorRegistry, key string, kind contracts.ArtifactKind, extractor contracts.Extractor[T]) {
|
func mustRegisterTypedExtractor[T any](t *testing.T, registry *ExtractorRegistry, key string, kind contracts.ArtifactKind, extractor contracts.Extractor[T]) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := RegisterExtractor(registry, ModuleSpec{Key: key, Stage: StageExtract, ArtifactKind: kind}, func() (contracts.Extractor[T], error) { return extractor, nil }); err != nil {
|
if err := RegisterExtractor(registry, ModuleSpec{Key: key, Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: kind}, func() (contracts.Extractor[T], error) { return extractor, nil }); err != nil {
|
||||||
t.Fatalf("RegisterExtractor() error = %v", err)
|
t.Fatalf("RegisterExtractor() error = %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func mustRegisterTypedMerger[T any](t *testing.T, registry *MergerRegistry, kind contracts.ArtifactKind, merger contracts.Merger[T]) {
|
func mustRegisterTypedMerger[T any](t *testing.T, registry *MergerRegistry, kind contracts.ArtifactKind, merger contracts.Merger[T]) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := RegisterMerger(registry, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: kind}, func() (contracts.Merger[T], error) { return merger, nil }); err != nil {
|
if err := RegisterMerger(registry, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: kind}, func() (contracts.Merger[T], error) { return merger, nil }); err != nil {
|
||||||
t.Fatalf("RegisterMerger() error = %v", err)
|
t.Fatalf("RegisterMerger() error = %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func mustRegisterTypedNormalizer[T any](t *testing.T, registry *NormalizerRegistry, kind contracts.ArtifactKind, normalizer contracts.Normalizer[T]) {
|
func mustRegisterTypedNormalizer[T any](t *testing.T, registry *NormalizerRegistry, kind contracts.ArtifactKind, normalizer contracts.Normalizer[T]) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := RegisterNormalizer(registry, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: kind}, func() (contracts.Normalizer[T], error) { return normalizer, nil }); err != nil {
|
if err := RegisterNormalizer(registry, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: kind}, func() (contracts.Normalizer[T], error) { return normalizer, nil }); err != nil {
|
||||||
t.Fatalf("RegisterNormalizer() error = %v", err)
|
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -11,15 +11,15 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ModulePromptFile maps a module-owned embedded prompt file into the
|
// ModulePromptFile maps a module-owned embedded prompt file into the registered
|
||||||
// Scriptorium-visible module prompt directory.
|
// module prompt directory.
|
||||||
type ModulePromptFile struct {
|
type ModulePromptFile struct {
|
||||||
Name string
|
Name string
|
||||||
Path string
|
Path string
|
||||||
}
|
}
|
||||||
|
|
||||||
// SharedPromptFile maps a caller-owned shared prompt file into a module's
|
// SharedPromptFile maps a caller-owned shared prompt file into a module's
|
||||||
// Scriptorium-visible sharedassets prompt subdirectory.
|
// registered sharedassets prompt subdirectory.
|
||||||
type SharedPromptFile struct {
|
type SharedPromptFile struct {
|
||||||
Name string
|
Name string
|
||||||
FS fs.FS
|
FS fs.FS
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
id: dnd.scenes
|
id: dnd.scenes
|
||||||
version: "v1"
|
version: "v1"
|
||||||
default_profile: gemini-2-flash
|
default_profile: dnd-extraction
|
||||||
inputs:
|
inputs:
|
||||||
- name: transcript
|
- name: transcript
|
||||||
required: true
|
required: true
|
||||||
|
|||||||
@@ -47,16 +47,12 @@ func (c *Chunker) Key() string {
|
|||||||
return Key
|
return Key
|
||||||
}
|
}
|
||||||
|
|
||||||
func (*Chunker) ExecutionClass() contracts.ExecutionClass {
|
|
||||||
return contracts.ExecutionClassLLMBacked
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
|
func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
|
||||||
return shared.ReferenceSlots(referenceSlotDescriptions)
|
return shared.ReferenceSlots(referenceSlotDescriptions)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *Chunker) ManifestMetadata() map[string]any {
|
func (c *Chunker) ManifestMetadata() map[string]any {
|
||||||
promptSHA, err := scriptoriumPromptMetadata()
|
promptSHA, err := promptAssetMetadata()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
promptSHA = ""
|
promptSHA = ""
|
||||||
}
|
}
|
||||||
@@ -120,6 +116,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
|||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageChunk,
|
Stage: pipeline.StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
ReferenceSlots: shared.ReferenceSlots(referenceSlotDescriptions),
|
ReferenceSlots: shared.ReferenceSlots(referenceSlotDescriptions),
|
||||||
|
|||||||
@@ -24,6 +24,7 @@ func TestNewModuleSpecAndRegister(t *testing.T) {
|
|||||||
want := pipeline.ModuleSpec{
|
want := pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageChunk,
|
Stage: pipeline.StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: []string{"source.transcript"},
|
Requires: []string{"source.transcript"},
|
||||||
Provides: []string{"chunks"},
|
Provides: []string{"chunks"},
|
||||||
ReferenceSlots: wantReferenceSlots(),
|
ReferenceSlots: wantReferenceSlots(),
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
)
|
)
|
||||||
|
|
||||||
const scriptoriumPromptRoot = "assets/prompts"
|
const promptAssetRoot = "assets/prompts"
|
||||||
|
|
||||||
var promptAssetManifest = shared.PromptAssetManifest{
|
var promptAssetManifest = shared.PromptAssetManifest{
|
||||||
ModuleDir: "dnd.scenes",
|
ModuleDir: "dnd.scenes",
|
||||||
@@ -30,21 +30,21 @@ func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("prepare scene prompt assets: %w", err)
|
return fmt.Errorf("prepare scene prompt assets: %w", err)
|
||||||
}
|
}
|
||||||
if err := registry.RegisterPromptFS(promptFS, scriptoriumPromptRoot); err != nil {
|
if err := registry.RegisterPromptFS(promptFS, promptAssetRoot); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumPromptMetadata() (string, error) {
|
func promptAssetMetadata() (string, error) {
|
||||||
scriptoriumPromptHashOnce.Do(func() {
|
promptAssetHashOnce.Do(func() {
|
||||||
scriptoriumPromptHash, scriptoriumPromptHashErr = promptAssetManifest.Hash(embeddedAssets)
|
promptAssetHash, promptAssetHashErr = promptAssetManifest.Hash(embeddedAssets)
|
||||||
})
|
})
|
||||||
return scriptoriumPromptHash, scriptoriumPromptHashErr
|
return promptAssetHash, promptAssetHashErr
|
||||||
}
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
scriptoriumPromptHashOnce sync.Once
|
promptAssetHashOnce sync.Once
|
||||||
scriptoriumPromptHash string
|
promptAssetHash string
|
||||||
scriptoriumPromptHashErr error
|
promptAssetHashErr error
|
||||||
)
|
)
|
||||||
@@ -8,10 +8,10 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestScriptoriumPromptPreparesTranscriptAndTaskMessages(t *testing.T) {
|
func TestPromptAssetsPrepareTranscriptAndTaskMessages(t *testing.T) {
|
||||||
transcript := []byte(`{"sentinel":"scene-transcript"}`)
|
transcript := []byte(`{"sentinel":"scene-transcript"}`)
|
||||||
prepared := prepareScenesPrompt(t, transcript, "scene-players", "scene-party", "scene-glossary")
|
prepared := prepareScenesPrompt(t, transcript, "scene-players", "scene-party", "scene-glossary")
|
||||||
|
|
||||||
@@ -27,7 +27,7 @@ func TestScriptoriumPromptPreparesTranscriptAndTaskMessages(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
for _, index := range []int{1, 4} {
|
for _, index := range []int{1, 4} {
|
||||||
if cache := prepared.Messages[index].CacheControl; cache == nil || cache.Type != scriptorium.CacheControlEphemeral {
|
if cache := prepared.Messages[index].CacheControl; cache == nil || cache.Type != promptkit.CacheControlEphemeral {
|
||||||
t.Errorf("message %d cache control = %#v, want ephemeral", index, cache)
|
t.Errorf("message %d cache control = %#v, want ephemeral", index, cache)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -52,7 +52,7 @@ func TestScriptoriumPromptPreparesTranscriptAndTaskMessages(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestScriptoriumPromptDiagnosticsOmitRawMaterials(t *testing.T) {
|
func TestPromptAssetDiagnosticsOmitRawMaterials(t *testing.T) {
|
||||||
transcript := []byte(`{"secret":"source text"}`)
|
transcript := []byte(`{"secret":"source text"}`)
|
||||||
prepared := prepareScenesPrompt(t, transcript, "private player note", "private party note", "private glossary note")
|
prepared := prepareScenesPrompt(t, transcript, "private player note", "private party note", "private glossary note")
|
||||||
metadata := newChunker(t, &fakeScenesLLMClient{}).ManifestMetadata()
|
metadata := newChunker(t, &fakeScenesLLMClient{}).ManifestMetadata()
|
||||||
@@ -94,22 +94,22 @@ func TestScriptoriumPromptDiagnosticsOmitRawMaterials(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func prepareScenesPrompt(t *testing.T, transcript []byte, players string, party string, glossary string) *scriptorium.PreparedRun {
|
func prepareScenesPrompt(t *testing.T, transcript []byte, players string, party string, glossary string) *promptkit.PreparedRun {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
registry := llm.NewAssetRegistry()
|
registry := llm.NewAssetRegistry()
|
||||||
if err := RegisterPromptAssets(registry); err != nil {
|
if err := RegisterPromptAssets(registry); err != nil {
|
||||||
t.Fatalf("register scene prompt assets: %v", err)
|
t.Fatalf("register scene prompt assets: %v", err)
|
||||||
}
|
}
|
||||||
engine := newScenesScriptoriumEngine(t, registry)
|
engine := newScenesPromptEngine(t, registry)
|
||||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: PromptID,
|
PromptID: PromptID,
|
||||||
PromptVersion: ResponseSchemaVersion,
|
PromptVersion: ResponseSchemaVersion,
|
||||||
ProfileID: "scene-test-profile",
|
ProfileID: "scene-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.InlineWithURI("file:///session.json", string(transcript)),
|
"transcript": promptkit.InlineWithURI("file:///session.json", string(transcript)),
|
||||||
"players": scriptorium.Inline(players),
|
"players": promptkit.Inline(players),
|
||||||
"party": scriptorium.Inline(party),
|
"party": promptkit.Inline(party),
|
||||||
"glossary": scriptorium.Inline(glossary),
|
"glossary": promptkit.Inline(glossary),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -118,18 +118,18 @@ func prepareScenesPrompt(t *testing.T, transcript []byte, players string, party
|
|||||||
return prepared
|
return prepared
|
||||||
}
|
}
|
||||||
|
|
||||||
func newScenesScriptoriumEngine(t *testing.T, registry *llm.AssetRegistry) *scriptorium.Engine {
|
func newScenesPromptEngine(t *testing.T, registry *llm.AssetRegistry) *promptkit.Engine {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
options, err := registry.ScriptoriumOptions()
|
options, err := registry.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ScriptoriumOptions() error = %v, want nil", err)
|
t.Fatalf("PromptKitOptions() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
ID: "scene-test-profile",
|
ID: "scene-test-profile",
|
||||||
Endpoint: "http://127.0.0.1:1/v1",
|
Endpoint: "http://127.0.0.1:1/v1",
|
||||||
Model: "scene-test-model",
|
Model: "scene-test-model",
|
||||||
})))
|
})))
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("NewEngine() error = %v, want nil", err)
|
t.Fatalf("NewEngine() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
id: dnd.combat_turns
|
id: dnd.combat_turns
|
||||||
version: "v1"
|
version: "v1"
|
||||||
default_profile: gemini-2-flash
|
default_profile: dnd-extraction
|
||||||
inputs:
|
inputs:
|
||||||
- name: transcript
|
- name: transcript
|
||||||
required: true
|
required: true
|
||||||
|
|||||||
@@ -96,7 +96,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, extractorErrorf("prepare scene eligibility: %w", err)
|
return nil, extractorErrorf("prepare scene eligibility: %w", err)
|
||||||
}
|
}
|
||||||
promptSHA, err := scriptoriumPromptMetadata()
|
promptSHA, err := promptAssetMetadata()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, extractorErrorf("load prompt metadata: %w", err)
|
return nil, extractorErrorf("load prompt metadata: %w", err)
|
||||||
}
|
}
|
||||||
@@ -226,6 +226,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
|||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageExtract,
|
Stage: pipeline.StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
ArtifactKind: dnd.CombatTurnListKind,
|
ArtifactKind: dnd.CombatTurnListKind,
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
)
|
)
|
||||||
|
|
||||||
const scriptoriumPromptRoot = "assets/prompts"
|
const promptAssetRoot = "assets/prompts"
|
||||||
|
|
||||||
var promptAssetManifest = shared.PromptAssetManifest{
|
var promptAssetManifest = shared.PromptAssetManifest{
|
||||||
ModuleDir: "dnd.combat_turns",
|
ModuleDir: "dnd.combat_turns",
|
||||||
@@ -33,21 +33,21 @@ func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("prepare combat-turn prompt assets: %w", err)
|
return fmt.Errorf("prepare combat-turn prompt assets: %w", err)
|
||||||
}
|
}
|
||||||
if err := registry.RegisterPromptFS(promptFS, scriptoriumPromptRoot); err != nil {
|
if err := registry.RegisterPromptFS(promptFS, promptAssetRoot); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumPromptMetadata() (string, error) {
|
func promptAssetMetadata() (string, error) {
|
||||||
scriptoriumPromptHashOnce.Do(func() {
|
promptAssetHashOnce.Do(func() {
|
||||||
scriptoriumPromptHash, scriptoriumPromptHashErr = promptAssetManifest.Hash(embeddedAssets)
|
promptAssetHash, promptAssetHashErr = promptAssetManifest.Hash(embeddedAssets)
|
||||||
})
|
})
|
||||||
return scriptoriumPromptHash, scriptoriumPromptHashErr
|
return promptAssetHash, promptAssetHashErr
|
||||||
}
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
scriptoriumPromptHashOnce sync.Once
|
promptAssetHashOnce sync.Once
|
||||||
scriptoriumPromptHash string
|
promptAssetHash string
|
||||||
scriptoriumPromptHashErr error
|
promptAssetHashErr error
|
||||||
)
|
)
|
||||||
@@ -8,7 +8,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestRegisterPromptAssetsAndPrepareCombatPrompt(t *testing.T) {
|
func TestRegisterPromptAssetsAndPrepareCombatPrompt(t *testing.T) {
|
||||||
@@ -23,37 +23,37 @@ func TestRegisterPromptAssetsAndPrepareCombatPrompt(t *testing.T) {
|
|||||||
if _, err := fs.ReadFile(schemaFS, "dnd_combat_turns_llm.v1.json"); err != nil {
|
if _, err := fs.ReadFile(schemaFS, "dnd_combat_turns_llm.v1.json"); err != nil {
|
||||||
t.Fatalf("response schema asset: %v", err)
|
t.Fatalf("response schema asset: %v", err)
|
||||||
}
|
}
|
||||||
hash, err := scriptoriumPromptMetadata()
|
hash, err := promptAssetMetadata()
|
||||||
if err != nil || !strings.HasPrefix(hash, "sha256:") {
|
if err != nil || !strings.HasPrefix(hash, "sha256:") {
|
||||||
t.Fatalf("scriptoriumPromptMetadata() = %q, %v; want digest", hash, err)
|
t.Fatalf("promptAssetMetadata() = %q, %v; want digest", hash, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestScriptoriumPromptPreparesRequiredInputs(t *testing.T) {
|
func TestPromptAssetsPrepareRequiredInputs(t *testing.T) {
|
||||||
registry := llm.NewAssetRegistry()
|
registry := llm.NewAssetRegistry()
|
||||||
if err := RegisterPromptAssets(registry); err != nil {
|
if err := RegisterPromptAssets(registry); err != nil {
|
||||||
t.Fatalf("RegisterPromptAssets() error = %v, want nil", err)
|
t.Fatalf("RegisterPromptAssets() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
options, err := registry.ScriptoriumOptions()
|
options, err := registry.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ScriptoriumOptions() error = %v, want nil", err)
|
t.Fatalf("PromptKitOptions() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
ID: "combat-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "combat-test-model",
|
ID: "combat-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "combat-test-model",
|
||||||
})))
|
})))
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("NewEngine() error = %v, want nil", err)
|
t.Fatalf("NewEngine() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
transcript := `{"units":[1]}`
|
transcript := `{"units":[1]}`
|
||||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "combat-test-profile",
|
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "combat-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.InlineWithURI("file:///session.json", transcript),
|
"transcript": promptkit.InlineWithURI("file:///session.json", transcript),
|
||||||
"players": scriptorium.Inline(" "),
|
"players": promptkit.Inline(" "),
|
||||||
"party": scriptorium.Inline(" "),
|
"party": promptkit.Inline(" "),
|
||||||
"glossary": scriptorium.Inline(" "),
|
"glossary": promptkit.Inline(" "),
|
||||||
"npcs": scriptorium.Inline(" "),
|
"npcs": promptkit.Inline(" "),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
id: dnd.item_events
|
id: dnd.item_events
|
||||||
version: "v1"
|
version: "v1"
|
||||||
default_profile: gemini-2-flash
|
default_profile: dnd-extraction
|
||||||
inputs:
|
inputs:
|
||||||
- name: transcript
|
- name: transcript
|
||||||
required: true
|
required: true
|
||||||
|
|||||||
@@ -54,7 +54,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
|
|||||||
if len(references) > 1 {
|
if len(references) > 1 {
|
||||||
return nil, extractorErrorf("at most one reference set may be supplied")
|
return nil, extractorErrorf("at most one reference set may be supplied")
|
||||||
}
|
}
|
||||||
promptSHA, err := scriptoriumPromptMetadata()
|
promptSHA, err := promptAssetMetadata()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, extractorErrorf("load prompt metadata: %w", err)
|
return nil, extractorErrorf("load prompt metadata: %w", err)
|
||||||
}
|
}
|
||||||
@@ -129,6 +129,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
|||||||
return pipeline.ModuleSpec{
|
return pipeline.ModuleSpec{
|
||||||
Key: Key,
|
Key: Key,
|
||||||
Stage: pipeline.StageExtract,
|
Stage: pipeline.StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
ArtifactKind: dnd.ItemEventListKind,
|
ArtifactKind: dnd.ItemEventListKind,
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
)
|
)
|
||||||
|
|
||||||
const scriptoriumPromptRoot = "assets/prompts"
|
const promptAssetRoot = "assets/prompts"
|
||||||
|
|
||||||
var promptAssetManifest = shared.PromptAssetManifest{
|
var promptAssetManifest = shared.PromptAssetManifest{
|
||||||
ModuleDir: "dnd.item_events",
|
ModuleDir: "dnd.item_events",
|
||||||
@@ -32,21 +32,21 @@ func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("prepare item event prompt assets: %w", err)
|
return fmt.Errorf("prepare item event prompt assets: %w", err)
|
||||||
}
|
}
|
||||||
if err := registry.RegisterPromptFS(promptFS, scriptoriumPromptRoot); err != nil {
|
if err := registry.RegisterPromptFS(promptFS, promptAssetRoot); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumPromptMetadata() (string, error) {
|
func promptAssetMetadata() (string, error) {
|
||||||
scriptoriumPromptHashOnce.Do(func() {
|
promptAssetHashOnce.Do(func() {
|
||||||
scriptoriumPromptHash, scriptoriumPromptHashErr = promptAssetManifest.Hash(embeddedAssets)
|
promptAssetHash, promptAssetHashErr = promptAssetManifest.Hash(embeddedAssets)
|
||||||
})
|
})
|
||||||
return scriptoriumPromptHash, scriptoriumPromptHashErr
|
return promptAssetHash, promptAssetHashErr
|
||||||
}
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
scriptoriumPromptHashOnce sync.Once
|
promptAssetHashOnce sync.Once
|
||||||
scriptoriumPromptHash string
|
promptAssetHash string
|
||||||
scriptoriumPromptHashErr error
|
promptAssetHashErr error
|
||||||
)
|
)
|
||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestPromptAssetsPrepareItemEventPrompt(t *testing.T) {
|
func TestPromptAssetsPrepareItemEventPrompt(t *testing.T) {
|
||||||
@@ -17,24 +17,24 @@ func TestPromptAssetsPrepareItemEventPrompt(t *testing.T) {
|
|||||||
if err := RegisterPromptAssets(registry); err != nil {
|
if err := RegisterPromptAssets(registry); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
options, err := registry.ScriptoriumOptions()
|
options, err := registry.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
ID: "item-events-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "item-events-test-model",
|
ID: "item-events-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "item-events-test-model",
|
||||||
})))
|
})))
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "item-events-test-profile",
|
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "item-events-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.InlineWithURI("file:///session.json", `{"segments":[1]}`),
|
"transcript": promptkit.InlineWithURI("file:///session.json", `{"segments":[1]}`),
|
||||||
"players": scriptorium.Inline(" "),
|
"players": promptkit.Inline(" "),
|
||||||
"party": scriptorium.Inline(" "),
|
"party": promptkit.Inline(" "),
|
||||||
"glossary": scriptorium.Inline(" "),
|
"glossary": promptkit.Inline(" "),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -46,9 +46,9 @@ func TestPromptAssetsPrepareItemEventPrompt(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestPromptAssetsDoNotLeakIntoMetadata(t *testing.T) {
|
func TestPromptAssetsDoNotLeakIntoMetadata(t *testing.T) {
|
||||||
hash, err := scriptoriumPromptMetadata()
|
hash, err := promptAssetMetadata()
|
||||||
if err != nil || !strings.HasPrefix(hash, "sha256:") {
|
if err != nil || !strings.HasPrefix(hash, "sha256:") {
|
||||||
t.Fatalf("scriptoriumPromptMetadata() = %q, %v", hash, err)
|
t.Fatalf("promptAssetMetadata() = %q, %v", hash, err)
|
||||||
}
|
}
|
||||||
metadata := newExtractor(t, &fakeItemEventsLLMClient{}).ManifestMetadata()
|
metadata := newExtractor(t, &fakeItemEventsLLMClient{}).ManifestMetadata()
|
||||||
payload, err := json.Marshal(metadata)
|
payload, err := json.Marshal(metadata)
|
||||||
@@ -18,7 +18,7 @@ func TestConstructorSpecOptionsAndMetadata(t *testing.T) {
|
|||||||
t.Fatalf("New() error = %v", err)
|
t.Fatalf("New() error = %v", err)
|
||||||
}
|
}
|
||||||
want := pipeline.ModuleSpec{
|
want := pipeline.ModuleSpec{
|
||||||
Key: Key, Stage: pipeline.StageExtract, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.item_events"}, ArtifactKind: dnd.ItemEventListKind,
|
Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.item_events"}, ArtifactKind: dnd.ItemEventListKind,
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "glossary", Description: referenceSlotDescriptions.Glossary, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
{Name: "glossary", Description: referenceSlotDescriptions.Glossary, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||||
{Name: "party", Description: referenceSlotDescriptions.Party, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
{Name: "party", Description: referenceSlotDescriptions.Party, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
id: dnd.npc_interactions
|
id: dnd.npc_interactions
|
||||||
version: "v1"
|
version: "v1"
|
||||||
default_profile: gemini-2-flash
|
default_profile: dnd-extraction
|
||||||
inputs:
|
inputs:
|
||||||
- name: transcript
|
- name: transcript
|
||||||
required: true
|
required: true
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user