Compare commits

20 Commits

Author SHA1 Message Date
39388e96d4 Make PromptKit profile handling safer and more consistent 2026-08-03 18:35:40 +00:00
12ac25bd63 Sanitize PromptKit profile fingerprint errors 2026-08-03 17:25:44 +00:00
394278e1f2 Document workload-oriented LLM profile deployment 2026-08-03 17:21:42 +00:00
5cd7f8e737 Expose pipeline LLM profile defaults 2026-08-03 17:15:07 +00:00
bf3fadf9ae Resolve pipeline LLM profile defaults 2026-08-03 17:09:16 +00:00
58815aaf33 Require explicit module execution classes 2026-08-03 17:00:08 +00:00
ce857966f1 Add execution metadata to module specifications 2026-08-03 16:49:51 +00:00
a3bd0c1867 Add D&D extraction fallback profile 2026-08-03 16:39:45 +00:00
b05634ee86 Add fallback PromptKit profile assets 2026-08-03 16:34:48 +00:00
4829f94157 Inspect PromptKit profiles during preflight 2026-08-03 16:26:15 +00:00
67b315099d Execute PromptKit requests from prepared snapshots 2026-08-03 16:17:40 +00:00
b5c86de4d7 Upgrade PromptKit to version 0.5.0 2026-08-03 16:12:29 +00:00
2eeca2ed5a Plan the PromptKit upgrade and profile workflow 2026-08-03 16:07:37 +00:00
b5aaeb1c78 Update future roadmap document with ideas for new feature developments 2026-07-30 16:27:36 +00:00
9171b66a41 Clarify PromptKit configuration and retire the completed plan 2026-07-30 15:38:31 +00:00
b4363b3b73 Document local PromptKit backend support 2026-07-30 05:22:41 +00:00
241e9d2a89 Include local backend target in checkpoint identity 2026-07-30 05:17:39 +00:00
715fff7b72 Register configured local PromptKit backend 2026-07-30 05:14:22 +00:00
d627b91b4f Add local PromptKit backend configuration 2026-07-30 05:09:54 +00:00
a67b3aa76d Upgrade PromptKit to version 0.3.0 2026-07-30 05:06:32 +00:00
109 changed files with 4152 additions and 651 deletions

View File

@@ -0,0 +1,49 @@
# ADR-0010: Use workload-oriented LLM profile defaults
**Status:** Accepted
**Date:** 2026-08-03
## Context
LLM-backed D&D operations share an execution-policy choice, but repeating a
provider or model-named profile on every module binding ties pipeline structure
to a deployment decision. Different environments may require different model,
backend, timeout, or reasoning settings while retaining the same workload.
Notarius also needs a usable default for maintained D&D prompts without making
an operator profile mandatory. That default must remain owned by the D&D
family, while generic LLM infrastructure stays unaware of domain-specific
policy.
## Decision
Pipelines may name one workload-oriented default profile, inherited only by
selected LLM-backed bindings and validators. Binding-level profile IDs remain
intentional exceptions, and the run-wide CLI profile override has highest
precedence.
The D&D family owns an embedded fallback profile named `dnd-extraction`.
Operators may provide a complete profile with the same ID through a PromptKit
filesystem source. PromptKit selects the higher-precedence matching definition;
Notarius does not merge profile documents. Production, development, and local
deployments can therefore use different execution policy behind one unchanged
pipeline ID.
## Alternatives considered
- Repeat a model-named profile on every binding. This makes routine deployment
policy changes noisy and obscures the shared workload intent.
- Require every deployment to install a profile file. This adds configuration
friction and leaves maintained D&D prompts without an application-owned
fallback.
- Put D&D profile policy in generic LLM infrastructure. This breaks domain
ownership and makes generic code depend on one workload.
## Consequences
Pipeline configuration expresses workload intent rather than a specific
provider or model. Operators can replace the complete execution policy without
editing bindings, while binding-level and run-wide exceptions remain available.
Profile changes affect resolved pipeline and checkpoint identity, so they may
intentionally cause work to be recomputed. The D&D fallback becomes a
maintained application execution-policy asset.

View File

@@ -39,7 +39,7 @@ pipeline ID and **--input** are required.
| **--debug** | Retain a debug bundle for this run. |
| **--debug-dir path** | Override the debug-bundle root. Requires **--debug**. |
| **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. |
| **--llm-profile id** | Override effective LLM-capable module bindings with one configured profile. |
| **--llm-profile id** | Highest-precedence configured profile for selected LLM-backed bindings and validators; it replaces binding and [pipeline](config.md#pipelines) defaults. |
| **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. |
| **--reasoning-effort value** | Replace the selected PromptKit profile's reasoning effort for every LLM-backed call in this run. The value must be non-empty and the flag may be specified only once. |
| **--clear-reasoning-effort** | Clear reasoning effort inherited from the selected PromptKit profile for every LLM-backed call in this run. |

View File

@@ -32,7 +32,8 @@ override the fields listed below.
single-lane Seriatim-to-spell pipeline.
- [Complete D&D configuration](../examples/dnd-complete.config.yml) uses
ordered steps, all implemented D&D lanes, generated references, state
settings, and bounded LLM concurrency.
settings, bounded LLM concurrency, and the maintained
[operator profile](../examples/profiles/dnd-extraction.yml).
Use these complete files as starting points rather than combining the
illustrative fragments in this reference.
@@ -46,7 +47,7 @@ other than **version** is optional.
| Field | Type | Default | Rules |
| --- | --- | --- | --- |
| **version** | integer | none | Required; must be 4. |
| **promptkit** | object | none | Profile source configuration. |
| **promptkit** | object | none | Profile source and optional local-backend configuration. |
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
| **concurrency** | object | see below | Global LLM and extraction limits. |
| **output** | object | see below | Published output settings. |
@@ -71,27 +72,76 @@ per-user root. An explicit empty output or debug directory is invalid.
## PromptKit Profiles
The optional **promptkit** object selects one source of profile definitions:
The optional **promptkit** object selects one source of profile definitions and
may register one conventional local OpenAI-compatible backend:
~~~yaml
version: 4
promptkit:
profile_dir: /path/to/profiles
# profile_file: /path/to/profiles.yml
profile_dir: ./profiles
# profile_file: ./profiles.yml
local_backend:
endpoint: http://localhost:8000/v1
concurrency_limit: 2
~~~
| Field | Type | Rules |
| --- | --- | --- |
| **profile_dir** | string | Non-empty directory containing profile files. |
| **profile_file** | string | Non-empty profile file. |
| **local_backend** | object | Optional registration for the conventional PromptKit backend ID **local**. |
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
Set at most one of these fields. Profile IDs used by a binding must be available
from the selected PromptKit profile source when the pipeline is resolved.
Keep credentials out of this file: configure a profile to read its credential
from an environment variable, then set that environment variable only in the
run environment. PromptKit owns the profile-file format; see the
[PromptKit upstream boundary](integrations/pkg-promptkit.md) for the pinned
package and canonical format reference.
Set at most one of **profile_dir** and **profile_file**. Relative values use
the process working directory, not the configuration file's directory. The
complete example's `./examples/profiles/dnd-extraction.yml` value is therefore
valid when Notarius is launched from the repository root; use an absolute path
for services and containers.
An operator source is optional. For a requested ID, PromptKit checks the
configured operator source first, then Notarius's embedded fallback profiles,
then its own built-in catalog. A matching profile is complete: it replaces a
lower-precedence definition rather than merging with it. The maintained
[`dnd-extraction` operator profile](../examples/profiles/dnd-extraction.yml)
is a secret-free deployment artifact; production, development, and local
deployments can each provide a complete definition with that same workload ID.
Use workload-oriented IDs for new profiles instead of model names.
[Operations](operations.md#promptkit-profile-deployment) owns the deployment
workflow and credential-handling guidance.
When **local_backend** is present, its endpoint is trimmed and must use HTTP or
HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are
allowed. User information, queries, and fragments are rejected. A zero
**concurrency_limit** leaves the local backend unrestricted inside PromptKit;
a positive value limits simultaneous local generations. The application-wide
**concurrency.total_llm** limit still applies in both cases. Neither local
backend field has an environment override. Omitting **local_backend** registers
nothing and preserves existing built-in and endpoint-only profile behavior.
A file-backed PromptKit profile selects the registration by its case-sensitive
backend ID:
~~~yaml
id: local-summary
backend: local
model: example-model
~~~
Keep credentials out of the local-backend object. A PromptKit profile may name
its credential environment variable through `api_key_env`; set that variable
only in the run environment. PromptKit owns the
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md).
The [PromptKit upstream boundary](integrations/pkg-promptkit.md) identifies the
supported package API, and [Operations](operations.md#operational-limits)
describes the effective concurrency layers.
`notarius config validate --pipeline <id>` resolves the selected pipeline and
inspects every explicit effective profile without contacting a provider or
requiring credential values. It rejects absent, malformed, or incompatible
profiles before a run prepares modules. Credential availability is checked only
when a generation is prepared.
## Migrating Version 3 Configuration
@@ -154,6 +204,7 @@ Each **pipelines** entry has a unique, non-empty ID and the following shape:
~~~yaml
pipelines:
dnd-session:
llm_profile: dnd-extraction
input: seriatim
chunk: generic
output: json
@@ -166,6 +217,7 @@ pipelines:
| Field | Type | Default | Rules |
| --- | --- | --- | --- |
| **llm_profile** | string | none | Optional non-empty default PromptKit profile ID for selected LLM-backed bindings and validators. An explicitly present blank value is invalid. |
| **input** | module binding | none | Required. |
| **chunk** | module binding | **generic** | Optional. |
| **output** | module binding | **json** | Optional. |
@@ -179,6 +231,12 @@ needs a unique non-empty **id**, an **artifacts** map, and may have
**references**. A lane ID must not appear more than once in a pipeline,
including across explicit steps.
For each selected LLM-backed binding or validator, profile selection occurs
after module, validator, and `--only` lane selection. It uses the
run-level **--llm-profile** value first, then the binding's **llm_profile**,
then the pipeline's **llm_profile**, and finally the PromptKit default.
Deterministic bindings do not receive these defaults or run overrides.
A lane has these fields:
| Field | Type | Default | Rules |
@@ -206,7 +264,7 @@ Use an object for fields:
~~~yaml
extract:
module: dnd/spells
llm_profile: gemini-2-flash
llm_profile: dnd-extraction
retries: 2
references:
spell_catalog: ./dnd-spell-catalog.json
@@ -215,7 +273,7 @@ extract:
| Binding field | Type | Default | Rules |
| --- | --- | --- | --- |
| **module** | string | none | Required for an object binding. Must be a registered compatible key. |
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID. |
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID for an LLM-backed binding. It overrides the pipeline default unless the run supplies **--llm-profile**. |
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. |
| **options** | object | none | Must satisfy the selected module. |
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
@@ -225,7 +283,8 @@ Omitting **validators** uses the registered chain. **validators: []** selects
an empty chain; a non-empty list replaces the chain in the listed order.
Validator bindings accept only **module**, **llm_profile**, and **options**.
They reject **references**, **retries**, and nested **validators**. Deterministic
validators reject an explicit **llm_profile**.
validators reject an explicit **llm_profile**. Deterministic module bindings
also reject an explicit **llm_profile**.
The **json** output module accepts optional **include_chunk_map** and
**evidence_context** settings:

View File

@@ -1,11 +1,11 @@
# PromptKit Integration
Notarius pins
[`gitea.maximumdirect.net/eric/promptkit` v0.2.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0)
[`gitea.maximumdirect.net/eric/promptkit` v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0)
as its in-process prompt engine. The upstream
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0/docs/consumers/pkg-promptkit.md)
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/consumers/pkg-promptkit.md)
owns the public engine API, and the upstream
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.2.0/docs/formats.md)
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
owns prompt, profile, and schema file contracts.
## Supported Boundary
@@ -13,15 +13,35 @@ owns prompt, profile, and schema file contracts.
Notarius relies on the root `promptkit` package to:
- construct an `Engine` with filesystem-backed prompt, schema, and optional
profile sources;
- prepare and run a `RunRequest` with named inline artifacts, variables,
a direct session ID, prompt identity, and profile selection;
operator and application-fallback profile sources;
- prepare one frozen execution from a `RunRequest` with named inline artifacts,
variables, a direct session ID, prompt identity, and profile selection, then
record credential-redacted details and run that exact execution;
- return rendered debug material, validated structured output, selected
profile, backend, effective model metadata, and token usage;
- register the optional conventional `local` backend through `BackendLocal`,
`LocalBackend`, and `WithBackend`;
- distinguish structured-output validation failure from execution failure; and
- identify a missing explicit profile through `ErrProfileNotFound` and backend
admission exhaustion through `ErrCapacityExceeded`.
The pinned
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/backends.go)
owns the registration and backend-capacity contract.
For one completion, the adapter calls `PrepareExecution`, takes a
caller-owned `Details` snapshot, and calls `RunPrepared` for that same opaque
prepared execution. It defers `Discard` for every unexecuted handle. Explicit
profile preflight uses `Engine.InspectProfile`; it does not prepare a synthetic
prompt. PromptKit's prepared handle, inspection result, and capacity-error
types stay inside the Notarius LLM adapter.
When a PromptKit profile and runtime override leave `temperature`, `max_tokens`,
or `top_p` unset, Notarius leaves that control unset as well. Compatible
providers therefore apply their own defaults; an operator that requires a
specific sampling value must select it explicitly in the profile or runtime
override.
Notarius does not use PromptKit's optional `ArtifactReader`. It materializes
source and reference content itself and supplies owned inline artifacts at the
adapter boundary. It also retains responsibility for pipeline retries,
@@ -38,15 +58,36 @@ Notarius records PromptKit's selected backend ID and effective reasoning
setting as optional run-manifest provenance. Endpoint-only profiles have no
backend ID. Debug prompt material also retains the selected backend ID and
PromptKit's stable lower-case `effective_model_params` JSON, which may include
`backend_id`. Notarius production configuration does not expose user-defined
PromptKit backend registration.
`backend_id`. Notarius production configuration exposes one optional
conventional `local` registration. It does not expose a general user-defined
PromptKit backend registry. Endpoint-only profiles remain supported unchanged.
Notarius retains its application-wide scheduled client around the PromptKit
adapter. PromptKit may apply a narrower limit for the selected backend;
endpoint-only profiles have no such backend limit. The adapter translates
PromptKit capacity rejection into the provider-neutral Notarius
`ErrLLMCapacityExceeded` contract and leaves retries to the calling pipeline
stage.
`ErrLLMCapacityExceeded` contract. It may include the normalized selected
backend ID in safe diagnostic context, without exposing PromptKit's capacity
error type, and leaves retries to the calling pipeline stage.
## Profile Sources And Compatibility
Notarius gives PromptKit the configured operator profile source, registered
application fallback profile assets, and optional backend registration through
the same construction path for inspection and execution. PromptKit owns the
resulting source precedence and strict profile parsing: a matching operator
profile is a complete replacement for a fallback or built-in profile, while an
invalid matching document fails instead of falling through. The operator
configuration and deployment workflow are defined in
[Configuration](../config.md#promptkit-profiles) and
[Operations](../operations.md#promptkit-profile-deployment).
Notarius supports this boundary against PromptKit v0.5.0. Its fallback source,
prepared-execution, inspection, and typed capacity APIs are used as public
upstream contracts; other PromptKit APIs or file-format behavior are not
implicitly supported. A dependency upgrade requires reviewing the adapter,
profile-source construction, and this compatibility statement against the
pinned upstream documentation.
## Notarius Ownership
@@ -54,8 +95,11 @@ stage.
module assets, maps its transport-neutral completion contract, prepares and
executes requests, validates output, records provenance, captures debug
material, redacts errors, and preserves timeout ownership.
[D&D Module Internals](../internal/dnd.md) owns the embedded
`dnd-extraction` fallback profile and the maintained D&D prompt defaults.
[Configuration](../config.md#promptkit-profiles) defines how a Notarius
configuration selects one PromptKit profile source.
configuration selects one PromptKit profile source and optionally registers
the conventional local backend.
PromptKit API or format changes outside this boundary are not implicitly
supported. Updating the pinned version requires reviewing the adapter and

View File

@@ -38,11 +38,14 @@ in [Configuration Internals](configuration.md).
Configuration validation without a selected pipeline checks structural
configuration only. Validation with a selected pipeline also builds the
effective catalog, resolves the pipeline, and verifies explicitly selected
PromptKit profiles. Each explicit binding or validator profile is prepared
against the configured PromptKit source without performing generation, so an
unknown profile fails before pipeline preparation. Pipeline listing validates
configuration before returning normalized, sorted identifiers.
effective catalog, resolves the pipeline, and verifies every explicit effective
PromptKit profile. Selected LLM-backed input, chunk, lane, output, and validator
profiles are inspected
against the configured PromptKit source and backend registrations without
loading a prompt or performing generation, so an unknown or invalid profile
fails before pipeline preparation. Credential availability remains an
execution-time concern. Pipeline listing validates configuration before
returning normalized, sorted identifiers.
## Production Composition
@@ -59,7 +62,8 @@ profile-provenance recorder, creates one scheduler from the effective global
LLM limit, and wraps the client before it reaches modules. Registration and LLM
construction errors are returned before a pipeline is prepared. Configuration
field definitions remain in [Configuration](../config.md#promptkit-profiles);
the adapter mechanics remain in [LLM Runtime](llm.md).
the D&D registrar's fallback profile assets and the adapter mechanics remain in
[LLM Runtime](llm.md).
The factory also accepts `LLMRuntimeOverrides`, whose reasoning pointer
preserves inherit, replace, and clear states across the composition boundary.
@@ -81,7 +85,8 @@ handoff:
2. create and validate a safe run identity, then allocate a debug bundle only
when requested;
3. build the effective catalog, resolve requested reference changes, resolve
the effective pipeline, and verify explicit PromptKit profiles;
the effective pipeline, and inspect its explicit effective PromptKit
profiles;
4. materialize external or generated references and record redacted invocation
and resolution provenance when debug capture is enabled;
5. construct registries, the scheduled LLM client, prepared modules, and the

View File

@@ -39,14 +39,22 @@ This establishes the public precedence order without giving environment input a
second file schema. Loading and application reject malformed YAML, unsupported
file versions, unknown fields, invalid values, and identifiers that are empty
or collide after whitespace normalization. The file application also makes the
effective extraction-worker default follow the effective LLM limit.
effective extraction-worker default follow the effective LLM limit. A present
PromptKit local-backend object requires and trims its endpoint, defaults its
omitted concurrency limit to zero, and is copied so the parsed file model
cannot alias the populated **Config**. A pipeline `llm_profile` is
presence-aware: omission remains empty, while a present blank value is
rejected and a non-empty file value is trimmed before it reaches **Config**.
**Config.Validate** checks configuration-only invariants before resolution. It
rejects incompatible profile sources, invalid state-surface values, unsupported
concurrency settings, malformed bindings and references, invalid retries, and
invalid pipeline, step, or lane structure. Its errors retain the closest known
pipeline, lane, and binding context. It deliberately does not require modules
to be registered: that requires a catalog and belongs to resolution.
invalid pipeline, step, or lane structure. PromptKit local-backend validation
accepts only an absolute HTTP or HTTPS endpoint with a host and no user
information, query, or fragment, and rejects a negative local concurrency
limit. Its errors retain the closest known pipeline, lane, and binding context.
It deliberately does not require modules to be registered: that requires a
catalog and belongs to resolution.
The exact user-selectable values and validation rules are defined in
[Configuration](../config.md). Keep additions to the file model, an
@@ -56,13 +64,15 @@ environment override, its validation, and that reference in the same change.
**Config.Resolve** first recomputes derived concurrency defaults and validates
the configuration. It normalizes the requested pipeline ID, copies the selected
profile, applies a non-empty command-level LLM profile override to the
LLM-capable stage bindings, and calls the framework resolver with the requested
lane selection and reference changes.
profile, and passes the non-empty command-level LLM profile override, requested
lane selection, and reference changes to the framework resolver.
The command-level override does not replace an explicitly selected validator
profile. Validator bindings remain part of the resolved validator chain and
are resolved under their own declared configuration.
After module and validator selection, the resolver applies the effective
profile policy to LLM-backed bindings only: command override, binding profile,
pipeline profile, then the prompt default. Deterministic bindings remain
profile-free, and no second inheritance decision occurs during execution. The
public field definitions and precedence are owned by
[Configuration](../config.md#pipelines).
The framework resolver supplies defaults, selects lanes, resolves validator
chains, checks registered module and artifact compatibility, validates module
@@ -71,7 +81,8 @@ options, and returns the fixed ordered pipeline shape. The resulting
changes, a clone of the input configuration, and the resolved pipeline.
Callers may therefore retain or modify their input slices and maps without
changing the resolved result, and later consumers cannot mutate the original
configuration through the effective value.
configuration through the effective value. This ownership includes the nested
PromptKit local-backend value.
Resolution failures stop before module construction and source parsing. They
include an error path for an unconfigured pipeline, missing module, missing
@@ -83,7 +94,8 @@ runtime error class described in the [CLI reference](../cli.md#output-streams-an
The framework assigns the resolved pipeline a deterministic SHA-256 digest
after defaults, lane selection, module bindings, reference bindings, validator
chains, and artifact schema identity have been resolved. The digest excludes
chains, effective LLM profiles, and artifact schema identity have been
resolved. The digest excludes
its own stored value. It identifies resolved composition rather than raw YAML
bytes, a debug payload, or all runtime state. The CLI records it as invocation
provenance before execution; cache and checkpoint identity have additional
@@ -94,9 +106,12 @@ Configuration summaries must use **Redacted**, **RedactedSummaryPayload**, or
Those methods copy every binding and nested option container, replace values
whose key is credential-shaped with **[REDACTED]**, and omit materialized
reference content while retaining safe binding and reference provenance. The
payload must not alias the source configuration or resolved pipeline. This
redaction is deliberately narrow: it protects configuration summaries and does
not authorize recording arbitrary environment values or provider requests.
payload must not alias the source configuration or resolved pipeline.
PromptKit's local endpoint and concurrency limit are preserved as non-secret
configuration metadata in the independently owned summary; the object contains
no credential value. This redaction is deliberately narrow: it protects
configuration summaries and does not authorize recording arbitrary environment
values or provider requests.
## Invariants To Preserve

View File

@@ -22,10 +22,15 @@ does not repeat their JSON shapes or schemas.
## Family Composition
The D&D registrar registers the familys artifact codecs, extractors, typed
append-order mergers, normalizers, validators, prompt assets, and default
validator chains. Each extractor and normalizer has a stable module spec,
strict option decoding, and a typed builder. Configuration remains the
canonical owner of the exact keys and validator order.
append-order mergers, normalizers, validators, prompt assets, fallback LLM
profile asset, and default validator chains. Each extractor and normalizer has
a stable module spec, explicit execution class, strict option decoding, and a
typed builder. Scene chunking, every extractor, and NPC normalization are
registered as `llm_backed`; the remaining current D&D mergers and normalizers
are `deterministic`. The metadata is available to catalog inspection and
resolved-pipeline debug data and determines which selected bindings inherit the
pipeline profile. Configuration remains the canonical owner of the exact keys,
profile precedence, and validator order.
Private structured-LLM response schemas are deliberately minimal. They reject
invalid JSON structure, missing required fields, incompatible types, and
@@ -41,6 +46,15 @@ reference, and transcript assets instead of copying their text into individual
modules. A manifests declared sequence, including cache-control placement, is
part of the prompt behavior.
Every maintained D&D LLM prompt selects `dnd-extraction` as its default
profile. The D&D registrar embeds that fallback profile with the maintained
OpenRouter model, timeout, and service-tier policy. An operator may provide a
complete profile with the same ID through the configured PromptKit source; that
definition replaces the fallback rather than merging with it. The fallback
leaves reasoning and optional sampling controls unspecified. Deployment profile
selection and the maintained operator example are documented in
[Configuration](../config.md#promptkit-profiles).
All extraction prompts share this four-message rendered prefix: the system
message without cache control, the identity message without cache control, the
campaign-reference message with ephemeral cache control, and the chunk

View File

@@ -28,13 +28,15 @@ durable schemas. Those responsibilities remain with the module and its
named material to a PromptKit inline artifact while preserving its origin URI,
maps the trimmed request session to PromptKit's direct per-run session field,
retains the same value as the `session_id` prompt variable for maintained
prompt compatibility, forwards profile selection, then prepares and runs the
prompt. The direct field is authoritative for provider session behavior. A
session ID is a stable, non-secret correlation identifier and may be exposed
to providers and provider observability. The adapter returns PromptKits
validated raw bytes rather than re-encoding the decoded target. An empty
optional material is represented as one space so its named input is retained
by PromptKit.
prompt compatibility, and forwards profile selection. It then creates one
frozen prepared execution, captures its caller-owned credential-redacted
details for debug material, and executes that exact snapshot through
PromptKit's prepared-execution boundary. The direct field
is authoritative for provider session behavior. A session ID is a stable,
non-secret correlation identifier and may be exposed to providers and provider
observability. The adapter returns PromptKits validated raw bytes rather than
re-encoding the decoded target. An empty optional material is represented as
one space so its named input is retained by PromptKit.
Client construction may also receive a run-wide reasoning-effort override from
the CLI factory boundary. The adapter copies the caller-owned pointer and
@@ -46,26 +48,50 @@ states. With neither flag, profile behavior remains unchanged. Because
production constructs one shared client, the selected state applies uniformly
to module calls, retries, and LLM-backed validators for the whole run.
An empty request profile lets the prompt select its configured default. The CLI
prepares every explicitly selected binding profile before a run begins, so a
missing explicit profile fails before stage execution. Calls record the profile
actually selected by PromptKit. The recorder trims and deduplicates non-secret
profile identity, provider, model, selected backend ID, and effective reasoning
values for manifest use. Entries that differ in backend or reasoning remain
distinct and deterministically ordered. Endpoint-only profiles retain an empty
backend ID, which the published JSON omits. Successful completion responses and
recorded profile manifests identify the adapter provider as `promptkit`.
An empty request profile lets the prompt select its configured default. Before a
run begins, the CLI asks the adapter to inspect every explicit profile on the
resolved selected LLM-backed bindings and validators, including inherited
pipeline profiles. Inspection resolves the profile and its selected backend and
target without loading a prompt, reading credentials, admitting capacity, or
contacting a provider, so a missing or invalid explicit profile fails before
stage execution while a valid `api_key_env` may remain unset. Calls record the
profile actually selected by PromptKit. The recorder trims and deduplicates
non-secret profile identity, provider, model, selected backend ID, and
effective reasoning values for manifest use. Entries that differ in backend or
reasoning remain distinct and deterministically ordered. Endpoint-only profiles
retain an empty backend ID, which the published JSON omits. Successful
completion responses and recorded profile manifests identify the adapter
provider as `promptkit`.
The CLI's profile-inspection engine and the production adapter use the same
profile-source construction to apply the configured profile directory or file,
the optional registered fallback profile assets, and the optional conventional
`local` backend. Preflight therefore resolves the same profile sources and
backend membership as runtime without performing generation. Fallback assets
are mounted only when at least one source is registered. The production D&D
registrar contributes its `dnd-extraction` fallback, and the maintained D&D
prompts select that logical ID by default. PromptKit owns source precedence and
profile parsing: an operator-provided matching profile takes precedence over a
fallback profile without Notarius merging either document.
When the registration is absent, a profile selecting `backend: local` fails
inspection instead of falling back to a built-in or endpoint-only target.
Before execution, the adapter also contributes a non-secret checkpoint
fingerprint for the effective PromptKit profile source. It combines the
identity of PromptKit's compiled-in profile catalog with a deterministic digest
of every YAML profile in the configured profile directory, or of the configured
profile file. The fingerprint contains neither profile content nor source
paths. It covers both explicit binding profiles and prompt-selected defaults,
so changing a model or other profile setting cannot reuse checkpoints created
under the prior profile source. This cache identity is independent of durable
profile file, and a deterministic digest of the flattened fallback profile
assets. The fingerprint contains neither profile content nor source paths. It
covers inherited pipeline profiles, explicit binding profiles, and
prompt-selected defaults, so changing a model or other profile setting cannot
reuse checkpoints created under the
prior profile source. This cache identity is independent of durable
profile provenance: run manifests continue to list only profiles actually
observed during LLM calls.
observed during LLM calls. When the local backend is registered, a second
fingerprint hashes its trimmed endpoint behind a stable marker. Changing that
semantic execution target invalidates checkpoint reuse. The raw endpoint is not
stored in checkpoint identity, and the local concurrency limit is excluded
because it changes scheduling rather than execution semantics.
## Shared Provider-Call Limit
@@ -85,15 +111,19 @@ PromptKit applies a second, independent admission limit when the selected
profile names a limited backend. It sits beneath the Notarius scheduled client,
so it may narrow but cannot expand the application-wide limit. Built-in
OpenRouter profiles select PromptKit's reserved backend and its upstream
capacity policy. Endpoint-only profiles do not select a PromptKit backend and
remain limited only by the Notarius scheduler.
capacity policy. A positive configured local-backend limit bounds active local
generations inside PromptKit; zero leaves that backend unlimited there.
Endpoint-only profiles do not select a PromptKit backend and remain limited
only by the Notarius scheduler.
## Prompt And Schema Assets
An `AssetRegistry` collects prompt and schema filesystems from production module
families. It flattens registered roots into the PromptKit filesystems and
rejects invalid roots, unreadable assets, duplicate paths, and missing prompt
or schema files during preparation. The frameworks `promptfs` helper combines
An `AssetRegistry` collects prompt, schema, and optional fallback-profile
filesystems from production module families. It flattens registered roots into
the corresponding PromptKit filesystems and rejects invalid roots, unreadable
assets, duplicate paths, and missing prompt or schema files during preparation.
Fallback assets receive a safe content digest for checkpoint identity; raw
paths and bytes are never included. The frameworks `promptfs` helper combines
module-owned prompt files with reusable domain fragments without making the
framework depend on D&D content.
@@ -142,10 +172,12 @@ than output-validation failures.
When PromptKit rejects backend admission before generation, the adapter maps
`promptkit.ErrCapacityExceeded` to
`contracts.ErrLLMCapacityExceeded`, retaining prompt context and a redacted
upstream diagnostic without exposing the PromptKit sentinel as a framework
contract. A canceled caller context takes precedence. The adapter does not
retry capacity failures; the pipeline's existing binding attempt policy sees
the operational error and decides whether to rerun the complete operation.
upstream diagnostic without exposing the PromptKit sentinel or capacity-error
type as a framework contract. When supplied, the normalized selected backend
ID appears only in that safe application-owned diagnostic context. A canceled
caller context takes precedence. The adapter does not retry capacity failures;
the pipeline's existing binding attempt policy sees the operational error and
decides whether to rerun the complete operation.
Prompt-declared repair is executed within PromptKits structured-output flow.
The current production D&D prompt manifests set repair attempts to zero. That

View File

@@ -12,9 +12,15 @@ exceptions. See [D&D Module Internals](dnd.md) rather than adding them here.
A module is a typed implementation registered for one pipeline stage. Its
`ModuleSpec` is the public-to-the-framework declaration of its stable key,
stage, required and provided capabilities, artifact kind, and accepted
reference slots. The framework uses that declaration to resolve a configured
binding before it builds the implementation.
stage, execution class, required and provided capabilities, artifact kind, and
accepted reference slots. The execution class states whether a module is
`deterministic` or `llm_backed`; registries retain it for catalog inspection and
resolved-pipeline debug data without constructing the module. The framework
uses the declaration to resolve a configured binding before it builds the
implementation. After selection, the resolver applies profile inheritance only
to bindings whose declared execution class is `llm_backed` and rejects a
binding-specific profile on a deterministic module. The user-facing precedence
contract belongs in [Configuration](../config.md#pipelines).
Implementations that accept options must provide both an option validator and
a builder. The validator is used while resolving configuration; the builder
@@ -47,7 +53,7 @@ Production composition is intentionally split by family:
- The Seriatim registrar provides the transcript input adapter. Its external
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
- The D&D registrar provides its codecs, extractors, mergers, normalizers,
validators, prompt assets, and default chains. Its behavioral conventions
validators, prompt assets, fallback profile asset, and default chains. Its behavioral conventions
are documented in [D&D Module Internals](dnd.md).
The CLI owns the composition that invokes these registrars. A module package
@@ -59,10 +65,10 @@ packages depend on production extensions.
1. Choose the pipeline stage and the typed artifact boundary. Put external
input or durable artifact formats in the relevant integration contract,
not in this guide or in a private LLM response type.
2. Define a stable `ModuleSpec` with the exact capabilities and reference
slots needed for the operation. Model a producer/consumer handoff as an
artifact-compatible slot; configuration then chooses an external file or a
generated binding.
2. Define a stable `ModuleSpec` with an explicit execution class, the exact
capabilities, and reference slots needed for the operation. Model a
producer/consumer handoff as an artifact-compatible slot; configuration
then chooses an external file or a generated binding.
3. Implement strict option decoding, construction, and the typed stage
interface. Preserve caller ownership: do not retain mutable request data
and return defensive copies where an implementation exposes stored data.

View File

@@ -34,6 +34,11 @@ requested lanes where that is supported, resolves validator chains, checks
module capabilities and typed artifact compatibility, validates options, and
assigns a deterministic resolved-composition digest. The resolved pipeline
contains bindings and declared reference targets, not external reference bytes.
After selection, the resolver applies command, binding, and pipeline profile
precedence to LLM-backed bindings and validators only; prompt defaults remain
an empty resolved binding profile. Deterministic bindings remain profile-free.
These effective values are part of the digest, so execution and checkpoint
consumers do not repeat profile inheritance.
Configuration resolution supplies the selected profile and catalog; see
[Configuration Internals](configuration.md).

View File

@@ -42,6 +42,42 @@ evidence publication. Apply an appropriate umask and output-root access policy
before enabling that option; the requested output modes alone may not be
suitable for transcript-bearing bundles.
## PromptKit Profile Deployment
Profile deployment has four distinct layers:
| Layer | Owner | Operational role |
| --- | --- | --- |
| Prompts and schemas | Notarius module families | Embedded request and structured-output definitions. They are not deployment profile files. |
| Fallback profiles | Notarius module families | Embedded application defaults, including D&D's `dnd-extraction` profile. |
| Built-in profiles | PromptKit | Upstream catalog entries available when no higher-precedence source defines an ID. |
| Operator profiles | Deployment filesystem | Complete environment-specific definitions selected by `promptkit.profile_file` or `promptkit.profile_dir`. |
The maintained D&D pipeline uses the workload ID `dnd-extraction`. The
embedded fallback makes that ID usable without an operator file. Production,
development, and local deployments can each install a different complete
definition for the same ID, retaining the pipeline while choosing their own
model, backend, timeout, or reasoning policy. An operator definition wins over
the fallback; it is not merged with it. The configuration field and full
precedence rules are owned by [Configuration](config.md#promptkit-profiles).
Use a profile source owned by the service account, keep it readable only by
the intended operator, and supply provider credentials through the service
environment—not in the Notarius configuration or profile YAML. The maintained
[operator profile](../examples/profiles/dnd-extraction.yml) is secret-free and
can be copied as a format starting point. Validate a deployment without a
provider call or credentials:
~~~sh
notarius config validate --config /etc/notarius/config.yml --pipeline dnd-session
~~~
Profile paths are currently resolved from the process working directory, not
from the configuration file. The complete example's
`./examples/profiles/dnd-extraction.yml` path is valid for a repository-root
invocation only. Use absolute paths such as
`/etc/notarius/profiles/dnd-extraction.yml` for services and containers.
## Run Lifecycle
Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads
@@ -118,8 +154,10 @@ disabled. Without **--resume**, a recording-enabled run executes normally and
does not load checkpoint state. Compatibility includes the resolved pipeline,
input, selected lanes, runtime overrides, reference provenance, LLM-profile
provenance, the effective PromptKit profile-source fingerprint, and
prepared-component fingerprints. Changing profile content causes a cold miss
even when its profile ID is unchanged. A changed identity produces a cold miss;
prepared-component fingerprints. When a local PromptKit backend is configured,
compatibility also includes a non-secret fingerprint of its endpoint. Changing
profile content or the local endpoint causes a cold miss; changing only the
local concurrency limit does not. A changed identity produces a cold miss;
Notarius does not migrate, rewrite, or delete older checkpoint directories.
Reasoning-effort inheritance, replacement, and explicit clearing are distinct
runtime identities, so checkpoints created under one state are not reused by
@@ -217,7 +255,7 @@ Provider execution settings and the generation timeout come from the selected
PromptKit profile. The invocation-only **--reasoning-effort** and
**--clear-reasoning-effort** controls may replace or clear that profile setting
for all LLM-backed calls in one run without changing the profile. PromptKit
v0.2.0 does not add a provider retry loop. Notarius binding retries rerun the
v0.5.0 does not add a provider retry loop. Notarius binding retries rerun the
complete module operation and validation chain as defined by
[module bindings](config.md#module-bindings-and-validators).
@@ -234,13 +272,19 @@ and validators. PromptKit may impose a narrower admission limit for the
selected backend. The effective active-generation bound is the intersection of
both limits and can therefore be lower than **total_llm**. Built-in OpenRouter
profiles use PromptKit's upstream backend limit; endpoint-only profiles have no
PromptKit backend limit and remain bounded by Notarius.
PromptKit backend limit and remain bounded by Notarius. For the configured
local backend, a zero **concurrency_limit** leaves only the Notarius scheduler
as a call limit. A positive value makes the effective active local-generation
bound the smaller of **total_llm** and that local limit.
When a PromptKit backend has admitted all active and queued work, a new call
fails as capacity exhaustion before generation. The adapter does not retry it.
The calling stage's configured retry policy applies normally, and the run fails
if those attempts are exhausted. Caller cancellation remains authoritative.
Configuration contracts are documented under
For a positive local limit, PromptKit owns its default waiting capacity and
admission behavior. When a PromptKit backend has admitted all active and queued
work, a new call fails as capacity exhaustion before generation. The adapter
maps that failure to Notarius's existing provider-neutral capacity error and
does not retry it. The calling stage's configured retry policy applies
normally, and the run fails if those attempts are exhausted. Caller
cancellation remains authoritative. Configuration contracts are documented
under [PromptKit profiles](config.md#promptkit-profiles) and
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
limits and actual provider-call limits are independent. Notarius writes local
filesystem state only; remote storage, archival, and retention automation are

View File

@@ -53,6 +53,71 @@ not as committed release dates.
spell, combat, interaction, and scene-description lanes after real-world use.
Add more complex chunking only in response to demonstrated failures.
## Cross-Cutting LLM Runtime
### Deterministic Prompt Session Identity
- Replace the source-document-ID default for prompt sessions with one
predictable, procedurally generated session ID for the complete
source-processing workload.
- Preserve an explicit non-empty `--session-id` as the highest-precedence
override. Otherwise, derive the default only from the effective input module
identity and the exact raw input bytes.
- Use a versioned, bounded representation such as
`notarius:v1:<sha256(input-module + NUL + raw-input)>`. The exact encoding
must fit PromptKit's session length contract and must not embed source
content.
- Keep the derived session stable across runs, pipelines, selected lanes,
ordered steps, retries, resume, recomputation, LLM profiles, reasoning
overrides, and output, debug, or cache settings.
- Do not include file-backed references, generated references, reference
contents, or the composition of a reference bundle in session derivation.
References may change between prompt calls within one pipeline without
changing routing affinity.
- Resolve the authoritative session before checkpoint construction and use the
same value for checkpoint runtime identity, every prompt-facing module,
PromptKit's direct session field, the compatibility `session_id` prompt
variable, run-manifest metadata, and debug metadata.
- Keep routing identity separate from cache and checkpoint content identity.
Exact prompt prefixes, reference contents, model settings, and other
generation-affecting inputs must continue to participate in their existing
hashes and checkpoint fingerprints even though they do not change the
session.
- Treat the generated value as a provider-visible, stable pseudonymous
correlation identifier. Do not introduce an installation-specific HMAC or
secret unless a concrete multi-tenant or privacy requirement justifies
sacrificing deterministic identity across installations.
### Raise The Default Application-Wide LLM Limit
- Raise the default `concurrency.total_llm` value from 1 to 16 so ordinary
single-backend runs can use PromptKit's expected OpenRouter capacity and
lower-capacity local backends without an unnecessarily narrower Notarius
limit.
- Keep the Notarius application-wide scheduler mandatory and require
`total_llm` to remain a positive integer. Do not make the default unlimited:
endpoint-only profiles, an unrestricted local backend, injected clients, and
aggregate work across several backends may have no narrower PromptKit limit.
- Continue defaulting `concurrency.stage_workers.extract` to the effective
`total_llm`, making its default 16 as part of the same change. Preserve an
explicit lower extract-worker setting when an operator wants less queued or
concurrent extraction work.
- Define effective provider concurrency as the intersection of the Notarius
application-wide limit, the selected PromptKit backend limit when present,
and the work made available by stage execution. A Notarius limit of 16 does
not narrow a backend already limited to 16, while a local backend limited to
4 remains bounded at 4.
- Treat the default as an application-wide safety ceiling across profiles,
backends, modules, retries, and validators. A run that intentionally needs
the combined capacity of several backends may configure a higher
`total_llm` and an appropriate extract-worker count explicitly.
- Retain the existing configuration and environment override surfaces. Update
canonical configuration, operations, and internal documentation together
when the default changes.
- Reconsider decoupling the extract-worker default from `total_llm` only after
mixed-backend workloads demonstrate a need for a high global emergency
ceiling with a lower default work-production rate.
## Shared Normalization And Quality Work
### Generic LLM-Assisted Deduplication

View File

@@ -0,0 +1,586 @@
# PromptKit v0.5 Implementation Plan
## Objective
Implement the target state in
[PromptKit v0.5 Integration And LLM Profile Policy](promptkit.md). Each numbered
stage is intended to be one implementation prompt for a GPT-5.6-Terra coding
agent. Complete stages in order and leave the repository buildable, tested, and
internally coherent after every stage.
Follow [Architecture](../policy/architecture.md),
[Testing Policy](../policy/testing.md), and
[Documentation Policy](../policy/documentation.md) throughout. Preserve
unrelated user changes. Use `apply_patch` for source and documentation edits,
run `gofmt` on changed Go files, and add only tests that protect the behaviors
and risks assigned to that stage.
Do not implement the separate deterministic session-ID or default-concurrency
roadmap items as part of this plan. Do not perform paid or credentialed LLM
calls.
## Background Summary
Notarius currently pins PromptKit v0.3.0, calls `Prepare` and then `Run` for one
completion, validates profiles through a synthetic prompt, has no application
fallback profile source, and accepts LLM profiles only at individual bindings
or through the run-wide CLI override. PromptKit v0.5.0 is source-compatible
with the current tree; a temporary v0.5.0 module override has already passed
`go test ./...`.
The implementation must nevertheless treat the upstream optional-parameter
change as intentional: unset `temperature`, `max_tokens`, and `top_p` remain
unset and are omitted from compatible provider requests. Do not restore the old
implicit `top_p: 1` default.
## Stage 1: Upgrade The PromptKit Dependency
### Goal
Establish a clean PromptKit v0.5.0 baseline before adopting its new APIs.
### Work
- Update `go.mod` and `go.sum` from PromptKit v0.3.0 to v0.5.0 and run
`go mod tidy`.
- Change the PromptKit built-in profile-catalog marker in
`internal/framework/llm/promptkit_profile_fingerprint.go` to identify
v0.5.0. This deliberately invalidates LLM checkpoints tied to the prior
catalog identity.
- Review PromptKit-facing compile errors or test failures against the v0.4.0
and v0.5.0 release guides. Do not adopt prepared execution, inspection, or
fallback profiles in this stage.
- Replace the existing test assertion for one exact built-in fingerprint hash
with durable assertions that the fingerprint is deterministic, non-empty,
non-secret, and changes when a semantic profile source changes. Do not add a
new version-constant or exact-hash change detector.
- Update `docs/integrations/pkg-promptkit.md` to pin and link v0.5.0 and state
the implemented dependency-level behavior: unset optional sampling controls
are provider defaults. Do not document later stages as implemented.
- Update any other canonical text that explicitly claims the dependency is
v0.3.0, but defer descriptions of unimplemented v0.5 APIs.
### Tests And Validation
- `go test ./internal/framework/llm ./internal/cli`
- `go test ./...`
- `go vet ./...`
- `go build ./cmd/notarius`
- `rg -n 'promptkit v0\.3\.0|promptkit@v0\.3\.0|PromptKit v0\.3\.0' .`
- `git diff --check`
### Completion Criteria
- The repository directly pins v0.5.0 and all default offline checks pass.
- The profile-source fingerprint identifies the new upstream catalog without a
brittle literal-hash test.
- Current documentation no longer identifies v0.3.0 as the supported version.
## Stage 2: Execute One Frozen Prepared Snapshot
### Goal
Make Notarius debug details and generation use one exact PromptKit preparation.
### Work
- Refactor `PromptKitClient.CompleteStructured` to call
`PrepareExecution`, immediately defer `Discard`, obtain a caller-owned
`Details` value, and execute with `RunPrepared`.
- Preserve the existing Notarius request mapping, cancellation precedence,
validation classification, raw structured bytes, response decoding,
profile recording, usage reporting, and credential redaction.
- Ensure every preparation, execution, validation, empty-result, and decode
error retains useful Notarius prompt context without exposing prepared handle
state or secrets.
- Use `errors.As` to obtain `*promptkit.CapacityError` on admission rejection.
Preserve `contracts.ErrLLMCapacityExceeded` as the stable classification and
add a nonblank backend ID only to safe application-owned diagnostic context.
Do not expose `promptkit.CapacityError` outside the LLM adapter.
- Update `docs/internal/llm.md` and the implemented-mechanics portion of
`docs/integrations/pkg-promptkit.md` to describe the single frozen execution
snapshot and structured capacity adaptation.
### Tests And Validation
- Adapt existing PromptKit client tests to the prepared-execution path.
- Retain or add one behavioral test proving that the debug prompt details match
the request actually passed to generation when a backing prompt source could
otherwise change between independent preparations. Test the resulting
snapshot consistency, not a private helper call count.
- Retain capacity tests proving `errors.Is` reaches
`contracts.ErrLLMCapacityExceeded`, the selected backend can appear in safe
diagnostic context, and provider calls are not made after rejected
admission.
- Run `go test ./internal/framework/llm` and
`go test -race ./internal/framework/llm`.
- Run `go test ./...` and `git diff --check`.
### Completion Criteria
- `CompleteStructured` no longer calls independent `Prepare` and `Run`
operations for one request.
- Debug prompt material and generation result originate from the same frozen
PromptKit snapshot.
- Capacity remains a provider-neutral Notarius error classification.
## Stage 3: Replace Synthetic Profile Validation With Inspection
### Goal
Validate profiles through PromptKit's exact profile-inspection boundary and
centralize engine profile-source construction.
### Work
- Introduce a small provider-adapter-owned profile inspection or validation
function in `internal/framework/llm`. Its public internal signature must use
Notarius-owned configuration and result/error types rather than returning
PromptKit types to the CLI.
- Share the code that applies `profile_dir`, `profile_file`, and registered
backend options between the production PromptKit engine and the inspection
engine. Preserve the mutual-exclusion and local-backend rules.
- Change CLI explicit-profile preflight to use `Engine.InspectProfile` through
that LLM boundary.
- Remove `profileCheckPromptID`, `profileCheckPromptFS`, the `testing/fstest`
production dependency, and the synthetic `Prepare` request.
- Preserve distinct, useful errors for an absent profile, invalid profile,
unknown backend registration, cancellation, and invalid profile source.
- Do not require `api_key_env` to be populated during configuration validation.
Inspection may report credential requirements internally, but actual
preparation remains responsible for credential availability before a model
call.
- Update current-behavior sections in `docs/internal/cli.md` and
`docs/internal/llm.md`. Keep field definitions in `docs/config.md`.
### Tests And Validation
- Replace synthetic-prompt tests with profile inspection tests covering:
configured local backend success; missing local backend failure; absent
profile; malformed profile; and an otherwise valid profile whose credential
environment variable is intentionally unset.
- Prove validation performs no provider HTTP call and remains offline.
- Run `go test ./internal/framework/llm ./internal/cli` and `go test ./...`.
- Run `git diff --check`.
### Completion Criteria
- No production synthetic profile-check prompt remains.
- Profile validation uses the same ordinary profile source and backend
registrations as execution.
- Configuration validation succeeds for structurally valid profiles without
reading credential values.
## Stage 4: Add Application Fallback Profile Asset Plumbing
### Goal
Allow module families to register application-owned fallback profile YAML
without placing domain policy in generic LLM code.
### Work
- Extend `internal/framework/llm.AssetRegistry` with a separate fallback
profile source collection, registration method, flattened filesystem, and
safe content digest.
- Reuse the existing asset-source path validation and flattening behavior where
appropriate. Reject invalid roots, unreadable assets, and duplicate flattened
paths. Do not parse PromptKit profile YAML in Notarius.
- Add `promptkit.WithFallbackProfileFS` to production engine options only when
at least one fallback profile source is registered.
- Supply the identical assembled fallback source to the profile-inspection
engine. Adjust CLI composition so pipeline-aware profile validation can use
the production LLM asset registry without exposing PromptKit types.
- Extend profile-source checkpoint identity to include the exact fallback
profile asset digest in addition to the PromptKit catalog marker and operator
source. Keep the resulting fingerprint hash-only and path/content/credential
free.
- Keep operator source precedence owned by PromptKit. Do not implement profile
merging or duplicate PromptKit source resolution in Notarius.
- Update `docs/internal/llm.md` only for the new implemented generic asset and
fingerprint mechanics. No domain fallback exists until Stage 5.
### Tests And Validation
- Add focused AssetRegistry tests for successful flattening, invalid roots,
duplicate paths, and hash changes when fallback bytes change.
- Add adapter-level tests showing that the fallback filesystem reaches both
execution construction and inspection construction.
- Extend checkpoint tests to prove fallback content changes profile-source
identity without exposing raw YAML or paths. Use relational comparisons, not
a fixed hash literal.
- Run `go test ./internal/framework/llm ./internal/cli` and `go test ./...`.
- Run `git diff --check`.
### Completion Criteria
- Generic plumbing can carry application fallback profiles while remaining
unaware of D&D IDs or model settings.
- Inspection, execution, and checkpoint identity use the same fallback asset
source.
## Stage 5: Adopt The D&D `dnd-extraction` Fallback
### Goal
Give the D&D module family one stable embedded workload profile that operators
can replace.
### Work
- Add a D&D-owned embedded PromptKit profile asset with ID `dnd-extraction`
under `internal/modules/dnd`. Use the exact baseline defined in
`promptkit.md`: OpenRouter, `openai/gpt-5.6-luna`, no explicit reasoning
effort, a 240-second timeout, flex service tier, and no selected temperature,
token limit, or `top_p`. The omitted reasoning value intentionally allows
OpenAI's backend to apply its `medium` default.
- Register the profile filesystem from the D&D registrar through the generic
fallback profile asset boundary. Keep D&D policy out of
`internal/framework/llm` and the CLI composition root.
- Change every maintained D&D LLM prompt definition—including scene chunking,
all D&D extractors, and NPC normalization—from the model-named default to
`default_profile: dnd-extraction`.
- Add an integration-level profile-resolution test proving that:
- the fallback resolves when no operator source defines the ID;
- a valid operator profile with the same ID wins completely; and
- an invalid matching operator profile fails rather than falling through.
- Test through Notarius's assembled production assets and PromptKit boundary;
do not duplicate every upstream source-precedence case.
- Update the implemented profile ownership and prompt-default behavior in
`docs/internal/dnd.md`, `docs/internal/llm.md`, and
`docs/integrations/pkg-promptkit.md`. Defer the complete operator walkthrough
and examples to Stage 10.
### Tests And Validation
- Run focused D&D prompt preparation tests and the production composition
tests.
- Run `go test ./internal/modules/dnd/... ./internal/framework/llm
./internal/cli`.
- Run `go test ./...`.
- Verify `rg -n 'default_profile: gemini-2-flash' internal/modules/dnd`
returns no matches.
- Run `git diff --check`.
### Completion Criteria
- All maintained D&D prompts use the application-owned logical profile ID.
- The fallback works without an operator profile and remains authoritatively
overridable by a matching valid operator definition.
## Stage 6: Introduce Module Execution-Class Metadata
### Goal
Make each production module's ability to use an LLM statically discoverable
without yet changing profile inheritance.
### Work
- Add `ExecutionClass contracts.ExecutionClass` to `pipeline.ModuleSpec` and
preserve it through normalization, cloning, catalogs, registries, JSON/debug
views, and lookup helpers.
- In this transitional stage only, allow an omitted execution class to
normalize to deterministic so existing test-only fixtures can be migrated in
Stage 7 without breaking the repository midway.
- Explicitly classify every production module:
- D&D scene chunking, every D&D extractor, and D&D NPC normalization as
`llm_backed`;
- all other current production input, chunk, merge, normalize, and output
modules as `deterministic`.
- Update production module specification tests and production catalog tests to
assert the semantic class alongside stage, artifact kind, and capabilities.
- Add catalog lookup support needed by later resolution to retrieve a selected
module's execution class by stage and key without constructing it.
- Do not implement pipeline-level profile inheritance or reject deterministic
profiles yet.
- Update `docs/internal/modules.md` and `docs/internal/dnd.md` to identify
execution class as registered module metadata, while noting only implemented
uses.
### Tests And Validation
- Run module registration/spec tests across generic, Seriatim, and D&D
families.
- Run `go test ./internal/framework/pipeline ./internal/modules/...`.
- Run `go test ./...` and `git diff --check`.
### Completion Criteria
- Every production module has an explicit correct execution class.
- Catalog consumers can retrieve that class without a concrete module
instance.
- Test-only omitted classes remain the only temporary compatibility behavior.
## Stage 7: Enforce Execution Metadata And Remove Runtime Probing
### Goal
Finish the execution-class contract so missing metadata cannot cause future
profile drift.
### Work
- Update every framework, CLI, and integration test module specification to
declare an explicit execution class appropriate to the fake behavior.
- Change module-spec validation so an empty or unsupported execution class is a
registration error. Remove the transitional deterministic default from
Stage 6.
- Replace the chunk runner's special `ChunkExecutionClassProvider` probe with
specification-derived behavior. Remove the now-redundant provider interface,
implementation methods, and tests when they have no remaining consumer.
- Ensure chunk producer provenance remains unchanged: it records a non-empty
effective binding profile for an LLM-backed chunker, while a deterministic
chunker records no profile. A profile selected only through the prompt
default remains represented by PromptKit's actual-profile manifest rather
than being invented as an explicit chunk binding.
- Review helper constructors and fixtures for opportunities to set execution
class once without obscuring the class under test. Do not introduce an
elaborate test-spec framework.
- Update internal documentation if the removal changes any described runtime
mechanics.
### Tests And Validation
- Add or retain focused registration tests for missing and invalid execution
classes.
- Retain chunk-plan provenance tests for LLM-backed and deterministic
chunkers.
- Run `go test ./internal/framework/pipeline ./internal/modules/...`.
- Run `go test ./...`, `go vet ./...`, and `git diff --check`.
### Completion Criteria
- No registered module specification relies on an implicit execution class.
- Pipeline metadata, not a concrete runtime type assertion, owns module
execution classification.
## Stage 8: Resolve Programmatic Pipeline Profile Defaults
### Goal
Implement profile inheritance and precedence inside the pipeline resolver
before exposing the field through YAML configuration.
### Work
- Add an optional trimmed `LLMProfile` field to
`pipeline.PipelineProfile`. Add a non-empty runtime override field to
`pipeline.ResolveOptions` so all precedence decisions occur in the resolver
rather than through pre-resolution mutation.
- After module selection, `--only` filtering, default validator-chain
selection, and validator compatibility resolution, apply effective profiles
to every selected input, chunk, extract, merge, normalize, output, and
validator binding according to the precedence in `promptkit.md`.
- Apply profiles only when the selected module or validator execution class is
`llm_backed`.
- Reject a binding-specific `llm_profile` on any deterministic module or
validator. Do not reject or inspect an unused pipeline default when no
selected LLM-backed binding consumes it.
- Leave an LLM-backed binding empty when no CLI, binding, or pipeline profile is
selected so PromptKit can use the prompt's `default_profile`.
- Store the effective values on resolved bindings before digest construction.
Do not add a second inheritance decision to execution.
- Ensure semantically equivalent repeated binding profiles and one inherited
default produce the same resolved pipeline digest. Ensure any changed
effective profile changes the digest.
- Do not modify file configuration or CLI parsing in this stage.
### Tests And Validation
- Add pipeline package tests for the complete precedence matrix:
runtime override; binding-specific exception; pipeline default; prompt
fallback; and deterministic bindings.
- Cover default and explicitly configured validator chains, all relevant stage
categories, `--only` lane selection, unused defaults, deterministic-profile
rejection, and semantic digest equivalence.
- Prefer table-driven package-level tests over assertions on private traversal
helpers.
- Run `go test ./internal/framework/pipeline` and `go test ./...`.
- Run `git diff --check`.
### Completion Criteria
- Programmatic pipelines resolve one canonical effective profile policy.
- Only LLM-backed resolved bindings can contain a profile.
- Runtime override, binding, pipeline, and prompt precedence is unambiguous and
digest-stable.
## Stage 9: Expose Pipeline Defaults Through Configuration And CLI
### Goal
Make the profile-default workflow available to operators while preserving
validation and override behavior.
### Work
- Add optional `pipelines.<id>.llm_profile` support to the version 4 file
configuration model. Use presence-aware decoding so an explicitly set blank
value is rejected, while omission remains valid.
- Preserve the field through file application, configuration cloning,
effective configuration, and programmatic profile copies without aliasing or
trimming drift.
- Remove `applyLLMProfileOverride`. Pass the CLI override through the resolver's
runtime-override input so deterministic bindings are never populated.
- Update effective profile-ID collection to cover every selected LLM-backed
module stage and LLM-backed validator, including future LLM-backed input and
output modules. Do not inspect deterministic or unselected profiles.
- Ensure `run`, `config validate --pipeline`, resume/checkpoint identity, and
relevant dry preflight paths all use the same resolved effective profiles.
- Preserve `--llm-profile` as the highest-precedence non-empty run-wide
override and preserve binding-specific profiles as exceptions when no CLI
override is present.
- Do not increment the configuration version.
- Update current configuration and CLI contracts in `docs/config.md` and
`docs/cli.md` in the same stage. Link to operations for the deployment
workflow rather than duplicating it prematurely.
### Tests And Validation
- Add file-config tests for omission, trimming, explicit blank rejection,
unknown-key behavior, cloning, and round-trip application.
- Add effective-config and CLI contract tests for precedence, LLM-only
application, inherited-profile inspection failure before factory execution,
`--only`, and digest changes.
- Retain offline operation and do not require credentials for
`config validate --pipeline`.
- Run `go test ./internal/core/config ./internal/framework/pipeline
./internal/cli`.
- Run `go test ./...`, `go vet ./...`, and `git diff --check`.
### Completion Criteria
- Operators can select `dnd-extraction` once per pipeline.
- Configuration and CLI paths share the resolver's precedence policy.
- Unknown effective profiles fail preflight, while deterministic and unused
profiles do not cause spurious inspection.
## Stage 10: Complete Operator Documentation, Examples, And Decision Record
### Goal
Make the implemented workflow understandable, copyable, and maintainable
without duplicating canonical facts.
### Work
- Create an ADR using the next sequential number for the durable decision to
use workload-oriented pipeline defaults with operator-overridable application
fallback profiles. Record context, decision, alternatives, and consequences;
do not turn the ADR into a field reference or implementation log.
- Complete `docs/config.md` as the canonical owner of profile-source fields,
`pipelines.<id>.llm_profile`, validation, and precedence.
- Complete `docs/operations.md` with an operator workflow that distinguishes
Notarius embedded prompts, Notarius fallback profiles, PromptKit built-ins,
and deployment filesystem profiles. Include production/development/local use
of the same `dnd-extraction` ID, credential handling, absolute-path guidance,
and the fact that current relative profile paths use the process working
directory rather than the configuration file's directory.
- Complete `docs/integrations/pkg-promptkit.md` with the v0.5.0 boundary,
prepared execution, inspection, fallback and ordinary source precedence,
optional provider controls, capacity adaptation, and compatibility policy.
- Update `docs/internal/configuration.md`, `docs/internal/pipeline.md`,
`docs/internal/cli.md`, `docs/internal/llm.md`, `docs/internal/modules.md`, and
`docs/internal/dnd.md` only for their owned implementation details. Link to
canonical configuration, operations, and upstream format contracts rather
than restating them.
- Keep exactly the existing two D&D configuration examples. Add
`llm_profile: dnd-extraction` to the minimal and complete pipelines and remove
the now-redundant model-named binding override from the complete example.
- Add one secret-free maintained operator profile at
`examples/profiles/dnd-extraction.yml`. It should be a complete valid profile
for the same logical ID and may mirror the embedded baseline; its purpose is
to demonstrate file ownership and format, not claim automatic environment
detection. Link it from the configuration and operations documentation.
- If the complete example selects the external profile file, use a path that
is valid for the documented repository-root invocation and explicitly note
the working-directory rule. Keep the minimal example dependent only on the
embedded fallback.
- Add or extend maintained-example validation so both configuration examples
and the profile YAML are checked without generation or credentials.
- Remove the now-implemented `Pipeline-Level LLM Profile Defaults` section from
`docs/roadmap/future.md`. Preserve the unrelated deterministic session and
concurrency items.
- Do not delete `promptkit.md` or this implementation plan during the feature
implementation; retire them only after post-implementation review.
### Tests And Validation
- Run maintained example/configuration tests and relevant CLI help/parser
tests.
- Run `go test ./...`.
- Run `rg -n 'gemini-2-flash' examples docs` and review every remaining match
for intentional model-policy or historical context.
- Run `rg -n 'v0\.3\.0|profileCheckPrompt|applyLLMProfileOverride' .` and resolve
stale production or current-documentation matches.
- Verify all new links and `git diff --check`.
### Completion Criteria
- Every current fact has one canonical documentation owner.
- Operators can distinguish and deploy all profile layers without reading Go
source.
- Both maintained configurations and the maintained external profile are valid,
secret-free, and tested offline.
- Implemented profile work no longer remains in `future.md`.
## Stage 11: Final Verification And Quality Review
### Goal
Verify the complete migration as one integrated change and correct only defects
or omissions found during that review.
### Work
- Review the final diff against every acceptance criterion in `promptkit.md`.
- Confirm provider-specific PromptKit types remain inside the LLM integration
boundary and D&D policy remains inside the D&D module family.
- Confirm execution and inspection receive identical ordinary, fallback, and
backend configuration.
- Confirm no paths, profile YAML, endpoints, credentials, or prepared handle
state leak into fingerprints or ordinary diagnostics.
- Confirm all production module specs have explicit correct execution classes
and every resolved deterministic binding is profile-free.
- Confirm prompt default, pipeline default, binding override, and CLI override
behavior through representative assembled configurations.
- Review tests for redundancy and remove obsolete synthetic-prompt,
runtime-probe, exact-hash, or duplicated upstream-behavior tests superseded by
stronger contract tests.
- Perform an optional manual D&D quality comparison if credentials and an
evaluation transcript are deliberately supplied. Record no private input or
credential material, and do not make this comparison a completion gate.
### Validation Commands
```sh
gofmt -w <changed-go-files>
go test ./...
go test -race ./internal/framework/llm ./internal/core/config ./internal/framework/pipeline ./internal/cli
go vet ./...
go build ./cmd/notarius
git diff --check
```
Also run focused stale-contract searches:
```sh
rg -n 'gitea.maximumdirect.net/eric/promptkit v0\.3\.0|PromptKit v0\.3\.0' .
rg -n 'default_profile: gemini-2-flash|profileCheckPrompt|applyLLMProfileOverride' internal docs examples
```
Review any matches rather than deleting intentional historical references
blindly.
### Completion Criteria
- All automated checks pass offline and without real credentials.
- The implemented behavior matches `promptkit.md` with no known architecture,
provenance, checkpoint, profile-precedence, or documentation gap.
- Any optional live evaluation is clearly separate from correctness testing.
## Open Questions
None. The roadmap decisions are sufficient to implement every stage without an
additional product or architecture choice.

318
docs/roadmap/promptkit.md Normal file
View File

@@ -0,0 +1,318 @@
# PromptKit v0.5 Integration And LLM Profile Policy
## Purpose
This roadmap defines the target state for upgrading Notarius from PromptKit
v0.3.0 to v0.5.0 and adopting the upstream runtime and profile facilities that
directly improve Notarius. It also defines the application policy for stable,
domain-oriented LLM profile names, operator overrides, pipeline inheritance,
profile validation, provider defaults, checkpoint identity, and documentation.
The ordered work needed to reach this state belongs in
[the implementation plan](implementation.md). Current behavior remains defined
by the canonical documentation outside `docs/roadmap/` until the corresponding
work is implemented.
## Background
Notarius currently pins PromptKit v0.3.0. Its adapter prepares a request once
for debug material and then independently runs the original request, causing
PromptKit to prepare the same logical call a second time. The CLI validates an
explicit profile by preparing a synthetic prompt. PromptKit profile selection
can be repeated on individual module bindings or replaced for one invocation
with `--llm-profile`, but a configured pipeline cannot yet declare one inherited
profile policy.
PromptKit v0.4.0 and v0.5.0 add the upstream boundaries needed to improve these
areas:
- [v0.4.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.4.0.md)
adds opaque prepared executions, exact profile and prompt inspection, and a
typed backend-capacity error;
- [v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.5.0.md)
adds application fallback profile filesystems and stops sending unset
optional sampling controls as framework-selected provider values; and
- the [v0.5.0 format contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
defines the resulting profile-source and execution-setting precedence.
A source-compatibility test of the current Notarius repository against
PromptKit v0.5.0 completed successfully. The work is therefore primarily an
intentional runtime and configuration migration rather than a repair for a
breaking Go API change.
## Goals
- Pin and document PromptKit v0.5.0 as Notarius's supported upstream contract.
- Execute the exact prepared request snapshot whose safe details are recorded
in Notarius debug material.
- Validate configured PromptKit profiles through the upstream inspection API
without synthetic prompts, provider calls, or credential-value access.
- Give Notarius an application-owned, operator-overridable
`dnd-extraction` profile fallback.
- Let a pipeline choose one default LLM profile without repeating that ID on
every LLM-backed binding.
- Apply profile inheritance and run-wide overrides only where the resolved
module or validator can use an LLM.
- Preserve accurate checkpoint invalidation, effective profile provenance,
redaction, cancellation, concurrency, and provider-neutral module contracts.
- Provide operators with one clear deployment pattern for production,
development, and local profile definitions.
## Target End State
### PromptKit Runtime Boundary
Notarius depends on PromptKit v0.5.0 and uses its public APIs rather than
reimplementing source or execution resolution.
For each structured completion, the adapter:
1. builds one PromptKit run request from the provider-neutral Notarius request;
2. calls `PrepareExecution` once;
3. immediately arranges an idempotent `Discard` for every unexecuted handle;
4. obtains credential-redacted `Details` for debug and response metadata; and
5. calls `RunPrepared` so generation uses that exact frozen snapshot.
The debug prompt and successful result therefore describe the same selected
profile, rendered messages, input bytes, session, output contract, and effective
settings even when a filesystem-backed source changes concurrently. PromptKit
handle types remain private to `internal/framework/llm`.
PromptKit admission failures continue to match Notarius's provider-neutral
`ErrLLMCapacityExceeded` contract. When PromptKit supplies a `CapacityError`,
the adapter obtains the normalized backend ID through `errors.As` and may add it
to safe application-owned diagnostics without parsing upstream error wording.
The backend ID does not become a provider-specific module contract.
### Optional Provider Controls
Notarius accepts PromptKit v0.5.0's new behavior for `temperature`,
`max_tokens`, and `top_p`: an unset setting is omitted from compatible provider
requests and the provider chooses its own default. Notarius does not restore
PromptKit's former implicit `top_p: 1` value globally.
An operator who requires a particular value specifies it in the selected
PromptKit profile. The application fallback described below intentionally
leaves these controls unset. A human-reviewed D&D extraction comparison should
be performed after the upgrade, but paid or nondeterministic model output is
not part of the default automated test suite.
### Profile Inspection
Pipeline-aware configuration validation uses `Engine.InspectProfile` for every
effective explicit profile ID. It verifies that the profile exists, parses and
validates, resolves its backend and target, and is compatible with the engine's
registered backends. It does not create a synthetic prompt, load prompt inputs,
contact a provider, or require credential values to exist in the validation
process environment.
Credential availability is execution-time state. PromptKit preparation still
enforces the selected profile's credential contract before generation. This
keeps `notarius config validate` useful in build and deployment validation
environments where secrets are deliberately absent.
PromptKit construction for inspection and execution uses one shared internal
profile-source and backend-option path. The CLI does not expose PromptKit public
types across the Notarius LLM boundary merely to perform inspection.
`InspectPrompt` is not adopted merely because it exists. It remains available
for a later, separately defined module-to-prompt interface preflight if a
concrete validation requirement justifies that additional contract.
### Application And Operator Profile Sources
Notarius embeds one ordinary PromptKit YAML profile with the stable ID
`dnd-extraction`. It is an application fallback registered through
`WithFallbackProfileFS`, is owned by the D&D module family, and initially
preserves the current effective D&D baseline:
- backend: PromptKit's built-in `openrouter` backend;
- model: `openai/gpt-5.6-luna`;
- reasoning effort: unset, allowing OpenAI's backend to apply its default of
`medium`;
- generation timeout: 240 seconds;
- service tier: `flex`; and
- no application-selected `temperature`, `max_tokens`, or `top_p`.
All maintained D&D LLM prompt definitions use `dnd-extraction` as their
`default_profile`. The ID communicates workload intent rather than a provider,
model, or environment. Changing the embedded fallback is an intentional
Notarius execution-policy change and participates in checkpoint identity.
Effective profile definitions resolve in PromptKit's order:
1. programmatic in-memory profiles used by tests or explicit consumers;
2. the operator source configured by `promptkit.profile_file` or
`promptkit.profile_dir`;
3. the Notarius application fallback source; and
4. PromptKit's embedded built-in catalog.
Only an absent ID falls through to the next source. A matching profile is a
complete definition: fields are not merged with a lower-precedence definition,
and a malformed matching operator profile fails rather than silently selecting
the application fallback.
Production, development, and local deployments should normally provide
different complete definitions for the same `dnd-extraction` ID. An operator
source is optional because the application fallback keeps the maintained D&D
workflow usable, but a deployment that needs an intentional model or backend
policy should configure its own definition.
### Domain Ownership And Asset Assembly
The D&D fallback profile remains under `internal/modules/dnd` and is registered
by the D&D registrar, consistent with ADR-0004. Generic LLM plumbing knows how
to collect and flatten application fallback profile filesystems but contains no
D&D model or policy knowledge.
The shared asset registry detects invalid roots, unreadable sources, and
duplicate flattened paths. PromptKit remains responsible for strict profile
YAML parsing, duplicate profile-ID detection, source precedence, and effective
target resolution. The same assembled fallback source is supplied to runtime
execution and CLI profile inspection.
### Explicit Module Execution Metadata
Every registered input, chunk, extract, merge, normalize, and output module
declares one required execution class: `deterministic` or `llm_backed`.
Validator registrations continue to declare the same distinction through their
validator specifications.
The registered specification is authoritative for configuration resolution.
Current production classifications are:
- the D&D scene chunker, all D&D extractors, and the D&D NPC normalizer are
LLM-backed;
- the Seriatim input adapter, generic chunker, all current mergers, all other
current normalizers, and the JSON output encoder are deterministic; and
- current validators retain their declared classifications.
Missing or unsupported execution metadata is a registration error. Explicitly
assigning `llm_profile` to a deterministic module or validator is a pipeline
resolution error. The framework does not infer execution class by inspecting
domain package names or concrete implementation types at runtime.
The module specification replaces the chunk runner's special runtime
execution-class probe. Effective resolved bindings already express the result:
only LLM-backed bindings may retain a non-empty profile.
### Pipeline-Level Profile Default
Configuration version 4 gains one optional non-empty pipeline field:
```yaml
pipelines:
dnd-session:
llm_profile: dnd-extraction
```
No configuration-version increment is required because the field is additive
and existing files remain valid. An explicitly present blank value is invalid.
For every selected LLM-backed module and validator, the effective profile uses
this precedence:
1. non-empty run-wide `--llm-profile` override;
2. binding-specific `llm_profile`;
3. pipeline-level `llm_profile`; and
4. the prompt definition's `default_profile`, represented by an empty effective
Notarius binding profile.
The run-wide override and inherited pipeline default never attach to a
deterministic binding. Binding-specific exceptions remain available when one
operation needs a different cost, latency, quality, backend, or reasoning
policy.
Inheritance is resolved after module and validator selection, including
`--only` lane filtering, but before effective-pipeline validation, digest
construction, explicit-profile inspection, checkpoint construction,
preparation, execution, or provenance capture. Only profiles used by selected
LLM-backed bindings are inspected. An unused pipeline default in a pipeline
with no selected LLM-backed work does not require an otherwise unused profile
to exist.
The resolved pipeline contains effective binding profiles rather than a second
runtime inheritance mechanism. Two pipelines that differ only by spelling the
same effective policy once as a pipeline default and once on every LLM-backed
binding have the same semantic resolved digest. Changing an effective profile
changes the digest and applicable checkpoint identity.
### Provenance And Checkpoints
The PromptKit profile-source checkpoint fingerprint covers:
- the PromptKit v0.5.0 built-in profile catalog identity;
- exact application fallback profile asset content; and
- exact configured operator profile YAML content, when present.
The existing local-backend target fingerprint remains separate and continues
to exclude scheduling-only concurrency limits. Fingerprints contain hashes and
stable markers, not profile contents, filesystem paths, endpoints, credentials,
or other secrets.
Changing the PromptKit version, application fallback, operator profile, or
effective pipeline profile makes incompatible LLM checkpoints ineligible for
reuse. The dependency upgrade is expected to invalidate checkpoints produced
under v0.3.0.
Successful run manifests continue to record only profiles actually selected by
PromptKit, including their effective model, backend, and reasoning metadata.
Debug output reports the same effective execution snapshot used for generation.
### Operator Documentation And Examples
Canonical documentation clearly distinguishes:
- Notarius prompt and schema assets embedded in the application;
- Notarius application fallback profiles embedded in the application;
- PromptKit's own embedded built-in profiles; and
- operator profile files on the deployment filesystem.
The configuration reference owns the pipeline field, profile-source fields,
validation rules, and precedence. Operations owns deployment layout, working
directory behavior, credentials, and environment-specific profile management.
The PromptKit integration document owns the pinned upstream contract and
source-precedence boundary. Internal documents describe asset registration,
resolution, inspection, prepared execution, fingerprinting, and tests without
duplicating user-facing field definitions.
The maintained examples continue to include only the minimal and complete D&D
configurations. They use the stable `dnd-extraction` policy, and one maintained
PromptKit profile file under `examples/` demonstrates an operator override.
Examples remain secret-free and are validated without live provider calls.
## Out Of Scope
- Implementing the separate deterministic prompt-session identity roadmap
item.
- Changing the default `concurrency.total_llm` value; PromptKit's retained
OpenRouter capacity of 16 remains relevant to that separate item.
- Adding model evaluation as a deterministic or CI correctness gate.
- Automatically selecting production, development, or local environments.
Deployment configuration chooses the operator profile source.
- Profile inheritance, partial profile merging, or cross-profile aliases.
- Exposing PromptKit types to modules, validators, durable output contracts, or
public configuration structures.
- Adopting `InspectPrompt` without a separately justified prompt-interface
validation contract.
## Acceptance Criteria
- Notarius builds and its offline test suite passes with PromptKit v0.5.0.
- Every structured completion executes the exact snapshot used for safe debug
prompt details.
- Profile preflight uses profile inspection and no synthetic prompt.
- The embedded `dnd-extraction` fallback resolves without an operator source,
and a matching valid operator profile replaces it completely.
- Every production module has explicit, correct execution metadata.
- Pipeline, binding, CLI, and prompt-default precedence behaves as defined for
modules and validators, while deterministic bindings remain profile-free.
- Effective profiles participate in pipeline digests, profile inspection,
checkpoint identity, debug records, and run provenance at the appropriate
boundaries.
- The dependency and application fallback changes invalidate incompatible old
checkpoints without exposing profile or credential content.
- Canonical documentation and maintained examples accurately describe and
exercise the implemented operator workflow.
- Default tests remain deterministic, offline, credential-free, and focused on
Notarius-owned behavior rather than duplicating PromptKit's upstream suite.

View File

@@ -1,4 +1,6 @@
version: 4
promptkit:
profile_file: ./examples/profiles/dnd-extraction.yml
concurrency:
total_llm: 2
stage_workers:
@@ -16,6 +18,7 @@ debug:
directory: ./notarius-debug
pipelines:
dnd-session:
llm_profile: dnd-extraction
input: seriatim
# Stable campaign context is shared by every module that accepts these slots.
references:
@@ -54,7 +57,6 @@ pipelines:
merge: appendorder
normalize:
module: dnd/npcs
llm_profile: gemini-2-flash
retries: 2
scene-descriptions:
extract:

View File

@@ -1,6 +1,7 @@
version: 4
pipelines:
dnd-session:
llm_profile: dnd-extraction
input: seriatim
artifacts:
spells:

View File

@@ -0,0 +1,5 @@
id: dnd-extraction
backend: openrouter
model: openai/gpt-5.6-luna
timeout_seconds: 240
service_tier: flex

2
go.mod
View File

@@ -3,7 +3,7 @@ module gitea.maximumdirect.net/eric/notarius
go 1.25.5
require (
gitea.maximumdirect.net/eric/promptkit v0.2.0
gitea.maximumdirect.net/eric/promptkit v0.5.0
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
gopkg.in/yaml.v3 v3.0.1
)

4
go.sum
View File

@@ -1,5 +1,5 @@
gitea.maximumdirect.net/eric/promptkit v0.2.0 h1:6bUATKnVgLlHSbKjGb8EnCt2r7jV5Xw6t+pX7G9S/lU=
gitea.maximumdirect.net/eric/promptkit v0.2.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
gitea.maximumdirect.net/eric/promptkit v0.5.0 h1:jnpazLyyNhWrB2xzwwtUkNUfktkTdkENTwuSPnKiYrc=
gitea.maximumdirect.net/eric/promptkit v0.5.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=

View File

@@ -213,11 +213,12 @@ func assembledSpellPipeline(t *testing.T, options assembledSpellPipelineOptions)
components := productionTestComponents(t)
extractor := &assembledSpellExtractor{unknownSpell: options.unknownSpell}
if err := pipeline.RegisterExtractor[dnd.SpellList](components.registries.Extractors, pipeline.ModuleSpec{
Key: assembledSpellExtractorKey,
Stage: pipeline.StageExtract,
Requires: []string{"chunks", "source.transcript"},
Provides: []string{"dnd.spell_casts"},
ArtifactKind: dnd.SpellListKind,
Key: assembledSpellExtractorKey,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunks", "source.transcript"},
Provides: []string{"dnd.spell_casts"},
ArtifactKind: dnd.SpellListKind,
}, func() (contracts.Extractor[dnd.SpellList], error) {
return extractor, nil
}); err != nil {

View File

@@ -143,17 +143,6 @@ func isEmptyRegistries(registries pipeline.Registries) bool {
registries.Outputs == nil
}
func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
if err := ctx.Err(); err != nil {
return nil, nil, err
}
assets, err := productionPromptAssets()
if err != nil {
return nil, nil, err
}
return buildProductionLLMClient(ctx, cfg, profileID, overrides, assets)
}
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
return func(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
return buildProductionLLMClient(ctx, cfg, profileID, overrides, assets)
@@ -171,6 +160,7 @@ func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID
client, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{
ProfileDir: cfg.PromptKit.ProfileDir,
ProfileFile: cfg.PromptKit.ProfileFile,
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
Assets: assets,
Recorder: recorder,
ReasoningEffort: overrides.ReasoningEffort,

View File

@@ -154,6 +154,23 @@ func TestConfigValidateResolvesPipelineAndChecksSelection(t *testing.T) {
}
}
func TestConfigValidatePipelineDefaultProfileIsOffline(t *testing.T) {
configPath := writeCommandConfigContent(t, `version: 4
pipelines:
demo:
llm_profile: dnd-extraction
input: seriatim
artifacts:
spells:
extract: dnd/spells
`)
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "demo"}, &stdout, &stderr, Options{})
if code != 0 || !strings.Contains(stdout.String(), "valid for pipeline \"demo\"") || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
}
func TestPipelinesListSortsNormalizedIDsInTextAndJSON(t *testing.T) {
configPath := writeCommandConfig(t, " zeta ", "alpha")
options := commandContractOptions(t)

View File

@@ -94,6 +94,37 @@ func TestMaintainedConfigurationExampleSet(t *testing.T) {
if got := strings.Join(names, ","); got != "dnd-complete.config.yml,dnd-minimal.config.yml" {
t.Fatalf("maintained configuration examples = %q, want only the minimal and complete D&D examples", got)
}
profileEntries, err := os.ReadDir(repositoryPath("examples", "profiles"))
if err != nil {
t.Fatal(err)
}
names = names[:0]
for _, entry := range profileEntries {
if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".yml") {
names = append(names, entry.Name())
}
}
sort.Strings(names)
if got := strings.Join(names, ","); got != "dnd-extraction.yml" {
t.Fatalf("maintained operator profiles = %q, want dnd-extraction.yml", got)
}
}
func TestMaintainedExamplesValidateEffectiveProfilesOffline(t *testing.T) {
t.Chdir(repositoryPath())
t.Setenv("OPENROUTER_API_KEY", "")
for _, example := range maintainedExampleFiles(t) {
t.Run(example.name, func(t *testing.T) {
var stdout, stderr strings.Builder
code := RunWithOptions([]string{
"config", "validate", "--config", example.path, "--pipeline", "dnd-session",
}, &stdout, &stderr, Options{})
if code != 0 || stderr.Len() != 0 || !strings.Contains(stdout.String(), `valid for pipeline "dnd-session"`) {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})
}
}
func exampleStepLaneIDs(resolved pipeline.ResolvedPipeline) []string {

View File

@@ -6,6 +6,8 @@ import (
"errors"
"fmt"
"io/fs"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"reflect"
@@ -13,7 +15,9 @@ import (
"sort"
"strings"
"sync"
"sync/atomic"
"testing"
"testing/fstest"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
@@ -34,6 +38,7 @@ import (
itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents"
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
"gitea.maximumdirect.net/eric/promptkit"
)
func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
@@ -161,6 +166,35 @@ func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
assertProductionContains(t, "production prompt assets", assetNames, requiredAssets)
catalog := catalogFromRegistries(registries)
for _, test := range []struct {
stage pipeline.ModuleStage
key string
want contracts.ExecutionClass
}{
{stage: pipeline.StageInput, key: "seriatim", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageChunk, key: "generic", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageChunk, key: "dnd/scenes", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageExtract, key: "dnd/spells", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageExtract, key: "dnd/npcs", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageExtract, key: "dnd/combat-turns", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageExtract, key: "dnd/item-events", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageExtract, key: "dnd/npc-interactions", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageExtract, key: "dnd/scene-descriptions", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageMerge, key: "appendorder", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageNormalize, key: "noop", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageNormalize, key: "dnd/spells", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageNormalize, key: "dnd/npcs", want: contracts.ExecutionClassLLMBacked},
{stage: pipeline.StageNormalize, key: "dnd/combat-turns", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageNormalize, key: "dnd/item-events", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageNormalize, key: "dnd/npc-interactions", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageNormalize, key: "dnd/scene-descriptions", want: contracts.ExecutionClassDeterministic},
{stage: pipeline.StageOutput, key: "json", want: contracts.ExecutionClassDeterministic},
} {
got, ok := catalog.ExecutionClass(test.stage, test.key)
if !ok || got != test.want {
t.Fatalf("production execution class for %s/%s = %q, %t; want %q, true", test.stage, test.key, got, ok, test.want)
}
}
converted := registriesFromCatalog(catalog)
if converted.ArtifactCodecs != registries.ArtifactCodecs || converted.ArtifactEvidence != registries.ArtifactEvidence || converted.ValidatorChains != registries.ValidatorChains {
t.Fatal("catalog/registry conversion did not preserve artifact and validator registries")
@@ -196,6 +230,76 @@ func TestDefaultCLICompositionValidatesRepresentativeConfiguration(t *testing.T)
}
}
func TestProductionAssetsResolveDNDExtractionProfile(t *testing.T) {
components := productionTestComponents(t)
newEngine := func(profileFile string) (*promptkit.Engine, error) {
t.Helper()
options, err := components.assets.PromptKitOptions()
if err != nil {
return nil, err
}
if profileFile != "" {
options = append(options, promptkit.WithProfileFile(profileFile))
}
return promptkit.NewEngine(promptkit.Config{}, options...)
}
t.Run("fallback", func(t *testing.T) {
engine, err := newEngine("")
if err != nil {
t.Fatal(err)
}
inspection, err := engine.InspectProfile(context.Background(), "dnd-extraction")
if err != nil {
t.Fatalf("InspectProfile() error = %v, want fallback profile", err)
}
params := inspection.EffectiveModelParams
if params.BackendID != "openrouter" || params.Model != "openai/gpt-5.6-luna" || params.TimeoutSeconds != 240 || params.ServiceTier != "flex" {
t.Fatalf("fallback profile parameters = %#v", params)
}
if params.ReasoningEffort != "" || params.Temperature != 0 || params.MaxTokens != 0 || params.TopP != 0 {
t.Fatalf("fallback profile selected optional provider controls: %#v", params)
}
})
t.Run("valid operator profile wins", func(t *testing.T) {
profilePath := filepath.Join(t.TempDir(), "profiles.yaml")
if err := os.WriteFile(profilePath, []byte(`id: dnd-extraction
endpoint: http://operator.example.test/v1
model: operator-model
timeout_seconds: 75
`), 0o600); err != nil {
t.Fatal(err)
}
engine, err := newEngine(profilePath)
if err != nil {
t.Fatal(err)
}
inspection, err := engine.InspectProfile(context.Background(), "dnd-extraction")
if err != nil {
t.Fatalf("InspectProfile() error = %v, want operator profile", err)
}
params := inspection.EffectiveModelParams
if params.BackendID != "" || params.Endpoint != "http://operator.example.test/v1" || params.Model != "operator-model" || params.TimeoutSeconds != 75 || params.ServiceTier != "" {
t.Fatalf("operator profile parameters = %#v, want complete replacement", params)
}
})
t.Run("invalid operator profile does not fall through", func(t *testing.T) {
profilePath := filepath.Join(t.TempDir(), "profiles.yaml")
if err := os.WriteFile(profilePath, []byte("id: dnd-extraction\nendpoint: http://operator.example.test/v1\nmodel: operator-model\nunknown: value\n"), 0o600); err != nil {
t.Fatal(err)
}
engine, err := newEngine(profilePath)
if err == nil {
_, err = engine.InspectProfile(context.Background(), "dnd-extraction")
}
if err == nil {
t.Fatal("operator profile error = nil, want failure instead of fallback")
}
})
}
func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) {
components := productionTestComponents(t)
cfg := config.Default()
@@ -370,39 +474,150 @@ func setNormalizeSpellCatalogSource(t *testing.T, resolved *pipeline.ResolvedPip
resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings = bindings
}
func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) {
func TestProductionLLMClientFactoryBuildsOfflineRuntime(t *testing.T) {
components := productionTestComponents(t)
factories := []struct {
name string
factory LLMClientFactory
}{
{name: "default production assets", factory: productionLLMClientFactory},
{name: "provided production assets", factory: productionLLMClientFactoryWithAssets(components.assets)},
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
if err != nil {
t.Fatalf("build production LLM runtime: %v", err)
}
for _, tt := range factories {
t.Run(tt.name, func(t *testing.T) {
client, manifests, err := tt.factory(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
if err != nil {
t.Fatalf("build production LLM runtime: %v", err)
}
if client == nil {
t.Fatal("production LLM runtime returned a nil client")
}
if len(manifests) != 0 {
t.Fatalf("eager profile manifests = %#v, want none", manifests)
}
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
if !ok {
t.Fatalf("production LLM client %T does not provide one profile-source checkpoint fingerprint", client)
}
fingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
if err != nil || len(fingerprints) != 1 {
t.Fatalf("production LLM checkpoint fingerprints = %#v, error = %v, want one profile-source identity", fingerprints, err)
}
if _, ok := client.(contracts.LLMProfileManifestProvider); !ok {
t.Fatalf("production LLM client %T does not provide profile manifests", client)
}
})
if client == nil {
t.Fatal("production LLM runtime returned a nil client")
}
if len(manifests) != 0 {
t.Fatalf("eager profile manifests = %#v, want none", manifests)
}
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
if !ok {
t.Fatalf("production LLM client %T does not provide one profile-source checkpoint fingerprint", client)
}
fingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
if err != nil || len(fingerprints) != 1 {
t.Fatalf("production LLM checkpoint fingerprints = %#v, error = %v, want one profile-source identity", fingerprints, err)
}
if _, ok := client.(contracts.LLMProfileManifestProvider); !ok {
t.Fatalf("production LLM client %T does not provide profile manifests", client)
}
}
func TestNormalizeOptionsSharesProductionProfileAssetsWithDefaultRuntime(t *testing.T) {
opts, err := normalizeOptions(Options{
Catalog: pipeline.ModuleCatalog{Inputs: pipeline.NewInputAdapterRegistry()},
})
if err != nil {
t.Fatal(err)
}
if opts.promptKitAssets == nil || opts.LLMClientFactory == nil {
t.Fatalf("normalized options = %#v, want shared profile assets and default runtime factory", opts)
}
if err := validateExplicitPromptKitProfiles(context.Background(), config.Default(), []string{"dnd-extraction"}, opts.promptKitAssets); err != nil {
t.Fatalf("inspect application fallback profile: %v", err)
}
client, _, err := opts.LLMClientFactory(context.Background(), config.Default(), "dnd-extraction", LLMRuntimeOverrides{})
if err != nil {
t.Fatalf("build default runtime: %v", err)
}
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
if !ok {
t.Fatalf("default runtime client %T does not provide checkpoint fingerprints", client)
}
runtimeFingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
directClient, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{Assets: opts.promptKitAssets})
if err != nil {
t.Fatal(err)
}
inspectionFingerprints, err := directClient.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(runtimeFingerprints, inspectionFingerprints) {
t.Fatalf("runtime profile fingerprints = %#v, inspection profile fingerprints = %#v", runtimeFingerprints, inspectionFingerprints)
}
}
func TestProductionLLMClientFactoryUsesConfiguredLocalBackend(t *testing.T) {
var providerCalls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
providerCalls.Add(1)
if r.URL.Path != "/v1/chat/completions" {
t.Errorf("provider path = %q, want /v1/chat/completions", r.URL.Path)
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{
"choices": [{"message": {"role": "assistant", "content": "{\"ok\":true}"}}],
"usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7}
}`))
}))
defer server.Close()
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
if err := os.WriteFile(profilePath, []byte(`id: local-profile
backend: local
model: local-model
`), 0o600); err != nil {
t.Fatal(err)
}
assets := llm.NewAssetRegistry()
if err := assets.RegisterPromptFS(fstest.MapFS{
"production.local.yaml": {Data: []byte(`id: production.local
version: "v1"
inputs:
- name: transcript
required: true
messages:
- role: user
content: '{{ input "transcript" }}'
output:
format: json
validation_mode: json
`)},
}, "."); err != nil {
t.Fatalf("register prompt assets: %v", err)
}
cfg := config.Default()
cfg.PromptKit.ProfileFile = profilePath
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
Endpoint: server.URL + "/v1",
ConcurrencyLimit: 2,
}
client, manifests, err := productionLLMClientFactoryWithAssets(assets)(
context.Background(),
cfg,
"local-profile",
LLMRuntimeOverrides{},
)
if err != nil {
t.Fatalf("build production LLM runtime: %v", err)
}
if len(manifests) != 0 {
t.Fatalf("eager profile manifests = %#v, want none", manifests)
}
var out map[string]any
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
PromptID: "production.local",
ProfileID: "local-profile",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "text/plain", []byte("local request"), "", ""),
},
}, &out)
if err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", err)
}
if providerCalls.Load() != 1 {
t.Fatalf("provider calls = %d, want 1", providerCalls.Load())
}
provider, ok := client.(contracts.LLMProfileManifestProvider)
if !ok {
t.Fatalf("production client %T does not provide profile manifests", client)
}
recorded := provider.LLMProfileManifests()
if len(recorded) != 1 || recorded[0].BackendID != promptkit.BackendLocal {
t.Fatalf("production profile manifests = %#v, want local backend", recorded)
}
}
@@ -410,7 +625,8 @@ func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
t.Run("canceled context", func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
client, manifests, err := productionLLMClientFactory(ctx, config.Default(), "test-profile", LLMRuntimeOverrides{})
components := productionTestComponents(t)
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(ctx, config.Default(), "test-profile", LLMRuntimeOverrides{})
if !errors.Is(err, context.Canceled) || client != nil || len(manifests) != 0 {
t.Fatalf("client=%T manifests=%#v error=%v, want canceled construction", client, manifests, err)
}
@@ -657,9 +873,10 @@ func productionCLIOptions(t *testing.T) Options {
func productionOptionsFromComponents(components productionComponents) Options {
return Options{
Catalog: catalogFromRegistries(components.registries),
Registries: components.registries,
LookupEnv: emptyLookup,
Catalog: catalogFromRegistries(components.registries),
Registries: components.registries,
LookupEnv: emptyLookup,
promptKitAssets: components.assets,
}
}

View File

@@ -2,67 +2,45 @@ package cli
import (
"context"
"errors"
"fmt"
"testing/fstest"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/promptkit"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
)
const profileCheckPromptID = "notarius.profile.check"
var profileCheckPromptFS = fstest.MapFS{
"prompts/profile-check.yaml": &fstest.MapFile{Data: []byte(`id: notarius.profile.check
version: "1.0.0"
default_profile: mistral-small-3
inputs:
- name: transcript
required: true
messages:
- role: user
content: "{{input \"transcript\"}}"
output:
format: text
validation_mode: none
repair_attempts: 0
`)},
}
func validateExplicitPromptKitProfiles(ctx context.Context, cfg config.Config, profileIDs []string) error {
func validateExplicitPromptKitProfiles(ctx context.Context, cfg config.Config, profileIDs []string, assets *llm.AssetRegistry) error {
if len(profileIDs) == 0 {
return nil
}
engine, err := newProfileValidationEngine(cfg)
inspector, err := llm.NewPromptKitProfileInspector(llm.PromptKitProfileInspectorConfig{
Source: promptKitProfileSourceConfig(cfg),
Assets: assets,
})
if err != nil {
return fmt.Errorf("load PromptKit profiles: %w", err)
}
for _, profileID := range profileIDs {
if _, err := engine.Prepare(ctx, promptkit.RunRequest{
PromptID: profileCheckPromptID,
ProfileID: profileID,
Inputs: map[string]promptkit.ArtifactRef{
"transcript": promptkit.Inline("profile check"),
},
}); err != nil {
if errors.Is(err, promptkit.ErrProfileNotFound) {
return fmt.Errorf("PromptKit profile %q is not configured", profileID)
}
return fmt.Errorf("validate PromptKit profile %q: %w", profileID, err)
if _, err := inspector.InspectProfile(ctx, profileID); err != nil {
return err
}
}
return nil
}
func newProfileValidationEngine(cfg config.Config) (*promptkit.Engine, error) {
opts := []promptkit.Option{
promptkit.WithPromptFS(profileCheckPromptFS, "prompts"),
func promptKitProfileSourceConfig(cfg config.Config) llm.PromptKitProfileSourceConfig {
return llm.PromptKitProfileSourceConfig{
ProfileDir: cfg.PromptKit.ProfileDir,
ProfileFile: cfg.PromptKit.ProfileFile,
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
}
}
func mapPromptKitLocalBackend(cfg *config.PromptKitLocalBackendConfig) *llm.PromptKitLocalBackendConfig {
if cfg == nil {
return nil
}
return &llm.PromptKitLocalBackendConfig{
Endpoint: cfg.Endpoint,
ConcurrencyLimit: cfg.ConcurrencyLimit,
}
if cfg.PromptKit.ProfileFile != "" {
opts = append(opts, promptkit.WithProfileFile(cfg.PromptKit.ProfileFile))
}
return promptkit.NewEngine(promptkit.Config{
PromptDir: "unused",
ProfileDir: cfg.PromptKit.ProfileDir,
}, opts...)
}

View File

@@ -0,0 +1,159 @@
package cli
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync/atomic"
"testing"
"testing/fstest"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
)
func TestExplicitPromptKitProfileValidationInspectsProfilesWithoutGeneration(t *testing.T) {
var providerCalls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
providerCalls.Add(1)
}))
defer server.Close()
writeProfile := func(t *testing.T, name, content string) string {
t.Helper()
profilePath := filepath.Join(t.TempDir(), name+".yaml")
if err := os.WriteFile(profilePath, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
return profilePath
}
localProfile := "id: local-profile\nbackend: local\nmodel: local-model\n"
credentialProfile := `id: credential-profile
endpoint: ` + server.URL + `/v1
model: credential-model
api_key_env: NOTARIUS_PROMPTKIT_PROFILE_INSPECTION_TEST_KEY
`
t.Setenv("NOTARIUS_PROMPTKIT_PROFILE_INSPECTION_TEST_KEY", "")
tests := []struct {
name string
profilePath string
profileID string
profileDir bool
localBackend bool
canceled bool
wantErr []string
rejectErr []string
}{
{
name: "configured local backend",
profilePath: writeProfile(t, "local-profile", localProfile),
profileID: "local-profile",
profileDir: true,
localBackend: true,
},
{
name: "missing local backend registration",
profilePath: writeProfile(t, "local-profile", localProfile),
profileID: "local-profile",
wantErr: []string{`PromptKit profile "local-profile" is invalid or unreadable`},
},
{
name: "absent profile",
profilePath: writeProfile(t, "local-profile", localProfile),
profileID: "absent-profile",
localBackend: true,
wantErr: []string{`PromptKit profile "absent-profile" is not configured`},
},
{
name: "malformed profile",
profilePath: writeProfile(t, "malformed-profile", "id: malformed-profile\nbackend: [\n"),
profileID: "malformed-profile",
wantErr: []string{`PromptKit profile "malformed-profile" is invalid or unreadable`},
rejectErr: []string{"malformed-profile.yaml", "backend: ["},
},
{
name: "invalid profile source",
profilePath: filepath.Join(t.TempDir(), "missing-profile.yaml"),
profileID: "missing-profile",
wantErr: []string{"load PromptKit profiles", "profile configuration is invalid or unreadable"},
},
{
name: "credential environment intentionally unset",
profilePath: writeProfile(t, "credential-profile", credentialProfile),
profileID: "credential-profile",
},
{
name: "canceled inspection",
profilePath: writeProfile(t, "local-profile", localProfile),
profileID: "local-profile",
localBackend: true,
canceled: true,
wantErr: []string{"context canceled"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := config.Default()
if tt.profileDir {
cfg.PromptKit.ProfileDir = filepath.Dir(tt.profilePath)
} else {
cfg.PromptKit.ProfileFile = tt.profilePath
}
if tt.localBackend {
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
Endpoint: server.URL + "/v1",
ConcurrencyLimit: 2,
}
}
ctx := context.Background()
if tt.canceled {
var cancel context.CancelFunc
ctx, cancel = context.WithCancel(ctx)
cancel()
}
err := validateExplicitPromptKitProfiles(ctx, cfg, []string{tt.profileID}, nil)
if len(tt.wantErr) == 0 {
if err != nil {
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
}
return
}
if err == nil {
t.Fatal("validateExplicitPromptKitProfiles() error = nil, want failure")
}
if tt.canceled && !errors.Is(err, context.Canceled) {
t.Fatalf("canceled inspection error = %v, want context canceled", err)
}
for _, want := range tt.wantErr {
if !strings.Contains(err.Error(), want) {
t.Fatalf("validation error = %q, want %q", err, want)
}
}
for _, rejected := range append(tt.rejectErr, tt.profilePath) {
if rejected != "" && strings.Contains(err.Error(), rejected) {
t.Fatalf("validation error = %q, must not expose %q", err, rejected)
}
}
})
}
if providerCalls.Load() != 0 {
t.Fatalf("provider calls during profile inspection = %d, want 0", providerCalls.Load())
}
}
func TestExplicitPromptKitProfileValidationUsesFallbackAssets(t *testing.T) {
assets := llm.NewAssetRegistry()
if err := assets.RegisterFallbackProfileFS(fstest.MapFS{
"profiles/fallback.yaml": {Data: []byte("id: fallback-profile\nendpoint: http://promptkit.test/v1\nmodel: fallback-model\n")},
}, "profiles"); err != nil {
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
}
if err := validateExplicitPromptKitProfiles(context.Background(), config.Default(), []string{"fallback-profile"}, assets); err != nil {
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
}
}

View File

@@ -116,7 +116,7 @@ func (h *recomputeTestHarness) options() Options {
for _, key := range []string{"test/extract/producer", "test/extract/unrelated", "test/extract/middle", "test/extract/dependent"} {
moduleKey := key
spec := pipeline.ModuleSpec{
Key: moduleKey, Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind,
Key: moduleKey, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind,
ReferenceSlots: []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}},
}
if err := pipeline.RegisterExtractor(opts.Registries.Extractors, spec, func() (contracts.Extractor[stateTestArtifact], error) {
@@ -125,7 +125,7 @@ func (h *recomputeTestHarness) options() Options {
panic(err)
}
}
if err := opts.Registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/recompute-output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
if err := opts.Registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/recompute-output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
return recomputeTestOutput{}, nil
}); err != nil {
panic(err)

View File

@@ -364,21 +364,21 @@ func referenceContractCatalog(t *testing.T, includeBetaMerger, includeBetaNormal
t.Fatal(err)
}
}
register(registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "reference/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }))
register(registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/chunk", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-slot"}, {Name: "required-chunk", Required: true}}}, func() (contracts.Chunker, error) { return stateTestChunker{}, nil }))
register(registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "reference/input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }))
register(registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-slot"}, {Name: "required-chunk", Required: true}}}, func() (contracts.Chunker, error) { return stateTestChunker{}, nil }))
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecA{}))
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecB{}))
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-alpha", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-beta", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-alpha", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-beta", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
if includeBetaMerger {
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
}
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
if includeBetaNormalizer {
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
}
register(registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { return stateTestOutput{}, nil }))
register(registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { return stateTestOutput{}, nil }))
return catalogFromRegistries(registries)
}

View File

@@ -46,6 +46,7 @@ type Options struct {
ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory
DebugRecorderFactory func(string) (pipeline.DebugRecorder, error)
DebugTerminalFactory func(*debugbundle.SummaryWriter) DebugTerminalWriter
promptKitAssets *frameworkllm.AssetRegistry
}
type LLMRuntimeOverrides struct {
@@ -121,12 +122,17 @@ func normalizeOptions(opts Options) (Options, error) {
}
opts.Registries = components.registries
opts.Catalog = catalogFromRegistries(components.registries)
if opts.LLMClientFactory == nil {
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(components.assets)
}
opts.promptKitAssets = components.assets
}
if opts.LLMClientFactory == nil {
opts.LLMClientFactory = productionLLMClientFactory
if opts.promptKitAssets == nil {
assets, err := productionPromptAssets()
if err != nil {
return Options{}, err
}
opts.promptKitAssets = assets
}
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(opts.promptKitAssets)
}
return opts, nil
}
@@ -340,7 +346,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
return failPipelineCommand(stderr, commandState, terminalWriter, err)
}
profileIDs := effectiveLLMProfileIDs(effective.ResolvedPipeline)
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, profileIDs); err != nil {
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, profileIDs, opts.promptKitAssets); err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, err)
}
workingDir, err := os.Getwd()
@@ -952,11 +958,22 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
seen[id] = struct{}{}
}
}
add(resolved.Chunk)
if resolved.InputExecutionClass == contracts.ExecutionClassLLMBacked {
add(resolved.Input)
}
if resolved.ChunkExecutionClass == contracts.ExecutionClassLLMBacked {
add(resolved.Chunk)
}
for _, lane := range resolved.AllArtifactLanes() {
add(lane.Extract)
add(lane.Merge)
add(lane.Normalize)
if lane.ExtractExecutionClass == contracts.ExecutionClassLLMBacked {
add(lane.Extract)
}
if lane.MergeExecutionClass == contracts.ExecutionClassLLMBacked {
add(lane.Merge)
}
if lane.NormalizeExecutionClass == contracts.ExecutionClassLLMBacked {
add(lane.Normalize)
}
}
for _, chain := range resolved.ValidatorChains {
for _, validator := range chain.Validators {
@@ -965,6 +982,9 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
}
}
}
if resolved.OutputExecutionClass == contracts.ExecutionClassLLMBacked {
add(resolved.Output)
}
ids := make([]string, 0, len(seen))
for id := range seen {
ids = append(ids, id)
@@ -1048,7 +1068,7 @@ func runConfigValidate(args []string, stdout, stderr io.Writer, opts Options) in
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline)); err != nil {
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline), opts.promptKitAssets); err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}

View File

@@ -276,7 +276,7 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
}
})
t.Run("validator profile remains distinct", func(t *testing.T) {
t.Run("runtime override applies to validators", func(t *testing.T) {
roots := newStateTestRoots(t)
profileDir := writeRunContractProfiles(t, "override-profile", "validator-profile")
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
@@ -294,11 +294,11 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
if code != 0 || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
if len(factoryProfiles) != 1 || factoryProfiles[0] != "" {
t.Fatalf("factory profiles = %#v, want one call without a unique profile", factoryProfiles)
if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" {
t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles)
}
if len(validatorProfiles) != 1 || validatorProfiles[0] != "validator-profile" {
t.Fatalf("validator profiles = %#v, want configured validator profile", validatorProfiles)
if len(validatorProfiles) != 1 || validatorProfiles[0] != "override-profile" {
t.Fatalf("validator profiles = %#v, want runtime override", validatorProfiles)
}
})
@@ -318,6 +318,24 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
}
})
t.Run("pipeline default is rejected before factory access", func(t *testing.T) {
roots := newStateTestRoots(t)
profileDir := writeRunContractProfiles(t, "configured-profile")
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
replaceStateTestConfigLine(t, roots.config, " sample:\n", " sample:\n llm_profile: missing-profile\n")
factoryCalls := 0
opts := newStateTestHarness().options()
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
factoryCalls++
return nil, nil, nil
}
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, &stdout, &stderr, opts)
if code != 1 || !strings.Contains(stderr.String(), "not configured") || factoryCalls != 0 || stdout.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
}
})
}
func TestRunReasoningEffortOverrideReachesFactory(t *testing.T) {
@@ -445,24 +463,30 @@ func TestReasoningEffortOverrideSeparatesCheckpointIdentities(t *testing.T) {
func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
resolved := pipeline.ResolvedPipeline{
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "},
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
InputExecutionClass: contracts.ExecutionClassLLMBacked,
Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "},
ChunkExecutionClass: contracts.ExecutionClassLLMBacked,
Steps: []pipeline.ResolvedPipelineStep{{
ID: "default",
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
Extract: pipeline.ModuleBinding{LLMProfile: "alpha"},
Merge: pipeline.ModuleBinding{LLMProfile: "zeta"},
Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "},
Extract: pipeline.ModuleBinding{LLMProfile: "alpha"},
ExtractExecutionClass: contracts.ExecutionClassLLMBacked,
Merge: pipeline.ModuleBinding{LLMProfile: "deterministic-merge"},
MergeExecutionClass: contracts.ExecutionClassDeterministic,
Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "},
NormalizeExecutionClass: contracts.ExecutionClassLLMBacked,
}},
}},
ValidatorChains: []pipeline.ResolvedValidatorChain{{Validators: []pipeline.ResolvedValidator{
{Binding: pipeline.ModuleBinding{LLMProfile: "deterministic-profile"}, ExecutionClass: contracts.ExecutionClassDeterministic},
{Binding: pipeline.ModuleBinding{LLMProfile: "beta"}, ExecutionClass: contracts.ExecutionClassLLMBacked},
}}},
Output: pipeline.ModuleBinding{LLMProfile: "output-profile"},
Output: pipeline.ModuleBinding{LLMProfile: "output-profile"},
OutputExecutionClass: contracts.ExecutionClassLLMBacked,
}
got := effectiveLLMProfileIDs(resolved)
want := []string{"alpha", "beta", "gamma", "zeta"}
want := []string{"alpha", "beta", "gamma", "input-profile", "output-profile", "zeta"}
if strings.Join(got, ",") != strings.Join(want, ",") {
t.Fatalf("effective profiles = %#v, want %#v", got, want)
}
@@ -526,7 +550,7 @@ func TestRunFactoryAndPreparationFailuresAreProcessFailures(t *testing.T) {
t.Fatal(err)
}
opts := newStateTestHarness().options()
if err := pipeline.RegisterExtractorBuilder(opts.Registries.Extractors, pipeline.ModuleSpec{Key: "test/failing-extract", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Extractor[stateTestArtifact], error) {
if err := pipeline.RegisterExtractorBuilder(opts.Registries.Extractors, pipeline.ModuleSpec{Key: "test/failing-extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Extractor[stateTestArtifact], error) {
return nil, errors.New("injected extractor construction failure")
}); err != nil {
t.Fatal(err)

View File

@@ -823,22 +823,22 @@ func (h *stateTestHarness) options() Options {
if err := pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, stateTestCodec{}); err != nil {
panic(err)
}
if err := registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }); err != nil {
if err := registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }); err != nil {
panic(err)
}
if err := registries.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/chunk", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "cache-reference"}}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Chunker, error) { return stateTestChunker{h}, nil }); err != nil {
if err := registries.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "cache-reference"}}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Chunker, error) { return stateTestChunker{h}, nil }); err != nil {
panic(err)
}
if err := pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "test/extract", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{h}, nil }); err != nil {
if err := pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "test/extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{h}, nil }); err != nil {
panic(err)
}
if err := pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "test/merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{harness: h}, nil }); err != nil {
if err := pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "test/merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{harness: h}, nil }); err != nil {
panic(err)
}
if err := pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "test/normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{harness: h}, nil }); err != nil {
if err := pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "test/normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{harness: h}, nil }); err != nil {
panic(err)
}
if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
return stateTestOutput{harness: h, includeWarnings: h.includeWarnings}, nil
}); err != nil {
panic(err)

View File

@@ -16,8 +16,14 @@ type Config struct {
}
type PromptKitConfig struct {
ProfileDir string `json:"profile_dir,omitempty"`
ProfileFile string `json:"profile_file,omitempty"`
ProfileDir string `json:"profile_dir,omitempty"`
ProfileFile string `json:"profile_file,omitempty"`
LocalBackend *PromptKitLocalBackendConfig `json:"local_backend,omitempty"`
}
type PromptKitLocalBackendConfig struct {
Endpoint string `json:"endpoint"`
ConcurrencyLimit int `json:"concurrency_limit"`
}
type ConcurrencyConfig struct {
@@ -66,6 +72,10 @@ func Default() Config {
func cloneConfig(in Config) Config {
out := in
if in.PromptKit.LocalBackend != nil {
localBackend := *in.PromptKit.LocalBackend
out.PromptKit.LocalBackend = &localBackend
}
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
for key, profile := range in.Pipelines {

View File

@@ -42,12 +42,10 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
}
profile = clonePipelineProfile(profile)
profile.ID = pipelineID
if override := strings.TrimSpace(input.LLMProfileOverride); override != "" {
applyLLMProfileOverride(&profile, override)
}
resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{
Only: input.Only,
LLMProfileOverride: input.LLMProfileOverride,
ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...),
ReferenceUnbinds: append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...),
}, input.Catalog)
@@ -65,22 +63,6 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
}, nil
}
func applyLLMProfileOverride(profile *pipeline.PipelineProfile, profileID string) {
profile.Chunk.LLMProfile = profileID
apply := func(artifacts map[string]pipeline.ArtifactLaneProfile) {
for laneID, lane := range artifacts {
lane.Extract.LLMProfile = profileID
lane.Merge.LLMProfile = profileID
lane.Normalize.LLMProfile = profileID
artifacts[laneID] = lane
}
}
apply(profile.Artifacts)
for index := range profile.Steps {
apply(profile.Steps[index].Artifacts)
}
}
func lookupPipelineProfile(profiles map[string]pipeline.PipelineProfile, pipelineID string) (pipeline.PipelineProfile, bool) {
pipelineID = strings.TrimSpace(pipelineID)
for rawID, profile := range profiles {

View File

@@ -107,6 +107,33 @@ func TestEffectiveConfigPreservesPromptKitProfileSource(t *testing.T) {
}
}
func TestEffectiveConfigOwnsPromptKitLocalBackend(t *testing.T) {
cfg := configForEffectiveTests(t, effectiveProfile())
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
Endpoint: "http://localhost:8000/v1",
ConcurrencyLimit: 2,
}
effective, err := cfg.Resolve(ResolveInput{PipelineID: "main", Catalog: effectiveCatalog(t)})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if effective.Config.PromptKit.LocalBackend == nil {
t.Fatal("effective local backend = nil")
}
if effective.Config.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend {
t.Fatal("effective local backend aliases input config")
}
cfg.PromptKit.LocalBackend.Endpoint = "http://changed-input.example/v1"
if effective.Config.PromptKit.LocalBackend.Endpoint != "http://localhost:8000/v1" {
t.Fatalf("input mutation changed effective config: %#v", effective.Config.PromptKit.LocalBackend)
}
effective.Config.PromptKit.LocalBackend.ConcurrencyLimit = 9
if cfg.PromptKit.LocalBackend.ConcurrencyLimit != 2 {
t.Fatalf("effective mutation changed input config: %#v", cfg.PromptKit.LocalBackend)
}
}
func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
tests := []struct {
name string
@@ -177,7 +204,7 @@ func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
}
}
func TestEffectiveConfigLLMProfileOverrideChangesDigestWithoutOverridingValidators(t *testing.T) {
func TestEffectiveConfigLLMProfileOverrideChangesDigestAndOverridesValidators(t *testing.T) {
profile := effectiveProfile()
profile.Chunk.LLMProfile = "chunk-profile"
lane := profile.Artifacts["lane"]
@@ -210,8 +237,27 @@ func TestEffectiveConfigLLMProfileOverrideChangesDigestWithoutOverridingValidato
t.Fatalf("pipeline profile override was not applied: %#v", resolved)
}
validators := findEffectiveValidatorChain(resolved, pipeline.StageExtract, "lane")
if len(validators.Validators) != 1 || validators.Validators[0].Binding.LLMProfile != "validator-profile" {
t.Fatalf("validator profile was overridden: %#v", validators)
if len(validators.Validators) != 1 || validators.Validators[0].Binding.LLMProfile != "override-profile" {
t.Fatalf("validator profile = %#v, want runtime override", validators)
}
}
func TestEffectiveConfigPipelineLLMProfileIsInheritedWithoutMutatingConfig(t *testing.T) {
profile := effectiveProfile()
profile.LLMProfile = " configured-profile "
effective, err := resolveEffectiveProfile(t, profile, ResolveInput{})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if got := effective.Config.Pipelines["main"].LLMProfile; got != " configured-profile " {
t.Fatalf("effective config pipeline llm profile = %q, want preserved programmatic value", got)
}
resolved := effective.ResolvedPipeline
if got := resolved.Chunk.LLMProfile; got != "configured-profile" {
t.Fatalf("resolved chunk profile = %q, want inherited profile", got)
}
if got := resolved.Steps[0].ArtifactLanes[0].Extract.LLMProfile; got != "configured-profile" {
t.Fatalf("resolved extract profile = %q, want inherited profile", got)
}
}
@@ -458,7 +504,7 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
if err := pipeline.RegisterArtifactCodec(catalog.ArtifactCodecs, effectiveCodec{}); err != nil {
t.Fatal(err)
}
if err := catalog.Inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
if err := catalog.Inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
return effectiveInput{key: "input"}, nil
}); err != nil {
t.Fatal(err)
@@ -466,6 +512,7 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
chunkSpec := pipeline.ModuleSpec{
Key: pipeline.DefaultChunkModule,
Stage: pipeline.StageChunk,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: []string{"source"},
Provides: []string{"chunk"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-ref"}},
@@ -476,32 +523,32 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
}); err != nil {
t.Fatal(err)
}
if err := catalog.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "needs-capability", Stage: pipeline.StageChunk, Requires: []string{"missing"}}, chunkOptions, func(pipeline.BuildRequest) (contracts.Chunker, error) {
if err := catalog.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "needs-capability", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}}, chunkOptions, func(pipeline.BuildRequest) (contracts.Chunker, error) {
return effectiveChunker{key: "needs-capability"}, nil
}); err != nil {
t.Fatal(err)
}
if err := pipeline.RegisterExtractor(catalog.Extractors, pipeline.ModuleSpec{Key: "extract", Stage: pipeline.StageExtract, ArtifactKind: effectiveArtifactKind, Requires: []string{"chunk"}, Provides: []string{"candidate"}}, func() (contracts.Extractor[effectiveArtifact], error) {
if err := pipeline.RegisterExtractor(catalog.Extractors, pipeline.ModuleSpec{Key: "extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"chunk"}, Provides: []string{"candidate"}}, func() (contracts.Extractor[effectiveArtifact], error) {
return effectiveExtractor{key: "extract"}, nil
}); err != nil {
t.Fatal(err)
}
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: pipeline.DefaultMergeModule, Stage: pipeline.StageMerge, ArtifactKind: effectiveArtifactKind, Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: pipeline.DefaultMergeModule, Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
return effectiveMerger{key: pipeline.DefaultMergeModule}, nil
}); err != nil {
t.Fatal(err)
}
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: "other-merge", Stage: pipeline.StageMerge, ArtifactKind: "other-kind", Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: "other-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "other-kind", Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
return effectiveMerger{key: "other-merge"}, nil
}); err != nil {
t.Fatal(err)
}
if err := pipeline.RegisterNormalizer(catalog.Normalizers, pipeline.ModuleSpec{Key: pipeline.DefaultNormalizeModule, Stage: pipeline.StageNormalize, ArtifactKind: effectiveArtifactKind, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer[effectiveArtifact], error) {
if err := pipeline.RegisterNormalizer(catalog.Normalizers, pipeline.ModuleSpec{Key: pipeline.DefaultNormalizeModule, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer[effectiveArtifact], error) {
return effectiveNormalizer{key: pipeline.DefaultNormalizeModule}, nil
}); err != nil {
t.Fatal(err)
}
if err := catalog.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: pipeline.DefaultOutputModule, Stage: pipeline.StageOutput, Requires: []string{"normalized"}}, func() (contracts.OutputEncoder, error) {
if err := catalog.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: pipeline.DefaultOutputModule, Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}}, func() (contracts.OutputEncoder, error) {
return effectiveOutput{key: pipeline.DefaultOutputModule}, nil
}); err != nil {
t.Fatal(err)

View File

@@ -23,26 +23,34 @@ type FileConfig struct {
}
type FilePromptKitConfig struct {
ProfileDir *string `yaml:"profile_dir,omitempty"`
ProfileFile *string `yaml:"profile_file,omitempty"`
ProfileDir *string `yaml:"profile_dir,omitempty"`
ProfileFile *string `yaml:"profile_file,omitempty"`
LocalBackend *FilePromptKitLocalBackendConfig `yaml:"local_backend,omitempty"`
}
type FilePromptKitLocalBackendConfig struct {
Endpoint *string `yaml:"endpoint,omitempty"`
ConcurrencyLimit *int `yaml:"concurrency_limit,omitempty"`
}
type FilePipelineProfile struct {
Input fileModuleBinding `yaml:"input"`
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
Steps []FilePipelineStepProfile `yaml:"steps,omitempty"`
Output *fileModuleBinding `yaml:"output,omitempty"`
References map[string]fileReferenceSource `yaml:"references,omitempty"`
artifactsSet bool `yaml:"-"`
stepsSet bool `yaml:"-"`
LLMProfile *string `yaml:"llm_profile,omitempty"`
Input fileModuleBinding `yaml:"input"`
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
Steps []FilePipelineStepProfile `yaml:"steps,omitempty"`
Output *fileModuleBinding `yaml:"output,omitempty"`
References map[string]fileReferenceSource `yaml:"references,omitempty"`
artifactsSet bool `yaml:"-"`
stepsSet bool `yaml:"-"`
llmProfileSet bool `yaml:"-"`
}
func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
type plainFilePipelineProfile FilePipelineProfile
var decoded plainFilePipelineProfile
seen, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
"input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
"llm_profile": {}, "input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
}, "pipeline profile")
if err != nil {
return err
@@ -50,6 +58,7 @@ func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
*p = FilePipelineProfile(decoded)
_, p.artifactsSet = seen["artifacts"]
_, p.stepsSet = seen["steps"]
_, p.llmProfileSet = seen["llm_profile"]
return nil
}
@@ -251,6 +260,9 @@ func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
return err
}
b.LLMProfile = strings.TrimSpace(llmProfile)
if b.LLMProfile == "" {
return fmt.Errorf("llm_profile must not be empty when set")
}
case "retries":
var retries int
if err := valueNode.Decode(&retries); err != nil {
@@ -468,10 +480,31 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
}
c.PromptKit.ProfileFile = value
}
if fileCfg.PromptKit.LocalBackend != nil {
if fileCfg.PromptKit.LocalBackend.Endpoint == nil {
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
}
endpoint := strings.TrimSpace(*fileCfg.PromptKit.LocalBackend.Endpoint)
if endpoint == "" {
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
}
localBackend := PromptKitLocalBackendConfig{Endpoint: endpoint}
if fileCfg.PromptKit.LocalBackend.ConcurrencyLimit != nil {
localBackend.ConcurrencyLimit = *fileCfg.PromptKit.LocalBackend.ConcurrencyLimit
}
c.PromptKit.LocalBackend = &localBackend
}
}
for _, pipelineID := range pipelineIDs {
filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]]
llmProfile := ""
if filePipeline.llmProfileSet || filePipeline.LLMProfile != nil {
if filePipeline.LLMProfile == nil || strings.TrimSpace(*filePipeline.LLMProfile) == "" {
return fmt.Errorf("pipeline %q llm_profile must not be empty when set", pipelineID)
}
llmProfile = strings.TrimSpace(*filePipeline.LLMProfile)
}
hasSteps := filePipeline.stepsSet || filePipeline.Steps != nil
laneIDs, rawLaneIDs, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID))
if err != nil {
@@ -479,6 +512,7 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
}
profile := pipeline.PipelineProfile{
ID: pipelineID,
LLMProfile: llmProfile,
Input: filePipeline.Input.toPipelineBinding(),
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
References: fileReferenceSourcesToPipeline(filePipeline.References),

View File

@@ -2,6 +2,7 @@ package config
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
@@ -49,6 +50,93 @@ func TestFileConfigMinimalVersion4AppliesOverDefaults(t *testing.T) {
}
}
func TestFilePipelineLLMProfileIsPresenceAwareAndDetached(t *testing.T) {
const pipelineYAML = `version: 4
pipelines:
main:
%s
input: input
artifacts:
lane:
extract: extract
`
t.Run("omitted", func(t *testing.T) {
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, ""))
if file.Pipelines["main"].LLMProfile != nil || file.Pipelines["main"].llmProfileSet {
t.Fatalf("parsed pipeline profile = %#v, want omitted llm profile", file.Pipelines["main"])
}
cfg := Default()
if err := cfg.ApplyFileConfig(file); err != nil {
t.Fatal(err)
}
if got := cfg.Pipelines["main"].LLMProfile; got != "" {
t.Fatalf("pipeline llm profile = %q, want empty", got)
}
})
t.Run("trimmed and detached", func(t *testing.T) {
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, "llm_profile: ' configured-profile '"))
cfg := Default()
if err := cfg.ApplyFileConfig(file); err != nil {
t.Fatal(err)
}
if got := cfg.Pipelines["main"].LLMProfile; got != "configured-profile" {
t.Fatalf("pipeline llm profile = %q, want trimmed value", got)
}
*file.Pipelines["main"].LLMProfile = "changed-profile"
if got := cfg.Pipelines["main"].LLMProfile; got != "configured-profile" {
t.Fatalf("effective config aliases parsed file: %q", got)
}
if got := cloneConfig(cfg).Pipelines["main"].LLMProfile; got != "configured-profile" {
t.Fatalf("cloned pipeline llm profile = %q", got)
}
data, err := json.Marshal(cfg)
if err != nil {
t.Fatal(err)
}
var roundTripped Config
if err := json.Unmarshal(data, &roundTripped); err != nil {
t.Fatal(err)
}
if got := roundTripped.Pipelines["main"].LLMProfile; got != "configured-profile" {
t.Fatalf("round-tripped pipeline llm profile = %q", got)
}
})
for _, value := range []string{"''", "' '", "null"} {
t.Run("explicit empty "+value, func(t *testing.T) {
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, "llm_profile: "+value))
cfg := Default()
err := cfg.ApplyFileConfig(file)
if err == nil || !strings.Contains(err.Error(), `pipeline "main" llm_profile must not be empty`) {
t.Fatalf("ApplyFileConfig() error = %v, want explicit-empty rejection", err)
}
})
}
}
func TestFileModuleBindingRejectsExplicitEmptyLLMProfile(t *testing.T) {
const configYAML = `version: 4
pipelines:
main:
input:
module: input
llm_profile: %s
artifacts:
lane:
extract: extract
`
for _, value := range []string{"''", "' '", "null"} {
t.Run(value, func(t *testing.T) {
_, err := ParseFileConfigYAML([]byte(fmt.Sprintf(configYAML, value)))
if err == nil || !strings.Contains(err.Error(), "llm_profile must not be empty when set") {
t.Fatalf("ParseFileConfigYAML() error = %v, want explicit-empty binding profile rejection", err)
}
})
}
}
func TestFilePromptKitProfileSourcesSurviveConfigBoundaries(t *testing.T) {
tests := []struct {
name string
@@ -97,6 +185,100 @@ func TestFilePromptKitProfileSourcesSurviveConfigBoundaries(t *testing.T) {
}
}
func TestFilePromptKitLocalBackendSurvivesConfigBoundaries(t *testing.T) {
tests := []struct {
name string
concurrencyYAML string
wantConcurrency int
}{
{name: "omitted concurrency defaults to zero"},
{name: "positive concurrency is preserved", concurrencyYAML: " concurrency_limit: 2\n", wantConcurrency: 2},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
file := parseFileConfig(t, "version: 4\npromptkit:\n local_backend:\n endpoint: ' http://localhost:8000/v1 '\n"+tt.concurrencyYAML)
cfg := Default()
if err := cfg.ApplyFileConfig(file); err != nil {
t.Fatalf("ApplyFileConfig() error = %v", err)
}
want := PromptKitLocalBackendConfig{
Endpoint: "http://localhost:8000/v1",
ConcurrencyLimit: tt.wantConcurrency,
}
if cfg.PromptKit.LocalBackend == nil || *cfg.PromptKit.LocalBackend != want {
t.Fatalf("local backend config = %#v, want %#v", cfg.PromptKit.LocalBackend, want)
}
*file.PromptKit.LocalBackend.Endpoint = "http://changed.example/v1"
if file.PromptKit.LocalBackend.ConcurrencyLimit != nil {
*file.PromptKit.LocalBackend.ConcurrencyLimit = 99
}
if *cfg.PromptKit.LocalBackend != want {
t.Fatalf("effective config aliases parsed file model: %#v", cfg.PromptKit.LocalBackend)
}
cloned := cloneConfig(cfg)
if cloned.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *cloned.PromptKit.LocalBackend != want {
t.Fatalf("cloned local backend = %#v, want detached %#v", cloned.PromptKit.LocalBackend, want)
}
cloned.PromptKit.LocalBackend.Endpoint = "http://clone.example/v1"
if *cfg.PromptKit.LocalBackend != want {
t.Fatalf("mutating clone changed source config: %#v", cfg.PromptKit.LocalBackend)
}
redacted := cfg.Redacted()
if redacted.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *redacted.PromptKit.LocalBackend != want {
t.Fatalf("redacted local backend = %#v, want detached %#v", redacted.PromptKit.LocalBackend, want)
}
data, err := json.Marshal(cfg)
if err != nil {
t.Fatalf("json.Marshal() error = %v", err)
}
var payload struct {
PromptKit map[string]json.RawMessage `json:"promptkit"`
}
if err := json.Unmarshal(data, &payload); err != nil {
t.Fatalf("json.Unmarshal() error = %v", err)
}
localJSON, ok := payload.PromptKit["local_backend"]
if !ok {
t.Fatalf("runtime PromptKit JSON keys = %v, want local_backend", payload.PromptKit)
}
var localPayload map[string]json.RawMessage
if err := json.Unmarshal(localJSON, &localPayload); err != nil {
t.Fatalf("unmarshal local_backend JSON: %v", err)
}
if _, ok := localPayload["endpoint"]; !ok {
t.Fatalf("runtime local_backend JSON keys = %v, want endpoint", localPayload)
}
if _, ok := localPayload["concurrency_limit"]; !ok {
t.Fatalf("runtime local_backend JSON keys = %v, want concurrency_limit", localPayload)
}
})
}
}
func TestFilePromptKitLocalBackendRequiresEndpoint(t *testing.T) {
for _, tt := range []struct {
name string
yaml string
}{
{name: "missing", yaml: "version: 4\npromptkit:\n local_backend: {}\n"},
{name: "empty", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ''\n"},
{name: "blank", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ' '\n"},
} {
t.Run(tt.name, func(t *testing.T) {
file := parseFileConfig(t, tt.yaml)
cfg := Default()
err := cfg.ApplyFileConfig(file)
if err == nil || !strings.Contains(err.Error(), "promptkit.local_backend.endpoint") {
t.Fatalf("ApplyFileConfig() error = %v, want endpoint field context", err)
}
})
}
}
func TestFilePromptKitExplicitEmptyProfileSourcesAreRejected(t *testing.T) {
for _, field := range []string{"profile_dir", "profile_file"} {
t.Run(field, func(t *testing.T) {
@@ -188,6 +370,11 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
yaml: "version: 4\ncache:\n checkpoints:\n enabled: definitely\n",
want: "cannot unmarshal",
},
{
name: "local backend field",
yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: http://localhost:8000/v1\n unknown: true\n",
want: "field unknown not found",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {

View File

@@ -19,23 +19,28 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
}
resolved := pipeline.ResolvedPipeline{
ID: "redaction-test",
Digest: "sha256:safe-digest",
Input: bindings["input"],
Chunk: bindings["chunk"],
ChunkReferences: redactionTestReferenceTarget(pipeline.StageChunk, "", "chunk-reference-content"),
ID: "redaction-test",
Digest: "sha256:safe-digest",
Input: bindings["input"],
InputExecutionClass: contracts.ExecutionClassDeterministic,
Chunk: bindings["chunk"],
ChunkExecutionClass: contracts.ExecutionClassLLMBacked,
ChunkReferences: redactionTestReferenceTarget(pipeline.StageChunk, "", "chunk-reference-content"),
Steps: []pipeline.ResolvedPipelineStep{{
ID: "default",
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
ID: "safe-lane",
ArtifactKind: "safe/artifact",
Extract: bindings["extract"],
Merge: bindings["merge"],
Normalize: bindings["normalize"],
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"),
MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"),
NormalizeReferences: redactionTestReferenceTarget(pipeline.StageNormalize, "safe-lane", "normalize-reference-content"),
ID: "safe-lane",
ArtifactKind: "safe/artifact",
Extract: bindings["extract"],
ExtractExecutionClass: contracts.ExecutionClassLLMBacked,
Merge: bindings["merge"],
MergeExecutionClass: contracts.ExecutionClassDeterministic,
Normalize: bindings["normalize"],
NormalizeExecutionClass: contracts.ExecutionClassLLMBacked,
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"),
MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"),
NormalizeReferences: redactionTestReferenceTarget(pipeline.StageNormalize, "safe-lane", "normalize-reference-content"),
}},
}},
ValidatorChains: []pipeline.ResolvedValidatorChain{{
@@ -49,7 +54,8 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
ArtifactKind: "safe/artifact",
}},
}},
Output: bindings["output"],
Output: bindings["output"],
OutputExecutionClass: contracts.ExecutionClassDeterministic,
}
effective := EffectiveConfig{
Config: Config{Pipelines: map[string]pipeline.PipelineProfile{
@@ -88,6 +94,11 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
t.Fatalf("resolved pipeline summary does not retain %q: %s", safe, text)
}
}
for _, executionClass := range []string{"input_execution_class\":\"deterministic", "chunk_execution_class\":\"llm_backed", "extract_execution_class\":\"llm_backed", "merge_execution_class\":\"deterministic", "normalize_execution_class\":\"llm_backed", "output_execution_class\":\"deterministic"} {
if !strings.Contains(text, executionClass) {
t.Fatalf("resolved pipeline summary does not retain %q: %s", executionClass, text)
}
}
payload.Input.Options["safe"] = "mutated"
nested := payload.Input.Options["nested"].([]any)[0].([]any)[0].(map[string]any)

View File

@@ -2,6 +2,7 @@ package config
import (
"fmt"
"net/url"
"sort"
"strings"
@@ -53,6 +54,27 @@ func validatePromptKit(cfg PromptKitConfig) error {
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
return fmt.Errorf("promptkit profile_dir and profile_file are mutually exclusive")
}
if cfg.LocalBackend == nil {
return nil
}
endpoint := strings.TrimSpace(cfg.LocalBackend.Endpoint)
if endpoint == "" {
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
}
parsed, err := url.Parse(endpoint)
if err != nil ||
(!strings.EqualFold(parsed.Scheme, "http") && !strings.EqualFold(parsed.Scheme, "https")) ||
!parsed.IsAbs() ||
parsed.Hostname() == "" ||
parsed.User != nil ||
parsed.RawQuery != "" ||
parsed.ForceQuery ||
strings.Contains(endpoint, "#") {
return fmt.Errorf("promptkit.local_backend.endpoint must be an absolute HTTP or HTTPS URL with a host and no user information, query, or fragment")
}
if cfg.LocalBackend.ConcurrencyLimit < 0 {
return fmt.Errorf("promptkit.local_backend.concurrency_limit must not be negative")
}
return nil
}

View File

@@ -93,6 +93,73 @@ func TestValidatePromptKitSourcesAreMutuallyExclusive(t *testing.T) {
assertValidationContains(t, cfg, "promptkit profile_dir and profile_file are mutually exclusive")
}
func TestValidatePromptKitLocalBackendEndpoints(t *testing.T) {
tests := []struct {
name string
endpoint string
profileSource PromptKitConfig
}{
{
name: "HTTP endpoint with path and profile directory",
endpoint: "http://localhost:8000/v1",
profileSource: PromptKitConfig{ProfileDir: "./profiles"},
},
{
name: "case-insensitive HTTPS endpoint and profile file",
endpoint: "HTTPS://inference.example.test/api",
profileSource: PromptKitConfig{ProfileFile: "./profiles.yml"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := Default()
cfg.PromptKit = tt.profileSource
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
Endpoint: tt.endpoint,
ConcurrencyLimit: 2,
}
if err := cfg.Validate(); err != nil {
t.Fatalf("Validate() error = %v", err)
}
})
}
}
func TestValidatePromptKitLocalBackendRejectsInvalidValues(t *testing.T) {
tests := []struct {
name string
endpoint string
concurrencyLimit int
want string
}{
{name: "blank endpoint", endpoint: " ", want: "promptkit.local_backend.endpoint"},
{name: "relative URL", endpoint: "localhost:8000/v1", want: "promptkit.local_backend.endpoint"},
{name: "unsupported scheme", endpoint: "ftp://localhost/model", want: "promptkit.local_backend.endpoint"},
{name: "missing host", endpoint: "http:///v1", want: "promptkit.local_backend.endpoint"},
{name: "user information", endpoint: "http://user:secret@localhost/v1", want: "promptkit.local_backend.endpoint"},
{name: "query", endpoint: "http://localhost/v1?model=example", want: "promptkit.local_backend.endpoint"},
{name: "empty query", endpoint: "http://localhost/v1?", want: "promptkit.local_backend.endpoint"},
{name: "fragment", endpoint: "http://localhost/v1#model", want: "promptkit.local_backend.endpoint"},
{name: "empty fragment", endpoint: "http://localhost/v1#", want: "promptkit.local_backend.endpoint"},
{
name: "negative concurrency",
endpoint: "http://localhost:8000/v1",
concurrencyLimit: -1,
want: "promptkit.local_backend.concurrency_limit",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := Default()
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
Endpoint: tt.endpoint,
ConcurrencyLimit: tt.concurrencyLimit,
}
assertValidationContains(t, cfg, tt.want)
})
}
}
func TestValidateStateSurfaceRules(t *testing.T) {
tests := []struct {
name string

View File

@@ -267,10 +267,6 @@ const (
ExecutionClassLLMBacked ExecutionClass = "llm_backed"
)
type ChunkExecutionClassProvider interface {
ExecutionClass() ExecutionClass
}
type ValidationResult struct {
Approved bool `json:"approved"`
ReasonCode string `json:"reason_code,omitempty"`

View File

@@ -21,8 +21,9 @@ type AssetSource struct {
}
type AssetRegistry struct {
prompts []AssetSource
schemas []AssetSource
prompts []AssetSource
schemas []AssetSource
fallbackProfiles []AssetSource
}
type AssetHashPart struct {
@@ -58,6 +59,20 @@ func (r *AssetRegistry) RegisterSchemaFS(fsys fs.FS, root string) error {
return nil
}
// RegisterFallbackProfileFS registers profile assets that PromptKit uses only
// when an operator-configured source does not provide a matching profile.
func (r *AssetRegistry) RegisterFallbackProfileFS(fsys fs.FS, root string) error {
if r == nil {
return fmt.Errorf("asset registry must not be nil")
}
source, err := newAssetSource(fsys, root)
if err != nil {
return fmt.Errorf("register fallback profile assets: %w", err)
}
r.fallbackProfiles = append(r.fallbackProfiles, source)
return nil
}
func (r *AssetRegistry) PromptFS() (fs.FS, error) {
if r == nil {
return nil, fmt.Errorf("asset registry must not be nil")
@@ -72,19 +87,76 @@ func (r *AssetRegistry) SchemaFS() (fs.FS, error) {
return flattenAssetSources(r.schemas)
}
func (r *AssetRegistry) FallbackProfileFS() (fs.FS, error) {
if r == nil {
return nil, fmt.Errorf("asset registry must not be nil")
}
return flattenAssetSources(r.fallbackProfiles)
}
// FallbackProfileDigest returns a deterministic, non-secret identity for the
// flattened fallback profile assets.
func (r *AssetRegistry) FallbackProfileDigest() (string, error) {
_, digest, _, err := r.fallbackProfileAssets()
return digest, err
}
func (r *AssetRegistry) PromptKitOptions() ([]promptkit.Option, error) {
options, _, err := r.promptKitOptions()
return options, err
}
func (r *AssetRegistry) promptKitOptions() ([]promptkit.Option, string, error) {
promptFS, err := r.PromptFS()
if err != nil {
return nil, fmt.Errorf("prepare prompt assets: %w", err)
return nil, "", fmt.Errorf("prepare prompt assets: %w", err)
}
schemaFS, err := r.SchemaFS()
if err != nil {
return nil, fmt.Errorf("prepare schema assets: %w", err)
return nil, "", fmt.Errorf("prepare schema assets: %w", err)
}
return []promptkit.Option{
options := []promptkit.Option{
promptkit.WithPromptFS(promptFS, "."),
promptkit.WithSchemaFS(schemaFS, "."),
}, nil
}
fallbackFS, fallbackDigest, hasFallback, err := r.fallbackProfileAssets()
if err != nil {
return nil, "", err
}
if hasFallback {
options = append(options, promptkit.WithFallbackProfileFS(fallbackFS, "."))
}
return options, fallbackDigest, nil
}
func (r *AssetRegistry) promptKitFallbackProfileOption() (promptkit.Option, bool, error) {
fallbackFS, _, hasFallback, err := r.fallbackProfileAssets()
if err != nil {
return nil, false, err
}
if !hasFallback {
return nil, false, nil
}
return promptkit.WithFallbackProfileFS(fallbackFS, "."), true, nil
}
func (r *AssetRegistry) fallbackProfileAssets() (fs.FS, string, bool, error) {
if r == nil {
return nil, "", false, fmt.Errorf("asset registry must not be nil")
}
if len(r.fallbackProfiles) == 0 {
empty := sha256.Sum256([]byte("notarius:fallback-profile-assets:empty"))
return nil, "sha256:" + hex.EncodeToString(empty[:]), false, nil
}
fallbackFS, err := r.FallbackProfileFS()
if err != nil {
return nil, "", false, fmt.Errorf("prepare fallback profile assets: %w", err)
}
digest, err := hashAssetFilesystem(fallbackFS)
if err != nil {
return nil, "", false, err
}
return fallbackFS, digest, true, nil
}
func HashAssets(parts []AssetHashPart) (string, error) {
@@ -117,6 +189,28 @@ func HashAssets(parts []AssetHashPart) (string, error) {
return "sha256:" + hex.EncodeToString(hash.Sum(nil)), nil
}
func hashAssetFilesystem(fsys fs.FS) (string, error) {
var parts []AssetHashPart
err := fs.WalkDir(fsys, ".", func(name string, entry fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if entry.IsDir() {
return nil
}
parts = append(parts, AssetHashPart{FS: fsys, Path: name})
return nil
})
if err != nil {
return "", fmt.Errorf("walk assets for digest: %w", err)
}
if len(parts) > 0 {
return HashAssets(parts)
}
empty := sha256.Sum256([]byte("notarius:fallback-profile-assets:empty"))
return "sha256:" + hex.EncodeToString(empty[:]), nil
}
func newAssetSource(fsys fs.FS, root string) (AssetSource, error) {
if fsys == nil {
return AssetSource{}, fmt.Errorf("filesystem must not be nil")
@@ -248,6 +342,9 @@ func (m assetMapFS) dirEntries(dir string) []fs.DirEntry {
children[childName] = entry
}
if len(children) == 0 {
if dir == "." {
return []fs.DirEntry{}
}
return nil
}
names := make([]string, 0, len(children))

View File

@@ -2,6 +2,7 @@ package llm
import (
"context"
"io/fs"
"strings"
"testing"
"testing/fstest"
@@ -98,6 +99,89 @@ func TestAssetRegistryRejectsDuplicateAssetPaths(t *testing.T) {
}
}
func TestAssetRegistryCombinesFallbackProfileSources(t *testing.T) {
registry := NewAssetRegistry()
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
"first/profiles/one.yaml": {Data: []byte("id: one\nmodel: first\n")},
}, "first/profiles"); err != nil {
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
}
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
"second/two.yaml": {Data: []byte("id: two\nmodel: second\n")},
}, "second"); err != nil {
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
}
fallbackFS, err := registry.FallbackProfileFS()
if err != nil {
t.Fatalf("FallbackProfileFS() error = %v, want nil", err)
}
for _, name := range []string{"one.yaml", "two.yaml"} {
if _, err := fs.ReadFile(fallbackFS, name); err != nil {
t.Fatalf("FallbackProfileFS().ReadFile(%q) error = %v, want nil", name, err)
}
}
}
func TestAssetRegistryRejectsInvalidFallbackProfileRoot(t *testing.T) {
registry := NewAssetRegistry()
err := registry.RegisterFallbackProfileFS(fstest.MapFS{}, "../profiles")
if err == nil || !strings.Contains(err.Error(), "invalid path") {
t.Fatalf("RegisterFallbackProfileFS() error = %v, want invalid root error", err)
}
}
func TestAssetRegistryRejectsUnreadableFallbackProfileAssets(t *testing.T) {
registry := NewAssetRegistry()
if err := registry.RegisterFallbackProfileFS(unreadableAssetFS{}, "."); err != nil {
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
}
_, err := registry.FallbackProfileFS()
if err == nil || !strings.Contains(err.Error(), "permission denied") {
t.Fatalf("FallbackProfileFS() error = %v, want unreadable asset error", err)
}
}
func TestAssetRegistryRejectsDuplicateFallbackProfilePaths(t *testing.T) {
registry := NewAssetRegistry()
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{"first/profile.yaml": {Data: []byte("id: first\n")}}, "first"); err != nil {
t.Fatal(err)
}
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{"second/profile.yaml": {Data: []byte("id: second\n")}}, "second"); err != nil {
t.Fatal(err)
}
_, err := registry.FallbackProfileFS()
if err == nil || !strings.Contains(err.Error(), "duplicate asset path") {
t.Fatalf("FallbackProfileFS() error = %v, want duplicate path error", err)
}
}
func TestAssetRegistryFallbackProfileDigestTracksContentWithoutLeakingIt(t *testing.T) {
digestFor := func(content string) string {
t.Helper()
registry := NewAssetRegistry()
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
"profiles/profile.yaml": {Data: []byte(content)},
}, "profiles"); err != nil {
t.Fatal(err)
}
digest, err := registry.FallbackProfileDigest()
if err != nil {
t.Fatal(err)
}
return digest
}
first := digestFor("id: fallback\nmodel: model-one\n")
second := digestFor("id: fallback\nmodel: model-two\n")
if first == second {
t.Fatalf("fallback profile digests = %q and %q, want content change", first, second)
}
if !strings.HasPrefix(first, "sha256:") || strings.Contains(first, "model-one") || strings.Contains(first, "profile.yaml") {
t.Fatalf("fallback profile digest leaked source details: %q", first)
}
}
func TestAssetRegistryCombinesNamespacedPromptSources(t *testing.T) {
registry := NewAssetRegistry()
mustRegisterPromptFS(t, registry, fstest.MapFS{
@@ -196,3 +280,9 @@ output:
repair_attempts: 0
`
}
type unreadableAssetFS struct{}
func (unreadableAssetFS) Open(name string) (fs.File, error) {
return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrPermission}
}

View File

@@ -19,9 +19,15 @@ import (
const promptKitProviderName = "promptkit"
type PromptKitLocalBackendConfig struct {
Endpoint string
ConcurrencyLimit int
}
type PromptKitClientConfig struct {
ProfileDir string
ProfileFile string
LocalBackend *PromptKitLocalBackendConfig
Assets *AssetRegistry
Timeout time.Duration
HTTPClient *http.Client
@@ -31,11 +37,13 @@ type PromptKitClientConfig struct {
}
type PromptKitClient struct {
engine *promptkit.Engine
recorder *LLMProfileRecorder
profileDir string
profileFile string
reasoningEffort *string
engine *promptkit.Engine
recorder *LLMProfileRecorder
profileDir string
profileFile string
localEndpoint string
fallbackProfileDigest string
reasoningEffort *string
}
type LLMProfileRecorder struct {
@@ -46,24 +54,31 @@ type LLMProfileRecorder struct {
var _ contracts.StructuredLLMClient = (*PromptKitClient)(nil)
var _ contracts.LLMProfileManifestProvider = (*PromptKitClient)(nil)
func PromptKitLocalBackendOption(cfg PromptKitLocalBackendConfig) promptkit.Option {
return promptkit.WithBackend(promptkit.LocalBackend(cfg.Endpoint, cfg.ConcurrencyLimit))
}
func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
if cfg.Assets == nil {
return nil, fmt.Errorf("PromptKit client assets must not be nil")
}
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
return nil, fmt.Errorf("PromptKit profile_dir and profile_file are mutually exclusive")
}
options, err := cfg.Assets.PromptKitOptions()
profileSource, profileOptions, err := promptKitProfileSourceEngineOptions(PromptKitProfileSourceConfig{
ProfileDir: cfg.ProfileDir,
ProfileFile: cfg.ProfileFile,
LocalBackend: cfg.LocalBackend,
})
if err != nil {
return nil, err
}
if profileFile := strings.TrimSpace(cfg.ProfileFile); profileFile != "" {
options = append(options, promptkit.WithProfileFile(profileFile))
options, fallbackProfileDigest, err := cfg.Assets.promptKitOptions()
if err != nil {
return nil, err
}
options = append(options, profileOptions...)
options = append(options, cfg.EngineOptions...)
engine, err := promptkit.NewEngine(promptkit.Config{
ProfileDir: strings.TrimSpace(cfg.ProfileDir),
ProfileDir: profileSource.ProfileDir,
Timeout: cfg.Timeout,
HTTPClient: cfg.HTTPClient,
}, options...)
@@ -80,11 +95,13 @@ func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
reasoningEffort = &value
}
return &PromptKitClient{
engine: engine,
recorder: recorder,
profileDir: strings.TrimSpace(cfg.ProfileDir),
profileFile: strings.TrimSpace(cfg.ProfileFile),
reasoningEffort: reasoningEffort,
engine: engine,
recorder: recorder,
profileDir: profileSource.ProfileDir,
profileFile: profileSource.ProfileFile,
localEndpoint: profileSource.localEndpoint(),
fallbackProfileDigest: fallbackProfileDigest,
reasoningEffort: reasoningEffort,
}, nil
}
@@ -120,19 +137,31 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
Vars: promptKitVars(req, sessionID),
Execution: execution,
}
prepared, err := c.engine.Prepare(ctx, runReq)
prepared, err := c.engine.PrepareExecution(ctx, runReq)
if err != nil {
if ctxErr := ctx.Err(); ctxErr != nil {
return contracts.StructuredCompletionResponse{}, ctxErr
}
return contracts.StructuredCompletionResponse{}, fmt.Errorf("prepare PromptKit prompt %q: %w", promptID, redactPromptKitError(err))
}
result, err := c.engine.Run(ctx, runReq)
defer prepared.Discard()
preparedDetails := prepared.Details()
result, err := c.engine.RunPrepared(ctx, prepared)
if err != nil {
if ctxErr := ctx.Err(); ctxErr != nil {
return contracts.StructuredCompletionResponse{}, ctxErr
}
if errors.Is(err, promptkit.ErrCapacityExceeded) {
var capacityErr *promptkit.CapacityError
if errors.As(err, &capacityErr) && strings.TrimSpace(capacityErr.BackendID) != "" {
return contracts.StructuredCompletionResponse{}, fmt.Errorf(
"run PromptKit prompt %q on backend %q: %w: %v",
promptID,
strings.TrimSpace(capacityErr.BackendID),
contracts.ErrLLMCapacityExceeded,
redactPromptKitError(err),
)
}
return contracts.StructuredCompletionResponse{}, fmt.Errorf(
"run PromptKit prompt %q: %w: %v",
promptID,
@@ -145,7 +174,7 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
if result == nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w: empty result", promptID, contracts.ErrInvalidStructuredOutput)
}
response := c.responseFromResult(result, prepared)
response := c.responseFromResult(result, &preparedDetails)
if result.Validation.Status == promptkit.ValidationFailed || !result.Validation.IsValid {
return response, fmt.Errorf("run PromptKit prompt %q: %w: validation failed: %s", promptID, contracts.ErrInvalidStructuredOutput, strings.Join(result.Validation.Errors, "; "))
}
@@ -298,11 +327,15 @@ func (c *PromptKitClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint,
if c == nil {
return nil, nil
}
fingerprint, err := promptKitProfileFingerprint(c.profileDir, c.profileFile)
fingerprint, err := promptKitProfileFingerprint(c.profileDir, c.profileFile, c.fallbackProfileDigest)
if err != nil {
return nil, err
}
return []CheckpointFingerprint{fingerprint}, nil
fingerprints := []CheckpointFingerprint{fingerprint}
if c.localEndpoint != "" {
fingerprints = append(fingerprints, promptKitLocalBackendFingerprint(c.localEndpoint))
}
return fingerprints, nil
}
func NewLLMProfileRecorder() *LLMProfileRecorder {

View File

@@ -5,7 +5,9 @@ import (
"encoding/json"
"errors"
"io"
"io/fs"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"reflect"
@@ -113,6 +115,104 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
}
}
func TestPromptKitClientUsesOnePreparedSnapshotForDebugAndGeneration(t *testing.T) {
const initialPrompt = `id: snapshot.test
version: "v1"
default_profile: snapshot-profile
inputs:
- name: transcript
required: true
content_type: application/json
messages:
- role: user
content: "Snapshot A: {{ input \"transcript\" }}"
output:
format: json
validation_mode: json_schema
schema_path: adapter.schema.json
repair_attempts: 0
`
const updatedPrompt = `id: snapshot.test
version: "v1"
default_profile: snapshot-profile
inputs:
- name: transcript
required: true
content_type: application/json
messages:
- role: user
content: "Snapshot B: {{ input \"transcript\" }}"
output:
format: json
validation_mode: json_schema
schema_path: adapter.schema.json
repair_attempts: 0
`
source := &switchingPromptFS{files: fstest.MapFS{
"snapshot.test.yaml": {Data: []byte(initialPrompt)},
}}
fake := &fakePromptKitLLM{content: `{"ok":true}`}
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
EngineOptions: []promptkit.Option{
promptkit.WithPromptFS(source, "."),
promptkit.WithBackend(promptkit.Backend{ID: "snapshot-backend", Endpoint: "http://promptkit.test/v1"}),
promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
ID: "snapshot-profile",
BackendID: "snapshot-backend",
Model: "snapshot-model",
})),
promptkit.WithLLMClient(fake),
},
})
if err != nil {
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
}
opened := source.holdNextPromptRead()
defer source.resumePromptRead()
type completion struct {
response contracts.StructuredCompletionResponse
err error
}
completed := make(chan completion, 1)
go func() {
var out struct {
OK bool `json:"ok"`
}
response, callErr := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
PromptID: "snapshot.test",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
}, &out)
completed <- completion{response: response, err: callErr}
}()
select {
case <-opened:
case <-time.After(time.Second):
t.Fatal("PromptKit did not read the prompt source")
}
source.replacePrompt([]byte(updatedPrompt))
source.resumePromptRead()
result := <-completed
if result.err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", result.err)
}
if result.response.Debug == nil || result.response.Debug.Prompt == nil || len(result.response.Debug.Prompt.Messages) != 1 {
t.Fatalf("debug prompt = %#v, want one prepared message", result.response.Debug)
}
debugContent := result.response.Debug.Prompt.Messages[0].Content
generatedContent := fake.lastRequest().Prompt.Messages[0].Content
if debugContent != generatedContent {
t.Fatalf("debug content = %q, generation content = %q, want one snapshot", debugContent, generatedContent)
}
if !strings.Contains(debugContent, "Snapshot A") || strings.Contains(debugContent, "Snapshot B") {
t.Fatalf("snapshot content = %q, want the source read before it changed", debugContent)
}
}
func TestPromptKitClientRetainsSessionPromptVariable(t *testing.T) {
fake := &fakePromptKitLLM{content: `{"ok":true}`}
client := newTestPromptKitClient(t, fake)
@@ -252,9 +352,10 @@ func TestNewPromptKitClientReportsAssetAndEngineConstructionFailures(t *testing.
func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
const credentialEnvironment = "PROMPTKIT_TEST_API_KEY"
writeProfile := func(model string) {
t.Helper()
content := "id: checkpoint-profile\nendpoint: http://promptkit.test/v1\nmodel: " + model + "\n"
content := "id: checkpoint-profile\nendpoint: http://promptkit.test/v1\nmodel: " + model + "\napi_key_env: " + credentialEnvironment + "\n"
if err := os.WriteFile(profilePath, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
@@ -280,12 +381,21 @@ func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
writeProfile("model-one")
first := fingerprintFor()
repeated := fingerprintFor()
if first != repeated {
t.Fatalf("profile-source fingerprint = %#v then %#v for unchanged source", first, repeated)
}
writeProfile("model-two")
second := fingerprintFor()
if first == second {
t.Fatalf("profile-source fingerprint = %#v for both profile models", first)
}
if strings.Contains(first.Value, profilePath) || strings.Contains(first.Value, "model-one") {
if strings.TrimSpace(first.Value) == "" {
t.Fatal("profile-source fingerprint is empty")
}
if strings.Contains(first.Value, profilePath) ||
strings.Contains(first.Value, "model-one") ||
strings.Contains(first.Value, credentialEnvironment) {
t.Fatalf("profile-source fingerprint exposes source details: %#v", first)
}
@@ -305,9 +415,229 @@ func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
if fresh[0].Value == "mutated" {
t.Fatal("LLMCheckpointFingerprints exposed mutable backing storage")
}
const wantBuiltinFingerprint = "sha256:4b48cd1ef010b587cd8d56bc13b73a0cdd84da1414a8de9f9a2adc695dfcf0a7"
if fresh[0].Value != wantBuiltinFingerprint {
t.Fatalf("built-in profile fingerprint = %q, want %q", fresh[0].Value, wantBuiltinFingerprint)
secondClient, err := NewPromptKitClient(PromptKitClientConfig{Assets: newTestPromptKitAssets(t)})
if err != nil {
t.Fatal(err)
}
secondBuiltin, err := secondClient.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
if len(secondBuiltin) != 1 || fresh[0] != secondBuiltin[0] {
t.Fatalf("built-in profile fingerprints = %#v and %#v, want deterministic identity", fresh, secondBuiltin)
}
if strings.TrimSpace(fresh[0].Value) == "" {
t.Fatal("built-in profile fingerprint is empty")
}
t.Run("directory layout", func(t *testing.T) {
profileDir := t.TempDir()
firstPath := filepath.Join(profileDir, "first-profile.yaml")
secondPath := filepath.Join(profileDir, "second-profile.yaml")
content := []byte("id: directory-profile\nendpoint: http://promptkit.test/v1\nmodel: directory-model\n")
if err := os.WriteFile(firstPath, content, 0o600); err != nil {
t.Fatal(err)
}
first, err := promptKitProfileFingerprint(profileDir, "", "")
if err != nil {
t.Fatal(err)
}
if err := os.Rename(firstPath, secondPath); err != nil {
t.Fatal(err)
}
second, err := promptKitProfileFingerprint(profileDir, "", "")
if err != nil {
t.Fatal(err)
}
if first == second {
t.Fatalf("profile-source fingerprint = %#v after source filename changed", first)
}
if strings.Contains(first.Value, firstPath) || strings.Contains(second.Value, secondPath) {
t.Fatalf("profile-source fingerprint exposes source path: %#v, %#v", first, second)
}
})
}
func TestPromptKitProfileFingerprintReadErrorsDoNotExposeSourcePaths(t *testing.T) {
for _, source := range []struct {
name string
profileDir string
profileFile string
}{
{
name: "file",
profileFile: filepath.Join(t.TempDir(), "missing-profile.yaml"),
},
{
name: "directory",
profileDir: filepath.Join(t.TempDir(), "missing-profiles"),
},
} {
t.Run(source.name, func(t *testing.T) {
_, err := promptKitProfileFingerprint(source.profileDir, source.profileFile, "")
if err == nil {
t.Fatal("promptKitProfileFingerprint() error = nil, want source read failure")
}
if (source.profileDir != "" && strings.Contains(err.Error(), source.profileDir)) ||
(source.profileFile != "" && strings.Contains(err.Error(), source.profileFile)) {
t.Fatalf("fingerprint error exposes profile source path: %q", err)
}
})
}
}
func TestPromptKitClientUsesFallbackProfilesForExecutionAndInspection(t *testing.T) {
assets := newTestPromptKitAssets(t)
const profileID = "fallback-profile"
if err := assets.RegisterFallbackProfileFS(fstest.MapFS{
"profiles/fallback.yaml": {Data: []byte("id: " + profileID + "\nendpoint: http://promptkit.test/v1\nmodel: fallback-model\n")},
}, "profiles"); err != nil {
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
}
fake := &fakePromptKitLLM{content: `{"ok":true}`}
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: assets,
EngineOptions: []promptkit.Option{promptkit.WithLLMClient(fake)},
})
if err != nil {
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
}
var out map[string]any
response, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
PromptID: "adapter.test",
ProfileID: profileID,
SessionID: "fallback-profile-test",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
}, &out)
if err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", err)
}
if response.ProfileID != profileID || response.Model != "fallback-model" {
t.Fatalf("completion response = %#v, want fallback profile", response)
}
inspector, err := NewPromptKitProfileInspector(PromptKitProfileInspectorConfig{Assets: assets})
if err != nil {
t.Fatalf("NewPromptKitProfileInspector() error = %v, want nil", err)
}
inspection, err := inspector.InspectProfile(context.Background(), profileID)
if err != nil {
t.Fatalf("InspectProfile() error = %v, want nil", err)
}
if inspection.ProfileID != profileID || inspection.Model != "fallback-model" {
t.Fatalf("profile inspection = %#v, want fallback profile", inspection)
}
}
func TestPromptKitClientCheckpointFingerprintTracksFallbackProfileAssets(t *testing.T) {
fingerprintFor := func(content string) CheckpointFingerprint {
t.Helper()
assets := newTestPromptKitAssets(t)
if err := assets.RegisterFallbackProfileFS(fstest.MapFS{
"profiles/fallback.yaml": {Data: []byte(content)},
}, "profiles"); err != nil {
t.Fatal(err)
}
client, err := NewPromptKitClient(PromptKitClientConfig{Assets: assets})
if err != nil {
t.Fatal(err)
}
fingerprints, err := client.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
if len(fingerprints) != 1 || fingerprints[0].Name != promptKitProfileFingerprintName {
t.Fatalf("checkpoint fingerprints = %#v, want profile source identity", fingerprints)
}
return fingerprints[0]
}
first := fingerprintFor("id: fallback\nendpoint: http://promptkit.test/v1\nmodel: model-one\n")
second := fingerprintFor("id: fallback\nendpoint: http://promptkit.test/v1\nmodel: model-two\n")
if first == second {
t.Fatalf("checkpoint fingerprints = %#v and %#v, want fallback asset change", first, second)
}
if strings.Contains(first.Value, "model-one") || strings.Contains(first.Value, "fallback.yaml") {
t.Fatalf("checkpoint fingerprint leaked fallback source details: %#v", first)
}
}
func TestPromptKitClientCheckpointFingerprintTracksLocalBackendTarget(t *testing.T) {
const (
firstEndpoint = "http://localhost:8000/v1"
secondEndpoint = "https://inference.example.test/v1"
)
fingerprintsFor := func(endpoint string, concurrencyLimit int) []CheckpointFingerprint {
t.Helper()
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
LocalBackend: &PromptKitLocalBackendConfig{
Endpoint: endpoint,
ConcurrencyLimit: concurrencyLimit,
},
})
if err != nil {
t.Fatal(err)
}
values, err := client.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
return values
}
baseline := fingerprintsFor(firstEndpoint, 0)
if len(baseline) != 2 ||
baseline[0].Name != promptKitProfileFingerprintName ||
baseline[1].Name != promptKitLocalBackendFingerprintName {
t.Fatalf("checkpoint fingerprints = %#v, want profile source then local backend target", baseline)
}
endpointChanged := fingerprintsFor(secondEndpoint, 0)
if baseline[0] != endpointChanged[0] || baseline[1] == endpointChanged[1] {
t.Fatalf("endpoint fingerprints = %#v and %#v, want only local target to change", baseline, endpointChanged)
}
concurrencyChanged := fingerprintsFor(firstEndpoint, 4)
if !reflect.DeepEqual(baseline, concurrencyChanged) {
t.Fatalf("concurrency fingerprints = %#v, want %#v", concurrencyChanged, baseline)
}
for _, values := range [][]CheckpointFingerprint{baseline, endpointChanged} {
for _, fingerprint := range values {
if strings.Contains(fingerprint.Value, firstEndpoint) ||
strings.Contains(fingerprint.Value, secondEndpoint) {
t.Fatalf("checkpoint fingerprint exposes endpoint: %#v", fingerprint)
}
}
}
localBackend := &PromptKitLocalBackendConfig{
Endpoint: " " + firstEndpoint + " ",
ConcurrencyLimit: 0,
}
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
LocalBackend: localBackend,
})
if err != nil {
t.Fatal(err)
}
localBackend.Endpoint = secondEndpoint
copy, err := client.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(copy, baseline) {
t.Fatalf("fingerprints after input mutation = %#v, want retained target %#v", copy, baseline)
}
copy[0].Value = "mutated-profile"
copy[1].Value = "mutated-target"
fresh, err := client.LLMCheckpointFingerprints()
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(fresh, baseline) {
t.Fatalf("fingerprints after returned-slice mutation = %#v, want %#v", fresh, baseline)
}
}
@@ -334,6 +664,88 @@ func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testi
}
}
func TestPromptKitClientUsesConfiguredLocalBackend(t *testing.T) {
var providerCalls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
providerCalls.Add(1)
if r.URL.Path != "/v1/chat/completions" {
t.Errorf("provider path = %q, want /v1/chat/completions", r.URL.Path)
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{
"choices": [{"message": {"role": "assistant", "content": "{\"ok\":true}"}}],
"usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7}
}`))
}))
defer server.Close()
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
if err := os.WriteFile(profilePath, []byte(`id: local-profile
backend: local
model: local-model
`), 0o600); err != nil {
t.Fatal(err)
}
localBackend := &PromptKitLocalBackendConfig{
Endpoint: " " + server.URL + "/v1 ",
ConcurrencyLimit: 2,
}
client, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
ProfileFile: profilePath,
LocalBackend: localBackend,
})
if err != nil {
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
}
localBackend.Endpoint = "http://127.0.0.1:1/v1"
request := contracts.StructuredCompletionRequest{
PromptID: "adapter.direct-session",
ProfileID: "local-profile",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
Vars: map[string]any{"custom": "value"},
}
var out map[string]any
response, err := client.CompleteStructured(context.Background(), request, &out)
if err != nil {
t.Fatalf("CompleteStructured() error = %v, want nil", err)
}
if providerCalls.Load() != 1 {
t.Fatalf("provider calls = %d, want 1", providerCalls.Load())
}
if response.ProfileID != "local-profile" || response.Model != "local-model" {
t.Fatalf("response metadata = %#v", response)
}
if response.Debug == nil || response.Debug.Prompt == nil ||
response.Debug.Prompt.SelectedBackendID != promptkit.BackendLocal {
t.Fatalf("response debug prompt = %#v, want local backend", response.Debug)
}
manifests := client.LLMProfileManifests()
if len(manifests) != 1 || manifests[0].BackendID != promptkit.BackendLocal {
t.Fatalf("profile manifests = %#v, want local backend", manifests)
}
clientWithoutRegistration, err := NewPromptKitClient(PromptKitClientConfig{
Assets: newTestPromptKitAssets(t),
ProfileFile: profilePath,
})
if err != nil {
t.Fatalf("NewPromptKitClient() without local registration error = %v, want nil", err)
}
_, err = clientWithoutRegistration.CompleteStructured(context.Background(), request, &out)
if err == nil ||
!strings.Contains(err.Error(), "prepare PromptKit prompt") ||
!strings.Contains(err.Error(), promptkit.BackendLocal) {
t.Fatalf("CompleteStructured() without registration error = %v, want preparation failure with local backend context", err)
}
if providerCalls.Load() != 1 {
t.Fatalf("provider calls after missing-registration failure = %d, want 1", providerCalls.Load())
}
}
func TestLLMProfileRecorderDistinguishesEffectiveTargets(t *testing.T) {
recorder := NewLLMProfileRecorder()
for _, profile := range []artifacts.LLMProfileManifest{
@@ -494,8 +906,13 @@ func TestPromptKitClientTranslatesBackendCapacityExhaustion(t *testing.T) {
t.Fatalf("capacity error exposes PromptKit sentinel: %v", capacityErr)
}
if !strings.Contains(capacityErr.Error(), `run PromptKit prompt "adapter.direct-session"`) ||
!strings.Contains(capacityErr.Error(), `backend "limited-backend"`) ||
!strings.Contains(capacityErr.Error(), "backend capacity exceeded") {
t.Fatalf("capacity error = %q, want prompt context and upstream diagnostic", capacityErr)
t.Fatalf("capacity error = %q, want prompt, backend, and upstream diagnostic context", capacityErr)
}
var upstreamCapacityErr *promptkit.CapacityError
if errors.As(capacityErr, &upstreamCapacityErr) {
t.Fatalf("capacity error exposes PromptKit capacity type: %v", capacityErr)
}
if calls := atomic.LoadInt32(&fake.calls); calls != 1 {
t.Fatalf("provider calls after capacity rejection = %d, want 1", calls)
@@ -730,6 +1147,61 @@ output:
return registry
}
type switchingPromptFS struct {
mu sync.Mutex
files fstest.MapFS
pauseNextOpen bool
opened chan struct{}
resume chan struct{}
}
func (f *switchingPromptFS) Open(name string) (fs.File, error) {
f.mu.Lock()
file, err := f.files.Open(name)
pause := f.pauseNextOpen && name == "snapshot.test.yaml"
resume := f.resume
if pause {
f.pauseNextOpen = false
close(f.opened)
}
f.mu.Unlock()
if pause {
<-resume
}
return file, err
}
func (f *switchingPromptFS) ReadDir(name string) ([]fs.DirEntry, error) {
f.mu.Lock()
defer f.mu.Unlock()
return f.files.ReadDir(name)
}
func (f *switchingPromptFS) holdNextPromptRead() <-chan struct{} {
f.mu.Lock()
defer f.mu.Unlock()
f.pauseNextOpen = true
f.opened = make(chan struct{})
f.resume = make(chan struct{})
return f.opened
}
func (f *switchingPromptFS) replacePrompt(content []byte) {
f.mu.Lock()
defer f.mu.Unlock()
f.files["snapshot.test.yaml"] = &fstest.MapFile{Data: append([]byte(nil), content...)}
}
func (f *switchingPromptFS) resumePromptRead() {
f.mu.Lock()
resume := f.resume
f.resume = nil
f.mu.Unlock()
if resume != nil {
close(resume)
}
}
type fakePromptKitLLM struct {
content string
allowEmpty bool

View File

@@ -12,30 +12,36 @@ import (
)
const (
promptKitProfileFingerprintName = "promptkit_profile_source"
promptKitProfileFingerprintName = "promptkit_profile_source"
promptKitLocalBackendFingerprintName = "promptkit_local_backend_target"
promptKitLocalBackendMarker = "notarius:promptkit-local-backend:v1"
// The built-in profile catalog is compiled into this pinned PromptKit
// release. Update this identity when the dependency is upgraded.
promptKitBuiltinProfileCatalogID = "promptkit:v0.2.0:builtin-profiles"
promptKitBuiltinProfileCatalogID = "promptkit:v0.5.0:builtin-profiles"
)
func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFingerprint, error) {
func promptKitProfileFingerprint(profileDir, profileFile, fallbackProfileDigest string) (CheckpointFingerprint, error) {
hasher := sha256.New()
writeFingerprintPart(hasher, []byte(promptKitBuiltinProfileCatalogID))
writeFingerprintPart(hasher, []byte(strings.TrimSpace(fallbackProfileDigest)))
switch {
case strings.TrimSpace(profileFile) != "":
data, err := os.ReadFile(strings.TrimSpace(profileFile))
cleanProfileFile := strings.TrimSpace(profileFile)
data, err := os.ReadFile(cleanProfileFile)
if err != nil {
return CheckpointFingerprint{}, fmt.Errorf("read PromptKit profile file for checkpoint identity: %w", err)
return CheckpointFingerprint{}, fmt.Errorf("read PromptKit profile file for checkpoint identity")
}
writeFingerprintPart(hasher, []byte(filepath.ToSlash(filepath.Base(cleanProfileFile))))
writeFingerprintPart(hasher, data)
case strings.TrimSpace(profileDir) != "":
digests, err := promptKitProfileFileDigests(strings.TrimSpace(profileDir))
files, err := promptKitProfileFiles(strings.TrimSpace(profileDir))
if err != nil {
return CheckpointFingerprint{}, err
}
for _, digest := range digests {
writeFingerprintPart(hasher, digest)
for _, file := range files {
writeFingerprintPart(hasher, []byte(file.path))
writeFingerprintPart(hasher, file.digest)
}
}
@@ -45,8 +51,23 @@ func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFing
}, nil
}
func promptKitProfileFileDigests(root string) ([][]byte, error) {
var digests [][]byte
func promptKitLocalBackendFingerprint(endpoint string) CheckpointFingerprint {
hasher := sha256.New()
writeFingerprintPart(hasher, []byte(promptKitLocalBackendMarker))
writeFingerprintPart(hasher, []byte(strings.TrimSpace(endpoint)))
return CheckpointFingerprint{
Name: promptKitLocalBackendFingerprintName,
Value: "sha256:" + hex.EncodeToString(hasher.Sum(nil)),
}
}
type promptKitProfileFile struct {
path string
digest []byte
}
func promptKitProfileFiles(root string) ([]promptKitProfileFile, error) {
var files []promptKitProfileFile
err := filepath.WalkDir(root, func(name string, entry fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
@@ -62,17 +83,24 @@ func promptKitProfileFileDigests(root string) ([][]byte, error) {
if err != nil {
return err
}
relativePath, err := filepath.Rel(root, name)
if err != nil {
return err
}
sum := sha256.Sum256(data)
digests = append(digests, append([]byte(nil), sum[:]...))
files = append(files, promptKitProfileFile{
path: filepath.ToSlash(relativePath),
digest: append([]byte(nil), sum[:]...),
})
return nil
})
if err != nil {
return nil, fmt.Errorf("read PromptKit profile directory for checkpoint identity: %w", err)
return nil, fmt.Errorf("read PromptKit profile directory for checkpoint identity")
}
sort.Slice(digests, func(i, j int) bool {
return string(digests[i]) < string(digests[j])
sort.Slice(files, func(i, j int) bool {
return files[i].path < files[j].path
})
return digests, nil
return files, nil
}
func writeFingerprintPart(hasher interface{ Write([]byte) (int, error) }, value []byte) {

View File

@@ -0,0 +1,146 @@
package llm
import (
"context"
"errors"
"fmt"
"strings"
"gitea.maximumdirect.net/eric/promptkit"
)
type PromptKitProfileSourceConfig struct {
ProfileDir string
ProfileFile string
LocalBackend *PromptKitLocalBackendConfig
}
type PromptKitProfileInspectorConfig struct {
Source PromptKitProfileSourceConfig
Assets *AssetRegistry
}
func (c PromptKitProfileSourceConfig) localEndpoint() string {
if c.LocalBackend == nil {
return ""
}
return c.LocalBackend.Endpoint
}
type PromptKitProfileInspector struct {
engine *promptkit.Engine
}
type PromptKitProfileInspection struct {
ProfileID string
BackendID string
Model string
CredentialEnvironment string
CredentialRequired bool
}
type PromptKitProfileInspectionError struct {
ProfileID string
err error
}
func (e *PromptKitProfileInspectionError) Error() string {
switch {
case errors.Is(e.err, promptkit.ErrProfileNotFound):
return fmt.Sprintf("PromptKit profile %q is not configured", e.ProfileID)
case errors.Is(e.err, promptkit.ErrInvalidRequest):
return fmt.Sprintf("PromptKit profile ID %q is invalid", e.ProfileID)
case errors.Is(e.err, promptkit.ErrProfileLoad):
return fmt.Sprintf("PromptKit profile %q is invalid or unreadable", e.ProfileID)
default:
return fmt.Sprintf("PromptKit profile %q could not be inspected", e.ProfileID)
}
}
func (e *PromptKitProfileInspectionError) Unwrap() error {
return e.err
}
type promptKitProfileConfigurationError struct {
err error
}
func (e *promptKitProfileConfigurationError) Error() string {
return "PromptKit profile configuration is invalid or unreadable"
}
func (e *promptKitProfileConfigurationError) Unwrap() error {
return e.err
}
func NewPromptKitProfileInspector(cfg PromptKitProfileInspectorConfig) (*PromptKitProfileInspector, error) {
source, options, err := promptKitProfileSourceEngineOptions(cfg.Source)
if err != nil {
return nil, err
}
if cfg.Assets != nil {
fallbackOption, hasFallback, err := cfg.Assets.promptKitFallbackProfileOption()
if err != nil {
return nil, err
}
if hasFallback {
options = append(options, fallbackOption)
}
}
engine, err := promptkit.NewEngine(promptkit.Config{
PromptDir: ".",
ProfileDir: source.ProfileDir,
}, options...)
if err != nil {
return nil, &promptKitProfileConfigurationError{err: err}
}
return &PromptKitProfileInspector{engine: engine}, nil
}
func (i *PromptKitProfileInspector) InspectProfile(ctx context.Context, profileID string) (PromptKitProfileInspection, error) {
if i == nil || i.engine == nil {
return PromptKitProfileInspection{}, fmt.Errorf("PromptKit profile inspector must not be nil")
}
profileID = strings.TrimSpace(profileID)
inspection, err := i.engine.InspectProfile(ctx, profileID)
if err != nil {
if ctxErr := ctx.Err(); ctxErr != nil {
return PromptKitProfileInspection{}, ctxErr
}
return PromptKitProfileInspection{}, &PromptKitProfileInspectionError{
ProfileID: profileID,
err: err,
}
}
return PromptKitProfileInspection{
ProfileID: inspection.ProfileID,
BackendID: strings.TrimSpace(inspection.EffectiveModelParams.BackendID),
Model: strings.TrimSpace(inspection.EffectiveModelParams.Model),
CredentialEnvironment: strings.TrimSpace(inspection.EffectiveModelParams.APIKeyEnv),
CredentialRequired: inspection.APIKeyRequired,
}, nil
}
func promptKitProfileSourceEngineOptions(cfg PromptKitProfileSourceConfig) (PromptKitProfileSourceConfig, []promptkit.Option, error) {
source := PromptKitProfileSourceConfig{
ProfileDir: strings.TrimSpace(cfg.ProfileDir),
ProfileFile: strings.TrimSpace(cfg.ProfileFile),
}
if source.ProfileDir != "" && source.ProfileFile != "" {
return PromptKitProfileSourceConfig{}, nil, fmt.Errorf("PromptKit profile_dir and profile_file are mutually exclusive")
}
if cfg.LocalBackend != nil {
localBackend := *cfg.LocalBackend
localBackend.Endpoint = strings.TrimSpace(localBackend.Endpoint)
source.LocalBackend = &localBackend
}
var options []promptkit.Option
if source.ProfileFile != "" {
options = append(options, promptkit.WithProfileFile(source.ProfileFile))
}
if source.LocalBackend != nil {
options = append(options, PromptKitLocalBackendOption(*source.LocalBackend))
}
return source, options, nil
}

View File

@@ -81,7 +81,10 @@ func TestScheduledClientPreservesCheckpointFingerprints(t *testing.T) {
t.Fatal(err)
}
inner := &fingerprintedStructuredClient{
fingerprints: []CheckpointFingerprint{{Name: "profile_source", Value: "sha256:one"}},
fingerprints: []CheckpointFingerprint{
{Name: "profile_source", Value: "sha256:one"},
{Name: "backend_target", Value: "sha256:two"},
},
}
client := NewScheduledClient(inner, scheduler)
provider, ok := client.(CheckpointFingerprintProvider)
@@ -92,7 +95,9 @@ func TestScheduledClientPreservesCheckpointFingerprints(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if len(got) != 1 || got[0] != inner.fingerprints[0] {
if len(got) != len(inner.fingerprints) ||
got[0] != inner.fingerprints[0] ||
got[1] != inner.fingerprints[1] {
t.Fatalf("checkpoint fingerprints = %#v, want %#v", got, inner.fingerprints)
}
}

View File

@@ -24,10 +24,6 @@ func NewChunkerRegistry() *ChunkerRegistry {
}
}
func (r *ChunkerRegistry) Register(key string, constructor ChunkerConstructor) error {
return r.RegisterWithSpec(defaultModuleSpec(key, StageChunk), constructor)
}
func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerConstructor) error {
if constructor == nil {
return fmt.Errorf("chunker constructor for %q must not be nil", strings.TrimSpace(spec.Key))

View File

@@ -29,6 +29,10 @@ type registryBehaviorCase[M any] struct {
moduleKey func(M) string
}
func testModuleSpec(key string, stage ModuleStage) ModuleSpec {
return ModuleSpec{Key: key, Stage: stage, ExecutionClass: contracts.ExecutionClassDeterministic}
}
func TestChunkerRegistryBehavior(t *testing.T) {
runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Chunker]{
name: "ChunkerRegistry",
@@ -39,7 +43,9 @@ func TestChunkerRegistryBehavior(t *testing.T) {
return NewChunkerRegistry()
},
register: func(registry any, key string, constructor func() (contracts.Chunker, error)) error {
return registry.(*ChunkerRegistry).Register(key, constructor)
return registry.(*ChunkerRegistry).RegisterWithSpec(ModuleSpec{
Key: key, Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic,
}, constructor)
},
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Chunker, error)) error {
return registry.(*ChunkerRegistry).RegisterWithSpec(spec, constructor)
@@ -55,7 +61,9 @@ func TestChunkerRegistryBehavior(t *testing.T) {
},
nilRegister: func(key string, constructor func() (contracts.Chunker, error)) error {
var registry *ChunkerRegistry
return registry.Register(key, constructor)
return registry.RegisterWithSpec(ModuleSpec{
Key: key, Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic,
}, constructor)
},
nilBuild: func(key string) (contracts.Chunker, error) {
var registry *ChunkerRegistry
@@ -101,10 +109,11 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
t.Run(testCase.name+"/metadata registration and lookup", func(t *testing.T) {
registry := testCase.newRegistry()
spec := ModuleSpec{
Key: " " + testCase.key + " ",
Stage: testCase.stage,
Provides: []string{" beta ", "alpha", "", "beta"},
Requires: []string{" source ", "source", ""},
Key: " " + testCase.key + " ",
Stage: testCase.stage,
ExecutionClass: contracts.ExecutionClassDeterministic,
Provides: []string{" beta ", "alpha", "", "beta"},
Requires: []string{" source ", "source", ""},
}
if err := testCase.registerWithSpec(registry, spec, testCase.constructor(testCase.key)); err != nil {
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
@@ -115,10 +124,11 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
t.Fatal("Spec() ok = false, want true")
}
want := ModuleSpec{
Key: testCase.key,
Stage: testCase.stage,
Provides: []string{"alpha", "beta"},
Requires: []string{"source"},
Key: testCase.key,
Stage: testCase.stage,
ExecutionClass: contracts.ExecutionClassDeterministic,
Provides: []string{"alpha", "beta"},
Requires: []string{"source"},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("Spec() = %#v, want %#v", got, want)
@@ -134,7 +144,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
}
})
t.Run(testCase.name+"/default spec from register", func(t *testing.T) {
t.Run(testCase.name+"/minimal explicit spec registration", func(t *testing.T) {
registry := testCase.newRegistry()
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
@@ -144,7 +154,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
if !ok {
t.Fatal("Spec() ok = false, want true")
}
want := ModuleSpec{Key: testCase.key, Stage: testCase.stage}
want := ModuleSpec{Key: testCase.key, Stage: testCase.stage, ExecutionClass: contracts.ExecutionClassDeterministic}
if !reflect.DeepEqual(spec, want) {
t.Fatalf("Spec() = %#v, want %#v", spec, want)
}
@@ -152,7 +162,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
t.Run(testCase.name+"/wrong stage rejection", func(t *testing.T) {
registry := testCase.newRegistry()
err := testCase.registerWithSpec(registry, ModuleSpec{Key: testCase.key, Stage: testCase.wrongStage}, testCase.constructor(testCase.key))
err := testCase.registerWithSpec(registry, ModuleSpec{Key: testCase.key, Stage: testCase.wrongStage, ExecutionClass: contracts.ExecutionClassDeterministic}, testCase.constructor(testCase.key))
if err == nil {
t.Fatal("RegisterWithSpec() error = nil, want error")
}
@@ -161,6 +171,22 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
}
})
t.Run(testCase.name+"/execution class rejection", func(t *testing.T) {
for _, spec := range []ModuleSpec{
{Key: testCase.key, Stage: testCase.stage},
{Key: testCase.key, Stage: testCase.stage, ExecutionClass: contracts.ExecutionClass("unsupported")},
} {
registry := testCase.newRegistry()
err := testCase.registerWithSpec(registry, spec, testCase.constructor(testCase.key))
if err == nil {
t.Fatal("RegisterWithSpec() error = nil, want execution class error")
}
if !strings.Contains(err.Error(), "execution class") {
t.Fatalf("RegisterWithSpec() error = %q, want execution class error", err.Error())
}
}
})
t.Run(testCase.name+"/key trimming", func(t *testing.T) {
registry := testCase.newRegistry()
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {

View File

@@ -108,7 +108,7 @@ func registerTestEvidenceOutput(t *testing.T, registries *Registries, policy Evi
func registerTestEvidenceOutputWithProfileValidation(t *testing.T, registries *Registries, policy EvidenceContextPolicy, validateProfile OutputProfileOptionValidator) {
t.Helper()
registry := NewOutputEncoderRegistry()
if err := registry.RegisterBuilderWithProfileValidation(defaultModuleSpec("output", StageOutput), func(options map[string]any) error {
if err := registry.RegisterBuilderWithProfileValidation(testModuleSpec("output", StageOutput), func(options map[string]any) error {
return RejectUnknownOptions(options, "known")
}, validateProfile, func(BuildRequest) (contracts.OutputEncoder, error) {
return testEvidenceOutput{policy: cloneEvidenceContextPolicy(policy)}, nil

View File

@@ -24,10 +24,6 @@ func NewInputAdapterRegistry() *InputAdapterRegistry {
}
}
func (r *InputAdapterRegistry) Register(key string, constructor InputAdapterConstructor) error {
return r.RegisterWithSpec(defaultModuleSpec(key, StageInput), constructor)
}
func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor InputAdapterConstructor) error {
if constructor == nil {
return fmt.Errorf("input adapter constructor for %q must not be nil", strings.TrimSpace(spec.Key))

View File

@@ -11,10 +11,14 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func registerTestInput(registry *InputAdapterRegistry, key string, constructor InputAdapterConstructor) error {
return registry.RegisterWithSpec(testModuleSpec(key, StageInput), constructor)
}
func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) {
registry := NewInputAdapterRegistry()
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
@@ -30,7 +34,7 @@ func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) {
func TestInputAdapterRegistryRegisterAndBuildTrimKeys(t *testing.T) {
registry := NewInputAdapterRegistry()
if err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
if err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
@@ -46,10 +50,11 @@ func TestInputAdapterRegistryRegisterAndBuildTrimKeys(t *testing.T) {
func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
registry := NewInputAdapterRegistry()
spec := ModuleSpec{
Key: " generic-input ",
Stage: StageInput,
Provides: []string{" parsed-source ", "source-document", "parsed-source", ""},
Requires: []string{" raw-bytes ", "raw-bytes", ""},
Key: " generic-input ",
Stage: StageInput,
ExecutionClass: contracts.ExecutionClassDeterministic,
Provides: []string{" parsed-source ", "source-document", "parsed-source", ""},
Requires: []string{" raw-bytes ", "raw-bytes", ""},
}
if err := registry.RegisterWithSpec(spec, fakeInputAdapterConstructor("generic-input")); err != nil {
@@ -61,10 +66,11 @@ func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
t.Fatal("Spec() ok = false, want true")
}
want := ModuleSpec{
Key: "generic-input",
Stage: StageInput,
Provides: []string{"parsed-source", "source-document"},
Requires: []string{"raw-bytes"},
Key: "generic-input",
Stage: StageInput,
ExecutionClass: contracts.ExecutionClassDeterministic,
Provides: []string{"parsed-source", "source-document"},
Requires: []string{"raw-bytes"},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("Spec() = %#v, want %#v", got, want)
@@ -80,10 +86,10 @@ func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
}
}
func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
func TestInputAdapterRegistryRegisterWithSpecStoresMinimalMetadata(t *testing.T) {
registry := NewInputAdapterRegistry()
if err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
if err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
@@ -91,7 +97,7 @@ func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
if !ok {
t.Fatal("Spec() ok = false, want true")
}
want := ModuleSpec{Key: "generic-input", Stage: StageInput}
want := ModuleSpec{Key: "generic-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic}
if !reflect.DeepEqual(got, want) {
t.Fatalf("Spec() = %#v, want %#v", got, want)
}
@@ -100,7 +106,7 @@ func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
func TestInputAdapterRegistryRegisterWithSpecRejectsWrongStage(t *testing.T) {
registry := NewInputAdapterRegistry()
err := registry.RegisterWithSpec(ModuleSpec{Key: "generic-input", Stage: StageExtract}, fakeInputAdapterConstructor("generic-input"))
err := registry.RegisterWithSpec(ModuleSpec{Key: "generic-input", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic}, fakeInputAdapterConstructor("generic-input"))
if err == nil {
t.Fatal("RegisterWithSpec() error = nil, want error")
@@ -121,7 +127,7 @@ func TestInputAdapterRegistrySpecRejectsUnknownKey(t *testing.T) {
func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) {
registry := NewInputAdapterRegistry()
err := registry.Register(" \t", fakeInputAdapterConstructor("generic-input"))
err := registerTestInput(registry, " \t", fakeInputAdapterConstructor("generic-input"))
if err == nil {
t.Fatal("Register() error = nil, want error")
@@ -133,11 +139,11 @@ func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) {
func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) {
registry := NewInputAdapterRegistry()
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input"))
err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input"))
if err == nil {
t.Fatal("Register() error = nil, want error")
@@ -150,7 +156,7 @@ func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) {
func TestInputAdapterRegistryRegisterRejectsNilConstructor(t *testing.T) {
registry := NewInputAdapterRegistry()
err := registry.Register("generic-input", nil)
err := registerTestInput(registry, "generic-input", nil)
if err == nil {
t.Fatal("Register() error = nil, want error")
@@ -176,7 +182,7 @@ func TestInputAdapterRegistryBuildRejectsUnknownKey(t *testing.T) {
func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) {
registry := NewInputAdapterRegistry()
constructorErr := errors.New("constructor failed")
if err := registry.Register("generic-input", func() (contracts.InputAdapter, error) {
if err := registerTestInput(registry, "generic-input", func() (contracts.InputAdapter, error) {
return nil, constructorErr
}); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
@@ -197,7 +203,7 @@ func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) {
func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) {
registry := NewInputAdapterRegistry()
if err := registry.Register("generic-input", func() (contracts.InputAdapter, error) {
if err := registerTestInput(registry, "generic-input", func() (contracts.InputAdapter, error) {
return nil, nil
}); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
@@ -215,7 +221,7 @@ func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) {
func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) {
registry := NewInputAdapterRegistry()
if err := registry.Register("generic-input", fakeInputAdapterConstructor("other-input")); err != nil {
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("other-input")); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
@@ -232,7 +238,7 @@ func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) {
func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
registry := NewInputAdapterRegistry()
for _, key := range []string{"zeta", "alpha", "middle"} {
if err := registry.Register(key, fakeInputAdapterConstructor(key)); err != nil {
if err := registerTestInput(registry, key, fakeInputAdapterConstructor(key)); err != nil {
t.Fatalf("Register(%q) error = %v, want nil", key, err)
}
}
@@ -253,7 +259,7 @@ func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
func TestInputAdapterRegistryNilRegistryBehavior(t *testing.T) {
var registry *InputAdapterRegistry
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err == nil {
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err == nil {
t.Fatal("Register() error = nil, want error")
}
if _, err := registry.Build("generic-input"); err == nil {

View File

@@ -23,23 +23,19 @@ const (
type ModuleSpec struct {
Key string
Stage ModuleStage
ExecutionClass contracts.ExecutionClass
ArtifactKind contracts.ArtifactKind
Provides []string
Requires []string
ReferenceSlots []contracts.ReferenceSlot
}
func defaultModuleSpec(key string, stage ModuleStage) ModuleSpec {
return ModuleSpec{
Key: key,
Stage: stage,
}
}
func normalizeModuleSpec(spec ModuleSpec) ModuleSpec {
executionClass := contracts.ExecutionClass(strings.TrimSpace(string(spec.ExecutionClass)))
return ModuleSpec{
Key: strings.TrimSpace(spec.Key),
Stage: spec.Stage,
ExecutionClass: executionClass,
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
Provides: normalizeCapabilities(spec.Provides),
Requires: normalizeCapabilities(spec.Requires),
@@ -76,6 +72,7 @@ func cloneModuleSpec(spec ModuleSpec) ModuleSpec {
return ModuleSpec{
Key: spec.Key,
Stage: spec.Stage,
ExecutionClass: spec.ExecutionClass,
ArtifactKind: spec.ArtifactKind,
Provides: append([]string(nil), spec.Provides...),
Requires: append([]string(nil), spec.Requires...),
@@ -90,6 +87,12 @@ func validateModuleSpec(kind string, expectedStage ModuleStage, spec ModuleSpec)
if spec.Stage != expectedStage {
return fmt.Errorf("%s %q must use %q stage, got %q", kind, spec.Key, expectedStage, spec.Stage)
}
if spec.ExecutionClass == "" {
return fmt.Errorf("%s %q execution class must not be empty", kind, spec.Key)
}
if spec.ExecutionClass != contracts.ExecutionClassDeterministic && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
return fmt.Errorf("%s %q has unsupported execution class %q", kind, spec.Key, spec.ExecutionClass)
}
if spec.ArtifactKind != "" && spec.Stage != StageExtract && spec.Stage != StageMerge && spec.Stage != StageNormalize {
return fmt.Errorf("%s %q must not declare an artifact kind", kind, spec.Key)
}

View File

@@ -1,12 +1,45 @@
package pipeline
import (
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestValidateModuleSpecRequiresSupportedExecutionClass(t *testing.T) {
for _, test := range []struct {
name string
class contracts.ExecutionClass
want string
}{
{name: "missing", want: "execution class"},
{name: "unsupported", class: "remote", want: "unsupported"},
{name: "deterministic", class: contracts.ExecutionClassDeterministic},
{name: "llm backed", class: contracts.ExecutionClassLLMBacked},
} {
t.Run(test.name, func(t *testing.T) {
spec := normalizeModuleSpec(ModuleSpec{Key: "module", Stage: StageChunk, ExecutionClass: test.class})
err := validateModuleSpec("chunker", StageChunk, spec)
if test.want == "" && err != nil {
t.Fatalf("validateModuleSpec() error = %v, want nil", err)
}
if test.want != "" && (err == nil || !strings.Contains(err.Error(), test.want)) {
t.Fatalf("validateModuleSpec() error = %v, want %q", err, test.want)
}
})
}
}
func TestCloneModuleSpecPreservesExecutionClass(t *testing.T) {
spec := normalizeModuleSpec(ModuleSpec{Key: " module ", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked})
cloned := cloneModuleSpec(spec)
if !reflect.DeepEqual(cloned, spec) {
t.Fatalf("cloneModuleSpec() = %#v, want %#v", cloned, spec)
}
}
func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
tests := []struct {
name string
@@ -22,8 +55,9 @@ func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
spec := normalizeModuleSpec(ModuleSpec{
Key: "module",
Stage: test.stage,
Key: "module",
Stage: test.stage,
ExecutionClass: contracts.ExecutionClassDeterministic,
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster", Description: "Character roster", MaxBytes: 1024},
},
@@ -50,8 +84,9 @@ func TestValidateModuleSpecRejectsReferenceSlotsForIneligibleStages(t *testing.T
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
spec := normalizeModuleSpec(ModuleSpec{
Key: "module",
Stage: test.stage,
Key: "module",
Stage: test.stage,
ExecutionClass: contracts.ExecutionClassDeterministic,
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster"},
},
@@ -109,6 +144,7 @@ func TestValidateModuleSpecRejectsInvalidReferenceSlotsForEligibleStages(t *test
spec := normalizeModuleSpec(ModuleSpec{
Key: "module",
Stage: stage.stage,
ExecutionClass: contracts.ExecutionClassDeterministic,
ReferenceSlots: invalid.slots,
})
err := validateModuleSpec(stage.kind, stage.stage, spec)

View File

@@ -26,7 +26,7 @@ func TestNormalizerRegistryErasureClonesRetryDirective(t *testing.T) {
FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "omitted", Message: "fallback warning"}},
}
registry := NewNormalizerRegistry()
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
return retryingNotesNormalizer{warnings: warnings, retry: retry}, nil
}); err != nil {
t.Fatalf("RegisterNormalizer() error = %v", err)

View File

@@ -32,10 +32,6 @@ func NewOutputEncoderRegistry() *OutputEncoderRegistry {
}
}
func (r *OutputEncoderRegistry) Register(key string, constructor OutputEncoderConstructor) error {
return r.RegisterWithSpec(defaultModuleSpec(key, StageOutput), constructor)
}
func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor OutputEncoderConstructor) error {
if constructor == nil {
return fmt.Errorf("output encoder constructor for %q must not be nil", strings.TrimSpace(spec.Key))

View File

@@ -17,7 +17,9 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
return NewOutputEncoderRegistry()
},
register: func(registry any, key string, constructor func() (contracts.OutputEncoder, error)) error {
return registry.(*OutputEncoderRegistry).Register(key, constructor)
return registry.(*OutputEncoderRegistry).RegisterWithSpec(ModuleSpec{
Key: key, Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic,
}, constructor)
},
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.OutputEncoder, error)) error {
return registry.(*OutputEncoderRegistry).RegisterWithSpec(spec, constructor)
@@ -33,7 +35,9 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
},
nilRegister: func(key string, constructor func() (contracts.OutputEncoder, error)) error {
var registry *OutputEncoderRegistry
return registry.Register(key, constructor)
return registry.RegisterWithSpec(ModuleSpec{
Key: key, Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic,
}, constructor)
},
nilBuild: func(key string) (contracts.OutputEncoder, error) {
var registry *OutputEncoderRegistry
@@ -60,7 +64,7 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
func TestOutputProfileValidationReceivesOwnedOptionsAndLaneIDs(t *testing.T) {
registry := NewOutputEncoderRegistry()
if err := registry.RegisterBuilderWithProfileValidation(defaultModuleSpec("profile-output", StageOutput), func(options map[string]any) error {
if err := registry.RegisterBuilderWithProfileValidation(testModuleSpec("profile-output", StageOutput), func(options map[string]any) error {
options["nested"].(map[string]any)["value"] = "changed"
return nil
}, func(context OutputProfileOptionContext, options map[string]any) error {

View File

@@ -480,19 +480,19 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
if err := RegisterArtifactCodec(registries.ArtifactCodecs, notesCodec()); err != nil {
t.Fatal(err)
}
if err := registries.Inputs.RegisterBuilderWithSpec(defaultModuleSpec("input", StageInput), strict, func(request BuildRequest) (contracts.InputAdapter, error) {
if err := registries.Inputs.RegisterBuilderWithSpec(testModuleSpec("input", StageInput), strict, func(request BuildRequest) (contracts.InputAdapter, error) {
record("input", &request)
return input, nil
}); err != nil {
t.Fatal(err)
}
if err := registries.Chunkers.RegisterBuilderWithSpec(defaultModuleSpec("chunk", StageChunk), strict, func(request BuildRequest) (contracts.Chunker, error) {
if err := registries.Chunkers.RegisterBuilderWithSpec(testModuleSpec("chunk", StageChunk), strict, func(request BuildRequest) (contracts.Chunker, error) {
record("chunk", &request)
return &typedTestChunker{key: "chunk"}, nil
}); err != nil {
t.Fatal(err)
}
extractSpec := defaultModuleSpec("extract", StageExtract)
extractSpec := testModuleSpec("extract", StageExtract)
extractSpec.ArtifactKind = "test/notes"
if err := RegisterExtractorBuilder(registries.Extractors, extractSpec, strict, func(request BuildRequest) (contracts.Extractor[codecNotes], error) {
record("extract", &request)
@@ -503,7 +503,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
}); err != nil {
t.Fatal(err)
}
mergeSpec := defaultModuleSpec("merge", StageMerge)
mergeSpec := testModuleSpec("merge", StageMerge)
mergeSpec.ArtifactKind = "test/notes"
if err := RegisterMergerBuilder(registries.Mergers, mergeSpec, strict, func(request BuildRequest) (contracts.Merger[codecNotes], error) {
record("merge", &request)
@@ -511,7 +511,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
}); err != nil {
t.Fatal(err)
}
normalizeSpec := defaultModuleSpec("normalize", StageNormalize)
normalizeSpec := testModuleSpec("normalize", StageNormalize)
normalizeSpec.ArtifactKind = "test/notes"
if err := RegisterNormalizerBuilder(registries.Normalizers, normalizeSpec, strict, func(request BuildRequest) (contracts.Normalizer[codecNotes], error) {
record("normalize", &request)
@@ -532,7 +532,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
}); err != nil {
t.Fatal(err)
}
if err := registries.Outputs.RegisterBuilderWithSpec(defaultModuleSpec("output", StageOutput), strict, func(request BuildRequest) (contracts.OutputEncoder, error) {
if err := registries.Outputs.RegisterBuilderWithSpec(testModuleSpec("output", StageOutput), strict, func(request BuildRequest) (contracts.OutputEncoder, error) {
record("output", &request)
if failure.output != nil {
return nil, failure.output

View File

@@ -117,6 +117,7 @@ type PipelineStepProfile struct {
type PipelineProfile struct {
ID string `json:"id"`
LLMProfile string `json:"llm_profile,omitempty"`
Input ModuleBinding `json:"input"`
Chunk ModuleBinding `json:"chunk,omitempty"`
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
@@ -127,6 +128,7 @@ type PipelineProfile struct {
type ResolveOptions struct {
Only []string
LLMProfileOverride string
ReferenceOverrides []ReferenceBinding
ReferenceUnbinds []ReferenceUnbind
}
@@ -156,20 +158,23 @@ type ResolvedReferenceTarget struct {
}
type ResolvedArtifactLane struct {
StepID string
ID string
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
ArtifactSchemaName string `json:"artifact_schema_name,omitempty"`
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
Extract ModuleBinding
Merge ModuleBinding
Normalize ModuleBinding
Validators []ModuleBinding
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
StepID string
ID string
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
ArtifactSchemaName string `json:"artifact_schema_name,omitempty"`
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
Extract ModuleBinding
ExtractExecutionClass contracts.ExecutionClass `json:"extract_execution_class"`
Merge ModuleBinding
MergeExecutionClass contracts.ExecutionClass `json:"merge_execution_class"`
Normalize ModuleBinding
NormalizeExecutionClass contracts.ExecutionClass `json:"normalize_execution_class"`
Validators []ModuleBinding
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
}
type ResolvedPipelineStep struct {
@@ -192,14 +197,17 @@ type ResolvedValidator struct {
}
type ResolvedPipeline struct {
ID string
Digest string
Input ModuleBinding
Chunk ModuleBinding
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
Steps []ResolvedPipelineStep
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
Output ModuleBinding
ID string
Digest string
Input ModuleBinding
InputExecutionClass contracts.ExecutionClass `json:"input_execution_class"`
Chunk ModuleBinding
ChunkExecutionClass contracts.ExecutionClass `json:"chunk_execution_class"`
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
Steps []ResolvedPipelineStep
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
Output ModuleBinding
OutputExecutionClass contracts.ExecutionClass `json:"output_execution_class"`
}
// AllArtifactLanes returns lanes in deterministic step order for read-only
@@ -225,6 +233,46 @@ type ModuleCatalog struct {
Outputs *OutputEncoderRegistry
}
// ExecutionClass returns the registered execution class for a module selected
// by stage and key without constructing the module.
func (catalog ModuleCatalog) ExecutionClass(stage ModuleStage, key string) (contracts.ExecutionClass, bool) {
var executionClass contracts.ExecutionClass
var ok bool
switch stage {
case StageInput:
var spec ModuleSpec
spec, ok = catalog.Inputs.Spec(key)
executionClass = spec.ExecutionClass
case StageChunk:
var spec ModuleSpec
spec, ok = catalog.Chunkers.Spec(key)
executionClass = spec.ExecutionClass
case StageExtract:
var spec ModuleSpec
spec, ok = catalog.Extractors.Spec(key)
executionClass = spec.ExecutionClass
case StageMerge:
var spec ModuleSpec
spec, ok = catalog.Mergers.Spec(key)
executionClass = spec.ExecutionClass
case StageNormalize:
var spec ModuleSpec
spec, ok = catalog.Normalizers.Spec(key)
executionClass = spec.ExecutionClass
case StageValidate:
var spec ValidatorSpec
spec, ok = catalog.Validators.Spec(key)
executionClass = spec.ExecutionClass
case StageOutput:
var spec ModuleSpec
spec, ok = catalog.Outputs.Spec(key)
executionClass = spec.ExecutionClass
}
return executionClass, ok
}
func Binding(module string) ModuleBinding {
return ModuleBinding{Module: strings.TrimSpace(module)}
}
@@ -316,11 +364,13 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
return ResolvedPipeline{}, err
}
resolved := ResolvedPipeline{
ID: pipelineID,
Input: input,
Chunk: chunk,
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
Output: resolveBinding(profile.Output, DefaultOutputModule),
ID: pipelineID,
Input: input,
InputExecutionClass: inputModuleSpec.ExecutionClass,
Chunk: chunk,
ChunkExecutionClass: chunkSpec.ExecutionClass,
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
Output: resolveBinding(profile.Output, DefaultOutputModule),
}
chunkValidatorChain, err := resolveValidatorChain(pipelineID, "", StageChunk, chunk.Module, chunk.Validators, "", nil, catalog)
if err != nil {
@@ -386,6 +436,10 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
if missing, ok := outputCapabilities.missing(outputSpec.Requires); ok {
return ResolvedPipeline{}, capabilityError(pipelineID, "", StageOutput, resolved.Output.Module, missing)
}
resolved.OutputExecutionClass = outputSpec.ExecutionClass
if err := applyEffectiveLLMProfiles(&resolved, profile.LLMProfile, options.LLMProfileOverride); err != nil {
return ResolvedPipeline{}, err
}
if err := validateResolvedOptions(resolved, catalog, configuredLaneIDs); err != nil {
return ResolvedPipeline{}, err
}
@@ -475,6 +529,7 @@ func resolveArtifactLane(
}
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
lane.ExtractReferences.StepID = strings.TrimSpace(stepID)
lane.ExtractExecutionClass = extractSpec.ExecutionClass
capabilities.add(extractSpec.Provides...)
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
@@ -499,6 +554,7 @@ func resolveArtifactLane(
}
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
lane.MergeReferences.StepID = strings.TrimSpace(stepID)
lane.MergeExecutionClass = mergeSpec.ExecutionClass
capabilities.add(mergeSpec.Provides...)
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
@@ -523,6 +579,7 @@ func resolveArtifactLane(
}
lane.NormalizeReferences = referenceTarget(StageNormalize, laneID, lane.Normalize.Module, normalizeReferences)
lane.NormalizeReferences.StepID = strings.TrimSpace(stepID)
lane.NormalizeExecutionClass = normalizeSpec.ExecutionClass
capabilities.add(normalizeSpec.Provides...)
if len(lane.Validators) > 0 {
@@ -784,9 +841,6 @@ func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage,
if err != nil {
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q: %w", pipelineID, stage, chain.ModuleKey, err)
}
if strings.TrimSpace(validator.LLMProfile) != "" && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q assigns llm_profile to deterministic validator %q", pipelineID, stage, chain.ModuleKey, validator.Module)
}
chain.Validators = append(chain.Validators, ResolvedValidator{
Binding: cloneModuleBinding(validator),
ExecutionClass: spec.ExecutionClass,
@@ -1198,6 +1252,66 @@ func resolveBinding(binding ModuleBinding, defaultModule string) ModuleBinding {
}
}
func applyEffectiveLLMProfiles(resolved *ResolvedPipeline, pipelineProfile, overrideProfile string) error {
pipelineProfile = strings.TrimSpace(pipelineProfile)
overrideProfile = strings.TrimSpace(overrideProfile)
apply := func(stage ModuleStage, laneID, module string, binding *ModuleBinding, executionClass contracts.ExecutionClass, kind string) error {
binding.LLMProfile = strings.TrimSpace(binding.LLMProfile)
if executionClass != contracts.ExecutionClassLLMBacked {
if binding.LLMProfile != "" {
if laneID == "" {
return fmt.Errorf("pipeline %q %s %q assigns llm_profile to deterministic %s %q", resolved.ID, stage, module, kind, binding.Module)
}
return fmt.Errorf("pipeline %q lane %q %s %q assigns llm_profile to deterministic %s %q", resolved.ID, laneID, stage, module, kind, binding.Module)
}
return nil
}
if overrideProfile != "" {
binding.LLMProfile = overrideProfile
return nil
}
if binding.LLMProfile == "" {
binding.LLMProfile = pipelineProfile
}
return nil
}
if err := apply(StageInput, "", resolved.Input.Module, &resolved.Input, resolved.InputExecutionClass, "module"); err != nil {
return err
}
if err := apply(StageChunk, "", resolved.Chunk.Module, &resolved.Chunk, resolved.ChunkExecutionClass, "module"); err != nil {
return err
}
for stepIndex := range resolved.Steps {
for laneIndex := range resolved.Steps[stepIndex].ArtifactLanes {
lane := &resolved.Steps[stepIndex].ArtifactLanes[laneIndex]
if err := apply(StageExtract, lane.ID, lane.Extract.Module, &lane.Extract, lane.ExtractExecutionClass, "module"); err != nil {
return err
}
if err := apply(StageMerge, lane.ID, lane.Merge.Module, &lane.Merge, lane.MergeExecutionClass, "module"); err != nil {
return err
}
if err := apply(StageNormalize, lane.ID, lane.Normalize.Module, &lane.Normalize, lane.NormalizeExecutionClass, "module"); err != nil {
return err
}
}
}
if err := apply(StageOutput, "", resolved.Output.Module, &resolved.Output, resolved.OutputExecutionClass, "module"); err != nil {
return err
}
for chainIndex := range resolved.ValidatorChains {
chain := &resolved.ValidatorChains[chainIndex]
for validatorIndex := range chain.Validators {
validator := &chain.Validators[validatorIndex]
if err := apply(chain.Stage, chain.LaneID, chain.ModuleKey, &validator.Binding, validator.ExecutionClass, "validator"); err != nil {
return err
}
}
}
return nil
}
func resolveBindings(bindings []ModuleBinding, defaultModule string) []ModuleBinding {
if len(bindings) == 0 {
return nil
@@ -1307,21 +1421,27 @@ func selectedArtifactLanes(pipelineID string, artifacts map[string]ArtifactLaneP
func resolvedPipelineDigest(resolved ResolvedPipeline) (string, error) {
withoutDigest := struct {
ID string
Input ModuleBinding
Chunk ModuleBinding
ChunkReferences ResolvedReferenceTarget
Steps []ResolvedPipelineStep
ValidatorChains []ResolvedValidatorChain
Output ModuleBinding
ID string
Input ModuleBinding
InputExecutionClass contracts.ExecutionClass
Chunk ModuleBinding
ChunkExecutionClass contracts.ExecutionClass
ChunkReferences ResolvedReferenceTarget
Steps []ResolvedPipelineStep
ValidatorChains []ResolvedValidatorChain
Output ModuleBinding
OutputExecutionClass contracts.ExecutionClass
}{
ID: resolved.ID,
Input: resolved.Input,
Chunk: resolved.Chunk,
ChunkReferences: resolved.ChunkReferences,
Steps: resolved.Steps,
ValidatorChains: resolved.ValidatorChains,
Output: resolved.Output,
ID: resolved.ID,
Input: resolved.Input,
InputExecutionClass: resolved.InputExecutionClass,
Chunk: resolved.Chunk,
ChunkExecutionClass: resolved.ChunkExecutionClass,
ChunkReferences: resolved.ChunkReferences,
Steps: resolved.Steps,
ValidatorChains: resolved.ValidatorChains,
Output: resolved.Output,
OutputExecutionClass: resolved.OutputExecutionClass,
}
encoded, err := json.Marshal(withoutDigest)
if err != nil {

View File

@@ -15,22 +15,22 @@ import (
func TestResolvePipelineWithExplicitModules(t *testing.T) {
catalog := newProfileCatalog(t)
registerProfileSpecs(t, catalog,
ModuleSpec{Key: "window", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}},
ModuleSpec{Key: "record-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "dedupe", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}},
ModuleSpec{Key: "canonical", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}},
ModuleSpec{Key: "ndjson", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
ModuleSpec{Key: "window", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}},
ModuleSpec{Key: "record-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "dedupe", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"candidate"}, Provides: []string{"merged"}},
ModuleSpec{Key: "canonical", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}},
ModuleSpec{Key: "ndjson", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
)
resolved, err := ResolvePipeline(PipelineProfile{
ID: " campaign ",
Input: ModuleBinding{Module: " text ", LLMProfile: " fast "},
Input: Binding(" text "),
Chunk: ModuleBinding{Module: " window ", Options: map[string]any{
"size": 10,
}},
Artifacts: map[string]ArtifactLaneProfile{
" records ": {
Extract: ModuleBinding{Module: " record-extractor ", LLMProfile: " careful "},
Extract: Binding(" record-extractor "),
Merge: Binding(" dedupe "),
Normalize: Binding(" canonical "),
},
@@ -44,15 +44,21 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
if resolved.ID != "campaign" {
t.Fatalf("ID = %q, want campaign", resolved.ID)
}
if !reflect.DeepEqual(resolved.Input, ModuleBinding{Module: "text", LLMProfile: "fast"}) {
if !reflect.DeepEqual(resolved.Input, ModuleBinding{Module: "text"}) {
t.Fatalf("Input = %#v, want trimmed explicit input", resolved.Input)
}
if resolved.InputExecutionClass != contracts.ExecutionClassDeterministic {
t.Fatalf("InputExecutionClass = %q, want deterministic", resolved.InputExecutionClass)
}
if resolved.Chunk.Module != "window" || resolved.Chunk.LLMProfile != "" {
t.Fatalf("Chunk = %#v, want explicit module and empty LLM profile", resolved.Chunk)
}
if resolved.Chunk.Options["size"] != 10 {
t.Fatalf("Chunk.Options = %#v, want size option", resolved.Chunk.Options)
}
if resolved.ChunkExecutionClass != contracts.ExecutionClassDeterministic {
t.Fatalf("ChunkExecutionClass = %q, want deterministic", resolved.ChunkExecutionClass)
}
if len(resolved.Steps) != 1 || resolved.Steps[0].ID != "default" || len(resolved.Steps[0].ArtifactLanes) != 1 {
t.Fatalf("resolved steps = %#v, want one default step with one lane", resolved.Steps)
}
@@ -60,9 +66,12 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
if lane.ID != "records" {
t.Fatalf("lane.ID = %q, want records", lane.ID)
}
if !reflect.DeepEqual(lane.Extract, ModuleBinding{Module: "record-extractor", LLMProfile: "careful"}) {
if !reflect.DeepEqual(lane.Extract, ModuleBinding{Module: "record-extractor"}) {
t.Fatalf("lane.Extract = %#v, want explicit extractor", lane.Extract)
}
if lane.ExtractExecutionClass != contracts.ExecutionClassDeterministic || lane.MergeExecutionClass != contracts.ExecutionClassDeterministic || lane.NormalizeExecutionClass != contracts.ExecutionClassDeterministic {
t.Fatalf("lane execution classes = %q/%q/%q, want deterministic", lane.ExtractExecutionClass, lane.MergeExecutionClass, lane.NormalizeExecutionClass)
}
if lane.Merge.Module != "dedupe" || lane.Normalize.Module != "canonical" {
t.Fatalf("lane merge/normalize = %#v/%#v, want explicit modules", lane.Merge, lane.Normalize)
}
@@ -72,11 +81,51 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
if resolved.Output.Module != "ndjson" {
t.Fatalf("Output.Module = %q, want ndjson", resolved.Output.Module)
}
if resolved.OutputExecutionClass != contracts.ExecutionClassDeterministic {
t.Fatalf("OutputExecutionClass = %q, want deterministic", resolved.OutputExecutionClass)
}
if !strings.HasPrefix(resolved.Digest, "sha256:") {
t.Fatalf("Digest = %q, want sha256 digest", resolved.Digest)
}
}
func TestModuleCatalogExecutionClassLooksUpRegisteredMetadata(t *testing.T) {
catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked},
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked},
ModuleSpec{Key: "llm-extract", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked},
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked},
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked},
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked},
)
registerProfileValidatorSpec(t, catalog, ValidatorSpec{Key: "llm-validator", ExecutionClass: contracts.ExecutionClassLLMBacked})
for _, test := range []struct {
stage ModuleStage
key string
want contracts.ExecutionClass
}{
{stage: StageInput, key: "llm-input", want: contracts.ExecutionClassLLMBacked},
{stage: StageChunk, key: "llm-chunk", want: contracts.ExecutionClassLLMBacked},
{stage: StageExtract, key: "llm-extract", want: contracts.ExecutionClassLLMBacked},
{stage: StageMerge, key: "llm-merge", want: contracts.ExecutionClassLLMBacked},
{stage: StageNormalize, key: "llm-normalize", want: contracts.ExecutionClassLLMBacked},
{stage: StageValidate, key: "llm-validator", want: contracts.ExecutionClassLLMBacked},
{stage: StageOutput, key: "llm-output", want: contracts.ExecutionClassLLMBacked},
} {
t.Run(string(test.stage), func(t *testing.T) {
got, ok := catalog.ExecutionClass(test.stage, test.key)
if !ok || got != test.want {
t.Fatalf("ExecutionClass(%q, %q) = %q, %t; want %q, true", test.stage, test.key, got, ok, test.want)
}
})
}
if _, ok := catalog.ExecutionClass(StageExtract, "missing"); ok {
t.Fatal("ExecutionClass() found an unregistered module")
}
}
func TestResolvePipelineAppliesDefaults(t *testing.T) {
resolved, err := ResolvePipeline(PipelineProfile{
ID: "defaulted",
@@ -110,6 +159,176 @@ func TestResolvePipelineAppliesDefaults(t *testing.T) {
}
}
func TestResolvePipelineAppliesEffectiveLLMProfiles(t *testing.T) {
for _, test := range []struct {
name string
profile PipelineProfile
options ResolveOptions
want map[string]string
}{
{
name: "runtime override",
profile: func() PipelineProfile {
profile := llmProfilePipeline()
profile.LLMProfile = " pipeline "
profile.Chunk.LLMProfile = "binding"
return profile
}(),
options: ResolveOptions{LLMProfileOverride: " runtime "},
want: llmProfileValues("runtime"),
},
{
name: "binding exception",
profile: func() PipelineProfile {
profile := llmProfilePipeline()
profile.LLMProfile = "pipeline"
lane := profile.Artifacts["events"]
lane.Extract.LLMProfile = "extract"
lane.Extract.Validators = ValidatorOverride{
Set: true,
Validators: []ModuleBinding{{Module: "llm-validator", LLMProfile: "validator"}},
}
profile.Artifacts["events"] = lane
return profile
}(),
want: func() map[string]string {
values := llmProfileValues("pipeline")
values["extract"] = "extract"
values["validator:extract:events"] = "validator"
return values
}(),
},
{
name: "pipeline default",
profile: func() PipelineProfile {
profile := llmProfilePipeline()
profile.LLMProfile = "pipeline"
return profile
}(),
want: llmProfileValues("pipeline"),
},
{
name: "prompt fallback",
profile: llmProfilePipeline(),
want: llmProfileValues(""),
},
} {
t.Run(test.name, func(t *testing.T) {
resolved, err := ResolvePipeline(test.profile, test.options, llmProfileCatalog(t))
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
if got := resolvedLLMProfileValues(resolved); !reflect.DeepEqual(got, test.want) {
t.Fatalf("resolved profiles = %#v, want %#v", got, test.want)
}
})
}
}
func TestResolvePipelineAppliesProfilesOnlyToSelectedLLMBackedBindings(t *testing.T) {
profile := llmProfilePipeline()
profile.LLMProfile = "pipeline"
profile.Artifacts["notes"] = ArtifactLaneProfile{Extract: Binding("llm-extractor")}
resolved, err := ResolvePipeline(profile, ResolveOptions{Only: []string{"events"}}, llmProfileCatalog(t))
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
if got := laneIDs(resolved.Steps[0].ArtifactLanes); !reflect.DeepEqual(got, []string{"events"}) {
t.Fatalf("selected lanes = %#v, want events only", got)
}
if got := resolvedLLMProfileValues(resolved); !reflect.DeepEqual(got, llmProfileValues("pipeline")) {
t.Fatalf("resolved profiles = %#v, want selected LLM bindings only", got)
}
}
func TestResolvePipelineLeavesUnusedProfilesOffDeterministicBindings(t *testing.T) {
profile := baselineProfile()
profile.LLMProfile = "unused"
resolved, err := ResolvePipeline(profile, ResolveOptions{LLMProfileOverride: "also-unused"}, newProfileCatalog(t))
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
if resolved.Input.LLMProfile != "" || resolved.Chunk.LLMProfile != "" || resolved.Output.LLMProfile != "" {
t.Fatalf("deterministic pipeline profiles = input %q chunk %q output %q, want empty", resolved.Input.LLMProfile, resolved.Chunk.LLMProfile, resolved.Output.LLMProfile)
}
lane := resolved.Steps[0].ArtifactLanes[0]
if lane.Extract.LLMProfile != "" || lane.Merge.LLMProfile != "" || lane.Normalize.LLMProfile != "" {
t.Fatalf("deterministic lane profiles = extract %q merge %q normalize %q, want empty", lane.Extract.LLMProfile, lane.Merge.LLMProfile, lane.Normalize.LLMProfile)
}
}
func TestResolvePipelineRejectsLLMProfileForDeterministicModule(t *testing.T) {
for _, test := range []struct {
name string
mutate func(*PipelineProfile)
}{
{name: "input", mutate: func(profile *PipelineProfile) { profile.Input.LLMProfile = "invalid" }},
{name: "chunk", mutate: func(profile *PipelineProfile) { profile.Chunk.LLMProfile = "invalid" }},
{name: "extract", mutate: func(profile *PipelineProfile) {
lane := profile.Artifacts["events"]
lane.Extract.LLMProfile = "invalid"
profile.Artifacts["events"] = lane
}},
{name: "merge", mutate: func(profile *PipelineProfile) {
lane := profile.Artifacts["events"]
lane.Merge.LLMProfile = "invalid"
profile.Artifacts["events"] = lane
}},
{name: "normalize", mutate: func(profile *PipelineProfile) {
lane := profile.Artifacts["events"]
lane.Normalize.LLMProfile = "invalid"
profile.Artifacts["events"] = lane
}},
{name: "output", mutate: func(profile *PipelineProfile) { profile.Output.LLMProfile = "invalid" }},
} {
t.Run(test.name, func(t *testing.T) {
profile := baselineProfile()
test.mutate(&profile)
_, err := ResolvePipeline(profile, ResolveOptions{}, newProfileCatalog(t))
if err == nil || !strings.Contains(err.Error(), "llm_profile") || !strings.Contains(err.Error(), "deterministic") {
t.Fatalf("ResolvePipeline() error = %v, want deterministic profile rejection", err)
}
})
}
}
func TestResolvePipelineDigestUsesEffectiveLLMProfiles(t *testing.T) {
inherited := llmProfilePipeline()
inherited.LLMProfile = "shared"
explicit := llmProfilePipeline()
explicit.Input.LLMProfile = "shared"
explicit.Chunk.LLMProfile = "shared"
explicit.Output.LLMProfile = "shared"
lane := explicit.Artifacts["events"]
lane.Extract.LLMProfile = "shared"
lane.Merge.LLMProfile = "shared"
lane.Normalize.LLMProfile = "shared"
explicit.Artifacts["events"] = lane
inheritedResolved, err := ResolvePipeline(inherited, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
if err != nil {
t.Fatalf("ResolvePipeline(inherited) error = %v, want nil", err)
}
explicitResolved, err := ResolvePipeline(explicit, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
if err != nil {
t.Fatalf("ResolvePipeline(explicit) error = %v, want nil", err)
}
if inheritedResolved.Digest != explicitResolved.Digest {
t.Fatalf("effective profile digests differ: %q != %q", inheritedResolved.Digest, explicitResolved.Digest)
}
explicit.Chunk.LLMProfile = "different"
changedResolved, err := ResolvePipeline(explicit, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
if err != nil {
t.Fatalf("ResolvePipeline(changed) error = %v, want nil", err)
}
if explicitResolved.Digest == changedResolved.Digest {
t.Fatalf("digest = %q after effective profile change, want different", explicitResolved.Digest)
}
}
func TestResolvePipelineRecordsValidatorChains(t *testing.T) {
catalog := newProfileCatalog(t)
if err := catalog.ValidatorChains.Register(ValidatorChainMapping{
@@ -337,7 +556,8 @@ func TestResolvePipelinePreservesOrderedStepsAndExpandsGeneratedBindings(t *test
catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{
Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes",
Requires: []string{"chunk"}, Provides: []string{"candidate"},
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"}, Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}, AcceptedMediaTypes: []string{"application/json"}}},
})
resolved, err := ResolvePipeline(PipelineProfile{
@@ -365,7 +585,7 @@ func TestResolvePipelinePreservesOrderedStepsAndExpandsGeneratedBindings(t *test
func TestResolvePipelineRejectsGeneratedBindingOrderingAndKind(t *testing.T) {
catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/other"}}}},
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/other"}}}},
)
_, err := ResolvePipeline(PipelineProfile{
ID: "invalid-order", Input: Binding("text"), Steps: []PipelineStepProfile{
@@ -400,10 +620,11 @@ func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
profile.Artifacts["events"] = lane
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
Key: "event-extractor",
Stage: StageExtract,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
Key: "event-extractor",
Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster", Required: true},
{Name: "lore"},
@@ -446,6 +667,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToChunkTarget(t *testing.
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "generic",
Stage: StageChunk,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"source"},
Provides: []string{"chunk"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide"}},
@@ -474,6 +696,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToExtractorTarget(t *test
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "event-extractor",
Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
@@ -502,6 +725,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToNormalizerTarget(t *tes
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "noop",
Stage: StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"merged"},
Provides: []string{"normalized"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
@@ -530,6 +754,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToMergeTarget(t *testing.
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "appendorder",
Stage: StageMerge,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"candidate"},
Provides: []string{"merged"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "merge_notes"}},
@@ -559,10 +784,10 @@ func TestResolvePipelineAppliesOnePipelineReferenceDefaultToMultipleTargets(t *t
profile := baselineProfile()
profile.References = ExternalReferenceMap(map[string]string{"context": "./context.md"})
catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"candidate"}, Provides: []string{"merged"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
)
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
@@ -580,10 +805,11 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedLane(t *t
profile := multiLaneProfile()
profile.References = ExternalReferenceMap(map[string]string{"notes_context": "./notes.md"})
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
Key: "note-extractor",
Stage: StageExtract,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
Key: "note-extractor",
Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "notes_context"},
},
@@ -605,10 +831,11 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedNormalize
lane.Normalize = Binding("note-normalizer")
profile.Artifacts["notes"] = lane
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "note-normalizer",
Stage: StageNormalize,
Requires: []string{"merged"},
Provides: []string{"normalized"},
Key: "note-normalizer",
Stage: StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"merged"},
Provides: []string{"normalized"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "notes_context"},
},
@@ -658,6 +885,7 @@ func TestResolvePipelineRejectsExtractLocalReferenceDeclaredOnlyByNormalizer(t *
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "noop",
Stage: StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"merged"},
Provides: []string{"normalized"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
@@ -676,10 +904,11 @@ func TestResolvePipelineRejectsMergeLocalReferenceDeclaredOnlyByNormalizer(t *te
lane.Merge.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./normalize.md"})
profile.Artifacts["events"] = lane
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "noop",
Stage: StageNormalize,
Requires: []string{"merged"},
Provides: []string{"normalized"},
Key: "noop",
Stage: StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"merged"},
Provides: []string{"normalized"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "normalization_notes"},
},
@@ -700,6 +929,7 @@ func TestResolvePipelineRejectsNormalizeLocalReferenceDeclaredOnlyByExtractor(t
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "event-extractor",
Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
@@ -716,6 +946,7 @@ func TestResolvePipelineRequiresBoundChunkReference(t *testing.T) {
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "generic",
Stage: StageChunk,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"source"},
Provides: []string{"chunk"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide", Required: true}},
@@ -732,6 +963,7 @@ func TestResolvePipelineRequiresBoundNormalizeReference(t *testing.T) {
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "noop",
Stage: StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"merged"},
Provides: []string{"normalized"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes", Required: true}},
@@ -757,9 +989,9 @@ func TestResolvePipelineLocalReferencesOverridePipelineDefaultsForEligibleTarget
lane.Normalize.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./local-normalize.md"})
profile.Artifacts["events"] = lane
catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}}},
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}}},
ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}}},
ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}}},
)
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
@@ -774,10 +1006,11 @@ func TestResolvePipelineLocalReferencesOverridePipelineDefaultsForEligibleTarget
func TestResolvePipelineRequiresBoundReferenceSlotsForSelectedLanes(t *testing.T) {
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
Key: "event-extractor",
Stage: StageExtract,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
Key: "event-extractor",
Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster", Required: true},
},
@@ -798,10 +1031,11 @@ func TestResolvePipelineReferenceUnbindCanLeaveRequiredSlotMissing(t *testing.T)
profile := baselineProfile()
profile.References = ExternalReferenceMap(map[string]string{"roster": "./roster.yml"})
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
Key: "event-extractor",
Stage: StageExtract,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
Key: "event-extractor",
Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster", Required: true},
},
@@ -827,11 +1061,12 @@ func TestResolvePipelineUsesReferenceSlotsFromSpecWithoutConstructingExtractor(t
}
}
if err := RegisterExtractor[codecNotes](catalog.Extractors, ModuleSpec{
Key: "event-extractor",
Stage: StageExtract,
ArtifactKind: "test/notes",
Requires: []string{"chunk"},
Provides: []string{"candidate"},
Key: "event-extractor",
Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
ArtifactKind: "test/notes",
Requires: []string{"chunk"},
Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster", Required: true},
},
@@ -989,32 +1224,32 @@ func TestResolvePipelineRejectsMissingCapabilities(t *testing.T) {
}{
{
name: "input",
spec: ModuleSpec{Key: "text", Stage: StageInput, Requires: []string{"raw"}},
spec: ModuleSpec{Key: "text", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"raw"}},
want: []string{"baseline", "input", "text", "raw"},
},
{
name: "chunk",
spec: ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"missing"}},
spec: ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
want: []string{"baseline", "chunk", "generic", "missing"},
},
{
name: "extract",
spec: ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"missing"}},
spec: ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
want: []string{"baseline", "events", "extract", "event-extractor", "missing"},
},
{
name: "merge",
spec: ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"missing"}},
spec: ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
want: []string{"baseline", "events", "merge", "appendorder", "missing"},
},
{
name: "normalize",
spec: ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"missing"}},
spec: ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
want: []string{"baseline", "events", "normalize", "noop", "missing"},
},
{
name: "output",
spec: ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"missing"}},
spec: ModuleSpec{Key: "json", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
want: []string{"baseline", "output", "json", "missing"},
},
}
@@ -1100,7 +1335,7 @@ func TestResolvePipelineDigestChangesWhenBindingChanges(t *testing.T) {
right := baselineProfile()
right.Chunk = Binding("window")
catalog := newProfileCatalog(t)
registerProfileSpecs(t, catalog, ModuleSpec{Key: "window", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}})
registerProfileSpecs(t, catalog, ModuleSpec{Key: "window", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}})
leftResolved, err := ResolvePipeline(left, ResolveOptions{}, catalog)
if err != nil {
@@ -1467,17 +1702,103 @@ func emptyProfileCatalog() ModuleCatalog {
func defaultProfileSpecs() []ModuleSpec {
return []ModuleSpec{
ModuleSpec{Key: "text", Stage: StageInput, Provides: []string{"source"}},
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}},
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "appendorder", Stage: StageMerge, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
ModuleSpec{Key: "noop", Stage: StageNormalize, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
ModuleSpec{Key: "grounded", Stage: StageValidate, Requires: []string{"normalized"}, Provides: []string{"validated"}},
ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
ModuleSpec{Key: "text", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}},
ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}},
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
ModuleSpec{Key: "grounded", Stage: StageValidate, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"validated"}},
ModuleSpec{Key: "json", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
}
}
func llmProfileCatalog(t *testing.T) ModuleCatalog {
t.Helper()
catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked, Provides: []string{"source"}},
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"source"}, Provides: []string{"chunk"}},
ModuleSpec{Key: "llm-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
)
if err := RegisterChunkValidator(catalog.Validators, ValidatorSpec{Key: "llm-chunk-validator", ExecutionClass: contracts.ExecutionClassLLMBacked}, func() (contracts.ChunkValidator, error) {
return typedTestChunkValidator{key: "llm-chunk-validator"}, nil
}); err != nil {
t.Fatalf("register chunk validator: %v", err)
}
registerProfileValidatorSpec(t, catalog, ValidatorSpec{Key: "llm-validator", ExecutionClass: contracts.ExecutionClassLLMBacked})
for _, mapping := range []ValidatorChainMapping{
{Stage: StageChunk, Module: "llm-chunk", Validators: []ModuleBinding{Binding("llm-chunk-validator")}},
{Stage: StageExtract, Module: "llm-extractor", Validators: []ModuleBinding{Binding("llm-validator")}},
{Stage: StageMerge, Module: "llm-merge", Validators: []ModuleBinding{Binding("llm-validator")}},
{Stage: StageNormalize, Module: "llm-normalize", Validators: []ModuleBinding{Binding("llm-validator")}},
} {
if err := catalog.ValidatorChains.Register(mapping); err != nil {
t.Fatalf("register validator chain %#v: %v", mapping, err)
}
}
return catalog
}
func llmProfileCatalogWithoutValidatorChains(t *testing.T) ModuleCatalog {
catalog := llmProfileCatalog(t)
catalog.ValidatorChains = NewValidatorChainRegistry()
return catalog
}
func llmProfilePipeline() PipelineProfile {
return PipelineProfile{
ID: "llm-profile",
Input: Binding("llm-input"),
Chunk: Binding("llm-chunk"),
Output: Binding("llm-output"),
Artifacts: map[string]ArtifactLaneProfile{
"events": {
Extract: Binding("llm-extractor"),
Merge: Binding("llm-merge"),
Normalize: Binding("llm-normalize"),
},
},
}
}
func llmProfileValues(profile string) map[string]string {
return map[string]string{
"input": profile,
"chunk": profile,
"extract": profile,
"merge": profile,
"normalize": profile,
"output": profile,
"validator:chunk:": profile,
"validator:extract:events": profile,
"validator:merge:events": profile,
"validator:normalize:events": profile,
}
}
func resolvedLLMProfileValues(resolved ResolvedPipeline) map[string]string {
values := map[string]string{
"input": resolved.Input.LLMProfile,
"chunk": resolved.Chunk.LLMProfile,
"output": resolved.Output.LLMProfile,
}
lane := resolved.Steps[0].ArtifactLanes[0]
values["extract"] = lane.Extract.LLMProfile
values["merge"] = lane.Merge.LLMProfile
values["normalize"] = lane.Normalize.LLMProfile
for _, chain := range resolved.ValidatorChains {
for _, validator := range chain.Validators {
if validator.ExecutionClass == contracts.ExecutionClassLLMBacked {
values["validator:"+string(chain.Stage)+":"+chain.LaneID] = validator.Binding.LLMProfile
}
}
}
return values
}
func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSpec) {
t.Helper()
@@ -1566,7 +1887,13 @@ func TestResolvedPipelineCanMarshalToCanonicalJSON(t *testing.T) {
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
if _, err := json.Marshal(resolved); err != nil {
encoded, err := json.Marshal(resolved)
if err != nil {
t.Fatalf("json.Marshal(resolved) error = %v, want nil", err)
}
for _, field := range []string{"input_execution_class", "chunk_execution_class", "extract_execution_class", "merge_execution_class", "normalize_execution_class", "output_execution_class"} {
if !strings.Contains(string(encoded), `"`+field+`":"deterministic"`) {
t.Fatalf("resolved JSON does not retain %q: %s", field, encoded)
}
}
}

View File

@@ -189,12 +189,12 @@ func TestMaterializeReferencesUsesLaneArtifactVariant(t *testing.T) {
{kind: "test/beta", mergeSlot: "beta_merge", normalizeSlot: "beta_normalize"},
{kind: "test/alpha", mergeSlot: "alpha_merge", normalizeSlot: "alpha_normalize"},
} {
if err := RegisterMerger(mergers, ModuleSpec{Key: "shared/merge", Stage: StageMerge, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.mergeSlot}}}, func() (contracts.Merger[codecNotes], error) {
if err := RegisterMerger(mergers, ModuleSpec{Key: "shared/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.mergeSlot}}}, func() (contracts.Merger[codecNotes], error) {
return typedTestMerger[codecNotes]{key: "shared/merge"}, nil
}); err != nil {
t.Fatalf("RegisterMerger(%s): %v", item.kind, err)
}
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "shared/normalize", Stage: StageNormalize, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.normalizeSlot}}}, func() (contracts.Normalizer[codecNotes], error) {
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "shared/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.normalizeSlot}}}, func() (contracts.Normalizer[codecNotes], error) {
return typedTestNormalizer[codecNotes]{key: "shared/normalize"}, nil
}); err != nil {
t.Fatalf("RegisterNormalizer(%s): %v", item.kind, err)
@@ -531,6 +531,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
ModuleSpec{
Key: "generic",
Stage: StageChunk,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"source"},
Provides: []string{"chunk"},
ReferenceSlots: chunkSlots,
@@ -538,6 +539,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
ModuleSpec{
Key: "event-extractor",
Stage: StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
ReferenceSlots: extractSlots,
@@ -545,6 +547,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
ModuleSpec{
Key: "appendorder",
Stage: StageMerge,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"candidate"},
Provides: []string{"merged"},
ReferenceSlots: mergeSlots,
@@ -552,6 +555,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
ModuleSpec{
Key: "noop",
Stage: StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"merged"},
Provides: []string{"normalized"},
ReferenceSlots: normalizeSlots,

View File

@@ -116,7 +116,7 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
return false, nil, terminal.record(nil, fmt.Errorf("clone chunker manifest metadata: %w", metadataErr))
}
profile := ""
if provider, ok := chunker.(contracts.ChunkExecutionClassProvider); ok && provider.ExecutionClass() == contracts.ExecutionClassLLMBacked {
if input.pipeline.ChunkExecutionClass == contracts.ExecutionClassLLMBacked {
profile = input.pipeline.Chunk.LLMProfile
}
candidate := ChunkPlanRecord{

View File

@@ -67,30 +67,17 @@ type manifestChunker struct {
}
func (c manifestChunker) ManifestMetadata() map[string]any { return c.metadata }
func (manifestChunker) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassLLMBacked
}
type llmCountingChunker struct {
terminalChunker
llmCalls *int
}
func (llmCountingChunker) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassLLMBacked
}
func (c llmCountingChunker) Plan(ctx context.Context, request contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
(*c.llmCalls)++
return c.terminalChunker.Plan(ctx, request)
}
type deterministicChunker struct{ terminalChunker }
func (deterministicChunker) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassDeterministic
}
type retryingChunker struct {
key string
plan source.ChunkPlan
@@ -535,6 +522,7 @@ func TestRunnerPublishesOnlyAcceptedGeneratedPlans(t *testing.T) {
func TestRunnerStoresProducerProvenanceAndProducerWarnings(t *testing.T) {
prepared, plan := preparedTerminalDebugPipeline(t)
prepared.resolved.Chunk.LLMProfile = "chunk-profile"
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
prepared.resolved.ChunkReferences = ResolvedReferenceTarget{
Stage: StageChunk,
ReferenceSet: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
@@ -577,7 +565,8 @@ func TestRunnerRejectsUncloneableModuleManifestMetadata(t *testing.T) {
func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
prepared, plan := preparedTerminalDebugPipeline(t)
prepared.resolved.Chunk.LLMProfile = "configured-but-unused"
prepared.chunker = deterministicChunker{terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}}
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassDeterministic
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}
store := &recordingChunkPlanStore{}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store})
if err != nil {
@@ -588,6 +577,22 @@ func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
}
}
func TestRunnerDoesNotInventProducerProfileForLLMChunker(t *testing.T) {
prepared, plan := preparedTerminalDebugPipeline(t)
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
prepared.resolved.Chunk.LLMProfile = ""
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}
store := &recordingChunkPlanStore{}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store})
if err != nil {
t.Fatal(err)
}
if store.saved.Producer.LLMProfile != "" || output.Manifest.ChunkPlan.ProducerLLMProfile != "" {
t.Fatalf("LLM producer profile = stored %q manifest %q, want empty", store.saved.Producer.LLMProfile, output.Manifest.ChunkPlan.ProducerLLMProfile)
}
}
func TestRunnerRefreshChangesDownstreamChunkFingerprint(t *testing.T) {
doc := typedTestDocumentWithUnits(2)
prepared := preparedConcurrentPipeline(t, 1)

View File

@@ -229,7 +229,7 @@ func TestResolveTypedLaneRejectsIncompatibleComposition(t *testing.T) {
func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) {
registry := NewMergerRegistry()
spec := ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"}
spec := ModuleSpec{Key: "typed/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}
constructor := func() (contracts.Merger[codecNotes], error) {
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
}
@@ -243,7 +243,7 @@ func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) {
func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
extractors := NewExtractorRegistry()
if err := RegisterExtractor(extractors, ModuleSpec{Key: "typed/extract", Stage: StageExtract, ArtifactKind: "test/notes"}, func() (contracts.Extractor[codecNotes], error) {
if err := RegisterExtractor(extractors, ModuleSpec{Key: "typed/extract", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Extractor[codecNotes], error) {
return typedTestExtractor[codecNotes]{key: "typed/extract"}, nil
}); err != nil {
t.Fatalf("RegisterExtractor() error = %v", err)
@@ -253,7 +253,7 @@ func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
}
mergers := NewMergerRegistry()
if err := RegisterMerger(mergers, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"}, func() (contracts.Merger[codecNotes], error) {
if err := RegisterMerger(mergers, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Merger[codecNotes], error) {
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
}); err != nil {
t.Fatalf("RegisterMerger() error = %v", err)
@@ -263,7 +263,7 @@ func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
}
normalizers := NewNormalizerRegistry()
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
return typedTestNormalizer[codecNotes]{key: "typed/normalize"}, nil
}); err != nil {
t.Fatalf("RegisterNormalizer() error = %v", err)
@@ -298,7 +298,7 @@ func TestTypedRegistrySpecLookupUsesArtifactKindAndStableCatalogOrder(t *testing
mergers := NewMergerRegistry()
normalizers := NewNormalizerRegistry()
for _, item := range order {
base := ModuleSpec{Key: "typed/shared", ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.slot}}}
base := ModuleSpec{Key: "typed/shared", ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.slot}}}
mergeSpec := base
mergeSpec.Stage = StageMerge
normalizeSpec := base
@@ -474,18 +474,18 @@ func typedResolutionCatalog(t *testing.T, options typedCatalogOptions) ModuleCat
func mustRegisterTypedTestBase(t *testing.T, catalog ModuleCatalog) {
t.Helper()
if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{Key: "typed/input", Stage: StageInput}, func() (contracts.InputAdapter, error) {
if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{Key: "typed/input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.InputAdapter, error) {
return &typedTestInput{key: "typed/input", doc: typedTestDocument()}, nil
}); err != nil {
t.Fatalf("register input: %v", err)
}
if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{Key: "typed/chunk", Stage: StageChunk}, func() (contracts.Chunker, error) {
if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{Key: "typed/chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.Chunker, error) {
doc := typedTestDocument()
return &typedTestChunker{key: "typed/chunk", plan: typedTestPlan(doc)}, nil
}); err != nil {
t.Fatalf("register chunker: %v", err)
}
if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{Key: "typed/output", Stage: StageOutput}, func() (contracts.OutputEncoder, error) {
if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{Key: "typed/output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.OutputEncoder, error) {
return &typedTestOutput{key: "typed/output"}, nil
}); err != nil {
t.Fatalf("register output: %v", err)
@@ -509,21 +509,21 @@ func mustRegisterArtifactCodec[T any](t *testing.T, registry *ArtifactCodecRegis
func mustRegisterTypedExtractor[T any](t *testing.T, registry *ExtractorRegistry, key string, kind contracts.ArtifactKind, extractor contracts.Extractor[T]) {
t.Helper()
if err := RegisterExtractor(registry, ModuleSpec{Key: key, Stage: StageExtract, ArtifactKind: kind}, func() (contracts.Extractor[T], error) { return extractor, nil }); err != nil {
if err := RegisterExtractor(registry, ModuleSpec{Key: key, Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: kind}, func() (contracts.Extractor[T], error) { return extractor, nil }); err != nil {
t.Fatalf("RegisterExtractor() error = %v", err)
}
}
func mustRegisterTypedMerger[T any](t *testing.T, registry *MergerRegistry, kind contracts.ArtifactKind, merger contracts.Merger[T]) {
t.Helper()
if err := RegisterMerger(registry, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: kind}, func() (contracts.Merger[T], error) { return merger, nil }); err != nil {
if err := RegisterMerger(registry, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: kind}, func() (contracts.Merger[T], error) { return merger, nil }); err != nil {
t.Fatalf("RegisterMerger() error = %v", err)
}
}
func mustRegisterTypedNormalizer[T any](t *testing.T, registry *NormalizerRegistry, kind contracts.ArtifactKind, normalizer contracts.Normalizer[T]) {
t.Helper()
if err := RegisterNormalizer(registry, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: kind}, func() (contracts.Normalizer[T], error) { return normalizer, nil }); err != nil {
if err := RegisterNormalizer(registry, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: kind}, func() (contracts.Normalizer[T], error) { return normalizer, nil }); err != nil {
t.Fatalf("RegisterNormalizer() error = %v", err)
}
}

View File

@@ -1,6 +1,6 @@
id: dnd.scenes
version: "v1"
default_profile: gemini-2-flash
default_profile: dnd-extraction
inputs:
- name: transcript
required: true

View File

@@ -47,10 +47,6 @@ func (c *Chunker) Key() string {
return Key
}
func (*Chunker) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassLLMBacked
}
func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
return shared.ReferenceSlots(referenceSlotDescriptions)
}
@@ -120,6 +116,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageChunk,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ReferenceSlots: shared.ReferenceSlots(referenceSlotDescriptions),

View File

@@ -24,6 +24,7 @@ func TestNewModuleSpecAndRegister(t *testing.T) {
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageChunk,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: []string{"source.transcript"},
Provides: []string{"chunks"},
ReferenceSlots: wantReferenceSlots(),

View File

@@ -1,6 +1,6 @@
id: dnd.combat_turns
version: "v1"
default_profile: gemini-2-flash
default_profile: dnd-extraction
inputs:
- name: transcript
required: true

View File

@@ -226,6 +226,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.CombatTurnListKind,

View File

@@ -1,6 +1,6 @@
id: dnd.item_events
version: "v1"
default_profile: gemini-2-flash
default_profile: dnd-extraction
inputs:
- name: transcript
required: true

View File

@@ -129,6 +129,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.ItemEventListKind,

View File

@@ -18,7 +18,7 @@ func TestConstructorSpecOptionsAndMetadata(t *testing.T) {
t.Fatalf("New() error = %v", err)
}
want := pipeline.ModuleSpec{
Key: Key, Stage: pipeline.StageExtract, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.item_events"}, ArtifactKind: dnd.ItemEventListKind,
Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.item_events"}, ArtifactKind: dnd.ItemEventListKind,
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "glossary", Description: referenceSlotDescriptions.Glossary, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
{Name: "party", Description: referenceSlotDescriptions.Party, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},

View File

@@ -1,6 +1,6 @@
id: dnd.npc_interactions
version: "v1"
default_profile: gemini-2-flash
default_profile: dnd-extraction
inputs:
- name: transcript
required: true

View File

@@ -177,6 +177,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.NPCInteractionListKind,

View File

@@ -1,6 +1,6 @@
id: dnd.npcs
version: "v1"
default_profile: gemini-2-flash
default_profile: dnd-extraction
inputs:
- name: transcript
required: true

View File

@@ -135,6 +135,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.NPCListKind,

View File

@@ -25,11 +25,12 @@ func TestNewRequiresLLMClientAndRejectsAmbiguousReferences(t *testing.T) {
func TestModuleSpecAndReferenceSlots(t *testing.T) {
got := ModuleSpec()
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageExtract,
Requires: []string{"chunks", "source.transcript"},
Provides: []string{"dnd.npcs"},
ArtifactKind: dnd.NPCListKind,
Key: Key,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: []string{"chunks", "source.transcript"},
Provides: []string{"dnd.npcs"},
ArtifactKind: dnd.NPCListKind,
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "glossary", Description: "Optional campaign glossary reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
{Name: "party", Description: "Optional party roster reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},

View File

@@ -1,6 +1,6 @@
id: dnd.scene_descriptions
version: "v1"
default_profile: gemini-2-flash
default_profile: dnd-extraction
inputs:
- name: transcript
required: true

View File

@@ -146,6 +146,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.SceneDescriptionListKind,

View File

@@ -24,7 +24,7 @@ func TestNewRequiresLLMClientAndRejectsAmbiguousReferences(t *testing.T) {
func TestModuleSpecAndReferenceSlots(t *testing.T) {
want := pipeline.ModuleSpec{
Key: Key, Stage: pipeline.StageExtract, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.scene_descriptions"}, ArtifactKind: dnd.SceneDescriptionListKind,
Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.scene_descriptions"}, ArtifactKind: dnd.SceneDescriptionListKind,
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "glossary", Description: "Optional campaign glossary reference material used only to disambiguate scene descriptions.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
{Name: "party", Description: "Optional party roster reference material used only to disambiguate scene descriptions.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},

View File

@@ -1,6 +1,6 @@
id: dnd.spells
version: "v1"
default_profile: gemini-2-flash
default_profile: dnd-extraction
inputs:
- name: transcript
required: true

View File

@@ -200,6 +200,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.SpellListKind,

View File

@@ -23,8 +23,9 @@ func TestNewRequiresLLMClientAndReturnsExtractor(t *testing.T) {
func TestModuleSpec(t *testing.T) {
got := ModuleSpec()
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageExtract,
Key: Key,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: []string{
"chunks",
"source.transcript",

View File

@@ -336,6 +336,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.CombatTurnListKind,

View File

@@ -222,11 +222,12 @@ func eventScope(index int) string { return fmt.Sprintf("events[%d]", index) }
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageNormalize,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.ItemEventListKind,
Key: Key,
Stage: pipeline.StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.ItemEventListKind,
}
}

View File

@@ -278,6 +278,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.NPCInteractionListKind,

View File

@@ -1,6 +1,6 @@
id: dnd.npcs.normalize
version: "v1"
default_profile: gemini-2-flash
default_profile: dnd-extraction
inputs:
- name: candidates
required: true

View File

@@ -360,7 +360,7 @@ func duplicateWarning(retainedIndex int, removed []int) contracts.Warning {
func npcScope(index int) string { return fmt.Sprintf("npcs[%d]", index) }
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, Requires: append([]string(nil), requiredCapabilities...), Provides: append([]string(nil), providedCapabilities...), ArtifactKind: dnd.NPCListKind}
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: append([]string(nil), requiredCapabilities...), Provides: append([]string(nil), providedCapabilities...), ArtifactKind: dnd.NPCListKind}
}
func Register(registry *pipeline.NormalizerRegistry) error {

View File

@@ -21,7 +21,7 @@ func TestModuleContractAndIdentity(t *testing.T) {
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
t.Fatal("DecodeOptions() accepted unknown option")
}
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: dnd.NPCListKind}
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: dnd.NPCListKind}
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
}

View File

@@ -138,11 +138,12 @@ func sourceRefLabel(ref source.SourceRef) string {
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageNormalize,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.SceneDescriptionListKind,
Key: Key,
Stage: pipeline.StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.SceneDescriptionListKind,
}
}

View File

@@ -90,7 +90,7 @@ func TestNormalizerContractAndCancellation(t *testing.T) {
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
t.Fatal("DecodeOptions() accepted unknown options")
}
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: dnd.SceneDescriptionListKind}
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: dnd.SceneDescriptionListKind}
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
}

View File

@@ -297,6 +297,7 @@ func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.SpellListKind,

View File

@@ -24,11 +24,12 @@ func TestModuleContractAndStrictOptions(t *testing.T) {
}
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageNormalize,
Requires: []string{"merged"},
Provides: []string{"normalized"},
ArtifactKind: dnd.SpellListKind,
Key: Key,
Stage: pipeline.StageNormalize,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"merged"},
Provides: []string{"normalized"},
ArtifactKind: dnd.SpellListKind,
ReferenceSlots: []contracts.ReferenceSlot{{
Name: spellcatalog.SpellCatalogReferenceSlot,
Description: "Optional canonical spell-name catalog used for normalization and duplicate identity.",

View File

@@ -0,0 +1,5 @@
id: dnd-extraction
backend: openrouter
model: openai/gpt-5.6-luna
timeout_seconds: 240
service_tier: flex

View File

@@ -0,0 +1,14 @@
package register
import (
"embed"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
)
//go:embed assets/profiles/*.yaml
var embeddedProfileAssets embed.FS
func registerFallbackProfiles(assets *llm.AssetRegistry) error {
return assets.RegisterFallbackProfileFS(embeddedProfileAssets, "assets/profiles")
}

View File

@@ -30,6 +30,9 @@ func Register(registries pipeline.Registries, assets *llm.AssetRegistry) error {
if err := registerPromptAssets(assets); err != nil {
return err
}
if err := registerFallbackProfiles(assets); err != nil {
return err
}
return registerDefaultChains(registries.ValidatorChains)
}

View File

@@ -39,6 +39,31 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
if _, err := fs.ReadFile(promptFS, "dnd.npcs.normalize/dnd.npcs.normalize.yaml"); err != nil {
t.Fatalf("normalization prompt asset = %v, want registered private prompt", err)
}
for _, name := range []string{
"dnd.scenes/dnd.scenes.yaml",
"dnd.spells/dnd.spells.yaml",
"dnd.npcs/dnd.npcs.yaml",
"dnd.combat_turns/dnd.combat_turns.yaml",
"dnd.item_events/dnd.item_events.yaml",
"dnd.npc_interactions/dnd.npc_interactions.yaml",
"dnd.scene_descriptions/dnd.scene_descriptions.yaml",
"dnd.npcs.normalize/dnd.npcs.normalize.yaml",
} {
content, err := fs.ReadFile(promptFS, name)
if err != nil {
t.Fatalf("read prompt asset %q: %v", name, err)
}
if !strings.Contains(string(content), "default_profile: dnd-extraction") {
t.Fatalf("prompt asset %q does not select dnd-extraction", name)
}
}
fallbackFS, err := assets.FallbackProfileFS()
if err != nil {
t.Fatalf("FallbackProfileFS() error = %v", err)
}
if _, err := fs.ReadFile(fallbackFS, "dnd-extraction.yaml"); err != nil {
t.Fatalf("fallback profile asset = %v, want registered D&D profile", err)
}
schemaFS, err := assets.SchemaFS()
if err != nil {
t.Fatalf("SchemaFS() error = %v", err)

View File

@@ -38,10 +38,6 @@ func (c *Chunker) Key() string {
return Key
}
func (*Chunker) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassDeterministic
}
func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
@@ -90,9 +86,10 @@ func (c *Chunker) Plan(ctx context.Context, req contracts.ChunkRequest) (contrac
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageChunk,
Provides: []string{"chunks"},
Key: Key,
Stage: pipeline.StageChunk,
ExecutionClass: contracts.ExecutionClassDeterministic,
Provides: []string{"chunks"},
}
}

View File

@@ -15,9 +15,10 @@ import (
func TestModuleSpecAndRegister(t *testing.T) {
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageChunk,
Provides: []string{"chunks"},
Key: Key,
Stage: pipeline.StageChunk,
ExecutionClass: contracts.ExecutionClassDeterministic,
Provides: []string{"chunks"},
}
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)

View File

@@ -3,13 +3,14 @@ package appendorder
import (
"fmt"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const Key = "appendorder"
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageMerge, Provides: []string{"merged"}}
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"merged"}}
}
func mergerErrorf(format string, args ...any) error {

Some files were not shown because too many files have changed in this diff Show More