Compare commits
13 Commits
b5aaeb1c78
...
v0.2.0
| Author | SHA1 | Date | |
|---|---|---|---|
| 39388e96d4 | |||
| 12ac25bd63 | |||
| 394278e1f2 | |||
| 5cd7f8e737 | |||
| bf3fadf9ae | |||
| 58815aaf33 | |||
| ce857966f1 | |||
| a3bd0c1867 | |||
| b05634ee86 | |||
| 4829f94157 | |||
| 67b315099d | |||
| b5c86de4d7 | |||
| 2eeca2ed5a |
49
docs/adr/0010-workload-oriented-llm-profile-defaults.md
Normal file
49
docs/adr/0010-workload-oriented-llm-profile-defaults.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# ADR-0010: Use workload-oriented LLM profile defaults
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-08-03
|
||||
|
||||
## Context
|
||||
|
||||
LLM-backed D&D operations share an execution-policy choice, but repeating a
|
||||
provider or model-named profile on every module binding ties pipeline structure
|
||||
to a deployment decision. Different environments may require different model,
|
||||
backend, timeout, or reasoning settings while retaining the same workload.
|
||||
|
||||
Notarius also needs a usable default for maintained D&D prompts without making
|
||||
an operator profile mandatory. That default must remain owned by the D&D
|
||||
family, while generic LLM infrastructure stays unaware of domain-specific
|
||||
policy.
|
||||
|
||||
## Decision
|
||||
|
||||
Pipelines may name one workload-oriented default profile, inherited only by
|
||||
selected LLM-backed bindings and validators. Binding-level profile IDs remain
|
||||
intentional exceptions, and the run-wide CLI profile override has highest
|
||||
precedence.
|
||||
|
||||
The D&D family owns an embedded fallback profile named `dnd-extraction`.
|
||||
Operators may provide a complete profile with the same ID through a PromptKit
|
||||
filesystem source. PromptKit selects the higher-precedence matching definition;
|
||||
Notarius does not merge profile documents. Production, development, and local
|
||||
deployments can therefore use different execution policy behind one unchanged
|
||||
pipeline ID.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- Repeat a model-named profile on every binding. This makes routine deployment
|
||||
policy changes noisy and obscures the shared workload intent.
|
||||
- Require every deployment to install a profile file. This adds configuration
|
||||
friction and leaves maintained D&D prompts without an application-owned
|
||||
fallback.
|
||||
- Put D&D profile policy in generic LLM infrastructure. This breaks domain
|
||||
ownership and makes generic code depend on one workload.
|
||||
|
||||
## Consequences
|
||||
|
||||
Pipeline configuration expresses workload intent rather than a specific
|
||||
provider or model. Operators can replace the complete execution policy without
|
||||
editing bindings, while binding-level and run-wide exceptions remain available.
|
||||
Profile changes affect resolved pipeline and checkpoint identity, so they may
|
||||
intentionally cause work to be recomputed. The D&D fallback becomes a
|
||||
maintained application execution-policy asset.
|
||||
@@ -39,7 +39,7 @@ pipeline ID and **--input** are required.
|
||||
| **--debug** | Retain a debug bundle for this run. |
|
||||
| **--debug-dir path** | Override the debug-bundle root. Requires **--debug**. |
|
||||
| **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. |
|
||||
| **--llm-profile id** | Override effective LLM-capable module bindings with one configured profile. |
|
||||
| **--llm-profile id** | Highest-precedence configured profile for selected LLM-backed bindings and validators; it replaces binding and [pipeline](config.md#pipelines) defaults. |
|
||||
| **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. |
|
||||
| **--reasoning-effort value** | Replace the selected PromptKit profile's reasoning effort for every LLM-backed call in this run. The value must be non-empty and the flag may be specified only once. |
|
||||
| **--clear-reasoning-effort** | Clear reasoning effort inherited from the selected PromptKit profile for every LLM-backed call in this run. |
|
||||
|
||||
@@ -32,7 +32,8 @@ override the fields listed below.
|
||||
single-lane Seriatim-to-spell pipeline.
|
||||
- [Complete D&D configuration](../examples/dnd-complete.config.yml) uses
|
||||
ordered steps, all implemented D&D lanes, generated references, state
|
||||
settings, and bounded LLM concurrency.
|
||||
settings, bounded LLM concurrency, and the maintained
|
||||
[operator profile](../examples/profiles/dnd-extraction.yml).
|
||||
|
||||
Use these complete files as starting points rather than combining the
|
||||
illustrative fragments in this reference.
|
||||
@@ -93,10 +94,22 @@ promptkit:
|
||||
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
|
||||
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
|
||||
|
||||
Set at most one of **profile_dir** and **profile_file**. Profile IDs used by a
|
||||
binding must be available from the selected PromptKit profile source when the
|
||||
pipeline is resolved. The optional local registration may coexist with either
|
||||
profile source or with PromptKit's built-in profiles.
|
||||
Set at most one of **profile_dir** and **profile_file**. Relative values use
|
||||
the process working directory, not the configuration file's directory. The
|
||||
complete example's `./examples/profiles/dnd-extraction.yml` value is therefore
|
||||
valid when Notarius is launched from the repository root; use an absolute path
|
||||
for services and containers.
|
||||
|
||||
An operator source is optional. For a requested ID, PromptKit checks the
|
||||
configured operator source first, then Notarius's embedded fallback profiles,
|
||||
then its own built-in catalog. A matching profile is complete: it replaces a
|
||||
lower-precedence definition rather than merging with it. The maintained
|
||||
[`dnd-extraction` operator profile](../examples/profiles/dnd-extraction.yml)
|
||||
is a secret-free deployment artifact; production, development, and local
|
||||
deployments can each provide a complete definition with that same workload ID.
|
||||
Use workload-oriented IDs for new profiles instead of model names.
|
||||
[Operations](operations.md#promptkit-profile-deployment) owns the deployment
|
||||
workflow and credential-handling guidance.
|
||||
|
||||
When **local_backend** is present, its endpoint is trimmed and must use HTTP or
|
||||
HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are
|
||||
@@ -119,11 +132,17 @@ model: example-model
|
||||
Keep credentials out of the local-backend object. A PromptKit profile may name
|
||||
its credential environment variable through `api_key_env`; set that variable
|
||||
only in the run environment. PromptKit owns the
|
||||
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/formats.md).
|
||||
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md).
|
||||
The [PromptKit upstream boundary](integrations/pkg-promptkit.md) identifies the
|
||||
supported package API, and [Operations](operations.md#operational-limits)
|
||||
describes the effective concurrency layers.
|
||||
|
||||
`notarius config validate --pipeline <id>` resolves the selected pipeline and
|
||||
inspects every explicit effective profile without contacting a provider or
|
||||
requiring credential values. It rejects absent, malformed, or incompatible
|
||||
profiles before a run prepares modules. Credential availability is checked only
|
||||
when a generation is prepared.
|
||||
|
||||
## Migrating Version 3 Configuration
|
||||
|
||||
Version 3 files are not decoded or rewritten. Change **version: 3** to
|
||||
@@ -185,6 +204,7 @@ Each **pipelines** entry has a unique, non-empty ID and the following shape:
|
||||
~~~yaml
|
||||
pipelines:
|
||||
dnd-session:
|
||||
llm_profile: dnd-extraction
|
||||
input: seriatim
|
||||
chunk: generic
|
||||
output: json
|
||||
@@ -197,6 +217,7 @@ pipelines:
|
||||
|
||||
| Field | Type | Default | Rules |
|
||||
| --- | --- | --- | --- |
|
||||
| **llm_profile** | string | none | Optional non-empty default PromptKit profile ID for selected LLM-backed bindings and validators. An explicitly present blank value is invalid. |
|
||||
| **input** | module binding | none | Required. |
|
||||
| **chunk** | module binding | **generic** | Optional. |
|
||||
| **output** | module binding | **json** | Optional. |
|
||||
@@ -210,6 +231,12 @@ needs a unique non-empty **id**, an **artifacts** map, and may have
|
||||
**references**. A lane ID must not appear more than once in a pipeline,
|
||||
including across explicit steps.
|
||||
|
||||
For each selected LLM-backed binding or validator, profile selection occurs
|
||||
after module, validator, and `--only` lane selection. It uses the
|
||||
run-level **--llm-profile** value first, then the binding's **llm_profile**,
|
||||
then the pipeline's **llm_profile**, and finally the PromptKit default.
|
||||
Deterministic bindings do not receive these defaults or run overrides.
|
||||
|
||||
A lane has these fields:
|
||||
|
||||
| Field | Type | Default | Rules |
|
||||
@@ -237,7 +264,7 @@ Use an object for fields:
|
||||
~~~yaml
|
||||
extract:
|
||||
module: dnd/spells
|
||||
llm_profile: gemini-2-flash
|
||||
llm_profile: dnd-extraction
|
||||
retries: 2
|
||||
references:
|
||||
spell_catalog: ./dnd-spell-catalog.json
|
||||
@@ -246,7 +273,7 @@ extract:
|
||||
| Binding field | Type | Default | Rules |
|
||||
| --- | --- | --- | --- |
|
||||
| **module** | string | none | Required for an object binding. Must be a registered compatible key. |
|
||||
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID. |
|
||||
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID for an LLM-backed binding. It overrides the pipeline default unless the run supplies **--llm-profile**. |
|
||||
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. |
|
||||
| **options** | object | none | Must satisfy the selected module. |
|
||||
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
|
||||
@@ -256,7 +283,8 @@ Omitting **validators** uses the registered chain. **validators: []** selects
|
||||
an empty chain; a non-empty list replaces the chain in the listed order.
|
||||
Validator bindings accept only **module**, **llm_profile**, and **options**.
|
||||
They reject **references**, **retries**, and nested **validators**. Deterministic
|
||||
validators reject an explicit **llm_profile**.
|
||||
validators reject an explicit **llm_profile**. Deterministic module bindings
|
||||
also reject an explicit **llm_profile**.
|
||||
|
||||
The **json** output module accepts optional **include_chunk_map** and
|
||||
**evidence_context** settings:
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
# PromptKit Integration
|
||||
|
||||
Notarius pins
|
||||
[`gitea.maximumdirect.net/eric/promptkit` v0.3.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0)
|
||||
[`gitea.maximumdirect.net/eric/promptkit` v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0)
|
||||
as its in-process prompt engine. The upstream
|
||||
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/consumers/pkg-promptkit.md)
|
||||
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/consumers/pkg-promptkit.md)
|
||||
owns the public engine API, and the upstream
|
||||
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/formats.md)
|
||||
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
|
||||
owns prompt, profile, and schema file contracts.
|
||||
|
||||
## Supported Boundary
|
||||
@@ -13,9 +13,10 @@ owns prompt, profile, and schema file contracts.
|
||||
Notarius relies on the root `promptkit` package to:
|
||||
|
||||
- construct an `Engine` with filesystem-backed prompt, schema, and optional
|
||||
profile sources;
|
||||
- prepare and run a `RunRequest` with named inline artifacts, variables,
|
||||
a direct session ID, prompt identity, and profile selection;
|
||||
operator and application-fallback profile sources;
|
||||
- prepare one frozen execution from a `RunRequest` with named inline artifacts,
|
||||
variables, a direct session ID, prompt identity, and profile selection, then
|
||||
record credential-redacted details and run that exact execution;
|
||||
- return rendered debug material, validated structured output, selected
|
||||
profile, backend, effective model metadata, and token usage;
|
||||
- register the optional conventional `local` backend through `BackendLocal`,
|
||||
@@ -25,9 +26,22 @@ Notarius relies on the root `promptkit` package to:
|
||||
admission exhaustion through `ErrCapacityExceeded`.
|
||||
|
||||
The pinned
|
||||
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/backends.go)
|
||||
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/backends.go)
|
||||
owns the registration and backend-capacity contract.
|
||||
|
||||
For one completion, the adapter calls `PrepareExecution`, takes a
|
||||
caller-owned `Details` snapshot, and calls `RunPrepared` for that same opaque
|
||||
prepared execution. It defers `Discard` for every unexecuted handle. Explicit
|
||||
profile preflight uses `Engine.InspectProfile`; it does not prepare a synthetic
|
||||
prompt. PromptKit's prepared handle, inspection result, and capacity-error
|
||||
types stay inside the Notarius LLM adapter.
|
||||
|
||||
When a PromptKit profile and runtime override leave `temperature`, `max_tokens`,
|
||||
or `top_p` unset, Notarius leaves that control unset as well. Compatible
|
||||
providers therefore apply their own defaults; an operator that requires a
|
||||
specific sampling value must select it explicitly in the profile or runtime
|
||||
override.
|
||||
|
||||
Notarius does not use PromptKit's optional `ArtifactReader`. It materializes
|
||||
source and reference content itself and supplies owned inline artifacts at the
|
||||
adapter boundary. It also retains responsibility for pipeline retries,
|
||||
@@ -52,8 +66,28 @@ Notarius retains its application-wide scheduled client around the PromptKit
|
||||
adapter. PromptKit may apply a narrower limit for the selected backend;
|
||||
endpoint-only profiles have no such backend limit. The adapter translates
|
||||
PromptKit capacity rejection into the provider-neutral Notarius
|
||||
`ErrLLMCapacityExceeded` contract and leaves retries to the calling pipeline
|
||||
stage.
|
||||
`ErrLLMCapacityExceeded` contract. It may include the normalized selected
|
||||
backend ID in safe diagnostic context, without exposing PromptKit's capacity
|
||||
error type, and leaves retries to the calling pipeline stage.
|
||||
|
||||
## Profile Sources And Compatibility
|
||||
|
||||
Notarius gives PromptKit the configured operator profile source, registered
|
||||
application fallback profile assets, and optional backend registration through
|
||||
the same construction path for inspection and execution. PromptKit owns the
|
||||
resulting source precedence and strict profile parsing: a matching operator
|
||||
profile is a complete replacement for a fallback or built-in profile, while an
|
||||
invalid matching document fails instead of falling through. The operator
|
||||
configuration and deployment workflow are defined in
|
||||
[Configuration](../config.md#promptkit-profiles) and
|
||||
[Operations](../operations.md#promptkit-profile-deployment).
|
||||
|
||||
Notarius supports this boundary against PromptKit v0.5.0. Its fallback source,
|
||||
prepared-execution, inspection, and typed capacity APIs are used as public
|
||||
upstream contracts; other PromptKit APIs or file-format behavior are not
|
||||
implicitly supported. A dependency upgrade requires reviewing the adapter,
|
||||
profile-source construction, and this compatibility statement against the
|
||||
pinned upstream documentation.
|
||||
|
||||
## Notarius Ownership
|
||||
|
||||
@@ -61,6 +95,8 @@ stage.
|
||||
module assets, maps its transport-neutral completion contract, prepares and
|
||||
executes requests, validates output, records provenance, captures debug
|
||||
material, redacts errors, and preserves timeout ownership.
|
||||
[D&D Module Internals](../internal/dnd.md) owns the embedded
|
||||
`dnd-extraction` fallback profile and the maintained D&D prompt defaults.
|
||||
[Configuration](../config.md#promptkit-profiles) defines how a Notarius
|
||||
configuration selects one PromptKit profile source and optionally registers
|
||||
the conventional local backend.
|
||||
|
||||
@@ -38,11 +38,14 @@ in [Configuration Internals](configuration.md).
|
||||
|
||||
Configuration validation without a selected pipeline checks structural
|
||||
configuration only. Validation with a selected pipeline also builds the
|
||||
effective catalog, resolves the pipeline, and verifies explicitly selected
|
||||
PromptKit profiles. Each explicit binding or validator profile is prepared
|
||||
against the configured PromptKit source without performing generation, so an
|
||||
unknown profile fails before pipeline preparation. Pipeline listing validates
|
||||
configuration before returning normalized, sorted identifiers.
|
||||
effective catalog, resolves the pipeline, and verifies every explicit effective
|
||||
PromptKit profile. Selected LLM-backed input, chunk, lane, output, and validator
|
||||
profiles are inspected
|
||||
against the configured PromptKit source and backend registrations without
|
||||
loading a prompt or performing generation, so an unknown or invalid profile
|
||||
fails before pipeline preparation. Credential availability remains an
|
||||
execution-time concern. Pipeline listing validates configuration before
|
||||
returning normalized, sorted identifiers.
|
||||
|
||||
## Production Composition
|
||||
|
||||
@@ -59,7 +62,8 @@ profile-provenance recorder, creates one scheduler from the effective global
|
||||
LLM limit, and wraps the client before it reaches modules. Registration and LLM
|
||||
construction errors are returned before a pipeline is prepared. Configuration
|
||||
field definitions remain in [Configuration](../config.md#promptkit-profiles);
|
||||
the adapter mechanics remain in [LLM Runtime](llm.md).
|
||||
the D&D registrar's fallback profile assets and the adapter mechanics remain in
|
||||
[LLM Runtime](llm.md).
|
||||
|
||||
The factory also accepts `LLMRuntimeOverrides`, whose reasoning pointer
|
||||
preserves inherit, replace, and clear states across the composition boundary.
|
||||
@@ -81,7 +85,8 @@ handoff:
|
||||
2. create and validate a safe run identity, then allocate a debug bundle only
|
||||
when requested;
|
||||
3. build the effective catalog, resolve requested reference changes, resolve
|
||||
the effective pipeline, and verify explicit PromptKit profiles;
|
||||
the effective pipeline, and inspect its explicit effective PromptKit
|
||||
profiles;
|
||||
4. materialize external or generated references and record redacted invocation
|
||||
and resolution provenance when debug capture is enabled;
|
||||
5. construct registries, the scheduled LLM client, prepared modules, and the
|
||||
|
||||
@@ -42,7 +42,9 @@ or collide after whitespace normalization. The file application also makes the
|
||||
effective extraction-worker default follow the effective LLM limit. A present
|
||||
PromptKit local-backend object requires and trims its endpoint, defaults its
|
||||
omitted concurrency limit to zero, and is copied so the parsed file model
|
||||
cannot alias the populated **Config**.
|
||||
cannot alias the populated **Config**. A pipeline `llm_profile` is
|
||||
presence-aware: omission remains empty, while a present blank value is
|
||||
rejected and a non-empty file value is trimmed before it reaches **Config**.
|
||||
|
||||
**Config.Validate** checks configuration-only invariants before resolution. It
|
||||
rejects incompatible profile sources, invalid state-surface values, unsupported
|
||||
@@ -62,13 +64,15 @@ environment override, its validation, and that reference in the same change.
|
||||
|
||||
**Config.Resolve** first recomputes derived concurrency defaults and validates
|
||||
the configuration. It normalizes the requested pipeline ID, copies the selected
|
||||
profile, applies a non-empty command-level LLM profile override to the
|
||||
LLM-capable stage bindings, and calls the framework resolver with the requested
|
||||
lane selection and reference changes.
|
||||
profile, and passes the non-empty command-level LLM profile override, requested
|
||||
lane selection, and reference changes to the framework resolver.
|
||||
|
||||
The command-level override does not replace an explicitly selected validator
|
||||
profile. Validator bindings remain part of the resolved validator chain and
|
||||
are resolved under their own declared configuration.
|
||||
After module and validator selection, the resolver applies the effective
|
||||
profile policy to LLM-backed bindings only: command override, binding profile,
|
||||
pipeline profile, then the prompt default. Deterministic bindings remain
|
||||
profile-free, and no second inheritance decision occurs during execution. The
|
||||
public field definitions and precedence are owned by
|
||||
[Configuration](../config.md#pipelines).
|
||||
|
||||
The framework resolver supplies defaults, selects lanes, resolves validator
|
||||
chains, checks registered module and artifact compatibility, validates module
|
||||
@@ -90,7 +94,8 @@ runtime error class described in the [CLI reference](../cli.md#output-streams-an
|
||||
|
||||
The framework assigns the resolved pipeline a deterministic SHA-256 digest
|
||||
after defaults, lane selection, module bindings, reference bindings, validator
|
||||
chains, and artifact schema identity have been resolved. The digest excludes
|
||||
chains, effective LLM profiles, and artifact schema identity have been
|
||||
resolved. The digest excludes
|
||||
its own stored value. It identifies resolved composition rather than raw YAML
|
||||
bytes, a debug payload, or all runtime state. The CLI records it as invocation
|
||||
provenance before execution; cache and checkpoint identity have additional
|
||||
|
||||
@@ -22,10 +22,15 @@ does not repeat their JSON shapes or schemas.
|
||||
## Family Composition
|
||||
|
||||
The D&D registrar registers the family’s artifact codecs, extractors, typed
|
||||
append-order mergers, normalizers, validators, prompt assets, and default
|
||||
validator chains. Each extractor and normalizer has a stable module spec,
|
||||
strict option decoding, and a typed builder. Configuration remains the
|
||||
canonical owner of the exact keys and validator order.
|
||||
append-order mergers, normalizers, validators, prompt assets, fallback LLM
|
||||
profile asset, and default validator chains. Each extractor and normalizer has
|
||||
a stable module spec, explicit execution class, strict option decoding, and a
|
||||
typed builder. Scene chunking, every extractor, and NPC normalization are
|
||||
registered as `llm_backed`; the remaining current D&D mergers and normalizers
|
||||
are `deterministic`. The metadata is available to catalog inspection and
|
||||
resolved-pipeline debug data and determines which selected bindings inherit the
|
||||
pipeline profile. Configuration remains the canonical owner of the exact keys,
|
||||
profile precedence, and validator order.
|
||||
|
||||
Private structured-LLM response schemas are deliberately minimal. They reject
|
||||
invalid JSON structure, missing required fields, incompatible types, and
|
||||
@@ -41,6 +46,15 @@ reference, and transcript assets instead of copying their text into individual
|
||||
modules. A manifest’s declared sequence, including cache-control placement, is
|
||||
part of the prompt behavior.
|
||||
|
||||
Every maintained D&D LLM prompt selects `dnd-extraction` as its default
|
||||
profile. The D&D registrar embeds that fallback profile with the maintained
|
||||
OpenRouter model, timeout, and service-tier policy. An operator may provide a
|
||||
complete profile with the same ID through the configured PromptKit source; that
|
||||
definition replaces the fallback rather than merging with it. The fallback
|
||||
leaves reasoning and optional sampling controls unspecified. Deployment profile
|
||||
selection and the maintained operator example are documented in
|
||||
[Configuration](../config.md#promptkit-profiles).
|
||||
|
||||
All extraction prompts share this four-message rendered prefix: the system
|
||||
message without cache control, the identity message without cache control, the
|
||||
campaign-reference message with ephemeral cache control, and the chunk
|
||||
|
||||
@@ -28,13 +28,15 @@ durable schemas. Those responsibilities remain with the module and its
|
||||
named material to a PromptKit inline artifact while preserving its origin URI,
|
||||
maps the trimmed request session to PromptKit's direct per-run session field,
|
||||
retains the same value as the `session_id` prompt variable for maintained
|
||||
prompt compatibility, forwards profile selection, then prepares and runs the
|
||||
prompt. The direct field is authoritative for provider session behavior. A
|
||||
session ID is a stable, non-secret correlation identifier and may be exposed
|
||||
to providers and provider observability. The adapter returns PromptKit’s
|
||||
validated raw bytes rather than re-encoding the decoded target. An empty
|
||||
optional material is represented as one space so its named input is retained
|
||||
by PromptKit.
|
||||
prompt compatibility, and forwards profile selection. It then creates one
|
||||
frozen prepared execution, captures its caller-owned credential-redacted
|
||||
details for debug material, and executes that exact snapshot through
|
||||
PromptKit's prepared-execution boundary. The direct field
|
||||
is authoritative for provider session behavior. A session ID is a stable,
|
||||
non-secret correlation identifier and may be exposed to providers and provider
|
||||
observability. The adapter returns PromptKit’s validated raw bytes rather than
|
||||
re-encoding the decoded target. An empty optional material is represented as
|
||||
one space so its named input is retained by PromptKit.
|
||||
|
||||
Client construction may also receive a run-wide reasoning-effort override from
|
||||
the CLI factory boundary. The adapter copies the caller-owned pointer and
|
||||
@@ -46,31 +48,44 @@ states. With neither flag, profile behavior remains unchanged. Because
|
||||
production constructs one shared client, the selected state applies uniformly
|
||||
to module calls, retries, and LLM-backed validators for the whole run.
|
||||
|
||||
An empty request profile lets the prompt select its configured default. The CLI
|
||||
prepares every explicitly selected binding profile before a run begins, so a
|
||||
missing explicit profile fails before stage execution. Calls record the profile
|
||||
actually selected by PromptKit. The recorder trims and deduplicates non-secret
|
||||
profile identity, provider, model, selected backend ID, and effective reasoning
|
||||
values for manifest use. Entries that differ in backend or reasoning remain
|
||||
distinct and deterministically ordered. Endpoint-only profiles retain an empty
|
||||
backend ID, which the published JSON omits. Successful completion responses and
|
||||
recorded profile manifests identify the adapter provider as `promptkit`.
|
||||
An empty request profile lets the prompt select its configured default. Before a
|
||||
run begins, the CLI asks the adapter to inspect every explicit profile on the
|
||||
resolved selected LLM-backed bindings and validators, including inherited
|
||||
pipeline profiles. Inspection resolves the profile and its selected backend and
|
||||
target without loading a prompt, reading credentials, admitting capacity, or
|
||||
contacting a provider, so a missing or invalid explicit profile fails before
|
||||
stage execution while a valid `api_key_env` may remain unset. Calls record the
|
||||
profile actually selected by PromptKit. The recorder trims and deduplicates
|
||||
non-secret profile identity, provider, model, selected backend ID, and
|
||||
effective reasoning values for manifest use. Entries that differ in backend or
|
||||
reasoning remain distinct and deterministically ordered. Endpoint-only profiles
|
||||
retain an empty backend ID, which the published JSON omits. Successful
|
||||
completion responses and recorded profile manifests identify the adapter
|
||||
provider as `promptkit`.
|
||||
|
||||
The CLI's preparation-only engine and the production adapter use the same
|
||||
conversion helper to register the optional conventional `local` backend.
|
||||
Preflight therefore resolves the same backend membership as runtime without
|
||||
performing generation. When the registration is absent, a profile selecting
|
||||
`backend: local` fails preparation instead of falling back to a built-in or
|
||||
endpoint-only target.
|
||||
The CLI's profile-inspection engine and the production adapter use the same
|
||||
profile-source construction to apply the configured profile directory or file,
|
||||
the optional registered fallback profile assets, and the optional conventional
|
||||
`local` backend. Preflight therefore resolves the same profile sources and
|
||||
backend membership as runtime without performing generation. Fallback assets
|
||||
are mounted only when at least one source is registered. The production D&D
|
||||
registrar contributes its `dnd-extraction` fallback, and the maintained D&D
|
||||
prompts select that logical ID by default. PromptKit owns source precedence and
|
||||
profile parsing: an operator-provided matching profile takes precedence over a
|
||||
fallback profile without Notarius merging either document.
|
||||
When the registration is absent, a profile selecting `backend: local` fails
|
||||
inspection instead of falling back to a built-in or endpoint-only target.
|
||||
|
||||
Before execution, the adapter also contributes a non-secret checkpoint
|
||||
fingerprint for the effective PromptKit profile source. It combines the
|
||||
identity of PromptKit's compiled-in profile catalog with a deterministic digest
|
||||
of every YAML profile in the configured profile directory, or of the configured
|
||||
profile file. The fingerprint contains neither profile content nor source
|
||||
paths. It covers both explicit binding profiles and prompt-selected defaults,
|
||||
so changing a model or other profile setting cannot reuse checkpoints created
|
||||
under the prior profile source. This cache identity is independent of durable
|
||||
profile file, and a deterministic digest of the flattened fallback profile
|
||||
assets. The fingerprint contains neither profile content nor source paths. It
|
||||
covers inherited pipeline profiles, explicit binding profiles, and
|
||||
prompt-selected defaults, so changing a model or other profile setting cannot
|
||||
reuse checkpoints created under the
|
||||
prior profile source. This cache identity is independent of durable
|
||||
profile provenance: run manifests continue to list only profiles actually
|
||||
observed during LLM calls. When the local backend is registered, a second
|
||||
fingerprint hashes its trimmed endpoint behind a stable marker. Changing that
|
||||
@@ -103,10 +118,12 @@ only by the Notarius scheduler.
|
||||
|
||||
## Prompt And Schema Assets
|
||||
|
||||
An `AssetRegistry` collects prompt and schema filesystems from production module
|
||||
families. It flattens registered roots into the PromptKit filesystems and
|
||||
rejects invalid roots, unreadable assets, duplicate paths, and missing prompt
|
||||
or schema files during preparation. The framework’s `promptfs` helper combines
|
||||
An `AssetRegistry` collects prompt, schema, and optional fallback-profile
|
||||
filesystems from production module families. It flattens registered roots into
|
||||
the corresponding PromptKit filesystems and rejects invalid roots, unreadable
|
||||
assets, duplicate paths, and missing prompt or schema files during preparation.
|
||||
Fallback assets receive a safe content digest for checkpoint identity; raw
|
||||
paths and bytes are never included. The framework’s `promptfs` helper combines
|
||||
module-owned prompt files with reusable domain fragments without making the
|
||||
framework depend on D&D content.
|
||||
|
||||
@@ -155,10 +172,12 @@ than output-validation failures.
|
||||
When PromptKit rejects backend admission before generation, the adapter maps
|
||||
`promptkit.ErrCapacityExceeded` to
|
||||
`contracts.ErrLLMCapacityExceeded`, retaining prompt context and a redacted
|
||||
upstream diagnostic without exposing the PromptKit sentinel as a framework
|
||||
contract. A canceled caller context takes precedence. The adapter does not
|
||||
retry capacity failures; the pipeline's existing binding attempt policy sees
|
||||
the operational error and decides whether to rerun the complete operation.
|
||||
upstream diagnostic without exposing the PromptKit sentinel or capacity-error
|
||||
type as a framework contract. When supplied, the normalized selected backend
|
||||
ID appears only in that safe application-owned diagnostic context. A canceled
|
||||
caller context takes precedence. The adapter does not retry capacity failures;
|
||||
the pipeline's existing binding attempt policy sees the operational error and
|
||||
decides whether to rerun the complete operation.
|
||||
|
||||
Prompt-declared repair is executed within PromptKit’s structured-output flow.
|
||||
The current production D&D prompt manifests set repair attempts to zero. That
|
||||
|
||||
@@ -12,9 +12,15 @@ exceptions. See [D&D Module Internals](dnd.md) rather than adding them here.
|
||||
|
||||
A module is a typed implementation registered for one pipeline stage. Its
|
||||
`ModuleSpec` is the public-to-the-framework declaration of its stable key,
|
||||
stage, required and provided capabilities, artifact kind, and accepted
|
||||
reference slots. The framework uses that declaration to resolve a configured
|
||||
binding before it builds the implementation.
|
||||
stage, execution class, required and provided capabilities, artifact kind, and
|
||||
accepted reference slots. The execution class states whether a module is
|
||||
`deterministic` or `llm_backed`; registries retain it for catalog inspection and
|
||||
resolved-pipeline debug data without constructing the module. The framework
|
||||
uses the declaration to resolve a configured binding before it builds the
|
||||
implementation. After selection, the resolver applies profile inheritance only
|
||||
to bindings whose declared execution class is `llm_backed` and rejects a
|
||||
binding-specific profile on a deterministic module. The user-facing precedence
|
||||
contract belongs in [Configuration](../config.md#pipelines).
|
||||
|
||||
Implementations that accept options must provide both an option validator and
|
||||
a builder. The validator is used while resolving configuration; the builder
|
||||
@@ -47,7 +53,7 @@ Production composition is intentionally split by family:
|
||||
- The Seriatim registrar provides the transcript input adapter. Its external
|
||||
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
|
||||
- The D&D registrar provides its codecs, extractors, mergers, normalizers,
|
||||
validators, prompt assets, and default chains. Its behavioral conventions
|
||||
validators, prompt assets, fallback profile asset, and default chains. Its behavioral conventions
|
||||
are documented in [D&D Module Internals](dnd.md).
|
||||
|
||||
The CLI owns the composition that invokes these registrars. A module package
|
||||
@@ -59,10 +65,10 @@ packages depend on production extensions.
|
||||
1. Choose the pipeline stage and the typed artifact boundary. Put external
|
||||
input or durable artifact formats in the relevant integration contract,
|
||||
not in this guide or in a private LLM response type.
|
||||
2. Define a stable `ModuleSpec` with the exact capabilities and reference
|
||||
slots needed for the operation. Model a producer/consumer handoff as an
|
||||
artifact-compatible slot; configuration then chooses an external file or a
|
||||
generated binding.
|
||||
2. Define a stable `ModuleSpec` with an explicit execution class, the exact
|
||||
capabilities, and reference slots needed for the operation. Model a
|
||||
producer/consumer handoff as an artifact-compatible slot; configuration
|
||||
then chooses an external file or a generated binding.
|
||||
3. Implement strict option decoding, construction, and the typed stage
|
||||
interface. Preserve caller ownership: do not retain mutable request data
|
||||
and return defensive copies where an implementation exposes stored data.
|
||||
|
||||
@@ -34,6 +34,11 @@ requested lanes where that is supported, resolves validator chains, checks
|
||||
module capabilities and typed artifact compatibility, validates options, and
|
||||
assigns a deterministic resolved-composition digest. The resolved pipeline
|
||||
contains bindings and declared reference targets, not external reference bytes.
|
||||
After selection, the resolver applies command, binding, and pipeline profile
|
||||
precedence to LLM-backed bindings and validators only; prompt defaults remain
|
||||
an empty resolved binding profile. Deterministic bindings remain profile-free.
|
||||
These effective values are part of the digest, so execution and checkpoint
|
||||
consumers do not repeat profile inheritance.
|
||||
Configuration resolution supplies the selected profile and catalog; see
|
||||
[Configuration Internals](configuration.md).
|
||||
|
||||
|
||||
@@ -42,6 +42,42 @@ evidence publication. Apply an appropriate umask and output-root access policy
|
||||
before enabling that option; the requested output modes alone may not be
|
||||
suitable for transcript-bearing bundles.
|
||||
|
||||
## PromptKit Profile Deployment
|
||||
|
||||
Profile deployment has four distinct layers:
|
||||
|
||||
| Layer | Owner | Operational role |
|
||||
| --- | --- | --- |
|
||||
| Prompts and schemas | Notarius module families | Embedded request and structured-output definitions. They are not deployment profile files. |
|
||||
| Fallback profiles | Notarius module families | Embedded application defaults, including D&D's `dnd-extraction` profile. |
|
||||
| Built-in profiles | PromptKit | Upstream catalog entries available when no higher-precedence source defines an ID. |
|
||||
| Operator profiles | Deployment filesystem | Complete environment-specific definitions selected by `promptkit.profile_file` or `promptkit.profile_dir`. |
|
||||
|
||||
The maintained D&D pipeline uses the workload ID `dnd-extraction`. The
|
||||
embedded fallback makes that ID usable without an operator file. Production,
|
||||
development, and local deployments can each install a different complete
|
||||
definition for the same ID, retaining the pipeline while choosing their own
|
||||
model, backend, timeout, or reasoning policy. An operator definition wins over
|
||||
the fallback; it is not merged with it. The configuration field and full
|
||||
precedence rules are owned by [Configuration](config.md#promptkit-profiles).
|
||||
|
||||
Use a profile source owned by the service account, keep it readable only by
|
||||
the intended operator, and supply provider credentials through the service
|
||||
environment—not in the Notarius configuration or profile YAML. The maintained
|
||||
[operator profile](../examples/profiles/dnd-extraction.yml) is secret-free and
|
||||
can be copied as a format starting point. Validate a deployment without a
|
||||
provider call or credentials:
|
||||
|
||||
~~~sh
|
||||
notarius config validate --config /etc/notarius/config.yml --pipeline dnd-session
|
||||
~~~
|
||||
|
||||
Profile paths are currently resolved from the process working directory, not
|
||||
from the configuration file. The complete example's
|
||||
`./examples/profiles/dnd-extraction.yml` path is valid for a repository-root
|
||||
invocation only. Use absolute paths such as
|
||||
`/etc/notarius/profiles/dnd-extraction.yml` for services and containers.
|
||||
|
||||
## Run Lifecycle
|
||||
|
||||
Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads
|
||||
@@ -219,7 +255,7 @@ Provider execution settings and the generation timeout come from the selected
|
||||
PromptKit profile. The invocation-only **--reasoning-effort** and
|
||||
**--clear-reasoning-effort** controls may replace or clear that profile setting
|
||||
for all LLM-backed calls in one run without changing the profile. PromptKit
|
||||
v0.3.0 does not add a provider retry loop. Notarius binding retries rerun the
|
||||
v0.5.0 does not add a provider retry loop. Notarius binding retries rerun the
|
||||
complete module operation and validation chain as defined by
|
||||
[module bindings](config.md#module-bindings-and-validators).
|
||||
|
||||
|
||||
@@ -88,55 +88,6 @@ not as committed release dates.
|
||||
secret unless a concrete multi-tenant or privacy requirement justifies
|
||||
sacrificing deterministic identity across installations.
|
||||
|
||||
### Pipeline-Level LLM Profile Defaults
|
||||
|
||||
- Add an optional pipeline-level `llm_profile` default so an operator can
|
||||
select one PromptKit execution policy for the pipeline without repeating the
|
||||
same profile ID on every LLM-backed module binding.
|
||||
- Apply the following precedence consistently: an explicit run-wide
|
||||
`--llm-profile` override, then a binding-specific `llm_profile`, then the
|
||||
pipeline-level default, then the prompt definition's embedded
|
||||
`default_profile`.
|
||||
- Apply inheritance only to bindings whose resolved modules are LLM-backed,
|
||||
including applicable chunk, extraction, merge, normalization, and validation
|
||||
bindings. Do not attach an inherited profile to deterministic modules or
|
||||
weaken existing validation that rejects profiles where generation is not
|
||||
supported.
|
||||
- Resolve inherited profiles before effective-pipeline validation, digest and
|
||||
checkpoint construction, execution, and provenance capture. Validate every
|
||||
resulting explicit profile ID against the selected PromptKit profile source,
|
||||
and ensure manifests and debug output report the profile actually selected
|
||||
for each generation target.
|
||||
- Preserve binding-specific profiles as intentional exceptions for modules
|
||||
that require a different quality, latency, cost, provider, or reasoning
|
||||
policy. Preserve `--llm-profile` as the convenient highest-precedence
|
||||
experiment or incident-response override for an entire run.
|
||||
- Treat PromptKit profiles as execution-policy configuration and prefer stable,
|
||||
workload-oriented IDs such as `dnd-extraction` over model names. A pipeline
|
||||
should express the kind of work it performs rather than encode a particular
|
||||
provider, model, or deployment environment.
|
||||
- Establish deployment-managed PromptKit profile files as the recommended
|
||||
environment-specific configuration mechanism. Production, development, and
|
||||
local deployments may each define the same logical `dnd-extraction` ID with
|
||||
different model and generation settings, while retaining one unchanged
|
||||
Notarius pipeline definition.
|
||||
- Keep deployment profiles distinct from both Notarius prompt assets embedded
|
||||
in the application binary and PromptKit's built-in profile catalog. Document
|
||||
that `promptkit.profile_dir` or `promptkit.profile_file` selects an external
|
||||
filesystem source whose definitions overlay PromptKit built-ins, and
|
||||
recommend application-owned IDs rather than silently replacing built-in
|
||||
profile IDs.
|
||||
- Document an operator-friendly layout in which the Notarius configuration and
|
||||
a profile subdirectory are deployed together. Until profile paths are
|
||||
explicitly resolved relative to the configuration file, clearly state that
|
||||
relative paths use the process working directory and recommend absolute
|
||||
paths for services and containers.
|
||||
- Update the configuration reference, operations guide, PromptKit integration
|
||||
boundary, relevant internal configuration and pipeline documentation, and
|
||||
maintained D&D examples together. Include one concrete external-profile
|
||||
example and explain the profile precedence and environment-neutral pipeline
|
||||
pattern without duplicating PromptKit's complete profile-format reference.
|
||||
|
||||
### Raise The Default Application-Wide LLM Limit
|
||||
|
||||
- Raise the default `concurrency.total_llm` value from 1 to 16 so ordinary
|
||||
|
||||
586
docs/roadmap/implementation.md
Normal file
586
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,586 @@
|
||||
# PromptKit v0.5 Implementation Plan
|
||||
|
||||
## Objective
|
||||
|
||||
Implement the target state in
|
||||
[PromptKit v0.5 Integration And LLM Profile Policy](promptkit.md). Each numbered
|
||||
stage is intended to be one implementation prompt for a GPT-5.6-Terra coding
|
||||
agent. Complete stages in order and leave the repository buildable, tested, and
|
||||
internally coherent after every stage.
|
||||
|
||||
Follow [Architecture](../policy/architecture.md),
|
||||
[Testing Policy](../policy/testing.md), and
|
||||
[Documentation Policy](../policy/documentation.md) throughout. Preserve
|
||||
unrelated user changes. Use `apply_patch` for source and documentation edits,
|
||||
run `gofmt` on changed Go files, and add only tests that protect the behaviors
|
||||
and risks assigned to that stage.
|
||||
|
||||
Do not implement the separate deterministic session-ID or default-concurrency
|
||||
roadmap items as part of this plan. Do not perform paid or credentialed LLM
|
||||
calls.
|
||||
|
||||
## Background Summary
|
||||
|
||||
Notarius currently pins PromptKit v0.3.0, calls `Prepare` and then `Run` for one
|
||||
completion, validates profiles through a synthetic prompt, has no application
|
||||
fallback profile source, and accepts LLM profiles only at individual bindings
|
||||
or through the run-wide CLI override. PromptKit v0.5.0 is source-compatible
|
||||
with the current tree; a temporary v0.5.0 module override has already passed
|
||||
`go test ./...`.
|
||||
|
||||
The implementation must nevertheless treat the upstream optional-parameter
|
||||
change as intentional: unset `temperature`, `max_tokens`, and `top_p` remain
|
||||
unset and are omitted from compatible provider requests. Do not restore the old
|
||||
implicit `top_p: 1` default.
|
||||
|
||||
## Stage 1: Upgrade The PromptKit Dependency
|
||||
|
||||
### Goal
|
||||
|
||||
Establish a clean PromptKit v0.5.0 baseline before adopting its new APIs.
|
||||
|
||||
### Work
|
||||
|
||||
- Update `go.mod` and `go.sum` from PromptKit v0.3.0 to v0.5.0 and run
|
||||
`go mod tidy`.
|
||||
- Change the PromptKit built-in profile-catalog marker in
|
||||
`internal/framework/llm/promptkit_profile_fingerprint.go` to identify
|
||||
v0.5.0. This deliberately invalidates LLM checkpoints tied to the prior
|
||||
catalog identity.
|
||||
- Review PromptKit-facing compile errors or test failures against the v0.4.0
|
||||
and v0.5.0 release guides. Do not adopt prepared execution, inspection, or
|
||||
fallback profiles in this stage.
|
||||
- Replace the existing test assertion for one exact built-in fingerprint hash
|
||||
with durable assertions that the fingerprint is deterministic, non-empty,
|
||||
non-secret, and changes when a semantic profile source changes. Do not add a
|
||||
new version-constant or exact-hash change detector.
|
||||
- Update `docs/integrations/pkg-promptkit.md` to pin and link v0.5.0 and state
|
||||
the implemented dependency-level behavior: unset optional sampling controls
|
||||
are provider defaults. Do not document later stages as implemented.
|
||||
- Update any other canonical text that explicitly claims the dependency is
|
||||
v0.3.0, but defer descriptions of unimplemented v0.5 APIs.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- `go test ./internal/framework/llm ./internal/cli`
|
||||
- `go test ./...`
|
||||
- `go vet ./...`
|
||||
- `go build ./cmd/notarius`
|
||||
- `rg -n 'promptkit v0\.3\.0|promptkit@v0\.3\.0|PromptKit v0\.3\.0' .`
|
||||
- `git diff --check`
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- The repository directly pins v0.5.0 and all default offline checks pass.
|
||||
- The profile-source fingerprint identifies the new upstream catalog without a
|
||||
brittle literal-hash test.
|
||||
- Current documentation no longer identifies v0.3.0 as the supported version.
|
||||
|
||||
## Stage 2: Execute One Frozen Prepared Snapshot
|
||||
|
||||
### Goal
|
||||
|
||||
Make Notarius debug details and generation use one exact PromptKit preparation.
|
||||
|
||||
### Work
|
||||
|
||||
- Refactor `PromptKitClient.CompleteStructured` to call
|
||||
`PrepareExecution`, immediately defer `Discard`, obtain a caller-owned
|
||||
`Details` value, and execute with `RunPrepared`.
|
||||
- Preserve the existing Notarius request mapping, cancellation precedence,
|
||||
validation classification, raw structured bytes, response decoding,
|
||||
profile recording, usage reporting, and credential redaction.
|
||||
- Ensure every preparation, execution, validation, empty-result, and decode
|
||||
error retains useful Notarius prompt context without exposing prepared handle
|
||||
state or secrets.
|
||||
- Use `errors.As` to obtain `*promptkit.CapacityError` on admission rejection.
|
||||
Preserve `contracts.ErrLLMCapacityExceeded` as the stable classification and
|
||||
add a nonblank backend ID only to safe application-owned diagnostic context.
|
||||
Do not expose `promptkit.CapacityError` outside the LLM adapter.
|
||||
- Update `docs/internal/llm.md` and the implemented-mechanics portion of
|
||||
`docs/integrations/pkg-promptkit.md` to describe the single frozen execution
|
||||
snapshot and structured capacity adaptation.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Adapt existing PromptKit client tests to the prepared-execution path.
|
||||
- Retain or add one behavioral test proving that the debug prompt details match
|
||||
the request actually passed to generation when a backing prompt source could
|
||||
otherwise change between independent preparations. Test the resulting
|
||||
snapshot consistency, not a private helper call count.
|
||||
- Retain capacity tests proving `errors.Is` reaches
|
||||
`contracts.ErrLLMCapacityExceeded`, the selected backend can appear in safe
|
||||
diagnostic context, and provider calls are not made after rejected
|
||||
admission.
|
||||
- Run `go test ./internal/framework/llm` and
|
||||
`go test -race ./internal/framework/llm`.
|
||||
- Run `go test ./...` and `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- `CompleteStructured` no longer calls independent `Prepare` and `Run`
|
||||
operations for one request.
|
||||
- Debug prompt material and generation result originate from the same frozen
|
||||
PromptKit snapshot.
|
||||
- Capacity remains a provider-neutral Notarius error classification.
|
||||
|
||||
## Stage 3: Replace Synthetic Profile Validation With Inspection
|
||||
|
||||
### Goal
|
||||
|
||||
Validate profiles through PromptKit's exact profile-inspection boundary and
|
||||
centralize engine profile-source construction.
|
||||
|
||||
### Work
|
||||
|
||||
- Introduce a small provider-adapter-owned profile inspection or validation
|
||||
function in `internal/framework/llm`. Its public internal signature must use
|
||||
Notarius-owned configuration and result/error types rather than returning
|
||||
PromptKit types to the CLI.
|
||||
- Share the code that applies `profile_dir`, `profile_file`, and registered
|
||||
backend options between the production PromptKit engine and the inspection
|
||||
engine. Preserve the mutual-exclusion and local-backend rules.
|
||||
- Change CLI explicit-profile preflight to use `Engine.InspectProfile` through
|
||||
that LLM boundary.
|
||||
- Remove `profileCheckPromptID`, `profileCheckPromptFS`, the `testing/fstest`
|
||||
production dependency, and the synthetic `Prepare` request.
|
||||
- Preserve distinct, useful errors for an absent profile, invalid profile,
|
||||
unknown backend registration, cancellation, and invalid profile source.
|
||||
- Do not require `api_key_env` to be populated during configuration validation.
|
||||
Inspection may report credential requirements internally, but actual
|
||||
preparation remains responsible for credential availability before a model
|
||||
call.
|
||||
- Update current-behavior sections in `docs/internal/cli.md` and
|
||||
`docs/internal/llm.md`. Keep field definitions in `docs/config.md`.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Replace synthetic-prompt tests with profile inspection tests covering:
|
||||
configured local backend success; missing local backend failure; absent
|
||||
profile; malformed profile; and an otherwise valid profile whose credential
|
||||
environment variable is intentionally unset.
|
||||
- Prove validation performs no provider HTTP call and remains offline.
|
||||
- Run `go test ./internal/framework/llm ./internal/cli` and `go test ./...`.
|
||||
- Run `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- No production synthetic profile-check prompt remains.
|
||||
- Profile validation uses the same ordinary profile source and backend
|
||||
registrations as execution.
|
||||
- Configuration validation succeeds for structurally valid profiles without
|
||||
reading credential values.
|
||||
|
||||
## Stage 4: Add Application Fallback Profile Asset Plumbing
|
||||
|
||||
### Goal
|
||||
|
||||
Allow module families to register application-owned fallback profile YAML
|
||||
without placing domain policy in generic LLM code.
|
||||
|
||||
### Work
|
||||
|
||||
- Extend `internal/framework/llm.AssetRegistry` with a separate fallback
|
||||
profile source collection, registration method, flattened filesystem, and
|
||||
safe content digest.
|
||||
- Reuse the existing asset-source path validation and flattening behavior where
|
||||
appropriate. Reject invalid roots, unreadable assets, and duplicate flattened
|
||||
paths. Do not parse PromptKit profile YAML in Notarius.
|
||||
- Add `promptkit.WithFallbackProfileFS` to production engine options only when
|
||||
at least one fallback profile source is registered.
|
||||
- Supply the identical assembled fallback source to the profile-inspection
|
||||
engine. Adjust CLI composition so pipeline-aware profile validation can use
|
||||
the production LLM asset registry without exposing PromptKit types.
|
||||
- Extend profile-source checkpoint identity to include the exact fallback
|
||||
profile asset digest in addition to the PromptKit catalog marker and operator
|
||||
source. Keep the resulting fingerprint hash-only and path/content/credential
|
||||
free.
|
||||
- Keep operator source precedence owned by PromptKit. Do not implement profile
|
||||
merging or duplicate PromptKit source resolution in Notarius.
|
||||
- Update `docs/internal/llm.md` only for the new implemented generic asset and
|
||||
fingerprint mechanics. No domain fallback exists until Stage 5.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Add focused AssetRegistry tests for successful flattening, invalid roots,
|
||||
duplicate paths, and hash changes when fallback bytes change.
|
||||
- Add adapter-level tests showing that the fallback filesystem reaches both
|
||||
execution construction and inspection construction.
|
||||
- Extend checkpoint tests to prove fallback content changes profile-source
|
||||
identity without exposing raw YAML or paths. Use relational comparisons, not
|
||||
a fixed hash literal.
|
||||
- Run `go test ./internal/framework/llm ./internal/cli` and `go test ./...`.
|
||||
- Run `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Generic plumbing can carry application fallback profiles while remaining
|
||||
unaware of D&D IDs or model settings.
|
||||
- Inspection, execution, and checkpoint identity use the same fallback asset
|
||||
source.
|
||||
|
||||
## Stage 5: Adopt The D&D `dnd-extraction` Fallback
|
||||
|
||||
### Goal
|
||||
|
||||
Give the D&D module family one stable embedded workload profile that operators
|
||||
can replace.
|
||||
|
||||
### Work
|
||||
|
||||
- Add a D&D-owned embedded PromptKit profile asset with ID `dnd-extraction`
|
||||
under `internal/modules/dnd`. Use the exact baseline defined in
|
||||
`promptkit.md`: OpenRouter, `openai/gpt-5.6-luna`, no explicit reasoning
|
||||
effort, a 240-second timeout, flex service tier, and no selected temperature,
|
||||
token limit, or `top_p`. The omitted reasoning value intentionally allows
|
||||
OpenAI's backend to apply its `medium` default.
|
||||
- Register the profile filesystem from the D&D registrar through the generic
|
||||
fallback profile asset boundary. Keep D&D policy out of
|
||||
`internal/framework/llm` and the CLI composition root.
|
||||
- Change every maintained D&D LLM prompt definition—including scene chunking,
|
||||
all D&D extractors, and NPC normalization—from the model-named default to
|
||||
`default_profile: dnd-extraction`.
|
||||
- Add an integration-level profile-resolution test proving that:
|
||||
- the fallback resolves when no operator source defines the ID;
|
||||
- a valid operator profile with the same ID wins completely; and
|
||||
- an invalid matching operator profile fails rather than falling through.
|
||||
- Test through Notarius's assembled production assets and PromptKit boundary;
|
||||
do not duplicate every upstream source-precedence case.
|
||||
- Update the implemented profile ownership and prompt-default behavior in
|
||||
`docs/internal/dnd.md`, `docs/internal/llm.md`, and
|
||||
`docs/integrations/pkg-promptkit.md`. Defer the complete operator walkthrough
|
||||
and examples to Stage 10.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Run focused D&D prompt preparation tests and the production composition
|
||||
tests.
|
||||
- Run `go test ./internal/modules/dnd/... ./internal/framework/llm
|
||||
./internal/cli`.
|
||||
- Run `go test ./...`.
|
||||
- Verify `rg -n 'default_profile: gemini-2-flash' internal/modules/dnd`
|
||||
returns no matches.
|
||||
- Run `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- All maintained D&D prompts use the application-owned logical profile ID.
|
||||
- The fallback works without an operator profile and remains authoritatively
|
||||
overridable by a matching valid operator definition.
|
||||
|
||||
## Stage 6: Introduce Module Execution-Class Metadata
|
||||
|
||||
### Goal
|
||||
|
||||
Make each production module's ability to use an LLM statically discoverable
|
||||
without yet changing profile inheritance.
|
||||
|
||||
### Work
|
||||
|
||||
- Add `ExecutionClass contracts.ExecutionClass` to `pipeline.ModuleSpec` and
|
||||
preserve it through normalization, cloning, catalogs, registries, JSON/debug
|
||||
views, and lookup helpers.
|
||||
- In this transitional stage only, allow an omitted execution class to
|
||||
normalize to deterministic so existing test-only fixtures can be migrated in
|
||||
Stage 7 without breaking the repository midway.
|
||||
- Explicitly classify every production module:
|
||||
- D&D scene chunking, every D&D extractor, and D&D NPC normalization as
|
||||
`llm_backed`;
|
||||
- all other current production input, chunk, merge, normalize, and output
|
||||
modules as `deterministic`.
|
||||
- Update production module specification tests and production catalog tests to
|
||||
assert the semantic class alongside stage, artifact kind, and capabilities.
|
||||
- Add catalog lookup support needed by later resolution to retrieve a selected
|
||||
module's execution class by stage and key without constructing it.
|
||||
- Do not implement pipeline-level profile inheritance or reject deterministic
|
||||
profiles yet.
|
||||
- Update `docs/internal/modules.md` and `docs/internal/dnd.md` to identify
|
||||
execution class as registered module metadata, while noting only implemented
|
||||
uses.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Run module registration/spec tests across generic, Seriatim, and D&D
|
||||
families.
|
||||
- Run `go test ./internal/framework/pipeline ./internal/modules/...`.
|
||||
- Run `go test ./...` and `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Every production module has an explicit correct execution class.
|
||||
- Catalog consumers can retrieve that class without a concrete module
|
||||
instance.
|
||||
- Test-only omitted classes remain the only temporary compatibility behavior.
|
||||
|
||||
## Stage 7: Enforce Execution Metadata And Remove Runtime Probing
|
||||
|
||||
### Goal
|
||||
|
||||
Finish the execution-class contract so missing metadata cannot cause future
|
||||
profile drift.
|
||||
|
||||
### Work
|
||||
|
||||
- Update every framework, CLI, and integration test module specification to
|
||||
declare an explicit execution class appropriate to the fake behavior.
|
||||
- Change module-spec validation so an empty or unsupported execution class is a
|
||||
registration error. Remove the transitional deterministic default from
|
||||
Stage 6.
|
||||
- Replace the chunk runner's special `ChunkExecutionClassProvider` probe with
|
||||
specification-derived behavior. Remove the now-redundant provider interface,
|
||||
implementation methods, and tests when they have no remaining consumer.
|
||||
- Ensure chunk producer provenance remains unchanged: it records a non-empty
|
||||
effective binding profile for an LLM-backed chunker, while a deterministic
|
||||
chunker records no profile. A profile selected only through the prompt
|
||||
default remains represented by PromptKit's actual-profile manifest rather
|
||||
than being invented as an explicit chunk binding.
|
||||
- Review helper constructors and fixtures for opportunities to set execution
|
||||
class once without obscuring the class under test. Do not introduce an
|
||||
elaborate test-spec framework.
|
||||
- Update internal documentation if the removal changes any described runtime
|
||||
mechanics.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Add or retain focused registration tests for missing and invalid execution
|
||||
classes.
|
||||
- Retain chunk-plan provenance tests for LLM-backed and deterministic
|
||||
chunkers.
|
||||
- Run `go test ./internal/framework/pipeline ./internal/modules/...`.
|
||||
- Run `go test ./...`, `go vet ./...`, and `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- No registered module specification relies on an implicit execution class.
|
||||
- Pipeline metadata, not a concrete runtime type assertion, owns module
|
||||
execution classification.
|
||||
|
||||
## Stage 8: Resolve Programmatic Pipeline Profile Defaults
|
||||
|
||||
### Goal
|
||||
|
||||
Implement profile inheritance and precedence inside the pipeline resolver
|
||||
before exposing the field through YAML configuration.
|
||||
|
||||
### Work
|
||||
|
||||
- Add an optional trimmed `LLMProfile` field to
|
||||
`pipeline.PipelineProfile`. Add a non-empty runtime override field to
|
||||
`pipeline.ResolveOptions` so all precedence decisions occur in the resolver
|
||||
rather than through pre-resolution mutation.
|
||||
- After module selection, `--only` filtering, default validator-chain
|
||||
selection, and validator compatibility resolution, apply effective profiles
|
||||
to every selected input, chunk, extract, merge, normalize, output, and
|
||||
validator binding according to the precedence in `promptkit.md`.
|
||||
- Apply profiles only when the selected module or validator execution class is
|
||||
`llm_backed`.
|
||||
- Reject a binding-specific `llm_profile` on any deterministic module or
|
||||
validator. Do not reject or inspect an unused pipeline default when no
|
||||
selected LLM-backed binding consumes it.
|
||||
- Leave an LLM-backed binding empty when no CLI, binding, or pipeline profile is
|
||||
selected so PromptKit can use the prompt's `default_profile`.
|
||||
- Store the effective values on resolved bindings before digest construction.
|
||||
Do not add a second inheritance decision to execution.
|
||||
- Ensure semantically equivalent repeated binding profiles and one inherited
|
||||
default produce the same resolved pipeline digest. Ensure any changed
|
||||
effective profile changes the digest.
|
||||
- Do not modify file configuration or CLI parsing in this stage.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Add pipeline package tests for the complete precedence matrix:
|
||||
runtime override; binding-specific exception; pipeline default; prompt
|
||||
fallback; and deterministic bindings.
|
||||
- Cover default and explicitly configured validator chains, all relevant stage
|
||||
categories, `--only` lane selection, unused defaults, deterministic-profile
|
||||
rejection, and semantic digest equivalence.
|
||||
- Prefer table-driven package-level tests over assertions on private traversal
|
||||
helpers.
|
||||
- Run `go test ./internal/framework/pipeline` and `go test ./...`.
|
||||
- Run `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Programmatic pipelines resolve one canonical effective profile policy.
|
||||
- Only LLM-backed resolved bindings can contain a profile.
|
||||
- Runtime override, binding, pipeline, and prompt precedence is unambiguous and
|
||||
digest-stable.
|
||||
|
||||
## Stage 9: Expose Pipeline Defaults Through Configuration And CLI
|
||||
|
||||
### Goal
|
||||
|
||||
Make the profile-default workflow available to operators while preserving
|
||||
validation and override behavior.
|
||||
|
||||
### Work
|
||||
|
||||
- Add optional `pipelines.<id>.llm_profile` support to the version 4 file
|
||||
configuration model. Use presence-aware decoding so an explicitly set blank
|
||||
value is rejected, while omission remains valid.
|
||||
- Preserve the field through file application, configuration cloning,
|
||||
effective configuration, and programmatic profile copies without aliasing or
|
||||
trimming drift.
|
||||
- Remove `applyLLMProfileOverride`. Pass the CLI override through the resolver's
|
||||
runtime-override input so deterministic bindings are never populated.
|
||||
- Update effective profile-ID collection to cover every selected LLM-backed
|
||||
module stage and LLM-backed validator, including future LLM-backed input and
|
||||
output modules. Do not inspect deterministic or unselected profiles.
|
||||
- Ensure `run`, `config validate --pipeline`, resume/checkpoint identity, and
|
||||
relevant dry preflight paths all use the same resolved effective profiles.
|
||||
- Preserve `--llm-profile` as the highest-precedence non-empty run-wide
|
||||
override and preserve binding-specific profiles as exceptions when no CLI
|
||||
override is present.
|
||||
- Do not increment the configuration version.
|
||||
- Update current configuration and CLI contracts in `docs/config.md` and
|
||||
`docs/cli.md` in the same stage. Link to operations for the deployment
|
||||
workflow rather than duplicating it prematurely.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Add file-config tests for omission, trimming, explicit blank rejection,
|
||||
unknown-key behavior, cloning, and round-trip application.
|
||||
- Add effective-config and CLI contract tests for precedence, LLM-only
|
||||
application, inherited-profile inspection failure before factory execution,
|
||||
`--only`, and digest changes.
|
||||
- Retain offline operation and do not require credentials for
|
||||
`config validate --pipeline`.
|
||||
- Run `go test ./internal/core/config ./internal/framework/pipeline
|
||||
./internal/cli`.
|
||||
- Run `go test ./...`, `go vet ./...`, and `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Operators can select `dnd-extraction` once per pipeline.
|
||||
- Configuration and CLI paths share the resolver's precedence policy.
|
||||
- Unknown effective profiles fail preflight, while deterministic and unused
|
||||
profiles do not cause spurious inspection.
|
||||
|
||||
## Stage 10: Complete Operator Documentation, Examples, And Decision Record
|
||||
|
||||
### Goal
|
||||
|
||||
Make the implemented workflow understandable, copyable, and maintainable
|
||||
without duplicating canonical facts.
|
||||
|
||||
### Work
|
||||
|
||||
- Create an ADR using the next sequential number for the durable decision to
|
||||
use workload-oriented pipeline defaults with operator-overridable application
|
||||
fallback profiles. Record context, decision, alternatives, and consequences;
|
||||
do not turn the ADR into a field reference or implementation log.
|
||||
- Complete `docs/config.md` as the canonical owner of profile-source fields,
|
||||
`pipelines.<id>.llm_profile`, validation, and precedence.
|
||||
- Complete `docs/operations.md` with an operator workflow that distinguishes
|
||||
Notarius embedded prompts, Notarius fallback profiles, PromptKit built-ins,
|
||||
and deployment filesystem profiles. Include production/development/local use
|
||||
of the same `dnd-extraction` ID, credential handling, absolute-path guidance,
|
||||
and the fact that current relative profile paths use the process working
|
||||
directory rather than the configuration file's directory.
|
||||
- Complete `docs/integrations/pkg-promptkit.md` with the v0.5.0 boundary,
|
||||
prepared execution, inspection, fallback and ordinary source precedence,
|
||||
optional provider controls, capacity adaptation, and compatibility policy.
|
||||
- Update `docs/internal/configuration.md`, `docs/internal/pipeline.md`,
|
||||
`docs/internal/cli.md`, `docs/internal/llm.md`, `docs/internal/modules.md`, and
|
||||
`docs/internal/dnd.md` only for their owned implementation details. Link to
|
||||
canonical configuration, operations, and upstream format contracts rather
|
||||
than restating them.
|
||||
- Keep exactly the existing two D&D configuration examples. Add
|
||||
`llm_profile: dnd-extraction` to the minimal and complete pipelines and remove
|
||||
the now-redundant model-named binding override from the complete example.
|
||||
- Add one secret-free maintained operator profile at
|
||||
`examples/profiles/dnd-extraction.yml`. It should be a complete valid profile
|
||||
for the same logical ID and may mirror the embedded baseline; its purpose is
|
||||
to demonstrate file ownership and format, not claim automatic environment
|
||||
detection. Link it from the configuration and operations documentation.
|
||||
- If the complete example selects the external profile file, use a path that
|
||||
is valid for the documented repository-root invocation and explicitly note
|
||||
the working-directory rule. Keep the minimal example dependent only on the
|
||||
embedded fallback.
|
||||
- Add or extend maintained-example validation so both configuration examples
|
||||
and the profile YAML are checked without generation or credentials.
|
||||
- Remove the now-implemented `Pipeline-Level LLM Profile Defaults` section from
|
||||
`docs/roadmap/future.md`. Preserve the unrelated deterministic session and
|
||||
concurrency items.
|
||||
- Do not delete `promptkit.md` or this implementation plan during the feature
|
||||
implementation; retire them only after post-implementation review.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Run maintained example/configuration tests and relevant CLI help/parser
|
||||
tests.
|
||||
- Run `go test ./...`.
|
||||
- Run `rg -n 'gemini-2-flash' examples docs` and review every remaining match
|
||||
for intentional model-policy or historical context.
|
||||
- Run `rg -n 'v0\.3\.0|profileCheckPrompt|applyLLMProfileOverride' .` and resolve
|
||||
stale production or current-documentation matches.
|
||||
- Verify all new links and `git diff --check`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Every current fact has one canonical documentation owner.
|
||||
- Operators can distinguish and deploy all profile layers without reading Go
|
||||
source.
|
||||
- Both maintained configurations and the maintained external profile are valid,
|
||||
secret-free, and tested offline.
|
||||
- Implemented profile work no longer remains in `future.md`.
|
||||
|
||||
## Stage 11: Final Verification And Quality Review
|
||||
|
||||
### Goal
|
||||
|
||||
Verify the complete migration as one integrated change and correct only defects
|
||||
or omissions found during that review.
|
||||
|
||||
### Work
|
||||
|
||||
- Review the final diff against every acceptance criterion in `promptkit.md`.
|
||||
- Confirm provider-specific PromptKit types remain inside the LLM integration
|
||||
boundary and D&D policy remains inside the D&D module family.
|
||||
- Confirm execution and inspection receive identical ordinary, fallback, and
|
||||
backend configuration.
|
||||
- Confirm no paths, profile YAML, endpoints, credentials, or prepared handle
|
||||
state leak into fingerprints or ordinary diagnostics.
|
||||
- Confirm all production module specs have explicit correct execution classes
|
||||
and every resolved deterministic binding is profile-free.
|
||||
- Confirm prompt default, pipeline default, binding override, and CLI override
|
||||
behavior through representative assembled configurations.
|
||||
- Review tests for redundancy and remove obsolete synthetic-prompt,
|
||||
runtime-probe, exact-hash, or duplicated upstream-behavior tests superseded by
|
||||
stronger contract tests.
|
||||
- Perform an optional manual D&D quality comparison if credentials and an
|
||||
evaluation transcript are deliberately supplied. Record no private input or
|
||||
credential material, and do not make this comparison a completion gate.
|
||||
|
||||
### Validation Commands
|
||||
|
||||
```sh
|
||||
gofmt -w <changed-go-files>
|
||||
go test ./...
|
||||
go test -race ./internal/framework/llm ./internal/core/config ./internal/framework/pipeline ./internal/cli
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Also run focused stale-contract searches:
|
||||
|
||||
```sh
|
||||
rg -n 'gitea.maximumdirect.net/eric/promptkit v0\.3\.0|PromptKit v0\.3\.0' .
|
||||
rg -n 'default_profile: gemini-2-flash|profileCheckPrompt|applyLLMProfileOverride' internal docs examples
|
||||
```
|
||||
|
||||
Review any matches rather than deleting intentional historical references
|
||||
blindly.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- All automated checks pass offline and without real credentials.
|
||||
- The implemented behavior matches `promptkit.md` with no known architecture,
|
||||
provenance, checkpoint, profile-precedence, or documentation gap.
|
||||
- Any optional live evaluation is clearly separate from correctness testing.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The roadmap decisions are sufficient to implement every stage without an
|
||||
additional product or architecture choice.
|
||||
318
docs/roadmap/promptkit.md
Normal file
318
docs/roadmap/promptkit.md
Normal file
@@ -0,0 +1,318 @@
|
||||
# PromptKit v0.5 Integration And LLM Profile Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap defines the target state for upgrading Notarius from PromptKit
|
||||
v0.3.0 to v0.5.0 and adopting the upstream runtime and profile facilities that
|
||||
directly improve Notarius. It also defines the application policy for stable,
|
||||
domain-oriented LLM profile names, operator overrides, pipeline inheritance,
|
||||
profile validation, provider defaults, checkpoint identity, and documentation.
|
||||
|
||||
The ordered work needed to reach this state belongs in
|
||||
[the implementation plan](implementation.md). Current behavior remains defined
|
||||
by the canonical documentation outside `docs/roadmap/` until the corresponding
|
||||
work is implemented.
|
||||
|
||||
## Background
|
||||
|
||||
Notarius currently pins PromptKit v0.3.0. Its adapter prepares a request once
|
||||
for debug material and then independently runs the original request, causing
|
||||
PromptKit to prepare the same logical call a second time. The CLI validates an
|
||||
explicit profile by preparing a synthetic prompt. PromptKit profile selection
|
||||
can be repeated on individual module bindings or replaced for one invocation
|
||||
with `--llm-profile`, but a configured pipeline cannot yet declare one inherited
|
||||
profile policy.
|
||||
|
||||
PromptKit v0.4.0 and v0.5.0 add the upstream boundaries needed to improve these
|
||||
areas:
|
||||
|
||||
- [v0.4.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.4.0.md)
|
||||
adds opaque prepared executions, exact profile and prompt inspection, and a
|
||||
typed backend-capacity error;
|
||||
- [v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.5.0.md)
|
||||
adds application fallback profile filesystems and stops sending unset
|
||||
optional sampling controls as framework-selected provider values; and
|
||||
- the [v0.5.0 format contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
|
||||
defines the resulting profile-source and execution-setting precedence.
|
||||
|
||||
A source-compatibility test of the current Notarius repository against
|
||||
PromptKit v0.5.0 completed successfully. The work is therefore primarily an
|
||||
intentional runtime and configuration migration rather than a repair for a
|
||||
breaking Go API change.
|
||||
|
||||
## Goals
|
||||
|
||||
- Pin and document PromptKit v0.5.0 as Notarius's supported upstream contract.
|
||||
- Execute the exact prepared request snapshot whose safe details are recorded
|
||||
in Notarius debug material.
|
||||
- Validate configured PromptKit profiles through the upstream inspection API
|
||||
without synthetic prompts, provider calls, or credential-value access.
|
||||
- Give Notarius an application-owned, operator-overridable
|
||||
`dnd-extraction` profile fallback.
|
||||
- Let a pipeline choose one default LLM profile without repeating that ID on
|
||||
every LLM-backed binding.
|
||||
- Apply profile inheritance and run-wide overrides only where the resolved
|
||||
module or validator can use an LLM.
|
||||
- Preserve accurate checkpoint invalidation, effective profile provenance,
|
||||
redaction, cancellation, concurrency, and provider-neutral module contracts.
|
||||
- Provide operators with one clear deployment pattern for production,
|
||||
development, and local profile definitions.
|
||||
|
||||
## Target End State
|
||||
|
||||
### PromptKit Runtime Boundary
|
||||
|
||||
Notarius depends on PromptKit v0.5.0 and uses its public APIs rather than
|
||||
reimplementing source or execution resolution.
|
||||
|
||||
For each structured completion, the adapter:
|
||||
|
||||
1. builds one PromptKit run request from the provider-neutral Notarius request;
|
||||
2. calls `PrepareExecution` once;
|
||||
3. immediately arranges an idempotent `Discard` for every unexecuted handle;
|
||||
4. obtains credential-redacted `Details` for debug and response metadata; and
|
||||
5. calls `RunPrepared` so generation uses that exact frozen snapshot.
|
||||
|
||||
The debug prompt and successful result therefore describe the same selected
|
||||
profile, rendered messages, input bytes, session, output contract, and effective
|
||||
settings even when a filesystem-backed source changes concurrently. PromptKit
|
||||
handle types remain private to `internal/framework/llm`.
|
||||
|
||||
PromptKit admission failures continue to match Notarius's provider-neutral
|
||||
`ErrLLMCapacityExceeded` contract. When PromptKit supplies a `CapacityError`,
|
||||
the adapter obtains the normalized backend ID through `errors.As` and may add it
|
||||
to safe application-owned diagnostics without parsing upstream error wording.
|
||||
The backend ID does not become a provider-specific module contract.
|
||||
|
||||
### Optional Provider Controls
|
||||
|
||||
Notarius accepts PromptKit v0.5.0's new behavior for `temperature`,
|
||||
`max_tokens`, and `top_p`: an unset setting is omitted from compatible provider
|
||||
requests and the provider chooses its own default. Notarius does not restore
|
||||
PromptKit's former implicit `top_p: 1` value globally.
|
||||
|
||||
An operator who requires a particular value specifies it in the selected
|
||||
PromptKit profile. The application fallback described below intentionally
|
||||
leaves these controls unset. A human-reviewed D&D extraction comparison should
|
||||
be performed after the upgrade, but paid or nondeterministic model output is
|
||||
not part of the default automated test suite.
|
||||
|
||||
### Profile Inspection
|
||||
|
||||
Pipeline-aware configuration validation uses `Engine.InspectProfile` for every
|
||||
effective explicit profile ID. It verifies that the profile exists, parses and
|
||||
validates, resolves its backend and target, and is compatible with the engine's
|
||||
registered backends. It does not create a synthetic prompt, load prompt inputs,
|
||||
contact a provider, or require credential values to exist in the validation
|
||||
process environment.
|
||||
|
||||
Credential availability is execution-time state. PromptKit preparation still
|
||||
enforces the selected profile's credential contract before generation. This
|
||||
keeps `notarius config validate` useful in build and deployment validation
|
||||
environments where secrets are deliberately absent.
|
||||
|
||||
PromptKit construction for inspection and execution uses one shared internal
|
||||
profile-source and backend-option path. The CLI does not expose PromptKit public
|
||||
types across the Notarius LLM boundary merely to perform inspection.
|
||||
|
||||
`InspectPrompt` is not adopted merely because it exists. It remains available
|
||||
for a later, separately defined module-to-prompt interface preflight if a
|
||||
concrete validation requirement justifies that additional contract.
|
||||
|
||||
### Application And Operator Profile Sources
|
||||
|
||||
Notarius embeds one ordinary PromptKit YAML profile with the stable ID
|
||||
`dnd-extraction`. It is an application fallback registered through
|
||||
`WithFallbackProfileFS`, is owned by the D&D module family, and initially
|
||||
preserves the current effective D&D baseline:
|
||||
|
||||
- backend: PromptKit's built-in `openrouter` backend;
|
||||
- model: `openai/gpt-5.6-luna`;
|
||||
- reasoning effort: unset, allowing OpenAI's backend to apply its default of
|
||||
`medium`;
|
||||
- generation timeout: 240 seconds;
|
||||
- service tier: `flex`; and
|
||||
- no application-selected `temperature`, `max_tokens`, or `top_p`.
|
||||
|
||||
All maintained D&D LLM prompt definitions use `dnd-extraction` as their
|
||||
`default_profile`. The ID communicates workload intent rather than a provider,
|
||||
model, or environment. Changing the embedded fallback is an intentional
|
||||
Notarius execution-policy change and participates in checkpoint identity.
|
||||
|
||||
Effective profile definitions resolve in PromptKit's order:
|
||||
|
||||
1. programmatic in-memory profiles used by tests or explicit consumers;
|
||||
2. the operator source configured by `promptkit.profile_file` or
|
||||
`promptkit.profile_dir`;
|
||||
3. the Notarius application fallback source; and
|
||||
4. PromptKit's embedded built-in catalog.
|
||||
|
||||
Only an absent ID falls through to the next source. A matching profile is a
|
||||
complete definition: fields are not merged with a lower-precedence definition,
|
||||
and a malformed matching operator profile fails rather than silently selecting
|
||||
the application fallback.
|
||||
|
||||
Production, development, and local deployments should normally provide
|
||||
different complete definitions for the same `dnd-extraction` ID. An operator
|
||||
source is optional because the application fallback keeps the maintained D&D
|
||||
workflow usable, but a deployment that needs an intentional model or backend
|
||||
policy should configure its own definition.
|
||||
|
||||
### Domain Ownership And Asset Assembly
|
||||
|
||||
The D&D fallback profile remains under `internal/modules/dnd` and is registered
|
||||
by the D&D registrar, consistent with ADR-0004. Generic LLM plumbing knows how
|
||||
to collect and flatten application fallback profile filesystems but contains no
|
||||
D&D model or policy knowledge.
|
||||
|
||||
The shared asset registry detects invalid roots, unreadable sources, and
|
||||
duplicate flattened paths. PromptKit remains responsible for strict profile
|
||||
YAML parsing, duplicate profile-ID detection, source precedence, and effective
|
||||
target resolution. The same assembled fallback source is supplied to runtime
|
||||
execution and CLI profile inspection.
|
||||
|
||||
### Explicit Module Execution Metadata
|
||||
|
||||
Every registered input, chunk, extract, merge, normalize, and output module
|
||||
declares one required execution class: `deterministic` or `llm_backed`.
|
||||
Validator registrations continue to declare the same distinction through their
|
||||
validator specifications.
|
||||
|
||||
The registered specification is authoritative for configuration resolution.
|
||||
Current production classifications are:
|
||||
|
||||
- the D&D scene chunker, all D&D extractors, and the D&D NPC normalizer are
|
||||
LLM-backed;
|
||||
- the Seriatim input adapter, generic chunker, all current mergers, all other
|
||||
current normalizers, and the JSON output encoder are deterministic; and
|
||||
- current validators retain their declared classifications.
|
||||
|
||||
Missing or unsupported execution metadata is a registration error. Explicitly
|
||||
assigning `llm_profile` to a deterministic module or validator is a pipeline
|
||||
resolution error. The framework does not infer execution class by inspecting
|
||||
domain package names or concrete implementation types at runtime.
|
||||
|
||||
The module specification replaces the chunk runner's special runtime
|
||||
execution-class probe. Effective resolved bindings already express the result:
|
||||
only LLM-backed bindings may retain a non-empty profile.
|
||||
|
||||
### Pipeline-Level Profile Default
|
||||
|
||||
Configuration version 4 gains one optional non-empty pipeline field:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
dnd-session:
|
||||
llm_profile: dnd-extraction
|
||||
```
|
||||
|
||||
No configuration-version increment is required because the field is additive
|
||||
and existing files remain valid. An explicitly present blank value is invalid.
|
||||
|
||||
For every selected LLM-backed module and validator, the effective profile uses
|
||||
this precedence:
|
||||
|
||||
1. non-empty run-wide `--llm-profile` override;
|
||||
2. binding-specific `llm_profile`;
|
||||
3. pipeline-level `llm_profile`; and
|
||||
4. the prompt definition's `default_profile`, represented by an empty effective
|
||||
Notarius binding profile.
|
||||
|
||||
The run-wide override and inherited pipeline default never attach to a
|
||||
deterministic binding. Binding-specific exceptions remain available when one
|
||||
operation needs a different cost, latency, quality, backend, or reasoning
|
||||
policy.
|
||||
|
||||
Inheritance is resolved after module and validator selection, including
|
||||
`--only` lane filtering, but before effective-pipeline validation, digest
|
||||
construction, explicit-profile inspection, checkpoint construction,
|
||||
preparation, execution, or provenance capture. Only profiles used by selected
|
||||
LLM-backed bindings are inspected. An unused pipeline default in a pipeline
|
||||
with no selected LLM-backed work does not require an otherwise unused profile
|
||||
to exist.
|
||||
|
||||
The resolved pipeline contains effective binding profiles rather than a second
|
||||
runtime inheritance mechanism. Two pipelines that differ only by spelling the
|
||||
same effective policy once as a pipeline default and once on every LLM-backed
|
||||
binding have the same semantic resolved digest. Changing an effective profile
|
||||
changes the digest and applicable checkpoint identity.
|
||||
|
||||
### Provenance And Checkpoints
|
||||
|
||||
The PromptKit profile-source checkpoint fingerprint covers:
|
||||
|
||||
- the PromptKit v0.5.0 built-in profile catalog identity;
|
||||
- exact application fallback profile asset content; and
|
||||
- exact configured operator profile YAML content, when present.
|
||||
|
||||
The existing local-backend target fingerprint remains separate and continues
|
||||
to exclude scheduling-only concurrency limits. Fingerprints contain hashes and
|
||||
stable markers, not profile contents, filesystem paths, endpoints, credentials,
|
||||
or other secrets.
|
||||
|
||||
Changing the PromptKit version, application fallback, operator profile, or
|
||||
effective pipeline profile makes incompatible LLM checkpoints ineligible for
|
||||
reuse. The dependency upgrade is expected to invalidate checkpoints produced
|
||||
under v0.3.0.
|
||||
|
||||
Successful run manifests continue to record only profiles actually selected by
|
||||
PromptKit, including their effective model, backend, and reasoning metadata.
|
||||
Debug output reports the same effective execution snapshot used for generation.
|
||||
|
||||
### Operator Documentation And Examples
|
||||
|
||||
Canonical documentation clearly distinguishes:
|
||||
|
||||
- Notarius prompt and schema assets embedded in the application;
|
||||
- Notarius application fallback profiles embedded in the application;
|
||||
- PromptKit's own embedded built-in profiles; and
|
||||
- operator profile files on the deployment filesystem.
|
||||
|
||||
The configuration reference owns the pipeline field, profile-source fields,
|
||||
validation rules, and precedence. Operations owns deployment layout, working
|
||||
directory behavior, credentials, and environment-specific profile management.
|
||||
The PromptKit integration document owns the pinned upstream contract and
|
||||
source-precedence boundary. Internal documents describe asset registration,
|
||||
resolution, inspection, prepared execution, fingerprinting, and tests without
|
||||
duplicating user-facing field definitions.
|
||||
|
||||
The maintained examples continue to include only the minimal and complete D&D
|
||||
configurations. They use the stable `dnd-extraction` policy, and one maintained
|
||||
PromptKit profile file under `examples/` demonstrates an operator override.
|
||||
Examples remain secret-free and are validated without live provider calls.
|
||||
|
||||
## Out Of Scope
|
||||
|
||||
- Implementing the separate deterministic prompt-session identity roadmap
|
||||
item.
|
||||
- Changing the default `concurrency.total_llm` value; PromptKit's retained
|
||||
OpenRouter capacity of 16 remains relevant to that separate item.
|
||||
- Adding model evaluation as a deterministic or CI correctness gate.
|
||||
- Automatically selecting production, development, or local environments.
|
||||
Deployment configuration chooses the operator profile source.
|
||||
- Profile inheritance, partial profile merging, or cross-profile aliases.
|
||||
- Exposing PromptKit types to modules, validators, durable output contracts, or
|
||||
public configuration structures.
|
||||
- Adopting `InspectPrompt` without a separately justified prompt-interface
|
||||
validation contract.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Notarius builds and its offline test suite passes with PromptKit v0.5.0.
|
||||
- Every structured completion executes the exact snapshot used for safe debug
|
||||
prompt details.
|
||||
- Profile preflight uses profile inspection and no synthetic prompt.
|
||||
- The embedded `dnd-extraction` fallback resolves without an operator source,
|
||||
and a matching valid operator profile replaces it completely.
|
||||
- Every production module has explicit, correct execution metadata.
|
||||
- Pipeline, binding, CLI, and prompt-default precedence behaves as defined for
|
||||
modules and validators, while deterministic bindings remain profile-free.
|
||||
- Effective profiles participate in pipeline digests, profile inspection,
|
||||
checkpoint identity, debug records, and run provenance at the appropriate
|
||||
boundaries.
|
||||
- The dependency and application fallback changes invalidate incompatible old
|
||||
checkpoints without exposing profile or credential content.
|
||||
- Canonical documentation and maintained examples accurately describe and
|
||||
exercise the implemented operator workflow.
|
||||
- Default tests remain deterministic, offline, credential-free, and focused on
|
||||
Notarius-owned behavior rather than duplicating PromptKit's upstream suite.
|
||||
@@ -1,4 +1,6 @@
|
||||
version: 4
|
||||
promptkit:
|
||||
profile_file: ./examples/profiles/dnd-extraction.yml
|
||||
concurrency:
|
||||
total_llm: 2
|
||||
stage_workers:
|
||||
@@ -16,6 +18,7 @@ debug:
|
||||
directory: ./notarius-debug
|
||||
pipelines:
|
||||
dnd-session:
|
||||
llm_profile: dnd-extraction
|
||||
input: seriatim
|
||||
# Stable campaign context is shared by every module that accepts these slots.
|
||||
references:
|
||||
@@ -54,7 +57,6 @@ pipelines:
|
||||
merge: appendorder
|
||||
normalize:
|
||||
module: dnd/npcs
|
||||
llm_profile: gemini-2-flash
|
||||
retries: 2
|
||||
scene-descriptions:
|
||||
extract:
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
version: 4
|
||||
pipelines:
|
||||
dnd-session:
|
||||
llm_profile: dnd-extraction
|
||||
input: seriatim
|
||||
artifacts:
|
||||
spells:
|
||||
|
||||
5
examples/profiles/dnd-extraction.yml
Normal file
5
examples/profiles/dnd-extraction.yml
Normal file
@@ -0,0 +1,5 @@
|
||||
id: dnd-extraction
|
||||
backend: openrouter
|
||||
model: openai/gpt-5.6-luna
|
||||
timeout_seconds: 240
|
||||
service_tier: flex
|
||||
2
go.mod
2
go.mod
@@ -3,7 +3,7 @@ module gitea.maximumdirect.net/eric/notarius
|
||||
go 1.25.5
|
||||
|
||||
require (
|
||||
gitea.maximumdirect.net/eric/promptkit v0.3.0
|
||||
gitea.maximumdirect.net/eric/promptkit v0.5.0
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
|
||||
4
go.sum
4
go.sum
@@ -1,5 +1,5 @@
|
||||
gitea.maximumdirect.net/eric/promptkit v0.3.0 h1:P5/GJ6fVIVsBLJx8NxoaGV34ZWeAkU8lm4Ht6aVMOFg=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.3.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.5.0 h1:jnpazLyyNhWrB2xzwwtUkNUfktkTdkENTwuSPnKiYrc=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.5.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
||||
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
||||
|
||||
@@ -213,11 +213,12 @@ func assembledSpellPipeline(t *testing.T, options assembledSpellPipelineOptions)
|
||||
components := productionTestComponents(t)
|
||||
extractor := &assembledSpellExtractor{unknownSpell: options.unknownSpell}
|
||||
if err := pipeline.RegisterExtractor[dnd.SpellList](components.registries.Extractors, pipeline.ModuleSpec{
|
||||
Key: assembledSpellExtractorKey,
|
||||
Stage: pipeline.StageExtract,
|
||||
Requires: []string{"chunks", "source.transcript"},
|
||||
Provides: []string{"dnd.spell_casts"},
|
||||
ArtifactKind: dnd.SpellListKind,
|
||||
Key: assembledSpellExtractorKey,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"chunks", "source.transcript"},
|
||||
Provides: []string{"dnd.spell_casts"},
|
||||
ArtifactKind: dnd.SpellListKind,
|
||||
}, func() (contracts.Extractor[dnd.SpellList], error) {
|
||||
return extractor, nil
|
||||
}); err != nil {
|
||||
|
||||
@@ -143,17 +143,6 @@ func isEmptyRegistries(registries pipeline.Registries) bool {
|
||||
registries.Outputs == nil
|
||||
}
|
||||
|
||||
func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
assets, err := productionPromptAssets()
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
return buildProductionLLMClient(ctx, cfg, profileID, overrides, assets)
|
||||
}
|
||||
|
||||
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
|
||||
return func(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
return buildProductionLLMClient(ctx, cfg, profileID, overrides, assets)
|
||||
|
||||
@@ -154,6 +154,23 @@ func TestConfigValidateResolvesPipelineAndChecksSelection(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigValidatePipelineDefaultProfileIsOffline(t *testing.T) {
|
||||
configPath := writeCommandConfigContent(t, `version: 4
|
||||
pipelines:
|
||||
demo:
|
||||
llm_profile: dnd-extraction
|
||||
input: seriatim
|
||||
artifacts:
|
||||
spells:
|
||||
extract: dnd/spells
|
||||
`)
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "demo"}, &stdout, &stderr, Options{})
|
||||
if code != 0 || !strings.Contains(stdout.String(), "valid for pipeline \"demo\"") || stderr.Len() != 0 {
|
||||
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestPipelinesListSortsNormalizedIDsInTextAndJSON(t *testing.T) {
|
||||
configPath := writeCommandConfig(t, " zeta ", "alpha")
|
||||
options := commandContractOptions(t)
|
||||
|
||||
@@ -94,6 +94,37 @@ func TestMaintainedConfigurationExampleSet(t *testing.T) {
|
||||
if got := strings.Join(names, ","); got != "dnd-complete.config.yml,dnd-minimal.config.yml" {
|
||||
t.Fatalf("maintained configuration examples = %q, want only the minimal and complete D&D examples", got)
|
||||
}
|
||||
|
||||
profileEntries, err := os.ReadDir(repositoryPath("examples", "profiles"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
names = names[:0]
|
||||
for _, entry := range profileEntries {
|
||||
if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".yml") {
|
||||
names = append(names, entry.Name())
|
||||
}
|
||||
}
|
||||
sort.Strings(names)
|
||||
if got := strings.Join(names, ","); got != "dnd-extraction.yml" {
|
||||
t.Fatalf("maintained operator profiles = %q, want dnd-extraction.yml", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMaintainedExamplesValidateEffectiveProfilesOffline(t *testing.T) {
|
||||
t.Chdir(repositoryPath())
|
||||
t.Setenv("OPENROUTER_API_KEY", "")
|
||||
for _, example := range maintainedExampleFiles(t) {
|
||||
t.Run(example.name, func(t *testing.T) {
|
||||
var stdout, stderr strings.Builder
|
||||
code := RunWithOptions([]string{
|
||||
"config", "validate", "--config", example.path, "--pipeline", "dnd-session",
|
||||
}, &stdout, &stderr, Options{})
|
||||
if code != 0 || stderr.Len() != 0 || !strings.Contains(stdout.String(), `valid for pipeline "dnd-session"`) {
|
||||
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func exampleStepLaneIDs(resolved pipeline.ResolvedPipeline) []string {
|
||||
|
||||
@@ -166,6 +166,35 @@ func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
|
||||
assertProductionContains(t, "production prompt assets", assetNames, requiredAssets)
|
||||
|
||||
catalog := catalogFromRegistries(registries)
|
||||
for _, test := range []struct {
|
||||
stage pipeline.ModuleStage
|
||||
key string
|
||||
want contracts.ExecutionClass
|
||||
}{
|
||||
{stage: pipeline.StageInput, key: "seriatim", want: contracts.ExecutionClassDeterministic},
|
||||
{stage: pipeline.StageChunk, key: "generic", want: contracts.ExecutionClassDeterministic},
|
||||
{stage: pipeline.StageChunk, key: "dnd/scenes", want: contracts.ExecutionClassLLMBacked},
|
||||
{stage: pipeline.StageExtract, key: "dnd/spells", want: contracts.ExecutionClassLLMBacked},
|
||||
{stage: pipeline.StageExtract, key: "dnd/npcs", want: contracts.ExecutionClassLLMBacked},
|
||||
{stage: pipeline.StageExtract, key: "dnd/combat-turns", want: contracts.ExecutionClassLLMBacked},
|
||||
{stage: pipeline.StageExtract, key: "dnd/item-events", want: contracts.ExecutionClassLLMBacked},
|
||||
{stage: pipeline.StageExtract, key: "dnd/npc-interactions", want: contracts.ExecutionClassLLMBacked},
|
||||
{stage: pipeline.StageExtract, key: "dnd/scene-descriptions", want: contracts.ExecutionClassLLMBacked},
|
||||
{stage: pipeline.StageMerge, key: "appendorder", want: contracts.ExecutionClassDeterministic},
|
||||
{stage: pipeline.StageNormalize, key: "noop", want: contracts.ExecutionClassDeterministic},
|
||||
{stage: pipeline.StageNormalize, key: "dnd/spells", want: contracts.ExecutionClassDeterministic},
|
||||
{stage: pipeline.StageNormalize, key: "dnd/npcs", want: contracts.ExecutionClassLLMBacked},
|
||||
{stage: pipeline.StageNormalize, key: "dnd/combat-turns", want: contracts.ExecutionClassDeterministic},
|
||||
{stage: pipeline.StageNormalize, key: "dnd/item-events", want: contracts.ExecutionClassDeterministic},
|
||||
{stage: pipeline.StageNormalize, key: "dnd/npc-interactions", want: contracts.ExecutionClassDeterministic},
|
||||
{stage: pipeline.StageNormalize, key: "dnd/scene-descriptions", want: contracts.ExecutionClassDeterministic},
|
||||
{stage: pipeline.StageOutput, key: "json", want: contracts.ExecutionClassDeterministic},
|
||||
} {
|
||||
got, ok := catalog.ExecutionClass(test.stage, test.key)
|
||||
if !ok || got != test.want {
|
||||
t.Fatalf("production execution class for %s/%s = %q, %t; want %q, true", test.stage, test.key, got, ok, test.want)
|
||||
}
|
||||
}
|
||||
converted := registriesFromCatalog(catalog)
|
||||
if converted.ArtifactCodecs != registries.ArtifactCodecs || converted.ArtifactEvidence != registries.ArtifactEvidence || converted.ValidatorChains != registries.ValidatorChains {
|
||||
t.Fatal("catalog/registry conversion did not preserve artifact and validator registries")
|
||||
@@ -201,6 +230,76 @@ func TestDefaultCLICompositionValidatesRepresentativeConfiguration(t *testing.T)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProductionAssetsResolveDNDExtractionProfile(t *testing.T) {
|
||||
components := productionTestComponents(t)
|
||||
newEngine := func(profileFile string) (*promptkit.Engine, error) {
|
||||
t.Helper()
|
||||
options, err := components.assets.PromptKitOptions()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if profileFile != "" {
|
||||
options = append(options, promptkit.WithProfileFile(profileFile))
|
||||
}
|
||||
return promptkit.NewEngine(promptkit.Config{}, options...)
|
||||
}
|
||||
|
||||
t.Run("fallback", func(t *testing.T) {
|
||||
engine, err := newEngine("")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inspection, err := engine.InspectProfile(context.Background(), "dnd-extraction")
|
||||
if err != nil {
|
||||
t.Fatalf("InspectProfile() error = %v, want fallback profile", err)
|
||||
}
|
||||
params := inspection.EffectiveModelParams
|
||||
if params.BackendID != "openrouter" || params.Model != "openai/gpt-5.6-luna" || params.TimeoutSeconds != 240 || params.ServiceTier != "flex" {
|
||||
t.Fatalf("fallback profile parameters = %#v", params)
|
||||
}
|
||||
if params.ReasoningEffort != "" || params.Temperature != 0 || params.MaxTokens != 0 || params.TopP != 0 {
|
||||
t.Fatalf("fallback profile selected optional provider controls: %#v", params)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("valid operator profile wins", func(t *testing.T) {
|
||||
profilePath := filepath.Join(t.TempDir(), "profiles.yaml")
|
||||
if err := os.WriteFile(profilePath, []byte(`id: dnd-extraction
|
||||
endpoint: http://operator.example.test/v1
|
||||
model: operator-model
|
||||
timeout_seconds: 75
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
engine, err := newEngine(profilePath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inspection, err := engine.InspectProfile(context.Background(), "dnd-extraction")
|
||||
if err != nil {
|
||||
t.Fatalf("InspectProfile() error = %v, want operator profile", err)
|
||||
}
|
||||
params := inspection.EffectiveModelParams
|
||||
if params.BackendID != "" || params.Endpoint != "http://operator.example.test/v1" || params.Model != "operator-model" || params.TimeoutSeconds != 75 || params.ServiceTier != "" {
|
||||
t.Fatalf("operator profile parameters = %#v, want complete replacement", params)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("invalid operator profile does not fall through", func(t *testing.T) {
|
||||
profilePath := filepath.Join(t.TempDir(), "profiles.yaml")
|
||||
if err := os.WriteFile(profilePath, []byte("id: dnd-extraction\nendpoint: http://operator.example.test/v1\nmodel: operator-model\nunknown: value\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
engine, err := newEngine(profilePath)
|
||||
if err == nil {
|
||||
_, err = engine.InspectProfile(context.Background(), "dnd-extraction")
|
||||
}
|
||||
if err == nil {
|
||||
t.Fatal("operator profile error = nil, want failure instead of fallback")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) {
|
||||
components := productionTestComponents(t)
|
||||
cfg := config.Default()
|
||||
@@ -375,39 +474,67 @@ func setNormalizeSpellCatalogSource(t *testing.T, resolved *pipeline.ResolvedPip
|
||||
resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings = bindings
|
||||
}
|
||||
|
||||
func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) {
|
||||
func TestProductionLLMClientFactoryBuildsOfflineRuntime(t *testing.T) {
|
||||
components := productionTestComponents(t)
|
||||
factories := []struct {
|
||||
name string
|
||||
factory LLMClientFactory
|
||||
}{
|
||||
{name: "default production assets", factory: productionLLMClientFactory},
|
||||
{name: "provided production assets", factory: productionLLMClientFactoryWithAssets(components.assets)},
|
||||
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
|
||||
if err != nil {
|
||||
t.Fatalf("build production LLM runtime: %v", err)
|
||||
}
|
||||
for _, tt := range factories {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
client, manifests, err := tt.factory(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
|
||||
if err != nil {
|
||||
t.Fatalf("build production LLM runtime: %v", err)
|
||||
}
|
||||
if client == nil {
|
||||
t.Fatal("production LLM runtime returned a nil client")
|
||||
}
|
||||
if len(manifests) != 0 {
|
||||
t.Fatalf("eager profile manifests = %#v, want none", manifests)
|
||||
}
|
||||
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
|
||||
if !ok {
|
||||
t.Fatalf("production LLM client %T does not provide one profile-source checkpoint fingerprint", client)
|
||||
}
|
||||
fingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
|
||||
if err != nil || len(fingerprints) != 1 {
|
||||
t.Fatalf("production LLM checkpoint fingerprints = %#v, error = %v, want one profile-source identity", fingerprints, err)
|
||||
}
|
||||
if _, ok := client.(contracts.LLMProfileManifestProvider); !ok {
|
||||
t.Fatalf("production LLM client %T does not provide profile manifests", client)
|
||||
}
|
||||
})
|
||||
if client == nil {
|
||||
t.Fatal("production LLM runtime returned a nil client")
|
||||
}
|
||||
if len(manifests) != 0 {
|
||||
t.Fatalf("eager profile manifests = %#v, want none", manifests)
|
||||
}
|
||||
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
|
||||
if !ok {
|
||||
t.Fatalf("production LLM client %T does not provide one profile-source checkpoint fingerprint", client)
|
||||
}
|
||||
fingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
|
||||
if err != nil || len(fingerprints) != 1 {
|
||||
t.Fatalf("production LLM checkpoint fingerprints = %#v, error = %v, want one profile-source identity", fingerprints, err)
|
||||
}
|
||||
if _, ok := client.(contracts.LLMProfileManifestProvider); !ok {
|
||||
t.Fatalf("production LLM client %T does not provide profile manifests", client)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeOptionsSharesProductionProfileAssetsWithDefaultRuntime(t *testing.T) {
|
||||
opts, err := normalizeOptions(Options{
|
||||
Catalog: pipeline.ModuleCatalog{Inputs: pipeline.NewInputAdapterRegistry()},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if opts.promptKitAssets == nil || opts.LLMClientFactory == nil {
|
||||
t.Fatalf("normalized options = %#v, want shared profile assets and default runtime factory", opts)
|
||||
}
|
||||
if err := validateExplicitPromptKitProfiles(context.Background(), config.Default(), []string{"dnd-extraction"}, opts.promptKitAssets); err != nil {
|
||||
t.Fatalf("inspect application fallback profile: %v", err)
|
||||
}
|
||||
|
||||
client, _, err := opts.LLMClientFactory(context.Background(), config.Default(), "dnd-extraction", LLMRuntimeOverrides{})
|
||||
if err != nil {
|
||||
t.Fatalf("build default runtime: %v", err)
|
||||
}
|
||||
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
|
||||
if !ok {
|
||||
t.Fatalf("default runtime client %T does not provide checkpoint fingerprints", client)
|
||||
}
|
||||
runtimeFingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
directClient, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{Assets: opts.promptKitAssets})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inspectionFingerprints, err := directClient.LLMCheckpointFingerprints()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !reflect.DeepEqual(runtimeFingerprints, inspectionFingerprints) {
|
||||
t.Fatalf("runtime profile fingerprints = %#v, inspection profile fingerprints = %#v", runtimeFingerprints, inspectionFingerprints)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -498,7 +625,8 @@ func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
|
||||
t.Run("canceled context", func(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
client, manifests, err := productionLLMClientFactory(ctx, config.Default(), "test-profile", LLMRuntimeOverrides{})
|
||||
components := productionTestComponents(t)
|
||||
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(ctx, config.Default(), "test-profile", LLMRuntimeOverrides{})
|
||||
if !errors.Is(err, context.Canceled) || client != nil || len(manifests) != 0 {
|
||||
t.Fatalf("client=%T manifests=%#v error=%v, want canceled construction", client, manifests, err)
|
||||
}
|
||||
@@ -745,9 +873,10 @@ func productionCLIOptions(t *testing.T) Options {
|
||||
|
||||
func productionOptionsFromComponents(components productionComponents) Options {
|
||||
return Options{
|
||||
Catalog: catalogFromRegistries(components.registries),
|
||||
Registries: components.registries,
|
||||
LookupEnv: emptyLookup,
|
||||
Catalog: catalogFromRegistries(components.registries),
|
||||
Registries: components.registries,
|
||||
LookupEnv: emptyLookup,
|
||||
promptKitAssets: components.assets,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -2,73 +2,37 @@ package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"testing/fstest"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
)
|
||||
|
||||
const profileCheckPromptID = "notarius.profile.check"
|
||||
|
||||
var profileCheckPromptFS = fstest.MapFS{
|
||||
"prompts/profile-check.yaml": &fstest.MapFile{Data: []byte(`id: notarius.profile.check
|
||||
version: "1.0.0"
|
||||
default_profile: mistral-small-3
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
messages:
|
||||
- role: user
|
||||
content: "{{input \"transcript\"}}"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
repair_attempts: 0
|
||||
`)},
|
||||
}
|
||||
|
||||
func validateExplicitPromptKitProfiles(ctx context.Context, cfg config.Config, profileIDs []string) error {
|
||||
func validateExplicitPromptKitProfiles(ctx context.Context, cfg config.Config, profileIDs []string, assets *llm.AssetRegistry) error {
|
||||
if len(profileIDs) == 0 {
|
||||
return nil
|
||||
}
|
||||
engine, err := newProfileValidationEngine(cfg)
|
||||
inspector, err := llm.NewPromptKitProfileInspector(llm.PromptKitProfileInspectorConfig{
|
||||
Source: promptKitProfileSourceConfig(cfg),
|
||||
Assets: assets,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("load PromptKit profiles: %w", err)
|
||||
}
|
||||
for _, profileID := range profileIDs {
|
||||
if _, err := engine.Prepare(ctx, promptkit.RunRequest{
|
||||
PromptID: profileCheckPromptID,
|
||||
ProfileID: profileID,
|
||||
Inputs: map[string]promptkit.ArtifactRef{
|
||||
"transcript": promptkit.Inline("profile check"),
|
||||
},
|
||||
}); err != nil {
|
||||
if errors.Is(err, promptkit.ErrProfileNotFound) {
|
||||
return fmt.Errorf("PromptKit profile %q is not configured", profileID)
|
||||
}
|
||||
return fmt.Errorf("validate PromptKit profile %q: %w", profileID, err)
|
||||
if _, err := inspector.InspectProfile(ctx, profileID); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func newProfileValidationEngine(cfg config.Config) (*promptkit.Engine, error) {
|
||||
opts := []promptkit.Option{
|
||||
promptkit.WithPromptFS(profileCheckPromptFS, "prompts"),
|
||||
func promptKitProfileSourceConfig(cfg config.Config) llm.PromptKitProfileSourceConfig {
|
||||
return llm.PromptKitProfileSourceConfig{
|
||||
ProfileDir: cfg.PromptKit.ProfileDir,
|
||||
ProfileFile: cfg.PromptKit.ProfileFile,
|
||||
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
|
||||
}
|
||||
if cfg.PromptKit.ProfileFile != "" {
|
||||
opts = append(opts, promptkit.WithProfileFile(cfg.PromptKit.ProfileFile))
|
||||
}
|
||||
if localBackend := mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend); localBackend != nil {
|
||||
opts = append(opts, llm.PromptKitLocalBackendOption(*localBackend))
|
||||
}
|
||||
return promptkit.NewEngine(promptkit.Config{
|
||||
PromptDir: "unused",
|
||||
ProfileDir: cfg.PromptKit.ProfileDir,
|
||||
}, opts...)
|
||||
}
|
||||
|
||||
func mapPromptKitLocalBackend(cfg *config.PromptKitLocalBackendConfig) *llm.PromptKitLocalBackendConfig {
|
||||
|
||||
@@ -2,6 +2,7 @@ package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
@@ -9,46 +10,150 @@ import (
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
)
|
||||
|
||||
func TestExplicitPromptKitProfileValidationUsesConfiguredLocalBackendWithoutGeneration(t *testing.T) {
|
||||
func TestExplicitPromptKitProfileValidationInspectsProfilesWithoutGeneration(t *testing.T) {
|
||||
var providerCalls atomic.Int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
|
||||
providerCalls.Add(1)
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
|
||||
if err := os.WriteFile(profilePath, []byte(`id: local-profile
|
||||
backend: local
|
||||
model: local-model
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfg := config.Default()
|
||||
cfg.PromptKit.ProfileFile = profilePath
|
||||
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
|
||||
Endpoint: server.URL + "/v1",
|
||||
ConcurrencyLimit: 2,
|
||||
}
|
||||
if err := validateExplicitPromptKitProfiles(context.Background(), cfg, []string{"local-profile"}); err != nil {
|
||||
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
|
||||
}
|
||||
if providerCalls.Load() != 0 {
|
||||
t.Fatalf("provider calls during configured profile validation = %d, want 0", providerCalls.Load())
|
||||
writeProfile := func(t *testing.T, name, content string) string {
|
||||
t.Helper()
|
||||
profilePath := filepath.Join(t.TempDir(), name+".yaml")
|
||||
if err := os.WriteFile(profilePath, []byte(content), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return profilePath
|
||||
}
|
||||
localProfile := "id: local-profile\nbackend: local\nmodel: local-model\n"
|
||||
credentialProfile := `id: credential-profile
|
||||
endpoint: ` + server.URL + `/v1
|
||||
model: credential-model
|
||||
api_key_env: NOTARIUS_PROMPTKIT_PROFILE_INSPECTION_TEST_KEY
|
||||
`
|
||||
t.Setenv("NOTARIUS_PROMPTKIT_PROFILE_INSPECTION_TEST_KEY", "")
|
||||
|
||||
cfg.PromptKit.LocalBackend = nil
|
||||
err := validateExplicitPromptKitProfiles(context.Background(), cfg, []string{"local-profile"})
|
||||
if err == nil ||
|
||||
!strings.Contains(err.Error(), `validate PromptKit profile "local-profile"`) ||
|
||||
!strings.Contains(err.Error(), promptkit.BackendLocal) {
|
||||
t.Fatalf("validation without registration error = %v, want profile and local backend context", err)
|
||||
tests := []struct {
|
||||
name string
|
||||
profilePath string
|
||||
profileID string
|
||||
profileDir bool
|
||||
localBackend bool
|
||||
canceled bool
|
||||
wantErr []string
|
||||
rejectErr []string
|
||||
}{
|
||||
{
|
||||
name: "configured local backend",
|
||||
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||
profileID: "local-profile",
|
||||
profileDir: true,
|
||||
localBackend: true,
|
||||
},
|
||||
{
|
||||
name: "missing local backend registration",
|
||||
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||
profileID: "local-profile",
|
||||
wantErr: []string{`PromptKit profile "local-profile" is invalid or unreadable`},
|
||||
},
|
||||
{
|
||||
name: "absent profile",
|
||||
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||
profileID: "absent-profile",
|
||||
localBackend: true,
|
||||
wantErr: []string{`PromptKit profile "absent-profile" is not configured`},
|
||||
},
|
||||
{
|
||||
name: "malformed profile",
|
||||
profilePath: writeProfile(t, "malformed-profile", "id: malformed-profile\nbackend: [\n"),
|
||||
profileID: "malformed-profile",
|
||||
wantErr: []string{`PromptKit profile "malformed-profile" is invalid or unreadable`},
|
||||
rejectErr: []string{"malformed-profile.yaml", "backend: ["},
|
||||
},
|
||||
{
|
||||
name: "invalid profile source",
|
||||
profilePath: filepath.Join(t.TempDir(), "missing-profile.yaml"),
|
||||
profileID: "missing-profile",
|
||||
wantErr: []string{"load PromptKit profiles", "profile configuration is invalid or unreadable"},
|
||||
},
|
||||
{
|
||||
name: "credential environment intentionally unset",
|
||||
profilePath: writeProfile(t, "credential-profile", credentialProfile),
|
||||
profileID: "credential-profile",
|
||||
},
|
||||
{
|
||||
name: "canceled inspection",
|
||||
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||
profileID: "local-profile",
|
||||
localBackend: true,
|
||||
canceled: true,
|
||||
wantErr: []string{"context canceled"},
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
cfg := config.Default()
|
||||
if tt.profileDir {
|
||||
cfg.PromptKit.ProfileDir = filepath.Dir(tt.profilePath)
|
||||
} else {
|
||||
cfg.PromptKit.ProfileFile = tt.profilePath
|
||||
}
|
||||
if tt.localBackend {
|
||||
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
|
||||
Endpoint: server.URL + "/v1",
|
||||
ConcurrencyLimit: 2,
|
||||
}
|
||||
}
|
||||
ctx := context.Background()
|
||||
if tt.canceled {
|
||||
var cancel context.CancelFunc
|
||||
ctx, cancel = context.WithCancel(ctx)
|
||||
cancel()
|
||||
}
|
||||
err := validateExplicitPromptKitProfiles(ctx, cfg, []string{tt.profileID}, nil)
|
||||
if len(tt.wantErr) == 0 {
|
||||
if err != nil {
|
||||
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err == nil {
|
||||
t.Fatal("validateExplicitPromptKitProfiles() error = nil, want failure")
|
||||
}
|
||||
if tt.canceled && !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("canceled inspection error = %v, want context canceled", err)
|
||||
}
|
||||
for _, want := range tt.wantErr {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Fatalf("validation error = %q, want %q", err, want)
|
||||
}
|
||||
}
|
||||
for _, rejected := range append(tt.rejectErr, tt.profilePath) {
|
||||
if rejected != "" && strings.Contains(err.Error(), rejected) {
|
||||
t.Fatalf("validation error = %q, must not expose %q", err, rejected)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
if providerCalls.Load() != 0 {
|
||||
t.Fatalf("provider calls after missing-registration validation = %d, want 0", providerCalls.Load())
|
||||
t.Fatalf("provider calls during profile inspection = %d, want 0", providerCalls.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExplicitPromptKitProfileValidationUsesFallbackAssets(t *testing.T) {
|
||||
assets := llm.NewAssetRegistry()
|
||||
if err := assets.RegisterFallbackProfileFS(fstest.MapFS{
|
||||
"profiles/fallback.yaml": {Data: []byte("id: fallback-profile\nendpoint: http://promptkit.test/v1\nmodel: fallback-model\n")},
|
||||
}, "profiles"); err != nil {
|
||||
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||
}
|
||||
if err := validateExplicitPromptKitProfiles(context.Background(), config.Default(), []string{"fallback-profile"}, assets); err != nil {
|
||||
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -116,7 +116,7 @@ func (h *recomputeTestHarness) options() Options {
|
||||
for _, key := range []string{"test/extract/producer", "test/extract/unrelated", "test/extract/middle", "test/extract/dependent"} {
|
||||
moduleKey := key
|
||||
spec := pipeline.ModuleSpec{
|
||||
Key: moduleKey, Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind,
|
||||
Key: moduleKey, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind,
|
||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}},
|
||||
}
|
||||
if err := pipeline.RegisterExtractor(opts.Registries.Extractors, spec, func() (contracts.Extractor[stateTestArtifact], error) {
|
||||
@@ -125,7 +125,7 @@ func (h *recomputeTestHarness) options() Options {
|
||||
panic(err)
|
||||
}
|
||||
}
|
||||
if err := opts.Registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/recompute-output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
||||
if err := opts.Registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/recompute-output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
||||
return recomputeTestOutput{}, nil
|
||||
}); err != nil {
|
||||
panic(err)
|
||||
|
||||
@@ -364,21 +364,21 @@ func referenceContractCatalog(t *testing.T, includeBetaMerger, includeBetaNormal
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
register(registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "reference/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }))
|
||||
register(registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/chunk", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-slot"}, {Name: "required-chunk", Required: true}}}, func() (contracts.Chunker, error) { return stateTestChunker{}, nil }))
|
||||
register(registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "reference/input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }))
|
||||
register(registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-slot"}, {Name: "required-chunk", Required: true}}}, func() (contracts.Chunker, error) { return stateTestChunker{}, nil }))
|
||||
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecA{}))
|
||||
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecB{}))
|
||||
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-alpha", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
||||
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-beta", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
||||
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
||||
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-alpha", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
||||
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-beta", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
||||
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
||||
if includeBetaMerger {
|
||||
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
||||
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
||||
}
|
||||
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
||||
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
||||
if includeBetaNormalizer {
|
||||
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
||||
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
||||
}
|
||||
register(registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { return stateTestOutput{}, nil }))
|
||||
register(registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { return stateTestOutput{}, nil }))
|
||||
return catalogFromRegistries(registries)
|
||||
}
|
||||
|
||||
|
||||
@@ -46,6 +46,7 @@ type Options struct {
|
||||
ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory
|
||||
DebugRecorderFactory func(string) (pipeline.DebugRecorder, error)
|
||||
DebugTerminalFactory func(*debugbundle.SummaryWriter) DebugTerminalWriter
|
||||
promptKitAssets *frameworkllm.AssetRegistry
|
||||
}
|
||||
|
||||
type LLMRuntimeOverrides struct {
|
||||
@@ -121,12 +122,17 @@ func normalizeOptions(opts Options) (Options, error) {
|
||||
}
|
||||
opts.Registries = components.registries
|
||||
opts.Catalog = catalogFromRegistries(components.registries)
|
||||
if opts.LLMClientFactory == nil {
|
||||
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(components.assets)
|
||||
}
|
||||
opts.promptKitAssets = components.assets
|
||||
}
|
||||
if opts.LLMClientFactory == nil {
|
||||
opts.LLMClientFactory = productionLLMClientFactory
|
||||
if opts.promptKitAssets == nil {
|
||||
assets, err := productionPromptAssets()
|
||||
if err != nil {
|
||||
return Options{}, err
|
||||
}
|
||||
opts.promptKitAssets = assets
|
||||
}
|
||||
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(opts.promptKitAssets)
|
||||
}
|
||||
return opts, nil
|
||||
}
|
||||
@@ -340,7 +346,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||
}
|
||||
profileIDs := effectiveLLMProfileIDs(effective.ResolvedPipeline)
|
||||
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, profileIDs); err != nil {
|
||||
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, profileIDs, opts.promptKitAssets); err != nil {
|
||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||
}
|
||||
workingDir, err := os.Getwd()
|
||||
@@ -952,11 +958,22 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
|
||||
seen[id] = struct{}{}
|
||||
}
|
||||
}
|
||||
add(resolved.Chunk)
|
||||
if resolved.InputExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||
add(resolved.Input)
|
||||
}
|
||||
if resolved.ChunkExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||
add(resolved.Chunk)
|
||||
}
|
||||
for _, lane := range resolved.AllArtifactLanes() {
|
||||
add(lane.Extract)
|
||||
add(lane.Merge)
|
||||
add(lane.Normalize)
|
||||
if lane.ExtractExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||
add(lane.Extract)
|
||||
}
|
||||
if lane.MergeExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||
add(lane.Merge)
|
||||
}
|
||||
if lane.NormalizeExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||
add(lane.Normalize)
|
||||
}
|
||||
}
|
||||
for _, chain := range resolved.ValidatorChains {
|
||||
for _, validator := range chain.Validators {
|
||||
@@ -965,6 +982,9 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
|
||||
}
|
||||
}
|
||||
}
|
||||
if resolved.OutputExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||
add(resolved.Output)
|
||||
}
|
||||
ids := make([]string, 0, len(seen))
|
||||
for id := range seen {
|
||||
ids = append(ids, id)
|
||||
@@ -1048,7 +1068,7 @@ func runConfigValidate(args []string, stdout, stderr io.Writer, opts Options) in
|
||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline)); err != nil {
|
||||
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline), opts.promptKitAssets); err != nil {
|
||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
|
||||
@@ -276,7 +276,7 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("validator profile remains distinct", func(t *testing.T) {
|
||||
t.Run("runtime override applies to validators", func(t *testing.T) {
|
||||
roots := newStateTestRoots(t)
|
||||
profileDir := writeRunContractProfiles(t, "override-profile", "validator-profile")
|
||||
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||
@@ -294,11 +294,11 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
||||
if code != 0 || stderr.Len() != 0 {
|
||||
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||
}
|
||||
if len(factoryProfiles) != 1 || factoryProfiles[0] != "" {
|
||||
t.Fatalf("factory profiles = %#v, want one call without a unique profile", factoryProfiles)
|
||||
if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" {
|
||||
t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles)
|
||||
}
|
||||
if len(validatorProfiles) != 1 || validatorProfiles[0] != "validator-profile" {
|
||||
t.Fatalf("validator profiles = %#v, want configured validator profile", validatorProfiles)
|
||||
if len(validatorProfiles) != 1 || validatorProfiles[0] != "override-profile" {
|
||||
t.Fatalf("validator profiles = %#v, want runtime override", validatorProfiles)
|
||||
}
|
||||
})
|
||||
|
||||
@@ -318,6 +318,24 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
||||
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("pipeline default is rejected before factory access", func(t *testing.T) {
|
||||
roots := newStateTestRoots(t)
|
||||
profileDir := writeRunContractProfiles(t, "configured-profile")
|
||||
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||
replaceStateTestConfigLine(t, roots.config, " sample:\n", " sample:\n llm_profile: missing-profile\n")
|
||||
factoryCalls := 0
|
||||
opts := newStateTestHarness().options()
|
||||
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
factoryCalls++
|
||||
return nil, nil, nil
|
||||
}
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, &stdout, &stderr, opts)
|
||||
if code != 1 || !strings.Contains(stderr.String(), "not configured") || factoryCalls != 0 || stdout.Len() != 0 {
|
||||
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestRunReasoningEffortOverrideReachesFactory(t *testing.T) {
|
||||
@@ -445,24 +463,30 @@ func TestReasoningEffortOverrideSeparatesCheckpointIdentities(t *testing.T) {
|
||||
|
||||
func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
|
||||
resolved := pipeline.ResolvedPipeline{
|
||||
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
|
||||
Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "},
|
||||
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
|
||||
InputExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "},
|
||||
ChunkExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Steps: []pipeline.ResolvedPipelineStep{{
|
||||
ID: "default",
|
||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
||||
Extract: pipeline.ModuleBinding{LLMProfile: "alpha"},
|
||||
Merge: pipeline.ModuleBinding{LLMProfile: "zeta"},
|
||||
Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "},
|
||||
Extract: pipeline.ModuleBinding{LLMProfile: "alpha"},
|
||||
ExtractExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Merge: pipeline.ModuleBinding{LLMProfile: "deterministic-merge"},
|
||||
MergeExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "},
|
||||
NormalizeExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
}},
|
||||
}},
|
||||
ValidatorChains: []pipeline.ResolvedValidatorChain{{Validators: []pipeline.ResolvedValidator{
|
||||
{Binding: pipeline.ModuleBinding{LLMProfile: "deterministic-profile"}, ExecutionClass: contracts.ExecutionClassDeterministic},
|
||||
{Binding: pipeline.ModuleBinding{LLMProfile: "beta"}, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||
}}},
|
||||
Output: pipeline.ModuleBinding{LLMProfile: "output-profile"},
|
||||
Output: pipeline.ModuleBinding{LLMProfile: "output-profile"},
|
||||
OutputExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
}
|
||||
got := effectiveLLMProfileIDs(resolved)
|
||||
want := []string{"alpha", "beta", "gamma", "zeta"}
|
||||
want := []string{"alpha", "beta", "gamma", "input-profile", "output-profile", "zeta"}
|
||||
if strings.Join(got, ",") != strings.Join(want, ",") {
|
||||
t.Fatalf("effective profiles = %#v, want %#v", got, want)
|
||||
}
|
||||
@@ -526,7 +550,7 @@ func TestRunFactoryAndPreparationFailuresAreProcessFailures(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
opts := newStateTestHarness().options()
|
||||
if err := pipeline.RegisterExtractorBuilder(opts.Registries.Extractors, pipeline.ModuleSpec{Key: "test/failing-extract", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Extractor[stateTestArtifact], error) {
|
||||
if err := pipeline.RegisterExtractorBuilder(opts.Registries.Extractors, pipeline.ModuleSpec{Key: "test/failing-extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Extractor[stateTestArtifact], error) {
|
||||
return nil, errors.New("injected extractor construction failure")
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
|
||||
@@ -823,22 +823,22 @@ func (h *stateTestHarness) options() Options {
|
||||
if err := pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, stateTestCodec{}); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
if err := registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }); err != nil {
|
||||
if err := registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
if err := registries.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/chunk", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "cache-reference"}}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Chunker, error) { return stateTestChunker{h}, nil }); err != nil {
|
||||
if err := registries.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "cache-reference"}}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Chunker, error) { return stateTestChunker{h}, nil }); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
if err := pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "test/extract", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{h}, nil }); err != nil {
|
||||
if err := pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "test/extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{h}, nil }); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
if err := pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "test/merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{harness: h}, nil }); err != nil {
|
||||
if err := pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "test/merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{harness: h}, nil }); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
if err := pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "test/normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{harness: h}, nil }); err != nil {
|
||||
if err := pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "test/normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{harness: h}, nil }); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
||||
if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
||||
return stateTestOutput{harness: h, includeWarnings: h.includeWarnings}, nil
|
||||
}); err != nil {
|
||||
panic(err)
|
||||
|
||||
@@ -42,12 +42,10 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
|
||||
}
|
||||
profile = clonePipelineProfile(profile)
|
||||
profile.ID = pipelineID
|
||||
if override := strings.TrimSpace(input.LLMProfileOverride); override != "" {
|
||||
applyLLMProfileOverride(&profile, override)
|
||||
}
|
||||
|
||||
resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{
|
||||
Only: input.Only,
|
||||
LLMProfileOverride: input.LLMProfileOverride,
|
||||
ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...),
|
||||
ReferenceUnbinds: append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...),
|
||||
}, input.Catalog)
|
||||
@@ -65,22 +63,6 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
|
||||
}, nil
|
||||
}
|
||||
|
||||
func applyLLMProfileOverride(profile *pipeline.PipelineProfile, profileID string) {
|
||||
profile.Chunk.LLMProfile = profileID
|
||||
apply := func(artifacts map[string]pipeline.ArtifactLaneProfile) {
|
||||
for laneID, lane := range artifacts {
|
||||
lane.Extract.LLMProfile = profileID
|
||||
lane.Merge.LLMProfile = profileID
|
||||
lane.Normalize.LLMProfile = profileID
|
||||
artifacts[laneID] = lane
|
||||
}
|
||||
}
|
||||
apply(profile.Artifacts)
|
||||
for index := range profile.Steps {
|
||||
apply(profile.Steps[index].Artifacts)
|
||||
}
|
||||
}
|
||||
|
||||
func lookupPipelineProfile(profiles map[string]pipeline.PipelineProfile, pipelineID string) (pipeline.PipelineProfile, bool) {
|
||||
pipelineID = strings.TrimSpace(pipelineID)
|
||||
for rawID, profile := range profiles {
|
||||
|
||||
@@ -204,7 +204,7 @@ func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestEffectiveConfigLLMProfileOverrideChangesDigestWithoutOverridingValidators(t *testing.T) {
|
||||
func TestEffectiveConfigLLMProfileOverrideChangesDigestAndOverridesValidators(t *testing.T) {
|
||||
profile := effectiveProfile()
|
||||
profile.Chunk.LLMProfile = "chunk-profile"
|
||||
lane := profile.Artifacts["lane"]
|
||||
@@ -237,8 +237,27 @@ func TestEffectiveConfigLLMProfileOverrideChangesDigestWithoutOverridingValidato
|
||||
t.Fatalf("pipeline profile override was not applied: %#v", resolved)
|
||||
}
|
||||
validators := findEffectiveValidatorChain(resolved, pipeline.StageExtract, "lane")
|
||||
if len(validators.Validators) != 1 || validators.Validators[0].Binding.LLMProfile != "validator-profile" {
|
||||
t.Fatalf("validator profile was overridden: %#v", validators)
|
||||
if len(validators.Validators) != 1 || validators.Validators[0].Binding.LLMProfile != "override-profile" {
|
||||
t.Fatalf("validator profile = %#v, want runtime override", validators)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEffectiveConfigPipelineLLMProfileIsInheritedWithoutMutatingConfig(t *testing.T) {
|
||||
profile := effectiveProfile()
|
||||
profile.LLMProfile = " configured-profile "
|
||||
effective, err := resolveEffectiveProfile(t, profile, ResolveInput{})
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve() error = %v", err)
|
||||
}
|
||||
if got := effective.Config.Pipelines["main"].LLMProfile; got != " configured-profile " {
|
||||
t.Fatalf("effective config pipeline llm profile = %q, want preserved programmatic value", got)
|
||||
}
|
||||
resolved := effective.ResolvedPipeline
|
||||
if got := resolved.Chunk.LLMProfile; got != "configured-profile" {
|
||||
t.Fatalf("resolved chunk profile = %q, want inherited profile", got)
|
||||
}
|
||||
if got := resolved.Steps[0].ArtifactLanes[0].Extract.LLMProfile; got != "configured-profile" {
|
||||
t.Fatalf("resolved extract profile = %q, want inherited profile", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -485,7 +504,7 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
|
||||
if err := pipeline.RegisterArtifactCodec(catalog.ArtifactCodecs, effectiveCodec{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := catalog.Inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
|
||||
if err := catalog.Inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
|
||||
return effectiveInput{key: "input"}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -493,6 +512,7 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
|
||||
chunkSpec := pipeline.ModuleSpec{
|
||||
Key: pipeline.DefaultChunkModule,
|
||||
Stage: pipeline.StageChunk,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: []string{"source"},
|
||||
Provides: []string{"chunk"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-ref"}},
|
||||
@@ -503,32 +523,32 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := catalog.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "needs-capability", Stage: pipeline.StageChunk, Requires: []string{"missing"}}, chunkOptions, func(pipeline.BuildRequest) (contracts.Chunker, error) {
|
||||
if err := catalog.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "needs-capability", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}}, chunkOptions, func(pipeline.BuildRequest) (contracts.Chunker, error) {
|
||||
return effectiveChunker{key: "needs-capability"}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := pipeline.RegisterExtractor(catalog.Extractors, pipeline.ModuleSpec{Key: "extract", Stage: pipeline.StageExtract, ArtifactKind: effectiveArtifactKind, Requires: []string{"chunk"}, Provides: []string{"candidate"}}, func() (contracts.Extractor[effectiveArtifact], error) {
|
||||
if err := pipeline.RegisterExtractor(catalog.Extractors, pipeline.ModuleSpec{Key: "extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"chunk"}, Provides: []string{"candidate"}}, func() (contracts.Extractor[effectiveArtifact], error) {
|
||||
return effectiveExtractor{key: "extract"}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: pipeline.DefaultMergeModule, Stage: pipeline.StageMerge, ArtifactKind: effectiveArtifactKind, Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
||||
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: pipeline.DefaultMergeModule, Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
||||
return effectiveMerger{key: pipeline.DefaultMergeModule}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: "other-merge", Stage: pipeline.StageMerge, ArtifactKind: "other-kind", Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
||||
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: "other-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "other-kind", Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
||||
return effectiveMerger{key: "other-merge"}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := pipeline.RegisterNormalizer(catalog.Normalizers, pipeline.ModuleSpec{Key: pipeline.DefaultNormalizeModule, Stage: pipeline.StageNormalize, ArtifactKind: effectiveArtifactKind, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer[effectiveArtifact], error) {
|
||||
if err := pipeline.RegisterNormalizer(catalog.Normalizers, pipeline.ModuleSpec{Key: pipeline.DefaultNormalizeModule, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer[effectiveArtifact], error) {
|
||||
return effectiveNormalizer{key: pipeline.DefaultNormalizeModule}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := catalog.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: pipeline.DefaultOutputModule, Stage: pipeline.StageOutput, Requires: []string{"normalized"}}, func() (contracts.OutputEncoder, error) {
|
||||
if err := catalog.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: pipeline.DefaultOutputModule, Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}}, func() (contracts.OutputEncoder, error) {
|
||||
return effectiveOutput{key: pipeline.DefaultOutputModule}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
|
||||
@@ -34,21 +34,23 @@ type FilePromptKitLocalBackendConfig struct {
|
||||
}
|
||||
|
||||
type FilePipelineProfile struct {
|
||||
Input fileModuleBinding `yaml:"input"`
|
||||
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
|
||||
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
|
||||
Steps []FilePipelineStepProfile `yaml:"steps,omitempty"`
|
||||
Output *fileModuleBinding `yaml:"output,omitempty"`
|
||||
References map[string]fileReferenceSource `yaml:"references,omitempty"`
|
||||
artifactsSet bool `yaml:"-"`
|
||||
stepsSet bool `yaml:"-"`
|
||||
LLMProfile *string `yaml:"llm_profile,omitempty"`
|
||||
Input fileModuleBinding `yaml:"input"`
|
||||
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
|
||||
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
|
||||
Steps []FilePipelineStepProfile `yaml:"steps,omitempty"`
|
||||
Output *fileModuleBinding `yaml:"output,omitempty"`
|
||||
References map[string]fileReferenceSource `yaml:"references,omitempty"`
|
||||
artifactsSet bool `yaml:"-"`
|
||||
stepsSet bool `yaml:"-"`
|
||||
llmProfileSet bool `yaml:"-"`
|
||||
}
|
||||
|
||||
func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
|
||||
type plainFilePipelineProfile FilePipelineProfile
|
||||
var decoded plainFilePipelineProfile
|
||||
seen, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
|
||||
"input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
|
||||
"llm_profile": {}, "input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
|
||||
}, "pipeline profile")
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -56,6 +58,7 @@ func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
|
||||
*p = FilePipelineProfile(decoded)
|
||||
_, p.artifactsSet = seen["artifacts"]
|
||||
_, p.stepsSet = seen["steps"]
|
||||
_, p.llmProfileSet = seen["llm_profile"]
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -257,6 +260,9 @@ func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
|
||||
return err
|
||||
}
|
||||
b.LLMProfile = strings.TrimSpace(llmProfile)
|
||||
if b.LLMProfile == "" {
|
||||
return fmt.Errorf("llm_profile must not be empty when set")
|
||||
}
|
||||
case "retries":
|
||||
var retries int
|
||||
if err := valueNode.Decode(&retries); err != nil {
|
||||
@@ -492,6 +498,13 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
||||
|
||||
for _, pipelineID := range pipelineIDs {
|
||||
filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]]
|
||||
llmProfile := ""
|
||||
if filePipeline.llmProfileSet || filePipeline.LLMProfile != nil {
|
||||
if filePipeline.LLMProfile == nil || strings.TrimSpace(*filePipeline.LLMProfile) == "" {
|
||||
return fmt.Errorf("pipeline %q llm_profile must not be empty when set", pipelineID)
|
||||
}
|
||||
llmProfile = strings.TrimSpace(*filePipeline.LLMProfile)
|
||||
}
|
||||
hasSteps := filePipeline.stepsSet || filePipeline.Steps != nil
|
||||
laneIDs, rawLaneIDs, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID))
|
||||
if err != nil {
|
||||
@@ -499,6 +512,7 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
||||
}
|
||||
profile := pipeline.PipelineProfile{
|
||||
ID: pipelineID,
|
||||
LLMProfile: llmProfile,
|
||||
Input: filePipeline.Input.toPipelineBinding(),
|
||||
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
|
||||
References: fileReferenceSourcesToPipeline(filePipeline.References),
|
||||
|
||||
@@ -2,6 +2,7 @@ package config
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
@@ -49,6 +50,93 @@ func TestFileConfigMinimalVersion4AppliesOverDefaults(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilePipelineLLMProfileIsPresenceAwareAndDetached(t *testing.T) {
|
||||
const pipelineYAML = `version: 4
|
||||
pipelines:
|
||||
main:
|
||||
%s
|
||||
input: input
|
||||
artifacts:
|
||||
lane:
|
||||
extract: extract
|
||||
`
|
||||
|
||||
t.Run("omitted", func(t *testing.T) {
|
||||
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, ""))
|
||||
if file.Pipelines["main"].LLMProfile != nil || file.Pipelines["main"].llmProfileSet {
|
||||
t.Fatalf("parsed pipeline profile = %#v, want omitted llm profile", file.Pipelines["main"])
|
||||
}
|
||||
cfg := Default()
|
||||
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := cfg.Pipelines["main"].LLMProfile; got != "" {
|
||||
t.Fatalf("pipeline llm profile = %q, want empty", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("trimmed and detached", func(t *testing.T) {
|
||||
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, "llm_profile: ' configured-profile '"))
|
||||
cfg := Default()
|
||||
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := cfg.Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||
t.Fatalf("pipeline llm profile = %q, want trimmed value", got)
|
||||
}
|
||||
*file.Pipelines["main"].LLMProfile = "changed-profile"
|
||||
if got := cfg.Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||
t.Fatalf("effective config aliases parsed file: %q", got)
|
||||
}
|
||||
if got := cloneConfig(cfg).Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||
t.Fatalf("cloned pipeline llm profile = %q", got)
|
||||
}
|
||||
data, err := json.Marshal(cfg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var roundTripped Config
|
||||
if err := json.Unmarshal(data, &roundTripped); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := roundTripped.Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||
t.Fatalf("round-tripped pipeline llm profile = %q", got)
|
||||
}
|
||||
})
|
||||
|
||||
for _, value := range []string{"''", "' '", "null"} {
|
||||
t.Run("explicit empty "+value, func(t *testing.T) {
|
||||
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, "llm_profile: "+value))
|
||||
cfg := Default()
|
||||
err := cfg.ApplyFileConfig(file)
|
||||
if err == nil || !strings.Contains(err.Error(), `pipeline "main" llm_profile must not be empty`) {
|
||||
t.Fatalf("ApplyFileConfig() error = %v, want explicit-empty rejection", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileModuleBindingRejectsExplicitEmptyLLMProfile(t *testing.T) {
|
||||
const configYAML = `version: 4
|
||||
pipelines:
|
||||
main:
|
||||
input:
|
||||
module: input
|
||||
llm_profile: %s
|
||||
artifacts:
|
||||
lane:
|
||||
extract: extract
|
||||
`
|
||||
for _, value := range []string{"''", "' '", "null"} {
|
||||
t.Run(value, func(t *testing.T) {
|
||||
_, err := ParseFileConfigYAML([]byte(fmt.Sprintf(configYAML, value)))
|
||||
if err == nil || !strings.Contains(err.Error(), "llm_profile must not be empty when set") {
|
||||
t.Fatalf("ParseFileConfigYAML() error = %v, want explicit-empty binding profile rejection", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilePromptKitProfileSourcesSurviveConfigBoundaries(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
|
||||
@@ -19,23 +19,28 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
|
||||
}
|
||||
|
||||
resolved := pipeline.ResolvedPipeline{
|
||||
ID: "redaction-test",
|
||||
Digest: "sha256:safe-digest",
|
||||
Input: bindings["input"],
|
||||
Chunk: bindings["chunk"],
|
||||
ChunkReferences: redactionTestReferenceTarget(pipeline.StageChunk, "", "chunk-reference-content"),
|
||||
ID: "redaction-test",
|
||||
Digest: "sha256:safe-digest",
|
||||
Input: bindings["input"],
|
||||
InputExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Chunk: bindings["chunk"],
|
||||
ChunkExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
ChunkReferences: redactionTestReferenceTarget(pipeline.StageChunk, "", "chunk-reference-content"),
|
||||
Steps: []pipeline.ResolvedPipelineStep{{
|
||||
ID: "default",
|
||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
||||
ID: "safe-lane",
|
||||
ArtifactKind: "safe/artifact",
|
||||
Extract: bindings["extract"],
|
||||
Merge: bindings["merge"],
|
||||
Normalize: bindings["normalize"],
|
||||
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
|
||||
ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"),
|
||||
MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"),
|
||||
NormalizeReferences: redactionTestReferenceTarget(pipeline.StageNormalize, "safe-lane", "normalize-reference-content"),
|
||||
ID: "safe-lane",
|
||||
ArtifactKind: "safe/artifact",
|
||||
Extract: bindings["extract"],
|
||||
ExtractExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Merge: bindings["merge"],
|
||||
MergeExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Normalize: bindings["normalize"],
|
||||
NormalizeExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
|
||||
ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"),
|
||||
MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"),
|
||||
NormalizeReferences: redactionTestReferenceTarget(pipeline.StageNormalize, "safe-lane", "normalize-reference-content"),
|
||||
}},
|
||||
}},
|
||||
ValidatorChains: []pipeline.ResolvedValidatorChain{{
|
||||
@@ -49,7 +54,8 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
|
||||
ArtifactKind: "safe/artifact",
|
||||
}},
|
||||
}},
|
||||
Output: bindings["output"],
|
||||
Output: bindings["output"],
|
||||
OutputExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
}
|
||||
effective := EffectiveConfig{
|
||||
Config: Config{Pipelines: map[string]pipeline.PipelineProfile{
|
||||
@@ -88,6 +94,11 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
|
||||
t.Fatalf("resolved pipeline summary does not retain %q: %s", safe, text)
|
||||
}
|
||||
}
|
||||
for _, executionClass := range []string{"input_execution_class\":\"deterministic", "chunk_execution_class\":\"llm_backed", "extract_execution_class\":\"llm_backed", "merge_execution_class\":\"deterministic", "normalize_execution_class\":\"llm_backed", "output_execution_class\":\"deterministic"} {
|
||||
if !strings.Contains(text, executionClass) {
|
||||
t.Fatalf("resolved pipeline summary does not retain %q: %s", executionClass, text)
|
||||
}
|
||||
}
|
||||
|
||||
payload.Input.Options["safe"] = "mutated"
|
||||
nested := payload.Input.Options["nested"].([]any)[0].([]any)[0].(map[string]any)
|
||||
|
||||
@@ -267,10 +267,6 @@ const (
|
||||
ExecutionClassLLMBacked ExecutionClass = "llm_backed"
|
||||
)
|
||||
|
||||
type ChunkExecutionClassProvider interface {
|
||||
ExecutionClass() ExecutionClass
|
||||
}
|
||||
|
||||
type ValidationResult struct {
|
||||
Approved bool `json:"approved"`
|
||||
ReasonCode string `json:"reason_code,omitempty"`
|
||||
|
||||
@@ -21,8 +21,9 @@ type AssetSource struct {
|
||||
}
|
||||
|
||||
type AssetRegistry struct {
|
||||
prompts []AssetSource
|
||||
schemas []AssetSource
|
||||
prompts []AssetSource
|
||||
schemas []AssetSource
|
||||
fallbackProfiles []AssetSource
|
||||
}
|
||||
|
||||
type AssetHashPart struct {
|
||||
@@ -58,6 +59,20 @@ func (r *AssetRegistry) RegisterSchemaFS(fsys fs.FS, root string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// RegisterFallbackProfileFS registers profile assets that PromptKit uses only
|
||||
// when an operator-configured source does not provide a matching profile.
|
||||
func (r *AssetRegistry) RegisterFallbackProfileFS(fsys fs.FS, root string) error {
|
||||
if r == nil {
|
||||
return fmt.Errorf("asset registry must not be nil")
|
||||
}
|
||||
source, err := newAssetSource(fsys, root)
|
||||
if err != nil {
|
||||
return fmt.Errorf("register fallback profile assets: %w", err)
|
||||
}
|
||||
r.fallbackProfiles = append(r.fallbackProfiles, source)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *AssetRegistry) PromptFS() (fs.FS, error) {
|
||||
if r == nil {
|
||||
return nil, fmt.Errorf("asset registry must not be nil")
|
||||
@@ -72,19 +87,76 @@ func (r *AssetRegistry) SchemaFS() (fs.FS, error) {
|
||||
return flattenAssetSources(r.schemas)
|
||||
}
|
||||
|
||||
func (r *AssetRegistry) FallbackProfileFS() (fs.FS, error) {
|
||||
if r == nil {
|
||||
return nil, fmt.Errorf("asset registry must not be nil")
|
||||
}
|
||||
return flattenAssetSources(r.fallbackProfiles)
|
||||
}
|
||||
|
||||
// FallbackProfileDigest returns a deterministic, non-secret identity for the
|
||||
// flattened fallback profile assets.
|
||||
func (r *AssetRegistry) FallbackProfileDigest() (string, error) {
|
||||
_, digest, _, err := r.fallbackProfileAssets()
|
||||
return digest, err
|
||||
}
|
||||
|
||||
func (r *AssetRegistry) PromptKitOptions() ([]promptkit.Option, error) {
|
||||
options, _, err := r.promptKitOptions()
|
||||
return options, err
|
||||
}
|
||||
|
||||
func (r *AssetRegistry) promptKitOptions() ([]promptkit.Option, string, error) {
|
||||
promptFS, err := r.PromptFS()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("prepare prompt assets: %w", err)
|
||||
return nil, "", fmt.Errorf("prepare prompt assets: %w", err)
|
||||
}
|
||||
schemaFS, err := r.SchemaFS()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("prepare schema assets: %w", err)
|
||||
return nil, "", fmt.Errorf("prepare schema assets: %w", err)
|
||||
}
|
||||
return []promptkit.Option{
|
||||
options := []promptkit.Option{
|
||||
promptkit.WithPromptFS(promptFS, "."),
|
||||
promptkit.WithSchemaFS(schemaFS, "."),
|
||||
}, nil
|
||||
}
|
||||
fallbackFS, fallbackDigest, hasFallback, err := r.fallbackProfileAssets()
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
if hasFallback {
|
||||
options = append(options, promptkit.WithFallbackProfileFS(fallbackFS, "."))
|
||||
}
|
||||
return options, fallbackDigest, nil
|
||||
}
|
||||
|
||||
func (r *AssetRegistry) promptKitFallbackProfileOption() (promptkit.Option, bool, error) {
|
||||
fallbackFS, _, hasFallback, err := r.fallbackProfileAssets()
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
if !hasFallback {
|
||||
return nil, false, nil
|
||||
}
|
||||
return promptkit.WithFallbackProfileFS(fallbackFS, "."), true, nil
|
||||
}
|
||||
|
||||
func (r *AssetRegistry) fallbackProfileAssets() (fs.FS, string, bool, error) {
|
||||
if r == nil {
|
||||
return nil, "", false, fmt.Errorf("asset registry must not be nil")
|
||||
}
|
||||
if len(r.fallbackProfiles) == 0 {
|
||||
empty := sha256.Sum256([]byte("notarius:fallback-profile-assets:empty"))
|
||||
return nil, "sha256:" + hex.EncodeToString(empty[:]), false, nil
|
||||
}
|
||||
fallbackFS, err := r.FallbackProfileFS()
|
||||
if err != nil {
|
||||
return nil, "", false, fmt.Errorf("prepare fallback profile assets: %w", err)
|
||||
}
|
||||
digest, err := hashAssetFilesystem(fallbackFS)
|
||||
if err != nil {
|
||||
return nil, "", false, err
|
||||
}
|
||||
return fallbackFS, digest, true, nil
|
||||
}
|
||||
|
||||
func HashAssets(parts []AssetHashPart) (string, error) {
|
||||
@@ -117,6 +189,28 @@ func HashAssets(parts []AssetHashPart) (string, error) {
|
||||
return "sha256:" + hex.EncodeToString(hash.Sum(nil)), nil
|
||||
}
|
||||
|
||||
func hashAssetFilesystem(fsys fs.FS) (string, error) {
|
||||
var parts []AssetHashPart
|
||||
err := fs.WalkDir(fsys, ".", func(name string, entry fs.DirEntry, walkErr error) error {
|
||||
if walkErr != nil {
|
||||
return walkErr
|
||||
}
|
||||
if entry.IsDir() {
|
||||
return nil
|
||||
}
|
||||
parts = append(parts, AssetHashPart{FS: fsys, Path: name})
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("walk assets for digest: %w", err)
|
||||
}
|
||||
if len(parts) > 0 {
|
||||
return HashAssets(parts)
|
||||
}
|
||||
empty := sha256.Sum256([]byte("notarius:fallback-profile-assets:empty"))
|
||||
return "sha256:" + hex.EncodeToString(empty[:]), nil
|
||||
}
|
||||
|
||||
func newAssetSource(fsys fs.FS, root string) (AssetSource, error) {
|
||||
if fsys == nil {
|
||||
return AssetSource{}, fmt.Errorf("filesystem must not be nil")
|
||||
@@ -248,6 +342,9 @@ func (m assetMapFS) dirEntries(dir string) []fs.DirEntry {
|
||||
children[childName] = entry
|
||||
}
|
||||
if len(children) == 0 {
|
||||
if dir == "." {
|
||||
return []fs.DirEntry{}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
names := make([]string, 0, len(children))
|
||||
|
||||
@@ -2,6 +2,7 @@ package llm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io/fs"
|
||||
"strings"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
@@ -98,6 +99,89 @@ func TestAssetRegistryRejectsDuplicateAssetPaths(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssetRegistryCombinesFallbackProfileSources(t *testing.T) {
|
||||
registry := NewAssetRegistry()
|
||||
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
|
||||
"first/profiles/one.yaml": {Data: []byte("id: one\nmodel: first\n")},
|
||||
}, "first/profiles"); err != nil {
|
||||
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||
}
|
||||
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
|
||||
"second/two.yaml": {Data: []byte("id: two\nmodel: second\n")},
|
||||
}, "second"); err != nil {
|
||||
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
fallbackFS, err := registry.FallbackProfileFS()
|
||||
if err != nil {
|
||||
t.Fatalf("FallbackProfileFS() error = %v, want nil", err)
|
||||
}
|
||||
for _, name := range []string{"one.yaml", "two.yaml"} {
|
||||
if _, err := fs.ReadFile(fallbackFS, name); err != nil {
|
||||
t.Fatalf("FallbackProfileFS().ReadFile(%q) error = %v, want nil", name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssetRegistryRejectsInvalidFallbackProfileRoot(t *testing.T) {
|
||||
registry := NewAssetRegistry()
|
||||
err := registry.RegisterFallbackProfileFS(fstest.MapFS{}, "../profiles")
|
||||
if err == nil || !strings.Contains(err.Error(), "invalid path") {
|
||||
t.Fatalf("RegisterFallbackProfileFS() error = %v, want invalid root error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssetRegistryRejectsUnreadableFallbackProfileAssets(t *testing.T) {
|
||||
registry := NewAssetRegistry()
|
||||
if err := registry.RegisterFallbackProfileFS(unreadableAssetFS{}, "."); err != nil {
|
||||
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||
}
|
||||
_, err := registry.FallbackProfileFS()
|
||||
if err == nil || !strings.Contains(err.Error(), "permission denied") {
|
||||
t.Fatalf("FallbackProfileFS() error = %v, want unreadable asset error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssetRegistryRejectsDuplicateFallbackProfilePaths(t *testing.T) {
|
||||
registry := NewAssetRegistry()
|
||||
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{"first/profile.yaml": {Data: []byte("id: first\n")}}, "first"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{"second/profile.yaml": {Data: []byte("id: second\n")}}, "second"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, err := registry.FallbackProfileFS()
|
||||
if err == nil || !strings.Contains(err.Error(), "duplicate asset path") {
|
||||
t.Fatalf("FallbackProfileFS() error = %v, want duplicate path error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssetRegistryFallbackProfileDigestTracksContentWithoutLeakingIt(t *testing.T) {
|
||||
digestFor := func(content string) string {
|
||||
t.Helper()
|
||||
registry := NewAssetRegistry()
|
||||
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
|
||||
"profiles/profile.yaml": {Data: []byte(content)},
|
||||
}, "profiles"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
digest, err := registry.FallbackProfileDigest()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return digest
|
||||
}
|
||||
|
||||
first := digestFor("id: fallback\nmodel: model-one\n")
|
||||
second := digestFor("id: fallback\nmodel: model-two\n")
|
||||
if first == second {
|
||||
t.Fatalf("fallback profile digests = %q and %q, want content change", first, second)
|
||||
}
|
||||
if !strings.HasPrefix(first, "sha256:") || strings.Contains(first, "model-one") || strings.Contains(first, "profile.yaml") {
|
||||
t.Fatalf("fallback profile digest leaked source details: %q", first)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssetRegistryCombinesNamespacedPromptSources(t *testing.T) {
|
||||
registry := NewAssetRegistry()
|
||||
mustRegisterPromptFS(t, registry, fstest.MapFS{
|
||||
@@ -196,3 +280,9 @@ output:
|
||||
repair_attempts: 0
|
||||
`
|
||||
}
|
||||
|
||||
type unreadableAssetFS struct{}
|
||||
|
||||
func (unreadableAssetFS) Open(name string) (fs.File, error) {
|
||||
return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrPermission}
|
||||
}
|
||||
|
||||
@@ -37,12 +37,13 @@ type PromptKitClientConfig struct {
|
||||
}
|
||||
|
||||
type PromptKitClient struct {
|
||||
engine *promptkit.Engine
|
||||
recorder *LLMProfileRecorder
|
||||
profileDir string
|
||||
profileFile string
|
||||
localEndpoint string
|
||||
reasoningEffort *string
|
||||
engine *promptkit.Engine
|
||||
recorder *LLMProfileRecorder
|
||||
profileDir string
|
||||
profileFile string
|
||||
localEndpoint string
|
||||
fallbackProfileDigest string
|
||||
reasoningEffort *string
|
||||
}
|
||||
|
||||
type LLMProfileRecorder struct {
|
||||
@@ -61,27 +62,23 @@ func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
||||
if cfg.Assets == nil {
|
||||
return nil, fmt.Errorf("PromptKit client assets must not be nil")
|
||||
}
|
||||
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
||||
return nil, fmt.Errorf("PromptKit profile_dir and profile_file are mutually exclusive")
|
||||
}
|
||||
options, err := cfg.Assets.PromptKitOptions()
|
||||
profileSource, profileOptions, err := promptKitProfileSourceEngineOptions(PromptKitProfileSourceConfig{
|
||||
ProfileDir: cfg.ProfileDir,
|
||||
ProfileFile: cfg.ProfileFile,
|
||||
LocalBackend: cfg.LocalBackend,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if profileFile := strings.TrimSpace(cfg.ProfileFile); profileFile != "" {
|
||||
options = append(options, promptkit.WithProfileFile(profileFile))
|
||||
}
|
||||
var localEndpoint string
|
||||
if cfg.LocalBackend != nil {
|
||||
localBackend := *cfg.LocalBackend
|
||||
localBackend.Endpoint = strings.TrimSpace(localBackend.Endpoint)
|
||||
localEndpoint = localBackend.Endpoint
|
||||
options = append(options, PromptKitLocalBackendOption(localBackend))
|
||||
options, fallbackProfileDigest, err := cfg.Assets.promptKitOptions()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
options = append(options, profileOptions...)
|
||||
options = append(options, cfg.EngineOptions...)
|
||||
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||
ProfileDir: strings.TrimSpace(cfg.ProfileDir),
|
||||
ProfileDir: profileSource.ProfileDir,
|
||||
Timeout: cfg.Timeout,
|
||||
HTTPClient: cfg.HTTPClient,
|
||||
}, options...)
|
||||
@@ -98,12 +95,13 @@ func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
||||
reasoningEffort = &value
|
||||
}
|
||||
return &PromptKitClient{
|
||||
engine: engine,
|
||||
recorder: recorder,
|
||||
profileDir: strings.TrimSpace(cfg.ProfileDir),
|
||||
profileFile: strings.TrimSpace(cfg.ProfileFile),
|
||||
localEndpoint: localEndpoint,
|
||||
reasoningEffort: reasoningEffort,
|
||||
engine: engine,
|
||||
recorder: recorder,
|
||||
profileDir: profileSource.ProfileDir,
|
||||
profileFile: profileSource.ProfileFile,
|
||||
localEndpoint: profileSource.localEndpoint(),
|
||||
fallbackProfileDigest: fallbackProfileDigest,
|
||||
reasoningEffort: reasoningEffort,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -139,19 +137,31 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
|
||||
Vars: promptKitVars(req, sessionID),
|
||||
Execution: execution,
|
||||
}
|
||||
prepared, err := c.engine.Prepare(ctx, runReq)
|
||||
prepared, err := c.engine.PrepareExecution(ctx, runReq)
|
||||
if err != nil {
|
||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||
return contracts.StructuredCompletionResponse{}, ctxErr
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("prepare PromptKit prompt %q: %w", promptID, redactPromptKitError(err))
|
||||
}
|
||||
result, err := c.engine.Run(ctx, runReq)
|
||||
defer prepared.Discard()
|
||||
preparedDetails := prepared.Details()
|
||||
result, err := c.engine.RunPrepared(ctx, prepared)
|
||||
if err != nil {
|
||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||
return contracts.StructuredCompletionResponse{}, ctxErr
|
||||
}
|
||||
if errors.Is(err, promptkit.ErrCapacityExceeded) {
|
||||
var capacityErr *promptkit.CapacityError
|
||||
if errors.As(err, &capacityErr) && strings.TrimSpace(capacityErr.BackendID) != "" {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf(
|
||||
"run PromptKit prompt %q on backend %q: %w: %v",
|
||||
promptID,
|
||||
strings.TrimSpace(capacityErr.BackendID),
|
||||
contracts.ErrLLMCapacityExceeded,
|
||||
redactPromptKitError(err),
|
||||
)
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf(
|
||||
"run PromptKit prompt %q: %w: %v",
|
||||
promptID,
|
||||
@@ -164,7 +174,7 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
|
||||
if result == nil {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w: empty result", promptID, contracts.ErrInvalidStructuredOutput)
|
||||
}
|
||||
response := c.responseFromResult(result, prepared)
|
||||
response := c.responseFromResult(result, &preparedDetails)
|
||||
if result.Validation.Status == promptkit.ValidationFailed || !result.Validation.IsValid {
|
||||
return response, fmt.Errorf("run PromptKit prompt %q: %w: validation failed: %s", promptID, contracts.ErrInvalidStructuredOutput, strings.Join(result.Validation.Errors, "; "))
|
||||
}
|
||||
@@ -317,7 +327,7 @@ func (c *PromptKitClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint,
|
||||
if c == nil {
|
||||
return nil, nil
|
||||
}
|
||||
fingerprint, err := promptKitProfileFingerprint(c.profileDir, c.profileFile)
|
||||
fingerprint, err := promptKitProfileFingerprint(c.profileDir, c.profileFile, c.fallbackProfileDigest)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"io/fs"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
@@ -114,6 +115,104 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientUsesOnePreparedSnapshotForDebugAndGeneration(t *testing.T) {
|
||||
const initialPrompt = `id: snapshot.test
|
||||
version: "v1"
|
||||
default_profile: snapshot-profile
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: application/json
|
||||
messages:
|
||||
- role: user
|
||||
content: "Snapshot A: {{ input \"transcript\" }}"
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: adapter.schema.json
|
||||
repair_attempts: 0
|
||||
`
|
||||
const updatedPrompt = `id: snapshot.test
|
||||
version: "v1"
|
||||
default_profile: snapshot-profile
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: application/json
|
||||
messages:
|
||||
- role: user
|
||||
content: "Snapshot B: {{ input \"transcript\" }}"
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: adapter.schema.json
|
||||
repair_attempts: 0
|
||||
`
|
||||
source := &switchingPromptFS{files: fstest.MapFS{
|
||||
"snapshot.test.yaml": {Data: []byte(initialPrompt)},
|
||||
}}
|
||||
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||
Assets: newTestPromptKitAssets(t),
|
||||
EngineOptions: []promptkit.Option{
|
||||
promptkit.WithPromptFS(source, "."),
|
||||
promptkit.WithBackend(promptkit.Backend{ID: "snapshot-backend", Endpoint: "http://promptkit.test/v1"}),
|
||||
promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||
ID: "snapshot-profile",
|
||||
BackendID: "snapshot-backend",
|
||||
Model: "snapshot-model",
|
||||
})),
|
||||
promptkit.WithLLMClient(fake),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
opened := source.holdNextPromptRead()
|
||||
defer source.resumePromptRead()
|
||||
type completion struct {
|
||||
response contracts.StructuredCompletionResponse
|
||||
err error
|
||||
}
|
||||
completed := make(chan completion, 1)
|
||||
go func() {
|
||||
var out struct {
|
||||
OK bool `json:"ok"`
|
||||
}
|
||||
response, callErr := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
PromptID: "snapshot.test",
|
||||
Inputs: contracts.LLMInputSet{
|
||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||
},
|
||||
}, &out)
|
||||
completed <- completion{response: response, err: callErr}
|
||||
}()
|
||||
select {
|
||||
case <-opened:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("PromptKit did not read the prompt source")
|
||||
}
|
||||
source.replacePrompt([]byte(updatedPrompt))
|
||||
source.resumePromptRead()
|
||||
|
||||
result := <-completed
|
||||
if result.err != nil {
|
||||
t.Fatalf("CompleteStructured() error = %v, want nil", result.err)
|
||||
}
|
||||
if result.response.Debug == nil || result.response.Debug.Prompt == nil || len(result.response.Debug.Prompt.Messages) != 1 {
|
||||
t.Fatalf("debug prompt = %#v, want one prepared message", result.response.Debug)
|
||||
}
|
||||
debugContent := result.response.Debug.Prompt.Messages[0].Content
|
||||
generatedContent := fake.lastRequest().Prompt.Messages[0].Content
|
||||
if debugContent != generatedContent {
|
||||
t.Fatalf("debug content = %q, generation content = %q, want one snapshot", debugContent, generatedContent)
|
||||
}
|
||||
if !strings.Contains(debugContent, "Snapshot A") || strings.Contains(debugContent, "Snapshot B") {
|
||||
t.Fatalf("snapshot content = %q, want the source read before it changed", debugContent)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientRetainsSessionPromptVariable(t *testing.T) {
|
||||
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||
client := newTestPromptKitClient(t, fake)
|
||||
@@ -253,9 +352,10 @@ func TestNewPromptKitClientReportsAssetAndEngineConstructionFailures(t *testing.
|
||||
|
||||
func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
|
||||
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
|
||||
const credentialEnvironment = "PROMPTKIT_TEST_API_KEY"
|
||||
writeProfile := func(model string) {
|
||||
t.Helper()
|
||||
content := "id: checkpoint-profile\nendpoint: http://promptkit.test/v1\nmodel: " + model + "\n"
|
||||
content := "id: checkpoint-profile\nendpoint: http://promptkit.test/v1\nmodel: " + model + "\napi_key_env: " + credentialEnvironment + "\n"
|
||||
if err := os.WriteFile(profilePath, []byte(content), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -281,12 +381,21 @@ func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
|
||||
|
||||
writeProfile("model-one")
|
||||
first := fingerprintFor()
|
||||
repeated := fingerprintFor()
|
||||
if first != repeated {
|
||||
t.Fatalf("profile-source fingerprint = %#v then %#v for unchanged source", first, repeated)
|
||||
}
|
||||
writeProfile("model-two")
|
||||
second := fingerprintFor()
|
||||
if first == second {
|
||||
t.Fatalf("profile-source fingerprint = %#v for both profile models", first)
|
||||
}
|
||||
if strings.Contains(first.Value, profilePath) || strings.Contains(first.Value, "model-one") {
|
||||
if strings.TrimSpace(first.Value) == "" {
|
||||
t.Fatal("profile-source fingerprint is empty")
|
||||
}
|
||||
if strings.Contains(first.Value, profilePath) ||
|
||||
strings.Contains(first.Value, "model-one") ||
|
||||
strings.Contains(first.Value, credentialEnvironment) {
|
||||
t.Fatalf("profile-source fingerprint exposes source details: %#v", first)
|
||||
}
|
||||
|
||||
@@ -306,9 +415,152 @@ func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
|
||||
if fresh[0].Value == "mutated" {
|
||||
t.Fatal("LLMCheckpointFingerprints exposed mutable backing storage")
|
||||
}
|
||||
const wantBuiltinFingerprint = "sha256:5218b1dec48f5fdd46836826e0b25906c33efbdf943c5f18085b8d82467e0276"
|
||||
if fresh[0].Value != wantBuiltinFingerprint {
|
||||
t.Fatalf("built-in profile fingerprint = %q, want %q", fresh[0].Value, wantBuiltinFingerprint)
|
||||
secondClient, err := NewPromptKitClient(PromptKitClientConfig{Assets: newTestPromptKitAssets(t)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
secondBuiltin, err := secondClient.LLMCheckpointFingerprints()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(secondBuiltin) != 1 || fresh[0] != secondBuiltin[0] {
|
||||
t.Fatalf("built-in profile fingerprints = %#v and %#v, want deterministic identity", fresh, secondBuiltin)
|
||||
}
|
||||
if strings.TrimSpace(fresh[0].Value) == "" {
|
||||
t.Fatal("built-in profile fingerprint is empty")
|
||||
}
|
||||
|
||||
t.Run("directory layout", func(t *testing.T) {
|
||||
profileDir := t.TempDir()
|
||||
firstPath := filepath.Join(profileDir, "first-profile.yaml")
|
||||
secondPath := filepath.Join(profileDir, "second-profile.yaml")
|
||||
content := []byte("id: directory-profile\nendpoint: http://promptkit.test/v1\nmodel: directory-model\n")
|
||||
if err := os.WriteFile(firstPath, content, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
first, err := promptKitProfileFingerprint(profileDir, "", "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Rename(firstPath, secondPath); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
second, err := promptKitProfileFingerprint(profileDir, "", "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if first == second {
|
||||
t.Fatalf("profile-source fingerprint = %#v after source filename changed", first)
|
||||
}
|
||||
if strings.Contains(first.Value, firstPath) || strings.Contains(second.Value, secondPath) {
|
||||
t.Fatalf("profile-source fingerprint exposes source path: %#v, %#v", first, second)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestPromptKitProfileFingerprintReadErrorsDoNotExposeSourcePaths(t *testing.T) {
|
||||
for _, source := range []struct {
|
||||
name string
|
||||
profileDir string
|
||||
profileFile string
|
||||
}{
|
||||
{
|
||||
name: "file",
|
||||
profileFile: filepath.Join(t.TempDir(), "missing-profile.yaml"),
|
||||
},
|
||||
{
|
||||
name: "directory",
|
||||
profileDir: filepath.Join(t.TempDir(), "missing-profiles"),
|
||||
},
|
||||
} {
|
||||
t.Run(source.name, func(t *testing.T) {
|
||||
_, err := promptKitProfileFingerprint(source.profileDir, source.profileFile, "")
|
||||
if err == nil {
|
||||
t.Fatal("promptKitProfileFingerprint() error = nil, want source read failure")
|
||||
}
|
||||
if (source.profileDir != "" && strings.Contains(err.Error(), source.profileDir)) ||
|
||||
(source.profileFile != "" && strings.Contains(err.Error(), source.profileFile)) {
|
||||
t.Fatalf("fingerprint error exposes profile source path: %q", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientUsesFallbackProfilesForExecutionAndInspection(t *testing.T) {
|
||||
assets := newTestPromptKitAssets(t)
|
||||
const profileID = "fallback-profile"
|
||||
if err := assets.RegisterFallbackProfileFS(fstest.MapFS{
|
||||
"profiles/fallback.yaml": {Data: []byte("id: " + profileID + "\nendpoint: http://promptkit.test/v1\nmodel: fallback-model\n")},
|
||||
}, "profiles"); err != nil {
|
||||
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||
}
|
||||
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||
Assets: assets,
|
||||
EngineOptions: []promptkit.Option{promptkit.WithLLMClient(fake)},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
|
||||
}
|
||||
var out map[string]any
|
||||
response, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
PromptID: "adapter.test",
|
||||
ProfileID: profileID,
|
||||
SessionID: "fallback-profile-test",
|
||||
Inputs: contracts.LLMInputSet{
|
||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||
},
|
||||
}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
||||
}
|
||||
if response.ProfileID != profileID || response.Model != "fallback-model" {
|
||||
t.Fatalf("completion response = %#v, want fallback profile", response)
|
||||
}
|
||||
|
||||
inspector, err := NewPromptKitProfileInspector(PromptKitProfileInspectorConfig{Assets: assets})
|
||||
if err != nil {
|
||||
t.Fatalf("NewPromptKitProfileInspector() error = %v, want nil", err)
|
||||
}
|
||||
inspection, err := inspector.InspectProfile(context.Background(), profileID)
|
||||
if err != nil {
|
||||
t.Fatalf("InspectProfile() error = %v, want nil", err)
|
||||
}
|
||||
if inspection.ProfileID != profileID || inspection.Model != "fallback-model" {
|
||||
t.Fatalf("profile inspection = %#v, want fallback profile", inspection)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientCheckpointFingerprintTracksFallbackProfileAssets(t *testing.T) {
|
||||
fingerprintFor := func(content string) CheckpointFingerprint {
|
||||
t.Helper()
|
||||
assets := newTestPromptKitAssets(t)
|
||||
if err := assets.RegisterFallbackProfileFS(fstest.MapFS{
|
||||
"profiles/fallback.yaml": {Data: []byte(content)},
|
||||
}, "profiles"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
client, err := NewPromptKitClient(PromptKitClientConfig{Assets: assets})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fingerprints, err := client.LLMCheckpointFingerprints()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(fingerprints) != 1 || fingerprints[0].Name != promptKitProfileFingerprintName {
|
||||
t.Fatalf("checkpoint fingerprints = %#v, want profile source identity", fingerprints)
|
||||
}
|
||||
return fingerprints[0]
|
||||
}
|
||||
|
||||
first := fingerprintFor("id: fallback\nendpoint: http://promptkit.test/v1\nmodel: model-one\n")
|
||||
second := fingerprintFor("id: fallback\nendpoint: http://promptkit.test/v1\nmodel: model-two\n")
|
||||
if first == second {
|
||||
t.Fatalf("checkpoint fingerprints = %#v and %#v, want fallback asset change", first, second)
|
||||
}
|
||||
if strings.Contains(first.Value, "model-one") || strings.Contains(first.Value, "fallback.yaml") {
|
||||
t.Fatalf("checkpoint fingerprint leaked fallback source details: %#v", first)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -654,8 +906,13 @@ func TestPromptKitClientTranslatesBackendCapacityExhaustion(t *testing.T) {
|
||||
t.Fatalf("capacity error exposes PromptKit sentinel: %v", capacityErr)
|
||||
}
|
||||
if !strings.Contains(capacityErr.Error(), `run PromptKit prompt "adapter.direct-session"`) ||
|
||||
!strings.Contains(capacityErr.Error(), `backend "limited-backend"`) ||
|
||||
!strings.Contains(capacityErr.Error(), "backend capacity exceeded") {
|
||||
t.Fatalf("capacity error = %q, want prompt context and upstream diagnostic", capacityErr)
|
||||
t.Fatalf("capacity error = %q, want prompt, backend, and upstream diagnostic context", capacityErr)
|
||||
}
|
||||
var upstreamCapacityErr *promptkit.CapacityError
|
||||
if errors.As(capacityErr, &upstreamCapacityErr) {
|
||||
t.Fatalf("capacity error exposes PromptKit capacity type: %v", capacityErr)
|
||||
}
|
||||
if calls := atomic.LoadInt32(&fake.calls); calls != 1 {
|
||||
t.Fatalf("provider calls after capacity rejection = %d, want 1", calls)
|
||||
@@ -890,6 +1147,61 @@ output:
|
||||
return registry
|
||||
}
|
||||
|
||||
type switchingPromptFS struct {
|
||||
mu sync.Mutex
|
||||
files fstest.MapFS
|
||||
pauseNextOpen bool
|
||||
opened chan struct{}
|
||||
resume chan struct{}
|
||||
}
|
||||
|
||||
func (f *switchingPromptFS) Open(name string) (fs.File, error) {
|
||||
f.mu.Lock()
|
||||
file, err := f.files.Open(name)
|
||||
pause := f.pauseNextOpen && name == "snapshot.test.yaml"
|
||||
resume := f.resume
|
||||
if pause {
|
||||
f.pauseNextOpen = false
|
||||
close(f.opened)
|
||||
}
|
||||
f.mu.Unlock()
|
||||
if pause {
|
||||
<-resume
|
||||
}
|
||||
return file, err
|
||||
}
|
||||
|
||||
func (f *switchingPromptFS) ReadDir(name string) ([]fs.DirEntry, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.files.ReadDir(name)
|
||||
}
|
||||
|
||||
func (f *switchingPromptFS) holdNextPromptRead() <-chan struct{} {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.pauseNextOpen = true
|
||||
f.opened = make(chan struct{})
|
||||
f.resume = make(chan struct{})
|
||||
return f.opened
|
||||
}
|
||||
|
||||
func (f *switchingPromptFS) replacePrompt(content []byte) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.files["snapshot.test.yaml"] = &fstest.MapFile{Data: append([]byte(nil), content...)}
|
||||
}
|
||||
|
||||
func (f *switchingPromptFS) resumePromptRead() {
|
||||
f.mu.Lock()
|
||||
resume := f.resume
|
||||
f.resume = nil
|
||||
f.mu.Unlock()
|
||||
if resume != nil {
|
||||
close(resume)
|
||||
}
|
||||
}
|
||||
|
||||
type fakePromptKitLLM struct {
|
||||
content string
|
||||
allowEmpty bool
|
||||
|
||||
@@ -17,27 +17,31 @@ const (
|
||||
promptKitLocalBackendMarker = "notarius:promptkit-local-backend:v1"
|
||||
// The built-in profile catalog is compiled into this pinned PromptKit
|
||||
// release. Update this identity when the dependency is upgraded.
|
||||
promptKitBuiltinProfileCatalogID = "promptkit:v0.3.0:builtin-profiles"
|
||||
promptKitBuiltinProfileCatalogID = "promptkit:v0.5.0:builtin-profiles"
|
||||
)
|
||||
|
||||
func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFingerprint, error) {
|
||||
func promptKitProfileFingerprint(profileDir, profileFile, fallbackProfileDigest string) (CheckpointFingerprint, error) {
|
||||
hasher := sha256.New()
|
||||
writeFingerprintPart(hasher, []byte(promptKitBuiltinProfileCatalogID))
|
||||
writeFingerprintPart(hasher, []byte(strings.TrimSpace(fallbackProfileDigest)))
|
||||
|
||||
switch {
|
||||
case strings.TrimSpace(profileFile) != "":
|
||||
data, err := os.ReadFile(strings.TrimSpace(profileFile))
|
||||
cleanProfileFile := strings.TrimSpace(profileFile)
|
||||
data, err := os.ReadFile(cleanProfileFile)
|
||||
if err != nil {
|
||||
return CheckpointFingerprint{}, fmt.Errorf("read PromptKit profile file for checkpoint identity: %w", err)
|
||||
return CheckpointFingerprint{}, fmt.Errorf("read PromptKit profile file for checkpoint identity")
|
||||
}
|
||||
writeFingerprintPart(hasher, []byte(filepath.ToSlash(filepath.Base(cleanProfileFile))))
|
||||
writeFingerprintPart(hasher, data)
|
||||
case strings.TrimSpace(profileDir) != "":
|
||||
digests, err := promptKitProfileFileDigests(strings.TrimSpace(profileDir))
|
||||
files, err := promptKitProfileFiles(strings.TrimSpace(profileDir))
|
||||
if err != nil {
|
||||
return CheckpointFingerprint{}, err
|
||||
}
|
||||
for _, digest := range digests {
|
||||
writeFingerprintPart(hasher, digest)
|
||||
for _, file := range files {
|
||||
writeFingerprintPart(hasher, []byte(file.path))
|
||||
writeFingerprintPart(hasher, file.digest)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -57,8 +61,13 @@ func promptKitLocalBackendFingerprint(endpoint string) CheckpointFingerprint {
|
||||
}
|
||||
}
|
||||
|
||||
func promptKitProfileFileDigests(root string) ([][]byte, error) {
|
||||
var digests [][]byte
|
||||
type promptKitProfileFile struct {
|
||||
path string
|
||||
digest []byte
|
||||
}
|
||||
|
||||
func promptKitProfileFiles(root string) ([]promptKitProfileFile, error) {
|
||||
var files []promptKitProfileFile
|
||||
err := filepath.WalkDir(root, func(name string, entry fs.DirEntry, walkErr error) error {
|
||||
if walkErr != nil {
|
||||
return walkErr
|
||||
@@ -74,17 +83,24 @@ func promptKitProfileFileDigests(root string) ([][]byte, error) {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
relativePath, err := filepath.Rel(root, name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sum := sha256.Sum256(data)
|
||||
digests = append(digests, append([]byte(nil), sum[:]...))
|
||||
files = append(files, promptKitProfileFile{
|
||||
path: filepath.ToSlash(relativePath),
|
||||
digest: append([]byte(nil), sum[:]...),
|
||||
})
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read PromptKit profile directory for checkpoint identity: %w", err)
|
||||
return nil, fmt.Errorf("read PromptKit profile directory for checkpoint identity")
|
||||
}
|
||||
sort.Slice(digests, func(i, j int) bool {
|
||||
return string(digests[i]) < string(digests[j])
|
||||
sort.Slice(files, func(i, j int) bool {
|
||||
return files[i].path < files[j].path
|
||||
})
|
||||
return digests, nil
|
||||
return files, nil
|
||||
}
|
||||
|
||||
func writeFingerprintPart(hasher interface{ Write([]byte) (int, error) }, value []byte) {
|
||||
|
||||
146
internal/framework/llm/promptkit_profiles.go
Normal file
146
internal/framework/llm/promptkit_profiles.go
Normal file
@@ -0,0 +1,146 @@
|
||||
package llm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
)
|
||||
|
||||
type PromptKitProfileSourceConfig struct {
|
||||
ProfileDir string
|
||||
ProfileFile string
|
||||
LocalBackend *PromptKitLocalBackendConfig
|
||||
}
|
||||
|
||||
type PromptKitProfileInspectorConfig struct {
|
||||
Source PromptKitProfileSourceConfig
|
||||
Assets *AssetRegistry
|
||||
}
|
||||
|
||||
func (c PromptKitProfileSourceConfig) localEndpoint() string {
|
||||
if c.LocalBackend == nil {
|
||||
return ""
|
||||
}
|
||||
return c.LocalBackend.Endpoint
|
||||
}
|
||||
|
||||
type PromptKitProfileInspector struct {
|
||||
engine *promptkit.Engine
|
||||
}
|
||||
|
||||
type PromptKitProfileInspection struct {
|
||||
ProfileID string
|
||||
BackendID string
|
||||
Model string
|
||||
CredentialEnvironment string
|
||||
CredentialRequired bool
|
||||
}
|
||||
|
||||
type PromptKitProfileInspectionError struct {
|
||||
ProfileID string
|
||||
err error
|
||||
}
|
||||
|
||||
func (e *PromptKitProfileInspectionError) Error() string {
|
||||
switch {
|
||||
case errors.Is(e.err, promptkit.ErrProfileNotFound):
|
||||
return fmt.Sprintf("PromptKit profile %q is not configured", e.ProfileID)
|
||||
case errors.Is(e.err, promptkit.ErrInvalidRequest):
|
||||
return fmt.Sprintf("PromptKit profile ID %q is invalid", e.ProfileID)
|
||||
case errors.Is(e.err, promptkit.ErrProfileLoad):
|
||||
return fmt.Sprintf("PromptKit profile %q is invalid or unreadable", e.ProfileID)
|
||||
default:
|
||||
return fmt.Sprintf("PromptKit profile %q could not be inspected", e.ProfileID)
|
||||
}
|
||||
}
|
||||
|
||||
func (e *PromptKitProfileInspectionError) Unwrap() error {
|
||||
return e.err
|
||||
}
|
||||
|
||||
type promptKitProfileConfigurationError struct {
|
||||
err error
|
||||
}
|
||||
|
||||
func (e *promptKitProfileConfigurationError) Error() string {
|
||||
return "PromptKit profile configuration is invalid or unreadable"
|
||||
}
|
||||
|
||||
func (e *promptKitProfileConfigurationError) Unwrap() error {
|
||||
return e.err
|
||||
}
|
||||
|
||||
func NewPromptKitProfileInspector(cfg PromptKitProfileInspectorConfig) (*PromptKitProfileInspector, error) {
|
||||
source, options, err := promptKitProfileSourceEngineOptions(cfg.Source)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if cfg.Assets != nil {
|
||||
fallbackOption, hasFallback, err := cfg.Assets.promptKitFallbackProfileOption()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if hasFallback {
|
||||
options = append(options, fallbackOption)
|
||||
}
|
||||
}
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||
PromptDir: ".",
|
||||
ProfileDir: source.ProfileDir,
|
||||
}, options...)
|
||||
if err != nil {
|
||||
return nil, &promptKitProfileConfigurationError{err: err}
|
||||
}
|
||||
return &PromptKitProfileInspector{engine: engine}, nil
|
||||
}
|
||||
|
||||
func (i *PromptKitProfileInspector) InspectProfile(ctx context.Context, profileID string) (PromptKitProfileInspection, error) {
|
||||
if i == nil || i.engine == nil {
|
||||
return PromptKitProfileInspection{}, fmt.Errorf("PromptKit profile inspector must not be nil")
|
||||
}
|
||||
profileID = strings.TrimSpace(profileID)
|
||||
inspection, err := i.engine.InspectProfile(ctx, profileID)
|
||||
if err != nil {
|
||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||
return PromptKitProfileInspection{}, ctxErr
|
||||
}
|
||||
return PromptKitProfileInspection{}, &PromptKitProfileInspectionError{
|
||||
ProfileID: profileID,
|
||||
err: err,
|
||||
}
|
||||
}
|
||||
return PromptKitProfileInspection{
|
||||
ProfileID: inspection.ProfileID,
|
||||
BackendID: strings.TrimSpace(inspection.EffectiveModelParams.BackendID),
|
||||
Model: strings.TrimSpace(inspection.EffectiveModelParams.Model),
|
||||
CredentialEnvironment: strings.TrimSpace(inspection.EffectiveModelParams.APIKeyEnv),
|
||||
CredentialRequired: inspection.APIKeyRequired,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func promptKitProfileSourceEngineOptions(cfg PromptKitProfileSourceConfig) (PromptKitProfileSourceConfig, []promptkit.Option, error) {
|
||||
source := PromptKitProfileSourceConfig{
|
||||
ProfileDir: strings.TrimSpace(cfg.ProfileDir),
|
||||
ProfileFile: strings.TrimSpace(cfg.ProfileFile),
|
||||
}
|
||||
if source.ProfileDir != "" && source.ProfileFile != "" {
|
||||
return PromptKitProfileSourceConfig{}, nil, fmt.Errorf("PromptKit profile_dir and profile_file are mutually exclusive")
|
||||
}
|
||||
if cfg.LocalBackend != nil {
|
||||
localBackend := *cfg.LocalBackend
|
||||
localBackend.Endpoint = strings.TrimSpace(localBackend.Endpoint)
|
||||
source.LocalBackend = &localBackend
|
||||
}
|
||||
|
||||
var options []promptkit.Option
|
||||
if source.ProfileFile != "" {
|
||||
options = append(options, promptkit.WithProfileFile(source.ProfileFile))
|
||||
}
|
||||
if source.LocalBackend != nil {
|
||||
options = append(options, PromptKitLocalBackendOption(*source.LocalBackend))
|
||||
}
|
||||
return source, options, nil
|
||||
}
|
||||
@@ -24,10 +24,6 @@ func NewChunkerRegistry() *ChunkerRegistry {
|
||||
}
|
||||
}
|
||||
|
||||
func (r *ChunkerRegistry) Register(key string, constructor ChunkerConstructor) error {
|
||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageChunk), constructor)
|
||||
}
|
||||
|
||||
func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerConstructor) error {
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("chunker constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||
|
||||
@@ -29,6 +29,10 @@ type registryBehaviorCase[M any] struct {
|
||||
moduleKey func(M) string
|
||||
}
|
||||
|
||||
func testModuleSpec(key string, stage ModuleStage) ModuleSpec {
|
||||
return ModuleSpec{Key: key, Stage: stage, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||
}
|
||||
|
||||
func TestChunkerRegistryBehavior(t *testing.T) {
|
||||
runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Chunker]{
|
||||
name: "ChunkerRegistry",
|
||||
@@ -39,7 +43,9 @@ func TestChunkerRegistryBehavior(t *testing.T) {
|
||||
return NewChunkerRegistry()
|
||||
},
|
||||
register: func(registry any, key string, constructor func() (contracts.Chunker, error)) error {
|
||||
return registry.(*ChunkerRegistry).Register(key, constructor)
|
||||
return registry.(*ChunkerRegistry).RegisterWithSpec(ModuleSpec{
|
||||
Key: key, Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
}, constructor)
|
||||
},
|
||||
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Chunker, error)) error {
|
||||
return registry.(*ChunkerRegistry).RegisterWithSpec(spec, constructor)
|
||||
@@ -55,7 +61,9 @@ func TestChunkerRegistryBehavior(t *testing.T) {
|
||||
},
|
||||
nilRegister: func(key string, constructor func() (contracts.Chunker, error)) error {
|
||||
var registry *ChunkerRegistry
|
||||
return registry.Register(key, constructor)
|
||||
return registry.RegisterWithSpec(ModuleSpec{
|
||||
Key: key, Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
}, constructor)
|
||||
},
|
||||
nilBuild: func(key string) (contracts.Chunker, error) {
|
||||
var registry *ChunkerRegistry
|
||||
@@ -101,10 +109,11 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
||||
t.Run(testCase.name+"/metadata registration and lookup", func(t *testing.T) {
|
||||
registry := testCase.newRegistry()
|
||||
spec := ModuleSpec{
|
||||
Key: " " + testCase.key + " ",
|
||||
Stage: testCase.stage,
|
||||
Provides: []string{" beta ", "alpha", "", "beta"},
|
||||
Requires: []string{" source ", "source", ""},
|
||||
Key: " " + testCase.key + " ",
|
||||
Stage: testCase.stage,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Provides: []string{" beta ", "alpha", "", "beta"},
|
||||
Requires: []string{" source ", "source", ""},
|
||||
}
|
||||
if err := testCase.registerWithSpec(registry, spec, testCase.constructor(testCase.key)); err != nil {
|
||||
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
||||
@@ -115,10 +124,11 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
||||
t.Fatal("Spec() ok = false, want true")
|
||||
}
|
||||
want := ModuleSpec{
|
||||
Key: testCase.key,
|
||||
Stage: testCase.stage,
|
||||
Provides: []string{"alpha", "beta"},
|
||||
Requires: []string{"source"},
|
||||
Key: testCase.key,
|
||||
Stage: testCase.stage,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Provides: []string{"alpha", "beta"},
|
||||
Requires: []string{"source"},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||
@@ -134,7 +144,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
||||
}
|
||||
})
|
||||
|
||||
t.Run(testCase.name+"/default spec from register", func(t *testing.T) {
|
||||
t.Run(testCase.name+"/minimal explicit spec registration", func(t *testing.T) {
|
||||
registry := testCase.newRegistry()
|
||||
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
@@ -144,7 +154,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
||||
if !ok {
|
||||
t.Fatal("Spec() ok = false, want true")
|
||||
}
|
||||
want := ModuleSpec{Key: testCase.key, Stage: testCase.stage}
|
||||
want := ModuleSpec{Key: testCase.key, Stage: testCase.stage, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||
if !reflect.DeepEqual(spec, want) {
|
||||
t.Fatalf("Spec() = %#v, want %#v", spec, want)
|
||||
}
|
||||
@@ -152,7 +162,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
||||
|
||||
t.Run(testCase.name+"/wrong stage rejection", func(t *testing.T) {
|
||||
registry := testCase.newRegistry()
|
||||
err := testCase.registerWithSpec(registry, ModuleSpec{Key: testCase.key, Stage: testCase.wrongStage}, testCase.constructor(testCase.key))
|
||||
err := testCase.registerWithSpec(registry, ModuleSpec{Key: testCase.key, Stage: testCase.wrongStage, ExecutionClass: contracts.ExecutionClassDeterministic}, testCase.constructor(testCase.key))
|
||||
if err == nil {
|
||||
t.Fatal("RegisterWithSpec() error = nil, want error")
|
||||
}
|
||||
@@ -161,6 +171,22 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
||||
}
|
||||
})
|
||||
|
||||
t.Run(testCase.name+"/execution class rejection", func(t *testing.T) {
|
||||
for _, spec := range []ModuleSpec{
|
||||
{Key: testCase.key, Stage: testCase.stage},
|
||||
{Key: testCase.key, Stage: testCase.stage, ExecutionClass: contracts.ExecutionClass("unsupported")},
|
||||
} {
|
||||
registry := testCase.newRegistry()
|
||||
err := testCase.registerWithSpec(registry, spec, testCase.constructor(testCase.key))
|
||||
if err == nil {
|
||||
t.Fatal("RegisterWithSpec() error = nil, want execution class error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "execution class") {
|
||||
t.Fatalf("RegisterWithSpec() error = %q, want execution class error", err.Error())
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
t.Run(testCase.name+"/key trimming", func(t *testing.T) {
|
||||
registry := testCase.newRegistry()
|
||||
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
|
||||
|
||||
@@ -108,7 +108,7 @@ func registerTestEvidenceOutput(t *testing.T, registries *Registries, policy Evi
|
||||
func registerTestEvidenceOutputWithProfileValidation(t *testing.T, registries *Registries, policy EvidenceContextPolicy, validateProfile OutputProfileOptionValidator) {
|
||||
t.Helper()
|
||||
registry := NewOutputEncoderRegistry()
|
||||
if err := registry.RegisterBuilderWithProfileValidation(defaultModuleSpec("output", StageOutput), func(options map[string]any) error {
|
||||
if err := registry.RegisterBuilderWithProfileValidation(testModuleSpec("output", StageOutput), func(options map[string]any) error {
|
||||
return RejectUnknownOptions(options, "known")
|
||||
}, validateProfile, func(BuildRequest) (contracts.OutputEncoder, error) {
|
||||
return testEvidenceOutput{policy: cloneEvidenceContextPolicy(policy)}, nil
|
||||
|
||||
@@ -24,10 +24,6 @@ func NewInputAdapterRegistry() *InputAdapterRegistry {
|
||||
}
|
||||
}
|
||||
|
||||
func (r *InputAdapterRegistry) Register(key string, constructor InputAdapterConstructor) error {
|
||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageInput), constructor)
|
||||
}
|
||||
|
||||
func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor InputAdapterConstructor) error {
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("input adapter constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||
|
||||
@@ -11,10 +11,14 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func registerTestInput(registry *InputAdapterRegistry, key string, constructor InputAdapterConstructor) error {
|
||||
return registry.RegisterWithSpec(testModuleSpec(key, StageInput), constructor)
|
||||
}
|
||||
|
||||
func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) {
|
||||
registry := NewInputAdapterRegistry()
|
||||
|
||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
@@ -30,7 +34,7 @@ func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) {
|
||||
func TestInputAdapterRegistryRegisterAndBuildTrimKeys(t *testing.T) {
|
||||
registry := NewInputAdapterRegistry()
|
||||
|
||||
if err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||
if err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
@@ -46,10 +50,11 @@ func TestInputAdapterRegistryRegisterAndBuildTrimKeys(t *testing.T) {
|
||||
func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
||||
registry := NewInputAdapterRegistry()
|
||||
spec := ModuleSpec{
|
||||
Key: " generic-input ",
|
||||
Stage: StageInput,
|
||||
Provides: []string{" parsed-source ", "source-document", "parsed-source", ""},
|
||||
Requires: []string{" raw-bytes ", "raw-bytes", ""},
|
||||
Key: " generic-input ",
|
||||
Stage: StageInput,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Provides: []string{" parsed-source ", "source-document", "parsed-source", ""},
|
||||
Requires: []string{" raw-bytes ", "raw-bytes", ""},
|
||||
}
|
||||
|
||||
if err := registry.RegisterWithSpec(spec, fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||
@@ -61,10 +66,11 @@ func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
||||
t.Fatal("Spec() ok = false, want true")
|
||||
}
|
||||
want := ModuleSpec{
|
||||
Key: "generic-input",
|
||||
Stage: StageInput,
|
||||
Provides: []string{"parsed-source", "source-document"},
|
||||
Requires: []string{"raw-bytes"},
|
||||
Key: "generic-input",
|
||||
Stage: StageInput,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Provides: []string{"parsed-source", "source-document"},
|
||||
Requires: []string{"raw-bytes"},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||
@@ -80,10 +86,10 @@ func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
|
||||
func TestInputAdapterRegistryRegisterWithSpecStoresMinimalMetadata(t *testing.T) {
|
||||
registry := NewInputAdapterRegistry()
|
||||
|
||||
if err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||
if err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
@@ -91,7 +97,7 @@ func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
|
||||
if !ok {
|
||||
t.Fatal("Spec() ok = false, want true")
|
||||
}
|
||||
want := ModuleSpec{Key: "generic-input", Stage: StageInput}
|
||||
want := ModuleSpec{Key: "generic-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||
}
|
||||
@@ -100,7 +106,7 @@ func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
|
||||
func TestInputAdapterRegistryRegisterWithSpecRejectsWrongStage(t *testing.T) {
|
||||
registry := NewInputAdapterRegistry()
|
||||
|
||||
err := registry.RegisterWithSpec(ModuleSpec{Key: "generic-input", Stage: StageExtract}, fakeInputAdapterConstructor("generic-input"))
|
||||
err := registry.RegisterWithSpec(ModuleSpec{Key: "generic-input", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic}, fakeInputAdapterConstructor("generic-input"))
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("RegisterWithSpec() error = nil, want error")
|
||||
@@ -121,7 +127,7 @@ func TestInputAdapterRegistrySpecRejectsUnknownKey(t *testing.T) {
|
||||
func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) {
|
||||
registry := NewInputAdapterRegistry()
|
||||
|
||||
err := registry.Register(" \t", fakeInputAdapterConstructor("generic-input"))
|
||||
err := registerTestInput(registry, " \t", fakeInputAdapterConstructor("generic-input"))
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("Register() error = nil, want error")
|
||||
@@ -133,11 +139,11 @@ func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) {
|
||||
|
||||
func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) {
|
||||
registry := NewInputAdapterRegistry()
|
||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input"))
|
||||
err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input"))
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("Register() error = nil, want error")
|
||||
@@ -150,7 +156,7 @@ func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) {
|
||||
func TestInputAdapterRegistryRegisterRejectsNilConstructor(t *testing.T) {
|
||||
registry := NewInputAdapterRegistry()
|
||||
|
||||
err := registry.Register("generic-input", nil)
|
||||
err := registerTestInput(registry, "generic-input", nil)
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("Register() error = nil, want error")
|
||||
@@ -176,7 +182,7 @@ func TestInputAdapterRegistryBuildRejectsUnknownKey(t *testing.T) {
|
||||
func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) {
|
||||
registry := NewInputAdapterRegistry()
|
||||
constructorErr := errors.New("constructor failed")
|
||||
if err := registry.Register("generic-input", func() (contracts.InputAdapter, error) {
|
||||
if err := registerTestInput(registry, "generic-input", func() (contracts.InputAdapter, error) {
|
||||
return nil, constructorErr
|
||||
}); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
@@ -197,7 +203,7 @@ func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) {
|
||||
|
||||
func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) {
|
||||
registry := NewInputAdapterRegistry()
|
||||
if err := registry.Register("generic-input", func() (contracts.InputAdapter, error) {
|
||||
if err := registerTestInput(registry, "generic-input", func() (contracts.InputAdapter, error) {
|
||||
return nil, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
@@ -215,7 +221,7 @@ func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) {
|
||||
|
||||
func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) {
|
||||
registry := NewInputAdapterRegistry()
|
||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("other-input")); err != nil {
|
||||
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("other-input")); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
@@ -232,7 +238,7 @@ func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) {
|
||||
func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
|
||||
registry := NewInputAdapterRegistry()
|
||||
for _, key := range []string{"zeta", "alpha", "middle"} {
|
||||
if err := registry.Register(key, fakeInputAdapterConstructor(key)); err != nil {
|
||||
if err := registerTestInput(registry, key, fakeInputAdapterConstructor(key)); err != nil {
|
||||
t.Fatalf("Register(%q) error = %v, want nil", key, err)
|
||||
}
|
||||
}
|
||||
@@ -253,7 +259,7 @@ func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
|
||||
func TestInputAdapterRegistryNilRegistryBehavior(t *testing.T) {
|
||||
var registry *InputAdapterRegistry
|
||||
|
||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err == nil {
|
||||
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err == nil {
|
||||
t.Fatal("Register() error = nil, want error")
|
||||
}
|
||||
if _, err := registry.Build("generic-input"); err == nil {
|
||||
|
||||
@@ -23,23 +23,19 @@ const (
|
||||
type ModuleSpec struct {
|
||||
Key string
|
||||
Stage ModuleStage
|
||||
ExecutionClass contracts.ExecutionClass
|
||||
ArtifactKind contracts.ArtifactKind
|
||||
Provides []string
|
||||
Requires []string
|
||||
ReferenceSlots []contracts.ReferenceSlot
|
||||
}
|
||||
|
||||
func defaultModuleSpec(key string, stage ModuleStage) ModuleSpec {
|
||||
return ModuleSpec{
|
||||
Key: key,
|
||||
Stage: stage,
|
||||
}
|
||||
}
|
||||
|
||||
func normalizeModuleSpec(spec ModuleSpec) ModuleSpec {
|
||||
executionClass := contracts.ExecutionClass(strings.TrimSpace(string(spec.ExecutionClass)))
|
||||
return ModuleSpec{
|
||||
Key: strings.TrimSpace(spec.Key),
|
||||
Stage: spec.Stage,
|
||||
ExecutionClass: executionClass,
|
||||
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
|
||||
Provides: normalizeCapabilities(spec.Provides),
|
||||
Requires: normalizeCapabilities(spec.Requires),
|
||||
@@ -76,6 +72,7 @@ func cloneModuleSpec(spec ModuleSpec) ModuleSpec {
|
||||
return ModuleSpec{
|
||||
Key: spec.Key,
|
||||
Stage: spec.Stage,
|
||||
ExecutionClass: spec.ExecutionClass,
|
||||
ArtifactKind: spec.ArtifactKind,
|
||||
Provides: append([]string(nil), spec.Provides...),
|
||||
Requires: append([]string(nil), spec.Requires...),
|
||||
@@ -90,6 +87,12 @@ func validateModuleSpec(kind string, expectedStage ModuleStage, spec ModuleSpec)
|
||||
if spec.Stage != expectedStage {
|
||||
return fmt.Errorf("%s %q must use %q stage, got %q", kind, spec.Key, expectedStage, spec.Stage)
|
||||
}
|
||||
if spec.ExecutionClass == "" {
|
||||
return fmt.Errorf("%s %q execution class must not be empty", kind, spec.Key)
|
||||
}
|
||||
if spec.ExecutionClass != contracts.ExecutionClassDeterministic && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
|
||||
return fmt.Errorf("%s %q has unsupported execution class %q", kind, spec.Key, spec.ExecutionClass)
|
||||
}
|
||||
if spec.ArtifactKind != "" && spec.Stage != StageExtract && spec.Stage != StageMerge && spec.Stage != StageNormalize {
|
||||
return fmt.Errorf("%s %q must not declare an artifact kind", kind, spec.Key)
|
||||
}
|
||||
|
||||
@@ -1,12 +1,45 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func TestValidateModuleSpecRequiresSupportedExecutionClass(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
class contracts.ExecutionClass
|
||||
want string
|
||||
}{
|
||||
{name: "missing", want: "execution class"},
|
||||
{name: "unsupported", class: "remote", want: "unsupported"},
|
||||
{name: "deterministic", class: contracts.ExecutionClassDeterministic},
|
||||
{name: "llm backed", class: contracts.ExecutionClassLLMBacked},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
spec := normalizeModuleSpec(ModuleSpec{Key: "module", Stage: StageChunk, ExecutionClass: test.class})
|
||||
err := validateModuleSpec("chunker", StageChunk, spec)
|
||||
if test.want == "" && err != nil {
|
||||
t.Fatalf("validateModuleSpec() error = %v, want nil", err)
|
||||
}
|
||||
if test.want != "" && (err == nil || !strings.Contains(err.Error(), test.want)) {
|
||||
t.Fatalf("validateModuleSpec() error = %v, want %q", err, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCloneModuleSpecPreservesExecutionClass(t *testing.T) {
|
||||
spec := normalizeModuleSpec(ModuleSpec{Key: " module ", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked})
|
||||
cloned := cloneModuleSpec(spec)
|
||||
if !reflect.DeepEqual(cloned, spec) {
|
||||
t.Fatalf("cloneModuleSpec() = %#v, want %#v", cloned, spec)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
@@ -22,8 +55,9 @@ func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
spec := normalizeModuleSpec(ModuleSpec{
|
||||
Key: "module",
|
||||
Stage: test.stage,
|
||||
Key: "module",
|
||||
Stage: test.stage,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{Name: "roster", Description: "Character roster", MaxBytes: 1024},
|
||||
},
|
||||
@@ -50,8 +84,9 @@ func TestValidateModuleSpecRejectsReferenceSlotsForIneligibleStages(t *testing.T
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
spec := normalizeModuleSpec(ModuleSpec{
|
||||
Key: "module",
|
||||
Stage: test.stage,
|
||||
Key: "module",
|
||||
Stage: test.stage,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{Name: "roster"},
|
||||
},
|
||||
@@ -109,6 +144,7 @@ func TestValidateModuleSpecRejectsInvalidReferenceSlotsForEligibleStages(t *test
|
||||
spec := normalizeModuleSpec(ModuleSpec{
|
||||
Key: "module",
|
||||
Stage: stage.stage,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
ReferenceSlots: invalid.slots,
|
||||
})
|
||||
err := validateModuleSpec(stage.kind, stage.stage, spec)
|
||||
|
||||
@@ -26,7 +26,7 @@ func TestNormalizerRegistryErasureClonesRetryDirective(t *testing.T) {
|
||||
FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "omitted", Message: "fallback warning"}},
|
||||
}
|
||||
registry := NewNormalizerRegistry()
|
||||
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
||||
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
||||
return retryingNotesNormalizer{warnings: warnings, retry: retry}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||
|
||||
@@ -32,10 +32,6 @@ func NewOutputEncoderRegistry() *OutputEncoderRegistry {
|
||||
}
|
||||
}
|
||||
|
||||
func (r *OutputEncoderRegistry) Register(key string, constructor OutputEncoderConstructor) error {
|
||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageOutput), constructor)
|
||||
}
|
||||
|
||||
func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor OutputEncoderConstructor) error {
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("output encoder constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||
|
||||
@@ -17,7 +17,9 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
|
||||
return NewOutputEncoderRegistry()
|
||||
},
|
||||
register: func(registry any, key string, constructor func() (contracts.OutputEncoder, error)) error {
|
||||
return registry.(*OutputEncoderRegistry).Register(key, constructor)
|
||||
return registry.(*OutputEncoderRegistry).RegisterWithSpec(ModuleSpec{
|
||||
Key: key, Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
}, constructor)
|
||||
},
|
||||
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.OutputEncoder, error)) error {
|
||||
return registry.(*OutputEncoderRegistry).RegisterWithSpec(spec, constructor)
|
||||
@@ -33,7 +35,9 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
|
||||
},
|
||||
nilRegister: func(key string, constructor func() (contracts.OutputEncoder, error)) error {
|
||||
var registry *OutputEncoderRegistry
|
||||
return registry.Register(key, constructor)
|
||||
return registry.RegisterWithSpec(ModuleSpec{
|
||||
Key: key, Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
}, constructor)
|
||||
},
|
||||
nilBuild: func(key string) (contracts.OutputEncoder, error) {
|
||||
var registry *OutputEncoderRegistry
|
||||
@@ -60,7 +64,7 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
|
||||
|
||||
func TestOutputProfileValidationReceivesOwnedOptionsAndLaneIDs(t *testing.T) {
|
||||
registry := NewOutputEncoderRegistry()
|
||||
if err := registry.RegisterBuilderWithProfileValidation(defaultModuleSpec("profile-output", StageOutput), func(options map[string]any) error {
|
||||
if err := registry.RegisterBuilderWithProfileValidation(testModuleSpec("profile-output", StageOutput), func(options map[string]any) error {
|
||||
options["nested"].(map[string]any)["value"] = "changed"
|
||||
return nil
|
||||
}, func(context OutputProfileOptionContext, options map[string]any) error {
|
||||
|
||||
@@ -480,19 +480,19 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
||||
if err := RegisterArtifactCodec(registries.ArtifactCodecs, notesCodec()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := registries.Inputs.RegisterBuilderWithSpec(defaultModuleSpec("input", StageInput), strict, func(request BuildRequest) (contracts.InputAdapter, error) {
|
||||
if err := registries.Inputs.RegisterBuilderWithSpec(testModuleSpec("input", StageInput), strict, func(request BuildRequest) (contracts.InputAdapter, error) {
|
||||
record("input", &request)
|
||||
return input, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := registries.Chunkers.RegisterBuilderWithSpec(defaultModuleSpec("chunk", StageChunk), strict, func(request BuildRequest) (contracts.Chunker, error) {
|
||||
if err := registries.Chunkers.RegisterBuilderWithSpec(testModuleSpec("chunk", StageChunk), strict, func(request BuildRequest) (contracts.Chunker, error) {
|
||||
record("chunk", &request)
|
||||
return &typedTestChunker{key: "chunk"}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
extractSpec := defaultModuleSpec("extract", StageExtract)
|
||||
extractSpec := testModuleSpec("extract", StageExtract)
|
||||
extractSpec.ArtifactKind = "test/notes"
|
||||
if err := RegisterExtractorBuilder(registries.Extractors, extractSpec, strict, func(request BuildRequest) (contracts.Extractor[codecNotes], error) {
|
||||
record("extract", &request)
|
||||
@@ -503,7 +503,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mergeSpec := defaultModuleSpec("merge", StageMerge)
|
||||
mergeSpec := testModuleSpec("merge", StageMerge)
|
||||
mergeSpec.ArtifactKind = "test/notes"
|
||||
if err := RegisterMergerBuilder(registries.Mergers, mergeSpec, strict, func(request BuildRequest) (contracts.Merger[codecNotes], error) {
|
||||
record("merge", &request)
|
||||
@@ -511,7 +511,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
normalizeSpec := defaultModuleSpec("normalize", StageNormalize)
|
||||
normalizeSpec := testModuleSpec("normalize", StageNormalize)
|
||||
normalizeSpec.ArtifactKind = "test/notes"
|
||||
if err := RegisterNormalizerBuilder(registries.Normalizers, normalizeSpec, strict, func(request BuildRequest) (contracts.Normalizer[codecNotes], error) {
|
||||
record("normalize", &request)
|
||||
@@ -532,7 +532,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := registries.Outputs.RegisterBuilderWithSpec(defaultModuleSpec("output", StageOutput), strict, func(request BuildRequest) (contracts.OutputEncoder, error) {
|
||||
if err := registries.Outputs.RegisterBuilderWithSpec(testModuleSpec("output", StageOutput), strict, func(request BuildRequest) (contracts.OutputEncoder, error) {
|
||||
record("output", &request)
|
||||
if failure.output != nil {
|
||||
return nil, failure.output
|
||||
|
||||
@@ -117,6 +117,7 @@ type PipelineStepProfile struct {
|
||||
|
||||
type PipelineProfile struct {
|
||||
ID string `json:"id"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
Input ModuleBinding `json:"input"`
|
||||
Chunk ModuleBinding `json:"chunk,omitempty"`
|
||||
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
|
||||
@@ -127,6 +128,7 @@ type PipelineProfile struct {
|
||||
|
||||
type ResolveOptions struct {
|
||||
Only []string
|
||||
LLMProfileOverride string
|
||||
ReferenceOverrides []ReferenceBinding
|
||||
ReferenceUnbinds []ReferenceUnbind
|
||||
}
|
||||
@@ -156,20 +158,23 @@ type ResolvedReferenceTarget struct {
|
||||
}
|
||||
|
||||
type ResolvedArtifactLane struct {
|
||||
StepID string
|
||||
ID string
|
||||
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
|
||||
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
|
||||
ArtifactSchemaName string `json:"artifact_schema_name,omitempty"`
|
||||
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
|
||||
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
|
||||
Extract ModuleBinding
|
||||
Merge ModuleBinding
|
||||
Normalize ModuleBinding
|
||||
Validators []ModuleBinding
|
||||
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
|
||||
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
|
||||
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
|
||||
StepID string
|
||||
ID string
|
||||
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
|
||||
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
|
||||
ArtifactSchemaName string `json:"artifact_schema_name,omitempty"`
|
||||
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
|
||||
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
|
||||
Extract ModuleBinding
|
||||
ExtractExecutionClass contracts.ExecutionClass `json:"extract_execution_class"`
|
||||
Merge ModuleBinding
|
||||
MergeExecutionClass contracts.ExecutionClass `json:"merge_execution_class"`
|
||||
Normalize ModuleBinding
|
||||
NormalizeExecutionClass contracts.ExecutionClass `json:"normalize_execution_class"`
|
||||
Validators []ModuleBinding
|
||||
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
|
||||
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
|
||||
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
|
||||
}
|
||||
|
||||
type ResolvedPipelineStep struct {
|
||||
@@ -192,14 +197,17 @@ type ResolvedValidator struct {
|
||||
}
|
||||
|
||||
type ResolvedPipeline struct {
|
||||
ID string
|
||||
Digest string
|
||||
Input ModuleBinding
|
||||
Chunk ModuleBinding
|
||||
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
|
||||
Steps []ResolvedPipelineStep
|
||||
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
|
||||
Output ModuleBinding
|
||||
ID string
|
||||
Digest string
|
||||
Input ModuleBinding
|
||||
InputExecutionClass contracts.ExecutionClass `json:"input_execution_class"`
|
||||
Chunk ModuleBinding
|
||||
ChunkExecutionClass contracts.ExecutionClass `json:"chunk_execution_class"`
|
||||
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
|
||||
Steps []ResolvedPipelineStep
|
||||
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
|
||||
Output ModuleBinding
|
||||
OutputExecutionClass contracts.ExecutionClass `json:"output_execution_class"`
|
||||
}
|
||||
|
||||
// AllArtifactLanes returns lanes in deterministic step order for read-only
|
||||
@@ -225,6 +233,46 @@ type ModuleCatalog struct {
|
||||
Outputs *OutputEncoderRegistry
|
||||
}
|
||||
|
||||
// ExecutionClass returns the registered execution class for a module selected
|
||||
// by stage and key without constructing the module.
|
||||
func (catalog ModuleCatalog) ExecutionClass(stage ModuleStage, key string) (contracts.ExecutionClass, bool) {
|
||||
var executionClass contracts.ExecutionClass
|
||||
var ok bool
|
||||
|
||||
switch stage {
|
||||
case StageInput:
|
||||
var spec ModuleSpec
|
||||
spec, ok = catalog.Inputs.Spec(key)
|
||||
executionClass = spec.ExecutionClass
|
||||
case StageChunk:
|
||||
var spec ModuleSpec
|
||||
spec, ok = catalog.Chunkers.Spec(key)
|
||||
executionClass = spec.ExecutionClass
|
||||
case StageExtract:
|
||||
var spec ModuleSpec
|
||||
spec, ok = catalog.Extractors.Spec(key)
|
||||
executionClass = spec.ExecutionClass
|
||||
case StageMerge:
|
||||
var spec ModuleSpec
|
||||
spec, ok = catalog.Mergers.Spec(key)
|
||||
executionClass = spec.ExecutionClass
|
||||
case StageNormalize:
|
||||
var spec ModuleSpec
|
||||
spec, ok = catalog.Normalizers.Spec(key)
|
||||
executionClass = spec.ExecutionClass
|
||||
case StageValidate:
|
||||
var spec ValidatorSpec
|
||||
spec, ok = catalog.Validators.Spec(key)
|
||||
executionClass = spec.ExecutionClass
|
||||
case StageOutput:
|
||||
var spec ModuleSpec
|
||||
spec, ok = catalog.Outputs.Spec(key)
|
||||
executionClass = spec.ExecutionClass
|
||||
}
|
||||
|
||||
return executionClass, ok
|
||||
}
|
||||
|
||||
func Binding(module string) ModuleBinding {
|
||||
return ModuleBinding{Module: strings.TrimSpace(module)}
|
||||
}
|
||||
@@ -316,11 +364,13 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
||||
return ResolvedPipeline{}, err
|
||||
}
|
||||
resolved := ResolvedPipeline{
|
||||
ID: pipelineID,
|
||||
Input: input,
|
||||
Chunk: chunk,
|
||||
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
|
||||
Output: resolveBinding(profile.Output, DefaultOutputModule),
|
||||
ID: pipelineID,
|
||||
Input: input,
|
||||
InputExecutionClass: inputModuleSpec.ExecutionClass,
|
||||
Chunk: chunk,
|
||||
ChunkExecutionClass: chunkSpec.ExecutionClass,
|
||||
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
|
||||
Output: resolveBinding(profile.Output, DefaultOutputModule),
|
||||
}
|
||||
chunkValidatorChain, err := resolveValidatorChain(pipelineID, "", StageChunk, chunk.Module, chunk.Validators, "", nil, catalog)
|
||||
if err != nil {
|
||||
@@ -386,6 +436,10 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
||||
if missing, ok := outputCapabilities.missing(outputSpec.Requires); ok {
|
||||
return ResolvedPipeline{}, capabilityError(pipelineID, "", StageOutput, resolved.Output.Module, missing)
|
||||
}
|
||||
resolved.OutputExecutionClass = outputSpec.ExecutionClass
|
||||
if err := applyEffectiveLLMProfiles(&resolved, profile.LLMProfile, options.LLMProfileOverride); err != nil {
|
||||
return ResolvedPipeline{}, err
|
||||
}
|
||||
if err := validateResolvedOptions(resolved, catalog, configuredLaneIDs); err != nil {
|
||||
return ResolvedPipeline{}, err
|
||||
}
|
||||
@@ -475,6 +529,7 @@ func resolveArtifactLane(
|
||||
}
|
||||
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
|
||||
lane.ExtractReferences.StepID = strings.TrimSpace(stepID)
|
||||
lane.ExtractExecutionClass = extractSpec.ExecutionClass
|
||||
capabilities.add(extractSpec.Provides...)
|
||||
|
||||
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
|
||||
@@ -499,6 +554,7 @@ func resolveArtifactLane(
|
||||
}
|
||||
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
|
||||
lane.MergeReferences.StepID = strings.TrimSpace(stepID)
|
||||
lane.MergeExecutionClass = mergeSpec.ExecutionClass
|
||||
capabilities.add(mergeSpec.Provides...)
|
||||
|
||||
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
|
||||
@@ -523,6 +579,7 @@ func resolveArtifactLane(
|
||||
}
|
||||
lane.NormalizeReferences = referenceTarget(StageNormalize, laneID, lane.Normalize.Module, normalizeReferences)
|
||||
lane.NormalizeReferences.StepID = strings.TrimSpace(stepID)
|
||||
lane.NormalizeExecutionClass = normalizeSpec.ExecutionClass
|
||||
capabilities.add(normalizeSpec.Provides...)
|
||||
|
||||
if len(lane.Validators) > 0 {
|
||||
@@ -784,9 +841,6 @@ func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage,
|
||||
if err != nil {
|
||||
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q: %w", pipelineID, stage, chain.ModuleKey, err)
|
||||
}
|
||||
if strings.TrimSpace(validator.LLMProfile) != "" && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
|
||||
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q assigns llm_profile to deterministic validator %q", pipelineID, stage, chain.ModuleKey, validator.Module)
|
||||
}
|
||||
chain.Validators = append(chain.Validators, ResolvedValidator{
|
||||
Binding: cloneModuleBinding(validator),
|
||||
ExecutionClass: spec.ExecutionClass,
|
||||
@@ -1198,6 +1252,66 @@ func resolveBinding(binding ModuleBinding, defaultModule string) ModuleBinding {
|
||||
}
|
||||
}
|
||||
|
||||
func applyEffectiveLLMProfiles(resolved *ResolvedPipeline, pipelineProfile, overrideProfile string) error {
|
||||
pipelineProfile = strings.TrimSpace(pipelineProfile)
|
||||
overrideProfile = strings.TrimSpace(overrideProfile)
|
||||
|
||||
apply := func(stage ModuleStage, laneID, module string, binding *ModuleBinding, executionClass contracts.ExecutionClass, kind string) error {
|
||||
binding.LLMProfile = strings.TrimSpace(binding.LLMProfile)
|
||||
if executionClass != contracts.ExecutionClassLLMBacked {
|
||||
if binding.LLMProfile != "" {
|
||||
if laneID == "" {
|
||||
return fmt.Errorf("pipeline %q %s %q assigns llm_profile to deterministic %s %q", resolved.ID, stage, module, kind, binding.Module)
|
||||
}
|
||||
return fmt.Errorf("pipeline %q lane %q %s %q assigns llm_profile to deterministic %s %q", resolved.ID, laneID, stage, module, kind, binding.Module)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if overrideProfile != "" {
|
||||
binding.LLMProfile = overrideProfile
|
||||
return nil
|
||||
}
|
||||
if binding.LLMProfile == "" {
|
||||
binding.LLMProfile = pipelineProfile
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
if err := apply(StageInput, "", resolved.Input.Module, &resolved.Input, resolved.InputExecutionClass, "module"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := apply(StageChunk, "", resolved.Chunk.Module, &resolved.Chunk, resolved.ChunkExecutionClass, "module"); err != nil {
|
||||
return err
|
||||
}
|
||||
for stepIndex := range resolved.Steps {
|
||||
for laneIndex := range resolved.Steps[stepIndex].ArtifactLanes {
|
||||
lane := &resolved.Steps[stepIndex].ArtifactLanes[laneIndex]
|
||||
if err := apply(StageExtract, lane.ID, lane.Extract.Module, &lane.Extract, lane.ExtractExecutionClass, "module"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := apply(StageMerge, lane.ID, lane.Merge.Module, &lane.Merge, lane.MergeExecutionClass, "module"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := apply(StageNormalize, lane.ID, lane.Normalize.Module, &lane.Normalize, lane.NormalizeExecutionClass, "module"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := apply(StageOutput, "", resolved.Output.Module, &resolved.Output, resolved.OutputExecutionClass, "module"); err != nil {
|
||||
return err
|
||||
}
|
||||
for chainIndex := range resolved.ValidatorChains {
|
||||
chain := &resolved.ValidatorChains[chainIndex]
|
||||
for validatorIndex := range chain.Validators {
|
||||
validator := &chain.Validators[validatorIndex]
|
||||
if err := apply(chain.Stage, chain.LaneID, chain.ModuleKey, &validator.Binding, validator.ExecutionClass, "validator"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func resolveBindings(bindings []ModuleBinding, defaultModule string) []ModuleBinding {
|
||||
if len(bindings) == 0 {
|
||||
return nil
|
||||
@@ -1307,21 +1421,27 @@ func selectedArtifactLanes(pipelineID string, artifacts map[string]ArtifactLaneP
|
||||
|
||||
func resolvedPipelineDigest(resolved ResolvedPipeline) (string, error) {
|
||||
withoutDigest := struct {
|
||||
ID string
|
||||
Input ModuleBinding
|
||||
Chunk ModuleBinding
|
||||
ChunkReferences ResolvedReferenceTarget
|
||||
Steps []ResolvedPipelineStep
|
||||
ValidatorChains []ResolvedValidatorChain
|
||||
Output ModuleBinding
|
||||
ID string
|
||||
Input ModuleBinding
|
||||
InputExecutionClass contracts.ExecutionClass
|
||||
Chunk ModuleBinding
|
||||
ChunkExecutionClass contracts.ExecutionClass
|
||||
ChunkReferences ResolvedReferenceTarget
|
||||
Steps []ResolvedPipelineStep
|
||||
ValidatorChains []ResolvedValidatorChain
|
||||
Output ModuleBinding
|
||||
OutputExecutionClass contracts.ExecutionClass
|
||||
}{
|
||||
ID: resolved.ID,
|
||||
Input: resolved.Input,
|
||||
Chunk: resolved.Chunk,
|
||||
ChunkReferences: resolved.ChunkReferences,
|
||||
Steps: resolved.Steps,
|
||||
ValidatorChains: resolved.ValidatorChains,
|
||||
Output: resolved.Output,
|
||||
ID: resolved.ID,
|
||||
Input: resolved.Input,
|
||||
InputExecutionClass: resolved.InputExecutionClass,
|
||||
Chunk: resolved.Chunk,
|
||||
ChunkExecutionClass: resolved.ChunkExecutionClass,
|
||||
ChunkReferences: resolved.ChunkReferences,
|
||||
Steps: resolved.Steps,
|
||||
ValidatorChains: resolved.ValidatorChains,
|
||||
Output: resolved.Output,
|
||||
OutputExecutionClass: resolved.OutputExecutionClass,
|
||||
}
|
||||
encoded, err := json.Marshal(withoutDigest)
|
||||
if err != nil {
|
||||
|
||||
@@ -15,22 +15,22 @@ import (
|
||||
func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
||||
catalog := newProfileCatalog(t)
|
||||
registerProfileSpecs(t, catalog,
|
||||
ModuleSpec{Key: "window", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||
ModuleSpec{Key: "record-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||
ModuleSpec{Key: "dedupe", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||
ModuleSpec{Key: "canonical", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||
ModuleSpec{Key: "ndjson", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||
ModuleSpec{Key: "window", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||
ModuleSpec{Key: "record-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||
ModuleSpec{Key: "dedupe", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||
ModuleSpec{Key: "canonical", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||
ModuleSpec{Key: "ndjson", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||
)
|
||||
|
||||
resolved, err := ResolvePipeline(PipelineProfile{
|
||||
ID: " campaign ",
|
||||
Input: ModuleBinding{Module: " text ", LLMProfile: " fast "},
|
||||
Input: Binding(" text "),
|
||||
Chunk: ModuleBinding{Module: " window ", Options: map[string]any{
|
||||
"size": 10,
|
||||
}},
|
||||
Artifacts: map[string]ArtifactLaneProfile{
|
||||
" records ": {
|
||||
Extract: ModuleBinding{Module: " record-extractor ", LLMProfile: " careful "},
|
||||
Extract: Binding(" record-extractor "),
|
||||
Merge: Binding(" dedupe "),
|
||||
Normalize: Binding(" canonical "),
|
||||
},
|
||||
@@ -44,15 +44,21 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
||||
if resolved.ID != "campaign" {
|
||||
t.Fatalf("ID = %q, want campaign", resolved.ID)
|
||||
}
|
||||
if !reflect.DeepEqual(resolved.Input, ModuleBinding{Module: "text", LLMProfile: "fast"}) {
|
||||
if !reflect.DeepEqual(resolved.Input, ModuleBinding{Module: "text"}) {
|
||||
t.Fatalf("Input = %#v, want trimmed explicit input", resolved.Input)
|
||||
}
|
||||
if resolved.InputExecutionClass != contracts.ExecutionClassDeterministic {
|
||||
t.Fatalf("InputExecutionClass = %q, want deterministic", resolved.InputExecutionClass)
|
||||
}
|
||||
if resolved.Chunk.Module != "window" || resolved.Chunk.LLMProfile != "" {
|
||||
t.Fatalf("Chunk = %#v, want explicit module and empty LLM profile", resolved.Chunk)
|
||||
}
|
||||
if resolved.Chunk.Options["size"] != 10 {
|
||||
t.Fatalf("Chunk.Options = %#v, want size option", resolved.Chunk.Options)
|
||||
}
|
||||
if resolved.ChunkExecutionClass != contracts.ExecutionClassDeterministic {
|
||||
t.Fatalf("ChunkExecutionClass = %q, want deterministic", resolved.ChunkExecutionClass)
|
||||
}
|
||||
if len(resolved.Steps) != 1 || resolved.Steps[0].ID != "default" || len(resolved.Steps[0].ArtifactLanes) != 1 {
|
||||
t.Fatalf("resolved steps = %#v, want one default step with one lane", resolved.Steps)
|
||||
}
|
||||
@@ -60,9 +66,12 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
||||
if lane.ID != "records" {
|
||||
t.Fatalf("lane.ID = %q, want records", lane.ID)
|
||||
}
|
||||
if !reflect.DeepEqual(lane.Extract, ModuleBinding{Module: "record-extractor", LLMProfile: "careful"}) {
|
||||
if !reflect.DeepEqual(lane.Extract, ModuleBinding{Module: "record-extractor"}) {
|
||||
t.Fatalf("lane.Extract = %#v, want explicit extractor", lane.Extract)
|
||||
}
|
||||
if lane.ExtractExecutionClass != contracts.ExecutionClassDeterministic || lane.MergeExecutionClass != contracts.ExecutionClassDeterministic || lane.NormalizeExecutionClass != contracts.ExecutionClassDeterministic {
|
||||
t.Fatalf("lane execution classes = %q/%q/%q, want deterministic", lane.ExtractExecutionClass, lane.MergeExecutionClass, lane.NormalizeExecutionClass)
|
||||
}
|
||||
if lane.Merge.Module != "dedupe" || lane.Normalize.Module != "canonical" {
|
||||
t.Fatalf("lane merge/normalize = %#v/%#v, want explicit modules", lane.Merge, lane.Normalize)
|
||||
}
|
||||
@@ -72,11 +81,51 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
||||
if resolved.Output.Module != "ndjson" {
|
||||
t.Fatalf("Output.Module = %q, want ndjson", resolved.Output.Module)
|
||||
}
|
||||
if resolved.OutputExecutionClass != contracts.ExecutionClassDeterministic {
|
||||
t.Fatalf("OutputExecutionClass = %q, want deterministic", resolved.OutputExecutionClass)
|
||||
}
|
||||
if !strings.HasPrefix(resolved.Digest, "sha256:") {
|
||||
t.Fatalf("Digest = %q, want sha256 digest", resolved.Digest)
|
||||
}
|
||||
}
|
||||
|
||||
func TestModuleCatalogExecutionClassLooksUpRegisteredMetadata(t *testing.T) {
|
||||
catalog := newProfileCatalogWithOverrides(t,
|
||||
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||
ModuleSpec{Key: "llm-extract", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||
)
|
||||
registerProfileValidatorSpec(t, catalog, ValidatorSpec{Key: "llm-validator", ExecutionClass: contracts.ExecutionClassLLMBacked})
|
||||
|
||||
for _, test := range []struct {
|
||||
stage ModuleStage
|
||||
key string
|
||||
want contracts.ExecutionClass
|
||||
}{
|
||||
{stage: StageInput, key: "llm-input", want: contracts.ExecutionClassLLMBacked},
|
||||
{stage: StageChunk, key: "llm-chunk", want: contracts.ExecutionClassLLMBacked},
|
||||
{stage: StageExtract, key: "llm-extract", want: contracts.ExecutionClassLLMBacked},
|
||||
{stage: StageMerge, key: "llm-merge", want: contracts.ExecutionClassLLMBacked},
|
||||
{stage: StageNormalize, key: "llm-normalize", want: contracts.ExecutionClassLLMBacked},
|
||||
{stage: StageValidate, key: "llm-validator", want: contracts.ExecutionClassLLMBacked},
|
||||
{stage: StageOutput, key: "llm-output", want: contracts.ExecutionClassLLMBacked},
|
||||
} {
|
||||
t.Run(string(test.stage), func(t *testing.T) {
|
||||
got, ok := catalog.ExecutionClass(test.stage, test.key)
|
||||
if !ok || got != test.want {
|
||||
t.Fatalf("ExecutionClass(%q, %q) = %q, %t; want %q, true", test.stage, test.key, got, ok, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
if _, ok := catalog.ExecutionClass(StageExtract, "missing"); ok {
|
||||
t.Fatal("ExecutionClass() found an unregistered module")
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineAppliesDefaults(t *testing.T) {
|
||||
resolved, err := ResolvePipeline(PipelineProfile{
|
||||
ID: "defaulted",
|
||||
@@ -110,6 +159,176 @@ func TestResolvePipelineAppliesDefaults(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineAppliesEffectiveLLMProfiles(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
profile PipelineProfile
|
||||
options ResolveOptions
|
||||
want map[string]string
|
||||
}{
|
||||
{
|
||||
name: "runtime override",
|
||||
profile: func() PipelineProfile {
|
||||
profile := llmProfilePipeline()
|
||||
profile.LLMProfile = " pipeline "
|
||||
profile.Chunk.LLMProfile = "binding"
|
||||
return profile
|
||||
}(),
|
||||
options: ResolveOptions{LLMProfileOverride: " runtime "},
|
||||
want: llmProfileValues("runtime"),
|
||||
},
|
||||
{
|
||||
name: "binding exception",
|
||||
profile: func() PipelineProfile {
|
||||
profile := llmProfilePipeline()
|
||||
profile.LLMProfile = "pipeline"
|
||||
lane := profile.Artifacts["events"]
|
||||
lane.Extract.LLMProfile = "extract"
|
||||
lane.Extract.Validators = ValidatorOverride{
|
||||
Set: true,
|
||||
Validators: []ModuleBinding{{Module: "llm-validator", LLMProfile: "validator"}},
|
||||
}
|
||||
profile.Artifacts["events"] = lane
|
||||
return profile
|
||||
}(),
|
||||
want: func() map[string]string {
|
||||
values := llmProfileValues("pipeline")
|
||||
values["extract"] = "extract"
|
||||
values["validator:extract:events"] = "validator"
|
||||
return values
|
||||
}(),
|
||||
},
|
||||
{
|
||||
name: "pipeline default",
|
||||
profile: func() PipelineProfile {
|
||||
profile := llmProfilePipeline()
|
||||
profile.LLMProfile = "pipeline"
|
||||
return profile
|
||||
}(),
|
||||
want: llmProfileValues("pipeline"),
|
||||
},
|
||||
{
|
||||
name: "prompt fallback",
|
||||
profile: llmProfilePipeline(),
|
||||
want: llmProfileValues(""),
|
||||
},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
resolved, err := ResolvePipeline(test.profile, test.options, llmProfileCatalog(t))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
if got := resolvedLLMProfileValues(resolved); !reflect.DeepEqual(got, test.want) {
|
||||
t.Fatalf("resolved profiles = %#v, want %#v", got, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineAppliesProfilesOnlyToSelectedLLMBackedBindings(t *testing.T) {
|
||||
profile := llmProfilePipeline()
|
||||
profile.LLMProfile = "pipeline"
|
||||
profile.Artifacts["notes"] = ArtifactLaneProfile{Extract: Binding("llm-extractor")}
|
||||
|
||||
resolved, err := ResolvePipeline(profile, ResolveOptions{Only: []string{"events"}}, llmProfileCatalog(t))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
if got := laneIDs(resolved.Steps[0].ArtifactLanes); !reflect.DeepEqual(got, []string{"events"}) {
|
||||
t.Fatalf("selected lanes = %#v, want events only", got)
|
||||
}
|
||||
if got := resolvedLLMProfileValues(resolved); !reflect.DeepEqual(got, llmProfileValues("pipeline")) {
|
||||
t.Fatalf("resolved profiles = %#v, want selected LLM bindings only", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineLeavesUnusedProfilesOffDeterministicBindings(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
profile.LLMProfile = "unused"
|
||||
|
||||
resolved, err := ResolvePipeline(profile, ResolveOptions{LLMProfileOverride: "also-unused"}, newProfileCatalog(t))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
if resolved.Input.LLMProfile != "" || resolved.Chunk.LLMProfile != "" || resolved.Output.LLMProfile != "" {
|
||||
t.Fatalf("deterministic pipeline profiles = input %q chunk %q output %q, want empty", resolved.Input.LLMProfile, resolved.Chunk.LLMProfile, resolved.Output.LLMProfile)
|
||||
}
|
||||
lane := resolved.Steps[0].ArtifactLanes[0]
|
||||
if lane.Extract.LLMProfile != "" || lane.Merge.LLMProfile != "" || lane.Normalize.LLMProfile != "" {
|
||||
t.Fatalf("deterministic lane profiles = extract %q merge %q normalize %q, want empty", lane.Extract.LLMProfile, lane.Merge.LLMProfile, lane.Normalize.LLMProfile)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineRejectsLLMProfileForDeterministicModule(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
mutate func(*PipelineProfile)
|
||||
}{
|
||||
{name: "input", mutate: func(profile *PipelineProfile) { profile.Input.LLMProfile = "invalid" }},
|
||||
{name: "chunk", mutate: func(profile *PipelineProfile) { profile.Chunk.LLMProfile = "invalid" }},
|
||||
{name: "extract", mutate: func(profile *PipelineProfile) {
|
||||
lane := profile.Artifacts["events"]
|
||||
lane.Extract.LLMProfile = "invalid"
|
||||
profile.Artifacts["events"] = lane
|
||||
}},
|
||||
{name: "merge", mutate: func(profile *PipelineProfile) {
|
||||
lane := profile.Artifacts["events"]
|
||||
lane.Merge.LLMProfile = "invalid"
|
||||
profile.Artifacts["events"] = lane
|
||||
}},
|
||||
{name: "normalize", mutate: func(profile *PipelineProfile) {
|
||||
lane := profile.Artifacts["events"]
|
||||
lane.Normalize.LLMProfile = "invalid"
|
||||
profile.Artifacts["events"] = lane
|
||||
}},
|
||||
{name: "output", mutate: func(profile *PipelineProfile) { profile.Output.LLMProfile = "invalid" }},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
test.mutate(&profile)
|
||||
_, err := ResolvePipeline(profile, ResolveOptions{}, newProfileCatalog(t))
|
||||
if err == nil || !strings.Contains(err.Error(), "llm_profile") || !strings.Contains(err.Error(), "deterministic") {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want deterministic profile rejection", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineDigestUsesEffectiveLLMProfiles(t *testing.T) {
|
||||
inherited := llmProfilePipeline()
|
||||
inherited.LLMProfile = "shared"
|
||||
explicit := llmProfilePipeline()
|
||||
explicit.Input.LLMProfile = "shared"
|
||||
explicit.Chunk.LLMProfile = "shared"
|
||||
explicit.Output.LLMProfile = "shared"
|
||||
lane := explicit.Artifacts["events"]
|
||||
lane.Extract.LLMProfile = "shared"
|
||||
lane.Merge.LLMProfile = "shared"
|
||||
lane.Normalize.LLMProfile = "shared"
|
||||
explicit.Artifacts["events"] = lane
|
||||
|
||||
inheritedResolved, err := ResolvePipeline(inherited, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline(inherited) error = %v, want nil", err)
|
||||
}
|
||||
explicitResolved, err := ResolvePipeline(explicit, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline(explicit) error = %v, want nil", err)
|
||||
}
|
||||
if inheritedResolved.Digest != explicitResolved.Digest {
|
||||
t.Fatalf("effective profile digests differ: %q != %q", inheritedResolved.Digest, explicitResolved.Digest)
|
||||
}
|
||||
|
||||
explicit.Chunk.LLMProfile = "different"
|
||||
changedResolved, err := ResolvePipeline(explicit, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline(changed) error = %v, want nil", err)
|
||||
}
|
||||
if explicitResolved.Digest == changedResolved.Digest {
|
||||
t.Fatalf("digest = %q after effective profile change, want different", explicitResolved.Digest)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineRecordsValidatorChains(t *testing.T) {
|
||||
catalog := newProfileCatalog(t)
|
||||
if err := catalog.ValidatorChains.Register(ValidatorChainMapping{
|
||||
@@ -337,7 +556,8 @@ func TestResolvePipelinePreservesOrderedStepsAndExpandsGeneratedBindings(t *test
|
||||
catalog := newProfileCatalogWithOverrides(t,
|
||||
ModuleSpec{
|
||||
Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes",
|
||||
Requires: []string{"chunk"}, Provides: []string{"candidate"},
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"chunk"}, Provides: []string{"candidate"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}, AcceptedMediaTypes: []string{"application/json"}}},
|
||||
})
|
||||
resolved, err := ResolvePipeline(PipelineProfile{
|
||||
@@ -365,7 +585,7 @@ func TestResolvePipelinePreservesOrderedStepsAndExpandsGeneratedBindings(t *test
|
||||
|
||||
func TestResolvePipelineRejectsGeneratedBindingOrderingAndKind(t *testing.T) {
|
||||
catalog := newProfileCatalogWithOverrides(t,
|
||||
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/other"}}}},
|
||||
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/other"}}}},
|
||||
)
|
||||
_, err := ResolvePipeline(PipelineProfile{
|
||||
ID: "invalid-order", Input: Binding("text"), Steps: []PipelineStepProfile{
|
||||
@@ -400,10 +620,11 @@ func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
|
||||
profile.Artifacts["events"] = lane
|
||||
|
||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||
Key: "event-extractor",
|
||||
Stage: StageExtract,
|
||||
Requires: []string{"chunk"},
|
||||
Provides: []string{"candidate"},
|
||||
Key: "event-extractor",
|
||||
Stage: StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"chunk"},
|
||||
Provides: []string{"candidate"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{Name: "roster", Required: true},
|
||||
{Name: "lore"},
|
||||
@@ -446,6 +667,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToChunkTarget(t *testing.
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "generic",
|
||||
Stage: StageChunk,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"source"},
|
||||
Provides: []string{"chunk"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide"}},
|
||||
@@ -474,6 +696,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToExtractorTarget(t *test
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "event-extractor",
|
||||
Stage: StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"chunk"},
|
||||
Provides: []string{"candidate"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
|
||||
@@ -502,6 +725,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToNormalizerTarget(t *tes
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "noop",
|
||||
Stage: StageNormalize,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"merged"},
|
||||
Provides: []string{"normalized"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
||||
@@ -530,6 +754,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToMergeTarget(t *testing.
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "appendorder",
|
||||
Stage: StageMerge,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"candidate"},
|
||||
Provides: []string{"merged"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "merge_notes"}},
|
||||
@@ -559,10 +784,10 @@ func TestResolvePipelineAppliesOnePipelineReferenceDefaultToMultipleTargets(t *t
|
||||
profile := baselineProfile()
|
||||
profile.References = ExternalReferenceMap(map[string]string{"context": "./context.md"})
|
||||
catalog := newProfileCatalogWithOverrides(t,
|
||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||
ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||
ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||
ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"candidate"}, Provides: []string{"merged"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||
ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||
)
|
||||
|
||||
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
||||
@@ -580,10 +805,11 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedLane(t *t
|
||||
profile := multiLaneProfile()
|
||||
profile.References = ExternalReferenceMap(map[string]string{"notes_context": "./notes.md"})
|
||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||
Key: "note-extractor",
|
||||
Stage: StageExtract,
|
||||
Requires: []string{"chunk"},
|
||||
Provides: []string{"candidate"},
|
||||
Key: "note-extractor",
|
||||
Stage: StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"chunk"},
|
||||
Provides: []string{"candidate"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{Name: "notes_context"},
|
||||
},
|
||||
@@ -605,10 +831,11 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedNormalize
|
||||
lane.Normalize = Binding("note-normalizer")
|
||||
profile.Artifacts["notes"] = lane
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "note-normalizer",
|
||||
Stage: StageNormalize,
|
||||
Requires: []string{"merged"},
|
||||
Provides: []string{"normalized"},
|
||||
Key: "note-normalizer",
|
||||
Stage: StageNormalize,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"merged"},
|
||||
Provides: []string{"normalized"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{Name: "notes_context"},
|
||||
},
|
||||
@@ -658,6 +885,7 @@ func TestResolvePipelineRejectsExtractLocalReferenceDeclaredOnlyByNormalizer(t *
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "noop",
|
||||
Stage: StageNormalize,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"merged"},
|
||||
Provides: []string{"normalized"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
||||
@@ -676,10 +904,11 @@ func TestResolvePipelineRejectsMergeLocalReferenceDeclaredOnlyByNormalizer(t *te
|
||||
lane.Merge.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./normalize.md"})
|
||||
profile.Artifacts["events"] = lane
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "noop",
|
||||
Stage: StageNormalize,
|
||||
Requires: []string{"merged"},
|
||||
Provides: []string{"normalized"},
|
||||
Key: "noop",
|
||||
Stage: StageNormalize,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"merged"},
|
||||
Provides: []string{"normalized"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{Name: "normalization_notes"},
|
||||
},
|
||||
@@ -700,6 +929,7 @@ func TestResolvePipelineRejectsNormalizeLocalReferenceDeclaredOnlyByExtractor(t
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "event-extractor",
|
||||
Stage: StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"chunk"},
|
||||
Provides: []string{"candidate"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
|
||||
@@ -716,6 +946,7 @@ func TestResolvePipelineRequiresBoundChunkReference(t *testing.T) {
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "generic",
|
||||
Stage: StageChunk,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"source"},
|
||||
Provides: []string{"chunk"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide", Required: true}},
|
||||
@@ -732,6 +963,7 @@ func TestResolvePipelineRequiresBoundNormalizeReference(t *testing.T) {
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "noop",
|
||||
Stage: StageNormalize,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"merged"},
|
||||
Provides: []string{"normalized"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes", Required: true}},
|
||||
@@ -757,9 +989,9 @@ func TestResolvePipelineLocalReferencesOverridePipelineDefaultsForEligibleTarget
|
||||
lane.Normalize.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./local-normalize.md"})
|
||||
profile.Artifacts["events"] = lane
|
||||
catalog := newProfileCatalogWithOverrides(t,
|
||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}}},
|
||||
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}}},
|
||||
ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}}},
|
||||
ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}}},
|
||||
)
|
||||
|
||||
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
||||
@@ -774,10 +1006,11 @@ func TestResolvePipelineLocalReferencesOverridePipelineDefaultsForEligibleTarget
|
||||
|
||||
func TestResolvePipelineRequiresBoundReferenceSlotsForSelectedLanes(t *testing.T) {
|
||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||
Key: "event-extractor",
|
||||
Stage: StageExtract,
|
||||
Requires: []string{"chunk"},
|
||||
Provides: []string{"candidate"},
|
||||
Key: "event-extractor",
|
||||
Stage: StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"chunk"},
|
||||
Provides: []string{"candidate"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{Name: "roster", Required: true},
|
||||
},
|
||||
@@ -798,10 +1031,11 @@ func TestResolvePipelineReferenceUnbindCanLeaveRequiredSlotMissing(t *testing.T)
|
||||
profile := baselineProfile()
|
||||
profile.References = ExternalReferenceMap(map[string]string{"roster": "./roster.yml"})
|
||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||
Key: "event-extractor",
|
||||
Stage: StageExtract,
|
||||
Requires: []string{"chunk"},
|
||||
Provides: []string{"candidate"},
|
||||
Key: "event-extractor",
|
||||
Stage: StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"chunk"},
|
||||
Provides: []string{"candidate"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{Name: "roster", Required: true},
|
||||
},
|
||||
@@ -827,11 +1061,12 @@ func TestResolvePipelineUsesReferenceSlotsFromSpecWithoutConstructingExtractor(t
|
||||
}
|
||||
}
|
||||
if err := RegisterExtractor[codecNotes](catalog.Extractors, ModuleSpec{
|
||||
Key: "event-extractor",
|
||||
Stage: StageExtract,
|
||||
ArtifactKind: "test/notes",
|
||||
Requires: []string{"chunk"},
|
||||
Provides: []string{"candidate"},
|
||||
Key: "event-extractor",
|
||||
Stage: StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
ArtifactKind: "test/notes",
|
||||
Requires: []string{"chunk"},
|
||||
Provides: []string{"candidate"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{Name: "roster", Required: true},
|
||||
},
|
||||
@@ -989,32 +1224,32 @@ func TestResolvePipelineRejectsMissingCapabilities(t *testing.T) {
|
||||
}{
|
||||
{
|
||||
name: "input",
|
||||
spec: ModuleSpec{Key: "text", Stage: StageInput, Requires: []string{"raw"}},
|
||||
spec: ModuleSpec{Key: "text", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"raw"}},
|
||||
want: []string{"baseline", "input", "text", "raw"},
|
||||
},
|
||||
{
|
||||
name: "chunk",
|
||||
spec: ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"missing"}},
|
||||
spec: ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||
want: []string{"baseline", "chunk", "generic", "missing"},
|
||||
},
|
||||
{
|
||||
name: "extract",
|
||||
spec: ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"missing"}},
|
||||
spec: ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||
want: []string{"baseline", "events", "extract", "event-extractor", "missing"},
|
||||
},
|
||||
{
|
||||
name: "merge",
|
||||
spec: ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"missing"}},
|
||||
spec: ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||
want: []string{"baseline", "events", "merge", "appendorder", "missing"},
|
||||
},
|
||||
{
|
||||
name: "normalize",
|
||||
spec: ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"missing"}},
|
||||
spec: ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||
want: []string{"baseline", "events", "normalize", "noop", "missing"},
|
||||
},
|
||||
{
|
||||
name: "output",
|
||||
spec: ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"missing"}},
|
||||
spec: ModuleSpec{Key: "json", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||
want: []string{"baseline", "output", "json", "missing"},
|
||||
},
|
||||
}
|
||||
@@ -1100,7 +1335,7 @@ func TestResolvePipelineDigestChangesWhenBindingChanges(t *testing.T) {
|
||||
right := baselineProfile()
|
||||
right.Chunk = Binding("window")
|
||||
catalog := newProfileCatalog(t)
|
||||
registerProfileSpecs(t, catalog, ModuleSpec{Key: "window", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}})
|
||||
registerProfileSpecs(t, catalog, ModuleSpec{Key: "window", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}})
|
||||
|
||||
leftResolved, err := ResolvePipeline(left, ResolveOptions{}, catalog)
|
||||
if err != nil {
|
||||
@@ -1467,17 +1702,103 @@ func emptyProfileCatalog() ModuleCatalog {
|
||||
|
||||
func defaultProfileSpecs() []ModuleSpec {
|
||||
return []ModuleSpec{
|
||||
ModuleSpec{Key: "text", Stage: StageInput, Provides: []string{"source"}},
|
||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||
ModuleSpec{Key: "appendorder", Stage: StageMerge, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||
ModuleSpec{Key: "noop", Stage: StageNormalize, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||
ModuleSpec{Key: "grounded", Stage: StageValidate, Requires: []string{"normalized"}, Provides: []string{"validated"}},
|
||||
ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||
ModuleSpec{Key: "text", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}},
|
||||
ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||
ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||
ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||
ModuleSpec{Key: "grounded", Stage: StageValidate, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"validated"}},
|
||||
ModuleSpec{Key: "json", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||
}
|
||||
}
|
||||
|
||||
func llmProfileCatalog(t *testing.T) ModuleCatalog {
|
||||
t.Helper()
|
||||
catalog := newProfileCatalogWithOverrides(t,
|
||||
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked, Provides: []string{"source"}},
|
||||
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||
ModuleSpec{Key: "llm-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||
)
|
||||
if err := RegisterChunkValidator(catalog.Validators, ValidatorSpec{Key: "llm-chunk-validator", ExecutionClass: contracts.ExecutionClassLLMBacked}, func() (contracts.ChunkValidator, error) {
|
||||
return typedTestChunkValidator{key: "llm-chunk-validator"}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register chunk validator: %v", err)
|
||||
}
|
||||
registerProfileValidatorSpec(t, catalog, ValidatorSpec{Key: "llm-validator", ExecutionClass: contracts.ExecutionClassLLMBacked})
|
||||
for _, mapping := range []ValidatorChainMapping{
|
||||
{Stage: StageChunk, Module: "llm-chunk", Validators: []ModuleBinding{Binding("llm-chunk-validator")}},
|
||||
{Stage: StageExtract, Module: "llm-extractor", Validators: []ModuleBinding{Binding("llm-validator")}},
|
||||
{Stage: StageMerge, Module: "llm-merge", Validators: []ModuleBinding{Binding("llm-validator")}},
|
||||
{Stage: StageNormalize, Module: "llm-normalize", Validators: []ModuleBinding{Binding("llm-validator")}},
|
||||
} {
|
||||
if err := catalog.ValidatorChains.Register(mapping); err != nil {
|
||||
t.Fatalf("register validator chain %#v: %v", mapping, err)
|
||||
}
|
||||
}
|
||||
return catalog
|
||||
}
|
||||
|
||||
func llmProfileCatalogWithoutValidatorChains(t *testing.T) ModuleCatalog {
|
||||
catalog := llmProfileCatalog(t)
|
||||
catalog.ValidatorChains = NewValidatorChainRegistry()
|
||||
return catalog
|
||||
}
|
||||
|
||||
func llmProfilePipeline() PipelineProfile {
|
||||
return PipelineProfile{
|
||||
ID: "llm-profile",
|
||||
Input: Binding("llm-input"),
|
||||
Chunk: Binding("llm-chunk"),
|
||||
Output: Binding("llm-output"),
|
||||
Artifacts: map[string]ArtifactLaneProfile{
|
||||
"events": {
|
||||
Extract: Binding("llm-extractor"),
|
||||
Merge: Binding("llm-merge"),
|
||||
Normalize: Binding("llm-normalize"),
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func llmProfileValues(profile string) map[string]string {
|
||||
return map[string]string{
|
||||
"input": profile,
|
||||
"chunk": profile,
|
||||
"extract": profile,
|
||||
"merge": profile,
|
||||
"normalize": profile,
|
||||
"output": profile,
|
||||
"validator:chunk:": profile,
|
||||
"validator:extract:events": profile,
|
||||
"validator:merge:events": profile,
|
||||
"validator:normalize:events": profile,
|
||||
}
|
||||
}
|
||||
|
||||
func resolvedLLMProfileValues(resolved ResolvedPipeline) map[string]string {
|
||||
values := map[string]string{
|
||||
"input": resolved.Input.LLMProfile,
|
||||
"chunk": resolved.Chunk.LLMProfile,
|
||||
"output": resolved.Output.LLMProfile,
|
||||
}
|
||||
lane := resolved.Steps[0].ArtifactLanes[0]
|
||||
values["extract"] = lane.Extract.LLMProfile
|
||||
values["merge"] = lane.Merge.LLMProfile
|
||||
values["normalize"] = lane.Normalize.LLMProfile
|
||||
for _, chain := range resolved.ValidatorChains {
|
||||
for _, validator := range chain.Validators {
|
||||
if validator.ExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||
values["validator:"+string(chain.Stage)+":"+chain.LaneID] = validator.Binding.LLMProfile
|
||||
}
|
||||
}
|
||||
}
|
||||
return values
|
||||
}
|
||||
|
||||
func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSpec) {
|
||||
t.Helper()
|
||||
|
||||
@@ -1566,7 +1887,13 @@ func TestResolvedPipelineCanMarshalToCanonicalJSON(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
if _, err := json.Marshal(resolved); err != nil {
|
||||
encoded, err := json.Marshal(resolved)
|
||||
if err != nil {
|
||||
t.Fatalf("json.Marshal(resolved) error = %v, want nil", err)
|
||||
}
|
||||
for _, field := range []string{"input_execution_class", "chunk_execution_class", "extract_execution_class", "merge_execution_class", "normalize_execution_class", "output_execution_class"} {
|
||||
if !strings.Contains(string(encoded), `"`+field+`":"deterministic"`) {
|
||||
t.Fatalf("resolved JSON does not retain %q: %s", field, encoded)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -189,12 +189,12 @@ func TestMaterializeReferencesUsesLaneArtifactVariant(t *testing.T) {
|
||||
{kind: "test/beta", mergeSlot: "beta_merge", normalizeSlot: "beta_normalize"},
|
||||
{kind: "test/alpha", mergeSlot: "alpha_merge", normalizeSlot: "alpha_normalize"},
|
||||
} {
|
||||
if err := RegisterMerger(mergers, ModuleSpec{Key: "shared/merge", Stage: StageMerge, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.mergeSlot}}}, func() (contracts.Merger[codecNotes], error) {
|
||||
if err := RegisterMerger(mergers, ModuleSpec{Key: "shared/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.mergeSlot}}}, func() (contracts.Merger[codecNotes], error) {
|
||||
return typedTestMerger[codecNotes]{key: "shared/merge"}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterMerger(%s): %v", item.kind, err)
|
||||
}
|
||||
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "shared/normalize", Stage: StageNormalize, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.normalizeSlot}}}, func() (contracts.Normalizer[codecNotes], error) {
|
||||
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "shared/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.normalizeSlot}}}, func() (contracts.Normalizer[codecNotes], error) {
|
||||
return typedTestNormalizer[codecNotes]{key: "shared/normalize"}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterNormalizer(%s): %v", item.kind, err)
|
||||
@@ -531,6 +531,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
||||
ModuleSpec{
|
||||
Key: "generic",
|
||||
Stage: StageChunk,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"source"},
|
||||
Provides: []string{"chunk"},
|
||||
ReferenceSlots: chunkSlots,
|
||||
@@ -538,6 +539,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
||||
ModuleSpec{
|
||||
Key: "event-extractor",
|
||||
Stage: StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"chunk"},
|
||||
Provides: []string{"candidate"},
|
||||
ReferenceSlots: extractSlots,
|
||||
@@ -545,6 +547,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
||||
ModuleSpec{
|
||||
Key: "appendorder",
|
||||
Stage: StageMerge,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"candidate"},
|
||||
Provides: []string{"merged"},
|
||||
ReferenceSlots: mergeSlots,
|
||||
@@ -552,6 +555,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
||||
ModuleSpec{
|
||||
Key: "noop",
|
||||
Stage: StageNormalize,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"merged"},
|
||||
Provides: []string{"normalized"},
|
||||
ReferenceSlots: normalizeSlots,
|
||||
|
||||
@@ -116,7 +116,7 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
|
||||
return false, nil, terminal.record(nil, fmt.Errorf("clone chunker manifest metadata: %w", metadataErr))
|
||||
}
|
||||
profile := ""
|
||||
if provider, ok := chunker.(contracts.ChunkExecutionClassProvider); ok && provider.ExecutionClass() == contracts.ExecutionClassLLMBacked {
|
||||
if input.pipeline.ChunkExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||
profile = input.pipeline.Chunk.LLMProfile
|
||||
}
|
||||
candidate := ChunkPlanRecord{
|
||||
|
||||
@@ -67,30 +67,17 @@ type manifestChunker struct {
|
||||
}
|
||||
|
||||
func (c manifestChunker) ManifestMetadata() map[string]any { return c.metadata }
|
||||
func (manifestChunker) ExecutionClass() contracts.ExecutionClass {
|
||||
return contracts.ExecutionClassLLMBacked
|
||||
}
|
||||
|
||||
type llmCountingChunker struct {
|
||||
terminalChunker
|
||||
llmCalls *int
|
||||
}
|
||||
|
||||
func (llmCountingChunker) ExecutionClass() contracts.ExecutionClass {
|
||||
return contracts.ExecutionClassLLMBacked
|
||||
}
|
||||
|
||||
func (c llmCountingChunker) Plan(ctx context.Context, request contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
|
||||
(*c.llmCalls)++
|
||||
return c.terminalChunker.Plan(ctx, request)
|
||||
}
|
||||
|
||||
type deterministicChunker struct{ terminalChunker }
|
||||
|
||||
func (deterministicChunker) ExecutionClass() contracts.ExecutionClass {
|
||||
return contracts.ExecutionClassDeterministic
|
||||
}
|
||||
|
||||
type retryingChunker struct {
|
||||
key string
|
||||
plan source.ChunkPlan
|
||||
@@ -535,6 +522,7 @@ func TestRunnerPublishesOnlyAcceptedGeneratedPlans(t *testing.T) {
|
||||
func TestRunnerStoresProducerProvenanceAndProducerWarnings(t *testing.T) {
|
||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||
prepared.resolved.Chunk.LLMProfile = "chunk-profile"
|
||||
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
|
||||
prepared.resolved.ChunkReferences = ResolvedReferenceTarget{
|
||||
Stage: StageChunk,
|
||||
ReferenceSet: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||
@@ -577,7 +565,8 @@ func TestRunnerRejectsUncloneableModuleManifestMetadata(t *testing.T) {
|
||||
func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
|
||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||
prepared.resolved.Chunk.LLMProfile = "configured-but-unused"
|
||||
prepared.chunker = deterministicChunker{terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}}
|
||||
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassDeterministic
|
||||
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}
|
||||
store := &recordingChunkPlanStore{}
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store})
|
||||
if err != nil {
|
||||
@@ -588,6 +577,22 @@ func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerDoesNotInventProducerProfileForLLMChunker(t *testing.T) {
|
||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
|
||||
prepared.resolved.Chunk.LLMProfile = ""
|
||||
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}
|
||||
store := &recordingChunkPlanStore{}
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if store.saved.Producer.LLMProfile != "" || output.Manifest.ChunkPlan.ProducerLLMProfile != "" {
|
||||
t.Fatalf("LLM producer profile = stored %q manifest %q, want empty", store.saved.Producer.LLMProfile, output.Manifest.ChunkPlan.ProducerLLMProfile)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerRefreshChangesDownstreamChunkFingerprint(t *testing.T) {
|
||||
doc := typedTestDocumentWithUnits(2)
|
||||
prepared := preparedConcurrentPipeline(t, 1)
|
||||
|
||||
@@ -229,7 +229,7 @@ func TestResolveTypedLaneRejectsIncompatibleComposition(t *testing.T) {
|
||||
|
||||
func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) {
|
||||
registry := NewMergerRegistry()
|
||||
spec := ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"}
|
||||
spec := ModuleSpec{Key: "typed/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}
|
||||
constructor := func() (contracts.Merger[codecNotes], error) {
|
||||
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
|
||||
}
|
||||
@@ -243,7 +243,7 @@ func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) {
|
||||
|
||||
func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
|
||||
extractors := NewExtractorRegistry()
|
||||
if err := RegisterExtractor(extractors, ModuleSpec{Key: "typed/extract", Stage: StageExtract, ArtifactKind: "test/notes"}, func() (contracts.Extractor[codecNotes], error) {
|
||||
if err := RegisterExtractor(extractors, ModuleSpec{Key: "typed/extract", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Extractor[codecNotes], error) {
|
||||
return typedTestExtractor[codecNotes]{key: "typed/extract"}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterExtractor() error = %v", err)
|
||||
@@ -253,7 +253,7 @@ func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
|
||||
}
|
||||
|
||||
mergers := NewMergerRegistry()
|
||||
if err := RegisterMerger(mergers, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"}, func() (contracts.Merger[codecNotes], error) {
|
||||
if err := RegisterMerger(mergers, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Merger[codecNotes], error) {
|
||||
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterMerger() error = %v", err)
|
||||
@@ -263,7 +263,7 @@ func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
|
||||
}
|
||||
|
||||
normalizers := NewNormalizerRegistry()
|
||||
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
||||
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
||||
return typedTestNormalizer[codecNotes]{key: "typed/normalize"}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||
@@ -298,7 +298,7 @@ func TestTypedRegistrySpecLookupUsesArtifactKindAndStableCatalogOrder(t *testing
|
||||
mergers := NewMergerRegistry()
|
||||
normalizers := NewNormalizerRegistry()
|
||||
for _, item := range order {
|
||||
base := ModuleSpec{Key: "typed/shared", ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.slot}}}
|
||||
base := ModuleSpec{Key: "typed/shared", ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.slot}}}
|
||||
mergeSpec := base
|
||||
mergeSpec.Stage = StageMerge
|
||||
normalizeSpec := base
|
||||
@@ -474,18 +474,18 @@ func typedResolutionCatalog(t *testing.T, options typedCatalogOptions) ModuleCat
|
||||
|
||||
func mustRegisterTypedTestBase(t *testing.T, catalog ModuleCatalog) {
|
||||
t.Helper()
|
||||
if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{Key: "typed/input", Stage: StageInput}, func() (contracts.InputAdapter, error) {
|
||||
if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{Key: "typed/input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.InputAdapter, error) {
|
||||
return &typedTestInput{key: "typed/input", doc: typedTestDocument()}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register input: %v", err)
|
||||
}
|
||||
if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{Key: "typed/chunk", Stage: StageChunk}, func() (contracts.Chunker, error) {
|
||||
if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{Key: "typed/chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.Chunker, error) {
|
||||
doc := typedTestDocument()
|
||||
return &typedTestChunker{key: "typed/chunk", plan: typedTestPlan(doc)}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register chunker: %v", err)
|
||||
}
|
||||
if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{Key: "typed/output", Stage: StageOutput}, func() (contracts.OutputEncoder, error) {
|
||||
if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{Key: "typed/output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.OutputEncoder, error) {
|
||||
return &typedTestOutput{key: "typed/output"}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register output: %v", err)
|
||||
@@ -509,21 +509,21 @@ func mustRegisterArtifactCodec[T any](t *testing.T, registry *ArtifactCodecRegis
|
||||
|
||||
func mustRegisterTypedExtractor[T any](t *testing.T, registry *ExtractorRegistry, key string, kind contracts.ArtifactKind, extractor contracts.Extractor[T]) {
|
||||
t.Helper()
|
||||
if err := RegisterExtractor(registry, ModuleSpec{Key: key, Stage: StageExtract, ArtifactKind: kind}, func() (contracts.Extractor[T], error) { return extractor, nil }); err != nil {
|
||||
if err := RegisterExtractor(registry, ModuleSpec{Key: key, Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: kind}, func() (contracts.Extractor[T], error) { return extractor, nil }); err != nil {
|
||||
t.Fatalf("RegisterExtractor() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustRegisterTypedMerger[T any](t *testing.T, registry *MergerRegistry, kind contracts.ArtifactKind, merger contracts.Merger[T]) {
|
||||
t.Helper()
|
||||
if err := RegisterMerger(registry, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: kind}, func() (contracts.Merger[T], error) { return merger, nil }); err != nil {
|
||||
if err := RegisterMerger(registry, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: kind}, func() (contracts.Merger[T], error) { return merger, nil }); err != nil {
|
||||
t.Fatalf("RegisterMerger() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustRegisterTypedNormalizer[T any](t *testing.T, registry *NormalizerRegistry, kind contracts.ArtifactKind, normalizer contracts.Normalizer[T]) {
|
||||
t.Helper()
|
||||
if err := RegisterNormalizer(registry, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: kind}, func() (contracts.Normalizer[T], error) { return normalizer, nil }); err != nil {
|
||||
if err := RegisterNormalizer(registry, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: kind}, func() (contracts.Normalizer[T], error) { return normalizer, nil }); err != nil {
|
||||
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
id: dnd.scenes
|
||||
version: "v1"
|
||||
default_profile: gemini-2-flash
|
||||
default_profile: dnd-extraction
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
|
||||
@@ -47,10 +47,6 @@ func (c *Chunker) Key() string {
|
||||
return Key
|
||||
}
|
||||
|
||||
func (*Chunker) ExecutionClass() contracts.ExecutionClass {
|
||||
return contracts.ExecutionClassLLMBacked
|
||||
}
|
||||
|
||||
func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return shared.ReferenceSlots(referenceSlotDescriptions)
|
||||
}
|
||||
@@ -120,6 +116,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageChunk,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ReferenceSlots: shared.ReferenceSlots(referenceSlotDescriptions),
|
||||
|
||||
@@ -24,6 +24,7 @@ func TestNewModuleSpecAndRegister(t *testing.T) {
|
||||
want := pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageChunk,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: []string{"source.transcript"},
|
||||
Provides: []string{"chunks"},
|
||||
ReferenceSlots: wantReferenceSlots(),
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
id: dnd.combat_turns
|
||||
version: "v1"
|
||||
default_profile: gemini-2-flash
|
||||
default_profile: dnd-extraction
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
|
||||
@@ -226,6 +226,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.CombatTurnListKind,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
id: dnd.item_events
|
||||
version: "v1"
|
||||
default_profile: gemini-2-flash
|
||||
default_profile: dnd-extraction
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
|
||||
@@ -129,6 +129,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.ItemEventListKind,
|
||||
|
||||
@@ -18,7 +18,7 @@ func TestConstructorSpecOptionsAndMetadata(t *testing.T) {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
want := pipeline.ModuleSpec{
|
||||
Key: Key, Stage: pipeline.StageExtract, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.item_events"}, ArtifactKind: dnd.ItemEventListKind,
|
||||
Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.item_events"}, ArtifactKind: dnd.ItemEventListKind,
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{Name: "glossary", Description: referenceSlotDescriptions.Glossary, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||
{Name: "party", Description: referenceSlotDescriptions.Party, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
id: dnd.npc_interactions
|
||||
version: "v1"
|
||||
default_profile: gemini-2-flash
|
||||
default_profile: dnd-extraction
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
|
||||
@@ -177,6 +177,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.NPCInteractionListKind,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
id: dnd.npcs
|
||||
version: "v1"
|
||||
default_profile: gemini-2-flash
|
||||
default_profile: dnd-extraction
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
|
||||
@@ -135,6 +135,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.NPCListKind,
|
||||
|
||||
@@ -25,11 +25,12 @@ func TestNewRequiresLLMClientAndRejectsAmbiguousReferences(t *testing.T) {
|
||||
func TestModuleSpecAndReferenceSlots(t *testing.T) {
|
||||
got := ModuleSpec()
|
||||
want := pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
Requires: []string{"chunks", "source.transcript"},
|
||||
Provides: []string{"dnd.npcs"},
|
||||
ArtifactKind: dnd.NPCListKind,
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: []string{"chunks", "source.transcript"},
|
||||
Provides: []string{"dnd.npcs"},
|
||||
ArtifactKind: dnd.NPCListKind,
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{Name: "glossary", Description: "Optional campaign glossary reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||
{Name: "party", Description: "Optional party roster reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
id: dnd.scene_descriptions
|
||||
version: "v1"
|
||||
default_profile: gemini-2-flash
|
||||
default_profile: dnd-extraction
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
|
||||
@@ -146,6 +146,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.SceneDescriptionListKind,
|
||||
|
||||
@@ -24,7 +24,7 @@ func TestNewRequiresLLMClientAndRejectsAmbiguousReferences(t *testing.T) {
|
||||
|
||||
func TestModuleSpecAndReferenceSlots(t *testing.T) {
|
||||
want := pipeline.ModuleSpec{
|
||||
Key: Key, Stage: pipeline.StageExtract, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.scene_descriptions"}, ArtifactKind: dnd.SceneDescriptionListKind,
|
||||
Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.scene_descriptions"}, ArtifactKind: dnd.SceneDescriptionListKind,
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{Name: "glossary", Description: "Optional campaign glossary reference material used only to disambiguate scene descriptions.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||
{Name: "party", Description: "Optional party roster reference material used only to disambiguate scene descriptions.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
id: dnd.spells
|
||||
version: "v1"
|
||||
default_profile: gemini-2-flash
|
||||
default_profile: dnd-extraction
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
|
||||
@@ -200,6 +200,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.SpellListKind,
|
||||
|
||||
@@ -23,8 +23,9 @@ func TestNewRequiresLLMClientAndReturnsExtractor(t *testing.T) {
|
||||
func TestModuleSpec(t *testing.T) {
|
||||
got := ModuleSpec()
|
||||
want := pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: []string{
|
||||
"chunks",
|
||||
"source.transcript",
|
||||
|
||||
@@ -336,6 +336,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageNormalize,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.CombatTurnListKind,
|
||||
|
||||
@@ -222,11 +222,12 @@ func eventScope(index int) string { return fmt.Sprintf("events[%d]", index) }
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageNormalize,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.ItemEventListKind,
|
||||
Key: Key,
|
||||
Stage: pipeline.StageNormalize,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.ItemEventListKind,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -278,6 +278,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageNormalize,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.NPCInteractionListKind,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
id: dnd.npcs.normalize
|
||||
version: "v1"
|
||||
default_profile: gemini-2-flash
|
||||
default_profile: dnd-extraction
|
||||
inputs:
|
||||
- name: candidates
|
||||
required: true
|
||||
|
||||
@@ -360,7 +360,7 @@ func duplicateWarning(retainedIndex int, removed []int) contracts.Warning {
|
||||
func npcScope(index int) string { return fmt.Sprintf("npcs[%d]", index) }
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, Requires: append([]string(nil), requiredCapabilities...), Provides: append([]string(nil), providedCapabilities...), ArtifactKind: dnd.NPCListKind}
|
||||
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: append([]string(nil), requiredCapabilities...), Provides: append([]string(nil), providedCapabilities...), ArtifactKind: dnd.NPCListKind}
|
||||
}
|
||||
|
||||
func Register(registry *pipeline.NormalizerRegistry) error {
|
||||
|
||||
@@ -21,7 +21,7 @@ func TestModuleContractAndIdentity(t *testing.T) {
|
||||
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
|
||||
t.Fatal("DecodeOptions() accepted unknown option")
|
||||
}
|
||||
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: dnd.NPCListKind}
|
||||
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: dnd.NPCListKind}
|
||||
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||
}
|
||||
|
||||
@@ -138,11 +138,12 @@ func sourceRefLabel(ref source.SourceRef) string {
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageNormalize,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.SceneDescriptionListKind,
|
||||
Key: Key,
|
||||
Stage: pipeline.StageNormalize,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.SceneDescriptionListKind,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -90,7 +90,7 @@ func TestNormalizerContractAndCancellation(t *testing.T) {
|
||||
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
|
||||
t.Fatal("DecodeOptions() accepted unknown options")
|
||||
}
|
||||
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: dnd.SceneDescriptionListKind}
|
||||
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: dnd.SceneDescriptionListKind}
|
||||
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||
}
|
||||
|
||||
@@ -297,6 +297,7 @@ func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageNormalize,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.SpellListKind,
|
||||
|
||||
@@ -24,11 +24,12 @@ func TestModuleContractAndStrictOptions(t *testing.T) {
|
||||
}
|
||||
|
||||
want := pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageNormalize,
|
||||
Requires: []string{"merged"},
|
||||
Provides: []string{"normalized"},
|
||||
ArtifactKind: dnd.SpellListKind,
|
||||
Key: Key,
|
||||
Stage: pipeline.StageNormalize,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"merged"},
|
||||
Provides: []string{"normalized"},
|
||||
ArtifactKind: dnd.SpellListKind,
|
||||
ReferenceSlots: []contracts.ReferenceSlot{{
|
||||
Name: spellcatalog.SpellCatalogReferenceSlot,
|
||||
Description: "Optional canonical spell-name catalog used for normalization and duplicate identity.",
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
id: dnd-extraction
|
||||
backend: openrouter
|
||||
model: openai/gpt-5.6-luna
|
||||
timeout_seconds: 240
|
||||
service_tier: flex
|
||||
14
internal/modules/dnd/register/profiles.go
Normal file
14
internal/modules/dnd/register/profiles.go
Normal file
@@ -0,0 +1,14 @@
|
||||
package register
|
||||
|
||||
import (
|
||||
"embed"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
)
|
||||
|
||||
//go:embed assets/profiles/*.yaml
|
||||
var embeddedProfileAssets embed.FS
|
||||
|
||||
func registerFallbackProfiles(assets *llm.AssetRegistry) error {
|
||||
return assets.RegisterFallbackProfileFS(embeddedProfileAssets, "assets/profiles")
|
||||
}
|
||||
@@ -30,6 +30,9 @@ func Register(registries pipeline.Registries, assets *llm.AssetRegistry) error {
|
||||
if err := registerPromptAssets(assets); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := registerFallbackProfiles(assets); err != nil {
|
||||
return err
|
||||
}
|
||||
return registerDefaultChains(registries.ValidatorChains)
|
||||
}
|
||||
|
||||
|
||||
@@ -39,6 +39,31 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
|
||||
if _, err := fs.ReadFile(promptFS, "dnd.npcs.normalize/dnd.npcs.normalize.yaml"); err != nil {
|
||||
t.Fatalf("normalization prompt asset = %v, want registered private prompt", err)
|
||||
}
|
||||
for _, name := range []string{
|
||||
"dnd.scenes/dnd.scenes.yaml",
|
||||
"dnd.spells/dnd.spells.yaml",
|
||||
"dnd.npcs/dnd.npcs.yaml",
|
||||
"dnd.combat_turns/dnd.combat_turns.yaml",
|
||||
"dnd.item_events/dnd.item_events.yaml",
|
||||
"dnd.npc_interactions/dnd.npc_interactions.yaml",
|
||||
"dnd.scene_descriptions/dnd.scene_descriptions.yaml",
|
||||
"dnd.npcs.normalize/dnd.npcs.normalize.yaml",
|
||||
} {
|
||||
content, err := fs.ReadFile(promptFS, name)
|
||||
if err != nil {
|
||||
t.Fatalf("read prompt asset %q: %v", name, err)
|
||||
}
|
||||
if !strings.Contains(string(content), "default_profile: dnd-extraction") {
|
||||
t.Fatalf("prompt asset %q does not select dnd-extraction", name)
|
||||
}
|
||||
}
|
||||
fallbackFS, err := assets.FallbackProfileFS()
|
||||
if err != nil {
|
||||
t.Fatalf("FallbackProfileFS() error = %v", err)
|
||||
}
|
||||
if _, err := fs.ReadFile(fallbackFS, "dnd-extraction.yaml"); err != nil {
|
||||
t.Fatalf("fallback profile asset = %v, want registered D&D profile", err)
|
||||
}
|
||||
schemaFS, err := assets.SchemaFS()
|
||||
if err != nil {
|
||||
t.Fatalf("SchemaFS() error = %v", err)
|
||||
|
||||
@@ -38,10 +38,6 @@ func (c *Chunker) Key() string {
|
||||
return Key
|
||||
}
|
||||
|
||||
func (*Chunker) ExecutionClass() contracts.ExecutionClass {
|
||||
return contracts.ExecutionClassDeterministic
|
||||
}
|
||||
|
||||
func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
@@ -90,9 +86,10 @@ func (c *Chunker) Plan(ctx context.Context, req contracts.ChunkRequest) (contrac
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageChunk,
|
||||
Provides: []string{"chunks"},
|
||||
Key: Key,
|
||||
Stage: pipeline.StageChunk,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Provides: []string{"chunks"},
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -15,9 +15,10 @@ import (
|
||||
|
||||
func TestModuleSpecAndRegister(t *testing.T) {
|
||||
want := pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageChunk,
|
||||
Provides: []string{"chunks"},
|
||||
Key: Key,
|
||||
Stage: pipeline.StageChunk,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Provides: []string{"chunks"},
|
||||
}
|
||||
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||
|
||||
@@ -3,13 +3,14 @@ package appendorder
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
)
|
||||
|
||||
const Key = "appendorder"
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageMerge, Provides: []string{"merged"}}
|
||||
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"merged"}}
|
||||
}
|
||||
|
||||
func mergerErrorf(format string, args ...any) error {
|
||||
|
||||
@@ -3,13 +3,14 @@ package noop
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
)
|
||||
|
||||
const Key = "noop"
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}}
|
||||
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}}
|
||||
}
|
||||
|
||||
func normalizerErrorf(format string, args ...any) error {
|
||||
|
||||
@@ -80,10 +80,11 @@ func (e *Encoder) Encode(ctx context.Context, req contracts.OutputRequest) (cont
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageOutput,
|
||||
Requires: []string{"normalized"},
|
||||
Provides: []string{"encoded"},
|
||||
Key: Key,
|
||||
Stage: pipeline.StageOutput,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"normalized"},
|
||||
Provides: []string{"encoded"},
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -17,10 +17,11 @@ import (
|
||||
|
||||
func TestModuleSpecAndRegister(t *testing.T) {
|
||||
want := pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageOutput,
|
||||
Requires: []string{"normalized"},
|
||||
Provides: []string{"encoded"},
|
||||
Key: Key,
|
||||
Stage: pipeline.StageOutput,
|
||||
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: []string{"normalized"},
|
||||
Provides: []string{"encoded"},
|
||||
}
|
||||
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||
|
||||
@@ -123,11 +123,11 @@ func TestRunnerIndependentlyBoundsWorkersAndProviderCallsAcrossRegisteredModules
|
||||
}
|
||||
client := frameworkllm.NewScheduledClient(instrumentedProvider{tracker: tracker}, scheduler)
|
||||
catalog := dndSpellsTestCatalog(t, dndSpellsCatalogSpecs{})
|
||||
if err := catalog.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: concurrentChunkerKey, Stage: pipeline.StageChunk, Requires: []string{"source.transcript"}, Provides: []string{"chunks"}}, func() (contracts.Chunker, error) { return concurrentChunker{}, nil }); err != nil {
|
||||
if err := catalog.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: concurrentChunkerKey, Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source.transcript"}, Provides: []string{"chunks"}}, func() (contracts.Chunker, error) { return concurrentChunker{}, nil }); err != nil {
|
||||
t.Fatalf("register chunker: %v", err)
|
||||
}
|
||||
validateOptions := func(options map[string]any) error { return pipeline.RejectUnknownOptions(options) }
|
||||
if err := pipeline.RegisterExtractorBuilder(catalog.Extractors, pipeline.ModuleSpec{Key: concurrentExtractorKey, Stage: pipeline.StageExtract, ArtifactKind: dnd.SpellListKind, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.spell_casts"}}, validateOptions, func(request pipeline.BuildRequest) (contracts.Extractor[dnd.SpellList], error) {
|
||||
if err := pipeline.RegisterExtractorBuilder(catalog.Extractors, pipeline.ModuleSpec{Key: concurrentExtractorKey, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: dnd.SpellListKind, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.spell_casts"}}, validateOptions, func(request pipeline.BuildRequest) (contracts.Extractor[dnd.SpellList], error) {
|
||||
return &concurrentExtractor{client: request.Dependencies.LLM, tracker: tracker}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register extractor: %v", err)
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user