Compare commits
5 Commits
7cfab8ada0
...
f1a6574013
| Author | SHA1 | Date | |
|---|---|---|---|
| f1a6574013 | |||
| 4bca6d3103 | |||
| 7c569a3d8c | |||
| 8e04ef9e2b | |||
| 53a330587b |
@@ -1,7 +1,7 @@
|
|||||||
# Configuration
|
# Configuration
|
||||||
|
|
||||||
This is the canonical reference for Notarius configuration. Configuration files
|
This is the canonical reference for Notarius configuration. Configuration files
|
||||||
are YAML and must declare version 3. They select pipelines and their modules;
|
are YAML and must declare version 4. They select pipelines and their modules;
|
||||||
the [CLI reference](cli.md) owns invocation syntax, and
|
the [CLI reference](cli.md) owns invocation syntax, and
|
||||||
[Operations](operations.md) owns run-state procedures.
|
[Operations](operations.md) owns run-state procedures.
|
||||||
|
|
||||||
@@ -45,8 +45,8 @@ other than **version** is optional.
|
|||||||
|
|
||||||
| Field | Type | Default | Rules |
|
| Field | Type | Default | Rules |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| **version** | integer | none | Required; must be 3. |
|
| **version** | integer | none | Required; must be 4. |
|
||||||
| **scriptorium** | object | none | Profile source configuration. |
|
| **promptkit** | object | none | Profile source configuration. |
|
||||||
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
|
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
|
||||||
| **concurrency** | object | see below | Global LLM and extraction limits. |
|
| **concurrency** | object | see below | Global LLM and extraction limits. |
|
||||||
| **output** | object | see below | Published output settings. |
|
| **output** | object | see below | Published output settings. |
|
||||||
@@ -69,9 +69,16 @@ Built-in defaults are:
|
|||||||
An empty cache directory in YAML deliberately selects the corresponding
|
An empty cache directory in YAML deliberately selects the corresponding
|
||||||
per-user root. An explicit empty output or debug directory is invalid.
|
per-user root. An explicit empty output or debug directory is invalid.
|
||||||
|
|
||||||
## Scriptorium Profiles
|
## PromptKit Profiles
|
||||||
|
|
||||||
The optional **scriptorium** object selects one source of profile definitions:
|
The optional **promptkit** object selects one source of profile definitions:
|
||||||
|
|
||||||
|
~~~yaml
|
||||||
|
version: 4
|
||||||
|
promptkit:
|
||||||
|
profile_dir: /path/to/profiles
|
||||||
|
# profile_file: /path/to/profiles.yml
|
||||||
|
~~~
|
||||||
|
|
||||||
| Field | Type | Rules |
|
| Field | Type | Rules |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
@@ -79,10 +86,19 @@ The optional **scriptorium** object selects one source of profile definitions:
|
|||||||
| **profile_file** | string | Non-empty profile file. |
|
| **profile_file** | string | Non-empty profile file. |
|
||||||
|
|
||||||
Set at most one of these fields. Profile IDs used by a binding must be available
|
Set at most one of these fields. Profile IDs used by a binding must be available
|
||||||
from the selected Scriptorium profile source when the pipeline is resolved.
|
from the selected PromptKit profile source when the pipeline is resolved.
|
||||||
Keep credentials out of this file: configure a profile to read its credential
|
Keep credentials out of this file: configure a profile to read its credential
|
||||||
from an environment variable, then set that environment variable only in the
|
from an environment variable, then set that environment variable only in the
|
||||||
run environment.
|
run environment. PromptKit owns the profile-file format; see the
|
||||||
|
[PromptKit upstream boundary](integrations/pkg-promptkit.md) for the pinned
|
||||||
|
package and canonical format reference.
|
||||||
|
|
||||||
|
## Migrating Version 3 Configuration
|
||||||
|
|
||||||
|
Version 3 files are not decoded or rewritten. Change **version: 3** to
|
||||||
|
**version: 4** and rename the top-level **scriptorium:** section to
|
||||||
|
**promptkit:**. Version 4 decoding is strict, so a remaining **scriptorium**
|
||||||
|
field is rejected as unknown.
|
||||||
|
|
||||||
## Operational Environment Variables
|
## Operational Environment Variables
|
||||||
|
|
||||||
@@ -199,7 +215,7 @@ extract:
|
|||||||
| Binding field | Type | Default | Rules |
|
| Binding field | Type | Default | Rules |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| **module** | string | none | Required for an object binding. Must be a registered compatible key. |
|
| **module** | string | none | Required for an object binding. Must be a registered compatible key. |
|
||||||
| **llm_profile** | string | none | Optional non-empty Scriptorium profile ID. |
|
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID. |
|
||||||
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. |
|
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. |
|
||||||
| **options** | object | none | Must satisfy the selected module. |
|
| **options** | object | none | Must satisfy the selected module. |
|
||||||
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
|
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ implemented component map.
|
|||||||
| Configuration loading, resolution, or user-visible configuration behavior | [Configuration Internals](internal/configuration.md) and [Configuration](config.md) | The internal guide owns loading and resolution mechanics; the reference owns the configuration contract. |
|
| Configuration loading, resolution, or user-visible configuration behavior | [Configuration Internals](internal/configuration.md) and [Configuration](config.md) | The internal guide owns loading and resolution mechanics; the reference owns the configuration contract. |
|
||||||
| Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
|
| Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
|
||||||
| Production modules or validators | [Module Internals](internal/modules.md), [D&D Module Internals](internal/dnd.md), and [D&D integration contracts](integrations/) | The generic guide owns extension mechanics, the D&D guide owns shared family conventions, and the contracts own durable output shapes. |
|
| Production modules or validators | [Module Internals](internal/modules.md), [D&D Module Internals](internal/dnd.md), and [D&D integration contracts](integrations/) | The generic guide owns extension mechanics, the D&D guide owns shared family conventions, and the contracts own durable output shapes. |
|
||||||
| LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and Scriptorium integration. |
|
| LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and PromptKit integration. |
|
||||||
| Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. |
|
| Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. |
|
||||||
| External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. |
|
| External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. |
|
||||||
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
|
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
|
||||||
|
|||||||
@@ -1,151 +1,41 @@
|
|||||||
# Package `promptkit`
|
# PromptKit Integration
|
||||||
|
|
||||||
Import path:
|
Notarius pins
|
||||||
|
[`gitea.maximumdirect.net/eric/promptkit` v0.1.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0)
|
||||||
|
as its in-process prompt engine. The upstream
|
||||||
|
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/consumers/pkg-promptkit.md)
|
||||||
|
owns the public engine API, and the upstream
|
||||||
|
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.1.0/docs/formats.md)
|
||||||
|
owns prompt, profile, and schema file contracts.
|
||||||
|
|
||||||
```go
|
## Supported Boundary
|
||||||
import "gitea.maximumdirect.net/eric/promptkit"
|
|
||||||
```
|
|
||||||
|
|
||||||
Package `promptkit` is the supported Go contract for in-process prompt
|
Notarius relies on the root `promptkit` package to:
|
||||||
preparation and execution. The declarations and their GoDoc in the
|
|
||||||
[root package](../../doc.go) own the exact API; this guide explains how the
|
|
||||||
pieces are used together. The [framework format reference](../formats.md) owns
|
|
||||||
prompt, profile, and schema file contracts.
|
|
||||||
|
|
||||||
## Engine Construction And Sources
|
- construct an `Engine` with filesystem-backed prompt, schema, and optional
|
||||||
|
profile sources;
|
||||||
|
- prepare and run a `RunRequest` with named inline artifacts, variables,
|
||||||
|
metadata, prompt identity, and profile selection;
|
||||||
|
- return rendered debug material, validated structured output, selected
|
||||||
|
profile and model metadata, and token usage;
|
||||||
|
- distinguish structured-output validation failure from execution failure; and
|
||||||
|
- identify a missing explicit profile through `ErrProfileNotFound`.
|
||||||
|
|
||||||
Construct an engine with [`NewEngine`, `Config`, and
|
Notarius does not use PromptKit's optional `ArtifactReader`. It materializes
|
||||||
`Option`](../../engine.go). `PromptDir` is required unless a prompt source
|
source and reference content itself and supplies owned inline artifacts at the
|
||||||
option is supplied. `ProfileDir` optionally overlays built-in profiles, and an
|
adapter boundary. It also retains responsibility for pipeline retries,
|
||||||
empty `SchemaDir` uses the current directory. `Timeout` is the transport-wide
|
scheduling, debug persistence, redaction, profile provenance, and conversion
|
||||||
safety cap for the built-in OpenAI-compatible client. An optional `HTTPClient`
|
from private model responses into durable domain artifacts.
|
||||||
is cloned; its positive timeout takes precedence.
|
|
||||||
|
|
||||||
Nil options are ignored. Invalid construction, including a nil injected client
|
## Notarius Ownership
|
||||||
or artifact reader, returns an error matching `ErrInvalidConfig`.
|
|
||||||
|
|
||||||
The [source options](../../engine.go) replace their matching directory source:
|
[LLM Runtime Internals](../internal/llm.md) describes how Notarius mounts
|
||||||
|
module assets, maps its transport-neutral completion contract, prepares and
|
||||||
|
executes requests, validates output, records provenance, captures debug
|
||||||
|
material, redacts errors, and preserves timeout ownership.
|
||||||
|
[Configuration](../config.md#promptkit-profiles) defines how a Notarius
|
||||||
|
configuration selects one PromptKit profile source.
|
||||||
|
|
||||||
- `WithPromptFS` and `WithPromptFile` select prompt definitions;
|
PromptKit API or format changes outside this boundary are not implicitly
|
||||||
- `WithProfileFS` and `WithProfileFile` overlay built-in profiles;
|
supported. Updating the pinned version requires reviewing the adapter and
|
||||||
- `WithProfiles` adds in-memory profiles ahead of file and built-in profiles;
|
profile/configuration contracts against the upstream documentation.
|
||||||
- `WithSchemaFS` and `WithSchemaFile` select JSON Schema documents;
|
|
||||||
- `WithLLMClient` replaces the built-in model client; and
|
|
||||||
- `WithArtifactReader` replaces the default reader for every input.
|
|
||||||
|
|
||||||
Source selection, path resolution, strict decoding, profile overlays, and
|
|
||||||
file-to-request precedence are defined in the
|
|
||||||
[framework format reference](../formats.md).
|
|
||||||
|
|
||||||
Per-generation timeout values from profiles or requests are independent of
|
|
||||||
the transport cap and caller context. An explicit request value of zero
|
|
||||||
disables only the per-generation deadline. The
|
|
||||||
[outbound integration contract](../integrations/openai-compatible-chat.md#timeout-and-cancellation)
|
|
||||||
defines the complete timeout layering.
|
|
||||||
|
|
||||||
## Preparation And Execution
|
|
||||||
|
|
||||||
[`Engine.Prepare` and `Engine.Run`](../../engine.go) accept the public
|
|
||||||
[`RunRequest`](../../types.go). `Prepare` resolves the prompt, profile, input
|
|
||||||
artifacts, validation contract, and rendered messages without calling an LLM.
|
|
||||||
`Run` performs the same preparation, calls the configured client, and validates
|
|
||||||
the generated content. The maintained
|
|
||||||
[offline preparation example](../../examples/go-library/prepare/main.go)
|
|
||||||
provides a complete runnable workflow using a prompt file, in-memory profile,
|
|
||||||
and inline input.
|
|
||||||
|
|
||||||
[`PreparedRun` and `RunResult`](../../types.go) expose copied public values.
|
|
||||||
Preparation returns effective settings, hashes, rendered messages, selected
|
|
||||||
profile, structured-output information, and timing without resolved secrets or
|
|
||||||
model output. Execution adds the generated artifact and raw output, validation
|
|
||||||
state, model metadata, usage, run ID, and duration.
|
|
||||||
|
|
||||||
A generated-content validation failure returns a result with
|
|
||||||
`Validation.Status == ValidationFailed`. An inability to perform validation
|
|
||||||
returns an error matching `ErrValidation`.
|
|
||||||
|
|
||||||
## Requests, Inputs, And Overrides
|
|
||||||
|
|
||||||
The [request and value declarations](../../types.go) own the available fields,
|
|
||||||
serialized constants, and result shapes. Use `File`, `Inline`, or
|
|
||||||
`InlineWithURI` to construct artifact references. The
|
|
||||||
[framework format reference](../formats.md) defines declared inputs, template
|
|
||||||
references, output contracts, and the relationship between file values and
|
|
||||||
request overrides.
|
|
||||||
|
|
||||||
`ExecutionTargetOverride` uses pointers for numeric settings so an explicit
|
|
||||||
zero remains distinct from no override. `ExtraParams` accepts JSON-compatible
|
|
||||||
strings, booleans, finite numbers, string-keyed objects, arrays or slices, and
|
|
||||||
nil. Unsupported values, non-string map keys, non-finite numbers, and cycles
|
|
||||||
match `ErrInvalidConfig` in profiles or `ErrInvalidRequest` in request
|
|
||||||
overrides.
|
|
||||||
|
|
||||||
Returned requests, profiles, prepared values, results, artifacts, maps, and
|
|
||||||
slices are isolated from internal engine state. Consumers and injected
|
|
||||||
extensions should not retain or mutate values owned by another caller.
|
|
||||||
|
|
||||||
## Profiles And Credentials
|
|
||||||
|
|
||||||
[`OpenAICompatibleProfile`](../../profiles.go) constructs an ordinary
|
|
||||||
in-memory profile for an OpenAI-compatible chat-completions endpoint.
|
|
||||||
`WithProfiles` rejects duplicate IDs in one call and gives in-memory profiles
|
|
||||||
precedence over explicit file sources and built-ins.
|
|
||||||
|
|
||||||
Raw API keys do not belong in profiles. File-backed profiles may name an
|
|
||||||
environment variable, while an in-memory profile can require a request key.
|
|
||||||
A direct `RunRequest.APIKey` is request-scoped and takes precedence over an
|
|
||||||
environment lookup for the built-in client. Profile fields, ranges, built-ins,
|
|
||||||
precedence, and credential rules are owned by the
|
|
||||||
[framework format reference](../formats.md).
|
|
||||||
|
|
||||||
API keys are excluded from JSON, prepared values, and results. The public
|
|
||||||
`String` and `GoString` methods report only whether a direct key is present.
|
|
||||||
Avoid reflection-based dumps of request structs, which can bypass that
|
|
||||||
redaction.
|
|
||||||
|
|
||||||
## Extension Interfaces
|
|
||||||
|
|
||||||
The [`LLMClient`, `GenerateRequest`, and
|
|
||||||
`GenerateResponse`](../../types.go) boundary lets a consumer replace model
|
|
||||||
generation. Injected clients receive copied rendered messages, effective
|
|
||||||
settings, explicit numeric-setting presence, structured-output constraints,
|
|
||||||
and the request-scoped key. They return generated content and token usage.
|
|
||||||
|
|
||||||
The [`ArtifactReader`](../../types.go) boundary replaces the default inline and
|
|
||||||
file reader for every input. Readers provide artifact content and metadata; the
|
|
||||||
engine fills an empty artifact name from the input-map key. A reader error
|
|
||||||
matches `ErrArtifactLoad` while preserving the original identity for
|
|
||||||
`errors.Is`. A nil artifact with a nil error is also an artifact-load failure.
|
|
||||||
|
|
||||||
Extensions should honor context cancellation and avoid logging raw prompts,
|
|
||||||
artifacts, or credentials.
|
|
||||||
|
|
||||||
## Errors
|
|
||||||
|
|
||||||
The [public error declarations](../../engine.go) and
|
|
||||||
[mapping](../../errors.go) preserve these sentinel checks through `errors.Is`:
|
|
||||||
|
|
||||||
- `ErrInvalidConfig`
|
|
||||||
- `ErrInvalidRequest`
|
|
||||||
- `ErrPromptNotFound`
|
|
||||||
- `ErrProfileNotFound`
|
|
||||||
- `ErrProfileRequired`
|
|
||||||
- `ErrPromptLoad`
|
|
||||||
- `ErrProfileLoad`
|
|
||||||
- `ErrAPIKeyEnvMissing`
|
|
||||||
- `ErrArtifactLoad`
|
|
||||||
- `ErrPromptRender`
|
|
||||||
- `ErrLLMGenerate`
|
|
||||||
- `ErrValidation`
|
|
||||||
|
|
||||||
`ErrProfileRequired` and `ErrAPIKeyEnvMissing` also match
|
|
||||||
`ErrInvalidRequest`, allowing either broad request handling or a specific
|
|
||||||
condition. Wrapped collaborator errors retain their identity where the public
|
|
||||||
contract promises it.
|
|
||||||
|
|
||||||
## Consumer Boundary
|
|
||||||
|
|
||||||
Promptkit is an importable library. It does not own a command, inbound HTTP
|
|
||||||
API, process configuration, or deployment policy. Scriptorium is one
|
|
||||||
downstream application that maps this root package contract into those
|
|
||||||
application concerns.
|
|
||||||
|
|||||||
@@ -39,8 +39,10 @@ in [Configuration Internals](configuration.md).
|
|||||||
Configuration validation without a selected pipeline checks structural
|
Configuration validation without a selected pipeline checks structural
|
||||||
configuration only. Validation with a selected pipeline also builds the
|
configuration only. Validation with a selected pipeline also builds the
|
||||||
effective catalog, resolves the pipeline, and verifies explicitly selected
|
effective catalog, resolves the pipeline, and verifies explicitly selected
|
||||||
Scriptorium profiles. Pipeline listing validates configuration before returning
|
PromptKit profiles. Each explicit binding or validator profile is prepared
|
||||||
normalized, sorted identifiers.
|
against the configured PromptKit source without performing generation, so an
|
||||||
|
unknown profile fails before pipeline preparation. Pipeline listing validates
|
||||||
|
configuration before returning normalized, sorted identifiers.
|
||||||
|
|
||||||
## Production Composition
|
## Production Composition
|
||||||
|
|
||||||
@@ -51,12 +53,13 @@ catalog used for resolution and the concrete constructors used for preparation.
|
|||||||
Tests may provide a catalog or registries instead; production code must not
|
Tests may provide a catalog or registries instead; production code must not
|
||||||
silently merge an injected partial catalog with production registrations.
|
silently merge an injected partial catalog with production registrations.
|
||||||
|
|
||||||
The production LLM factory builds the Scriptorium-backed client from resolved
|
The production LLM factory builds one PromptKit-backed client from the resolved
|
||||||
configuration, creates one scheduler from the effective global LLM limit, and
|
**promptkit.profile_dir** or **promptkit.profile_file** source, attaches the
|
||||||
wraps the client before it reaches modules. Registration and LLM construction
|
profile-provenance recorder, creates one scheduler from the effective global
|
||||||
errors are returned before a pipeline is prepared. Concrete module keys and
|
LLM limit, and wraps the client before it reaches modules. Registration and LLM
|
||||||
validator chains are public configuration choices and remain documented in
|
construction errors are returned before a pipeline is prepared. Configuration
|
||||||
[Configuration](../config.md).
|
field definitions remain in [Configuration](../config.md#promptkit-profiles);
|
||||||
|
the adapter mechanics remain in [LLM Runtime](llm.md).
|
||||||
|
|
||||||
## Run Orchestration
|
## Run Orchestration
|
||||||
|
|
||||||
@@ -68,7 +71,7 @@ handoff:
|
|||||||
2. create and validate a safe run identity, then allocate a debug bundle only
|
2. create and validate a safe run identity, then allocate a debug bundle only
|
||||||
when requested;
|
when requested;
|
||||||
3. build the effective catalog, resolve requested reference changes, resolve
|
3. build the effective catalog, resolve requested reference changes, resolve
|
||||||
the effective pipeline, and verify explicit Scriptorium profiles;
|
the effective pipeline, and verify explicit PromptKit profiles;
|
||||||
4. materialize external or generated references and record redacted invocation
|
4. materialize external or generated references and record redacted invocation
|
||||||
and resolution provenance when debug capture is enabled;
|
and resolution provenance when debug capture is enabled;
|
||||||
5. construct registries, the scheduled LLM client, prepared modules, and the
|
5. construct registries, the scheduled LLM client, prepared modules, and the
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
# LLM Runtime Internals
|
# LLM Runtime Internals
|
||||||
|
|
||||||
`internal/framework/llm` is Notarius’s provider-independent structured
|
`internal/framework/llm` is Notarius’s provider-independent structured
|
||||||
completion boundary. It adapts framework requests to Scriptorium, bounds
|
completion boundary. It adapts framework requests to PromptKit, bounds
|
||||||
provider calls, assembles registered prompt and schema assets, records selected
|
provider calls, assembles registered prompt and schema assets, records selected
|
||||||
profiles, and redacts provider errors. The architectural boundary is defined in
|
profiles, and redacts provider errors. The architectural boundary is defined in
|
||||||
[Architecture](../policy/architecture.md#llm-boundary); profile sources,
|
[Architecture](../policy/architecture.md#llm-boundary); profile sources,
|
||||||
credentials, and concurrency settings belong in
|
credentials, and concurrency settings belong in
|
||||||
[Configuration](../config.md#scriptorium-profiles) and
|
[Configuration](../config.md#promptkit-profiles) and
|
||||||
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||||||
|
|
||||||
## Structured Completion Boundary
|
## Structured Completion Boundary
|
||||||
@@ -24,22 +24,26 @@ adapter does not own source evidence, artifact conversion, normalization, or
|
|||||||
durable schemas. Those responsibilities remain with the module and its
|
durable schemas. Those responsibilities remain with the module and its
|
||||||
[integration contract](../integrations/).
|
[integration contract](../integrations/).
|
||||||
|
|
||||||
`ScriptoriumClient` validates the request target and prompt identity, maps each
|
`PromptKitClient` validates the request target and prompt identity, maps each
|
||||||
named material to a Scriptorium inline artifact while preserving its origin URI,
|
named material to a PromptKit inline artifact while preserving its origin URI,
|
||||||
forwards session and profile selection, then prepares and runs the prompt. It
|
maps the request session to the existing `session_id` prompt variable, forwards
|
||||||
returns Scriptorium’s validated raw bytes rather than re-encoding the decoded
|
profile selection, then prepares and runs the prompt. PromptKit v0.1.0 has no
|
||||||
target. An empty optional material is represented as one space so its named
|
direct request-level session field. The adapter returns PromptKit’s validated
|
||||||
input is retained by Scriptorium.
|
raw bytes rather than re-encoding the decoded target. An empty optional
|
||||||
|
material is represented as one space so its named input is retained by
|
||||||
|
PromptKit.
|
||||||
|
|
||||||
An empty request profile lets the prompt select its configured default. The CLI
|
An empty request profile lets the prompt select its configured default. The CLI
|
||||||
prepares every explicitly selected binding profile before a run begins, so a
|
prepares every explicitly selected binding profile before a run begins, so a
|
||||||
missing explicit profile fails before stage execution. Calls record the profile
|
missing explicit profile fails before stage execution. Calls record the profile
|
||||||
actually selected by Scriptorium; the recorder deduplicates non-secret profile
|
actually selected by PromptKit; the recorder deduplicates non-secret profile
|
||||||
identity, provider, and model values for manifest use.
|
identity, provider, and model values for manifest use. Successful completion
|
||||||
|
responses and recorded profile manifests identify the adapter provider as
|
||||||
|
`promptkit`.
|
||||||
|
|
||||||
## Shared Provider-Call Limit
|
## Shared Provider-Call Limit
|
||||||
|
|
||||||
Production construction creates one Scriptorium client and wraps it in one
|
Production construction creates one PromptKit client and wraps it in one
|
||||||
scheduled client. The scheduler has a fixed, positive permit limit, serves
|
scheduled client. The scheduler has a fixed, positive permit limit, serves
|
||||||
queued calls in FIFO order, and removes a queued call when its context is
|
queued calls in FIFO order, and removes a queued call when its context is
|
||||||
cancelled. A granted permit is released exactly once on every completion path.
|
cancelled. A granted permit is released exactly once on every completion path.
|
||||||
@@ -54,7 +58,7 @@ and its effective default are owned by
|
|||||||
## Prompt And Schema Assets
|
## Prompt And Schema Assets
|
||||||
|
|
||||||
An `AssetRegistry` collects prompt and schema filesystems from production module
|
An `AssetRegistry` collects prompt and schema filesystems from production module
|
||||||
families. It flattens registered roots into the Scriptorium filesystems and
|
families. It flattens registered roots into the PromptKit filesystems and
|
||||||
rejects invalid roots, unreadable assets, duplicate paths, and missing prompt
|
rejects invalid roots, unreadable assets, duplicate paths, and missing prompt
|
||||||
or schema files during preparation. The framework’s `promptfs` helper combines
|
or schema files during preparation. The framework’s `promptfs` helper combines
|
||||||
module-owned prompt files with reusable domain fragments without making the
|
module-owned prompt files with reusable domain fragments without making the
|
||||||
@@ -95,14 +99,14 @@ meaningful input placement and cache controls of the prompt being changed.
|
|||||||
|
|
||||||
## Validation, Repair, And Retries
|
## Validation, Repair, And Retries
|
||||||
|
|
||||||
Scriptorium performs prompt rendering, provider execution, and the prompt’s
|
PromptKit performs prompt rendering, provider execution, and the prompt’s
|
||||||
structured-output validation. The adapter reports an empty result, validation
|
structured-output validation. The adapter reports an empty result, validation
|
||||||
failure, empty structured body, or decode failure as
|
failure, empty structured body, or decode failure as
|
||||||
`ErrInvalidStructuredOutput`, while retaining the returned raw bytes and debug
|
`ErrInvalidStructuredOutput`, while retaining the returned raw bytes and debug
|
||||||
material when they exist. Provider failures remain operational errors rather
|
material when they exist. Provider failures remain operational errors rather
|
||||||
than output-validation failures.
|
than output-validation failures.
|
||||||
|
|
||||||
Prompt-declared repair is executed within Scriptorium’s structured-output flow.
|
Prompt-declared repair is executed within PromptKit’s structured-output flow.
|
||||||
The current production D&D prompt manifests set repair attempts to zero. That
|
The current production D&D prompt manifests set repair attempts to zero. That
|
||||||
setting does not replace pipeline retry behavior: a binding’s configured retry
|
setting does not replace pipeline retry behavior: a binding’s configured retry
|
||||||
count reruns its stage attempt after an error or rejection, and an exhausted
|
count reruns its stage attempt after an error or rejection, and an exhausted
|
||||||
@@ -111,6 +115,22 @@ attempt lifecycle, validation chains, and retry diagnostics; see
|
|||||||
[Pipeline Internals](pipeline.md#validation-retries-and-output) and the
|
[Pipeline Internals](pipeline.md#validation-retries-and-output) and the
|
||||||
[binding reference](../config.md#module-bindings-and-validators).
|
[binding reference](../config.md#module-bindings-and-validators).
|
||||||
|
|
||||||
|
## Timeout Ownership
|
||||||
|
|
||||||
|
The caller context remains the outer cancellation authority. PromptKit applies
|
||||||
|
a positive effective generation timeout as an inner request deadline; an
|
||||||
|
explicit zero disables only that generation deadline. The HTTP client timeout
|
||||||
|
is a separate transport-wide cap. Notarius forwards the caller context and
|
||||||
|
does not install another timeout wrapper around PromptKit.
|
||||||
|
|
||||||
|
The selected PromptKit profile owns generation settings. Notarius binding
|
||||||
|
retries remain outside the adapter and repeat the complete module operation
|
||||||
|
and validation chain. PromptKit v0.1.0 does not add a provider retry loop.
|
||||||
|
Operator-facing behavior is summarized in
|
||||||
|
[Operations](../operations.md#operational-limits), and the pinned upstream
|
||||||
|
contract is identified in
|
||||||
|
[PromptKit Integration](../integrations/pkg-promptkit.md).
|
||||||
|
|
||||||
## Observability And Redaction
|
## Observability And Redaction
|
||||||
|
|
||||||
When debug recording is enabled, the pipeline decorates the shared client. The
|
When debug recording is enabled, the pipeline decorates the shared client. The
|
||||||
|
|||||||
@@ -207,9 +207,19 @@ or automatic cleanup command.
|
|||||||
|
|
||||||
## Operational Limits
|
## Operational Limits
|
||||||
|
|
||||||
Provider retries and timeouts are supplied by the selected Scriptorium profile.
|
Provider execution settings and the generation timeout come from the selected
|
||||||
Module retry settings and concurrency limits are configuration contracts; see
|
PromptKit profile. PromptKit v0.1.0 does not add a provider retry loop;
|
||||||
[module bindings](config.md#module-bindings-and-validators) and
|
Notarius binding retries rerun the complete module operation and validation
|
||||||
|
chain as defined by [module bindings](config.md#module-bindings-and-validators).
|
||||||
|
|
||||||
|
Timeouts are layered. Caller cancellation is the outer authority. A positive
|
||||||
|
effective generation timeout adds an inner request deadline, while zero
|
||||||
|
disables only that generation deadline. The HTTP client timeout remains a
|
||||||
|
transport-wide cap. Notarius does not add another timeout around PromptKit.
|
||||||
|
The pinned upstream boundary and profile-format links are in
|
||||||
|
[PromptKit Integration](integrations/pkg-promptkit.md).
|
||||||
|
|
||||||
|
Concurrency limits are configuration contracts; see
|
||||||
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
|
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
|
||||||
limits and actual provider-call limits are independent. Notarius writes local
|
limits and actual provider-call limits are independent. Notarius writes local
|
||||||
filesystem state only; remote storage, archival, and retention automation are
|
filesystem state only; remote storage, archival, and retention automation are
|
||||||
|
|||||||
@@ -95,9 +95,9 @@ safety checks, and deterministic application of accepted changes.
|
|||||||
|
|
||||||
### Native Session Propagation
|
### Native Session Propagation
|
||||||
|
|
||||||
- Once upstream Scriptorium exposes a session identifier on its run request,
|
- Once upstream PromptKit exposes a direct request-level session identifier,
|
||||||
propagate the existing `StructuredCompletionRequest.SessionID` through the
|
propagate the existing `StructuredCompletionRequest.SessionID` through the
|
||||||
Scriptorium adapter's native session field.
|
PromptKit adapter's native session field.
|
||||||
- Preserve the current `--session-id` invocation contract and its run-wide
|
- Preserve the current `--session-id` invocation contract and its run-wide
|
||||||
propagation to every prompt-facing module and validator. Do not introduce a
|
propagation to every prompt-facing module and validator. Do not introduce a
|
||||||
second session configuration surface.
|
second session configuration surface.
|
||||||
@@ -110,10 +110,11 @@ safety checks, and deterministic application of accepted changes.
|
|||||||
concurrent-run isolation, and unsupported-provider behavior once the
|
concurrent-run isolation, and unsupported-provider behavior once the
|
||||||
upstream contract is available.
|
upstream contract is available.
|
||||||
|
|
||||||
This work is blocked on native session support in the upstream Scriptorium
|
This work is blocked because PromptKit v0.1.0 does not expose the required
|
||||||
package. Notarius already carries a run-scoped session ID through its CLI,
|
direct request-level session field. Notarius already carries a run-scoped
|
||||||
pipeline requests, checkpoint identity, and prompt variables; the missing
|
session ID through its CLI, pipeline requests, checkpoint identity, and a
|
||||||
capability is native propagation across the LLM adapter boundary.
|
`session_id` prompt variable; that prompt-variable propagation is not native
|
||||||
|
provider session support.
|
||||||
|
|
||||||
## Further Reference Evolution
|
## Further Reference Evolution
|
||||||
|
|
||||||
|
|||||||
@@ -2,7 +2,10 @@
|
|||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
Planned.
|
In progress. The dependency, framework adapter, version 4 PromptKit
|
||||||
|
configuration migration, provider-neutral module prompt-asset support,
|
||||||
|
provenance alignment, and canonical documentation are implemented; final
|
||||||
|
repository verification is still planned.
|
||||||
|
|
||||||
## Objective
|
## Objective
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
version: 3
|
version: 4
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 2
|
total_llm: 2
|
||||||
stage_workers:
|
stage_workers:
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
version: 3
|
version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
|
|||||||
2
go.mod
2
go.mod
@@ -3,7 +3,7 @@ module gitea.maximumdirect.net/eric/notarius
|
|||||||
go 1.25.5
|
go 1.25.5
|
||||||
|
|
||||||
require (
|
require (
|
||||||
gitea.maximumdirect.net/eric/scriptorium v0.11.1
|
gitea.maximumdirect.net/eric/promptkit v0.1.0
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
||||||
gopkg.in/yaml.v3 v3.0.1
|
gopkg.in/yaml.v3 v3.0.1
|
||||||
)
|
)
|
||||||
|
|||||||
8
go.sum
8
go.sum
@@ -1,13 +1,9 @@
|
|||||||
gitea.maximumdirect.net/eric/scriptorium v0.11.0 h1:rjvbt9FTaWHxYlHq7QlUzmMVUt3QdbTmeCkmH81N//o=
|
gitea.maximumdirect.net/eric/promptkit v0.1.0 h1:vuKeBxkiY8E54LRFbLQFjlJJCiOfMvB1++DYBCrD/ug=
|
||||||
gitea.maximumdirect.net/eric/scriptorium v0.11.0/go.mod h1:FQ5lEuNxmrQyNgIomkpZdxvfTC0jWjbXYuq3tbJWF64=
|
gitea.maximumdirect.net/eric/promptkit v0.1.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||||
gitea.maximumdirect.net/eric/scriptorium v0.11.1 h1:zBKtB3+fP8FcHGI8DJD99CiTL6crAGitBhWtE+xYJHc=
|
|
||||||
gitea.maximumdirect.net/eric/scriptorium v0.11.1/go.mod h1:FQ5lEuNxmrQyNgIomkpZdxvfTC0jWjbXYuq3tbJWF64=
|
|
||||||
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
||||||
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
|
||||||
golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
|
|
||||||
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
|
||||||
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
||||||
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||||
|
|||||||
@@ -168,14 +168,14 @@ func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID
|
|||||||
return nil, nil, fmt.Errorf("production asset registry must not be nil")
|
return nil, nil, fmt.Errorf("production asset registry must not be nil")
|
||||||
}
|
}
|
||||||
recorder := llm.NewLLMProfileRecorder()
|
recorder := llm.NewLLMProfileRecorder()
|
||||||
client, err := llm.NewScriptoriumClient(llm.ScriptoriumClientConfig{
|
client, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{
|
||||||
ProfileDir: cfg.Scriptorium.ProfileDir,
|
ProfileDir: cfg.PromptKit.ProfileDir,
|
||||||
ProfileFile: cfg.Scriptorium.ProfileFile,
|
ProfileFile: cfg.PromptKit.ProfileFile,
|
||||||
Assets: assets,
|
Assets: assets,
|
||||||
Recorder: recorder,
|
Recorder: recorder,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, nil, fmt.Errorf("create Scriptorium-backed LLM client: %w", err)
|
return nil, nil, fmt.Errorf("create PromptKit-backed LLM client: %w", err)
|
||||||
}
|
}
|
||||||
scheduler, err := llm.NewScheduler(cfg.Concurrency.TotalLLM)
|
scheduler, err := llm.NewScheduler(cfg.Concurrency.TotalLLM)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -214,13 +214,13 @@ func commandContractOptionsWithLookup(t *testing.T, lookup func(string) (string,
|
|||||||
|
|
||||||
func writeCommandConfig(t *testing.T, firstID, secondID string) string {
|
func writeCommandConfig(t *testing.T, firstID, secondID string) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
content := fmt.Sprintf("version: 3\npipelines:\n %q:\n input: seriatim\n %q:\n input: seriatim\n", firstID, secondID)
|
content := fmt.Sprintf("version: 4\npipelines:\n %q:\n input: seriatim\n %q:\n input: seriatim\n", firstID, secondID)
|
||||||
return writeCommandConfigContent(t, content)
|
return writeCommandConfigContent(t, content)
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeResolvableCommandConfig(t *testing.T) string {
|
func writeResolvableCommandConfig(t *testing.T) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
return writeCommandConfigContent(t, `version: 3
|
return writeCommandConfigContent(t, `version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
demo:
|
demo:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
|
|||||||
@@ -6,8 +6,10 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"reflect"
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
@@ -28,7 +30,7 @@ func TestProductionSceneDescriptionWorkflow(t *testing.T) {
|
|||||||
Output: pipeline.Binding("json"),
|
Output: pipeline.Binding("json"),
|
||||||
Artifacts: map[string]pipeline.ArtifactLaneProfile{
|
Artifacts: map[string]pipeline.ArtifactLaneProfile{
|
||||||
"scene-descriptions": {
|
"scene-descriptions": {
|
||||||
Extract: pipeline.Binding(sceneextract.Key),
|
Extract: pipeline.ModuleBinding{Module: sceneextract.Key, LLMProfile: "scene-description-profile"},
|
||||||
Normalize: pipeline.Binding(scenenormalize.Key),
|
Normalize: pipeline.Binding(scenenormalize.Key),
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
@@ -45,7 +47,8 @@ func TestProductionSceneDescriptionWorkflow(t *testing.T) {
|
|||||||
t.Fatalf("resolved references = %#v / %#v, want no generated or required references", lane.ExtractReferences, lane.NormalizeReferences)
|
t.Fatalf("resolved references = %#v / %#v, want no generated or required references", lane.ExtractReferences, lane.NormalizeReferences)
|
||||||
}
|
}
|
||||||
|
|
||||||
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, components.registries, pipeline.ModuleDependencies{LLM: sceneDescriptionLLM{}})
|
llmClient := &sceneDescriptionLLM{}
|
||||||
|
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, components.registries, pipeline.ModuleDependencies{LLM: llmClient})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Prepare() error = %v", err)
|
t.Fatalf("Prepare() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -60,6 +63,14 @@ func TestProductionSceneDescriptionWorkflow(t *testing.T) {
|
|||||||
if output.Manifest.ValidationStatus != "approved" || len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 1 {
|
if output.Manifest.ValidationStatus != "approved" || len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 1 {
|
||||||
t.Fatalf("run output = %#v, want one approved normalized artifact", output)
|
t.Fatalf("run output = %#v, want one approved normalized artifact", output)
|
||||||
}
|
}
|
||||||
|
wantProfiles := []artifacts.LLMProfileManifest{{
|
||||||
|
ID: "scene-description-profile",
|
||||||
|
Provider: "promptkit",
|
||||||
|
Model: "deterministic",
|
||||||
|
}}
|
||||||
|
if !reflect.DeepEqual(output.Manifest.LLMProfiles, wantProfiles) {
|
||||||
|
t.Fatalf("manifest LLM profiles = %#v, want %#v", output.Manifest.LLMProfiles, wantProfiles)
|
||||||
|
}
|
||||||
normalizedOutput := output.NormalizeOutputs[0]
|
normalizedOutput := output.NormalizeOutputs[0]
|
||||||
if normalizedOutput.NormalizerKey != scenenormalize.Key || normalizedOutput.Artifact.Kind != dnd.SceneDescriptionListKind || normalizedOutput.Artifact.Schema.ID != scenecodec.SchemaID || normalizedOutput.Artifact.Schema.Name != scenecodec.SchemaName || normalizedOutput.Artifact.Schema.Version != scenecodec.SchemaVersion {
|
if normalizedOutput.NormalizerKey != scenenormalize.Key || normalizedOutput.Artifact.Kind != dnd.SceneDescriptionListKind || normalizedOutput.Artifact.Schema.ID != scenecodec.SchemaID || normalizedOutput.Artifact.Schema.Name != scenecodec.SchemaName || normalizedOutput.Artifact.Schema.Version != scenecodec.SchemaVersion {
|
||||||
t.Fatalf("normalized output = %#v, want registered durable scene-description schema", normalizedOutput)
|
t.Fatalf("normalized output = %#v, want registered durable scene-description schema", normalizedOutput)
|
||||||
@@ -85,9 +96,12 @@ func TestProductionSceneDescriptionWorkflow(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
type sceneDescriptionLLM struct{}
|
type sceneDescriptionLLM struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
profile *artifacts.LLMProfileManifest
|
||||||
|
}
|
||||||
|
|
||||||
func (sceneDescriptionLLM) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
func (client *sceneDescriptionLLM) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
if err := ctx.Err(); err != nil {
|
if err := ctx.Err(); err != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, err
|
return contracts.StructuredCompletionResponse{}, err
|
||||||
}
|
}
|
||||||
@@ -107,5 +121,27 @@ func (sceneDescriptionLLM) CompleteStructured(ctx context.Context, req contracts
|
|||||||
if err := json.Unmarshal([]byte(content), out); err != nil {
|
if err := json.Unmarshal([]byte(content), out); err != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate structured response: %w", err)
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate structured response: %w", err)
|
||||||
}
|
}
|
||||||
return contracts.StructuredCompletionResponse{Content: []byte(content), Provider: "test", Model: "deterministic", ProfileID: req.ProfileID}, nil
|
profile := artifacts.LLMProfileManifest{
|
||||||
|
ID: req.ProfileID,
|
||||||
|
Provider: "promptkit",
|
||||||
|
Model: "deterministic",
|
||||||
|
}
|
||||||
|
client.mu.Lock()
|
||||||
|
client.profile = &profile
|
||||||
|
client.mu.Unlock()
|
||||||
|
return contracts.StructuredCompletionResponse{
|
||||||
|
Content: []byte(content),
|
||||||
|
Provider: profile.Provider,
|
||||||
|
Model: profile.Model,
|
||||||
|
ProfileID: profile.ID,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (client *sceneDescriptionLLM) LLMProfileManifests() []artifacts.LLMProfileManifest {
|
||||||
|
client.mu.Lock()
|
||||||
|
defer client.mu.Unlock()
|
||||||
|
if client.profile == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return []artifacts.LLMProfileManifest{*client.profile}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ func TestOversizedNPCRegistryFailsBeforeRuntimeAndCheckpointConstruction(t *test
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
checkpointRoot := filepath.Join(t.TempDir(), "checkpoints")
|
checkpointRoot := filepath.Join(t.TempDir(), "checkpoints")
|
||||||
content := fmt.Sprintf(`version: 3
|
content := fmt.Sprintf(`version: 4
|
||||||
cache:
|
cache:
|
||||||
chunk_plans:
|
chunk_plans:
|
||||||
mode: bypass
|
mode: bypass
|
||||||
|
|||||||
@@ -583,7 +583,7 @@ func maintainedExampleFiles(t *testing.T) []maintainedExample {
|
|||||||
|
|
||||||
func productionSpellCatalogContractConfig(t *testing.T) string {
|
func productionSpellCatalogContractConfig(t *testing.T) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
return fmt.Sprintf(`version: 3
|
return fmt.Sprintf(`version: 4
|
||||||
cache:
|
cache:
|
||||||
chunk_plans:
|
chunk_plans:
|
||||||
mode: bypass
|
mode: bypass
|
||||||
@@ -680,7 +680,7 @@ func productionRunOptions(t *testing.T, fake *productionFakeLLMClient) Options {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func productionRunConfig(outputRoot, chunkModule string) string {
|
func productionRunConfig(outputRoot, chunkModule string) string {
|
||||||
return fmt.Sprintf(`version: 3
|
return fmt.Sprintf(`version: 4
|
||||||
output:
|
output:
|
||||||
directory: %q
|
directory: %q
|
||||||
cache:
|
cache:
|
||||||
|
|||||||
68
internal/cli/promptkit_profiles.go
Normal file
68
internal/cli/promptkit_profiles.go
Normal file
@@ -0,0 +1,68 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"testing/fstest"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
|
)
|
||||||
|
|
||||||
|
const profileCheckPromptID = "notarius.profile.check"
|
||||||
|
|
||||||
|
var profileCheckPromptFS = fstest.MapFS{
|
||||||
|
"prompts/profile-check.yaml": &fstest.MapFile{Data: []byte(`id: notarius.profile.check
|
||||||
|
version: "1.0.0"
|
||||||
|
default_profile: mistral-small-3
|
||||||
|
inputs:
|
||||||
|
- name: transcript
|
||||||
|
required: true
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "{{input \"transcript\"}}"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
repair_attempts: 0
|
||||||
|
`)},
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateExplicitPromptKitProfiles(ctx context.Context, cfg config.Config, profileIDs []string) error {
|
||||||
|
if len(profileIDs) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
engine, err := newProfileValidationEngine(cfg)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("load PromptKit profiles: %w", err)
|
||||||
|
}
|
||||||
|
for _, profileID := range profileIDs {
|
||||||
|
if _, err := engine.Prepare(ctx, promptkit.RunRequest{
|
||||||
|
PromptID: profileCheckPromptID,
|
||||||
|
ProfileID: profileID,
|
||||||
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
|
"transcript": promptkit.Inline("profile check"),
|
||||||
|
},
|
||||||
|
}); err != nil {
|
||||||
|
if errors.Is(err, promptkit.ErrProfileNotFound) {
|
||||||
|
return fmt.Errorf("PromptKit profile %q is not configured", profileID)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("validate PromptKit profile %q: %w", profileID, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func newProfileValidationEngine(cfg config.Config) (*promptkit.Engine, error) {
|
||||||
|
opts := []promptkit.Option{
|
||||||
|
promptkit.WithPromptFS(profileCheckPromptFS, "prompts"),
|
||||||
|
}
|
||||||
|
if cfg.PromptKit.ProfileFile != "" {
|
||||||
|
opts = append(opts, promptkit.WithProfileFile(cfg.PromptKit.ProfileFile))
|
||||||
|
}
|
||||||
|
return promptkit.NewEngine(promptkit.Config{
|
||||||
|
PromptDir: "unused",
|
||||||
|
ProfileDir: cfg.PromptKit.ProfileDir,
|
||||||
|
}, opts...)
|
||||||
|
}
|
||||||
@@ -194,7 +194,7 @@ func (recomputeTestOutput) Encode(_ context.Context, req contracts.OutputRequest
|
|||||||
func newRecomputeTestRoots(t *testing.T) stateTestRoots {
|
func newRecomputeTestRoots(t *testing.T) stateTestRoots {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
config := fmt.Sprintf(`version: 3
|
config := fmt.Sprintf(`version: 4
|
||||||
output:
|
output:
|
||||||
directory: %q
|
directory: %q
|
||||||
cache:
|
cache:
|
||||||
|
|||||||
@@ -217,7 +217,7 @@ func TestReferenceMaterializationSeparatesCLIAndConfigPathOrigins(t *testing.T)
|
|||||||
workingDir := t.TempDir()
|
workingDir := t.TempDir()
|
||||||
cfg := referenceContractConfig()
|
cfg := referenceContractConfig()
|
||||||
configPath := filepath.Join(configDir, "config.yml")
|
configPath := filepath.Join(configDir, "config.yml")
|
||||||
if err := os.WriteFile(configPath, []byte("version: 3\n"), 0o600); err != nil {
|
if err := os.WriteFile(configPath, []byte("version: 4\n"), 0o600); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(filepath.Join(configDir, "required.txt"), []byte("config reference"), 0o600); err != nil {
|
if err := os.WriteFile(filepath.Join(configDir, "required.txt"), []byte("config reference"), 0o600); err != nil {
|
||||||
|
|||||||
@@ -315,7 +315,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||||
}
|
}
|
||||||
profileIDs := effectiveLLMProfileIDs(effective.ResolvedPipeline)
|
profileIDs := effectiveLLMProfileIDs(effective.ResolvedPipeline)
|
||||||
if err := validateExplicitScriptoriumProfiles(context.Background(), effective.Config, profileIDs); err != nil {
|
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, profileIDs); err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||||
}
|
}
|
||||||
workingDir, err := os.Getwd()
|
workingDir, err := os.Getwd()
|
||||||
@@ -979,7 +979,7 @@ func runConfigValidate(args []string, stdout, stderr io.Writer, opts Options) in
|
|||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
if err := validateExplicitScriptoriumProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline)); err != nil {
|
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline)); err != nil {
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -241,7 +241,7 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
t.Run("one effective profile reaches the factory and modules", func(t *testing.T) {
|
t.Run("one effective profile reaches the factory and modules", func(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
profileDir := writeRunContractProfiles(t, "override-profile")
|
profileDir := writeRunContractProfiles(t, "override-profile")
|
||||||
prependRunContractConfig(t, roots, fmt.Sprintf("scriptorium:\n profile_dir: %q\n", profileDir))
|
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||||
harness := newStateTestHarness()
|
harness := newStateTestHarness()
|
||||||
var factoryProfiles []string
|
var factoryProfiles []string
|
||||||
opts := harness.options()
|
opts := harness.options()
|
||||||
@@ -273,7 +273,7 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
t.Run("validator profile remains distinct", func(t *testing.T) {
|
t.Run("validator profile remains distinct", func(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
profileDir := writeRunContractProfiles(t, "override-profile", "validator-profile")
|
profileDir := writeRunContractProfiles(t, "override-profile", "validator-profile")
|
||||||
prependRunContractConfig(t, roots, fmt.Sprintf("scriptorium:\n profile_dir: %q\n", profileDir))
|
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||||
harness := newStateTestHarness()
|
harness := newStateTestHarness()
|
||||||
var validatorProfiles []string
|
var validatorProfiles []string
|
||||||
opts := harness.options()
|
opts := harness.options()
|
||||||
@@ -299,7 +299,7 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
t.Run("unknown profile is rejected without factory access", func(t *testing.T) {
|
t.Run("unknown profile is rejected without factory access", func(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
profileDir := writeRunContractProfiles(t, "override-profile")
|
profileDir := writeRunContractProfiles(t, "override-profile")
|
||||||
prependRunContractConfig(t, roots, fmt.Sprintf("scriptorium:\n profile_dir: %q\n", profileDir))
|
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||||
factoryCalls := 0
|
factoryCalls := 0
|
||||||
opts := newStateTestHarness().options()
|
opts := newStateTestHarness().options()
|
||||||
opts.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
opts.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
|
|||||||
@@ -1,68 +0,0 @@
|
|||||||
package cli
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"testing/fstest"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
|
||||||
)
|
|
||||||
|
|
||||||
const profileCheckPromptID = "notarius.profile.check"
|
|
||||||
|
|
||||||
var profileCheckPromptFS = fstest.MapFS{
|
|
||||||
"prompts/profile-check.yaml": &fstest.MapFile{Data: []byte(`id: notarius.profile.check
|
|
||||||
version: "1.0.0"
|
|
||||||
default_profile: mistral-small-3
|
|
||||||
inputs:
|
|
||||||
- name: transcript
|
|
||||||
required: true
|
|
||||||
messages:
|
|
||||||
- role: user
|
|
||||||
content: "{{input \"transcript\"}}"
|
|
||||||
output:
|
|
||||||
format: text
|
|
||||||
validation_mode: none
|
|
||||||
repair_attempts: 0
|
|
||||||
`)},
|
|
||||||
}
|
|
||||||
|
|
||||||
func validateExplicitScriptoriumProfiles(ctx context.Context, cfg config.Config, profileIDs []string) error {
|
|
||||||
if len(profileIDs) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
engine, err := newProfileValidationEngine(cfg)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("load Scriptorium profiles: %w", err)
|
|
||||||
}
|
|
||||||
for _, profileID := range profileIDs {
|
|
||||||
if _, err := engine.Prepare(ctx, scriptorium.RunRequest{
|
|
||||||
PromptID: profileCheckPromptID,
|
|
||||||
ProfileID: profileID,
|
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
|
||||||
"transcript": scriptorium.Inline("profile check"),
|
|
||||||
},
|
|
||||||
}); err != nil {
|
|
||||||
if errors.Is(err, scriptorium.ErrProfileNotFound) {
|
|
||||||
return fmt.Errorf("Scriptorium profile %q is not configured", profileID)
|
|
||||||
}
|
|
||||||
return fmt.Errorf("validate Scriptorium profile %q: %w", profileID, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func newProfileValidationEngine(cfg config.Config) (*scriptorium.Engine, error) {
|
|
||||||
opts := []scriptorium.Option{
|
|
||||||
scriptorium.WithPromptFS(profileCheckPromptFS, "prompts"),
|
|
||||||
}
|
|
||||||
if cfg.Scriptorium.ProfileFile != "" {
|
|
||||||
opts = append(opts, scriptorium.WithProfileFile(cfg.Scriptorium.ProfileFile))
|
|
||||||
}
|
|
||||||
return scriptorium.NewEngine(scriptorium.Config{
|
|
||||||
PromptDir: "unused",
|
|
||||||
ProfileDir: cfg.Scriptorium.ProfileDir,
|
|
||||||
}, opts...)
|
|
||||||
}
|
|
||||||
@@ -635,7 +635,7 @@ func newStateTestRoots(t *testing.T) stateTestRoots {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
roots.config = filepath.Join(base, "config.yml")
|
roots.config = filepath.Join(base, "config.yml")
|
||||||
config := fmt.Sprintf("version: 3\noutput:\n directory: %q\ncache:\n chunk_plans:\n directory: %q\n mode: auto\n checkpoints:\n enabled: true\n directory: %q\ndebug:\n directory: %q\npipelines:\n sample:\n input: test/input\n chunk: test/chunk\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n output: test/output\n", roots.output, roots.plans, roots.checkpoints, roots.debug)
|
config := fmt.Sprintf("version: 4\noutput:\n directory: %q\ncache:\n chunk_plans:\n directory: %q\n mode: auto\n checkpoints:\n enabled: true\n directory: %q\ndebug:\n directory: %q\npipelines:\n sample:\n input: test/input\n chunk: test/chunk\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n output: test/output\n", roots.output, roots.plans, roots.checkpoints, roots.debug)
|
||||||
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
|
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -53,7 +53,7 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
|
|||||||
PipelineID: "pipeline-1",
|
PipelineID: "pipeline-1",
|
||||||
PipelineDigest: "sha256:abc123",
|
PipelineDigest: "sha256:abc123",
|
||||||
LLMProfiles: []LLMProfileManifest{
|
LLMProfiles: []LLMProfileManifest{
|
||||||
{ID: "default", Provider: "scriptorium", Model: "model-a"},
|
{ID: "default", Provider: "promptkit", Model: "model-a"},
|
||||||
},
|
},
|
||||||
ArtifactLanes: []ArtifactLaneManifest{
|
ArtifactLanes: []ArtifactLaneManifest{
|
||||||
{
|
{
|
||||||
@@ -102,6 +102,9 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
|
|||||||
t.Fatalf("llm_profiles[0] = %#v, want object", profiles[0])
|
t.Fatalf("llm_profiles[0] = %#v, want object", profiles[0])
|
||||||
}
|
}
|
||||||
assertHasKeys(t, profile, "id", "provider", "model")
|
assertHasKeys(t, profile, "id", "provider", "model")
|
||||||
|
if profile["provider"] != "promptkit" {
|
||||||
|
t.Fatalf("llm_profiles[0].provider = %#v, want promptkit", profile["provider"])
|
||||||
|
}
|
||||||
|
|
||||||
lanes, ok := got["artifact_lanes"].([]any)
|
lanes, ok := got["artifact_lanes"].([]any)
|
||||||
if !ok {
|
if !ok {
|
||||||
|
|||||||
@@ -4,10 +4,10 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
)
|
)
|
||||||
|
|
||||||
const SupportedFileConfigVersion = 3
|
const SupportedFileConfigVersion = 4
|
||||||
|
|
||||||
type Config struct {
|
type Config struct {
|
||||||
Scriptorium ScriptoriumConfig `json:"scriptorium,omitempty"`
|
PromptKit PromptKitConfig `json:"promptkit,omitempty"`
|
||||||
Pipelines map[string]pipeline.PipelineProfile `json:"pipelines"`
|
Pipelines map[string]pipeline.PipelineProfile `json:"pipelines"`
|
||||||
Concurrency ConcurrencyConfig `json:"concurrency"`
|
Concurrency ConcurrencyConfig `json:"concurrency"`
|
||||||
Output OutputConfig `json:"output"`
|
Output OutputConfig `json:"output"`
|
||||||
@@ -15,7 +15,7 @@ type Config struct {
|
|||||||
Debug DebugConfig `json:"debug"`
|
Debug DebugConfig `json:"debug"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type ScriptoriumConfig struct {
|
type PromptKitConfig struct {
|
||||||
ProfileDir string `json:"profile_dir,omitempty"`
|
ProfileDir string `json:"profile_dir,omitempty"`
|
||||||
ProfileFile string `json:"profile_file,omitempty"`
|
ProfileFile string `json:"profile_file,omitempty"`
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -84,6 +84,29 @@ func TestEffectiveConfigMaterializesDefaultBindingsThroughCatalog(t *testing.T)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestEffectiveConfigPreservesPromptKitProfileSource(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
profileSource PromptKitConfig
|
||||||
|
}{
|
||||||
|
{name: "profile directory", profileSource: PromptKitConfig{ProfileDir: "./profiles"}},
|
||||||
|
{name: "profile file", profileSource: PromptKitConfig{ProfileFile: "./profiles.yml"}},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := configForEffectiveTests(t, effectiveProfile())
|
||||||
|
cfg.PromptKit = tt.profileSource
|
||||||
|
effective, err := cfg.Resolve(ResolveInput{PipelineID: "main", Catalog: effectiveCatalog(t)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v", err)
|
||||||
|
}
|
||||||
|
if effective.Config.PromptKit != cfg.PromptKit {
|
||||||
|
t.Fatalf("effective PromptKit config = %#v, want %#v", effective.Config.PromptKit, cfg.PromptKit)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
|
func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
func TestPrecedenceFileValuesOverrideBuiltInDefaults(t *testing.T) {
|
func TestPrecedenceFileValuesOverrideBuiltInDefaults(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 4
|
total_llm: 4
|
||||||
stage_workers:
|
stage_workers:
|
||||||
@@ -35,7 +35,7 @@ debug:
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestPrecedenceOperationalEnvironmentOverridesFileValues(t *testing.T) {
|
func TestPrecedenceOperationalEnvironmentOverridesFileValues(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 2
|
total_llm: 2
|
||||||
stage_workers:
|
stage_workers:
|
||||||
@@ -81,21 +81,21 @@ func TestPrecedenceExtractWorkersFollowEffectiveConcurrencyUnlessExplicit(t *tes
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "default follows environment total",
|
name: "default follows environment total",
|
||||||
file: "version: 3\n",
|
file: "version: 4\n",
|
||||||
env: map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "5"},
|
env: map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "5"},
|
||||||
wantTotal: 5,
|
wantTotal: 5,
|
||||||
wantWorker: 5,
|
wantWorker: 5,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "file worker is retained",
|
name: "file worker is retained",
|
||||||
file: "version: 3\nconcurrency:\n total_llm: 3\n stage_workers:\n extract: 2\n",
|
file: "version: 4\nconcurrency:\n total_llm: 3\n stage_workers:\n extract: 2\n",
|
||||||
env: map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6"},
|
env: map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6"},
|
||||||
wantTotal: 6,
|
wantTotal: 6,
|
||||||
wantWorker: 2,
|
wantWorker: 2,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "environment worker is retained",
|
name: "environment worker is retained",
|
||||||
file: "version: 3\nconcurrency:\n total_llm: 2\n",
|
file: "version: 4\nconcurrency:\n total_llm: 2\n",
|
||||||
env: map[string]string{
|
env: map[string]string{
|
||||||
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6",
|
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6",
|
||||||
"NOTARIUS_STAGE_WORKERS_EXTRACT": "4",
|
"NOTARIUS_STAGE_WORKERS_EXTRACT": "4",
|
||||||
@@ -118,7 +118,7 @@ func TestPrecedenceExtractWorkersFollowEffectiveConcurrencyUnlessExplicit(t *tes
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestPrecedenceEmptyFileCacheDirectoriesDeferPerUserResolution(t *testing.T) {
|
func TestPrecedenceEmptyFileCacheDirectoriesDeferPerUserResolution(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
cache:
|
cache:
|
||||||
chunk_plans:
|
chunk_plans:
|
||||||
directory: ""
|
directory: ""
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ import (
|
|||||||
|
|
||||||
type FileConfig struct {
|
type FileConfig struct {
|
||||||
Version int `yaml:"version"`
|
Version int `yaml:"version"`
|
||||||
Scriptorium *FileScriptoriumConfig `yaml:"scriptorium,omitempty"`
|
PromptKit *FilePromptKitConfig `yaml:"promptkit,omitempty"`
|
||||||
Pipelines map[string]FilePipelineProfile `yaml:"pipelines,omitempty"`
|
Pipelines map[string]FilePipelineProfile `yaml:"pipelines,omitempty"`
|
||||||
Concurrency *FileConcurrencyConfig `yaml:"concurrency,omitempty"`
|
Concurrency *FileConcurrencyConfig `yaml:"concurrency,omitempty"`
|
||||||
Output *FileOutputConfig `yaml:"output,omitempty"`
|
Output *FileOutputConfig `yaml:"output,omitempty"`
|
||||||
@@ -22,7 +22,7 @@ type FileConfig struct {
|
|||||||
Debug *FileDebugConfig `yaml:"debug,omitempty"`
|
Debug *FileDebugConfig `yaml:"debug,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type FileScriptoriumConfig struct {
|
type FilePromptKitConfig struct {
|
||||||
ProfileDir *string `yaml:"profile_dir,omitempty"`
|
ProfileDir *string `yaml:"profile_dir,omitempty"`
|
||||||
ProfileFile *string `yaml:"profile_file,omitempty"`
|
ProfileFile *string `yaml:"profile_file,omitempty"`
|
||||||
}
|
}
|
||||||
@@ -325,6 +325,9 @@ func ParseFileConfigYAML(data []byte) (FileConfig, error) {
|
|||||||
if header.Version == 2 {
|
if header.Version == 2 {
|
||||||
return FileConfig{}, fmt.Errorf("config version 2 is no longer supported; migrate the file using the version 2-to-3 migration in docs/config.md")
|
return FileConfig{}, fmt.Errorf("config version 2 is no longer supported; migrate the file using the version 2-to-3 migration in docs/config.md")
|
||||||
}
|
}
|
||||||
|
if header.Version == 3 {
|
||||||
|
return FileConfig{}, fmt.Errorf("config version 3 is no longer supported; change \"version: 3\" to \"version: 4\" and rename \"scriptorium:\" to \"promptkit:\"")
|
||||||
|
}
|
||||||
if header.Version != SupportedFileConfigVersion {
|
if header.Version != SupportedFileConfigVersion {
|
||||||
return FileConfig{}, fmt.Errorf("unsupported config version %d (supported version is %d)", header.Version, SupportedFileConfigVersion)
|
return FileConfig{}, fmt.Errorf("unsupported config version %d (supported version is %d)", header.Version, SupportedFileConfigVersion)
|
||||||
}
|
}
|
||||||
@@ -450,20 +453,20 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if fileCfg.Scriptorium != nil {
|
if fileCfg.PromptKit != nil {
|
||||||
if fileCfg.Scriptorium.ProfileDir != nil {
|
if fileCfg.PromptKit.ProfileDir != nil {
|
||||||
value := strings.TrimSpace(*fileCfg.Scriptorium.ProfileDir)
|
value := strings.TrimSpace(*fileCfg.PromptKit.ProfileDir)
|
||||||
if value == "" {
|
if value == "" {
|
||||||
return fmt.Errorf("scriptorium.profile_dir must not be empty when set")
|
return fmt.Errorf("promptkit.profile_dir must not be empty when set")
|
||||||
}
|
}
|
||||||
c.Scriptorium.ProfileDir = value
|
c.PromptKit.ProfileDir = value
|
||||||
}
|
}
|
||||||
if fileCfg.Scriptorium.ProfileFile != nil {
|
if fileCfg.PromptKit.ProfileFile != nil {
|
||||||
value := strings.TrimSpace(*fileCfg.Scriptorium.ProfileFile)
|
value := strings.TrimSpace(*fileCfg.PromptKit.ProfileFile)
|
||||||
if value == "" {
|
if value == "" {
|
||||||
return fmt.Errorf("scriptorium.profile_file must not be empty when set")
|
return fmt.Errorf("promptkit.profile_file must not be empty when set")
|
||||||
}
|
}
|
||||||
c.Scriptorium.ProfileFile = value
|
c.PromptKit.ProfileFile = value
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package config
|
package config
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"encoding/json"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"reflect"
|
"reflect"
|
||||||
@@ -34,8 +35,8 @@ func TestDefaultReturnsDocumentedValuesAndIndependentMaps(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigMinimalVersion3AppliesOverDefaults(t *testing.T) {
|
func TestFileConfigMinimalVersion4AppliesOverDefaults(t *testing.T) {
|
||||||
file := parseFileConfig(t, "version: 3\n")
|
file := parseFileConfig(t, "version: 4\n")
|
||||||
cfg := Default()
|
cfg := Default()
|
||||||
if err := cfg.ApplyFileConfig(file); err != nil {
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
@@ -48,6 +49,78 @@ func TestFileConfigMinimalVersion3AppliesOverDefaults(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitProfileSourcesSurviveConfigBoundaries(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
yaml string
|
||||||
|
want PromptKitConfig
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "profile directory",
|
||||||
|
yaml: "version: 4\npromptkit:\n profile_dir: ' ./profiles '\n",
|
||||||
|
want: PromptKitConfig{ProfileDir: "./profiles"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "profile file",
|
||||||
|
yaml: "version: 4\npromptkit:\n profile_file: ' ./profiles.yml '\n",
|
||||||
|
want: PromptKitConfig{ProfileFile: "./profiles.yml"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := applyFileConfig(t, tt.yaml)
|
||||||
|
if cfg.PromptKit != tt.want {
|
||||||
|
t.Fatalf("PromptKit config = %#v, want %#v", cfg.PromptKit, tt.want)
|
||||||
|
}
|
||||||
|
if got := cloneConfig(cfg).PromptKit; got != tt.want {
|
||||||
|
t.Fatalf("cloned PromptKit config = %#v, want %#v", got, tt.want)
|
||||||
|
}
|
||||||
|
if got := cfg.Redacted().PromptKit; got != tt.want {
|
||||||
|
t.Fatalf("redacted PromptKit config = %#v, want %#v", got, tt.want)
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := json.Marshal(cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("json.Marshal() error = %v", err)
|
||||||
|
}
|
||||||
|
var payload map[string]json.RawMessage
|
||||||
|
if err := json.Unmarshal(data, &payload); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal() error = %v", err)
|
||||||
|
}
|
||||||
|
if _, ok := payload["promptkit"]; !ok {
|
||||||
|
t.Fatalf("runtime JSON keys = %v, want promptkit", payload)
|
||||||
|
}
|
||||||
|
if _, ok := payload["scriptorium"]; ok {
|
||||||
|
t.Fatalf("runtime JSON keys = %v, must not contain removed section", payload)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitExplicitEmptyProfileSourcesAreRejected(t *testing.T) {
|
||||||
|
for _, field := range []string{"profile_dir", "profile_file"} {
|
||||||
|
t.Run(field, func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, "version: 4\npromptkit:\n "+field+": ''\n")
|
||||||
|
cfg := Default()
|
||||||
|
err := cfg.ApplyFileConfig(file)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "promptkit."+field+" must not be empty") {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v, want explicit-empty rejection", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitProfileSourcesRemainMutuallyExclusive(t *testing.T) {
|
||||||
|
cfg := applyFileConfig(t, `version: 4
|
||||||
|
promptkit:
|
||||||
|
profile_dir: ./profiles
|
||||||
|
profile_file: ./profiles.yml
|
||||||
|
`)
|
||||||
|
if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "promptkit profile_dir and profile_file are mutually exclusive") {
|
||||||
|
t.Fatalf("Validate() error = %v, want mutually exclusive profile sources", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestFileConfigMissingVersionIsReportedBeforeFieldDecoding(t *testing.T) {
|
func TestFileConfigMissingVersionIsReportedBeforeFieldDecoding(t *testing.T) {
|
||||||
_, err := ParseFileConfigYAML([]byte("workspace:\n directory: /tmp/old\n"))
|
_, err := ParseFileConfigYAML([]byte("workspace:\n directory: /tmp/old\n"))
|
||||||
if err == nil || !strings.Contains(err.Error(), "config version is required") {
|
if err == nil || !strings.Contains(err.Error(), "config version is required") {
|
||||||
@@ -55,6 +128,15 @@ func TestFileConfigMissingVersionIsReportedBeforeFieldDecoding(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFileConfigVersion3ReportsPromptKitMigration(t *testing.T) {
|
||||||
|
_, err := ParseFileConfigYAML([]byte("version: 3\nscriptorium:\n profile_dir: ./profiles\n"))
|
||||||
|
if err == nil ||
|
||||||
|
!strings.Contains(err.Error(), `change "version: 3" to "version: 4"`) ||
|
||||||
|
!strings.Contains(err.Error(), `rename "scriptorium:" to "promptkit:"`) {
|
||||||
|
t.Fatalf("version 3 error = %v, want actionable version and section migration", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -63,14 +145,19 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "removed diagnostics",
|
name: "removed diagnostics",
|
||||||
yaml: "version: 3\ndiagnostics: {}\n",
|
yaml: "version: 4\ndiagnostics: {}\n",
|
||||||
want: "field diagnostics not found",
|
want: "field diagnostics not found",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "removed llm profiles",
|
name: "removed llm profiles",
|
||||||
yaml: "version: 3\nllm_profiles: {}\n",
|
yaml: "version: 4\nllm_profiles: {}\n",
|
||||||
want: "field llm_profiles not found",
|
want: "field llm_profiles not found",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: "removed scriptorium section",
|
||||||
|
yaml: "version: 4\nscriptorium: {}\n",
|
||||||
|
want: "field scriptorium not found",
|
||||||
|
},
|
||||||
{
|
{
|
||||||
name: "version 2 migration",
|
name: "version 2 migration",
|
||||||
yaml: "version: 2\nworkspace:\n directory: /tmp/old\n",
|
yaml: "version: 2\nworkspace:\n directory: /tmp/old\n",
|
||||||
@@ -78,27 +165,27 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "pipeline field",
|
name: "pipeline field",
|
||||||
yaml: "version: 3\npipelines:\n main:\n unknown: true\n",
|
yaml: "version: 4\npipelines:\n main:\n unknown: true\n",
|
||||||
want: "field unknown not found",
|
want: "field unknown not found",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "lane field",
|
name: "lane field",
|
||||||
yaml: "version: 3\npipelines:\n main:\n artifacts:\n spells:\n unknown: true\n",
|
yaml: "version: 4\npipelines:\n main:\n artifacts:\n spells:\n unknown: true\n",
|
||||||
want: "field unknown not found",
|
want: "field unknown not found",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "module binding field",
|
name: "module binding field",
|
||||||
yaml: "version: 3\npipelines:\n main:\n input:\n module: seriatim\n unknown: true\n",
|
yaml: "version: 4\npipelines:\n main:\n input:\n module: seriatim\n unknown: true\n",
|
||||||
want: "field unknown not found in module binding",
|
want: "field unknown not found in module binding",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "checkpoint field",
|
name: "checkpoint field",
|
||||||
yaml: "version: 3\ncache:\n checkpoints:\n unknown: true\n",
|
yaml: "version: 4\ncache:\n checkpoints:\n unknown: true\n",
|
||||||
want: "field unknown not found",
|
want: "field unknown not found",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "checkpoint enabled type",
|
name: "checkpoint enabled type",
|
||||||
yaml: "version: 3\ncache:\n checkpoints:\n enabled: definitely\n",
|
yaml: "version: 4\ncache:\n checkpoints:\n enabled: definitely\n",
|
||||||
want: "cannot unmarshal",
|
want: "cannot unmarshal",
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -113,7 +200,7 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigModuleBindingsPreserveFormsAndValidatorPresence(t *testing.T) {
|
func TestFileConfigModuleBindingsPreserveFormsAndValidatorPresence(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
main:
|
main:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
@@ -160,7 +247,7 @@ pipelines:
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigReferencePrecedenceIsRetained(t *testing.T) {
|
func TestFileConfigReferencePrecedenceIsRetained(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
main:
|
main:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
@@ -224,7 +311,7 @@ pipelines:
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigStageLocalValidatorsPreserveOrderAndFields(t *testing.T) {
|
func TestFileConfigStageLocalValidatorsPreserveOrderAndFields(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
main:
|
main:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
@@ -277,8 +364,8 @@ pipelines:
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigStateSectionsApplyIndependently(t *testing.T) {
|
func TestFileConfigStateSectionsApplyIndependently(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
scriptorium:
|
promptkit:
|
||||||
profile_dir: ./profiles
|
profile_dir: ./profiles
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 7
|
total_llm: 7
|
||||||
@@ -294,15 +381,15 @@ cache:
|
|||||||
debug:
|
debug:
|
||||||
directory: ./debug
|
directory: ./debug
|
||||||
`)
|
`)
|
||||||
if cfg.Scriptorium.ProfileDir != "./profiles" || cfg.Scriptorium.ProfileFile != "" {
|
if cfg.PromptKit.ProfileDir != "./profiles" || cfg.PromptKit.ProfileFile != "" {
|
||||||
t.Fatalf("scriptorium = %#v", cfg.Scriptorium)
|
t.Fatalf("promptkit = %#v", cfg.PromptKit)
|
||||||
}
|
}
|
||||||
if cfg.Concurrency.TotalLLM != 7 || cfg.Concurrency.StageWorkers["extract"] != 7 {
|
if cfg.Concurrency.TotalLLM != 7 || cfg.Concurrency.StageWorkers["extract"] != 7 {
|
||||||
t.Fatalf("concurrency = %#v", cfg.Concurrency)
|
t.Fatalf("concurrency = %#v", cfg.Concurrency)
|
||||||
}
|
}
|
||||||
if cfg.Output.Directory != "./output" || cfg.Cache.ChunkPlans.Directory != "plans" || cfg.Cache.ChunkPlans.Mode != pipeline.ChunkCacheBypass ||
|
if cfg.Output.Directory != "./output" || cfg.Cache.ChunkPlans.Directory != "plans" || cfg.Cache.ChunkPlans.Mode != pipeline.ChunkCacheBypass ||
|
||||||
!cfg.Cache.Checkpoints.Enabled || cfg.Cache.Checkpoints.Directory != "checkpoints" || cfg.Debug.Directory != "./debug" {
|
!cfg.Cache.Checkpoints.Enabled || cfg.Cache.Checkpoints.Directory != "checkpoints" || cfg.Debug.Directory != "./debug" {
|
||||||
t.Fatalf("state sections = %#v, %#v, %#v, %#v", cfg.Output, cfg.Cache, cfg.Debug, cfg.Scriptorium)
|
t.Fatalf("state sections = %#v, %#v, %#v, %#v", cfg.Output, cfg.Cache, cfg.Debug, cfg.PromptKit)
|
||||||
}
|
}
|
||||||
if cfg.Output.Directory == cfg.Cache.ChunkPlans.Directory || cfg.Cache.ChunkPlans.Directory == cfg.Cache.Checkpoints.Directory || cfg.Cache.Checkpoints.Directory == cfg.Debug.Directory {
|
if cfg.Output.Directory == cfg.Cache.ChunkPlans.Directory || cfg.Cache.ChunkPlans.Directory == cfg.Cache.Checkpoints.Directory || cfg.Cache.Checkpoints.Directory == cfg.Debug.Directory {
|
||||||
t.Fatal("state roots were coupled")
|
t.Fatal("state roots were coupled")
|
||||||
@@ -310,11 +397,11 @@ debug:
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigCheckpointEnabledCanBeExplicitlyDisabled(t *testing.T) {
|
func TestFileConfigCheckpointEnabledCanBeExplicitlyDisabled(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, "version: 3\ncache:\n checkpoints:\n enabled: true\n")
|
cfg := applyFileConfig(t, "version: 4\ncache:\n checkpoints:\n enabled: true\n")
|
||||||
if !cfg.Cache.Checkpoints.Enabled || !cloneConfig(cfg).Cache.Checkpoints.Enabled {
|
if !cfg.Cache.Checkpoints.Enabled || !cloneConfig(cfg).Cache.Checkpoints.Enabled {
|
||||||
t.Fatalf("enabled checkpoint config was not retained: %#v", cfg.Cache.Checkpoints)
|
t.Fatalf("enabled checkpoint config was not retained: %#v", cfg.Cache.Checkpoints)
|
||||||
}
|
}
|
||||||
file := parseFileConfig(t, "version: 3\ncache:\n checkpoints:\n enabled: false\n")
|
file := parseFileConfig(t, "version: 4\ncache:\n checkpoints:\n enabled: false\n")
|
||||||
if err := cfg.ApplyFileConfig(file); err != nil {
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -331,17 +418,17 @@ func TestFileConfigRejectsTrimmedKeyCollisions(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "pipeline ids",
|
name: "pipeline ids",
|
||||||
yaml: "version: 3\npipelines:\n main: {}\n ' main ': {}\n",
|
yaml: "version: 4\npipelines:\n main: {}\n ' main ': {}\n",
|
||||||
want: "pipeline id \"main\" is duplicated after trimming",
|
want: "pipeline id \"main\" is duplicated after trimming",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "lane ids",
|
name: "lane ids",
|
||||||
yaml: "version: 3\npipelines:\n main:\n artifacts:\n spells: {}\n ' spells ': {}\n",
|
yaml: "version: 4\npipelines:\n main:\n artifacts:\n spells: {}\n ' spells ': {}\n",
|
||||||
want: "artifact lane id \"spells\" is duplicated after trimming",
|
want: "artifact lane id \"spells\" is duplicated after trimming",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "reference slots",
|
name: "reference slots",
|
||||||
yaml: "version: 3\npipelines:\n main:\n references:\n slot: ./one.txt\n ' slot ': ./two.txt\n",
|
yaml: "version: 4\npipelines:\n main:\n references:\n slot: ./one.txt\n ' slot ': ./two.txt\n",
|
||||||
want: "reference slot \"slot\" is duplicated after trimming",
|
want: "reference slot \"slot\" is duplicated after trimming",
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -358,7 +445,7 @@ func TestFileConfigRejectsTrimmedKeyCollisions(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigParsesOrderedStepsAndReferenceSources(t *testing.T) {
|
func TestFileConfigParsesOrderedStepsAndReferenceSources(t *testing.T) {
|
||||||
file := parseFileConfig(t, `version: 3
|
file := parseFileConfig(t, `version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
session:
|
session:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
@@ -397,7 +484,7 @@ func TestFileConfigRejectsAmbiguousReferenceSourceForms(t *testing.T) {
|
|||||||
"artifact: {step: 1, lane: b}",
|
"artifact: {step: 1, lane: b}",
|
||||||
"1",
|
"1",
|
||||||
} {
|
} {
|
||||||
_, err := ParseFileConfigYAML([]byte("version: 3\npipelines:\n p:\n input: text\n references:\n slot: " + source + "\n"))
|
_, err := ParseFileConfigYAML([]byte("version: 4\npipelines:\n p:\n input: text\n references:\n slot: " + source + "\n"))
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatalf("ParseFileConfigYAML(%q) error = nil", source)
|
t.Fatalf("ParseFileConfigYAML(%q) error = nil", source)
|
||||||
}
|
}
|
||||||
@@ -412,12 +499,12 @@ func TestFileConfigRejectsEmptyAndAmbiguousPipelineShapes(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "empty steps",
|
name: "empty steps",
|
||||||
yaml: "version: 3\npipelines:\n p:\n input: text\n steps: []\n",
|
yaml: "version: 4\npipelines:\n p:\n input: text\n steps: []\n",
|
||||||
want: "at least one ordered step",
|
want: "at least one ordered step",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "both forms",
|
name: "both forms",
|
||||||
yaml: "version: 3\npipelines:\n p:\n input: text\n artifacts: {}\n steps: []\n",
|
yaml: "version: 4\npipelines:\n p:\n input: text\n artifacts: {}\n steps: []\n",
|
||||||
want: "both artifacts and steps",
|
want: "both artifacts and steps",
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ import (
|
|||||||
|
|
||||||
func (c Config) Validate() error {
|
func (c Config) Validate() error {
|
||||||
c.Concurrency.recomputeStageWorkerDefaults()
|
c.Concurrency.recomputeStageWorkerDefaults()
|
||||||
if err := validateScriptorium(c.Scriptorium); err != nil {
|
if err := validatePromptKit(c.PromptKit); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if err := validateStateSurfaces(c); err != nil {
|
if err := validateStateSurfaces(c); err != nil {
|
||||||
@@ -49,9 +49,9 @@ func validateStageWorkers(cfg ConcurrencyConfig) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func validateScriptorium(cfg ScriptoriumConfig) error {
|
func validatePromptKit(cfg PromptKitConfig) error {
|
||||||
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
||||||
return fmt.Errorf("scriptorium profile_dir and profile_file are mutually exclusive")
|
return fmt.Errorf("promptkit profile_dir and profile_file are mutually exclusive")
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -87,10 +87,10 @@ func TestValidateConcurrencyRules(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestValidateScriptoriumSourcesAreMutuallyExclusive(t *testing.T) {
|
func TestValidatePromptKitSourcesAreMutuallyExclusive(t *testing.T) {
|
||||||
cfg := Default()
|
cfg := Default()
|
||||||
cfg.Scriptorium = ScriptoriumConfig{ProfileDir: "./profiles", ProfileFile: "./profile.yml"}
|
cfg.PromptKit = PromptKitConfig{ProfileDir: "./profiles", ProfileFile: "./profile.yml"}
|
||||||
assertValidationContains(t, cfg, "scriptorium profile_dir and profile_file are mutually exclusive")
|
assertValidationContains(t, cfg, "promptkit profile_dir and profile_file are mutually exclusive")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestValidateStateSurfaceRules(t *testing.T) {
|
func TestValidateStateSurfaceRules(t *testing.T) {
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
type AssetSource struct {
|
type AssetSource struct {
|
||||||
@@ -72,7 +72,7 @@ func (r *AssetRegistry) SchemaFS() (fs.FS, error) {
|
|||||||
return flattenAssetSources(r.schemas)
|
return flattenAssetSources(r.schemas)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *AssetRegistry) ScriptoriumOptions() ([]scriptorium.Option, error) {
|
func (r *AssetRegistry) PromptKitOptions() ([]promptkit.Option, error) {
|
||||||
promptFS, err := r.PromptFS()
|
promptFS, err := r.PromptFS()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("prepare prompt assets: %w", err)
|
return nil, fmt.Errorf("prepare prompt assets: %w", err)
|
||||||
@@ -81,9 +81,9 @@ func (r *AssetRegistry) ScriptoriumOptions() ([]scriptorium.Option, error) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("prepare schema assets: %w", err)
|
return nil, fmt.Errorf("prepare schema assets: %w", err)
|
||||||
}
|
}
|
||||||
return []scriptorium.Option{
|
return []promptkit.Option{
|
||||||
scriptorium.WithPromptFS(promptFS, "."),
|
promptkit.WithPromptFS(promptFS, "."),
|
||||||
scriptorium.WithSchemaFS(schemaFS, "."),
|
promptkit.WithSchemaFS(schemaFS, "."),
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ import (
|
|||||||
"testing/fstest"
|
"testing/fstest"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestAssetRegistryCombinesPromptAndSchemaSources(t *testing.T) {
|
func TestAssetRegistryCombinesPromptAndSchemaSources(t *testing.T) {
|
||||||
@@ -22,12 +22,12 @@ func TestAssetRegistryCombinesPromptAndSchemaSources(t *testing.T) {
|
|||||||
"root/schemas/out.json": {Data: []byte(`{"type":"object"}`)},
|
"root/schemas/out.json": {Data: []byte(`{"type":"object"}`)},
|
||||||
}, "root")
|
}, "root")
|
||||||
|
|
||||||
engine := newAssetTestEngine(t, registry)
|
engine := newPromptKitAssetTestEngine(t, registry)
|
||||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: "asset.test",
|
PromptID: "asset.test",
|
||||||
ProfileID: "asset-test-profile",
|
ProfileID: "asset-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.Inline(`{"ok":true}`),
|
"transcript": promptkit.Inline(`{"ok":true}`),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -50,12 +50,12 @@ func TestAssetRegistryPrepareFailsForMissingPromptAsset(t *testing.T) {
|
|||||||
"out.json": {Data: []byte(`{"type":"object"}`)},
|
"out.json": {Data: []byte(`{"type":"object"}`)},
|
||||||
}, ".")
|
}, ".")
|
||||||
|
|
||||||
engine := newAssetTestEngine(t, registry)
|
engine := newPromptKitAssetTestEngine(t, registry)
|
||||||
_, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
_, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: "asset.test",
|
PromptID: "asset.test",
|
||||||
ProfileID: "asset-test-profile",
|
ProfileID: "asset-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.Inline(`{"ok":true}`),
|
"transcript": promptkit.Inline(`{"ok":true}`),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err == nil || !strings.Contains(err.Error(), "content_file") {
|
if err == nil || !strings.Contains(err.Error(), "content_file") {
|
||||||
@@ -74,12 +74,12 @@ func TestAssetRegistryPrepareFailsForMissingSchemaAsset(t *testing.T) {
|
|||||||
"present.json": {Data: []byte(`{"type":"object"}`)},
|
"present.json": {Data: []byte(`{"type":"object"}`)},
|
||||||
}, ".")
|
}, ".")
|
||||||
|
|
||||||
engine := newAssetTestEngine(t, registry)
|
engine := newPromptKitAssetTestEngine(t, registry)
|
||||||
_, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
_, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: "asset.test",
|
PromptID: "asset.test",
|
||||||
ProfileID: "asset-test-profile",
|
ProfileID: "asset-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.Inline(`{"ok":true}`),
|
"transcript": promptkit.Inline(`{"ok":true}`),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err == nil || !strings.Contains(err.Error(), "missing.json") {
|
if err == nil || !strings.Contains(err.Error(), "missing.json") {
|
||||||
@@ -145,18 +145,18 @@ func TestHashAssetsOmitsRawAssetContent(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func newAssetTestEngine(t *testing.T, registry *AssetRegistry) *scriptorium.Engine {
|
func newPromptKitAssetTestEngine(t *testing.T, registry *AssetRegistry) *promptkit.Engine {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
options, err := registry.ScriptoriumOptions()
|
options, err := registry.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ScriptoriumOptions() error = %v, want nil", err)
|
t.Fatalf("PromptKitOptions() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
ID: "asset-test-profile",
|
ID: "asset-test-profile",
|
||||||
Endpoint: "http://127.0.0.1:1/v1",
|
Endpoint: "http://127.0.0.1:1/v1",
|
||||||
Model: "asset-test-model",
|
Model: "asset-test-model",
|
||||||
})))
|
})))
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("NewEngine() error = %v, want nil", err)
|
t.Fatalf("NewEngine() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -13,23 +13,23 @@ import (
|
|||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
const scriptoriumProviderName = "scriptorium"
|
const promptKitProviderName = "promptkit"
|
||||||
|
|
||||||
type ScriptoriumClientConfig struct {
|
type PromptKitClientConfig struct {
|
||||||
ProfileDir string
|
ProfileDir string
|
||||||
ProfileFile string
|
ProfileFile string
|
||||||
Assets *AssetRegistry
|
Assets *AssetRegistry
|
||||||
Timeout time.Duration
|
Timeout time.Duration
|
||||||
HTTPClient *http.Client
|
HTTPClient *http.Client
|
||||||
EngineOptions []scriptorium.Option
|
EngineOptions []promptkit.Option
|
||||||
Recorder *LLMProfileRecorder
|
Recorder *LLMProfileRecorder
|
||||||
}
|
}
|
||||||
|
|
||||||
type ScriptoriumClient struct {
|
type PromptKitClient struct {
|
||||||
engine *scriptorium.Engine
|
engine *promptkit.Engine
|
||||||
recorder *LLMProfileRecorder
|
recorder *LLMProfileRecorder
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -38,49 +38,49 @@ type LLMProfileRecorder struct {
|
|||||||
profiles map[string]artifacts.LLMProfileManifest
|
profiles map[string]artifacts.LLMProfileManifest
|
||||||
}
|
}
|
||||||
|
|
||||||
var _ contracts.StructuredLLMClient = (*ScriptoriumClient)(nil)
|
var _ contracts.StructuredLLMClient = (*PromptKitClient)(nil)
|
||||||
var _ contracts.LLMProfileManifestProvider = (*ScriptoriumClient)(nil)
|
var _ contracts.LLMProfileManifestProvider = (*PromptKitClient)(nil)
|
||||||
|
|
||||||
func NewScriptoriumClient(cfg ScriptoriumClientConfig) (*ScriptoriumClient, error) {
|
func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
||||||
if cfg.Assets == nil {
|
if cfg.Assets == nil {
|
||||||
return nil, fmt.Errorf("scriptorium client assets must not be nil")
|
return nil, fmt.Errorf("PromptKit client assets must not be nil")
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
||||||
return nil, fmt.Errorf("scriptorium profile_dir and profile_file are mutually exclusive")
|
return nil, fmt.Errorf("PromptKit profile_dir and profile_file are mutually exclusive")
|
||||||
}
|
}
|
||||||
options, err := cfg.Assets.ScriptoriumOptions()
|
options, err := cfg.Assets.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
if profileFile := strings.TrimSpace(cfg.ProfileFile); profileFile != "" {
|
if profileFile := strings.TrimSpace(cfg.ProfileFile); profileFile != "" {
|
||||||
options = append(options, scriptorium.WithProfileFile(profileFile))
|
options = append(options, promptkit.WithProfileFile(profileFile))
|
||||||
}
|
}
|
||||||
options = append(options, cfg.EngineOptions...)
|
options = append(options, cfg.EngineOptions...)
|
||||||
|
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{
|
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||||
ProfileDir: strings.TrimSpace(cfg.ProfileDir),
|
ProfileDir: strings.TrimSpace(cfg.ProfileDir),
|
||||||
Timeout: cfg.Timeout,
|
Timeout: cfg.Timeout,
|
||||||
HTTPClient: cfg.HTTPClient,
|
HTTPClient: cfg.HTTPClient,
|
||||||
}, options...)
|
}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("create Scriptorium engine: %w", err)
|
return nil, fmt.Errorf("create PromptKit engine: %w", err)
|
||||||
}
|
}
|
||||||
recorder := cfg.Recorder
|
recorder := cfg.Recorder
|
||||||
if recorder == nil {
|
if recorder == nil {
|
||||||
recorder = NewLLMProfileRecorder()
|
recorder = NewLLMProfileRecorder()
|
||||||
}
|
}
|
||||||
return &ScriptoriumClient{
|
return &PromptKitClient{
|
||||||
engine: engine,
|
engine: engine,
|
||||||
recorder: recorder,
|
recorder: recorder,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *ScriptoriumClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
if c == nil {
|
if c == nil {
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("scriptorium client must not be nil")
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("PromptKit client must not be nil")
|
||||||
}
|
}
|
||||||
if c.engine == nil {
|
if c.engine == nil {
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("scriptorium client engine must not be nil")
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("PromptKit client engine must not be nil")
|
||||||
}
|
}
|
||||||
if err := validateOutputTarget(out); err != nil {
|
if err := validateOutputTarget(out); err != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, err
|
return contracts.StructuredCompletionResponse{}, err
|
||||||
@@ -90,52 +90,52 @@ func (c *ScriptoriumClient) CompleteStructured(ctx context.Context, req contract
|
|||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion prompt_id must not be empty")
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion prompt_id must not be empty")
|
||||||
}
|
}
|
||||||
|
|
||||||
runReq := scriptorium.RunRequest{
|
runReq := promptkit.RunRequest{
|
||||||
PromptID: promptID,
|
PromptID: promptID,
|
||||||
PromptVersion: strings.TrimSpace(req.PromptVersion),
|
PromptVersion: strings.TrimSpace(req.PromptVersion),
|
||||||
ProfileID: strings.TrimSpace(req.ProfileID),
|
ProfileID: strings.TrimSpace(req.ProfileID),
|
||||||
Inputs: scriptoriumInputs(req.Inputs),
|
Inputs: promptKitInputs(req.Inputs),
|
||||||
Vars: scriptoriumVars(req),
|
Vars: promptKitVars(req),
|
||||||
Metadata: scriptoriumMetadata(req),
|
Metadata: promptKitMetadata(req),
|
||||||
}
|
}
|
||||||
prepared, err := c.engine.Prepare(ctx, runReq)
|
prepared, err := c.engine.Prepare(ctx, runReq)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, ctxErr
|
return contracts.StructuredCompletionResponse{}, ctxErr
|
||||||
}
|
}
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("prepare Scriptorium prompt %q: %w", promptID, redactScriptoriumError(err))
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("prepare PromptKit prompt %q: %w", promptID, redactPromptKitError(err))
|
||||||
}
|
}
|
||||||
result, err := c.engine.Run(ctx, runReq)
|
result, err := c.engine.Run(ctx, runReq)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, ctxErr
|
return contracts.StructuredCompletionResponse{}, ctxErr
|
||||||
}
|
}
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run Scriptorium prompt %q: %w", promptID, redactScriptoriumError(err))
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w", promptID, redactPromptKitError(err))
|
||||||
}
|
}
|
||||||
if result == nil {
|
if result == nil {
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run Scriptorium prompt %q: %w: empty result", promptID, contracts.ErrInvalidStructuredOutput)
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w: empty result", promptID, contracts.ErrInvalidStructuredOutput)
|
||||||
}
|
}
|
||||||
response := c.responseFromResult(result, prepared)
|
response := c.responseFromResult(result, prepared)
|
||||||
if result.Validation.Status == scriptorium.ValidationFailed || !result.Validation.IsValid {
|
if result.Validation.Status == promptkit.ValidationFailed || !result.Validation.IsValid {
|
||||||
return response, fmt.Errorf("run Scriptorium prompt %q: %w: validation failed: %s", promptID, contracts.ErrInvalidStructuredOutput, strings.Join(result.Validation.Errors, "; "))
|
return response, fmt.Errorf("run PromptKit prompt %q: %w: validation failed: %s", promptID, contracts.ErrInvalidStructuredOutput, strings.Join(result.Validation.Errors, "; "))
|
||||||
}
|
}
|
||||||
if len(strings.TrimSpace(string(response.Content))) == 0 {
|
if len(strings.TrimSpace(string(response.Content))) == 0 {
|
||||||
return response, fmt.Errorf("run Scriptorium prompt %q: %w: empty structured output", promptID, contracts.ErrInvalidStructuredOutput)
|
return response, fmt.Errorf("run PromptKit prompt %q: %w: empty structured output", promptID, contracts.ErrInvalidStructuredOutput)
|
||||||
}
|
}
|
||||||
if err := json.Unmarshal(response.Content, out); err != nil {
|
if err := json.Unmarshal(response.Content, out); err != nil {
|
||||||
return response, fmt.Errorf("decode Scriptorium structured output for prompt %q: %w: %w", promptID, contracts.ErrInvalidStructuredOutput, err)
|
return response, fmt.Errorf("decode PromptKit structured output for prompt %q: %w: %w", promptID, contracts.ErrInvalidStructuredOutput, err)
|
||||||
}
|
}
|
||||||
return response, nil
|
return response, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *ScriptoriumClient) responseFromResult(result *scriptorium.RunResult, prepared *scriptorium.PreparedRun) contracts.StructuredCompletionResponse {
|
func (c *PromptKitClient) responseFromResult(result *promptkit.RunResult, prepared *promptkit.PreparedRun) contracts.StructuredCompletionResponse {
|
||||||
content := result.Artifact.Body
|
content := result.Artifact.Body
|
||||||
if len(content) == 0 {
|
if len(content) == 0 {
|
||||||
content = []byte(result.RawOutput)
|
content = []byte(result.RawOutput)
|
||||||
}
|
}
|
||||||
profile := artifacts.LLMProfileManifest{
|
profile := artifacts.LLMProfileManifest{
|
||||||
ID: strings.TrimSpace(result.SelectedProfileID),
|
ID: strings.TrimSpace(result.SelectedProfileID),
|
||||||
Provider: scriptoriumProviderName,
|
Provider: promptKitProviderName,
|
||||||
Model: firstNonEmpty(result.ModelName, result.EffectiveModelParams.Model),
|
Model: firstNonEmpty(result.ModelName, result.EffectiveModelParams.Model),
|
||||||
}
|
}
|
||||||
if c.recorder != nil {
|
if c.recorder != nil {
|
||||||
@@ -149,17 +149,17 @@ func (c *ScriptoriumClient) responseFromResult(result *scriptorium.RunResult, pr
|
|||||||
PromptTokens: result.Usage.PromptTokens,
|
PromptTokens: result.Usage.PromptTokens,
|
||||||
CompletionTokens: result.Usage.CompletionTokens,
|
CompletionTokens: result.Usage.CompletionTokens,
|
||||||
TotalTokens: result.Usage.TotalTokens,
|
TotalTokens: result.Usage.TotalTokens,
|
||||||
Debug: scriptoriumDebugMaterial(prepared, result),
|
Debug: promptKitDebugMaterial(prepared, result),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumDebugMaterial(prepared *scriptorium.PreparedRun, result *scriptorium.RunResult) *contracts.LLMDebugMaterial {
|
func promptKitDebugMaterial(prepared *promptkit.PreparedRun, result *promptkit.RunResult) *contracts.LLMDebugMaterial {
|
||||||
material := &contracts.LLMDebugMaterial{}
|
material := &contracts.LLMDebugMaterial{}
|
||||||
if prepared != nil {
|
if prepared != nil {
|
||||||
material.Prompt = scriptoriumDebugPrompt(prepared)
|
material.Prompt = promptKitDebugPrompt(prepared)
|
||||||
}
|
}
|
||||||
if result != nil {
|
if result != nil {
|
||||||
material.Response = scriptoriumDebugResponse(result)
|
material.Response = promptKitDebugResponse(result)
|
||||||
}
|
}
|
||||||
if material.Prompt == nil && material.Response == nil {
|
if material.Prompt == nil && material.Response == nil {
|
||||||
return nil
|
return nil
|
||||||
@@ -167,7 +167,7 @@ func scriptoriumDebugMaterial(prepared *scriptorium.PreparedRun, result *scripto
|
|||||||
return material
|
return material
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumDebugPrompt(prepared *scriptorium.PreparedRun) *contracts.LLMDebugPrompt {
|
func promptKitDebugPrompt(prepared *promptkit.PreparedRun) *contracts.LLMDebugPrompt {
|
||||||
if prepared == nil {
|
if prepared == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -194,7 +194,7 @@ func scriptoriumDebugPrompt(prepared *scriptorium.PreparedRun) *contracts.LLMDeb
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumDebugResponse(result *scriptorium.RunResult) *contracts.LLMDebugResponse {
|
func promptKitDebugResponse(result *promptkit.RunResult) *contracts.LLMDebugResponse {
|
||||||
if result == nil {
|
if result == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -254,7 +254,7 @@ func cloneStringMap(values map[string]string) map[string]string {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *ScriptoriumClient) LLMProfileManifests() []artifacts.LLMProfileManifest {
|
func (c *PromptKitClient) LLMProfileManifests() []artifacts.LLMProfileManifest {
|
||||||
if c == nil || c.recorder == nil {
|
if c == nil || c.recorder == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -302,11 +302,11 @@ func (r *LLMProfileRecorder) Manifests() []artifacts.LLMProfileManifest {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumInputs(inputs contracts.LLMInputSet) map[string]scriptorium.ArtifactRef {
|
func promptKitInputs(inputs contracts.LLMInputSet) map[string]promptkit.ArtifactRef {
|
||||||
if len(inputs) == 0 {
|
if len(inputs) == 0 {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
out := make(map[string]scriptorium.ArtifactRef, len(inputs))
|
out := make(map[string]promptkit.ArtifactRef, len(inputs))
|
||||||
for key, material := range inputs {
|
for key, material := range inputs {
|
||||||
name := strings.TrimSpace(key)
|
name := strings.TrimSpace(key)
|
||||||
if name == "" {
|
if name == "" {
|
||||||
@@ -320,15 +320,15 @@ func scriptoriumInputs(inputs contracts.LLMInputSet) map[string]scriptorium.Arti
|
|||||||
body = " "
|
body = " "
|
||||||
}
|
}
|
||||||
if origin := strings.TrimSpace(material.OriginURI); origin != "" {
|
if origin := strings.TrimSpace(material.OriginURI); origin != "" {
|
||||||
out[name] = scriptorium.InlineWithURI(origin, body)
|
out[name] = promptkit.InlineWithURI(origin, body)
|
||||||
} else {
|
} else {
|
||||||
out[name] = scriptorium.Inline(body)
|
out[name] = promptkit.Inline(body)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumVars(req contracts.StructuredCompletionRequest) map[string]string {
|
func promptKitVars(req contracts.StructuredCompletionRequest) map[string]string {
|
||||||
vars := make(map[string]string, len(req.Vars)+1)
|
vars := make(map[string]string, len(req.Vars)+1)
|
||||||
for key, value := range req.Vars {
|
for key, value := range req.Vars {
|
||||||
name := strings.TrimSpace(key)
|
name := strings.TrimSpace(key)
|
||||||
@@ -346,7 +346,7 @@ func scriptoriumVars(req contracts.StructuredCompletionRequest) map[string]strin
|
|||||||
return vars
|
return vars
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumMetadata(req contracts.StructuredCompletionRequest) map[string]string {
|
func promptKitMetadata(req contracts.StructuredCompletionRequest) map[string]string {
|
||||||
metadata := map[string]string{}
|
metadata := map[string]string{}
|
||||||
if stageName := strings.TrimSpace(req.StageName); stageName != "" {
|
if stageName := strings.TrimSpace(req.StageName); stageName != "" {
|
||||||
metadata["stage_name"] = stageName
|
metadata["stage_name"] = stageName
|
||||||
@@ -359,7 +359,7 @@ func scriptoriumMetadata(req contracts.StructuredCompletionRequest) map[string]s
|
|||||||
|
|
||||||
var bearerTokenPattern = regexp.MustCompile(`(?i)Bearer\s+[A-Za-z0-9._~+/=-]+`)
|
var bearerTokenPattern = regexp.MustCompile(`(?i)Bearer\s+[A-Za-z0-9._~+/=-]+`)
|
||||||
|
|
||||||
func redactScriptoriumError(err error) error {
|
func redactPromptKitError(err error) error {
|
||||||
if err == nil {
|
if err == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -4,6 +4,8 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
@@ -12,12 +14,12 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestScriptoriumClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
|
func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
|
||||||
fake := &fakeScriptoriumLLM{content: `{"ok":true}`}
|
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||||
client := newTestScriptoriumClient(t, fake)
|
client := newTestPromptKitClient(t, fake)
|
||||||
|
|
||||||
var out struct {
|
var out struct {
|
||||||
OK bool `json:"ok"`
|
OK bool `json:"ok"`
|
||||||
@@ -39,7 +41,7 @@ func TestScriptoriumClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
|
|||||||
if !out.OK {
|
if !out.OK {
|
||||||
t.Fatalf("decoded output OK = false, want true")
|
t.Fatalf("decoded output OK = false, want true")
|
||||||
}
|
}
|
||||||
if resp.Provider != scriptoriumProviderName || resp.Model != "explicit-model" || resp.ProfileID != "explicit-profile" {
|
if resp.Provider != "promptkit" || resp.Model != "explicit-model" || resp.ProfileID != "explicit-profile" {
|
||||||
t.Fatalf("response metadata = %#v", resp)
|
t.Fatalf("response metadata = %#v", resp)
|
||||||
}
|
}
|
||||||
if resp.PromptTokens != 11 || resp.CompletionTokens != 7 || resp.TotalTokens != 18 {
|
if resp.PromptTokens != 11 || resp.CompletionTokens != 7 || resp.TotalTokens != 18 {
|
||||||
@@ -57,6 +59,9 @@ func TestScriptoriumClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
|
|||||||
if resp.Debug.Response == nil || resp.Debug.Response.Content != `{"ok":true}` {
|
if resp.Debug.Response == nil || resp.Debug.Response.Content != `{"ok":true}` {
|
||||||
t.Fatalf("debug response = %#v, want raw response content", resp.Debug.Response)
|
t.Fatalf("debug response = %#v, want raw response content", resp.Debug.Response)
|
||||||
}
|
}
|
||||||
|
if resp.Debug.Response.Usage.CachedTokens != 5 || resp.Debug.Response.Usage.CacheWriteTokens != 3 {
|
||||||
|
t.Fatalf("debug usage = %#v, want cached token counts", resp.Debug.Response.Usage)
|
||||||
|
}
|
||||||
debugJSON, err := json.Marshal(resp.Debug)
|
debugJSON, err := json.Marshal(resp.Debug)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("marshal debug material: %v", err)
|
t.Fatalf("marshal debug material: %v", err)
|
||||||
@@ -71,21 +76,52 @@ func TestScriptoriumClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
|
|||||||
if gotReq.Target.Model != "explicit-model" {
|
if gotReq.Target.Model != "explicit-model" {
|
||||||
t.Fatalf("model = %q, want explicit-model", gotReq.Target.Model)
|
t.Fatalf("model = %q, want explicit-model", gotReq.Target.Model)
|
||||||
}
|
}
|
||||||
if len(gotReq.Prompt.Messages) != 1 || !strings.Contains(gotReq.Prompt.Messages[0].Content, `{"source":true}`) {
|
if len(gotReq.Prompt.Messages) != 1 ||
|
||||||
t.Fatalf("rendered messages = %#v, want transcript input content", gotReq.Prompt.Messages)
|
!strings.Contains(gotReq.Prompt.Messages[0].Content, `{"source":true}`) ||
|
||||||
|
!strings.Contains(gotReq.Prompt.Messages[0].Content, "value") {
|
||||||
|
t.Fatalf("rendered messages = %#v, want transcript input and variable content", gotReq.Prompt.Messages)
|
||||||
}
|
}
|
||||||
if gotReq.StructuredOutput == nil {
|
if gotReq.StructuredOutput == nil {
|
||||||
t.Fatalf("structured output = nil, want JSON schema")
|
t.Fatalf("structured output = nil, want JSON schema")
|
||||||
}
|
}
|
||||||
manifests := client.LLMProfileManifests()
|
manifests := client.LLMProfileManifests()
|
||||||
if len(manifests) != 1 || manifests[0].ID != "explicit-profile" || manifests[0].Model != "explicit-model" {
|
if len(manifests) != 1 ||
|
||||||
|
manifests[0].ID != "explicit-profile" ||
|
||||||
|
manifests[0].Provider != "promptkit" ||
|
||||||
|
manifests[0].Model != "explicit-model" {
|
||||||
t.Fatalf("profile manifests = %#v", manifests)
|
t.Fatalf("profile manifests = %#v", manifests)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestScriptoriumClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) {
|
func TestNewPromptKitClientReportsAssetAndEngineConstructionFailures(t *testing.T) {
|
||||||
fake := &fakeScriptoriumLLM{content: `{"ok":true}`}
|
t.Run("assets", func(t *testing.T) {
|
||||||
client := newTestScriptoriumClient(t, fake)
|
registry := NewAssetRegistry()
|
||||||
|
for _, root := range []string{"one", "two"} {
|
||||||
|
if err := registry.RegisterPromptFS(fstest.MapFS{
|
||||||
|
root + "/prompt.yaml": {Data: []byte("id: duplicate")},
|
||||||
|
}, root); err != nil {
|
||||||
|
t.Fatalf("RegisterPromptFS() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, err := NewPromptKitClient(PromptKitClientConfig{Assets: registry}); err == nil ||
|
||||||
|
!strings.Contains(err.Error(), "duplicate asset path") {
|
||||||
|
t.Fatalf("NewPromptKitClient() error = %v, want asset construction failure", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("engine", func(t *testing.T) {
|
||||||
|
if _, err := NewPromptKitClient(PromptKitClientConfig{
|
||||||
|
Assets: newTestPromptKitAssets(t),
|
||||||
|
EngineOptions: []promptkit.Option{promptkit.WithProfileFile("")},
|
||||||
|
}); err == nil || !strings.Contains(err.Error(), "create PromptKit engine") {
|
||||||
|
t.Fatalf("NewPromptKitClient() error = %v, want engine construction failure", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) {
|
||||||
|
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||||
|
client := newTestPromptKitClient(t, fake)
|
||||||
|
|
||||||
var out map[string]any
|
var out map[string]any
|
||||||
if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||||
@@ -102,8 +138,8 @@ func TestScriptoriumClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *tes
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestScriptoriumClientValidationFailureReturnsError(t *testing.T) {
|
func TestPromptKitClientValidationFailureReturnsError(t *testing.T) {
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"bad":true}`})
|
client := newTestPromptKitClient(t, &fakePromptKitLLM{content: `{"bad":true}`})
|
||||||
|
|
||||||
var out map[string]any
|
var out map[string]any
|
||||||
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||||
@@ -127,8 +163,8 @@ func TestScriptoriumClientValidationFailureReturnsError(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestScriptoriumClientDecodeFailureReturnsRawResponse(t *testing.T) {
|
func TestPromptKitClientDecodeFailureReturnsRawResponse(t *testing.T) {
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"ok":true}`})
|
client := newTestPromptKitClient(t, &fakePromptKitLLM{content: `{"ok":true}`})
|
||||||
|
|
||||||
var out []any
|
var out []any
|
||||||
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||||
@@ -138,7 +174,7 @@ func TestScriptoriumClientDecodeFailureReturnsRawResponse(t *testing.T) {
|
|||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||||
},
|
},
|
||||||
}, &out)
|
}, &out)
|
||||||
if err == nil || !errors.Is(err, contracts.ErrInvalidStructuredOutput) || !strings.Contains(err.Error(), "decode Scriptorium structured output") {
|
if err == nil || !errors.Is(err, contracts.ErrInvalidStructuredOutput) || !strings.Contains(err.Error(), "decode PromptKit structured output") {
|
||||||
t.Fatalf("CompleteStructured() error = %v, want decode failure", err)
|
t.Fatalf("CompleteStructured() error = %v, want decode failure", err)
|
||||||
}
|
}
|
||||||
if got := string(resp.Content); got != `{"ok":true}` {
|
if got := string(resp.Content); got != `{"ok":true}` {
|
||||||
@@ -149,8 +185,8 @@ func TestScriptoriumClientDecodeFailureReturnsRawResponse(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestScriptoriumClientProviderFailureIncludesContextAndRedactsBearerToken(t *testing.T) {
|
func TestPromptKitClientProviderFailureIncludesContextAndRedactsBearerToken(t *testing.T) {
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{err: errors.New("provider failed with Bearer secret-token")})
|
client := newTestPromptKitClient(t, &fakePromptKitLLM{err: errors.New("provider failed with Bearer secret-token")})
|
||||||
|
|
||||||
var out map[string]any
|
var out map[string]any
|
||||||
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||||
@@ -166,7 +202,7 @@ func TestScriptoriumClientProviderFailureIncludesContextAndRedactsBearerToken(t
|
|||||||
if errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
if errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
||||||
t.Fatalf("provider error = %v, must not be classified as invalid structured output", err)
|
t.Fatalf("provider error = %v, must not be classified as invalid structured output", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(err.Error(), `run Scriptorium prompt "adapter.test"`) {
|
if !strings.Contains(err.Error(), `run PromptKit prompt "adapter.test"`) {
|
||||||
t.Fatalf("error = %q, want operation context", err.Error())
|
t.Fatalf("error = %q, want operation context", err.Error())
|
||||||
}
|
}
|
||||||
if strings.Contains(err.Error(), "secret-token") || !strings.Contains(err.Error(), "Bearer [REDACTED]") {
|
if strings.Contains(err.Error(), "secret-token") || !strings.Contains(err.Error(), "Bearer [REDACTED]") {
|
||||||
@@ -177,10 +213,10 @@ func TestScriptoriumClientProviderFailureIncludesContextAndRedactsBearerToken(t
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestScriptoriumClientContextCancellationIsRespected(t *testing.T) {
|
func TestPromptKitClientContextCancellationIsRespected(t *testing.T) {
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
cancel()
|
cancel()
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"ok":true}`})
|
client := newTestPromptKitClient(t, &fakePromptKitLLM{content: `{"ok":true}`})
|
||||||
|
|
||||||
var out map[string]any
|
var out map[string]any
|
||||||
_, err := client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
_, err := client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||||
@@ -194,8 +230,57 @@ func TestScriptoriumClientContextCancellationIsRespected(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestScriptoriumClientClassifiesEmptyStructuredCompletion(t *testing.T) {
|
func TestPromptKitClientForwardsConfiguredTransportTimeout(t *testing.T) {
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{allowEmpty: true})
|
var remaining time.Duration
|
||||||
|
transport := roundTripFunc(func(req *http.Request) (*http.Response, error) {
|
||||||
|
deadline, ok := req.Context().Deadline()
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("outbound request context has no deadline")
|
||||||
|
}
|
||||||
|
remaining = time.Until(deadline)
|
||||||
|
return &http.Response{
|
||||||
|
StatusCode: http.StatusOK,
|
||||||
|
Status: "200 OK",
|
||||||
|
Header: make(http.Header),
|
||||||
|
Body: io.NopCloser(strings.NewReader(
|
||||||
|
`{"choices":[{"message":{"content":"{\"ok\":true}"}}]}`,
|
||||||
|
)),
|
||||||
|
Request: req,
|
||||||
|
}, nil
|
||||||
|
})
|
||||||
|
const configuredTimeout = 2 * time.Second
|
||||||
|
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||||
|
Assets: newTestPromptKitAssets(t),
|
||||||
|
Timeout: configuredTimeout,
|
||||||
|
HTTPClient: &http.Client{Transport: transport},
|
||||||
|
EngineOptions: []promptkit.Option{
|
||||||
|
promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
|
ID: "default-profile",
|
||||||
|
Endpoint: "http://promptkit.test/v1",
|
||||||
|
Model: "default-model",
|
||||||
|
})),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewPromptKitClient() error = %v", err)
|
||||||
|
}
|
||||||
|
var out map[string]any
|
||||||
|
if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||||
|
PromptID: "adapter.test",
|
||||||
|
SessionID: "session-123",
|
||||||
|
Inputs: contracts.LLMInputSet{
|
||||||
|
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||||
|
},
|
||||||
|
}, &out); err != nil {
|
||||||
|
t.Fatalf("CompleteStructured() error = %v", err)
|
||||||
|
}
|
||||||
|
if remaining < configuredTimeout-time.Second || remaining > configuredTimeout {
|
||||||
|
t.Fatalf("transport deadline remaining = %v, want near %v", remaining, configuredTimeout)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptKitClientClassifiesEmptyStructuredCompletion(t *testing.T) {
|
||||||
|
client := newTestPromptKitClient(t, &fakePromptKitLLM{allowEmpty: true})
|
||||||
|
|
||||||
var out map[string]any
|
var out map[string]any
|
||||||
_, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
_, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||||
@@ -210,12 +295,12 @@ func TestScriptoriumClientClassifiesEmptyStructuredCompletion(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestScheduledScriptoriumClientBoundsConcurrentCalls(t *testing.T) {
|
func TestScheduledPromptKitClientBoundsConcurrentCalls(t *testing.T) {
|
||||||
fake := &fakeScriptoriumLLM{
|
fake := &fakePromptKitLLM{
|
||||||
content: `{"ok":true}`,
|
content: `{"ok":true}`,
|
||||||
block: make(chan struct{}),
|
block: make(chan struct{}),
|
||||||
}
|
}
|
||||||
client := newTestScriptoriumClient(t, fake)
|
client := newTestPromptKitClient(t, fake)
|
||||||
scheduler, err := NewScheduler(1)
|
scheduler, err := NewScheduler(1)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("NewScheduler() error = %v, want nil", err)
|
t.Fatalf("NewScheduler() error = %v, want nil", err)
|
||||||
@@ -249,8 +334,8 @@ func TestScheduledScriptoriumClientBoundsConcurrentCalls(t *testing.T) {
|
|||||||
wg.Wait()
|
wg.Wait()
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestScriptoriumClientValidatesRequest(t *testing.T) {
|
func TestPromptKitClientValidatesRequest(t *testing.T) {
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"ok":true}`})
|
client := newTestPromptKitClient(t, &fakePromptKitLLM{content: `{"ok":true}`})
|
||||||
var out map[string]any
|
var out map[string]any
|
||||||
if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{}, &out); err == nil || !strings.Contains(err.Error(), "prompt_id") {
|
if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{}, &out); err == nil || !strings.Contains(err.Error(), "prompt_id") {
|
||||||
t.Fatalf("missing prompt id error = %v, want prompt_id validation", err)
|
t.Fatalf("missing prompt id error = %v, want prompt_id validation", err)
|
||||||
@@ -260,7 +345,34 @@ func TestScriptoriumClientValidatesRequest(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func newTestScriptoriumClient(t *testing.T, fake *fakeScriptoriumLLM) *ScriptoriumClient {
|
func newTestPromptKitClient(t *testing.T, fake *fakePromptKitLLM) *PromptKitClient {
|
||||||
|
t.Helper()
|
||||||
|
registry := newTestPromptKitAssets(t)
|
||||||
|
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||||
|
Assets: registry,
|
||||||
|
EngineOptions: []promptkit.Option{
|
||||||
|
promptkit.WithProfiles(
|
||||||
|
promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
|
ID: "default-profile",
|
||||||
|
Endpoint: "http://127.0.0.1:1/v1",
|
||||||
|
Model: "default-model",
|
||||||
|
}),
|
||||||
|
promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
|
ID: "explicit-profile",
|
||||||
|
Endpoint: "http://127.0.0.1:1/v1",
|
||||||
|
Model: "explicit-model",
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
promptkit.WithLLMClient(fake),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewPromptKitClient() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
return client
|
||||||
|
}
|
||||||
|
|
||||||
|
func newTestPromptKitAssets(t *testing.T) *AssetRegistry {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
registry := NewAssetRegistry()
|
registry := NewAssetRegistry()
|
||||||
if err := registry.RegisterPromptFS(fstest.MapFS{
|
if err := registry.RegisterPromptFS(fstest.MapFS{
|
||||||
@@ -274,7 +386,7 @@ inputs:
|
|||||||
content_type: application/json
|
content_type: application/json
|
||||||
messages:
|
messages:
|
||||||
- role: user
|
- role: user
|
||||||
content: "Transcript: {{ input \"transcript\" }}"
|
content: "Transcript: {{ input \"transcript\" }} Custom: {{ index . \"custom\" }}"
|
||||||
output:
|
output:
|
||||||
format: json
|
format: json
|
||||||
validation_mode: json_schema
|
validation_mode: json_schema
|
||||||
@@ -289,43 +401,22 @@ output:
|
|||||||
}, "."); err != nil {
|
}, "."); err != nil {
|
||||||
t.Fatalf("RegisterSchemaFS() error = %v", err)
|
t.Fatalf("RegisterSchemaFS() error = %v", err)
|
||||||
}
|
}
|
||||||
client, err := NewScriptoriumClient(ScriptoriumClientConfig{
|
return registry
|
||||||
Assets: registry,
|
|
||||||
EngineOptions: []scriptorium.Option{
|
|
||||||
scriptorium.WithProfiles(
|
|
||||||
scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
|
||||||
ID: "default-profile",
|
|
||||||
Endpoint: "http://127.0.0.1:1/v1",
|
|
||||||
Model: "default-model",
|
|
||||||
}),
|
|
||||||
scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
|
||||||
ID: "explicit-profile",
|
|
||||||
Endpoint: "http://127.0.0.1:1/v1",
|
|
||||||
Model: "explicit-model",
|
|
||||||
}),
|
|
||||||
),
|
|
||||||
scriptorium.WithLLMClient(fake),
|
|
||||||
},
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("NewScriptoriumClient() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
return client
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type fakeScriptoriumLLM struct {
|
type fakePromptKitLLM struct {
|
||||||
content string
|
content string
|
||||||
allowEmpty bool
|
allowEmpty bool
|
||||||
err error
|
err error
|
||||||
block chan struct{}
|
block chan struct{}
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
last scriptorium.GenerateRequest
|
last promptkit.GenerateRequest
|
||||||
calls int32
|
calls int32
|
||||||
inFlight int32
|
inFlight int32
|
||||||
maxInFlight int32
|
maxInFlight int32
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeScriptoriumLLM) Generate(ctx context.Context, req scriptorium.GenerateRequest) (*scriptorium.GenerateResponse, error) {
|
func (f *fakePromptKitLLM) Generate(ctx context.Context, req promptkit.GenerateRequest) (*promptkit.GenerateResponse, error) {
|
||||||
f.mu.Lock()
|
f.mu.Lock()
|
||||||
f.last = req
|
f.last = req
|
||||||
f.mu.Unlock()
|
f.mu.Unlock()
|
||||||
@@ -355,17 +446,25 @@ func (f *fakeScriptoriumLLM) Generate(ctx context.Context, req scriptorium.Gener
|
|||||||
if !f.allowEmpty && !json.Valid([]byte(content)) {
|
if !f.allowEmpty && !json.Valid([]byte(content)) {
|
||||||
return nil, errors.New("test fake must return JSON content")
|
return nil, errors.New("test fake must return JSON content")
|
||||||
}
|
}
|
||||||
return &scriptorium.GenerateResponse{
|
return &promptkit.GenerateResponse{
|
||||||
Content: content,
|
Content: content,
|
||||||
Usage: scriptorium.TokenUsage{
|
Usage: promptkit.TokenUsage{
|
||||||
PromptTokens: 11,
|
PromptTokens: 11,
|
||||||
CompletionTokens: 7,
|
CompletionTokens: 7,
|
||||||
TotalTokens: 18,
|
TotalTokens: 18,
|
||||||
|
CachedTokens: 5,
|
||||||
|
CacheWriteTokens: 3,
|
||||||
},
|
},
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeScriptoriumLLM) lastRequest() scriptorium.GenerateRequest {
|
type roundTripFunc func(*http.Request) (*http.Response, error)
|
||||||
|
|
||||||
|
func (f roundTripFunc) RoundTrip(req *http.Request) (*http.Response, error) {
|
||||||
|
return f(req)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakePromptKitLLM) lastRequest() promptkit.GenerateRequest {
|
||||||
f.mu.Lock()
|
f.mu.Lock()
|
||||||
defer f.mu.Unlock()
|
defer f.mu.Unlock()
|
||||||
return f.last
|
return f.last
|
||||||
@@ -12,14 +12,14 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// ModulePromptFile maps a module-owned embedded prompt file into the
|
// ModulePromptFile maps a module-owned embedded prompt file into the
|
||||||
// Scriptorium-visible module prompt directory.
|
// PromptKit-visible module prompt directory.
|
||||||
type ModulePromptFile struct {
|
type ModulePromptFile struct {
|
||||||
Name string
|
Name string
|
||||||
Path string
|
Path string
|
||||||
}
|
}
|
||||||
|
|
||||||
// SharedPromptFile maps a caller-owned shared prompt file into a module's
|
// SharedPromptFile maps a caller-owned shared prompt file into a module's
|
||||||
// Scriptorium-visible sharedassets prompt subdirectory.
|
// PromptKit-visible sharedassets prompt subdirectory.
|
||||||
type SharedPromptFile struct {
|
type SharedPromptFile struct {
|
||||||
Name string
|
Name string
|
||||||
FS fs.FS
|
FS fs.FS
|
||||||
|
|||||||
@@ -56,7 +56,7 @@ func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (c *Chunker) ManifestMetadata() map[string]any {
|
func (c *Chunker) ManifestMetadata() map[string]any {
|
||||||
promptSHA, err := scriptoriumPromptMetadata()
|
promptSHA, err := promptAssetMetadata()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
promptSHA = ""
|
promptSHA = ""
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
)
|
)
|
||||||
|
|
||||||
const scriptoriumPromptRoot = "assets/prompts"
|
const promptAssetRoot = "assets/prompts"
|
||||||
|
|
||||||
var promptAssetManifest = shared.PromptAssetManifest{
|
var promptAssetManifest = shared.PromptAssetManifest{
|
||||||
ModuleDir: "dnd.scenes",
|
ModuleDir: "dnd.scenes",
|
||||||
@@ -30,21 +30,21 @@ func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("prepare scene prompt assets: %w", err)
|
return fmt.Errorf("prepare scene prompt assets: %w", err)
|
||||||
}
|
}
|
||||||
if err := registry.RegisterPromptFS(promptFS, scriptoriumPromptRoot); err != nil {
|
if err := registry.RegisterPromptFS(promptFS, promptAssetRoot); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumPromptMetadata() (string, error) {
|
func promptAssetMetadata() (string, error) {
|
||||||
scriptoriumPromptHashOnce.Do(func() {
|
promptAssetHashOnce.Do(func() {
|
||||||
scriptoriumPromptHash, scriptoriumPromptHashErr = promptAssetManifest.Hash(embeddedAssets)
|
promptAssetHash, promptAssetHashErr = promptAssetManifest.Hash(embeddedAssets)
|
||||||
})
|
})
|
||||||
return scriptoriumPromptHash, scriptoriumPromptHashErr
|
return promptAssetHash, promptAssetHashErr
|
||||||
}
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
scriptoriumPromptHashOnce sync.Once
|
promptAssetHashOnce sync.Once
|
||||||
scriptoriumPromptHash string
|
promptAssetHash string
|
||||||
scriptoriumPromptHashErr error
|
promptAssetHashErr error
|
||||||
)
|
)
|
||||||
@@ -8,10 +8,10 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestScriptoriumPromptPreparesTranscriptAndTaskMessages(t *testing.T) {
|
func TestPromptAssetsPrepareTranscriptAndTaskMessages(t *testing.T) {
|
||||||
transcript := []byte(`{"sentinel":"scene-transcript"}`)
|
transcript := []byte(`{"sentinel":"scene-transcript"}`)
|
||||||
prepared := prepareScenesPrompt(t, transcript, "scene-players", "scene-party", "scene-glossary")
|
prepared := prepareScenesPrompt(t, transcript, "scene-players", "scene-party", "scene-glossary")
|
||||||
|
|
||||||
@@ -27,7 +27,7 @@ func TestScriptoriumPromptPreparesTranscriptAndTaskMessages(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
for _, index := range []int{1, 4} {
|
for _, index := range []int{1, 4} {
|
||||||
if cache := prepared.Messages[index].CacheControl; cache == nil || cache.Type != scriptorium.CacheControlEphemeral {
|
if cache := prepared.Messages[index].CacheControl; cache == nil || cache.Type != promptkit.CacheControlEphemeral {
|
||||||
t.Errorf("message %d cache control = %#v, want ephemeral", index, cache)
|
t.Errorf("message %d cache control = %#v, want ephemeral", index, cache)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -52,7 +52,7 @@ func TestScriptoriumPromptPreparesTranscriptAndTaskMessages(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestScriptoriumPromptDiagnosticsOmitRawMaterials(t *testing.T) {
|
func TestPromptAssetDiagnosticsOmitRawMaterials(t *testing.T) {
|
||||||
transcript := []byte(`{"secret":"source text"}`)
|
transcript := []byte(`{"secret":"source text"}`)
|
||||||
prepared := prepareScenesPrompt(t, transcript, "private player note", "private party note", "private glossary note")
|
prepared := prepareScenesPrompt(t, transcript, "private player note", "private party note", "private glossary note")
|
||||||
metadata := newChunker(t, &fakeScenesLLMClient{}).ManifestMetadata()
|
metadata := newChunker(t, &fakeScenesLLMClient{}).ManifestMetadata()
|
||||||
@@ -94,22 +94,22 @@ func TestScriptoriumPromptDiagnosticsOmitRawMaterials(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func prepareScenesPrompt(t *testing.T, transcript []byte, players string, party string, glossary string) *scriptorium.PreparedRun {
|
func prepareScenesPrompt(t *testing.T, transcript []byte, players string, party string, glossary string) *promptkit.PreparedRun {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
registry := llm.NewAssetRegistry()
|
registry := llm.NewAssetRegistry()
|
||||||
if err := RegisterPromptAssets(registry); err != nil {
|
if err := RegisterPromptAssets(registry); err != nil {
|
||||||
t.Fatalf("register scene prompt assets: %v", err)
|
t.Fatalf("register scene prompt assets: %v", err)
|
||||||
}
|
}
|
||||||
engine := newScenesScriptoriumEngine(t, registry)
|
engine := newScenesPromptEngine(t, registry)
|
||||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: PromptID,
|
PromptID: PromptID,
|
||||||
PromptVersion: ResponseSchemaVersion,
|
PromptVersion: ResponseSchemaVersion,
|
||||||
ProfileID: "scene-test-profile",
|
ProfileID: "scene-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.InlineWithURI("file:///session.json", string(transcript)),
|
"transcript": promptkit.InlineWithURI("file:///session.json", string(transcript)),
|
||||||
"players": scriptorium.Inline(players),
|
"players": promptkit.Inline(players),
|
||||||
"party": scriptorium.Inline(party),
|
"party": promptkit.Inline(party),
|
||||||
"glossary": scriptorium.Inline(glossary),
|
"glossary": promptkit.Inline(glossary),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -118,18 +118,18 @@ func prepareScenesPrompt(t *testing.T, transcript []byte, players string, party
|
|||||||
return prepared
|
return prepared
|
||||||
}
|
}
|
||||||
|
|
||||||
func newScenesScriptoriumEngine(t *testing.T, registry *llm.AssetRegistry) *scriptorium.Engine {
|
func newScenesPromptEngine(t *testing.T, registry *llm.AssetRegistry) *promptkit.Engine {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
options, err := registry.ScriptoriumOptions()
|
options, err := registry.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ScriptoriumOptions() error = %v, want nil", err)
|
t.Fatalf("PromptKitOptions() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
ID: "scene-test-profile",
|
ID: "scene-test-profile",
|
||||||
Endpoint: "http://127.0.0.1:1/v1",
|
Endpoint: "http://127.0.0.1:1/v1",
|
||||||
Model: "scene-test-model",
|
Model: "scene-test-model",
|
||||||
})))
|
})))
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("NewEngine() error = %v, want nil", err)
|
t.Fatalf("NewEngine() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
@@ -96,7 +96,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, extractorErrorf("prepare scene eligibility: %w", err)
|
return nil, extractorErrorf("prepare scene eligibility: %w", err)
|
||||||
}
|
}
|
||||||
promptSHA, err := scriptoriumPromptMetadata()
|
promptSHA, err := promptAssetMetadata()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, extractorErrorf("load prompt metadata: %w", err)
|
return nil, extractorErrorf("load prompt metadata: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
)
|
)
|
||||||
|
|
||||||
const scriptoriumPromptRoot = "assets/prompts"
|
const promptAssetRoot = "assets/prompts"
|
||||||
|
|
||||||
var promptAssetManifest = shared.PromptAssetManifest{
|
var promptAssetManifest = shared.PromptAssetManifest{
|
||||||
ModuleDir: "dnd.combat_turns",
|
ModuleDir: "dnd.combat_turns",
|
||||||
@@ -33,21 +33,21 @@ func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("prepare combat-turn prompt assets: %w", err)
|
return fmt.Errorf("prepare combat-turn prompt assets: %w", err)
|
||||||
}
|
}
|
||||||
if err := registry.RegisterPromptFS(promptFS, scriptoriumPromptRoot); err != nil {
|
if err := registry.RegisterPromptFS(promptFS, promptAssetRoot); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumPromptMetadata() (string, error) {
|
func promptAssetMetadata() (string, error) {
|
||||||
scriptoriumPromptHashOnce.Do(func() {
|
promptAssetHashOnce.Do(func() {
|
||||||
scriptoriumPromptHash, scriptoriumPromptHashErr = promptAssetManifest.Hash(embeddedAssets)
|
promptAssetHash, promptAssetHashErr = promptAssetManifest.Hash(embeddedAssets)
|
||||||
})
|
})
|
||||||
return scriptoriumPromptHash, scriptoriumPromptHashErr
|
return promptAssetHash, promptAssetHashErr
|
||||||
}
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
scriptoriumPromptHashOnce sync.Once
|
promptAssetHashOnce sync.Once
|
||||||
scriptoriumPromptHash string
|
promptAssetHash string
|
||||||
scriptoriumPromptHashErr error
|
promptAssetHashErr error
|
||||||
)
|
)
|
||||||
@@ -8,7 +8,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestRegisterPromptAssetsAndPrepareCombatPrompt(t *testing.T) {
|
func TestRegisterPromptAssetsAndPrepareCombatPrompt(t *testing.T) {
|
||||||
@@ -23,37 +23,37 @@ func TestRegisterPromptAssetsAndPrepareCombatPrompt(t *testing.T) {
|
|||||||
if _, err := fs.ReadFile(schemaFS, "dnd_combat_turns_llm.v1.json"); err != nil {
|
if _, err := fs.ReadFile(schemaFS, "dnd_combat_turns_llm.v1.json"); err != nil {
|
||||||
t.Fatalf("response schema asset: %v", err)
|
t.Fatalf("response schema asset: %v", err)
|
||||||
}
|
}
|
||||||
hash, err := scriptoriumPromptMetadata()
|
hash, err := promptAssetMetadata()
|
||||||
if err != nil || !strings.HasPrefix(hash, "sha256:") {
|
if err != nil || !strings.HasPrefix(hash, "sha256:") {
|
||||||
t.Fatalf("scriptoriumPromptMetadata() = %q, %v; want digest", hash, err)
|
t.Fatalf("promptAssetMetadata() = %q, %v; want digest", hash, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestScriptoriumPromptPreparesRequiredInputs(t *testing.T) {
|
func TestPromptAssetsPrepareRequiredInputs(t *testing.T) {
|
||||||
registry := llm.NewAssetRegistry()
|
registry := llm.NewAssetRegistry()
|
||||||
if err := RegisterPromptAssets(registry); err != nil {
|
if err := RegisterPromptAssets(registry); err != nil {
|
||||||
t.Fatalf("RegisterPromptAssets() error = %v, want nil", err)
|
t.Fatalf("RegisterPromptAssets() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
options, err := registry.ScriptoriumOptions()
|
options, err := registry.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ScriptoriumOptions() error = %v, want nil", err)
|
t.Fatalf("PromptKitOptions() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
ID: "combat-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "combat-test-model",
|
ID: "combat-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "combat-test-model",
|
||||||
})))
|
})))
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("NewEngine() error = %v, want nil", err)
|
t.Fatalf("NewEngine() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
transcript := `{"units":[1]}`
|
transcript := `{"units":[1]}`
|
||||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "combat-test-profile",
|
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "combat-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.InlineWithURI("file:///session.json", transcript),
|
"transcript": promptkit.InlineWithURI("file:///session.json", transcript),
|
||||||
"players": scriptorium.Inline(" "),
|
"players": promptkit.Inline(" "),
|
||||||
"party": scriptorium.Inline(" "),
|
"party": promptkit.Inline(" "),
|
||||||
"glossary": scriptorium.Inline(" "),
|
"glossary": promptkit.Inline(" "),
|
||||||
"npcs": scriptorium.Inline(" "),
|
"npcs": promptkit.Inline(" "),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -54,7 +54,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
|
|||||||
if len(references) > 1 {
|
if len(references) > 1 {
|
||||||
return nil, extractorErrorf("at most one reference set may be supplied")
|
return nil, extractorErrorf("at most one reference set may be supplied")
|
||||||
}
|
}
|
||||||
promptSHA, err := scriptoriumPromptMetadata()
|
promptSHA, err := promptAssetMetadata()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, extractorErrorf("load prompt metadata: %w", err)
|
return nil, extractorErrorf("load prompt metadata: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
)
|
)
|
||||||
|
|
||||||
const scriptoriumPromptRoot = "assets/prompts"
|
const promptAssetRoot = "assets/prompts"
|
||||||
|
|
||||||
var promptAssetManifest = shared.PromptAssetManifest{
|
var promptAssetManifest = shared.PromptAssetManifest{
|
||||||
ModuleDir: "dnd.item_events",
|
ModuleDir: "dnd.item_events",
|
||||||
@@ -32,21 +32,21 @@ func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("prepare item event prompt assets: %w", err)
|
return fmt.Errorf("prepare item event prompt assets: %w", err)
|
||||||
}
|
}
|
||||||
if err := registry.RegisterPromptFS(promptFS, scriptoriumPromptRoot); err != nil {
|
if err := registry.RegisterPromptFS(promptFS, promptAssetRoot); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumPromptMetadata() (string, error) {
|
func promptAssetMetadata() (string, error) {
|
||||||
scriptoriumPromptHashOnce.Do(func() {
|
promptAssetHashOnce.Do(func() {
|
||||||
scriptoriumPromptHash, scriptoriumPromptHashErr = promptAssetManifest.Hash(embeddedAssets)
|
promptAssetHash, promptAssetHashErr = promptAssetManifest.Hash(embeddedAssets)
|
||||||
})
|
})
|
||||||
return scriptoriumPromptHash, scriptoriumPromptHashErr
|
return promptAssetHash, promptAssetHashErr
|
||||||
}
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
scriptoriumPromptHashOnce sync.Once
|
promptAssetHashOnce sync.Once
|
||||||
scriptoriumPromptHash string
|
promptAssetHash string
|
||||||
scriptoriumPromptHashErr error
|
promptAssetHashErr error
|
||||||
)
|
)
|
||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestPromptAssetsPrepareItemEventPrompt(t *testing.T) {
|
func TestPromptAssetsPrepareItemEventPrompt(t *testing.T) {
|
||||||
@@ -17,24 +17,24 @@ func TestPromptAssetsPrepareItemEventPrompt(t *testing.T) {
|
|||||||
if err := RegisterPromptAssets(registry); err != nil {
|
if err := RegisterPromptAssets(registry); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
options, err := registry.ScriptoriumOptions()
|
options, err := registry.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
ID: "item-events-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "item-events-test-model",
|
ID: "item-events-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "item-events-test-model",
|
||||||
})))
|
})))
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "item-events-test-profile",
|
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "item-events-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.InlineWithURI("file:///session.json", `{"segments":[1]}`),
|
"transcript": promptkit.InlineWithURI("file:///session.json", `{"segments":[1]}`),
|
||||||
"players": scriptorium.Inline(" "),
|
"players": promptkit.Inline(" "),
|
||||||
"party": scriptorium.Inline(" "),
|
"party": promptkit.Inline(" "),
|
||||||
"glossary": scriptorium.Inline(" "),
|
"glossary": promptkit.Inline(" "),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -46,9 +46,9 @@ func TestPromptAssetsPrepareItemEventPrompt(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestPromptAssetsDoNotLeakIntoMetadata(t *testing.T) {
|
func TestPromptAssetsDoNotLeakIntoMetadata(t *testing.T) {
|
||||||
hash, err := scriptoriumPromptMetadata()
|
hash, err := promptAssetMetadata()
|
||||||
if err != nil || !strings.HasPrefix(hash, "sha256:") {
|
if err != nil || !strings.HasPrefix(hash, "sha256:") {
|
||||||
t.Fatalf("scriptoriumPromptMetadata() = %q, %v", hash, err)
|
t.Fatalf("promptAssetMetadata() = %q, %v", hash, err)
|
||||||
}
|
}
|
||||||
metadata := newExtractor(t, &fakeItemEventsLLMClient{}).ManifestMetadata()
|
metadata := newExtractor(t, &fakeItemEventsLLMClient{}).ManifestMetadata()
|
||||||
payload, err := json.Marshal(metadata)
|
payload, err := json.Marshal(metadata)
|
||||||
@@ -80,7 +80,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, extractorErrorf("prepare NPC registry prompt input: %w", err)
|
return nil, extractorErrorf("prepare NPC registry prompt input: %w", err)
|
||||||
}
|
}
|
||||||
promptSHA, err := scriptoriumPromptMetadata()
|
promptSHA, err := promptAssetMetadata()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, extractorErrorf("load prompt metadata: %w", err)
|
return nil, extractorErrorf("load prompt metadata: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
)
|
)
|
||||||
|
|
||||||
const scriptoriumPromptRoot = "assets/prompts"
|
const promptAssetRoot = "assets/prompts"
|
||||||
|
|
||||||
var promptAssetManifest = shared.PromptAssetManifest{
|
var promptAssetManifest = shared.PromptAssetManifest{
|
||||||
ModuleDir: "dnd.npc_interactions",
|
ModuleDir: "dnd.npc_interactions",
|
||||||
@@ -33,21 +33,21 @@ func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("prepare NPC-interaction prompt assets: %w", err)
|
return fmt.Errorf("prepare NPC-interaction prompt assets: %w", err)
|
||||||
}
|
}
|
||||||
if err := registry.RegisterPromptFS(promptFS, scriptoriumPromptRoot); err != nil {
|
if err := registry.RegisterPromptFS(promptFS, promptAssetRoot); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumPromptMetadata() (string, error) {
|
func promptAssetMetadata() (string, error) {
|
||||||
scriptoriumPromptHashOnce.Do(func() {
|
promptAssetHashOnce.Do(func() {
|
||||||
scriptoriumPromptHash, scriptoriumPromptHashErr = promptAssetManifest.Hash(embeddedAssets)
|
promptAssetHash, promptAssetHashErr = promptAssetManifest.Hash(embeddedAssets)
|
||||||
})
|
})
|
||||||
return scriptoriumPromptHash, scriptoriumPromptHashErr
|
return promptAssetHash, promptAssetHashErr
|
||||||
}
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
scriptoriumPromptHashOnce sync.Once
|
promptAssetHashOnce sync.Once
|
||||||
scriptoriumPromptHash string
|
promptAssetHash string
|
||||||
scriptoriumPromptHashErr error
|
promptAssetHashErr error
|
||||||
)
|
)
|
||||||
@@ -8,7 +8,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestRegisterPromptAssetsAndPrepareInteractionPrompt(t *testing.T) {
|
func TestRegisterPromptAssetsAndPrepareInteractionPrompt(t *testing.T) {
|
||||||
@@ -23,26 +23,26 @@ func TestRegisterPromptAssetsAndPrepareInteractionPrompt(t *testing.T) {
|
|||||||
if _, err := fs.ReadFile(schemaFS, "dnd_npc_interactions_llm.v1.json"); err != nil {
|
if _, err := fs.ReadFile(schemaFS, "dnd_npc_interactions_llm.v1.json"); err != nil {
|
||||||
t.Fatalf("response schema asset: %v", err)
|
t.Fatalf("response schema asset: %v", err)
|
||||||
}
|
}
|
||||||
options, err := registry.ScriptoriumOptions()
|
options, err := registry.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
ID: "npc-interactions-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "npc-interactions-test-model",
|
ID: "npc-interactions-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "npc-interactions-test-model",
|
||||||
})))
|
})))
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
transcript := `{"units":[1]}`
|
transcript := `{"units":[1]}`
|
||||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "npc-interactions-test-profile",
|
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "npc-interactions-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.InlineWithURI("file:///session.json", transcript),
|
"transcript": promptkit.InlineWithURI("file:///session.json", transcript),
|
||||||
"players": scriptorium.Inline("Dana: Mira"),
|
"players": promptkit.Inline("Dana: Mira"),
|
||||||
"party": scriptorium.Inline("Mira: ranger"),
|
"party": promptkit.Inline("Mira: ranger"),
|
||||||
"glossary": scriptorium.Inline("Greencloak: title"),
|
"glossary": promptkit.Inline("Greencloak: title"),
|
||||||
"npcs": scriptorium.Inline(`{"npcs":[{"name":"Mira Thorn"}]}`),
|
"npcs": promptkit.Inline(`{"npcs":[{"name":"Mira Thorn"}]}`),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -54,9 +54,9 @@ func TestRegisterPromptAssetsAndPrepareInteractionPrompt(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestPromptMetadataDoesNotExposeAssetContent(t *testing.T) {
|
func TestPromptMetadataDoesNotExposeAssetContent(t *testing.T) {
|
||||||
hash, err := scriptoriumPromptMetadata()
|
hash, err := promptAssetMetadata()
|
||||||
if err != nil || !strings.HasPrefix(hash, "sha256:") {
|
if err != nil || !strings.HasPrefix(hash, "sha256:") {
|
||||||
t.Fatalf("scriptoriumPromptMetadata() = %q, %v", hash, err)
|
t.Fatalf("promptAssetMetadata() = %q, %v", hash, err)
|
||||||
}
|
}
|
||||||
metadata := newExtractor(t, &fakeInteractionsLLMClient{}).ManifestMetadata()
|
metadata := newExtractor(t, &fakeInteractionsLLMClient{}).ManifestMetadata()
|
||||||
for _, forbidden := range []string{"common-dnd-system", "dnd_npc_interactions_llm.v1.json"} {
|
for _, forbidden := range []string{"common-dnd-system", "dnd_npc_interactions_llm.v1.json"} {
|
||||||
@@ -54,7 +54,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
|
|||||||
if len(references) > 1 {
|
if len(references) > 1 {
|
||||||
return nil, extractorErrorf("at most one reference set may be supplied")
|
return nil, extractorErrorf("at most one reference set may be supplied")
|
||||||
}
|
}
|
||||||
promptSHA, err := scriptoriumPromptMetadata()
|
promptSHA, err := promptAssetMetadata()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, extractorErrorf("load prompt metadata: %w", err)
|
return nil, extractorErrorf("load prompt metadata: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
)
|
)
|
||||||
|
|
||||||
const scriptoriumPromptRoot = "assets/prompts"
|
const promptAssetRoot = "assets/prompts"
|
||||||
|
|
||||||
var promptAssetManifest = shared.PromptAssetManifest{
|
var promptAssetManifest = shared.PromptAssetManifest{
|
||||||
ModuleDir: "dnd.npcs",
|
ModuleDir: "dnd.npcs",
|
||||||
@@ -32,21 +32,21 @@ func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("prepare NPC prompt assets: %w", err)
|
return fmt.Errorf("prepare NPC prompt assets: %w", err)
|
||||||
}
|
}
|
||||||
if err := registry.RegisterPromptFS(promptFS, scriptoriumPromptRoot); err != nil {
|
if err := registry.RegisterPromptFS(promptFS, promptAssetRoot); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumPromptMetadata() (string, error) {
|
func promptAssetMetadata() (string, error) {
|
||||||
scriptoriumPromptHashOnce.Do(func() {
|
promptAssetHashOnce.Do(func() {
|
||||||
scriptoriumPromptHash, scriptoriumPromptHashErr = promptAssetManifest.Hash(embeddedAssets)
|
promptAssetHash, promptAssetHashErr = promptAssetManifest.Hash(embeddedAssets)
|
||||||
})
|
})
|
||||||
return scriptoriumPromptHash, scriptoriumPromptHashErr
|
return promptAssetHash, promptAssetHashErr
|
||||||
}
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
scriptoriumPromptHashOnce sync.Once
|
promptAssetHashOnce sync.Once
|
||||||
scriptoriumPromptHash string
|
promptAssetHash string
|
||||||
scriptoriumPromptHashErr error
|
promptAssetHashErr error
|
||||||
)
|
)
|
||||||
@@ -8,7 +8,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestRegisterPromptAssetsAndPrepareNPCPrompt(t *testing.T) {
|
func TestRegisterPromptAssetsAndPrepareNPCPrompt(t *testing.T) {
|
||||||
@@ -16,24 +16,24 @@ func TestRegisterPromptAssetsAndPrepareNPCPrompt(t *testing.T) {
|
|||||||
if err := RegisterPromptAssets(registry); err != nil {
|
if err := RegisterPromptAssets(registry); err != nil {
|
||||||
t.Fatalf("RegisterPromptAssets() error = %v, want nil", err)
|
t.Fatalf("RegisterPromptAssets() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
options, err := registry.ScriptoriumOptions()
|
options, err := registry.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ScriptoriumOptions() error = %v, want nil", err)
|
t.Fatalf("PromptKitOptions() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
ID: "npc-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "npc-test-model",
|
ID: "npc-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "npc-test-model",
|
||||||
})))
|
})))
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("NewEngine() error = %v, want nil", err)
|
t.Fatalf("NewEngine() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "npc-test-profile",
|
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "npc-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.InlineWithURI("file:///session.json", `{"units":[1]}`),
|
"transcript": promptkit.InlineWithURI("file:///session.json", `{"units":[1]}`),
|
||||||
"players": scriptorium.Inline(" "),
|
"players": promptkit.Inline(" "),
|
||||||
"party": scriptorium.Inline(" "),
|
"party": promptkit.Inline(" "),
|
||||||
"glossary": scriptorium.Inline(" "),
|
"glossary": promptkit.Inline(" "),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -45,9 +45,9 @@ func TestRegisterPromptAssetsAndPrepareNPCPrompt(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestPromptMetadataAndDiagnosticsDoNotContainRawAssets(t *testing.T) {
|
func TestPromptMetadataAndDiagnosticsDoNotContainRawAssets(t *testing.T) {
|
||||||
hash, err := scriptoriumPromptMetadata()
|
hash, err := promptAssetMetadata()
|
||||||
if err != nil || !strings.HasPrefix(hash, "sha256:") {
|
if err != nil || !strings.HasPrefix(hash, "sha256:") {
|
||||||
t.Fatalf("scriptoriumPromptMetadata() = %q, %v; want hash", hash, err)
|
t.Fatalf("promptAssetMetadata() = %q, %v; want hash", hash, err)
|
||||||
}
|
}
|
||||||
metadata := newExtractor(t, &fakeNPCsLLMClient{}).ManifestMetadata()
|
metadata := newExtractor(t, &fakeNPCsLLMClient{}).ManifestMetadata()
|
||||||
payload, err := json.Marshal(metadata)
|
payload, err := json.Marshal(metadata)
|
||||||
@@ -59,7 +59,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
|
|||||||
if len(references) > 1 {
|
if len(references) > 1 {
|
||||||
return nil, extractorErrorf("at most one reference set may be supplied")
|
return nil, extractorErrorf("at most one reference set may be supplied")
|
||||||
}
|
}
|
||||||
promptSHA, err := scriptoriumPromptMetadata()
|
promptSHA, err := promptAssetMetadata()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, extractorErrorf("load prompt metadata: %w", err)
|
return nil, extractorErrorf("load prompt metadata: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
)
|
)
|
||||||
|
|
||||||
const scriptoriumPromptRoot = "assets/prompts"
|
const promptAssetRoot = "assets/prompts"
|
||||||
|
|
||||||
var promptAssetManifest = shared.PromptAssetManifest{
|
var promptAssetManifest = shared.PromptAssetManifest{
|
||||||
ModuleDir: "dnd.scene_descriptions",
|
ModuleDir: "dnd.scene_descriptions",
|
||||||
@@ -31,21 +31,21 @@ func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("prepare scene-description prompt assets: %w", err)
|
return fmt.Errorf("prepare scene-description prompt assets: %w", err)
|
||||||
}
|
}
|
||||||
if err := registry.RegisterPromptFS(promptFS, scriptoriumPromptRoot); err != nil {
|
if err := registry.RegisterPromptFS(promptFS, promptAssetRoot); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumPromptMetadata() (string, error) {
|
func promptAssetMetadata() (string, error) {
|
||||||
scriptoriumPromptHashOnce.Do(func() {
|
promptAssetHashOnce.Do(func() {
|
||||||
scriptoriumPromptHash, scriptoriumPromptHashErr = promptAssetManifest.Hash(embeddedAssets)
|
promptAssetHash, promptAssetHashErr = promptAssetManifest.Hash(embeddedAssets)
|
||||||
})
|
})
|
||||||
return scriptoriumPromptHash, scriptoriumPromptHashErr
|
return promptAssetHash, promptAssetHashErr
|
||||||
}
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
scriptoriumPromptHashOnce sync.Once
|
promptAssetHashOnce sync.Once
|
||||||
scriptoriumPromptHash string
|
promptAssetHash string
|
||||||
scriptoriumPromptHashErr error
|
promptAssetHashErr error
|
||||||
)
|
)
|
||||||
@@ -8,7 +8,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestRegisterPromptAssetsPreparesSceneDescriptionPrompt(t *testing.T) {
|
func TestRegisterPromptAssetsPreparesSceneDescriptionPrompt(t *testing.T) {
|
||||||
@@ -16,24 +16,24 @@ func TestRegisterPromptAssetsPreparesSceneDescriptionPrompt(t *testing.T) {
|
|||||||
if err := RegisterPromptAssets(registry); err != nil {
|
if err := RegisterPromptAssets(registry); err != nil {
|
||||||
t.Fatalf("RegisterPromptAssets() error = %v, want nil", err)
|
t.Fatalf("RegisterPromptAssets() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
options, err := registry.ScriptoriumOptions()
|
options, err := registry.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ScriptoriumOptions() error = %v, want nil", err)
|
t.Fatalf("PromptKitOptions() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
ID: "scene-description-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "scene-description-test-model",
|
ID: "scene-description-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "scene-description-test-model",
|
||||||
})))
|
})))
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("NewEngine() error = %v, want nil", err)
|
t.Fatalf("NewEngine() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "scene-description-test-profile",
|
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "scene-description-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.InlineWithURI("file:///session.json", `{"units":[1]}`),
|
"transcript": promptkit.InlineWithURI("file:///session.json", `{"units":[1]}`),
|
||||||
"players": scriptorium.Inline(" "),
|
"players": promptkit.Inline(" "),
|
||||||
"party": scriptorium.Inline(" "),
|
"party": promptkit.Inline(" "),
|
||||||
"glossary": scriptorium.Inline(" "),
|
"glossary": promptkit.Inline(" "),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -45,9 +45,9 @@ func TestRegisterPromptAssetsPreparesSceneDescriptionPrompt(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestPromptMetadataAndDiagnosticsDoNotContainRawAssets(t *testing.T) {
|
func TestPromptMetadataAndDiagnosticsDoNotContainRawAssets(t *testing.T) {
|
||||||
hash, err := scriptoriumPromptMetadata()
|
hash, err := promptAssetMetadata()
|
||||||
if err != nil || !strings.HasPrefix(hash, "sha256:") {
|
if err != nil || !strings.HasPrefix(hash, "sha256:") {
|
||||||
t.Fatalf("scriptoriumPromptMetadata() = %q, %v; want hash", hash, err)
|
t.Fatalf("promptAssetMetadata() = %q, %v; want hash", hash, err)
|
||||||
}
|
}
|
||||||
payload, err := json.Marshal(newExtractor(t, &fakeSceneDescriptionsLLMClient{}).ManifestMetadata())
|
payload, err := json.Marshal(newExtractor(t, &fakeSceneDescriptionsLLMClient{}).ManifestMetadata())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -94,7 +94,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, extractorErrorf("prepare NPC registry prompt input: %w", err)
|
return nil, extractorErrorf("prepare NPC registry prompt input: %w", err)
|
||||||
}
|
}
|
||||||
promptSHA, err := scriptoriumPromptMetadata()
|
promptSHA, err := promptAssetMetadata()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, extractorErrorf("load prompt metadata: %w", err)
|
return nil, extractorErrorf("load prompt metadata: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
)
|
)
|
||||||
|
|
||||||
const scriptoriumPromptRoot = "assets/prompts"
|
const promptAssetRoot = "assets/prompts"
|
||||||
|
|
||||||
var promptAssetManifest = shared.PromptAssetManifest{
|
var promptAssetManifest = shared.PromptAssetManifest{
|
||||||
ModuleDir: "dnd.spells",
|
ModuleDir: "dnd.spells",
|
||||||
@@ -34,21 +34,21 @@ func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("prepare spell prompt assets: %w", err)
|
return fmt.Errorf("prepare spell prompt assets: %w", err)
|
||||||
}
|
}
|
||||||
if err := registry.RegisterPromptFS(promptFS, scriptoriumPromptRoot); err != nil {
|
if err := registry.RegisterPromptFS(promptFS, promptAssetRoot); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumPromptMetadata() (string, error) {
|
func promptAssetMetadata() (string, error) {
|
||||||
scriptoriumPromptHashOnce.Do(func() {
|
promptAssetHashOnce.Do(func() {
|
||||||
scriptoriumPromptHash, scriptoriumPromptHashErr = promptAssetManifest.Hash(embeddedAssets)
|
promptAssetHash, promptAssetHashErr = promptAssetManifest.Hash(embeddedAssets)
|
||||||
})
|
})
|
||||||
return scriptoriumPromptHash, scriptoriumPromptHashErr
|
return promptAssetHash, promptAssetHashErr
|
||||||
}
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
scriptoriumPromptHashOnce sync.Once
|
promptAssetHashOnce sync.Once
|
||||||
scriptoriumPromptHash string
|
promptAssetHash string
|
||||||
scriptoriumPromptHashErr error
|
promptAssetHashErr error
|
||||||
)
|
)
|
||||||
@@ -8,10 +8,10 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestScriptoriumPromptPreparesSpellPrompt(t *testing.T) {
|
func TestPromptAssetsPrepareSpellPrompt(t *testing.T) {
|
||||||
transcript := []byte(`{"id":"session-1","segments":[{"id":"u1","text":"Mira casts shield."}]}`)
|
transcript := []byte(`{"id":"session-1","segments":[{"id":"u1","text":"Mira casts shield."}]}`)
|
||||||
prepared := prepareSpellsPrompt(t, transcript, "Dana: Mira", "Mira: wizard", "Shield: abjuration")
|
prepared := prepareSpellsPrompt(t, transcript, "Dana: Mira", "Mira: wizard", "Shield: abjuration")
|
||||||
|
|
||||||
@@ -23,7 +23,7 @@ func TestScriptoriumPromptPreparesSpellPrompt(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestScriptoriumPromptPreparesWithMissingOptionalReferences(t *testing.T) {
|
func TestPromptAssetsPrepareWithMissingOptionalReferences(t *testing.T) {
|
||||||
transcript := []byte(`{"id":"session-1","segments":[]}`)
|
transcript := []byte(`{"id":"session-1","segments":[]}`)
|
||||||
prepared := prepareSpellsPrompt(t, transcript, " ", " ", " ")
|
prepared := prepareSpellsPrompt(t, transcript, " ", " ", " ")
|
||||||
|
|
||||||
@@ -35,7 +35,7 @@ func TestScriptoriumPromptPreparesWithMissingOptionalReferences(t *testing.T) {
|
|||||||
t.Fatalf("reference message did not render empty optional reference placeholders")
|
t.Fatalf("reference message did not render empty optional reference placeholders")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestScriptoriumPromptDiagnosticsOmitRawMaterials(t *testing.T) {
|
func TestPromptAssetDiagnosticsOmitRawMaterials(t *testing.T) {
|
||||||
transcript := []byte(`{"secret":"source text"}`)
|
transcript := []byte(`{"secret":"source text"}`)
|
||||||
reference := "private party note"
|
reference := "private party note"
|
||||||
prepared := prepareSpellsPrompt(t, transcript, "private player note", reference, " ")
|
prepared := prepareSpellsPrompt(t, transcript, "private player note", reference, " ")
|
||||||
@@ -82,36 +82,36 @@ func TestScriptoriumPromptDiagnosticsOmitRawMaterials(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func prepareSpellsPrompt(t *testing.T, transcript []byte, players string, party string, glossary string) *scriptorium.PreparedRun {
|
func prepareSpellsPrompt(t *testing.T, transcript []byte, players string, party string, glossary string) *promptkit.PreparedRun {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
registry := llm.NewAssetRegistry()
|
registry := llm.NewAssetRegistry()
|
||||||
if err := RegisterPromptAssets(registry); err != nil {
|
if err := RegisterPromptAssets(registry); err != nil {
|
||||||
t.Fatalf("register spell prompt assets: %v", err)
|
t.Fatalf("register spell prompt assets: %v", err)
|
||||||
}
|
}
|
||||||
options, err := registry.ScriptoriumOptions()
|
options, err := registry.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ScriptoriumOptions() error = %v, want nil", err)
|
t.Fatalf("PromptKitOptions() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
ID: "spell-test-profile",
|
ID: "spell-test-profile",
|
||||||
Endpoint: "http://127.0.0.1:1/v1",
|
Endpoint: "http://127.0.0.1:1/v1",
|
||||||
Model: "spell-test-model",
|
Model: "spell-test-model",
|
||||||
})))
|
})))
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("NewEngine() error = %v, want nil", err)
|
t.Fatalf("NewEngine() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: PromptID,
|
PromptID: PromptID,
|
||||||
PromptVersion: SchemaVersion,
|
PromptVersion: SchemaVersion,
|
||||||
ProfileID: "spell-test-profile",
|
ProfileID: "spell-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.InlineWithURI("file:///session.json", string(transcript)),
|
"transcript": promptkit.InlineWithURI("file:///session.json", string(transcript)),
|
||||||
"spell_catalog": scriptorium.Inline(`{"spell_names":["Cure Wounds"]}`),
|
"spell_catalog": promptkit.Inline(`{"spell_names":["Cure Wounds"]}`),
|
||||||
"npcs": scriptorium.Inline(`{"npcs":[]}`),
|
"npcs": promptkit.Inline(`{"npcs":[]}`),
|
||||||
"players": scriptorium.Inline(players),
|
"players": promptkit.Inline(players),
|
||||||
"party": scriptorium.Inline(party),
|
"party": promptkit.Inline(party),
|
||||||
"glossary": scriptorium.Inline(glossary),
|
"glossary": promptkit.Inline(glossary),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -53,7 +53,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options) (*Normalizer, error
|
|||||||
if llmClient == nil {
|
if llmClient == nil {
|
||||||
return nil, normalizerErrorf("LLM client must not be nil")
|
return nil, normalizerErrorf("LLM client must not be nil")
|
||||||
}
|
}
|
||||||
promptSHA, err := scriptoriumPromptMetadata()
|
promptSHA, err := promptAssetMetadata()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, normalizerErrorf("load prompt metadata: %w", err)
|
return nil, normalizerErrorf("load prompt metadata: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
)
|
)
|
||||||
|
|
||||||
const scriptoriumPromptRoot = "assets/prompts"
|
const promptAssetRoot = "assets/prompts"
|
||||||
|
|
||||||
var promptAssetManifest = shared.PromptAssetManifest{
|
var promptAssetManifest = shared.PromptAssetManifest{
|
||||||
ModuleDir: PromptID,
|
ModuleDir: PromptID,
|
||||||
@@ -30,21 +30,21 @@ func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("prepare NPC normalization prompt assets: %w", err)
|
return fmt.Errorf("prepare NPC normalization prompt assets: %w", err)
|
||||||
}
|
}
|
||||||
if err := registry.RegisterPromptFS(promptFS, scriptoriumPromptRoot); err != nil {
|
if err := registry.RegisterPromptFS(promptFS, promptAssetRoot); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumPromptMetadata() (string, error) {
|
func promptAssetMetadata() (string, error) {
|
||||||
scriptoriumPromptHashOnce.Do(func() {
|
promptAssetHashOnce.Do(func() {
|
||||||
scriptoriumPromptHash, scriptoriumPromptHashErr = promptAssetManifest.Hash(embeddedAssets)
|
promptAssetHash, promptAssetHashErr = promptAssetManifest.Hash(embeddedAssets)
|
||||||
})
|
})
|
||||||
return scriptoriumPromptHash, scriptoriumPromptHashErr
|
return promptAssetHash, promptAssetHashErr
|
||||||
}
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
scriptoriumPromptHashOnce sync.Once
|
promptAssetHashOnce sync.Once
|
||||||
scriptoriumPromptHash string
|
promptAssetHash string
|
||||||
scriptoriumPromptHashErr error
|
promptAssetHashErr error
|
||||||
)
|
)
|
||||||
@@ -8,36 +8,36 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestRegisterPromptAssetsPreparesNormalizationPrompt(t *testing.T) {
|
func TestRegisterPromptAssetsPreparesNormalizationPrompt(t *testing.T) {
|
||||||
if want := []string{"common-dnd-system.md", "common-dnd-transcript.md"}; !reflect.DeepEqual(promptAssetManifest.SharedFiles, want) {
|
if want := []string{"common-dnd-system.md", "common-dnd-transcript.md"}; !reflect.DeepEqual(promptAssetManifest.SharedFiles, want) {
|
||||||
t.Fatalf("shared prompt assets = %#v, want %#v", promptAssetManifest.SharedFiles, want)
|
t.Fatalf("shared prompt assets = %#v, want %#v", promptAssetManifest.SharedFiles, want)
|
||||||
}
|
}
|
||||||
if promptHash, err := scriptoriumPromptMetadata(); err != nil || promptHash == "" {
|
if promptHash, err := promptAssetMetadata(); err != nil || promptHash == "" {
|
||||||
t.Fatalf("scriptoriumPromptMetadata() = %q, %v; want prompt fingerprint", promptHash, err)
|
t.Fatalf("promptAssetMetadata() = %q, %v; want prompt fingerprint", promptHash, err)
|
||||||
}
|
}
|
||||||
registry := llm.NewAssetRegistry()
|
registry := llm.NewAssetRegistry()
|
||||||
if err := RegisterPromptAssets(registry); err != nil {
|
if err := RegisterPromptAssets(registry); err != nil {
|
||||||
t.Fatalf("RegisterPromptAssets() error = %v", err)
|
t.Fatalf("RegisterPromptAssets() error = %v", err)
|
||||||
}
|
}
|
||||||
options, err := registry.ScriptoriumOptions()
|
options, err := registry.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ScriptoriumOptions() error = %v", err)
|
t.Fatalf("PromptKitOptions() error = %v", err)
|
||||||
}
|
}
|
||||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
ID: "normalize-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "normalize-test-model",
|
ID: "normalize-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "normalize-test-model",
|
||||||
})))
|
})))
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("NewEngine() error = %v", err)
|
t.Fatalf("NewEngine() error = %v", err)
|
||||||
}
|
}
|
||||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "normalize-test-profile",
|
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "normalize-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"candidates": scriptorium.Inline(`{"npcs":[{"name":"Mira","source_refs":[]}]}`),
|
"candidates": promptkit.Inline(`{"npcs":[{"name":"Mira","source_refs":[]}]}`),
|
||||||
"transcript": scriptorium.Inline(`{"windows":[{"units":[]}]}`),
|
"transcript": promptkit.Inline(`{"windows":[{"units":[]}]}`),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -55,7 +55,7 @@ func TestRegisterPromptAssetsPreparesNormalizationPrompt(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
for _, index := range []int{2, 4} {
|
for _, index := range []int{2, 4} {
|
||||||
if cache := prepared.Messages[index].CacheControl; cache == nil || cache.Type != scriptorium.CacheControlEphemeral {
|
if cache := prepared.Messages[index].CacheControl; cache == nil || cache.Type != promptkit.CacheControlEphemeral {
|
||||||
t.Errorf("message %d cache control = %#v, want ephemeral", index, cache)
|
t.Errorf("message %d cache control = %#v, want ephemeral", index, cache)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -14,7 +14,7 @@ import (
|
|||||||
npcextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
|
npcextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
|
||||||
scenedescriptionextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/scenedescriptions"
|
scenedescriptionextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/scenedescriptions"
|
||||||
spellextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
|
spellextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestExtractionPromptsShareRenderedPrefix(t *testing.T) {
|
func TestExtractionPromptsShareRenderedPrefix(t *testing.T) {
|
||||||
@@ -31,17 +31,17 @@ func TestExtractionPromptsShareRenderedPrefix(t *testing.T) {
|
|||||||
t.Fatalf("registerPromptAssets() error = %v", err)
|
t.Fatalf("registerPromptAssets() error = %v", err)
|
||||||
}
|
}
|
||||||
engine := newPromptCacheEngine(t, registry)
|
engine := newPromptCacheEngine(t, registry)
|
||||||
commonInputs := map[string]scriptorium.ArtifactRef{
|
commonInputs := map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.InlineWithURI("file:///session.json", `{"sentinel":"`+transcriptSentinel+`"}`),
|
"transcript": promptkit.InlineWithURI("file:///session.json", `{"sentinel":"`+transcriptSentinel+`"}`),
|
||||||
"players": scriptorium.Inline(playersSentinel),
|
"players": promptkit.Inline(playersSentinel),
|
||||||
"party": scriptorium.Inline(partySentinel),
|
"party": promptkit.Inline(partySentinel),
|
||||||
"glossary": scriptorium.Inline(glossarySentinel),
|
"glossary": promptkit.Inline(glossarySentinel),
|
||||||
}
|
}
|
||||||
cases := []struct {
|
cases := []struct {
|
||||||
name string
|
name string
|
||||||
promptID string
|
promptID string
|
||||||
promptVersion string
|
promptVersion string
|
||||||
inputs map[string]scriptorium.ArtifactRef
|
inputs map[string]promptkit.ArtifactRef
|
||||||
npcInput bool
|
npcInput bool
|
||||||
spellCatalogInput bool
|
spellCatalogInput bool
|
||||||
}{
|
}{
|
||||||
@@ -52,8 +52,8 @@ func TestExtractionPromptsShareRenderedPrefix(t *testing.T) {
|
|||||||
name: "combat turns",
|
name: "combat turns",
|
||||||
promptID: combatextract.PromptID,
|
promptID: combatextract.PromptID,
|
||||||
promptVersion: combatextract.SchemaVersion,
|
promptVersion: combatextract.SchemaVersion,
|
||||||
inputs: withPromptInputs(commonInputs, map[string]scriptorium.ArtifactRef{
|
inputs: withPromptInputs(commonInputs, map[string]promptkit.ArtifactRef{
|
||||||
"npcs": scriptorium.Inline(`{"sentinel":"` + npcSentinel + `"}`),
|
"npcs": promptkit.Inline(`{"sentinel":"` + npcSentinel + `"}`),
|
||||||
}),
|
}),
|
||||||
npcInput: true,
|
npcInput: true,
|
||||||
},
|
},
|
||||||
@@ -61,8 +61,8 @@ func TestExtractionPromptsShareRenderedPrefix(t *testing.T) {
|
|||||||
name: "npc interactions",
|
name: "npc interactions",
|
||||||
promptID: interactionextract.PromptID,
|
promptID: interactionextract.PromptID,
|
||||||
promptVersion: interactionextract.SchemaVersion,
|
promptVersion: interactionextract.SchemaVersion,
|
||||||
inputs: withPromptInputs(commonInputs, map[string]scriptorium.ArtifactRef{
|
inputs: withPromptInputs(commonInputs, map[string]promptkit.ArtifactRef{
|
||||||
"npcs": scriptorium.Inline(`{"sentinel":"` + npcSentinel + `"}`),
|
"npcs": promptkit.Inline(`{"sentinel":"` + npcSentinel + `"}`),
|
||||||
}),
|
}),
|
||||||
npcInput: true,
|
npcInput: true,
|
||||||
},
|
},
|
||||||
@@ -70,19 +70,19 @@ func TestExtractionPromptsShareRenderedPrefix(t *testing.T) {
|
|||||||
name: "spells",
|
name: "spells",
|
||||||
promptID: spellextract.PromptID,
|
promptID: spellextract.PromptID,
|
||||||
promptVersion: spellextract.SchemaVersion,
|
promptVersion: spellextract.SchemaVersion,
|
||||||
inputs: withPromptInputs(commonInputs, map[string]scriptorium.ArtifactRef{
|
inputs: withPromptInputs(commonInputs, map[string]promptkit.ArtifactRef{
|
||||||
"npcs": scriptorium.Inline(`{"sentinel":"` + npcSentinel + `"}`),
|
"npcs": promptkit.Inline(`{"sentinel":"` + npcSentinel + `"}`),
|
||||||
"spell_catalog": scriptorium.Inline(`{"sentinel":"` + catalogSentinel + `"}`),
|
"spell_catalog": promptkit.Inline(`{"sentinel":"` + catalogSentinel + `"}`),
|
||||||
}),
|
}),
|
||||||
npcInput: true,
|
npcInput: true,
|
||||||
spellCatalogInput: true,
|
spellCatalogInput: true,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
var sharedPrefix []scriptorium.RenderedMessage
|
var sharedPrefix []promptkit.RenderedMessage
|
||||||
for _, testCase := range cases {
|
for _, testCase := range cases {
|
||||||
t.Run(testCase.name, func(t *testing.T) {
|
t.Run(testCase.name, func(t *testing.T) {
|
||||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: testCase.promptID,
|
PromptID: testCase.promptID,
|
||||||
PromptVersion: testCase.promptVersion,
|
PromptVersion: testCase.promptVersion,
|
||||||
ProfileID: "prompt-cache-test-profile",
|
ProfileID: "prompt-cache-test-profile",
|
||||||
@@ -100,7 +100,7 @@ func TestExtractionPromptsShareRenderedPrefix(t *testing.T) {
|
|||||||
t.Fatalf("prepared prompt has %d messages, want lane-specific suffix after transcript", len(prepared.Messages))
|
t.Fatalf("prepared prompt has %d messages, want lane-specific suffix after transcript", len(prepared.Messages))
|
||||||
}
|
}
|
||||||
if sharedPrefix == nil {
|
if sharedPrefix == nil {
|
||||||
sharedPrefix = append([]scriptorium.RenderedMessage(nil), prefix...)
|
sharedPrefix = append([]promptkit.RenderedMessage(nil), prefix...)
|
||||||
} else if !reflect.DeepEqual(prefix, sharedPrefix) {
|
} else if !reflect.DeepEqual(prefix, sharedPrefix) {
|
||||||
t.Fatalf("rendered prefix = %#v, want %#v", prefix, sharedPrefix)
|
t.Fatalf("rendered prefix = %#v, want %#v", prefix, sharedPrefix)
|
||||||
}
|
}
|
||||||
@@ -114,24 +114,24 @@ func TestExtractionPromptsShareRenderedPrefix(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func newPromptCacheEngine(t *testing.T, registry *llm.AssetRegistry) *scriptorium.Engine {
|
func newPromptCacheEngine(t *testing.T, registry *llm.AssetRegistry) *promptkit.Engine {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
options, err := registry.ScriptoriumOptions()
|
options, err := registry.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ScriptoriumOptions() error = %v", err)
|
t.Fatalf("PromptKitOptions() error = %v", err)
|
||||||
}
|
}
|
||||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
ID: "prompt-cache-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "prompt-cache-test-model",
|
ID: "prompt-cache-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "prompt-cache-test-model",
|
||||||
})))
|
})))
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("NewEngine() error = %v", err)
|
t.Fatalf("NewEngine() error = %v", err)
|
||||||
}
|
}
|
||||||
return engine
|
return engine
|
||||||
}
|
}
|
||||||
|
|
||||||
func withPromptInputs(inputs, extras map[string]scriptorium.ArtifactRef) map[string]scriptorium.ArtifactRef {
|
func withPromptInputs(inputs, extras map[string]promptkit.ArtifactRef) map[string]promptkit.ArtifactRef {
|
||||||
merged := make(map[string]scriptorium.ArtifactRef, len(inputs)+len(extras))
|
merged := make(map[string]promptkit.ArtifactRef, len(inputs)+len(extras))
|
||||||
for name, input := range inputs {
|
for name, input := range inputs {
|
||||||
merged[name] = input
|
merged[name] = input
|
||||||
}
|
}
|
||||||
@@ -141,14 +141,14 @@ func withPromptInputs(inputs, extras map[string]scriptorium.ArtifactRef) map[str
|
|||||||
return merged
|
return merged
|
||||||
}
|
}
|
||||||
|
|
||||||
func assertRenderedInputAfter(t *testing.T, messages []scriptorium.RenderedMessage, sentinel string, index int) {
|
func assertRenderedInputAfter(t *testing.T, messages []promptkit.RenderedMessage, sentinel string, index int) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if inputIndex := renderedInputMessageIndex(t, messages, sentinel); inputIndex <= index {
|
if inputIndex := renderedInputMessageIndex(t, messages, sentinel); inputIndex <= index {
|
||||||
t.Fatalf("input sentinel %q rendered at message %d, want after transcript message %d", sentinel, inputIndex, index)
|
t.Fatalf("input sentinel %q rendered at message %d, want after transcript message %d", sentinel, inputIndex, index)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func renderedInputMessageIndex(t *testing.T, messages []scriptorium.RenderedMessage, sentinel string) int {
|
func renderedInputMessageIndex(t *testing.T, messages []promptkit.RenderedMessage, sentinel string) int {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
index := -1
|
index := -1
|
||||||
occurrences := 0
|
occurrences := 0
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
version: 3
|
version: 4
|
||||||
output:
|
output:
|
||||||
directory: ./notarius-output
|
directory: ./notarius-output
|
||||||
cache:
|
cache:
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
version: 3
|
version: 4
|
||||||
output:
|
output:
|
||||||
directory: ./notarius-output
|
directory: ./notarius-output
|
||||||
cache:
|
cache:
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
version: 3
|
version: 4
|
||||||
output:
|
output:
|
||||||
directory: ./notarius-output
|
directory: ./notarius-output
|
||||||
cache:
|
cache:
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
version: 3
|
version: 4
|
||||||
output:
|
output:
|
||||||
directory: ./notarius-output
|
directory: ./notarius-output
|
||||||
cache:
|
cache:
|
||||||
|
|||||||
Reference in New Issue
Block a user