eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Finish Scriptorium runtime documentation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-05 18:34:21 +00:00
parent c9fbb331e2
commit 31d70a2dd7
15 changed files with 221 additions and 755 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
README.md
View File

@@ -4,21 +4,19 @@ Notarius is a Go CLI for extracting structured artifacts from source material
with explicit, configurable pipeline modules.
The current implementation reads Seriatim transcript JSON, chunks the source
units, extracts D&D spell-cast artifacts with an OpenAI-compatible LLM, and
writes JSON output plus diagnostics for each run.
units, extracts D&D spell-cast artifacts with a Scriptorium-backed LLM runtime,
and writes JSON output plus diagnostics for each run.
```sh
NOTARIUS_LLM_DEFAULT_BASE_URL=http://127.0.0.1:8080/v1 \
NOTARIUS_LLM_DEFAULT_MODEL=your-model \
OPENROUTER_API_KEY=... \
go run ./cmd/notarius run dnd-session \
--config examples/dnd-spells.config.yml \
--input examples/seriatim-minimal-transcript.json
```
If the provider requires authentication, set
`NOTARIUS_LLM_DEFAULT_API_KEY` in the environment before running the command.
Outputs are written under `./notarius-output/<run-id>/` unless `--output-dir`
is provided.
The maintained example uses Scriptorium's built-in `mistral-small-3` profile,
which reads `OPENROUTER_API_KEY`. Outputs are written under
`./notarius-output/<run-id>/` unless `--output-dir` is provided.
Useful references:
@@ -27,7 +25,6 @@ Useful references:
- [Operations](docs/operations.md)
- [Troubleshooting](docs/troubleshooting.md)
- [Seriatim input contract](docs/integrations/seriatim.md)
- [OpenAI-compatible provider contract](docs/integrations/openai-compatible.md)
- [JSON output contract](docs/integrations/json-output.md)
- [D&D spell artifact contract](docs/integrations/dnd-spell-artifacts.md)
- [Developer workflow](docs/policy/development.md)

34
docs/cli.md
View File

@@ -6,21 +6,22 @@ interface.
## Quick Run
```sh
NOTARIUS_LLM_DEFAULT_BASE_URL=http://127.0.0.1:8080/v1 \
NOTARIUS_LLM_DEFAULT_MODEL=your-model \
OPENROUTER_API_KEY=... \
go run ./cmd/notarius run dnd-session \
--config examples/dnd-spells.config.yml \
--input examples/seriatim-minimal-transcript.json
```
Set `NOTARIUS_LLM_DEFAULT_API_KEY` if the OpenAI-compatible provider requires
a bearer token.
The maintained example uses prompt defaults and Scriptorium's built-in
`mistral-small-3` profile, which reads `OPENROUTER_API_KEY`. To use another
endpoint or model, configure a Scriptorium profile source and select its profile
ID in config or with `--llm-profile`.
## Commands
```text
notarius help
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--reference selector=path] [--without-reference selector]
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--session-id id] [--reference selector=path] [--without-reference selector]
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list --config path/to/config.yml [--json]
```
@@ -44,8 +45,10 @@ Flags:
Defaults to `./notarius-output`.
- `--diagnostics-dir path`: diagnostics work directory override for this
invocation.
- `--llm-profile id`: override every effective module binding to use one LLM
profile.
- `--llm-profile id`: override every effective LLM-capable module binding to
use one Scriptorium profile ID.
- `--session-id id`: pass a stable prompt session identifier through LLM-backed
module calls.
- `--reference selector=path`: bind a reference path to a chunk, extractor, or
normalizer reference slot. Repeatable.
- `--without-reference selector`: remove a configured optional reference binding.
@@ -122,12 +125,19 @@ go run ./cmd/notarius run dnd-session \
--without-reference glossary
```
Use `--session-id` when an external orchestrator needs all prompt calls from one
run to share an identifier:
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-spells.config.yml \
--input examples/seriatim-minimal-transcript.json \
--session-id campaign-17-session-04
```
For durable output, diagnostics, retention, and failure inspection, see
[Operations](operations.md).
The current `run` command requires the resolved pipeline to use exactly one
distinct LLM profile after defaults and overrides are applied.
## `config validate`
`notarius config validate` loads and validates configuration.
@@ -196,5 +206,5 @@ The production CLI currently registers these module keys:
The production CLI does not currently register validator modules.
For YAML structure, defaults, environment overrides, and module binding syntax,
see [Configuration](config.md).
For YAML structure, Scriptorium profile sources, environment overrides, and
module binding syntax, see [Configuration](config.md).

106
docs/config.md
View File

@@ -2,7 +2,7 @@
This is the canonical reference for implemented Notarius configuration.
Notarius reads YAML config files with `version: 1`. File config is applied over
Notarius reads YAML config files with `version: 2`. File config is applied over
built-in defaults, then environment overrides are applied.
## Discovery
@@ -18,12 +18,7 @@ If none is available, the command fails with a config file not found error.
## Minimal Example
```yaml
version: 1
llm_profiles:
default:
provider: openai-compatible
base_url: http://127.0.0.1:8080/v1
model: your-model
version: 2
pipelines:
dnd-session:
input: seriatim
@@ -43,25 +38,20 @@ The maintained fixture is [examples/dnd-spells.config.yml](../examples/dnd-spell
## Top-Level Fields
- `version`: required. The only supported value is `1`.
- `llm_profiles`: optional map of LLM profile IDs to profile settings.
- `version`: required. The only supported value is `2`.
- `scriptorium`: optional Scriptorium profile source settings.
- `pipelines`: optional map of pipeline IDs to pipeline definitions.
- `concurrency`: optional global concurrency settings.
- `diagnostics`: optional diagnostics settings.
Unknown YAML fields are rejected.
Unknown YAML fields are rejected. The removed top-level `llm_profiles` field is
rejected; execution profiles now come from Scriptorium.
## Defaults
Built-in defaults:
```yaml
llm_profiles:
default:
provider: openai-compatible
timeout: 600
max_retries: 3
max_concurrency: 1
concurrency:
total_llm: 1
diagnostics:
@@ -71,44 +61,52 @@ diagnostics:
No pipelines are built in. A run requires a configured pipeline.
## LLM Profiles
If `scriptorium` is omitted, Notarius uses Scriptorium's built-in profile
catalog. Prompt definitions may also name default profile IDs. The current D&D
scene and spell prompts use Scriptorium prompt defaults when a module binding
does not set `llm_profile`.
Each `llm_profiles` entry may contain:
## Scriptorium Profiles
- `provider`: optional provider key. Empty means `openai-compatible`; any other
non-empty value must be `openai-compatible`.
- `base_url`: provider base URL. Required for actual LLM calls.
- `model`: provider model name. Required for actual LLM calls.
- `api_key_env`: environment variable name to read for the API key.
- `timeout`: request timeout as whole seconds or a Go-style duration string such
as `10m`.
- `max_retries`: retry count for provider calls. Must be zero or greater.
- `max_concurrency`: per-profile LLM concurrency. Must be zero or greater; when
zero, Notarius uses `concurrency.total_llm`.
`scriptorium` fields:
Raw API keys are not accepted as file config fields. Use `api_key_env` or an
environment override.
- `profile_dir`: optional directory containing Scriptorium profile YAML files.
- `profile_file`: optional Scriptorium profile YAML file.
`profile_dir` and `profile_file` are mutually exclusive. Custom profiles
overlay Scriptorium built-in profiles by profile ID.
Scriptorium profile files use Scriptorium's profile schema. A minimal profile
looks like:
```yaml
id: local-fast
endpoint: http://127.0.0.1:8080/v1
model: your-model
api_key_env: SCRIPTORIUM_API_KEY
timeout_seconds: 180
```
Notarius does not accept raw API keys in Notarius config. For file-backed
Scriptorium profiles, store the environment variable name in `api_key_env` and
set that variable in the run environment. Scriptorium rejects raw `api_key`
fields in profile YAML.
## Environment Overrides
These environment variables are applied after the config file:
- `NOTARIUS_CONFIG`: config discovery path.
- `NOTARIUS_LLM_DEFAULT_API_KEY`: API key for the `default` LLM profile.
- `NOTARIUS_LLM_DEFAULT_BASE_URL`: base URL for the `default` LLM profile.
- `NOTARIUS_LLM_DEFAULT_MODEL`: model for the `default` LLM profile.
- `NOTARIUS_LLM_DEFAULT_TIMEOUT_SECONDS`: integer timeout seconds for the
`default` LLM profile.
- `NOTARIUS_LLM_DEFAULT_MAX_RETRIES`: integer retry count for the `default` LLM
profile.
- `NOTARIUS_LLM_DEFAULT_MAX_CONCURRENCY`: integer max concurrency for the
`default` LLM profile.
- `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency.
- `NOTARIUS_WORK_DIR`: diagnostics work directory.
- `NOTARIUS_DIAGNOSTICS_RETENTION`: diagnostics retention mode.
Integer environment values must parse as base-10 integers.
The removed `NOTARIUS_LLM_DEFAULT_*` variables are not read. Configure provider
endpoint, model, and credential environment variable names through Scriptorium
profiles.
## Pipelines
A pipeline defines the fixed Notarius workflow:
@@ -152,10 +150,9 @@ directory. Materialized bound files must be UTF-8 text. Materialized reference
provenance is recorded for chunk, extractor, and normalizer targets, and runtime
reference content is passed to the target that declares the slot. Reference
media types are inferred from file extensions, recorded as canonical base media
types, and checked only when a module declares
`AcceptedMediaTypes`; unknown extensions are recorded as
`application/octet-stream`. Reference content is not written to diagnostics,
logs, errors, or manifests.
types, and checked only when a module declares `AcceptedMediaTypes`; unknown
extensions are recorded as `application/octet-stream`. Reference content is not
written to diagnostics, logs, errors, or manifests.
Pipeline-level `references` are defaults. They are valid when at least one
eligible target in the full configured pipeline declares the slot, including
@@ -199,7 +196,7 @@ bindings. They override pipeline-level defaults for slots declared by the chunk
or normalizer module. Extractor-local references apply only to the extractor,
and normalizer-local references apply only to the normalizer.
Stage-local reference fields use the same map shape at:
Target-local reference fields use the same map shape at:
- `pipelines.<id>.chunk.references`
- `pipelines.<id>.artifacts.<lane>.extract.references`
@@ -219,23 +216,22 @@ or object form:
```yaml
chunk:
module: generic
llm_profile: default
options:
max_units: 50
module: dnd/scenes
llm_profile: local-fast
```
Binding fields:
- `module`: module key.
- `llm_profile`: optional LLM profile ID. Empty means `default`.
- `llm_profile`: optional Scriptorium profile ID. Empty or omitted lets the
Scriptorium prompt default select the profile.
- `options`: optional module-specific settings.
- `references`: optional reference bindings. Supported only for `chunk`,
`extract`, and `normalize` bindings. `input`, `merge`, validator, and
`output` bindings reject this field during validation.
The `--llm-profile` run flag overrides every effective module binding to use
one configured profile.
The `--llm-profile` run flag overrides every effective LLM-capable module
binding to use one Scriptorium profile ID.
## Implemented Production Modules
@@ -256,7 +252,7 @@ The `generic` chunker accepts:
`max_units`.
The `dnd/scenes` chunker requires transcript source capabilities, calls the
configured structured LLM provider, and does not accept module options.
configured structured LLM runtime, and does not accept module options.
The `dnd/spells` extractor declares optional text reference slots:
@@ -285,11 +281,11 @@ invocation.
Configuration validation checks:
- supported config version and known YAML fields;
- mutually exclusive `scriptorium.profile_dir` and `scriptorium.profile_file`;
- non-empty, non-duplicated IDs after trimming;
- supported LLM provider and non-negative profile limits;
- positive global LLM concurrency;
- supported diagnostics retention and non-empty work directory;
- module binding LLM profiles refer to configured profiles.
- stale removed fields such as `llm_profiles`.
Pipeline resolution additionally checks:
@@ -298,7 +294,7 @@ Pipeline resolution additionally checks:
- selected lanes exist when `--only` is used;
- required module keys are present;
- module keys are registered for the expected slot;
- module capability requirements are satisfied.
- module capability requirements are satisfied;
- bound reference slots are declared by selected chunk, extractor, or
normalizer targets;
- required reference slots are bound for selected targets.

4
docs/integrations/json-output.md
View File

@@ -85,8 +85,8 @@ approved.
],
"llm_profiles": [
{
"id": "default",
"provider": "openai-compatible",
"id": "mistral-small-3",
"provider": "scriptorium",
"model": "configured-model"
}
],

128
docs/integrations/openai-compatible.md
View File

@@ -1,128 +0,0 @@
# OpenAI-Compatible Structured Output
This document describes the external LLM provider contract implemented by the
production Notarius LLM client.
## Provider
- Provider key: `openai-compatible`
- HTTP method: `POST`
- Endpoint: `<base_url>/chat/completions`
- Request body: JSON
- Response mode: chat completions with structured JSON schema output
`base_url` is trimmed of trailing slashes before `/chat/completions` is
appended. Configure provider settings in [Configuration](../config.md).
## Request
The client sends a JSON object with:
```json
{
"model": "configured-model",
"messages": [
{
"role": "system",
"content": "..."
},
{
"role": "user",
"content": "..."
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "schema_name",
"strict": true,
"schema": {}
}
}
}
```
Implemented request behavior:
- `model` comes from the structured completion request when set, otherwise from
the configured LLM profile.
- `messages` must be non-empty; each role and content must be non-empty after
trimming.
- `response_format.type` is always `json_schema`.
- `response_format.json_schema.strict` is always `true`.
- `response_format.json_schema.name` and `schema` come from the extractor or
validator making the call.
If an API key is configured, the client sends:
```text
Authorization: Bearer <api-key>
```
The client always sends `Content-Type: application/json`.
## Response
The client expects a JSON response with at least one choice:
```json
{
"model": "provider-model",
"choices": [
{
"message": {
"content": "{\"field\":\"value\"}"
}
}
],
"usage": {
"prompt_tokens": 10,
"completion_tokens": 5,
"total_tokens": 15
}
}
```
`choices[0].message.content` may be either:
- a JSON string whose contents are valid JSON; or
- raw JSON.
The decoded content is unmarshaled into the caller-provided structured output
target. If `usage` is present, prompt, completion, and total token counts are
copied into the completion response.
## Errors And Retries
The client validates base URL, model, response schema name, response schema
JSON, messages, and output target before or during the call.
Retryable failures:
- HTTP request failure;
- response body read failure;
- HTTP `429`;
- HTTP `5xx`;
- malformed provider response envelope;
- missing choices;
- missing, empty, or invalid assistant JSON content;
- structured-output decode failure.
Non-retryable provider status codes include non-`429` `4xx` responses.
Provider error bodies are parsed for `error.message` or `message` when present.
Configured API key values and bearer-token values are redacted from returned
provider errors.
## Timeouts And Concurrency
The configured profile timeout is applied per provider request when greater
than zero. Context cancellation is respected.
The production CLI wraps the provider client with the LLM scheduler. Effective
concurrency is described in [LLM runtime internals](../internal/llm.md).
## Limits
This contract documents only the fields the implemented client sends and reads.
Provider-specific extensions are ignored unless they affect those fields.

17
docs/internal/llm.md
View File

@@ -21,13 +21,18 @@ Modules that call the LLM own their prompts, schemas, prompt IDs, validators,
and domain-specific interpretation. Provider adapters should not contain
domain-specific prompt logic.
Prompt input materials carry source or reference bytes with optional origin
metadata. The Scriptorium-backed runtime receives them as named artifacts rather
than rendered prompt strings owned by Notarius modules.
## Production Client Construction
`internal/cli` builds the production LLM client from the effective config:
1. collect production Scriptorium prompt and schema assets from module packages;
2. create a Scriptorium-backed structured client using effective Scriptorium
profile source settings;
profile source settings from `scriptorium.profile_dir` or
`scriptorium.profile_file`;
3. create a scheduler from global LLM concurrency;
4. wrap the client with `NewScheduledClient`;
5. let the runtime report non-secret profile manifest metadata after calls.
@@ -42,8 +47,10 @@ profile ID before pipeline execution.
Notarius prompt requests into Scriptorium `RunRequest` values. It:
- validates the caller output target and prompt ID;
- converts `LLMInputMaterial` values into inline Scriptorium artifacts;
- passes `session_id` through Scriptorium variables when present;
- converts `LLMInputMaterial` values into inline Scriptorium artifacts, using a
single space for empty material so optional blank references remain explicit;
- passes `session_id` through Scriptorium variables and request metadata when
present;
- sends explicit profile IDs only when the request supplies one;
- lets Scriptorium render prompts, call the configured provider, and validate
structured output;
@@ -53,7 +60,9 @@ Notarius prompt requests into Scriptorium `RunRequest` values. It:
Generated-output validation failures are returned as Notarius errors. Provider
and runtime errors are wrapped with prompt context and bearer tokens are
redacted from error strings.
redacted from error strings. Prompt text, raw source input, reference content,
schema JSON, API keys, and bearer tokens are not added to default diagnostics or
run manifests.
## Scheduler

34
docs/internal/modules.md
View File

@@ -40,12 +40,20 @@ uses references, pass them as prompt input materials through the structured LLM
request. Prompt metadata hashes remain based on prompt asset source, not
rendered reference bytes.
Chunk modules receive the structured LLM client through `contracts.ChunkRequest`
when they need model-backed chunking. The pipeline runner validates generic
chunk result invariants before extraction; module-owned policies may be stricter
but must stay within the module package.
LLM-backed modules own Scriptorium prompt definitions and response schemas in
their embedded assets. Module contracts should expose prompt IDs, versions,
input material names, and non-secret prompt/schema hashes through manifest
metadata; they should not expose Scriptorium public types through chunk,
extract, or normalize contracts.
Normalize modules receive the structured LLM client through
Chunk modules receive the structured LLM client, configured Scriptorium profile
ID, prompt session ID, and raw source input material through
`contracts.ChunkRequest` when they need model-backed chunking. The pipeline
runner validates generic chunk result invariants before extraction; module-owned
policies may be stricter but must stay within the module package.
Normalize modules receive the structured LLM client, configured Scriptorium
profile ID, prompt session ID, and reference material through
`contracts.NormalizeRequest` when they need model-backed reconciliation.
## `seriatim` Input
@@ -91,9 +99,10 @@ Provides:
Package: `internal/modules/chunk/dnd/scenes`
The `dnd/scenes` chunker uses the structured LLM client to divide transcript
source units into coherent D&D scenes. It renders embedded prompts, loads the
embedded structured response schema, validates model-authored source-unit
boundaries, and converts each scene into a deterministic source chunk.
source units into coherent D&D scenes. It supplies the embedded Scriptorium
prompt ID, prompt version, transcript input material, response schema, and
session ID to the runtime; validates model-authored source-unit boundaries; and
converts each scene into a deterministic source chunk.
Requires:
@@ -123,10 +132,11 @@ text, or secrets.
Package: `internal/modules/extract/dnd/spells`
The `dnd/spells` extractor owns D&D spell-cast artifact semantics. It renders
embedded prompts, loads the embedded structured response schema, calls the
structured LLM client, converts spell-cast responses into artifact candidates,
and supplies deterministic validators.
The `dnd/spells` extractor owns D&D spell-cast artifact semantics. It supplies
the embedded Scriptorium prompt ID, prompt version, transcript and reference
input materials, response schema, and session ID to the runtime; converts
spell-cast responses into artifact candidates; and supplies deterministic
validators.
Requires:

21
docs/internal/pipeline.md
View File

@@ -23,7 +23,8 @@ before execution:
- merge: `appendorder`
- normalize: `noop`
- output: `json`
- LLM profile: `default`
- LLM profile: empty, which lets Scriptorium prompt defaults choose a
profile.
4. The module catalog is checked for each bound module key.
5. Module capabilities are checked in workflow order.
6. A digest is calculated from the resolved pipeline without the digest field.
@@ -54,13 +55,19 @@ empty bound files. Media-type acceptance is checked only when a slot declares
manifests. The CLI writes provenance-only resolved reference diagnostics, and
the run manifest records target-stage reference provenance separately from
source digests. Runtime reference content is passed to the matching chunker,
extractor, or normalizer request.
extractor, or normalizer request. LLM-backed modules pass that material onward
as named Scriptorium prompt inputs.
Prompt bundles can declare reference slots and use `reference` and
`hasreference` template functions. Bundle loading validates string-literal slot
names against the declaration. Rendering receives a target reference set from the
caller; unbound optional slots render as empty strings, and `hasreference`
returns true only when at least one bound item has content.
The CLI carries raw input bytes into `pipeline.RunInput`. Input adapters parse
those bytes into the source document, while LLM-backed modules that need the
original transcript material can pass the same bytes as a prompt input with
origin metadata. The raw input payload is not written to manifests or default
diagnostics.
The CLI also carries an optional run `session_id`. The runner makes it available
to chunk, extract, and normalize requests; LLM-backed modules forward it through
their structured completion requests so Scriptorium can include it in prompt
execution metadata.
## Registries And Module Specs

10
docs/operations.md
View File

@@ -5,8 +5,8 @@ This is the canonical reference for operating implemented Notarius runs.
## Normal Run
A run reads one source file, resolves one configured pipeline, calls the
configured OpenAI-compatible LLM profile, writes durable JSON output, and writes
diagnostics for inspection.
configured Scriptorium-backed LLM runtime, writes durable JSON output, and
writes diagnostics for inspection.
```sh
go run ./cmd/notarius run dnd-session \
@@ -61,7 +61,7 @@ Implemented diagnostics artifacts:
- `invocation.json`: command metadata such as operation, config path, input
path, selected lanes, run ID, and pipeline digest when available.
- `effective-config.json`: resolved config with API keys redacted.
- `effective-config.json`: resolved config without raw API keys.
- `resolved-pipeline.json`: resolved module bindings and pipeline digest.
- `resolved-references.json`: resolved reference provenance, including target
stage, lane ID when present, origin, digest, media type, byte size, and
@@ -133,8 +133,8 @@ directories unless they are part of your own operational policy.
There is no command to resume a failed run. Re-run `notarius run` after fixing
the cause.
Provider retries are limited to the OpenAI-compatible client retry behavior
configured by the effective LLM profile. There is no separate CLI retry command.
Provider retries and timeouts are handled by Scriptorium according to the
selected execution profile. There is no separate CLI retry command.
Notarius writes local files only. Remote storage and archive management are not
part of the implemented CLI.

8
docs/roadmap/future.md
View File

@@ -21,17 +21,9 @@ future work only.
- Cross-lane entity normalization.
- Cross-chunk semantic deduplication.
- Configurable validator chains with production validator modules.
- Multiple effective LLM profiles in one run.
- Parallel execution where it preserves deterministic manifests and diagnostics.
- Additional output encoders.
## Candidate Architecture Work
- Evaluate replacing the local LLM adapter with an import from
`gitea.maximumdirect.net/eric/scriptorium`, provided it preserves Notarius
boundaries around provider plumbing, prompt ownership, diagnostics, and secret
handling.
## Candidate Operational Work
- Packaged release artifacts for alpha distribution.

482
docs/roadmap/implementation.md
View File

@@ -1,474 +1,16 @@
# Scriptorium Cutover Implementation Plan
# Scriptorium Runtime Migration
This plan implements the target state in
[scriptorium.md](scriptorium.md): a hard cutover from Notarius' local
OpenAI-compatible adapter, local LLM profile schema, and local system/user
prompt renderer to Scriptorium-backed prompt execution.
The Scriptorium-backed LLM runtime migration is complete.
The target implementer is an LLM coding agent. Complete each stage in order.
Do not preserve backward compatibility for removed local LLM-profile or
prompt-rendering behavior unless a stage explicitly says to keep a temporary
test seam.
Current implemented behavior is documented in:
## Global Constraints
- [Configuration](../config.md)
- [CLI Reference](../cli.md)
- [Operations](../operations.md)
- [Troubleshooting](../troubleshooting.md)
- [LLM Runtime Internals](../internal/llm.md)
- [Module Internals](../internal/modules.md)
- [Pipeline Internals](../internal/pipeline.md)
- [JSON Output](../integrations/json-output.md)
- Keep Scriptorium types out of chunk, extract, and normalize module contracts.
Module-facing requests should use Notarius-owned contract types.
- Keep provider plumbing behind `internal/framework/llm` and CLI construction.
- Keep module-owned prompt intent, prompt IDs, response schemas, validators, and
source-reference validation with the relevant modules.
- Keep raw input material, raw references, raw prompts, raw schemas, and API
keys out of manifests and default diagnostics.
- Treat the original source input and large references as prompt input
materials. They may be passed to Scriptorium as inline/file inputs, but should
not be copied into durable provenance.
- Use Scriptorium profile configuration directly. Do not keep Notarius'
`llm_profiles` schema as a second profile system.
- Use Scriptorium structured-output execution and validation behavior for
production calls. Notarius should still unmarshal successful structured JSON
into module-owned response structs and run module-owned deterministic
validation afterward.
## Stage 1: Dependency And API Grounding
Goal: add the Scriptorium dependency and establish the exact public API surface
used by the rest of the implementation.
Work:
- Add `gitea.maximumdirect.net/eric/scriptorium` to `go.mod` with `go get`.
- Inspect the installed package with `go doc` or source reads before coding the
adapter. Confirm names and fields for:
- `NewEngine`;
- `Config`;
- `WithPromptFS`, `WithPromptFile`, or equivalent prompt source options;
- `WithProfileFile`, `WithProfileFS`, profile directory config, and built-in
profile behavior;
- `WithSchemaFS` or equivalent schema source options;
- `RunRequest`, including prompt ID, prompt version, profile ID, inputs,
variables, metadata, direct API key, and validation override fields;
- `RunResult`, including raw output, output artifact, validation status,
prompt/profile/model metadata, usage, and timings;
- public sentinel errors.
- Add a short internal implementation note as a code comment only where needed;
do not add user-facing docs in this stage except if a package-level test
helper needs explanation.
Acceptance checks:
- `go test ./internal/framework/llm`
- `go test ./internal/cli`
- `go test ./...`
## Stage 2: Prompt Input Materials And Session IDs
Goal: make raw source input and references available to module prompt execution
without storing large content in manifests or default diagnostics.
Work:
- Add Notarius-owned contract types under `internal/framework/contracts`:
- `LLMInputMaterial` with at least `Name`, `MediaType`, `Content`, `Digest`,
`OriginURI`, and `SizeBytes`;
- a helper-friendly collection type, such as `LLMInputSet`, if useful.
- Add prompt-execution fields to `StructuredCompletionRequest`:
- `PromptID`;
- `PromptVersion`;
- `ProfileID` or reuse the request `LLMProfile` value when called from stage
requests;
- `SessionID`;
- `Inputs map[string]LLMInputMaterial`;
- `Vars map[string]any`.
- Keep or remove the old `Messages`, `Model`, `ResponseSchemaName`, and
`ResponseSchema` fields according to what makes the cutover cleanest. The
final production path must not require modules to pass rendered messages or
raw schema JSON directly to the provider adapter.
- Add `SourceInput contracts.LLMInputMaterial` and `SessionID string` to
`ChunkRequest`, `ExtractionRequest`, and `NormalizeRequest`.
- Add `SessionID string` to `pipeline.RunInput`.
- In `pipeline.Run`, build a source input material from `RunInput.RawInput` and
`RunInput.Path`:
- preserve the bytes exactly;
- infer media type from the input path extension, using `application/json`
for `.json` and a deterministic fallback for unknown extensions;
- compute a SHA-256 digest;
- use a file URI or path-derived origin URI;
- do not put the bytes in manifest metadata.
- Pass the same source input material and session ID to chunk, extract, and
normalize requests.
- Reuse existing materialized `ReferenceSet` content for reference prompt
inputs. Do not introduce a second file-reading path for references.
- Add CLI support for `--session-id <id>` on `notarius run`.
- Empty means use a deterministic default derived from the parsed source
document ID.
- The deterministic default must be stable across runs over the same parsed
source document.
- The explicit value should be trimmed and rejected if empty after trimming.
- Because the default depends on the parsed source document, compute and attach
the final session ID inside the runner after input parsing, or return enough
information from parsing for the CLI to resolve it before LLM calls.
- Record the non-secret session ID in run metadata or manifest metadata, but
never record raw prompt content.
Acceptance tests:
- Contract tests proving `LLMInputMaterial.Content` is defensively copied where
relevant and omitted from JSON.
- Pipeline tests proving chunk, extract, and normalize requests receive the same
source input bytes and session ID.
- CLI tests for `--session-id`, including explicit value, missing value,
trimming, and default behavior.
- Redaction/diagnostics tests proving raw source bytes and reference content do
not appear in manifests or default diagnostics.
Focused checks:
- `go test ./internal/framework/contracts`
- `go test ./internal/framework/pipeline`
- `go test ./internal/cli`
## Stage 3: Configuration Hard Cutover To Scriptorium Profiles
Goal: replace Notarius' local LLM profile schema with Scriptorium profile
selection and profile sources.
Work:
- Bump the Notarius file config version because this is an incompatible config
schema change.
- Remove the top-level `llm_profiles` file-config schema and the local
`config.LLMProfile` model.
- Add a top-level Scriptorium config block. Use this shape unless Scriptorium's
installed API requires a small naming adjustment:
```yaml
scriptorium:
profile_dir: ./profiles
profile_file: ./profiles.yml
```
- Treat `profile_dir` and `profile_file` as mutually exclusive in Notarius
config validation. Built-in Scriptorium profiles remain available when neither
is set.
- Keep existing module binding field name `llm_profile`; it now names a
Scriptorium profile ID.
- Stop forcing empty module `llm_profile` bindings to
`pipeline.DefaultLLMProfile`. Empty means "use the Scriptorium prompt's
`default_profile`."
- Keep `--llm-profile` as a run-level operational override. It should set the
same explicit Scriptorium profile ID for every LLM-eligible selected target.
- Remove `OpenAICompatibleClientConfig` construction from `internal/core/config`.
- Remove local OpenAI-compatible provider validation from config validation.
- Add config validation for:
- config version;
- mutually exclusive `scriptorium.profile_dir` and
`scriptorium.profile_file`;
- non-empty profile source paths when fields are present;
- non-empty explicit `llm_profile` strings after trimming.
- Add CLI/config validation that uses Scriptorium profile loading to reject
unknown explicit profile IDs when possible. Prompt-default profile failures
may surface during Scriptorium prepare/run if Scriptorium owns that lookup.
- Remove or replace environment override behavior tied to
`NOTARIUS_LLM_DEFAULT_*`. Secrets should come from Scriptorium profile
`api_key_env` values or direct request-scoped Scriptorium behavior.
- Update redacted effective config behavior for the new Scriptorium config
shape.
Acceptance tests:
- Config parsing accepts `scriptorium.profile_dir`.
- Config parsing accepts `scriptorium.profile_file`.
- Config validation rejects both fields set at once.
- Config validation rejects stale `llm_profiles`.
- Existing pipeline bindings with explicit `llm_profile` resolve to trimmed
Scriptorium profile IDs.
- Empty `llm_profile` remains empty through resolution unless `--llm-profile`
is supplied.
- CLI config validation fails cleanly for an unknown explicit profile ID.
- Redacted config diagnostics do not contain raw API keys.
Focused checks:
- `go test ./internal/core/config`
- `go test ./internal/cli`
## Stage 4: Scriptorium Prompt And Schema Assets
Goal: move production LLM prompts and schemas to Scriptorium-compatible assets
while preserving module ownership.
Work:
- Create a prompt/schema asset registration mechanism that does not put
D&D-specific prompt content in framework packages.
- Recommended shape: add a small prompt-asset registry in
`internal/framework/llm` or a sibling framework package that can collect
`fs.FS` roots for Scriptorium prompt and schema sources.
- Production module packages should register their own Scriptorium prompt and
schema assets through production catalog/registry wiring in `internal/cli`.
- Shared D&D prompt assets may live in a D&D-specific module package such as
`internal/modules/dnd/promptassets`; they must not live in `internal/core`
or source-agnostic framework packages.
- Add shared D&D prompt assets:
- stable shared system message, if needed;
- stable cacheable transcript user message:
```text
A transcript of a Dungeons & Dragons gameplay session is provided below.
{{ input "transcript" }}
```
- stable cacheable reference/context message templates for roster, glossary,
previous recap, or other large references used by current D&D modules.
- Convert `dnd/scenes` to a Scriptorium prompt definition:
- prompt ID: `dnd.scenes`;
- prompt version: current module prompt version;
- input `transcript`, required, `application/json`;
- messages ordered for cache reuse: shared system, shared transcript user
message with cache control, scene task, scene instructions;
- output JSON schema path pointing at the existing scene schema asset;
- schema IDs/names/versions remain module-owned and manifest-safe.
- Convert `dnd/spells` to a Scriptorium prompt definition:
- prompt ID: `dnd.spells`;
- prompt version: current module prompt version;
- input `transcript`, required, `application/json`;
- optional inputs for `roster` and `glossary`;
- messages ordered for cache reuse: shared system, shared transcript user
message with cache control, optional/reference context message with cache
control, spell task, spell instructions;
- output JSON schema path pointing at the existing spell schema asset.
- For optional references, pass empty inline input material when the slot is
unbound unless Scriptorium's template/input semantics support missing
optional inputs cleanly. Do not let optional missing references make prompt
rendering fail.
- For multiple reference items in one slot, concatenate deterministically with
stable headings that include only non-secret provenance, then pass the result
as that prompt input. Existing single-item slots should keep their current
behavior.
- Replace current prompt hash metadata with hashes derived from the
Scriptorium prompt definition plus message assets, or with Scriptorium
prepared-run prompt metadata if it is available without raw prompt content.
- Preserve existing manifest metadata keys where practical:
- `prompt_id`;
- `prompt_version`;
- `prompt_sha256`;
- `response_schema_key`;
- `response_schema_id`;
- `response_schema_name`;
- `response_schema_version`;
- `response_schema_sha256`.
Acceptance tests:
- Prompt asset loading fails fast for missing prompt files or schemas.
- `dnd/scenes` prepared prompt contains separate transcript and task messages.
- `dnd/spells` prepared prompt contains separate transcript, reference, and
task messages.
- The transcript message body is byte-identical to the expected shared template
plus original Seriatim JSON bytes.
- Reference prompt input rendering is deterministic.
- Prompt/schema diagnostics omit raw prompt text, raw transcript bytes, raw
reference content, and raw schema JSON.
Focused checks:
- `go test ./internal/framework/llm`
- `go test ./internal/modules/chunk/dnd/scenes`
- `go test ./internal/modules/extract/dnd/spells`
## Stage 5: Scriptorium-Backed LLM Runtime
Goal: implement the production `StructuredLLMClient` using Scriptorium.
Work:
- Add a Scriptorium-backed client under `internal/framework/llm`.
- Its constructor should accept:
- Scriptorium profile source settings from effective Notarius config;
- registered prompt/schema assets;
- request timeout or HTTP client settings only if still owned by Notarius
after the profile cutover;
- optional Scriptorium engine options for tests.
- Implement `CompleteStructured(ctx, req, out)` by:
- validating `out` is a non-nil pointer;
- validating `req.PromptID` is non-empty;
- converting Notarius `LLMInputMaterial` values to Scriptorium artifact refs;
- adding `session_id` to Scriptorium request vars when non-empty;
- passing explicit profile ID only when the module binding or CLI override
supplied one;
- passing no raw API key unless a deliberate request-scoped secret path is
implemented;
- calling Scriptorium `Run`;
- converting final validation failure into a Notarius error;
- unmarshaling successful structured JSON into `out`;
- returning `StructuredCompletionResponse` with raw JSON content, provider,
model, profile ID when available, token usage, and non-secret metadata.
- Add a profile recorder or response accumulator so `RunManifest.LLMProfiles`
records the actual Scriptorium profile/provider/model values used during the
run.
- Do not rely on one precomputed profile ID before pipeline execution.
- Deduplicate profile manifest entries deterministically.
- Keep the existing `Scheduler` and scheduled client wrapper unless Scriptorium
provides an equivalent Notarius-approved concurrency mechanism.
- Ensure all Scriptorium errors are wrapped with context and converted to
concise CLI-facing errors. Preserve `errors.Is` checks internally when
practical.
- Apply Notarius secret redaction to errors before writing diagnostics.
- Remove production construction of `OpenAICompatibleClient`.
Acceptance tests:
- Scriptorium adapter maps Notarius prompt request fields into the expected
Scriptorium run request using an injected fake Scriptorium LLM client.
- Successful structured output unmarshals into the caller target.
- Scriptorium validation failure returns an error.
- Provider/runtime failure returns an error with operation context.
- Context cancellation is respected.
- Token usage maps into `StructuredCompletionResponse`.
- Used Scriptorium profile/provider/model metadata appears in the run manifest
without secrets.
- API keys or bearer tokens in synthetic errors are redacted.
- Scheduler still bounds concurrent Scriptorium-backed calls.
Focused checks:
- `go test ./internal/framework/llm`
- `go test ./internal/framework/pipeline`
- `go test ./internal/cli`
## Stage 6: Module Cutover And Legacy Runtime Removal
Goal: update production modules to call the prompt-based contract and remove
obsolete local prompt/runtime code.
Work:
- Update `dnd/scenes`:
- stop rendering local system/user prompt strings;
- call `CompleteStructured` with `PromptID`, `PromptVersion`, `SessionID`,
`ProfileID` or request LLM profile, `transcript` input material, and any
required vars;
- keep existing response validation, chunk canonicalization, caveat handling,
and manifest metadata policy.
- Update `dnd/spells`:
- stop rendering local system/user prompt strings;
- call `CompleteStructured` with `transcript`, optional `roster`, optional
`glossary`, session ID, profile ID, and vars;
- keep existing spell response validation, source-reference validation, and
manifest metadata policy.
- Update any LLM-backed normalize modules if present. If only noop normalize is
present, ensure the contract and tests prove normalizers can receive the same
Scriptorium-capable client and prompt inputs.
- Remove the old `internal/framework/prompt` renderer if no remaining code uses
it. If generic tests still need prompt rendering, replace them with
Scriptorium prompt asset tests or delete obsolete tests.
- Remove `internal/framework/llm/openai_compatible_client.go` and its tests
after the Scriptorium adapter tests cover replacement behavior.
- Remove local OpenAI-compatible integration docs after current-behavior docs
are updated in Stage 7.
- Remove stale schema registry helpers only if they are no longer needed for
module-owned schema metadata. Keep lightweight schema hashing/loading helpers
if modules still use them for manifest metadata.
Acceptance tests:
- `dnd/scenes` fake-client tests assert the module sends prompt ID,
transcript input, session ID, and schema/prompt metadata rather than rendered
message text.
- `dnd/spells` fake-client tests assert roster/glossary inputs are passed as
inputs and not interpolated locally.
- Existing malformed LLM response tests still fail as malformed structured
output.
- Existing source-reference validation tests still pass.
- No production code imports the old prompt renderer or local
OpenAI-compatible client.
Focused checks:
- `go test ./internal/modules/chunk/dnd/scenes`
- `go test ./internal/modules/extract/dnd/spells`
- If `internal/framework/prompt` is deleted, do not run a package-specific test
for it; instead verify with `rg -n "internal/framework/prompt|RenderUserSystem" internal`
that no production code still depends on it.
## Stage 7: CLI, Examples, Docs, And Full Validation
Goal: finish user-facing behavior, examples, and canonical docs for the new
runtime.
Work:
- Update `examples/dnd-spells.config.yml` and any maintained test configs to
the new config version and Scriptorium profile source behavior.
- Update `docs/config.md`:
- new config version;
- `scriptorium.profile_dir` and `scriptorium.profile_file`;
- `llm_profile` now means Scriptorium profile ID;
- empty `llm_profile` behavior;
- removed `llm_profiles`;
- removed `NOTARIUS_LLM_DEFAULT_*` env behavior if removed;
- secret-handling policy through Scriptorium profile `api_key_env`.
- Update `docs/cli.md`:
- add `--session-id`;
- update `--llm-profile` wording to Scriptorium profile override;
- update run examples if needed.
- Update `docs/internal/llm.md`:
- Scriptorium-backed runtime;
- prompt asset loading;
- input material handling;
- scheduling;
- structured-output validation;
- profile manifest recording;
- secret redaction.
- Update `docs/internal/modules.md`:
- module prompt ownership through Scriptorium prompt definitions;
- chunk/extract/normalize request input materials and session ID;
- D&D scenes/spells prompt metadata.
- Update `docs/internal/pipeline.md`:
- raw input material lifecycle;
- reference material lifecycle into prompt inputs;
- session ID lifecycle;
- actual LLM profile provenance.
- Update `docs/integrations/json-output.md` for any manifest changes.
- Remove or rewrite `docs/integrations/openai-compatible.md`. If no local
OpenAI-compatible adapter remains, do not document it as current Notarius
behavior.
- Update `docs/troubleshooting.md` for:
- Scriptorium profile-not-found;
- prompt-not-found;
- schema/validation failures;
- missing API key env values;
- session ID usage if relevant.
- Update `docs/operations.md` if diagnostics or retention behavior changes.
- Remove the Scriptorium migration item from `docs/roadmap/future.md` once the
feature is implemented.
- Replace `docs/roadmap/implementation.md` with a completed note or remove it
after implementation is complete, according to the repository's current
roadmap cleanup pattern.
Acceptance tests and inspections:
- `rg -n "llm_profiles|OpenAICompatibleClient|openai-compatible|RenderUserSystem|NOTARIUS_LLM_DEFAULT" internal docs examples`
should return only intentional historical/deferred references, if any.
- `rg -n "scriptorium|session-id|profile_dir|profile_file" docs examples`
should show current-behavior docs and examples are updated.
- `go test ./...`
- `go vet ./...`
- `go build ./cmd/notarius`
## Cross-Stage Review Checklist
Before considering the implementation complete, verify:
- No Scriptorium public types appear in chunk, extract, or normalize contracts.
- No raw source input, reference content, prompt text, schema JSON, API key, or
bearer token appears in manifests or default diagnostics.
- The shared transcript message uses original Seriatim JSON bytes exactly.
- Large reference content can be supplied as Scriptorium prompt inputs without
changing module-facing reference slot contracts.
- `--session-id` is easy for an external orchestrator to pass.
- Empty `llm_profile` lets prompt defaults work; explicit `llm_profile` and
`--llm-profile` select Scriptorium profile IDs.
- Production Notarius LLM execution goes through Scriptorium.
- Current-behavior documentation describes only implemented behavior.
Remaining product and operational ideas belong in [Future Work](future.md).

101
docs/troubleshooting.md
View File

@@ -34,7 +34,7 @@ Symptoms include:
Fix:
- Use `version: 1`.
- Use `version: 2`.
- Remove unknown YAML fields.
- Validate with:
@@ -157,21 +157,31 @@ Fix:
whitespace.
- `end` must be greater than or equal to `start`.
## Missing LLM Base URL Or Model
## Scriptorium Profile Source Failure
Symptoms include:
- `LLM profile "default" base URL must not be empty`
- `LLM profile "default" model must not be empty`
- `base URL must be valid`
- `scriptorium profile_dir and profile_file are mutually exclusive`
- `scriptorium.profile_dir must not be empty when set`
- `scriptorium.profile_file must not be empty when set`
- `profile load`
- `profile not found`
Fix:
- Set `base_url` and `model` in `llm_profiles.default`.
- Or set `NOTARIUS_LLM_DEFAULT_BASE_URL` and
`NOTARIUS_LLM_DEFAULT_MODEL`.
- If a profile needs authentication, set `api_key_env` in YAML or set
`NOTARIUS_LLM_DEFAULT_API_KEY`.
- Configure at most one of `scriptorium.profile_dir` or
`scriptorium.profile_file`.
- Confirm the selected Scriptorium profile ID exists in the configured profile
source or Scriptorium built-in profiles.
- If using `--llm-profile`, pass a Scriptorium profile ID, not a removed
Notarius profile ID.
- Validate the config and selected pipeline:
```sh
go run ./cmd/notarius config validate \
--config path/to/config.yml \
--pipeline dnd-session
```
## LLM Profile Override Failure
@@ -183,38 +193,49 @@ notarius: LLM profile override "..." is not configured
Fix:
- Add the profile under `llm_profiles`.
- Or use an existing profile ID with `--llm-profile`.
- Add the profile to the configured Scriptorium profile source.
- Or use an existing Scriptorium profile ID with `--llm-profile`.
Current runs require exactly one distinct effective LLM profile. If a pipeline
uses several profiles, run with `--llm-profile <id>` or align the bindings in
configuration.
Use `--llm-profile <id>` when one run should force every LLM-backed binding to
the same Scriptorium profile.
## Provider HTTP Or Response Failure
## Missing API Key Environment Variable
Symptoms include:
- `provider request failed`
- `provider returned status 400`
- `provider returned status 403`
- `provider response missing choices`
- `provider response assistant message content is not valid JSON`
- `api_key_env`
- `unset environment variable`
- provider authentication failures after selecting a profile that needs a key
Fix:
- Check the selected Scriptorium profile's `api_key_env` field.
- Set that environment variable before running Notarius.
- Do not put raw API keys in Notarius config or file-backed Scriptorium
profiles.
## Prompt Or Structured Output Failure
Symptoms include:
- `prompt not found`
- `prompt render`
- `schema`
- `validation`
- `decode structured output`
Fix:
- Confirm the `base_url` points to an OpenAI-compatible endpoint root. Notarius
posts to `<base_url>/chat/completions`.
- Check `model` and provider credentials.
- Inspect the retained diagnostics `error.log`.
- For 400 and 403 responses, fix the request configuration or credentials.
- For 429 and 5xx responses, the client retries according to `max_retries`; if
the failure persists, inspect the provider response and adjust capacity,
credentials, or model settings.
- The assistant message content must decode as JSON matching the extractor's
structured response schema.
Provider error messages are redacted for configured API key values.
- Ensure production modules register their embedded Scriptorium prompt and
schema assets.
- If the error names a profile, select a Scriptorium profile that is available
through the configured profile source or built-in catalog.
- If the error names generated output validation, retry with a model that
follows JSON schema instructions reliably.
- Inspect retained diagnostics `error.log`, `resolved-pipeline.json`, and
`run-manifest.json` when available. Prompt text, source text, reference
content, raw schema JSON, and secrets are not written to default diagnostics.
- Provider errors are redacted for bearer tokens and configured API key values.
## Scene Chunking Failure
@@ -234,8 +255,8 @@ Fix:
- Validate the pipeline configuration and confirm the input module provides a
transcript source when using `chunk: dnd/scenes`.
- Confirm the LLM profile has a working OpenAI-compatible `base_url`, `model`,
and credentials.
- Confirm the selected Scriptorium profile has a working endpoint, model, and
credentials.
- Inspect retained diagnostics for the run error and resolved pipeline.
- If the error names malformed structured output, retry with a model that
follows structured response schemas reliably.
@@ -244,6 +265,16 @@ Fix:
- Scene boundaries must use exact source-unit IDs, cover the full source
document, be contiguous, and not overlap.
## Session ID
Symptom: external logs or provider traces cannot be correlated with a Notarius
run.
Fix:
- Pass `--session-id <id>` to `notarius run`.
- Use a stable, non-secret identifier from the external orchestrator.
## Output Write Failure
Symptoms include:

2
internal/core/artifacts/artifacts_test.go
View File

@@ -128,7 +128,7 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
PipelineID: "pipeline-1",
PipelineDigest: "sha256:abc123",
LLMProfiles: []LLMProfileManifest{
{ID: "default", Provider: "openai-compatible", Model: "model-a"},
{ID: "default", Provider: "scriptorium", Model: "model-a"},
},
ArtifactLanes: []ArtifactLaneManifest{
{

2
internal/framework/llm/scriptorium_client.go
View File

@@ -16,7 +16,7 @@ import (
"gitea.maximumdirect.net/eric/scriptorium"
)
const scriptoriumProviderName = "openai-compatible"
const scriptoriumProviderName = "scriptorium"
type ScriptoriumClientConfig struct {
ProfileDir string

12
internal/framework/pipeline/runner_test.go
View File

@@ -1184,7 +1184,7 @@ func TestRunManifestIncludesPipelineAndLaneDetails(t *testing.T) {
func TestRunManifestIncludesRunTimingAndLLMProfiles(t *testing.T) {
startedAt := time.Now().Add(-time.Minute).UTC()
profiles := []artifacts.LLMProfileManifest{
{ID: "default", Provider: "openai-compatible", Model: "model-a"},
{ID: "default", Provider: "scriptorium", Model: "model-a"},
}
output, err := New(newRunnerRegistries(t, nil)).Run(context.Background(), RunInput{
@@ -1216,9 +1216,9 @@ func TestRunManifestIncludesProfilesReportedByLLMClient(t *testing.T) {
output, err := New(newRunnerRegistries(t, nil)).Run(context.Background(), RunInput{
Pipeline: resolvedPipeline(),
LLMClient: manifestReportingLLMClient{profiles: []artifacts.LLMProfileManifest{
{ID: "profile-b", Provider: "openai-compatible", Model: "model-b"},
{ID: "profile-a", Provider: "openai-compatible", Model: "model-a"},
{ID: "profile-b", Provider: "openai-compatible", Model: "model-b"},
{ID: "profile-b", Provider: "scriptorium", Model: "model-b"},
{ID: "profile-a", Provider: "scriptorium", Model: "model-a"},
{ID: "profile-b", Provider: "scriptorium", Model: "model-b"},
}},
})
if err != nil {
@@ -1226,8 +1226,8 @@ func TestRunManifestIncludesProfilesReportedByLLMClient(t *testing.T) {
}
want := []artifacts.LLMProfileManifest{
{ID: "profile-a", Provider: "openai-compatible", Model: "model-a"},
{ID: "profile-b", Provider: "openai-compatible", Model: "model-b"},
{ID: "profile-a", Provider: "scriptorium", Model: "model-a"},
{ID: "profile-b", Provider: "scriptorium", Model: "model-b"},
}
if !reflect.DeepEqual(output.Manifest.LLMProfiles, want) {
t.Fatalf("LLMProfiles = %#v, want %#v", output.Manifest.LLMProfiles, want)