Compare commits
49 Commits
1da29e6788
...
v0.2.0
| Author | SHA1 | Date | |
|---|---|---|---|
| 39388e96d4 | |||
| 12ac25bd63 | |||
| 394278e1f2 | |||
| 5cd7f8e737 | |||
| bf3fadf9ae | |||
| 58815aaf33 | |||
| ce857966f1 | |||
| a3bd0c1867 | |||
| b05634ee86 | |||
| 4829f94157 | |||
| 67b315099d | |||
| b5c86de4d7 | |||
| 2eeca2ed5a | |||
| b5aaeb1c78 | |||
| 9171b66a41 | |||
| b4363b3b73 | |||
| 241e9d2a89 | |||
| 715fff7b72 | |||
| d627b91b4f | |||
| a67b3aa76d | |||
| a16dcdfa52 | |||
| 46e4466d28 | |||
| 71a004bfc8 | |||
| f8333f2c15 | |||
| f603f7ac64 | |||
| 7a00e7049c | |||
| 2a9db9a957 | |||
| de046a8f13 | |||
| f1a6574013 | |||
| 4bca6d3103 | |||
| 7c569a3d8c | |||
| 8e04ef9e2b | |||
| 53a330587b | |||
| 7cfab8ada0 | |||
| 5c82b62856 | |||
| de8ed41b34 | |||
| d1eaec4dad | |||
| c0ec068f53 | |||
| 5cbd9e56e4 | |||
| 53490cdb59 | |||
| 7c94b5eeed | |||
| 4f2864fc96 | |||
| 893b03fccf | |||
| 256cc98ddb | |||
| e61e522662 | |||
| a4c7eca87b | |||
| 224a8292c4 | |||
| 64d461fc18 | |||
| fb1134e591 |
2
.gitignore
vendored
2
.gitignore
vendored
@@ -2,6 +2,7 @@
|
|||||||
notarius
|
notarius
|
||||||
notarius-output
|
notarius-output
|
||||||
workspace/
|
workspace/
|
||||||
|
.codebase-memory/
|
||||||
|
|
||||||
# ---> Go
|
# ---> Go
|
||||||
# If you prefer the allow list template instead of the deny list, see community template:
|
# If you prefer the allow list template instead of the deny list, see community template:
|
||||||
@@ -73,4 +74,3 @@ Icon
|
|||||||
Network Trash Folder
|
Network Trash Folder
|
||||||
Temporary Items
|
Temporary Items
|
||||||
.apdisk
|
.apdisk
|
||||||
|
|
||||||
|
|||||||
49
docs/adr/0010-workload-oriented-llm-profile-defaults.md
Normal file
49
docs/adr/0010-workload-oriented-llm-profile-defaults.md
Normal file
@@ -0,0 +1,49 @@
|
|||||||
|
# ADR-0010: Use workload-oriented LLM profile defaults
|
||||||
|
|
||||||
|
**Status:** Accepted
|
||||||
|
**Date:** 2026-08-03
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
LLM-backed D&D operations share an execution-policy choice, but repeating a
|
||||||
|
provider or model-named profile on every module binding ties pipeline structure
|
||||||
|
to a deployment decision. Different environments may require different model,
|
||||||
|
backend, timeout, or reasoning settings while retaining the same workload.
|
||||||
|
|
||||||
|
Notarius also needs a usable default for maintained D&D prompts without making
|
||||||
|
an operator profile mandatory. That default must remain owned by the D&D
|
||||||
|
family, while generic LLM infrastructure stays unaware of domain-specific
|
||||||
|
policy.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Pipelines may name one workload-oriented default profile, inherited only by
|
||||||
|
selected LLM-backed bindings and validators. Binding-level profile IDs remain
|
||||||
|
intentional exceptions, and the run-wide CLI profile override has highest
|
||||||
|
precedence.
|
||||||
|
|
||||||
|
The D&D family owns an embedded fallback profile named `dnd-extraction`.
|
||||||
|
Operators may provide a complete profile with the same ID through a PromptKit
|
||||||
|
filesystem source. PromptKit selects the higher-precedence matching definition;
|
||||||
|
Notarius does not merge profile documents. Production, development, and local
|
||||||
|
deployments can therefore use different execution policy behind one unchanged
|
||||||
|
pipeline ID.
|
||||||
|
|
||||||
|
## Alternatives considered
|
||||||
|
|
||||||
|
- Repeat a model-named profile on every binding. This makes routine deployment
|
||||||
|
policy changes noisy and obscures the shared workload intent.
|
||||||
|
- Require every deployment to install a profile file. This adds configuration
|
||||||
|
friction and leaves maintained D&D prompts without an application-owned
|
||||||
|
fallback.
|
||||||
|
- Put D&D profile policy in generic LLM infrastructure. This breaks domain
|
||||||
|
ownership and makes generic code depend on one workload.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
Pipeline configuration expresses workload intent rather than a specific
|
||||||
|
provider or model. Operators can replace the complete execution policy without
|
||||||
|
editing bindings, while binding-level and run-wide exceptions remain available.
|
||||||
|
Profile changes affect resolved pipeline and checkpoint identity, so they may
|
||||||
|
intentionally cause work to be recomputed. The D&D fallback becomes a
|
||||||
|
maintained application execution-policy asset.
|
||||||
15
docs/cli.md
15
docs/cli.md
@@ -39,16 +39,23 @@ pipeline ID and **--input** are required.
|
|||||||
| **--debug** | Retain a debug bundle for this run. |
|
| **--debug** | Retain a debug bundle for this run. |
|
||||||
| **--debug-dir path** | Override the debug-bundle root. Requires **--debug**. |
|
| **--debug-dir path** | Override the debug-bundle root. Requires **--debug**. |
|
||||||
| **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. |
|
| **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. |
|
||||||
| **--llm-profile id** | Override effective LLM-capable module bindings with one configured profile. |
|
| **--llm-profile id** | Highest-precedence configured profile for selected LLM-backed bindings and validators; it replaces binding and [pipeline](config.md#pipelines) defaults. |
|
||||||
| **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. |
|
| **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. |
|
||||||
|
| **--reasoning-effort value** | Replace the selected PromptKit profile's reasoning effort for every LLM-backed call in this run. The value must be non-empty and the flag may be specified only once. |
|
||||||
|
| **--clear-reasoning-effort** | Clear reasoning effort inherited from the selected PromptKit profile for every LLM-backed call in this run. |
|
||||||
| **--reference selector=path** | Add or replace a file reference binding. Repeatable. |
|
| **--reference selector=path** | Add or replace a file reference binding. Repeatable. |
|
||||||
| **--without-reference selector** | Remove a configured optional reference binding. Repeatable. |
|
| **--without-reference selector** | Remove a configured optional reference binding. Repeatable. |
|
||||||
|
|
||||||
**--chunk_cache** accepts only **auto**, **bypass**, or **refresh**.
|
**--chunk_cache** accepts only **auto**, **bypass**, or **refresh**.
|
||||||
**--debug-dir**, **--output-dir**, **--session-id**, and
|
**--debug-dir**, **--output-dir**, **--session-id**, and
|
||||||
**--recompute-step** reject explicit empty values. **--recompute-step**
|
**--reasoning-effort**, and **--recompute-step** reject explicit empty values.
|
||||||
requires **--resume**; checkpoint requirements and reuse behavior are
|
**--reasoning-effort** and **--clear-reasoning-effort** are mutually exclusive.
|
||||||
documented in [Operations](operations.md).
|
When neither is present, reasoning effort comes from the selected PromptKit
|
||||||
|
profile. These controls apply to the shared run client, including retries and
|
||||||
|
LLM-backed validators, and do not modify configuration or profile files.
|
||||||
|
Persistent reasoning settings remain a PromptKit profile concern.
|
||||||
|
**--recompute-step** requires **--resume**; checkpoint requirements and reuse
|
||||||
|
behavior are documented in [Operations](operations.md).
|
||||||
|
|
||||||
### Reference selectors
|
### Reference selectors
|
||||||
|
|
||||||
|
|||||||
127
docs/config.md
127
docs/config.md
@@ -1,7 +1,7 @@
|
|||||||
# Configuration
|
# Configuration
|
||||||
|
|
||||||
This is the canonical reference for Notarius configuration. Configuration files
|
This is the canonical reference for Notarius configuration. Configuration files
|
||||||
are YAML and must declare version 3. They select pipelines and their modules;
|
are YAML and must declare version 4. They select pipelines and their modules;
|
||||||
the [CLI reference](cli.md) owns invocation syntax, and
|
the [CLI reference](cli.md) owns invocation syntax, and
|
||||||
[Operations](operations.md) owns run-state procedures.
|
[Operations](operations.md) owns run-state procedures.
|
||||||
|
|
||||||
@@ -32,7 +32,8 @@ override the fields listed below.
|
|||||||
single-lane Seriatim-to-spell pipeline.
|
single-lane Seriatim-to-spell pipeline.
|
||||||
- [Complete D&D configuration](../examples/dnd-complete.config.yml) uses
|
- [Complete D&D configuration](../examples/dnd-complete.config.yml) uses
|
||||||
ordered steps, all implemented D&D lanes, generated references, state
|
ordered steps, all implemented D&D lanes, generated references, state
|
||||||
settings, and bounded LLM concurrency.
|
settings, bounded LLM concurrency, and the maintained
|
||||||
|
[operator profile](../examples/profiles/dnd-extraction.yml).
|
||||||
|
|
||||||
Use these complete files as starting points rather than combining the
|
Use these complete files as starting points rather than combining the
|
||||||
illustrative fragments in this reference.
|
illustrative fragments in this reference.
|
||||||
@@ -45,8 +46,8 @@ other than **version** is optional.
|
|||||||
|
|
||||||
| Field | Type | Default | Rules |
|
| Field | Type | Default | Rules |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| **version** | integer | none | Required; must be 3. |
|
| **version** | integer | none | Required; must be 4. |
|
||||||
| **scriptorium** | object | none | Profile source configuration. |
|
| **promptkit** | object | none | Profile source and optional local-backend configuration. |
|
||||||
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
|
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
|
||||||
| **concurrency** | object | see below | Global LLM and extraction limits. |
|
| **concurrency** | object | see below | Global LLM and extraction limits. |
|
||||||
| **output** | object | see below | Published output settings. |
|
| **output** | object | see below | Published output settings. |
|
||||||
@@ -69,20 +70,85 @@ Built-in defaults are:
|
|||||||
An empty cache directory in YAML deliberately selects the corresponding
|
An empty cache directory in YAML deliberately selects the corresponding
|
||||||
per-user root. An explicit empty output or debug directory is invalid.
|
per-user root. An explicit empty output or debug directory is invalid.
|
||||||
|
|
||||||
## Scriptorium Profiles
|
## PromptKit Profiles
|
||||||
|
|
||||||
The optional **scriptorium** object selects one source of profile definitions:
|
The optional **promptkit** object selects one source of profile definitions and
|
||||||
|
may register one conventional local OpenAI-compatible backend:
|
||||||
|
|
||||||
|
~~~yaml
|
||||||
|
version: 4
|
||||||
|
|
||||||
|
promptkit:
|
||||||
|
profile_dir: ./profiles
|
||||||
|
# profile_file: ./profiles.yml
|
||||||
|
local_backend:
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
concurrency_limit: 2
|
||||||
|
~~~
|
||||||
|
|
||||||
| Field | Type | Rules |
|
| Field | Type | Rules |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| **profile_dir** | string | Non-empty directory containing profile files. |
|
| **profile_dir** | string | Non-empty directory containing profile files. |
|
||||||
| **profile_file** | string | Non-empty profile file. |
|
| **profile_file** | string | Non-empty profile file. |
|
||||||
|
| **local_backend** | object | Optional registration for the conventional PromptKit backend ID **local**. |
|
||||||
|
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
|
||||||
|
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
|
||||||
|
|
||||||
Set at most one of these fields. Profile IDs used by a binding must be available
|
Set at most one of **profile_dir** and **profile_file**. Relative values use
|
||||||
from the selected Scriptorium profile source when the pipeline is resolved.
|
the process working directory, not the configuration file's directory. The
|
||||||
Keep credentials out of this file: configure a profile to read its credential
|
complete example's `./examples/profiles/dnd-extraction.yml` value is therefore
|
||||||
from an environment variable, then set that environment variable only in the
|
valid when Notarius is launched from the repository root; use an absolute path
|
||||||
run environment.
|
for services and containers.
|
||||||
|
|
||||||
|
An operator source is optional. For a requested ID, PromptKit checks the
|
||||||
|
configured operator source first, then Notarius's embedded fallback profiles,
|
||||||
|
then its own built-in catalog. A matching profile is complete: it replaces a
|
||||||
|
lower-precedence definition rather than merging with it. The maintained
|
||||||
|
[`dnd-extraction` operator profile](../examples/profiles/dnd-extraction.yml)
|
||||||
|
is a secret-free deployment artifact; production, development, and local
|
||||||
|
deployments can each provide a complete definition with that same workload ID.
|
||||||
|
Use workload-oriented IDs for new profiles instead of model names.
|
||||||
|
[Operations](operations.md#promptkit-profile-deployment) owns the deployment
|
||||||
|
workflow and credential-handling guidance.
|
||||||
|
|
||||||
|
When **local_backend** is present, its endpoint is trimmed and must use HTTP or
|
||||||
|
HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are
|
||||||
|
allowed. User information, queries, and fragments are rejected. A zero
|
||||||
|
**concurrency_limit** leaves the local backend unrestricted inside PromptKit;
|
||||||
|
a positive value limits simultaneous local generations. The application-wide
|
||||||
|
**concurrency.total_llm** limit still applies in both cases. Neither local
|
||||||
|
backend field has an environment override. Omitting **local_backend** registers
|
||||||
|
nothing and preserves existing built-in and endpoint-only profile behavior.
|
||||||
|
|
||||||
|
A file-backed PromptKit profile selects the registration by its case-sensitive
|
||||||
|
backend ID:
|
||||||
|
|
||||||
|
~~~yaml
|
||||||
|
id: local-summary
|
||||||
|
backend: local
|
||||||
|
model: example-model
|
||||||
|
~~~
|
||||||
|
|
||||||
|
Keep credentials out of the local-backend object. A PromptKit profile may name
|
||||||
|
its credential environment variable through `api_key_env`; set that variable
|
||||||
|
only in the run environment. PromptKit owns the
|
||||||
|
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md).
|
||||||
|
The [PromptKit upstream boundary](integrations/pkg-promptkit.md) identifies the
|
||||||
|
supported package API, and [Operations](operations.md#operational-limits)
|
||||||
|
describes the effective concurrency layers.
|
||||||
|
|
||||||
|
`notarius config validate --pipeline <id>` resolves the selected pipeline and
|
||||||
|
inspects every explicit effective profile without contacting a provider or
|
||||||
|
requiring credential values. It rejects absent, malformed, or incompatible
|
||||||
|
profiles before a run prepares modules. Credential availability is checked only
|
||||||
|
when a generation is prepared.
|
||||||
|
|
||||||
|
## Migrating Version 3 Configuration
|
||||||
|
|
||||||
|
Version 3 files are not decoded or rewritten. Change **version: 3** to
|
||||||
|
**version: 4** and rename the top-level **scriptorium:** section to
|
||||||
|
**promptkit:**. Version 4 decoding is strict, so a remaining **scriptorium**
|
||||||
|
field is rejected as unknown.
|
||||||
|
|
||||||
## Operational Environment Variables
|
## Operational Environment Variables
|
||||||
|
|
||||||
@@ -138,6 +204,7 @@ Each **pipelines** entry has a unique, non-empty ID and the following shape:
|
|||||||
~~~yaml
|
~~~yaml
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input: seriatim
|
||||||
chunk: generic
|
chunk: generic
|
||||||
output: json
|
output: json
|
||||||
@@ -150,6 +217,7 @@ pipelines:
|
|||||||
|
|
||||||
| Field | Type | Default | Rules |
|
| Field | Type | Default | Rules |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
|
| **llm_profile** | string | none | Optional non-empty default PromptKit profile ID for selected LLM-backed bindings and validators. An explicitly present blank value is invalid. |
|
||||||
| **input** | module binding | none | Required. |
|
| **input** | module binding | none | Required. |
|
||||||
| **chunk** | module binding | **generic** | Optional. |
|
| **chunk** | module binding | **generic** | Optional. |
|
||||||
| **output** | module binding | **json** | Optional. |
|
| **output** | module binding | **json** | Optional. |
|
||||||
@@ -163,6 +231,12 @@ needs a unique non-empty **id**, an **artifacts** map, and may have
|
|||||||
**references**. A lane ID must not appear more than once in a pipeline,
|
**references**. A lane ID must not appear more than once in a pipeline,
|
||||||
including across explicit steps.
|
including across explicit steps.
|
||||||
|
|
||||||
|
For each selected LLM-backed binding or validator, profile selection occurs
|
||||||
|
after module, validator, and `--only` lane selection. It uses the
|
||||||
|
run-level **--llm-profile** value first, then the binding's **llm_profile**,
|
||||||
|
then the pipeline's **llm_profile**, and finally the PromptKit default.
|
||||||
|
Deterministic bindings do not receive these defaults or run overrides.
|
||||||
|
|
||||||
A lane has these fields:
|
A lane has these fields:
|
||||||
|
|
||||||
| Field | Type | Default | Rules |
|
| Field | Type | Default | Rules |
|
||||||
@@ -190,7 +264,7 @@ Use an object for fields:
|
|||||||
~~~yaml
|
~~~yaml
|
||||||
extract:
|
extract:
|
||||||
module: dnd/spells
|
module: dnd/spells
|
||||||
llm_profile: gemini-2-flash
|
llm_profile: dnd-extraction
|
||||||
retries: 2
|
retries: 2
|
||||||
references:
|
references:
|
||||||
spell_catalog: ./dnd-spell-catalog.json
|
spell_catalog: ./dnd-spell-catalog.json
|
||||||
@@ -199,7 +273,7 @@ extract:
|
|||||||
| Binding field | Type | Default | Rules |
|
| Binding field | Type | Default | Rules |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| **module** | string | none | Required for an object binding. Must be a registered compatible key. |
|
| **module** | string | none | Required for an object binding. Must be a registered compatible key. |
|
||||||
| **llm_profile** | string | none | Optional non-empty Scriptorium profile ID. |
|
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID for an LLM-backed binding. It overrides the pipeline default unless the run supplies **--llm-profile**. |
|
||||||
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. |
|
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. |
|
||||||
| **options** | object | none | Must satisfy the selected module. |
|
| **options** | object | none | Must satisfy the selected module. |
|
||||||
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
|
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
|
||||||
@@ -209,21 +283,44 @@ Omitting **validators** uses the registered chain. **validators: []** selects
|
|||||||
an empty chain; a non-empty list replaces the chain in the listed order.
|
an empty chain; a non-empty list replaces the chain in the listed order.
|
||||||
Validator bindings accept only **module**, **llm_profile**, and **options**.
|
Validator bindings accept only **module**, **llm_profile**, and **options**.
|
||||||
They reject **references**, **retries**, and nested **validators**. Deterministic
|
They reject **references**, **retries**, and nested **validators**. Deterministic
|
||||||
validators reject an explicit **llm_profile**.
|
validators reject an explicit **llm_profile**. Deterministic module bindings
|
||||||
|
also reject an explicit **llm_profile**.
|
||||||
|
|
||||||
The **json** output module accepts one option:
|
The **json** output module accepts optional **include_chunk_map** and
|
||||||
|
**evidence_context** settings:
|
||||||
|
|
||||||
~~~yaml
|
~~~yaml
|
||||||
output:
|
output:
|
||||||
module: json
|
module: json
|
||||||
options:
|
options:
|
||||||
include_chunk_map: true
|
include_chunk_map: true
|
||||||
|
evidence_context:
|
||||||
|
enabled: true
|
||||||
|
window_units: 3
|
||||||
|
lanes:
|
||||||
|
- npcs
|
||||||
|
- spells
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
**include_chunk_map** is a boolean and defaults to false. It adds the accepted
|
**include_chunk_map** is a boolean and defaults to false. It adds the accepted
|
||||||
chunk map when one exists; its wire format is defined in the
|
chunk map when one exists; its wire format is defined in the
|
||||||
[chunk-map contract](integrations/chunk-map.md).
|
[chunk-map contract](integrations/chunk-map.md).
|
||||||
|
|
||||||
|
Omitting **evidence_context** disables evidence publication. When present, it
|
||||||
|
is an object with these strict fields:
|
||||||
|
|
||||||
|
| Field | Type | Rules |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| **enabled** | boolean | Required. `false` permits no other evidence fields. |
|
||||||
|
| **lanes** | array of strings | Required and non-empty when enabled. Each value is trimmed and must be unique; every value must name a configured pipeline lane. |
|
||||||
|
| **window_units** | non-negative integer | Optional when enabled; defaults to 3. Zero retains only directly cited units. |
|
||||||
|
|
||||||
|
Unknown outer or nested option fields are rejected, as are incompatible YAML
|
||||||
|
types. The allowlist remains valid when a run uses lane filtering: a configured
|
||||||
|
lane that is not active for that invocation simply contributes no evidence.
|
||||||
|
Evidence publication is opt-in because it can persist source text and metadata.
|
||||||
|
Its payload contract is [Published Evidence Context](integrations/evidence-context.md).
|
||||||
|
|
||||||
## References And Ordered Handoffs
|
## References And Ordered Handoffs
|
||||||
|
|
||||||
Reference maps bind named slots that the selected target declares. A scalar is
|
Reference maps bind named slots that the selected target declares. A scalar is
|
||||||
|
|||||||
@@ -49,6 +49,14 @@ guessed filename. Before decoding a selected payload, verify its descriptor's
|
|||||||
media type and schema identity against the relevant published artifact
|
media type and schema identity against the relevant published artifact
|
||||||
contract. The JSON bundle contract links to the available lane contracts.
|
contract. The JSON bundle contract links to the available lane contracts.
|
||||||
|
|
||||||
|
If `index.json` has an `evidence_context` descriptor, treat it as a
|
||||||
|
pipeline-wide artifact rather than a lane entry. Verify its six descriptor
|
||||||
|
fields before decoding the linked file according to the [Published Evidence
|
||||||
|
Context contract](../integrations/evidence-context.md). Use each
|
||||||
|
`evidence_refs` entry as the citation to source material. Its surrounding
|
||||||
|
context range and included units explain the citation, but do not widen or
|
||||||
|
replace the cited source reference.
|
||||||
|
|
||||||
A zero exit status may still report rejected outputs, warnings, or absent
|
A zero exit status may still report rejected outputs, warnings, or absent
|
||||||
lanes. The caller decides which lane IDs are required for its own work and
|
lanes. The caller decides which lane IDs are required for its own work and
|
||||||
which are optional; it should make that decision explicitly rather than infer
|
which are optional; it should make that decision explicitly rather than infer
|
||||||
@@ -61,4 +69,6 @@ Keep the receipt with the published `manifest.json`, and retain
|
|||||||
them. Treat the input, output bundle, cache, debug bundle, and captured process
|
them. Treat the input, output bundle, cache, debug bundle, and captured process
|
||||||
logs as potentially sensitive data. Apply the caller's access controls and
|
logs as potentially sensitive data. Apply the caller's access controls and
|
||||||
retention policy, and avoid copying secrets into arguments, logs, or
|
retention policy, and avoid copying secrets into arguments, logs, or
|
||||||
provenance records.
|
provenance records. An evidence-context artifact contains source-unit text and
|
||||||
|
metadata, and selected lanes can cover most of an input; preserve and share it
|
||||||
|
only when that source content is authorized for the recipient.
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ implemented component map.
|
|||||||
| Configuration loading, resolution, or user-visible configuration behavior | [Configuration Internals](internal/configuration.md) and [Configuration](config.md) | The internal guide owns loading and resolution mechanics; the reference owns the configuration contract. |
|
| Configuration loading, resolution, or user-visible configuration behavior | [Configuration Internals](internal/configuration.md) and [Configuration](config.md) | The internal guide owns loading and resolution mechanics; the reference owns the configuration contract. |
|
||||||
| Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
|
| Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
|
||||||
| Production modules or validators | [Module Internals](internal/modules.md), [D&D Module Internals](internal/dnd.md), and [D&D integration contracts](integrations/) | The generic guide owns extension mechanics, the D&D guide owns shared family conventions, and the contracts own durable output shapes. |
|
| Production modules or validators | [Module Internals](internal/modules.md), [D&D Module Internals](internal/dnd.md), and [D&D integration contracts](integrations/) | The generic guide owns extension mechanics, the D&D guide owns shared family conventions, and the contracts own durable output shapes. |
|
||||||
| LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and Scriptorium integration. |
|
| LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and PromptKit integration. |
|
||||||
| Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. |
|
| Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. |
|
||||||
| External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. |
|
| External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. |
|
||||||
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
|
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
|
||||||
|
|||||||
116
docs/integrations/evidence-context.md
Normal file
116
docs/integrations/evidence-context.md
Normal file
@@ -0,0 +1,116 @@
|
|||||||
|
# Published Evidence Context
|
||||||
|
|
||||||
|
This contract defines the optional `source/evidence-context` artifact emitted
|
||||||
|
by the production JSON output. Its configuration is owned by
|
||||||
|
[Configuration](../config.md#module-bindings-and-validators); its logical-file
|
||||||
|
discovery is owned by [Published JSON Output](json-output.md).
|
||||||
|
|
||||||
|
## Identity And Discovery
|
||||||
|
|
||||||
|
When enabled, the JSON bundle contains `evidence-context.json` and an
|
||||||
|
`index.json` `evidence_context` descriptor with the same six fields as other
|
||||||
|
pipeline-wide artifact descriptors.
|
||||||
|
|
||||||
|
| Property | Value |
|
||||||
|
| --- | --- |
|
||||||
|
| Artifact kind | `source/evidence-context` |
|
||||||
|
| Media type | `application/json` |
|
||||||
|
| Schema ID | `notarius.source.evidence_context` |
|
||||||
|
| Schema name | `notarius_source_evidence_context_v1` |
|
||||||
|
| Schema version | `v1` |
|
||||||
|
| Logical file | `evidence-context.json` |
|
||||||
|
|
||||||
|
Consumers must discover the file from the descriptor, verify all six descriptor
|
||||||
|
fields, and decode only a supported schema version. The descriptor is optional:
|
||||||
|
its absence means evidence publication was not enabled for that bundle.
|
||||||
|
|
||||||
|
## Payload
|
||||||
|
|
||||||
|
The v1 payload is a JSON object with required `source_id`, `source_digest`,
|
||||||
|
`window_units`, `selected_lanes`, and `contexts` fields. `selected_lanes` and
|
||||||
|
`contexts` are always arrays; an enabled configuration with no accepted direct
|
||||||
|
evidence publishes `contexts: []`.
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"source_id": "session-alpha",
|
||||||
|
"source_digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
|
||||||
|
"window_units": 1,
|
||||||
|
"selected_lanes": ["npcs", "spells"],
|
||||||
|
"contexts": [
|
||||||
|
{
|
||||||
|
"context_ref": {
|
||||||
|
"source_id": "session-alpha",
|
||||||
|
"start_unit_id": 10,
|
||||||
|
"end_unit_id": 20
|
||||||
|
},
|
||||||
|
"evidence_refs": [
|
||||||
|
{
|
||||||
|
"lane_id": "spells",
|
||||||
|
"source_ref": {
|
||||||
|
"source_id": "session-alpha",
|
||||||
|
"start_unit_id": 10,
|
||||||
|
"end_unit_id": 10
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"units": [
|
||||||
|
{
|
||||||
|
"id": 10,
|
||||||
|
"kind": "transcript_segment",
|
||||||
|
"text": "Aria casts Cure Wounds.",
|
||||||
|
"ref": {
|
||||||
|
"source_id": "session-alpha",
|
||||||
|
"start_unit_id": 10,
|
||||||
|
"end_unit_id": 10
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 20,
|
||||||
|
"kind": "transcript_segment",
|
||||||
|
"text": "The party regroups.",
|
||||||
|
"ref": {
|
||||||
|
"source_id": "session-alpha",
|
||||||
|
"start_unit_id": 20,
|
||||||
|
"end_unit_id": 20
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Each context requires a `context_ref` object and `evidence_refs` and `units`
|
||||||
|
arrays. `context_ref` identifies the first and last included unit. Each
|
||||||
|
evidence entry contains a selected `lane_id` and an original `source_ref`. A
|
||||||
|
unit uses the existing source-unit shape: required `id`, `kind`, `text`, and
|
||||||
|
self `ref`, plus optional JSON-object `metadata`. Fixed payload objects reject
|
||||||
|
unknown fields; unit metadata may contain application-defined JSON values.
|
||||||
|
|
||||||
|
## Citations And Context
|
||||||
|
|
||||||
|
`evidence_refs` are the authoritative citations. They identify the direct
|
||||||
|
references emitted by accepted normalized artifacts. `context_ref` and the
|
||||||
|
units collection include those cited units plus nearby source units selected by
|
||||||
|
the configured window. They are explanatory context, not widened citations.
|
||||||
|
|
||||||
|
Only accepted outputs from the configured lane allowlist contribute. Rejected,
|
||||||
|
failed, absent, and lane-filtered outputs do not contribute. The artifact never
|
||||||
|
contains raw input bytes, prompts, model responses, auxiliary reference
|
||||||
|
content, credentials, or filesystem paths.
|
||||||
|
|
||||||
|
## Ordering And Compatibility
|
||||||
|
|
||||||
|
The selected lane allowlist is lexical. Contexts and units are in source
|
||||||
|
document position order, not numeric unit-ID order. Direct evidence entries
|
||||||
|
are deterministically ordered by lane and source reference. Overlapping or
|
||||||
|
contiguous windows merge, and each source unit appears at most once in the
|
||||||
|
resulting contexts.
|
||||||
|
|
||||||
|
The artifact is additive to the JSON bundle and is not a lane payload,
|
||||||
|
normalized-output count, checkpoint, or generated reference. Consumers that
|
||||||
|
do not need it must tolerate the absent optional descriptor. Consumers that do
|
||||||
|
use it should preserve the artifact and its schema identity with the run
|
||||||
|
provenance, and should treat its source text and metadata as sensitive durable
|
||||||
|
content.
|
||||||
@@ -3,14 +3,14 @@
|
|||||||
This document defines the logical JSON bundle emitted by the production JSON
|
This document defines the logical JSON bundle emitted by the production JSON
|
||||||
output encoder. The bundle’s physical destination, atomic publication, and
|
output encoder. The bundle’s physical destination, atomic publication, and
|
||||||
retention are operational concerns; see [Operations](../operations.md#output-bundles).
|
retention are operational concerns; see [Operations](../operations.md#output-bundles).
|
||||||
Output configuration, including chunk-map export, belongs in
|
Output configuration, including chunk-map and evidence-context publication, belongs in
|
||||||
[Configuration](../config.md#module-bindings-and-validators).
|
[Configuration](../config.md#module-bindings-and-validators).
|
||||||
|
|
||||||
## Bundle Layout
|
## Bundle Layout
|
||||||
|
|
||||||
All paths below are logical, relative, slash-separated bundle paths. The
|
All paths below are logical, relative, slash-separated bundle paths. The
|
||||||
encoder always emits the first four JSON files below and adds lane or chunk-map
|
encoder always emits the first four JSON files below and adds lane or
|
||||||
files when their corresponding artifacts are available:
|
pipeline-wide artifact files when their corresponding artifacts are available:
|
||||||
|
|
||||||
A subprocess caller first obtains the physical bundle root from the
|
A subprocess caller first obtains the physical bundle root from the
|
||||||
[run-result receipt](run-result.md), then resolves `index.json` beneath that
|
[run-result receipt](run-result.md), then resolves `index.json` beneath that
|
||||||
@@ -24,6 +24,7 @@ root for the logical discovery described here.
|
|||||||
| `warnings.json` | Accepted-output and run warnings. |
|
| `warnings.json` | Accepted-output and run warnings. |
|
||||||
| `lanes/<safe-lane-id>.json` | One normalized artifact payload for each lane. |
|
| `lanes/<safe-lane-id>.json` | One normalized artifact payload for each lane. |
|
||||||
| `chunk-map.json` | Optional accepted chunk map, when its export is enabled and available. |
|
| `chunk-map.json` | Optional accepted chunk map, when its export is enabled and available. |
|
||||||
|
| `evidence-context.json` | Optional source-context artifact, when evidence publication is enabled. |
|
||||||
|
|
||||||
JSON files are pretty-printed with a trailing newline. Lane payloads are
|
JSON files are pretty-printed with a trailing newline. Lane payloads are
|
||||||
accepted only when their media type is `application/json`.
|
accepted only when their media type is `application/json`.
|
||||||
@@ -49,13 +50,15 @@ normalized lanes has this valid minimal index:
|
|||||||
| `rejected_file` | Yes | Always `rejected.json`. |
|
| `rejected_file` | Yes | Always `rejected.json`. |
|
||||||
| `warnings_file` | Yes | Always `warnings.json`. |
|
| `warnings_file` | Yes | Always `warnings.json`. |
|
||||||
| `chunk_map` | No | Descriptor for the pipeline-wide `chunk-map.json`; never a lane descriptor. |
|
| `chunk_map` | No | Descriptor for the pipeline-wide `chunk-map.json`; never a lane descriptor. |
|
||||||
|
| `evidence_context` | No | Descriptor for the pipeline-wide `evidence-context.json`; never a lane descriptor. |
|
||||||
|
|
||||||
Each lane descriptor has required `lane_id` and `file`. It may also include
|
Each lane descriptor has required `lane_id` and `file`. It may also include
|
||||||
`media_type`, `module_key`, `schema_id`, `schema_name`, and `schema_version`
|
`media_type`, `module_key`, `schema_id`, `schema_name`, and `schema_version`
|
||||||
when supplied by the normalized artifact. A `chunk_map` descriptor contains
|
when supplied by the normalized artifact. Each pipeline-wide artifact
|
||||||
`artifact_kind`, `file`, `media_type`, `schema_id`, `schema_name`, and
|
descriptor (`chunk_map` or `evidence_context`) contains `artifact_kind`,
|
||||||
`schema_version`; its payload is defined by the
|
`file`, `media_type`, `schema_id`, `schema_name`, and `schema_version`. Their
|
||||||
[Accepted Chunk Map contract](chunk-map.md).
|
payloads are defined by the [Accepted Chunk Map contract](chunk-map.md) and
|
||||||
|
[Published Evidence Context](evidence-context.md), respectively.
|
||||||
|
|
||||||
The lane path is derived from its lane ID. Characters outside letters, digits,
|
The lane path is derived from its lane ID. Characters outside letters, digits,
|
||||||
periods, underscores, and hyphens become underscores; `..` sequences are
|
periods, underscores, and hyphens become underscores; `..` sequences are
|
||||||
@@ -95,6 +98,21 @@ summarize results without embedding lane payload bytes. A chunk-plan summary is
|
|||||||
provenance for the plan used by this run; cache records, debug artifacts, and
|
provenance for the plan used by this run; cache records, debug artifacts, and
|
||||||
other operational state are not published as bundle files.
|
other operational state are not published as bundle files.
|
||||||
|
|
||||||
|
Each `llm_profiles` entry identifies effective, non-secret LLM execution
|
||||||
|
provenance:
|
||||||
|
|
||||||
|
| Field | Required | Meaning |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `id` | Yes | Selected PromptKit profile identifier. |
|
||||||
|
| `provider` | No | Notarius adapter provider identifier. |
|
||||||
|
| `model` | No | Effective provider model identifier. |
|
||||||
|
| `backend_id` | No | Effective PromptKit backend registration identifier. Endpoint-only profiles omit it. |
|
||||||
|
| `reasoning_effort` | No | Effective opaque provider reasoning setting. An empty or explicitly cleared setting is omitted. |
|
||||||
|
|
||||||
|
These values describe observed execution; they are not a backend-registration
|
||||||
|
interface. Entries that differ by backend or effective reasoning remain
|
||||||
|
distinct even when their profile, provider, and model are otherwise equal.
|
||||||
|
|
||||||
## Rejections And Warnings
|
## Rejections And Warnings
|
||||||
|
|
||||||
`rejected.json` is always an object with a `rejected` array. Each entry has
|
`rejected.json` is always an object with a `rejected` array. Each entry has
|
||||||
|
|||||||
106
docs/integrations/pkg-promptkit.md
Normal file
106
docs/integrations/pkg-promptkit.md
Normal file
@@ -0,0 +1,106 @@
|
|||||||
|
# PromptKit Integration
|
||||||
|
|
||||||
|
Notarius pins
|
||||||
|
[`gitea.maximumdirect.net/eric/promptkit` v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0)
|
||||||
|
as its in-process prompt engine. The upstream
|
||||||
|
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/consumers/pkg-promptkit.md)
|
||||||
|
owns the public engine API, and the upstream
|
||||||
|
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
|
||||||
|
owns prompt, profile, and schema file contracts.
|
||||||
|
|
||||||
|
## Supported Boundary
|
||||||
|
|
||||||
|
Notarius relies on the root `promptkit` package to:
|
||||||
|
|
||||||
|
- construct an `Engine` with filesystem-backed prompt, schema, and optional
|
||||||
|
operator and application-fallback profile sources;
|
||||||
|
- prepare one frozen execution from a `RunRequest` with named inline artifacts,
|
||||||
|
variables, a direct session ID, prompt identity, and profile selection, then
|
||||||
|
record credential-redacted details and run that exact execution;
|
||||||
|
- return rendered debug material, validated structured output, selected
|
||||||
|
profile, backend, effective model metadata, and token usage;
|
||||||
|
- register the optional conventional `local` backend through `BackendLocal`,
|
||||||
|
`LocalBackend`, and `WithBackend`;
|
||||||
|
- distinguish structured-output validation failure from execution failure; and
|
||||||
|
- identify a missing explicit profile through `ErrProfileNotFound` and backend
|
||||||
|
admission exhaustion through `ErrCapacityExceeded`.
|
||||||
|
|
||||||
|
The pinned
|
||||||
|
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/backends.go)
|
||||||
|
owns the registration and backend-capacity contract.
|
||||||
|
|
||||||
|
For one completion, the adapter calls `PrepareExecution`, takes a
|
||||||
|
caller-owned `Details` snapshot, and calls `RunPrepared` for that same opaque
|
||||||
|
prepared execution. It defers `Discard` for every unexecuted handle. Explicit
|
||||||
|
profile preflight uses `Engine.InspectProfile`; it does not prepare a synthetic
|
||||||
|
prompt. PromptKit's prepared handle, inspection result, and capacity-error
|
||||||
|
types stay inside the Notarius LLM adapter.
|
||||||
|
|
||||||
|
When a PromptKit profile and runtime override leave `temperature`, `max_tokens`,
|
||||||
|
or `top_p` unset, Notarius leaves that control unset as well. Compatible
|
||||||
|
providers therefore apply their own defaults; an operator that requires a
|
||||||
|
specific sampling value must select it explicitly in the profile or runtime
|
||||||
|
override.
|
||||||
|
|
||||||
|
Notarius does not use PromptKit's optional `ArtifactReader`. It materializes
|
||||||
|
source and reference content itself and supplies owned inline artifacts at the
|
||||||
|
adapter boundary. It also retains responsibility for pipeline retries,
|
||||||
|
scheduling, debug persistence, redaction, profile provenance, and conversion
|
||||||
|
from private model responses into durable domain artifacts.
|
||||||
|
|
||||||
|
Notarius sends its trimmed run session through PromptKit's direct session
|
||||||
|
field, which is authoritative for provider session behavior. It also retains
|
||||||
|
the same value as the `session_id` prompt variable for maintained prompt
|
||||||
|
compatibility. Session IDs are stable, non-secret correlation identifiers and
|
||||||
|
may be exposed to providers and provider observability.
|
||||||
|
|
||||||
|
Notarius records PromptKit's selected backend ID and effective reasoning
|
||||||
|
setting as optional run-manifest provenance. Endpoint-only profiles have no
|
||||||
|
backend ID. Debug prompt material also retains the selected backend ID and
|
||||||
|
PromptKit's stable lower-case `effective_model_params` JSON, which may include
|
||||||
|
`backend_id`. Notarius production configuration exposes one optional
|
||||||
|
conventional `local` registration. It does not expose a general user-defined
|
||||||
|
PromptKit backend registry. Endpoint-only profiles remain supported unchanged.
|
||||||
|
|
||||||
|
Notarius retains its application-wide scheduled client around the PromptKit
|
||||||
|
adapter. PromptKit may apply a narrower limit for the selected backend;
|
||||||
|
endpoint-only profiles have no such backend limit. The adapter translates
|
||||||
|
PromptKit capacity rejection into the provider-neutral Notarius
|
||||||
|
`ErrLLMCapacityExceeded` contract. It may include the normalized selected
|
||||||
|
backend ID in safe diagnostic context, without exposing PromptKit's capacity
|
||||||
|
error type, and leaves retries to the calling pipeline stage.
|
||||||
|
|
||||||
|
## Profile Sources And Compatibility
|
||||||
|
|
||||||
|
Notarius gives PromptKit the configured operator profile source, registered
|
||||||
|
application fallback profile assets, and optional backend registration through
|
||||||
|
the same construction path for inspection and execution. PromptKit owns the
|
||||||
|
resulting source precedence and strict profile parsing: a matching operator
|
||||||
|
profile is a complete replacement for a fallback or built-in profile, while an
|
||||||
|
invalid matching document fails instead of falling through. The operator
|
||||||
|
configuration and deployment workflow are defined in
|
||||||
|
[Configuration](../config.md#promptkit-profiles) and
|
||||||
|
[Operations](../operations.md#promptkit-profile-deployment).
|
||||||
|
|
||||||
|
Notarius supports this boundary against PromptKit v0.5.0. Its fallback source,
|
||||||
|
prepared-execution, inspection, and typed capacity APIs are used as public
|
||||||
|
upstream contracts; other PromptKit APIs or file-format behavior are not
|
||||||
|
implicitly supported. A dependency upgrade requires reviewing the adapter,
|
||||||
|
profile-source construction, and this compatibility statement against the
|
||||||
|
pinned upstream documentation.
|
||||||
|
|
||||||
|
## Notarius Ownership
|
||||||
|
|
||||||
|
[LLM Runtime Internals](../internal/llm.md) describes how Notarius mounts
|
||||||
|
module assets, maps its transport-neutral completion contract, prepares and
|
||||||
|
executes requests, validates output, records provenance, captures debug
|
||||||
|
material, redacts errors, and preserves timeout ownership.
|
||||||
|
[D&D Module Internals](../internal/dnd.md) owns the embedded
|
||||||
|
`dnd-extraction` fallback profile and the maintained D&D prompt defaults.
|
||||||
|
[Configuration](../config.md#promptkit-profiles) defines how a Notarius
|
||||||
|
configuration selects one PromptKit profile source and optionally registers
|
||||||
|
the conventional local backend.
|
||||||
|
|
||||||
|
PromptKit API or format changes outside this boundary are not implicitly
|
||||||
|
supported. Updating the pinned version requires reviewing the adapter and
|
||||||
|
profile/configuration contracts against the upstream documentation.
|
||||||
@@ -38,7 +38,7 @@ For another output module, its absence does not indicate a failed run.
|
|||||||
"normalized_output_count": 6,
|
"normalized_output_count": 6,
|
||||||
"rejected_output_count": 2,
|
"rejected_output_count": 2,
|
||||||
"warning_count": 1,
|
"warning_count": 1,
|
||||||
"validation_status": "approved"
|
"validation_status": "rejected"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -38,9 +38,14 @@ in [Configuration Internals](configuration.md).
|
|||||||
|
|
||||||
Configuration validation without a selected pipeline checks structural
|
Configuration validation without a selected pipeline checks structural
|
||||||
configuration only. Validation with a selected pipeline also builds the
|
configuration only. Validation with a selected pipeline also builds the
|
||||||
effective catalog, resolves the pipeline, and verifies explicitly selected
|
effective catalog, resolves the pipeline, and verifies every explicit effective
|
||||||
Scriptorium profiles. Pipeline listing validates configuration before returning
|
PromptKit profile. Selected LLM-backed input, chunk, lane, output, and validator
|
||||||
normalized, sorted identifiers.
|
profiles are inspected
|
||||||
|
against the configured PromptKit source and backend registrations without
|
||||||
|
loading a prompt or performing generation, so an unknown or invalid profile
|
||||||
|
fails before pipeline preparation. Credential availability remains an
|
||||||
|
execution-time concern. Pipeline listing validates configuration before
|
||||||
|
returning normalized, sorted identifiers.
|
||||||
|
|
||||||
## Production Composition
|
## Production Composition
|
||||||
|
|
||||||
@@ -51,12 +56,24 @@ catalog used for resolution and the concrete constructors used for preparation.
|
|||||||
Tests may provide a catalog or registries instead; production code must not
|
Tests may provide a catalog or registries instead; production code must not
|
||||||
silently merge an injected partial catalog with production registrations.
|
silently merge an injected partial catalog with production registrations.
|
||||||
|
|
||||||
The production LLM factory builds the Scriptorium-backed client from resolved
|
The production LLM factory builds one PromptKit-backed client from the resolved
|
||||||
configuration, creates one scheduler from the effective global LLM limit, and
|
**promptkit.profile_dir** or **promptkit.profile_file** source, attaches the
|
||||||
wraps the client before it reaches modules. Registration and LLM construction
|
profile-provenance recorder, creates one scheduler from the effective global
|
||||||
errors are returned before a pipeline is prepared. Concrete module keys and
|
LLM limit, and wraps the client before it reaches modules. Registration and LLM
|
||||||
validator chains are public configuration choices and remain documented in
|
construction errors are returned before a pipeline is prepared. Configuration
|
||||||
[Configuration](../config.md).
|
field definitions remain in [Configuration](../config.md#promptkit-profiles);
|
||||||
|
the D&D registrar's fallback profile assets and the adapter mechanics remain in
|
||||||
|
[LLM Runtime](llm.md).
|
||||||
|
|
||||||
|
The factory also accepts `LLMRuntimeOverrides`, whose reasoning pointer
|
||||||
|
preserves inherit, replace, and clear states across the composition boundary.
|
||||||
|
Run orchestration constructs this value from the mutually exclusive
|
||||||
|
`--reasoning-effort` and `--clear-reasoning-effort` controls. Absence preserves
|
||||||
|
a nil pointer, replacement is trimmed, and clear uses a non-nil empty string.
|
||||||
|
The same override reaches the one shared production client, checkpoint
|
||||||
|
identity, and debug invocation metadata. Persistent reasoning configuration
|
||||||
|
remains owned by PromptKit profiles; Notarius configuration has no reasoning
|
||||||
|
field.
|
||||||
|
|
||||||
## Run Orchestration
|
## Run Orchestration
|
||||||
|
|
||||||
@@ -68,7 +85,8 @@ handoff:
|
|||||||
2. create and validate a safe run identity, then allocate a debug bundle only
|
2. create and validate a safe run identity, then allocate a debug bundle only
|
||||||
when requested;
|
when requested;
|
||||||
3. build the effective catalog, resolve requested reference changes, resolve
|
3. build the effective catalog, resolve requested reference changes, resolve
|
||||||
the effective pipeline, and verify explicit Scriptorium profiles;
|
the effective pipeline, and inspect its explicit effective PromptKit
|
||||||
|
profiles;
|
||||||
4. materialize external or generated references and record redacted invocation
|
4. materialize external or generated references and record redacted invocation
|
||||||
and resolution provenance when debug capture is enabled;
|
and resolution provenance when debug capture is enabled;
|
||||||
5. construct registries, the scheduled LLM client, prepared modules, and the
|
5. construct registries, the scheduled LLM client, prepared modules, and the
|
||||||
@@ -88,10 +106,12 @@ For `run --json`, the CLI constructs and encodes its private run-result receipt
|
|||||||
after a successful runner result is available, before it publishes logical
|
after a successful runner result is available, before it publishes logical
|
||||||
output files. It writes the prepared receipt to standard output only after
|
output files. It writes the prepared receipt to standard output only after
|
||||||
output publication and requested debug terminalization succeed. A receipt-write
|
output publication and requested debug terminalization succeed. A receipt-write
|
||||||
failure is a runtime command failure; output may be partial, the CLI reports a
|
failure exits with runtime status 1 and may leave partial standard-output bytes,
|
||||||
bounded command-owned error, and it does not repeat terminal reporting. The
|
but the already-published output bundle remains complete and requested debug
|
||||||
receipt remains a CLI reporting concern rather than a framework or output-module
|
reporting remains successfully terminalized. The CLI reports a bounded
|
||||||
responsibility; its public contract is the
|
command-owned error and does not repeat terminal reporting. The receipt remains
|
||||||
|
a CLI reporting concern rather than a framework or output-module responsibility;
|
||||||
|
its public contract is the
|
||||||
[run-result receipt](../integrations/run-result.md).
|
[run-result receipt](../integrations/run-result.md).
|
||||||
|
|
||||||
## Failure Mapping And Terminal Reporting
|
## Failure Mapping And Terminal Reporting
|
||||||
|
|||||||
@@ -39,14 +39,22 @@ This establishes the public precedence order without giving environment input a
|
|||||||
second file schema. Loading and application reject malformed YAML, unsupported
|
second file schema. Loading and application reject malformed YAML, unsupported
|
||||||
file versions, unknown fields, invalid values, and identifiers that are empty
|
file versions, unknown fields, invalid values, and identifiers that are empty
|
||||||
or collide after whitespace normalization. The file application also makes the
|
or collide after whitespace normalization. The file application also makes the
|
||||||
effective extraction-worker default follow the effective LLM limit.
|
effective extraction-worker default follow the effective LLM limit. A present
|
||||||
|
PromptKit local-backend object requires and trims its endpoint, defaults its
|
||||||
|
omitted concurrency limit to zero, and is copied so the parsed file model
|
||||||
|
cannot alias the populated **Config**. A pipeline `llm_profile` is
|
||||||
|
presence-aware: omission remains empty, while a present blank value is
|
||||||
|
rejected and a non-empty file value is trimmed before it reaches **Config**.
|
||||||
|
|
||||||
**Config.Validate** checks configuration-only invariants before resolution. It
|
**Config.Validate** checks configuration-only invariants before resolution. It
|
||||||
rejects incompatible profile sources, invalid state-surface values, unsupported
|
rejects incompatible profile sources, invalid state-surface values, unsupported
|
||||||
concurrency settings, malformed bindings and references, invalid retries, and
|
concurrency settings, malformed bindings and references, invalid retries, and
|
||||||
invalid pipeline, step, or lane structure. Its errors retain the closest known
|
invalid pipeline, step, or lane structure. PromptKit local-backend validation
|
||||||
pipeline, lane, and binding context. It deliberately does not require modules
|
accepts only an absolute HTTP or HTTPS endpoint with a host and no user
|
||||||
to be registered: that requires a catalog and belongs to resolution.
|
information, query, or fragment, and rejects a negative local concurrency
|
||||||
|
limit. Its errors retain the closest known pipeline, lane, and binding context.
|
||||||
|
It deliberately does not require modules to be registered: that requires a
|
||||||
|
catalog and belongs to resolution.
|
||||||
|
|
||||||
The exact user-selectable values and validation rules are defined in
|
The exact user-selectable values and validation rules are defined in
|
||||||
[Configuration](../config.md). Keep additions to the file model, an
|
[Configuration](../config.md). Keep additions to the file model, an
|
||||||
@@ -56,13 +64,15 @@ environment override, its validation, and that reference in the same change.
|
|||||||
|
|
||||||
**Config.Resolve** first recomputes derived concurrency defaults and validates
|
**Config.Resolve** first recomputes derived concurrency defaults and validates
|
||||||
the configuration. It normalizes the requested pipeline ID, copies the selected
|
the configuration. It normalizes the requested pipeline ID, copies the selected
|
||||||
profile, applies a non-empty command-level LLM profile override to the
|
profile, and passes the non-empty command-level LLM profile override, requested
|
||||||
LLM-capable stage bindings, and calls the framework resolver with the requested
|
lane selection, and reference changes to the framework resolver.
|
||||||
lane selection and reference changes.
|
|
||||||
|
|
||||||
The command-level override does not replace an explicitly selected validator
|
After module and validator selection, the resolver applies the effective
|
||||||
profile. Validator bindings remain part of the resolved validator chain and
|
profile policy to LLM-backed bindings only: command override, binding profile,
|
||||||
are resolved under their own declared configuration.
|
pipeline profile, then the prompt default. Deterministic bindings remain
|
||||||
|
profile-free, and no second inheritance decision occurs during execution. The
|
||||||
|
public field definitions and precedence are owned by
|
||||||
|
[Configuration](../config.md#pipelines).
|
||||||
|
|
||||||
The framework resolver supplies defaults, selects lanes, resolves validator
|
The framework resolver supplies defaults, selects lanes, resolves validator
|
||||||
chains, checks registered module and artifact compatibility, validates module
|
chains, checks registered module and artifact compatibility, validates module
|
||||||
@@ -71,7 +81,8 @@ options, and returns the fixed ordered pipeline shape. The resulting
|
|||||||
changes, a clone of the input configuration, and the resolved pipeline.
|
changes, a clone of the input configuration, and the resolved pipeline.
|
||||||
Callers may therefore retain or modify their input slices and maps without
|
Callers may therefore retain or modify their input slices and maps without
|
||||||
changing the resolved result, and later consumers cannot mutate the original
|
changing the resolved result, and later consumers cannot mutate the original
|
||||||
configuration through the effective value.
|
configuration through the effective value. This ownership includes the nested
|
||||||
|
PromptKit local-backend value.
|
||||||
|
|
||||||
Resolution failures stop before module construction and source parsing. They
|
Resolution failures stop before module construction and source parsing. They
|
||||||
include an error path for an unconfigured pipeline, missing module, missing
|
include an error path for an unconfigured pipeline, missing module, missing
|
||||||
@@ -83,7 +94,8 @@ runtime error class described in the [CLI reference](../cli.md#output-streams-an
|
|||||||
|
|
||||||
The framework assigns the resolved pipeline a deterministic SHA-256 digest
|
The framework assigns the resolved pipeline a deterministic SHA-256 digest
|
||||||
after defaults, lane selection, module bindings, reference bindings, validator
|
after defaults, lane selection, module bindings, reference bindings, validator
|
||||||
chains, and artifact schema identity have been resolved. The digest excludes
|
chains, effective LLM profiles, and artifact schema identity have been
|
||||||
|
resolved. The digest excludes
|
||||||
its own stored value. It identifies resolved composition rather than raw YAML
|
its own stored value. It identifies resolved composition rather than raw YAML
|
||||||
bytes, a debug payload, or all runtime state. The CLI records it as invocation
|
bytes, a debug payload, or all runtime state. The CLI records it as invocation
|
||||||
provenance before execution; cache and checkpoint identity have additional
|
provenance before execution; cache and checkpoint identity have additional
|
||||||
@@ -94,9 +106,12 @@ Configuration summaries must use **Redacted**, **RedactedSummaryPayload**, or
|
|||||||
Those methods copy every binding and nested option container, replace values
|
Those methods copy every binding and nested option container, replace values
|
||||||
whose key is credential-shaped with **[REDACTED]**, and omit materialized
|
whose key is credential-shaped with **[REDACTED]**, and omit materialized
|
||||||
reference content while retaining safe binding and reference provenance. The
|
reference content while retaining safe binding and reference provenance. The
|
||||||
payload must not alias the source configuration or resolved pipeline. This
|
payload must not alias the source configuration or resolved pipeline.
|
||||||
redaction is deliberately narrow: it protects configuration summaries and does
|
PromptKit's local endpoint and concurrency limit are preserved as non-secret
|
||||||
not authorize recording arbitrary environment values or provider requests.
|
configuration metadata in the independently owned summary; the object contains
|
||||||
|
no credential value. This redaction is deliberately narrow: it protects
|
||||||
|
configuration summaries and does not authorize recording arbitrary environment
|
||||||
|
values or provider requests.
|
||||||
|
|
||||||
## Invariants To Preserve
|
## Invariants To Preserve
|
||||||
|
|
||||||
|
|||||||
@@ -22,10 +22,15 @@ does not repeat their JSON shapes or schemas.
|
|||||||
## Family Composition
|
## Family Composition
|
||||||
|
|
||||||
The D&D registrar registers the family’s artifact codecs, extractors, typed
|
The D&D registrar registers the family’s artifact codecs, extractors, typed
|
||||||
append-order mergers, normalizers, validators, prompt assets, and default
|
append-order mergers, normalizers, validators, prompt assets, fallback LLM
|
||||||
validator chains. Each extractor and normalizer has a stable module spec,
|
profile asset, and default validator chains. Each extractor and normalizer has
|
||||||
strict option decoding, and a typed builder. Configuration remains the
|
a stable module spec, explicit execution class, strict option decoding, and a
|
||||||
canonical owner of the exact keys and validator order.
|
typed builder. Scene chunking, every extractor, and NPC normalization are
|
||||||
|
registered as `llm_backed`; the remaining current D&D mergers and normalizers
|
||||||
|
are `deterministic`. The metadata is available to catalog inspection and
|
||||||
|
resolved-pipeline debug data and determines which selected bindings inherit the
|
||||||
|
pipeline profile. Configuration remains the canonical owner of the exact keys,
|
||||||
|
profile precedence, and validator order.
|
||||||
|
|
||||||
Private structured-LLM response schemas are deliberately minimal. They reject
|
Private structured-LLM response schemas are deliberately minimal. They reject
|
||||||
invalid JSON structure, missing required fields, incompatible types, and
|
invalid JSON structure, missing required fields, incompatible types, and
|
||||||
@@ -39,9 +44,40 @@ D&D extractors assemble prompts from an ordered manifest of shared and
|
|||||||
module-owned assets. Reuse the shared D&D system, evidence, identity,
|
module-owned assets. Reuse the shared D&D system, evidence, identity,
|
||||||
reference, and transcript assets instead of copying their text into individual
|
reference, and transcript assets instead of copying their text into individual
|
||||||
modules. A manifest’s declared sequence, including cache-control placement, is
|
modules. A manifest’s declared sequence, including cache-control placement, is
|
||||||
part of the prompt behavior, and the chunk transcript is the final message.
|
part of the prompt behavior.
|
||||||
Preserve that order when changing an extractor or its assets so prompt-cache
|
|
||||||
behavior remains stable.
|
Every maintained D&D LLM prompt selects `dnd-extraction` as its default
|
||||||
|
profile. The D&D registrar embeds that fallback profile with the maintained
|
||||||
|
OpenRouter model, timeout, and service-tier policy. An operator may provide a
|
||||||
|
complete profile with the same ID through the configured PromptKit source; that
|
||||||
|
definition replaces the fallback rather than merging with it. The fallback
|
||||||
|
leaves reasoning and optional sampling controls unspecified. Deployment profile
|
||||||
|
selection and the maintained operator example are documented in
|
||||||
|
[Configuration](../config.md#promptkit-profiles).
|
||||||
|
|
||||||
|
All extraction prompts share this four-message rendered prefix: the system
|
||||||
|
message without cache control, the identity message without cache control, the
|
||||||
|
campaign-reference message with ephemeral cache control, and the chunk
|
||||||
|
transcript message with ephemeral cache control. This gives equivalent
|
||||||
|
extraction requests the same reusable prefix through their source material.
|
||||||
|
|
||||||
|
Extraction-evidence policy, generated NPC registries, spell catalogs, module
|
||||||
|
tasks, and instructions follow the transcript because they are not universal
|
||||||
|
across all extraction lanes. The final instructions message carries ephemeral
|
||||||
|
cache control; evidence, registry, catalog, and task messages do not. Preserve
|
||||||
|
this division when changing an extractor or its assets so prompt-cache behavior
|
||||||
|
remains stable.
|
||||||
|
|
||||||
|
The other D&D LLM prompts intentionally follow different patterns. Scene
|
||||||
|
chunking has no sibling extraction lane with which to share its full transcript,
|
||||||
|
so it renders campaign references before its task and instructions, then places
|
||||||
|
the cacheable full transcript last. NPC normalization keeps its task and
|
||||||
|
cacheable instructions before the candidate collection, followed by the
|
||||||
|
cacheable transcript windows: candidates must be available before their
|
||||||
|
supporting evidence is evaluated, and those windows are not a cross-lane
|
||||||
|
prefix. Mounted assets and their declared message order determine the prompt
|
||||||
|
fingerprint, so intentional prompt edits continue to invalidate stale
|
||||||
|
checkpoints.
|
||||||
|
|
||||||
All extractors use the shared prompt-input preparation rules. The current chunk
|
All extractors use the shared prompt-input preparation rules. The current chunk
|
||||||
is copied into transcript material; player, party, glossary, and compatible
|
is copied into transcript material; player, party, glossary, and compatible
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
# LLM Runtime Internals
|
# LLM Runtime Internals
|
||||||
|
|
||||||
`internal/framework/llm` is Notarius’s provider-independent structured
|
`internal/framework/llm` is Notarius’s provider-independent structured
|
||||||
completion boundary. It adapts framework requests to Scriptorium, bounds
|
completion boundary. It adapts framework requests to PromptKit, bounds
|
||||||
provider calls, assembles registered prompt and schema assets, records selected
|
provider calls, assembles registered prompt and schema assets, records selected
|
||||||
profiles, and redacts provider errors. The architectural boundary is defined in
|
profiles, and redacts provider errors. The architectural boundary is defined in
|
||||||
[Architecture](../policy/architecture.md#llm-boundary); profile sources,
|
[Architecture](../policy/architecture.md#llm-boundary); profile sources,
|
||||||
credentials, and concurrency settings belong in
|
credentials, and concurrency settings belong in
|
||||||
[Configuration](../config.md#scriptorium-profiles) and
|
[Configuration](../config.md#promptkit-profiles) and
|
||||||
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||||||
|
|
||||||
## Structured Completion Boundary
|
## Structured Completion Boundary
|
||||||
@@ -24,22 +24,78 @@ adapter does not own source evidence, artifact conversion, normalization, or
|
|||||||
durable schemas. Those responsibilities remain with the module and its
|
durable schemas. Those responsibilities remain with the module and its
|
||||||
[integration contract](../integrations/).
|
[integration contract](../integrations/).
|
||||||
|
|
||||||
`ScriptoriumClient` validates the request target and prompt identity, maps each
|
`PromptKitClient` validates the request target and prompt identity, maps each
|
||||||
named material to a Scriptorium inline artifact while preserving its origin URI,
|
named material to a PromptKit inline artifact while preserving its origin URI,
|
||||||
forwards session and profile selection, then prepares and runs the prompt. It
|
maps the trimmed request session to PromptKit's direct per-run session field,
|
||||||
returns Scriptorium’s validated raw bytes rather than re-encoding the decoded
|
retains the same value as the `session_id` prompt variable for maintained
|
||||||
target. An empty optional material is represented as one space so its named
|
prompt compatibility, and forwards profile selection. It then creates one
|
||||||
input is retained by Scriptorium.
|
frozen prepared execution, captures its caller-owned credential-redacted
|
||||||
|
details for debug material, and executes that exact snapshot through
|
||||||
|
PromptKit's prepared-execution boundary. The direct field
|
||||||
|
is authoritative for provider session behavior. A session ID is a stable,
|
||||||
|
non-secret correlation identifier and may be exposed to providers and provider
|
||||||
|
observability. The adapter returns PromptKit’s validated raw bytes rather than
|
||||||
|
re-encoding the decoded target. An empty optional material is represented as
|
||||||
|
one space so its named input is retained by PromptKit.
|
||||||
|
|
||||||
An empty request profile lets the prompt select its configured default. The CLI
|
Client construction may also receive a run-wide reasoning-effort override from
|
||||||
prepares every explicitly selected binding profile before a run begins, so a
|
the CLI factory boundary. The adapter copies the caller-owned pointer and
|
||||||
missing explicit profile fails before stage execution. Calls record the profile
|
creates a fresh PromptKit execution override for each request: a nil pointer
|
||||||
actually selected by Scriptorium; the recorder deduplicates non-secret profile
|
inherits the selected profile, a non-empty value replaces it, and an empty
|
||||||
identity, provider, and model values for manifest use.
|
value clears inherited reasoning. The CLI's mutually exclusive
|
||||||
|
`--reasoning-effort` and `--clear-reasoning-effort` controls select those
|
||||||
|
states. With neither flag, profile behavior remains unchanged. Because
|
||||||
|
production constructs one shared client, the selected state applies uniformly
|
||||||
|
to module calls, retries, and LLM-backed validators for the whole run.
|
||||||
|
|
||||||
|
An empty request profile lets the prompt select its configured default. Before a
|
||||||
|
run begins, the CLI asks the adapter to inspect every explicit profile on the
|
||||||
|
resolved selected LLM-backed bindings and validators, including inherited
|
||||||
|
pipeline profiles. Inspection resolves the profile and its selected backend and
|
||||||
|
target without loading a prompt, reading credentials, admitting capacity, or
|
||||||
|
contacting a provider, so a missing or invalid explicit profile fails before
|
||||||
|
stage execution while a valid `api_key_env` may remain unset. Calls record the
|
||||||
|
profile actually selected by PromptKit. The recorder trims and deduplicates
|
||||||
|
non-secret profile identity, provider, model, selected backend ID, and
|
||||||
|
effective reasoning values for manifest use. Entries that differ in backend or
|
||||||
|
reasoning remain distinct and deterministically ordered. Endpoint-only profiles
|
||||||
|
retain an empty backend ID, which the published JSON omits. Successful
|
||||||
|
completion responses and recorded profile manifests identify the adapter
|
||||||
|
provider as `promptkit`.
|
||||||
|
|
||||||
|
The CLI's profile-inspection engine and the production adapter use the same
|
||||||
|
profile-source construction to apply the configured profile directory or file,
|
||||||
|
the optional registered fallback profile assets, and the optional conventional
|
||||||
|
`local` backend. Preflight therefore resolves the same profile sources and
|
||||||
|
backend membership as runtime without performing generation. Fallback assets
|
||||||
|
are mounted only when at least one source is registered. The production D&D
|
||||||
|
registrar contributes its `dnd-extraction` fallback, and the maintained D&D
|
||||||
|
prompts select that logical ID by default. PromptKit owns source precedence and
|
||||||
|
profile parsing: an operator-provided matching profile takes precedence over a
|
||||||
|
fallback profile without Notarius merging either document.
|
||||||
|
When the registration is absent, a profile selecting `backend: local` fails
|
||||||
|
inspection instead of falling back to a built-in or endpoint-only target.
|
||||||
|
|
||||||
|
Before execution, the adapter also contributes a non-secret checkpoint
|
||||||
|
fingerprint for the effective PromptKit profile source. It combines the
|
||||||
|
identity of PromptKit's compiled-in profile catalog with a deterministic digest
|
||||||
|
of every YAML profile in the configured profile directory, or of the configured
|
||||||
|
profile file, and a deterministic digest of the flattened fallback profile
|
||||||
|
assets. The fingerprint contains neither profile content nor source paths. It
|
||||||
|
covers inherited pipeline profiles, explicit binding profiles, and
|
||||||
|
prompt-selected defaults, so changing a model or other profile setting cannot
|
||||||
|
reuse checkpoints created under the
|
||||||
|
prior profile source. This cache identity is independent of durable
|
||||||
|
profile provenance: run manifests continue to list only profiles actually
|
||||||
|
observed during LLM calls. When the local backend is registered, a second
|
||||||
|
fingerprint hashes its trimmed endpoint behind a stable marker. Changing that
|
||||||
|
semantic execution target invalidates checkpoint reuse. The raw endpoint is not
|
||||||
|
stored in checkpoint identity, and the local concurrency limit is excluded
|
||||||
|
because it changes scheduling rather than execution semantics.
|
||||||
|
|
||||||
## Shared Provider-Call Limit
|
## Shared Provider-Call Limit
|
||||||
|
|
||||||
Production construction creates one Scriptorium client and wraps it in one
|
Production construction creates one PromptKit client and wraps it in one
|
||||||
scheduled client. The scheduler has a fixed, positive permit limit, serves
|
scheduled client. The scheduler has a fixed, positive permit limit, serves
|
||||||
queued calls in FIFO order, and removes a queued call when its context is
|
queued calls in FIFO order, and removes a queued call when its context is
|
||||||
cancelled. A granted permit is released exactly once on every completion path.
|
cancelled. A granted permit is released exactly once on every completion path.
|
||||||
@@ -51,12 +107,23 @@ worker counts cannot exceed the configured LLM limit. The configuration field
|
|||||||
and its effective default are owned by
|
and its effective default are owned by
|
||||||
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||||||
|
|
||||||
|
PromptKit applies a second, independent admission limit when the selected
|
||||||
|
profile names a limited backend. It sits beneath the Notarius scheduled client,
|
||||||
|
so it may narrow but cannot expand the application-wide limit. Built-in
|
||||||
|
OpenRouter profiles select PromptKit's reserved backend and its upstream
|
||||||
|
capacity policy. A positive configured local-backend limit bounds active local
|
||||||
|
generations inside PromptKit; zero leaves that backend unlimited there.
|
||||||
|
Endpoint-only profiles do not select a PromptKit backend and remain limited
|
||||||
|
only by the Notarius scheduler.
|
||||||
|
|
||||||
## Prompt And Schema Assets
|
## Prompt And Schema Assets
|
||||||
|
|
||||||
An `AssetRegistry` collects prompt and schema filesystems from production module
|
An `AssetRegistry` collects prompt, schema, and optional fallback-profile
|
||||||
families. It flattens registered roots into the Scriptorium filesystems and
|
filesystems from production module families. It flattens registered roots into
|
||||||
rejects invalid roots, unreadable assets, duplicate paths, and missing prompt
|
the corresponding PromptKit filesystems and rejects invalid roots, unreadable
|
||||||
or schema files during preparation. The framework’s `promptfs` helper combines
|
assets, duplicate paths, and missing prompt or schema files during preparation.
|
||||||
|
Fallback assets receive a safe content digest for checkpoint identity; raw
|
||||||
|
paths and bytes are never included. The framework’s `promptfs` helper combines
|
||||||
module-owned prompt files with reusable domain fragments without making the
|
module-owned prompt files with reusable domain fragments without making the
|
||||||
framework depend on D&D content.
|
framework depend on D&D content.
|
||||||
|
|
||||||
@@ -76,30 +143,43 @@ contract. Durable formats and compatibility rules remain in the
|
|||||||
## Prompt Maintenance And Backend Caching
|
## Prompt Maintenance And Backend Caching
|
||||||
|
|
||||||
Prompt message order and shared asset bytes are runtime behavior. Backend cache
|
Prompt message order and shared asset bytes are runtime behavior. Backend cache
|
||||||
reuse depends on the same preceding messages and content, not merely equivalent
|
reuse depends on identical preceding roles, rendered bytes, and cache-control
|
||||||
meaning. Keep reusable shared assets byte-identical and keep stable material
|
metadata—not merely equivalent meaning. Keep reusable shared assets
|
||||||
before the inputs that vary per request wherever a prompt’s declared sequence
|
byte-identical and preserve each prompt’s declared ordering and cache controls
|
||||||
supports caching. Preserve the existing manifest order and cache-control hints
|
when editing it.
|
||||||
when editing a prompt.
|
|
||||||
|
|
||||||
D&D extraction manifests place the changing chunk transcript at the end of the
|
For sibling prompts that can reuse the same source material, order universal
|
||||||
prompt after their reusable context. Scene chunking and NPC normalization use
|
shared context first, request source material next, and module-specific
|
||||||
their own declared message sequences because their inputs and work differ. The
|
suffixes last. Put a cache boundary at a reusable prefix that is useful to the
|
||||||
family-specific asset and ordering rules belong in [D&D Module Internals](dnd.md).
|
backend. Redundant intermediate cache boundaries do not extend that reusable
|
||||||
Do not add tests that enforce a fixed message-prefix length; prompt-asset tests
|
prefix and add no value.
|
||||||
should instead verify the meaningful asset sequence, inputs, and cache controls
|
|
||||||
of the prompt being changed.
|
Prompt-family owners may choose a different sequence when their inputs and
|
||||||
|
reuse pattern differ. The D&D family’s extraction, scene-chunking, and NPC
|
||||||
|
normalization policies are maintained in [D&D Module Internals](dnd.md#prompt-construction).
|
||||||
|
Do not add tests that enforce prompt prose; prompt tests should verify the
|
||||||
|
meaningful input placement and cache controls of the prompt being changed.
|
||||||
|
|
||||||
## Validation, Repair, And Retries
|
## Validation, Repair, And Retries
|
||||||
|
|
||||||
Scriptorium performs prompt rendering, provider execution, and the prompt’s
|
PromptKit performs prompt rendering, provider execution, and the prompt’s
|
||||||
structured-output validation. The adapter reports an empty result, validation
|
structured-output validation. The adapter reports an empty result, validation
|
||||||
failure, empty structured body, or decode failure as
|
failure, empty structured body, or decode failure as
|
||||||
`ErrInvalidStructuredOutput`, while retaining the returned raw bytes and debug
|
`ErrInvalidStructuredOutput`, while retaining the returned raw bytes and debug
|
||||||
material when they exist. Provider failures remain operational errors rather
|
material when they exist. Provider failures remain operational errors rather
|
||||||
than output-validation failures.
|
than output-validation failures.
|
||||||
|
|
||||||
Prompt-declared repair is executed within Scriptorium’s structured-output flow.
|
When PromptKit rejects backend admission before generation, the adapter maps
|
||||||
|
`promptkit.ErrCapacityExceeded` to
|
||||||
|
`contracts.ErrLLMCapacityExceeded`, retaining prompt context and a redacted
|
||||||
|
upstream diagnostic without exposing the PromptKit sentinel or capacity-error
|
||||||
|
type as a framework contract. When supplied, the normalized selected backend
|
||||||
|
ID appears only in that safe application-owned diagnostic context. A canceled
|
||||||
|
caller context takes precedence. The adapter does not retry capacity failures;
|
||||||
|
the pipeline's existing binding attempt policy sees the operational error and
|
||||||
|
decides whether to rerun the complete operation.
|
||||||
|
|
||||||
|
Prompt-declared repair is executed within PromptKit’s structured-output flow.
|
||||||
The current production D&D prompt manifests set repair attempts to zero. That
|
The current production D&D prompt manifests set repair attempts to zero. That
|
||||||
setting does not replace pipeline retry behavior: a binding’s configured retry
|
setting does not replace pipeline retry behavior: a binding’s configured retry
|
||||||
count reruns its stage attempt after an error or rejection, and an exhausted
|
count reruns its stage attempt after an error or rejection, and an exhausted
|
||||||
@@ -108,22 +188,42 @@ attempt lifecycle, validation chains, and retry diagnostics; see
|
|||||||
[Pipeline Internals](pipeline.md#validation-retries-and-output) and the
|
[Pipeline Internals](pipeline.md#validation-retries-and-output) and the
|
||||||
[binding reference](../config.md#module-bindings-and-validators).
|
[binding reference](../config.md#module-bindings-and-validators).
|
||||||
|
|
||||||
|
## Timeout Ownership
|
||||||
|
|
||||||
|
The caller context remains the outer cancellation authority. PromptKit applies
|
||||||
|
a positive effective generation timeout as an inner request deadline; an
|
||||||
|
explicit zero disables only that generation deadline. The HTTP client timeout
|
||||||
|
is a separate transport-wide cap. Notarius forwards the caller context and
|
||||||
|
does not install another timeout wrapper around PromptKit.
|
||||||
|
|
||||||
|
The selected PromptKit profile owns generation settings. Notarius binding
|
||||||
|
retries remain outside the adapter and repeat the complete module operation
|
||||||
|
and validation chain. PromptKit does not add a provider retry loop.
|
||||||
|
Operator-facing behavior is summarized in
|
||||||
|
[Operations](../operations.md#operational-limits), and the pinned upstream
|
||||||
|
contract is identified in
|
||||||
|
[PromptKit Integration](../integrations/pkg-promptkit.md).
|
||||||
|
|
||||||
## Observability And Redaction
|
## Observability And Redaction
|
||||||
|
|
||||||
When debug recording is enabled, the pipeline decorates the shared client. The
|
When debug recording is enabled, the pipeline decorates the shared client. The
|
||||||
wrapper records prepared prompt and response material, timing, selected profile
|
wrapper records prepared prompt and response material, timing, selected profile
|
||||||
and model, and call identifiers in the run’s debug bundle, including material
|
and backend, effective model parameters, and call identifiers in the run’s
|
||||||
available from a failed structured completion. For a successful completion, a
|
debug bundle, including material available from a failed structured completion.
|
||||||
debug-write failure is surfaced; when the completion already failed, its call
|
Effective parameters use PromptKit's stable lower-case JSON field names and may
|
||||||
error remains the result. Debug-bundle location, retention, and handling are
|
include `backend_id`. For a successful completion, a debug-write failure is
|
||||||
operational concerns documented in [Operations](../operations.md#debug-bundles).
|
surfaced; when the completion already failed, its call error remains the
|
||||||
|
result. Debug-bundle location, retention, and handling are operational concerns
|
||||||
|
documented in [Operations](../operations.md#debug-bundles).
|
||||||
|
|
||||||
Run manifests receive selected profile summaries and component identities, not
|
Run manifests receive selected profile summaries, including optional effective
|
||||||
prompt, schema, source, reference, or response content. Provider error text is
|
backend and reasoning provenance, and component identities—not prompt, schema,
|
||||||
wrapped with prompt context and bearer credentials are redacted before it
|
source, reference, or response content. The published field semantics belong
|
||||||
crosses the runtime boundary. Known-secret redaction is available to other
|
to the [JSON output contract](../integrations/json-output.md#manifestjson).
|
||||||
runtime collaborators; it does not make prompt or response contents safe for
|
Provider error text is wrapped with prompt context and bearer credentials are
|
||||||
general logging.
|
redacted before it crosses the runtime boundary. Known-secret redaction is
|
||||||
|
available to other runtime collaborators; it does not make prompt or response
|
||||||
|
contents safe for general logging.
|
||||||
|
|
||||||
## Failure Boundaries
|
## Failure Boundaries
|
||||||
|
|
||||||
@@ -131,6 +231,8 @@ general logging.
|
|||||||
sources, invalid asset registration, or a non-positive scheduler limit.
|
sources, invalid asset registration, or a non-positive scheduler limit.
|
||||||
- Preparation failures, unavailable explicit profiles, provider failures, and
|
- Preparation failures, unavailable explicit profiles, provider failures, and
|
||||||
context cancellation propagate to the calling stage with context.
|
context cancellation propagate to the calling stage with context.
|
||||||
|
- Backend admission exhaustion is a provider-neutral operational error and is
|
||||||
|
not classified as invalid structured output or validator rejection.
|
||||||
- Malformed or schema-invalid provider output is classified separately as
|
- Malformed or schema-invalid provider output is classified separately as
|
||||||
invalid structured output so the module or pipeline can apply its own retry
|
invalid structured output so the module or pipeline can apply its own retry
|
||||||
and rejection policy.
|
and rejection policy.
|
||||||
|
|||||||
@@ -12,9 +12,15 @@ exceptions. See [D&D Module Internals](dnd.md) rather than adding them here.
|
|||||||
|
|
||||||
A module is a typed implementation registered for one pipeline stage. Its
|
A module is a typed implementation registered for one pipeline stage. Its
|
||||||
`ModuleSpec` is the public-to-the-framework declaration of its stable key,
|
`ModuleSpec` is the public-to-the-framework declaration of its stable key,
|
||||||
stage, required and provided capabilities, artifact kind, and accepted
|
stage, execution class, required and provided capabilities, artifact kind, and
|
||||||
reference slots. The framework uses that declaration to resolve a configured
|
accepted reference slots. The execution class states whether a module is
|
||||||
binding before it builds the implementation.
|
`deterministic` or `llm_backed`; registries retain it for catalog inspection and
|
||||||
|
resolved-pipeline debug data without constructing the module. The framework
|
||||||
|
uses the declaration to resolve a configured binding before it builds the
|
||||||
|
implementation. After selection, the resolver applies profile inheritance only
|
||||||
|
to bindings whose declared execution class is `llm_backed` and rejects a
|
||||||
|
binding-specific profile on a deterministic module. The user-facing precedence
|
||||||
|
contract belongs in [Configuration](../config.md#pipelines).
|
||||||
|
|
||||||
Implementations that accept options must provide both an option validator and
|
Implementations that accept options must provide both an option validator and
|
||||||
a builder. The validator is used while resolving configuration; the builder
|
a builder. The validator is used while resolving configuration; the builder
|
||||||
@@ -30,6 +36,14 @@ they need, register each leaf implementation, and add any family-owned assets
|
|||||||
or default validator chains. They return contextual errors so production
|
or default validator chains. They return contextual errors so production
|
||||||
composition fails at startup rather than at the first run.
|
composition fails at startup rather than at the first run.
|
||||||
|
|
||||||
|
An artifact family can register an optional typed evidence projector alongside
|
||||||
|
its codec. The projector returns defensive copies of the artifact's direct
|
||||||
|
generic source references and must use the codec's exact Go type. It does not
|
||||||
|
interpret surrounding context or publish files; the pipeline validates the
|
||||||
|
capability during preparation and the output boundary owns publication. See
|
||||||
|
the [Published Evidence Context contract](../integrations/evidence-context.md)
|
||||||
|
for the durable result.
|
||||||
|
|
||||||
## Production Composition
|
## Production Composition
|
||||||
|
|
||||||
Production composition is intentionally split by family:
|
Production composition is intentionally split by family:
|
||||||
@@ -39,7 +53,7 @@ Production composition is intentionally split by family:
|
|||||||
- The Seriatim registrar provides the transcript input adapter. Its external
|
- The Seriatim registrar provides the transcript input adapter. Its external
|
||||||
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
|
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
|
||||||
- The D&D registrar provides its codecs, extractors, mergers, normalizers,
|
- The D&D registrar provides its codecs, extractors, mergers, normalizers,
|
||||||
validators, prompt assets, and default chains. Its behavioral conventions
|
validators, prompt assets, fallback profile asset, and default chains. Its behavioral conventions
|
||||||
are documented in [D&D Module Internals](dnd.md).
|
are documented in [D&D Module Internals](dnd.md).
|
||||||
|
|
||||||
The CLI owns the composition that invokes these registrars. A module package
|
The CLI owns the composition that invokes these registrars. A module package
|
||||||
@@ -51,10 +65,10 @@ packages depend on production extensions.
|
|||||||
1. Choose the pipeline stage and the typed artifact boundary. Put external
|
1. Choose the pipeline stage and the typed artifact boundary. Put external
|
||||||
input or durable artifact formats in the relevant integration contract,
|
input or durable artifact formats in the relevant integration contract,
|
||||||
not in this guide or in a private LLM response type.
|
not in this guide or in a private LLM response type.
|
||||||
2. Define a stable `ModuleSpec` with the exact capabilities and reference
|
2. Define a stable `ModuleSpec` with an explicit execution class, the exact
|
||||||
slots needed for the operation. Model a producer/consumer handoff as an
|
capabilities, and reference slots needed for the operation. Model a
|
||||||
artifact-compatible slot; configuration then chooses an external file or a
|
producer/consumer handoff as an artifact-compatible slot; configuration
|
||||||
generated binding.
|
then chooses an external file or a generated binding.
|
||||||
3. Implement strict option decoding, construction, and the typed stage
|
3. Implement strict option decoding, construction, and the typed stage
|
||||||
interface. Preserve caller ownership: do not retain mutable request data
|
interface. Preserve caller ownership: do not retain mutable request data
|
||||||
and return defensive copies where an implementation exposes stored data.
|
and return defensive copies where an implementation exposes stored data.
|
||||||
|
|||||||
@@ -34,6 +34,11 @@ requested lanes where that is supported, resolves validator chains, checks
|
|||||||
module capabilities and typed artifact compatibility, validates options, and
|
module capabilities and typed artifact compatibility, validates options, and
|
||||||
assigns a deterministic resolved-composition digest. The resolved pipeline
|
assigns a deterministic resolved-composition digest. The resolved pipeline
|
||||||
contains bindings and declared reference targets, not external reference bytes.
|
contains bindings and declared reference targets, not external reference bytes.
|
||||||
|
After selection, the resolver applies command, binding, and pipeline profile
|
||||||
|
precedence to LLM-backed bindings and validators only; prompt defaults remain
|
||||||
|
an empty resolved binding profile. Deterministic bindings remain profile-free.
|
||||||
|
These effective values are part of the digest, so execution and checkpoint
|
||||||
|
consumers do not repeat profile inheritance.
|
||||||
Configuration resolution supplies the selected profile and catalog; see
|
Configuration resolution supplies the selected profile and catalog; see
|
||||||
[Configuration Internals](configuration.md).
|
[Configuration Internals](configuration.md).
|
||||||
|
|
||||||
@@ -52,6 +57,13 @@ checkpoint fingerprints. Missing registrations, incompatible typed entries,
|
|||||||
nil implementations, and constructor failures are reported before source
|
nil implementations, and constructor failures are reported before source
|
||||||
parsing or any stage operation begins.
|
parsing or any stage operation begins.
|
||||||
|
|
||||||
|
An output encoder can opt into source-evidence publication through its output
|
||||||
|
policy. Preparation keeps the configured lane allowlist and active lanes
|
||||||
|
separate, then verifies an exact typed evidence projector and registered codec
|
||||||
|
for each active lane. The resulting private plan is immutable; lanes excluded
|
||||||
|
by invocation filtering remain configured but do not acquire a projector for
|
||||||
|
that run.
|
||||||
|
|
||||||
## Typed Lanes And References
|
## Typed Lanes And References
|
||||||
|
|
||||||
Each resolved lane has one artifact kind, codec, and exact Go type. The
|
Each resolved lane has one artifact kind, codec, and exact Go type. The
|
||||||
@@ -110,11 +122,16 @@ directives consume this same budget and validate any final safe fallback through
|
|||||||
the normalizer chain.
|
the normalizer chain.
|
||||||
|
|
||||||
After terminal lane work, the runner assembles manifest provenance, normalized
|
After terminal lane work, the runner assembles manifest provenance, normalized
|
||||||
artifacts, rejections, warnings, and an optional accepted chunk map. The output
|
artifacts, rejections, warnings, and an optional accepted chunk map. When an
|
||||||
encoder returns logical files; it does not choose a physical directory. The CLI
|
output policy selected evidence lanes, it decodes accepted serialized normalize
|
||||||
publishes those files only after the runner returns without a framework error.
|
outputs through their registered codecs and invokes the prepared typed
|
||||||
Logical file names and schemas are defined by the
|
projectors. Rejected or absent lanes contribute nothing. This reconstruction is
|
||||||
[output integration contracts](../integrations/).
|
also used after normalized-checkpoint reuse, so no second typed output channel
|
||||||
|
is retained. The runner passes the resulting owned artifact to the output
|
||||||
|
encoder, which returns logical files and does not choose a physical directory.
|
||||||
|
The CLI publishes those files only after the runner returns without a framework
|
||||||
|
error. Logical file names and schemas are defined by the [output integration
|
||||||
|
contracts](../integrations/).
|
||||||
|
|
||||||
## Checkpoint And Debug Hooks
|
## Checkpoint And Debug Hooks
|
||||||
|
|
||||||
|
|||||||
@@ -39,7 +39,9 @@ codecs, loader, and recorder. The CLI constructs a recorder whenever checkpoint
|
|||||||
recording is enabled and constructs a loader only for a `--resume` invocation.
|
recording is enabled and constructs a loader only for a `--resume` invocation.
|
||||||
Identity incorporates explicit stable semantic fingerprints collected from
|
Identity incorporates explicit stable semantic fingerprints collected from
|
||||||
prepared modules and validators in addition to configuration, input,
|
prepared modules and validators in addition to configuration, input,
|
||||||
references, runtime overrides, and LLM profiles.
|
references, runtime overrides, observed LLM profiles, and the LLM runtime's
|
||||||
|
non-secret effective profile-source identity. A profile source change therefore
|
||||||
|
causes a cold miss even when the configured profile ID remains unchanged.
|
||||||
The serialized
|
The serialized
|
||||||
`workspace_schema_version` identifiers are frozen wire-compatibility fields;
|
`workspace_schema_version` identifiers are frozen wire-compatibility fields;
|
||||||
they do not describe a current public state surface.
|
they do not describe a current public state surface.
|
||||||
|
|||||||
@@ -36,7 +36,47 @@ On supported Unix systems, output directories and files are created with
|
|||||||
requested modes **0755** and **0644**. Chunk-plan, checkpoint, and debug
|
requested modes **0755** and **0644**. Chunk-plan, checkpoint, and debug
|
||||||
directories and files use **0700** and **0600**. The operating system's umask
|
directories and files use **0700** and **0600**. The operating system's umask
|
||||||
may impose stricter output modes. Cache and debug roots may contain sensitive
|
may impose stricter output modes. Cache and debug roots may contain sensitive
|
||||||
source-derived data, so provision them for one trusted account or service.
|
source-derived data, so provision them for one trusted account or service. An
|
||||||
|
output bundle can also contain source content when its JSON output enables
|
||||||
|
evidence publication. Apply an appropriate umask and output-root access policy
|
||||||
|
before enabling that option; the requested output modes alone may not be
|
||||||
|
suitable for transcript-bearing bundles.
|
||||||
|
|
||||||
|
## PromptKit Profile Deployment
|
||||||
|
|
||||||
|
Profile deployment has four distinct layers:
|
||||||
|
|
||||||
|
| Layer | Owner | Operational role |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Prompts and schemas | Notarius module families | Embedded request and structured-output definitions. They are not deployment profile files. |
|
||||||
|
| Fallback profiles | Notarius module families | Embedded application defaults, including D&D's `dnd-extraction` profile. |
|
||||||
|
| Built-in profiles | PromptKit | Upstream catalog entries available when no higher-precedence source defines an ID. |
|
||||||
|
| Operator profiles | Deployment filesystem | Complete environment-specific definitions selected by `promptkit.profile_file` or `promptkit.profile_dir`. |
|
||||||
|
|
||||||
|
The maintained D&D pipeline uses the workload ID `dnd-extraction`. The
|
||||||
|
embedded fallback makes that ID usable without an operator file. Production,
|
||||||
|
development, and local deployments can each install a different complete
|
||||||
|
definition for the same ID, retaining the pipeline while choosing their own
|
||||||
|
model, backend, timeout, or reasoning policy. An operator definition wins over
|
||||||
|
the fallback; it is not merged with it. The configuration field and full
|
||||||
|
precedence rules are owned by [Configuration](config.md#promptkit-profiles).
|
||||||
|
|
||||||
|
Use a profile source owned by the service account, keep it readable only by
|
||||||
|
the intended operator, and supply provider credentials through the service
|
||||||
|
environment—not in the Notarius configuration or profile YAML. The maintained
|
||||||
|
[operator profile](../examples/profiles/dnd-extraction.yml) is secret-free and
|
||||||
|
can be copied as a format starting point. Validate a deployment without a
|
||||||
|
provider call or credentials:
|
||||||
|
|
||||||
|
~~~sh
|
||||||
|
notarius config validate --config /etc/notarius/config.yml --pipeline dnd-session
|
||||||
|
~~~
|
||||||
|
|
||||||
|
Profile paths are currently resolved from the process working directory, not
|
||||||
|
from the configuration file. The complete example's
|
||||||
|
`./examples/profiles/dnd-extraction.yml` path is valid for a repository-root
|
||||||
|
invocation only. Use absolute paths such as
|
||||||
|
`/etc/notarius/profiles/dnd-extraction.yml` for services and containers.
|
||||||
|
|
||||||
## Run Lifecycle
|
## Run Lifecycle
|
||||||
|
|
||||||
@@ -66,7 +106,13 @@ run directory remains for inspection and is never removed automatically.
|
|||||||
Treat an output bundle as durable user data. Do not use cache-cleanup policy to
|
Treat an output bundle as durable user data. Do not use cache-cleanup policy to
|
||||||
remove it. An optional accepted chunk map is also durable output and can carry
|
remove it. An optional accepted chunk map is also durable output and can carry
|
||||||
source- or model-derived annotations; its content and compatibility contract
|
source- or model-derived annotations; its content and compatibility contract
|
||||||
are defined in [Accepted Chunk Map](integrations/chunk-map.md).
|
are defined in [Accepted Chunk Map](integrations/chunk-map.md). An optional
|
||||||
|
[evidence context](integrations/evidence-context.md) contains source-unit text
|
||||||
|
and metadata. It is not a cache or debug artifact: retain it with the output
|
||||||
|
bundle only for as long as consumers need it, and apply source-content access
|
||||||
|
controls to the entire bundle. Selected lanes may collectively cite most of a
|
||||||
|
transcript, so a broad allowlist can make the evidence artifact nearly as
|
||||||
|
sensitive and large as the source itself.
|
||||||
|
|
||||||
## Chunk-Plan Cache
|
## Chunk-Plan Cache
|
||||||
|
|
||||||
@@ -107,9 +153,15 @@ compatible recorded work. A resume request fails when checkpoint recording is
|
|||||||
disabled. Without **--resume**, a recording-enabled run executes normally and
|
disabled. Without **--resume**, a recording-enabled run executes normally and
|
||||||
does not load checkpoint state. Compatibility includes the resolved pipeline,
|
does not load checkpoint state. Compatibility includes the resolved pipeline,
|
||||||
input, selected lanes, runtime overrides, reference provenance, LLM-profile
|
input, selected lanes, runtime overrides, reference provenance, LLM-profile
|
||||||
provenance, and prepared-component fingerprints. A changed identity produces a
|
provenance, the effective PromptKit profile-source fingerprint, and
|
||||||
cold miss; Notarius does not migrate, rewrite, or delete older checkpoint
|
prepared-component fingerprints. When a local PromptKit backend is configured,
|
||||||
directories.
|
compatibility also includes a non-secret fingerprint of its endpoint. Changing
|
||||||
|
profile content or the local endpoint causes a cold miss; changing only the
|
||||||
|
local concurrency limit does not. A changed identity produces a cold miss;
|
||||||
|
Notarius does not migrate, rewrite, or delete older checkpoint directories.
|
||||||
|
Reasoning-effort inheritance, replacement, and explicit clearing are distinct
|
||||||
|
runtime identities, so checkpoints created under one state are not reused by
|
||||||
|
either of the others.
|
||||||
|
|
||||||
Checkpoint state is confined below an identity-specific path:
|
Checkpoint state is confined below an identity-specific path:
|
||||||
|
|
||||||
@@ -169,7 +221,9 @@ warning, checkpoint, chunk-plan, and terminal reporting artifacts. The trace
|
|||||||
contains allowlisted application diagnostic records and can include source or
|
contains allowlisted application diagnostic records and can include source or
|
||||||
derived application data. Neither surface is a cache input. Do not treat a
|
derived application data. Neither surface is a cache input. Do not treat a
|
||||||
debug bundle as safe to share merely because its configuration summary is
|
debug bundle as safe to share merely because its configuration summary is
|
||||||
redacted.
|
redacted. Invocation metadata omits reasoning effort when it is inherited,
|
||||||
|
records the replacement value when one is supplied, and records an empty value
|
||||||
|
when inherited reasoning was explicitly cleared.
|
||||||
|
|
||||||
Notarius never creates debug state without an explicit request and never
|
Notarius never creates debug state without an explicit request and never
|
||||||
automatically deletes a requested bundle. If allocation succeeds, the command
|
automatically deletes a requested bundle. If allocation succeeds, the command
|
||||||
@@ -197,9 +251,40 @@ or automatic cleanup command.
|
|||||||
|
|
||||||
## Operational Limits
|
## Operational Limits
|
||||||
|
|
||||||
Provider retries and timeouts are supplied by the selected Scriptorium profile.
|
Provider execution settings and the generation timeout come from the selected
|
||||||
Module retry settings and concurrency limits are configuration contracts; see
|
PromptKit profile. The invocation-only **--reasoning-effort** and
|
||||||
[module bindings](config.md#module-bindings-and-validators) and
|
**--clear-reasoning-effort** controls may replace or clear that profile setting
|
||||||
|
for all LLM-backed calls in one run without changing the profile. PromptKit
|
||||||
|
v0.5.0 does not add a provider retry loop. Notarius binding retries rerun the
|
||||||
|
complete module operation and validation chain as defined by
|
||||||
|
[module bindings](config.md#module-bindings-and-validators).
|
||||||
|
|
||||||
|
Timeouts are layered. Caller cancellation is the outer authority. A positive
|
||||||
|
effective generation timeout adds an inner request deadline, while zero
|
||||||
|
disables only that generation deadline. The HTTP client timeout remains a
|
||||||
|
transport-wide cap. Notarius does not add another timeout around PromptKit.
|
||||||
|
The pinned upstream boundary and profile-format links are in
|
||||||
|
[PromptKit Integration](integrations/pkg-promptkit.md).
|
||||||
|
|
||||||
|
Concurrency has two independent layers. Notarius **total_llm** is the
|
||||||
|
application-wide provider-call limit shared by all backends, modules, retries,
|
||||||
|
and validators. PromptKit may impose a narrower admission limit for the
|
||||||
|
selected backend. The effective active-generation bound is the intersection of
|
||||||
|
both limits and can therefore be lower than **total_llm**. Built-in OpenRouter
|
||||||
|
profiles use PromptKit's upstream backend limit; endpoint-only profiles have no
|
||||||
|
PromptKit backend limit and remain bounded by Notarius. For the configured
|
||||||
|
local backend, a zero **concurrency_limit** leaves only the Notarius scheduler
|
||||||
|
as a call limit. A positive value makes the effective active local-generation
|
||||||
|
bound the smaller of **total_llm** and that local limit.
|
||||||
|
|
||||||
|
For a positive local limit, PromptKit owns its default waiting capacity and
|
||||||
|
admission behavior. When a PromptKit backend has admitted all active and queued
|
||||||
|
work, a new call fails as capacity exhaustion before generation. The adapter
|
||||||
|
maps that failure to Notarius's existing provider-neutral capacity error and
|
||||||
|
does not retry it. The calling stage's configured retry policy applies
|
||||||
|
normally, and the run fails if those attempts are exhausted. Caller
|
||||||
|
cancellation remains authoritative. Configuration contracts are documented
|
||||||
|
under [PromptKit profiles](config.md#promptkit-profiles) and
|
||||||
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
|
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
|
||||||
limits and actual provider-call limits are independent. Notarius writes local
|
limits and actual provider-call limits are independent. Notarius writes local
|
||||||
filesystem state only; remote storage, archival, and retention automation are
|
filesystem state only; remote storage, archival, and retention automation are
|
||||||
|
|||||||
@@ -79,6 +79,14 @@ Pipeline resolution requires a compatible codec and matching kind-specific
|
|||||||
variants before a typed lane can be accepted. Framework-owned erasure remains
|
variants before a typed lane can be accepted. Framework-owned erasure remains
|
||||||
private and must report type incompatibility as an error rather than a panic.
|
private and must report type incompatibility as an error rather than a panic.
|
||||||
|
|
||||||
|
An artifact kind may additionally provide a typed evidence projection that
|
||||||
|
copies its direct generic source references. Preparation proves that projection
|
||||||
|
matches the artifact codec's exact Go type before retaining it for an output
|
||||||
|
policy. The runner reconstructs evidence only from accepted serialized
|
||||||
|
normalized artifacts, and the output boundary owns any resulting publication.
|
||||||
|
Generic framework code never infers evidence by inspecting domain JSON or
|
||||||
|
depends on domain artifact types.
|
||||||
|
|
||||||
Auxiliary references provide context or disambiguation. They are not source
|
Auxiliary references provide context or disambiguation. They are not source
|
||||||
evidence and must not be converted into source references.
|
evidence and must not be converted into source references.
|
||||||
|
|
||||||
@@ -169,7 +177,8 @@ individual modules.
|
|||||||
The application-wide LLM scheduler bounds actual provider calls independently
|
The application-wide LLM scheduler bounds actual provider calls independently
|
||||||
of framework worker limits. Every LLM-backed module, retry, and validator uses
|
of framework worker limits. Every LLM-backed module, retry, and validator uses
|
||||||
the single injected scheduled client, including work performed by overlapping
|
the single injected scheduled client, including work performed by overlapping
|
||||||
lanes.
|
lanes. Provider runtime adapters may enforce a narrower backend-specific limit
|
||||||
|
beneath this mandatory application-wide scheduler.
|
||||||
|
|
||||||
## Configuration And Provenance
|
## Configuration And Provenance
|
||||||
|
|
||||||
|
|||||||
@@ -7,6 +7,37 @@ not as committed release dates.
|
|||||||
|
|
||||||
## Near-Term D&D Pipeline
|
## Near-Term D&D Pipeline
|
||||||
|
|
||||||
|
### Combat Enemy Ledger
|
||||||
|
|
||||||
|
- Add a D&D artifact that identifies enemies faced during combat and supports
|
||||||
|
an end-of-session encounter ledger.
|
||||||
|
- Track each enemy's observed state using a small controlled vocabulary such as
|
||||||
|
`active`, `killed`, `fled`, `captured`, or `incapacitated`, while preserving
|
||||||
|
an explicit unresolved state when the transcript does not establish an
|
||||||
|
outcome.
|
||||||
|
- Preserve the evidence for enemy participation and state changes rather than
|
||||||
|
inferring a terminal outcome from combat ending or an enemy disappearing
|
||||||
|
from the conversation.
|
||||||
|
- Define how repeated mentions, groups of unnamed enemies, summoned or allied
|
||||||
|
creatures, and the same enemy appearing in multiple combats affect identity
|
||||||
|
and ledger entries.
|
||||||
|
- Evaluate whether the ledger should be extracted directly, derived from
|
||||||
|
combat-turn artifacts, or use a sequential pipeline that consumes combat
|
||||||
|
turns and the normalized NPC registry as grounding references.
|
||||||
|
|
||||||
|
### Location Extraction
|
||||||
|
|
||||||
|
- Add a D&D artifact for locations visited by the party or otherwise mentioned
|
||||||
|
in the transcript.
|
||||||
|
- Distinguish observed visits from references, plans, recalled places, and
|
||||||
|
uncertain or inferred locations so a mention alone is not reported as a
|
||||||
|
visit.
|
||||||
|
- Preserve transcript evidence for each visit or mention and reconcile aliases,
|
||||||
|
nested places, and repeated appearances without collapsing distinct
|
||||||
|
locations that share a generic name.
|
||||||
|
- Define how the location artifact should ground later narrative reports and
|
||||||
|
whether future event artifacts should retain canonical location identities.
|
||||||
|
|
||||||
### Evaluate Spell Extraction And Normalization
|
### Evaluate Spell Extraction And Normalization
|
||||||
|
|
||||||
- Evaluate ordinary extraction retries and the completed normalization path
|
- Evaluate ordinary extraction retries and the completed normalization path
|
||||||
@@ -22,6 +53,71 @@ not as committed release dates.
|
|||||||
spell, combat, interaction, and scene-description lanes after real-world use.
|
spell, combat, interaction, and scene-description lanes after real-world use.
|
||||||
Add more complex chunking only in response to demonstrated failures.
|
Add more complex chunking only in response to demonstrated failures.
|
||||||
|
|
||||||
|
## Cross-Cutting LLM Runtime
|
||||||
|
|
||||||
|
### Deterministic Prompt Session Identity
|
||||||
|
|
||||||
|
- Replace the source-document-ID default for prompt sessions with one
|
||||||
|
predictable, procedurally generated session ID for the complete
|
||||||
|
source-processing workload.
|
||||||
|
- Preserve an explicit non-empty `--session-id` as the highest-precedence
|
||||||
|
override. Otherwise, derive the default only from the effective input module
|
||||||
|
identity and the exact raw input bytes.
|
||||||
|
- Use a versioned, bounded representation such as
|
||||||
|
`notarius:v1:<sha256(input-module + NUL + raw-input)>`. The exact encoding
|
||||||
|
must fit PromptKit's session length contract and must not embed source
|
||||||
|
content.
|
||||||
|
- Keep the derived session stable across runs, pipelines, selected lanes,
|
||||||
|
ordered steps, retries, resume, recomputation, LLM profiles, reasoning
|
||||||
|
overrides, and output, debug, or cache settings.
|
||||||
|
- Do not include file-backed references, generated references, reference
|
||||||
|
contents, or the composition of a reference bundle in session derivation.
|
||||||
|
References may change between prompt calls within one pipeline without
|
||||||
|
changing routing affinity.
|
||||||
|
- Resolve the authoritative session before checkpoint construction and use the
|
||||||
|
same value for checkpoint runtime identity, every prompt-facing module,
|
||||||
|
PromptKit's direct session field, the compatibility `session_id` prompt
|
||||||
|
variable, run-manifest metadata, and debug metadata.
|
||||||
|
- Keep routing identity separate from cache and checkpoint content identity.
|
||||||
|
Exact prompt prefixes, reference contents, model settings, and other
|
||||||
|
generation-affecting inputs must continue to participate in their existing
|
||||||
|
hashes and checkpoint fingerprints even though they do not change the
|
||||||
|
session.
|
||||||
|
- Treat the generated value as a provider-visible, stable pseudonymous
|
||||||
|
correlation identifier. Do not introduce an installation-specific HMAC or
|
||||||
|
secret unless a concrete multi-tenant or privacy requirement justifies
|
||||||
|
sacrificing deterministic identity across installations.
|
||||||
|
|
||||||
|
### Raise The Default Application-Wide LLM Limit
|
||||||
|
|
||||||
|
- Raise the default `concurrency.total_llm` value from 1 to 16 so ordinary
|
||||||
|
single-backend runs can use PromptKit's expected OpenRouter capacity and
|
||||||
|
lower-capacity local backends without an unnecessarily narrower Notarius
|
||||||
|
limit.
|
||||||
|
- Keep the Notarius application-wide scheduler mandatory and require
|
||||||
|
`total_llm` to remain a positive integer. Do not make the default unlimited:
|
||||||
|
endpoint-only profiles, an unrestricted local backend, injected clients, and
|
||||||
|
aggregate work across several backends may have no narrower PromptKit limit.
|
||||||
|
- Continue defaulting `concurrency.stage_workers.extract` to the effective
|
||||||
|
`total_llm`, making its default 16 as part of the same change. Preserve an
|
||||||
|
explicit lower extract-worker setting when an operator wants less queued or
|
||||||
|
concurrent extraction work.
|
||||||
|
- Define effective provider concurrency as the intersection of the Notarius
|
||||||
|
application-wide limit, the selected PromptKit backend limit when present,
|
||||||
|
and the work made available by stage execution. A Notarius limit of 16 does
|
||||||
|
not narrow a backend already limited to 16, while a local backend limited to
|
||||||
|
4 remains bounded at 4.
|
||||||
|
- Treat the default as an application-wide safety ceiling across profiles,
|
||||||
|
backends, modules, retries, and validators. A run that intentionally needs
|
||||||
|
the combined capacity of several backends may configure a higher
|
||||||
|
`total_llm` and an appropriate extract-worker count explicitly.
|
||||||
|
- Retain the existing configuration and environment override surfaces. Update
|
||||||
|
canonical configuration, operations, and internal documentation together
|
||||||
|
when the default changes.
|
||||||
|
- Reconsider decoupling the extract-worker default from `total_llm` only after
|
||||||
|
mixed-backend workloads demonstrate a need for a high global emergency
|
||||||
|
ceiling with a lower default work-production rate.
|
||||||
|
|
||||||
## Shared Normalization And Quality Work
|
## Shared Normalization And Quality Work
|
||||||
|
|
||||||
### Generic LLM-Assisted Deduplication
|
### Generic LLM-Assisted Deduplication
|
||||||
|
|||||||
@@ -1,284 +1,586 @@
|
|||||||
# Subprocess Integration Implementation Plan
|
# PromptKit v0.5 Implementation Plan
|
||||||
|
|
||||||
## Status
|
|
||||||
|
|
||||||
Completed.
|
|
||||||
|
|
||||||
## Objective
|
## Objective
|
||||||
|
|
||||||
Implement the accepted [Subprocess Integration Contract](subprocess.md) as a
|
Implement the target state in
|
||||||
small, generic extension of the existing CLI boundary. The completed work must
|
[PromptKit v0.5 Integration And LLM Profile Policy](promptkit.md). Each numbered
|
||||||
let a subprocess caller discover a successful Notarius output bundle through a
|
stage is intended to be one implementation prompt for a GPT-5.6-Terra coding
|
||||||
versioned JSON receipt without parsing human prose, while preserving current
|
agent. Complete stages in order and leave the repository buildable, tested, and
|
||||||
pipeline, output, rejection, warning, configuration, and interactive CLI
|
internally coherent after every stage.
|
||||||
behavior.
|
|
||||||
|
|
||||||
Complete the stages below in order. Each stage must leave its affected package
|
Follow [Architecture](../policy/architecture.md),
|
||||||
passing before the next begins. Do not add Narratio-specific production code,
|
[Testing Policy](../policy/testing.md), and
|
||||||
a public Go client, a new output encoder, or any feature listed as out of scope
|
[Documentation Policy](../policy/documentation.md) throughout. Preserve
|
||||||
in the feature roadmap.
|
unrelated user changes. Use `apply_patch` for source and documentation edits,
|
||||||
|
run `gofmt` on changed Go files, and add only tests that protect the behaviors
|
||||||
|
and risks assigned to that stage.
|
||||||
|
|
||||||
## Decisions And Invariants
|
Do not implement the separate deterministic session-ID or default-concurrency
|
||||||
|
roadmap items as part of this plan. Do not perform paid or credentialed LLM
|
||||||
|
calls.
|
||||||
|
|
||||||
- `--json` is a boolean flag on `notarius run`; it does not affect pipeline
|
## Background Summary
|
||||||
resolution, execution, checkpoint identity, output encoding, or publication.
|
|
||||||
- The machine result is a private CLI Go type implementing the public
|
|
||||||
`notarius.run-result.v1` wire contract. Do not expose framework or CLI Go
|
|
||||||
packages for external import.
|
|
||||||
- The finalized run manifest is authoritative for `run_id`, `pipeline_id`, and
|
|
||||||
`validation_status`. Verify that its pipeline ID matches the effective
|
|
||||||
resolved pipeline rather than emitting conflicting provenance.
|
|
||||||
- Human-oriented stdout remains byte-for-byte governed by its current path when
|
|
||||||
`--json` is absent. Machine mode emits no human status or debug-path line on
|
|
||||||
stdout.
|
|
||||||
- Warnings and errors remain on stderr. Exit statuses remain 0 for success, 1
|
|
||||||
for runtime failure, and 2 for syntax failure.
|
|
||||||
- Validation rejection remains a successful pipeline outcome. The receipt
|
|
||||||
reports counts but does not decide which lanes an orchestrator requires.
|
|
||||||
- The receipt points to the existing bundle; it does not duplicate output
|
|
||||||
descriptors, lane payloads, warnings, rejections, or manifest content.
|
|
||||||
- Reported filesystem paths are lexical absolute paths produced with
|
|
||||||
`filepath.Abs`. Do not resolve symlinks or change the physical directories
|
|
||||||
used for publication.
|
|
||||||
- For the production `json` output module, `index_file` is exactly
|
|
||||||
`index.json` and is included only after confirming that the runner returned
|
|
||||||
exactly one logical file with that clean name. For any injected or future
|
|
||||||
non-`json` output module, omit `index_file`; the machine receipt remains
|
|
||||||
generic.
|
|
||||||
- Machine-result construction and JSON encoding occur before debug success
|
|
||||||
terminalization, but stdout writing occurs only after output publication and
|
|
||||||
successful requested debug terminalization.
|
|
||||||
- Standard output is not transactional. A final writer failure returns exit 1
|
|
||||||
and may leave partial bytes. Consumers must ignore stdout unless the process
|
|
||||||
exits 0. Because debug terminalization must precede receipt emission, a
|
|
||||||
result-delivery failure does not rewrite an already persisted successful run
|
|
||||||
report; it is additionally reported on stderr and the process exits 1.
|
|
||||||
- Do not add a stable JSON error envelope or promote diagnostic wording into a
|
|
||||||
compatibility contract.
|
|
||||||
|
|
||||||
## Stage 1: Define The Run-Result Wire Model
|
Notarius currently pins PromptKit v0.3.0, calls `Prepare` and then `Run` for one
|
||||||
|
completion, validates profiles through a synthetic prompt, has no application
|
||||||
|
fallback profile source, and accepts LLM profiles only at individual bindings
|
||||||
|
or through the run-wide CLI override. PromptKit v0.5.0 is source-compatible
|
||||||
|
with the current tree; a temporary v0.5.0 module override has already passed
|
||||||
|
`go test ./...`.
|
||||||
|
|
||||||
Add a focused private implementation under `internal/cli`, preferably
|
The implementation must nevertheless treat the upstream optional-parameter
|
||||||
`run_result.go`, with:
|
change as intentional: unset `temperature`, `max_tokens`, and `top_p` remain
|
||||||
|
unset and are omitted from compatible provider requests. Do not restore the old
|
||||||
|
implicit `top_p: 1` default.
|
||||||
|
|
||||||
- a constant for `notarius.run-result.v1`;
|
## Stage 1: Upgrade The PromptKit Dependency
|
||||||
- a private result struct whose JSON fields and requiredness match the feature
|
|
||||||
roadmap;
|
|
||||||
- a constructor that accepts the resolved pipeline, completed runner output,
|
|
||||||
run output directory, and optional debug directory; and
|
|
||||||
- serialization into an owned byte slice containing exactly one compact JSON
|
|
||||||
object followed by one newline; and
|
|
||||||
- a complete-write helper for delivering those prepared bytes.
|
|
||||||
|
|
||||||
The result fields are:
|
### Goal
|
||||||
|
|
||||||
- required `schema_version`, `run_id`, `pipeline_id`, `output_directory`,
|
Establish a clean PromptKit v0.5.0 baseline before adopting its new APIs.
|
||||||
`normalized_output_count`, `rejected_output_count`, and `warning_count`;
|
|
||||||
- required non-empty `validation_status`, copied without reinterpretation from
|
|
||||||
the final run manifest;
|
|
||||||
- optional `index_file`, set to `index.json` for the resolved production JSON
|
|
||||||
output module; and
|
|
||||||
- optional `debug_directory`, present only when debug capture was requested and
|
|
||||||
its path is non-empty.
|
|
||||||
|
|
||||||
The constructor must:
|
### Work
|
||||||
|
|
||||||
1. validate the required run ID, pipeline ID, final validation status, and
|
- Update `go.mod` and `go.sum` from PromptKit v0.3.0 to v0.5.0 and run
|
||||||
non-empty run output directory rather than emit a malformed receipt;
|
`go mod tidy`.
|
||||||
2. verify that the final manifest pipeline ID matches the effective resolved
|
- Change the PromptKit built-in profile-catalog marker in
|
||||||
pipeline ID;
|
`internal/framework/llm/promptkit_profile_fingerprint.go` to identify
|
||||||
3. convert the output bundle and optional debug bundle paths to lexical absolute
|
v0.5.0. This deliberately invalidates LLM checkpoints tied to the prior
|
||||||
paths;
|
catalog identity.
|
||||||
4. derive all counts from the completed `pipeline.RunOutput`;
|
- Review PromptKit-facing compile errors or test failures against the v0.4.0
|
||||||
5. identify the output module through the resolved module key, not concrete
|
and v0.5.0 release guides. Do not adopt prepared execution, inspection, or
|
||||||
encoder types or D&D knowledge; and
|
fallback profiles in this stage.
|
||||||
6. for module key `pipeline.DefaultOutputModule`, verify exactly one returned
|
- Replace the existing test assertion for one exact built-in fingerprint hash
|
||||||
logical output file is named `index.json` before setting `index_file`.
|
with durable assertions that the fingerprint is deterministic, non-empty,
|
||||||
|
non-secret, and changes when a semantic profile source changes. Do not add a
|
||||||
|
new version-constant or exact-hash change detector.
|
||||||
|
- Update `docs/integrations/pkg-promptkit.md` to pin and link v0.5.0 and state
|
||||||
|
the implemented dependency-level behavior: unset optional sampling controls
|
||||||
|
are provider defaults. Do not document later stages as implemented.
|
||||||
|
- Update any other canonical text that explicitly claims the dependency is
|
||||||
|
v0.3.0, but defer descriptions of unimplemented v0.5 APIs.
|
||||||
|
|
||||||
Treat a missing or duplicate `index.json` from the production JSON output as a
|
### Tests And Validation
|
||||||
runtime contract error. Do not inspect or decode the index contents here; the
|
|
||||||
output encoder and its existing tests own that format.
|
|
||||||
|
|
||||||
Implement the writer with `encoding/json` and complete-write handling. It must
|
- `go test ./internal/framework/llm ./internal/cli`
|
||||||
surface encoding errors, zero-progress writes, short writes, and underlying
|
|
||||||
writer errors to the command boundary. The command must map a delivery failure
|
|
||||||
to a fixed, bounded result-write error without wrapping caller-supplied writer
|
|
||||||
text into user or debug diagnostics. Do not add an injected serializer or a
|
|
||||||
generic serialization framework merely to manufacture unreachable error cases
|
|
||||||
for tests.
|
|
||||||
|
|
||||||
Add lean focused tests for the wire-model boundary:
|
|
||||||
|
|
||||||
- required fields, counts, schema identity, and compact newline-terminated JSON;
|
|
||||||
- blank required manifest identities, blank validation status, and a mismatch
|
|
||||||
between final-manifest and effective pipeline IDs;
|
|
||||||
- lexical absolute conversion for relative output and debug paths;
|
|
||||||
- production JSON entry-point discovery;
|
|
||||||
- omission of `index_file` for a non-JSON output module;
|
|
||||||
- missing and duplicate production entry points; and
|
|
||||||
- a representative failing or zero-progress writer.
|
|
||||||
|
|
||||||
Decode structured output in tests rather than snapshotting an entire JSON
|
|
||||||
string. Assert literal field names and the schema identity because they are the
|
|
||||||
public compatibility contract.
|
|
||||||
|
|
||||||
Stage completion:
|
|
||||||
|
|
||||||
- `go test ./internal/cli`
|
|
||||||
|
|
||||||
## Stage 2: Integrate Machine Mode With Run Finalization
|
|
||||||
|
|
||||||
Update the run command in `internal/cli/run.go`:
|
|
||||||
|
|
||||||
- register `--json` with the existing `flag.FlagSet`;
|
|
||||||
- include it in root usage without adding it to `runFlagTakesValue`;
|
|
||||||
- leave all existing flag combinations valid; and
|
|
||||||
- do not add configuration or environment equivalents.
|
|
||||||
|
|
||||||
After the runner succeeds and requested debug summaries have been written,
|
|
||||||
apply this exact finalization order:
|
|
||||||
|
|
||||||
1. when `--json` is selected, construct and encode the complete receipt into
|
|
||||||
owned memory; fail before physical output publication if its required fields
|
|
||||||
or production entry point are invalid;
|
|
||||||
2. publish the runner's logical output files through the existing confined
|
|
||||||
output writer;
|
|
||||||
3. terminalize requested debug reporting as successful and stop without
|
|
||||||
writing stdout if terminalization fails;
|
|
||||||
4. in machine mode, write the prepared receipt to stdout and return a runtime
|
|
||||||
failure if that write does not complete;
|
|
||||||
5. otherwise, use the unchanged human-oriented success reporting path; and
|
|
||||||
6. report successful-run warning counts to stderr as today.
|
|
||||||
|
|
||||||
The receipt's `output_directory` is the absolute path to the run-specific
|
|
||||||
bundle, not the configured output root. `debug_directory` is the absolute path
|
|
||||||
to the allocated run-specific debug bundle.
|
|
||||||
|
|
||||||
On a machine-result stdout failure after terminalization:
|
|
||||||
|
|
||||||
- report a fixed, bounded, code-owned result-write error on stderr through the
|
|
||||||
existing CLI failure reporting surface without including the underlying
|
|
||||||
writer error;
|
|
||||||
- include the existing debug-path discovery line when applicable;
|
|
||||||
- return exit 1;
|
|
||||||
- do not attempt terminal reporting a second time; and
|
|
||||||
- leave the successfully published output and debug bundle intact.
|
|
||||||
|
|
||||||
Do not change `pipelineCommandState`, the debug run-report wire shape, the
|
|
||||||
output encoder contract, or `writeOutputFiles` solely to support this feature.
|
|
||||||
The machine receipt belongs to CLI reporting after the runner and output module
|
|
||||||
have completed their existing responsibilities.
|
|
||||||
|
|
||||||
Stage completion:
|
|
||||||
|
|
||||||
- `go test ./internal/cli`
|
|
||||||
|
|
||||||
## Stage 3: Protect The Public CLI Contract
|
|
||||||
|
|
||||||
Add or extend behavior-level CLI tests at the narrowest stable boundaries.
|
|
||||||
Reuse existing production components, fakes, state harnesses, and maintained
|
|
||||||
examples rather than creating a second subprocess fixture framework.
|
|
||||||
|
|
||||||
Cover:
|
|
||||||
|
|
||||||
- a representative maintained production invocation with `--json`, decoding
|
|
||||||
exactly one stdout document and verifying the schema identity, run and
|
|
||||||
pipeline IDs, absolute output directory, `index_file`, counts, validation
|
|
||||||
status, and the existence of the referenced bundle entry point;
|
|
||||||
- a warning-bearing debug run, proving a correct warning count, warning
|
|
||||||
reporting on stderr, an absolute optional debug path, and no human prose in
|
|
||||||
stdout;
|
|
||||||
- a rejection-bearing successful run using an explicit deterministic rejecting
|
|
||||||
validator, proving zero or partial normalized outputs and the correct
|
|
||||||
rejection count without changing exit status;
|
|
||||||
- one representative syntax failure and one representative runtime failure
|
|
||||||
with `--json`, proving the established exit class, stderr ownership, and no
|
|
||||||
completed success document;
|
|
||||||
- an injected stdout writer failure after successful publication, proving exit
|
|
||||||
1, retained output files, fixed diagnostic reporting, and omission of a
|
|
||||||
recognizable writer-error sentinel from stderr and debug diagnostics; and
|
|
||||||
- the existing no-`--json` tests continuing to protect interactive output.
|
|
||||||
|
|
||||||
For the writer-failure case, use a writer that fails before accepting bytes when
|
|
||||||
asserting empty stdout. The public contract nevertheless remains that arbitrary
|
|
||||||
writers may leave partial bytes and consumers must ignore stdout on nonzero
|
|
||||||
exit.
|
|
||||||
|
|
||||||
Do not:
|
|
||||||
|
|
||||||
- duplicate the JSON output encoder's lane, manifest, warning, or rejection
|
|
||||||
serialization matrix;
|
|
||||||
- assert complete human error strings;
|
|
||||||
- add a golden file for the small receipt;
|
|
||||||
- test private helper call order; or
|
|
||||||
- add a prompt-language or unrelated end-to-end test.
|
|
||||||
|
|
||||||
Stage completion:
|
|
||||||
|
|
||||||
- `go test ./internal/cli`
|
|
||||||
- `go test ./internal/modules/generic/output/json`
|
|
||||||
- `go test ./internal/modules/integration`
|
|
||||||
|
|
||||||
## Stage 4: Publish The Consumer-Facing Documentation
|
|
||||||
|
|
||||||
After the code and behavior tests pass, update the canonical current-behavior
|
|
||||||
documentation:
|
|
||||||
|
|
||||||
- `docs/cli.md`
|
|
||||||
- add `--json` to run syntax and its flag table;
|
|
||||||
- define stdout/stderr behavior and the requirement to parse machine stdout
|
|
||||||
only after exit 0;
|
|
||||||
- preserve the existing exit-status table and link to the run-result
|
|
||||||
contract.
|
|
||||||
- `docs/integrations/run-result.md`
|
|
||||||
- own the complete `notarius.run-result.v1` field table, requiredness,
|
|
||||||
example, path semantics, production `index_file` rule, partial-write rule,
|
|
||||||
and additive compatibility policy;
|
|
||||||
- link to the published JSON bundle contract rather than repeating its
|
|
||||||
descriptors or payload schemas.
|
|
||||||
- `docs/integrations/json-output.md`
|
|
||||||
- add only a narrow cross-link explaining that a subprocess caller obtains
|
|
||||||
the physical bundle root from the run-result receipt before using
|
|
||||||
`index.json` for logical discovery;
|
|
||||||
- do not duplicate run-result fields or stream semantics.
|
|
||||||
- `docs/consumers/subprocess.md`
|
|
||||||
- provide a concise task workflow for preflight, invocation, separate stream
|
|
||||||
capture, exit checking, receipt decoding, confined index resolution, lane
|
|
||||||
lookup by ID, media-type and schema-identity checks, required-versus-optional
|
|
||||||
consumer policy, provenance retention, and sensitive-data handling;
|
|
||||||
- use a generic orchestrator example with placeholders, not private Narratio
|
|
||||||
paths, credentials, or a duplicate complete configuration.
|
|
||||||
- `docs/internal/cli.md`
|
|
||||||
- document receipt construction before publication, emission after
|
|
||||||
publication and debug terminalization, and the stdout failure behavior;
|
|
||||||
- preserve the CLI composition-root boundary.
|
|
||||||
- `README.md` and `docs/development.md`
|
|
||||||
- add only the smallest links needed to make subprocess consumer guidance and
|
|
||||||
its implementation owner discoverable.
|
|
||||||
|
|
||||||
Do not otherwise change `docs/config.md`, `docs/operations.md`, or the JSON
|
|
||||||
output bundle contract unless implementation reveals an actual change to
|
|
||||||
behavior they own. The machine receipt adds no configuration field, filesystem
|
|
||||||
surface, retention policy, or lane-output shape.
|
|
||||||
|
|
||||||
After all code, tests, and current-behavior documentation are complete:
|
|
||||||
|
|
||||||
- change [the feature roadmap](subprocess.md) status to `Implemented`;
|
|
||||||
- change this plan's status to `Completed`; and
|
|
||||||
- ensure neither roadmap is used as the canonical description of current
|
|
||||||
behavior.
|
|
||||||
|
|
||||||
Stage completion:
|
|
||||||
|
|
||||||
- validate documentation links and command examples against the implementation;
|
|
||||||
- `git diff --check`
|
|
||||||
- `go test ./...`
|
- `go test ./...`
|
||||||
- `go vet ./...`
|
- `go vet ./...`
|
||||||
- `go build ./cmd/notarius`
|
- `go build ./cmd/notarius`
|
||||||
- `go test -race ./internal/cli`
|
- `rg -n 'promptkit v0\.3\.0|promptkit@v0\.3\.0|PromptKit v0\.3\.0' .`
|
||||||
|
- `git diff --check`
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- The repository directly pins v0.5.0 and all default offline checks pass.
|
||||||
|
- The profile-source fingerprint identifies the new upstream catalog without a
|
||||||
|
brittle literal-hash test.
|
||||||
|
- Current documentation no longer identifies v0.3.0 as the supported version.
|
||||||
|
|
||||||
|
## Stage 2: Execute One Frozen Prepared Snapshot
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Make Notarius debug details and generation use one exact PromptKit preparation.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Refactor `PromptKitClient.CompleteStructured` to call
|
||||||
|
`PrepareExecution`, immediately defer `Discard`, obtain a caller-owned
|
||||||
|
`Details` value, and execute with `RunPrepared`.
|
||||||
|
- Preserve the existing Notarius request mapping, cancellation precedence,
|
||||||
|
validation classification, raw structured bytes, response decoding,
|
||||||
|
profile recording, usage reporting, and credential redaction.
|
||||||
|
- Ensure every preparation, execution, validation, empty-result, and decode
|
||||||
|
error retains useful Notarius prompt context without exposing prepared handle
|
||||||
|
state or secrets.
|
||||||
|
- Use `errors.As` to obtain `*promptkit.CapacityError` on admission rejection.
|
||||||
|
Preserve `contracts.ErrLLMCapacityExceeded` as the stable classification and
|
||||||
|
add a nonblank backend ID only to safe application-owned diagnostic context.
|
||||||
|
Do not expose `promptkit.CapacityError` outside the LLM adapter.
|
||||||
|
- Update `docs/internal/llm.md` and the implemented-mechanics portion of
|
||||||
|
`docs/integrations/pkg-promptkit.md` to describe the single frozen execution
|
||||||
|
snapshot and structured capacity adaptation.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Adapt existing PromptKit client tests to the prepared-execution path.
|
||||||
|
- Retain or add one behavioral test proving that the debug prompt details match
|
||||||
|
the request actually passed to generation when a backing prompt source could
|
||||||
|
otherwise change between independent preparations. Test the resulting
|
||||||
|
snapshot consistency, not a private helper call count.
|
||||||
|
- Retain capacity tests proving `errors.Is` reaches
|
||||||
|
`contracts.ErrLLMCapacityExceeded`, the selected backend can appear in safe
|
||||||
|
diagnostic context, and provider calls are not made after rejected
|
||||||
|
admission.
|
||||||
|
- Run `go test ./internal/framework/llm` and
|
||||||
|
`go test -race ./internal/framework/llm`.
|
||||||
|
- Run `go test ./...` and `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- `CompleteStructured` no longer calls independent `Prepare` and `Run`
|
||||||
|
operations for one request.
|
||||||
|
- Debug prompt material and generation result originate from the same frozen
|
||||||
|
PromptKit snapshot.
|
||||||
|
- Capacity remains a provider-neutral Notarius error classification.
|
||||||
|
|
||||||
|
## Stage 3: Replace Synthetic Profile Validation With Inspection
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Validate profiles through PromptKit's exact profile-inspection boundary and
|
||||||
|
centralize engine profile-source construction.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Introduce a small provider-adapter-owned profile inspection or validation
|
||||||
|
function in `internal/framework/llm`. Its public internal signature must use
|
||||||
|
Notarius-owned configuration and result/error types rather than returning
|
||||||
|
PromptKit types to the CLI.
|
||||||
|
- Share the code that applies `profile_dir`, `profile_file`, and registered
|
||||||
|
backend options between the production PromptKit engine and the inspection
|
||||||
|
engine. Preserve the mutual-exclusion and local-backend rules.
|
||||||
|
- Change CLI explicit-profile preflight to use `Engine.InspectProfile` through
|
||||||
|
that LLM boundary.
|
||||||
|
- Remove `profileCheckPromptID`, `profileCheckPromptFS`, the `testing/fstest`
|
||||||
|
production dependency, and the synthetic `Prepare` request.
|
||||||
|
- Preserve distinct, useful errors for an absent profile, invalid profile,
|
||||||
|
unknown backend registration, cancellation, and invalid profile source.
|
||||||
|
- Do not require `api_key_env` to be populated during configuration validation.
|
||||||
|
Inspection may report credential requirements internally, but actual
|
||||||
|
preparation remains responsible for credential availability before a model
|
||||||
|
call.
|
||||||
|
- Update current-behavior sections in `docs/internal/cli.md` and
|
||||||
|
`docs/internal/llm.md`. Keep field definitions in `docs/config.md`.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Replace synthetic-prompt tests with profile inspection tests covering:
|
||||||
|
configured local backend success; missing local backend failure; absent
|
||||||
|
profile; malformed profile; and an otherwise valid profile whose credential
|
||||||
|
environment variable is intentionally unset.
|
||||||
|
- Prove validation performs no provider HTTP call and remains offline.
|
||||||
|
- Run `go test ./internal/framework/llm ./internal/cli` and `go test ./...`.
|
||||||
|
- Run `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- No production synthetic profile-check prompt remains.
|
||||||
|
- Profile validation uses the same ordinary profile source and backend
|
||||||
|
registrations as execution.
|
||||||
|
- Configuration validation succeeds for structurally valid profiles without
|
||||||
|
reading credential values.
|
||||||
|
|
||||||
|
## Stage 4: Add Application Fallback Profile Asset Plumbing
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Allow module families to register application-owned fallback profile YAML
|
||||||
|
without placing domain policy in generic LLM code.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Extend `internal/framework/llm.AssetRegistry` with a separate fallback
|
||||||
|
profile source collection, registration method, flattened filesystem, and
|
||||||
|
safe content digest.
|
||||||
|
- Reuse the existing asset-source path validation and flattening behavior where
|
||||||
|
appropriate. Reject invalid roots, unreadable assets, and duplicate flattened
|
||||||
|
paths. Do not parse PromptKit profile YAML in Notarius.
|
||||||
|
- Add `promptkit.WithFallbackProfileFS` to production engine options only when
|
||||||
|
at least one fallback profile source is registered.
|
||||||
|
- Supply the identical assembled fallback source to the profile-inspection
|
||||||
|
engine. Adjust CLI composition so pipeline-aware profile validation can use
|
||||||
|
the production LLM asset registry without exposing PromptKit types.
|
||||||
|
- Extend profile-source checkpoint identity to include the exact fallback
|
||||||
|
profile asset digest in addition to the PromptKit catalog marker and operator
|
||||||
|
source. Keep the resulting fingerprint hash-only and path/content/credential
|
||||||
|
free.
|
||||||
|
- Keep operator source precedence owned by PromptKit. Do not implement profile
|
||||||
|
merging or duplicate PromptKit source resolution in Notarius.
|
||||||
|
- Update `docs/internal/llm.md` only for the new implemented generic asset and
|
||||||
|
fingerprint mechanics. No domain fallback exists until Stage 5.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Add focused AssetRegistry tests for successful flattening, invalid roots,
|
||||||
|
duplicate paths, and hash changes when fallback bytes change.
|
||||||
|
- Add adapter-level tests showing that the fallback filesystem reaches both
|
||||||
|
execution construction and inspection construction.
|
||||||
|
- Extend checkpoint tests to prove fallback content changes profile-source
|
||||||
|
identity without exposing raw YAML or paths. Use relational comparisons, not
|
||||||
|
a fixed hash literal.
|
||||||
|
- Run `go test ./internal/framework/llm ./internal/cli` and `go test ./...`.
|
||||||
|
- Run `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- Generic plumbing can carry application fallback profiles while remaining
|
||||||
|
unaware of D&D IDs or model settings.
|
||||||
|
- Inspection, execution, and checkpoint identity use the same fallback asset
|
||||||
|
source.
|
||||||
|
|
||||||
|
## Stage 5: Adopt The D&D `dnd-extraction` Fallback
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Give the D&D module family one stable embedded workload profile that operators
|
||||||
|
can replace.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Add a D&D-owned embedded PromptKit profile asset with ID `dnd-extraction`
|
||||||
|
under `internal/modules/dnd`. Use the exact baseline defined in
|
||||||
|
`promptkit.md`: OpenRouter, `openai/gpt-5.6-luna`, no explicit reasoning
|
||||||
|
effort, a 240-second timeout, flex service tier, and no selected temperature,
|
||||||
|
token limit, or `top_p`. The omitted reasoning value intentionally allows
|
||||||
|
OpenAI's backend to apply its `medium` default.
|
||||||
|
- Register the profile filesystem from the D&D registrar through the generic
|
||||||
|
fallback profile asset boundary. Keep D&D policy out of
|
||||||
|
`internal/framework/llm` and the CLI composition root.
|
||||||
|
- Change every maintained D&D LLM prompt definition—including scene chunking,
|
||||||
|
all D&D extractors, and NPC normalization—from the model-named default to
|
||||||
|
`default_profile: dnd-extraction`.
|
||||||
|
- Add an integration-level profile-resolution test proving that:
|
||||||
|
- the fallback resolves when no operator source defines the ID;
|
||||||
|
- a valid operator profile with the same ID wins completely; and
|
||||||
|
- an invalid matching operator profile fails rather than falling through.
|
||||||
|
- Test through Notarius's assembled production assets and PromptKit boundary;
|
||||||
|
do not duplicate every upstream source-precedence case.
|
||||||
|
- Update the implemented profile ownership and prompt-default behavior in
|
||||||
|
`docs/internal/dnd.md`, `docs/internal/llm.md`, and
|
||||||
|
`docs/integrations/pkg-promptkit.md`. Defer the complete operator walkthrough
|
||||||
|
and examples to Stage 10.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Run focused D&D prompt preparation tests and the production composition
|
||||||
|
tests.
|
||||||
|
- Run `go test ./internal/modules/dnd/... ./internal/framework/llm
|
||||||
|
./internal/cli`.
|
||||||
|
- Run `go test ./...`.
|
||||||
|
- Verify `rg -n 'default_profile: gemini-2-flash' internal/modules/dnd`
|
||||||
|
returns no matches.
|
||||||
|
- Run `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- All maintained D&D prompts use the application-owned logical profile ID.
|
||||||
|
- The fallback works without an operator profile and remains authoritatively
|
||||||
|
overridable by a matching valid operator definition.
|
||||||
|
|
||||||
|
## Stage 6: Introduce Module Execution-Class Metadata
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Make each production module's ability to use an LLM statically discoverable
|
||||||
|
without yet changing profile inheritance.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Add `ExecutionClass contracts.ExecutionClass` to `pipeline.ModuleSpec` and
|
||||||
|
preserve it through normalization, cloning, catalogs, registries, JSON/debug
|
||||||
|
views, and lookup helpers.
|
||||||
|
- In this transitional stage only, allow an omitted execution class to
|
||||||
|
normalize to deterministic so existing test-only fixtures can be migrated in
|
||||||
|
Stage 7 without breaking the repository midway.
|
||||||
|
- Explicitly classify every production module:
|
||||||
|
- D&D scene chunking, every D&D extractor, and D&D NPC normalization as
|
||||||
|
`llm_backed`;
|
||||||
|
- all other current production input, chunk, merge, normalize, and output
|
||||||
|
modules as `deterministic`.
|
||||||
|
- Update production module specification tests and production catalog tests to
|
||||||
|
assert the semantic class alongside stage, artifact kind, and capabilities.
|
||||||
|
- Add catalog lookup support needed by later resolution to retrieve a selected
|
||||||
|
module's execution class by stage and key without constructing it.
|
||||||
|
- Do not implement pipeline-level profile inheritance or reject deterministic
|
||||||
|
profiles yet.
|
||||||
|
- Update `docs/internal/modules.md` and `docs/internal/dnd.md` to identify
|
||||||
|
execution class as registered module metadata, while noting only implemented
|
||||||
|
uses.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Run module registration/spec tests across generic, Seriatim, and D&D
|
||||||
|
families.
|
||||||
|
- Run `go test ./internal/framework/pipeline ./internal/modules/...`.
|
||||||
|
- Run `go test ./...` and `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- Every production module has an explicit correct execution class.
|
||||||
|
- Catalog consumers can retrieve that class without a concrete module
|
||||||
|
instance.
|
||||||
|
- Test-only omitted classes remain the only temporary compatibility behavior.
|
||||||
|
|
||||||
|
## Stage 7: Enforce Execution Metadata And Remove Runtime Probing
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Finish the execution-class contract so missing metadata cannot cause future
|
||||||
|
profile drift.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Update every framework, CLI, and integration test module specification to
|
||||||
|
declare an explicit execution class appropriate to the fake behavior.
|
||||||
|
- Change module-spec validation so an empty or unsupported execution class is a
|
||||||
|
registration error. Remove the transitional deterministic default from
|
||||||
|
Stage 6.
|
||||||
|
- Replace the chunk runner's special `ChunkExecutionClassProvider` probe with
|
||||||
|
specification-derived behavior. Remove the now-redundant provider interface,
|
||||||
|
implementation methods, and tests when they have no remaining consumer.
|
||||||
|
- Ensure chunk producer provenance remains unchanged: it records a non-empty
|
||||||
|
effective binding profile for an LLM-backed chunker, while a deterministic
|
||||||
|
chunker records no profile. A profile selected only through the prompt
|
||||||
|
default remains represented by PromptKit's actual-profile manifest rather
|
||||||
|
than being invented as an explicit chunk binding.
|
||||||
|
- Review helper constructors and fixtures for opportunities to set execution
|
||||||
|
class once without obscuring the class under test. Do not introduce an
|
||||||
|
elaborate test-spec framework.
|
||||||
|
- Update internal documentation if the removal changes any described runtime
|
||||||
|
mechanics.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Add or retain focused registration tests for missing and invalid execution
|
||||||
|
classes.
|
||||||
|
- Retain chunk-plan provenance tests for LLM-backed and deterministic
|
||||||
|
chunkers.
|
||||||
|
- Run `go test ./internal/framework/pipeline ./internal/modules/...`.
|
||||||
|
- Run `go test ./...`, `go vet ./...`, and `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- No registered module specification relies on an implicit execution class.
|
||||||
|
- Pipeline metadata, not a concrete runtime type assertion, owns module
|
||||||
|
execution classification.
|
||||||
|
|
||||||
|
## Stage 8: Resolve Programmatic Pipeline Profile Defaults
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Implement profile inheritance and precedence inside the pipeline resolver
|
||||||
|
before exposing the field through YAML configuration.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Add an optional trimmed `LLMProfile` field to
|
||||||
|
`pipeline.PipelineProfile`. Add a non-empty runtime override field to
|
||||||
|
`pipeline.ResolveOptions` so all precedence decisions occur in the resolver
|
||||||
|
rather than through pre-resolution mutation.
|
||||||
|
- After module selection, `--only` filtering, default validator-chain
|
||||||
|
selection, and validator compatibility resolution, apply effective profiles
|
||||||
|
to every selected input, chunk, extract, merge, normalize, output, and
|
||||||
|
validator binding according to the precedence in `promptkit.md`.
|
||||||
|
- Apply profiles only when the selected module or validator execution class is
|
||||||
|
`llm_backed`.
|
||||||
|
- Reject a binding-specific `llm_profile` on any deterministic module or
|
||||||
|
validator. Do not reject or inspect an unused pipeline default when no
|
||||||
|
selected LLM-backed binding consumes it.
|
||||||
|
- Leave an LLM-backed binding empty when no CLI, binding, or pipeline profile is
|
||||||
|
selected so PromptKit can use the prompt's `default_profile`.
|
||||||
|
- Store the effective values on resolved bindings before digest construction.
|
||||||
|
Do not add a second inheritance decision to execution.
|
||||||
|
- Ensure semantically equivalent repeated binding profiles and one inherited
|
||||||
|
default produce the same resolved pipeline digest. Ensure any changed
|
||||||
|
effective profile changes the digest.
|
||||||
|
- Do not modify file configuration or CLI parsing in this stage.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Add pipeline package tests for the complete precedence matrix:
|
||||||
|
runtime override; binding-specific exception; pipeline default; prompt
|
||||||
|
fallback; and deterministic bindings.
|
||||||
|
- Cover default and explicitly configured validator chains, all relevant stage
|
||||||
|
categories, `--only` lane selection, unused defaults, deterministic-profile
|
||||||
|
rejection, and semantic digest equivalence.
|
||||||
|
- Prefer table-driven package-level tests over assertions on private traversal
|
||||||
|
helpers.
|
||||||
|
- Run `go test ./internal/framework/pipeline` and `go test ./...`.
|
||||||
|
- Run `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- Programmatic pipelines resolve one canonical effective profile policy.
|
||||||
|
- Only LLM-backed resolved bindings can contain a profile.
|
||||||
|
- Runtime override, binding, pipeline, and prompt precedence is unambiguous and
|
||||||
|
digest-stable.
|
||||||
|
|
||||||
|
## Stage 9: Expose Pipeline Defaults Through Configuration And CLI
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Make the profile-default workflow available to operators while preserving
|
||||||
|
validation and override behavior.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Add optional `pipelines.<id>.llm_profile` support to the version 4 file
|
||||||
|
configuration model. Use presence-aware decoding so an explicitly set blank
|
||||||
|
value is rejected, while omission remains valid.
|
||||||
|
- Preserve the field through file application, configuration cloning,
|
||||||
|
effective configuration, and programmatic profile copies without aliasing or
|
||||||
|
trimming drift.
|
||||||
|
- Remove `applyLLMProfileOverride`. Pass the CLI override through the resolver's
|
||||||
|
runtime-override input so deterministic bindings are never populated.
|
||||||
|
- Update effective profile-ID collection to cover every selected LLM-backed
|
||||||
|
module stage and LLM-backed validator, including future LLM-backed input and
|
||||||
|
output modules. Do not inspect deterministic or unselected profiles.
|
||||||
|
- Ensure `run`, `config validate --pipeline`, resume/checkpoint identity, and
|
||||||
|
relevant dry preflight paths all use the same resolved effective profiles.
|
||||||
|
- Preserve `--llm-profile` as the highest-precedence non-empty run-wide
|
||||||
|
override and preserve binding-specific profiles as exceptions when no CLI
|
||||||
|
override is present.
|
||||||
|
- Do not increment the configuration version.
|
||||||
|
- Update current configuration and CLI contracts in `docs/config.md` and
|
||||||
|
`docs/cli.md` in the same stage. Link to operations for the deployment
|
||||||
|
workflow rather than duplicating it prematurely.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Add file-config tests for omission, trimming, explicit blank rejection,
|
||||||
|
unknown-key behavior, cloning, and round-trip application.
|
||||||
|
- Add effective-config and CLI contract tests for precedence, LLM-only
|
||||||
|
application, inherited-profile inspection failure before factory execution,
|
||||||
|
`--only`, and digest changes.
|
||||||
|
- Retain offline operation and do not require credentials for
|
||||||
|
`config validate --pipeline`.
|
||||||
|
- Run `go test ./internal/core/config ./internal/framework/pipeline
|
||||||
|
./internal/cli`.
|
||||||
|
- Run `go test ./...`, `go vet ./...`, and `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- Operators can select `dnd-extraction` once per pipeline.
|
||||||
|
- Configuration and CLI paths share the resolver's precedence policy.
|
||||||
|
- Unknown effective profiles fail preflight, while deterministic and unused
|
||||||
|
profiles do not cause spurious inspection.
|
||||||
|
|
||||||
|
## Stage 10: Complete Operator Documentation, Examples, And Decision Record
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Make the implemented workflow understandable, copyable, and maintainable
|
||||||
|
without duplicating canonical facts.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Create an ADR using the next sequential number for the durable decision to
|
||||||
|
use workload-oriented pipeline defaults with operator-overridable application
|
||||||
|
fallback profiles. Record context, decision, alternatives, and consequences;
|
||||||
|
do not turn the ADR into a field reference or implementation log.
|
||||||
|
- Complete `docs/config.md` as the canonical owner of profile-source fields,
|
||||||
|
`pipelines.<id>.llm_profile`, validation, and precedence.
|
||||||
|
- Complete `docs/operations.md` with an operator workflow that distinguishes
|
||||||
|
Notarius embedded prompts, Notarius fallback profiles, PromptKit built-ins,
|
||||||
|
and deployment filesystem profiles. Include production/development/local use
|
||||||
|
of the same `dnd-extraction` ID, credential handling, absolute-path guidance,
|
||||||
|
and the fact that current relative profile paths use the process working
|
||||||
|
directory rather than the configuration file's directory.
|
||||||
|
- Complete `docs/integrations/pkg-promptkit.md` with the v0.5.0 boundary,
|
||||||
|
prepared execution, inspection, fallback and ordinary source precedence,
|
||||||
|
optional provider controls, capacity adaptation, and compatibility policy.
|
||||||
|
- Update `docs/internal/configuration.md`, `docs/internal/pipeline.md`,
|
||||||
|
`docs/internal/cli.md`, `docs/internal/llm.md`, `docs/internal/modules.md`, and
|
||||||
|
`docs/internal/dnd.md` only for their owned implementation details. Link to
|
||||||
|
canonical configuration, operations, and upstream format contracts rather
|
||||||
|
than restating them.
|
||||||
|
- Keep exactly the existing two D&D configuration examples. Add
|
||||||
|
`llm_profile: dnd-extraction` to the minimal and complete pipelines and remove
|
||||||
|
the now-redundant model-named binding override from the complete example.
|
||||||
|
- Add one secret-free maintained operator profile at
|
||||||
|
`examples/profiles/dnd-extraction.yml`. It should be a complete valid profile
|
||||||
|
for the same logical ID and may mirror the embedded baseline; its purpose is
|
||||||
|
to demonstrate file ownership and format, not claim automatic environment
|
||||||
|
detection. Link it from the configuration and operations documentation.
|
||||||
|
- If the complete example selects the external profile file, use a path that
|
||||||
|
is valid for the documented repository-root invocation and explicitly note
|
||||||
|
the working-directory rule. Keep the minimal example dependent only on the
|
||||||
|
embedded fallback.
|
||||||
|
- Add or extend maintained-example validation so both configuration examples
|
||||||
|
and the profile YAML are checked without generation or credentials.
|
||||||
|
- Remove the now-implemented `Pipeline-Level LLM Profile Defaults` section from
|
||||||
|
`docs/roadmap/future.md`. Preserve the unrelated deterministic session and
|
||||||
|
concurrency items.
|
||||||
|
- Do not delete `promptkit.md` or this implementation plan during the feature
|
||||||
|
implementation; retire them only after post-implementation review.
|
||||||
|
|
||||||
|
### Tests And Validation
|
||||||
|
|
||||||
|
- Run maintained example/configuration tests and relevant CLI help/parser
|
||||||
|
tests.
|
||||||
|
- Run `go test ./...`.
|
||||||
|
- Run `rg -n 'gemini-2-flash' examples docs` and review every remaining match
|
||||||
|
for intentional model-policy or historical context.
|
||||||
|
- Run `rg -n 'v0\.3\.0|profileCheckPrompt|applyLLMProfileOverride' .` and resolve
|
||||||
|
stale production or current-documentation matches.
|
||||||
|
- Verify all new links and `git diff --check`.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- Every current fact has one canonical documentation owner.
|
||||||
|
- Operators can distinguish and deploy all profile layers without reading Go
|
||||||
|
source.
|
||||||
|
- Both maintained configurations and the maintained external profile are valid,
|
||||||
|
secret-free, and tested offline.
|
||||||
|
- Implemented profile work no longer remains in `future.md`.
|
||||||
|
|
||||||
|
## Stage 11: Final Verification And Quality Review
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Verify the complete migration as one integrated change and correct only defects
|
||||||
|
or omissions found during that review.
|
||||||
|
|
||||||
|
### Work
|
||||||
|
|
||||||
|
- Review the final diff against every acceptance criterion in `promptkit.md`.
|
||||||
|
- Confirm provider-specific PromptKit types remain inside the LLM integration
|
||||||
|
boundary and D&D policy remains inside the D&D module family.
|
||||||
|
- Confirm execution and inspection receive identical ordinary, fallback, and
|
||||||
|
backend configuration.
|
||||||
|
- Confirm no paths, profile YAML, endpoints, credentials, or prepared handle
|
||||||
|
state leak into fingerprints or ordinary diagnostics.
|
||||||
|
- Confirm all production module specs have explicit correct execution classes
|
||||||
|
and every resolved deterministic binding is profile-free.
|
||||||
|
- Confirm prompt default, pipeline default, binding override, and CLI override
|
||||||
|
behavior through representative assembled configurations.
|
||||||
|
- Review tests for redundancy and remove obsolete synthetic-prompt,
|
||||||
|
runtime-probe, exact-hash, or duplicated upstream-behavior tests superseded by
|
||||||
|
stronger contract tests.
|
||||||
|
- Perform an optional manual D&D quality comparison if credentials and an
|
||||||
|
evaluation transcript are deliberately supplied. Record no private input or
|
||||||
|
credential material, and do not make this comparison a completion gate.
|
||||||
|
|
||||||
|
### Validation Commands
|
||||||
|
|
||||||
|
```sh
|
||||||
|
gofmt -w <changed-go-files>
|
||||||
|
go test ./...
|
||||||
|
go test -race ./internal/framework/llm ./internal/core/config ./internal/framework/pipeline ./internal/cli
|
||||||
|
go vet ./...
|
||||||
|
go build ./cmd/notarius
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
Also run focused stale-contract searches:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
rg -n 'gitea.maximumdirect.net/eric/promptkit v0\.3\.0|PromptKit v0\.3\.0' .
|
||||||
|
rg -n 'default_profile: gemini-2-flash|profileCheckPrompt|applyLLMProfileOverride' internal docs examples
|
||||||
|
```
|
||||||
|
|
||||||
|
Review any matches rather than deleting intentional historical references
|
||||||
|
blindly.
|
||||||
|
|
||||||
|
### Completion Criteria
|
||||||
|
|
||||||
|
- All automated checks pass offline and without real credentials.
|
||||||
|
- The implemented behavior matches `promptkit.md` with no known architecture,
|
||||||
|
provenance, checkpoint, profile-precedence, or documentation gap.
|
||||||
|
- Any optional live evaluation is clearly separate from correctness testing.
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
None. The decisions above resolve stdout partial-write behavior, output-module
|
None. The roadmap decisions are sufficient to implement every stage without an
|
||||||
generality, path normalization, partial pipeline success, terminal reporting,
|
additional product or architecture choice.
|
||||||
documentation ownership, and test boundaries.
|
|
||||||
|
|||||||
318
docs/roadmap/promptkit.md
Normal file
318
docs/roadmap/promptkit.md
Normal file
@@ -0,0 +1,318 @@
|
|||||||
|
# PromptKit v0.5 Integration And LLM Profile Policy
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This roadmap defines the target state for upgrading Notarius from PromptKit
|
||||||
|
v0.3.0 to v0.5.0 and adopting the upstream runtime and profile facilities that
|
||||||
|
directly improve Notarius. It also defines the application policy for stable,
|
||||||
|
domain-oriented LLM profile names, operator overrides, pipeline inheritance,
|
||||||
|
profile validation, provider defaults, checkpoint identity, and documentation.
|
||||||
|
|
||||||
|
The ordered work needed to reach this state belongs in
|
||||||
|
[the implementation plan](implementation.md). Current behavior remains defined
|
||||||
|
by the canonical documentation outside `docs/roadmap/` until the corresponding
|
||||||
|
work is implemented.
|
||||||
|
|
||||||
|
## Background
|
||||||
|
|
||||||
|
Notarius currently pins PromptKit v0.3.0. Its adapter prepares a request once
|
||||||
|
for debug material and then independently runs the original request, causing
|
||||||
|
PromptKit to prepare the same logical call a second time. The CLI validates an
|
||||||
|
explicit profile by preparing a synthetic prompt. PromptKit profile selection
|
||||||
|
can be repeated on individual module bindings or replaced for one invocation
|
||||||
|
with `--llm-profile`, but a configured pipeline cannot yet declare one inherited
|
||||||
|
profile policy.
|
||||||
|
|
||||||
|
PromptKit v0.4.0 and v0.5.0 add the upstream boundaries needed to improve these
|
||||||
|
areas:
|
||||||
|
|
||||||
|
- [v0.4.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.4.0.md)
|
||||||
|
adds opaque prepared executions, exact profile and prompt inspection, and a
|
||||||
|
typed backend-capacity error;
|
||||||
|
- [v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.5.0.md)
|
||||||
|
adds application fallback profile filesystems and stops sending unset
|
||||||
|
optional sampling controls as framework-selected provider values; and
|
||||||
|
- the [v0.5.0 format contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
|
||||||
|
defines the resulting profile-source and execution-setting precedence.
|
||||||
|
|
||||||
|
A source-compatibility test of the current Notarius repository against
|
||||||
|
PromptKit v0.5.0 completed successfully. The work is therefore primarily an
|
||||||
|
intentional runtime and configuration migration rather than a repair for a
|
||||||
|
breaking Go API change.
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
- Pin and document PromptKit v0.5.0 as Notarius's supported upstream contract.
|
||||||
|
- Execute the exact prepared request snapshot whose safe details are recorded
|
||||||
|
in Notarius debug material.
|
||||||
|
- Validate configured PromptKit profiles through the upstream inspection API
|
||||||
|
without synthetic prompts, provider calls, or credential-value access.
|
||||||
|
- Give Notarius an application-owned, operator-overridable
|
||||||
|
`dnd-extraction` profile fallback.
|
||||||
|
- Let a pipeline choose one default LLM profile without repeating that ID on
|
||||||
|
every LLM-backed binding.
|
||||||
|
- Apply profile inheritance and run-wide overrides only where the resolved
|
||||||
|
module or validator can use an LLM.
|
||||||
|
- Preserve accurate checkpoint invalidation, effective profile provenance,
|
||||||
|
redaction, cancellation, concurrency, and provider-neutral module contracts.
|
||||||
|
- Provide operators with one clear deployment pattern for production,
|
||||||
|
development, and local profile definitions.
|
||||||
|
|
||||||
|
## Target End State
|
||||||
|
|
||||||
|
### PromptKit Runtime Boundary
|
||||||
|
|
||||||
|
Notarius depends on PromptKit v0.5.0 and uses its public APIs rather than
|
||||||
|
reimplementing source or execution resolution.
|
||||||
|
|
||||||
|
For each structured completion, the adapter:
|
||||||
|
|
||||||
|
1. builds one PromptKit run request from the provider-neutral Notarius request;
|
||||||
|
2. calls `PrepareExecution` once;
|
||||||
|
3. immediately arranges an idempotent `Discard` for every unexecuted handle;
|
||||||
|
4. obtains credential-redacted `Details` for debug and response metadata; and
|
||||||
|
5. calls `RunPrepared` so generation uses that exact frozen snapshot.
|
||||||
|
|
||||||
|
The debug prompt and successful result therefore describe the same selected
|
||||||
|
profile, rendered messages, input bytes, session, output contract, and effective
|
||||||
|
settings even when a filesystem-backed source changes concurrently. PromptKit
|
||||||
|
handle types remain private to `internal/framework/llm`.
|
||||||
|
|
||||||
|
PromptKit admission failures continue to match Notarius's provider-neutral
|
||||||
|
`ErrLLMCapacityExceeded` contract. When PromptKit supplies a `CapacityError`,
|
||||||
|
the adapter obtains the normalized backend ID through `errors.As` and may add it
|
||||||
|
to safe application-owned diagnostics without parsing upstream error wording.
|
||||||
|
The backend ID does not become a provider-specific module contract.
|
||||||
|
|
||||||
|
### Optional Provider Controls
|
||||||
|
|
||||||
|
Notarius accepts PromptKit v0.5.0's new behavior for `temperature`,
|
||||||
|
`max_tokens`, and `top_p`: an unset setting is omitted from compatible provider
|
||||||
|
requests and the provider chooses its own default. Notarius does not restore
|
||||||
|
PromptKit's former implicit `top_p: 1` value globally.
|
||||||
|
|
||||||
|
An operator who requires a particular value specifies it in the selected
|
||||||
|
PromptKit profile. The application fallback described below intentionally
|
||||||
|
leaves these controls unset. A human-reviewed D&D extraction comparison should
|
||||||
|
be performed after the upgrade, but paid or nondeterministic model output is
|
||||||
|
not part of the default automated test suite.
|
||||||
|
|
||||||
|
### Profile Inspection
|
||||||
|
|
||||||
|
Pipeline-aware configuration validation uses `Engine.InspectProfile` for every
|
||||||
|
effective explicit profile ID. It verifies that the profile exists, parses and
|
||||||
|
validates, resolves its backend and target, and is compatible with the engine's
|
||||||
|
registered backends. It does not create a synthetic prompt, load prompt inputs,
|
||||||
|
contact a provider, or require credential values to exist in the validation
|
||||||
|
process environment.
|
||||||
|
|
||||||
|
Credential availability is execution-time state. PromptKit preparation still
|
||||||
|
enforces the selected profile's credential contract before generation. This
|
||||||
|
keeps `notarius config validate` useful in build and deployment validation
|
||||||
|
environments where secrets are deliberately absent.
|
||||||
|
|
||||||
|
PromptKit construction for inspection and execution uses one shared internal
|
||||||
|
profile-source and backend-option path. The CLI does not expose PromptKit public
|
||||||
|
types across the Notarius LLM boundary merely to perform inspection.
|
||||||
|
|
||||||
|
`InspectPrompt` is not adopted merely because it exists. It remains available
|
||||||
|
for a later, separately defined module-to-prompt interface preflight if a
|
||||||
|
concrete validation requirement justifies that additional contract.
|
||||||
|
|
||||||
|
### Application And Operator Profile Sources
|
||||||
|
|
||||||
|
Notarius embeds one ordinary PromptKit YAML profile with the stable ID
|
||||||
|
`dnd-extraction`. It is an application fallback registered through
|
||||||
|
`WithFallbackProfileFS`, is owned by the D&D module family, and initially
|
||||||
|
preserves the current effective D&D baseline:
|
||||||
|
|
||||||
|
- backend: PromptKit's built-in `openrouter` backend;
|
||||||
|
- model: `openai/gpt-5.6-luna`;
|
||||||
|
- reasoning effort: unset, allowing OpenAI's backend to apply its default of
|
||||||
|
`medium`;
|
||||||
|
- generation timeout: 240 seconds;
|
||||||
|
- service tier: `flex`; and
|
||||||
|
- no application-selected `temperature`, `max_tokens`, or `top_p`.
|
||||||
|
|
||||||
|
All maintained D&D LLM prompt definitions use `dnd-extraction` as their
|
||||||
|
`default_profile`. The ID communicates workload intent rather than a provider,
|
||||||
|
model, or environment. Changing the embedded fallback is an intentional
|
||||||
|
Notarius execution-policy change and participates in checkpoint identity.
|
||||||
|
|
||||||
|
Effective profile definitions resolve in PromptKit's order:
|
||||||
|
|
||||||
|
1. programmatic in-memory profiles used by tests or explicit consumers;
|
||||||
|
2. the operator source configured by `promptkit.profile_file` or
|
||||||
|
`promptkit.profile_dir`;
|
||||||
|
3. the Notarius application fallback source; and
|
||||||
|
4. PromptKit's embedded built-in catalog.
|
||||||
|
|
||||||
|
Only an absent ID falls through to the next source. A matching profile is a
|
||||||
|
complete definition: fields are not merged with a lower-precedence definition,
|
||||||
|
and a malformed matching operator profile fails rather than silently selecting
|
||||||
|
the application fallback.
|
||||||
|
|
||||||
|
Production, development, and local deployments should normally provide
|
||||||
|
different complete definitions for the same `dnd-extraction` ID. An operator
|
||||||
|
source is optional because the application fallback keeps the maintained D&D
|
||||||
|
workflow usable, but a deployment that needs an intentional model or backend
|
||||||
|
policy should configure its own definition.
|
||||||
|
|
||||||
|
### Domain Ownership And Asset Assembly
|
||||||
|
|
||||||
|
The D&D fallback profile remains under `internal/modules/dnd` and is registered
|
||||||
|
by the D&D registrar, consistent with ADR-0004. Generic LLM plumbing knows how
|
||||||
|
to collect and flatten application fallback profile filesystems but contains no
|
||||||
|
D&D model or policy knowledge.
|
||||||
|
|
||||||
|
The shared asset registry detects invalid roots, unreadable sources, and
|
||||||
|
duplicate flattened paths. PromptKit remains responsible for strict profile
|
||||||
|
YAML parsing, duplicate profile-ID detection, source precedence, and effective
|
||||||
|
target resolution. The same assembled fallback source is supplied to runtime
|
||||||
|
execution and CLI profile inspection.
|
||||||
|
|
||||||
|
### Explicit Module Execution Metadata
|
||||||
|
|
||||||
|
Every registered input, chunk, extract, merge, normalize, and output module
|
||||||
|
declares one required execution class: `deterministic` or `llm_backed`.
|
||||||
|
Validator registrations continue to declare the same distinction through their
|
||||||
|
validator specifications.
|
||||||
|
|
||||||
|
The registered specification is authoritative for configuration resolution.
|
||||||
|
Current production classifications are:
|
||||||
|
|
||||||
|
- the D&D scene chunker, all D&D extractors, and the D&D NPC normalizer are
|
||||||
|
LLM-backed;
|
||||||
|
- the Seriatim input adapter, generic chunker, all current mergers, all other
|
||||||
|
current normalizers, and the JSON output encoder are deterministic; and
|
||||||
|
- current validators retain their declared classifications.
|
||||||
|
|
||||||
|
Missing or unsupported execution metadata is a registration error. Explicitly
|
||||||
|
assigning `llm_profile` to a deterministic module or validator is a pipeline
|
||||||
|
resolution error. The framework does not infer execution class by inspecting
|
||||||
|
domain package names or concrete implementation types at runtime.
|
||||||
|
|
||||||
|
The module specification replaces the chunk runner's special runtime
|
||||||
|
execution-class probe. Effective resolved bindings already express the result:
|
||||||
|
only LLM-backed bindings may retain a non-empty profile.
|
||||||
|
|
||||||
|
### Pipeline-Level Profile Default
|
||||||
|
|
||||||
|
Configuration version 4 gains one optional non-empty pipeline field:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
pipelines:
|
||||||
|
dnd-session:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
|
```
|
||||||
|
|
||||||
|
No configuration-version increment is required because the field is additive
|
||||||
|
and existing files remain valid. An explicitly present blank value is invalid.
|
||||||
|
|
||||||
|
For every selected LLM-backed module and validator, the effective profile uses
|
||||||
|
this precedence:
|
||||||
|
|
||||||
|
1. non-empty run-wide `--llm-profile` override;
|
||||||
|
2. binding-specific `llm_profile`;
|
||||||
|
3. pipeline-level `llm_profile`; and
|
||||||
|
4. the prompt definition's `default_profile`, represented by an empty effective
|
||||||
|
Notarius binding profile.
|
||||||
|
|
||||||
|
The run-wide override and inherited pipeline default never attach to a
|
||||||
|
deterministic binding. Binding-specific exceptions remain available when one
|
||||||
|
operation needs a different cost, latency, quality, backend, or reasoning
|
||||||
|
policy.
|
||||||
|
|
||||||
|
Inheritance is resolved after module and validator selection, including
|
||||||
|
`--only` lane filtering, but before effective-pipeline validation, digest
|
||||||
|
construction, explicit-profile inspection, checkpoint construction,
|
||||||
|
preparation, execution, or provenance capture. Only profiles used by selected
|
||||||
|
LLM-backed bindings are inspected. An unused pipeline default in a pipeline
|
||||||
|
with no selected LLM-backed work does not require an otherwise unused profile
|
||||||
|
to exist.
|
||||||
|
|
||||||
|
The resolved pipeline contains effective binding profiles rather than a second
|
||||||
|
runtime inheritance mechanism. Two pipelines that differ only by spelling the
|
||||||
|
same effective policy once as a pipeline default and once on every LLM-backed
|
||||||
|
binding have the same semantic resolved digest. Changing an effective profile
|
||||||
|
changes the digest and applicable checkpoint identity.
|
||||||
|
|
||||||
|
### Provenance And Checkpoints
|
||||||
|
|
||||||
|
The PromptKit profile-source checkpoint fingerprint covers:
|
||||||
|
|
||||||
|
- the PromptKit v0.5.0 built-in profile catalog identity;
|
||||||
|
- exact application fallback profile asset content; and
|
||||||
|
- exact configured operator profile YAML content, when present.
|
||||||
|
|
||||||
|
The existing local-backend target fingerprint remains separate and continues
|
||||||
|
to exclude scheduling-only concurrency limits. Fingerprints contain hashes and
|
||||||
|
stable markers, not profile contents, filesystem paths, endpoints, credentials,
|
||||||
|
or other secrets.
|
||||||
|
|
||||||
|
Changing the PromptKit version, application fallback, operator profile, or
|
||||||
|
effective pipeline profile makes incompatible LLM checkpoints ineligible for
|
||||||
|
reuse. The dependency upgrade is expected to invalidate checkpoints produced
|
||||||
|
under v0.3.0.
|
||||||
|
|
||||||
|
Successful run manifests continue to record only profiles actually selected by
|
||||||
|
PromptKit, including their effective model, backend, and reasoning metadata.
|
||||||
|
Debug output reports the same effective execution snapshot used for generation.
|
||||||
|
|
||||||
|
### Operator Documentation And Examples
|
||||||
|
|
||||||
|
Canonical documentation clearly distinguishes:
|
||||||
|
|
||||||
|
- Notarius prompt and schema assets embedded in the application;
|
||||||
|
- Notarius application fallback profiles embedded in the application;
|
||||||
|
- PromptKit's own embedded built-in profiles; and
|
||||||
|
- operator profile files on the deployment filesystem.
|
||||||
|
|
||||||
|
The configuration reference owns the pipeline field, profile-source fields,
|
||||||
|
validation rules, and precedence. Operations owns deployment layout, working
|
||||||
|
directory behavior, credentials, and environment-specific profile management.
|
||||||
|
The PromptKit integration document owns the pinned upstream contract and
|
||||||
|
source-precedence boundary. Internal documents describe asset registration,
|
||||||
|
resolution, inspection, prepared execution, fingerprinting, and tests without
|
||||||
|
duplicating user-facing field definitions.
|
||||||
|
|
||||||
|
The maintained examples continue to include only the minimal and complete D&D
|
||||||
|
configurations. They use the stable `dnd-extraction` policy, and one maintained
|
||||||
|
PromptKit profile file under `examples/` demonstrates an operator override.
|
||||||
|
Examples remain secret-free and are validated without live provider calls.
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
- Implementing the separate deterministic prompt-session identity roadmap
|
||||||
|
item.
|
||||||
|
- Changing the default `concurrency.total_llm` value; PromptKit's retained
|
||||||
|
OpenRouter capacity of 16 remains relevant to that separate item.
|
||||||
|
- Adding model evaluation as a deterministic or CI correctness gate.
|
||||||
|
- Automatically selecting production, development, or local environments.
|
||||||
|
Deployment configuration chooses the operator profile source.
|
||||||
|
- Profile inheritance, partial profile merging, or cross-profile aliases.
|
||||||
|
- Exposing PromptKit types to modules, validators, durable output contracts, or
|
||||||
|
public configuration structures.
|
||||||
|
- Adopting `InspectPrompt` without a separately justified prompt-interface
|
||||||
|
validation contract.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- Notarius builds and its offline test suite passes with PromptKit v0.5.0.
|
||||||
|
- Every structured completion executes the exact snapshot used for safe debug
|
||||||
|
prompt details.
|
||||||
|
- Profile preflight uses profile inspection and no synthetic prompt.
|
||||||
|
- The embedded `dnd-extraction` fallback resolves without an operator source,
|
||||||
|
and a matching valid operator profile replaces it completely.
|
||||||
|
- Every production module has explicit, correct execution metadata.
|
||||||
|
- Pipeline, binding, CLI, and prompt-default precedence behaves as defined for
|
||||||
|
modules and validators, while deterministic bindings remain profile-free.
|
||||||
|
- Effective profiles participate in pipeline digests, profile inspection,
|
||||||
|
checkpoint identity, debug records, and run provenance at the appropriate
|
||||||
|
boundaries.
|
||||||
|
- The dependency and application fallback changes invalidate incompatible old
|
||||||
|
checkpoints without exposing profile or credential content.
|
||||||
|
- Canonical documentation and maintained examples accurately describe and
|
||||||
|
exercise the implemented operator workflow.
|
||||||
|
- Default tests remain deterministic, offline, credential-free, and focused on
|
||||||
|
Notarius-owned behavior rather than duplicating PromptKit's upstream suite.
|
||||||
@@ -1,208 +0,0 @@
|
|||||||
# Subprocess Integration Contract
|
|
||||||
|
|
||||||
## Status
|
|
||||||
|
|
||||||
Implemented.
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
Make Notarius straightforward to invoke as a subprocess from an orchestrator
|
|
||||||
such as Narratio. A caller should be able to run a configured pipeline, discover
|
|
||||||
the published output bundle without parsing human prose or scanning a
|
|
||||||
directory, and hand selected structured artifacts to a later stage.
|
|
||||||
|
|
||||||
This work strengthens the public CLI boundary. It does not turn Notarius into a
|
|
||||||
Go library, embed Narratio-specific behavior, or change pipeline execution and
|
|
||||||
artifact semantics.
|
|
||||||
|
|
||||||
## Desired End State
|
|
||||||
|
|
||||||
A subprocess caller can:
|
|
||||||
|
|
||||||
1. validate a Notarius configuration and selected pipeline before execution;
|
|
||||||
2. invoke `notarius run` with explicit input, output-root, session, and
|
|
||||||
reference arguments;
|
|
||||||
3. request one versioned, machine-readable success result on standard output;
|
|
||||||
4. use that result to locate the published output bundle;
|
|
||||||
5. discover normalized lane payloads through the bundle's authoritative
|
|
||||||
`index.json`;
|
|
||||||
6. distinguish process failure from successful partial pipeline outcomes; and
|
|
||||||
7. record Notarius run provenance in its own manifest without depending on
|
|
||||||
internal packages, cache formats, debug formats, or human-readable messages.
|
|
||||||
|
|
||||||
The existing human-oriented command output remains the default for interactive
|
|
||||||
use.
|
|
||||||
|
|
||||||
## Machine-Readable Run Result
|
|
||||||
|
|
||||||
`notarius run` supports `--json`. On success, the flag makes standard output
|
|
||||||
contain exactly one JSON object followed by a newline. No human-oriented status
|
|
||||||
line is mixed into that stream.
|
|
||||||
|
|
||||||
The result uses the schema identity `notarius.run-result.v1` and contains:
|
|
||||||
|
|
||||||
| Field | Presence | Meaning |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `schema_version` | Required | Exactly `notarius.run-result.v1`. |
|
|
||||||
| `run_id` | Required | The Notarius run identifier. |
|
|
||||||
| `pipeline_id` | Required | The effective pipeline identifier. |
|
|
||||||
| `output_directory` | Required | Absolute path to the successfully published output bundle. |
|
|
||||||
| `index_file` | Required for the production JSON output | Logical bundle path `index.json`. |
|
|
||||||
| `normalized_output_count` | Required | Number of final normalized lane outputs returned by the pipeline. |
|
|
||||||
| `rejected_output_count` | Required | Number of recorded rejected outputs. |
|
|
||||||
| `warning_count` | Required | Number of final run warnings returned by the pipeline. |
|
|
||||||
| `validation_status` | Required | The run manifest's final validation status without reinterpretation. |
|
|
||||||
| `debug_directory` | Optional | Absolute debug-bundle path when debug capture was requested and completed. |
|
|
||||||
|
|
||||||
An illustrative successful result is:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"schema_version": "notarius.run-result.v1",
|
|
||||||
"run_id": "run-1770000000000000000-0123456789abcdef0123456789abcdef",
|
|
||||||
"pipeline_id": "dnd-session",
|
|
||||||
"output_directory": "/srv/narratio/runs/session-7/notarius/run-1770000000000000000-0123456789abcdef0123456789abcdef",
|
|
||||||
"index_file": "index.json",
|
|
||||||
"normalized_output_count": 6,
|
|
||||||
"rejected_output_count": 2,
|
|
||||||
"warning_count": 1,
|
|
||||||
"validation_status": "approved"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
The receipt is a discovery and summary document, not a duplicate output
|
|
||||||
envelope. It does not embed lane payloads, rejection entries, warnings, the run
|
|
||||||
manifest, or output-file contents. Consumers use `index_file` and the existing
|
|
||||||
published JSON output contract for those records.
|
|
||||||
|
|
||||||
The result contract must tolerate future additive optional fields. Any
|
|
||||||
incompatible field or semantic change requires a new run-result schema version.
|
|
||||||
|
|
||||||
## Stream, Publication, And Failure Semantics
|
|
||||||
|
|
||||||
Machine-readable output is emitted only after:
|
|
||||||
|
|
||||||
- the pipeline has completed without a framework error;
|
|
||||||
- all logical output files have been successfully published;
|
|
||||||
- requested debug terminal reporting has completed; and
|
|
||||||
- all result fields are known.
|
|
||||||
|
|
||||||
Writing or encoding the machine-readable result is part of successful command
|
|
||||||
completion. Failure to write it produces the existing runtime-failure exit
|
|
||||||
class.
|
|
||||||
|
|
||||||
With `--json`:
|
|
||||||
|
|
||||||
- successful stdout is exclusively the run-result JSON document;
|
|
||||||
- successful warnings remain on stderr under the existing CLI contract;
|
|
||||||
- syntax and runtime errors retain their existing exit statuses and stderr
|
|
||||||
diagnostics;
|
|
||||||
- consumers treat stdout as a valid result only when the process exits with
|
|
||||||
status 0; failures before result writing emit no result, while a failure
|
|
||||||
during the stdout write may leave incomplete bytes that must be ignored; and
|
|
||||||
- human-readable diagnostic wording is not promoted into a machine contract.
|
|
||||||
|
|
||||||
Without `--json`, current interactive stdout and stderr behavior remains
|
|
||||||
unchanged.
|
|
||||||
|
|
||||||
Successful runs may contain rejected outputs or omit some normalized lanes.
|
|
||||||
That remains a valid pipeline outcome. The run result reports counts, while
|
|
||||||
`index.json`, `rejected.json`, and `warnings.json` remain authoritative for
|
|
||||||
details. Notarius will not add a generic `--fail-on-rejection` policy as part
|
|
||||||
of this work.
|
|
||||||
|
|
||||||
## Output Discovery And Consumer Responsibilities
|
|
||||||
|
|
||||||
The production JSON encoder's `index.json` remains the authoritative mapping
|
|
||||||
from lane IDs to published payloads. A subprocess consumer should:
|
|
||||||
|
|
||||||
- resolve `index_file` beneath `output_directory` and reject path escape;
|
|
||||||
- locate expected outputs by `lane_id`, not by guessing filenames;
|
|
||||||
- check each selected descriptor's media type and schema identity;
|
|
||||||
- decode payloads according to their published integration contracts;
|
|
||||||
- decide which lanes are required or optional for its own later stages; and
|
|
||||||
- retain rejection, warning, and manifest files when they are needed for
|
|
||||||
provenance or review.
|
|
||||||
|
|
||||||
For Narratio, required report inputs and partial-success policy remain Narratio
|
|
||||||
stage configuration and orchestration concerns. Notarius does not acquire
|
|
||||||
knowledge of Narratio stages, manifests, workspace layout, publication policy,
|
|
||||||
or report formats.
|
|
||||||
|
|
||||||
## Invocation Guidance
|
|
||||||
|
|
||||||
The consumer documentation recommends that subprocess callers:
|
|
||||||
|
|
||||||
- use `notarius config validate --pipeline` as an optional preflight;
|
|
||||||
- pass explicit absolute paths for the input, configuration, output root, and
|
|
||||||
CLI-supplied references;
|
|
||||||
- use a stable, non-secret prompt session identifier when useful for provider
|
|
||||||
routing or caching;
|
|
||||||
- capture stdout and stderr separately;
|
|
||||||
- supply credentials through the configured environment mechanism rather than
|
|
||||||
command arguments or generated configuration containing secret values;
|
|
||||||
- place output, cache, debug, and subprocess logs under intentional
|
|
||||||
sensitivity and retention policies; and
|
|
||||||
- treat the Notarius manifest and run-result receipt as provenance while
|
|
||||||
leaving the caller's own manifest authoritative for its stage lifecycle.
|
|
||||||
|
|
||||||
Notarius configuration remains owned by Notarius. An orchestrator may select a
|
|
||||||
configuration and pass supported operational overrides, but should not
|
|
||||||
duplicate the complete Notarius configuration schema.
|
|
||||||
|
|
||||||
## Documentation End State
|
|
||||||
|
|
||||||
- `docs/cli.md` owns `run --json`, stream behavior, and exit semantics;
|
|
||||||
- a new `docs/integrations/run-result.md` owns the versioned run-result wire
|
|
||||||
contract and compatibility policy;
|
|
||||||
- `docs/integrations/json-output.md` remains the sole owner of output-bundle
|
|
||||||
discovery and lane publication;
|
|
||||||
- a new `docs/consumers/subprocess.md` provides the task-oriented invocation and
|
|
||||||
consumption workflow; and
|
|
||||||
- `docs/internal/cli.md` describes how the CLI constructs and emits the result
|
|
||||||
only after successful publication.
|
|
||||||
|
|
||||||
Other documents should link to these owners instead of repeating volatile
|
|
||||||
fields or command details.
|
|
||||||
|
|
||||||
## Acceptance Criteria
|
|
||||||
|
|
||||||
- An ordinary successful `run` retains its existing human-readable output.
|
|
||||||
- A successful `run --json` emits one valid `notarius.run-result.v1` document
|
|
||||||
and no human prose on stdout.
|
|
||||||
- Relative configured or overridden output and debug roots are reported as
|
|
||||||
absolute bundle paths.
|
|
||||||
- The receipt identifies the production JSON bundle entry point without
|
|
||||||
copying its lane descriptors or payloads.
|
|
||||||
- Warning-bearing and rejection-bearing runs remain successful and report
|
|
||||||
accurate counts.
|
|
||||||
- Syntax, configuration, provider, pipeline, publication, debug, and result
|
|
||||||
writing failures retain the correct nonzero exit class. Consumers are
|
|
||||||
explicitly required to ignore stdout from a nonzero invocation.
|
|
||||||
- The implementation does not expose internal Go types or couple generic CLI
|
|
||||||
code to D&D or Narratio concepts.
|
|
||||||
- Public and internal documentation assigns each new contract to one canonical
|
|
||||||
owner.
|
|
||||||
- Offline behavioral tests protect the structured-output contract, default
|
|
||||||
human behavior, absolute path reporting, stream separation, and failure to
|
|
||||||
serialize or write the success result without duplicating lower-level output
|
|
||||||
encoder tests.
|
|
||||||
|
|
||||||
## Out Of Scope
|
|
||||||
|
|
||||||
The following may be useful later but are not prerequisites for the Narratio
|
|
||||||
integration:
|
|
||||||
|
|
||||||
- a result-file flag in addition to machine-readable stdout;
|
|
||||||
- a JSON failure envelope or stable machine-readable error taxonomy;
|
|
||||||
- caller-supplied Notarius run IDs or exact output-bundle paths;
|
|
||||||
- a generic `--fail-on-rejection` or required-lane CLI policy;
|
|
||||||
- a public Go client package or importable Narratio adapter;
|
|
||||||
- Narratio stage, configuration, manifest, or report-generation changes;
|
|
||||||
- `notarius version --json`;
|
|
||||||
- installable or queryable artifact JSON Schemas;
|
|
||||||
- signal-aware CLI contexts and graceful SIGINT or SIGTERM handling;
|
|
||||||
- packaged release artifacts and a broader application-versioning policy.
|
|
||||||
|
|
||||||
These items should be promoted only in response to a demonstrated integration
|
|
||||||
need rather than bundled into the initial subprocess contract.
|
|
||||||
@@ -1,4 +1,6 @@
|
|||||||
version: 3
|
version: 4
|
||||||
|
promptkit:
|
||||||
|
profile_file: ./examples/profiles/dnd-extraction.yml
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 2
|
total_llm: 2
|
||||||
stage_workers:
|
stage_workers:
|
||||||
@@ -16,6 +18,7 @@ debug:
|
|||||||
directory: ./notarius-debug
|
directory: ./notarius-debug
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input: seriatim
|
||||||
# Stable campaign context is shared by every module that accepts these slots.
|
# Stable campaign context is shared by every module that accepts these slots.
|
||||||
references:
|
references:
|
||||||
@@ -28,6 +31,15 @@ pipelines:
|
|||||||
module: json
|
module: json
|
||||||
options:
|
options:
|
||||||
include_chunk_map: true
|
include_chunk_map: true
|
||||||
|
evidence_context:
|
||||||
|
enabled: true
|
||||||
|
window_units: 3
|
||||||
|
lanes:
|
||||||
|
- item-events
|
||||||
|
- npcs
|
||||||
|
- spells
|
||||||
|
- combat-turns
|
||||||
|
- npc-interactions
|
||||||
steps:
|
steps:
|
||||||
# Establish session-wide reference artifacts alongside independent item events.
|
# Establish session-wide reference artifacts alongside independent item events.
|
||||||
- id: describe-session
|
- id: describe-session
|
||||||
@@ -45,7 +57,6 @@ pipelines:
|
|||||||
merge: appendorder
|
merge: appendorder
|
||||||
normalize:
|
normalize:
|
||||||
module: dnd/npcs
|
module: dnd/npcs
|
||||||
llm_profile: gemini-2-flash
|
|
||||||
retries: 2
|
retries: 2
|
||||||
scene-descriptions:
|
scene-descriptions:
|
||||||
extract:
|
extract:
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
version: 3
|
version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input: seriatim
|
||||||
artifacts:
|
artifacts:
|
||||||
spells:
|
spells:
|
||||||
|
|||||||
5
examples/profiles/dnd-extraction.yml
Normal file
5
examples/profiles/dnd-extraction.yml
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
id: dnd-extraction
|
||||||
|
backend: openrouter
|
||||||
|
model: openai/gpt-5.6-luna
|
||||||
|
timeout_seconds: 240
|
||||||
|
service_tier: flex
|
||||||
2
go.mod
2
go.mod
@@ -3,7 +3,7 @@ module gitea.maximumdirect.net/eric/notarius
|
|||||||
go 1.25.5
|
go 1.25.5
|
||||||
|
|
||||||
require (
|
require (
|
||||||
gitea.maximumdirect.net/eric/scriptorium v0.11.1
|
gitea.maximumdirect.net/eric/promptkit v0.5.0
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
||||||
gopkg.in/yaml.v3 v3.0.1
|
gopkg.in/yaml.v3 v3.0.1
|
||||||
)
|
)
|
||||||
|
|||||||
8
go.sum
8
go.sum
@@ -1,13 +1,9 @@
|
|||||||
gitea.maximumdirect.net/eric/scriptorium v0.11.0 h1:rjvbt9FTaWHxYlHq7QlUzmMVUt3QdbTmeCkmH81N//o=
|
gitea.maximumdirect.net/eric/promptkit v0.5.0 h1:jnpazLyyNhWrB2xzwwtUkNUfktkTdkENTwuSPnKiYrc=
|
||||||
gitea.maximumdirect.net/eric/scriptorium v0.11.0/go.mod h1:FQ5lEuNxmrQyNgIomkpZdxvfTC0jWjbXYuq3tbJWF64=
|
gitea.maximumdirect.net/eric/promptkit v0.5.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||||
gitea.maximumdirect.net/eric/scriptorium v0.11.1 h1:zBKtB3+fP8FcHGI8DJD99CiTL6crAGitBhWtE+xYJHc=
|
|
||||||
gitea.maximumdirect.net/eric/scriptorium v0.11.1/go.mod h1:FQ5lEuNxmrQyNgIomkpZdxvfTC0jWjbXYuq3tbJWF64=
|
|
||||||
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
||||||
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
|
||||||
golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
|
|
||||||
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
|
||||||
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
||||||
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||||
|
|||||||
@@ -215,6 +215,7 @@ func assembledSpellPipeline(t *testing.T, options assembledSpellPipelineOptions)
|
|||||||
if err := pipeline.RegisterExtractor[dnd.SpellList](components.registries.Extractors, pipeline.ModuleSpec{
|
if err := pipeline.RegisterExtractor[dnd.SpellList](components.registries.Extractors, pipeline.ModuleSpec{
|
||||||
Key: assembledSpellExtractorKey,
|
Key: assembledSpellExtractorKey,
|
||||||
Stage: pipeline.StageExtract,
|
Stage: pipeline.StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunks", "source.transcript"},
|
Requires: []string{"chunks", "source.transcript"},
|
||||||
Provides: []string{"dnd.spell_casts"},
|
Provides: []string{"dnd.spell_casts"},
|
||||||
ArtifactKind: dnd.SpellListKind,
|
ArtifactKind: dnd.SpellListKind,
|
||||||
|
|||||||
@@ -24,6 +24,7 @@ func newProductionComponents() (productionComponents, error) {
|
|||||||
Inputs: pipeline.NewInputAdapterRegistry(),
|
Inputs: pipeline.NewInputAdapterRegistry(),
|
||||||
Chunkers: pipeline.NewChunkerRegistry(),
|
Chunkers: pipeline.NewChunkerRegistry(),
|
||||||
ArtifactCodecs: pipeline.NewArtifactCodecRegistry(),
|
ArtifactCodecs: pipeline.NewArtifactCodecRegistry(),
|
||||||
|
ArtifactEvidence: pipeline.NewArtifactEvidenceRegistry(),
|
||||||
Extractors: pipeline.NewExtractorRegistry(),
|
Extractors: pipeline.NewExtractorRegistry(),
|
||||||
Mergers: pipeline.NewMergerRegistry(),
|
Mergers: pipeline.NewMergerRegistry(),
|
||||||
Normalizers: pipeline.NewNormalizerRegistry(),
|
Normalizers: pipeline.NewNormalizerRegistry(),
|
||||||
@@ -91,6 +92,7 @@ func catalogFromRegistries(registries pipeline.Registries) pipeline.ModuleCatalo
|
|||||||
Inputs: registries.Inputs,
|
Inputs: registries.Inputs,
|
||||||
Chunkers: registries.Chunkers,
|
Chunkers: registries.Chunkers,
|
||||||
ArtifactCodecs: registries.ArtifactCodecs,
|
ArtifactCodecs: registries.ArtifactCodecs,
|
||||||
|
ArtifactEvidence: registries.ArtifactEvidence,
|
||||||
Extractors: registries.Extractors,
|
Extractors: registries.Extractors,
|
||||||
Mergers: registries.Mergers,
|
Mergers: registries.Mergers,
|
||||||
Normalizers: registries.Normalizers,
|
Normalizers: registries.Normalizers,
|
||||||
@@ -105,6 +107,7 @@ func registriesFromCatalog(catalog pipeline.ModuleCatalog) pipeline.Registries {
|
|||||||
Inputs: catalog.Inputs,
|
Inputs: catalog.Inputs,
|
||||||
Chunkers: catalog.Chunkers,
|
Chunkers: catalog.Chunkers,
|
||||||
ArtifactCodecs: catalog.ArtifactCodecs,
|
ArtifactCodecs: catalog.ArtifactCodecs,
|
||||||
|
ArtifactEvidence: catalog.ArtifactEvidence,
|
||||||
Extractors: catalog.Extractors,
|
Extractors: catalog.Extractors,
|
||||||
Mergers: catalog.Mergers,
|
Mergers: catalog.Mergers,
|
||||||
Normalizers: catalog.Normalizers,
|
Normalizers: catalog.Normalizers,
|
||||||
@@ -118,6 +121,7 @@ func isEmptyCatalog(catalog pipeline.ModuleCatalog) bool {
|
|||||||
return catalog.Inputs == nil &&
|
return catalog.Inputs == nil &&
|
||||||
catalog.Chunkers == nil &&
|
catalog.Chunkers == nil &&
|
||||||
catalog.ArtifactCodecs == nil &&
|
catalog.ArtifactCodecs == nil &&
|
||||||
|
catalog.ArtifactEvidence == nil &&
|
||||||
catalog.Extractors == nil &&
|
catalog.Extractors == nil &&
|
||||||
catalog.Mergers == nil &&
|
catalog.Mergers == nil &&
|
||||||
catalog.Normalizers == nil &&
|
catalog.Normalizers == nil &&
|
||||||
@@ -130,6 +134,7 @@ func isEmptyRegistries(registries pipeline.Registries) bool {
|
|||||||
return registries.Inputs == nil &&
|
return registries.Inputs == nil &&
|
||||||
registries.Chunkers == nil &&
|
registries.Chunkers == nil &&
|
||||||
registries.ArtifactCodecs == nil &&
|
registries.ArtifactCodecs == nil &&
|
||||||
|
registries.ArtifactEvidence == nil &&
|
||||||
registries.Extractors == nil &&
|
registries.Extractors == nil &&
|
||||||
registries.Mergers == nil &&
|
registries.Mergers == nil &&
|
||||||
registries.Normalizers == nil &&
|
registries.Normalizers == nil &&
|
||||||
@@ -138,24 +143,13 @@ func isEmptyRegistries(registries pipeline.Registries) bool {
|
|||||||
registries.Outputs == nil
|
registries.Outputs == nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
|
||||||
if err := ctx.Err(); err != nil {
|
|
||||||
return nil, nil, err
|
|
||||||
}
|
|
||||||
assets, err := productionPromptAssets()
|
|
||||||
if err != nil {
|
|
||||||
return nil, nil, err
|
|
||||||
}
|
|
||||||
return buildProductionLLMClient(ctx, cfg, profileID, assets)
|
|
||||||
}
|
|
||||||
|
|
||||||
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
|
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
|
||||||
return func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
return func(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
return buildProductionLLMClient(ctx, cfg, profileID, assets)
|
return buildProductionLLMClient(ctx, cfg, profileID, overrides, assets)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
if err := ctx.Err(); err != nil {
|
if err := ctx.Err(); err != nil {
|
||||||
return nil, nil, err
|
return nil, nil, err
|
||||||
}
|
}
|
||||||
@@ -163,14 +157,16 @@ func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID
|
|||||||
return nil, nil, fmt.Errorf("production asset registry must not be nil")
|
return nil, nil, fmt.Errorf("production asset registry must not be nil")
|
||||||
}
|
}
|
||||||
recorder := llm.NewLLMProfileRecorder()
|
recorder := llm.NewLLMProfileRecorder()
|
||||||
client, err := llm.NewScriptoriumClient(llm.ScriptoriumClientConfig{
|
client, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{
|
||||||
ProfileDir: cfg.Scriptorium.ProfileDir,
|
ProfileDir: cfg.PromptKit.ProfileDir,
|
||||||
ProfileFile: cfg.Scriptorium.ProfileFile,
|
ProfileFile: cfg.PromptKit.ProfileFile,
|
||||||
|
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
|
||||||
Assets: assets,
|
Assets: assets,
|
||||||
Recorder: recorder,
|
Recorder: recorder,
|
||||||
|
ReasoningEffort: overrides.ReasoningEffort,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, nil, fmt.Errorf("create Scriptorium-backed LLM client: %w", err)
|
return nil, nil, fmt.Errorf("create PromptKit-backed LLM client: %w", err)
|
||||||
}
|
}
|
||||||
scheduler, err := llm.NewScheduler(cfg.Concurrency.TotalLLM)
|
scheduler, err := llm.NewScheduler(cfg.Concurrency.TotalLLM)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -154,6 +154,23 @@ func TestConfigValidateResolvesPipelineAndChecksSelection(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestConfigValidatePipelineDefaultProfileIsOffline(t *testing.T) {
|
||||||
|
configPath := writeCommandConfigContent(t, `version: 4
|
||||||
|
pipelines:
|
||||||
|
demo:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
|
input: seriatim
|
||||||
|
artifacts:
|
||||||
|
spells:
|
||||||
|
extract: dnd/spells
|
||||||
|
`)
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
code := RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "demo"}, &stdout, &stderr, Options{})
|
||||||
|
if code != 0 || !strings.Contains(stdout.String(), "valid for pipeline \"demo\"") || stderr.Len() != 0 {
|
||||||
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestPipelinesListSortsNormalizedIDsInTextAndJSON(t *testing.T) {
|
func TestPipelinesListSortsNormalizedIDsInTextAndJSON(t *testing.T) {
|
||||||
configPath := writeCommandConfig(t, " zeta ", "alpha")
|
configPath := writeCommandConfig(t, " zeta ", "alpha")
|
||||||
options := commandContractOptions(t)
|
options := commandContractOptions(t)
|
||||||
@@ -214,13 +231,13 @@ func commandContractOptionsWithLookup(t *testing.T, lookup func(string) (string,
|
|||||||
|
|
||||||
func writeCommandConfig(t *testing.T, firstID, secondID string) string {
|
func writeCommandConfig(t *testing.T, firstID, secondID string) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
content := fmt.Sprintf("version: 3\npipelines:\n %q:\n input: seriatim\n %q:\n input: seriatim\n", firstID, secondID)
|
content := fmt.Sprintf("version: 4\npipelines:\n %q:\n input: seriatim\n %q:\n input: seriatim\n", firstID, secondID)
|
||||||
return writeCommandConfigContent(t, content)
|
return writeCommandConfigContent(t, content)
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeResolvableCommandConfig(t *testing.T) string {
|
func writeResolvableCommandConfig(t *testing.T) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
return writeCommandConfigContent(t, `version: 3
|
return writeCommandConfigContent(t, `version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
demo:
|
demo:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
|
|||||||
@@ -6,8 +6,10 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"reflect"
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
@@ -28,7 +30,7 @@ func TestProductionSceneDescriptionWorkflow(t *testing.T) {
|
|||||||
Output: pipeline.Binding("json"),
|
Output: pipeline.Binding("json"),
|
||||||
Artifacts: map[string]pipeline.ArtifactLaneProfile{
|
Artifacts: map[string]pipeline.ArtifactLaneProfile{
|
||||||
"scene-descriptions": {
|
"scene-descriptions": {
|
||||||
Extract: pipeline.Binding(sceneextract.Key),
|
Extract: pipeline.ModuleBinding{Module: sceneextract.Key, LLMProfile: "scene-description-profile"},
|
||||||
Normalize: pipeline.Binding(scenenormalize.Key),
|
Normalize: pipeline.Binding(scenenormalize.Key),
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
@@ -45,7 +47,8 @@ func TestProductionSceneDescriptionWorkflow(t *testing.T) {
|
|||||||
t.Fatalf("resolved references = %#v / %#v, want no generated or required references", lane.ExtractReferences, lane.NormalizeReferences)
|
t.Fatalf("resolved references = %#v / %#v, want no generated or required references", lane.ExtractReferences, lane.NormalizeReferences)
|
||||||
}
|
}
|
||||||
|
|
||||||
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, components.registries, pipeline.ModuleDependencies{LLM: sceneDescriptionLLM{}})
|
llmClient := &sceneDescriptionLLM{}
|
||||||
|
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, components.registries, pipeline.ModuleDependencies{LLM: llmClient})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Prepare() error = %v", err)
|
t.Fatalf("Prepare() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -60,6 +63,14 @@ func TestProductionSceneDescriptionWorkflow(t *testing.T) {
|
|||||||
if output.Manifest.ValidationStatus != "approved" || len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 1 {
|
if output.Manifest.ValidationStatus != "approved" || len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 1 {
|
||||||
t.Fatalf("run output = %#v, want one approved normalized artifact", output)
|
t.Fatalf("run output = %#v, want one approved normalized artifact", output)
|
||||||
}
|
}
|
||||||
|
wantProfiles := []artifacts.LLMProfileManifest{{
|
||||||
|
ID: "scene-description-profile",
|
||||||
|
Provider: "promptkit",
|
||||||
|
Model: "deterministic",
|
||||||
|
}}
|
||||||
|
if !reflect.DeepEqual(output.Manifest.LLMProfiles, wantProfiles) {
|
||||||
|
t.Fatalf("manifest LLM profiles = %#v, want %#v", output.Manifest.LLMProfiles, wantProfiles)
|
||||||
|
}
|
||||||
normalizedOutput := output.NormalizeOutputs[0]
|
normalizedOutput := output.NormalizeOutputs[0]
|
||||||
if normalizedOutput.NormalizerKey != scenenormalize.Key || normalizedOutput.Artifact.Kind != dnd.SceneDescriptionListKind || normalizedOutput.Artifact.Schema.ID != scenecodec.SchemaID || normalizedOutput.Artifact.Schema.Name != scenecodec.SchemaName || normalizedOutput.Artifact.Schema.Version != scenecodec.SchemaVersion {
|
if normalizedOutput.NormalizerKey != scenenormalize.Key || normalizedOutput.Artifact.Kind != dnd.SceneDescriptionListKind || normalizedOutput.Artifact.Schema.ID != scenecodec.SchemaID || normalizedOutput.Artifact.Schema.Name != scenecodec.SchemaName || normalizedOutput.Artifact.Schema.Version != scenecodec.SchemaVersion {
|
||||||
t.Fatalf("normalized output = %#v, want registered durable scene-description schema", normalizedOutput)
|
t.Fatalf("normalized output = %#v, want registered durable scene-description schema", normalizedOutput)
|
||||||
@@ -85,9 +96,12 @@ func TestProductionSceneDescriptionWorkflow(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
type sceneDescriptionLLM struct{}
|
type sceneDescriptionLLM struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
profile *artifacts.LLMProfileManifest
|
||||||
|
}
|
||||||
|
|
||||||
func (sceneDescriptionLLM) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
func (client *sceneDescriptionLLM) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
if err := ctx.Err(); err != nil {
|
if err := ctx.Err(); err != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, err
|
return contracts.StructuredCompletionResponse{}, err
|
||||||
}
|
}
|
||||||
@@ -107,5 +121,27 @@ func (sceneDescriptionLLM) CompleteStructured(ctx context.Context, req contracts
|
|||||||
if err := json.Unmarshal([]byte(content), out); err != nil {
|
if err := json.Unmarshal([]byte(content), out); err != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate structured response: %w", err)
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate structured response: %w", err)
|
||||||
}
|
}
|
||||||
return contracts.StructuredCompletionResponse{Content: []byte(content), Provider: "test", Model: "deterministic", ProfileID: req.ProfileID}, nil
|
profile := artifacts.LLMProfileManifest{
|
||||||
|
ID: req.ProfileID,
|
||||||
|
Provider: "promptkit",
|
||||||
|
Model: "deterministic",
|
||||||
|
}
|
||||||
|
client.mu.Lock()
|
||||||
|
client.profile = &profile
|
||||||
|
client.mu.Unlock()
|
||||||
|
return contracts.StructuredCompletionResponse{
|
||||||
|
Content: []byte(content),
|
||||||
|
Provider: profile.Provider,
|
||||||
|
Model: profile.Model,
|
||||||
|
ProfileID: profile.ID,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (client *sceneDescriptionLLM) LLMProfileManifests() []artifacts.LLMProfileManifest {
|
||||||
|
client.mu.Lock()
|
||||||
|
defer client.mu.Unlock()
|
||||||
|
if client.profile == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return []artifacts.LLMProfileManifest{*client.profile}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -94,6 +94,37 @@ func TestMaintainedConfigurationExampleSet(t *testing.T) {
|
|||||||
if got := strings.Join(names, ","); got != "dnd-complete.config.yml,dnd-minimal.config.yml" {
|
if got := strings.Join(names, ","); got != "dnd-complete.config.yml,dnd-minimal.config.yml" {
|
||||||
t.Fatalf("maintained configuration examples = %q, want only the minimal and complete D&D examples", got)
|
t.Fatalf("maintained configuration examples = %q, want only the minimal and complete D&D examples", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
profileEntries, err := os.ReadDir(repositoryPath("examples", "profiles"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
names = names[:0]
|
||||||
|
for _, entry := range profileEntries {
|
||||||
|
if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".yml") {
|
||||||
|
names = append(names, entry.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
if got := strings.Join(names, ","); got != "dnd-extraction.yml" {
|
||||||
|
t.Fatalf("maintained operator profiles = %q, want dnd-extraction.yml", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMaintainedExamplesValidateEffectiveProfilesOffline(t *testing.T) {
|
||||||
|
t.Chdir(repositoryPath())
|
||||||
|
t.Setenv("OPENROUTER_API_KEY", "")
|
||||||
|
for _, example := range maintainedExampleFiles(t) {
|
||||||
|
t.Run(example.name, func(t *testing.T) {
|
||||||
|
var stdout, stderr strings.Builder
|
||||||
|
code := RunWithOptions([]string{
|
||||||
|
"config", "validate", "--config", example.path, "--pipeline", "dnd-session",
|
||||||
|
}, &stdout, &stderr, Options{})
|
||||||
|
if code != 0 || stderr.Len() != 0 || !strings.Contains(stdout.String(), `valid for pipeline "dnd-session"`) {
|
||||||
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func exampleStepLaneIDs(resolved pipeline.ResolvedPipeline) []string {
|
func exampleStepLaneIDs(resolved pipeline.ResolvedPipeline) []string {
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ func TestOversizedNPCRegistryFailsBeforeRuntimeAndCheckpointConstruction(t *test
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
checkpointRoot := filepath.Join(t.TempDir(), "checkpoints")
|
checkpointRoot := filepath.Join(t.TempDir(), "checkpoints")
|
||||||
content := fmt.Sprintf(`version: 3
|
content := fmt.Sprintf(`version: 4
|
||||||
cache:
|
cache:
|
||||||
chunk_plans:
|
chunk_plans:
|
||||||
mode: bypass
|
mode: bypass
|
||||||
@@ -51,7 +51,7 @@ pipelines:
|
|||||||
options := Options{
|
options := Options{
|
||||||
Catalog: catalogFromRegistries(components.registries),
|
Catalog: catalogFromRegistries(components.registries),
|
||||||
Registries: components.registries,
|
Registries: components.registries,
|
||||||
LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
llmConstructed = true
|
llmConstructed = true
|
||||||
return nil, nil, errors.New("LLM client must not be constructed")
|
return nil, nil, errors.New("LLM client must not be constructed")
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io/fs"
|
"io/fs"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"reflect"
|
"reflect"
|
||||||
@@ -13,13 +15,16 @@ import (
|
|||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
|
"testing/fstest"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkmap"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkmap"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
|
||||||
@@ -33,6 +38,7 @@ import (
|
|||||||
itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents"
|
itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents"
|
||||||
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
|
func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
|
||||||
@@ -160,9 +166,38 @@ func TestProductionCatalogCoversMaintainedConfigurations(t *testing.T) {
|
|||||||
assertProductionContains(t, "production prompt assets", assetNames, requiredAssets)
|
assertProductionContains(t, "production prompt assets", assetNames, requiredAssets)
|
||||||
|
|
||||||
catalog := catalogFromRegistries(registries)
|
catalog := catalogFromRegistries(registries)
|
||||||
|
for _, test := range []struct {
|
||||||
|
stage pipeline.ModuleStage
|
||||||
|
key string
|
||||||
|
want contracts.ExecutionClass
|
||||||
|
}{
|
||||||
|
{stage: pipeline.StageInput, key: "seriatim", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageChunk, key: "generic", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageChunk, key: "dnd/scenes", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/spells", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/npcs", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/combat-turns", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/item-events", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/npc-interactions", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageExtract, key: "dnd/scene-descriptions", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageMerge, key: "appendorder", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "noop", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/spells", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/npcs", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/combat-turns", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/item-events", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/npc-interactions", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageNormalize, key: "dnd/scene-descriptions", want: contracts.ExecutionClassDeterministic},
|
||||||
|
{stage: pipeline.StageOutput, key: "json", want: contracts.ExecutionClassDeterministic},
|
||||||
|
} {
|
||||||
|
got, ok := catalog.ExecutionClass(test.stage, test.key)
|
||||||
|
if !ok || got != test.want {
|
||||||
|
t.Fatalf("production execution class for %s/%s = %q, %t; want %q, true", test.stage, test.key, got, ok, test.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
converted := registriesFromCatalog(catalog)
|
converted := registriesFromCatalog(catalog)
|
||||||
if converted.ArtifactCodecs != registries.ArtifactCodecs || converted.ValidatorChains != registries.ValidatorChains {
|
if converted.ArtifactCodecs != registries.ArtifactCodecs || converted.ArtifactEvidence != registries.ArtifactEvidence || converted.ValidatorChains != registries.ValidatorChains {
|
||||||
t.Fatal("catalog/registry conversion did not preserve codec and validator-chain registries")
|
t.Fatal("catalog/registry conversion did not preserve artifact and validator registries")
|
||||||
}
|
}
|
||||||
codecSpec, ok := catalog.ArtifactCodecs.Spec(dnd.SpellListKind)
|
codecSpec, ok := catalog.ArtifactCodecs.Spec(dnd.SpellListKind)
|
||||||
if !ok || codecSpec.Kind != dnd.SpellListKind || codecSpec.Schema.ID != spellcodec.SchemaID {
|
if !ok || codecSpec.Kind != dnd.SpellListKind || codecSpec.Schema.ID != spellcodec.SchemaID {
|
||||||
@@ -195,6 +230,76 @@ func TestDefaultCLICompositionValidatesRepresentativeConfiguration(t *testing.T)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestProductionAssetsResolveDNDExtractionProfile(t *testing.T) {
|
||||||
|
components := productionTestComponents(t)
|
||||||
|
newEngine := func(profileFile string) (*promptkit.Engine, error) {
|
||||||
|
t.Helper()
|
||||||
|
options, err := components.assets.PromptKitOptions()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if profileFile != "" {
|
||||||
|
options = append(options, promptkit.WithProfileFile(profileFile))
|
||||||
|
}
|
||||||
|
return promptkit.NewEngine(promptkit.Config{}, options...)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Run("fallback", func(t *testing.T) {
|
||||||
|
engine, err := newEngine("")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
inspection, err := engine.InspectProfile(context.Background(), "dnd-extraction")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("InspectProfile() error = %v, want fallback profile", err)
|
||||||
|
}
|
||||||
|
params := inspection.EffectiveModelParams
|
||||||
|
if params.BackendID != "openrouter" || params.Model != "openai/gpt-5.6-luna" || params.TimeoutSeconds != 240 || params.ServiceTier != "flex" {
|
||||||
|
t.Fatalf("fallback profile parameters = %#v", params)
|
||||||
|
}
|
||||||
|
if params.ReasoningEffort != "" || params.Temperature != 0 || params.MaxTokens != 0 || params.TopP != 0 {
|
||||||
|
t.Fatalf("fallback profile selected optional provider controls: %#v", params)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("valid operator profile wins", func(t *testing.T) {
|
||||||
|
profilePath := filepath.Join(t.TempDir(), "profiles.yaml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte(`id: dnd-extraction
|
||||||
|
endpoint: http://operator.example.test/v1
|
||||||
|
model: operator-model
|
||||||
|
timeout_seconds: 75
|
||||||
|
`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
engine, err := newEngine(profilePath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
inspection, err := engine.InspectProfile(context.Background(), "dnd-extraction")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("InspectProfile() error = %v, want operator profile", err)
|
||||||
|
}
|
||||||
|
params := inspection.EffectiveModelParams
|
||||||
|
if params.BackendID != "" || params.Endpoint != "http://operator.example.test/v1" || params.Model != "operator-model" || params.TimeoutSeconds != 75 || params.ServiceTier != "" {
|
||||||
|
t.Fatalf("operator profile parameters = %#v, want complete replacement", params)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("invalid operator profile does not fall through", func(t *testing.T) {
|
||||||
|
profilePath := filepath.Join(t.TempDir(), "profiles.yaml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte("id: dnd-extraction\nendpoint: http://operator.example.test/v1\nmodel: operator-model\nunknown: value\n"), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
engine, err := newEngine(profilePath)
|
||||||
|
if err == nil {
|
||||||
|
_, err = engine.InspectProfile(context.Background(), "dnd-extraction")
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("operator profile error = nil, want failure instead of fallback")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) {
|
func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) {
|
||||||
components := productionTestComponents(t)
|
components := productionTestComponents(t)
|
||||||
cfg := config.Default()
|
cfg := config.Default()
|
||||||
@@ -321,7 +426,7 @@ func TestProductionSpellNormalizerRejectsInvalidCatalogReferencesBeforeExecution
|
|||||||
options := Options{
|
options := Options{
|
||||||
Catalog: catalogFromRegistries(components.registries),
|
Catalog: catalogFromRegistries(components.registries),
|
||||||
Registries: components.registries,
|
Registries: components.registries,
|
||||||
LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
llmConstructed = true
|
llmConstructed = true
|
||||||
return nil, nil, errors.New("LLM client must not be constructed")
|
return nil, nil, errors.New("LLM client must not be constructed")
|
||||||
},
|
},
|
||||||
@@ -369,18 +474,9 @@ func setNormalizeSpellCatalogSource(t *testing.T, resolved *pipeline.ResolvedPip
|
|||||||
resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings = bindings
|
resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings = bindings
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) {
|
func TestProductionLLMClientFactoryBuildsOfflineRuntime(t *testing.T) {
|
||||||
components := productionTestComponents(t)
|
components := productionTestComponents(t)
|
||||||
factories := []struct {
|
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
|
||||||
name string
|
|
||||||
factory LLMClientFactory
|
|
||||||
}{
|
|
||||||
{name: "default production assets", factory: productionLLMClientFactory},
|
|
||||||
{name: "provided production assets", factory: productionLLMClientFactoryWithAssets(components.assets)},
|
|
||||||
}
|
|
||||||
for _, tt := range factories {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
client, manifests, err := tt.factory(context.Background(), config.Default(), "test-profile")
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("build production LLM runtime: %v", err)
|
t.Fatalf("build production LLM runtime: %v", err)
|
||||||
}
|
}
|
||||||
@@ -390,10 +486,138 @@ func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) {
|
|||||||
if len(manifests) != 0 {
|
if len(manifests) != 0 {
|
||||||
t.Fatalf("eager profile manifests = %#v, want none", manifests)
|
t.Fatalf("eager profile manifests = %#v, want none", manifests)
|
||||||
}
|
}
|
||||||
|
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("production LLM client %T does not provide one profile-source checkpoint fingerprint", client)
|
||||||
|
}
|
||||||
|
fingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
|
||||||
|
if err != nil || len(fingerprints) != 1 {
|
||||||
|
t.Fatalf("production LLM checkpoint fingerprints = %#v, error = %v, want one profile-source identity", fingerprints, err)
|
||||||
|
}
|
||||||
if _, ok := client.(contracts.LLMProfileManifestProvider); !ok {
|
if _, ok := client.(contracts.LLMProfileManifestProvider); !ok {
|
||||||
t.Fatalf("production LLM client %T does not provide profile manifests", client)
|
t.Fatalf("production LLM client %T does not provide profile manifests", client)
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNormalizeOptionsSharesProductionProfileAssetsWithDefaultRuntime(t *testing.T) {
|
||||||
|
opts, err := normalizeOptions(Options{
|
||||||
|
Catalog: pipeline.ModuleCatalog{Inputs: pipeline.NewInputAdapterRegistry()},
|
||||||
})
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if opts.promptKitAssets == nil || opts.LLMClientFactory == nil {
|
||||||
|
t.Fatalf("normalized options = %#v, want shared profile assets and default runtime factory", opts)
|
||||||
|
}
|
||||||
|
if err := validateExplicitPromptKitProfiles(context.Background(), config.Default(), []string{"dnd-extraction"}, opts.promptKitAssets); err != nil {
|
||||||
|
t.Fatalf("inspect application fallback profile: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
client, _, err := opts.LLMClientFactory(context.Background(), config.Default(), "dnd-extraction", LLMRuntimeOverrides{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("build default runtime: %v", err)
|
||||||
|
}
|
||||||
|
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("default runtime client %T does not provide checkpoint fingerprints", client)
|
||||||
|
}
|
||||||
|
runtimeFingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
directClient, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{Assets: opts.promptKitAssets})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
inspectionFingerprints, err := directClient.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(runtimeFingerprints, inspectionFingerprints) {
|
||||||
|
t.Fatalf("runtime profile fingerprints = %#v, inspection profile fingerprints = %#v", runtimeFingerprints, inspectionFingerprints)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProductionLLMClientFactoryUsesConfiguredLocalBackend(t *testing.T) {
|
||||||
|
var providerCalls atomic.Int32
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
providerCalls.Add(1)
|
||||||
|
if r.URL.Path != "/v1/chat/completions" {
|
||||||
|
t.Errorf("provider path = %q, want /v1/chat/completions", r.URL.Path)
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_, _ = w.Write([]byte(`{
|
||||||
|
"choices": [{"message": {"role": "assistant", "content": "{\"ok\":true}"}}],
|
||||||
|
"usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7}
|
||||||
|
}`))
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte(`id: local-profile
|
||||||
|
backend: local
|
||||||
|
model: local-model
|
||||||
|
`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
assets := llm.NewAssetRegistry()
|
||||||
|
if err := assets.RegisterPromptFS(fstest.MapFS{
|
||||||
|
"production.local.yaml": {Data: []byte(`id: production.local
|
||||||
|
version: "v1"
|
||||||
|
inputs:
|
||||||
|
- name: transcript
|
||||||
|
required: true
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: '{{ input "transcript" }}'
|
||||||
|
output:
|
||||||
|
format: json
|
||||||
|
validation_mode: json
|
||||||
|
`)},
|
||||||
|
}, "."); err != nil {
|
||||||
|
t.Fatalf("register prompt assets: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg := config.Default()
|
||||||
|
cfg.PromptKit.ProfileFile = profilePath
|
||||||
|
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: server.URL + "/v1",
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
client, manifests, err := productionLLMClientFactoryWithAssets(assets)(
|
||||||
|
context.Background(),
|
||||||
|
cfg,
|
||||||
|
"local-profile",
|
||||||
|
LLMRuntimeOverrides{},
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("build production LLM runtime: %v", err)
|
||||||
|
}
|
||||||
|
if len(manifests) != 0 {
|
||||||
|
t.Fatalf("eager profile manifests = %#v, want none", manifests)
|
||||||
|
}
|
||||||
|
|
||||||
|
var out map[string]any
|
||||||
|
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||||
|
PromptID: "production.local",
|
||||||
|
ProfileID: "local-profile",
|
||||||
|
Inputs: contracts.LLMInputSet{
|
||||||
|
"transcript": contracts.NewLLMInputMaterial("transcript", "text/plain", []byte("local request"), "", ""),
|
||||||
|
},
|
||||||
|
}, &out)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if providerCalls.Load() != 1 {
|
||||||
|
t.Fatalf("provider calls = %d, want 1", providerCalls.Load())
|
||||||
|
}
|
||||||
|
provider, ok := client.(contracts.LLMProfileManifestProvider)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("production client %T does not provide profile manifests", client)
|
||||||
|
}
|
||||||
|
recorded := provider.LLMProfileManifests()
|
||||||
|
if len(recorded) != 1 || recorded[0].BackendID != promptkit.BackendLocal {
|
||||||
|
t.Fatalf("production profile manifests = %#v, want local backend", recorded)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -401,14 +625,15 @@ func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
|
|||||||
t.Run("canceled context", func(t *testing.T) {
|
t.Run("canceled context", func(t *testing.T) {
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
cancel()
|
cancel()
|
||||||
client, manifests, err := productionLLMClientFactory(ctx, config.Default(), "test-profile")
|
components := productionTestComponents(t)
|
||||||
|
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(ctx, config.Default(), "test-profile", LLMRuntimeOverrides{})
|
||||||
if !errors.Is(err, context.Canceled) || client != nil || len(manifests) != 0 {
|
if !errors.Is(err, context.Canceled) || client != nil || len(manifests) != 0 {
|
||||||
t.Fatalf("client=%T manifests=%#v error=%v, want canceled construction", client, manifests, err)
|
t.Fatalf("client=%T manifests=%#v error=%v, want canceled construction", client, manifests, err)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
t.Run("nil assets", func(t *testing.T) {
|
t.Run("nil assets", func(t *testing.T) {
|
||||||
client, manifests, err := productionLLMClientFactoryWithAssets(nil)(context.Background(), config.Default(), "test-profile")
|
client, manifests, err := productionLLMClientFactoryWithAssets(nil)(context.Background(), config.Default(), "test-profile", LLMRuntimeOverrides{})
|
||||||
if err == nil || !strings.Contains(err.Error(), "asset registry must not be nil") || client != nil || len(manifests) != 0 {
|
if err == nil || !strings.Contains(err.Error(), "asset registry must not be nil") || client != nil || len(manifests) != 0 {
|
||||||
t.Fatalf("client=%T manifests=%#v error=%v, want nil-assets failure", client, manifests, err)
|
t.Fatalf("client=%T manifests=%#v error=%v, want nil-assets failure", client, manifests, err)
|
||||||
}
|
}
|
||||||
@@ -418,7 +643,7 @@ func TestProductionLLMClientFactoriesRejectInvalidConstruction(t *testing.T) {
|
|||||||
components := productionTestComponents(t)
|
components := productionTestComponents(t)
|
||||||
cfg := config.Default()
|
cfg := config.Default()
|
||||||
cfg.Concurrency.TotalLLM = 0
|
cfg.Concurrency.TotalLLM = 0
|
||||||
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), cfg, "test-profile")
|
client, manifests, err := productionLLMClientFactoryWithAssets(components.assets)(context.Background(), cfg, "test-profile", LLMRuntimeOverrides{})
|
||||||
if err == nil || !strings.Contains(err.Error(), "create LLM scheduler") || !strings.Contains(err.Error(), "greater than zero") || client != nil || len(manifests) != 0 {
|
if err == nil || !strings.Contains(err.Error(), "create LLM scheduler") || !strings.Contains(err.Error(), "greater than zero") || client != nil || len(manifests) != 0 {
|
||||||
t.Fatalf("client=%T manifests=%#v error=%v, want scheduler-construction failure", client, manifests, err)
|
t.Fatalf("client=%T manifests=%#v error=%v, want scheduler-construction failure", client, manifests, err)
|
||||||
}
|
}
|
||||||
@@ -583,7 +808,7 @@ func maintainedExampleFiles(t *testing.T) []maintainedExample {
|
|||||||
|
|
||||||
func productionSpellCatalogContractConfig(t *testing.T) string {
|
func productionSpellCatalogContractConfig(t *testing.T) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
return fmt.Sprintf(`version: 3
|
return fmt.Sprintf(`version: 4
|
||||||
cache:
|
cache:
|
||||||
chunk_plans:
|
chunk_plans:
|
||||||
mode: bypass
|
mode: bypass
|
||||||
@@ -651,6 +876,7 @@ func productionOptionsFromComponents(components productionComponents) Options {
|
|||||||
Catalog: catalogFromRegistries(components.registries),
|
Catalog: catalogFromRegistries(components.registries),
|
||||||
Registries: components.registries,
|
Registries: components.registries,
|
||||||
LookupEnv: emptyLookup,
|
LookupEnv: emptyLookup,
|
||||||
|
promptKitAssets: components.assets,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -673,14 +899,14 @@ func productionRunOptions(t *testing.T, fake *productionFakeLLMClient) Options {
|
|||||||
options.Now = func() time.Time { return time.Unix(1700000000, 0).UTC() }
|
options.Now = func() time.Time { return time.Unix(1700000000, 0).UTC() }
|
||||||
options.RunIDGenerator = func(time.Time) (string, error) { return productionRunID, nil }
|
options.RunIDGenerator = func(time.Time) (string, error) { return productionRunID, nil }
|
||||||
options.UserCacheDir = func() (string, error) { return "", errors.New("user cache must not be used") }
|
options.UserCacheDir = func() (string, error) { return "", errors.New("user cache must not be used") }
|
||||||
options.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
options.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
return fake, nil, nil
|
return fake, nil, nil
|
||||||
}
|
}
|
||||||
return options
|
return options
|
||||||
}
|
}
|
||||||
|
|
||||||
func productionRunConfig(outputRoot, chunkModule string) string {
|
func productionRunConfig(outputRoot, chunkModule string) string {
|
||||||
return fmt.Sprintf(`version: 3
|
return fmt.Sprintf(`version: 4
|
||||||
output:
|
output:
|
||||||
directory: %q
|
directory: %q
|
||||||
cache:
|
cache:
|
||||||
|
|||||||
46
internal/cli/promptkit_profiles.go
Normal file
46
internal/cli/promptkit_profiles.go
Normal file
@@ -0,0 +1,46 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
|
)
|
||||||
|
|
||||||
|
func validateExplicitPromptKitProfiles(ctx context.Context, cfg config.Config, profileIDs []string, assets *llm.AssetRegistry) error {
|
||||||
|
if len(profileIDs) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
inspector, err := llm.NewPromptKitProfileInspector(llm.PromptKitProfileInspectorConfig{
|
||||||
|
Source: promptKitProfileSourceConfig(cfg),
|
||||||
|
Assets: assets,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("load PromptKit profiles: %w", err)
|
||||||
|
}
|
||||||
|
for _, profileID := range profileIDs {
|
||||||
|
if _, err := inspector.InspectProfile(ctx, profileID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptKitProfileSourceConfig(cfg config.Config) llm.PromptKitProfileSourceConfig {
|
||||||
|
return llm.PromptKitProfileSourceConfig{
|
||||||
|
ProfileDir: cfg.PromptKit.ProfileDir,
|
||||||
|
ProfileFile: cfg.PromptKit.ProfileFile,
|
||||||
|
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mapPromptKitLocalBackend(cfg *config.PromptKitLocalBackendConfig) *llm.PromptKitLocalBackendConfig {
|
||||||
|
if cfg == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return &llm.PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: cfg.Endpoint,
|
||||||
|
ConcurrencyLimit: cfg.ConcurrencyLimit,
|
||||||
|
}
|
||||||
|
}
|
||||||
159
internal/cli/promptkit_profiles_test.go
Normal file
159
internal/cli/promptkit_profiles_test.go
Normal file
@@ -0,0 +1,159 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"testing/fstest"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestExplicitPromptKitProfileValidationInspectsProfilesWithoutGeneration(t *testing.T) {
|
||||||
|
var providerCalls atomic.Int32
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
|
||||||
|
providerCalls.Add(1)
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
writeProfile := func(t *testing.T, name, content string) string {
|
||||||
|
t.Helper()
|
||||||
|
profilePath := filepath.Join(t.TempDir(), name+".yaml")
|
||||||
|
if err := os.WriteFile(profilePath, []byte(content), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return profilePath
|
||||||
|
}
|
||||||
|
localProfile := "id: local-profile\nbackend: local\nmodel: local-model\n"
|
||||||
|
credentialProfile := `id: credential-profile
|
||||||
|
endpoint: ` + server.URL + `/v1
|
||||||
|
model: credential-model
|
||||||
|
api_key_env: NOTARIUS_PROMPTKIT_PROFILE_INSPECTION_TEST_KEY
|
||||||
|
`
|
||||||
|
t.Setenv("NOTARIUS_PROMPTKIT_PROFILE_INSPECTION_TEST_KEY", "")
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
profilePath string
|
||||||
|
profileID string
|
||||||
|
profileDir bool
|
||||||
|
localBackend bool
|
||||||
|
canceled bool
|
||||||
|
wantErr []string
|
||||||
|
rejectErr []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "configured local backend",
|
||||||
|
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||||
|
profileID: "local-profile",
|
||||||
|
profileDir: true,
|
||||||
|
localBackend: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "missing local backend registration",
|
||||||
|
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||||
|
profileID: "local-profile",
|
||||||
|
wantErr: []string{`PromptKit profile "local-profile" is invalid or unreadable`},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "absent profile",
|
||||||
|
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||||
|
profileID: "absent-profile",
|
||||||
|
localBackend: true,
|
||||||
|
wantErr: []string{`PromptKit profile "absent-profile" is not configured`},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "malformed profile",
|
||||||
|
profilePath: writeProfile(t, "malformed-profile", "id: malformed-profile\nbackend: [\n"),
|
||||||
|
profileID: "malformed-profile",
|
||||||
|
wantErr: []string{`PromptKit profile "malformed-profile" is invalid or unreadable`},
|
||||||
|
rejectErr: []string{"malformed-profile.yaml", "backend: ["},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "invalid profile source",
|
||||||
|
profilePath: filepath.Join(t.TempDir(), "missing-profile.yaml"),
|
||||||
|
profileID: "missing-profile",
|
||||||
|
wantErr: []string{"load PromptKit profiles", "profile configuration is invalid or unreadable"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "credential environment intentionally unset",
|
||||||
|
profilePath: writeProfile(t, "credential-profile", credentialProfile),
|
||||||
|
profileID: "credential-profile",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "canceled inspection",
|
||||||
|
profilePath: writeProfile(t, "local-profile", localProfile),
|
||||||
|
profileID: "local-profile",
|
||||||
|
localBackend: true,
|
||||||
|
canceled: true,
|
||||||
|
wantErr: []string{"context canceled"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := config.Default()
|
||||||
|
if tt.profileDir {
|
||||||
|
cfg.PromptKit.ProfileDir = filepath.Dir(tt.profilePath)
|
||||||
|
} else {
|
||||||
|
cfg.PromptKit.ProfileFile = tt.profilePath
|
||||||
|
}
|
||||||
|
if tt.localBackend {
|
||||||
|
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: server.URL + "/v1",
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
if tt.canceled {
|
||||||
|
var cancel context.CancelFunc
|
||||||
|
ctx, cancel = context.WithCancel(ctx)
|
||||||
|
cancel()
|
||||||
|
}
|
||||||
|
err := validateExplicitPromptKitProfiles(ctx, cfg, []string{tt.profileID}, nil)
|
||||||
|
if len(tt.wantErr) == 0 {
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("validateExplicitPromptKitProfiles() error = nil, want failure")
|
||||||
|
}
|
||||||
|
if tt.canceled && !errors.Is(err, context.Canceled) {
|
||||||
|
t.Fatalf("canceled inspection error = %v, want context canceled", err)
|
||||||
|
}
|
||||||
|
for _, want := range tt.wantErr {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("validation error = %q, want %q", err, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, rejected := range append(tt.rejectErr, tt.profilePath) {
|
||||||
|
if rejected != "" && strings.Contains(err.Error(), rejected) {
|
||||||
|
t.Fatalf("validation error = %q, must not expose %q", err, rejected)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if providerCalls.Load() != 0 {
|
||||||
|
t.Fatalf("provider calls during profile inspection = %d, want 0", providerCalls.Load())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExplicitPromptKitProfileValidationUsesFallbackAssets(t *testing.T) {
|
||||||
|
assets := llm.NewAssetRegistry()
|
||||||
|
if err := assets.RegisterFallbackProfileFS(fstest.MapFS{
|
||||||
|
"profiles/fallback.yaml": {Data: []byte("id: fallback-profile\nendpoint: http://promptkit.test/v1\nmodel: fallback-model\n")},
|
||||||
|
}, "profiles"); err != nil {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if err := validateExplicitPromptKitProfiles(context.Background(), config.Default(), []string{"fallback-profile"}, assets); err != nil {
|
||||||
|
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -116,7 +116,7 @@ func (h *recomputeTestHarness) options() Options {
|
|||||||
for _, key := range []string{"test/extract/producer", "test/extract/unrelated", "test/extract/middle", "test/extract/dependent"} {
|
for _, key := range []string{"test/extract/producer", "test/extract/unrelated", "test/extract/middle", "test/extract/dependent"} {
|
||||||
moduleKey := key
|
moduleKey := key
|
||||||
spec := pipeline.ModuleSpec{
|
spec := pipeline.ModuleSpec{
|
||||||
Key: moduleKey, Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind,
|
Key: moduleKey, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind,
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}},
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterExtractor(opts.Registries.Extractors, spec, func() (contracts.Extractor[stateTestArtifact], error) {
|
if err := pipeline.RegisterExtractor(opts.Registries.Extractors, spec, func() (contracts.Extractor[stateTestArtifact], error) {
|
||||||
@@ -125,7 +125,7 @@ func (h *recomputeTestHarness) options() Options {
|
|||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if err := opts.Registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/recompute-output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
if err := opts.Registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/recompute-output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
||||||
return recomputeTestOutput{}, nil
|
return recomputeTestOutput{}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
@@ -194,7 +194,7 @@ func (recomputeTestOutput) Encode(_ context.Context, req contracts.OutputRequest
|
|||||||
func newRecomputeTestRoots(t *testing.T) stateTestRoots {
|
func newRecomputeTestRoots(t *testing.T) stateTestRoots {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
config := fmt.Sprintf(`version: 3
|
config := fmt.Sprintf(`version: 4
|
||||||
output:
|
output:
|
||||||
directory: %q
|
directory: %q
|
||||||
cache:
|
cache:
|
||||||
|
|||||||
@@ -217,7 +217,7 @@ func TestReferenceMaterializationSeparatesCLIAndConfigPathOrigins(t *testing.T)
|
|||||||
workingDir := t.TempDir()
|
workingDir := t.TempDir()
|
||||||
cfg := referenceContractConfig()
|
cfg := referenceContractConfig()
|
||||||
configPath := filepath.Join(configDir, "config.yml")
|
configPath := filepath.Join(configDir, "config.yml")
|
||||||
if err := os.WriteFile(configPath, []byte("version: 3\n"), 0o600); err != nil {
|
if err := os.WriteFile(configPath, []byte("version: 4\n"), 0o600); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(filepath.Join(configDir, "required.txt"), []byte("config reference"), 0o600); err != nil {
|
if err := os.WriteFile(filepath.Join(configDir, "required.txt"), []byte("config reference"), 0o600); err != nil {
|
||||||
@@ -364,21 +364,21 @@ func referenceContractCatalog(t *testing.T, includeBetaMerger, includeBetaNormal
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
register(registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "reference/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }))
|
register(registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "reference/input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }))
|
||||||
register(registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/chunk", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-slot"}, {Name: "required-chunk", Required: true}}}, func() (contracts.Chunker, error) { return stateTestChunker{}, nil }))
|
register(registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-slot"}, {Name: "required-chunk", Required: true}}}, func() (contracts.Chunker, error) { return stateTestChunker{}, nil }))
|
||||||
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecA{}))
|
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecA{}))
|
||||||
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecB{}))
|
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecB{}))
|
||||||
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-alpha", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-alpha", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
||||||
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-beta", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-beta", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
|
||||||
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
||||||
if includeBetaMerger {
|
if includeBetaMerger {
|
||||||
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
|
||||||
}
|
}
|
||||||
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
||||||
if includeBetaNormalizer {
|
if includeBetaNormalizer {
|
||||||
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
|
||||||
}
|
}
|
||||||
register(registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { return stateTestOutput{}, nil }))
|
register(registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { return stateTestOutput{}, nil }))
|
||||||
return catalogFromRegistries(registries)
|
return catalogFromRegistries(registries)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -23,13 +23,14 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkplan"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkplan"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
frameworkdebug "gitea.maximumdirect.net/eric/notarius/internal/framework/debug"
|
frameworkdebug "gitea.maximumdirect.net/eric/notarius/internal/framework/debug"
|
||||||
|
frameworkllm "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
)
|
)
|
||||||
|
|
||||||
const defaultConfigPath = "/usr/local/etc/notarius/config.yml"
|
const defaultConfigPath = "/usr/local/etc/notarius/config.yml"
|
||||||
const usage = `Usage:
|
const usage = `Usage:
|
||||||
notarius help
|
notarius help
|
||||||
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--output-dir path] [--json] [--chunk_cache auto|bypass|refresh] [--resume] [--recompute-step step-id] [--debug [--debug-dir path]] [--only lane-a,lane-b] [--session-id id] [--reference selector=path] [--without-reference selector]
|
notarius run <pipeline-id> --input path/to/source.json [--json] [flags]
|
||||||
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
|
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
|
||||||
notarius pipelines list --config path/to/config.yml [--json]
|
notarius pipelines list --config path/to/config.yml [--json]
|
||||||
`
|
`
|
||||||
@@ -45,9 +46,14 @@ type Options struct {
|
|||||||
ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory
|
ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory
|
||||||
DebugRecorderFactory func(string) (pipeline.DebugRecorder, error)
|
DebugRecorderFactory func(string) (pipeline.DebugRecorder, error)
|
||||||
DebugTerminalFactory func(*debugbundle.SummaryWriter) DebugTerminalWriter
|
DebugTerminalFactory func(*debugbundle.SummaryWriter) DebugTerminalWriter
|
||||||
|
promptKitAssets *frameworkllm.AssetRegistry
|
||||||
}
|
}
|
||||||
|
|
||||||
type LLMClientFactory func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error)
|
type LLMRuntimeOverrides struct {
|
||||||
|
ReasoningEffort *string
|
||||||
|
}
|
||||||
|
|
||||||
|
type LLMClientFactory func(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error)
|
||||||
|
|
||||||
// Run executes the command-line interface and returns a process exit code.
|
// Run executes the command-line interface and returns a process exit code.
|
||||||
func Run(args []string, stdout, stderr io.Writer) int {
|
func Run(args []string, stdout, stderr io.Writer) int {
|
||||||
@@ -116,12 +122,17 @@ func normalizeOptions(opts Options) (Options, error) {
|
|||||||
}
|
}
|
||||||
opts.Registries = components.registries
|
opts.Registries = components.registries
|
||||||
opts.Catalog = catalogFromRegistries(components.registries)
|
opts.Catalog = catalogFromRegistries(components.registries)
|
||||||
if opts.LLMClientFactory == nil {
|
opts.promptKitAssets = components.assets
|
||||||
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(components.assets)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if opts.LLMClientFactory == nil {
|
if opts.LLMClientFactory == nil {
|
||||||
opts.LLMClientFactory = productionLLMClientFactory
|
if opts.promptKitAssets == nil {
|
||||||
|
assets, err := productionPromptAssets()
|
||||||
|
if err != nil {
|
||||||
|
return Options{}, err
|
||||||
|
}
|
||||||
|
opts.promptKitAssets = assets
|
||||||
|
}
|
||||||
|
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(opts.promptKitAssets)
|
||||||
}
|
}
|
||||||
return opts, nil
|
return opts, nil
|
||||||
}
|
}
|
||||||
@@ -137,13 +148,16 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
debug := fs.Bool("debug", false, "write a debug bundle")
|
debug := fs.Bool("debug", false, "write a debug bundle")
|
||||||
debugDir := fs.String("debug-dir", "", "debug bundle directory")
|
debugDir := fs.String("debug-dir", "", "debug bundle directory")
|
||||||
llmProfile := fs.String("llm-profile", "", "LLM profile override")
|
llmProfile := fs.String("llm-profile", "", "LLM profile override")
|
||||||
|
reasoningEffort := singleValueFlag{name: "--reasoning-effort"}
|
||||||
|
clearReasoningEffort := fs.Bool("clear-reasoning-effort", false, "clear the LLM profile reasoning effort")
|
||||||
resume := fs.Bool("resume", false, "reuse compatible recorded checkpoints")
|
resume := fs.Bool("resume", false, "reuse compatible recorded checkpoints")
|
||||||
recomputeStep := singleValueFlag{}
|
recomputeStep := singleValueFlag{name: "--recompute-step"}
|
||||||
chunkCache := chunkCacheFlag{}
|
chunkCache := chunkCacheFlag{}
|
||||||
sessionID := sessionIDFlag{}
|
sessionID := sessionIDFlag{}
|
||||||
referenceFlags := stringListFlag{}
|
referenceFlags := stringListFlag{}
|
||||||
withoutReferenceFlags := stringListFlag{}
|
withoutReferenceFlags := stringListFlag{}
|
||||||
fs.Var(&sessionID, "session-id", "prompt session identifier")
|
fs.Var(&sessionID, "session-id", "prompt session identifier")
|
||||||
|
fs.Var(&reasoningEffort, "reasoning-effort", "reasoning effort override")
|
||||||
fs.Var(&chunkCache, "chunk_cache", "chunk plan cache mode: auto, bypass, or refresh")
|
fs.Var(&chunkCache, "chunk_cache", "chunk plan cache mode: auto, bypass, or refresh")
|
||||||
fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, merge.slot=path, lane.slot=path, lane.extract.slot=path, lane.merge.slot=path, or lane.normalize.slot=path")
|
fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, merge.slot=path, lane.slot=path, lane.extract.slot=path, lane.merge.slot=path, or lane.normalize.slot=path")
|
||||||
fs.Var(&withoutReferenceFlags, "without-reference", "unbind a reference, using the same selector forms as --reference")
|
fs.Var(&withoutReferenceFlags, "without-reference", "unbind a reference, using the same selector forms as --reference")
|
||||||
@@ -189,6 +203,22 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
fmt.Fprintln(stderr, "notarius: --session-id must not be empty")
|
fmt.Fprintln(stderr, "notarius: --session-id must not be empty")
|
||||||
return 2
|
return 2
|
||||||
}
|
}
|
||||||
|
if reasoningEffort.set && *clearReasoningEffort {
|
||||||
|
fmt.Fprintln(stderr, "notarius: --reasoning-effort cannot be combined with --clear-reasoning-effort")
|
||||||
|
return 2
|
||||||
|
}
|
||||||
|
if reasoningEffort.set && strings.TrimSpace(reasoningEffort.value) == "" {
|
||||||
|
fmt.Fprintln(stderr, "notarius: --reasoning-effort must not be empty")
|
||||||
|
return 2
|
||||||
|
}
|
||||||
|
runtimeOverrides := LLMRuntimeOverrides{}
|
||||||
|
if reasoningEffort.set {
|
||||||
|
value := strings.TrimSpace(reasoningEffort.value)
|
||||||
|
runtimeOverrides.ReasoningEffort = &value
|
||||||
|
} else if *clearReasoningEffort {
|
||||||
|
value := ""
|
||||||
|
runtimeOverrides.ReasoningEffort = &value
|
||||||
|
}
|
||||||
only, err := parseOnly(*onlyRaw)
|
only, err := parseOnly(*onlyRaw)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||||
@@ -286,6 +316,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
ConfigSource: configSource(*configPath),
|
ConfigSource: configSource(*configPath),
|
||||||
OnlyLanes: append([]string(nil), only...),
|
OnlyLanes: append([]string(nil), only...),
|
||||||
ChunkCacheOverride: chunkCache.explicitValue(),
|
ChunkCacheOverride: chunkCache.explicitValue(),
|
||||||
|
ReasoningEffortOverride: runtimeOverrides.ReasoningEffort,
|
||||||
Resume: *resume,
|
Resume: *resume,
|
||||||
RecomputeStep: strings.TrimSpace(recomputeStep.value),
|
RecomputeStep: strings.TrimSpace(recomputeStep.value),
|
||||||
RunID: runID,
|
RunID: runID,
|
||||||
@@ -315,7 +346,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||||
}
|
}
|
||||||
profileIDs := effectiveLLMProfileIDs(effective.ResolvedPipeline)
|
profileIDs := effectiveLLMProfileIDs(effective.ResolvedPipeline)
|
||||||
if err := validateExplicitScriptoriumProfiles(context.Background(), effective.Config, profileIDs); err != nil {
|
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, profileIDs, opts.promptKitAssets); err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||||
}
|
}
|
||||||
workingDir, err := os.Getwd()
|
workingDir, err := os.Getwd()
|
||||||
@@ -363,10 +394,17 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
if len(profileIDs) == 1 {
|
if len(profileIDs) == 1 {
|
||||||
factoryProfileID = profileIDs[0]
|
factoryProfileID = profileIDs[0]
|
||||||
}
|
}
|
||||||
llmClient, llmProfiles, err := opts.LLMClientFactory(ctx, effective.Config, factoryProfileID)
|
llmClient, llmProfiles, err := opts.LLMClientFactory(ctx, effective.Config, factoryProfileID, runtimeOverrides)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("create LLM client for profile %q: %w", factoryProfileID, err))
|
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("create LLM client for profile %q: %w", factoryProfileID, err))
|
||||||
}
|
}
|
||||||
|
var llmFingerprints []checkpoint.Fingerprint
|
||||||
|
if effective.Config.Cache.Checkpoints.Enabled {
|
||||||
|
llmFingerprints, err = llmCheckpointFingerprints(llmClient)
|
||||||
|
if err != nil {
|
||||||
|
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("prepare LLM checkpoint identity: %w", err))
|
||||||
|
}
|
||||||
|
}
|
||||||
llmClient = pipeline.WithDebugLLMRecording(llmClient, debugRecorder)
|
llmClient = pipeline.WithDebugLLMRecording(llmClient, debugRecorder)
|
||||||
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, registries, pipeline.ModuleDependencies{LLM: llmClient})
|
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, registries, pipeline.ModuleDependencies{LLM: llmClient})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -380,7 +418,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||||
}
|
}
|
||||||
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(effective.Config.Cache.Checkpoints, opts, effective.ResolvedPipeline, prepared.CheckpointFingerprints(), rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), *resume)
|
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(effective.Config.Cache.Checkpoints, opts, effective.ResolvedPipeline, prepared.CheckpointFingerprints(), llmFingerprints, rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), runtimeOverrides, *resume)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||||
}
|
}
|
||||||
@@ -481,11 +519,13 @@ func checkpointHandlersForRun(
|
|||||||
opts Options,
|
opts Options,
|
||||||
resolved pipeline.ResolvedPipeline,
|
resolved pipeline.ResolvedPipeline,
|
||||||
componentFingerprints []pipeline.CheckpointFingerprint,
|
componentFingerprints []pipeline.CheckpointFingerprint,
|
||||||
|
llmFingerprints []checkpoint.Fingerprint,
|
||||||
rawInput []byte,
|
rawInput []byte,
|
||||||
only []string,
|
only []string,
|
||||||
llmProfiles []artifacts.LLMProfileManifest,
|
llmProfiles []artifacts.LLMProfileManifest,
|
||||||
llmProfileOverride string,
|
llmProfileOverride string,
|
||||||
sessionID string,
|
sessionID string,
|
||||||
|
runtimeOverrides LLMRuntimeOverrides,
|
||||||
resume bool,
|
resume bool,
|
||||||
) (pipeline.CheckpointRecorder, pipeline.CheckpointLoader, error) {
|
) (pipeline.CheckpointRecorder, pipeline.CheckpointLoader, error) {
|
||||||
if !settings.Enabled {
|
if !settings.Enabled {
|
||||||
@@ -499,9 +539,13 @@ func checkpointHandlersForRun(
|
|||||||
InputKey: resolved.Input.Module,
|
InputKey: resolved.Input.Module,
|
||||||
RawInputDigest: rawInputDigest(rawInput),
|
RawInputDigest: rawInputDigest(rawInput),
|
||||||
SelectedLanes: only,
|
SelectedLanes: only,
|
||||||
RuntimeOverrides: runtimeOverrideFingerprints(llmProfileOverride, sessionID),
|
RuntimeOverrides: runtimeOverrideFingerprints(llmProfileOverride, sessionID, runtimeOverrides),
|
||||||
References: pipeline.ReferenceProvenance(resolved),
|
References: pipeline.ReferenceProvenance(resolved),
|
||||||
ProvenanceFingerprints: append(llmProfileFingerprints(llmProfiles), checkpointIdentityFingerprints(componentFingerprints)...),
|
ProvenanceFingerprints: combineCheckpointFingerprints(
|
||||||
|
llmProfileFingerprints(llmProfiles),
|
||||||
|
llmFingerprints,
|
||||||
|
checkpointIdentityFingerprints(componentFingerprints),
|
||||||
|
),
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, nil, fmt.Errorf("create checkpoint identity: %w", err)
|
return nil, nil, fmt.Errorf("create checkpoint identity: %w", err)
|
||||||
@@ -527,6 +571,30 @@ func checkpointHandlersForRun(
|
|||||||
return recorder, loader, nil
|
return recorder, loader, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func llmCheckpointFingerprints(client contracts.StructuredLLMClient) ([]checkpoint.Fingerprint, error) {
|
||||||
|
provider, ok := client.(frameworkllm.CheckpointFingerprintProvider)
|
||||||
|
if !ok {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
values, err := provider.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out := make([]checkpoint.Fingerprint, 0, len(values))
|
||||||
|
for _, value := range values {
|
||||||
|
out = append(out, checkpoint.Fingerprint{Name: value.Name, Value: value.Value})
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func combineCheckpointFingerprints(sources ...[]checkpoint.Fingerprint) []checkpoint.Fingerprint {
|
||||||
|
var out []checkpoint.Fingerprint
|
||||||
|
for _, source := range sources {
|
||||||
|
out = append(out, source...)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
func recomputePolicy(resolved pipeline.ResolvedPipeline, requestedStep string) (pipeline.CheckpointExecutionPolicy, error) {
|
func recomputePolicy(resolved pipeline.ResolvedPipeline, requestedStep string) (pipeline.CheckpointExecutionPolicy, error) {
|
||||||
requestedStep = strings.TrimSpace(requestedStep)
|
requestedStep = strings.TrimSpace(requestedStep)
|
||||||
if requestedStep == "" {
|
if requestedStep == "" {
|
||||||
@@ -632,7 +700,7 @@ func rawInputDigest(data []byte) string {
|
|||||||
return "sha256:" + hex.EncodeToString(sum[:])
|
return "sha256:" + hex.EncodeToString(sum[:])
|
||||||
}
|
}
|
||||||
|
|
||||||
func runtimeOverrideFingerprints(llmProfileOverride string, sessionID string) []checkpoint.Fingerprint {
|
func runtimeOverrideFingerprints(llmProfileOverride string, sessionID string, runtimeOverrides LLMRuntimeOverrides) []checkpoint.Fingerprint {
|
||||||
var values []checkpoint.Fingerprint
|
var values []checkpoint.Fingerprint
|
||||||
if strings.TrimSpace(llmProfileOverride) != "" {
|
if strings.TrimSpace(llmProfileOverride) != "" {
|
||||||
values = append(values, checkpoint.Fingerprint{Name: "llm_profile_override", Value: strings.TrimSpace(llmProfileOverride)})
|
values = append(values, checkpoint.Fingerprint{Name: "llm_profile_override", Value: strings.TrimSpace(llmProfileOverride)})
|
||||||
@@ -640,6 +708,13 @@ func runtimeOverrideFingerprints(llmProfileOverride string, sessionID string) []
|
|||||||
if strings.TrimSpace(sessionID) != "" {
|
if strings.TrimSpace(sessionID) != "" {
|
||||||
values = append(values, checkpoint.Fingerprint{Name: "session_id", Value: strings.TrimSpace(sessionID)})
|
values = append(values, checkpoint.Fingerprint{Name: "session_id", Value: strings.TrimSpace(sessionID)})
|
||||||
}
|
}
|
||||||
|
if runtimeOverrides.ReasoningEffort != nil {
|
||||||
|
value := strings.TrimSpace(*runtimeOverrides.ReasoningEffort)
|
||||||
|
if value == "" {
|
||||||
|
value = "<cleared>"
|
||||||
|
}
|
||||||
|
values = append(values, checkpoint.Fingerprint{Name: "reasoning_effort_override", Value: value})
|
||||||
|
}
|
||||||
return values
|
return values
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -796,7 +871,7 @@ func reorderRunArgs(args []string) []string {
|
|||||||
|
|
||||||
func runFlagTakesValue(arg string) bool {
|
func runFlagTakesValue(arg string) bool {
|
||||||
switch arg {
|
switch arg {
|
||||||
case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--chunk_cache", "--reference", "--without-reference", "--recompute-step":
|
case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--reasoning-effort", "--chunk_cache", "--reference", "--without-reference", "--recompute-step":
|
||||||
return true
|
return true
|
||||||
default:
|
default:
|
||||||
return false
|
return false
|
||||||
@@ -856,11 +931,11 @@ func chunkPlanStoreForRun(cfg config.ChunkPlanCacheConfig, opts Options) (pipeli
|
|||||||
|
|
||||||
func validateRunFlagValues(args []string) error {
|
func validateRunFlagValues(args []string) error {
|
||||||
for i, arg := range args {
|
for i, arg := range args {
|
||||||
if arg != "--session-id" {
|
if arg != "--session-id" && arg != "--reasoning-effort" {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if i+1 >= len(args) || strings.HasPrefix(args[i+1], "-") {
|
if i+1 >= len(args) || strings.HasPrefix(args[i+1], "-") {
|
||||||
return fmt.Errorf("flag needs an argument: --session-id")
|
return fmt.Errorf("flag needs an argument: %s", arg)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
@@ -883,12 +958,23 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
|
|||||||
seen[id] = struct{}{}
|
seen[id] = struct{}{}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if resolved.InputExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
|
add(resolved.Input)
|
||||||
|
}
|
||||||
|
if resolved.ChunkExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
add(resolved.Chunk)
|
add(resolved.Chunk)
|
||||||
|
}
|
||||||
for _, lane := range resolved.AllArtifactLanes() {
|
for _, lane := range resolved.AllArtifactLanes() {
|
||||||
|
if lane.ExtractExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
add(lane.Extract)
|
add(lane.Extract)
|
||||||
|
}
|
||||||
|
if lane.MergeExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
add(lane.Merge)
|
add(lane.Merge)
|
||||||
|
}
|
||||||
|
if lane.NormalizeExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
add(lane.Normalize)
|
add(lane.Normalize)
|
||||||
}
|
}
|
||||||
|
}
|
||||||
for _, chain := range resolved.ValidatorChains {
|
for _, chain := range resolved.ValidatorChains {
|
||||||
for _, validator := range chain.Validators {
|
for _, validator := range chain.Validators {
|
||||||
if validator.ExecutionClass == contracts.ExecutionClassLLMBacked {
|
if validator.ExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
@@ -896,6 +982,9 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if resolved.OutputExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
|
add(resolved.Output)
|
||||||
|
}
|
||||||
ids := make([]string, 0, len(seen))
|
ids := make([]string, 0, len(seen))
|
||||||
for id := range seen {
|
for id := range seen {
|
||||||
ids = append(ids, id)
|
ids = append(ids, id)
|
||||||
@@ -979,7 +1068,7 @@ func runConfigValidate(args []string, stdout, stderr io.Writer, opts Options) in
|
|||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
if err := validateExplicitScriptoriumProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline)); err != nil {
|
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline), opts.promptKitAssets); err != nil {
|
||||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
@@ -1144,6 +1233,7 @@ type sessionIDFlag struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type singleValueFlag struct {
|
type singleValueFlag struct {
|
||||||
|
name string
|
||||||
value string
|
value string
|
||||||
set bool
|
set bool
|
||||||
}
|
}
|
||||||
@@ -1157,7 +1247,7 @@ func (flag *singleValueFlag) String() string {
|
|||||||
|
|
||||||
func (flag *singleValueFlag) Set(value string) error {
|
func (flag *singleValueFlag) Set(value string) error {
|
||||||
if flag.set {
|
if flag.set {
|
||||||
return fmt.Errorf("--recompute-step may be specified only once")
|
return fmt.Errorf("%s may be specified only once", flag.name)
|
||||||
}
|
}
|
||||||
flag.value = value
|
flag.value = value
|
||||||
flag.set = true
|
flag.set = true
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ import (
|
|||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/checkpoint"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
)
|
)
|
||||||
@@ -241,12 +242,14 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
t.Run("one effective profile reaches the factory and modules", func(t *testing.T) {
|
t.Run("one effective profile reaches the factory and modules", func(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
profileDir := writeRunContractProfiles(t, "override-profile")
|
profileDir := writeRunContractProfiles(t, "override-profile")
|
||||||
prependRunContractConfig(t, roots, fmt.Sprintf("scriptorium:\n profile_dir: %q\n", profileDir))
|
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||||
harness := newStateTestHarness()
|
harness := newStateTestHarness()
|
||||||
var factoryProfiles []string
|
var factoryProfiles []string
|
||||||
opts := harness.options()
|
opts := harness.options()
|
||||||
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
var factoryOverrides []LLMRuntimeOverrides
|
||||||
|
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
factoryProfiles = append(factoryProfiles, profileID)
|
factoryProfiles = append(factoryProfiles, profileID)
|
||||||
|
factoryOverrides = append(factoryOverrides, overrides)
|
||||||
return nil, nil, nil
|
return nil, nil, nil
|
||||||
}
|
}
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
@@ -257,6 +260,9 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" {
|
if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" {
|
||||||
t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles)
|
t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles)
|
||||||
}
|
}
|
||||||
|
if len(factoryOverrides) != 1 || factoryOverrides[0].ReasoningEffort != nil {
|
||||||
|
t.Fatalf("factory overrides = %#v, want inherited reasoning", factoryOverrides)
|
||||||
|
}
|
||||||
harness.mu.Lock()
|
harness.mu.Lock()
|
||||||
profiles := append([]string(nil), harness.moduleProfiles...)
|
profiles := append([]string(nil), harness.moduleProfiles...)
|
||||||
harness.mu.Unlock()
|
harness.mu.Unlock()
|
||||||
@@ -270,16 +276,16 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
t.Run("validator profile remains distinct", func(t *testing.T) {
|
t.Run("runtime override applies to validators", func(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
profileDir := writeRunContractProfiles(t, "override-profile", "validator-profile")
|
profileDir := writeRunContractProfiles(t, "override-profile", "validator-profile")
|
||||||
prependRunContractConfig(t, roots, fmt.Sprintf("scriptorium:\n profile_dir: %q\n", profileDir))
|
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||||
harness := newStateTestHarness()
|
harness := newStateTestHarness()
|
||||||
var validatorProfiles []string
|
var validatorProfiles []string
|
||||||
opts := harness.options()
|
opts := harness.options()
|
||||||
registerRunContractValidator(t, &opts, &validatorProfiles)
|
registerRunContractValidator(t, &opts, &validatorProfiles)
|
||||||
factoryProfiles := []string{}
|
factoryProfiles := []string{}
|
||||||
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string, _ LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
factoryProfiles = append(factoryProfiles, profileID)
|
factoryProfiles = append(factoryProfiles, profileID)
|
||||||
return nil, nil, nil
|
return nil, nil, nil
|
||||||
}
|
}
|
||||||
@@ -288,21 +294,21 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
if code != 0 || stderr.Len() != 0 {
|
if code != 0 || stderr.Len() != 0 {
|
||||||
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
}
|
}
|
||||||
if len(factoryProfiles) != 1 || factoryProfiles[0] != "" {
|
if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" {
|
||||||
t.Fatalf("factory profiles = %#v, want one call without a unique profile", factoryProfiles)
|
t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles)
|
||||||
}
|
}
|
||||||
if len(validatorProfiles) != 1 || validatorProfiles[0] != "validator-profile" {
|
if len(validatorProfiles) != 1 || validatorProfiles[0] != "override-profile" {
|
||||||
t.Fatalf("validator profiles = %#v, want configured validator profile", validatorProfiles)
|
t.Fatalf("validator profiles = %#v, want runtime override", validatorProfiles)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
t.Run("unknown profile is rejected without factory access", func(t *testing.T) {
|
t.Run("unknown profile is rejected without factory access", func(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
profileDir := writeRunContractProfiles(t, "override-profile")
|
profileDir := writeRunContractProfiles(t, "override-profile")
|
||||||
prependRunContractConfig(t, roots, fmt.Sprintf("scriptorium:\n profile_dir: %q\n", profileDir))
|
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||||
factoryCalls := 0
|
factoryCalls := 0
|
||||||
opts := newStateTestHarness().options()
|
opts := newStateTestHarness().options()
|
||||||
opts.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
factoryCalls++
|
factoryCalls++
|
||||||
return nil, nil, nil
|
return nil, nil, nil
|
||||||
}
|
}
|
||||||
@@ -312,18 +318,164 @@ func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
|
|||||||
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
|
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run("pipeline default is rejected before factory access", func(t *testing.T) {
|
||||||
|
roots := newStateTestRoots(t)
|
||||||
|
profileDir := writeRunContractProfiles(t, "configured-profile")
|
||||||
|
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
|
||||||
|
replaceStateTestConfigLine(t, roots.config, " sample:\n", " sample:\n llm_profile: missing-profile\n")
|
||||||
|
factoryCalls := 0
|
||||||
|
opts := newStateTestHarness().options()
|
||||||
|
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
|
factoryCalls++
|
||||||
|
return nil, nil, nil
|
||||||
|
}
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, &stdout, &stderr, opts)
|
||||||
|
if code != 1 || !strings.Contains(stderr.String(), "not configured") || factoryCalls != 0 || stdout.Len() != 0 {
|
||||||
|
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunReasoningEffortOverrideReachesFactory(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
flags []string
|
||||||
|
wantValue string
|
||||||
|
wantSet bool
|
||||||
|
}{
|
||||||
|
{name: "inherit"},
|
||||||
|
{name: "replace", flags: []string{"--reasoning-effort", " focused "}, wantValue: "focused", wantSet: true},
|
||||||
|
{name: "clear", flags: []string{"--clear-reasoning-effort"}, wantSet: true},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
roots := newStateTestRoots(t)
|
||||||
|
opts := newStateTestHarness().options()
|
||||||
|
var got []LLMRuntimeOverrides
|
||||||
|
opts.LLMClientFactory = func(_ context.Context, _ config.Config, _ string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
|
got = append(got, overrides)
|
||||||
|
return nil, nil, nil
|
||||||
|
}
|
||||||
|
args := append([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, tt.flags...)
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
if code := RunWithOptions(args, &stdout, &stderr, opts); code != 0 || stderr.Len() != 0 {
|
||||||
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
|
}
|
||||||
|
if len(got) != 1 {
|
||||||
|
t.Fatalf("factory overrides = %#v, want one call", got)
|
||||||
|
}
|
||||||
|
if !tt.wantSet {
|
||||||
|
if got[0].ReasoningEffort != nil {
|
||||||
|
t.Fatalf("reasoning effort = %q, want inherit", *got[0].ReasoningEffort)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if got[0].ReasoningEffort == nil || *got[0].ReasoningEffort != tt.wantValue {
|
||||||
|
t.Fatalf("reasoning effort = %#v, want %q", got[0].ReasoningEffort, tt.wantValue)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunReasoningEffortOverrideRejectsInvalidSyntax(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
flags []string
|
||||||
|
wantError string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "mutually exclusive controls",
|
||||||
|
flags: []string{"--reasoning-effort", "focused", "--clear-reasoning-effort"},
|
||||||
|
wantError: "cannot be combined",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "empty replacement",
|
||||||
|
flags: []string{"--reasoning-effort", " "},
|
||||||
|
wantError: "must not be empty",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "duplicate replacement",
|
||||||
|
flags: []string{"--reasoning-effort", "low", "--reasoning-effort", "high"},
|
||||||
|
wantError: "may be specified only once",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "missing replacement",
|
||||||
|
flags: []string{"--reasoning-effort"},
|
||||||
|
wantError: "flag needs an argument",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
roots := newStateTestRoots(t)
|
||||||
|
args := append([]string{"run", "sample", "--config", roots.config, "--input", roots.input}, tt.flags...)
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
code := RunWithOptions(args, &stdout, &stderr, newStateTestHarness().options())
|
||||||
|
if code != 2 || stdout.Len() != 0 || !strings.Contains(stderr.String(), tt.wantError) {
|
||||||
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
|
}
|
||||||
|
assertNoRunState(t, roots)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReasoningEffortOverrideSeparatesCheckpointIdentities(t *testing.T) {
|
||||||
|
replacement := " focused "
|
||||||
|
cleared := ""
|
||||||
|
states := []struct {
|
||||||
|
name string
|
||||||
|
overrides LLMRuntimeOverrides
|
||||||
|
wantValue string
|
||||||
|
wantSet bool
|
||||||
|
}{
|
||||||
|
{name: "inherit"},
|
||||||
|
{name: "replace", overrides: LLMRuntimeOverrides{ReasoningEffort: &replacement}, wantValue: "focused", wantSet: true},
|
||||||
|
{name: "clear", overrides: LLMRuntimeOverrides{ReasoningEffort: &cleared}, wantValue: "<cleared>", wantSet: true},
|
||||||
|
}
|
||||||
|
digests := make(map[string]string, len(states))
|
||||||
|
for _, state := range states {
|
||||||
|
fingerprints := runtimeOverrideFingerprints("", "", state.overrides)
|
||||||
|
var value string
|
||||||
|
var found bool
|
||||||
|
for _, fingerprint := range fingerprints {
|
||||||
|
if fingerprint.Name == "reasoning_effort_override" {
|
||||||
|
value, found = fingerprint.Value, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if found != state.wantSet || (found && value != state.wantValue) {
|
||||||
|
t.Fatalf("%s fingerprint found=%t value=%q, want found=%t value=%q", state.name, found, value, state.wantSet, state.wantValue)
|
||||||
|
}
|
||||||
|
identity, err := checkpoint.NewIdentity(checkpoint.IdentityInput{
|
||||||
|
Pipeline: pipeline.ResolvedPipeline{ID: "sample", Digest: "sha256:pipeline", Input: pipeline.Binding("test/input")},
|
||||||
|
RawInputDigest: "sha256:input",
|
||||||
|
RuntimeOverrides: fingerprints,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
digests[state.name] = identity.Digest
|
||||||
|
}
|
||||||
|
if digests["inherit"] == digests["replace"] || digests["inherit"] == digests["clear"] || digests["replace"] == digests["clear"] {
|
||||||
|
t.Fatalf("checkpoint identity digests are not distinct: %#v", digests)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
|
func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
|
||||||
resolved := pipeline.ResolvedPipeline{
|
resolved := pipeline.ResolvedPipeline{
|
||||||
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
|
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
|
||||||
|
InputExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "},
|
Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "},
|
||||||
|
ChunkExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Steps: []pipeline.ResolvedPipelineStep{{
|
Steps: []pipeline.ResolvedPipelineStep{{
|
||||||
ID: "default",
|
ID: "default",
|
||||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
||||||
Extract: pipeline.ModuleBinding{LLMProfile: "alpha"},
|
Extract: pipeline.ModuleBinding{LLMProfile: "alpha"},
|
||||||
Merge: pipeline.ModuleBinding{LLMProfile: "zeta"},
|
ExtractExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
|
Merge: pipeline.ModuleBinding{LLMProfile: "deterministic-merge"},
|
||||||
|
MergeExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "},
|
Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "},
|
||||||
|
NormalizeExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
}},
|
}},
|
||||||
}},
|
}},
|
||||||
ValidatorChains: []pipeline.ResolvedValidatorChain{{Validators: []pipeline.ResolvedValidator{
|
ValidatorChains: []pipeline.ResolvedValidatorChain{{Validators: []pipeline.ResolvedValidator{
|
||||||
@@ -331,9 +483,10 @@ func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
|
|||||||
{Binding: pipeline.ModuleBinding{LLMProfile: "beta"}, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
{Binding: pipeline.ModuleBinding{LLMProfile: "beta"}, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
}}},
|
}}},
|
||||||
Output: pipeline.ModuleBinding{LLMProfile: "output-profile"},
|
Output: pipeline.ModuleBinding{LLMProfile: "output-profile"},
|
||||||
|
OutputExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
}
|
}
|
||||||
got := effectiveLLMProfileIDs(resolved)
|
got := effectiveLLMProfileIDs(resolved)
|
||||||
want := []string{"alpha", "beta", "gamma", "zeta"}
|
want := []string{"alpha", "beta", "gamma", "input-profile", "output-profile", "zeta"}
|
||||||
if strings.Join(got, ",") != strings.Join(want, ",") {
|
if strings.Join(got, ",") != strings.Join(want, ",") {
|
||||||
t.Fatalf("effective profiles = %#v, want %#v", got, want)
|
t.Fatalf("effective profiles = %#v, want %#v", got, want)
|
||||||
}
|
}
|
||||||
@@ -376,7 +529,7 @@ func TestRunFactoryAndPreparationFailuresAreProcessFailures(t *testing.T) {
|
|||||||
t.Run("LLM factory", func(t *testing.T) {
|
t.Run("LLM factory", func(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
opts := newStateTestHarness().options()
|
opts := newStateTestHarness().options()
|
||||||
opts.LLMClientFactory = func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
return nil, nil, errors.New("injected LLM factory failure")
|
return nil, nil, errors.New("injected LLM factory failure")
|
||||||
}
|
}
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
@@ -397,7 +550,7 @@ func TestRunFactoryAndPreparationFailuresAreProcessFailures(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
opts := newStateTestHarness().options()
|
opts := newStateTestHarness().options()
|
||||||
if err := pipeline.RegisterExtractorBuilder(opts.Registries.Extractors, pipeline.ModuleSpec{Key: "test/failing-extract", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Extractor[stateTestArtifact], error) {
|
if err := pipeline.RegisterExtractorBuilder(opts.Registries.Extractors, pipeline.ModuleSpec{Key: "test/failing-extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Extractor[stateTestArtifact], error) {
|
||||||
return nil, errors.New("injected extractor construction failure")
|
return nil, errors.New("injected extractor construction failure")
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
|
|||||||
@@ -53,7 +53,7 @@ func newRunResult(resolved pipeline.ResolvedPipeline, output pipeline.RunOutput,
|
|||||||
result := runResult{
|
result := runResult{
|
||||||
SchemaVersion: runResultSchemaVersion,
|
SchemaVersion: runResultSchemaVersion,
|
||||||
RunID: output.Manifest.RunID,
|
RunID: output.Manifest.RunID,
|
||||||
PipelineID: resolved.ID,
|
PipelineID: output.Manifest.PipelineID,
|
||||||
OutputDirectory: absOutputDirectory,
|
OutputDirectory: absOutputDirectory,
|
||||||
NormalizedOutputCount: len(output.NormalizeOutputs),
|
NormalizedOutputCount: len(output.NormalizeOutputs),
|
||||||
RejectedOutputCount: len(output.Rejected),
|
RejectedOutputCount: len(output.Rejected),
|
||||||
|
|||||||
@@ -1,68 +0,0 @@
|
|||||||
package cli
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"testing/fstest"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
|
||||||
)
|
|
||||||
|
|
||||||
const profileCheckPromptID = "notarius.profile.check"
|
|
||||||
|
|
||||||
var profileCheckPromptFS = fstest.MapFS{
|
|
||||||
"prompts/profile-check.yaml": &fstest.MapFile{Data: []byte(`id: notarius.profile.check
|
|
||||||
version: "1.0.0"
|
|
||||||
default_profile: mistral-small-3
|
|
||||||
inputs:
|
|
||||||
- name: transcript
|
|
||||||
required: true
|
|
||||||
messages:
|
|
||||||
- role: user
|
|
||||||
content: "{{input \"transcript\"}}"
|
|
||||||
output:
|
|
||||||
format: text
|
|
||||||
validation_mode: none
|
|
||||||
repair_attempts: 0
|
|
||||||
`)},
|
|
||||||
}
|
|
||||||
|
|
||||||
func validateExplicitScriptoriumProfiles(ctx context.Context, cfg config.Config, profileIDs []string) error {
|
|
||||||
if len(profileIDs) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
engine, err := newProfileValidationEngine(cfg)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("load Scriptorium profiles: %w", err)
|
|
||||||
}
|
|
||||||
for _, profileID := range profileIDs {
|
|
||||||
if _, err := engine.Prepare(ctx, scriptorium.RunRequest{
|
|
||||||
PromptID: profileCheckPromptID,
|
|
||||||
ProfileID: profileID,
|
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
|
||||||
"transcript": scriptorium.Inline("profile check"),
|
|
||||||
},
|
|
||||||
}); err != nil {
|
|
||||||
if errors.Is(err, scriptorium.ErrProfileNotFound) {
|
|
||||||
return fmt.Errorf("Scriptorium profile %q is not configured", profileID)
|
|
||||||
}
|
|
||||||
return fmt.Errorf("validate Scriptorium profile %q: %w", profileID, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func newProfileValidationEngine(cfg config.Config) (*scriptorium.Engine, error) {
|
|
||||||
opts := []scriptorium.Option{
|
|
||||||
scriptorium.WithPromptFS(profileCheckPromptFS, "prompts"),
|
|
||||||
}
|
|
||||||
if cfg.Scriptorium.ProfileFile != "" {
|
|
||||||
opts = append(opts, scriptorium.WithProfileFile(cfg.Scriptorium.ProfileFile))
|
|
||||||
}
|
|
||||||
return scriptorium.NewEngine(scriptorium.Config{
|
|
||||||
PromptDir: "unused",
|
|
||||||
ProfileDir: cfg.Scriptorium.ProfileDir,
|
|
||||||
}, opts...)
|
|
||||||
}
|
|
||||||
@@ -214,8 +214,9 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
fingerprints := prepared.CheckpointFingerprints()
|
fingerprints := prepared.CheckpointFingerprints()
|
||||||
|
llmFingerprints := []checkpoint.Fingerprint{{Name: "promptkit_profile_source", Value: "sha256:profile-source-one"}}
|
||||||
settings := config.CheckpointCacheConfig{Enabled: true, Directory: t.TempDir()}
|
settings := config.CheckpointCacheConfig{Enabled: true, Directory: t.TempDir()}
|
||||||
recorder, _, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, []byte("same input"), nil, nil, "", "", false)
|
recorder, _, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -241,7 +242,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
_, sameLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, []byte("same input"), nil, nil, "", "", true)
|
_, sameLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -253,7 +254,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
|||||||
}
|
}
|
||||||
changed := replaceCheckpointFingerprintValue(t, fingerprints, normalizeSpellCatalogFingerprintName(), "sha256:changed-effective-catalog")
|
changed := replaceCheckpointFingerprintValue(t, fingerprints, normalizeSpellCatalogFingerprintName(), "sha256:changed-effective-catalog")
|
||||||
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changed, normalizeSpellCatalogFingerprintName())
|
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changed, normalizeSpellCatalogFingerprintName())
|
||||||
_, changedLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changed, []byte("same input"), nil, nil, "", "", true)
|
_, changedLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changed, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -265,13 +266,25 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
|||||||
}
|
}
|
||||||
changedMapping := replaceCheckpointFingerprintValue(t, fingerprints, extractSpellMappingFingerprintName(), "dnd.spells.extract_mapping.v3")
|
changedMapping := replaceCheckpointFingerprintValue(t, fingerprints, extractSpellMappingFingerprintName(), "dnd.spells.extract_mapping.v3")
|
||||||
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changedMapping, extractSpellMappingFingerprintName())
|
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changedMapping, extractSpellMappingFingerprintName())
|
||||||
_, mappingLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changedMapping, []byte("same input"), nil, nil, "", "", true)
|
_, mappingLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changedMapping, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, decision := mappingLoader.Source(materialized.Input.Module); decision.Reused {
|
if _, decision := mappingLoader.Source(materialized.Input.Module); decision.Reused {
|
||||||
t.Fatalf("changed mapping policy decision = %#v, want cold miss", decision)
|
t.Fatalf("changed mapping policy decision = %#v, want cold miss", decision)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
changedLLMFingerprints := []checkpoint.Fingerprint{{Name: "promptkit_profile_source", Value: "sha256:profile-source-two"}}
|
||||||
|
_, profileLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, changedLLMFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, decision := profileLoader.Source(materialized.Input.Module); decision.Reused {
|
||||||
|
t.Fatalf("changed PromptKit profile source decision = %#v, want cold miss", decision)
|
||||||
|
}
|
||||||
|
if _, decision := profileLoader.Normalize("spells", spellnormalize.Key, normalizeDependencies); decision.Reused {
|
||||||
|
t.Fatalf("changed PromptKit profile normalize decision = %#v, want cold miss", decision)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeSpellCatalogFingerprintName() string {
|
func normalizeSpellCatalogFingerprintName() string {
|
||||||
|
|||||||
@@ -635,7 +635,7 @@ func newStateTestRoots(t *testing.T) stateTestRoots {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
roots.config = filepath.Join(base, "config.yml")
|
roots.config = filepath.Join(base, "config.yml")
|
||||||
config := fmt.Sprintf("version: 3\noutput:\n directory: %q\ncache:\n chunk_plans:\n directory: %q\n mode: auto\n checkpoints:\n enabled: true\n directory: %q\ndebug:\n directory: %q\npipelines:\n sample:\n input: test/input\n chunk: test/chunk\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n output: test/output\n", roots.output, roots.plans, roots.checkpoints, roots.debug)
|
config := fmt.Sprintf("version: 4\noutput:\n directory: %q\ncache:\n chunk_plans:\n directory: %q\n mode: auto\n checkpoints:\n enabled: true\n directory: %q\ndebug:\n directory: %q\npipelines:\n sample:\n input: test/input\n chunk: test/chunk\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n output: test/output\n", roots.output, roots.plans, roots.checkpoints, roots.debug)
|
||||||
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
|
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -823,22 +823,22 @@ func (h *stateTestHarness) options() Options {
|
|||||||
if err := pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, stateTestCodec{}); err != nil {
|
if err := pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, stateTestCodec{}); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }); err != nil {
|
if err := registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := registries.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/chunk", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "cache-reference"}}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Chunker, error) { return stateTestChunker{h}, nil }); err != nil {
|
if err := registries.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "cache-reference"}}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Chunker, error) { return stateTestChunker{h}, nil }); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "test/extract", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{h}, nil }); err != nil {
|
if err := pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "test/extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{h}, nil }); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "test/merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{harness: h}, nil }); err != nil {
|
if err := pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "test/merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{harness: h}, nil }); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "test/normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{harness: h}, nil }); err != nil {
|
if err := pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "test/normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{harness: h}, nil }); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
||||||
return stateTestOutput{harness: h, includeWarnings: h.includeWarnings}, nil
|
return stateTestOutput{harness: h, includeWarnings: h.includeWarnings}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
@@ -848,7 +848,7 @@ func (h *stateTestHarness) options() Options {
|
|||||||
defer h.mu.Unlock()
|
defer h.mu.Unlock()
|
||||||
h.runIDCalls++
|
h.runIDCalls++
|
||||||
return fmt.Sprintf("run-%d-%032x", startedAt.UnixNano(), h.runIDCalls), nil
|
return fmt.Sprintf("run-%d-%032x", startedAt.UnixNano(), h.runIDCalls), nil
|
||||||
}, UserCacheDir: func() (string, error) { return "", errors.New("unexpected user cache lookup") }, LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
}, UserCacheDir: func() (string, error) { return "", errors.New("unexpected user cache lookup") }, LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
return nil, nil, nil
|
return nil, nil, nil
|
||||||
}}
|
}}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package artifacts
|
package artifacts
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -29,6 +30,29 @@ type LLMProfileManifest struct {
|
|||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
Provider string `json:"provider,omitempty"`
|
Provider string `json:"provider,omitempty"`
|
||||||
Model string `json:"model,omitempty"`
|
Model string `json:"model,omitempty"`
|
||||||
|
BackendID string `json:"backend_id,omitempty"`
|
||||||
|
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Normalized returns the canonical representation used for manifest identity
|
||||||
|
// and publication.
|
||||||
|
func (profile LLMProfileManifest) Normalized() LLMProfileManifest {
|
||||||
|
profile.ID = strings.TrimSpace(profile.ID)
|
||||||
|
profile.Provider = strings.TrimSpace(profile.Provider)
|
||||||
|
profile.Model = strings.TrimSpace(profile.Model)
|
||||||
|
profile.BackendID = strings.TrimSpace(profile.BackendID)
|
||||||
|
profile.ReasoningEffort = strings.TrimSpace(profile.ReasoningEffort)
|
||||||
|
return profile
|
||||||
|
}
|
||||||
|
|
||||||
|
// IdentityKey returns an opaque, deterministic key for the effective profile.
|
||||||
|
func (profile LLMProfileManifest) IdentityKey() string {
|
||||||
|
profile = profile.Normalized()
|
||||||
|
return profile.ID + "\x00" +
|
||||||
|
profile.Provider + "\x00" +
|
||||||
|
profile.Model + "\x00" +
|
||||||
|
profile.BackendID + "\x00" +
|
||||||
|
profile.ReasoningEffort
|
||||||
}
|
}
|
||||||
|
|
||||||
type ReferenceProvenance struct {
|
type ReferenceProvenance struct {
|
||||||
|
|||||||
@@ -53,7 +53,13 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
|
|||||||
PipelineID: "pipeline-1",
|
PipelineID: "pipeline-1",
|
||||||
PipelineDigest: "sha256:abc123",
|
PipelineDigest: "sha256:abc123",
|
||||||
LLMProfiles: []LLMProfileManifest{
|
LLMProfiles: []LLMProfileManifest{
|
||||||
{ID: "default", Provider: "scriptorium", Model: "model-a"},
|
{
|
||||||
|
ID: "default",
|
||||||
|
Provider: "promptkit",
|
||||||
|
Model: "model-a",
|
||||||
|
BackendID: "openrouter",
|
||||||
|
ReasoningEffort: "high",
|
||||||
|
},
|
||||||
},
|
},
|
||||||
ArtifactLanes: []ArtifactLaneManifest{
|
ArtifactLanes: []ArtifactLaneManifest{
|
||||||
{
|
{
|
||||||
@@ -101,7 +107,13 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
|
|||||||
if !ok {
|
if !ok {
|
||||||
t.Fatalf("llm_profiles[0] = %#v, want object", profiles[0])
|
t.Fatalf("llm_profiles[0] = %#v, want object", profiles[0])
|
||||||
}
|
}
|
||||||
assertHasKeys(t, profile, "id", "provider", "model")
|
assertHasKeys(t, profile, "id", "provider", "model", "backend_id", "reasoning_effort")
|
||||||
|
if profile["provider"] != "promptkit" {
|
||||||
|
t.Fatalf("llm_profiles[0].provider = %#v, want promptkit", profile["provider"])
|
||||||
|
}
|
||||||
|
if profile["backend_id"] != "openrouter" || profile["reasoning_effort"] != "high" {
|
||||||
|
t.Fatalf("llm_profiles[0] = %#v, want backend and reasoning provenance", profile)
|
||||||
|
}
|
||||||
|
|
||||||
lanes, ok := got["artifact_lanes"].([]any)
|
lanes, ok := got["artifact_lanes"].([]any)
|
||||||
if !ok {
|
if !ok {
|
||||||
|
|||||||
@@ -4,10 +4,10 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
)
|
)
|
||||||
|
|
||||||
const SupportedFileConfigVersion = 3
|
const SupportedFileConfigVersion = 4
|
||||||
|
|
||||||
type Config struct {
|
type Config struct {
|
||||||
Scriptorium ScriptoriumConfig `json:"scriptorium,omitempty"`
|
PromptKit PromptKitConfig `json:"promptkit,omitempty"`
|
||||||
Pipelines map[string]pipeline.PipelineProfile `json:"pipelines"`
|
Pipelines map[string]pipeline.PipelineProfile `json:"pipelines"`
|
||||||
Concurrency ConcurrencyConfig `json:"concurrency"`
|
Concurrency ConcurrencyConfig `json:"concurrency"`
|
||||||
Output OutputConfig `json:"output"`
|
Output OutputConfig `json:"output"`
|
||||||
@@ -15,9 +15,15 @@ type Config struct {
|
|||||||
Debug DebugConfig `json:"debug"`
|
Debug DebugConfig `json:"debug"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type ScriptoriumConfig struct {
|
type PromptKitConfig struct {
|
||||||
ProfileDir string `json:"profile_dir,omitempty"`
|
ProfileDir string `json:"profile_dir,omitempty"`
|
||||||
ProfileFile string `json:"profile_file,omitempty"`
|
ProfileFile string `json:"profile_file,omitempty"`
|
||||||
|
LocalBackend *PromptKitLocalBackendConfig `json:"local_backend,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitLocalBackendConfig struct {
|
||||||
|
Endpoint string `json:"endpoint"`
|
||||||
|
ConcurrencyLimit int `json:"concurrency_limit"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type ConcurrencyConfig struct {
|
type ConcurrencyConfig struct {
|
||||||
@@ -66,6 +72,10 @@ func Default() Config {
|
|||||||
|
|
||||||
func cloneConfig(in Config) Config {
|
func cloneConfig(in Config) Config {
|
||||||
out := in
|
out := in
|
||||||
|
if in.PromptKit.LocalBackend != nil {
|
||||||
|
localBackend := *in.PromptKit.LocalBackend
|
||||||
|
out.PromptKit.LocalBackend = &localBackend
|
||||||
|
}
|
||||||
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
|
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
|
||||||
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
|
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
|
||||||
for key, profile := range in.Pipelines {
|
for key, profile := range in.Pipelines {
|
||||||
|
|||||||
@@ -42,12 +42,10 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
|
|||||||
}
|
}
|
||||||
profile = clonePipelineProfile(profile)
|
profile = clonePipelineProfile(profile)
|
||||||
profile.ID = pipelineID
|
profile.ID = pipelineID
|
||||||
if override := strings.TrimSpace(input.LLMProfileOverride); override != "" {
|
|
||||||
applyLLMProfileOverride(&profile, override)
|
|
||||||
}
|
|
||||||
|
|
||||||
resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{
|
resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{
|
||||||
Only: input.Only,
|
Only: input.Only,
|
||||||
|
LLMProfileOverride: input.LLMProfileOverride,
|
||||||
ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...),
|
ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...),
|
||||||
ReferenceUnbinds: append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...),
|
ReferenceUnbinds: append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...),
|
||||||
}, input.Catalog)
|
}, input.Catalog)
|
||||||
@@ -65,22 +63,6 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func applyLLMProfileOverride(profile *pipeline.PipelineProfile, profileID string) {
|
|
||||||
profile.Chunk.LLMProfile = profileID
|
|
||||||
apply := func(artifacts map[string]pipeline.ArtifactLaneProfile) {
|
|
||||||
for laneID, lane := range artifacts {
|
|
||||||
lane.Extract.LLMProfile = profileID
|
|
||||||
lane.Merge.LLMProfile = profileID
|
|
||||||
lane.Normalize.LLMProfile = profileID
|
|
||||||
artifacts[laneID] = lane
|
|
||||||
}
|
|
||||||
}
|
|
||||||
apply(profile.Artifacts)
|
|
||||||
for index := range profile.Steps {
|
|
||||||
apply(profile.Steps[index].Artifacts)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func lookupPipelineProfile(profiles map[string]pipeline.PipelineProfile, pipelineID string) (pipeline.PipelineProfile, bool) {
|
func lookupPipelineProfile(profiles map[string]pipeline.PipelineProfile, pipelineID string) (pipeline.PipelineProfile, bool) {
|
||||||
pipelineID = strings.TrimSpace(pipelineID)
|
pipelineID = strings.TrimSpace(pipelineID)
|
||||||
for rawID, profile := range profiles {
|
for rawID, profile := range profiles {
|
||||||
|
|||||||
@@ -84,6 +84,56 @@ func TestEffectiveConfigMaterializesDefaultBindingsThroughCatalog(t *testing.T)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestEffectiveConfigPreservesPromptKitProfileSource(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
profileSource PromptKitConfig
|
||||||
|
}{
|
||||||
|
{name: "profile directory", profileSource: PromptKitConfig{ProfileDir: "./profiles"}},
|
||||||
|
{name: "profile file", profileSource: PromptKitConfig{ProfileFile: "./profiles.yml"}},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := configForEffectiveTests(t, effectiveProfile())
|
||||||
|
cfg.PromptKit = tt.profileSource
|
||||||
|
effective, err := cfg.Resolve(ResolveInput{PipelineID: "main", Catalog: effectiveCatalog(t)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v", err)
|
||||||
|
}
|
||||||
|
if effective.Config.PromptKit != cfg.PromptKit {
|
||||||
|
t.Fatalf("effective PromptKit config = %#v, want %#v", effective.Config.PromptKit, cfg.PromptKit)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEffectiveConfigOwnsPromptKitLocalBackend(t *testing.T) {
|
||||||
|
cfg := configForEffectiveTests(t, effectiveProfile())
|
||||||
|
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: "http://localhost:8000/v1",
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
effective, err := cfg.Resolve(ResolveInput{PipelineID: "main", Catalog: effectiveCatalog(t)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v", err)
|
||||||
|
}
|
||||||
|
if effective.Config.PromptKit.LocalBackend == nil {
|
||||||
|
t.Fatal("effective local backend = nil")
|
||||||
|
}
|
||||||
|
if effective.Config.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend {
|
||||||
|
t.Fatal("effective local backend aliases input config")
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg.PromptKit.LocalBackend.Endpoint = "http://changed-input.example/v1"
|
||||||
|
if effective.Config.PromptKit.LocalBackend.Endpoint != "http://localhost:8000/v1" {
|
||||||
|
t.Fatalf("input mutation changed effective config: %#v", effective.Config.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
effective.Config.PromptKit.LocalBackend.ConcurrencyLimit = 9
|
||||||
|
if cfg.PromptKit.LocalBackend.ConcurrencyLimit != 2 {
|
||||||
|
t.Fatalf("effective mutation changed input config: %#v", cfg.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
|
func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -154,7 +204,7 @@ func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestEffectiveConfigLLMProfileOverrideChangesDigestWithoutOverridingValidators(t *testing.T) {
|
func TestEffectiveConfigLLMProfileOverrideChangesDigestAndOverridesValidators(t *testing.T) {
|
||||||
profile := effectiveProfile()
|
profile := effectiveProfile()
|
||||||
profile.Chunk.LLMProfile = "chunk-profile"
|
profile.Chunk.LLMProfile = "chunk-profile"
|
||||||
lane := profile.Artifacts["lane"]
|
lane := profile.Artifacts["lane"]
|
||||||
@@ -187,8 +237,27 @@ func TestEffectiveConfigLLMProfileOverrideChangesDigestWithoutOverridingValidato
|
|||||||
t.Fatalf("pipeline profile override was not applied: %#v", resolved)
|
t.Fatalf("pipeline profile override was not applied: %#v", resolved)
|
||||||
}
|
}
|
||||||
validators := findEffectiveValidatorChain(resolved, pipeline.StageExtract, "lane")
|
validators := findEffectiveValidatorChain(resolved, pipeline.StageExtract, "lane")
|
||||||
if len(validators.Validators) != 1 || validators.Validators[0].Binding.LLMProfile != "validator-profile" {
|
if len(validators.Validators) != 1 || validators.Validators[0].Binding.LLMProfile != "override-profile" {
|
||||||
t.Fatalf("validator profile was overridden: %#v", validators)
|
t.Fatalf("validator profile = %#v, want runtime override", validators)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEffectiveConfigPipelineLLMProfileIsInheritedWithoutMutatingConfig(t *testing.T) {
|
||||||
|
profile := effectiveProfile()
|
||||||
|
profile.LLMProfile = " configured-profile "
|
||||||
|
effective, err := resolveEffectiveProfile(t, profile, ResolveInput{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v", err)
|
||||||
|
}
|
||||||
|
if got := effective.Config.Pipelines["main"].LLMProfile; got != " configured-profile " {
|
||||||
|
t.Fatalf("effective config pipeline llm profile = %q, want preserved programmatic value", got)
|
||||||
|
}
|
||||||
|
resolved := effective.ResolvedPipeline
|
||||||
|
if got := resolved.Chunk.LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("resolved chunk profile = %q, want inherited profile", got)
|
||||||
|
}
|
||||||
|
if got := resolved.Steps[0].ArtifactLanes[0].Extract.LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("resolved extract profile = %q, want inherited profile", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -435,7 +504,7 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
|
|||||||
if err := pipeline.RegisterArtifactCodec(catalog.ArtifactCodecs, effectiveCodec{}); err != nil {
|
if err := pipeline.RegisterArtifactCodec(catalog.ArtifactCodecs, effectiveCodec{}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := catalog.Inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
|
if err := catalog.Inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
|
||||||
return effectiveInput{key: "input"}, nil
|
return effectiveInput{key: "input"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
@@ -443,6 +512,7 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
|
|||||||
chunkSpec := pipeline.ModuleSpec{
|
chunkSpec := pipeline.ModuleSpec{
|
||||||
Key: pipeline.DefaultChunkModule,
|
Key: pipeline.DefaultChunkModule,
|
||||||
Stage: pipeline.StageChunk,
|
Stage: pipeline.StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Requires: []string{"source"},
|
Requires: []string{"source"},
|
||||||
Provides: []string{"chunk"},
|
Provides: []string{"chunk"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-ref"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-ref"}},
|
||||||
@@ -453,32 +523,32 @@ func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := catalog.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "needs-capability", Stage: pipeline.StageChunk, Requires: []string{"missing"}}, chunkOptions, func(pipeline.BuildRequest) (contracts.Chunker, error) {
|
if err := catalog.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "needs-capability", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}}, chunkOptions, func(pipeline.BuildRequest) (contracts.Chunker, error) {
|
||||||
return effectiveChunker{key: "needs-capability"}, nil
|
return effectiveChunker{key: "needs-capability"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterExtractor(catalog.Extractors, pipeline.ModuleSpec{Key: "extract", Stage: pipeline.StageExtract, ArtifactKind: effectiveArtifactKind, Requires: []string{"chunk"}, Provides: []string{"candidate"}}, func() (contracts.Extractor[effectiveArtifact], error) {
|
if err := pipeline.RegisterExtractor(catalog.Extractors, pipeline.ModuleSpec{Key: "extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"chunk"}, Provides: []string{"candidate"}}, func() (contracts.Extractor[effectiveArtifact], error) {
|
||||||
return effectiveExtractor{key: "extract"}, nil
|
return effectiveExtractor{key: "extract"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: pipeline.DefaultMergeModule, Stage: pipeline.StageMerge, ArtifactKind: effectiveArtifactKind, Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: pipeline.DefaultMergeModule, Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
||||||
return effectiveMerger{key: pipeline.DefaultMergeModule}, nil
|
return effectiveMerger{key: pipeline.DefaultMergeModule}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: "other-merge", Stage: pipeline.StageMerge, ArtifactKind: "other-kind", Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: "other-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "other-kind", Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
|
||||||
return effectiveMerger{key: "other-merge"}, nil
|
return effectiveMerger{key: "other-merge"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := pipeline.RegisterNormalizer(catalog.Normalizers, pipeline.ModuleSpec{Key: pipeline.DefaultNormalizeModule, Stage: pipeline.StageNormalize, ArtifactKind: effectiveArtifactKind, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer[effectiveArtifact], error) {
|
if err := pipeline.RegisterNormalizer(catalog.Normalizers, pipeline.ModuleSpec{Key: pipeline.DefaultNormalizeModule, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer[effectiveArtifact], error) {
|
||||||
return effectiveNormalizer{key: pipeline.DefaultNormalizeModule}, nil
|
return effectiveNormalizer{key: pipeline.DefaultNormalizeModule}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := catalog.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: pipeline.DefaultOutputModule, Stage: pipeline.StageOutput, Requires: []string{"normalized"}}, func() (contracts.OutputEncoder, error) {
|
if err := catalog.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: pipeline.DefaultOutputModule, Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}}, func() (contracts.OutputEncoder, error) {
|
||||||
return effectiveOutput{key: pipeline.DefaultOutputModule}, nil
|
return effectiveOutput{key: pipeline.DefaultOutputModule}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
func TestPrecedenceFileValuesOverrideBuiltInDefaults(t *testing.T) {
|
func TestPrecedenceFileValuesOverrideBuiltInDefaults(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 4
|
total_llm: 4
|
||||||
stage_workers:
|
stage_workers:
|
||||||
@@ -35,7 +35,7 @@ debug:
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestPrecedenceOperationalEnvironmentOverridesFileValues(t *testing.T) {
|
func TestPrecedenceOperationalEnvironmentOverridesFileValues(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 2
|
total_llm: 2
|
||||||
stage_workers:
|
stage_workers:
|
||||||
@@ -81,21 +81,21 @@ func TestPrecedenceExtractWorkersFollowEffectiveConcurrencyUnlessExplicit(t *tes
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "default follows environment total",
|
name: "default follows environment total",
|
||||||
file: "version: 3\n",
|
file: "version: 4\n",
|
||||||
env: map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "5"},
|
env: map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "5"},
|
||||||
wantTotal: 5,
|
wantTotal: 5,
|
||||||
wantWorker: 5,
|
wantWorker: 5,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "file worker is retained",
|
name: "file worker is retained",
|
||||||
file: "version: 3\nconcurrency:\n total_llm: 3\n stage_workers:\n extract: 2\n",
|
file: "version: 4\nconcurrency:\n total_llm: 3\n stage_workers:\n extract: 2\n",
|
||||||
env: map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6"},
|
env: map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6"},
|
||||||
wantTotal: 6,
|
wantTotal: 6,
|
||||||
wantWorker: 2,
|
wantWorker: 2,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "environment worker is retained",
|
name: "environment worker is retained",
|
||||||
file: "version: 3\nconcurrency:\n total_llm: 2\n",
|
file: "version: 4\nconcurrency:\n total_llm: 2\n",
|
||||||
env: map[string]string{
|
env: map[string]string{
|
||||||
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6",
|
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6",
|
||||||
"NOTARIUS_STAGE_WORKERS_EXTRACT": "4",
|
"NOTARIUS_STAGE_WORKERS_EXTRACT": "4",
|
||||||
@@ -118,7 +118,7 @@ func TestPrecedenceExtractWorkersFollowEffectiveConcurrencyUnlessExplicit(t *tes
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestPrecedenceEmptyFileCacheDirectoriesDeferPerUserResolution(t *testing.T) {
|
func TestPrecedenceEmptyFileCacheDirectoriesDeferPerUserResolution(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
cache:
|
cache:
|
||||||
chunk_plans:
|
chunk_plans:
|
||||||
directory: ""
|
directory: ""
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ import (
|
|||||||
|
|
||||||
type FileConfig struct {
|
type FileConfig struct {
|
||||||
Version int `yaml:"version"`
|
Version int `yaml:"version"`
|
||||||
Scriptorium *FileScriptoriumConfig `yaml:"scriptorium,omitempty"`
|
PromptKit *FilePromptKitConfig `yaml:"promptkit,omitempty"`
|
||||||
Pipelines map[string]FilePipelineProfile `yaml:"pipelines,omitempty"`
|
Pipelines map[string]FilePipelineProfile `yaml:"pipelines,omitempty"`
|
||||||
Concurrency *FileConcurrencyConfig `yaml:"concurrency,omitempty"`
|
Concurrency *FileConcurrencyConfig `yaml:"concurrency,omitempty"`
|
||||||
Output *FileOutputConfig `yaml:"output,omitempty"`
|
Output *FileOutputConfig `yaml:"output,omitempty"`
|
||||||
@@ -22,12 +22,19 @@ type FileConfig struct {
|
|||||||
Debug *FileDebugConfig `yaml:"debug,omitempty"`
|
Debug *FileDebugConfig `yaml:"debug,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type FileScriptoriumConfig struct {
|
type FilePromptKitConfig struct {
|
||||||
ProfileDir *string `yaml:"profile_dir,omitempty"`
|
ProfileDir *string `yaml:"profile_dir,omitempty"`
|
||||||
ProfileFile *string `yaml:"profile_file,omitempty"`
|
ProfileFile *string `yaml:"profile_file,omitempty"`
|
||||||
|
LocalBackend *FilePromptKitLocalBackendConfig `yaml:"local_backend,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type FilePromptKitLocalBackendConfig struct {
|
||||||
|
Endpoint *string `yaml:"endpoint,omitempty"`
|
||||||
|
ConcurrencyLimit *int `yaml:"concurrency_limit,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type FilePipelineProfile struct {
|
type FilePipelineProfile struct {
|
||||||
|
LLMProfile *string `yaml:"llm_profile,omitempty"`
|
||||||
Input fileModuleBinding `yaml:"input"`
|
Input fileModuleBinding `yaml:"input"`
|
||||||
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
|
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
|
||||||
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
|
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
|
||||||
@@ -36,13 +43,14 @@ type FilePipelineProfile struct {
|
|||||||
References map[string]fileReferenceSource `yaml:"references,omitempty"`
|
References map[string]fileReferenceSource `yaml:"references,omitempty"`
|
||||||
artifactsSet bool `yaml:"-"`
|
artifactsSet bool `yaml:"-"`
|
||||||
stepsSet bool `yaml:"-"`
|
stepsSet bool `yaml:"-"`
|
||||||
|
llmProfileSet bool `yaml:"-"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
|
func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
|
||||||
type plainFilePipelineProfile FilePipelineProfile
|
type plainFilePipelineProfile FilePipelineProfile
|
||||||
var decoded plainFilePipelineProfile
|
var decoded plainFilePipelineProfile
|
||||||
seen, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
|
seen, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
|
||||||
"input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
|
"llm_profile": {}, "input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
|
||||||
}, "pipeline profile")
|
}, "pipeline profile")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -50,6 +58,7 @@ func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
|
|||||||
*p = FilePipelineProfile(decoded)
|
*p = FilePipelineProfile(decoded)
|
||||||
_, p.artifactsSet = seen["artifacts"]
|
_, p.artifactsSet = seen["artifacts"]
|
||||||
_, p.stepsSet = seen["steps"]
|
_, p.stepsSet = seen["steps"]
|
||||||
|
_, p.llmProfileSet = seen["llm_profile"]
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -251,6 +260,9 @@ func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
b.LLMProfile = strings.TrimSpace(llmProfile)
|
b.LLMProfile = strings.TrimSpace(llmProfile)
|
||||||
|
if b.LLMProfile == "" {
|
||||||
|
return fmt.Errorf("llm_profile must not be empty when set")
|
||||||
|
}
|
||||||
case "retries":
|
case "retries":
|
||||||
var retries int
|
var retries int
|
||||||
if err := valueNode.Decode(&retries); err != nil {
|
if err := valueNode.Decode(&retries); err != nil {
|
||||||
@@ -325,6 +337,9 @@ func ParseFileConfigYAML(data []byte) (FileConfig, error) {
|
|||||||
if header.Version == 2 {
|
if header.Version == 2 {
|
||||||
return FileConfig{}, fmt.Errorf("config version 2 is no longer supported; migrate the file using the version 2-to-3 migration in docs/config.md")
|
return FileConfig{}, fmt.Errorf("config version 2 is no longer supported; migrate the file using the version 2-to-3 migration in docs/config.md")
|
||||||
}
|
}
|
||||||
|
if header.Version == 3 {
|
||||||
|
return FileConfig{}, fmt.Errorf("config version 3 is no longer supported; change \"version: 3\" to \"version: 4\" and rename \"scriptorium:\" to \"promptkit:\"")
|
||||||
|
}
|
||||||
if header.Version != SupportedFileConfigVersion {
|
if header.Version != SupportedFileConfigVersion {
|
||||||
return FileConfig{}, fmt.Errorf("unsupported config version %d (supported version is %d)", header.Version, SupportedFileConfigVersion)
|
return FileConfig{}, fmt.Errorf("unsupported config version %d (supported version is %d)", header.Version, SupportedFileConfigVersion)
|
||||||
}
|
}
|
||||||
@@ -450,25 +465,46 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if fileCfg.Scriptorium != nil {
|
if fileCfg.PromptKit != nil {
|
||||||
if fileCfg.Scriptorium.ProfileDir != nil {
|
if fileCfg.PromptKit.ProfileDir != nil {
|
||||||
value := strings.TrimSpace(*fileCfg.Scriptorium.ProfileDir)
|
value := strings.TrimSpace(*fileCfg.PromptKit.ProfileDir)
|
||||||
if value == "" {
|
if value == "" {
|
||||||
return fmt.Errorf("scriptorium.profile_dir must not be empty when set")
|
return fmt.Errorf("promptkit.profile_dir must not be empty when set")
|
||||||
}
|
}
|
||||||
c.Scriptorium.ProfileDir = value
|
c.PromptKit.ProfileDir = value
|
||||||
}
|
}
|
||||||
if fileCfg.Scriptorium.ProfileFile != nil {
|
if fileCfg.PromptKit.ProfileFile != nil {
|
||||||
value := strings.TrimSpace(*fileCfg.Scriptorium.ProfileFile)
|
value := strings.TrimSpace(*fileCfg.PromptKit.ProfileFile)
|
||||||
if value == "" {
|
if value == "" {
|
||||||
return fmt.Errorf("scriptorium.profile_file must not be empty when set")
|
return fmt.Errorf("promptkit.profile_file must not be empty when set")
|
||||||
}
|
}
|
||||||
c.Scriptorium.ProfileFile = value
|
c.PromptKit.ProfileFile = value
|
||||||
|
}
|
||||||
|
if fileCfg.PromptKit.LocalBackend != nil {
|
||||||
|
if fileCfg.PromptKit.LocalBackend.Endpoint == nil {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
|
||||||
|
}
|
||||||
|
endpoint := strings.TrimSpace(*fileCfg.PromptKit.LocalBackend.Endpoint)
|
||||||
|
if endpoint == "" {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
|
||||||
|
}
|
||||||
|
localBackend := PromptKitLocalBackendConfig{Endpoint: endpoint}
|
||||||
|
if fileCfg.PromptKit.LocalBackend.ConcurrencyLimit != nil {
|
||||||
|
localBackend.ConcurrencyLimit = *fileCfg.PromptKit.LocalBackend.ConcurrencyLimit
|
||||||
|
}
|
||||||
|
c.PromptKit.LocalBackend = &localBackend
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, pipelineID := range pipelineIDs {
|
for _, pipelineID := range pipelineIDs {
|
||||||
filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]]
|
filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]]
|
||||||
|
llmProfile := ""
|
||||||
|
if filePipeline.llmProfileSet || filePipeline.LLMProfile != nil {
|
||||||
|
if filePipeline.LLMProfile == nil || strings.TrimSpace(*filePipeline.LLMProfile) == "" {
|
||||||
|
return fmt.Errorf("pipeline %q llm_profile must not be empty when set", pipelineID)
|
||||||
|
}
|
||||||
|
llmProfile = strings.TrimSpace(*filePipeline.LLMProfile)
|
||||||
|
}
|
||||||
hasSteps := filePipeline.stepsSet || filePipeline.Steps != nil
|
hasSteps := filePipeline.stepsSet || filePipeline.Steps != nil
|
||||||
laneIDs, rawLaneIDs, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID))
|
laneIDs, rawLaneIDs, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -476,6 +512,7 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
|||||||
}
|
}
|
||||||
profile := pipeline.PipelineProfile{
|
profile := pipeline.PipelineProfile{
|
||||||
ID: pipelineID,
|
ID: pipelineID,
|
||||||
|
LLMProfile: llmProfile,
|
||||||
Input: filePipeline.Input.toPipelineBinding(),
|
Input: filePipeline.Input.toPipelineBinding(),
|
||||||
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
|
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
|
||||||
References: fileReferenceSourcesToPipeline(filePipeline.References),
|
References: fileReferenceSourcesToPipeline(filePipeline.References),
|
||||||
|
|||||||
@@ -1,6 +1,8 @@
|
|||||||
package config
|
package config
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"reflect"
|
"reflect"
|
||||||
@@ -34,8 +36,8 @@ func TestDefaultReturnsDocumentedValuesAndIndependentMaps(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigMinimalVersion3AppliesOverDefaults(t *testing.T) {
|
func TestFileConfigMinimalVersion4AppliesOverDefaults(t *testing.T) {
|
||||||
file := parseFileConfig(t, "version: 3\n")
|
file := parseFileConfig(t, "version: 4\n")
|
||||||
cfg := Default()
|
cfg := Default()
|
||||||
if err := cfg.ApplyFileConfig(file); err != nil {
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
@@ -48,6 +50,259 @@ func TestFileConfigMinimalVersion3AppliesOverDefaults(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFilePipelineLLMProfileIsPresenceAwareAndDetached(t *testing.T) {
|
||||||
|
const pipelineYAML = `version: 4
|
||||||
|
pipelines:
|
||||||
|
main:
|
||||||
|
%s
|
||||||
|
input: input
|
||||||
|
artifacts:
|
||||||
|
lane:
|
||||||
|
extract: extract
|
||||||
|
`
|
||||||
|
|
||||||
|
t.Run("omitted", func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, ""))
|
||||||
|
if file.Pipelines["main"].LLMProfile != nil || file.Pipelines["main"].llmProfileSet {
|
||||||
|
t.Fatalf("parsed pipeline profile = %#v, want omitted llm profile", file.Pipelines["main"])
|
||||||
|
}
|
||||||
|
cfg := Default()
|
||||||
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got := cfg.Pipelines["main"].LLMProfile; got != "" {
|
||||||
|
t.Fatalf("pipeline llm profile = %q, want empty", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("trimmed and detached", func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, "llm_profile: ' configured-profile '"))
|
||||||
|
cfg := Default()
|
||||||
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got := cfg.Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("pipeline llm profile = %q, want trimmed value", got)
|
||||||
|
}
|
||||||
|
*file.Pipelines["main"].LLMProfile = "changed-profile"
|
||||||
|
if got := cfg.Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("effective config aliases parsed file: %q", got)
|
||||||
|
}
|
||||||
|
if got := cloneConfig(cfg).Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("cloned pipeline llm profile = %q", got)
|
||||||
|
}
|
||||||
|
data, err := json.Marshal(cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var roundTripped Config
|
||||||
|
if err := json.Unmarshal(data, &roundTripped); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got := roundTripped.Pipelines["main"].LLMProfile; got != "configured-profile" {
|
||||||
|
t.Fatalf("round-tripped pipeline llm profile = %q", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
for _, value := range []string{"''", "' '", "null"} {
|
||||||
|
t.Run("explicit empty "+value, func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, fmt.Sprintf(pipelineYAML, "llm_profile: "+value))
|
||||||
|
cfg := Default()
|
||||||
|
err := cfg.ApplyFileConfig(file)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), `pipeline "main" llm_profile must not be empty`) {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v, want explicit-empty rejection", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFileModuleBindingRejectsExplicitEmptyLLMProfile(t *testing.T) {
|
||||||
|
const configYAML = `version: 4
|
||||||
|
pipelines:
|
||||||
|
main:
|
||||||
|
input:
|
||||||
|
module: input
|
||||||
|
llm_profile: %s
|
||||||
|
artifacts:
|
||||||
|
lane:
|
||||||
|
extract: extract
|
||||||
|
`
|
||||||
|
for _, value := range []string{"''", "' '", "null"} {
|
||||||
|
t.Run(value, func(t *testing.T) {
|
||||||
|
_, err := ParseFileConfigYAML([]byte(fmt.Sprintf(configYAML, value)))
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "llm_profile must not be empty when set") {
|
||||||
|
t.Fatalf("ParseFileConfigYAML() error = %v, want explicit-empty binding profile rejection", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitProfileSourcesSurviveConfigBoundaries(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
yaml string
|
||||||
|
want PromptKitConfig
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "profile directory",
|
||||||
|
yaml: "version: 4\npromptkit:\n profile_dir: ' ./profiles '\n",
|
||||||
|
want: PromptKitConfig{ProfileDir: "./profiles"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "profile file",
|
||||||
|
yaml: "version: 4\npromptkit:\n profile_file: ' ./profiles.yml '\n",
|
||||||
|
want: PromptKitConfig{ProfileFile: "./profiles.yml"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := applyFileConfig(t, tt.yaml)
|
||||||
|
if cfg.PromptKit != tt.want {
|
||||||
|
t.Fatalf("PromptKit config = %#v, want %#v", cfg.PromptKit, tt.want)
|
||||||
|
}
|
||||||
|
if got := cloneConfig(cfg).PromptKit; got != tt.want {
|
||||||
|
t.Fatalf("cloned PromptKit config = %#v, want %#v", got, tt.want)
|
||||||
|
}
|
||||||
|
if got := cfg.Redacted().PromptKit; got != tt.want {
|
||||||
|
t.Fatalf("redacted PromptKit config = %#v, want %#v", got, tt.want)
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := json.Marshal(cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("json.Marshal() error = %v", err)
|
||||||
|
}
|
||||||
|
var payload map[string]json.RawMessage
|
||||||
|
if err := json.Unmarshal(data, &payload); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal() error = %v", err)
|
||||||
|
}
|
||||||
|
if _, ok := payload["promptkit"]; !ok {
|
||||||
|
t.Fatalf("runtime JSON keys = %v, want promptkit", payload)
|
||||||
|
}
|
||||||
|
if _, ok := payload["scriptorium"]; ok {
|
||||||
|
t.Fatalf("runtime JSON keys = %v, must not contain removed section", payload)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitLocalBackendSurvivesConfigBoundaries(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
concurrencyYAML string
|
||||||
|
wantConcurrency int
|
||||||
|
}{
|
||||||
|
{name: "omitted concurrency defaults to zero"},
|
||||||
|
{name: "positive concurrency is preserved", concurrencyYAML: " concurrency_limit: 2\n", wantConcurrency: 2},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, "version: 4\npromptkit:\n local_backend:\n endpoint: ' http://localhost:8000/v1 '\n"+tt.concurrencyYAML)
|
||||||
|
cfg := Default()
|
||||||
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v", err)
|
||||||
|
}
|
||||||
|
want := PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: "http://localhost:8000/v1",
|
||||||
|
ConcurrencyLimit: tt.wantConcurrency,
|
||||||
|
}
|
||||||
|
if cfg.PromptKit.LocalBackend == nil || *cfg.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("local backend config = %#v, want %#v", cfg.PromptKit.LocalBackend, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
*file.PromptKit.LocalBackend.Endpoint = "http://changed.example/v1"
|
||||||
|
if file.PromptKit.LocalBackend.ConcurrencyLimit != nil {
|
||||||
|
*file.PromptKit.LocalBackend.ConcurrencyLimit = 99
|
||||||
|
}
|
||||||
|
if *cfg.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("effective config aliases parsed file model: %#v", cfg.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
|
||||||
|
cloned := cloneConfig(cfg)
|
||||||
|
if cloned.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *cloned.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("cloned local backend = %#v, want detached %#v", cloned.PromptKit.LocalBackend, want)
|
||||||
|
}
|
||||||
|
cloned.PromptKit.LocalBackend.Endpoint = "http://clone.example/v1"
|
||||||
|
if *cfg.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("mutating clone changed source config: %#v", cfg.PromptKit.LocalBackend)
|
||||||
|
}
|
||||||
|
|
||||||
|
redacted := cfg.Redacted()
|
||||||
|
if redacted.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend || *redacted.PromptKit.LocalBackend != want {
|
||||||
|
t.Fatalf("redacted local backend = %#v, want detached %#v", redacted.PromptKit.LocalBackend, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := json.Marshal(cfg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("json.Marshal() error = %v", err)
|
||||||
|
}
|
||||||
|
var payload struct {
|
||||||
|
PromptKit map[string]json.RawMessage `json:"promptkit"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(data, &payload); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal() error = %v", err)
|
||||||
|
}
|
||||||
|
localJSON, ok := payload.PromptKit["local_backend"]
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("runtime PromptKit JSON keys = %v, want local_backend", payload.PromptKit)
|
||||||
|
}
|
||||||
|
var localPayload map[string]json.RawMessage
|
||||||
|
if err := json.Unmarshal(localJSON, &localPayload); err != nil {
|
||||||
|
t.Fatalf("unmarshal local_backend JSON: %v", err)
|
||||||
|
}
|
||||||
|
if _, ok := localPayload["endpoint"]; !ok {
|
||||||
|
t.Fatalf("runtime local_backend JSON keys = %v, want endpoint", localPayload)
|
||||||
|
}
|
||||||
|
if _, ok := localPayload["concurrency_limit"]; !ok {
|
||||||
|
t.Fatalf("runtime local_backend JSON keys = %v, want concurrency_limit", localPayload)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitLocalBackendRequiresEndpoint(t *testing.T) {
|
||||||
|
for _, tt := range []struct {
|
||||||
|
name string
|
||||||
|
yaml string
|
||||||
|
}{
|
||||||
|
{name: "missing", yaml: "version: 4\npromptkit:\n local_backend: {}\n"},
|
||||||
|
{name: "empty", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ''\n"},
|
||||||
|
{name: "blank", yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: ' '\n"},
|
||||||
|
} {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, tt.yaml)
|
||||||
|
cfg := Default()
|
||||||
|
err := cfg.ApplyFileConfig(file)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "promptkit.local_backend.endpoint") {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v, want endpoint field context", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitExplicitEmptyProfileSourcesAreRejected(t *testing.T) {
|
||||||
|
for _, field := range []string{"profile_dir", "profile_file"} {
|
||||||
|
t.Run(field, func(t *testing.T) {
|
||||||
|
file := parseFileConfig(t, "version: 4\npromptkit:\n "+field+": ''\n")
|
||||||
|
cfg := Default()
|
||||||
|
err := cfg.ApplyFileConfig(file)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "promptkit."+field+" must not be empty") {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v, want explicit-empty rejection", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilePromptKitProfileSourcesRemainMutuallyExclusive(t *testing.T) {
|
||||||
|
cfg := applyFileConfig(t, `version: 4
|
||||||
|
promptkit:
|
||||||
|
profile_dir: ./profiles
|
||||||
|
profile_file: ./profiles.yml
|
||||||
|
`)
|
||||||
|
if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "promptkit profile_dir and profile_file are mutually exclusive") {
|
||||||
|
t.Fatalf("Validate() error = %v, want mutually exclusive profile sources", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestFileConfigMissingVersionIsReportedBeforeFieldDecoding(t *testing.T) {
|
func TestFileConfigMissingVersionIsReportedBeforeFieldDecoding(t *testing.T) {
|
||||||
_, err := ParseFileConfigYAML([]byte("workspace:\n directory: /tmp/old\n"))
|
_, err := ParseFileConfigYAML([]byte("workspace:\n directory: /tmp/old\n"))
|
||||||
if err == nil || !strings.Contains(err.Error(), "config version is required") {
|
if err == nil || !strings.Contains(err.Error(), "config version is required") {
|
||||||
@@ -55,6 +310,15 @@ func TestFileConfigMissingVersionIsReportedBeforeFieldDecoding(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFileConfigVersion3ReportsPromptKitMigration(t *testing.T) {
|
||||||
|
_, err := ParseFileConfigYAML([]byte("version: 3\nscriptorium:\n profile_dir: ./profiles\n"))
|
||||||
|
if err == nil ||
|
||||||
|
!strings.Contains(err.Error(), `change "version: 3" to "version: 4"`) ||
|
||||||
|
!strings.Contains(err.Error(), `rename "scriptorium:" to "promptkit:"`) {
|
||||||
|
t.Fatalf("version 3 error = %v, want actionable version and section migration", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -63,14 +327,19 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "removed diagnostics",
|
name: "removed diagnostics",
|
||||||
yaml: "version: 3\ndiagnostics: {}\n",
|
yaml: "version: 4\ndiagnostics: {}\n",
|
||||||
want: "field diagnostics not found",
|
want: "field diagnostics not found",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "removed llm profiles",
|
name: "removed llm profiles",
|
||||||
yaml: "version: 3\nllm_profiles: {}\n",
|
yaml: "version: 4\nllm_profiles: {}\n",
|
||||||
want: "field llm_profiles not found",
|
want: "field llm_profiles not found",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: "removed scriptorium section",
|
||||||
|
yaml: "version: 4\nscriptorium: {}\n",
|
||||||
|
want: "field scriptorium not found",
|
||||||
|
},
|
||||||
{
|
{
|
||||||
name: "version 2 migration",
|
name: "version 2 migration",
|
||||||
yaml: "version: 2\nworkspace:\n directory: /tmp/old\n",
|
yaml: "version: 2\nworkspace:\n directory: /tmp/old\n",
|
||||||
@@ -78,29 +347,34 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "pipeline field",
|
name: "pipeline field",
|
||||||
yaml: "version: 3\npipelines:\n main:\n unknown: true\n",
|
yaml: "version: 4\npipelines:\n main:\n unknown: true\n",
|
||||||
want: "field unknown not found",
|
want: "field unknown not found",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "lane field",
|
name: "lane field",
|
||||||
yaml: "version: 3\npipelines:\n main:\n artifacts:\n spells:\n unknown: true\n",
|
yaml: "version: 4\npipelines:\n main:\n artifacts:\n spells:\n unknown: true\n",
|
||||||
want: "field unknown not found",
|
want: "field unknown not found",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "module binding field",
|
name: "module binding field",
|
||||||
yaml: "version: 3\npipelines:\n main:\n input:\n module: seriatim\n unknown: true\n",
|
yaml: "version: 4\npipelines:\n main:\n input:\n module: seriatim\n unknown: true\n",
|
||||||
want: "field unknown not found in module binding",
|
want: "field unknown not found in module binding",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "checkpoint field",
|
name: "checkpoint field",
|
||||||
yaml: "version: 3\ncache:\n checkpoints:\n unknown: true\n",
|
yaml: "version: 4\ncache:\n checkpoints:\n unknown: true\n",
|
||||||
want: "field unknown not found",
|
want: "field unknown not found",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "checkpoint enabled type",
|
name: "checkpoint enabled type",
|
||||||
yaml: "version: 3\ncache:\n checkpoints:\n enabled: definitely\n",
|
yaml: "version: 4\ncache:\n checkpoints:\n enabled: definitely\n",
|
||||||
want: "cannot unmarshal",
|
want: "cannot unmarshal",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: "local backend field",
|
||||||
|
yaml: "version: 4\npromptkit:\n local_backend:\n endpoint: http://localhost:8000/v1\n unknown: true\n",
|
||||||
|
want: "field unknown not found",
|
||||||
|
},
|
||||||
}
|
}
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
@@ -113,7 +387,7 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigModuleBindingsPreserveFormsAndValidatorPresence(t *testing.T) {
|
func TestFileConfigModuleBindingsPreserveFormsAndValidatorPresence(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
main:
|
main:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
@@ -160,7 +434,7 @@ pipelines:
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigReferencePrecedenceIsRetained(t *testing.T) {
|
func TestFileConfigReferencePrecedenceIsRetained(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
main:
|
main:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
@@ -224,7 +498,7 @@ pipelines:
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigStageLocalValidatorsPreserveOrderAndFields(t *testing.T) {
|
func TestFileConfigStageLocalValidatorsPreserveOrderAndFields(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
main:
|
main:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
@@ -277,8 +551,8 @@ pipelines:
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigStateSectionsApplyIndependently(t *testing.T) {
|
func TestFileConfigStateSectionsApplyIndependently(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, `version: 3
|
cfg := applyFileConfig(t, `version: 4
|
||||||
scriptorium:
|
promptkit:
|
||||||
profile_dir: ./profiles
|
profile_dir: ./profiles
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 7
|
total_llm: 7
|
||||||
@@ -294,15 +568,15 @@ cache:
|
|||||||
debug:
|
debug:
|
||||||
directory: ./debug
|
directory: ./debug
|
||||||
`)
|
`)
|
||||||
if cfg.Scriptorium.ProfileDir != "./profiles" || cfg.Scriptorium.ProfileFile != "" {
|
if cfg.PromptKit.ProfileDir != "./profiles" || cfg.PromptKit.ProfileFile != "" {
|
||||||
t.Fatalf("scriptorium = %#v", cfg.Scriptorium)
|
t.Fatalf("promptkit = %#v", cfg.PromptKit)
|
||||||
}
|
}
|
||||||
if cfg.Concurrency.TotalLLM != 7 || cfg.Concurrency.StageWorkers["extract"] != 7 {
|
if cfg.Concurrency.TotalLLM != 7 || cfg.Concurrency.StageWorkers["extract"] != 7 {
|
||||||
t.Fatalf("concurrency = %#v", cfg.Concurrency)
|
t.Fatalf("concurrency = %#v", cfg.Concurrency)
|
||||||
}
|
}
|
||||||
if cfg.Output.Directory != "./output" || cfg.Cache.ChunkPlans.Directory != "plans" || cfg.Cache.ChunkPlans.Mode != pipeline.ChunkCacheBypass ||
|
if cfg.Output.Directory != "./output" || cfg.Cache.ChunkPlans.Directory != "plans" || cfg.Cache.ChunkPlans.Mode != pipeline.ChunkCacheBypass ||
|
||||||
!cfg.Cache.Checkpoints.Enabled || cfg.Cache.Checkpoints.Directory != "checkpoints" || cfg.Debug.Directory != "./debug" {
|
!cfg.Cache.Checkpoints.Enabled || cfg.Cache.Checkpoints.Directory != "checkpoints" || cfg.Debug.Directory != "./debug" {
|
||||||
t.Fatalf("state sections = %#v, %#v, %#v, %#v", cfg.Output, cfg.Cache, cfg.Debug, cfg.Scriptorium)
|
t.Fatalf("state sections = %#v, %#v, %#v, %#v", cfg.Output, cfg.Cache, cfg.Debug, cfg.PromptKit)
|
||||||
}
|
}
|
||||||
if cfg.Output.Directory == cfg.Cache.ChunkPlans.Directory || cfg.Cache.ChunkPlans.Directory == cfg.Cache.Checkpoints.Directory || cfg.Cache.Checkpoints.Directory == cfg.Debug.Directory {
|
if cfg.Output.Directory == cfg.Cache.ChunkPlans.Directory || cfg.Cache.ChunkPlans.Directory == cfg.Cache.Checkpoints.Directory || cfg.Cache.Checkpoints.Directory == cfg.Debug.Directory {
|
||||||
t.Fatal("state roots were coupled")
|
t.Fatal("state roots were coupled")
|
||||||
@@ -310,11 +584,11 @@ debug:
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigCheckpointEnabledCanBeExplicitlyDisabled(t *testing.T) {
|
func TestFileConfigCheckpointEnabledCanBeExplicitlyDisabled(t *testing.T) {
|
||||||
cfg := applyFileConfig(t, "version: 3\ncache:\n checkpoints:\n enabled: true\n")
|
cfg := applyFileConfig(t, "version: 4\ncache:\n checkpoints:\n enabled: true\n")
|
||||||
if !cfg.Cache.Checkpoints.Enabled || !cloneConfig(cfg).Cache.Checkpoints.Enabled {
|
if !cfg.Cache.Checkpoints.Enabled || !cloneConfig(cfg).Cache.Checkpoints.Enabled {
|
||||||
t.Fatalf("enabled checkpoint config was not retained: %#v", cfg.Cache.Checkpoints)
|
t.Fatalf("enabled checkpoint config was not retained: %#v", cfg.Cache.Checkpoints)
|
||||||
}
|
}
|
||||||
file := parseFileConfig(t, "version: 3\ncache:\n checkpoints:\n enabled: false\n")
|
file := parseFileConfig(t, "version: 4\ncache:\n checkpoints:\n enabled: false\n")
|
||||||
if err := cfg.ApplyFileConfig(file); err != nil {
|
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -331,17 +605,17 @@ func TestFileConfigRejectsTrimmedKeyCollisions(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "pipeline ids",
|
name: "pipeline ids",
|
||||||
yaml: "version: 3\npipelines:\n main: {}\n ' main ': {}\n",
|
yaml: "version: 4\npipelines:\n main: {}\n ' main ': {}\n",
|
||||||
want: "pipeline id \"main\" is duplicated after trimming",
|
want: "pipeline id \"main\" is duplicated after trimming",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "lane ids",
|
name: "lane ids",
|
||||||
yaml: "version: 3\npipelines:\n main:\n artifacts:\n spells: {}\n ' spells ': {}\n",
|
yaml: "version: 4\npipelines:\n main:\n artifacts:\n spells: {}\n ' spells ': {}\n",
|
||||||
want: "artifact lane id \"spells\" is duplicated after trimming",
|
want: "artifact lane id \"spells\" is duplicated after trimming",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "reference slots",
|
name: "reference slots",
|
||||||
yaml: "version: 3\npipelines:\n main:\n references:\n slot: ./one.txt\n ' slot ': ./two.txt\n",
|
yaml: "version: 4\npipelines:\n main:\n references:\n slot: ./one.txt\n ' slot ': ./two.txt\n",
|
||||||
want: "reference slot \"slot\" is duplicated after trimming",
|
want: "reference slot \"slot\" is duplicated after trimming",
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -358,7 +632,7 @@ func TestFileConfigRejectsTrimmedKeyCollisions(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFileConfigParsesOrderedStepsAndReferenceSources(t *testing.T) {
|
func TestFileConfigParsesOrderedStepsAndReferenceSources(t *testing.T) {
|
||||||
file := parseFileConfig(t, `version: 3
|
file := parseFileConfig(t, `version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
session:
|
session:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
@@ -397,7 +671,7 @@ func TestFileConfigRejectsAmbiguousReferenceSourceForms(t *testing.T) {
|
|||||||
"artifact: {step: 1, lane: b}",
|
"artifact: {step: 1, lane: b}",
|
||||||
"1",
|
"1",
|
||||||
} {
|
} {
|
||||||
_, err := ParseFileConfigYAML([]byte("version: 3\npipelines:\n p:\n input: text\n references:\n slot: " + source + "\n"))
|
_, err := ParseFileConfigYAML([]byte("version: 4\npipelines:\n p:\n input: text\n references:\n slot: " + source + "\n"))
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatalf("ParseFileConfigYAML(%q) error = nil", source)
|
t.Fatalf("ParseFileConfigYAML(%q) error = nil", source)
|
||||||
}
|
}
|
||||||
@@ -412,12 +686,12 @@ func TestFileConfigRejectsEmptyAndAmbiguousPipelineShapes(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "empty steps",
|
name: "empty steps",
|
||||||
yaml: "version: 3\npipelines:\n p:\n input: text\n steps: []\n",
|
yaml: "version: 4\npipelines:\n p:\n input: text\n steps: []\n",
|
||||||
want: "at least one ordered step",
|
want: "at least one ordered step",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "both forms",
|
name: "both forms",
|
||||||
yaml: "version: 3\npipelines:\n p:\n input: text\n artifacts: {}\n steps: []\n",
|
yaml: "version: 4\npipelines:\n p:\n input: text\n artifacts: {}\n steps: []\n",
|
||||||
want: "both artifacts and steps",
|
want: "both artifacts and steps",
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -22,7 +22,9 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
|
|||||||
ID: "redaction-test",
|
ID: "redaction-test",
|
||||||
Digest: "sha256:safe-digest",
|
Digest: "sha256:safe-digest",
|
||||||
Input: bindings["input"],
|
Input: bindings["input"],
|
||||||
|
InputExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Chunk: bindings["chunk"],
|
Chunk: bindings["chunk"],
|
||||||
|
ChunkExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
ChunkReferences: redactionTestReferenceTarget(pipeline.StageChunk, "", "chunk-reference-content"),
|
ChunkReferences: redactionTestReferenceTarget(pipeline.StageChunk, "", "chunk-reference-content"),
|
||||||
Steps: []pipeline.ResolvedPipelineStep{{
|
Steps: []pipeline.ResolvedPipelineStep{{
|
||||||
ID: "default",
|
ID: "default",
|
||||||
@@ -30,8 +32,11 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
|
|||||||
ID: "safe-lane",
|
ID: "safe-lane",
|
||||||
ArtifactKind: "safe/artifact",
|
ArtifactKind: "safe/artifact",
|
||||||
Extract: bindings["extract"],
|
Extract: bindings["extract"],
|
||||||
|
ExtractExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Merge: bindings["merge"],
|
Merge: bindings["merge"],
|
||||||
|
MergeExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Normalize: bindings["normalize"],
|
Normalize: bindings["normalize"],
|
||||||
|
NormalizeExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||||
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
|
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
|
||||||
ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"),
|
ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"),
|
||||||
MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"),
|
MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"),
|
||||||
@@ -50,6 +55,7 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
|
|||||||
}},
|
}},
|
||||||
}},
|
}},
|
||||||
Output: bindings["output"],
|
Output: bindings["output"],
|
||||||
|
OutputExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
}
|
}
|
||||||
effective := EffectiveConfig{
|
effective := EffectiveConfig{
|
||||||
Config: Config{Pipelines: map[string]pipeline.PipelineProfile{
|
Config: Config{Pipelines: map[string]pipeline.PipelineProfile{
|
||||||
@@ -88,6 +94,11 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
|
|||||||
t.Fatalf("resolved pipeline summary does not retain %q: %s", safe, text)
|
t.Fatalf("resolved pipeline summary does not retain %q: %s", safe, text)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
for _, executionClass := range []string{"input_execution_class\":\"deterministic", "chunk_execution_class\":\"llm_backed", "extract_execution_class\":\"llm_backed", "merge_execution_class\":\"deterministic", "normalize_execution_class\":\"llm_backed", "output_execution_class\":\"deterministic"} {
|
||||||
|
if !strings.Contains(text, executionClass) {
|
||||||
|
t.Fatalf("resolved pipeline summary does not retain %q: %s", executionClass, text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
payload.Input.Options["safe"] = "mutated"
|
payload.Input.Options["safe"] = "mutated"
|
||||||
nested := payload.Input.Options["nested"].([]any)[0].([]any)[0].(map[string]any)
|
nested := payload.Input.Options["nested"].([]any)[0].([]any)[0].(map[string]any)
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package config
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"net/url"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
@@ -10,7 +11,7 @@ import (
|
|||||||
|
|
||||||
func (c Config) Validate() error {
|
func (c Config) Validate() error {
|
||||||
c.Concurrency.recomputeStageWorkerDefaults()
|
c.Concurrency.recomputeStageWorkerDefaults()
|
||||||
if err := validateScriptorium(c.Scriptorium); err != nil {
|
if err := validatePromptKit(c.PromptKit); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if err := validateStateSurfaces(c); err != nil {
|
if err := validateStateSurfaces(c); err != nil {
|
||||||
@@ -49,9 +50,30 @@ func validateStageWorkers(cfg ConcurrencyConfig) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func validateScriptorium(cfg ScriptoriumConfig) error {
|
func validatePromptKit(cfg PromptKitConfig) error {
|
||||||
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
||||||
return fmt.Errorf("scriptorium profile_dir and profile_file are mutually exclusive")
|
return fmt.Errorf("promptkit profile_dir and profile_file are mutually exclusive")
|
||||||
|
}
|
||||||
|
if cfg.LocalBackend == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
endpoint := strings.TrimSpace(cfg.LocalBackend.Endpoint)
|
||||||
|
if endpoint == "" {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must not be empty when set")
|
||||||
|
}
|
||||||
|
parsed, err := url.Parse(endpoint)
|
||||||
|
if err != nil ||
|
||||||
|
(!strings.EqualFold(parsed.Scheme, "http") && !strings.EqualFold(parsed.Scheme, "https")) ||
|
||||||
|
!parsed.IsAbs() ||
|
||||||
|
parsed.Hostname() == "" ||
|
||||||
|
parsed.User != nil ||
|
||||||
|
parsed.RawQuery != "" ||
|
||||||
|
parsed.ForceQuery ||
|
||||||
|
strings.Contains(endpoint, "#") {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.endpoint must be an absolute HTTP or HTTPS URL with a host and no user information, query, or fragment")
|
||||||
|
}
|
||||||
|
if cfg.LocalBackend.ConcurrencyLimit < 0 {
|
||||||
|
return fmt.Errorf("promptkit.local_backend.concurrency_limit must not be negative")
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -87,10 +87,77 @@ func TestValidateConcurrencyRules(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestValidateScriptoriumSourcesAreMutuallyExclusive(t *testing.T) {
|
func TestValidatePromptKitSourcesAreMutuallyExclusive(t *testing.T) {
|
||||||
cfg := Default()
|
cfg := Default()
|
||||||
cfg.Scriptorium = ScriptoriumConfig{ProfileDir: "./profiles", ProfileFile: "./profile.yml"}
|
cfg.PromptKit = PromptKitConfig{ProfileDir: "./profiles", ProfileFile: "./profile.yml"}
|
||||||
assertValidationContains(t, cfg, "scriptorium profile_dir and profile_file are mutually exclusive")
|
assertValidationContains(t, cfg, "promptkit profile_dir and profile_file are mutually exclusive")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatePromptKitLocalBackendEndpoints(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
endpoint string
|
||||||
|
profileSource PromptKitConfig
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "HTTP endpoint with path and profile directory",
|
||||||
|
endpoint: "http://localhost:8000/v1",
|
||||||
|
profileSource: PromptKitConfig{ProfileDir: "./profiles"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "case-insensitive HTTPS endpoint and profile file",
|
||||||
|
endpoint: "HTTPS://inference.example.test/api",
|
||||||
|
profileSource: PromptKitConfig{ProfileFile: "./profiles.yml"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := Default()
|
||||||
|
cfg.PromptKit = tt.profileSource
|
||||||
|
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: tt.endpoint,
|
||||||
|
ConcurrencyLimit: 2,
|
||||||
|
}
|
||||||
|
if err := cfg.Validate(); err != nil {
|
||||||
|
t.Fatalf("Validate() error = %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatePromptKitLocalBackendRejectsInvalidValues(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
endpoint string
|
||||||
|
concurrencyLimit int
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "blank endpoint", endpoint: " ", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "relative URL", endpoint: "localhost:8000/v1", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "unsupported scheme", endpoint: "ftp://localhost/model", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "missing host", endpoint: "http:///v1", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "user information", endpoint: "http://user:secret@localhost/v1", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "query", endpoint: "http://localhost/v1?model=example", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "empty query", endpoint: "http://localhost/v1?", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "fragment", endpoint: "http://localhost/v1#model", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{name: "empty fragment", endpoint: "http://localhost/v1#", want: "promptkit.local_backend.endpoint"},
|
||||||
|
{
|
||||||
|
name: "negative concurrency",
|
||||||
|
endpoint: "http://localhost:8000/v1",
|
||||||
|
concurrencyLimit: -1,
|
||||||
|
want: "promptkit.local_backend.concurrency_limit",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
cfg := Default()
|
||||||
|
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
|
||||||
|
Endpoint: tt.endpoint,
|
||||||
|
ConcurrencyLimit: tt.concurrencyLimit,
|
||||||
|
}
|
||||||
|
assertValidationContains(t, cfg, tt.want)
|
||||||
|
})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestValidateStateSurfaceRules(t *testing.T) {
|
func TestValidateStateSurfaceRules(t *testing.T) {
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package debugbundle
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -137,6 +138,47 @@ func TestSummaryWriterWritesEverySummaryArtifact(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestWriteInvocationPreservesReasoningEffortOverrideStates(t *testing.T) {
|
||||||
|
replacement := "focused"
|
||||||
|
cleared := ""
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
override *string
|
||||||
|
wantValue string
|
||||||
|
wantSet bool
|
||||||
|
}{
|
||||||
|
{name: "inherit"},
|
||||||
|
{name: "replace", override: &replacement, wantValue: "focused", wantSet: true},
|
||||||
|
{name: "clear", override: &cleared, wantSet: true},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
bundle, err := Allocate(t.TempDir(), testBundleRunID, time.Unix(0, 42))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := bundle.Summary().WriteInvocation(Invocation{
|
||||||
|
Operation: "run",
|
||||||
|
ReasoningEffortOverride: tt.override,
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
data, err := os.ReadFile(filepath.Join(bundle.SummaryRoot(), ArtifactInvocationMetadata))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var payload map[string]any
|
||||||
|
if err := json.Unmarshal(data, &payload); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
value, found := payload["reasoning_effort_override"]
|
||||||
|
if found != tt.wantSet || (found && value != tt.wantValue) {
|
||||||
|
t.Fatalf("reasoning override found=%t value=%#v, want found=%t value=%q; JSON=%s", found, value, tt.wantSet, tt.wantValue, data)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
func TestSummaryWriterInternalWritesConfineArtifacts(t *testing.T) {
|
func TestSummaryWriterInternalWritesConfineArtifacts(t *testing.T) {
|
||||||
bundle, err := Allocate(t.TempDir(), testBundleRunID, time.Unix(0, 42))
|
bundle, err := Allocate(t.TempDir(), testBundleRunID, time.Unix(0, 42))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -39,6 +39,7 @@ type Invocation struct {
|
|||||||
ConfigSource string `json:"config_source,omitempty"`
|
ConfigSource string `json:"config_source,omitempty"`
|
||||||
OnlyLanes []string `json:"only_lanes,omitempty"`
|
OnlyLanes []string `json:"only_lanes,omitempty"`
|
||||||
ChunkCacheOverride string `json:"chunk_cache_override,omitempty"`
|
ChunkCacheOverride string `json:"chunk_cache_override,omitempty"`
|
||||||
|
ReasoningEffortOverride *string `json:"reasoning_effort_override,omitempty"`
|
||||||
RunID string `json:"run_id"`
|
RunID string `json:"run_id"`
|
||||||
StartedAt time.Time `json:"started_at"`
|
StartedAt time.Time `json:"started_at"`
|
||||||
}
|
}
|
||||||
@@ -68,6 +69,10 @@ func (w *SummaryWriter) WriteInvocation(payload Invocation) error {
|
|||||||
if payload.StartedAt.IsZero() {
|
if payload.StartedAt.IsZero() {
|
||||||
payload.StartedAt = w.createdAt
|
payload.StartedAt = w.createdAt
|
||||||
}
|
}
|
||||||
|
if payload.ReasoningEffortOverride != nil {
|
||||||
|
value := *payload.ReasoningEffortOverride
|
||||||
|
payload.ReasoningEffortOverride = &value
|
||||||
|
}
|
||||||
return w.writeJSON(ArtifactInvocationMetadata, payload)
|
return w.writeJSON(ArtifactInvocationMetadata, payload)
|
||||||
}
|
}
|
||||||
func (w *SummaryWriter) WriteRedactedEffectiveConfig(payload RedactedSummaryPayload) error {
|
func (w *SummaryWriter) WriteRedactedEffectiveConfig(payload RedactedSummaryPayload) error {
|
||||||
|
|||||||
@@ -43,6 +43,7 @@ type LLMDebugPrompt struct {
|
|||||||
PromptVersion string `json:"prompt_version,omitempty"`
|
PromptVersion string `json:"prompt_version,omitempty"`
|
||||||
PromptHash string `json:"prompt_hash,omitempty"`
|
PromptHash string `json:"prompt_hash,omitempty"`
|
||||||
SelectedProfileID string `json:"selected_profile_id,omitempty"`
|
SelectedProfileID string `json:"selected_profile_id,omitempty"`
|
||||||
|
SelectedBackendID string `json:"selected_backend_id,omitempty"`
|
||||||
SessionID string `json:"session_id,omitempty"`
|
SessionID string `json:"session_id,omitempty"`
|
||||||
RenderedPromptHash string `json:"rendered_prompt_hash,omitempty"`
|
RenderedPromptHash string `json:"rendered_prompt_hash,omitempty"`
|
||||||
Messages []LLMDebugMessage `json:"messages,omitempty"`
|
Messages []LLMDebugMessage `json:"messages,omitempty"`
|
||||||
@@ -266,10 +267,6 @@ const (
|
|||||||
ExecutionClassLLMBacked ExecutionClass = "llm_backed"
|
ExecutionClassLLMBacked ExecutionClass = "llm_backed"
|
||||||
)
|
)
|
||||||
|
|
||||||
type ChunkExecutionClassProvider interface {
|
|
||||||
ExecutionClass() ExecutionClass
|
|
||||||
}
|
|
||||||
|
|
||||||
type ValidationResult struct {
|
type ValidationResult struct {
|
||||||
Approved bool `json:"approved"`
|
Approved bool `json:"approved"`
|
||||||
ReasonCode string `json:"reason_code,omitempty"`
|
ReasonCode string `json:"reason_code,omitempty"`
|
||||||
@@ -292,6 +289,7 @@ type OutputRequest struct {
|
|||||||
LLMProfile string `json:"llm_profile,omitempty"`
|
LLMProfile string `json:"llm_profile,omitempty"`
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
Metadata map[string]any `json:"metadata,omitempty"`
|
||||||
ChunkMap *SerializedArtifact `json:"chunk_map,omitempty"`
|
ChunkMap *SerializedArtifact `json:"chunk_map,omitempty"`
|
||||||
|
EvidenceContext *SerializedArtifact `json:"evidence_context,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type OutputFile struct {
|
type OutputFile struct {
|
||||||
|
|||||||
@@ -5,3 +5,7 @@ import "errors"
|
|||||||
// ErrInvalidStructuredOutput identifies a provider response that cannot satisfy
|
// ErrInvalidStructuredOutput identifies a provider response that cannot satisfy
|
||||||
// the caller's declared structured-output contract.
|
// the caller's declared structured-output contract.
|
||||||
var ErrInvalidStructuredOutput = errors.New("invalid structured output")
|
var ErrInvalidStructuredOutput = errors.New("invalid structured output")
|
||||||
|
|
||||||
|
// ErrLLMCapacityExceeded identifies backend admission exhaustion before model
|
||||||
|
// generation begins.
|
||||||
|
var ErrLLMCapacityExceeded = errors.New("LLM capacity exceeded")
|
||||||
|
|||||||
@@ -0,0 +1,66 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"$id": "notarius.source.evidence_context",
|
||||||
|
"title": "notarius_source_evidence_context_v1",
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["source_id", "source_digest", "window_units", "selected_lanes", "contexts"],
|
||||||
|
"properties": {
|
||||||
|
"source_id": {"type": "string", "minLength": 1},
|
||||||
|
"source_digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
|
||||||
|
"window_units": {"type": "integer", "minimum": 0},
|
||||||
|
"selected_lanes": {
|
||||||
|
"type": "array",
|
||||||
|
"minItems": 1,
|
||||||
|
"uniqueItems": true,
|
||||||
|
"items": {"type": "string", "minLength": 1}
|
||||||
|
},
|
||||||
|
"contexts": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {"$ref": "#/$defs/context"}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"$defs": {
|
||||||
|
"source_ref": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["source_id", "start_unit_id", "end_unit_id"],
|
||||||
|
"properties": {
|
||||||
|
"source_id": {"type": "string", "minLength": 1},
|
||||||
|
"start_unit_id": {"type": "integer", "minimum": 1},
|
||||||
|
"end_unit_id": {"type": "integer", "minimum": 1}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"unit": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["id", "kind", "text", "ref"],
|
||||||
|
"properties": {
|
||||||
|
"id": {"type": "integer", "minimum": 1},
|
||||||
|
"kind": {"type": "string", "minLength": 1},
|
||||||
|
"text": {"type": "string", "minLength": 1},
|
||||||
|
"ref": {"$ref": "#/$defs/source_ref"},
|
||||||
|
"metadata": {"type": "object", "additionalProperties": true}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"evidence_ref": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["lane_id", "source_ref"],
|
||||||
|
"properties": {
|
||||||
|
"lane_id": {"type": "string", "minLength": 1},
|
||||||
|
"source_ref": {"$ref": "#/$defs/source_ref"}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"context": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["context_ref", "evidence_refs", "units"],
|
||||||
|
"properties": {
|
||||||
|
"context_ref": {"$ref": "#/$defs/source_ref"},
|
||||||
|
"evidence_refs": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/evidence_ref"}},
|
||||||
|
"units": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/unit"}}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
196
internal/framework/evidencecontext/build.go
Normal file
196
internal/framework/evidencecontext/build.go
Normal file
@@ -0,0 +1,196 @@
|
|||||||
|
package evidencecontext
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
)
|
||||||
|
|
||||||
|
type contribution struct {
|
||||||
|
laneID string
|
||||||
|
ref source.SourceRef
|
||||||
|
startPos int
|
||||||
|
endPos int
|
||||||
|
}
|
||||||
|
|
||||||
|
type expandedRange struct {
|
||||||
|
startPos int
|
||||||
|
endPos int
|
||||||
|
contributions []contribution
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build validates accepted direct references, expands them by source-document
|
||||||
|
// position, and returns their deterministic context union.
|
||||||
|
func Build(request BuildRequest) (Document, error) {
|
||||||
|
if request.WindowUnits < 0 {
|
||||||
|
return Document{}, fmt.Errorf("window_units must not be negative")
|
||||||
|
}
|
||||||
|
lanes, err := normalizeSelectedLanes(request.SelectedLanes)
|
||||||
|
if err != nil {
|
||||||
|
return Document{}, err
|
||||||
|
}
|
||||||
|
if err := source.ValidateDocument(request.Source); err != nil {
|
||||||
|
return Document{}, fmt.Errorf("validate source document: %w", err)
|
||||||
|
}
|
||||||
|
digest, err := source.DigestDocument(request.Source)
|
||||||
|
if err != nil {
|
||||||
|
return Document{}, fmt.Errorf("digest source document: %w", err)
|
||||||
|
}
|
||||||
|
if digest != request.Source.Digest {
|
||||||
|
return Document{}, fmt.Errorf("source digest does not match source document digest")
|
||||||
|
}
|
||||||
|
|
||||||
|
selected := make(map[string]struct{}, len(lanes))
|
||||||
|
for _, laneID := range lanes {
|
||||||
|
selected[laneID] = struct{}{}
|
||||||
|
}
|
||||||
|
index := source.NewDocumentIndex(request.Source)
|
||||||
|
seen := make(map[evidenceKey]struct{})
|
||||||
|
contributions := make([]contribution, 0)
|
||||||
|
for laneIndex, laneEvidence := range request.LaneEvidence {
|
||||||
|
laneID := strings.TrimSpace(laneEvidence.LaneID)
|
||||||
|
if _, ok := selected[laneID]; !ok {
|
||||||
|
return Document{}, fmt.Errorf("lane evidence[%d] lane %q is not selected", laneIndex, laneID)
|
||||||
|
}
|
||||||
|
for refIndex, ref := range laneEvidence.SourceRefs {
|
||||||
|
if err := index.ValidateRef(ref); err != nil {
|
||||||
|
return Document{}, fmt.Errorf("lane %q source reference[%d]: %w", laneID, refIndex, err)
|
||||||
|
}
|
||||||
|
key := evidenceKey{laneID: laneID, ref: ref}
|
||||||
|
if _, exists := seen[key]; exists {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[key] = struct{}{}
|
||||||
|
startPos, _ := index.Position(ref.StartUnitID)
|
||||||
|
endPos, _ := index.Position(ref.EndUnitID)
|
||||||
|
contributions = append(contributions, contribution{laneID: laneID, ref: ref, startPos: expandStart(startPos, request.WindowUnits), endPos: expandEnd(endPos, len(request.Source.Units), request.WindowUnits)})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
sort.Slice(contributions, func(i, j int) bool { return lessContribution(contributions[i], contributions[j]) })
|
||||||
|
document := Document{
|
||||||
|
SourceID: request.Source.ID,
|
||||||
|
SourceDigest: digest,
|
||||||
|
WindowUnits: request.WindowUnits,
|
||||||
|
SelectedLanes: lanes,
|
||||||
|
Contexts: make([]Context, 0),
|
||||||
|
}
|
||||||
|
for _, rangeValue := range mergeRanges(contributions) {
|
||||||
|
context, err := buildContext(request.Source, rangeValue)
|
||||||
|
if err != nil {
|
||||||
|
return Document{}, err
|
||||||
|
}
|
||||||
|
document.Contexts = append(document.Contexts, context)
|
||||||
|
}
|
||||||
|
canonical, err := canonicalize(document)
|
||||||
|
if err != nil {
|
||||||
|
return Document{}, fmt.Errorf("validate evidence context: %w", err)
|
||||||
|
}
|
||||||
|
return clone(canonical)
|
||||||
|
}
|
||||||
|
|
||||||
|
type evidenceKey struct {
|
||||||
|
laneID string
|
||||||
|
ref source.SourceRef
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizeSelectedLanes(values []string) ([]string, error) {
|
||||||
|
if len(values) == 0 {
|
||||||
|
return nil, fmt.Errorf("selected_lanes must not be empty")
|
||||||
|
}
|
||||||
|
seen := make(map[string]struct{}, len(values))
|
||||||
|
lanes := make([]string, 0, len(values))
|
||||||
|
for index, raw := range values {
|
||||||
|
laneID := strings.TrimSpace(raw)
|
||||||
|
if laneID == "" {
|
||||||
|
return nil, fmt.Errorf("selected_lanes[%d] must not be empty", index)
|
||||||
|
}
|
||||||
|
if _, exists := seen[laneID]; exists {
|
||||||
|
return nil, fmt.Errorf("selected_lanes lane %q is duplicated", laneID)
|
||||||
|
}
|
||||||
|
seen[laneID] = struct{}{}
|
||||||
|
lanes = append(lanes, laneID)
|
||||||
|
}
|
||||||
|
sort.Strings(lanes)
|
||||||
|
return lanes, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func expandStart(position, window int) int {
|
||||||
|
if window > position {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return position - window
|
||||||
|
}
|
||||||
|
|
||||||
|
func expandEnd(position, length, window int) int {
|
||||||
|
last := length - 1
|
||||||
|
if window > last-position {
|
||||||
|
return last
|
||||||
|
}
|
||||||
|
return position + window
|
||||||
|
}
|
||||||
|
|
||||||
|
func lessContribution(left, right contribution) bool {
|
||||||
|
if left.startPos != right.startPos {
|
||||||
|
return left.startPos < right.startPos
|
||||||
|
}
|
||||||
|
if left.endPos != right.endPos {
|
||||||
|
return left.endPos < right.endPos
|
||||||
|
}
|
||||||
|
return lessEvidenceRef(EvidenceRef{LaneID: left.laneID, SourceRef: left.ref}, EvidenceRef{LaneID: right.laneID, SourceRef: right.ref})
|
||||||
|
}
|
||||||
|
|
||||||
|
func mergeRanges(values []contribution) []expandedRange {
|
||||||
|
if len(values) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
ranges := make([]expandedRange, 0, len(values))
|
||||||
|
for _, value := range values {
|
||||||
|
if len(ranges) == 0 || value.startPos > ranges[len(ranges)-1].endPos+1 {
|
||||||
|
ranges = append(ranges, expandedRange{startPos: value.startPos, endPos: value.endPos, contributions: []contribution{value}})
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
current := &ranges[len(ranges)-1]
|
||||||
|
if value.endPos > current.endPos {
|
||||||
|
current.endPos = value.endPos
|
||||||
|
}
|
||||||
|
current.contributions = append(current.contributions, value)
|
||||||
|
}
|
||||||
|
return ranges
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildContext(document *source.SourceDocument, value expandedRange) (Context, error) {
|
||||||
|
evidenceRefs := make([]EvidenceRef, 0, len(value.contributions))
|
||||||
|
for _, contribution := range value.contributions {
|
||||||
|
evidenceRefs = append(evidenceRefs, EvidenceRef{LaneID: contribution.laneID, SourceRef: contribution.ref})
|
||||||
|
}
|
||||||
|
sort.Slice(evidenceRefs, func(i, j int) bool { return lessEvidenceRef(evidenceRefs[i], evidenceRefs[j]) })
|
||||||
|
units := make([]source.SourceUnit, 0, value.endPos-value.startPos+1)
|
||||||
|
for position := value.startPos; position <= value.endPos; position++ {
|
||||||
|
unit, err := cloneSourceUnit(document.Units[position])
|
||||||
|
if err != nil {
|
||||||
|
return Context{}, fmt.Errorf("clone source unit at position %d: %w", position, err)
|
||||||
|
}
|
||||||
|
units = append(units, unit)
|
||||||
|
}
|
||||||
|
return Context{
|
||||||
|
ContextRef: source.SourceRef{SourceID: document.ID, StartUnitID: units[0].ID, EndUnitID: units[len(units)-1].ID},
|
||||||
|
EvidenceRefs: evidenceRefs,
|
||||||
|
Units: units,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func lessEvidenceRef(left, right EvidenceRef) bool {
|
||||||
|
if left.LaneID != right.LaneID {
|
||||||
|
return left.LaneID < right.LaneID
|
||||||
|
}
|
||||||
|
if left.SourceRef.SourceID != right.SourceRef.SourceID {
|
||||||
|
return left.SourceRef.SourceID < right.SourceRef.SourceID
|
||||||
|
}
|
||||||
|
if left.SourceRef.StartUnitID != right.SourceRef.StartUnitID {
|
||||||
|
return left.SourceRef.StartUnitID < right.SourceRef.StartUnitID
|
||||||
|
}
|
||||||
|
return left.SourceRef.EndUnitID < right.SourceRef.EndUnitID
|
||||||
|
}
|
||||||
329
internal/framework/evidencecontext/codec.go
Normal file
329
internal/framework/evidencecontext/codec.go
Normal file
@@ -0,0 +1,329 @@
|
|||||||
|
package evidencecontext
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"embed"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"github.com/santhosh-tekuri/jsonschema/v6"
|
||||||
|
)
|
||||||
|
|
||||||
|
//go:embed assets/schemas/source_evidence_context.v1.json
|
||||||
|
var schemaAssets embed.FS
|
||||||
|
|
||||||
|
var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
|
||||||
|
|
||||||
|
var (
|
||||||
|
loadSchemaOnce sync.Once
|
||||||
|
loadedSchema []byte
|
||||||
|
compiledSchema *jsonschema.Schema
|
||||||
|
loadSchemaErr error
|
||||||
|
)
|
||||||
|
|
||||||
|
// Codec owns strict serialization for the durable evidence-context contract.
|
||||||
|
type Codec struct{}
|
||||||
|
|
||||||
|
func New() *Codec { return &Codec{} }
|
||||||
|
|
||||||
|
func (c *Codec) Kind() contracts.ArtifactKind { return ArtifactKind }
|
||||||
|
|
||||||
|
func (c *Codec) Schema() contracts.ArtifactSchema {
|
||||||
|
raw, err := c.schemaBytes()
|
||||||
|
if err != nil {
|
||||||
|
return contracts.ArtifactSchema{}
|
||||||
|
}
|
||||||
|
return contracts.ArtifactSchema{ID: SchemaID, Name: SchemaName, Version: SchemaVersion, JSONSchema: raw}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Codec) MediaType() string { return MediaType }
|
||||||
|
|
||||||
|
// Serialize builds and encodes the framework-owned serialized artifact.
|
||||||
|
func Serialize(request BuildRequest) (contracts.SerializedArtifact, error) {
|
||||||
|
value, err := Build(request)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.SerializedArtifact{}, err
|
||||||
|
}
|
||||||
|
codec := New()
|
||||||
|
content, err := codec.Encode(value)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.SerializedArtifact{}, err
|
||||||
|
}
|
||||||
|
return contracts.SerializedArtifact{Kind: ArtifactKind, Schema: codec.Schema(), MediaType: MediaType, Content: content}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Codec) Encode(value Document) ([]byte, error) {
|
||||||
|
if _, err := c.schemaBytes(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
canonical, err := canonicalize(value)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("encode evidence context: %w", err)
|
||||||
|
}
|
||||||
|
content, err := json.Marshal(canonical)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("encode evidence context: %w", err)
|
||||||
|
}
|
||||||
|
if err := validateSchemaInstance(content); err != nil {
|
||||||
|
return nil, fmt.Errorf("encode evidence context: %w", err)
|
||||||
|
}
|
||||||
|
return content, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Codec) Decode(content []byte) (Document, error) {
|
||||||
|
if _, err := c.schemaBytes(); err != nil {
|
||||||
|
return Document{}, err
|
||||||
|
}
|
||||||
|
if err := validateSchemaInstance(content); err != nil {
|
||||||
|
return Document{}, fmt.Errorf("decode evidence context: %w", err)
|
||||||
|
}
|
||||||
|
decoder := json.NewDecoder(bytes.NewReader(content))
|
||||||
|
decoder.DisallowUnknownFields()
|
||||||
|
var value Document
|
||||||
|
if err := decoder.Decode(&value); err != nil {
|
||||||
|
return Document{}, fmt.Errorf("decode evidence context: %w", err)
|
||||||
|
}
|
||||||
|
var trailing any
|
||||||
|
if err := decoder.Decode(&trailing); err != io.EOF {
|
||||||
|
return Document{}, fmt.Errorf("decode evidence context: multiple JSON values")
|
||||||
|
}
|
||||||
|
canonical, err := canonicalize(value)
|
||||||
|
if err != nil {
|
||||||
|
return Document{}, fmt.Errorf("decode evidence context: %w", err)
|
||||||
|
}
|
||||||
|
return clone(canonical)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Codec) schemaBytes() ([]byte, error) {
|
||||||
|
loadSchemaOnce.Do(loadAndCompileSchema)
|
||||||
|
if loadSchemaErr != nil {
|
||||||
|
return nil, loadSchemaErr
|
||||||
|
}
|
||||||
|
return append([]byte(nil), loadedSchema...), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadAndCompileSchema() {
|
||||||
|
raw, err := schemaAssets.ReadFile("assets/schemas/source_evidence_context.v1.json")
|
||||||
|
if err != nil {
|
||||||
|
loadSchemaErr = fmt.Errorf("read source evidence context schema: %w", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var identity struct {
|
||||||
|
ID string `json:"$id"`
|
||||||
|
Title string `json:"title"`
|
||||||
|
Type string `json:"type"`
|
||||||
|
Required []string `json:"required"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(raw, &identity); err != nil {
|
||||||
|
loadSchemaErr = fmt.Errorf("decode source evidence context schema: %w", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if identity.ID != SchemaID || identity.Title != SchemaName || identity.Type != "object" || !hasRequiredFields(identity.Required) {
|
||||||
|
loadSchemaErr = fmt.Errorf("source evidence context schema identity or required fields are invalid")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
schemaDocument, err := jsonschema.UnmarshalJSON(bytes.NewReader(raw))
|
||||||
|
if err != nil {
|
||||||
|
loadSchemaErr = fmt.Errorf("parse source evidence context schema: %w", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
compiler := jsonschema.NewCompiler()
|
||||||
|
if err := compiler.AddResource("source-evidence-context-schema.json", schemaDocument); err != nil {
|
||||||
|
loadSchemaErr = fmt.Errorf("load source evidence context schema: %w", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
compiled, err := compiler.Compile("source-evidence-context-schema.json")
|
||||||
|
if err != nil {
|
||||||
|
loadSchemaErr = fmt.Errorf("compile source evidence context schema: %w", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
loadedSchema = append([]byte(nil), raw...)
|
||||||
|
compiledSchema = compiled
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateSchemaInstance(content []byte) error {
|
||||||
|
instance, err := jsonschema.UnmarshalJSON(bytes.NewReader(content))
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("payload is not valid JSON: %w", err)
|
||||||
|
}
|
||||||
|
if err := compiledSchema.Validate(instance); err != nil {
|
||||||
|
return fmt.Errorf("payload does not conform to source evidence context schema: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func hasRequiredFields(required []string) bool {
|
||||||
|
want := map[string]bool{"source_id": true, "source_digest": true, "window_units": true, "selected_lanes": true, "contexts": true}
|
||||||
|
for _, field := range required {
|
||||||
|
delete(want, field)
|
||||||
|
}
|
||||||
|
return len(want) == 0
|
||||||
|
}
|
||||||
|
|
||||||
|
func canonicalize(value Document) (Document, error) {
|
||||||
|
owned, err := clone(value)
|
||||||
|
if err != nil {
|
||||||
|
return Document{}, err
|
||||||
|
}
|
||||||
|
value = owned
|
||||||
|
if err := requireIdentity("source_id", value.SourceID); err != nil {
|
||||||
|
return Document{}, err
|
||||||
|
}
|
||||||
|
if !digestPattern.MatchString(value.SourceDigest) {
|
||||||
|
return Document{}, fmt.Errorf("source_digest must be a sha256 digest")
|
||||||
|
}
|
||||||
|
if value.WindowUnits < 0 {
|
||||||
|
return Document{}, fmt.Errorf("window_units must not be negative")
|
||||||
|
}
|
||||||
|
if err := validateSelectedLanes(value.SelectedLanes); err != nil {
|
||||||
|
return Document{}, err
|
||||||
|
}
|
||||||
|
if value.Contexts == nil {
|
||||||
|
value.Contexts = make([]Context, 0)
|
||||||
|
}
|
||||||
|
selected := make(map[string]struct{}, len(value.SelectedLanes))
|
||||||
|
for _, laneID := range value.SelectedLanes {
|
||||||
|
selected[laneID] = struct{}{}
|
||||||
|
}
|
||||||
|
seenUnits := make(map[int]struct{})
|
||||||
|
for contextIndex := range value.Contexts {
|
||||||
|
context, err := canonicalizeContext(value.SourceID, selected, seenUnits, value.Contexts[contextIndex], contextIndex)
|
||||||
|
if err != nil {
|
||||||
|
return Document{}, err
|
||||||
|
}
|
||||||
|
value.Contexts[contextIndex] = context
|
||||||
|
}
|
||||||
|
return value, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateSelectedLanes(lanes []string) error {
|
||||||
|
if len(lanes) == 0 {
|
||||||
|
return fmt.Errorf("selected_lanes must not be empty")
|
||||||
|
}
|
||||||
|
for index, laneID := range lanes {
|
||||||
|
if err := requireIdentity(fmt.Sprintf("selected_lanes[%d]", index), laneID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if index > 0 && lanes[index-1] >= laneID {
|
||||||
|
return fmt.Errorf("selected_lanes must be unique and in lexical order")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func canonicalizeContext(sourceID string, selected map[string]struct{}, seenUnits map[int]struct{}, value Context, contextIndex int) (Context, error) {
|
||||||
|
prefix := fmt.Sprintf("contexts[%d]", contextIndex)
|
||||||
|
if len(value.EvidenceRefs) == 0 {
|
||||||
|
return Context{}, fmt.Errorf("%s.evidence_refs must not be empty", prefix)
|
||||||
|
}
|
||||||
|
if len(value.Units) == 0 {
|
||||||
|
return Context{}, fmt.Errorf("%s.units must not be empty", prefix)
|
||||||
|
}
|
||||||
|
if err := validateRefIdentity(sourceID, value.ContextRef, prefix+".context_ref"); err != nil {
|
||||||
|
return Context{}, err
|
||||||
|
}
|
||||||
|
positions := make(map[int]int, len(value.Units))
|
||||||
|
for unitIndex := range value.Units {
|
||||||
|
unit, err := cloneSourceUnit(value.Units[unitIndex])
|
||||||
|
if err != nil {
|
||||||
|
return Context{}, fmt.Errorf("%s.units[%d]: %w", prefix, unitIndex, err)
|
||||||
|
}
|
||||||
|
if unit.ID <= 0 || strings.TrimSpace(unit.Kind) == "" || strings.TrimSpace(unit.Text) == "" {
|
||||||
|
return Context{}, fmt.Errorf("%s.units[%d] has invalid required fields", prefix, unitIndex)
|
||||||
|
}
|
||||||
|
if err := validateRefIdentity(sourceID, unit.Ref, fmt.Sprintf("%s.units[%d].ref", prefix, unitIndex)); err != nil {
|
||||||
|
return Context{}, err
|
||||||
|
}
|
||||||
|
if unit.Ref.StartUnitID != unit.ID || unit.Ref.EndUnitID != unit.ID {
|
||||||
|
return Context{}, fmt.Errorf("%s.units[%d].ref must identify unit id %d", prefix, unitIndex, unit.ID)
|
||||||
|
}
|
||||||
|
if _, exists := positions[unit.ID]; exists {
|
||||||
|
return Context{}, fmt.Errorf("%s.units contains duplicate unit id %d", prefix, unit.ID)
|
||||||
|
}
|
||||||
|
if _, exists := seenUnits[unit.ID]; exists {
|
||||||
|
return Context{}, fmt.Errorf("contexts contain duplicate unit id %d", unit.ID)
|
||||||
|
}
|
||||||
|
positions[unit.ID] = unitIndex
|
||||||
|
seenUnits[unit.ID] = struct{}{}
|
||||||
|
value.Units[unitIndex] = unit
|
||||||
|
}
|
||||||
|
if value.ContextRef.StartUnitID != value.Units[0].ID || value.ContextRef.EndUnitID != value.Units[len(value.Units)-1].ID {
|
||||||
|
return Context{}, fmt.Errorf("%s.context_ref must identify the first and last units", prefix)
|
||||||
|
}
|
||||||
|
for evidenceIndex := range value.EvidenceRefs {
|
||||||
|
evidence := value.EvidenceRefs[evidenceIndex]
|
||||||
|
if _, ok := selected[evidence.LaneID]; !ok {
|
||||||
|
return Context{}, fmt.Errorf("%s.evidence_refs[%d].lane_id is not selected", prefix, evidenceIndex)
|
||||||
|
}
|
||||||
|
if err := requireIdentity(fmt.Sprintf("%s.evidence_refs[%d].lane_id", prefix, evidenceIndex), evidence.LaneID); err != nil {
|
||||||
|
return Context{}, err
|
||||||
|
}
|
||||||
|
if err := validateRefIdentity(sourceID, evidence.SourceRef, fmt.Sprintf("%s.evidence_refs[%d].source_ref", prefix, evidenceIndex)); err != nil {
|
||||||
|
return Context{}, err
|
||||||
|
}
|
||||||
|
start, startOK := positions[evidence.SourceRef.StartUnitID]
|
||||||
|
end, endOK := positions[evidence.SourceRef.EndUnitID]
|
||||||
|
if !startOK || !endOK || start > end {
|
||||||
|
return Context{}, fmt.Errorf("%s.evidence_refs[%d].source_ref is outside context units", prefix, evidenceIndex)
|
||||||
|
}
|
||||||
|
if evidenceIndex > 0 && !lessEvidenceRef(value.EvidenceRefs[evidenceIndex-1], evidence) {
|
||||||
|
return Context{}, fmt.Errorf("%s.evidence_refs must be unique and in deterministic order", prefix)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return value, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateRefIdentity(sourceID string, ref source.SourceRef, field string) error {
|
||||||
|
if ref.SourceID != sourceID {
|
||||||
|
return fmt.Errorf("%s.source_id does not match source_id", field)
|
||||||
|
}
|
||||||
|
if ref.StartUnitID <= 0 || ref.EndUnitID <= 0 {
|
||||||
|
return fmt.Errorf("%s endpoints must be positive", field)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireIdentity(field, value string) error {
|
||||||
|
if strings.TrimSpace(value) == "" || strings.TrimSpace(value) != value {
|
||||||
|
return fmt.Errorf("%s must be a non-empty trimmed string", field)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func clone(value Document) (Document, error) {
|
||||||
|
value.SelectedLanes = append([]string(nil), value.SelectedLanes...)
|
||||||
|
if value.Contexts == nil {
|
||||||
|
value.Contexts = make([]Context, 0)
|
||||||
|
} else {
|
||||||
|
contexts := make([]Context, len(value.Contexts))
|
||||||
|
for contextIndex, context := range value.Contexts {
|
||||||
|
contexts[contextIndex].ContextRef = context.ContextRef
|
||||||
|
contexts[contextIndex].EvidenceRefs = append([]EvidenceRef(nil), context.EvidenceRefs...)
|
||||||
|
contexts[contextIndex].Units = make([]source.SourceUnit, len(context.Units))
|
||||||
|
for unitIndex, unit := range context.Units {
|
||||||
|
cloned, err := cloneSourceUnit(unit)
|
||||||
|
if err != nil {
|
||||||
|
return Document{}, fmt.Errorf("clone contexts[%d].units[%d]: %w", contextIndex, unitIndex, err)
|
||||||
|
}
|
||||||
|
contexts[contextIndex].Units[unitIndex] = cloned
|
||||||
|
}
|
||||||
|
}
|
||||||
|
value.Contexts = contexts
|
||||||
|
}
|
||||||
|
return value, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneSourceUnit(unit source.SourceUnit) (source.SourceUnit, error) {
|
||||||
|
metadata, err := source.CloneMetadata(unit.Metadata)
|
||||||
|
if err != nil {
|
||||||
|
return source.SourceUnit{}, fmt.Errorf("clone metadata: %w", err)
|
||||||
|
}
|
||||||
|
unit.Metadata = metadata
|
||||||
|
return unit, nil
|
||||||
|
}
|
||||||
350
internal/framework/evidencecontext/evidencecontext_test.go
Normal file
350
internal/framework/evidencecontext/evidencecontext_test.go
Normal file
@@ -0,0 +1,350 @@
|
|||||||
|
package evidencecontext
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"math"
|
||||||
|
"os"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestBuildExpandsAndMergesEvidenceByDocumentPosition(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
window int
|
||||||
|
evidence []LaneEvidence
|
||||||
|
wantUnits [][]int
|
||||||
|
wantRefs [][]EvidenceRef
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "zero window",
|
||||||
|
evidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3)}}},
|
||||||
|
wantUnits: [][]int{{3}},
|
||||||
|
wantRefs: [][]EvidenceRef{{{LaneID: "npcs", SourceRef: ref(3, 3)}}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "non monotonic ids use positions and clip boundaries",
|
||||||
|
window: 1,
|
||||||
|
evidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3)}}},
|
||||||
|
wantUnits: [][]int{{10, 3, 30}},
|
||||||
|
wantRefs: [][]EvidenceRef{{{LaneID: "npcs", SourceRef: ref(3, 3)}}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "separate gaps stay separate",
|
||||||
|
evidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(10, 10), ref(50, 50)}}},
|
||||||
|
wantUnits: [][]int{{10}, {50}},
|
||||||
|
wantRefs: [][]EvidenceRef{{{LaneID: "npcs", SourceRef: ref(10, 10)}}, {{LaneID: "npcs", SourceRef: ref(50, 50)}}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "overlapping windows merge",
|
||||||
|
window: 1,
|
||||||
|
evidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3), ref(30, 30)}}},
|
||||||
|
wantUnits: [][]int{{10, 3, 30, 7}},
|
||||||
|
wantRefs: [][]EvidenceRef{{{LaneID: "npcs", SourceRef: ref(3, 3)}, {LaneID: "npcs", SourceRef: ref(30, 30)}}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "contiguous windows merge",
|
||||||
|
window: 1,
|
||||||
|
evidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(10, 10), ref(7, 7)}}},
|
||||||
|
wantUnits: [][]int{{10, 3, 30, 7, 50}},
|
||||||
|
wantRefs: [][]EvidenceRef{{{LaneID: "npcs", SourceRef: ref(7, 7)}, {LaneID: "npcs", SourceRef: ref(10, 10)}}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "duplicate contributions retain unique lane attribution",
|
||||||
|
evidence: []LaneEvidence{
|
||||||
|
{LaneID: "spells", SourceRefs: []source.SourceRef{ref(30, 30), ref(30, 30)}},
|
||||||
|
{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(30, 30)}},
|
||||||
|
},
|
||||||
|
wantUnits: [][]int{{30}},
|
||||||
|
wantRefs: [][]EvidenceRef{{{LaneID: "npcs", SourceRef: ref(30, 30)}, {LaneID: "spells", SourceRef: ref(30, 30)}}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "empty contributions retain explicit empty contexts",
|
||||||
|
evidence: []LaneEvidence{{LaneID: "npcs"}},
|
||||||
|
wantUnits: [][]int{},
|
||||||
|
wantRefs: [][]EvidenceRef{},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "largest window clips without overflow",
|
||||||
|
window: math.MaxInt,
|
||||||
|
evidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(30, 30)}}},
|
||||||
|
wantUnits: [][]int{{10, 3, 30, 7, 50}},
|
||||||
|
wantRefs: [][]EvidenceRef{{{LaneID: "npcs", SourceRef: ref(30, 30)}}},
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
document := testDocument(t)
|
||||||
|
got, err := Build(BuildRequest{Source: document, WindowUnits: test.window, SelectedLanes: []string{"spells", "npcs"}, LaneEvidence: test.evidence})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Build() error = %v", err)
|
||||||
|
}
|
||||||
|
if want := []string{"npcs", "spells"}; !reflect.DeepEqual(got.SelectedLanes, want) {
|
||||||
|
t.Fatalf("SelectedLanes = %#v, want %#v", got.SelectedLanes, want)
|
||||||
|
}
|
||||||
|
if got.WindowUnits != test.window || got.SourceID != document.ID || got.SourceDigest != document.Digest {
|
||||||
|
t.Fatalf("Build() identity = %#v, want source and window identity", got)
|
||||||
|
}
|
||||||
|
if actual := contextUnitIDs(got.Contexts); !reflect.DeepEqual(actual, test.wantUnits) {
|
||||||
|
t.Fatalf("context unit ids = %#v, want %#v", actual, test.wantUnits)
|
||||||
|
}
|
||||||
|
if actual := contextEvidenceRefs(got.Contexts); !reflect.DeepEqual(actual, test.wantRefs) {
|
||||||
|
t.Fatalf("context evidence refs = %#v, want %#v", actual, test.wantRefs)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildIsStableAndOwnsSourceAndInputs(t *testing.T) {
|
||||||
|
document := testDocument(t)
|
||||||
|
refs := []source.SourceRef{ref(30, 30), ref(3, 3)}
|
||||||
|
request := BuildRequest{
|
||||||
|
Source: document,
|
||||||
|
WindowUnits: 1,
|
||||||
|
SelectedLanes: []string{"spells", "npcs"},
|
||||||
|
LaneEvidence: []LaneEvidence{{LaneID: "spells", SourceRefs: refs}, {LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3)}}},
|
||||||
|
}
|
||||||
|
first, err := Build(request)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
secondRequest := request
|
||||||
|
secondRequest.LaneEvidence = []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3)}}, {LaneID: "spells", SourceRefs: []source.SourceRef{ref(3, 3), ref(30, 30)}}}
|
||||||
|
second, err := Build(secondRequest)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(first, second) {
|
||||||
|
t.Fatalf("Build() order differs:\nfirst: %#v\nsecond: %#v", first, second)
|
||||||
|
}
|
||||||
|
first.SelectedLanes[0] = "changed"
|
||||||
|
first.Contexts[0].Units[0].Metadata["nested"].(map[string]any)["value"] = "changed"
|
||||||
|
if document.Units[0].Metadata["nested"].(map[string]any)["value"] != "original" {
|
||||||
|
t.Fatal("Build() returned metadata aliases to source document")
|
||||||
|
}
|
||||||
|
document.Units[0].Metadata["nested"].(map[string]any)["value"] = "later"
|
||||||
|
if second.Contexts[0].Units[0].Metadata["nested"].(map[string]any)["value"] != "original" {
|
||||||
|
t.Fatal("Build() retained metadata aliases to source document")
|
||||||
|
}
|
||||||
|
refs[0].StartUnitID = 999
|
||||||
|
if !containsEvidenceRef(second.Contexts[0].EvidenceRefs, ref(30, 30)) {
|
||||||
|
t.Fatal("Build() retained source-reference input aliases")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildRejectsInvalidInputs(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
mutate func(*BuildRequest)
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "negative window", mutate: func(request *BuildRequest) { request.WindowUnits = -1 }, want: "window_units"},
|
||||||
|
{name: "blank selected lane", mutate: func(request *BuildRequest) { request.SelectedLanes = []string{" "} }, want: "selected_lanes"},
|
||||||
|
{name: "duplicate selected lane", mutate: func(request *BuildRequest) { request.SelectedLanes = []string{"npcs", " npcs "} }, want: "duplicated"},
|
||||||
|
{name: "unselected contribution", mutate: func(request *BuildRequest) {
|
||||||
|
request.LaneEvidence = []LaneEvidence{{LaneID: "other", SourceRefs: []source.SourceRef{ref(3, 3)}}}
|
||||||
|
}, want: "not selected"},
|
||||||
|
{name: "source digest mismatch", mutate: func(request *BuildRequest) { request.Source.Digest = "sha256:" + strings.Repeat("0", 64) }, want: "does not match"},
|
||||||
|
{name: "invalid reference", mutate: func(request *BuildRequest) {
|
||||||
|
request.LaneEvidence = []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(99, 99)}}}
|
||||||
|
}, want: "source reference[0]"},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
request := BuildRequest{Source: testDocument(t), SelectedLanes: []string{"npcs"}, LaneEvidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3)}}}}
|
||||||
|
test.mutate(&request)
|
||||||
|
if _, err := Build(request); err == nil || !strings.Contains(err.Error(), test.want) {
|
||||||
|
t.Fatalf("Build() error = %v, want %q", err, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCodecRoundTripsCompactFixtureAndOwnsDecodedValues(t *testing.T) {
|
||||||
|
fixture, err := os.ReadFile("testdata/source_evidence_context.v1.json")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
codec := New()
|
||||||
|
value, err := codec.Decode(fixture)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Decode(fixture) error = %v", err)
|
||||||
|
}
|
||||||
|
encoded, err := codec.Encode(value)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !bytes.Equal(encoded, bytes.TrimSpace(fixture)) {
|
||||||
|
t.Fatalf("fixture does not use canonical encoding\nwant: %s\n got: %s", fixture, encoded)
|
||||||
|
}
|
||||||
|
value.Contexts[0].Units[0].Text = "changed"
|
||||||
|
decoded, err := codec.Decode(fixture)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if decoded.Contexts[0].Units[0].Text != "The party meets Rowan." {
|
||||||
|
t.Fatal("Decode() reused mutable document storage")
|
||||||
|
}
|
||||||
|
built, err := Build(BuildRequest{Source: testDocument(t), WindowUnits: 1, SelectedLanes: []string{"npcs"}, LaneEvidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3)}}}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
content, err := codec.Encode(built)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
first, err := codec.Decode(content)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
second, err := codec.Decode(content)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
first.Contexts[0].Units[0].Metadata["nested"].(map[string]any)["value"] = "changed"
|
||||||
|
if second.Contexts[0].Units[0].Metadata["nested"].(map[string]any)["value"] != "original" {
|
||||||
|
t.Fatal("Decode() returned metadata aliases")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCodecRejectsInvalidDurableBoundaries(t *testing.T) {
|
||||||
|
value, err := Build(BuildRequest{Source: testDocument(t), SelectedLanes: []string{"npcs"}, LaneEvidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3)}}}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
mutate func(*Document)
|
||||||
|
}{
|
||||||
|
{name: "unsorted lanes", mutate: func(value *Document) { value.SelectedLanes = []string{"z", "a"} }},
|
||||||
|
{name: "context range mismatch", mutate: func(value *Document) { value.Contexts[0].ContextRef.EndUnitID = 999 }},
|
||||||
|
{name: "mismatched evidence source", mutate: func(value *Document) { value.Contexts[0].EvidenceRefs[0].SourceRef.SourceID = "other" }},
|
||||||
|
{name: "invalid evidence range", mutate: func(value *Document) {
|
||||||
|
value.Contexts[0].EvidenceRefs[0].SourceRef.StartUnitID = 10
|
||||||
|
}},
|
||||||
|
{name: "duplicate context unit", mutate: func(value *Document) { value.Contexts = append(value.Contexts, value.Contexts[0]) }},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
candidate, err := clone(value)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
test.mutate(&candidate)
|
||||||
|
if _, err := New().Encode(candidate); err == nil {
|
||||||
|
t.Fatal("Encode() error = nil, want durable model rejection")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
content, err := New().Encode(value)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
mutate func(map[string]any)
|
||||||
|
}{
|
||||||
|
{name: "missing contexts", mutate: func(value map[string]any) { delete(value, "contexts") }},
|
||||||
|
{name: "null contexts", mutate: func(value map[string]any) { value["contexts"] = nil }},
|
||||||
|
{name: "unknown fixed field", mutate: func(value map[string]any) { value["unknown"] = true }},
|
||||||
|
{name: "missing units", mutate: func(value map[string]any) { delete(contextObject(value, 0), "units") }},
|
||||||
|
{name: "null evidence refs", mutate: func(value map[string]any) { contextObject(value, 0)["evidence_refs"] = nil }},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
raw := decodeJSON(t, content)
|
||||||
|
test.mutate(raw)
|
||||||
|
mutated, err := json.Marshal(raw)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := New().Decode(mutated); err == nil {
|
||||||
|
t.Fatal("Decode() error = nil, want strict payload rejection")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if _, err := New().Decode(append(content, []byte(" {}")...)); err == nil {
|
||||||
|
t.Fatal("Decode() error = nil, want trailing JSON rejection")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSerializeUsesFixedArtifactIdentity(t *testing.T) {
|
||||||
|
artifact, err := Serialize(BuildRequest{Source: testDocument(t), SelectedLanes: []string{"npcs"}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if artifact.Kind != ArtifactKind || artifact.MediaType != MediaType || artifact.Schema.ID != SchemaID || artifact.Schema.Name != SchemaName || artifact.Schema.Version != SchemaVersion {
|
||||||
|
t.Fatalf("Serialize() = %#v, want fixed artifact identity", artifact)
|
||||||
|
}
|
||||||
|
decoded, err := New().Decode(artifact.Content)
|
||||||
|
if err != nil || len(decoded.Contexts) != 0 || decoded.Contexts == nil {
|
||||||
|
t.Fatalf("Decode(Serialize()) = %#v, %v; want explicit empty contexts", decoded, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func testDocument(t *testing.T) *source.SourceDocument {
|
||||||
|
t.Helper()
|
||||||
|
document := &source.SourceDocument{
|
||||||
|
ID: "session", Kind: "transcript", Format: "application/json",
|
||||||
|
Units: []source.SourceUnit{
|
||||||
|
{ID: 10, Kind: "segment", Text: "first", Ref: ref(10, 10), Metadata: map[string]any{"nested": map[string]any{"value": "original"}}},
|
||||||
|
{ID: 3, Kind: "segment", Text: "second", Ref: ref(3, 3)},
|
||||||
|
{ID: 30, Kind: "segment", Text: "third", Ref: ref(30, 30)},
|
||||||
|
{ID: 7, Kind: "segment", Text: "fourth", Ref: ref(7, 7)},
|
||||||
|
{ID: 50, Kind: "segment", Text: "fifth", Ref: ref(50, 50)},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
digest, err := source.DigestDocument(document)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
document.Digest = digest
|
||||||
|
return document
|
||||||
|
}
|
||||||
|
|
||||||
|
func ref(start, end int) source.SourceRef {
|
||||||
|
return source.SourceRef{SourceID: "session", StartUnitID: start, EndUnitID: end}
|
||||||
|
}
|
||||||
|
|
||||||
|
func contextUnitIDs(contexts []Context) [][]int {
|
||||||
|
values := make([][]int, len(contexts))
|
||||||
|
for index, context := range contexts {
|
||||||
|
values[index] = make([]int, len(context.Units))
|
||||||
|
for unitIndex, unit := range context.Units {
|
||||||
|
values[index][unitIndex] = unit.ID
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return values
|
||||||
|
}
|
||||||
|
|
||||||
|
func contextEvidenceRefs(contexts []Context) [][]EvidenceRef {
|
||||||
|
values := make([][]EvidenceRef, len(contexts))
|
||||||
|
for index, context := range contexts {
|
||||||
|
values[index] = append([]EvidenceRef(nil), context.EvidenceRefs...)
|
||||||
|
}
|
||||||
|
return values
|
||||||
|
}
|
||||||
|
|
||||||
|
func containsEvidenceRef(values []EvidenceRef, want source.SourceRef) bool {
|
||||||
|
for _, value := range values {
|
||||||
|
if value.SourceRef == want {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeJSON(t *testing.T, content []byte) map[string]any {
|
||||||
|
t.Helper()
|
||||||
|
decoder := json.NewDecoder(bytes.NewReader(content))
|
||||||
|
decoder.UseNumber()
|
||||||
|
var value map[string]any
|
||||||
|
if err := decoder.Decode(&value); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return value
|
||||||
|
}
|
||||||
|
|
||||||
|
func contextObject(value map[string]any, index int) map[string]any {
|
||||||
|
return value["contexts"].([]any)[index].(map[string]any)
|
||||||
|
}
|
||||||
50
internal/framework/evidencecontext/model.go
Normal file
50
internal/framework/evidencecontext/model.go
Normal file
@@ -0,0 +1,50 @@
|
|||||||
|
// Package evidencecontext owns the durable source evidence-context contract.
|
||||||
|
package evidencecontext
|
||||||
|
|
||||||
|
import (
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
ArtifactKind contracts.ArtifactKind = "source/evidence-context"
|
||||||
|
SchemaID = "notarius.source.evidence_context"
|
||||||
|
SchemaName = "notarius_source_evidence_context_v1"
|
||||||
|
SchemaVersion = "v1"
|
||||||
|
MediaType = "application/json"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Document is the durable union of direct evidence and surrounding source
|
||||||
|
// context selected for one accepted source document.
|
||||||
|
type Document struct {
|
||||||
|
SourceID string `json:"source_id"`
|
||||||
|
SourceDigest string `json:"source_digest"`
|
||||||
|
WindowUnits int `json:"window_units"`
|
||||||
|
SelectedLanes []string `json:"selected_lanes"`
|
||||||
|
Contexts []Context `json:"contexts"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type Context struct {
|
||||||
|
ContextRef source.SourceRef `json:"context_ref"`
|
||||||
|
EvidenceRefs []EvidenceRef `json:"evidence_refs"`
|
||||||
|
Units []source.SourceUnit `json:"units"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type EvidenceRef struct {
|
||||||
|
LaneID string `json:"lane_id"`
|
||||||
|
SourceRef source.SourceRef `json:"source_ref"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// LaneEvidence attributes direct source references to one selected lane.
|
||||||
|
type LaneEvidence struct {
|
||||||
|
LaneID string `json:"lane_id"`
|
||||||
|
SourceRefs []source.SourceRef `json:"source_refs"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// BuildRequest supplies accepted source material and direct lane evidence.
|
||||||
|
type BuildRequest struct {
|
||||||
|
Source *source.SourceDocument
|
||||||
|
WindowUnits int
|
||||||
|
SelectedLanes []string
|
||||||
|
LaneEvidence []LaneEvidence
|
||||||
|
}
|
||||||
1
internal/framework/evidencecontext/testdata/source_evidence_context.v1.json
vendored
Normal file
1
internal/framework/evidencecontext/testdata/source_evidence_context.v1.json
vendored
Normal file
@@ -0,0 +1 @@
|
|||||||
|
{"source_id":"session-alpha","source_digest":"sha256:0000000000000000000000000000000000000000000000000000000000000000","window_units":0,"selected_lanes":["npcs"],"contexts":[{"context_ref":{"source_id":"session-alpha","start_unit_id":13,"end_unit_id":13},"evidence_refs":[{"lane_id":"npcs","source_ref":{"source_id":"session-alpha","start_unit_id":13,"end_unit_id":13}}],"units":[{"id":13,"kind":"transcript_segment","text":"The party meets Rowan.","ref":{"source_id":"session-alpha","start_unit_id":13,"end_unit_id":13}}]}]}
|
||||||
@@ -12,7 +12,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
type AssetSource struct {
|
type AssetSource struct {
|
||||||
@@ -23,6 +23,7 @@ type AssetSource struct {
|
|||||||
type AssetRegistry struct {
|
type AssetRegistry struct {
|
||||||
prompts []AssetSource
|
prompts []AssetSource
|
||||||
schemas []AssetSource
|
schemas []AssetSource
|
||||||
|
fallbackProfiles []AssetSource
|
||||||
}
|
}
|
||||||
|
|
||||||
type AssetHashPart struct {
|
type AssetHashPart struct {
|
||||||
@@ -58,6 +59,20 @@ func (r *AssetRegistry) RegisterSchemaFS(fsys fs.FS, root string) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RegisterFallbackProfileFS registers profile assets that PromptKit uses only
|
||||||
|
// when an operator-configured source does not provide a matching profile.
|
||||||
|
func (r *AssetRegistry) RegisterFallbackProfileFS(fsys fs.FS, root string) error {
|
||||||
|
if r == nil {
|
||||||
|
return fmt.Errorf("asset registry must not be nil")
|
||||||
|
}
|
||||||
|
source, err := newAssetSource(fsys, root)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("register fallback profile assets: %w", err)
|
||||||
|
}
|
||||||
|
r.fallbackProfiles = append(r.fallbackProfiles, source)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func (r *AssetRegistry) PromptFS() (fs.FS, error) {
|
func (r *AssetRegistry) PromptFS() (fs.FS, error) {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return nil, fmt.Errorf("asset registry must not be nil")
|
return nil, fmt.Errorf("asset registry must not be nil")
|
||||||
@@ -72,19 +87,76 @@ func (r *AssetRegistry) SchemaFS() (fs.FS, error) {
|
|||||||
return flattenAssetSources(r.schemas)
|
return flattenAssetSources(r.schemas)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *AssetRegistry) ScriptoriumOptions() ([]scriptorium.Option, error) {
|
func (r *AssetRegistry) FallbackProfileFS() (fs.FS, error) {
|
||||||
|
if r == nil {
|
||||||
|
return nil, fmt.Errorf("asset registry must not be nil")
|
||||||
|
}
|
||||||
|
return flattenAssetSources(r.fallbackProfiles)
|
||||||
|
}
|
||||||
|
|
||||||
|
// FallbackProfileDigest returns a deterministic, non-secret identity for the
|
||||||
|
// flattened fallback profile assets.
|
||||||
|
func (r *AssetRegistry) FallbackProfileDigest() (string, error) {
|
||||||
|
_, digest, _, err := r.fallbackProfileAssets()
|
||||||
|
return digest, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *AssetRegistry) PromptKitOptions() ([]promptkit.Option, error) {
|
||||||
|
options, _, err := r.promptKitOptions()
|
||||||
|
return options, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *AssetRegistry) promptKitOptions() ([]promptkit.Option, string, error) {
|
||||||
promptFS, err := r.PromptFS()
|
promptFS, err := r.PromptFS()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("prepare prompt assets: %w", err)
|
return nil, "", fmt.Errorf("prepare prompt assets: %w", err)
|
||||||
}
|
}
|
||||||
schemaFS, err := r.SchemaFS()
|
schemaFS, err := r.SchemaFS()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("prepare schema assets: %w", err)
|
return nil, "", fmt.Errorf("prepare schema assets: %w", err)
|
||||||
}
|
}
|
||||||
return []scriptorium.Option{
|
options := []promptkit.Option{
|
||||||
scriptorium.WithPromptFS(promptFS, "."),
|
promptkit.WithPromptFS(promptFS, "."),
|
||||||
scriptorium.WithSchemaFS(schemaFS, "."),
|
promptkit.WithSchemaFS(schemaFS, "."),
|
||||||
}, nil
|
}
|
||||||
|
fallbackFS, fallbackDigest, hasFallback, err := r.fallbackProfileAssets()
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", err
|
||||||
|
}
|
||||||
|
if hasFallback {
|
||||||
|
options = append(options, promptkit.WithFallbackProfileFS(fallbackFS, "."))
|
||||||
|
}
|
||||||
|
return options, fallbackDigest, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *AssetRegistry) promptKitFallbackProfileOption() (promptkit.Option, bool, error) {
|
||||||
|
fallbackFS, _, hasFallback, err := r.fallbackProfileAssets()
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
if !hasFallback {
|
||||||
|
return nil, false, nil
|
||||||
|
}
|
||||||
|
return promptkit.WithFallbackProfileFS(fallbackFS, "."), true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *AssetRegistry) fallbackProfileAssets() (fs.FS, string, bool, error) {
|
||||||
|
if r == nil {
|
||||||
|
return nil, "", false, fmt.Errorf("asset registry must not be nil")
|
||||||
|
}
|
||||||
|
if len(r.fallbackProfiles) == 0 {
|
||||||
|
empty := sha256.Sum256([]byte("notarius:fallback-profile-assets:empty"))
|
||||||
|
return nil, "sha256:" + hex.EncodeToString(empty[:]), false, nil
|
||||||
|
}
|
||||||
|
fallbackFS, err := r.FallbackProfileFS()
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", false, fmt.Errorf("prepare fallback profile assets: %w", err)
|
||||||
|
}
|
||||||
|
digest, err := hashAssetFilesystem(fallbackFS)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", false, err
|
||||||
|
}
|
||||||
|
return fallbackFS, digest, true, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func HashAssets(parts []AssetHashPart) (string, error) {
|
func HashAssets(parts []AssetHashPart) (string, error) {
|
||||||
@@ -117,6 +189,28 @@ func HashAssets(parts []AssetHashPart) (string, error) {
|
|||||||
return "sha256:" + hex.EncodeToString(hash.Sum(nil)), nil
|
return "sha256:" + hex.EncodeToString(hash.Sum(nil)), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func hashAssetFilesystem(fsys fs.FS) (string, error) {
|
||||||
|
var parts []AssetHashPart
|
||||||
|
err := fs.WalkDir(fsys, ".", func(name string, entry fs.DirEntry, walkErr error) error {
|
||||||
|
if walkErr != nil {
|
||||||
|
return walkErr
|
||||||
|
}
|
||||||
|
if entry.IsDir() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
parts = append(parts, AssetHashPart{FS: fsys, Path: name})
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("walk assets for digest: %w", err)
|
||||||
|
}
|
||||||
|
if len(parts) > 0 {
|
||||||
|
return HashAssets(parts)
|
||||||
|
}
|
||||||
|
empty := sha256.Sum256([]byte("notarius:fallback-profile-assets:empty"))
|
||||||
|
return "sha256:" + hex.EncodeToString(empty[:]), nil
|
||||||
|
}
|
||||||
|
|
||||||
func newAssetSource(fsys fs.FS, root string) (AssetSource, error) {
|
func newAssetSource(fsys fs.FS, root string) (AssetSource, error) {
|
||||||
if fsys == nil {
|
if fsys == nil {
|
||||||
return AssetSource{}, fmt.Errorf("filesystem must not be nil")
|
return AssetSource{}, fmt.Errorf("filesystem must not be nil")
|
||||||
@@ -248,6 +342,9 @@ func (m assetMapFS) dirEntries(dir string) []fs.DirEntry {
|
|||||||
children[childName] = entry
|
children[childName] = entry
|
||||||
}
|
}
|
||||||
if len(children) == 0 {
|
if len(children) == 0 {
|
||||||
|
if dir == "." {
|
||||||
|
return []fs.DirEntry{}
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
names := make([]string, 0, len(children))
|
names := make([]string, 0, len(children))
|
||||||
|
|||||||
@@ -2,12 +2,13 @@ package llm
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"io/fs"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"testing/fstest"
|
"testing/fstest"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestAssetRegistryCombinesPromptAndSchemaSources(t *testing.T) {
|
func TestAssetRegistryCombinesPromptAndSchemaSources(t *testing.T) {
|
||||||
@@ -22,12 +23,12 @@ func TestAssetRegistryCombinesPromptAndSchemaSources(t *testing.T) {
|
|||||||
"root/schemas/out.json": {Data: []byte(`{"type":"object"}`)},
|
"root/schemas/out.json": {Data: []byte(`{"type":"object"}`)},
|
||||||
}, "root")
|
}, "root")
|
||||||
|
|
||||||
engine := newAssetTestEngine(t, registry)
|
engine := newPromptKitAssetTestEngine(t, registry)
|
||||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: "asset.test",
|
PromptID: "asset.test",
|
||||||
ProfileID: "asset-test-profile",
|
ProfileID: "asset-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.Inline(`{"ok":true}`),
|
"transcript": promptkit.Inline(`{"ok":true}`),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -50,12 +51,12 @@ func TestAssetRegistryPrepareFailsForMissingPromptAsset(t *testing.T) {
|
|||||||
"out.json": {Data: []byte(`{"type":"object"}`)},
|
"out.json": {Data: []byte(`{"type":"object"}`)},
|
||||||
}, ".")
|
}, ".")
|
||||||
|
|
||||||
engine := newAssetTestEngine(t, registry)
|
engine := newPromptKitAssetTestEngine(t, registry)
|
||||||
_, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
_, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: "asset.test",
|
PromptID: "asset.test",
|
||||||
ProfileID: "asset-test-profile",
|
ProfileID: "asset-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.Inline(`{"ok":true}`),
|
"transcript": promptkit.Inline(`{"ok":true}`),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err == nil || !strings.Contains(err.Error(), "content_file") {
|
if err == nil || !strings.Contains(err.Error(), "content_file") {
|
||||||
@@ -74,12 +75,12 @@ func TestAssetRegistryPrepareFailsForMissingSchemaAsset(t *testing.T) {
|
|||||||
"present.json": {Data: []byte(`{"type":"object"}`)},
|
"present.json": {Data: []byte(`{"type":"object"}`)},
|
||||||
}, ".")
|
}, ".")
|
||||||
|
|
||||||
engine := newAssetTestEngine(t, registry)
|
engine := newPromptKitAssetTestEngine(t, registry)
|
||||||
_, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
_, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||||
PromptID: "asset.test",
|
PromptID: "asset.test",
|
||||||
ProfileID: "asset-test-profile",
|
ProfileID: "asset-test-profile",
|
||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
"transcript": scriptorium.Inline(`{"ok":true}`),
|
"transcript": promptkit.Inline(`{"ok":true}`),
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err == nil || !strings.Contains(err.Error(), "missing.json") {
|
if err == nil || !strings.Contains(err.Error(), "missing.json") {
|
||||||
@@ -98,6 +99,89 @@ func TestAssetRegistryRejectsDuplicateAssetPaths(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestAssetRegistryCombinesFallbackProfileSources(t *testing.T) {
|
||||||
|
registry := NewAssetRegistry()
|
||||||
|
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
|
||||||
|
"first/profiles/one.yaml": {Data: []byte("id: one\nmodel: first\n")},
|
||||||
|
}, "first/profiles"); err != nil {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
|
||||||
|
"second/two.yaml": {Data: []byte("id: two\nmodel: second\n")},
|
||||||
|
}, "second"); err != nil {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
fallbackFS, err := registry.FallbackProfileFS()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("FallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
for _, name := range []string{"one.yaml", "two.yaml"} {
|
||||||
|
if _, err := fs.ReadFile(fallbackFS, name); err != nil {
|
||||||
|
t.Fatalf("FallbackProfileFS().ReadFile(%q) error = %v, want nil", name, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAssetRegistryRejectsInvalidFallbackProfileRoot(t *testing.T) {
|
||||||
|
registry := NewAssetRegistry()
|
||||||
|
err := registry.RegisterFallbackProfileFS(fstest.MapFS{}, "../profiles")
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "invalid path") {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want invalid root error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAssetRegistryRejectsUnreadableFallbackProfileAssets(t *testing.T) {
|
||||||
|
registry := NewAssetRegistry()
|
||||||
|
if err := registry.RegisterFallbackProfileFS(unreadableAssetFS{}, "."); err != nil {
|
||||||
|
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
_, err := registry.FallbackProfileFS()
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "permission denied") {
|
||||||
|
t.Fatalf("FallbackProfileFS() error = %v, want unreadable asset error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAssetRegistryRejectsDuplicateFallbackProfilePaths(t *testing.T) {
|
||||||
|
registry := NewAssetRegistry()
|
||||||
|
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{"first/profile.yaml": {Data: []byte("id: first\n")}}, "first"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{"second/profile.yaml": {Data: []byte("id: second\n")}}, "second"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
_, err := registry.FallbackProfileFS()
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "duplicate asset path") {
|
||||||
|
t.Fatalf("FallbackProfileFS() error = %v, want duplicate path error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAssetRegistryFallbackProfileDigestTracksContentWithoutLeakingIt(t *testing.T) {
|
||||||
|
digestFor := func(content string) string {
|
||||||
|
t.Helper()
|
||||||
|
registry := NewAssetRegistry()
|
||||||
|
if err := registry.RegisterFallbackProfileFS(fstest.MapFS{
|
||||||
|
"profiles/profile.yaml": {Data: []byte(content)},
|
||||||
|
}, "profiles"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
digest, err := registry.FallbackProfileDigest()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return digest
|
||||||
|
}
|
||||||
|
|
||||||
|
first := digestFor("id: fallback\nmodel: model-one\n")
|
||||||
|
second := digestFor("id: fallback\nmodel: model-two\n")
|
||||||
|
if first == second {
|
||||||
|
t.Fatalf("fallback profile digests = %q and %q, want content change", first, second)
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(first, "sha256:") || strings.Contains(first, "model-one") || strings.Contains(first, "profile.yaml") {
|
||||||
|
t.Fatalf("fallback profile digest leaked source details: %q", first)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestAssetRegistryCombinesNamespacedPromptSources(t *testing.T) {
|
func TestAssetRegistryCombinesNamespacedPromptSources(t *testing.T) {
|
||||||
registry := NewAssetRegistry()
|
registry := NewAssetRegistry()
|
||||||
mustRegisterPromptFS(t, registry, fstest.MapFS{
|
mustRegisterPromptFS(t, registry, fstest.MapFS{
|
||||||
@@ -145,18 +229,18 @@ func TestHashAssetsOmitsRawAssetContent(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func newAssetTestEngine(t *testing.T, registry *AssetRegistry) *scriptorium.Engine {
|
func newPromptKitAssetTestEngine(t *testing.T, registry *AssetRegistry) *promptkit.Engine {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
options, err := registry.ScriptoriumOptions()
|
options, err := registry.PromptKitOptions()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ScriptoriumOptions() error = %v, want nil", err)
|
t.Fatalf("PromptKitOptions() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||||
ID: "asset-test-profile",
|
ID: "asset-test-profile",
|
||||||
Endpoint: "http://127.0.0.1:1/v1",
|
Endpoint: "http://127.0.0.1:1/v1",
|
||||||
Model: "asset-test-model",
|
Model: "asset-test-model",
|
||||||
})))
|
})))
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("NewEngine() error = %v, want nil", err)
|
t.Fatalf("NewEngine() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
@@ -196,3 +280,9 @@ output:
|
|||||||
repair_attempts: 0
|
repair_attempts: 0
|
||||||
`
|
`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type unreadableAssetFS struct{}
|
||||||
|
|
||||||
|
func (unreadableAssetFS) Open(name string) (fs.File, error) {
|
||||||
|
return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrPermission}
|
||||||
|
}
|
||||||
|
|||||||
14
internal/framework/llm/checkpoint_fingerprint.go
Normal file
14
internal/framework/llm/checkpoint_fingerprint.go
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
package llm
|
||||||
|
|
||||||
|
// CheckpointFingerprint is a stable, non-secret semantic identity contributed
|
||||||
|
// by the LLM runtime before pipeline execution.
|
||||||
|
type CheckpointFingerprint struct {
|
||||||
|
Name string
|
||||||
|
Value string
|
||||||
|
}
|
||||||
|
|
||||||
|
// CheckpointFingerprintProvider exposes LLM-runtime identities that must
|
||||||
|
// participate in checkpoint composition.
|
||||||
|
type CheckpointFingerprintProvider interface {
|
||||||
|
LLMCheckpointFingerprints() ([]CheckpointFingerprint, error)
|
||||||
|
}
|
||||||
@@ -3,6 +3,7 @@ package llm
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"net/http"
|
"net/http"
|
||||||
"regexp"
|
"regexp"
|
||||||
@@ -13,24 +14,36 @@ import (
|
|||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
const scriptoriumProviderName = "scriptorium"
|
const promptKitProviderName = "promptkit"
|
||||||
|
|
||||||
type ScriptoriumClientConfig struct {
|
type PromptKitLocalBackendConfig struct {
|
||||||
|
Endpoint string
|
||||||
|
ConcurrencyLimit int
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitClientConfig struct {
|
||||||
ProfileDir string
|
ProfileDir string
|
||||||
ProfileFile string
|
ProfileFile string
|
||||||
|
LocalBackend *PromptKitLocalBackendConfig
|
||||||
Assets *AssetRegistry
|
Assets *AssetRegistry
|
||||||
Timeout time.Duration
|
Timeout time.Duration
|
||||||
HTTPClient *http.Client
|
HTTPClient *http.Client
|
||||||
EngineOptions []scriptorium.Option
|
EngineOptions []promptkit.Option
|
||||||
Recorder *LLMProfileRecorder
|
Recorder *LLMProfileRecorder
|
||||||
|
ReasoningEffort *string
|
||||||
}
|
}
|
||||||
|
|
||||||
type ScriptoriumClient struct {
|
type PromptKitClient struct {
|
||||||
engine *scriptorium.Engine
|
engine *promptkit.Engine
|
||||||
recorder *LLMProfileRecorder
|
recorder *LLMProfileRecorder
|
||||||
|
profileDir string
|
||||||
|
profileFile string
|
||||||
|
localEndpoint string
|
||||||
|
fallbackProfileDigest string
|
||||||
|
reasoningEffort *string
|
||||||
}
|
}
|
||||||
|
|
||||||
type LLMProfileRecorder struct {
|
type LLMProfileRecorder struct {
|
||||||
@@ -38,49 +51,66 @@ type LLMProfileRecorder struct {
|
|||||||
profiles map[string]artifacts.LLMProfileManifest
|
profiles map[string]artifacts.LLMProfileManifest
|
||||||
}
|
}
|
||||||
|
|
||||||
var _ contracts.StructuredLLMClient = (*ScriptoriumClient)(nil)
|
var _ contracts.StructuredLLMClient = (*PromptKitClient)(nil)
|
||||||
var _ contracts.LLMProfileManifestProvider = (*ScriptoriumClient)(nil)
|
var _ contracts.LLMProfileManifestProvider = (*PromptKitClient)(nil)
|
||||||
|
|
||||||
func NewScriptoriumClient(cfg ScriptoriumClientConfig) (*ScriptoriumClient, error) {
|
func PromptKitLocalBackendOption(cfg PromptKitLocalBackendConfig) promptkit.Option {
|
||||||
|
return promptkit.WithBackend(promptkit.LocalBackend(cfg.Endpoint, cfg.ConcurrencyLimit))
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
||||||
if cfg.Assets == nil {
|
if cfg.Assets == nil {
|
||||||
return nil, fmt.Errorf("scriptorium client assets must not be nil")
|
return nil, fmt.Errorf("PromptKit client assets must not be nil")
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
profileSource, profileOptions, err := promptKitProfileSourceEngineOptions(PromptKitProfileSourceConfig{
|
||||||
return nil, fmt.Errorf("scriptorium profile_dir and profile_file are mutually exclusive")
|
ProfileDir: cfg.ProfileDir,
|
||||||
}
|
ProfileFile: cfg.ProfileFile,
|
||||||
options, err := cfg.Assets.ScriptoriumOptions()
|
LocalBackend: cfg.LocalBackend,
|
||||||
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
if profileFile := strings.TrimSpace(cfg.ProfileFile); profileFile != "" {
|
options, fallbackProfileDigest, err := cfg.Assets.promptKitOptions()
|
||||||
options = append(options, scriptorium.WithProfileFile(profileFile))
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
}
|
}
|
||||||
|
options = append(options, profileOptions...)
|
||||||
options = append(options, cfg.EngineOptions...)
|
options = append(options, cfg.EngineOptions...)
|
||||||
|
|
||||||
engine, err := scriptorium.NewEngine(scriptorium.Config{
|
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||||
ProfileDir: strings.TrimSpace(cfg.ProfileDir),
|
ProfileDir: profileSource.ProfileDir,
|
||||||
Timeout: cfg.Timeout,
|
Timeout: cfg.Timeout,
|
||||||
HTTPClient: cfg.HTTPClient,
|
HTTPClient: cfg.HTTPClient,
|
||||||
}, options...)
|
}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("create Scriptorium engine: %w", err)
|
return nil, fmt.Errorf("create PromptKit engine: %w", err)
|
||||||
}
|
}
|
||||||
recorder := cfg.Recorder
|
recorder := cfg.Recorder
|
||||||
if recorder == nil {
|
if recorder == nil {
|
||||||
recorder = NewLLMProfileRecorder()
|
recorder = NewLLMProfileRecorder()
|
||||||
}
|
}
|
||||||
return &ScriptoriumClient{
|
var reasoningEffort *string
|
||||||
|
if cfg.ReasoningEffort != nil {
|
||||||
|
value := *cfg.ReasoningEffort
|
||||||
|
reasoningEffort = &value
|
||||||
|
}
|
||||||
|
return &PromptKitClient{
|
||||||
engine: engine,
|
engine: engine,
|
||||||
recorder: recorder,
|
recorder: recorder,
|
||||||
|
profileDir: profileSource.ProfileDir,
|
||||||
|
profileFile: profileSource.ProfileFile,
|
||||||
|
localEndpoint: profileSource.localEndpoint(),
|
||||||
|
fallbackProfileDigest: fallbackProfileDigest,
|
||||||
|
reasoningEffort: reasoningEffort,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *ScriptoriumClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
if c == nil {
|
if c == nil {
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("scriptorium client must not be nil")
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("PromptKit client must not be nil")
|
||||||
}
|
}
|
||||||
if c.engine == nil {
|
if c.engine == nil {
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("scriptorium client engine must not be nil")
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("PromptKit client engine must not be nil")
|
||||||
}
|
}
|
||||||
if err := validateOutputTarget(out); err != nil {
|
if err := validateOutputTarget(out); err != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, err
|
return contracts.StructuredCompletionResponse{}, err
|
||||||
@@ -89,54 +119,85 @@ func (c *ScriptoriumClient) CompleteStructured(ctx context.Context, req contract
|
|||||||
if promptID == "" {
|
if promptID == "" {
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion prompt_id must not be empty")
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion prompt_id must not be empty")
|
||||||
}
|
}
|
||||||
|
sessionID := strings.TrimSpace(req.SessionID)
|
||||||
|
var execution *promptkit.ExecutionTargetOverride
|
||||||
|
if c.reasoningEffort != nil {
|
||||||
|
reasoningEffort := *c.reasoningEffort
|
||||||
|
execution = &promptkit.ExecutionTargetOverride{
|
||||||
|
ReasoningEffort: &reasoningEffort,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
runReq := scriptorium.RunRequest{
|
runReq := promptkit.RunRequest{
|
||||||
PromptID: promptID,
|
PromptID: promptID,
|
||||||
PromptVersion: strings.TrimSpace(req.PromptVersion),
|
PromptVersion: strings.TrimSpace(req.PromptVersion),
|
||||||
ProfileID: strings.TrimSpace(req.ProfileID),
|
ProfileID: strings.TrimSpace(req.ProfileID),
|
||||||
Inputs: scriptoriumInputs(req.Inputs),
|
SessionID: sessionID,
|
||||||
Vars: scriptoriumVars(req),
|
Inputs: promptKitInputs(req.Inputs),
|
||||||
Metadata: scriptoriumMetadata(req),
|
Vars: promptKitVars(req, sessionID),
|
||||||
|
Execution: execution,
|
||||||
}
|
}
|
||||||
prepared, err := c.engine.Prepare(ctx, runReq)
|
prepared, err := c.engine.PrepareExecution(ctx, runReq)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, ctxErr
|
return contracts.StructuredCompletionResponse{}, ctxErr
|
||||||
}
|
}
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("prepare Scriptorium prompt %q: %w", promptID, redactScriptoriumError(err))
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("prepare PromptKit prompt %q: %w", promptID, redactPromptKitError(err))
|
||||||
}
|
}
|
||||||
result, err := c.engine.Run(ctx, runReq)
|
defer prepared.Discard()
|
||||||
|
preparedDetails := prepared.Details()
|
||||||
|
result, err := c.engine.RunPrepared(ctx, prepared)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
return contracts.StructuredCompletionResponse{}, ctxErr
|
return contracts.StructuredCompletionResponse{}, ctxErr
|
||||||
}
|
}
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run Scriptorium prompt %q: %w", promptID, redactScriptoriumError(err))
|
if errors.Is(err, promptkit.ErrCapacityExceeded) {
|
||||||
|
var capacityErr *promptkit.CapacityError
|
||||||
|
if errors.As(err, &capacityErr) && strings.TrimSpace(capacityErr.BackendID) != "" {
|
||||||
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf(
|
||||||
|
"run PromptKit prompt %q on backend %q: %w: %v",
|
||||||
|
promptID,
|
||||||
|
strings.TrimSpace(capacityErr.BackendID),
|
||||||
|
contracts.ErrLLMCapacityExceeded,
|
||||||
|
redactPromptKitError(err),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf(
|
||||||
|
"run PromptKit prompt %q: %w: %v",
|
||||||
|
promptID,
|
||||||
|
contracts.ErrLLMCapacityExceeded,
|
||||||
|
redactPromptKitError(err),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w", promptID, redactPromptKitError(err))
|
||||||
}
|
}
|
||||||
if result == nil {
|
if result == nil {
|
||||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run Scriptorium prompt %q: %w: empty result", promptID, contracts.ErrInvalidStructuredOutput)
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w: empty result", promptID, contracts.ErrInvalidStructuredOutput)
|
||||||
}
|
}
|
||||||
response := c.responseFromResult(result, prepared)
|
response := c.responseFromResult(result, &preparedDetails)
|
||||||
if result.Validation.Status == scriptorium.ValidationFailed || !result.Validation.IsValid {
|
if result.Validation.Status == promptkit.ValidationFailed || !result.Validation.IsValid {
|
||||||
return response, fmt.Errorf("run Scriptorium prompt %q: %w: validation failed: %s", promptID, contracts.ErrInvalidStructuredOutput, strings.Join(result.Validation.Errors, "; "))
|
return response, fmt.Errorf("run PromptKit prompt %q: %w: validation failed: %s", promptID, contracts.ErrInvalidStructuredOutput, strings.Join(result.Validation.Errors, "; "))
|
||||||
}
|
}
|
||||||
if len(strings.TrimSpace(string(response.Content))) == 0 {
|
if len(strings.TrimSpace(string(response.Content))) == 0 {
|
||||||
return response, fmt.Errorf("run Scriptorium prompt %q: %w: empty structured output", promptID, contracts.ErrInvalidStructuredOutput)
|
return response, fmt.Errorf("run PromptKit prompt %q: %w: empty structured output", promptID, contracts.ErrInvalidStructuredOutput)
|
||||||
}
|
}
|
||||||
if err := json.Unmarshal(response.Content, out); err != nil {
|
if err := json.Unmarshal(response.Content, out); err != nil {
|
||||||
return response, fmt.Errorf("decode Scriptorium structured output for prompt %q: %w: %w", promptID, contracts.ErrInvalidStructuredOutput, err)
|
return response, fmt.Errorf("decode PromptKit structured output for prompt %q: %w: %w", promptID, contracts.ErrInvalidStructuredOutput, err)
|
||||||
}
|
}
|
||||||
return response, nil
|
return response, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *ScriptoriumClient) responseFromResult(result *scriptorium.RunResult, prepared *scriptorium.PreparedRun) contracts.StructuredCompletionResponse {
|
func (c *PromptKitClient) responseFromResult(result *promptkit.RunResult, prepared *promptkit.PreparedRun) contracts.StructuredCompletionResponse {
|
||||||
content := result.Artifact.Body
|
content := result.Artifact.Body
|
||||||
if len(content) == 0 {
|
if len(content) == 0 {
|
||||||
content = []byte(result.RawOutput)
|
content = []byte(result.RawOutput)
|
||||||
}
|
}
|
||||||
profile := artifacts.LLMProfileManifest{
|
profile := artifacts.LLMProfileManifest{
|
||||||
ID: strings.TrimSpace(result.SelectedProfileID),
|
ID: strings.TrimSpace(result.SelectedProfileID),
|
||||||
Provider: scriptoriumProviderName,
|
Provider: promptKitProviderName,
|
||||||
Model: firstNonEmpty(result.ModelName, result.EffectiveModelParams.Model),
|
Model: firstNonEmpty(result.ModelName, result.EffectiveModelParams.Model),
|
||||||
|
BackendID: strings.TrimSpace(result.SelectedBackendID),
|
||||||
|
ReasoningEffort: strings.TrimSpace(result.EffectiveModelParams.ReasoningEffort),
|
||||||
}
|
}
|
||||||
if c.recorder != nil {
|
if c.recorder != nil {
|
||||||
c.recorder.Record(profile)
|
c.recorder.Record(profile)
|
||||||
@@ -149,17 +210,17 @@ func (c *ScriptoriumClient) responseFromResult(result *scriptorium.RunResult, pr
|
|||||||
PromptTokens: result.Usage.PromptTokens,
|
PromptTokens: result.Usage.PromptTokens,
|
||||||
CompletionTokens: result.Usage.CompletionTokens,
|
CompletionTokens: result.Usage.CompletionTokens,
|
||||||
TotalTokens: result.Usage.TotalTokens,
|
TotalTokens: result.Usage.TotalTokens,
|
||||||
Debug: scriptoriumDebugMaterial(prepared, result),
|
Debug: promptKitDebugMaterial(prepared, result),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumDebugMaterial(prepared *scriptorium.PreparedRun, result *scriptorium.RunResult) *contracts.LLMDebugMaterial {
|
func promptKitDebugMaterial(prepared *promptkit.PreparedRun, result *promptkit.RunResult) *contracts.LLMDebugMaterial {
|
||||||
material := &contracts.LLMDebugMaterial{}
|
material := &contracts.LLMDebugMaterial{}
|
||||||
if prepared != nil {
|
if prepared != nil {
|
||||||
material.Prompt = scriptoriumDebugPrompt(prepared)
|
material.Prompt = promptKitDebugPrompt(prepared)
|
||||||
}
|
}
|
||||||
if result != nil {
|
if result != nil {
|
||||||
material.Response = scriptoriumDebugResponse(result)
|
material.Response = promptKitDebugResponse(result)
|
||||||
}
|
}
|
||||||
if material.Prompt == nil && material.Response == nil {
|
if material.Prompt == nil && material.Response == nil {
|
||||||
return nil
|
return nil
|
||||||
@@ -167,7 +228,7 @@ func scriptoriumDebugMaterial(prepared *scriptorium.PreparedRun, result *scripto
|
|||||||
return material
|
return material
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumDebugPrompt(prepared *scriptorium.PreparedRun) *contracts.LLMDebugPrompt {
|
func promptKitDebugPrompt(prepared *promptkit.PreparedRun) *contracts.LLMDebugPrompt {
|
||||||
if prepared == nil {
|
if prepared == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -184,6 +245,7 @@ func scriptoriumDebugPrompt(prepared *scriptorium.PreparedRun) *contracts.LLMDeb
|
|||||||
PromptVersion: prepared.PromptVersion,
|
PromptVersion: prepared.PromptVersion,
|
||||||
PromptHash: prepared.PromptHash,
|
PromptHash: prepared.PromptHash,
|
||||||
SelectedProfileID: prepared.SelectedProfileID,
|
SelectedProfileID: prepared.SelectedProfileID,
|
||||||
|
SelectedBackendID: prepared.SelectedBackendID,
|
||||||
SessionID: prepared.SessionID,
|
SessionID: prepared.SessionID,
|
||||||
RenderedPromptHash: prepared.RenderedPromptHash,
|
RenderedPromptHash: prepared.RenderedPromptHash,
|
||||||
Messages: messages,
|
Messages: messages,
|
||||||
@@ -194,7 +256,7 @@ func scriptoriumDebugPrompt(prepared *scriptorium.PreparedRun) *contracts.LLMDeb
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumDebugResponse(result *scriptorium.RunResult) *contracts.LLMDebugResponse {
|
func promptKitDebugResponse(result *promptkit.RunResult) *contracts.LLMDebugResponse {
|
||||||
if result == nil {
|
if result == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -254,13 +316,28 @@ func cloneStringMap(values map[string]string) map[string]string {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *ScriptoriumClient) LLMProfileManifests() []artifacts.LLMProfileManifest {
|
func (c *PromptKitClient) LLMProfileManifests() []artifacts.LLMProfileManifest {
|
||||||
if c == nil || c.recorder == nil {
|
if c == nil || c.recorder == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return c.recorder.Manifests()
|
return c.recorder.Manifests()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (c *PromptKitClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint, error) {
|
||||||
|
if c == nil {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
fingerprint, err := promptKitProfileFingerprint(c.profileDir, c.profileFile, c.fallbackProfileDigest)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
fingerprints := []CheckpointFingerprint{fingerprint}
|
||||||
|
if c.localEndpoint != "" {
|
||||||
|
fingerprints = append(fingerprints, promptKitLocalBackendFingerprint(c.localEndpoint))
|
||||||
|
}
|
||||||
|
return fingerprints, nil
|
||||||
|
}
|
||||||
|
|
||||||
func NewLLMProfileRecorder() *LLMProfileRecorder {
|
func NewLLMProfileRecorder() *LLMProfileRecorder {
|
||||||
return &LLMProfileRecorder{profiles: map[string]artifacts.LLMProfileManifest{}}
|
return &LLMProfileRecorder{profiles: map[string]artifacts.LLMProfileManifest{}}
|
||||||
}
|
}
|
||||||
@@ -269,10 +346,8 @@ func (r *LLMProfileRecorder) Record(profile artifacts.LLMProfileManifest) {
|
|||||||
if r == nil {
|
if r == nil {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
profile.ID = strings.TrimSpace(profile.ID)
|
profile = profile.Normalized()
|
||||||
profile.Provider = strings.TrimSpace(profile.Provider)
|
key := profile.IdentityKey()
|
||||||
profile.Model = strings.TrimSpace(profile.Model)
|
|
||||||
key := profile.ID + "\x00" + profile.Provider + "\x00" + profile.Model
|
|
||||||
r.mu.Lock()
|
r.mu.Lock()
|
||||||
defer r.mu.Unlock()
|
defer r.mu.Unlock()
|
||||||
if r.profiles == nil {
|
if r.profiles == nil {
|
||||||
@@ -302,11 +377,11 @@ func (r *LLMProfileRecorder) Manifests() []artifacts.LLMProfileManifest {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumInputs(inputs contracts.LLMInputSet) map[string]scriptorium.ArtifactRef {
|
func promptKitInputs(inputs contracts.LLMInputSet) map[string]promptkit.ArtifactRef {
|
||||||
if len(inputs) == 0 {
|
if len(inputs) == 0 {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
out := make(map[string]scriptorium.ArtifactRef, len(inputs))
|
out := make(map[string]promptkit.ArtifactRef, len(inputs))
|
||||||
for key, material := range inputs {
|
for key, material := range inputs {
|
||||||
name := strings.TrimSpace(key)
|
name := strings.TrimSpace(key)
|
||||||
if name == "" {
|
if name == "" {
|
||||||
@@ -320,15 +395,15 @@ func scriptoriumInputs(inputs contracts.LLMInputSet) map[string]scriptorium.Arti
|
|||||||
body = " "
|
body = " "
|
||||||
}
|
}
|
||||||
if origin := strings.TrimSpace(material.OriginURI); origin != "" {
|
if origin := strings.TrimSpace(material.OriginURI); origin != "" {
|
||||||
out[name] = scriptorium.InlineWithURI(origin, body)
|
out[name] = promptkit.InlineWithURI(origin, body)
|
||||||
} else {
|
} else {
|
||||||
out[name] = scriptorium.Inline(body)
|
out[name] = promptkit.Inline(body)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumVars(req contracts.StructuredCompletionRequest) map[string]string {
|
func promptKitVars(req contracts.StructuredCompletionRequest, sessionID string) map[string]string {
|
||||||
vars := make(map[string]string, len(req.Vars)+1)
|
vars := make(map[string]string, len(req.Vars)+1)
|
||||||
for key, value := range req.Vars {
|
for key, value := range req.Vars {
|
||||||
name := strings.TrimSpace(key)
|
name := strings.TrimSpace(key)
|
||||||
@@ -337,7 +412,7 @@ func scriptoriumVars(req contracts.StructuredCompletionRequest) map[string]strin
|
|||||||
}
|
}
|
||||||
vars[name] = fmt.Sprint(value)
|
vars[name] = fmt.Sprint(value)
|
||||||
}
|
}
|
||||||
if sessionID := strings.TrimSpace(req.SessionID); sessionID != "" {
|
if sessionID != "" {
|
||||||
vars["session_id"] = sessionID
|
vars["session_id"] = sessionID
|
||||||
}
|
}
|
||||||
if len(vars) == 0 {
|
if len(vars) == 0 {
|
||||||
@@ -346,20 +421,9 @@ func scriptoriumVars(req contracts.StructuredCompletionRequest) map[string]strin
|
|||||||
return vars
|
return vars
|
||||||
}
|
}
|
||||||
|
|
||||||
func scriptoriumMetadata(req contracts.StructuredCompletionRequest) map[string]string {
|
|
||||||
metadata := map[string]string{}
|
|
||||||
if stageName := strings.TrimSpace(req.StageName); stageName != "" {
|
|
||||||
metadata["stage_name"] = stageName
|
|
||||||
}
|
|
||||||
if len(metadata) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
return metadata
|
|
||||||
}
|
|
||||||
|
|
||||||
var bearerTokenPattern = regexp.MustCompile(`(?i)Bearer\s+[A-Za-z0-9._~+/=-]+`)
|
var bearerTokenPattern = regexp.MustCompile(`(?i)Bearer\s+[A-Za-z0-9._~+/=-]+`)
|
||||||
|
|
||||||
func redactScriptoriumError(err error) error {
|
func redactPromptKitError(err error) error {
|
||||||
if err == nil {
|
if err == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
1269
internal/framework/llm/promptkit_client_test.go
Normal file
1269
internal/framework/llm/promptkit_client_test.go
Normal file
File diff suppressed because it is too large
Load Diff
110
internal/framework/llm/promptkit_profile_fingerprint.go
Normal file
110
internal/framework/llm/promptkit_profile_fingerprint.go
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
package llm
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"fmt"
|
||||||
|
"io/fs"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
promptKitProfileFingerprintName = "promptkit_profile_source"
|
||||||
|
promptKitLocalBackendFingerprintName = "promptkit_local_backend_target"
|
||||||
|
promptKitLocalBackendMarker = "notarius:promptkit-local-backend:v1"
|
||||||
|
// The built-in profile catalog is compiled into this pinned PromptKit
|
||||||
|
// release. Update this identity when the dependency is upgraded.
|
||||||
|
promptKitBuiltinProfileCatalogID = "promptkit:v0.5.0:builtin-profiles"
|
||||||
|
)
|
||||||
|
|
||||||
|
func promptKitProfileFingerprint(profileDir, profileFile, fallbackProfileDigest string) (CheckpointFingerprint, error) {
|
||||||
|
hasher := sha256.New()
|
||||||
|
writeFingerprintPart(hasher, []byte(promptKitBuiltinProfileCatalogID))
|
||||||
|
writeFingerprintPart(hasher, []byte(strings.TrimSpace(fallbackProfileDigest)))
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case strings.TrimSpace(profileFile) != "":
|
||||||
|
cleanProfileFile := strings.TrimSpace(profileFile)
|
||||||
|
data, err := os.ReadFile(cleanProfileFile)
|
||||||
|
if err != nil {
|
||||||
|
return CheckpointFingerprint{}, fmt.Errorf("read PromptKit profile file for checkpoint identity")
|
||||||
|
}
|
||||||
|
writeFingerprintPart(hasher, []byte(filepath.ToSlash(filepath.Base(cleanProfileFile))))
|
||||||
|
writeFingerprintPart(hasher, data)
|
||||||
|
case strings.TrimSpace(profileDir) != "":
|
||||||
|
files, err := promptKitProfileFiles(strings.TrimSpace(profileDir))
|
||||||
|
if err != nil {
|
||||||
|
return CheckpointFingerprint{}, err
|
||||||
|
}
|
||||||
|
for _, file := range files {
|
||||||
|
writeFingerprintPart(hasher, []byte(file.path))
|
||||||
|
writeFingerprintPart(hasher, file.digest)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return CheckpointFingerprint{
|
||||||
|
Name: promptKitProfileFingerprintName,
|
||||||
|
Value: "sha256:" + hex.EncodeToString(hasher.Sum(nil)),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptKitLocalBackendFingerprint(endpoint string) CheckpointFingerprint {
|
||||||
|
hasher := sha256.New()
|
||||||
|
writeFingerprintPart(hasher, []byte(promptKitLocalBackendMarker))
|
||||||
|
writeFingerprintPart(hasher, []byte(strings.TrimSpace(endpoint)))
|
||||||
|
return CheckpointFingerprint{
|
||||||
|
Name: promptKitLocalBackendFingerprintName,
|
||||||
|
Value: "sha256:" + hex.EncodeToString(hasher.Sum(nil)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type promptKitProfileFile struct {
|
||||||
|
path string
|
||||||
|
digest []byte
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptKitProfileFiles(root string) ([]promptKitProfileFile, error) {
|
||||||
|
var files []promptKitProfileFile
|
||||||
|
err := filepath.WalkDir(root, func(name string, entry fs.DirEntry, walkErr error) error {
|
||||||
|
if walkErr != nil {
|
||||||
|
return walkErr
|
||||||
|
}
|
||||||
|
if entry.IsDir() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
extension := filepath.Ext(entry.Name())
|
||||||
|
if extension != ".yaml" && extension != ".yml" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
data, err := os.ReadFile(name)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
relativePath, err := filepath.Rel(root, name)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
sum := sha256.Sum256(data)
|
||||||
|
files = append(files, promptKitProfileFile{
|
||||||
|
path: filepath.ToSlash(relativePath),
|
||||||
|
digest: append([]byte(nil), sum[:]...),
|
||||||
|
})
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read PromptKit profile directory for checkpoint identity")
|
||||||
|
}
|
||||||
|
sort.Slice(files, func(i, j int) bool {
|
||||||
|
return files[i].path < files[j].path
|
||||||
|
})
|
||||||
|
return files, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeFingerprintPart(hasher interface{ Write([]byte) (int, error) }, value []byte) {
|
||||||
|
length := []byte(fmt.Sprintf("%d:", len(value)))
|
||||||
|
_, _ = hasher.Write(length)
|
||||||
|
_, _ = hasher.Write(value)
|
||||||
|
}
|
||||||
146
internal/framework/llm/promptkit_profiles.go
Normal file
146
internal/framework/llm/promptkit_profiles.go
Normal file
@@ -0,0 +1,146 @@
|
|||||||
|
package llm
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
|
)
|
||||||
|
|
||||||
|
type PromptKitProfileSourceConfig struct {
|
||||||
|
ProfileDir string
|
||||||
|
ProfileFile string
|
||||||
|
LocalBackend *PromptKitLocalBackendConfig
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitProfileInspectorConfig struct {
|
||||||
|
Source PromptKitProfileSourceConfig
|
||||||
|
Assets *AssetRegistry
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c PromptKitProfileSourceConfig) localEndpoint() string {
|
||||||
|
if c.LocalBackend == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return c.LocalBackend.Endpoint
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitProfileInspector struct {
|
||||||
|
engine *promptkit.Engine
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitProfileInspection struct {
|
||||||
|
ProfileID string
|
||||||
|
BackendID string
|
||||||
|
Model string
|
||||||
|
CredentialEnvironment string
|
||||||
|
CredentialRequired bool
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptKitProfileInspectionError struct {
|
||||||
|
ProfileID string
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *PromptKitProfileInspectionError) Error() string {
|
||||||
|
switch {
|
||||||
|
case errors.Is(e.err, promptkit.ErrProfileNotFound):
|
||||||
|
return fmt.Sprintf("PromptKit profile %q is not configured", e.ProfileID)
|
||||||
|
case errors.Is(e.err, promptkit.ErrInvalidRequest):
|
||||||
|
return fmt.Sprintf("PromptKit profile ID %q is invalid", e.ProfileID)
|
||||||
|
case errors.Is(e.err, promptkit.ErrProfileLoad):
|
||||||
|
return fmt.Sprintf("PromptKit profile %q is invalid or unreadable", e.ProfileID)
|
||||||
|
default:
|
||||||
|
return fmt.Sprintf("PromptKit profile %q could not be inspected", e.ProfileID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *PromptKitProfileInspectionError) Unwrap() error {
|
||||||
|
return e.err
|
||||||
|
}
|
||||||
|
|
||||||
|
type promptKitProfileConfigurationError struct {
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *promptKitProfileConfigurationError) Error() string {
|
||||||
|
return "PromptKit profile configuration is invalid or unreadable"
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *promptKitProfileConfigurationError) Unwrap() error {
|
||||||
|
return e.err
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewPromptKitProfileInspector(cfg PromptKitProfileInspectorConfig) (*PromptKitProfileInspector, error) {
|
||||||
|
source, options, err := promptKitProfileSourceEngineOptions(cfg.Source)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if cfg.Assets != nil {
|
||||||
|
fallbackOption, hasFallback, err := cfg.Assets.promptKitFallbackProfileOption()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if hasFallback {
|
||||||
|
options = append(options, fallbackOption)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||||
|
PromptDir: ".",
|
||||||
|
ProfileDir: source.ProfileDir,
|
||||||
|
}, options...)
|
||||||
|
if err != nil {
|
||||||
|
return nil, &promptKitProfileConfigurationError{err: err}
|
||||||
|
}
|
||||||
|
return &PromptKitProfileInspector{engine: engine}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (i *PromptKitProfileInspector) InspectProfile(ctx context.Context, profileID string) (PromptKitProfileInspection, error) {
|
||||||
|
if i == nil || i.engine == nil {
|
||||||
|
return PromptKitProfileInspection{}, fmt.Errorf("PromptKit profile inspector must not be nil")
|
||||||
|
}
|
||||||
|
profileID = strings.TrimSpace(profileID)
|
||||||
|
inspection, err := i.engine.InspectProfile(ctx, profileID)
|
||||||
|
if err != nil {
|
||||||
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
|
return PromptKitProfileInspection{}, ctxErr
|
||||||
|
}
|
||||||
|
return PromptKitProfileInspection{}, &PromptKitProfileInspectionError{
|
||||||
|
ProfileID: profileID,
|
||||||
|
err: err,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return PromptKitProfileInspection{
|
||||||
|
ProfileID: inspection.ProfileID,
|
||||||
|
BackendID: strings.TrimSpace(inspection.EffectiveModelParams.BackendID),
|
||||||
|
Model: strings.TrimSpace(inspection.EffectiveModelParams.Model),
|
||||||
|
CredentialEnvironment: strings.TrimSpace(inspection.EffectiveModelParams.APIKeyEnv),
|
||||||
|
CredentialRequired: inspection.APIKeyRequired,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptKitProfileSourceEngineOptions(cfg PromptKitProfileSourceConfig) (PromptKitProfileSourceConfig, []promptkit.Option, error) {
|
||||||
|
source := PromptKitProfileSourceConfig{
|
||||||
|
ProfileDir: strings.TrimSpace(cfg.ProfileDir),
|
||||||
|
ProfileFile: strings.TrimSpace(cfg.ProfileFile),
|
||||||
|
}
|
||||||
|
if source.ProfileDir != "" && source.ProfileFile != "" {
|
||||||
|
return PromptKitProfileSourceConfig{}, nil, fmt.Errorf("PromptKit profile_dir and profile_file are mutually exclusive")
|
||||||
|
}
|
||||||
|
if cfg.LocalBackend != nil {
|
||||||
|
localBackend := *cfg.LocalBackend
|
||||||
|
localBackend.Endpoint = strings.TrimSpace(localBackend.Endpoint)
|
||||||
|
source.LocalBackend = &localBackend
|
||||||
|
}
|
||||||
|
|
||||||
|
var options []promptkit.Option
|
||||||
|
if source.ProfileFile != "" {
|
||||||
|
options = append(options, promptkit.WithProfileFile(source.ProfileFile))
|
||||||
|
}
|
||||||
|
if source.LocalBackend != nil {
|
||||||
|
options = append(options, PromptKitLocalBackendOption(*source.LocalBackend))
|
||||||
|
}
|
||||||
|
return source, options, nil
|
||||||
|
}
|
||||||
@@ -53,3 +53,14 @@ func (c *scheduledClient) LLMProfileManifests() []artifacts.LLMProfileManifest {
|
|||||||
}
|
}
|
||||||
return provider.LLMProfileManifests()
|
return provider.LLMProfileManifests()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (c *scheduledClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint, error) {
|
||||||
|
if c == nil || c.client == nil {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
provider, ok := c.client.(CheckpointFingerprintProvider)
|
||||||
|
if !ok {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return provider.LLMCheckpointFingerprints()
|
||||||
|
}
|
||||||
|
|||||||
@@ -75,6 +75,33 @@ func TestScheduledClientPropagatesSchedulerError(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestScheduledClientPreservesCheckpointFingerprints(t *testing.T) {
|
||||||
|
scheduler, err := NewScheduler(1)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
inner := &fingerprintedStructuredClient{
|
||||||
|
fingerprints: []CheckpointFingerprint{
|
||||||
|
{Name: "profile_source", Value: "sha256:one"},
|
||||||
|
{Name: "backend_target", Value: "sha256:two"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
client := NewScheduledClient(inner, scheduler)
|
||||||
|
provider, ok := client.(CheckpointFingerprintProvider)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("scheduled client %T does not preserve checkpoint fingerprints", client)
|
||||||
|
}
|
||||||
|
got, err := provider.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(got) != len(inner.fingerprints) ||
|
||||||
|
got[0] != inner.fingerprints[0] ||
|
||||||
|
got[1] != inner.fingerprints[1] {
|
||||||
|
t.Fatalf("checkpoint fingerprints = %#v, want %#v", got, inner.fingerprints)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
type blockingStructuredClient struct {
|
type blockingStructuredClient struct {
|
||||||
release chan struct{}
|
release chan struct{}
|
||||||
inFlight int32
|
inFlight int32
|
||||||
@@ -110,6 +137,15 @@ type errorStructuredClient struct {
|
|||||||
err error
|
err error
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type fingerprintedStructuredClient struct {
|
||||||
|
errorStructuredClient
|
||||||
|
fingerprints []CheckpointFingerprint
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *fingerprintedStructuredClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint, error) {
|
||||||
|
return append([]CheckpointFingerprint(nil), c.fingerprints...), nil
|
||||||
|
}
|
||||||
|
|
||||||
func (c *errorStructuredClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
func (c *errorStructuredClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
return contracts.StructuredCompletionResponse{}, c.err
|
return contracts.StructuredCompletionResponse{}, c.err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,372 +0,0 @@
|
|||||||
package llm
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"strings"
|
|
||||||
"sync"
|
|
||||||
"sync/atomic"
|
|
||||||
"testing"
|
|
||||||
"testing/fstest"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestScriptoriumClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
|
|
||||||
fake := &fakeScriptoriumLLM{content: `{"ok":true}`}
|
|
||||||
client := newTestScriptoriumClient(t, fake)
|
|
||||||
|
|
||||||
var out struct {
|
|
||||||
OK bool `json:"ok"`
|
|
||||||
}
|
|
||||||
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
|
||||||
StageName: "test-stage",
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
PromptVersion: "v1",
|
|
||||||
ProfileID: "explicit-profile",
|
|
||||||
SessionID: "session-123",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "sha256:source", "file:///source.json"),
|
|
||||||
},
|
|
||||||
Vars: map[string]any{"custom": "value"},
|
|
||||||
}, &out)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
if !out.OK {
|
|
||||||
t.Fatalf("decoded output OK = false, want true")
|
|
||||||
}
|
|
||||||
if resp.Provider != scriptoriumProviderName || resp.Model != "explicit-model" || resp.ProfileID != "explicit-profile" {
|
|
||||||
t.Fatalf("response metadata = %#v", resp)
|
|
||||||
}
|
|
||||||
if resp.PromptTokens != 11 || resp.CompletionTokens != 7 || resp.TotalTokens != 18 {
|
|
||||||
t.Fatalf("usage = %#v, want mapped token counts", resp)
|
|
||||||
}
|
|
||||||
if resp.Debug == nil || resp.Debug.Prompt == nil {
|
|
||||||
t.Fatalf("debug prompt = nil, want prepared prompt material")
|
|
||||||
}
|
|
||||||
if resp.Debug.Prompt.PromptID != "adapter.test" || resp.Debug.Prompt.SelectedProfileID != "explicit-profile" {
|
|
||||||
t.Fatalf("debug prompt metadata = %#v, want prompt/profile", resp.Debug.Prompt)
|
|
||||||
}
|
|
||||||
if len(resp.Debug.Prompt.Messages) != 1 || !strings.Contains(resp.Debug.Prompt.Messages[0].Content, `{"source":true}`) {
|
|
||||||
t.Fatalf("debug prompt messages = %#v, want rendered input content", resp.Debug.Prompt.Messages)
|
|
||||||
}
|
|
||||||
if resp.Debug.Response == nil || resp.Debug.Response.Content != `{"ok":true}` {
|
|
||||||
t.Fatalf("debug response = %#v, want raw response content", resp.Debug.Response)
|
|
||||||
}
|
|
||||||
debugJSON, err := json.Marshal(resp.Debug)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("marshal debug material: %v", err)
|
|
||||||
}
|
|
||||||
if strings.Contains(string(debugJSON), "secret-token") || strings.Contains(string(debugJSON), "sk-") {
|
|
||||||
t.Fatalf("debug material contains secret material: %s", debugJSON)
|
|
||||||
}
|
|
||||||
gotReq := fake.lastRequest()
|
|
||||||
if gotReq.Prompt.SessionID != "session-123" {
|
|
||||||
t.Fatalf("session id = %q, want session-123", gotReq.Prompt.SessionID)
|
|
||||||
}
|
|
||||||
if gotReq.Target.Model != "explicit-model" {
|
|
||||||
t.Fatalf("model = %q, want explicit-model", gotReq.Target.Model)
|
|
||||||
}
|
|
||||||
if len(gotReq.Prompt.Messages) != 1 || !strings.Contains(gotReq.Prompt.Messages[0].Content, `{"source":true}`) {
|
|
||||||
t.Fatalf("rendered messages = %#v, want transcript input content", gotReq.Prompt.Messages)
|
|
||||||
}
|
|
||||||
if gotReq.StructuredOutput == nil {
|
|
||||||
t.Fatalf("structured output = nil, want JSON schema")
|
|
||||||
}
|
|
||||||
manifests := client.LLMProfileManifests()
|
|
||||||
if len(manifests) != 1 || manifests[0].ID != "explicit-profile" || manifests[0].Model != "explicit-model" {
|
|
||||||
t.Fatalf("profile manifests = %#v", manifests)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScriptoriumClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) {
|
|
||||||
fake := &fakeScriptoriumLLM{content: `{"ok":true}`}
|
|
||||||
client := newTestScriptoriumClient(t, fake)
|
|
||||||
|
|
||||||
var out map[string]any
|
|
||||||
if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
SessionID: "session-123",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
|
||||||
},
|
|
||||||
}, &out); err != nil {
|
|
||||||
t.Fatalf("CompleteStructured() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
if got := fake.lastRequest().Target.Model; got != "default-model" {
|
|
||||||
t.Fatalf("model = %q, want prompt default profile model", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScriptoriumClientValidationFailureReturnsError(t *testing.T) {
|
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"bad":true}`})
|
|
||||||
|
|
||||||
var out map[string]any
|
|
||||||
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
SessionID: "session-123",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
|
||||||
},
|
|
||||||
}, &out)
|
|
||||||
if err == nil || !errors.Is(err, contracts.ErrInvalidStructuredOutput) || !strings.Contains(err.Error(), "validation failed") {
|
|
||||||
t.Fatalf("CompleteStructured() error = %v, want validation failure", err)
|
|
||||||
}
|
|
||||||
if got := string(resp.Content); got != `{"bad":true}` {
|
|
||||||
t.Fatalf("response content = %q, want raw failed output", got)
|
|
||||||
}
|
|
||||||
if resp.Debug == nil || resp.Debug.Response == nil || resp.Debug.Response.Content != `{"bad":true}` {
|
|
||||||
t.Fatalf("debug response = %#v, want raw failed output", resp.Debug)
|
|
||||||
}
|
|
||||||
if resp.Debug.Prompt == nil || len(resp.Debug.Prompt.Messages) == 0 {
|
|
||||||
t.Fatalf("debug prompt = %#v, want prepared prompt material", resp.Debug.Prompt)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScriptoriumClientDecodeFailureReturnsRawResponse(t *testing.T) {
|
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"ok":true}`})
|
|
||||||
|
|
||||||
var out []any
|
|
||||||
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
SessionID: "session-123",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
|
||||||
},
|
|
||||||
}, &out)
|
|
||||||
if err == nil || !errors.Is(err, contracts.ErrInvalidStructuredOutput) || !strings.Contains(err.Error(), "decode Scriptorium structured output") {
|
|
||||||
t.Fatalf("CompleteStructured() error = %v, want decode failure", err)
|
|
||||||
}
|
|
||||||
if got := string(resp.Content); got != `{"ok":true}` {
|
|
||||||
t.Fatalf("response content = %q, want raw decode-failed output", got)
|
|
||||||
}
|
|
||||||
if resp.Debug == nil || resp.Debug.Response == nil || resp.Debug.Response.Content != `{"ok":true}` {
|
|
||||||
t.Fatalf("debug response = %#v, want raw decode-failed output", resp.Debug)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScriptoriumClientProviderFailureIncludesContextAndRedactsBearerToken(t *testing.T) {
|
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{err: errors.New("provider failed with Bearer secret-token")})
|
|
||||||
|
|
||||||
var out map[string]any
|
|
||||||
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
SessionID: "session-123",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
|
||||||
},
|
|
||||||
}, &out)
|
|
||||||
if err == nil {
|
|
||||||
t.Fatalf("CompleteStructured() error = nil, want provider error")
|
|
||||||
}
|
|
||||||
if errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
|
||||||
t.Fatalf("provider error = %v, must not be classified as invalid structured output", err)
|
|
||||||
}
|
|
||||||
if !strings.Contains(err.Error(), `run Scriptorium prompt "adapter.test"`) {
|
|
||||||
t.Fatalf("error = %q, want operation context", err.Error())
|
|
||||||
}
|
|
||||||
if strings.Contains(err.Error(), "secret-token") || !strings.Contains(err.Error(), "Bearer [REDACTED]") {
|
|
||||||
t.Fatalf("error = %q, want redacted bearer token", err.Error())
|
|
||||||
}
|
|
||||||
if resp.Debug != nil {
|
|
||||||
t.Fatalf("debug material = %#v, want none for provider failure without result", resp.Debug)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScriptoriumClientContextCancellationIsRespected(t *testing.T) {
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
cancel()
|
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"ok":true}`})
|
|
||||||
|
|
||||||
var out map[string]any
|
|
||||||
_, err := client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
|
||||||
},
|
|
||||||
}, &out)
|
|
||||||
if !errors.Is(err, context.Canceled) || errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
|
||||||
t.Fatalf("CompleteStructured() error = %v, want context canceled", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScriptoriumClientClassifiesEmptyStructuredCompletion(t *testing.T) {
|
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{allowEmpty: true})
|
|
||||||
|
|
||||||
var out map[string]any
|
|
||||||
_, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
SessionID: "session-123",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
|
||||||
},
|
|
||||||
}, &out)
|
|
||||||
if !errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
|
||||||
t.Fatalf("CompleteStructured() error = %v, want invalid structured output", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScheduledScriptoriumClientBoundsConcurrentCalls(t *testing.T) {
|
|
||||||
fake := &fakeScriptoriumLLM{
|
|
||||||
content: `{"ok":true}`,
|
|
||||||
block: make(chan struct{}),
|
|
||||||
}
|
|
||||||
client := newTestScriptoriumClient(t, fake)
|
|
||||||
scheduler, err := NewScheduler(1)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("NewScheduler() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
scheduled := NewScheduledClient(client, scheduler)
|
|
||||||
|
|
||||||
var wg sync.WaitGroup
|
|
||||||
for i := 0; i < 3; i++ {
|
|
||||||
wg.Add(1)
|
|
||||||
go func() {
|
|
||||||
defer wg.Done()
|
|
||||||
var out map[string]any
|
|
||||||
_, callErr := scheduled.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
|
||||||
PromptID: "adapter.test",
|
|
||||||
SessionID: "session-123",
|
|
||||||
Inputs: contracts.LLMInputSet{
|
|
||||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
|
||||||
},
|
|
||||||
}, &out)
|
|
||||||
if callErr != nil {
|
|
||||||
t.Errorf("CompleteStructured() error = %v, want nil", callErr)
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
}
|
|
||||||
waitForAtomicAtLeast(t, &fake.calls, 1)
|
|
||||||
time.Sleep(20 * time.Millisecond)
|
|
||||||
if got := atomic.LoadInt32(&fake.maxInFlight); got > 1 {
|
|
||||||
t.Fatalf("max in-flight calls = %d, want <= 1", got)
|
|
||||||
}
|
|
||||||
close(fake.block)
|
|
||||||
wg.Wait()
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScriptoriumClientValidatesRequest(t *testing.T) {
|
|
||||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"ok":true}`})
|
|
||||||
var out map[string]any
|
|
||||||
if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{}, &out); err == nil || !strings.Contains(err.Error(), "prompt_id") {
|
|
||||||
t.Fatalf("missing prompt id error = %v, want prompt_id validation", err)
|
|
||||||
}
|
|
||||||
if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{PromptID: "adapter.test"}, nil); err == nil || !strings.Contains(err.Error(), "non-nil pointer") {
|
|
||||||
t.Fatalf("nil output error = %v, want output validation", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func newTestScriptoriumClient(t *testing.T, fake *fakeScriptoriumLLM) *ScriptoriumClient {
|
|
||||||
t.Helper()
|
|
||||||
registry := NewAssetRegistry()
|
|
||||||
if err := registry.RegisterPromptFS(fstest.MapFS{
|
|
||||||
"adapter.test.yaml": {Data: []byte(`id: adapter.test
|
|
||||||
version: "v1"
|
|
||||||
default_profile: default-profile
|
|
||||||
session_id: "{{ .session_id }}"
|
|
||||||
inputs:
|
|
||||||
- name: transcript
|
|
||||||
required: true
|
|
||||||
content_type: application/json
|
|
||||||
messages:
|
|
||||||
- role: user
|
|
||||||
content: "Transcript: {{ input \"transcript\" }}"
|
|
||||||
output:
|
|
||||||
format: json
|
|
||||||
validation_mode: json_schema
|
|
||||||
schema_path: adapter.schema.json
|
|
||||||
repair_attempts: 0
|
|
||||||
`)},
|
|
||||||
}, "."); err != nil {
|
|
||||||
t.Fatalf("RegisterPromptFS() error = %v", err)
|
|
||||||
}
|
|
||||||
if err := registry.RegisterSchemaFS(fstest.MapFS{
|
|
||||||
"adapter.schema.json": {Data: []byte(`{"type":"object","required":["ok"],"properties":{"ok":{"type":"boolean"}}}`)},
|
|
||||||
}, "."); err != nil {
|
|
||||||
t.Fatalf("RegisterSchemaFS() error = %v", err)
|
|
||||||
}
|
|
||||||
client, err := NewScriptoriumClient(ScriptoriumClientConfig{
|
|
||||||
Assets: registry,
|
|
||||||
EngineOptions: []scriptorium.Option{
|
|
||||||
scriptorium.WithProfiles(
|
|
||||||
scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
|
||||||
ID: "default-profile",
|
|
||||||
Endpoint: "http://127.0.0.1:1/v1",
|
|
||||||
Model: "default-model",
|
|
||||||
}),
|
|
||||||
scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
|
||||||
ID: "explicit-profile",
|
|
||||||
Endpoint: "http://127.0.0.1:1/v1",
|
|
||||||
Model: "explicit-model",
|
|
||||||
}),
|
|
||||||
),
|
|
||||||
scriptorium.WithLLMClient(fake),
|
|
||||||
},
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("NewScriptoriumClient() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
return client
|
|
||||||
}
|
|
||||||
|
|
||||||
type fakeScriptoriumLLM struct {
|
|
||||||
content string
|
|
||||||
allowEmpty bool
|
|
||||||
err error
|
|
||||||
block chan struct{}
|
|
||||||
mu sync.Mutex
|
|
||||||
last scriptorium.GenerateRequest
|
|
||||||
calls int32
|
|
||||||
inFlight int32
|
|
||||||
maxInFlight int32
|
|
||||||
}
|
|
||||||
|
|
||||||
func (f *fakeScriptoriumLLM) Generate(ctx context.Context, req scriptorium.GenerateRequest) (*scriptorium.GenerateResponse, error) {
|
|
||||||
f.mu.Lock()
|
|
||||||
f.last = req
|
|
||||||
f.mu.Unlock()
|
|
||||||
atomic.AddInt32(&f.calls, 1)
|
|
||||||
current := atomic.AddInt32(&f.inFlight, 1)
|
|
||||||
for {
|
|
||||||
seen := atomic.LoadInt32(&f.maxInFlight)
|
|
||||||
if current <= seen || atomic.CompareAndSwapInt32(&f.maxInFlight, seen, current) {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
defer atomic.AddInt32(&f.inFlight, -1)
|
|
||||||
if f.block != nil {
|
|
||||||
select {
|
|
||||||
case <-f.block:
|
|
||||||
case <-ctx.Done():
|
|
||||||
return nil, ctx.Err()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if f.err != nil {
|
|
||||||
return nil, f.err
|
|
||||||
}
|
|
||||||
content := f.content
|
|
||||||
if content == "" && !f.allowEmpty {
|
|
||||||
content = `{"ok":true}`
|
|
||||||
}
|
|
||||||
if !f.allowEmpty && !json.Valid([]byte(content)) {
|
|
||||||
return nil, errors.New("test fake must return JSON content")
|
|
||||||
}
|
|
||||||
return &scriptorium.GenerateResponse{
|
|
||||||
Content: content,
|
|
||||||
Usage: scriptorium.TokenUsage{
|
|
||||||
PromptTokens: 11,
|
|
||||||
CompletionTokens: 7,
|
|
||||||
TotalTokens: 18,
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (f *fakeScriptoriumLLM) lastRequest() scriptorium.GenerateRequest {
|
|
||||||
f.mu.Lock()
|
|
||||||
defer f.mu.Unlock()
|
|
||||||
return f.last
|
|
||||||
}
|
|
||||||
148
internal/framework/pipeline/artifact_evidence_registry.go
Normal file
148
internal/framework/pipeline/artifact_evidence_registry.go
Normal file
@@ -0,0 +1,148 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"reflect"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ArtifactEvidenceProjector returns the direct source references represented
|
||||||
|
// by one normalized artifact value.
|
||||||
|
type ArtifactEvidenceProjector[T any] func(T) []source.SourceRef
|
||||||
|
|
||||||
|
// ArtifactEvidenceRegistry keeps the typed projection boundary private while
|
||||||
|
// allowing preparation and execution to discover registered capabilities.
|
||||||
|
type ArtifactEvidenceRegistry struct {
|
||||||
|
entries map[contracts.ArtifactKind]artifactEvidenceEntry
|
||||||
|
}
|
||||||
|
|
||||||
|
type artifactEvidenceEntry struct {
|
||||||
|
valueType reflect.Type
|
||||||
|
project func(any) ([]source.SourceRef, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewArtifactEvidenceRegistry() *ArtifactEvidenceRegistry {
|
||||||
|
return &ArtifactEvidenceRegistry{entries: make(map[contracts.ArtifactKind]artifactEvidenceEntry)}
|
||||||
|
}
|
||||||
|
|
||||||
|
// RegisterArtifactEvidence registers one evidence projector for an artifact
|
||||||
|
// kind. Projectors are invoked only after an exact Go-type check.
|
||||||
|
func RegisterArtifactEvidence[T any](registry *ArtifactEvidenceRegistry, kind contracts.ArtifactKind, projector ArtifactEvidenceProjector[T]) error {
|
||||||
|
if registry == nil {
|
||||||
|
return fmt.Errorf("artifact evidence registry must not be nil")
|
||||||
|
}
|
||||||
|
if projector == nil {
|
||||||
|
return fmt.Errorf("artifact evidence projector must not be nil")
|
||||||
|
}
|
||||||
|
kind = normalizeArtifactKind(kind)
|
||||||
|
if kind == "" {
|
||||||
|
return fmt.Errorf("artifact evidence kind must not be empty")
|
||||||
|
}
|
||||||
|
if _, ok := registry.entries[kind]; ok {
|
||||||
|
return fmt.Errorf("artifact evidence %q is already registered", kind)
|
||||||
|
}
|
||||||
|
valueType := reflect.TypeFor[T]()
|
||||||
|
if registry.entries == nil {
|
||||||
|
registry.entries = make(map[contracts.ArtifactKind]artifactEvidenceEntry)
|
||||||
|
}
|
||||||
|
registry.entries[kind] = artifactEvidenceEntry{
|
||||||
|
valueType: valueType,
|
||||||
|
project: func(value any) ([]source.SourceRef, error) {
|
||||||
|
actualType := reflect.TypeOf(value)
|
||||||
|
if actualType != valueType {
|
||||||
|
return nil, newArtifactEvidenceTypeError(kind, valueType, actualType)
|
||||||
|
}
|
||||||
|
typed, ok := value.(T)
|
||||||
|
if !ok {
|
||||||
|
return nil, newArtifactEvidenceTypeError(kind, valueType, actualType)
|
||||||
|
}
|
||||||
|
return append([]source.SourceRef(nil), projector(typed)...), nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *ArtifactEvidenceRegistry) RegisteredKinds() []contracts.ArtifactKind {
|
||||||
|
if r == nil || len(r.entries) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
kinds := make([]contracts.ArtifactKind, 0, len(r.entries))
|
||||||
|
for kind := range r.entries {
|
||||||
|
kinds = append(kinds, kind)
|
||||||
|
}
|
||||||
|
sort.Slice(kinds, func(i, j int) bool { return kinds[i] < kinds[j] })
|
||||||
|
return kinds
|
||||||
|
}
|
||||||
|
|
||||||
|
// Project returns independently owned direct references for a registered
|
||||||
|
// artifact value.
|
||||||
|
func (r *ArtifactEvidenceRegistry) Project(kind contracts.ArtifactKind, value any) ([]source.SourceRef, error) {
|
||||||
|
entry, normalizedKind, err := r.entry(kind)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
refs, err := entry.project(value)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("project artifact evidence %q: %w", normalizedKind, err)
|
||||||
|
}
|
||||||
|
return append([]source.SourceRef(nil), refs...), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *ArtifactEvidenceRegistry) entry(kind contracts.ArtifactKind) (artifactEvidenceEntry, contracts.ArtifactKind, error) {
|
||||||
|
if r == nil {
|
||||||
|
return artifactEvidenceEntry{}, "", fmt.Errorf("artifact evidence registry must not be nil")
|
||||||
|
}
|
||||||
|
kind = normalizeArtifactKind(kind)
|
||||||
|
if kind == "" {
|
||||||
|
return artifactEvidenceEntry{}, "", fmt.Errorf("artifact evidence kind must not be empty")
|
||||||
|
}
|
||||||
|
entry, ok := r.entries[kind]
|
||||||
|
if !ok {
|
||||||
|
return artifactEvidenceEntry{}, kind, fmt.Errorf("artifact evidence %q is not registered", kind)
|
||||||
|
}
|
||||||
|
return entry, kind, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *ArtifactEvidenceRegistry) valueType(kind contracts.ArtifactKind) (reflect.Type, bool) {
|
||||||
|
if r == nil {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
entry, ok := r.entries[normalizeArtifactKind(kind)]
|
||||||
|
if !ok {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
return entry.valueType, true
|
||||||
|
}
|
||||||
|
|
||||||
|
func newArtifactEvidenceTypeError(kind contracts.ArtifactKind, expected, actual reflect.Type) error {
|
||||||
|
actualName := "<nil>"
|
||||||
|
if actual != nil {
|
||||||
|
actualName = actual.String()
|
||||||
|
}
|
||||||
|
return fmt.Errorf("project artifact evidence %q: expected exact Go type %s, got %s", kind, expected, actualName)
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizeEvidenceLaneIDs(values []string) ([]string, error) {
|
||||||
|
if len(values) == 0 {
|
||||||
|
return nil, fmt.Errorf("evidence lane ids must not be empty")
|
||||||
|
}
|
||||||
|
seen := make(map[string]struct{}, len(values))
|
||||||
|
lanes := make([]string, 0, len(values))
|
||||||
|
for _, raw := range values {
|
||||||
|
lane := strings.TrimSpace(raw)
|
||||||
|
if lane == "" {
|
||||||
|
return nil, fmt.Errorf("evidence lane id must not be empty")
|
||||||
|
}
|
||||||
|
if _, ok := seen[lane]; ok {
|
||||||
|
return nil, fmt.Errorf("evidence lane id %q is duplicated", lane)
|
||||||
|
}
|
||||||
|
seen[lane] = struct{}{}
|
||||||
|
lanes = append(lanes, lane)
|
||||||
|
}
|
||||||
|
sort.Strings(lanes)
|
||||||
|
return lanes, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestArtifactEvidenceRegistryProjectsExactRegisteredTypesWithOwnedReferences(t *testing.T) {
|
||||||
|
registry := NewArtifactEvidenceRegistry()
|
||||||
|
input := codecNotes{Items: []string{"one"}}
|
||||||
|
refs := []source.SourceRef{{SourceID: "source", StartUnitID: 2, EndUnitID: 2}}
|
||||||
|
if err := RegisterArtifactEvidence(registry, "test/notes", func(codecNotes) []source.SourceRef { return refs }); err != nil {
|
||||||
|
t.Fatalf("RegisterArtifactEvidence() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if err := RegisterArtifactEvidence(registry, "test/score", func(codecScore) []source.SourceRef { return nil }); err != nil {
|
||||||
|
t.Fatalf("RegisterArtifactEvidence() second kind error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if got, want := registry.RegisteredKinds(), []contracts.ArtifactKind{"test/notes", "test/score"}; !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("RegisteredKinds() = %#v, want %#v", got, want)
|
||||||
|
}
|
||||||
|
projected, err := registry.Project(" test/notes ", input)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Project() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
projected[0].StartUnitID = 99
|
||||||
|
if refs[0].StartUnitID != 2 {
|
||||||
|
t.Fatal("Project() retained projector reference storage")
|
||||||
|
}
|
||||||
|
if _, err := registry.Project("test/notes", codecNotesAlias(input)); err == nil || !strings.Contains(err.Error(), "expected exact Go type") {
|
||||||
|
t.Fatalf("Project() exact type error = %v, want exact type failure", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestArtifactEvidenceRegistryRejectsInvalidRegistration(t *testing.T) {
|
||||||
|
registry := NewArtifactEvidenceRegistry()
|
||||||
|
if err := RegisterArtifactEvidence[codecNotes](nil, "test/notes", func(codecNotes) []source.SourceRef { return nil }); err == nil || !strings.Contains(err.Error(), "must not be nil") {
|
||||||
|
t.Fatalf("nil registry error = %v, want failure", err)
|
||||||
|
}
|
||||||
|
if err := RegisterArtifactEvidence(registry, " ", func(codecNotes) []source.SourceRef { return nil }); err == nil || !strings.Contains(err.Error(), "must not be empty") {
|
||||||
|
t.Fatalf("blank kind error = %v, want failure", err)
|
||||||
|
}
|
||||||
|
if err := RegisterArtifactEvidence(registry, "test/notes", ArtifactEvidenceProjector[codecNotes](nil)); err == nil || !strings.Contains(err.Error(), "must not be nil") {
|
||||||
|
t.Fatalf("nil projector error = %v, want failure", err)
|
||||||
|
}
|
||||||
|
if err := RegisterArtifactEvidence(registry, "test/notes", func(codecNotes) []source.SourceRef { return nil }); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := RegisterArtifactEvidence(registry, "test/notes", func(codecNotes) []source.SourceRef { return nil }); err == nil || !strings.Contains(err.Error(), "already registered") {
|
||||||
|
t.Fatalf("duplicate kind error = %v, want failure", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -24,10 +24,6 @@ func NewChunkerRegistry() *ChunkerRegistry {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *ChunkerRegistry) Register(key string, constructor ChunkerConstructor) error {
|
|
||||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageChunk), constructor)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerConstructor) error {
|
func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerConstructor) error {
|
||||||
if constructor == nil {
|
if constructor == nil {
|
||||||
return fmt.Errorf("chunker constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
return fmt.Errorf("chunker constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
|
|||||||
@@ -29,6 +29,10 @@ type registryBehaviorCase[M any] struct {
|
|||||||
moduleKey func(M) string
|
moduleKey func(M) string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func testModuleSpec(key string, stage ModuleStage) ModuleSpec {
|
||||||
|
return ModuleSpec{Key: key, Stage: stage, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
|
}
|
||||||
|
|
||||||
func TestChunkerRegistryBehavior(t *testing.T) {
|
func TestChunkerRegistryBehavior(t *testing.T) {
|
||||||
runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Chunker]{
|
runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Chunker]{
|
||||||
name: "ChunkerRegistry",
|
name: "ChunkerRegistry",
|
||||||
@@ -39,7 +43,9 @@ func TestChunkerRegistryBehavior(t *testing.T) {
|
|||||||
return NewChunkerRegistry()
|
return NewChunkerRegistry()
|
||||||
},
|
},
|
||||||
register: func(registry any, key string, constructor func() (contracts.Chunker, error)) error {
|
register: func(registry any, key string, constructor func() (contracts.Chunker, error)) error {
|
||||||
return registry.(*ChunkerRegistry).Register(key, constructor)
|
return registry.(*ChunkerRegistry).RegisterWithSpec(ModuleSpec{
|
||||||
|
Key: key, Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
}, constructor)
|
||||||
},
|
},
|
||||||
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Chunker, error)) error {
|
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Chunker, error)) error {
|
||||||
return registry.(*ChunkerRegistry).RegisterWithSpec(spec, constructor)
|
return registry.(*ChunkerRegistry).RegisterWithSpec(spec, constructor)
|
||||||
@@ -55,7 +61,9 @@ func TestChunkerRegistryBehavior(t *testing.T) {
|
|||||||
},
|
},
|
||||||
nilRegister: func(key string, constructor func() (contracts.Chunker, error)) error {
|
nilRegister: func(key string, constructor func() (contracts.Chunker, error)) error {
|
||||||
var registry *ChunkerRegistry
|
var registry *ChunkerRegistry
|
||||||
return registry.Register(key, constructor)
|
return registry.RegisterWithSpec(ModuleSpec{
|
||||||
|
Key: key, Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
}, constructor)
|
||||||
},
|
},
|
||||||
nilBuild: func(key string) (contracts.Chunker, error) {
|
nilBuild: func(key string) (contracts.Chunker, error) {
|
||||||
var registry *ChunkerRegistry
|
var registry *ChunkerRegistry
|
||||||
@@ -103,6 +111,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
spec := ModuleSpec{
|
spec := ModuleSpec{
|
||||||
Key: " " + testCase.key + " ",
|
Key: " " + testCase.key + " ",
|
||||||
Stage: testCase.stage,
|
Stage: testCase.stage,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{" beta ", "alpha", "", "beta"},
|
Provides: []string{" beta ", "alpha", "", "beta"},
|
||||||
Requires: []string{" source ", "source", ""},
|
Requires: []string{" source ", "source", ""},
|
||||||
}
|
}
|
||||||
@@ -117,6 +126,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
want := ModuleSpec{
|
want := ModuleSpec{
|
||||||
Key: testCase.key,
|
Key: testCase.key,
|
||||||
Stage: testCase.stage,
|
Stage: testCase.stage,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"alpha", "beta"},
|
Provides: []string{"alpha", "beta"},
|
||||||
Requires: []string{"source"},
|
Requires: []string{"source"},
|
||||||
}
|
}
|
||||||
@@ -134,7 +144,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
t.Run(testCase.name+"/default spec from register", func(t *testing.T) {
|
t.Run(testCase.name+"/minimal explicit spec registration", func(t *testing.T) {
|
||||||
registry := testCase.newRegistry()
|
registry := testCase.newRegistry()
|
||||||
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
|
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
@@ -144,7 +154,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
if !ok {
|
if !ok {
|
||||||
t.Fatal("Spec() ok = false, want true")
|
t.Fatal("Spec() ok = false, want true")
|
||||||
}
|
}
|
||||||
want := ModuleSpec{Key: testCase.key, Stage: testCase.stage}
|
want := ModuleSpec{Key: testCase.key, Stage: testCase.stage, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
if !reflect.DeepEqual(spec, want) {
|
if !reflect.DeepEqual(spec, want) {
|
||||||
t.Fatalf("Spec() = %#v, want %#v", spec, want)
|
t.Fatalf("Spec() = %#v, want %#v", spec, want)
|
||||||
}
|
}
|
||||||
@@ -152,7 +162,7 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
|
|
||||||
t.Run(testCase.name+"/wrong stage rejection", func(t *testing.T) {
|
t.Run(testCase.name+"/wrong stage rejection", func(t *testing.T) {
|
||||||
registry := testCase.newRegistry()
|
registry := testCase.newRegistry()
|
||||||
err := testCase.registerWithSpec(registry, ModuleSpec{Key: testCase.key, Stage: testCase.wrongStage}, testCase.constructor(testCase.key))
|
err := testCase.registerWithSpec(registry, ModuleSpec{Key: testCase.key, Stage: testCase.wrongStage, ExecutionClass: contracts.ExecutionClassDeterministic}, testCase.constructor(testCase.key))
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("RegisterWithSpec() error = nil, want error")
|
t.Fatal("RegisterWithSpec() error = nil, want error")
|
||||||
}
|
}
|
||||||
@@ -161,6 +171,22 @@ func runRegistryBehaviorTests[M any](t *testing.T, testCase registryBehaviorCase
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run(testCase.name+"/execution class rejection", func(t *testing.T) {
|
||||||
|
for _, spec := range []ModuleSpec{
|
||||||
|
{Key: testCase.key, Stage: testCase.stage},
|
||||||
|
{Key: testCase.key, Stage: testCase.stage, ExecutionClass: contracts.ExecutionClass("unsupported")},
|
||||||
|
} {
|
||||||
|
registry := testCase.newRegistry()
|
||||||
|
err := testCase.registerWithSpec(registry, spec, testCase.constructor(testCase.key))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("RegisterWithSpec() error = nil, want execution class error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "execution class") {
|
||||||
|
t.Fatalf("RegisterWithSpec() error = %q, want execution class error", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
t.Run(testCase.name+"/key trimming", func(t *testing.T) {
|
t.Run(testCase.name+"/key trimming", func(t *testing.T) {
|
||||||
registry := testCase.newRegistry()
|
registry := testCase.newRegistry()
|
||||||
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
|
if err := testCase.register(registry, " "+testCase.key+" ", testCase.constructor(testCase.key)); err != nil {
|
||||||
|
|||||||
111
internal/framework/pipeline/evidence_output.go
Normal file
111
internal/framework/pipeline/evidence_output.go
Normal file
@@ -0,0 +1,111 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/evidencecontext"
|
||||||
|
)
|
||||||
|
|
||||||
|
// debugEvidenceContextSummary intentionally contains only publication-safe
|
||||||
|
// identifiers and aggregate counts. The evidence document itself can include
|
||||||
|
// source text and must never be written to this debug envelope.
|
||||||
|
type debugEvidenceContextSummary struct {
|
||||||
|
ArtifactKind contracts.ArtifactKind `json:"artifact_kind"`
|
||||||
|
MediaType string `json:"media_type"`
|
||||||
|
SchemaID string `json:"schema_id"`
|
||||||
|
SchemaName string `json:"schema_name"`
|
||||||
|
SchemaVersion string `json:"schema_version"`
|
||||||
|
SelectedLanes []string `json:"selected_lanes"`
|
||||||
|
WindowUnits int `json:"window_units"`
|
||||||
|
ContextCount int `json:"context_count"`
|
||||||
|
UnitCount int `json:"unit_count"`
|
||||||
|
SourceDigest string `json:"source_digest"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// buildOutputEvidenceContext projects the prepared output policy from accepted
|
||||||
|
// normalized artifacts. It is intentionally separate from lane execution so
|
||||||
|
// checkpointed normalized outputs use the same reconstruction path.
|
||||||
|
func buildOutputEvidenceContext(prepared *PreparedPipeline, doc *source.SourceDocument, outputs []contracts.SerializedOutput) (*contracts.SerializedArtifact, *debugEvidenceContextSummary, error) {
|
||||||
|
if prepared == nil || prepared.evidencePlan == nil {
|
||||||
|
return nil, nil, nil
|
||||||
|
}
|
||||||
|
if doc == nil {
|
||||||
|
return nil, nil, fmt.Errorf("evidence context output: source document is unavailable")
|
||||||
|
}
|
||||||
|
if prepared.artifactCodecs == nil {
|
||||||
|
return nil, nil, fmt.Errorf("evidence context output: artifact codecs are unavailable")
|
||||||
|
}
|
||||||
|
|
||||||
|
byLane := make(map[string]contracts.SerializedOutput, len(outputs))
|
||||||
|
for _, output := range outputs {
|
||||||
|
laneID := strings.TrimSpace(output.LaneID)
|
||||||
|
if _, exists := byLane[laneID]; exists {
|
||||||
|
return nil, nil, fmt.Errorf("evidence context output: accepted normalized outputs contain duplicate lane %q", laneID)
|
||||||
|
}
|
||||||
|
byLane[laneID] = contracts.CloneSerializedOutput(output)
|
||||||
|
}
|
||||||
|
|
||||||
|
request := evidencecontext.BuildRequest{
|
||||||
|
Source: doc,
|
||||||
|
WindowUnits: prepared.evidencePlan.policy.WindowUnits,
|
||||||
|
SelectedLanes: append([]string(nil), prepared.evidencePlan.policy.LaneIDs...),
|
||||||
|
LaneEvidence: make([]evidencecontext.LaneEvidence, 0, len(prepared.evidencePlan.lanes)),
|
||||||
|
}
|
||||||
|
for _, lane := range prepared.evidencePlan.lanes {
|
||||||
|
output, ok := byLane[lane.laneID]
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if output.SourceID != doc.ID {
|
||||||
|
return nil, nil, fmt.Errorf("evidence context output lane %q: accepted normalized output source is incompatible", lane.laneID)
|
||||||
|
}
|
||||||
|
if output.Artifact.Kind != lane.kind {
|
||||||
|
return nil, nil, fmt.Errorf("evidence context output lane %q: accepted normalized output artifact kind is incompatible", lane.laneID)
|
||||||
|
}
|
||||||
|
value, err := prepared.artifactCodecs.Decode(contracts.CloneSerializedArtifact(output.Artifact))
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, fmt.Errorf("evidence context output lane %q: accepted normalized artifact cannot be decoded", lane.laneID)
|
||||||
|
}
|
||||||
|
references, err := lane.project(value)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, fmt.Errorf("evidence context output lane %q: accepted normalized artifact cannot be projected", lane.laneID)
|
||||||
|
}
|
||||||
|
request.LaneEvidence = append(request.LaneEvidence, evidencecontext.LaneEvidence{
|
||||||
|
LaneID: lane.laneID,
|
||||||
|
SourceRefs: append([]source.SourceRef(nil), references...),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
document, err := evidencecontext.Build(request)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, fmt.Errorf("evidence context output: accepted evidence references are invalid")
|
||||||
|
}
|
||||||
|
content, err := evidencecontext.New().Encode(document)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, fmt.Errorf("evidence context output: evidence context serialization failed")
|
||||||
|
}
|
||||||
|
artifact := &contracts.SerializedArtifact{
|
||||||
|
Kind: evidencecontext.ArtifactKind,
|
||||||
|
Schema: evidencecontext.New().Schema(),
|
||||||
|
MediaType: evidencecontext.MediaType,
|
||||||
|
Content: content,
|
||||||
|
}
|
||||||
|
summary := debugEvidenceContextSummary{
|
||||||
|
ArtifactKind: artifact.Kind,
|
||||||
|
MediaType: artifact.MediaType,
|
||||||
|
SchemaID: artifact.Schema.ID,
|
||||||
|
SchemaName: artifact.Schema.Name,
|
||||||
|
SchemaVersion: artifact.Schema.Version,
|
||||||
|
SelectedLanes: append([]string(nil), document.SelectedLanes...),
|
||||||
|
WindowUnits: document.WindowUnits,
|
||||||
|
ContextCount: len(document.Contexts),
|
||||||
|
SourceDigest: document.SourceDigest,
|
||||||
|
}
|
||||||
|
for _, context := range document.Contexts {
|
||||||
|
summary.UnitCount += len(context.Units)
|
||||||
|
}
|
||||||
|
return contracts.CloneSerializedArtifactPointer(artifact), &summary, nil
|
||||||
|
}
|
||||||
252
internal/framework/pipeline/evidence_output_test.go
Normal file
252
internal/framework/pipeline/evidence_output_test.go
Normal file
@@ -0,0 +1,252 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/evidencecontext"
|
||||||
|
)
|
||||||
|
|
||||||
|
type capturingEvidenceOutput struct {
|
||||||
|
requests []contracts.OutputRequest
|
||||||
|
}
|
||||||
|
|
||||||
|
func (*capturingEvidenceOutput) Key() string { return "capture/evidence-context" }
|
||||||
|
|
||||||
|
func (output *capturingEvidenceOutput) Encode(_ context.Context, request contracts.OutputRequest) (contracts.OutputResult, error) {
|
||||||
|
request.EvidenceContext = contracts.CloneSerializedArtifactPointer(request.EvidenceContext)
|
||||||
|
output.requests = append(output.requests, request)
|
||||||
|
return contracts.OutputResult{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func installEvidencePlan(prepared *PreparedPipeline, window int, selected []string, project func(codecNotes) ([]source.SourceRef, error)) {
|
||||||
|
lanes := make([]preparedEvidenceLane, 0, len(selected))
|
||||||
|
for _, laneID := range selected {
|
||||||
|
for _, step := range prepared.Steps {
|
||||||
|
for _, lane := range step.lanes {
|
||||||
|
if lane.resolved.ID != laneID {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
lanes = append(lanes, preparedEvidenceLane{
|
||||||
|
laneID: laneID,
|
||||||
|
kind: lane.resolved.ArtifactKind,
|
||||||
|
project: func(value any) ([]source.SourceRef, error) {
|
||||||
|
notes, ok := value.(codecNotes)
|
||||||
|
if !ok {
|
||||||
|
return nil, errors.New("unexpected artifact type")
|
||||||
|
}
|
||||||
|
return project(notes)
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
prepared.evidencePlan = &preparedEvidencePlan{policy: EvidenceContextPolicy{Enabled: true, WindowUnits: window, LaneIDs: append([]string(nil), selected...)}, lanes: lanes}
|
||||||
|
}
|
||||||
|
|
||||||
|
func setNormalizedNotes(prepared *PreparedPipeline, values map[string]codecNotes) {
|
||||||
|
for stepIndex := range prepared.Steps {
|
||||||
|
for laneIndex := range prepared.Steps[stepIndex].lanes {
|
||||||
|
lane := &prepared.Steps[stepIndex].lanes[laneIndex]
|
||||||
|
value, ok := values[lane.resolved.ID]
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
lane.typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||||
|
return erasedTypedResult{Value: value}, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeCapturedEvidence(t *testing.T, output *capturingEvidenceOutput) evidencecontext.Document {
|
||||||
|
t.Helper()
|
||||||
|
if len(output.requests) != 1 || output.requests[0].EvidenceContext == nil {
|
||||||
|
t.Fatalf("output requests = %#v, want one evidence context", output.requests)
|
||||||
|
}
|
||||||
|
value, err := evidencecontext.New().Decode(output.requests[0].EvidenceContext.Content)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Decode(evidence context): %v", err)
|
||||||
|
}
|
||||||
|
return value
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerBuildsEvidenceContextFromSelectedNormalizedOutputs(t *testing.T) {
|
||||||
|
prepared := preparedOrderedPipeline(t, 3, orderedLaneSpec{id: "alpha", profile: "notes"}, orderedLaneSpec{id: "beta", profile: "notes"})
|
||||||
|
encoder := &capturingEvidenceOutput{}
|
||||||
|
prepared.output = encoder
|
||||||
|
setNormalizedNotes(prepared, map[string]codecNotes{"alpha": {Items: []string{"one"}}, "beta": {Items: []string{"two"}}})
|
||||||
|
installEvidencePlan(prepared, 1, []string{"alpha", "beta", "inactive"}, func(notes codecNotes) ([]source.SourceRef, error) {
|
||||||
|
switch notes.Items[0] {
|
||||||
|
case "one":
|
||||||
|
return []source.SourceRef{{SourceID: "source", StartUnitID: 1, EndUnitID: 1}}, nil
|
||||||
|
case "two":
|
||||||
|
return []source.SourceRef{{SourceID: "source", StartUnitID: 2, EndUnitID: 2}}, nil
|
||||||
|
default:
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
})
|
||||||
|
debug := newCapturedDebugRecorder()
|
||||||
|
if _, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Debug: debug}); err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
value := decodeCapturedEvidence(t, encoder)
|
||||||
|
if !reflect.DeepEqual(value.SelectedLanes, []string{"alpha", "beta", "inactive"}) || len(value.Contexts) != 1 || len(value.Contexts[0].Units) != 3 {
|
||||||
|
t.Fatalf("evidence context = %#v, want selected union", value)
|
||||||
|
}
|
||||||
|
if got := value.Contexts[0].EvidenceRefs; len(got) != 2 || got[0].LaneID != "alpha" || got[1].LaneID != "beta" {
|
||||||
|
t.Fatalf("evidence refs = %#v, want both selected lanes", got)
|
||||||
|
}
|
||||||
|
debugJSON := string(debug.json["output/evidence-context.json"])
|
||||||
|
if strings.Contains(debugJSON, "text-1") || strings.Contains(debugJSON, "metadata") || !strings.Contains(debugJSON, `"artifact_kind":"source/evidence-context"`) || !strings.Contains(debugJSON, `"schema_id":"notarius.source.evidence_context"`) || !strings.Contains(debugJSON, `"context_count":1`) || !strings.Contains(debugJSON, `"unit_count":3`) {
|
||||||
|
t.Fatalf("evidence debug envelope = %s, want only allowlisted summary", debugJSON)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerEvidenceContextOmitsAbsentAndRejectedLanes(t *testing.T) {
|
||||||
|
prepared := preparedOrderedPipeline(t, 2, orderedLaneSpec{id: "present", profile: "notes"}, orderedLaneSpec{id: "rejected", profile: "notes"})
|
||||||
|
encoder := &capturingEvidenceOutput{}
|
||||||
|
prepared.output = encoder
|
||||||
|
setNormalizedNotes(prepared, map[string]codecNotes{"present": {Items: []string{"present"}}, "rejected": {Items: []string{"rejected"}}})
|
||||||
|
prepared.Steps[1].lanes[0].mergeValidators.validators = []preparedValidator{{
|
||||||
|
resolved: ResolvedValidator{Binding: Binding("reject"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||||
|
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||||
|
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted"}, nil
|
||||||
|
},
|
||||||
|
}}
|
||||||
|
installEvidencePlan(prepared, 0, []string{"absent", "present", "rejected"}, func(notes codecNotes) ([]source.SourceRef, error) {
|
||||||
|
if len(notes.Items) > 0 && notes.Items[0] == "present" {
|
||||||
|
return []source.SourceRef{{SourceID: "source", StartUnitID: 1, EndUnitID: 1}}, nil
|
||||||
|
}
|
||||||
|
return []source.SourceRef{{SourceID: "source", StartUnitID: 2, EndUnitID: 2}}, nil
|
||||||
|
})
|
||||||
|
result, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(result.Rejected) != 1 || result.Rejected[0].LaneID != "rejected" {
|
||||||
|
t.Fatalf("rejections = %#v, want rejected lane unchanged", result.Rejected)
|
||||||
|
}
|
||||||
|
value := decodeCapturedEvidence(t, encoder)
|
||||||
|
if len(value.Contexts) != 1 || len(value.Contexts[0].EvidenceRefs) != 1 || value.Contexts[0].EvidenceRefs[0].LaneID != "present" {
|
||||||
|
t.Fatalf("evidence context = %#v, want present lane only", value)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerEvidenceContextFailurePreventsOutputEncoding(t *testing.T) {
|
||||||
|
prepared := preparedConcurrentPipeline(t, 1)
|
||||||
|
encoder := &capturingEvidenceOutput{}
|
||||||
|
prepared.output = encoder
|
||||||
|
setNormalizedNotes(prepared, map[string]codecNotes{"notes": {Items: []string{"invalid"}}})
|
||||||
|
installEvidencePlan(prepared, 0, []string{"notes"}, func(codecNotes) ([]source.SourceRef, error) {
|
||||||
|
return []source.SourceRef{{SourceID: "source", StartUnitID: 99, EndUnitID: 99}}, nil
|
||||||
|
})
|
||||||
|
result, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "accepted evidence references are invalid") {
|
||||||
|
t.Fatalf("Run() error = %v, want evidence context failure", err)
|
||||||
|
}
|
||||||
|
if len(encoder.requests) != 0 || result.Manifest.ValidationStatus != "failed" || len(result.Rejected) != 0 {
|
||||||
|
t.Fatalf("output requests = %#v manifest = %#v rejected = %#v, want failed run before output encoding", encoder.requests, result.Manifest, result.Rejected)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildOutputEvidenceContextRejectsIncompatibleAcceptedOutputs(t *testing.T) {
|
||||||
|
prepared := preparedConcurrentPipeline(t, 1)
|
||||||
|
doc := prepared.input.(*typedTestInput).doc
|
||||||
|
installEvidencePlan(prepared, 0, []string{"notes"}, func(codecNotes) ([]source.SourceRef, error) {
|
||||||
|
return []source.SourceRef{{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 1}}, nil
|
||||||
|
})
|
||||||
|
artifact, err := prepared.artifactCodecs.Encode("test/notes", codecNotes{Items: []string{"valid"}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
valid := contracts.SerializedOutput{LaneID: "notes", SourceID: doc.ID, Artifact: artifact}
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
outputs []contracts.SerializedOutput
|
||||||
|
mutate func(*contracts.SerializedOutput)
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "duplicate lane", outputs: []contracts.SerializedOutput{valid, valid}, want: "duplicate lane"},
|
||||||
|
{name: "foreign source", outputs: []contracts.SerializedOutput{valid}, mutate: func(output *contracts.SerializedOutput) { output.SourceID = "other" }, want: "source is incompatible"},
|
||||||
|
{name: "wrong artifact kind", outputs: []contracts.SerializedOutput{valid}, mutate: func(output *contracts.SerializedOutput) { output.Artifact.Kind = "test/score" }, want: "artifact kind is incompatible"},
|
||||||
|
{name: "invalid artifact payload", outputs: []contracts.SerializedOutput{valid}, mutate: func(output *contracts.SerializedOutput) { output.Artifact.Content = []byte("not JSON") }, want: "cannot be decoded"},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
outputs := append([]contracts.SerializedOutput(nil), test.outputs...)
|
||||||
|
for index := range outputs {
|
||||||
|
outputs[index] = contracts.CloneSerializedOutput(outputs[index])
|
||||||
|
}
|
||||||
|
if test.mutate != nil {
|
||||||
|
test.mutate(&outputs[0])
|
||||||
|
}
|
||||||
|
_, _, err := buildOutputEvidenceContext(prepared, doc, outputs)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||||
|
t.Fatalf("buildOutputEvidenceContext() error = %v, want %q", err, test.want)
|
||||||
|
}
|
||||||
|
if strings.Contains(err.Error(), "not JSON") {
|
||||||
|
t.Fatalf("buildOutputEvidenceContext() exposed artifact payload: %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEvidenceContextOutputRequestOwnsArtifactBytes(t *testing.T) {
|
||||||
|
artifact := &contracts.SerializedArtifact{Content: []byte("original"), Metadata: map[string]any{"source": "original"}}
|
||||||
|
request := contracts.OutputRequest{EvidenceContext: contracts.CloneSerializedArtifactPointer(artifact)}
|
||||||
|
request.EvidenceContext.Content[0] = 'X'
|
||||||
|
request.EvidenceContext.Metadata["source"] = "changed"
|
||||||
|
if string(artifact.Content) != "original" || artifact.Metadata["source"] != "original" {
|
||||||
|
t.Fatalf("output request evidence context aliases source artifact: %#v", artifact)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerEvidenceContextRebuildsFromAcceptedCheckpoint(t *testing.T) {
|
||||||
|
prepared := preparedConcurrentPipeline(t, 1)
|
||||||
|
encoder := &capturingEvidenceOutput{}
|
||||||
|
prepared.output = encoder
|
||||||
|
doc := prepared.input.(*typedTestInput).doc
|
||||||
|
lane := prepared.Steps[0].lanes[0]
|
||||||
|
stored, err := checkpointArtifact(lane.typed.codec, lane.resolved.ID, lane.resolved.Normalize.Module, doc.ID, codecNotes{Items: []string{"stored"}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
loader := newAcceptedCheckpointLoader()
|
||||||
|
key := CheckpointLaneKey(lane.resolved.StepID, lane.resolved.ID)
|
||||||
|
loader.accepted[key] = NormalizeCheckpoint{Output: stored}
|
||||||
|
loader.acceptedDecision[key] = NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused)
|
||||||
|
installEvidencePlan(prepared, 0, []string{"notes"}, func(notes codecNotes) ([]source.SourceRef, error) {
|
||||||
|
if !reflect.DeepEqual(notes.Items, []string{"stored"}) {
|
||||||
|
return nil, errors.New("checkpoint artifact was not projected")
|
||||||
|
}
|
||||||
|
return []source.SourceRef{{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 1}}, nil
|
||||||
|
})
|
||||||
|
policy := CheckpointExecutionPolicy{RequireReusableLanes: map[string]struct{}{key: {}}}
|
||||||
|
if _, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: loader, CheckpointPolicy: policy}); err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
value := decodeCapturedEvidence(t, encoder)
|
||||||
|
if len(value.Contexts) != 1 || value.Contexts[0].EvidenceRefs[0].LaneID != "notes" {
|
||||||
|
t.Fatalf("evidence context = %#v, want checkpointed normalized output", value)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerSkipsEvidenceContextWhenOutputDoesNotOptIn(t *testing.T) {
|
||||||
|
prepared := preparedConcurrentPipeline(t, 1)
|
||||||
|
encoder := &capturingEvidenceOutput{}
|
||||||
|
prepared.output = encoder
|
||||||
|
projected := 0
|
||||||
|
prepared.evidencePlan = nil
|
||||||
|
if _, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")}); err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
if projected != 0 || len(encoder.requests) != 1 || encoder.requests[0].EvidenceContext != nil {
|
||||||
|
t.Fatalf("projected = %d requests = %#v, want no evidence work", projected, encoder.requests)
|
||||||
|
}
|
||||||
|
}
|
||||||
20
internal/framework/pipeline/evidence_policy.go
Normal file
20
internal/framework/pipeline/evidence_policy.go
Normal file
@@ -0,0 +1,20 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
// EvidenceContextPolicy controls optional source-context publication by an
|
||||||
|
// output encoder.
|
||||||
|
type EvidenceContextPolicy struct {
|
||||||
|
Enabled bool
|
||||||
|
WindowUnits int
|
||||||
|
LaneIDs []string
|
||||||
|
}
|
||||||
|
|
||||||
|
// EvidenceContextPolicyProvider is implemented by output encoders that opt in
|
||||||
|
// to evidence-context publication.
|
||||||
|
type EvidenceContextPolicyProvider interface {
|
||||||
|
EvidenceContextPolicy() EvidenceContextPolicy
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneEvidenceContextPolicy(policy EvidenceContextPolicy) EvidenceContextPolicy {
|
||||||
|
policy.LaneIDs = append([]string(nil), policy.LaneIDs...)
|
||||||
|
return policy
|
||||||
|
}
|
||||||
119
internal/framework/pipeline/evidence_preparation_test.go
Normal file
119
internal/framework/pipeline/evidence_preparation_test.go
Normal file
@@ -0,0 +1,119 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
type testEvidenceOutput struct {
|
||||||
|
policy EvidenceContextPolicy
|
||||||
|
}
|
||||||
|
|
||||||
|
func (output testEvidenceOutput) Key() string { return "output" }
|
||||||
|
func (output testEvidenceOutput) Encode(context.Context, contracts.OutputRequest) (contracts.OutputResult, error) {
|
||||||
|
return contracts.OutputResult{}, nil
|
||||||
|
}
|
||||||
|
func (output testEvidenceOutput) EvidenceContextPolicy() EvidenceContextPolicy {
|
||||||
|
return cloneEvidenceContextPolicy(output.policy)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEvidencePolicyResolutionAndPreparationSurviveLaneFiltering(t *testing.T) {
|
||||||
|
registries, _ := constructionRegistries(t, nil, nil)
|
||||||
|
registries.ArtifactEvidence = NewArtifactEvidenceRegistry()
|
||||||
|
policy := EvidenceContextPolicy{Enabled: true, WindowUnits: 2, LaneIDs: []string{"artifact", "inactive"}}
|
||||||
|
profileValidated := false
|
||||||
|
registerTestEvidenceOutputWithProfileValidation(t, ®istries, policy, func(context OutputProfileOptionContext, _ map[string]any) error {
|
||||||
|
profileValidated = true
|
||||||
|
if want := []string{"artifact", "inactive"}; !reflect.DeepEqual(context.LaneIDs, want) {
|
||||||
|
return fmt.Errorf("configured lane ids = %#v, want %#v", context.LaneIDs, want)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err := RegisterArtifactEvidence(registries.ArtifactEvidence, "test/notes", func(codecNotes) []source.SourceRef { return nil }); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
profile := constructionProfile()
|
||||||
|
profile.Artifacts["inactive"] = profile.Artifacts["artifact"]
|
||||||
|
profile.Output.Options = map[string]any{"known": true}
|
||||||
|
resolved, err := ResolvePipeline(profile, ResolveOptions{Only: []string{"artifact"}}, registries.catalog())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !profileValidated {
|
||||||
|
t.Fatal("output profile options were not validated")
|
||||||
|
}
|
||||||
|
if lanes := resolved.AllArtifactLanes(); len(lanes) != 1 || lanes[0].ID != "artifact" {
|
||||||
|
t.Fatalf("resolved lanes = %#v, want only the invocation-selected lane", lanes)
|
||||||
|
}
|
||||||
|
prepared, err := Prepare(resolved, registries, ModuleDependencies{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Prepare() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if prepared.evidencePlan == nil || !reflect.DeepEqual(prepared.evidencePlan.policy.LaneIDs, []string{"artifact", "inactive"}) {
|
||||||
|
t.Fatalf("evidence plan = %#v, want complete configured policy", prepared.evidencePlan)
|
||||||
|
}
|
||||||
|
if got := prepared.evidencePlan.lanes; len(got) != 1 || got[0].laneID != "artifact" {
|
||||||
|
t.Fatalf("active evidence lanes = %#v, want artifact only", got)
|
||||||
|
}
|
||||||
|
prepared.evidencePlan.policy.LaneIDs[0] = "mutated"
|
||||||
|
if policy := prepared.output.(EvidenceContextPolicyProvider).EvidenceContextPolicy(); policy.LaneIDs[0] != "artifact" {
|
||||||
|
t.Fatalf("prepared plan mutated provider policy: %#v", policy)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPrepareEvidencePlanRejectsMissingAndMismatchedCapabilities(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
configure func(*Registries)
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "missing registry", configure: func(registries *Registries) { registries.ArtifactEvidence = nil }, want: "artifact evidence registry"},
|
||||||
|
{name: "unsupported kind", configure: func(registries *Registries) {}, want: "artifact evidence \"test/notes\" is not registered"},
|
||||||
|
{name: "mismatched type", configure: func(registries *Registries) {
|
||||||
|
if err := RegisterArtifactEvidence(registries.ArtifactEvidence, "test/notes", func(codecScore) []source.SourceRef { return nil }); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}, want: "requires Go type"},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
registries, _ := constructionRegistries(t, nil, nil)
|
||||||
|
registries.ArtifactEvidence = NewArtifactEvidenceRegistry()
|
||||||
|
registerTestEvidenceOutput(t, ®istries, EvidenceContextPolicy{Enabled: true, LaneIDs: []string{"artifact"}})
|
||||||
|
test.configure(®istries)
|
||||||
|
profile := constructionProfile()
|
||||||
|
profile.Output.Options = map[string]any{"known": true}
|
||||||
|
resolved, err := ResolvePipeline(profile, ResolveOptions{}, registries.catalog())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
_, err = Prepare(resolved, registries, ModuleDependencies{})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||||
|
t.Fatalf("Prepare() error = %v, want %q", err, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func registerTestEvidenceOutput(t *testing.T, registries *Registries, policy EvidenceContextPolicy) {
|
||||||
|
t.Helper()
|
||||||
|
registerTestEvidenceOutputWithProfileValidation(t, registries, policy, nil)
|
||||||
|
}
|
||||||
|
|
||||||
|
func registerTestEvidenceOutputWithProfileValidation(t *testing.T, registries *Registries, policy EvidenceContextPolicy, validateProfile OutputProfileOptionValidator) {
|
||||||
|
t.Helper()
|
||||||
|
registry := NewOutputEncoderRegistry()
|
||||||
|
if err := registry.RegisterBuilderWithProfileValidation(testModuleSpec("output", StageOutput), func(options map[string]any) error {
|
||||||
|
return RejectUnknownOptions(options, "known")
|
||||||
|
}, validateProfile, func(BuildRequest) (contracts.OutputEncoder, error) {
|
||||||
|
return testEvidenceOutput{policy: cloneEvidenceContextPolicy(policy)}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
registries.Outputs = registry
|
||||||
|
}
|
||||||
@@ -24,10 +24,6 @@ func NewInputAdapterRegistry() *InputAdapterRegistry {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *InputAdapterRegistry) Register(key string, constructor InputAdapterConstructor) error {
|
|
||||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageInput), constructor)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor InputAdapterConstructor) error {
|
func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor InputAdapterConstructor) error {
|
||||||
if constructor == nil {
|
if constructor == nil {
|
||||||
return fmt.Errorf("input adapter constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
return fmt.Errorf("input adapter constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
|
|||||||
@@ -11,10 +11,14 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
func registerTestInput(registry *InputAdapterRegistry, key string, constructor InputAdapterConstructor) error {
|
||||||
|
return registry.RegisterWithSpec(testModuleSpec(key, StageInput), constructor)
|
||||||
|
}
|
||||||
|
|
||||||
func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) {
|
func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -30,7 +34,7 @@ func TestInputAdapterRegistryRegisterAndBuild(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisterAndBuildTrimKeys(t *testing.T) {
|
func TestInputAdapterRegistryRegisterAndBuildTrimKeys(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
if err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
if err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -48,6 +52,7 @@ func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
|||||||
spec := ModuleSpec{
|
spec := ModuleSpec{
|
||||||
Key: " generic-input ",
|
Key: " generic-input ",
|
||||||
Stage: StageInput,
|
Stage: StageInput,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{" parsed-source ", "source-document", "parsed-source", ""},
|
Provides: []string{" parsed-source ", "source-document", "parsed-source", ""},
|
||||||
Requires: []string{" raw-bytes ", "raw-bytes", ""},
|
Requires: []string{" raw-bytes ", "raw-bytes", ""},
|
||||||
}
|
}
|
||||||
@@ -63,6 +68,7 @@ func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
|||||||
want := ModuleSpec{
|
want := ModuleSpec{
|
||||||
Key: "generic-input",
|
Key: "generic-input",
|
||||||
Stage: StageInput,
|
Stage: StageInput,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Provides: []string{"parsed-source", "source-document"},
|
Provides: []string{"parsed-source", "source-document"},
|
||||||
Requires: []string{"raw-bytes"},
|
Requires: []string{"raw-bytes"},
|
||||||
}
|
}
|
||||||
@@ -80,10 +86,10 @@ func TestInputAdapterRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
|
func TestInputAdapterRegistryRegisterWithSpecStoresMinimalMetadata(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
if err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
if err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -91,7 +97,7 @@ func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
|
|||||||
if !ok {
|
if !ok {
|
||||||
t.Fatal("Spec() ok = false, want true")
|
t.Fatal("Spec() ok = false, want true")
|
||||||
}
|
}
|
||||||
want := ModuleSpec{Key: "generic-input", Stage: StageInput}
|
want := ModuleSpec{Key: "generic-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
if !reflect.DeepEqual(got, want) {
|
if !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||||
}
|
}
|
||||||
@@ -100,7 +106,7 @@ func TestInputAdapterRegistryRegisterStoresDefaultSpec(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisterWithSpecRejectsWrongStage(t *testing.T) {
|
func TestInputAdapterRegistryRegisterWithSpecRejectsWrongStage(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
err := registry.RegisterWithSpec(ModuleSpec{Key: "generic-input", Stage: StageExtract}, fakeInputAdapterConstructor("generic-input"))
|
err := registry.RegisterWithSpec(ModuleSpec{Key: "generic-input", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic}, fakeInputAdapterConstructor("generic-input"))
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("RegisterWithSpec() error = nil, want error")
|
t.Fatal("RegisterWithSpec() error = nil, want error")
|
||||||
@@ -121,7 +127,7 @@ func TestInputAdapterRegistrySpecRejectsUnknownKey(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) {
|
func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
err := registry.Register(" \t", fakeInputAdapterConstructor("generic-input"))
|
err := registerTestInput(registry, " \t", fakeInputAdapterConstructor("generic-input"))
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("Register() error = nil, want error")
|
t.Fatal("Register() error = nil, want error")
|
||||||
@@ -133,11 +139,11 @@ func TestInputAdapterRegistryRegisterRejectsEmptyKey(t *testing.T) {
|
|||||||
|
|
||||||
func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) {
|
func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
err := registry.Register(" generic-input ", fakeInputAdapterConstructor("generic-input"))
|
err := registerTestInput(registry, " generic-input ", fakeInputAdapterConstructor("generic-input"))
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("Register() error = nil, want error")
|
t.Fatal("Register() error = nil, want error")
|
||||||
@@ -150,7 +156,7 @@ func TestInputAdapterRegistryRegisterRejectsDuplicateKey(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisterRejectsNilConstructor(t *testing.T) {
|
func TestInputAdapterRegistryRegisterRejectsNilConstructor(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
|
|
||||||
err := registry.Register("generic-input", nil)
|
err := registerTestInput(registry, "generic-input", nil)
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("Register() error = nil, want error")
|
t.Fatal("Register() error = nil, want error")
|
||||||
@@ -176,7 +182,7 @@ func TestInputAdapterRegistryBuildRejectsUnknownKey(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) {
|
func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
constructorErr := errors.New("constructor failed")
|
constructorErr := errors.New("constructor failed")
|
||||||
if err := registry.Register("generic-input", func() (contracts.InputAdapter, error) {
|
if err := registerTestInput(registry, "generic-input", func() (contracts.InputAdapter, error) {
|
||||||
return nil, constructorErr
|
return nil, constructorErr
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
@@ -197,7 +203,7 @@ func TestInputAdapterRegistryBuildWrapsConstructorError(t *testing.T) {
|
|||||||
|
|
||||||
func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) {
|
func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
if err := registry.Register("generic-input", func() (contracts.InputAdapter, error) {
|
if err := registerTestInput(registry, "generic-input", func() (contracts.InputAdapter, error) {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
@@ -215,7 +221,7 @@ func TestInputAdapterRegistryBuildRejectsNilAdapter(t *testing.T) {
|
|||||||
|
|
||||||
func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) {
|
func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("other-input")); err != nil {
|
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("other-input")); err != nil {
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -232,7 +238,7 @@ func TestInputAdapterRegistryBuildRejectsAdapterKeyMismatch(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
|
func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
|
||||||
registry := NewInputAdapterRegistry()
|
registry := NewInputAdapterRegistry()
|
||||||
for _, key := range []string{"zeta", "alpha", "middle"} {
|
for _, key := range []string{"zeta", "alpha", "middle"} {
|
||||||
if err := registry.Register(key, fakeInputAdapterConstructor(key)); err != nil {
|
if err := registerTestInput(registry, key, fakeInputAdapterConstructor(key)); err != nil {
|
||||||
t.Fatalf("Register(%q) error = %v, want nil", key, err)
|
t.Fatalf("Register(%q) error = %v, want nil", key, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -253,7 +259,7 @@ func TestInputAdapterRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
|
|||||||
func TestInputAdapterRegistryNilRegistryBehavior(t *testing.T) {
|
func TestInputAdapterRegistryNilRegistryBehavior(t *testing.T) {
|
||||||
var registry *InputAdapterRegistry
|
var registry *InputAdapterRegistry
|
||||||
|
|
||||||
if err := registry.Register("generic-input", fakeInputAdapterConstructor("generic-input")); err == nil {
|
if err := registerTestInput(registry, "generic-input", fakeInputAdapterConstructor("generic-input")); err == nil {
|
||||||
t.Fatal("Register() error = nil, want error")
|
t.Fatal("Register() error = nil, want error")
|
||||||
}
|
}
|
||||||
if _, err := registry.Build("generic-input"); err == nil {
|
if _, err := registry.Build("generic-input"); err == nil {
|
||||||
|
|||||||
@@ -23,23 +23,19 @@ const (
|
|||||||
type ModuleSpec struct {
|
type ModuleSpec struct {
|
||||||
Key string
|
Key string
|
||||||
Stage ModuleStage
|
Stage ModuleStage
|
||||||
|
ExecutionClass contracts.ExecutionClass
|
||||||
ArtifactKind contracts.ArtifactKind
|
ArtifactKind contracts.ArtifactKind
|
||||||
Provides []string
|
Provides []string
|
||||||
Requires []string
|
Requires []string
|
||||||
ReferenceSlots []contracts.ReferenceSlot
|
ReferenceSlots []contracts.ReferenceSlot
|
||||||
}
|
}
|
||||||
|
|
||||||
func defaultModuleSpec(key string, stage ModuleStage) ModuleSpec {
|
|
||||||
return ModuleSpec{
|
|
||||||
Key: key,
|
|
||||||
Stage: stage,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func normalizeModuleSpec(spec ModuleSpec) ModuleSpec {
|
func normalizeModuleSpec(spec ModuleSpec) ModuleSpec {
|
||||||
|
executionClass := contracts.ExecutionClass(strings.TrimSpace(string(spec.ExecutionClass)))
|
||||||
return ModuleSpec{
|
return ModuleSpec{
|
||||||
Key: strings.TrimSpace(spec.Key),
|
Key: strings.TrimSpace(spec.Key),
|
||||||
Stage: spec.Stage,
|
Stage: spec.Stage,
|
||||||
|
ExecutionClass: executionClass,
|
||||||
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
|
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
|
||||||
Provides: normalizeCapabilities(spec.Provides),
|
Provides: normalizeCapabilities(spec.Provides),
|
||||||
Requires: normalizeCapabilities(spec.Requires),
|
Requires: normalizeCapabilities(spec.Requires),
|
||||||
@@ -76,6 +72,7 @@ func cloneModuleSpec(spec ModuleSpec) ModuleSpec {
|
|||||||
return ModuleSpec{
|
return ModuleSpec{
|
||||||
Key: spec.Key,
|
Key: spec.Key,
|
||||||
Stage: spec.Stage,
|
Stage: spec.Stage,
|
||||||
|
ExecutionClass: spec.ExecutionClass,
|
||||||
ArtifactKind: spec.ArtifactKind,
|
ArtifactKind: spec.ArtifactKind,
|
||||||
Provides: append([]string(nil), spec.Provides...),
|
Provides: append([]string(nil), spec.Provides...),
|
||||||
Requires: append([]string(nil), spec.Requires...),
|
Requires: append([]string(nil), spec.Requires...),
|
||||||
@@ -90,6 +87,12 @@ func validateModuleSpec(kind string, expectedStage ModuleStage, spec ModuleSpec)
|
|||||||
if spec.Stage != expectedStage {
|
if spec.Stage != expectedStage {
|
||||||
return fmt.Errorf("%s %q must use %q stage, got %q", kind, spec.Key, expectedStage, spec.Stage)
|
return fmt.Errorf("%s %q must use %q stage, got %q", kind, spec.Key, expectedStage, spec.Stage)
|
||||||
}
|
}
|
||||||
|
if spec.ExecutionClass == "" {
|
||||||
|
return fmt.Errorf("%s %q execution class must not be empty", kind, spec.Key)
|
||||||
|
}
|
||||||
|
if spec.ExecutionClass != contracts.ExecutionClassDeterministic && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
|
||||||
|
return fmt.Errorf("%s %q has unsupported execution class %q", kind, spec.Key, spec.ExecutionClass)
|
||||||
|
}
|
||||||
if spec.ArtifactKind != "" && spec.Stage != StageExtract && spec.Stage != StageMerge && spec.Stage != StageNormalize {
|
if spec.ArtifactKind != "" && spec.Stage != StageExtract && spec.Stage != StageMerge && spec.Stage != StageNormalize {
|
||||||
return fmt.Errorf("%s %q must not declare an artifact kind", kind, spec.Key)
|
return fmt.Errorf("%s %q must not declare an artifact kind", kind, spec.Key)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,12 +1,45 @@
|
|||||||
package pipeline
|
package pipeline
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
func TestValidateModuleSpecRequiresSupportedExecutionClass(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
class contracts.ExecutionClass
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "missing", want: "execution class"},
|
||||||
|
{name: "unsupported", class: "remote", want: "unsupported"},
|
||||||
|
{name: "deterministic", class: contracts.ExecutionClassDeterministic},
|
||||||
|
{name: "llm backed", class: contracts.ExecutionClassLLMBacked},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
spec := normalizeModuleSpec(ModuleSpec{Key: "module", Stage: StageChunk, ExecutionClass: test.class})
|
||||||
|
err := validateModuleSpec("chunker", StageChunk, spec)
|
||||||
|
if test.want == "" && err != nil {
|
||||||
|
t.Fatalf("validateModuleSpec() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if test.want != "" && (err == nil || !strings.Contains(err.Error(), test.want)) {
|
||||||
|
t.Fatalf("validateModuleSpec() error = %v, want %q", err, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCloneModuleSpecPreservesExecutionClass(t *testing.T) {
|
||||||
|
spec := normalizeModuleSpec(ModuleSpec{Key: " module ", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked})
|
||||||
|
cloned := cloneModuleSpec(spec)
|
||||||
|
if !reflect.DeepEqual(cloned, spec) {
|
||||||
|
t.Fatalf("cloneModuleSpec() = %#v, want %#v", cloned, spec)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
|
func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -24,6 +57,7 @@ func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
|
|||||||
spec := normalizeModuleSpec(ModuleSpec{
|
spec := normalizeModuleSpec(ModuleSpec{
|
||||||
Key: "module",
|
Key: "module",
|
||||||
Stage: test.stage,
|
Stage: test.stage,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster", Description: "Character roster", MaxBytes: 1024},
|
{Name: "roster", Description: "Character roster", MaxBytes: 1024},
|
||||||
},
|
},
|
||||||
@@ -52,6 +86,7 @@ func TestValidateModuleSpecRejectsReferenceSlotsForIneligibleStages(t *testing.T
|
|||||||
spec := normalizeModuleSpec(ModuleSpec{
|
spec := normalizeModuleSpec(ModuleSpec{
|
||||||
Key: "module",
|
Key: "module",
|
||||||
Stage: test.stage,
|
Stage: test.stage,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster"},
|
{Name: "roster"},
|
||||||
},
|
},
|
||||||
@@ -109,6 +144,7 @@ func TestValidateModuleSpecRejectsInvalidReferenceSlotsForEligibleStages(t *test
|
|||||||
spec := normalizeModuleSpec(ModuleSpec{
|
spec := normalizeModuleSpec(ModuleSpec{
|
||||||
Key: "module",
|
Key: "module",
|
||||||
Stage: stage.stage,
|
Stage: stage.stage,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
ReferenceSlots: invalid.slots,
|
ReferenceSlots: invalid.slots,
|
||||||
})
|
})
|
||||||
err := validateModuleSpec(stage.kind, stage.stage, spec)
|
err := validateModuleSpec(stage.kind, stage.stage, spec)
|
||||||
|
|||||||
@@ -26,7 +26,7 @@ func TestNormalizerRegistryErasureClonesRetryDirective(t *testing.T) {
|
|||||||
FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "omitted", Message: "fallback warning"}},
|
FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "omitted", Message: "fallback warning"}},
|
||||||
}
|
}
|
||||||
registry := NewNormalizerRegistry()
|
registry := NewNormalizerRegistry()
|
||||||
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
||||||
return retryingNotesNormalizer{warnings: warnings, retry: retry}, nil
|
return retryingNotesNormalizer{warnings: warnings, retry: retry}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterNormalizer() error = %v", err)
|
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||||
|
|||||||
@@ -2,7 +2,7 @@ package pipeline
|
|||||||
|
|
||||||
import "fmt"
|
import "fmt"
|
||||||
|
|
||||||
func validateResolvedOptions(resolved ResolvedPipeline, catalog ModuleCatalog) error {
|
func validateResolvedOptions(resolved ResolvedPipeline, catalog ModuleCatalog, configuredLaneIDs []string) error {
|
||||||
if err := catalog.Inputs.ValidateOptions(resolved.Input.Module, resolved.Input.Options); err != nil {
|
if err := catalog.Inputs.ValidateOptions(resolved.Input.Module, resolved.Input.Options); err != nil {
|
||||||
return moduleOptionsError(resolved.ID, "", StageInput, resolved.Input.Module, err)
|
return moduleOptionsError(resolved.ID, "", StageInput, resolved.Input.Module, err)
|
||||||
}
|
}
|
||||||
@@ -42,6 +42,9 @@ func validateResolvedOptions(resolved ResolvedPipeline, catalog ModuleCatalog) e
|
|||||||
if err := catalog.Outputs.ValidateOptions(resolved.Output.Module, resolved.Output.Options); err != nil {
|
if err := catalog.Outputs.ValidateOptions(resolved.Output.Module, resolved.Output.Options); err != nil {
|
||||||
return moduleOptionsError(resolved.ID, "", StageOutput, resolved.Output.Module, err)
|
return moduleOptionsError(resolved.ID, "", StageOutput, resolved.Output.Module, err)
|
||||||
}
|
}
|
||||||
|
if err := catalog.Outputs.ValidateProfileOptions(resolved.Output.Module, OutputProfileOptionContext{LaneIDs: configuredLaneIDs}, resolved.Output.Options); err != nil {
|
||||||
|
return moduleOptionsError(resolved.ID, "", StageOutput, resolved.Output.Module, err)
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -10,9 +10,16 @@ import (
|
|||||||
type OutputEncoderConstructor func() (contracts.OutputEncoder, error)
|
type OutputEncoderConstructor func() (contracts.OutputEncoder, error)
|
||||||
type OutputEncoderBuilder func(BuildRequest) (contracts.OutputEncoder, error)
|
type OutputEncoderBuilder func(BuildRequest) (contracts.OutputEncoder, error)
|
||||||
|
|
||||||
|
type OutputProfileOptionContext struct {
|
||||||
|
LaneIDs []string
|
||||||
|
}
|
||||||
|
|
||||||
|
type OutputProfileOptionValidator func(OutputProfileOptionContext, map[string]any) error
|
||||||
|
|
||||||
type OutputEncoderRegistry struct {
|
type OutputEncoderRegistry struct {
|
||||||
builders map[string]OutputEncoderBuilder
|
builders map[string]OutputEncoderBuilder
|
||||||
optionValidators map[string]OptionValidator
|
optionValidators map[string]OptionValidator
|
||||||
|
profileValidators map[string]OutputProfileOptionValidator
|
||||||
specs map[string]ModuleSpec
|
specs map[string]ModuleSpec
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -20,14 +27,11 @@ func NewOutputEncoderRegistry() *OutputEncoderRegistry {
|
|||||||
return &OutputEncoderRegistry{
|
return &OutputEncoderRegistry{
|
||||||
builders: make(map[string]OutputEncoderBuilder),
|
builders: make(map[string]OutputEncoderBuilder),
|
||||||
optionValidators: make(map[string]OptionValidator),
|
optionValidators: make(map[string]OptionValidator),
|
||||||
|
profileValidators: make(map[string]OutputProfileOptionValidator),
|
||||||
specs: make(map[string]ModuleSpec),
|
specs: make(map[string]ModuleSpec),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *OutputEncoderRegistry) Register(key string, constructor OutputEncoderConstructor) error {
|
|
||||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageOutput), constructor)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor OutputEncoderConstructor) error {
|
func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor OutputEncoderConstructor) error {
|
||||||
if constructor == nil {
|
if constructor == nil {
|
||||||
return fmt.Errorf("output encoder constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
return fmt.Errorf("output encoder constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
@@ -38,6 +42,12 @@ func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor Ou
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (r *OutputEncoderRegistry) RegisterBuilderWithSpec(spec ModuleSpec, validateOptions OptionValidator, builder OutputEncoderBuilder) error {
|
func (r *OutputEncoderRegistry) RegisterBuilderWithSpec(spec ModuleSpec, validateOptions OptionValidator, builder OutputEncoderBuilder) error {
|
||||||
|
return r.RegisterBuilderWithProfileValidation(spec, validateOptions, nil, builder)
|
||||||
|
}
|
||||||
|
|
||||||
|
// RegisterBuilderWithProfileValidation registers an output builder with an
|
||||||
|
// optional validator that can inspect all configured lane identities.
|
||||||
|
func (r *OutputEncoderRegistry) RegisterBuilderWithProfileValidation(spec ModuleSpec, validateOptions OptionValidator, validateProfile OutputProfileOptionValidator, builder OutputEncoderBuilder) error {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return fmt.Errorf("output encoder registry must not be nil")
|
return fmt.Errorf("output encoder registry must not be nil")
|
||||||
}
|
}
|
||||||
@@ -62,11 +72,15 @@ func (r *OutputEncoderRegistry) RegisterBuilderWithSpec(spec ModuleSpec, validat
|
|||||||
if r.optionValidators == nil {
|
if r.optionValidators == nil {
|
||||||
r.optionValidators = make(map[string]OptionValidator)
|
r.optionValidators = make(map[string]OptionValidator)
|
||||||
}
|
}
|
||||||
|
if r.profileValidators == nil {
|
||||||
|
r.profileValidators = make(map[string]OutputProfileOptionValidator)
|
||||||
|
}
|
||||||
if r.specs == nil {
|
if r.specs == nil {
|
||||||
r.specs = make(map[string]ModuleSpec)
|
r.specs = make(map[string]ModuleSpec)
|
||||||
}
|
}
|
||||||
r.builders[normalizedSpec.Key] = builder
|
r.builders[normalizedSpec.Key] = builder
|
||||||
r.optionValidators[normalizedSpec.Key] = validateOptions
|
r.optionValidators[normalizedSpec.Key] = validateOptions
|
||||||
|
r.profileValidators[normalizedSpec.Key] = validateProfile
|
||||||
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -116,6 +130,21 @@ func (r *OutputEncoderRegistry) ValidateOptions(key string, options map[string]a
|
|||||||
return validateRegisteredOptions(validator, options)
|
return validateRegisteredOptions(validator, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (r *OutputEncoderRegistry) ValidateProfileOptions(key string, context OutputProfileOptionContext, options map[string]any) error {
|
||||||
|
if r == nil {
|
||||||
|
return fmt.Errorf("output encoder registry must not be nil")
|
||||||
|
}
|
||||||
|
normalizedKey := strings.TrimSpace(key)
|
||||||
|
validator, ok := r.profileValidators[normalizedKey]
|
||||||
|
if !ok {
|
||||||
|
return fmt.Errorf("output encoder %q is not registered", normalizedKey)
|
||||||
|
}
|
||||||
|
if validator == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return validator(OutputProfileOptionContext{LaneIDs: append([]string(nil), context.LaneIDs...)}, cloneOptions(options))
|
||||||
|
}
|
||||||
|
|
||||||
func (r *OutputEncoderRegistry) Spec(key string) (ModuleSpec, bool) {
|
func (r *OutputEncoderRegistry) Spec(key string) (ModuleSpec, bool) {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return ModuleSpec{}, false
|
return ModuleSpec{}, false
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package pipeline
|
package pipeline
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"reflect"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
@@ -16,7 +17,9 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
|
|||||||
return NewOutputEncoderRegistry()
|
return NewOutputEncoderRegistry()
|
||||||
},
|
},
|
||||||
register: func(registry any, key string, constructor func() (contracts.OutputEncoder, error)) error {
|
register: func(registry any, key string, constructor func() (contracts.OutputEncoder, error)) error {
|
||||||
return registry.(*OutputEncoderRegistry).Register(key, constructor)
|
return registry.(*OutputEncoderRegistry).RegisterWithSpec(ModuleSpec{
|
||||||
|
Key: key, Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
}, constructor)
|
||||||
},
|
},
|
||||||
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.OutputEncoder, error)) error {
|
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.OutputEncoder, error)) error {
|
||||||
return registry.(*OutputEncoderRegistry).RegisterWithSpec(spec, constructor)
|
return registry.(*OutputEncoderRegistry).RegisterWithSpec(spec, constructor)
|
||||||
@@ -32,7 +35,9 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
|
|||||||
},
|
},
|
||||||
nilRegister: func(key string, constructor func() (contracts.OutputEncoder, error)) error {
|
nilRegister: func(key string, constructor func() (contracts.OutputEncoder, error)) error {
|
||||||
var registry *OutputEncoderRegistry
|
var registry *OutputEncoderRegistry
|
||||||
return registry.Register(key, constructor)
|
return registry.RegisterWithSpec(ModuleSpec{
|
||||||
|
Key: key, Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
|
}, constructor)
|
||||||
},
|
},
|
||||||
nilBuild: func(key string) (contracts.OutputEncoder, error) {
|
nilBuild: func(key string) (contracts.OutputEncoder, error) {
|
||||||
var registry *OutputEncoderRegistry
|
var registry *OutputEncoderRegistry
|
||||||
@@ -56,3 +61,33 @@ func TestOutputEncoderRegistryBehavior(t *testing.T) {
|
|||||||
},
|
},
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestOutputProfileValidationReceivesOwnedOptionsAndLaneIDs(t *testing.T) {
|
||||||
|
registry := NewOutputEncoderRegistry()
|
||||||
|
if err := registry.RegisterBuilderWithProfileValidation(testModuleSpec("profile-output", StageOutput), func(options map[string]any) error {
|
||||||
|
options["nested"].(map[string]any)["value"] = "changed"
|
||||||
|
return nil
|
||||||
|
}, func(context OutputProfileOptionContext, options map[string]any) error {
|
||||||
|
context.LaneIDs[0] = "changed"
|
||||||
|
options["nested"].(map[string]any)["value"] = "changed-again"
|
||||||
|
return nil
|
||||||
|
}, func(BuildRequest) (contracts.OutputEncoder, error) {
|
||||||
|
return registryOutputEncoder{key: "profile-output"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
options := map[string]any{"nested": map[string]any{"value": "original"}}
|
||||||
|
if err := registry.ValidateOptions("profile-output", options); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
context := OutputProfileOptionContext{LaneIDs: []string{"artifact"}}
|
||||||
|
if err := registry.ValidateProfileOptions("profile-output", context, options); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got := options["nested"].(map[string]any)["value"]; got != "original" {
|
||||||
|
t.Fatalf("options mutated by validator: %q", got)
|
||||||
|
}
|
||||||
|
if want := []string{"artifact"}; !reflect.DeepEqual(context.LaneIDs, want) {
|
||||||
|
t.Fatalf("lane ids mutated by validator: %#v, want %#v", context.LaneIDs, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -480,19 +480,19 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
|||||||
if err := RegisterArtifactCodec(registries.ArtifactCodecs, notesCodec()); err != nil {
|
if err := RegisterArtifactCodec(registries.ArtifactCodecs, notesCodec()); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := registries.Inputs.RegisterBuilderWithSpec(defaultModuleSpec("input", StageInput), strict, func(request BuildRequest) (contracts.InputAdapter, error) {
|
if err := registries.Inputs.RegisterBuilderWithSpec(testModuleSpec("input", StageInput), strict, func(request BuildRequest) (contracts.InputAdapter, error) {
|
||||||
record("input", &request)
|
record("input", &request)
|
||||||
return input, nil
|
return input, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := registries.Chunkers.RegisterBuilderWithSpec(defaultModuleSpec("chunk", StageChunk), strict, func(request BuildRequest) (contracts.Chunker, error) {
|
if err := registries.Chunkers.RegisterBuilderWithSpec(testModuleSpec("chunk", StageChunk), strict, func(request BuildRequest) (contracts.Chunker, error) {
|
||||||
record("chunk", &request)
|
record("chunk", &request)
|
||||||
return &typedTestChunker{key: "chunk"}, nil
|
return &typedTestChunker{key: "chunk"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
extractSpec := defaultModuleSpec("extract", StageExtract)
|
extractSpec := testModuleSpec("extract", StageExtract)
|
||||||
extractSpec.ArtifactKind = "test/notes"
|
extractSpec.ArtifactKind = "test/notes"
|
||||||
if err := RegisterExtractorBuilder(registries.Extractors, extractSpec, strict, func(request BuildRequest) (contracts.Extractor[codecNotes], error) {
|
if err := RegisterExtractorBuilder(registries.Extractors, extractSpec, strict, func(request BuildRequest) (contracts.Extractor[codecNotes], error) {
|
||||||
record("extract", &request)
|
record("extract", &request)
|
||||||
@@ -503,7 +503,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
mergeSpec := defaultModuleSpec("merge", StageMerge)
|
mergeSpec := testModuleSpec("merge", StageMerge)
|
||||||
mergeSpec.ArtifactKind = "test/notes"
|
mergeSpec.ArtifactKind = "test/notes"
|
||||||
if err := RegisterMergerBuilder(registries.Mergers, mergeSpec, strict, func(request BuildRequest) (contracts.Merger[codecNotes], error) {
|
if err := RegisterMergerBuilder(registries.Mergers, mergeSpec, strict, func(request BuildRequest) (contracts.Merger[codecNotes], error) {
|
||||||
record("merge", &request)
|
record("merge", &request)
|
||||||
@@ -511,7 +511,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
normalizeSpec := defaultModuleSpec("normalize", StageNormalize)
|
normalizeSpec := testModuleSpec("normalize", StageNormalize)
|
||||||
normalizeSpec.ArtifactKind = "test/notes"
|
normalizeSpec.ArtifactKind = "test/notes"
|
||||||
if err := RegisterNormalizerBuilder(registries.Normalizers, normalizeSpec, strict, func(request BuildRequest) (contracts.Normalizer[codecNotes], error) {
|
if err := RegisterNormalizerBuilder(registries.Normalizers, normalizeSpec, strict, func(request BuildRequest) (contracts.Normalizer[codecNotes], error) {
|
||||||
record("normalize", &request)
|
record("normalize", &request)
|
||||||
@@ -532,7 +532,7 @@ func constructionRegistriesWithHooks(t *testing.T, built *[]string, failure *con
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := registries.Outputs.RegisterBuilderWithSpec(defaultModuleSpec("output", StageOutput), strict, func(request BuildRequest) (contracts.OutputEncoder, error) {
|
if err := registries.Outputs.RegisterBuilderWithSpec(testModuleSpec("output", StageOutput), strict, func(request BuildRequest) (contracts.OutputEncoder, error) {
|
||||||
record("output", &request)
|
record("output", &request)
|
||||||
if failure.output != nil {
|
if failure.output != nil {
|
||||||
return nil, failure.output
|
return nil, failure.output
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"reflect"
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -24,6 +25,7 @@ type PreparedPipeline struct {
|
|||||||
chunkValidators preparedValidatorChain
|
chunkValidators preparedValidatorChain
|
||||||
output contracts.OutputEncoder
|
output contracts.OutputEncoder
|
||||||
artifactCodecs *ArtifactCodecRegistry
|
artifactCodecs *ArtifactCodecRegistry
|
||||||
|
evidencePlan *preparedEvidencePlan
|
||||||
checkpointFingerprints []CheckpointFingerprint
|
checkpointFingerprints []CheckpointFingerprint
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -55,6 +57,17 @@ type preparedTypedLane struct {
|
|||||||
codec artifactCodecEntry
|
codec artifactCodecEntry
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type preparedEvidencePlan struct {
|
||||||
|
policy EvidenceContextPolicy
|
||||||
|
lanes []preparedEvidenceLane
|
||||||
|
}
|
||||||
|
|
||||||
|
type preparedEvidenceLane struct {
|
||||||
|
laneID string
|
||||||
|
kind contracts.ArtifactKind
|
||||||
|
project func(any) ([]source.SourceRef, error)
|
||||||
|
}
|
||||||
|
|
||||||
type preparedValidatorChain struct {
|
type preparedValidatorChain struct {
|
||||||
resolved ResolvedValidatorChain
|
resolved ResolvedValidatorChain
|
||||||
validators []preparedValidator
|
validators []preparedValidator
|
||||||
@@ -129,6 +142,10 @@ func Prepare(resolved ResolvedPipeline, registries Registries, deps ModuleDepend
|
|||||||
return nil, constructionError(stable.ID, "", StageOutput, stable.Output.Module, "", err)
|
return nil, constructionError(stable.ID, "", StageOutput, stable.Output.Module, "", err)
|
||||||
}
|
}
|
||||||
prepared.output = output
|
prepared.output = output
|
||||||
|
prepared.evidencePlan, err = prepareEvidencePlan(stable, registries, output)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
prepared.checkpointFingerprints, err = collectPreparedCheckpointFingerprints(prepared)
|
prepared.checkpointFingerprints, err = collectPreparedCheckpointFingerprints(prepared)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@@ -136,6 +153,52 @@ func Prepare(resolved ResolvedPipeline, registries Registries, deps ModuleDepend
|
|||||||
return prepared, nil
|
return prepared, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func prepareEvidencePlan(resolved ResolvedPipeline, registries Registries, output contracts.OutputEncoder) (*preparedEvidencePlan, error) {
|
||||||
|
provider, ok := output.(EvidenceContextPolicyProvider)
|
||||||
|
if !ok {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
policy := cloneEvidenceContextPolicy(provider.EvidenceContextPolicy())
|
||||||
|
if !policy.Enabled {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
if policy.WindowUnits < 0 {
|
||||||
|
return nil, constructionError(resolved.ID, "", StageOutput, resolved.Output.Module, "", fmt.Errorf("evidence window units must not be negative"))
|
||||||
|
}
|
||||||
|
lanes, err := normalizeEvidenceLaneIDs(policy.LaneIDs)
|
||||||
|
if err != nil {
|
||||||
|
return nil, constructionError(resolved.ID, "", StageOutput, resolved.Output.Module, "", err)
|
||||||
|
}
|
||||||
|
policy.LaneIDs = lanes
|
||||||
|
active := make(map[string]ResolvedArtifactLane)
|
||||||
|
for _, lane := range resolved.AllArtifactLanes() {
|
||||||
|
active[lane.ID] = lane
|
||||||
|
}
|
||||||
|
plan := &preparedEvidencePlan{policy: cloneEvidenceContextPolicy(policy)}
|
||||||
|
for _, laneID := range policy.LaneIDs {
|
||||||
|
lane, ok := active[laneID]
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if registries.ArtifactEvidence == nil {
|
||||||
|
return nil, constructionError(resolved.ID, laneID, StageOutput, resolved.Output.Module, "", fmt.Errorf("artifact evidence registry must not be nil for active evidence lane"))
|
||||||
|
}
|
||||||
|
evidence, _, evidenceErr := registries.ArtifactEvidence.entry(lane.ArtifactKind)
|
||||||
|
if evidenceErr != nil {
|
||||||
|
return nil, constructionError(resolved.ID, laneID, StageOutput, resolved.Output.Module, "", evidenceErr)
|
||||||
|
}
|
||||||
|
codecType, ok := registries.ArtifactCodecs.valueType(lane.ArtifactKind)
|
||||||
|
if !ok {
|
||||||
|
return nil, constructionError(resolved.ID, laneID, StageOutput, resolved.Output.Module, "", fmt.Errorf("artifact codec %q has no Go type", lane.ArtifactKind))
|
||||||
|
}
|
||||||
|
if evidence.valueType != codecType {
|
||||||
|
return nil, constructionError(resolved.ID, laneID, StageOutput, resolved.Output.Module, "", fmt.Errorf("artifact evidence kind %q requires Go type %s, but active artifact codec uses %s", lane.ArtifactKind, typeName(evidence.valueType), typeName(codecType)))
|
||||||
|
}
|
||||||
|
plan.lanes = append(plan.lanes, preparedEvidenceLane{laneID: laneID, kind: lane.ArtifactKind, project: evidence.project})
|
||||||
|
}
|
||||||
|
return plan, nil
|
||||||
|
}
|
||||||
|
|
||||||
func prepareLane(pipeline ResolvedPipeline, lane ResolvedArtifactLane, registries Registries, deps ModuleDependencies) (preparedLaneExecutor, error) {
|
func prepareLane(pipeline ResolvedPipeline, lane ResolvedArtifactLane, registries Registries, deps ModuleDependencies) (preparedLaneExecutor, error) {
|
||||||
executor := preparedLaneExecutor{resolved: cloneResolvedArtifactLane(lane)}
|
executor := preparedLaneExecutor{resolved: cloneResolvedArtifactLane(lane)}
|
||||||
request := func(binding ModuleBinding, references contracts.ReferenceSet) BuildRequest {
|
request := func(binding ModuleBinding, references contracts.ReferenceSet) BuildRequest {
|
||||||
@@ -303,7 +366,7 @@ func constructionError(pipelineID, laneID string, stage ModuleStage, moduleKey,
|
|||||||
|
|
||||||
func (registries Registries) catalog() ModuleCatalog {
|
func (registries Registries) catalog() ModuleCatalog {
|
||||||
return ModuleCatalog{
|
return ModuleCatalog{
|
||||||
Inputs: registries.Inputs, Chunkers: registries.Chunkers, ArtifactCodecs: registries.ArtifactCodecs,
|
Inputs: registries.Inputs, Chunkers: registries.Chunkers, ArtifactCodecs: registries.ArtifactCodecs, ArtifactEvidence: registries.ArtifactEvidence,
|
||||||
Extractors: registries.Extractors, Mergers: registries.Mergers, Normalizers: registries.Normalizers,
|
Extractors: registries.Extractors, Mergers: registries.Mergers, Normalizers: registries.Normalizers,
|
||||||
Validators: registries.Validators, ValidatorChains: registries.ValidatorChains, Outputs: registries.Outputs,
|
Validators: registries.Validators, ValidatorChains: registries.ValidatorChains, Outputs: registries.Outputs,
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -117,6 +117,7 @@ type PipelineStepProfile struct {
|
|||||||
|
|
||||||
type PipelineProfile struct {
|
type PipelineProfile struct {
|
||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
|
LLMProfile string `json:"llm_profile,omitempty"`
|
||||||
Input ModuleBinding `json:"input"`
|
Input ModuleBinding `json:"input"`
|
||||||
Chunk ModuleBinding `json:"chunk,omitempty"`
|
Chunk ModuleBinding `json:"chunk,omitempty"`
|
||||||
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
|
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
|
||||||
@@ -127,6 +128,7 @@ type PipelineProfile struct {
|
|||||||
|
|
||||||
type ResolveOptions struct {
|
type ResolveOptions struct {
|
||||||
Only []string
|
Only []string
|
||||||
|
LLMProfileOverride string
|
||||||
ReferenceOverrides []ReferenceBinding
|
ReferenceOverrides []ReferenceBinding
|
||||||
ReferenceUnbinds []ReferenceUnbind
|
ReferenceUnbinds []ReferenceUnbind
|
||||||
}
|
}
|
||||||
@@ -164,8 +166,11 @@ type ResolvedArtifactLane struct {
|
|||||||
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
|
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
|
||||||
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
|
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
|
||||||
Extract ModuleBinding
|
Extract ModuleBinding
|
||||||
|
ExtractExecutionClass contracts.ExecutionClass `json:"extract_execution_class"`
|
||||||
Merge ModuleBinding
|
Merge ModuleBinding
|
||||||
|
MergeExecutionClass contracts.ExecutionClass `json:"merge_execution_class"`
|
||||||
Normalize ModuleBinding
|
Normalize ModuleBinding
|
||||||
|
NormalizeExecutionClass contracts.ExecutionClass `json:"normalize_execution_class"`
|
||||||
Validators []ModuleBinding
|
Validators []ModuleBinding
|
||||||
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
|
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
|
||||||
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
|
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
|
||||||
@@ -195,11 +200,14 @@ type ResolvedPipeline struct {
|
|||||||
ID string
|
ID string
|
||||||
Digest string
|
Digest string
|
||||||
Input ModuleBinding
|
Input ModuleBinding
|
||||||
|
InputExecutionClass contracts.ExecutionClass `json:"input_execution_class"`
|
||||||
Chunk ModuleBinding
|
Chunk ModuleBinding
|
||||||
|
ChunkExecutionClass contracts.ExecutionClass `json:"chunk_execution_class"`
|
||||||
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
|
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
|
||||||
Steps []ResolvedPipelineStep
|
Steps []ResolvedPipelineStep
|
||||||
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
|
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
|
||||||
Output ModuleBinding
|
Output ModuleBinding
|
||||||
|
OutputExecutionClass contracts.ExecutionClass `json:"output_execution_class"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// AllArtifactLanes returns lanes in deterministic step order for read-only
|
// AllArtifactLanes returns lanes in deterministic step order for read-only
|
||||||
@@ -216,6 +224,7 @@ type ModuleCatalog struct {
|
|||||||
Inputs *InputAdapterRegistry
|
Inputs *InputAdapterRegistry
|
||||||
Chunkers *ChunkerRegistry
|
Chunkers *ChunkerRegistry
|
||||||
ArtifactCodecs *ArtifactCodecRegistry
|
ArtifactCodecs *ArtifactCodecRegistry
|
||||||
|
ArtifactEvidence *ArtifactEvidenceRegistry
|
||||||
Extractors *ExtractorRegistry
|
Extractors *ExtractorRegistry
|
||||||
Mergers *MergerRegistry
|
Mergers *MergerRegistry
|
||||||
Normalizers *NormalizerRegistry
|
Normalizers *NormalizerRegistry
|
||||||
@@ -224,6 +233,46 @@ type ModuleCatalog struct {
|
|||||||
Outputs *OutputEncoderRegistry
|
Outputs *OutputEncoderRegistry
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ExecutionClass returns the registered execution class for a module selected
|
||||||
|
// by stage and key without constructing the module.
|
||||||
|
func (catalog ModuleCatalog) ExecutionClass(stage ModuleStage, key string) (contracts.ExecutionClass, bool) {
|
||||||
|
var executionClass contracts.ExecutionClass
|
||||||
|
var ok bool
|
||||||
|
|
||||||
|
switch stage {
|
||||||
|
case StageInput:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Inputs.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageChunk:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Chunkers.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageExtract:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Extractors.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageMerge:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Mergers.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageNormalize:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Normalizers.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageValidate:
|
||||||
|
var spec ValidatorSpec
|
||||||
|
spec, ok = catalog.Validators.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
case StageOutput:
|
||||||
|
var spec ModuleSpec
|
||||||
|
spec, ok = catalog.Outputs.Spec(key)
|
||||||
|
executionClass = spec.ExecutionClass
|
||||||
|
}
|
||||||
|
|
||||||
|
return executionClass, ok
|
||||||
|
}
|
||||||
|
|
||||||
func Binding(module string) ModuleBinding {
|
func Binding(module string) ModuleBinding {
|
||||||
return ModuleBinding{Module: strings.TrimSpace(module)}
|
return ModuleBinding{Module: strings.TrimSpace(module)}
|
||||||
}
|
}
|
||||||
@@ -251,6 +300,10 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
|||||||
if len(steps) == 0 {
|
if len(steps) == 0 {
|
||||||
return ResolvedPipeline{}, fmt.Errorf("pipeline %q must declare at least one artifact lane", pipelineID)
|
return ResolvedPipeline{}, fmt.Errorf("pipeline %q must declare at least one artifact lane", pipelineID)
|
||||||
}
|
}
|
||||||
|
configuredLaneIDs, err := configuredProfileLaneIDs(pipelineID, steps)
|
||||||
|
if err != nil {
|
||||||
|
return ResolvedPipeline{}, err
|
||||||
|
}
|
||||||
|
|
||||||
input := resolveBinding(profile.Input, "")
|
input := resolveBinding(profile.Input, "")
|
||||||
if input.Module == "" {
|
if input.Module == "" {
|
||||||
@@ -313,7 +366,9 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
|||||||
resolved := ResolvedPipeline{
|
resolved := ResolvedPipeline{
|
||||||
ID: pipelineID,
|
ID: pipelineID,
|
||||||
Input: input,
|
Input: input,
|
||||||
|
InputExecutionClass: inputModuleSpec.ExecutionClass,
|
||||||
Chunk: chunk,
|
Chunk: chunk,
|
||||||
|
ChunkExecutionClass: chunkSpec.ExecutionClass,
|
||||||
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
|
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
|
||||||
Output: resolveBinding(profile.Output, DefaultOutputModule),
|
Output: resolveBinding(profile.Output, DefaultOutputModule),
|
||||||
}
|
}
|
||||||
@@ -381,7 +436,11 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
|||||||
if missing, ok := outputCapabilities.missing(outputSpec.Requires); ok {
|
if missing, ok := outputCapabilities.missing(outputSpec.Requires); ok {
|
||||||
return ResolvedPipeline{}, capabilityError(pipelineID, "", StageOutput, resolved.Output.Module, missing)
|
return ResolvedPipeline{}, capabilityError(pipelineID, "", StageOutput, resolved.Output.Module, missing)
|
||||||
}
|
}
|
||||||
if err := validateResolvedOptions(resolved, catalog); err != nil {
|
resolved.OutputExecutionClass = outputSpec.ExecutionClass
|
||||||
|
if err := applyEffectiveLLMProfiles(&resolved, profile.LLMProfile, options.LLMProfileOverride); err != nil {
|
||||||
|
return ResolvedPipeline{}, err
|
||||||
|
}
|
||||||
|
if err := validateResolvedOptions(resolved, catalog, configuredLaneIDs); err != nil {
|
||||||
return ResolvedPipeline{}, err
|
return ResolvedPipeline{}, err
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -393,6 +452,29 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
|||||||
return resolved, nil
|
return resolved, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// configuredProfileLaneIDs validates the complete configured lane identity
|
||||||
|
// set before invocation-level selection removes legacy-profile lanes.
|
||||||
|
func configuredProfileLaneIDs(pipelineID string, steps []PipelineStepProfile) ([]string, error) {
|
||||||
|
seen := make(map[string]string)
|
||||||
|
var laneIDs []string
|
||||||
|
for _, step := range steps {
|
||||||
|
_, ids, err := selectedArtifactLanes(pipelineID, step.Artifacts, ResolveOptions{})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
stepID := strings.TrimSpace(step.ID)
|
||||||
|
for _, laneID := range ids {
|
||||||
|
if previous, ok := seen[laneID]; ok {
|
||||||
|
return nil, fmt.Errorf("pipeline %q artifact lane id %q is duplicated across steps %q and %q", pipelineID, laneID, previous, stepID)
|
||||||
|
}
|
||||||
|
seen[laneID] = stepID
|
||||||
|
laneIDs = append(laneIDs, laneID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Strings(laneIDs)
|
||||||
|
return laneIDs, nil
|
||||||
|
}
|
||||||
|
|
||||||
func resolveArtifactLane(
|
func resolveArtifactLane(
|
||||||
pipelineID string,
|
pipelineID string,
|
||||||
stepID string,
|
stepID string,
|
||||||
@@ -447,6 +529,7 @@ func resolveArtifactLane(
|
|||||||
}
|
}
|
||||||
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
|
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
|
||||||
lane.ExtractReferences.StepID = strings.TrimSpace(stepID)
|
lane.ExtractReferences.StepID = strings.TrimSpace(stepID)
|
||||||
|
lane.ExtractExecutionClass = extractSpec.ExecutionClass
|
||||||
capabilities.add(extractSpec.Provides...)
|
capabilities.add(extractSpec.Provides...)
|
||||||
|
|
||||||
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
|
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
|
||||||
@@ -471,6 +554,7 @@ func resolveArtifactLane(
|
|||||||
}
|
}
|
||||||
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
|
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
|
||||||
lane.MergeReferences.StepID = strings.TrimSpace(stepID)
|
lane.MergeReferences.StepID = strings.TrimSpace(stepID)
|
||||||
|
lane.MergeExecutionClass = mergeSpec.ExecutionClass
|
||||||
capabilities.add(mergeSpec.Provides...)
|
capabilities.add(mergeSpec.Provides...)
|
||||||
|
|
||||||
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
|
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
|
||||||
@@ -495,6 +579,7 @@ func resolveArtifactLane(
|
|||||||
}
|
}
|
||||||
lane.NormalizeReferences = referenceTarget(StageNormalize, laneID, lane.Normalize.Module, normalizeReferences)
|
lane.NormalizeReferences = referenceTarget(StageNormalize, laneID, lane.Normalize.Module, normalizeReferences)
|
||||||
lane.NormalizeReferences.StepID = strings.TrimSpace(stepID)
|
lane.NormalizeReferences.StepID = strings.TrimSpace(stepID)
|
||||||
|
lane.NormalizeExecutionClass = normalizeSpec.ExecutionClass
|
||||||
capabilities.add(normalizeSpec.Provides...)
|
capabilities.add(normalizeSpec.Provides...)
|
||||||
|
|
||||||
if len(lane.Validators) > 0 {
|
if len(lane.Validators) > 0 {
|
||||||
@@ -756,9 +841,6 @@ func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage,
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q: %w", pipelineID, stage, chain.ModuleKey, err)
|
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q: %w", pipelineID, stage, chain.ModuleKey, err)
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(validator.LLMProfile) != "" && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
|
|
||||||
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q assigns llm_profile to deterministic validator %q", pipelineID, stage, chain.ModuleKey, validator.Module)
|
|
||||||
}
|
|
||||||
chain.Validators = append(chain.Validators, ResolvedValidator{
|
chain.Validators = append(chain.Validators, ResolvedValidator{
|
||||||
Binding: cloneModuleBinding(validator),
|
Binding: cloneModuleBinding(validator),
|
||||||
ExecutionClass: spec.ExecutionClass,
|
ExecutionClass: spec.ExecutionClass,
|
||||||
@@ -1170,6 +1252,66 @@ func resolveBinding(binding ModuleBinding, defaultModule string) ModuleBinding {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func applyEffectiveLLMProfiles(resolved *ResolvedPipeline, pipelineProfile, overrideProfile string) error {
|
||||||
|
pipelineProfile = strings.TrimSpace(pipelineProfile)
|
||||||
|
overrideProfile = strings.TrimSpace(overrideProfile)
|
||||||
|
|
||||||
|
apply := func(stage ModuleStage, laneID, module string, binding *ModuleBinding, executionClass contracts.ExecutionClass, kind string) error {
|
||||||
|
binding.LLMProfile = strings.TrimSpace(binding.LLMProfile)
|
||||||
|
if executionClass != contracts.ExecutionClassLLMBacked {
|
||||||
|
if binding.LLMProfile != "" {
|
||||||
|
if laneID == "" {
|
||||||
|
return fmt.Errorf("pipeline %q %s %q assigns llm_profile to deterministic %s %q", resolved.ID, stage, module, kind, binding.Module)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("pipeline %q lane %q %s %q assigns llm_profile to deterministic %s %q", resolved.ID, laneID, stage, module, kind, binding.Module)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if overrideProfile != "" {
|
||||||
|
binding.LLMProfile = overrideProfile
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if binding.LLMProfile == "" {
|
||||||
|
binding.LLMProfile = pipelineProfile
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := apply(StageInput, "", resolved.Input.Module, &resolved.Input, resolved.InputExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := apply(StageChunk, "", resolved.Chunk.Module, &resolved.Chunk, resolved.ChunkExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for stepIndex := range resolved.Steps {
|
||||||
|
for laneIndex := range resolved.Steps[stepIndex].ArtifactLanes {
|
||||||
|
lane := &resolved.Steps[stepIndex].ArtifactLanes[laneIndex]
|
||||||
|
if err := apply(StageExtract, lane.ID, lane.Extract.Module, &lane.Extract, lane.ExtractExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := apply(StageMerge, lane.ID, lane.Merge.Module, &lane.Merge, lane.MergeExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := apply(StageNormalize, lane.ID, lane.Normalize.Module, &lane.Normalize, lane.NormalizeExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := apply(StageOutput, "", resolved.Output.Module, &resolved.Output, resolved.OutputExecutionClass, "module"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for chainIndex := range resolved.ValidatorChains {
|
||||||
|
chain := &resolved.ValidatorChains[chainIndex]
|
||||||
|
for validatorIndex := range chain.Validators {
|
||||||
|
validator := &chain.Validators[validatorIndex]
|
||||||
|
if err := apply(chain.Stage, chain.LaneID, chain.ModuleKey, &validator.Binding, validator.ExecutionClass, "validator"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func resolveBindings(bindings []ModuleBinding, defaultModule string) []ModuleBinding {
|
func resolveBindings(bindings []ModuleBinding, defaultModule string) []ModuleBinding {
|
||||||
if len(bindings) == 0 {
|
if len(bindings) == 0 {
|
||||||
return nil
|
return nil
|
||||||
@@ -1190,11 +1332,32 @@ func cloneOptions(options map[string]any) map[string]any {
|
|||||||
|
|
||||||
copied := make(map[string]any, len(options))
|
copied := make(map[string]any, len(options))
|
||||||
for key, value := range options {
|
for key, value := range options {
|
||||||
copied[key] = value
|
copied[key] = cloneOptionValue(value)
|
||||||
}
|
}
|
||||||
return copied
|
return copied
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func cloneOptionValue(value any) any {
|
||||||
|
switch typed := value.(type) {
|
||||||
|
case map[string]any:
|
||||||
|
return cloneOptions(typed)
|
||||||
|
case []any:
|
||||||
|
out := make([]any, len(typed))
|
||||||
|
for i := range typed {
|
||||||
|
out[i] = cloneOptionValue(typed[i])
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
case []string:
|
||||||
|
return append([]string(nil), typed...)
|
||||||
|
case []byte:
|
||||||
|
return append([]byte(nil), typed...)
|
||||||
|
case json.RawMessage:
|
||||||
|
return append(json.RawMessage(nil), typed...)
|
||||||
|
default:
|
||||||
|
return value
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func normalizeReferenceMap(values map[string]ReferenceSource) map[string]ReferenceSource {
|
func normalizeReferenceMap(values map[string]ReferenceSource) map[string]ReferenceSource {
|
||||||
if len(values) == 0 {
|
if len(values) == 0 {
|
||||||
return nil
|
return nil
|
||||||
@@ -1260,19 +1423,25 @@ func resolvedPipelineDigest(resolved ResolvedPipeline) (string, error) {
|
|||||||
withoutDigest := struct {
|
withoutDigest := struct {
|
||||||
ID string
|
ID string
|
||||||
Input ModuleBinding
|
Input ModuleBinding
|
||||||
|
InputExecutionClass contracts.ExecutionClass
|
||||||
Chunk ModuleBinding
|
Chunk ModuleBinding
|
||||||
|
ChunkExecutionClass contracts.ExecutionClass
|
||||||
ChunkReferences ResolvedReferenceTarget
|
ChunkReferences ResolvedReferenceTarget
|
||||||
Steps []ResolvedPipelineStep
|
Steps []ResolvedPipelineStep
|
||||||
ValidatorChains []ResolvedValidatorChain
|
ValidatorChains []ResolvedValidatorChain
|
||||||
Output ModuleBinding
|
Output ModuleBinding
|
||||||
|
OutputExecutionClass contracts.ExecutionClass
|
||||||
}{
|
}{
|
||||||
ID: resolved.ID,
|
ID: resolved.ID,
|
||||||
Input: resolved.Input,
|
Input: resolved.Input,
|
||||||
|
InputExecutionClass: resolved.InputExecutionClass,
|
||||||
Chunk: resolved.Chunk,
|
Chunk: resolved.Chunk,
|
||||||
|
ChunkExecutionClass: resolved.ChunkExecutionClass,
|
||||||
ChunkReferences: resolved.ChunkReferences,
|
ChunkReferences: resolved.ChunkReferences,
|
||||||
Steps: resolved.Steps,
|
Steps: resolved.Steps,
|
||||||
ValidatorChains: resolved.ValidatorChains,
|
ValidatorChains: resolved.ValidatorChains,
|
||||||
Output: resolved.Output,
|
Output: resolved.Output,
|
||||||
|
OutputExecutionClass: resolved.OutputExecutionClass,
|
||||||
}
|
}
|
||||||
encoded, err := json.Marshal(withoutDigest)
|
encoded, err := json.Marshal(withoutDigest)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -15,22 +15,22 @@ import (
|
|||||||
func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
||||||
catalog := newProfileCatalog(t)
|
catalog := newProfileCatalog(t)
|
||||||
registerProfileSpecs(t, catalog,
|
registerProfileSpecs(t, catalog,
|
||||||
ModuleSpec{Key: "window", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
ModuleSpec{Key: "window", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||||
ModuleSpec{Key: "record-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
ModuleSpec{Key: "record-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||||
ModuleSpec{Key: "dedupe", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
ModuleSpec{Key: "dedupe", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||||
ModuleSpec{Key: "canonical", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
ModuleSpec{Key: "canonical", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||||
ModuleSpec{Key: "ndjson", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
ModuleSpec{Key: "ndjson", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||||
)
|
)
|
||||||
|
|
||||||
resolved, err := ResolvePipeline(PipelineProfile{
|
resolved, err := ResolvePipeline(PipelineProfile{
|
||||||
ID: " campaign ",
|
ID: " campaign ",
|
||||||
Input: ModuleBinding{Module: " text ", LLMProfile: " fast "},
|
Input: Binding(" text "),
|
||||||
Chunk: ModuleBinding{Module: " window ", Options: map[string]any{
|
Chunk: ModuleBinding{Module: " window ", Options: map[string]any{
|
||||||
"size": 10,
|
"size": 10,
|
||||||
}},
|
}},
|
||||||
Artifacts: map[string]ArtifactLaneProfile{
|
Artifacts: map[string]ArtifactLaneProfile{
|
||||||
" records ": {
|
" records ": {
|
||||||
Extract: ModuleBinding{Module: " record-extractor ", LLMProfile: " careful "},
|
Extract: Binding(" record-extractor "),
|
||||||
Merge: Binding(" dedupe "),
|
Merge: Binding(" dedupe "),
|
||||||
Normalize: Binding(" canonical "),
|
Normalize: Binding(" canonical "),
|
||||||
},
|
},
|
||||||
@@ -44,15 +44,21 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
|||||||
if resolved.ID != "campaign" {
|
if resolved.ID != "campaign" {
|
||||||
t.Fatalf("ID = %q, want campaign", resolved.ID)
|
t.Fatalf("ID = %q, want campaign", resolved.ID)
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(resolved.Input, ModuleBinding{Module: "text", LLMProfile: "fast"}) {
|
if !reflect.DeepEqual(resolved.Input, ModuleBinding{Module: "text"}) {
|
||||||
t.Fatalf("Input = %#v, want trimmed explicit input", resolved.Input)
|
t.Fatalf("Input = %#v, want trimmed explicit input", resolved.Input)
|
||||||
}
|
}
|
||||||
|
if resolved.InputExecutionClass != contracts.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("InputExecutionClass = %q, want deterministic", resolved.InputExecutionClass)
|
||||||
|
}
|
||||||
if resolved.Chunk.Module != "window" || resolved.Chunk.LLMProfile != "" {
|
if resolved.Chunk.Module != "window" || resolved.Chunk.LLMProfile != "" {
|
||||||
t.Fatalf("Chunk = %#v, want explicit module and empty LLM profile", resolved.Chunk)
|
t.Fatalf("Chunk = %#v, want explicit module and empty LLM profile", resolved.Chunk)
|
||||||
}
|
}
|
||||||
if resolved.Chunk.Options["size"] != 10 {
|
if resolved.Chunk.Options["size"] != 10 {
|
||||||
t.Fatalf("Chunk.Options = %#v, want size option", resolved.Chunk.Options)
|
t.Fatalf("Chunk.Options = %#v, want size option", resolved.Chunk.Options)
|
||||||
}
|
}
|
||||||
|
if resolved.ChunkExecutionClass != contracts.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("ChunkExecutionClass = %q, want deterministic", resolved.ChunkExecutionClass)
|
||||||
|
}
|
||||||
if len(resolved.Steps) != 1 || resolved.Steps[0].ID != "default" || len(resolved.Steps[0].ArtifactLanes) != 1 {
|
if len(resolved.Steps) != 1 || resolved.Steps[0].ID != "default" || len(resolved.Steps[0].ArtifactLanes) != 1 {
|
||||||
t.Fatalf("resolved steps = %#v, want one default step with one lane", resolved.Steps)
|
t.Fatalf("resolved steps = %#v, want one default step with one lane", resolved.Steps)
|
||||||
}
|
}
|
||||||
@@ -60,9 +66,12 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
|||||||
if lane.ID != "records" {
|
if lane.ID != "records" {
|
||||||
t.Fatalf("lane.ID = %q, want records", lane.ID)
|
t.Fatalf("lane.ID = %q, want records", lane.ID)
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(lane.Extract, ModuleBinding{Module: "record-extractor", LLMProfile: "careful"}) {
|
if !reflect.DeepEqual(lane.Extract, ModuleBinding{Module: "record-extractor"}) {
|
||||||
t.Fatalf("lane.Extract = %#v, want explicit extractor", lane.Extract)
|
t.Fatalf("lane.Extract = %#v, want explicit extractor", lane.Extract)
|
||||||
}
|
}
|
||||||
|
if lane.ExtractExecutionClass != contracts.ExecutionClassDeterministic || lane.MergeExecutionClass != contracts.ExecutionClassDeterministic || lane.NormalizeExecutionClass != contracts.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("lane execution classes = %q/%q/%q, want deterministic", lane.ExtractExecutionClass, lane.MergeExecutionClass, lane.NormalizeExecutionClass)
|
||||||
|
}
|
||||||
if lane.Merge.Module != "dedupe" || lane.Normalize.Module != "canonical" {
|
if lane.Merge.Module != "dedupe" || lane.Normalize.Module != "canonical" {
|
||||||
t.Fatalf("lane merge/normalize = %#v/%#v, want explicit modules", lane.Merge, lane.Normalize)
|
t.Fatalf("lane merge/normalize = %#v/%#v, want explicit modules", lane.Merge, lane.Normalize)
|
||||||
}
|
}
|
||||||
@@ -72,11 +81,51 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
|||||||
if resolved.Output.Module != "ndjson" {
|
if resolved.Output.Module != "ndjson" {
|
||||||
t.Fatalf("Output.Module = %q, want ndjson", resolved.Output.Module)
|
t.Fatalf("Output.Module = %q, want ndjson", resolved.Output.Module)
|
||||||
}
|
}
|
||||||
|
if resolved.OutputExecutionClass != contracts.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("OutputExecutionClass = %q, want deterministic", resolved.OutputExecutionClass)
|
||||||
|
}
|
||||||
if !strings.HasPrefix(resolved.Digest, "sha256:") {
|
if !strings.HasPrefix(resolved.Digest, "sha256:") {
|
||||||
t.Fatalf("Digest = %q, want sha256 digest", resolved.Digest)
|
t.Fatalf("Digest = %q, want sha256 digest", resolved.Digest)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestModuleCatalogExecutionClassLooksUpRegisteredMetadata(t *testing.T) {
|
||||||
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
|
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
ModuleSpec{Key: "llm-extract", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||||
|
)
|
||||||
|
registerProfileValidatorSpec(t, catalog, ValidatorSpec{Key: "llm-validator", ExecutionClass: contracts.ExecutionClassLLMBacked})
|
||||||
|
|
||||||
|
for _, test := range []struct {
|
||||||
|
stage ModuleStage
|
||||||
|
key string
|
||||||
|
want contracts.ExecutionClass
|
||||||
|
}{
|
||||||
|
{stage: StageInput, key: "llm-input", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageChunk, key: "llm-chunk", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageExtract, key: "llm-extract", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageMerge, key: "llm-merge", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageNormalize, key: "llm-normalize", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageValidate, key: "llm-validator", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
{stage: StageOutput, key: "llm-output", want: contracts.ExecutionClassLLMBacked},
|
||||||
|
} {
|
||||||
|
t.Run(string(test.stage), func(t *testing.T) {
|
||||||
|
got, ok := catalog.ExecutionClass(test.stage, test.key)
|
||||||
|
if !ok || got != test.want {
|
||||||
|
t.Fatalf("ExecutionClass(%q, %q) = %q, %t; want %q, true", test.stage, test.key, got, ok, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, ok := catalog.ExecutionClass(StageExtract, "missing"); ok {
|
||||||
|
t.Fatal("ExecutionClass() found an unregistered module")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestResolvePipelineAppliesDefaults(t *testing.T) {
|
func TestResolvePipelineAppliesDefaults(t *testing.T) {
|
||||||
resolved, err := ResolvePipeline(PipelineProfile{
|
resolved, err := ResolvePipeline(PipelineProfile{
|
||||||
ID: "defaulted",
|
ID: "defaulted",
|
||||||
@@ -110,6 +159,176 @@ func TestResolvePipelineAppliesDefaults(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestResolvePipelineAppliesEffectiveLLMProfiles(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
profile PipelineProfile
|
||||||
|
options ResolveOptions
|
||||||
|
want map[string]string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "runtime override",
|
||||||
|
profile: func() PipelineProfile {
|
||||||
|
profile := llmProfilePipeline()
|
||||||
|
profile.LLMProfile = " pipeline "
|
||||||
|
profile.Chunk.LLMProfile = "binding"
|
||||||
|
return profile
|
||||||
|
}(),
|
||||||
|
options: ResolveOptions{LLMProfileOverride: " runtime "},
|
||||||
|
want: llmProfileValues("runtime"),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "binding exception",
|
||||||
|
profile: func() PipelineProfile {
|
||||||
|
profile := llmProfilePipeline()
|
||||||
|
profile.LLMProfile = "pipeline"
|
||||||
|
lane := profile.Artifacts["events"]
|
||||||
|
lane.Extract.LLMProfile = "extract"
|
||||||
|
lane.Extract.Validators = ValidatorOverride{
|
||||||
|
Set: true,
|
||||||
|
Validators: []ModuleBinding{{Module: "llm-validator", LLMProfile: "validator"}},
|
||||||
|
}
|
||||||
|
profile.Artifacts["events"] = lane
|
||||||
|
return profile
|
||||||
|
}(),
|
||||||
|
want: func() map[string]string {
|
||||||
|
values := llmProfileValues("pipeline")
|
||||||
|
values["extract"] = "extract"
|
||||||
|
values["validator:extract:events"] = "validator"
|
||||||
|
return values
|
||||||
|
}(),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "pipeline default",
|
||||||
|
profile: func() PipelineProfile {
|
||||||
|
profile := llmProfilePipeline()
|
||||||
|
profile.LLMProfile = "pipeline"
|
||||||
|
return profile
|
||||||
|
}(),
|
||||||
|
want: llmProfileValues("pipeline"),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "prompt fallback",
|
||||||
|
profile: llmProfilePipeline(),
|
||||||
|
want: llmProfileValues(""),
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
resolved, err := ResolvePipeline(test.profile, test.options, llmProfileCatalog(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if got := resolvedLLMProfileValues(resolved); !reflect.DeepEqual(got, test.want) {
|
||||||
|
t.Fatalf("resolved profiles = %#v, want %#v", got, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePipelineAppliesProfilesOnlyToSelectedLLMBackedBindings(t *testing.T) {
|
||||||
|
profile := llmProfilePipeline()
|
||||||
|
profile.LLMProfile = "pipeline"
|
||||||
|
profile.Artifacts["notes"] = ArtifactLaneProfile{Extract: Binding("llm-extractor")}
|
||||||
|
|
||||||
|
resolved, err := ResolvePipeline(profile, ResolveOptions{Only: []string{"events"}}, llmProfileCatalog(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if got := laneIDs(resolved.Steps[0].ArtifactLanes); !reflect.DeepEqual(got, []string{"events"}) {
|
||||||
|
t.Fatalf("selected lanes = %#v, want events only", got)
|
||||||
|
}
|
||||||
|
if got := resolvedLLMProfileValues(resolved); !reflect.DeepEqual(got, llmProfileValues("pipeline")) {
|
||||||
|
t.Fatalf("resolved profiles = %#v, want selected LLM bindings only", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePipelineLeavesUnusedProfilesOffDeterministicBindings(t *testing.T) {
|
||||||
|
profile := baselineProfile()
|
||||||
|
profile.LLMProfile = "unused"
|
||||||
|
|
||||||
|
resolved, err := ResolvePipeline(profile, ResolveOptions{LLMProfileOverride: "also-unused"}, newProfileCatalog(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if resolved.Input.LLMProfile != "" || resolved.Chunk.LLMProfile != "" || resolved.Output.LLMProfile != "" {
|
||||||
|
t.Fatalf("deterministic pipeline profiles = input %q chunk %q output %q, want empty", resolved.Input.LLMProfile, resolved.Chunk.LLMProfile, resolved.Output.LLMProfile)
|
||||||
|
}
|
||||||
|
lane := resolved.Steps[0].ArtifactLanes[0]
|
||||||
|
if lane.Extract.LLMProfile != "" || lane.Merge.LLMProfile != "" || lane.Normalize.LLMProfile != "" {
|
||||||
|
t.Fatalf("deterministic lane profiles = extract %q merge %q normalize %q, want empty", lane.Extract.LLMProfile, lane.Merge.LLMProfile, lane.Normalize.LLMProfile)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePipelineRejectsLLMProfileForDeterministicModule(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
mutate func(*PipelineProfile)
|
||||||
|
}{
|
||||||
|
{name: "input", mutate: func(profile *PipelineProfile) { profile.Input.LLMProfile = "invalid" }},
|
||||||
|
{name: "chunk", mutate: func(profile *PipelineProfile) { profile.Chunk.LLMProfile = "invalid" }},
|
||||||
|
{name: "extract", mutate: func(profile *PipelineProfile) {
|
||||||
|
lane := profile.Artifacts["events"]
|
||||||
|
lane.Extract.LLMProfile = "invalid"
|
||||||
|
profile.Artifacts["events"] = lane
|
||||||
|
}},
|
||||||
|
{name: "merge", mutate: func(profile *PipelineProfile) {
|
||||||
|
lane := profile.Artifacts["events"]
|
||||||
|
lane.Merge.LLMProfile = "invalid"
|
||||||
|
profile.Artifacts["events"] = lane
|
||||||
|
}},
|
||||||
|
{name: "normalize", mutate: func(profile *PipelineProfile) {
|
||||||
|
lane := profile.Artifacts["events"]
|
||||||
|
lane.Normalize.LLMProfile = "invalid"
|
||||||
|
profile.Artifacts["events"] = lane
|
||||||
|
}},
|
||||||
|
{name: "output", mutate: func(profile *PipelineProfile) { profile.Output.LLMProfile = "invalid" }},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
profile := baselineProfile()
|
||||||
|
test.mutate(&profile)
|
||||||
|
_, err := ResolvePipeline(profile, ResolveOptions{}, newProfileCatalog(t))
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "llm_profile") || !strings.Contains(err.Error(), "deterministic") {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want deterministic profile rejection", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePipelineDigestUsesEffectiveLLMProfiles(t *testing.T) {
|
||||||
|
inherited := llmProfilePipeline()
|
||||||
|
inherited.LLMProfile = "shared"
|
||||||
|
explicit := llmProfilePipeline()
|
||||||
|
explicit.Input.LLMProfile = "shared"
|
||||||
|
explicit.Chunk.LLMProfile = "shared"
|
||||||
|
explicit.Output.LLMProfile = "shared"
|
||||||
|
lane := explicit.Artifacts["events"]
|
||||||
|
lane.Extract.LLMProfile = "shared"
|
||||||
|
lane.Merge.LLMProfile = "shared"
|
||||||
|
lane.Normalize.LLMProfile = "shared"
|
||||||
|
explicit.Artifacts["events"] = lane
|
||||||
|
|
||||||
|
inheritedResolved, err := ResolvePipeline(inherited, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline(inherited) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
explicitResolved, err := ResolvePipeline(explicit, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline(explicit) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if inheritedResolved.Digest != explicitResolved.Digest {
|
||||||
|
t.Fatalf("effective profile digests differ: %q != %q", inheritedResolved.Digest, explicitResolved.Digest)
|
||||||
|
}
|
||||||
|
|
||||||
|
explicit.Chunk.LLMProfile = "different"
|
||||||
|
changedResolved, err := ResolvePipeline(explicit, ResolveOptions{}, llmProfileCatalogWithoutValidatorChains(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline(changed) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if explicitResolved.Digest == changedResolved.Digest {
|
||||||
|
t.Fatalf("digest = %q after effective profile change, want different", explicitResolved.Digest)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestResolvePipelineRecordsValidatorChains(t *testing.T) {
|
func TestResolvePipelineRecordsValidatorChains(t *testing.T) {
|
||||||
catalog := newProfileCatalog(t)
|
catalog := newProfileCatalog(t)
|
||||||
if err := catalog.ValidatorChains.Register(ValidatorChainMapping{
|
if err := catalog.ValidatorChains.Register(ValidatorChainMapping{
|
||||||
@@ -337,6 +556,7 @@ func TestResolvePipelinePreservesOrderedStepsAndExpandsGeneratedBindings(t *test
|
|||||||
catalog := newProfileCatalogWithOverrides(t,
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
ModuleSpec{
|
ModuleSpec{
|
||||||
Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes",
|
Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes",
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"}, Provides: []string{"candidate"},
|
Requires: []string{"chunk"}, Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}, AcceptedMediaTypes: []string{"application/json"}}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}, AcceptedMediaTypes: []string{"application/json"}}},
|
||||||
})
|
})
|
||||||
@@ -365,7 +585,7 @@ func TestResolvePipelinePreservesOrderedStepsAndExpandsGeneratedBindings(t *test
|
|||||||
|
|
||||||
func TestResolvePipelineRejectsGeneratedBindingOrderingAndKind(t *testing.T) {
|
func TestResolvePipelineRejectsGeneratedBindingOrderingAndKind(t *testing.T) {
|
||||||
catalog := newProfileCatalogWithOverrides(t,
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/other"}}}},
|
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/other"}}}},
|
||||||
)
|
)
|
||||||
_, err := ResolvePipeline(PipelineProfile{
|
_, err := ResolvePipeline(PipelineProfile{
|
||||||
ID: "invalid-order", Input: Binding("text"), Steps: []PipelineStepProfile{
|
ID: "invalid-order", Input: Binding("text"), Steps: []PipelineStepProfile{
|
||||||
@@ -402,6 +622,7 @@ func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
|
|||||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"},
|
Requires: []string{"chunk"},
|
||||||
Provides: []string{"candidate"},
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
@@ -446,6 +667,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToChunkTarget(t *testing.
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "generic",
|
Key: "generic",
|
||||||
Stage: StageChunk,
|
Stage: StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"source"},
|
Requires: []string{"source"},
|
||||||
Provides: []string{"chunk"},
|
Provides: []string{"chunk"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide"}},
|
||||||
@@ -474,6 +696,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToExtractorTarget(t *test
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"},
|
Requires: []string{"chunk"},
|
||||||
Provides: []string{"candidate"},
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
|
||||||
@@ -502,6 +725,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToNormalizerTarget(t *tes
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "noop",
|
Key: "noop",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"merged"},
|
Requires: []string{"merged"},
|
||||||
Provides: []string{"normalized"},
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
||||||
@@ -530,6 +754,7 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToMergeTarget(t *testing.
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "appendorder",
|
Key: "appendorder",
|
||||||
Stage: StageMerge,
|
Stage: StageMerge,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"candidate"},
|
Requires: []string{"candidate"},
|
||||||
Provides: []string{"merged"},
|
Provides: []string{"merged"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "merge_notes"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "merge_notes"}},
|
||||||
@@ -559,10 +784,10 @@ func TestResolvePipelineAppliesOnePipelineReferenceDefaultToMultipleTargets(t *t
|
|||||||
profile := baselineProfile()
|
profile := baselineProfile()
|
||||||
profile.References = ExternalReferenceMap(map[string]string{"context": "./context.md"})
|
profile.References = ExternalReferenceMap(map[string]string{"context": "./context.md"})
|
||||||
catalog := newProfileCatalogWithOverrides(t,
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||||
ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"candidate"}, Provides: []string{"merged"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||||
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||||
)
|
)
|
||||||
|
|
||||||
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
||||||
@@ -582,6 +807,7 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedLane(t *t
|
|||||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||||
Key: "note-extractor",
|
Key: "note-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"},
|
Requires: []string{"chunk"},
|
||||||
Provides: []string{"candidate"},
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
@@ -607,6 +833,7 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedNormalize
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "note-normalizer",
|
Key: "note-normalizer",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"merged"},
|
Requires: []string{"merged"},
|
||||||
Provides: []string{"normalized"},
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
@@ -658,6 +885,7 @@ func TestResolvePipelineRejectsExtractLocalReferenceDeclaredOnlyByNormalizer(t *
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "noop",
|
Key: "noop",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"merged"},
|
Requires: []string{"merged"},
|
||||||
Provides: []string{"normalized"},
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
||||||
@@ -678,6 +906,7 @@ func TestResolvePipelineRejectsMergeLocalReferenceDeclaredOnlyByNormalizer(t *te
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "noop",
|
Key: "noop",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"merged"},
|
Requires: []string{"merged"},
|
||||||
Provides: []string{"normalized"},
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
@@ -700,6 +929,7 @@ func TestResolvePipelineRejectsNormalizeLocalReferenceDeclaredOnlyByExtractor(t
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"},
|
Requires: []string{"chunk"},
|
||||||
Provides: []string{"candidate"},
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}},
|
||||||
@@ -716,6 +946,7 @@ func TestResolvePipelineRequiresBoundChunkReference(t *testing.T) {
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "generic",
|
Key: "generic",
|
||||||
Stage: StageChunk,
|
Stage: StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"source"},
|
Requires: []string{"source"},
|
||||||
Provides: []string{"chunk"},
|
Provides: []string{"chunk"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide", Required: true}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "scene_guide", Required: true}},
|
||||||
@@ -732,6 +963,7 @@ func TestResolvePipelineRequiresBoundNormalizeReference(t *testing.T) {
|
|||||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||||
Key: "noop",
|
Key: "noop",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"merged"},
|
Requires: []string{"merged"},
|
||||||
Provides: []string{"normalized"},
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes", Required: true}},
|
ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes", Required: true}},
|
||||||
@@ -757,9 +989,9 @@ func TestResolvePipelineLocalReferencesOverridePipelineDefaultsForEligibleTarget
|
|||||||
lane.Normalize.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./local-normalize.md"})
|
lane.Normalize.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./local-normalize.md"})
|
||||||
profile.Artifacts["events"] = lane
|
profile.Artifacts["events"] = lane
|
||||||
catalog := newProfileCatalogWithOverrides(t,
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}}},
|
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "roster"}}},
|
||||||
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}}},
|
ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "normalization_notes"}}},
|
||||||
)
|
)
|
||||||
|
|
||||||
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
||||||
@@ -776,6 +1008,7 @@ func TestResolvePipelineRequiresBoundReferenceSlotsForSelectedLanes(t *testing.T
|
|||||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"},
|
Requires: []string{"chunk"},
|
||||||
Provides: []string{"candidate"},
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
@@ -800,6 +1033,7 @@ func TestResolvePipelineReferenceUnbindCanLeaveRequiredSlotMissing(t *testing.T)
|
|||||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"},
|
Requires: []string{"chunk"},
|
||||||
Provides: []string{"candidate"},
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
@@ -829,6 +1063,7 @@ func TestResolvePipelineUsesReferenceSlotsFromSpecWithoutConstructingExtractor(t
|
|||||||
if err := RegisterExtractor[codecNotes](catalog.Extractors, ModuleSpec{
|
if err := RegisterExtractor[codecNotes](catalog.Extractors, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
ArtifactKind: "test/notes",
|
ArtifactKind: "test/notes",
|
||||||
Requires: []string{"chunk"},
|
Requires: []string{"chunk"},
|
||||||
Provides: []string{"candidate"},
|
Provides: []string{"candidate"},
|
||||||
@@ -989,32 +1224,32 @@ func TestResolvePipelineRejectsMissingCapabilities(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "input",
|
name: "input",
|
||||||
spec: ModuleSpec{Key: "text", Stage: StageInput, Requires: []string{"raw"}},
|
spec: ModuleSpec{Key: "text", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"raw"}},
|
||||||
want: []string{"baseline", "input", "text", "raw"},
|
want: []string{"baseline", "input", "text", "raw"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "chunk",
|
name: "chunk",
|
||||||
spec: ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"missing"}},
|
spec: ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||||
want: []string{"baseline", "chunk", "generic", "missing"},
|
want: []string{"baseline", "chunk", "generic", "missing"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "extract",
|
name: "extract",
|
||||||
spec: ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"missing"}},
|
spec: ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||||
want: []string{"baseline", "events", "extract", "event-extractor", "missing"},
|
want: []string{"baseline", "events", "extract", "event-extractor", "missing"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "merge",
|
name: "merge",
|
||||||
spec: ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"missing"}},
|
spec: ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||||
want: []string{"baseline", "events", "merge", "appendorder", "missing"},
|
want: []string{"baseline", "events", "merge", "appendorder", "missing"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "normalize",
|
name: "normalize",
|
||||||
spec: ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"missing"}},
|
spec: ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||||
want: []string{"baseline", "events", "normalize", "noop", "missing"},
|
want: []string{"baseline", "events", "normalize", "noop", "missing"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "output",
|
name: "output",
|
||||||
spec: ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"missing"}},
|
spec: ModuleSpec{Key: "json", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}},
|
||||||
want: []string{"baseline", "output", "json", "missing"},
|
want: []string{"baseline", "output", "json", "missing"},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -1100,7 +1335,7 @@ func TestResolvePipelineDigestChangesWhenBindingChanges(t *testing.T) {
|
|||||||
right := baselineProfile()
|
right := baselineProfile()
|
||||||
right.Chunk = Binding("window")
|
right.Chunk = Binding("window")
|
||||||
catalog := newProfileCatalog(t)
|
catalog := newProfileCatalog(t)
|
||||||
registerProfileSpecs(t, catalog, ModuleSpec{Key: "window", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}})
|
registerProfileSpecs(t, catalog, ModuleSpec{Key: "window", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}})
|
||||||
|
|
||||||
leftResolved, err := ResolvePipeline(left, ResolveOptions{}, catalog)
|
leftResolved, err := ResolvePipeline(left, ResolveOptions{}, catalog)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1467,17 +1702,103 @@ func emptyProfileCatalog() ModuleCatalog {
|
|||||||
|
|
||||||
func defaultProfileSpecs() []ModuleSpec {
|
func defaultProfileSpecs() []ModuleSpec {
|
||||||
return []ModuleSpec{
|
return []ModuleSpec{
|
||||||
ModuleSpec{Key: "text", Stage: StageInput, Provides: []string{"source"}},
|
ModuleSpec{Key: "text", Stage: StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}},
|
||||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
ModuleSpec{Key: "generic", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||||
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||||
ModuleSpec{Key: "appendorder", Stage: StageMerge, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
ModuleSpec{Key: "appendorder", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||||
ModuleSpec{Key: "noop", Stage: StageNormalize, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
ModuleSpec{Key: "noop", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||||
ModuleSpec{Key: "grounded", Stage: StageValidate, Requires: []string{"normalized"}, Provides: []string{"validated"}},
|
ModuleSpec{Key: "grounded", Stage: StageValidate, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"validated"}},
|
||||||
ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
ModuleSpec{Key: "json", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func llmProfileCatalog(t *testing.T) ModuleCatalog {
|
||||||
|
t.Helper()
|
||||||
|
catalog := newProfileCatalogWithOverrides(t,
|
||||||
|
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked, Provides: []string{"source"}},
|
||||||
|
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||||
|
ModuleSpec{Key: "llm-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||||
|
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||||
|
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||||
|
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||||
|
)
|
||||||
|
if err := RegisterChunkValidator(catalog.Validators, ValidatorSpec{Key: "llm-chunk-validator", ExecutionClass: contracts.ExecutionClassLLMBacked}, func() (contracts.ChunkValidator, error) {
|
||||||
|
return typedTestChunkValidator{key: "llm-chunk-validator"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register chunk validator: %v", err)
|
||||||
|
}
|
||||||
|
registerProfileValidatorSpec(t, catalog, ValidatorSpec{Key: "llm-validator", ExecutionClass: contracts.ExecutionClassLLMBacked})
|
||||||
|
for _, mapping := range []ValidatorChainMapping{
|
||||||
|
{Stage: StageChunk, Module: "llm-chunk", Validators: []ModuleBinding{Binding("llm-chunk-validator")}},
|
||||||
|
{Stage: StageExtract, Module: "llm-extractor", Validators: []ModuleBinding{Binding("llm-validator")}},
|
||||||
|
{Stage: StageMerge, Module: "llm-merge", Validators: []ModuleBinding{Binding("llm-validator")}},
|
||||||
|
{Stage: StageNormalize, Module: "llm-normalize", Validators: []ModuleBinding{Binding("llm-validator")}},
|
||||||
|
} {
|
||||||
|
if err := catalog.ValidatorChains.Register(mapping); err != nil {
|
||||||
|
t.Fatalf("register validator chain %#v: %v", mapping, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return catalog
|
||||||
|
}
|
||||||
|
|
||||||
|
func llmProfileCatalogWithoutValidatorChains(t *testing.T) ModuleCatalog {
|
||||||
|
catalog := llmProfileCatalog(t)
|
||||||
|
catalog.ValidatorChains = NewValidatorChainRegistry()
|
||||||
|
return catalog
|
||||||
|
}
|
||||||
|
|
||||||
|
func llmProfilePipeline() PipelineProfile {
|
||||||
|
return PipelineProfile{
|
||||||
|
ID: "llm-profile",
|
||||||
|
Input: Binding("llm-input"),
|
||||||
|
Chunk: Binding("llm-chunk"),
|
||||||
|
Output: Binding("llm-output"),
|
||||||
|
Artifacts: map[string]ArtifactLaneProfile{
|
||||||
|
"events": {
|
||||||
|
Extract: Binding("llm-extractor"),
|
||||||
|
Merge: Binding("llm-merge"),
|
||||||
|
Normalize: Binding("llm-normalize"),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func llmProfileValues(profile string) map[string]string {
|
||||||
|
return map[string]string{
|
||||||
|
"input": profile,
|
||||||
|
"chunk": profile,
|
||||||
|
"extract": profile,
|
||||||
|
"merge": profile,
|
||||||
|
"normalize": profile,
|
||||||
|
"output": profile,
|
||||||
|
"validator:chunk:": profile,
|
||||||
|
"validator:extract:events": profile,
|
||||||
|
"validator:merge:events": profile,
|
||||||
|
"validator:normalize:events": profile,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolvedLLMProfileValues(resolved ResolvedPipeline) map[string]string {
|
||||||
|
values := map[string]string{
|
||||||
|
"input": resolved.Input.LLMProfile,
|
||||||
|
"chunk": resolved.Chunk.LLMProfile,
|
||||||
|
"output": resolved.Output.LLMProfile,
|
||||||
|
}
|
||||||
|
lane := resolved.Steps[0].ArtifactLanes[0]
|
||||||
|
values["extract"] = lane.Extract.LLMProfile
|
||||||
|
values["merge"] = lane.Merge.LLMProfile
|
||||||
|
values["normalize"] = lane.Normalize.LLMProfile
|
||||||
|
for _, chain := range resolved.ValidatorChains {
|
||||||
|
for _, validator := range chain.Validators {
|
||||||
|
if validator.ExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
|
values["validator:"+string(chain.Stage)+":"+chain.LaneID] = validator.Binding.LLMProfile
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return values
|
||||||
|
}
|
||||||
|
|
||||||
func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSpec) {
|
func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSpec) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -1566,7 +1887,13 @@ func TestResolvedPipelineCanMarshalToCanonicalJSON(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
if _, err := json.Marshal(resolved); err != nil {
|
encoded, err := json.Marshal(resolved)
|
||||||
|
if err != nil {
|
||||||
t.Fatalf("json.Marshal(resolved) error = %v, want nil", err)
|
t.Fatalf("json.Marshal(resolved) error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
for _, field := range []string{"input_execution_class", "chunk_execution_class", "extract_execution_class", "merge_execution_class", "normalize_execution_class", "output_execution_class"} {
|
||||||
|
if !strings.Contains(string(encoded), `"`+field+`":"deterministic"`) {
|
||||||
|
t.Fatalf("resolved JSON does not retain %q: %s", field, encoded)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -189,12 +189,12 @@ func TestMaterializeReferencesUsesLaneArtifactVariant(t *testing.T) {
|
|||||||
{kind: "test/beta", mergeSlot: "beta_merge", normalizeSlot: "beta_normalize"},
|
{kind: "test/beta", mergeSlot: "beta_merge", normalizeSlot: "beta_normalize"},
|
||||||
{kind: "test/alpha", mergeSlot: "alpha_merge", normalizeSlot: "alpha_normalize"},
|
{kind: "test/alpha", mergeSlot: "alpha_merge", normalizeSlot: "alpha_normalize"},
|
||||||
} {
|
} {
|
||||||
if err := RegisterMerger(mergers, ModuleSpec{Key: "shared/merge", Stage: StageMerge, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.mergeSlot}}}, func() (contracts.Merger[codecNotes], error) {
|
if err := RegisterMerger(mergers, ModuleSpec{Key: "shared/merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.mergeSlot}}}, func() (contracts.Merger[codecNotes], error) {
|
||||||
return typedTestMerger[codecNotes]{key: "shared/merge"}, nil
|
return typedTestMerger[codecNotes]{key: "shared/merge"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterMerger(%s): %v", item.kind, err)
|
t.Fatalf("RegisterMerger(%s): %v", item.kind, err)
|
||||||
}
|
}
|
||||||
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "shared/normalize", Stage: StageNormalize, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.normalizeSlot}}}, func() (contracts.Normalizer[codecNotes], error) {
|
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "shared/normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: item.kind, ReferenceSlots: []contracts.ReferenceSlot{{Name: item.normalizeSlot}}}, func() (contracts.Normalizer[codecNotes], error) {
|
||||||
return typedTestNormalizer[codecNotes]{key: "shared/normalize"}, nil
|
return typedTestNormalizer[codecNotes]{key: "shared/normalize"}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterNormalizer(%s): %v", item.kind, err)
|
t.Fatalf("RegisterNormalizer(%s): %v", item.kind, err)
|
||||||
@@ -531,6 +531,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
|||||||
ModuleSpec{
|
ModuleSpec{
|
||||||
Key: "generic",
|
Key: "generic",
|
||||||
Stage: StageChunk,
|
Stage: StageChunk,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"source"},
|
Requires: []string{"source"},
|
||||||
Provides: []string{"chunk"},
|
Provides: []string{"chunk"},
|
||||||
ReferenceSlots: chunkSlots,
|
ReferenceSlots: chunkSlots,
|
||||||
@@ -538,6 +539,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
|||||||
ModuleSpec{
|
ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"chunk"},
|
Requires: []string{"chunk"},
|
||||||
Provides: []string{"candidate"},
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: extractSlots,
|
ReferenceSlots: extractSlots,
|
||||||
@@ -545,6 +547,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
|||||||
ModuleSpec{
|
ModuleSpec{
|
||||||
Key: "appendorder",
|
Key: "appendorder",
|
||||||
Stage: StageMerge,
|
Stage: StageMerge,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"candidate"},
|
Requires: []string{"candidate"},
|
||||||
Provides: []string{"merged"},
|
Provides: []string{"merged"},
|
||||||
ReferenceSlots: mergeSlots,
|
ReferenceSlots: mergeSlots,
|
||||||
@@ -552,6 +555,7 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
|||||||
ModuleSpec{
|
ModuleSpec{
|
||||||
Key: "noop",
|
Key: "noop",
|
||||||
Stage: StageNormalize,
|
Stage: StageNormalize,
|
||||||
|
ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||||
Requires: []string{"merged"},
|
Requires: []string{"merged"},
|
||||||
Provides: []string{"normalized"},
|
Provides: []string{"normalized"},
|
||||||
ReferenceSlots: normalizeSlots,
|
ReferenceSlots: normalizeSlots,
|
||||||
|
|||||||
@@ -24,6 +24,7 @@ type Registries struct {
|
|||||||
Inputs *InputAdapterRegistry
|
Inputs *InputAdapterRegistry
|
||||||
Chunkers *ChunkerRegistry
|
Chunkers *ChunkerRegistry
|
||||||
ArtifactCodecs *ArtifactCodecRegistry
|
ArtifactCodecs *ArtifactCodecRegistry
|
||||||
|
ArtifactEvidence *ArtifactEvidenceRegistry
|
||||||
Extractors *ExtractorRegistry
|
Extractors *ExtractorRegistry
|
||||||
Mergers *MergerRegistry
|
Mergers *MergerRegistry
|
||||||
Normalizers *NormalizerRegistry
|
Normalizers *NormalizerRegistry
|
||||||
@@ -293,6 +294,20 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
|||||||
return failOutput(output), err
|
return failOutput(output), err
|
||||||
}
|
}
|
||||||
outputStarted := time.Now().UTC()
|
outputStarted := time.Now().UTC()
|
||||||
|
evidenceArtifact, evidenceSummary, err := buildOutputEvidenceContext(input.Prepared, doc, output.NormalizeOutputs)
|
||||||
|
if err != nil {
|
||||||
|
return failOutput(output), err
|
||||||
|
}
|
||||||
|
if evidenceSummary != nil {
|
||||||
|
if err := writeDebugTimed(debugRecorder, "output/evidence-context.json", debugTimedEnvelope{
|
||||||
|
Stage: string(StageOutput),
|
||||||
|
ModuleKey: encoder.Key(),
|
||||||
|
StartedAt: outputStarted,
|
||||||
|
Payload: *evidenceSummary,
|
||||||
|
}); err != nil {
|
||||||
|
return failOutput(output), fmt.Errorf("write evidence context debug artifact: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
outputDebugPayload := map[string]any{
|
outputDebugPayload := map[string]any{
|
||||||
"manifest": output.Manifest,
|
"manifest": output.Manifest,
|
||||||
"normalize_outputs": debugSerializedOutputEnvelopes(output.NormalizeOutputs),
|
"normalize_outputs": debugSerializedOutputEnvelopes(output.NormalizeOutputs),
|
||||||
@@ -324,6 +339,7 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
|||||||
LLMProfile: input.pipeline.Output.LLMProfile,
|
LLMProfile: input.pipeline.Output.LLMProfile,
|
||||||
Metadata: outputMetadata,
|
Metadata: outputMetadata,
|
||||||
ChunkMap: contracts.CloneSerializedArtifactPointer(acceptedChunkMap),
|
ChunkMap: contracts.CloneSerializedArtifactPointer(acceptedChunkMap),
|
||||||
|
EvidenceContext: contracts.CloneSerializedArtifactPointer(evidenceArtifact),
|
||||||
})
|
})
|
||||||
output.Warnings = append(output.Warnings, encoded.Warnings...)
|
output.Warnings = append(output.Warnings, encoded.Warnings...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -813,18 +829,12 @@ func mergeLLMProfileManifests(sources ...[]artifacts.LLMProfileManifest) []artif
|
|||||||
merged := make(map[string]artifacts.LLMProfileManifest)
|
merged := make(map[string]artifacts.LLMProfileManifest)
|
||||||
for _, source := range sources {
|
for _, source := range sources {
|
||||||
for _, profile := range source {
|
for _, profile := range source {
|
||||||
id := strings.TrimSpace(profile.ID)
|
profile = profile.Normalized()
|
||||||
provider := strings.TrimSpace(profile.Provider)
|
key := profile.IdentityKey()
|
||||||
model := strings.TrimSpace(profile.Model)
|
|
||||||
key := id + "\x00" + provider + "\x00" + model
|
|
||||||
if _, exists := merged[key]; exists {
|
if _, exists := merged[key]; exists {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
merged[key] = artifacts.LLMProfileManifest{
|
merged[key] = profile
|
||||||
ID: id,
|
|
||||||
Provider: provider,
|
|
||||||
Model: model,
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if len(merged) == 0 {
|
if len(merged) == 0 {
|
||||||
|
|||||||
@@ -116,7 +116,7 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
|
|||||||
return false, nil, terminal.record(nil, fmt.Errorf("clone chunker manifest metadata: %w", metadataErr))
|
return false, nil, terminal.record(nil, fmt.Errorf("clone chunker manifest metadata: %w", metadataErr))
|
||||||
}
|
}
|
||||||
profile := ""
|
profile := ""
|
||||||
if provider, ok := chunker.(contracts.ChunkExecutionClassProvider); ok && provider.ExecutionClass() == contracts.ExecutionClassLLMBacked {
|
if input.pipeline.ChunkExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||||
profile = input.pipeline.Chunk.LLMProfile
|
profile = input.pipeline.Chunk.LLMProfile
|
||||||
}
|
}
|
||||||
candidate := ChunkPlanRecord{
|
candidate := ChunkPlanRecord{
|
||||||
|
|||||||
@@ -67,30 +67,17 @@ type manifestChunker struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (c manifestChunker) ManifestMetadata() map[string]any { return c.metadata }
|
func (c manifestChunker) ManifestMetadata() map[string]any { return c.metadata }
|
||||||
func (manifestChunker) ExecutionClass() contracts.ExecutionClass {
|
|
||||||
return contracts.ExecutionClassLLMBacked
|
|
||||||
}
|
|
||||||
|
|
||||||
type llmCountingChunker struct {
|
type llmCountingChunker struct {
|
||||||
terminalChunker
|
terminalChunker
|
||||||
llmCalls *int
|
llmCalls *int
|
||||||
}
|
}
|
||||||
|
|
||||||
func (llmCountingChunker) ExecutionClass() contracts.ExecutionClass {
|
|
||||||
return contracts.ExecutionClassLLMBacked
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c llmCountingChunker) Plan(ctx context.Context, request contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
|
func (c llmCountingChunker) Plan(ctx context.Context, request contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
|
||||||
(*c.llmCalls)++
|
(*c.llmCalls)++
|
||||||
return c.terminalChunker.Plan(ctx, request)
|
return c.terminalChunker.Plan(ctx, request)
|
||||||
}
|
}
|
||||||
|
|
||||||
type deterministicChunker struct{ terminalChunker }
|
|
||||||
|
|
||||||
func (deterministicChunker) ExecutionClass() contracts.ExecutionClass {
|
|
||||||
return contracts.ExecutionClassDeterministic
|
|
||||||
}
|
|
||||||
|
|
||||||
type retryingChunker struct {
|
type retryingChunker struct {
|
||||||
key string
|
key string
|
||||||
plan source.ChunkPlan
|
plan source.ChunkPlan
|
||||||
@@ -535,6 +522,7 @@ func TestRunnerPublishesOnlyAcceptedGeneratedPlans(t *testing.T) {
|
|||||||
func TestRunnerStoresProducerProvenanceAndProducerWarnings(t *testing.T) {
|
func TestRunnerStoresProducerProvenanceAndProducerWarnings(t *testing.T) {
|
||||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||||
prepared.resolved.Chunk.LLMProfile = "chunk-profile"
|
prepared.resolved.Chunk.LLMProfile = "chunk-profile"
|
||||||
|
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
|
||||||
prepared.resolved.ChunkReferences = ResolvedReferenceTarget{
|
prepared.resolved.ChunkReferences = ResolvedReferenceTarget{
|
||||||
Stage: StageChunk,
|
Stage: StageChunk,
|
||||||
ReferenceSet: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
ReferenceSet: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||||
@@ -577,7 +565,8 @@ func TestRunnerRejectsUncloneableModuleManifestMetadata(t *testing.T) {
|
|||||||
func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
|
func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
|
||||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||||
prepared.resolved.Chunk.LLMProfile = "configured-but-unused"
|
prepared.resolved.Chunk.LLMProfile = "configured-but-unused"
|
||||||
prepared.chunker = deterministicChunker{terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}}
|
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassDeterministic
|
||||||
|
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}
|
||||||
store := &recordingChunkPlanStore{}
|
store := &recordingChunkPlanStore{}
|
||||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store})
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -588,6 +577,22 @@ func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunnerDoesNotInventProducerProfileForLLMChunker(t *testing.T) {
|
||||||
|
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||||
|
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
|
||||||
|
prepared.resolved.Chunk.LLMProfile = ""
|
||||||
|
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}
|
||||||
|
store := &recordingChunkPlanStore{}
|
||||||
|
|
||||||
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if store.saved.Producer.LLMProfile != "" || output.Manifest.ChunkPlan.ProducerLLMProfile != "" {
|
||||||
|
t.Fatalf("LLM producer profile = stored %q manifest %q, want empty", store.saved.Producer.LLMProfile, output.Manifest.ChunkPlan.ProducerLLMProfile)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunnerRefreshChangesDownstreamChunkFingerprint(t *testing.T) {
|
func TestRunnerRefreshChangesDownstreamChunkFingerprint(t *testing.T) {
|
||||||
doc := typedTestDocumentWithUnits(2)
|
doc := typedTestDocumentWithUnits(2)
|
||||||
prepared := preparedConcurrentPipeline(t, 1)
|
prepared := preparedConcurrentPipeline(t, 1)
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user