Compare commits

...

8 Commits

59 changed files with 3401 additions and 88 deletions

View File

@@ -20,7 +20,7 @@ a bearer token.
```text
notarius help
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b]
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--reference slot=path] [--without-reference slot]
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list --config path/to/config.yml [--json]
```
@@ -46,11 +46,59 @@ Flags:
invocation.
- `--llm-profile id`: override every effective module binding to use one LLM
profile.
- `--reference slot=path`: bind a reference path to an extractor reference
slot. Repeatable. Use `lane.slot=path` when multiple selected lanes declare
the same slot.
- `--without-reference slot`: remove a configured optional reference binding.
Repeatable. Use `lane.slot` when multiple selected lanes declare the same
slot.
On success, the command prints the completed pipeline ID, approved and rejected
artifact counts, and the output directory. If the run completes with warnings,
the warning count is printed to stderr.
Reference flags are resolved against selected artifact lanes before the run
starts. Flat slot names are accepted only when exactly one selected lane
declares that slot. Bound reference files are read before extraction, validated
as UTF-8 text, and passed only to the lane extractor that declares the slot.
Reference content is not written to diagnostics, logs, errors, or manifests.
Reference binding precedence is:
1. pipeline-level config `references`;
2. lane-level config `references`;
3. `--reference` run flags;
4. `--without-reference` run flags.
`--reference` binds or replaces one slot for one selected lane. Use
`slot=path` when the selected lanes declare the slot unambiguously:
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-spells.config.yml \
--input examples/seriatim-minimal-transcript.json \
--reference roster=./campaign-roster.txt
```
Use `lane.slot=path` when multiple selected lanes declare the same slot or when
you want to target a specific lane:
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-spells.config.yml \
--input examples/seriatim-minimal-transcript.json \
--reference spells.glossary=./campaign-glossary.txt
```
Use `--without-reference` to remove a configured optional binding for a run:
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-spells.config.yml \
--input examples/seriatim-minimal-transcript.json \
--without-reference glossary
```
For durable output, diagnostics, retention, and failure inspection, see
[Operations](operations.md).

View File

@@ -27,6 +27,9 @@ llm_profiles:
pipelines:
dnd-session:
input: seriatim
references:
roster: ./dnd-spells-roster.txt
glossary: ./dnd-spells-glossary.txt
chunk:
module: generic
options:
@@ -121,6 +124,9 @@ Pipeline fields:
- `artifacts`: required for pipeline resolution. It maps artifact lane IDs to
lane definitions.
- `output`: optional module binding. Default module is `json`.
- `references`: optional map of extractor reference slot names to reference
paths. These bindings are defaults for artifact lanes whose extractor declares
the matching slot.
Artifact lane fields:
@@ -129,11 +135,52 @@ Artifact lane fields:
- `normalize`: optional module binding. Default module is `noop`.
- `validators`: optional list of module bindings. The production CLI currently
does not register validator modules.
- `references`: optional map of extractor reference slot names to reference
paths. Lane bindings override pipeline-level bindings for the same slot.
`notarius run` and `notarius config validate --pipeline` resolve the pipeline
against the production module catalog and fail fast for unknown or incompatible
module keys.
Reference bindings are validated against extractor-declared slots during
pipeline resolution. Required slots must be bound after config defaults,
lane-level bindings, and run-time `--reference` or `--without-reference`
overrides are applied. Config-relative paths are resolved relative to the
config file; CLI reference paths are resolved relative to the current working
directory. Bound files must be UTF-8 text and are passed only to lane
extractors that declare the slot. Reference content is not written to
diagnostics, logs, errors, or manifests.
Pipeline-level `references` are defaults. They apply only to selected lanes
whose extractor declares the slot:
```yaml
pipelines:
dnd-session:
input: seriatim
references:
roster: ./campaign/party-roster.txt
glossary: ./campaign/glossary.txt
artifacts:
spells:
extract: dnd/spells
```
Lane-level `references` override or add bindings for one lane:
```yaml
pipelines:
dnd-session:
input: seriatim
references:
glossary: ./campaign/glossary.txt
artifacts:
spells:
extract: dnd/spells
references:
roster: ./campaign/session-roster.txt
```
## Module Bindings
Every module binding may use shorthand:
@@ -182,6 +229,14 @@ The `generic` chunker accepts:
The `dnd/scenes` chunker requires transcript source capabilities, calls the
configured structured LLM provider, and does not accept module options.
The `dnd/spells` extractor declares optional text reference slots:
- `roster`
- `glossary`
The extractor uses these references only as supporting disambiguation material;
spell casts still must be present in the source transcript.
## Diagnostics
`diagnostics` fields:
@@ -215,3 +270,5 @@ Pipeline resolution additionally checks:
- required module keys are present;
- module keys are registered for the expected slot;
- module capability requirements are satisfied.
- bound reference slots are declared by selected lane extractors;
- required reference slots are bound for selected lanes.

View File

@@ -17,6 +17,14 @@ The extractor requires source chunks and transcript source capability. It
returns generic artifact candidates that are serialized by the JSON output
module.
The extractor accepts optional UTF-8 text references:
- `roster`: campaign roster or player-character notes.
- `glossary`: campaign glossary or spell/name notes.
References are supporting disambiguation material only. They are not source
evidence and are not addressable through `source_refs`.
## Artifact Envelope
Approved artifacts use the generic artifact envelope documented in
@@ -120,6 +128,11 @@ Rejection reason codes:
- `invalid_source_ref`: at least one source reference fails generic source
reference validation.
Warning reason codes:
- `spell_not_near_source`: the extracted spell name was not found in the cited
source text.
Rejected candidates are written to `rejected.json` by the JSON output module.
## Manifest Metadata

View File

@@ -98,6 +98,30 @@ approved.
Fields with empty values may be omitted by JSON encoding.
`source_digests` contains source document digests only. Bound references are
recorded separately under `references`, which contains provenance only:
lane ID, slot name, origin type and URI, digest, media type, byte size, and
binding source. Reference content is not written to durable output.
When references are bound, the manifest section has this shape:
```json
{
"references": [
{
"lane_id": "spells",
"slot_name": "roster",
"origin_type": "file",
"origin_uri": "file:///absolute/path/roster.txt",
"digest": "sha256:...",
"media_type": "text/plain; charset=utf-8",
"size_bytes": 123,
"binding_source": "config"
}
]
}
```
`module_metadata` is omitted when no singleton module provides metadata.
`validation_status` is `approved` when no candidates were rejected and

View File

@@ -33,6 +33,7 @@ Implemented artifact names:
- `invocation.json`
- `effective-config.json`
- `resolved-pipeline.json`
- `resolved-references.json`
- `source-document.json`
- `run-manifest.json`
- `run-report.json`

View File

@@ -20,6 +20,22 @@ A production module package should provide:
Module specs should describe capabilities accurately. Resolution uses specs to
reject incompatible pipelines before execution.
Extractor modules that accept auxiliary reference material must declare slots
through both `ReferenceSlots()` and `ModuleSpec().ReferenceSlots`. The runtime
slot list and registry metadata should match so config validation can inspect
slots without constructing extractor instances. A slot declaration names the
slot, whether it is required, accepted media types, whether multiple items are
allowed, and any byte limit.
Reference content is delivered only to the lane extractor through
`contracts.ExtractionRequest.References`. It is not source evidence and must not
be converted into `SourceRef` values. If a module prompt uses references, load
the prompt bundle with the same declared slots and render with
`RenderUserSystemWithReferences`. Prompt templates may use the `reference`
function for content and the `hasreference` function for conditional sections.
Prompt metadata hashes remain based on template source, not rendered reference
bytes.
Chunk modules receive the structured LLM client through `contracts.ChunkRequest`
when they need model-backed chunking. The pipeline runner validates generic
chunk result invariants before extraction; module-owned policies may be stricter
@@ -124,12 +140,18 @@ metadata under `artifact_lanes[].metadata.extractor`. Durable artifact payload
details belong in the
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
The extractor declares optional `roster` and `glossary` reference slots accepting
UTF-8 text. Its prompt frames references as supporting disambiguation material
only; spell-cast artifacts must still be grounded in the source transcript.
## D&D Spell Validators
The spell extractor returns two built-in validators:
- `dnd/spells/shape`: rejects malformed payloads and missing required fields.
- `dnd/spells/source_refs`: rejects candidates without valid source references.
It also emits a warning when the extracted spell name is not found in the
cited source text.
Reason codes include:
@@ -137,6 +159,7 @@ Reason codes include:
- `missing_required_field`
- `missing_source_ref`
- `invalid_source_ref`
- `spell_not_near_source`
These validators are supplied by the extractor when no validators are configured
for the lane.

View File

@@ -30,6 +30,33 @@ before execution:
The CLI writes the resolved pipeline and digest to diagnostics.
Pipeline profiles and artifact lanes may include reference binding maps keyed by
extractor reference slot name. During resolution, pipeline-level bindings act as
defaults for selected lanes whose extractor declares the slot, lane-level
bindings override or add lane bindings, runtime `--reference` requests override
config bindings, and runtime unbinds remove optional bindings. Flat runtime slot
names are resolved only when exactly one selected lane declares the slot;
otherwise the CLI requires `lane.slot`. Resolution validates bindings against
extractor specs and records lane-scoped binding metadata. It does not read
reference files or include reference bytes in source digests.
During run preparation, resolved file references are materialized before any
LLM-backed pipeline work. Config bindings resolve relative to the config file,
CLI bindings resolve relative to the current working directory, and materialized
reference content is passed only to the matching lane extractor through
`ExtractionRequest`. Materialization accepts UTF-8 text files, computes
`sha256:` content digests, records file origins, enforces declared byte limits,
and warns for empty bound files. Reference content is omitted from diagnostics
and manifests. The CLI writes provenance-only resolved reference diagnostics,
and the run manifest records lane-scoped reference provenance separately from
source digests.
Prompt bundles can declare reference slots and use `reference` and
`hasreference` template functions. Bundle loading validates string-literal slot
names against the declaration. Rendering receives a lane reference set from the
caller; unbound optional slots render as empty strings, and `hasreference`
returns true only when at least one bound item has content.
## Registries And Module Specs
`pipeline.Registries` holds concrete constructors for execution. A
@@ -44,6 +71,10 @@ Every production module registers a `ModuleSpec` with:
- `Provides`: capabilities added after that module runs;
- `Requires`: capabilities that must already be available.
Extractor specs may also declare reference slots. Slot declarations are
available from registry metadata without constructing extractor instances.
Non-extractor module specs must not declare reference slots.
Capability checks prevent incompatible pipeline composition before a run starts.
## Runner Input And Output
@@ -152,7 +183,7 @@ On successful execution, the manifest validation status is:
The manifest records run ID, pipeline ID, pipeline digest, module keys, top-level
module metadata, artifact lanes, LLM profile metadata, source digest,
validation status, and timing.
reference provenance, validation status, and timing.
Singleton pipeline modules may add non-secret metadata by implementing
`contracts.ManifestMetadataProvider`. The runner records that metadata under

View File

@@ -35,7 +35,8 @@ The `json` output module writes these files:
- `index.json`: file index with paths to the manifest, artifact files,
rejected artifacts, and warnings.
- `manifest.json`: run manifest with resolved pipeline provenance, top-level
module metadata, module keys, validation status, and timing.
module metadata, module keys, reference provenance, validation status, and
timing.
- `artifacts/<artifact-type>.json`: approved artifacts grouped by artifact
type. For the current D&D spell extractor, this includes
`artifacts/dnd.spell_cast.json` when spell-cast artifacts are approved.
@@ -62,6 +63,9 @@ Implemented diagnostics artifacts:
path, selected lanes, run ID, and pipeline digest when available.
- `effective-config.json`: resolved config with API keys redacted.
- `resolved-pipeline.json`: resolved module bindings and pipeline digest.
- `resolved-references.json`: lane-scoped resolved reference provenance,
including origin, digest, media type, byte size, and binding source, without
reference content.
- `run-manifest.json`: the same run manifest written to durable output when it
is available, including top-level module metadata when present.
- `warnings.json`: warning list.
@@ -107,6 +111,10 @@ stderr, and writes warnings to durable output and diagnostics when retained.
The run manifest `validation_status` indicates whether final artifacts were
approved or rejected after validation.
Reference-related warnings include empty bound reference files and D&D spell
relatedness warnings such as `spell_not_near_source`. Empty references are still
passed to extractors so optional slots can be intentionally blank.
## Cleanup
It is safe to remove specific old run directories after their output and

View File

@@ -102,6 +102,34 @@ go run ./cmd/notarius run dnd-session \
- For `config validate`, include `--pipeline` when using `--only`.
- Confirm the lane ID exists under `pipelines.<id>.artifacts`.
## Reference Binding Failure
Symptoms include:
- `reference slot "..." is not declared`
- `reference slot "..." is declared by multiple selected lanes`
- `required reference slot "..." is not bound`
- `--reference must use slot=path or lane.slot=path`
- `--without-reference must use slot or lane.slot`
- `read "...": no such file`
- `must be UTF-8 text`
- `is ... bytes, limit ...`
Fix:
- Confirm the selected extractor declares the slot. The implemented
`dnd/spells` extractor declares optional `roster` and `glossary` slots.
- Use `lane.slot=path` when more than one selected lane declares the same slot.
- Use `--without-reference slot` to remove optional config bindings; do not pass
an empty `--reference slot=`.
- Check whether a path came from config or CLI. Config paths are relative to
the config file. CLI reference paths are relative to the current working
directory.
- Ensure the file is readable UTF-8 text and within any byte limit declared by
the extractor.
- If diagnostics are retained, inspect `resolved-pipeline.json`,
`resolved-references.json`, and `error.log`.
## Seriatim Input Validation Failure
Symptoms include `seriatim input`, `parse JSON`, `segments must not be empty`,

View File

@@ -0,0 +1,2 @@
Cure Wounds: healing spell cast by touch.
Shield: defensive reaction spell.

View File

@@ -0,0 +1,3 @@
Aria: party cleric and recurring healer.
Borin: fighter ally.
Bandit mage: hostile spellcaster.

View File

@@ -7,6 +7,9 @@ llm_profiles:
pipelines:
dnd-session:
input: seriatim
references:
roster: ./dnd-spells-roster.txt
glossary: ./dnd-spells-glossary.txt
chunk:
module: generic
options:

View File

@@ -25,7 +25,7 @@ const defaultOutputRoot = "./notarius-output"
const usage = `Usage:
notarius help
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b]
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--reference slot=path] [--without-reference slot]
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list --config path/to/config.yml [--json]
`
@@ -95,6 +95,10 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
outputDir := fs.String("output-dir", "", "output directory")
diagnosticsDir := fs.String("diagnostics-dir", "", "diagnostics directory")
llmProfile := fs.String("llm-profile", "", "LLM profile override")
referenceFlags := stringListFlag{}
withoutReferenceFlags := stringListFlag{}
fs.Var(&referenceFlags, "reference", "reference binding, as slot=path or lane.slot=path")
fs.Var(&withoutReferenceFlags, "without-reference", "unbind a reference, as slot or lane.slot")
if err := fs.Parse(reorderRunArgs(args)); err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 2
@@ -121,6 +125,16 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 2
}
referenceRequests, err := parseReferenceFlags(referenceFlags)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 2
}
referenceUnbindRequests, err := parseReferenceUnbindFlags(withoutReferenceFlags)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 2
}
cfg, loadedConfigPath, err := loadConfig(*configPath, opts)
if err != nil {
@@ -155,15 +169,33 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
}
referenceOverrides, referenceUnbinds, err := resolveCLIReferenceRequests(cfg, pipelineID, only, catalog, referenceRequests, referenceUnbindRequests)
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
}
effective, err := cfg.Resolve(config.ResolveInput{
PipelineID: pipelineID,
Only: only,
Catalog: catalog,
LLMProfileOverride: *llmProfile,
ReferenceOverrides: referenceOverrides,
ReferenceUnbinds: referenceUnbinds,
})
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
}
workingDir, err := os.Getwd()
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("resolve working directory: %w", err))
}
materialized, referenceWarnings, err := pipeline.MaterializeReferences(effective.ResolvedPipeline, catalog, pipeline.ReferenceMaterializationOptions{
ConfigPath: loadedConfigPath,
WorkingDir: workingDir,
})
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
}
effective.ResolvedPipeline = materialized
invocation.PipelineDigest = effective.ResolvedPipeline.Digest
if err := runDir.WriteInvocationMetadata(invocation); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics invocation metadata: %w", err))
@@ -174,6 +206,9 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
if err := runDir.WriteResolvedPipeline(effective.ResolvedPipeline); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics resolved pipeline: %w", err))
}
if err := runDir.WriteResolvedReferences(pipeline.ReferenceProvenance(effective.ResolvedPipeline)); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics resolved references: %w", err))
}
profileIDs := effectiveLLMProfileIDs(effective.ResolvedPipeline)
if len(profileIDs) != 1 {
@@ -205,6 +240,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
StartedAt: startedAt,
LLMProfiles: llmProfiles,
Metadata: runMetadata(*outputDir, *diagnosticsDir),
Warnings: referenceWarnings,
})
if err != nil {
if output.Manifest.PipelineID != "" {
@@ -412,7 +448,7 @@ func reorderRunArgs(args []string) []string {
func runFlagTakesValue(arg string) bool {
switch arg {
case "--config", "--input", "--only", "--output-dir", "--diagnostics-dir", "--llm-profile":
case "--config", "--input", "--only", "--output-dir", "--diagnostics-dir", "--llm-profile", "--reference", "--without-reference":
return true
default:
return false
@@ -661,6 +697,255 @@ func parseOnly(raw string) ([]string, error) {
return result, nil
}
type stringListFlag []string
func (flag *stringListFlag) String() string {
if flag == nil {
return ""
}
return strings.Join(*flag, ",")
}
func (flag *stringListFlag) Set(value string) error {
*flag = append(*flag, value)
return nil
}
type cliReferenceRequest struct {
LaneID string
SlotName string
Source string
}
type cliReferenceUnbindRequest struct {
LaneID string
SlotName string
}
func parseReferenceFlags(values []string) ([]cliReferenceRequest, error) {
if len(values) == 0 {
return nil, nil
}
requests := make([]cliReferenceRequest, 0, len(values))
for _, raw := range values {
name, source, ok := strings.Cut(raw, "=")
if !ok {
return nil, fmt.Errorf("--reference must use slot=path or lane.slot=path")
}
if strings.TrimSpace(source) == "" {
return nil, fmt.Errorf("--reference path must not be empty; use --without-reference to unbind")
}
laneID, slotName, err := parseReferenceSelector(name, "--reference")
if err != nil {
return nil, err
}
requests = append(requests, cliReferenceRequest{
LaneID: laneID,
SlotName: slotName,
Source: strings.TrimSpace(source),
})
}
return requests, nil
}
func parseReferenceUnbindFlags(values []string) ([]cliReferenceUnbindRequest, error) {
if len(values) == 0 {
return nil, nil
}
requests := make([]cliReferenceUnbindRequest, 0, len(values))
for _, raw := range values {
if strings.Contains(raw, "=") {
return nil, fmt.Errorf("--without-reference must use slot or lane.slot")
}
laneID, slotName, err := parseReferenceSelector(raw, "--without-reference")
if err != nil {
return nil, err
}
requests = append(requests, cliReferenceUnbindRequest{
LaneID: laneID,
SlotName: slotName,
})
}
return requests, nil
}
func parseReferenceSelector(raw string, flagName string) (string, string, error) {
selector := strings.TrimSpace(raw)
if selector == "" {
return "", "", fmt.Errorf("%s reference slot must not be empty", flagName)
}
if strings.Count(selector, ".") > 1 {
return "", "", fmt.Errorf("%s must use slot or lane.slot", flagName)
}
laneID := ""
slotName := selector
if strings.Contains(selector, ".") {
before, after, _ := strings.Cut(selector, ".")
laneID = strings.TrimSpace(before)
slotName = strings.TrimSpace(after)
if laneID == "" || slotName == "" {
return "", "", fmt.Errorf("%s must use non-empty lane.slot values", flagName)
}
}
return laneID, slotName, nil
}
func resolveCLIReferenceRequests(
cfg config.Config,
pipelineID string,
only []string,
catalog pipeline.ModuleCatalog,
referenceRequests []cliReferenceRequest,
unbindRequests []cliReferenceUnbindRequest,
) ([]pipeline.ReferenceBinding, []pipeline.ReferenceUnbind, error) {
if len(referenceRequests) == 0 && len(unbindRequests) == 0 {
return nil, nil, nil
}
selected, err := selectedReferenceLanes(cfg, pipelineID, only, catalog)
if err != nil {
return nil, nil, err
}
overrides := make([]pipeline.ReferenceBinding, 0, len(referenceRequests))
for _, request := range referenceRequests {
laneID, err := resolveCLIReferenceLane(selected, request.LaneID, request.SlotName)
if err != nil {
return nil, nil, err
}
overrides = append(overrides, pipeline.ReferenceBinding{
LaneID: laneID,
SlotName: request.SlotName,
Source: request.Source,
BindingSource: contracts.ReferenceBindingSourceCLI,
})
}
unbinds := make([]pipeline.ReferenceUnbind, 0, len(unbindRequests))
for _, request := range unbindRequests {
laneID, err := resolveCLIReferenceLane(selected, request.LaneID, request.SlotName)
if err != nil {
return nil, nil, err
}
unbinds = append(unbinds, pipeline.ReferenceUnbind{
LaneID: laneID,
SlotName: request.SlotName,
})
}
return overrides, unbinds, nil
}
type selectedReferenceLane struct {
id string
slots map[string]struct{}
}
func selectedReferenceLanes(cfg config.Config, pipelineID string, only []string, catalog pipeline.ModuleCatalog) ([]selectedReferenceLane, error) {
profile, ok := lookupCLIReferencePipeline(cfg.Pipelines, pipelineID)
if !ok {
return nil, fmt.Errorf("pipeline %q is not configured", strings.TrimSpace(pipelineID))
}
lanesByID := make(map[string]pipeline.ArtifactLaneProfile, len(profile.Artifacts))
for rawLaneID, lane := range profile.Artifacts {
laneID := strings.TrimSpace(rawLaneID)
if laneID == "" {
return nil, fmt.Errorf("pipeline %q artifact lane id must not be empty", strings.TrimSpace(pipelineID))
}
if _, ok := lanesByID[laneID]; ok {
return nil, fmt.Errorf("pipeline %q artifact lane %q is duplicated after trimming", strings.TrimSpace(pipelineID), laneID)
}
lanesByID[laneID] = lane
}
selectedIDs := make([]string, 0, len(lanesByID))
if len(only) == 0 {
for laneID := range lanesByID {
selectedIDs = append(selectedIDs, laneID)
}
} else {
seen := make(map[string]struct{}, len(only))
for _, rawLaneID := range only {
laneID := strings.TrimSpace(rawLaneID)
if laneID == "" {
return nil, fmt.Errorf("pipeline %q selected artifact lane id must not be empty", strings.TrimSpace(pipelineID))
}
if _, ok := lanesByID[laneID]; !ok {
return nil, fmt.Errorf("pipeline %q selected artifact lane %q is not declared", strings.TrimSpace(pipelineID), laneID)
}
if _, ok := seen[laneID]; !ok {
selectedIDs = append(selectedIDs, laneID)
seen[laneID] = struct{}{}
}
}
}
sort.Strings(selectedIDs)
selected := make([]selectedReferenceLane, 0, len(selectedIDs))
for _, laneID := range selectedIDs {
lane := lanesByID[laneID]
extractModule := strings.TrimSpace(lane.Extract.Module)
if extractModule == "" {
return nil, fmt.Errorf("pipeline %q lane %q extract module must not be empty", strings.TrimSpace(pipelineID), laneID)
}
if catalog.Extractors == nil {
return nil, fmt.Errorf("pipeline %q lane %q extract module %q: module %q is not registered", strings.TrimSpace(pipelineID), laneID, extractModule, extractModule)
}
spec, ok := catalog.Extractors.Spec(extractModule)
if !ok {
return nil, fmt.Errorf("pipeline %q lane %q extract module %q: module %q is not registered", strings.TrimSpace(pipelineID), laneID, extractModule, extractModule)
}
slotSet := make(map[string]struct{}, len(spec.ReferenceSlots))
for _, slot := range spec.ReferenceSlots {
slotSet[slot.Name] = struct{}{}
}
selected = append(selected, selectedReferenceLane{id: laneID, slots: slotSet})
}
return selected, nil
}
func lookupCLIReferencePipeline(profiles map[string]pipeline.PipelineProfile, pipelineID string) (pipeline.PipelineProfile, bool) {
pipelineID = strings.TrimSpace(pipelineID)
for rawID, profile := range profiles {
if strings.TrimSpace(rawID) == pipelineID {
return profile, true
}
}
return pipeline.PipelineProfile{}, false
}
func resolveCLIReferenceLane(selected []selectedReferenceLane, requestedLaneID string, slotName string) (string, error) {
slotName = strings.TrimSpace(slotName)
requestedLaneID = strings.TrimSpace(requestedLaneID)
if requestedLaneID != "" {
for _, lane := range selected {
if lane.id == requestedLaneID {
if _, ok := lane.slots[slotName]; !ok {
return "", fmt.Errorf("reference slot %q is not declared by selected lane %q", slotName, requestedLaneID)
}
return requestedLaneID, nil
}
}
return "", fmt.Errorf("reference lane %q is not selected", requestedLaneID)
}
matches := make([]string, 0, 1)
for _, lane := range selected {
if _, ok := lane.slots[slotName]; ok {
matches = append(matches, lane.id)
}
}
switch len(matches) {
case 0:
return "", fmt.Errorf("reference slot %q is not declared by any selected lane", slotName)
case 1:
return matches[0], nil
default:
return "", fmt.Errorf("reference slot %q is declared by multiple selected lanes (%s); use lane.slot", slotName, strings.Join(matches, ", "))
}
}
func sortedPipelineIDs(cfg config.Config) []string {
ids := make([]string, 0, len(cfg.Pipelines))
for id := range cfg.Pipelines {

View File

@@ -8,12 +8,14 @@ import (
"os"
"path/filepath"
"reflect"
"sort"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/dnd/scenes"
@@ -737,6 +739,219 @@ func TestRunPipelineLLMProfileOverrideSelectsFactoryProfile(t *testing.T) {
}
}
func TestRunPipelineReferenceFlagBindsUnambiguousSlot(t *testing.T) {
configPath := writeTestConfig(t, testConfigYAML("example", "events"))
inputPath := filepath.Join(t.TempDir(), "missing.json")
referencePath := writeFile(t, "roster.yml", "Aria\n")
diagnosticsDir := t.TempDir()
var stdout bytes.Buffer
var stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "example",
"--config", configPath,
"--input", inputPath,
"--diagnostics-dir", diagnosticsDir,
"--reference", "roster=" + referencePath,
}, &stdout, &stderr, Options{
Catalog: fakeCatalog(t, pipeline.ModuleSpec{
Key: "fake/extract",
Stage: pipeline.StageExtract,
Requires: []string{"chunks"},
Provides: []string{"artifact"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster"},
},
}),
})
if code != 1 || !strings.Contains(stderr.String(), "read input") {
t.Fatalf("RunWithOptions() code = %d stderr=%q, want read input failure after resolution", code, stderr.String())
}
resolved := readResolvedPipeline(t, diagnosticsDir)
refs := resolved.ArtifactLanes[0].References
want := []pipeline.ReferenceBinding{
{LaneID: "events", SlotName: "roster", Source: referencePath, BindingSource: contracts.ReferenceBindingSourceCLI},
}
if !reflect.DeepEqual(refs, want) {
t.Fatalf("resolved references = %#v, want %#v", refs, want)
}
}
func TestRunPipelineReferenceFlagBindsLaneQualifiedSlot(t *testing.T) {
configPath := writeTestConfig(t, testConfigYAML("example", "events", "notes"))
inputPath := filepath.Join(t.TempDir(), "missing.json")
referencePath := writeFile(t, "notes.yml", "Notes\n")
diagnosticsDir := t.TempDir()
var stdout bytes.Buffer
var stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "example",
"--config", configPath,
"--input", inputPath,
"--only", "events,notes",
"--diagnostics-dir", diagnosticsDir,
"--reference", "notes.roster=" + referencePath,
}, &stdout, &stderr, Options{
Catalog: fakeCatalog(t, pipeline.ModuleSpec{
Key: "fake/extract",
Stage: pipeline.StageExtract,
Requires: []string{"chunks"},
Provides: []string{"artifact"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster"},
},
}),
})
if code != 1 || !strings.Contains(stderr.String(), "read input") {
t.Fatalf("RunWithOptions() code = %d stderr=%q, want read input failure after resolution", code, stderr.String())
}
resolved := readResolvedPipeline(t, diagnosticsDir)
events := resolvedArtifactLane(t, resolved, "events")
if len(events.References) != 0 {
t.Fatalf("events references = %#v, want none", events.References)
}
notes := resolvedArtifactLane(t, resolved, "notes")
if len(notes.References) != 1 || notes.References[0].Source != referencePath {
t.Fatalf("notes references = %#v, want lane-qualified binding", notes.References)
}
}
func TestRunPipelineReferenceFlagRejectsAmbiguousFlatSlot(t *testing.T) {
configPath := writeTestConfig(t, testConfigYAML("example", "events", "notes"))
inputPath := writeSeriatimInput(t)
diagnosticsDir := t.TempDir()
var stdout bytes.Buffer
var stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "example",
"--config", configPath,
"--input", inputPath,
"--diagnostics-dir", diagnosticsDir,
"--reference", "roster=./roster.yml",
}, &stdout, &stderr, Options{
Catalog: fakeCatalog(t, pipeline.ModuleSpec{
Key: "fake/extract",
Stage: pipeline.StageExtract,
Requires: []string{"chunks"},
Provides: []string{"artifact"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster"},
},
}),
})
if code != 1 {
t.Fatalf("RunWithOptions() code = %d, want 1", code)
}
if !strings.Contains(stderr.String(), "multiple selected lanes") || !strings.Contains(stderr.String(), "lane.slot") {
t.Fatalf("stderr = %q, want ambiguous reference error", stderr.String())
}
}
func TestRunPipelineReferenceFlagsRejectMalformedValues(t *testing.T) {
tests := []struct {
name string
args []string
want string
}{
{name: "missing equals", args: []string{"--reference", "roster"}, want: "slot=path"},
{name: "empty path", args: []string{"--reference", "roster="}, want: "path must not be empty"},
{name: "empty slot", args: []string{"--reference", "=./roster.yml"}, want: "slot must not be empty"},
{name: "too many selector parts", args: []string{"--reference", "a.b.c=./roster.yml"}, want: "lane.slot"},
{name: "unbind with equals", args: []string{"--without-reference", "roster=./roster.yml"}, want: "slot or lane.slot"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
configPath := writeTestConfig(t, testConfigYAML("example", "events"))
inputPath := writeSeriatimInput(t)
var stdout bytes.Buffer
var stderr bytes.Buffer
args := []string{"run", "example", "--config", configPath, "--input", inputPath}
args = append(args, test.args...)
code := RunWithOptions(args, &stdout, &stderr, Options{Catalog: fakeCatalog(t)})
if code != 2 {
t.Fatalf("RunWithOptions() code = %d, want 2", code)
}
if !strings.Contains(stderr.String(), test.want) {
t.Fatalf("stderr = %q, want substring %q", stderr.String(), test.want)
}
})
}
}
func TestRunPipelineWithoutReferenceRemovesOptionalConfigBinding(t *testing.T) {
configPath := writeTestConfig(t, testConfigYAMLWithReferences("example", "events", map[string]string{"roster": "./config-roster.yml"}))
inputPath := filepath.Join(t.TempDir(), "missing.json")
diagnosticsDir := t.TempDir()
var stdout bytes.Buffer
var stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "example",
"--config", configPath,
"--input", inputPath,
"--diagnostics-dir", diagnosticsDir,
"--without-reference", "roster",
}, &stdout, &stderr, Options{
Catalog: fakeCatalog(t, pipeline.ModuleSpec{
Key: "fake/extract",
Stage: pipeline.StageExtract,
Requires: []string{"chunks"},
Provides: []string{"artifact"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster"},
},
}),
})
if code != 1 || !strings.Contains(stderr.String(), "read input") {
t.Fatalf("RunWithOptions() code = %d stderr=%q, want read input failure after resolution", code, stderr.String())
}
resolved := readResolvedPipeline(t, diagnosticsDir)
if refs := resolved.ArtifactLanes[0].References; len(refs) != 0 {
t.Fatalf("references = %#v, want unbound optional slot", refs)
}
}
func TestRunPipelineWithoutReferenceFailsWhenRequiredSlotWouldBeMissing(t *testing.T) {
configPath := writeTestConfig(t, testConfigYAMLWithReferences("example", "events", map[string]string{"roster": "./config-roster.yml"}))
inputPath := writeSeriatimInput(t)
diagnosticsDir := t.TempDir()
var stdout bytes.Buffer
var stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "example",
"--config", configPath,
"--input", inputPath,
"--diagnostics-dir", diagnosticsDir,
"--without-reference", "roster",
}, &stdout, &stderr, Options{
Catalog: fakeCatalog(t, pipeline.ModuleSpec{
Key: "fake/extract",
Stage: pipeline.StageExtract,
Requires: []string{"chunks"},
Provides: []string{"artifact"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster", Required: true},
},
}),
})
if code != 1 {
t.Fatalf("RunWithOptions() code = %d, want 1", code)
}
if !strings.Contains(stderr.String(), "required reference slot") || !strings.Contains(stderr.String(), "roster") {
t.Fatalf("stderr = %q, want required reference error", stderr.String())
}
}
func TestRunPipelineWritesDurableOutputFiles(t *testing.T) {
diagnosticsDir := t.TempDir()
outputDir := t.TempDir()
@@ -770,6 +985,74 @@ func TestRunPipelineWritesDurableOutputFiles(t *testing.T) {
assertNoTemporaryFiles(t, runOutputDir)
}
func TestRunPipelineReferenceBytesProduceDistinctManifests(t *testing.T) {
run := func(t *testing.T, referenceText string) artifacts.RunManifest {
t.Helper()
diagnosticsDir := t.TempDir()
outputDir := t.TempDir()
configDir := t.TempDir()
referencePath := filepath.Join(configDir, "roster.txt")
if err := os.WriteFile(referencePath, []byte(referenceText), 0o644); err != nil {
t.Fatalf("write reference: %v", err)
}
configPath := filepath.Join(configDir, "config.yml")
if err := os.WriteFile(configPath, []byte(testConfigYAMLWithReferencesAndDiagnostics("example", "events", diagnosticsDir, map[string]string{"roster": "roster.txt"})), 0o644); err != nil {
t.Fatalf("write config: %v", err)
}
inputPath := writeFile(t, "source.txt", "source text")
var stdout bytes.Buffer
var stderr bytes.Buffer
code := RunWithOptions([]string{"run", "example", "--config", configPath, "--input", inputPath, "--output-dir", outputDir}, &stdout, &stderr, Options{
Catalog: fakeCatalog(t, pipeline.ModuleSpec{
Key: "fake/extract",
Stage: pipeline.StageExtract,
Requires: []string{"chunks"},
Provides: []string{"artifact"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster"},
},
}),
Registries: fakeExecutionRegistries(t),
LLMClientFactory: fakeLLMFactory(newFakeRunLLMClient(false), nil),
})
if code != 0 {
t.Fatalf("RunWithOptions() code = %d, stderr=%q", code, stderr.String())
}
var manifest artifacts.RunManifest
readJSONFile(t, filepath.Join(onlyChildDir(t, outputDir), "manifest.json"), &manifest)
if len(manifest.References) != 1 {
t.Fatalf("manifest references = %#v, want one entry", manifest.References)
}
if !reflect.DeepEqual(manifest.SourceDigests, []string{"sha256:source"}) {
t.Fatalf("source digests = %#v, want source-only digest", manifest.SourceDigests)
}
var resolvedReferences []artifacts.ReferenceProvenance
readJSONFile(t, filepath.Join(onlyChildDir(t, diagnosticsDir), diagnostics.ArtifactResolvedReferences), &resolvedReferences)
if !reflect.DeepEqual(resolvedReferences, manifest.References) {
t.Fatalf("resolved references = %#v, want manifest references %#v", resolvedReferences, manifest.References)
}
resolvedReferenceJSON := string(readFile(t, filepath.Join(onlyChildDir(t, diagnosticsDir), diagnostics.ArtifactResolvedReferences)))
if strings.Contains(resolvedReferenceJSON, referenceText) || strings.Contains(resolvedReferenceJSON, "content") {
t.Fatalf("resolved references diagnostics contains content: %s", resolvedReferenceJSON)
}
return manifest
}
first := run(t, "first roster")
second := run(t, "second roster")
if first.References[0].Digest == second.References[0].Digest {
t.Fatalf("reference digests match for different bytes: %q", first.References[0].Digest)
}
if first.PipelineDigest != second.PipelineDigest {
t.Fatalf("pipeline digests differ = %q vs %q, want reference bytes outside pipeline identity", first.PipelineDigest, second.PipelineDigest)
}
}
func TestRunPipelineRejectsUnsafeOutputFileName(t *testing.T) {
diagnosticsDir := t.TempDir()
outputDir := t.TempDir()
@@ -1305,6 +1588,51 @@ func testConfigYAMLForPipelines(pipelines map[string][]string) string {
return b.String()
}
func testConfigYAMLWithReferences(pipelineID string, laneID string, references map[string]string) string {
var b strings.Builder
b.WriteString("version: 1\n")
b.WriteString("pipelines:\n")
b.WriteString(" " + pipelineID + ":\n")
b.WriteString(" input: fake/input\n")
b.WriteString(" artifacts:\n")
b.WriteString(" " + laneID + ":\n")
b.WriteString(" extract: fake/extract\n")
b.WriteString(" references:\n")
keys := make([]string, 0, len(references))
for key := range references {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
b.WriteString(" " + key + ": " + references[key] + "\n")
}
return b.String()
}
func testConfigYAMLWithReferencesAndDiagnostics(pipelineID string, laneID string, diagnosticsDir string, references map[string]string) string {
var b strings.Builder
b.WriteString("version: 1\n")
b.WriteString("diagnostics:\n")
b.WriteString(" work_dir: " + diagnosticsDir + "\n")
b.WriteString(" retention: always\n")
b.WriteString("pipelines:\n")
b.WriteString(" " + pipelineID + ":\n")
b.WriteString(" input: fake/input\n")
b.WriteString(" artifacts:\n")
b.WriteString(" " + laneID + ":\n")
b.WriteString(" extract: fake/extract\n")
b.WriteString(" references:\n")
keys := make([]string, 0, len(references))
for key := range references {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
b.WriteString(" " + key + ": " + references[key] + "\n")
}
return b.String()
}
func mvpConfigYAML(pipelineID string, extractor string) string {
return `version: 1
pipelines:
@@ -1553,6 +1881,153 @@ func registriesWithOutput(t *testing.T, encoder contracts.OutputEncoder) pipelin
return registries
}
func fakeExecutionRegistries(t *testing.T) pipeline.Registries {
t.Helper()
inputs := pipeline.NewInputAdapterRegistry()
chunkers := pipeline.NewChunkerRegistry()
extractors := pipeline.NewExtractorRegistry()
mergers := pipeline.NewMergerRegistry()
normalizers := pipeline.NewNormalizerRegistry()
outputs := pipeline.NewOutputEncoderRegistry()
if err := inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "fake/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
return fakeRunInputAdapter{}, nil
}); err != nil {
t.Fatalf("register fake input: %v", err)
}
if err := chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "generic", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}}, func() (contracts.Chunker, error) {
return fakeRunChunker{}, nil
}); err != nil {
t.Fatalf("register fake chunker: %v", err)
}
if err := extractors.RegisterWithSpec(pipeline.ModuleSpec{
Key: "fake/extract",
Stage: pipeline.StageExtract,
Requires: []string{"chunks"},
Provides: []string{"artifact"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster"},
},
}, func() (contracts.Extractor, error) {
return fakeRunExtractor{}, nil
}); err != nil {
t.Fatalf("register fake extractor: %v", err)
}
if err := mergers.RegisterWithSpec(pipeline.ModuleSpec{Key: "appendorder", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}}, func() (contracts.Merger, error) {
return fakeRunMerger{}, nil
}); err != nil {
t.Fatalf("register fake merger: %v", err)
}
if err := normalizers.RegisterWithSpec(pipeline.ModuleSpec{Key: "noop", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer, error) {
return fakeRunNormalizer{}, nil
}); err != nil {
t.Fatalf("register fake normalizer: %v", err)
}
if err := jsonoutput.Register(outputs); err != nil {
t.Fatalf("register json output: %v", err)
}
return pipeline.Registries{
Inputs: inputs,
Chunkers: chunkers,
Extractors: extractors,
Mergers: mergers,
Normalizers: normalizers,
Outputs: outputs,
}
}
type fakeRunInputAdapter struct{}
func (fakeRunInputAdapter) Key() string {
return "fake/input"
}
func (fakeRunInputAdapter) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
return &source.SourceDocument{
ID: "source",
Kind: "text",
Format: "test",
Digest: "sha256:source",
Units: []source.SourceUnit{
{ID: "unit-1", Kind: "text", Text: string(req.Raw)},
},
}, nil
}
type fakeRunChunker struct{}
func (fakeRunChunker) Key() string {
return "generic"
}
func (fakeRunChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
return contracts.ChunkResult{
Chunks: []contracts.SourceChunk{
{ID: "chunk-1", SourceID: req.Source.ID, Index: 0, Units: req.Source.Units},
},
}, nil
}
type fakeRunExtractor struct{}
func (fakeRunExtractor) Key() string {
return "fake/extract"
}
func (fakeRunExtractor) ArtifactType() string {
return "fake.artifact"
}
func (fakeRunExtractor) SchemaVersion() string {
return "v1"
}
func (fakeRunExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return []contracts.ReferenceSlot{{Name: "roster"}}
}
func (fakeRunExtractor) Validators() []contracts.Validator {
return nil
}
func (fakeRunExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
return contracts.ExtractionResult{
Candidates: []artifacts.ArtifactCandidate{
{
Payload: []byte(`{"value":true}`),
SourceRefs: []source.SourceRef{
{SourceID: "source", StartUnitID: "unit-1", EndUnitID: "unit-1"},
},
},
},
}, nil
}
type fakeRunMerger struct{}
func (fakeRunMerger) Key() string {
return "appendorder"
}
func (fakeRunMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
var candidates []artifacts.ArtifactCandidate
for _, chunkArtifacts := range req.ChunkArtifacts {
candidates = append(candidates, chunkArtifacts.Candidates...)
}
return contracts.MergeResult{Candidates: candidates}, nil
}
type fakeRunNormalizer struct{}
func (fakeRunNormalizer) Key() string {
return "noop"
}
func (fakeRunNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
return contracts.NormalizeResult{Candidates: append([]artifacts.ArtifactCandidate(nil), req.Candidates...)}, nil
}
func onlyChildDir(t *testing.T, root string) string {
t.Helper()
children := childDirs(t, root)
@@ -1596,6 +2071,25 @@ func readJSONFile(t *testing.T, path string, out any) {
}
}
func readResolvedPipeline(t *testing.T, diagnosticsDir string) pipeline.ResolvedPipeline {
t.Helper()
runDir := onlyChildDir(t, diagnosticsDir)
var resolved pipeline.ResolvedPipeline
readJSONFile(t, filepath.Join(runDir, diagnostics.ArtifactResolvedPipeline), &resolved)
return resolved
}
func resolvedArtifactLane(t *testing.T, resolved pipeline.ResolvedPipeline, laneID string) pipeline.ResolvedArtifactLane {
t.Helper()
for _, lane := range resolved.ArtifactLanes {
if lane.ID == laneID {
return lane
}
}
t.Fatalf("lane %q not found in resolved pipeline", laneID)
return pipeline.ResolvedArtifactLane{}
}
func assertNoTemporaryFiles(t *testing.T, root string) {
t.Helper()
if err := filepath.WalkDir(root, func(path string, entry os.DirEntry, err error) error {
@@ -1611,7 +2105,7 @@ func assertNoTemporaryFiles(t *testing.T, root string) {
}
}
func fakeCatalog(t *testing.T) pipeline.ModuleCatalog {
func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.ModuleCatalog {
t.Helper()
inputs := pipeline.NewInputAdapterRegistry()
chunkers := pipeline.NewChunkerRegistry()
@@ -1621,12 +2115,24 @@ func fakeCatalog(t *testing.T) pipeline.ModuleCatalog {
validators := pipeline.NewValidatorRegistry()
outputs := pipeline.NewOutputEncoderRegistry()
mustRegisterInput(t, inputs, pipeline.ModuleSpec{Key: "fake/input", Stage: pipeline.StageInput, Provides: []string{"source"}})
mustRegisterChunker(t, chunkers, pipeline.ModuleSpec{Key: "generic", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}})
mustRegisterExtractor(t, extractors, pipeline.ModuleSpec{Key: "fake/extract", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}})
mustRegisterMerger(t, mergers, pipeline.ModuleSpec{Key: "appendorder", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}})
mustRegisterNormalizer(t, normalizers, pipeline.ModuleSpec{Key: "noop", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}})
mustRegisterOutput(t, outputs, pipeline.ModuleSpec{Key: "json", Stage: pipeline.StageOutput, Requires: []string{"normalized"}})
specs := map[string]pipeline.ModuleSpec{
"fake/input": {Key: "fake/input", Stage: pipeline.StageInput, Provides: []string{"source"}},
"generic": {Key: "generic", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}},
"fake/extract": {Key: "fake/extract", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}},
"appendorder": {Key: "appendorder", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}},
"noop": {Key: "noop", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}},
"json": {Key: "json", Stage: pipeline.StageOutput, Requires: []string{"normalized"}},
}
for _, override := range overrides {
specs[override.Key] = override
}
mustRegisterInput(t, inputs, specs["fake/input"])
mustRegisterChunker(t, chunkers, specs["generic"])
mustRegisterExtractor(t, extractors, specs["fake/extract"])
mustRegisterMerger(t, mergers, specs["appendorder"])
mustRegisterNormalizer(t, normalizers, specs["noop"])
mustRegisterOutput(t, outputs, specs["json"])
return pipeline.ModuleCatalog{
Inputs: inputs,

View File

@@ -48,6 +48,17 @@ type LLMProfileManifest struct {
Model string `json:"model,omitempty"`
}
type ReferenceProvenance struct {
LaneID string `json:"lane_id"`
SlotName string `json:"slot_name"`
OriginType string `json:"origin_type"`
OriginURI string `json:"origin_uri,omitempty"`
Digest string `json:"digest,omitempty"`
MediaType string `json:"media_type,omitempty"`
SizeBytes int64 `json:"size_bytes,omitempty"`
BindingSource string `json:"binding_source,omitempty"`
}
type RunManifest struct {
RunID string `json:"run_id,omitempty"`
PipelineID string `json:"pipeline_id,omitempty"`
@@ -61,6 +72,7 @@ type RunManifest struct {
OutputEncoder string `json:"output_encoder,omitempty"`
ModuleMetadata map[string]map[string]any `json:"module_metadata,omitempty"`
ArtifactLanes []ArtifactLaneManifest `json:"artifact_lanes,omitempty"`
References []ReferenceProvenance `json:"references,omitempty"`
LLMProfiles []LLMProfileManifest `json:"llm_profiles,omitempty"`
SchemaVersion string `json:"schema_version,omitempty"`
ValidationStatus string `json:"validation_status,omitempty"`

View File

@@ -183,6 +183,43 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
assertHasKeys(t, lane, "id", "extractor", "merger", "normalizer", "validators", "metadata")
}
func TestRunManifestIncludesReferenceProvenance(t *testing.T) {
manifest := RunManifest{
References: []ReferenceProvenance{
{
LaneID: "events",
SlotName: "roster",
OriginType: "file",
OriginURI: "file:///tmp/roster.txt",
Digest: "sha256:reference",
MediaType: "text/plain; charset=utf-8",
SizeBytes: 12,
BindingSource: "config",
},
},
}
gotJSON, err := json.Marshal(manifest)
if err != nil {
t.Fatalf("json.Marshal() error = %v", err)
}
var got RunManifest
if err := json.Unmarshal(gotJSON, &got); err != nil {
t.Fatalf("json.Unmarshal() error = %v", err)
}
if len(got.References) != 1 {
t.Fatalf("len(References) = %d, want 1", len(got.References))
}
reference := got.References[0]
if reference.LaneID != "events" || reference.SlotName != "roster" || reference.OriginType != "file" || reference.OriginURI != "file:///tmp/roster.txt" {
t.Fatalf("reference provenance = %#v, want lane-scoped origin details", reference)
}
if reference.Digest != "sha256:reference" || reference.MediaType != "text/plain; charset=utf-8" || reference.SizeBytes != 12 || reference.BindingSource != "config" {
t.Fatalf("reference provenance = %#v, want digest/media/size/source details", reference)
}
}
func TestRunManifestIncludesTopLevelModuleMetadata(t *testing.T) {
manifest := RunManifest{
ModuleMetadata: map[string]map[string]any{

View File

@@ -73,6 +73,7 @@ func clonePipelineProfile(in pipeline.PipelineProfile) pipeline.PipelineProfile
out.Input = cloneModuleBinding(in.Input)
out.Chunk = cloneModuleBinding(in.Chunk)
out.Output = cloneModuleBinding(in.Output)
out.References = cloneStringMap(in.References)
if len(in.Artifacts) > 0 {
out.Artifacts = make(map[string]pipeline.ArtifactLaneProfile, len(in.Artifacts))
for key, lane := range in.Artifacts {
@@ -87,6 +88,7 @@ func cloneArtifactLaneProfile(in pipeline.ArtifactLaneProfile) pipeline.Artifact
out.Extract = cloneModuleBinding(in.Extract)
out.Merge = cloneModuleBinding(in.Merge)
out.Normalize = cloneModuleBinding(in.Normalize)
out.References = cloneStringMap(in.References)
if len(in.Validators) > 0 {
out.Validators = make([]pipeline.ModuleBinding, len(in.Validators))
for i, binding := range in.Validators {
@@ -96,6 +98,17 @@ func cloneArtifactLaneProfile(in pipeline.ArtifactLaneProfile) pipeline.Artifact
return out
}
func cloneStringMap(in map[string]string) map[string]string {
if len(in) == 0 {
return nil
}
out := make(map[string]string, len(in))
for key, value := range in {
out[key] = value
}
return out
}
func cloneModuleBinding(in pipeline.ModuleBinding) pipeline.ModuleBinding {
out := in
if len(in.Options) > 0 {

View File

@@ -14,13 +14,17 @@ type ResolveInput struct {
Only []string
Catalog pipeline.ModuleCatalog
LLMProfileOverride string
ReferenceOverrides []pipeline.ReferenceBinding
ReferenceUnbinds []pipeline.ReferenceUnbind
}
type EffectiveConfig struct {
Config Config
PipelineID string
Only []string
ResolvedPipeline pipeline.ResolvedPipeline
Config Config
PipelineID string
Only []string
ReferenceOverrides []pipeline.ReferenceBinding
ReferenceUnbinds []pipeline.ReferenceUnbind
ResolvedPipeline pipeline.ResolvedPipeline
}
func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
@@ -46,16 +50,22 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
applyLLMProfileOverride(&profile, override)
}
resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{Only: input.Only}, input.Catalog)
resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{
Only: input.Only,
ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...),
ReferenceUnbinds: append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...),
}, input.Catalog)
if err != nil {
return EffectiveConfig{}, fmt.Errorf("resolve pipeline %q: %w", pipelineID, err)
}
return EffectiveConfig{
Config: cloneConfig(c),
PipelineID: pipelineID,
Only: append([]string(nil), input.Only...),
ResolvedPipeline: resolved,
Config: cloneConfig(c),
PipelineID: pipelineID,
Only: append([]string(nil), input.Only...),
ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...),
ReferenceUnbinds: append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...),
ResolvedPipeline: resolved,
}, nil
}

View File

@@ -35,10 +35,11 @@ type FileLLMProfile struct {
}
type FilePipelineProfile struct {
Input fileModuleBinding `yaml:"input"`
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
Output *fileModuleBinding `yaml:"output,omitempty"`
Input fileModuleBinding `yaml:"input"`
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
Output *fileModuleBinding `yaml:"output,omitempty"`
References map[string]string `yaml:"references,omitempty"`
}
type FileArtifactLaneProfile struct {
@@ -46,6 +47,7 @@ type FileArtifactLaneProfile struct {
Merge *fileModuleBinding `yaml:"merge,omitempty"`
Normalize *fileModuleBinding `yaml:"normalize,omitempty"`
Validators []fileModuleBinding `yaml:"validators,omitempty"`
References map[string]string `yaml:"references,omitempty"`
}
type FileConcurrencyConfig struct {
@@ -212,6 +214,18 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
if _, _, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID)); err != nil {
return err
}
if _, _, err := normalizedMapKeys(filePipeline.References, fmt.Sprintf("pipeline %q reference slot", pipelineID)); err != nil {
return err
}
for rawLaneID, fileLane := range filePipeline.Artifacts {
laneID := strings.TrimSpace(rawLaneID)
if laneID == "" {
continue
}
if _, _, err := normalizedMapKeys(fileLane.References, fmt.Sprintf("pipeline %q lane %q reference slot", pipelineID, laneID)); err != nil {
return err
}
}
}
for _, profileID := range profileIDs {
@@ -253,9 +267,10 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
return err
}
profile := pipeline.PipelineProfile{
ID: pipelineID,
Input: filePipeline.Input.toPipelineBinding(),
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
ID: pipelineID,
Input: filePipeline.Input.toPipelineBinding(),
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
References: normalizedStringMap(filePipeline.References),
}
if filePipeline.Chunk != nil {
profile.Chunk = filePipeline.Chunk.toPipelineBinding()
@@ -266,7 +281,8 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
for _, laneID := range laneIDs {
fileLane := filePipeline.Artifacts[rawLaneIDs[laneID]]
lane := pipeline.ArtifactLaneProfile{
Extract: fileLane.Extract.toPipelineBinding(),
Extract: fileLane.Extract.toPipelineBinding(),
References: normalizedStringMap(fileLane.References),
}
if fileLane.Merge != nil {
lane.Merge = fileLane.Merge.toPipelineBinding()
@@ -318,6 +334,25 @@ func normalizedMapKeys[T any](values map[string]T, keyName string) ([]string, ma
return keys, rawByNormalized, nil
}
func normalizedStringMap(values map[string]string) map[string]string {
if len(values) == 0 {
return nil
}
out := make(map[string]string, len(values))
keys := make([]string, 0, len(values))
rawByNormalized := make(map[string]string, len(values))
for rawKey := range values {
key := strings.TrimSpace(rawKey)
rawByNormalized[key] = rawKey
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
out[key] = strings.TrimSpace(values[rawByNormalized[key]])
}
return out
}
func resolveAPIKeyEnv(envName string, lookup func(string) (string, bool)) (string, error) {
name := strings.TrimSpace(envName)
if name == "" {

View File

@@ -144,6 +144,31 @@ pipelines:
}
}
func TestParseFileConfigReferenceMaps(t *testing.T) {
cfg := parseAndApplyConfig(t, `
version: 1
pipelines:
example:
input: fake/input
references:
" roster ": " ./shared-roster.yml "
artifacts:
events:
extract: fake/extract
references:
" lore ": " ./lore.md "
`)
profile := cfg.Pipelines["example"]
if !reflect.DeepEqual(profile.References, map[string]string{"roster": "./shared-roster.yml"}) {
t.Fatalf("pipeline references = %#v, want trimmed map", profile.References)
}
gotLaneRefs := profile.Artifacts["events"].References
if !reflect.DeepEqual(gotLaneRefs, map[string]string{"lore": "./lore.md"}) {
t.Fatalf("lane references = %#v, want trimmed map", gotLaneRefs)
}
}
func TestParseFileConfigValidatorMixedBindingForms(t *testing.T) {
cfg := parseAndApplyConfig(t, `
version: 1
@@ -297,6 +322,58 @@ pipelines:
}
}
func TestApplyFileConfigRejectsDuplicateTrimmedReferenceSlots(t *testing.T) {
tests := []struct {
name string
raw string
want string
}{
{
name: "pipeline",
raw: `
version: 1
pipelines:
example:
input: fake/input
references:
roster: ./first.yml
" roster ": ./second.yml
`,
want: `pipeline "example" reference slot`,
},
{
name: "lane",
raw: `
version: 1
pipelines:
example:
input: fake/input
artifacts:
events:
extract: fake/extract
references:
roster: ./first.yml
" roster ": ./second.yml
`,
want: `pipeline "example" lane "events" reference slot`,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
fileCfg, err := ParseFileConfigYAML([]byte(tc.raw))
if err != nil {
t.Fatalf("ParseFileConfigYAML: %v", err)
}
cfg := Default()
err = cfg.applyFileConfigWithLookup(fileCfg, emptyLookup)
if err == nil || !strings.Contains(err.Error(), tc.want) || !strings.Contains(err.Error(), "duplicated") {
t.Fatalf("expected duplicate reference slot error, got %v", err)
}
})
}
}
func TestApplyFileConfigAllowsRetryOnlyLLMProfile(t *testing.T) {
cfg := parseAndApplyConfig(t, `
version: 1

View File

@@ -21,10 +21,12 @@ func (c Config) RedactedDiagnosticsPayload() any {
func (e EffectiveConfig) RedactedDiagnosticsPayload() any {
return EffectiveConfig{
Config: e.Config.Redacted(),
PipelineID: e.PipelineID,
Only: append([]string(nil), e.Only...),
ResolvedPipeline: cloneResolvedPipeline(e.ResolvedPipeline),
Config: e.Config.Redacted(),
PipelineID: e.PipelineID,
Only: append([]string(nil), e.Only...),
ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), e.ReferenceOverrides...),
ReferenceUnbinds: append([]pipeline.ReferenceUnbind(nil), e.ReferenceUnbinds...),
ResolvedPipeline: cloneResolvedPipeline(e.ResolvedPipeline),
}
}
@@ -47,6 +49,8 @@ func cloneResolvedArtifactLane(in pipeline.ResolvedArtifactLane) pipeline.Resolv
out.Extract = cloneModuleBinding(in.Extract)
out.Merge = cloneModuleBinding(in.Merge)
out.Normalize = cloneModuleBinding(in.Normalize)
out.References = append([]pipeline.ReferenceBinding(nil), in.References...)
out.ReferenceSet = pipeline.CloneReferenceSet(in.ReferenceSet)
if len(in.Validators) > 0 {
out.Validators = make([]pipeline.ModuleBinding, len(in.Validators))
for i, binding := range in.Validators {

View File

@@ -3,6 +3,7 @@ package config
import (
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
@@ -65,12 +66,21 @@ func TestEffectiveConfigRedactedDiagnosticsPayloadRedactsAndCopies(t *testing.T)
cfg.LLMProfiles[pipeline.DefaultLLMProfile] = profile
lane := cfg.Pipelines["example"].Artifacts["events"]
lane.Extract.Options = map[string]any{"temperature": 0.2}
lane.References = map[string]string{"roster": "./roster.yml"}
cfg.Pipelines["example"].Artifacts["events"] = lane
effective, err := cfg.Resolve(ResolveInput{
PipelineID: "example",
Only: []string{"events"},
Catalog: fakeCatalog(t),
Catalog: fakeCatalog(t, pipeline.ModuleSpec{
Key: "fake/extract",
Stage: pipeline.StageExtract,
Requires: []string{"chunks"},
Provides: []string{"artifact"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster"},
},
}),
})
if err != nil {
t.Fatalf("Resolve: %v", err)
@@ -98,4 +108,8 @@ func TestEffectiveConfigRedactedDiagnosticsPayloadRedactsAndCopies(t *testing.T)
if effective.ResolvedPipeline.ArtifactLanes[0].Extract.Options["temperature"] != 0.2 {
t.Fatalf("expected resolved pipeline options to be copied")
}
payload.ResolvedPipeline.ArtifactLanes[0].References[0].Source = "./changed.yml"
if effective.ResolvedPipeline.ArtifactLanes[0].References[0].Source != "./roster.yml" {
t.Fatalf("expected resolved pipeline references to be copied")
}
}

View File

@@ -98,11 +98,17 @@ func validatePipelineProfiles(profiles map[string]pipeline.PipelineProfile, llmP
if err := validateBindingLLMProfile(id, "", "output", profile.Output, llmProfiles); err != nil {
return err
}
if err := validateReferenceMap(id, "", profile.References); err != nil {
return err
}
for rawLaneID, lane := range profile.Artifacts {
laneID := strings.TrimSpace(rawLaneID)
if laneID == "" {
return fmt.Errorf("pipeline %q artifact lane id must not be empty", id)
}
if err := validateReferenceMap(id, laneID, lane.References); err != nil {
return err
}
if err := validateBindingLLMProfile(id, laneID, "extract", lane.Extract, llmProfiles); err != nil {
return err
}
@@ -122,6 +128,33 @@ func validatePipelineProfiles(profiles map[string]pipeline.PipelineProfile, llmP
return nil
}
func validateReferenceMap(pipelineID string, laneID string, references map[string]string) error {
seen := make(map[string]struct{}, len(references))
for rawSlotName, rawSource := range references {
slotName := strings.TrimSpace(rawSlotName)
if slotName == "" {
if laneID != "" {
return fmt.Errorf("pipeline %q lane %q reference slot name must not be empty", pipelineID, laneID)
}
return fmt.Errorf("pipeline %q reference slot name must not be empty", pipelineID)
}
if _, ok := seen[slotName]; ok {
if laneID != "" {
return fmt.Errorf("pipeline %q lane %q reference slot %q is duplicated after trimming", pipelineID, laneID, slotName)
}
return fmt.Errorf("pipeline %q reference slot %q is duplicated after trimming", pipelineID, slotName)
}
seen[slotName] = struct{}{}
if strings.TrimSpace(rawSource) == "" {
if laneID != "" {
return fmt.Errorf("pipeline %q lane %q reference slot %q source must not be empty", pipelineID, laneID, slotName)
}
return fmt.Errorf("pipeline %q reference slot %q source must not be empty", pipelineID, slotName)
}
}
return nil
}
func validateBindingLLMProfile(
pipelineID string,
laneID string,

View File

@@ -116,6 +116,73 @@ func TestValidateRejectsInvalidDiagnosticsRetention(t *testing.T) {
}
}
func TestValidateRejectsInvalidReferenceMaps(t *testing.T) {
tests := []struct {
name string
mutate func(Config) Config
want []string
}{
{
name: "empty pipeline slot",
mutate: func(cfg Config) Config {
profile := cfg.Pipelines["example"]
profile.References = map[string]string{" ": "./roster.yml"}
cfg.Pipelines["example"] = profile
return cfg
},
want: []string{"example", "reference slot", "empty"},
},
{
name: "empty pipeline source",
mutate: func(cfg Config) Config {
profile := cfg.Pipelines["example"]
profile.References = map[string]string{"roster": " "}
cfg.Pipelines["example"] = profile
return cfg
},
want: []string{"example", "roster", "source", "empty"},
},
{
name: "empty lane slot",
mutate: func(cfg Config) Config {
profile := cfg.Pipelines["example"]
lane := profile.Artifacts["events"]
lane.References = map[string]string{" ": "./roster.yml"}
profile.Artifacts["events"] = lane
cfg.Pipelines["example"] = profile
return cfg
},
want: []string{"example", "events", "reference slot", "empty"},
},
{
name: "empty lane source",
mutate: func(cfg Config) Config {
profile := cfg.Pipelines["example"]
lane := profile.Artifacts["events"]
lane.References = map[string]string{"roster": " "}
profile.Artifacts["events"] = lane
cfg.Pipelines["example"] = profile
return cfg
},
want: []string{"example", "events", "roster", "source", "empty"},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
err := tc.mutate(validConfig()).Validate()
if err == nil {
t.Fatal("Validate() error = nil, want error")
}
for _, want := range tc.want {
if !strings.Contains(err.Error(), want) {
t.Fatalf("Validate() error = %q, want substring %q", err.Error(), want)
}
}
})
}
}
func TestValidateRejectsEmptyIDs(t *testing.T) {
tests := []struct {
name string

View File

@@ -4,6 +4,7 @@ const (
ArtifactInvocationMetadata = "invocation.json"
ArtifactEffectiveConfig = "effective-config.json"
ArtifactResolvedPipeline = "resolved-pipeline.json"
ArtifactResolvedReferences = "resolved-references.json"
ArtifactSourceDocument = "source-document.json"
ArtifactRunManifest = "run-manifest.json"
ArtifactRunReport = "run-report.json"

View File

@@ -7,6 +7,7 @@ func TestArtifactNamesUseExtractionOrientedNames(t *testing.T) {
ArtifactInvocationMetadata,
ArtifactEffectiveConfig,
ArtifactResolvedPipeline,
ArtifactResolvedReferences,
ArtifactSourceDocument,
ArtifactRunManifest,
ArtifactRunReport,

View File

@@ -149,6 +149,10 @@ func (r *RunDirectory) WriteResolvedPipeline(payload any) error {
return r.WriteJSONArtifact(ArtifactResolvedPipeline, payload)
}
func (r *RunDirectory) WriteResolvedReferences(payload any) error {
return r.WriteJSONArtifact(ArtifactResolvedReferences, payload)
}
func (r *RunDirectory) WriteSourceDocument(payload any) error {
return r.WriteJSONArtifact(ArtifactSourceDocument, payload)
}

View File

@@ -191,6 +191,9 @@ func TestWriteTypedArtifacts(t *testing.T) {
if err := runDir.WriteResolvedPipeline(map[string]any{"pipeline": "test"}); err != nil {
t.Fatalf("WriteResolvedPipeline: %v", err)
}
if err := runDir.WriteResolvedReferences([]artifacts.ReferenceProvenance{{LaneID: "events", SlotName: "roster"}}); err != nil {
t.Fatalf("WriteResolvedReferences: %v", err)
}
if err := runDir.WriteSourceDocument(map[string]any{"source_id": "source-1"}); err != nil {
t.Fatalf("WriteSourceDocument: %v", err)
}
@@ -207,6 +210,7 @@ func TestWriteTypedArtifacts(t *testing.T) {
for _, name := range []string{
ArtifactEffectiveConfig,
ArtifactResolvedPipeline,
ArtifactResolvedReferences,
ArtifactSourceDocument,
ArtifactRunManifest,
ArtifactRunReport,

View File

@@ -203,6 +203,10 @@ func (extractor compositionExtractor) SchemaVersion() string {
return "v1"
}
func (extractor compositionExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (extractor compositionExtractor) Validators() []contracts.Validator {
return []contracts.Validator{compositionValidator{}}
}

View File

@@ -74,10 +74,49 @@ type Chunker interface {
Chunk(ctx context.Context, req ChunkRequest) (ChunkResult, error)
}
const (
ReferenceBindingSourceConfig = "config"
ReferenceBindingSourceCLI = "cli"
)
type ReferenceSlot struct {
Name string `json:"name"`
Description string `json:"description,omitempty"`
Required bool `json:"required,omitempty"`
AcceptedMediaTypes []string `json:"accepted_media_types,omitempty"`
Multiple bool `json:"multiple,omitempty"`
MaxBytes int64 `json:"max_bytes,omitempty"`
}
type ReferenceOrigin struct {
Type string `json:"type"`
URI string `json:"uri,omitempty"`
}
type ReferenceItem struct {
SlotName string `json:"slot_name"`
MediaType string `json:"media_type,omitempty"`
Content []byte `json:"-"`
Digest string `json:"digest,omitempty"`
Origin ReferenceOrigin `json:"origin"`
SizeBytes int64 `json:"size_bytes,omitempty"`
BindingSource string `json:"binding_source,omitempty"`
}
type ResolvedReferenceSlot struct {
Slot ReferenceSlot `json:"slot"`
Items []ReferenceItem `json:"items,omitempty"`
}
type ReferenceSet struct {
Slots map[string]ResolvedReferenceSlot `json:"slots,omitempty"`
}
type ExtractionRequest struct {
Source *source.SourceDocument `json:"-"`
Chunk *SourceChunk `json:"chunk,omitempty"`
AmbientContext map[string]any `json:"ambient_context,omitempty"`
References ReferenceSet `json:"references,omitempty"`
LLMClient StructuredLLMClient `json:"-"`
LLMProfile string `json:"llm_profile,omitempty"`
Options map[string]any `json:"options,omitempty"`
@@ -93,6 +132,7 @@ type Extractor interface {
Key() string
ArtifactType() string
SchemaVersion() string
ReferenceSlots() []ReferenceSlot
Validators() []Validator
Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error)
}

View File

@@ -186,6 +186,71 @@ func TestFakeExtractorReceivesChunkAndAmbientContext(t *testing.T) {
}
}
func TestReferenceSetDataTypes(t *testing.T) {
references := ReferenceSet{
Slots: map[string]ResolvedReferenceSlot{
"roster": {
Slot: ReferenceSlot{
Name: "roster",
Description: "Known characters",
Required: true,
AcceptedMediaTypes: []string{"text/plain"},
Multiple: true,
MaxBytes: 4096,
},
Items: []ReferenceItem{
{
SlotName: "roster",
MediaType: "text/plain",
Content: []byte("Aria\nBryn\n"),
Digest: "sha256:reference",
Origin: ReferenceOrigin{
Type: "file",
URI: "file:///tmp/roster.txt",
},
SizeBytes: 10,
BindingSource: ReferenceBindingSourceConfig,
},
},
},
},
}
item := references.Slots["roster"].Items[0]
if item.SlotName != "roster" || item.MediaType != "text/plain" || string(item.Content) != "Aria\nBryn\n" {
t.Fatalf("reference item = %#v, want constructed item fields", item)
}
if item.BindingSource != ReferenceBindingSourceConfig {
t.Fatalf("BindingSource = %q, want %q", item.BindingSource, ReferenceBindingSourceConfig)
}
}
func TestReferenceItemJSONOmitsContent(t *testing.T) {
item := ReferenceItem{
SlotName: "roster",
MediaType: "text/plain",
Content: []byte("reference content"),
Digest: "sha256:reference",
Origin: ReferenceOrigin{Type: "file", URI: "file:///tmp/roster.txt"},
}
encoded, err := json.Marshal(item)
if err != nil {
t.Fatalf("json.Marshal() error = %v, want nil", err)
}
var got map[string]any
if err := json.Unmarshal(encoded, &got); err != nil {
t.Fatalf("json.Unmarshal() error = %v, want nil", err)
}
if _, ok := got["content"]; ok {
t.Fatalf("encoded reference item leaked content: %s", encoded)
}
if _, ok := got["Content"]; ok {
t.Fatalf("encoded reference item leaked Content: %s", encoded)
}
}
func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
candidate := artifacts.ArtifactCandidate{
Index: 0,
@@ -359,6 +424,10 @@ func (extractor fakeExtractor) SchemaVersion() string {
return extractor.schemaVersion
}
func (extractor fakeExtractor) ReferenceSlots() []ReferenceSlot {
return nil
}
func (extractor fakeExtractor) Validators() []Validator {
return extractor.validators
}

View File

@@ -117,6 +117,8 @@ func (defaultExtractor) ArtifactType() string { return "record" }
func (defaultExtractor) SchemaVersion() string { return "v1" }
func (defaultExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (defaultExtractor) Validators() []contracts.Validator { return nil }
func (defaultExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {

View File

@@ -49,6 +49,20 @@ func TestExtractorRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
Stage: StageExtract,
Provides: []string{" generic-artifact ", "source-citations", "generic-artifact", ""},
Requires: []string{" source-document ", "source-document", ""},
ReferenceSlots: []contracts.ReferenceSlot{
{
Name: " glossary ",
Description: " Supporting terms ",
AcceptedMediaTypes: []string{" text/plain ", "text/markdown", "text/plain", ""},
MaxBytes: 1024,
},
{
Name: " roster ",
Description: " Characters ",
Required: true,
Multiple: true,
},
},
}
if err := registry.RegisterWithSpec(spec, fakeExtractorConstructor("generic-extractor")); err != nil {
@@ -64,12 +78,28 @@ func TestExtractorRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
Stage: StageExtract,
Provides: []string{"generic-artifact", "source-citations"},
Requires: []string{"source-document"},
ReferenceSlots: []contracts.ReferenceSlot{
{
Name: "glossary",
Description: "Supporting terms",
AcceptedMediaTypes: []string{"text/markdown", "text/plain"},
MaxBytes: 1024,
},
{
Name: "roster",
Description: "Characters",
Required: true,
Multiple: true,
},
},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("Spec() = %#v, want %#v", got, want)
}
got.Provides[0] = "changed"
got.ReferenceSlots[0].Name = "changed"
got.ReferenceSlots[0].AcceptedMediaTypes[0] = "changed"
again, ok := registry.Spec("generic-extractor")
if !ok {
t.Fatal("Spec() after caller mutation ok = false, want true")
@@ -109,6 +139,50 @@ func TestExtractorRegistryRegisterWithSpecRejectsWrongStage(t *testing.T) {
}
}
func TestExtractorRegistryRejectsInvalidReferenceSlots(t *testing.T) {
tests := []struct {
name string
slots []contracts.ReferenceSlot
want string
}{
{
name: "empty name",
slots: []contracts.ReferenceSlot{{Name: " "}},
want: "name",
},
{
name: "duplicate name after trim",
slots: []contracts.ReferenceSlot{
{Name: "roster"},
{Name: " roster "},
},
want: "duplicated",
},
{
name: "negative max bytes",
slots: []contracts.ReferenceSlot{{Name: "roster", MaxBytes: -1}},
want: "max_bytes",
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
registry := NewExtractorRegistry()
err := registry.RegisterWithSpec(ModuleSpec{
Key: "generic-extractor",
Stage: StageExtract,
ReferenceSlots: test.slots,
}, fakeExtractorConstructor("generic-extractor"))
if err == nil {
t.Fatal("RegisterWithSpec() error = nil, want error")
}
if !strings.Contains(err.Error(), test.want) {
t.Fatalf("RegisterWithSpec() error = %q, want %q", err.Error(), test.want)
}
})
}
}
func TestExtractorRegistrySpecRejectsUnknownKey(t *testing.T) {
registry := NewExtractorRegistry()
@@ -301,6 +375,10 @@ func (extractor registryFakeExtractor) SchemaVersion() string {
return "v1"
}
func (extractor registryFakeExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (extractor registryFakeExtractor) Validators() []contracts.Validator {
return nil
}

View File

@@ -4,6 +4,8 @@ import (
"fmt"
"sort"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type ModuleStage string
@@ -19,10 +21,11 @@ const (
)
type ModuleSpec struct {
Key string
Stage ModuleStage
Provides []string
Requires []string
Key string
Stage ModuleStage
Provides []string
Requires []string
ReferenceSlots []contracts.ReferenceSlot
}
func defaultModuleSpec(key string, stage ModuleStage) ModuleSpec {
@@ -34,10 +37,11 @@ func defaultModuleSpec(key string, stage ModuleStage) ModuleSpec {
func normalizeModuleSpec(spec ModuleSpec) ModuleSpec {
return ModuleSpec{
Key: strings.TrimSpace(spec.Key),
Stage: spec.Stage,
Provides: normalizeCapabilities(spec.Provides),
Requires: normalizeCapabilities(spec.Requires),
Key: strings.TrimSpace(spec.Key),
Stage: spec.Stage,
Provides: normalizeCapabilities(spec.Provides),
Requires: normalizeCapabilities(spec.Requires),
ReferenceSlots: normalizeReferenceSlots(spec.ReferenceSlots),
}
}
@@ -68,10 +72,11 @@ func normalizeCapabilities(values []string) []string {
func cloneModuleSpec(spec ModuleSpec) ModuleSpec {
return ModuleSpec{
Key: spec.Key,
Stage: spec.Stage,
Provides: append([]string(nil), spec.Provides...),
Requires: append([]string(nil), spec.Requires...),
Key: spec.Key,
Stage: spec.Stage,
Provides: append([]string(nil), spec.Provides...),
Requires: append([]string(nil), spec.Requires...),
ReferenceSlots: cloneReferenceSlots(spec.ReferenceSlots),
}
}
@@ -82,6 +87,12 @@ func validateModuleSpec(kind string, expectedStage ModuleStage, spec ModuleSpec)
if spec.Stage != expectedStage {
return fmt.Errorf("%s %q must use %q stage, got %q", kind, spec.Key, expectedStage, spec.Stage)
}
if spec.Stage != StageExtract && len(spec.ReferenceSlots) > 0 {
return fmt.Errorf("%s %q must not declare reference slots", kind, spec.Key)
}
if err := validateReferenceSlots(spec.ReferenceSlots); err != nil {
return fmt.Errorf("%s %q reference slots: %w", kind, spec.Key, err)
}
return nil
}
@@ -97,3 +108,74 @@ func sortedRegistryKeys[C any](constructors map[string]C) []string {
sort.Strings(keys)
return keys
}
func normalizeReferenceSlots(slots []contracts.ReferenceSlot) []contracts.ReferenceSlot {
if len(slots) == 0 {
return nil
}
normalized := make([]contracts.ReferenceSlot, 0, len(slots))
for _, slot := range slots {
slot.Name = strings.TrimSpace(slot.Name)
slot.Description = strings.TrimSpace(slot.Description)
slot.AcceptedMediaTypes = normalizeStringSet(slot.AcceptedMediaTypes)
normalized = append(normalized, slot)
}
sort.SliceStable(normalized, func(i, j int) bool {
return normalized[i].Name < normalized[j].Name
})
return normalized
}
func normalizeStringSet(values []string) []string {
if len(values) == 0 {
return nil
}
seen := make(map[string]struct{}, len(values))
for _, value := range values {
normalized := strings.TrimSpace(value)
if normalized == "" {
continue
}
seen[normalized] = struct{}{}
}
if len(seen) == 0 {
return nil
}
out := make([]string, 0, len(seen))
for value := range seen {
out = append(out, value)
}
sort.Strings(out)
return out
}
func validateReferenceSlots(slots []contracts.ReferenceSlot) error {
seen := make(map[string]struct{}, len(slots))
for i, slot := range slots {
if slot.Name == "" {
return fmt.Errorf("slot[%d].name must not be empty", i)
}
if _, ok := seen[slot.Name]; ok {
return fmt.Errorf("slot name %q is duplicated", slot.Name)
}
seen[slot.Name] = struct{}{}
if slot.MaxBytes < 0 {
return fmt.Errorf("slot %q max_bytes must not be negative", slot.Name)
}
}
return nil
}
func cloneReferenceSlots(slots []contracts.ReferenceSlot) []contracts.ReferenceSlot {
if len(slots) == 0 {
return nil
}
out := make([]contracts.ReferenceSlot, 0, len(slots))
for _, slot := range slots {
slot.AcceptedMediaTypes = append([]string(nil), slot.AcceptedMediaTypes...)
out = append(out, slot)
}
return out
}

View File

@@ -0,0 +1,25 @@
package pipeline
import (
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestValidateModuleSpecRejectsReferenceSlotsForNonExtractors(t *testing.T) {
err := validateModuleSpec("chunker", StageChunk, ModuleSpec{
Key: "generic",
Stage: StageChunk,
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster"},
},
})
if err == nil {
t.Fatal("validateModuleSpec() error = nil, want error")
}
if !strings.Contains(err.Error(), "reference slots") {
t.Fatalf("validateModuleSpec() error = %q, want reference slots context", err.Error())
}
}

View File

@@ -7,6 +7,8 @@ import (
"fmt"
"sort"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
const (
@@ -24,30 +26,48 @@ type ModuleBinding struct {
}
type ArtifactLaneProfile struct {
Extract ModuleBinding `json:"extract"`
Merge ModuleBinding `json:"merge,omitempty"`
Normalize ModuleBinding `json:"normalize,omitempty"`
Validators []ModuleBinding `json:"validators,omitempty"`
Extract ModuleBinding `json:"extract"`
Merge ModuleBinding `json:"merge,omitempty"`
Normalize ModuleBinding `json:"normalize,omitempty"`
Validators []ModuleBinding `json:"validators,omitempty"`
References map[string]string `json:"references,omitempty"`
}
type PipelineProfile struct {
ID string `json:"id"`
Input ModuleBinding `json:"input"`
Chunk ModuleBinding `json:"chunk,omitempty"`
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
Output ModuleBinding `json:"output,omitempty"`
ID string `json:"id"`
Input ModuleBinding `json:"input"`
Chunk ModuleBinding `json:"chunk,omitempty"`
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
Output ModuleBinding `json:"output,omitempty"`
References map[string]string `json:"references,omitempty"`
}
type ResolveOptions struct {
Only []string
Only []string
ReferenceOverrides []ReferenceBinding
ReferenceUnbinds []ReferenceUnbind
}
type ReferenceBinding struct {
LaneID string `json:"lane_id,omitempty"`
SlotName string `json:"slot_name"`
Source string `json:"source"`
BindingSource string `json:"binding_source,omitempty"`
}
type ReferenceUnbind struct {
LaneID string `json:"lane_id"`
SlotName string `json:"slot_name"`
}
type ResolvedArtifactLane struct {
ID string
Extract ModuleBinding
Merge ModuleBinding
Normalize ModuleBinding
Validators []ModuleBinding
ID string
Extract ModuleBinding
Merge ModuleBinding
Normalize ModuleBinding
Validators []ModuleBinding
References []ReferenceBinding `json:"references,omitempty"`
ReferenceSet contracts.ReferenceSet `json:"-"`
}
type ResolvedPipeline struct {
@@ -126,7 +146,7 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
for _, laneID := range selectedLaneIDs {
laneProfile := lanesByID[laneID]
lane, laneCapabilities, err := resolveArtifactLane(pipelineID, laneID, laneProfile, capabilities, catalog)
lane, laneCapabilities, err := resolveArtifactLane(pipelineID, laneID, laneProfile, profile.References, options, capabilities, catalog)
if err != nil {
return ResolvedPipeline{}, err
}
@@ -150,7 +170,15 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
return resolved, nil
}
func resolveArtifactLane(pipelineID, laneID string, profile ArtifactLaneProfile, inherited capabilitySet, catalog ModuleCatalog) (ResolvedArtifactLane, capabilitySet, error) {
func resolveArtifactLane(
pipelineID string,
laneID string,
profile ArtifactLaneProfile,
pipelineReferences map[string]string,
options ResolveOptions,
inherited capabilitySet,
catalog ModuleCatalog,
) (ResolvedArtifactLane, capabilitySet, error) {
lane := ResolvedArtifactLane{
ID: laneID,
Extract: resolveBinding(profile.Extract, ""),
@@ -171,6 +199,11 @@ func resolveArtifactLane(pipelineID, laneID string, profile ArtifactLaneProfile,
if missing, ok := capabilities.missing(extractSpec.Requires); ok {
return ResolvedArtifactLane{}, nil, capabilityError(pipelineID, laneID, StageExtract, lane.Extract.Module, missing)
}
references, err := resolveReferenceBindings(pipelineID, laneID, lane.Extract.Module, extractSpec.ReferenceSlots, pipelineReferences, profile.References, options)
if err != nil {
return ResolvedArtifactLane{}, nil, err
}
lane.References = references
capabilities.add(extractSpec.Provides...)
mergeSpec, err := mergerSpec(catalog, lane.Merge.Module)
@@ -205,6 +238,162 @@ func resolveArtifactLane(pipelineID, laneID string, profile ArtifactLaneProfile,
return lane, capabilities, nil
}
func resolveReferenceBindings(
pipelineID string,
laneID string,
extractorModule string,
slots []contracts.ReferenceSlot,
pipelineReferences map[string]string,
laneReferences map[string]string,
options ResolveOptions,
) ([]ReferenceBinding, error) {
slotByName := make(map[string]contracts.ReferenceSlot, len(slots))
for _, slot := range slots {
slotByName[slot.Name] = slot
}
bindings := make(map[string]ReferenceBinding)
addBinding := func(slotName, source, bindingSource string) error {
slotName = strings.TrimSpace(slotName)
source = strings.TrimSpace(source)
if slotName == "" {
return fmt.Errorf("pipeline %q lane %q reference slot name must not be empty", pipelineID, laneID)
}
if source == "" {
return fmt.Errorf("pipeline %q lane %q reference slot %q source must not be empty", pipelineID, laneID, slotName)
}
if _, ok := slotByName[slotName]; !ok {
return fmt.Errorf("pipeline %q lane %q reference slot %q is not declared by extractor %q", pipelineID, laneID, slotName, extractorModule)
}
bindings[slotName] = ReferenceBinding{
LaneID: laneID,
SlotName: slotName,
Source: source,
BindingSource: bindingSource,
}
return nil
}
normalizedPipelineReferences, err := normalizedReferenceMap(pipelineReferences, fmt.Sprintf("pipeline %q reference slot", pipelineID))
if err != nil {
return nil, err
}
for _, slotName := range sortedStringMapKeys(normalizedPipelineReferences) {
if _, ok := slotByName[slotName]; !ok {
continue
}
if err := addBinding(slotName, normalizedPipelineReferences[slotName], contracts.ReferenceBindingSourceConfig); err != nil {
return nil, err
}
}
normalizedLaneReferences, err := normalizedReferenceMap(laneReferences, fmt.Sprintf("pipeline %q lane %q reference slot", pipelineID, laneID))
if err != nil {
return nil, err
}
for _, slotName := range sortedStringMapKeys(normalizedLaneReferences) {
if err := addBinding(slotName, normalizedLaneReferences[slotName], contracts.ReferenceBindingSourceConfig); err != nil {
return nil, err
}
}
for _, override := range options.ReferenceOverrides {
optionLaneID := strings.TrimSpace(override.LaneID)
if optionLaneID == "" {
return nil, fmt.Errorf("pipeline %q reference override lane id must not be empty", pipelineID)
}
if optionLaneID != laneID {
continue
}
source := override.BindingSource
if strings.TrimSpace(source) == "" {
source = contracts.ReferenceBindingSourceCLI
}
if err := addBinding(override.SlotName, override.Source, strings.TrimSpace(source)); err != nil {
return nil, err
}
}
for _, unbind := range options.ReferenceUnbinds {
optionLaneID := strings.TrimSpace(unbind.LaneID)
if optionLaneID == "" {
return nil, fmt.Errorf("pipeline %q reference unbind lane id must not be empty", pipelineID)
}
if optionLaneID != laneID {
continue
}
slotName := strings.TrimSpace(unbind.SlotName)
if slotName == "" {
return nil, fmt.Errorf("pipeline %q lane %q reference unbind slot name must not be empty", pipelineID, laneID)
}
if _, ok := slotByName[slotName]; !ok {
return nil, fmt.Errorf("pipeline %q lane %q reference slot %q is not declared", pipelineID, laneID, slotName)
}
delete(bindings, slotName)
}
for _, slot := range slots {
if slot.Required {
if _, ok := bindings[slot.Name]; !ok {
return nil, fmt.Errorf("pipeline %q lane %q required reference slot %q is not bound", pipelineID, laneID, slot.Name)
}
}
}
keys := sortedReferenceBindingKeys(bindings)
resolved := make([]ReferenceBinding, 0, len(keys))
for _, slotName := range keys {
resolved = append(resolved, bindings[slotName])
}
return resolved, nil
}
func normalizedReferenceMap(values map[string]string, keyName string) (map[string]string, error) {
if len(values) == 0 {
return nil, nil
}
out := make(map[string]string, len(values))
for rawSlotName, rawSource := range values {
slotName := strings.TrimSpace(rawSlotName)
if slotName == "" {
return nil, fmt.Errorf("%s must not be empty", keyName)
}
if _, ok := out[slotName]; ok {
return nil, fmt.Errorf("%s %q is duplicated after trimming", keyName, slotName)
}
source := strings.TrimSpace(rawSource)
if source == "" {
return nil, fmt.Errorf("%s %q source must not be empty", keyName, slotName)
}
out[slotName] = source
}
return out, nil
}
func sortedStringMapKeys(values map[string]string) []string {
if len(values) == 0 {
return nil
}
keys := make([]string, 0, len(values))
for key := range values {
keys = append(keys, key)
}
sort.Strings(keys)
return keys
}
func sortedReferenceBindingKeys(values map[string]ReferenceBinding) []string {
if len(values) == 0 {
return nil
}
keys := make([]string, 0, len(values))
for key := range values {
keys = append(keys, key)
}
sort.Strings(keys)
return keys
}
func resolveBinding(binding ModuleBinding, defaultModule string) ModuleBinding {
module := strings.TrimSpace(binding.Module)
if module == "" {

View File

@@ -3,6 +3,7 @@ package pipeline
import (
"context"
"encoding/json"
"errors"
"reflect"
"strings"
"testing"
@@ -125,6 +126,137 @@ func TestResolvePipelineSelectsOnlyRequestedLanes(t *testing.T) {
}
}
func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
profile := multiLaneProfile()
profile.References = map[string]string{
" roster ": " ./shared-roster.yml ",
"unclaimed": "./ignored.yml",
}
lane := profile.Artifacts["events"]
lane.References = map[string]string{
"roster": "./lane-roster.yml",
" lore ": " ./lore.md ",
}
profile.Artifacts["events"] = lane
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
Key: "event-extractor",
Stage: StageExtract,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster", Required: true},
{Name: "lore"},
},
})
resolved, err := ResolvePipeline(profile, ResolveOptions{Only: []string{"events", "summaries"}}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
events := resolvedLane(t, resolved.ArtifactLanes, "events")
want := []ReferenceBinding{
{LaneID: "events", SlotName: "lore", Source: "./lore.md", BindingSource: contracts.ReferenceBindingSourceConfig},
{LaneID: "events", SlotName: "roster", Source: "./lane-roster.yml", BindingSource: contracts.ReferenceBindingSourceConfig},
}
if !reflect.DeepEqual(events.References, want) {
t.Fatalf("events references = %#v, want %#v", events.References, want)
}
summaries := resolvedLane(t, resolved.ArtifactLanes, "summaries")
if len(summaries.References) != 0 {
t.Fatalf("summaries references = %#v, want none", summaries.References)
}
}
func TestResolvePipelineRejectsUndeclaredReferenceSlot(t *testing.T) {
profile := baselineProfile()
lane := profile.Artifacts["events"]
lane.References = map[string]string{"missing": "./missing.yml"}
profile.Artifacts["events"] = lane
_, err := ResolvePipeline(profile, ResolveOptions{}, newProfileCatalog(t))
if err == nil {
t.Fatal("ResolvePipeline() error = nil, want error")
}
assertErrorContains(t, err, "events", "missing", "not declared")
}
func TestResolvePipelineRequiresBoundReferenceSlotsForSelectedLanes(t *testing.T) {
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
Key: "event-extractor",
Stage: StageExtract,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster", Required: true},
},
})
if _, err := ResolvePipeline(multiLaneProfile(), ResolveOptions{Only: []string{"notes"}}, catalog); err != nil {
t.Fatalf("ResolvePipeline(unselected required slot) error = %v, want nil", err)
}
_, err := ResolvePipeline(multiLaneProfile(), ResolveOptions{Only: []string{"events"}}, catalog)
if err == nil {
t.Fatal("ResolvePipeline(selected required slot) error = nil, want error")
}
assertErrorContains(t, err, "events", "required", "roster", "not bound")
}
func TestResolvePipelineReferenceUnbindCanLeaveRequiredSlotMissing(t *testing.T) {
profile := baselineProfile()
profile.References = map[string]string{"roster": "./roster.yml"}
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
Key: "event-extractor",
Stage: StageExtract,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster", Required: true},
},
})
_, err := ResolvePipeline(profile, ResolveOptions{
ReferenceUnbinds: []ReferenceUnbind{{LaneID: "events", SlotName: "roster"}},
}, catalog)
if err == nil {
t.Fatal("ResolvePipeline() error = nil, want error")
}
assertErrorContains(t, err, "events", "required", "roster", "not bound")
}
func TestResolvePipelineUsesReferenceSlotsFromSpecWithoutConstructingExtractor(t *testing.T) {
profile := baselineProfile()
profile.References = map[string]string{"roster": "./roster.yml"}
catalog := emptyProfileCatalog()
for _, spec := range defaultProfileSpecs() {
if spec.Key != "event-extractor" {
registerProfileSpecs(t, catalog, spec)
}
}
if err := catalog.Extractors.RegisterWithSpec(ModuleSpec{
Key: "event-extractor",
Stage: StageExtract,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster", Required: true},
},
}, func() (contracts.Extractor, error) {
return nil, errors.New("constructor should not run")
}); err != nil {
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
}
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
if got := resolved.ArtifactLanes[0].References[0].Source; got != "./roster.yml" {
t.Fatalf("reference source = %q, want ./roster.yml", got)
}
}
func TestResolvePipelineRejectsUnknownOnlyLane(t *testing.T) {
_, err := ResolvePipeline(multiLaneProfile(), ResolveOptions{Only: []string{"missing"}}, newProfileCatalog(t))
if err == nil {
@@ -463,6 +595,17 @@ func laneIDs(lanes []ResolvedArtifactLane) []string {
return ids
}
func resolvedLane(t *testing.T, lanes []ResolvedArtifactLane, laneID string) ResolvedArtifactLane {
t.Helper()
for _, lane := range lanes {
if lane.ID == laneID {
return lane
}
}
t.Fatalf("lane %q not found in %#v", laneID, laneIDs(lanes))
return ResolvedArtifactLane{}
}
func assertErrorContains(t *testing.T, err error, values ...string) {
t.Helper()

View File

@@ -0,0 +1,218 @@
package pipeline
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"net/url"
"os"
"path/filepath"
"sort"
"strings"
"unicode/utf8"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
const (
referenceOriginFile = "file"
referenceMediaType = "text/plain; charset=utf-8"
)
type ReferenceMaterializationOptions struct {
ConfigPath string
WorkingDir string
}
func MaterializeReferences(resolved ResolvedPipeline, catalog ModuleCatalog, options ReferenceMaterializationOptions) (ResolvedPipeline, []contracts.Warning, error) {
out := resolved
if len(resolved.ArtifactLanes) == 0 {
return out, nil, nil
}
warnings := []contracts.Warning(nil)
out.ArtifactLanes = make([]ResolvedArtifactLane, len(resolved.ArtifactLanes))
for i, lane := range resolved.ArtifactLanes {
materializedLane := lane
referenceSet, laneWarnings, err := materializeLaneReferences(resolved.ID, lane, catalog, options)
if err != nil {
return ResolvedPipeline{}, nil, err
}
materializedLane.ReferenceSet = referenceSet
out.ArtifactLanes[i] = materializedLane
warnings = append(warnings, laneWarnings...)
}
return out, warnings, nil
}
func materializeLaneReferences(
pipelineID string,
lane ResolvedArtifactLane,
catalog ModuleCatalog,
options ReferenceMaterializationOptions,
) (contracts.ReferenceSet, []contracts.Warning, error) {
if len(lane.References) == 0 {
return contracts.ReferenceSet{}, nil, nil
}
if catalog.Extractors == nil {
return contracts.ReferenceSet{}, nil, fmt.Errorf("pipeline %q lane %q extract module %q: module %q is not registered", pipelineID, lane.ID, lane.Extract.Module, lane.Extract.Module)
}
spec, ok := catalog.Extractors.Spec(lane.Extract.Module)
if !ok {
return contracts.ReferenceSet{}, nil, fmt.Errorf("pipeline %q lane %q extract module %q: module %q is not registered", pipelineID, lane.ID, lane.Extract.Module, lane.Extract.Module)
}
slotByName := make(map[string]contracts.ReferenceSlot, len(spec.ReferenceSlots))
for _, slot := range spec.ReferenceSlots {
slotByName[slot.Name] = slot
}
set := contracts.ReferenceSet{Slots: make(map[string]contracts.ResolvedReferenceSlot, len(lane.References))}
var warnings []contracts.Warning
for _, binding := range lane.References {
slotName := strings.TrimSpace(binding.SlotName)
slot, ok := slotByName[slotName]
if !ok {
return contracts.ReferenceSet{}, nil, fmt.Errorf("pipeline %q lane %q reference slot %q is not declared by extractor %q", pipelineID, lane.ID, slotName, lane.Extract.Module)
}
path, err := referencePath(binding, options)
if err != nil {
return contracts.ReferenceSet{}, nil, fmt.Errorf("pipeline %q lane %q reference slot %q path %q: %w", pipelineID, lane.ID, slotName, binding.Source, err)
}
content, err := os.ReadFile(path)
if err != nil {
return contracts.ReferenceSet{}, nil, fmt.Errorf("pipeline %q lane %q reference slot %q read %q: %w", pipelineID, lane.ID, slotName, path, err)
}
if !utf8.Valid(content) {
return contracts.ReferenceSet{}, nil, fmt.Errorf("pipeline %q lane %q reference slot %q path %q must be UTF-8 text", pipelineID, lane.ID, slotName, path)
}
if slot.MaxBytes > 0 && int64(len(content)) > slot.MaxBytes {
return contracts.ReferenceSet{}, nil, fmt.Errorf("pipeline %q lane %q reference slot %q path %q is %d bytes, limit %d", pipelineID, lane.ID, slotName, path, len(content), slot.MaxBytes)
}
if len(content) == 0 {
warnings = append(warnings, contracts.Warning{
Scope: fmt.Sprintf("pipeline.%s.lane.%s.reference.%s", pipelineID, lane.ID, slotName),
ReasonCode: "empty_reference",
Message: fmt.Sprintf("reference slot %q for lane %q is bound to an empty file", slotName, lane.ID),
})
}
item := contracts.ReferenceItem{
SlotName: slotName,
MediaType: referenceMediaType,
Content: append([]byte(nil), content...),
Digest: referenceDigest(content),
Origin: contracts.ReferenceOrigin{Type: referenceOriginFile, URI: fileURI(path)},
SizeBytes: int64(len(content)),
BindingSource: strings.TrimSpace(binding.BindingSource),
}
set.Slots[slotName] = contracts.ResolvedReferenceSlot{
Slot: cloneReferenceSlot(slot),
Items: []contracts.ReferenceItem{item},
}
}
return set, warnings, nil
}
func referencePath(binding ReferenceBinding, options ReferenceMaterializationOptions) (string, error) {
source := strings.TrimSpace(binding.Source)
if source == "" {
return "", fmt.Errorf("must not be empty")
}
if filepath.IsAbs(source) {
return filepath.Clean(source), nil
}
base := strings.TrimSpace(options.WorkingDir)
if strings.TrimSpace(binding.BindingSource) == contracts.ReferenceBindingSourceConfig {
base = filepath.Dir(strings.TrimSpace(options.ConfigPath))
}
if base == "" {
var err error
base, err = os.Getwd()
if err != nil {
return "", fmt.Errorf("resolve working directory: %w", err)
}
}
return filepath.Clean(filepath.Join(base, source)), nil
}
func referenceDigest(content []byte) string {
sum := sha256.Sum256(content)
return "sha256:" + hex.EncodeToString(sum[:])
}
func fileURI(path string) string {
absolute, err := filepath.Abs(path)
if err != nil {
absolute = path
}
absolute = filepath.ToSlash(filepath.Clean(absolute))
if strings.HasPrefix(absolute, "/") {
return "file://" + (&url.URL{Path: absolute}).EscapedPath()
}
return "file:///" + (&url.URL{Path: absolute}).EscapedPath()
}
func cloneReferenceSlot(slot contracts.ReferenceSlot) contracts.ReferenceSlot {
slot.AcceptedMediaTypes = append([]string(nil), slot.AcceptedMediaTypes...)
return slot
}
func CloneReferenceSet(in contracts.ReferenceSet) contracts.ReferenceSet {
if len(in.Slots) == 0 {
return contracts.ReferenceSet{}
}
out := contracts.ReferenceSet{Slots: make(map[string]contracts.ResolvedReferenceSlot, len(in.Slots))}
keys := make([]string, 0, len(in.Slots))
for key := range in.Slots {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
slot := in.Slots[key]
slot.Slot = cloneReferenceSlot(slot.Slot)
if len(slot.Items) > 0 {
items := make([]contracts.ReferenceItem, len(slot.Items))
for i, item := range slot.Items {
item.Content = append([]byte(nil), item.Content...)
items[i] = item
}
slot.Items = items
}
out.Slots[key] = slot
}
return out
}
func ReferenceProvenance(resolved ResolvedPipeline) []artifacts.ReferenceProvenance {
provenance := []artifacts.ReferenceProvenance{}
for _, lane := range resolved.ArtifactLanes {
if len(lane.ReferenceSet.Slots) == 0 {
continue
}
slotNames := make([]string, 0, len(lane.ReferenceSet.Slots))
for slotName := range lane.ReferenceSet.Slots {
slotNames = append(slotNames, slotName)
}
sort.Strings(slotNames)
for _, slotName := range slotNames {
slot := lane.ReferenceSet.Slots[slotName]
for _, item := range slot.Items {
provenance = append(provenance, artifacts.ReferenceProvenance{
LaneID: lane.ID,
SlotName: item.SlotName,
OriginType: item.Origin.Type,
OriginURI: item.Origin.URI,
Digest: item.Digest,
MediaType: item.MediaType,
SizeBytes: item.SizeBytes,
BindingSource: item.BindingSource,
})
}
}
}
return provenance
}

View File

@@ -0,0 +1,189 @@
package pipeline
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestMaterializeReferencesResolvesPathsAndDigestsContent(t *testing.T) {
configDir := t.TempDir()
workingDir := t.TempDir()
configReference := filepath.Join(configDir, "config-reference.txt")
cliReference := filepath.Join(workingDir, "cli-reference.txt")
writeReferenceFile(t, configReference, []byte("config text"))
writeReferenceFile(t, cliReference, []byte("cli text"))
pipeline := baselineProfile()
pipeline.References = map[string]string{"roster": "config-reference.txt"}
lane := pipeline.Artifacts["events"]
lane.References = map[string]string{"glossary": "cli-reference.txt"}
pipeline.Artifacts["events"] = lane
catalog := referenceCatalog(t, []contracts.ReferenceSlot{
{Name: "roster"},
{Name: "glossary"},
})
resolved, err := ResolvePipeline(pipeline, ResolveOptions{
ReferenceOverrides: []ReferenceBinding{
{LaneID: "events", SlotName: "glossary", Source: "cli-reference.txt", BindingSource: contracts.ReferenceBindingSourceCLI},
},
}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
first, warnings, err := MaterializeReferences(resolved, catalog, ReferenceMaterializationOptions{
ConfigPath: filepath.Join(configDir, "config.yml"),
WorkingDir: workingDir,
})
if err != nil {
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
}
if len(warnings) != 0 {
t.Fatalf("warnings = %#v, want none", warnings)
}
second, _, err := MaterializeReferences(resolved, catalog, ReferenceMaterializationOptions{
ConfigPath: filepath.Join(configDir, "config.yml"),
WorkingDir: workingDir,
})
if err != nil {
t.Fatalf("MaterializeReferences(second) error = %v, want nil", err)
}
referenceSet := first.ArtifactLanes[0].ReferenceSet
roster := referenceSet.Slots["roster"].Items[0]
if string(roster.Content) != "config text" {
t.Fatalf("roster content = %q, want config text", roster.Content)
}
if roster.Digest != referenceDigest([]byte("config text")) || roster.Digest != second.ArtifactLanes[0].ReferenceSet.Slots["roster"].Items[0].Digest {
t.Fatalf("roster digest = %q, want stable digest", roster.Digest)
}
if roster.BindingSource != contracts.ReferenceBindingSourceConfig {
t.Fatalf("roster binding source = %q, want config", roster.BindingSource)
}
if roster.MediaType != referenceMediaType || roster.Origin.Type != referenceOriginFile || roster.SizeBytes != int64(len("config text")) {
t.Fatalf("roster metadata = %#v, want text file metadata", roster)
}
if !strings.Contains(roster.Origin.URI, "config-reference.txt") {
t.Fatalf("roster origin URI = %q, want config reference path", roster.Origin.URI)
}
glossary := referenceSet.Slots["glossary"].Items[0]
if string(glossary.Content) != "cli text" {
t.Fatalf("glossary content = %q, want cli text", glossary.Content)
}
if glossary.BindingSource != contracts.ReferenceBindingSourceCLI {
t.Fatalf("glossary binding source = %q, want cli", glossary.BindingSource)
}
if !strings.Contains(glossary.Origin.URI, "cli-reference.txt") {
t.Fatalf("glossary origin URI = %q, want cli reference path", glossary.Origin.URI)
}
provenance := ReferenceProvenance(first)
if len(provenance) != 2 {
t.Fatalf("ReferenceProvenance() = %#v, want two entries", provenance)
}
if provenance[0].LaneID != "events" || provenance[0].SlotName != "glossary" || provenance[0].Digest != glossary.Digest {
t.Fatalf("ReferenceProvenance()[0] = %#v, want sorted glossary provenance", provenance[0])
}
if provenance[1].LaneID != "events" || provenance[1].SlotName != "roster" || provenance[1].Digest != roster.Digest {
t.Fatalf("ReferenceProvenance()[1] = %#v, want roster provenance", provenance[1])
}
encoded, err := json.Marshal(first)
if err != nil {
t.Fatalf("json.Marshal(materialized) error = %v, want nil", err)
}
if strings.Contains(string(encoded), "config text") || strings.Contains(string(encoded), "cli text") {
t.Fatalf("materialized pipeline JSON contains reference content: %s", encoded)
}
}
func TestMaterializeReferencesRejectsNonUTF8Content(t *testing.T) {
configDir := t.TempDir()
path := filepath.Join(configDir, "bad.txt")
writeReferenceFile(t, path, []byte{0xff, 0xfe})
resolved := resolvedPipelineWithReference(t, "roster", "bad.txt", contracts.ReferenceBindingSourceConfig, contracts.ReferenceSlot{Name: "roster"})
_, _, err := MaterializeReferences(resolved, referenceCatalog(t, []contracts.ReferenceSlot{{Name: "roster"}}), ReferenceMaterializationOptions{
ConfigPath: filepath.Join(configDir, "config.yml"),
})
if err == nil || !strings.Contains(err.Error(), "UTF-8") || !strings.Contains(err.Error(), "roster") || !strings.Contains(err.Error(), path) {
t.Fatalf("error = %v, want UTF-8 path error", err)
}
}
func TestMaterializeReferencesWarnsForEmptyFiles(t *testing.T) {
configDir := t.TempDir()
path := filepath.Join(configDir, "empty.txt")
writeReferenceFile(t, path, nil)
resolved := resolvedPipelineWithReference(t, "roster", "empty.txt", contracts.ReferenceBindingSourceConfig, contracts.ReferenceSlot{Name: "roster"})
materialized, warnings, err := MaterializeReferences(resolved, referenceCatalog(t, []contracts.ReferenceSlot{{Name: "roster"}}), ReferenceMaterializationOptions{
ConfigPath: filepath.Join(configDir, "config.yml"),
})
if err != nil {
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
}
if len(warnings) != 1 || warnings[0].ReasonCode != "empty_reference" {
t.Fatalf("warnings = %#v, want empty reference warning", warnings)
}
item := materialized.ArtifactLanes[0].ReferenceSet.Slots["roster"].Items[0]
if item.SizeBytes != 0 || item.Digest != referenceDigest(nil) {
t.Fatalf("empty item = %#v, want zero size and empty digest", item)
}
}
func TestMaterializeReferencesEnforcesMaxBytes(t *testing.T) {
configDir := t.TempDir()
path := filepath.Join(configDir, "large.txt")
writeReferenceFile(t, path, []byte("too large"))
slot := contracts.ReferenceSlot{Name: "roster", MaxBytes: 3}
resolved := resolvedPipelineWithReference(t, "roster", "large.txt", contracts.ReferenceBindingSourceConfig, slot)
_, _, err := MaterializeReferences(resolved, referenceCatalog(t, []contracts.ReferenceSlot{slot}), ReferenceMaterializationOptions{
ConfigPath: filepath.Join(configDir, "config.yml"),
})
if err == nil || !strings.Contains(err.Error(), "9 bytes") || !strings.Contains(err.Error(), "limit 3") || !strings.Contains(err.Error(), "roster") {
t.Fatalf("error = %v, want max bytes error", err)
}
}
func resolvedPipelineWithReference(t *testing.T, slotName, source, bindingSource string, slot contracts.ReferenceSlot) ResolvedPipeline {
t.Helper()
profile := baselineProfile()
lane := profile.Artifacts["events"]
lane.References = map[string]string{slotName: source}
profile.Artifacts["events"] = lane
catalog := referenceCatalog(t, []contracts.ReferenceSlot{slot})
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
if bindingSource != contracts.ReferenceBindingSourceConfig {
resolved.ArtifactLanes[0].References[0].BindingSource = bindingSource
}
return resolved
}
func referenceCatalog(t *testing.T, slots []contracts.ReferenceSlot) ModuleCatalog {
t.Helper()
return newProfileCatalogWithOverride(t, ModuleSpec{
Key: "event-extractor",
Stage: StageExtract,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
ReferenceSlots: slots,
})
}
func writeReferenceFile(t *testing.T, path string, content []byte) {
t.Helper()
if err := os.WriteFile(path, content, 0o644); err != nil {
t.Fatalf("write reference %q: %v", path, err)
}
}

View File

@@ -149,6 +149,10 @@ func (extractor integrationExtractor) SchemaVersion() string {
return "v1"
}
func (extractor integrationExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (extractor integrationExtractor) Validators() []contracts.Validator {
return extractor.validators
}

View File

@@ -41,6 +41,7 @@ type RunInput struct {
StartedAt time.Time
LLMProfiles []artifacts.LLMProfileManifest
Metadata map[string]any
Warnings []contracts.Warning
}
type RunOutput struct {
@@ -63,6 +64,7 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error) {
return output, err
}
output.Warnings = append(output.Warnings, cloneWarnings(input.Warnings)...)
output.Manifest = manifestFromPipeline(input)
adapter, err := r.registries.Inputs.Build(input.Pipeline.Input.Module)
@@ -184,6 +186,7 @@ func (r *Runner) runLane(ctx context.Context, input RunInput, doc *source.Source
result, err := extractor.Extract(ctx, contracts.ExtractionRequest{
Source: doc,
Chunk: &chunk,
References: CloneReferenceSet(lane.ReferenceSet),
LLMClient: input.LLMClient,
LLMProfile: lane.Extract.LLMProfile,
Options: cloneOptions(lane.Extract.Options),
@@ -353,8 +356,12 @@ func manifestFromPipeline(input RunInput) artifacts.RunManifest {
ArtifactLanes: make([]artifacts.ArtifactLaneManifest, 0, len(pipeline.ArtifactLanes)),
RunID: runID,
StartedAt: timePtr(startedAt),
References: ReferenceProvenance(pipeline),
LLMProfiles: cloneLLMProfiles(input.LLMProfiles),
}
// The runner does not currently maintain a cache or idempotency key. Reference
// digests are recorded in manifest provenance and intentionally kept separate
// from source_digests.
for _, lane := range pipeline.ArtifactLanes {
laneManifest := artifacts.ArtifactLaneManifest{
@@ -500,6 +507,13 @@ func cloneLLMProfiles(profiles []artifacts.LLMProfileManifest) []artifacts.LLMPr
return append([]artifacts.LLMProfileManifest(nil), profiles...)
}
func cloneWarnings(warnings []contracts.Warning) []contracts.Warning {
if len(warnings) == 0 {
return nil
}
return append([]contracts.Warning(nil), warnings...)
}
func timePtr(t time.Time) *time.Time {
return &t
}

View File

@@ -559,6 +559,60 @@ func TestRunPassesModuleBindingConfigToStageRequests(t *testing.T) {
}
}
func TestRunPassesLaneReferencesToExtractorRequests(t *testing.T) {
modules := defaultRunnerModules()
pipeline := resolvedPipeline()
pipeline.ArtifactLanes[0].ReferenceSet = contracts.ReferenceSet{
Slots: map[string]contracts.ResolvedReferenceSlot{
"roster": {
Slot: contracts.ReferenceSlot{Name: "roster"},
Items: []contracts.ReferenceItem{
{
SlotName: "roster",
MediaType: "text/plain; charset=utf-8",
Content: []byte("reference text"),
Digest: "sha256:test",
Origin: contracts.ReferenceOrigin{Type: "file", URI: "file:///tmp/reference.txt"},
SizeBytes: int64(len("reference text")),
BindingSource: contracts.ReferenceBindingSourceConfig,
},
},
},
},
}
_, err := New(newRunnerRegistries(t, modules)).Run(context.Background(), RunInput{Pipeline: pipeline})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
req := modules.extractors["extract-alpha"].requests[0]
item := req.References.Slots["roster"].Items[0]
if string(item.Content) != "reference text" {
t.Fatalf("reference content = %q, want reference text", item.Content)
}
item.Content[0] = 'R'
if got := string(pipeline.ArtifactLanes[0].ReferenceSet.Slots["roster"].Items[0].Content); got != "reference text" {
t.Fatalf("runner mutated reference set content = %q", got)
}
}
func TestRunIncludesInputWarnings(t *testing.T) {
modules := defaultRunnerModules()
warning := contracts.Warning{Scope: "reference", ReasonCode: "empty_reference", Message: "empty reference"}
output, err := New(newRunnerRegistries(t, modules)).Run(context.Background(), RunInput{
Pipeline: resolvedPipeline(),
Warnings: []contracts.Warning{warning},
})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if len(output.Warnings) != 1 || output.Warnings[0] != warning {
t.Fatalf("warnings = %#v, want input warning", output.Warnings)
}
}
func TestRunRecordsTopLevelModuleMetadataForSingletonModules(t *testing.T) {
modules := defaultRunnerModules()
modules.input.manifestMetadata = map[string]any{
@@ -906,7 +960,27 @@ func TestRunReturnsFailedManifestWhenOutputEncoderFails(t *testing.T) {
}
func TestRunManifestIncludesPipelineAndLaneDetails(t *testing.T) {
output, err := New(newRunnerRegistries(t, nil)).Run(context.Background(), RunInput{Pipeline: resolvedPipelineWithValidators("configured")})
resolved := resolvedPipelineWithValidators("configured")
resolved.ArtifactLanes[0].ReferenceSet = contracts.ReferenceSet{
Slots: map[string]contracts.ResolvedReferenceSlot{
"roster": {
Slot: contracts.ReferenceSlot{Name: "roster"},
Items: []contracts.ReferenceItem{
{
SlotName: "roster",
MediaType: "text/plain; charset=utf-8",
Content: []byte("reference content"),
Digest: "sha256:reference",
Origin: contracts.ReferenceOrigin{Type: "file", URI: "file:///tmp/roster.txt"},
SizeBytes: int64(len("reference content")),
BindingSource: contracts.ReferenceBindingSourceConfig,
},
},
},
},
}
output, err := New(newRunnerRegistries(t, nil)).Run(context.Background(), RunInput{Pipeline: resolved})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
@@ -921,6 +995,16 @@ func TestRunManifestIncludesPipelineAndLaneDetails(t *testing.T) {
if !reflect.DeepEqual(manifest.SourceDigests, []string{"sha256:source"}) {
t.Fatalf("SourceDigests = %#v, want source digest", manifest.SourceDigests)
}
if len(manifest.References) != 1 {
t.Fatalf("References = %#v, want one reference provenance entry", manifest.References)
}
reference := manifest.References[0]
if reference.LaneID != "alpha" || reference.SlotName != "roster" || reference.Digest != "sha256:reference" {
t.Fatalf("reference provenance = %#v, want lane slot digest", reference)
}
if reference.OriginType != "file" || reference.OriginURI != "file:///tmp/roster.txt" || reference.MediaType != "text/plain; charset=utf-8" || reference.SizeBytes != int64(len("reference content")) || reference.BindingSource != contracts.ReferenceBindingSourceConfig {
t.Fatalf("reference provenance = %#v, want origin/media/size/source", reference)
}
if manifest.ValidationStatus != "approved" {
t.Fatalf("ValidationStatus = %q, want approved", manifest.ValidationStatus)
}
@@ -1269,6 +1353,10 @@ func (extractor *runnerExtractor) SchemaVersion() string {
return extractor.schemaVersion
}
func (extractor *runnerExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (extractor *runnerExtractor) ManifestMetadata() map[string]any {
return extractor.manifestMetadata
}

View File

@@ -252,6 +252,10 @@ func (extractor walkingSkeletonExtractor) SchemaVersion() string {
return "v1"
}
func (extractor walkingSkeletonExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (extractor walkingSkeletonExtractor) Validators() []contracts.Validator {
return nil
}

View File

@@ -7,9 +7,13 @@ import (
"fmt"
"io/fs"
"path"
"reflect"
"sort"
"strings"
"text/template"
"text/template/parse"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
//go:embed assets/**
@@ -43,18 +47,20 @@ func (m Metadata) DiagnosticsMap() map[string]any {
// Definition identifies a caller-owned system/user prompt bundle.
type Definition struct {
PromptID string
Version string
EmbeddedPath string
SystemPath string
UserPath string
PromptID string
Version string
EmbeddedPath string
SystemPath string
UserPath string
ReferenceSlots []contracts.ReferenceSlot
}
// Bundle is a compiled system/user prompt pair.
type Bundle struct {
systemTmpl *template.Template
userTmpl *template.Template
metadata Metadata
systemTmpl *template.Template
userTmpl *template.Template
metadata Metadata
referenceSlots map[string]contracts.ReferenceSlot
}
// Metadata returns metadata for the compiled prompt bundle.
@@ -168,7 +174,9 @@ func LoadBundle(fsys fs.FS, def Definition) (*Bundle, error) {
}
funcs := template.FuncMap{
"hardening": func() string { return sharedHardening },
"hardening": func() string { return sharedHardening },
"reference": func(string) (string, error) { return "", nil },
"hasreference": func(string) (bool, error) { return false, nil },
}
systemTmpl, err := template.New(path.Base(systemPath)).Option("missingkey=error").Funcs(funcs).Parse(systemSource)
if err != nil {
@@ -178,6 +186,13 @@ func LoadBundle(fsys fs.FS, def Definition) (*Bundle, error) {
if err != nil {
return nil, fmt.Errorf("parse embedded user prompt %q: %w", userPath, err)
}
referenceSlots := referenceSlotMap(def.ReferenceSlots)
if err := validateTemplateReferenceSlots(systemTmpl, referenceSlots); err != nil {
return nil, fmt.Errorf("validate embedded system prompt %q: %w", systemPath, err)
}
if err := validateTemplateReferenceSlots(userTmpl, referenceSlots); err != nil {
return nil, fmt.Errorf("validate embedded user prompt %q: %w", userPath, err)
}
hashInput := systemSource + "\n\n" + userSource
hash := sha256.Sum256([]byte(hashInput))
@@ -190,12 +205,127 @@ func LoadBundle(fsys fs.FS, def Definition) (*Bundle, error) {
}
return &Bundle{
systemTmpl: systemTmpl,
userTmpl: userTmpl,
metadata: metadata,
systemTmpl: systemTmpl,
userTmpl: userTmpl,
metadata: metadata,
referenceSlots: referenceSlots,
}, nil
}
func referenceSlotMap(slots []contracts.ReferenceSlot) map[string]contracts.ReferenceSlot {
if len(slots) == 0 {
return nil
}
out := make(map[string]contracts.ReferenceSlot, len(slots))
for _, slot := range slots {
name := strings.TrimSpace(slot.Name)
if name == "" {
continue
}
slot.Name = name
slot.AcceptedMediaTypes = append([]string(nil), slot.AcceptedMediaTypes...)
out[name] = slot
}
return out
}
func validateTemplateReferenceSlots(tmpl *template.Template, declared map[string]contracts.ReferenceSlot) error {
if tmpl == nil || tmpl.Tree == nil || tmpl.Tree.Root == nil {
return nil
}
return validateReferenceNodes(tmpl.Tree.Root, declared)
}
func validateReferenceNodes(node parse.Node, declared map[string]contracts.ReferenceSlot) error {
if node == nil || reflect.ValueOf(node).IsNil() {
return nil
}
switch typed := node.(type) {
case *parse.ListNode:
for _, child := range typed.Nodes {
if err := validateReferenceNodes(child, declared); err != nil {
return err
}
}
case *parse.ActionNode:
return validateReferencePipeline(typed.Pipe, declared)
case *parse.IfNode:
if err := validateReferencePipeline(typed.Pipe, declared); err != nil {
return err
}
if err := validateReferenceNodes(typed.List, declared); err != nil {
return err
}
return validateReferenceNodes(typed.ElseList, declared)
case *parse.RangeNode:
if err := validateReferencePipeline(typed.Pipe, declared); err != nil {
return err
}
if err := validateReferenceNodes(typed.List, declared); err != nil {
return err
}
return validateReferenceNodes(typed.ElseList, declared)
case *parse.WithNode:
if err := validateReferencePipeline(typed.Pipe, declared); err != nil {
return err
}
if err := validateReferenceNodes(typed.List, declared); err != nil {
return err
}
return validateReferenceNodes(typed.ElseList, declared)
case *parse.TemplateNode:
return nil
}
return nil
}
func validateReferencePipeline(pipe *parse.PipeNode, declared map[string]contracts.ReferenceSlot) error {
if pipe == nil {
return nil
}
for _, cmd := range pipe.Cmds {
if err := validateReferenceCommand(cmd, declared); err != nil {
return err
}
}
return nil
}
func validateReferenceCommand(cmd *parse.CommandNode, declared map[string]contracts.ReferenceSlot) error {
if cmd == nil || len(cmd.Args) == 0 {
return nil
}
for _, arg := range cmd.Args[1:] {
if nested, ok := arg.(*parse.PipeNode); ok {
if err := validateReferencePipeline(nested, declared); err != nil {
return err
}
}
}
identifier, ok := cmd.Args[0].(*parse.IdentifierNode)
if !ok {
return nil
}
if identifier.Ident != "reference" && identifier.Ident != "hasreference" {
return nil
}
if len(cmd.Args) != 2 {
return fmt.Errorf("%s requires one string slot name", identifier.Ident)
}
slotArg, ok := cmd.Args[1].(*parse.StringNode)
if !ok {
return fmt.Errorf("%s requires a string literal slot name", identifier.Ident)
}
slotName := strings.TrimSpace(slotArg.Text)
if slotName == "" {
return fmt.Errorf("%s slot name must not be empty", identifier.Ident)
}
if _, ok := declared[slotName]; !ok {
return fmt.Errorf("%s slot %q is not declared", identifier.Ident, slotName)
}
return nil
}
func readPromptAsset(fsys fs.FS, assetPath string) (string, error) {
if strings.TrimSpace(assetPath) == "" {
return "", fmt.Errorf("prompt asset path must not be empty")

View File

@@ -4,6 +4,9 @@ import (
"bytes"
"fmt"
"strings"
"text/template"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
// RenderUserSystem renders the system and user prompt pair for promptID.
@@ -16,20 +19,105 @@ func RenderUserSystem(promptID string, data any) (system string, user string, me
return compiled.RenderUserSystem(data)
}
// RenderUserSystemWithReferences renders the system and user prompt pair for promptID with reference template functions.
func RenderUserSystemWithReferences(promptID string, data any, references contracts.ReferenceSet) (system string, user string, metadata Metadata, err error) {
trimmedID := strings.TrimSpace(promptID)
compiled, ok := promptRegistry[trimmedID]
if !ok {
return "", "", Metadata{}, fmt.Errorf("unknown prompt id %q", promptID)
}
return compiled.RenderUserSystemWithReferences(data, references)
}
// RenderUserSystem renders the bundle's system and user prompts.
func (b *Bundle) RenderUserSystem(data any) (system string, user string, metadata Metadata, err error) {
return b.RenderUserSystemWithReferences(data, contracts.ReferenceSet{})
}
// RenderUserSystemWithReferences renders the bundle's system and user prompts with reference template functions.
func (b *Bundle) RenderUserSystemWithReferences(data any, references contracts.ReferenceSet) (system string, user string, metadata Metadata, err error) {
if b == nil {
return "", "", Metadata{}, fmt.Errorf("prompt bundle must not be nil")
}
systemTmpl, userTmpl, err := b.renderTemplates(references)
if err != nil {
return "", "", Metadata{}, err
}
var systemBuf bytes.Buffer
if err := b.systemTmpl.Execute(&systemBuf, data); err != nil {
if err := systemTmpl.Execute(&systemBuf, data); err != nil {
return "", "", Metadata{}, fmt.Errorf("render system prompt %q: %w", b.metadata.PromptID, err)
}
var userBuf bytes.Buffer
if err := b.userTmpl.Execute(&userBuf, data); err != nil {
if err := userTmpl.Execute(&userBuf, data); err != nil {
return "", "", Metadata{}, fmt.Errorf("render user prompt %q: %w", b.metadata.PromptID, err)
}
return strings.TrimSpace(systemBuf.String()), strings.TrimSpace(userBuf.String()), b.metadata, nil
}
func (b *Bundle) renderTemplates(references contracts.ReferenceSet) (*template.Template, *template.Template, error) {
funcs := b.referenceFuncs(references)
systemTmpl, err := b.systemTmpl.Clone()
if err != nil {
return nil, nil, fmt.Errorf("clone system prompt %q: %w", b.metadata.PromptID, err)
}
userTmpl, err := b.userTmpl.Clone()
if err != nil {
return nil, nil, fmt.Errorf("clone user prompt %q: %w", b.metadata.PromptID, err)
}
systemTmpl.Funcs(funcs)
userTmpl.Funcs(funcs)
return systemTmpl, userTmpl, nil
}
func (b *Bundle) referenceFuncs(references contracts.ReferenceSet) template.FuncMap {
return template.FuncMap{
"hardening": func() string { return sharedHardening },
"hasreference": func(slotName string) (bool, error) {
items, _, err := b.referenceItems(slotName, references)
if err != nil {
return false, err
}
for _, item := range items {
if len(item.Content) > 0 {
return true, nil
}
}
return false, nil
},
"reference": func(slotName string) (string, error) {
items, slot, err := b.referenceItems(slotName, references)
if err != nil {
return "", err
}
if len(items) == 0 {
return "", nil
}
if len(items) > 1 && !slot.Multiple {
return "", fmt.Errorf("reference slot %q has %d bound items but does not allow multiple", slot.Name, len(items))
}
parts := make([]string, 0, len(items))
for _, item := range items {
parts = append(parts, string(item.Content))
}
return strings.Join(parts, "\n"), nil
},
}
}
func (b *Bundle) referenceItems(slotName string, references contracts.ReferenceSet) ([]contracts.ReferenceItem, contracts.ReferenceSlot, error) {
slotName = strings.TrimSpace(slotName)
slot, ok := b.referenceSlots[slotName]
if !ok {
return nil, contracts.ReferenceSlot{}, fmt.Errorf("reference slot %q is not declared", slotName)
}
if len(references.Slots) == 0 {
return nil, slot, nil
}
resolved, ok := references.Slots[slotName]
if !ok {
return nil, slot, nil
}
return append([]contracts.ReferenceItem(nil), resolved.Items...), slot, nil
}

View File

@@ -1,8 +1,13 @@
package prompt
import (
"crypto/sha256"
"encoding/hex"
"strings"
"testing"
"testing/fstest"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestRenderUserSystemReturnsTextAndMetadata(t *testing.T) {
@@ -64,3 +69,175 @@ func TestRenderUserSystemIncludesHardeningText(t *testing.T) {
t.Fatalf("expected rendered system prompt to include hardening text: %q", system)
}
}
func TestRenderUserSystemWithReferencesRendersDeclaredSlots(t *testing.T) {
bundle := loadReferenceBundle(t,
[]contracts.ReferenceSlot{{Name: "roster"}, {Name: "glossary"}},
`System has roster={{ hasreference "roster" }} has glossary={{ hasreference "glossary" }}`,
`Roster={{ reference "roster" }} Glossary={{ reference "glossary" }}`,
)
references := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
"roster": {
Slot: contracts.ReferenceSlot{Name: "roster"},
Items: []contracts.ReferenceItem{
{SlotName: "roster", Content: []byte("Aria")},
},
},
}}
system, user, _, err := bundle.RenderUserSystemWithReferences(map[string]any{}, references)
if err != nil {
t.Fatalf("RenderUserSystemWithReferences: %v", err)
}
if !strings.Contains(system, "has roster=true") || !strings.Contains(system, "has glossary=false") {
t.Fatalf("system = %q, want reference presence flags", system)
}
if !strings.Contains(user, "Roster=Aria") || !strings.Contains(user, "Glossary=") {
t.Fatalf("user = %q, want rendered and empty optional references", user)
}
}
func TestRenderUserSystemReferenceHasReferenceRequiresContent(t *testing.T) {
bundle := loadReferenceBundle(t,
[]contracts.ReferenceSlot{{Name: "roster"}},
`System`,
`{{ hasreference "roster" }} {{ reference "roster" }}`,
)
references := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
"roster": {
Slot: contracts.ReferenceSlot{Name: "roster"},
Items: []contracts.ReferenceItem{
{SlotName: "roster", Content: nil},
},
},
}}
_, user, _, err := bundle.RenderUserSystemWithReferences(map[string]any{}, references)
if err != nil {
t.Fatalf("RenderUserSystemWithReferences: %v", err)
}
if user != "false" {
t.Fatalf("user = %q, want false with empty reference content", user)
}
}
func TestLoadBundleRejectsUndeclaredReferenceSlots(t *testing.T) {
_, err := LoadBundle(referenceBundleFS(`System`, `{{ reference "roster" }}`), referenceBundleDefinition(nil))
if err == nil || !strings.Contains(err.Error(), "roster") || !strings.Contains(err.Error(), "not declared") {
t.Fatalf("LoadBundle() error = %v, want undeclared reference slot error", err)
}
}
func TestLoadBundleRejectsDynamicReferenceSlotNames(t *testing.T) {
_, err := LoadBundle(referenceBundleFS(`System`, `{{ reference .SlotName }}`), referenceBundleDefinition([]contracts.ReferenceSlot{{Name: "roster"}}))
if err == nil || !strings.Contains(err.Error(), "string literal") {
t.Fatalf("LoadBundle() error = %v, want string literal error", err)
}
}
func TestLoadBundleRejectsNestedUndeclaredReferenceSlots(t *testing.T) {
_, err := LoadBundle(referenceBundleFS(`System`, `{{ printf "%s" (reference "roster") }}`), referenceBundleDefinition(nil))
if err == nil || !strings.Contains(err.Error(), "roster") || !strings.Contains(err.Error(), "not declared") {
t.Fatalf("LoadBundle() error = %v, want nested undeclared reference slot error", err)
}
}
func TestRenderUserSystemRejectsMultipleReferenceItemsUnlessDeclared(t *testing.T) {
bundle := loadReferenceBundle(t,
[]contracts.ReferenceSlot{{Name: "roster"}},
`System`,
`{{ reference "roster" }}`,
)
references := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
"roster": {
Slot: contracts.ReferenceSlot{Name: "roster"},
Items: []contracts.ReferenceItem{
{SlotName: "roster", Content: []byte("Aria")},
{SlotName: "roster", Content: []byte("Bryn")},
},
},
}}
_, _, _, err := bundle.RenderUserSystemWithReferences(map[string]any{}, references)
if err == nil || !strings.Contains(err.Error(), "does not allow multiple") {
t.Fatalf("RenderUserSystemWithReferences() error = %v, want multiple item error", err)
}
}
func TestRenderUserSystemRendersMultipleReferenceItemsDeterministicallyWhenDeclared(t *testing.T) {
bundle := loadReferenceBundle(t,
[]contracts.ReferenceSlot{{Name: "roster", Multiple: true}},
`System`,
`{{ reference "roster" }}`,
)
references := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
"roster": {
Slot: contracts.ReferenceSlot{Name: "roster", Multiple: true},
Items: []contracts.ReferenceItem{
{SlotName: "roster", Content: []byte("Aria")},
{SlotName: "roster", Content: []byte("Bryn")},
},
},
}}
_, first, _, err := bundle.RenderUserSystemWithReferences(map[string]any{}, references)
if err != nil {
t.Fatalf("RenderUserSystemWithReferences(first): %v", err)
}
_, second, _, err := bundle.RenderUserSystemWithReferences(map[string]any{}, references)
if err != nil {
t.Fatalf("RenderUserSystemWithReferences(second): %v", err)
}
if first != "Aria\nBryn" || first != second {
t.Fatalf("rendered references = %q/%q, want deterministic item order", first, second)
}
}
func TestPromptMetadataHashIgnoresRenderedReferenceContent(t *testing.T) {
systemSource := `System`
userSource := `{{ reference "roster" }}`
bundle := loadReferenceBundle(t, []contracts.ReferenceSlot{{Name: "roster"}}, systemSource, userSource)
references := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
"roster": {
Slot: contracts.ReferenceSlot{Name: "roster"},
Items: []contracts.ReferenceItem{{SlotName: "roster", Content: []byte("Aria")}},
},
}}
_, _, metadata, err := bundle.RenderUserSystemWithReferences(map[string]any{}, references)
if err != nil {
t.Fatalf("RenderUserSystemWithReferences: %v", err)
}
hash := sha256.Sum256([]byte(systemSource + "\n\n" + userSource))
want := "sha256:" + hex.EncodeToString(hash[:])
if metadata.SHA256 != want {
t.Fatalf("metadata.SHA256 = %q, want template source hash %q", metadata.SHA256, want)
}
}
func loadReferenceBundle(t *testing.T, slots []contracts.ReferenceSlot, systemSource string, userSource string) *Bundle {
t.Helper()
bundle, err := LoadBundle(referenceBundleFS(systemSource, userSource), referenceBundleDefinition(slots))
if err != nil {
t.Fatalf("LoadBundle() error = %v, want nil", err)
}
return bundle
}
func referenceBundleDefinition(slots []contracts.ReferenceSlot) Definition {
return Definition{
PromptID: "test.references",
Version: VersionV1,
EmbeddedPath: "assets/test/references",
SystemPath: "assets/test/references/system.md",
UserPath: "assets/test/references/user.md",
ReferenceSlots: slots,
}
}
func referenceBundleFS(systemSource string, userSource string) fstest.MapFS {
return fstest.MapFS{
"assets/test/references/system.md": {Data: []byte(systemSource)},
"assets/test/references/user.md": {Data: []byte(userSource)},
}
}

View File

@@ -6,4 +6,9 @@ Extract only spell casts that are supported by the provided source text. Do not
infer spells from general D&D knowledge or from table chatter that does not
identify a spell being cast.
Reference material, when present, is supporting context only. Use it only to
disambiguate names, aliases, speakers, campaign terms, or spell names already
present in the source text. Do not extract a spell cast solely because it appears
in reference material.
Source references must use the source-unit IDs exactly as provided.

View File

@@ -16,6 +16,19 @@ Source units:
{{ end }}
{{ end }}
{{ if hasreference "roster" }}
Roster reference material:
{{ reference "roster" }}
{{ end }}
{{ if hasreference "glossary" }}
Glossary reference material:
{{ reference "glossary" }}
{{ end }}
Return only D&D spell-cast artifacts. For each spell cast, identify the in-world
caster, spell name, effect, narrative description, and source references using
source_id, start_unit_id, and end_unit_id.
Use roster and glossary reference material only to clarify source text. Do not
return spells, casters, or effects that are mentioned only in reference material.

View File

@@ -25,6 +25,19 @@ var providedCapabilities = []string{
"dnd.spell_casts",
}
var referenceSlots = []contracts.ReferenceSlot{
{
Name: "glossary",
Description: "Optional campaign glossary reference material used only for disambiguation.",
AcceptedMediaTypes: []string{"text/plain; charset=utf-8"},
},
{
Name: "roster",
Description: "Optional campaign roster or player-character reference material used only for disambiguation.",
AcceptedMediaTypes: []string{"text/plain; charset=utf-8"},
},
}
var _ contracts.Extractor = (*Extractor)(nil)
type Extractor struct{}
@@ -45,6 +58,10 @@ func (e *Extractor) SchemaVersion() string {
return SchemaVersion
}
func (e *Extractor) ReferenceSlots() []contracts.ReferenceSlot {
return cloneReferenceSlots(referenceSlots)
}
func (e *Extractor) ManifestMetadata() map[string]any {
promptMetadata := spellsPromptBundle.Metadata()
metadata := map[string]any{
@@ -136,10 +153,11 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.ExtractionRequest
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageExtract,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
Key: Key,
Stage: pipeline.StageExtract,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ReferenceSlots: cloneReferenceSlots(referenceSlots),
}
}
@@ -161,3 +179,15 @@ func spellCastPayload(spellCast spellCastResponse) (json.RawMessage, error) {
func extractorErrorf(format string, args ...any) error {
return fmt.Errorf("dnd spells extractor: "+format, args...)
}
func cloneReferenceSlots(slots []contracts.ReferenceSlot) []contracts.ReferenceSlot {
if len(slots) == 0 {
return nil
}
out := make([]contracts.ReferenceSlot, len(slots))
for i, slot := range slots {
slot.AcceptedMediaTypes = append([]string(nil), slot.AcceptedMediaTypes...)
out[i] = slot
}
return out
}

View File

@@ -116,6 +116,56 @@ func TestExtractorManifestMetadataIncludesPromptAndSchemaProvenance(t *testing.T
}
}
func TestExtractIncludesReferencesInPrompt(t *testing.T) {
client := &fakeSpellsLLMClient{response: extractionResponse{SpellCasts: []spellCastResponse{}}}
req := extractionRequestWithClient(client)
req.References = contracts.ReferenceSet{
Slots: map[string]contracts.ResolvedReferenceSlot{
"roster": {
Slot: contracts.ReferenceSlot{Name: "roster"},
Items: []contracts.ReferenceItem{
{SlotName: "roster", Content: []byte("Aria Brightmantle: party cleric")},
},
},
"glossary": {
Slot: contracts.ReferenceSlot{Name: "glossary"},
Items: []contracts.ReferenceItem{
{SlotName: "glossary", Content: []byte("Brightmantle: local temple name")},
},
},
},
}
if _, err := New().Extract(context.Background(), req); err != nil {
t.Fatalf("Extract() error = %v, want nil", err)
}
if len(client.requests) != 1 {
t.Fatalf("LLM calls = %d, want 1", len(client.requests))
}
system := client.requests[0].Messages[0].Content
for _, want := range []string{
"Reference material, when present, is supporting context only.",
"in reference material.",
} {
if !strings.Contains(system, want) {
t.Fatalf("system prompt = %q, want substring %q", system, want)
}
}
user := client.requests[0].Messages[1].Content
for _, want := range []string{
"Roster reference material:",
"Aria Brightmantle: party cleric",
"Glossary reference material:",
"Brightmantle: local temple name",
"mentioned only in reference material.",
} {
if !strings.Contains(user, want) {
t.Fatalf("user prompt = %q, want substring %q", user, want)
}
}
}
func TestExtractReturnsNoCandidatesForEmptyResponse(t *testing.T) {
client := &fakeSpellsLLMClient{response: extractionResponse{SpellCasts: []spellCastResponse{}}}

View File

@@ -32,11 +32,12 @@ var spellsPromptBundle = mustLoadPromptBundle()
func mustLoadPromptBundle() *prompt.Bundle {
bundle, err := prompt.LoadBundle(embeddedAssets, prompt.Definition{
PromptID: PromptID,
Version: SchemaVersion,
EmbeddedPath: "assets/prompts",
SystemPath: "assets/prompts/system.md",
UserPath: "assets/prompts/user.md",
PromptID: PromptID,
Version: SchemaVersion,
EmbeddedPath: "assets/prompts",
SystemPath: "assets/prompts/system.md",
UserPath: "assets/prompts/user.md",
ReferenceSlots: cloneReferenceSlots(referenceSlots),
})
if err != nil {
panic(err)
@@ -74,7 +75,7 @@ func renderPrompt(req contracts.ExtractionRequest) (system string, user string,
if err != nil {
return "", "", prompt.Metadata{}, err
}
system, user, metadata, err = spellsPromptBundle.RenderUserSystem(data)
system, user, metadata, err = spellsPromptBundle.RenderUserSystemWithReferences(data, req.References)
if err != nil {
return "", "", prompt.Metadata{}, fmt.Errorf("dnd spells prompt: %w", err)
}

View File

@@ -101,6 +101,50 @@ func TestRenderPromptIncludesSourceContext(t *testing.T) {
if metadata.EmbeddedPath != "assets/prompts" {
t.Fatalf("metadata.EmbeddedPath = %q, want assets/prompts", metadata.EmbeddedPath)
}
if strings.Contains(user, "Roster reference material") || strings.Contains(user, "Glossary reference material") {
t.Fatalf("user prompt = %q, want no optional reference sections without bindings", user)
}
}
func TestRenderPromptIncludesBoundReferences(t *testing.T) {
req := promptExtractionRequest()
req.References = contracts.ReferenceSet{
Slots: map[string]contracts.ResolvedReferenceSlot{
"roster": {
Slot: contracts.ReferenceSlot{Name: "roster"},
Items: []contracts.ReferenceItem{
{SlotName: "roster", Content: []byte("Aria: cleric, also known as Sister Aria")},
},
},
"glossary": {
Slot: contracts.ReferenceSlot{Name: "glossary"},
Items: []contracts.ReferenceItem{
{SlotName: "glossary", Content: []byte("Cure Wounds: healing spell")},
},
},
},
}
_, user, metadata, err := renderPrompt(req)
if err != nil {
t.Fatalf("renderPrompt() error = %v, want nil", err)
}
for _, want := range []string{
"Roster reference material:",
"Aria: cleric, also known as Sister Aria",
"Glossary reference material:",
"Cure Wounds: healing spell",
"Use roster and glossary reference material only to clarify source text.",
"mentioned only in reference material.",
} {
if !strings.Contains(user, want) {
t.Fatalf("user prompt = %q, want substring %q", user, want)
}
}
if metadata.SHA256 != spellsPromptBundle.Metadata().SHA256 {
t.Fatalf("metadata.SHA256 = %q, want template hash %q", metadata.SHA256, spellsPromptBundle.Metadata().SHA256)
}
}
func TestBuildPromptDataRejectsMissingSourceContext(t *testing.T) {

View File

@@ -5,6 +5,7 @@ import (
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
@@ -36,6 +37,18 @@ func TestModuleSpec(t *testing.T) {
Provides: []string{
"dnd.spell_casts",
},
ReferenceSlots: []contracts.ReferenceSlot{
{
Name: "glossary",
Description: "Optional campaign glossary reference material used only for disambiguation.",
AcceptedMediaTypes: []string{"text/plain; charset=utf-8"},
},
{
Name: "roster",
Description: "Optional campaign roster or player-character reference material used only for disambiguation.",
AcceptedMediaTypes: []string{"text/plain; charset=utf-8"},
},
},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
@@ -43,6 +56,7 @@ func TestModuleSpec(t *testing.T) {
got.Requires[0] = "changed"
got.Provides[0] = "changed"
got.ReferenceSlots[0].AcceptedMediaTypes[0] = "changed"
again := ModuleSpec()
if !reflect.DeepEqual(again, want) {
t.Fatalf("ModuleSpec() after caller mutation = %#v, want %#v", again, want)
@@ -82,6 +96,15 @@ func TestRegisterStoresModuleSpec(t *testing.T) {
}
}
func TestRuntimeReferenceSlotsMatchModuleSpec(t *testing.T) {
extractor := New()
spec := ModuleSpec()
if !reflect.DeepEqual(extractor.ReferenceSlots(), spec.ReferenceSlots) {
t.Fatalf("ReferenceSlots() = %#v, want spec slots %#v", extractor.ReferenceSlots(), spec.ReferenceSlots)
}
}
func TestRegisterNilRegistryReturnsError(t *testing.T) {
err := Register(nil)
if err == nil {

View File

@@ -110,6 +110,96 @@ func TestRunnerProcessesSeriatimInputWithDNDSpellsExtractor(t *testing.T) {
}
}
func TestRunnerPassesRosterAndGlossaryReferencesToDNDSpellsPrompt(t *testing.T) {
raw := readDNDSpellsFixture(t)
expectedDoc := parseDNDSpellsFixture(t, raw)
resolved := resolveDNDSpellsPipeline(t)
resolved.ResolvedPipeline.ArtifactLanes[0].ReferenceSet = dndSpellsReferenceSet(
"Aria: party cleric\nBorin: fighter",
"Fire Bolt: evocation cantrip",
)
llmClient := &fakeSpellsLLMClient{
response: extractionResponse{
SpellCasts: []spellCastResponse{
{
Caster: "Borin",
Spell: "Fire Bolt",
Effect: "Scorches the wight.",
NarrativeDescription: "Borin hurls fire at the wight.",
SourceRefs: []source.SourceRef{
{SourceID: expectedDoc.ID, StartUnitID: "seg-003", EndUnitID: "seg-003"},
},
},
},
},
}
output, err := pipeline.New(dndSpellsRunnerRegistries(t)).Run(context.Background(), pipeline.RunInput{
Pipeline: resolved.ResolvedPipeline,
RawInput: raw,
LLMClient: llmClient,
})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if len(output.Approved) != 1 {
t.Fatalf("len(Approved) = %d, want 1", len(output.Approved))
}
if len(output.Manifest.References) != 2 {
t.Fatalf("manifest references = %#v, want roster and glossary provenance", output.Manifest.References)
}
if len(llmClient.requests) != 1 {
t.Fatalf("LLM calls = %d, want 1", len(llmClient.requests))
}
user := llmClient.requests[0].Messages[1].Content
for _, want := range []string{
"Roster reference material:",
"Aria: party cleric",
"Glossary reference material:",
"Fire Bolt: evocation cantrip",
} {
if !strings.Contains(user, want) {
t.Fatalf("user prompt = %q, want substring %q", user, want)
}
}
}
func TestRunnerDoesNotExtractSpellMentionedOnlyInRoster(t *testing.T) {
raw := readDNDSpellsFixture(t)
resolved := resolveDNDSpellsPipeline(t)
resolved.ResolvedPipeline.ArtifactLanes[0].ReferenceSet = dndSpellsReferenceSet(
"Mira: wizard who can cast Lightning Bolt",
"",
)
llmClient := &fakeSpellsLLMClient{
response: extractionResponse{SpellCasts: []spellCastResponse{}},
}
output, err := pipeline.New(dndSpellsRunnerRegistries(t)).Run(context.Background(), pipeline.RunInput{
Pipeline: resolved.ResolvedPipeline,
RawInput: raw,
LLMClient: llmClient,
})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if len(output.Approved) != 0 {
t.Fatalf("approved artifacts = %#v, want no roster-only spell casts", output.Approved)
}
if len(llmClient.requests) != 1 {
t.Fatalf("LLM calls = %d, want 1", len(llmClient.requests))
}
user := llmClient.requests[0].Messages[1].Content
if !strings.Contains(user, "Lightning Bolt") {
t.Fatalf("user prompt = %q, want roster-only spell in reference section", user)
}
if output.Manifest.ValidationStatus != "approved" {
t.Fatalf("ValidationStatus = %q, want approved empty extraction", output.Manifest.ValidationStatus)
}
}
func TestRunnerRejectsDNDSpellCastWithInvalidSourceRef(t *testing.T) {
raw := readDNDSpellsFixture(t)
resolved := resolveDNDSpellsPipeline(t)
@@ -155,6 +245,43 @@ func TestRunnerRejectsDNDSpellCastWithInvalidSourceRef(t *testing.T) {
}
}
func dndSpellsReferenceSet(roster string, glossary string) contracts.ReferenceSet {
slots := make(map[string]contracts.ResolvedReferenceSlot)
if strings.TrimSpace(roster) != "" {
slots["roster"] = contracts.ResolvedReferenceSlot{
Slot: contracts.ReferenceSlot{Name: "roster"},
Items: []contracts.ReferenceItem{
{
SlotName: "roster",
MediaType: "text/plain; charset=utf-8",
Content: []byte(roster),
Digest: "sha256:roster",
Origin: contracts.ReferenceOrigin{Type: "file", URI: "file:///tmp/roster.txt"},
SizeBytes: int64(len(roster)),
BindingSource: contracts.ReferenceBindingSourceConfig,
},
},
}
}
if strings.TrimSpace(glossary) != "" {
slots["glossary"] = contracts.ResolvedReferenceSlot{
Slot: contracts.ReferenceSlot{Name: "glossary"},
Items: []contracts.ReferenceItem{
{
SlotName: "glossary",
MediaType: "text/plain; charset=utf-8",
Content: []byte(glossary),
Digest: "sha256:glossary",
Origin: contracts.ReferenceOrigin{Type: "file", URI: "file:///tmp/glossary.txt"},
SizeBytes: int64(len(glossary)),
BindingSource: contracts.ReferenceBindingSourceConfig,
},
},
}
}
return contracts.ReferenceSet{Slots: slots}
}
func TestRunnerFailsWhenDNDSpellsExtractorReturnsMalformedOutput(t *testing.T) {
raw := readDNDSpellsFixture(t)
resolved := resolveDNDSpellsPipeline(t)

View File

@@ -20,6 +20,7 @@ const (
reasonMissingRequiredField = "missing_required_field"
reasonMissingSourceRef = "missing_source_ref"
reasonInvalidSourceRef = "invalid_source_ref"
reasonSpellNotNearSource = "spell_not_near_source"
)
var _ contracts.Validator = ShapeValidator{}
@@ -54,12 +55,15 @@ func (validator SourceRefValidator) Validate(ctx context.Context, req contracts.
}
decisions := make([]contracts.ValidationDecision, 0, len(req.Candidates))
var warnings []contracts.Warning
for _, candidate := range req.Candidates {
decisions = append(decisions, validateSourceRefs(req.Source, candidate))
warnings = append(warnings, sourceRelatednessWarnings(req.Source, candidate)...)
}
return contracts.ValidationResult{
ValidatorName: validator.Name(),
Decisions: decisions,
Warnings: warnings,
}, nil
}
@@ -88,6 +92,66 @@ func validateSourceRefs(doc *source.SourceDocument, candidate artifacts.Artifact
return validate.Approved(candidate.Index)
}
func sourceRelatednessWarnings(doc *source.SourceDocument, candidate artifacts.ArtifactCandidate) []contracts.Warning {
if doc == nil || len(candidate.SourceRefs) == 0 {
return nil
}
var payload SpellCast
if err := json.Unmarshal(candidate.Payload, &payload); err != nil {
return nil
}
spell := strings.TrimSpace(payload.Spell)
if spell == "" {
return nil
}
needle := strings.ToLower(spell)
for _, ref := range candidate.SourceRefs {
text, ok := sourceRefText(doc, ref)
if !ok {
continue
}
if strings.Contains(strings.ToLower(text), needle) {
return nil
}
}
return []contracts.Warning{
{
Scope: fmt.Sprintf("candidate.%d", candidate.Index),
ReasonCode: reasonSpellNotNearSource,
Message: fmt.Sprintf("spell %q was not found in the cited source text", spell),
},
}
}
func sourceRefText(doc *source.SourceDocument, ref source.SourceRef) (string, bool) {
if err := source.ValidateRef(doc, ref); err != nil {
return "", false
}
start := -1
end := -1
for i, unit := range doc.Units {
if unit.ID == ref.StartUnitID {
start = i
}
if unit.ID == ref.EndUnitID {
end = i
}
}
if start < 0 || end < start {
return "", false
}
var b strings.Builder
for i := start; i <= end; i++ {
if b.Len() > 0 {
b.WriteString("\n")
}
b.WriteString(doc.Units[i].Text)
}
return b.String(), true
}
func requiredSpellCastFields(payload SpellCast) []struct {
name string
value string

View File

@@ -51,6 +51,9 @@ func TestValidatorsApproveValidCandidate(t *testing.T) {
t.Fatalf("SourceRefValidator.Validate() error = %v, want nil", err)
}
assertSingleDecision(t, sourceRefResult, sourceRefValidatorName, 7, true, validate.ReasonApproved)
if len(sourceRefResult.Warnings) != 0 {
t.Fatalf("warnings = %#v, want none", sourceRefResult.Warnings)
}
}
func TestShapeValidatorRejectsMalformedPayload(t *testing.T) {
@@ -167,6 +170,29 @@ func TestSourceRefValidatorRejectsInvalidRefs(t *testing.T) {
}
}
func TestSourceRefValidatorWarnsWhenSpellNameIsNotInCitedSource(t *testing.T) {
candidate := validSpellCandidate(31)
payload := validSpellPayload()
payload.Spell = "Shield"
candidate.Payload = mustSpellPayload(t, payload)
result, err := SourceRefValidator{}.Validate(context.Background(), contracts.ValidationRequest{
Source: promptSourceDocument(),
Candidates: []artifacts.ArtifactCandidate{candidate},
})
if err != nil {
t.Fatalf("SourceRefValidator.Validate() error = %v, want nil", err)
}
assertSingleDecision(t, result, sourceRefValidatorName, 31, true, validate.ReasonApproved)
if len(result.Warnings) != 1 {
t.Fatalf("warnings = %#v, want one relatedness warning", result.Warnings)
}
warning := result.Warnings[0]
if warning.ReasonCode != reasonSpellNotNearSource || !strings.Contains(warning.Message, "Shield") {
t.Fatalf("warning = %#v, want spell relatedness warning", warning)
}
}
func TestSourceRefValidatorRequiresSourceDocument(t *testing.T) {
_, err := SourceRefValidator{}.Validate(context.Background(), contracts.ValidationRequest{
Candidates: []artifacts.ArtifactCandidate{validSpellCandidate(19)},

View File

@@ -221,6 +221,8 @@ func (fakeExtractor) ArtifactType() string { return "fake" }
func (fakeExtractor) SchemaVersion() string { return "v1" }
func (fakeExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (fakeExtractor) Validators() []contracts.Validator { return nil }
func (fakeExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {

View File

@@ -186,6 +186,10 @@ func (e *runnerSeriatimExtractor) SchemaVersion() string {
return "v1"
}
func (e *runnerSeriatimExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (e *runnerSeriatimExtractor) Validators() []contracts.Validator {
return nil
}

View File

@@ -142,6 +142,42 @@ func TestEncodePrettyPrintsJSON(t *testing.T) {
}
}
func TestEncodeIncludesManifestReferences(t *testing.T) {
result, err := New().Encode(context.Background(), contracts.OutputRequest{
Manifest: artifacts.RunManifest{
RunID: "run-1",
References: []artifacts.ReferenceProvenance{
{
LaneID: "events",
SlotName: "roster",
OriginType: "file",
OriginURI: "file:///tmp/roster.txt",
Digest: "sha256:reference",
MediaType: "text/plain; charset=utf-8",
SizeBytes: 12,
BindingSource: "config",
},
},
},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
manifest := decodeObject(t, fileBytes(t, result.Files, "manifest.json"))
references := manifest["references"].([]any)
if len(references) != 1 {
t.Fatalf("references = %#v, want one entry", references)
}
reference := references[0].(map[string]any)
if reference["lane_id"] != "events" || reference["slot_name"] != "roster" || reference["digest"] != "sha256:reference" {
t.Fatalf("reference manifest = %#v, want lane slot digest", reference)
}
if _, ok := reference["content"]; ok {
t.Fatalf("reference manifest = %#v, want no content field", reference)
}
}
func TestEncodeRejectsArtifactTypeWithoutSafeFileName(t *testing.T) {
_, err := New().Encode(context.Background(), contracts.OutputRequest{
Approved: []artifacts.Artifact{artifact("///", "unsafe")},