Materialize references for all eligible targets
This commit is contained in:
11
docs/cli.md
11
docs/cli.md
@@ -58,11 +58,12 @@ the warning count is printed to stderr.
|
|||||||
|
|
||||||
Reference flags are resolved against selected chunk, extractor, and normalizer
|
Reference flags are resolved against selected chunk, extractor, and normalizer
|
||||||
targets before the run starts. Flat slot names are accepted only when exactly
|
targets before the run starts. Flat slot names are accepted only when exactly
|
||||||
one selected target declares that slot. Bound extractor reference files are read
|
one selected target declares that slot. Bound reference files are read before
|
||||||
before extraction, validated as UTF-8 text, and passed only to the lane
|
pipeline work starts, validated as UTF-8 text, and recorded as provenance for
|
||||||
extractor that declares the slot. Notarius infers reference media types from
|
the target that declares the slot. Runtime reference content is currently passed
|
||||||
file extensions for provenance and for optional slot checks. Reference content
|
only to lane extractors. Notarius infers reference media types from file
|
||||||
is not written to diagnostics, logs, errors, or manifests.
|
extensions for provenance and for optional slot checks. Reference content is not
|
||||||
|
written to diagnostics, logs, errors, or manifests.
|
||||||
|
|
||||||
Reference binding precedence is:
|
Reference binding precedence is:
|
||||||
|
|
||||||
|
|||||||
@@ -148,12 +148,14 @@ must be bound after config defaults, target-local references, lane-level
|
|||||||
compatibility bindings, and run-time `--reference` or `--without-reference`
|
compatibility bindings, and run-time `--reference` or `--without-reference`
|
||||||
overrides are applied. Config-relative paths are resolved relative to the config
|
overrides are applied. Config-relative paths are resolved relative to the config
|
||||||
file; CLI reference paths are resolved relative to the current working
|
file; CLI reference paths are resolved relative to the current working
|
||||||
directory. Materialized bound files must be UTF-8 text and are currently passed
|
directory. Materialized bound files must be UTF-8 text. Materialized reference
|
||||||
only to lane extractors that declare the slot. Reference media types are
|
provenance is recorded for chunk, extractor, and normalizer targets; runtime
|
||||||
inferred from file extensions, recorded as canonical base media types, and
|
reference content is currently passed only to lane extractors that declare the
|
||||||
checked only when a module declares `AcceptedMediaTypes`; unknown extensions are
|
slot. Reference media types are inferred from file extensions, recorded as
|
||||||
recorded as `application/octet-stream`. Reference content is not written to
|
canonical base media types, and checked only when a module declares
|
||||||
diagnostics, logs, errors, or manifests.
|
`AcceptedMediaTypes`; unknown extensions are recorded as
|
||||||
|
`application/octet-stream`. Reference content is not written to diagnostics,
|
||||||
|
logs, errors, or manifests.
|
||||||
|
|
||||||
Pipeline-level `references` are defaults. They are valid when at least one
|
Pipeline-level `references` are defaults. They are valid when at least one
|
||||||
eligible target in the full configured pipeline declares the slot, including
|
eligible target in the full configured pipeline declares the slot, including
|
||||||
|
|||||||
@@ -104,6 +104,9 @@ target stage, lane ID when present, slot name, origin type and URI, digest,
|
|||||||
media type, byte size, and binding source. Reference content is not written to
|
media type, byte size, and binding source. Reference content is not written to
|
||||||
durable output.
|
durable output.
|
||||||
|
|
||||||
|
Reference `stage` is `chunk`, `extract`, or `normalize`. `lane_id` is omitted
|
||||||
|
for chunk references and present for extract and normalize references.
|
||||||
|
|
||||||
When references are bound, the manifest section has this shape:
|
When references are bound, the manifest section has this shape:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
|
|||||||
@@ -31,14 +31,15 @@ must still be UTF-8 text. When a slot declares accepted media types, Notarius
|
|||||||
compares the canonical base media type inferred from the file extension,
|
compares the canonical base media type inferred from the file extension,
|
||||||
case-insensitively and without parameters.
|
case-insensitively and without parameters.
|
||||||
|
|
||||||
The current resolver materializes reference content only for lane extractors
|
The resolver materializes reference content for chunk, extractor, and
|
||||||
through `contracts.ExtractionRequest.References`. It is not source evidence and
|
normalizer targets. Runtime delivery is currently implemented only for lane
|
||||||
must not be converted into `SourceRef` values. If a module prompt uses
|
extractors through `contracts.ExtractionRequest.References`. Reference material
|
||||||
references, load the prompt bundle with the same declared slots and render with
|
is not source evidence and must not be converted into `SourceRef` values. If a
|
||||||
`RenderUserSystemWithReferences`. Prompt templates may use the `reference`
|
module prompt uses references, load the prompt bundle with the same declared
|
||||||
function for content and the `hasreference` function for conditional sections.
|
slots and render with `RenderUserSystemWithReferences`. Prompt templates may use
|
||||||
Prompt metadata hashes remain based on template source, not rendered reference
|
the `reference` function for content and the `hasreference` function for
|
||||||
bytes.
|
conditional sections. Prompt metadata hashes remain based on template source,
|
||||||
|
not rendered reference bytes.
|
||||||
|
|
||||||
Chunk modules receive the structured LLM client through `contracts.ChunkRequest`
|
Chunk modules receive the structured LLM client through `contracts.ChunkRequest`
|
||||||
when they need model-backed chunking. The pipeline runner validates generic
|
when they need model-backed chunking. The pipeline runner validates generic
|
||||||
|
|||||||
@@ -42,19 +42,19 @@ specific selector such as `chunk.slot`, `lane.extract.slot`, or
|
|||||||
target specs and stores the bindings in target-aware resolved reference holders.
|
target specs and stores the bindings in target-aware resolved reference holders.
|
||||||
It does not read reference files or include reference bytes in source digests.
|
It does not read reference files or include reference bytes in source digests.
|
||||||
|
|
||||||
During run preparation, resolved extractor file references are materialized
|
During run preparation, resolved file references for chunk, extractor, and
|
||||||
before any LLM-backed pipeline work. Config bindings resolve relative to the
|
normalizer targets are materialized before any LLM-backed pipeline work. Config
|
||||||
config file, CLI bindings resolve relative to the current working directory, and
|
bindings resolve relative to the config file, and CLI bindings resolve relative
|
||||||
materialized reference content is passed only to the matching lane extractor
|
to the current working directory. Materialization accepts UTF-8 text files,
|
||||||
through `ExtractionRequest`. Materialization accepts UTF-8 text files, computes
|
computes `sha256:` content digests, records file origins, infers canonical base
|
||||||
`sha256:` content digests, records file origins, infers canonical base media
|
media types from file extensions, enforces declared byte limits, and warns for
|
||||||
types from file extensions, enforces declared byte limits, and warns for empty
|
empty bound files. Media-type acceptance is checked only when a slot declares
|
||||||
bound files. Media-type acceptance is checked only when a slot declares
|
|
||||||
`AcceptedMediaTypes`; unknown extensions are recorded as
|
`AcceptedMediaTypes`; unknown extensions are recorded as
|
||||||
`application/octet-stream`. Reference content is omitted from diagnostics and
|
`application/octet-stream`. Reference content is omitted from diagnostics and
|
||||||
manifests. The CLI writes provenance-only resolved reference diagnostics, and
|
manifests. The CLI writes provenance-only resolved reference diagnostics, and
|
||||||
the run manifest records target-stage reference provenance separately from
|
the run manifest records target-stage reference provenance separately from
|
||||||
source digests.
|
source digests. Runtime reference content is currently passed only to the
|
||||||
|
matching lane extractor through `ExtractionRequest`.
|
||||||
|
|
||||||
Prompt bundles can declare reference slots and use `reference` and
|
Prompt bundles can declare reference slots and use `reference` and
|
||||||
`hasreference` template functions. Bundle loading validates string-literal slot
|
`hasreference` template functions. Bundle loading validates string-literal slot
|
||||||
|
|||||||
@@ -30,23 +30,35 @@ type ReferenceMaterializationOptions struct {
|
|||||||
func MaterializeReferences(resolved ResolvedPipeline, catalog ModuleCatalog, options ReferenceMaterializationOptions) (ResolvedPipeline, []contracts.Warning, error) {
|
func MaterializeReferences(resolved ResolvedPipeline, catalog ModuleCatalog, options ReferenceMaterializationOptions) (ResolvedPipeline, []contracts.Warning, error) {
|
||||||
out := resolved
|
out := resolved
|
||||||
out.ChunkReferences = CloneReferenceTarget(resolved.ChunkReferences)
|
out.ChunkReferences = CloneReferenceTarget(resolved.ChunkReferences)
|
||||||
|
chunkReferenceSet, chunkWarnings, err := materializeReferenceTarget(resolved.ID, resolved.ChunkReferences, catalog, options)
|
||||||
|
if err != nil {
|
||||||
|
return ResolvedPipeline{}, nil, err
|
||||||
|
}
|
||||||
|
out.ChunkReferences.ReferenceSet = chunkReferenceSet
|
||||||
|
warnings := append([]contracts.Warning(nil), chunkWarnings...)
|
||||||
if len(resolved.ArtifactLanes) == 0 {
|
if len(resolved.ArtifactLanes) == 0 {
|
||||||
return out, nil, nil
|
return out, warnings, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
warnings := []contracts.Warning(nil)
|
|
||||||
out.ArtifactLanes = make([]ResolvedArtifactLane, len(resolved.ArtifactLanes))
|
out.ArtifactLanes = make([]ResolvedArtifactLane, len(resolved.ArtifactLanes))
|
||||||
for i, lane := range resolved.ArtifactLanes {
|
for i, lane := range resolved.ArtifactLanes {
|
||||||
materializedLane := lane
|
materializedLane := lane
|
||||||
materializedLane.ExtractReferences = CloneReferenceTarget(lane.ExtractReferences)
|
materializedLane.ExtractReferences = CloneReferenceTarget(lane.ExtractReferences)
|
||||||
materializedLane.NormalizeReferences = CloneReferenceTarget(lane.NormalizeReferences)
|
materializedLane.NormalizeReferences = CloneReferenceTarget(lane.NormalizeReferences)
|
||||||
referenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.ExtractReferences, catalog, options)
|
extractReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.ExtractReferences, catalog, options)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ResolvedPipeline{}, nil, err
|
return ResolvedPipeline{}, nil, err
|
||||||
}
|
}
|
||||||
materializedLane.ExtractReferences.ReferenceSet = referenceSet
|
materializedLane.ExtractReferences.ReferenceSet = extractReferenceSet
|
||||||
out.ArtifactLanes[i] = materializedLane
|
|
||||||
warnings = append(warnings, laneWarnings...)
|
warnings = append(warnings, laneWarnings...)
|
||||||
|
|
||||||
|
normalizeReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.NormalizeReferences, catalog, options)
|
||||||
|
if err != nil {
|
||||||
|
return ResolvedPipeline{}, nil, err
|
||||||
|
}
|
||||||
|
materializedLane.NormalizeReferences.ReferenceSet = normalizeReferenceSet
|
||||||
|
warnings = append(warnings, laneWarnings...)
|
||||||
|
out.ArtifactLanes[i] = materializedLane
|
||||||
}
|
}
|
||||||
return out, warnings, nil
|
return out, warnings, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,6 +4,8 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
@@ -103,6 +105,66 @@ func TestMaterializeReferencesResolvesPathsAndDigestsContent(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestMaterializeReferencesStoresSetsAndProvenanceForAllTargets(t *testing.T) {
|
||||||
|
configDir := t.TempDir()
|
||||||
|
writeReferenceFile(t, filepath.Join(configDir, "chunk.txt"), []byte("chunk text"))
|
||||||
|
writeReferenceFile(t, filepath.Join(configDir, "extract.txt"), []byte("extract text"))
|
||||||
|
writeReferenceFile(t, filepath.Join(configDir, "normalize.txt"), []byte("normalize text"))
|
||||||
|
|
||||||
|
profile := baselineProfile()
|
||||||
|
profile.References = map[string]string{
|
||||||
|
"scene_guide": "chunk.txt",
|
||||||
|
"roster": "extract.txt",
|
||||||
|
"normalization_notes": "normalize.txt",
|
||||||
|
}
|
||||||
|
catalog := referenceCatalogForTargets(t,
|
||||||
|
[]contracts.ReferenceSlot{{Name: "scene_guide"}},
|
||||||
|
[]contracts.ReferenceSlot{{Name: "roster"}},
|
||||||
|
[]contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
||||||
|
)
|
||||||
|
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
materialized, warnings, err := MaterializeReferences(resolved, catalog, ReferenceMaterializationOptions{
|
||||||
|
ConfigPath: filepath.Join(configDir, "config.yml"),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if len(warnings) != 0 {
|
||||||
|
t.Fatalf("warnings = %#v, want none", warnings)
|
||||||
|
}
|
||||||
|
|
||||||
|
chunkItem := materialized.ChunkReferences.ReferenceSet.Slots["scene_guide"].Items[0]
|
||||||
|
if string(chunkItem.Content) != "chunk text" {
|
||||||
|
t.Fatalf("chunk content = %q, want chunk text", chunkItem.Content)
|
||||||
|
}
|
||||||
|
extractItem := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
|
||||||
|
if string(extractItem.Content) != "extract text" {
|
||||||
|
t.Fatalf("extract content = %q, want extract text", extractItem.Content)
|
||||||
|
}
|
||||||
|
normalizeItem := materialized.ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["normalization_notes"].Items[0]
|
||||||
|
if string(normalizeItem.Content) != "normalize text" {
|
||||||
|
t.Fatalf("normalize content = %q, want normalize text", normalizeItem.Content)
|
||||||
|
}
|
||||||
|
|
||||||
|
provenance := ReferenceProvenance(materialized)
|
||||||
|
if len(provenance) != 3 {
|
||||||
|
t.Fatalf("ReferenceProvenance() = %#v, want three entries", provenance)
|
||||||
|
}
|
||||||
|
if provenance[0].Stage != string(StageChunk) || provenance[0].LaneID != "" || provenance[0].SlotName != "scene_guide" || provenance[0].Digest != chunkItem.Digest {
|
||||||
|
t.Fatalf("ReferenceProvenance()[0] = %#v, want chunk scene guide provenance", provenance[0])
|
||||||
|
}
|
||||||
|
if provenance[1].Stage != string(StageExtract) || provenance[1].LaneID != "events" || provenance[1].SlotName != "roster" || provenance[1].Digest != extractItem.Digest {
|
||||||
|
t.Fatalf("ReferenceProvenance()[1] = %#v, want extract roster provenance", provenance[1])
|
||||||
|
}
|
||||||
|
if provenance[2].Stage != string(StageNormalize) || provenance[2].LaneID != "events" || provenance[2].SlotName != "normalization_notes" || provenance[2].Digest != normalizeItem.Digest {
|
||||||
|
t.Fatalf("ReferenceProvenance()[2] = %#v, want normalize notes provenance", provenance[2])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestMaterializeReferencesRejectsNonUTF8Content(t *testing.T) {
|
func TestMaterializeReferencesRejectsNonUTF8Content(t *testing.T) {
|
||||||
configDir := t.TempDir()
|
configDir := t.TempDir()
|
||||||
path := filepath.Join(configDir, "bad.txt")
|
path := filepath.Join(configDir, "bad.txt")
|
||||||
@@ -117,6 +179,20 @@ func TestMaterializeReferencesRejectsNonUTF8Content(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestMaterializeReferencesRejectsNonUTF8ContentForChunkTarget(t *testing.T) {
|
||||||
|
configDir := t.TempDir()
|
||||||
|
path := filepath.Join(configDir, "bad.txt")
|
||||||
|
writeReferenceFile(t, path, []byte{0xff, 0xfe})
|
||||||
|
|
||||||
|
resolved := resolvedPipelineWithTargetReference(t, StageChunk, "", "scene_guide", "bad.txt", contracts.ReferenceBindingSourceConfig, contracts.ReferenceSlot{Name: "scene_guide"})
|
||||||
|
_, _, err := MaterializeReferences(resolved, referenceCatalogForTargets(t, []contracts.ReferenceSlot{{Name: "scene_guide"}}, nil, nil), ReferenceMaterializationOptions{
|
||||||
|
ConfigPath: filepath.Join(configDir, "config.yml"),
|
||||||
|
})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "chunk") || !strings.Contains(err.Error(), "UTF-8") || !strings.Contains(err.Error(), "scene_guide") || !strings.Contains(err.Error(), path) {
|
||||||
|
t.Fatalf("error = %v, want chunk UTF-8 path error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestMaterializeReferencesAllowsAnyMediaTypeWhenSlotDoesNotRestrictIt(t *testing.T) {
|
func TestMaterializeReferencesAllowsAnyMediaTypeWhenSlotDoesNotRestrictIt(t *testing.T) {
|
||||||
configDir := t.TempDir()
|
configDir := t.TempDir()
|
||||||
path := filepath.Join(configDir, "roster.reference")
|
path := filepath.Join(configDir, "roster.reference")
|
||||||
@@ -207,6 +283,21 @@ func TestMaterializeReferencesMatchesAcceptedMediaTypesIgnoringParameters(t *tes
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestMaterializeReferencesRejectsUnacceptedMediaTypeForNormalizeTarget(t *testing.T) {
|
||||||
|
configDir := t.TempDir()
|
||||||
|
path := filepath.Join(configDir, "notes.json")
|
||||||
|
writeReferenceFile(t, path, []byte(`{"notes":true}`))
|
||||||
|
|
||||||
|
slot := contracts.ReferenceSlot{Name: "normalization_notes", AcceptedMediaTypes: []string{"text/markdown"}}
|
||||||
|
resolved := resolvedPipelineWithTargetReference(t, StageNormalize, "events", "normalization_notes", "notes.json", contracts.ReferenceBindingSourceConfig, slot)
|
||||||
|
_, _, err := MaterializeReferences(resolved, referenceCatalogForTargets(t, nil, nil, []contracts.ReferenceSlot{slot}), ReferenceMaterializationOptions{
|
||||||
|
ConfigPath: filepath.Join(configDir, "config.yml"),
|
||||||
|
})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "normalize") || !strings.Contains(err.Error(), "media type") || !strings.Contains(err.Error(), "application/json") || !strings.Contains(err.Error(), "normalization_notes") {
|
||||||
|
t.Fatalf("error = %v, want normalize media type rejection", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestMaterializeReferencesWarnsForEmptyFiles(t *testing.T) {
|
func TestMaterializeReferencesWarnsForEmptyFiles(t *testing.T) {
|
||||||
configDir := t.TempDir()
|
configDir := t.TempDir()
|
||||||
path := filepath.Join(configDir, "empty.txt")
|
path := filepath.Join(configDir, "empty.txt")
|
||||||
@@ -228,6 +319,45 @@ func TestMaterializeReferencesWarnsForEmptyFiles(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestMaterializeReferencesWarningScopesIncludeTargetContext(t *testing.T) {
|
||||||
|
configDir := t.TempDir()
|
||||||
|
writeReferenceFile(t, filepath.Join(configDir, "chunk.txt"), nil)
|
||||||
|
writeReferenceFile(t, filepath.Join(configDir, "extract.txt"), nil)
|
||||||
|
writeReferenceFile(t, filepath.Join(configDir, "normalize.txt"), nil)
|
||||||
|
|
||||||
|
profile := baselineProfile()
|
||||||
|
profile.References = map[string]string{
|
||||||
|
"scene_guide": "chunk.txt",
|
||||||
|
"roster": "extract.txt",
|
||||||
|
"normalization_notes": "normalize.txt",
|
||||||
|
}
|
||||||
|
catalog := referenceCatalogForTargets(t,
|
||||||
|
[]contracts.ReferenceSlot{{Name: "scene_guide"}},
|
||||||
|
[]contracts.ReferenceSlot{{Name: "roster"}},
|
||||||
|
[]contracts.ReferenceSlot{{Name: "normalization_notes"}},
|
||||||
|
)
|
||||||
|
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, warnings, err := MaterializeReferences(resolved, catalog, ReferenceMaterializationOptions{
|
||||||
|
ConfigPath: filepath.Join(configDir, "config.yml"),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
got := warningScopes(warnings)
|
||||||
|
want := []string{
|
||||||
|
"pipeline.baseline.chunk.reference.scene_guide",
|
||||||
|
"pipeline.baseline.lane.events.extract.reference.roster",
|
||||||
|
"pipeline.baseline.lane.events.normalize.reference.normalization_notes",
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("warning scopes = %#v, want %#v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestMaterializeReferencesEnforcesMaxBytes(t *testing.T) {
|
func TestMaterializeReferencesEnforcesMaxBytes(t *testing.T) {
|
||||||
configDir := t.TempDir()
|
configDir := t.TempDir()
|
||||||
path := filepath.Join(configDir, "large.txt")
|
path := filepath.Join(configDir, "large.txt")
|
||||||
@@ -244,31 +374,99 @@ func TestMaterializeReferencesEnforcesMaxBytes(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func resolvedPipelineWithReference(t *testing.T, slotName, source, bindingSource string, slot contracts.ReferenceSlot) ResolvedPipeline {
|
func resolvedPipelineWithReference(t *testing.T, slotName, source, bindingSource string, slot contracts.ReferenceSlot) ResolvedPipeline {
|
||||||
|
t.Helper()
|
||||||
|
return resolvedPipelineWithTargetReference(t, StageExtract, "events", slotName, source, bindingSource, slot)
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolvedPipelineWithTargetReference(t *testing.T, stage ModuleStage, laneID string, slotName, source, bindingSource string, slot contracts.ReferenceSlot) ResolvedPipeline {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
profile := baselineProfile()
|
profile := baselineProfile()
|
||||||
lane := profile.Artifacts["events"]
|
switch stage {
|
||||||
lane.References = map[string]string{slotName: source}
|
case StageChunk:
|
||||||
profile.Artifacts["events"] = lane
|
profile.Chunk.References = map[string]string{slotName: source}
|
||||||
catalog := referenceCatalog(t, []contracts.ReferenceSlot{slot})
|
case StageExtract:
|
||||||
|
lane := profile.Artifacts[laneID]
|
||||||
|
lane.References = map[string]string{slotName: source}
|
||||||
|
profile.Artifacts[laneID] = lane
|
||||||
|
case StageNormalize:
|
||||||
|
lane := profile.Artifacts[laneID]
|
||||||
|
lane.Normalize.References = map[string]string{slotName: source}
|
||||||
|
profile.Artifacts[laneID] = lane
|
||||||
|
default:
|
||||||
|
t.Fatalf("unsupported reference target stage %q", stage)
|
||||||
|
}
|
||||||
|
catalog := referenceCatalogForStage(t, stage, []contracts.ReferenceSlot{slot})
|
||||||
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
if bindingSource != contracts.ReferenceBindingSourceConfig {
|
if bindingSource != contracts.ReferenceBindingSourceConfig {
|
||||||
resolved.ArtifactLanes[0].ExtractReferences.Bindings[0].BindingSource = bindingSource
|
switch stage {
|
||||||
|
case StageChunk:
|
||||||
|
resolved.ChunkReferences.Bindings[0].BindingSource = bindingSource
|
||||||
|
case StageExtract:
|
||||||
|
resolved.ArtifactLanes[0].ExtractReferences.Bindings[0].BindingSource = bindingSource
|
||||||
|
case StageNormalize:
|
||||||
|
resolved.ArtifactLanes[0].NormalizeReferences.Bindings[0].BindingSource = bindingSource
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return resolved
|
return resolved
|
||||||
}
|
}
|
||||||
|
|
||||||
func referenceCatalog(t *testing.T, slots []contracts.ReferenceSlot) ModuleCatalog {
|
func referenceCatalog(t *testing.T, slots []contracts.ReferenceSlot) ModuleCatalog {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
return newProfileCatalogWithOverride(t, ModuleSpec{
|
return referenceCatalogForStage(t, StageExtract, slots)
|
||||||
Key: "event-extractor",
|
}
|
||||||
Stage: StageExtract,
|
|
||||||
Requires: []string{"chunk"},
|
func referenceCatalogForStage(t *testing.T, stage ModuleStage, slots []contracts.ReferenceSlot) ModuleCatalog {
|
||||||
Provides: []string{"candidate"},
|
t.Helper()
|
||||||
ReferenceSlots: slots,
|
switch stage {
|
||||||
})
|
case StageChunk:
|
||||||
|
return referenceCatalogForTargets(t, slots, nil, nil)
|
||||||
|
case StageExtract:
|
||||||
|
return referenceCatalogForTargets(t, nil, slots, nil)
|
||||||
|
case StageNormalize:
|
||||||
|
return referenceCatalogForTargets(t, nil, nil, slots)
|
||||||
|
default:
|
||||||
|
t.Fatalf("unsupported reference target stage %q", stage)
|
||||||
|
return ModuleCatalog{}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, normalizeSlots []contracts.ReferenceSlot) ModuleCatalog {
|
||||||
|
t.Helper()
|
||||||
|
return newProfileCatalogWithOverrides(t,
|
||||||
|
ModuleSpec{
|
||||||
|
Key: "generic",
|
||||||
|
Stage: StageChunk,
|
||||||
|
Requires: []string{"source"},
|
||||||
|
Provides: []string{"chunk"},
|
||||||
|
ReferenceSlots: chunkSlots,
|
||||||
|
},
|
||||||
|
ModuleSpec{
|
||||||
|
Key: "event-extractor",
|
||||||
|
Stage: StageExtract,
|
||||||
|
Requires: []string{"chunk"},
|
||||||
|
Provides: []string{"candidate"},
|
||||||
|
ReferenceSlots: extractSlots,
|
||||||
|
},
|
||||||
|
ModuleSpec{
|
||||||
|
Key: "noop",
|
||||||
|
Stage: StageNormalize,
|
||||||
|
Requires: []string{"merged"},
|
||||||
|
Provides: []string{"normalized"},
|
||||||
|
ReferenceSlots: normalizeSlots,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
func warningScopes(warnings []contracts.Warning) []string {
|
||||||
|
scopes := make([]string, 0, len(warnings))
|
||||||
|
for _, warning := range warnings {
|
||||||
|
scopes = append(scopes, warning.Scope)
|
||||||
|
}
|
||||||
|
sort.Strings(scopes)
|
||||||
|
return scopes
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeReferenceFile(t *testing.T, path string, content []byte) {
|
func writeReferenceFile(t *testing.T, path string, content []byte) {
|
||||||
|
|||||||
@@ -965,6 +965,24 @@ func TestRunReturnsFailedManifestWhenOutputEncoderFails(t *testing.T) {
|
|||||||
|
|
||||||
func TestRunManifestIncludesPipelineAndLaneDetails(t *testing.T) {
|
func TestRunManifestIncludesPipelineAndLaneDetails(t *testing.T) {
|
||||||
resolved := resolvedPipelineWithValidators("configured")
|
resolved := resolvedPipelineWithValidators("configured")
|
||||||
|
resolved.ChunkReferences.ReferenceSet = contracts.ReferenceSet{
|
||||||
|
Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||||
|
"scene_guide": {
|
||||||
|
Slot: contracts.ReferenceSlot{Name: "scene_guide"},
|
||||||
|
Items: []contracts.ReferenceItem{
|
||||||
|
{
|
||||||
|
SlotName: "scene_guide",
|
||||||
|
MediaType: "text/plain; charset=utf-8",
|
||||||
|
Content: []byte("chunk reference content"),
|
||||||
|
Digest: "sha256:chunk-reference",
|
||||||
|
Origin: contracts.ReferenceOrigin{Type: "file", URI: "file:///tmp/scene-guide.txt"},
|
||||||
|
SizeBytes: int64(len("chunk reference content")),
|
||||||
|
BindingSource: contracts.ReferenceBindingSourceCLI,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
resolved.ArtifactLanes[0].ExtractReferences.ReferenceSet = contracts.ReferenceSet{
|
resolved.ArtifactLanes[0].ExtractReferences.ReferenceSet = contracts.ReferenceSet{
|
||||||
Slots: map[string]contracts.ResolvedReferenceSlot{
|
Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||||
"roster": {
|
"roster": {
|
||||||
@@ -983,6 +1001,24 @@ func TestRunManifestIncludesPipelineAndLaneDetails(t *testing.T) {
|
|||||||
},
|
},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
resolved.ArtifactLanes[0].NormalizeReferences.ReferenceSet = contracts.ReferenceSet{
|
||||||
|
Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||||
|
"normalization_notes": {
|
||||||
|
Slot: contracts.ReferenceSlot{Name: "normalization_notes"},
|
||||||
|
Items: []contracts.ReferenceItem{
|
||||||
|
{
|
||||||
|
SlotName: "normalization_notes",
|
||||||
|
MediaType: "text/plain; charset=utf-8",
|
||||||
|
Content: []byte("normalize reference content"),
|
||||||
|
Digest: "sha256:normalize-reference",
|
||||||
|
Origin: contracts.ReferenceOrigin{Type: "file", URI: "file:///tmp/normalize.txt"},
|
||||||
|
SizeBytes: int64(len("normalize reference content")),
|
||||||
|
BindingSource: contracts.ReferenceBindingSourceConfig,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
output, err := New(newRunnerRegistries(t, nil)).Run(context.Background(), RunInput{Pipeline: resolved})
|
output, err := New(newRunnerRegistries(t, nil)).Run(context.Background(), RunInput{Pipeline: resolved})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -999,15 +1035,26 @@ func TestRunManifestIncludesPipelineAndLaneDetails(t *testing.T) {
|
|||||||
if !reflect.DeepEqual(manifest.SourceDigests, []string{"sha256:source"}) {
|
if !reflect.DeepEqual(manifest.SourceDigests, []string{"sha256:source"}) {
|
||||||
t.Fatalf("SourceDigests = %#v, want source digest", manifest.SourceDigests)
|
t.Fatalf("SourceDigests = %#v, want source digest", manifest.SourceDigests)
|
||||||
}
|
}
|
||||||
if len(manifest.References) != 1 {
|
if len(manifest.References) != 3 {
|
||||||
t.Fatalf("References = %#v, want one reference provenance entry", manifest.References)
|
t.Fatalf("References = %#v, want three reference provenance entries", manifest.References)
|
||||||
}
|
}
|
||||||
reference := manifest.References[0]
|
chunkReference := manifest.References[0]
|
||||||
if reference.Stage != string(StageExtract) || reference.LaneID != "alpha" || reference.SlotName != "roster" || reference.Digest != "sha256:reference" {
|
if chunkReference.Stage != string(StageChunk) || chunkReference.LaneID != "" || chunkReference.SlotName != "scene_guide" || chunkReference.Digest != "sha256:chunk-reference" {
|
||||||
t.Fatalf("reference provenance = %#v, want lane slot digest", reference)
|
t.Fatalf("chunk reference provenance = %#v, want chunk slot digest", chunkReference)
|
||||||
}
|
}
|
||||||
if reference.OriginType != "file" || reference.OriginURI != "file:///tmp/roster.txt" || reference.MediaType != "text/plain; charset=utf-8" || reference.SizeBytes != int64(len("reference content")) || reference.BindingSource != contracts.ReferenceBindingSourceConfig {
|
if chunkReference.OriginType != "file" || chunkReference.OriginURI != "file:///tmp/scene-guide.txt" || chunkReference.MediaType != "text/plain; charset=utf-8" || chunkReference.SizeBytes != int64(len("chunk reference content")) || chunkReference.BindingSource != contracts.ReferenceBindingSourceCLI {
|
||||||
t.Fatalf("reference provenance = %#v, want origin/media/size/source", reference)
|
t.Fatalf("chunk reference provenance = %#v, want origin/media/size/source", chunkReference)
|
||||||
|
}
|
||||||
|
extractReference := manifest.References[1]
|
||||||
|
if extractReference.Stage != string(StageExtract) || extractReference.LaneID != "alpha" || extractReference.SlotName != "roster" || extractReference.Digest != "sha256:reference" {
|
||||||
|
t.Fatalf("extract reference provenance = %#v, want lane slot digest", extractReference)
|
||||||
|
}
|
||||||
|
if extractReference.OriginType != "file" || extractReference.OriginURI != "file:///tmp/roster.txt" || extractReference.MediaType != "text/plain; charset=utf-8" || extractReference.SizeBytes != int64(len("reference content")) || extractReference.BindingSource != contracts.ReferenceBindingSourceConfig {
|
||||||
|
t.Fatalf("extract reference provenance = %#v, want origin/media/size/source", extractReference)
|
||||||
|
}
|
||||||
|
normalizeReference := manifest.References[2]
|
||||||
|
if normalizeReference.Stage != string(StageNormalize) || normalizeReference.LaneID != "alpha" || normalizeReference.SlotName != "normalization_notes" || normalizeReference.Digest != "sha256:normalize-reference" {
|
||||||
|
t.Fatalf("normalize reference provenance = %#v, want lane slot digest", normalizeReference)
|
||||||
}
|
}
|
||||||
if manifest.ValidationStatus != "approved" {
|
if manifest.ValidationStatus != "approved" {
|
||||||
t.Fatalf("ValidationStatus = %q, want approved", manifest.ValidationStatus)
|
t.Fatalf("ValidationStatus = %q, want approved", manifest.ValidationStatus)
|
||||||
|
|||||||
Reference in New Issue
Block a user