Materialize references for all eligible targets

This commit is contained in:
2026-07-05 16:35:24 +00:00
parent 8c623b7ad8
commit 4cafde2502
8 changed files with 316 additions and 52 deletions

View File

@@ -58,11 +58,12 @@ the warning count is printed to stderr.
Reference flags are resolved against selected chunk, extractor, and normalizer Reference flags are resolved against selected chunk, extractor, and normalizer
targets before the run starts. Flat slot names are accepted only when exactly targets before the run starts. Flat slot names are accepted only when exactly
one selected target declares that slot. Bound extractor reference files are read one selected target declares that slot. Bound reference files are read before
before extraction, validated as UTF-8 text, and passed only to the lane pipeline work starts, validated as UTF-8 text, and recorded as provenance for
extractor that declares the slot. Notarius infers reference media types from the target that declares the slot. Runtime reference content is currently passed
file extensions for provenance and for optional slot checks. Reference content only to lane extractors. Notarius infers reference media types from file
is not written to diagnostics, logs, errors, or manifests. extensions for provenance and for optional slot checks. Reference content is not
written to diagnostics, logs, errors, or manifests.
Reference binding precedence is: Reference binding precedence is:

View File

@@ -148,12 +148,14 @@ must be bound after config defaults, target-local references, lane-level
compatibility bindings, and run-time `--reference` or `--without-reference` compatibility bindings, and run-time `--reference` or `--without-reference`
overrides are applied. Config-relative paths are resolved relative to the config overrides are applied. Config-relative paths are resolved relative to the config
file; CLI reference paths are resolved relative to the current working file; CLI reference paths are resolved relative to the current working
directory. Materialized bound files must be UTF-8 text and are currently passed directory. Materialized bound files must be UTF-8 text. Materialized reference
only to lane extractors that declare the slot. Reference media types are provenance is recorded for chunk, extractor, and normalizer targets; runtime
inferred from file extensions, recorded as canonical base media types, and reference content is currently passed only to lane extractors that declare the
checked only when a module declares `AcceptedMediaTypes`; unknown extensions are slot. Reference media types are inferred from file extensions, recorded as
recorded as `application/octet-stream`. Reference content is not written to canonical base media types, and checked only when a module declares
diagnostics, logs, errors, or manifests. `AcceptedMediaTypes`; unknown extensions are recorded as
`application/octet-stream`. Reference content is not written to diagnostics,
logs, errors, or manifests.
Pipeline-level `references` are defaults. They are valid when at least one Pipeline-level `references` are defaults. They are valid when at least one
eligible target in the full configured pipeline declares the slot, including eligible target in the full configured pipeline declares the slot, including

View File

@@ -104,6 +104,9 @@ target stage, lane ID when present, slot name, origin type and URI, digest,
media type, byte size, and binding source. Reference content is not written to media type, byte size, and binding source. Reference content is not written to
durable output. durable output.
Reference `stage` is `chunk`, `extract`, or `normalize`. `lane_id` is omitted
for chunk references and present for extract and normalize references.
When references are bound, the manifest section has this shape: When references are bound, the manifest section has this shape:
```json ```json

View File

@@ -31,14 +31,15 @@ must still be UTF-8 text. When a slot declares accepted media types, Notarius
compares the canonical base media type inferred from the file extension, compares the canonical base media type inferred from the file extension,
case-insensitively and without parameters. case-insensitively and without parameters.
The current resolver materializes reference content only for lane extractors The resolver materializes reference content for chunk, extractor, and
through `contracts.ExtractionRequest.References`. It is not source evidence and normalizer targets. Runtime delivery is currently implemented only for lane
must not be converted into `SourceRef` values. If a module prompt uses extractors through `contracts.ExtractionRequest.References`. Reference material
references, load the prompt bundle with the same declared slots and render with is not source evidence and must not be converted into `SourceRef` values. If a
`RenderUserSystemWithReferences`. Prompt templates may use the `reference` module prompt uses references, load the prompt bundle with the same declared
function for content and the `hasreference` function for conditional sections. slots and render with `RenderUserSystemWithReferences`. Prompt templates may use
Prompt metadata hashes remain based on template source, not rendered reference the `reference` function for content and the `hasreference` function for
bytes. conditional sections. Prompt metadata hashes remain based on template source,
not rendered reference bytes.
Chunk modules receive the structured LLM client through `contracts.ChunkRequest` Chunk modules receive the structured LLM client through `contracts.ChunkRequest`
when they need model-backed chunking. The pipeline runner validates generic when they need model-backed chunking. The pipeline runner validates generic

View File

@@ -42,19 +42,19 @@ specific selector such as `chunk.slot`, `lane.extract.slot`, or
target specs and stores the bindings in target-aware resolved reference holders. target specs and stores the bindings in target-aware resolved reference holders.
It does not read reference files or include reference bytes in source digests. It does not read reference files or include reference bytes in source digests.
During run preparation, resolved extractor file references are materialized During run preparation, resolved file references for chunk, extractor, and
before any LLM-backed pipeline work. Config bindings resolve relative to the normalizer targets are materialized before any LLM-backed pipeline work. Config
config file, CLI bindings resolve relative to the current working directory, and bindings resolve relative to the config file, and CLI bindings resolve relative
materialized reference content is passed only to the matching lane extractor to the current working directory. Materialization accepts UTF-8 text files,
through `ExtractionRequest`. Materialization accepts UTF-8 text files, computes computes `sha256:` content digests, records file origins, infers canonical base
`sha256:` content digests, records file origins, infers canonical base media media types from file extensions, enforces declared byte limits, and warns for
types from file extensions, enforces declared byte limits, and warns for empty empty bound files. Media-type acceptance is checked only when a slot declares
bound files. Media-type acceptance is checked only when a slot declares
`AcceptedMediaTypes`; unknown extensions are recorded as `AcceptedMediaTypes`; unknown extensions are recorded as
`application/octet-stream`. Reference content is omitted from diagnostics and `application/octet-stream`. Reference content is omitted from diagnostics and
manifests. The CLI writes provenance-only resolved reference diagnostics, and manifests. The CLI writes provenance-only resolved reference diagnostics, and
the run manifest records target-stage reference provenance separately from the run manifest records target-stage reference provenance separately from
source digests. source digests. Runtime reference content is currently passed only to the
matching lane extractor through `ExtractionRequest`.
Prompt bundles can declare reference slots and use `reference` and Prompt bundles can declare reference slots and use `reference` and
`hasreference` template functions. Bundle loading validates string-literal slot `hasreference` template functions. Bundle loading validates string-literal slot

View File

@@ -30,23 +30,35 @@ type ReferenceMaterializationOptions struct {
func MaterializeReferences(resolved ResolvedPipeline, catalog ModuleCatalog, options ReferenceMaterializationOptions) (ResolvedPipeline, []contracts.Warning, error) { func MaterializeReferences(resolved ResolvedPipeline, catalog ModuleCatalog, options ReferenceMaterializationOptions) (ResolvedPipeline, []contracts.Warning, error) {
out := resolved out := resolved
out.ChunkReferences = CloneReferenceTarget(resolved.ChunkReferences) out.ChunkReferences = CloneReferenceTarget(resolved.ChunkReferences)
chunkReferenceSet, chunkWarnings, err := materializeReferenceTarget(resolved.ID, resolved.ChunkReferences, catalog, options)
if err != nil {
return ResolvedPipeline{}, nil, err
}
out.ChunkReferences.ReferenceSet = chunkReferenceSet
warnings := append([]contracts.Warning(nil), chunkWarnings...)
if len(resolved.ArtifactLanes) == 0 { if len(resolved.ArtifactLanes) == 0 {
return out, nil, nil return out, warnings, nil
} }
warnings := []contracts.Warning(nil)
out.ArtifactLanes = make([]ResolvedArtifactLane, len(resolved.ArtifactLanes)) out.ArtifactLanes = make([]ResolvedArtifactLane, len(resolved.ArtifactLanes))
for i, lane := range resolved.ArtifactLanes { for i, lane := range resolved.ArtifactLanes {
materializedLane := lane materializedLane := lane
materializedLane.ExtractReferences = CloneReferenceTarget(lane.ExtractReferences) materializedLane.ExtractReferences = CloneReferenceTarget(lane.ExtractReferences)
materializedLane.NormalizeReferences = CloneReferenceTarget(lane.NormalizeReferences) materializedLane.NormalizeReferences = CloneReferenceTarget(lane.NormalizeReferences)
referenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.ExtractReferences, catalog, options) extractReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.ExtractReferences, catalog, options)
if err != nil { if err != nil {
return ResolvedPipeline{}, nil, err return ResolvedPipeline{}, nil, err
} }
materializedLane.ExtractReferences.ReferenceSet = referenceSet materializedLane.ExtractReferences.ReferenceSet = extractReferenceSet
out.ArtifactLanes[i] = materializedLane
warnings = append(warnings, laneWarnings...) warnings = append(warnings, laneWarnings...)
normalizeReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.NormalizeReferences, catalog, options)
if err != nil {
return ResolvedPipeline{}, nil, err
}
materializedLane.NormalizeReferences.ReferenceSet = normalizeReferenceSet
warnings = append(warnings, laneWarnings...)
out.ArtifactLanes[i] = materializedLane
} }
return out, warnings, nil return out, warnings, nil
} }

View File

@@ -4,6 +4,8 @@ import (
"encoding/json" "encoding/json"
"os" "os"
"path/filepath" "path/filepath"
"reflect"
"sort"
"strings" "strings"
"testing" "testing"
@@ -103,6 +105,66 @@ func TestMaterializeReferencesResolvesPathsAndDigestsContent(t *testing.T) {
} }
} }
func TestMaterializeReferencesStoresSetsAndProvenanceForAllTargets(t *testing.T) {
configDir := t.TempDir()
writeReferenceFile(t, filepath.Join(configDir, "chunk.txt"), []byte("chunk text"))
writeReferenceFile(t, filepath.Join(configDir, "extract.txt"), []byte("extract text"))
writeReferenceFile(t, filepath.Join(configDir, "normalize.txt"), []byte("normalize text"))
profile := baselineProfile()
profile.References = map[string]string{
"scene_guide": "chunk.txt",
"roster": "extract.txt",
"normalization_notes": "normalize.txt",
}
catalog := referenceCatalogForTargets(t,
[]contracts.ReferenceSlot{{Name: "scene_guide"}},
[]contracts.ReferenceSlot{{Name: "roster"}},
[]contracts.ReferenceSlot{{Name: "normalization_notes"}},
)
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
materialized, warnings, err := MaterializeReferences(resolved, catalog, ReferenceMaterializationOptions{
ConfigPath: filepath.Join(configDir, "config.yml"),
})
if err != nil {
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
}
if len(warnings) != 0 {
t.Fatalf("warnings = %#v, want none", warnings)
}
chunkItem := materialized.ChunkReferences.ReferenceSet.Slots["scene_guide"].Items[0]
if string(chunkItem.Content) != "chunk text" {
t.Fatalf("chunk content = %q, want chunk text", chunkItem.Content)
}
extractItem := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
if string(extractItem.Content) != "extract text" {
t.Fatalf("extract content = %q, want extract text", extractItem.Content)
}
normalizeItem := materialized.ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["normalization_notes"].Items[0]
if string(normalizeItem.Content) != "normalize text" {
t.Fatalf("normalize content = %q, want normalize text", normalizeItem.Content)
}
provenance := ReferenceProvenance(materialized)
if len(provenance) != 3 {
t.Fatalf("ReferenceProvenance() = %#v, want three entries", provenance)
}
if provenance[0].Stage != string(StageChunk) || provenance[0].LaneID != "" || provenance[0].SlotName != "scene_guide" || provenance[0].Digest != chunkItem.Digest {
t.Fatalf("ReferenceProvenance()[0] = %#v, want chunk scene guide provenance", provenance[0])
}
if provenance[1].Stage != string(StageExtract) || provenance[1].LaneID != "events" || provenance[1].SlotName != "roster" || provenance[1].Digest != extractItem.Digest {
t.Fatalf("ReferenceProvenance()[1] = %#v, want extract roster provenance", provenance[1])
}
if provenance[2].Stage != string(StageNormalize) || provenance[2].LaneID != "events" || provenance[2].SlotName != "normalization_notes" || provenance[2].Digest != normalizeItem.Digest {
t.Fatalf("ReferenceProvenance()[2] = %#v, want normalize notes provenance", provenance[2])
}
}
func TestMaterializeReferencesRejectsNonUTF8Content(t *testing.T) { func TestMaterializeReferencesRejectsNonUTF8Content(t *testing.T) {
configDir := t.TempDir() configDir := t.TempDir()
path := filepath.Join(configDir, "bad.txt") path := filepath.Join(configDir, "bad.txt")
@@ -117,6 +179,20 @@ func TestMaterializeReferencesRejectsNonUTF8Content(t *testing.T) {
} }
} }
func TestMaterializeReferencesRejectsNonUTF8ContentForChunkTarget(t *testing.T) {
configDir := t.TempDir()
path := filepath.Join(configDir, "bad.txt")
writeReferenceFile(t, path, []byte{0xff, 0xfe})
resolved := resolvedPipelineWithTargetReference(t, StageChunk, "", "scene_guide", "bad.txt", contracts.ReferenceBindingSourceConfig, contracts.ReferenceSlot{Name: "scene_guide"})
_, _, err := MaterializeReferences(resolved, referenceCatalogForTargets(t, []contracts.ReferenceSlot{{Name: "scene_guide"}}, nil, nil), ReferenceMaterializationOptions{
ConfigPath: filepath.Join(configDir, "config.yml"),
})
if err == nil || !strings.Contains(err.Error(), "chunk") || !strings.Contains(err.Error(), "UTF-8") || !strings.Contains(err.Error(), "scene_guide") || !strings.Contains(err.Error(), path) {
t.Fatalf("error = %v, want chunk UTF-8 path error", err)
}
}
func TestMaterializeReferencesAllowsAnyMediaTypeWhenSlotDoesNotRestrictIt(t *testing.T) { func TestMaterializeReferencesAllowsAnyMediaTypeWhenSlotDoesNotRestrictIt(t *testing.T) {
configDir := t.TempDir() configDir := t.TempDir()
path := filepath.Join(configDir, "roster.reference") path := filepath.Join(configDir, "roster.reference")
@@ -207,6 +283,21 @@ func TestMaterializeReferencesMatchesAcceptedMediaTypesIgnoringParameters(t *tes
} }
} }
func TestMaterializeReferencesRejectsUnacceptedMediaTypeForNormalizeTarget(t *testing.T) {
configDir := t.TempDir()
path := filepath.Join(configDir, "notes.json")
writeReferenceFile(t, path, []byte(`{"notes":true}`))
slot := contracts.ReferenceSlot{Name: "normalization_notes", AcceptedMediaTypes: []string{"text/markdown"}}
resolved := resolvedPipelineWithTargetReference(t, StageNormalize, "events", "normalization_notes", "notes.json", contracts.ReferenceBindingSourceConfig, slot)
_, _, err := MaterializeReferences(resolved, referenceCatalogForTargets(t, nil, nil, []contracts.ReferenceSlot{slot}), ReferenceMaterializationOptions{
ConfigPath: filepath.Join(configDir, "config.yml"),
})
if err == nil || !strings.Contains(err.Error(), "normalize") || !strings.Contains(err.Error(), "media type") || !strings.Contains(err.Error(), "application/json") || !strings.Contains(err.Error(), "normalization_notes") {
t.Fatalf("error = %v, want normalize media type rejection", err)
}
}
func TestMaterializeReferencesWarnsForEmptyFiles(t *testing.T) { func TestMaterializeReferencesWarnsForEmptyFiles(t *testing.T) {
configDir := t.TempDir() configDir := t.TempDir()
path := filepath.Join(configDir, "empty.txt") path := filepath.Join(configDir, "empty.txt")
@@ -228,6 +319,45 @@ func TestMaterializeReferencesWarnsForEmptyFiles(t *testing.T) {
} }
} }
func TestMaterializeReferencesWarningScopesIncludeTargetContext(t *testing.T) {
configDir := t.TempDir()
writeReferenceFile(t, filepath.Join(configDir, "chunk.txt"), nil)
writeReferenceFile(t, filepath.Join(configDir, "extract.txt"), nil)
writeReferenceFile(t, filepath.Join(configDir, "normalize.txt"), nil)
profile := baselineProfile()
profile.References = map[string]string{
"scene_guide": "chunk.txt",
"roster": "extract.txt",
"normalization_notes": "normalize.txt",
}
catalog := referenceCatalogForTargets(t,
[]contracts.ReferenceSlot{{Name: "scene_guide"}},
[]contracts.ReferenceSlot{{Name: "roster"}},
[]contracts.ReferenceSlot{{Name: "normalization_notes"}},
)
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
_, warnings, err := MaterializeReferences(resolved, catalog, ReferenceMaterializationOptions{
ConfigPath: filepath.Join(configDir, "config.yml"),
})
if err != nil {
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
}
got := warningScopes(warnings)
want := []string{
"pipeline.baseline.chunk.reference.scene_guide",
"pipeline.baseline.lane.events.extract.reference.roster",
"pipeline.baseline.lane.events.normalize.reference.normalization_notes",
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("warning scopes = %#v, want %#v", got, want)
}
}
func TestMaterializeReferencesEnforcesMaxBytes(t *testing.T) { func TestMaterializeReferencesEnforcesMaxBytes(t *testing.T) {
configDir := t.TempDir() configDir := t.TempDir()
path := filepath.Join(configDir, "large.txt") path := filepath.Join(configDir, "large.txt")
@@ -244,31 +374,99 @@ func TestMaterializeReferencesEnforcesMaxBytes(t *testing.T) {
} }
func resolvedPipelineWithReference(t *testing.T, slotName, source, bindingSource string, slot contracts.ReferenceSlot) ResolvedPipeline { func resolvedPipelineWithReference(t *testing.T, slotName, source, bindingSource string, slot contracts.ReferenceSlot) ResolvedPipeline {
t.Helper()
return resolvedPipelineWithTargetReference(t, StageExtract, "events", slotName, source, bindingSource, slot)
}
func resolvedPipelineWithTargetReference(t *testing.T, stage ModuleStage, laneID string, slotName, source, bindingSource string, slot contracts.ReferenceSlot) ResolvedPipeline {
t.Helper() t.Helper()
profile := baselineProfile() profile := baselineProfile()
lane := profile.Artifacts["events"] switch stage {
lane.References = map[string]string{slotName: source} case StageChunk:
profile.Artifacts["events"] = lane profile.Chunk.References = map[string]string{slotName: source}
catalog := referenceCatalog(t, []contracts.ReferenceSlot{slot}) case StageExtract:
lane := profile.Artifacts[laneID]
lane.References = map[string]string{slotName: source}
profile.Artifacts[laneID] = lane
case StageNormalize:
lane := profile.Artifacts[laneID]
lane.Normalize.References = map[string]string{slotName: source}
profile.Artifacts[laneID] = lane
default:
t.Fatalf("unsupported reference target stage %q", stage)
}
catalog := referenceCatalogForStage(t, stage, []contracts.ReferenceSlot{slot})
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog) resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
if err != nil { if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err) t.Fatalf("ResolvePipeline() error = %v, want nil", err)
} }
if bindingSource != contracts.ReferenceBindingSourceConfig { if bindingSource != contracts.ReferenceBindingSourceConfig {
resolved.ArtifactLanes[0].ExtractReferences.Bindings[0].BindingSource = bindingSource switch stage {
case StageChunk:
resolved.ChunkReferences.Bindings[0].BindingSource = bindingSource
case StageExtract:
resolved.ArtifactLanes[0].ExtractReferences.Bindings[0].BindingSource = bindingSource
case StageNormalize:
resolved.ArtifactLanes[0].NormalizeReferences.Bindings[0].BindingSource = bindingSource
}
} }
return resolved return resolved
} }
func referenceCatalog(t *testing.T, slots []contracts.ReferenceSlot) ModuleCatalog { func referenceCatalog(t *testing.T, slots []contracts.ReferenceSlot) ModuleCatalog {
t.Helper() t.Helper()
return newProfileCatalogWithOverride(t, ModuleSpec{ return referenceCatalogForStage(t, StageExtract, slots)
Key: "event-extractor", }
Stage: StageExtract,
Requires: []string{"chunk"}, func referenceCatalogForStage(t *testing.T, stage ModuleStage, slots []contracts.ReferenceSlot) ModuleCatalog {
Provides: []string{"candidate"}, t.Helper()
ReferenceSlots: slots, switch stage {
}) case StageChunk:
return referenceCatalogForTargets(t, slots, nil, nil)
case StageExtract:
return referenceCatalogForTargets(t, nil, slots, nil)
case StageNormalize:
return referenceCatalogForTargets(t, nil, nil, slots)
default:
t.Fatalf("unsupported reference target stage %q", stage)
return ModuleCatalog{}
}
}
func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, normalizeSlots []contracts.ReferenceSlot) ModuleCatalog {
t.Helper()
return newProfileCatalogWithOverrides(t,
ModuleSpec{
Key: "generic",
Stage: StageChunk,
Requires: []string{"source"},
Provides: []string{"chunk"},
ReferenceSlots: chunkSlots,
},
ModuleSpec{
Key: "event-extractor",
Stage: StageExtract,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
ReferenceSlots: extractSlots,
},
ModuleSpec{
Key: "noop",
Stage: StageNormalize,
Requires: []string{"merged"},
Provides: []string{"normalized"},
ReferenceSlots: normalizeSlots,
},
)
}
func warningScopes(warnings []contracts.Warning) []string {
scopes := make([]string, 0, len(warnings))
for _, warning := range warnings {
scopes = append(scopes, warning.Scope)
}
sort.Strings(scopes)
return scopes
} }
func writeReferenceFile(t *testing.T, path string, content []byte) { func writeReferenceFile(t *testing.T, path string, content []byte) {

View File

@@ -965,6 +965,24 @@ func TestRunReturnsFailedManifestWhenOutputEncoderFails(t *testing.T) {
func TestRunManifestIncludesPipelineAndLaneDetails(t *testing.T) { func TestRunManifestIncludesPipelineAndLaneDetails(t *testing.T) {
resolved := resolvedPipelineWithValidators("configured") resolved := resolvedPipelineWithValidators("configured")
resolved.ChunkReferences.ReferenceSet = contracts.ReferenceSet{
Slots: map[string]contracts.ResolvedReferenceSlot{
"scene_guide": {
Slot: contracts.ReferenceSlot{Name: "scene_guide"},
Items: []contracts.ReferenceItem{
{
SlotName: "scene_guide",
MediaType: "text/plain; charset=utf-8",
Content: []byte("chunk reference content"),
Digest: "sha256:chunk-reference",
Origin: contracts.ReferenceOrigin{Type: "file", URI: "file:///tmp/scene-guide.txt"},
SizeBytes: int64(len("chunk reference content")),
BindingSource: contracts.ReferenceBindingSourceCLI,
},
},
},
},
}
resolved.ArtifactLanes[0].ExtractReferences.ReferenceSet = contracts.ReferenceSet{ resolved.ArtifactLanes[0].ExtractReferences.ReferenceSet = contracts.ReferenceSet{
Slots: map[string]contracts.ResolvedReferenceSlot{ Slots: map[string]contracts.ResolvedReferenceSlot{
"roster": { "roster": {
@@ -983,6 +1001,24 @@ func TestRunManifestIncludesPipelineAndLaneDetails(t *testing.T) {
}, },
}, },
} }
resolved.ArtifactLanes[0].NormalizeReferences.ReferenceSet = contracts.ReferenceSet{
Slots: map[string]contracts.ResolvedReferenceSlot{
"normalization_notes": {
Slot: contracts.ReferenceSlot{Name: "normalization_notes"},
Items: []contracts.ReferenceItem{
{
SlotName: "normalization_notes",
MediaType: "text/plain; charset=utf-8",
Content: []byte("normalize reference content"),
Digest: "sha256:normalize-reference",
Origin: contracts.ReferenceOrigin{Type: "file", URI: "file:///tmp/normalize.txt"},
SizeBytes: int64(len("normalize reference content")),
BindingSource: contracts.ReferenceBindingSourceConfig,
},
},
},
},
}
output, err := New(newRunnerRegistries(t, nil)).Run(context.Background(), RunInput{Pipeline: resolved}) output, err := New(newRunnerRegistries(t, nil)).Run(context.Background(), RunInput{Pipeline: resolved})
if err != nil { if err != nil {
@@ -999,15 +1035,26 @@ func TestRunManifestIncludesPipelineAndLaneDetails(t *testing.T) {
if !reflect.DeepEqual(manifest.SourceDigests, []string{"sha256:source"}) { if !reflect.DeepEqual(manifest.SourceDigests, []string{"sha256:source"}) {
t.Fatalf("SourceDigests = %#v, want source digest", manifest.SourceDigests) t.Fatalf("SourceDigests = %#v, want source digest", manifest.SourceDigests)
} }
if len(manifest.References) != 1 { if len(manifest.References) != 3 {
t.Fatalf("References = %#v, want one reference provenance entry", manifest.References) t.Fatalf("References = %#v, want three reference provenance entries", manifest.References)
} }
reference := manifest.References[0] chunkReference := manifest.References[0]
if reference.Stage != string(StageExtract) || reference.LaneID != "alpha" || reference.SlotName != "roster" || reference.Digest != "sha256:reference" { if chunkReference.Stage != string(StageChunk) || chunkReference.LaneID != "" || chunkReference.SlotName != "scene_guide" || chunkReference.Digest != "sha256:chunk-reference" {
t.Fatalf("reference provenance = %#v, want lane slot digest", reference) t.Fatalf("chunk reference provenance = %#v, want chunk slot digest", chunkReference)
} }
if reference.OriginType != "file" || reference.OriginURI != "file:///tmp/roster.txt" || reference.MediaType != "text/plain; charset=utf-8" || reference.SizeBytes != int64(len("reference content")) || reference.BindingSource != contracts.ReferenceBindingSourceConfig { if chunkReference.OriginType != "file" || chunkReference.OriginURI != "file:///tmp/scene-guide.txt" || chunkReference.MediaType != "text/plain; charset=utf-8" || chunkReference.SizeBytes != int64(len("chunk reference content")) || chunkReference.BindingSource != contracts.ReferenceBindingSourceCLI {
t.Fatalf("reference provenance = %#v, want origin/media/size/source", reference) t.Fatalf("chunk reference provenance = %#v, want origin/media/size/source", chunkReference)
}
extractReference := manifest.References[1]
if extractReference.Stage != string(StageExtract) || extractReference.LaneID != "alpha" || extractReference.SlotName != "roster" || extractReference.Digest != "sha256:reference" {
t.Fatalf("extract reference provenance = %#v, want lane slot digest", extractReference)
}
if extractReference.OriginType != "file" || extractReference.OriginURI != "file:///tmp/roster.txt" || extractReference.MediaType != "text/plain; charset=utf-8" || extractReference.SizeBytes != int64(len("reference content")) || extractReference.BindingSource != contracts.ReferenceBindingSourceConfig {
t.Fatalf("extract reference provenance = %#v, want origin/media/size/source", extractReference)
}
normalizeReference := manifest.References[2]
if normalizeReference.Stage != string(StageNormalize) || normalizeReference.LaneID != "alpha" || normalizeReference.SlotName != "normalization_notes" || normalizeReference.Digest != "sha256:normalize-reference" {
t.Fatalf("normalize reference provenance = %#v, want lane slot digest", normalizeReference)
} }
if manifest.ValidationStatus != "approved" { if manifest.ValidationStatus != "approved" {
t.Fatalf("ValidationStatus = %q, want approved", manifest.ValidationStatus) t.Fatalf("ValidationStatus = %q, want approved", manifest.ValidationStatus)