Materialize references for all eligible targets
This commit is contained in:
11
docs/cli.md
11
docs/cli.md
@@ -58,11 +58,12 @@ the warning count is printed to stderr.
|
||||
|
||||
Reference flags are resolved against selected chunk, extractor, and normalizer
|
||||
targets before the run starts. Flat slot names are accepted only when exactly
|
||||
one selected target declares that slot. Bound extractor reference files are read
|
||||
before extraction, validated as UTF-8 text, and passed only to the lane
|
||||
extractor that declares the slot. Notarius infers reference media types from
|
||||
file extensions for provenance and for optional slot checks. Reference content
|
||||
is not written to diagnostics, logs, errors, or manifests.
|
||||
one selected target declares that slot. Bound reference files are read before
|
||||
pipeline work starts, validated as UTF-8 text, and recorded as provenance for
|
||||
the target that declares the slot. Runtime reference content is currently passed
|
||||
only to lane extractors. Notarius infers reference media types from file
|
||||
extensions for provenance and for optional slot checks. Reference content is not
|
||||
written to diagnostics, logs, errors, or manifests.
|
||||
|
||||
Reference binding precedence is:
|
||||
|
||||
|
||||
@@ -148,12 +148,14 @@ must be bound after config defaults, target-local references, lane-level
|
||||
compatibility bindings, and run-time `--reference` or `--without-reference`
|
||||
overrides are applied. Config-relative paths are resolved relative to the config
|
||||
file; CLI reference paths are resolved relative to the current working
|
||||
directory. Materialized bound files must be UTF-8 text and are currently passed
|
||||
only to lane extractors that declare the slot. Reference media types are
|
||||
inferred from file extensions, recorded as canonical base media types, and
|
||||
checked only when a module declares `AcceptedMediaTypes`; unknown extensions are
|
||||
recorded as `application/octet-stream`. Reference content is not written to
|
||||
diagnostics, logs, errors, or manifests.
|
||||
directory. Materialized bound files must be UTF-8 text. Materialized reference
|
||||
provenance is recorded for chunk, extractor, and normalizer targets; runtime
|
||||
reference content is currently passed only to lane extractors that declare the
|
||||
slot. Reference media types are inferred from file extensions, recorded as
|
||||
canonical base media types, and checked only when a module declares
|
||||
`AcceptedMediaTypes`; unknown extensions are recorded as
|
||||
`application/octet-stream`. Reference content is not written to diagnostics,
|
||||
logs, errors, or manifests.
|
||||
|
||||
Pipeline-level `references` are defaults. They are valid when at least one
|
||||
eligible target in the full configured pipeline declares the slot, including
|
||||
|
||||
@@ -104,6 +104,9 @@ target stage, lane ID when present, slot name, origin type and URI, digest,
|
||||
media type, byte size, and binding source. Reference content is not written to
|
||||
durable output.
|
||||
|
||||
Reference `stage` is `chunk`, `extract`, or `normalize`. `lane_id` is omitted
|
||||
for chunk references and present for extract and normalize references.
|
||||
|
||||
When references are bound, the manifest section has this shape:
|
||||
|
||||
```json
|
||||
|
||||
@@ -31,14 +31,15 @@ must still be UTF-8 text. When a slot declares accepted media types, Notarius
|
||||
compares the canonical base media type inferred from the file extension,
|
||||
case-insensitively and without parameters.
|
||||
|
||||
The current resolver materializes reference content only for lane extractors
|
||||
through `contracts.ExtractionRequest.References`. It is not source evidence and
|
||||
must not be converted into `SourceRef` values. If a module prompt uses
|
||||
references, load the prompt bundle with the same declared slots and render with
|
||||
`RenderUserSystemWithReferences`. Prompt templates may use the `reference`
|
||||
function for content and the `hasreference` function for conditional sections.
|
||||
Prompt metadata hashes remain based on template source, not rendered reference
|
||||
bytes.
|
||||
The resolver materializes reference content for chunk, extractor, and
|
||||
normalizer targets. Runtime delivery is currently implemented only for lane
|
||||
extractors through `contracts.ExtractionRequest.References`. Reference material
|
||||
is not source evidence and must not be converted into `SourceRef` values. If a
|
||||
module prompt uses references, load the prompt bundle with the same declared
|
||||
slots and render with `RenderUserSystemWithReferences`. Prompt templates may use
|
||||
the `reference` function for content and the `hasreference` function for
|
||||
conditional sections. Prompt metadata hashes remain based on template source,
|
||||
not rendered reference bytes.
|
||||
|
||||
Chunk modules receive the structured LLM client through `contracts.ChunkRequest`
|
||||
when they need model-backed chunking. The pipeline runner validates generic
|
||||
|
||||
@@ -42,19 +42,19 @@ specific selector such as `chunk.slot`, `lane.extract.slot`, or
|
||||
target specs and stores the bindings in target-aware resolved reference holders.
|
||||
It does not read reference files or include reference bytes in source digests.
|
||||
|
||||
During run preparation, resolved extractor file references are materialized
|
||||
before any LLM-backed pipeline work. Config bindings resolve relative to the
|
||||
config file, CLI bindings resolve relative to the current working directory, and
|
||||
materialized reference content is passed only to the matching lane extractor
|
||||
through `ExtractionRequest`. Materialization accepts UTF-8 text files, computes
|
||||
`sha256:` content digests, records file origins, infers canonical base media
|
||||
types from file extensions, enforces declared byte limits, and warns for empty
|
||||
bound files. Media-type acceptance is checked only when a slot declares
|
||||
During run preparation, resolved file references for chunk, extractor, and
|
||||
normalizer targets are materialized before any LLM-backed pipeline work. Config
|
||||
bindings resolve relative to the config file, and CLI bindings resolve relative
|
||||
to the current working directory. Materialization accepts UTF-8 text files,
|
||||
computes `sha256:` content digests, records file origins, infers canonical base
|
||||
media types from file extensions, enforces declared byte limits, and warns for
|
||||
empty bound files. Media-type acceptance is checked only when a slot declares
|
||||
`AcceptedMediaTypes`; unknown extensions are recorded as
|
||||
`application/octet-stream`. Reference content is omitted from diagnostics and
|
||||
manifests. The CLI writes provenance-only resolved reference diagnostics, and
|
||||
the run manifest records target-stage reference provenance separately from
|
||||
source digests.
|
||||
source digests. Runtime reference content is currently passed only to the
|
||||
matching lane extractor through `ExtractionRequest`.
|
||||
|
||||
Prompt bundles can declare reference slots and use `reference` and
|
||||
`hasreference` template functions. Bundle loading validates string-literal slot
|
||||
|
||||
Reference in New Issue
Block a user