Resolve references across eligible pipeline targets
This commit is contained in:
@@ -124,9 +124,9 @@ Pipeline fields:
|
||||
- `artifacts`: required for pipeline resolution. It maps artifact lane IDs to
|
||||
lane definitions.
|
||||
- `output`: optional module binding. Default module is `json`.
|
||||
- `references`: optional map of extractor reference slot names to reference
|
||||
paths. These bindings are defaults for artifact lanes whose extractor declares
|
||||
the matching slot.
|
||||
- `references`: optional map of reference slot names to reference paths. These
|
||||
bindings are defaults for eligible pipeline targets that declare the matching
|
||||
slot.
|
||||
|
||||
Artifact lane fields:
|
||||
|
||||
@@ -142,21 +142,23 @@ Artifact lane fields:
|
||||
against the production module catalog and fail fast for unknown or incompatible
|
||||
module keys.
|
||||
|
||||
Reference bindings are validated against extractor-declared slots during
|
||||
pipeline resolution. Required slots must be bound after config defaults,
|
||||
extractor binding references, lane-level compatibility bindings, and run-time
|
||||
`--reference` or `--without-reference` overrides are applied. Config-relative
|
||||
paths are resolved relative to the config file; CLI reference paths are resolved
|
||||
relative to the current working directory. Bound files must be UTF-8 text and
|
||||
are passed only to lane extractors that declare the slot. Reference media types
|
||||
are inferred from file extensions, recorded as canonical base media types, and
|
||||
checked only when a module declares `AcceptedMediaTypes`; unknown extensions are
|
||||
recorded as `application/octet-stream`. Reference content is not written to
|
||||
Reference bindings are validated against reference slots declared by eligible
|
||||
chunk, extract, and normalize targets during pipeline resolution. Required slots
|
||||
must be bound after config defaults, target-local references, lane-level
|
||||
compatibility bindings, and extractor run-time `--reference` or
|
||||
`--without-reference` overrides are applied. Config-relative paths are resolved
|
||||
relative to the config file; CLI reference paths are resolved relative to the
|
||||
current working directory. Materialized bound files must be UTF-8 text and are
|
||||
currently passed only to lane extractors that declare the slot. Reference media
|
||||
types are inferred from file extensions, recorded as canonical base media types,
|
||||
and checked only when a module declares `AcceptedMediaTypes`; unknown extensions
|
||||
are recorded as `application/octet-stream`. Reference content is not written to
|
||||
diagnostics, logs, errors, or manifests.
|
||||
|
||||
Pipeline-level `references` are defaults. They are valid when at least one
|
||||
declared lane in the pipeline has an extractor that declares the slot. During a
|
||||
run, they apply only to selected lanes whose extractor declares the slot:
|
||||
eligible target in the full configured pipeline declares the slot, including
|
||||
chunk, extractor, and normalizer targets. During a run, they apply only to the
|
||||
selected targets that declare the slot:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
@@ -191,9 +193,9 @@ pipelines:
|
||||
```
|
||||
|
||||
`chunk.references` and `normalize.references` are accepted in object-form
|
||||
bindings and preserved in the effective configuration. They are validated as
|
||||
reference maps, but current reference materialization still delivers content
|
||||
only to extractor bindings.
|
||||
bindings. They override pipeline-level defaults for slots declared by the chunk
|
||||
or normalizer module. Extractor-local references apply only to the extractor,
|
||||
and normalizer-local references apply only to the normalizer.
|
||||
|
||||
## Module Bindings
|
||||
|
||||
|
||||
@@ -31,21 +31,21 @@ before execution:
|
||||
The CLI writes the resolved pipeline and digest to diagnostics.
|
||||
|
||||
Pipeline profiles and artifact lanes may include reference binding maps keyed by
|
||||
extractor reference slot name. During resolution, pipeline-level bindings act as
|
||||
defaults for selected lanes whose extractor declares the slot, lane-level
|
||||
bindings override or add lane bindings, runtime `--reference` requests override
|
||||
config bindings, and runtime unbinds remove optional bindings. Flat runtime slot
|
||||
names are resolved only when exactly one selected lane declares the slot;
|
||||
otherwise the CLI requires `lane.slot`. Resolution validates bindings against
|
||||
extractor specs and stores the bindings in target-aware resolved reference
|
||||
holders. It does not read reference files or include reference bytes in source
|
||||
digests.
|
||||
reference slot name. During resolution, pipeline-level bindings act as defaults
|
||||
for selected chunk, extractor, and normalizer targets that declare the slot;
|
||||
target-local bindings override or add bindings for that target. Runtime
|
||||
`--reference` requests override extractor config bindings, and runtime unbinds
|
||||
remove optional extractor bindings. Flat runtime slot names are resolved only
|
||||
when exactly one selected extractor lane declares the slot; otherwise the CLI
|
||||
requires `lane.slot`. Resolution validates bindings against the declaring
|
||||
target specs and stores the bindings in target-aware resolved reference holders.
|
||||
It does not read reference files or include reference bytes in source digests.
|
||||
|
||||
During run preparation, resolved file references are materialized before any
|
||||
LLM-backed pipeline work. Config bindings resolve relative to the config file,
|
||||
CLI bindings resolve relative to the current working directory, and materialized
|
||||
reference content is passed only to the matching lane extractor through
|
||||
`ExtractionRequest`. Materialization accepts UTF-8 text files, computes
|
||||
During run preparation, resolved extractor file references are materialized
|
||||
before any LLM-backed pipeline work. Config bindings resolve relative to the
|
||||
config file, CLI bindings resolve relative to the current working directory, and
|
||||
materialized reference content is passed only to the matching lane extractor
|
||||
through `ExtractionRequest`. Materialization accepts UTF-8 text files, computes
|
||||
`sha256:` content digests, records file origins, infers canonical base media
|
||||
types from file extensions, enforces declared byte limits, and warns for empty
|
||||
bound files. Media-type acceptance is checked only when a slot declares
|
||||
|
||||
Reference in New Issue
Block a user