Resolve references across eligible pipeline targets

This commit is contained in:
2026-07-05 16:24:49 +00:00
parent 51053d390d
commit 43dc954440
5 changed files with 443 additions and 86 deletions

View File

@@ -124,9 +124,9 @@ Pipeline fields:
- `artifacts`: required for pipeline resolution. It maps artifact lane IDs to
lane definitions.
- `output`: optional module binding. Default module is `json`.
- `references`: optional map of extractor reference slot names to reference
paths. These bindings are defaults for artifact lanes whose extractor declares
the matching slot.
- `references`: optional map of reference slot names to reference paths. These
bindings are defaults for eligible pipeline targets that declare the matching
slot.
Artifact lane fields:
@@ -142,21 +142,23 @@ Artifact lane fields:
against the production module catalog and fail fast for unknown or incompatible
module keys.
Reference bindings are validated against extractor-declared slots during
pipeline resolution. Required slots must be bound after config defaults,
extractor binding references, lane-level compatibility bindings, and run-time
`--reference` or `--without-reference` overrides are applied. Config-relative
paths are resolved relative to the config file; CLI reference paths are resolved
relative to the current working directory. Bound files must be UTF-8 text and
are passed only to lane extractors that declare the slot. Reference media types
are inferred from file extensions, recorded as canonical base media types, and
checked only when a module declares `AcceptedMediaTypes`; unknown extensions are
recorded as `application/octet-stream`. Reference content is not written to
Reference bindings are validated against reference slots declared by eligible
chunk, extract, and normalize targets during pipeline resolution. Required slots
must be bound after config defaults, target-local references, lane-level
compatibility bindings, and extractor run-time `--reference` or
`--without-reference` overrides are applied. Config-relative paths are resolved
relative to the config file; CLI reference paths are resolved relative to the
current working directory. Materialized bound files must be UTF-8 text and are
currently passed only to lane extractors that declare the slot. Reference media
types are inferred from file extensions, recorded as canonical base media types,
and checked only when a module declares `AcceptedMediaTypes`; unknown extensions
are recorded as `application/octet-stream`. Reference content is not written to
diagnostics, logs, errors, or manifests.
Pipeline-level `references` are defaults. They are valid when at least one
declared lane in the pipeline has an extractor that declares the slot. During a
run, they apply only to selected lanes whose extractor declares the slot:
eligible target in the full configured pipeline declares the slot, including
chunk, extractor, and normalizer targets. During a run, they apply only to the
selected targets that declare the slot:
```yaml
pipelines:
@@ -191,9 +193,9 @@ pipelines:
```
`chunk.references` and `normalize.references` are accepted in object-form
bindings and preserved in the effective configuration. They are validated as
reference maps, but current reference materialization still delivers content
only to extractor bindings.
bindings. They override pipeline-level defaults for slots declared by the chunk
or normalizer module. Extractor-local references apply only to the extractor,
and normalizer-local references apply only to the normalizer.
## Module Bindings

View File

@@ -31,21 +31,21 @@ before execution:
The CLI writes the resolved pipeline and digest to diagnostics.
Pipeline profiles and artifact lanes may include reference binding maps keyed by
extractor reference slot name. During resolution, pipeline-level bindings act as
defaults for selected lanes whose extractor declares the slot, lane-level
bindings override or add lane bindings, runtime `--reference` requests override
config bindings, and runtime unbinds remove optional bindings. Flat runtime slot
names are resolved only when exactly one selected lane declares the slot;
otherwise the CLI requires `lane.slot`. Resolution validates bindings against
extractor specs and stores the bindings in target-aware resolved reference
holders. It does not read reference files or include reference bytes in source
digests.
reference slot name. During resolution, pipeline-level bindings act as defaults
for selected chunk, extractor, and normalizer targets that declare the slot;
target-local bindings override or add bindings for that target. Runtime
`--reference` requests override extractor config bindings, and runtime unbinds
remove optional extractor bindings. Flat runtime slot names are resolved only
when exactly one selected extractor lane declares the slot; otherwise the CLI
requires `lane.slot`. Resolution validates bindings against the declaring
target specs and stores the bindings in target-aware resolved reference holders.
It does not read reference files or include reference bytes in source digests.
During run preparation, resolved file references are materialized before any
LLM-backed pipeline work. Config bindings resolve relative to the config file,
CLI bindings resolve relative to the current working directory, and materialized
reference content is passed only to the matching lane extractor through
`ExtractionRequest`. Materialization accepts UTF-8 text files, computes
During run preparation, resolved extractor file references are materialized
before any LLM-backed pipeline work. Config bindings resolve relative to the
config file, CLI bindings resolve relative to the current working directory, and
materialized reference content is passed only to the matching lane extractor
through `ExtractionRequest`. Materialization accepts UTF-8 text files, computes
`sha256:` content digests, records file origins, infers canonical base media
types from file extensions, enforces declared byte limits, and warns for empty
bound files. Media-type acceptance is checked only when a slot declares