Prepare pipelines before source execution
This commit is contained in:
@@ -7,7 +7,8 @@ defaults, and selectable keys are defined in
|
||||
[Configuration](../config.md#pipelines).
|
||||
|
||||
Pipeline execution is serial. Resolution fixes the selected lanes and all
|
||||
stage bindings before the runner constructs stage implementations.
|
||||
stage bindings; preparation constructs every selected implementation before the
|
||||
runner begins source work.
|
||||
|
||||
## Resolution
|
||||
|
||||
@@ -24,7 +25,9 @@ calls `pipeline.ResolvePipeline`.
|
||||
selects exact-type merger, normalizer, and validator variants;
|
||||
5. checks required and provided capabilities in workflow order;
|
||||
6. resolves target-aware reference bindings and validator chains;
|
||||
7. calculates a digest over the resolved structure, including typed artifact
|
||||
7. validates each selected module and validator option set through its registry
|
||||
entry; and
|
||||
8. calculates a digest over the resolved structure, including typed artifact
|
||||
kind and schema identity.
|
||||
|
||||
Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
|
||||
@@ -53,7 +56,8 @@ runtime sensitive-data handling belongs in [Operations](../operations.md).
|
||||
|
||||
## Registries And Specs
|
||||
|
||||
`pipeline.Registries` holds constructors used during execution.
|
||||
`pipeline.Registries` holds option validators and run-local builders used during
|
||||
resolution and preparation.
|
||||
`pipeline.ModuleCatalog` exposes their specs during configuration validation and
|
||||
resolution. Separate registries exist for every stage and for validators;
|
||||
`ValidatorChainRegistry` stores production default-chain mappings. Both
|
||||
@@ -77,24 +81,37 @@ A `ModuleSpec` declares its stage plus required and provided capabilities.
|
||||
Chunk, extract, merge, and normalize specs may also declare reference slots.
|
||||
Registry implementations defensively copy spec metadata, reject duplicate keys,
|
||||
and verify that a constructed implementation reports the registered key.
|
||||
Builder registrations accept `ModuleDependencies` and cloned raw options through
|
||||
one `BuildRequest`. Existing production registrations are adapted from their
|
||||
zero-argument constructors while their implementation-owned option decoders are
|
||||
migrated separately.
|
||||
|
||||
A `ValidatorSpec` declares a validator key and execution class. Resolution uses
|
||||
the execution class to reject incompatible profile bindings before execution.
|
||||
The current production catalog and default chain are listed only in
|
||||
[Configuration](../config.md#implemented-production-validators).
|
||||
|
||||
## Runner Boundary
|
||||
## Preparation And Runner Boundary
|
||||
|
||||
`pipeline.RunInput` carries the resolved pipeline, raw source input, structured
|
||||
LLM client, run identity and timing, optional session and profile metadata, and
|
||||
checkpoint/debug collaborators. The runner parses source bytes through the
|
||||
selected input adapter. Later stage requests receive the generic source model;
|
||||
extract requests receive chunk-scoped input material, while chunk, merge, and
|
||||
normalize requests retain access to the original source material.
|
||||
`pipeline.Prepare` receives a resolved pipeline, the registries, and shared
|
||||
module dependencies. It constructs input; chunk and its validators; each lane's
|
||||
extract, merge, and normalize modules and validator chains in resolved order;
|
||||
then output. It stops at the first error with pipeline, stage, lane, module, and
|
||||
validator context as applicable. It never invokes an operation method.
|
||||
|
||||
Typed lanes can be composed and resolved but are not passed to the current raw
|
||||
runner. The runner rejects such input before source work; production lanes are
|
||||
still resolved and executed exclusively through the legacy raw path.
|
||||
`PreparedPipeline` keeps private constructed executors and exposes cloned
|
||||
resolved input, chunk, lane, and output identities. `pipeline.RunInput` carries
|
||||
that prepared pipeline, raw source input, run identity and timing, optional
|
||||
session and profile metadata, and checkpoint/debug collaborators. The runner
|
||||
parses source bytes through the already constructed input adapter. Later stage
|
||||
requests receive the generic source model; extract requests receive
|
||||
chunk-scoped input material, while chunk, merge, and normalize requests retain
|
||||
access to the original source material.
|
||||
|
||||
Typed lanes can be composed, resolved, and prepared but are not executed by the
|
||||
current raw runner. The runner rejects such input before source work;
|
||||
production lanes are still resolved and executed exclusively through the
|
||||
legacy raw path.
|
||||
|
||||
Source validation requires every unit to carry a canonical self-reference to
|
||||
its containing document and its own unit ID. Explicit clone, checkpoint, and
|
||||
@@ -111,17 +128,17 @@ runner returns.
|
||||
|
||||
The runner:
|
||||
|
||||
1. validates its input and registries;
|
||||
2. builds the input adapter, parses the raw input, and validates the generic
|
||||
1. validates its prepared input;
|
||||
2. parses the raw input with the prepared adapter and validates the generic
|
||||
source document;
|
||||
3. obtains or executes the chunk result;
|
||||
4. validates and canonicalizes chunks;
|
||||
5. executes each resolved artifact lane in order;
|
||||
6. builds the output encoder and validates its logical file results;
|
||||
6. invokes the prepared output encoder and validates its logical file results;
|
||||
7. returns the assembled manifest, outcomes, warnings, and files.
|
||||
|
||||
Within each artifact lane, it builds the extractor, merger, and normalizer,
|
||||
then performs these transitions:
|
||||
Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
|
||||
and validators while performing these transitions:
|
||||
|
||||
1. extract once per accepted chunk and add runner-owned lane, source, and chunk
|
||||
provenance;
|
||||
@@ -208,8 +225,10 @@ durable manifest and logical file schemas are defined in the
|
||||
- `internal/framework/pipeline/artifact_codec_registry_test.go`: typed codec
|
||||
metadata, registration, erasure safety, strict decoding, and cloning.
|
||||
- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed
|
||||
lane resolution, target-specific validators, incompatibilities, ordering, and
|
||||
schema-sensitive pipeline identity.
|
||||
lane resolution and preparation, target-specific validators,
|
||||
incompatibilities, ordering, and schema-sensitive pipeline identity.
|
||||
- `internal/framework/pipeline/preparation_test.go`: option validation,
|
||||
construction order, dependency failures, and the before-source-work boundary.
|
||||
- `internal/framework/pipeline/references_test.go`: target resolution and
|
||||
materialization.
|
||||
- `internal/framework/pipeline/runner_test.go`: stage transitions, retries,
|
||||
|
||||
Reference in New Issue
Block a user