Finalize bounded typed pipeline implementation
This commit is contained in:
@@ -142,7 +142,8 @@ The runner:
|
||||
source document;
|
||||
3. obtains or executes the chunk result;
|
||||
4. validates and canonicalizes chunks;
|
||||
5. executes each resolved artifact lane in order;
|
||||
5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
|
||||
bounded lane continuation when all extracts for that lane are terminal;
|
||||
6. invokes the prepared output encoder and validates its logical file results;
|
||||
7. returns the assembled manifest, outcomes, warnings, and files.
|
||||
|
||||
@@ -161,6 +162,13 @@ and validators while performing these transitions:
|
||||
Module-provided warnings and payload warnings are promoted only from attempts
|
||||
whose results are accepted and used.
|
||||
|
||||
The extract job channel has the same capacity as the effective extract worker
|
||||
count, so dispatch applies backpressure. A fixed continuation executor prevents
|
||||
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
|
||||
continuations publish lane-local results; the coordinator is the only writer of
|
||||
aggregate output and merges those results in resolved lane and source-chunk
|
||||
order.
|
||||
|
||||
## Chunk Canonicalization
|
||||
|
||||
Before lane execution, generic validation requires unique chunk IDs, matching
|
||||
@@ -234,6 +242,12 @@ time. Successful status reflects whether any result was rejected. The
|
||||
durable manifest and logical file schemas are defined in the
|
||||
[JSON output contract](../integrations/json-output.md).
|
||||
|
||||
On a framework failure, the runner cancels its derived context, stops submitting
|
||||
new extract work, drains started tasks, and skips the output encoder. Parent
|
||||
cancellation takes precedence. Otherwise context-cancellation fallout is
|
||||
discarded when a substantive error exists, and the primary error is selected by
|
||||
stage, resolved lane, and source chunk rather than completion time.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
|
||||
@@ -244,6 +258,9 @@ durable manifest and logical file schemas are defined in the
|
||||
- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed
|
||||
lane resolution and preparation, target-specific validators,
|
||||
incompatibilities, ordering, and schema-sensitive pipeline identity.
|
||||
- `internal/framework/pipeline/runner_concurrency_test.go`: bounded dispatch and
|
||||
continuations, reverse completion, stable errors, rejection, cancellation,
|
||||
retries, and independent provider-call limits.
|
||||
- `internal/framework/pipeline/preparation_test.go`: option validation,
|
||||
construction order, dependency failures, and the before-source-work boundary.
|
||||
- `internal/framework/pipeline/references_test.go`: target resolution and
|
||||
|
||||
Reference in New Issue
Block a user