Finalize bounded typed pipeline implementation
This commit is contained in:
@@ -83,6 +83,12 @@ cancelled waiter that has already received a permit releases it.
|
||||
defers release on every result path. The effective limit and default are
|
||||
configuration facts in [Configuration](../config.md#defaults).
|
||||
|
||||
This provider-call ceiling is independent of the pipeline's extract worker
|
||||
limit. Concurrent lanes, retries, and validators all use the same scheduled
|
||||
client, so increasing framework workers cannot exceed `total_llm`. Pipeline
|
||||
dispatch and cancellation mechanics are documented in
|
||||
[Pipeline Internals](pipeline.md#execution-flow).
|
||||
|
||||
## Prompt And Schema Assets
|
||||
|
||||
`AssetRegistry` combines caller-owned prompt filesystems under stable prefixes
|
||||
|
||||
@@ -23,8 +23,13 @@ implementation-owned values and injects dependencies. The spell extractor is
|
||||
typed over the canonical D&D model. D&D validators, merge, and normalize use
|
||||
typed variants; JSON representation validators use serialized requests; and
|
||||
unconditional validators expose separate chunk and typed variants. The D&D
|
||||
production registrar does not register raw spell-stage implementations in
|
||||
parallel.
|
||||
production registrar registers only the canonical typed spell implementations.
|
||||
|
||||
Prepared extractors, extract validators, and codecs may be reused concurrently
|
||||
by the run-wide extract pool. Production implementations are immutable after
|
||||
construction: they retain only typed options, immutable assets, or the shared
|
||||
concurrency-safe LLM client. Implementations that introduce mutable state must
|
||||
synchronize that state without creating a separate provider scheduler.
|
||||
|
||||
Specs expose capability and execution metadata without constructing an
|
||||
implementation. Registry entries separately expose option validation and
|
||||
|
||||
@@ -142,7 +142,8 @@ The runner:
|
||||
source document;
|
||||
3. obtains or executes the chunk result;
|
||||
4. validates and canonicalizes chunks;
|
||||
5. executes each resolved artifact lane in order;
|
||||
5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
|
||||
bounded lane continuation when all extracts for that lane are terminal;
|
||||
6. invokes the prepared output encoder and validates its logical file results;
|
||||
7. returns the assembled manifest, outcomes, warnings, and files.
|
||||
|
||||
@@ -161,6 +162,13 @@ and validators while performing these transitions:
|
||||
Module-provided warnings and payload warnings are promoted only from attempts
|
||||
whose results are accepted and used.
|
||||
|
||||
The extract job channel has the same capacity as the effective extract worker
|
||||
count, so dispatch applies backpressure. A fixed continuation executor prevents
|
||||
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
|
||||
continuations publish lane-local results; the coordinator is the only writer of
|
||||
aggregate output and merges those results in resolved lane and source-chunk
|
||||
order.
|
||||
|
||||
## Chunk Canonicalization
|
||||
|
||||
Before lane execution, generic validation requires unique chunk IDs, matching
|
||||
@@ -234,6 +242,12 @@ time. Successful status reflects whether any result was rejected. The
|
||||
durable manifest and logical file schemas are defined in the
|
||||
[JSON output contract](../integrations/json-output.md).
|
||||
|
||||
On a framework failure, the runner cancels its derived context, stops submitting
|
||||
new extract work, drains started tasks, and skips the output encoder. Parent
|
||||
cancellation takes precedence. Otherwise context-cancellation fallout is
|
||||
discarded when a substantive error exists, and the primary error is selected by
|
||||
stage, resolved lane, and source chunk rather than completion time.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
|
||||
@@ -244,6 +258,9 @@ durable manifest and logical file schemas are defined in the
|
||||
- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed
|
||||
lane resolution and preparation, target-specific validators,
|
||||
incompatibilities, ordering, and schema-sensitive pipeline identity.
|
||||
- `internal/framework/pipeline/runner_concurrency_test.go`: bounded dispatch and
|
||||
continuations, reverse completion, stable errors, rejection, cancellation,
|
||||
retries, and independent provider-call limits.
|
||||
- `internal/framework/pipeline/preparation_test.go`: option validation,
|
||||
construction order, dependency failures, and the before-source-work boundary.
|
||||
- `internal/framework/pipeline/references_test.go`: target resolution and
|
||||
|
||||
Reference in New Issue
Block a user