Finalize bounded typed pipeline implementation

This commit is contained in:
2026-07-17 08:49:08 +00:00
parent adfd3bd052
commit 3013ee044d
11 changed files with 369 additions and 857 deletions

View File

@@ -83,6 +83,12 @@ cancelled waiter that has already received a permit releases it.
defers release on every result path. The effective limit and default are
configuration facts in [Configuration](../config.md#defaults).
This provider-call ceiling is independent of the pipeline's extract worker
limit. Concurrent lanes, retries, and validators all use the same scheduled
client, so increasing framework workers cannot exceed `total_llm`. Pipeline
dispatch and cancellation mechanics are documented in
[Pipeline Internals](pipeline.md#execution-flow).
## Prompt And Schema Assets
`AssetRegistry` combines caller-owned prompt filesystems under stable prefixes

View File

@@ -23,8 +23,13 @@ implementation-owned values and injects dependencies. The spell extractor is
typed over the canonical D&D model. D&D validators, merge, and normalize use
typed variants; JSON representation validators use serialized requests; and
unconditional validators expose separate chunk and typed variants. The D&D
production registrar does not register raw spell-stage implementations in
parallel.
production registrar registers only the canonical typed spell implementations.
Prepared extractors, extract validators, and codecs may be reused concurrently
by the run-wide extract pool. Production implementations are immutable after
construction: they retain only typed options, immutable assets, or the shared
concurrency-safe LLM client. Implementations that introduce mutable state must
synchronize that state without creating a separate provider scheduler.
Specs expose capability and execution metadata without constructing an
implementation. Registry entries separately expose option validation and

View File

@@ -142,7 +142,8 @@ The runner:
source document;
3. obtains or executes the chunk result;
4. validates and canonicalizes chunks;
5. executes each resolved artifact lane in order;
5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
bounded lane continuation when all extracts for that lane are terminal;
6. invokes the prepared output encoder and validates its logical file results;
7. returns the assembled manifest, outcomes, warnings, and files.
@@ -161,6 +162,13 @@ and validators while performing these transitions:
Module-provided warnings and payload warnings are promoted only from attempts
whose results are accepted and used.
The extract job channel has the same capacity as the effective extract worker
count, so dispatch applies backpressure. A fixed continuation executor prevents
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
continuations publish lane-local results; the coordinator is the only writer of
aggregate output and merges those results in resolved lane and source-chunk
order.
## Chunk Canonicalization
Before lane execution, generic validation requires unique chunk IDs, matching
@@ -234,6 +242,12 @@ time. Successful status reflects whether any result was rejected. The
durable manifest and logical file schemas are defined in the
[JSON output contract](../integrations/json-output.md).
On a framework failure, the runner cancels its derived context, stops submitting
new extract work, drains started tasks, and skips the output encoder. Parent
cancellation takes precedence. Otherwise context-cancellation fallout is
discarded when a substantive error exists, and the primary error is selected by
stage, resolved lane, and source chunk rather than completion time.
## Tests To Inspect
- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
@@ -244,6 +258,9 @@ durable manifest and logical file schemas are defined in the
- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed
lane resolution and preparation, target-specific validators,
incompatibilities, ordering, and schema-sensitive pipeline identity.
- `internal/framework/pipeline/runner_concurrency_test.go`: bounded dispatch and
continuations, reverse completion, stable errors, rejection, cancellation,
retries, and independent provider-call limits.
- `internal/framework/pipeline/preparation_test.go`: option validation,
construction order, dependency failures, and the before-source-work boundary.
- `internal/framework/pipeline/references_test.go`: target resolution and