eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add bounded concurrent pipeline execution

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-17 08:37:52 +00:00
parent 4023c66508
commit adfd3bd052
10 changed files with 958 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
docs/config.md
View File

@@ -123,9 +123,10 @@ Unknown or empty stage-worker keys are rejected. The environment override
`NOTARIUS_STAGE_WORKERS_EXTRACT` takes precedence over the file value, as does
`NOTARIUS_TOTAL_LLM_CONCURRENCY` for the global ceiling.
The worker value is present in effective and redacted configuration. The
current runner remains serial; this setting does not yet change execution
scheduling.
The worker value is present in effective and redacted configuration. It bounds
the fixed run-wide extract pool and its bounded dispatch queue. Extract jobs are
submitted by source chunk and then resolved lane; `total_llm` independently
bounds actual provider calls made by extracts, retries, and validators.
## Pipelines

13
docs/internal/overview.md
View File

@@ -14,9 +14,11 @@ collaborators, invokes `internal/framework/pipeline`, and places the logical
output files returned by the runner. Diagnostics, checkpoints, and debug
recorders are optional side-channel collaborators supplied at this boundary.
Pipeline execution is serial. Resolution produces a fixed ordered workflow and
a sorted set of artifact lanes. Preparation constructs the complete module and
validator set before the runner receives source bytes.
Resolution produces a fixed ordered workflow and a sorted set of artifact
lanes. Preparation constructs the complete module and validator set before the
runner receives source bytes. Source parsing and chunking are serial; extraction
uses a bounded run-wide worker pool, followed by serial per-lane merge and
normalize continuations that may overlap across lanes.
## Application Boundary
@@ -65,8 +67,9 @@ LLM-backed modules retain the injected shared client. The D&D family registers
the canonical `dnd/spell-list` codec, typed spell extractor and validators, and
kind-specific generic merge and normalize strategies; generic JSON validators
use the serialized-validation contract. The runner executes lanes through
private exact-type-checked closures and serializes artifacts only through their
codec at checkpoint, debug, and output boundaries.
private exact-type-checked closures, coordinates extract results independently
of completion timing, and serializes artifacts only through their codec at
checkpoint, debug, and output boundaries.
## Production Extensions

9
docs/internal/pipeline.md
View File

@@ -6,9 +6,12 @@ Their fixed workflow and ownership boundaries are defined by
defaults, and selectable keys are defined in
[Configuration](../config.md#pipelines).
Pipeline execution is serial. Resolution fixes the selected lanes and all
stage bindings; preparation constructs every selected implementation before the
runner begins source work.
Resolution fixes the selected lanes and all stage bindings; preparation
constructs every selected implementation before the runner begins source work.
After serial input parsing and chunking, the runner dispatches extract work to
one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's
merge and normalize operations remain serial and may overlap other lanes once
all extracts for that lane are terminal.
## Resolution