Add extract worker configuration controls

This commit is contained in:
2026-07-17 08:21:24 +00:00
parent adfe3825ee
commit 4023c66508
17 changed files with 343 additions and 22 deletions

View File

@@ -41,6 +41,7 @@ rejected; execution profiles now come from Scriptorium.
Built-in defaults:
- `concurrency.total_llm`: `1`
- `concurrency.stage_workers.extract`: effective `concurrency.total_llm`
- `diagnostics.work_dir`: `/tmp/notarius`
- `diagnostics.retention`: `auto`
- `workspace.directory`: unset
@@ -88,6 +89,7 @@ These environment variables are applied after the config file:
- `NOTARIUS_CONFIG`: config discovery path.
- `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency.
- `NOTARIUS_STAGE_WORKERS_EXTRACT`: integer extract worker limit.
- `NOTARIUS_WORKSPACE_DIR`: workspace directory.
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`: boolean diagnostics enablement.
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`: workspace diagnostics retention
@@ -107,6 +109,24 @@ The removed `NOTARIUS_LLM_DEFAULT_*` variables are not read. Configure provider
endpoint, model, and credential environment variable names through Scriptorium
profiles.
## Concurrency
`concurrency` fields:
- `total_llm`: positive integer ceiling on concurrent provider calls.
- `stage_workers`: optional map of framework worker limits. The only supported
key is `extract`.
`stage_workers.extract` defaults to the effective `total_llm` value after file
and environment precedence. It must be between `1` and `total_llm`, inclusive.
Unknown or empty stage-worker keys are rejected. The environment override
`NOTARIUS_STAGE_WORKERS_EXTRACT` takes precedence over the file value, as does
`NOTARIUS_TOTAL_LLM_CONCURRENCY` for the global ceiling.
The worker value is present in effective and redacted configuration. The
current runner remains serial; this setting does not yet change execution
scheduling.
## Pipelines
A pipeline selects implementations for the fixed workflow defined by
@@ -262,9 +282,9 @@ production validators do not call the LLM and must not set `llm_profile`.
| input | `seriatim` | Reads Seriatim transcript JSON. |
| chunk | `generic` | Splits source units into ordered chunks. |
| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. |
| extract | `dnd/spells` | Extracts D&D spell raw outputs. |
| merge | `appendorder` | Merges JSON raw extract outputs in chunk order. |
| normalize | `noop` | Passes merged raw outputs through unchanged. |
| extract | `dnd/spells` | Extracts typed D&D spell-list artifacts. |
| merge | `appendorder` | Combines typed artifacts in chunk order. |
| normalize | `noop` | Passes merged typed artifacts through unchanged. |
| output | `json` | Produces JSON output files for normalized `application/json` lanes. |
## Implemented Production Validators
@@ -275,7 +295,7 @@ production validators do not call the LLM and must not set `llm_profile`.
| `generic/always_reject` | deterministic | Rejects returned module output with reason `always_reject`. |
| `generic/valid_json` | deterministic | Rejects payloads that are not syntactically valid JSON. |
| `generic/valid_json_schema` | deterministic | Rejects invalid JSON or JSON that does not conform to the module response schema. |
| `extract/dnd/spells/shape` | deterministic | Rejects malformed D&D spell-cast JSON payloads. |
| `extract/dnd/spells/shape` | deterministic | Rejects malformed D&D spell-list artifacts. |
| `extract/dnd/spells/source_refs` | deterministic | Rejects missing or invalid D&D spell source references. |
| `extract/dnd/spells/source_relatedness` | deterministic | Emits warnings when a spell name is not found near its cited source text. |
@@ -366,6 +386,8 @@ Configuration validation checks:
- mutually exclusive `scriptorium.profile_dir` and `scriptorium.profile_file`;
- non-empty, non-duplicated IDs after trimming;
- positive global LLM concurrency;
- supported stage-worker keys and an effective extract worker count in the
inclusive range `1..concurrency.total_llm`;
- supported diagnostics retention and non-empty work directory;
- stale removed fields such as `llm_profiles`.

View File

@@ -209,8 +209,8 @@ does not inventory implementations.
## Tests To Inspect
- Package-local `*_test.go` files under the module or validator being changed.
- `internal/framework/pipeline/registry_integration_test.go`: registry and spec
composition.
- `internal/framework/pipeline/typed_resolution_test.go`: typed registry, spec,
and heterogeneous artifact composition.
- `internal/framework/pipeline/default_modules_test.go`: framework binding
defaults.
- `internal/cli/run_test.go`: production catalog, config resolution, and

View File

@@ -245,9 +245,10 @@ durable manifest and logical file schemas are defined in the
construction order, dependency failures, and the before-source-work boundary.
- `internal/framework/pipeline/references_test.go`: target resolution and
materialization.
- `internal/framework/pipeline/runner_test.go`: stage transitions, retries,
rejections, warnings, checkpoints, debug hooks, and manifests.
- `internal/framework/pipeline/walking_skeleton_test.go`: fake-backed complete
workflow composition.
- `internal/cli/run_test.go`: production stage transitions, retries, rejections,
warnings, debug hooks, manifests, and end-to-end composition.
- `internal/modules/integration/*_test.go` and
`internal/modules/seriatim/input/transcript/runner_test.go`: typed runner
composition across concrete module families.
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse
collaborators.

View File

@@ -111,8 +111,8 @@ file placement, checkpoints, or diagnostics.
## Validation
Validation is a framework-managed boundary around raw outputs from chunk,
extract, merge, and normalize stages. Validators receive immutable stage output
Validation is a framework-managed boundary around outputs from chunk, extract,
merge, and normalize stages. Validators receive immutable stage output
and make an explicit whole-output decision: approve, approve with warnings, or
reject.