Cleanup and complete the pipeline refactor

This commit is contained in:
2026-07-07 15:32:35 -05:00
parent a9d8505cdb
commit 582c5dceed
14 changed files with 164 additions and 160 deletions

View File

@@ -207,7 +207,8 @@ The production CLI currently registers these module keys:
- normalize: `noop`
- output: `json`
The production CLI does not currently register validator modules.
Configured validator module lists are reserved for a future validator-chain
feature and are rejected by current configuration validation.
For YAML structure, Scriptorium profile sources, environment overrides, and
module binding syntax, see [Configuration](config.md).

View File

@@ -131,8 +131,8 @@ Artifact lane fields:
- `extract`: required module binding.
- `merge`: optional module binding. Default module is `appendorder`.
- `normalize`: optional module binding. Default module is `noop`.
- `validators`: optional list of module bindings. The production CLI currently
does not register validator modules.
- `validators`: reserved for future configurable validator chains. Non-empty
lists are rejected by current configuration validation.
- `references`: optional compatibility alias for extractor reference bindings.
Lane bindings override pipeline-level bindings for the same slot.
@@ -233,8 +233,9 @@ Binding fields:
`normalize` bindings.
- `options`: optional module-specific settings.
- `references`: optional reference bindings. Supported only for `chunk`,
`extract`, `merge`, and `normalize` bindings. `input`, validator, and
`output` bindings reject this field during validation.
`extract`, `merge`, and `normalize` bindings. `input` and `output` bindings
reject this field during validation. Validator bindings are reserved for a
future validator-chain feature and are rejected when configured.
The `--llm-profile` run flag overrides every effective LLM-capable module
binding to use one Scriptorium profile ID: chunk, every selected lane extract,

View File

@@ -149,9 +149,10 @@ text, or secrets.
Package: `internal/modules/extract/dnd/spells`
The `dnd/spells` extractor owns D&D spell-cast extraction semantics. It
supplies the embedded Scriptorium prompt ID, prompt version, transcript and
reference input materials, response schema, and session ID to the runtime; then
returns the structured LLM `spell_casts` response as raw JSON.
supplies the embedded Scriptorium prompt ID, prompt version, chunk-scoped
transcript input material, reference input materials, response schema, and
session ID to the runtime; then returns the structured LLM `spell_casts`
response as raw JSON.
Its LLM-facing source-reference schema uses integer `start_unit_id` and
`end_unit_id` values matching source-unit IDs.

View File

@@ -60,10 +60,11 @@ extractor, merger, or normalizer request. LLM-backed modules pass that material
onward as named Scriptorium prompt inputs.
The CLI carries raw input bytes into `pipeline.RunInput`. Input adapters parse
those bytes into the source document, while LLM-backed modules that need the
original transcript material can pass the same bytes as a prompt input with
origin metadata. The raw input payload is not written to manifests or default
diagnostics.
those bytes into the source document. Chunk, merge, and normalize requests
receive the original source material as `SourceInput`; extraction requests
receive chunk-scoped source material built from the current `SourceChunk`
content, media type, and origin metadata. The raw input payload is not written
to manifests or default diagnostics.
The CLI also carries an optional run `session_id`. The runner makes it available
to chunk, extract, merge, and normalize requests; LLM-backed modules forward it
@@ -186,6 +187,11 @@ Runner-side raw validation chains receive the raw module output plus
stage, lane, module, source, and chunk provenance. Empty raw validation chains
approve output by default.
Pipeline-configured validator lists are not part of the current runner
contract. Non-empty configured validator lists are rejected during configuration
validation or resolved-run validation so they cannot appear in manifests without
executing.
Validator rejection is a non-fatal run outcome: the rejected output is recorded
in `RunOutput.Rejected` and does not pass to the next stage. Validator execution
errors are framework-level errors and retry according to the relevant binding.

View File

@@ -320,7 +320,7 @@ Explanation and fixes:
- Increase a module binding's `retries` only when re-running the same module
input can reasonably produce an acceptable output.
- If rejection is deterministic, fix the source input, module configuration, or
validator configuration rather than adding retries.
validation policy rather than adding retries.
## Retry Exhaustion