Cleanup and complete the pipeline refactor
This commit is contained in:
@@ -207,7 +207,8 @@ The production CLI currently registers these module keys:
|
||||
- normalize: `noop`
|
||||
- output: `json`
|
||||
|
||||
The production CLI does not currently register validator modules.
|
||||
Configured validator module lists are reserved for a future validator-chain
|
||||
feature and are rejected by current configuration validation.
|
||||
|
||||
For YAML structure, Scriptorium profile sources, environment overrides, and
|
||||
module binding syntax, see [Configuration](config.md).
|
||||
|
||||
@@ -131,8 +131,8 @@ Artifact lane fields:
|
||||
- `extract`: required module binding.
|
||||
- `merge`: optional module binding. Default module is `appendorder`.
|
||||
- `normalize`: optional module binding. Default module is `noop`.
|
||||
- `validators`: optional list of module bindings. The production CLI currently
|
||||
does not register validator modules.
|
||||
- `validators`: reserved for future configurable validator chains. Non-empty
|
||||
lists are rejected by current configuration validation.
|
||||
- `references`: optional compatibility alias for extractor reference bindings.
|
||||
Lane bindings override pipeline-level bindings for the same slot.
|
||||
|
||||
@@ -233,8 +233,9 @@ Binding fields:
|
||||
`normalize` bindings.
|
||||
- `options`: optional module-specific settings.
|
||||
- `references`: optional reference bindings. Supported only for `chunk`,
|
||||
`extract`, `merge`, and `normalize` bindings. `input`, validator, and
|
||||
`output` bindings reject this field during validation.
|
||||
`extract`, `merge`, and `normalize` bindings. `input` and `output` bindings
|
||||
reject this field during validation. Validator bindings are reserved for a
|
||||
future validator-chain feature and are rejected when configured.
|
||||
|
||||
The `--llm-profile` run flag overrides every effective LLM-capable module
|
||||
binding to use one Scriptorium profile ID: chunk, every selected lane extract,
|
||||
|
||||
@@ -149,9 +149,10 @@ text, or secrets.
|
||||
Package: `internal/modules/extract/dnd/spells`
|
||||
|
||||
The `dnd/spells` extractor owns D&D spell-cast extraction semantics. It
|
||||
supplies the embedded Scriptorium prompt ID, prompt version, transcript and
|
||||
reference input materials, response schema, and session ID to the runtime; then
|
||||
returns the structured LLM `spell_casts` response as raw JSON.
|
||||
supplies the embedded Scriptorium prompt ID, prompt version, chunk-scoped
|
||||
transcript input material, reference input materials, response schema, and
|
||||
session ID to the runtime; then returns the structured LLM `spell_casts`
|
||||
response as raw JSON.
|
||||
Its LLM-facing source-reference schema uses integer `start_unit_id` and
|
||||
`end_unit_id` values matching source-unit IDs.
|
||||
|
||||
|
||||
@@ -60,10 +60,11 @@ extractor, merger, or normalizer request. LLM-backed modules pass that material
|
||||
onward as named Scriptorium prompt inputs.
|
||||
|
||||
The CLI carries raw input bytes into `pipeline.RunInput`. Input adapters parse
|
||||
those bytes into the source document, while LLM-backed modules that need the
|
||||
original transcript material can pass the same bytes as a prompt input with
|
||||
origin metadata. The raw input payload is not written to manifests or default
|
||||
diagnostics.
|
||||
those bytes into the source document. Chunk, merge, and normalize requests
|
||||
receive the original source material as `SourceInput`; extraction requests
|
||||
receive chunk-scoped source material built from the current `SourceChunk`
|
||||
content, media type, and origin metadata. The raw input payload is not written
|
||||
to manifests or default diagnostics.
|
||||
|
||||
The CLI also carries an optional run `session_id`. The runner makes it available
|
||||
to chunk, extract, merge, and normalize requests; LLM-backed modules forward it
|
||||
@@ -186,6 +187,11 @@ Runner-side raw validation chains receive the raw module output plus
|
||||
stage, lane, module, source, and chunk provenance. Empty raw validation chains
|
||||
approve output by default.
|
||||
|
||||
Pipeline-configured validator lists are not part of the current runner
|
||||
contract. Non-empty configured validator lists are rejected during configuration
|
||||
validation or resolved-run validation so they cannot appear in manifests without
|
||||
executing.
|
||||
|
||||
Validator rejection is a non-fatal run outcome: the rejected output is recorded
|
||||
in `RunOutput.Rejected` and does not pass to the next stage. Validator execution
|
||||
errors are framework-level errors and retry according to the relevant binding.
|
||||
|
||||
@@ -320,7 +320,7 @@ Explanation and fixes:
|
||||
- Increase a module binding's `retries` only when re-running the same module
|
||||
input can reasonably produce an acceptable output.
|
||||
- If rejection is deterministic, fix the source input, module configuration, or
|
||||
validator configuration rather than adding retries.
|
||||
validation policy rather than adding retries.
|
||||
|
||||
## Retry Exhaustion
|
||||
|
||||
|
||||
Reference in New Issue
Block a user