|
|
|
|
@@ -116,8 +116,9 @@ The runner:
|
|
|
|
|
1. validates run input and registries;
|
|
|
|
|
2. builds the input adapter and parses the raw input into a source document;
|
|
|
|
|
3. validates the source document;
|
|
|
|
|
4. builds the chunker and produces source chunks;
|
|
|
|
|
5. validates source chunks against framework invariants;
|
|
|
|
|
4. builds the chunker and produces source chunks, retrying when configured;
|
|
|
|
|
5. validates source chunks against framework invariants and any registered raw
|
|
|
|
|
chunk validators;
|
|
|
|
|
6. runs each selected artifact lane in sorted resolved order;
|
|
|
|
|
7. builds the output encoder and validates logical output file names.
|
|
|
|
|
|
|
|
|
|
@@ -129,9 +130,8 @@ configured LLM profile, module options, and run metadata. Deterministic and
|
|
|
|
|
LLM-backed chunkers use the same contract; provider construction stays outside
|
|
|
|
|
chunk modules.
|
|
|
|
|
|
|
|
|
|
After `Chunk` returns, the runner appends chunker warnings before returning any
|
|
|
|
|
chunker error. When chunking succeeds, the runner validates generic chunk
|
|
|
|
|
invariants before running extractors:
|
|
|
|
|
When chunking succeeds, the runner validates generic chunk invariants before
|
|
|
|
|
running extractors:
|
|
|
|
|
|
|
|
|
|
- chunk IDs must be non-empty and unique in the chunk result;
|
|
|
|
|
- each chunk `SourceID` must match the source document ID;
|
|
|
|
|
@@ -150,6 +150,10 @@ chunk metadata. Extractors and downstream stages therefore see canonical source
|
|
|
|
|
units, while `SourceChunk.Metadata` remains the supported place for
|
|
|
|
|
chunker-owned context.
|
|
|
|
|
|
|
|
|
|
If chunk validation rejects a chunk after configured retries, the runner records
|
|
|
|
|
a rejected raw output and skips downstream lane execution. Framework-level
|
|
|
|
|
chunking or validation errors that remain after configured retries fail the run.
|
|
|
|
|
|
|
|
|
|
The framework does not require complete source-unit coverage and does not reject
|
|
|
|
|
overlap between different chunks. Stricter policies, such as full coverage or
|
|
|
|
|
non-overlap, belong to individual chunk modules when they are part of that
|
|
|
|
|
@@ -159,24 +163,37 @@ Within an artifact lane, the runner:
|
|
|
|
|
|
|
|
|
|
1. builds the extractor, merger, and normalizer;
|
|
|
|
|
2. records module manifest metadata when modules provide it;
|
|
|
|
|
3. extracts one raw `ExtractOutput` from each chunk;
|
|
|
|
|
3. extracts one raw `ExtractOutput` from each accepted chunk, retrying when
|
|
|
|
|
configured;
|
|
|
|
|
4. fills runner-owned provenance on each extract output, including lane ID,
|
|
|
|
|
extractor key, source ID, chunk ID, and chunk index;
|
|
|
|
|
5. merges ordered extract outputs into one raw `MergeOutput`;
|
|
|
|
|
6. normalizes the merge output into one raw `NormalizeOutput`;
|
|
|
|
|
7. appends the normalized raw output to `RunOutput.NormalizeOutputs`.
|
|
|
|
|
5. validates raw extract outputs and omits rejected outputs from merge input;
|
|
|
|
|
6. merges ordered accepted extract outputs into one raw `MergeOutput`, retrying
|
|
|
|
|
when configured;
|
|
|
|
|
7. validates raw merge output and skips normalization for rejected merge output;
|
|
|
|
|
8. normalizes the accepted merge output into one raw `NormalizeOutput`,
|
|
|
|
|
retrying when configured;
|
|
|
|
|
9. validates raw normalize output and appends accepted normalized raw output to
|
|
|
|
|
`RunOutput.NormalizeOutputs`.
|
|
|
|
|
|
|
|
|
|
## Validators
|
|
|
|
|
|
|
|
|
|
The current runner handoff is raw-output based. Extractors, mergers, and
|
|
|
|
|
normalizers do not advertise candidate validator chains through their module
|
|
|
|
|
interfaces. `RunOutput.Rejected` is reserved for rejected raw outputs when raw
|
|
|
|
|
validation is wired into the runner.
|
|
|
|
|
interfaces. Runner-side raw validation chains receive the raw module output plus
|
|
|
|
|
stage, lane, module, source, and chunk provenance. Empty raw validation chains
|
|
|
|
|
approve output by default.
|
|
|
|
|
|
|
|
|
|
Validator rejection is a non-fatal run outcome: the rejected output is recorded
|
|
|
|
|
in `RunOutput.Rejected` and does not pass to the next stage. Validator execution
|
|
|
|
|
errors are framework-level errors and retry according to the relevant binding.
|
|
|
|
|
|
|
|
|
|
## Warnings And Failures
|
|
|
|
|
|
|
|
|
|
Warnings from chunking, extraction, merging, normalization, and output encoding
|
|
|
|
|
are accumulated in `RunOutput.Warnings`.
|
|
|
|
|
Warnings from the successful chunking, extraction, merging, and normalization
|
|
|
|
|
attempts whose outputs are used are accumulated in `RunOutput.Warnings`, along
|
|
|
|
|
with output encoder warnings. Warnings from discarded retry attempts are not
|
|
|
|
|
promoted to final warnings.
|
|
|
|
|
|
|
|
|
|
Errors wrap the operation and module key or lane context. If execution fails
|
|
|
|
|
after a manifest exists, the returned manifest is marked `failed` and receives a
|
|
|
|
|
|