eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement runner retries and raw validation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-07 19:19:04 +00:00
parent bcedf19a08
commit cc6b050367
8 changed files with 805 additions and 129 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
docs/internal/pipeline.md
View File

@@ -116,8 +116,9 @@ The runner:
1. validates run input and registries;
2. builds the input adapter and parses the raw input into a source document;
3. validates the source document;
4. builds the chunker and produces source chunks;
5. validates source chunks against framework invariants;
4. builds the chunker and produces source chunks, retrying when configured;
5. validates source chunks against framework invariants and any registered raw
chunk validators;
6. runs each selected artifact lane in sorted resolved order;
7. builds the output encoder and validates logical output file names.
@@ -129,9 +130,8 @@ configured LLM profile, module options, and run metadata. Deterministic and
LLM-backed chunkers use the same contract; provider construction stays outside
chunk modules.
After `Chunk` returns, the runner appends chunker warnings before returning any
chunker error. When chunking succeeds, the runner validates generic chunk
invariants before running extractors:
When chunking succeeds, the runner validates generic chunk invariants before
running extractors:
- chunk IDs must be non-empty and unique in the chunk result;
- each chunk `SourceID` must match the source document ID;
@@ -150,6 +150,10 @@ chunk metadata. Extractors and downstream stages therefore see canonical source
units, while `SourceChunk.Metadata` remains the supported place for
chunker-owned context.
If chunk validation rejects a chunk after configured retries, the runner records
a rejected raw output and skips downstream lane execution. Framework-level
chunking or validation errors that remain after configured retries fail the run.
The framework does not require complete source-unit coverage and does not reject
overlap between different chunks. Stricter policies, such as full coverage or
non-overlap, belong to individual chunk modules when they are part of that
@@ -159,24 +163,37 @@ Within an artifact lane, the runner:
1. builds the extractor, merger, and normalizer;
2. records module manifest metadata when modules provide it;
3. extracts one raw `ExtractOutput` from each chunk;
3. extracts one raw `ExtractOutput` from each accepted chunk, retrying when
configured;
4. fills runner-owned provenance on each extract output, including lane ID,
extractor key, source ID, chunk ID, and chunk index;
5. merges ordered extract outputs into one raw `MergeOutput`;
6. normalizes the merge output into one raw `NormalizeOutput`;
7. appends the normalized raw output to `RunOutput.NormalizeOutputs`.
5. validates raw extract outputs and omits rejected outputs from merge input;
6. merges ordered accepted extract outputs into one raw `MergeOutput`, retrying
when configured;
7. validates raw merge output and skips normalization for rejected merge output;
8. normalizes the accepted merge output into one raw `NormalizeOutput`,
retrying when configured;
9. validates raw normalize output and appends accepted normalized raw output to
`RunOutput.NormalizeOutputs`.
## Validators
The current runner handoff is raw-output based. Extractors, mergers, and
normalizers do not advertise candidate validator chains through their module
interfaces. `RunOutput.Rejected` is reserved for rejected raw outputs when raw
validation is wired into the runner.
interfaces. Runner-side raw validation chains receive the raw module output plus
stage, lane, module, source, and chunk provenance. Empty raw validation chains
approve output by default.
Validator rejection is a non-fatal run outcome: the rejected output is recorded
in `RunOutput.Rejected` and does not pass to the next stage. Validator execution
errors are framework-level errors and retry according to the relevant binding.
## Warnings And Failures
Warnings from chunking, extraction, merging, normalization, and output encoding
are accumulated in `RunOutput.Warnings`.
Warnings from the successful chunking, extraction, merging, and normalization
attempts whose outputs are used are accumulated in `RunOutput.Warnings`, along
with output encoder warnings. Warnings from discarded retry attempts are not
promoted to final warnings.
Errors wrap the operation and module key or lane context. If execution fails
after a manifest exists, the returned manifest is marked `failed` and receives a