eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document accepted chunk map export

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-23 15:06:33 +00:00
parent 16a998055c
commit 06148074a2
13 changed files with 228 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
docs/internal/pipeline.md
View File

@@ -208,15 +208,18 @@ The runner:
3. selects a stored plan or executes the configured chunker's `Plan` operation;
4. canonicalizes and materializes the plan, then validates the resulting
chunks;
5. executes each resolved step in configuration order. For one step, it
5. builds the framework-owned accepted chunk map from the accepted source,
logical plan, and exact materialized chunks, then supplies it to the output
request independently of output-module options;
6. executes each resolved step in configuration order. For one step, it
dispatches extract jobs in source-chunk then resolved-lane order, starts a
bounded lane continuation when all extracts for that lane are terminal, and
waits for every lane to become terminal;
6. encodes and validates each accepted normalized producer artifact, then
7. encodes and validates each accepted normalized producer artifact, then
builds the immutable generated reference sets for the next step;
7. invokes the prepared output encoder only after every step succeeds and
8. invokes the prepared output encoder only after every step succeeds and
validates its logical file results;
8. returns the assembled manifest, outcomes, warnings, and files.
9. returns the assembled manifest, outcomes, warnings, and files.
Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
and validators while performing these transitions:
@@ -265,6 +268,14 @@ currently requested chunker and the effective plan producer. Cache state and
paths are configured and operated outside the runner; see
[Configuration](../config.md#state-surfaces) and [Operations](../operations.md).
For an accepted plan, the runner also constructs the strict framework-owned
[Accepted Chunk Map](../integrations/chunk-map.md) before lane execution. It
uses the current resolved chunker as `requested_chunker` and the stored or
generated record as `producer`, preserving that distinction on reuse. Chunk
rejection supplies no map; later lane rejection does not discard it. Output
encoders receive a defensively owned serialized value and may explicitly
ignore it.
The extract job channel has the same capacity as the effective extract worker
count, so dispatch applies backpressure. A fixed continuation executor prevents
ready or checkpoint-reused lanes from creating one goroutine each. Workers and