Prepare pipelines before source execution

This commit is contained in:
2026-07-17 06:18:46 +00:00
parent 1c84d19e5f
commit ce3a07512f
26 changed files with 1562 additions and 404 deletions

View File

@@ -7,7 +7,8 @@ defaults, and selectable keys are defined in
[Configuration](../config.md#pipelines).
Pipeline execution is serial. Resolution fixes the selected lanes and all
stage bindings before the runner constructs stage implementations.
stage bindings; preparation constructs every selected implementation before the
runner begins source work.
## Resolution
@@ -24,7 +25,9 @@ calls `pipeline.ResolvePipeline`.
selects exact-type merger, normalizer, and validator variants;
5. checks required and provided capabilities in workflow order;
6. resolves target-aware reference bindings and validator chains;
7. calculates a digest over the resolved structure, including typed artifact
7. validates each selected module and validator option set through its registry
entry; and
8. calculates a digest over the resolved structure, including typed artifact
kind and schema identity.
Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
@@ -53,7 +56,8 @@ runtime sensitive-data handling belongs in [Operations](../operations.md).
## Registries And Specs
`pipeline.Registries` holds constructors used during execution.
`pipeline.Registries` holds option validators and run-local builders used during
resolution and preparation.
`pipeline.ModuleCatalog` exposes their specs during configuration validation and
resolution. Separate registries exist for every stage and for validators;
`ValidatorChainRegistry` stores production default-chain mappings. Both
@@ -77,24 +81,37 @@ A `ModuleSpec` declares its stage plus required and provided capabilities.
Chunk, extract, merge, and normalize specs may also declare reference slots.
Registry implementations defensively copy spec metadata, reject duplicate keys,
and verify that a constructed implementation reports the registered key.
Builder registrations accept `ModuleDependencies` and cloned raw options through
one `BuildRequest`. Existing production registrations are adapted from their
zero-argument constructors while their implementation-owned option decoders are
migrated separately.
A `ValidatorSpec` declares a validator key and execution class. Resolution uses
the execution class to reject incompatible profile bindings before execution.
The current production catalog and default chain are listed only in
[Configuration](../config.md#implemented-production-validators).
## Runner Boundary
## Preparation And Runner Boundary
`pipeline.RunInput` carries the resolved pipeline, raw source input, structured
LLM client, run identity and timing, optional session and profile metadata, and
checkpoint/debug collaborators. The runner parses source bytes through the
selected input adapter. Later stage requests receive the generic source model;
extract requests receive chunk-scoped input material, while chunk, merge, and
normalize requests retain access to the original source material.
`pipeline.Prepare` receives a resolved pipeline, the registries, and shared
module dependencies. It constructs input; chunk and its validators; each lane's
extract, merge, and normalize modules and validator chains in resolved order;
then output. It stops at the first error with pipeline, stage, lane, module, and
validator context as applicable. It never invokes an operation method.
Typed lanes can be composed and resolved but are not passed to the current raw
runner. The runner rejects such input before source work; production lanes are
still resolved and executed exclusively through the legacy raw path.
`PreparedPipeline` keeps private constructed executors and exposes cloned
resolved input, chunk, lane, and output identities. `pipeline.RunInput` carries
that prepared pipeline, raw source input, run identity and timing, optional
session and profile metadata, and checkpoint/debug collaborators. The runner
parses source bytes through the already constructed input adapter. Later stage
requests receive the generic source model; extract requests receive
chunk-scoped input material, while chunk, merge, and normalize requests retain
access to the original source material.
Typed lanes can be composed, resolved, and prepared but are not executed by the
current raw runner. The runner rejects such input before source work;
production lanes are still resolved and executed exclusively through the
legacy raw path.
Source validation requires every unit to carry a canonical self-reference to
its containing document and its own unit ID. Explicit clone, checkpoint, and
@@ -111,17 +128,17 @@ runner returns.
The runner:
1. validates its input and registries;
2. builds the input adapter, parses the raw input, and validates the generic
1. validates its prepared input;
2. parses the raw input with the prepared adapter and validates the generic
source document;
3. obtains or executes the chunk result;
4. validates and canonicalizes chunks;
5. executes each resolved artifact lane in order;
6. builds the output encoder and validates its logical file results;
6. invokes the prepared output encoder and validates its logical file results;
7. returns the assembled manifest, outcomes, warnings, and files.
Within each artifact lane, it builds the extractor, merger, and normalizer,
then performs these transitions:
Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
and validators while performing these transitions:
1. extract once per accepted chunk and add runner-owned lane, source, and chunk
provenance;
@@ -208,8 +225,10 @@ durable manifest and logical file schemas are defined in the
- `internal/framework/pipeline/artifact_codec_registry_test.go`: typed codec
metadata, registration, erasure safety, strict decoding, and cloning.
- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed
lane resolution, target-specific validators, incompatibilities, ordering, and
schema-sensitive pipeline identity.
lane resolution and preparation, target-specific validators,
incompatibilities, ordering, and schema-sensitive pipeline identity.
- `internal/framework/pipeline/preparation_test.go`: option validation,
construction order, dependency failures, and the before-source-work boundary.
- `internal/framework/pipeline/references_test.go`: target resolution and
materialization.
- `internal/framework/pipeline/runner_test.go`: stage transitions, retries,