Prepare pipelines before source execution
This commit is contained in:
@@ -33,7 +33,9 @@ without exposing Scriptorium types through stage contracts.
|
||||
2. creating a `ScriptoriumClient` from the effective profile source;
|
||||
3. attaching an `LLMProfileRecorder`;
|
||||
4. creating a scheduler from the effective concurrency limit;
|
||||
5. returning a `ScheduledClient` wrapper.
|
||||
5. returning a `ScheduledClient` wrapper; and
|
||||
6. injecting that one shared client into complete pipeline preparation before
|
||||
the source file is read or the runner is invoked.
|
||||
|
||||
The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
|
||||
and validator bindings. It prepares a small internal check prompt for each ID so
|
||||
|
||||
@@ -19,10 +19,15 @@ validator chains and prompt asset collection.
|
||||
|
||||
Production extract, merge, normalize, and validator packages currently use the
|
||||
explicit legacy raw registration APIs. Typed registration is framework-ready,
|
||||
but no production artifact kind or codec is registered yet.
|
||||
but no production artifact kind or codec is registered yet. All selected
|
||||
production implementations are constructed during pipeline preparation through
|
||||
temporary adapters around their existing zero-argument constructors. Their raw
|
||||
option maps and LLM clients remain operation inputs until each implementation
|
||||
migrates to its construction-owned decoder and injected dependencies.
|
||||
|
||||
Specs expose capability and execution metadata without constructing an
|
||||
implementation. Chunk, extract, merge, and normalize modules that accept
|
||||
implementation. Registry entries separately expose option validation and
|
||||
run-local construction. Chunk, extract, merge, and normalize modules that accept
|
||||
auxiliary material declare identical reference slots from both
|
||||
`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
|
||||
that agreement. Runtime delivery uses the corresponding stage request's
|
||||
|
||||
@@ -15,7 +15,8 @@ output files returned by the runner. Diagnostics, checkpoints, and debug
|
||||
recorders are optional side-channel collaborators supplied at this boundary.
|
||||
|
||||
Pipeline execution is serial. Resolution produces a fixed ordered workflow and
|
||||
a sorted set of artifact lanes before the runner constructs any stage module.
|
||||
a sorted set of artifact lanes. Preparation constructs the complete module and
|
||||
validator set before the runner receives source bytes.
|
||||
|
||||
## Application Boundary
|
||||
|
||||
@@ -39,7 +40,7 @@ a sorted set of artifact lanes before the runner constructs any stage module.
|
||||
| Package | Implemented responsibility |
|
||||
| --- | --- |
|
||||
| `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. |
|
||||
| `internal/framework/pipeline` | Module and artifact-codec registries, profile resolution, capability checks, reference materialization, validator-chain resolution, retries, orchestration, warnings, and manifest population. |
|
||||
| `internal/framework/pipeline` | Module and artifact-codec registries, option validation, profile resolution, capability checks, reference materialization, complete pipeline preparation, retries, orchestration, warnings, and manifest population. |
|
||||
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
|
||||
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
|
||||
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
|
||||
@@ -56,9 +57,13 @@ Artifact registries support heterogeneous typed extraction entries and
|
||||
kind-specific merger, normalizer, and validator variants. Resolution derives a
|
||||
lane's kind from its extractor, requires the matching codec, verifies exact Go
|
||||
type equality across the lane, and records schema identity in the resolved lane
|
||||
and pipeline digest. Production module families do not register typed variants
|
||||
yet and continue through explicitly named legacy raw registrations. The current
|
||||
runner rejects a typed resolved lane instead of routing it through raw execution.
|
||||
and pipeline digest. Registry entries carry separate option-validation and
|
||||
run-local construction closures. Preparation injects shared dependencies and
|
||||
constructs input, chunk, validators, ordered lanes, and output before source
|
||||
parsing. Production module families do not register typed variants yet and
|
||||
continue through explicitly named legacy raw registrations and temporary
|
||||
zero-argument constructor adapters. The current runner rejects a typed prepared
|
||||
lane instead of routing it through raw execution.
|
||||
|
||||
## Production Extensions
|
||||
|
||||
|
||||
@@ -7,7 +7,8 @@ defaults, and selectable keys are defined in
|
||||
[Configuration](../config.md#pipelines).
|
||||
|
||||
Pipeline execution is serial. Resolution fixes the selected lanes and all
|
||||
stage bindings before the runner constructs stage implementations.
|
||||
stage bindings; preparation constructs every selected implementation before the
|
||||
runner begins source work.
|
||||
|
||||
## Resolution
|
||||
|
||||
@@ -24,7 +25,9 @@ calls `pipeline.ResolvePipeline`.
|
||||
selects exact-type merger, normalizer, and validator variants;
|
||||
5. checks required and provided capabilities in workflow order;
|
||||
6. resolves target-aware reference bindings and validator chains;
|
||||
7. calculates a digest over the resolved structure, including typed artifact
|
||||
7. validates each selected module and validator option set through its registry
|
||||
entry; and
|
||||
8. calculates a digest over the resolved structure, including typed artifact
|
||||
kind and schema identity.
|
||||
|
||||
Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
|
||||
@@ -53,7 +56,8 @@ runtime sensitive-data handling belongs in [Operations](../operations.md).
|
||||
|
||||
## Registries And Specs
|
||||
|
||||
`pipeline.Registries` holds constructors used during execution.
|
||||
`pipeline.Registries` holds option validators and run-local builders used during
|
||||
resolution and preparation.
|
||||
`pipeline.ModuleCatalog` exposes their specs during configuration validation and
|
||||
resolution. Separate registries exist for every stage and for validators;
|
||||
`ValidatorChainRegistry` stores production default-chain mappings. Both
|
||||
@@ -77,24 +81,37 @@ A `ModuleSpec` declares its stage plus required and provided capabilities.
|
||||
Chunk, extract, merge, and normalize specs may also declare reference slots.
|
||||
Registry implementations defensively copy spec metadata, reject duplicate keys,
|
||||
and verify that a constructed implementation reports the registered key.
|
||||
Builder registrations accept `ModuleDependencies` and cloned raw options through
|
||||
one `BuildRequest`. Existing production registrations are adapted from their
|
||||
zero-argument constructors while their implementation-owned option decoders are
|
||||
migrated separately.
|
||||
|
||||
A `ValidatorSpec` declares a validator key and execution class. Resolution uses
|
||||
the execution class to reject incompatible profile bindings before execution.
|
||||
The current production catalog and default chain are listed only in
|
||||
[Configuration](../config.md#implemented-production-validators).
|
||||
|
||||
## Runner Boundary
|
||||
## Preparation And Runner Boundary
|
||||
|
||||
`pipeline.RunInput` carries the resolved pipeline, raw source input, structured
|
||||
LLM client, run identity and timing, optional session and profile metadata, and
|
||||
checkpoint/debug collaborators. The runner parses source bytes through the
|
||||
selected input adapter. Later stage requests receive the generic source model;
|
||||
extract requests receive chunk-scoped input material, while chunk, merge, and
|
||||
normalize requests retain access to the original source material.
|
||||
`pipeline.Prepare` receives a resolved pipeline, the registries, and shared
|
||||
module dependencies. It constructs input; chunk and its validators; each lane's
|
||||
extract, merge, and normalize modules and validator chains in resolved order;
|
||||
then output. It stops at the first error with pipeline, stage, lane, module, and
|
||||
validator context as applicable. It never invokes an operation method.
|
||||
|
||||
Typed lanes can be composed and resolved but are not passed to the current raw
|
||||
runner. The runner rejects such input before source work; production lanes are
|
||||
still resolved and executed exclusively through the legacy raw path.
|
||||
`PreparedPipeline` keeps private constructed executors and exposes cloned
|
||||
resolved input, chunk, lane, and output identities. `pipeline.RunInput` carries
|
||||
that prepared pipeline, raw source input, run identity and timing, optional
|
||||
session and profile metadata, and checkpoint/debug collaborators. The runner
|
||||
parses source bytes through the already constructed input adapter. Later stage
|
||||
requests receive the generic source model; extract requests receive
|
||||
chunk-scoped input material, while chunk, merge, and normalize requests retain
|
||||
access to the original source material.
|
||||
|
||||
Typed lanes can be composed, resolved, and prepared but are not executed by the
|
||||
current raw runner. The runner rejects such input before source work;
|
||||
production lanes are still resolved and executed exclusively through the
|
||||
legacy raw path.
|
||||
|
||||
Source validation requires every unit to carry a canonical self-reference to
|
||||
its containing document and its own unit ID. Explicit clone, checkpoint, and
|
||||
@@ -111,17 +128,17 @@ runner returns.
|
||||
|
||||
The runner:
|
||||
|
||||
1. validates its input and registries;
|
||||
2. builds the input adapter, parses the raw input, and validates the generic
|
||||
1. validates its prepared input;
|
||||
2. parses the raw input with the prepared adapter and validates the generic
|
||||
source document;
|
||||
3. obtains or executes the chunk result;
|
||||
4. validates and canonicalizes chunks;
|
||||
5. executes each resolved artifact lane in order;
|
||||
6. builds the output encoder and validates its logical file results;
|
||||
6. invokes the prepared output encoder and validates its logical file results;
|
||||
7. returns the assembled manifest, outcomes, warnings, and files.
|
||||
|
||||
Within each artifact lane, it builds the extractor, merger, and normalizer,
|
||||
then performs these transitions:
|
||||
Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
|
||||
and validators while performing these transitions:
|
||||
|
||||
1. extract once per accepted chunk and add runner-owned lane, source, and chunk
|
||||
provenance;
|
||||
@@ -208,8 +225,10 @@ durable manifest and logical file schemas are defined in the
|
||||
- `internal/framework/pipeline/artifact_codec_registry_test.go`: typed codec
|
||||
metadata, registration, erasure safety, strict decoding, and cloning.
|
||||
- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed
|
||||
lane resolution, target-specific validators, incompatibilities, ordering, and
|
||||
schema-sensitive pipeline identity.
|
||||
lane resolution and preparation, target-specific validators,
|
||||
incompatibilities, ordering, and schema-sensitive pipeline identity.
|
||||
- `internal/framework/pipeline/preparation_test.go`: option validation,
|
||||
construction order, dependency failures, and the before-source-work boundary.
|
||||
- `internal/framework/pipeline/references_test.go`: target resolution and
|
||||
materialization.
|
||||
- `internal/framework/pipeline/runner_test.go`: stage transitions, retries,
|
||||
|
||||
@@ -90,6 +90,12 @@ and verifies module availability and capabilities before execution. Structural
|
||||
pipeline choices must not be scattered through conditionals or hidden behind
|
||||
ad hoc command flags.
|
||||
|
||||
Resolution validates every selected module and validator option set. A separate
|
||||
preparation boundary then constructs the complete input, chunk, lane,
|
||||
validation, and output implementation set in pipeline order. The runner accepts
|
||||
only that prepared set, so construction and dependency failures occur before
|
||||
source parsing or any other module operation.
|
||||
|
||||
Stage ownership is explicit:
|
||||
|
||||
- input modules convert external material into the generic source model;
|
||||
|
||||
Reference in New Issue
Block a user