Prepare pipelines before source execution

This commit is contained in:
2026-07-17 06:18:46 +00:00
parent 1c84d19e5f
commit ce3a07512f
26 changed files with 1562 additions and 404 deletions

View File

@@ -33,7 +33,9 @@ without exposing Scriptorium types through stage contracts.
2. creating a `ScriptoriumClient` from the effective profile source;
3. attaching an `LLMProfileRecorder`;
4. creating a scheduler from the effective concurrency limit;
5. returning a `ScheduledClient` wrapper.
5. returning a `ScheduledClient` wrapper; and
6. injecting that one shared client into complete pipeline preparation before
the source file is read or the runner is invoked.
The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
and validator bindings. It prepares a small internal check prompt for each ID so

View File

@@ -19,10 +19,15 @@ validator chains and prompt asset collection.
Production extract, merge, normalize, and validator packages currently use the
explicit legacy raw registration APIs. Typed registration is framework-ready,
but no production artifact kind or codec is registered yet.
but no production artifact kind or codec is registered yet. All selected
production implementations are constructed during pipeline preparation through
temporary adapters around their existing zero-argument constructors. Their raw
option maps and LLM clients remain operation inputs until each implementation
migrates to its construction-owned decoder and injected dependencies.
Specs expose capability and execution metadata without constructing an
implementation. Chunk, extract, merge, and normalize modules that accept
implementation. Registry entries separately expose option validation and
run-local construction. Chunk, extract, merge, and normalize modules that accept
auxiliary material declare identical reference slots from both
`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
that agreement. Runtime delivery uses the corresponding stage request's

View File

@@ -15,7 +15,8 @@ output files returned by the runner. Diagnostics, checkpoints, and debug
recorders are optional side-channel collaborators supplied at this boundary.
Pipeline execution is serial. Resolution produces a fixed ordered workflow and
a sorted set of artifact lanes before the runner constructs any stage module.
a sorted set of artifact lanes. Preparation constructs the complete module and
validator set before the runner receives source bytes.
## Application Boundary
@@ -39,7 +40,7 @@ a sorted set of artifact lanes before the runner constructs any stage module.
| Package | Implemented responsibility |
| --- | --- |
| `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. |
| `internal/framework/pipeline` | Module and artifact-codec registries, profile resolution, capability checks, reference materialization, validator-chain resolution, retries, orchestration, warnings, and manifest population. |
| `internal/framework/pipeline` | Module and artifact-codec registries, option validation, profile resolution, capability checks, reference materialization, complete pipeline preparation, retries, orchestration, warnings, and manifest population. |
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
@@ -56,9 +57,13 @@ Artifact registries support heterogeneous typed extraction entries and
kind-specific merger, normalizer, and validator variants. Resolution derives a
lane's kind from its extractor, requires the matching codec, verifies exact Go
type equality across the lane, and records schema identity in the resolved lane
and pipeline digest. Production module families do not register typed variants
yet and continue through explicitly named legacy raw registrations. The current
runner rejects a typed resolved lane instead of routing it through raw execution.
and pipeline digest. Registry entries carry separate option-validation and
run-local construction closures. Preparation injects shared dependencies and
constructs input, chunk, validators, ordered lanes, and output before source
parsing. Production module families do not register typed variants yet and
continue through explicitly named legacy raw registrations and temporary
zero-argument constructor adapters. The current runner rejects a typed prepared
lane instead of routing it through raw execution.
## Production Extensions

View File

@@ -7,7 +7,8 @@ defaults, and selectable keys are defined in
[Configuration](../config.md#pipelines).
Pipeline execution is serial. Resolution fixes the selected lanes and all
stage bindings before the runner constructs stage implementations.
stage bindings; preparation constructs every selected implementation before the
runner begins source work.
## Resolution
@@ -24,7 +25,9 @@ calls `pipeline.ResolvePipeline`.
selects exact-type merger, normalizer, and validator variants;
5. checks required and provided capabilities in workflow order;
6. resolves target-aware reference bindings and validator chains;
7. calculates a digest over the resolved structure, including typed artifact
7. validates each selected module and validator option set through its registry
entry; and
8. calculates a digest over the resolved structure, including typed artifact
kind and schema identity.
Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
@@ -53,7 +56,8 @@ runtime sensitive-data handling belongs in [Operations](../operations.md).
## Registries And Specs
`pipeline.Registries` holds constructors used during execution.
`pipeline.Registries` holds option validators and run-local builders used during
resolution and preparation.
`pipeline.ModuleCatalog` exposes their specs during configuration validation and
resolution. Separate registries exist for every stage and for validators;
`ValidatorChainRegistry` stores production default-chain mappings. Both
@@ -77,24 +81,37 @@ A `ModuleSpec` declares its stage plus required and provided capabilities.
Chunk, extract, merge, and normalize specs may also declare reference slots.
Registry implementations defensively copy spec metadata, reject duplicate keys,
and verify that a constructed implementation reports the registered key.
Builder registrations accept `ModuleDependencies` and cloned raw options through
one `BuildRequest`. Existing production registrations are adapted from their
zero-argument constructors while their implementation-owned option decoders are
migrated separately.
A `ValidatorSpec` declares a validator key and execution class. Resolution uses
the execution class to reject incompatible profile bindings before execution.
The current production catalog and default chain are listed only in
[Configuration](../config.md#implemented-production-validators).
## Runner Boundary
## Preparation And Runner Boundary
`pipeline.RunInput` carries the resolved pipeline, raw source input, structured
LLM client, run identity and timing, optional session and profile metadata, and
checkpoint/debug collaborators. The runner parses source bytes through the
selected input adapter. Later stage requests receive the generic source model;
extract requests receive chunk-scoped input material, while chunk, merge, and
normalize requests retain access to the original source material.
`pipeline.Prepare` receives a resolved pipeline, the registries, and shared
module dependencies. It constructs input; chunk and its validators; each lane's
extract, merge, and normalize modules and validator chains in resolved order;
then output. It stops at the first error with pipeline, stage, lane, module, and
validator context as applicable. It never invokes an operation method.
Typed lanes can be composed and resolved but are not passed to the current raw
runner. The runner rejects such input before source work; production lanes are
still resolved and executed exclusively through the legacy raw path.
`PreparedPipeline` keeps private constructed executors and exposes cloned
resolved input, chunk, lane, and output identities. `pipeline.RunInput` carries
that prepared pipeline, raw source input, run identity and timing, optional
session and profile metadata, and checkpoint/debug collaborators. The runner
parses source bytes through the already constructed input adapter. Later stage
requests receive the generic source model; extract requests receive
chunk-scoped input material, while chunk, merge, and normalize requests retain
access to the original source material.
Typed lanes can be composed, resolved, and prepared but are not executed by the
current raw runner. The runner rejects such input before source work;
production lanes are still resolved and executed exclusively through the
legacy raw path.
Source validation requires every unit to carry a canonical self-reference to
its containing document and its own unit ID. Explicit clone, checkpoint, and
@@ -111,17 +128,17 @@ runner returns.
The runner:
1. validates its input and registries;
2. builds the input adapter, parses the raw input, and validates the generic
1. validates its prepared input;
2. parses the raw input with the prepared adapter and validates the generic
source document;
3. obtains or executes the chunk result;
4. validates and canonicalizes chunks;
5. executes each resolved artifact lane in order;
6. builds the output encoder and validates its logical file results;
6. invokes the prepared output encoder and validates its logical file results;
7. returns the assembled manifest, outcomes, warnings, and files.
Within each artifact lane, it builds the extractor, merger, and normalizer,
then performs these transitions:
Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
and validators while performing these transitions:
1. extract once per accepted chunk and add runner-owned lane, source, and chunk
provenance;
@@ -208,8 +225,10 @@ durable manifest and logical file schemas are defined in the
- `internal/framework/pipeline/artifact_codec_registry_test.go`: typed codec
metadata, registration, erasure safety, strict decoding, and cloning.
- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed
lane resolution, target-specific validators, incompatibilities, ordering, and
schema-sensitive pipeline identity.
lane resolution and preparation, target-specific validators,
incompatibilities, ordering, and schema-sensitive pipeline identity.
- `internal/framework/pipeline/preparation_test.go`: option validation,
construction order, dependency failures, and the before-source-work boundary.
- `internal/framework/pipeline/references_test.go`: target resolution and
materialization.
- `internal/framework/pipeline/runner_test.go`: stage transitions, retries,