282 lines
13 KiB
Markdown
282 lines
13 KiB
Markdown
# Pipeline Internals
|
|
|
|
The implemented pipeline runner lives in `internal/framework/pipeline`. It
|
|
executes the fixed workflow defined by the architecture policy:
|
|
|
|
```text
|
|
input -> chunk -> extract -> merge -> normalize -> output
|
|
```
|
|
|
|
Pipeline execution is serial. The runner executes the resolved lanes one after
|
|
another in the fixed workflow order.
|
|
|
|
## Profile Resolution
|
|
|
|
Config loading produces `pipeline.PipelineProfile` values. Resolution happens
|
|
before execution:
|
|
|
|
1. `internal/core/config.Config.Resolve` validates config and finds the named
|
|
pipeline.
|
|
2. The optional lane selection is passed to `pipeline.ResolvePipeline`.
|
|
3. Module bindings are defaulted:
|
|
- chunk: `generic`
|
|
- merge: `appendorder`
|
|
- normalize: `noop`
|
|
- output: `json`
|
|
- LLM profile: empty, which lets Scriptorium prompt defaults choose a
|
|
profile.
|
|
4. The module catalog is checked for each bound module key.
|
|
5. Module capabilities are checked in workflow order.
|
|
6. A digest is calculated from the resolved pipeline without the digest field.
|
|
|
|
The CLI writes the resolved pipeline and digest to diagnostics.
|
|
|
|
Pipeline profiles and artifact lanes may include reference binding maps keyed by
|
|
reference slot name. During resolution, pipeline-level bindings act as defaults
|
|
for selected chunk, extractor, merger, and normalizer targets that declare the
|
|
slot; target-local bindings override or add bindings for that target. Runtime
|
|
`--reference` requests override target config bindings, and runtime unbinds
|
|
remove optional target bindings. Flat runtime slot names are resolved only when
|
|
exactly one selected target declares the slot; otherwise the CLI requires a more
|
|
specific selector such as `chunk.slot`, `lane.extract.slot`,
|
|
`lane.merge.slot`, or `lane.normalize.slot`. Resolution validates bindings
|
|
against the declaring target specs and stores the bindings in target-aware
|
|
resolved reference holders. It does not read reference files or include
|
|
reference bytes in source digests.
|
|
|
|
During run preparation, resolved file references for chunk, extractor, merger,
|
|
and normalizer targets are materialized before any LLM-backed pipeline work. Config
|
|
bindings resolve relative to the config file, and CLI bindings resolve relative
|
|
to the current working directory. Materialization accepts UTF-8 text files,
|
|
computes `sha256:` content digests, records file origins, infers canonical base
|
|
media types from file extensions, enforces declared byte limits, and warns for
|
|
empty bound files. Media-type acceptance is checked only when a slot declares
|
|
`AcceptedMediaTypes`; unknown extensions are recorded as
|
|
`application/octet-stream`. Reference content is omitted from diagnostics and
|
|
manifests. The CLI writes provenance-only resolved reference diagnostics, and
|
|
the run manifest records target-stage reference provenance separately from
|
|
source digests. Runtime reference content is passed to the matching chunker,
|
|
extractor, merger, or normalizer request. LLM-backed modules pass that material
|
|
onward as named Scriptorium prompt inputs.
|
|
|
|
The CLI carries raw input bytes into `pipeline.RunInput`. Input adapters parse
|
|
those bytes into the source document. Chunk, merge, and normalize requests
|
|
receive the original source material as `SourceInput`; extraction requests
|
|
receive chunk-scoped source material built from the current `SourceChunk`
|
|
content, media type, and origin metadata. The raw input payload is not written
|
|
to manifests or default diagnostics.
|
|
|
|
The CLI also carries an optional run `session_id`. The runner makes it available
|
|
to chunk, extract, merge, and normalize requests; LLM-backed modules forward it
|
|
through their structured completion requests so Scriptorium can include it in
|
|
prompt execution metadata.
|
|
|
|
When workspace resume checkpointing is enabled, the CLI constructs a checkpoint
|
|
recorder after pipeline resolution and reference materialization and passes it
|
|
through `pipeline.RunInput`. The runner records source, chunk, extract, merge,
|
|
and normalize outcomes through that interface. Concrete modules do not receive
|
|
workspace paths and do not write checkpoint files directly.
|
|
|
|
For `run --resume`, the CLI also passes a checkpoint loader. The runner consults
|
|
the loader in workflow order and reuses only checkpoints whose manifest schema,
|
|
status, identity digest, dependency fingerprints, payload files, and payload
|
|
digests validate for the current invocation. Missing or invalid checkpoints fall
|
|
back to normal execution and are refreshed by the recorder.
|
|
|
|
## Registries And Module Specs
|
|
|
|
`pipeline.Registries` holds concrete constructors for execution. A
|
|
`pipeline.ModuleCatalog` exposes module specs for config validation and
|
|
resolution. The catalog also exposes validator specs and central default
|
|
validator-chain mappings without constructing modules or validators.
|
|
|
|
Every production module registers a `ModuleSpec` with:
|
|
|
|
- `Key`: module key used in config;
|
|
- `Stage`: module kind such as input, chunk, extract, merge, normalize,
|
|
validate, or output;
|
|
- `Provides`: capabilities added after that module runs;
|
|
- `Requires`: capabilities that must already be available.
|
|
|
|
Chunk, extract, merge, and normalize specs may also declare reference slots. Slot
|
|
declarations are available from registry metadata without constructing module
|
|
instances. Input, validate, and output specs must not declare reference slots.
|
|
|
|
Capability checks prevent incompatible pipeline composition before a run starts.
|
|
|
|
Every production validator registers a `ValidatorSpec` with:
|
|
|
|
- `Key`: validator key used in config and manifests;
|
|
- `ExecutionClass`: `deterministic` or `llm_backed`.
|
|
|
|
Default validator chains are keyed by workflow stage and module key. Production
|
|
currently registers a default chain for `extract` module `dnd/spells` only.
|
|
|
|
## Runner Input And Output
|
|
|
|
`pipeline.RunInput` carries:
|
|
|
|
- a `ResolvedPipeline`;
|
|
- optional source ID, input path, and raw input bytes;
|
|
- a structured LLM client;
|
|
- run ID, start time, LLM profile manifest metadata, and CLI metadata.
|
|
|
|
`pipeline.RunOutput` carries:
|
|
|
|
- run manifest;
|
|
- normalized raw outputs;
|
|
- rejected raw outputs;
|
|
- warnings;
|
|
- logical output files returned by the output encoder.
|
|
|
|
The CLI owns durable file writes and diagnostics writes after the runner returns.
|
|
|
|
## Execution
|
|
|
|
The runner:
|
|
|
|
1. validates run input and registries;
|
|
2. builds the input adapter and parses the raw input into a source document;
|
|
3. validates the source document;
|
|
4. builds the chunker and produces source chunks, retrying when configured;
|
|
5. validates source chunks against framework invariants and the resolved chunk
|
|
validator chain;
|
|
6. runs each selected artifact lane in sorted resolved order;
|
|
7. builds the output encoder and validates logical output file names.
|
|
8. passes accepted normalized raw outputs, rejected output records, warnings,
|
|
and the manifest to the output encoder.
|
|
|
|
## Chunk Results
|
|
|
|
Chunkers implement `contracts.Chunker` and receive a `contracts.ChunkRequest`
|
|
with the validated source document, reference set, structured LLM client, the
|
|
configured LLM profile, module options, and run metadata. Deterministic and
|
|
LLM-backed chunkers use the same contract; provider construction stays outside
|
|
chunk modules.
|
|
|
|
When chunking succeeds, the runner validates generic chunk invariants before
|
|
running extractors:
|
|
|
|
- chunk IDs must be non-empty and unique in the chunk result;
|
|
- each chunk `SourceID` must match the source document ID;
|
|
- each chunk `Index` must match its zero-based returned order;
|
|
- each chunk start and end unit ID must exist in the source document, with the
|
|
start unit at or before the end unit;
|
|
- each chunk must include non-empty extraction content and media type;
|
|
- each chunk must contain at least one source unit;
|
|
- a chunk must not repeat a source unit;
|
|
- every chunk source unit must exist in the source document;
|
|
- source units inside each chunk must appear in source-document order.
|
|
|
|
After validation, the runner rebuilds each chunk from source-document units by
|
|
integer ID, preserving chunk boundaries, content bytes, media type, and cloned
|
|
chunk metadata. Extractors and downstream stages therefore see canonical source
|
|
units, while `SourceChunk.Metadata` remains the supported place for
|
|
chunker-owned context.
|
|
|
|
If chunk validation rejects the chunk result after configured retries, the runner
|
|
records a rejected raw output and skips downstream lane execution. Framework-level
|
|
chunking or validation errors that remain after configured retries fail the run.
|
|
|
|
The framework does not require complete source-unit coverage and does not reject
|
|
overlap between different chunks. Stricter policies, such as full coverage or
|
|
non-overlap, belong to individual chunk modules when they are part of that
|
|
module's contract.
|
|
|
|
Within an artifact lane, the runner:
|
|
|
|
1. builds the extractor, merger, and normalizer;
|
|
2. records module manifest metadata when modules provide it;
|
|
3. extracts one raw `ExtractOutput` from each accepted chunk, retrying when
|
|
configured;
|
|
4. fills runner-owned provenance on each extract output, including lane ID,
|
|
extractor key, source ID, chunk ID, and chunk index;
|
|
5. validates raw extract outputs and omits rejected outputs from merge input;
|
|
6. merges ordered accepted extract outputs into one raw `MergeOutput`, retrying
|
|
when configured;
|
|
7. validates raw merge output and skips normalization for rejected merge output;
|
|
8. normalizes the accepted merge output into one raw `NormalizeOutput`,
|
|
retrying when configured;
|
|
9. validates raw normalize output and appends accepted normalized raw output to
|
|
`RunOutput.NormalizeOutputs`.
|
|
|
|
## Validators
|
|
|
|
The runner handoff is raw-output based. Chunkers, extractors, mergers, and
|
|
normalizers do not advertise validator chains through their module interfaces.
|
|
Resolved validation chains receive the raw module output plus stage, lane,
|
|
module, source, chunk, schema, session, reference, LLM client/profile, binding
|
|
option, and run metadata context. Chunk validators receive the chunk result
|
|
collection, merge validators receive the ordered extract outputs used by the
|
|
merge, and normalize validators receive the accepted merge output. Empty chains
|
|
approve output by default. Response-schema provenance may include in-memory JSON
|
|
schema bytes for validators. Those bytes are omitted from manifests,
|
|
diagnostics, and encoded output files.
|
|
|
|
Resolved validator chains come from central default mappings unless a
|
|
stage-local config override is set on `chunk`, lane `extract`, lane `merge`, or
|
|
lane `normalize`. Explicit empty overrides are valid and are recorded as empty
|
|
chains in manifests. Explicit non-empty overrides replace the default chain and
|
|
preserve configured order.
|
|
|
|
The production default chain for `extract` module `dnd/spells` is:
|
|
|
|
1. `generic/valid_json`
|
|
2. `generic/valid_json_schema`
|
|
3. `extract/dnd/spells/shape`
|
|
4. `extract/dnd/spells/source_refs`
|
|
5. `extract/dnd/spells/source_relatedness`
|
|
|
|
No other production module currently has a default validator chain.
|
|
|
|
Validator rejection is a non-fatal run outcome: the rejected output is recorded
|
|
in `RunOutput.Rejected` and does not pass to the next stage. Validator execution
|
|
errors are framework-level errors and retry according to the relevant binding.
|
|
Warning-only validators return approved results with warnings; those warnings
|
|
are promoted only from successful attempts whose outputs are used.
|
|
|
|
## Warnings And Failures
|
|
|
|
Warnings from the successful chunking, extraction, merging, and normalization
|
|
attempts whose outputs are used are accumulated in `RunOutput.Warnings`, along
|
|
with output encoder warnings. Warnings from discarded retry attempts are not
|
|
promoted to final warnings.
|
|
|
|
Errors wrap the operation and module key or lane context. If execution fails
|
|
after a manifest exists, the returned manifest is marked `failed` and receives a
|
|
completion timestamp.
|
|
|
|
On successful execution, the manifest validation status is:
|
|
|
|
- `approved` when no raw outputs were rejected;
|
|
- `rejected` when at least one raw output was rejected.
|
|
|
|
## Manifest Population
|
|
|
|
The manifest records run ID, pipeline ID, pipeline digest, module keys, top-level
|
|
module metadata, artifact lanes, LLM profile metadata, source digest,
|
|
reference provenance, normalized raw output summaries, rejected output
|
|
summaries, validation status, and timing. Raw output summaries include lane ID,
|
|
normalizer module key, media type, source ID, and response-schema provenance
|
|
when present. Rejected output summaries include stage, lane, module, chunk,
|
|
validator or reason, message, attempt count, and optional diagnostic artifact
|
|
path. The manifest does not include raw output payload bytes.
|
|
|
|
Singleton pipeline modules may add non-secret metadata by implementing
|
|
`contracts.ManifestMetadataProvider`. The runner records that metadata under
|
|
`module_metadata` with stable keys for `input`, `chunker`, and `output`.
|
|
|
|
Lane-owned modules may add non-secret metadata through
|
|
`artifact_lanes[].metadata`. The runner records extractor, merger, and
|
|
normalizer metadata there. The D&D spell extractor uses lane metadata for
|
|
prompt and response-schema provenance.
|
|
|
|
## JSON Output
|
|
|
|
The production JSON output encoder writes `manifest.json`, `index.json`,
|
|
`warnings.json`, `rejected.json`, and one pretty-printed JSON file per accepted
|
|
normalized lane output under `lanes/`. It accepts only normalized outputs with
|
|
valid `application/json` payloads. Unsupported media types, invalid JSON, unsafe
|
|
logical paths, and duplicate sanitized lane file names fail the run before
|
|
durable output files are written.
|