154 lines
5.5 KiB
Markdown
154 lines
5.5 KiB
Markdown
# Pipeline Internals
|
|
|
|
The implemented pipeline runner lives in `internal/framework/pipeline`. It
|
|
executes the fixed workflow defined by the architecture policy:
|
|
|
|
```text
|
|
input -> chunk -> extract -> merge -> normalize -> output
|
|
```
|
|
|
|
Pipeline execution is serial. The runner executes the resolved lanes one after
|
|
another in the fixed workflow order.
|
|
|
|
## Profile Resolution
|
|
|
|
Config loading produces `pipeline.PipelineProfile` values. Resolution happens
|
|
before execution:
|
|
|
|
1. `internal/core/config.Config.Resolve` validates config and finds the named
|
|
pipeline.
|
|
2. The optional lane selection is passed to `pipeline.ResolvePipeline`.
|
|
3. Module bindings are defaulted:
|
|
- chunk: `generic`
|
|
- merge: `appendorder`
|
|
- normalize: `noop`
|
|
- output: `json`
|
|
- LLM profile: `default`
|
|
4. The module catalog is checked for each bound module key.
|
|
5. Module capabilities are checked in workflow order.
|
|
6. A digest is calculated from the resolved pipeline without the digest field.
|
|
|
|
The CLI writes the resolved pipeline and digest to diagnostics.
|
|
|
|
## Registries And Module Specs
|
|
|
|
`pipeline.Registries` holds concrete constructors for execution. A
|
|
`pipeline.ModuleCatalog` exposes module specs for config validation and
|
|
resolution.
|
|
|
|
Every production module registers a `ModuleSpec` with:
|
|
|
|
- `Key`: module key used in config;
|
|
- `Stage`: module kind such as input, chunk, extract, merge, normalize,
|
|
validate, or output;
|
|
- `Provides`: capabilities added after that module runs;
|
|
- `Requires`: capabilities that must already be available.
|
|
|
|
Capability checks prevent incompatible pipeline composition before a run starts.
|
|
|
|
## Runner Input And Output
|
|
|
|
`pipeline.RunInput` carries:
|
|
|
|
- a `ResolvedPipeline`;
|
|
- optional source ID, input path, and raw input bytes;
|
|
- a structured LLM client;
|
|
- run ID, start time, LLM profile manifest metadata, and CLI metadata.
|
|
|
|
`pipeline.RunOutput` carries:
|
|
|
|
- run manifest;
|
|
- approved artifacts;
|
|
- rejected artifacts;
|
|
- warnings;
|
|
- logical output files returned by the output encoder.
|
|
|
|
The CLI owns durable file writes and diagnostics writes after the runner returns.
|
|
|
|
## Execution
|
|
|
|
The runner:
|
|
|
|
1. validates run input and registries;
|
|
2. builds the input adapter and parses the raw input into a source document;
|
|
3. validates the source document;
|
|
4. builds the chunker and produces source chunks;
|
|
5. validates source chunks against framework invariants;
|
|
6. runs each selected artifact lane in sorted resolved order;
|
|
7. builds the output encoder and validates logical output file names.
|
|
|
|
## Chunk Results
|
|
|
|
Chunkers implement `contracts.Chunker` and receive a `contracts.ChunkRequest`
|
|
with the validated source document, the structured LLM client, the configured
|
|
LLM profile, module options, and run metadata. Deterministic and LLM-backed
|
|
chunkers use the same contract; provider construction stays outside chunk
|
|
modules.
|
|
|
|
After `Chunk` returns, the runner appends chunker warnings before returning any
|
|
chunker error. When chunking succeeds, the runner validates generic chunk
|
|
invariants before running extractors:
|
|
|
|
- chunk IDs must be non-empty and unique in the chunk result;
|
|
- each chunk `SourceID` must match the source document ID;
|
|
- each chunk `Index` must match its zero-based returned order;
|
|
- each chunk must contain at least one source unit;
|
|
- a chunk must not repeat a source unit;
|
|
- every chunk source unit must exist in the source document;
|
|
- source units inside each chunk must appear in source-document order.
|
|
|
|
The framework does not require complete source-unit coverage and does not reject
|
|
overlap between different chunks. Stricter policies, such as full coverage or
|
|
non-overlap, belong to individual chunk modules when they are part of that
|
|
module's contract.
|
|
|
|
Within an artifact lane, the runner:
|
|
|
|
1. builds the extractor, merger, and normalizer;
|
|
2. records module manifest metadata when modules provide it;
|
|
3. extracts candidates from each chunk;
|
|
4. normalizes candidate envelope fields such as index, extractor key, artifact
|
|
type, and schema version;
|
|
5. merges candidates;
|
|
6. normalizes merged candidates;
|
|
7. validates candidate envelope consistency;
|
|
8. runs validators;
|
|
9. converts approved candidates to artifacts.
|
|
|
|
## Validators
|
|
|
|
If a lane declares validators in config, the runner builds those validators from
|
|
the validator registry. Otherwise it uses validators returned by the extractor.
|
|
|
|
Each validator must return exactly one decision for each eligible candidate. The
|
|
runner enforces decision cardinality with `internal/framework/validate`.
|
|
Rejected candidates are removed before the next validator runs. Approved
|
|
candidates continue through the chain.
|
|
|
|
The production CLI currently registers no standalone validator modules. The
|
|
current D&D spell extractor supplies deterministic shape and source-reference
|
|
validators.
|
|
|
|
## Warnings And Failures
|
|
|
|
Warnings from chunking, extraction, merging, normalization, validation, and
|
|
output encoding are accumulated in `RunOutput.Warnings`.
|
|
|
|
Errors wrap the operation and module key or lane context. If execution fails
|
|
after a manifest exists, the returned manifest is marked `failed` and receives a
|
|
completion timestamp.
|
|
|
|
On successful execution, the manifest validation status is:
|
|
|
|
- `approved` when no candidates were rejected;
|
|
- `rejected` when at least one candidate was rejected.
|
|
|
|
## Manifest Population
|
|
|
|
The manifest records run ID, pipeline ID, pipeline digest, module keys, artifact
|
|
lanes, LLM profile metadata, source digest, validation status, and timing.
|
|
|
|
Modules can add non-secret manifest metadata by implementing
|
|
`contracts.ManifestMetadataProvider`. The D&D spell extractor uses this for
|
|
prompt and response-schema provenance.
|