191 lines
9.1 KiB
Markdown
191 lines
9.1 KiB
Markdown
# Pipeline Internals
|
|
|
|
The implemented resolver and runner live in `internal/framework/pipeline`.
|
|
Their fixed workflow and ownership boundaries are defined by
|
|
[Architecture](../policy/architecture.md#system-shape). Configuration fields,
|
|
defaults, and selectable keys are defined in
|
|
[Configuration](../config.md#pipelines).
|
|
|
|
Pipeline execution is serial. Resolution fixes the selected lanes and all
|
|
stage bindings before the runner constructs stage implementations.
|
|
|
|
## Resolution
|
|
|
|
`internal/core/config.Config.Resolve` validates the loaded configuration,
|
|
selects the named profile, applies the runtime inputs supplied by the CLI, and
|
|
calls `pipeline.ResolvePipeline`.
|
|
|
|
`ResolvePipeline`:
|
|
|
|
1. selects and sorts artifact lanes;
|
|
2. completes omitted bindings using the documented configuration defaults;
|
|
3. looks up each module and validator spec without constructing it;
|
|
4. checks required and provided capabilities in workflow order;
|
|
5. resolves target-aware reference bindings and validator chains;
|
|
6. calculates a digest over the resolved structure.
|
|
|
|
Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
|
|
bindings, validator chains, reference targets, and the digest. It does not read
|
|
reference bytes or construct runtime modules. CLI lane and reference selector
|
|
syntax is defined in the [CLI reference](../cli.md#run).
|
|
|
|
## Reference Materialization
|
|
|
|
The CLI calls `MaterializeReferences` after resolution and before constructing
|
|
the LLM client or running the pipeline. The materializer checks each binding
|
|
against its resolved target declaration, reads and validates the file, and
|
|
builds both a `contracts.ReferenceSet` and provenance-only metadata on the
|
|
corresponding `ResolvedReferenceTarget`.
|
|
|
|
The runner clones the resulting set into the chunk, extract, merge, or normalize
|
|
request that owns the target. LLM-backed extensions may convert those items into
|
|
named prompt inputs. Reference content remains separate from source evidence and
|
|
source digests.
|
|
|
|
Binding precedence, path resolution, accepted content, and media-type behavior
|
|
are configuration contracts; see [Configuration](../config.md#pipelines).
|
|
Durable provenance is defined in the
|
|
[JSON output contract](../integrations/json-output.md#manifestjson), while
|
|
runtime sensitive-data handling belongs in [Operations](../operations.md).
|
|
|
|
## Registries And Specs
|
|
|
|
`pipeline.Registries` holds constructors used during execution.
|
|
`pipeline.ModuleCatalog` exposes their specs during configuration validation and
|
|
resolution. Separate registries exist for every stage and for validators;
|
|
`ValidatorChainRegistry` stores production default-chain mappings.
|
|
|
|
A `ModuleSpec` declares its stage plus required and provided capabilities.
|
|
Chunk, extract, merge, and normalize specs may also declare reference slots.
|
|
Registry implementations defensively copy spec metadata, reject duplicate keys,
|
|
and verify that a constructed implementation reports the registered key.
|
|
|
|
A `ValidatorSpec` declares a validator key and execution class. Resolution uses
|
|
the execution class to reject incompatible profile bindings before execution.
|
|
The current production catalog and default chain are listed only in
|
|
[Configuration](../config.md#implemented-production-validators).
|
|
|
|
## Runner Boundary
|
|
|
|
`pipeline.RunInput` carries the resolved pipeline, raw source input, structured
|
|
LLM client, run identity and timing, optional session and profile metadata, and
|
|
checkpoint/debug collaborators. The runner parses source bytes through the
|
|
selected input adapter. Later stage requests receive the generic source model;
|
|
extract requests receive chunk-scoped input material, while chunk, merge, and
|
|
normalize requests retain access to the original source material.
|
|
|
|
Source validation requires every unit to carry a canonical self-reference to
|
|
its containing document and its own unit ID. Explicit clone, checkpoint, and
|
|
debug boundaries retain that reference, and the canonical source digest covers
|
|
it deterministically.
|
|
|
|
`pipeline.RunOutput` carries the run manifest, accepted normalized results,
|
|
rejected results, warnings, checkpoint events, and logical files returned by the
|
|
output encoder. The CLI owns diagnostics and durable filesystem writes after the
|
|
runner returns.
|
|
|
|
## Execution Flow
|
|
|
|
The runner:
|
|
|
|
1. validates its input and registries;
|
|
2. builds the input adapter, parses the raw input, and validates the generic
|
|
source document;
|
|
3. obtains or executes the chunk result;
|
|
4. validates and canonicalizes chunks;
|
|
5. executes each resolved artifact lane in order;
|
|
6. builds the output encoder and validates its logical file results;
|
|
7. returns the assembled manifest, outcomes, warnings, and files.
|
|
|
|
Within each artifact lane, it builds the extractor, merger, and normalizer,
|
|
then performs these transitions:
|
|
|
|
1. extract once per accepted chunk and add runner-owned lane, source, and chunk
|
|
provenance;
|
|
2. validate each raw extract result and omit rejected results from merge input;
|
|
3. skip the rest of the lane when no extract result is accepted;
|
|
4. merge accepted extract results in their existing order;
|
|
5. validate the merge result and skip normalization on rejection;
|
|
6. normalize the accepted merge result;
|
|
7. validate and append the accepted normalized result.
|
|
|
|
Module-provided warnings and payload warnings are promoted only from attempts
|
|
whose results are accepted and used.
|
|
|
|
## Chunk Canonicalization
|
|
|
|
Before lane execution, generic validation requires unique chunk IDs, matching
|
|
source identity, indexes matching returned order, valid ordered boundaries,
|
|
non-empty content and media type, and at least one valid source unit per chunk.
|
|
Units may not repeat inside a chunk and must preserve source-document order.
|
|
|
|
The runner then rebuilds each chunk's unit slice from the source document by
|
|
unit ID. It preserves the module-owned boundaries, content, media type, and
|
|
cloned metadata. The framework permits gaps and overlap between separate
|
|
chunks; stricter coverage policy belongs to the chunk implementation.
|
|
|
|
## Validation And Retries
|
|
|
|
Chunk, extract, merge, and normalize results pass through the resolved validator
|
|
chain for their stage and module. Each validator receives the raw payload plus
|
|
the relevant source, chunk, prior-stage, schema, reference, session, LLM, option,
|
|
and run context. Validators execute in resolved order and stop at the first
|
|
error or rejection. An empty chain approves the result.
|
|
|
|
`runWithRetry` applies the effective retry policy around module execution and
|
|
its complete validation chain. A module or validator error becomes a framework
|
|
error when attempts are exhausted. A rejection becomes a recorded
|
|
`RejectedOutput` when attempts are exhausted. Cancellation stops retry
|
|
processing immediately.
|
|
|
|
Rejected output is a non-fatal pipeline outcome and does not advance. Warnings
|
|
from discarded attempts are not promoted. Configuration owns retry counts and
|
|
validator overrides; see [Module Bindings](../config.md#module-bindings).
|
|
|
|
## Checkpoint And Debug Hooks
|
|
|
|
The runner depends on recorder and loader interfaces, using no-op
|
|
implementations when collaborators are absent. Each checkpointed workflow
|
|
boundary records a running, succeeded, or failed transition. Reuse decisions
|
|
are consulted in workflow order and accepted payloads are cloned before
|
|
entering the normal handoff path. Dependency fingerprints connect later
|
|
checkpoints to the exact accepted results on which they depend.
|
|
|
|
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
|
|
boundaries. Context scopes associate nested LLM calls with the module or
|
|
validator attempt that made them. Debug-write failures are framework errors;
|
|
debug data is never used as a checkpoint source.
|
|
|
|
Checkpoint identity, physical layout, reuse behavior, and debug artifact
|
|
handling are operator contracts in [Operations](../operations.md). Serialization
|
|
and recorder implementation are inventoried in
|
|
[Internal Overview](overview.md#run-state-components).
|
|
|
|
## Results And Failures
|
|
|
|
The runner owns manifest assembly and handoff summaries but not the durable JSON
|
|
schema. It records resolved module and lane provenance, validator chains,
|
|
source/reference identities, selected LLM profiles, normalized and rejected
|
|
summaries, status, and timing. Raw payload bytes remain outside the manifest.
|
|
Module metadata providers may add non-secret singleton or lane-scoped metadata.
|
|
|
|
Execution errors include stage, module, lane, or validator context. Once a
|
|
manifest exists, a failing run returns it with failed status and completion
|
|
time. Successful status reflects whether any raw result was rejected. The
|
|
durable manifest and logical file schemas are defined in the
|
|
[JSON output contract](../integrations/json-output.md).
|
|
|
|
## Tests To Inspect
|
|
|
|
- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
|
|
- `internal/framework/pipeline/profile_test.go`: selection, defaults,
|
|
capabilities, validator chains, and digest behavior.
|
|
- `internal/framework/pipeline/references_test.go`: target resolution and
|
|
materialization.
|
|
- `internal/framework/pipeline/runner_test.go`: stage transitions, retries,
|
|
rejections, warnings, checkpoints, debug hooks, and manifests.
|
|
- `internal/framework/pipeline/walking_skeleton_test.go`: fake-backed complete
|
|
workflow composition.
|
|
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse
|
|
collaborators.
|