# Pipeline Internals

The implemented resolver and runner live in `internal/framework/pipeline`.
Their fixed workflow and ownership boundaries are defined by
[Architecture](../policy/architecture.md#system-shape). Configuration fields,
defaults, and selectable keys are defined in
[Configuration](../config.md#pipelines).

Pipeline execution is serial. Resolution fixes the selected lanes and all
stage bindings before the runner constructs stage implementations.

## Resolution

`internal/core/config.Config.Resolve` validates the loaded configuration,
selects the named profile, applies the runtime inputs supplied by the CLI, and
calls `pipeline.ResolvePipeline`.

`ResolvePipeline`:

1. selects and sorts artifact lanes;
2. completes omitted bindings using the documented configuration defaults;
3. looks up each module and validator spec without constructing it;
4. checks required and provided capabilities in workflow order;
5. resolves target-aware reference bindings and validator chains;
6. calculates a digest over the resolved structure.

Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
bindings, validator chains, reference targets, and the digest. It does not read
reference bytes or construct runtime modules. CLI lane and reference selector
syntax is defined in the [CLI reference](../cli.md#run).

## Reference Materialization

The CLI calls `MaterializeReferences` after resolution and before constructing
the LLM client or running the pipeline. The materializer checks each binding
against its resolved target declaration, reads and validates the file, and
builds both a `contracts.ReferenceSet` and provenance-only metadata on the
corresponding `ResolvedReferenceTarget`.

The runner clones the resulting set into the chunk, extract, merge, or normalize
request that owns the target. LLM-backed extensions may convert those items into
named prompt inputs. Reference content remains separate from source evidence and
source digests.

Binding precedence, path resolution, accepted content, and media-type behavior
are configuration contracts; see [Configuration](../config.md#pipelines).
Durable provenance is defined in the
[JSON output contract](../integrations/json-output.md#manifestjson), while
runtime sensitive-data handling belongs in [Operations](../operations.md).

## Registries And Specs

`pipeline.Registries` holds constructors used during execution.
`pipeline.ModuleCatalog` exposes their specs during configuration validation and
resolution. Separate registries exist for every stage and for validators;
`ValidatorChainRegistry` stores production default-chain mappings.

A `ModuleSpec` declares its stage plus required and provided capabilities.
Chunk, extract, merge, and normalize specs may also declare reference slots.
Registry implementations defensively copy spec metadata, reject duplicate keys,
and verify that a constructed implementation reports the registered key.

A `ValidatorSpec` declares a validator key and execution class. Resolution uses
the execution class to reject incompatible profile bindings before execution.
The current production catalog and default chain are listed only in
[Configuration](../config.md#implemented-production-validators).

## Runner Boundary

`pipeline.RunInput` carries the resolved pipeline, raw source input, structured
LLM client, run identity and timing, optional session and profile metadata, and
checkpoint/debug collaborators. The runner parses source bytes through the
selected input adapter. Later stage requests receive the generic source model;
extract requests receive chunk-scoped input material, while chunk, merge, and
normalize requests retain access to the original source material.

`pipeline.RunOutput` carries the run manifest, accepted normalized results,
rejected results, warnings, checkpoint events, and logical files returned by the
output encoder. The CLI owns diagnostics and durable filesystem writes after the
runner returns.

## Execution Flow

The runner:

1. validates its input and registries;
2. builds the input adapter, parses the raw input, and validates the generic
   source document;
3. obtains or executes the chunk result;
4. validates and canonicalizes chunks;
5. executes each resolved artifact lane in order;
6. builds the output encoder and validates its logical file results;
7. returns the assembled manifest, outcomes, warnings, and files.

Within each artifact lane, it builds the extractor, merger, and normalizer,
then performs these transitions:

1. extract once per accepted chunk and add runner-owned lane, source, and chunk
   provenance;
2. validate each raw extract result and omit rejected results from merge input;
3. skip the rest of the lane when no extract result is accepted;
4. merge accepted extract results in their existing order;
5. validate the merge result and skip normalization on rejection;
6. normalize the accepted merge result;
7. validate and append the accepted normalized result.

Module-provided warnings and payload warnings are promoted only from attempts
whose results are accepted and used.

## Chunk Canonicalization

Before lane execution, generic validation requires unique chunk IDs, matching
source identity, indexes matching returned order, valid ordered boundaries,
non-empty content and media type, and at least one valid source unit per chunk.
Units may not repeat inside a chunk and must preserve source-document order.

The runner then rebuilds each chunk's unit slice from the source document by
unit ID. It preserves the module-owned boundaries, content, media type, and
cloned metadata. The framework permits gaps and overlap between separate
chunks; stricter coverage policy belongs to the chunk implementation.

## Validation And Retries

Chunk, extract, merge, and normalize results pass through the resolved validator
chain for their stage and module. Each validator receives the raw payload plus
the relevant source, chunk, prior-stage, schema, reference, session, LLM, option,
and run context. Validators execute in resolved order and stop at the first
error or rejection. An empty chain approves the result.

`runWithRetry` performs the initial module call plus the configured additional
attempts. Each attempt includes module execution and its complete validation
chain. A module or validator error retries and becomes a framework error after
the final attempt. A rejection retries and becomes a recorded `RejectedOutput`
after the final attempt. Cancellation stops retry processing immediately.

Rejected output is a non-fatal pipeline outcome and does not advance. Warnings
from discarded attempts are not promoted. Configuration owns retry counts and
validator overrides; see [Module Bindings](../config.md#module-bindings).

## Checkpoint And Debug Hooks

The runner depends on recorder and loader interfaces, using no-op
implementations when collaborators are absent. Each checkpointed workflow
boundary records a running, succeeded, or failed transition. Reuse decisions
are consulted in workflow order and accepted payloads are cloned before
entering the normal handoff path. Dependency fingerprints connect later
checkpoints to the exact accepted results on which they depend.

Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
boundaries. Context scopes associate nested LLM calls with the module or
validator attempt that made them. Debug-write failures are framework errors;
debug data is never used as a checkpoint source.

Checkpoint identity, physical layout, reuse behavior, and debug artifact
handling are operator contracts in [Operations](../operations.md). Serialization
and recorder implementation are inventoried in
[Internal Overview](overview.md#run-state-components).

## Results And Failures

The runner owns manifest assembly and handoff summaries but not the durable JSON
schema. It records resolved module and lane provenance, validator chains,
source/reference identities, selected LLM profiles, normalized and rejected
summaries, status, and timing. Raw payload bytes remain outside the manifest.
Module metadata providers may add non-secret singleton or lane-scoped metadata.

Execution errors include stage, module, lane, or validator context. Once a
manifest exists, a failing run returns it with failed status and completion
time. Successful status reflects whether any raw result was rejected. The
durable manifest and logical file schemas are defined in the
[JSON output contract](../integrations/json-output.md).

## Tests To Inspect

- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
- `internal/framework/pipeline/profile_test.go`: selection, defaults,
  capabilities, validator chains, and digest behavior.
- `internal/framework/pipeline/references_test.go`: target resolution and
  materialization.
- `internal/framework/pipeline/runner_test.go`: stage transitions, retries,
  rejections, warnings, checkpoints, debug hooks, and manifests.
- `internal/framework/pipeline/walking_skeleton_test.go`: fake-backed complete
  workflow composition.
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse
  collaborators.