# Pipeline Internals

The implemented pipeline runner lives in `internal/framework/pipeline`. It
executes the fixed workflow defined by the architecture policy:

```text
input -> chunk -> extract -> merge -> normalize -> output
```

Pipeline execution is serial. The runner executes the resolved lanes one after
another in the fixed workflow order.

## Profile Resolution

Config loading produces `pipeline.PipelineProfile` values. Resolution happens
before execution:

1. `internal/core/config.Config.Resolve` validates config and finds the named
   pipeline.
2. The optional lane selection is passed to `pipeline.ResolvePipeline`.
3. Module bindings are defaulted:
   - chunk: `generic`
   - merge: `appendorder`
   - normalize: `noop`
   - output: `json`
   - LLM profile: `default`
4. The module catalog is checked for each bound module key.
5. Module capabilities are checked in workflow order.
6. A digest is calculated from the resolved pipeline without the digest field.

The CLI writes the resolved pipeline and digest to diagnostics.

## Registries And Module Specs

`pipeline.Registries` holds concrete constructors for execution. A
`pipeline.ModuleCatalog` exposes module specs for config validation and
resolution.

Every production module registers a `ModuleSpec` with:

- `Key`: module key used in config;
- `Stage`: module kind such as input, chunk, extract, merge, normalize,
  validate, or output;
- `Provides`: capabilities added after that module runs;
- `Requires`: capabilities that must already be available.

Capability checks prevent incompatible pipeline composition before a run starts.

## Runner Input And Output

`pipeline.RunInput` carries:

- a `ResolvedPipeline`;
- optional source ID, input path, and raw input bytes;
- a structured LLM client;
- run ID, start time, LLM profile manifest metadata, and CLI metadata.

`pipeline.RunOutput` carries:

- run manifest;
- approved artifacts;
- rejected artifacts;
- warnings;
- logical output files returned by the output encoder.

The CLI owns durable file writes and diagnostics writes after the runner returns.

## Execution

The runner:

1. validates run input and registries;
2. builds the input adapter and parses the raw input into a source document;
3. validates the source document;
4. builds the chunker and produces source chunks;
5. validates source chunks against framework invariants;
6. runs each selected artifact lane in sorted resolved order;
7. builds the output encoder and validates logical output file names.

## Chunk Results

Chunkers implement `contracts.Chunker` and receive a `contracts.ChunkRequest`
with the validated source document, the structured LLM client, the configured
LLM profile, module options, and run metadata. Deterministic and LLM-backed
chunkers use the same contract; provider construction stays outside chunk
modules.

After `Chunk` returns, the runner appends chunker warnings before returning any
chunker error. When chunking succeeds, the runner validates generic chunk
invariants before running extractors:

- chunk IDs must be non-empty and unique in the chunk result;
- each chunk `SourceID` must match the source document ID;
- each chunk `Index` must match its zero-based returned order;
- each chunk must contain at least one source unit;
- a chunk must not repeat a source unit;
- every chunk source unit must exist in the source document;
- source units inside each chunk must appear in source-document order.

The framework does not require complete source-unit coverage and does not reject
overlap between different chunks. Stricter policies, such as full coverage or
non-overlap, belong to individual chunk modules when they are part of that
module's contract.

Within an artifact lane, the runner:

1. builds the extractor, merger, and normalizer;
2. records module manifest metadata when modules provide it;
3. extracts candidates from each chunk;
4. normalizes candidate envelope fields such as index, extractor key, artifact
   type, and schema version;
5. merges candidates;
6. normalizes merged candidates;
7. validates candidate envelope consistency;
8. runs validators;
9. converts approved candidates to artifacts.

## Validators

If a lane declares validators in config, the runner builds those validators from
the validator registry. Otherwise it uses validators returned by the extractor.

Each validator must return exactly one decision for each eligible candidate. The
runner enforces decision cardinality with `internal/framework/validate`.
Rejected candidates are removed before the next validator runs. Approved
candidates continue through the chain.

The production CLI currently registers no standalone validator modules. The
current D&D spell extractor supplies deterministic shape and source-reference
validators.

## Warnings And Failures

Warnings from chunking, extraction, merging, normalization, validation, and
output encoding are accumulated in `RunOutput.Warnings`.

Errors wrap the operation and module key or lane context. If execution fails
after a manifest exists, the returned manifest is marked `failed` and receives a
completion timestamp.

On successful execution, the manifest validation status is:

- `approved` when no candidates were rejected;
- `rejected` when at least one candidate was rejected.

## Manifest Population

The manifest records run ID, pipeline ID, pipeline digest, module keys, artifact
lanes, LLM profile metadata, source digest, validation status, and timing.

Modules can add non-secret manifest metadata by implementing
`contracts.ManifestMetadataProvider`. The D&D spell extractor uses this for
prompt and response-schema provenance.