# Pipeline Internals

The implemented pipeline runner lives in `internal/framework/pipeline`. It
executes the fixed workflow defined by the architecture policy:

```text
input -> chunk -> extract -> merge -> normalize -> output
```

Pipeline execution is serial. The runner executes the resolved lanes one after
another in the fixed workflow order.

## Profile Resolution

Config loading produces `pipeline.PipelineProfile` values. Resolution happens
before execution:

1. `internal/core/config.Config.Resolve` validates config and finds the named
   pipeline.
2. The optional lane selection is passed to `pipeline.ResolvePipeline`.
3. Module bindings are defaulted:
   - chunk: `generic`
   - merge: `appendorder`
   - normalize: `noop`
   - output: `json`
   - LLM profile: empty, which lets Scriptorium prompt defaults choose a
     profile.
4. The module catalog is checked for each bound module key.
5. Module capabilities are checked in workflow order.
6. A digest is calculated from the resolved pipeline without the digest field.

The CLI writes the resolved pipeline and digest to diagnostics.

Pipeline profiles and artifact lanes may include reference binding maps keyed by
reference slot name. During resolution, pipeline-level bindings act as defaults
for selected chunk, extractor, merger, and normalizer targets that declare the
slot; target-local bindings override or add bindings for that target. Runtime
`--reference` requests override target config bindings, and runtime unbinds
remove optional target bindings. Flat runtime slot names are resolved only when
exactly one selected target declares the slot; otherwise the CLI requires a more
specific selector such as `chunk.slot`, `lane.extract.slot`,
`lane.merge.slot`, or `lane.normalize.slot`. Resolution validates bindings
against the declaring target specs and stores the bindings in target-aware
resolved reference holders. It does not read reference files or include
reference bytes in source digests.

During run preparation, resolved file references for chunk, extractor, merger,
and normalizer targets are materialized before any LLM-backed pipeline work. Config
bindings resolve relative to the config file, and CLI bindings resolve relative
to the current working directory. Materialization accepts UTF-8 text files,
computes `sha256:` content digests, records file origins, infers canonical base
media types from file extensions, enforces declared byte limits, and warns for
empty bound files. Media-type acceptance is checked only when a slot declares
`AcceptedMediaTypes`; unknown extensions are recorded as
`application/octet-stream`. Reference content is omitted from diagnostics and
manifests. The CLI writes provenance-only resolved reference diagnostics, and
the run manifest records target-stage reference provenance separately from
source digests. Runtime reference content is passed to the matching chunker,
extractor, merger, or normalizer request. LLM-backed modules pass that material
onward as named Scriptorium prompt inputs.

The CLI carries raw input bytes into `pipeline.RunInput`. Input adapters parse
those bytes into the source document. Chunk, merge, and normalize requests
receive the original source material as `SourceInput`; extraction requests
receive chunk-scoped source material built from the current `SourceChunk`
content, media type, and origin metadata. The raw input payload is not written
to manifests or default diagnostics.

The CLI also carries an optional run `session_id`. The runner makes it available
to chunk, extract, merge, and normalize requests; LLM-backed modules forward it
through their structured completion requests so Scriptorium can include it in
prompt execution metadata.

## Registries And Module Specs

`pipeline.Registries` holds concrete constructors for execution. A
`pipeline.ModuleCatalog` exposes module specs for config validation and
resolution.

Every production module registers a `ModuleSpec` with:

- `Key`: module key used in config;
- `Stage`: module kind such as input, chunk, extract, merge, normalize,
  validate, or output;
- `Provides`: capabilities added after that module runs;
- `Requires`: capabilities that must already be available.

Chunk, extract, merge, and normalize specs may also declare reference slots. Slot
declarations are available from registry metadata without constructing module
instances. Input, validate, and output specs must not declare reference slots.

Capability checks prevent incompatible pipeline composition before a run starts.

## Runner Input And Output

`pipeline.RunInput` carries:

- a `ResolvedPipeline`;
- optional source ID, input path, and raw input bytes;
- a structured LLM client;
- run ID, start time, LLM profile manifest metadata, and CLI metadata.

`pipeline.RunOutput` carries:

- run manifest;
- normalized raw outputs;
- rejected raw outputs;
- warnings;
- logical output files returned by the output encoder.

The CLI owns durable file writes and diagnostics writes after the runner returns.

## Execution

The runner:

1. validates run input and registries;
2. builds the input adapter and parses the raw input into a source document;
3. validates the source document;
4. builds the chunker and produces source chunks, retrying when configured;
5. validates source chunks against framework invariants and any registered raw
   chunk validators;
6. runs each selected artifact lane in sorted resolved order;
7. builds the output encoder and validates logical output file names.
8. passes accepted normalized raw outputs, rejected output records, warnings,
   and the manifest to the output encoder.

## Chunk Results

Chunkers implement `contracts.Chunker` and receive a `contracts.ChunkRequest`
with the validated source document, reference set, structured LLM client, the
configured LLM profile, module options, and run metadata. Deterministic and
LLM-backed chunkers use the same contract; provider construction stays outside
chunk modules.

When chunking succeeds, the runner validates generic chunk invariants before
running extractors:

- chunk IDs must be non-empty and unique in the chunk result;
- each chunk `SourceID` must match the source document ID;
- each chunk `Index` must match its zero-based returned order;
- each chunk start and end unit ID must exist in the source document, with the
  start unit at or before the end unit;
- each chunk must include non-empty extraction content and media type;
- each chunk must contain at least one source unit;
- a chunk must not repeat a source unit;
- every chunk source unit must exist in the source document;
- source units inside each chunk must appear in source-document order.

After validation, the runner rebuilds each chunk from source-document units by
integer ID, preserving chunk boundaries, content bytes, media type, and cloned
chunk metadata. Extractors and downstream stages therefore see canonical source
units, while `SourceChunk.Metadata` remains the supported place for
chunker-owned context.

If chunk validation rejects a chunk after configured retries, the runner records
a rejected raw output and skips downstream lane execution. Framework-level
chunking or validation errors that remain after configured retries fail the run.

The framework does not require complete source-unit coverage and does not reject
overlap between different chunks. Stricter policies, such as full coverage or
non-overlap, belong to individual chunk modules when they are part of that
module's contract.

Within an artifact lane, the runner:

1. builds the extractor, merger, and normalizer;
2. records module manifest metadata when modules provide it;
3. extracts one raw `ExtractOutput` from each accepted chunk, retrying when
   configured;
4. fills runner-owned provenance on each extract output, including lane ID,
   extractor key, source ID, chunk ID, and chunk index;
5. validates raw extract outputs and omits rejected outputs from merge input;
6. merges ordered accepted extract outputs into one raw `MergeOutput`, retrying
   when configured;
7. validates raw merge output and skips normalization for rejected merge output;
8. normalizes the accepted merge output into one raw `NormalizeOutput`,
   retrying when configured;
9. validates raw normalize output and appends accepted normalized raw output to
   `RunOutput.NormalizeOutputs`.

## Validators

The current runner handoff is raw-output based. Extractors, mergers, and
normalizers do not advertise validator chains through their module interfaces.
Runner-side raw validation chains receive the raw module output plus
stage, lane, module, source, and chunk provenance. Empty raw validation chains
approve output by default.

Pipeline-configured validator lists are not part of the current runner
contract. Non-empty configured validator lists are rejected during configuration
validation or resolved-run validation so they cannot appear in manifests without
executing.

Validator rejection is a non-fatal run outcome: the rejected output is recorded
in `RunOutput.Rejected` and does not pass to the next stage. Validator execution
errors are framework-level errors and retry according to the relevant binding.

## Warnings And Failures

Warnings from the successful chunking, extraction, merging, and normalization
attempts whose outputs are used are accumulated in `RunOutput.Warnings`, along
with output encoder warnings. Warnings from discarded retry attempts are not
promoted to final warnings.

Errors wrap the operation and module key or lane context. If execution fails
after a manifest exists, the returned manifest is marked `failed` and receives a
completion timestamp.

On successful execution, the manifest validation status is:

- `approved` when no raw outputs were rejected;
- `rejected` when at least one raw output was rejected.

## Manifest Population

The manifest records run ID, pipeline ID, pipeline digest, module keys, top-level
module metadata, artifact lanes, LLM profile metadata, source digest,
reference provenance, normalized raw output summaries, rejected output
summaries, validation status, and timing. Raw output summaries include lane ID,
normalizer module key, media type, source ID, and response-schema provenance
when present. Rejected output summaries include stage, lane, module, chunk,
validator or reason, message, attempt count, and optional diagnostic artifact
path. The manifest does not include raw output payload bytes.

Singleton pipeline modules may add non-secret metadata by implementing
`contracts.ManifestMetadataProvider`. The runner records that metadata under
`module_metadata` with stable keys for `input`, `chunker`, and `output`.

Lane-owned modules may add non-secret metadata through
`artifact_lanes[].metadata`. The runner records extractor, merger, and
normalizer metadata there. The D&D spell extractor uses lane metadata for
prompt and response-schema provenance.

## JSON Output

The production JSON output encoder writes `manifest.json`, `index.json`,
`warnings.json`, `rejected.json`, and one pretty-printed JSON file per accepted
normalized lane output under `lanes/`. It accepts only normalized outputs with
valid `application/json` payloads. Unsupported media types, invalid JSON, unsafe
logical paths, and duplicate sanitized lane file names fail the run before
durable output files are written.