# Pipeline Internals

This document describes the framework-owned pipeline mechanics in
**internal/framework/pipeline**. [Configuration](../config.md) owns selectable
profiles, bindings, and retry settings; [Operations](../operations.md) owns
state lifecycle and recovery; and the [integration contracts](../integrations/)
own durable output shapes. Concrete production extensions are covered by
[Module Internals](modules.md).

## Boundary

The pipeline framework accepts a resolved composition, registries, shared
dependencies, input bytes, a supplied prompt session, and state/debug
collaborators. It returns logical output files, normalized artifacts, recorded
rejections and warnings, manifest provenance, and checkpoint decisions. The
CLI owns process arguments, configuration discovery, session resolution,
physical roots, and placement of returned output files.

The framework has one fixed shape:

~~~
input -> chunk -> extract -> merge -> normalize -> output
~~~

Input and chunking are pipeline-wide. A selected artifact lane owns extract,
merge, and normalize; output aggregates the terminal lane outcomes. A pipeline
is an ordered list of steps, not an arbitrary workflow graph.

## Resolve, Materialize, Prepare

Resolution turns a configured pipeline profile into a **ResolvedPipeline**.
It normalizes the pipeline and lane identities, applies stage defaults, selects
requested lanes where that is supported, resolves validator chains, checks
module capabilities and typed artifact compatibility, validates options, and
assigns a deterministic resolved-composition digest. The resolved pipeline
contains bindings and declared reference targets, not external reference bytes.
After selection, the resolver applies command, binding, and pipeline profile
precedence to LLM-backed bindings and validators only; prompt defaults remain
an empty resolved binding profile. Deterministic bindings remain profile-free.
These effective values are part of the digest, so execution and checkpoint
consumers do not repeat profile inheritance.
Configuration resolution supplies the selected profile and catalog; see
[Configuration Internals](configuration.md).

External reference materialization happens before preparation. The materializer
checks that each slot is declared by the selected module, resolves a file path
relative to the correct configuration or working-directory origin, reads
UTF-8 text, verifies media type and size limits, and retains bounded
provenance. A generated-artifact selector remains declared but has no bytes
until its producing step completes.

Preparation is the construction boundary. It validates the resolved shape and
registry set, clones the resolved data, then constructs the input adapter,
chunker, stage-local validators, every typed lane, and output encoder with
cloned options, references, and shared dependencies. It also collects stable
checkpoint fingerprints. Missing registrations, incompatible typed entries,
nil implementations, and constructor failures are reported before source
parsing or any stage operation begins.

An output encoder can opt into source-evidence publication through its output
policy. Preparation keeps the configured lane allowlist and active lanes
separate, then verifies an exact typed evidence projector and registered codec
for each active lane. The resulting private plan is immutable; lanes excluded
by invocation filtering remain configured but do not acquire a projector for
that run.

## Typed Lanes And References

Each resolved lane has one artifact kind, codec, and exact Go type. The
framework uses private type erasure only around those typed operations; every
handoff checks exact type and codec identity and reports incompatibility as an
error rather than panicking. Encoding through the registered codec is the
boundary for output, checkpoints, debug records, and generated references.

Reference targets are stage- and lane-specific. External reference bytes are
cloned into the operation request. Generated references are built at the next
step boundary from exactly one accepted normalized producer output. The
framework decodes and re-encodes that output with the registered producer
codec, checks its complete schema and media identity, and records a content
digest plus bounded producer provenance. A missing, ambiguous, invalid, or
incompatible producer prevents the consumer step from starting.

## Execution And Ordering

The runner validates its input, installs no-op state collaborators when none
were supplied, and serially performs source parsing and chunk-plan selection.
It transports the supplied session unchanged to prompt-facing operations and
run-manifest metadata; it neither derives a session nor substitutes a parsed
source document identifier. The public session contract is owned by the
[CLI reference](../cli.md#run).
An accepted plan is materialized into source-addressed chunks and passes the
configured chunk validators before any lane runs. A chunk rejection is a
recorded pipeline outcome: lanes do not start, but the output stage can encode
the terminal result.

For each ordered step, the runner first builds generated reference sets from
the accepted normalized outputs of earlier steps. It then executes the step's
lanes. Later steps do not begin until the current step is terminal and its
generated handoffs have succeeded.

Within a step, the lane engine dispatches extraction jobs in deterministic
chunk-first, lane-second order to a bounded worker group. When all extraction
jobs for one lane are terminal, a bounded continuation group can run that
lane's merge and normalize work while extraction for other lanes continues.
The framework does not create an unbounded goroutine per chunk or lane.

Completion timing does not determine public results. The coordinator restores
lane and chunk order before merging results, and selects a framework error by
stable stage, lane, and chunk position. A validator rejection records a lane
outcome without cancelling unrelated work. A framework error or parent
cancellation cancels derived work, prevents queued work from starting, waits
for started workers, and prevents output encoding.

## Validation, Retries, And Output

Every chunk, extract, merge, and normalize candidate passes its resolved
validator chain. Validators receive immutable canonical input appropriate to
their target: chunks, typed values, or serialized codec bytes. They may
approve, approve with warnings, reject, or fail. A rejection is an ordinary
pipeline result; a validator error is a framework error.

The runner applies the binding's retry policy around a stage operation and its
complete validation chain. It preserves warnings only from the final accepted
or rejected attempt. Cancellation stops retries. Normalizer-specific retry
directives consume this same budget and validate any final safe fallback through
the normalizer chain.

After terminal lane work, the runner assembles manifest provenance, normalized
artifacts, rejections, warnings, and an optional accepted chunk map. When an
output policy selected evidence lanes, it decodes accepted serialized normalize
outputs through their registered codecs and invokes the prepared typed
projectors. Rejected or absent lanes contribute nothing. This reconstruction is
also used after normalized-checkpoint reuse, so no second typed output channel
is retained. The runner passes the resulting owned artifact to the output
encoder, which returns logical files and does not choose a physical directory.
The CLI publishes those files only after the runner returns without a framework
error. Logical file names and schemas are defined by the [output integration
contracts](../integrations/).

## Checkpoint And Debug Hooks

The runner receives checkpoint and debug interfaces rather than roots. It
records workflow transitions and reuse decisions through the supplied
collaborators, and clones reusable artifacts before they re-enter normal typed
handoff. Generated-reference dependencies participate in checkpoint decisions.
Selective recomputation can require a canonical accepted normalized predecessor
before a dependent lane starts.

Debug recording is attempt-scoped and application-owned. A failure to persist
required debug data is a framework error. State roots, persistence, reason-code
meanings, resume, and cleanup are intentionally owned by
[Run State Internals](state.md) and [Operations](../operations.md).

## Invariants To Preserve

- The six fixed stages remain explicit; a pipeline is not a general DAG.
- Resolution and preparation reject statically discoverable incompatibility
  before parsing or execution.
- Every typed lane uses one compatible artifact kind, codec, and exact Go type.
- Generated references come only from one earlier accepted normalized producer
  and carry canonical identity rather than an unverified value.
- Rejections are recorded outcomes; framework errors cancel derived work and
  prevent output encoding.
- Public ordering and selected errors are independent of goroutine completion
  order.
- Pipeline modules receive collaborators and data, never CLI streams or
  physical output, cache, or debug roots.

## Focused Tests

- **internal/framework/pipeline/profile_test.go** and
  **typed_resolution_test.go** cover resolution, defaults, ordered steps,
  compatibility, validators, references, and resolved identity.
- **internal/framework/pipeline/preparation_test.go** covers complete
  construction before execution and contextual construction failures.
- **internal/framework/pipeline/references_test.go** and **handoff_test.go**
  cover external materialization, generated references, provenance, and typed
  producer checks.
- **internal/framework/pipeline/runner_concurrency_test.go** covers bounded
  execution, ordered steps, stable error selection, rejections, and
  cancellation.
- **internal/framework/pipeline/runner_chunk_plan_test.go**,
  **runner_typed_checkpoint_test.go**, and
  **runner_accepted_checkpoint_test.go** cover state hooks and reuse behavior.
- **internal/framework/pipeline/runner_attempt_debug_test.go** and
  **runner_terminal_debug_test.go** cover attempt and terminal debug behavior.

Run **go test ./internal/framework/pipeline ./internal/cli** after changing a
pipeline boundary. Use the more focused tests above while iterating.
