notarius/docs/internal/pipeline.md

# Pipeline Internals

The implemented resolver and runner live in `internal/framework/pipeline`.
Their fixed workflow and ownership boundaries are defined by
[Architecture](../policy/architecture.md#system-shape). Configuration fields,
defaults, and selectable keys are defined in
[Configuration](../config.md#pipelines).

Resolution fixes the selected lanes and all stage bindings; preparation
constructs every selected implementation before the runner begins source work.
After serial input parsing and plan selection or generation, the runner
materializes chunks and dispatches extract work to
one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's
merge and normalize operations remain serial and may overlap other lanes once
all extracts for that lane are terminal.

## Resolution

`internal/core/config.Config.Resolve` validates the loaded configuration,
selects the named profile, applies the runtime inputs supplied by the CLI, and
calls `pipeline.ResolvePipeline`.

`ResolvePipeline`:

1. selects and sorts artifact lanes;
2. completes omitted bindings using the documented configuration defaults;
3. looks up each module and validator spec without constructing it;
4. for a typed extractor, derives its artifact kind, requires the codec, and
   selects exact-type merger, normalizer, and validator variants;
5. checks required and provided capabilities in workflow order;
6. resolves target-aware reference bindings and validator chains;
7. validates each selected module and validator option set through its registry
   entry; and
8. calculates a digest over the resolved structure, including typed artifact
   kind and schema identity and the effective validator policy in its resolved
   execution order.

Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
bindings, validator chains, reference targets, and the digest. It does not read
reference bytes or construct runtime modules. CLI lane and reference selector
syntax is defined in the [CLI reference](../cli.md#run).

The digest includes each resolved validator chain's stage, lane, owning module,
ordered validator bindings, execution classes, targets, and artifact kinds.
Changing a default chain or an explicit override therefore changes pipeline
identity whenever it changes the effective validator policy.

## Reference Materialization

The CLI calls `MaterializeReferences` after resolution and before constructing
the LLM client or running the pipeline. The materializer checks each binding
against its resolved target declaration, reads and validates the file, and
builds both a `contracts.ReferenceSet` and provenance-only metadata on the
corresponding `ResolvedReferenceTarget`.

The runner clones the resulting set into the chunk, extract, merge, or normalize
request that owns the target. LLM-backed extensions may convert those items into
named prompt inputs. Reference content remains separate from source evidence and
source digests.

Binding precedence, path resolution, accepted content, and media-type behavior
are configuration contracts; see [Configuration](../config.md#pipelines).
Durable provenance is defined in the
[JSON output contract](../integrations/json-output.md#manifestjson), while
runtime sensitive-data handling belongs in [Operations](../operations.md).

## Registries And Specs

`pipeline.Registries` holds option validators and run-local builders used during
resolution and preparation.
`pipeline.ModuleCatalog` exposes their specs during configuration validation and
resolution. Separate registries exist for every stage and for validators;
`ValidatorChainRegistry` stores production default-chain mappings. Both
containers also carry an `ArtifactCodecRegistry`. Generic registration records
one codec per stable artifact kind, validates its schema metadata and JSON
Schema, retains the exact schema digest and Go type, and safely encodes or
decodes framework-erased values with typed errors on incompatibility.

Typed extractor entries are keyed by module key and declare one artifact kind.
Merger, normalizer, and typed-validator variants are keyed by module or
validator key plus artifact kind. Chunk and serialized validators occupy
separate target namespaces; serialized registrations declare whether they
support chunks, artifacts, or both. Duplicate variants and exact Go-type
mismatches are rejected deterministically.

Lane-sensitive merger and normalizer spec discovery always supplies the
extractor's artifact kind, so variants under one reusable key may declare
different capabilities and reference slots. Kind-neutral registry inspection
selects the first registered artifact kind in sorted order.

Production composition registers the D&D spell-list codec and typed extractor,
matching typed merge, normalize, and semantic-validator variants, and
serialized JSON validators. Every artifact lane resolves through the typed
registries and a matching codec.

A `ModuleSpec` declares its stage plus required and provided capabilities.
Chunk, extract, merge, and normalize specs may also declare reference slots.
Registry implementations defensively copy spec metadata, reject duplicate keys,
and verify that a constructed implementation reports the registered key.
Builder registrations accept `ModuleDependencies` and cloned configuration
options through one `BuildRequest`. Builders decode those options and retain
typed values or injected dependencies in the constructed implementation.
Extractors declare their artifact kind, and merger, normalizer, and validator
resolution selects the matching typed variant.

A `ValidatorSpec` declares a validator key and execution class. Resolution uses
the execution class to reject incompatible profile bindings before execution.
The current production catalog and default chain are listed only in
[Configuration](../config.md#implemented-production-validators).

## Preparation And Runner Boundary

`pipeline.Prepare` receives a resolved pipeline, the registries, and shared
module dependencies. It constructs input; chunk and its validators; each lane's
extract, merge, and normalize modules and validator chains in resolved order;
then output. It stops at the first error with pipeline, stage, lane, module, and
validator context as applicable. It never invokes an operation method.

`PreparedPipeline` keeps private constructed executors and exposes cloned
resolved input, chunk, lane, and output identities. `pipeline.RunInput` carries
that prepared pipeline, raw source input, run identity and timing, optional
session and profile metadata, a chunk-plan store and mode, and checkpoint/debug
collaborators. The runner
parses source bytes through the already constructed input adapter. Later stage
requests receive the generic source model; extract requests receive
chunk-scoped input material, while chunk, merge, and normalize requests retain
access to the original source material. Input, chunk, and output operation
requests do not carry raw module options. The chunk request also does not carry
an LLM client; an LLM-backed chunker receives the shared client during
preparation. Their operation requests retain run-specific source, reference,
profile, session, and metadata context as applicable.

Prepared lanes retain exact-type-checked erased operation closures. The runner
uses those closures to keep each value typed through extraction, validation,
merge, and normalization.

Source validation requires every unit to carry a canonical self-reference to
its containing document and its own unit ID. Explicit clone, checkpoint, and
debug boundaries retain that reference, and the canonical source digest covers
it deterministically. Chunks use the same source model and carry one canonical
reference spanning the first selected unit through the last.

`pipeline.RunOutput` carries the run manifest, accepted normalized serialized
artifacts with lane and normalizer provenance,
rejected results, warnings, checkpoint events, and logical files returned by the
output encoder. The CLI owns diagnostics and durable filesystem writes after the
runner returns.

## Execution Flow

The runner:

1. validates its prepared input;
2. parses the raw input with the prepared adapter and validates the generic
   source document;
3. selects a stored plan or executes the configured chunker's `Plan` operation;
4. canonicalizes and materializes the plan, then validates the resulting
   chunks;
5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
   bounded lane continuation when all extracts for that lane are terminal;
6. invokes the prepared output encoder and validates its logical file results;
7. returns the assembled manifest, outcomes, warnings, and files.

Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
and validators while performing these transitions:

1. extract once per accepted chunk and add runner-owned lane, source, and chunk
   provenance;
2. validate each extract result and omit rejected results from merge input;
3. skip the rest of the lane when no extract result is accepted;
4. merge accepted extract results in their existing order;
5. validate the merge result and skip normalization on rejection;
6. normalize the accepted merge result;
7. validate and append the accepted normalized result.

Module-provided warnings and payload warnings are promoted only from attempts
whose results are accepted and used.

## Chunk Plans And Reuse

`Chunker.Plan` returns a `source.ChunkPlan`: the canonical source digest,
ordered unit-ID ranges, and optional plan or range annotations. The framework
owns plan canonicalization and materialization. It creates the generic chunks
and therefore owns their IDs, indexes, source references, JSON content, units,
media type, and generic metadata. Plan and range annotations are independently
owned raw JSON and become `Chunk.PlanAnnotations` and `Chunk.Annotations`.

In `auto`, the runner looks up the source digest before invoking the chunker. A
valid hit is materialized and sent through the current run's configured chunk
validators; it does not invoke the chunk module, consume its retry budget, or
make a chunk-stage LLM call. A missing, invalid, or unmaterializable record
generates a candidate. `refresh` generates without lookup; `bypass` generates
without cache access. Generated plans are published only after the full chunk
validator chain approves them. A validator rejection is a regular rejected
pipeline outcome and never replaces a cached plan.

The store is source-addressed, not pipeline-addressed. Changes to pipeline
configuration, requested chunker, options, references, lanes, validators, or
LLM profile do not prevent a source-digest hit. The manifest records both the
currently requested chunker and the effective plan producer. Cache state and
paths are configured and operated outside the runner; see
[Configuration](../config.md#workspace) and [Operations](../operations.md).

The extract job channel has the same capacity as the effective extract worker
count, so dispatch applies backpressure. A fixed continuation executor prevents
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
continuations publish lane-local results; the coordinator is the only writer of
aggregate output and merges those results in resolved lane and source-chunk
order.

## Plan Canonicalization And Chunk Materialization

Plan canonicalization requires canonical JSON annotations, a matching source
digest, at least one range, existing ordered boundaries, and increasing range
starts. Ranges may overlap or leave gaps; a chunker may impose stricter policy.
Materialization deterministically reconstructs each range from the current
source document and copies annotations without interpreting their namespaces.

Before lane execution, generic chunk validation checks the materialized chunks'
identities, order, source references, content, media type, units, and metadata.
No chunk checkpoint participates in plan selection: plan storage is the only
chunk-reuse mechanism. Extract, merge, and normalize checkpoints continue to
use materialized chunk digests as their dependencies.

## Validation And Retries

Chunk, extract, merge, and normalize results pass through the resolved validator
chain for their stage and module. Chunk validators receive canonical chunks;
typed validators receive the domain value; and serialized validators receive
canonical chunk JSON or artifact codec bytes. Validators execute in resolved
order and stop at the first error or rejection. An empty chain approves the
result.

`runWithRetry` applies the effective retry policy around module execution and
its complete validation chain. A module or validator error becomes a framework
error when attempts are exhausted. A rejection becomes a recorded
`RejectedOutput` when attempts are exhausted. Cancellation stops retry
processing immediately.

Rejected output is a non-fatal pipeline outcome and does not advance. Warnings
from discarded attempts are not promoted. Configuration owns retry counts and
validator overrides; see [Module Bindings](../config.md#module-bindings).

## Checkpoint And Debug Hooks

The runner depends on recorder and loader interfaces, using no-op
implementations when collaborators are absent. Each checkpointed workflow
boundary records a running, succeeded, or failed transition. Reuse decisions
are consulted in workflow order and accepted payloads are cloned before
entering the normal handoff path. Typed extract, merge, and normalize
checkpoints store codec bytes with artifact kind, schema ID and version, exact
schema digest, and media type. Reuse compares that identity with the prepared
codec and decodes through the codec; missing identity, mismatches, corrupt
bytes, and decode failures become explicit reuse misses and execute the step
normally. Dependency fingerprints and debug content digests use the same stable
codec bytes that cross those boundaries.

Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
boundaries. Every executed chunk, extract, merge, and normalize attempt writes
one terminal envelope for acceptance, validator rejection, module or validator
error, or applicable candidate or final serialization error. The envelope
contains its attempt-local warnings, any available candidate and rejection,
and terminal error text; failures before a candidate exists omit that payload.
Only LLM calls made by the module operation belong to the module attempt.
Validator calls retain independent scopes under `validate/` and are not
duplicated into the module envelope. A failed terminal-envelope write is a
non-retryable framework error and is joined with any primary attempt error.
Debug data is never used as a checkpoint source. Typed artifact debug envelopes
are domain-neutral, redact sensitive metadata and bytes through the common
debug policy, and record codec identity plus schema and content digests.

Merge and normalize attempts serialize their in-memory candidate with the
codec's required candidate encoder before typed validation. Serialized
validators and attempt debug use that candidate representation, which carries
the codec media type and schema identity but is never checkpointed or passed
downstream. Only a validator-approved value is encoded through the strict final
codec and made eligible for a checkpoint or stage output.

Checkpoint identity, physical layout, reuse behavior, and debug artifact
handling are operator contracts in [Operations](../operations.md). Serialization
and recorder implementation are inventoried in
[Internal Overview](overview.md#run-state-components).

## Results And Failures

The runner owns manifest assembly and handoff summaries but not the durable JSON
schema. It records resolved module and lane provenance, validator chains,
source/reference identities, selected LLM profiles, normalized and rejected
summaries, status, and timing. Serialized artifact content remains outside the manifest.
Module metadata providers may add non-secret singleton or lane-scoped metadata.

Execution errors include stage, module, lane, or validator context. Once a
manifest exists, a failing run returns it with failed status and completion
time. Successful status reflects whether any result was rejected. The
durable manifest and logical file schemas are defined in the
[JSON output contract](../integrations/json-output.md).

On a framework failure, the runner cancels its derived context, stops submitting
new extract work, drains started tasks, and skips the output encoder. Parent
cancellation takes precedence. Otherwise context-cancellation fallout is
discarded when a substantive error exists, and the primary error is selected by
stage, resolved lane, and source chunk rather than completion time.

## Tests To Inspect

- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
- `internal/framework/pipeline/profile_test.go`: selection, defaults,
  capabilities, validator chains, and digest behavior.
- `internal/framework/pipeline/artifact_codec_registry_test.go`: typed codec
  metadata, registration, erasure safety, strict decoding, and cloning.
- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed
  lane resolution and preparation, target-specific validators,
  incompatibilities, ordering, and schema-sensitive pipeline identity.
- `internal/framework/pipeline/runner_concurrency_test.go`: bounded dispatch and
  continuations, reverse completion, stable errors, rejection, cancellation,
  retries, and independent provider-call limits.
- `internal/framework/pipeline/preparation_test.go`: option validation,
  construction order, dependency failures, and the before-source-work boundary.
- `internal/framework/pipeline/references_test.go`: target resolution and
  materialization.
- `internal/cli/run_test.go`: production stage transitions, retries, rejections,
  warnings, debug hooks, manifests, and end-to-end composition.
- `internal/modules/integration/*_test.go` and
  `internal/modules/seriatim/input/transcript/runner_test.go`: typed runner
  composition across concrete module families.
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse
  collaborators.