notarius/docs/internal/pipeline.md

Pipeline Internals

This document describes the framework-owned pipeline mechanics in internal/framework/pipeline. Configuration owns selectable profiles, bindings, and retry settings; Operations owns state lifecycle and recovery; and the integration contracts own durable output shapes. Concrete production extensions are covered by Module Internals.

Boundary

The pipeline framework accepts a resolved composition, registries, shared dependencies, input bytes, a supplied prompt session, and state/debug collaborators. It returns logical output files, normalized artifacts, recorded rejections and warnings, manifest provenance, and checkpoint decisions. The CLI owns process arguments, configuration discovery, session resolution, physical roots, and placement of returned output files.

The framework has one fixed shape:

input -> chunk -> extract -> merge -> normalize -> output

Input and chunking are pipeline-wide. A selected artifact lane owns extract, merge, and normalize; output aggregates the terminal lane outcomes. A pipeline is an ordered list of steps, not an arbitrary workflow graph.

Resolve, Materialize, Prepare

Resolution turns a configured pipeline profile into a ResolvedPipeline. It normalizes the pipeline and lane identities, applies stage defaults, selects requested lanes where that is supported, resolves validator chains, checks module capabilities and typed artifact compatibility, validates options, and assigns a deterministic resolved-composition digest. The resolved pipeline contains bindings and declared reference targets, not external reference bytes. After selection, the resolver applies command, binding, and pipeline profile precedence to LLM-backed bindings and validators only; prompt defaults remain an empty resolved binding profile. Deterministic bindings remain profile-free. These effective values are part of the digest, so execution and checkpoint consumers do not repeat profile inheritance. Configuration resolution supplies the selected profile and catalog; see Configuration Internals.

External reference materialization happens before preparation. The materializer checks that each slot is declared by the selected module, resolves a file path relative to the correct configuration or working-directory origin, reads UTF-8 text, verifies media type and size limits, and retains bounded provenance. A generated-artifact selector remains declared but has no bytes until its producing step completes.

Preparation is the construction boundary. It validates the resolved shape and registry set, clones the resolved data, then constructs the input adapter, chunker, stage-local validators, every typed lane, and output encoder with cloned options, references, and shared dependencies. It also collects stable checkpoint fingerprints. Missing registrations, incompatible typed entries, nil implementations, and constructor failures are reported before source parsing or any stage operation begins.

An output encoder can opt into source-evidence publication through its output policy. Preparation keeps the configured lane allowlist and active lanes separate, then verifies an exact typed evidence projector and registered codec for each active lane. The resulting private plan is immutable; lanes excluded by invocation filtering remain configured but do not acquire a projector for that run.

Typed Lanes And References

Each resolved lane has one artifact kind, codec, and exact Go type. The framework uses private type erasure only around those typed operations; every handoff checks exact type and codec identity and reports incompatibility as an error rather than panicking. Encoding through the registered codec is the boundary for output, checkpoints, debug records, and generated references.

Reference targets are stage- and lane-specific. External reference bytes are cloned into the operation request. Generated references are built at the next step boundary from exactly one accepted normalized producer output. The framework decodes and re-encodes that output with the registered producer codec, checks its complete schema and media identity, and records a content digest plus bounded producer provenance. A missing, ambiguous, invalid, or incompatible producer prevents the consumer step from starting.

Execution And Ordering

The runner validates its input, installs no-op state collaborators when none were supplied, and serially performs source parsing and chunk-plan selection. It transports the supplied session unchanged to prompt-facing operations and run-manifest metadata; it neither derives a session nor substitutes a parsed source document identifier. The public session contract is owned by the CLI reference. An accepted plan is materialized into source-addressed chunks and passes the configured chunk validators before any lane runs. A chunk rejection is a recorded pipeline outcome: lanes do not start, but the output stage can encode the terminal result.

For each ordered step, the runner first builds generated reference sets from the accepted normalized outputs of earlier steps. It then executes the step's lanes. Later steps do not begin until the current step is terminal and its generated handoffs have succeeded.

Within a step, the lane engine dispatches extraction jobs in deterministic chunk-first, lane-second order to a bounded worker group. When all extraction jobs for one lane are terminal, a bounded continuation group can run that lane's merge and normalize work while extraction for other lanes continues. The framework does not create an unbounded goroutine per chunk or lane.

Completion timing does not determine public results. The coordinator restores lane and chunk order before merging results, and selects a framework error by stable stage, lane, and chunk position. A validator rejection records a lane outcome without cancelling unrelated work. A framework error or parent cancellation cancels derived work, prevents queued work from starting, waits for started workers, and prevents output encoding.

Validation, Retries, And Output

Every chunk, extract, merge, and normalize candidate passes its resolved validator chain. Validators receive immutable canonical input appropriate to their target: chunks, typed values, or serialized codec bytes. They may approve, approve with warnings, reject, or fail. A rejection is an ordinary pipeline result; a validator error is a framework error.

The runner applies the binding's retry policy around a stage operation and its complete validation chain. It preserves warnings only from the final accepted or rejected attempt. Cancellation stops retries. Normalizer-specific retry directives consume this same budget and validate any final safe fallback through the normalizer chain.

After terminal lane work, the runner assembles manifest provenance, normalized artifacts, rejections, warnings, and an optional accepted chunk map. When an output policy selected evidence lanes, it decodes accepted serialized normalize outputs through their registered codecs and invokes the prepared typed projectors. Rejected or absent lanes contribute nothing. This reconstruction is also used after normalized-checkpoint reuse, so no second typed output channel is retained. The runner passes the resulting owned artifact to the output encoder, which returns logical files and does not choose a physical directory. The CLI publishes those files only after the runner returns without a framework error. Logical file names and schemas are defined by the output integration contracts.

Checkpoint And Debug Hooks

The runner receives checkpoint and debug interfaces rather than roots. It records workflow transitions and reuse decisions through the supplied collaborators, and clones reusable artifacts before they re-enter normal typed handoff. Generated-reference dependencies participate in checkpoint decisions. Selective recomputation can require a canonical accepted normalized predecessor before a dependent lane starts.

Debug recording is attempt-scoped and application-owned. A failure to persist required debug data is a framework error. State roots, persistence, reason-code meanings, resume, and cleanup are intentionally owned by Run State Internals and Operations.

Invariants To Preserve

• The six fixed stages remain explicit; a pipeline is not a general DAG.
• Resolution and preparation reject statically discoverable incompatibility before parsing or execution.
• Every typed lane uses one compatible artifact kind, codec, and exact Go type.
• Generated references come only from one earlier accepted normalized producer and carry canonical identity rather than an unverified value.
• Rejections are recorded outcomes; framework errors cancel derived work and prevent output encoding.
• Public ordering and selected errors are independent of goroutine completion order.
• Pipeline modules receive collaborators and data, never CLI streams or physical output, cache, or debug roots.

Focused Tests

• internal/framework/pipeline/profile_test.go and typed_resolution_test.go cover resolution, defaults, ordered steps, compatibility, validators, references, and resolved identity.
• internal/framework/pipeline/preparation_test.go covers complete construction before execution and contextual construction failures.
• internal/framework/pipeline/references_test.go and handoff_test.go cover external materialization, generated references, provenance, and typed producer checks.
• internal/framework/pipeline/runner_concurrency_test.go covers bounded execution, ordered steps, stable error selection, rejections, and cancellation.
• internal/framework/pipeline/runner_chunk_plan_test.go, runner_typed_checkpoint_test.go, and runner_accepted_checkpoint_test.go cover state hooks and reuse behavior.
• internal/framework/pipeline/runner_attempt_debug_test.go and runner_terminal_debug_test.go cover attempt and terminal debug behavior.

Run go test ./internal/framework/pipeline ./internal/cli after changing a pipeline boundary. Use the more focused tests above while iterating.