notarius/docs/internal/pipeline.md

Pipeline Internals

The implemented resolver and runner live in internal/framework/pipeline. Their fixed workflow and ownership boundaries are defined by Architecture. Configuration fields, defaults, and selectable keys are defined in Configuration.

Resolution fixes the ordered steps, selected lanes, and all stage bindings; preparation constructs every selected implementation before the runner begins source work. After serial input parsing and plan selection or generation, the runner materializes chunks and executes one step at a time. Within a step, extract work uses one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's merge and normalize operations remain serial, and lanes in the same step may overlap once their extracts are terminal. A later step cannot start across its barrier until every earlier lane is terminal and each required generated artifact has been accepted and handed off.

Resolution

internal/core/config.Config.Resolve validates the loaded configuration, selects the named profile, applies the runtime inputs supplied by the CLI, and calls pipeline.ResolvePipeline.

ResolvePipeline:

1. selects the explicit ordered steps, or creates the implicit default step from the legacy top-level artifacts map;
2. selects and sorts artifact lanes within each step while enforcing global lane identity;
3. completes omitted bindings using the documented configuration defaults;
4. looks up each module and validator spec without constructing it;
5. for a typed extractor, derives its artifact kind, requires the codec, and selects exact-type merger, normalizer, and validator variants;
6. checks required and provided capabilities in workflow order;
7. resolves external and generated target-aware reference bindings and validates producer order, consumer slot declarations, and artifact-kind compatibility;
8. validates each selected module and validator option set through its registry entry; and
9. calculates a digest over the resolved structure, including step order, step IDs, lane membership, generated topology, producer and consumer identities, typed artifact kind and schema identity, and the effective validator policy in its resolved execution order.

Resolution returns a ResolvedPipeline containing ordered steps, lanes, concrete bindings, validator chains, reference targets, and the digest. It does not read external reference bytes or construct runtime modules. CLI lane and reference selector syntax is defined in the CLI reference.

The digest includes each resolved step's ID and lane membership, generated producer/consumer topology, and each validator chain's stage, lane, owning module, ordered validator bindings, execution classes, targets, and artifact kinds. Changing step order, a dependency, a default chain, or an explicit override therefore changes pipeline identity whenever it changes effective execution policy.

Reference Materialization

The CLI calls MaterializeReferences after resolution and before constructing the LLM client or running the pipeline. For external bindings, the materializer checks each binding against its resolved target declaration, reads and validates the file, and builds both a contracts.ReferenceSet and provenance-only metadata on the corresponding ResolvedReferenceTarget. A structured generated binding is declaration-only at this point: its producer bytes do not exist until the producer lane reaches an accepted normalized result.

Preparation delivers the materialized external set for each target through pipeline.BuildRequest: chunkers and chunk validators receive the chunk target; extractors and extract validators receive the lane extract target; mergers and merge validators receive the lane merge target; and normalizers and normalize validators receive the lane normalize target. Input and output builders receive an empty set because those stages cannot declare references. Every builder gets an isolated deep clone of its target set, so construction-time mutation cannot change another builder, the resolved pipeline, or later runtime requests.

Prepared consumers do not need to be reconstructed when generated content is available. At the step boundary, the runner encodes the accepted producer value through its registered canonical codec, validates the generated bytes against each target slot's kind, schema, media type, and size, and clones one immutable reference item into the operation request. The item includes canonical digest, size, and bounded producer provenance but no filesystem URI. A handoff failure is a framework dependency error and prevents every consumer in that step from starting.

The runner continues to clone the resulting set into the chunk, extract, merge, or normalize request that owns the target. LLM-backed extensions may convert those items into named prompt inputs. Reference content remains separate from source evidence and source digests, whether the item came from a file or a generated handoff.

Binding precedence, path resolution, accepted content, and media-type behavior are configuration contracts; see Configuration. Durable provenance is defined in the JSON output contract, while runtime sensitive-data handling belongs in Operations.

Registries And Specs

pipeline.Registries holds option validators and run-local builders used during resolution and preparation. pipeline.ModuleCatalog exposes their specs during configuration validation and resolution. Separate registries exist for every stage and for validators; ValidatorChainRegistry stores production default-chain mappings. Both containers also carry an ArtifactCodecRegistry. Generic registration records one codec per stable artifact kind, validates its schema metadata and JSON Schema, retains the exact schema digest and Go type, and safely encodes or decodes framework-erased values with typed errors on incompatibility.

Typed extractor entries are keyed by module key and declare one artifact kind. Merger, normalizer, and typed-validator variants are keyed by module or validator key plus artifact kind. Chunk and serialized validators occupy separate target namespaces; serialized registrations declare whether they support chunks, artifacts, or both. Duplicate variants and exact Go-type mismatches are rejected deterministically.

Lane-sensitive merger and normalizer spec discovery always supplies the extractor's artifact kind, so variants under one reusable key may declare different capabilities and reference slots. Kind-neutral registry inspection selects the first registered artifact kind in sorted order.

Production composition registers the D&D spell-list codec and typed extractor, matching typed merge, normalize, and semantic-validator variants, and serialized JSON validators. Every artifact lane resolves through the typed registries and a matching codec.

A ModuleSpec declares its stage plus required and provided capabilities. Chunk, extract, merge, and normalize specs may also declare reference slots. Registry implementations defensively copy spec metadata, reject duplicate keys, and verify that a constructed implementation reports the registered key. Builder registrations accept ModuleDependencies and cloned configuration options through one BuildRequest. Builders decode those options and retain typed values or injected dependencies in the constructed implementation. Extractors declare their artifact kind, and merger, normalizer, and validator resolution selects the matching typed variant.

A ValidatorSpec declares a validator key and execution class. Resolution uses the execution class to reject incompatible profile bindings before execution. The current production catalog and default chain are listed only in Configuration.

Preparation And Runner Boundary

pipeline.Prepare receives a resolved pipeline, the registries, and shared module dependencies. It constructs input; chunk and its validators; every step's lane extract, merge, and normalize modules and validator chains in resolved order; then output. It stops at the first error with pipeline, step, stage, lane, module, and validator context as applicable. It never invokes an operation method. Generated references are not available during preparation; the operation request is the handoff boundary.

PreparedPipeline keeps private constructed executors and exposes cloned resolved input, chunk, lane, and output identities. Prepared components may implement pipeline.CheckpointFingerprintProvider to contribute explicit semantic identities to checkpoint reuse. Preparation trims and validates each non-secret name and value, prefixes it with the component's stage, lane, module, and validator scope, rejects duplicates, and retains the resulting sorted collection behind a defensive-copy accessor. Fingerprints must be stable and must not contain source content, credentials, local paths, timestamps, or other invocation-specific values.

pipeline.RunInput carries that prepared pipeline, raw source input, run identity and timing, optional session and profile metadata, a chunk-plan store and mode, a checkpoint execution policy, and checkpoint/debug collaborators. The runner parses source bytes through the already constructed input adapter. Later stage requests receive the generic source model; extract requests receive chunk-scoped input material, while chunk, merge, and normalize requests retain access to the original source material. Input, chunk, and output operation requests do not carry raw module options. The chunk request also does not carry an LLM client; an LLM-backed chunker receives the shared client during preparation. Their operation requests retain run-specific source, reference, profile, session, metadata, and step-handoff context as applicable. A generated reference is cloned into each compatible consumer request and is never exposed as a path.

Prepared lanes retain exact-type-checked erased operation closures. The runner uses those closures to keep each value typed through extraction, validation, merge, and normalization.

Source validation requires every unit to carry a canonical self-reference to its containing document and its own unit ID. Explicit clone, checkpoint, and debug boundaries retain that reference, and the canonical source digest covers it deterministically. Chunks use the same source model and carry one canonical reference spanning the first selected unit through the last.

pipeline.RunOutput carries the run manifest, accepted normalized serialized artifacts with lane and normalizer provenance, rejected results, warnings, checkpoint events, and logical files returned by the output encoder. The CLI owns debug-summary and durable filesystem writes after the runner returns.

Execution Flow

The pipeline-wide coordinator owns the ordered step loop, generated-reference sets at each barrier, and deterministic merging of step outcomes. For one step, one run-local lane engine owns worker lifecycle, cancellation, dispatch, continuation queues, and result collection. It initializes checkpoint state in lane order, dispatches bounded extract work, advances terminal lanes through serial merge and normalize work, selects failures by stable pipeline scope, and merges lane-local outcomes back in resolved order. Completion timing never becomes public ordering.

The runner:

1. validates its prepared input;
2. parses the raw input with the prepared adapter and validates the generic source document;
3. selects a stored plan or executes the configured chunker's Plan operation;
4. canonicalizes and materializes the plan, then validates the resulting chunks;
5. builds the framework-owned accepted chunk map from the accepted source, logical plan, and exact materialized chunks, then supplies it to the output request independently of output-module options;
6. executes each resolved step in configuration order. For one step, it dispatches extract jobs in source-chunk then resolved-lane order, starts a bounded lane continuation when all extracts for that lane are terminal, and waits for every lane to become terminal;
7. encodes and validates each accepted normalized producer artifact, then builds the immutable generated reference sets for the next step;
8. invokes the prepared output encoder only after every step succeeds and validates its logical file results;
9. returns the assembled manifest, outcomes, warnings, and files.

Within each artifact lane, it reuses the prepared extractor, merger, normalizer, and validators while performing these transitions:

1. extract once per accepted chunk and add runner-owned lane, source, and chunk provenance;
2. validate each extract result and omit rejected results from merge input;
3. skip the rest of the lane when no extract result is accepted;
4. merge accepted extract results in their existing order;
5. validate the merge result and skip normalization on rejection;
6. normalize the accepted merge result;
7. validate and append the accepted normalized result.

At a step barrier, a lane with no accepted normalized output is still a regular rejection unless a later generated binding names that lane as a required producer. In that case the runner raises a deterministic dependency error and does not start the consumer step. One accepted typed artifact may fan out to multiple compatible target slots. Consumers in the same step may run concurrently after the handoff; no work crosses the barrier early.

Module-provided warnings and payload warnings are promoted only from attempts whose results are accepted and used.

Chunk Plans And Reuse

Chunker.Plan returns a source.ChunkPlan: the canonical source digest, ordered unit-ID ranges, and optional plan or range annotations. The framework owns plan canonicalization and materialization. It creates the generic chunks and therefore owns their IDs, indexes, source references, JSON content, units, media type, and generic metadata. Plan and range annotations are independently owned raw JSON and become Chunk.PlanAnnotations and Chunk.Annotations.

In auto, the runner looks up the source digest before invoking the chunker. A valid hit is materialized and sent through the current run's configured chunk validators; it does not invoke the chunk module, consume its retry budget, or make a chunk-stage LLM call. A missing, invalid, or unmaterializable record generates a candidate. refresh generates without lookup; bypass generates without cache access. Generated plans are published only after the full chunk validator chain approves them. A validator rejection is a regular rejected pipeline outcome and never replaces a cached plan.

The store is source-addressed, not pipeline-addressed. Changes to pipeline configuration, requested chunker, options, references, lanes, validators, or LLM profile do not prevent a source-digest hit. The manifest records both the currently requested chunker and the effective plan producer. Cache state and paths are configured and operated outside the runner; see Configuration and Operations.

For an accepted plan, the runner also constructs the strict framework-owned Accepted Chunk Map before lane execution. It uses the current resolved chunker as requested_chunker and the stored or generated record as producer, preserving that distinction on reuse. Chunk rejection supplies no map; later lane rejection does not discard it. Output encoders receive a defensively owned serialized value and may explicitly ignore it.

The extract job channel has the same capacity as the effective extract worker count, so dispatch applies backpressure. A fixed continuation executor prevents ready or checkpoint-reused lanes from creating one goroutine each. Workers and continuations publish lane-local results; the coordinator is the only writer of aggregate output and merges those results in resolved lane and source-chunk order.

Plan Canonicalization And Chunk Materialization

Plan canonicalization requires canonical JSON annotations, a matching source digest, at least one range, existing ordered boundaries, and increasing range starts. Ranges may overlap or leave gaps; a chunker may impose stricter policy. Materialization deterministically reconstructs each range from the current source document, deep-clones JSON-shaped source-unit metadata, and copies annotations without interpreting their namespaces. Materialized chunks and separate materializations do not share mutable unit metadata; unsupported or cyclic metadata fails materialization with context.

Before lane execution, generic chunk validation checks the materialized chunks' identities, order, source references, content, media type, units, and metadata. No chunk checkpoint participates in plan selection: plan storage is the only chunk-reuse mechanism. Extract, merge, and normalize checkpoints continue to use materialized chunk digests as their dependencies.

Validation And Retries

Chunk, extract, merge, and normalize results pass through the resolved validator chain for their stage and module. Chunk validators receive canonical chunks; typed validators receive the domain value; and serialized validators receive canonical chunk JSON or artifact codec bytes. Validators execute in resolved order and stop at the first error or rejection. An empty chain approves the result.

Production D&D artifact chains keep generic JSON syntax validation first, then run every rejecting domain validator before generic JSON Schema validation. The domain validator therefore owns expected semantic diagnostics; the generic schema validator remains the final rejecting representation backstop, before warning-only relatedness validation. Explicitly configured validator chains retain their configured order.

runWithRetry applies the effective retry policy around module execution and its complete validation chain. A module or validator error becomes a framework error when attempts are exhausted. A rejection becomes a recorded RejectedOutput when attempts are exhausted. Cancellation stops retry processing immediately.

Rejected output is a non-fatal pipeline outcome and does not advance. Warnings from discarded attempts are not promoted. Configuration owns retry counts and validator overrides; see Module Bindings.

Checkpoint And Debug Hooks

The runner depends on recorder and loader interfaces, using no-op implementations when collaborators are absent. Each checkpointed workflow boundary records a running, succeeded, or failed transition. Reuse decisions are consulted in workflow order and accepted payloads are cloned before entering the normal handoff path. Typed extract, merge, and normalize checkpoints store codec bytes with artifact kind, schema ID, name, version and exact digest, and media type. Reuse compares that identity with the prepared codec and decodes through the codec; missing identity, mismatches, corrupt bytes, and decode failures become explicit reuse misses and execute the lane normally. Dependency fingerprints and debug content digests use the same stable codec bytes that cross those boundaries.

That progressive extract, merge, and normalize reuse is the ordinary resume path. A lane marked as a required predecessor for selective recomputation takes a separate accepted-output path before extract scheduling. The loader reads the existing successful normalize manifest and payload by step, lane, and normalizer, without consulting extract or merge dependencies. It requires the current non-empty checkpoint identity to match, so the invocation identity still binds the input, resolved topology and configuration, references, runtime overrides, profiles, and component fingerprints.

The runner decodes and canonically re-encodes each reusable artifact once with the prepared codec, requiring exact kind, schema identity and digest, media type, canonical bytes, content digest, and producer provenance. A valid accepted producer becomes a runner-owned cloned normalized output, restores only normalize-checkpoint warnings, and records one accepted_artifact_reused normalize decision. It does not invoke or record extract, merge, normalize, or their validators. Invalid or unavailable accepted state records its decision and fails the producer step; the dependent step never starts and the producer is not implicitly rerun. If a later required lane fails during initialization, already hydrated terminal lanes remain in the failed output in resolved order.

Generated references add downstream dependencies containing the producer's artifact kind, complete schema identity, media type, canonical content digest, and size. Compatible accepted producer outputs may therefore feed a later step without re-executing the producer. Forced lanes bypass accepted-output hydration and execute normally. A missing, rejected, corrupt, incompatible, or changed producer blocks its dependent while leaving independent work eligible for reuse. The runner records bounded decision categories: reused, executed, forced_recompute, and dependency_invalidated. Operator meanings for the stable reason codes belong to Operations.

The CLI includes prepared-component fingerprints in the run-wide checkpoint identity alongside resolved configuration, raw input, reference provenance, runtime overrides, and LLM-profile fingerprints. Module metadata is not used implicitly for cache identity: components opt in only with stable semantic values that can change accepted output. Adding or changing a component fingerprint intentionally produces a cold cache miss. Existing checkpoint schemas and paths remain unchanged.

The CLI's --recompute-step policy forces the selected step and all transitive dependents, but requires accepted normalized artifacts for every unselected producer on which that closure depends. It changes execution policy only; it does not alter persistent checkpoint identity.

Debug instrumentation wraps run, stage, attempt, validator, and structured LLM boundaries. Every executed chunk, extract, merge, and normalize attempt writes one terminal envelope for acceptance, validator rejection, module or validator error, or applicable candidate or final serialization error. The envelope contains its attempt-local warnings, any available candidate and rejection, and terminal error text; failures before a candidate exists omit that payload. Only LLM calls made by the module operation belong to the module attempt. Validator calls retain independent scopes under validate/ and are not duplicated into the module envelope. A failed terminal-envelope write is a non-retryable framework error and is joined with any primary attempt error. Debug data is never used as a checkpoint source. Typed artifact debug envelopes are domain-neutral, redact sensitive metadata and bytes through the common debug policy, and record codec identity plus schema and content digests.

Merge and normalize attempts serialize their in-memory candidate with the codec's required candidate encoder before typed validation. Serialized validators and attempt debug use that candidate representation, which carries the codec media type and schema identity but is never checkpointed or passed downstream. Only a validator-approved value is encoded through the strict final codec and made eligible for a checkpoint or stage output.

Checkpoint identity, physical layout, reuse behavior, and debug artifact handling are operator contracts in Operations. Serialization and recorder implementation are inventoried in Internal Overview.

Results And Failures

The runner owns manifest assembly and handoff summaries but not the durable JSON schema. It records resolved module and lane provenance, validator chains, source/reference identities, selected LLM profiles, normalized and rejected summaries, status, and timing. Serialized artifact content remains outside the manifest. Module metadata providers may add non-secret singleton or lane-scoped metadata.

Execution errors include stage, module, lane, or validator context. Once a manifest exists, a failing run returns it with failed status and completion time. Successful status reflects whether any result was rejected. The durable manifest and logical file schemas are defined in the JSON output contract.

On a framework failure, the runner cancels its derived context, stops submitting new extract work, drains started tasks, and skips the output encoder. Parent cancellation takes precedence. Otherwise context-cancellation fallout is discarded when a substantive error exists, and the primary error is selected by stage, resolved lane, and source chunk rather than completion time.

Tests To Inspect

• internal/core/config/effective_config_test.go: config-to-resolution boundary.
• internal/framework/pipeline/profile_test.go: selection, defaults, capabilities, validator chains, and digest behavior.
• internal/framework/pipeline/artifact_codec_registry_test.go: typed codec metadata, registration, erasure safety, strict decoding, and cloning.
• internal/framework/pipeline/typed_resolution_test.go: heterogeneous typed lane resolution and preparation, target-specific validators, incompatibilities, ordering, and schema-sensitive pipeline identity.
• internal/framework/pipeline/runner_concurrency_test.go: bounded dispatch and continuations, reverse completion, stable errors, rejection, cancellation, retries, and independent provider-call limits.
• internal/framework/pipeline/preparation_test.go: option validation, construction order, dependency failures, and the before-source-work boundary.
• internal/framework/pipeline/references_test.go: target resolution and materialization.
• internal/cli/run_contract_test.go: production run transitions, retries, rejections, warnings, CLI recomputation controls, debug hooks, and manifests.
• internal/cli/recompute_execution_contract_test.go: filesystem-backed selective recomputation and accepted-producer recovery.
• internal/cli/production_contract_test.go: production composition and configuration-resolution smoke coverage.
• internal/cli/example_contract_test.go: maintained example resolution and execution ownership.
• internal/modules/integration/*_test.go and internal/modules/seriatim/input/transcript/runner_test.go: typed runner composition across concrete module families.
• internal/framework/checkpoint/*_test.go: checkpoint serialization and reuse collaborators.