seriatim/docs/internal/pipeline.md

Pipeline Internals

Purpose

Describes implemented merge pipeline orchestration in internal/pipeline.

Inputs and outputs

Input:

• config.Config
• registry-resolved modules from internal/builtin

Output:

• selected public artifact written by output writer modules
• optional report JSON when cfg.ReportFile is set

Stage contracts

The runner executes these contracts in order:

1. InputReader: external inputs -> []model.RawTranscript
2. Preprocessor: PreprocessState transformations (raw -> canonical)
3. Merger: canonical transcripts -> model.MergedTranscript
4. Postprocessor: merged transcript transformations
5. OutputWriter: serialized artifact writes

PreprocessState must end in StateCanonical before merge.

Registry resolution

resolvePlan maps configured names to modules:

• input reader: cfg.InputReader
• preprocessors: cfg.PreprocessingModules
• postprocessors: cfg.PostprocessingModules
• output writers: cfg.OutputModules
• merger: single registered merger

Unknown names fail fast with contextual errors.

Execution order and reporting

• Modules run sequentially in configured order.
• Events returned by modules are appended in execution order.
• Report metadata includes input reader, input files, and module lists.
• Output writer events are appended before optional report write.

Config fields used

Runner-level fields:

• InputReader
• InputFiles
• PreprocessingModules
• PostprocessingModules
• OutputModules
• OutputSchema (via artifact.SelectedFromMerged)
• ReportFile

Module-specific settings are consumed inside builtin modules (for example coalesce gap and overlap thresholds).

Adapters used

• Input adapters: registered InputReader implementations (default json-files).
• Output adapters: registered OutputWriter implementations (default json).
• Report adapter: report.WriteJSON when cfg.ReportFile is provided.

Boundaries

• Pipeline does not parse CLI flags.
• Pipeline does not normalize raw CLI strings.
• Pipeline delegates conversion to public output contracts to internal/artifact.
• Artifact-level commands trim and normalize are outside this pipeline.

Failure behavior

Pipeline returns errors from:

• registry resolution (unknown modules, missing merger)
• invalid preprocessing state transitions
• module read/process/merge/write failures
• optional report write failure

No retry/resume state is stored.

Tests to inspect before changes

• internal/pipeline/runner_test.go
• internal/builtin/preprocess_test.go
• internal/builtin/postprocess_test.go
• internal/cli/merge_test.go

Invariants

• Sequential deterministic execution order.
• Preprocessing state must type-check from raw to canonical.
• Module selection is explicit by canonical names.
• Report event order reflects actual execution order.
• Output artifact selection is schema-driven via internal/artifact.