103 lines
2.9 KiB
Markdown
103 lines
2.9 KiB
Markdown
# Pipeline Internals
|
|
|
|
## Purpose
|
|
|
|
Describes implemented merge pipeline orchestration in `internal/pipeline`.
|
|
|
|
## Inputs and outputs
|
|
|
|
Input:
|
|
|
|
- `config.Config`
|
|
- registry-resolved modules from `internal/builtin`
|
|
|
|
Output:
|
|
|
|
- selected public artifact written by output writer modules
|
|
- optional report JSON when `cfg.ReportFile` is set
|
|
|
|
## Stage contracts
|
|
|
|
The runner executes these contracts in order:
|
|
|
|
1. `InputReader`: external inputs -> `[]model.RawTranscript`
|
|
2. `Preprocessor`: `PreprocessState` transformations (`raw` -> `canonical`)
|
|
3. `Merger`: canonical transcripts -> `model.MergedTranscript`
|
|
4. `Postprocessor`: merged transcript transformations
|
|
5. `OutputWriter`: serialized artifact writes
|
|
|
|
`PreprocessState` must end in `StateCanonical` before merge.
|
|
|
|
## Registry resolution
|
|
|
|
`resolvePlan` maps configured names to modules:
|
|
|
|
- input reader: `cfg.InputReader`
|
|
- preprocessors: `cfg.PreprocessingModules`
|
|
- postprocessors: `cfg.PostprocessingModules`
|
|
- output writers: `cfg.OutputModules`
|
|
- merger: single registered merger
|
|
|
|
Unknown names fail fast with contextual errors.
|
|
|
|
## Execution order and reporting
|
|
|
|
- Modules run sequentially in configured order.
|
|
- Events returned by modules are appended in execution order.
|
|
- Report metadata includes input reader, input files, and module lists.
|
|
- Output writer events are appended before optional report write.
|
|
|
|
## Config fields used
|
|
|
|
Runner-level fields:
|
|
|
|
- `InputReader`
|
|
- `InputFiles`
|
|
- `PreprocessingModules`
|
|
- `PostprocessingModules`
|
|
- `OutputModules`
|
|
- `OutputSchema` (via `artifact.SelectedFromMerged`)
|
|
- `ReportFile`
|
|
|
|
Module-specific settings are consumed inside builtin modules (for example
|
|
coalesce gap and overlap thresholds).
|
|
|
|
## Adapters used
|
|
|
|
- Input adapters: registered `InputReader` implementations (default `json-files`).
|
|
- Output adapters: registered `OutputWriter` implementations (default `json`).
|
|
- Report adapter: `report.WriteJSON` when `cfg.ReportFile` is provided.
|
|
|
|
## Boundaries
|
|
|
|
- Pipeline does not parse CLI flags.
|
|
- Pipeline does not normalize raw CLI strings.
|
|
- Pipeline delegates conversion to public output contracts to `internal/artifact`.
|
|
- Artifact-level commands `trim` and `normalize` are outside this pipeline.
|
|
|
|
## Failure behavior
|
|
|
|
Pipeline returns errors from:
|
|
|
|
- registry resolution (unknown modules, missing merger)
|
|
- invalid preprocessing state transitions
|
|
- module read/process/merge/write failures
|
|
- optional report write failure
|
|
|
|
No retry/resume state is stored.
|
|
|
|
## Tests to inspect before changes
|
|
|
|
- `internal/pipeline/runner_test.go`
|
|
- `internal/builtin/preprocess_test.go`
|
|
- `internal/builtin/postprocess_test.go`
|
|
- `internal/cli/merge_test.go`
|
|
|
|
## Invariants
|
|
|
|
- Sequential deterministic execution order.
|
|
- Preprocessing state must type-check from `raw` to `canonical`.
|
|
- Module selection is explicit by canonical names.
|
|
- Report event order reflects actual execution order.
|
|
- Output artifact selection is schema-driven via `internal/artifact`.
|