121 lines
3.6 KiB
Markdown
121 lines
3.6 KiB
Markdown
# Built-In Modules
|
|
|
|
## Purpose
|
|
|
|
Describes implemented built-in module behavior and boundaries in
|
|
`internal/builtin`.
|
|
|
|
## Implemented module set
|
|
|
|
Input reader:
|
|
|
|
- `json-files`
|
|
|
|
Preprocessing:
|
|
|
|
- `validate-raw`
|
|
- `normalize-speakers`
|
|
- `trim-text`
|
|
|
|
Merger:
|
|
|
|
- `chronological-merge`
|
|
|
|
Postprocessing:
|
|
|
|
- `detect-overlaps`
|
|
- `resolve-overlaps`
|
|
- `backchannel`
|
|
- `filler`
|
|
- `resolve-danglers`
|
|
- `coalesce`
|
|
- `autocorrect`
|
|
- `assign-ids`
|
|
- `validate-output`
|
|
|
|
Output writer:
|
|
|
|
- `json`
|
|
|
|
## Inputs, outputs, and side effects
|
|
|
|
- `json-files`: reads JSON files from `cfg.InputFiles`, parses supported
|
|
segment/word fields, emits warnings for untimed words.
|
|
- `validate-raw`: validates raw source/timing invariants.
|
|
- `normalize-speakers`: converts raw transcripts to canonical segments,
|
|
optionally resolving speakers from `cfg.SpeakersFile`.
|
|
- `trim-text`: trims canonical segment text whitespace.
|
|
- `chronological-merge`: flattens canonical segments and applies deterministic
|
|
sort (`model.SegmentLess`).
|
|
- `detect-overlaps`: annotates overlap groups.
|
|
- `resolve-overlaps`: rewrites overlap groups using timed words and thresholds.
|
|
- `backchannel`/`filler`: classify short utterances using duration thresholds.
|
|
- `resolve-danglers`: merges dangling derived fragments.
|
|
- `coalesce`: merges adjacent same-speaker segments within configured gap.
|
|
- `autocorrect`: applies YAML replacement rules when configured.
|
|
- `assign-ids`: assigns final sequential IDs.
|
|
- `validate-output`: validates selected public artifact shape.
|
|
- `json`: writes artifact JSON to `cfg.OutputFile` through shared deterministic
|
|
JSON file writing.
|
|
|
|
Filesystem side effects are limited to:
|
|
|
|
- reading configured input/YAML files
|
|
- writing configured output artifact
|
|
|
|
## Config fields used
|
|
|
|
Primary module inputs from `config.Config`:
|
|
|
|
- file paths: `InputFiles`, `SpeakersFile`, `AutocorrectFile`, `OutputFile`
|
|
- schema/modules: `OutputSchema`, `OutputModules`
|
|
- overlap/coalesce thresholds: `OverlapWordRunGap`,
|
|
`WordRunReorderWindow`, `CoalesceGap`
|
|
- category thresholds: `BackchannelMaxDuration`, `FillerMaxDuration`
|
|
|
|
## Ordering constraints
|
|
|
|
- Preprocessing must satisfy state contracts from `raw` to `canonical`.
|
|
- Order-sensitive transforms should run before `assign-ids`.
|
|
- `validate-output` should run after final ID assignment and output-shape
|
|
mutations.
|
|
- Default configuration includes a second `detect-overlaps` pass after
|
|
transformations.
|
|
|
|
## Boundaries
|
|
|
|
- Modules implement behavior; CLI/config parsing remains outside modules.
|
|
- Modules communicate through explicit model contracts and report events.
|
|
- Output modules operate on final artifacts and do not re-run transform logic.
|
|
|
|
## Failure behavior
|
|
|
|
Representative failures:
|
|
|
|
- invalid input JSON shape or typed field errors (`json-files`)
|
|
- invalid YAML or unmatched speaker map entries
|
|
- unknown module names during registry resolution
|
|
- invalid ordering/state transitions in preprocessing chain
|
|
- validation failure in `validate-output`
|
|
- output write failure in `json` writer
|
|
|
|
## Tests to inspect before changes
|
|
|
|
- `internal/builtin/preprocess_test.go`
|
|
- `internal/builtin/postprocess_test.go`
|
|
- `internal/overlap/resolve_test.go`
|
|
- `internal/overlap/detect_test.go`
|
|
- `internal/coalesce/coalesce_test.go`
|
|
- `internal/danglers/danglers_test.go`
|
|
- `internal/backchannel/backchannel_test.go`
|
|
- `internal/filler/filler_test.go`
|
|
- `internal/autocorrect/autocorrect_test.go`
|
|
- `internal/cli/merge_test.go`
|
|
|
|
## Invariants
|
|
|
|
- Modules are selected by canonical name through the registry.
|
|
- Execution is sequential and deterministic for a fixed configuration.
|
|
- `assign-ids` defines final public segment IDs.
|
|
- `validate-output` enforces public artifact contracts through `schema`.
|