diff --git a/README.md b/README.md index 7b246b0..bf839ed 100644 --- a/README.md +++ b/README.md @@ -1,513 +1,34 @@ # seriatim -`seriatim` merges per-speaker WhisperX-style JSON transcripts into a single JSON transcript that preserves speaker identity and chronological order. It also trims existing seriatim output artifacts by segment ID and normalizes external transcript-like JSON into standard seriatim output schemas. +`seriatim` is a Go CLI for transcript artifact processing. -The current implementation supports the `merge`, `trim`, and `normalize` commands. `merge` reads one or more input JSON files, optionally maps each input file to a canonical speaker using `speakers.yml`, sorts all segments by timestamp, detects and resolves overlaps when word-level timing is available, assigns consecutive numeric `id` values, and writes a merged JSON artifact. `trim` reads an existing seriatim output artifact and projects it to a retained segment subset. `normalize` reads transcript-like JSON input, validates required segment fields, sorts deterministically, assigns fresh IDs, and emits a selected seriatim output schema. +It merges per-speaker WhisperX-style JSON into one deterministic transcript, trims existing seriatim artifacts by segment ID, and normalizes transcript-like JSON into standard seriatim output schemas. -## Usage +## Quickstart -Run from source: +Shortest useful merge command: ```sh go run ./cmd/seriatim merge \ - --input-file samples/raw/2026-04-19-Eric_Rakestraw.json \ - --input-file samples/raw/2026-04-19-Mike_Brown.json \ + --input-file speaker-a.json \ + --input-file speaker-b.json \ --output-file merged.json ``` -Optional report output: - -```sh -go run ./cmd/seriatim merge \ - --input-file eric.json \ - --input-file mike.json \ - --output-file merged.json \ - --report-file report.json -``` - -Trim an existing seriatim artifact: - -```sh -go run ./cmd/seriatim trim \ - --input-file merged.json \ - --output-file trimmed.json \ - --keep "1-10, 15, 20-25" -``` - -Normalize external transcript-style JSON: - -```sh -go run ./cmd/seriatim normalize \ - --input-file transcript.json \ - --output-file normalized.json -``` - -Normalize an Audita-style bare segment array to full schema with report output: - -```sh -go run ./cmd/seriatim normalize \ - --input-file audita-segments.json \ - --output-file normalized-full.json \ - --output-schema seriatim-full \ - --report-file normalize-report.json -``` - -## CLI - -```text -seriatim merge [flags] -seriatim trim [flags] -seriatim normalize [flags] -``` - -Global flags: - -| Flag | Description | -| --- | --- | -| `--help` | Show command help. | -| `--version` | Show application version. Local builds default to `dev`; release builds inject the release version. | - -`merge` flags: - -| Flag | Required | Default | Description | -| --- | --- | --- | --- | -| `--input-file` | Yes | none | Input transcript JSON file. Repeat once per speaker/input file. | -| `--output-file` | Yes | none | Merged transcript JSON output path. | -| `--report-file` | No | none | Optional report JSON output path. | -| `--speakers` | No | none | Speaker map YAML file. When omitted, input file basenames are used as speaker labels. | -| `--autocorrect` | No | none | Autocorrect rules YAML file. When omitted, the default `autocorrect` module leaves text unchanged. | -| `--input-reader` | No | `json-files` | Input reader module. | -| `--output-modules` | No | `json` | Comma-separated output modules. | -| `--output-schema` | No | `seriatim-intermediate` | JSON output contract. Allowed values are `seriatim-minimal`, `seriatim-intermediate`, and `seriatim-full`. If omitted, the runtime default is used; consumers that depend on a specific shape should set this explicitly. | -| `--preprocessing-modules` | No | `validate-raw,normalize-speakers,trim-text` | Comma-separated preprocessing modules, evaluated in order. | -| `--postprocessing-modules` | No | `detect-overlaps,resolve-overlaps,backchannel,filler,resolve-danglers,coalesce,detect-overlaps,autocorrect,assign-ids,validate-output` | Comma-separated postprocessing modules, evaluated in order. | -| `--coalesce-gap` | No | `3.0` | Maximum same-speaker gap in seconds for `coalesce`; also used as the `resolve-overlaps` context window. Must be a non-negative float. | - -`trim` flags: - -| Flag | Required | Default | Description | -| --- | --- | --- | --- | -| `--input-file` | Yes | none | Input seriatim output artifact JSON file. | -| `--output-file` | Yes | none | Trimmed transcript JSON output path. | -| `--keep` | Exactly one of `--keep` or `--remove` is required | none | Segment ID selector to retain. | -| `--remove` | Exactly one of `--keep` or `--remove` is required | none | Segment ID selector to drop. | -| `--output-schema` | No | preserve input artifact schema | Optional output schema override: `seriatim-minimal`, `seriatim-intermediate`, or `seriatim-full`. | -| `--report-file` | No | none | Optional report JSON output path. | -| `--allow-empty` | No | `false` | Allow trimming to zero retained segments. | - -`trim` selection rules: - -- `--keep` and `--remove` are mutually exclusive. -- Exactly one of `--keep` or `--remove` is required. -- Selection is by segment ID only. -- Invalid selected segment IDs fail the command by default. - -`trim` selector syntax: - -- Segment IDs are positive 1-based integers. -- Inclusive ranges are supported: `1-10`. -- Comma-separated selectors are supported: `1-10,15,20-25`. -- Whitespace around numbers, commas, and hyphens is allowed: `1 - 10, 15, 20 - 25`. -- Duplicate and overlapping ranges are accepted and normalized as a union. -- Descending ranges (for example `10-1`) are rejected. - -`trim` behavior: - -- `trim` consumes existing seriatim JSON output artifacts only. -- `trim` does not accept raw WhisperX transcript JSON as input. -- Retained output segment IDs are renumbered sequentially from `1` to `N`. -- Transcript order is preserved from input transcript order; selector order does not reorder output. -- When output schema is `seriatim-full`, overlap groups are recomputed from retained segments. -- `--output-schema seriatim-full` is supported when trim has full-schema artifact data to emit; trim does not synthesize missing full-schema provenance from minimal/intermediate input artifacts. -- `trim` does not run merge postprocessors such as `resolve-overlaps`, `coalesce`, or `autocorrect`. - -`trim` report output: - -- When `--report-file` is provided, the report includes standard trim/validation/output events. -- The report includes a `trim-audit` event containing trim operation metadata, including selected IDs, retained/removed counts, removed IDs, and old-to-new segment ID mapping. -- Old-to-new ID mapping is emitted as a deterministic ordered array of `{old_id, new_id}` pairs. - -`normalize` flags: - -| Flag | Required | Default | Description | -| --- | --- | --- | --- | -| `--input-file` | Yes | none | Input transcript JSON file. | -| `--output-file` | Yes | none | Normalized transcript JSON output path. | -| `--output-schema` | No | `seriatim-intermediate` (resolved via `SERIATIM_OUTPUT_SCHEMA` when set) | Output JSON schema: `seriatim-minimal`, `seriatim-intermediate`, or `seriatim-full`. | -| `--output-modules` | No | `json` | Comma-separated output modules. Current normalize support is `json` only. | -| `--report-file` | No | none | Optional report JSON output path. | - -`normalize` input shapes: - -- Top-level object with a `segments` array. -- Bare top-level array of segment objects (for example, Audita-style output). - -`normalize` behavior: - -- Repairs missing timing fields deterministically: - if one of `start`/`end` is present, sets both to that value; - if both are missing, uses midpoint of previous `end` and next `start`, - with edge fallback to available neighbor and `0.0` for single-segment inputs. -- If `end < start`, swaps them. -- Fills missing/empty `speaker` with `Unknown_Speaker`. -- Drops segments with missing, empty, or whitespace-only `text`. -- Validates repaired timing with `start >= 0`. -- Accepts existing input `id` values as provenance only. -- Reassigns output segment IDs sequentially from `1` to `N`. -- Sorts deterministically by `(start, end, original_input_index, speaker)`. -- Uses original input order only as a tie-breaker. -- Does not run merge postprocessors such as overlap detection, overlap resolution, coalescing, or autocorrect. -- Useful for converting external transcript outputs into standard seriatim artifacts. - -`normalize` report output: - -- When `--report-file` is provided, normalize emits deterministic report events with input shape detection, segment counts, schema/module selections, sorting/ID diagnostics, and output write/validation summaries. -- A machine-readable `normalize-audit` event is included for downstream tooling. - -Environment variables: - -| Environment Variable | Default | Description | -| --- | --- | --- | -| `SERIATIM_OUTPUT_SCHEMA` | `seriatim-intermediate` | Output schema used when `--output-schema` is not explicitly provided. Allowed values are `seriatim-minimal`, `seriatim-intermediate`, and `seriatim-full`. The CLI flag takes precedence. | -| `SERIATIM_OVERLAP_WORD_RUN_GAP` | `1.0` | Maximum gap in seconds between adjacent timed words when `resolve-overlaps` builds word-run replacement segments. Must be a positive float. | -| `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW` | `1.0` | Near-start window in seconds for ordering replacement word runs shortest-first. Must be a positive float. | -| `SERIATIM_BACKCHANNEL_MAX_DURATION` | `2.0` | Maximum duration in seconds for `backchannel` classification. Must be a positive float. | -| `SERIATIM_FILLER_MAX_DURATION` | `1.25` | Maximum duration in seconds for `filler` classification. Must be a positive float. | - -## Input JSON Format - -Each input file must be valid JSON with a top-level `segments` array. The current parser accepts the WhisperX segment subset needed for merging: - -```json -{ - "segments": [ - { - "start": 1.25, - "end": 3.5, - "text": "Hello there.", - "words": [ - {"word": "Hello", "start": 1.25, "end": 1.55, "score": 0.98}, - {"word": "there.", "start": 1.7, "end": 2.0} - ] - } - ] -} -``` - -Required segment fields: - -- `start`: number, must be `>= 0`. -- `end`: number, must be `>= start`. -- `text`: string. - -Optional word fields: - -- `words`: array of word timing objects. -- `words[].word`: string. -- `words[].start`: optional number, must be `>= 0` when present. -- `words[].end`: optional number, must be `>= start` when present with `start`. -- `words[].score`: optional number. -- `words[].speaker`: optional raw speaker label string. - -Word-level timing is preserved internally for overlap resolution. If a word is missing `start` or `end`, seriatim keeps the word text, emits a warning in the optional report, and does not use that word as a timing anchor. Word timing is not emitted in the final JSON artifact. - -## Speaker Map Format - -`speakers.yml` maps input files to canonical speaker names using ordered substring rules: - -This file is optional. If `--speakers` is omitted, `seriatim` uses each input file basename as the segment speaker label. - -```yaml -match: - - speaker: "Eric Rakestraw" - match: - - "Eric_Rakestraw" - - "Eric" - - - speaker: "Mike Brown" - match: - - "Mike_Brown" - - "mb" -``` - -For each `--input-file`, `seriatim` takes the file basename and evaluates the rules in order. The first rule with a matching substring wins, and no later rules are evaluated. - -For example, this input: - -```text -samples/raw/2026-04-19-Eric_Rakestraw.json -``` - -matches this rule because the basename contains `Eric_Rakestraw`: - -```yaml -- speaker: "Eric Rakestraw" - match: - - "Eric_Rakestraw" -``` - -Important details: - -- Matching is against the input file basename, not the full path. -- Matching is case-insensitive. -- Rules are evaluated from first to last. -- Each rule must have a non-empty `speaker`. -- Each rule must have at least one non-empty `match` string. -- Duplicate speaker names are invalid. -- Every input file must match at least one rule or the command fails. - -Deprecated old format: - -```yaml -inputs: - eric.json: - speaker: "Eric Rakestraw" -``` - -The old `inputs:` direct mapping format is no longer supported. - -## Output JSON Format - -`--output-modules json` controls the writer. `--output-schema` controls the JSON contract that writer serializes. - -The named schemas are stable public contracts. If a consumer depends on a specific shape, it should request that schema explicitly at runtime. The runtime default selection may change in a future release. - -The `seriatim-intermediate` schema is the current default selection when neither `--output-schema` nor `SERIATIM_OUTPUT_SCHEMA` is set. It stays close to the minimal schema, but adds optional `categories` on each segment: - -```json -{ - "metadata": { - "application": "seriatim", - "version": "dev", - "output_schema": "seriatim-intermediate" - }, - "segments": [ - { - "id": 1, - "start": 1.25, - "end": 3.5, - "speaker": "Eric Rakestraw", - "text": "Hello there.", - "categories": ["backchannel"] - } - ] -} -``` - -The `seriatim-full` schema uses the full seriatim envelope: - -```json -{ - "metadata": { - "application": "seriatim", - "version": "dev", - "input_reader": "json-files", - "input_files": ["eric.json", "mike.json"], - "preprocessing_modules": ["validate-raw", "normalize-speakers", "trim-text"], - "postprocessing_modules": ["detect-overlaps", "resolve-overlaps", "backchannel", "filler", "resolve-danglers", "coalesce", "detect-overlaps", "autocorrect", "assign-ids", "validate-output"], - "output_modules": ["json"] - }, - "segments": [ - { - "id": 1, - "source": "eric.json", - "source_segment_index": 0, - "speaker": "Eric Rakestraw", - "start": 1.25, - "end": 3.5, - "text": "Hello there.", - "overlap_group_id": 1 - }, - { - "id": 2, - "source": "eric.json", - "source_ref": "word-run:1:1:1", - "derived_from": ["eric.json#0"], - "speaker": "Eric Rakestraw", - "start": 2.0, - "end": 2.5, - "text": "Resolved word run", - "categories": ["backchannel"] - } - ], - "overlap_groups": [ - { - "id": 1, - "start": 1.25, - "end": 4.0, - "segments": ["eric.json#0", "mike.json#0"], - "speakers": ["Eric Rakestraw", "Mike Brown"], - "class": "unknown", - "resolution": "unresolved" - } - ] -} -``` - -The `seriatim-minimal` schema emits minimal metadata and compact ordered segments: - -```json -{ - "metadata": { - "application": "seriatim", - "version": "dev", - "output_schema": "seriatim-minimal" - }, - "segments": [ - { - "id": 1, - "start": 1.25, - "end": 3.5, - "speaker": "Eric Rakestraw", - "text": "Hello there." - } - ] -} -``` - -Minimal output intentionally omits categories, overlap groups, source/provenance fields, and pipeline configuration metadata. - -Intermediate output intentionally omits overlap groups and source/provenance fields, but keeps optional `categories` and minimal metadata. - -Segments are sorted deterministically by: - -```text -(start, end, source, source_segment_index/source_ref, speaker) -``` - -Final segment IDs are assigned after sorting and start at `1`. - -The public Go output contract is available from: - -```go -import "gitea.maximumdirect.net/eric/seriatim/schema" -``` - -The same package embeds machine-readable JSON Schemas in `schema/full-output.schema.json`, `schema/intermediate-output.schema.json`, and `schema/minimal-output.schema.json`. The default `validate-output` postprocessor validates the selected output shape and verifies final segment IDs are present, sequential, and start at `1`. - -## Overlap Detection - -The default postprocessing pipeline detects overlapping segment groups. - -Overlap behavior: - -- A strict timing overlap is required: `next.start < current_group_end`. -- Segments that only touch at a boundary are not grouped. -- Groups require at least two distinct speakers. -- Transitive overlaps are grouped together. -- Segments in detected groups receive `overlap_group_id`. -- `overlap_groups[].segments` contains stable references in `source#source_segment_index` format. -- `class` is currently `unknown`. -- `resolution` is `unresolved` until `resolve-overlaps` replaces the group. - -## Overlap Resolution - -The default postprocessing pipeline runs `detect-overlaps`, then `resolve-overlaps`, then `backchannel`, then `filler`, then `resolve-danglers`, then `coalesce`, then a second `detect-overlaps` pass. - -For each detected overlap group, `resolve-overlaps` uses preserved WhisperX word timing to build smaller word-run replacement segments: - -- The resolution window expands the detected overlap group by `--coalesce-gap` seconds on both sides. -- Nearby same-speaker context segments are included when they intersect the expanded window and their start or end is within `--coalesce-gap` of the original overlap boundary. -- Once a segment is selected for replacement, all timed words from that segment participate in word-run construction; the window controls segment selection, not per-word clipping. -- Context segments that are part of another detected overlap group are not pulled into the current group. -- Untimed words are included in replacement text in original word order when nearby timed words create a replacement run. -- Untimed words do not affect replacement segment start/end times or word-run gap splitting. -- Words for the same speaker are merged into one run when the gap between adjacent words is no greater than `SERIATIM_OVERLAP_WORD_RUN_GAP`. -- The default word-run gap is `1.0` seconds. -- Set `SERIATIM_OVERLAP_WORD_RUN_GAP` to a positive number of seconds to override the default. -- Near-start replacement word runs are reordered so shorter segments come first when adjacent starts are within `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW`. -- The default word-run reorder window is `1.0` seconds. -- Set `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW` to a positive number of seconds to override the default. -- Replacement segment text is built by joining word text with single spaces. -- Replacement segments include `source_ref` and `derived_from`. -- Replacement segments omit `source_segment_index` because they are derived from one or more original segments. -- Resolved overlap groups are removed before the second detection pass. -- Replacement segments are left without `overlap_group_id` until the second detection pass annotates any remaining overlap. -- If a speaker has no usable word timing in a group, that speaker's original segment is kept. -- If no speakers in a group have usable word timing, the original group and annotations remain unchanged. - -## Backchannels - -The default pipeline runs `backchannel` before `coalesce`. It tags short acknowledgement segments with: - -```json -"categories": ["backchannel"] -``` - -Backchannel matching is case-insensitive, ignores punctuation for matching and word-count purposes, trims surrounding whitespace, and requires a matching acknowledgement phrase, no more than three whitespace-delimited words, and duration no greater than `SERIATIM_BACKCHANNEL_MAX_DURATION` seconds. The default maximum duration is `2.0` seconds. - -## Fillers - -The default pipeline runs `filler` after `backchannel` and before `coalesce`. It tags short filler utterances with: - -```json -"categories": ["filler"] -``` - -Filler matching is case-insensitive, ignores punctuation for matching and word-count purposes, trims surrounding whitespace, and requires only filler tokens such as `um`, `uh`, `er`, `erm`, `ah`, `eh`, `hmm`, `mm`, or repeated combinations of those tokens. Matching segments must contain no more than three whitespace-delimited words and have duration no greater than `SERIATIM_FILLER_MAX_DURATION` seconds. The default maximum duration is `1.25` seconds. - -## Dangler Resolution - -The default pipeline runs `resolve-danglers` before `coalesce` and before the second overlap detection pass. It repairs short derived fragments when they share provenance with a nearby segment: - -- Dangling-end fragments have no more than two words and end in punctuation. -- Dangling-start fragments have no more than two words. -- Matching uses same-speaker segments with any shared `derived_from` value. -- Merged segments use `source_ref` values such as `resolve-danglers:1`, keep the target segment's transcript position, and union `derived_from`. - -## Coalescing - -The default pipeline runs `coalesce` after `resolve-danglers` and before the second overlap detection pass. It merges adjacent same-speaker segments in the transcript's current order when `next.start - current.end <= --coalesce-gap`. - -Coalesced segments use `source_ref` values such as `coalesce:1`, include `derived_from`, and omit `source_segment_index`. - -Different-speaker backchannel and filler segments do not block coalescing of surrounding same-speaker segments. Same-speaker backchannel and filler segments are merged normally when they are within `--coalesce-gap`. When same-speaker segments are coalesced, any `backchannel` or `filler` category from the merged inputs is dropped from the coalesced segment. - -## Autocorrect - -Autocorrect is included in the default postprocessing pipeline. If `--autocorrect` is omitted, the module leaves transcript text unchanged and records a skip event in the optional report. - -Enable corrections by passing `--autocorrect`: - -```sh -go run ./cmd/seriatim merge \ - --input-file input.json \ - --autocorrect autocorrect.yml \ - --output-file merged.json -``` - -`autocorrect.yml` format: - -```yaml -autocorrect: - - target: "Hrank" - match: - - "hrank" - - "Frank" - - - target: "Mike Brown" - match: - - "Mike Pat" -``` - -Matching behavior: - -- Matching is case-sensitive. -- Matches apply only to whole tokens, not substrings inside larger words. -- Punctuation and whitespace can surround a match. -- Multi-word and hyphenated matches are supported. -- Duplicate match strings are invalid, including duplicates across separate rules. - -## Current Limitations - -- Only JSON input is supported. -- Overlap resolution depends on WhisperX word timing; groups without usable word timing remain unresolved. -- Alternate output formats are not implemented yet. - -## Release Builds - -Local builds record version metadata as `dev`. Release builds should inject the release version with `ldflags`: - -```sh -go build -ldflags "-X gitea.maximumdirect.net/eric/seriatim/internal/buildinfo.Version=v1.0.0" ./cmd/seriatim -``` +## Commands + +- `merge`: merge one or more input transcript JSON files. +- `trim`: keep/remove segment IDs from an existing seriatim artifact. +- `normalize`: canonicalize transcript-like JSON into a seriatim artifact. + +## Documentation + +- CLI reference: [docs/cli.md](docs/cli.md) +- Configuration reference: [docs/config.md](docs/config.md) +- Development architecture policy: [docs/policy/architecture.md](docs/policy/architecture.md) +- Documentation policy: [docs/policy/documentation.md](docs/policy/documentation.md) +- Public JSON schemas: + - [schema/minimal-output.schema.json](schema/minimal-output.schema.json) + - [schema/intermediate-output.schema.json](schema/intermediate-output.schema.json) + - [schema/full-output.schema.json](schema/full-output.schema.json) +- Remaining documentation migration work: [docs/roadmap/documentation.md](docs/roadmap/documentation.md) diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 0000000..37cbf46 --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,176 @@ +# CLI Reference + +## Shortest useful command + +```sh +go run ./cmd/seriatim merge \ + --input-file speaker-a.json \ + --input-file speaker-b.json \ + --output-file merged.json +``` + +## Command overview + +| Command | Purpose | +| --- | --- | +| `merge` | Merge one or more raw transcript JSON inputs into one seriatim artifact. | +| `trim` | Keep or remove segment IDs from an existing seriatim artifact. | +| `normalize` | Canonicalize transcript-like JSON into a seriatim artifact. | + +Root usage: + +```text +seriatim [command] +``` + +## Global flags + +| Flag | Description | +| --- | --- | +| `-h, --help` | Show help. | +| `-v, --version` | Show build version. | + +## `merge` + +Usage: + +```text +seriatim merge [flags] +``` + +Flags: + +| Flag | Required | Default | Description | +| --- | --- | --- | --- | +| `--input-file stringArray` | Yes, repeat at least once | none | Input transcript JSON file(s). | +| `--output-file string` | Yes | none | Output transcript JSON file path. | +| `--report-file string` | No | none | Optional report JSON path. | +| `--speakers string` | No | none | Speaker-map YAML file. | +| `--autocorrect string` | No | none | Autocorrect YAML file. | +| `--input-reader string` | No | `json-files` | Input reader module name. | +| `--output-modules string` | No | `json` | Comma-separated output module names. | +| `--output-schema string` | No | `seriatim-intermediate` | Output schema name: `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`. | +| `--preprocessing-modules string` | No | `validate-raw,normalize-speakers,trim-text` | Comma-separated preprocessing module names, run in order. | +| `--postprocessing-modules string` | No | `detect-overlaps,resolve-overlaps,backchannel,filler,resolve-danglers,coalesce,detect-overlaps,autocorrect,assign-ids,validate-output` | Comma-separated postprocessing module names, run in order. | +| `--coalesce-gap string` | No | `3.0` | Non-negative seconds for coalescing and overlap-resolution context. | + +`merge` behavior and validation: + +- Unknown input reader, preprocessing module, postprocessing module, or output module fails the command. +- Preprocessing order must satisfy module state requirements (`raw` -> `canonical`); invalid order fails. +- Input files are validated, deduplicated, normalized, then sorted for deterministic processing. +- Optional report output is written only when `--report-file` is set. +- When `--output-schema` is omitted, schema resolution is: `SERIATIM_OUTPUT_SCHEMA` -> default `seriatim-intermediate`. + +## `trim` + +Usage: + +```text +seriatim trim [flags] +``` + +Flags: + +| Flag | Required | Default | Description | +| --- | --- | --- | --- | +| `--input-file string` | Yes | none | Input seriatim artifact JSON file. | +| `--output-file string` | Yes | none | Output transcript JSON file path. | +| `--keep string` | Exactly one of `--keep` / `--remove` | none | Segment ID selector to keep. | +| `--remove string` | Exactly one of `--keep` / `--remove` | none | Segment ID selector to remove. | +| `--output-schema string` | No | preserve input artifact schema | Output schema override: `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`. | +| `--report-file string` | No | none | Optional report JSON path. | +| `--allow-empty` | No | `false` | Allow output with zero segments. | + +Selector rules: + +- IDs must be positive integers. +- Single IDs and inclusive ranges are supported: `1`, `1-10`. +- Comma-separated selectors are supported: `1-10,15,20-25`. +- Whitespace around commas and hyphens is allowed. +- Descending ranges (example `10-1`) are invalid. +- Duplicates and overlapping ranges are normalized as a union. + +`trim` behavior: + +- Input must already be a valid seriatim artifact (not raw merge input JSON). +- Output keeps transcript order from input and renumbers retained segment IDs sequentially. +- If `--output-schema` is omitted, the input artifact schema is preserved. +- `trim` never runs merge preprocessing/postprocessing modules. + +## `normalize` + +Usage: + +```text +seriatim normalize [flags] +``` + +Flags: + +| Flag | Required | Default | Description | +| --- | --- | --- | --- | +| `--input-file string` | Yes | none | Input transcript JSON file. | +| `--output-file string` | Yes | none | Output transcript JSON file path. | +| `--output-schema string` | No | `seriatim-intermediate` | Output schema name: `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`. | +| `--output-modules string` | No | `json` | Comma-separated output module names (`json` only). | +| `--report-file string` | No | none | Optional report JSON path. | + +`normalize` input shapes: + +- Object with top-level `segments` array. +- Bare top-level segment array. + +`normalize` behavior: + +- Sorts deterministically and reassigns output IDs sequentially from `1`. +- Fills missing/blank speakers with `Unknown_Speaker`. +- Repairs/sanitizes timing fields deterministically; rejects invalid repaired timing. +- Drops segments with missing or blank text. +- Does not run merge modules. +- When `--output-schema` is omitted, schema resolution is: `SERIATIM_OUTPUT_SCHEMA` -> default `seriatim-intermediate`. + +## Common workflows + +Merge with a speaker map and report output: + +```sh +go run ./cmd/seriatim merge \ + --input-file speaker-a.json \ + --input-file speaker-b.json \ + --speakers speakers.yml \ + --output-file merged.json \ + --report-file merge-report.json +``` + +Trim to a segment subset: + +```sh +go run ./cmd/seriatim trim \ + --input-file merged.json \ + --output-file trimmed.json \ + --keep "1-20,25" +``` + +Normalize an external transcript JSON file: + +```sh +go run ./cmd/seriatim normalize \ + --input-file external.json \ + --output-file normalized.json +``` + +## Exit and errors + +- Commands return exit code `0` on success. +- On error, the CLI prints one error line to stderr and exits with status `1`. +- Cobra usage text is silenced on runtime errors; use `--help` for command usage. + +## Related docs + +- Configuration reference: [config.md](config.md) +- Public output schemas: + - [../schema/minimal-output.schema.json](../schema/minimal-output.schema.json) + - [../schema/intermediate-output.schema.json](../schema/intermediate-output.schema.json) + - [../schema/full-output.schema.json](../schema/full-output.schema.json) +- Remaining docs migration work (operations, troubleshooting, examples): [roadmap/documentation.md](roadmap/documentation.md) diff --git a/docs/config.md b/docs/config.md new file mode 100644 index 0000000..ede1637 --- /dev/null +++ b/docs/config.md @@ -0,0 +1,162 @@ +# Configuration Reference + +## Configuration surfaces + +seriatim has no central JSON/TOML/YAML application config file. + +Runtime configuration comes from: + +1. CLI flags +2. Environment variables (`SERIATIM_*`) +3. Optional YAML rule files referenced by CLI flags (`--speakers`, `--autocorrect`) + +## Output schema precedence + +For `merge` and `normalize`: + +1. `--output-schema` flag (when explicitly set) +2. `SERIATIM_OUTPUT_SCHEMA` +3. default `seriatim-intermediate` + +For `trim`: + +- If `--output-schema` is omitted, output preserves the input artifact schema. +- If `--output-schema` is set, it must be one of `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`. + +## Merge module defaults + +Default merge module selections: + +- `--input-reader`: `json-files` +- `--preprocessing-modules`: `validate-raw,normalize-speakers,trim-text` +- `--postprocessing-modules`: `detect-overlaps,resolve-overlaps,backchannel,filler,resolve-danglers,coalesce,detect-overlaps,autocorrect,assign-ids,validate-output` +- `--output-modules`: `json` + +Module-list notes: + +- Lists are comma-separated. +- Empty module names are invalid. +- Unknown module names fail the command. +- Preprocessing order must satisfy state requirements. + +## Environment variables + +| Variable | Default | Used by | Rules | +| --- | --- | --- | --- | +| `SERIATIM_OUTPUT_SCHEMA` | `seriatim-intermediate` | `merge`, `normalize` | Must be `seriatim-minimal`, `seriatim-intermediate`, or `seriatim-full`. Ignored when `--output-schema` is explicitly set. | +| `SERIATIM_OVERLAP_WORD_RUN_GAP` | `1.0` | `merge` | Positive float (`> 0`). | +| `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW` | `1.0` | `merge` | Positive float (`> 0`). | +| `SERIATIM_BACKCHANNEL_MAX_DURATION` | `2.0` | `merge` | Positive float (`> 0`). | +| `SERIATIM_FILLER_MAX_DURATION` | `1.25` | `merge` | Positive float (`> 0`). | + +Additional merge threshold flag: + +- `--coalesce-gap` defaults to `3.0` and must be a non-negative float (`>= 0`). + +## `speakers.yml` + +Purpose: + +- Maps each merge input filename basename to a canonical speaker label. + +Top-level key: + +- `match` (array of ordered rules) + +Rule fields: + +- `speaker` (required, non-empty) +- `match` (required, non-empty array of non-empty strings) + +Example: + +```yaml +match: + - speaker: "Alice" + match: + - "alice_track" + - "alice" + + - speaker: "Bob" + match: + - "bob_track" +``` + +Behavior: + +- Matching is case-insensitive. +- Matching is against basename only (not full path). +- First matching rule wins. +- Duplicate `speaker` values are invalid. +- If any input file has no match, merge fails. + +## `autocorrect.yml` + +Purpose: + +- Applies ordered token-level text replacements during merge `autocorrect` postprocessing. + +Top-level key: + +- `autocorrect` (array of rules) + +Rule fields: + +- `target` (required, non-empty) +- `match` (required, non-empty array of non-empty strings) + +Example: + +```yaml +autocorrect: + - target: "Godfrey" + match: + - "God-free" + + - target: "Mike Brown" + match: + - "Mike Pat" +``` + +Behavior: + +- Match strings are case-sensitive. +- Replacements are whole-token only (no substring replacement inside larger tokens). +- Duplicate match strings within one rule are invalid. +- Duplicate match strings across different rules are invalid. +- If `--autocorrect` is not provided, the autocorrect module is skipped. + +## Path and validation rules + +All commands: + +- `--input-file` paths must exist and must be files. +- Output/report parent directories must already exist. +- Paths are normalized before use. + +`merge`: + +- Requires at least one `--input-file`. +- Rejects duplicate `--input-file` paths. +- Sorts normalized input file paths for deterministic execution. +- `--speakers` and `--autocorrect` are optional, but when set they must point to existing files. + +`trim`: + +- Requires exactly one of `--keep` or `--remove`. +- `--keep` and `--remove` are mutually exclusive. +- Validates optional `--output-schema` when provided. + +`normalize`: + +- Validates `--output-schema` through the same schema set as `merge`. +- Currently accepts only `json` in `--output-modules`. + +## Related docs + +- CLI reference: [cli.md](cli.md) +- Public output schemas: + - [../schema/minimal-output.schema.json](../schema/minimal-output.schema.json) + - [../schema/intermediate-output.schema.json](../schema/intermediate-output.schema.json) + - [../schema/full-output.schema.json](../schema/full-output.schema.json) +- Remaining docs migration work (operations, troubleshooting, examples): [roadmap/documentation.md](roadmap/documentation.md)