diff --git a/README.md b/README.md index b8b6043..667031e 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,12 @@ go run ./cmd/seriatim merge \ - Operations guide: [docs/operations.md](docs/operations.md) - Troubleshooting: [docs/troubleshooting.md](docs/troubleshooting.md) - Development architecture policy: [docs/policy/architecture.md](docs/policy/architecture.md) +- Contributor workflow: [docs/policy/development.md](docs/policy/development.md) - Documentation policy: [docs/policy/documentation.md](docs/policy/documentation.md) +- Internal implementation docs: + - [docs/internal/pipeline.md](docs/internal/pipeline.md) + - [docs/internal/artifacts.md](docs/internal/artifacts.md) + - [docs/internal/modules.md](docs/internal/modules.md) - Public JSON schemas: - [schema/minimal-output.schema.json](schema/minimal-output.schema.json) - [schema/intermediate-output.schema.json](schema/intermediate-output.schema.json) diff --git a/docs/internal/artifacts.md b/docs/internal/artifacts.md new file mode 100644 index 0000000..421b7a8 --- /dev/null +++ b/docs/internal/artifacts.md @@ -0,0 +1,140 @@ +# Artifact Internals + +## Purpose + +Describes public artifact conversion and validation internals for merge output, +trim, and normalize. + +## Artifact contracts + +Public contracts live in `schema/`: + +- full: `schema.Transcript` +- intermediate: `schema.IntermediateTranscript` +- minimal: `schema.MinimalTranscript` + +Machine-readable schemas: + +- `schema/full-output.schema.json` +- `schema/intermediate-output.schema.json` +- `schema/minimal-output.schema.json` + +## Schema selection + +Merge pipeline conversion uses `internal/artifact.SelectedFromMerged`: + +- `seriatim-full` -> `artifact.FromMerged` +- `seriatim-intermediate` -> `artifact.IntermediateFromMerged` +- `seriatim-minimal` -> `artifact.MinimalFromMerged` + +Unknown/empty selection falls back to intermediate conversion. + +## Merge conversion behavior + +`internal/artifact` converts `model.MergedTranscript` to public contracts: + +- full schema preserves source/provenance, overlap groups, and metadata module + lists. +- intermediate schema emits segment timing/text/speaker with optional + categories and compact metadata. +- minimal schema emits compact segment timing/text/speaker and compact + metadata. + +## Validation behavior + +`schema/output.go` validates both structure and semantics: + +- embedded JSON Schema validation via `jsonschema/v6` +- semantic checks for sequential segment IDs starting at `1` +- semantic checks for non-inverted segment timing (`end >= start`) +- full schema overlap-group timing checks (`group.end >= group.start`) + +## Trim internals + +`internal/trim` is artifact-level projection, not merge reprocessing. + +Core flow: + +1. Parse selector (`internal/trim/selector.go`). +2. Parse input artifact and detect schema (`ParseArtifactJSON`). +3. Apply keep/remove projection with sequential ID renumbering. +4. Recompute overlap groups only for full-schema artifacts. +5. Optionally convert output schema when supported. +6. Validate output artifact before write. + +Schema-conversion limits: + +- full -> intermediate/minimal supported. +- intermediate -> minimal supported. +- minimal -> intermediate supported. +- intermediate/minimal -> full is rejected. + +Trim invariants: + +- selected IDs must exist in input. +- input IDs must be positive, unique, sequential. +- retained order follows input transcript order. +- output IDs are reassigned to `1..N`. + +## Normalize internals + +`internal/normalize` canonicalizes transcript-like JSON input into a selected +public schema. + +Parse layer (`parse.go`): + +- accepts object-with-`segments` or bare segment array +- repairs missing timing deterministically +- swaps inverted timing +- fills missing/blank speaker with `Unknown_Speaker` +- drops missing/blank text segments + +Build layer (`build.go`): + +- sorts deterministically by `(start, end, input_index, speaker)` +- reassigns output IDs sequentially +- builds minimal/intermediate/full output shape +- validates selected output schema before write + +Run layer (`normalize.go`): + +- writes output JSON +- optionally writes report with `normalize-audit` + +Normalize invariant: + +- report events do not embed transcript text. + +## Boundaries + +- CLI flag semantics belong to `docs/cli.md`. +- Runtime config/env surfaces belong to `docs/config.md`. +- This doc describes internal conversion/validation behavior only. + +## Failure behavior + +Representative failure classes: + +- malformed or unsupported input JSON shape +- schema validation failure for parsed artifact or built output +- unsupported schema conversion path (trim) +- selector or input-ID consistency errors (trim) +- output/report file write failures from command paths + +## Tests to inspect before changes + +- `schema/output_test.go` +- `internal/artifact/transcript_test.go` +- `internal/trim/selector_test.go` +- `internal/trim/artifact_test.go` +- `internal/trim/apply_test.go` +- `internal/normalize/parse_test.go` +- `internal/cli/trim_test.go` +- `internal/cli/normalize_test.go` + +## Invariants + +- Public artifacts are validated through `schema` before acceptance. +- Segment IDs in emitted artifacts are sequential and deterministic. +- Internal-only fields are not emitted in minimal/intermediate contracts. +- Trim and normalize stay artifact-level and do not execute merge modules. diff --git a/docs/internal/modules.md b/docs/internal/modules.md new file mode 100644 index 0000000..98b5e21 --- /dev/null +++ b/docs/internal/modules.md @@ -0,0 +1,119 @@ +# Built-In Modules + +## Purpose + +Describes implemented built-in module behavior and boundaries in +`internal/builtin`. + +## Implemented module set + +Input reader: + +- `json-files` + +Preprocessing: + +- `validate-raw` +- `normalize-speakers` +- `trim-text` + +Merger: + +- `chronological-merge` + +Postprocessing: + +- `detect-overlaps` +- `resolve-overlaps` +- `backchannel` +- `filler` +- `resolve-danglers` +- `coalesce` +- `autocorrect` +- `assign-ids` +- `validate-output` + +Output writer: + +- `json` + +## Inputs, outputs, and side effects + +- `json-files`: reads JSON files from `cfg.InputFiles`, parses supported + segment/word fields, emits warnings for untimed words. +- `validate-raw`: validates raw source/timing invariants. +- `normalize-speakers`: converts raw transcripts to canonical segments, + optionally resolving speakers from `cfg.SpeakersFile`. +- `trim-text`: trims canonical segment text whitespace. +- `chronological-merge`: flattens canonical segments and applies deterministic + sort (`model.SegmentLess`). +- `detect-overlaps`: annotates overlap groups. +- `resolve-overlaps`: rewrites overlap groups using timed words and thresholds. +- `backchannel`/`filler`: classify short utterances using duration thresholds. +- `resolve-danglers`: merges dangling derived fragments. +- `coalesce`: merges adjacent same-speaker segments within configured gap. +- `autocorrect`: applies YAML replacement rules when configured. +- `assign-ids`: assigns final sequential IDs. +- `validate-output`: validates selected public artifact shape. +- `json`: writes artifact JSON to `cfg.OutputFile`. + +Filesystem side effects are limited to: + +- reading configured input/YAML files +- writing configured output artifact + +## Config fields used + +Primary module inputs from `config.Config`: + +- file paths: `InputFiles`, `SpeakersFile`, `AutocorrectFile`, `OutputFile` +- schema/modules: `OutputSchema`, `OutputModules` +- overlap/coalesce thresholds: `OverlapWordRunGap`, + `WordRunReorderWindow`, `CoalesceGap` +- category thresholds: `BackchannelMaxDuration`, `FillerMaxDuration` + +## Ordering constraints + +- Preprocessing must satisfy state contracts from `raw` to `canonical`. +- Order-sensitive transforms should run before `assign-ids`. +- `validate-output` should run after final ID assignment and output-shape + mutations. +- Default configuration includes a second `detect-overlaps` pass after + transformations. + +## Boundaries + +- Modules implement behavior; CLI/config parsing remains outside modules. +- Modules communicate through explicit model contracts and report events. +- Output modules operate on final artifacts and do not re-run transform logic. + +## Failure behavior + +Representative failures: + +- invalid input JSON shape or typed field errors (`json-files`) +- invalid YAML or unmatched speaker map entries +- unknown module names during registry resolution +- invalid ordering/state transitions in preprocessing chain +- validation failure in `validate-output` +- output write failure in `json` writer + +## Tests to inspect before changes + +- `internal/builtin/preprocess_test.go` +- `internal/builtin/postprocess_test.go` +- `internal/overlap/resolve_test.go` +- `internal/overlap/detect_test.go` +- `internal/coalesce/coalesce_test.go` +- `internal/danglers/danglers_test.go` +- `internal/backchannel/backchannel_test.go` +- `internal/filler/filler_test.go` +- `internal/autocorrect/autocorrect_test.go` +- `internal/cli/merge_test.go` + +## Invariants + +- Modules are selected by canonical name through the registry. +- Execution is sequential and deterministic for a fixed configuration. +- `assign-ids` defines final public segment IDs. +- `validate-output` enforces public artifact contracts through `schema`. diff --git a/docs/internal/pipeline.md b/docs/internal/pipeline.md new file mode 100644 index 0000000..386a063 --- /dev/null +++ b/docs/internal/pipeline.md @@ -0,0 +1,102 @@ +# Pipeline Internals + +## Purpose + +Describes implemented merge pipeline orchestration in `internal/pipeline`. + +## Inputs and outputs + +Input: + +- `config.Config` +- registry-resolved modules from `internal/builtin` + +Output: + +- selected public artifact written by output writer modules +- optional report JSON when `cfg.ReportFile` is set + +## Stage contracts + +The runner executes these contracts in order: + +1. `InputReader`: external inputs -> `[]model.RawTranscript` +2. `Preprocessor`: `PreprocessState` transformations (`raw` -> `canonical`) +3. `Merger`: canonical transcripts -> `model.MergedTranscript` +4. `Postprocessor`: merged transcript transformations +5. `OutputWriter`: serialized artifact writes + +`PreprocessState` must end in `StateCanonical` before merge. + +## Registry resolution + +`resolvePlan` maps configured names to modules: + +- input reader: `cfg.InputReader` +- preprocessors: `cfg.PreprocessingModules` +- postprocessors: `cfg.PostprocessingModules` +- output writers: `cfg.OutputModules` +- merger: single registered merger + +Unknown names fail fast with contextual errors. + +## Execution order and reporting + +- Modules run sequentially in configured order. +- Events returned by modules are appended in execution order. +- Report metadata includes input reader, input files, and module lists. +- Output writer events are appended before optional report write. + +## Config fields used + +Runner-level fields: + +- `InputReader` +- `InputFiles` +- `PreprocessingModules` +- `PostprocessingModules` +- `OutputModules` +- `OutputSchema` (via `artifact.SelectedFromMerged`) +- `ReportFile` + +Module-specific settings are consumed inside builtin modules (for example +coalesce gap and overlap thresholds). + +## Adapters used + +- Input adapters: registered `InputReader` implementations (default `json-files`). +- Output adapters: registered `OutputWriter` implementations (default `json`). +- Report adapter: `report.WriteJSON` when `cfg.ReportFile` is provided. + +## Boundaries + +- Pipeline does not parse CLI flags. +- Pipeline does not normalize raw CLI strings. +- Pipeline delegates conversion to public output contracts to `internal/artifact`. +- Artifact-level commands `trim` and `normalize` are outside this pipeline. + +## Failure behavior + +Pipeline returns errors from: + +- registry resolution (unknown modules, missing merger) +- invalid preprocessing state transitions +- module read/process/merge/write failures +- optional report write failure + +No retry/resume state is stored. + +## Tests to inspect before changes + +- `internal/pipeline/runner_test.go` +- `internal/builtin/preprocess_test.go` +- `internal/builtin/postprocess_test.go` +- `internal/cli/merge_test.go` + +## Invariants + +- Sequential deterministic execution order. +- Preprocessing state must type-check from `raw` to `canonical`. +- Module selection is explicit by canonical names. +- Report event order reflects actual execution order. +- Output artifact selection is schema-driven via `internal/artifact`. diff --git a/docs/policy/development.md b/docs/policy/development.md new file mode 100644 index 0000000..6cef3ef --- /dev/null +++ b/docs/policy/development.md @@ -0,0 +1,110 @@ +# Development Policy + +## Purpose + +This document defines contributor workflow for maintainers and coding agents. +It complements [architecture policy](architecture.md) and +[documentation policy](documentation.md). + +## Repository layout + +- `cmd/seriatim/`: process entrypoint. +- `internal/cli/`: Cobra commands and flag wiring. +- `internal/config/`: option normalization and validation. +- `internal/pipeline/`: orchestration interfaces, registry, runner. +- `internal/builtin/`: implemented input/pre/post/output modules and merger. +- `internal/artifact/`: conversion from internal merged model to public shapes. +- `internal/trim/`: artifact-level trim logic. +- `internal/normalize/`: artifact-level normalize parsing/building. +- `internal/*` domain packages: overlap, coalesce, danglers, filler, + backchannel, speaker, autocorrect, report, model. +- `schema/`: public structs plus embedded JSON Schemas and validation. +- `docs/`: policy, user docs, roadmap, and internal docs. + +## Local checks + +Primary repository check: + +```sh +go test ./... +``` + +Useful manual checks for CLI-facing changes: + +```sh +go run ./cmd/seriatim --help +go run ./cmd/seriatim merge --help +go run ./cmd/seriatim trim --help +go run ./cmd/seriatim normalize --help +``` + +Current toolchain note: + +- There is no Makefile. +- There is no taskfile. +- There is no committed linter configuration. +- There is no automated documentation checker. + +## Coding conventions + +- Keep core behavior deterministic for identical inputs/config/version. +- Keep CLI command functions thin: parse flags, construct config, delegate. +- Keep validation in `internal/config` and package-specific validators. +- Return errors from deep logic; do not print inside internal packages. +- Preserve clear package boundaries between adapters and domain transforms. + +## Dependency policy + +Prefer the Go standard library first. + +Third-party dependencies should stay narrow and justified. Current direct +runtime dependencies are: + +- `github.com/spf13/cobra` for CLI structure. +- `gopkg.in/yaml.v3` for YAML rule files. +- `github.com/santhosh-tekuri/jsonschema/v6` for public schema validation. + +## Adding CLI flags + +1. Add the flag in the relevant `internal/cli/*.go` command. +2. Thread the raw value through `config.*Options`. +3. Add normalization/validation in `internal/config/config.go`. +4. Update or add CLI/config tests. +5. Update canonical docs (`docs/cli.md`, `docs/config.md`) if user-visible. + +## Adding config fields or environment variables + +1. Add field(s) to the relevant config struct(s). +2. Parse and validate in `internal/config/config.go`. +3. Add tests in `internal/config/config_test.go`. +4. Thread validated values into consuming modules. +5. Update `docs/config.md` and related docs. + +## Adding modules or pipeline behavior + +1. Implement the module in the appropriate package (often `internal/builtin`). +2. Expose a stable module name via `Name()`. +3. Register it in `internal/builtin/registry.go`. +4. Ensure preprocessing modules declare correct `Requires()`/`Produces()` + states. +5. Add/adjust tests in module packages and `internal/cli/merge_test.go`. +6. Document internal behavior changes in `docs/internal/`. + +## Schema and artifact changes + +1. Update public structs and validation logic in `schema/`. +2. Update embedded JSON Schema files (`schema/*.schema.json`) if contract + changes. +3. Update conversion behavior in `internal/artifact`, `internal/trim`, and/or + `internal/normalize` as needed. +4. Add tests in `schema/`, `internal/artifact/`, `internal/trim/`, + `internal/normalize/`, and CLI tests. +5. Update user and internal docs that reference output contracts. + +## Documentation expectations + +- Outside `docs/roadmap/`, document only implemented behavior. +- Keep canonical homes: CLI in `docs/cli.md`, config in `docs/config.md`, + operations in `docs/operations.md`, troubleshooting in + `docs/troubleshooting.md`, internals in `docs/internal/`. +- When behavior changes, update docs in the same change.