eric/seriatim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Add development policy and internal implementation documentation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-24 13:38:27 +00:00
parent b3e7dc3136
commit 88018c9e76
5 changed files with 476 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
README.md
View File

@@ -28,7 +28,12 @@ go run ./cmd/seriatim merge \
- Operations guide: [docs/operations.md](docs/operations.md) - Operations guide: [docs/operations.md](docs/operations.md)
- Troubleshooting: [docs/troubleshooting.md](docs/troubleshooting.md) - Troubleshooting: [docs/troubleshooting.md](docs/troubleshooting.md)
- Development architecture policy: [docs/policy/architecture.md](docs/policy/architecture.md) - Development architecture policy: [docs/policy/architecture.md](docs/policy/architecture.md)
- Contributor workflow: [docs/policy/development.md](docs/policy/development.md)
- Documentation policy: [docs/policy/documentation.md](docs/policy/documentation.md) - Documentation policy: [docs/policy/documentation.md](docs/policy/documentation.md)
- Internal implementation docs:
- [docs/internal/pipeline.md](docs/internal/pipeline.md)
- [docs/internal/artifacts.md](docs/internal/artifacts.md)
- [docs/internal/modules.md](docs/internal/modules.md)
- Public JSON schemas: - Public JSON schemas:
- [schema/minimal-output.schema.json](schema/minimal-output.schema.json) - [schema/minimal-output.schema.json](schema/minimal-output.schema.json)
- [schema/intermediate-output.schema.json](schema/intermediate-output.schema.json) - [schema/intermediate-output.schema.json](schema/intermediate-output.schema.json)

140
docs/internal/artifacts.md Normal file
View File

@@ -0,0 +1,140 @@
# Artifact Internals
## Purpose
Describes public artifact conversion and validation internals for merge output,
trim, and normalize.
## Artifact contracts
Public contracts live in `schema/`:
- full: `schema.Transcript`
- intermediate: `schema.IntermediateTranscript`
- minimal: `schema.MinimalTranscript`
Machine-readable schemas:
- `schema/full-output.schema.json`
- `schema/intermediate-output.schema.json`
- `schema/minimal-output.schema.json`
## Schema selection
Merge pipeline conversion uses `internal/artifact.SelectedFromMerged`:
- `seriatim-full` -> `artifact.FromMerged`
- `seriatim-intermediate` -> `artifact.IntermediateFromMerged`
- `seriatim-minimal` -> `artifact.MinimalFromMerged`
Unknown/empty selection falls back to intermediate conversion.
## Merge conversion behavior
`internal/artifact` converts `model.MergedTranscript` to public contracts:
- full schema preserves source/provenance, overlap groups, and metadata module
lists.
- intermediate schema emits segment timing/text/speaker with optional
categories and compact metadata.
- minimal schema emits compact segment timing/text/speaker and compact
metadata.
## Validation behavior
`schema/output.go` validates both structure and semantics:
- embedded JSON Schema validation via `jsonschema/v6`
- semantic checks for sequential segment IDs starting at `1`
- semantic checks for non-inverted segment timing (`end >= start`)
- full schema overlap-group timing checks (`group.end >= group.start`)
## Trim internals
`internal/trim` is artifact-level projection, not merge reprocessing.
Core flow:
1. Parse selector (`internal/trim/selector.go`).
2. Parse input artifact and detect schema (`ParseArtifactJSON`).
3. Apply keep/remove projection with sequential ID renumbering.
4. Recompute overlap groups only for full-schema artifacts.
5. Optionally convert output schema when supported.
6. Validate output artifact before write.
Schema-conversion limits:
- full -> intermediate/minimal supported.
- intermediate -> minimal supported.
- minimal -> intermediate supported.
- intermediate/minimal -> full is rejected.
Trim invariants:
- selected IDs must exist in input.
- input IDs must be positive, unique, sequential.
- retained order follows input transcript order.
- output IDs are reassigned to `1..N`.
## Normalize internals
`internal/normalize` canonicalizes transcript-like JSON input into a selected
public schema.
Parse layer (`parse.go`):
- accepts object-with-`segments` or bare segment array
- repairs missing timing deterministically
- swaps inverted timing
- fills missing/blank speaker with `Unknown_Speaker`
- drops missing/blank text segments
Build layer (`build.go`):
- sorts deterministically by `(start, end, input_index, speaker)`
- reassigns output IDs sequentially
- builds minimal/intermediate/full output shape
- validates selected output schema before write
Run layer (`normalize.go`):
- writes output JSON
- optionally writes report with `normalize-audit`
Normalize invariant:
- report events do not embed transcript text.
## Boundaries
- CLI flag semantics belong to `docs/cli.md`.
- Runtime config/env surfaces belong to `docs/config.md`.
- This doc describes internal conversion/validation behavior only.
## Failure behavior
Representative failure classes:
- malformed or unsupported input JSON shape
- schema validation failure for parsed artifact or built output
- unsupported schema conversion path (trim)
- selector or input-ID consistency errors (trim)
- output/report file write failures from command paths
## Tests to inspect before changes
- `schema/output_test.go`
- `internal/artifact/transcript_test.go`
- `internal/trim/selector_test.go`
- `internal/trim/artifact_test.go`
- `internal/trim/apply_test.go`
- `internal/normalize/parse_test.go`
- `internal/cli/trim_test.go`
- `internal/cli/normalize_test.go`
## Invariants
- Public artifacts are validated through `schema` before acceptance.
- Segment IDs in emitted artifacts are sequential and deterministic.
- Internal-only fields are not emitted in minimal/intermediate contracts.
- Trim and normalize stay artifact-level and do not execute merge modules.

119
docs/internal/modules.md Normal file
View File

@@ -0,0 +1,119 @@
# Built-In Modules
## Purpose
Describes implemented built-in module behavior and boundaries in
`internal/builtin`.
## Implemented module set
Input reader:
- `json-files`
Preprocessing:
- `validate-raw`
- `normalize-speakers`
- `trim-text`
Merger:
- `chronological-merge`
Postprocessing:
- `detect-overlaps`
- `resolve-overlaps`
- `backchannel`
- `filler`
- `resolve-danglers`
- `coalesce`
- `autocorrect`
- `assign-ids`
- `validate-output`
Output writer:
- `json`
## Inputs, outputs, and side effects
- `json-files`: reads JSON files from `cfg.InputFiles`, parses supported
segment/word fields, emits warnings for untimed words.
- `validate-raw`: validates raw source/timing invariants.
- `normalize-speakers`: converts raw transcripts to canonical segments,
optionally resolving speakers from `cfg.SpeakersFile`.
- `trim-text`: trims canonical segment text whitespace.
- `chronological-merge`: flattens canonical segments and applies deterministic
sort (`model.SegmentLess`).
- `detect-overlaps`: annotates overlap groups.
- `resolve-overlaps`: rewrites overlap groups using timed words and thresholds.
- `backchannel`/`filler`: classify short utterances using duration thresholds.
- `resolve-danglers`: merges dangling derived fragments.
- `coalesce`: merges adjacent same-speaker segments within configured gap.
- `autocorrect`: applies YAML replacement rules when configured.
- `assign-ids`: assigns final sequential IDs.
- `validate-output`: validates selected public artifact shape.
- `json`: writes artifact JSON to `cfg.OutputFile`.
Filesystem side effects are limited to:
- reading configured input/YAML files
- writing configured output artifact
## Config fields used
Primary module inputs from `config.Config`:
- file paths: `InputFiles`, `SpeakersFile`, `AutocorrectFile`, `OutputFile`
- schema/modules: `OutputSchema`, `OutputModules`
- overlap/coalesce thresholds: `OverlapWordRunGap`,
`WordRunReorderWindow`, `CoalesceGap`
- category thresholds: `BackchannelMaxDuration`, `FillerMaxDuration`
## Ordering constraints
- Preprocessing must satisfy state contracts from `raw` to `canonical`.
- Order-sensitive transforms should run before `assign-ids`.
- `validate-output` should run after final ID assignment and output-shape
mutations.
- Default configuration includes a second `detect-overlaps` pass after
transformations.
## Boundaries
- Modules implement behavior; CLI/config parsing remains outside modules.
- Modules communicate through explicit model contracts and report events.
- Output modules operate on final artifacts and do not re-run transform logic.
## Failure behavior
Representative failures:
- invalid input JSON shape or typed field errors (`json-files`)
- invalid YAML or unmatched speaker map entries
- unknown module names during registry resolution
- invalid ordering/state transitions in preprocessing chain
- validation failure in `validate-output`
- output write failure in `json` writer
## Tests to inspect before changes
- `internal/builtin/preprocess_test.go`
- `internal/builtin/postprocess_test.go`
- `internal/overlap/resolve_test.go`
- `internal/overlap/detect_test.go`
- `internal/coalesce/coalesce_test.go`
- `internal/danglers/danglers_test.go`
- `internal/backchannel/backchannel_test.go`
- `internal/filler/filler_test.go`
- `internal/autocorrect/autocorrect_test.go`
- `internal/cli/merge_test.go`
## Invariants
- Modules are selected by canonical name through the registry.
- Execution is sequential and deterministic for a fixed configuration.
- `assign-ids` defines final public segment IDs.
- `validate-output` enforces public artifact contracts through `schema`.

102
docs/internal/pipeline.md Normal file
View File

@@ -0,0 +1,102 @@
# Pipeline Internals
## Purpose
Describes implemented merge pipeline orchestration in `internal/pipeline`.
## Inputs and outputs
Input:
- `config.Config`
- registry-resolved modules from `internal/builtin`
Output:
- selected public artifact written by output writer modules
- optional report JSON when `cfg.ReportFile` is set
## Stage contracts
The runner executes these contracts in order:
1. `InputReader`: external inputs -> `[]model.RawTranscript`
2. `Preprocessor`: `PreprocessState` transformations (`raw` -> `canonical`)
3. `Merger`: canonical transcripts -> `model.MergedTranscript`
4. `Postprocessor`: merged transcript transformations
5. `OutputWriter`: serialized artifact writes
`PreprocessState` must end in `StateCanonical` before merge.
## Registry resolution
`resolvePlan` maps configured names to modules:
- input reader: `cfg.InputReader`
- preprocessors: `cfg.PreprocessingModules`
- postprocessors: `cfg.PostprocessingModules`
- output writers: `cfg.OutputModules`
- merger: single registered merger
Unknown names fail fast with contextual errors.
## Execution order and reporting
- Modules run sequentially in configured order.
- Events returned by modules are appended in execution order.
- Report metadata includes input reader, input files, and module lists.
- Output writer events are appended before optional report write.
## Config fields used
Runner-level fields:
- `InputReader`
- `InputFiles`
- `PreprocessingModules`
- `PostprocessingModules`
- `OutputModules`
- `OutputSchema` (via `artifact.SelectedFromMerged`)
- `ReportFile`
Module-specific settings are consumed inside builtin modules (for example
coalesce gap and overlap thresholds).
## Adapters used
- Input adapters: registered `InputReader` implementations (default `json-files`).
- Output adapters: registered `OutputWriter` implementations (default `json`).
- Report adapter: `report.WriteJSON` when `cfg.ReportFile` is provided.
## Boundaries
- Pipeline does not parse CLI flags.
- Pipeline does not normalize raw CLI strings.
- Pipeline delegates conversion to public output contracts to `internal/artifact`.
- Artifact-level commands `trim` and `normalize` are outside this pipeline.
## Failure behavior
Pipeline returns errors from:
- registry resolution (unknown modules, missing merger)
- invalid preprocessing state transitions
- module read/process/merge/write failures
- optional report write failure
No retry/resume state is stored.
## Tests to inspect before changes
- `internal/pipeline/runner_test.go`
- `internal/builtin/preprocess_test.go`
- `internal/builtin/postprocess_test.go`
- `internal/cli/merge_test.go`
## Invariants
- Sequential deterministic execution order.
- Preprocessing state must type-check from `raw` to `canonical`.
- Module selection is explicit by canonical names.
- Report event order reflects actual execution order.
- Output artifact selection is schema-driven via `internal/artifact`.

110
docs/policy/development.md Normal file
View File

@@ -0,0 +1,110 @@
# Development Policy
## Purpose
This document defines contributor workflow for maintainers and coding agents.
It complements [architecture policy](architecture.md) and
[documentation policy](documentation.md).
## Repository layout
- `cmd/seriatim/`: process entrypoint.
- `internal/cli/`: Cobra commands and flag wiring.
- `internal/config/`: option normalization and validation.
- `internal/pipeline/`: orchestration interfaces, registry, runner.
- `internal/builtin/`: implemented input/pre/post/output modules and merger.
- `internal/artifact/`: conversion from internal merged model to public shapes.
- `internal/trim/`: artifact-level trim logic.
- `internal/normalize/`: artifact-level normalize parsing/building.
- `internal/*` domain packages: overlap, coalesce, danglers, filler,
backchannel, speaker, autocorrect, report, model.
- `schema/`: public structs plus embedded JSON Schemas and validation.
- `docs/`: policy, user docs, roadmap, and internal docs.
## Local checks
Primary repository check:
```sh
go test ./...
```
Useful manual checks for CLI-facing changes:
```sh
go run ./cmd/seriatim --help
go run ./cmd/seriatim merge --help
go run ./cmd/seriatim trim --help
go run ./cmd/seriatim normalize --help
```
Current toolchain note:
- There is no Makefile.
- There is no taskfile.
- There is no committed linter configuration.
- There is no automated documentation checker.
## Coding conventions
- Keep core behavior deterministic for identical inputs/config/version.
- Keep CLI command functions thin: parse flags, construct config, delegate.
- Keep validation in `internal/config` and package-specific validators.
- Return errors from deep logic; do not print inside internal packages.
- Preserve clear package boundaries between adapters and domain transforms.
## Dependency policy
Prefer the Go standard library first.
Third-party dependencies should stay narrow and justified. Current direct
runtime dependencies are:
- `github.com/spf13/cobra` for CLI structure.
- `gopkg.in/yaml.v3` for YAML rule files.
- `github.com/santhosh-tekuri/jsonschema/v6` for public schema validation.
## Adding CLI flags
1. Add the flag in the relevant `internal/cli/*.go` command.
2. Thread the raw value through `config.*Options`.
3. Add normalization/validation in `internal/config/config.go`.
4. Update or add CLI/config tests.
5. Update canonical docs (`docs/cli.md`, `docs/config.md`) if user-visible.
## Adding config fields or environment variables
1. Add field(s) to the relevant config struct(s).
2. Parse and validate in `internal/config/config.go`.
3. Add tests in `internal/config/config_test.go`.
4. Thread validated values into consuming modules.
5. Update `docs/config.md` and related docs.
## Adding modules or pipeline behavior
1. Implement the module in the appropriate package (often `internal/builtin`).
2. Expose a stable module name via `Name()`.
3. Register it in `internal/builtin/registry.go`.
4. Ensure preprocessing modules declare correct `Requires()`/`Produces()`
states.
5. Add/adjust tests in module packages and `internal/cli/merge_test.go`.
6. Document internal behavior changes in `docs/internal/`.
## Schema and artifact changes
1. Update public structs and validation logic in `schema/`.
2. Update embedded JSON Schema files (`schema/*.schema.json`) if contract
changes.
3. Update conversion behavior in `internal/artifact`, `internal/trim`, and/or
`internal/normalize` as needed.
4. Add tests in `schema/`, `internal/artifact/`, `internal/trim/`,
`internal/normalize/`, and CLI tests.
5. Update user and internal docs that reference output contracts.
## Documentation expectations
- Outside `docs/roadmap/`, document only implemented behavior.
- Keep canonical homes: CLI in `docs/cli.md`, config in `docs/config.md`,
operations in `docs/operations.md`, troubleshooting in
`docs/troubleshooting.md`, internals in `docs/internal/`.
- When behavior changes, update docs in the same change.