Add development policy and internal implementation documentation
This commit is contained in:
@@ -28,7 +28,12 @@ go run ./cmd/seriatim merge \
|
||||
- Operations guide: [docs/operations.md](docs/operations.md)
|
||||
- Troubleshooting: [docs/troubleshooting.md](docs/troubleshooting.md)
|
||||
- Development architecture policy: [docs/policy/architecture.md](docs/policy/architecture.md)
|
||||
- Contributor workflow: [docs/policy/development.md](docs/policy/development.md)
|
||||
- Documentation policy: [docs/policy/documentation.md](docs/policy/documentation.md)
|
||||
- Internal implementation docs:
|
||||
- [docs/internal/pipeline.md](docs/internal/pipeline.md)
|
||||
- [docs/internal/artifacts.md](docs/internal/artifacts.md)
|
||||
- [docs/internal/modules.md](docs/internal/modules.md)
|
||||
- Public JSON schemas:
|
||||
- [schema/minimal-output.schema.json](schema/minimal-output.schema.json)
|
||||
- [schema/intermediate-output.schema.json](schema/intermediate-output.schema.json)
|
||||
|
||||
140
docs/internal/artifacts.md
Normal file
140
docs/internal/artifacts.md
Normal file
@@ -0,0 +1,140 @@
|
||||
# Artifact Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
Describes public artifact conversion and validation internals for merge output,
|
||||
trim, and normalize.
|
||||
|
||||
## Artifact contracts
|
||||
|
||||
Public contracts live in `schema/`:
|
||||
|
||||
- full: `schema.Transcript`
|
||||
- intermediate: `schema.IntermediateTranscript`
|
||||
- minimal: `schema.MinimalTranscript`
|
||||
|
||||
Machine-readable schemas:
|
||||
|
||||
- `schema/full-output.schema.json`
|
||||
- `schema/intermediate-output.schema.json`
|
||||
- `schema/minimal-output.schema.json`
|
||||
|
||||
## Schema selection
|
||||
|
||||
Merge pipeline conversion uses `internal/artifact.SelectedFromMerged`:
|
||||
|
||||
- `seriatim-full` -> `artifact.FromMerged`
|
||||
- `seriatim-intermediate` -> `artifact.IntermediateFromMerged`
|
||||
- `seriatim-minimal` -> `artifact.MinimalFromMerged`
|
||||
|
||||
Unknown/empty selection falls back to intermediate conversion.
|
||||
|
||||
## Merge conversion behavior
|
||||
|
||||
`internal/artifact` converts `model.MergedTranscript` to public contracts:
|
||||
|
||||
- full schema preserves source/provenance, overlap groups, and metadata module
|
||||
lists.
|
||||
- intermediate schema emits segment timing/text/speaker with optional
|
||||
categories and compact metadata.
|
||||
- minimal schema emits compact segment timing/text/speaker and compact
|
||||
metadata.
|
||||
|
||||
## Validation behavior
|
||||
|
||||
`schema/output.go` validates both structure and semantics:
|
||||
|
||||
- embedded JSON Schema validation via `jsonschema/v6`
|
||||
- semantic checks for sequential segment IDs starting at `1`
|
||||
- semantic checks for non-inverted segment timing (`end >= start`)
|
||||
- full schema overlap-group timing checks (`group.end >= group.start`)
|
||||
|
||||
## Trim internals
|
||||
|
||||
`internal/trim` is artifact-level projection, not merge reprocessing.
|
||||
|
||||
Core flow:
|
||||
|
||||
1. Parse selector (`internal/trim/selector.go`).
|
||||
2. Parse input artifact and detect schema (`ParseArtifactJSON`).
|
||||
3. Apply keep/remove projection with sequential ID renumbering.
|
||||
4. Recompute overlap groups only for full-schema artifacts.
|
||||
5. Optionally convert output schema when supported.
|
||||
6. Validate output artifact before write.
|
||||
|
||||
Schema-conversion limits:
|
||||
|
||||
- full -> intermediate/minimal supported.
|
||||
- intermediate -> minimal supported.
|
||||
- minimal -> intermediate supported.
|
||||
- intermediate/minimal -> full is rejected.
|
||||
|
||||
Trim invariants:
|
||||
|
||||
- selected IDs must exist in input.
|
||||
- input IDs must be positive, unique, sequential.
|
||||
- retained order follows input transcript order.
|
||||
- output IDs are reassigned to `1..N`.
|
||||
|
||||
## Normalize internals
|
||||
|
||||
`internal/normalize` canonicalizes transcript-like JSON input into a selected
|
||||
public schema.
|
||||
|
||||
Parse layer (`parse.go`):
|
||||
|
||||
- accepts object-with-`segments` or bare segment array
|
||||
- repairs missing timing deterministically
|
||||
- swaps inverted timing
|
||||
- fills missing/blank speaker with `Unknown_Speaker`
|
||||
- drops missing/blank text segments
|
||||
|
||||
Build layer (`build.go`):
|
||||
|
||||
- sorts deterministically by `(start, end, input_index, speaker)`
|
||||
- reassigns output IDs sequentially
|
||||
- builds minimal/intermediate/full output shape
|
||||
- validates selected output schema before write
|
||||
|
||||
Run layer (`normalize.go`):
|
||||
|
||||
- writes output JSON
|
||||
- optionally writes report with `normalize-audit`
|
||||
|
||||
Normalize invariant:
|
||||
|
||||
- report events do not embed transcript text.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- CLI flag semantics belong to `docs/cli.md`.
|
||||
- Runtime config/env surfaces belong to `docs/config.md`.
|
||||
- This doc describes internal conversion/validation behavior only.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
Representative failure classes:
|
||||
|
||||
- malformed or unsupported input JSON shape
|
||||
- schema validation failure for parsed artifact or built output
|
||||
- unsupported schema conversion path (trim)
|
||||
- selector or input-ID consistency errors (trim)
|
||||
- output/report file write failures from command paths
|
||||
|
||||
## Tests to inspect before changes
|
||||
|
||||
- `schema/output_test.go`
|
||||
- `internal/artifact/transcript_test.go`
|
||||
- `internal/trim/selector_test.go`
|
||||
- `internal/trim/artifact_test.go`
|
||||
- `internal/trim/apply_test.go`
|
||||
- `internal/normalize/parse_test.go`
|
||||
- `internal/cli/trim_test.go`
|
||||
- `internal/cli/normalize_test.go`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Public artifacts are validated through `schema` before acceptance.
|
||||
- Segment IDs in emitted artifacts are sequential and deterministic.
|
||||
- Internal-only fields are not emitted in minimal/intermediate contracts.
|
||||
- Trim and normalize stay artifact-level and do not execute merge modules.
|
||||
119
docs/internal/modules.md
Normal file
119
docs/internal/modules.md
Normal file
@@ -0,0 +1,119 @@
|
||||
# Built-In Modules
|
||||
|
||||
## Purpose
|
||||
|
||||
Describes implemented built-in module behavior and boundaries in
|
||||
`internal/builtin`.
|
||||
|
||||
## Implemented module set
|
||||
|
||||
Input reader:
|
||||
|
||||
- `json-files`
|
||||
|
||||
Preprocessing:
|
||||
|
||||
- `validate-raw`
|
||||
- `normalize-speakers`
|
||||
- `trim-text`
|
||||
|
||||
Merger:
|
||||
|
||||
- `chronological-merge`
|
||||
|
||||
Postprocessing:
|
||||
|
||||
- `detect-overlaps`
|
||||
- `resolve-overlaps`
|
||||
- `backchannel`
|
||||
- `filler`
|
||||
- `resolve-danglers`
|
||||
- `coalesce`
|
||||
- `autocorrect`
|
||||
- `assign-ids`
|
||||
- `validate-output`
|
||||
|
||||
Output writer:
|
||||
|
||||
- `json`
|
||||
|
||||
## Inputs, outputs, and side effects
|
||||
|
||||
- `json-files`: reads JSON files from `cfg.InputFiles`, parses supported
|
||||
segment/word fields, emits warnings for untimed words.
|
||||
- `validate-raw`: validates raw source/timing invariants.
|
||||
- `normalize-speakers`: converts raw transcripts to canonical segments,
|
||||
optionally resolving speakers from `cfg.SpeakersFile`.
|
||||
- `trim-text`: trims canonical segment text whitespace.
|
||||
- `chronological-merge`: flattens canonical segments and applies deterministic
|
||||
sort (`model.SegmentLess`).
|
||||
- `detect-overlaps`: annotates overlap groups.
|
||||
- `resolve-overlaps`: rewrites overlap groups using timed words and thresholds.
|
||||
- `backchannel`/`filler`: classify short utterances using duration thresholds.
|
||||
- `resolve-danglers`: merges dangling derived fragments.
|
||||
- `coalesce`: merges adjacent same-speaker segments within configured gap.
|
||||
- `autocorrect`: applies YAML replacement rules when configured.
|
||||
- `assign-ids`: assigns final sequential IDs.
|
||||
- `validate-output`: validates selected public artifact shape.
|
||||
- `json`: writes artifact JSON to `cfg.OutputFile`.
|
||||
|
||||
Filesystem side effects are limited to:
|
||||
|
||||
- reading configured input/YAML files
|
||||
- writing configured output artifact
|
||||
|
||||
## Config fields used
|
||||
|
||||
Primary module inputs from `config.Config`:
|
||||
|
||||
- file paths: `InputFiles`, `SpeakersFile`, `AutocorrectFile`, `OutputFile`
|
||||
- schema/modules: `OutputSchema`, `OutputModules`
|
||||
- overlap/coalesce thresholds: `OverlapWordRunGap`,
|
||||
`WordRunReorderWindow`, `CoalesceGap`
|
||||
- category thresholds: `BackchannelMaxDuration`, `FillerMaxDuration`
|
||||
|
||||
## Ordering constraints
|
||||
|
||||
- Preprocessing must satisfy state contracts from `raw` to `canonical`.
|
||||
- Order-sensitive transforms should run before `assign-ids`.
|
||||
- `validate-output` should run after final ID assignment and output-shape
|
||||
mutations.
|
||||
- Default configuration includes a second `detect-overlaps` pass after
|
||||
transformations.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Modules implement behavior; CLI/config parsing remains outside modules.
|
||||
- Modules communicate through explicit model contracts and report events.
|
||||
- Output modules operate on final artifacts and do not re-run transform logic.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
Representative failures:
|
||||
|
||||
- invalid input JSON shape or typed field errors (`json-files`)
|
||||
- invalid YAML or unmatched speaker map entries
|
||||
- unknown module names during registry resolution
|
||||
- invalid ordering/state transitions in preprocessing chain
|
||||
- validation failure in `validate-output`
|
||||
- output write failure in `json` writer
|
||||
|
||||
## Tests to inspect before changes
|
||||
|
||||
- `internal/builtin/preprocess_test.go`
|
||||
- `internal/builtin/postprocess_test.go`
|
||||
- `internal/overlap/resolve_test.go`
|
||||
- `internal/overlap/detect_test.go`
|
||||
- `internal/coalesce/coalesce_test.go`
|
||||
- `internal/danglers/danglers_test.go`
|
||||
- `internal/backchannel/backchannel_test.go`
|
||||
- `internal/filler/filler_test.go`
|
||||
- `internal/autocorrect/autocorrect_test.go`
|
||||
- `internal/cli/merge_test.go`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Modules are selected by canonical name through the registry.
|
||||
- Execution is sequential and deterministic for a fixed configuration.
|
||||
- `assign-ids` defines final public segment IDs.
|
||||
- `validate-output` enforces public artifact contracts through `schema`.
|
||||
102
docs/internal/pipeline.md
Normal file
102
docs/internal/pipeline.md
Normal file
@@ -0,0 +1,102 @@
|
||||
# Pipeline Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
Describes implemented merge pipeline orchestration in `internal/pipeline`.
|
||||
|
||||
## Inputs and outputs
|
||||
|
||||
Input:
|
||||
|
||||
- `config.Config`
|
||||
- registry-resolved modules from `internal/builtin`
|
||||
|
||||
Output:
|
||||
|
||||
- selected public artifact written by output writer modules
|
||||
- optional report JSON when `cfg.ReportFile` is set
|
||||
|
||||
## Stage contracts
|
||||
|
||||
The runner executes these contracts in order:
|
||||
|
||||
1. `InputReader`: external inputs -> `[]model.RawTranscript`
|
||||
2. `Preprocessor`: `PreprocessState` transformations (`raw` -> `canonical`)
|
||||
3. `Merger`: canonical transcripts -> `model.MergedTranscript`
|
||||
4. `Postprocessor`: merged transcript transformations
|
||||
5. `OutputWriter`: serialized artifact writes
|
||||
|
||||
`PreprocessState` must end in `StateCanonical` before merge.
|
||||
|
||||
## Registry resolution
|
||||
|
||||
`resolvePlan` maps configured names to modules:
|
||||
|
||||
- input reader: `cfg.InputReader`
|
||||
- preprocessors: `cfg.PreprocessingModules`
|
||||
- postprocessors: `cfg.PostprocessingModules`
|
||||
- output writers: `cfg.OutputModules`
|
||||
- merger: single registered merger
|
||||
|
||||
Unknown names fail fast with contextual errors.
|
||||
|
||||
## Execution order and reporting
|
||||
|
||||
- Modules run sequentially in configured order.
|
||||
- Events returned by modules are appended in execution order.
|
||||
- Report metadata includes input reader, input files, and module lists.
|
||||
- Output writer events are appended before optional report write.
|
||||
|
||||
## Config fields used
|
||||
|
||||
Runner-level fields:
|
||||
|
||||
- `InputReader`
|
||||
- `InputFiles`
|
||||
- `PreprocessingModules`
|
||||
- `PostprocessingModules`
|
||||
- `OutputModules`
|
||||
- `OutputSchema` (via `artifact.SelectedFromMerged`)
|
||||
- `ReportFile`
|
||||
|
||||
Module-specific settings are consumed inside builtin modules (for example
|
||||
coalesce gap and overlap thresholds).
|
||||
|
||||
## Adapters used
|
||||
|
||||
- Input adapters: registered `InputReader` implementations (default `json-files`).
|
||||
- Output adapters: registered `OutputWriter` implementations (default `json`).
|
||||
- Report adapter: `report.WriteJSON` when `cfg.ReportFile` is provided.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Pipeline does not parse CLI flags.
|
||||
- Pipeline does not normalize raw CLI strings.
|
||||
- Pipeline delegates conversion to public output contracts to `internal/artifact`.
|
||||
- Artifact-level commands `trim` and `normalize` are outside this pipeline.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
Pipeline returns errors from:
|
||||
|
||||
- registry resolution (unknown modules, missing merger)
|
||||
- invalid preprocessing state transitions
|
||||
- module read/process/merge/write failures
|
||||
- optional report write failure
|
||||
|
||||
No retry/resume state is stored.
|
||||
|
||||
## Tests to inspect before changes
|
||||
|
||||
- `internal/pipeline/runner_test.go`
|
||||
- `internal/builtin/preprocess_test.go`
|
||||
- `internal/builtin/postprocess_test.go`
|
||||
- `internal/cli/merge_test.go`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Sequential deterministic execution order.
|
||||
- Preprocessing state must type-check from `raw` to `canonical`.
|
||||
- Module selection is explicit by canonical names.
|
||||
- Report event order reflects actual execution order.
|
||||
- Output artifact selection is schema-driven via `internal/artifact`.
|
||||
110
docs/policy/development.md
Normal file
110
docs/policy/development.md
Normal file
@@ -0,0 +1,110 @@
|
||||
# Development Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines contributor workflow for maintainers and coding agents.
|
||||
It complements [architecture policy](architecture.md) and
|
||||
[documentation policy](documentation.md).
|
||||
|
||||
## Repository layout
|
||||
|
||||
- `cmd/seriatim/`: process entrypoint.
|
||||
- `internal/cli/`: Cobra commands and flag wiring.
|
||||
- `internal/config/`: option normalization and validation.
|
||||
- `internal/pipeline/`: orchestration interfaces, registry, runner.
|
||||
- `internal/builtin/`: implemented input/pre/post/output modules and merger.
|
||||
- `internal/artifact/`: conversion from internal merged model to public shapes.
|
||||
- `internal/trim/`: artifact-level trim logic.
|
||||
- `internal/normalize/`: artifact-level normalize parsing/building.
|
||||
- `internal/*` domain packages: overlap, coalesce, danglers, filler,
|
||||
backchannel, speaker, autocorrect, report, model.
|
||||
- `schema/`: public structs plus embedded JSON Schemas and validation.
|
||||
- `docs/`: policy, user docs, roadmap, and internal docs.
|
||||
|
||||
## Local checks
|
||||
|
||||
Primary repository check:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Useful manual checks for CLI-facing changes:
|
||||
|
||||
```sh
|
||||
go run ./cmd/seriatim --help
|
||||
go run ./cmd/seriatim merge --help
|
||||
go run ./cmd/seriatim trim --help
|
||||
go run ./cmd/seriatim normalize --help
|
||||
```
|
||||
|
||||
Current toolchain note:
|
||||
|
||||
- There is no Makefile.
|
||||
- There is no taskfile.
|
||||
- There is no committed linter configuration.
|
||||
- There is no automated documentation checker.
|
||||
|
||||
## Coding conventions
|
||||
|
||||
- Keep core behavior deterministic for identical inputs/config/version.
|
||||
- Keep CLI command functions thin: parse flags, construct config, delegate.
|
||||
- Keep validation in `internal/config` and package-specific validators.
|
||||
- Return errors from deep logic; do not print inside internal packages.
|
||||
- Preserve clear package boundaries between adapters and domain transforms.
|
||||
|
||||
## Dependency policy
|
||||
|
||||
Prefer the Go standard library first.
|
||||
|
||||
Third-party dependencies should stay narrow and justified. Current direct
|
||||
runtime dependencies are:
|
||||
|
||||
- `github.com/spf13/cobra` for CLI structure.
|
||||
- `gopkg.in/yaml.v3` for YAML rule files.
|
||||
- `github.com/santhosh-tekuri/jsonschema/v6` for public schema validation.
|
||||
|
||||
## Adding CLI flags
|
||||
|
||||
1. Add the flag in the relevant `internal/cli/*.go` command.
|
||||
2. Thread the raw value through `config.*Options`.
|
||||
3. Add normalization/validation in `internal/config/config.go`.
|
||||
4. Update or add CLI/config tests.
|
||||
5. Update canonical docs (`docs/cli.md`, `docs/config.md`) if user-visible.
|
||||
|
||||
## Adding config fields or environment variables
|
||||
|
||||
1. Add field(s) to the relevant config struct(s).
|
||||
2. Parse and validate in `internal/config/config.go`.
|
||||
3. Add tests in `internal/config/config_test.go`.
|
||||
4. Thread validated values into consuming modules.
|
||||
5. Update `docs/config.md` and related docs.
|
||||
|
||||
## Adding modules or pipeline behavior
|
||||
|
||||
1. Implement the module in the appropriate package (often `internal/builtin`).
|
||||
2. Expose a stable module name via `Name()`.
|
||||
3. Register it in `internal/builtin/registry.go`.
|
||||
4. Ensure preprocessing modules declare correct `Requires()`/`Produces()`
|
||||
states.
|
||||
5. Add/adjust tests in module packages and `internal/cli/merge_test.go`.
|
||||
6. Document internal behavior changes in `docs/internal/`.
|
||||
|
||||
## Schema and artifact changes
|
||||
|
||||
1. Update public structs and validation logic in `schema/`.
|
||||
2. Update embedded JSON Schema files (`schema/*.schema.json`) if contract
|
||||
changes.
|
||||
3. Update conversion behavior in `internal/artifact`, `internal/trim`, and/or
|
||||
`internal/normalize` as needed.
|
||||
4. Add tests in `schema/`, `internal/artifact/`, `internal/trim/`,
|
||||
`internal/normalize/`, and CLI tests.
|
||||
5. Update user and internal docs that reference output contracts.
|
||||
|
||||
## Documentation expectations
|
||||
|
||||
- Outside `docs/roadmap/`, document only implemented behavior.
|
||||
- Keep canonical homes: CLI in `docs/cli.md`, config in `docs/config.md`,
|
||||
operations in `docs/operations.md`, troubleshooting in
|
||||
`docs/troubleshooting.md`, internals in `docs/internal/`.
|
||||
- When behavior changes, update docs in the same change.
|
||||
Reference in New Issue
Block a user