Add development policy and internal implementation documentation
This commit is contained in:
140
docs/internal/artifacts.md
Normal file
140
docs/internal/artifacts.md
Normal file
@@ -0,0 +1,140 @@
|
||||
# Artifact Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
Describes public artifact conversion and validation internals for merge output,
|
||||
trim, and normalize.
|
||||
|
||||
## Artifact contracts
|
||||
|
||||
Public contracts live in `schema/`:
|
||||
|
||||
- full: `schema.Transcript`
|
||||
- intermediate: `schema.IntermediateTranscript`
|
||||
- minimal: `schema.MinimalTranscript`
|
||||
|
||||
Machine-readable schemas:
|
||||
|
||||
- `schema/full-output.schema.json`
|
||||
- `schema/intermediate-output.schema.json`
|
||||
- `schema/minimal-output.schema.json`
|
||||
|
||||
## Schema selection
|
||||
|
||||
Merge pipeline conversion uses `internal/artifact.SelectedFromMerged`:
|
||||
|
||||
- `seriatim-full` -> `artifact.FromMerged`
|
||||
- `seriatim-intermediate` -> `artifact.IntermediateFromMerged`
|
||||
- `seriatim-minimal` -> `artifact.MinimalFromMerged`
|
||||
|
||||
Unknown/empty selection falls back to intermediate conversion.
|
||||
|
||||
## Merge conversion behavior
|
||||
|
||||
`internal/artifact` converts `model.MergedTranscript` to public contracts:
|
||||
|
||||
- full schema preserves source/provenance, overlap groups, and metadata module
|
||||
lists.
|
||||
- intermediate schema emits segment timing/text/speaker with optional
|
||||
categories and compact metadata.
|
||||
- minimal schema emits compact segment timing/text/speaker and compact
|
||||
metadata.
|
||||
|
||||
## Validation behavior
|
||||
|
||||
`schema/output.go` validates both structure and semantics:
|
||||
|
||||
- embedded JSON Schema validation via `jsonschema/v6`
|
||||
- semantic checks for sequential segment IDs starting at `1`
|
||||
- semantic checks for non-inverted segment timing (`end >= start`)
|
||||
- full schema overlap-group timing checks (`group.end >= group.start`)
|
||||
|
||||
## Trim internals
|
||||
|
||||
`internal/trim` is artifact-level projection, not merge reprocessing.
|
||||
|
||||
Core flow:
|
||||
|
||||
1. Parse selector (`internal/trim/selector.go`).
|
||||
2. Parse input artifact and detect schema (`ParseArtifactJSON`).
|
||||
3. Apply keep/remove projection with sequential ID renumbering.
|
||||
4. Recompute overlap groups only for full-schema artifacts.
|
||||
5. Optionally convert output schema when supported.
|
||||
6. Validate output artifact before write.
|
||||
|
||||
Schema-conversion limits:
|
||||
|
||||
- full -> intermediate/minimal supported.
|
||||
- intermediate -> minimal supported.
|
||||
- minimal -> intermediate supported.
|
||||
- intermediate/minimal -> full is rejected.
|
||||
|
||||
Trim invariants:
|
||||
|
||||
- selected IDs must exist in input.
|
||||
- input IDs must be positive, unique, sequential.
|
||||
- retained order follows input transcript order.
|
||||
- output IDs are reassigned to `1..N`.
|
||||
|
||||
## Normalize internals
|
||||
|
||||
`internal/normalize` canonicalizes transcript-like JSON input into a selected
|
||||
public schema.
|
||||
|
||||
Parse layer (`parse.go`):
|
||||
|
||||
- accepts object-with-`segments` or bare segment array
|
||||
- repairs missing timing deterministically
|
||||
- swaps inverted timing
|
||||
- fills missing/blank speaker with `Unknown_Speaker`
|
||||
- drops missing/blank text segments
|
||||
|
||||
Build layer (`build.go`):
|
||||
|
||||
- sorts deterministically by `(start, end, input_index, speaker)`
|
||||
- reassigns output IDs sequentially
|
||||
- builds minimal/intermediate/full output shape
|
||||
- validates selected output schema before write
|
||||
|
||||
Run layer (`normalize.go`):
|
||||
|
||||
- writes output JSON
|
||||
- optionally writes report with `normalize-audit`
|
||||
|
||||
Normalize invariant:
|
||||
|
||||
- report events do not embed transcript text.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- CLI flag semantics belong to `docs/cli.md`.
|
||||
- Runtime config/env surfaces belong to `docs/config.md`.
|
||||
- This doc describes internal conversion/validation behavior only.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
Representative failure classes:
|
||||
|
||||
- malformed or unsupported input JSON shape
|
||||
- schema validation failure for parsed artifact or built output
|
||||
- unsupported schema conversion path (trim)
|
||||
- selector or input-ID consistency errors (trim)
|
||||
- output/report file write failures from command paths
|
||||
|
||||
## Tests to inspect before changes
|
||||
|
||||
- `schema/output_test.go`
|
||||
- `internal/artifact/transcript_test.go`
|
||||
- `internal/trim/selector_test.go`
|
||||
- `internal/trim/artifact_test.go`
|
||||
- `internal/trim/apply_test.go`
|
||||
- `internal/normalize/parse_test.go`
|
||||
- `internal/cli/trim_test.go`
|
||||
- `internal/cli/normalize_test.go`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Public artifacts are validated through `schema` before acceptance.
|
||||
- Segment IDs in emitted artifacts are sequential and deterministic.
|
||||
- Internal-only fields are not emitted in minimal/intermediate contracts.
|
||||
- Trim and normalize stay artifact-level and do not execute merge modules.
|
||||
119
docs/internal/modules.md
Normal file
119
docs/internal/modules.md
Normal file
@@ -0,0 +1,119 @@
|
||||
# Built-In Modules
|
||||
|
||||
## Purpose
|
||||
|
||||
Describes implemented built-in module behavior and boundaries in
|
||||
`internal/builtin`.
|
||||
|
||||
## Implemented module set
|
||||
|
||||
Input reader:
|
||||
|
||||
- `json-files`
|
||||
|
||||
Preprocessing:
|
||||
|
||||
- `validate-raw`
|
||||
- `normalize-speakers`
|
||||
- `trim-text`
|
||||
|
||||
Merger:
|
||||
|
||||
- `chronological-merge`
|
||||
|
||||
Postprocessing:
|
||||
|
||||
- `detect-overlaps`
|
||||
- `resolve-overlaps`
|
||||
- `backchannel`
|
||||
- `filler`
|
||||
- `resolve-danglers`
|
||||
- `coalesce`
|
||||
- `autocorrect`
|
||||
- `assign-ids`
|
||||
- `validate-output`
|
||||
|
||||
Output writer:
|
||||
|
||||
- `json`
|
||||
|
||||
## Inputs, outputs, and side effects
|
||||
|
||||
- `json-files`: reads JSON files from `cfg.InputFiles`, parses supported
|
||||
segment/word fields, emits warnings for untimed words.
|
||||
- `validate-raw`: validates raw source/timing invariants.
|
||||
- `normalize-speakers`: converts raw transcripts to canonical segments,
|
||||
optionally resolving speakers from `cfg.SpeakersFile`.
|
||||
- `trim-text`: trims canonical segment text whitespace.
|
||||
- `chronological-merge`: flattens canonical segments and applies deterministic
|
||||
sort (`model.SegmentLess`).
|
||||
- `detect-overlaps`: annotates overlap groups.
|
||||
- `resolve-overlaps`: rewrites overlap groups using timed words and thresholds.
|
||||
- `backchannel`/`filler`: classify short utterances using duration thresholds.
|
||||
- `resolve-danglers`: merges dangling derived fragments.
|
||||
- `coalesce`: merges adjacent same-speaker segments within configured gap.
|
||||
- `autocorrect`: applies YAML replacement rules when configured.
|
||||
- `assign-ids`: assigns final sequential IDs.
|
||||
- `validate-output`: validates selected public artifact shape.
|
||||
- `json`: writes artifact JSON to `cfg.OutputFile`.
|
||||
|
||||
Filesystem side effects are limited to:
|
||||
|
||||
- reading configured input/YAML files
|
||||
- writing configured output artifact
|
||||
|
||||
## Config fields used
|
||||
|
||||
Primary module inputs from `config.Config`:
|
||||
|
||||
- file paths: `InputFiles`, `SpeakersFile`, `AutocorrectFile`, `OutputFile`
|
||||
- schema/modules: `OutputSchema`, `OutputModules`
|
||||
- overlap/coalesce thresholds: `OverlapWordRunGap`,
|
||||
`WordRunReorderWindow`, `CoalesceGap`
|
||||
- category thresholds: `BackchannelMaxDuration`, `FillerMaxDuration`
|
||||
|
||||
## Ordering constraints
|
||||
|
||||
- Preprocessing must satisfy state contracts from `raw` to `canonical`.
|
||||
- Order-sensitive transforms should run before `assign-ids`.
|
||||
- `validate-output` should run after final ID assignment and output-shape
|
||||
mutations.
|
||||
- Default configuration includes a second `detect-overlaps` pass after
|
||||
transformations.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Modules implement behavior; CLI/config parsing remains outside modules.
|
||||
- Modules communicate through explicit model contracts and report events.
|
||||
- Output modules operate on final artifacts and do not re-run transform logic.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
Representative failures:
|
||||
|
||||
- invalid input JSON shape or typed field errors (`json-files`)
|
||||
- invalid YAML or unmatched speaker map entries
|
||||
- unknown module names during registry resolution
|
||||
- invalid ordering/state transitions in preprocessing chain
|
||||
- validation failure in `validate-output`
|
||||
- output write failure in `json` writer
|
||||
|
||||
## Tests to inspect before changes
|
||||
|
||||
- `internal/builtin/preprocess_test.go`
|
||||
- `internal/builtin/postprocess_test.go`
|
||||
- `internal/overlap/resolve_test.go`
|
||||
- `internal/overlap/detect_test.go`
|
||||
- `internal/coalesce/coalesce_test.go`
|
||||
- `internal/danglers/danglers_test.go`
|
||||
- `internal/backchannel/backchannel_test.go`
|
||||
- `internal/filler/filler_test.go`
|
||||
- `internal/autocorrect/autocorrect_test.go`
|
||||
- `internal/cli/merge_test.go`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Modules are selected by canonical name through the registry.
|
||||
- Execution is sequential and deterministic for a fixed configuration.
|
||||
- `assign-ids` defines final public segment IDs.
|
||||
- `validate-output` enforces public artifact contracts through `schema`.
|
||||
102
docs/internal/pipeline.md
Normal file
102
docs/internal/pipeline.md
Normal file
@@ -0,0 +1,102 @@
|
||||
# Pipeline Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
Describes implemented merge pipeline orchestration in `internal/pipeline`.
|
||||
|
||||
## Inputs and outputs
|
||||
|
||||
Input:
|
||||
|
||||
- `config.Config`
|
||||
- registry-resolved modules from `internal/builtin`
|
||||
|
||||
Output:
|
||||
|
||||
- selected public artifact written by output writer modules
|
||||
- optional report JSON when `cfg.ReportFile` is set
|
||||
|
||||
## Stage contracts
|
||||
|
||||
The runner executes these contracts in order:
|
||||
|
||||
1. `InputReader`: external inputs -> `[]model.RawTranscript`
|
||||
2. `Preprocessor`: `PreprocessState` transformations (`raw` -> `canonical`)
|
||||
3. `Merger`: canonical transcripts -> `model.MergedTranscript`
|
||||
4. `Postprocessor`: merged transcript transformations
|
||||
5. `OutputWriter`: serialized artifact writes
|
||||
|
||||
`PreprocessState` must end in `StateCanonical` before merge.
|
||||
|
||||
## Registry resolution
|
||||
|
||||
`resolvePlan` maps configured names to modules:
|
||||
|
||||
- input reader: `cfg.InputReader`
|
||||
- preprocessors: `cfg.PreprocessingModules`
|
||||
- postprocessors: `cfg.PostprocessingModules`
|
||||
- output writers: `cfg.OutputModules`
|
||||
- merger: single registered merger
|
||||
|
||||
Unknown names fail fast with contextual errors.
|
||||
|
||||
## Execution order and reporting
|
||||
|
||||
- Modules run sequentially in configured order.
|
||||
- Events returned by modules are appended in execution order.
|
||||
- Report metadata includes input reader, input files, and module lists.
|
||||
- Output writer events are appended before optional report write.
|
||||
|
||||
## Config fields used
|
||||
|
||||
Runner-level fields:
|
||||
|
||||
- `InputReader`
|
||||
- `InputFiles`
|
||||
- `PreprocessingModules`
|
||||
- `PostprocessingModules`
|
||||
- `OutputModules`
|
||||
- `OutputSchema` (via `artifact.SelectedFromMerged`)
|
||||
- `ReportFile`
|
||||
|
||||
Module-specific settings are consumed inside builtin modules (for example
|
||||
coalesce gap and overlap thresholds).
|
||||
|
||||
## Adapters used
|
||||
|
||||
- Input adapters: registered `InputReader` implementations (default `json-files`).
|
||||
- Output adapters: registered `OutputWriter` implementations (default `json`).
|
||||
- Report adapter: `report.WriteJSON` when `cfg.ReportFile` is provided.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Pipeline does not parse CLI flags.
|
||||
- Pipeline does not normalize raw CLI strings.
|
||||
- Pipeline delegates conversion to public output contracts to `internal/artifact`.
|
||||
- Artifact-level commands `trim` and `normalize` are outside this pipeline.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
Pipeline returns errors from:
|
||||
|
||||
- registry resolution (unknown modules, missing merger)
|
||||
- invalid preprocessing state transitions
|
||||
- module read/process/merge/write failures
|
||||
- optional report write failure
|
||||
|
||||
No retry/resume state is stored.
|
||||
|
||||
## Tests to inspect before changes
|
||||
|
||||
- `internal/pipeline/runner_test.go`
|
||||
- `internal/builtin/preprocess_test.go`
|
||||
- `internal/builtin/postprocess_test.go`
|
||||
- `internal/cli/merge_test.go`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Sequential deterministic execution order.
|
||||
- Preprocessing state must type-check from `raw` to `canonical`.
|
||||
- Module selection is explicit by canonical names.
|
||||
- Report event order reflects actual execution order.
|
||||
- Output artifact selection is schema-driven via `internal/artifact`.
|
||||
Reference in New Issue
Block a user