# Documentation Roadmap ## Purpose This roadmap defines the work required to bring seriatim's documentation into compliance with `docs/policy/documentation.md` and the current implementation. It is grounded in the repository as it exists now: the Go CLI, config loading, pipeline modules, artifact commands, schemas, reports, samples, and tests. Outside `docs/roadmap/`, documentation must describe only implemented behavior. Planned, future, deprecated, experimental, or unimplemented work must remain in roadmap documents until the code exists. ## Repository Documentation Inventory - `README.md` - keep and rewrite. It currently mixes project orientation, quickstart, full CLI reference, config/env reference, file formats, module internals, limitations, and release build notes. Policy says README should be concise and link to canonical docs. - `docs/policy/documentation.md` - keep and lightly update only if the policy itself changes. It is the controlling documentation layout and maintenance policy. - `docs/policy/architecture.md` - keep and lightly update as implementation changes. It is the canonical development architecture policy. - Root `architecture.md` - delete after salvage, or move only truly roadmap material into `docs/roadmap/`. It is in the wrong canonical home and contains future-oriented and aspirational claims. - `docs/roadmap/documentation.md` - create new. This file is the planning artifact for the documentation migration. - `samples/` - split or move after audit. It contains sample raw transcripts, merged artifacts, reports, `speakers.yml`, and `autocorrect.yml`, but copyable examples belong under `examples/`. The raw sample data is large and should be reviewed for privacy and maintainability before linking from docs. - `schema/*.schema.json` - keep. These are public output contracts and should be linked from documentation instead of duplicated in full. - Missing canonical docs - create `docs/cli.md`, `docs/config.md`, `docs/operations.md`, `docs/policy/development.md`, `docs/internal/`, and likely `docs/troubleshooting.md`, `docs/integrations/`, and `examples/`. ## Policy Compliance Assessment Required documents missing for seriatim's current shape as a modular, staged, CLI/config-driven project: - `docs/cli.md` - `docs/config.md` - `docs/operations.md` - `docs/internal/` - `docs/policy/development.md` Recommended documents and directories missing: - `docs/troubleshooting.md` - maintained copyable examples under `examples/` - concise integration notes under `docs/integrations/` Existing compliance issues: - `README.md` is too broad for its canonical scope. It should keep project purpose, quickstart, and links, then delegate CLI, config, operations, internals, and schema details. - Root `architecture.md` is stale and in the wrong home. It includes future input methods and formats, future output formats, dynamic plugin speculation, an LLM non-goal, interface sketches that diverge from code, and other development-policy content now covered by `docs/policy/architecture.md`. - Non-roadmap docs should not carry forward claims about future defaults, future formats, unimplemented plugin systems, or unimplemented alternate input/output methods. - Historical or deprecated wording, such as the old speaker map format, should move out of the README unless it is still needed in troubleshooting or a narrow migration note. - There is no `examples/` directory. `samples/` exists but is not the canonical examples home and should not be treated as copyable public examples without a privacy and size audit. - Links need verification after migration: README should link to all new canonical docs, docs should link to schema files and maintained examples, and no doc should link to the deleted root `architecture.md`. ## Target Documentation Set ### `README.md` - Audience: users, administrators, and operators. - Purpose: project orientation and shortest useful quickstart. - Canonical scope: concise project purpose, elevator pitch, one minimal command, and links to targeted docs. - Recommended outline: project description; shortest merge command; command summary; links to CLI, config, operations, architecture, development, schemas, examples, and troubleshooting. - Source of truth: current `README.md`, `internal/cli`, `internal/config`, `cmd/seriatim/main.go`, and CLI tests. - Acceptance criteria: no full flag tables, no full config reference, no module manual, no future-feature claims, and all links resolve. ### `docs/cli.md` - Audience: users, administrators, and operators. - Purpose: canonical CLI reference and workflows. - Canonical scope: shortest useful command, command overview, complete flag reference, common workflows, diagnostics and report flags. - Recommended outline: shortest useful command; global flags; `merge`; `trim`; `normalize`; common workflows; exit/error behavior; links to config, operations, examples, and schemas. - Source of truth: `internal/cli/root.go`, `internal/cli/merge.go`, `internal/cli/trim.go`, `internal/cli/normalize.go`, `internal/config`, and `internal/cli/*_test.go`. - Acceptance criteria: every documented flag, default, and required/mutually exclusive rule matches code; package internals are linked rather than explained in depth. ### `docs/config.md` - Audience: administrators, operators, and advanced users. - Purpose: canonical runtime configuration reference. - Canonical scope: environment variables, default module lists, output schema selection, `speakers.yml`, `autocorrect.yml`, path validation, and precedence. - Recommended outline: config surfaces; output schema precedence; merge module defaults; environment variables; speaker map YAML; autocorrect YAML; path and validation rules; links to examples. - Source of truth: `internal/config/config.go`, `internal/speaker/map.go`, `internal/autocorrect/autocorrect.go`, `internal/config/config_test.go`, `internal/speaker/map_test.go`, and `internal/autocorrect/autocorrect_test.go`. - Acceptance criteria: all config fields and `SERIATIM_*` env vars match code; unsupported config files or unimplemented formats are not described. ### `docs/operations.md` - Audience: administrators and operators. - Purpose: operational behavior for running commands safely. - Canonical scope: file workflow, filesystem layout expectations, output and report files, retry behavior, cleanup, validation failures, and operational caveats. - Recommended outline: normal workflow; input/output/report files; no durable state; failure and retry behavior; reports and diagnostics; cleanup; privacy considerations for transcript artifacts. - Source of truth: `cmd/seriatim/main.go`, `internal/cli`, `internal/config`, `internal/report`, `internal/builtin/output.go`, `internal/normalize`, and trim/merge/normalize CLI tests. - Acceptance criteria: clearly states there is no daemon, database, resume state, remote storage, or background job state; does not invent recovery workflows. ### `docs/policy/development.md` - Audience: developers and coding agents. - Purpose: contributor workflow and change guidance. - Canonical scope: repository layout, build/test commands, coding conventions, dependency policy, adding flags/config fields/modules/docs/examples. - Recommended outline: repo layout; local checks; coding conventions; adding CLI flags; adding config/env vars; adding modules/stages; schema changes; examples and documentation updates. - Source of truth: `docs/policy/documentation.md`, `docs/policy/architecture.md`, `go.mod`, package layout, and test layout. - Acceptance criteria: includes `go test ./...`; states there is no current Makefile, taskfile, linter config, or automated doc checker; aligns with the architecture policy. ### `docs/internal/pipeline.md` - Audience: developers and coding agents. - Purpose: implemented merge pipeline internals. - Canonical scope: registry, stage interfaces, preprocessing state transitions, module order, report event accumulation, final output/report writing. - Recommended outline: purpose; inputs and outputs; stage contracts; registry resolution; execution order; config fields used; adapters; failure behavior; tests; invariants. - Source of truth: `internal/pipeline`, `internal/builtin`, `internal/model`, `internal/report`, `internal/pipeline/runner_test.go`, `internal/builtin/*_test.go`, and `internal/cli/merge_test.go`. - Acceptance criteria: describes only implemented sequential execution; does not document concurrency, plugins, or future formats. ### `docs/internal/artifacts.md` - Audience: developers and coding agents. - Purpose: public artifact conversion and validation internals. - Canonical scope: schema structs, embedded JSON Schemas, conversion from merged model, trim/normalize artifact handling, and output validation. - Recommended outline: artifact contracts; schema selection; conversion; validation; trim projection; normalize canonicalization; tests; invariants. - Source of truth: `schema`, `internal/artifact`, `internal/trim`, `internal/normalize`, and related tests. - Acceptance criteria: links to `schema/*.schema.json`; does not duplicate full schemas or describe unavailable output formats. ### `docs/internal/modules.md` - Audience: developers and coding agents. - Purpose: implemented built-in module behavior and boundaries. - Canonical scope: `json-files`, preprocessing modules, chronological merge, postprocessing modules, and JSON output writer. - Recommended outline: module list; inputs/outputs; config fields used; allowed side effects; ordering constraints; failure behavior; tests; invariants. - Source of truth: `internal/builtin`, `internal/overlap`, `internal/coalesce`, `internal/danglers`, `internal/backchannel`, `internal/filler`, `internal/autocorrect`, and package tests. - Acceptance criteria: avoids full CLI/config duplication; identifies order-sensitive transforms that must run before `assign-ids`. ### `docs/troubleshooting.md` - Audience: users, administrators, and operators. - Purpose: common failure symptoms and safe fixes. - Canonical scope: implemented validation and runtime failures observed in error paths and tests. - Recommended outline: invalid JSON/input shape; missing required flags; invalid output parent directory; invalid speaker/autocorrect YAML; unknown module; invalid output schema; invalid trim selector; schema validation failure; report write failure. - Source of truth: `internal/config`, `internal/cli/*_test.go`, `internal/trim/*_test.go`, `internal/normalize/*_test.go`, `internal/speaker/*_test.go`, and `internal/autocorrect/*_test.go`. - Acceptance criteria: each entry has symptom, likely cause, inspection step, safe fix, and link; no speculative failure modes. ### `docs/integrations/whisperx-json.md` - Audience: developers and coding agents. - Purpose: external input JSON contract used by `merge`. - Canonical scope: the supported WhisperX-like subset only. - Recommended outline: top-level shape; required segment fields; optional word timing fields; validation/failure behavior; how word timing affects overlap resolution; links to CLI and examples. - Source of truth: `internal/builtin/input.go`, merge CLI tests, and README input-format material. - Acceptance criteria: does not attempt to document full WhisperX behavior or unsupported input formats. ### `docs/integrations/output-schemas.md` - Audience: developers, coding agents, and artifact consumers. - Purpose: orientation to public JSON output contracts. - Canonical scope: minimal/intermediate/full schema roles and links to schema files. - Recommended outline: schema selection; minimal; intermediate; full; semantic invariants; validation APIs; links to `schema/*.schema.json`. - Source of truth: `schema/output.go`, `schema/*.schema.json`, `schema/output_test.go`, and `internal/artifact`. - Acceptance criteria: links to machine-readable schemas instead of copying them in full. ### `examples/` - Audience: users, administrators, operators, developers, and coding agents. - Purpose: maintained copyable examples. - Canonical scope: small synthetic inputs and config files for implemented commands only. - Source of truth: examples created during the documentation migration and validated through actual command invocations. - Acceptance criteria: examples are valid, free of secrets/private transcript data, and linked from README, CLI, config, and operations docs. ## File-by-File Rewrite Guidance ### README Cover what seriatim is, the shortest useful `merge` command, a brief command summary, and links to canonical docs. Avoid full flag tables, config/env reference, module internals, schema examples, troubleshooting details, future formats, or release-history narrative. Inspect `internal/cli`, `internal/config`, and CLI tests before updating commands. ### CLI Reference Document actual `merge`, `trim`, and `normalize` flags from `internal/cli`. Include required flags, defaults, mutually exclusive selector rules, schema selection, report flags, and common workflows. Link to `docs/config.md` for environment variables and YAML formats. Avoid internal package explanations. Inspect `internal/cli/*_test.go` for edge cases and examples. ### Config Reference Document all implemented config surfaces: flags that become config values, `SERIATIM_OUTPUT_SCHEMA`, `SERIATIM_OVERLAP_WORD_RUN_GAP`, `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW`, `SERIATIM_BACKCHANNEL_MAX_DURATION`, `SERIATIM_FILLER_MAX_DURATION`, module lists, output schemas, `speakers.yml`, and `autocorrect.yml`. Avoid command tutorials and unimplemented config files. Inspect `internal/config`, `internal/speaker`, `internal/autocorrect`, and tests. ### Operations Document filesystem-only command execution, output/report artifacts, validation failures, retry behavior, and cleanup. Explicitly say there is no daemon, database, remote storage, resume state, or background job state. Avoid unimplemented recovery procedures. ### Development Policy Document repository layout, `go test ./...`, package conventions, standard-library-first dependency guidance, how to add flags/config/modules, and documentation update expectations. State that no Makefile, taskfile, linter config, or automated documentation checker currently exists. ### Internal Docs Keep internal docs behavior-level and concise. Describe implemented inputs, outputs, boundaries, config fields used, adapters, failure behavior, tests, and invariants. Avoid future plugins, future input/output formats, concurrency, or duplicating CLI/config reference material. ### Root `architecture.md` Do not carry forward future input methods, future formats, future output formats, LLM text, dynamic plugin speculation, or interface sketches that diverge from code. Salvage only current-behavior details that are not already covered in `docs/policy/architecture.md` and move any legitimate future ideas under `docs/roadmap/`. ## Examples Plan Create small synthetic examples under `examples/` rather than relying on the current large `samples/raw` data. - `examples/minimal-merge/` - Purpose: shortest complete merge workflow with two small raw JSON files and optional `speakers.yml`. - Expected validity check: run `go run ./cmd/seriatim merge` with the example files and validate JSON output is produced. - Docs to link: README, `docs/cli.md`, `docs/config.md`, `docs/operations.md`. - `examples/normalize/` - Purpose: normalize object-with-`segments` and bare segment array inputs. - Expected validity check: run `go run ./cmd/seriatim normalize` for both shapes. - Docs to link: `docs/cli.md`, `docs/operations.md`, and any Audita/bare array integration note if created. - `examples/trim/` - Purpose: trim a small existing seriatim artifact by `--keep` and/or `--remove`. - Expected validity check: run `go run ./cmd/seriatim trim` and validate sequential retained IDs. - Docs to link: `docs/cli.md`, `docs/operations.md`. - `examples/speakers.yml` and `examples/autocorrect.yml` - Purpose: copyable YAML rule examples if linked from `docs/config.md`. - Expected validity check: load through merge command or package tests. - Docs to link: `docs/config.md`, `docs/cli.md`. Do not invent examples for unimplemented input methods, output formats, services, or plugin systems. Do not reuse `samples/raw` as public examples without privacy and size review. ## Internal Documentation Plan ### Pipeline - Path: `docs/internal/pipeline.md` - Purpose: document implemented merge pipeline orchestration. - Inputs and outputs: `config.Config`, raw transcripts, canonical transcripts, merged transcript, selected public artifact, optional report. - Boundaries: registry and runner orchestration; no CLI flag parsing; no schema details beyond output selection. - Config fields used: input reader, module lists, output modules, output schema, input/output/report files, timing thresholds passed through modules. - Adapters used: input reader, output writer, report writer. - Failure behavior: unknown modules, invalid preprocessing state, stage errors, output/report write failures. - Tests to inspect: `internal/pipeline/runner_test.go`, `internal/builtin/*_test.go`, `internal/cli/merge_test.go`. - Architectural invariants: deterministic sequential stage order, explicit raw-to-canonical preprocessing state, output validation before acceptance. ### Artifacts and Schemas - Path: `docs/internal/artifacts.md` - Purpose: document public artifact conversion and validation internals. - Inputs and outputs: merged model, schema structs, serialized JSON artifacts, parsed trim/normalize artifacts. - Boundaries: conversion and validation only; CLI docs own user-facing flags. - Config fields used: output schema, output modules, input files for metadata. - Adapters used: embedded JSON Schema files and JSON encoders/decoders. - Failure behavior: schema validation errors, unsupported artifact/schema conversion, invalid IDs/timing. - Tests to inspect: `schema/output_test.go`, `internal/artifact/transcript_test.go`, `internal/trim/*_test.go`, `internal/normalize/*_test.go`. - Architectural invariants: sequential IDs, selected schema validation, no internal-only fields in public schemas. ### Built-In Modules - Path: `docs/internal/modules.md` - Purpose: document implemented module responsibilities and ordering constraints. - Inputs and outputs: raw transcripts, preprocess state, merged transcript, report events, selected JSON output. - Boundaries: module behavior only; no full CLI/config reference. - Config fields used: speaker file, autocorrect file, coalesce gap, overlap word gap, word run reorder window, backchannel/filler max durations. - Adapters used: JSON input/output, speaker YAML, autocorrect YAML, report events. - Failure behavior: input validation errors, invalid YAML, unknown module names, invalid output schema before write. - Tests to inspect: `internal/builtin`, `internal/overlap`, `internal/coalesce`, `internal/danglers`, `internal/backchannel`, `internal/filler`, `internal/autocorrect`, and CLI merge tests. - Architectural invariants: order-sensitive transforms run before `assign-ids`; modules stay narrow and explicitly configured. ### Trim - Path: include in `docs/internal/artifacts.md` or create `docs/internal/trim.md` if artifacts doc grows too large. - Purpose: document artifact-level segment projection. - Inputs and outputs: existing seriatim artifact, selector, selected output schema, optional report. - Boundaries: no merge postprocessors; no raw WhisperX input. - Config fields used: input/output/report files, keep/remove selector, optional output schema, allow-empty. - Adapters used: file I/O in CLI, artifact parsing/validation, report writer. - Failure behavior: malformed selector, invalid artifact, missing selected IDs, non-sequential input IDs, empty output unless allowed, unsupported schema up-conversion. - Tests to inspect: `internal/trim/*_test.go`, `internal/cli/trim_test.go`. - Architectural invariants: preserve transcript order, renumber retained IDs, recompute full-schema overlap groups, never run merge modules. ### Normalize - Path: include in `docs/internal/artifacts.md` or create `docs/internal/normalize.md` if artifacts doc grows too large. - Purpose: document artifact-level transcript canonicalization. - Inputs and outputs: transcript-like JSON object or bare array, selected seriatim output schema, optional report. - Boundaries: no merge preprocessing or postprocessing modules. - Config fields used: input/output/report files, output schema, output modules. - Adapters used: file I/O, JSON parsing, schema validation, report writer. - Failure behavior: invalid JSON, unsupported top-level shape, invalid timing after repair, unsupported output module/schema, report write failure. - Tests to inspect: `internal/normalize/*_test.go`, `internal/cli/normalize_test.go`. - Architectural invariants: deterministic repair/sort/ID assignment, no transcript text in normalize report events, no merge modules. ## Integration Documentation Plan - `docs/integrations/whisperx-json.md` - External system or contract: WhisperX-like JSON transcript subset. - Current usage: `merge` reads a top-level `segments` array with required segment timing/text and optional word timing. - Version or compatibility notes: no explicit WhisperX version is encoded in the repository; document only the accepted subset. - Document: supported fields, validation, word timing behavior, errors. - Do not document: full WhisperX schema, audio diarization, non-JSON formats. - `docs/integrations/output-schemas.md` - External system or contract: seriatim public JSON output contracts. - Current usage: `merge`, `trim`, and `normalize` emit `seriatim-minimal`, `seriatim-intermediate`, or `seriatim-full`. - Version or compatibility notes: schemas are embedded from `schema/`; release version metadata is injected through build info. - Document: schema roles, semantic invariants, validation APIs, links to schema files. - Do not document: unimplemented output formats or full schema copies. - YAML rule files - Prefer documenting speaker and autocorrect YAML contracts in `docs/config.md`. Create `docs/integrations/yaml-rule-files.md` only if the config reference becomes too large. - Audita-style bare arrays - Cover under `docs/cli.md` normalize behavior unless maintainers need a separate integration note. Do not generalize beyond implemented bare segment arrays. - No external CLI/API/service docs are needed now. The repository implements no external CLI, network API, daemon, remote storage, or service integration. ## Recommended Implementation Sequence ### Stage 1: Write Documentation Roadmap - Goal: create this roadmap. - Files: `docs/roadmap/documentation.md`. - Repository areas to inspect: documentation policy, architecture policy, README, root `architecture.md`, CLI/config/pipeline/schema/report/tests. - Acceptance criteria: roadmap exists, no other files changed by this stage, and the roadmap is action-oriented. - Suggested validation commands: `go test ./...`; `git status --short`. - Prompt size: one implementation prompt. ### Stage 2: User-Facing Canonical Docs and Slim README - Goal: move user reference material out of README into canonical docs. - Files: update `README.md`; create `docs/cli.md` and `docs/config.md`. - Repository areas to inspect: `internal/cli`, `internal/config`, `internal/speaker`, `internal/autocorrect`, CLI/config tests. - Acceptance criteria: README is concise; CLI/config docs match flags, defaults, env vars, YAML formats, and validation; no roadmap-only content appears. - Suggested validation commands: `go test ./...`; `go run ./cmd/seriatim --help`; `go run ./cmd/seriatim merge --help`; `go run ./cmd/seriatim trim --help`; `go run ./cmd/seriatim normalize --help`; stale-term grep from the validation plan. - Prompt size: one prompt if concise; split if README rewrite or config reference grows too large. ### Stage 3: Operations and Troubleshooting - Goal: document runtime operation, reports, failure behavior, and common fixes. - Files: create `docs/operations.md` and `docs/troubleshooting.md`. - Repository areas to inspect: `cmd/seriatim/main.go`, `internal/cli`, `internal/config`, `internal/report`, output writer, normalize/trim/merge tests. - Acceptance criteria: docs describe filesystem-only operation and current failure modes; no daemon, resume, remote storage, or recovery behavior is invented. - Suggested validation commands: `go test ./...`; manual link review. - Prompt size: one prompt. ### Stage 4: Developer and Internal Docs - Goal: create developer workflow and implemented internal component docs. - Files: create `docs/policy/development.md`, `docs/internal/pipeline.md`, `docs/internal/artifacts.md`, and `docs/internal/modules.md`. - Repository areas to inspect: architecture policy, pipeline, modules, schema, artifact conversion, trim/normalize packages, tests. - Acceptance criteria: docs preserve boundaries, avoid CLI/config duplication, and identify tests/invariants for future changes. - Suggested validation commands: `go test ./...`; grep for unimplemented future-format/plugin/concurrency claims outside roadmap. - Prompt size: split into development policy and internal docs if needed. ### Stage 5: Integrations and Examples - Goal: add concise integration notes and maintained synthetic examples. - Files: create `docs/integrations/whisperx-json.md`, `docs/integrations/output-schemas.md`, and `examples/*`; decide whether `samples/` should remain separate. - Repository areas to inspect: `internal/builtin/input.go`, `schema`, `internal/artifact`, CLI tests, existing `samples/`. - Acceptance criteria: examples are small, synthetic, valid, and linked from relevant docs; integration docs document only implemented contracts. - Suggested validation commands: `go test ./...`; run documented example `go run` commands; validate example YAML through command paths. - Prompt size: split if examples need tests or sample cleanup decisions. ### Stage 6: Stale Documentation Cleanup - Goal: remove wrong-home and stale documentation after canonical replacements exist. - Files: delete or relocate root `architecture.md`; remove stale material from README; update links across docs. - Repository areas to inspect: all docs, README, roadmap, root files. - Acceptance criteria: no links to deleted root `architecture.md`; no unimplemented behavior outside `docs/roadmap/`; canonical homes are respected. - Suggested validation commands: `go test ./...`; stale-term grep; manual link check; `git status --short`. - Prompt size: one prompt. ## Validation Plan Use these checks during or after documentation migration: - Run `go test ./...`. - Run `go run ./cmd/seriatim --help`. - Run `go run ./cmd/seriatim merge --help`. - Run `go run ./cmd/seriatim trim --help`. - Run `go run ./cmd/seriatim normalize --help`. - Once examples exist, run each documented example command and verify output is produced in a temporary path. - Load example YAML through the merge command or package tests. - Validate example JSON through existing CLI/schema paths where practical. - Grep outside `docs/roadmap/` for stale or roadmap-only terms: `Future input`, `Future output`, `LLM`, `plugin`, `SRT`, `VTT`, `.tar.gz`, `URI`, `old format`, `not implemented yet`, and `runtime default may change`. - Manually check links unless a link checker is added. No automated documentation checker currently exists. - Verify docs and examples contain no secrets, private transcript data, API keys, tokens, passwords, or private infrastructure details. ## Open Questions - Should `samples/` be removed, kept as non-doc sample data, or replaced by small synthetic `examples/`? Recommendation: create small synthetic examples first, then audit `samples/` for privacy, size, and ongoing maintenance before deleting or linking it. - Should Audita-style bare-array normalization have a separate integration doc? Recommendation: cover it in `docs/cli.md` normalize behavior unless a stronger external-contract requirement emerges.