seriatim/docs/roadmap/documentation.md

Documentation Roadmap

Purpose

This roadmap defines the work required to bring seriatim's documentation into compliance with docs/policy/documentation.md and the current implementation. It is grounded in the repository as it exists now: the Go CLI, config loading, pipeline modules, artifact commands, schemas, reports, samples, and tests.

Outside docs/roadmap/, documentation must describe only implemented behavior. Planned, future, deprecated, experimental, or unimplemented work must remain in roadmap documents until the code exists.

Repository Documentation Inventory

• README.md - keep and rewrite. It currently mixes project orientation, quickstart, full CLI reference, config/env reference, file formats, module internals, limitations, and release build notes. Policy says README should be concise and link to canonical docs.
• docs/policy/documentation.md - keep and lightly update only if the policy itself changes. It is the controlling documentation layout and maintenance policy.
• docs/policy/architecture.md - keep and lightly update as implementation changes. It is the canonical development architecture policy.
• Root architecture.md - delete after salvage, or move only truly roadmap material into docs/roadmap/. It is in the wrong canonical home and contains future-oriented and aspirational claims.
• docs/roadmap/documentation.md - create new. This file is the planning artifact for the documentation migration.
• samples/ - split or move after audit. It contains sample raw transcripts, merged artifacts, reports, speakers.yml, and autocorrect.yml, but copyable examples belong under examples/. The raw sample data is large and should be reviewed for privacy and maintainability before linking from docs.
• schema/*.schema.json - keep. These are public output contracts and should be linked from documentation instead of duplicated in full.
• Missing canonical docs - create docs/cli.md, docs/config.md, docs/operations.md, docs/policy/development.md, docs/internal/, and likely docs/troubleshooting.md, docs/integrations/, and examples/.

Policy Compliance Assessment

Required documents missing for seriatim's current shape as a modular, staged, CLI/config-driven project:

• docs/cli.md
• docs/config.md
• docs/operations.md
• docs/internal/
• docs/policy/development.md

Recommended documents and directories missing:

• docs/troubleshooting.md
• maintained copyable examples under examples/
• concise integration notes under docs/integrations/

Existing compliance issues:

• README.md is too broad for its canonical scope. It should keep project purpose, quickstart, and links, then delegate CLI, config, operations, internals, and schema details.
• Root architecture.md is stale and in the wrong home. It includes future input methods and formats, future output formats, dynamic plugin speculation, an LLM non-goal, interface sketches that diverge from code, and other development-policy content now covered by docs/policy/architecture.md.
• Non-roadmap docs should not carry forward claims about future defaults, future formats, unimplemented plugin systems, or unimplemented alternate input/output methods.
• Historical or deprecated wording, such as the old speaker map format, should move out of the README unless it is still needed in troubleshooting or a narrow migration note.
• There is no examples/ directory. samples/ exists but is not the canonical examples home and should not be treated as copyable public examples without a privacy and size audit.
• Links need verification after migration: README should link to all new canonical docs, docs should link to schema files and maintained examples, and no doc should link to the deleted root architecture.md.

Target Documentation Set

README.md

• Audience: users, administrators, and operators.
• Purpose: project orientation and shortest useful quickstart.
• Canonical scope: concise project purpose, elevator pitch, one minimal command, and links to targeted docs.
• Recommended outline: project description; shortest merge command; command summary; links to CLI, config, operations, architecture, development, schemas, examples, and troubleshooting.
• Source of truth: current README.md, internal/cli, internal/config, cmd/seriatim/main.go, and CLI tests.
• Acceptance criteria: no full flag tables, no full config reference, no module manual, no future-feature claims, and all links resolve.

docs/cli.md

• Audience: users, administrators, and operators.
• Purpose: canonical CLI reference and workflows.
• Canonical scope: shortest useful command, command overview, complete flag reference, common workflows, diagnostics and report flags.
• Recommended outline: shortest useful command; global flags; merge; trim; normalize; common workflows; exit/error behavior; links to config, operations, examples, and schemas.
• Source of truth: internal/cli/root.go, internal/cli/merge.go, internal/cli/trim.go, internal/cli/normalize.go, internal/config, and internal/cli/*_test.go.
• Acceptance criteria: every documented flag, default, and required/mutually exclusive rule matches code; package internals are linked rather than explained in depth.

docs/config.md

• Audience: administrators, operators, and advanced users.
• Purpose: canonical runtime configuration reference.
• Canonical scope: environment variables, default module lists, output schema selection, speakers.yml, autocorrect.yml, path validation, and precedence.
• Recommended outline: config surfaces; output schema precedence; merge module defaults; environment variables; speaker map YAML; autocorrect YAML; path and validation rules; links to examples.
• Source of truth: internal/config/config.go, internal/speaker/map.go, internal/autocorrect/autocorrect.go, internal/config/config_test.go, internal/speaker/map_test.go, and internal/autocorrect/autocorrect_test.go.
• Acceptance criteria: all config fields and SERIATIM_* env vars match code; unsupported config files or unimplemented formats are not described.

docs/operations.md

• Audience: administrators and operators.
• Purpose: operational behavior for running commands safely.
• Canonical scope: file workflow, filesystem layout expectations, output and report files, retry behavior, cleanup, validation failures, and operational caveats.
• Recommended outline: normal workflow; input/output/report files; no durable state; failure and retry behavior; reports and diagnostics; cleanup; privacy considerations for transcript artifacts.
• Source of truth: cmd/seriatim/main.go, internal/cli, internal/config, internal/report, internal/builtin/output.go, internal/normalize, and trim/merge/normalize CLI tests.
• Acceptance criteria: clearly states there is no daemon, database, resume state, remote storage, or background job state; does not invent recovery workflows.

docs/policy/development.md

• Audience: developers and coding agents.
• Purpose: contributor workflow and change guidance.
• Canonical scope: repository layout, build/test commands, coding conventions, dependency policy, adding flags/config fields/modules/docs/examples.
• Recommended outline: repo layout; local checks; coding conventions; adding CLI flags; adding config/env vars; adding modules/stages; schema changes; examples and documentation updates.
• Source of truth: docs/policy/documentation.md, docs/policy/architecture.md, go.mod, package layout, and test layout.
• Acceptance criteria: includes go test ./...; states there is no current Makefile, taskfile, linter config, or automated doc checker; aligns with the architecture policy.

docs/internal/pipeline.md

• Audience: developers and coding agents.
• Purpose: implemented merge pipeline internals.
• Canonical scope: registry, stage interfaces, preprocessing state transitions, module order, report event accumulation, final output/report writing.
• Recommended outline: purpose; inputs and outputs; stage contracts; registry resolution; execution order; config fields used; adapters; failure behavior; tests; invariants.
• Source of truth: internal/pipeline, internal/builtin, internal/model, internal/report, internal/pipeline/runner_test.go, internal/builtin/*_test.go, and internal/cli/merge_test.go.
• Acceptance criteria: describes only implemented sequential execution; does not document concurrency, plugins, or future formats.

docs/internal/artifacts.md

• Audience: developers and coding agents.
• Purpose: public artifact conversion and validation internals.
• Canonical scope: schema structs, embedded JSON Schemas, conversion from merged model, trim/normalize artifact handling, and output validation.
• Recommended outline: artifact contracts; schema selection; conversion; validation; trim projection; normalize canonicalization; tests; invariants.
• Source of truth: schema, internal/artifact, internal/trim, internal/normalize, and related tests.
• Acceptance criteria: links to schema/*.schema.json; does not duplicate full schemas or describe unavailable output formats.

docs/internal/modules.md

• Audience: developers and coding agents.
• Purpose: implemented built-in module behavior and boundaries.
• Canonical scope: json-files, preprocessing modules, chronological merge, postprocessing modules, and JSON output writer.
• Recommended outline: module list; inputs/outputs; config fields used; allowed side effects; ordering constraints; failure behavior; tests; invariants.
• Source of truth: internal/builtin, internal/overlap, internal/coalesce, internal/danglers, internal/backchannel, internal/filler, internal/autocorrect, and package tests.
• Acceptance criteria: avoids full CLI/config duplication; identifies order-sensitive transforms that must run before assign-ids.

docs/troubleshooting.md

• Audience: users, administrators, and operators.
• Purpose: common failure symptoms and safe fixes.
• Canonical scope: implemented validation and runtime failures observed in error paths and tests.
• Recommended outline: invalid JSON/input shape; missing required flags; invalid output parent directory; invalid speaker/autocorrect YAML; unknown module; invalid output schema; invalid trim selector; schema validation failure; report write failure.
• Source of truth: internal/config, internal/cli/*_test.go, internal/trim/*_test.go, internal/normalize/*_test.go, internal/speaker/*_test.go, and internal/autocorrect/*_test.go.
• Acceptance criteria: each entry has symptom, likely cause, inspection step, safe fix, and link; no speculative failure modes.

docs/integrations/whisperx-json.md

• Audience: developers and coding agents.
• Purpose: external input JSON contract used by merge.
• Canonical scope: the supported WhisperX-like subset only.
• Recommended outline: top-level shape; required segment fields; optional word timing fields; validation/failure behavior; how word timing affects overlap resolution; links to CLI and examples.
• Source of truth: internal/builtin/input.go, merge CLI tests, and README input-format material.
• Acceptance criteria: does not attempt to document full WhisperX behavior or unsupported input formats.

docs/integrations/output-schemas.md

• Audience: developers, coding agents, and artifact consumers.
• Purpose: orientation to public JSON output contracts.
• Canonical scope: minimal/intermediate/full schema roles and links to schema files.
• Recommended outline: schema selection; minimal; intermediate; full; semantic invariants; validation APIs; links to schema/*.schema.json.
• Source of truth: schema/output.go, schema/*.schema.json, schema/output_test.go, and internal/artifact.
• Acceptance criteria: links to machine-readable schemas instead of copying them in full.

examples/

• Audience: users, administrators, operators, developers, and coding agents.
• Purpose: maintained copyable examples.
• Canonical scope: small synthetic inputs and config files for implemented commands only.
• Source of truth: examples created during the documentation migration and validated through actual command invocations.
• Acceptance criteria: examples are valid, free of secrets/private transcript data, and linked from README, CLI, config, and operations docs.

File-by-File Rewrite Guidance

README

Cover what seriatim is, the shortest useful merge command, a brief command summary, and links to canonical docs. Avoid full flag tables, config/env reference, module internals, schema examples, troubleshooting details, future formats, or release-history narrative. Inspect internal/cli, internal/config, and CLI tests before updating commands.

CLI Reference

Document actual merge, trim, and normalize flags from internal/cli. Include required flags, defaults, mutually exclusive selector rules, schema selection, report flags, and common workflows. Link to docs/config.md for environment variables and YAML formats. Avoid internal package explanations. Inspect internal/cli/*_test.go for edge cases and examples.

Config Reference

Document all implemented config surfaces: flags that become config values, SERIATIM_OUTPUT_SCHEMA, SERIATIM_OVERLAP_WORD_RUN_GAP, SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW, SERIATIM_BACKCHANNEL_MAX_DURATION, SERIATIM_FILLER_MAX_DURATION, module lists, output schemas, speakers.yml, and autocorrect.yml. Avoid command tutorials and unimplemented config files. Inspect internal/config, internal/speaker, internal/autocorrect, and tests.

Operations

Document filesystem-only command execution, output/report artifacts, validation failures, retry behavior, and cleanup. Explicitly say there is no daemon, database, remote storage, resume state, or background job state. Avoid unimplemented recovery procedures.

Development Policy

Document repository layout, go test ./..., package conventions, standard-library-first dependency guidance, how to add flags/config/modules, and documentation update expectations. State that no Makefile, taskfile, linter config, or automated documentation checker currently exists.

Internal Docs

Keep internal docs behavior-level and concise. Describe implemented inputs, outputs, boundaries, config fields used, adapters, failure behavior, tests, and invariants. Avoid future plugins, future input/output formats, concurrency, or duplicating CLI/config reference material.

Root architecture.md

Do not carry forward future input methods, future formats, future output formats, LLM text, dynamic plugin speculation, or interface sketches that diverge from code. Salvage only current-behavior details that are not already covered in docs/policy/architecture.md and move any legitimate future ideas under docs/roadmap/.

Examples Plan

Create small synthetic examples under examples/ rather than relying on the current large samples/raw data.

• examples/minimal-merge/
  • Purpose: shortest complete merge workflow with two small raw JSON files and optional speakers.yml.
  • Expected validity check: run go run ./cmd/seriatim merge with the example files and validate JSON output is produced.
  • Docs to link: README, docs/cli.md, docs/config.md, docs/operations.md.
• examples/normalize/
  • Purpose: normalize object-with-segments and bare segment array inputs.
  • Expected validity check: run go run ./cmd/seriatim normalize for both shapes.
  • Docs to link: docs/cli.md, docs/operations.md, and any Audita/bare array integration note if created.
• examples/trim/
  • Purpose: trim a small existing seriatim artifact by --keep and/or --remove.
  • Expected validity check: run go run ./cmd/seriatim trim and validate sequential retained IDs.
  • Docs to link: docs/cli.md, docs/operations.md.
• examples/speakers.yml and examples/autocorrect.yml
  • Purpose: copyable YAML rule examples if linked from docs/config.md.
  • Expected validity check: load through merge command or package tests.
  • Docs to link: docs/config.md, docs/cli.md.

Do not invent examples for unimplemented input methods, output formats, services, or plugin systems. Do not reuse samples/raw as public examples without privacy and size review.

Internal Documentation Plan

Pipeline

• Path: docs/internal/pipeline.md
• Purpose: document implemented merge pipeline orchestration.
• Inputs and outputs: config.Config, raw transcripts, canonical transcripts, merged transcript, selected public artifact, optional report.
• Boundaries: registry and runner orchestration; no CLI flag parsing; no schema details beyond output selection.
• Config fields used: input reader, module lists, output modules, output schema, input/output/report files, timing thresholds passed through modules.
• Adapters used: input reader, output writer, report writer.
• Failure behavior: unknown modules, invalid preprocessing state, stage errors, output/report write failures.
• Tests to inspect: internal/pipeline/runner_test.go, internal/builtin/*_test.go, internal/cli/merge_test.go.
• Architectural invariants: deterministic sequential stage order, explicit raw-to-canonical preprocessing state, output validation before acceptance.

Artifacts and Schemas

• Path: docs/internal/artifacts.md
• Purpose: document public artifact conversion and validation internals.
• Inputs and outputs: merged model, schema structs, serialized JSON artifacts, parsed trim/normalize artifacts.
• Boundaries: conversion and validation only; CLI docs own user-facing flags.
• Config fields used: output schema, output modules, input files for metadata.
• Adapters used: embedded JSON Schema files and JSON encoders/decoders.
• Failure behavior: schema validation errors, unsupported artifact/schema conversion, invalid IDs/timing.
• Tests to inspect: schema/output_test.go, internal/artifact/transcript_test.go, internal/trim/*_test.go, internal/normalize/*_test.go.
• Architectural invariants: sequential IDs, selected schema validation, no internal-only fields in public schemas.

Built-In Modules

• Path: docs/internal/modules.md
• Purpose: document implemented module responsibilities and ordering constraints.
• Inputs and outputs: raw transcripts, preprocess state, merged transcript, report events, selected JSON output.
• Boundaries: module behavior only; no full CLI/config reference.
• Config fields used: speaker file, autocorrect file, coalesce gap, overlap word gap, word run reorder window, backchannel/filler max durations.
• Adapters used: JSON input/output, speaker YAML, autocorrect YAML, report events.
• Failure behavior: input validation errors, invalid YAML, unknown module names, invalid output schema before write.
• Tests to inspect: internal/builtin, internal/overlap, internal/coalesce, internal/danglers, internal/backchannel, internal/filler, internal/autocorrect, and CLI merge tests.
• Architectural invariants: order-sensitive transforms run before assign-ids; modules stay narrow and explicitly configured.

Trim

• Path: include in docs/internal/artifacts.md or create docs/internal/trim.md if artifacts doc grows too large.
• Purpose: document artifact-level segment projection.
• Inputs and outputs: existing seriatim artifact, selector, selected output schema, optional report.
• Boundaries: no merge postprocessors; no raw WhisperX input.
• Config fields used: input/output/report files, keep/remove selector, optional output schema, allow-empty.
• Adapters used: file I/O in CLI, artifact parsing/validation, report writer.
• Failure behavior: malformed selector, invalid artifact, missing selected IDs, non-sequential input IDs, empty output unless allowed, unsupported schema up-conversion.
• Tests to inspect: internal/trim/*_test.go, internal/cli/trim_test.go.
• Architectural invariants: preserve transcript order, renumber retained IDs, recompute full-schema overlap groups, never run merge modules.

Normalize

• Path: include in docs/internal/artifacts.md or create docs/internal/normalize.md if artifacts doc grows too large.
• Purpose: document artifact-level transcript canonicalization.
• Inputs and outputs: transcript-like JSON object or bare array, selected seriatim output schema, optional report.
• Boundaries: no merge preprocessing or postprocessing modules.
• Config fields used: input/output/report files, output schema, output modules.
• Adapters used: file I/O, JSON parsing, schema validation, report writer.
• Failure behavior: invalid JSON, unsupported top-level shape, invalid timing after repair, unsupported output module/schema, report write failure.
• Tests to inspect: internal/normalize/*_test.go, internal/cli/normalize_test.go.
• Architectural invariants: deterministic repair/sort/ID assignment, no transcript text in normalize report events, no merge modules.

Integration Documentation Plan

• docs/integrations/whisperx-json.md
  • External system or contract: WhisperX-like JSON transcript subset.
  • Current usage: merge reads a top-level segments array with required segment timing/text and optional word timing.
  • Version or compatibility notes: no explicit WhisperX version is encoded in the repository; document only the accepted subset.
  • Document: supported fields, validation, word timing behavior, errors.
  • Do not document: full WhisperX schema, audio diarization, non-JSON formats.
• docs/integrations/output-schemas.md
  • External system or contract: seriatim public JSON output contracts.
  • Current usage: merge, trim, and normalize emit seriatim-minimal, seriatim-intermediate, or seriatim-full.
  • Version or compatibility notes: schemas are embedded from schema/; release version metadata is injected through build info.
  • Document: schema roles, semantic invariants, validation APIs, links to schema files.
  • Do not document: unimplemented output formats or full schema copies.
• YAML rule files
  • Prefer documenting speaker and autocorrect YAML contracts in docs/config.md. Create docs/integrations/yaml-rule-files.md only if the config reference becomes too large.
• Audita-style bare arrays
  • Cover under docs/cli.md normalize behavior unless maintainers need a separate integration note. Do not generalize beyond implemented bare segment arrays.
• No external CLI/API/service docs are needed now. The repository implements no external CLI, network API, daemon, remote storage, or service integration.

Recommended Implementation Sequence

Stage 1: Write Documentation Roadmap

• Goal: create this roadmap.
• Files: docs/roadmap/documentation.md.
• Repository areas to inspect: documentation policy, architecture policy, README, root architecture.md, CLI/config/pipeline/schema/report/tests.
• Acceptance criteria: roadmap exists, no other files changed by this stage, and the roadmap is action-oriented.
• Suggested validation commands: go test ./...; git status --short.
• Prompt size: one implementation prompt.

Stage 2: User-Facing Canonical Docs and Slim README

• Goal: move user reference material out of README into canonical docs.
• Files: update README.md; create docs/cli.md and docs/config.md.
• Repository areas to inspect: internal/cli, internal/config, internal/speaker, internal/autocorrect, CLI/config tests.
• Acceptance criteria: README is concise; CLI/config docs match flags, defaults, env vars, YAML formats, and validation; no roadmap-only content appears.
• Suggested validation commands: go test ./...; go run ./cmd/seriatim --help; go run ./cmd/seriatim merge --help; go run ./cmd/seriatim trim --help; go run ./cmd/seriatim normalize --help; stale-term grep from the validation plan.
• Prompt size: one prompt if concise; split if README rewrite or config reference grows too large.

Stage 3: Operations and Troubleshooting

• Goal: document runtime operation, reports, failure behavior, and common fixes.
• Files: create docs/operations.md and docs/troubleshooting.md.
• Repository areas to inspect: cmd/seriatim/main.go, internal/cli, internal/config, internal/report, output writer, normalize/trim/merge tests.
• Acceptance criteria: docs describe filesystem-only operation and current failure modes; no daemon, resume, remote storage, or recovery behavior is invented.
• Suggested validation commands: go test ./...; manual link review.
• Prompt size: one prompt.

Stage 4: Developer and Internal Docs

• Goal: create developer workflow and implemented internal component docs.
• Files: create docs/policy/development.md, docs/internal/pipeline.md, docs/internal/artifacts.md, and docs/internal/modules.md.
• Repository areas to inspect: architecture policy, pipeline, modules, schema, artifact conversion, trim/normalize packages, tests.
• Acceptance criteria: docs preserve boundaries, avoid CLI/config duplication, and identify tests/invariants for future changes.
• Suggested validation commands: go test ./...; grep for unimplemented future-format/plugin/concurrency claims outside roadmap.
• Prompt size: split into development policy and internal docs if needed.

Stage 5: Integrations and Examples

• Goal: add concise integration notes and maintained synthetic examples.
• Files: create docs/integrations/whisperx-json.md, docs/integrations/output-schemas.md, and examples/*; decide whether samples/ should remain separate.
• Repository areas to inspect: internal/builtin/input.go, schema, internal/artifact, CLI tests, existing samples/.
• Acceptance criteria: examples are small, synthetic, valid, and linked from relevant docs; integration docs document only implemented contracts.
• Suggested validation commands: go test ./...; run documented example go run commands; validate example YAML through command paths.
• Prompt size: split if examples need tests or sample cleanup decisions.

Stage 6: Stale Documentation Cleanup

• Goal: remove wrong-home and stale documentation after canonical replacements exist.
• Files: delete or relocate root architecture.md; remove stale material from README; update links across docs.
• Repository areas to inspect: all docs, README, roadmap, root files.
• Acceptance criteria: no links to deleted root architecture.md; no unimplemented behavior outside docs/roadmap/; canonical homes are respected.
• Suggested validation commands: go test ./...; stale-term grep; manual link check; git status --short.
• Prompt size: one prompt.

Validation Plan

Use these checks during or after documentation migration:

• Run go test ./....
• Run go run ./cmd/seriatim --help.
• Run go run ./cmd/seriatim merge --help.
• Run go run ./cmd/seriatim trim --help.
• Run go run ./cmd/seriatim normalize --help.
• Once examples exist, run each documented example command and verify output is produced in a temporary path.
• Load example YAML through the merge command or package tests.
• Validate example JSON through existing CLI/schema paths where practical.
• Grep outside docs/roadmap/ for stale or roadmap-only terms: Future input, Future output, LLM, plugin, SRT, VTT, .tar.gz, URI, old format, not implemented yet, and runtime default may change.
• Manually check links unless a link checker is added. No automated documentation checker currently exists.
• Verify docs and examples contain no secrets, private transcript data, API keys, tokens, passwords, or private infrastructure details.

Open Questions

• Should samples/ be removed, kept as non-doc sample data, or replaced by small synthetic examples/? Recommendation: create small synthetic examples first, then audit samples/ for privacy, size, and ongoing maintenance before deleting or linking it.
• Should Audita-style bare-array normalization have a separate integration doc? Recommendation: cover it in docs/cli.md normalize behavior unless a stronger external-contract requirement emerges.