From b7cc5fb980043094b1f3a767b6d0bf737e620602 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 24 May 2026 08:19:39 -0500 Subject: [PATCH] Added the standard documentation policy and a roadmap for adding documentation --- LICENSE | 2 +- docs/policy/architecture.md | 219 +++++++++++++ docs/policy/documentation.md | 356 +++++++++++++++++++++ docs/roadmap/documentation.md | 579 ++++++++++++++++++++++++++++++++++ 4 files changed, 1155 insertions(+), 1 deletion(-) create mode 100644 docs/policy/architecture.md create mode 100644 docs/policy/documentation.md create mode 100644 docs/roadmap/documentation.md diff --git a/LICENSE b/LICENSE index a64b5c1..49bdd71 100644 --- a/LICENSE +++ b/LICENSE @@ -1,4 +1,4 @@ -Copyright (c) 2026 eric. +Copyright (c) 2026 Eric Rakestraw. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md new file mode 100644 index 0000000..cdd334c --- /dev/null +++ b/docs/policy/architecture.md @@ -0,0 +1,219 @@ +# Architecture Policy + +## Purpose + +This document defines seriatim's development architecture and invariants for +maintainers and automated coding agents. It describes how the implemented +system is intended to be built and changed. It is not a user manual, CLI +reference, config reference, or roadmap. + +Keep this document aligned with [documentation policy](documentation.md). It +must describe current behavior only; planned or speculative work belongs under +`docs/roadmap/`. + +## Project Shape + +seriatim is a Go CLI for transcript artifact processing. The implemented +commands are `merge`, `trim`, and `normalize`. + +`merge` reads one or more JSON transcript files, optionally maps input files to +canonical speakers, runs a registry-selected preprocessing chain, merges +canonical segments into deterministic chronological order, runs a +registry-selected postprocessing chain, validates the selected output schema, +and writes JSON output plus an optional JSON report. + +`trim` and `normalize` are artifact-level commands outside the merge pipeline. +`trim` reads an existing seriatim output artifact and projects it by segment ID. +`normalize` reads transcript-like JSON and emits one of seriatim's supported +output schemas. Neither command runs merge preprocessing or postprocessing +modules. + +The supported public output schemas are `seriatim-minimal`, +`seriatim-intermediate`, and `seriatim-full`. For current command and flag +details, use [the README](../../README.md) until dedicated `docs/cli.md` and +`docs/config.md` files exist. + +## Core Design Principles + +- Keep a hexagonal architecture boundary. Domain models, stage contracts, and + deterministic transformations must stay separate from CLI parsing, + filesystem access, config loading, reporting, and other external adapters. +- Keep stages and modules composable. Built-in modules are selected by + canonical registry names and implement explicit interfaces for their pipeline + role. +- Preserve deterministic behavior. Given the same inputs, configuration, and + version, output ordering, segment IDs, schema validation, and report event + ordering should remain stable. +- Current command execution is sequential. There is no scheduler, worker pool, + or concurrent module execution in the implemented pipeline. Any concurrency + added later must be bounded, observable, and must not make output handling + nondeterministic. +- Prefer the Go standard library. Third-party dependencies should remain narrow + and justified, such as Cobra for CLI structure, YAML parsing, and JSON Schema + validation. +- Document current behavior. Architecture, user, and internal docs must not + describe planned features as implemented behavior. + +## Architectural Boundaries + +Core transcript data belongs in `internal/model` and public artifact contracts +belong in `schema`. Conversion from internal merged data to public JSON shapes +belongs at the artifact boundary, not inside CLI code or transformation +packages. + +Pipeline orchestration belongs in `internal/pipeline`. It resolves registered +modules, validates preprocessing state transitions, executes stages in order, +collects report events, converts the final transcript, and writes optional +reports. Built-in adapters and modules are registered from `internal/builtin`. + +CLI code in `internal/cli` should parse flags, build validated config values, +and delegate. `merge` delegates to `pipeline.Run`; `trim` and `normalize` +perform artifact-level orchestration and delegate deterministic parsing, +validation, and transformation work to their internal packages. + +Config loading and validation belongs in `internal/config`. Filesystem reads and +writes are adapter concerns and should not spread into pure transformation +helpers. Existing built-in modules that load configured YAML files must keep +that I/O narrow and explicit. + +Reports belong in `internal/report`. Modules and commands should emit concise +events for validation findings, corrections, and transformations without +turning report messages into a duplicate output artifact. + +Tests and samples are supporting evidence for behavior. Tests should verify +stable contracts and edge cases; samples should remain valid examples, not +hidden architecture dependencies. + +## Modules or Stages + +The merge pipeline has these implemented stages: + +- `InputReader`: reads configured external input into raw transcripts. +- `Preprocessor`: transforms `PreprocessState` from raw to canonical state. +- `Merger`: combines canonical transcripts into one merged transcript. +- `Postprocessor`: transforms or annotates the merged transcript. +- `OutputWriter`: writes the selected output artifact. + +Modules must keep narrow responsibilities, declare their stage through the +interface they implement, and use explicit config values. Preprocessors must +declare `Requires()` and `Produces()` states; the runner rejects invalid +raw/canonical ordering before processing completes. + +Modules run in the configured order. Order-affecting modules must run before +`assign-ids`, and `validate-output` must see final IDs that match the selected +schema. Accepted and rejected transformations should be deterministic and, when +observable, recorded through report events. + +Transformation helpers should avoid hidden global state. Shared caches, such as +compiled JSON schemas, must be protected and must not affect output ordering. + +## State, Inputs, and Outputs + +seriatim is file-based. It reads JSON inputs and optional YAML rule files, then +writes JSON transcript artifacts and optional JSON reports. + +The implemented application has no durable database, daemon state, resume +state, remote storage, or background job state. Runtime state is held in memory +for the current command invocation and serialized only through requested output +and report files. + +Input file paths are normalized and validated during config construction. +`merge` sorts input file paths before processing, then uses stable segment sort +keys. `trim` preserves transcript order while renumbering retained IDs. +`normalize` sorts by implemented deterministic keys and assigns fresh IDs. + +## Configuration and CLI Boundaries + +The CLI surface is an adapter over validated config structs. Cobra command code +should stay thin: parse flags, account for flag/default precedence, call config +constructors, and delegate. + +Config constructors validate required paths, output parent directories, module +lists, selected schemas, mutually exclusive trim selector options, and supported +environment-derived settings. Module name validation is split between config +where command-specific names are fixed and the pipeline registry where module +composition is resolved. + +Do not duplicate full CLI or config reference material here. Use +[the README](../../README.md) for the current user-facing reference until the +canonical `docs/cli.md` and `docs/config.md` files exist. + +## Errors, Logging, and Diagnostics + +Commands return errors instead of printing inside deep logic. The root command +silences Cobra usage/error output, and `cmd/seriatim/main.go` prints one error +to stderr and exits with status `1`. + +Validation failures should fail fast with contextual errors. Correctable +conditions should be deterministic and, where reports are requested, reflected +as report events. Optional reports contain metadata and ordered events; they are +not required for command success unless the report file itself cannot be +written. + +The implemented code does not use a logging subsystem. Diagnostics are returned +as errors or written to optional report JSON. Normalize report events avoid +embedding transcript text; keep that privacy-oriented behavior when changing +normalize diagnostics. + +## Testing Expectations + +`go test ./...` is the repository-wide check. There is currently no Makefile, +taskfile, linter config, or dedicated documentation check. + +When changing config or CLI behavior, inspect `internal/config` and +`internal/cli` tests. When changing pipeline composition or stage contracts, +inspect `internal/pipeline` and `internal/builtin` tests. When changing +correction or annotation modules, inspect the package tests for overlap, +coalesce, danglers, backchannel, filler, and autocorrect behavior. + +When changing artifact-level commands, inspect `internal/trim`, +`internal/normalize`, and their CLI tests. When changing public output shape or +schema validation, inspect `schema` and `internal/artifact` tests. Report and +diagnostic changes should be covered through the command or package tests that +emit the affected events. + +## Dependency Policy + +Prefer the Go standard library for parsing, data transformation, concurrency +primitives, filesystem work, and testing wherever it is reasonable. + +Third-party dependencies must be narrow, justified, and preferably de facto +standard for their purpose. Existing examples include Cobra for CLI structure, +`gopkg.in/yaml.v3` for YAML files, and `jsonschema/v6` for validating embedded +public JSON schemas. Avoid broad framework dependencies for behavior that is +already simple and local. + +## Documentation Expectations + +Architecture docs must stay aligned with [documentation policy](documentation.md). +Current-behavior docs must not become aspirational. If code and docs disagree, +fix the inaccurate current-behavior doc or put planned work under +`docs/roadmap/`. + +Prefer links to canonical docs instead of repeating full CLI, config, schema, or +operations reference material. Keep examples real, tested where practical, and +free of secrets or private transcript data. + +## Architectural Invariants + +- Keep core/domain logic separate from CLI, config, filesystem, reporting, and + other adapter concerns. +- Keep modules narrowly scoped, explicitly configured, and composable by + registry name. +- Preserve deterministic ordering, final segment ID assignment, and schema + validation before output acceptance. +- Keep `trim` and `normalize` artifact-level; do not run merge modules from + those commands. +- Keep public output schemas validated through `schema`. +- Keep optional reports ordered, concise, and diagnostic. +- Avoid broad dependencies without a concrete maintainability benefit. +- Do not document unimplemented behavior outside `docs/roadmap/`. + +## Non-Goals + +The implemented application does not perform transcription, audio diarization, +speaker inference from audio or text, summarization, daemon operation, remote +storage, dynamic external plugin loading, or concurrent pipeline execution. + +The architecture policy is not a package-by-package reference, CLI manual, +config reference, schema reference, or roadmap. diff --git a/docs/policy/documentation.md b/docs/policy/documentation.md new file mode 100644 index 0000000..d9ed7fa --- /dev/null +++ b/docs/policy/documentation.md @@ -0,0 +1,356 @@ +# Go Project Documentation Policy + +## Purpose + +Project documentation must help four audiences: + +1. users who need to run the application; +2. administrators/operators who need to configure and operate it; +3. developers who need to understand and change it safely; +4. LLM coding agents that need clear scope, boundaries, and invariants. + +Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition. + +## Core Rules + +### 1. Keep docs concise + +Each document should cover a defined scope and only the essentials for that scope. + +Avoid: +- long background explanations; +- repeated reference material; +- implementation detail in user-facing docs; +- aspirational language outside roadmap docs; +- verbose examples where one minimal example is clearer. + +### 2. Document only implemented behavior outside roadmap files + +Unimplemented, planned, aspirational, experimental, or future work may be described only under: + +- `docs/roadmap/` + +No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist. + +If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary. + +### 3. Use canonical homes + +Each type of information should have one canonical location. + +Canonical homes: + +- project purpose and quickstart: `README.md` +- development principles: `docs/policy/architecture.md` +- configuration reference: `docs/config.md` +- CLI reference: `docs/cli.md` +- operations and recovery: `docs/operations.md` +- troubleshooting: `docs/troubleshooting.md` +- implemented internals: `docs/internal/` +- future work: `docs/roadmap/` +- contributor workflow: `docs/policy/development.md` +- copyable examples: `examples/` + +Other files should summarize briefly and link to the canonical source. + +### 4. Keep examples real + +Examples should be valid, maintained, and free of secrets. + +Where practical: +- example configs should load successfully; +- example commands should match real CLI syntax; +- important examples should be covered by tests. + +## Documentation Profiles + +All projects require: + +- `README.md` +- `docs/policy/architecture.md` + +Additional docs depend on the project. + +### Small library + +Recommended: +- `docs/policy/development.md`, if contributor conventions are non-obvious + +### Simple CLI + +Required: +- `docs/cli.md` + +Recommended: +- `docs/policy/development.md` + +### Config-driven CLI + +Required: +- `docs/cli.md` +- `docs/config.md` + +Recommended: +- `examples/` +- `docs/policy/development.md` + +### Stateful or operator-facing application + +Required: +- `docs/cli.md`, if CLI-based +- `docs/config.md`, if config-driven +- `docs/operations.md` + +Recommended: +- `docs/troubleshooting.md` +- `examples/` +- `docs/policy/development.md` + +### Modular, staged, service-oriented, or orchestration application + +Required: +- `docs/cli.md`, if CLI-based +- `docs/config.md`, if config-driven +- `docs/operations.md` +- `docs/internal/` +- `docs/policy/development.md` + +Recommended: +- `docs/troubleshooting.md` +- validated examples under `examples/` + +## Required Documents + +### README.md + +**Audience:** users, administrators, operators + +The README is the outward-facing project orientation page. + +It should include, in order: + +1. concise description; +2. elevator pitch; +3. shortest useful command or usage example; +4. links to targeted docs. + +The README should be short. It is not a manual. + +The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.) + +### docs/policy/architecture.md + +**Audience:** developers, LLM coding agents + +`docs/policy/architecture.md` is required for every project. + +It is an inward-facing development policy document. It should describe how the project is intended to be built and changed. + +It should include: + +- project shape; +- core design principles; +- package and boundary philosophy; +- state/persistence philosophy, if applicable; +- external integration philosophy, if applicable; +- error-handling and logging principles; +- testing expectations; +- documentation expectations; +- architectural invariants; +- explicit non-goals, if useful. + +For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light. + +### docs/policy/development.md + +**Audience:** developers, LLM coding agents + +Required for projects maintained by humans and LLM coding agents. + +It should include: + +- repository layout; +- build/test commands; +- coding conventions; +- dependency policy; +- how to add config fields; +- how to add CLI flags; +- how to add stages/modules/adapters, if applicable; +- how to update examples; +- documentation update expectations. + +### docs/config.md + +**Audience:** administrators, operators, advanced users + +Required for applications with configuration files. + +It should include, in order: + +1. config file locations and discovery precedence; +2. minimal working config; +3. production-oriented config; +4. full configuration reference; +5. secrets handling, if applicable; +6. links to maintained examples. + +The full configuration reference should be canonical. + +### docs/cli.md + +**Audience:** users, administrators, operators + +Required for CLI applications. + +It should include, in order: + +1. shortest useful command; +2. command overview; +3. complete flag reference; +4. common workflows; +5. diagnostic or recovery commands, if applicable. + +Explain when commands are useful, not just their syntax. + +### docs/operations.md + +**Audience:** administrators, operators + +Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures. + +It should cover: + +- normal workflow; +- filesystem layout; +- remote storage layout, if applicable; +- logs and manifests; +- resume/retry behavior; +- cleanup behavior; +- archive/backup behavior; +- safe recovery procedures; +- operational caveats. + +### docs/troubleshooting.md + +**Audience:** administrators, operators + +Recommended once recurring failure modes exist. + +Each entry should include: + +- symptom; +- likely cause; +- diagnostic command or inspection step; +- safe fix; +- relevant links. + +### docs/internal/ + +**Audience:** developers, LLM coding agents + +Required for modular, staged, service-oriented, or orchestration projects. + +This directory describes implemented internal components. It is not the roadmap. + +Use one file per major component where useful. + +Each component doc should include: + +1. purpose; +2. inputs and outputs; +3. boundaries; +4. config fields used; +5. external adapters used; +6. state or manifest behavior, if applicable; +7. skip/resume behavior, if applicable; +8. failure behavior; +9. tests to inspect before changing; +10. architectural invariants. + +### docs/roadmap/ + +**Audience:** maintainers, developers, LLM coding agents + +This is the only place for planned, future, aspirational, experimental, or unimplemented work. + +Roadmap docs should clearly distinguish: + +- proposed work; +- accepted plans; +- deferred ideas; +- rejected ideas; +- implementation prompts or task breakdowns, if useful. + +Roadmap docs should not be confused with current behavior. + +### docs/integrations/ + +**Audience:** developers, LLM coding agents + +Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain. + +This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses. + +Use one file per integration where useful. + +## Examples Directory + +Projects with non-trivial configuration or workflows should include `examples/`. + +Useful examples include: + +- minimal working config; +- production-oriented config; +- full annotated config; +- local development config; +- remote/object-storage config; +- minimal session/input file. + +Examples should be valid, maintained, tested when practical, and linked from relevant docs. + +## Security and Privacy + +Docs and examples must not include: + +- real API keys; +- tokens; +- passwords; +- private keys; +- private environment dumps; +- sensitive user data; +- raw private transcripts; +- private infrastructure details unless intentionally public. + +Document secret-handling mechanisms, not actual secret values. + +## Maintenance Rules + +When docs change, verify the affected behavior. + +Where practical: + +- load example config files in tests; +- test CLI examples or command parser behavior; +- validate documented flags against real flags; +- remove stale references; +- update links after renames; +- keep roadmap content out of non-roadmap docs. + +If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs. + +Documentation is complete only when it matches the current code. + +## Documentation Change Checklist + +Before merging documentation changes, verify: + +- README is concise and orientation-focused. +- `docs/policy/architecture.md` describes development principles. +- Future work appears only under `docs/roadmap/`. +- User-facing docs avoid unnecessary internals. +- Developer-facing docs preserve boundaries and invariants. +- Config examples match the schema. +- CLI examples match real commands and flags. +- Defaults appear in the canonical config reference. +- No secrets or private data are included. +- Links are accurate. diff --git a/docs/roadmap/documentation.md b/docs/roadmap/documentation.md new file mode 100644 index 0000000..d20b490 --- /dev/null +++ b/docs/roadmap/documentation.md @@ -0,0 +1,579 @@ +# Documentation Roadmap + +## Purpose + +This roadmap defines the work required to bring seriatim's documentation into +compliance with `docs/policy/documentation.md` and the current implementation. +It is grounded in the repository as it exists now: the Go CLI, config loading, +pipeline modules, artifact commands, schemas, reports, samples, and tests. + +Outside `docs/roadmap/`, documentation must describe only implemented +behavior. Planned, future, deprecated, experimental, or unimplemented work must +remain in roadmap documents until the code exists. + +## Repository Documentation Inventory + +- `README.md` - keep and rewrite. It currently mixes project orientation, + quickstart, full CLI reference, config/env reference, file formats, module + internals, limitations, and release build notes. Policy says README should be + concise and link to canonical docs. +- `docs/policy/documentation.md` - keep and lightly update only if the policy + itself changes. It is the controlling documentation layout and maintenance + policy. +- `docs/policy/architecture.md` - keep and lightly update as implementation + changes. It is the canonical development architecture policy. +- Root `architecture.md` - delete after salvage, or move only truly roadmap + material into `docs/roadmap/`. It is in the wrong canonical home and contains + future-oriented and aspirational claims. +- `docs/roadmap/documentation.md` - create new. This file is the planning + artifact for the documentation migration. +- `samples/` - split or move after audit. It contains sample raw transcripts, + merged artifacts, reports, `speakers.yml`, and `autocorrect.yml`, but + copyable examples belong under `examples/`. The raw sample data is large and + should be reviewed for privacy and maintainability before linking from docs. +- `schema/*.schema.json` - keep. These are public output contracts and should + be linked from documentation instead of duplicated in full. +- Missing canonical docs - create `docs/cli.md`, `docs/config.md`, + `docs/operations.md`, `docs/policy/development.md`, `docs/internal/`, and + likely `docs/troubleshooting.md`, `docs/integrations/`, and `examples/`. + +## Policy Compliance Assessment + +Required documents missing for seriatim's current shape as a modular, staged, +CLI/config-driven project: + +- `docs/cli.md` +- `docs/config.md` +- `docs/operations.md` +- `docs/internal/` +- `docs/policy/development.md` + +Recommended documents and directories missing: + +- `docs/troubleshooting.md` +- maintained copyable examples under `examples/` +- concise integration notes under `docs/integrations/` + +Existing compliance issues: + +- `README.md` is too broad for its canonical scope. It should keep project + purpose, quickstart, and links, then delegate CLI, config, operations, + internals, and schema details. +- Root `architecture.md` is stale and in the wrong home. It includes future + input methods and formats, future output formats, dynamic plugin speculation, + an LLM non-goal, interface sketches that diverge from code, and other + development-policy content now covered by `docs/policy/architecture.md`. +- Non-roadmap docs should not carry forward claims about future defaults, + future formats, unimplemented plugin systems, or unimplemented alternate + input/output methods. +- Historical or deprecated wording, such as the old speaker map format, should + move out of the README unless it is still needed in troubleshooting or a + narrow migration note. +- There is no `examples/` directory. `samples/` exists but is not the canonical + examples home and should not be treated as copyable public examples without a + privacy and size audit. +- Links need verification after migration: README should link to all new + canonical docs, docs should link to schema files and maintained examples, and + no doc should link to the deleted root `architecture.md`. + +## Target Documentation Set + +### `README.md` + +- Audience: users, administrators, and operators. +- Purpose: project orientation and shortest useful quickstart. +- Canonical scope: concise project purpose, elevator pitch, one minimal command, + and links to targeted docs. +- Recommended outline: project description; shortest merge command; command + summary; links to CLI, config, operations, architecture, development, schemas, + examples, and troubleshooting. +- Source of truth: current `README.md`, `internal/cli`, `internal/config`, + `cmd/seriatim/main.go`, and CLI tests. +- Acceptance criteria: no full flag tables, no full config reference, no module + manual, no future-feature claims, and all links resolve. + +### `docs/cli.md` + +- Audience: users, administrators, and operators. +- Purpose: canonical CLI reference and workflows. +- Canonical scope: shortest useful command, command overview, complete flag + reference, common workflows, diagnostics and report flags. +- Recommended outline: shortest useful command; global flags; `merge`; `trim`; + `normalize`; common workflows; exit/error behavior; links to config, + operations, examples, and schemas. +- Source of truth: `internal/cli/root.go`, `internal/cli/merge.go`, + `internal/cli/trim.go`, `internal/cli/normalize.go`, `internal/config`, and + `internal/cli/*_test.go`. +- Acceptance criteria: every documented flag, default, and required/mutually + exclusive rule matches code; package internals are linked rather than + explained in depth. + +### `docs/config.md` + +- Audience: administrators, operators, and advanced users. +- Purpose: canonical runtime configuration reference. +- Canonical scope: environment variables, default module lists, output schema + selection, `speakers.yml`, `autocorrect.yml`, path validation, and precedence. +- Recommended outline: config surfaces; output schema precedence; merge module + defaults; environment variables; speaker map YAML; autocorrect YAML; path and + validation rules; links to examples. +- Source of truth: `internal/config/config.go`, `internal/speaker/map.go`, + `internal/autocorrect/autocorrect.go`, `internal/config/config_test.go`, + `internal/speaker/map_test.go`, and `internal/autocorrect/autocorrect_test.go`. +- Acceptance criteria: all config fields and `SERIATIM_*` env vars match code; + unsupported config files or unimplemented formats are not described. + +### `docs/operations.md` + +- Audience: administrators and operators. +- Purpose: operational behavior for running commands safely. +- Canonical scope: file workflow, filesystem layout expectations, output and + report files, retry behavior, cleanup, validation failures, and operational + caveats. +- Recommended outline: normal workflow; input/output/report files; no durable + state; failure and retry behavior; reports and diagnostics; cleanup; privacy + considerations for transcript artifacts. +- Source of truth: `cmd/seriatim/main.go`, `internal/cli`, `internal/config`, + `internal/report`, `internal/builtin/output.go`, `internal/normalize`, and + trim/merge/normalize CLI tests. +- Acceptance criteria: clearly states there is no daemon, database, resume + state, remote storage, or background job state; does not invent recovery + workflows. + +### `docs/policy/development.md` + +- Audience: developers and coding agents. +- Purpose: contributor workflow and change guidance. +- Canonical scope: repository layout, build/test commands, coding conventions, + dependency policy, adding flags/config fields/modules/docs/examples. +- Recommended outline: repo layout; local checks; coding conventions; adding + CLI flags; adding config/env vars; adding modules/stages; schema changes; + examples and documentation updates. +- Source of truth: `docs/policy/documentation.md`, + `docs/policy/architecture.md`, `go.mod`, package layout, and test layout. +- Acceptance criteria: includes `go test ./...`; states there is no current + Makefile, taskfile, linter config, or automated doc checker; aligns with the + architecture policy. + +### `docs/internal/pipeline.md` + +- Audience: developers and coding agents. +- Purpose: implemented merge pipeline internals. +- Canonical scope: registry, stage interfaces, preprocessing state transitions, + module order, report event accumulation, final output/report writing. +- Recommended outline: purpose; inputs and outputs; stage contracts; registry + resolution; execution order; config fields used; adapters; failure behavior; + tests; invariants. +- Source of truth: `internal/pipeline`, `internal/builtin`, `internal/model`, + `internal/report`, `internal/pipeline/runner_test.go`, + `internal/builtin/*_test.go`, and `internal/cli/merge_test.go`. +- Acceptance criteria: describes only implemented sequential execution; does + not document concurrency, plugins, or future formats. + +### `docs/internal/artifacts.md` + +- Audience: developers and coding agents. +- Purpose: public artifact conversion and validation internals. +- Canonical scope: schema structs, embedded JSON Schemas, conversion from merged + model, trim/normalize artifact handling, and output validation. +- Recommended outline: artifact contracts; schema selection; conversion; + validation; trim projection; normalize canonicalization; tests; invariants. +- Source of truth: `schema`, `internal/artifact`, `internal/trim`, + `internal/normalize`, and related tests. +- Acceptance criteria: links to `schema/*.schema.json`; does not duplicate full + schemas or describe unavailable output formats. + +### `docs/internal/modules.md` + +- Audience: developers and coding agents. +- Purpose: implemented built-in module behavior and boundaries. +- Canonical scope: `json-files`, preprocessing modules, chronological merge, + postprocessing modules, and JSON output writer. +- Recommended outline: module list; inputs/outputs; config fields used; allowed + side effects; ordering constraints; failure behavior; tests; invariants. +- Source of truth: `internal/builtin`, `internal/overlap`, `internal/coalesce`, + `internal/danglers`, `internal/backchannel`, `internal/filler`, + `internal/autocorrect`, and package tests. +- Acceptance criteria: avoids full CLI/config duplication; identifies + order-sensitive transforms that must run before `assign-ids`. + +### `docs/troubleshooting.md` + +- Audience: users, administrators, and operators. +- Purpose: common failure symptoms and safe fixes. +- Canonical scope: implemented validation and runtime failures observed in + error paths and tests. +- Recommended outline: invalid JSON/input shape; missing required flags; invalid + output parent directory; invalid speaker/autocorrect YAML; unknown module; + invalid output schema; invalid trim selector; schema validation failure; + report write failure. +- Source of truth: `internal/config`, `internal/cli/*_test.go`, + `internal/trim/*_test.go`, `internal/normalize/*_test.go`, + `internal/speaker/*_test.go`, and `internal/autocorrect/*_test.go`. +- Acceptance criteria: each entry has symptom, likely cause, inspection step, + safe fix, and link; no speculative failure modes. + +### `docs/integrations/whisperx-json.md` + +- Audience: developers and coding agents. +- Purpose: external input JSON contract used by `merge`. +- Canonical scope: the supported WhisperX-like subset only. +- Recommended outline: top-level shape; required segment fields; optional word + timing fields; validation/failure behavior; how word timing affects overlap + resolution; links to CLI and examples. +- Source of truth: `internal/builtin/input.go`, merge CLI tests, and README + input-format material. +- Acceptance criteria: does not attempt to document full WhisperX behavior or + unsupported input formats. + +### `docs/integrations/output-schemas.md` + +- Audience: developers, coding agents, and artifact consumers. +- Purpose: orientation to public JSON output contracts. +- Canonical scope: minimal/intermediate/full schema roles and links to schema + files. +- Recommended outline: schema selection; minimal; intermediate; full; semantic + invariants; validation APIs; links to `schema/*.schema.json`. +- Source of truth: `schema/output.go`, `schema/*.schema.json`, + `schema/output_test.go`, and `internal/artifact`. +- Acceptance criteria: links to machine-readable schemas instead of copying + them in full. + +### `examples/` + +- Audience: users, administrators, operators, developers, and coding agents. +- Purpose: maintained copyable examples. +- Canonical scope: small synthetic inputs and config files for implemented + commands only. +- Source of truth: examples created during the documentation migration and + validated through actual command invocations. +- Acceptance criteria: examples are valid, free of secrets/private transcript + data, and linked from README, CLI, config, and operations docs. + +## File-by-File Rewrite Guidance + +### README + +Cover what seriatim is, the shortest useful `merge` command, a brief command +summary, and links to canonical docs. Avoid full flag tables, config/env +reference, module internals, schema examples, troubleshooting details, future +formats, or release-history narrative. Inspect `internal/cli`, `internal/config`, +and CLI tests before updating commands. + +### CLI Reference + +Document actual `merge`, `trim`, and `normalize` flags from `internal/cli`. +Include required flags, defaults, mutually exclusive selector rules, schema +selection, report flags, and common workflows. Link to `docs/config.md` for +environment variables and YAML formats. Avoid internal package explanations. +Inspect `internal/cli/*_test.go` for edge cases and examples. + +### Config Reference + +Document all implemented config surfaces: flags that become config values, +`SERIATIM_OUTPUT_SCHEMA`, `SERIATIM_OVERLAP_WORD_RUN_GAP`, +`SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW`, +`SERIATIM_BACKCHANNEL_MAX_DURATION`, `SERIATIM_FILLER_MAX_DURATION`, module +lists, output schemas, `speakers.yml`, and `autocorrect.yml`. Avoid command +tutorials and unimplemented config files. Inspect `internal/config`, +`internal/speaker`, `internal/autocorrect`, and tests. + +### Operations + +Document filesystem-only command execution, output/report artifacts, validation +failures, retry behavior, and cleanup. Explicitly say there is no daemon, +database, remote storage, resume state, or background job state. Avoid +unimplemented recovery procedures. + +### Development Policy + +Document repository layout, `go test ./...`, package conventions, +standard-library-first dependency guidance, how to add flags/config/modules, +and documentation update expectations. State that no Makefile, taskfile, +linter config, or automated documentation checker currently exists. + +### Internal Docs + +Keep internal docs behavior-level and concise. Describe implemented inputs, +outputs, boundaries, config fields used, adapters, failure behavior, tests, and +invariants. Avoid future plugins, future input/output formats, concurrency, or +duplicating CLI/config reference material. + +### Root `architecture.md` + +Do not carry forward future input methods, future formats, future output +formats, LLM text, dynamic plugin speculation, or interface sketches that +diverge from code. Salvage only current-behavior details that are not already +covered in `docs/policy/architecture.md` and move any legitimate future ideas +under `docs/roadmap/`. + +## Examples Plan + +Create small synthetic examples under `examples/` rather than relying on the +current large `samples/raw` data. + +- `examples/minimal-merge/` + - Purpose: shortest complete merge workflow with two small raw JSON files and + optional `speakers.yml`. + - Expected validity check: run `go run ./cmd/seriatim merge` with the example + files and validate JSON output is produced. + - Docs to link: README, `docs/cli.md`, `docs/config.md`, + `docs/operations.md`. +- `examples/normalize/` + - Purpose: normalize object-with-`segments` and bare segment array inputs. + - Expected validity check: run `go run ./cmd/seriatim normalize` for both + shapes. + - Docs to link: `docs/cli.md`, `docs/operations.md`, and any Audita/bare + array integration note if created. +- `examples/trim/` + - Purpose: trim a small existing seriatim artifact by `--keep` and/or + `--remove`. + - Expected validity check: run `go run ./cmd/seriatim trim` and validate + sequential retained IDs. + - Docs to link: `docs/cli.md`, `docs/operations.md`. +- `examples/speakers.yml` and `examples/autocorrect.yml` + - Purpose: copyable YAML rule examples if linked from `docs/config.md`. + - Expected validity check: load through merge command or package tests. + - Docs to link: `docs/config.md`, `docs/cli.md`. + +Do not invent examples for unimplemented input methods, output formats, +services, or plugin systems. Do not reuse `samples/raw` as public examples +without privacy and size review. + +## Internal Documentation Plan + +### Pipeline + +- Path: `docs/internal/pipeline.md` +- Purpose: document implemented merge pipeline orchestration. +- Inputs and outputs: `config.Config`, raw transcripts, canonical transcripts, + merged transcript, selected public artifact, optional report. +- Boundaries: registry and runner orchestration; no CLI flag parsing; no schema + details beyond output selection. +- Config fields used: input reader, module lists, output modules, output schema, + input/output/report files, timing thresholds passed through modules. +- Adapters used: input reader, output writer, report writer. +- Failure behavior: unknown modules, invalid preprocessing state, stage errors, + output/report write failures. +- Tests to inspect: `internal/pipeline/runner_test.go`, + `internal/builtin/*_test.go`, `internal/cli/merge_test.go`. +- Architectural invariants: deterministic sequential stage order, explicit + raw-to-canonical preprocessing state, output validation before acceptance. + +### Artifacts and Schemas + +- Path: `docs/internal/artifacts.md` +- Purpose: document public artifact conversion and validation internals. +- Inputs and outputs: merged model, schema structs, serialized JSON artifacts, + parsed trim/normalize artifacts. +- Boundaries: conversion and validation only; CLI docs own user-facing flags. +- Config fields used: output schema, output modules, input files for metadata. +- Adapters used: embedded JSON Schema files and JSON encoders/decoders. +- Failure behavior: schema validation errors, unsupported artifact/schema + conversion, invalid IDs/timing. +- Tests to inspect: `schema/output_test.go`, + `internal/artifact/transcript_test.go`, `internal/trim/*_test.go`, + `internal/normalize/*_test.go`. +- Architectural invariants: sequential IDs, selected schema validation, no + internal-only fields in public schemas. + +### Built-In Modules + +- Path: `docs/internal/modules.md` +- Purpose: document implemented module responsibilities and ordering + constraints. +- Inputs and outputs: raw transcripts, preprocess state, merged transcript, + report events, selected JSON output. +- Boundaries: module behavior only; no full CLI/config reference. +- Config fields used: speaker file, autocorrect file, coalesce gap, overlap word + gap, word run reorder window, backchannel/filler max durations. +- Adapters used: JSON input/output, speaker YAML, autocorrect YAML, report + events. +- Failure behavior: input validation errors, invalid YAML, unknown module names, + invalid output schema before write. +- Tests to inspect: `internal/builtin`, `internal/overlap`, + `internal/coalesce`, `internal/danglers`, `internal/backchannel`, + `internal/filler`, `internal/autocorrect`, and CLI merge tests. +- Architectural invariants: order-sensitive transforms run before `assign-ids`; + modules stay narrow and explicitly configured. + +### Trim + +- Path: include in `docs/internal/artifacts.md` or create + `docs/internal/trim.md` if artifacts doc grows too large. +- Purpose: document artifact-level segment projection. +- Inputs and outputs: existing seriatim artifact, selector, selected output + schema, optional report. +- Boundaries: no merge postprocessors; no raw WhisperX input. +- Config fields used: input/output/report files, keep/remove selector, + optional output schema, allow-empty. +- Adapters used: file I/O in CLI, artifact parsing/validation, report writer. +- Failure behavior: malformed selector, invalid artifact, missing selected IDs, + non-sequential input IDs, empty output unless allowed, unsupported schema + up-conversion. +- Tests to inspect: `internal/trim/*_test.go`, `internal/cli/trim_test.go`. +- Architectural invariants: preserve transcript order, renumber retained IDs, + recompute full-schema overlap groups, never run merge modules. + +### Normalize + +- Path: include in `docs/internal/artifacts.md` or create + `docs/internal/normalize.md` if artifacts doc grows too large. +- Purpose: document artifact-level transcript canonicalization. +- Inputs and outputs: transcript-like JSON object or bare array, selected + seriatim output schema, optional report. +- Boundaries: no merge preprocessing or postprocessing modules. +- Config fields used: input/output/report files, output schema, output modules. +- Adapters used: file I/O, JSON parsing, schema validation, report writer. +- Failure behavior: invalid JSON, unsupported top-level shape, invalid timing + after repair, unsupported output module/schema, report write failure. +- Tests to inspect: `internal/normalize/*_test.go`, + `internal/cli/normalize_test.go`. +- Architectural invariants: deterministic repair/sort/ID assignment, no + transcript text in normalize report events, no merge modules. + +## Integration Documentation Plan + +- `docs/integrations/whisperx-json.md` + - External system or contract: WhisperX-like JSON transcript subset. + - Current usage: `merge` reads a top-level `segments` array with required + segment timing/text and optional word timing. + - Version or compatibility notes: no explicit WhisperX version is encoded in + the repository; document only the accepted subset. + - Document: supported fields, validation, word timing behavior, errors. + - Do not document: full WhisperX schema, audio diarization, non-JSON formats. +- `docs/integrations/output-schemas.md` + - External system or contract: seriatim public JSON output contracts. + - Current usage: `merge`, `trim`, and `normalize` emit + `seriatim-minimal`, `seriatim-intermediate`, or `seriatim-full`. + - Version or compatibility notes: schemas are embedded from `schema/`; release + version metadata is injected through build info. + - Document: schema roles, semantic invariants, validation APIs, links to + schema files. + - Do not document: unimplemented output formats or full schema copies. +- YAML rule files + - Prefer documenting speaker and autocorrect YAML contracts in + `docs/config.md`. Create `docs/integrations/yaml-rule-files.md` only if the + config reference becomes too large. +- Audita-style bare arrays + - Cover under `docs/cli.md` normalize behavior unless maintainers need a + separate integration note. Do not generalize beyond implemented bare segment + arrays. +- No external CLI/API/service docs are needed now. The repository implements no + external CLI, network API, daemon, remote storage, or service integration. + +## Recommended Implementation Sequence + +### Stage 1: Write Documentation Roadmap + +- Goal: create this roadmap. +- Files: `docs/roadmap/documentation.md`. +- Repository areas to inspect: documentation policy, architecture policy, + README, root `architecture.md`, CLI/config/pipeline/schema/report/tests. +- Acceptance criteria: roadmap exists, no other files changed by this stage, + and the roadmap is action-oriented. +- Suggested validation commands: `go test ./...`; `git status --short`. +- Prompt size: one implementation prompt. + +### Stage 2: User-Facing Canonical Docs and Slim README + +- Goal: move user reference material out of README into canonical docs. +- Files: update `README.md`; create `docs/cli.md` and `docs/config.md`. +- Repository areas to inspect: `internal/cli`, `internal/config`, + `internal/speaker`, `internal/autocorrect`, CLI/config tests. +- Acceptance criteria: README is concise; CLI/config docs match flags, defaults, + env vars, YAML formats, and validation; no roadmap-only content appears. +- Suggested validation commands: `go test ./...`; + `go run ./cmd/seriatim --help`; + `go run ./cmd/seriatim merge --help`; + `go run ./cmd/seriatim trim --help`; + `go run ./cmd/seriatim normalize --help`; + stale-term grep from the validation plan. +- Prompt size: one prompt if concise; split if README rewrite or config + reference grows too large. + +### Stage 3: Operations and Troubleshooting + +- Goal: document runtime operation, reports, failure behavior, and common fixes. +- Files: create `docs/operations.md` and `docs/troubleshooting.md`. +- Repository areas to inspect: `cmd/seriatim/main.go`, `internal/cli`, + `internal/config`, `internal/report`, output writer, normalize/trim/merge + tests. +- Acceptance criteria: docs describe filesystem-only operation and current + failure modes; no daemon, resume, remote storage, or recovery behavior is + invented. +- Suggested validation commands: `go test ./...`; manual link review. +- Prompt size: one prompt. + +### Stage 4: Developer and Internal Docs + +- Goal: create developer workflow and implemented internal component docs. +- Files: create `docs/policy/development.md`, + `docs/internal/pipeline.md`, `docs/internal/artifacts.md`, and + `docs/internal/modules.md`. +- Repository areas to inspect: architecture policy, pipeline, modules, schema, + artifact conversion, trim/normalize packages, tests. +- Acceptance criteria: docs preserve boundaries, avoid CLI/config duplication, + and identify tests/invariants for future changes. +- Suggested validation commands: `go test ./...`; grep for unimplemented + future-format/plugin/concurrency claims outside roadmap. +- Prompt size: split into development policy and internal docs if needed. + +### Stage 5: Integrations and Examples + +- Goal: add concise integration notes and maintained synthetic examples. +- Files: create `docs/integrations/whisperx-json.md`, + `docs/integrations/output-schemas.md`, and `examples/*`; decide whether + `samples/` should remain separate. +- Repository areas to inspect: `internal/builtin/input.go`, `schema`, + `internal/artifact`, CLI tests, existing `samples/`. +- Acceptance criteria: examples are small, synthetic, valid, and linked from + relevant docs; integration docs document only implemented contracts. +- Suggested validation commands: `go test ./...`; run documented example + `go run` commands; validate example YAML through command paths. +- Prompt size: split if examples need tests or sample cleanup decisions. + +### Stage 6: Stale Documentation Cleanup + +- Goal: remove wrong-home and stale documentation after canonical replacements + exist. +- Files: delete or relocate root `architecture.md`; remove stale material from + README; update links across docs. +- Repository areas to inspect: all docs, README, roadmap, root files. +- Acceptance criteria: no links to deleted root `architecture.md`; no + unimplemented behavior outside `docs/roadmap/`; canonical homes are respected. +- Suggested validation commands: `go test ./...`; stale-term grep; manual link + check; `git status --short`. +- Prompt size: one prompt. + +## Validation Plan + +Use these checks during or after documentation migration: + +- Run `go test ./...`. +- Run `go run ./cmd/seriatim --help`. +- Run `go run ./cmd/seriatim merge --help`. +- Run `go run ./cmd/seriatim trim --help`. +- Run `go run ./cmd/seriatim normalize --help`. +- Once examples exist, run each documented example command and verify output is + produced in a temporary path. +- Load example YAML through the merge command or package tests. +- Validate example JSON through existing CLI/schema paths where practical. +- Grep outside `docs/roadmap/` for stale or roadmap-only terms: + `Future input`, `Future output`, `LLM`, `plugin`, `SRT`, `VTT`, `.tar.gz`, + `URI`, `old format`, `not implemented yet`, and + `runtime default may change`. +- Manually check links unless a link checker is added. No automated + documentation checker currently exists. +- Verify docs and examples contain no secrets, private transcript data, API + keys, tokens, passwords, or private infrastructure details. + +## Open Questions + +- Should `samples/` be removed, kept as non-doc sample data, or replaced by + small synthetic `examples/`? Recommendation: create small synthetic examples + first, then audit `samples/` for privacy, size, and ongoing maintenance before + deleting or linking it. +- Should Audita-style bare-array normalization have a separate integration doc? + Recommendation: cover it in `docs/cli.md` normalize behavior unless a + stronger external-contract requirement emerges.