Added the standard documentation policy and a roadmap for adding documentation

docs/policy/architecture.md

@@ -0,0 +1,219 @@
+# Architecture Policy
+
+## Purpose
+
+This document defines seriatim's development architecture and invariants for
+maintainers and automated coding agents. It describes how the implemented
+system is intended to be built and changed. It is not a user manual, CLI
+reference, config reference, or roadmap.
+
+Keep this document aligned with [documentation policy](documentation.md). It
+must describe current behavior only; planned or speculative work belongs under
+`docs/roadmap/`.
+
+## Project Shape
+
+seriatim is a Go CLI for transcript artifact processing. The implemented
+commands are `merge`, `trim`, and `normalize`.
+
+`merge` reads one or more JSON transcript files, optionally maps input files to
+canonical speakers, runs a registry-selected preprocessing chain, merges
+canonical segments into deterministic chronological order, runs a
+registry-selected postprocessing chain, validates the selected output schema,
+and writes JSON output plus an optional JSON report.
+
+`trim` and `normalize` are artifact-level commands outside the merge pipeline.
+`trim` reads an existing seriatim output artifact and projects it by segment ID.
+`normalize` reads transcript-like JSON and emits one of seriatim's supported
+output schemas. Neither command runs merge preprocessing or postprocessing
+modules.
+
+The supported public output schemas are `seriatim-minimal`,
+`seriatim-intermediate`, and `seriatim-full`. For current command and flag
+details, use [the README](../../README.md) until dedicated `docs/cli.md` and
+`docs/config.md` files exist.
+
+## Core Design Principles
+
+- Keep a hexagonal architecture boundary. Domain models, stage contracts, and
+  deterministic transformations must stay separate from CLI parsing,
+  filesystem access, config loading, reporting, and other external adapters.
+- Keep stages and modules composable. Built-in modules are selected by
+  canonical registry names and implement explicit interfaces for their pipeline
+  role.
+- Preserve deterministic behavior. Given the same inputs, configuration, and
+  version, output ordering, segment IDs, schema validation, and report event
+  ordering should remain stable.
+- Current command execution is sequential. There is no scheduler, worker pool,
+  or concurrent module execution in the implemented pipeline. Any concurrency
+  added later must be bounded, observable, and must not make output handling
+  nondeterministic.
+- Prefer the Go standard library. Third-party dependencies should remain narrow
+  and justified, such as Cobra for CLI structure, YAML parsing, and JSON Schema
+  validation.
+- Document current behavior. Architecture, user, and internal docs must not
+  describe planned features as implemented behavior.
+
+## Architectural Boundaries
+
+Core transcript data belongs in `internal/model` and public artifact contracts
+belong in `schema`. Conversion from internal merged data to public JSON shapes
+belongs at the artifact boundary, not inside CLI code or transformation
+packages.
+
+Pipeline orchestration belongs in `internal/pipeline`. It resolves registered
+modules, validates preprocessing state transitions, executes stages in order,
+collects report events, converts the final transcript, and writes optional
+reports. Built-in adapters and modules are registered from `internal/builtin`.
+
+CLI code in `internal/cli` should parse flags, build validated config values,
+and delegate. `merge` delegates to `pipeline.Run`; `trim` and `normalize`
+perform artifact-level orchestration and delegate deterministic parsing,
+validation, and transformation work to their internal packages.
+
+Config loading and validation belongs in `internal/config`. Filesystem reads and
+writes are adapter concerns and should not spread into pure transformation
+helpers. Existing built-in modules that load configured YAML files must keep
+that I/O narrow and explicit.
+
+Reports belong in `internal/report`. Modules and commands should emit concise
+events for validation findings, corrections, and transformations without
+turning report messages into a duplicate output artifact.
+
+Tests and samples are supporting evidence for behavior. Tests should verify
+stable contracts and edge cases; samples should remain valid examples, not
+hidden architecture dependencies.
+
+## Modules or Stages
+
+The merge pipeline has these implemented stages:
+
+- `InputReader`: reads configured external input into raw transcripts.
+- `Preprocessor`: transforms `PreprocessState` from raw to canonical state.
+- `Merger`: combines canonical transcripts into one merged transcript.
+- `Postprocessor`: transforms or annotates the merged transcript.
+- `OutputWriter`: writes the selected output artifact.
+
+Modules must keep narrow responsibilities, declare their stage through the
+interface they implement, and use explicit config values. Preprocessors must
+declare `Requires()` and `Produces()` states; the runner rejects invalid
+raw/canonical ordering before processing completes.
+
+Modules run in the configured order. Order-affecting modules must run before
+`assign-ids`, and `validate-output` must see final IDs that match the selected
+schema. Accepted and rejected transformations should be deterministic and, when
+observable, recorded through report events.
+
+Transformation helpers should avoid hidden global state. Shared caches, such as
+compiled JSON schemas, must be protected and must not affect output ordering.
+
+## State, Inputs, and Outputs
+
+seriatim is file-based. It reads JSON inputs and optional YAML rule files, then
+writes JSON transcript artifacts and optional JSON reports.
+
+The implemented application has no durable database, daemon state, resume
+state, remote storage, or background job state. Runtime state is held in memory
+for the current command invocation and serialized only through requested output
+and report files.
+
+Input file paths are normalized and validated during config construction.
+`merge` sorts input file paths before processing, then uses stable segment sort
+keys. `trim` preserves transcript order while renumbering retained IDs.
+`normalize` sorts by implemented deterministic keys and assigns fresh IDs.
+
+## Configuration and CLI Boundaries
+
+The CLI surface is an adapter over validated config structs. Cobra command code
+should stay thin: parse flags, account for flag/default precedence, call config
+constructors, and delegate.
+
+Config constructors validate required paths, output parent directories, module
+lists, selected schemas, mutually exclusive trim selector options, and supported
+environment-derived settings. Module name validation is split between config
+where command-specific names are fixed and the pipeline registry where module
+composition is resolved.
+
+Do not duplicate full CLI or config reference material here. Use
+[the README](../../README.md) for the current user-facing reference until the
+canonical `docs/cli.md` and `docs/config.md` files exist.
+
+## Errors, Logging, and Diagnostics
+
+Commands return errors instead of printing inside deep logic. The root command
+silences Cobra usage/error output, and `cmd/seriatim/main.go` prints one error
+to stderr and exits with status `1`.
+
+Validation failures should fail fast with contextual errors. Correctable
+conditions should be deterministic and, where reports are requested, reflected
+as report events. Optional reports contain metadata and ordered events; they are
+not required for command success unless the report file itself cannot be
+written.
+
+The implemented code does not use a logging subsystem. Diagnostics are returned
+as errors or written to optional report JSON. Normalize report events avoid
+embedding transcript text; keep that privacy-oriented behavior when changing
+normalize diagnostics.
+
+## Testing Expectations
+
+`go test ./...` is the repository-wide check. There is currently no Makefile,
+taskfile, linter config, or dedicated documentation check.
+
+When changing config or CLI behavior, inspect `internal/config` and
+`internal/cli` tests. When changing pipeline composition or stage contracts,
+inspect `internal/pipeline` and `internal/builtin` tests. When changing
+correction or annotation modules, inspect the package tests for overlap,
+coalesce, danglers, backchannel, filler, and autocorrect behavior.
+
+When changing artifact-level commands, inspect `internal/trim`,
+`internal/normalize`, and their CLI tests. When changing public output shape or
+schema validation, inspect `schema` and `internal/artifact` tests. Report and
+diagnostic changes should be covered through the command or package tests that
+emit the affected events.
+
+## Dependency Policy
+
+Prefer the Go standard library for parsing, data transformation, concurrency
+primitives, filesystem work, and testing wherever it is reasonable.
+
+Third-party dependencies must be narrow, justified, and preferably de facto
+standard for their purpose. Existing examples include Cobra for CLI structure,
+`gopkg.in/yaml.v3` for YAML files, and `jsonschema/v6` for validating embedded
+public JSON schemas. Avoid broad framework dependencies for behavior that is
+already simple and local.
+
+## Documentation Expectations
+
+Architecture docs must stay aligned with [documentation policy](documentation.md).
+Current-behavior docs must not become aspirational. If code and docs disagree,
+fix the inaccurate current-behavior doc or put planned work under
+`docs/roadmap/`.
+
+Prefer links to canonical docs instead of repeating full CLI, config, schema, or
+operations reference material. Keep examples real, tested where practical, and
+free of secrets or private transcript data.
+
+## Architectural Invariants
+
+- Keep core/domain logic separate from CLI, config, filesystem, reporting, and
+  other adapter concerns.
+- Keep modules narrowly scoped, explicitly configured, and composable by
+  registry name.
+- Preserve deterministic ordering, final segment ID assignment, and schema
+  validation before output acceptance.
+- Keep `trim` and `normalize` artifact-level; do not run merge modules from
+  those commands.
+- Keep public output schemas validated through `schema`.
+- Keep optional reports ordered, concise, and diagnostic.
+- Avoid broad dependencies without a concrete maintainability benefit.
+- Do not document unimplemented behavior outside `docs/roadmap/`.
+
+## Non-Goals
+
+The implemented application does not perform transcription, audio diarization,
+speaker inference from audio or text, summarization, daemon operation, remote
+storage, dynamic external plugin loading, or concurrent pipeline execution.
+
+The architecture policy is not a package-by-package reference, CLI manual,
+config reference, schema reference, or roadmap.

docs/policy/documentation.md

@@ -0,0 +1,356 @@
+# Go Project Documentation Policy
+
+## Purpose
+
+Project documentation must help four audiences:
+
+1. users who need to run the application;
+2. administrators/operators who need to configure and operate it;
+3. developers who need to understand and change it safely;
+4. LLM coding agents that need clear scope, boundaries, and invariants.
+
+Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
+
+## Core Rules
+
+### 1. Keep docs concise
+
+Each document should cover a defined scope and only the essentials for that scope.
+
+Avoid:
+- long background explanations;
+- repeated reference material;
+- implementation detail in user-facing docs;
+- aspirational language outside roadmap docs;
+- verbose examples where one minimal example is clearer.
+
+### 2. Document only implemented behavior outside roadmap files
+
+Unimplemented, planned, aspirational, experimental, or future work may be described only under:
+
+- `docs/roadmap/`
+
+No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
+
+If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
+
+### 3. Use canonical homes
+
+Each type of information should have one canonical location.
+
+Canonical homes:
+
+- project purpose and quickstart: `README.md`
+- development principles: `docs/policy/architecture.md`
+- configuration reference: `docs/config.md`
+- CLI reference: `docs/cli.md`
+- operations and recovery: `docs/operations.md`
+- troubleshooting: `docs/troubleshooting.md`
+- implemented internals: `docs/internal/`
+- future work: `docs/roadmap/`
+- contributor workflow: `docs/policy/development.md`
+- copyable examples: `examples/`
+
+Other files should summarize briefly and link to the canonical source.
+
+### 4. Keep examples real
+
+Examples should be valid, maintained, and free of secrets.
+
+Where practical:
+- example configs should load successfully;
+- example commands should match real CLI syntax;
+- important examples should be covered by tests.
+
+## Documentation Profiles
+
+All projects require:
+
+- `README.md`
+- `docs/policy/architecture.md`
+
+Additional docs depend on the project.
+
+### Small library
+
+Recommended:
+- `docs/policy/development.md`, if contributor conventions are non-obvious
+
+### Simple CLI
+
+Required:
+- `docs/cli.md`
+
+Recommended:
+- `docs/policy/development.md`
+
+### Config-driven CLI
+
+Required:
+- `docs/cli.md`
+- `docs/config.md`
+
+Recommended:
+- `examples/`
+- `docs/policy/development.md`
+
+### Stateful or operator-facing application
+
+Required:
+- `docs/cli.md`, if CLI-based
+- `docs/config.md`, if config-driven
+- `docs/operations.md`
+
+Recommended:
+- `docs/troubleshooting.md`
+- `examples/`
+- `docs/policy/development.md`
+
+### Modular, staged, service-oriented, or orchestration application
+
+Required:
+- `docs/cli.md`, if CLI-based
+- `docs/config.md`, if config-driven
+- `docs/operations.md`
+- `docs/internal/`
+- `docs/policy/development.md`
+
+Recommended:
+- `docs/troubleshooting.md`
+- validated examples under `examples/`
+
+## Required Documents
+
+### README.md
+
+**Audience:** users, administrators, operators
+
+The README is the outward-facing project orientation page.
+
+It should include, in order:
+
+1. concise description;
+2. elevator pitch;
+3. shortest useful command or usage example;
+4. links to targeted docs.
+
+The README should be short. It is not a manual.
+
+The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.)
+
+### docs/policy/architecture.md
+
+**Audience:** developers, LLM coding agents
+
+`docs/policy/architecture.md` is required for every project.
+
+It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
+
+It should include:
+
+- project shape;
+- core design principles;
+- package and boundary philosophy;
+- state/persistence philosophy, if applicable;
+- external integration philosophy, if applicable;
+- error-handling and logging principles;
+- testing expectations;
+- documentation expectations;
+- architectural invariants;
+- explicit non-goals, if useful.
+
+For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
+
+### docs/policy/development.md
+
+**Audience:** developers, LLM coding agents
+
+Required for projects maintained by humans and LLM coding agents.
+
+It should include:
+
+- repository layout;
+- build/test commands;
+- coding conventions;
+- dependency policy;
+- how to add config fields;
+- how to add CLI flags;
+- how to add stages/modules/adapters, if applicable;
+- how to update examples;
+- documentation update expectations.
+
+### docs/config.md
+
+**Audience:** administrators, operators, advanced users
+
+Required for applications with configuration files.
+
+It should include, in order:
+
+1. config file locations and discovery precedence;
+2. minimal working config;
+3. production-oriented config;
+4. full configuration reference;
+5. secrets handling, if applicable;
+6. links to maintained examples.
+
+The full configuration reference should be canonical.
+
+### docs/cli.md
+
+**Audience:** users, administrators, operators
+
+Required for CLI applications.
+
+It should include, in order:
+
+1. shortest useful command;
+2. command overview;
+3. complete flag reference;
+4. common workflows;
+5. diagnostic or recovery commands, if applicable.
+
+Explain when commands are useful, not just their syntax.
+
+### docs/operations.md
+
+**Audience:** administrators, operators
+
+Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures.
+
+It should cover:
+
+- normal workflow;
+- filesystem layout;
+- remote storage layout, if applicable;
+- logs and manifests;
+- resume/retry behavior;
+- cleanup behavior;
+- archive/backup behavior;
+- safe recovery procedures;
+- operational caveats.
+
+### docs/troubleshooting.md
+
+**Audience:** administrators, operators
+
+Recommended once recurring failure modes exist.
+
+Each entry should include:
+
+- symptom;
+- likely cause;
+- diagnostic command or inspection step;
+- safe fix;
+- relevant links.
+
+### docs/internal/
+
+**Audience:** developers, LLM coding agents
+
+Required for modular, staged, service-oriented, or orchestration projects.
+
+This directory describes implemented internal components. It is not the roadmap.
+
+Use one file per major component where useful.
+
+Each component doc should include:
+
+1. purpose;
+2. inputs and outputs;
+3. boundaries;
+4. config fields used;
+5. external adapters used;
+6. state or manifest behavior, if applicable;
+7. skip/resume behavior, if applicable;
+8. failure behavior;
+9. tests to inspect before changing;
+10. architectural invariants.
+
+### docs/roadmap/
+
+**Audience:** maintainers, developers, LLM coding agents
+
+This is the only place for planned, future, aspirational, experimental, or unimplemented work.
+
+Roadmap docs should clearly distinguish:
+
+- proposed work;
+- accepted plans;
+- deferred ideas;
+- rejected ideas;
+- implementation prompts or task breakdowns, if useful.
+
+Roadmap docs should not be confused with current behavior.
+
+### docs/integrations/
+
+**Audience:** developers, LLM coding agents
+
+Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
+
+This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses.
+
+Use one file per integration where useful.
+
+## Examples Directory
+
+Projects with non-trivial configuration or workflows should include `examples/`.
+
+Useful examples include:
+
+- minimal working config;
+- production-oriented config;
+- full annotated config;
+- local development config;
+- remote/object-storage config;
+- minimal session/input file.
+
+Examples should be valid, maintained, tested when practical, and linked from relevant docs.
+
+## Security and Privacy
+
+Docs and examples must not include:
+
+- real API keys;
+- tokens;
+- passwords;
+- private keys;
+- private environment dumps;
+- sensitive user data;
+- raw private transcripts;
+- private infrastructure details unless intentionally public.
+
+Document secret-handling mechanisms, not actual secret values.
+
+## Maintenance Rules
+
+When docs change, verify the affected behavior.
+
+Where practical:
+
+- load example config files in tests;
+- test CLI examples or command parser behavior;
+- validate documented flags against real flags;
+- remove stale references;
+- update links after renames;
+- keep roadmap content out of non-roadmap docs.
+
+If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
+
+Documentation is complete only when it matches the current code.
+
+## Documentation Change Checklist
+
+Before merging documentation changes, verify:
+
+- README is concise and orientation-focused.
+- `docs/policy/architecture.md` describes development principles.
+- Future work appears only under `docs/roadmap/`.
+- User-facing docs avoid unnecessary internals.
+- Developer-facing docs preserve boundaries and invariants.
+- Config examples match the schema.
+- CLI examples match real commands and flags.
+- Defaults appear in the canonical config reference.
+- No secrets or private data are included.
+- Links are accurate.