Plan documentation roadmap cleanup

docs/roadmap/documentation.md

@@ -0,0 +1,579 @@
+# Documentation Roadmap
+
+## Purpose
+
+This roadmap defines the work required to bring this repository's documentation into compliance with `docs/policy/documentation.md` and the current implementation. It is a planning artifact only: future implementation passes should rewrite or create the target documentation files in staged changes, while keeping unimplemented or aspirational material under `docs/roadmap/`.
+
+## Repository Documentation Inventory
+
+Current documentation and examples found during inspection:
+
+- `README.md` — keep and lightly update. It is already short and orientation-focused, but the final pass should verify the quickstart command, links, and wording against the rewritten canonical docs.
+- `docs/cli.md` — keep and rewrite. It is the canonical CLI reference, but it mixes command reference, workflows, HTTP upload details, output schema, and recovery links; rewrite around current command behavior in `internal/cli` and output behavior in `internal/app`.
+- `docs/config.md` — keep and rewrite. It is the canonical user-facing config reference and contains most implemented fields, but it is long and should be rebuilt from `internal/config` structs, defaults, validation, examples, and backend wiring.
+- `docs/operations.md` — keep and rewrite. It is the canonical operations/recovery doc, but it currently carries broad reference material that should link to `docs/cli.md`, `docs/config.md`, and component docs instead of duplicating them.
+- `docs/troubleshooting.md` — keep and rewrite. It exists and is symptom-oriented, but it is large enough to require a clean pass against current error behavior and should avoid becoming another operations manual.
+- `docs/policy/architecture.md` — keep and lightly update only if the code/docs rewrite reveals stale architectural statements. It is a controlling policy document, not user documentation.
+- `docs/policy/development.md` — keep and lightly update. It is the contributor workflow policy and should remain concise; verify commands and package list after documentation migration.
+- `docs/policy/documentation.md` — keep unchanged unless documentation policy itself needs a separate policy change. It is controlling for this roadmap.
+- `docs/internal/app.md` — keep and rewrite. It reflects recent upload coordination and run orchestration, but should be normalized to the required internal-doc sections from `docs/policy/documentation.md`.
+- `docs/internal/bundle.md` — keep and lightly update. It is close to the implemented component contract; verify path rules, nested manifest handling, and `pkg/bundle` boundary.
+- `docs/internal/config.md` — keep and lightly update. It is close to current config internals; verify defaults, HTTP upload source-only behavior, secret resolver behavior, and backend support.
+- `docs/internal/ingest.md` — keep and lightly update. It matches archive staging behavior after cleanup; verify tar entry types, root manifest rules, failure cleanup, and limits.
+- `docs/internal/link.md` — keep and lightly update. It is concise and maps to `internal/link`.
+- `docs/internal/notify.md` — keep and lightly update. It should remain explicit that only the no-op notifier is implemented.
+- `docs/internal/publish.md` — keep and rewrite. It is important and mostly current, but should be checked against `internal/publish` safety, link, transform, and force behavior.
+- `docs/internal/state.md` — keep and lightly update. It should be checked against state schema, URL validation, comparison outcomes, and output record rules.
+- `docs/internal/storage.md` — keep and rewrite or split. It covers storage plus local, SSH, S3, and fake backend behavior; consider keeping shared storage in this file and moving external backend protocol notes to integration docs.
+- `docs/internal/transform.md` — keep and lightly update. It should link to the Markdown integration doc and reflect only implemented transforms.
+- `docs/integrations/markdown.md` — keep and lightly update. It is the only current integration doc; verify it against `internal/transform/markdown` and `go.mod` Goldmark dependency.
+- `docs/roadmap/audit.md` — keep or delete after migration review. It is roadmap-scoped and may remain as an audit artifact, but completed cleanup findings should not be the only record of current behavior.
+- `docs/roadmap/cleanup.md` — delete or replace with a concise status note after all cleanup has landed. It describes completed implementation work and should not remain as the canonical description of current behavior.
+- `docs/roadmap/http.md` — delete or archive under roadmap only if it still contains deferred HTTP upload work. Implemented HTTP upload behavior belongs in current docs; stale planning prose should not be retained as current guidance.
+- `docs/roadmap/implementation.md` — delete or archive under roadmap only if it still contains deferred feature work. Implemented behavior belongs in current docs.
+- `docs/roadmap/documentation.md` — create. This file is the action plan for the documentation migration.
+- `examples/local-to-local.yml` — keep and lightly update. Minimal local config example; should remain load-tested.
+- `examples/local-publish.yml` — keep and lightly update. Runnable local source publication example used by README, CLI, and operations quickstarts.
+- `examples/local-html.yml` — keep and lightly update. Runnable local sidecar HTML publication example.
+- `examples/local-index.yml` — keep and lightly update. Runnable local `index.html` publication example.
+- `examples/fan-out.yml` — keep and lightly update. Runnable local fan-out example.
+- `examples/archive-and-latest.yml` — keep and lightly update. Runnable archive plus fixed latest example.
+- `examples/http-upload-local.yml` — keep and lightly update. Runnable local HTTP upload server config when a token is supplied.
+- `examples/ssh-destination.yml` — keep and lightly update. Environment-gated SSH/SFTP example; should remain load-tested but not presented as runnable without editing endpoint details.
+- `examples/s3-destination.yml` — keep and lightly update. Environment-gated S3-compatible example; should remain load-tested but not presented as runnable without editing endpoint and credentials.
+- `examples/source-bundle/manifest.json`, `examples/source-bundle/report.md`, `examples/source-bundle/summary.txt` — keep and lightly update only if manifest rules or fixture content changes. This is the maintained source bundle example used by examples and docs.
+
+No current documentation file obviously needs to be moved before rewriting. New integration docs should be created under `docs/integrations/` where the integration contract is important and already implemented.
+
+## Policy Compliance Assessment
+
+Required documents for this modular, CLI/config-driven, stateful orchestration project are present:
+
+- `README.md`
+- `docs/policy/architecture.md`
+- `docs/policy/development.md`
+- `docs/cli.md`
+- `docs/config.md`
+- `docs/operations.md`
+- `docs/internal/`
+
+Recommended documents are also present:
+
+- `docs/troubleshooting.md`
+- `examples/`
+
+Recommended integration coverage is incomplete. `docs/integrations/markdown.md` exists, but the project also has implemented integration contracts for source bundle manifests, destination state files, SSH/SFTP, S3-compatible storage, and the HTTP upload API. These are documented in current user/internal docs, but not consistently in `docs/integrations/`.
+
+Potential compliance issues to resolve during the rewrite:
+
+- Several current docs are accurate enough to use as input, but too much reference material is duplicated across `docs/cli.md`, `docs/config.md`, `docs/operations.md`, and `docs/internal/*.md`.
+- Completed roadmap docs under `docs/roadmap/` may read as active implementation plans. Keep only roadmap content that is still future/deferred, or explicitly convert completed roadmap material into current docs and delete obsolete plans.
+- `README.md` is concise, but should link only to canonical current docs and avoid implying that roadmap content is required to understand implemented behavior.
+- `docs/config.md` is the right canonical home for config fields, defaults, and examples, but should be rebuilt from `internal/config/config.go`, `internal/config/defaults.go`, and `internal/config/validate.go` so field lists and defaults remain defensible.
+- `docs/cli.md` should be validated against command help strings and parser tests in `internal/cli`, including `manifest create <path> --id x`, `manifest create --id x <path>`, missing flag values, invalid `--format`, and help output.
+- `docs/operations.md` should focus on operating, recovery, safety, storage layout, retries, and HTTP upload lifecycle, not full flag or config reference.
+- `docs/troubleshooting.md` should remain symptom-first and should link to canonical config/CLI/operations docs rather than repeating broad reference sections.
+- `docs/internal/storage.md` currently includes adapter details that may be clearer as short integration docs for SSH/SFTP and S3-compatible storage, while keeping internal storage interface boundaries in `docs/internal/storage.md`.
+- Examples are present and load-tested by `internal/config/load_test.go`; future docs should link to those examples and preserve the distinction between runnable local examples and environment-gated remote examples.
+- Link verification is manual today; no automated Markdown link checker was found.
+
+No clear non-roadmap current document was found describing obviously unimplemented features as available. However, the rewrite should explicitly check for stale planning terms such as `future`, `planned`, `deferred`, `experimental`, `deprecated`, and completed roadmap-only feature descriptions outside `docs/roadmap/`.
+
+Roadmap file disposition for the documentation migration:
+
+- `docs/roadmap/http.md` should remain for now. It already describes only deferred HTTP upload extensions and links current behavior back to canonical docs.
+- `docs/roadmap/implementation.md` should remain for now. It duplicates some deferred HTTP upload extension tracking, so the final roadmap hygiene pass should either merge it into `docs/roadmap/http.md` or delete it after confirming no unique deferred work is lost.
+- `docs/roadmap/audit.md` should remain until the current docs are rewritten. It is a historical audit artifact, not current behavior, and should be deleted or reduced in the final roadmap hygiene pass if it no longer contains useful deferred work.
+- `docs/roadmap/cleanup.md` should remain until the current docs are rewritten. It is a completed cleanup plan and should be deleted in the final roadmap hygiene pass after all implemented cleanup behavior is represented in current docs.
+- `docs/roadmap/documentation.md` should remain as the active documentation migration plan until this documentation migration is complete.
+
+## Target Documentation Set
+
+The desired final documentation tree should include the files below.
+
+### `README.md`
+
+- Audience: users, administrators, operators.
+- Purpose: project orientation and quickest useful command.
+- Canonical scope: concise project purpose, elevator pitch, shortest useful command, links to canonical docs.
+- Recommended outline:
+  - `# distributor`
+  - one-paragraph description
+  - shortest useful command using `examples/local-publish.yml`
+  - short list of links to CLI, config, operations, troubleshooting, and producer/integration contract docs
+- Source-of-truth repository areas: `cmd/distributor/main.go`, `internal/cli`, `examples/local-publish.yml`, `docs/policy/documentation.md`.
+- Acceptance criteria: under one page; no full config reference; no future behavior outside a short pointer that roadmap content exists under `docs/roadmap/`; all links resolve.
+
+### `docs/cli.md`
+
+- Audience: users, administrators, operators.
+- Purpose: canonical command syntax, flags, workflows, and output behavior.
+- Canonical scope: CLI command reference and command-level workflows only.
+- Recommended outline:
+  - shortest useful command
+  - command overview
+  - flag reference by command
+  - mode rules for `validate` and `inspect`
+  - `manifest create` positional parsing examples
+  - common workflows
+  - output and exit behavior
+  - links to config, operations, troubleshooting
+- Source-of-truth repository areas: `internal/cli/*.go`, `internal/cli/root_test.go`, `internal/app/output.go`, `internal/app/run_output.go`, `internal/app/validate.go`, `internal/app/inspect.go`, `internal/app/manifest.go`.
+- Acceptance criteria: every documented flag exists in `internal/cli`; no undocumented command appears; examples match parser tests; output descriptions match app tests; no config field reference beyond command usage and links.
+
+### `docs/config.md`
+
+- Audience: administrators, operators, advanced users.
+- Purpose: canonical user-facing YAML configuration reference.
+- Canonical scope: config file location, minimal config, production-oriented config, field reference, defaults, secrets, maintained examples.
+- Recommended outline:
+  - config file location and precedence for `--config`
+  - minimal local config
+  - production local config
+  - HTTP upload source config
+  - backend reference: local, SSH, S3, HTTP upload source
+  - publish, transform, path mapping, links, transfer policy
+  - field reference
+  - size and duration values
+  - defaults
+  - secrets and credential resolution
+  - examples list
+- Source-of-truth repository areas: `internal/config/config.go`, `internal/config/defaults.go`, `internal/config/validate.go`, `internal/config/load.go`, `internal/config/secrets.go`, `internal/app/backends.go`, `internal/app/serve.go`, `internal/config/load_test.go`, examples.
+- Acceptance criteria: every field listed exists in config structs; every default matches `ApplyDefaults`; unknown-field rejection is documented; `http_upload` is source-only; examples load with `go test ./internal/config`; no execution support is claimed beyond local, SSH/SFTP, S3, and HTTP upload server behavior.
+
+### `docs/operations.md`
+
+- Audience: administrators/operators.
+- Purpose: operating workflows, storage layout, safety, recovery, and upload lifecycle.
+- Canonical scope: how to operate implemented workflows safely, not exhaustive CLI/config reference.
+- Recommended outline:
+  - normal validation and dry-run workflow
+  - local publication workflow
+  - HTTP upload workflow and lifecycle
+  - filesystem/storage layout
+  - destination state and retry behavior
+  - dry-run and forced replacement safety
+  - remote backend operational notes
+  - logs/output interpretation
+  - safe cleanup and recovery procedures
+  - links to CLI, config, troubleshooting
+- Source-of-truth repository areas: `internal/app/run.go`, `internal/app/upload_coordinator.go`, `internal/app/upload_http.go`, `internal/publish`, `internal/state`, `internal/storage`, adapter packages, `internal/app/run_test.go`, `internal/app/upload_*_test.go`.
+- Acceptance criteria: describes only implemented operational behavior; does not duplicate full config reference; force/delete warnings align with publish and storage tests; upload status retention and memory-only behavior match code.
+
+### `docs/troubleshooting.md`
+
+- Audience: administrators/operators.
+- Purpose: symptom-oriented diagnostics and safe fixes.
+- Canonical scope: common observed failure modes, causes, diagnostic commands, safe fixes, and links.
+- Recommended outline:
+  - config load and unknown field failures
+  - source bundle validation failures
+  - destination state/conflict/unmanaged content failures
+  - forced replacement safety failures
+  - SSH host key/auth failures
+  - S3 credential/connectivity/path failures
+  - HTTP upload authentication/content/queue/status failures
+  - JSON output partial-result behavior
+- Source-of-truth repository areas: error strings and tests in `internal/config`, `pkg/bundle`, `internal/bundle`, `internal/publish`, `internal/state`, `internal/adapters/ssh`, `internal/adapters/s3`, `internal/app/upload_*_test.go`, `internal/cli/root_test.go`.
+- Acceptance criteria: entries follow symptom/likely cause/diagnostic/safe fix/link format; no broad reference duplication; no secret values; links resolve.
+
+### `docs/policy/architecture.md`
+
+- Audience: developers and LLM coding agents.
+- Purpose: controlling architecture policy and invariants.
+- Canonical scope: development architecture, boundaries, package ownership, safety invariants.
+- Recommended outline: keep existing outline unless a separate architecture-policy update is needed.
+- Source-of-truth repository areas: package layout, current implemented workflow, policy decisions in tests.
+- Acceptance criteria: remains policy-oriented; does not become user docs; any examples are representative and current.
+
+### `docs/policy/development.md`
+
+- Audience: developers and LLM coding agents.
+- Purpose: contributor workflow and coding conventions.
+- Canonical scope: repo layout, commands, coding rules, test/doc expectations.
+- Recommended outline: keep existing outline and verify package list and test commands.
+- Source-of-truth repository areas: repository tree, `go.mod`, tests, examples, AGENTS.md.
+- Acceptance criteria: commands work; package list matches current tree; documentation policy links remain accurate.
+
+### `docs/policy/documentation.md`
+
+- Audience: developers and LLM coding agents.
+- Purpose: controlling documentation policy.
+- Canonical scope: documentation layout, audience, canonical homes, maintenance rules.
+- Recommended outline: no migration changes unless policy itself changes.
+- Source-of-truth repository areas: policy intent only.
+- Acceptance criteria: unchanged in implementation passes unless explicitly requested.
+
+### `docs/internal/app.md`
+
+- Audience: developers and LLM coding agents.
+- Purpose: implemented app orchestration, command use cases, runtime setup, run reports, upload coordination, and boundaries.
+- Canonical scope: `internal/app` only.
+- Recommended outline:
+  - purpose
+  - inputs and outputs
+  - boundaries
+  - config fields used
+  - adapters used
+  - run/report behavior
+  - upload server and coordinator behavior
+  - failure behavior
+  - tests to inspect
+  - invariants
+- Source-of-truth repository areas: `internal/app/*.go`, especially `runtime.go`, `run.go`, `run_destination.go`, `run_output.go`, `serve.go`, `upload_coordinator.go`, `upload_http.go`, app tests.
+- Acceptance criteria: no references to removed duplicate-run coordinator; upload behavior matches staging-before-acceptance; app docs link to config/CLI docs for user syntax.
+
+### `docs/internal/bundle.md`
+
+- Audience: developers and LLM coding agents.
+- Purpose: source bundle discovery and validation internals.
+- Canonical scope: `internal/bundle` and its boundary with `pkg/bundle`.
+- Recommended outline: purpose; inputs/outputs; source manifest behavior; discovery; validation; boundaries; failure behavior; tests; invariants.
+- Source-of-truth repository areas: `internal/bundle`, `pkg/bundle`, `internal/bundle/*_test.go`, `pkg/bundle/*_test.go`.
+- Acceptance criteria: path safety and reserved manifest/state rules match code; nested manifest rejection is documented; producer API boundary is clear.
+
+### `docs/internal/config.md`
+
+- Audience: developers and LLM coding agents.
+- Purpose: config loading/defaulting/validation internals and secret resolver boundary.
+- Canonical scope: `internal/config` behavior, not full user config reference.
+- Recommended outline: purpose; inputs/outputs; loading flow; defaults ownership; validation responsibilities; executable support boundary; secrets; failure behavior; tests; invariants.
+- Source-of-truth repository areas: `internal/config/*.go`, `internal/config/*_test.go`, `internal/app/runtime.go`, `internal/app/backends.go`.
+- Acceptance criteria: links to `docs/config.md` for user-facing field reference; no duplicate full field table; source-only HTTP upload boundary is clear.
+
+### `docs/internal/ingest.md`
+
+- Audience: developers and LLM coding agents.
+- Purpose: upload archive staging internals.
+- Canonical scope: `internal/ingest` only.
+- Recommended outline: purpose; inputs/outputs; accepted content types; extraction rules; bundle validation; cleanup/failure behavior; tests; invariants.
+- Source-of-truth repository areas: `internal/ingest/archive.go`, `internal/ingest/archive_test.go`, `pkg/bundle` validation tests.
+- Acceptance criteria: documents root manifest rule, nested manifest rejection, duplicate file rejection, symlink/hardlink/special-entry rejection, safe directories, and limits exactly as implemented.
+
+### `docs/internal/link.md`
+
+- Audience: developers and LLM coding agents.
+- Purpose: shared URL validation policy.
+- Canonical scope: `internal/link`.
+- Recommended outline: purpose; inputs/outputs; validation behavior; boundaries; tests; invariants.
+- Source-of-truth repository areas: `internal/link`, callers in `internal/config`, `internal/state`, `internal/publish`.
+- Acceptance criteria: concise; URL rules match tests; link construction remains outside this package.
+
+### `docs/internal/notify.md`
+
+- Audience: developers and LLM coding agents.
+- Purpose: notification interface and no-op implementation.
+- Canonical scope: implemented notification behavior only.
+- Recommended outline: purpose; inputs/outputs; current behavior; boundaries; failure behavior; tests; invariants.
+- Source-of-truth repository areas: `internal/notify`, `internal/app/run_notify.go`, `internal/app/run_test.go`.
+- Acceptance criteria: does not document external notifier config/adapters as implemented.
+
+### `docs/internal/publish.md`
+
+- Audience: developers and LLM coding agents.
+- Purpose: publish planning/execution contract and safety behavior.
+- Canonical scope: `internal/publish`.
+- Recommended outline: purpose; inputs/outputs; actions; output planning; link planning; transfer policy handling; safety/deletion; execution order; failure behavior; boundaries; tests; invariants.
+- Source-of-truth repository areas: `internal/publish/*.go`, `internal/publish/*_test.go`, `internal/state`, `internal/storage`, `internal/app/run_destination.go`.
+- Acceptance criteria: force behavior and managed deletion are exact; app-owned selection/path mapping boundary is clear; no CLI/config reference duplication beyond naming consumed policies.
+
+### `docs/internal/state.md`
+
+- Audience: developers and LLM coding agents.
+- Purpose: destination state schema, parsing, validation, and comparison internals.
+- Canonical scope: `internal/state`.
+- Recommended outline: purpose; inputs/outputs; state schema; output records; link metadata; comparison outcomes; failure behavior; boundaries; tests; invariants.
+- Source-of-truth repository areas: `internal/state`, state tests, publish/state callers.
+- Acceptance criteria: `.distributor.json` remains destination state canonical; comparison does not imply mutation; URL/output validation matches code.
+
+### `docs/internal/storage.md`
+
+- Audience: developers and LLM coding agents.
+- Purpose: storage interface, logical paths, typed errors, traversal, deletion boundaries, fake backend.
+- Canonical scope: `internal/storage` and backend interface semantics.
+- Recommended outline: purpose; inputs/outputs; boundaries; logical paths; errors; traversal helpers; deletion helpers; fake backend; tests; invariants.
+- Source-of-truth repository areas: `internal/storage`, `internal/storage/fake`, adapter packages for interface compliance.
+- Acceptance criteria: keeps adapter protocol details brief and links to integration docs for SSH/SFTP and S3; path and deletion safety match tests.
+
+### `docs/internal/transform.md`
+
+- Audience: developers and LLM coding agents.
+- Purpose: transform registry and Markdown transform boundary.
+- Canonical scope: `internal/transform` and implemented Markdown transform.
+- Recommended outline: purpose; inputs/outputs; registry; Markdown behavior summary; failure behavior; boundaries; tests; invariants; link to `docs/integrations/markdown.md`.
+- Source-of-truth repository areas: `internal/transform`, `internal/transform/markdown`, publish callers.
+- Acceptance criteria: only Markdown-to-HTML is documented as implemented; generated output metadata matches code.
+
+### `docs/integrations/source-bundle.md`
+
+- Audience: producer developers and distributor maintainers.
+- Purpose: external producer-facing source bundle and `manifest.json` contract.
+- Canonical scope: implemented source bundle file format and `pkg/bundle` helper API overview.
+- Recommended outline: source bundle layout; manifest schema; path/digest/timestamp rules; reserved paths; local producer helper package; validation command; examples.
+- Source-of-truth repository areas: `pkg/bundle`, `internal/bundle`, `examples/source-bundle`, `docs/policy/architecture.md` source bundle contract.
+- Acceptance criteria: no destination config or publish policy; schema matches `pkg/bundle`; links to CLI validate and examples.
+
+### `docs/integrations/destination-state.md`
+
+- Audience: operators, destination consumers, maintainers.
+- Purpose: external destination `.distributor.json` state file contract.
+- Canonical scope: implemented destination state JSON format and compatibility expectations.
+- Recommended outline: file purpose; schema fields; embedded source manifest; outputs; links; comparison-relevant fields; safe manual inspection guidance.
+- Source-of-truth repository areas: `internal/state`, `internal/publish/output.go`, `docs/policy/architecture.md` destination state contract.
+- Acceptance criteria: does not instruct users to hand-edit state as normal workflow; describes implemented fields only.
+
+### `docs/integrations/http-upload.md`
+
+- Audience: producer developers and operators integrating with the upload API.
+- Purpose: HTTP upload API contract.
+- Canonical scope: implemented HTTP routes, auth, content types, responses, status lifecycle, and error boundary.
+- Recommended outline: configured token-to-pipeline mapping; routes; auth header; accepted content types; archive requirements; response shapes; status lifecycle; retention; security notes.
+- Source-of-truth repository areas: `internal/app/upload_http.go`, `internal/app/upload_coordinator.go`, `internal/ingest`, upload tests, `examples/http-upload-local.yml`.
+- Acceptance criteria: documents staging/validation before run id; no durable queue claim; no TLS/rate-limit/browser UI claims.
+
+### `docs/integrations/markdown.md`
+
+- Audience: maintainers of transform behavior.
+- Purpose: Goldmark integration contract.
+- Canonical scope: implemented Markdown rendering behavior and dependency notes.
+- Recommended outline: dependency; renderer options; raw HTML behavior; sidecar/index mapping; compatibility notes; tests.
+- Source-of-truth repository areas: `go.mod`, `internal/transform/markdown`, markdown tests.
+- Acceptance criteria: describes only behavior this project configures or depends on.
+
+### `docs/integrations/ssh-sftp.md`
+
+- Audience: operators and maintainers integrating SSH/SFTP storage.
+- Purpose: SSH/SFTP backend contract and operational compatibility notes.
+- Canonical scope: implemented SSH/SFTP protocol behavior, auth sources, host key policy, deletion safety.
+- Recommended outline: libraries/protocol; config fields link; auth order; host key policies; known hosts behavior; path root; dry-run host key behavior; tests/integration gating.
+- Source-of-truth repository areas: `internal/adapters/ssh`, `internal/app/backends.go`, `internal/config/ssh.go`, SSH adapter tests.
+- Acceptance criteria: no password auth claim; no shell/rsync behavior; documents opt-in integration tests only if present in tests.
+
+### `docs/integrations/s3.md`
+
+- Audience: operators and maintainers integrating S3-compatible storage.
+- Purpose: S3-compatible backend contract and operational compatibility notes.
+- Canonical scope: implemented AWS SDK usage, endpoint/bucket/prefix behavior, credentials, object semantics, deletion safety.
+- Recommended outline: dependency; endpoint/region/force path style; credentials; prefix/object key semantics; overwrite and content type behavior; tests/integration gating.
+- Source-of-truth repository areas: `internal/adapters/s3`, `internal/app/backends.go`, `internal/config/s3.go`, S3 adapter tests.
+- Acceptance criteria: no unsupported cloud-specific features; explicit credential resolver boundary is correct.
+
+### `examples/`
+
+- Audience: users, administrators, operators, developers.
+- Purpose: copyable, valid configs and a source bundle fixture.
+- Canonical scope: maintained examples for implemented behavior.
+- Recommended outline: no separate README required unless examples become hard to navigate; link from README/config/CLI/operations.
+- Source-of-truth repository areas: `examples/*.yml`, `examples/source-bundle`, `internal/config/load_test.go`.
+- Acceptance criteria: all example YAML files load in tests; local examples can be run without external services; remote examples are clearly environment-gated and contain no secrets.
+
+### `docs/roadmap/`
+
+- Audience: maintainers, developers, LLM coding agents.
+- Purpose: future/deferred work and implementation roadmaps only.
+- Canonical scope: not current behavior.
+- Recommended final contents: keep `docs/roadmap/documentation.md` until migration completes; keep any roadmap docs that still describe deferred work; delete completed roadmaps once their behavior is documented in current docs.
+- Source-of-truth repository areas: current roadmap files plus current implementation.
+- Acceptance criteria: no completed behavior is documented only in roadmap files; active roadmap files clearly distinguish future/deferred work from current behavior.
+
+## File-by-File Rewrite Guidance
+
+### `README.md`
+
+Cover the project purpose, the shortest local run command, and links to canonical docs. Avoid full CLI/config details, implementation history, and feature roadmap summaries. Inspect `examples/local-publish.yml`, `internal/cli/root.go`, and `docs/cli.md` after rewrite. Do not carry forward any wording that suggests roadmap docs are required to understand implemented behavior.
+
+### `docs/cli.md`
+
+Cover the exact commands implemented in `internal/cli`: `version`, `run`, `serve`, `validate`, `inspect`, and `manifest create`. Include `--format text|json`, `--config`, `--dry-run`, `--force`, source diagnostic flags, and manifest creation flags. Explicitly include both manifest positional forms: `manifest create <path> --id x` and `manifest create --id x <path>`. Avoid duplicating full YAML reference or operations recovery procedures. Link to `docs/config.md` for fields and `docs/operations.md` for safety/recovery. Inspect `internal/cli/*.go` and `internal/cli/root_test.go` before writing.
+
+### `docs/config.md`
+
+Rewrite from the config structs, defaults, validation, and load tests. Cover all implemented fields, defaults, accepted values, source/destination backend differences, HTTP upload source-only behavior, S3 credentials, SSH host key policy, path mapping, links, publish/transform, transfer policy, secrets, and examples. Avoid internal package implementation details except where needed to explain user-visible behavior. Link to integration docs for protocol-level details. Inspect `internal/config/config.go`, `internal/config/defaults.go`, `internal/config/validate.go`, `internal/config/secrets.go`, `internal/app/backends.go`, and `internal/config/load_test.go`.
+
+### `docs/operations.md`
+
+Rewrite as an operations guide. Cover validation-before-run, dry-run, normal publish, fixed/latest destination safety, destination state, retry behavior, forced replacement, HTTP upload server lifecycle, memory-only status, and remote backend operational caveats. Avoid listing every config field or every CLI flag. Link to CLI/config/troubleshooting and integration docs. Inspect app run/upload tests, publish safety tests, storage deletion tests, and adapter tests.
+
+### `docs/troubleshooting.md`
+
+Rewrite around symptoms and safe fixes. Keep entries concise and link out. Cover config parsing/validation, source manifest failures, reserved/unsafe paths, digest mismatches, destination unmanaged/conflict/newer state, force requirements, SSH host key/auth issues, S3 credential/connectivity issues, HTTP upload `401`/`413`/`415`/`503`/`400`/`404`, and JSON partial results. Avoid broad tutorials and avoid exposing secret values. Inspect tests where errors are asserted.
+
+### `docs/internal/*.md`
+
+Normalize each internal doc to the policy-required component structure. Keep boundaries explicit: CLI parsing in `internal/cli`, orchestration in `internal/app`, config in `internal/config`, source manifests in `pkg/bundle`/`internal/bundle`, destination state in `internal/state`, storage interface in `internal/storage`, adapter protocol behavior in adapters, transforms in `internal/transform`, publication in `internal/publish`, upload staging in `internal/ingest`, URL policy in `internal/link`, notification interface in `internal/notify`. Avoid user-facing how-to content except links to canonical user docs. Do not document unimplemented notification adapters or future backends.
+
+### `docs/integrations/*.md`
+
+Keep integration docs concise and contract-focused. Create new entries only for implemented integrations. Do not copy full external documentation. Document only how this project uses the integration and which code/tests define compatibility. Link from config, operations, and internal docs where useful.
+
+### `docs/roadmap/*.md`
+
+After current docs are rewritten, review existing roadmap files. Delete or replace completed roadmap documents when they no longer describe future/deferred work. Preserve genuinely deferred work under `docs/roadmap/`. Avoid using roadmap files as changelogs.
+
+## Examples Plan
+
+`examples/` exists and should remain the canonical home for maintained examples.
+
+Recommended examples:
+
+- `examples/source-bundle/` — purpose: minimal valid source bundle fixture. Validity check: `go run ./cmd/distributor validate examples/source-bundle` or existing bundle tests. Link from README, CLI, source bundle integration doc.
+- `examples/local-to-local.yml` — purpose: minimal local config shape. Validity check: `go test ./internal/config`; optional dry-run after replacing absolute paths with local temp paths is not directly runnable as-is. Link from config reference as a minimal production-shaped example.
+- `examples/local-publish.yml` — purpose: runnable local source publication. Validity check: `go run ./cmd/distributor run --config examples/local-publish.yml --dry-run`; already load-tested. Link from README, CLI, operations.
+- `examples/local-html.yml` — purpose: runnable sidecar Markdown-to-HTML publication. Validity check: dry-run command; load-tested. Link from config HTML section and CLI workflows.
+- `examples/local-index.yml` — purpose: runnable `index.html` Markdown publication. Validity check: dry-run command; load-tested. Link from config HTML section.
+- `examples/fan-out.yml` — purpose: one source to multiple local destinations. Validity check: dry-run command; load-tested. Link from operations and architecture/pipeline model if useful.
+- `examples/archive-and-latest.yml` — purpose: archive plus fixed latest destination. Validity check: dry-run command; load-tested. Link from operations fixed-path safety and config path mapping.
+- `examples/http-upload-local.yml` — purpose: local HTTP upload server with token environment reference. Validity check: `go test ./internal/config`; optional manual `serve` with `DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN`. Link from CLI, config HTTP upload, operations, HTTP upload integration doc.
+- `examples/ssh-destination.yml` — purpose: environment-gated SSH/SFTP destination shape. Validity check: `go test ./internal/config`; live execution only after replacing endpoint/path/key fields. Link from config SSH and SSH integration doc.
+- `examples/s3-destination.yml` — purpose: environment-gated S3-compatible destination shape. Validity check: `go test ./internal/config`; live execution only after replacing endpoint/bucket/credentials. Link from config S3 and S3 integration doc.
+
+Do not invent examples for unimplemented notification adapters, durable queues, TLS, zstd uploads, browser UI, or non-Markdown transforms.
+
+## Internal Documentation Plan
+
+Recommended internal component docs for implemented components:
+
+- Component: app orchestration. Path: `docs/internal/app.md`. Purpose: top-level use cases, runtime setup, run report, upload coordination. Inputs/outputs: app option structs, config, writers, reports, HTTP requests/status records. Boundaries: no config schema ownership, no adapter internals, no manifest rules. Config fields used: config path, server HTTP, source/destination backend fields through runtime setup. Adapters used: local, SSH/SFTP, S3 through backend factory. Failure behavior: fatal setup errors, partial destination failures, upload admission/staging/status errors. Tests: `internal/app/*_test.go`, `internal/cli/root_test.go`. Invariants: fan-out independence, same run path after selection, dry-run safety, no secret leakage.
+- Component: config. Path: `docs/internal/config.md`. Purpose: YAML structs, defaults, validation, secret resolver. Inputs/outputs: YAML path to `Config`, `Environment`, validation errors. Boundaries: no backend opening or upload execution. Config fields used: all config fields. Adapters used: none directly. Failure behavior: load/parse/validation/secret errors. Tests: `internal/config/*_test.go`, `internal/app/runtime_test.go`. Invariants: known fields only, defaults before validation, source-only HTTP upload.
+- Component: source bundles. Path: `docs/internal/bundle.md`. Purpose: storage-backed discovery/validation and public manifest contract delegation. Inputs/outputs: storage backend trees to validated bundle list. Boundaries: concrete adapters hidden behind storage; producer API in `pkg/bundle`. Config fields used: source root indirectly. Adapters used: storage interface only. Failure behavior: manifest parse/validation, file stat/read/digest failures. Tests: `internal/bundle`, `pkg/bundle`. Invariants: one root manifest per bundle, nested manifests rejected, path safety.
+- Component: ingestion. Path: `docs/internal/ingest.md`. Purpose: archive content-type validation, extraction, source bundle staging. Inputs/outputs: upload body/content type/limits/staging path/run id to committed local bundle root. Boundaries: no auth, queueing, HTTP routing, or publish. Config fields used: effective staging path and upload size supplied by app. Adapters used: local filesystem only. Failure behavior: unsupported content type, oversize, unsafe tar entries, validation failure, cleanup. Tests: `internal/ingest/archive_test.go`. Invariants: invalid archives cannot commit staged roots.
+- Component: publish. Path: `docs/internal/publish.md`. Purpose: plan/execute destination actions and safety. Inputs/outputs: source bundle/backend, destination backend/state/policies to plan and writes. Boundaries: no CLI parsing/config loading/source selection. Config fields used: publish, transform, transfer, links after app/config validation. Adapters used: storage interface and transform resolver. Failure behavior: conflicts, unmanaged content, output collisions, write/delete failures. Tests: `internal/publish`, `internal/app/run_test.go`. Invariants: state after outputs, bounded deletion, force explicit.
+- Component: state. Path: `docs/internal/state.md`. Purpose: `.distributor.json` schema and comparison. Inputs/outputs: JSON/current source identity to comparison outcome. Boundaries: no storage mutation. Config fields used: none directly. Adapters used: none. Failure behavior: invalid JSON/schema/URLs/output paths/conflicts. Tests: `internal/state`. Invariants: state is sentinel, comparison pure, embedded manifests validated.
+- Component: storage. Path: `docs/internal/storage.md`. Purpose: storage interface, logical path policy, errors, traversal, deletion helpers, fake backend. Inputs/outputs: logical paths and backend operations. Boundaries: adapters own protocol specifics. Config fields used: none directly. Adapters used: local/SSH/S3 implement interface; fake for tests. Failure behavior: typed storage errors, path validation, traversal stop. Tests: `internal/storage`, `internal/storage/fake`, adapter tests. Invariants: slash paths, root confinement, bounded deletion.
+- Component: transform. Path: `docs/internal/transform.md`. Purpose: transform interface/registry and Markdown generation summary. Inputs/outputs: validated source bundle/source backend/options to generated output records. Boundaries: no publication or state writing. Config fields used: transform policy supplied by publish. Adapters used: source storage backend for reads. Failure behavior: unresolved transform, invalid index input, read/render errors. Tests: `internal/transform`, `internal/transform/markdown`. Invariants: source immutability, deterministic output metadata.
+- Component: link. Path: `docs/internal/link.md`. Purpose: shared HTTP URL validation. Inputs/outputs: URL string to validation result. Boundaries: no URL construction. Config fields used: `links.base_url` via config. Adapters used: none. Failure behavior: parse/scheme/host/query/fragment validation. Tests: `internal/link`, callers. Invariants: same URL policy for config and persisted state.
+- Component: notify. Path: `docs/internal/notify.md`. Purpose: notification interface and no-op behavior. Inputs/outputs: notification event to error. Boundaries: no configured external notifications. Config fields used: none currently. Adapters used: none. Failure behavior: no-op returns context cancellation only. Tests: `internal/notify`, app notify tests. Invariants: only successful publish/replacement notifies; dry-run never notifies.
+
+Only recommend internal docs for implemented components. Do not add docs for nonexistent workflow engines, notification adapters, durable queues, or non-Markdown transforms.
+
+## Integration Documentation Plan
+
+Recommended integration docs for implemented external contracts:
+
+- Path: `docs/integrations/source-bundle.md`. External system or contract: producer-created source bundle directory and `manifest.json` schema. Current usage: producer-facing `pkg/bundle`, CLI `manifest create`, app/source validation. Version notes: `schema_version: 1`; digest format `sha256:<64 lowercase hex>`; RFC3339 timestamps. Document schema, validation, reserved paths, examples, and `pkg/bundle` helper role. Do not document destination routing or config policies as part of source manifests.
+- Path: `docs/integrations/destination-state.md`. External system or contract: destination `.distributor.json` state file. Current usage: destination sentinel, comparison state, output and link metadata. Version notes: `schema_version: 1`; `distributor_version` diagnostic. Document fields, output kinds, embedded manifest, links, comparison role. Do not recommend routine hand-editing or future state fields.
+- Path: `docs/integrations/http-upload.md`. External system or contract: HTTP upload API. Current usage: `serve` routes `/healthz`, `/upload`, `/runs/<run_id>`, bearer token auth, tar/tar.gz archive ingestion. Version notes: no explicit API version; response schema is implemented in app tests. Document routes, request/response shapes, status values, content types, queue/status retention. Do not document TLS, rate limiting, durable queues, idempotency, zstd, or browser UI as implemented.
+- Path: `docs/integrations/markdown.md`. External system or contract: Goldmark Markdown renderer. Current usage: Markdown-to-HTML transform. Version notes: dependency version is in `go.mod`; document project-configured behavior instead of the full Goldmark API. Document raw HTML behavior, sidecar/index modes, deterministic output, tests. Do not document unsupported Markdown extensions unless configured in code.
+- Path: `docs/integrations/ssh-sftp.md`. External system or contract: SSH/SFTP storage. Current usage: source and destination backend adapter using native SSH/SFTP libraries. Version notes: dependency versions in `go.mod`; live integration tests are opt-in. Document auth order, host key policies, known_hosts behavior, path root, dry-run host key persistence behavior. Do not document password auth, shell commands, rsync, SCP, or unsupported SSH features.
+- Path: `docs/integrations/s3.md`. External system or contract: S3-compatible object storage. Current usage: source and destination backend adapter using AWS SDK for Go v2. Version notes: dependency versions in `go.mod`; supports configured endpoint, bucket, prefix, region, force path style. Document credential resolution, prefix semantics, traversal/stat/write/delete behavior, content type inference, live test gating. Do not document cloud-provider-specific features not used by the adapter.
+
+No external CLI integrations were found. The project uses Go libraries and protocols directly.
+
+## Recommended Implementation Sequence
+
+### Stage 1: Documentation Structure And Roadmap Cleanup
+
+- Goal: establish the target documentation tree and remove or quarantine completed roadmap noise without rewriting user docs yet.
+- Files to create/update/delete/move: update `docs/roadmap/documentation.md` only if needed; decide which completed roadmap files remain, but delete them only in a later implementation pass after current docs are rewritten.
+- Repository areas to inspect: `docs/roadmap/*`, `docs/policy/documentation.md`, current docs links.
+- Acceptance criteria: roadmap docs contain only future/deferred/planning material; no current behavior is documented only in old roadmap files.
+- Suggested validation commands: `find docs -maxdepth 3 -type f | sort`; `rg -n -i "future|planned|deferred|experimental|deprecated|not implemented" README.md docs --glob '!docs/roadmap/**'`.
+- Prompt size: small enough for one implementation prompt.
+
+### Stage 2: README And Canonical User Entry Points
+
+- Goal: rewrite `README.md` and `docs/cli.md` as concise, accurate user entry points.
+- Files to create/update/delete/move: `README.md`, `docs/cli.md`.
+- Repository areas to inspect: `internal/cli`, `internal/cli/root_test.go`, `internal/app` output code, examples.
+- Acceptance criteria: README is concise; CLI flags and examples match parser code/tests; manifest positional parsing examples are included; no full config reference duplication.
+- Suggested validation commands: `go test ./internal/cli ./internal/app`; manual `go run ./cmd/distributor <command> --help` spot checks.
+- Prompt size: small enough for one implementation prompt.
+
+### Stage 3: Configuration Reference And Examples Links
+
+- Goal: rewrite the canonical config reference and verify examples list.
+- Files to create/update/delete/move: `docs/config.md`; examples only if stale or invalid.
+- Repository areas to inspect: `internal/config`, `internal/app/backends.go`, `internal/app/serve.go`, `examples`, config tests.
+- Acceptance criteria: all fields/defaults/accepted values match code; examples list is complete; remote examples are clearly environment-gated; no secrets.
+- Suggested validation commands: `go test ./internal/config`; `rg -n "examples/" docs/config.md README.md docs/cli.md docs/operations.md`.
+- Prompt size: one implementation prompt if examples remain unchanged; split if examples require edits.
+
+### Stage 4: Operations And Troubleshooting
+
+- Goal: rewrite operator-facing workflow, safety, recovery, and symptom docs.
+- Files to create/update/delete/move: `docs/operations.md`, `docs/troubleshooting.md`.
+- Repository areas to inspect: `internal/app/run*.go`, `internal/app/upload*.go`, `internal/publish`, `internal/state`, `internal/storage`, adapter tests, CLI tests.
+- Acceptance criteria: operations doc avoids full field/flag tables; troubleshooting entries use symptom/cause/diagnostic/safe fix/link; HTTP upload and forced replacement behavior match code.
+- Suggested validation commands: `go test ./internal/app ./internal/publish ./internal/state ./internal/storage`; stale-term grep outside roadmap.
+- Prompt size: likely too large for one prompt if both docs are long; split into operations first, troubleshooting second.
+
+### Stage 5: Integration Contracts
+
+- Goal: create concise integration docs for implemented external contracts.
+- Files to create/update/delete/move: create `docs/integrations/source-bundle.md`, `docs/integrations/destination-state.md`, `docs/integrations/http-upload.md`, `docs/integrations/ssh-sftp.md`, `docs/integrations/s3.md`; update `docs/integrations/markdown.md`.
+- Repository areas to inspect: `pkg/bundle`, `internal/state`, `internal/app/upload_http.go`, `internal/ingest`, `internal/adapters/ssh`, `internal/adapters/s3`, `internal/transform/markdown`, `go.mod`, tests.
+- Acceptance criteria: integration docs cover only implemented contracts; no external docs copied wholesale; current docs link to integration docs where appropriate.
+- Suggested validation commands: `go test ./pkg/bundle ./internal/state ./internal/app ./internal/ingest ./internal/adapters/ssh ./internal/adapters/s3 ./internal/transform/markdown`.
+- Prompt size: split into source/state/http and markdown/SSH/S3 if needed.
+
+### Stage 6: Internal Component Docs Normalization
+
+- Goal: normalize `docs/internal/` to policy-required sections and current component boundaries.
+- Files to create/update/delete/move: all `docs/internal/*.md`.
+- Repository areas to inspect: matching internal package code/tests and integration docs from Stage 5.
+- Acceptance criteria: each internal doc includes purpose, inputs/outputs, boundaries, config fields used, adapters used, failure behavior, tests, and invariants where applicable; user-facing how-to content is linked instead of duplicated.
+- Suggested validation commands: package-specific `go test` for touched components; `rg -n "future|planned|deferred|not implemented" docs/internal`.
+- Prompt size: too large for one prompt; split by package group: app/config/ingest, bundle/state/publish, storage/adapters/transform/link/notify.
+
+### Stage 7: Final Roadmap And Link Hygiene
+
+- Goal: remove completed roadmap files that no longer represent future work and check links/stale terminology.
+- Files to create/update/delete/move: `docs/roadmap/*` as appropriate; no code.
+- Repository areas to inspect: all docs, README, examples.
+- Acceptance criteria: completed behavior is documented in current docs, not only roadmaps; no stale completed roadmap instructions remain as active plans; links are accurate.
+- Suggested validation commands: `find docs -maxdepth 3 -type f | sort`; `rg -n -i "future|planned|deferred|experimental|deprecated|old behavior|Stage|Phase" README.md docs --glob '!docs/roadmap/**'`; manual link review.
+- Prompt size: small enough for one implementation prompt after prior rewrites.
+
+### Stage 8: Full Documentation Validation
+
+- Goal: verify docs against tests, examples, and policy checklist.
+- Files to create/update/delete/move: fixes only if validation finds gaps.
+- Repository areas to inspect: full repo.
+- Acceptance criteria: tests pass; examples load; CLI examples match parser; stale-term grep passes; documentation checklist in `docs/policy/documentation.md` is satisfied.
+- Suggested validation commands: `go test ./...`; `go test ./internal/config ./internal/cli ./internal/app`; stale-term and link greps listed below.
+- Prompt size: small enough for one implementation prompt if prior stages are complete.
+
+## Validation Plan
+
+Recommended validation checks during and after documentation implementation:
+
+- Full test suite: `go test ./...`.
+- Config/example loading: `go test ./internal/config` because `internal/config/load_test.go` loads maintained examples.
+- CLI/parser behavior: `go test ./internal/cli ./internal/app`.
+- App/report/upload behavior: `go test ./internal/app ./internal/ingest`.
+- Publish/state/storage safety: `go test ./internal/publish ./internal/state ./internal/storage ./internal/storage/fake`.
+- Adapter docs: `go test ./internal/adapters/local ./internal/adapters/ssh ./internal/adapters/s3`.
+- Transform docs: `go test ./internal/transform/markdown`.
+- Producer/source bundle docs: `go test ./pkg/bundle ./internal/bundle`.
+- Stale terminology outside roadmap:
+  ```sh
+  rg -n -i "future|planned|deferred|experimental|deprecated|not implemented|old behavior" README.md docs --glob '!docs/roadmap/**'
+  ```
+- Stale completed-work labels outside roadmap:
+  ```sh
+  rg -n -i "Stage|Phase" README.md docs --glob '!docs/roadmap/**'
+  ```
+- CLI flag consistency:
+  ```sh
+  rg -n "--config|--dry-run|--force|--format|--pipeline|--bundle|--id|--file|--created|--overwrite" docs/cli.md internal/cli
+  ```
+- Example references:
+  ```sh
+  rg -n "examples/" README.md docs examples internal/config/load_test.go
+  ```
+- Link checks: no automated Markdown link checker was found. Perform manual link review or add a link checker in a separate tooling change if desired.
+
+Manual review items:
+
+- Confirm README remains concise and orientation-focused.
+- Confirm `docs/config.md` is the only full config field/default reference.
+- Confirm `docs/cli.md` is the only full command/flag reference.
+- Confirm `docs/operations.md` focuses on operating and recovery.
+- Confirm `docs/troubleshooting.md` is symptom-first.
+- Confirm `docs/internal/` docs describe implemented component contracts and boundaries.
+- Confirm integration docs do not claim support for unimplemented external features.
+- Confirm examples contain no secrets and distinguish local runnable examples from environment-gated remote examples.
+
+## Open Questions
+
+No open questions block this roadmap. The recommended approach is to rewrite the documentation from the current code and tests, preserve all implemented behavior in current docs, move only actual integration contracts into `docs/integrations/`, and delete completed roadmap documents only after their implemented behavior is represented in canonical current docs.