eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Clean up documentation consistency

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-31 04:10:14 +00:00
parent 14444c152b
commit 783d094007
7 changed files with 43 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@@ -10,4 +10,4 @@ Run the local example pipeline:
go run ./cmd/distributor run --config examples/local-publish.yml
```
See [docs/cli.md](docs/cli.md), [docs/config.md](docs/config.md), [docs/operations.md](docs/operations.md), and [docs/troubleshooting.md](docs/troubleshooting.md) for the implemented CLI, configuration, operating notes, and common failure modes. Future planning lives under `docs/roadmap/`.
See [docs/cli.md](docs/cli.md), [docs/config.md](docs/config.md), [docs/operations.md](docs/operations.md), and [docs/troubleshooting.md](docs/troubleshooting.md) for the implemented CLI, configuration, operating notes, and common failure modes. Planning material lives under `docs/roadmap/`.

8
docs/config.md
View File

@@ -10,7 +10,7 @@ If `--config` is omitted, `run` uses:
/usr/local/etc/distributor/config.yml
```
Config parsing rejects unknown YAML fields. The current executable backend support is local only. SSH and S3 config fields are accepted by config validation, but runtime execution for those backends is not implemented.
Config parsing rejects unknown YAML fields. The current executable backend support is local only. SSH and S3 config fields are accepted by config validation, but runtime execution for those backends is unavailable.
## Minimal Local Config
@@ -105,8 +105,8 @@ Destination:
Accepted backend names:
- `local`: executable; requires `path`.
- `ssh`: config validation only; execution is not implemented.
- `s3`: config validation only; execution is not implemented.
- `ssh`: config validation only; execution is unavailable.
- `s3`: config validation only; execution is unavailable.
Publish policy:
@@ -141,7 +141,7 @@ Do not put literal secrets in config files. S3 credentials may name environment
- `credentials.access_key_id_env`
- `credentials.secret_access_key_env`
S3 execution is not implemented yet; these fields are accepted so config shape can be validated ahead of backend implementation.
S3 execution is unavailable; these fields are accepted so config shape can be validated.
## Examples

2
docs/operations.md
View File

@@ -80,6 +80,6 @@ After a successful publish or replacement, the internal notifier hook runs. The
## Caveats
Only local-to-local execution is implemented. SSH execution, S3 execution, external notification adapters, and force overwrite behavior are not implemented.
Only local-to-local execution is available. SSH execution, S3 execution, external notification adapters, and force overwrite behavior are unavailable.
For symptom-oriented fixes, see [troubleshooting](troubleshooting.md). For config details, see [configuration](config.md). For command syntax, see [CLI](cli.md).

36
docs/policy/architecture.md
View File

@@ -8,7 +8,7 @@ This document defines the development principles for `distributor`. It is inward
Producer applications generate manifested bundles. `distributor` discovers those bundles, validates them, optionally derives publication artifacts such as HTML, and publishes selected source and generated artifacts to one or more configured destinations.
`distributor` does not generate domain reports, interpret domain-specific report content, run producer pipelines, edit reports, or act as a CMS. Weather reports, D&D recaps, calendar summaries, email digests, and future report types should all enter `distributor` through the same bundle contract.
`distributor` does not generate domain reports, interpret domain-specific report content, run producer pipelines, edit reports, or act as a CMS. Weather reports, D&D recaps, calendar summaries, email digests, and additional report types should all enter `distributor` through the same bundle contract.
## Project Shape
@@ -16,7 +16,7 @@ Default to a small, explicit, dependency-light Go application. Keep the design m
Business logic should live outside CLI, transport, and external-adapter packages. The core application should reason in terms of pipelines, bundles, destination state, transforms, and publish plans—not S3 SDK calls, SFTP sessions, shell commands, or filesystem details.
The core workflow is:
The current core workflow is:
1. load configured pipelines;
2. open the source backend;
@@ -28,13 +28,13 @@ The core workflow is:
8. optionally transform Markdown to HTML for that destination;
9. publish selected source and generated artifacts;
10. write `.distributor.json` as the destination sentinel/state file;
11. run the notification stage, which is a no-op in the MVP.
11. run the notification hook, which is a no-op in the MVP.
## Pipeline Model
A pipeline has exactly one source and one or more destinations.
The source is discovered and validated once. Each destination has independent backend configuration, publication policy, transform policy, replacement behavior, state, and future notification behavior.
The source is discovered and validated once. Each destination has independent backend configuration, publication policy, transform policy, replacement behavior, state, and notification behavior.
The pipeline model is fan-out by design:
@@ -45,7 +45,7 @@ source bundle
-> destination C: source files + HTML
```
Destination-specific behavior must not leak back into the source bundle contract. A producer should not need to know whether a bundle will be published to local storage, SSH/SFTP, S3, a static site, email, RSS, or a future notification channel.
Destination-specific behavior must not leak back into the source bundle contract. A producer should not need to know whether a bundle will be published to local storage, another storage backend, a static site, email, RSS, or another notification channel.
## Source Bundle Contract
@@ -166,13 +166,13 @@ For example, one destination may publish source files only as a long-term archiv
## Backend Abstraction
Sources and destinations use the same storage abstraction. Local filesystem, SSH/SFTP, and S3-compatible object storage are peer backends. Any backend may appear as a source or a destination unless a specific limitation is documented.
Sources and destinations use the same storage abstraction. Current runtime execution uses the local filesystem backend. Additional storage backends should be peer implementations behind the same interface, and any backend-specific execution limitation must be documented.
Application logic must interact with storage through internal backend interfaces. Backend-specific behavior belongs in adapter packages. Pipeline, bundle, state, publish, and transform packages must not import S3, SSH/SFTP, or filesystem adapter implementation details.
Application logic must interact with storage through internal backend interfaces. Backend-specific behavior belongs in adapter packages. Pipeline, bundle, state, publish, and transform packages must not import service-specific or filesystem adapter implementation details.
Adapters should be thin. Backend adapters should implement storage operations and translate backend-specific errors, but should not make bundle comparison, transform, routing, or replacement decisions.
SSH support should prefer a native SFTP implementation over shelling out to `ssh`, `scp`, or `rsync`, unless a later design document records a reason to differ.
Remote file-transfer support should prefer native protocol implementations over shelling out, unless a later design document records a reason to differ.
## Dependency Policy
@@ -184,7 +184,7 @@ Avoid dependencies for small conveniences. Do not let external dependency types
## Package Layout
Use this layout unless the project has a documented reason to differ:
Use this current layout unless the project has a documented reason to differ:
- `cmd/distributor`: application entrypoint only.
- `internal/app`: application orchestration and top-level use cases.
@@ -194,14 +194,14 @@ Use this layout unless the project has a documented reason to differ:
- `internal/state`: `.distributor.json` parsing, validation, comparison, and output metadata.
- `internal/storage`: backend interfaces, shared path/resource types, backend registry, and storage errors.
- `internal/adapters/local`: local filesystem backend.
- `internal/adapters/ssh`: SSH/SFTP backend.
- `internal/adapters/s3`: S3-compatible object storage backend.
- `internal/transform`: transform interfaces, registry, planning, and shared transform models.
- `internal/transform/markdown`: Markdown-to-HTML implementation.
- `internal/publish`: destination planning, reconciliation, safety checks, and publish execution.
- `internal/notify`: notification interface and MVP no-op notifier.
- `internal/logging`: logging setup and shared logging helpers.
New storage adapters should live under `internal/adapters/<name>` and stay thin.
Package-private implementation constants may live near the package that owns them, preferably in `constants.go` when useful.
## Configuration
@@ -244,9 +244,9 @@ Each major stage should have an explicit input/output contract:
- publish execution;
- notification.
If users can select backends, transforms, notifiers, or future renderers, selection should go through a registry or equivalent mechanism rather than scattered conditionals.
If users can select backends, transforms, notifiers, or renderers, selection should go through a registry or equivalent mechanism rather than scattered conditionals.
The orchestrator should be able to plan, dry-run, and execute configured pipelines. Dry-run behavior should be first-class because the application may delete, overwrite, or publish files to remote storage.
The orchestrator should be able to plan, dry-run, and execute configured pipelines. Dry-run behavior should be first-class because the application may delete, overwrite, or publish files.
## Embedded Assets
@@ -264,7 +264,7 @@ Skip and no-op decisions should be logged at an appropriate level so operators c
## Context, Timeouts, and Cancellation
Long-running operations should accept `context.Context`. Storage operations, SSH/SFTP sessions, S3 requests, transforms, and multi-stage workflows should respect cancellation and timeouts.
Long-running operations should accept `context.Context`. Storage operations, service requests, transforms, and multi-step workflows should respect cancellation and timeouts.
## State, Files, and Safety
@@ -282,7 +282,7 @@ Where practical, publish operations should use staging paths or temporary object
Core logic should be testable without real external services. Use fakes, fixtures, or local test doubles for adapters where practical.
Config examples should be load-tested. Important CLI workflows should have parser or command tests. Stage/module contracts should have focused tests that do not require running the full application unless end-to-end coverage is intentional.
Config examples should be load-tested. Important CLI workflows should have parser or command tests. Component contracts should have focused tests that do not require running the full application unless end-to-end coverage is intentional.
Important tests include:
@@ -297,13 +297,13 @@ Important tests include:
- transform output planning and metadata recording;
- dry-run output;
- local backend behavior with temporary directories;
- fake backend behavior for S3 and SSH/SFTP-facing core logic.
- fake backend behavior for storage-facing core logic.
## Documentation
Documentation should follow the project documentation policy. Keep user docs focused on implemented behavior. Put future, planned, or aspirational work only under `docs/roadmap/`.
When changing architecture, config, CLI behavior, adapters, manifest/state contracts, transform behavior, publish behavior, or stage/module contracts, update the relevant docs and examples in the same change.
When changing architecture, config, CLI behavior, adapters, manifest/state contracts, transform behavior, publish behavior, or component contracts, update the relevant docs and examples in the same change.
The source manifest and destination `.distributor.json` schemas should have canonical documentation once implemented. Example configs should be valid and load-tested where practical.
@@ -321,4 +321,4 @@ The source manifest and destination `.distributor.json` schemas should have cano
- a backup system;
- a notification platform.
It may later support notification adapters, RSS/feed generation, richer HTML templates, or additional transforms, but those features must preserve the core bundle-distribution boundary.
Additional notification, feed, template, or transform behavior must preserve the core bundle-distribution boundary.

2
docs/policy/development.md
View File

@@ -167,7 +167,7 @@ look runnable but require unsupported backend execution.
When changing examples:
1. Keep paths relative to the repository where practical.
2. Prefer local examples until remote backends are implemented.
2. Prefer local examples until remote backend support exists.
3. Run `go test ./internal/config` because config tests load examples.
4. Update README, CLI, or config docs if links or recommended workflows change.

32
docs/roadmap/documentation.md
View File

@@ -18,9 +18,9 @@ The current documentation and examples reviewed are:
| `docs/cli.md` | Keep and update against CLI tests | Covers `version`, `run`, `validate`, and `inspect`; should be checked against command help and parser tests. |
| `docs/config.md` | Keep and tighten | Current config reference documents local execution and also accepted SSH/S3 config fields. It must clearly distinguish config validation from executable backend support. |
| `docs/operations.md` | Keep and expand slightly | Covers local workflow, destination state, retry behavior, cleanup, fan-out failure handling, and caveats. |
| `docs/troubleshooting.md` | Missing, recommended | Recurring failure modes now exist and should be documented. |
| `docs/troubleshooting.md` | Added | Covers recurring local MVP failure modes. |
| `docs/policy/architecture.md` | Keep and clarify where needed | Development policy is broad and includes future adapter direction. Wording should not imply SSH/S3 adapters currently exist. |
| `docs/policy/development.md` | Required rewrite | Currently contains only `# Not yet implemented`; this is the largest policy compliance gap. |
| `docs/policy/development.md` | Updated | Contains contributor and agent workflow guidance. |
| `docs/policy/documentation.md` | Keep | Canonical documentation policy. No change required unless the policy itself changes. |
| `docs/internal/bundle.md` | Keep and verify | Describes implemented bundle parsing, discovery, validation, and digest semantics. |
| `docs/internal/notify.md` | Keep and verify | Accurately states current no-op notification behavior. |
@@ -28,10 +28,10 @@ The current documentation and examples reviewed are:
| `docs/internal/state.md` | Keep and verify | Describes implemented `.distributor.json` state and comparison behavior. |
| `docs/internal/storage.md` | Keep and verify | Describes storage interface, typed errors, path rules, traversal, and managed deletion. |
| `docs/internal/transform.md` | Keep and verify | Describes Markdown-to-HTML sidecar behavior and transform boundaries. |
| `docs/internal/app.md` | Missing, recommended | Needed for orchestration, backend factory, transform registry, dry-run, summaries, and notifier handoff. |
| `docs/internal/config.md` | Missing, recommended | Needed for config loading, defaults, validation, accepted-but-not-executable backends, and example tests. |
| `docs/internal/app.md` | Added | Covers orchestration, backend factory, transform registry, dry-run, summaries, and notifier handoff. |
| `docs/internal/config.md` | Added | Covers config loading, defaults, validation, accepted-but-not-executable backends, and example tests. |
| `docs/internal/local-backend.md` | Missing, optional | Local adapter behavior may remain in `docs/internal/storage.md`; create this only if local filesystem safety detail outgrows that doc. |
| `docs/integrations/` | Missing, optional/recommended | Markdown rendering uses Goldmark. A concise Markdown integration note is useful because raw HTML behavior and deterministic rendering are externally visible. |
| `docs/integrations/markdown.md` | Added | Documents current Goldmark-backed Markdown rendering behavior. |
| `docs/roadmap/audit.md` | Historical roadmap/report | Keep under roadmap unless replaced by a new audit. |
| `docs/roadmap/cleanup.md` | Historical or completed roadmap | Keep under roadmap; optionally add completion status in the documentation refresh. |
| `docs/roadmap/config.md` | Roadmap | Keep as planning material; avoid linking to it as current config reference. |
@@ -42,7 +42,7 @@ The current documentation and examples reviewed are:
| `examples/local-to-local.yml` | Keep | Minimal local config; load-tested. |
| `examples/local-publish.yml` | Keep | Runnable local publication example used by README and CLI docs. |
| `examples/local-html.yml` | Keep | Runnable local HTML example. |
| `examples/fan-out.yml` | Needs decision in refresh | Currently load-tested but uses SSH/S3 destinations that are not executable. Replace with a local-only fan-out example or move remote fan-out material under roadmap. |
| `examples/fan-out.yml` | Updated | Local-only runnable fan-out example. |
| `examples/source-bundle/` | Keep | Copyable valid source bundle fixture for local CLI examples. |
Implementation source areas inspected for documentation truth:
@@ -73,20 +73,20 @@ Absent areas from earlier planning that should not be documented as implemented:
## Policy Compliance Assessment
Required current-behavior docs mostly exist for a config-driven, stateful, modular CLI, but three gaps should be closed before remote backend work resumes.
Required current-behavior docs now exist for the config-driven, stateful, modular CLI.
Required fixes:
Completed fixes from this refresh:
- Rewrite `docs/policy/development.md`; it is required by the documentation policy and is currently a placeholder.
- Keep all non-roadmap docs scoped to implemented behavior. In particular, SSH/S3 execution, force overwrite, and external notification adapters must remain described as unavailable unless the corresponding code exists.
- Resolve `examples/fan-out.yml`. It is valid config syntax, but it is not an executable example because SSH/S3 backends are not implemented for execution.
- `docs/policy/development.md` contains real workflow guidance.
- Non-roadmap docs are scoped to implemented behavior. SSH/S3 execution, force overwrite, and external notification adapters remain described as unavailable unless the corresponding code exists.
- `examples/fan-out.yml` is local-only and runnable.
Recommended fixes:
Completed recommended fixes from this refresh:
- Add `docs/troubleshooting.md` for recurring local MVP failure modes: invalid config, invalid source manifest, digest mismatch, unmanaged destination content, destination conflicts, unsupported remote execution, output path collisions, and failed writes.
- Add `docs/internal/app.md` and `docs/internal/config.md` so future agents have one current-behavior internal reference for orchestration and config semantics.
- Add a concise `docs/integrations/markdown.md` only if the project wants integration notes for Goldmark/CommonMark rendering behavior. This is recommended because Markdown rendering is externally visible and raw HTML handling is an important contract.
- Add status notes to roadmap files that are now historical or completed so future agents do not treat old MVP planning as current behavior or active instructions.
- `docs/troubleshooting.md` covers recurring local MVP failure modes.
- `docs/internal/app.md` and `docs/internal/config.md` provide current-behavior internal references.
- `docs/integrations/markdown.md` documents current Goldmark-backed rendering behavior.
- Historical roadmap status notes were added where useful.
No broad rewrite is needed for `README.md`, `docs/cli.md`, `docs/config.md`, or `docs/operations.md`. They are close to the implemented local MVP and should be tightened against code and tests.

4
docs/troubleshooting.md
View File

@@ -34,7 +34,7 @@ Diagnostic:
rg -n "backend:" <config-path>
```
Safe fix: use `backend: local` for executable workflows. SSH and S3 config shapes are accepted only for validation; runtime execution is not implemented.
Safe fix: use `backend: local` for executable workflows. SSH and S3 config shapes are accepted only for validation; runtime execution is unavailable.
## `backend ssh is not implemented for execution` or `backend s3 is not implemented for execution`
@@ -46,7 +46,7 @@ Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run
```
Safe fix: use local destinations for current executable workflows, or keep remote backend configs under roadmap material until those adapters exist. See [configuration](config.md).
Safe fix: use local destinations for current executable workflows, or keep remote backend configs under roadmap material unless those adapters are added. See [configuration](config.md).
## `validate command requires a path` or `inspect command requires a path`