Compare commits
6 Commits
edb9ac3a90
...
783d094007
| Author | SHA1 | Date | |
|---|---|---|---|
| 783d094007 | |||
| 14444c152b | |||
| 93c22884b0 | |||
| eac73a79a1 | |||
| b3044c5b7b | |||
| c36217d0df |
@@ -1,6 +1,8 @@
|
||||
# distributor
|
||||
|
||||
`distributor` validates and publishes manifested report bundles.
|
||||
`distributor` validates manifested report bundles and publishes selected source or generated artifacts to configured destinations.
|
||||
|
||||
It is currently a local-first CLI: source bundles are read from local storage, destinations are local directories, and Markdown files can be rendered to HTML sidecars.
|
||||
|
||||
Run the local example pipeline:
|
||||
|
||||
@@ -8,4 +10,4 @@ Run the local example pipeline:
|
||||
go run ./cmd/distributor run --config examples/local-publish.yml
|
||||
```
|
||||
|
||||
See [docs/cli.md](docs/cli.md), [docs/config.md](docs/config.md), and [docs/operations.md](docs/operations.md) for the implemented CLI, configuration, and operating notes. Current design and implementation planning lives under `docs/roadmap/`.
|
||||
See [docs/cli.md](docs/cli.md), [docs/config.md](docs/config.md), [docs/operations.md](docs/operations.md), and [docs/troubleshooting.md](docs/troubleshooting.md) for the implemented CLI, configuration, operating notes, and common failure modes. Planning material lives under `docs/roadmap/`.
|
||||
|
||||
65
docs/cli.md
65
docs/cli.md
@@ -6,44 +6,41 @@
|
||||
go run ./cmd/distributor run --config examples/local-publish.yml
|
||||
```
|
||||
|
||||
This validates and publishes the example source bundle to `workspace/published/source-bundle`.
|
||||
This discovers the example source bundle and publishes source files to `workspace/published/source-bundle`.
|
||||
|
||||
## Command overview
|
||||
|
||||
```sh
|
||||
distributor --help
|
||||
distributor [--help]
|
||||
distributor version
|
||||
distributor run
|
||||
distributor validate
|
||||
distributor inspect
|
||||
distributor run [--config <path>] [--dry-run]
|
||||
distributor validate <path>
|
||||
distributor inspect <path>
|
||||
```
|
||||
|
||||
`version` prints the application name and version. The default development version is `dev`; release builds may replace it at build time.
|
||||
- `version`: prints the application name and version. Development builds print `distributor dev`.
|
||||
- `run`: loads a YAML config, discovers local source bundles, plans each configured destination, writes selected outputs unless `--dry-run` is set, and prints a final status summary.
|
||||
- `validate`: validates a local source bundle directory or a local tree containing source bundles.
|
||||
- `inspect`: validates local source bundles and prints normalized bundle metadata.
|
||||
|
||||
`run --config <path>` executes configured local-to-local pipelines that publish source files, generated HTML files, or both.
|
||||
|
||||
`run --config <path> --dry-run` discovers source bundles, inspects destination state, and prints planned actions plus a final status summary without writing files.
|
||||
|
||||
`validate <path>` validates a local source bundle directory or a local tree containing source bundles.
|
||||
|
||||
`inspect <path>` validates discovered local source bundles and prints a concise normalized summary.
|
||||
|
||||
Remote backends are not implemented yet.
|
||||
`validate` and `inspect` accept local paths only. `run` currently executes local backends only. SSH and S3 config can be parsed and validated, but configured SSH or S3 execution fails with a clear unsupported-execution error.
|
||||
|
||||
## Flag reference
|
||||
|
||||
The root command supports:
|
||||
Root command:
|
||||
|
||||
- `--help`, `-h`: print root help.
|
||||
- `--help`, `-h`, or `help`: print root help.
|
||||
|
||||
Each subcommand supports:
|
||||
All subcommands:
|
||||
|
||||
- `--help`, `-h`: print command-specific help.
|
||||
|
||||
`run` supports:
|
||||
`run` flags:
|
||||
|
||||
- `--config <path>`: config file to load.
|
||||
- `--dry-run`: validate config, print planned actions and final status, and do not publish.
|
||||
- `--config <path>`: config file to load. If omitted, `run` uses `/usr/local/etc/distributor/config.yml`.
|
||||
- `--dry-run`: load config, discover bundles, inspect destination state, print planned actions and final status, and do not write files.
|
||||
|
||||
`run` does not accept positional arguments. `validate` and `inspect` accept at most one path; omitting the path returns a required-path error.
|
||||
|
||||
## Common workflows
|
||||
|
||||
@@ -65,7 +62,7 @@ Preview local publication without writing:
|
||||
go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
|
||||
```
|
||||
|
||||
Publish the local example:
|
||||
Publish the local source example:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config examples/local-publish.yml
|
||||
@@ -76,3 +73,27 @@ Publish the local HTML example:
|
||||
```sh
|
||||
go run ./cmd/distributor run --config examples/local-html.yml
|
||||
```
|
||||
|
||||
Preview local fan-out publication:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
`run` prints the number of configured pipelines, one line per pipeline, one line per planned destination action, and a final status line. Actions include:
|
||||
|
||||
- `publish_new`: destination has no managed state and is empty.
|
||||
- `replace_older`: destination state is older than the source manifest.
|
||||
- `skip_same`: destination state already matches the source manifest.
|
||||
- `skip_destination_newer`: destination state is newer than the source manifest.
|
||||
- `error`: planning or execution failed for that destination.
|
||||
|
||||
The command exits non-zero if any destination fails. Independent later destinations are still attempted.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
Use `validate` before publication when a producer has written a new bundle. Use `inspect` to confirm normalized ids, timestamps, digests, file paths, and file sizes.
|
||||
|
||||
For symptom-oriented recovery steps, see [troubleshooting](troubleshooting.md). For destination state and retry behavior, see [operations](operations.md). For config fields and defaults, see [configuration](config.md).
|
||||
|
||||
101
docs/config.md
101
docs/config.md
@@ -1,18 +1,18 @@
|
||||
# Distributor Configuration
|
||||
|
||||
## Config file location
|
||||
## Config File Location
|
||||
|
||||
`distributor run --config <path>` loads the YAML config at the path provided by `--config`.
|
||||
`distributor run --config <path>` loads the YAML config at the provided path.
|
||||
|
||||
If `--config` is omitted during run, the built-in default path is:
|
||||
If `--config` is omitted, `run` uses:
|
||||
|
||||
```text
|
||||
/usr/local/etc/distributor/config.yml
|
||||
```
|
||||
|
||||
The current implementation supports local-to-local publication of source files, generated HTML files, or both. Remote backends are not implemented yet.
|
||||
Config parsing rejects unknown YAML fields. The current executable backend support is local only. SSH and S3 config fields are accepted by config validation, but runtime execution for those backends is unavailable.
|
||||
|
||||
## Minimal config
|
||||
## Minimal Local Config
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
@@ -26,9 +26,9 @@ pipelines:
|
||||
path: /srv/reports/archive
|
||||
```
|
||||
|
||||
This uses the default publish policy of source files only and the default transfer policy.
|
||||
This publishes source files only. It uses the default validation and transfer policies.
|
||||
|
||||
## Production-oriented config
|
||||
## Production-Oriented Local Config
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
@@ -52,6 +52,22 @@ pipelines:
|
||||
on_conflict: fail
|
||||
```
|
||||
|
||||
## HTML Publication
|
||||
|
||||
To publish generated HTML from Markdown files:
|
||||
|
||||
```yaml
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
```
|
||||
|
||||
Sidecar generation writes `report.html` for `report.md`. It does not mutate the source bundle.
|
||||
|
||||
## Reference
|
||||
|
||||
Top level:
|
||||
@@ -60,37 +76,78 @@ Top level:
|
||||
|
||||
Pipeline:
|
||||
|
||||
- `id`: required unique identifier.
|
||||
- `id`: required unique slug-like identifier.
|
||||
- `source`: required backend config.
|
||||
- `validation.on_digest_mismatch`: optional; defaults to `fail`; only `fail` is supported.
|
||||
- `destinations`: required non-empty destination list.
|
||||
- `validation.on_digest_mismatch`: optional, defaults to `fail`; only `fail` is supported.
|
||||
|
||||
Backends:
|
||||
Source backend:
|
||||
|
||||
- `local`: requires `path`.
|
||||
- `ssh`: requires `uri` and `path`.
|
||||
- `s3`: requires `endpoint` and `bucket`; supports optional `prefix`, `region`, `force_path_style`, and `credentials`.
|
||||
- `backend`: required.
|
||||
- `path`: required for `local` and `ssh`.
|
||||
- `uri`: required for `ssh`.
|
||||
- `endpoint`: required for `s3`.
|
||||
- `bucket`: required for `s3`.
|
||||
- `prefix`: optional for `s3`.
|
||||
- `region`: optional for `s3`.
|
||||
- `force_path_style`: optional for `s3`.
|
||||
- `credentials.access_key_id_env`: optional S3 credential environment variable name.
|
||||
- `credentials.secret_access_key_env`: optional S3 credential environment variable name.
|
||||
|
||||
Destination policy:
|
||||
Destination:
|
||||
|
||||
- `id`: required unique slug-like identifier within the pipeline.
|
||||
- Backend fields: same accepted shape as source backends, with destination fields at the destination level.
|
||||
- `publish`: optional; defaults to source-only publication.
|
||||
- `transform`: required only for generated HTML publication.
|
||||
- `transfer`: optional; defaults described below.
|
||||
|
||||
Accepted backend names:
|
||||
|
||||
- `local`: executable; requires `path`.
|
||||
- `ssh`: config validation only; execution is unavailable.
|
||||
- `s3`: config validation only; execution is unavailable.
|
||||
|
||||
Publish policy:
|
||||
|
||||
- `publish.source`: publish source artifacts.
|
||||
- `publish.html`: publish generated HTML artifacts from Markdown source files.
|
||||
- `transfer.on_destination_same`: `skip` or `fail`, defaults to `skip`.
|
||||
- `transfer.on_destination_older`: `replace` or `fail`, defaults to `replace`.
|
||||
- `transfer.on_destination_newer`: `skip` or `fail`, defaults to `skip`.
|
||||
- `transfer.on_conflict`: only `fail`, defaults to `fail`.
|
||||
|
||||
When `publish.html` is true, `transform.markdown_to_html.enabled: true` and `transform.markdown_to_html.mode: sidecar` are required.
|
||||
At least one output type must be enabled. When `publish.html` is true, `transform.markdown_to_html.enabled` must be `true` and `transform.markdown_to_html.mode` must be `sidecar`.
|
||||
|
||||
Markdown-to-HTML sidecar generation writes `report.html` for `report.md` and does not mutate the source bundle.
|
||||
Transfer policy:
|
||||
|
||||
- `transfer.on_destination_same`: `skip` or `fail`; defaults to `skip`.
|
||||
- `transfer.on_destination_older`: `replace` or `fail`; defaults to `replace`.
|
||||
- `transfer.on_destination_newer`: `skip` or `fail`; defaults to `skip`.
|
||||
- `transfer.on_conflict`: only `fail`; defaults to `fail`.
|
||||
|
||||
## Defaults
|
||||
|
||||
Defaults are applied after YAML decoding and before validation:
|
||||
|
||||
- `validation.on_digest_mismatch: fail`
|
||||
- `publish.source: true`
|
||||
- `publish.html: false`
|
||||
- `transfer.on_destination_same: skip`
|
||||
- `transfer.on_destination_older: replace`
|
||||
- `transfer.on_destination_newer: skip`
|
||||
- `transfer.on_conflict: fail`
|
||||
|
||||
## Secrets
|
||||
|
||||
Do not put literal secrets in config files. S3 credentials may refer to environment variable names with:
|
||||
Do not put literal secrets in config files. S3 credentials may name environment variables:
|
||||
|
||||
- `credentials.access_key_id_env`
|
||||
- `credentials.secret_access_key_env`
|
||||
|
||||
S3 execution is unavailable; these fields are accepted so config shape can be validated.
|
||||
|
||||
## Examples
|
||||
|
||||
Maintained examples live under [examples/](../examples/).
|
||||
Maintained examples live under [examples](../examples/):
|
||||
|
||||
- `local-to-local.yml`: minimal local config.
|
||||
- `local-publish.yml`: runnable local source publication.
|
||||
- `local-html.yml`: runnable local HTML publication.
|
||||
- `fan-out.yml`: runnable local fan-out publication to source and HTML destinations.
|
||||
|
||||
55
docs/integrations/markdown.md
Normal file
55
docs/integrations/markdown.md
Normal file
@@ -0,0 +1,55 @@
|
||||
# Markdown Integration
|
||||
|
||||
## Purpose
|
||||
|
||||
Markdown-to-HTML is the only implemented external file-format integration. This note documents the renderer behavior that is externally visible in generated destination artifacts.
|
||||
|
||||
## Dependency
|
||||
|
||||
Rendering uses `github.com/yuin/goldmark`. The exact dependency version is pinned in `go.mod`; review that file before changing renderer behavior or diagnosing version-specific output changes.
|
||||
|
||||
## Renderer behavior
|
||||
|
||||
`internal/transform/markdown.New` constructs the renderer with `goldmark.New()` and no project-specific extensions or renderer options.
|
||||
|
||||
For each source bundle file ending in `.md`, the transform reads the Markdown source and generates an HTML sidecar in the same logical directory. The output path replaces the `.md` suffix with `.html`, so `report.md` produces `report.html`. Non-Markdown source files produce no Markdown outputs.
|
||||
|
||||
Raw HTML embedded in Markdown is not passed through by the current renderer behavior. Tests allow Goldmark's disabled-or-escaped raw HTML output forms and reject literal script tags in generated HTML.
|
||||
|
||||
## Wrapper
|
||||
|
||||
Rendered Markdown body HTML is wrapped in a fixed document shell:
|
||||
|
||||
- `<!doctype html>`
|
||||
- `<html lang="en">`
|
||||
- UTF-8 `<meta charset>`
|
||||
- empty `<title>`
|
||||
- `<body>` containing the rendered Markdown body
|
||||
|
||||
The wrapper is deterministic and does not read configuration, templates, CSS, or source manifest metadata.
|
||||
|
||||
## Output metadata
|
||||
|
||||
Generated outputs record:
|
||||
|
||||
- destination path;
|
||||
- source path;
|
||||
- transform id `markdown_to_html`;
|
||||
- SHA-256 digest of the wrapped HTML bytes;
|
||||
- byte size of the wrapped HTML bytes.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Markdown rendering does not mutate source bundles, publish files, write `.distributor.json`, select outputs, or choose transfer actions. Publish planning decides whether generated HTML is selected for a destination.
|
||||
|
||||
Only sidecar output mode is supported for current behavior.
|
||||
|
||||
## Tests
|
||||
|
||||
Before changing Markdown renderer behavior, inspect and run:
|
||||
|
||||
```bash
|
||||
go test ./internal/transform/markdown
|
||||
```
|
||||
|
||||
The tests cover sidecar naming, ignored non-Markdown files, raw HTML handling, deterministic output, digest metadata, and size metadata.
|
||||
64
docs/internal/app.md
Normal file
64
docs/internal/app.md
Normal file
@@ -0,0 +1,64 @@
|
||||
# Application Orchestration
|
||||
|
||||
## Purpose
|
||||
|
||||
`internal/app` owns top-level use cases for `run`, `validate`, and `inspect`. It wires configuration, storage backends, transforms, publish planning, execution, summaries, and notification handoff.
|
||||
|
||||
## Inputs and outputs
|
||||
|
||||
`Run` accepts a context, optional config path, dry-run flag, stdout writer, and optional notifier. It loads YAML config, discovers source bundles for each configured pipeline, plans each destination independently, optionally executes publish plans, writes summary output when stdout is supplied, and returns an aggregated error if any destination fails.
|
||||
|
||||
`Validate` and `Inspect` accept a local path. `Validate` discovers and validates bundles. `Inspect` writes bundle metadata and manifest file entries to stdout when provided.
|
||||
|
||||
## Run flow
|
||||
|
||||
The runner:
|
||||
|
||||
1. loads config from the supplied path or `config.DefaultConfigPath`;
|
||||
2. opens the configured source backend;
|
||||
3. discovers validated bundles from the source root;
|
||||
4. opens each destination backend independently;
|
||||
5. builds a publish plan for each bundle and destination;
|
||||
6. prints plan lines and records summary counters;
|
||||
7. executes publish or replacement plans unless dry-run is enabled;
|
||||
8. invokes the notifier after successful publish or replacement actions.
|
||||
|
||||
Destination failures are collected while later destinations continue to run. Source open and source discovery failures stop the run because there are no valid bundles to fan out.
|
||||
|
||||
## Backend and transform wiring
|
||||
|
||||
The app-level backend factory registers only the local backend for execution. Config validation accepts other backend shapes, but `Run` can execute only local sources and local destinations.
|
||||
|
||||
The app-level transform registry registers Markdown-to-HTML using `internal/transform/markdown`. Lower-level publish code receives a resolver and does not import concrete transform implementations.
|
||||
|
||||
## Dry-run behavior
|
||||
|
||||
Dry-run still loads config, opens backends, discovers bundles, inspects destinations, resolves transforms, and builds publish plans. It does not write destination outputs, write `.distributor.json`, delete managed outputs, or notify.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
`Run` returns immediately for config loading errors, context cancellation before work starts, source open errors, and source discovery errors. Per-destination backend, planning, execution, and notification errors are aggregated into one run error after remaining destinations have been attempted.
|
||||
|
||||
Stdout write errors are returned immediately because the caller's requested output stream can no longer be trusted.
|
||||
|
||||
## Boundaries
|
||||
|
||||
`internal/app` coordinates packages but does not own manifest validation rules, destination state comparison, storage path rules, output planning, transform rendering, or backend-specific filesystem behavior.
|
||||
|
||||
`Validate` and `Inspect` are local path commands. Remote execution wiring is outside current behavior.
|
||||
|
||||
## Tests
|
||||
|
||||
Before changing app orchestration, inspect tests under:
|
||||
|
||||
- `internal/app`
|
||||
- `internal/cli`
|
||||
- `internal/publish`
|
||||
|
||||
## Invariants
|
||||
|
||||
- One source fans out to each destination independently.
|
||||
- Destination failures do not prevent later destinations from being planned.
|
||||
- Dry-run must not mutate destination storage or invoke notifications.
|
||||
- Concrete backend and transform registration stays at the app layer.
|
||||
- The default notifier is `notify.Noop`.
|
||||
@@ -32,10 +32,22 @@ The bundle digest is SHA-256 of a deterministic JSON array of file records in ma
|
||||
|
||||
Discovery walks a storage backend beneath a source root, finds `manifest.json` files, sorts bundle paths lexically, and rejects nested manifests.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
Manifest parsing and validation fail before destination planning. Storage-backed validation fails when listed files are missing, are not regular files, have unexpected sizes, have unexpected SHA-256 digests, or when a source bundle includes unsafe or reserved paths.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Bundle code uses `internal/storage` and does not import local, SSH, or S3 adapters. CLI local path support is wired in `internal/app`.
|
||||
Bundle code uses `internal/storage` and does not import concrete adapters. CLI local path support is wired in `internal/app`.
|
||||
|
||||
## Tests
|
||||
|
||||
Before changing bundle behavior, inspect tests under `internal/bundle`.
|
||||
|
||||
## Invariants
|
||||
|
||||
- `manifest.json` is the only source bundle contract.
|
||||
- Source file paths must stay relative to the bundle root.
|
||||
- The top-level bundle digest is derived from manifest file records in order.
|
||||
- Discovery order is lexical and deterministic.
|
||||
- Nested manifests are rejected.
|
||||
|
||||
61
docs/internal/config.md
Normal file
61
docs/internal/config.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# Configuration Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
`internal/config` defines YAML-backed configuration structs, defaulting, and validation for distributor pipelines.
|
||||
|
||||
## Inputs and outputs
|
||||
|
||||
Input is a YAML file containing `pipelines`. Output is a `Config` value with defaults applied and validation completed. Load failures include the config path and whether the failure occurred during file loading, YAML parsing, or validation.
|
||||
|
||||
## Loading flow
|
||||
|
||||
`LoadFile` opens the requested path, decodes YAML with known-field checking enabled, applies defaults, and validates the result. The app uses `DefaultConfigPath` when the CLI does not supply a config path.
|
||||
|
||||
Known-field checking rejects misspelled or unknown YAML keys before defaults and validation run.
|
||||
|
||||
## Defaults
|
||||
|
||||
Defaults are applied in `ApplyDefaults`:
|
||||
|
||||
- pipeline validation defaults `on_digest_mismatch` to `fail`;
|
||||
- destination publish policy defaults to source output only;
|
||||
- `transfer.on_destination_same` defaults to `skip`;
|
||||
- `transfer.on_destination_older` defaults to `replace`;
|
||||
- `transfer.on_destination_newer` defaults to `skip`;
|
||||
- `transfer.on_conflict` defaults to `fail`.
|
||||
|
||||
## Validation responsibilities
|
||||
|
||||
Validation requires at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, and valid transfer actions.
|
||||
|
||||
`ValidatePublishTransformPolicy` is shared with publish planning so destination policy combinations are checked consistently. Publishing HTML requires an enabled Markdown-to-HTML transform in `sidecar` mode. A publish policy must select source output, HTML output, or both.
|
||||
|
||||
## Executable support boundary
|
||||
|
||||
Config validation accepts `local`, `ssh`, and `s3` backend shapes so config files can be validated as schemas. Runtime execution currently opens only local backends through `internal/app`.
|
||||
|
||||
The user-facing configuration reference is `docs/config.md`; this file documents package behavior for maintainers.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
Load errors wrap the underlying file, YAML, or validation error with context. Validation collects all detected field errors into one error value instead of stopping at the first invalid field.
|
||||
|
||||
Unsupported backend names fail validation. Accepted backend names without runtime execution support fail later during app backend opening.
|
||||
|
||||
## Tests
|
||||
|
||||
Before changing config behavior, inspect:
|
||||
|
||||
- `internal/config/load_test.go`
|
||||
- `internal/config/validate_test.go`
|
||||
- example-loading coverage in `internal/config`
|
||||
- user-facing examples under `examples/`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Defaults are applied before validation.
|
||||
- Unknown YAML fields are rejected.
|
||||
- `docs/config.md` remains the canonical user-facing config reference.
|
||||
- Runtime backend execution support is not inferred from config validation support.
|
||||
- New user-visible config behavior must be covered by tests and docs in the same change.
|
||||
@@ -4,10 +4,32 @@
|
||||
|
||||
`internal/notify` defines the internal notification interface used by the application runner.
|
||||
|
||||
## Inputs and outputs
|
||||
|
||||
Input is a notification event containing pipeline id, destination id, bundle id, bundle path, action, and output metadata. The interface returns an error so app orchestration can treat notification failures as destination failures.
|
||||
|
||||
## Current behavior
|
||||
|
||||
The implemented notifier is a no-op. It is invoked only after a successful publish or replacement. Dry-run, skipped destinations, and failed destinations do not invoke it.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
`notify.Noop` always succeeds unless the context is already canceled. If a configured notifier returns an error, `internal/app` records that destination as failed and continues with remaining destinations.
|
||||
|
||||
## Boundaries
|
||||
|
||||
No external notification adapters are implemented. Notification configuration is not part of the current user-facing config schema.
|
||||
External notification adapters and user-facing notification configuration are outside current behavior.
|
||||
|
||||
## Tests
|
||||
|
||||
Before changing notification behavior, inspect:
|
||||
|
||||
- `internal/notify`
|
||||
- `internal/app/run_test.go`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Notifications are emitted only after successful publish or replacement execution.
|
||||
- Dry-run never notifies.
|
||||
- Skipped and failed destinations never notify.
|
||||
- The default app notifier is `notify.Noop`.
|
||||
|
||||
@@ -12,11 +12,17 @@ Output is a plan with an action, reason, and selected source or generated output
|
||||
|
||||
## Actions
|
||||
|
||||
Supported actions are publish new, replace older destination, skip same source, skip newer destination, fail conflict, and fail unmanaged destination.
|
||||
Supported actions are `publish_new`, `replace_older`, `skip_same`, `skip_destination_newer`, `fail_conflict`, and `fail_unmanaged`.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
Planning fails when request fields are incomplete, publish and transform policies are invalid, selected outputs collide, HTML output is requested without Markdown inputs, destination state is invalid, destination content is unmanaged, or transfer policy maps the comparison outcome to failure.
|
||||
|
||||
Execution fails if a write, delete, state serialization, or context check fails. Outputs written during a failed publish attempt are cleaned up through managed deletion where possible.
|
||||
|
||||
## Boundaries
|
||||
|
||||
The current implementation publishes source files and Markdown-to-HTML sidecar outputs. Remote backend execution is not implemented.
|
||||
The current implementation publishes source files and Markdown-to-HTML sidecar outputs. Backend behavior is supplied through `internal/storage`; app runtime currently supplies local backends.
|
||||
|
||||
The package uses `internal/state` for destination comparison, `internal/storage` for IO, and the shared `internal/config` publish/transform policy helper for request validation. It resolves transforms through a narrow resolver supplied by the caller; concrete transform registration is owned by the app layer. It does not parse CLI flags or load config files.
|
||||
|
||||
@@ -27,3 +33,11 @@ Replacement deletes only outputs recorded in existing destination state plus `.d
|
||||
## Tests
|
||||
|
||||
Before changing publish behavior, inspect tests under `internal/publish` and local run tests under `internal/app`.
|
||||
|
||||
## Invariants
|
||||
|
||||
- Publish planning is deterministic for the same source, destination state, policies, and transform outputs.
|
||||
- Replacement deletes only managed paths recorded in existing state plus `.distributor.json`.
|
||||
- Publish execution writes destination state after selected outputs are written.
|
||||
- Transform implementations are resolved through an interface supplied by the caller.
|
||||
- Unmanaged destination content is never overwritten.
|
||||
|
||||
@@ -31,6 +31,10 @@ Each output records `path`, `kind`, `source_path`, `sha256`, and `size`. Support
|
||||
|
||||
Comparison outcomes cover absent destination state, unmanaged destination content, invalid state, pipeline or destination mismatch, same source manifest, older destination source, newer destination source, same-created digest conflict, and different source id conflict.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
Invalid JSON, invalid state schema, invalid embedded source manifests, unsafe output paths, unsupported output kinds, missing generated-output transform names, and mismatched pipeline or destination ids produce comparison outcomes that publish planning can turn into fail actions.
|
||||
|
||||
## Boundaries
|
||||
|
||||
This package does not publish files, delete files, inspect storage backends, or choose transfer policy actions. Publish planning consumes these comparison outcomes later.
|
||||
@@ -38,3 +42,11 @@ This package does not publish files, delete files, inspect storage backends, or
|
||||
## Tests
|
||||
|
||||
Before changing destination state behavior, inspect tests under `internal/state`.
|
||||
|
||||
## Invariants
|
||||
|
||||
- `.distributor.json` is the destination sentinel and state record.
|
||||
- Embedded source manifests use the same validation rules as source bundles.
|
||||
- Generated outputs always record a transform id.
|
||||
- Comparison returns outcomes and reasons; it does not mutate storage.
|
||||
- `distributor_version` is diagnostic metadata, not a comparison key.
|
||||
|
||||
@@ -24,10 +24,18 @@ Logical file paths must be non-empty, relative, clean, slash-separated, and must
|
||||
|
||||
Storage errors use typed categories such as not found, already exists, invalid path, conflict, permission, temporary, unsupported, and unknown. Callers should use helper predicates rather than matching error strings.
|
||||
|
||||
Backends may wrap implementation-specific errors, but callers should receive storage errors where practical. Traversal can stop cleanly with `ErrStopWalk`.
|
||||
|
||||
## Deletion
|
||||
|
||||
Backends expose guarded managed deletion only. `DeleteManagedBundle` may delete listed managed outputs plus `.distributor.json`; it does not provide broad recursive deletion.
|
||||
|
||||
## Local and fake backends
|
||||
|
||||
The local adapter maps logical paths to a configured filesystem root and keeps adapter-specific path handling behind the storage interface.
|
||||
|
||||
The fake backend is an in-memory implementation for package tests. It is not registered for runtime use.
|
||||
|
||||
## Tests
|
||||
|
||||
Before changing storage behavior, inspect tests under:
|
||||
@@ -35,3 +43,11 @@ Before changing storage behavior, inspect tests under:
|
||||
- `internal/storage`
|
||||
- `internal/storage/fake`
|
||||
- `internal/adapters/local`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Core packages depend on `internal/storage`, not concrete adapters.
|
||||
- Logical paths are slash-separated and confined to the backend root.
|
||||
- `storage.List` uses backend traversal and returns deterministic entries.
|
||||
- Managed deletion is limited to recorded outputs plus `.distributor.json`.
|
||||
- Runtime backend registration is owned by `internal/app`.
|
||||
|
||||
@@ -8,12 +8,22 @@
|
||||
|
||||
Inputs are a validated source bundle and source backend. Outputs include destination path, source path, transform id, generated bytes, SHA-256, and size.
|
||||
|
||||
## Registry
|
||||
|
||||
`internal/transform` defines the transform interface and registry. The app layer registers the Markdown implementation; publish planning receives only a resolver.
|
||||
|
||||
## Markdown behavior
|
||||
|
||||
Markdown files ending in `.md` generate `.html` files in the same logical directory. Non-Markdown files do not generate outputs. Raw HTML embedded in Markdown is not passed through by the renderer.
|
||||
|
||||
Generated HTML is deterministic for the same source content and transform configuration.
|
||||
|
||||
See `docs/integrations/markdown.md` for the Goldmark integration contract.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
Transform resolution fails when a requested transform id is not registered. Markdown rendering fails when the source file cannot be read or rendered. Publish planning fails when HTML output is requested and the selected transform produces no outputs for a bundle.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Transforms do not publish files, mutate source bundles, or write destination state. Publish planning selects and writes transform outputs.
|
||||
@@ -22,4 +32,15 @@ The app layer owns default transform registration. The transform package does no
|
||||
|
||||
## Tests
|
||||
|
||||
Before changing transform behavior, inspect tests under `internal/transform`.
|
||||
Before changing transform behavior, inspect tests under:
|
||||
|
||||
- `internal/transform`
|
||||
- `internal/transform/markdown`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Source bundle files are never mutated by transforms.
|
||||
- Generated outputs record destination path, source path, transform id, SHA-256, and size.
|
||||
- Markdown sidecar naming changes only the `.md` extension to `.html`.
|
||||
- Non-Markdown source files do not generate Markdown outputs.
|
||||
- Transform registration stays outside publish planning.
|
||||
|
||||
@@ -1,6 +1,12 @@
|
||||
# Distributor Operations
|
||||
|
||||
## Normal workflow
|
||||
## Normal Workflow
|
||||
|
||||
Validate a source bundle:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate examples/source-bundle
|
||||
```
|
||||
|
||||
Preview a local publication:
|
||||
|
||||
@@ -20,30 +26,60 @@ Run the local HTML publication:
|
||||
go run ./cmd/distributor run --config examples/local-html.yml
|
||||
```
|
||||
|
||||
## Filesystem layout
|
||||
Preview local fan-out publication:
|
||||
|
||||
Source bundles are discovered beneath the configured local source root. Destination bundle paths preserve the source bundle path relative to that source root.
|
||||
```sh
|
||||
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
|
||||
```
|
||||
|
||||
The maintained example writes under `workspace/`, which is ignored by Git.
|
||||
## Filesystem Layout
|
||||
|
||||
## Destination state
|
||||
Source bundles are discovered beneath the configured local source root. Each bundle is a directory containing `manifest.json`.
|
||||
|
||||
Each published destination bundle contains `.distributor.json`. This state file records the source manifest, copied source outputs, and generated outputs. It is the authoritative marker that a destination path is managed by `distributor`.
|
||||
Destination bundle paths preserve the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under `daily/` publishes under `daily/` at each destination.
|
||||
|
||||
The maintained examples write under `workspace/`, which is ignored by Git.
|
||||
|
||||
## Destination State
|
||||
|
||||
Each published destination bundle contains `.distributor.json`. This file is the managed sentinel and destination state record. It stores:
|
||||
|
||||
- pipeline and destination identity;
|
||||
- publication timestamp;
|
||||
- source manifest used for publication;
|
||||
- copied source output metadata;
|
||||
- generated output metadata.
|
||||
|
||||
`manifest.json` from the source bundle is not copied as destination state.
|
||||
|
||||
## Retry behavior
|
||||
Do not edit `.distributor.json` by hand during normal operation. If it is missing or invalid while destination files remain, `distributor` treats the destination as unmanaged or conflicted.
|
||||
|
||||
If a destination already has matching `.distributor.json`, publication skips it as already published.
|
||||
## Dry Runs
|
||||
|
||||
If destination state is older than the source manifest, publication replaces only managed outputs recorded in `.distributor.json` plus the state file.
|
||||
`--dry-run` loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files or destination state.
|
||||
|
||||
If a write fails during local publication, `distributor` removes outputs written during that failed attempt where possible so a retry does not see an unmanaged destination.
|
||||
Dry-run output is useful before publishing to confirm actions such as `publish_new`, `replace_older`, `skip_same`, and `skip_destination_newer`.
|
||||
|
||||
If one destination fails, later destinations in the same fan-out are still planned and run where they are independent. The command exits non-zero after printing the final status when any destination fails.
|
||||
## Retry and Replacement Behavior
|
||||
|
||||
After a successful publish or replacement, the internal notifier hook runs as a no-op. Skipped destinations do not invoke it.
|
||||
If a destination has matching `.distributor.json`, publication skips it as already published.
|
||||
|
||||
If destination state is older than the source manifest and transfer policy allows replacement, publication deletes only managed outputs recorded in `.distributor.json` plus the state file, then writes the new outputs and state.
|
||||
|
||||
If destination state is newer than the source manifest, the default behavior is to skip. If destination state has the same source id and created timestamp but a different digest, publication fails as a conflict.
|
||||
|
||||
If a destination path has files but no valid `.distributor.json`, publication fails as unmanaged content. There is no force overwrite option.
|
||||
|
||||
## Failure Handling
|
||||
|
||||
If one destination fails in a fan-out run, independent later destinations are still planned and executed. The command exits non-zero after printing the final status if any destination failed.
|
||||
|
||||
If a write fails during local publication, `distributor` attempts to remove outputs written during that failed attempt so a retry does not see those partial outputs as unmanaged destination content.
|
||||
|
||||
After a successful publish or replacement, the internal notifier hook runs. The current default notifier is a no-op. Skipped destinations do not invoke it.
|
||||
|
||||
## Caveats
|
||||
|
||||
Only local-to-local publication is implemented. SSH, S3, external notification adapters, and force overwrite behavior are not implemented.
|
||||
Only local-to-local execution is available. SSH execution, S3 execution, external notification adapters, and force overwrite behavior are unavailable.
|
||||
|
||||
For symptom-oriented fixes, see [troubleshooting](troubleshooting.md). For config details, see [configuration](config.md). For command syntax, see [CLI](cli.md).
|
||||
|
||||
@@ -8,7 +8,7 @@ This document defines the development principles for `distributor`. It is inward
|
||||
|
||||
Producer applications generate manifested bundles. `distributor` discovers those bundles, validates them, optionally derives publication artifacts such as HTML, and publishes selected source and generated artifacts to one or more configured destinations.
|
||||
|
||||
`distributor` does not generate domain reports, interpret domain-specific report content, run producer pipelines, edit reports, or act as a CMS. Weather reports, D&D recaps, calendar summaries, email digests, and future report types should all enter `distributor` through the same bundle contract.
|
||||
`distributor` does not generate domain reports, interpret domain-specific report content, run producer pipelines, edit reports, or act as a CMS. Weather reports, D&D recaps, calendar summaries, email digests, and additional report types should all enter `distributor` through the same bundle contract.
|
||||
|
||||
## Project Shape
|
||||
|
||||
@@ -16,7 +16,7 @@ Default to a small, explicit, dependency-light Go application. Keep the design m
|
||||
|
||||
Business logic should live outside CLI, transport, and external-adapter packages. The core application should reason in terms of pipelines, bundles, destination state, transforms, and publish plans—not S3 SDK calls, SFTP sessions, shell commands, or filesystem details.
|
||||
|
||||
The core workflow is:
|
||||
The current core workflow is:
|
||||
|
||||
1. load configured pipelines;
|
||||
2. open the source backend;
|
||||
@@ -28,13 +28,13 @@ The core workflow is:
|
||||
8. optionally transform Markdown to HTML for that destination;
|
||||
9. publish selected source and generated artifacts;
|
||||
10. write `.distributor.json` as the destination sentinel/state file;
|
||||
11. run the notification stage, which is a no-op in the MVP.
|
||||
11. run the notification hook, which is a no-op in the MVP.
|
||||
|
||||
## Pipeline Model
|
||||
|
||||
A pipeline has exactly one source and one or more destinations.
|
||||
|
||||
The source is discovered and validated once. Each destination has independent backend configuration, publication policy, transform policy, replacement behavior, state, and future notification behavior.
|
||||
The source is discovered and validated once. Each destination has independent backend configuration, publication policy, transform policy, replacement behavior, state, and notification behavior.
|
||||
|
||||
The pipeline model is fan-out by design:
|
||||
|
||||
@@ -45,7 +45,7 @@ source bundle
|
||||
-> destination C: source files + HTML
|
||||
```
|
||||
|
||||
Destination-specific behavior must not leak back into the source bundle contract. A producer should not need to know whether a bundle will be published to local storage, SSH/SFTP, S3, a static site, email, RSS, or a future notification channel.
|
||||
Destination-specific behavior must not leak back into the source bundle contract. A producer should not need to know whether a bundle will be published to local storage, another storage backend, a static site, email, RSS, or another notification channel.
|
||||
|
||||
## Source Bundle Contract
|
||||
|
||||
@@ -166,13 +166,13 @@ For example, one destination may publish source files only as a long-term archiv
|
||||
|
||||
## Backend Abstraction
|
||||
|
||||
Sources and destinations use the same storage abstraction. Local filesystem, SSH/SFTP, and S3-compatible object storage are peer backends. Any backend may appear as a source or a destination unless a specific limitation is documented.
|
||||
Sources and destinations use the same storage abstraction. Current runtime execution uses the local filesystem backend. Additional storage backends should be peer implementations behind the same interface, and any backend-specific execution limitation must be documented.
|
||||
|
||||
Application logic must interact with storage through internal backend interfaces. Backend-specific behavior belongs in adapter packages. Pipeline, bundle, state, publish, and transform packages must not import S3, SSH/SFTP, or filesystem adapter implementation details.
|
||||
Application logic must interact with storage through internal backend interfaces. Backend-specific behavior belongs in adapter packages. Pipeline, bundle, state, publish, and transform packages must not import service-specific or filesystem adapter implementation details.
|
||||
|
||||
Adapters should be thin. Backend adapters should implement storage operations and translate backend-specific errors, but should not make bundle comparison, transform, routing, or replacement decisions.
|
||||
|
||||
SSH support should prefer a native SFTP implementation over shelling out to `ssh`, `scp`, or `rsync`, unless a later design document records a reason to differ.
|
||||
Remote file-transfer support should prefer native protocol implementations over shelling out, unless a later design document records a reason to differ.
|
||||
|
||||
## Dependency Policy
|
||||
|
||||
@@ -184,7 +184,7 @@ Avoid dependencies for small conveniences. Do not let external dependency types
|
||||
|
||||
## Package Layout
|
||||
|
||||
Use this layout unless the project has a documented reason to differ:
|
||||
Use this current layout unless the project has a documented reason to differ:
|
||||
|
||||
- `cmd/distributor`: application entrypoint only.
|
||||
- `internal/app`: application orchestration and top-level use cases.
|
||||
@@ -194,14 +194,14 @@ Use this layout unless the project has a documented reason to differ:
|
||||
- `internal/state`: `.distributor.json` parsing, validation, comparison, and output metadata.
|
||||
- `internal/storage`: backend interfaces, shared path/resource types, backend registry, and storage errors.
|
||||
- `internal/adapters/local`: local filesystem backend.
|
||||
- `internal/adapters/ssh`: SSH/SFTP backend.
|
||||
- `internal/adapters/s3`: S3-compatible object storage backend.
|
||||
- `internal/transform`: transform interfaces, registry, planning, and shared transform models.
|
||||
- `internal/transform/markdown`: Markdown-to-HTML implementation.
|
||||
- `internal/publish`: destination planning, reconciliation, safety checks, and publish execution.
|
||||
- `internal/notify`: notification interface and MVP no-op notifier.
|
||||
- `internal/logging`: logging setup and shared logging helpers.
|
||||
|
||||
New storage adapters should live under `internal/adapters/<name>` and stay thin.
|
||||
|
||||
Package-private implementation constants may live near the package that owns them, preferably in `constants.go` when useful.
|
||||
|
||||
## Configuration
|
||||
@@ -244,9 +244,9 @@ Each major stage should have an explicit input/output contract:
|
||||
- publish execution;
|
||||
- notification.
|
||||
|
||||
If users can select backends, transforms, notifiers, or future renderers, selection should go through a registry or equivalent mechanism rather than scattered conditionals.
|
||||
If users can select backends, transforms, notifiers, or renderers, selection should go through a registry or equivalent mechanism rather than scattered conditionals.
|
||||
|
||||
The orchestrator should be able to plan, dry-run, and execute configured pipelines. Dry-run behavior should be first-class because the application may delete, overwrite, or publish files to remote storage.
|
||||
The orchestrator should be able to plan, dry-run, and execute configured pipelines. Dry-run behavior should be first-class because the application may delete, overwrite, or publish files.
|
||||
|
||||
## Embedded Assets
|
||||
|
||||
@@ -264,7 +264,7 @@ Skip and no-op decisions should be logged at an appropriate level so operators c
|
||||
|
||||
## Context, Timeouts, and Cancellation
|
||||
|
||||
Long-running operations should accept `context.Context`. Storage operations, SSH/SFTP sessions, S3 requests, transforms, and multi-stage workflows should respect cancellation and timeouts.
|
||||
Long-running operations should accept `context.Context`. Storage operations, service requests, transforms, and multi-step workflows should respect cancellation and timeouts.
|
||||
|
||||
## State, Files, and Safety
|
||||
|
||||
@@ -282,7 +282,7 @@ Where practical, publish operations should use staging paths or temporary object
|
||||
|
||||
Core logic should be testable without real external services. Use fakes, fixtures, or local test doubles for adapters where practical.
|
||||
|
||||
Config examples should be load-tested. Important CLI workflows should have parser or command tests. Stage/module contracts should have focused tests that do not require running the full application unless end-to-end coverage is intentional.
|
||||
Config examples should be load-tested. Important CLI workflows should have parser or command tests. Component contracts should have focused tests that do not require running the full application unless end-to-end coverage is intentional.
|
||||
|
||||
Important tests include:
|
||||
|
||||
@@ -297,13 +297,13 @@ Important tests include:
|
||||
- transform output planning and metadata recording;
|
||||
- dry-run output;
|
||||
- local backend behavior with temporary directories;
|
||||
- fake backend behavior for S3 and SSH/SFTP-facing core logic.
|
||||
- fake backend behavior for storage-facing core logic.
|
||||
|
||||
## Documentation
|
||||
|
||||
Documentation should follow the project documentation policy. Keep user docs focused on implemented behavior. Put future, planned, or aspirational work only under `docs/roadmap/`.
|
||||
|
||||
When changing architecture, config, CLI behavior, adapters, manifest/state contracts, transform behavior, publish behavior, or stage/module contracts, update the relevant docs and examples in the same change.
|
||||
When changing architecture, config, CLI behavior, adapters, manifest/state contracts, transform behavior, publish behavior, or component contracts, update the relevant docs and examples in the same change.
|
||||
|
||||
The source manifest and destination `.distributor.json` schemas should have canonical documentation once implemented. Example configs should be valid and load-tested where practical.
|
||||
|
||||
@@ -321,4 +321,4 @@ The source manifest and destination `.distributor.json` schemas should have cano
|
||||
- a backup system;
|
||||
- a notification platform.
|
||||
|
||||
It may later support notification adapters, RSS/feed generation, richer HTML templates, or additional transforms, but those features must preserve the core bundle-distribution boundary.
|
||||
Additional notification, feed, template, or transform behavior must preserve the core bundle-distribution boundary.
|
||||
|
||||
@@ -1 +1,184 @@
|
||||
# Not yet implemented
|
||||
# Development Policy
|
||||
|
||||
This document defines the day-to-day development workflow for `distributor`.
|
||||
Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`.
|
||||
|
||||
## Repository Layout
|
||||
|
||||
- `cmd/distributor`: executable entrypoint only.
|
||||
- `internal/app`: top-level use cases for `run`, `validate`, and `inspect`.
|
||||
- `internal/cli`: standard-library command parsing, flags, help text, and command wiring.
|
||||
- `internal/config`: YAML configuration structs, loading, defaults, and validation.
|
||||
- `internal/bundle`: source bundle discovery, manifest parsing, digest calculation, and validation.
|
||||
- `internal/state`: destination `.distributor.json` parsing, validation, and comparison.
|
||||
- `internal/storage`: backend interface, registry, logical path rules, typed errors, and shared storage helpers.
|
||||
- `internal/adapters/local`: local filesystem backend.
|
||||
- `internal/storage/fake`: in-memory backend for tests.
|
||||
- `internal/publish`: destination inspection, output planning, reconciliation, execution, and managed cleanup.
|
||||
- `internal/transform`: transform interface and registry.
|
||||
- `internal/transform/markdown`: Markdown-to-HTML sidecar transform.
|
||||
- `internal/notify`: notification interface and current no-op notifier.
|
||||
- `internal/testutil`: shared test fixtures. Production code must not import this package.
|
||||
- `docs`: current user, operator, policy, internal, and roadmap documentation.
|
||||
- `examples`: copyable example configs and source bundles.
|
||||
|
||||
Do not create new top-level package families such as `pkg`, `internal/stage`,
|
||||
`internal/modules`, or service-specific adapter directories unless the
|
||||
architecture policy or a current roadmap explicitly calls for them.
|
||||
|
||||
## Common Commands
|
||||
|
||||
Run the full test suite:
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Run targeted packages while developing:
|
||||
|
||||
```bash
|
||||
go test ./internal/config
|
||||
go test ./internal/cli ./internal/app
|
||||
go test ./internal/publish ./internal/state
|
||||
go test ./internal/transform/markdown
|
||||
```
|
||||
|
||||
Run the CLI against an example config:
|
||||
|
||||
```bash
|
||||
go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
|
||||
```
|
||||
|
||||
Validate or inspect a local source bundle:
|
||||
|
||||
```bash
|
||||
go run ./cmd/distributor validate examples/source-bundle
|
||||
go run ./cmd/distributor inspect examples/source-bundle
|
||||
```
|
||||
|
||||
If Go cache permissions fail in a restricted environment, use workspace-safe
|
||||
temporary caches:
|
||||
|
||||
```bash
|
||||
GOCACHE=/private/tmp/distributor-gocache GOMODCACHE=/private/tmp/distributor-gomodcache go test ./...
|
||||
```
|
||||
|
||||
## Coding Conventions
|
||||
|
||||
- Keep the application small, explicit, and dependency-light.
|
||||
- Prefer package-local helpers over broad abstractions until behavior is shared by multiple packages.
|
||||
- Keep CLI parsing in `internal/cli`; business decisions belong in `internal/app`, `internal/bundle`, `internal/publish`, `internal/state`, and related core packages.
|
||||
- Keep adapter packages thin. Backend-specific filesystem or service behavior belongs in adapters; bundle, state, transform, and publish policy belongs outside adapters.
|
||||
- Preserve public CLI behavior, config semantics, manifest schema, destination state schema, and local MVP behavior unless the current task explicitly changes them.
|
||||
- Use `storage.DisplayPath`, `storage.StateFileName`, `storage.StatePath`, and `storage.ManagedBundleTargets` instead of duplicating those conventions.
|
||||
- Use `bundle.ValidateManifest` for normalized source manifest semantics, including embedded source manifests in destination state.
|
||||
- Use `config.ValidatePublishTransformPolicy` for publish and transform policy combinations.
|
||||
- Do not import concrete transform implementations from `internal/publish`; app-level wiring owns transform registration.
|
||||
- Do not import `internal/testutil` from production code.
|
||||
|
||||
## Dependency Policy
|
||||
|
||||
The project currently depends on:
|
||||
|
||||
- `gopkg.in/yaml.v3` for YAML configuration loading.
|
||||
- `github.com/yuin/goldmark` for Markdown rendering.
|
||||
|
||||
Add external dependencies only when they materially improve correctness,
|
||||
security, interoperability, or implementation complexity. Avoid dependencies
|
||||
for small conveniences. Do not let dependency-specific types leak across
|
||||
internal package boundaries unless that dependency is the explicit package
|
||||
contract.
|
||||
|
||||
## Configuration Changes
|
||||
|
||||
When adding or changing configuration:
|
||||
|
||||
1. Update `internal/config/config.go` structs and YAML tags.
|
||||
2. Add defaults in `internal/config/defaults.go` only for built-in defaults.
|
||||
3. Add validation in `internal/config/validate.go` with clear field context.
|
||||
4. Update config load and validation tests.
|
||||
5. Update `docs/config.md` in the same change if current user-visible config behavior changes.
|
||||
6. Update examples only with configs that are valid and executable for implemented behavior.
|
||||
|
||||
Config validation may accept fields for backends that are not executable yet,
|
||||
but user-facing docs and examples must clearly state execution support. At the
|
||||
time of this policy, only the local backend is executable.
|
||||
|
||||
## CLI Changes
|
||||
|
||||
The CLI is hand-written with the Go standard library. Do not introduce a CLI
|
||||
framework without a documented reason.
|
||||
|
||||
When adding or changing commands or flags:
|
||||
|
||||
1. Keep parsing and help text in `internal/cli`.
|
||||
2. Keep command work in `internal/app` or a lower-level package.
|
||||
3. Add or update CLI tests in `internal/cli`.
|
||||
4. Update `docs/cli.md` if syntax, flags, output expectations, or workflows change.
|
||||
|
||||
`validate` and `inspect` are local path commands. `run` loads configured
|
||||
pipelines and currently executes local backends only.
|
||||
|
||||
## Storage Backends
|
||||
|
||||
Storage behavior is defined by `internal/storage.Backend` and shared path rules
|
||||
in `internal/storage`.
|
||||
|
||||
When adding a backend:
|
||||
|
||||
1. Implement the storage interface in an adapter package.
|
||||
2. Translate backend-specific errors into storage errors where practical.
|
||||
3. Keep bundle comparison, transform, routing, and replacement policy out of the adapter.
|
||||
4. Register runtime construction through app-level backend factory wiring.
|
||||
5. Add focused adapter tests and app-level wiring tests.
|
||||
6. Update user docs, operations docs, examples, and internal docs only for behavior that is actually implemented.
|
||||
|
||||
Do not document SSH/SFTP or S3 execution as available until corresponding
|
||||
adapter packages and app wiring exist.
|
||||
|
||||
## Transforms
|
||||
|
||||
Transforms use `internal/transform` interfaces and registry wiring.
|
||||
|
||||
When adding or changing a transform:
|
||||
|
||||
1. Keep the transform implementation in its own package under `internal/transform`.
|
||||
2. Register default runtime transforms from `internal/app`.
|
||||
3. Keep `internal/publish` dependent only on the transform interface or resolver.
|
||||
4. Record deterministic output metadata: path, source path, transform name, digest, and size.
|
||||
5. Add transform tests and app or publish tests for wiring and policy behavior.
|
||||
6. Update `docs/internal/transform.md` and any relevant integration docs for implemented behavior.
|
||||
|
||||
## Tests
|
||||
|
||||
Test close to the behavior being changed:
|
||||
|
||||
- Use package tests for parsing, validation, comparison, planning, and adapter behavior.
|
||||
- Use `internal/app` and `internal/cli` tests for user-facing workflows.
|
||||
- Use `internal/testutil` for shared valid fixtures only; keep edge cases near the package under test.
|
||||
- Run `go test ./...` after cross-package changes or documentation/example changes tied to tests.
|
||||
|
||||
## Examples
|
||||
|
||||
Examples under `examples/` must be valid, maintained, and free of secrets.
|
||||
They should be copyable for implemented behavior. Do not leave examples that
|
||||
look runnable but require unsupported backend execution.
|
||||
|
||||
When changing examples:
|
||||
|
||||
1. Keep paths relative to the repository where practical.
|
||||
2. Prefer local examples until remote backend support exists.
|
||||
3. Run `go test ./internal/config` because config tests load examples.
|
||||
4. Update README, CLI, or config docs if links or recommended workflows change.
|
||||
|
||||
## Documentation
|
||||
|
||||
Follow `docs/policy/documentation.md`.
|
||||
|
||||
- Document implemented behavior outside `docs/roadmap/`.
|
||||
- Keep future, planned, or aspirational behavior under `docs/roadmap/`.
|
||||
- Keep `docs/config.md` canonical for user-facing config reference.
|
||||
- Keep `docs/cli.md` canonical for command syntax and workflows.
|
||||
- Keep `docs/operations.md` canonical for operational and recovery behavior.
|
||||
- Keep `docs/internal/` focused on implemented package contracts.
|
||||
- Update docs in the same change as behavior when public behavior, config, CLI, examples, or internal contracts change.
|
||||
|
||||
@@ -18,9 +18,9 @@ The current documentation and examples reviewed are:
|
||||
| `docs/cli.md` | Keep and update against CLI tests | Covers `version`, `run`, `validate`, and `inspect`; should be checked against command help and parser tests. |
|
||||
| `docs/config.md` | Keep and tighten | Current config reference documents local execution and also accepted SSH/S3 config fields. It must clearly distinguish config validation from executable backend support. |
|
||||
| `docs/operations.md` | Keep and expand slightly | Covers local workflow, destination state, retry behavior, cleanup, fan-out failure handling, and caveats. |
|
||||
| `docs/troubleshooting.md` | Missing, recommended | Recurring failure modes now exist and should be documented. |
|
||||
| `docs/troubleshooting.md` | Added | Covers recurring local MVP failure modes. |
|
||||
| `docs/policy/architecture.md` | Keep and clarify where needed | Development policy is broad and includes future adapter direction. Wording should not imply SSH/S3 adapters currently exist. |
|
||||
| `docs/policy/development.md` | Required rewrite | Currently contains only `# Not yet implemented`; this is the largest policy compliance gap. |
|
||||
| `docs/policy/development.md` | Updated | Contains contributor and agent workflow guidance. |
|
||||
| `docs/policy/documentation.md` | Keep | Canonical documentation policy. No change required unless the policy itself changes. |
|
||||
| `docs/internal/bundle.md` | Keep and verify | Describes implemented bundle parsing, discovery, validation, and digest semantics. |
|
||||
| `docs/internal/notify.md` | Keep and verify | Accurately states current no-op notification behavior. |
|
||||
@@ -28,10 +28,10 @@ The current documentation and examples reviewed are:
|
||||
| `docs/internal/state.md` | Keep and verify | Describes implemented `.distributor.json` state and comparison behavior. |
|
||||
| `docs/internal/storage.md` | Keep and verify | Describes storage interface, typed errors, path rules, traversal, and managed deletion. |
|
||||
| `docs/internal/transform.md` | Keep and verify | Describes Markdown-to-HTML sidecar behavior and transform boundaries. |
|
||||
| `docs/internal/app.md` | Missing, recommended | Needed for orchestration, backend factory, transform registry, dry-run, summaries, and notifier handoff. |
|
||||
| `docs/internal/config.md` | Missing, recommended | Needed for config loading, defaults, validation, accepted-but-not-executable backends, and example tests. |
|
||||
| `docs/internal/app.md` | Added | Covers orchestration, backend factory, transform registry, dry-run, summaries, and notifier handoff. |
|
||||
| `docs/internal/config.md` | Added | Covers config loading, defaults, validation, accepted-but-not-executable backends, and example tests. |
|
||||
| `docs/internal/local-backend.md` | Missing, optional | Local adapter behavior may remain in `docs/internal/storage.md`; create this only if local filesystem safety detail outgrows that doc. |
|
||||
| `docs/integrations/` | Missing, optional/recommended | Markdown rendering uses Goldmark. A concise Markdown integration note is useful because raw HTML behavior and deterministic rendering are externally visible. |
|
||||
| `docs/integrations/markdown.md` | Added | Documents current Goldmark-backed Markdown rendering behavior. |
|
||||
| `docs/roadmap/audit.md` | Historical roadmap/report | Keep under roadmap unless replaced by a new audit. |
|
||||
| `docs/roadmap/cleanup.md` | Historical or completed roadmap | Keep under roadmap; optionally add completion status in the documentation refresh. |
|
||||
| `docs/roadmap/config.md` | Roadmap | Keep as planning material; avoid linking to it as current config reference. |
|
||||
@@ -42,7 +42,7 @@ The current documentation and examples reviewed are:
|
||||
| `examples/local-to-local.yml` | Keep | Minimal local config; load-tested. |
|
||||
| `examples/local-publish.yml` | Keep | Runnable local publication example used by README and CLI docs. |
|
||||
| `examples/local-html.yml` | Keep | Runnable local HTML example. |
|
||||
| `examples/fan-out.yml` | Needs decision in refresh | Currently load-tested but uses SSH/S3 destinations that are not executable. Replace with a local-only fan-out example or move remote fan-out material under roadmap. |
|
||||
| `examples/fan-out.yml` | Updated | Local-only runnable fan-out example. |
|
||||
| `examples/source-bundle/` | Keep | Copyable valid source bundle fixture for local CLI examples. |
|
||||
|
||||
Implementation source areas inspected for documentation truth:
|
||||
@@ -73,20 +73,20 @@ Absent areas from earlier planning that should not be documented as implemented:
|
||||
|
||||
## Policy Compliance Assessment
|
||||
|
||||
Required current-behavior docs mostly exist for a config-driven, stateful, modular CLI, but three gaps should be closed before remote backend work resumes.
|
||||
Required current-behavior docs now exist for the config-driven, stateful, modular CLI.
|
||||
|
||||
Required fixes:
|
||||
Completed fixes from this refresh:
|
||||
|
||||
- Rewrite `docs/policy/development.md`; it is required by the documentation policy and is currently a placeholder.
|
||||
- Keep all non-roadmap docs scoped to implemented behavior. In particular, SSH/S3 execution, force overwrite, and external notification adapters must remain described as unavailable unless the corresponding code exists.
|
||||
- Resolve `examples/fan-out.yml`. It is valid config syntax, but it is not an executable example because SSH/S3 backends are not implemented for execution.
|
||||
- `docs/policy/development.md` contains real workflow guidance.
|
||||
- Non-roadmap docs are scoped to implemented behavior. SSH/S3 execution, force overwrite, and external notification adapters remain described as unavailable unless the corresponding code exists.
|
||||
- `examples/fan-out.yml` is local-only and runnable.
|
||||
|
||||
Recommended fixes:
|
||||
Completed recommended fixes from this refresh:
|
||||
|
||||
- Add `docs/troubleshooting.md` for recurring local MVP failure modes: invalid config, invalid source manifest, digest mismatch, unmanaged destination content, destination conflicts, unsupported remote execution, output path collisions, and failed writes.
|
||||
- Add `docs/internal/app.md` and `docs/internal/config.md` so future agents have one current-behavior internal reference for orchestration and config semantics.
|
||||
- Add a concise `docs/integrations/markdown.md` only if the project wants integration notes for Goldmark/CommonMark rendering behavior. This is recommended because Markdown rendering is externally visible and raw HTML handling is an important contract.
|
||||
- Add status notes to roadmap files that are now historical or completed so future agents do not treat old MVP planning as current behavior or active instructions.
|
||||
- `docs/troubleshooting.md` covers recurring local MVP failure modes.
|
||||
- `docs/internal/app.md` and `docs/internal/config.md` provide current-behavior internal references.
|
||||
- `docs/integrations/markdown.md` documents current Goldmark-backed rendering behavior.
|
||||
- Historical roadmap status notes were added where useful.
|
||||
|
||||
No broad rewrite is needed for `README.md`, `docs/cli.md`, `docs/config.md`, or `docs/operations.md`. They are close to the implemented local MVP and should be tightened against code and tests.
|
||||
|
||||
|
||||
135
docs/troubleshooting.md
Normal file
135
docs/troubleshooting.md
Normal file
@@ -0,0 +1,135 @@
|
||||
# Distributor Troubleshooting
|
||||
|
||||
## `load config ... no such file or directory`
|
||||
|
||||
Likely cause: `run` could not find the config path. If `--config` is omitted, the default path is `/usr/local/etc/distributor/config.yml`.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
ls -l <config-path>
|
||||
```
|
||||
|
||||
Safe fix: pass an existing config path with `--config`, or install a config at the default path. See [configuration](config.md).
|
||||
|
||||
## `parse config ... field not found`
|
||||
|
||||
Likely cause: the YAML contains an unknown field. Config loading rejects unknown keys.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run
|
||||
```
|
||||
|
||||
Safe fix: compare the file to the reference in [configuration](config.md) and remove or rename unsupported fields.
|
||||
|
||||
## `validate config ... backend ... is unsupported`
|
||||
|
||||
Likely cause: a source or destination uses a backend name other than `local`, `ssh`, or `s3`.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
rg -n "backend:" <config-path>
|
||||
```
|
||||
|
||||
Safe fix: use `backend: local` for executable workflows. SSH and S3 config shapes are accepted only for validation; runtime execution is unavailable.
|
||||
|
||||
## `backend ssh is not implemented for execution` or `backend s3 is not implemented for execution`
|
||||
|
||||
Likely cause: the config validates but `run` tried to execute a remote backend.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run
|
||||
```
|
||||
|
||||
Safe fix: use local destinations for current executable workflows, or keep remote backend configs under roadmap material unless those adapters are added. See [configuration](config.md).
|
||||
|
||||
## `validate command requires a path` or `inspect command requires a path`
|
||||
|
||||
Likely cause: `validate` or `inspect` was run without a path.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate --help
|
||||
go run ./cmd/distributor inspect --help
|
||||
```
|
||||
|
||||
Safe fix: pass a local source bundle directory or a local tree containing source bundles.
|
||||
|
||||
## `no bundles found under "."`
|
||||
|
||||
Likely cause: the selected source root does not contain a `manifest.json` source bundle.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
find <source-root> -name manifest.json -print
|
||||
```
|
||||
|
||||
Safe fix: point the command or config at the directory containing the source bundle, or write a valid `manifest.json` and listed files. See [CLI](cli.md).
|
||||
|
||||
## `sha256 mismatch`, `size mismatch`, or `digest mismatch`
|
||||
|
||||
Likely cause: a listed source file changed after `manifest.json` was created, or the manifest digest does not match its file list.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate <source-root>
|
||||
```
|
||||
|
||||
Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source digest failures.
|
||||
|
||||
## `destination has content but no distributor state`
|
||||
|
||||
Likely cause: the destination path is not empty and has no `.distributor.json` state file, so `distributor` will not claim it as managed.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
find <destination-path> -maxdepth 2 -print
|
||||
```
|
||||
|
||||
Safe fix: choose an empty destination path or move existing files aside after confirming they are not needed. There is no force overwrite option.
|
||||
|
||||
## `fail_conflict`
|
||||
|
||||
Likely cause: existing `.distributor.json` belongs to a different pipeline, a different destination, a different source id, or a same-created source with a different digest.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
cat <destination-path>/.distributor.json
|
||||
go run ./cmd/distributor inspect <source-root>
|
||||
```
|
||||
|
||||
Safe fix: verify you are publishing the intended source to the intended destination. Use a separate destination path for unrelated content.
|
||||
|
||||
## `destination output path collision`
|
||||
|
||||
Likely cause: configured publication would write two outputs to the same destination path, such as publishing a source `report.html` while also generating `report.html` from `report.md`.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run
|
||||
```
|
||||
|
||||
Safe fix: adjust the source bundle contents or publish policy so source and generated outputs do not collide.
|
||||
|
||||
## A run failed after writing some files
|
||||
|
||||
Likely cause: a write failed partway through publication. Local execution attempts to clean up outputs written during the failed attempt.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
find <destination-path> -maxdepth 2 -print
|
||||
```
|
||||
|
||||
Safe fix: inspect the destination before retrying. If only unrelated unmanaged files remain, move them aside or choose a clean destination. Re-run with `--dry-run` before publishing again. See [operations](operations.md).
|
||||
@@ -1,28 +1,18 @@
|
||||
pipelines:
|
||||
- id: reports
|
||||
- id: example-fan-out
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/reports
|
||||
validation:
|
||||
on_digest_mismatch: fail
|
||||
path: examples/source-bundle
|
||||
destinations:
|
||||
- id: markdown-archive
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: archive
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
- id: local-source-archive
|
||||
backend: local
|
||||
path: workspace/published/fan-out/source
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
- id: static-site
|
||||
backend: ssh
|
||||
uri: ssh://deploy@example.com:22
|
||||
path: /srv/www/reports
|
||||
- id: local-html-site
|
||||
backend: local
|
||||
path: workspace/published/fan-out/html
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
|
||||
Reference in New Issue
Block a user