218 lines
10 KiB
Markdown
218 lines
10 KiB
Markdown
# Development Policy
|
|
|
|
This document defines the day-to-day development workflow for `distributor`.
|
|
Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`.
|
|
|
|
## Repository Layout
|
|
|
|
- `cmd/distributor`: executable entrypoint only.
|
|
- `pkg/bundle`: public producer-facing source manifest and local bundle writer helpers.
|
|
- `pkg/upload`: public producer-facing HTTP upload client built on `pkg/bundle`.
|
|
- `internal/app`: top-level use cases for `run`, `validate`, and `inspect`.
|
|
- `internal/cli`: standard-library command parsing, flags, help text, and command wiring.
|
|
- `internal/config`: YAML configuration structs, loading, defaults, and validation.
|
|
- `internal/bundle`: storage-backed source bundle discovery and validation using the public manifest contract.
|
|
- `internal/state`: destination `.distributor.json` parsing, validation, and comparison.
|
|
- `internal/storage`: backend interface, registry, logical path rules, typed errors, and shared storage helpers.
|
|
- `internal/adapters/local`: local filesystem backend.
|
|
- `internal/adapters/ssh`: SSH/SFTP backend.
|
|
- `internal/adapters/s3`: S3-compatible object storage backend.
|
|
- `internal/storage/fake`: in-memory backend for tests.
|
|
- `internal/publish`: destination inspection, output planning, catalog workflow planning, execution, managed cleanup, and explicit forced replacement.
|
|
- `internal/transform`: transform interface and registry.
|
|
- `internal/transform/markdown`: Markdown-to-HTML transform.
|
|
- `internal/notify`: notification interface and current no-op notifier.
|
|
- `internal/testutil`: shared test fixtures. Production code must not import this package.
|
|
- `docs`: current user, operator, consumer, integration, policy, internal, and roadmap documentation.
|
|
- `examples`: copyable example configs and source bundles.
|
|
|
|
Do not create new top-level package families such as public `pkg/...` packages
|
|
beyond `pkg/bundle` and `pkg/upload`, generic workflow containers, or
|
|
service-specific adapter directories unless the architecture policy or a
|
|
current roadmap explicitly calls for them.
|
|
|
|
## Common Commands
|
|
|
|
Run the full test suite:
|
|
|
|
```bash
|
|
go test ./...
|
|
```
|
|
|
|
Run targeted packages while developing:
|
|
|
|
```bash
|
|
go test ./internal/config
|
|
go test ./internal/cli ./internal/app
|
|
go test ./internal/publish ./internal/state
|
|
go test ./internal/transform/markdown
|
|
go test ./pkg/bundle ./pkg/upload
|
|
```
|
|
|
|
Run the CLI against an example config:
|
|
|
|
```bash
|
|
go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
|
|
```
|
|
|
|
Validate or inspect a local source bundle:
|
|
|
|
```bash
|
|
go run ./cmd/distributor validate examples/source-bundle
|
|
go run ./cmd/distributor inspect examples/source-bundle
|
|
```
|
|
|
|
If Go cache permissions fail in a restricted environment, use workspace-safe
|
|
temporary caches:
|
|
|
|
```bash
|
|
GOCACHE=/private/tmp/distributor-gocache GOMODCACHE=/private/tmp/distributor-gomodcache go test ./...
|
|
```
|
|
|
|
## Coding Conventions
|
|
|
|
- Keep the application small, explicit, and dependency-light.
|
|
- Prefer package-local helpers over broad abstractions until behavior is shared by multiple packages.
|
|
- Keep CLI parsing in `internal/cli`; business decisions belong in `internal/app`, `internal/bundle`, `internal/publish`, `internal/state`, and related core packages.
|
|
- Keep adapter packages thin. Backend-specific filesystem or service behavior belongs in adapters; bundle, state, transform, and publish policy belongs outside adapters.
|
|
- Preserve public CLI behavior, config semantics, manifest schema, destination state schema, and implemented backend behavior unless the current task explicitly changes them.
|
|
- Use `storage.DisplayPath`, `storage.StateFileName`, `storage.StatePath`, and `storage.ManagedBundleTargets` instead of duplicating those conventions.
|
|
- Use `pkg/bundle` for normalized source manifest semantics. Internal packages should reach those rules through `internal/bundle` when they also need storage-backed bundle discovery or validation.
|
|
- Keep `pkg/upload` as a producer-facing HTTP client. It should depend on `pkg/bundle` and standard HTTP/archive primitives, not on `internal/app`, `internal/ingest`, server config, storage backends, or destination state types.
|
|
- Use `config.ValidatePublishTransformPolicy` for publish and transform policy combinations.
|
|
- Do not import concrete transform implementations from `internal/publish`; app-level wiring owns transform registration.
|
|
- Do not import `internal/testutil` from production code.
|
|
|
|
## Dependency Policy
|
|
|
|
The project currently depends on:
|
|
|
|
- `gopkg.in/yaml.v3` for YAML configuration loading.
|
|
- `github.com/yuin/goldmark` for Markdown rendering.
|
|
- `golang.org/x/crypto/ssh`, `golang.org/x/crypto/ssh/agent`, and `golang.org/x/crypto/ssh/knownhosts` for native SSH support.
|
|
- `github.com/pkg/sftp` for native SFTP support.
|
|
- `github.com/aws/aws-sdk-go-v2/...` packages for S3-compatible storage support.
|
|
|
|
Add external dependencies only when they materially improve correctness,
|
|
security, interoperability, or implementation complexity. Avoid dependencies
|
|
for small conveniences. Do not let dependency-specific types leak across
|
|
internal package boundaries unless that dependency is the explicit package
|
|
contract.
|
|
|
|
## Configuration Changes
|
|
|
|
When adding or changing configuration:
|
|
|
|
1. Update `internal/config/config.go` structs and YAML tags.
|
|
2. Add defaults in `internal/config/defaults.go` only for built-in defaults.
|
|
3. Add validation in `internal/config/validate.go` with clear field context.
|
|
4. Update config load and validation tests.
|
|
5. Update `docs/config.md` in the same change if current user-visible config behavior changes.
|
|
6. Update examples only with configs that are valid and executable for implemented behavior.
|
|
|
|
Config validation may accept fields for roadmap backends before execution
|
|
support exists, but user-facing docs and examples must clearly state execution
|
|
support. Runtime executable backends are local, SSH, and S3.
|
|
|
|
Credential-consuming code must use the config-owned environment resolver for
|
|
explicit credential environment variable references. Do not call `os.Getenv`
|
|
directly for backend credentials, because `secrets.directory` values are
|
|
intentionally available through the resolver without mutating the process
|
|
environment.
|
|
|
|
## CLI Changes
|
|
|
|
The CLI is hand-written with the Go standard library. Do not introduce a CLI
|
|
framework without a documented reason.
|
|
|
|
When adding or changing commands or flags:
|
|
|
|
1. Keep parsing and help text in `internal/cli`.
|
|
2. Keep command work in `internal/app` or a lower-level package.
|
|
3. Add or update CLI tests in `internal/cli`.
|
|
4. Update `docs/cli.md` if syntax, flags, output expectations, or workflows change.
|
|
|
|
`validate` and `inspect` support a local path shortcut and configured
|
|
source-only diagnostics. `run` loads configured pipelines and executes local,
|
|
SSH, and S3 backends.
|
|
|
|
## Storage Backends
|
|
|
|
Storage behavior is defined by `internal/storage.Backend` and shared path rules
|
|
in `internal/storage`.
|
|
|
|
When adding a backend:
|
|
|
|
1. Implement the storage interface in an adapter package.
|
|
2. Translate backend-specific errors into storage errors where practical.
|
|
3. Keep bundle comparison, transform, routing, and replacement policy out of the adapter.
|
|
4. Register runtime construction through app-level backend factory wiring.
|
|
5. Add focused adapter tests and app-level wiring tests.
|
|
6. Update user docs, operations docs, examples, and internal docs only for behavior that is actually implemented.
|
|
|
|
Do not document future backend execution as available until the corresponding
|
|
adapter package and app wiring exist.
|
|
|
|
## Transforms
|
|
|
|
Transforms use `internal/transform` interfaces and registry wiring.
|
|
|
|
When adding or changing a transform:
|
|
|
|
1. Keep the transform implementation in its own package under `internal/transform`.
|
|
2. Register default runtime transforms from `internal/app`.
|
|
3. Keep `internal/publish` dependent only on the transform interface or resolver.
|
|
4. Record deterministic output metadata: path, source path, transform name, digest, and size.
|
|
5. Add transform tests and app or publish tests for wiring and policy behavior.
|
|
6. Update `docs/internal/transform.md` and any relevant integration docs for implemented behavior.
|
|
|
|
## Tests
|
|
|
|
Test close to the behavior being changed:
|
|
|
|
- Use package tests for parsing, validation, comparison, planning, and adapter behavior.
|
|
- Use `internal/app` and `internal/cli` tests for user-facing workflows.
|
|
- Use `internal/testutil` for shared valid fixtures only; keep edge cases near the package under test.
|
|
- Run `go test ./...` after cross-package changes or documentation/example changes tied to tests.
|
|
- Run `go test ./pkg/bundle ./pkg/upload` after changing producer-facing bundle or upload APIs.
|
|
|
|
Live integration tests must be opt-in and skipped during normal `go test ./...`
|
|
unless their required environment variables are set. Test-only environment
|
|
variables must use this prefix shape:
|
|
|
|
```text
|
|
DISTRIBUTOR_TEST_<BACKEND>_*
|
|
```
|
|
|
|
Examples include `DISTRIBUTOR_TEST_SSH_HOST` and
|
|
`DISTRIBUTOR_TEST_S3_ENDPOINT`. Do not use production credential variable names
|
|
for test-only controls.
|
|
|
|
## Examples
|
|
|
|
Examples under `examples/` must be valid, maintained, and free of secrets.
|
|
They should be copyable for implemented behavior. Remote examples must use
|
|
placeholders or environment variables for endpoint and credential material.
|
|
|
|
When changing examples:
|
|
|
|
1. Keep paths relative to the repository where practical.
|
|
2. Keep local examples runnable without external services; gate remote examples
|
|
behind user-provided endpoints and credentials.
|
|
3. Run `go test ./internal/config` because config tests load examples.
|
|
4. Update README, CLI, or config docs if links or recommended workflows change.
|
|
|
|
## Documentation
|
|
|
|
Follow `docs/policy/documentation.md`.
|
|
|
|
- Document implemented behavior outside `docs/roadmap/`.
|
|
- Keep future, planned, or aspirational behavior under `docs/roadmap/`.
|
|
- Keep `docs/config.md` canonical for user-facing config reference.
|
|
- Keep `docs/cli.md` canonical for command syntax and workflows.
|
|
- Keep `docs/operations.md` canonical for operational and recovery behavior.
|
|
- Keep `docs/consumers/` canonical for public package and consumer API workflows.
|
|
- Keep `docs/integrations/` canonical for external file-format and wire-protocol contracts.
|
|
- Keep `docs/internal/` focused on implemented package contracts.
|
|
- Update docs in the same change as behavior when public behavior, public packages/APIs, config, CLI, examples, or internal contracts change.
|