distributor/docs/policy/development.md

Development Policy

This document defines the day-to-day development workflow for distributor. Use it with docs/policy/architecture.md and docs/policy/documentation.md.

Repository Layout

• cmd/distributor: executable entrypoint only.
• pkg/bundle: public producer-facing source manifest and local bundle writer helpers.
• internal/app: top-level use cases for run, validate, and inspect.
• internal/cli: standard-library command parsing, flags, help text, and command wiring.
• internal/config: YAML configuration structs, loading, defaults, and validation.
• internal/bundle: storage-backed source bundle discovery and validation using the public manifest contract.
• internal/state: destination .distributor.json parsing, validation, and comparison.
• internal/storage: backend interface, registry, logical path rules, typed errors, and shared storage helpers.
• internal/adapters/local: local filesystem backend.
• internal/adapters/ssh: SSH/SFTP backend.
• internal/adapters/s3: S3-compatible object storage backend.
• internal/storage/fake: in-memory backend for tests.
• internal/publish: destination inspection, output planning, reconciliation, execution, managed cleanup, and explicit forced replacement.
• internal/transform: transform interface and registry.
• internal/transform/markdown: Markdown-to-HTML transform.
• internal/notify: notification interface and current no-op notifier.
• internal/testutil: shared test fixtures. Production code must not import this package.
• docs: current user, operator, policy, internal, and roadmap documentation.
• examples: copyable example configs and source bundles.

Do not create new top-level package families such as public pkg/... packages beyond pkg/bundle, generic workflow containers, or service-specific adapter directories unless the architecture policy or a current roadmap explicitly calls for them.

Common Commands

Run the full test suite:

go test ./...

Run targeted packages while developing:

go test ./internal/config
go test ./internal/cli ./internal/app
go test ./internal/publish ./internal/state
go test ./internal/transform/markdown

Run the CLI against an example config:

go run ./cmd/distributor run --config examples/local-publish.yml --dry-run

Validate or inspect a local source bundle:

go run ./cmd/distributor validate examples/source-bundle
go run ./cmd/distributor inspect examples/source-bundle

If Go cache permissions fail in a restricted environment, use workspace-safe temporary caches:

GOCACHE=/private/tmp/distributor-gocache GOMODCACHE=/private/tmp/distributor-gomodcache go test ./...

Coding Conventions

• Keep the application small, explicit, and dependency-light.
• Prefer package-local helpers over broad abstractions until behavior is shared by multiple packages.
• Keep CLI parsing in internal/cli; business decisions belong in internal/app, internal/bundle, internal/publish, internal/state, and related core packages.
• Keep adapter packages thin. Backend-specific filesystem or service behavior belongs in adapters; bundle, state, transform, and publish policy belongs outside adapters.
• Preserve public CLI behavior, config semantics, manifest schema, destination state schema, and implemented backend behavior unless the current task explicitly changes them.
• Use storage.DisplayPath, storage.StateFileName, storage.StatePath, and storage.ManagedBundleTargets instead of duplicating those conventions.
• Use pkg/bundle for normalized source manifest semantics. Internal packages should reach those rules through internal/bundle when they also need storage-backed bundle discovery or validation.
• Use config.ValidatePublishTransformPolicy for publish and transform policy combinations.
• Do not import concrete transform implementations from internal/publish; app-level wiring owns transform registration.
• Do not import internal/testutil from production code.

Dependency Policy

The project currently depends on:

• gopkg.in/yaml.v3 for YAML configuration loading.
• github.com/yuin/goldmark for Markdown rendering.
• golang.org/x/crypto/ssh, golang.org/x/crypto/ssh/agent, and golang.org/x/crypto/ssh/knownhosts for native SSH support.
• github.com/pkg/sftp for native SFTP support.
• github.com/aws/aws-sdk-go-v2/... packages for S3-compatible storage support.

Add external dependencies only when they materially improve correctness, security, interoperability, or implementation complexity. Avoid dependencies for small conveniences. Do not let dependency-specific types leak across internal package boundaries unless that dependency is the explicit package contract.

Configuration Changes

When adding or changing configuration:

1. Update internal/config/config.go structs and YAML tags.
2. Add defaults in internal/config/defaults.go only for built-in defaults.
3. Add validation in internal/config/validate.go with clear field context.
4. Update config load and validation tests.
5. Update docs/config.md in the same change if current user-visible config behavior changes.
6. Update examples only with configs that are valid and executable for implemented behavior.

Config validation may accept fields for roadmap backends before execution support exists, but user-facing docs and examples must clearly state execution support. Runtime executable backends are local, SSH, and S3.

Credential-consuming code must use the config-owned environment resolver for explicit credential environment variable references. Do not call os.Getenv directly for backend credentials, because secrets.directory values are intentionally available through the resolver without mutating the process environment.

CLI Changes

The CLI is hand-written with the Go standard library. Do not introduce a CLI framework without a documented reason.

When adding or changing commands or flags:

1. Keep parsing and help text in internal/cli.
2. Keep command work in internal/app or a lower-level package.
3. Add or update CLI tests in internal/cli.
4. Update docs/cli.md if syntax, flags, output expectations, or workflows change.

validate and inspect support a local path shortcut and configured source-only diagnostics. run loads configured pipelines and executes local, SSH, and S3 backends.

Storage Backends

Storage behavior is defined by internal/storage.Backend and shared path rules in internal/storage.

When adding a backend:

1. Implement the storage interface in an adapter package.
2. Translate backend-specific errors into storage errors where practical.
3. Keep bundle comparison, transform, routing, and replacement policy out of the adapter.
4. Register runtime construction through app-level backend factory wiring.
5. Add focused adapter tests and app-level wiring tests.
6. Update user docs, operations docs, examples, and internal docs only for behavior that is actually implemented.

Do not document future backend execution as available until the corresponding adapter package and app wiring exist.

Transforms

Transforms use internal/transform interfaces and registry wiring.

When adding or changing a transform:

1. Keep the transform implementation in its own package under internal/transform.
2. Register default runtime transforms from internal/app.
3. Keep internal/publish dependent only on the transform interface or resolver.
4. Record deterministic output metadata: path, source path, transform name, digest, and size.
5. Add transform tests and app or publish tests for wiring and policy behavior.
6. Update docs/internal/transform.md and any relevant integration docs for implemented behavior.

Tests

Test close to the behavior being changed:

• Use package tests for parsing, validation, comparison, planning, and adapter behavior.
• Use internal/app and internal/cli tests for user-facing workflows.
• Use internal/testutil for shared valid fixtures only; keep edge cases near the package under test.
• Run go test ./... after cross-package changes or documentation/example changes tied to tests.

Live integration tests must be opt-in and skipped during normal go test ./... unless their required environment variables are set. Test-only environment variables must use this prefix shape:

DISTRIBUTOR_TEST_<BACKEND>_*

Examples include DISTRIBUTOR_TEST_SSH_HOST and DISTRIBUTOR_TEST_S3_ENDPOINT. Do not use production credential variable names for test-only controls.

Examples

Examples under examples/ must be valid, maintained, and free of secrets. They should be copyable for implemented behavior. Remote examples must use placeholders or environment variables for endpoint and credential material.

When changing examples:

1. Keep paths relative to the repository where practical.
2. Keep local examples runnable without external services; gate remote examples behind user-provided endpoints and credentials.
3. Run go test ./internal/config because config tests load examples.
4. Update README, CLI, or config docs if links or recommended workflows change.

Documentation

Follow docs/policy/documentation.md.

• Document implemented behavior outside docs/roadmap/.
• Keep future, planned, or aspirational behavior under docs/roadmap/.
• Keep docs/config.md canonical for user-facing config reference.
• Keep docs/cli.md canonical for command syntax and workflows.
• Keep docs/operations.md canonical for operational and recovery behavior.
• Keep docs/internal/ focused on implemented package contracts.
• Update docs in the same change as behavior when public behavior, config, CLI, examples, or internal contracts change.