Refresh roadmap documentation
This commit is contained in:
@@ -71,7 +71,7 @@ GOCACHE=/private/tmp/distributor-gocache GOMODCACHE=/private/tmp/distributor-gom
|
||||
- Prefer package-local helpers over broad abstractions until behavior is shared by multiple packages.
|
||||
- Keep CLI parsing in `internal/cli`; business decisions belong in `internal/app`, `internal/bundle`, `internal/publish`, `internal/state`, and related core packages.
|
||||
- Keep adapter packages thin. Backend-specific filesystem or service behavior belongs in adapters; bundle, state, transform, and publish policy belongs outside adapters.
|
||||
- Preserve public CLI behavior, config semantics, manifest schema, destination state schema, and local MVP behavior unless the current task explicitly changes them.
|
||||
- Preserve public CLI behavior, config semantics, manifest schema, destination state schema, and implemented backend behavior unless the current task explicitly changes them.
|
||||
- Use `storage.DisplayPath`, `storage.StateFileName`, `storage.StatePath`, and `storage.ManagedBundleTargets` instead of duplicating those conventions.
|
||||
- Use `bundle.ValidateManifest` for normalized source manifest semantics, including embedded source manifests in destination state.
|
||||
- Use `config.ValidatePublishTransformPolicy` for publish and transform policy combinations.
|
||||
@@ -105,9 +105,9 @@ When adding or changing configuration:
|
||||
5. Update `docs/config.md` in the same change if current user-visible config behavior changes.
|
||||
6. Update examples only with configs that are valid and executable for implemented behavior.
|
||||
|
||||
Config validation may accept fields for backends that are not executable yet,
|
||||
but user-facing docs and examples must clearly state execution support. At the
|
||||
time of this policy, local, SSH, and S3 backends are executable.
|
||||
Config validation may accept fields for roadmap backends before execution
|
||||
support exists, but user-facing docs and examples must clearly state execution
|
||||
support. Runtime executable backends are local, SSH, and S3.
|
||||
|
||||
Credential-consuming code must use the config-owned environment resolver for
|
||||
explicit credential environment variable references. Do not call `os.Getenv`
|
||||
@@ -128,7 +128,7 @@ When adding or changing commands or flags:
|
||||
4. Update `docs/cli.md` if syntax, flags, output expectations, or workflows change.
|
||||
|
||||
`validate` and `inspect` are local path commands. `run` loads configured
|
||||
pipelines and currently executes local, SSH, and S3 backends.
|
||||
pipelines and executes local, SSH, and S3 backends.
|
||||
|
||||
## Storage Backends
|
||||
|
||||
@@ -184,13 +184,14 @@ for test-only controls.
|
||||
## Examples
|
||||
|
||||
Examples under `examples/` must be valid, maintained, and free of secrets.
|
||||
They should be copyable for implemented behavior. Do not leave examples that
|
||||
look runnable but require unsupported backend execution.
|
||||
They should be copyable for implemented behavior. Remote examples must use
|
||||
placeholders or environment variables for endpoint and credential material.
|
||||
|
||||
When changing examples:
|
||||
|
||||
1. Keep paths relative to the repository where practical.
|
||||
2. Prefer local examples until remote backend support exists.
|
||||
2. Keep local examples runnable without external services; gate remote examples
|
||||
behind user-provided endpoints and credentials.
|
||||
3. Run `go test ./internal/config` because config tests load examples.
|
||||
4. Update README, CLI, or config docs if links or recommended workflows change.
|
||||
|
||||
|
||||
@@ -1,656 +1,97 @@
|
||||
# Post-Local-MVP Implementation Roadmap
|
||||
# Implementation Roadmap
|
||||
|
||||
This is the canonical active roadmap for `distributor` after the local MVP checkpoint.
|
||||
This roadmap records current implementation status and deferred work for
|
||||
`distributor`. Implemented behavior is documented in the user, operator, and
|
||||
internal documentation listed below.
|
||||
|
||||
The original MVP stages 1-8 are complete and are no longer listed as pending implementation work. Current behavior is documented outside the roadmap in `README.md`, `docs/cli.md`, `docs/config.md`, `docs/operations.md`, `docs/troubleshooting.md`, `docs/internal/`, `docs/integrations/markdown.md`, and `docs/policy/development.md`.
|
||||
Canonical current-behavior docs:
|
||||
|
||||
Future, planned, or aspirational behavior should remain under `docs/roadmap/` until implemented.
|
||||
- `README.md`
|
||||
- `docs/cli.md`
|
||||
- `docs/config.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/troubleshooting.md`
|
||||
- `docs/internal/`
|
||||
- `docs/integrations/markdown.md`
|
||||
- `docs/policy/`
|
||||
- `examples/`
|
||||
|
||||
## Current Baseline
|
||||
Future, planned, or aspirational behavior belongs under `docs/roadmap/` until
|
||||
it is implemented.
|
||||
|
||||
The implemented local MVP includes:
|
||||
## Current State
|
||||
|
||||
- standard-library CLI commands for `version`, `run`, `validate`, and `inspect`;
|
||||
`distributor` is ready for routine use against producer pipelines using the
|
||||
implemented local, SSH/SFTP, and S3-compatible backends.
|
||||
|
||||
Implemented capabilities:
|
||||
|
||||
- standard-library CLI commands for `version`, `run`, `validate`, and
|
||||
`inspect`;
|
||||
- YAML config loading, defaults, known-field rejection, and validation;
|
||||
- accepted config shapes for `local`, `ssh`, and `s3`, with executable backend support currently limited to `local`;
|
||||
- backend-rooted storage interface with typed errors, safe logical paths, traversal, `HasAny`, managed deletion, local backend, and fake backend;
|
||||
- source bundle discovery, manifest parsing, RFC3339 timestamp handling, duplicate path checks, path safety checks, symlink rejection, per-file digest validation, and bundle digest validation;
|
||||
- destination `.distributor.json` state parsing, validation, output metadata, and source comparison;
|
||||
- local publication of source files, Markdown sidecar HTML, or both;
|
||||
- local, SSH/SFTP, and S3-compatible source and destination backends;
|
||||
- structured SSH/SFTP configuration with agent and key-file authentication,
|
||||
known-hosts handling, and backend-rooted path confinement;
|
||||
- S3-compatible configuration with endpoint, region, addressing mode, TLS
|
||||
controls, credential environment references, streaming reads and writes,
|
||||
listing, metadata, and managed deletion;
|
||||
- top-level `secrets.directory` loading for credential values, with process
|
||||
environment precedence and without mutating the process environment;
|
||||
- backend-rooted storage interfaces with typed errors, safe logical paths,
|
||||
traversal, collision checks, managed deletion, and explicit forced
|
||||
replacement;
|
||||
- source bundle discovery, manifest parsing, RFC3339 timestamp handling,
|
||||
duplicate path checks, path safety checks, symlink rejection, per-file digest
|
||||
validation, and bundle digest validation;
|
||||
- destination `.distributor.json` state parsing, validation, output metadata,
|
||||
and source comparison;
|
||||
- publication of source files, Markdown sidecar HTML, or both;
|
||||
- destination output collision detection before writes;
|
||||
- managed replacement for older destination state;
|
||||
- unmanaged destination and conflict failures by default;
|
||||
- conservative conflict failures by default for unmanaged destination content
|
||||
and conflicting managed destination content;
|
||||
- explicit forced replacement with `run --force`, limited to the configured
|
||||
destination bundle path;
|
||||
- deterministic dry-run output and final run summaries;
|
||||
- deterministic sequential fan-out with aggregated failures;
|
||||
- cleanup of outputs written during failed local publish attempts where practical;
|
||||
- deterministic sequential fan-out with aggregated destination failures;
|
||||
- cleanup of outputs written during failed publish attempts where practical;
|
||||
- no-op notification hook after successful publish or replacement;
|
||||
- current user, operator, internal, integration, and development documentation for implemented behavior.
|
||||
- current user, operator, internal, integration, and development documentation
|
||||
for implemented behavior.
|
||||
|
||||
The local MVP intentionally does not include executable SSH/SFTP backends, executable S3-compatible backends, force overwrite behavior, external notification adapters, warning-only digest mismatch behavior, or broad recursive destination deletion.
|
||||
## Active Roadmap
|
||||
|
||||
## Active Roadmap Stages
|
||||
There are no active implementation items in this roadmap.
|
||||
|
||||
Implement each stage independently. Unless a stage explicitly says otherwise:
|
||||
|
||||
1. read `docs/policy/architecture.md`, `docs/policy/documentation.md`, `docs/policy/development.md`, and this roadmap before editing;
|
||||
2. preserve current local MVP behavior;
|
||||
3. keep user-facing docs limited to implemented behavior;
|
||||
4. add or update focused tests for the behavior changed;
|
||||
5. run the relevant package tests and `go test ./...` for cross-package changes;
|
||||
6. avoid implementing later stages early.
|
||||
|
||||
## Stage 1: SSH/SFTP Backend
|
||||
|
||||
### Goal
|
||||
|
||||
Implement native SSH/SFTP storage backend support for sources and destinations through the existing storage interface and app-level backend factory.
|
||||
|
||||
### Implementation Scope
|
||||
|
||||
Add an SSH/SFTP adapter package under `internal/adapters/ssh`.
|
||||
|
||||
The backend must implement the current `internal/storage.Backend` contract:
|
||||
|
||||
- `ReadFile` and `OpenReader`;
|
||||
- `WriteFile` and `WriteFrom`;
|
||||
- `Stat`;
|
||||
- `Walk`;
|
||||
- `HasAny`;
|
||||
- `DeleteManagedBundle`.
|
||||
|
||||
Use native SFTP operations rather than shelling out to `ssh`, `scp`, or `rsync`.
|
||||
|
||||
Use Go SSH/SFTP libraries behind the adapter boundary:
|
||||
|
||||
- `golang.org/x/crypto/ssh`;
|
||||
- `golang.org/x/crypto/ssh/agent`;
|
||||
- `golang.org/x/crypto/ssh/knownhosts`;
|
||||
- `github.com/pkg/sftp`.
|
||||
|
||||
Authentication behavior:
|
||||
|
||||
- prefer SSH agent by default;
|
||||
- support `ssh_key_file` from the start and use it after SSH agent auth;
|
||||
- do not support password authentication in YAML in this stage;
|
||||
- support host key policies `strict`, `true`, `accept-new`, `off`, and `false`;
|
||||
- accept both YAML booleans and strings for `host_key_policy`;
|
||||
- normalize `true`, `"true"`, and `"strict"` to `strict`;
|
||||
- normalize `false`, `"false"`, and `"off"` to `off`;
|
||||
- normalize `"accept-new"` to `accept-new`;
|
||||
- default host key policy to `accept-new`;
|
||||
- support an optional `known_hosts` path, defaulting to the service user's OpenSSH known-hosts file where practical;
|
||||
- treat changed host keys as fatal for both `strict` and `accept-new`;
|
||||
- allow `off` and `false` only as explicit insecure modes, and surface that insecurity in docs and operation output where practical;
|
||||
- persist newly accepted host keys for `accept-new` when a writable known-hosts path is available;
|
||||
- fail clearly if `accept-new` needs to persist a new host key but no writable known-hosts path is available;
|
||||
- do not create a missing parent `.ssh` directory automatically in this stage.
|
||||
|
||||
Config execution behavior:
|
||||
|
||||
- switch SSH execution to structured config fields instead of URI-based config:
|
||||
|
||||
```yaml
|
||||
backend: ssh
|
||||
host: example.com
|
||||
user: distributor
|
||||
port: 2222
|
||||
path: /remote/root
|
||||
ssh_key_file: /home/distributor/.ssh/id_ed25519
|
||||
known_hosts: /home/distributor/.ssh/known_hosts
|
||||
host_key_policy: accept-new
|
||||
```
|
||||
|
||||
- require `host` and `path`;
|
||||
- make `user` optional, defaulting to the current OS user where available;
|
||||
- fail clearly at validation or backend-open time if `user` is omitted and the current OS user cannot be determined;
|
||||
- make `port` optional, defaulting to `22`;
|
||||
- make `ssh_key_file`, `known_hosts`, and `host_key_policy` optional;
|
||||
- do not keep `uri` as an SSH execution field unless a later compatibility stage explicitly reintroduces it;
|
||||
- keep secrets out of config files;
|
||||
- keep config loading and validation centralized in `internal/config`;
|
||||
- wire runtime construction through app-level backend factory and storage registry patterns.
|
||||
|
||||
Supported pipeline combinations:
|
||||
|
||||
- local source to SSH destination;
|
||||
- SSH source to local destination;
|
||||
- SSH source to SSH destination where feasible through streaming or backend-owned staging.
|
||||
|
||||
Safety requirements:
|
||||
|
||||
- enforce the same backend-rooted logical path rules as local storage;
|
||||
- reject path traversal, absolute logical paths, dot segments, and backslashes;
|
||||
- report or reject symlinks according to storage and bundle validation rules;
|
||||
- keep deletion limited to managed output paths and `.distributor.json`;
|
||||
- never delete a configured backend root;
|
||||
- preserve conservative non-force conflict behavior.
|
||||
|
||||
### Documentation Updates
|
||||
|
||||
After implementation, update only current-behavior docs:
|
||||
|
||||
- `docs/config.md`: mark SSH as executable and document any implemented SSH-only fields.
|
||||
- `docs/operations.md`: add SSH source/destination operating notes and recovery boundaries.
|
||||
- `docs/troubleshooting.md`: add common SSH authentication, known-hosts, and remote path failures.
|
||||
- `docs/internal/storage.md`: add implemented SSH adapter behavior and tests.
|
||||
- `docs/policy/development.md`: update backend addition guidance if implementation changes the pattern.
|
||||
- `examples/`: add only runnable or clearly environment-gated SSH examples.
|
||||
|
||||
Do not document S3 or force overwrite as implemented in this stage.
|
||||
|
||||
### Tests
|
||||
|
||||
Add unit tests for:
|
||||
|
||||
- SSH config execution wiring;
|
||||
- structured SSH config validation and defaulting;
|
||||
- host, user, port, remote root path, key file, known-hosts path, and host key policy handling;
|
||||
- logical path validation;
|
||||
- auth fallback order;
|
||||
- host key policy behavior through test doubles or isolated known-hosts files;
|
||||
- storage error translation where practical;
|
||||
- `Walk` and `HasAny` behavior through test doubles or controlled fixtures;
|
||||
- managed deletion boundaries;
|
||||
- app-level local-to-SSH and SSH-to-local planning or wiring using fakes/mocks where possible.
|
||||
|
||||
Add integration tests only if they are skipped unless explicit SSH test endpoint environment variables are configured. Use these environment variable names:
|
||||
|
||||
- `DISTRIBUTOR_TEST_SSH_HOST`;
|
||||
- `DISTRIBUTOR_TEST_SSH_USER`;
|
||||
- `DISTRIBUTOR_TEST_SSH_PORT`;
|
||||
- `DISTRIBUTOR_TEST_SSH_PATH`;
|
||||
- `DISTRIBUTOR_TEST_SSH_KEY_FILE`;
|
||||
- `DISTRIBUTOR_TEST_SSH_KNOWN_HOSTS`.
|
||||
|
||||
Normal `go test ./...` must not require a live SSH server.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- SSH/SFTP backend compiles and satisfies `storage.Backend`.
|
||||
- Runtime `run` can execute supported SSH source and destination flows.
|
||||
- Local MVP tests still pass.
|
||||
- Normal test runs do not require a live SSH server.
|
||||
- User docs accurately describe implemented SSH behavior and boundaries.
|
||||
|
||||
## Stage 2: Secrets Directory and Credential Environment Resolution
|
||||
|
||||
### Goal
|
||||
|
||||
Add a top-level secrets directory feature that lets deployments provide credential values as files while keeping config files free of literal secrets and without mutating the process environment.
|
||||
|
||||
This stage exists before S3 because S3 credentials are the first planned backend credentials that need environment-variable resolution at runtime.
|
||||
|
||||
### Implementation Scope
|
||||
|
||||
Add top-level config for a secrets directory:
|
||||
|
||||
```yaml
|
||||
secrets:
|
||||
directory: /run/secrets/distributor
|
||||
```
|
||||
|
||||
If `secrets.directory` is omitted, behavior must remain unchanged.
|
||||
|
||||
If `secrets.directory` is configured, load it before backend construction and credential resolution. If the configured directory is missing or unreadable, fail clearly before opening source or destination backends.
|
||||
|
||||
Secrets directory behavior:
|
||||
|
||||
- each valid filename becomes an internal environment key;
|
||||
- file contents become the corresponding internal environment value;
|
||||
- trim exactly one trailing `\n` or `\r\n`;
|
||||
- preserve all other whitespace;
|
||||
- valid filenames must match `[A-Za-z_][A-Za-z0-9_]*`;
|
||||
- ignore subdirectories;
|
||||
- follow symlinks to regular files;
|
||||
- ignore real directories and symlinks that resolve to directories;
|
||||
- do not enforce file owner, group, or mode permission policy;
|
||||
- never log or print secret values.
|
||||
|
||||
Conflict behavior:
|
||||
|
||||
- existing process environment values take precedence over secrets-directory values;
|
||||
- if the process environment value equals the loaded secret file value, emit no warning;
|
||||
- if the values differ, emit a clear warning during `run` output that the secret file was ignored because the real environment already has that variable;
|
||||
- the warning must include the variable name but neither value.
|
||||
|
||||
Resolver behavior:
|
||||
|
||||
- do not mutate `os.Environ`;
|
||||
- implement an internal resolver that checks the real process environment first and then secrets-directory values;
|
||||
- use the resolver for explicit credential environment variable references;
|
||||
- SDK default credential chains continue to see only the real process environment unless the variable exists there independently.
|
||||
|
||||
Per-pipeline and per-destination credential behavior should continue to come from backend config. No separate `pipelines[].env` map is needed in this stage because backend credential environment variable names are already configured per source or destination backend.
|
||||
|
||||
### Documentation Updates
|
||||
|
||||
After implementation, update only current-behavior docs:
|
||||
|
||||
- `docs/config.md`: document `secrets.directory` and credential environment resolution.
|
||||
- `docs/operations.md`: document deployment patterns for secrets directories and conflict warnings.
|
||||
- `docs/troubleshooting.md`: add missing directory, unreadable directory, invalid filename, missing referenced credential variable, and conflict-warning entries.
|
||||
- `docs/internal/config.md`: document config loading, secrets loading, and resolver behavior.
|
||||
- `docs/policy/development.md`: document how future credential-consuming features should use the resolver instead of reading `os.Getenv` directly.
|
||||
|
||||
User docs should describe `secrets.directory` as credential support, not as a general templating or shell environment feature.
|
||||
|
||||
### Tests
|
||||
|
||||
Add config tests for:
|
||||
|
||||
- omitted `secrets`;
|
||||
- valid `secrets.directory`;
|
||||
- unknown nested secrets fields.
|
||||
|
||||
Add resolver tests for:
|
||||
|
||||
- loading valid files;
|
||||
- trimming exactly one trailing newline or CRLF;
|
||||
- preserving other whitespace;
|
||||
- rejecting invalid filenames;
|
||||
- ignoring directories;
|
||||
- following symlinks;
|
||||
- missing configured directory failure;
|
||||
- existing process environment values taking precedence;
|
||||
- same process-environment and secret value producing no warning;
|
||||
- different process-environment and secret value producing a warning without either value;
|
||||
- secret values not appearing in errors, warnings, or logs.
|
||||
|
||||
Add integration-style credential resolution tests using fake credentials:
|
||||
|
||||
- explicit credential env names can be satisfied by secrets-directory files;
|
||||
- real process environment values take precedence over secrets-directory values;
|
||||
- default SDK credential chains are not fed by secrets-directory values unless those variables exist in the real process environment.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- `secrets.directory` config is accepted and validated.
|
||||
- Runtime `run` loads configured secrets before backend construction.
|
||||
- Explicit credential env references can resolve through the secrets-aware resolver.
|
||||
- Process environment is not mutated.
|
||||
- Warnings are emitted only for differing process-environment/secret conflicts.
|
||||
- Local MVP behavior remains unchanged when `secrets.directory` is omitted.
|
||||
|
||||
## Stage 3: S3-Compatible Backend
|
||||
|
||||
### Goal
|
||||
|
||||
Implement S3-compatible object storage backend support for sources and destinations through the existing storage interface and app-level backend factory.
|
||||
|
||||
### Implementation Scope
|
||||
|
||||
Add an S3-compatible adapter package under `internal/adapters/s3`.
|
||||
|
||||
This stage depends on Stage 2. Explicit S3 credential environment variable references must resolve through the secrets-aware resolver introduced there.
|
||||
|
||||
Use the AWS SDK for Go v2 behind the adapter boundary rather than hand-rolling S3 requests or signing:
|
||||
|
||||
- `github.com/aws/aws-sdk-go-v2/config`;
|
||||
- `github.com/aws/aws-sdk-go-v2/credentials`;
|
||||
- `github.com/aws/aws-sdk-go-v2/service/s3`;
|
||||
- add `github.com/aws/aws-sdk-go-v2/feature/s3/manager` only if multipart upload or download becomes necessary.
|
||||
|
||||
Do not implement a custom standard-library-only S3 client in this stage. The adapter should rely on the SDK for Signature V4, canonical request signing, endpoint behavior, retries where configured by the SDK, error decoding, pagination primitives, and credential-chain integration.
|
||||
|
||||
The backend must implement the current `internal/storage.Backend` contract:
|
||||
|
||||
- `ReadFile` and `OpenReader`;
|
||||
- `WriteFile` and `WriteFrom`;
|
||||
- `Stat`;
|
||||
- `Walk`;
|
||||
- `HasAny`;
|
||||
- `DeleteManagedBundle`.
|
||||
|
||||
Use the existing accepted config shape:
|
||||
|
||||
```yaml
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: some/prefix
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
```
|
||||
|
||||
Config semantics:
|
||||
|
||||
- require `endpoint` and `bucket`;
|
||||
- make `prefix` optional;
|
||||
- make `region` optional, defaulting to `us-east-1`;
|
||||
- make `force_path_style` optional, defaulting to `true` for S3-compatible storage;
|
||||
- distinguish omitted `force_path_style` from explicit `false` so users can disable path-style addressing;
|
||||
- normalize `prefix` as a slash-separated backend-root prefix;
|
||||
- allow an empty `prefix`;
|
||||
- trim leading and trailing slashes from `prefix`, then reject `.` or `..` segments, backslashes, and traversal;
|
||||
- do not add insecure TLS or TLS verification bypass config in this stage;
|
||||
- allow `http://` endpoints for local development or local S3-compatible test services when explicitly configured.
|
||||
|
||||
Credential behavior:
|
||||
|
||||
- if either `credentials.access_key_id_env` or `credentials.secret_access_key_env` is configured, require both fields;
|
||||
- when credential environment variable names are configured, require both referenced environment variables to be present and non-empty;
|
||||
- resolve explicit credential environment variable names through the secrets-aware resolver;
|
||||
- explicit environment-variable credentials take precedence over SDK credential discovery;
|
||||
- when explicit credential environment variable names are omitted, use the AWS SDK default credential chain;
|
||||
- do not feed secrets-directory values into the AWS SDK default credential chain unless those variables exist in the real process environment;
|
||||
- do not put literal secrets in YAML.
|
||||
|
||||
Object semantics:
|
||||
|
||||
- treat prefixes as object trees, not real directories;
|
||||
- normalize configured prefix plus logical path with exact path-boundary matching;
|
||||
- `Stat` must not synthesize directory metadata only because objects exist below a prefix;
|
||||
- `Walk` should use object-list pagination and should not load an entire prefix into memory;
|
||||
- `HasAny` should stop after the first matching object;
|
||||
- `DeleteManagedBundle` must delete only listed managed output objects plus `.distributor.json`;
|
||||
- ignore bucket versioning in this stage; normal deletes are sufficient and the adapter should not manage object versions or delete markers.
|
||||
|
||||
Write behavior:
|
||||
|
||||
- treat successful object PUT as publish-on-success;
|
||||
- set content type from `storage.WriteOptions` where available;
|
||||
- for `Overwrite: false`, perform `HeadObject` before `PutObject` and fail with already-exists if the object exists;
|
||||
- accept the small race between `HeadObject` and `PutObject` in this stage because `distributor` is not a multi-writer synchronization tool;
|
||||
- if `WriteFrom` has `SizeKnown`, stream with the known content length where the SDK allows;
|
||||
- if `WriteFrom` size is unknown, buffer or spool as needed; initial buffering is acceptable because current artifacts are expected to be small;
|
||||
- defer multipart upload unless large artifacts become a real requirement;
|
||||
- preserve overwrite checks and conservative conflict behavior.
|
||||
|
||||
Content type behavior should cover at least:
|
||||
|
||||
- `.md`: `text/markdown; charset=utf-8`;
|
||||
- `.html`: `text/html; charset=utf-8`;
|
||||
- `.json`: `application/json`;
|
||||
- `.txt`: `text/plain; charset=utf-8`;
|
||||
- unknown extensions: `application/octet-stream`.
|
||||
|
||||
Supported pipeline combinations:
|
||||
|
||||
- local source to S3 destination;
|
||||
- S3 source to local destination;
|
||||
- S3 source to S3 destination where feasible through streaming or backend-owned staging.
|
||||
|
||||
### Documentation Updates
|
||||
|
||||
After implementation, update only current-behavior docs:
|
||||
|
||||
- `docs/config.md`: mark S3 as executable and document implemented credential behavior.
|
||||
- `docs/operations.md`: add S3 source/destination layout, prefix, and recovery notes.
|
||||
- `docs/troubleshooting.md`: add common S3 credential, bucket, endpoint, prefix, and permission failures.
|
||||
- `docs/internal/storage.md`: add implemented S3 adapter behavior and tests.
|
||||
- `docs/policy/development.md`: update backend addition guidance if implementation changes the pattern.
|
||||
- `examples/`: add only safe S3 examples that use placeholder endpoints and environment variable names.
|
||||
|
||||
Do not document force overwrite or notification adapters as implemented in this stage.
|
||||
|
||||
### Tests
|
||||
|
||||
Add unit tests for:
|
||||
|
||||
- S3 config execution wiring;
|
||||
- default region and default path-style behavior;
|
||||
- explicit `force_path_style: false` behavior;
|
||||
- S3 prefix trimming and validation;
|
||||
- credential environment variable handling;
|
||||
- secrets-aware explicit credential resolution;
|
||||
- SDK credential-chain fallback when explicit credential environment variable names are omitted;
|
||||
- key and prefix normalization;
|
||||
- exact prefix boundary behavior;
|
||||
- exact-object `Stat` behavior that does not synthesize prefix directories;
|
||||
- path traversal rejection;
|
||||
- content type selection;
|
||||
- `Overwrite: false` `HeadObject` behavior;
|
||||
- paginated `Walk` behavior through mocks/fakes;
|
||||
- early-stop `HasAny`;
|
||||
- normal managed deletes that ignore bucket versioning;
|
||||
- managed deletion boundaries;
|
||||
- publish planning with S3 destination state fixtures.
|
||||
|
||||
Add integration tests only if they are skipped unless explicit S3-compatible endpoint credentials are configured. Use these environment variable names:
|
||||
|
||||
- `DISTRIBUTOR_TEST_S3_ENDPOINT`;
|
||||
- `DISTRIBUTOR_TEST_S3_BUCKET`;
|
||||
- `DISTRIBUTOR_TEST_S3_PREFIX`;
|
||||
- `DISTRIBUTOR_TEST_S3_REGION`;
|
||||
- `DISTRIBUTOR_TEST_S3_FORCE_PATH_STYLE`;
|
||||
- `DISTRIBUTOR_TEST_S3_ACCESS_KEY_ID`;
|
||||
- `DISTRIBUTOR_TEST_S3_SECRET_ACCESS_KEY`.
|
||||
|
||||
Normal `go test ./...` must not require live S3 credentials.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- S3-compatible backend compiles and satisfies `storage.Backend`.
|
||||
- Runtime `run` can execute supported S3 source and destination flows.
|
||||
- Local and SSH behavior, if implemented, remain unchanged.
|
||||
- Normal test runs do not require live S3.
|
||||
- User docs accurately describe implemented S3 behavior and boundaries.
|
||||
|
||||
## Stage 4: Cross-Backend Hardening and Documentation
|
||||
|
||||
### Goal
|
||||
|
||||
Harden behavior across implemented backend combinations, improve operator-facing failures, and synchronize current-behavior documentation and examples after remote backend support exists.
|
||||
|
||||
### Implementation Scope
|
||||
|
||||
Exercise and harden representative flows across all implemented backend types:
|
||||
|
||||
- local source to local archive destination;
|
||||
- local source to local HTML destination;
|
||||
- local source to multiple destinations with different publish policies;
|
||||
- local source to SSH destination, when SSH is implemented and test credentials exist;
|
||||
- SSH source to local destination, when SSH is implemented and test credentials exist;
|
||||
- local source to S3 destination, when S3 is implemented and test credentials exist;
|
||||
- S3 source to local destination, when S3 is implemented and test credentials exist.
|
||||
|
||||
Improve error context where practical for:
|
||||
|
||||
- invalid config;
|
||||
- invalid source manifest;
|
||||
- digest mismatch;
|
||||
- destination conflict;
|
||||
- unmanaged destination path;
|
||||
- backend read, write, stat, walk, and delete failures;
|
||||
- transform failures;
|
||||
- partial fan-out failures.
|
||||
|
||||
Ensure errors and logs identify pipeline id, destination id, bundle path or id, backend type, and logical path where useful without exposing secrets.
|
||||
|
||||
Do not add force overwrite behavior in this stage.
|
||||
|
||||
### Documentation Updates
|
||||
|
||||
Update current-behavior docs after hardening:
|
||||
|
||||
- `README.md`: keep the quickstart local unless remote examples become safe and concise.
|
||||
- `docs/cli.md`: document any changed output or diagnostics.
|
||||
- `docs/config.md`: ensure backend support status and config reference match implementation.
|
||||
- `docs/operations.md`: document cross-backend state layout, retry behavior, and recovery caveats.
|
||||
- `docs/troubleshooting.md`: add recurring SSH/S3 failure modes discovered during hardening.
|
||||
- `docs/internal/`: update storage, publish, app, and config internals where behavior changed.
|
||||
- `examples/`: keep examples copyable and free of secrets; remote examples must rely on placeholders and environment variables.
|
||||
|
||||
### Tests
|
||||
|
||||
Add or expand tests for:
|
||||
|
||||
- dry-run across multiple destinations and backend types;
|
||||
- partial failure behavior;
|
||||
- repeated run idempotency;
|
||||
- older/newer destination state behavior across backends;
|
||||
- destination state output metadata accuracy;
|
||||
- generated HTML output metadata accuracy;
|
||||
- destructive replacement safety across implemented backends;
|
||||
- error context for common failures.
|
||||
|
||||
Integration tests for SSH or S3 must remain opt-in through environment variables.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Implemented backend combinations behave consistently through the common pipeline path.
|
||||
- Repeated runs are idempotent.
|
||||
- Destructive paths remain bounded to managed destination bundle paths.
|
||||
- Operator-facing errors are actionable.
|
||||
- Current-behavior docs and examples match implemented backend support.
|
||||
|
||||
## Stage 5: Explicit Force Overwrite
|
||||
|
||||
### Goal
|
||||
|
||||
Introduce explicit operator-requested force behavior for controlled overwrite cases that remain intentionally unsupported by default.
|
||||
|
||||
### Implementation Scope
|
||||
|
||||
Add a CLI-only force option:
|
||||
|
||||
```bash
|
||||
distributor run --config config.yml --force
|
||||
```
|
||||
|
||||
Force must be explicit per run. Do not add a persistent config default for force behavior.
|
||||
|
||||
Define and implement force planning for:
|
||||
|
||||
- unmanaged non-empty destination paths;
|
||||
- destination state with a different source id;
|
||||
- destination state with matching source id and matching `created` timestamp but different digest;
|
||||
- destination state with mismatched `pipeline_id` or `destination_id`;
|
||||
- destination newer than source when transfer policy explicitly allows replacement.
|
||||
|
||||
Once force behavior exists, update transfer policy validation only for values supported by implemented force behavior:
|
||||
|
||||
- `on_destination_newer: replace`;
|
||||
- `on_conflict: replace`.
|
||||
|
||||
Safety requirements:
|
||||
|
||||
- non-force behavior remains unchanged and conservative;
|
||||
- dry-run must show destructive force actions before any forced run;
|
||||
- force must never delete above the resolved destination bundle path or configured destination prefix;
|
||||
- local replacement should remain staged where practical;
|
||||
- S3 replacement must remain constrained to the destination bundle prefix;
|
||||
- managed state should still be written only after successful output writes;
|
||||
- logs and output must clearly mark force decisions.
|
||||
|
||||
### Documentation Updates
|
||||
|
||||
After implementation, update:
|
||||
|
||||
- `docs/cli.md`: document `--force` syntax and dry-run workflow.
|
||||
- `docs/config.md`: document newly accepted transfer policy values and note force is CLI-only.
|
||||
- `docs/operations.md`: document safe force workflow and recovery boundaries.
|
||||
- `docs/troubleshooting.md`: describe when force may be appropriate and when it remains unsafe.
|
||||
- `docs/internal/publish.md` and `docs/internal/state.md`: document force planning and comparison handling.
|
||||
|
||||
Do not document force as a default or config-only behavior.
|
||||
|
||||
### Tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- force rejected or unavailable when the flag is absent;
|
||||
- unmanaged non-empty destination overwritten only with force;
|
||||
- different source id overwritten only with force and allowed policy;
|
||||
- same id and created timestamp with different digest overwritten only with force and allowed policy;
|
||||
- destination newer replaced only with force and allowed policy;
|
||||
- pipeline or destination id mismatch overwritten only with force and allowed policy;
|
||||
- dry-run reports destructive force actions without writing;
|
||||
- force deletes only bounded destination bundle paths;
|
||||
- local, SSH, and S3 backends, where implemented, preserve deletion boundaries.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Force overwrite behavior is explicit, logged, dry-runnable, and test-covered.
|
||||
- Default non-force behavior remains unchanged.
|
||||
- User docs clearly describe force risks and safe workflow.
|
||||
|
||||
## Stage 6: Release Readiness
|
||||
|
||||
### Goal
|
||||
|
||||
Perform a final quality pass before treating `distributor` as ready for routine use against real producer pipelines and implemented destination backends.
|
||||
|
||||
### Implementation Scope
|
||||
|
||||
Review:
|
||||
|
||||
- package boundaries against `docs/policy/architecture.md`;
|
||||
- contributor workflow against `docs/policy/development.md`;
|
||||
- user docs against `docs/policy/documentation.md`;
|
||||
- CLI UX and command output;
|
||||
- config validation and examples;
|
||||
- manifest and state compatibility;
|
||||
- destructive operation safety;
|
||||
- backend error handling;
|
||||
- logging and diagnostics for unattended operation;
|
||||
- test coverage for core invariants.
|
||||
|
||||
Do not add new product features in this stage.
|
||||
|
||||
### Documentation Updates
|
||||
|
||||
Update current-behavior docs only for issues found during the readiness review.
|
||||
|
||||
If release packaging, version injection, or installation workflow is added, document it in the appropriate current-behavior user or development docs.
|
||||
|
||||
### Tests
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Also verify representative CLI examples that are documented as runnable.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- A dry-run can be performed safely against real configured sources and destinations.
|
||||
- Repeated runs are idempotent.
|
||||
- Destructive replacement cannot occur outside managed destination bundle paths.
|
||||
- Current docs accurately reflect the application.
|
||||
- The project is ready to deploy against one real producer pipeline.
|
||||
Before implementing new product behavior, add a focused roadmap entry when the
|
||||
work changes storage semantics, config, CLI behavior, state schema, transform
|
||||
behavior, notification behavior, operational safety, or user-visible workflows.
|
||||
Keep those entries out of current-behavior docs until the behavior exists.
|
||||
|
||||
## Deferred Work
|
||||
|
||||
The following work remains intentionally deferred unless a future roadmap promotes it:
|
||||
These items are not implemented and should stay out of current-behavior docs
|
||||
until a roadmap entry is selected and implemented:
|
||||
|
||||
- external notification adapters such as email, ntfy, Gotify, or Pushover;
|
||||
- RSS or Atom feed generation;
|
||||
- static site index pages beyond sidecar HTML output;
|
||||
- destination path remapping rules;
|
||||
- HTML themes beyond the minimal deterministic template;
|
||||
- optional HTML sanitization with `bluemonday` or equivalent, implemented as an HTML post-processing step in the transform layer if richer or less-trusted HTML output is later supported;
|
||||
- full plugin architecture;
|
||||
- web UI;
|
||||
- report editing;
|
||||
- producer pipeline execution;
|
||||
- database-backed state;
|
||||
- complex retry queues;
|
||||
- concurrent publication workers;
|
||||
- symlink support;
|
||||
- external notification adapters;
|
||||
- warning-only digest mismatch handling;
|
||||
- password-based SSH authentication in YAML;
|
||||
- broad recursive or prefix deletion outside explicitly bounded force behavior.
|
||||
- additional auth mechanisms beyond the implemented SSH and S3 credential
|
||||
paths;
|
||||
- compatibility parsing for legacy SSH URI config;
|
||||
- broad recursive destination deletion outside managed bundle paths;
|
||||
- concurrent fan-out publishing;
|
||||
- resumable multipart S3 uploads;
|
||||
- cloud-provider-specific IAM integration docs;
|
||||
- repository-managed packaging, release, and deployment automation.
|
||||
|
||||
## Validation
|
||||
## Roadmap Maintenance
|
||||
|
||||
For roadmap-only edits:
|
||||
When adding future roadmap work:
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
git diff -- docs/roadmap
|
||||
rg -n "docs/roadmap/(packages|contracts|storage|config|documentation)\\.md" README.md docs examples
|
||||
rg -n "docs/roadmap/(packages|contracts|storage|config|documentation)\\.md" .
|
||||
rg -n "SSH|S3|--force|force overwrite|notification adapter|future|planned" README.md docs/*.md docs/internal docs/policy examples
|
||||
```
|
||||
|
||||
Also search `docs/roadmap` for old MVP stage headings and titles from deleted roadmap files. That check should return no matches.
|
||||
|
||||
The final SSH/S3/force/future-work search is not expected to return zero results. Review matches and confirm they are either under roadmap material or clearly marked as unsupported current behavior.
|
||||
|
||||
Go tests are not required for documentation-only roadmap rationalization unless examples, behavior docs, or code change.
|
||||
- describe user-visible behavior and safety boundaries;
|
||||
- define which existing docs must change after implementation;
|
||||
- keep examples secret-free and runnable or clearly environment-gated;
|
||||
- avoid workflow labels in production code, tests, config fields, and user
|
||||
documentation;
|
||||
- run focused tests for the changed behavior and `go test ./...` for
|
||||
cross-package changes.
|
||||
|
||||
Reference in New Issue
Block a user