Fixes and cleanup after implementation of the S3 and SSH roadmap
This commit is contained in:
@@ -38,7 +38,7 @@ All subcommands:
|
||||
`run` flags:
|
||||
|
||||
- `--config <path>`: config file to load. If omitted, `run` uses `/usr/local/etc/distributor/config.yml`.
|
||||
- `--dry-run`: load config, discover bundles, inspect destination state, print planned actions and final status, and do not write files.
|
||||
- `--dry-run`: load config, discover bundles, inspect destination state, print planned actions and final status, and do not write output files, destination state, or SSH `known_hosts` entries.
|
||||
- `--force`: allow explicit destructive replacement for supported conflict cases in this run only.
|
||||
|
||||
`run` does not accept positional arguments. `validate` and `inspect` accept at most one path; omitting the path returns a required-path error.
|
||||
|
||||
@@ -134,7 +134,7 @@ Authentication uses SSH agent identities first when `SSH_AUTH_SOCK` is set, then
|
||||
Host key policies:
|
||||
|
||||
- `strict`, `true`, and `"true"` require a matching known host key.
|
||||
- `accept-new` accepts and persists a new host key, but fails if an existing key changed.
|
||||
- `accept-new` accepts and persists a new host key, but fails if an existing key changed. During `run --dry-run`, new host keys are accepted only for the current connection and are not persisted.
|
||||
- `off`, `false`, and `"false"` disable host key checking and are insecure.
|
||||
|
||||
`accept-new` and `strict` use `known_hosts` when configured. If omitted, distributor uses the current service user's default OpenSSH `known_hosts` path where practical. `accept-new` fails when it needs to persist a new host key and no writable `known_hosts` path is available. It does not create a missing parent `.ssh` directory.
|
||||
|
||||
@@ -72,7 +72,7 @@ Do not edit `.distributor.json` by hand during normal operation. If it is missin
|
||||
|
||||
## Dry Runs
|
||||
|
||||
`--dry-run` loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files or destination state.
|
||||
`--dry-run` loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files, destination state, or SSH `known_hosts` entries.
|
||||
|
||||
Dry-run output is useful before publishing to confirm actions such as `publish_new`, `replace_older`, `force_replace`, `skip_same`, and `skip_destination_newer`.
|
||||
|
||||
@@ -117,7 +117,7 @@ SSH execution uses SFTP over `golang.org/x/crypto/ssh` and `github.com/pkg/sftp`
|
||||
|
||||
Configure `ssh_key_file`, an SSH agent, or both. Agent identities are attempted first, followed by the configured key file. YAML password authentication is not supported.
|
||||
|
||||
The default host key policy is `accept-new`. New host keys are written to `known_hosts` when the file path is writable. Changed host keys are fatal for both `strict` and `accept-new`. The `off` policy disables host key checking and `run` prints a warning when stdout is enabled.
|
||||
The default host key policy is `accept-new`. New host keys are written to `known_hosts` when the file path is writable. During `--dry-run`, unknown host keys may be accepted for the current connection but are not written to `known_hosts`; a later non-dry-run may persist the same key. Changed host keys are fatal for both `strict` and `accept-new`. The `off` policy disables host key checking and `run` prints a warning when stdout is enabled.
|
||||
|
||||
Recovery boundaries are the same as local storage: replacement deletes only managed output paths recorded in `.distributor.json` plus the state file, and failed writes are cleaned up where practical. Distributor never performs broad recursive remote deletion.
|
||||
|
||||
|
||||
@@ -24,41 +24,6 @@ it is implemented.
|
||||
`distributor` is ready for routine use against producer pipelines using the
|
||||
implemented local, SSH/SFTP, and S3-compatible backends.
|
||||
|
||||
Implemented capabilities:
|
||||
|
||||
- standard-library CLI commands for `version`, `run`, `validate`, and
|
||||
`inspect`;
|
||||
- YAML config loading, defaults, known-field rejection, and validation;
|
||||
- local, SSH/SFTP, and S3-compatible source and destination backends;
|
||||
- structured SSH/SFTP configuration with agent and key-file authentication,
|
||||
known-hosts handling, and backend-rooted path confinement;
|
||||
- S3-compatible configuration with endpoint, region, addressing mode, TLS
|
||||
controls, credential environment references, streaming reads and writes,
|
||||
listing, metadata, and managed deletion;
|
||||
- top-level `secrets.directory` loading for credential values, with process
|
||||
environment precedence and without mutating the process environment;
|
||||
- backend-rooted storage interfaces with typed errors, safe logical paths,
|
||||
traversal, collision checks, managed deletion, and explicit forced
|
||||
replacement;
|
||||
- source bundle discovery, manifest parsing, RFC3339 timestamp handling,
|
||||
duplicate path checks, path safety checks, symlink rejection, per-file digest
|
||||
validation, and bundle digest validation;
|
||||
- destination `.distributor.json` state parsing, validation, output metadata,
|
||||
and source comparison;
|
||||
- publication of source files, Markdown sidecar HTML, or both;
|
||||
- destination output collision detection before writes;
|
||||
- managed replacement for older destination state;
|
||||
- conservative conflict failures by default for unmanaged destination content
|
||||
and conflicting managed destination content;
|
||||
- explicit forced replacement with `run --force`, limited to the configured
|
||||
destination bundle path;
|
||||
- deterministic dry-run output and final run summaries;
|
||||
- deterministic sequential fan-out with aggregated destination failures;
|
||||
- cleanup of outputs written during failed publish attempts where practical;
|
||||
- no-op notification hook after successful publish or replacement;
|
||||
- current user, operator, internal, integration, and development documentation
|
||||
for implemented behavior.
|
||||
|
||||
## Active Roadmap
|
||||
|
||||
There are no active implementation items in this roadmap.
|
||||
@@ -80,7 +45,7 @@ until a roadmap entry is selected and implemented:
|
||||
- compatibility parsing for legacy SSH URI config;
|
||||
- broad recursive destination deletion outside managed bundle paths;
|
||||
- concurrent fan-out publishing;
|
||||
- resumable multipart S3 uploads;
|
||||
- streaming, resumable, or multipart S3 uploads;
|
||||
- cloud-provider-specific IAM integration docs;
|
||||
- repository-managed packaging, release, and deployment automation.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user