distributor/docs/roadmap/implementation.md

Post-Local-MVP Implementation Roadmap

This is the canonical active roadmap for distributor after the local MVP checkpoint.

The original MVP stages 1-8 are complete and are no longer listed as pending implementation work. Current behavior is documented outside the roadmap in README.md, docs/cli.md, docs/config.md, docs/operations.md, docs/troubleshooting.md, docs/internal/, docs/integrations/markdown.md, and docs/policy/development.md.

Future, planned, or aspirational behavior should remain under docs/roadmap/ until implemented.

Current Baseline

The implemented local MVP includes:

• standard-library CLI commands for version, run, validate, and inspect;
• YAML config loading, defaults, known-field rejection, and validation;
• accepted config shapes for local, ssh, and s3, with executable backend support currently limited to local;
• backend-rooted storage interface with typed errors, safe logical paths, traversal, HasAny, managed deletion, local backend, and fake backend;
• source bundle discovery, manifest parsing, RFC3339 timestamp handling, duplicate path checks, path safety checks, symlink rejection, per-file digest validation, and bundle digest validation;
• destination .distributor.json state parsing, validation, output metadata, and source comparison;
• local publication of source files, Markdown sidecar HTML, or both;
• destination output collision detection before writes;
• managed replacement for older destination state;
• unmanaged destination and conflict failures by default;
• deterministic dry-run output and final run summaries;
• deterministic sequential fan-out with aggregated failures;
• cleanup of outputs written during failed local publish attempts where practical;
• no-op notification hook after successful publish or replacement;
• current user, operator, internal, integration, and development documentation for implemented behavior.

The local MVP intentionally does not include executable SSH/SFTP backends, executable S3-compatible backends, force overwrite behavior, external notification adapters, warning-only digest mismatch behavior, or broad recursive destination deletion.

Active Roadmap Stages

Implement each stage independently. Unless a stage explicitly says otherwise:

1. read docs/policy/architecture.md, docs/policy/documentation.md, docs/policy/development.md, and this roadmap before editing;
2. preserve current local MVP behavior;
3. keep user-facing docs limited to implemented behavior;
4. add or update focused tests for the behavior changed;
5. run the relevant package tests and go test ./... for cross-package changes;
6. avoid implementing later stages early.

Stage 1: SSH/SFTP Backend

Goal

Implement native SSH/SFTP storage backend support for sources and destinations through the existing storage interface and app-level backend factory.

Implementation Scope

Add an SSH/SFTP adapter package under internal/adapters/ssh.

The backend must implement the current internal/storage.Backend contract:

• ReadFile and OpenReader;
• WriteFile and WriteFrom;
• Stat;
• Walk;
• HasAny;
• DeleteManagedBundle.

Use native SFTP operations rather than shelling out to ssh, scp, or rsync.

Use Go SSH/SFTP libraries behind the adapter boundary:

• golang.org/x/crypto/ssh;
• golang.org/x/crypto/ssh/agent;
• golang.org/x/crypto/ssh/knownhosts;
• github.com/pkg/sftp.

Authentication behavior:

• prefer SSH agent by default;
• support ssh_key_file from the start and use it after SSH agent auth;
• do not support password authentication in YAML in this stage;
• support host key policies strict, true, accept-new, off, and false;
• accept both YAML booleans and strings for host_key_policy;
• normalize true, "true", and "strict" to strict;
• normalize false, "false", and "off" to off;
• normalize "accept-new" to accept-new;
• default host key policy to accept-new;
• support an optional known_hosts path, defaulting to the service user's OpenSSH known-hosts file where practical;
• treat changed host keys as fatal for both strict and accept-new;
• allow off and false only as explicit insecure modes, and surface that insecurity in docs and operation output where practical;
• persist newly accepted host keys for accept-new when a writable known-hosts path is available;
• fail clearly if accept-new needs to persist a new host key but no writable known-hosts path is available;
• do not create a missing parent .ssh directory automatically in this stage.

Config execution behavior:

• switch SSH execution to structured config fields instead of URI-based config:

backend: ssh
host: example.com
user: distributor
port: 2222
path: /remote/root
ssh_key_file: /home/distributor/.ssh/id_ed25519
known_hosts: /home/distributor/.ssh/known_hosts
host_key_policy: accept-new

• require host and path;
• make user optional, defaulting to the current OS user where available;
• fail clearly at validation or backend-open time if user is omitted and the current OS user cannot be determined;
• make port optional, defaulting to 22;
• make ssh_key_file, known_hosts, and host_key_policy optional;
• do not keep uri as an SSH execution field unless a later compatibility stage explicitly reintroduces it;
• keep secrets out of config files;
• keep config loading and validation centralized in internal/config;
• wire runtime construction through app-level backend factory and storage registry patterns.

Supported pipeline combinations:

• local source to SSH destination;
• SSH source to local destination;
• SSH source to SSH destination where feasible through streaming or backend-owned staging.

Safety requirements:

• enforce the same backend-rooted logical path rules as local storage;
• reject path traversal, absolute logical paths, dot segments, and backslashes;
• report or reject symlinks according to storage and bundle validation rules;
• keep deletion limited to managed output paths and .distributor.json;
• never delete a configured backend root;
• preserve conservative non-force conflict behavior.

Documentation Updates

After implementation, update only current-behavior docs:

• docs/config.md: mark SSH as executable and document any implemented SSH-only fields.
• docs/operations.md: add SSH source/destination operating notes and recovery boundaries.
• docs/troubleshooting.md: add common SSH authentication, known-hosts, and remote path failures.
• docs/internal/storage.md: add implemented SSH adapter behavior and tests.
• docs/policy/development.md: update backend addition guidance if implementation changes the pattern.
• examples/: add only runnable or clearly environment-gated SSH examples.

Do not document S3 or force overwrite as implemented in this stage.

Tests

Add unit tests for:

• SSH config execution wiring;
• structured SSH config validation and defaulting;
• host, user, port, remote root path, key file, known-hosts path, and host key policy handling;
• logical path validation;
• auth fallback order;
• host key policy behavior through test doubles or isolated known-hosts files;
• storage error translation where practical;
• Walk and HasAny behavior through test doubles or controlled fixtures;
• managed deletion boundaries;
• app-level local-to-SSH and SSH-to-local planning or wiring using fakes/mocks where possible.

Add integration tests only if they are skipped unless explicit SSH test endpoint environment variables are configured. Use these environment variable names:

• DISTRIBUTOR_TEST_SSH_HOST;
• DISTRIBUTOR_TEST_SSH_USER;
• DISTRIBUTOR_TEST_SSH_PORT;
• DISTRIBUTOR_TEST_SSH_PATH;
• DISTRIBUTOR_TEST_SSH_KEY_FILE;
• DISTRIBUTOR_TEST_SSH_KNOWN_HOSTS.

Normal go test ./... must not require a live SSH server.

Completion Criteria

• SSH/SFTP backend compiles and satisfies storage.Backend.
• Runtime run can execute supported SSH source and destination flows.
• Local MVP tests still pass.
• Normal test runs do not require a live SSH server.
• User docs accurately describe implemented SSH behavior and boundaries.

Stage 2: Secrets Directory and Credential Environment Resolution

Goal

Add a top-level secrets directory feature that lets deployments provide credential values as files while keeping config files free of literal secrets and without mutating the process environment.

This stage exists before S3 because S3 credentials are the first planned backend credentials that need environment-variable resolution at runtime.

Implementation Scope

Add top-level config for a secrets directory:

secrets:
  directory: /run/secrets/distributor

If secrets.directory is omitted, behavior must remain unchanged.

If secrets.directory is configured, load it before backend construction and credential resolution. If the configured directory is missing or unreadable, fail clearly before opening source or destination backends.

Secrets directory behavior:

• each valid filename becomes an internal environment key;
• file contents become the corresponding internal environment value;
• trim exactly one trailing \n or \r\n;
• preserve all other whitespace;
• valid filenames must match [A-Za-z_][A-Za-z0-9_]*;
• ignore subdirectories;
• follow symlinks to regular files;
• ignore real directories and symlinks that resolve to directories;
• do not enforce file owner, group, or mode permission policy;
• never log or print secret values.

Conflict behavior:

• existing process environment values take precedence over secrets-directory values;
• if the process environment value equals the loaded secret file value, emit no warning;
• if the values differ, emit a clear warning during run output that the secret file was ignored because the real environment already has that variable;
• the warning must include the variable name but neither value.

Resolver behavior:

• do not mutate os.Environ;
• implement an internal resolver that checks the real process environment first and then secrets-directory values;
• use the resolver for explicit credential environment variable references;
• SDK default credential chains continue to see only the real process environment unless the variable exists there independently.

Per-pipeline and per-destination credential behavior should continue to come from backend config. No separate pipelines[].env map is needed in this stage because backend credential environment variable names are already configured per source or destination backend.

Documentation Updates

After implementation, update only current-behavior docs:

• docs/config.md: document secrets.directory and credential environment resolution.
• docs/operations.md: document deployment patterns for secrets directories and conflict warnings.
• docs/troubleshooting.md: add missing directory, unreadable directory, invalid filename, missing referenced credential variable, and conflict-warning entries.
• docs/internal/config.md: document config loading, secrets loading, and resolver behavior.
• docs/policy/development.md: document how future credential-consuming features should use the resolver instead of reading os.Getenv directly.

User docs should describe secrets.directory as credential support, not as a general templating or shell environment feature.

Tests

Add config tests for:

• omitted secrets;
• valid secrets.directory;
• unknown nested secrets fields.

Add resolver tests for:

• loading valid files;
• trimming exactly one trailing newline or CRLF;
• preserving other whitespace;
• rejecting invalid filenames;
• ignoring directories;
• following symlinks;
• missing configured directory failure;
• existing process environment values taking precedence;
• same process-environment and secret value producing no warning;
• different process-environment and secret value producing a warning without either value;
• secret values not appearing in errors, warnings, or logs.

Add integration-style credential resolution tests using fake credentials:

• explicit credential env names can be satisfied by secrets-directory files;
• real process environment values take precedence over secrets-directory values;
• default SDK credential chains are not fed by secrets-directory values unless those variables exist in the real process environment.

Completion Criteria

• secrets.directory config is accepted and validated.
• Runtime run loads configured secrets before backend construction.
• Explicit credential env references can resolve through the secrets-aware resolver.
• Process environment is not mutated.
• Warnings are emitted only for differing process-environment/secret conflicts.
• Local MVP behavior remains unchanged when secrets.directory is omitted.

Stage 3: S3-Compatible Backend

Goal

Implement S3-compatible object storage backend support for sources and destinations through the existing storage interface and app-level backend factory.

Implementation Scope

Add an S3-compatible adapter package under internal/adapters/s3.

This stage depends on Stage 2. Explicit S3 credential environment variable references must resolve through the secrets-aware resolver introduced there.

Use the AWS SDK for Go v2 behind the adapter boundary rather than hand-rolling S3 requests or signing:

• github.com/aws/aws-sdk-go-v2/config;
• github.com/aws/aws-sdk-go-v2/credentials;
• github.com/aws/aws-sdk-go-v2/service/s3;
• add github.com/aws/aws-sdk-go-v2/feature/s3/manager only if multipart upload or download becomes necessary.

Do not implement a custom standard-library-only S3 client in this stage. The adapter should rely on the SDK for Signature V4, canonical request signing, endpoint behavior, retries where configured by the SDK, error decoding, pagination primitives, and credential-chain integration.

The backend must implement the current internal/storage.Backend contract:

• ReadFile and OpenReader;
• WriteFile and WriteFrom;
• Stat;
• Walk;
• HasAny;
• DeleteManagedBundle.

Use the existing accepted config shape:

backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: some/prefix
region: us-east-1
force_path_style: true
credentials:
  access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
  secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY

Config semantics:

• require endpoint and bucket;
• make prefix optional;
• make region optional, defaulting to us-east-1;
• make force_path_style optional, defaulting to true for S3-compatible storage;
• distinguish omitted force_path_style from explicit false so users can disable path-style addressing;
• normalize prefix as a slash-separated backend-root prefix;
• allow an empty prefix;
• trim leading and trailing slashes from prefix, then reject . or .. segments, backslashes, and traversal;
• do not add insecure TLS or TLS verification bypass config in this stage;
• allow http:// endpoints for local development or local S3-compatible test services when explicitly configured.

Credential behavior:

• if either credentials.access_key_id_env or credentials.secret_access_key_env is configured, require both fields;
• when credential environment variable names are configured, require both referenced environment variables to be present and non-empty;
• resolve explicit credential environment variable names through the secrets-aware resolver;
• explicit environment-variable credentials take precedence over SDK credential discovery;
• when explicit credential environment variable names are omitted, use the AWS SDK default credential chain;
• do not feed secrets-directory values into the AWS SDK default credential chain unless those variables exist in the real process environment;
• do not put literal secrets in YAML.

Object semantics:

• treat prefixes as object trees, not real directories;
• normalize configured prefix plus logical path with exact path-boundary matching;
• Stat must not synthesize directory metadata only because objects exist below a prefix;
• Walk should use object-list pagination and should not load an entire prefix into memory;
• HasAny should stop after the first matching object;
• DeleteManagedBundle must delete only listed managed output objects plus .distributor.json;
• ignore bucket versioning in this stage; normal deletes are sufficient and the adapter should not manage object versions or delete markers.

Write behavior:

• treat successful object PUT as publish-on-success;
• set content type from storage.WriteOptions where available;
• for Overwrite: false, perform HeadObject before PutObject and fail with already-exists if the object exists;
• accept the small race between HeadObject and PutObject in this stage because distributor is not a multi-writer synchronization tool;
• if WriteFrom has SizeKnown, stream with the known content length where the SDK allows;
• if WriteFrom size is unknown, buffer or spool as needed; initial buffering is acceptable because current artifacts are expected to be small;
• defer multipart upload unless large artifacts become a real requirement;
• preserve overwrite checks and conservative conflict behavior.

Content type behavior should cover at least:

• .md: text/markdown; charset=utf-8;
• .html: text/html; charset=utf-8;
• .json: application/json;
• .txt: text/plain; charset=utf-8;
• unknown extensions: application/octet-stream.

Supported pipeline combinations:

• local source to S3 destination;
• S3 source to local destination;
• S3 source to S3 destination where feasible through streaming or backend-owned staging.

Documentation Updates

After implementation, update only current-behavior docs:

• docs/config.md: mark S3 as executable and document implemented credential behavior.
• docs/operations.md: add S3 source/destination layout, prefix, and recovery notes.
• docs/troubleshooting.md: add common S3 credential, bucket, endpoint, prefix, and permission failures.
• docs/internal/storage.md: add implemented S3 adapter behavior and tests.
• docs/policy/development.md: update backend addition guidance if implementation changes the pattern.
• examples/: add only safe S3 examples that use placeholder endpoints and environment variable names.

Do not document force overwrite or notification adapters as implemented in this stage.

Tests

Add unit tests for:

• S3 config execution wiring;
• default region and default path-style behavior;
• explicit force_path_style: false behavior;
• S3 prefix trimming and validation;
• credential environment variable handling;
• secrets-aware explicit credential resolution;
• SDK credential-chain fallback when explicit credential environment variable names are omitted;
• key and prefix normalization;
• exact prefix boundary behavior;
• exact-object Stat behavior that does not synthesize prefix directories;
• path traversal rejection;
• content type selection;
• Overwrite: false HeadObject behavior;
• paginated Walk behavior through mocks/fakes;
• early-stop HasAny;
• normal managed deletes that ignore bucket versioning;
• managed deletion boundaries;
• publish planning with S3 destination state fixtures.

Add integration tests only if they are skipped unless explicit S3-compatible endpoint credentials are configured. Use these environment variable names:

• DISTRIBUTOR_TEST_S3_ENDPOINT;
• DISTRIBUTOR_TEST_S3_BUCKET;
• DISTRIBUTOR_TEST_S3_PREFIX;
• DISTRIBUTOR_TEST_S3_REGION;
• DISTRIBUTOR_TEST_S3_FORCE_PATH_STYLE;
• DISTRIBUTOR_TEST_S3_ACCESS_KEY_ID;
• DISTRIBUTOR_TEST_S3_SECRET_ACCESS_KEY.

Normal go test ./... must not require live S3 credentials.

Completion Criteria

• S3-compatible backend compiles and satisfies storage.Backend.
• Runtime run can execute supported S3 source and destination flows.
• Local and SSH behavior, if implemented, remain unchanged.
• Normal test runs do not require live S3.
• User docs accurately describe implemented S3 behavior and boundaries.

Stage 4: Cross-Backend Hardening and Documentation

Goal

Harden behavior across implemented backend combinations, improve operator-facing failures, and synchronize current-behavior documentation and examples after remote backend support exists.

Implementation Scope

Exercise and harden representative flows across all implemented backend types:

• local source to local archive destination;
• local source to local HTML destination;
• local source to multiple destinations with different publish policies;
• local source to SSH destination, when SSH is implemented and test credentials exist;
• SSH source to local destination, when SSH is implemented and test credentials exist;
• local source to S3 destination, when S3 is implemented and test credentials exist;
• S3 source to local destination, when S3 is implemented and test credentials exist.

Improve error context where practical for:

• invalid config;
• invalid source manifest;
• digest mismatch;
• destination conflict;
• unmanaged destination path;
• backend read, write, stat, walk, and delete failures;
• transform failures;
• partial fan-out failures.

Ensure errors and logs identify pipeline id, destination id, bundle path or id, backend type, and logical path where useful without exposing secrets.

Do not add force overwrite behavior in this stage.

Documentation Updates

Update current-behavior docs after hardening:

• README.md: keep the quickstart local unless remote examples become safe and concise.
• docs/cli.md: document any changed output or diagnostics.
• docs/config.md: ensure backend support status and config reference match implementation.
• docs/operations.md: document cross-backend state layout, retry behavior, and recovery caveats.
• docs/troubleshooting.md: add recurring SSH/S3 failure modes discovered during hardening.
• docs/internal/: update storage, publish, app, and config internals where behavior changed.
• examples/: keep examples copyable and free of secrets; remote examples must rely on placeholders and environment variables.

Tests

Add or expand tests for:

• dry-run across multiple destinations and backend types;
• partial failure behavior;
• repeated run idempotency;
• older/newer destination state behavior across backends;
• destination state output metadata accuracy;
• generated HTML output metadata accuracy;
• destructive replacement safety across implemented backends;
• error context for common failures.

Integration tests for SSH or S3 must remain opt-in through environment variables.

Completion Criteria

• Implemented backend combinations behave consistently through the common pipeline path.
• Repeated runs are idempotent.
• Destructive paths remain bounded to managed destination bundle paths.
• Operator-facing errors are actionable.
• Current-behavior docs and examples match implemented backend support.

Stage 5: Explicit Force Overwrite

Goal

Introduce explicit operator-requested force behavior for controlled overwrite cases that remain intentionally unsupported by default.

Implementation Scope

Add a CLI-only force option:

distributor run --config config.yml --force

Force must be explicit per run. Do not add a persistent config default for force behavior.

Define and implement force planning for:

• unmanaged non-empty destination paths;
• destination state with a different source id;
• destination state with matching source id and matching created timestamp but different digest;
• destination state with mismatched pipeline_id or destination_id;
• destination newer than source when transfer policy explicitly allows replacement.

Once force behavior exists, update transfer policy validation only for values supported by implemented force behavior:

• on_destination_newer: replace;
• on_conflict: replace.

Safety requirements:

• non-force behavior remains unchanged and conservative;
• dry-run must show destructive force actions before any forced run;
• force must never delete above the resolved destination bundle path or configured destination prefix;
• local replacement should remain staged where practical;
• S3 replacement must remain constrained to the destination bundle prefix;
• managed state should still be written only after successful output writes;
• logs and output must clearly mark force decisions.

Documentation Updates

After implementation, update:

• docs/cli.md: document --force syntax and dry-run workflow.
• docs/config.md: document newly accepted transfer policy values and note force is CLI-only.
• docs/operations.md: document safe force workflow and recovery boundaries.
• docs/troubleshooting.md: describe when force may be appropriate and when it remains unsafe.
• docs/internal/publish.md and docs/internal/state.md: document force planning and comparison handling.

Do not document force as a default or config-only behavior.

Tests

Add tests for:

• force rejected or unavailable when the flag is absent;
• unmanaged non-empty destination overwritten only with force;
• different source id overwritten only with force and allowed policy;
• same id and created timestamp with different digest overwritten only with force and allowed policy;
• destination newer replaced only with force and allowed policy;
• pipeline or destination id mismatch overwritten only with force and allowed policy;
• dry-run reports destructive force actions without writing;
• force deletes only bounded destination bundle paths;
• local, SSH, and S3 backends, where implemented, preserve deletion boundaries.

Completion Criteria

• Force overwrite behavior is explicit, logged, dry-runnable, and test-covered.
• Default non-force behavior remains unchanged.
• User docs clearly describe force risks and safe workflow.

Stage 6: Release Readiness

Goal

Perform a final quality pass before treating distributor as ready for routine use against real producer pipelines and implemented destination backends.

Implementation Scope

Review:

• package boundaries against docs/policy/architecture.md;
• contributor workflow against docs/policy/development.md;
• user docs against docs/policy/documentation.md;
• CLI UX and command output;
• config validation and examples;
• manifest and state compatibility;
• destructive operation safety;
• backend error handling;
• logging and diagnostics for unattended operation;
• test coverage for core invariants.

Do not add new product features in this stage.

Documentation Updates

Update current-behavior docs only for issues found during the readiness review.

If release packaging, version injection, or installation workflow is added, document it in the appropriate current-behavior user or development docs.

Tests

Run:

go test ./...

Also verify representative CLI examples that are documented as runnable.

Completion Criteria

• A dry-run can be performed safely against real configured sources and destinations.
• Repeated runs are idempotent.
• Destructive replacement cannot occur outside managed destination bundle paths.
• Current docs accurately reflect the application.
• The project is ready to deploy against one real producer pipeline.

Deferred Work

The following work remains intentionally deferred unless a future roadmap promotes it:

• external notification adapters such as email, ntfy, Gotify, or Pushover;
• RSS or Atom feed generation;
• static site index pages beyond sidecar HTML output;
• destination path remapping rules;
• HTML themes beyond the minimal deterministic template;
• optional HTML sanitization with bluemonday or equivalent, implemented as an HTML post-processing step in the transform layer if richer or less-trusted HTML output is later supported;
• full plugin architecture;
• web UI;
• report editing;
• producer pipeline execution;
• database-backed state;
• complex retry queues;
• concurrent publication workers;
• symlink support;
• warning-only digest mismatch handling;
• password-based SSH authentication in YAML;
• broad recursive or prefix deletion outside explicitly bounded force behavior.

Validation

For roadmap-only edits:

git status --short
git diff -- docs/roadmap
rg -n "docs/roadmap/(packages|contracts|storage|config|documentation)\\.md" README.md docs examples
rg -n "docs/roadmap/(packages|contracts|storage|config|documentation)\\.md" .
rg -n "SSH|S3|--force|force overwrite|notification adapter|future|planned" README.md docs/*.md docs/internal docs/policy examples

Also search docs/roadmap for old MVP stage headings and titles from deleted roadmap files. That check should return no matches.

The final SSH/S3/force/future-work search is not expected to return zero results. Review matches and confirm they are either under roadmap material or clearly marked as unsupported current behavior.

Go tests are not required for documentation-only roadmap rationalization unless examples, behavior docs, or code change.