distributor/docs/roadmap/implementation.md

Post-Local-MVP Implementation Roadmap

This is the canonical active roadmap for distributor after the local MVP checkpoint.

The original MVP stages 1-8 are complete and are no longer listed as pending implementation work. Current behavior is documented outside the roadmap in README.md, docs/cli.md, docs/config.md, docs/operations.md, docs/troubleshooting.md, docs/internal/, docs/integrations/markdown.md, and docs/policy/development.md.

Future, planned, or aspirational behavior should remain under docs/roadmap/ until implemented.

Current Baseline

The implemented local MVP includes:

• standard-library CLI commands for version, run, validate, and inspect;
• YAML config loading, defaults, known-field rejection, and validation;
• accepted config shapes for local, ssh, and s3, with executable backend support currently limited to local;
• backend-rooted storage interface with typed errors, safe logical paths, traversal, HasAny, managed deletion, local backend, and fake backend;
• source bundle discovery, manifest parsing, RFC3339 timestamp handling, duplicate path checks, path safety checks, symlink rejection, per-file digest validation, and bundle digest validation;
• destination .distributor.json state parsing, validation, output metadata, and source comparison;
• local publication of source files, Markdown sidecar HTML, or both;
• destination output collision detection before writes;
• managed replacement for older destination state;
• unmanaged destination and conflict failures by default;
• deterministic dry-run output and final run summaries;
• deterministic sequential fan-out with aggregated failures;
• cleanup of outputs written during failed local publish attempts where practical;
• no-op notification hook after successful publish or replacement;
• current user, operator, internal, integration, and development documentation for implemented behavior.

The local MVP intentionally does not include executable SSH/SFTP backends, executable S3-compatible backends, force overwrite behavior, external notification adapters, warning-only digest mismatch behavior, or broad recursive destination deletion.

Active Roadmap Stages

Implement each stage independently. Unless a stage explicitly says otherwise:

1. read docs/policy/architecture.md, docs/policy/documentation.md, docs/policy/development.md, and this roadmap before editing;
2. preserve current local MVP behavior;
3. keep user-facing docs limited to implemented behavior;
4. add or update focused tests for the behavior changed;
5. run the relevant package tests and go test ./... for cross-package changes;
6. avoid implementing later stages early.

Stage 1: SSH/SFTP Backend

Goal

Implement native SSH/SFTP storage backend support for sources and destinations through the existing storage interface and app-level backend factory.

Implementation Scope

Add an SSH/SFTP adapter package under internal/adapters/ssh.

The backend must implement the current internal/storage.Backend contract:

• ReadFile and OpenReader;
• WriteFile and WriteFrom;
• Stat;
• Walk;
• HasAny;
• DeleteManagedBundle.

Use native SFTP operations rather than shelling out to ssh, scp, or rsync.

Authentication behavior:

• prefer SSH agent by default;
• use known_hosts validation by default where practical;
• support optional key-file configuration only if it can be added cleanly;
• do not support passwords in YAML in this stage.

Config execution behavior:

• use the existing accepted config shape:

backend: ssh
uri: ssh://user@example.com:22
path: /remote/root

• keep secrets out of config files;
• keep config loading and validation centralized in internal/config;
• wire runtime construction through app-level backend factory and storage registry patterns.

Supported pipeline combinations:

• local source to SSH destination;
• SSH source to local destination;
• SSH source to SSH destination where feasible through streaming or backend-owned staging.

Safety requirements:

• enforce the same backend-rooted logical path rules as local storage;
• reject path traversal, absolute logical paths, dot segments, and backslashes;
• report or reject symlinks according to storage and bundle validation rules;
• keep deletion limited to managed output paths and .distributor.json;
• never delete a configured backend root;
• preserve conservative non-force conflict behavior.

Documentation Updates

After implementation, update only current-behavior docs:

• docs/config.md: mark SSH as executable and document any implemented SSH-only fields.
• docs/operations.md: add SSH source/destination operating notes and recovery boundaries.
• docs/troubleshooting.md: add common SSH authentication, known-hosts, and remote path failures.
• docs/internal/storage.md: add implemented SSH adapter behavior and tests.
• docs/policy/development.md: update backend addition guidance if implementation changes the pattern.
• examples/: add only runnable or clearly environment-gated SSH examples.

Do not document S3 or force overwrite as implemented in this stage.

Tests

Add unit tests for:

• SSH config execution wiring;
• URI and path handling;
• logical path validation;
• storage error translation where practical;
• Walk and HasAny behavior through test doubles or controlled fixtures;
• managed deletion boundaries;
• app-level local-to-SSH and SSH-to-local planning or wiring using fakes/mocks where possible.

Add integration tests only if they are skipped unless explicit SSH test endpoint environment variables are configured. Normal go test ./... must not require a live SSH server.

Completion Criteria

• SSH/SFTP backend compiles and satisfies storage.Backend.
• Runtime run can execute supported SSH source and destination flows.
• Local MVP tests still pass.
• Normal test runs do not require a live SSH server.
• User docs accurately describe implemented SSH behavior and boundaries.

Stage 2: S3-Compatible Backend

Goal

Implement S3-compatible object storage backend support for sources and destinations through the existing storage interface and app-level backend factory.

Implementation Scope

Add an S3-compatible adapter package under internal/adapters/s3.

The backend must implement the current internal/storage.Backend contract:

• ReadFile and OpenReader;
• WriteFile and WriteFrom;
• Stat;
• Walk;
• HasAny;
• DeleteManagedBundle.

Use the existing accepted config shape:

backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: some/prefix
region: us-east-1
force_path_style: true
credentials:
  access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
  secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY

Credential behavior:

• read access key and secret key from the named environment variables when configured;
• support standard SDK credential discovery only if it does not weaken explicit config behavior;
• do not put literal secrets in YAML.

Object semantics:

• treat prefixes as object trees, not real directories;
• normalize configured prefix plus logical path with exact path-boundary matching;
• Stat must not synthesize directory metadata only because objects exist below a prefix;
• Walk should use object-list pagination and should not load an entire prefix into memory;
• HasAny should stop after the first matching object;
• DeleteManagedBundle must delete only listed managed output objects plus .distributor.json.

Write behavior:

• treat successful object PUT as publish-on-success;
• set content type from storage.WriteOptions where available;
• spool or buffer WriteFrom only when required by the SDK or backend;
• preserve overwrite checks and conservative conflict behavior.

Content type behavior should cover at least:

• .md: text/markdown; charset=utf-8;
• .html: text/html; charset=utf-8;
• .json: application/json;
• .txt: text/plain; charset=utf-8.

Supported pipeline combinations:

• local source to S3 destination;
• S3 source to local destination;
• S3 source to S3 destination where feasible through streaming or backend-owned staging.

Documentation Updates

After implementation, update only current-behavior docs:

• docs/config.md: mark S3 as executable and document implemented credential behavior.
• docs/operations.md: add S3 source/destination layout, prefix, and recovery notes.
• docs/troubleshooting.md: add common S3 credential, bucket, endpoint, prefix, and permission failures.
• docs/internal/storage.md: add implemented S3 adapter behavior and tests.
• docs/policy/development.md: update backend addition guidance if implementation changes the pattern.
• examples/: add only safe S3 examples that use placeholder endpoints and environment variable names.

Do not document force overwrite or notification adapters as implemented in this stage.

Tests

Add unit tests for:

• S3 config execution wiring;
• credential environment variable handling;
• key and prefix normalization;
• exact prefix boundary behavior;
• path traversal rejection;
• content type selection;
• paginated Walk behavior through mocks/fakes;
• early-stop HasAny;
• managed deletion boundaries;
• publish planning with S3 destination state fixtures.

Add integration tests only if they are skipped unless explicit S3-compatible endpoint credentials are configured. Normal go test ./... must not require live S3 credentials.

Completion Criteria

• S3-compatible backend compiles and satisfies storage.Backend.
• Runtime run can execute supported S3 source and destination flows.
• Local and SSH behavior, if implemented, remain unchanged.
• Normal test runs do not require live S3.
• User docs accurately describe implemented S3 behavior and boundaries.

Stage 3: Cross-Backend Hardening and Documentation

Goal

Harden behavior across implemented backend combinations, improve operator-facing failures, and synchronize current-behavior documentation and examples after remote backend support exists.

Implementation Scope

Exercise and harden representative flows across all implemented backend types:

• local source to local archive destination;
• local source to local HTML destination;
• local source to multiple destinations with different publish policies;
• local source to SSH destination, when SSH is implemented and test credentials exist;
• SSH source to local destination, when SSH is implemented and test credentials exist;
• local source to S3 destination, when S3 is implemented and test credentials exist;
• S3 source to local destination, when S3 is implemented and test credentials exist.

Improve error context where practical for:

• invalid config;
• invalid source manifest;
• digest mismatch;
• destination conflict;
• unmanaged destination path;
• backend read, write, stat, walk, and delete failures;
• transform failures;
• partial fan-out failures.

Ensure errors and logs identify pipeline id, destination id, bundle path or id, backend type, and logical path where useful without exposing secrets.

Do not add force overwrite behavior in this stage.

Documentation Updates

Update current-behavior docs after hardening:

• README.md: keep the quickstart local unless remote examples become safe and concise.
• docs/cli.md: document any changed output or diagnostics.
• docs/config.md: ensure backend support status and config reference match implementation.
• docs/operations.md: document cross-backend state layout, retry behavior, and recovery caveats.
• docs/troubleshooting.md: add recurring SSH/S3 failure modes discovered during hardening.
• docs/internal/: update storage, publish, app, and config internals where behavior changed.
• examples/: keep examples copyable and free of secrets; remote examples must rely on placeholders and environment variables.

Tests

Add or expand tests for:

• dry-run across multiple destinations and backend types;
• partial failure behavior;
• repeated run idempotency;
• older/newer destination state behavior across backends;
• destination state output metadata accuracy;
• generated HTML output metadata accuracy;
• destructive replacement safety across implemented backends;
• error context for common failures.

Integration tests for SSH or S3 must remain opt-in through environment variables.

Completion Criteria

• Implemented backend combinations behave consistently through the common pipeline path.
• Repeated runs are idempotent.
• Destructive paths remain bounded to managed destination bundle paths.
• Operator-facing errors are actionable.
• Current-behavior docs and examples match implemented backend support.

Stage 4: Explicit Force Overwrite

Goal

Introduce explicit operator-requested force behavior for controlled overwrite cases that remain intentionally unsupported by default.

Implementation Scope

Add a CLI-only force option:

distributor run --config config.yml --force

Force must be explicit per run. Do not add a persistent config default for force behavior.

Define and implement force planning for:

• unmanaged non-empty destination paths;
• destination state with a different source id;
• destination state with matching source id and matching created timestamp but different digest;
• destination state with mismatched pipeline_id or destination_id;
• destination newer than source when transfer policy explicitly allows replacement.

Once force behavior exists, update transfer policy validation only for values supported by implemented force behavior:

• on_destination_newer: replace;
• on_conflict: replace.

Safety requirements:

• non-force behavior remains unchanged and conservative;
• dry-run must show destructive force actions before any forced run;
• force must never delete above the resolved destination bundle path or configured destination prefix;
• local replacement should remain staged where practical;
• S3 replacement must remain constrained to the destination bundle prefix;
• managed state should still be written only after successful output writes;
• logs and output must clearly mark force decisions.

Documentation Updates

After implementation, update:

• docs/cli.md: document --force syntax and dry-run workflow.
• docs/config.md: document newly accepted transfer policy values and note force is CLI-only.
• docs/operations.md: document safe force workflow and recovery boundaries.
• docs/troubleshooting.md: describe when force may be appropriate and when it remains unsafe.
• docs/internal/publish.md and docs/internal/state.md: document force planning and comparison handling.

Do not document force as a default or config-only behavior.

Tests

Add tests for:

• force rejected or unavailable when the flag is absent;
• unmanaged non-empty destination overwritten only with force;
• different source id overwritten only with force and allowed policy;
• same id and created timestamp with different digest overwritten only with force and allowed policy;
• destination newer replaced only with force and allowed policy;
• pipeline or destination id mismatch overwritten only with force and allowed policy;
• dry-run reports destructive force actions without writing;
• force deletes only bounded destination bundle paths;
• local, SSH, and S3 backends, where implemented, preserve deletion boundaries.

Completion Criteria

• Force overwrite behavior is explicit, logged, dry-runnable, and test-covered.
• Default non-force behavior remains unchanged.
• User docs clearly describe force risks and safe workflow.

Stage 5: Release Readiness

Goal

Perform a final quality pass before treating distributor as ready for routine use against real producer pipelines and implemented destination backends.

Implementation Scope

Review:

• package boundaries against docs/policy/architecture.md;
• contributor workflow against docs/policy/development.md;
• user docs against docs/policy/documentation.md;
• CLI UX and command output;
• config validation and examples;
• manifest and state compatibility;
• destructive operation safety;
• backend error handling;
• logging and diagnostics for unattended operation;
• test coverage for core invariants.

Do not add new product features in this stage.

Documentation Updates

Update current-behavior docs only for issues found during the readiness review.

If release packaging, version injection, or installation workflow is added, document it in the appropriate current-behavior user or development docs.

Tests

Run:

go test ./...

Also verify representative CLI examples that are documented as runnable.

Completion Criteria

• A dry-run can be performed safely against real configured sources and destinations.
• Repeated runs are idempotent.
• Destructive replacement cannot occur outside managed destination bundle paths.
• Current docs accurately reflect the application.
• The project is ready to deploy against one real producer pipeline.

Deferred Work

The following work remains intentionally deferred unless a future roadmap promotes it:

• external notification adapters such as email, ntfy, Gotify, or Pushover;
• RSS or Atom feed generation;
• static site index pages beyond sidecar HTML output;
• destination path remapping rules;
• HTML themes beyond the minimal deterministic template;
• full plugin architecture;
• web UI;
• report editing;
• producer pipeline execution;
• database-backed state;
• complex retry queues;
• concurrent publication workers;
• symlink support;
• warning-only digest mismatch handling;
• password-based SSH authentication in YAML;
• broad recursive or prefix deletion outside explicitly bounded force behavior.

Validation

For roadmap-only edits:

git status --short
git diff -- docs/roadmap
rg -n "docs/roadmap/(packages|contracts|storage|config|documentation)\\.md" README.md docs examples
rg -n "docs/roadmap/(packages|contracts|storage|config|documentation)\\.md" .
rg -n "SSH|S3|--force|force overwrite|notification adapter|future|planned" README.md docs/*.md docs/internal docs/policy examples

Also search docs/roadmap for old MVP stage headings and titles from deleted roadmap files. That check should return no matches.

The final SSH/S3/force/future-work search is not expected to return zero results. Review matches and confirm they are either under roadmap material or clearly marked as unsupported current behavior.

Go tests are not required for documentation-only roadmap rationalization unless examples, behavior docs, or code change.