Update and refine the implementation roadmap

docs/roadmap/implementation.md

@@ -9,6 +9,7 @@ The roadmap assumes the project includes these planning documents before impleme
 - `docs/roadmap/packages.md`
 - `docs/roadmap/contracts.md`
 - `docs/roadmap/config.md`
+- `docs/roadmap/storage.md`
 
 The MVP goal is a domain-agnostic bundle distributor that discovers source bundles, validates `manifest.json`, optionally transforms Markdown to HTML, publishes selected outputs to one or more destinations, and records destination state in `.distributor.json`.
 
@@ -169,16 +170,7 @@ internal/adapters/local/backend.go
 internal/storage/fake/
 ```
 
-Define backend operations needed by the MVP, including:
-
-- read file;
-- write file;
-- test existence;
-- list files or tree entries;
-- read destination state file if present;
-- create directories/prefixes as needed;
-- delete explicit managed files safely;
-- write files using temporary/staged writes where practical.
+Implement the storage contract in `docs/roadmap/storage.md`, including backend-rooted logical paths, hybrid byte/stream IO, metadata, traversal, typed errors, managed deletion, and efficient destination emptiness helper behavior.
 
 The fake backend should exist for unit tests of config, bundle, state, and publish logic without real local, SSH, or S3 IO.
 
@@ -197,13 +189,16 @@ The local backend must:
 
 Add tests for:
 
-- backend read/write/list/exists behavior;
-- safe path joining;
+- path validation;
+- backend byte and stream read/write behavior;
+- `Stat`, `Walk`, and materialized deterministic traversal helper behavior;
 - traversal rejection;
-- symlink rejection for source reads;
-- explicit-file deletion guard behavior;
-- local destination emptiness detection;
-- fake backend behavior sufficient for core package tests.
+- symlink entry reporting and source-read rejection;
+- staged write behavior where testable;
+- typed storage errors and helper predicates;
+- managed deletion guard behavior;
+- early-stop destination emptiness helper behavior;
+- fake backend parity for core package tests.
 
 ### Completion Criteria
 
@@ -319,6 +314,7 @@ Implement comparison outcomes:
 
 - destination absent;
 - destination unmanaged/non-empty;
+- destination state pipeline or destination id mismatch;
 - same source manifest;
 - same source id, destination older;
 - same source id, destination newer;
@@ -339,6 +335,8 @@ Test validation for:
 - invalid output metadata;
 - malformed published timestamp.
 
+Timestamps should parse RFC3339 input and distributor-written timestamps should normalize to RFC3339 UTC.
+
 ### Completion Criteria
 
 - Destination state can be parsed and validated independently.
@@ -485,7 +483,7 @@ Add tests for:
 - Generated outputs are recorded in destination state.
 - Dry-run reports transform outputs that would be generated.
 
-## Stage 8: No-Op Notification Stage and Pipeline Orchestration Polish
+## Stage 8: No-Op Notification Stage, Pipeline Polish, and Local MVP Checkpoint
 
 ### Goal
 
@@ -500,7 +498,7 @@ internal/notify/notify.go
 internal/notify/noop.go
 ```
 
-Integrate a no-op notifier after successful publication/skip handling as appropriate.
+Integrate a no-op notifier after actual successful publication or replacement. Skipped destinations should not invoke the notifier in the MVP.
 
 Clarify orchestration behavior when one destination fails. For MVP, fan-out should be deterministic and sequential. Continue planning and reporting later destinations where safe, but return non-zero if any destination fails.
 
@@ -526,9 +524,11 @@ Add tests for:
 
 - The pipeline shape includes notification as an internal no-op stage.
 - Run output is useful for unattended operation logs.
-- Local MVP behavior remains passing.
+- Local MVP behavior remains passing and is ready for one real local producer pipeline.
 
-## Stage 9: Native SSH/SFTP Backend
+Stages 1 through 8 define the local MVP checkpoint. Later stages extend the local MVP with remote backends, cross-backend hardening, user-facing documentation sync, and release readiness.
+
+## Stage 9: Native SSH/SFTP Backend Roadmap Extension
 
 ### Goal
 
@@ -585,7 +585,7 @@ If practical, add integration tests that can be skipped unless an SSH test endpo
 - Normal tests do not depend on a live SSH server.
 - At least local-to-SSH and SSH-to-local flows are documented or manually testable.
 
-## Stage 10: S3-Compatible Backend
+## Stage 10: S3-Compatible Backend Roadmap Extension
 
 ### Goal
 
@@ -614,14 +614,15 @@ credentials:
   secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
 ```
 
-Implement storage operations over S3 object keys:
+Implement storage operations over S3 object keys through the common storage contract:
 
-- read object;
-- write object;
-- exists;
-- list prefix;
-- delete managed prefix or listed managed files;
-- read/write `.distributor.json`.
+- `ReadFile` and `OpenReader`;
+- `WriteFile` and `WriteFrom`;
+- `Stat`;
+- `Walk` using object-list pagination where available;
+- `HasAny` with early stop for destination emptiness checks;
+- `DeleteManagedBundle`;
+- read/write `.distributor.json` through normal storage operations.
 
 Set reasonable content types where available:
 
@@ -656,7 +657,7 @@ If practical, add integration tests gated by environment variables or a local S3
 - S3 source and destination flows are supported through the common pipeline path.
 - Normal tests do not require live S3.
 
-## Stage 11: Cross-Backend End-to-End Coverage and Hardening
+## Stage 11: Cross-Backend End-to-End Coverage and Hardening Roadmap Extension
 
 ### Goal
 
@@ -785,6 +786,61 @@ Do not add new product features in this stage.
 - Destructive replacement cannot occur outside managed destination bundle paths.
 - Final docs accurately reflect the application.
 
+## Stage 14: Explicit Force Overwrite Roadmap Extension
+
+### Goal
+
+Introduce explicit operator-requested force behavior for controlled overwrite cases that are intentionally outside the local MVP.
+
+### Scope
+
+Add a CLI-only force option such as:
+
+```bash
+distributor run --config config.yml --force
+```
+
+Define and implement force planning for:
+
+- unmanaged non-empty destination paths;
+- destination state with a different source id;
+- destination state with matching source id and created timestamp but different digest;
+- destination state with mismatched `pipeline_id` or `destination_id`;
+- destination newer than source when the transfer policy explicitly allows replacement.
+
+Force behavior must be explicit per run. It should not be a persistent default in config for this stage.
+
+Update transfer policy validation to allow broader values only when force behavior is implemented and documented:
+
+- `on_destination_newer: replace`
+- `on_conflict: replace`
+
+### Safety Requirements
+
+- Dry-run must show every file or object that would be written or deleted before a forced run.
+- Force must still never delete above the resolved destination bundle path or configured destination prefix.
+- Filesystem replacement should remain staged where practical.
+- S3 replacement must remain constrained to the destination bundle prefix.
+- Logs must clearly mark force decisions and include pipeline id, destination id, bundle id, and reason.
+
+### Tests
+
+Add tests for:
+
+- force rejected when the flag is absent;
+- unmanaged non-empty destination overwritten only with force;
+- different source id overwritten only with force and allowed policy;
+- same id and created but different digest overwritten only with force and allowed policy;
+- destination newer replaced only with force and allowed policy;
+- pipeline or destination id mismatch overwritten only with force and allowed policy;
+- dry-run reports destructive force actions without writing;
+- destructive paths remain bounded to the destination bundle path.
+
+### Completion Criteria
+
+- Force overwrite behavior is explicit, logged, dry-runnable, and test-covered.
+- Default non-force behavior remains unchanged and conservative.
+
 ## Deferred Post-MVP Work
 
 The following items are intentionally outside the MVP unless explicitly pulled into a later roadmap:
@@ -804,5 +860,4 @@ The following items are intentionally outside the MVP unless explicitly pulled i
 - concurrent publication workers;
 - symlink support;
 - warning-only digest mismatch handling;
-- force or unmanaged-overwrite destination behavior;
 - password-based SSH authentication in YAML.