Update and refine the implementation roadmap

docs/roadmap/config.md

@@ -396,11 +396,12 @@ Supported fields:
 - `on_destination_newer`
 - `on_conflict`
 
-Recommended supported values:
+MVP supported values are intentionally limited by field:
 
-- `skip`
+- `on_destination_same`: `skip` or `fail`
-- `replace`
+- `on_destination_older`: `replace` or `fail`
-- `fail`
+- `on_destination_newer`: `skip` or `fail`
+- `on_conflict`: `fail`
 
 Recommended MVP defaults:
 
@@ -417,6 +418,7 @@ Safety rule:
 - `replace` must never perform broad deletion against a destination root.
 - `replace` may only operate within a resolved destination bundle path and should delete only files recorded in existing `.distributor.json.outputs` plus `.distributor.json` where practical.
 - Unmanaged non-empty destination paths fail in the MVP. Force or unmanaged overwrite configuration is deferred.
+- Broader replacement values, including replacing newer destinations or conflicts, are deferred to a later explicit force-overwrite stage.
 
 ## Path Mapping
 
@@ -574,5 +576,5 @@ Future notification policy should require:
 The following configuration ideas are intentionally outside the MVP:
 
 - warning-only digest mismatch handling;
-- unmanaged destination overwrite flags;
+- unmanaged destination overwrite flags until the explicit force-overwrite roadmap stage;
-- force replacement of destinations with different source ids.
+- force replacement of destinations with different source ids until the explicit force-overwrite roadmap stage.

docs/roadmap/contracts.md

@@ -13,6 +13,8 @@ The MVP contract has two durable files:
 
 `manifest.json` is not copied to the destination as destination state. Instead, `.distributor.json` records the normalized source manifest, generated output metadata, and distributor-owned publication metadata.
 
+All manifest and state timestamps should be serialized as RFC3339. Internal comparison should use parsed timestamp values, and distributor-written timestamps should be normalized to RFC3339 UTC.
+
 ## Terminology
 
 - **Source root**: Configured root path for a pipeline source.
@@ -227,6 +229,12 @@ If no `.distributor.json` exists and the destination path is non-empty:
 
 Local destination paths are empty when the destination bundle directory does not exist or exists with no entries. S3-compatible destination prefixes are empty when no objects exist below the destination bundle prefix, ignoring objects outside that exact prefix.
 
+### Pipeline or destination mismatch
+
+If `.distributor.json` exists but its `pipeline_id` or `destination_id` differs from the current pipeline or destination config:
+
+- Fail as a conflict.
+
 ### Same source manifest
 
 If `.distributor.json` exists and `source.manifest` exactly matches the current normalized source manifest:

docs/roadmap/implementation.md

@@ -9,6 +9,7 @@ The roadmap assumes the project includes these planning documents before impleme
 - `docs/roadmap/packages.md`
 - `docs/roadmap/contracts.md`
 - `docs/roadmap/config.md`
+- `docs/roadmap/storage.md`
 
 The MVP goal is a domain-agnostic bundle distributor that discovers source bundles, validates `manifest.json`, optionally transforms Markdown to HTML, publishes selected outputs to one or more destinations, and records destination state in `.distributor.json`.
 
@@ -169,16 +170,7 @@ internal/adapters/local/backend.go
 internal/storage/fake/
 ```
 
-Define backend operations needed by the MVP, including:
+Implement the storage contract in `docs/roadmap/storage.md`, including backend-rooted logical paths, hybrid byte/stream IO, metadata, traversal, typed errors, managed deletion, and efficient destination emptiness helper behavior.
-
-- read file;
-- write file;
-- test existence;
-- list files or tree entries;
-- read destination state file if present;
-- create directories/prefixes as needed;
-- delete explicit managed files safely;
-- write files using temporary/staged writes where practical.
 
 The fake backend should exist for unit tests of config, bundle, state, and publish logic without real local, SSH, or S3 IO.
 
@@ -197,13 +189,16 @@ The local backend must:
 
 Add tests for:
 
-- backend read/write/list/exists behavior;
+- path validation;
-- safe path joining;
+- backend byte and stream read/write behavior;
+- `Stat`, `Walk`, and materialized deterministic traversal helper behavior;
 - traversal rejection;
-- symlink rejection for source reads;
+- symlink entry reporting and source-read rejection;
-- explicit-file deletion guard behavior;
+- staged write behavior where testable;
-- local destination emptiness detection;
+- typed storage errors and helper predicates;
-- fake backend behavior sufficient for core package tests.
+- managed deletion guard behavior;
+- early-stop destination emptiness helper behavior;
+- fake backend parity for core package tests.
 
 ### Completion Criteria
 
@@ -319,6 +314,7 @@ Implement comparison outcomes:
 
 - destination absent;
 - destination unmanaged/non-empty;
+- destination state pipeline or destination id mismatch;
 - same source manifest;
 - same source id, destination older;
 - same source id, destination newer;
@@ -339,6 +335,8 @@ Test validation for:
 - invalid output metadata;
 - malformed published timestamp.
 
+Timestamps should parse RFC3339 input and distributor-written timestamps should normalize to RFC3339 UTC.
+
 ### Completion Criteria
 
 - Destination state can be parsed and validated independently.
@@ -485,7 +483,7 @@ Add tests for:
 - Generated outputs are recorded in destination state.
 - Dry-run reports transform outputs that would be generated.
 
-## Stage 8: No-Op Notification Stage and Pipeline Orchestration Polish
+## Stage 8: No-Op Notification Stage, Pipeline Polish, and Local MVP Checkpoint
 
 ### Goal
 
@@ -500,7 +498,7 @@ internal/notify/notify.go
 internal/notify/noop.go
 ```
 
-Integrate a no-op notifier after successful publication/skip handling as appropriate.
+Integrate a no-op notifier after actual successful publication or replacement. Skipped destinations should not invoke the notifier in the MVP.
 
 Clarify orchestration behavior when one destination fails. For MVP, fan-out should be deterministic and sequential. Continue planning and reporting later destinations where safe, but return non-zero if any destination fails.
 
@@ -526,9 +524,11 @@ Add tests for:
 
 - The pipeline shape includes notification as an internal no-op stage.
 - Run output is useful for unattended operation logs.
-- Local MVP behavior remains passing.
+- Local MVP behavior remains passing and is ready for one real local producer pipeline.
 
-## Stage 9: Native SSH/SFTP Backend
+Stages 1 through 8 define the local MVP checkpoint. Later stages extend the local MVP with remote backends, cross-backend hardening, user-facing documentation sync, and release readiness.
+
+## Stage 9: Native SSH/SFTP Backend Roadmap Extension
 
 ### Goal
 
@@ -585,7 +585,7 @@ If practical, add integration tests that can be skipped unless an SSH test endpo
 - Normal tests do not depend on a live SSH server.
 - At least local-to-SSH and SSH-to-local flows are documented or manually testable.
 
-## Stage 10: S3-Compatible Backend
+## Stage 10: S3-Compatible Backend Roadmap Extension
 
 ### Goal
 
@@ -614,14 +614,15 @@ credentials:
   secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
 ```
 
-Implement storage operations over S3 object keys:
+Implement storage operations over S3 object keys through the common storage contract:
 
-- read object;
+- `ReadFile` and `OpenReader`;
-- write object;
+- `WriteFile` and `WriteFrom`;
-- exists;
+- `Stat`;
-- list prefix;
+- `Walk` using object-list pagination where available;
-- delete managed prefix or listed managed files;
+- `HasAny` with early stop for destination emptiness checks;
-- read/write `.distributor.json`.
+- `DeleteManagedBundle`;
+- read/write `.distributor.json` through normal storage operations.
 
 Set reasonable content types where available:
 
@@ -656,7 +657,7 @@ If practical, add integration tests gated by environment variables or a local S3
 - S3 source and destination flows are supported through the common pipeline path.
 - Normal tests do not require live S3.
 
-## Stage 11: Cross-Backend End-to-End Coverage and Hardening
+## Stage 11: Cross-Backend End-to-End Coverage and Hardening Roadmap Extension
 
 ### Goal
 
@@ -785,6 +786,61 @@ Do not add new product features in this stage.
 - Destructive replacement cannot occur outside managed destination bundle paths.
 - Final docs accurately reflect the application.
 
+## Stage 14: Explicit Force Overwrite Roadmap Extension
+
+### Goal
+
+Introduce explicit operator-requested force behavior for controlled overwrite cases that are intentionally outside the local MVP.
+
+### Scope
+
+Add a CLI-only force option such as:
+
+```bash
+distributor run --config config.yml --force
+```
+
+Define and implement force planning for:
+
+- unmanaged non-empty destination paths;
+- destination state with a different source id;
+- destination state with matching source id and created timestamp but different digest;
+- destination state with mismatched `pipeline_id` or `destination_id`;
+- destination newer than source when the transfer policy explicitly allows replacement.
+
+Force behavior must be explicit per run. It should not be a persistent default in config for this stage.
+
+Update transfer policy validation to allow broader values only when force behavior is implemented and documented:
+
+- `on_destination_newer: replace`
+- `on_conflict: replace`
+
+### Safety Requirements
+
+- Dry-run must show every file or object that would be written or deleted before a forced run.
+- Force must still never delete above the resolved destination bundle path or configured destination prefix.
+- Filesystem replacement should remain staged where practical.
+- S3 replacement must remain constrained to the destination bundle prefix.
+- Logs must clearly mark force decisions and include pipeline id, destination id, bundle id, and reason.
+
+### Tests
+
+Add tests for:
+
+- force rejected when the flag is absent;
+- unmanaged non-empty destination overwritten only with force;
+- different source id overwritten only with force and allowed policy;
+- same id and created but different digest overwritten only with force and allowed policy;
+- destination newer replaced only with force and allowed policy;
+- pipeline or destination id mismatch overwritten only with force and allowed policy;
+- dry-run reports destructive force actions without writing;
+- destructive paths remain bounded to the destination bundle path.
+
+### Completion Criteria
+
+- Force overwrite behavior is explicit, logged, dry-runnable, and test-covered.
+- Default non-force behavior remains unchanged and conservative.
+
 ## Deferred Post-MVP Work
 
 The following items are intentionally outside the MVP unless explicitly pulled into a later roadmap:
@@ -804,5 +860,4 @@ The following items are intentionally outside the MVP unless explicitly pulled i
 - concurrent publication workers;
 - symlink support;
 - warning-only digest mismatch handling;
-- force or unmanaged-overwrite destination behavior;
 - password-based SSH authentication in YAML.

docs/roadmap/packages.md

@@ -148,7 +148,7 @@ for each selected pipeline:
       build publish plan
       transform as required by that destination
       execute publish plan unless dry-run
-      run noop notifier
+      run noop notifier after actual publication or replacement
 ```
 
 Boundaries:
@@ -309,6 +309,7 @@ Comparison rules:
 - same source id, newer destination source `created`: skip;
 - same source id, same `created`, different digest: conflict;
 - different source id: conflict;
+- pipeline id or destination id mismatch: conflict;
 - absent state: publish only if safe;
 - unmanaged non-empty path: fail.
 
@@ -327,22 +328,12 @@ Responsibilities:
 - define object/file metadata types;
 - define path/prefix helpers;
 - define common storage errors;
-- provide backend registry mechanisms.
+- provide backend registry mechanisms;
 - provide a fake backend for core package tests.
 
-The core application should use storage interfaces such as:
+The detailed storage contract is defined in `docs/roadmap/storage.md`. Core application code should use that storage interface for backend-rooted logical paths, byte and stream IO, metadata, traversal, typed errors, emptiness checks, and managed deletion.
 
-```go
+Destructive APIs should remain narrow. Prefer managed deletion of files recorded in `.distributor.json` instead of broad recursive deletion.
-type Backend interface {
-    ReadFile(ctx context.Context, path string) ([]byte, error)
-    WriteFile(ctx context.Context, path string, data []byte, opts WriteOptions) error
-    Exists(ctx context.Context, path string) (bool, error)
-    List(ctx context.Context, prefix string) ([]Entry, error)
-    DeleteFiles(ctx context.Context, paths []string) error
-}
-```
-
-Destructive APIs should remain narrow. Prefer deleting explicit files recorded in `.distributor.json` instead of broad recursive deletion.
 
 Boundaries:
 
@@ -505,100 +496,6 @@ Responsibilities:
 - ensure logs omit secrets;
 - provide consistent fields for pipeline id, bundle id, destination id, backend, path, action, and reason.
 
-## Implementation Slices
-
-### Slice 1: Skeleton, config, storage, and source bundle validation
-
-Deliver:
-
-- basic CLI skeleton;
-- config loading and validation;
-- storage interface, local backend, and fake backend;
-- source manifest model;
-- source bundle discovery;
-- file size and SHA-256 validation;
-- bundle digest validation;
-- local backend sufficient for validation;
-- fixtures for valid and invalid bundles.
-
-Useful commands:
-
-```bash
-distributor validate ./examples/weather-bundle
-```
-
-### Slice 2: Destination state and dry-run planning
-
-Deliver:
-
-- `.distributor.json` model;
-- destination state comparison;
-- publish plan model;
-- dry-run output;
-- local source to local destination planning;
-- tests for same/older/newer/conflict/unmanaged cases.
-
-Useful command:
-
-```bash
-distributor run --config ./examples/local.yml --dry-run
-```
-
-### Slice 3: Local publish execution
-
-Deliver:
-
-- local destination writes;
-- source-file publication;
-- `.distributor.json` writes;
-- replacement safety checks;
-- skip behavior;
-- narrow deletion behavior based on destination state outputs.
-
-Useful command:
-
-```bash
-distributor run --config ./examples/local.yml
-```
-
-### Slice 4: Markdown-to-HTML transform
-
-Deliver:
-
-- Markdown transform registry;
-- Markdown-to-HTML implementation;
-- per-destination `publish.source` and `publish.html` behavior;
-- generated output metadata in `.distributor.json`;
-- tests for source-only, HTML-only, and source-plus-HTML destinations.
-
-### Slice 5: S3 backend
-
-Deliver:
-
-- S3-compatible backend;
-- endpoint/bucket/prefix/region/force-path-style config;
-- credential handling via environment or standard mechanisms;
-- object listing, read, write, and narrow delete operations;
-- dry-run and publish coverage using fake or local-compatible test strategy.
-
-### Slice 6: SSH/SFTP backend
-
-Deliver:
-
-- native SFTP backend;
-- `uri` and `path` config;
-- documented authentication behavior;
-- read/write/list/delete operations;
-- adapter tests or documented integration test strategy.
-
-### Slice 7: No-op notification stage and future extension seam
-
-Deliver:
-
-- no-op notifier wired into orchestration;
-- clear internal contract for future email/ntfy adapters;
-- no user-facing notification behavior beyond no-op unless implemented.
-
 ## Deferred Ideas
 
 The following are intentionally out of MVP unless separately accepted in a later roadmap:

docs/roadmap/storage.md

@@ -0,0 +1,230 @@
+# Storage Interface Roadmap
+
+This roadmap defines the planned `internal/storage` contract for the `distributor` MVP. The goal is to give bundle validation, destination state inspection, publish planning, and backend adapters one consistent IO boundary without leaking local filesystem, SSH/SFTP, or S3-specific behavior into core packages.
+
+## Purpose and Invariants
+
+The storage layer is responsible for safe, backend-rooted access to files, objects, prefixes, and destination bundle paths.
+
+Core invariants:
+
+- Backends are opened at configured roots.
+- Core packages operate on backend-rooted logical paths, not absolute filesystem paths or raw object keys.
+- Backend adapters translate native storage behavior into common storage entries and typed errors.
+- Destructive operations remain narrow and managed.
+- Staging or atomic write behavior belongs behind the storage interface where practical.
+- The fake backend exists for tests only and must not be registered as a runtime backend.
+
+## Logical Path Model
+
+Storage paths are slash-separated logical paths relative to an already configured backend root.
+
+File paths:
+
+- must be non-empty;
+- must be relative;
+- must be clean;
+- must not contain `.` or `..` segments;
+- must not start with `/`;
+- must not contain backslashes;
+- must not resolve outside the backend root.
+
+Prefix paths use the same slash-separated model. A prefix may be empty to represent the backend root for listing and destination emptiness checks.
+
+Backends own conversion from logical paths to native paths or object keys. Core packages should not construct local filesystem paths, SFTP paths, or S3 object keys directly.
+
+## Core Interface Shape
+
+The MVP should use a hybrid byte and stream interface:
+
+```go
+type Backend interface {
+    ReadFile(ctx context.Context, path string) ([]byte, error)
+    OpenReader(ctx context.Context, path string) (io.ReadCloser, error)
+    WriteFile(ctx context.Context, path string, data []byte, opts WriteOptions) (Entry, error)
+    WriteFrom(ctx context.Context, path string, r io.Reader, opts WriteOptions) (Entry, error)
+    Stat(ctx context.Context, path string) (Entry, error)
+    Walk(ctx context.Context, prefix string, opts WalkOptions, fn WalkFunc) error
+    HasAny(ctx context.Context, prefix string) (bool, error)
+    DeleteManagedBundle(ctx context.Context, bundlePath string, managedOutputPaths []string, opts DeleteOptions) error
+}
+```
+
+Byte helpers are expected to cover manifests, destination state, small source artifacts, and generated outputs. Stream methods are included from the start for backend flexibility and larger future artifacts.
+
+Write operations should create required parent directories or prefixes as needed.
+
+## Entries and Metadata
+
+Storage metadata should be represented by an `Entry` model with at least:
+
+- backend-relative logical path;
+- entry type;
+- size, where available.
+
+Entry types:
+
+- `file`: filesystem file or object-storage object;
+- `directory`: filesystem directory or logical prefix;
+- `symlink`: local filesystem symlink;
+- `other`: unknown or unsupported native entry type.
+
+`Stat` returns metadata for one logical path.
+
+`Walk` traverses entries below a prefix and calls a callback for each entry. `WalkOptions` should include:
+
+- whether traversal is recursive;
+- an optional entry limit for callers that only need to know whether content exists.
+
+Backends may stream or paginate traversal internally. S3-compatible adapters should not need to load a whole prefix into memory to satisfy traversal.
+
+Raw traversal is not required to be lexically sorted. A helper that materializes walk results for bundle discovery, tests, or CLI output should sort entries lexically by logical path before returning them.
+
+`HasAny` reports whether at least one entry exists below a prefix. It should stop as soon as content is found.
+
+Source validation must reject symlink entries reported by local `Stat` or `Walk`.
+
+## Read Behavior
+
+`ReadFile` reads the whole object into memory and is appropriate for MVP manifest, state, and ordinary artifact handling.
+
+`OpenReader` returns a stream for callers that need to copy or hash content without requiring a second storage-specific API. Callers must close the returned reader.
+
+Both read methods must:
+
+- validate logical paths before backend access;
+- reject directories, prefixes, symlinks, and unsupported entries;
+- return typed not-found and invalid-path errors where applicable.
+
+## Write Behavior
+
+`WriteOptions` should include:
+
+- content type, when the destination backend can use it;
+- overwrite permission;
+- atomic or staged write preference.
+
+Backends own staging and atomic behavior where practical:
+
+- Local backend writes to a temporary file in the destination directory and renames or promotes into place.
+- SSH/SFTP backend should use a temporary remote file and rename where available.
+- S3-compatible backend treats a successful object PUT as publish-on-success and applies content type metadata.
+
+If overwrite is false and the target exists, writes should fail with an already-exists error.
+
+`WriteFile` and `WriteFrom` should return the written `Entry`, including final path and size where available.
+
+`DeleteOptions` should include:
+
+- whether missing managed output paths are ignored;
+- whether empty parent directories may be pruned for filesystem-like backends.
+
+## Managed Deletion
+
+The storage interface should expose a guarded managed deletion operation rather than raw recursive delete.
+
+`DeleteManagedBundle(ctx, bundlePath, managedOutputPaths, opts)` may delete only:
+
+- files or objects listed in valid `.distributor.json.outputs`;
+- `.distributor.json` at the destination bundle path;
+- empty directories created by those files, for filesystem-like backends.
+
+`managedOutputPaths` are relative to the destination bundle path. The backend validates each path and resolves it under `bundlePath`.
+
+If `bundlePath == ""`, deletion may remove explicit managed files at the destination root, but must never delete the root itself.
+
+Prefix or recursive deletion is out of MVP scope. A future force-overwrite stage may add broader behavior, but it must remain explicit and separately documented.
+
+## Destination Emptiness
+
+Destination emptiness should use `HasAny(prefix)` and typed not-found behavior. Callers that only need emptiness must not materialize a full recursive traversal.
+
+Rules:
+
+- A local destination bundle path is empty when the directory does not exist or exists with no entries.
+- An S3-compatible prefix is empty when no objects exist below that exact destination bundle prefix.
+- Entries outside the exact destination bundle path or prefix do not affect emptiness.
+
+## Error Model
+
+Storage should expose typed error categories with wrapping context. Callers should use helper predicates rather than string matching.
+
+Required categories:
+
+- not found;
+- already exists;
+- not empty;
+- invalid path;
+- conflict;
+- permission;
+- temporary;
+- unsupported;
+- unknown.
+
+Adapters should translate backend-native errors into these categories while preserving useful operation, backend, path, and cause context.
+
+## Adapter Expectations
+
+### Local
+
+The local backend should:
+
+- constrain all operations beneath the configured root;
+- reject traversal and absolute logical paths;
+- report symlinks through metadata;
+- reject symlink reads for source artifacts;
+- use staged writes where practical;
+- perform managed deletion only for explicit managed files and `.distributor.json`;
+- clean up empty directories created by managed outputs where safe.
+
+### Fake
+
+The fake backend should:
+
+- be in-memory and deterministic;
+- implement the same logical path validation rules;
+- support `Stat`, `Walk`, `HasAny`, byte reads and writes, stream reads and writes, managed deletion, and destination emptiness helper behavior;
+- support configured symlink entries for validation tests;
+- be used only by tests.
+
+### SSH/SFTP
+
+The SSH/SFTP backend should:
+
+- use native SFTP operations;
+- enforce the same logical path rules as local storage;
+- use temporary file plus rename for staged writes where available;
+- translate remote errors into storage error categories;
+- avoid exposing SSH or SFTP dependency types through `internal/storage`.
+
+### S3-Compatible
+
+The S3-compatible backend should:
+
+- treat prefixes as object trees, not real directories;
+- normalize configured prefix plus logical path into object keys;
+- use object PUT as publish-on-success;
+- set content type from `WriteOptions`;
+- implement traversal and emptiness by exact prefix;
+- use backend pagination for traversal where available;
+- allow `HasAny` to stop after the first matching object;
+- constrain managed deletion to listed output objects and `.distributor.json`.
+
+## Tests and Fixtures
+
+Storage implementation stages should test:
+
+- path validation rejects absolute paths, traversal, empty file paths, backslashes, and dot segments;
+- `Walk` visits backend-rooted logical paths under a prefix and supports recursive traversal;
+- a materializing helper sorts walk results lexically for deterministic tests and CLI output;
+- `HasAny` returns quickly for non-empty prefixes without requiring full traversal;
+- `ReadFile` and `OpenReader` return equivalent bytes;
+- `WriteFile` and `WriteFrom` honor overwrite and content-type options;
+- local staged writes do not leave final files on failure where testable;
+- managed deletion deletes only state-listed files and `.distributor.json`;
+- managed deletion never deletes destination root or unlisted files;
+- destination emptiness helper handles missing, empty, and non-empty local paths;
+- S3-compatible traversal can use pagination without loading a whole prefix into memory;
+- symlink entries are reported and rejected by source validation;
+- typed errors are usable through helper predicates;
+- fake backend behavior matches local backend semantics relevant to core tests.