distributor/docs/roadmap/latest_paths.md

Roadmap: Latest Path Destinations

Purpose

Support stable "latest" publication paths as normal fan-out destinations.

A latest destination republishes the current winning source bundle to a fixed destination path such as /weather/latest/, while archive destinations preserve source-relative bundle paths such as /weather/daily/brentwood/2026-06-01/.

This feature should be implemented before link generation so URL construction can use final destination path semantics. It is operationally destructive and benefits from index-mode HTML, although it must not require index mode or link generation.

Current Implementation Grounding

Current run behavior computes the destination bundle path as the source bundle path relative to the source root. Publication then writes selected outputs and .distributor.json below that destination bundle path.

Backend roots differ by backend:

• local and SSH/SFTP use configured path as the backend root;
• S3 uses configured bucket plus optional prefix as the backend root.

Destination comparison uses .distributor.json at the destination bundle path. Replacement and skip decisions are based on the normalized source manifest recorded in destination state. Unmanaged non-empty destinations fail unless explicit force behavior is requested.

Goals

• Add destination-local path mapping.
• Preserve source-relative destination paths as the default.
• Add fixed path mapping for latest-style destinations.
• For fixed destinations, select only the newest discovered source bundle for publication.
• Make fixed mapping work for local, SSH/SFTP, and S3 backends.
• Continue using normal destination state comparison and replacement rules.
• Keep link generation and index-mode HTML complementary, not required.

Non-Goals

• Do not add symlink-based latest behavior.
• Do not add feed or index generation.
• Do not change source manifest schema.
• Do not make producers responsible for latest publication.
• Do not add domain-specific latest selection rules.

Configuration Shape

Add destination-level path mapping:

path_mapping:
  mode: preserve_relative

Modes:

• preserve_relative: existing/default behavior.
• fixed: publish selected outputs directly at the destination backend root.

Example fixed destination:

destinations:
  - id: latest-html
    backend: ssh
    host: web.example.com
    user: deploy
    path: /srv/www/weather/latest
    path_mapping:
      mode: fixed
    publish:
      source: false
      html: true

For S3, the fixed destination root is the configured bucket plus prefix.

Path Mapping Semantics

preserve_relative:

destination bundle path = source-root-relative bundle path

fixed:

destination bundle path = ""

That empty logical destination bundle path means the destination backend root. Storage writes still use normal backend-rooted logical paths for outputs and state.

Newest Bundle Selection

preserve_relative destinations should continue publishing every selected source bundle independently.

fixed destinations should publish only one bundle per run: the newest discovered source bundle selected for that destination. Newest selection should be deterministic:

1. choose the bundle with the greatest source manifest created timestamp;
2. if multiple bundles have the same greatest created timestamp, use the source-root-relative bundle path ascending as a tie-breaker.

The selected bundle then uses the existing destination comparison, skip, replacement, cleanup, and force rules. Older discovered bundles should not be planned or written to that fixed destination in the same run.

Dry-run output should identify the number of fixed-destination candidates and the selected source bundle so operators can see which bundle would become latest.

Safety Rules

Fixed path mapping is more destructive than archive-style publication because newer bundles replace prior contents at the same destination path.

Required behavior:

• dry-run output must identify fixed path mapping and planned replacement;
• dry-run output must include an extra fixed-path warning or summary count when destructive replacement is possible;
• replacement should delete only managed outputs recorded in valid destination state when possible;
• unmanaged non-empty fixed destinations must fail unless force is explicitly requested;
• --force may be used when fixed mapping targets the backend root, but force replacement must remain bounded to the configured destination backend root and must never delete above it;
• fixed mapping to the backend root requires especially clear dry-run reporting.

Relationship To Other Roadmaps

HTML index mode is useful for fixed web destinations because index.html supports stable directory-style URLs, but sidecar HTML and source-only outputs should still work.

Link generation should use fixed path semantics so latest URLs are based on the fixed destination root, not the original source-relative archive path.

Producer manifest creation and remote validation are independent of destination path mapping.

Implementation Stages

1. Add destination-level path_mapping config validation with preserve_relative as the default and fixed as the new mode.
2. Refactor run planning so destination bundle path mapping is destination-local and can be evaluated before writes.
3. Add fixed-destination newest selection so each fixed destination receives only the newest discovered source bundle.
4. Integrate fixed mapping with destination comparison, managed deletion, force replacement, and dry-run reporting.
5. Update local, SSH, and S3 tests to verify fixed root behavior under each backend root model.
6. Update current-behavior documentation after implementation.

Testing Expectations

Suggested coverage:

• omitted path_mapping preserves existing behavior;
• preserve_relative explicitly matches existing behavior;
• fixed publishes outputs and .distributor.json at the backend root;
• fixed destinations select only the newest discovered bundle;
• fixed destination tie-break behavior is deterministic;
• older discovered bundles are not planned or written to fixed destinations;
• newer source replaces older managed fixed destination state;
• older source skips newer fixed destination state;
• unmanaged non-empty fixed destination fails without force;
• dry-run reports fixed path mapping, candidate count, selected bundle, and destructive replacement warnings clearly;
• --force remains bounded when fixed mapping targets the backend root;
• S3 fixed mapping respects bucket plus prefix as backend root.

Documentation Updates After Implementation

• Update docs/config.md with path_mapping.
• Update docs/operations.md with archive-plus-latest fan-out examples.
• Update docs/cli.md if dry-run output gains fixed-path indicators.
• Add an environment-safe example only after behavior is implemented.

Keep this roadmap under docs/roadmap/ until implemented.

Decisions

• --force is allowed when fixed mapping targets the backend root, but dry-run and run output must make the fixed-root replacement explicit and force remains bounded to the configured backend root.
• Fixed destinations publish only the newest discovered bundle, rather than processing every discovered bundle and letting later writes replace earlier writes.
• Dry-run includes an extra warning or summary count for fixed-path destructive replacements.

Future Work

• Add richer source selection policies if deployments need something other than newest-by-created for fixed destinations.
• Consider stricter ambiguity handling for equal latest timestamps if real producer workflows make timestamp ties common.
• Add higher-level status or approval workflows for fixed-root replacements if dry-run output is not enough operational protection.