Add implementation roadmap for the next feature set
This commit is contained in:
212
docs/roadmap/latest_paths.md
Normal file
212
docs/roadmap/latest_paths.md
Normal file
@@ -0,0 +1,212 @@
|
||||
# Roadmap: Latest Path Destinations
|
||||
|
||||
## Purpose
|
||||
|
||||
Support stable "latest" publication paths as normal fan-out destinations.
|
||||
|
||||
A latest destination republishes the current winning source bundle to a fixed
|
||||
destination path such as `/weather/latest/`, while archive destinations preserve
|
||||
source-relative bundle paths such as `/weather/daily/brentwood/2026-06-01/`.
|
||||
|
||||
This feature should be implemented before link generation so URL construction
|
||||
can use final destination path semantics. It is operationally destructive and
|
||||
benefits from index-mode HTML, although it must not require index mode or link
|
||||
generation.
|
||||
|
||||
## Current Implementation Grounding
|
||||
|
||||
Current run behavior computes the destination bundle path as the source bundle
|
||||
path relative to the source root. Publication then writes selected outputs and
|
||||
`.distributor.json` below that destination bundle path.
|
||||
|
||||
Backend roots differ by backend:
|
||||
|
||||
- local and SSH/SFTP use configured `path` as the backend root;
|
||||
- S3 uses configured bucket plus optional `prefix` as the backend root.
|
||||
|
||||
Destination comparison uses `.distributor.json` at the destination bundle path.
|
||||
Replacement and skip decisions are based on the normalized source manifest
|
||||
recorded in destination state. Unmanaged non-empty destinations fail unless
|
||||
explicit force behavior is requested.
|
||||
|
||||
## Goals
|
||||
|
||||
- Add destination-local path mapping.
|
||||
- Preserve source-relative destination paths as the default.
|
||||
- Add fixed path mapping for latest-style destinations.
|
||||
- For fixed destinations, select only the newest discovered source bundle for
|
||||
publication.
|
||||
- Make fixed mapping work for local, SSH/SFTP, and S3 backends.
|
||||
- Continue using normal destination state comparison and replacement rules.
|
||||
- Keep link generation and index-mode HTML complementary, not required.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not add symlink-based latest behavior.
|
||||
- Do not add feed or index generation.
|
||||
- Do not change source manifest schema.
|
||||
- Do not make producers responsible for latest publication.
|
||||
- Do not add domain-specific latest selection rules.
|
||||
|
||||
## Configuration Shape
|
||||
|
||||
Add destination-level path mapping:
|
||||
|
||||
```yaml
|
||||
path_mapping:
|
||||
mode: preserve_relative
|
||||
```
|
||||
|
||||
Modes:
|
||||
|
||||
- `preserve_relative`: existing/default behavior.
|
||||
- `fixed`: publish selected outputs directly at the destination backend root.
|
||||
|
||||
Example fixed destination:
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: latest-html
|
||||
backend: ssh
|
||||
host: web.example.com
|
||||
user: deploy
|
||||
path: /srv/www/weather/latest
|
||||
path_mapping:
|
||||
mode: fixed
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
```
|
||||
|
||||
For S3, the fixed destination root is the configured bucket plus `prefix`.
|
||||
|
||||
## Path Mapping Semantics
|
||||
|
||||
`preserve_relative`:
|
||||
|
||||
```text
|
||||
destination bundle path = source-root-relative bundle path
|
||||
```
|
||||
|
||||
`fixed`:
|
||||
|
||||
```text
|
||||
destination bundle path = ""
|
||||
```
|
||||
|
||||
That empty logical destination bundle path means the destination backend root.
|
||||
Storage writes still use normal backend-rooted logical paths for outputs and
|
||||
state.
|
||||
|
||||
## Newest Bundle Selection
|
||||
|
||||
`preserve_relative` destinations should continue publishing every selected
|
||||
source bundle independently.
|
||||
|
||||
`fixed` destinations should publish only one bundle per run: the newest
|
||||
discovered source bundle selected for that destination. Newest selection should
|
||||
be deterministic:
|
||||
|
||||
1. choose the bundle with the greatest source manifest `created` timestamp;
|
||||
2. if multiple bundles have the same greatest `created` timestamp, use the
|
||||
source-root-relative bundle path ascending as a tie-breaker.
|
||||
|
||||
The selected bundle then uses the existing destination comparison, skip,
|
||||
replacement, cleanup, and force rules. Older discovered bundles should not be
|
||||
planned or written to that fixed destination in the same run.
|
||||
|
||||
Dry-run output should identify the number of fixed-destination candidates and
|
||||
the selected source bundle so operators can see which bundle would become
|
||||
latest.
|
||||
|
||||
## Safety Rules
|
||||
|
||||
Fixed path mapping is more destructive than archive-style publication because
|
||||
newer bundles replace prior contents at the same destination path.
|
||||
|
||||
Required behavior:
|
||||
|
||||
- dry-run output must identify fixed path mapping and planned replacement;
|
||||
- dry-run output must include an extra fixed-path warning or summary count when
|
||||
destructive replacement is possible;
|
||||
- replacement should delete only managed outputs recorded in valid destination
|
||||
state when possible;
|
||||
- unmanaged non-empty fixed destinations must fail unless force is explicitly
|
||||
requested;
|
||||
- `--force` may be used when fixed mapping targets the backend root, but force
|
||||
replacement must remain bounded to the configured destination backend root and
|
||||
must never delete above it;
|
||||
- fixed mapping to the backend root requires especially clear dry-run reporting.
|
||||
|
||||
## Relationship To Other Roadmaps
|
||||
|
||||
HTML index mode is useful for fixed web destinations because `index.html`
|
||||
supports stable directory-style URLs, but sidecar HTML and source-only outputs
|
||||
should still work.
|
||||
|
||||
Link generation should use fixed path semantics so latest URLs are based on the
|
||||
fixed destination root, not the original source-relative archive path.
|
||||
|
||||
Producer manifest creation and remote validation are independent of destination
|
||||
path mapping.
|
||||
|
||||
## Implementation Stages
|
||||
|
||||
1. Add destination-level `path_mapping` config validation with
|
||||
`preserve_relative` as the default and `fixed` as the new mode.
|
||||
2. Refactor run planning so destination bundle path mapping is destination-local
|
||||
and can be evaluated before writes.
|
||||
3. Add fixed-destination newest selection so each fixed destination receives
|
||||
only the newest discovered source bundle.
|
||||
4. Integrate fixed mapping with destination comparison, managed deletion, force
|
||||
replacement, and dry-run reporting.
|
||||
5. Update local, SSH, and S3 tests to verify fixed root behavior under each
|
||||
backend root model.
|
||||
6. Update current-behavior documentation after implementation.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
Suggested coverage:
|
||||
|
||||
- omitted `path_mapping` preserves existing behavior;
|
||||
- `preserve_relative` explicitly matches existing behavior;
|
||||
- `fixed` publishes outputs and `.distributor.json` at the backend root;
|
||||
- fixed destinations select only the newest discovered bundle;
|
||||
- fixed destination tie-break behavior is deterministic;
|
||||
- older discovered bundles are not planned or written to fixed destinations;
|
||||
- newer source replaces older managed fixed destination state;
|
||||
- older source skips newer fixed destination state;
|
||||
- unmanaged non-empty fixed destination fails without force;
|
||||
- dry-run reports fixed path mapping, candidate count, selected bundle, and
|
||||
destructive replacement warnings clearly;
|
||||
- `--force` remains bounded when fixed mapping targets the backend root;
|
||||
- S3 fixed mapping respects bucket plus prefix as backend root.
|
||||
|
||||
## Documentation Updates After Implementation
|
||||
|
||||
- Update `docs/config.md` with `path_mapping`.
|
||||
- Update `docs/operations.md` with archive-plus-latest fan-out examples.
|
||||
- Update `docs/cli.md` if dry-run output gains fixed-path indicators.
|
||||
- Add an environment-safe example only after behavior is implemented.
|
||||
|
||||
Keep this roadmap under `docs/roadmap/` until implemented.
|
||||
|
||||
## Decisions
|
||||
|
||||
- `--force` is allowed when fixed mapping targets the backend root, but dry-run
|
||||
and run output must make the fixed-root replacement explicit and force remains
|
||||
bounded to the configured backend root.
|
||||
- Fixed destinations publish only the newest discovered bundle, rather than
|
||||
processing every discovered bundle and letting later writes replace earlier
|
||||
writes.
|
||||
- Dry-run includes an extra warning or summary count for fixed-path destructive
|
||||
replacements.
|
||||
|
||||
## Future Work
|
||||
|
||||
- Add richer source selection policies if deployments need something other than
|
||||
newest-by-`created` for fixed destinations.
|
||||
- Consider stricter ambiguity handling for equal latest timestamps if real
|
||||
producer workflows make timestamp ties common.
|
||||
- Add higher-level status or approval workflows for fixed-root replacements if
|
||||
dry-run output is not enough operational protection.
|
||||
Reference in New Issue
Block a user