eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Added four new roadmaps related to state management and a corresponding implementation plan

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-08 13:09:00 -05:00
parent ee6a351960
commit 8f3ef33f18
5 changed files with 1295 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

180
docs/roadmap/reconciliation.md Normal file
View File

@@ -0,0 +1,180 @@
# Reconciliation Roadmap
This document records planned destination reconciliation work that is not part
of the current implementation. Current destination replacement, state, and
recovery behavior is documented in `docs/operations.md`,
`docs/config.md`, and `docs/integrations/destination-state.md`.
This work should be implemented before the shared-root multi-pipeline state
work described in `docs/roadmap/multipipeline.md`. The merge mode described
here is still scoped to one pipeline/destination owner for a destination bundle
path; it does not by itself allow multiple pipelines to share one
`.distributor.json`.
## Destination Reconciliation Modes
Current destination replacement behavior treats each publish as the complete
desired managed output set for the destination bundle path. When a newer source
replaces an older destination state, `distributor` deletes prior managed outputs
that are no longer present in the new plan, then writes the new outputs and
state.
That behavior is useful for snapshot-style producers, but it is not ideal for
incremental producers that publish new reports without retaining prior report
state.
Planned work:
- Add a per-destination reconciliation mode.
- Keep current behavior as the default mode.
- Add a mode that writes new outputs while retaining prior managed outputs that
are not present in the new plan.
- Keep reconciliation mode per destination. Replacement and merge behavior are
destination safety policies, and one pipeline may need both a snapshot-style
`latest` destination and an accumulating `archive` destination from the same
source bundle.
Proposed configuration:
```yaml
destinations:
- id: archive
backend: s3
bucket: reports
prefix: weather/morning/archive
reconciliation:
mode: merge
- id: latest
backend: s3
bucket: reports
prefix: weather/morning/latest
path_mapping:
mode: fixed
reconciliation:
mode: replace
```
Accepted modes:
- `replace`: current behavior. The destination bundle path is reconciled to the
new planned output set. Prior managed outputs not present in the new plan are
removed.
- `merge`: new planned outputs are written into the managed destination set.
Prior managed outputs not present in the new plan are retained.
## State Model
Destination state needs to represent cumulative managed outputs for merge mode.
The current state file already records an `outputs` array, but that array
currently represents the outputs from the most recent publication.
Planned state semantics:
- `source.manifest` continues to record the latest source manifest used for
destination comparison.
- `outputs` records the complete currently managed output set for the
destination bundle path.
- In `replace` mode, the new state `outputs` are exactly the newly planned
outputs.
- In `merge` mode, the new state `outputs` are:
- previous managed outputs not overwritten by the new plan;
- plus newly planned outputs.
- Persist reconciliation metadata so state interpretation remains explicit after
config changes and so later migration, reconcile-state, and prune tooling can
explain why historical outputs were retained, for example:
```json
{
"reconciliation": {
"mode": "merge"
}
}
```
If state schema changes are required, bump the destination state schema version
and keep parsing/validation rules explicit.
## Safety Rules
Merge mode must not become an unsafe overwrite path.
Required safety behavior:
- A new output may overwrite a prior output only when that path is already
recorded as distributor-managed in the existing `.distributor.json`.
- A new output path that exists in storage but is not recorded as managed should
fail as unmanaged content by default.
- Prior managed outputs not included in the new plan are retained in merge mode.
- Prior managed outputs not included in the new plan are deleted in replace
mode.
- Failed-write cleanup should delete only outputs written by the failed attempt,
not retained prior managed outputs.
- Forced replacement behavior remains explicit, bounded, dry-runnable, and
constrained to the destination bundle path.
## Planning And Execution Work
Implementation work:
- Add destination config type `reconciliation.mode`.
- Default `reconciliation.mode` to `replace`.
- Validate mode values as `replace` or `merge`.
- Thread reconciliation mode into publish planning and execution requests.
- During replace execution, preserve current managed cleanup behavior.
- During merge execution:
- inspect existing state outputs before writing;
- allow overwrites only for paths already managed by existing state;
- write new outputs with overwrite enabled only for managed existing paths;
- retain previous managed outputs that are not overwritten;
- write destination state with the cumulative managed output set.
- Ensure generated HTML outputs and copied source outputs use the same
reconciliation semantics.
- Ensure link metadata and primary URL behavior remain deterministic when
retained outputs exist. Derive `links.primary_url` from the newly planned
outputs for the current publish, not from retained historical outputs.
## Testing
Important tests:
- Config defaults `reconciliation.mode` to `replace`.
- Config rejects unknown reconciliation modes.
- Replace mode deletes prior managed outputs omitted from the new bundle.
- Merge mode retains prior managed outputs omitted from the new bundle.
- Merge mode overwrites a path already recorded as managed.
- Merge mode fails when a new output path collides with unmanaged destination
content.
- Merge mode state records the cumulative managed output set.
- Failed merge writes clean up only outputs from the failed attempt.
- Fixed-path destinations support both modes.
- Source-only, generated-HTML-only, and source-plus-HTML publish policies all
honor reconciliation mode.
- S3, SSH/SFTP, local, and fake backend tests cover any backend overwrite or
cleanup behavior that changes.
## Documentation Work
When implemented, update current-behavior docs in the same change:
- `docs/config.md`: document `reconciliation.mode`, defaults, and examples.
- `docs/operations.md`: explain replace versus merge behavior and recovery.
- `docs/integrations/destination-state.md`: document any state schema or
semantic changes.
- `docs/internal/publish.md` and related internal docs: record planning and
execution invariants.
- Relevant examples under `examples/`, especially archive/latest style
configurations.
## Boundaries
- Do not add lifecycle retention policies such as keep-latest-N or delete older
than a duration as part of this work.
- Do not let producers choose reconciliation mode through source manifests or
upload requests.
- Do not weaken unmanaged-content safety checks.
- Do not make merge mode the default; preserve current replacement behavior for
existing configurations.
- Do not claim unmanaged colliding files in merge mode. Claiming unmanaged
content should remain an explicit force or repair workflow, not normal publish
behavior.