181 lines
6.9 KiB
Markdown
181 lines
6.9 KiB
Markdown
# Reconciliation Roadmap
|
|
|
|
This document records planned destination reconciliation work that is not part
|
|
of the current implementation. Current destination replacement, state, and
|
|
recovery behavior is documented in `docs/operations.md`,
|
|
`docs/config.md`, and `docs/integrations/destination-state.md`.
|
|
|
|
This work should be implemented before the shared-root multi-pipeline state
|
|
work described in `docs/roadmap/multipipeline.md`. The merge mode described
|
|
here is still scoped to one pipeline/destination owner for a destination bundle
|
|
path; it does not by itself allow multiple pipelines to share one
|
|
`.distributor.json`.
|
|
|
|
## Destination Reconciliation Modes
|
|
|
|
Current destination replacement behavior treats each publish as the complete
|
|
desired managed output set for the destination bundle path. When a newer source
|
|
replaces an older destination state, `distributor` deletes prior managed outputs
|
|
that are no longer present in the new plan, then writes the new outputs and
|
|
state.
|
|
|
|
That behavior is useful for snapshot-style producers, but it is not ideal for
|
|
incremental producers that publish new reports without retaining prior report
|
|
state.
|
|
|
|
Planned work:
|
|
|
|
- Add a per-destination reconciliation mode.
|
|
- Keep current behavior as the default mode.
|
|
- Add a mode that writes new outputs while retaining prior managed outputs that
|
|
are not present in the new plan.
|
|
- Keep reconciliation mode per destination. Replacement and merge behavior are
|
|
destination safety policies, and one pipeline may need both a snapshot-style
|
|
`latest` destination and an accumulating `archive` destination from the same
|
|
source bundle.
|
|
|
|
Proposed configuration:
|
|
|
|
```yaml
|
|
destinations:
|
|
- id: archive
|
|
backend: s3
|
|
bucket: reports
|
|
prefix: weather/morning/archive
|
|
reconciliation:
|
|
mode: merge
|
|
|
|
- id: latest
|
|
backend: s3
|
|
bucket: reports
|
|
prefix: weather/morning/latest
|
|
path_mapping:
|
|
mode: fixed
|
|
reconciliation:
|
|
mode: replace
|
|
```
|
|
|
|
Accepted modes:
|
|
|
|
- `replace`: current behavior. The destination bundle path is reconciled to the
|
|
new planned output set. Prior managed outputs not present in the new plan are
|
|
removed.
|
|
- `merge`: new planned outputs are written into the managed destination set.
|
|
Prior managed outputs not present in the new plan are retained.
|
|
|
|
## State Model
|
|
|
|
Destination state needs to represent cumulative managed outputs for merge mode.
|
|
The current state file already records an `outputs` array, but that array
|
|
currently represents the outputs from the most recent publication.
|
|
|
|
Planned state semantics:
|
|
|
|
- `source.manifest` continues to record the latest source manifest used for
|
|
destination comparison.
|
|
- `outputs` records the complete currently managed output set for the
|
|
destination bundle path.
|
|
- In `replace` mode, the new state `outputs` are exactly the newly planned
|
|
outputs.
|
|
- In `merge` mode, the new state `outputs` are:
|
|
- previous managed outputs not overwritten by the new plan;
|
|
- plus newly planned outputs.
|
|
- Persist reconciliation metadata so state interpretation remains explicit after
|
|
config changes and so later migration, reconcile-state, and prune tooling can
|
|
explain why historical outputs were retained, for example:
|
|
|
|
```json
|
|
{
|
|
"reconciliation": {
|
|
"mode": "merge"
|
|
}
|
|
}
|
|
```
|
|
|
|
If state schema changes are required, bump the destination state schema version
|
|
and keep parsing/validation rules explicit.
|
|
|
|
## Safety Rules
|
|
|
|
Merge mode must not become an unsafe overwrite path.
|
|
|
|
Required safety behavior:
|
|
|
|
- A new output may overwrite a prior output only when that path is already
|
|
recorded as distributor-managed in the existing `.distributor.json`.
|
|
- A new output path that exists in storage but is not recorded as managed should
|
|
fail as unmanaged content by default.
|
|
- Prior managed outputs not included in the new plan are retained in merge mode.
|
|
- Prior managed outputs not included in the new plan are deleted in replace
|
|
mode.
|
|
- Failed-write cleanup should delete only outputs written by the failed attempt,
|
|
not retained prior managed outputs.
|
|
- Forced replacement behavior remains explicit, bounded, dry-runnable, and
|
|
constrained to the destination bundle path.
|
|
|
|
## Planning And Execution Work
|
|
|
|
Implementation work:
|
|
|
|
- Add destination config type `reconciliation.mode`.
|
|
- Default `reconciliation.mode` to `replace`.
|
|
- Validate mode values as `replace` or `merge`.
|
|
- Thread reconciliation mode into publish planning and execution requests.
|
|
- During replace execution, preserve current managed cleanup behavior.
|
|
- During merge execution:
|
|
- inspect existing state outputs before writing;
|
|
- allow overwrites only for paths already managed by existing state;
|
|
- write new outputs with overwrite enabled only for managed existing paths;
|
|
- retain previous managed outputs that are not overwritten;
|
|
- write destination state with the cumulative managed output set.
|
|
- Ensure generated HTML outputs and copied source outputs use the same
|
|
reconciliation semantics.
|
|
- Ensure link metadata and primary URL behavior remain deterministic when
|
|
retained outputs exist. Derive `links.primary_url` from the newly planned
|
|
outputs for the current publish, not from retained historical outputs.
|
|
|
|
## Testing
|
|
|
|
Important tests:
|
|
|
|
- Config defaults `reconciliation.mode` to `replace`.
|
|
- Config rejects unknown reconciliation modes.
|
|
- Replace mode deletes prior managed outputs omitted from the new bundle.
|
|
- Merge mode retains prior managed outputs omitted from the new bundle.
|
|
- Merge mode overwrites a path already recorded as managed.
|
|
- Merge mode fails when a new output path collides with unmanaged destination
|
|
content.
|
|
- Merge mode state records the cumulative managed output set.
|
|
- Failed merge writes clean up only outputs from the failed attempt.
|
|
- Fixed-path destinations support both modes.
|
|
- Source-only, generated-HTML-only, and source-plus-HTML publish policies all
|
|
honor reconciliation mode.
|
|
- S3, SSH/SFTP, local, and fake backend tests cover any backend overwrite or
|
|
cleanup behavior that changes.
|
|
|
|
## Documentation Work
|
|
|
|
When implemented, update current-behavior docs in the same change:
|
|
|
|
- `docs/config.md`: document `reconciliation.mode`, defaults, and examples.
|
|
- `docs/operations.md`: explain replace versus merge behavior and recovery.
|
|
- `docs/integrations/destination-state.md`: document any state schema or
|
|
semantic changes.
|
|
- `docs/internal/publish.md` and related internal docs: record planning and
|
|
execution invariants.
|
|
- Relevant examples under `examples/`, especially archive/latest style
|
|
configurations.
|
|
|
|
## Boundaries
|
|
|
|
- Do not add lifecycle retention policies such as keep-latest-N or delete older
|
|
than a duration as part of this work.
|
|
- Do not let producers choose reconciliation mode through source manifests or
|
|
upload requests.
|
|
- Do not weaken unmanaged-content safety checks.
|
|
- Do not make merge mode the default; preserve current replacement behavior for
|
|
existing configurations.
|
|
- Do not claim unmanaged colliding files in merge mode. Claiming unmanaged
|
|
content should remain an explicit force or repair workflow, not normal publish
|
|
behavior.
|