196 lines
8.4 KiB
Markdown
196 lines
8.4 KiB
Markdown
# Multi-Pipeline Shared Root Roadmap
|
|
|
|
This document records planned shared-root destination state work that is not
|
|
part of the current implementation. Current destination state behavior is
|
|
documented in `docs/integrations/destination-state.md`.
|
|
|
|
This work should not be implemented until the reconciliation work in
|
|
`docs/roadmap/reconciliation.md` is present and complete. Shared-root
|
|
multi-pipeline publishing should then be implemented before reconcile-state
|
|
tooling (`docs/roadmap/reconcile-state.md`) and pruning
|
|
(`docs/roadmap/prune.md`) because both tools should operate on the final shared
|
|
state model.
|
|
|
|
## Problem
|
|
|
|
Current `.distributor.json` state is a single-owner destination bundle record.
|
|
It stores one `pipeline_id`, one `destination_id`, one latest source manifest,
|
|
and one output set. That works when each pipeline writes to a separate
|
|
destination bundle path.
|
|
|
|
It does not work well when multiple pipelines intentionally publish into one
|
|
shared destination root, for example:
|
|
|
|
```text
|
|
2026-06-08/daily/report.md
|
|
2026-06-08/event/watch-123.md
|
|
```
|
|
|
|
If both pipelines resolve to the same destination bundle path, the second
|
|
pipeline encounters a state identity mismatch even when its output paths are
|
|
disjoint. Separating roots avoids the conflict, but can make retention,
|
|
cleanup, linking, and human browsing less natural.
|
|
|
|
## Goal
|
|
|
|
Support multiple configured pipelines writing to one shared destination root
|
|
while preserving distributor ownership, unmanaged-content safety, dry-run
|
|
behavior, and deterministic conflict handling.
|
|
|
|
The desired shape is a collection-oriented `.distributor.json` that records
|
|
managed outputs from multiple owners under one state file. Producers still
|
|
submit source bundles only. Producers must not choose destinations,
|
|
reconciliation mode, retention policy, transforms, links, or storage backends.
|
|
|
|
Shared-root behavior should be an explicit per-destination state mode, for
|
|
example `state.mode: single_owner|shared_root` or a similarly named field under
|
|
the destination. Shared-root behavior changes the meaning of `.distributor.json`,
|
|
so it should be visible at the destination boundary rather than inferred from
|
|
other configured pipelines.
|
|
|
|
## State Model
|
|
|
|
Add a new destination state schema that can represent a managed collection
|
|
rather than one latest source bundle.
|
|
|
|
State shape:
|
|
|
|
- top-level `schema_version`;
|
|
- top-level `distributor_version`;
|
|
- top-level `created_at` for the first time this managed root was created;
|
|
- top-level `updated_at` for the last successful state update;
|
|
- top-level collection metadata identifying the state as shared-root capable;
|
|
- an `outputs` array containing all managed output records in the shared root;
|
|
- per-output owner metadata:
|
|
- `pipeline_id`;
|
|
- `destination_id`;
|
|
- source manifest id;
|
|
- source manifest digest;
|
|
- source manifest created timestamp;
|
|
- output kind, source path, transform, URL, SHA-256, and size;
|
|
- `created_at` for first publication of that output path;
|
|
- `updated_at` for the most recent write of that output path.
|
|
- optional source-manifest history keyed by source digest, if compact per-output
|
|
source identity fields are not enough for later diagnostics or tooling.
|
|
|
|
Output ownership is keyed by both `pipeline_id` and `destination_id`. The
|
|
architecture treats destinations as independent policy boundaries, and the same
|
|
pipeline may publish different output sets to different destinations.
|
|
|
|
The state should be able to answer these questions without scanning the whole
|
|
destination:
|
|
|
|
- Which paths are distributor-managed?
|
|
- Which pipeline/destination/source currently owns each managed path?
|
|
- Which outputs belong to a specific pipeline/destination publish operation?
|
|
- Which outputs are old enough, or least recently updated enough, for later
|
|
pruning decisions?
|
|
- Which paths are absent from storage and should be removed by reconcile-state
|
|
tooling?
|
|
|
|
## Ownership And Conflict Rules
|
|
|
|
Shared-root support must keep ownership explicit.
|
|
|
|
Required rules:
|
|
|
|
- A publish may write only output paths planned from its configured pipeline and
|
|
destination.
|
|
- A publish may overwrite an existing managed output path only when ownership
|
|
rules allow it.
|
|
- A new output path that exists in storage but is not recorded in
|
|
`.distributor.json` remains unmanaged content and fails by default.
|
|
- Disjoint outputs from different pipelines can coexist in one state file.
|
|
- State updates must preserve unrelated managed outputs from other pipelines.
|
|
- Failed-write cleanup must delete only paths written by the failed attempt.
|
|
- Forced replacement must remain explicit, dry-runnable, and bounded to the
|
|
configured destination root.
|
|
|
|
Overwrite rule: an output path may be overwritten by the same `pipeline_id` and
|
|
`destination_id`; a different pipeline or destination writing the same path
|
|
fails as an ownership conflict unless a later explicit operator takeover
|
|
workflow is implemented.
|
|
|
|
## Planning And Execution Work
|
|
|
|
Implementation work:
|
|
|
|
- Introduce a new state schema and parser that can represent shared ownership.
|
|
- Keep current schema parsing available long enough to migrate or reject old
|
|
state with clear errors.
|
|
- Add explicit per-destination configuration that opts a destination into
|
|
shared-root behavior.
|
|
- Thread shared-root state semantics through destination inspection,
|
|
comparison, publish planning, and execution.
|
|
- Make comparison operate on the subset of state owned by the current
|
|
pipeline/destination where appropriate.
|
|
- Make output collision detection check both newly planned outputs and existing
|
|
managed outputs from other owners.
|
|
- Write state atomically with preserved unrelated owners and updated records for
|
|
the current publish.
|
|
- Preserve merge and replace semantics within the current owner scope:
|
|
- owner-scoped `replace` removes prior outputs for that owner that are absent
|
|
from the new plan;
|
|
- owner-scoped `merge` retains prior outputs for that owner that are absent
|
|
from the new plan.
|
|
- Ensure local, SSH/SFTP, S3, and fake backends support the necessary state read,
|
|
write, overwrite, and cleanup paths.
|
|
|
|
## Migration And Compatibility
|
|
|
|
The initial implementation should prefer explicit safety over automatic
|
|
conversion.
|
|
|
|
Migration behavior:
|
|
|
|
- A destination configured for shared-root behavior may read current single-owner
|
|
state and convert it only when the existing state identity matches the current
|
|
pipeline/destination or when an explicit migration command/flag is used.
|
|
- A destination not configured for shared-root behavior continues to use the
|
|
current single-owner state semantics.
|
|
- Invalid or ambiguous state fails with an actionable error and a dry-run report
|
|
path.
|
|
|
|
## Testing
|
|
|
|
Important tests:
|
|
|
|
- Two pipelines publish disjoint paths into the same destination root.
|
|
- One pipeline can replace its own older output without deleting another
|
|
pipeline's output.
|
|
- One pipeline can merge new outputs while retaining its prior outputs and
|
|
another pipeline's outputs.
|
|
- A pipeline cannot silently overwrite another pipeline's managed path.
|
|
- A pipeline cannot claim unmanaged storage content by default.
|
|
- State timestamps are set on creation and updated on subsequent writes.
|
|
- Per-output timestamps preserve original `created_at` and update `updated_at`
|
|
only when that output path is rewritten.
|
|
- Failed writes clean up only attempted outputs.
|
|
- Dry-run reports shared-root actions without writing outputs or state.
|
|
- Forced replacement remains bounded and explicitly reported.
|
|
- Current single-owner state behavior remains covered for non-shared
|
|
destinations.
|
|
|
|
## Documentation Work
|
|
|
|
When implemented, update current-behavior docs in the same change:
|
|
|
|
- `docs/integrations/destination-state.md`: document the new schema and
|
|
ownership rules.
|
|
- `docs/config.md`: document the shared-root opt-in configuration.
|
|
- `docs/operations.md`: explain shared-root publishing, conflicts, and recovery.
|
|
- `docs/troubleshooting.md`: add shared-root ownership conflict entries.
|
|
- `docs/internal/state.md` and `docs/internal/publish.md`: record the new
|
|
planning and state invariants.
|
|
- Maintained examples that demonstrate date-first shared roots only after the
|
|
behavior is implemented.
|
|
|
|
## Boundaries
|
|
|
|
- Do not let producers route to destinations or choose shared-root behavior.
|
|
- Do not add prune behavior in this work.
|
|
- Do not add broad unmanaged-content adoption in this work.
|
|
- Do not make shared-root behavior implicit for all destinations.
|
|
- Do not use storage listings as the primary source of truth during normal
|
|
publishing; `.distributor.json` remains the managed ownership index.
|