distributor/docs/roadmap/multipipeline.md

Multi-Pipeline Shared Root Roadmap

This document records planned shared-root destination state work that is not part of the current implementation. Current destination state behavior is documented in docs/integrations/destination-state.md.

This work should not be implemented until the reconciliation work in docs/roadmap/reconciliation.md is present and complete. Shared-root multi-pipeline publishing should then be implemented before reconcile-state tooling (docs/roadmap/reconcile-state.md) and pruning (docs/roadmap/prune.md) because both tools should operate on the final shared state model.

Problem

Current .distributor.json state is a single-owner destination bundle record. It stores one pipeline_id, one destination_id, one latest source manifest, and one output set. That works when each pipeline writes to a separate destination bundle path.

It does not work well when multiple pipelines intentionally publish into one shared destination root, for example:

2026-06-08/daily/report.md
2026-06-08/event/watch-123.md

If both pipelines resolve to the same destination bundle path, the second pipeline encounters a state identity mismatch even when its output paths are disjoint. Separating roots avoids the conflict, but can make retention, cleanup, linking, and human browsing less natural.

Goal

Support multiple configured pipelines writing to one shared destination root while preserving distributor ownership, unmanaged-content safety, dry-run behavior, and deterministic conflict handling.

The desired shape is a collection-oriented .distributor.json that records managed outputs from multiple owners under one state file. Producers still submit source bundles only. Producers must not choose destinations, reconciliation mode, retention policy, transforms, links, or storage backends.

Shared-root behavior should be an explicit per-destination state mode, for example state.mode: single_owner|shared_root or a similarly named field under the destination. Shared-root behavior changes the meaning of .distributor.json, so it should be visible at the destination boundary rather than inferred from other configured pipelines.

State Model

Add a new destination state schema that can represent a managed collection rather than one latest source bundle.

State shape:

• top-level schema_version;
• top-level distributor_version;
• top-level created_at for the first time this managed root was created;
• top-level updated_at for the last successful state update;
• top-level collection metadata identifying the state as shared-root capable;
• an outputs array containing all managed output records in the shared root;
• per-output owner metadata:
  • pipeline_id;
  • destination_id;
  • source manifest id;
  • source manifest digest;
  • source manifest created timestamp;
  • output kind, source path, transform, URL, SHA-256, and size;
  • created_at for first publication of that output path;
  • updated_at for the most recent write of that output path.
• optional source-manifest history keyed by source digest, if compact per-output source identity fields are not enough for later diagnostics or tooling.

Output ownership is keyed by both pipeline_id and destination_id. The architecture treats destinations as independent policy boundaries, and the same pipeline may publish different output sets to different destinations.

The state should be able to answer these questions without scanning the whole destination:

• Which paths are distributor-managed?
• Which pipeline/destination/source currently owns each managed path?
• Which outputs belong to a specific pipeline/destination publish operation?
• Which outputs are old enough, or least recently updated enough, for later pruning decisions?
• Which paths are absent from storage and should be removed by reconcile-state tooling?

Ownership And Conflict Rules

Shared-root support must keep ownership explicit.

Required rules:

• A publish may write only output paths planned from its configured pipeline and destination.
• A publish may overwrite an existing managed output path only when ownership rules allow it.
• A new output path that exists in storage but is not recorded in .distributor.json remains unmanaged content and fails by default.
• Disjoint outputs from different pipelines can coexist in one state file.
• State updates must preserve unrelated managed outputs from other pipelines.
• Failed-write cleanup must delete only paths written by the failed attempt.
• Forced replacement must remain explicit, dry-runnable, and bounded to the configured destination root.

Overwrite rule: an output path may be overwritten by the same pipeline_id and destination_id; a different pipeline or destination writing the same path fails as an ownership conflict unless a later explicit operator takeover workflow is implemented.

Planning And Execution Work

Implementation work:

• Introduce a new state schema and parser that can represent shared ownership.
• Keep current schema parsing available long enough to migrate or reject old state with clear errors.
• Add explicit per-destination configuration that opts a destination into shared-root behavior.
• Thread shared-root state semantics through destination inspection, comparison, publish planning, and execution.
• Make comparison operate on the subset of state owned by the current pipeline/destination where appropriate.
• Make output collision detection check both newly planned outputs and existing managed outputs from other owners.
• Write state atomically with preserved unrelated owners and updated records for the current publish.
• Preserve merge and replace semantics within the current owner scope:
  • owner-scoped replace removes prior outputs for that owner that are absent from the new plan;
  • owner-scoped merge retains prior outputs for that owner that are absent from the new plan.
• Ensure local, SSH/SFTP, S3, and fake backends support the necessary state read, write, overwrite, and cleanup paths.

Migration And Compatibility

The initial implementation should prefer explicit safety over automatic conversion.

Migration behavior:

• A destination configured for shared-root behavior may read current single-owner state and convert it only when the existing state identity matches the current pipeline/destination or when an explicit migration command/flag is used.
• A destination not configured for shared-root behavior continues to use the current single-owner state semantics.
• Invalid or ambiguous state fails with an actionable error and a dry-run report path.

Testing

Important tests:

• Two pipelines publish disjoint paths into the same destination root.
• One pipeline can replace its own older output without deleting another pipeline's output.
• One pipeline can merge new outputs while retaining its prior outputs and another pipeline's outputs.
• A pipeline cannot silently overwrite another pipeline's managed path.
• A pipeline cannot claim unmanaged storage content by default.
• State timestamps are set on creation and updated on subsequent writes.
• Per-output timestamps preserve original created_at and update updated_at only when that output path is rewritten.
• Failed writes clean up only attempted outputs.
• Dry-run reports shared-root actions without writing outputs or state.
• Forced replacement remains bounded and explicitly reported.
• Current single-owner state behavior remains covered for non-shared destinations.

Documentation Work

When implemented, update current-behavior docs in the same change:

• docs/integrations/destination-state.md: document the new schema and ownership rules.
• docs/config.md: document the shared-root opt-in configuration.
• docs/operations.md: explain shared-root publishing, conflicts, and recovery.
• docs/troubleshooting.md: add shared-root ownership conflict entries.
• docs/internal/state.md and docs/internal/publish.md: record the new planning and state invariants.
• Maintained examples that demonstrate date-first shared roots only after the behavior is implemented.

Boundaries

• Do not let producers route to destinations or choose shared-root behavior.
• Do not add prune behavior in this work.
• Do not add broad unmanaged-content adoption in this work.
• Do not make shared-root behavior implicit for all destinations.
• Do not use storage listings as the primary source of truth during normal publishing; .distributor.json remains the managed ownership index.