diff --git a/README.md b/README.md index 15584c2..b741e1a 100644 --- a/README.md +++ b/README.md @@ -10,10 +10,11 @@ Run the maintained local example: go run ./cmd/distributor run --config examples/local-publish.yml ``` -Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, write, parse, and validate local source bundles with the same manifest contract used by the CLI. +Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, write, parse, and validate local source bundles with the same manifest contract used by the CLI. See [Source bundle contract](docs/integrations/source-bundle.md). - [CLI reference](docs/cli.md) - [Configuration reference](docs/config.md) - [Operations guide](docs/operations.md) - [Troubleshooting](docs/troubleshooting.md) +- [Integration contracts](docs/integrations/source-bundle.md) - [Development architecture](docs/policy/architecture.md) diff --git a/docs/cli.md b/docs/cli.md index f22b056..f164fee 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -2,7 +2,7 @@ Audience: operators, integrators, and developers who run `distributor` from a shell or automation. -This document is the canonical command and flag reference. Configuration schema details live in [Configuration](config.md), operational recovery guidance lives in [Operations](operations.md), and failure diagnosis lives in [Troubleshooting](troubleshooting.md). +This document is the canonical command and flag reference. Configuration schema details live in [Configuration](config.md), operational recovery guidance lives in [Operations](operations.md), failure diagnosis lives in [Troubleshooting](troubleshooting.md), and external contracts live under [Integrations](integrations/source-bundle.md). ## Shortest Useful Command diff --git a/docs/config.md b/docs/config.md index 08fef31..e7752e0 100644 --- a/docs/config.md +++ b/docs/config.md @@ -2,7 +2,7 @@ Audience: administrators, operators, and advanced users who write YAML configuration for `distributor`. -This document is the canonical user-facing configuration reference. CLI syntax lives in [CLI](cli.md), operating procedures live in [Operations](operations.md), and symptom-oriented recovery lives in [Troubleshooting](troubleshooting.md). +This document is the canonical user-facing configuration reference. CLI syntax lives in [CLI](cli.md), operating procedures live in [Operations](operations.md), symptom-oriented recovery lives in [Troubleshooting](troubleshooting.md), and external contracts live under [Integrations](integrations/source-bundle.md). ## Config File Loading @@ -153,7 +153,7 @@ path: /srv/distributor/archive ### SSH/SFTP Backend -SSH backends use native SFTP and can be used as sources and destinations. +SSH backends use native SFTP and can be used as sources and destinations. Adapter protocol behavior is documented in [SSH/SFTP Integration](integrations/ssh-sftp.md). ```yaml backend: ssh @@ -185,7 +185,7 @@ Authentication uses SSH agent identities when `SSH_AUTH_SOCK` is available, then ### S3-Compatible Backend -S3 backends can be used as sources and destinations. +S3 backends can be used as sources and destinations. Adapter protocol behavior is documented in [S3-Compatible Storage Integration](integrations/s3.md). ```yaml backend: s3 @@ -212,7 +212,7 @@ The S3 credential variable names must either both be configured or both be omitt ### HTTP Upload Source Backend -HTTP upload backends are valid only as pipeline sources and are served by `distributor serve`. +HTTP upload backends are valid only as pipeline sources and are served by `distributor serve`. The API contract is documented in [HTTP Upload API Contract](integrations/http-upload.md). ```yaml backend: http_upload @@ -235,7 +235,7 @@ validation: - `validation.on_digest_mismatch`: optional. Default and only accepted value: `fail`. -Source bundle digest mismatches fail validation before destination writes occur. +Source bundle digest mismatches fail validation before destination writes occur. The manifest file-format contract is documented in [Source Bundle Contract](integrations/source-bundle.md). ## Destination Fields @@ -330,7 +330,7 @@ links: - `links.base_url`: required when `links` is present. It must be an absolute `http` or `https` URL with a host and no query string or fragment. - `links.primary`: optional. Accepted values are `auto`, `html`, and `source`; default is `auto` when `links` is present. -`distributor` does not infer public URLs from backend config. Output URLs are built from `links.base_url`, the destination bundle path, and output paths using URL path semantics. `index.html` outputs produce directory-style URLs that omit the filename. +`distributor` does not infer public URLs from backend config. Destination state URL fields are documented in [Destination State Contract](integrations/destination-state.md). Output URLs are built from `links.base_url`, the destination bundle path, and output paths using URL path semantics. `index.html` outputs produce directory-style URLs that omit the filename. Primary URL policies: diff --git a/docs/integrations/destination-state.md b/docs/integrations/destination-state.md new file mode 100644 index 0000000..3de3743 --- /dev/null +++ b/docs/integrations/destination-state.md @@ -0,0 +1,99 @@ +# Destination State Contract + +Audience: operators, integrators, and maintainers who inspect or reason about destination `.distributor.json` files. + +Each managed destination bundle path contains `.distributor.json`. This file is the destination sentinel and state record used for comparison, skip, replacement, and recovery decisions. + +## State Schema + +Current schema version: `1`. + +```json +{ + "schema_version": 1, + "distributor_version": "dev", + "pipeline_id": "reports", + "destination_id": "archive", + "published_at": "2026-06-04T12:00:00Z", + "source": { + "manifest": { + "schema_version": 1, + "id": "reports.example.2026-06-04", + "digest": "sha256:...", + "created": "2026-06-04T11:55:00Z", + "files": [ + {"path": "report.md", "sha256": "sha256:...", "size": 1234} + ] + } + }, + "links": { + "primary_url": "https://reports.example.com/archive/report.html" + }, + "outputs": [ + { + "path": "report.html", + "kind": "generated", + "source_path": "report.md", + "transform": "markdown_to_html", + "url": "https://reports.example.com/archive/report.html", + "sha256": "sha256:...", + "size": 2345 + } + ] +} +``` + +Required fields: + +- `schema_version`: must be `1`. +- `pipeline_id`: configured pipeline id that wrote the state. +- `destination_id`: configured destination id that wrote the state. +- `published_at`: RFC3339 publication timestamp. +- `source.manifest`: embedded source bundle manifest. +- `outputs`: output records array; empty is allowed, but the field is required. + +Optional fields: + +- `distributor_version`: application version string when available. +- `links.primary_url`: absolute HTTP or HTTPS URL selected by destination link policy. + +## Output Records + +Each output record has: + +- `path`: destination-relative output path. +- `kind`: `source` or `generated`. +- `source_path`: source manifest path used for the output. +- `transform`: required for `generated` outputs; omitted for copied source outputs. +- `url`: optional absolute HTTP or HTTPS URL for the output. +- `sha256`: lowercase `sha256:<64 hex>` digest of the output bytes. +- `size`: output byte size, zero or greater. + +Output paths must be unique and use clean relative slash-separated path rules. + +## Comparison Semantics + +`distributor` compares the current source manifest to destination state before writing: + +- No state and no content: publish new outputs. +- No state and existing content: treat the destination as unmanaged. +- Matching embedded source manifest: skip. +- Same source id with older `created`: replace if policy allows. +- Same source id with newer `created`: skip by default. +- Same source id and same `created` with different digest: conflict. +- Different source id, pipeline id, or destination id: conflict. +- Invalid state JSON or invalid state fields: conflict. + +Normal replacement deletes only managed output paths recorded in `outputs` plus `.distributor.json`. Forced replacement deletes the bounded destination bundle path. + +## Boundaries + +Destination state is internal managed state written by `distributor`. Operators may inspect it during recovery, but normal workflows should not edit it by hand. Source `manifest.json` is not copied as destination state. + +## Tests + +Before changing this contract, inspect and run: + +```sh +go test ./internal/state ./internal/publish +``` diff --git a/docs/integrations/http-upload.md b/docs/integrations/http-upload.md new file mode 100644 index 0000000..ed7911e --- /dev/null +++ b/docs/integrations/http-upload.md @@ -0,0 +1,108 @@ +# HTTP Upload API Contract + +Audience: producers, operators, and maintainers integrating with `distributor serve`. + +`distributor serve` exposes a local HTTP upload API for pipelines whose source backend is `http_upload`. Each bearer token maps to exactly one configured pipeline. + +## Authentication + +Uploads authenticate with: + +```text +Authorization: Bearer +``` + +Token values are resolved from the configured `source.token_env` through the process environment or `secrets.directory`. Tokens are not configured as YAML literal values. + +Requests that include `pipeline` or `pipeline_id` query parameters are rejected. The bearer token selects the pipeline. + +## Endpoints + +### `GET /healthz` + +Returns `200 OK` when the server is running: + +```json +{"status":"ok"} +``` + +### `POST /upload` + +Accepts one source bundle archive and returns after the archive is staged and validated. + +Accepted content types: + +- `application/x-tar` +- `application/gzip` +- `application/x-gzip` + +Successful admission returns `202 Accepted`: + +```json +{"run_id":"reports.20260604T120000Z.abcdef12","status":"accepted"} +``` + +Common error responses: + +- `400`: pipeline query supplied, archive rejected, malformed archive, or invalid staged source bundle. +- `401`: missing, empty, or unknown bearer token. +- `413`: upload body exceeds the selected pipeline size limit. +- `415`: unsupported content type. +- `503`: upload queue is full. + +Error bodies use: + +```json +{"error":""} +``` + +### `GET /runs/` + +Returns an in-memory status record while retained: + +```json +{ + "run_id": "reports.20260604T120000Z.abcdef12", + "pipeline_id": "reports", + "status": "succeeded", + "accepted_at": "2026-06-04T12:00:00Z", + "started_at": "2026-06-04T12:00:01Z", + "finished_at": "2026-06-04T12:00:02Z", + "report": {} +} +``` + +Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Failed records include `error`. Succeeded and failed records may include a run report. + +Unknown, malformed, expired, or process-lost run ids return `404`. + +## Archive Contract + +Upload archives must be uncompressed tar or gzip-compressed tar. The archive must contain exactly one root-level `manifest.json` and all manifest-listed files. + +Archive entry rules: + +- Paths must be clean relative slash-separated paths. +- Absolute paths, backslashes, `.` and `..` segments, duplicate files, and nested `manifest.json` entries are rejected. +- Only directories and regular files are accepted. +- Symlinks, hardlinks, devices, FIFOs, sockets, and other entry types are rejected. + +The uploaded archive size and extracted bundle size are bounded by the selected pipeline's `source.max_upload_size`. Extracted file count is also bounded by the implementation. + +## Queue And Retention + +`server.http.queue_size` bounds accepted-but-not-started uploads plus uploads being staged. `server.http.max_concurrency` bounds publishing concurrency. The coordinator does not run two uploads for the same pipeline concurrently. + +Completed status records expire after `server.http.retention`; expiration removes committed staged bundle directories for completed uploads. Server restart clears queue state and status records. + +## Boundaries + +The HTTP API does not expose pipeline selection by request parameter, TLS, public routing policy, or durable status storage. Put public access controls, TLS termination, and rate limiting in deployment infrastructure. + +## Tests + +Before changing this contract, inspect and run: + +```sh +go test ./internal/app ./internal/ingest +``` diff --git a/docs/integrations/markdown.md b/docs/integrations/markdown.md index ffce1c6..82a853b 100644 --- a/docs/integrations/markdown.md +++ b/docs/integrations/markdown.md @@ -1,27 +1,27 @@ # Markdown Integration -## Purpose +Audience: operators and maintainers who rely on generated HTML outputs from Markdown source files. -Markdown-to-HTML is the only implemented external file-format integration. This note documents the renderer behavior that is externally visible in generated destination artifacts. +Markdown-to-HTML is an implemented file-format integration used by destination transform policy. ## Dependency -Rendering uses `github.com/yuin/goldmark`. The exact dependency version is pinned in `go.mod`; review that file before changing renderer behavior or diagnosing version-specific output changes. +Rendering uses `github.com/yuin/goldmark`. The exact version is pinned in `go.mod`. -## Renderer behavior +## Renderer Behavior -`internal/transform/markdown.New` constructs the renderer with `goldmark.New()` and no project-specific extensions or renderer options. +The transformer constructs `goldmark.New()` with no project-specific extensions, parser options, renderer options, templates, CSS, or metadata injection. -The transform supports two output modes: +Supported output modes: -- `sidecar`: reads each source bundle file ending in `.md` and generates an HTML sidecar in the same logical directory. The output path replaces the `.md` suffix with `.html`, so `report.md` produces `report.html`. Non-Markdown source files produce no Markdown outputs. -- `index`: renders one selected Markdown source to `index.html` at the destination bundle path. +- `sidecar`: renders each source manifest file ending in `.md` and writes a generated output beside it with the `.md` suffix replaced by `.html`. +- `index`: renders one Markdown source file to `index.html` at the destination bundle path. -In `index` mode, `transform.markdown_to_html.input` can name the source manifest path to render. If `input` is omitted, the manifest must list exactly one Markdown file. The selected input must be a safe relative source path, must be listed in the source manifest, and must end in `.md`. +In `index` mode, `transform.markdown_to_html.input` may name the source manifest path to render. If `input` is omitted, the source manifest must list exactly one `.md` file. The selected input must be a clean relative source path, must be listed in the source manifest, and must end in `.md`. Raw HTML embedded in Markdown is not passed through by the current renderer behavior. Tests allow Goldmark's disabled-or-escaped raw HTML output forms and reject literal script tags in generated HTML. -## Wrapper +## HTML Wrapper Rendered Markdown body HTML is wrapped in a fixed document shell: @@ -33,7 +33,7 @@ Rendered Markdown body HTML is wrapped in a fixed document shell: The wrapper is deterministic and does not read configuration, templates, CSS, or source manifest metadata. -## Output metadata +## Output Metadata Generated outputs record: @@ -43,18 +43,16 @@ Generated outputs record: - SHA-256 digest of the wrapped HTML bytes; - byte size of the wrapped HTML bytes. +Destination state stores generated outputs with `kind: generated`, `source_path`, `transform`, `sha256`, `size`, and optional `url`. + ## Boundaries -Markdown rendering does not mutate source bundles, publish files, write `.distributor.json`, select outputs, or choose transfer actions. Publish planning decides whether generated HTML is selected for a destination. - -Publish planning chooses the configured mode and input for each destination. Markdown rendering does not inspect destinations, publish files, write `.distributor.json`, or choose transfer actions. +Markdown rendering does not mutate source bundles, publish files, write `.distributor.json`, select destination actions, or choose transfer policy. Publish planning decides whether generated HTML is selected for a destination and destination state records the generated output metadata. ## Tests Before changing Markdown renderer behavior, inspect and run: -```bash +```sh go test ./internal/transform/markdown ``` - -The tests cover sidecar naming, index input selection, ignored non-Markdown files, raw HTML handling, deterministic output, digest metadata, and size metadata. diff --git a/docs/integrations/s3.md b/docs/integrations/s3.md new file mode 100644 index 0000000..2acf28a --- /dev/null +++ b/docs/integrations/s3.md @@ -0,0 +1,73 @@ +# S3-Compatible Storage Integration + +Audience: operators and maintainers configuring S3-compatible sources or destinations. + +The S3 backend uses the AWS SDK for Go v2 against a configured S3-compatible endpoint. + +## Dependencies + +Runtime S3 support uses: + +- `github.com/aws/aws-sdk-go-v2` +- `github.com/aws/aws-sdk-go-v2/config` +- `github.com/aws/aws-sdk-go-v2/credentials` +- `github.com/aws/aws-sdk-go-v2/service/s3` +- `github.com/aws/smithy-go` + +Exact versions are pinned in `go.mod`. + +## Config Contract + +Required fields: + +- `backend: s3` +- `endpoint` +- `bucket` + +Optional fields: + +- `prefix`: backend root object-key prefix; leading and trailing slashes are trimmed. +- `region`: defaults to `us-east-1`. +- `force_path_style`: defaults to `true` through config defaults. +- `credentials.access_key_id_env` +- `credentials.secret_access_key_env` + +Credential environment variable names must either both be configured or both be omitted. When configured, values are resolved through the process environment or `secrets.directory` before opening the backend. When omitted, the AWS SDK default credential chain is used. + +## Object Key Mapping + +The configured `bucket` plus optional `prefix` is the backend root. Logical storage paths are joined under that prefix using slash-separated object keys. + +Prefixes and logical paths must be clean relative slash-separated paths. Prefixes are object-key prefixes, not real directories. + +## Storage Behavior + +The adapter uses these S3 operations: + +- `HeadObject` for stat and overwrite checks. +- `GetObject` for reads. +- `PutObject` for writes. +- `ListObjectsV2` for walks and prefix deletion planning. +- `DeleteObject` for managed cleanup and replacement. + +Writes buffer the input and set `ContentLength`. If no content type is supplied by the caller, the adapter infers a content type from the logical path. + +Normal replacement and failed-write cleanup delete only managed output objects plus `.distributor.json`. Forced replacement deletes objects under the bounded destination bundle prefix. The backend does not manage bucket versioning, lifecycle rules, object lock, or delete markers. + +## Error Mapping + +S3 not-found responses are translated into shared storage not-found errors where practical. Other service, credential, network, or endpoint errors are surfaced through storage errors with backend and logical path context. + +## Boundaries + +The S3 integration does not create buckets, alter bucket policy, configure TLS bypass, manage public URLs, or infer website URLs from endpoint or bucket settings. Public URL metadata is configured separately with destination `links`. + +## Tests + +Before changing this integration, inspect and run: + +```sh +go test ./internal/adapters/s3 +``` + +Live S3-compatible tests are opt-in and gated by environment variables in the adapter test package. diff --git a/docs/integrations/source-bundle.md b/docs/integrations/source-bundle.md new file mode 100644 index 0000000..3198c2b --- /dev/null +++ b/docs/integrations/source-bundle.md @@ -0,0 +1,92 @@ +# Source Bundle Contract + +Audience: producer developers, integrators, and maintainers who create or validate source bundles consumed by `distributor`. + +A source bundle is a directory containing `manifest.json` and every regular file listed by that manifest. This is the producer-to-`distributor` file-format contract. + +## Manifest Schema + +Current schema version: `1`. + +```json +{ + "schema_version": 1, + "id": "reports.example.2026-06-04", + "digest": "sha256:...", + "created": "2026-06-04T12:00:00Z", + "files": [ + { + "path": "report.md", + "sha256": "sha256:...", + "size": 1234 + } + ] +} +``` + +Required manifest fields: + +- `schema_version`: must be `1`. +- `id`: non-empty bundle identifier. +- `digest`: lowercase `sha256:<64 hex>` digest of the ordered `files` list. +- `created`: RFC3339 timestamp. +- `files`: non-empty ordered list of file records. + +Required file fields: + +- `path`: bundle-relative slash-separated file path. +- `sha256`: lowercase `sha256:<64 hex>` digest of the file bytes. +- `size`: file size in bytes, zero or greater. + +## Path Rules + +Manifest file paths must be clean relative slash-separated paths. They must not be empty, absolute, contain backslashes, contain `.` or `..` segments, include empty path segments, or normalize to a different path. + +Any basename of `manifest.json` or `.distributor.json` is reserved, including nested occurrences such as `nested/manifest.json`. + +Listed files must be regular files. Symlinks and other special file types are rejected during local bundle validation and manifest building. + +## Digest Rules + +File digests use SHA-256 over each file's raw bytes. + +The bundle digest is SHA-256 over the canonical JSON-like payload for the ordered file records. The payload is constructed as: + +```text +[{"path":"","sha256":"","size":},...] +``` + +File order is significant. Explicit file lists preserve caller order. Scan mode sorts paths in ascending slash-path order. + +## Producer APIs + +Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build and validate this contract: + +- `BuildManifest`: builds a manifest from explicit file paths or scan mode. +- `WriteManifest`: writes `manifest.json`, optionally replacing an existing manifest. +- `WriteBundle`: copies source files into a complete bundle, validates it, and promotes it into place. +- `LoadManifest`, `ParseManifest`, `ValidateManifest`, and `ValidateBundle`: parse and validate local bundles. +- `FileDigest`, `BundleDigest`, and `ValidateDigest`: digest helpers. + +CLI producers can use: + +```sh +go run ./cmd/distributor manifest create --id +go run ./cmd/distributor validate +``` + +## Scan Mode + +Manifest scan mode walks the local bundle root recursively, includes regular files, includes dotfiles, skips files whose basename is `manifest.json` or `.distributor.json`, rejects symlinks, and sorts paths before building the manifest. + +## Boundaries + +The source bundle manifest does not configure routing, destination selection, public URLs, credentials, transforms, notification behavior, or storage backends. Those concerns belong in `distributor` configuration and destination state. + +## Tests + +Before changing this contract, inspect and run: + +```sh +go test ./pkg/bundle ./internal/bundle +``` diff --git a/docs/integrations/ssh-sftp.md b/docs/integrations/ssh-sftp.md new file mode 100644 index 0000000..9b78419 --- /dev/null +++ b/docs/integrations/ssh-sftp.md @@ -0,0 +1,71 @@ +# SSH/SFTP Integration + +Audience: operators and maintainers configuring SSH/SFTP sources or destinations. + +The SSH backend uses native SSH and SFTP libraries. It does not call `ssh`, `scp`, or `rsync`. + +## Dependencies + +Runtime SSH support uses: + +- `golang.org/x/crypto/ssh` +- `golang.org/x/crypto/ssh/agent` +- `golang.org/x/crypto/ssh/knownhosts` +- `github.com/pkg/sftp` + +Exact versions are pinned in `go.mod`. + +## Config Contract + +Required fields: + +- `backend: ssh` +- `host` +- `path` + +Optional fields: + +- `user`: defaults to the current OS user when available. +- `port`: defaults to `22`. +- `ssh_key_file`: private key path. +- `known_hosts`: OpenSSH known-hosts file path. +- `host_key_policy`: `strict`, `accept-new`, or `off`; defaults to `accept-new`. + +## Authentication + +Authentication methods are attempted in this order: + +1. SSH agent identities when `SSH_AUTH_SOCK` is set. +2. The private key configured by `ssh_key_file`. + +Password authentication is not configured in YAML. If neither an agent nor key file is available, opening the backend fails. + +## Host Key Policy + +- `strict`: requires a matching known host key. +- `accept-new`: accepts and persists an unknown host key, but rejects changed known keys. +- `off`: disables host key checking. + +When `known_hosts` is omitted and checking is enabled, the adapter uses the current user's default OpenSSH `known_hosts` path when available. During dry runs, accepted unknown host keys are not persisted. + +## Storage Behavior + +The configured `path` is the backend root. All source discovery, destination paths, reads, writes, state files, and deletes operate on logical paths below that root. + +The adapter rejects symlink ancestors for reads and writes. Reads require regular files. Writes create parent directories and prefer atomic temp-file-plus-rename writes when requested. Walk output is sorted through the shared storage walker. + +Managed cleanup and normal replacement delete only managed output paths plus `.distributor.json`. Forced replacement deletes the bounded destination bundle path. + +## Boundaries + +The SSH backend does not configure passwords, jump hosts, shell commands, `rsync`, host-key bypass warnings beyond command output, or broad recursive deletion outside the destination bundle path. + +## Tests + +Before changing this integration, inspect and run: + +```sh +go test ./internal/adapters/ssh +``` + +Live SSH tests are opt-in and gated by environment variables in the adapter test package. diff --git a/docs/operations.md b/docs/operations.md index 991ccd6..a129161 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -2,7 +2,7 @@ Audience: administrators and operators who run `distributor`, publish bundles, operate the HTTP upload service, or recover from failed runs. -This document covers operating workflows, storage layout, safety behavior, and recovery. Command syntax lives in [CLI](cli.md), configuration fields live in [Configuration](config.md), and symptom-specific fixes live in [Troubleshooting](troubleshooting.md). +This document covers operating workflows, storage layout, safety behavior, and recovery. Command syntax lives in [CLI](cli.md), configuration fields live in [Configuration](config.md), symptom-specific fixes live in [Troubleshooting](troubleshooting.md), and external contracts live under [Integrations](integrations/source-bundle.md). ## Normal Workflow @@ -41,7 +41,7 @@ Remote examples under `examples/ssh-destination.yml` and `examples/s3-destinatio ## Filesystem And Storage Layout -A source bundle is a directory containing `manifest.json` and every file listed in that manifest. Source discovery walks beneath the configured source backend root and finds bundle directories. +A source bundle is a directory containing `manifest.json` and every file listed in that manifest. See [Source Bundle Contract](integrations/source-bundle.md). Source discovery walks beneath the configured source backend root and finds bundle directories. Each destination has its own backend root: @@ -56,7 +56,7 @@ Destination path mapping controls where each source bundle is published beneath Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple bundles have the same timestamp, the source-root-relative bundle path in ascending order wins. -Published destination bundle paths contain `.distributor.json`. This file is both the managed sentinel and the destination state record. It records the pipeline id, destination id, publication time, source manifest, copied outputs, generated outputs, and optional public URL metadata. +Published destination bundle paths contain `.distributor.json`. See [Destination State Contract](integrations/destination-state.md). This file is both the managed sentinel and the destination state record. It records the pipeline id, destination id, publication time, source manifest, copied outputs, generated outputs, and optional public URL metadata. `manifest.json` from the source bundle is not copied as destination state. @@ -114,7 +114,7 @@ Forced replacement deletes the current destination bundle path before writing ou ## HTTP Upload Operation -`distributor serve` runs the HTTP upload API for pipelines whose source backend is `http_upload`. Each bearer token maps to exactly one configured upload pipeline. Token values come from the process environment or `secrets.directory`, not from YAML literal values. +The [HTTP Upload API Contract](integrations/http-upload.md) defines request and response details. `distributor serve` runs the HTTP upload API for pipelines whose source backend is `http_upload`. Each bearer token maps to exactly one configured upload pipeline. Token values come from the process environment or `secrets.directory`, not from YAML literal values. Start the maintained local example: @@ -162,7 +162,7 @@ The default bind address is private loopback. Put TLS, public routing, rate limi ### SSH/SFTP -SSH execution uses native SFTP. It does not shell out to `ssh`, `scp`, or `rsync`. +SSH execution uses native SFTP. See [SSH/SFTP Integration](integrations/ssh-sftp.md). It does not shell out to `ssh`, `scp`, or `rsync`. Configure `ssh_key_file`, an SSH agent, or both. Agent identities are attempted first, followed by the configured key file. YAML password authentication is not supported. @@ -170,7 +170,7 @@ The default host key policy is `accept-new`. During dry runs, unknown host keys ### S3-Compatible Storage -S3 execution uses the AWS SDK for Go v2. Configure an endpoint, bucket, optional prefix, optional region, optional path-style setting, and optional explicit credential variable names. +S3 execution uses the AWS SDK for Go v2. See [S3-Compatible Storage Integration](integrations/s3.md). Configure an endpoint, bucket, optional prefix, optional region, optional path-style setting, and optional explicit credential variable names. When explicit S3 credential variable names are configured, both must resolve to non-empty values through the process environment or `secrets.directory`. When omitted, the AWS SDK default credential chain is used as-is.