Validate documentation roadmap completion

README.md

@@ -1,18 +1,20 @@
 # distributor
 
-`distributor` validates manifested report bundles and publishes selected source or generated artifacts to configured destinations.
+`distributor` validates manifested report bundles, plans destination updates, and publishes selected source files or generated HTML outputs to configured destinations.
 
-It is a local-first CLI with SSH/SFTP, S3-compatible storage, and HTTP upload
+It is a local-first Go CLI for report distribution. A pipeline reads one source bundle tree, validates `manifest.json`, fans out to one or more local, SSH/SFTP, or S3-compatible destinations, records destination state in `.distributor.json`, and can also accept authenticated tar or tar.gz uploads through the HTTP upload server.
-support: source bundles can be read from local or remote storage, pushed to the
-upload API, published to local directories or remote paths, and rendered from
-Markdown to HTML sidecars or `index.html`.
 
-Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, write, parse, and validate complete local source bundles with the same manifest contract used by `distributor`.
+Run the maintained local example:
-
-Run the local example pipeline:
 
 ```sh
 go run ./cmd/distributor run --config examples/local-publish.yml
 ```
 
-See [docs/cli.md](docs/cli.md), [docs/config.md](docs/config.md), [docs/operations.md](docs/operations.md), and [docs/troubleshooting.md](docs/troubleshooting.md) for the implemented CLI, configuration, operating notes, and common failure modes. Future and deferred work lives under `docs/roadmap/`.
+Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, write, parse, and validate local source bundles with the same manifest contract used by the CLI. See [Source bundle contract](docs/integrations/source-bundle.md).
+
+- [CLI reference](docs/cli.md)
+- [Configuration reference](docs/config.md)
+- [Operations guide](docs/operations.md)
+- [Troubleshooting](docs/troubleshooting.md)
+- [Integration contracts](docs/integrations/source-bundle.md)
+- [Development architecture](docs/policy/architecture.md)

docs/cli.md

@@ -1,17 +1,24 @@
 # Distributor CLI
 
-## Shortest useful command
+Audience: operators, integrators, and developers who run `distributor` from a shell or automation.
+
+This document is the canonical command and flag reference. Configuration schema details live in [Configuration](config.md), operational recovery guidance lives in [Operations](operations.md), failure diagnosis lives in [Troubleshooting](troubleshooting.md), and external contracts live under [Integrations](integrations/source-bundle.md).
+
+## Shortest Useful Command
+
+Run the maintained local publishing example from the repository root:
 
 ```sh
 go run ./cmd/distributor run --config examples/local-publish.yml
 ```
 
-This discovers the example source bundle and publishes source files to `workspace/published/source-bundle`.
+The example reads `examples/source-bundle/manifest.json`, publishes the configured files into `workspace/published/source-bundle`, and writes destination state metadata beside the published output.
 
-## Command overview
+## Command Overview
 
-```sh
+```text
 distributor [--help]
+distributor help
 distributor version [--format text|json]
 distributor run [--config <path>] [--dry-run] [--force] [--format text|json]
 distributor serve [--config <path>]
@@ -19,192 +26,176 @@ distributor validate [--format text|json] <path>
 distributor validate --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
 distributor inspect [--format text|json] <path>
 distributor inspect --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
+distributor manifest
 distributor manifest create <bundle-path> --id <bundle-id> [options]
+distributor manifest create --id <bundle-id> [options] <bundle-path>
 ```
 
-- `version`: prints the application name and version. Development builds print `distributor dev`.
+- `version` prints the application name and version.
-- `run`: loads a YAML config, discovers source bundles, plans each configured destination, writes selected outputs unless `--dry-run` is set, and prints a final status summary.
+- `run` executes configured pipelines against their destinations.
-- `serve`: loads a YAML config, resolves HTTP upload bearer tokens, and runs the HTTP upload API.
+- `serve` starts the authenticated HTTP upload API defined by the configuration file.
-- `validate`: validates a local source bundle directory, a local source bundle tree, or one configured pipeline source.
+- `validate` checks a local bundle path or a configured source bundle.
-- `inspect`: validates source bundles and prints normalized bundle metadata for a local path or one configured pipeline source.
+- `inspect` reports manifest and file metadata for a local bundle path or a configured source bundle.
-- `manifest create`: creates `manifest.json` for a local source bundle directory.
+- `manifest create` writes a `manifest.json` file for an existing bundle directory.
 
-`validate` and `inspect` have two mutually exclusive modes: a local path shortcut, or configured source mode with `--config <path> --pipeline <id>`. Configured source mode opens only the selected pipeline source and supports configured `local`, `ssh`, and `s3` sources. It does not open destinations. `run` executes configured `local`, `ssh`, and `s3` sources and destinations. `serve` executes configured `http_upload` sources through the upload API and normal destination fan-out.
+## Flag Reference
 
-## Flag reference
+### Help
 
-Root command:
+`distributor`, `distributor --help`, `distributor -h`, `distributor help`, and `distributor manifest` print command help. Unknown commands and invalid argument combinations print usage guidance and exit non-zero.
 
-- `--help`, `-h`, or `help`: print root help.
+### Common Output Format
 
-All subcommands:
+`--format text|json` is supported by `version`, `run`, `validate`, `inspect`, and `manifest create`.
 
-- `--help`, `-h`: print command-specific help.
+- `text` is the default human-readable output.
+- `json` emits one JSON document for successful command execution.
+- Invalid formats are rejected before command execution.
 
-Output-producing subcommands:
+### `version`
 
-- `--format text|json`: output format. `text` is the default. Help and usage output are always text.
+```sh
+distributor version [--format text|json]
+```
 
-`run` flags:
+`version` accepts no positional arguments. Text output prints the application name and version; JSON output includes `application` and `version` fields.
 
-- `--config <path>`: config file to load. If omitted, `run` uses `/usr/local/etc/distributor/config.yml`.
+### `run`
-- `--dry-run`: load config, discover bundles, inspect destination state, print planned actions and final status, and do not write output files, destination state, or SSH `known_hosts` entries.
-- `--force`: allow explicit destructive replacement for supported conflict cases in this run only.
 
-`serve` flags:
+```sh
+distributor run [--config <path>] [--dry-run] [--force] [--format text|json]
+```
 
-- `--config <path>`: config file to load. If omitted, `serve` uses `/usr/local/etc/distributor/config.yml`.
+- `--config <path>` loads the pipeline configuration. If omitted, the application uses `/usr/local/etc/distributor/config.yml`.
+- `--dry-run` validates inputs and reports destination actions without applying changes.
+- `--force` permits a run when destination state indicates a conservative safety check would otherwise block it.
+- `--format text|json` selects human-readable or machine-readable output.
 
-`validate` and `inspect` configured source flags:
+`run` accepts no positional arguments.
 
-- `--config <path>`: config file to load for source validation or inspection. Required in configured source mode.
+### `serve`
-- `--pipeline <id>`: pipeline source to validate or inspect. Required in configured source mode.
-- `--bundle <path>`: source-root-relative bundle directory to validate or inspect instead of discovering every bundle under the source root.
 
-`manifest create` flags:
+```sh
+distributor serve [--config <path>]
+```
 
-- `--id <bundle-id>`: source bundle id. Required.
+- `--config <path>` loads HTTP, source, destination, and pipeline configuration. If omitted, the application uses `/usr/local/etc/distributor/config.yml`.
-- `--file <path>`: bundle-relative file to include. Repeatable. If omitted, files are scanned recursively.
-- `--created <time>`: RFC3339 source created timestamp. If omitted, the current UTC time is used.
-- `--overwrite`: replace an existing `manifest.json`.
 
-`run` does not accept positional arguments. `validate` and `inspect` accept at most one path in local mode. Local paths cannot be combined with `--config`, `--pipeline`, or `--bundle`.
+`serve` accepts no positional arguments and runs until interrupted or until the server exits with an error.
 
-## Common workflows
+### `validate`
 
-Validate a source bundle:
+```sh
+distributor validate [--format text|json] <path>
+distributor validate --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
+```
+
+`validate` has two source modes:
+
+- Local path mode validates the bundle at `<path>`.
+- Configured source mode resolves the source from `--config <path>` and `--pipeline <id>`.
+
+Configured source flags:
+
+- `--config <path>` loads the configuration file.
+- `--pipeline <id>` selects the configured pipeline source to validate.
+- `--bundle <path>` overrides the configured source bundle path for the selected pipeline.
+- `--format text|json` selects output format.
+
+A local positional path cannot be combined with `--config`, `--pipeline`, or `--bundle`. When any configured source flag is used, both `--config` and `--pipeline` are required.
+
+### `inspect`
+
+```sh
+distributor inspect [--format text|json] <path>
+distributor inspect --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
+```
+
+`inspect` uses the same source mode rules as `validate`, then reports bundle metadata instead of only validation status. Local path mode requires exactly one bundle path. Configured source mode requires both `--config` and `--pipeline`; `--bundle` may override the selected pipeline source path.
+
+### `manifest create`
+
+```sh
+distributor manifest create <bundle-path> --id <bundle-id> [options]
+distributor manifest create --id <bundle-id> [options] <bundle-path>
+```
+
+Flags may appear before or after the bundle path. Both `--flag value` and `--flag=value` forms are accepted.
+
+- `--id <bundle-id>` sets the manifest bundle identifier and is required.
+- `--created <timestamp>` sets the manifest creation timestamp. If omitted, the current UTC time is used.
+- `--file <relative-path>` includes one file in the manifest. The flag may be repeated.
+- `--overwrite` allows replacing an existing `manifest.json` file.
+- `--format text|json` selects output format.
+
+If no `--file` flags are provided, `manifest create` scans the bundle directory recursively. The command requires exactly one bundle path, refuses unsafe manifest paths, and writes `manifest.json` at the bundle root.
+
+## Common Workflows
+
+### Validate Or Inspect A Local Bundle
 
 ```sh
 go run ./cmd/distributor validate examples/source-bundle
+go run ./cmd/distributor inspect --format json examples/source-bundle
 ```
 
-Inspect a source bundle:
+### Validate Or Inspect A Configured Source
-
-```sh
-go run ./cmd/distributor inspect examples/source-bundle
-```
-
-Validate a configured source without opening destinations:
 
 ```sh
 go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
+go run ./cmd/distributor inspect --config examples/local-publish.yml --pipeline example-source-bundle --format json
 ```
 
-Inspect one configured source bundle:
+Use `--bundle <path>` with configured source mode when automation needs to validate or inspect an alternate bundle path through the selected pipeline configuration.
+
+### Create A Manifest
 
 ```sh
-go run ./cmd/distributor inspect \
+go run ./cmd/distributor manifest create examples/source-bundle --id example-source-bundle --overwrite
-  --config <config-path> \
+go run ./cmd/distributor manifest create --id example-source-bundle --overwrite examples/source-bundle
-  --pipeline <pipeline-id> \
-  --bundle daily/2026-06-01
 ```
 
-Create a manifest for a local producer bundle:
+Use repeated `--file` flags when the manifest should include an explicit file list instead of the recursive directory scan:
 
 ```sh
-go run ./cmd/distributor manifest create <bundle-path> --id <bundle-id>
+go run ./cmd/distributor manifest create examples/source-bundle \
-```
+  --id example-source-bundle \
-
-Create a manifest with explicit file order:
-
-```sh
-go run ./cmd/distributor manifest create <bundle-path> \
-  --id <bundle-id> \
-  --created 2026-06-01T11:00:00Z \
   --file report.md \
-  --file summary.txt
+  --file summary.txt \
+  --overwrite
 ```
 
-Preview local publication without writing:
+### Preview Or Publish A Pipeline
 
 ```sh
 go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
-```
-
-Publish the local source example:
-
-```sh
 go run ./cmd/distributor run --config examples/local-publish.yml
 ```
 
-Publish the local HTML example:
+Use `--format json` when automation needs structured run results. Use `--force` only when the operator has reviewed the destination state conflict and intentionally wants to continue.
-
+
-```sh
+### Run HTML And Fan-Out Examples
-go run ./cmd/distributor run --config examples/local-html.yml
-```
-
-Start the HTTP upload API:
-
-```sh
-DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN=<token> \
-  go run ./cmd/distributor serve --config examples/http-upload-local.yml
-```
-
-Upload an archive to the configured `http_upload` pipeline associated with a bearer token:
-
-```sh
-curl -X POST http://127.0.0.1:8080/upload \
-  -H "Authorization: Bearer $DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN" \
-  -H "Content-Type: application/gzip" \
-  --data-binary @bundle.tar.gz
-```
-
-The upload response is accepted asynchronously:
-
-```json
-{"run_id":"reports.20260603T120000Z.abcdef12","status":"accepted"}
-```
-
-Check upload status:
-
-```sh
-curl http://127.0.0.1:8080/runs/<run-id>
-```
-
-Check server readiness:
-
-```sh
-curl http://127.0.0.1:8080/healthz
-```
-
-Preview local fan-out publication:
 
 ```sh
+go run ./cmd/distributor run --config examples/local-html.yml --dry-run
+go run ./cmd/distributor run --config examples/local-index.yml --dry-run
 go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
-```
-
-Preview local archive-plus-latest publication:
-
-```sh
 go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run
 ```
 
-Preview a forced replacement before publishing:
+These examples exercise implemented output rendering and destination planning behavior. They still use the same `run` flags and output contract described here.
+
+### Start The HTTP Upload Server
 
 ```sh
-go run ./cmd/distributor run --config <config-path> --dry-run --force
+go run ./cmd/distributor serve --config examples/http-upload-local.yml
 ```
 
-## Output
+The server exposes health, status, and authenticated upload endpoints according to the loaded configuration. Use [Operations](operations.md) for server operation and recovery guidance.
 
-Text output is the default and is intended for humans.
+## Output And Exit Behavior
 
-`run` text output prints the number of configured pipelines, one line per pipeline, one line per planned destination action, and a final status line. Destination action lines include the source bundle path, destination id, destination backend, action, outputs, and reason. Fixed path destinations also print `path_mapping=fixed target=.` to show that the selected bundle targets the destination backend root. Actions include:
+Text output is optimized for direct operator use. JSON output is optimized for automation and uses a command-specific result object with a shared envelope similar to:
-
-- `publish_new`: destination has no managed state and is empty.
-- `replace_older`: destination state is older than the source manifest.
-- `force_replace`: `--force` requested a supported destructive replacement.
-- `skip_same`: destination state already matches the source manifest.
-- `skip_destination_newer`: destination state is newer than the source manifest.
-- `error`: planning or execution failed for that destination.
-
-The command exits non-zero if any destination fails. Independent later destinations are still attempted.
-
-Dry-run output for fixed path destinations prints a warning with the candidate count and selected source bundle. If a fixed path dry run plans a destructive replacement, it prints an additional warning that the destination root would be replaced.
-
-The final status line includes counters for `publish_new`, `replace_older`, `force_replace`, skipped destinations, failures, whether the run was a dry run, and fixed path destinations.
-
-JSON output writes exactly one JSON document to stdout:
 
 ```json
 {
@@ -216,20 +207,16 @@ JSON output writes exactly one JSON document to stdout:
 }
 ```
 
-Warnings are objects in the top-level `warnings` array and are not printed again as text. Fatal setup errors, such as a missing config file or invalid arguments, write no JSON document and return a non-zero exit code with a text error on stderr.
+- Successful JSON commands emit one JSON document on stdout.
+- Usage errors and fatal setup errors exit non-zero and do not emit a JSON result document.
+- `run --format json` emits a JSON result for partial destination failures, sets `ok` to `false`, includes result details and errors, and exits non-zero.
+- Warnings are included in JSON output and are printed in text output when relevant.
 
-`run --format json` returns partial results when destination failures occur after planning or execution begins. In that case stdout contains `ok: false`, a `result` with pipeline summaries, destination actions, final counters, and a top-level `errors` array; the process still exits non-zero.
+## Diagnostics And Recovery
 
-Command-specific JSON results:
+- Use `validate` before `run` when checking a bundle supplied by another process.
-
+- Use `inspect --format json` when automation needs manifest metadata, normalized file details, or checksum information.
-- `version`: application name and version.
+- Use `run --dry-run` before publishing to review destination actions.
-- `validate`: bundle count and discovered bundle identifiers. Configured source results also include pipeline id and source backend.
+- Use [Configuration](config.md) for schema and default details.
-- `inspect`: bundle path, id, created timestamp, digest, file count, total size, and manifest file records. Configured source results also include pipeline id and source backend.
+- Use [Troubleshooting](troubleshooting.md) for common errors and corrective action.
-- `manifest create`: manifest path, bundle root, id, created timestamp, digest, file count, and file records.
+- Use [Operations](operations.md) for HTTP upload operation, state files, and recovery workflows.
-- `run`: dry-run status, pipeline summaries, destination action records, destination bundle paths, path mapping markers, optional primary URLs, output records with optional URLs, final counters, warnings, and partial failure records.
-
-## Diagnostics
-
-Use `manifest create` when a local producer has written bundle files but not `manifest.json`. Use `validate` before publication when a producer has written a new bundle; use configured source mode when the bundle is already on an SSH or S3 source. Use `inspect` to confirm normalized ids, timestamps, digests, file paths, and file sizes.
-
-For symptom-oriented recovery steps, see [troubleshooting](troubleshooting.md). For destination state and retry behavior, see [operations](operations.md). For config fields and defaults, see [configuration](config.md).

docs/config.md

@@ -1,19 +1,25 @@
 # Configuration Reference
 
-## Config File Location
+Audience: administrators, operators, and advanced users who write YAML configuration for `distributor`.
 
-`distributor run --config <path>` and `distributor serve --config <path>` load
+This document is the canonical user-facing configuration reference. CLI syntax lives in [CLI](cli.md), operating procedures live in [Operations](operations.md), symptom-oriented recovery lives in [Troubleshooting](troubleshooting.md), and external contracts live under [Integrations](integrations/source-bundle.md).
-the YAML config at the provided path.
 
-If `--config` is omitted, both commands use:
+## Config File Loading
+
+`distributor run --config <path>` and `distributor serve --config <path>` load the YAML file at `<path>`. If `--config` is omitted, both commands use:
 
 ```text
 /usr/local/etc/distributor/config.yml
 ```
 
-Config parsing rejects unknown YAML fields. The executable `run` backends are
+YAML decoding rejects unknown fields. Defaults are applied after decoding and before validation.
-`local`, `ssh`, and `s3`. The `serve` command executes `http_upload` sources
+
-through the HTTP upload API and normal destination fan-out.
+Runtime backend support is command-specific:
+
+- `run`, `validate --config`, and `inspect --config` execute `local`, `ssh`, and `s3` sources.
+- `run` executes `local`, `ssh`, and `s3` destinations.
+- `serve` uses `http_upload` sources through the HTTP upload API and publishes to configured `local`, `ssh`, and `s3` destinations.
+- `http_upload` is valid only as a source backend.
 
 ## Minimal Local Config
 
@@ -29,7 +35,7 @@ pipelines:
         path: /srv/reports/archive
 ```
 
-This publishes source files only. It uses the default validation and transfer policies.
+This config publishes source files only. It uses default validation, destination path mapping, publish, transfer, and HTTP server values.
 
 ## Production-Oriented Local Config
 
@@ -42,6 +48,8 @@ server:
     queue_size: 16
     max_concurrency: 1
     retention: 24h
+secrets:
+  directory: /run/secrets/distributor
 pipelines:
   - id: reports
     source:
@@ -56,6 +64,8 @@ pipelines:
         publish:
           source: true
           html: false
+        path_mapping:
+          mode: preserve_relative
         transfer:
           on_destination_same: skip
           on_destination_older: replace
@@ -63,11 +73,9 @@ pipelines:
           on_conflict: fail
 ```
 
-## HTTP Upload Source Configuration
+## HTTP Upload Source Config
 
-HTTP upload sources are configured as pipeline sources only. They are not valid
+HTTP upload sources are configured on pipelines and are served by `distributor serve`. Upload tokens are resolved from the process environment or `secrets.directory`; literal bearer tokens are not configured in YAML.
-destination backends. `distributor serve` maps each configured upload token to
-exactly one `http_upload` pipeline.
 
 ```yaml
 server:
@@ -91,59 +99,188 @@ pipelines:
         path: /srv/reports/archive
 ```
 
-`source.token_env` is required and names the environment variable or `secrets.directory` file that provides the bearer token. Literal upload tokens are not supported in YAML.
+`token_env` is required for `http_upload` sources. `staging_path` defaults to `<server.http.staging_root>/<pipeline id>`. `max_upload_size` defaults to `server.http.max_upload_size`.
 
-`source.staging_path` is optional. When omitted, it defaults to `<server.http.staging_root>/<pipeline id>`.
+`serve` maps each resolved bearer token to exactly one `http_upload` pipeline. Startup fails when a token is missing, empty, or duplicates another upload pipeline token.
 
-`source.max_upload_size` is optional. When omitted, it defaults to `server.http.max_upload_size`.
+## Top-Level Fields
 
-The server resolves each `token_env` through the real process environment and
+### `server.http`
-the configured `secrets.directory` resolver. Startup fails if any configured
-upload token is missing, empty, or resolves to the same value as another upload
-pipeline. Token values are not read from YAML and are not printed in API
-responses.
 
-## HTTP Upload API
+`server.http` controls the HTTP upload server used by `serve`.
 
-`distributor serve` binds to `server.http.bind`, which defaults to
+- `bind`: optional TCP bind address. Default: `127.0.0.1:8080`.
-`127.0.0.1:8080`.
+- `staging_root`: optional root used to default `http_upload` source staging paths. Default: `/var/spool/distributor`.
+- `max_upload_size`: optional default upload limit for HTTP upload sources. Default: `20MB`.
+- `queue_size`: optional upload admission queue size. Default: `16`.
+- `max_concurrency`: optional upload worker concurrency. Default: `1`.
+- `retention`: optional in-memory completed-run retention duration. Default: `24h`.
 
-Routes:
+Numeric server values and durations must be greater than zero after defaults are applied.
 
-- `GET /healthz`: returns readiness status after config and upload tokens load.
+### `secrets`
-- `POST /upload`: accepts one tar or tar.gz source bundle archive.
-- `GET /runs/<run_id>`: returns an in-memory upload status record, or `404` if the run id is unknown or expired.
 
-`POST /upload` authenticates with:
+- `directory`: optional directory of secret files used by the config-owned credential resolver.
 
-```text
+See [Secrets](#secrets) for resolution rules.
-Authorization: Bearer <token>
+
+### `pipelines`
+
+`pipelines` is required and must contain at least one pipeline.
+
+Each pipeline has:
+
+- `id`: required unique slug-like identifier. It must start with a letter or number and may contain letters, numbers, `.`, `_`, and `-`.
+- `source`: required source backend config.
+- `validation`: optional validation policy.
+- `destinations`: required non-empty destination list.
+
+Pipeline ids must be unique across the config.
+
+## Backend Reference
+
+### Local Backend
+
+Local backends can be used as sources and destinations.
+
+```yaml
+backend: local
+path: /srv/distributor/archive
 ```
 
-The token selects the configured `http_upload` pipeline. Producers do not send a
+- `backend`: required value `local`.
-pipeline id. Requests with a submitted `pipeline` or `pipeline_id` query value
+- `path`: required local filesystem root for this backend.
-are rejected.
 
-Accepted upload content types:
+### SSH/SFTP Backend
 
-- `application/x-tar`
+SSH backends use native SFTP and can be used as sources and destinations. Adapter protocol behavior is documented in [SSH/SFTP Integration](integrations/ssh-sftp.md).
-- `application/gzip`
-- `application/x-gzip`
 
-Accepted uploads return:
+```yaml
-
+backend: ssh
-```json
+host: ssh.example.com
-{"run_id":"<id>","status":"accepted"}
+user: distributor
+port: 22
+path: /srv/distributor/archive
+ssh_key_file: /home/distributor/.ssh/id_ed25519
+known_hosts: /home/distributor/.ssh/known_hosts
+host_key_policy: strict
 ```
 
-The run id can be queried through `GET /runs/<run_id>` while the status record
+- `backend`: required value `ssh`.
-is retained in memory. Completed records expire after `server.http.retention`;
+- `host`: required SSH host.
-expiration also removes committed staged bundle directories for completed
+- `path`: required remote root path.
-uploads.
+- `user`: optional SSH username. If omitted, the adapter uses the current OS user when available.
+- `port`: optional TCP port. Default: `22`.
+- `ssh_key_file`: optional private key path.
+- `known_hosts`: optional OpenSSH `known_hosts` path.
+- `host_key_policy`: optional host key policy. Default: `accept-new`.
 
-## HTML Publication
+Accepted host key policy values:
 
-To publish generated sidecar HTML from Markdown files:
+- `strict` or boolean `true`: require a matching known host key.
+- `accept-new`: accept and persist a new host key, but reject changed known keys.
+- `off` or boolean `false`: disable host key checking.
+
+Authentication uses SSH agent identities when `SSH_AUTH_SOCK` is available, then `ssh_key_file` when configured. Password authentication is not configured in YAML.
+
+### S3-Compatible Backend
+
+S3 backends can be used as sources and destinations. Adapter protocol behavior is documented in [S3-Compatible Storage Integration](integrations/s3.md).
+
+```yaml
+backend: s3
+endpoint: https://s3.example.com
+bucket: reports
+prefix: distributor/archive
+region: us-east-1
+force_path_style: true
+credentials:
+  access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
+  secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
+```
+
+- `backend`: required value `s3`.
+- `endpoint`: required S3-compatible endpoint URL.
+- `bucket`: required bucket name.
+- `prefix`: optional backend root prefix. Leading and trailing slashes are trimmed; the remaining value must be a clean relative slash-separated path.
+- `region`: optional region. Default: `us-east-1`.
+- `force_path_style`: optional addressing mode toggle. Default: `true`.
+- `credentials.access_key_id_env`: optional environment variable or secret-file name for the access key id.
+- `credentials.secret_access_key_env`: optional environment variable or secret-file name for the secret access key.
+
+The S3 credential variable names must either both be configured or both be omitted. When omitted, the AWS SDK default credential chain is used. When configured, both values must resolve to non-empty strings through the process environment or `secrets.directory`.
+
+### HTTP Upload Source Backend
+
+HTTP upload backends are valid only as pipeline sources and are served by `distributor serve`. The API contract is documented in [HTTP Upload API Contract](integrations/http-upload.md).
+
+```yaml
+backend: http_upload
+token_env: WEATHER_DAILY_UPLOAD_TOKEN
+staging_path: /var/spool/distributor/weather-daily
+max_upload_size: 20MB
+```
+
+- `backend`: required value `http_upload`.
+- `token_env`: required environment variable or secret-file name containing the bearer token.
+- `staging_path`: optional staging path. Default: `<server.http.staging_root>/<pipeline id>`.
+- `max_upload_size`: optional per-source upload limit. Default: `server.http.max_upload_size`.
+
+## Validation Policy
+
+```yaml
+validation:
+  on_digest_mismatch: fail
+```
+
+- `validation.on_digest_mismatch`: optional. Default and only accepted value: `fail`.
+
+Source bundle digest mismatches fail validation before destination writes occur. The manifest file-format contract is documented in [Source Bundle Contract](integrations/source-bundle.md).
+
+## Destination Fields
+
+Each destination embeds a backend config at the destination level and may also configure publishing, transforms, path mapping, links, and transfer behavior.
+
+```yaml
+destinations:
+  - id: archive
+    backend: local
+    path: /srv/reports/archive
+    publish:
+      source: true
+      html: false
+    path_mapping:
+      mode: preserve_relative
+    transfer:
+      on_destination_same: skip
+      on_destination_older: replace
+      on_destination_newer: skip
+      on_conflict: fail
+```
+
+- `id`: required unique slug-like identifier within the pipeline.
+- Backend fields: required according to the selected destination backend.
+- `publish`: optional publish policy. Default: source-only publication.
+- `transform`: required only when publishing generated HTML.
+- `path_mapping`: optional destination path mapping policy.
+- `links`: optional public URL metadata policy.
+- `transfer`: optional destination reconciliation policy.
+
+Destination ids must be unique within a pipeline.
+
+## Publish And Transform Policy
+
+### Source-Only Publication
+
+```yaml
+publish:
+  source: true
+  html: false
+```
+
+`publish.source` controls whether source manifest files are copied to the destination.
+
+### Markdown-To-HTML Publication
 
 ```yaml
 publish:
@@ -155,240 +292,106 @@ transform:
     mode: sidecar
 ```
 
-Sidecar generation writes `report.html` for `report.md`. It does not mutate the source bundle.
+`publish.html` controls whether generated HTML outputs are published. When `publish.html` is `true`, `transform.markdown_to_html.enabled` must also be `true`.
 
-To publish a single Markdown file as `index.html`:
+Markdown transform fields:
 
-```yaml
+- `transform.markdown_to_html.enabled`: enables Markdown-to-HTML generation for this destination.
-publish:
+- `transform.markdown_to_html.mode`: optional. Accepted values are `sidecar` and `index`; default is `sidecar` when a Markdown transform block is present.
-  source: false
+- `transform.markdown_to_html.input`: optional source manifest path for `index` mode only.
-  html: true
-transform:
-  markdown_to_html:
-    enabled: true
-    mode: index
-    input: report.md
-```
 
-When `mode: index` omits `input`, the source manifest must list exactly one Markdown file.
+`sidecar` mode renders every manifest-listed `.md` file to a same-directory `.html` output. `index` mode renders one Markdown source to `index.html` at the destination bundle path. If `index` mode omits `input`, the selected source bundle must contain exactly one Markdown file.
+
+At least one output type must be enabled. Enabled Markdown transforms are rejected when `publish.html` is `false`, and `input` is rejected unless `mode` is `index`.
 
 ## Destination Path Mapping
 
-Each destination chooses how source bundle paths map into that destination:
-
 ```yaml
 path_mapping:
   mode: preserve_relative
 ```
 
-`preserve_relative` is the default. It publishes each discovered source bundle at the same path relative to the destination backend root. A source bundle at `daily/2026-06-01` publishes below `daily/2026-06-01` for that destination.
+- `path_mapping.mode`: optional. Accepted values are `preserve_relative` and `fixed`; default is `preserve_relative`.
 
-`fixed` publishes one selected source bundle directly at the destination backend root:
+`preserve_relative` publishes each discovered source bundle at the same path relative to the destination backend root.
 
-```yaml
+`fixed` publishes one selected source bundle directly at the destination backend root. Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple bundles have the same timestamp, the source-root-relative bundle path in ascending order wins.
-destinations:
-  - id: latest-html
-    backend: local
-    path: /srv/www/reports/latest
-    path_mapping:
-      mode: fixed
-    publish:
-      source: false
-      html: true
-    transform:
-      markdown_to_html:
-        enabled: true
-        mode: index
-        input: report.md
-```
 
-Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple candidates have the same timestamp, the source-root-relative bundle path in ascending order wins. Older candidates are not planned or written for that destination.
+Fixed mapping is useful for stable latest-style paths. Preview fixed destinations with `run --dry-run`, especially before using `--force`.
-
-Fixed mapping is useful for stable latest-style paths. It is more destructive than archive-style publication because successive source bundles target the same destination root. Preview fixed destinations with `run --dry-run`, especially before using `--force`.
 
 ## Destination Links
 
-Destinations can record public URLs for published outputs:
-
 ```yaml
 links:
   base_url: https://reports.example.com/archive
   primary: auto
 ```
 
-`links.base_url` is an absolute `http` or `https` URL corresponding to the destination backend root. It may include a path prefix, but it must not include a query string or fragment. Distributor does not infer public URLs from backend config.
+- `links.base_url`: required when `links` is present. It must be an absolute `http` or `https` URL with a host and no query string or fragment.
+- `links.primary`: optional. Accepted values are `auto`, `html`, and `source`; default is `auto` when `links` is present.
 
-`links.primary` selects the top-level primary URL stored in destination state:
+`distributor` does not infer public URLs from backend config. Destination state URL fields are documented in [Destination State Contract](integrations/destination-state.md). Output URLs are built from `links.base_url`, the destination bundle path, and output paths using URL path semantics. `index.html` outputs produce directory-style URLs that omit the filename.
+
+Primary URL policies:
 
 - `auto`: prefer `index.html`, then generated HTML, then source outputs.
 - `html`: use the first generated HTML output.
 - `source`: use the first copied source output.
 
-If a destination has no `links` block, no URL metadata is generated. If a primary policy has no matching output, per-output URLs are still recorded and the top-level primary URL is omitted.
+If no output matches the primary policy, per-output URLs may still be recorded and the top-level primary URL is omitted.
 
-Output URLs are built from `links.base_url`, the destination bundle path, and the output path using URL path semantics. `index.html` outputs produce directory-style URLs that omit the filename.
+## Transfer Policy
 
-## Reference
+```yaml
+transfer:
+  on_destination_same: skip
+  on_destination_older: replace
+  on_destination_newer: skip
+  on_conflict: fail
+```
 
-Top level:
+Transfer fields and accepted values:
 
-- `server.http.bind`: optional HTTP bind address; defaults to `127.0.0.1:8080`.
+- `transfer.on_destination_same`: `skip` or `fail`. Default: `skip`.
-- `server.http.staging_root`: optional root for default HTTP upload staging paths; defaults to `/var/spool/distributor`.
+- `transfer.on_destination_older`: `replace` or `fail`. Default: `replace`.
-- `server.http.max_upload_size`: optional default upload size limit; defaults to `20MB`.
+- `transfer.on_destination_newer`: `skip`, `replace`, or `fail`. Default: `skip`.
-- `server.http.queue_size`: optional HTTP upload admission queue size; defaults to `16`.
+- `transfer.on_conflict`: `fail` or `replace`. Default: `fail`.
-- `server.http.max_concurrency`: optional HTTP upload worker concurrency; defaults to `1`.
-- `server.http.retention`: optional completed upload retention duration; defaults to `24h`.
-- `secrets.directory`: optional credential secrets directory.
-- `pipelines`: required non-empty list.
 
-Pipeline:
+`replace` for `on_destination_newer` and `on_conflict` is honored only when `run --force` is supplied. There is no config field that enables forced replacement by default.
-
-- `id`: required unique slug-like identifier.
-- `source`: required backend config.
-- `validation.on_digest_mismatch`: optional; defaults to `fail`; only `fail` is supported.
-- `destinations`: required non-empty destination list.
-
-Source backend:
-
-- `backend`: required.
-- `path`: required for `local` and `ssh`.
-- `host`: required for `ssh`.
-- `user`: optional for `ssh`; defaults to the current OS user when available.
-- `port`: optional for `ssh`; defaults to `22`.
-- `ssh_key_file`: optional for `ssh`.
-- `known_hosts`: optional for `ssh`; defaults to the service user's OpenSSH `known_hosts` path when available.
-- `host_key_policy`: optional for `ssh`; defaults to `accept-new`.
-- `endpoint`: required for `s3`.
-- `bucket`: required for `s3`.
-- `prefix`: optional for `s3`; leading and trailing slashes are trimmed.
-- `region`: optional for `s3`; defaults to `us-east-1`.
-- `force_path_style`: optional for `s3`; defaults to `true`. Set `false` only for services that require virtual-host addressing.
-- `credentials.access_key_id_env`: optional S3 credential environment variable name.
-- `credentials.secret_access_key_env`: optional S3 credential environment variable name.
-- `token_env`: required for `http_upload`; names the token environment variable or secret-file name.
-- `staging_path`: optional for `http_upload`; defaults below `server.http.staging_root` using the pipeline id.
-- `max_upload_size`: optional for `http_upload`; defaults to `server.http.max_upload_size`.
-
-Destination:
-
-- `id`: required unique slug-like identifier within the pipeline.
-- Backend fields: same accepted shape as source backends, with destination fields at the destination level.
-- `publish`: optional; defaults to source-only publication.
-- `transform`: required only for generated HTML publication.
-- `path_mapping.mode`: optional; defaults to `preserve_relative`. Accepted values are `preserve_relative` and `fixed`.
-- `links.base_url`: optional links block; when present, `base_url` is required and must be an absolute HTTP or HTTPS URL without query string or fragment.
-- `links.primary`: optional; defaults to `auto`. Accepted values are `auto`, `html`, and `source`.
-- `transfer`: optional; defaults described below.
-
-Accepted backend names:
-
-- `local`: executable; requires `path`.
-- `ssh`: executable; requires `host` and `path`.
-- `s3`: executable; requires `endpoint` and `bucket`.
-- `http_upload`: source-only configuration; requires `token_env`.
 
 ## Size And Duration Values
 
-Upload size fields use an integer plus one of the supported binary-size suffixes:
+Upload size fields must be YAML strings with an integer and one of these suffixes:
 
 - `B`
 - `KB`
 - `MB`
 - `GB`
 
-Suffix multipliers use powers of 1024. Size values must be greater than zero after defaults are applied.
+Suffix multipliers use powers of 1024. Values must be greater than zero after defaults are applied.
 
-HTTP retention uses Go-style duration strings such as `24h`, `90m`, or `168h`. Retention must be greater than zero after defaults are applied.
+Duration fields must be YAML strings accepted by Go duration parsing, such as `24h`, `90m`, or `168h`. Values must be greater than zero after defaults are applied.
-
-## SSH Backend
-
-SSH uses native SFTP. It can be used for sources, destinations, or both:
-
-```yaml
-backend: ssh
-host: example.com
-user: distributor
-port: 2222
-path: /remote/root
-ssh_key_file: /home/distributor/.ssh/id_ed25519
-known_hosts: /home/distributor/.ssh/known_hosts
-host_key_policy: accept-new
-```
-
-Authentication uses SSH agent identities first when `SSH_AUTH_SOCK` is set, then `ssh_key_file` if configured. Password authentication in YAML is not supported.
-
-Host key policies:
-
-- `strict`, `true`, and `"true"` require a matching known host key.
-- `accept-new` accepts and persists a new host key, but fails if an existing key changed. During `run --dry-run`, new host keys are accepted only for the current connection and are not persisted.
-- `off`, `false`, and `"false"` disable host key checking and are insecure.
-
-`accept-new` and `strict` use `known_hosts` when configured. If omitted, distributor uses the current service user's default OpenSSH `known_hosts` path where practical. `accept-new` fails when it needs to persist a new host key and no writable `known_hosts` path is available. It does not create a missing parent `.ssh` directory.
-
-## S3 Backend
-
-S3 uses the AWS SDK for Go v2 and supports S3-compatible endpoints:
-
-```yaml
-backend: s3
-endpoint: https://s3.example.com
-bucket: reports
-prefix: archive
-region: us-east-1
-force_path_style: true
-credentials:
-  access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
-  secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
-```
-
-`endpoint` and `bucket` are required. `prefix` is an optional backend root; it is treated as an object-key prefix, not a real directory. Prefixes must be clean slash-separated paths after trimming leading and trailing slashes. `http://` endpoints are allowed for explicitly configured local development or local S3-compatible test services.
-
-If either credential environment variable name is configured, both must be configured and both referenced variables must resolve to non-empty values through the real process environment or `secrets.directory`. Explicit credentials take precedence over the AWS SDK default credential chain. If credential environment variable names are omitted, the SDK default credential chain is used and `secrets.directory` values are not injected into the process environment.
-
-Publish policy:
-
-- `publish.source`: publish source artifacts.
-- `publish.html`: publish generated HTML artifacts from Markdown source files.
-
-At least one output type must be enabled. When `publish.html` is true, `transform.markdown_to_html.enabled` must be `true`.
-
-Markdown-to-HTML transform:
-
-- `transform.markdown_to_html.enabled`: enables Markdown-to-HTML generation for destinations with `publish.html: true`.
-- `transform.markdown_to_html.mode`: optional; defaults to `sidecar`. Accepted values are `sidecar` and `index`.
-- `transform.markdown_to_html.input`: optional source manifest path for `index` mode. It must identify a listed Markdown file.
-
-`sidecar` mode renders each manifest-listed `.md` file to a same-directory `.html` output. `index` mode renders one selected Markdown file to `index.html` at the destination bundle path. Enabled Markdown-to-HTML config is rejected when `publish.html` is false, and `input` is valid only with `mode: index`.
-
-Transfer policy:
-
-- `transfer.on_destination_same`: `skip` or `fail`; defaults to `skip`.
-- `transfer.on_destination_older`: `replace` or `fail`; defaults to `replace`.
-- `transfer.on_destination_newer`: `skip`, `replace`, or `fail`; defaults to `skip`.
-- `transfer.on_conflict`: `fail` or `replace`; defaults to `fail`.
-
-`replace` for `on_destination_newer` and `on_conflict` is honored only when `run --force` is used for that invocation. Force is CLI-only; there is no persistent config field that enables forced replacement by default.
 
 ## Defaults
 
 Defaults are applied after YAML decoding and before validation:
 
-- `validation.on_digest_mismatch: fail`
-- SSH `port: 22`
-- SSH `host_key_policy: accept-new`
-- S3 `region: us-east-1`
-- S3 `force_path_style: true`
 - `server.http.bind: 127.0.0.1:8080`
 - `server.http.staging_root: /var/spool/distributor`
 - `server.http.max_upload_size: 20MB`
 - `server.http.queue_size: 16`
 - `server.http.max_concurrency: 1`
 - `server.http.retention: 24h`
-- `source.staging_path: /var/spool/distributor/<pipeline id>` for `http_upload`
+- `validation.on_digest_mismatch: fail`
-- `source.max_upload_size: server.http.max_upload_size` for `http_upload`
+- SSH `port: 22`
-- `transform.markdown_to_html.mode: sidecar` when a Markdown-to-HTML transform block is present and mode is omitted
+- SSH `host_key_policy: accept-new`
-- `publish.source: true`
+- S3 `region: us-east-1`
-- `publish.html: false`
+- S3 `prefix`: leading and trailing slashes trimmed
+- S3 `force_path_style: true`
+- `http_upload` source `staging_path: <server.http.staging_root>/<pipeline id>`
+- `http_upload` source `max_upload_size: server.http.max_upload_size`
+- `publish.source: true` and `publish.html: false`
+- `transform.markdown_to_html.mode: sidecar` when a Markdown transform block is present and mode is omitted
 - `path_mapping.mode: preserve_relative`
 - `links.primary: auto` when a `links` block is present and `primary` is omitted
 - `transfer.on_destination_same: skip`
@@ -405,29 +408,31 @@ secrets:
   directory: /run/secrets/distributor
 ```
 
-Each regular file in the directory becomes an internal credential environment value named by the filename. Valid filenames must match `[A-Za-z_][A-Za-z0-9_]*`. Directories are ignored, and symlinks to regular files are followed. Exactly one trailing LF or CRLF is trimmed from each file; other whitespace is preserved.
+Each regular file in the directory becomes an internal credential environment value named by the filename. Valid filenames match `[A-Za-z_][A-Za-z0-9_]*`. Directories are ignored. Symlinks to regular files are followed. Exactly one trailing LF or CRLF is trimmed from each file; other whitespace is preserved.
 
-The resolver checks the real process environment first, then the secrets directory. If both define the same variable with different values, `run` prints a warning with the variable name and uses the real environment value. Secret values are not printed. The process environment is not modified, so SDK default credential chains see only real environment variables.
+Credential resolution checks the real process environment first, then `secrets.directory`. If both define the same name with different values, `run` emits a warning with the variable name and uses the real environment value. Secret values are not printed. The process environment is not modified, so SDK default credential chains see only real process environment variables.
 
-S3 credentials may name environment variables:
+Fields resolved through this resolver:
 
 - `credentials.access_key_id_env`
 - `credentials.secret_access_key_env`
+- `source.token_env` for `http_upload` sources
 
-HTTP upload tokens name one environment variable or secret-file name:
+## Maintained Examples
 
-- `source.token_env`
+Maintained examples live under [examples](../examples/). Config tests load every file listed here.
 
-## Examples
+Local examples:
 
-Maintained examples live under [examples](../examples/):
+- `local-to-local.yml`: minimal local-to-local config using absolute sample paths; load-tested, but paths should be adapted before running.
+- `local-publish.yml`: runnable local source publication used by the README quickstart.
+- `local-html.yml`: local sidecar HTML publication.
+- `local-index.yml`: local `index.html` publication.
+- `fan-out.yml`: local fan-out publication to source and HTML destinations.
+- `archive-and-latest.yml`: local archive plus fixed latest publication.
+- `http-upload-local.yml`: local HTTP upload server config; requires `DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN` in the process environment or as a secret-file name before running `serve`.
 
-- `local-to-local.yml`: minimal local config.
+Environment-gated remote examples:
-- `local-publish.yml`: runnable local source publication.
+
-- `local-html.yml`: runnable local HTML publication.
+- `ssh-destination.yml`: local-to-SSH publication; replace host, user, path, key, and known-host values for an SSH/SFTP endpoint you control.
-- `local-index.yml`: runnable local `index.html` publication.
+- `s3-destination.yml`: local-to-S3 publication; replace endpoint, bucket, prefix, region, and credential variable names for an S3-compatible service you control.
-- `fan-out.yml`: runnable local fan-out publication to source and HTML destinations.
-- `archive-and-latest.yml`: runnable local fan-out publication to an archive destination and a fixed latest destination.
-- `http-upload-local.yml`: local HTTP upload server example with a token environment variable reference.
-- `ssh-destination.yml`: environment-gated local-to-SSH publication example.
-- `s3-destination.yml`: environment-gated local-to-S3 publication example.

docs/integrations/destination-state.md

@@ -0,0 +1,99 @@
+# Destination State Contract
+
+Audience: operators, integrators, and maintainers who inspect or reason about destination `.distributor.json` files.
+
+Each managed destination bundle path contains `.distributor.json`. This file is the destination sentinel and state record used for comparison, skip, replacement, and recovery decisions.
+
+## State Schema
+
+Current schema version: `1`.
+
+```json
+{
+  "schema_version": 1,
+  "distributor_version": "dev",
+  "pipeline_id": "reports",
+  "destination_id": "archive",
+  "published_at": "2026-06-04T12:00:00Z",
+  "source": {
+    "manifest": {
+      "schema_version": 1,
+      "id": "reports.example.2026-06-04",
+      "digest": "sha256:...",
+      "created": "2026-06-04T11:55:00Z",
+      "files": [
+        {"path": "report.md", "sha256": "sha256:...", "size": 1234}
+      ]
+    }
+  },
+  "links": {
+    "primary_url": "https://reports.example.com/archive/report.html"
+  },
+  "outputs": [
+    {
+      "path": "report.html",
+      "kind": "generated",
+      "source_path": "report.md",
+      "transform": "markdown_to_html",
+      "url": "https://reports.example.com/archive/report.html",
+      "sha256": "sha256:...",
+      "size": 2345
+    }
+  ]
+}
+```
+
+Required fields:
+
+- `schema_version`: must be `1`.
+- `pipeline_id`: configured pipeline id that wrote the state.
+- `destination_id`: configured destination id that wrote the state.
+- `published_at`: RFC3339 publication timestamp.
+- `source.manifest`: embedded source bundle manifest.
+- `outputs`: output records array; empty is allowed, but the field is required.
+
+Optional fields:
+
+- `distributor_version`: application version string when available.
+- `links.primary_url`: absolute HTTP or HTTPS URL selected by destination link policy.
+
+## Output Records
+
+Each output record has:
+
+- `path`: destination-relative output path.
+- `kind`: `source` or `generated`.
+- `source_path`: source manifest path used for the output.
+- `transform`: required for `generated` outputs; omitted for copied source outputs.
+- `url`: optional absolute HTTP or HTTPS URL for the output.
+- `sha256`: lowercase `sha256:<64 hex>` digest of the output bytes.
+- `size`: output byte size, zero or greater.
+
+Output paths must be unique and use clean relative slash-separated path rules.
+
+## Comparison Semantics
+
+`distributor` compares the current source manifest to destination state before writing:
+
+- No state and no content: publish new outputs.
+- No state and existing content: treat the destination as unmanaged.
+- Matching embedded source manifest: skip.
+- Same source id with older `created`: replace if policy allows.
+- Same source id with newer `created`: skip by default.
+- Same source id and same `created` with different digest: conflict.
+- Different source id, pipeline id, or destination id: conflict.
+- Invalid state JSON or invalid state fields: conflict.
+
+Normal replacement deletes only managed output paths recorded in `outputs` plus `.distributor.json`. Forced replacement deletes the bounded destination bundle path.
+
+## Boundaries
+
+Destination state is internal managed state written by `distributor`. Operators may inspect it during recovery, but normal workflows should not edit it by hand. Source `manifest.json` is not copied as destination state.
+
+## Tests
+
+Before changing this contract, inspect and run:
+
+```sh
+go test ./internal/state ./internal/publish
+```

docs/integrations/http-upload.md

@@ -0,0 +1,108 @@
+# HTTP Upload API Contract
+
+Audience: producers, operators, and maintainers integrating with `distributor serve`.
+
+`distributor serve` exposes a local HTTP upload API for pipelines whose source backend is `http_upload`. Each bearer token maps to exactly one configured pipeline.
+
+## Authentication
+
+Uploads authenticate with:
+
+```text
+Authorization: Bearer <token>
+```
+
+Token values are resolved from the configured `source.token_env` through the process environment or `secrets.directory`. Tokens are not configured as YAML literal values.
+
+Requests that include `pipeline` or `pipeline_id` query parameters are rejected. The bearer token selects the pipeline.
+
+## Endpoints
+
+### `GET /healthz`
+
+Returns `200 OK` when the server is running:
+
+```json
+{"status":"ok"}
+```
+
+### `POST /upload`
+
+Accepts one source bundle archive and returns after the archive is staged and validated.
+
+Accepted content types:
+
+- `application/x-tar`
+- `application/gzip`
+- `application/x-gzip`
+
+Successful admission returns `202 Accepted`:
+
+```json
+{"run_id":"reports.20260604T120000Z.abcdef12","status":"accepted"}
+```
+
+Common error responses:
+
+- `400`: pipeline query supplied, archive rejected, malformed archive, or invalid staged source bundle.
+- `401`: missing, empty, or unknown bearer token.
+- `413`: upload body exceeds the selected pipeline size limit.
+- `415`: unsupported content type.
+- `503`: upload queue is full.
+
+Error bodies use:
+
+```json
+{"error":"<message>"}
+```
+
+### `GET /runs/<run-id>`
+
+Returns an in-memory status record while retained:
+
+```json
+{
+  "run_id": "reports.20260604T120000Z.abcdef12",
+  "pipeline_id": "reports",
+  "status": "succeeded",
+  "accepted_at": "2026-06-04T12:00:00Z",
+  "started_at": "2026-06-04T12:00:01Z",
+  "finished_at": "2026-06-04T12:00:02Z",
+  "report": {}
+}
+```
+
+Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Failed records include `error`. Succeeded and failed records may include a run report.
+
+Unknown, malformed, expired, or process-lost run ids return `404`.
+
+## Archive Contract
+
+Upload archives must be uncompressed tar or gzip-compressed tar. The archive must contain exactly one root-level `manifest.json` and all manifest-listed files.
+
+Archive entry rules:
+
+- Paths must be clean relative slash-separated paths.
+- Absolute paths, backslashes, `.` and `..` segments, duplicate files, and nested `manifest.json` entries are rejected.
+- Only directories and regular files are accepted.
+- Symlinks, hardlinks, devices, FIFOs, sockets, and other entry types are rejected.
+
+The uploaded archive size and extracted bundle size are bounded by the selected pipeline's `source.max_upload_size`. Extracted file count is also bounded by the implementation.
+
+## Queue And Retention
+
+`server.http.queue_size` bounds accepted-but-not-started uploads plus uploads being staged. `server.http.max_concurrency` bounds publishing concurrency. The coordinator does not run two uploads for the same pipeline concurrently.
+
+Completed status records expire after `server.http.retention`; expiration removes committed staged bundle directories for completed uploads. Server restart clears queue state and status records.
+
+## Boundaries
+
+The HTTP API does not expose pipeline selection by request parameter, TLS, public routing policy, or durable status storage. Put public access controls, TLS termination, and rate limiting in deployment infrastructure.
+
+## Tests
+
+Before changing this contract, inspect and run:
+
+```sh
+go test ./internal/app ./internal/ingest
+```

docs/integrations/markdown.md

@@ -1,27 +1,27 @@
 # Markdown Integration
 
-## Purpose
+Audience: operators and maintainers who rely on generated HTML outputs from Markdown source files.
 
-Markdown-to-HTML is the only implemented external file-format integration. This note documents the renderer behavior that is externally visible in generated destination artifacts.
+Markdown-to-HTML is an implemented file-format integration used by destination transform policy.
 
 ## Dependency
 
-Rendering uses `github.com/yuin/goldmark`. The exact dependency version is pinned in `go.mod`; review that file before changing renderer behavior or diagnosing version-specific output changes.
+Rendering uses `github.com/yuin/goldmark`. The exact version is pinned in `go.mod`.
 
-## Renderer behavior
+## Renderer Behavior
 
-`internal/transform/markdown.New` constructs the renderer with `goldmark.New()` and no project-specific extensions or renderer options.
+The transformer constructs `goldmark.New()` with no project-specific extensions, parser options, renderer options, templates, CSS, or metadata injection.
 
-The transform supports two output modes:
+Supported output modes:
 
-- `sidecar`: reads each source bundle file ending in `.md` and generates an HTML sidecar in the same logical directory. The output path replaces the `.md` suffix with `.html`, so `report.md` produces `report.html`. Non-Markdown source files produce no Markdown outputs.
+- `sidecar`: renders each source manifest file ending in `.md` and writes a generated output beside it with the `.md` suffix replaced by `.html`.
-- `index`: renders one selected Markdown source to `index.html` at the destination bundle path.
+- `index`: renders one Markdown source file to `index.html` at the destination bundle path.
 
-In `index` mode, `transform.markdown_to_html.input` can name the source manifest path to render. If `input` is omitted, the manifest must list exactly one Markdown file. The selected input must be a safe relative source path, must be listed in the source manifest, and must end in `.md`.
+In `index` mode, `transform.markdown_to_html.input` may name the source manifest path to render. If `input` is omitted, the source manifest must list exactly one `.md` file. The selected input must be a clean relative source path, must be listed in the source manifest, and must end in `.md`.
 
 Raw HTML embedded in Markdown is not passed through by the current renderer behavior. Tests allow Goldmark's disabled-or-escaped raw HTML output forms and reject literal script tags in generated HTML.
 
-## Wrapper
+## HTML Wrapper
 
 Rendered Markdown body HTML is wrapped in a fixed document shell:
 
@@ -33,7 +33,7 @@ Rendered Markdown body HTML is wrapped in a fixed document shell:
 
 The wrapper is deterministic and does not read configuration, templates, CSS, or source manifest metadata.
 
-## Output metadata
+## Output Metadata
 
 Generated outputs record:
 
@@ -43,18 +43,16 @@ Generated outputs record:
 - SHA-256 digest of the wrapped HTML bytes;
 - byte size of the wrapped HTML bytes.
 
+Destination state stores generated outputs with `kind: generated`, `source_path`, `transform`, `sha256`, `size`, and optional `url`.
+
 ## Boundaries
 
-Markdown rendering does not mutate source bundles, publish files, write `.distributor.json`, select outputs, or choose transfer actions. Publish planning decides whether generated HTML is selected for a destination.
+Markdown rendering does not mutate source bundles, publish files, write `.distributor.json`, select destination actions, or choose transfer policy. Publish planning decides whether generated HTML is selected for a destination and destination state records the generated output metadata.
-
-Publish planning chooses the configured mode and input for each destination. Markdown rendering does not inspect destinations, publish files, write `.distributor.json`, or choose transfer actions.
 
 ## Tests
 
 Before changing Markdown renderer behavior, inspect and run:
 
-```bash
+```sh
 go test ./internal/transform/markdown
 ```
-
-The tests cover sidecar naming, index input selection, ignored non-Markdown files, raw HTML handling, deterministic output, digest metadata, and size metadata.

docs/integrations/s3.md

@@ -0,0 +1,73 @@
+# S3-Compatible Storage Integration
+
+Audience: operators and maintainers configuring S3-compatible sources or destinations.
+
+The S3 backend uses the AWS SDK for Go v2 against a configured S3-compatible endpoint.
+
+## Dependencies
+
+Runtime S3 support uses:
+
+- `github.com/aws/aws-sdk-go-v2`
+- `github.com/aws/aws-sdk-go-v2/config`
+- `github.com/aws/aws-sdk-go-v2/credentials`
+- `github.com/aws/aws-sdk-go-v2/service/s3`
+- `github.com/aws/smithy-go`
+
+Exact versions are pinned in `go.mod`.
+
+## Config Contract
+
+Required fields:
+
+- `backend: s3`
+- `endpoint`
+- `bucket`
+
+Optional fields:
+
+- `prefix`: backend root object-key prefix; leading and trailing slashes are trimmed.
+- `region`: defaults to `us-east-1`.
+- `force_path_style`: defaults to `true` through config defaults.
+- `credentials.access_key_id_env`
+- `credentials.secret_access_key_env`
+
+Credential environment variable names must either both be configured or both be omitted. When configured, values are resolved through the process environment or `secrets.directory` before opening the backend. When omitted, the AWS SDK default credential chain is used.
+
+## Object Key Mapping
+
+The configured `bucket` plus optional `prefix` is the backend root. Logical storage paths are joined under that prefix using slash-separated object keys.
+
+Prefixes and logical paths must be clean relative slash-separated paths. Prefixes are object-key prefixes, not real directories.
+
+## Storage Behavior
+
+The adapter uses these S3 operations:
+
+- `HeadObject` for stat and overwrite checks.
+- `GetObject` for reads.
+- `PutObject` for writes.
+- `ListObjectsV2` for walks and prefix deletion planning.
+- `DeleteObject` for managed cleanup and replacement.
+
+Writes buffer the input and set `ContentLength`. If no content type is supplied by the caller, the adapter infers a content type from the logical path.
+
+Normal replacement and failed-write cleanup delete only managed output objects plus `.distributor.json`. Forced replacement deletes objects under the bounded destination bundle prefix. The backend does not manage bucket versioning, lifecycle rules, object lock, or delete markers.
+
+## Error Mapping
+
+S3 not-found responses are translated into shared storage not-found errors where practical. Other service, credential, network, or endpoint errors are surfaced through storage errors with backend and logical path context.
+
+## Boundaries
+
+The S3 integration does not create buckets, alter bucket policy, configure TLS bypass, manage public URLs, or infer website URLs from endpoint or bucket settings. Public URL metadata is configured separately with destination `links`.
+
+## Tests
+
+Before changing this integration, inspect and run:
+
+```sh
+go test ./internal/adapters/s3
+```
+
+Live S3-compatible tests are opt-in and gated by environment variables in the adapter test package.

docs/integrations/source-bundle.md

@@ -0,0 +1,92 @@
+# Source Bundle Contract
+
+Audience: producer developers, integrators, and maintainers who create or validate source bundles consumed by `distributor`.
+
+A source bundle is a directory containing `manifest.json` and every regular file listed by that manifest. This is the producer-to-`distributor` file-format contract.
+
+## Manifest Schema
+
+Current schema version: `1`.
+
+```json
+{
+  "schema_version": 1,
+  "id": "reports.example.2026-06-04",
+  "digest": "sha256:...",
+  "created": "2026-06-04T12:00:00Z",
+  "files": [
+    {
+      "path": "report.md",
+      "sha256": "sha256:...",
+      "size": 1234
+    }
+  ]
+}
+```
+
+Required manifest fields:
+
+- `schema_version`: must be `1`.
+- `id`: non-empty bundle identifier.
+- `digest`: lowercase `sha256:<64 hex>` digest of the ordered `files` list.
+- `created`: RFC3339 timestamp.
+- `files`: non-empty ordered list of file records.
+
+Required file fields:
+
+- `path`: bundle-relative slash-separated file path.
+- `sha256`: lowercase `sha256:<64 hex>` digest of the file bytes.
+- `size`: file size in bytes, zero or greater.
+
+## Path Rules
+
+Manifest file paths must be clean relative slash-separated paths. They must not be empty, absolute, contain backslashes, contain `.` or `..` segments, include empty path segments, or normalize to a different path.
+
+Any basename of `manifest.json` or `.distributor.json` is reserved, including nested occurrences such as `nested/manifest.json`.
+
+Listed files must be regular files. Symlinks and other special file types are rejected during local bundle validation and manifest building.
+
+## Digest Rules
+
+File digests use SHA-256 over each file's raw bytes.
+
+The bundle digest is SHA-256 over the canonical JSON-like payload for the ordered file records. The payload is constructed as:
+
+```text
+[{"path":"<path>","sha256":"<sha256>","size":<size>},...]
+```
+
+File order is significant. Explicit file lists preserve caller order. Scan mode sorts paths in ascending slash-path order.
+
+## Producer APIs
+
+Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build and validate this contract:
+
+- `BuildManifest`: builds a manifest from explicit file paths or scan mode.
+- `WriteManifest`: writes `manifest.json`, optionally replacing an existing manifest.
+- `WriteBundle`: copies source files into a complete bundle, validates it, and promotes it into place.
+- `LoadManifest`, `ParseManifest`, `ValidateManifest`, and `ValidateBundle`: parse and validate local bundles.
+- `FileDigest`, `BundleDigest`, and `ValidateDigest`: digest helpers.
+
+CLI producers can use:
+
+```sh
+go run ./cmd/distributor manifest create <bundle-path> --id <bundle-id>
+go run ./cmd/distributor validate <bundle-path>
+```
+
+## Scan Mode
+
+Manifest scan mode walks the local bundle root recursively, includes regular files, includes dotfiles, skips files whose basename is `manifest.json` or `.distributor.json`, rejects symlinks, and sorts paths before building the manifest.
+
+## Boundaries
+
+The source bundle manifest does not configure routing, destination selection, public URLs, credentials, transforms, notification behavior, or storage backends. Those concerns belong in `distributor` configuration and destination state.
+
+## Tests
+
+Before changing this contract, inspect and run:
+
+```sh
+go test ./pkg/bundle ./internal/bundle
+```

docs/integrations/ssh-sftp.md

@@ -0,0 +1,71 @@
+# SSH/SFTP Integration
+
+Audience: operators and maintainers configuring SSH/SFTP sources or destinations.
+
+The SSH backend uses native SSH and SFTP libraries. It does not call `ssh`, `scp`, or `rsync`.
+
+## Dependencies
+
+Runtime SSH support uses:
+
+- `golang.org/x/crypto/ssh`
+- `golang.org/x/crypto/ssh/agent`
+- `golang.org/x/crypto/ssh/knownhosts`
+- `github.com/pkg/sftp`
+
+Exact versions are pinned in `go.mod`.
+
+## Config Contract
+
+Required fields:
+
+- `backend: ssh`
+- `host`
+- `path`
+
+Optional fields:
+
+- `user`: defaults to the current OS user when available.
+- `port`: defaults to `22`.
+- `ssh_key_file`: private key path.
+- `known_hosts`: OpenSSH known-hosts file path.
+- `host_key_policy`: `strict`, `accept-new`, or `off`; defaults to `accept-new`.
+
+## Authentication
+
+Authentication methods are attempted in this order:
+
+1. SSH agent identities when `SSH_AUTH_SOCK` is set.
+2. The private key configured by `ssh_key_file`.
+
+Password authentication is not configured in YAML. If neither an agent nor key file is available, opening the backend fails.
+
+## Host Key Policy
+
+- `strict`: requires a matching known host key.
+- `accept-new`: accepts and persists an unknown host key, but rejects changed known keys.
+- `off`: disables host key checking.
+
+When `known_hosts` is omitted and checking is enabled, the adapter uses the current user's default OpenSSH `known_hosts` path when available. During dry runs, accepted unknown host keys are not persisted.
+
+## Storage Behavior
+
+The configured `path` is the backend root. All source discovery, destination paths, reads, writes, state files, and deletes operate on logical paths below that root.
+
+The adapter rejects symlink ancestors for reads and writes. Reads require regular files. Writes create parent directories and prefer atomic temp-file-plus-rename writes when requested. Walk output is sorted through the shared storage walker.
+
+Managed cleanup and normal replacement delete only managed output paths plus `.distributor.json`. Forced replacement deletes the bounded destination bundle path.
+
+## Boundaries
+
+The SSH backend does not configure passwords, jump hosts, shell commands, `rsync`, host-key bypass warnings beyond command output, or broad recursive deletion outside the destination bundle path.
+
+## Tests
+
+Before changing this integration, inspect and run:
+
+```sh
+go test ./internal/adapters/ssh
+```
+
+Live SSH tests are opt-in and gated by environment variables in the adapter test package.

docs/internal/app.md

@@ -1,254 +1,66 @@
 # Application Orchestration
 
+Audience: developers and LLM coding agents changing `internal/app`.
+
 ## Purpose
 
-`internal/app` owns the top-level application use cases. It coordinates
+`internal/app` owns top-level application use cases: run, single-pipeline run, staged-source run, validate, inspect, manifest creation, and HTTP upload serving. It coordinates config loading, secret resolution, backend construction, source discovery, destination selection, publish planning/execution, notification handoff, output projection, and upload coordination.
-configuration loading, secret resolution, backend construction, source bundle
-discovery, destination selection, publish planning, publish execution,
-notification handoff, run reporting, and in-memory run coordination.
 
-The package is the boundary between callers and lower-level domain packages. It
+## Inputs And Outputs
-does not own manifest validation rules, destination state comparison, storage
-path rules, output planning, transform rendering, or backend-specific behavior.
 
-## Use Cases
+Inputs include app option structs, contexts, config paths, pipeline ids, local source roots, dry-run/force flags, output format, stdout writers, HTTP requests, and optional notifier implementations.
 
-`Run` is the CLI-facing all-pipeline entrypoint. It accepts a context, optional
+Outputs include `RunReport`, validate/inspect/manifest results, CLI text/JSON projections, HTTP upload responses, upload status records, and errors. Destination-scoped failures can return a partial run report plus an aggregated error; fatal setup failures return before a complete report exists.
-config path, dry-run flag, force flag, stdout writer, output format, and
-optional notifier. It runs every configured pipeline, builds a `RunReport`, and
-projects the report to text or JSON when stdout is supplied.
 
-`RunPipeline` is the app-layer single-pipeline entrypoint. It accepts a context,
+## Boundaries
-config path, pipeline ID, dry-run flag, force flag, and optional notifier. It
-loads the same config as `Run`, narrows execution to exactly one configured
-pipeline, and returns a `RunReport` without writing command output.
 
-`RunPipelineWithLocalSource` is the app-layer single-pipeline entrypoint for an
+`internal/app` wires packages together but does not own manifest validation rules, destination state comparison, storage path rules, publish safety policy, transform rendering, config schema validation, or backend protocol behavior.
-already prepared local source bundle root. It accepts the same pipeline
-selection and execution options as `RunPipeline` plus a local source root path.
-It loads config, selects one configured pipeline, opens the supplied source
-root as a local backend, validates exactly that root bundle, and then uses the
-same destination fan-out path as normal runs.
 
-`Validate` and `Inspect` accept either a local path or one configured pipeline
+User-facing command parsing stays in `internal/cli`. User-facing config reference stays in `docs/config.md`. External contracts live under `docs/integrations/`.
-source. They share source backend construction with run workflows and never open
-destination backends.
 
-`Serve` is the CLI-facing HTTP upload server entrypoint. It loads config,
+## Config Fields Used
-loads the configured secrets directory, resolves upload bearer tokens for
-configured `http_upload` sources, creates an `UploadCoordinator`, binds
-`server.http.bind`, and serves the upload API until its context is cancelled.
 
-## Run Reports
+The package consumes the loaded `config.Config`: `server.http`, `secrets.directory`, pipeline ids, source and destination backend fields, validation policy, publish policy, transform policy, path mapping, links, and transfer policy.
 
-`RunReport` is the structured result model for run workflows. It includes
+Config fields are validated and defaulted by `internal/config` before app workflows use them.
-dry-run state, pipeline summaries, action records, output metadata, summary
-counters, warnings, and destination-scoped output errors.
 
-Text and JSON run output are projections of `RunReport`. JSON tags on report
+## Adapters Used
-records match the CLI JSON output contract. Text output preserves the CLI
-summary shape while keeping output rendering outside the core planning and
-execution loop.
 
-Destination-scoped failures produce a report plus an aggregated error. Fatal
+The app backend factory registers runtime storage adapters for local filesystem, SSH/SFTP, and S3-compatible storage. It resolves explicit credentials through the config-owned environment resolver before opening S3 backends.
-setup failures, such as config loading, source open, or source discovery
-failures, return before a complete run report is available.
 
-## Run Flow
+The app layer registers default transforms, including Markdown-to-HTML, and supplies a transform resolver to publish planning. It uses `notify.Noop` when no notifier is supplied.
 
-The app runner:
+## State And Manifest Behavior
 
-1. loads config from the supplied path or `config.DefaultConfigPath`;
+Run workflows discover and validate source bundles through `internal/bundle`. Destination state actions are prepared and written through `internal/publish` and `internal/state`; the app layer records report projections of those actions and results.
-2. loads configured secret files into a config-owned environment resolver;
-3. builds the app-level backend factory and transform registry;
-4. opens each selected pipeline source backend;
-5. discovers validated source bundles from the source root;
-6. selects source bundles for each destination according to path mapping;
-7. opens destination backends independently;
-8. builds publish plans for selected bundle and destination combinations;
-9. records warnings, action records, output metadata, and summary counters;
-10. executes publish or replacement plans unless dry-run is enabled;
-11. invokes the notifier after successful publish or replacement actions;
-12. returns the structured report and any aggregated destination failures.
 
-`RunPipeline` follows the same flow after selecting a single configured
+HTTP uploads stage and validate archives before enqueueing a pipeline run with a local staged source root.
-pipeline. It uses the same backend factory, secret loading, transform registry,
-warning generation, destination planning, publish execution, notification
-behavior, and failure aggregation as `Run`.
 
-`RunPipelineWithLocalSource` follows the same flow after pipeline selection
+## Skip And Resume Behavior
-except for source opening and source discovery. It opens the supplied local
-source root directly, validates the root bundle before opening any destinations,
-and passes the resulting local source backend and bundle into the same
-destination planning and execution loop. Destination code receives the normal
-storage backend and bundle values and does not depend on how the source root was
-prepared.
 
-## Upload Coordination
+Fan-out destinations are independent. A destination failure is recorded and does not prevent later destinations from being attempted. Dry-run builds plans and reports without destination writes, destination state writes, notifier calls, or SSH known-host persistence.
 
-`UploadCoordinator` owns in-memory coordination for asynchronous upload
+HTTP upload status is in memory. Accepted jobs move through accepted, queued, running, succeeded, or failed states and expire after configured retention.
-processing. It admits uploads for configured `http_upload` pipelines, generates
-run IDs, tracks status records, stages accepted archives through
-`internal/ingest`, and executes the selected pipeline through
-`RunPipelineWithLocalSource`.
 
-Upload run IDs use:
+## Failure Behavior
 
-```text
+Runtime setup fails for config load, config validation, secret loading, or credential resolution errors. Source setup failures stop the affected run before destination planning. Destination open, planning, execution, and notification failures are recorded as destination failures where a partial result exists.
-<pipeline id>.<UTC timestamp>.<random suffix>
-```
 
-The timestamp uses `YYYYMMDDThhmmssZ` UTC format and the suffix is filesystem
+HTTP upload startup fails if upload tokens are missing, empty, or duplicated. Upload requests can fail during authentication, content-type validation, queue admission, archive staging, source validation, or later publish execution.
-safe.
 
-The coordinator records these statuses:
+## Tests To Inspect
 
-- `accepted`
+- `internal/app/*_test.go`
-- `queued`
+- `internal/cli/root_test.go`
-- `running`
+- `internal/config/*_test.go`
-- `succeeded`
+- `internal/ingest/*_test.go`
-- `failed`
+- `internal/publish/*_test.go`
-- `expired`
 
-Admission is bounded by `server.http.queue_size`. Full queues are rejected
+## Architectural Invariants
-before the upload body is staged. Execution is bounded by
-`server.http.max_concurrency`, and only one upload for a given pipeline may run
-at a time. Later uploads for the same pipeline remain queued until the active
-run finishes.
 
-Completed records retain the final run report or error text until
+- App orchestration owns wiring, not low-level policy.
-`server.http.retention` elapses. Expiration removes completed status records and
+- Dry-run must not write outputs, destination state, notifier events, or SSH known-host entries.
-their committed staged bundle directories. The coordinator is memory-only and
+- Fan-out destinations remain independent after a destination-scoped failure.
-does not persist queue state, status records, or run reports.
+- Secret values are never printed; warnings may name variables only.
-
+- Upload admission stages and validates a bundle before returning a run id.
-## HTTP Upload Server
+- Runtime backend registration remains app-owned.
-
-The HTTP upload server is app-layer transport wiring around
-`UploadCoordinator`. It owns request authentication, route dispatch, HTTP status
-mapping, and JSON response projection. Bundle staging and publication remain in
-the coordinator and staged-source run path.
-
-Server startup resolves every configured `http_upload` source `token_env`
-through the config-owned environment resolver after `secrets.directory` has
-been loaded. Startup fails when a token is missing, empty, or duplicates another
-upload pipeline token. Error messages identify environment variable names and
-pipeline ids, but not token values.
-
-Routes:
-
-- `GET /healthz`: returns `200` after config, secrets, tokens, coordinator, and route setup succeed.
-- `POST /upload`: accepts authenticated tar and tar.gz archives and returns an accepted run id.
-- `GET /runs/<run_id>`: returns the current in-memory upload status record or `404`.
-
-The upload token maps to exactly one configured pipeline. Producers do not
-submit pipeline ids, and submitted `pipeline` or `pipeline_id` query values are
-rejected. Full queues are rejected before the request body is read. Oversized
-uploads, unsupported content types, invalid bearer tokens, full queues, and
-unknown status records are mapped to stable HTTP status codes without returning
-secret token values.
-
-## Coordination
-
-`PipelineRunCoordinator` wraps `RunPipeline` with in-memory admission control.
-It allows different pipeline IDs to run concurrently and rejects a second active
-run for the same pipeline ID.
-
-Coordinator records contain a run ID, pipeline ID, status, timestamps, completed
-report, and error text when applicable. Active state is memory-only and is
-cleared after success, failure, unknown pipeline ID, or context cancellation.
-
-The admission context is checked before a run is accepted. Once accepted, the
-run uses the coordinator lifetime context, so caller cancellation can stop
-waiting for admission without owning the actual run lifetime.
-
-The coordinator does not queue duplicate runs, persist run records, or define
-transport endpoints.
-
-## Errors
-
-`Run` returns immediately for config loading errors, context cancellation before
-work starts, source open errors, and source discovery errors.
-
-`RunPipeline` returns `PipelineNotFoundError` when the requested pipeline ID is
-not configured. Callers can detect that condition with `IsPipelineNotFound`.
-
-`RunPipelineWithLocalSource` also returns `PipelineNotFoundError` for an unknown
-pipeline ID. It returns before destination opening when the supplied local
-source root is missing, cannot be opened, or does not validate as one complete
-source bundle.
-
-Per-destination backend, planning, execution, and notification errors are
-aggregated into one run error after remaining destinations have been attempted.
-Destination diagnostics include pipeline ID, destination ID, backend, and
-bundle path.
-
-`PipelineRunCoordinator` returns `DuplicatePipelineRunError` when the same
-pipeline already has an active run. Callers can detect that condition with
-`IsDuplicatePipelineRun`.
-
-Stdout write errors are returned immediately because the caller's requested
-output stream can no longer be trusted.
-
-## Package Layout
-
-Run helpers are grouped by responsibility:
-
-- `run.go`: `Run`, `RunPipeline`, and shared run orchestration.
-- `run_output.go`: `RunReport`, action/output records, and text/JSON report projection.
-- `run_summary.go`: summary counters.
-- `run_failures.go`: destination failure aggregation and partial-result detection.
-- `run_selection.go`: destination bundle selection, path mapping decisions, and fixed-path warnings.
-- `run_warnings.go`: secret and SSH warning records.
-- `run_notify.go`: notification event projection and action filtering.
-- `run_coordinator.go`: in-memory run admission, run IDs, status records, and duplicate-run errors.
-- `upload_coordinator.go`: in-memory upload admission, queueing, status tracking, staging handoff, and staged-source execution.
-- `upload_http.go`: HTTP upload authentication, routes, JSON response projection, and HTTP error mapping.
-- `serve.go`: config/secrets loading and HTTP server startup.
-- `backends.go`: app-level backend factory wiring.
-- `transforms.go`: app-level transform registry wiring.
-- `source_select.go`: configured-source selection shared by `validate` and `inspect`.
-
-## Backend And Transform Wiring
-
-The app-level backend factory registers local, SSH, and S3 backends for runtime
-execution. Source and destination backend config is converted through a shared
-app-local open spec before adapter construction.
-
-Credential references are resolved through the config environment resolver.
-Production app code must not read backend credential environment variables
-directly.
-
-The app-level transform registry registers Markdown-to-HTML through
-`internal/transform/markdown`. Lower-level publish code receives a resolver and
-does not import concrete transform implementations.
-
-## Dry-Run Behavior
-
-Dry-run loads config, opens backends, discovers bundles, inspects destinations,
-resolves transforms, and builds publish plans. It does not write destination
-outputs, write `.distributor.json`, delete managed outputs, perform forced
-prefix deletion, or invoke notifications.
-
-## Tests
-
-Before changing app orchestration, inspect tests under:
-
-- `internal/app`
-- `internal/cli`
-- `internal/publish`
-
-Use focused app tests for report structure, single-pipeline execution,
-coordinator admission, warning generation, notification behavior, and
-partial-result aggregation.
-
-## Invariants
-
-- One source fans out to each destination independently.
-- Destination failures do not prevent later destinations from being planned.
-- Destination-scoped failures still produce a structured report plus an aggregated error.
-- Dry-run must not mutate destination storage or invoke notifications.
-- `RunPipeline` must use the same run path as `Run` after pipeline selection.
-- Duplicate in-flight runs are rejected only for the same pipeline ID.
-- Different pipeline IDs may run concurrently.
-- Concrete backend and transform registration stays at the app layer.
-- The default notifier is `notify.Noop`.

docs/internal/bundle.md

@@ -1,55 +1,53 @@
-# Bundles
+# Source Bundle Internals
+
+Audience: developers and LLM coding agents changing `internal/bundle`.
 
 ## Purpose
 
-`internal/bundle` discovers and validates source bundles through the storage interface. The source manifest model, manifest parsing, manifest validation, path rules, digest calculation, and producer-side local writer come from `pkg/bundle` so producer-facing APIs and distributor validation share one manifest contract.
+`internal/bundle` discovers and validates source bundles through the storage abstraction. It adapts the public producer-facing source manifest contract from `pkg/bundle` to local, SSH/SFTP, S3-compatible, and test storage backends.
 
-## Inputs and outputs
+## Inputs And Outputs
 
-Input is a backend-rooted directory tree containing one or more `manifest.json` files. Output is a deterministic list of validated bundles with relative bundle paths and normalized manifest data.
+Inputs are a context, a `storage.Backend`, and a source-root prefix or bundle root path. Outputs are sorted `Bundle` records containing the source-root-relative bundle path and validated manifest.
-
-## Manifest behavior
-
-The source manifest requires:
-
-- `schema_version: 1`
-- `id`
-- `digest`
-- `created`
-- non-empty `files`
-
-Each file requires `path`, `sha256`, and `size`. Digests must use lowercase `sha256:<64 hex>` format. `created` must parse as RFC3339.
-
-`pkg/bundle.ValidateDigest` is the canonical digest format validator for producer-facing and internal code. `internal/bundle.ValidateDigest` delegates to that public validator so source manifests and destination state use the same digest grammar.
-
-## Validation
-
-`pkg/bundle.ValidateManifest` owns normalized source manifest semantics: schema version, id, digest format, timestamp presence, file list presence, source path safety, duplicate file paths, reserved paths, file digest format, non-negative file sizes, and the top-level bundle digest.
-
-Storage-backed bundle validation in `internal/bundle` additionally checks file existence, regular-file type, file size, and per-file SHA-256 for configured storage backends.
-
-The bundle digest is SHA-256 of a deterministic JSON array of file records in manifest order with fields `path`, `sha256`, and `size`.
-
-## Discovery
-
-Discovery walks a storage backend beneath a source root, finds `manifest.json` files, sorts bundle paths lexically, and rejects nested manifests.
-
-## Failure behavior
-
-Manifest parsing and validation fail before destination planning. Storage-backed validation fails when listed files are missing, are not regular files, have unexpected sizes, have unexpected SHA-256 digests, or when a source bundle includes unsafe or reserved paths.
 
 ## Boundaries
 
-Internal bundle discovery uses `internal/storage` and does not import concrete adapters. Producer-side local filesystem manifest building, complete bundle writing, and validation belong to `pkg/bundle`. CLI local path support is wired in `internal/app`.
+`internal/bundle` delegates manifest parsing, digest calculation, source path validation, and manifest validation to `pkg/bundle`. It does not publish files, inspect destination state, choose pipelines, or know concrete backend implementations.
 
-## Tests
+The external source bundle file-format contract is documented in `docs/integrations/source-bundle.md`.
 
-Before changing bundle behavior, inspect tests under `pkg/bundle` and `internal/bundle`.
+## Config Fields Used
 
-## Invariants
+The package does not read config directly. App workflows pass it storage backends that were opened from configured source fields.
 
-- `manifest.json` is the only source bundle contract.
+## Adapters Used
-- Source file paths must stay relative to the bundle root.
+
-- The top-level bundle digest is derived from manifest file records in order.
+The package depends only on `internal/storage.Backend`. Concrete local, SSH/SFTP, S3-compatible, and fake backends are hidden behind that interface.
-- Discovery order is lexical and deterministic.
+
-- Nested manifests are rejected.
+## State And Manifest Behavior
+
+Discovery walks recursively under the source root, finds entries whose basename is `manifest.json`, converts each manifest path to a bundle root, sorts roots, rejects nested bundle roots, and validates each bundle.
+
+Validation reads `manifest.json`, parses it, stats each manifest-listed file, requires regular files, verifies file sizes, reads file bytes, checks per-file SHA-256 digests, and recomputes the bundle digest.
+
+## Skip And Resume Behavior
+
+The package has no skip or resume state. Each call performs discovery or validation from the supplied backend state.
+
+## Failure Behavior
+
+Failures include invalid storage prefixes, missing manifests, parse errors, nested manifests, unsafe manifest paths, non-regular files, size mismatches, digest mismatches, backend stat/read errors, and no discovered bundles.
+
+## Tests To Inspect
+
+- `internal/bundle/*_test.go`
+- `pkg/bundle/*_test.go`
+- `internal/storage/fake/*_test.go`
+
+## Architectural Invariants
+
+- Source manifest semantics remain owned by `pkg/bundle`.
+- Discovery order is deterministic.
+- Nested manifests are rejected before returning bundles.
+- Source paths stay clean, relative, slash-separated, and confined to the backend root.
+- Concrete adapters never leak into bundle validation logic.

docs/internal/config.md

@@ -1,101 +1,57 @@
 # Configuration Internals
 
+Audience: developers and LLM coding agents changing `internal/config`.
+
 ## Purpose
 
-`internal/config` defines YAML-backed configuration structs, defaulting, and validation for distributor pipelines.
+`internal/config` owns YAML config structs, config file loading, defaulting, validation, size/duration parsing, SSH/S3 normalization helpers, and the credential environment resolver.
 
-## Inputs and outputs
+## Inputs And Outputs
 
-Input is a YAML file containing optional `server`, optional `secrets`, and required `pipelines`. Output is a `Config` value with defaults applied and validation completed. Load failures include the config path and whether the failure occurred during file loading, YAML parsing, or validation.
+Inputs are YAML files, YAML scalar values, process environment lookup functions, optional secrets directories, and `Config` values. Outputs are defaulted `Config` values, validation errors, parsed byte sizes and durations, normalized backend options, loaded secret environments, secret conflict metadata, and resolved credentials.
 
-## Loading flow
+## Boundaries
 
-`LoadFile` opens the requested path, decodes YAML with known-field checking enabled, applies defaults, and validates the result. The app uses `DefaultConfigPath` when the CLI does not supply a config path.
+The package does not open storage backends, authenticate HTTP requests, start servers, publish destinations, or execute transforms. Runtime execution support is wired by `internal/app`.
 
-Known-field checking rejects misspelled or unknown YAML keys before defaults and validation run.
+The canonical user-facing config reference is `docs/config.md`.
 
-`LoadFile` does not read secret files. App entrypoints load the configured
+## Config Fields Used
-secrets directory after config validation and before credential-consuming work.
 
-## Defaults
+The package defines all user-visible config fields: `server.http`, `secrets`, `pipelines`, source and destination backend fields, validation policy, publish policy, transform policy, path mapping, links, and transfer policy.
 
-Defaults are applied in `ApplyDefaults`:
+## Adapters Used
 
-- HTTP server `bind` defaults to `127.0.0.1:8080`;
+No external storage adapters are used directly. The package exposes normalized config and credential values consumed by app-level adapter construction.
-- HTTP server `staging_root` defaults to `/var/spool/distributor`;
-- HTTP server `max_upload_size` defaults to `20MB`;
-- HTTP server `queue_size` defaults to `16`;
-- HTTP server `max_concurrency` defaults to `1`;
-- HTTP server `retention` defaults to `24h`;
-- `http_upload` source `staging_path` defaults to `<server.http.staging_root>/<pipeline id>`;
-- `http_upload` source `max_upload_size` defaults to `server.http.max_upload_size`;
-- pipeline validation defaults `on_digest_mismatch` to `fail`;
-- SSH backend `port` defaults to `22`;
-- SSH backend `host_key_policy` defaults to `accept-new`;
-- destination publish policy defaults to source output only;
-- Markdown-to-HTML mode defaults to `sidecar` when a transform block is present and mode is omitted;
-- destination path mapping defaults to `preserve_relative`;
-- destination link primary policy defaults to `auto` when a `links` block is present;
-- `transfer.on_destination_same` defaults to `skip`;
-- `transfer.on_destination_older` defaults to `replace`;
-- `transfer.on_destination_newer` defaults to `skip`;
-- `transfer.on_conflict` defaults to `fail`.
 
-## Validation responsibilities
+## State And Manifest Behavior
 
-Validation requires positive HTTP server limits and retention, at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, valid destination link config, and valid transfer actions.
+The package does not parse source manifests or destination state. It validates config values that later affect manifest validation and destination state, such as publish/transform combinations, links, transfer policy, backend roots, S3 prefix shape, and HTTP upload source settings.
 
-HTTP upload sources require `token_env`, a staging path after defaults, and a positive maximum upload size. Literal token fields are not part of the YAML schema. The `http_upload` backend is accepted only for sources and rejected for destinations.
+## Skip And Resume Behavior
 
-Upload size values are parsed from strings with `B`, `KB`, `MB`, or `GB` suffixes using 1024 multipliers. Retention values are parsed with `time.ParseDuration`. Explicit zero values fail validation; omitted values receive defaults before validation.
+The package has no runtime skip or resume behavior. It provides transfer policy values that publish planning later applies to destination comparison outcomes.
 
-Transfer validation accepts `replace` for `on_destination_newer` and `on_conflict`, but publish planning honors those destructive actions only when the current run explicitly requests force.
+## Failure Behavior
 
-`ValidatePublishTransformPolicy` is shared with publish planning so destination policy combinations are checked consistently. Publishing HTML requires an enabled Markdown-to-HTML transform in `sidecar` or `index` mode. Enabled Markdown-to-HTML config is rejected when `publish.html` is false. `input` is accepted only for enabled `index` mode. A publish policy must select source output, HTML output, or both.
+`LoadFile` wraps file open, YAML parse, and validation failures with config path context. YAML decoding rejects unknown fields. Validation collects all detected field errors into a single error value.
 
-Destination path mapping accepts `preserve_relative` and `fixed`. The app layer applies the mapping when it selects destination bundle paths; config owns only YAML shape, defaulting, and validation.
+Secret loading fails for unreadable secrets directories, invalid secret filenames, unreadable secret files, and missing or empty required credential values. Secret conflicts are returned as warnings metadata, not secret values.
 
-Destination links are optional. When a `links` block is present, `base_url` is required, must use `http` or `https`, and must not include a query string or fragment. `primary` accepts `auto`, `html`, and `source`.
+## Tests To Inspect
-
-## Executable support boundary
-
-Config validation accepts `local`, `ssh`, `s3`, and source-only `http_upload` backend shapes. Runtime `run`, `validate`, and `inspect` workflows open `local`, `ssh`, and `s3` through `internal/app`. Runtime `serve` workflows execute `http_upload` sources through the app upload coordinator and HTTP server.
-
-SSH config uses structured fields: `host`, optional `user`, optional `port`, `path`, optional `ssh_key_file`, optional `known_hosts`, and optional `host_key_policy`. `host_key_policy` accepts YAML booleans and strings and normalizes `true`/`strict`, `accept-new`, and `false`/`off`.
-
-S3 config requires `endpoint` and `bucket`, normalizes optional `prefix`, defaults `region` to `us-east-1`, and defaults omitted `force_path_style` to `true` while preserving explicit `false`.
-
-HTTP upload config is source-only. Config owns its YAML shape, defaulting, size and duration parsing, and validation. The config package does not authenticate requests, stage uploads, or execute HTTP upload sources. The app layer resolves `token_env` through the config-owned environment resolver before starting the HTTP server.
-
-## Secrets and credential resolution
-
-`secrets.directory` points to a directory of credential files. `LoadSecretEnvironment` reads regular files and symlinks to regular files, rejects invalid filenames, trims exactly one trailing LF or CRLF, and returns an `Environment` resolver plus conflict metadata.
-
-The resolver checks the real process environment first and loaded secret values second. Differing process/secret conflicts are reported by variable name only. The resolver does not mutate `os.Environ`; default SDK credential chains continue to see only real process environment values.
-
-Credential-consuming backend wiring should resolve explicit credential environment variable references through `Environment.ResolveCredentials` or the same resolver pattern instead of calling `os.Getenv` directly.
-
-The user-facing configuration reference is `docs/config.md`; this file documents package behavior for maintainers.
-
-## Failure behavior
-
-Load errors wrap the underlying file, YAML, or validation error with context. Validation collects all detected field errors into one error value instead of stopping at the first invalid field.
-
-Unsupported backend names fail validation. Accepted backend names without runtime execution support fail later during app backend opening.
-
-## Tests
-
-Before changing config behavior, inspect:
 
 - `internal/config/load_test.go`
 - `internal/config/validate_test.go`
-- example-loading coverage in `internal/config`
+- `internal/config/secrets_test.go`
-- user-facing examples under `examples/`
+- `internal/config/backend_view_test.go`
+- `internal/app/runtime_test.go`
+- example configs under `examples/`
 
-## Invariants
+## Architectural Invariants
 
 - Defaults are applied before validation.
 - Unknown YAML fields are rejected.
-- `docs/config.md` remains the canonical user-facing config reference.
+- `http_upload` is source-only config.
-- Runtime backend execution support is not inferred from config validation support.
+- Credential-consuming runtime code must use the config-owned environment resolver.
-- New user-visible config behavior must be covered by tests and docs in the same change.
+- Secret values are never printed by config warnings.
+- New user-visible config behavior must update `docs/config.md` and tests.

docs/internal/ingest.md

@@ -1,43 +1,51 @@
 # Ingestion Internals
 
+Audience: developers and LLM coding agents changing `internal/ingest`.
+
 ## Purpose
 
-`internal/ingest` stages uploaded source bundle archives into local per-run directories. It does not authenticate requests, manage upload queues, publish destinations, or start an HTTP server.
+`internal/ingest` validates upload content types, extracts uploaded source bundle archives into local temporary storage, validates extracted bundles, and commits accepted bundles to per-run staging directories.
 
-## Archive staging
+## Inputs And Outputs
 
-`StageArchive` accepts one upload body, content type, pipeline staging path, run id, and explicit size and file-count limits. It writes the request body to temporary storage while enforcing the configured upload size limit, extracts the archive into temporary local storage, validates the extracted source bundle, and then commits the validated bundle to:
+Inputs are a context, upload body reader, content type, pipeline staging path, run id, maximum uploaded size, maximum extracted size, and maximum file count. Output is a `StagedBundle` containing the committed local bundle root and parsed manifest.
 
-```text
+## Boundaries
-<pipeline staging path>/<run id>
-```
 
-The returned `StagedBundle.Root` is a local filesystem path to the validated source bundle root.
+The package does not authenticate HTTP requests, manage upload queues, track upload status, publish destinations, load config, or start an HTTP server. Those responsibilities live in `internal/app`.
 
-## Accepted archive formats
+The HTTP API contract is documented in `docs/integrations/http-upload.md`.
 
-The package accepts only:
+## Config Fields Used
 
-- `application/x-tar`
+The package does not read config directly. The app layer passes effective values derived from `source.staging_path`, `source.max_upload_size`, and HTTP server defaults.
-- `application/gzip`
-- `application/x-gzip`
 
-Gzip uploads must contain a tar archive.
+## Adapters Used
 
-## Extraction rules
+The package uses the local filesystem directly for temporary archive storage, extraction, validation, and final staging path promotion. It does not use the storage backend abstraction.
 
-Archive entry paths must be clean relative slash-separated paths. Extraction rejects absolute paths, path traversal, backslash paths, duplicate files, symlinks, hardlinks, devices, sockets, and other special entries.
+## State And Manifest Behavior
 
-The archive must contain exactly one root-level `manifest.json`. Nested manifests are rejected.
+Accepted archives must contain exactly one root-level `manifest.json`. After extraction, the package validates the staged root through `pkg/bundle`, including manifest parsing, source path rules, file existence, regular-file checks, file sizes, file SHA-256 digests, and bundle digest.
 
-Regular files and directories are the only accepted tar entries. Regular file extraction enforces the explicit maximum extracted byte count and maximum file count supplied by the caller.
+## Skip And Resume Behavior
 
-## Bundle validation
+The package has no resume behavior. A successful call commits one complete staged bundle root. Failed calls remove temporary data created by that call.
 
-After extraction, the package loads and validates the staged bundle through `pkg/bundle`. Manifest parsing, source path validation, file existence checks, regular-file checks, file sizes, file SHA-256 digests, and bundle digest validation use the existing source bundle contract.
+## Failure Behavior
 
-Validation happens before the staged bundle is committed to its final per-run path.
+Failures include unsupported content type, unsafe run id, missing staging path, non-positive limits, oversize upload body, oversize extracted content, too many files, unsafe archive paths, duplicate files, nested manifests, unsupported tar entry types, gzip/tar read errors, bundle validation errors, and filesystem errors.
 
-## Failure behavior
+## Tests To Inspect
 
-Failed staging removes temporary archive and extraction data created by the package. A failed call does not publish anything and does not leave a committed per-run bundle directory.
+- `internal/ingest/archive_test.go`
+- `internal/app/upload_*_test.go`
+- `pkg/bundle/*_test.go`
+
+## Architectural Invariants
+
+- Invalid archives never commit a staged root.
+- Archive paths remain clean relative slash-separated paths.
+- Only directories and regular files are accepted from tar archives.
+- Source bundle validation happens before final staging path promotion.
+- Upload authentication and queueing remain outside this package.

docs/internal/link.md

@@ -1,27 +1,46 @@
 # Link URL Policy
 
+Audience: developers and LLM coding agents changing `internal/link`.
+
 ## Purpose
 
-`internal/link` defines shared validation for configured and persisted HTTP link URLs.
+`internal/link` owns shared validation for configured and persisted HTTP link URLs.
 
-## Inputs and outputs
+## Inputs And Outputs
 
 Input is a URL string. Output is either nil for an accepted URL or a concise validation error that callers wrap with field context.
 
-## Validation behavior
-
-Accepted URLs must parse successfully, use `http` or `https`, include a host, and omit query strings and fragments.
-
 ## Boundaries
 
-This package validates URL shape only. It does not construct destination output URLs, choose primary URLs, infer public URLs from backend configuration, or read configuration files.
+The package validates URL shape only. It does not construct output URLs, choose primary URLs, infer public URLs from backend configuration, parse config files, or write destination state.
 
-## Tests
+## Config Fields Used
 
-Before changing link URL policy, inspect tests under `internal/link` and callers in `internal/config`, `internal/state`, and `internal/publish`.
+The package does not read config directly. `internal/config` uses it to validate `links.base_url`; `internal/state` uses it to validate persisted `links.primary_url` and output `url` fields.
 
-## Invariants
+## Adapters Used
 
-- Configured `links.base_url`, persisted `links.primary_url`, persisted output `url`, and publish link planning use the same URL policy.
+None.
+
+## State And Manifest Behavior
+
+Destination state URL fields and configured link URLs share the same URL validation policy. Source manifests are not involved.
+
+## Skip And Resume Behavior
+
+None.
+
+## Failure Behavior
+
+Validation rejects parse failures, non-HTTP(S) schemes, missing hosts, query strings, and fragments.
+
+## Tests To Inspect
+
+- `internal/link/url_test.go`
+- callers in `internal/config`, `internal/state`, and `internal/publish`
+
+## Architectural Invariants
+
+- Configured and persisted link URLs use one validation policy.
+- URL construction remains outside `internal/link`.
 - Callers own field-specific error context.
-- URL path construction remains in `internal/publish`.

docs/internal/notify.md

@@ -1,35 +1,48 @@
-# Notify
+# Notification Internals
+
+Audience: developers and LLM coding agents changing `internal/notify` or app notification wiring.
 
 ## Purpose
 
-`internal/notify` defines the internal notification interface used by the application runner.
+`internal/notify` defines the notification interface used by app orchestration after successful destination publication or replacement.
 
-## Inputs and outputs
+## Inputs And Outputs
 
-Input is a notification event containing pipeline id, destination id, bundle id, bundle path, action, and output metadata. The interface returns an error so app orchestration can treat notification failures as destination failures.
+Input is a context and notification event containing pipeline id, destination id, bundle id, bundle path, action, and output metadata. Output is an error that app orchestration can record as a destination-scoped failure.
-
-## Current behavior
-
-The implemented notifier is a no-op. It is invoked only after a successful publish or replacement. Dry-run, skipped destinations, and failed destinations do not invoke it.
-
-## Failure behavior
-
-`notify.Noop` always succeeds unless the context is already canceled. If a configured notifier returns an error, `internal/app` records that destination as failed and continues with remaining destinations.
 
 ## Boundaries
 
-External notification adapters and user-facing notification configuration are outside current behavior.
+Only the no-op notifier exists in the repository. The package does not load config, send network requests, write destination state, publish files, or own run reporting.
 
-## Tests
+## Config Fields Used
 
-Before changing notification behavior, inspect:
+None.
+
+## Adapters Used
+
+None.
+
+## State And Manifest Behavior
+
+Notification events carry output metadata projected from publish plans. The package does not inspect source manifests or destination state.
+
+## Skip And Resume Behavior
+
+Dry-run, skipped destinations, failed destinations, and planning failures do not notify. The no-op notifier has no durable state.
+
+## Failure Behavior
+
+`notify.Noop` returns the context error when the context is canceled; otherwise it succeeds. If another notifier implementation returns an error, `internal/app` records the affected destination as failed and continues with remaining destinations where applicable.
+
+## Tests To Inspect
 
 - `internal/notify`
-- `internal/app/run_test.go`
+- `internal/app/run_notify.go`
+- notification coverage in `internal/app/run_test.go`
 
-## Invariants
+## Architectural Invariants
 
-- Notifications are emitted only after successful publish or replacement execution.
+- Notifications occur only after successful publish or replacement execution.
 - Dry-run never notifies.
 - Skipped and failed destinations never notify.
 - The default app notifier is `notify.Noop`.

docs/internal/publish.md

@@ -1,52 +1,63 @@
-# Publish
+# Publish Internals
+
+Audience: developers and LLM coding agents changing `internal/publish`.
 
 ## Purpose
 
-`internal/publish` plans and executes publication for one validated source bundle and one destination.
+`internal/publish` plans and executes publication for one validated source bundle and one destination bundle path. It owns destination comparison mapping, output selection, URL planning, managed cleanup selection, replacement safety, and destination state projection.
 
-## Inputs and outputs
+## Inputs And Outputs
 
-Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transform policy, optional link policy, transformer resolver, transfer policy, path mapping mode, destination bundle path, existing destination state, and whether explicit force was requested for the current run.
+Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, destination bundle path, path mapping mode, publish policy, transform policy, optional link policy, transformer resolver, transfer policy, distributor version, and force flag.
 
-Output is a plan with an action, reason, optional primary URL, and selected source or generated outputs. Execution writes selected source files, generated files, and `.distributor.json` for publish or replacement actions.
+Output from planning is a `Plan` with action, reason, destination identity, selected outputs, optional existing state, optional primary URL, and force metadata. Execution writes selected source outputs, generated outputs, and `.distributor.json` for executable publish or replacement actions.
-
-## Actions
-
-Supported actions are `publish_new`, `replace_older`, `force_replace`, `skip_same`, `skip_destination_newer`, `fail_conflict`, and `fail_unmanaged`.
-
-## Failure behavior
-
-Planning fails when request fields are incomplete, publish and transform policies are invalid, selected outputs collide, HTML output is requested without Markdown inputs, destination state is invalid, destination content is unmanaged without force, or transfer policy maps the comparison outcome to failure.
-
-Execution fails if a write, delete, state serialization, or context check fails. Outputs written during a failed publish attempt are cleaned up through managed deletion where possible.
 
 ## Boundaries
 
-The package publishes source files and Markdown-to-HTML outputs. Markdown sidecar mode writes same-directory `.html` outputs, and Markdown index mode writes `index.html`. Backend behavior is supplied through `internal/storage`; app runtime supplies local, SSH, and S3 backends.
+The package does not parse CLI flags, load config files, open concrete adapters, discover source bundles, select fixed-path bundle candidates, register transforms, or render command output. The app layer supplies validated request data and concrete dependencies.
 
-The package uses `internal/state` for destination comparison, `internal/storage` for IO, and the shared `internal/config` publish/transform policy helper for request validation. It resolves transforms through a narrow resolver supplied by the caller; concrete transform registration is owned by the app layer. It does not parse CLI flags, load config files, or choose which source bundles a destination receives.
+External destination state semantics are documented in `docs/integrations/destination-state.md`.
 
-The package owns projection from planned publish outputs to destination state output records and managed destination output paths. App JSON results and notification events keep their own schemas, but may use the publish output projection to avoid field-mapping drift.
+## Config Fields Used
 
-The app layer computes the destination bundle path before planning. `preserve_relative` destinations pass the source-root-relative bundle path. `fixed` destinations pass an empty destination bundle path, which means the destination backend root, and pass only the newest selected source bundle for that destination.
+The package consumes already-defaulted config values for destination `publish`, `transform`, `links`, `transfer`, and path mapping mode. It uses `config.ValidatePublishTransformPolicy` for publish/transform consistency.
 
-When link config is present, publish planning builds per-output URLs from `links.base_url`, the destination bundle path, and each output path. `index.html` outputs use directory-style URLs. The primary URL is selected from planned outputs according to the destination primary policy.
+## Adapters Used
 
-## Safety
+The package depends on `internal/storage.Backend` for source and destination IO, and on a narrow transformer resolver interface for generated outputs. It does not import concrete storage adapters or concrete transform implementations.
 
-Normal replacement deletes only outputs recorded in existing destination state plus `.distributor.json`. Forced replacement deletes the bounded destination bundle path before writing outputs and state. Failed writes trigger cleanup of outputs written during the failed attempt where practical.
+## State And Manifest Behavior
 
-## Tests
+Planning inspects destination state through `internal/state`, compares it with the source manifest, and maps comparison outcomes plus transfer policy into actions: `publish_new`, `replace_older`, `force_replace`, `skip_same`, `skip_destination_newer`, `fail_conflict`, or `fail_unmanaged`.
 
-Before changing publish behavior, inspect tests under `internal/publish` and run tests under `internal/app`.
+Execution writes destination state after selected outputs are written. Destination state includes copied source output metadata, generated output metadata, embedded source manifest, link metadata when configured, pipeline id, destination id, and publication timestamp.
 
-## Invariants
+## Skip And Resume Behavior
 
-- Publish planning is deterministic for the same source, destination state, policies, and transform outputs.
+`skip_same` and `skip_destination_newer` execute as no-ops. Normal replacement removes only managed output paths from existing state plus `.distributor.json`; this allows retries without broad deletion. Failed writes trigger cleanup of outputs written during that failed attempt where practical.
-- Destination bundle paths are caller-supplied and are interpreted relative to the destination backend root.
+
-- URL generation uses URL path semantics and does not infer public URLs from backend configuration.
+Forced replacement is explicit per request and deletes the bounded destination bundle path before writing new outputs and state.
+
+## Failure Behavior
+
+Planning fails for incomplete requests, invalid publish/transform policy, output path collisions, invalid destination state, unmanaged destination content without force, conflict outcomes not allowed by transfer policy, unresolved transforms, invalid Markdown output selection, and invalid link URL planning.
+
+Execution fails on delete, read, transform output, write, state validation, state serialization, or context errors. Execution refuses actions that are not executable publish or replacement actions.
+
+## Tests To Inspect
+
+- `internal/publish/*_test.go`
+- `internal/app/run_test.go`
+- `internal/state/*_test.go`
+- `internal/transform/markdown/*_test.go`
+
+## Architectural Invariants
+
+- Planning is deterministic for the same request and destination state.
+- Destination bundle paths are caller-supplied and backend-root-relative.
+- URL generation uses URL path semantics and never infers public URLs from backend config.
 - Normal replacement deletes only managed paths recorded in existing state plus `.distributor.json`.
-- Forced replacement is explicit per run and deletes only within the destination bundle path.
+- Forced replacement deletes only within the supplied destination bundle path.
-- Publish execution writes destination state after selected outputs are written.
+- Destination state is written after selected outputs are written.
-- Transform implementations are resolved through an interface supplied by the caller.
+- Transform resolution stays behind a caller-supplied interface.
-- Unmanaged destination content is overwritten only by explicit forced replacement.
+- Unmanaged content is claimed only by explicit force.

docs/internal/state.md

@@ -1,55 +1,54 @@
-# Destination State
+# Destination State Internals
+
+Audience: developers and LLM coding agents changing `internal/state`.
 
 ## Purpose
 
-`internal/state` parses, validates, writes, and compares `.distributor.json` destination state.
+`internal/state` parses, validates, serializes, and compares `.distributor.json` destination state.
 
-## Inputs and outputs
+## Inputs And Outputs
 
-Input is JSON destination state plus the current source manifest, pipeline id, destination id, and whether the destination path has unmanaged content. Output is a deterministic comparison outcome and reason.
+Inputs are destination state JSON, constructed state values, current source manifest, pipeline id, destination id, and whether the destination path has content without state. Outputs are validated state values, JSON bytes, comparison outcomes, and human-readable reasons.
-
-## State behavior
-
-`.distributor.json` requires:
-
-- `schema_version: 1`
-- `pipeline_id`
-- `destination_id`
-- `published_at`
-- `source.manifest`
-- `outputs`
-
-`distributor_version` is optional diagnostic metadata. `links` is optional URL metadata. `published_at` parses as RFC3339 and distributor-written state serializes it as RFC3339 UTC.
-
-The embedded `source.manifest` is validated with the same source manifest rules used by `internal/bundle`.
-
-## Outputs
-
-Each output records `path`, `kind`, `source_path`, `sha256`, and `size`. Supported output kinds are `source` and `generated`. Generated outputs require `transform`. Outputs may record `url` when the destination has link generation configured.
-
-The optional top-level `links.primary_url` records the selected primary URL for the published destination bundle. It is omitted when link generation is not configured or when the destination primary policy has no matching output.
-
-## Comparison
-
-Comparison outcomes cover absent destination state, unmanaged destination content, invalid state, pipeline or destination mismatch, same source manifest, older destination source, newer destination source, same-created digest conflict, and different source id conflict.
-
-## Failure behavior
-
-Invalid JSON, invalid state schema, invalid embedded source manifests, unsafe output paths, invalid stored URLs, unsupported output kinds, missing generated-output transform names, and mismatched pipeline or destination ids produce comparison outcomes that publish planning can turn into fail actions. Supported identity and source-manifest conflicts can become forced replacement only when publish planning receives explicit force and compatible transfer policy.
 
 ## Boundaries
 
-This package does not publish files, delete files, inspect storage backends, or choose transfer policy actions. Publish planning consumes these comparison outcomes later.
+The package does not inspect storage backends, mutate files, choose transfer policy, build publish outputs, generate URLs, or parse config. Publish planning consumes state comparison outcomes.
 
-## Tests
+The external destination state contract is documented in `docs/integrations/destination-state.md`.
 
-Before changing destination state behavior, inspect tests under `internal/state`.
+## Config Fields Used
 
-## Invariants
+None directly. Destination ids, pipeline ids, and link URLs originate from config but are supplied as values by callers.
+
+## Adapters Used
+
+None.
+
+## State And Manifest Behavior
+
+`.distributor.json` schema version is `1`. Required fields are `pipeline_id`, `destination_id`, `published_at`, `source.manifest`, and `outputs`. `distributor_version` and `links` are optional.
+
+Embedded source manifests are parsed and validated through `internal/bundle`, which delegates source manifest semantics to `pkg/bundle`. Output records require clean paths, `source` or `generated` kind, valid source paths, lowercase SHA-256 digests, non-negative sizes, and transform ids for generated outputs. Stored URLs must pass `internal/link` validation.
+
+## Skip And Resume Behavior
+
+Comparison is pure. It returns outcomes for absent state, unmanaged content, invalid state, pipeline/destination mismatch, same source manifest, older destination, newer destination, same-created digest conflict, and different source id conflict. It does not decide whether to skip, replace, force, or fail; publish planning maps outcomes to actions.
+
+## Failure Behavior
+
+Parsing rejects invalid JSON, trailing data, missing required fields, invalid timestamps, invalid embedded manifests, duplicate outputs, invalid output paths, unsupported output kinds, missing generated transforms, invalid URLs, invalid digests, and negative sizes.
+
+## Tests To Inspect
+
+- `internal/state/distributor_test.go`
+- `internal/state/compare_test.go`
+- `internal/publish/*_test.go`
+
+## Architectural Invariants
 
 - `.distributor.json` is the destination sentinel and state record.
-- Embedded source manifests use the same validation rules as source bundles.
+- Comparison does not mutate storage.
+- Embedded source manifests use the source bundle contract.
 - Generated outputs always record a transform id.
 - Stored URLs are optional and must be absolute HTTP or HTTPS URLs when present.
-- Comparison returns outcomes and reasons; it does not mutate storage.
 - `distributor_version` is diagnostic metadata, not a comparison key.

docs/internal/storage.md

@@ -1,75 +1,56 @@
-# Storage
+# Storage Internals
+
+Audience: developers and LLM coding agents changing `internal/storage`, storage adapters, or storage-backed callers.
 
 ## Purpose
 
-`internal/storage` defines backend-rooted logical file access for core packages. Callers use slash-separated paths relative to a configured backend root.
+`internal/storage` defines backend-rooted logical file access, path validation, typed storage errors, traversal helpers, backend registration, managed deletion targets, and test fake storage behavior.
 
-## Inputs and outputs
+## Inputs And Outputs
 
-The storage interface supports byte reads, stream reads, byte writes, stream writes, exact metadata lookup, traversal, destination emptiness checks, guarded managed deletion, and bounded prefix deletion for explicit forced replacement.
+Inputs are contexts, logical paths or prefixes, byte slices or readers, write options, walk options, delete options, and backend open configs. Outputs are file bytes, readers, `Entry` metadata, walk callbacks, boolean content checks, registered backends, and typed errors.
-
-Entries report a logical path, type, and size when available. Entry types are `file`, `directory`, `symlink`, and `other`.
 
 ## Boundaries
 
-Core packages should depend on `internal/storage`, not adapter packages. Adapter-specific path handling stays behind backend implementations.
+Core packages depend on `internal/storage`, not concrete adapters. Adapter protocol behavior belongs in `internal/adapters/local`, `internal/adapters/ssh`, and `internal/adapters/s3`; external SSH/SFTP and S3 notes live under `docs/integrations/`.
 
-The local adapter lives in `internal/adapters/local`. The SSH/SFTP adapter lives in `internal/adapters/ssh`. The S3-compatible adapter lives in `internal/adapters/s3`. Runtime backend construction is wired through the app-level backend factory and storage registry. The fake backend lives in `internal/storage/fake` for tests and is not registered for runtime use.
+Runtime backend construction and registration are owned by `internal/app`. The fake backend is for tests only.
 
-## Paths
+## Config Fields Used
 
-Logical file paths must be non-empty, relative, clean, slash-separated, and must not contain `.` or `..` segments or backslashes. Prefix paths follow the same rules, except an empty prefix means the backend root.
+The storage package does not read config directly. App adapter wiring converts config fields into backend open config values.
 
-## Failure behavior
+## Adapters Used
 
-Storage errors use typed categories such as not found, already exists, invalid path, conflict, permission, temporary, unsupported, and unknown. Callers should use helper predicates rather than matching error strings.
+Local, SSH/SFTP, and S3-compatible adapters implement `storage.Backend`. `internal/storage/fake` implements the same interface for tests.
 
-Backends may wrap implementation-specific errors, but callers should receive storage errors where practical. Traversal can stop cleanly with `ErrStopWalk`.
+## State And Manifest Behavior
 
-## Traversal helpers
+Storage owns `.distributor.json` path helpers through `StateFileName`, `StatePath`, and `ManagedBundleTargets`. It does not parse source manifests or destination state.
 
-Backends own their traversal mechanics. The local adapter owns filesystem walking, the SSH adapter owns SFTP directory walking, and the S3 adapter owns object listing and pagination.
+Logical paths are slash-separated and relative to a backend root. Prefix validation allows an empty prefix to mean the backend root; file path validation requires a non-empty path.
 
-`internal/storage` owns the shared callback emission rules used by backends:
+## Skip And Resume Behavior
 
-- context cancellation is checked before callback emission;
+Storage has no publication skip policy. It supplies `HasAny` for unmanaged-content checks, `DeleteManagedBundle` target construction for normal replacement cleanup, and `DeletePrefix` semantics for explicit forced replacement.
-- `WalkOptions.Limit` bounds the number of emitted entries;
-- `ErrStopWalk` stops traversal without becoming a caller-visible error;
-- callback errors are wrapped as storage walk errors.
 
-`storage.HasAny(ctx, backend, prefix)` provides the shared destination-content check. It calls `Walk` with non-recursive, limit-one traversal and stops after the first emitted entry.
+## Failure Behavior
 
-## Deletion
+Storage errors use typed categories: not found, already exists, not empty, invalid path, conflict, permission, temporary, unsupported, and unknown. Callers should use helper predicates instead of matching strings. Traversal can stop cleanly with `ErrStopWalk`.
 
-`DeleteManagedBundle` may delete listed managed outputs plus `.distributor.json`.
+## Tests To Inspect
 
-`DeletePrefix` removes content at and below a logical prefix for explicit forced replacement. It must not delete above the requested prefix or above the configured backend root.
+- `internal/storage/*_test.go`
+- `internal/storage/fake/*_test.go`
+- `internal/adapters/local/*_test.go`
+- `internal/adapters/ssh/*_test.go`
+- `internal/adapters/s3/*_test.go`
 
-## Local, SSH, S3, and fake backends
+## Architectural Invariants
 
-The local adapter maps logical paths to a configured filesystem root and keeps adapter-specific path handling behind the storage interface.
+- Logical paths are clean relative slash-separated paths confined to the backend root.
-
+- Core packages never import concrete adapters.
-The SSH adapter maps logical paths to a configured remote SFTP root. It uses native SSH and SFTP libraries, supports SSH agent and key-file authentication, applies host-key policies, rejects unsafe logical paths, reports symlink entries from `Lstat`, and limits deletion to managed targets or explicit bounded prefixes.
+- `storage.List` returns deterministic sorted entries.
-
+- Managed deletion targets are recorded outputs plus `.distributor.json`.
-The S3 adapter maps logical paths to object keys below a configured bucket and optional prefix. It uses the AWS SDK for Go v2, treats prefixes as object trees, requires exact objects for `Stat`, paginates traversal, applies conservative overwrite checks with `HeadObject`, infers basic content types, and limits deletion to managed target objects or explicit bounded object-key prefixes.
+- Prefix deletion is bounded to the requested logical prefix.
-
+- Runtime registration remains app-owned.
-The fake backend is an in-memory implementation for package tests. It is not registered for runtime use.
-
-## Tests
-
-Before changing storage behavior, inspect tests under:
-
-- `internal/storage`
-- `internal/storage/fake`
-- `internal/adapters/local`
-- `internal/adapters/ssh`
-- `internal/adapters/s3`
-
-## Invariants
-
-- Core packages depend on `internal/storage`, not concrete adapters.
-- Logical paths are slash-separated and confined to the backend root.
-- `storage.List` uses backend traversal and returns deterministic entries.
-- Managed deletion is limited to recorded outputs plus `.distributor.json`.
-- Prefix deletion is limited to the requested logical prefix.
-- Runtime backend registration is owned by `internal/app`.

docs/internal/transform.md

@@ -1,47 +1,52 @@
-# Transform
+# Transform Internals
+
+Audience: developers and LLM coding agents changing `internal/transform` or transform implementations.
 
 ## Purpose
 
-`internal/transform` defines generated publication artifacts. `internal/transform/markdown` implements Markdown-to-HTML generation.
+`internal/transform` defines generated publication artifacts, transform request/response types, transform registry behavior, and transform identifiers. `internal/transform/markdown` implements Markdown-to-HTML generation.
 
-## Inputs and outputs
+## Inputs And Outputs
 
-Inputs are a validated source bundle, source backend, and transform options supplied by publish planning. Outputs include destination path, source path, transform id, generated bytes, SHA-256, and size.
+Inputs are a validated source bundle, source storage backend, and transform options supplied by publish planning. Outputs are generated artifact records containing destination path, source path, transform id, generated bytes, SHA-256 digest, and byte size.
-
-## Registry
-
-`internal/transform` defines the transform interface and registry. The app layer registers the Markdown implementation; publish planning receives only a resolver.
-
-## Markdown behavior
-
-Markdown sidecar mode renders files ending in `.md` to `.html` files in the same logical directory. Markdown index mode renders one selected manifest-listed Markdown file to `index.html`. Non-Markdown files do not generate sidecar outputs. Raw HTML embedded in Markdown is not passed through by the renderer.
-
-Generated HTML is deterministic for the same source content and transform configuration.
-
-See `docs/integrations/markdown.md` for the Goldmark integration contract.
-
-## Failure behavior
-
-Transform resolution fails when a requested transform id is not registered. Markdown rendering fails when the source file cannot be read or rendered. Index input selection fails when the configured input is unsafe, not listed, not Markdown, or when no configured input can be inferred from exactly one manifest-listed Markdown file. Publish planning fails when HTML output is requested and the selected transform produces no outputs for a bundle.
 
 ## Boundaries
 
-Transforms do not publish files, mutate source bundles, or write destination state. Publish planning selects and writes transform outputs.
+Transforms do not mutate source bundles, publish files, write destination state, choose destination actions, parse config, or inspect destinations. Publish planning decides whether generated outputs are selected and writes destination state later.
 
-The app layer owns default transform registration. The transform package does not import concrete transform implementations.
+The Goldmark renderer contract is documented in `docs/integrations/markdown.md`.
 
-## Tests
+## Config Fields Used
 
-Before changing transform behavior, inspect tests under:
+Transform packages do not read config directly. Publish planning passes effective `transform.markdown_to_html.mode` and `transform.markdown_to_html.input` values.
 
-- `internal/transform`
+## Adapters Used
-- `internal/transform/markdown`
 
-## Invariants
+Transforms read source files through `internal/storage.Backend`. The Markdown implementation uses `github.com/yuin/goldmark` for rendering.
+
+## State And Manifest Behavior
+
+Transform outputs carry metadata later projected into destination state. Markdown sidecar mode renders manifest-listed `.md` files to same-directory `.html` outputs. Markdown index mode renders one selected Markdown source to `index.html`.
+
+## Skip And Resume Behavior
+
+Transforms have no skip/resume state. They are deterministic for the same source bytes and transform options.
+
+## Failure Behavior
+
+Registry registration fails for empty names, nil transformers, and duplicate names. Transform resolution fails when publish planning requests an unregistered transform. Markdown rendering fails on source read errors, renderer errors, unsafe configured input, missing manifest input, non-Markdown input, ambiguous index input, or absent Markdown inputs.
+
+## Tests To Inspect
+
+- `internal/transform/*_test.go`
+- `internal/transform/markdown/*_test.go`
+- `internal/publish/*_test.go`
+
+## Architectural Invariants
 
 - Source bundle files are never mutated by transforms.
 - Generated outputs record destination path, source path, transform id, SHA-256, and size.
-- Markdown sidecar naming changes only the `.md` extension to `.html`.
+- Markdown sidecar naming changes only the `.md` suffix to `.html`.
 - Markdown index mode always writes `index.html`.
-- Non-Markdown source files do not generate Markdown outputs.
+- Non-Markdown source files do not generate sidecar outputs.
 - Transform registration stays outside publish planning.

docs/operations.md

@@ -1,107 +1,120 @@
 # Distributor Operations
 
+Audience: administrators and operators who run `distributor`, publish bundles, operate the HTTP upload service, or recover from failed runs.
+
+This document covers operating workflows, storage layout, safety behavior, and recovery. Command syntax lives in [CLI](cli.md), configuration fields live in [Configuration](config.md), symptom-specific fixes live in [Troubleshooting](troubleshooting.md), and external contracts live under [Integrations](integrations/source-bundle.md).
+
 ## Normal Workflow
 
-Validate a source bundle:
+Validate a producer bundle before publishing:
 
 ```sh
 go run ./cmd/distributor validate examples/source-bundle
 ```
 
-Preview a local publication:
+Preview a configured run before writing destination content:
 
 ```sh
 go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
 ```
 
-Run the local publication:
+Publish after reviewing the preview:
 
 ```sh
 go run ./cmd/distributor run --config examples/local-publish.yml
 ```
 
-Run the local HTML publication:
+Use JSON output for automation:
-
-```sh
-go run ./cmd/distributor run --config examples/local-html.yml
-```
-
-Run the local `index.html` publication:
-
-```sh
-go run ./cmd/distributor run --config examples/local-index.yml
-```
-
-Preview local fan-out publication:
-
-```sh
-go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
-```
-
-Preview local archive-plus-latest publication:
-
-```sh
-go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run
-```
-
-Preview a run for automation:
 
 ```sh
 go run ./cmd/distributor run --config examples/fan-out.yml --dry-run --format json
 ```
 
-Preview an environment-gated SSH destination config after editing it for an SSH/SFTP endpoint you control:
+Use configured source diagnostics when the source is defined in YAML and may be local, SSH/SFTP, or S3-compatible storage:
-
-```sh
-go run ./cmd/distributor run --config examples/ssh-destination.yml --dry-run
-```
-
-Preview an environment-gated S3 destination config after editing it for an S3-compatible endpoint and bucket you control:
-
-```sh
-go run ./cmd/distributor run --config examples/s3-destination.yml --dry-run
-```
-
-Validate one configured source without opening destinations:
 
 ```sh
 go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
+go run ./cmd/distributor inspect --config examples/local-publish.yml --pipeline example-source-bundle --format json
 ```
 
-## HTTP Upload Workflow
+Remote examples under `examples/ssh-destination.yml` and `examples/s3-destination.yml` are load-tested templates. Edit their endpoint, path, key, bucket, prefix, and credential values for storage you control before running them.
 
-`distributor serve` runs the HTTP upload API for pipelines whose source backend
+## Filesystem And Storage Layout
-is `http_upload`. Each upload token maps to one configured pipeline, and each
-accepted archive is staged, validated, and published through the same
-destination fan-out path used by local source runs.
 
-Minimal local HTTP upload configuration:
+A source bundle is a directory containing `manifest.json` and every file listed in that manifest. See [Source Bundle Contract](integrations/source-bundle.md). Source discovery walks beneath the configured source backend root and finds bundle directories.
 
-```yaml
+Each destination has its own backend root:
-server:
+
-  http:
+- Local destinations use the configured local `path`.
-    bind: 127.0.0.1:8080
+- SSH/SFTP destinations use the configured remote `path`.
-    staging_root: /var/spool/distributor
+- S3-compatible destinations use the configured `bucket` plus optional `prefix`.
-    max_upload_size: 20MB
+
-    queue_size: 16
+Destination path mapping controls where each source bundle is published beneath the destination root:
-    max_concurrency: 1
+
-    retention: 24h
+- `preserve_relative` publishes each source bundle at the same source-root-relative path.
-secrets:
+- `fixed` publishes one selected source bundle at the destination root.
-  directory: /run/secrets/distributor
+
-pipelines:
+Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple bundles have the same timestamp, the source-root-relative bundle path in ascending order wins.
-  - id: reports
+
-    source:
+Published destination bundle paths contain `.distributor.json`. See [Destination State Contract](integrations/destination-state.md). This file is both the managed sentinel and the destination state record. It records the pipeline id, destination id, publication time, source manifest, copied outputs, generated outputs, and optional public URL metadata.
-      backend: http_upload
+
-      token_env: DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN
+`manifest.json` from the source bundle is not copied as destination state.
-    destinations:
+
-      - id: archive
+## Destination State And Retry Behavior
-        backend: local
+
-        path: /srv/reports/archive
+`distributor` compares the source manifest to destination `.distributor.json` before writing:
+
+- No destination state and no destination content: publish new outputs.
+- Matching destination state: skip as already published.
+- Older destination state for the same source id: replace if transfer policy allows it.
+- Newer destination state: skip by default.
+- Invalid destination state, identity mismatch, different source id, or same-created digest mismatch: fail by default.
+- Content without `.distributor.json`: fail as unmanaged content by default.
+
+Normal replacement deletes only managed output paths recorded in `.distributor.json` plus the state file, then verifies the destination bundle path is empty before writing new outputs and state.
+
+If a write fails after some outputs were written, `distributor` attempts to delete outputs from that failed attempt so a retry does not treat partial outputs as unmanaged content. Operators should still inspect the destination after a failed write before retrying.
+
+Fan-out destinations are independent. If one destination fails after planning or execution begins, later destinations are still attempted. The command exits non-zero if any destination failed.
+
+## Dry Runs And Output Review
+
+`run --dry-run` loads config, resolves credentials, discovers source bundles, opens destinations, inspects destination state, builds publish plans, and prints actions. It does not write outputs, `.distributor.json`, or SSH `known_hosts` entries.
+
+Review these action labels before publishing:
+
+- `publish_new`: destination is empty and unmanaged.
+- `replace_older`: destination state is older than the source.
+- `skip_same`: destination state already matches the source.
+- `skip_destination_newer`: destination state is newer than the source and is skipped.
+- `force_replace`: destructive replacement selected because `--force` is present and policy permits it.
+- `error`: planning or execution failed for that destination.
+
+Fixed destinations add fixed-path warnings during dry runs, including the selected source bundle and replacement warnings when the destination root would be replaced.
+
+JSON output includes warnings, pipeline summaries, destination action records, output records, URLs when configured, final counters, and partial failure details. Fatal setup failures such as unreadable config or invalid secrets do not produce a JSON result document.
+
+## Forced Replacement Workflow
+
+Use `--force` only after a dry run shows the intended bounded `force_replace` action:
+
+```sh
+go run ./cmd/distributor run --config <config-path> --dry-run --force
+go run ./cmd/distributor run --config <config-path> --force
 ```
 
-Create `/run/secrets/distributor/DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN` or set the
+Forced replacement can claim unmanaged non-empty destination paths. State conflicts require both `--force` and transfer policy that permits replacement:
-real process environment variable before starting the server. Distributor does
+
-not read literal upload tokens from YAML.
+- newer destination state requires `transfer.on_destination_newer: replace`;
+- conflict outcomes require `transfer.on_conflict: replace`.
+
+Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete parent paths, sibling paths, or storage outside the destination bundle path. For fixed destinations, the destination bundle path is the backend root, so a forced replacement can clear that configured root.
+
+`--force` applies only to the current invocation. There is no config field that enables forced replacement by default.
+
+## HTTP Upload Operation
+
+The [HTTP Upload API Contract](integrations/http-upload.md) defines request and response details. `distributor serve` runs the HTTP upload API for pipelines whose source backend is `http_upload`. Each bearer token maps to exactly one configured upload pipeline. Token values come from the process environment or `secrets.directory`, not from YAML literal values.
 
 Start the maintained local example:
 
@@ -110,7 +123,13 @@ DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN=<token> \
   go run ./cmd/distributor serve --config examples/http-upload-local.yml
 ```
 
-Submit a tar or tar.gz source bundle:
+Readiness:
+
+```sh
+curl http://127.0.0.1:8080/healthz
+```
+
+Upload one tar or tar.gz source bundle archive:
 
 ```sh
 curl -X POST http://127.0.0.1:8080/upload \
@@ -119,296 +138,61 @@ curl -X POST http://127.0.0.1:8080/upload \
   --data-binary @bundle.tar.gz
 ```
 
-Successful admission returns a run id:
+Accepted uploads return after the archive is staged and validated:
 
 ```json
-{"run_id":"reports.20260603T120000Z.abcdef12","status":"accepted"}
+{"run_id":"example-http-upload.20260604T120000Z.abcdef12","status":"accepted"}
 ```
 
-Poll status until it reaches `succeeded` or `failed`:
+Poll status while the in-memory record is retained:
 
 ```sh
 curl http://127.0.0.1:8080/runs/<run-id>
 ```
 
-The status record includes the completed run report on successful publication
+Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire after `server.http.retention`. Expiration removes committed staged bundle directories for completed uploads. Restarting the process clears upload status, queue state, and in-memory records.
-or error details on failure. Status is memory-only and expires after
-`server.http.retention`; completed staged bundle directories are removed on
-expiry. Restarting the process clears upload status and queue state.
 
-Use `GET /healthz` for readiness after config and tokens load:
+Upload admission is bounded by `server.http.queue_size`. Publication concurrency is bounded by `server.http.max_concurrency`, and the coordinator does not run two uploads for the same pipeline at the same time.
 
-```sh
+The upload server accepts `application/x-tar`, `application/gzip`, and `application/x-gzip`. Archives are extracted into a temporary staging directory, must contain exactly one root-level `manifest.json`, and must validate as one complete source bundle before a run id is issued. Per-source `max_upload_size` bounds both uploaded archive size and extracted bundle size. The implementation also caps extracted file count.
-curl http://127.0.0.1:8080/healthz
-```
 
-The default bind address is private loopback. Put TLS, public routing,
+The default bind address is private loopback. Put TLS, public routing, rate limiting, and external access policy in a reverse proxy or deployment layer.
-rate-limiting, and external access policy in a reverse proxy or deployment
-layer.
 
-## Filesystem Layout
+## Remote Backend Notes
 
-Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`.
+### SSH/SFTP
 
-Destination bundle paths are configured per destination with `path_mapping.mode`.
+SSH execution uses native SFTP. See [SSH/SFTP Integration](integrations/ssh-sftp.md). It does not shell out to `ssh`, `scp`, or `rsync`.
-
-The default mode, `preserve_relative`, preserves the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under `daily/` publishes under `daily/` at that destination.
-
-The `fixed` mode publishes one selected source bundle at the destination backend root. A fixed destination with local `path: /srv/www/reports/latest` writes outputs and `.distributor.json` directly under `/srv/www/reports/latest`. Fixed destinations select the newest discovered source bundle by manifest `created` timestamp, with the source-root-relative bundle path as the deterministic tie-breaker.
-
-The maintained local examples write under `workspace/`, which is ignored by Git.
-
-SSH backends use the configured remote `path` as the backend root. Source bundle discovery and destination bundle paths are relative to that root, using the same logical path rules as local storage.
-
-S3 backends use the configured bucket plus optional `prefix` as the backend root. Source bundle discovery and destination bundle paths are relative to that object-key prefix. Prefixes are object-key prefixes, not real directories.
-
-## Destination State
-
-Each published destination bundle contains `.distributor.json`. This file is the managed sentinel and destination state record. It stores:
-
-- pipeline and destination identity;
-- publication timestamp;
-- source manifest used for publication;
-- copied source output metadata;
-- generated output metadata;
-- optional public URL metadata when destination links are configured.
-
-`manifest.json` from the source bundle is not copied as destination state.
-
-Do not edit `.distributor.json` by hand during normal operation. If it is missing or invalid while destination files remain, `distributor` treats the destination as unmanaged or conflicted.
-
-## Go Producer Bundles
-
-Go producer applications can import `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to create complete local source bundles with the same path, digest, timestamp, and validation rules used by `distributor`. The package also exposes digest helpers, including `ValidateDigest`, for producer code that needs to validate lowercase `sha256:<64 hex>` strings before writing manifests.
-
-Minimal producer-side bundle creation:
-
-```go
-manifest, err := bundle.WriteBundle(bundle.WriteBundleOptions{
-    Root:  outputDir,
-    ID:    "reports.example.2026-05-30",
-    Files: []bundle.BundleFile{
-        {SourcePath: reportPath, Path: "report.md"},
-        {SourcePath: summaryPath, Path: "summary.txt"},
-    },
-})
-if err != nil {
-    return err
-}
-```
-
-`WriteBundle` copies local producer files into a sibling temporary directory, writes `manifest.json`, validates the result, and promotes the completed bundle into place. It fails if `Root` already exists unless `Overwrite` is true. With overwrite enabled, it builds and validates the replacement before moving the existing root aside.
-
-Use `BuildManifest` and `WriteManifest` when a producer already wrote all bundle files into the final root. `BuildManifest` can preserve an explicit file order, or `Scan: true` can recursively include regular files under `Root` in deterministic slash-path order. Scan mode includes dotfiles, excludes files named `manifest.json` or `.distributor.json`, and rejects symlinks.
-
-Shell producers can create the same manifest through the CLI after writing bundle files:
-
-```sh
-go run ./cmd/distributor manifest create <bundle-path> --id reports.example.2026-05-30
-go run ./cmd/distributor validate <bundle-path>
-```
-
-Use repeated `--file` flags to preserve a specific file order. If no `--file` flags are provided, the command scans the bundle directory recursively using the same filtering rules as `pkg/bundle.BuildManifest`.
-
-## Static HTML Publication
-
-Markdown-to-HTML publication can write sidecar files or a fixed `index.html`.
-
-Use sidecar mode when each Markdown source should keep a matching HTML filename:
-
-```yaml
-publish:
-  source: false
-  html: true
-transform:
-  markdown_to_html:
-    enabled: true
-    mode: sidecar
-```
-
-Use index mode for static-site destinations that should serve a bundle through `index.html`:
-
-```yaml
-publish:
-  source: false
-  html: true
-transform:
-  markdown_to_html:
-    enabled: true
-    mode: index
-    input: report.md
-```
-
-If `input` is omitted in index mode, the source manifest must list exactly one Markdown file. Generated HTML is recorded in `.distributor.json` with `kind: generated`, `source_path`, `transform: markdown_to_html`, digest, and size metadata.
-
-## Archive And Latest Fan-Out
-
-A pipeline can publish the same source to an archive destination and a stable latest destination:
-
-```yaml
-pipelines:
-  - id: reports
-    source:
-      backend: local
-      path: /var/spool/distributor/reports
-    destinations:
-      - id: archive
-        backend: local
-        path: /srv/reports/archive
-        path_mapping:
-          mode: preserve_relative
-        publish:
-          source: true
-          html: false
-      - id: latest-html
-        backend: local
-        path: /srv/www/reports/latest
-        path_mapping:
-          mode: fixed
-        links:
-          base_url: https://reports.example.com/latest
-          primary: auto
-        publish:
-          source: false
-          html: true
-        transform:
-          markdown_to_html:
-            enabled: true
-            mode: index
-            input: report.md
-```
-
-The archive destination plans every discovered source bundle at its source-relative path. The fixed latest destination plans only the newest discovered bundle and writes `index.html` plus `.distributor.json` at its backend root.
-
-## Static Site URLs
-
-Use destination `links` when a destination backend root corresponds to a public HTTP or HTTPS URL:
-
-```yaml
-links:
-  base_url: https://reports.example.com/archive
-  primary: auto
-```
-
-Distributor records URLs in `.distributor.json`; it does not publish notifications or infer URLs from local, SSH, or S3 backend fields.
-
-For archive-style destinations, URLs include the destination bundle path. A source bundle under `daily/brentwood/2026-06-01` with `base_url: https://reports.example.com/archive` can produce:
-
-```text
-https://reports.example.com/archive/daily/brentwood/2026-06-01/report.html
-```
-
-For fixed destinations, URLs are rooted at `links.base_url`. A fixed HTML index destination with `base_url: https://reports.example.com/latest` records:
-
-```text
-https://reports.example.com/latest/
-```
-
-`index.html` outputs use directory-style URLs. Other outputs include their filename. The primary URL is selected from the published outputs using the destination `links.primary` policy.
-
-## Source Validation and Inspection
-
-`validate` and `inspect` can operate on a local path or on one configured pipeline source. Configured source mode requires both `--config` and `--pipeline`; it loads the normal config, resolves `secrets.directory`, opens only the selected source backend, and does not open any destinations.
-
-Configured source validation is useful when producers write directly to SSH or S3 storage:
-
-```sh
-go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
-go run ./cmd/distributor inspect --config <config-path> --pipeline <pipeline-id>
-```
-
-Use `--bundle <path>` to validate or inspect one source-root-relative bundle directory:
-
-```sh
-go run ./cmd/distributor validate \
-  --config <config-path> \
-  --pipeline <pipeline-id> \
-  --bundle daily/2026-06-01
-```
-
-For configured SSH sources, host key and authentication behavior matches `run`. For configured S3 sources, endpoint, bucket, prefix, region, path-style, explicit credential environment variables, and `secrets.directory` handling match `run`.
-
-## Dry Runs
-
-`--dry-run` loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files, destination state, or SSH `known_hosts` entries.
-
-Dry-run output is useful before publishing to confirm actions such as `publish_new`, `replace_older`, `force_replace`, `skip_same`, and `skip_destination_newer`.
-
-Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication. Fixed path destinations add `path_mapping=fixed target=.` to planned action lines. Dry-run also prints a warning with the fixed destination candidate count and selected source bundle; destructive fixed replacements print an additional warning that the destination root would be replaced.
-
-Use `--format json` when another process needs stable run data. JSON output includes warnings, pipeline summaries, destination actions, destination bundle paths, path mapping modes, optional link URLs, output records, final counters, and partial failure records. The summary includes `fixed_path`. If one destination fails after planning or execution begins, JSON output still contains the successful and failed destination records with `ok: false`, and the command exits non-zero.
-
-## Retry and Replacement Behavior
-
-If a destination has matching `.distributor.json`, publication skips it as already published.
-
-If destination state is older than the source manifest and transfer policy allows replacement, publication deletes only managed outputs recorded in `.distributor.json` plus the state file, then writes the new outputs and state.
-
-If destination state is newer than the source manifest, the default behavior is to skip. If destination state has the same source id and created timestamp but a different digest, publication fails as a conflict.
-
-If a destination path has files but no valid `.distributor.json`, publication fails as unmanaged content unless the current run explicitly uses `--force`.
-
-## Force Workflow
-
-Use `--force` only after a dry run shows the intended `force_replace` action:
-
-```sh
-go run ./cmd/distributor run --config <config-path> --dry-run --force
-go run ./cmd/distributor run --config <config-path> --force
-```
-
-Forced replacement can overwrite unmanaged non-empty destination paths. Destination state conflicts require `transfer.on_conflict: replace` plus `--force`. Newer destination state requires `transfer.on_destination_newer: replace` plus `--force`.
-
-Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete above that bundle path. For fixed destinations, the destination bundle path is the backend root, so forced replacement may clear that configured root but not its parent path, sibling directories, or anything outside the configured S3 bucket and prefix. Force is per run only and has no config default.
-
-## Failure Handling
-
-If one destination fails in a fan-out run, independent later destinations are still planned and executed. The command exits non-zero after printing the final status if any destination failed.
-
-Errors include the pipeline id, destination id, destination backend, and bundle path where applicable.
-
-In JSON mode, destination failures after planning or execution begins are reported in the top-level `errors` array and in the run result while preserving a non-zero exit code. Fatal setup errors such as an unreadable config or invalid secrets directory write no JSON document.
-
-If a write fails during publication, `distributor` attempts to remove outputs written during that failed attempt so a retry does not see those partial outputs as unmanaged destination content.
-
-After a successful publish or replacement, the internal notifier hook runs. The current default notifier is a no-op. Skipped destinations do not invoke it.
-
-## SSH Operation Notes
-
-SSH execution uses SFTP over `golang.org/x/crypto/ssh` and `github.com/pkg/sftp`. It does not shell out to `ssh`, `scp`, or `rsync`.
 
 Configure `ssh_key_file`, an SSH agent, or both. Agent identities are attempted first, followed by the configured key file. YAML password authentication is not supported.
 
-The default host key policy is `accept-new`. New host keys are written to `known_hosts` when the file path is writable. During `--dry-run`, unknown host keys may be accepted for the current connection but are not written to `known_hosts`; a later non-dry-run may persist the same key. Changed host keys are fatal for both `strict` and `accept-new`. The `off` policy disables host key checking and `run` prints a warning when stdout is enabled.
+The default host key policy is `accept-new`. During dry runs, unknown host keys may be accepted for the current connection but are not persisted. Changed host keys are fatal for `strict` and `accept-new`. `host_key_policy: off` disables host key checking and should be limited to controlled test environments.
 
-Recovery boundaries are the same as local storage: replacement deletes only managed output paths recorded in `.distributor.json` plus the state file, and failed writes are cleaned up where practical. Distributor never performs broad recursive remote deletion.
+### S3-Compatible Storage
 
-## S3 Operation Notes
+S3 execution uses the AWS SDK for Go v2. See [S3-Compatible Storage Integration](integrations/s3.md). Configure an endpoint, bucket, optional prefix, optional region, optional path-style setting, and optional explicit credential variable names.
 
-S3 execution uses the AWS SDK for Go v2. Configure `endpoint`, `bucket`, optional `prefix`, optional `region`, and optional explicit credential environment variable names.
+When explicit S3 credential variable names are configured, both must resolve to non-empty values through the process environment or `secrets.directory`. When omitted, the AWS SDK default credential chain is used as-is.
 
-When explicit credential env names are configured, both variables must resolve to non-empty values through the real process environment or `secrets.directory`. When they are omitted, the AWS SDK default credential chain is used as-is.
+Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.
 
-Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. For fixed destinations, that prefix is the configured bucket plus optional `prefix`. Distributor does not manage bucket versioning or delete markers.
+## Secrets Operation
 
-## Secrets Directory
+`secrets.directory` is loaded during `run`, `serve`, and configured-source `validate` or `inspect` before credential-consuming work starts. If the directory is missing, unreadable, or contains an invalid secret filename, the command fails before storage work starts.
 
-Configure `secrets.directory` when credential values should come from mounted files, such as deployment secrets:
+Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, commands emit a warning naming the ignored secret variable without printing either value. The process environment is not modified.
 
-```yaml
+## Cleanup And Recovery
-secrets:
-  directory: /run/secrets/distributor
-```
 
-The directory is loaded during `run`, `serve`, and configured-source `validate`
+Use these recovery boundaries:
-or `inspect` before credential-consuming work starts. If the directory is
-missing, unreadable, or contains an invalid secret filename, the command fails
-before storage work starts.
 
-Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, `run` and configured-source diagnostics print a warning naming the ignored secret file variable without printing either value. The process environment is not changed.
+- For source validation failures, regenerate the source bundle and manifest together.
+- For an empty or missing destination, rerun after fixing config or storage access.
+- For unmanaged destination content, move unrelated files aside or use a different destination path before publishing.
+- For failed writes, inspect the destination bundle path, remove only confirmed partial outputs if needed, then rerun `--dry-run`.
+- For state conflicts, verify the source, pipeline, destination, and existing `.distributor.json` before considering `--force`.
+- For HTTP upload failures, inspect `/runs/<run-id>` while retained; after expiry or restart, rely on destination state and logs/output from the publishing run.
 
-## Caveats
+Do not edit `.distributor.json` during normal recovery. Treat it as the managed state record used for comparison and safe cleanup.
 
-External notification adapters are unavailable. Force overwrite behavior is available only through the explicit `run --force` workflow.
+For symptom-specific fixes, see [Troubleshooting](troubleshooting.md).
-
-For symptom-oriented fixes, see [troubleshooting](troubleshooting.md). For config details, see [configuration](config.md). For command syntax, see [CLI](cli.md).

docs/roadmap/audit.md

@@ -1,592 +0,0 @@
-# Code Quality and Deduplication Audit
-
-## 1. Executive summary
-
-Overall code quality is strong. The repository has clear package boundaries, good current-behavior documentation, focused adapter packages, and tests close to most implemented behavior. The most important cleanup opportunities are narrow and behavior-preserving rather than architectural.
-
-Top three refactoring targets:
-
-1. HTTP upload admission, body staging, and archive validation are split across `internal/app` and `internal/ingest` in a way that duplicates size and content-type policy and buffers uploads in memory.
-2. Runtime config loading, default config path selection, secret loading, and warning projection are repeated across app entrypoints.
-3. Run orchestration mixes destination processing, failure aggregation, warning recording, and report event ordering in one large loop, making future changes harder to review safely.
-
-The codebase appears ready for a limited cleanup pass. I do not see a major architectural risk that requires a redesign before the next release.
-
-## 2. Repository map reviewed
-
-Reviewed policy and current-behavior documentation:
-
-- `AGENTS.md`
-- `README.md`
-- `docs/policy/architecture.md`
-- `docs/policy/development.md`
-- `docs/policy/documentation.md`
-- `docs/config.md`
-- `docs/cli.md`
-- `docs/operations.md`
-- `docs/troubleshooting.md`
-- `docs/internal/*.md`
-- `docs/roadmap/http.md`
-- `docs/roadmap/implementation.md`
-
-Reviewed implementation areas:
-
-- `cmd/distributor`: executable entrypoint.
-- `internal/cli`: root command, `version`, `run`, `serve`, `validate`, `inspect`, and `manifest create` parsing.
-- `internal/app`: run orchestration, configured source diagnostics, backend factory, manifest creation, CLI output, HTTP upload server, upload coordinator, and pipeline coordinator.
-- `internal/config`: config structs, defaults, validation, quantity parsing, S3/SSH helpers, and secrets resolver.
-- `internal/bundle`: storage-backed source discovery and validation.
-- `pkg/bundle`: public manifest model, digest logic, manifest building, local validation, and local bundle writer.
-- `internal/storage` and `internal/storage/fake`: backend interface, path helpers, walk helpers, typed errors, and fake backend.
-- `internal/adapters/local`, `internal/adapters/ssh`, and `internal/adapters/s3`: runtime storage adapters.
-- `internal/ingest`: HTTP upload archive staging.
-- `internal/publish`: destination planning, output selection, link projection, state writing, cleanup, and force replacement.
-- `internal/state`: destination state parsing, validation, comparison, and JSON projection.
-- `internal/transform` and `internal/transform/markdown`: transform registry and Markdown rendering.
-- `internal/link`, `internal/notify`, `internal/logging`, and `internal/testutil`.
-- `examples`, package tests, and package `testdata`.
-
-Requested areas that are absent as separate packages:
-
-- `internal/stage`
-- `internal/modules`
-- `internal/validators`
-- `internal/artifacts`
-- `internal/manifest`
-- `internal/schema`
-- `internal/report`
-- public `pkg` packages other than `pkg/bundle`
-
-Those absences are consistent with current architecture policy; the corresponding behavior lives in narrower existing packages.
-
-## 3. High-confidence deduplication opportunities
-
-### HTTP upload body handling should be owned by ingestion
-
-Affected files/packages:
-
-- `internal/app/upload_http.go`
-- `internal/app/upload_coordinator.go`
-- `internal/ingest/archive.go`
-- `internal/app/upload_http_test.go`
-- `internal/app/upload_http_integration_test.go`
-- `internal/ingest/archive_test.go`
-
-Duplicated or near-duplicated behavior:
-
-- `internal/app/upload_http.go` validates upload content types in `supportedUploadContentType`, while `internal/ingest/archive.go` validates the same content types in `archiveFormat`.
-- `internal/app/upload_http.go` enforces upload size in `readUploadBody`, while `internal/ingest/archive.go` enforces upload size again in `writeLimited`.
-- The HTTP handler reads the full upload body into memory before submission, then the coordinator passes a `bytes.Reader` to ingestion.
-
-Why it matters:
-
-- The app transport layer now partially owns archive policy that should belong to `internal/ingest`.
-- Large accepted uploads are buffered in memory even though ingestion already has streaming-to-disk mechanics.
-- Future archive formats, content types, or upload limit changes would need coordinated edits in multiple packages.
-
-Recommended refactor:
-
-- Move supported content-type checking behind an ingestion-owned helper, for example `ingest.ValidateContentType` or `ingest.IsSupportedContentType`.
-- Change upload admission so the request body is streamed to staging exactly once before the HTTP handler returns `202 Accepted`.
-- Keep queue-full rejection before reading the body.
-- Queue a staged local bundle root, not an unread request body. This preserves async distribution while keeping HTTP request lifetime separate from later pipeline execution.
-- Keep `UploadCoordinator` responsible for queueing, status, per-pipeline serialization, and execution. Keep `internal/ingest` responsible for archive format, size, extraction, cleanup, and source bundle validation.
-
-Suggested tests:
-
-- HTTP handler rejects full queues without reading the body.
-- HTTP handler streams a valid body to ingestion and returns `202` only after staging succeeds.
-- Unsupported content types are rejected through the ingestion-owned content-type policy.
-- Oversized uploads are rejected without retaining a staged run.
-- Accepted upload status still transitions through queued/running/succeeded or failed without depending on an open HTTP request body.
-
-Risk level:
-
-- Medium. The behavior change is internal but touches admission timing and async execution boundaries. It should be implemented in a focused prompt with existing HTTP integration tests extended first.
-
-### Runtime config and secret setup should have one app-level helper
-
-Affected files/packages:
-
-- `internal/app/run.go`
-- `internal/app/source_select.go`
-- `internal/app/serve.go`
-- `internal/app/backends.go`
-- `internal/app/run_warnings.go`
-- `internal/config`
-- `internal/app/*_test.go`
-
-Duplicated or near-duplicated behavior:
-
-- Defaulting an empty config path to `config.DefaultConfigPath` appears in `Run`, `RunPipeline`, `RunPipelineWithLocalSource`, and `Serve`.
-- Config loading and secret loading are separate repeated steps in `buildRunReport`, `selectSourceBundlesFromConfig`, and `Serve`.
-- Secret conflict warnings are projected in run and configured source diagnostics, while serve loads secrets without using or exposing conflict warning metadata.
-- Backend factory construction from a config environment is repeated through provider plumbing.
-
-Why it matters:
-
-- Config and secret precedence is a public operational policy.
-- A future change to config discovery, secret conflict reporting, or runtime environment construction could drift between `run`, `serve`, `validate`, and `inspect`.
-- Tests for secrets and credential resolution need to cover several entrypoints today.
-
-Recommended refactor:
-
-- Add a small app-level runtime setup helper, for example `loadRuntimeConfig(optionsConfigPath string) (runtimeConfig, error)`.
-- The helper should own default config path selection, `config.LoadFile`, `config.LoadSecretEnvironment`, and conversion of secret conflicts into `OutputWarning` values.
-- Keep config parsing and validation in `internal/config`; the helper should not duplicate config policy.
-- Let `run`, configured `validate`/`inspect`, and `serve` call the helper and then apply command-specific behavior.
-
-Suggested tests:
-
-- One focused app test proving default config path selection remains unchanged where injection permits it.
-- Existing secret conflict JSON/text warning tests for `run`, `validate`, and `inspect`.
-- Serve startup test proving duplicate and missing upload tokens still fail without leaking values.
-- S3 explicit credential tests proving the resolver is still used through the helper.
-
-Risk level:
-
-- Low to medium. This is a straightforward centralization but touches several command entrypoints.
-
-### Run destination processing needs a narrow helper boundary
-
-Affected files/packages:
-
-- `internal/app/run.go`
-- `internal/app/run_output.go`
-- `internal/app/run_failures.go`
-- `internal/app/run_selection.go`
-- `internal/publish`
-- `internal/app/run_test.go`
-
-Duplicated or near-duplicated behavior:
-
-- Destination backend open failures and publish planning/execution failures each manually add `runFailures`, record summary failure counts, append `RunActionRecord`, and append pipeline event indexes.
-- `publish.Build` error handling patches missing `Plan` identity fields inline before converting the plan to a run action.
-- Fixed-path warning emission is interleaved with destination selection and publish plan handling.
-
-Why it matters:
-
-- `run --format json` depends on exact action ordering, warnings, partial failures, and summary counters.
-- Future changes to actions, links, notifications, or HTTP upload reports could accidentally update one failure path but not another.
-- The current loop is correct but dense enough that small behavior changes are hard to review.
-
-Recommended refactor:
-
-- Extract a narrow `runDestination` or `destinationRunner` helper that processes one destination and returns action records, warnings, summary deltas, and failures.
-- Add a helper for recording a destination-scoped failure that updates `runFailures`, `runSummary`, `RunReport.Actions`, and pipeline events in one place.
-- Add a helper that normalizes partial `publish.Plan` identity fields before action projection.
-- Do not introduce a generic workflow engine or stage abstraction.
-
-Suggested tests:
-
-- Preserve existing run text output golden assertions.
-- Preserve JSON partial-result behavior when planning fails after destination processing begins.
-- Add one focused test where destination open fails for multiple selected bundles and verify action records, output errors, and summary counters stay aligned.
-- Add one fixed-path dry-run warning test after extraction to verify event ordering.
-
-Risk level:
-
-- Medium. The refactor is behavior-preserving but touches the most important user-facing workflow.
-
-### Archive path validation duplicates source path policy with a different error surface
-
-Affected files/packages:
-
-- `internal/ingest/archive.go`
-- `pkg/bundle/path.go`
-- `internal/storage/path.go`
-- `internal/ingest/archive_test.go`
-
-Duplicated or near-duplicated behavior:
-
-- `cleanArchivePath`, `pkg/bundle.ValidateSourcePath`, and `storage.ValidatePath` all enforce clean slash-separated relative paths with no backslashes, no absolute paths, and no dot segments.
-- Archive staging needs slightly different policy because directories are allowed and `manifest.json` is allowed only at the root, so the duplication is not completely mechanical.
-
-Why it matters:
-
-- Path safety is high-risk behavior.
-- Future changes to source path rules could miss archive extraction, especially around backslashes, reserved names, or dot segments.
-
-Recommended refactor:
-
-- Keep archive-specific rules in `internal/ingest`, but use a shared path-checking primitive where possible.
-- A good shape is an exported `pkg/bundle.ValidatePathSegmented` only if it fits the public producer API, or an internal helper in ingestion that delegates file-entry validation to `pkg/bundle.ValidateSourcePath` for regular files after handling directory-specific exceptions.
-- Preserve current archive-specific errors and tests.
-
-Suggested tests:
-
-- Table tests shared or mirrored across bundle path validation and archive path cleaning for absolute paths, traversal, backslashes, dot segments, empty names, root `manifest.json`, nested `manifest.json`, and `.distributor.json`.
-- Regression tests proving directories are still accepted in archives but symlinks and hardlinks remain rejected.
-
-Risk level:
-
-- Low to medium. Path validation changes need careful tests, but the desired change can be small.
-
-## 4. Medium-confidence opportunities
-
-### Source and destination backend config shapes could expose a normalized view
-
-Affected files/packages:
-
-- `internal/config/config.go`
-- `internal/config/defaults.go`
-- `internal/config/validate.go`
-- `internal/app/backends.go`
-- `internal/config/*_test.go`
-- `internal/app/backends_test.go`
-
-Duplicated or near-duplicated behavior:
-
-- `config.Backend` and `config.Destination` duplicate backend fields for local, SSH, S3, and credentials.
-- Defaults for source backends and destination backends are implemented in separate functions.
-- App backend opening converts both shapes into `backendOpenSpec`.
-
-Why it matters:
-
-- New backend fields must be added to both YAML structs, defaulting paths, validation paths, app open-spec conversion, docs, and tests.
-- The current pattern is easy to understand but likely to drift as more backend-specific fields are added.
-
-Recommended refactor:
-
-- Keep the YAML shape unchanged for compatibility.
-- Add package-local helpers in `internal/config` that return a normalized backend view for either source or destination.
-- Use that view for shared backend defaulting and validation where it improves clarity.
-- Keep destination-only fields such as `publish`, `transfer`, `links`, and `path_mapping` on `Destination`.
-
-Suggested tests:
-
-- Existing source and destination backend validation tests should continue to pass.
-- Add a table test that validates equivalent local, SSH, and S3 source/destination backend field requirements through the shared view.
-- Add a test that `http_upload` remains source-only.
-
-Risk level:
-
-- Medium. This reduces future drift, but the current duplication is understandable and does not need to be the first cleanup.
-
-### CLI command scaffolding is mostly shared, but manifest create has special parsing
-
-Affected files/packages:
-
-- `internal/cli/run.go`
-- `internal/cli/serve.go`
-- `internal/cli/source_mode.go`
-- `internal/cli/manifest.go`
-- `internal/cli/version.go`
-- `internal/cli/root_test.go`
-
-Duplicated or near-duplicated behavior:
-
-- Several commands repeat `flag.NewFlagSet`, `SetOutput`, help handling, format parsing, and usage exit handling.
-- `manifest create` uses `splitManifestCreateArgs` to allow a positional bundle path before flags, unlike Go's default `flag` behavior.
-
-Why it matters:
-
-- CLI syntax and error behavior are public.
-- A broad CLI helper could accidentally obscure command-specific parsing, but a narrow helper could reduce repeated setup and invalid-format handling.
-
-Recommended refactor:
-
-- Do not introduce a CLI framework.
-- Consider a tiny helper for common `FlagSet` creation and output-format parsing after higher-value app/config cleanup.
-- Keep `manifest create` custom parsing local unless another command needs the same interspersed positional behavior.
-
-Suggested tests:
-
-- Preserve current CLI usage-error tests.
-- Add explicit tests for `manifest create <path> --id x`, `manifest create --id x <path>`, and invalid missing flag values before any parser cleanup.
-
-Risk level:
-
-- Low if kept narrow; medium if over-generalized.
-
-### Output DTOs repeat bundle metadata projection
-
-Affected files/packages:
-
-- `internal/app/validate.go`
-- `internal/app/inspect.go`
-- `internal/app/manifest.go`
-- `internal/app/run_output.go`
-- `internal/app/output.go`
-
-Duplicated or near-duplicated behavior:
-
-- `inspect` and `manifest create` both project bundle file metadata into command-specific JSON structs.
-- `validate`, `inspect`, and `manifest create` each define local result types and file record types.
-- RFC3339 formatting uses both `time.RFC3339` and the equivalent literal layout string.
-
-Why it matters:
-
-- JSON output is now a public interface.
-- Repeated projection can drift in field names, timestamp formatting, or path display rules.
-
-Recommended refactor:
-
-- Add a small app-local projection helper for bundle summaries and manifest file records.
-- Use `time.RFC3339` instead of literal RFC3339 layouts.
-- Keep command-specific result structs where the command output semantics differ.
-
-Suggested tests:
-
-- JSON structural tests for `validate`, `inspect`, and `manifest create` before and after the helper.
-- A timestamp-format assertion using an offset timestamp to confirm current behavior is preserved.
-
-Risk level:
-
-- Low.
-
-### PipelineRunCoordinator overlaps conceptually with UploadCoordinator
-
-Affected files/packages:
-
-- `internal/app/run_coordinator.go`
-- `internal/app/upload_coordinator.go`
-- `docs/internal/app.md`
-- `internal/app/run_coordinator_test.go`
-- `internal/app/upload_coordinator_test.go`
-
-Duplicated or near-duplicated behavior:
-
-- Both coordinators define run records, statuses, timestamps, status transitions, context handling, and active pipeline protection.
-- The upload coordinator additionally queues, stages, expires status records, and serializes same-pipeline upload execution.
-
-Why it matters:
-
-- The concepts are similar enough to confuse future contributors.
-- However, the behavior is not identical: one rejects duplicate active runs, while the other queues accepted uploads.
-
-Recommended refactor:
-
-- Do not merge the coordinators now.
-- Review whether `PipelineRunCoordinator` is still needed as an exported app-level helper. If it is intended for future transports, document that role clearly. If not, remove it and its tests in a separate dead-code cleanup.
-- If both remain, extract only tiny shared timestamp/status helpers if a real third coordinator appears.
-
-Suggested tests:
-
-- If retained, keep existing duplicate-run tests.
-- If removed, run `go test ./internal/app ./internal/cli` and verify no current behavior depended on it.
-
-Risk level:
-
-- Low for documentation clarification, medium for removal because it is exported from an internal package and documented for maintainers.
-
-## 5. Boundary and responsibility concerns
-
-The major boundaries are sound:
-
-- CLI parsing stays in `internal/cli`.
-- Config defaults and validation stay in `internal/config`.
-- Backend-specific filesystem, SFTP, and S3 behavior stays in adapters.
-- Manifest semantics are centralized in `pkg/bundle`, with `internal/bundle` adding storage-backed discovery and validation.
-- Destination state comparison stays in `internal/state`.
-- Publish planning/execution stays in `internal/publish`.
-- Transform implementation is behind `internal/transform`.
-
-Concerns worth addressing:
-
-- HTTP upload request-body staging currently crosses the app/ingest boundary. The app transport layer should not own body buffering and archive size enforcement beyond admission and HTTP status projection.
-- Runtime config setup is app-layer behavior, but it is repeated rather than named. A runtime setup helper would clarify the boundary between `internal/config` and command-specific execution.
-- `internal/app/run.go` owns too many destination-loop details. Extracting a destination processing helper would keep orchestration in app while reducing local complexity.
-
-Recommended homes:
-
-- Upload archive policy: `internal/ingest`.
-- HTTP route/auth/status mapping: `internal/app/upload_http.go`.
-- Queueing/status/execution: `internal/app/upload_coordinator.go`.
-- Runtime config plus secret setup: a small helper in `internal/app`, using `internal/config`.
-- Path and state filenames: keep in `internal/storage`.
-
-## 6. Path, key, and naming construction review
-
-Centralized and healthy areas:
-
-- `storage.StateFileName`, `storage.StatePath`, `storage.ManagedBundleTargets`, `storage.Join`, `storage.DisplayPath`, and logical path validation are used in core publication and tests.
-- S3 object-key mapping is contained in `internal/adapters/s3`.
-- SSH and local native path conversion stay inside their adapters.
-- Link URL construction is isolated in `internal/publish/links.go` and URL validation in `internal/link`.
-- Manifest name and schema version are centralized in `pkg/bundle`, with `internal/bundle` aliases.
-
-Areas needing cleanup:
-
-- Archive path cleaning duplicates much of source/storage path policy and should either delegate to a shared primitive or be tightly covered by mirrored tests.
-- Upload run ID construction is isolated, but the shape is partly policy. Keep tests around `<pipeline_id>.<utc_timestamp>.<random_suffix>` before changing coordinator code.
-- Some app tests still construct destination state and source paths locally. `internal/testutil` already covers many cases; additional helper use should be opportunistic, not a sweeping test rewrite.
-
-## 7. Resolution and catalog review
-
-Named concept resolution is mostly consistent:
-
-- Backend names are defined in `internal/config/defaults.go`.
-- Runtime backend construction is app-owned through `backendFactory` and the storage registry.
-- Transform names are defined in `internal/transform`, and app wiring owns concrete registration.
-- Publish/transform policy combinations use `config.ValidatePublishTransformPolicy`.
-- Configured source selection for `validate` and `inspect` is shared in `source_select.go`.
-
-Potential refinements:
-
-- A normalized backend config view would make backend field resolution less repetitive across source and destination config.
-- Transform and backend registries should remain separate; there is no evidence that a generic registry abstraction would help.
-- No separate catalog package is needed for the current feature set.
-
-## 8. Config and command-loading review
-
-Config loading is reliable and strict:
-
-- YAML known-field checking is enabled.
-- Defaults are applied before validation.
-- Validation collects multiple field errors.
-- Secrets are loaded without mutating `os.Environ`.
-- Explicit S3 credential references use the config-owned resolver.
-
-Likely accidental duplication:
-
-- Default config path selection and `config.LoadFile` are repeated in several app entrypoints.
-- Secret loading is repeated in run, source diagnostics, and serve.
-- Secret conflict warning projection is not represented by one runtime setup result.
-
-Intentional differences:
-
-- `serve` loads upload tokens and does not produce CLI JSON output.
-- `validate` and `inspect` support local-path shortcut mode, while `run` and `serve` are config-driven.
-- `manifest create` is local filesystem producer tooling and does not load app config.
-
-Recommended cleanup:
-
-- Centralize runtime config and secret setup in `internal/app`.
-- Keep CLI flag parsing local to command files.
-- Keep `manifest create` outside runtime config loading.
-
-## 9. State, manifest, or progress handling review
-
-Manifest handling is in good shape:
-
-- `pkg/bundle` owns manifest parsing, digest grammar, source path validation, canonical bundle digest, local manifest building, and local bundle writing.
-- `internal/bundle` delegates normalized manifest semantics to `pkg/bundle` and adds storage-backed validation.
-- Destination state embeds the normalized manifest and validates through `internal/bundle`/`pkg/bundle`.
-
-State handling is in good shape:
-
-- `.distributor.json` parsing, validation, JSON projection, and comparison live in `internal/state`.
-- Publish execution writes destination state only after outputs are written.
-- Managed replacement deletes only state-listed outputs plus `.distributor.json`; forced replacement is explicit and bounded.
-
-Progress/status handling:
-
-- `RunReport` is the core run result model and supports JSON partial-result output.
-- HTTP upload status is memory-only and documented as such.
-- `PipelineRunCoordinator` and `UploadCoordinator` overlap conceptually but have different policies. Avoid merging unless product behavior converges.
-
-Gaps:
-
-- HTTP upload staging currently stores the request body in memory before queueing. This is both a quality gap and a mismatch with the intended ingestion boundary.
-- There is no durable upload status, but this is documented as deferred work and should not be addressed in cleanup.
-
-## 10. Refactors to avoid
-
-Avoid these changes in the cleanup pass:
-
-- Do not introduce a generic workflow engine or stage framework. The current explicit workflow is easier to audit.
-- Do not add a CLI framework. The standard-library CLI is sufficient and policy-approved.
-- Do not merge local, SSH, S3, and fake adapters behind a shared implementation layer. Their semantics differ enough that generic helpers would likely hide important behavior.
-- Do not collapse `pkg/bundle` and `internal/bundle`. The public producer API and storage-backed distributor validation have different responsibilities.
-- Do not move destination state comparison into `publish` or app orchestration.
-- Do not redesign JSON output envelopes while doing cleanup.
-- Do not add durable queues, retry workers, HTTP TLS, zstd, or browser UI under the banner of refactoring. These are feature work.
-- Do not rewrite tests wholesale to use a new fixture system. Add helpers only where they reduce immediate duplication around changed code.
-
-## 11. Recommended implementation sequence
-
-1. HTTP upload staging boundary cleanup.
-   - Move supported content-type policy to `internal/ingest`.
-   - Stop buffering accepted uploads in `upload_http.go`.
-   - Queue staged bundle roots rather than request bodies.
-   - Extend HTTP upload tests first.
-
-2. Runtime config setup helper.
-   - Add an app-level helper for default config path, config load, secret load, environment resolver, and secret warnings.
-   - Use it from `run`, configured `validate`/`inspect`, and `serve` where applicable.
-   - Preserve command-specific behavior.
-
-3. Run destination processing extraction.
-   - Add small helpers for destination-scoped failure recording and plan identity normalization.
-   - Extract one-destination processing only if the helper remains readable.
-   - Preserve action ordering and report output.
-
-4. Backend config normalized view.
-   - Add source/destination backend view helpers in `internal/config`.
-   - Use them for defaulting and validation if tests show the shape remains clear.
-   - Keep YAML structs and public config unchanged.
-
-5. Bundle output projection cleanup.
-   - Add app-local helpers for file record and bundle summary projection.
-   - Use `time.RFC3339` consistently.
-   - Preserve command-specific JSON field names.
-
-6. Archive/source path validation test alignment.
-   - Add mirrored path safety tests around ingestion and bundle path validation.
-   - Only centralize code if the helper does not blur archive directory semantics.
-
-7. Coordinator intent cleanup.
-   - Decide whether `PipelineRunCoordinator` is retained for internal future use.
-   - If retained, clarify comments/docs. If removed, do it as a separate dead-code commit.
-
-8. Test helper cleanup.
-   - Expand `internal/testutil` only for repeated setup touched by the previous refactors.
-   - Avoid moving every test fixture.
-
-## 12. Test strategy
-
-Tests to add before refactoring:
-
-- HTTP upload handler test proving queue-full rejection does not consume the body.
-- HTTP upload test proving accepted upload staging completes before `202 Accepted`.
-- Ingestion content-type policy tests exposed through the new helper.
-- Run report test covering destination open failure for multiple selected bundles.
-- CLI JSON tests for `inspect` and `manifest create` timestamp formatting before projection cleanup.
-
-Tests to run with each cleanup stage:
-
-- HTTP upload cleanup: `go test ./internal/ingest ./internal/app ./internal/cli`
-- Config setup cleanup: `go test ./internal/config ./internal/app ./internal/cli`
-- Run processing cleanup: `go test ./internal/app ./internal/publish ./internal/state`
-- Backend config view cleanup: `go test ./internal/config ./internal/app`
-- Output projection cleanup: `go test ./internal/app ./internal/cli`
-- Path validation cleanup: `go test ./pkg/bundle ./internal/bundle ./internal/ingest ./internal/storage`
-- Final cleanup validation: `go test ./...`
-
-Useful read-only checks:
-
-- `rg -n "LoadFile\\(|LoadSecretEnvironment\\(|DefaultConfigPath" internal/app internal/cli`
-- `rg -n "application/x-tar|application/gzip|application/x-gzip" internal docs`
-- `rg -n "2006-01-02T15:04:05Z07:00" internal pkg`
-- `rg -n "manifest.json|\\.distributor.json|StatePath|DisplayPath" internal pkg`
-
-## 13. Appendix: findings not worth acting on
-
-Adapter `ReadFile` and `WriteFile` wrappers:
-
-- Local, SSH, S3, and fake backends each implement byte helpers in terms of stream helpers. This is small duplication but appropriate because each adapter owns error translation and metadata semantics.
-
-Adapter traversal implementation:
-
-- Local filesystem walking, SFTP walking, and S3 pagination look similar at the interface level but are semantically different. Keep traversal mechanics in adapters and shared callback behavior in `storage.WalkEmitter`.
-
-State and manifest raw JSON parsing:
-
-- `pkg/bundle` and `internal/state` both parse raw JSON with pointer fields to detect missing required fields. The schemas and error contexts differ, so a generic required-field parser would not be worth the complexity.
-
-CLI help text:
-
-- Help text repeats command names and flags. This is acceptable in a small hand-written CLI and keeps command files readable.
-
-Test fixture strings:
-
-- Some tests inline YAML snippets or expected output strings despite `internal/testutil`. Inline data is often clearer for edge cases. Only centralize fixture setup when tests are already being changed for a behavior-preserving refactor.
-
-HTTP JSON response helpers:
-
-- HTTP API responses use simple JSON objects rather than the CLI JSON envelope. This is intentional because HTTP status codes and route-specific responses are not the same public interface as CLI command output.
-
-Public and internal bundle validation:
-
-- `pkg/bundle.ValidateBundle` is local-filesystem producer validation; `internal/bundle.Validate` is storage-backed distributor validation. Keep both, with shared manifest semantics delegated through `pkg/bundle`.

docs/roadmap/cleanup.md

@@ -1,400 +0,0 @@
-# Code Quality Cleanup Roadmap
-
-## Current Baseline
-
-The codebase has completed the local, SSH/SFTP, S3, public bundle package,
-manifest creation, JSON output, path mapping, link generation, and HTTP upload
-work documented in the current user and internal docs.
-
-The audit in `docs/roadmap/audit.md` found no major architectural risk. The
-remaining cleanup work should be narrow, behavior-preserving, and focused on
-reducing drift in upload staging, runtime config setup, run reporting, backend
-config handling, output projection, path validation tests, and internal
-coordination code.
-
-One intentional behavior change is part of this cleanup roadmap: malformed
-authenticated upload archives should be rejected before `202 Accepted`, rather
-than accepted and later marked failed. Valid staged uploads should still run
-asynchronously after admission.
-
-## Cleanup Principles
-
-- Preserve public CLI behavior, config schema, manifest schema, destination
-  state schema, backend behavior, and JSON envelopes unless a stage explicitly
-  says otherwise.
-- Keep config parsing and validation in `internal/config`.
-- Keep CLI parsing in `internal/cli`.
-- Keep upload archive policy in `internal/ingest`; keep HTTP routing,
-  authentication, and status projection in `internal/app`.
-- Keep backend-specific filesystem, SSH/SFTP, and S3 behavior in adapter
-  packages.
-- Prefer small package-local helpers over broad abstractions.
-- Add or strengthen tests before refactoring behavior that affects public
-  output, upload admission, path safety, or run reporting.
-
-## Active Cleanup Stages
-
-Implement these stages in order. Each stage should be small enough for one
-implementation prompt and should leave the repository passing the listed focused
-tests before moving to the next stage.
-
-## Stage 1: HTTP Upload Staging Boundary
-
-Goal:
-
-Move archive validation and upload body staging fully behind `internal/ingest`,
-stop app-layer full-body buffering, and reject malformed archives before
-returning `202 Accepted`.
-
-Implementation scope:
-
-- Add an ingestion-owned content-type helper, such as
-  `ValidateContentType(contentType string) error`, and remove duplicated
-  content-type policy from the HTTP handler.
-- Replace the current handler-side `readUploadBody` buffering with streaming
-  staging through `internal/ingest`.
-- Introduce a two-step upload coordinator admission model:
-  - reserve a run id and queue slot before consuming the request body;
-  - stage and validate the archive using that reserved run id;
-  - enqueue only a successfully staged local bundle root for async execution.
-- Keep queue-full rejection before reading the body.
-- Preserve `401` for missing or invalid bearer tokens, `415` for unsupported
-  content type, `413` for oversized uploads, and `503` for a full queue.
-- Return a pre-acceptance `400` for malformed tar/gzip content or invalid
-  staged bundles.
-- Preserve async queued/running/succeeded/failed status after a valid staged
-  bundle is accepted.
-- Do not add durable queues, idempotency keys, zstd, or new routes.
-
-Current-behavior documentation updates:
-
-- Update `docs/cli.md`, `docs/config.md`, `docs/operations.md`,
-  `docs/troubleshooting.md`, `docs/internal/app.md`, and
-  `docs/internal/ingest.md` only as needed to describe the new
-  pre-acceptance failure boundary.
-
-Tests:
-
-- `go test ./internal/ingest ./internal/app ./internal/cli`
-- Queue-full upload rejection does not read the request body.
-- Unsupported content type is rejected through ingestion-owned policy.
-- Oversized uploads return `413` and do not retain a staged run.
-- Malformed tar/gzip content returns `400` before a run id is issued.
-- Valid tar and tar.gz uploads return `202` after staging and still transition
-  through async status.
-- HTTP responses and status records do not leak bearer tokens or secret values.
-
-Completion criteria:
-
-- `internal/app` no longer buffers the full upload body before staging.
-- A valid accepted upload has a committed staged bundle root before the `202`
-  response is sent.
-- Invalid archive content cannot create an accepted run id.
-
-## Stage 2: Runtime Config And Secret Setup Helper
-
-Goal:
-
-Centralize runtime config path resolution, config loading, secret loading,
-environment resolver creation, and secret-conflict warning projection in one
-app-layer helper.
-
-Implementation scope:
-
-- Add a small `internal/app` runtime setup helper that:
-  - defaults an empty config path to `config.DefaultConfigPath`;
-  - calls `config.LoadFile`;
-  - calls `config.LoadSecretEnvironment`;
-  - exposes the loaded config, config path, `config.Environment`, and
-    `[]OutputWarning` for secret conflicts.
-- Use the helper from `Run`, `RunPipeline`, `RunPipelineWithLocalSource`,
-  configured `Validate`/`Inspect`, and `Serve` where applicable.
-- Keep `manifest create` outside runtime config loading.
-- Keep YAML structs, defaults, validation, and secret-directory parsing in
-  `internal/config`.
-- Preserve app test injection points for backend factories and upload handler
-  tests.
-
-Current-behavior documentation updates:
-
-- Update `docs/internal/app.md` if helper boundaries or flow descriptions
-  change. User-facing docs should not change unless observable behavior changes.
-
-Tests:
-
-- `go test ./internal/config ./internal/app ./internal/cli`
-- Default config path behavior remains unchanged.
-- Secret conflict warnings still appear in text and JSON output for `run`,
-  configured `validate`, and configured `inspect`.
-- `serve` still fails startup safely for missing, empty, or duplicate upload
-  tokens without leaking values.
-- Explicit S3 credential references still resolve through the config-owned
-  environment resolver.
-
-Completion criteria:
-
-- Runtime commands no longer repeat config path defaulting and secret loading.
-- Config policy remains owned by `internal/config`.
-
-## Stage 3: Run Destination Processing Extraction
-
-Goal:
-
-Reduce complexity in the main run loop while preserving run report behavior,
-warning ordering, action ordering, failure aggregation, and text/JSON output.
-
-Implementation scope:
-
-- Extract narrow helpers from `internal/app/run.go` for destination-scoped
-  processing.
-- Centralize destination-scoped failure recording so one helper updates
-  `runFailures`, `runSummary`, `RunReport.Actions`, and pipeline events.
-- Centralize normalization of partial `publish.Plan` identity fields before
-  converting plans to run action records.
-- Keep app orchestration explicit; do not introduce a generic workflow engine,
-  stage framework, or broad runner abstraction.
-- Preserve independent destination fan-out and partial-result behavior.
-
-Current-behavior documentation updates:
-
-- Update `docs/internal/app.md` only if helper names or package layout
-  descriptions materially change.
-
-Tests:
-
-- `go test ./internal/app ./internal/publish ./internal/state`
-- Destination open failures for multiple selected bundles keep action records,
-  output errors, summary counters, and pipeline events aligned.
-- JSON partial-result output remains unchanged when destination planning or
-  execution fails after a report exists.
-- Fixed-path dry-run warnings appear in the same order as before.
-- Existing run text output assertions continue to pass.
-
-Completion criteria:
-
-- `run.go` delegates destination-scoped record/failure bookkeeping to helpers.
-- No public output shape or ordering changes.
-
-## Stage 4: Backend Config Normalized View
-
-Goal:
-
-Reduce source/destination backend config drift while preserving the current YAML
-schema and public config behavior.
-
-Implementation scope:
-
-- Add package-local normalized backend view helpers in `internal/config` for
-  source and destination backend fields.
-- Use the normalized view to reduce duplication in backend defaulting and
-  validation where it remains clearer than the current paired code.
-- Keep `config.Backend` and `config.Destination` YAML structs and tags
-  unchanged.
-- Preserve destination-only policy fields on `Destination`.
-- Preserve `http_upload` as source-only and invalid for destinations.
-- Update app backend opening only if the normalized view provides clearer
-  handoff without leaking config internals.
-
-Current-behavior documentation updates:
-
-- None expected unless internal docs mention the old paired implementation
-  shape in a way that becomes misleading.
-
-Tests:
-
-- `go test ./internal/config ./internal/app`
-- Equivalent local, SSH, and S3 source/destination validation remains
-  consistent.
-- Defaults for SSH port/host key policy, S3 region/prefix/force-path-style, and
-  HTTP upload staging fields remain unchanged.
-- `http_upload` remains valid only for sources.
-
-Completion criteria:
-
-- Adding a future backend field has one obvious defaulting/validation path.
-- Public config files and examples continue to load unchanged.
-
-## Stage 5: Command Output Projection Cleanup
-
-Goal:
-
-Reduce drift in bundle and file metadata projection for app command JSON
-results.
-
-Implementation scope:
-
-- Add small app-local projection helpers for bundle summaries and manifest file
-  records used by `validate`, `inspect`, and `manifest create`.
-- Use `time.RFC3339` consistently instead of equivalent literal layouts.
-- Preserve existing JSON envelope fields, command names, command-specific result
-  field names, text output, and fatal error behavior.
-- Do not redesign CLI JSON output or HTTP JSON responses.
-
-Current-behavior documentation updates:
-
-- None expected unless tests reveal current docs are stale.
-
-Tests:
-
-- `go test ./internal/app ./internal/cli`
-- JSON output for `validate`, `inspect`, and `manifest create` remains
-  structurally stable.
-- RFC3339 timestamps remain unchanged, including offset-preserving source
-  timestamps where current behavior preserves them.
-- Text output remains unchanged.
-
-Completion criteria:
-
-- Bundle/file projection logic is shared where semantics match.
-- Command-specific result structs remain easy to read.
-
-## Stage 6: Archive And Source Path Validation Alignment
-
-Goal:
-
-Protect path safety by aligning archive path tests with source and storage path
-policy, without blurring archive-specific rules.
-
-Implementation scope:
-
-- Add mirrored path-safety table tests around `internal/ingest`, `pkg/bundle`,
-  `internal/bundle`, and `internal/storage` where useful.
-- Keep archive-specific directory handling, root-level manifest rules, duplicate
-  file rejection, symlink rejection, hardlink rejection, and special-entry
-  rejection in `internal/ingest`.
-- Centralize code only if the helper can preserve clear archive semantics and
-  current error behavior.
-- Do not add new public `pkg/bundle` APIs unless the existing public API cannot
-  safely support the needed shared behavior.
-
-Current-behavior documentation updates:
-
-- None expected unless implementation changes error boundaries or internal
-  package descriptions.
-
-Tests:
-
-- `go test ./pkg/bundle ./internal/bundle ./internal/ingest ./internal/storage`
-- Absolute paths, traversal, backslashes, empty paths, dot segments, nested
-  manifests, `.distributor.json` handling, symlinks, hardlinks, devices, and
-  sockets remain covered.
-- Archive directories remain accepted where safe.
-
-Completion criteria:
-
-- Path safety policy has regression coverage across archive staging, source
-  bundle validation, and storage logical path validation.
-- Any code sharing is smaller and clearer than the duplicated logic it replaces.
-
-## Stage 7: Pipeline Run Coordinator Removal
-
-Goal:
-
-Remove the currently unused internal `PipelineRunCoordinator` to avoid
-maintaining two similar coordination concepts.
-
-Implementation scope:
-
-- Delete `PipelineRunCoordinator`, `PipelineRunRecord`,
-  `DuplicatePipelineRunError`, related helpers, and their tests.
-- Remove or rewrite `docs/internal/app.md` sections that describe the removed
-  coordinator.
-- Keep `UploadCoordinator`; do not merge upload queueing with the removed
-  duplicate-run coordinator.
-- Before deletion, confirm with `rg` that production code does not reference
-  `NewPipelineRunCoordinator`, `PipelineRunCoordinator`, or
-  `DuplicatePipelineRunError`.
-
-Current-behavior documentation updates:
-
-- Update `docs/internal/app.md` because it currently documents the coordinator
-  as an internal implemented component.
-
-Tests:
-
-- `go test ./internal/app ./internal/cli`
-- `rg -n "PipelineRunCoordinator|NewPipelineRunCoordinator|DuplicatePipelineRunError" internal docs`
-  should show no stale references after removal.
-
-Completion criteria:
-
-- No production, test, or internal documentation references remain for the
-  removed coordinator.
-- Upload coordination behavior is unchanged.
-
-## Stage 8: Narrow CLI And Test Helper Cleanup
-
-Goal:
-
-Apply only low-risk CLI setup and test fixture cleanup that remains useful after
-the earlier stages.
-
-Implementation scope:
-
-- Add tiny CLI helpers for repeated `flag.FlagSet` setup or output-format
-  parsing only where command behavior remains obvious.
-- Keep the standard-library CLI; do not introduce a CLI framework.
-- Keep `manifest create` interspersed positional parsing local unless another
-  command now needs the same parsing behavior.
-- Expand `internal/testutil` only for repeated setup touched by earlier stages.
-- Do not rewrite tests wholesale just to use shared helpers.
-
-Current-behavior documentation updates:
-
-- None expected unless CLI help or syntax changes. This stage should avoid such
-  changes.
-
-Tests:
-
-- `go test ./internal/cli ./internal/app`
-- CLI usage-error tests remain stable.
-- `manifest create <path> --id x`, `manifest create --id x <path>`, missing
-  flag values, invalid `--format`, and help output remain covered.
-
-Completion criteria:
-
-- Remaining CLI/test cleanup is small, readable, and behavior-preserving.
-- No public CLI syntax or output changes.
-
-## Refactors To Avoid
-
-- Do not introduce a generic workflow engine or stage framework.
-- Do not add a CLI framework.
-- Do not merge local, SSH, S3, and fake backend adapter implementations.
-- Do not collapse `pkg/bundle` and `internal/bundle`.
-- Do not move destination state comparison into `internal/publish` or
-  `internal/app`.
-- Do not redesign CLI JSON envelopes.
-- Do not change HTTP JSON response shapes except where Stage 1 requires
-  pre-acceptance error behavior.
-- Do not add durable upload queues, retry workers, zstd support, in-app TLS,
-  idempotency keys, browser UI, or other feature work.
-- Do not rewrite tests wholesale to use new fixture helpers.
-
-## Validation
-
-After each implementation stage, run the stage-specific tests listed above.
-
-After all cleanup stages:
-
-```sh
-go test ./...
-```
-
-Recommended consistency checks:
-
-```sh
-rg -n "LoadFile\\(|LoadSecretEnvironment\\(|DefaultConfigPath" internal/app internal/cli
-rg -n "application/x-tar|application/gzip|application/x-gzip" internal docs
-rg -n "2006-01-02T15:04:05Z07:00" internal pkg
-rg -n "PipelineRunCoordinator|NewPipelineRunCoordinator|DuplicatePipelineRunError" internal docs
-```
-
-The cleanup is complete when:
-
-- all staged tests and `go test ./...` pass;
-- current-behavior docs describe the implemented Stage 1 upload failure
-  boundary;
-- `docs/roadmap/audit.md` findings have either been addressed or consciously
-  left in place as noted in this cleanup roadmap;
-- no completed cleanup behavior is documented only as future work.

docs/roadmap/documentation.md

@@ -0,0 +1,97 @@
+# Documentation Roadmap
+
+## Purpose
+
+This roadmap tracks the remaining work required to verify that project documentation complies with `docs/policy/documentation.md` and accurately reflects the current implementation.
+
+The documentation migration has rewritten the current user, operator, integration, and internal component docs. This file now records only remaining validation work. Current behavior belongs outside `docs/roadmap/`; deferred or unimplemented work belongs under `docs/roadmap/`.
+
+## Current Documentation Set
+
+Current documentation outside roadmap:
+
+- `README.md`: concise project orientation and quickstart.
+- `docs/cli.md`: canonical CLI command, flag, workflow, and output reference.
+- `docs/config.md`: canonical YAML configuration reference.
+- `docs/operations.md`: operating, safety, state, upload, and recovery guidance.
+- `docs/troubleshooting.md`: symptom-oriented diagnostic and safe-fix guide.
+- `docs/policy/architecture.md`: architecture and invariant policy.
+- `docs/policy/development.md`: contributor and coding workflow policy.
+- `docs/policy/documentation.md`: controlling documentation policy.
+- `docs/integrations/*.md`: implemented external/file-format/protocol contracts.
+- `docs/internal/*.md`: implemented internal component contracts.
+- `examples/*.yml` and `examples/source-bundle/*`: maintained example configs and source bundle fixture.
+
+Current roadmap files:
+
+- `docs/roadmap/documentation.md`: this remaining documentation validation plan.
+- `docs/roadmap/http.md`: deferred HTTP upload extensions only.
+
+Removed completed roadmap artifacts:
+
+- `docs/roadmap/audit.md`
+- `docs/roadmap/cleanup.md`
+- `docs/roadmap/implementation.md`
+
+## Remaining Documentation Validation
+
+Goal: verify the rewritten docs against tests, examples, code, links, and the documentation policy checklist.
+
+Files to create, update, delete, or move: fixes only if validation finds gaps.
+
+Repository areas to inspect:
+
+- `README.md`
+- `docs/cli.md`
+- `docs/config.md`
+- `docs/operations.md`
+- `docs/troubleshooting.md`
+- `docs/internal/`
+- `docs/integrations/`
+- `docs/policy/`
+- `examples/`
+- CLI parser code under `internal/cli`
+- config loading/defaulting/validation under `internal/config`
+- app/report/upload behavior under `internal/app`
+- source bundle, state, publish, storage, adapter, and transform packages
+
+Acceptance criteria:
+
+- Tests pass for the full repository.
+- Maintained example configs load.
+- CLI examples and flags match parser behavior.
+- Config fields and defaults match `internal/config`.
+- Operations and troubleshooting docs describe implemented behavior only.
+- Internal docs preserve package boundaries and policy-required sections.
+- Integration docs describe only implemented contracts.
+- Roadmap files contain only remaining or deferred work.
+- Links resolve.
+- No secrets or private data are present.
+
+Suggested validation commands:
+
+```sh
+go test ./...
+go test ./internal/config ./internal/cli ./internal/app
+go test ./pkg/bundle ./internal/bundle ./internal/state ./internal/publish ./internal/storage ./internal/storage/fake
+go test ./internal/adapters/local ./internal/adapters/ssh ./internal/adapters/s3 ./internal/ingest ./internal/transform/markdown
+rg -n -i "future|planned|deferred|experimental|deprecated|not implemented|old behavior" README.md docs --glob '!docs/roadmap/**' --glob '!docs/policy/**'
+rg -n "\\b(Stages?|Phases?)\\b" README.md docs --glob '!docs/roadmap/**' --glob '!docs/policy/**'
+rg -n -- "--config|--dry-run|--force|--format|--pipeline|--bundle|--id|--file|--created|--overwrite" docs/cli.md internal/cli
+rg -n "examples/" README.md docs examples internal/config/load_test.go
+```
+
+Manual review items:
+
+- Confirm README remains concise and orientation-focused.
+- Confirm `docs/config.md` is the only full config field/default reference.
+- Confirm `docs/cli.md` is the only full command/flag reference.
+- Confirm `docs/operations.md` focuses on operating and recovery.
+- Confirm `docs/troubleshooting.md` remains symptom-first.
+- Confirm `docs/internal/` describes implemented component contracts and boundaries.
+- Confirm integration docs do not claim support for unimplemented external features.
+- Confirm examples contain no secrets and distinguish local runnable examples from environment-gated remote examples.
+
+## Open Questions
+
+No open questions block the remaining validation work.

docs/roadmap/http.md

@@ -8,6 +8,7 @@ The HTTP upload API is implemented. Current behavior is documented in:
 - [Configuration](../config.md)
 - [Operations](../operations.md)
 - [Troubleshooting](../troubleshooting.md)
+- [HTTP upload contract](../integrations/http-upload.md)
 - [Application internals](../internal/app.md)
 - [Ingestion internals](../internal/ingest.md)
 

docs/roadmap/implementation.md

@@ -1,34 +0,0 @@
-# HTTP Upload Deferred Work
-
-## Purpose
-
-HTTP upload behavior is implemented and documented in the current-behavior
-manuals:
-
-- [CLI](../cli.md)
-- [Configuration](../config.md)
-- [Operations](../operations.md)
-- [Troubleshooting](../troubleshooting.md)
-- [Application internals](../internal/app.md)
-- [Configuration internals](../internal/config.md)
-- [Ingestion internals](../internal/ingest.md)
-
-This file tracks only HTTP upload work that is not implemented.
-
-## Deferred Work
-
-- URL-token authentication.
-- Zstandard-compressed archive support.
-- Durable status persistence across process restarts.
-- Database-backed queueing.
-- Producer-supplied idempotency keys.
-- Run listing, cancellation, and retry endpoints.
-- In-app TLS.
-- Public network exposure defaults.
-- Browser UI.
-
-## Documentation Rule
-
-Deferred behavior belongs under `docs/roadmap/` until implemented. Current
-behavior docs must describe only the active HTTP upload API, configuration,
-operation, troubleshooting, and internal package contracts.

docs/troubleshooting.md

@@ -1,8 +1,14 @@
 # Distributor Troubleshooting
 
-## `load config ... no such file or directory`
+Audience: administrators and operators diagnosing `distributor` command, configuration, publishing, storage, or HTTP upload failures.
 
-Likely cause: `run` could not find the config path. If `--config` is omitted, the default path is `/usr/local/etc/distributor/config.yml`.
+Each entry lists the symptom, likely cause, diagnostic step, safe fix, and relevant reference link. Command syntax lives in [CLI](cli.md), configuration fields live in [Configuration](config.md), and operating procedures live in [Operations](operations.md).
+
+## Config File Is Missing
+
+Symptom: `load config ... no such file or directory`.
+
+Likely cause: `--config` points to a missing file, or `--config` was omitted and `/usr/local/etc/distributor/config.yml` is not installed.
 
 Diagnostic:
 
@@ -10,11 +16,15 @@ Diagnostic:
 ls -l <config-path>
 ```
 
-Safe fix: pass an existing config path with `--config`, or install a config at the default path. See [configuration](config.md).
+Safe fix: pass an existing file with `--config`, or install a config at the default path.
 
-## `parse config ... field not found`
+Reference: [Configuration](config.md#config-file-loading).
 
-Likely cause: the YAML contains an unknown field. Config loading rejects unknown keys.
+## Config Contains An Unknown Field
+
+Symptom: `parse config ... field not found`.
+
+Likely cause: the YAML contains a key that is not part of the implemented config schema.
 
 Diagnostic:
 
@@ -22,14 +32,15 @@ Diagnostic:
 go run ./cmd/distributor run --config <config-path> --dry-run
 ```
 
-Safe fix: compare the file to the reference in [configuration](config.md) and remove or rename unsupported fields.
+Safe fix: remove or rename unsupported fields using the canonical config reference.
 
-## `validate config ... backend ... is unsupported`
+Reference: [Configuration](config.md).
 
-Likely cause: a source or destination uses an unsupported backend name, or a
+## Backend Name Or Placement Is Invalid
-command is trying to execute a backend that is valid only for another workflow.
+
-`run`, `validate`, and `inspect` execute `local`, `ssh`, and `s3` sources.
+Symptom: `backend ... is unsupported` or `http_upload is only supported for sources`.
-`serve` executes `http_upload` sources.
+
+Likely cause: a backend name is misspelled, not executable, or configured in the wrong role.
 
 Diagnostic:
 
@@ -37,13 +48,343 @@ Diagnostic:
 rg -n "backend:" <config-path>
 ```
 
-Safe fix: use `backend: local`, `backend: ssh`, or `backend: s3` for normal
+Safe fix: use `local`, `ssh`, or `s3` for executable sources and destinations. Use `http_upload` only as a source served by `distributor serve`.
-source and destination workflows. Use `backend: http_upload` only for sources
-handled by `distributor serve`.
 
-## `bind HTTP server ... address already in use`
+Reference: [Configuration](config.md#backend-reference).
 
-Likely cause: another process is already listening on `server.http.bind`.
+## CLI Arguments Select The Wrong Source Mode
+
+Symptom: `configured source mode requires --pipeline`, `does not accept a local path with --config, --pipeline, or --bundle`, `validate command requires a path`, or `inspect command requires a path`.
+
+Likely cause: `validate` or `inspect` mixed local path mode with configured source mode, or omitted the required source selector.
+
+Diagnostic:
+
+```sh
+go run ./cmd/distributor validate --help
+go run ./cmd/distributor inspect --help
+```
+
+Safe fix: use either `distributor validate <path>` / `distributor inspect <path>`, or use `--config <path> --pipeline <id>` with optional `--bundle <path>`.
+
+Reference: [CLI](cli.md#validate).
+
+## Output Format Is Invalid
+
+Symptom: `format must be text or json`.
+
+Likely cause: an unsupported value was passed to `--format`.
+
+Diagnostic:
+
+```sh
+go run ./cmd/distributor run --help
+```
+
+Safe fix: use `--format text` or `--format json`.
+
+Reference: [CLI](cli.md#common-output-format).
+
+## JSON Mode Wrote No JSON Document
+
+Symptom: `--format json` exits non-zero and stdout has no JSON result.
+
+Likely cause: the command failed before it could construct a result, such as invalid arguments, missing config, unreadable secrets, or source setup failure.
+
+Diagnostic:
+
+```sh
+go run ./cmd/distributor run --config <config-path> --format json
+```
+
+Safe fix: read stderr, fix the setup problem, then rerun. Partial destination failures during `run` can produce JSON; fatal setup failures do not.
+
+Reference: [CLI](cli.md#output-and-exit-behavior).
+
+## Source Pipeline Is Not Found
+
+Symptom: `pipeline "<id>" not found`.
+
+Likely cause: configured source diagnostics or upload processing selected a pipeline id that is absent from the loaded config.
+
+Diagnostic:
+
+```sh
+rg -n "id:" <config-path>
+```
+
+Safe fix: pass an existing `--pipeline` value or correct the pipeline id in config.
+
+Reference: [Configuration](config.md#pipelines).
+
+## Source Bundles Are Not Found
+
+Symptom: `no bundles found`, `no bundles found under "."`, `stat ssh ... not_found`, or `stat s3 ... not_found`.
+
+Likely cause: the source root, source-root-relative bundle path, S3 prefix, SSH path, or permissions do not expose a directory containing `manifest.json`.
+
+Diagnostic:
+
+```sh
+go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
+```
+
+Safe fix: correct the configured source backend root, permissions, prefix, or `--bundle` path. The selected bundle directory must contain `manifest.json`.
+
+Reference: [Operations](operations.md#filesystem-and-storage-layout).
+
+## Source Manifest Or Files Fail Validation
+
+Symptom: `sha256 mismatch`, `size mismatch`, `digest mismatch`, missing manifest fields, or unsafe source paths.
+
+Likely cause: files changed after `manifest.json` was written, the manifest digest is stale, or the producer wrote invalid bundle paths.
+
+Diagnostic:
+
+```sh
+go run ./cmd/distributor validate <source-root>
+```
+
+Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source validation failures.
+
+Reference: [Operations](operations.md#cleanup-and-recovery).
+
+## Destination Has Unmanaged Content
+
+Symptom: `destination has content but no distributor state` or a plan reason containing `fail_unmanaged`.
+
+Likely cause: the destination bundle path contains files but no valid `.distributor.json`, so `distributor` will not claim it by default.
+
+Diagnostic:
+
+```sh
+find <destination-path> -maxdepth 2 -print
+```
+
+Safe fix: choose an empty destination path, move unrelated files aside, or preview `run --dry-run --force` only after confirming the reported bundle path is safe to replace.
+
+Reference: [Operations](operations.md#forced-replacement-workflow).
+
+## Destination State Conflicts With Source
+
+Symptom: `fail_conflict`, `destination source id differs`, `same id and created time but different digest`, `pipeline id ... does not match`, or `destination id ... does not match`.
+
+Likely cause: `.distributor.json` belongs to a different pipeline, destination, source id, or same-created source with different content.
+
+Diagnostic:
+
+```sh
+cat <destination-path>/.distributor.json
+go run ./cmd/distributor inspect <source-root>
+```
+
+Safe fix: verify the source and destination are intended to match. Use a separate destination path for unrelated content. To replace the existing state, configure `transfer.on_conflict: replace`, preview with `--dry-run --force`, then publish with `--force`.
+
+Reference: [Operations](operations.md#destination-state-and-retry-behavior).
+
+## Destination Is Newer Than Source
+
+Symptom: `skip_destination_newer` or `destination is newer and replacement requires --force`.
+
+Likely cause: the destination state records a source manifest with a later `created` timestamp than the current source.
+
+Diagnostic:
+
+```sh
+go run ./cmd/distributor run --config <config-path> --dry-run --format json
+```
+
+Safe fix: keep the default skip behavior unless replacement is intentional. To replace newer state, configure `transfer.on_destination_newer: replace`, preview with `--dry-run --force`, then publish with `--force`.
+
+Reference: [Operations](operations.md#forced-replacement-workflow).
+
+## Forced Replacement Appears In A Plan
+
+Symptom: dry-run output includes `force_replace`.
+
+Likely cause: the run used `--force`, and planning selected a supported destructive replacement.
+
+Diagnostic:
+
+```sh
+go run ./cmd/distributor run --config <config-path> --dry-run --force
+```
+
+Safe fix: inspect the pipeline id, destination id, backend, and bundle path. Proceed only if deleting everything inside that destination bundle path is intended.
+
+Reference: [Operations](operations.md#forced-replacement-workflow).
+
+## Output Path Collision
+
+Symptom: `destination output path collision`.
+
+Likely cause: publication would write two outputs to the same destination path, such as copying `report.html` while also generating `report.html` from `report.md`.
+
+Diagnostic:
+
+```sh
+go run ./cmd/distributor run --config <config-path> --dry-run
+```
+
+Safe fix: adjust source files or publish/transform policy so copied and generated outputs do not collide.
+
+Reference: [Configuration](config.md#publish-and-transform-policy).
+
+## Run Failed After Writing Some Files
+
+Symptom: a destination write failed and the command exited non-zero after partial work.
+
+Likely cause: storage write failure, permission issue, network interruption, or object-store error during publish execution.
+
+Diagnostic:
+
+```sh
+find <destination-path> -maxdepth 2 -print
+```
+
+Safe fix: inspect the destination bundle path printed in the error. `distributor` attempts to remove outputs from the failed attempt, but operators should verify the destination before retrying. Rerun `--dry-run` before publishing again.
+
+Reference: [Operations](operations.md#destination-state-and-retry-behavior).
+
+## JSON Run Result Has `ok: false`
+
+Symptom: `run --format json` exits non-zero with a JSON result where `ok` is `false`.
+
+Likely cause: at least one destination failed after planning or execution began, while other destination results were still available.
+
+Diagnostic:
+
+```sh
+go run ./cmd/distributor run --config <config-path> --format json
+```
+
+Safe fix: inspect the top-level `errors` array, destination actions, output errors, and summary. Fix failed destinations, then preview with `--dry-run --format json`.
+
+Reference: [CLI](cli.md#output-and-exit-behavior).
+
+## Secrets Directory Is Missing Or Unreadable
+
+Symptom: `load secrets directory ... no such file or directory`, `permission denied`, or `secret filename ... is invalid`.
+
+Likely cause: `secrets.directory` points to a missing or unreadable directory, or it contains a regular file whose name is not a valid credential variable name.
+
+Diagnostic:
+
+```sh
+ls -ld <secrets-directory>
+find <secrets-directory> -maxdepth 1 -type f -printf '%f\n'
+```
+
+Safe fix: mount or create the directory, adjust permissions for the service user, or rename/remove invalid secret files. Secret filenames must match `[A-Za-z_][A-Za-z0-9_]*`.
+
+Reference: [Configuration](config.md#secrets).
+
+## Credential Variable Is Missing Or Empty
+
+Symptom: `credential environment variable ... is not set`, `credential environment variable ... is empty`, or S3 authentication errors such as `AccessDenied`, `InvalidAccessKeyId`, or `SignatureDoesNotMatch`.
+
+Likely cause: configured S3 credential variable names are not available through the process environment or `secrets.directory`, are empty, or do not authorize the requested bucket/prefix.
+
+Diagnostic:
+
+```sh
+env | cut -d= -f1 | rg '^(<access-key-variable>|<secret-key-variable>)$'
+ls -l <secrets-directory>
+```
+
+Safe fix: provide both configured S3 credential values, correct IAM/service permissions, or omit explicit credential fields to use the AWS SDK default credential chain.
+
+Reference: [Configuration](config.md#s3-compatible-backend).
+
+## Secret File Is Ignored In Favor Of Environment
+
+Symptom: `secret ... ignored because the real environment already has that variable`.
+
+Likely cause: the same credential name exists in the process environment and `secrets.directory` with different values.
+
+Diagnostic:
+
+```sh
+env | cut -d= -f1 | rg '^<variable-name>$'
+ls -l <secrets-directory>/<variable-name>
+```
+
+Safe fix: remove one source of the credential or intentionally keep the process environment value. `distributor` does not print either value.
+
+Reference: [Operations](operations.md#secrets-operation).
+
+## SSH Auth Is Not Configured
+
+Symptom: `no SSH auth methods configured`.
+
+Likely cause: no SSH agent is available and `ssh_key_file` is missing or unreadable.
+
+Diagnostic:
+
+```sh
+test -n "$SSH_AUTH_SOCK" && ssh-add -l
+ls -l <ssh-key-file>
+```
+
+Safe fix: start an SSH agent with an appropriate key loaded, or configure a readable private key with `ssh_key_file`.
+
+Reference: [Configuration](config.md#sshsftp-backend).
+
+## SSH Host Key Fails
+
+Symptom: `host key ... is unknown`, `known_hosts is required`, or `host key ... has changed`.
+
+Likely cause: strict host key checking has no trusted key, `accept-new` cannot persist a new key, or the remote host key differs from the stored key.
+
+Diagnostic:
+
+```sh
+ls -l <known-hosts-path>
+ssh-keygen -F <host> -f <known-hosts-path>
+```
+
+Safe fix: pre-populate `known_hosts` for `strict`, configure a writable `known_hosts` path for `accept-new`, or verify the server identity before updating a changed key. Do not disable host key checking to bypass an unexpected changed key.
+
+Reference: [Operations](operations.md#sshsftp).
+
+## S3 Prefix Is Invalid
+
+Symptom: `prefix must be a clean relative slash-separated path`.
+
+Likely cause: the S3 prefix contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed.
+
+Diagnostic:
+
+```sh
+go run ./cmd/distributor run --config <config-path> --dry-run
+```
+
+Safe fix: use a clean relative prefix such as `reports/archive`, or omit `prefix`.
+
+Reference: [Configuration](config.md#s3-compatible-backend).
+
+## S3 Location Or Connectivity Fails
+
+Symptom: `NoSuchBucket`, `InvalidBucketName`, `not_found`, endpoint connection failures, or TLS/network errors.
+
+Likely cause: endpoint, bucket, prefix, region, path-style mode, network routing, or credentials are wrong for the service.
+
+Diagnostic:
+
+```sh
+go run ./cmd/distributor run --config <config-path> --dry-run
+curl -I <endpoint>
+```
+
+Safe fix: verify `endpoint`, `bucket`, `region`, `prefix`, and `force_path_style`. For S3-compatible services, keep `force_path_style: true` unless the service requires virtual-host addressing. Distributor does not provide an insecure TLS bypass setting.
+
+Reference: [Operations](operations.md#s3-compatible-storage).
+
+## HTTP Server Cannot Bind
+
+Symptom: `bind HTTP server ... address already in use`.
+
+Likely cause: another process is listening on `server.http.bind`.
 
 Diagnostic:
 
@@ -51,44 +392,33 @@ Diagnostic:
 ss -ltnp | rg '<port>'
 ```
 
-Safe fix: stop the conflicting process or configure a different
+Safe fix: stop the conflicting process or configure a different bind address.
-`server.http.bind` value. The default bind address is `127.0.0.1:8080`.
 
-## `upload token environment variable ... is not set`
+Reference: [Configuration](config.md#serverhttp).
 
-Likely cause: a configured `http_upload` source references `token_env`, but the
+## HTTP Upload Token Is Missing Or Duplicated
-variable is absent from both the real process environment and
-`secrets.directory`.
 
-Diagnostic:
+Symptom: `upload token environment variable ... is not set`, `... is empty`, or `upload token environment variables ... resolve to the same value`.
 
-```sh
+Likely cause: an `http_upload` source references a missing/empty `token_env`, or two upload pipelines resolve to the same bearer token.
-env | cut -d= -f1 | rg '^<token-variable>$'
-ls -l <secrets-directory>/<token-variable>
-```
-
-Safe fix: set the real environment variable or create a readable
-secrets-directory file with the same name. Do not place literal token values in
-YAML.
-
-## `upload token environment variables ... resolve to the same value`
-
-Likely cause: two configured `http_upload` pipelines resolve to the same bearer
-token value.
 
 Diagnostic:
 
 ```sh
 rg -n 'token_env:' <config-path>
+env | cut -d= -f1 | rg '^<token-variable>$'
+ls -l <secrets-directory>/<token-variable>
 ```
 
-Safe fix: assign a distinct non-empty token value to each `http_upload`
+Safe fix: provide one distinct non-empty token value per upload pipeline through the process environment or `secrets.directory`. Do not put literal tokens in YAML.
-pipeline. Distributor does not print the duplicate token value.
 
-## `POST /upload` returns `401`
+Reference: [Configuration](config.md#http-upload-source-backend).
 
-Likely cause: the request is missing `Authorization: Bearer <token>` or the
+## Upload Request Is Unauthorized
-token does not match any configured `http_upload` pipeline.
+
+Symptom: `POST /upload` returns `401`.
+
+Likely cause: the request lacks `Authorization: Bearer <token>`, has an empty token, or uses a token that does not match any configured upload pipeline.
 
 Diagnostic:
 
@@ -99,55 +429,33 @@ curl -i -X POST http://127.0.0.1:8080/upload \
   --data-binary @bundle.tar
 ```
 
-Safe fix: use the token value resolved by the configured `token_env`. Do not
+Safe fix: use the token value resolved by the configured `token_env`. Do not include token values in logs or tickets.
-include token values in logs or tickets.
 
-## `POST /upload` returns `413`
+Reference: [Operations](operations.md#http-upload-operation).
 
-Likely cause: the request body exceeds the selected pipeline's
+## Upload Request Is Rejected Before A Run ID
-`source.max_upload_size` or the default `server.http.max_upload_size`.
+
+Symptom: `POST /upload` returns `400`, `413`, `415`, or `503`.
+
+Likely cause: the request included a `pipeline` or `pipeline_id` query, archive content is malformed, the body exceeds size limits, content type is unsupported, or the in-memory upload queue is full.
 
 Diagnostic:
 
 ```sh
-ls -lh bundle.tar bundle.tar.gz
+tar -tf bundle.tar
-rg -n 'max_upload_size:' <config-path>
+tar -tzf bundle.tar.gz
+rg -n 'max_upload_size|queue_size|max_concurrency' <config-path>
 ```
 
-Safe fix: upload a smaller archive, remove unnecessary files from the source
+Safe fix: send one valid tar or tar.gz source bundle archive with `Content-Type: application/x-tar`, `application/gzip`, or `application/x-gzip`; remove pipeline query parameters; reduce archive size or raise the configured limit; retry after queue pressure drops.
-bundle, or raise the configured upload size limit.
 
-## `POST /upload` returns `415`
+Reference: [Operations](operations.md#http-upload-operation).
 
-Likely cause: the upload uses an unsupported content type. The server accepts
+## Upload Status Is Missing
-uncompressed tar and gzip-compressed tar archives only.
 
-Diagnostic:
+Symptom: `GET /runs/<run_id>` returns `404`.
 
-```sh
+Likely cause: the run id is wrong, the process restarted, or the retained status record expired after `server.http.retention`.
-file bundle.tar.gz
-```
-
-Safe fix: send `Content-Type: application/x-tar`, `application/gzip`, or
-`application/x-gzip`, matching the archive format.
-
-## `POST /upload` returns `503`
-
-Likely cause: the in-memory upload queue is full.
-
-Diagnostic:
-
-```sh
-rg -n 'queue_size|max_concurrency' <config-path>
-```
-
-Safe fix: retry after active uploads finish, or increase `server.http.queue_size`
-for the deployment.
-
-## `GET /runs/<run_id>` returns `404`
-
-Likely cause: the run id is wrong, the process restarted, or the completed
-status record expired after `server.http.retention`.
 
 Diagnostic:
 
@@ -156,362 +464,6 @@ curl -i http://127.0.0.1:8080/runs/<run-id>
 rg -n 'retention:' <config-path>
 ```
 
-Safe fix: use the exact `run_id` returned by `POST /upload`. If status retention
+Safe fix: use the exact `run_id` returned by upload admission. Increase retention if operators need a longer status window.
-is too short for operators, increase `server.http.retention`.
 
-## `--format: format must be text or json`
+Reference: [Operations](operations.md#http-upload-operation).
-
-Likely cause: a command was run with an unsupported output format.
-
-Diagnostic:
-
-```sh
-go run ./cmd/distributor run --help
-```
-
-Safe fix: use `--format text` or `--format json`. Help and usage output are always text.
-
-## `--format json` wrote no JSON output
-
-Likely cause: the command failed before it could construct a result, such as a missing config file, invalid arguments, unreadable secrets directory, or source setup failure.
-
-Diagnostic:
-
-```sh
-go run ./cmd/distributor run --config <config-path> --format json
-```
-
-Safe fix: read the stderr error and fix the setup problem. JSON mode writes a document only after the command has enough information to construct a result.
-
-## `configured source mode requires --pipeline`
-
-Likely cause: `validate` or `inspect` was run with `--config` but without an explicit pipeline id.
-
-Diagnostic:
-
-```sh
-go run ./cmd/distributor validate --help
-go run ./cmd/distributor inspect --help
-```
-
-Safe fix: add `--pipeline <pipeline-id>`. Configured source diagnostics require an explicit pipeline even when the config contains one pipeline.
-
-## `does not accept a local path with --config, --pipeline, or --bundle`
-
-Likely cause: local-path mode and configured source mode were mixed in one `validate` or `inspect` command.
-
-Diagnostic:
-
-```sh
-go run ./cmd/distributor inspect --help
-```
-
-Safe fix: use either `distributor inspect <local-path>` or `distributor inspect --config <path> --pipeline <id>`, not both.
-
-## `--format json` exited non-zero with `ok: false`
-
-Likely cause: `run` began planning or executing destinations, and at least one destination failed while other destination results were still available.
-
-Diagnostic:
-
-```sh
-go run ./cmd/distributor run --config <config-path> --format json
-```
-
-Safe fix: inspect the top-level `errors` array, `result.actions`, and `result.summary`. Fix the failed destination, then preview with `--dry-run --format json` before retrying.
-
-## `prefix must be a clean relative slash-separated path`
-
-Likely cause: S3 `prefix` contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed.
-
-Diagnostic:
-
-```sh
-go run ./cmd/distributor run --config <config-path> --dry-run
-```
-
-Safe fix: use a clean relative prefix such as `reports/archive`, or omit `prefix`.
-
-## `NoSuchBucket`, `InvalidBucketName`, or `not_found`
-
-Likely cause: the S3 bucket, endpoint, or prefix is wrong, or the configured credentials cannot see the requested object.
-
-Diagnostic:
-
-```sh
-go run ./cmd/distributor run --config <config-path> --dry-run
-```
-
-Safe fix: verify `endpoint`, `bucket`, `region`, `force_path_style`, and `prefix`. For S3-compatible services, keep `force_path_style: true` unless the service requires virtual-host addressing.
-
-## `AccessDenied`, `InvalidAccessKeyId`, or `SignatureDoesNotMatch`
-
-Likely cause: S3 credentials are missing, wrong, empty, or lack permission for the bucket or prefix.
-
-Diagnostic:
-
-```sh
-env | cut -d= -f1 | rg '^(<access-key-variable>|<secret-key-variable>)$'
-ls -l <secrets-directory>
-```
-
-Safe fix: provide both configured credential environment variables through the real environment or `secrets.directory`, or omit explicit credential fields to use the AWS SDK default credential chain.
-
-## S3 endpoint connection failures
-
-Likely cause: the endpoint URL is unreachable, uses the wrong scheme, or does not match the configured path-style mode.
-
-Diagnostic:
-
-```sh
-curl -I <endpoint>
-```
-
-Safe fix: correct `endpoint`, network routing, TLS settings outside distributor, or `force_path_style`. Distributor does not provide insecure TLS bypass configuration.
-
-## `load secrets directory ... no such file or directory`
-
-Likely cause: `secrets.directory` points to a missing directory.
-
-Diagnostic:
-
-```sh
-ls -ld <secrets-directory>
-```
-
-Safe fix: create or mount the directory before running, or remove `secrets.directory` if no credential files are needed.
-
-## `load secrets directory ... permission denied`
-
-Likely cause: the service user cannot read the configured secrets directory.
-
-Diagnostic:
-
-```sh
-ls -ld <secrets-directory>
-namei -l <secrets-directory>
-```
-
-Safe fix: adjust the directory path or deployment permissions so the service user can read the directory. Distributor does not enforce owner, group, or mode policy beyond OS read access.
-
-## `secret filename ... is invalid`
-
-Likely cause: a regular file in `secrets.directory` does not match `[A-Za-z_][A-Za-z0-9_]*`.
-
-Diagnostic:
-
-```sh
-find <secrets-directory> -maxdepth 1 -type f -printf '%f\n'
-```
-
-Safe fix: rename the file to a valid credential environment variable name, or remove it from the secrets directory.
-
-## `credential environment variable ... is not set`
-
-Likely cause: a backend credential field references an environment variable that is absent from both the real process environment and the configured secrets directory.
-
-Diagnostic:
-
-```sh
-env | cut -d= -f1 | rg '^<variable-name>$'
-ls -l <secrets-directory>/<variable-name>
-```
-
-Safe fix: set the real environment variable or create a readable secrets-directory file with the same name.
-
-## `secret ... ignored because the real environment already has that variable`
-
-Likely cause: the real process environment and secrets directory both define the variable with different values.
-
-Diagnostic:
-
-```sh
-env | cut -d= -f1 | rg '^<variable-name>$'
-ls -l <secrets-directory>/<variable-name>
-```
-
-Safe fix: remove one source of the credential or make the deployment intentionally prefer the real environment value. Distributor does not print either value.
-
-## `host is required for ssh backend`
-
-Likely cause: SSH config is missing the structured `host` field, or an old URL-style SSH config is still in use.
-
-Diagnostic:
-
-```sh
-go run ./cmd/distributor run --config <config-path> --dry-run
-```
-
-Safe fix: configure SSH with `host`, optional `user` and `port`, and `path`. SSH URLs are not part of the active config schema.
-
-## `no SSH auth methods configured`
-
-Likely cause: neither an SSH agent nor `ssh_key_file` is available.
-
-Diagnostic:
-
-```sh
-test -n "$SSH_AUTH_SOCK" && ssh-add -l
-ls -l <ssh-key-file>
-```
-
-Safe fix: start an SSH agent with an appropriate key loaded, or configure `ssh_key_file` with a readable private key.
-
-## `host key ... is unknown` or `known_hosts is required`
-
-Likely cause: strict host key checking has no known host key, or `accept-new` cannot persist a new key.
-
-Diagnostic:
-
-```sh
-ls -l <known-hosts-path>
-ssh-keygen -F <host> -f <known-hosts-path>
-```
-
-Safe fix: configure a writable `known_hosts` path for `accept-new`, pre-populate `known_hosts` for `strict`, or explicitly use `host_key_policy: off` only for insecure test environments.
-
-## `host key ... has changed`
-
-Likely cause: the remote server presented a different host key than the one recorded in `known_hosts`.
-
-Diagnostic:
-
-```sh
-ssh-keygen -F <host> -f <known-hosts-path>
-```
-
-Safe fix: verify the server identity out of band before updating `known_hosts`. Do not switch to `host_key_policy: off` to bypass an unexpected changed key.
-
-## `pipeline "<id>" not found`
-
-Likely cause: configured source validation or inspection requested a pipeline id that is not present in the config file.
-
-Diagnostic:
-
-```sh
-rg -n "id:" <config-path>
-```
-
-Safe fix: pass an existing pipeline id with `--pipeline`, or update the config.
-
-## `stat ssh ... not_found`, `stat s3 ... not_found`, or `no bundles found`
-
-Likely cause: the configured source root is wrong, unreadable, or does not contain source bundles.
-
-Diagnostic:
-
-```sh
-go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
-```
-
-Safe fix: correct the configured source root, S3 prefix, permissions, or source bundle location. Use `--bundle <path>` only with a source-root-relative bundle directory that contains `manifest.json`.
-
-## `validate command requires a path` or `inspect command requires a path`
-
-Likely cause: `validate` or `inspect` was run without a local path and without configured source mode.
-
-Diagnostic:
-
-```sh
-go run ./cmd/distributor validate --help
-go run ./cmd/distributor inspect --help
-```
-
-Safe fix: pass a local source bundle directory or local tree, or pass both `--config <path>` and `--pipeline <id>`.
-
-## `no bundles found under "."`
-
-Likely cause: the selected source root does not contain a `manifest.json` source bundle.
-
-Diagnostic:
-
-```sh
-find <source-root> -name manifest.json -print
-```
-
-Safe fix: point the command or config at the directory containing the source bundle, or write a valid `manifest.json` and listed files. See [CLI](cli.md).
-
-## `sha256 mismatch`, `size mismatch`, or `digest mismatch`
-
-Likely cause: a listed source file changed after `manifest.json` was created, or the manifest digest does not match its file list.
-
-Diagnostic:
-
-```sh
-go run ./cmd/distributor validate <source-root>
-```
-
-Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source digest failures.
-
-## `destination has content but no distributor state`
-
-Likely cause: the destination path is not empty and has no `.distributor.json` state file, so `distributor` will not claim it as managed.
-
-Diagnostic:
-
-```sh
-find <destination-path> -maxdepth 2 -print
-```
-
-Safe fix: choose an empty destination path or move existing files aside after confirming they are not needed. If the destination should be claimed by distributor, preview with `run --dry-run --force` and publish with `run --force` only after confirming the reported `force_replace` action is bounded to the intended bundle path.
-
-## `fail_conflict`
-
-Likely cause: existing `.distributor.json` belongs to a different pipeline, a different destination, a different source id, or a same-created source with a different digest.
-
-Diagnostic:
-
-```sh
-cat <destination-path>/.distributor.json
-go run ./cmd/distributor inspect <source-root>
-```
-
-Safe fix: verify you are publishing the intended source to the intended destination. Use a separate destination path for unrelated content. If the existing state should be replaced, configure `transfer.on_conflict: replace`, preview with `run --dry-run --force`, then publish with `run --force`.
-
-## `destination is newer and replacement requires --force`
-
-Likely cause: config explicitly allows newer-destination replacement, but the current run did not include `--force`.
-
-Diagnostic:
-
-```sh
-go run ./cmd/distributor run --config <config-path> --dry-run --force
-```
-
-Safe fix: prefer the default `transfer.on_destination_newer: skip` unless replacing newer destination state is intentional. To replace it, keep `transfer.on_destination_newer: replace`, confirm the dry-run output shows `force_replace`, then run with `--force`.
-
-## `force_replace`
-
-Likely cause: the current run used `--force` and publish planning selected a supported destructive replacement.
-
-Diagnostic:
-
-```sh
-go run ./cmd/distributor run --config <config-path> --dry-run --force
-```
-
-Safe fix: inspect the printed pipeline id, destination id, backend, and bundle path. Proceed only if deleting all content within that destination bundle path is intended.
-
-## `destination output path collision`
-
-Likely cause: configured publication would write two outputs to the same destination path, such as publishing a source `report.html` while also generating `report.html` from `report.md`.
-
-Diagnostic:
-
-```sh
-go run ./cmd/distributor run --config <config-path> --dry-run
-```
-
-Safe fix: adjust the source bundle contents or publish policy so source and generated outputs do not collide.
-
-## A run failed after writing some files
-
-Likely cause: a write failed partway through publication. Local, SSH, and S3 execution attempt to clean up outputs written during the failed attempt.
-
-Diagnostic:
-
-```sh
-find <destination-path> -maxdepth 2 -print
-```
-
-Safe fix: use the pipeline id, destination id, backend, and bundle path printed in the run error to inspect the destination before retrying. If only unrelated unmanaged files remain, move them aside or choose a clean destination. Re-run with `--dry-run` before publishing again. See [operations](operations.md).

internal/app/inspect.go

@@ -65,55 +65,19 @@ func writeInspectResult(options InspectOptions, selection sourceSelection) error
 }
 
 type inspectResult struct {
-	PipelineID    string                `json:"pipeline_id,omitempty"`
+	PipelineID    string               `json:"pipeline_id,omitempty"`
-	SourceBackend string                `json:"source_backend,omitempty"`
+	SourceBackend string               `json:"source_backend,omitempty"`
-	BundleCount   int                   `json:"bundle_count"`
+	BundleCount   int                  `json:"bundle_count"`
-	Bundles       []inspectBundleResult `json:"bundles"`
+	Bundles       []bundleDetailResult `json:"bundles"`
-}
-
-type inspectBundleResult struct {
-	Path      string              `json:"path"`
-	ID        string              `json:"id"`
-	Created   string              `json:"created"`
-	Digest    string              `json:"digest"`
-	FileCount int                 `json:"file_count"`
-	TotalSize int64               `json:"total_size"`
-	Files     []inspectFileResult `json:"files"`
-}
-
-type inspectFileResult struct {
-	Path   string `json:"path"`
-	SHA256 string `json:"sha256"`
-	Size   int64  `json:"size"`
 }
 
 func inspectResultFromSelection(selection sourceSelection) inspectResult {
-	result := inspectResult{
+	return inspectResult{
 		PipelineID:    selection.PipelineID,
 		SourceBackend: selection.SourceBackend,
 		BundleCount:   len(selection.Bundles),
-		Bundles:       make([]inspectBundleResult, 0, len(selection.Bundles)),
+		Bundles:       bundleDetailsFromBundles(selection.Bundles),
 	}
-	for _, sourceBundle := range selection.Bundles {
-		bundleResult := inspectBundleResult{
-			Path:      storage.DisplayPath(sourceBundle.RootRelativePath),
-			ID:        sourceBundle.Manifest.ID,
-			Created:   sourceBundle.Manifest.Created.Format("2006-01-02T15:04:05Z07:00"),
-			Digest:    sourceBundle.Manifest.Digest,
-			FileCount: len(sourceBundle.Manifest.Files),
-			Files:     make([]inspectFileResult, 0, len(sourceBundle.Manifest.Files)),
-		}
-		for _, file := range sourceBundle.Manifest.Files {
-			bundleResult.TotalSize += file.Size
-			bundleResult.Files = append(bundleResult.Files, inspectFileResult{
-				Path:   file.Path,
-				SHA256: file.SHA256,
-				Size:   file.Size,
-			})
-		}
-		result.Bundles = append(result.Bundles, bundleResult)
-	}
-	return result
 }
 
 func writeInspection(w io.Writer, selection sourceSelection) error {
@@ -134,7 +98,7 @@ func writeInspection(w io.Writer, selection sourceSelection) error {
 			"- path=%s id=%s created=%s digest=%s files=%d\n",
 			storage.DisplayPath(sourceBundle.RootRelativePath),
 			sourceBundle.Manifest.ID,
-			sourceBundle.Manifest.Created.Format("2006-01-02T15:04:05Z07:00"),
+			formatManifestCreated(sourceBundle.Manifest.Created),
 			sourceBundle.Manifest.Digest,
 			len(sourceBundle.Manifest.Files),
 		); err != nil {

internal/app/inspect_test.go

@@ -3,9 +3,12 @@ package app
 import (
 	"bytes"
 	"context"
+	"encoding/json"
+	"os"
 	"path/filepath"
 	"strings"
 	"testing"
+	"time"
 
 	"gitea.maximumdirect.net/eric/distributor/internal/testutil"
 )
@@ -85,6 +88,98 @@ func TestInspectConfiguredSourceJSON(t *testing.T) {
 	}
 }
 
+func TestInspectJSONPreservesCreatedOffsetAndFileMetadata(t *testing.T) {
+	sourceRoot := t.TempDir()
+	created := time.Date(2026, 6, 1, 6, 30, 0, 0, time.FixedZone("CDT", -5*60*60))
+	testutil.WriteSourceBundle(t, sourceRoot, "daily", testutil.BundleOptions{
+		ID:      "reports.offset",
+		Created: created,
+		Files: []testutil.SourceFile{
+			{Path: "report.md", Data: "# Report\n"},
+		},
+	})
+	var stdout bytes.Buffer
+
+	err := Inspect(context.Background(), InspectOptions{
+		Path:         sourceRoot,
+		Stdout:       &stdout,
+		OutputFormat: OutputFormatJSON,
+	})
+
+	if err != nil {
+		t.Fatalf("Inspect() error = %v", err)
+	}
+	result := decodeAppResult(t, stdout.String())
+	bundles, ok := result["bundles"].([]any)
+	if !ok || len(bundles) != 1 {
+		t.Fatalf("bundles = %#v, want one bundle", result["bundles"])
+	}
+	bundle, ok := bundles[0].(map[string]any)
+	if !ok {
+		t.Fatalf("bundle = %#v, want object", bundles[0])
+	}
+	if bundle["created"] != "2026-06-01T06:30:00-05:00" || bundle["file_count"] != float64(1) {
+		t.Fatalf("bundle = %#v, want offset timestamp and file count", bundle)
+	}
+	files, ok := bundle["files"].([]any)
+	if !ok || len(files) != 1 {
+		t.Fatalf("files = %#v, want one file", bundle["files"])
+	}
+	file, ok := files[0].(map[string]any)
+	if !ok || file["path"] != "report.md" || file["sha256"] == "" || file["size"] != float64(9) {
+		t.Fatalf("file = %#v, want projected file metadata", file)
+	}
+}
+
+func TestInspectConfiguredSourceJSONIncludesSecretConflictWarningWithoutValues(t *testing.T) {
+	name := "DISTRIBUTOR_TEST_INSPECT_SECRET"
+	t.Setenv(name, "process-value")
+	sourceRoot := t.TempDir()
+	destinationRoot := t.TempDir()
+	secretsRoot := t.TempDir()
+	if err := os.WriteFile(filepath.Join(secretsRoot, name), []byte("secret-value\n"), 0o600); err != nil {
+		t.Fatalf("write secret: %v", err)
+	}
+	testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{ID: "reports.json"})
+	configPath := writeConfigFile(t, `
+secrets:
+  directory: `+secretsRoot+`
+pipelines:
+  - id: reports
+    source:
+      backend: local
+      path: `+sourceRoot+`
+    destinations:
+      - id: archive
+        backend: local
+        path: `+destinationRoot+`
+`)
+
+	var stdout bytes.Buffer
+	err := Inspect(context.Background(), InspectOptions{
+		ConfigPath:   configPath,
+		PipelineID:   "reports",
+		Stdout:       &stdout,
+		OutputFormat: OutputFormatJSON,
+	})
+	if err != nil {
+		t.Fatalf("Inspect() error = %v", err)
+	}
+	var envelope struct {
+		Warnings []OutputWarning `json:"warnings"`
+	}
+	if err := json.Unmarshal(stdout.Bytes(), &envelope); err != nil {
+		t.Fatalf("decode output: %v; output = %q", err, stdout.String())
+	}
+	if len(envelope.Warnings) != 1 || !strings.Contains(envelope.Warnings[0].Message, "secret "+name+" ignored") {
+		t.Fatalf("warnings = %#v, want secret conflict warning", envelope.Warnings)
+	}
+	output := stdout.String()
+	if strings.Contains(output, "process-value") || strings.Contains(output, "secret-value") {
+		t.Fatalf("stdout exposed secret values: %q", output)
+	}
+}
+
 func TestInspectRequiresPath(t *testing.T) {
 	err := Inspect(context.Background(), InspectOptions{})
 	if err == nil || !strings.Contains(err.Error(), "requires a path") {

internal/app/manifest.go

@@ -92,37 +92,23 @@ func normalizeManifestFiles(files []string) []string {
 }
 
 type manifestCreateResult struct {
-	ManifestPath string                     `json:"manifest_path"`
+	ManifestPath string               `json:"manifest_path"`
-	Root         string                     `json:"root"`
+	Root         string               `json:"root"`
-	ID           string                     `json:"id"`
+	ID           string               `json:"id"`
-	Created      string                     `json:"created"`
+	Created      string               `json:"created"`
-	Digest       string                     `json:"digest"`
+	Digest       string               `json:"digest"`
-	FileCount    int                        `json:"file_count"`
+	FileCount    int                  `json:"file_count"`
-	Files        []manifestCreateFileResult `json:"files"`
+	Files        []manifestFileResult `json:"files"`
-}
-
-type manifestCreateFileResult struct {
-	Path   string `json:"path"`
-	SHA256 string `json:"sha256"`
-	Size   int64  `json:"size"`
 }
 
 func manifestCreateResultFromManifest(root string, manifest producerbundle.Manifest) manifestCreateResult {
-	result := manifestCreateResult{
+	return manifestCreateResult{
 		ManifestPath: filepath.ToSlash(filepath.Join(root, producerbundle.ManifestName)),
 		Root:         filepath.ToSlash(root),
 		ID:           manifest.ID,
-		Created:      manifest.Created.Format(time.RFC3339),
+		Created:      formatManifestCreated(manifest.Created),
 		Digest:       manifest.Digest,
 		FileCount:    len(manifest.Files),
-		Files:        make([]manifestCreateFileResult, 0, len(manifest.Files)),
+		Files:        manifestFileResults(manifest.Files),
 	}
-	for _, file := range manifest.Files {
-		result.Files = append(result.Files, manifestCreateFileResult{
-			Path:   file.Path,
-			SHA256: file.SHA256,
-			Size:   file.Size,
-		})
-	}
-	return result
 }

internal/app/manifest_test.go

@@ -0,0 +1,42 @@
+package app
+
+import (
+	"bytes"
+	"context"
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+func TestManifestCreateJSONPreservesCreatedOffsetAndFileMetadata(t *testing.T) {
+	root := t.TempDir()
+	if err := os.WriteFile(filepath.Join(root, "report.md"), []byte("# Report\n"), 0o600); err != nil {
+		t.Fatalf("write report: %v", err)
+	}
+	var stdout bytes.Buffer
+
+	err := ManifestCreate(context.Background(), ManifestCreateOptions{
+		Root:         root,
+		ID:           "reports.offset",
+		Created:      "2026-06-01T06:30:00-05:00",
+		Files:        []string{"report.md"},
+		Stdout:       &stdout,
+		OutputFormat: OutputFormatJSON,
+	})
+
+	if err != nil {
+		t.Fatalf("ManifestCreate() error = %v", err)
+	}
+	result := decodeAppResult(t, stdout.String())
+	if result["id"] != "reports.offset" || result["created"] != "2026-06-01T06:30:00-05:00" || result["file_count"] != float64(1) {
+		t.Fatalf("result = %#v, want manifest metadata", result)
+	}
+	files, ok := result["files"].([]any)
+	if !ok || len(files) != 1 {
+		t.Fatalf("files = %#v, want one file", result["files"])
+	}
+	file, ok := files[0].(map[string]any)
+	if !ok || file["path"] != "report.md" || file["sha256"] == "" || file["size"] != float64(9) {
+		t.Fatalf("file = %#v, want projected file metadata", file)
+	}
+}

internal/app/output_projection.go

@@ -0,0 +1,83 @@
+package app
+
+import (
+	"time"
+
+	"gitea.maximumdirect.net/eric/distributor/internal/bundle"
+	"gitea.maximumdirect.net/eric/distributor/internal/storage"
+)
+
+type bundleSummaryResult struct {
+	Path string `json:"path"`
+	ID   string `json:"id"`
+}
+
+type bundleDetailResult struct {
+	Path      string               `json:"path"`
+	ID        string               `json:"id"`
+	Created   string               `json:"created"`
+	Digest    string               `json:"digest"`
+	FileCount int                  `json:"file_count"`
+	TotalSize int64                `json:"total_size"`
+	Files     []manifestFileResult `json:"files"`
+}
+
+type manifestFileResult struct {
+	Path   string `json:"path"`
+	SHA256 string `json:"sha256"`
+	Size   int64  `json:"size"`
+}
+
+func bundleSummaryFromBundle(sourceBundle bundle.Bundle) bundleSummaryResult {
+	return bundleSummaryResult{
+		Path: storage.DisplayPath(sourceBundle.RootRelativePath),
+		ID:   sourceBundle.Manifest.ID,
+	}
+}
+
+func bundleSummariesFromBundles(sourceBundles []bundle.Bundle) []bundleSummaryResult {
+	results := make([]bundleSummaryResult, 0, len(sourceBundles))
+	for _, sourceBundle := range sourceBundles {
+		results = append(results, bundleSummaryFromBundle(sourceBundle))
+	}
+	return results
+}
+
+func bundleDetailFromBundle(sourceBundle bundle.Bundle) bundleDetailResult {
+	result := bundleDetailResult{
+		Path:      storage.DisplayPath(sourceBundle.RootRelativePath),
+		ID:        sourceBundle.Manifest.ID,
+		Created:   formatManifestCreated(sourceBundle.Manifest.Created),
+		Digest:    sourceBundle.Manifest.Digest,
+		FileCount: len(sourceBundle.Manifest.Files),
+		Files:     manifestFileResults(sourceBundle.Manifest.Files),
+	}
+	for _, file := range sourceBundle.Manifest.Files {
+		result.TotalSize += file.Size
+	}
+	return result
+}
+
+func bundleDetailsFromBundles(sourceBundles []bundle.Bundle) []bundleDetailResult {
+	results := make([]bundleDetailResult, 0, len(sourceBundles))
+	for _, sourceBundle := range sourceBundles {
+		results = append(results, bundleDetailFromBundle(sourceBundle))
+	}
+	return results
+}
+
+func manifestFileResults(files []bundle.ManifestFile) []manifestFileResult {
+	results := make([]manifestFileResult, 0, len(files))
+	for _, file := range files {
+		results = append(results, manifestFileResult{
+			Path:   file.Path,
+			SHA256: file.SHA256,
+			Size:   file.Size,
+		})
+	}
+	return results
+}
+
+func formatManifestCreated(created time.Time) string {
+	return created.Format(time.RFC3339)
+}

internal/app/run.go

@@ -8,7 +8,6 @@ import (
 	"gitea.maximumdirect.net/eric/distributor/internal/bundle"
 	"gitea.maximumdirect.net/eric/distributor/internal/config"
 	"gitea.maximumdirect.net/eric/distributor/internal/notify"
-	"gitea.maximumdirect.net/eric/distributor/internal/publish"
 	"gitea.maximumdirect.net/eric/distributor/internal/storage"
 )
 
@@ -46,15 +45,11 @@ func Run(ctx context.Context, options RunOptions) error {
 		return err
 	}
 
-	configPath := options.ConfigPath
+	setup, err := loadRuntimeSetup(options.ConfigPath)
-	if configPath == "" {
-		configPath = config.DefaultConfigPath
-	}
-	cfg, err := config.LoadFile(configPath)
 	if err != nil {
 		return err
 	}
-	return runConfig(ctx, cfg, options)
+	return runSetup(ctx, setup, options)
 }
 
 func RunPipeline(ctx context.Context, options RunPipelineOptions) (RunReport, error) {
@@ -62,15 +57,11 @@ func RunPipeline(ctx context.Context, options RunPipelineOptions) (RunReport, er
 		return RunReport{}, err
 	}
 
-	configPath := options.ConfigPath
+	setup, err := loadRuntimeSetup(options.ConfigPath)
-	if configPath == "" {
-		configPath = config.DefaultConfigPath
-	}
-	cfg, err := config.LoadFile(configPath)
 	if err != nil {
 		return RunReport{}, err
 	}
-	return runPipelineConfig(ctx, cfg, options)
+	return runPipelineSetup(ctx, setup, options)
 }
 
 func RunPipelineWithLocalSource(ctx context.Context, options RunPipelineWithLocalSourceOptions) (RunReport, error) {
@@ -81,57 +72,81 @@ func RunPipelineWithLocalSource(ctx context.Context, options RunPipelineWithLoca
 		return RunReport{}, fmt.Errorf("source root is required")
 	}
 
-	configPath := options.ConfigPath
+	setup, err := loadRuntimeSetup(options.ConfigPath)
-	if configPath == "" {
-		configPath = config.DefaultConfigPath
-	}
-	cfg, err := config.LoadFile(configPath)
 	if err != nil {
 		return RunReport{}, err
 	}
-	return runPipelineConfigWithLocalSource(ctx, cfg, options)
+	return runPipelineSetupWithLocalSource(ctx, setup, options)
 }
 
 func runConfig(ctx context.Context, cfg config.Config, options RunOptions) error {
-	return runConfigWithBackendFactory(ctx, cfg, options, newBackendFactoryWithEnvironment)
+	setup, err := runtimeSetupFromConfig("", cfg)
+	if err != nil {
+		return err
+	}
+	return runSetupWithBackendFactory(ctx, setup, options, newBackendFactoryWithEnvironment)
 }
 
 type backendFactoryProvider func(config.Environment) *backendFactory
 
 func runPipelineConfig(ctx context.Context, cfg config.Config, options RunPipelineOptions) (RunReport, error) {
-	return runPipelineConfigWithBackendFactory(ctx, cfg, options, newBackendFactoryWithEnvironment)
+	setup, err := runtimeSetupFromConfig("", cfg)
+	if err != nil {
+		return RunReport{}, err
+	}
+	return runPipelineSetupWithBackendFactory(ctx, setup, options, newBackendFactoryWithEnvironment)
 }
 
 func runPipelineConfigWithLocalSource(ctx context.Context, cfg config.Config, options RunPipelineWithLocalSourceOptions) (RunReport, error) {
-	return runPipelineConfigWithLocalSourceAndBackendFactory(ctx, cfg, options, newBackendFactoryWithEnvironment)
+	setup, err := runtimeSetupFromConfig("", cfg)
+	if err != nil {
+		return RunReport{}, err
+	}
+	return runPipelineSetupWithLocalSourceAndBackendFactory(ctx, setup, options, newBackendFactoryWithEnvironment)
 }
 
 func runPipelineConfigWithBackendFactory(ctx context.Context, cfg config.Config, options RunPipelineOptions, provider backendFactoryProvider) (RunReport, error) {
-	pipeline, ok := findPipeline(cfg, options.PipelineID)
+	setup, err := runtimeSetupFromConfig("", cfg)
+	if err != nil {
+		return RunReport{}, err
+	}
+	return runPipelineSetupWithBackendFactory(ctx, setup, options, provider)
+}
+
+func runPipelineSetup(ctx context.Context, setup runtimeSetup, options RunPipelineOptions) (RunReport, error) {
+	return runPipelineSetupWithBackendFactory(ctx, setup, options, newBackendFactoryWithEnvironment)
+}
+
+func runPipelineSetupWithBackendFactory(ctx context.Context, setup runtimeSetup, options RunPipelineOptions, provider backendFactoryProvider) (RunReport, error) {
+	pipeline, ok := findPipeline(setup.Config, options.PipelineID)
 	if !ok {
 		return RunReport{}, PipelineNotFoundError{ID: options.PipelineID}
 	}
-	return buildRunReportWithBackendFactory(ctx, config.Config{
+	return buildRunReportWithSetup(ctx, setup.withPipelines([]config.Pipeline{pipeline}), RunOptions{
-		Server:    cfg.Server,
-		Secrets:   cfg.Secrets,
-		Pipelines: []config.Pipeline{pipeline},
-	}, RunOptions{
 		DryRun:   options.DryRun,
 		Force:    options.Force,
 		Notifier: options.Notifier,
-	}, provider)
+	}, provider, nil)
 }
 
 func runPipelineConfigWithLocalSourceAndBackendFactory(ctx context.Context, cfg config.Config, options RunPipelineWithLocalSourceOptions, provider backendFactoryProvider) (RunReport, error) {
-	pipeline, ok := findPipeline(cfg, options.PipelineID)
+	setup, err := runtimeSetupFromConfig("", cfg)
+	if err != nil {
+		return RunReport{}, err
+	}
+	return runPipelineSetupWithLocalSourceAndBackendFactory(ctx, setup, options, provider)
+}
+
+func runPipelineSetupWithLocalSource(ctx context.Context, setup runtimeSetup, options RunPipelineWithLocalSourceOptions) (RunReport, error) {
+	return runPipelineSetupWithLocalSourceAndBackendFactory(ctx, setup, options, newBackendFactoryWithEnvironment)
+}
+
+func runPipelineSetupWithLocalSourceAndBackendFactory(ctx context.Context, setup runtimeSetup, options RunPipelineWithLocalSourceOptions, provider backendFactoryProvider) (RunReport, error) {
+	pipeline, ok := findPipeline(setup.Config, options.PipelineID)
 	if !ok {
 		return RunReport{}, PipelineNotFoundError{ID: options.PipelineID}
 	}
-	return buildRunReport(ctx, config.Config{
+	return buildRunReportWithSetup(ctx, setup.withPipelines([]config.Pipeline{pipeline}), RunOptions{
-		Server:    cfg.Server,
-		Secrets:   cfg.Secrets,
-		Pipelines: []config.Pipeline{pipeline},
-	}, RunOptions{
 		DryRun:   options.DryRun,
 		Force:    options.Force,
 		Notifier: options.Notifier,
@@ -142,7 +157,19 @@ func runPipelineConfigWithLocalSourceAndBackendFactory(ctx context.Context, cfg
 }
 
 func runConfigWithBackendFactory(ctx context.Context, cfg config.Config, options RunOptions, provider backendFactoryProvider) error {
-	report, err := buildRunReportWithBackendFactory(ctx, cfg, options, provider)
+	setup, err := runtimeSetupFromConfig("", cfg)
+	if err != nil {
+		return err
+	}
+	return runSetupWithBackendFactory(ctx, setup, options, provider)
+}
+
+func runSetup(ctx context.Context, setup runtimeSetup, options RunOptions) error {
+	return runSetupWithBackendFactory(ctx, setup, options, newBackendFactoryWithEnvironment)
+}
+
+func runSetupWithBackendFactory(ctx context.Context, setup runtimeSetup, options RunOptions, provider backendFactoryProvider) error {
+	report, err := buildRunReportWithSetup(ctx, setup, options, provider, nil)
 	if err != nil && !IsPartialResultError(err) {
 		return err
 	}
@@ -153,7 +180,11 @@ func runConfigWithBackendFactory(ctx context.Context, cfg config.Config, options
 }
 
 func buildRunReportWithBackendFactory(ctx context.Context, cfg config.Config, options RunOptions, provider backendFactoryProvider) (RunReport, error) {
-	return buildRunReport(ctx, cfg, options, provider, nil)
+	setup, err := runtimeSetupFromConfig("", cfg)
+	if err != nil {
+		return RunReport{}, err
+	}
+	return buildRunReportWithSetup(ctx, setup, options, provider, nil)
 }
 
 type localSourceRoot struct {
@@ -162,6 +193,14 @@ type localSourceRoot struct {
 }
 
 func buildRunReport(ctx context.Context, cfg config.Config, options RunOptions, provider backendFactoryProvider, sourceRoot *localSourceRoot) (RunReport, error) {
+	setup, err := runtimeSetupFromConfig("", cfg)
+	if err != nil {
+		return RunReport{}, err
+	}
+	return buildRunReportWithSetup(ctx, setup, options, provider, sourceRoot)
+}
+
+func buildRunReportWithSetup(ctx context.Context, setup runtimeSetup, options RunOptions, provider backendFactoryProvider, sourceRoot *localSourceRoot) (RunReport, error) {
 	notifier := options.Notifier
 	if notifier == nil {
 		notifier = notify.Noop{}
@@ -173,17 +212,17 @@ func buildRunReport(ctx context.Context, cfg config.Config, options RunOptions,
 		Actions:   []RunActionRecord{},
 	}
 	var failures runFailures
-	secretLoad, err := config.LoadSecretEnvironment(cfg.Secrets.Directory, nil)
+	recorder := runReportRecorder{
-	if err != nil {
+		report:   &report,
-		return report, err
+		summary:  &summary,
+		failures: &failures,
 	}
-	secretWarnings := secretConflictWarnings(secretLoad.Conflicts)
+	report.PreambleWarnings = append(report.PreambleWarnings, setup.Warnings...)
-	report.PreambleWarnings = append(report.PreambleWarnings, secretWarnings...)
+	report.addWarnings(setup.Warnings)
-	report.addWarnings(secretWarnings)
+	backends := provider(setup.Environment)
-	backends := provider(secretLoad.Environment)
 	backends.readOnlyKnownHosts = options.DryRun
 	transforms := newTransformRegistry()
-	for _, pipeline := range cfg.Pipelines {
+	for _, pipeline := range setup.Config.Pipelines {
 		pipelineWarnings := sshWarnings(pipeline)
 		report.addWarnings(pipelineWarnings)
 		sourceBackend, bundles, sourceBackendName, err := openPipelineSource(ctx, backends, pipeline, sourceRoot)
@@ -199,103 +238,18 @@ func buildRunReport(ctx context.Context, cfg config.Config, options RunOptions,
 		})
 		pipelineIndex := len(report.Pipelines) - 1
 		for _, destination := range pipeline.Destinations {
-			selections := selectDestinationBundles(destination, bundles)
+			processDestination(ctx, runDestinationRequest{
-			if isFixedPathDestination(destination) {
+				options:       options,
-				summary.recordFixedPath()
+				notifier:      notifier,
-				if options.DryRun {
+				backends:      backends,
-					warning := fixedPathSelectionWarning(pipeline.ID, destination.ID, selections, len(bundles))
+				transforms:    transforms,
-					report.addWarning(warning)
+				pipeline:      pipeline,
-					report.Pipelines[pipelineIndex].events = append(report.Pipelines[pipelineIndex].events, warningEvent(warning))
+				pipelineIndex: pipelineIndex,
-				}
+				sourceBackend: sourceBackend,
-			}
+				bundles:       bundles,
-			if len(selections) == 0 {
+				destination:   destination,
-				continue
+				recorder:      &recorder,
-			}
+			})
-			destinationBackend, err := backends.openDestination(ctx, destination)
-			if err != nil {
-				for _, selection := range selections {
-					failures.add(pipeline.ID, destination.ID, destination.Backend, storage.DisplayPath(selection.SourceBundle.RootRelativePath), err)
-					summary.recordFailure()
-					report.Actions = append(report.Actions, errorAction(pipeline.ID, destination.ID, destination.Backend, selection.SourceBundle.RootRelativePath, err))
-					report.Pipelines[pipelineIndex].events = append(report.Pipelines[pipelineIndex].events, actionEvent(len(report.Actions)-1))
-				}
-				continue
-			}
-			closeDestination := true
-			deferCloseDestination := func() {
-				if closeDestination {
-					closeBackend(destinationBackend)
-					closeDestination = false
-				}
-			}
-			for _, selection := range selections {
-				sourceBundle := selection.SourceBundle
-				req := publish.Request{
-					PipelineID:            pipeline.ID,
-					DestinationID:         destination.ID,
-					SourceBundle:          sourceBundle,
-					SourceBackend:         sourceBackend,
-					DestinationBackend:    destinationBackend,
-					DestinationBundlePath: selection.DestinationBundlePath,
-					PathMapping:           destination.PathMap.Mode,
-					Publish:               *destination.Publish,
-					Transform:             destination.Transform,
-					Links:                 destination.Links,
-					Transformers:          transforms,
-					Transfer:              destination.Transfer,
-					DistributorVersion:    Version,
-					Force:                 options.Force,
-				}
-				plan, err := publish.Build(ctx, req)
-				if err != nil {
-					if plan.PipelineID == "" {
-						plan.PipelineID = pipeline.ID
-					}
-					if plan.DestinationID == "" {
-						plan.DestinationID = destination.ID
-					}
-					if plan.BundleID == "" {
-						plan.BundleID = sourceBundle.Manifest.ID
-					}
-					if plan.BundlePath == "" {
-						plan.BundlePath = sourceBundle.RootRelativePath
-					}
-					if plan.DestinationBundlePath == "" {
-						plan.DestinationBundlePath = selection.DestinationBundlePath
-					}
-				}
-				if isFixedPathDestination(destination) {
-					plan.PathMapping = config.PathMappingFixed
-					if options.DryRun && isDestructiveFixedPathAction(plan.Action) {
-						warning := fixedPathReplacementWarning(plan)
-						report.addWarning(warning)
-						report.Pipelines[pipelineIndex].events = append(report.Pipelines[pipelineIndex].events, warningEvent(warning))
-					}
-				}
-				report.Actions = append(report.Actions, runActionFromPlan(destination.Backend, plan, err))
-				report.Pipelines[pipelineIndex].events = append(report.Pipelines[pipelineIndex].events, actionEvent(len(report.Actions)-1))
-				if err != nil {
-					failures.add(pipeline.ID, destination.ID, destination.Backend, storage.DisplayPath(sourceBundle.RootRelativePath), err)
-					summary.recordFailure()
-					continue
-				}
-				summary.recordPlan(plan.Action)
-				if !options.DryRun {
-					if err := publish.Execute(ctx, req, plan); err != nil {
-						failures.add(pipeline.ID, destination.ID, destination.Backend, storage.DisplayPath(sourceBundle.RootRelativePath), err)
-						summary.recordFailure()
-						continue
-					}
-					if shouldNotify(plan.Action) {
-						if err := notifier.Notify(ctx, notifyEvent(plan)); err != nil {
-							failures.add(pipeline.ID, destination.ID, destination.Backend, storage.DisplayPath(sourceBundle.RootRelativePath), err)
-							summary.recordFailure()
-							continue
-						}
-					}
-				}
-			}
-			deferCloseDestination()
 		}
 		closeBackend(sourceBackend)
 	}
@@ -307,6 +261,12 @@ func buildRunReport(ctx context.Context, cfg config.Config, options RunOptions,
 	return report, nil
 }
 
+type runReportRecorder struct {
+	report   *RunReport
+	summary  *runSummary
+	failures *runFailures
+}
+
 func openPipelineSource(ctx context.Context, backends *backendFactory, pipeline config.Pipeline, sourceRoot *localSourceRoot) (storage.Backend, []bundle.Bundle, string, error) {
 	if sourceRoot != nil && sourceRoot.pipelineID == pipeline.ID {
 		sourceBackend, err := backends.openLocalPath(ctx, sourceRoot.root)

internal/app/run_coordinator.go

@@ -1,131 +0,0 @@
-package app
-
-import (
-	"context"
-	"errors"
-	"fmt"
-	"sync"
-	"time"
-)
-
-type PipelineRunID string
-
-type PipelineRunStatus string
-
-const (
-	PipelineRunRunning   PipelineRunStatus = "running"
-	PipelineRunSucceeded PipelineRunStatus = "succeeded"
-	PipelineRunFailed    PipelineRunStatus = "failed"
-)
-
-type PipelineRunRecord struct {
-	ID         PipelineRunID     `json:"id"`
-	PipelineID string            `json:"pipeline_id"`
-	Status     PipelineRunStatus `json:"status"`
-	StartedAt  time.Time         `json:"started_at"`
-	FinishedAt *time.Time        `json:"finished_at,omitempty"`
-	Report     RunReport         `json:"report,omitempty"`
-	Error      string            `json:"error,omitempty"`
-}
-
-type DuplicatePipelineRunError struct {
-	PipelineID string
-	RunID      PipelineRunID
-}
-
-func (err DuplicatePipelineRunError) Error() string {
-	if err.RunID == "" {
-		return fmt.Sprintf("pipeline %q already has an active run", err.PipelineID)
-	}
-	return fmt.Sprintf("pipeline %q already has active run %s", err.PipelineID, err.RunID)
-}
-
-func IsDuplicatePipelineRun(err error) bool {
-	var duplicate DuplicatePipelineRunError
-	return errors.As(err, &duplicate)
-}
-
-type PipelineRunCoordinator struct {
-	ctx    context.Context
-	run    pipelineRunFunc
-	now    func() time.Time
-	mu     sync.Mutex
-	nextID uint64
-	active map[string]PipelineRunRecord
-}
-
-type pipelineRunFunc func(context.Context, RunPipelineOptions) (RunReport, error)
-
-func NewPipelineRunCoordinator(ctx context.Context) *PipelineRunCoordinator {
-	return newPipelineRunCoordinator(ctx, RunPipeline)
-}
-
-func newPipelineRunCoordinator(ctx context.Context, run pipelineRunFunc) *PipelineRunCoordinator {
-	if ctx == nil {
-		ctx = context.Background()
-	}
-	return &PipelineRunCoordinator{
-		ctx:    ctx,
-		run:    run,
-		now:    time.Now,
-		active: map[string]PipelineRunRecord{},
-	}
-}
-
-func (coordinator *PipelineRunCoordinator) RunPipeline(ctx context.Context, options RunPipelineOptions) (PipelineRunRecord, error) {
-	if ctx == nil {
-		ctx = context.Background()
-	}
-	if err := ctx.Err(); err != nil {
-		return PipelineRunRecord{}, err
-	}
-	record, err := coordinator.admit(options.PipelineID)
-	if err != nil {
-		return PipelineRunRecord{}, err
-	}
-	defer coordinator.clear(options.PipelineID)
-
-	report, runErr := coordinator.run(coordinator.ctx, options)
-	record.Report = report
-	finishedAt := coordinator.now().UTC()
-	record.FinishedAt = &finishedAt
-	if runErr != nil {
-		record.Status = PipelineRunFailed
-		record.Error = runErr.Error()
-		return record, runErr
-	}
-	record.Status = PipelineRunSucceeded
-	return record, nil
-}
-
-func (coordinator *PipelineRunCoordinator) admit(pipelineID string) (PipelineRunRecord, error) {
-	coordinator.mu.Lock()
-	defer coordinator.mu.Unlock()
-	if active, ok := coordinator.active[pipelineID]; ok {
-		return PipelineRunRecord{}, DuplicatePipelineRunError{
-			PipelineID: pipelineID,
-			RunID:      active.ID,
-		}
-	}
-	coordinator.nextID++
-	record := PipelineRunRecord{
-		ID:         PipelineRunID(fmt.Sprintf("run-%016d", coordinator.nextID)),
-		PipelineID: pipelineID,
-		Status:     PipelineRunRunning,
-		StartedAt:  coordinator.now().UTC(),
-	}
-	coordinator.active[pipelineID] = record
-	return record, nil
-}
-
-func (coordinator *PipelineRunCoordinator) clear(pipelineID string) {
-	coordinator.mu.Lock()
-	defer coordinator.mu.Unlock()
-	delete(coordinator.active, pipelineID)
-}
-
-func (coordinator *PipelineRunCoordinator) activeCount() int {
-	coordinator.mu.Lock()
-	defer coordinator.mu.Unlock()
-	return len(coordinator.active)
-}

internal/app/run_coordinator_test.go

@@ -1,223 +0,0 @@
-package app
-
-import (
-	"context"
-	"errors"
-	"sync"
-	"testing"
-	"time"
-)
-
-func TestPipelineRunCoordinatorRejectsDuplicateActiveRun(t *testing.T) {
-	started := make(chan struct{})
-	release := make(chan struct{})
-	var startedOnce sync.Once
-	coordinator := newPipelineRunCoordinator(context.Background(), func(ctx context.Context, options RunPipelineOptions) (RunReport, error) {
-		startedOnce.Do(func() {
-			close(started)
-		})
-		<-release
-		return RunReport{}, nil
-	})
-	firstResult := make(chan runCoordinatorTestResult, 1)
-
-	go func() {
-		record, err := coordinator.RunPipeline(context.Background(), RunPipelineOptions{PipelineID: "reports"})
-		firstResult <- runCoordinatorTestResult{record: record, err: err}
-	}()
-	waitForSignal(t, started, "first run to start")
-
-	_, err := coordinator.RunPipeline(context.Background(), RunPipelineOptions{PipelineID: "reports"})
-	if err == nil || !IsDuplicatePipelineRun(err) {
-		t.Fatalf("RunPipeline() error = %v, want duplicate active run", err)
-	}
-	close(release)
-	result := waitForRunResult(t, firstResult)
-	if result.err != nil {
-		t.Fatalf("first RunPipeline() error = %v", result.err)
-	}
-	if result.record.Status != PipelineRunSucceeded || result.record.ID == "" || result.record.FinishedAt == nil {
-		t.Fatalf("first record = %#v, want succeeded completed record", result.record)
-	}
-	if got := coordinator.activeCount(); got != 0 {
-		t.Fatalf("active count = %d, want 0", got)
-	}
-}
-
-func TestPipelineRunCoordinatorAllowsDifferentActivePipelines(t *testing.T) {
-	started := make(chan string, 2)
-	release := make(chan struct{})
-	coordinator := newPipelineRunCoordinator(context.Background(), func(ctx context.Context, options RunPipelineOptions) (RunReport, error) {
-		started <- options.PipelineID
-		<-release
-		return RunReport{}, nil
-	})
-	firstResult := make(chan runCoordinatorTestResult, 1)
-	secondResult := make(chan runCoordinatorTestResult, 1)
-
-	go func() {
-		record, err := coordinator.RunPipeline(context.Background(), RunPipelineOptions{PipelineID: "reports-one"})
-		firstResult <- runCoordinatorTestResult{record: record, err: err}
-	}()
-	go func() {
-		record, err := coordinator.RunPipeline(context.Background(), RunPipelineOptions{PipelineID: "reports-two"})
-		secondResult <- runCoordinatorTestResult{record: record, err: err}
-	}()
-	startedPipelines := map[string]bool{
-		waitForPipelineID(t, started): true,
-		waitForPipelineID(t, started): true,
-	}
-	if !startedPipelines["reports-one"] || !startedPipelines["reports-two"] {
-		t.Fatalf("started pipelines = %#v, want both requested pipelines", startedPipelines)
-	}
-	if got := coordinator.activeCount(); got != 2 {
-		t.Fatalf("active count = %d, want 2", got)
-	}
-
-	close(release)
-	first := waitForRunResult(t, firstResult)
-	second := waitForRunResult(t, secondResult)
-	if first.err != nil || second.err != nil {
-		t.Fatalf("RunPipeline() errors = %v, %v; want nil", first.err, second.err)
-	}
-	if first.record.ID == second.record.ID {
-		t.Fatalf("run IDs matched: %q", first.record.ID)
-	}
-	if got := coordinator.activeCount(); got != 0 {
-		t.Fatalf("active count = %d, want 0", got)
-	}
-}
-
-func TestPipelineRunCoordinatorClearsActiveRunAfterSuccess(t *testing.T) {
-	coordinator := newPipelineRunCoordinator(context.Background(), func(ctx context.Context, options RunPipelineOptions) (RunReport, error) {
-		return RunReport{DryRun: options.DryRun}, nil
-	})
-
-	first, err := coordinator.RunPipeline(context.Background(), RunPipelineOptions{PipelineID: "reports", DryRun: true})
-	if err != nil {
-		t.Fatalf("first RunPipeline() error = %v", err)
-	}
-	second, err := coordinator.RunPipeline(context.Background(), RunPipelineOptions{PipelineID: "reports"})
-	if err != nil {
-		t.Fatalf("second RunPipeline() error = %v", err)
-	}
-	if first.Status != PipelineRunSucceeded || second.Status != PipelineRunSucceeded {
-		t.Fatalf("statuses = %s, %s; want succeeded", first.Status, second.Status)
-	}
-	if !first.Report.DryRun {
-		t.Fatalf("first report dry_run = false, want true")
-	}
-	if got := coordinator.activeCount(); got != 0 {
-		t.Fatalf("active count = %d, want 0", got)
-	}
-}
-
-func TestPipelineRunCoordinatorClearsActiveRunAfterFailure(t *testing.T) {
-	runError := errors.New("run failed")
-	attempt := 0
-	coordinator := newPipelineRunCoordinator(context.Background(), func(ctx context.Context, options RunPipelineOptions) (RunReport, error) {
-		attempt++
-		if attempt == 1 {
-			return RunReport{}, runError
-		}
-		return RunReport{}, nil
-	})
-
-	first, err := coordinator.RunPipeline(context.Background(), RunPipelineOptions{PipelineID: "reports"})
-	if !errors.Is(err, runError) {
-		t.Fatalf("first RunPipeline() error = %v, want run failure", err)
-	}
-	if first.Status != PipelineRunFailed || first.Error != runError.Error() || first.FinishedAt == nil {
-		t.Fatalf("first record = %#v, want failed completed record", first)
-	}
-	second, err := coordinator.RunPipeline(context.Background(), RunPipelineOptions{PipelineID: "reports"})
-	if err != nil {
-		t.Fatalf("second RunPipeline() error = %v", err)
-	}
-	if second.Status != PipelineRunSucceeded {
-		t.Fatalf("second status = %s, want succeeded", second.Status)
-	}
-	if got := coordinator.activeCount(); got != 0 {
-		t.Fatalf("active count = %d, want 0", got)
-	}
-}
-
-func TestPipelineRunCoordinatorClearsActiveRunAfterCancellation(t *testing.T) {
-	runContext, cancel := context.WithCancel(context.Background())
-	coordinator := newPipelineRunCoordinator(runContext, func(ctx context.Context, options RunPipelineOptions) (RunReport, error) {
-		return RunReport{}, ctx.Err()
-	})
-	cancel()
-
-	record, err := coordinator.RunPipeline(context.Background(), RunPipelineOptions{PipelineID: "reports"})
-	if !errors.Is(err, context.Canceled) {
-		t.Fatalf("RunPipeline() error = %v, want context canceled", err)
-	}
-	if record.Status != PipelineRunFailed || record.Error != context.Canceled.Error() {
-		t.Fatalf("record = %#v, want failed cancellation record", record)
-	}
-	_, err = coordinator.RunPipeline(context.Background(), RunPipelineOptions{PipelineID: "reports"})
-	if !errors.Is(err, context.Canceled) {
-		t.Fatalf("second RunPipeline() error = %v, want context canceled", err)
-	}
-	if IsDuplicatePipelineRun(err) {
-		t.Fatalf("second RunPipeline() error = %v, want cancellation instead of duplicate", err)
-	}
-	if got := coordinator.activeCount(); got != 0 {
-		t.Fatalf("active count = %d, want 0", got)
-	}
-}
-
-func TestPipelineRunCoordinatorUnknownPipelineDoesNotRemainActive(t *testing.T) {
-	coordinator := newPipelineRunCoordinator(context.Background(), func(ctx context.Context, options RunPipelineOptions) (RunReport, error) {
-		return RunReport{}, PipelineNotFoundError{ID: options.PipelineID}
-	})
-
-	_, err := coordinator.RunPipeline(context.Background(), RunPipelineOptions{PipelineID: "missing"})
-	if err == nil || !IsPipelineNotFound(err) {
-		t.Fatalf("RunPipeline() error = %v, want pipeline not found", err)
-	}
-	_, err = coordinator.RunPipeline(context.Background(), RunPipelineOptions{PipelineID: "missing"})
-	if err == nil || !IsPipelineNotFound(err) || IsDuplicatePipelineRun(err) {
-		t.Fatalf("second RunPipeline() error = %v, want pipeline not found without duplicate", err)
-	}
-	if got := coordinator.activeCount(); got != 0 {
-		t.Fatalf("active count = %d, want 0", got)
-	}
-}
-
-type runCoordinatorTestResult struct {
-	record PipelineRunRecord
-	err    error
-}
-
-func waitForSignal(t *testing.T, signal <-chan struct{}, name string) {
-	t.Helper()
-	select {
-	case <-signal:
-	case <-time.After(time.Second):
-		t.Fatalf("timed out waiting for %s", name)
-	}
-}
-
-func waitForPipelineID(t *testing.T, pipelineIDs <-chan string) string {
-	t.Helper()
-	select {
-	case pipelineID := <-pipelineIDs:
-		return pipelineID
-	case <-time.After(time.Second):
-		t.Fatalf("timed out waiting for pipeline start")
-		return ""
-	}
-}
-
-func waitForRunResult(t *testing.T, results <-chan runCoordinatorTestResult) runCoordinatorTestResult {
-	t.Helper()
-	select {
-	case result := <-results:
-		return result
-	case <-time.After(time.Second):
-		t.Fatalf("timed out waiting for run result")
-		return runCoordinatorTestResult{}
-	}
-}

internal/app/run_destination.go

@@ -0,0 +1,163 @@
+package app
+
+import (
+	"context"
+
+	"gitea.maximumdirect.net/eric/distributor/internal/bundle"
+	"gitea.maximumdirect.net/eric/distributor/internal/config"
+	"gitea.maximumdirect.net/eric/distributor/internal/notify"
+	"gitea.maximumdirect.net/eric/distributor/internal/publish"
+	"gitea.maximumdirect.net/eric/distributor/internal/storage"
+)
+
+type runDestinationRequest struct {
+	options       RunOptions
+	notifier      notify.Notifier
+	backends      *backendFactory
+	transforms    publish.TransformerResolver
+	pipeline      config.Pipeline
+	pipelineIndex int
+	sourceBackend storage.Backend
+	bundles       []bundle.Bundle
+	destination   config.Destination
+	recorder      *runReportRecorder
+}
+
+func processDestination(ctx context.Context, request runDestinationRequest) {
+	selections := selectDestinationBundles(request.destination, request.bundles)
+	if isFixedPathDestination(request.destination) {
+		request.recorder.summary.recordFixedPath()
+		if request.options.DryRun {
+			warning := fixedPathSelectionWarning(request.pipeline.ID, request.destination.ID, selections, len(request.bundles))
+			request.recorder.addPipelineWarning(request.pipelineIndex, warning)
+		}
+	}
+	if len(selections) == 0 {
+		return
+	}
+	destinationBackend, err := request.backends.openDestination(ctx, request.destination)
+	if err != nil {
+		for _, selection := range selections {
+			sourceBundle := selection.SourceBundle
+			request.recorder.recordDestinationFailure(request.pipelineIndex, runFailure{
+				pipelineID:    request.pipeline.ID,
+				destinationID: request.destination.ID,
+				backend:       request.destination.Backend,
+				bundlePath:    sourceBundle.RootRelativePath,
+				err:           err,
+			}, errorAction(request.pipeline.ID, request.destination.ID, request.destination.Backend, sourceBundle.RootRelativePath, err), true)
+		}
+		return
+	}
+	defer closeBackend(destinationBackend)
+
+	for _, selection := range selections {
+		processDestinationSelection(ctx, request, destinationBackend, selection)
+	}
+}
+
+func processDestinationSelection(ctx context.Context, request runDestinationRequest, destinationBackend storage.Backend, selection destinationBundleSelection) {
+	sourceBundle := selection.SourceBundle
+	publishRequest := publish.Request{
+		PipelineID:            request.pipeline.ID,
+		DestinationID:         request.destination.ID,
+		SourceBundle:          sourceBundle,
+		SourceBackend:         request.sourceBackend,
+		DestinationBackend:    destinationBackend,
+		DestinationBundlePath: selection.DestinationBundlePath,
+		PathMapping:           request.destination.PathMap.Mode,
+		Publish:               *request.destination.Publish,
+		Transform:             request.destination.Transform,
+		Links:                 request.destination.Links,
+		Transformers:          request.transforms,
+		Transfer:              request.destination.Transfer,
+		DistributorVersion:    Version,
+		Force:                 request.options.Force,
+	}
+	plan, err := publish.Build(ctx, publishRequest)
+	if err != nil {
+		plan = completePlanIdentity(plan, request.pipeline, request.destination, selection)
+	}
+	if isFixedPathDestination(request.destination) {
+		plan.PathMapping = config.PathMappingFixed
+		if request.options.DryRun && isDestructiveFixedPathAction(plan.Action) {
+			warning := fixedPathReplacementWarning(plan)
+			request.recorder.addPipelineWarning(request.pipelineIndex, warning)
+		}
+	}
+	action := runActionFromPlan(request.destination.Backend, plan, err)
+	if err != nil {
+		request.recorder.recordDestinationFailure(request.pipelineIndex, runFailure{
+			pipelineID:    request.pipeline.ID,
+			destinationID: request.destination.ID,
+			backend:       request.destination.Backend,
+			bundlePath:    sourceBundle.RootRelativePath,
+			err:           err,
+		}, action, true)
+		return
+	}
+	request.recorder.addPipelineAction(request.pipelineIndex, action)
+	request.recorder.summary.recordPlan(plan.Action)
+	if request.options.DryRun {
+		return
+	}
+	if err := publish.Execute(ctx, publishRequest, plan); err != nil {
+		request.recorder.recordDestinationFailure(request.pipelineIndex, runFailure{
+			pipelineID:    request.pipeline.ID,
+			destinationID: request.destination.ID,
+			backend:       request.destination.Backend,
+			bundlePath:    sourceBundle.RootRelativePath,
+			err:           err,
+		}, RunActionRecord{}, false)
+		return
+	}
+	if shouldNotify(plan.Action) {
+		if err := request.notifier.Notify(ctx, notifyEvent(plan)); err != nil {
+			request.recorder.recordDestinationFailure(request.pipelineIndex, runFailure{
+				pipelineID:    request.pipeline.ID,
+				destinationID: request.destination.ID,
+				backend:       request.destination.Backend,
+				bundlePath:    sourceBundle.RootRelativePath,
+				err:           err,
+			}, RunActionRecord{}, false)
+			return
+		}
+	}
+}
+
+func (recorder *runReportRecorder) addPipelineWarning(pipelineIndex int, warning OutputWarning) {
+	recorder.report.addWarning(warning)
+	recorder.report.Pipelines[pipelineIndex].events = append(recorder.report.Pipelines[pipelineIndex].events, warningEvent(warning))
+}
+
+func (recorder *runReportRecorder) addPipelineAction(pipelineIndex int, action RunActionRecord) {
+	recorder.report.Actions = append(recorder.report.Actions, action)
+	recorder.report.Pipelines[pipelineIndex].events = append(recorder.report.Pipelines[pipelineIndex].events, actionEvent(len(recorder.report.Actions)-1))
+}
+
+func (recorder *runReportRecorder) recordDestinationFailure(pipelineIndex int, failure runFailure, action RunActionRecord, includeAction bool) {
+	recorder.failures.add(failure.pipelineID, failure.destinationID, failure.backend, storage.DisplayPath(failure.bundlePath), failure.err)
+	recorder.summary.recordFailure()
+	if includeAction {
+		recorder.addPipelineAction(pipelineIndex, action)
+	}
+}
+
+func completePlanIdentity(plan publish.Plan, pipeline config.Pipeline, destination config.Destination, selection destinationBundleSelection) publish.Plan {
+	if plan.PipelineID == "" {
+		plan.PipelineID = pipeline.ID
+	}
+	if plan.DestinationID == "" {
+		plan.DestinationID = destination.ID
+	}
+	if plan.BundleID == "" {
+		plan.BundleID = selection.SourceBundle.Manifest.ID
+	}
+	if plan.BundlePath == "" {
+		plan.BundlePath = selection.SourceBundle.RootRelativePath
+	}
+	if plan.DestinationBundlePath == "" {
+		plan.DestinationBundlePath = selection.DestinationBundlePath
+	}
+	return plan
+}

internal/app/run_test.go

@@ -922,6 +922,50 @@ func TestBuildRunReportIncludesPartialFailures(t *testing.T) {
 	}
 }
 
+func TestBuildRunReportAlignsDestinationOpenFailuresForSelectedBundles(t *testing.T) {
+	sourceRoot := t.TempDir()
+	writeSourceBundle(t, sourceRoot, "daily/one", testBundleOptions{ID: "reports.one"})
+	writeSourceBundle(t, sourceRoot, "daily/two", testBundleOptions{ID: "reports.two", Created: testutil.DefaultCreated.Add(time.Hour)})
+	cfg := config.Config{Pipelines: []config.Pipeline{{
+		ID:     "reports",
+		Source: config.Backend{Backend: config.BackendLocal, Path: sourceRoot},
+		Destinations: []config.Destination{{
+			ID:       "object-archive",
+			Backend:  config.BackendS3,
+			Endpoint: "http://s3.test",
+			Bucket:   "missing-destination",
+		}},
+	}}}
+	config.ApplyDefaults(&cfg)
+
+	report, err := buildRunReportWithBackendFactory(context.Background(), cfg, RunOptions{}, fakeBackendFactoryProvider(t, nil))
+	if err == nil || !IsPartialResultError(err) {
+		t.Fatalf("buildRunReportWithBackendFactory() error = %v, want partial result error", err)
+	}
+	if report.Summary.Status != "failed" || report.Summary.Planned != 0 || report.Summary.Failed != 2 {
+		t.Fatalf("summary = %#v, want two destination open failures", report.Summary)
+	}
+	if got, want := len(report.Actions), 2; got != want {
+		t.Fatalf("action count = %d, want %d", got, want)
+	}
+	if got, want := len(report.OutputErrors), 2; got != want {
+		t.Fatalf("output error count = %d, want %d", got, want)
+	}
+	if got, want := len(report.Pipelines[0].events), 2; got != want {
+		t.Fatalf("pipeline event count = %d, want %d", got, want)
+	}
+	for index, bundlePath := range []string{"daily/one", "daily/two"} {
+		action := report.Actions[index]
+		if action.PipelineID != "reports" || action.DestinationID != "object-archive" || action.Backend != config.BackendS3 || action.BundlePath != bundlePath || action.Action != "error" {
+			t.Fatalf("action[%d] = %#v, want %s destination open error", index, action, bundlePath)
+		}
+		outputError := report.OutputErrors[index]
+		if outputError.PipelineID != action.PipelineID || outputError.DestinationID != action.DestinationID || outputError.Backend != action.Backend || outputError.BundlePath != action.BundlePath {
+			t.Fatalf("output error[%d] = %#v, action = %#v, want aligned identity", index, outputError, action)
+		}
+	}
+}
+
 func TestRunPipelineRunsOnlyRequestedPipeline(t *testing.T) {
 	firstSource := t.TempDir()
 	secondSource := t.TempDir()

internal/app/runtime.go

@@ -0,0 +1,44 @@
+package app
+
+import "gitea.maximumdirect.net/eric/distributor/internal/config"
+
+type runtimeSetup struct {
+	ConfigPath  string
+	Config      config.Config
+	Environment config.Environment
+	Warnings    []OutputWarning
+}
+
+func loadRuntimeSetup(configPath string) (runtimeSetup, error) {
+	resolvedPath := runtimeConfigPath(configPath)
+	cfg, err := config.LoadFile(resolvedPath)
+	if err != nil {
+		return runtimeSetup{}, err
+	}
+	return runtimeSetupFromConfig(resolvedPath, cfg)
+}
+
+func runtimeSetupFromConfig(configPath string, cfg config.Config) (runtimeSetup, error) {
+	secretLoad, err := config.LoadSecretEnvironment(cfg.Secrets.Directory, nil)
+	if err != nil {
+		return runtimeSetup{}, err
+	}
+	return runtimeSetup{
+		ConfigPath:  configPath,
+		Config:      cfg,
+		Environment: secretLoad.Environment,
+		Warnings:    secretConflictWarnings(secretLoad.Conflicts),
+	}, nil
+}
+
+func runtimeConfigPath(configPath string) string {
+	if configPath == "" {
+		return config.DefaultConfigPath
+	}
+	return configPath
+}
+
+func (setup runtimeSetup) withPipelines(pipelines []config.Pipeline) runtimeSetup {
+	setup.Config.Pipelines = pipelines
+	return setup
+}

internal/app/runtime_test.go

@@ -0,0 +1,32 @@
+package app
+
+import (
+	"testing"
+
+	"gitea.maximumdirect.net/eric/distributor/internal/config"
+	"gitea.maximumdirect.net/eric/distributor/internal/testutil"
+)
+
+func TestRuntimeConfigPathDefaultsEmptyPath(t *testing.T) {
+	if got, want := runtimeConfigPath(""), config.DefaultConfigPath; got != want {
+		t.Fatalf("runtimeConfigPath(\"\") = %q, want %q", got, want)
+	}
+	if got, want := runtimeConfigPath("/tmp/distributor.yml"), "/tmp/distributor.yml"; got != want {
+		t.Fatalf("runtimeConfigPath(explicit) = %q, want %q", got, want)
+	}
+}
+
+func TestLoadRuntimeSetupReturnsLoadedConfigPath(t *testing.T) {
+	configPath := testutil.WriteMinimalLocalConfig(t, t.TempDir(), t.TempDir())
+
+	setup, err := loadRuntimeSetup(configPath)
+	if err != nil {
+		t.Fatalf("loadRuntimeSetup() error = %v", err)
+	}
+	if setup.ConfigPath != configPath {
+		t.Fatalf("ConfigPath = %q, want %q", setup.ConfigPath, configPath)
+	}
+	if len(setup.Config.Pipelines) != 1 {
+		t.Fatalf("pipeline count = %d, want 1", len(setup.Config.Pipelines))
+	}
+}

internal/app/serve.go

@@ -6,8 +6,6 @@ import (
 	"fmt"
 	"net"
 	"net/http"
-
-	"gitea.maximumdirect.net/eric/distributor/internal/config"
 )
 
 type ServeOptions struct {
@@ -22,26 +20,18 @@ func Serve(ctx context.Context, options ServeOptions) error {
 		return err
 	}
 
-	configPath := options.ConfigPath
+	setup, err := loadRuntimeSetup(options.ConfigPath)
-	if configPath == "" {
-		configPath = config.DefaultConfigPath
-	}
-	cfg, err := config.LoadFile(configPath)
-	if err != nil {
-		return err
-	}
-	secretLoad, err := config.LoadSecretEnvironment(cfg.Secrets.Directory, nil)
 	if err != nil {
 		return err
 	}
 
-	handler, err := newUploadHTTPHandler(ctx, cfg, secretLoad.Environment)
+	handler, err := newUploadHTTPHandler(ctx, setup.Config, setup.Environment)
 	if err != nil {
 		return err
 	}
-	listener, err := net.Listen("tcp", cfg.Server.HTTP.Bind)
+	listener, err := net.Listen("tcp", setup.Config.Server.HTTP.Bind)
 	if err != nil {
-		return fmt.Errorf("bind HTTP server %q: %w", cfg.Server.HTTP.Bind, err)
+		return fmt.Errorf("bind HTTP server %q: %w", setup.Config.Server.HTTP.Bind, err)
 	}
 	defer listener.Close()
 

internal/app/serve_test.go

@@ -0,0 +1,88 @@
+package app
+
+import (
+	"context"
+	"strings"
+	"testing"
+)
+
+func TestServeFailsForUnsafeUploadTokensWithoutLeakingValues(t *testing.T) {
+	duplicateSecret := "duplicate-secret"
+	tests := []struct {
+		name       string
+		configPath func(*testing.T) string
+		env        map[string]string
+		want       string
+	}{
+		{
+			name: "missing token",
+			configPath: func(t *testing.T) string {
+				return writeServeUploadConfig(t, []string{"DISTRIBUTOR_TEST_MISSING_UPLOAD_TOKEN"})
+			},
+			want: "DISTRIBUTOR_TEST_MISSING_UPLOAD_TOKEN",
+		},
+		{
+			name: "empty token",
+			configPath: func(t *testing.T) string {
+				return writeServeUploadConfig(t, []string{"DISTRIBUTOR_TEST_EMPTY_UPLOAD_TOKEN"})
+			},
+			env:  map[string]string{"DISTRIBUTOR_TEST_EMPTY_UPLOAD_TOKEN": ""},
+			want: "DISTRIBUTOR_TEST_EMPTY_UPLOAD_TOKEN",
+		},
+		{
+			name: "duplicate token",
+			configPath: func(t *testing.T) string {
+				return writeServeUploadConfig(t, []string{
+					"DISTRIBUTOR_TEST_FIRST_UPLOAD_TOKEN",
+					"DISTRIBUTOR_TEST_SECOND_UPLOAD_TOKEN",
+				})
+			},
+			env: map[string]string{
+				"DISTRIBUTOR_TEST_FIRST_UPLOAD_TOKEN":  duplicateSecret,
+				"DISTRIBUTOR_TEST_SECOND_UPLOAD_TOKEN": duplicateSecret,
+			},
+			want: "same value",
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			for name, value := range tt.env {
+				t.Setenv(name, value)
+			}
+
+			err := Serve(context.Background(), ServeOptions{ConfigPath: tt.configPath(t)})
+			if err == nil {
+				t.Fatal("Serve() error = nil, want token startup error")
+			}
+			if !strings.Contains(err.Error(), tt.want) {
+				t.Fatalf("Serve() error = %v, want %q", err, tt.want)
+			}
+			if strings.Contains(err.Error(), duplicateSecret) {
+				t.Fatalf("Serve() error exposed token value: %v", err)
+			}
+		})
+	}
+}
+
+func writeServeUploadConfig(t *testing.T, tokenEnvs []string) string {
+	t.Helper()
+	body := `
+server:
+  http:
+    bind: 127.0.0.1:0
+pipelines:
+`
+	for index, tokenEnv := range tokenEnvs {
+		body += `
+  - id: reports-` + string(rune('a'+index)) + `
+    source:
+      backend: http_upload
+      token_env: ` + tokenEnv + `
+    destinations:
+      - id: archive
+        backend: local
+        path: ` + t.TempDir() + `
+`
+	}
+	return writeConfigFile(t, body)
+}

internal/app/source_select.go

@@ -44,11 +44,11 @@ func selectSourceBundles(ctx context.Context, options sourceCommandOptions, prov
 		return sourceSelection{}, err
 	}
 	if options.ConfigPath != "" {
-		cfg, err := config.LoadFile(options.ConfigPath)
+		setup, err := loadRuntimeSetup(options.ConfigPath)
 		if err != nil {
 			return sourceSelection{}, err
 		}
-		return selectSourceBundlesFromConfig(ctx, cfg, options, provider)
+		return selectSourceBundlesFromSetup(ctx, setup, options, provider)
 	}
 	if options.PipelineID != "" {
 		return sourceSelection{}, fmt.Errorf("configured source mode requires --config")
@@ -72,21 +72,25 @@ func selectSourceBundles(ctx context.Context, options sourceCommandOptions, prov
 }
 
 func selectSourceBundlesFromConfig(ctx context.Context, cfg config.Config, options sourceCommandOptions, provider backendFactoryProvider) (sourceSelection, error) {
+	setup, err := runtimeSetupFromConfig("", cfg)
+	if err != nil {
+		return sourceSelection{}, err
+	}
+	return selectSourceBundlesFromSetup(ctx, setup, options, provider)
+}
+
+func selectSourceBundlesFromSetup(ctx context.Context, setup runtimeSetup, options sourceCommandOptions, provider backendFactoryProvider) (sourceSelection, error) {
 	if options.Path != "" {
 		return sourceSelection{}, fmt.Errorf("configured source mode does not accept a local path")
 	}
 	if options.PipelineID == "" {
 		return sourceSelection{}, fmt.Errorf("configured source mode requires --pipeline")
 	}
-	secretLoad, err := config.LoadSecretEnvironment(cfg.Secrets.Directory, nil)
+	pipeline, ok := findPipeline(setup.Config, options.PipelineID)
-	if err != nil {
-		return sourceSelection{}, err
-	}
-	pipeline, ok := findPipeline(cfg, options.PipelineID)
 	if !ok {
 		return sourceSelection{}, PipelineNotFoundError{ID: options.PipelineID}
 	}
-	backends := provider(secretLoad.Environment)
+	backends := provider(setup.Environment)
 	sourceBackend, err := backends.openSource(ctx, pipeline.Source)
 	if err != nil {
 		return sourceSelection{}, fmt.Errorf("pipeline %s source backend %s: %w", pipeline.ID, pipeline.Source.Backend, err)
@@ -111,7 +115,7 @@ func selectSourceBundlesFromConfig(ctx context.Context, cfg config.Config, optio
 		PipelineID:    pipeline.ID,
 		SourceBackend: pipeline.Source.Backend,
 		ConfigMode:    true,
-		Warnings:      append(secretConflictWarnings(secretLoad.Conflicts), sourceSSHWarnings(pipeline)...),
+		Warnings:      append(setup.Warnings, sourceSSHWarnings(pipeline)...),
 	}, nil
 }
 

internal/app/upload_coordinator.go

@@ -78,6 +78,7 @@ type UploadCoordinator struct {
 	queueSize      int
 	maxConcurrency int
 	runningCount   int
+	reservedCount  int
 	activePipeline map[string]bool
 	pending        []*uploadJob
 	records        map[UploadRunID]UploadRunRecord
@@ -88,9 +89,10 @@ type uploadStageFunc func(context.Context, ingest.StageOptions) (ingest.StagedBu
 type uploadRunFunc func(context.Context, config.Config, RunPipelineWithLocalSourceOptions) (RunReport, error)
 
 type uploadJob struct {
-	recordID UploadRunID
+	recordID   UploadRunID
-	request  UploadRequest
+	request    UploadRequest
-	pipeline config.Pipeline
+	pipeline   config.Pipeline
+	stagedRoot string
 }
 
 type uploadCoordinatorHooks struct {
@@ -164,24 +166,49 @@ func (coordinator *UploadCoordinator) Submit(ctx context.Context, request Upload
 	if err != nil {
 		return UploadRunRecord{}, err
 	}
+	if err := ingest.ValidateContentType(request.ContentType); err != nil {
+		return UploadRunRecord{}, err
+	}
+
+	coordinator.mu.Lock()
+	coordinator.expireLocked(coordinator.now().UTC())
+	if coordinator.queueFullLocked() {
+		coordinator.mu.Unlock()
+		return UploadRunRecord{}, UploadQueueFullError{QueueSize: coordinator.queueSize}
+	}
+	coordinator.reservedCount++
+	coordinator.mu.Unlock()
+
+	staged, err := coordinator.stage(ctx, ingest.StageOptions{
+		Body:                request.Body,
+		ContentType:         request.ContentType,
+		PipelineStagingPath: pipeline.Source.Upload.StagingPath,
+		RunID:               string(runID),
+		MaxUploadSize:       int64(*pipeline.Source.Upload.MaxUploadSize),
+		MaxExtractedSize:    int64(*pipeline.Source.Upload.MaxUploadSize),
+		MaxFileCount:        uploadMaxFileCount(request.MaxFileCount),
+	})
+	if err != nil {
+		coordinator.releaseReservation()
+		return UploadRunRecord{}, err
+	}
 
 	coordinator.mu.Lock()
 	defer coordinator.mu.Unlock()
-	coordinator.expireLocked(coordinator.now().UTC())
+	coordinator.reservedCount--
-	if len(coordinator.pending) >= coordinator.queueSize {
-		return UploadRunRecord{}, UploadQueueFullError{QueueSize: coordinator.queueSize}
-	}
 	record := UploadRunRecord{
 		ID:         runID,
 		PipelineID: pipeline.ID,
 		Status:     UploadStatusAccepted,
 		AcceptedAt: coordinator.now().UTC(),
+		StagedRoot: staged.Root,
 	}
 	coordinator.records[runID] = record
 	coordinator.pending = append(coordinator.pending, &uploadJob{
-		recordID: runID,
+		recordID:   runID,
-		request:  request,
+		request:    request,
-		pipeline: pipeline,
+		pipeline:   pipeline,
+		stagedRoot: staged.Root,
 	})
 	coordinator.notify()
 	return record, nil
@@ -205,13 +232,13 @@ func (coordinator *UploadCoordinator) CanAccept() bool {
 	coordinator.mu.Lock()
 	defer coordinator.mu.Unlock()
 	coordinator.expireLocked(coordinator.now().UTC())
-	return len(coordinator.pending) < coordinator.queueSize
+	return !coordinator.queueFullLocked()
 }
 
 func (coordinator *UploadCoordinator) QueueDepth() int {
 	coordinator.mu.Lock()
 	defer coordinator.mu.Unlock()
-	return len(coordinator.pending)
+	return len(coordinator.pending) + coordinator.reservedCount
 }
 
 func (coordinator *UploadCoordinator) RunningCount() int {
@@ -290,47 +317,30 @@ func (coordinator *UploadCoordinator) markPendingQueuedLocked() {
 }
 
 func (coordinator *UploadCoordinator) runJob(job *uploadJob) {
-	record := coordinator.currentRecord(job.recordID)
+	report, err := coordinator.run(coordinator.ctx, coordinator.cfg, RunPipelineWithLocalSourceOptions{
-	maxFileCount := job.request.MaxFileCount
+		PipelineID: job.pipeline.ID,
-	if maxFileCount <= 0 {
+		SourceRoot: job.stagedRoot,
-		maxFileCount = DefaultUploadMaxFileCount
+		DryRun:     job.request.DryRun,
-	}
+		Force:      job.request.Force,
-	staged, err := coordinator.stage(coordinator.ctx, ingest.StageOptions{
-		Body:                job.request.Body,
-		ContentType:         job.request.ContentType,
-		PipelineStagingPath: job.pipeline.Source.Upload.StagingPath,
-		RunID:               string(record.ID),
-		MaxUploadSize:       int64(*job.pipeline.Source.Upload.MaxUploadSize),
-		MaxExtractedSize:    int64(*job.pipeline.Source.Upload.MaxUploadSize),
-		MaxFileCount:        maxFileCount,
 	})
-	if err == nil {
+	coordinator.complete(job, &report, err)
-		coordinator.setStagedRoot(job.recordID, staged.Root)
+}
-		var report RunReport
+
-		report, err = coordinator.run(coordinator.ctx, coordinator.cfg, RunPipelineWithLocalSourceOptions{
+func (coordinator *UploadCoordinator) releaseReservation() {
-			PipelineID: job.pipeline.ID,
+	coordinator.mu.Lock()
-			SourceRoot: staged.Root,
+	defer coordinator.mu.Unlock()
-			DryRun:     job.request.DryRun,
+	coordinator.reservedCount--
-			Force:      job.request.Force,
+}
-		})
+
-		coordinator.complete(job, &report, err)
+func (coordinator *UploadCoordinator) queueFullLocked() bool {
-		return
+	return len(coordinator.pending)+coordinator.reservedCount >= coordinator.queueSize
+}
+
+func uploadMaxFileCount(value int) int {
+	if value > 0 {
+		return value
 	}
-	coordinator.complete(job, nil, err)
+	return DefaultUploadMaxFileCount
-}
-
-func (coordinator *UploadCoordinator) currentRecord(runID UploadRunID) UploadRunRecord {
-	coordinator.mu.Lock()
-	defer coordinator.mu.Unlock()
-	return coordinator.records[runID]
-}
-
-func (coordinator *UploadCoordinator) setStagedRoot(runID UploadRunID, root string) {
-	coordinator.mu.Lock()
-	defer coordinator.mu.Unlock()
-	record := coordinator.records[runID]
-	record.StagedRoot = root
-	coordinator.records[runID] = record
 }
 
 func (coordinator *UploadCoordinator) complete(job *uploadJob, report *RunReport, runErr error) {

internal/app/upload_http.go

@@ -1,13 +1,10 @@
 package app
 
 import (
-	"bytes"
 	"context"
 	"encoding/json"
 	"errors"
 	"fmt"
-	"io"
-	"mime"
 	"net/http"
 	"strings"
 
@@ -24,7 +21,6 @@ type uploadCoordinator interface {
 type uploadHTTPHandler struct {
 	coordinator uploadCoordinator
 	tokens      map[string]string
-	limits      map[string]int64
 }
 
 type uploadAcceptedResponse struct {
@@ -38,20 +34,18 @@ type httpErrorResponse struct {
 
 func newUploadHTTPHandler(ctx context.Context, cfg config.Config, environment config.Environment) (http.Handler, error) {
 	config.ApplyDefaults(&cfg)
-	tokens, limits, err := resolveUploadTokens(cfg, environment)
+	tokens, err := resolveUploadTokens(cfg, environment)
 	if err != nil {
 		return nil, err
 	}
 	return uploadHTTPHandler{
 		coordinator: NewUploadCoordinator(ctx, cfg),
 		tokens:      tokens,
-		limits:      limits,
 	}, nil
 }
 
-func resolveUploadTokens(cfg config.Config, environment config.Environment) (map[string]string, map[string]int64, error) {
+func resolveUploadTokens(cfg config.Config, environment config.Environment) (map[string]string, error) {
 	tokens := make(map[string]string)
-	limits := make(map[string]int64)
 	for _, pipeline := range cfg.Pipelines {
 		if pipeline.Source.Backend != config.BackendHTTPUpload {
 			continue
@@ -59,18 +53,17 @@ func resolveUploadTokens(cfg config.Config, environment config.Environment) (map
 		tokenName := pipeline.Source.Upload.TokenEnv
 		token, ok := environment.Lookup(tokenName)
 		if !ok {
-			return nil, nil, fmt.Errorf("upload token environment variable %s is not set", tokenName)
+			return nil, fmt.Errorf("upload token environment variable %s is not set", tokenName)
 		}
 		if token == "" {
-			return nil, nil, fmt.Errorf("upload token environment variable %s is empty", tokenName)
+			return nil, fmt.Errorf("upload token environment variable %s is empty", tokenName)
 		}
 		if existing, exists := tokens[token]; exists {
-			return nil, nil, fmt.Errorf("upload token environment variables for pipelines %s and %s resolve to the same value", existing, pipeline.ID)
+			return nil, fmt.Errorf("upload token environment variables for pipelines %s and %s resolve to the same value", existing, pipeline.ID)
 		}
 		tokens[token] = pipeline.ID
-		limits[pipeline.ID] = int64(*pipeline.Source.Upload.MaxUploadSize)
 	}
-	return tokens, limits, nil
+	return tokens, nil
 }
 
 func (handler uploadHTTPHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
@@ -101,7 +94,7 @@ func (handler uploadHTTPHandler) handleUpload(w http.ResponseWriter, r *http.Req
 		return
 	}
 	contentType := r.Header.Get("Content-Type")
-	if !supportedUploadContentType(contentType) {
+	if err := ingest.ValidateContentType(contentType); err != nil {
 		writeHTTPError(w, http.StatusUnsupportedMediaType, "unsupported content type")
 		return
 	}
@@ -109,19 +102,10 @@ func (handler uploadHTTPHandler) handleUpload(w http.ResponseWriter, r *http.Req
 		writeHTTPError(w, http.StatusServiceUnavailable, "upload queue is full")
 		return
 	}
-	body, err := readUploadBody(r.Body, handler.limits[pipelineID])
-	if err != nil {
-		if errors.Is(err, ingest.ErrUploadTooLarge) {
-			writeHTTPError(w, http.StatusRequestEntityTooLarge, "upload exceeds maximum size")
-			return
-		}
-		writeHTTPError(w, http.StatusBadRequest, "read upload body failed")
-		return
-	}
 	record, err := handler.coordinator.Submit(r.Context(), UploadRequest{
 		PipelineID:  pipelineID,
 		ContentType: contentType,
-		Body:        bytes.NewReader(body),
+		Body:        r.Body,
 	})
 	if err != nil {
 		writeUploadSubmitError(w, err)
@@ -160,31 +144,6 @@ func (handler uploadHTTPHandler) authenticate(header string) (string, bool) {
 	return pipelineID, ok
 }
 
-func supportedUploadContentType(contentType string) bool {
-	mediaType, _, err := mime.ParseMediaType(contentType)
-	if err != nil {
-		mediaType = contentType
-	}
-	switch mediaType {
-	case ingest.ContentTypeTar, ingest.ContentTypeGzip, ingest.ContentTypeXGzip:
-		return true
-	default:
-		return false
-	}
-}
-
-func readUploadBody(body io.Reader, maxSize int64) ([]byte, error) {
-	limited := &io.LimitedReader{R: body, N: maxSize + 1}
-	data, err := io.ReadAll(limited)
-	if err != nil {
-		return nil, err
-	}
-	if int64(len(data)) > maxSize {
-		return nil, ingest.ErrUploadTooLarge
-	}
-	return data, nil
-}
-
 func writeUploadSubmitError(w http.ResponseWriter, err error) {
 	switch {
 	case IsUploadQueueFull(err):

internal/app/upload_http_integration_test.go

@@ -13,6 +13,7 @@ import (
 	"net/http/httptest"
 	"os"
 	"path/filepath"
+	"strings"
 	"testing"
 	"time"
 
@@ -68,32 +69,65 @@ func TestHTTPUploadPublishesTarAndGzipFanout(t *testing.T) {
 	}
 }
 
-func TestHTTPUploadInvalidArchiveFailsWithoutPublishing(t *testing.T) {
+func TestHTTPUploadInvalidArchiveIsRejectedWithoutRunID(t *testing.T) {
 	destination := t.TempDir()
-	cfg := httpUploadIntegrationConfig(t, []httpUploadPipelineSpec{{
+	coordinator := NewUploadCoordinator(context.Background(), httpUploadIntegrationConfig(t, []httpUploadPipelineSpec{{
 		id:           "reports",
 		tokenEnv:     "REPORTS_TOKEN",
 		stagingPath:  filepath.Join(t.TempDir(), "reports"),
 		destinations: []string{destination},
-	}}, 4, 1)
+	}}, 4, 1))
-	handler, err := newUploadHTTPHandler(context.Background(), cfg, uploadHTTPTestEnvironment(map[string]string{
+	handler := uploadHTTPHandler{
-		"REPORTS_TOKEN": "reports-secret",
+		coordinator: coordinator,
-	}))
+		tokens:      map[string]string{"reports-secret": "reports"},
-	if err != nil {
-		t.Fatalf("newUploadHTTPHandler() error = %v", err)
 	}
 	server := httptest.NewServer(handler)
 	defer server.Close()
 
-	runID := submitHTTPUpload(t, server, "reports-secret", ingest.ContentTypeTar, []byte("not a tar archive"))
+	status, body := postHTTPUpload(t, server, "reports-secret", ingest.ContentTypeTar, []byte("not a tar archive"))
-	record := waitForHTTPUploadStatus(t, server, runID, UploadStatusFailed)
+	if status != http.StatusBadRequest {
+		t.Fatalf("POST /upload status = %d, want %d; body = %s", status, http.StatusBadRequest, body)
+	}
+	if strings.Contains(body, "run_id") || strings.Contains(body, "reports-secret") {
+		t.Fatalf("invalid archive response exposed run id or token: %s", body)
+	}
+	if got := coordinator.QueueDepth(); got != 0 {
+		t.Fatalf("queue depth = %d, want 0", got)
+	}
+	assertDirectoryEmpty(t, destination)
+}
 
-	if record.Error == "" {
+func TestHTTPUploadOversizedArchiveIsRejectedWithoutRunID(t *testing.T) {
-		t.Fatal("failed status error is empty")
+	destination := t.TempDir()
+	stagingPath := filepath.Join(t.TempDir(), "reports")
+	cfg := httpUploadIntegrationConfig(t, []httpUploadPipelineSpec{{
+		id:           "reports",
+		tokenEnv:     "REPORTS_TOKEN",
+		stagingPath:  stagingPath,
+		destinations: []string{destination},
+	}}, 4, 1)
+	size := config.ByteSize(4)
+	cfg.Server.HTTP.MaxUploadSize = &size
+	cfg.Pipelines[0].Source.Upload.MaxUploadSize = &size
+	coordinator := NewUploadCoordinator(context.Background(), cfg)
+	handler := uploadHTTPHandler{
+		coordinator: coordinator,
+		tokens:      map[string]string{"reports-secret": "reports"},
 	}
-	if record.Report != nil {
+	server := httptest.NewServer(handler)
-		t.Fatalf("failed staging report = %#v, want nil", record.Report)
+	defer server.Close()
+
+	status, body := postHTTPUpload(t, server, "reports-secret", ingest.ContentTypeTar, bundleArchive(t, false, testutil.BundleOptions{}))
+	if status != http.StatusRequestEntityTooLarge {
+		t.Fatalf("POST /upload status = %d, want %d; body = %s", status, http.StatusRequestEntityTooLarge, body)
 	}
+	if strings.Contains(body, "run_id") || strings.Contains(body, "reports-secret") {
+		t.Fatalf("oversized response exposed run id or token: %s", body)
+	}
+	if got := coordinator.QueueDepth(); got != 0 {
+		t.Fatalf("queue depth = %d, want 0", got)
+	}
+	assertDirectoryEmpty(t, stagingPath)
 	assertDirectoryEmpty(t, destination)
 }
 
@@ -122,7 +156,6 @@ func TestHTTPUploadSamePipelineRequestsSerialize(t *testing.T) {
 	handler := uploadHTTPHandler{
 		coordinator: coordinator,
 		tokens:      map[string]string{"reports-secret": "reports"},
-		limits:      map[string]int64{"reports": 1024},
 	}
 	server := httptest.NewServer(handler)
 	defer server.Close()
@@ -177,10 +210,6 @@ func TestHTTPUploadDifferentPipelinesRunConcurrently(t *testing.T) {
 			"one-secret": "reports-one",
 			"two-secret": "reports-two",
 		},
-		limits: map[string]int64{
-			"reports-one": 1024,
-			"reports-two": 1024,
-		},
 	}
 	server := httptest.NewServer(handler)
 	defer server.Close()
@@ -247,6 +276,22 @@ func httpUploadIntegrationConfig(t *testing.T, pipelines []httpUploadPipelineSpe
 }
 
 func submitHTTPUpload(t *testing.T, server *httptest.Server, token, contentType string, body []byte) UploadRunID {
+	t.Helper()
+	status, responseBody := postHTTPUpload(t, server, token, contentType, body)
+	if status != http.StatusAccepted {
+		t.Fatalf("POST /upload status = %d, want %d; body = %s", status, http.StatusAccepted, responseBody)
+	}
+	var accepted uploadAcceptedResponse
+	if err := json.Unmarshal([]byte(responseBody), &accepted); err != nil {
+		t.Fatalf("decode accepted response: %v", err)
+	}
+	if accepted.RunID == "" || accepted.Status != UploadStatusAccepted {
+		t.Fatalf("accepted response = %#v, want run id and accepted status", accepted)
+	}
+	return accepted.RunID
+}
+
+func postHTTPUpload(t *testing.T, server *httptest.Server, token, contentType string, body []byte) (int, string) {
 	t.Helper()
 	request, err := http.NewRequest(http.MethodPost, server.URL+"/upload", bytes.NewReader(body))
 	if err != nil {
@@ -259,17 +304,11 @@ func submitHTTPUpload(t *testing.T, server *httptest.Server, token, contentType
 		t.Fatalf("POST /upload error = %v", err)
 	}
 	defer response.Body.Close()
-	if response.StatusCode != http.StatusAccepted {
+	data, err := io.ReadAll(response.Body)
-		t.Fatalf("POST /upload status = %d, want %d", response.StatusCode, http.StatusAccepted)
+	if err != nil {
+		t.Fatalf("read response body: %v", err)
 	}
-	var accepted uploadAcceptedResponse
+	return response.StatusCode, string(data)
-	if err := json.NewDecoder(response.Body).Decode(&accepted); err != nil {
-		t.Fatalf("decode accepted response: %v", err)
-	}
-	if accepted.RunID == "" || accepted.Status != UploadStatusAccepted {
-		t.Fatalf("accepted response = %#v, want run id and accepted status", accepted)
-	}
-	return accepted.RunID
 }
 
 func waitForHTTPUploadStatus(t *testing.T, server *httptest.Server, runID UploadRunID, status UploadStatus) UploadRunRecord {

internal/app/upload_http_test.go

@@ -12,6 +12,7 @@ import (
 	"time"
 
 	"gitea.maximumdirect.net/eric/distributor/internal/config"
+	"gitea.maximumdirect.net/eric/distributor/internal/ingest"
 )
 
 type fakeUploadCoordinator struct {
@@ -41,7 +42,7 @@ func (fake fakeUploadCoordinator) Status(runID UploadRunID) (UploadRunRecord, bo
 func TestResolveUploadTokensFailsForMissingAndDuplicateTokens(t *testing.T) {
 	cfg := uploadHTTPTestConfig()
 
-	_, _, err := resolveUploadTokens(cfg, config.NewEnvironment(nil, func(string) (string, bool) {
+	_, err := resolveUploadTokens(cfg, config.NewEnvironment(nil, func(string) (string, bool) {
 		return "", false
 	}))
 	if err == nil || !strings.Contains(err.Error(), "UPLOAD_TOKEN") {
@@ -58,7 +59,7 @@ func TestResolveUploadTokensFailsForMissingAndDuplicateTokens(t *testing.T) {
 	})
 	config.ApplyDefaults(&cfg)
 	secret := "super-secret-token"
-	_, _, err = resolveUploadTokens(cfg, uploadHTTPTestEnvironment(map[string]string{
+	_, err = resolveUploadTokens(cfg, uploadHTTPTestEnvironment(map[string]string{
 		"UPLOAD_TOKEN":       secret,
 		"OTHER_UPLOAD_TOKEN": secret,
 	}))
@@ -110,7 +111,6 @@ func TestUploadHTTPHandlerAuthenticatesAndAcceptsUpload(t *testing.T) {
 			},
 		},
 		tokens: map[string]string{"valid-token": "reports"},
-		limits: map[string]int64{"reports": 1024},
 	}
 	recorder := httptest.NewRecorder()
 	request := httptest.NewRequest(http.MethodPost, "/upload", strings.NewReader("archive"))
@@ -141,7 +141,6 @@ func TestUploadHTTPHandlerRejectsUnauthorizedRequests(t *testing.T) {
 	handler := uploadHTTPHandler{
 		coordinator: fakeUploadCoordinator{canAccept: true},
 		tokens:      map[string]string{"valid-token": "reports"},
-		limits:      map[string]int64{"reports": 1024},
 	}
 
 	for _, authHeader := range []string{"", "Bearer wrong-token"} {
@@ -178,14 +177,6 @@ func TestUploadHTTPHandlerRejectsUnsupportedOversizedFullQueueAndPipelineID(t *t
 			body:        strings.NewReader("archive"),
 			wantStatus:  http.StatusUnsupportedMediaType,
 		},
-		{
-			name:        "oversized",
-			canAccept:   true,
-			url:         "/upload",
-			contentType: "application/x-tar",
-			body:        strings.NewReader("too-large"),
-			wantStatus:  http.StatusRequestEntityTooLarge,
-		},
 		{
 			name:        "full queue",
 			canAccept:   false,
@@ -214,7 +205,6 @@ func TestUploadHTTPHandlerRejectsUnsupportedOversizedFullQueueAndPipelineID(t *t
 					},
 				},
 				tokens: map[string]string{"valid-token": "reports"},
-				limits: map[string]int64{"reports": 4},
 			}
 			recorder := httptest.NewRecorder()
 			request := httptest.NewRequest(http.MethodPost, tt.url, tt.body)
@@ -233,6 +223,41 @@ func TestUploadHTTPHandlerRejectsUnsupportedOversizedFullQueueAndPipelineID(t *t
 	}
 }
 
+func TestUploadHTTPHandlerMapsSubmitErrors(t *testing.T) {
+	tests := []struct {
+		name       string
+		err        error
+		wantStatus int
+	}{
+		{name: "oversized", err: ingest.ErrUploadTooLarge, wantStatus: http.StatusRequestEntityTooLarge},
+		{name: "unsupported", err: ingest.ErrUnsupportedContentType, wantStatus: http.StatusUnsupportedMediaType},
+		{name: "malformed", err: errors.New("malformed archive"), wantStatus: http.StatusBadRequest},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			handler := uploadHTTPHandler{
+				coordinator: fakeUploadCoordinator{
+					canAccept: true,
+					submit: func(context.Context, UploadRequest) (UploadRunRecord, error) {
+						return UploadRunRecord{}, tt.err
+					},
+				},
+				tokens: map[string]string{"valid-token": "reports"},
+			}
+			recorder := httptest.NewRecorder()
+			request := httptest.NewRequest(http.MethodPost, "/upload", strings.NewReader("archive"))
+			request.Header.Set("Authorization", "Bearer valid-token")
+			request.Header.Set("Content-Type", "application/x-tar")
+
+			handler.ServeHTTP(recorder, request)
+
+			if recorder.Code != tt.wantStatus {
+				t.Fatalf("status = %d, want %d; body = %q", recorder.Code, tt.wantStatus, recorder.Body.String())
+			}
+		})
+	}
+}
+
 func TestUploadHTTPHandlerRunStatusAndHealth(t *testing.T) {
 	finishedAt := time.Date(2026, 6, 3, 12, 0, 0, 0, time.UTC)
 	handler := uploadHTTPHandler{
@@ -251,7 +276,6 @@ func TestUploadHTTPHandlerRunStatusAndHealth(t *testing.T) {
 			},
 		},
 		tokens: map[string]string{"valid-token": "reports"},
-		limits: map[string]int64{"reports": 1024},
 	}
 
 	recorder := httptest.NewRecorder()

internal/app/validate.go

@@ -6,7 +6,6 @@ import (
 	"io"
 
 	"gitea.maximumdirect.net/eric/distributor/internal/config"
-	"gitea.maximumdirect.net/eric/distributor/internal/storage"
 )
 
 type ValidateOptions struct {
@@ -73,29 +72,17 @@ func writeValidateResult(options ValidateOptions, selection sourceSelection) err
 }
 
 type validateResult struct {
-	PipelineID    string                 `json:"pipeline_id,omitempty"`
+	PipelineID    string                `json:"pipeline_id,omitempty"`
-	SourceBackend string                 `json:"source_backend,omitempty"`
+	SourceBackend string                `json:"source_backend,omitempty"`
-	BundleCount   int                    `json:"bundle_count"`
+	BundleCount   int                   `json:"bundle_count"`
-	Bundles       []validateBundleResult `json:"bundles"`
+	Bundles       []bundleSummaryResult `json:"bundles"`
-}
-
-type validateBundleResult struct {
-	Path string `json:"path"`
-	ID   string `json:"id"`
 }
 
 func validateResultFromSelection(selection sourceSelection) validateResult {
-	result := validateResult{
+	return validateResult{
 		PipelineID:    selection.PipelineID,
 		SourceBackend: selection.SourceBackend,
 		BundleCount:   len(selection.Bundles),
-		Bundles:       make([]validateBundleResult, 0, len(selection.Bundles)),
+		Bundles:       bundleSummariesFromBundles(selection.Bundles),
 	}
-	for _, sourceBundle := range selection.Bundles {
-		result.Bundles = append(result.Bundles, validateBundleResult{
-			Path: storage.DisplayPath(sourceBundle.RootRelativePath),
-			ID:   sourceBundle.Manifest.ID,
-		})
-	}
-	return result
 }

internal/app/validate_test.go

@@ -4,6 +4,7 @@ import (
 	"bytes"
 	"context"
 	"encoding/json"
+	"os"
 	"path/filepath"
 	"strings"
 	"testing"
@@ -183,6 +184,51 @@ pipelines:
 	}
 }
 
+func TestValidateConfiguredSourcePrintsSecretConflictWarningWithoutValues(t *testing.T) {
+	name := "DISTRIBUTOR_TEST_VALIDATE_SECRET"
+	t.Setenv(name, "process-value")
+	sourceRoot := t.TempDir()
+	destinationRoot := t.TempDir()
+	secretsRoot := t.TempDir()
+	if err := os.WriteFile(filepath.Join(secretsRoot, name), []byte("secret-value\n"), 0o600); err != nil {
+		t.Fatalf("write secret: %v", err)
+	}
+	testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
+	configPath := writeConfigFile(t, `
+secrets:
+  directory: `+secretsRoot+`
+pipelines:
+  - id: reports
+    source:
+      backend: local
+      path: `+sourceRoot+`
+    destinations:
+      - id: archive
+        backend: local
+        path: `+destinationRoot+`
+`)
+
+	var stdout bytes.Buffer
+	err := Validate(context.Background(), ValidateOptions{
+		ConfigPath: configPath,
+		PipelineID: "reports",
+		Stdout:     &stdout,
+	})
+	if err != nil {
+		t.Fatalf("Validate() error = %v", err)
+	}
+	output := stdout.String()
+	if !strings.Contains(output, "secret "+name+" ignored because the real environment already has that variable") {
+		t.Fatalf("stdout = %q, want secret conflict warning", output)
+	}
+	if !strings.Contains(output, "Validated 1 bundle(s) for pipeline reports source local") {
+		t.Fatalf("stdout = %q, want validate summary", output)
+	}
+	if strings.Contains(output, "process-value") || strings.Contains(output, "secret-value") {
+		t.Fatalf("stdout exposed secret values: %q", output)
+	}
+}
+
 func TestValidateConfiguredSourceRequiresPipeline(t *testing.T) {
 	sourceRoot := t.TempDir()
 	destinationRoot := t.TempDir()

internal/bundle/manifest_test.go

@@ -70,11 +70,19 @@ func TestParseManifestRejectsInvalidDigestFormat(t *testing.T) {
 
 func TestParseManifestRejectsUnsafeFilePaths(t *testing.T) {
 	tests := []string{
+		`"path": ""`,
+		`"path": "."`,
+		`"path": "./report.md"`,
 		`"path": "../report.md"`,
 		`"path": "/report.md"`,
 		`"path": "nested/../report.md"`,
+		`"path": "nested/./report.md"`,
+		`"path": "nested//report.md"`,
+		`"path": "nested\\report.md"`,
 		`"path": "manifest.json"`,
+		`"path": "nested/manifest.json"`,
 		`"path": "` + storage.StateFileName + `"`,
+		`"path": "nested/` + storage.StateFileName + `"`,
 	}
 	for _, replacement := range tests {
 		t.Run(replacement, func(t *testing.T) {
@@ -128,6 +136,16 @@ func TestValidateManifestRejectsInvalidManifest(t *testing.T) {
 			manifest.Digest = BundleDigest(manifest.Files)
 			return manifest
 		},
+		"nested manifest path": func(manifest Manifest) Manifest {
+			manifest.Files[0].Path = "nested/manifest.json"
+			manifest.Digest = BundleDigest(manifest.Files)
+			return manifest
+		},
+		"nested state path": func(manifest Manifest) Manifest {
+			manifest.Files[0].Path = "nested/" + storage.StateFileName
+			manifest.Digest = BundleDigest(manifest.Files)
+			return manifest
+		},
 		"duplicate path": func(manifest Manifest) Manifest {
 			manifest.Files[1].Path = manifest.Files[0].Path
 			manifest.Digest = BundleDigest(manifest.Files)

internal/bundle/validate_test.go

@@ -65,6 +65,38 @@ func TestValidateRejectsSymlinkFile(t *testing.T) {
 	assertErrorContains(t, err, "regular file")
 }
 
+func TestValidateRejectsUnsafeManifestPaths(t *testing.T) {
+	tests := []string{
+		"",
+		".",
+		"./report.md",
+		"../report.md",
+		"/report.md",
+		"nested/../report.md",
+		"nested/./report.md",
+		"nested//report.md",
+		`nested\report.md`,
+		ManifestName,
+		storage.StateFileName,
+		"nested/" + ManifestName,
+		"nested/" + storage.StateFileName,
+	}
+	for _, path := range tests {
+		t.Run(path, func(t *testing.T) {
+			backend := validFakeBundle(t)
+			manifest := validFixtureManifest(t)
+			manifest.Files[0].Path = path
+			manifest.Digest = BundleDigest(manifest.Files)
+			writeManifest(t, backend, manifest)
+
+			_, err := Validate(context.Background(), backend, "")
+			if err == nil {
+				t.Fatal("Validate() error = nil, want unsafe path error")
+			}
+		})
+	}
+}
+
 func validFakeBundle(t *testing.T) *fake.Backend {
 	t.Helper()
 	backend := fake.New()

internal/cli/flags.go

@@ -0,0 +1,12 @@
+package cli
+
+import (
+	"flag"
+	"io"
+)
+
+func newFlagSet(name string, stderr io.Writer) *flag.FlagSet {
+	flags := flag.NewFlagSet(name, flag.ContinueOnError)
+	flags.SetOutput(stderr)
+	return flags
+}

internal/cli/manifest.go

@@ -2,7 +2,6 @@ package cli
 
 import (
 	"context"
-	"flag"
 	"fmt"
 	"io"
 	"strings"
@@ -30,8 +29,7 @@ func manifestCreateCommand(ctx context.Context, args []string, stdout, stderr io
 		printManifestCreateHelp(stdout)
 		return exitOK
 	}
-	flags := flag.NewFlagSet("manifest create", flag.ContinueOnError)
+	flags := newFlagSet("manifest create", stderr)
-	flags.SetOutput(stderr)
 	id := flags.String("id", "", "source bundle id")
 	created := flags.String("created", "", "source created timestamp")
 	overwrite := flags.Bool("overwrite", false, "replace an existing manifest.json")

internal/cli/run.go

@@ -2,7 +2,6 @@ package cli
 
 import (
 	"context"
-	"flag"
 	"fmt"
 	"io"
 
@@ -15,8 +14,7 @@ func runCommand(ctx context.Context, args []string, stdout, stderr io.Writer) in
 		return exitOK
 	}
 
-	flags := flag.NewFlagSet("run", flag.ContinueOnError)
+	flags := newFlagSet("run", stderr)
-	flags.SetOutput(stderr)
 	configPath := flags.String("config", "", "path to config file")
 	dryRun := flags.Bool("dry-run", false, "load and validate config without publishing")
 	force := flags.Bool("force", false, "allow explicit destructive replacement for supported conflicts")

internal/cli/serve.go

@@ -2,7 +2,6 @@ package cli
 
 import (
 	"context"
-	"flag"
 	"fmt"
 	"io"
 
@@ -17,8 +16,7 @@ func serveCommand(ctx context.Context, args []string, stdout, stderr io.Writer)
 		return exitOK
 	}
 
-	flags := flag.NewFlagSet("serve", flag.ContinueOnError)
+	flags := newFlagSet("serve", stderr)
-	flags.SetOutput(stderr)
 	configPath := flags.String("config", "", "path to config file")
 	if err := flags.Parse(args); err != nil {
 		return exitUsage

internal/cli/source_mode.go

@@ -1,7 +1,6 @@
 package cli
 
 import (
-	"flag"
 	"fmt"
 	"io"
 
@@ -17,8 +16,7 @@ type sourceDiagnosticArgs struct {
 }
 
 func parseSourceDiagnosticArgs(stderr io.Writer, command string, args []string) (sourceDiagnosticArgs, bool) {
-	flags := flag.NewFlagSet(command, flag.ContinueOnError)
+	flags := newFlagSet(command, stderr)
-	flags.SetOutput(stderr)
 	configPath := flags.String("config", "", "path to config file")
 	pipelineID := flags.String("pipeline", "", "pipeline id")
 	bundlePath := flags.String("bundle", "", "source-root-relative bundle path")

internal/cli/version.go

@@ -2,7 +2,6 @@ package cli
 
 import (
 	"context"
-	"flag"
 	"fmt"
 	"io"
 
@@ -14,8 +13,7 @@ func versionCommand(_ context.Context, args []string, stdout, stderr io.Writer)
 		printVersionHelp(stdout)
 		return exitOK
 	}
-	flags := flag.NewFlagSet("version", flag.ContinueOnError)
+	flags := newFlagSet("version", stderr)
-	flags.SetOutput(stderr)
 	formatFlag := addFormatFlag(flags)
 	if err := flags.Parse(args); err != nil {
 		return exitUsage

internal/config/backend_view.go

@@ -0,0 +1,50 @@
+package config
+
+type backendView struct {
+	Backend   string
+	Host      string
+	User      string
+	Port      int
+	Path      string
+	Endpoint  string
+	Bucket    string
+	Prefix    string
+	Region    string
+	ForcePath *bool
+	Creds     Credentials
+	SSH       SSH
+}
+
+func backendViewFromSource(source Backend) backendView {
+	return backendView{
+		Backend:   source.Backend,
+		Host:      source.Host,
+		User:      source.User,
+		Port:      source.Port,
+		Path:      source.Path,
+		Endpoint:  source.Endpoint,
+		Bucket:    source.Bucket,
+		Prefix:    source.Prefix,
+		Region:    source.Region,
+		ForcePath: source.ForcePath,
+		Creds:     source.Creds,
+		SSH:       source.SSH,
+	}
+}
+
+func backendViewFromDestination(destination Destination) backendView {
+	return backendView{
+		Backend:   destination.Backend,
+		Host:      destination.Host,
+		User:      destination.User,
+		Port:      destination.Port,
+		Path:      destination.Path,
+		Endpoint:  destination.Endpoint,
+		Bucket:    destination.Bucket,
+		Prefix:    destination.Prefix,
+		Region:    destination.Region,
+		ForcePath: destination.ForcePath,
+		Creds:     destination.Creds,
+		SSH:       destination.SSH,
+	}
+}

internal/config/backend_view_test.go

@@ -0,0 +1,187 @@
+package config
+
+import "testing"
+
+func TestBackendViewsPreserveEquivalentStorageFields(t *testing.T) {
+	forcePathStyle := false
+	source := Backend{
+		Backend:   BackendS3,
+		Host:      "storage.example.com",
+		User:      "reports",
+		Port:      2222,
+		Path:      "/reports",
+		Endpoint:  "https://s3.example.com",
+		Bucket:    "source",
+		Prefix:    "incoming",
+		Region:    "us-west-2",
+		ForcePath: &forcePathStyle,
+		Creds: Credentials{
+			AccessKeyIDEnv:     "ACCESS_KEY_ID",
+			SecretAccessKeyEnv: "SECRET_ACCESS_KEY",
+		},
+		SSH: SSH{
+			KeyFile:       "/home/reports/.ssh/id_ed25519",
+			KnownHosts:    "/home/reports/.ssh/known_hosts",
+			HostKeyPolicy: HostKeyPolicyStrict,
+		},
+	}
+	destination := Destination{
+		Backend:   source.Backend,
+		Host:      source.Host,
+		User:      source.User,
+		Port:      source.Port,
+		Path:      source.Path,
+		Endpoint:  source.Endpoint,
+		Bucket:    source.Bucket,
+		Prefix:    source.Prefix,
+		Region:    source.Region,
+		ForcePath: source.ForcePath,
+		Creds:     source.Creds,
+		SSH:       source.SSH,
+	}
+
+	sourceView := backendViewFromSource(source)
+	destinationView := backendViewFromDestination(destination)
+
+	if sourceView != destinationView {
+		t.Fatalf("source view = %#v, destination view = %#v, want equivalent storage fields", sourceView, destinationView)
+	}
+}
+
+func TestBackendViewValidationKeepsHTTPUploadSourceOnly(t *testing.T) {
+	cfg := Config{Pipelines: []Pipeline{{
+		ID: "reports",
+		Source: Backend{
+			Backend: BackendHTTPUpload,
+			Upload:  HTTPUpload{TokenEnv: "UPLOAD_TOKEN"},
+		},
+		Destinations: []Destination{{
+			ID:      "archive",
+			Backend: BackendHTTPUpload,
+		}},
+	}}}
+	ApplyDefaults(&cfg)
+
+	err := Validate(cfg)
+	if err == nil {
+		t.Fatal("Validate() error = nil, want destination http_upload error")
+	}
+	if got, want := err.Error(), "pipelines[0].destinations[0].backend http_upload is only supported for sources"; got != want {
+		t.Fatalf("Validate() error = %q, want %q", got, want)
+	}
+}
+
+func TestBackendViewValidationAppliesStorageRulesToSourcesAndDestinations(t *testing.T) {
+	forcePathStyle := false
+	tests := []struct {
+		name        string
+		source      Backend
+		destination Destination
+		wantErr     bool
+	}{
+		{
+			name: "local valid",
+			source: Backend{
+				Backend: BackendLocal,
+				Path:    "/source",
+			},
+			destination: Destination{
+				Backend: BackendLocal,
+				Path:    "/destination",
+			},
+		},
+		{
+			name: "local missing path",
+			source: Backend{
+				Backend: BackendLocal,
+			},
+			destination: Destination{
+				Backend: BackendLocal,
+			},
+			wantErr: true,
+		},
+		{
+			name: "ssh valid",
+			source: Backend{
+				Backend: BackendSSH,
+				Host:    "source.example.com",
+				Port:    22,
+				Path:    "/source",
+				SSH:     SSH{HostKeyPolicy: HostKeyPolicyAcceptNew},
+			},
+			destination: Destination{
+				Backend: BackendSSH,
+				Host:    "destination.example.com",
+				Port:    22,
+				Path:    "/destination",
+				SSH:     SSH{HostKeyPolicy: HostKeyPolicyAcceptNew},
+			},
+		},
+		{
+			name: "ssh missing host",
+			source: Backend{
+				Backend: BackendSSH,
+				Port:    22,
+				Path:    "/source",
+				SSH:     SSH{HostKeyPolicy: HostKeyPolicyAcceptNew},
+			},
+			destination: Destination{
+				Backend: BackendSSH,
+				Port:    22,
+				Path:    "/destination",
+				SSH:     SSH{HostKeyPolicy: HostKeyPolicyAcceptNew},
+			},
+			wantErr: true,
+		},
+		{
+			name: "s3 valid",
+			source: Backend{
+				Backend:   BackendS3,
+				Endpoint:  "https://s3.example.com",
+				Bucket:    "source",
+				Prefix:    "incoming",
+				Region:    DefaultS3Region,
+				ForcePath: &forcePathStyle,
+			},
+			destination: Destination{
+				Backend:   BackendS3,
+				Endpoint:  "https://s3.example.com",
+				Bucket:    "destination",
+				Prefix:    "archive",
+				Region:    DefaultS3Region,
+				ForcePath: &forcePathStyle,
+			},
+		},
+		{
+			name: "s3 partial credentials",
+			source: Backend{
+				Backend:  BackendS3,
+				Endpoint: "https://s3.example.com",
+				Bucket:   "source",
+				Region:   DefaultS3Region,
+				Creds:    Credentials{AccessKeyIDEnv: "ACCESS_KEY_ID"},
+			},
+			destination: Destination{
+				Backend:  BackendS3,
+				Endpoint: "https://s3.example.com",
+				Bucket:   "destination",
+				Region:   DefaultS3Region,
+				Creds:    Credentials{AccessKeyIDEnv: "ACCESS_KEY_ID"},
+			},
+			wantErr: true,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			sourceErrors := validateBackend(nil, "source", backendViewFromSource(tt.source))
+			destinationErrors := validateBackend(nil, "destination", backendViewFromDestination(tt.destination))
+			if got := len(sourceErrors) > 0; got != tt.wantErr {
+				t.Fatalf("source validation errors = %#v, wantErr %t", sourceErrors, tt.wantErr)
+			}
+			if got := len(destinationErrors) > 0; got != tt.wantErr {
+				t.Fatalf("destination validation errors = %#v, wantErr %t", destinationErrors, tt.wantErr)
+			}
+		})
+	}
+}

internal/config/defaults.go

@@ -134,30 +134,24 @@ func duration(value Duration) *Duration {
 }
 
 func applyBackendDefaults(backend *Backend) {
-	if backend.Backend == BackendSSH {
+	applyStorageBackendDefaults(backend.Backend, &backend.Port, &backend.SSH, &backend.Region, &backend.Prefix, &backend.ForcePath)
-		if backend.Port == 0 {
-			backend.Port = 22
-		}
-		if backend.SSH.HostKeyPolicy == "" {
-			backend.SSH.HostKeyPolicy = HostKeyPolicyAcceptNew
-		}
-	}
-	if backend.Backend == BackendS3 {
-		applyS3Defaults(&backend.Region, &backend.Prefix, &backend.ForcePath)
-	}
 }
 
 func applyDestinationDefaults(destination *Destination) {
-	if destination.Backend == BackendSSH {
+	applyStorageBackendDefaults(destination.Backend, &destination.Port, &destination.SSH, &destination.Region, &destination.Prefix, &destination.ForcePath)
-		if destination.Port == 0 {
+}
-			destination.Port = 22
+
+func applyStorageBackendDefaults(backend string, port *int, ssh *SSH, region, prefix *string, forcePath **bool) {
+	if backend == BackendSSH {
+		if *port == 0 {
+			*port = 22
 		}
-		if destination.SSH.HostKeyPolicy == "" {
+		if ssh.HostKeyPolicy == "" {
-			destination.SSH.HostKeyPolicy = HostKeyPolicyAcceptNew
+			ssh.HostKeyPolicy = HostKeyPolicyAcceptNew
 		}
 	}
-	if destination.Backend == BackendS3 {
+	if backend == BackendS3 {
-		applyS3Defaults(&destination.Region, &destination.Prefix, &destination.ForcePath)
+		applyS3Defaults(region, prefix, forcePath)
 	}
 }
 

internal/config/validate.go

@@ -100,7 +100,7 @@ func validateSourceBackend(errs ValidationErrors, context string, backend Backen
 	if backend.Backend == BackendHTTPUpload {
 		return validateHTTPUploadSource(errs, context, backend.Upload)
 	}
-	return validateBackend(errs, context, backend.Backend, backend.Host, backend.Port, backend.Path, backend.Endpoint, backend.Bucket, backend.Prefix, backend.SSH.HostKeyPolicy, backend.Creds)
+	return validateBackend(errs, context, backendViewFromSource(backend))
 }
 
 func validateDestinationBackend(errs ValidationErrors, context string, destination Destination) ValidationErrors {
@@ -108,7 +108,7 @@ func validateDestinationBackend(errs ValidationErrors, context string, destinati
 		errs = append(errs, context+".backend "+BackendHTTPUpload+" is only supported for sources")
 		return errs
 	}
-	return validateBackend(errs, context, destination.Backend, destination.Host, destination.Port, destination.Path, destination.Endpoint, destination.Bucket, destination.Prefix, destination.SSH.HostKeyPolicy, destination.Creds)
+	return validateBackend(errs, context, backendViewFromDestination(destination))
 }
 
 func validateHTTPUploadSource(errs ValidationErrors, context string, upload HTTPUpload) ValidationErrors {
@@ -124,47 +124,47 @@ func validateHTTPUploadSource(errs ValidationErrors, context string, upload HTTP
 	return errs
 }
 
-func validateBackend(errs ValidationErrors, context, backend, host string, port int, path, endpoint, bucket, prefix string, hostKeyPolicy HostKeyPolicy, creds Credentials) ValidationErrors {
+func validateBackend(errs ValidationErrors, context string, backend backendView) ValidationErrors {
-	switch backend {
+	switch backend.Backend {
 	case "":
 		errs = append(errs, context+".backend is required")
 	case BackendLocal:
-		if path == "" {
+		if backend.Path == "" {
 			errs = append(errs, context+".path is required for local backend")
 		}
 	case BackendSSH:
-		if host == "" {
+		if backend.Host == "" {
 			errs = append(errs, context+".host is required for ssh backend")
 		}
-		if path == "" {
+		if backend.Path == "" {
 			errs = append(errs, context+".path is required for ssh backend")
 		}
-		if port < 0 || port > 65535 {
+		if backend.Port < 0 || backend.Port > 65535 {
 			errs = append(errs, context+".port must be between 1 and 65535")
 		}
-		if port == 0 {
+		if backend.Port == 0 {
 			errs = append(errs, context+".port is required for ssh backend after defaults are applied")
 		}
-		if hostKeyPolicy != "" {
+		if backend.SSH.HostKeyPolicy != "" {
-			if _, ok := NormalizeHostKeyPolicy(string(hostKeyPolicy)); !ok {
+			if _, ok := NormalizeHostKeyPolicy(string(backend.SSH.HostKeyPolicy)); !ok {
 				errs = append(errs, context+".host_key_policy must be strict, true, accept-new, off, or false")
 			}
 		}
 	case BackendS3:
-		if endpoint == "" {
+		if backend.Endpoint == "" {
 			errs = append(errs, context+".endpoint is required for s3 backend")
 		}
-		if bucket == "" {
+		if backend.Bucket == "" {
 			errs = append(errs, context+".bucket is required for s3 backend")
 		}
-		if err := ValidateS3Prefix(prefix); err != nil {
+		if err := ValidateS3Prefix(backend.Prefix); err != nil {
 			errs = append(errs, context+".prefix must be a clean relative slash-separated path")
 		}
-		if (creds.AccessKeyIDEnv == "") != (creds.SecretAccessKeyEnv == "") {
+		if (backend.Creds.AccessKeyIDEnv == "") != (backend.Creds.SecretAccessKeyEnv == "") {
 			errs = append(errs, context+".credentials.access_key_id_env and credentials.secret_access_key_env must be configured together")
 		}
 	default:
-		errs = append(errs, context+".backend "+backend+" is unsupported")
+		errs = append(errs, context+".backend "+backend.Backend+" is unsupported")
 	}
 	return errs
 }

internal/ingest/archive.go

@@ -138,6 +138,11 @@ func validateRunID(value string) error {
 	return nil
 }
 
+func ValidateContentType(contentType string) error {
+	_, err := archiveFormat(contentType)
+	return err
+}
+
 type archiveKind int
 
 const (

internal/ingest/archive_test.go

@@ -5,6 +5,7 @@ import (
 	"bytes"
 	"compress/gzip"
 	"context"
+	"encoding/json"
 	"errors"
 	"io/fs"
 	"os"
@@ -47,6 +48,25 @@ func TestStageArchiveRejectsUnsupportedContentType(t *testing.T) {
 	}
 }
 
+func TestValidateContentType(t *testing.T) {
+	for _, contentType := range []string{
+		ContentTypeTar,
+		ContentTypeGzip,
+		ContentTypeXGzip,
+		ContentTypeGzip + "; charset=binary",
+	} {
+		t.Run(contentType, func(t *testing.T) {
+			if err := ValidateContentType(contentType); err != nil {
+				t.Fatalf("ValidateContentType() error = %v", err)
+			}
+		})
+	}
+
+	if err := ValidateContentType("application/zip"); !errors.Is(err, ErrUnsupportedContentType) {
+		t.Fatalf("ValidateContentType() error = %v, want ErrUnsupportedContentType", err)
+	}
+}
+
 func TestStageArchiveEnforcesMaxUploadSize(t *testing.T) {
 	archive := validArchive(t, false)
 	err := stageArchiveError(t, archive, ContentTypeTar, func(opts *StageOptions) {
@@ -94,9 +114,19 @@ func TestStageArchiveRejectsUnsafeEntries(t *testing.T) {
 		"path traversal": {
 			fileEntry("../report.md", "report"),
 		},
+		"dot path": {
+			fileEntry("./report.md", "report"),
+		},
+		"dot segment": {
+			fileEntry("nested/./report.md", "report"),
+		},
 		"backslash path": {
 			fileEntry(`nested\report.md`, "report"),
 		},
+		"duplicate file": {
+			fileEntry("report.md", "report"),
+			fileEntry("report.md", "report"),
+		},
 		"symlink": {
 			{name: "link.md", typeflag: tar.TypeSymlink, linkname: "report.md"},
 		},
@@ -106,6 +136,12 @@ func TestStageArchiveRejectsUnsafeEntries(t *testing.T) {
 		"device": {
 			{name: "device", typeflag: tar.TypeChar},
 		},
+		"fifo": {
+			{name: "socket", typeflag: tar.TypeFifo},
+		},
+		"socket": {
+			{name: "socket", typeflag: 'S'},
+		},
 	}
 	for name, entries := range tests {
 		t.Run(name, func(t *testing.T) {
@@ -127,6 +163,14 @@ func TestStageArchiveRejectsBundleValidationFailures(t *testing.T) {
 			fileEntry("nested/manifest.json", "{}"),
 			fileEntry("report.md", "report"),
 		},
+		"listed nested manifest": {
+			fileEntry("manifest.json", uncheckedManifestJSON(t, manifestFor("reports.listed.nested", fileSpec{path: "nested/manifest.json", body: "{}"}))),
+			fileEntry("nested/manifest.json", "{}"),
+		},
+		"listed state file": {
+			fileEntry("manifest.json", uncheckedManifestJSON(t, manifestFor("reports.listed.state", fileSpec{path: ".distributor.json", body: "{}"}))),
+			fileEntry(".distributor.json", "{}"),
+		},
 		"missing listed file": {
 			fileEntry("manifest.json", manifestJSON(t, manifestFor("reports.missing", fileSpec{path: "missing.md", body: "missing"}))),
 		},
@@ -149,6 +193,20 @@ func TestStageArchiveRejectsBundleValidationFailures(t *testing.T) {
 	}
 }
 
+func TestStageArchiveAcceptsSafeDirectories(t *testing.T) {
+	archive := makeArchive(t, false,
+		tarEntry{name: "nested", typeflag: tar.TypeDir},
+		tarEntry{name: "nested/assets", typeflag: tar.TypeDir},
+		fileEntry("manifest.json", manifestJSON(t, manifestFor("reports.directories", fileSpec{path: "nested/assets/report.md", body: "report"}))),
+		fileEntry("nested/assets/report.md", "report"),
+	)
+	staged := stageArchive(t, archive, ContentTypeTar)
+
+	if got := readFile(t, staged.Root, "nested/assets/report.md"); got != "report" {
+		t.Fatalf("report = %q", got)
+	}
+}
+
 func TestStageArchiveCleansUpFailedExtraction(t *testing.T) {
 	stagingPath := filepath.Join(t.TempDir(), "staging")
 	archive := makeArchive(t, false, fileEntry("../report.md", "report"))
@@ -336,6 +394,15 @@ func manifestJSON(t *testing.T, manifest sourcebundle.Manifest) string {
 	return string(data)
 }
 
+func uncheckedManifestJSON(t *testing.T, manifest sourcebundle.Manifest) string {
+	t.Helper()
+	data, err := json.MarshalIndent(manifest, "", "  ")
+	if err != nil {
+		t.Fatalf("MarshalIndent() error = %v", err)
+	}
+	return string(append(data, '\n'))
+}
+
 func writeFile(t *testing.T, root, relative, body string) {
 	t.Helper()
 	fullPath := filepath.Join(root, filepath.FromSlash(relative))

internal/storage/path_test.go

@@ -12,6 +12,10 @@ func TestValidatePath(t *testing.T) {
 		"report.md",
 		"daily/report.md",
 		"a-b_1.2/report.html",
+		"manifest.json",
+		StateFileName,
+		"nested/manifest.json",
+		"nested/" + StateFileName,
 	}
 	for _, path := range valid {
 		t.Run("valid "+path, func(t *testing.T) {
@@ -23,10 +27,14 @@ func TestValidatePath(t *testing.T) {
 
 	invalid := []string{
 		"",
+		".",
+		"./report.md",
 		"/absolute",
 		"../outside",
 		"nested/../outside",
+		"nested/.",
 		"nested/./file",
+		"nested/",
 		"nested//file",
 		`nested\file`,
 	}

pkg/bundle/bundle_test.go

@@ -90,17 +90,33 @@ func TestBuildManifestRequiresOneFileMode(t *testing.T) {
 	}
 }
 
-func TestBuildManifestRejectsUnsafePath(t *testing.T) {
+func TestBuildManifestRejectsUnsafeExplicitPaths(t *testing.T) {
 	root := t.TempDir()
 	writeFile(t, root, "report.txt", "report")
 
-	_, err := BuildManifest(BuildOptions{
+	tests := []string{
-		Root:  root,
+		"",
-		ID:    "reports.unsafe",
+		"../report.txt",
-		Files: []string{"../report.txt"},
+		"/report.txt",
-	})
+		"nested/../report.txt",
-	if err == nil {
+		"nested/./report.txt",
-		t.Fatal("BuildManifest() error = nil, want unsafe path error")
+		`nested\report.txt`,
+		ManifestName,
+		distributorStateName,
+		"nested/" + ManifestName,
+		"nested/" + distributorStateName,
+	}
+	for _, sourcePath := range tests {
+		t.Run(sourcePath, func(t *testing.T) {
+			_, err := BuildManifest(BuildOptions{
+				Root:  root,
+				ID:    "reports.unsafe",
+				Files: []string{sourcePath},
+			})
+			if err == nil {
+				t.Fatal("BuildManifest() error = nil, want unsafe path error")
+			}
+		})
 	}
 }
 
@@ -330,7 +346,21 @@ func TestValidateSourcePath(t *testing.T) {
 			t.Fatalf("ValidateSourcePath(%q) error = %v", path, err)
 		}
 	}
-	invalid := []string{"", "../report.md", "/report.md", "nested/../report.md", `nested\report.md`, ManifestName, distributorStateName}
+	invalid := []string{
+		"",
+		".",
+		"./report.md",
+		"../report.md",
+		"/report.md",
+		"nested/../report.md",
+		"nested/./report.md",
+		"nested//report.md",
+		`nested\report.md`,
+		ManifestName,
+		distributorStateName,
+		"nested/" + ManifestName,
+		"nested/" + distributorStateName,
+	}
 	for _, path := range invalid {
 		if err := ValidateSourcePath(path); err == nil {
 			t.Fatalf("ValidateSourcePath(%q) error = nil, want error", path)

pkg/bundle/path.go

@@ -23,8 +23,7 @@ func ValidateSourcePath(value string) error {
 			return fmt.Errorf("source path %q must be a clean relative slash-separated path", value)
 		}
 	}
-	switch value {
+	if path.Base(value) == ManifestName || path.Base(value) == distributorStateName {
-	case ManifestName, distributorStateName:
 		return fmt.Errorf("%q is reserved", value)
 	}
 	return nil