eric/distributor

Fork 0

Files

Eric Rakestraw 0d346dcdf5 Stage uploads before accepting HTTP runs

2026-06-04 00:14:32 +00:00

18 KiB

Raw Blame History

Distributor Operations

Normal Workflow

Validate a source bundle:

go run ./cmd/distributor validate examples/source-bundle

Preview a local publication:

go run ./cmd/distributor run --config examples/local-publish.yml --dry-run

Run the local publication:

go run ./cmd/distributor run --config examples/local-publish.yml

Run the local HTML publication:

go run ./cmd/distributor run --config examples/local-html.yml

Run the local index.html publication:

go run ./cmd/distributor run --config examples/local-index.yml

Preview local fan-out publication:

go run ./cmd/distributor run --config examples/fan-out.yml --dry-run

Preview local archive-plus-latest publication:

go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run

Preview a run for automation:

go run ./cmd/distributor run --config examples/fan-out.yml --dry-run --format json

Preview an environment-gated SSH destination config after editing it for an SSH/SFTP endpoint you control:

go run ./cmd/distributor run --config examples/ssh-destination.yml --dry-run

Preview an environment-gated S3 destination config after editing it for an S3-compatible endpoint and bucket you control:

go run ./cmd/distributor run --config examples/s3-destination.yml --dry-run

Validate one configured source without opening destinations:

go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle

HTTP Upload Workflow

distributor serve runs the HTTP upload API for pipelines whose source backend is http_upload. Each upload token maps to one configured pipeline. A valid archive is staged and validated before a run id is returned, then published through the same destination fan-out path used by local source runs.

Minimal local HTTP upload configuration:

server:
  http:
    bind: 127.0.0.1:8080
    staging_root: /var/spool/distributor
    max_upload_size: 20MB
    queue_size: 16
    max_concurrency: 1
    retention: 24h
secrets:
  directory: /run/secrets/distributor
pipelines:
  - id: reports
    source:
      backend: http_upload
      token_env: DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN
    destinations:
      - id: archive
        backend: local
        path: /srv/reports/archive

Create /run/secrets/distributor/DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN or set the real process environment variable before starting the server. Distributor does not read literal upload tokens from YAML.

Start the maintained local example:

DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN=<token> \
  go run ./cmd/distributor serve --config examples/http-upload-local.yml

Submit a tar or tar.gz source bundle:

curl -X POST http://127.0.0.1:8080/upload \
  -H "Authorization: Bearer $DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN" \
  -H "Content-Type: application/gzip" \
  --data-binary @bundle.tar.gz

Successful staging and admission returns a run id:

{"run_id":"reports.20260603T120000Z.abcdef12","status":"accepted"}

Poll status until it reaches succeeded or failed:

curl http://127.0.0.1:8080/runs/<run-id>

The status record includes the completed run report on successful publication or error details on failure. Status is memory-only and expires after server.http.retention; completed staged bundle directories are removed on expiry. Restarting the process clears upload status and queue state.

Malformed archives and invalid source bundles are rejected by POST /upload before a run id is issued.

Use GET /healthz for readiness after config and tokens load:

curl http://127.0.0.1:8080/healthz

The default bind address is private loopback. Put TLS, public routing, rate-limiting, and external access policy in a reverse proxy or deployment layer.

Filesystem Layout

Source bundles are discovered beneath the configured source root. Each bundle is a directory containing manifest.json.

Destination bundle paths are configured per destination with path_mapping.mode.

The default mode, preserve_relative, preserves the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under daily/ publishes under daily/ at that destination.

The fixed mode publishes one selected source bundle at the destination backend root. A fixed destination with local path: /srv/www/reports/latest writes outputs and .distributor.json directly under /srv/www/reports/latest. Fixed destinations select the newest discovered source bundle by manifest created timestamp, with the source-root-relative bundle path as the deterministic tie-breaker.

The maintained local examples write under workspace/, which is ignored by Git.

SSH backends use the configured remote path as the backend root. Source bundle discovery and destination bundle paths are relative to that root, using the same logical path rules as local storage.

S3 backends use the configured bucket plus optional prefix as the backend root. Source bundle discovery and destination bundle paths are relative to that object-key prefix. Prefixes are object-key prefixes, not real directories.

Destination State

Each published destination bundle contains .distributor.json. This file is the managed sentinel and destination state record. It stores:

pipeline and destination identity;
publication timestamp;
source manifest used for publication;
copied source output metadata;
generated output metadata;
optional public URL metadata when destination links are configured.

manifest.json from the source bundle is not copied as destination state.

Do not edit .distributor.json by hand during normal operation. If it is missing or invalid while destination files remain, distributor treats the destination as unmanaged or conflicted.

Go Producer Bundles

Go producer applications can import gitea.maximumdirect.net/eric/distributor/pkg/bundle to create complete local source bundles with the same path, digest, timestamp, and validation rules used by distributor. The package also exposes digest helpers, including ValidateDigest, for producer code that needs to validate lowercase sha256:<64 hex> strings before writing manifests.

Minimal producer-side bundle creation:

manifest, err := bundle.WriteBundle(bundle.WriteBundleOptions{
    Root:  outputDir,
    ID:    "reports.example.2026-05-30",
    Files: []bundle.BundleFile{
        {SourcePath: reportPath, Path: "report.md"},
        {SourcePath: summaryPath, Path: "summary.txt"},
    },
})
if err != nil {
    return err
}

WriteBundle copies local producer files into a sibling temporary directory, writes manifest.json, validates the result, and promotes the completed bundle into place. It fails if Root already exists unless Overwrite is true. With overwrite enabled, it builds and validates the replacement before moving the existing root aside.

Use BuildManifest and WriteManifest when a producer already wrote all bundle files into the final root. BuildManifest can preserve an explicit file order, or Scan: true can recursively include regular files under Root in deterministic slash-path order. Scan mode includes dotfiles, excludes files named manifest.json or .distributor.json, and rejects symlinks.

Shell producers can create the same manifest through the CLI after writing bundle files:

go run ./cmd/distributor manifest create <bundle-path> --id reports.example.2026-05-30
go run ./cmd/distributor validate <bundle-path>

Use repeated --file flags to preserve a specific file order. If no --file flags are provided, the command scans the bundle directory recursively using the same filtering rules as pkg/bundle.BuildManifest.

Static HTML Publication

Markdown-to-HTML publication can write sidecar files or a fixed index.html.

Use sidecar mode when each Markdown source should keep a matching HTML filename:

publish:
  source: false
  html: true
transform:
  markdown_to_html:
    enabled: true
    mode: sidecar

Use index mode for static-site destinations that should serve a bundle through index.html:

publish:
  source: false
  html: true
transform:
  markdown_to_html:
    enabled: true
    mode: index
    input: report.md

If input is omitted in index mode, the source manifest must list exactly one Markdown file. Generated HTML is recorded in .distributor.json with kind: generated, source_path, transform: markdown_to_html, digest, and size metadata.

Archive And Latest Fan-Out

A pipeline can publish the same source to an archive destination and a stable latest destination:

pipelines:
  - id: reports
    source:
      backend: local
      path: /var/spool/distributor/reports
    destinations:
      - id: archive
        backend: local
        path: /srv/reports/archive
        path_mapping:
          mode: preserve_relative
        publish:
          source: true
          html: false
      - id: latest-html
        backend: local
        path: /srv/www/reports/latest
        path_mapping:
          mode: fixed
        links:
          base_url: https://reports.example.com/latest
          primary: auto
        publish:
          source: false
          html: true
        transform:
          markdown_to_html:
            enabled: true
            mode: index
            input: report.md

The archive destination plans every discovered source bundle at its source-relative path. The fixed latest destination plans only the newest discovered bundle and writes index.html plus .distributor.json at its backend root.

Static Site URLs

Use destination links when a destination backend root corresponds to a public HTTP or HTTPS URL:

links:
  base_url: https://reports.example.com/archive
  primary: auto

Distributor records URLs in .distributor.json; it does not publish notifications or infer URLs from local, SSH, or S3 backend fields.

For archive-style destinations, URLs include the destination bundle path. A source bundle under daily/brentwood/2026-06-01 with base_url: https://reports.example.com/archive can produce:

https://reports.example.com/archive/daily/brentwood/2026-06-01/report.html

For fixed destinations, URLs are rooted at links.base_url. A fixed HTML index destination with base_url: https://reports.example.com/latest records:

https://reports.example.com/latest/

index.html outputs use directory-style URLs. Other outputs include their filename. The primary URL is selected from the published outputs using the destination links.primary policy.

Source Validation and Inspection

validate and inspect can operate on a local path or on one configured pipeline source. Configured source mode requires both --config and --pipeline; it loads the normal config, resolves secrets.directory, opens only the selected source backend, and does not open any destinations.

Configured source validation is useful when producers write directly to SSH or S3 storage:

go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
go run ./cmd/distributor inspect --config <config-path> --pipeline <pipeline-id>

Use --bundle <path> to validate or inspect one source-root-relative bundle directory:

go run ./cmd/distributor validate \
  --config <config-path> \
  --pipeline <pipeline-id> \
  --bundle daily/2026-06-01

For configured SSH sources, host key and authentication behavior matches run. For configured S3 sources, endpoint, bucket, prefix, region, path-style, explicit credential environment variables, and secrets.directory handling match run.

Dry Runs

--dry-run loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files, destination state, or SSH known_hosts entries.

Dry-run output is useful before publishing to confirm actions such as publish_new, replace_older, force_replace, skip_same, and skip_destination_newer.

Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication. Fixed path destinations add path_mapping=fixed target=. to planned action lines. Dry-run also prints a warning with the fixed destination candidate count and selected source bundle; destructive fixed replacements print an additional warning that the destination root would be replaced.

Use --format json when another process needs stable run data. JSON output includes warnings, pipeline summaries, destination actions, destination bundle paths, path mapping modes, optional link URLs, output records, final counters, and partial failure records. The summary includes fixed_path. If one destination fails after planning or execution begins, JSON output still contains the successful and failed destination records with ok: false, and the command exits non-zero.

Retry and Replacement Behavior

If a destination has matching .distributor.json, publication skips it as already published.

If destination state is older than the source manifest and transfer policy allows replacement, publication deletes only managed outputs recorded in .distributor.json plus the state file, then writes the new outputs and state.

If destination state is newer than the source manifest, the default behavior is to skip. If destination state has the same source id and created timestamp but a different digest, publication fails as a conflict.

If a destination path has files but no valid .distributor.json, publication fails as unmanaged content unless the current run explicitly uses --force.

Force Workflow

Use --force only after a dry run shows the intended force_replace action:

go run ./cmd/distributor run --config <config-path> --dry-run --force
go run ./cmd/distributor run --config <config-path> --force

Forced replacement can overwrite unmanaged non-empty destination paths. Destination state conflicts require transfer.on_conflict: replace plus --force. Newer destination state requires transfer.on_destination_newer: replace plus --force.

Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete above that bundle path. For fixed destinations, the destination bundle path is the backend root, so forced replacement may clear that configured root but not its parent path, sibling directories, or anything outside the configured S3 bucket and prefix. Force is per run only and has no config default.

Failure Handling

If one destination fails in a fan-out run, independent later destinations are still planned and executed. The command exits non-zero after printing the final status if any destination failed.

Errors include the pipeline id, destination id, destination backend, and bundle path where applicable.

In JSON mode, destination failures after planning or execution begins are reported in the top-level errors array and in the run result while preserving a non-zero exit code. Fatal setup errors such as an unreadable config or invalid secrets directory write no JSON document.

If a write fails during publication, distributor attempts to remove outputs written during that failed attempt so a retry does not see those partial outputs as unmanaged destination content.

After a successful publish or replacement, the internal notifier hook runs. The current default notifier is a no-op. Skipped destinations do not invoke it.

SSH Operation Notes

SSH execution uses SFTP over golang.org/x/crypto/ssh and github.com/pkg/sftp. It does not shell out to ssh, scp, or rsync.

Configure ssh_key_file, an SSH agent, or both. Agent identities are attempted first, followed by the configured key file. YAML password authentication is not supported.

The default host key policy is accept-new. New host keys are written to known_hosts when the file path is writable. During --dry-run, unknown host keys may be accepted for the current connection but are not written to known_hosts; a later non-dry-run may persist the same key. Changed host keys are fatal for both strict and accept-new. The off policy disables host key checking and run prints a warning when stdout is enabled.

Recovery boundaries are the same as local storage: replacement deletes only managed output paths recorded in .distributor.json plus the state file, and failed writes are cleaned up where practical. Distributor never performs broad recursive remote deletion.

S3 Operation Notes

S3 execution uses the AWS SDK for Go v2. Configure endpoint, bucket, optional prefix, optional region, and optional explicit credential environment variable names.

When explicit credential env names are configured, both variables must resolve to non-empty values through the real process environment or secrets.directory. When they are omitted, the AWS SDK default credential chain is used as-is.

Normal replacement and failed-write cleanup delete only managed output objects recorded in .distributor.json plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. For fixed destinations, that prefix is the configured bucket plus optional prefix. Distributor does not manage bucket versioning or delete markers.

Secrets Directory

Configure secrets.directory when credential values should come from mounted files, such as deployment secrets:

secrets:
  directory: /run/secrets/distributor

The directory is loaded during run, serve, and configured-source validate or inspect before credential-consuming work starts. If the directory is missing, unreadable, or contains an invalid secret filename, the command fails before storage work starts.

Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, run and configured-source diagnostics print a warning naming the ignored secret file variable without printing either value. The process environment is not changed.

Caveats

External notification adapters are unavailable. Force overwrite behavior is available only through the explicit run --force workflow.

For symptom-oriented fixes, see troubleshooting. For config details, see configuration. For command syntax, see CLI.

18 KiB Raw Blame History