distributor/docs/internal/ingest.md

Ingestion Internals

Audience: developers and LLM coding agents changing internal/ingest.

Purpose

internal/ingest validates upload content types, extracts uploaded source bundle archives into local temporary storage, validates extracted bundles, and commits accepted bundles to per-run staging directories.

Inputs And Outputs

Inputs are a context, upload body reader, content type, pipeline staging path, run id, maximum uploaded size, maximum extracted size, and maximum file count. Output is a StagedBundle containing the committed local bundle root and parsed manifest.

Boundaries

The package does not authenticate HTTP requests, manage upload queues, track upload status, publish destinations, load config, or start an HTTP server. Those responsibilities live in internal/app.

The HTTP API contract is documented in docs/integrations/http-upload.md.

Config Fields Used

The package does not read config directly. The app layer passes effective values derived from source.staging_path, source.max_upload_size, and HTTP server defaults.

Adapters Used

The package uses the local filesystem directly for temporary archive storage, extraction, validation, and final staging path promotion. It does not use the storage backend abstraction.

State And Manifest Behavior

Accepted archives must contain exactly one root-level manifest.json. After extraction, the package validates the staged root through pkg/bundle, including manifest parsing, source path rules, file existence, regular-file checks, file sizes, file SHA-256 digests, and bundle digest.

Skip And Resume Behavior

The package has no resume behavior. A successful call commits one complete staged bundle root. Failed calls remove temporary data created by that call.

Failure Behavior

Failures include unsupported content type, unsafe run id, missing staging path, non-positive limits, oversize upload body, oversize extracted content, too many files, unsafe archive paths, duplicate files, nested manifests, unsupported tar entry types, gzip/tar read errors, bundle validation errors, and filesystem errors.

Tests To Inspect

• internal/ingest/archive_test.go
• internal/app/upload_*_test.go
• pkg/bundle/*_test.go

Architectural Invariants

• Invalid archives never commit a staged root.
• Archive paths remain clean relative slash-separated paths.
• Only directories and regular files are accepted from tar archives.
• Source bundle validation happens before final staging path promotion.
• Upload authentication and queueing remain outside this package.