52 lines
2.5 KiB
Markdown
52 lines
2.5 KiB
Markdown
# Ingestion Internals
|
|
|
|
Audience: developers and LLM coding agents changing `internal/ingest`.
|
|
|
|
## Purpose
|
|
|
|
`internal/ingest` validates upload content types, extracts uploaded source bundle archives into local temporary storage, validates extracted bundles, and commits accepted bundles to per-run staging directories.
|
|
|
|
## Inputs And Outputs
|
|
|
|
Inputs are a context, upload body reader, content type, pipeline staging path, run id, maximum uploaded size, maximum extracted size, and maximum file count. Output is a `StagedBundle` containing the committed local bundle root and parsed manifest.
|
|
|
|
## Boundaries
|
|
|
|
The package does not authenticate HTTP requests, manage upload queues, track upload status, publish destinations, load config, or start an HTTP server. Those responsibilities live in `internal/app`.
|
|
|
|
The HTTP API contract is documented in `docs/integrations/http-upload.md`.
|
|
|
|
## Config Fields Used
|
|
|
|
The package does not read config directly. The app layer passes effective values derived from `source.staging_path`, `source.max_upload_size`, and HTTP server defaults.
|
|
|
|
## Adapters Used
|
|
|
|
The package uses the local filesystem directly for temporary archive storage, extraction, validation, and final staging path promotion. It does not use the storage backend abstraction.
|
|
|
|
## State And Manifest Behavior
|
|
|
|
Accepted archives must contain exactly one root-level `manifest.json`. After extraction, the package validates the staged root through `pkg/bundle`, including manifest parsing, source path rules, file existence, regular-file checks, file sizes, file SHA-256 digests, and bundle digest.
|
|
|
|
## Skip And Resume Behavior
|
|
|
|
The package has no resume behavior. A successful call commits one complete staged bundle root. Failed calls remove temporary data created by that call.
|
|
|
|
## Failure Behavior
|
|
|
|
Failures include unsupported content type, unsafe run id, missing staging path, non-positive limits, oversize upload body, oversize extracted content, too many files, unsafe archive paths, duplicate files, nested manifests, unsupported tar entry types, gzip/tar read errors, bundle validation errors, and filesystem errors.
|
|
|
|
## Tests To Inspect
|
|
|
|
- `internal/ingest/archive_test.go`
|
|
- `internal/app/upload_*_test.go`
|
|
- `pkg/bundle/*_test.go`
|
|
|
|
## Architectural Invariants
|
|
|
|
- Invalid archives never commit a staged root.
|
|
- Archive paths remain clean relative slash-separated paths.
|
|
- Only directories and regular files are accepted from tar archives.
|
|
- Source bundle validation happens before final staging path promotion.
|
|
- Upload authentication and queueing remain outside this package.
|