Normalize internal component documentation

This commit is contained in:
2026-06-04 12:18:30 +00:00
parent a81f686fae
commit bed425ab78
10 changed files with 345 additions and 533 deletions

View File

@@ -1,46 +1,51 @@
# Ingestion Internals
Audience: developers and LLM coding agents changing `internal/ingest`.
## Purpose
`internal/ingest` stages uploaded source bundle archives into local per-run directories. It does not authenticate requests, manage upload queues, publish destinations, or start an HTTP server.
`internal/ingest` validates upload content types, extracts uploaded source bundle archives into local temporary storage, validates extracted bundles, and commits accepted bundles to per-run staging directories.
## Archive staging
## Inputs And Outputs
`ValidateContentType` owns accepted upload content-type policy for archive
staging callers.
Inputs are a context, upload body reader, content type, pipeline staging path, run id, maximum uploaded size, maximum extracted size, and maximum file count. Output is a `StagedBundle` containing the committed local bundle root and parsed manifest.
`StageArchive` accepts one upload body, content type, pipeline staging path, run id, and explicit size and file-count limits. It writes the request body to temporary storage while enforcing the configured upload size limit, extracts the archive into temporary local storage, validates the extracted source bundle, and then commits the validated bundle to:
## Boundaries
```text
<pipeline staging path>/<run id>
```
The package does not authenticate HTTP requests, manage upload queues, track upload status, publish destinations, load config, or start an HTTP server. Those responsibilities live in `internal/app`.
The returned `StagedBundle.Root` is a local filesystem path to the validated source bundle root.
The HTTP API contract is documented in `docs/integrations/http-upload.md`.
## Accepted archive formats
## Config Fields Used
The package accepts only:
The package does not read config directly. The app layer passes effective values derived from `source.staging_path`, `source.max_upload_size`, and HTTP server defaults.
- `application/x-tar`
- `application/gzip`
- `application/x-gzip`
## Adapters Used
Gzip uploads must contain a tar archive.
The package uses the local filesystem directly for temporary archive storage, extraction, validation, and final staging path promotion. It does not use the storage backend abstraction.
## Extraction rules
## State And Manifest Behavior
Archive entry paths must be clean relative slash-separated paths. Extraction rejects absolute paths, path traversal, backslash paths, duplicate files, symlinks, hardlinks, devices, sockets, and other special entries.
Accepted archives must contain exactly one root-level `manifest.json`. After extraction, the package validates the staged root through `pkg/bundle`, including manifest parsing, source path rules, file existence, regular-file checks, file sizes, file SHA-256 digests, and bundle digest.
The archive must contain exactly one root-level `manifest.json`. Nested manifests are rejected.
## Skip And Resume Behavior
Regular files and directories are the only accepted tar entries. Regular file extraction enforces the explicit maximum extracted byte count and maximum file count supplied by the caller.
The package has no resume behavior. A successful call commits one complete staged bundle root. Failed calls remove temporary data created by that call.
## Bundle validation
## Failure Behavior
After extraction, the package loads and validates the staged bundle through `pkg/bundle`. Manifest parsing, source path validation, file existence checks, regular-file checks, file sizes, file SHA-256 digests, and bundle digest validation use the existing source bundle contract.
Failures include unsupported content type, unsafe run id, missing staging path, non-positive limits, oversize upload body, oversize extracted content, too many files, unsafe archive paths, duplicate files, nested manifests, unsupported tar entry types, gzip/tar read errors, bundle validation errors, and filesystem errors.
Validation happens before the staged bundle is committed to its final per-run path.
## Tests To Inspect
## Failure behavior
- `internal/ingest/archive_test.go`
- `internal/app/upload_*_test.go`
- `pkg/bundle/*_test.go`
Failed staging removes temporary archive and extraction data created by the package. A failed call does not publish anything and does not leave a committed per-run bundle directory.
## Architectural Invariants
- Invalid archives never commit a staged root.
- Archive paths remain clean relative slash-separated paths.
- Only directories and regular files are accepted from tar archives.
- Source bundle validation happens before final staging path promotion.
- Upload authentication and queueing remain outside this package.