Blob storage and file uploads

Databases are great for rows, but bad for 5 MB photos and 2 GB videos: they bloat backups, slow replication and cost far more per GB. Big binary data (blobs) belongs in object storage.

Block, file and object storage

Object storage is a flat namespace: bucket/key → bytes + metadata. It scales almost without limit, is extremely durable, and is cheap, but objects are replaced whole rather than edited in place.

Uploading: pre-signed URLs

1. The client asks your API for permission to upload.
2. The API creates a metadata row (status = pending) and returns a pre-signed URL: a time-limited, signed URL that allows one PUT to one key.
3. The client uploads the bytes directly to the object store. Your servers never touch the file, so they don’t need the bandwidth or memory.
4. The store emits an event; a worker validates the file, generates thumbnails or transcodes it, and marks the record ready.

Downloads work the same way: a pre-signed GET URL, or a CDN in front of the bucket.

Large files: multipart upload

Split the file into parts (say 5–100 MB), upload them in parallel, and retry only failed parts; then call “complete”. Uploads can be resumed after network drops, which matters on mobile. Pair it with checksums to detect corruption.

Durability and cost

• Replication: several full copies across devices or zones. Simple, but 3× storage.
• Erasure coding: split data into k data chunks plus m parity chunks; any k of the k + m can rebuild it. For example 10 + 4 survives losing 4 chunks with only 1.4× overhead. This is how stores reach “eleven nines” of durability cheaply.
• Storage tiers: hot (frequent access) → infrequent → archive (cheap, slow retrieval). Lifecycle rules move objects down tiers automatically, for example “after 30 days, move to infrequent access”.
• Deduplication: store by content hash so identical files are kept once (see the Dropbox question).