Designing Instagram

Every case study in this chapter follows the same seven steps from System Design Interviews: A step by step guide, and builds its design from the components in the System Design Master Template. Keep both open beside you, and try each problem yourself before reading the solution.

What is Instagram?

Instagram is a social networking service. Users upload photos and videos and share them with other users. A user can share content publicly or privately. Private content is visible only to a chosen set of people.

We will design a simpler version of Instagram. In our version, a user can upload photos and follow other users. The system builds a News Feed for each user. The News Feed shows the top photos from all the people the user follows.

Try it yourself

Before reading the solution, try designing it. Aim to produce four things: the requirements, a rough load estimate, the API, and a first drawing of the components.

Sketch it here

Designing Instagram (video)

Here is a video discussing how to design Instagram:

Step 1: Clarify the Requirements

💡 Always clarify requirements at the beginning of the interview. Ask questions until you know the exact scope of the system the interviewer has in mind.

Functional requirements

1. Users can upload, download, and view photos.
2. Users can search photos by title.
3. Users can follow other users.
4. The system generates a News Feed for each user. The feed contains top photos from all the people the user follows.

Non-functional requirements

1. High availability. The service must stay up, even when servers fail.
2. Low latency on the feed. News Feed generation should take at most 200ms.
3. Consistency can be relaxed. If a user does not see a new photo for a while, that is fine. We accept this in exchange for availability.
4. High durability. Once a photo or video is uploaded successfully, it must never be lost.

Not in scope: adding tags to photos, searching photos by tags, commenting on photos, tagging users in photos, and follow suggestions.

Two observations shape the whole design. The system is read-heavy. Users view photos far more often than they upload them. So the design must retrieve photos quickly. Also, users can upload as many photos as they like. So the system must manage storage efficiently.

Step 2: Estimate the Scale

Ask the interviewer for the size of the user base and the upload rate. Assume 500 million total users, with 1 million daily active users. Assume 2 million new photos per day, with an average photo file size of 200 KB.

Load.

2M / 86,400 seconds           = ~23 new photos per second

Photo storage.

2M x 200KB                    = 400 GB per day
400 GB x 365 days x 10 years  = ~1425 TB

One number changes the design: 1425 TB. Photo files are far too big to store as database rows. They need their own storage, which step 4 picks. Everything else about the load is modest.

Step 3: Define the API

Five endpoints cover the four functional requirements.

POST /photos
  body: { user_id, photo_data, title }
  201:  { photo_id, photo_url }

GET /photos/{photo_id}
  200:  the photo file

GET /photos/search?query=<title>
  200:  a list of matching photos

POST /follow
  body: { follower_id, followee_id }
  204:  no content

GET /feed/{user_id}
  200:  the top photos from the people the user follows

The feed endpoint has the 200ms latency target. Step 6 shows how that feed is built.

Step 4: Define the Data Model

The data splits into two kinds with very different shapes.

1. Photo files. Large binary objects, about 200 KB each. A file is written once, read many times, and never edited.
2. Metadata. Small rows that describe users, photos, and follow relationships.

That split decides the storage. Photo files go into a distributed file storage system like HDFS or S3. The metadata goes into database tables: User, Photo, and UserFollow.

One read pattern matters most: fetch the latest photos from the people a user follows. That query will drive the schema, the photo IDs, and the sharding.

💡 Defining the database schema early in the interview helps you understand the data flow between the parts of the system. It also guides the data partitioning discussion later.

We store data about users, their uploaded photos, and the people they follow. Three tables cover it: User, Photo, and UserFollow.

Photo holds one row per uploaded photo. PhotoPath is the path to the file in object storage; the 200 KB file itself never enters a database row. The location fields and the creation date complete the row. The table needs an index on (PhotoID, CreationDate), because the News Feed fetches recent photos first.

UserFollow stores the follow relationship as a pair of UserIDs: the follower and the followee. User holds one row per account.

The three tables look like this:

How big should PhotoID be? A 32-bit int holds about 2.1 billion values, and we expect far more photos than that:

2M x 365 days x 10 years      = ~7.3 billion photos

So PhotoID must be a bigint, a 64-bit integer column, which takes 8 bytes. UserID can stay a 4-byte int, because 500 million users fit comfortably.

Which database? A straightforward choice is a relational database like MySQL, since we need joins between users, photos, and follows. But relational databases are hard to scale to this size. The SQL vs. NoSQL lesson covers that trade in detail.

The alternative is a distributed key-value store, a NoSQL database that maps one key to one value. All photo metadata can go into one table there. The key is the PhotoID. The value is an object holding PhotoLocation, UserLocation, CreationTimestamp, and similar fields.

NoSQL stores scale across many machines easily. They also keep a set number of replicas of every row, which gives durability without extra work. Deletes are not applied instantly. A deleted row is retained for a certain number of days to support undeleting, then removed permanently.

Step 5: Draw the High-Level Design

At a high level, we support two scenarios: uploading photos, and viewing or searching photos. So our service needs object storage servers to store the photo files. It also needs database servers to store the metadata.

Read the diagram as two paths. An upload writes the photo file to object storage and the photo's metadata to the database. A view or search reads metadata from the database, then fetches the photo file from object storage.

Step 6: Go Deep

How big is the metadata?

Let's size each table for ten years. Assume each int and dateTime field takes four bytes.

User. Each row is 68 bytes:

UserID (4 bytes) + Name (20 bytes) + Email (32 bytes) + DateOfBirth (4 bytes)
+ CreationDate (4 bytes) + LastLogin (4 bytes) = 68 bytes

500 million x 68 bytes        = ~32 GB

Photo. With the 8-byte bigint PhotoID, each row is 288 bytes. The 256-byte path to the file in object storage takes most of that:

PhotoID (8 bytes) + UserID (4 bytes) + PhotoPath (256 bytes)
+ PhotoLatitude (4 bytes) + PhotoLongitude (4 bytes)
+ UserLatitude (4 bytes) + UserLongitude (4 bytes)
+ CreationDate (4 bytes) = 288 bytes

2M x 288 bytes                        = ~0.6 GB per day
2M x 288 bytes x 365 days x 10 years  = ~1.9 TB

UserFollow. A follow relationship is just two 4-byte UserIDs, so each row is 8 bytes. Assume each user follows 500 people on average:

500 million users x 500 follows x 8 bytes  = ~1.82 TB

Total metadata for ten years:

32 GB + 1.9 TB + 1.82 TB      = ~3.8 TB

Notice the gap: about 3.8 TB of metadata against 1425 TB of photo files. The files are the real storage problem, and object storage holds them.

Separate reads from writes

Photo uploads (writes) are slow, because they have to go to the disk. Reads are fast, especially when they are served from a cache.

Slow uploads create a second problem: connections. Web servers have a connection limit. Assume a server can hold at most 500 concurrent connections. Each upload is slow, so it occupies its connection for a long time. If uploads fill all 500 connections, the server cannot serve any reads.

The fix is to split reads and writes into separate services. Dedicated servers handle uploads. Different dedicated servers handle reads. Now a burst of slow uploads can never block photo views. The split also lets us scale and optimize each side independently.

The diagram below shows the split:

💡 In the interview: spend your time on the read and write split, the sharding choice, and feed generation. Two moments give the interviewer the most information. The first is when you name the hot-user problem and switch from UserID sharding to PhotoID sharding. The second is when you name the celebrity fan-out problem and propose the hybrid push and pull model. Both show the same skill: you find the failure in your own first answer. If you are asked what breaks first, say feed staleness, and name the consistency trade you agreed on in step 1.

Key takeaway: Instagram is a read-heavy system with two kinds of data. Photo files are large and never change, so they belong in replicated object storage. Metadata rows are small, so they belong in a sharded database. Separate read services from write services, so slow uploads never block fast views. Shard by PhotoID rather than UserID to avoid hot users. Make PhotoIDs a 64-bit bigint that starts with epoch time, so the newest photos sort first by primary index. Pre-generate each feed into a UserNewsFeed table. Push updates for ordinary accounts, and pull posts of very high-follower accounts at feed build time. Cache the popular 20 percent of photos and metadata, evicting with LRU.