Step 9: Scalability and Performance

Problem 1: The tables are too big for one machine

The reactions table grows to hundreds of billions of rows. One database server cannot hold that many, so we split the table across many servers. This is called sharding, and a field in each row, the shard key, decides which server gets that row.

The choice of shard key matters. If we split the reactions table by video, every reaction to XYZ1 goes to the same server, and that server receives all 50,000 writes a second while the others wait. If we split it by user instead, the millions of people reacting to XYZ1 are spread across all the servers, and so is the load. When Alice clicks like, the service still finds her earlier reaction with one lookup, because all of Alice's rows are on the same server.

The totals table is split by video, because each video's total stands alone. The server that holds XYZ1 gets only a little extra traffic, because the worker from Step 7 collects thousands of reactions and writes one new total every second or two.

The worker overwrites the total with a number it has computed. It does not use a Cassandra counter column, which adds one each time it is called. If a network timeout made the driver send the same counter update twice, the count would go up twice, and nothing would report the mistake.

Problem 2: One count lives on one cache server

Sharding spreads different videos across servers, but the count for XYZ1 is still one entry in the cache, on one server. At 50,000 updates a second, that server cannot keep up. An entry with more traffic than one server can handle is called a hot key.

The fix is to split the one count into ten smaller counts, video:XYZ1:0 to video:XYZ1:9. Each new like adds one to a piece chosen at random, so each piece receives about 5,000 updates a second. To show the count, the service reads all ten pieces in one request and adds them up. Every read now does ten lookups, so we split only the videos that are actually hot.

Problem 3: Every page asks for many counts

A watch page shows the count for the video and for about twenty comments. Across the site, that comes to about 250,000 count reads a second.

The batch endpoint from Step 4 asks for all 21 counts in one request, so 250,000 reads become about 11,500 requests a second. The page also needs to know which buttons this viewer has already pressed. Because the reactions table is split by user, all of this viewer's reactions sit on one server and come back in one read.

Fake likes

Counts affect ranking, so some people will try to inflate them. A per-minute limit for each account stops one account from sending thousands of reactions. Because every reaction is stored with its time, a later job can find groups of accounts that always like the same videos and recount the total without them.

💡 In the interview: Many answers stop at sharding. Explain that sharding does not help one hot cache entry, describe splitting it into ten pieces, and show how one batched request per page cuts the read load.